U.

Washington AMATH 352 - Spring 2009
Applied Linear Algebra and Numerical Analysis
AMATH 352 - Spring 2009
Prof. U. Hetmaniuk
University of Washington
The course goals are to understand the basic concepts of linear algebra and
to obtain an introduction to some aspects of computational techniques used in
matrix methods. We will study basic concepts in linear algebra, including vec-
tors, vector spaces, linear transformations, matrix-vector manipulations, solving
linear systems, least squares problems, and eigenvalue problems. Matrix decom-
positions (e.g. LU, QR, SVD, etc.) will play a fundamental role throughout the
course. The emphasis will be on practical aspects of linear algebra and nu-
merical methods for solving these problems. Such problems arise constantly in
science, engineering, finance, computer graphics.
Professor Randall J. LeVeque (U. Washington) created the notes. He, kindly,
allowed me to modify them. All the mistakes and typos are mine.
1
U. Washington AMATH 352 - Spring 2009
1 Column Vectors
In geometry, the two-dimensional plane is denoted R
2
and the three-dimensional
space R
3
. A vector is an object comprised of a magnitude and a direction. In
the plane, we can draw a vector as an arrow with some length and pointing
somewhere . A vector can also be thought of as a displacement. A displacement
does not depend where it starts. Consequently, the vectors are equal, even
though they start from different places, because they have equal length and equal
direction. The basic idea here, combining magnitude with direction, is the key
to extending to higher dimensions. In this section, we define the generalization
of vectors in the two-dimensional plane and three-dimensional space.
Let m be a positive integer. We denote R
m
the set of all real m-tuples, i.e.
the set of all sequences with m components, each of which is a real number.
The standard notation for an element x of R
m
is the column vector notation:
∀x ∈ R
m
⇐⇒x =
_
¸
_
x
1
.
.
.
x
m
_
¸
_. (1)
It is important to remember that, in many applications of linear algebra, the
elements of the vector represent something different from the three physical co-
ordinates of ordinary space. There is often nothing unphysical about considering
vectors with many more than 3 components.
Example. We have
_
−1
3
_
∈ R
2
,
_
_

7
0

3
_
_
∈ R
3
, and
_
¸
¸
_
1
−2/5
−3/5
4
_
¸
¸
_
∈ R
4
.
2
U. Washington AMATH 352 - Spring 2009
1.1 Addition of column vectors
We can define the addition of two vectors x and y of R
m
:
x =
_
¸
_
x
1
.
.
.
x
m
_
¸
_ y =
_
¸
_
y
1
.
.
.
y
m
_
¸
_, x+y =
_
¸
_
x
1
+ y
1
.
.
.
x
m
+ y
m
_
¸
_. (2)
The set R
m
is closed under addition, meaning that whenever the addition is
applied to vectors in R
m
, we obtain another vector in the same set R
m
. For
example, we have
_
_
1
2
3
_
_
+
_
_
2
−4
8
_
_
=
_
_
1 + 2
2 −4
3 + 8
_
_
=
_
_
3
−2
11
_
_
.
Exercise 1. Prove that the addition is associative: (x +y) +z = x + (y +z).
Exercise 2. Prove that the addition is commutative: x +y = y +x.
Exercise 3. Find the unique vector z such that x +z = x, for any vector x. z
is called the zero vector and, sometimes, denoted 0.
Exercise 4. Show that every vector x ∈ R
m
has an additive inverse y ∈ R
m
such that x +y = 0.
1.2 Scalar multiplication of a column vector
We can also define the scalar multiplication: if x ∈ R
m
and α ∈ R, then the
vector αx belongs to R
m
and is defined by multiplying each component of x by
the scalar α:
x =
_
¸
_
x
1
.
.
.
x
m
_
¸
_ α ∈ R, αx =
_
¸
_
αx
1
.
.
.
αx
m
_
¸
_. (3)
The set R
m
is also closed under scalar multiplication. For example, we have
2
_
_
1
2
3
_
_
=
_
_
2 1
2 2
2 3
_
_
=
_
_
2
4
6
_
_
, (−3)
_
_
2
−4
8
_
_
=
_
_
(−3) 2
(−3) (−4)
(−3) 8
_
_
=
_
_
−6
12
−24
_
_
.
Exercise 5. Prove that ∀ α, β ∈ R, α(βx) = (αβ)x.
Exercise 6. Prove that ∀α ∈ R, ∀ x, y ∈ R
m
, α(x +y) = αx + αy.
Exercise 7. Prove that ∀α, β ∈ R, ∀ x ∈ R
m
, (α + β)x = αx + βx.
Exercise 8. Show that, for every vector x ∈ R
m
, 1x = x.
3
U. Washington AMATH 352 - Spring 2009
1.3 Norms
To measure the magnitude or the length of a vector, we use a vector norm. A
vector norm is simply a map from vectors in R
m
to nonnegative real numbers,
satisfying the following conditions (which generalize important properties of the
absolute value for scalars):
∀x ∈ R
m
, |x| ≥ 0, (4a)
|x| = 0 if and only if x = 0, (4b)
∀α ∈ R, |αx| = [α[ |x| , (4c)
∀x, y ∈ R
m
, |x +y| ≤ |x| +|y| (Triangle inequality). (4d)
Note that a norm satisfies the following properties
|−x| = |x| and |x +x| = |2x| = 2 |x| .
The triangle inequality can not be an equality because we have, for any non-zero
vector x,
0 = |x −x| < |x| +|−x| = 2 |x| .
One common choice is the max-norm (or infinity-norm) denoted by
|x|

= max
1≤i≤m
[x
i
[ . (5)
A bound on the max-norm of the error is nice because we know that every
component of the error can be no greater than the max-norm,
∀ i, 1 ≤ i ≤ m, [x
i
[ ≤ |x|

.
It is easy to verify that ||

satisfies the required properties (4).
• ∀x ∈ R
m
, |x|

≥ 0 because the absolute value is always greater or equal
to zero.
• |x|

= 0 if and only if x = 0,
– If x = 0, then all the components of x are zero and, consequently,
|x|

= 0.
– If |x|

= 0, then every component x
i
will be zero. This implies that
the vector x is 0.
• ∀α ∈ R, |αx|

= [α[ |x|

,
– Note that, for every entry i, we have
[αx
i
[ = [α[ [x
i
[ .
4
U. Washington AMATH 352 - Spring 2009
We get
max
1≤i≤m
[αx
i
[ = max
1≤i≤m
[α[ [x
i
[ = [α[ max
1≤i≤m
[x
i
[
since we are multiplying every entry by the same factor [α[. We
obtain
|αx|

= [α[ |x|

• ∀x, y ∈ R
m
, |x +y|

≤ |x|

+|y|

– The absolute value satisfies a triangle inequality. Consequently, we
have, for every i,
[x
i
+ y
i
[ ≤ [x
i
[ +[y
i
[ ≤ |x|

+|y|

.
So every entry in the vector x +y is smaller than |x|

+|y|

. In
particular, the largest entry will also satisfies that. So we get
max
1≤i≤m
[x
i
+ y
i
[ = |x +y|

≤ |x|

+|y|

For some problems, however, there are other norms which are either more ap-
propriate or easier to bound using our analytical tools. The 2-norm is frequently
used,
|x|
2
=
¸
¸
¸
_
m

i=1
[x
i
[
2
. (6)
The 2-norm is often called the Euclidean norm. The 1-norm is defined as follows
|x|
1
=
m

i=1
[x
i
[ . (7)
The 1-norm is also known as the Manhattan norm because it corresponds to
the distance traveled on a grid of city streets.
These norms are special cases of the general family of p-norms, defined by
|x|
p
=
_
m

i=1
[x
i
[
p
_1
p
. (8)
Note that the max-norm can be obtained as the limit as p →+∞of the p-norm.
For example, we have
_
_
_
_
_
_
_
_
1
2
3
_
_
_
_
_
_
_
_

= 3,
_
_
_
_
_
_
_
_
1
2
3
_
_
_
_
_
_
_
_
1
= 6,
_
_
_
_
_
_
_
_
1
2
3
_
_
_
_
_
_
_
_
2
=

14,
and
_
_
_
_
_
_
_
_
−2
4
−8
_
_
_
_
_
_
_
_

= 8,
_
_
_
_
_
_
_
_
−2
4
−8
_
_
_
_
_
_
_
_
1
= 14,
_
_
_
_
_
_
_
_
−2
4
−8
_
_
_
_
_
_
_
_
2
=

84.
5
U. Washington AMATH 352 - Spring 2009
Exercise 9. Check whether the following maps, defined on R
3
, are norms or
not
• x −→x
1
+ x
2
+ x
3
• x −→[x
1
+ x
2
+ x
3
[
• x −→
_
x
4
1
+ x
4
2
+ x
4
3
Exercise 10. Prove that the p-norm satisfies the properties (4).
The closed unit ball ¦x ∈ R
m
[ |x| ≤ 1¦ is the set of all vectors with a norm
smaller than 1. The shape of this ball depends on the norm. The unit circle (or
sphere) is the set of all vectors with a norm equal to 1,
S
1
= ¦x ∈ R
m
[ |x| = 1¦. (9)
In R
2
, we can draw the unit circle as a curve composed of the points (x, y) such
that _
_
_
_
_
x
y
__
_
_
_
= 1.
There exists an infinite number of points on this curve. For example, for the
Euclidian norm, the unit circle contains the vectors
_
1
0
_
,
_
0
1
_
,
_ √
2
/2

2
/2
_
,
_
1
/2


3
/2
_
, . . .
The equation governing the unit circle for the Euclidian norm is
_
_
_
_
_
x
y
__
_
_
_
2
= 1 ⇒
_
x
2
+ y
2
= 1.
Several ways are possible to draw this curve. Among them, we can parametrize
x and y as follows
x = cos(θ)
y = sin(θ)
where θ belongs to [0, 2π]. Figure 1 illustrates the unit circle for the Euclidian
norm. The equation governing the unit circle for the 1-norm is
_
_
_
_
_
x
y
__
_
_
_
1
= 1 ⇒ [x[ +[y[ = 1.
The curve is included in the square [−1, 1] [−1, 1] and it is composed of the
following four branches:
x + y = 1 0 ≤ x, y ≤ 1
x −y = 1 x ∈ [0, 1] and y ∈ [−1, 0]
−x −y = 1 x, y ∈ [−1, 0]
−x + y = 1 x ∈ [−1, 0] and y ∈ [0, 1]
Figure 2 illustrates the unit circle for the 1-norm.
6
U. Washington AMATH 352 - Spring 2009
Figure 1: Unit circle for the Euclidian norm.
Figure 2: Unit circle for the Euclidian norm.
Exercise 11. Draw the unit closed ball ¦x ∈ R
2
[ |x| ≤ 1¦ corresponding to
the 4-norm and ∞-norm.
Trefethen and Bau
1
note that
“The Sergel plaza in Stockholm, Sweden, has the shape of the unit
ball in the 4-norm. The Danis poet Piet Hein popularized this su-
perellipse as a pleasing shape for objects such as conference tables.”
Other useful norms include the weighted p-norms, where each component of
a vector is weighted. For example, a weighted 2-norm can be specified as follows
|x| =
¸
¸
¸
_
m

i=1
w
i
[x
i
[
2
(10)
where the weights w
i
are strictly positive real numbers.
1
L. N. Trefethen and D. Bau, Numerical linear algebra, SIAM, Philadelphia, 1997.
7
U. Washington AMATH 352 - Spring 2009
Exercise 12. Show that the map
_
[x
1
[
2
+ 3 [x
2
[
2
is a norm. Draw its closed
unit ball.
Exercise 13. Show that the map
_
¸
¸
x
1


3x
2
¸
¸
2
+
¸
¸

3x
1
+ x
2
¸
¸
2
is a norm.
Draw its closed unit ball.
For a given non-zero vector, its length depends on the norm chosen to mea-
sure it. However, in R
m
, all the norms are related. For example, we have
|x|

≤ |x|
2


m|x|

. (11)
In R
3
, consider the vector
x =
_
_
1
−8
2
_
_
.
Then we have
_
_
_
_
_
_
_
_
1
−8
2
_
_
_
_
_
_
_
_

= 8 =

8
2

_
1
2
+ (−8)
2
+ 2
2
=
_
_
_
_
_
_
_
_
1
−8
2
_
_
_
_
_
_
_
_
2

_
8
2
+ 8
2
+ 8
2
=

3
_
_
_
_
_
_
_
_
1
−8
2
_
_
_
_
_
_
_
_

(12)
where the first inequality is due to the fact that we add positive numbers to 8
2
.
The second inequality comes from the fact that any component is smaller than
8. To extend this proof to R
m
, we assume that the max-norm is reached at the
component I. Then we have
|x|

= [x
I
[ =
_
[x
I
[
2

¸
¸
¸
_
_
I−1

i=1
[x
i
[
2
_
+[x
I
[
2
+
_
m

i=I+1
[x
i
[
2
_
= |x|
2

_
[x
I
[
2
+ +[x
I
[
2
=

m|x|

. (13)
Note that these inequalities are sharp because they are attained with the fol-
lowing vectors
_
_
_
_
_
_
_
_
_
_
¸
¸
¸
_
1
0
.
.
.
0
_
¸
¸
¸
_
_
_
_
_
_
_
_
_
_

=
_
_
_
_
_
_
_
_
_
_
¸
¸
¸
_
1
0
.
.
.
0
_
¸
¸
¸
_
_
_
_
_
_
_
_
_
_
2
= 1 and
_
_
_
_
_
_
_
_
¸
_
1
.
.
.
1
_
¸
_
_
_
_
_
_
_
_

= 1,
_
_
_
_
_
_
_
_
¸
_
1
.
.
.
1
_
¸
_
_
_
_
_
_
_
_
2
=

m.
Exercise 14. Prove that |x|

≤ |x|
1
≤ m|x|

. Find vectors with equality.
Exercise 15. Prove that
1

m
|x|
1
≤ |x|
2
≤ |x|
1
. Find vectors with equality.
8
U. Washington AMATH 352 - Spring 2009
1.4 Inner product
The Euclidean inner product of two column vectors in R
m
is defined as
x y =
m

i=1
x
i
y
i
. (14)
It is a map into real numbers. Note that x x ≥ 0. The Euclidean norm can be
defined as the square root of the inner product of x with itself
|x|
2
=

x x =
¸
¸
¸
_
m

i=1
[x
i
[
2
. (15)
The inner product of two vectors is bounded by the product of their lengths.
This inequality is the Cauchy-Schwarz inequality. It is one of the most important
inequalities in mathematics. For any vectors x and y of R
m
, we write
[x y[ ≤ |x|
2
|y|
2
, (16)
with equality if and only if y is proportional to x. For example, in R
2
, we have
(x y)
2
= (x
1
y
1
+ x
2
y
2
)
2
= x
2
1
y
2
1
+ x
2
2
y
2
2
+ 2x
1
x
2
y
1
y
2
.
Then we exploit the very useful result
2αβ ≤ α
2
+ β
2
=⇒ 2x
1
x
2
y
1
y
2
≤ x
2
1
y
2
2
+ x
2
2
y
2
1
to obtain
(x y)
2
≤ (x
2
1
+ x
2
2
)(y
2
1
+ y
2
2
) = |x|
2
2
|y|
2
2
.
The proof extends to R
m
.
Another useful inequality is the Hölder inequality
[x y[ ≤ |x|
p
|y|
q
, (17)
where p and q satisfy 1/p + 1/q = 1 and 1 ≤ p, q ≤ ∞.
Inner products allow the rigorous introduction of intuitive geometrical no-
tions such as the length of a vector or the angle between two vectors. It also
provides the means of defining orthogonality between vectors (zero inner prod-
uct). We will generalize its definition later in the notes.
The cosine of the angle θ between two column vectors is expressed in terms
of the inner product (14)
cos θ =
x y
|x|
2
|y|
2
. (18)
Note that, because of the Cauchy-Schwarz inequality (16), the right hand side
is always in the range [−1, 1]. The angle θ depends on the inner product chosen.
For the same two column vectors, two different inner products can give two
different angle θ.
9
U. Washington AMATH 352 - Spring 2009
Another important concept in linear algebra is the idea of two vectors being
orthogonal to one another, which is a generalization of “perpendicular”. We say
that x, y ∈ R
m
are orthogonal for the inner product (14) when x y = 0, i.e.
their inner product is 0. In R
2
or R
3
, two vectors are orthogonal if and only if
the lines drawn from the origin to the points with coordinates defined by x and
y are perpendicular to one another.
Example 16. Consider the vectors
_
1
3
_
and
_
2
−2
_
.
For the Euclidian inner product, the angle between the two vectors is
θ = arccos
_
1 2 + 3 (−2)

1
2
+ 3
2

2
2
+ 2
2
_
= arccos
_
−4

10

8
_
= arccos
_
−1

5
_
= 2.0344,
approximately 116 degrees.
Example 17. The vectors
x =
_
1
3
_
y =
_
−3
1
_
are orthogonal,
x y = 1 (−3) + 3 1 = −3 + 3 = 0.
So are the vectors
x =
_
¸
¸
_
1
3
−2
4
_
¸
¸
_
y =
_
¸
¸
_
4
1
4
1/4
_
¸
¸
_
.
Indeed, we have
x y = 1 4 + 3 1 + (−2) 4 + 4 = 4 + 3 −8 + 1 = 0.
Exercise 18. Compute the angle between the following pairs of vectors
__
1
0
_
,
_
−1
0
__
,
__
1
0
_
,
_
1
1
__
,
__
1

3
_
,
_
−1
0
__
.
1.5 Useful commands in Matlab
Here are a few commands in Matlab useful for this section.
• x = [1;2;3]; defines the vector x ∈ R
3
with components 1, 2, and 3.
• z = zeros(m,1) defines the zero vector in R
m
.
• x = ones(m,1) defines the vector x ∈ R
m
with components equal to 1.
10
U. Washington AMATH 352 - Spring 2009
• size(x) returns the dimensions of x in a row vector [m 1].
• x = rand(m,1) makes a vector of length m with random values, uniformly
distributed between 0 and 1.
• x = randn(m,1) makes a vector of length m with random values, normally
distributed (with mean 0 and variance 1).
• norm(x) computes the 2-norm of vector x. norm(x,inf), norm(x,1),
norm(x,p) computes, respectively, the ∞-norm, the 1-norm, and the p-
norm.
• dot(x,y) computes the Euclidian inner product between vectors x and y.
• max([1;-4;3]) computes the maximum entry in the vector (here 3).
• min([1;-4;3]) computes the minimum entry in the vector (here -4).
• The next sequence generates a vector x with 1000 components linearly
distributed between 0 and 5
x = [];
for i=1:1000,
x = [x; (i-1)*5.0/1000.0];
end
Whenever possible, it is recommended to declare x with its final size as
follows
x = zeros(1000,1);
for i=1:1000,
x(i) = (i-1)*5.0/1000.0;
end
11
U. Washington AMATH 352 - Spring 2009
2 Linear Spaces
Recall that R
m
denotes the set of all real m-vectors, i.e. the set of all vectors
with m components, each of which is a real number. We have also defined what
we mean by addition of two vectors x, y ∈ R
m
: we obtain the sum by adding
each component, and the sum x + y is another vector in R
m
. We have also
defined scalar multiplication: if x ∈ R
m
and α ∈ R then the vector αx ∈ R
m
is defined by multiplying each component of x by the scalar α. The set R
m
is
closed under addition and scalar multiplication. This just means that whenever
these operations are applied to vectors in R
m
we obtain another vector in the
same set R
m
.
The set R
m
is an example of linear space, which can be defined more generally
as follows:
Definition 19. A real linear space, or R-linear space, consists of a set of objects
V along with two operations ‘+’ (addition) and ‘’ (scalar multiplication) subject
to these conditions:
1. If u, v ∈ V then u + v ∈ V (closed under addition);
2. If u, v ∈ V then u + v = v + u (addition is commutative);
3. If u, v, w ∈ V then (u + v) + w = u + (v + w) (addition is associative);
4. There is a zero vector
¯
0 ∈ V such that v +
¯
0 = v for every v ∈ V ;
5. Every v ∈ V has an additive inverse w ∈ V such that v + w =
¯
0;
6. If v ∈ V and α ∈ R then α v ∈ V (closed under scalar multiplication);
7. If v ∈ V and α, β ∈ R then (α + β) v = α v + β v;
8. If u, v ∈ V and α ∈ R then α (u + v) = α u + α v;
9. If v ∈ V and α, β ∈ R then (αβ) v = α (β v);
10. If v ∈ V then 1 v = v.
It is also possible to define a complex linear space or C-linear space where the
scalars are now complex.
Example 20. Verify the properties of Definition 19 for the set of column vectors
R
m
.
1. For any vector x and any vector y in R
m
, we have
x =
_
¸
_
x
1
.
.
.
x
m
_
¸
_ y =
_
¸
_
y
1
.
.
.
y
m
_
¸
_ ⇒x +y =
_
¸
_
x
1
+ y
1
.
.
.
x
m
+ y
m
_
¸
_.
So the vector x + y is also a column vector with m rows and belongs to
R
m
.
12
U. Washington AMATH 352 - Spring 2009
2. For any vector x and any vector y in R
m
, the vectors x+y and y +x are
equal. Indeed, we have
x +y =
_
¸
_
x
1
+ y
1
.
.
.
x
m
+ y
m
_
¸
_ y +x =
_
¸
_
y
1
+ x
1
.
.
.
y
m
+ x
m
_
¸
_
and the m components are equal because the addition of scalar numbers
is commutative.
3. The addition of column vectors is associative. Indeed, we have
(x+y)+z =
_
¸
_
(x
1
+ y
1
) + z
1
.
.
.
(x
m
+ y
m
) + z
m
_
¸
_ x+(y+z) =
_
¸
_
x
1
+ (y
1
+ z
1
)
.
.
.
x
m
+ (y
m
+ z
m
)
_
¸
_.
The m components are equal because the addition of scalar numbers is
associative.
4. The zero vector is the vector with all its m components equal to 0,
0 =
_
¸
_
0
.
.
.
0
_
¸
_.
5. For any vector x in R
m
, the vector y, defined by
x =
_
¸
_
x
1
.
.
.
x
m
_
¸
_ y =
_
¸
_
−x
1
.
.
.
−x
m
_
¸
_,
is such that x +y = 0. The vector y is the additive inverse of x.
6. For any vector x in R
m
and any scalar α in R, we have
x =
_
¸
_
x
1
.
.
.
x
m
_
¸
_ ⇒αx =
_
¸
_
αx
1
.
.
.
αx
m
_
¸
_.
So the vector αx is also a column vector with m rows and belongs to R
m
.
7. For any vector x in R
m
and any scalar α and β in R, we have
(α+β)x =
_
¸
_
(α + β)x
1
.
.
.
(α + β)x
m
_
¸
_ =
_
¸
_
αx
1
+ βx
1
.
.
.
αx
m
+ βx
m
_
¸
_ =
_
¸
_
αx
1
.
.
.
αx
m
_
¸
_+
_
¸
_
βx
1
.
.
.
βx
m
_
¸
_ = αx+βx.
13
U. Washington AMATH 352 - Spring 2009
8. For any vector x and y in R
m
and any scalar α in R, we have
α(x+y) = α
_
¸
_
x
1
+ y
1
.
.
.
x
m
+ y
m
_
¸
_ =
_
¸
_
α(x
1
+ y
1
)
.
.
.
α(x
m
+ y
m
)
_
¸
_ =
_
¸
_
αx
1
+ αy
1
.
.
.
αx
m
+ αy
m
_
¸
_ = αx+αy.
9. For any vector x in R
m
and any scalar α and β in R, we have
(αβ)x =
_
¸
_
(αβ)x
1
.
.
.
(αβ)x
m
_
¸
_ =
_
¸
_
α(βx
1
)
.
.
.
α(βx
m
)
_
¸
_ = α
_
¸
_
βx
1
.
.
.
βx
m
_
¸
_ = α(βx).
10. For any vector x in R
m
, we have
1 x = 1
_
¸
_
x
1
.
.
.
x
m
_
¸
_ =
_
¸
_
1 x
1
.
.
.
1 x
m
_
¸
_ =
_
¸
_
x
1
.
.
.
x
m
_
¸
_ = x.
So R
m
is a real linear space.
Example 21. The set of all possible functions mapping real numbers to real
numbers, f : R −→ R, is denoted T(R, R). We will verifty that T(R, R) is a
real linear space.
1. For any function f and g in T(R, R), we have
(f + g)(x) = f(x) + g(x).
So the function f + g is a function whose input is a real number and its
output is also a real number. f + g belongs to T(R, R). For instance, if
f(x) = 3x
2
and g(x) = cos(x), then f + g is the function defined by
(f + g)(x) = 3x
2
+ cos(x), ∀x ∈ R.
2. For any function f and g in T(R, R), the functions f + g and g + f are
equal. Indeed, we have
∀x ∈ R, (f + g)(x) = f(x) + g(x) = g(x) + f(x) = (g + f)(x).
We used the fact that the addition of scalar numbers is commutative.
3. The addition of functions is associative. Indeed, we have
∀x ∈ R, [(f + g) + h](x) = (f + g)(x) + h(x) = f(x) + g(x) + h(x)
= f(x) + (g + h)(x) = [f + (g + h)](x).
We used the fact that the addition of scalar numbers is associative.
14
U. Washington AMATH 352 - Spring 2009
4. The zero function
¯
0 is the function identically 0 in R,
¯
0(x) = 0, ∀x ∈ R. (19)
5. The additive inverse of a function f is the continuous function f
(−)
defined
by
f
(−)
(x) = −f(x), ∀x ∈ R. (20)
Indeed, we have f(x) + f
(−)
(x) = f(x) −f(x) = 0 for every x in R.
6. For any function f in T(R, R) and any scalar α in R, we have
∀x ∈ R, (αf)(x) = αf(x).
So the function αf is a function whose input is a real number and its
output is also a real number. αf belongs to T(R, R). For example, if
g(x) = cos(x), 5g is the function defined by
(5g)(x) = 5 cos(x), ∀x ∈ R.
7. For any function f in T(R, R) and any scalar α and β in R, we have
∀x ∈ R, [(α+β)f](x) = (α+β)f(x) = αf(x)+βf(x) = (αf)(x)+(βf)(x).
So the functions (α + β)f and αf + βf are equal.
8. For any function f and g in T(R, R) and any scalar α in R, we have
∀x ∈ R, [α(f + g)](x) = αf(x) + αg(x) = (αf)(x) + (αg)(x).
So the functions α(f + g) and αf + αg are equal.
9. For any function f in T(R, R) and any scalar α and β in R, we have
∀x ∈ R, [(αβ)f](x) = (αβ)f(x) = αβf(x) = α[βf(x)] = α(βf)(x).
So the functions (αβ)f and α(βf) are equal.
10. For any function f in T(R, R), we have
∀x ∈ R, (1 f)(x) = 1 f(x) = f(x).
Example 22. The set of all continuous functions mapping real numbers to real
numbers, f : R −→R, is denoted C
0
(R, R). C
0
(R, R) is a real linear space.
We emphasize that the zero vector takes different meanings according to the
linear space V . When V = R
m
, the zero vector is
¯
0 =
_
¸
¸
¸
_
0
0
.
.
.
0
_
¸
¸
¸
_
, (21)
15
U. Washington AMATH 352 - Spring 2009
the vector with m zero components. When V = (
0
(R, R), the zero vector
¯
0 is
the function identically 0 in R,
¯
0(x) = 0, ∀x ∈ R. (22)
We will mostly study linear algebra in the context of linear spaces of vectors,
but the study of other linear spaces, particularly function spaces, is extremely
important in many branches of mathematics and many of the ideas introduced
here carry over to other linear spaces.
Exercise 23. Verify that R
m
is a Q-linear space, where Q is the set of signed
rational numbers.
Exercise 24. Verify that C is a real linear space.
Exercise 25. Verify that C is a complex linear space.
Exercise 26. Show that the set T
n
of all polynomials with real coefficients of
degree n is a real linear space. What is the zero vector?
2.1 Subsets and subspaces
Suppose we have a linear space V and o is a subset of V , which just means
every element of o is also an element of V ,
v ∈ o =⇒v ∈ V. (23)
A subset might contain a finite or infinite number of elements.
Example 27. The subset,
o
1
=
__
1
2
_
,
_
−2.3
7
_
,
_
π
1
__
, (24)
is a subset of R
2
with 3 elements.
Example 28. The subset,
o
2
=
_
x ∈ R
3
: x
2
= x
2
1
+ 3x
3
_
, (25)
is a subset of R
3
with an infinite number of elements, including
_
_
0
0
0
_
_
,
_
_
−2
1
−1
_
_
,
_
_
1
4
1
_
_
.
Example 29. The subset,
o
3
=
_
x ∈ R
2
: x
2
= 3x
1
_
, (26)
is a subset of R
2
with an infinite number of elements, including
_
0
0
_
,
_
−2.3
−6.9
_
,
_
π

_
.
16
U. Washington AMATH 352 - Spring 2009
Definition 30. If o is a subset of a linear space V and o is closed under
addition and scalar multiplication, then we say that o is a subspace of V .
Consider the previous subsets.
• The subset o
1
is not a subspace of R
2
because adding two vectors from
o
1
does not give a vector in o
1
.
• The subset o
2
is not a subspace of R
3
because
2
_
_
−2
1
−1
_
_
=
_
_
−4
2
−2
_
_
/ ∈ o
2
.
• The subset o
3
is a subspace of R
2
.
Example 31. C
0
(R, R) denotes the set of functions f,
f : R −→R,
that are continuous. For instance, f(x) = 3x
2
+ cos(x) and g(x) = [x[ belong
to C
0
(R, R). C
0
(R, R) is a subspace of T(R, R). Indeed, it is a subset of
T(R, R), which is a real linear space. For any function f and g in C
0
(R, R),
the sum function f + g is continuous. So the sum function f + g belongs to
C
0
(R, R). C
0
(R, R) is closed for the addition. For any function f in C
0
(R, R)
and any scalar α in R, the function αf is continuous. So the function αf belongs
to C
0
(R, R). C
0
(R, R) is closed for the scalar multiplication. Consequently,
C
0
(R, R) is a subspace of T(R, R). It is also a real linear space.
Example 32. C
1
(R, R) denotes the set of functions f,
f : R −→R,
that are continuous and differentiable and whose derivative f

is also continuous.
For instance, f(x) = 3x
2
+cos(x) is in C
1
(R, R). On the other hand, g(x) = [x[
belongs to C
0
(R, R) but not to C
1
(R, R). C
1
(R, R) is a subspace of T(R, R).
Indeed, it is a subset of T(R, R), which is a real linear space. For any function f
and g in C
1
(R, R), the sum function f +g is continuous. It is also differentiable
and its derivative, equal to f

+ g

, is also continuous. So the sum function
f +g belongs to C
1
(R, R). C
1
(R, R) is closed for the addition. For any function
f in C
1
(R, R) and any scalar α in R, the function αf is continuous. It is
also differentiable and its derivative, equal to αf

, is also continuous. So the
function αf belongs to C
1
(R, R). C
1
(R, R) is closed for the scalar multiplication.
Consequently, C
1
(R, R) is a subspace of T(R, R). It is also a real linear space.
Example 33. C
p
(R, R) denotes the set of functions f,
f : R −→R,
that are continuous and differentiable p times and whose p-th derivative f
(p)
is
also continuous. C
p
(R, R) is a subspace of C
0
(R, R).
17
U. Washington AMATH 352 - Spring 2009
Exercise 34. Show that the set T
n
of all polynomials with real coefficients of
degree n is a subspace of C
0
(R, R).
Note the following about subspaces:
• The set o = V is a subspace of V (R
2
is a subspace of R
2
). If o is a
subspace of V that is not all of V then it is called a proper subspace of V .
• A subspace o of a real linear space V is also a real linear space.
• The set Z = ¦
¯
0¦ that contains only the zero element of the linear space V
is a subspace of V since
¯
0 +
¯
0 =
¯
0 and α
¯
0 =
¯
0 so this set is closed under
these operations.
• The set Z = ¦
¯
0¦ is the only subspace of V that contains a finite number of
elements (just 1 element). All other subspaces contain an infinite number
of elements. Why? Because if v ∈ o then αv ∈ o for any real number α
(of which there are infinitely many). If α
1
v = α
2
v then by the rules of
Definition 19, we can rewrite this as (α
1
−α
2
)v =
¯
0. But this can be true
only if v =
¯
0 or if α
1
−α
2
= 0.
The fact that a subspace is also a real linear space suggests a technique to prove
that a set U is a real linear space:
1. Find a real linear space | such that U ⊂ |;
2. Show that U is a subspace of |.
Example 35. Consider the set
o =
_
f ∈ C
0
(R, R) [ f(0) = 0
_
.
We would like to check that o is a real linear space. Two approaches are possible:
1. Use Definition 19 and check all 10 items.
2. Find a superset U, which is a real linear space, such that o ⊂ U. Show
that o is a subspace of U by checking the properties of being closed for
the addition and the multiplication (only 2 checks):
(a) ∀u, v ∈ o, u + v ∈ o
(b) ∀α ∈ R, ∀u ∈ o, αu ∈ o
To prove that o is a real linear space, we will use the second approach. We need
to find a superset. The definition of o suggests C
0
(R, R) as superset. Indeed,
every function in o is a continuous function. We know that C
0
(R, R) is a real
linear space. So we just need to check the properties of being closed.
Let f and g be two functions in o. The sum f + g is a continuous function
because f and g are continuous. To check whether f +g belongs to o, we need
also to compute the value of f + g at 0:
(f + g)(0) = f(0) + g(0) = 0 + 0 = 0.
18
U. Washington AMATH 352 - Spring 2009
So f + g belongs to o.
Let f be a functions in o and α ∈ R. The product αf is a continuous
function because f is continuous. To check whether αf belongs to o, we need
also to compute the value of αf at 0:
(αf)(0) = αf(0) = α0 = 0.
So αf belongs to o.
We conclude that o is a subspace of C
0
(R, R) and it is also a real linear
space.
2.2 Linear dependence and independence
If x, y ∈ R
m
are any two vectors and α = β = 0, then
αx + βy = 0x + 0y =
¯
0 +
¯
0 =
¯
0.
So this trivial linear combination of x and y is always the zero vector.
In a real linear space V , two vectors u, v ∈ V are said to be linearly dependent
if there is some nontrivial linear combination of u and v that gives
¯
0, i.e. if there
are scalars α, β ∈ R that are not both equal to zero but for which αu +βv =
¯
0.
In general, two vectors in V are linearly dependent if (and only if) one is a
scalar multiple of the other, for example if v = γu for some scalar γ. Since then
γu −v =
¯
0 (or any other linear combination with β ,= 0 and α = −γβ gives the
zero vector).
Two vectors are said to be linearly independent when there is no nontrivial
linear combination of u and v that gives
¯
0. In other words, they are linearly
independent if the equation
αu + βv =
¯
0 (27)
has only the trivial solution α = β = 0. Two vectors are linearly independent
when one is not a scalar multiple of the other.
Example 36. The vectors
x =
_
1
2
_
, y =
_
3
0
_
are linearly independent since neither one is a scalar multiple of the other.
Another way to see this is that the equation αu + βv =
¯
0 is
_
α + 3β

_
=
_
0
0
_
.
The second component is zero only if α = 0. But then the first component
becomes 3β, which is zero only if β = 0. So (27) is satisfied only when α = β = 0.
19
U. Washington AMATH 352 - Spring 2009
Example 37. The polynomials p and q, defined by
p(x) = 1, q(x) = x, ∀x ∈ R,
are linearly independent. Indeed, the equation αp + βq =
¯
0 becomes
α + βx = 0, ∀x ∈ R.
Taking x = 0 gives that α must be 0. Then any nonzero value of x implies that
β = 0. So (27) is satisfied only when α = β = 0.
The idea of linear dependence and independence can be extended to sets of
more than 2 vectors.
Definition 38. The set of r vectors u
(1)
, u
(2)
, , u
(r)
∈ V is linearly indepen-
dent if the equation
α
1
u
(1)
+ α
2
u
(2)
+ + α
r
u
(r)
=
¯
0 (28)
has only the trivial solution α
1
= α
2
= = α
r
= 0, i.e. if every nontriv-
ial linear combination of the vectors is nonzero. When the set is not linearly
independent, it is said to be linearly dependent.
Example 39. The vectors
x =
_
1
2
_
, y =
_
3
0
_
, z =
_
1
1
_
are linearly dependent. Consider a linear combination resulting in the zero
vector
αx + βy + γz =
_
0
0
_
.
Then we have
_
α + 3β + γ
2α + γ
_
=
_
0
0
_
.
Identifying each entry results in the following system of equations
_
α + 3β + γ = 0
2α + γ = 0
These equations imply that γ = −2α and β = α/3. However, α remains arbi-
trary. For example, a linear combination with α =3, β = 1, and γ = −6 results
in the zero vector. This non-trivial combination implies that the 3 vectors are
linearly dependent.
Example 40. The vectors,
x
(1)
=
_
_
7
0
0
_
_
, x
(2)
=
_
_
1
2
0
_
_
, x
(3)
=
_
_
3
4
5
_
_
,
20
U. Washington AMATH 352 - Spring 2009
are linearly independent. Indeeed, α
1
x
(1)
+ α
2
x
(2)
+ α
3
x
(3)
= 0 if and only
if α
1
= α
2
= α
3
= 0. Let us prove this result. If α
1
= α
2
= α
3
= 0, then
α
1
x
(1)
+ α
2
x
(2)
+ α
3
x
(3)
= 0. If a linear combination of x
(1)
, x
(2)
, and x
(3)
is
zero, then we have
α
1
x
(1)
+ α
2
x
(2)
+ α
3
x
(3)
=
_
_

1
+ α
2
+ 3α
3

2
+ 4α
3

3
_
_
=
_
_
0
0
0
_
_
.
By matching the third component, we get that α
3
has to be 0. Plugging this
value into the second component, we obtain that α
2
is also 0. Then the first
component implies that α
1
is 0. A linear combination of x
(1)
, x
(2)
, and x
(3)
is
zero if and only if α
1
= α
2
= α
3
= 0.
Exercise 41. Check the linear dependency of the vectors
x
(1)
=
_
_
1
2
3
_
_
, x
(2)
=
_
_
−2.1
−4.2
−6.3
_
_
.
Exercise 42. Check the linear dependency of the vectors
x
(1)
=
_
_
1
2
3
_
_
, x
(2)
=
_
_
−2
1
1.5
_
_
, x
(3)
=
_
_
1
0
0
_
_
.
Example 43. The functions 1, x, and x
2
are linearly independent. Indeed,
consider the linear combination
α + βx + γx
2
= 0, ∀x ∈ R (29)
Taking x = 0 implies that α must be 0. Then taking x = 1 and x = −1 gives
β + γ = 0
−β + γ = 0
which implies that β = γ = 0. Summing the two equations results in β = 0.
Plugging α = β = 0 in any equation gives that γ = 0. So the linear combination
(29) is zero if and only if α = β = γ = 0.
Example 44. We will check the linear dependency of the functions 1, e
x
, and
e
−x
. Consider a linear combination that is equal to the zero function:
α + βe
x
+ γe
−x
= 0, ∀x ∈ R.
Differentiating this relationship, we have also
βe
x
−γe
−x
= 0, ∀x ∈ R.
Taking a second derivative, we get
βe
x
+ γe
−x
= 0, ∀x ∈ R.
21
U. Washington AMATH 352 - Spring 2009
Summing the last two equations, we obtain
2βe
x
= 0, ∀x ∈ R,
which implies that β = 0. Plugging this value into the equation after one
differentiation, we have
−γe
−x
= 0, ∀x ∈ R
and γ = 0. Finally, the first relation gives that α = 0. Consequently, the
functions 1, e
x
, and e
−x
are linearly independent.
Exercise 45. Check the linear dependency of the functions 1, cos(πx), and
sin(πx) on [−1, 1].
2.3 Span of a set of vectors
Let V denote a real linear space and u
(1)
, , u
(r)
∈ V be a set of r vectors.
Then the span of this set of vectors is the space of all linear combinations of
these vectors,
span(u
(1)
, , u
(r)
) =
_
α
1
u
(1)
+ + α
r
u
(r)
; α
1
, , α
r
∈ R
_
. (30)
This is a subspace of V . Since any linear combination of vectors in this set is
again a linear combination of u
(1)
, , u
(r)
∈ V .
Example 46. The subspace o
3
, given by (26), can be written as
o
3
= span
__
1
3
__
.
Example 47. The space
o = span
__
1
0
_
,
_
0
1
__
is all of R
2
, since any vector x ∈ R
2
can be written as a linear combination of
these two vectors:
_
x
1
x
2
_
= x
1
_
1
0
_
+ x
2
_
0
1
_
.
Example 48. The space
o = span
__
1
0
_
,
_
0
1
_
,
_
2
3
__
is all of R
2
, since any vector x ∈ R
2
can be written as a linear combination of
these three vectors:
_
x
1
x
2
_
= x
1
_
1
0
_
+ x
2
_
0
1
_
+ 0
_
2
3
_
.
22
U. Washington AMATH 352 - Spring 2009
Actually, in this case, there are infinitely many different ways to write an arbi-
trary vector x ∈ R
2
as a linear combination of these three vectors. For example,
we could write it as
_
x
1
x
2
_
= −x
1
_
1
0
_
+ (x
2
−3x
1
)
_
0
1
_
+ 2x
1
_
2
3
_
.
Example 49. The set T
2
of real polynomials of degree at most 2 is a real linear
space. Any polynomial in T
2
is written as
p(x) = α + βx + γx
2
.
It is a linear combination of functions 1, x, and x
2
. So we have
T
2
= span
_
1, x, x
2
_
.
2.4 Basis vectors
Consider V a real linear space and u
(1)
, , u
(r)
∈ V be a set of r vectors.
Then the span of this set of vectors,
span(u
(1)
, , u
(r)
) =
_
α
1
u
(1)
+ + α
r
u
(r)
; α
1
, , α
r
∈ R
_
, (31)
defines a subspace of V . In fact it can be shown that any subspace of R
m
has
this form — it is the span of some set of vectors. A minimal set of vectors that
define a space is called a basis for the space. What do we mean by minimal?
The subspace o
3
, given by (26), can be written as
o
3
= span
__
1
3
__
= span
__
2
6
__
= span
__
1
3
_
,
_
2
6
__
.
In the latter case, note that the last two vectors are linearly dependent. Clearly,
we require at least one vector to define this particular space, but specifying two
vectors is redundant. We say that either one of these vectors alone is a basis for
this particular space. More generally we make the following definition to make
this idea precise:
Definition 50. If o is a subspace of a linear space V , then the set of vectors
u
(1)
, , u
(r)
∈ V form a basis for o if
o = span(u
(1)
, , u
(r)
), (32a)
u
(1)
, , u
(r)
are linearly independent. (32b)
If a set of vectors spanning the space are not linearly independent, then we
can find a basis consisting of fewer vectors.
23
U. Washington AMATH 352 - Spring 2009
Example 51. The vectors
__
1
0
_
,
_
0
1
__
form a basis of R
2
. Indeed they span R
2
and they are linearly independent.
Example 52. The space T
2
is spanned by the functions 1, x, and x
2
. These
functions are linearly independent. Consequently, (1, x, x
2
) form a basis for T
2
.
2.5 Dimension of a space
There are typically infinitely many choices of basis vectors for a given space.
However, it can be shown that all choices of basis vectors consist of the same
number of vectors.
Definition 53. The number of basis vectors needed to define a space is called
the dimension of the space.
Example 54. The space o
3
, given by (26), is a 1-dimensional subspace of R
2
.
The single vector
_
1
3
_
is one possible basis for this space.
Example 55. The space o
0
= ¦
¯
0¦, consisting only of the zero vector, is a
special case. We say it is a 0-dimensional space. This is the only linear space
that consists of a single vector and the only linear space that has no basis.
Example 56. The space T
2
, consisting of polynomials of degree at most 2, is
of dimension 3. Indeed, (1, x, x
2
) form a basis for T
2
.
Example 57. The vector space R
m
has dimension m. The “standard basis”
consists of the “unit vectors”,
_
¸
¸
¸
¸
¸
_
1
0
0
.
.
.
0
_
¸
¸
¸
¸
¸
_
,
_
¸
¸
¸
¸
¸
_
0
1
0
.
.
.
0
_
¸
¸
¸
¸
¸
_
,
_
¸
¸
¸
¸
¸
_
0
0
1
.
.
.
0
_
¸
¸
¸
¸
¸
_
, ,
_
¸
¸
¸
¸
¸
_
0
0
0
.
.
.
1
_
¸
¸
¸
¸
¸
_
. (33)
There are infinitely many other bases: any set of m linearly independent vectors
will do.
Example 58. The set of complex values C is a real linear space of dimension
2. It is also a complex linear space of dimension 1.
Exercise 59. What is the dimension of T
n
the real linear space of polynomials
with real coefficients of degree less or equal to n?
24
U. Washington AMATH 352 - Spring 2009
Exercise 60. What is the dimension of the real linear space R? Exhibit a basis.
What is the dimension of R as a Q-linear space?
Finally, we have the following important result comparing dimensions be-
tween a real linear space and one of its subspaces.
Proposition 61. Let V be a real linear space and o ⊂ V be a subspace. Then
we have
dimo ≤ dimV (34)
where dim denotes the dimension of a real linear space.
We will look for the dimension and a basis for a couple of spaces. Note that
an infinite number of bases exist. However, all the bases are composed of the
same number of elements. For each space, a systematic approach to find a basis
is defined in the following steps.
1. Check that the space o is indeed a real linear space. For example, verify
that it is a subspace of a bigger real linear space.
2. Write a general formula characterizing any element of o (an element can
be a function, a vector, ...).
3. Identify the independent parameters in the formula.
4. Write a new formula characterizing any element of o as a linear combina-
tion of elements where each parameter is multiplying one element. This
formula should indicate that these elements span the space o.
5. Check whether these particular elements are linearly independent. If they
are linearly independent, these elements form a basis for o. If not, look
for a subset of these elements that still spans o and that contains linearly
independent elements.
Next, we apply these steps to the different examples.
Example 62. Consider the set o =
_
x ∈ R
3
[ x
3
= 0
_
. o is composed of vectors
with 3 rows, whose last component, x
3
, is equal to 0. o is a real linear space.
Indeed it is a subset of R
3
, which is a real linear space. o is closed for the
addition. For any vectors x and y in o, we have
x =
_
_
x
1
x
2
0
_
_
y =
_
_
y
1
y
2
0
_
_
⇒ x +y =
_
_
x
1
+ y
1
x
2
+ y
2
0
_
_
.
The vector x + y is a vector with 3 rows and its last component is equal to 0.
So x +y belongs to o and o is closed for the addition. It is also closed for the
scalar multiplication. For any vector x in o and any scalar α in R, we have
x =
_
_
x
1
x
2
0
_
_
⇒ αx =
_
_
αx
1
αx
2
0
_
_
.
25
U. Washington AMATH 352 - Spring 2009
The vector αx is a vector with 3 rows and its last component is equal to 0. So
αx belongs to o and o is closed for the scalar multiplication. So o is a subspace
of R
3
and a real linear space. To find a basis for o and the dimension, we write
a general formula for any vector in o
x ∈ o ⇐⇒ x =
_
_
x
1
x
2
0
_
_
.
This general formula has two parameters x
1
and x
2
. So we can write
x ∈ o ⇐⇒ x =
_
_
x
1
x
2
0
_
_
= x
1
_
_
1
0
0
_
_
+ x
2
_
_
0
1
0
_
_
.
So any vector in o is a linear combination of the two vectors
_
_
1
0
0
_
_
and
_
_
0
1
0
_
_
,
which implies that
o = span
_
_
_
_
1
0
0
_
_
,
_
_
0
1
0
_
_
_
_
.
To obtain a basis, we need to check whether these two vectors are linearly
independent. Consider a linear combination equal to the vector 0
α
_
_
1
0
0
_
_
+ β
_
_
0
1
0
_
_
=
_
_
0
0
0
_
_

_
_
α
β
0
_
_
=
_
_
0
0
0
_
_
.
Identifying the entries, we obtain that α = β = 0. The vectors are linearly
independent and they span o. So they form a basis for o. The dimension of o
is 2.
Example 63. Check whether the set
_
f ∈ C
1
(R, R) [ f

(x) = f(x)
_
is a real
linear space. Determine a basis and the dimension. Denote o the set to study.
We notice that o is a subset of C
1
(R, R), which is a real linear space. So we
will show that o is a subspace of C
1
(R, R). This will prove also that o is a
real linear space. So we need to check whether o is closed for the addition and
closed for the scalar multiplication. Consider any function f and g in o, then
the function f + g is a function in C
1
(R, R) because C
1
(R, R) is a real linear
space. To check whether f +g belongs to o, we have to compute the derivative
to see if it is equal to f + g:
(f + g)

(t) = f

(t) + g

(t) = f(t) + g(t) = (f(t) + g(t)) = (f + g)(t)
So f + g belongs to o. Finally, we need to check whether o is closed for the
multiplication. Consider any function f in o and any real number α, then the
function αf is a function in C
1
(R, R) because C
1
(R, R) is a real linear space.
26
U. Washington AMATH 352 - Spring 2009
To check whether αf belongs to o, we have to compute the derivative to see if
it is equal to αf:
(αf)

(t) = αf

(t) = α(f(t)) = αf(t) = (αf)(t).
So αf belongs to o. We conclude that o is a subspace of C
1
(R, R) and it is also
a real linear space. Any function in o is solution to the equation f

(t) = f(t).
A general solution is f(t) = αe
t
, for any α ∈ R. No other solution exists. Any
solution is proportional to e
t
. So we have o = span(e
t
). A non-zero function is
necessarily linearly independent. So (e
t
) is a basis for o. The dimension of o is
1.
Example 64. Consider the set o = ¦p ∈ T
3
[ p(0) = p(1) = p(2) = 0¦. o
is composed of polynomials of degree at most 3 that take the value 0 when
evaluated at x = 0, x = 1, and x = 2. o is a real linear space. Indeed it is a
subset of T
3
, which is a real linear space. o is closed for the addition because,
for any polynomials p and q in o, p+q is a polynomial in T
3
. When evaluated at
0 (or 1 or 2), p + q is equal to zero because p(0) = q(0) = 0 (or p(1) = q(1) = 0
or p(2) = q(2) = 0). So o is closed for the addition. It is also closed for the
multiplication because the values at 0 of αp is equal to αp(0) = 0 (the same
thing holds for x = 1 and x = 2). So o is a real linear space. We can write a
general formula for any polynomial in o.
α + βx + γx
2
+ δx
3
= 0
(x = 0) α = 0
(x = 1) β + γ + δ = 0
(x = 2) 2β + 4γ + 8δ = 0
Substracting two times the next-to-last equation from the last equation, we get
2β + 4γ + 8δ −(2β + 2γ + 2δ) = 2γ + 6δ = 0 =⇒ γ = −3δ.
Then we get for β = −γ − δ = 3δ − δ = 2δ. So any polynomial in o has the
form
p(x) = δ(2x −3x
2
+ x
3
).
The space o is spanned by the polynomial 2x − 3x
2
+ x
3
, which is non-zero.
The polynomial 2x −3x
2
+ x
3
forms a basis. The dimension of o is 1.
27
U. Washington AMATH 352 - Spring 2009
3 Linear Functions
First, recall what we mean by the notation
f : U →V. (35)
The function f takes an element of U as input. It is defined for any element of
U. The function value f(u) is an element of V .
We are now ready to define what we mean by a linear function.
Definition 65. Consider U and V two real linear spaces. The function f :
U →V is a linear function if both of the following conditions are satisfied:
∀u
(1)
, u
(2)
∈ U, f(u
(1)
+ u
(2)
) = f(u
(1)
) + f(u
(2)
) (36a)
∀u ∈ U and α ∈ R, f(αu) = αf(u) (36b)
When U and V are two complex linear spaces, condition (36b) is modified such
that α belongs to C. Note that f(
¯
0) =
¯
0, by taking α = 0.
3.1 Linear functions from R to R
The conditions (36) are very restrictive. The only functions satisfying these
conditions are functions of the form
f(x) = ax (37)
where a is some fixed real number. Indeed, we have
f(x) = f(x 1) = xf(1) (38)
and
f(x+y) = f(x) +f(y) = f(x 1) +f(y 1) = xf(1) +yf(1) = (x+y)f(1). (39)
The graph of such a function is simply a line through the origin with slope a,
as illustrated in Figure 3 for two choices of a. The fact that the graph is a line
helps us remember why these functions are called linear.
Now consider the function g(x) = 6x −3, whose graph is shown in Figure 4.
This graph is also a straight line but this is not a linear function according to the
strict terms of Definition 65. It cannot be since g(0) ,= 0 and we can also easily
check that g(1 +2) = 15 while g(1) +g(2) = 12, for example. The function g(x)
is properly called an affine function although people are often imprecise and call
it linear. It really consists of a linear function 6x plus a translation (shifting
each point downwards by 3), and this is what is meant by an affine function
more generally: a linear function shifted a constant (in this case −3).
Another way to define an affine function is to say that g(x) is affine if g(x) −
g(y) is a linear function of x − y, or equivalently if, for any fixed point x
0
the
function f(s) = g(x
0
+ s) − g(x
0
) is a linear function of s. You can check
28
U. Washington AMATH 352 - Spring 2009
Figure 3: (Left) Graph of f(x) = 2x. (Right) Graph of f(x) = −0.5x.
Figure 4: Graph of the affine function g(x) = 6x −3.
that this is true for any function of the form g(x) = ax + b, and these are the
only affine functions for the case we are currently considering, the simple case
of a function mapping R to R. (Soon these ideas will be generalized to more
interesting situations, so make sure you understand these basic ideas even if
they seem trivial now!)
3.2 Linear functions from R to R
m
We might have a situation where there are two different output values f
1
(x)
and f
2
(x) that depend on all the same input value x. We could then use the
symbol f (x) to denote the vector
f (x) =
_
f
1
(x)
f
2
(x)
_
(40)
containing these two output values. If the outputs are real numbers and are
defined for any real number x, then we would say that f maps R to R
2
.
This function is said to be linear if both real functions f
1
(x) and f
2
(x) are
linear functions in the sense of Definition 65. From what we noticed in the
previous section, we see that a function f : R → R
2
is linear only if it has the
29
U. Washington AMATH 352 - Spring 2009
form
f (x) =
_
a
1
x
a
2
x
_
(41)
where the values a
1
and a
2
are two scalar constants.
More generally, a function f : R →R
m
is linear if and only if it has the form
f (x) =
_
¸
_
a
1
x
.
.
.
a
m
x
_
¸
_ (42)
where the values a
1
, , a
m
are m scalar constants.
3.3 Linear functions from R
n
to R
Suppose f is a function that depends on several input values, say x
1
, x
2
, and
x
3
. To make the notation simple, we can still talk about the function f(x) if
we now think of x as the vector
x =
_
_
x
1
x
2
x
3
_
_
. (43)
Using the standard unit basis (33), we write a linear combination in terms
of the unit basis vectors
_
_
x
1
x
2
x
3
_
_
=
_
_
x
1
0
0
_
_
+
_
_
0
x
2
0
_
_
+
_
_
0
0
x
3
_
_
= x
1
_
_
1
0
0
_
_
+ x
2
_
_
0
1
0
_
_
+ x
3
_
_
0
0
1
_
_
. (44)
It is easy to show
f(x) = x
1
f
_
_
_
_
1
0
0
_
_
_
_
+ x
2
f
_
_
_
_
0
1
0
_
_
_
_
+ x
3
f
_
_
_
_
0
0
1
_
_
_
_
. (45)
An essential property of a linear function is the following: if we know the values
of
f
_
_
_
_
1
0
0
_
_
_
_
, f
_
_
_
_
0
1
0
_
_
_
_
, and f
_
_
_
_
0
0
1
_
_
_
_
, (46)
then we can easily find the value of f(x) for any x ∈ R
3
. Indeed, any vector of
R
3
is a linear combination of basis vectors
_
_
1
0
0
_
_
,
_
_
0
1
0
_
_
, and
_
_
0
0
1
_
_
. (47)
Taking linear combinations of vectors is a fundamental operation in linear alge-
bra.
30
U. Washington AMATH 352 - Spring 2009
It turns out that if f : R
n
→ R is a linear function, then all we need to
do is evaluate the function f for some well chosen set of n linearly independent
vectors and we can evaluate f(x) for any x in all of R
n
. This is one of the reasons
why linear problems and linear equations are so important in applications.
You might guess that every linear function mapping R
3
to R must have the
form
f(x) = a
1
x
1
+ a
2
x
2
+ a
3
x
3
(48)
for some constant real numbers a
1
, a
2
, and a
3
. You would be right! More
generally, a function f : R
n
→R is linear if and only if it has the form
f(x) = a
1
x
1
+ a
2
x
2
+ + a
n
x
n
(49)
where the values a
1
, , a
n
are n scalar constants.
3.4 Linear functions from R
n
to R
m
Now suppose that the function f is a vector with m components that depends
on n input values. We can write
f (x) =
_
¸
_
f
1
(x)
.
.
.
f
m
(x)
_
¸
_. (50)
According to Definition 65, f is a linear function if and only if the m com-
ponents f
1
, , f
m
are linear functions of x.
From what we noticed in the previous sections, the ith function f
i
(x) is then
given by
f
i
(x) = a
i1
x
1
+ a
i2
x
2
+ + a
in
x
n
=
n

j=1
a
ij
x
j
(51)
for i = 1, 2, . . . , m.
We often work with linear functions with many inputs and outputs and
so it is nice to simplify the notation for describing these functions. This was
recognized many years ago and so the notation of a matrix was invented. The
function is uniquely determined by mn numbers that are naturally arranged in
a matrix with m rows and n columns:
A =
_
¸
¸
¸
_
a
11
a
12
a
1n
a
21
a
22
a
2n
.
.
.
.
.
.
.
.
.
a
m1
a
m2
a
mn
_
¸
¸
¸
_
. (52)
A common notation is
A = (a
ij
)
i=1,··· ,m;j=1,··· ,n
= (a
ij
)
m×n
(53)
31
U. Washington AMATH 352 - Spring 2009
For shorthand, we might write
f (x) = Ax, (54)
where x is the vector of inputs with n components. This notation suggests that
we multiply the mn matrix A by the vector x to obtain the vector f (x).
Definition 66. The set of all real matrices with m rows and n columns is
denoted R
m×n
.
We define the concept of matrix-vector multiplication so that this is correct:
Definition 67. If A is an m n matrix of the form (52) and x ∈ R
n
is an
n-vector, then the product b = Ax is an m-vector (b ∈ R
m
) and the ith
component of b is
b
i
= a
i1
x
1
+ a
i2
x
2
+ + a
in
x
n
=
n

j=1
a
ij
x
j
. (55)
The matrix-vector multiplication can also be displayed as follows:
b =
_
¸
¸
¸
¸
_
a
1
a
2
a
n
_
¸
¸
¸
¸
_
_
¸
¸
¸
_
x
1
x
2
.
.
.
x
n
_
¸
¸
¸
_
= x
1
a
1
+ x
2
a
2
+ . . . + x
n
a
n
where b is expressed as a linear combination of the columns a
j
. It is a slight
change of notation to highlight that x acts of A to produce b. For example, we
have
_
1 0 2
−1 3 1
_
_
_
3
2
1
_
_
=
_
1 3 + 0 2 + 2 1
(−1) 3 + 3 2 + 1 1
_
=
_
5
4
_
. (56)
Note that once we have defined matrix-vector multiplication properly, the
fact that any linear function can be written in the form (54) (for some particular
matrix A) is a nice generalization of the fact that any single linear function of
a single variable has the form f(x) = ax for some single number a. That simple
case is just the case m = n = 1 in which the “matrix” a is 1 1.
Next we define the addition between two matrices and the scalar multipli-
cation.
Definition 68. For any matrices A = (a
ij
)
m×n
and B = (b
ij
)
m×n
in R
m×n
,
the addition A+B is a matrix in R
m×n
whose entries are the sum of the entries
of A and B, i.e. A+B = (a
ij
+ b
ij
)
m×n
.
Definition 69. For any matrix A = (a
ij
)
m×n
in R
m×n
and any scalar α in R,
the scalar multiplication αA is a matrix in R
m×n
whose entries are the entries
of A multiplied by α, i.e. αA = (αa
ij
)
m×n
.
32
U. Washington AMATH 352 - Spring 2009
An alternative definition of the addition of two matrices uses linear functions.
Suppose we have two linear functions f (x) = Ax and g(x) = Bx that are defined
by two different matrices A and B (both of which are m n, so both f and g
map R
n
to R
m
). Now define a new function h(x) by
h(x) = f (x) +g(x).
This means that, for any vector x ∈ R
n
, to compute the value of h(x) ∈ R
m
,
we first compute the two vectors f (x) and g(x). Then we add them together,
using the “vector addition” rule from R
m
. So the ith component of h(x) is just
the ith component of f (x) added to the ith component of g(x).
Since f and g are both linear functions, it turns out that the function h is
also a linear function. To see this, note that the ith component of h(x) is
h
i
(x) = f
i
(x) + g
i
(x)
=
n

j=1
a
ij
x
j
+
n

j=1
b
ij
x
j
=
n

j=1
(a
ij
+ b
ij
)x
j
.
(57)
But this means that h(x) is defined by the matrix-vector multiplication
h(x) = Cx
where C is the mn matrix with components c
ij
= a
ij
+ b
ij
. In other words,
C = A+B
where we define the sum of two matrices of the same shape in the obvious way,
by adding the corresponding elements of the two matrices. So h is a linear
function and the matrix that defines it is simply the sum of the matrices A and
B defining the functions f and g.
Proposition 70. R
m×n
is a real linear space.
Remark. The set of all matrices with complex entries, m rows, and n columns
is denoted C
m×n
, which is a complex linear space.
Next, we give a proof that R
m×n
is a real linear space.
1. For any matrix A and any matrix B in R
m×n
, we have
A = (a
ij
)
m×n
B = (b
ij
)
m×n
⇒ A+B = (a
ij
+ b
ij
)
m×n
.
So the matrix A+B is also a matrix with m rows and n columns and it
belongs to R
m×n
.
33
U. Washington AMATH 352 - Spring 2009
2. For any matrix A and any matrix B in R
m×n
, the vectors A + B and
B+A are equal. Indeed, we have
A+B = (a
ij
+ b
ij
)
m×n
B+A = (b
ij
+ a
ij
)
m×n
and the mn components are equal because the addition of scalar numbers
is commutative.
3. The addition of matrices is associative. Indeed, we have
(A+B)+C = ((a
ij
+ b
ij
) + c
ij
)
m×n
A+(B+C) = (a
ij
+ (b
ij
+ c
ij
))
m×n
.
The mn components are equal because the addition of scalar numbers is
associative.
4. The zero matrix in R
m×n
is the matrix with all its mn components equal
to 0,
0 =
_
¸
_
0 0
.
.
.
.
.
.
0 0
_
¸
_.
5. For any matrix A in R
m×n
, the matrix B, defined by
A = (a
ij
)
m×n
B = (−a
ij
)
m×n
,
is such that A+B = 0. The matrix B is the additive inverse of A.
6. For any matrix A in R
m×n
and any scalar α in R, we have
A = (a
ij
)
m×n
⇒ αA = (αa
ij
)
m×n
.
So the matrix αA is also a matrix with m rows and n columns and it
belongs to R
m×n
.
7. For any matrix A in R
m×n
and any scalar α and β in R, we have
(α + β)A = ((α + β) a
ij
)
m×n
= (αa
ij
+ βa
ij
)
m×n
= (αa
ij
)
m×n
+ (βa
ij
)
m×n
= αA+ βA.
8. For any matrices A and B in R
m×n
and any scalar α in R, we have
α(A+B) = α(a
ij
+ b
ij
)
m×n
= (αa
ij
+ αb
ij
)
m×n
= (αa
ij
)
m×n
+ (αb
ij
)
m×n
= αA+ αB.
9. For any matrix A in R
m×n
and any scalar α and β in R, we have
(αβ)A = ((αβ) a
ij
)
m×n
= (αβa
ij
)
m×n
= α(βa
ij
)
m×n
= α(βA) .
34
U. Washington AMATH 352 - Spring 2009
10. For any matrix A in R
m×n
, we have
1 A = 1 (a
ij
)
m×n
= (1 a
ij
)
m×n
= (a
ij
)
m×n
.
So R
m×n
is a real linear space.
Remark. Note that R
m
is the set of column vectors with m rows and 1 column.
R
m×1
denotes the set of matrices with m rows and 1 column.
Proposition 71. The matrix-vector multiplication satisfies
Ax +Bx = (A+B)x (58)
and
Ax +Ay = A(x +y). (59)
The proofs simply use the associativity of real numbers to rewrite a
ij
x
j
+
b
ij
x
j
as (a
ij
+ b
ij
)x
j
. Indeed we have
Ax =
_
¸
_

n
j=1
a
1j
x
j
.
.
.

n
j=1
a
mj
x
j
_
¸
_ Bx =
_
¸
_

n
j=1
b
1j
x
j
.
.
.

n
j=1
b
mj
x
j
_
¸
_
⇒ Ax+Bx =
_
¸
_

n
j=1
a
1j
x
j
+

n
j=1
b
1j
x
j
.
.
.

n
j=1
a
mj
x
j
+

n
j=1
b
mj
x
j
_
¸
_ =
_
¸
_

n
j=1
(a
1j
+ b
1j
)x
j
.
.
.

n
j=1
(a
mj
+ b
mj
)x
j
_
¸
_
So we have proved something nontrivial about matrix-vector algebra using the
rules of standard algebra of real numbers. It is also true for the other property
of matrix-vector multiplication.
Exercise 72. Consider the matrices A = (a
ij
)
m×n
and B = (b
ij
)
m×n
. Let
α, β ∈ R. What are the entries of the matrix αA+ βB?
Exercise 73. Consider the square matrix A = (a
ij
)
n×n
, i.e. m = n. The
function,
tr(A) = a
11
+ a
22
+ + a
nn
=
n

i=1
a
ii
, (60)
evaluates the trace of a square matrix. Show that the trace function is a linear
function on R
n×n
.
3.5 Linear differential operators
Recall that algebra and the idea of linearity applies to to situations other than
vectors and matrices. Many of the fundamental concepts we learn about linear
algebra for vectors and matrices carry over directly to functions and differen-
tial operators. Seeing these ideas in a different context now may help you to
understand what “linearity” means more generally and why it is important.
35
U. Washington AMATH 352 - Spring 2009
Let us introduce the concept of a operator, which is just a function that takes
a function as input and produces some other function as output. This gets a
bit confusing so we call it an “operator” instead of a function. A differential
operator computes the output function by combining various derivatives of the
input function.
The simplest differential operator is just D =
d
dx
. For example, if f(x) = x
3
then D(f) is the function f

(x) = 3x
2
. The operator D is a linear operator, it
satisfies the same linearity properties as in Definition 65, where U = C
1
(R, R)
and V = C
0
(R, R).
The operator D is a linear operator because it is true that
D(u+v) = D(u) +D(v), since
d
dx
(u(x) +v(x)) =
d
dx
u(x) +
d
dx
v(x), (61)
and
D(αu) = αD(u), since
d
dx
(αu(x)) = α
d
dx
u(x). (62)
These linearity properties extend of course to arbitrary linear combinations.
You are used to using this when finding the derivatives of complicated functions
by splitting them up, e.g.
d
dx
_
5x
4
+ 2 cos(6x)
_
= 5
d
dx
(x
4
) + 2
d
dx
cos(6x) = 20x
3
−12 sin(6x).
Linearity was used in the first step.
The second derivative operator
d
2
dx
2
and all higher order derivative operators
are also linear operators. We obtain a general linear differential operator by
taking a linear combination of these differential operators (and also the zero’th
order derivative operator, which is just the identity operator that maps any
function to itself). For example, the operator
L = 8
d
2
dx
2
−4
d
dx
+ 6I (63)
is a linear differential operator, where I denotes the identity operator. Applying
L to a function u(x) results in the function
(Lu)(x) = 8u

(x) −4u

(x) + 6u(x). (64)
Now consider the differential equation
8u

(x) −4u

(x) + 6u(x) = x
3
(65)
for 0 ≤ x ≤ 1 with u(0) = 1 and u(1) = 3. The problem is to find a funtion u(x)
that satisfies the equation (65) everywhere in the interval and also satisfies the
two boundary conditions. This is a linear differential equation since it has the
form (Lu)(x) = g(x) where L is the linear operator of (64) and g(x) is a given
function g(x) = x
3
.
36
U. Washington AMATH 352 - Spring 2009
Exercise 74. Consider the operator O : C
0
(R, R) →C
0
(R, R) defined by
O(f)(x) = xf(x) ∀x ∈ R.
Check whether the operator O is linear.
Exercise 75. Consider the operator O : C
1
(R, R) →C
0
(R, R) defined by
O(f)(x) = 2f

(x)f(x) ∀x ∈ R.
Check whether the operator O is linear.
3.6 Useful commands in Matlab
Here are a few commands in Matlab useful for this section.
• A = [1, 2;3, 4;5, 6]; defines the matrix A ∈ R
3×2
. Note that com-
mas separate the column entries on a same row and that semi-columns
finish every row.
• A = zeros(m,n) defines the matrix in R
m×n
with entries equal to 0.
• A = ones(m,n) defines the matrix in R
m×n
with entries equal to 1.
• size(A) returns the dimensions of A in a row vector [m n].
• A = rand(m,n) makes a matrix of size m n with random values, uni-
formly distributed between 0 and 1.
• A = randn(m,n) makes a matrix of size m n with random values, nor-
mally distributed (with mean 0 and variance 1).
• A*x computes the matrix-vector product.
• tr(A) computes the trace of matrix A.
37
U. Washington AMATH 352 - Spring 2009
4 Matrices
4.1 Space of mn matrices
The set of m n real matrices is denoted R
m×n
. The previous section proved
that R
m×n
is a real linear space. Its dimension is equal to mn. A simple basis
for R
m×n
consists of the mn matrices with only one non-zero entry.
For example, consider R
3×2
. Any matrix with 3 columns and 2 rows is
written as follows
A =
_
_
a
11
a
12
a
21
a
22
a
31
a
32
_
_
, where a
11
, a
12
, a
21
, a
22
, a
31
, a
32
∈ R.
The matrix A depends on 6 parameters: a
11
, a
12
, a
21
, a
22
, a
31
, and a
32
. We
write a linear combination of matrices where each parameter is multiplying a
matrix
A = a
11
_
_
1 0
0 0
0 0
_
_
+ a
12
_
_
0 1
0 0
0 0
_
_
+ a
21
_
_
0 0
1 0
0 0
_
_
+ a
22
_
_
0 0
0 1
0 0
_
_
+ a
31
_
_
0 0
0 0
1 0
_
_
+ a
32
_
_
0 0
0 0
0 1
_
_
.
This last formula illustrates that the 6 matrices span R
3×2
,
R
3×2
= span
_
_
_
_
1 0
0 0
0 0
_
_
,
_
_
0 1
0 0
0 0
_
_
,
_
_
0 0
1 0
0 0
_
_
,
_
_
0 0
0 1
0 0
_
_
,
_
_
0 0
0 0
1 0
_
_
,
_
_
0 0
0 0
0 1
_
_
_
_
.
These 6 matrices are also linearly independent. Indeed, we have
α
_
_
1 0
0 0
0 0
_
_
+ β
_
_
0 1
0 0
0 0
_
_
+ γ
_
_
0 0
1 0
0 0
_
_
+ δ
_
_
0 0
0 1
0 0
_
_
+
_
_
0 0
0 0
1 0
_
_
+ ζ
_
_
0 0
0 0
0 1
_
_
=
_
_
α β
γ δ
ζ
_
_
=
_
_
0 0
0 0
0 0
_
_
.
By identifying each entry to zero, we obtain the following 6 equations
_
¸
¸
¸
¸
¸
¸
_
¸
¸
¸
¸
¸
¸
_
α = 0 (from entry (1,1))
β = 0 (from entry (1,2))
γ = 0 (from entry (2,1))
δ = 0 (from entry (2,2))
= 0 (from entry (3,1))
ζ = 0 (from entry (3,2))
38
U. Washington AMATH 352 - Spring 2009
So these 6 matrices are linearly independent because the only linear combination
resulting in the zero matrix is the trivial combination with
α = β = γ = δ = = ζ = 0.
Consequently, a basis for R
3×2
is given by
_
_
_
_
1 0
0 0
0 0
_
_
,
_
_
0 1
0 0
0 0
_
_
,
_
_
0 0
1 0
0 0
_
_
,
_
_
0 0
0 1
0 0
_
_
,
_
_
0 0
0 0
1 0
_
_
,
_
_
0 0
0 0
0 1
_
_
_
_
. (66)
The dimension of R
3×2
is 6.
Remark. The set of mn complex matrices is denoted C
m×n
. It is a real linear
space of dimension 2mn.
Remark. The set of mn complex matrices is denoted C
m×n
. It is a complex
linear space of dimension mn.
4.2 Matrix-matrix multiplication
If A ∈ R
m×r
and B ∈ R
r×n
, then we can define the product C = AB, which
will be a matrix in R
m×n
. Note that this product is only defined if the number
of columns in A is equal to the number of rows in B.
The elements of the product matrix C are given by
c
ij
=
r

k=1
a
ik
b
kj
. (67)
For example, if
A =
_
¸
¸
_
a
11
a
12
a
21
a
22
a
31
a
32
a
41
a
42
_
¸
¸
_
, B =
_
b
11
b
12
b
13
b
21
b
22
b
23
_
, (68)
then C ∈ R
3×4
and
C =
_
¸
¸
_
c
11
c
12
c
13
c
21
c
22
c
23
c
31
c
32
c
33
c
41
c
42
c
43
_
¸
¸
_
, (69)
where, for example,
c
11
= a
11
b
11
+ a
12
b
21
,
c
12
= a
11
b
12
+ a
12
b
22
,
.
.
.
c
43
= a
41
b
13
+ a
42
b
23
.
39
U. Washington AMATH 352 - Spring 2009
Figure 5: Illustration of Matrix-Matrix product (Diagram from Wikipedia).
Figure 6: Example of Matrix-Matrix product (Diagram from Wikipedia).
The matrix-matrix product is illustrated in Figure 5. A numerical example is
depicted in Figure 6. The diagram highlights the origins of each coefficient.
Written in terms of columns, the product is
_
_
c
1
¸
¸
¸
¸
¸
¸

¸
¸
¸
¸
¸
¸
c
n
_
_
= A
_
_
b
1
¸
¸
¸
¸
¸
¸

¸
¸
¸
¸
¸
¸
b
n
_
_
=
_
_
Ab
1
¸
¸
¸
¸
¸
¸

¸
¸
¸
¸
¸
¸
Ab
n
_
_
.
(70)
Remark. In the case where n = 1, B only has one column, the matrix-matrix
multiplication AB agrees with matrix-vector multiplication.
Remark. Yet another way to view matrix-matrix multiplication is in terms of
rows. The ith row of the product C = AB is the ith row of A multiplied by
the matrix B.
Suppose A, B ∈ R
n×n
are both square and of the same size. Then the
products AB and BA are both defined. Each product is again an n n ma-
trix. Note, however, that in general AB ,= BA. Matrix multiplication is not
commutative in general!
40
U. Washington AMATH 352 - Spring 2009
Example 76. Let
A =
_
1 2
0 3
_
, B =
_
4 5
1 0
_
. (71)
Then
AB =
_
6 5
3 0
_
, BA =
_
4 23
1 2
_
. (72)
Definition 77. Consider A, B ∈ R
n×n
. When the products AB and BA are
equal, we say that A and B commute.
Example 78. A particular example of matrix-matrix product is the outer prod-
uct. Consider the product of an m-dimensional column vector, u ∈ R
m×1
, with
an n-dimensional row vector, v ∈ R
1×n
. The outer product is an mn matrix
that can be written
_
_
u
_
_
_
v
1
v
n
¸
=
_
_
v
1
u
¸
¸
¸
¸
¸
¸

¸
¸
¸
¸
¸
¸
v
n
u
_
_
=
_
¸
_
u
1
v
1
u
1
v
n
.
.
.
.
.
.
u
n
v
1
u
n
v
n
_
¸
_
(73)
The columns are all multiple of the same vector u, and similarly, the rows
are all multiple of the same vector v.
4.3 Range and rank of a matrix
Definition 79. Let a
(1)
, , a
(n)
∈ R
m
denote the columns of a matrix A ∈
R
m×n
. The column space of A, also called the range of A, is the subspace of
R
m
spanned by the columns of A. It is denoted by
1(A) = span(a
(1)
, , a
(n)
). (74)
Definition 80. The dimension of the range of A is called the rank of A:
rank(A) = dim(1(A)). (75)
Note the following:
• rank(A) ≤ m, since 1(A) is a subspace of R
m
;
• rank(A) ≤ n, since 1(A) is spanned by the n columns of A. So a basis
for 1(A) has at most n vectors;
• rank(A) = n if and only if the columns of A are linearly independent. In
this case, the columns form a basis for 1(A);
• if rank(A) < n, then the columns are linearly dependent and there exists
a vector z ,=
¯
0 such that Az =
¯
0 = 0
m×1
.
41
U. Washington AMATH 352 - Spring 2009
Similarly, the row rank of a matrix is the dimension of of the space spanned
by its rows. Row rank always equals column rank. So we refer to this number
simply as the rank of a matrix. An mn matrix of full rank is a matrix with
the maximal possible rank (min(m, n)). For example, a matrix of full rank with
m ≥ n must have n linearly independent columns.
Remark. The rank is defined similarly for complex matrices in C
m×n
.
Example 81. Consider the matrix
A =
_
_
2 6
−5 3
1 2
_
_
.
We denote a
1
and a
2
the two columns of A. The range of A is spanned by these
two column vectors:
1(A) = span(a
1
, a
2
).
To find the dimension of 1(A), we need to find a basis. The dimension will
be the number of vectors in the basis. To get a basis, we beed to find linearly
independent vectors that span the range of A. Assume there exists a linear
combination of a
1
and a
2
equal to the zero vector:
α
_
_
2
−5
1
_
_
+ β
_
_
6
3
2
_
_
=
_
_
0
0
0
_
_

_
_
_
2α + 6β = 0
−5α + 3β = 0
α + 2β = 0
The last equation gives that α = −2β. Replacing this value into the first
equation gives 6β − 4β = 0, which implies that β = 0. Consequently, we have
α = β = 0 and the two column vectors are linearly independent. So (a
1
, a
2
) is
a basis for 1(A). The rank of A is 2.
Example 82. Consider the matrix
A =
_
3 0
2 4
_
.
We denote a
1
and a
2
the two columns of A. The range of A is spanned by these
two column vectors:
1(A) = span(a
1
, a
2
).
To find the dimension of 1(A), we need to find a basis. The dimension will
be the number of vectors in the basis. To get a basis, we beed to find linearly
independent vectors that span the range of A. Assume there exists a linear
combination of a
1
and a
2
equal to the zero vector:
α
_
3
2
_
+ β
_
0
4
_
=
_
0
0
_

_
3α = 0
2α + 4β = 0
The first equation gives that α = 0. Replacing this value into the last equation
gives β = 0. The two column vectors are linearly independent. So (a
1
, a
2
) is a
basis for 1(A). The rank of A is 2.
42
U. Washington AMATH 352 - Spring 2009
Example 83. Consider the matrix
A =
_
_
3 0 −1
2 −8 2
−1 −2 1
_
_
.
We denote a
1
, a
2
, and a
3
the three columns of A. The range of A is spanned
by these three column vectors:
1(A) = span(a
1
, a
2
, a
3
).
To find the dimension of 1(A), we need to find a basis. The dimension will
be the number of vectors in the basis. To get a basis, we beed to find linearly
independent vectors that span the range of A. Assume there exists a linear
combination of a
1
, a
2
, and a
3
equal to the zero vector:
α
_
_
3
2
−1
_
_
+ β
_
_
0
−8
−2
_
_
+ γ
_
_
−1
2
1
_
_
=
_
_
0
0
0
_
_

_
_
_
3α −γ = 0
2α −8β + 2γ = 0
−α −2β + γ = 0
The first equation gives that γ = 3α. Replacing this value into the second and
third equation gives
_
_
_
γ = 3α
2α −8β + 6α = 0
−α −2β + 3α = 0

_
_
_
γ = 3α
8α −8β = 0
2α −2β = 0
which implies that β = α but leaves the value of α arbitrary. For example, we
can choose α = 1, β = 1, and γ = 3. Then we have
1
_
_
3
2
−1
_
_
+ 1
_
_
0
−8
−2
_
_
+ 3
_
_
−1
2
1
_
_
=
_
_
3 + 0 −3
2 −8 + 6
−1 −2 + 3
_
_
=
_
_
0
0
0
_
_
.
These three vectors are linearly dependent. So the rank of A is smaller than 3.
To check whether the rank is 2, we need to find two vectors that are linearly
independent. Three choices are possible. Here we choose a
1
and a
2
. Assume
there exists a linear combination ofa
1
and a
2
equal to the zero vector:
α
_
_
3
2
−1
_
_
+ β
_
_
0
−8
−2
_
_
=
_
_
0
0
0
_
_

_
_
_
3α = 0
2α −8β = 0
−α −2β = 0
which implies that α = 0. Then we get α = β = 0 and the two column vectors
are linearly independent. So (a
1
, a
2
) is a basis for 1(A). The rank of A is 2.
Exercise 84. Compute the rank of the following matrices
_
_
2 6
1 3
2/3 2
_
_
,
_
1 2
3 4
_
,
_
1 0 2
−1 3 1
_
.
Exercise 85. Compute the range and rank of a nonzero outer product.
43
U. Washington AMATH 352 - Spring 2009
4.4 Null space of a matrix
Note that the set of all vectors z for which Az =
¯
0 is a linear space. Since if
Az =
¯
0 and Aw =
¯
0, then
A(αz + βw) = αAz + βAw =
¯
0.
This space is called the null space of A.
Definition 86. The null space of A ∈ R
m×n
is the subspace of R
n
defined by
^(A) = ¦z ∈ R
n
; Az =
¯
0¦ . (76)
Note the following:
• if rank(A) = n, then ^(A) = ¦
¯
0¦ ;
• if rank(A) < n, then ^(A) has dimension at least 1. In fact, we can show
dim(^(A)) = n −rank(A). (77)
This result is sometimes referred to as the rank-nullity theorem or the
dimension theorem.
Example 87. Consider the matrix
A =
_
_
2 6
−5 3
1 2
_
_
.
We have seen that the rank of this matrix is 2. The rank-nullity theorem gives
dim^(A) = 2 −2 = 0.
The matrix A is full rank. The dimension of its null space is 0. We have
^(A) = ¦0¦ . There is no basis for this null space.
Example 88. Consider the matrix
A =
_
3 0
2 4
_
.
We have seen that the rank of this matrix is 2. The rank-nullity theorem gives
dim^(A) = 2 −2 = 0.
The matrix A is full rank. The dimension of its null space is 0. We have
^(A) = ¦0¦ . There is no basis for this null space.
44
U. Washington AMATH 352 - Spring 2009
Example 89. Consider the matrix
A =
_
_
3 0 −1
2 −8 2
−1 −2 1
_
_
.
We have seen that the rank of this matrix is 2. The rank-nullity theorem gives
dim^(A) = 3 −2 = 1.
The dimension of its null space is 1. We have ^(A) = span(z), where z is a
nonzero vector. Assume z =
_
_
z
1
z
2
z
3
_
_
. We have
Az = 0 ⇒
_
_
3 0 −1
2 −8 2
−1 −2 1
_
_
_
_
z
1
z
2
z
3
_
_
=
_
_
3z
1
−z
3
2z
1
−8z
2
+ z
3
−z
1
−2z
2
+ z
3
_
_
The first equation gives that z
3
= 3z
1
. Replacing this value into the second and
third equation gives
_
_
_
z
3
= 3z
1
2z
1
−8z
2
+ 6z
1
= 0
−z
1
−2z
2
+ 3z
1
= 0

_
_
_
z
3
= 3z
1
8z
1
−8z
2
= 0
2z
1
−2z
2
= 0
which implies that z
2
= z
1
but leaves the value of z
1
arbitrary. It is normal
that z
1
is arbitrary because the dimension of the null space is 1. So we should
expect 1 parameter. For example, we can choose z
1
= 1, z
2
= 1, and z
3
= 3.
Then we have
_
_
3 0 −1
2 −8 2
−1 −2 1
_
_
_
_
1
1
3
_
_
=
_
_
3 + 0 −3
2 −8 + 6
−1 −2 + 3
_
_
=
_
_
0
0
0
_
_
.
This vector is nonzero. It spans the null space. So a basis of ^(A) is
_
_
_
_
1
1
3
_
_
_
_
.
Example 90. Consider the matrix
A =
_
_
1 1 1
1 1 1
1 1 1
_
_
.
Assume z =
_
_
z
1
z
2
z
3
_
_
is a nonzero vector belonging to the null space of A. We
have
Az = 0 ⇒
_
_
1 1 1
1 1 1
1 1 1
_
_
_
_
z
1
z
2
z
3
_
_
=
_
_
z
1
+ z
2
+ z
3
z
1
+ z
2
+ z
3
z
1
+ z
2
+ z
3
_
_
45
U. Washington AMATH 352 - Spring 2009
All three equations give that z
3
= −z
1
− z
2
. So a general formula for the null
space of A is
^(A) =
_
_
_
_
_
z
1
z
2
−z
1
−z
2
_
_
; ∀z
1
, z
2
∈ R
_
_
_
.
We look for a basis for ^(A). We notice that this formula depends on two
parameters z
1
and z
2
. We have
∀z ∈ ^(A), z = z
1
_
_
1
0
−1
_
_
+ z
2
_
_
0
1
−1
_
_
.
So we can write
^(A) = span
_
_
_
_
1
0
−1
_
_
,
_
_
0
1
−1
_
_
_
_
.
Next we check whether these two vectors are linearly independent. Consider a
linear combination of these two vectors equal to the zero vector
α
_
_
1
0
−1
_
_
+ β
_
_
0
1
−1
_
_
=
_
_
0
0
0
_
_

_
_
_
α = 0
β = 0
−α −β = 0
which implies that α = β = 0. So these two vectors are linearly independent.
They form a basis for ^(A). So the dimension of the null space is 2. Using the
rank nullity theorem, we get that the rank of A is equal to 1.
Exercise 91. Compute the null space of the following matrix
_
_
2 6
1 3
2/3 2
_
_
,
_
1 2
3 4
_
,
_
1 0 2
−1 3 1
_
.
4.5 Transpose/Adjoint of a matrix
Definition 92. The transpose of an mn matrix A, denoted A
T
, is the nm
matrix whose (i, j) entry is the (j, i) entry of A.
For example,
A =
_
_
a
11
a
12
a
21
a
22
a
31
a
32
_
_
=⇒ A
T
=
_
a
11
a
21
a
31
a
12
a
22
a
32
_
. (78)
Definition 93. When A = A
T
, the matrix A is called symmetric. When
A = −A
T
, the matrix A is called skew-symmetric.
46
U. Washington AMATH 352 - Spring 2009
Note that rank(A
T
) = rank(A), since the column rank and row rank are
equal. However, the dimensions for ^(A) and ^(A
T
) can differ. For example,
the matrix
A =
_
3 0 −1
2 4 6
_
has a null space of dimension 1 and its transpose a null space of dimension 0.
The null spaces are
^(A) = span
_
_
_
_
1
−5
3
_
_
_
_
^(A
T
) = ¦
¯
0¦ = ¦0
2×1
¦.
Remark 94. A symmetric matrix must be square.
Note that we have:

_
A
T
_
T
= A;
• the identity matrix is a symmetric matrix;
• a diagonal matrix, whose non-zero entries are only on the diagonal, is
symmetric.
If A ∈ R
m×r
and B ∈ R
r×n
, then the product C = AB ∈ R
m×n
exists. We
can take the transpose of this matrix and will get an nm matrix C
T
∈ R
n×m
.
Note that in this case B
T
∈ R
n×r
and A
T
∈ R
r×m
. So the matrix product
B
T
A
T
is defined and is an n m matrix. In fact, this is just equal to C
T

R
n×m
. So, in general, it is true that
(AB)
T
= B
T
A
T
. (79)
The transpose of the product is the product of the transposes, but with the
order reversed!
If A ∈ R
m×n
is a matrix, then the product A
T
A exists and it is a square
matrix of dimension n. The matrix A
T
A is a symmetric matrix. Indeed, we
have
_
A
T
A
_
T
= (A)
T
_
A
T
_
T
= A
T
A.
Definition 95. The adjoint of a complex mn matrix A, denoted A

, is the
n m matrix whose (i, j) entry is the conjugate (j, i) entry of A,
(A

)
ij
= a
ij
, (80)
obtained by negating the imaginary part of a
ij
.
Definition 96. When A = A

, the matrix A is called hermitian. When A =
−A

, the matrix A is called skew-hermitian.
Note that an hermitian matrix must be square. Similarly to the transpose,
the adjoint of the product of matrices is the product of the adjoint, with the
order reversed.
47
U. Washington AMATH 352 - Spring 2009
Exercise 97. Let A be a square real matrix. Show that A+A
T
is a symmetric
matrix. Show that A−A
T
is a skew-symmetric matrix. Prove that any square
matrix is the sum of a symmetric and a skew-symmetric matrix.
Exercise 98. Characterize the diagonal entries of a real skew-symmetric ma-
trix.
Exercise 99. Characterize the diagonal entries of a complex hermitian matrix.
Consider x ∈ R
3
,
x =
_
_
x
1
x
2
x
3
_
_
,
then we have
x
T
x =
_
x
1
x
2
x
3
¸
_
_
x
1
x
2
x
3
_
_
= x
2
1
+ x
2
2
+ x
2
3
.
We have proved that, for x ∈ R
3
,
|x|
2
=

x
T
x. (81)
This relation is also true in R
m
. For the inner product defined in (14), we
similarly prove that
∀x, y ∈ R
m
, x y = x
T
y. (82)
In C
m
, we replace the transpose operation
T
with the adjoint operator

.
4.6 Matrix inverse
The n n matrix that has 1’s on the diagonal and 0’s everywhere else is called
the identity matrix and denoted by I.
It is called the identity matrix because for any x ∈ R
n
,
Ix = x.
Also, if B ∈ R
n×n
,
IB =
_
_
Ib
1
¸
¸
¸
¸
¸
¸

¸
¸
¸
¸
¸
¸
Ib
n
_
_
=
_
_
b
1
¸
¸
¸
¸
¸
¸

¸
¸
¸
¸
¸
¸
b
n
_
_
= B.
So multiplying a matrix by I leaves it unchanged.
More generally, if B ∈ R
m×n
then multiplying B on the left by an m m
identity matrix or on the right by an nn identity matrix leaves B unchanged.
Definition 100. A nonsingular or invertible matrix is a square matrix of full
rank.
48
U. Washington AMATH 352 - Spring 2009
Note that a full rank n n square matrix has columns that form a basis
for the linear space R
n
(or C
n
for complex matrices). Therefore, any vector in
R
n
has a unique expression as a linear combination of the column vectors. In
particular, every basis vector e
j
of the standard unit basis (33) has a unique
expansion in column vectors,
e
j
=
n

i=1
a
i
z
ij
. (83)
Let z
j
denote the column vector with entries z
ij
. Then we have
e
j
= Az
j
and combining all the vectors z
j
in the matrix Z, we get
_
_
e
1
¸
¸
¸
¸
¸
¸

¸
¸
¸
¸
¸
¸
e
n
_
_
= I = AZ.
The matrix Z is called the inverse matrix of A and written as A
−1
. It should
not be confused with the additive inverse of A that is −A. The matrix A
−1
is
the inverse for the multiplication and satisfies the relation
AA
−1
= I. (84)
It is the matrix version of the familiar expression
aa
−1
= 1
for any nonzero scalar a ∈ R.
Theorem 101. For A ∈ R
n×n
, the following conditions are equivalent:
• A has a unique inverse A
−1
such that AA
−1
= A
−1
A = I,
• rank(A) = n,
• 1(A) = R
n
,
• ^(A) = ¦
¯
0¦.
A similar result holds for complex matrices when replacing R with C.
For any nonzero scalar a ∈ R, the expression
aa
−1
= a
−1
a = 1
holds. The same is true for invertible matrices, i.e.
AA
−1
= A
−1
A = I. (85)
49
U. Washington AMATH 352 - Spring 2009
Starting from (84), we multiply on the right by A to get
AA
−1
A = A and A
_
A
−1
Ax −x
_
= 0 ∀x ∈ R
n
.
Since the null space of A is trivial, we get
A
−1
Ax = x ∀x ∈ R
n
,
proving that A
−1
A = I. The inverse matrix commutes with A and the product
in either order is the identity matrix.
Corollary 102. If Ais invertible, then A
−1
is invertible and we have
_
A
−1
_
−1
=
A .
Recall, from Section 4.5, that the transpose of a matrix product is the in-
verted product of the transposes. A similar formula holds for inverses in the
square case. If A, B ∈ R
n×n
are both nonsingular, then so is their product AB
and
(AB)
−1
= B
−1
A
−1
. (86)
Again the order is reversed. Indeed, we can write
B
−1
A
−1
AB = B
−1
_
A
−1
A
_
B = B
−1
IB = B
−1
B = I
and use the uniqueness of the inverse matrix. If either A or B is singular, then
the products AB and BAwill also be singular and noninvertible. We emphasize
that the inverse of a matrix is defined only for square matrices. So equation
(86) holds only when A and B are both invertible and, in particular, square.
Note that, for any square invertible matrix A, we have
_
A
T
_
−1
=
_
A
−1
_
T
. (87)
Proposition 103. The inverse of a 2 2 matrix is equal to
_
a b
c d
_
−1
=
1
ad −bc
_
d −b
−c a
_
(88)
if and only if ad −bc ,= 0.
Exercise 104. Prove formula (88).
The scalar ad −bc is called the determinant of the matrix,
det
__
a b
c d
__
= ad −bc. (89)
A 2 2 matrix is invertible if and only if its determinant is nonzero. The
determinant can be defined for any square nn matrix. For example, for 3 3
50
U. Washington AMATH 352 - Spring 2009
matrices, we have
det
_
_
_
_
a b c
d e f
g h i
_
_
_
_
= a det
__
e f
h i
__
−b det
__
d f
g i
__
+ c det
__
d e
g h
__
det
_
_
_
_
a b c
d e f
g h i
_
_
_
_
= a(ei −hf) −b(di −gf) + c(dh −ge)
det
_
_
_
_
a b c
d e f
g h i
_
_
_
_
= (aei + bfg + cdh) −(ahf + bdi + cge)
We can generalize that a n n matrix is invertible if and only if its de-
terminant is nonzero. However, the determinant rarely finds a useful role in
numerical algorithms.
The determinant satisfies the following properties:
• det(I) = 1;
• det(AB) = det(A) det(B);
• det(A
T
) = det(A);
• det(A
−1
) = 1/ det(A);
• det(αA) = α
n
det(A), where A is a square matrix of dimension n.
Remark. When writing the product x = A
−1
b, we should not think of x =
A
−1
b as the result of applying A
−1
to b. Instead, we should understand x as
the unique vector that satisfies the equation Ax = b. This means that x is
the vector of coefficients of the unique linear combination of b in the basis of
columns of A. Multiplication by A
−1
is a change of basis operation.
Remark. Numerically, we rarely work with inverse matrices. If we knew the
inverse matrix, then we could solve any linear system Ax = b simply by mul-
tiplying b by A
−1
. However, in practice, there are better ways to solve the
system (e.g., Gaussian elimination) that require less work than computing the
inverse matrix and often give more accurate solutions (when the rounding errors
of computer arithmetic are taken into account).
4.7 Orthogonal/Unitary matrices
Orthogonal and unitary matrices play an important role in numerical algo-
rithms. We give their definitions here. Some of their properties will be described
later in the notes.
Definition 105. A square n n real matrix O is orthogonal if O
T
= O
−1
or
O
T
O = I.
51
U. Washington AMATH 352 - Spring 2009
Definition 106. A square n n complex matrix Q is unitary if Q

= Q
−1
or
Q

Q = I.
When n = 2, the orthogonal matrices are
_
cos θ sin θ
−sin θ cos θ
_
and
_
cos θ sin θ
sin θ −cos θ
_
(90)
where θ belongs to R.
Let O denote an n n real orthogonal matrix and x an n 1 real vector.
The 2-norm of Ox is equal to
|Ox|
2
=
_
(Ox)
T
Ox =

x
T
O
T
Ox =

x
T
x = |x|
2
. (91)
The multiplication by an orthogonal matrix preserves the 2-norm of a vector.
Orthogonal matrices preserve also the inner product (14). Note that they do
not preserve the ∞-norm or the 1-norm. For example,
_
_
_
_
_
1
0
__
_
_
_

= 1 and
_
_
_
_
_ √
2/2

2/2


2/2

2/2
_ _
1
0
__
_
_
_

=

2
2
.
In C
m
, the unitary matrices preserve the 2-norm of complex vectors and the
inner product.
4.8 Useful commands in Matlab
Here are a few commands in Matlab useful for this section.
• C = A*B computes the matrix-matrix product.
• rank(A) computes the rank of a matrix.
• null(A) generates an orthonormal basis for the null space of A.
• A’ computes the adjoint matrix A

. When A has only real entries, the
adjoint matrix A

is equal to the transpose matrix A
T
. To compute the
matrix (without the complex conjuguate), Matlab uses A.’.
• I = eye(n) creates the identity matrix in R
n×n
.
• inv(A) computes the inverse of matrix A when it exists.
• det(A) computes the determinant of a matrix.
52
U. Washington AMATH 352 - Spring 2009
5 Norms and Inner Products
In this section, we generalize the norms and the inner product defined on R
m
(see Section 1.3) to linear spaces. In particular, we discuss norm and inner
products for matrices.
5.1 Norms
Definition 107. Consider U a real linear space. A norm || is a map from
U →R
+
, satisfying the following conditions :
∀u ∈ U, |u| ≥ 0, (92a)
|u| = 0 if and only if u =
¯
0, (92b)
∀α ∈ R, |αu| = [α[ |u| , (92c)
∀u
(1)
, u
(2)
∈ U,
_
_
_u
(1)
+ u
(2)
_
_
_ ≤
_
_
_u
(1)
_
_
_ +
_
_
_u
(2)
_
_
_ (Triangle inequality). (92d)
To define a norm for matrices, several approaches are possible. First, we
look at the matrix just through its entries. In that case, the resulting matrix
norms are very similar to vector norms. For example, the map
A = (a
ij
)
m×n
−→
_
_
m

i=1
n

j=1
[a
ij
[
p
_
_
1/p
is a norm on R
m×n
. Note that it is similar to the vector p-norm on R
mn
.
Example 108. Consider the linear space of real matrices U = R
m×n
and the
matrix A = (a
ij
)
m×n
. The map,
|A|
F
=
¸
¸
¸
_
m

i=1
n

j=1
[a
ij
[
2
, (93)
is called the Frobenius norm. It is similar to the 2-norm of R
mn
. It is easy to
show that this map is a norm according to Definition 107. The Frobenius norm
can be used to bound the product of matrices. Let C = AB with entries c
ij
and let a
row,i
be the ith row of A and b
j
the jth column of B. Then we have
c
ij
= a
row,i
b
j
. So, by the Cauchy-Schwarz inequality, we have
[c
ij
[ ≤ |a
row,i
|
2
|b
j
|
2
.
53
U. Washington AMATH 352 - Spring 2009
Squaring both sides and summing over i and j, we get
|AB|
2
F
=
m

i=1
n

j=1
[c
ij
[
2
,

m

i=1
n

j=1
|a
row,i
|
2
2
|b
j
|
2
2
,

_
m

i=1
|a
row,i
|
2
2
_
_
_
n

j=1
|b
j
|
2
2
_
_
= |A|
2
F
|B|
2
F
.
Another approach to define a matrix norm is to assess the effect of the
matrix on the length of a vector. This approach is intimately related to a vector
norm. The resulting norm is called the induced or associated matrix norm.
Considering all the nonzero vectors x, the induced matrix norm compares the
length of the vectors x and Ax and is equal to the largest possible ratio between
these lengths.
Definition 109. Consider the linear space of real matrices R
m×n
and the vector
norms ||
(n)
on the domain space R
n
→ R
+
and ||
(m)
on the range space
R
m
→R
+
. The induced matrix norm is defined as
|A|
(m,n)
= max
x=0
|Ax|
(m)
|x|
(n)
= max
x
(n)
=1
|Ax|
(m)
. (94)
The induced matrix norm is the largest value of the ratios |Ax|
(m)
/ |x|
(n)
.
It is easy to verify that ||
(m,n)
satisfy the properties (92) defining a norm.
Note that we used the subscripts (n) and (m) to avoid any confusion with the
vector p-norms.
Sometimes, the ratio |Ax|
(m)
/ |x|
(n)
is called the amplification factor or
gain in the direction x. So the norm |A|
(m,n)
is the maximum gain over all
directions.
Exercise 110. Prove that the norm ||
(m,n)
satisfies the properties (92).
Example 111. Consider the matrix A =
_
2 0
0 1
_
. We have the following
relations using, for example, the 1-norm of the vectors, i.e. the (1,1) induced
matrix norm:

_
2 0
0 1
_ _
1
0
_
=
_
2
0
_

_
_
_
_
_
1
0
__
_
_
_
1
= 1
_
_
_
_
A
_
1
0
__
_
_
_
1
= 2. The
ratio is here 2.

_
2 0
0 1
_ _
0
1
_
=
_
0
1
_

_
_
_
_
_
0
1
__
_
_
_
1
= 1
_
_
_
_
A
_
0
1
__
_
_
_
1
= 1. The
ratio is here 1.
54
U. Washington AMATH 352 - Spring 2009

_
2 0
0 1
_ _
1
1
_
=
_
2
1
_

_
_
_
_
_
1
1
__
_
_
_
1
= 2
_
_
_
_
A
_
1
1
__
_
_
_
1
= 3. The
ratio is here
3
/2.

_
2 0
0 1
_ _
2
1
_
=
_
4
1
_

_
_
_
_
_
2
1
__
_
_
_
1
= 3
_
_
_
_
A
_
2
1
__
_
_
_
1
= 5. The
ratio is here
5
/3.
From these examples, we see that the ratio depends on the vector x. It seems
that 2 will be the largest one. It is, indeed, the case. Consider an arbitrary
vector x. Then we have
x =
_
x
1
x
2
_
|x|
1
= [x
1
[ +[x
2
[
and
Ax =
_
2x
1
x
2
_
|Ax|
1
= 2 [x
1
[ +[x
2
[ .
So the ratio is equal to
|Ax|
1
|x|
1
=
2 [x
1
[ +[x
2
[
[x
1
[ +[x
2
[

2 [x
1
[ + 2 [x
2
[
[x
1
[ +[x
2
[
= 2.
This bound shows that the ratio can not be larger than 2. Since we have found
one vector for which the ratio is exactly 2, the (1,1) induced matrix norm for
A is equal to 2. This norm value indicates that the matrix A can multiply
the length of a vector by 2, at most. It could be less than 2 for some vectors.
However, it does not say anything about the direction of the vector.
Example 112. Let A be an m n matrix. The (1, 1) induced norm is often
denoted |A|
(1,1)
= |A|
1
. This notation is not confusing with the vector 1-norm
because A is a matrix. For any vector x ∈ R
n
, we have
|Ax|
1
=
_
_
_
_
_
_
n

j=1
a
j
x
j
_
_
_
_
_
_
1

n

j=1
[x
j
[ |a
j
|
1
≤ |x|
1
max
1≤j≤n
|a
j
|
1
,
where a
j
is the jth colulumn vector of A. Note that |a
j
|
1
is the vector 1-norm
because a
j
is a vector. Therefore, the induced matrix 1-norm satisfies
|A|
1
≤ max
1≤j≤n
|a
j
|
1
.
By choosing x = e
j
, the unit standard basis vector of , where j is maximizing
|a
j
|
1
, we attain the bound. Thus the matrix 1-norm is
|A|
1
= max
1≤j≤n
|a
j
|
1
= max
1≤j≤n
m

i=1
[a
ij
[ , (95)
which is equal to the maximum column sum. We can write
|Ax|
1
≤ |A|
1
|x|
1
.
55
U. Washington AMATH 352 - Spring 2009
Example 113. Let A be an mn matrix. The (∞, ∞) induced norm is often
denoted |A|
(∞,∞)
= |A|

. This notation is not confusing with the vector
∞-norm because A is a matrix. The matrix ∞-norm is
|A|

= max
1≤i≤m
n

j=1
[a
ij
[ , (96)
which is equal to the maximum row sum. We can write
|Ax|

≤ |A|

|x|

.
When comparing equations (95) and (96), we remark that, for any matrix
A ∈ R
m×n
, we have
|A|

=
_
_
A
T
_
_
1
. (97)
A similar property holds for complex matrices when using the adjoint matrix
A

.
Example 114. Let A be an mn matrix. The (2, 2) induced norm is denoted
|A|
(2,2)
= |A|
2
. This notation is not confusing with the vector 2-norm because
A is a matrix. The matrix 2-norm is defined by
|A|
2
= max
x=0
|Ax|
2
|x|
2
=
¸
max
x=0
x
T
A
T
Ax
x
T
x
. (98)
We can write
|Ax|
2
≤ |A|
2
|x|
2
.
Later in the notes, we will introduce the spectral radius that describes precisely
the 2-norm of a matrix. For the moment, just remember that the 2-norm of a
matrix A is related to the matrix A
T
A. As a consequence of (98), we remark
that the 2-norm of a real matrix is invariant under multiplication by orthogonal
matrices,
∀A ∈ R
m×n
, |OA|
2
= |A|
2
, ∀O ∈ R
m×m
satisfying O
T
O = I. (99)
The 2-norm of a complex matrix is invariant under multiplication by unitary
matrices.
Similarly to norms in R
m
, all the matrix norms are related to each other.
Recall that we have
|x|

≤ |x|
2


n|x|

, (100)
for any vector x ∈ R
n
. We can write also, for x ,= 0,
1

n|x|


1
|x|
2

1
|x|

.
For any vector y ∈ R
m
, we have
|y|

≤ |y|
2


m|y|

.
56
U. Washington AMATH 352 - Spring 2009
For any matrix in R
m×n
, we have
1

n
|A|

≤ |A|
2


m|A|

. (101)
Let us prove the right side of the bound. Consider any non-zero vector x ∈ R
n
.
We have
|Ax|
2
|x|
2


m
|Ax|

|x|
2


m
|Ax|

|x|



m|A|

(102)
where we used twice the equation (100) and where the last inequality is due to
the supremum. So we have
∀ x ∈ R
n
, x ,= 0,
|Ax|
2
|x|
2


m|A|

.
Note that the supremum is also the minimal upper bound. So we obtain
|A|
2


m|A|

.
Next, we consider the left hand side. We have
|Ax|

|x|


|Ax|
2
|x|


|Ax|
2
1

n
|x|
2


n|A|
2
(103)
where we used twice the equation (100) and where the last inequality is due to
the supremum. So we have
∀ x ∈ R
n
, x ,= 0,
|Ax|

|x|



n|A|
2
.
Using again the supremum as the minimal upper bound, we obtain
1

n
|A|

≤ |A|
2
.
Exercise 115. Prove that, for any mn matrix, we have
1
n
|A|

≤ |A|
1
≤ m|A|

.
Exercise 116. Prove that, for any mn matrix, we have
1

m
|A|
1
≤ |A|
2


n|A|
1
.
Exercise 117. Prove that, for any mn matrix, we have
|A|
2
≤ |A|
F


n|A|
2
.
57
U. Washington AMATH 352 - Spring 2009
The induced matrix norm of a matrix product can also be bounded. Let A
be an m r real matrix and B be an r n real matrix. For any x ∈ R
n
, we
have
|ABx|
m
≤ |A|
(m,r)
|Bx|
r
≤ |A|
(m,r)
|B|
(r,n)
|x|
n
.
Therefore, the induced matrix norm satisfy
|AB|
(m,n)
≤ |A|
(m,r)
|B|
(r,n)
. (104)
In particular, we have
_
_
_
|AB|
1
≤ |A|
1
|B|
1
|AB|
2
≤ |A|
2
|B|
2
|AB|

≤ |A|

|B|

(105)
Exercise 118. Consider U = R
n
and W a nonsingular matrix. Show that the
map
|x|
W
= |Wx|
2
is a norm.
Remark. When U is a complex linear space, a norm remains a map from U →
R
+
. The notions introduced in this Section can be extended to C
m×n
.
Exercise 119. Consider U = C
0
([0, 1], R). Show that the map
f −→
¸
ˆ
1
0
f(t)
2
dt
is a norm.
5.2 Inner products
Definition 120. Consider U a real linear space. An inner product < , > is a
map U U →R satisfying the following properties
∀u ∈ U, < u, u >≥ 0, (106a)
< u, u >= 0 if and only if u =
¯
0, (106b)
∀α ∈ R, ∀u, v ∈ U, < αu, v >= α < u, v >, (106c)
∀u, v ∈ U, < u, v >=< v, u >, (106d)
∀u, v, w ∈ U, < u + v, w >=< u, w > + < v, w > . (106e)
An inner product is a symmetric positive-definite bilinear form.
Exercise 121. Prove that an inner product < , > on a real linear space U
satisfies
∀u, v, w ∈ U, ∀α, β ∈ R, < u, αv + βw >= α < u, v > +β < u, w > . (107)
58
U. Washington AMATH 352 - Spring 2009
Example 122. The Euclidean inner product of two column vectors in R
m
(14)
satisfy the properties (113).
The Cauchy-Schwarz inequality holds for any inner product. For any vectors
u and v of U, we write
[< u, v >[ ≤

< u, u >

< v, v >, (108)
with equality if and only if v is proportional to u. The proof uses
< u −λv, u −λv >≥ 0
with
λ =
< u, v >
< v, v >
.
Exercise 123. Prove the Cauchy-Schwarz inequality (108).
Proposition 124. Given U a real linear space and an inner product < , >
on U. The map from U to R
+
, defined by
u −→

< u, u >, (109)
is a norm.
Exercise 125. Prove Proposition 124.
Example 126. Consider U = R
m×n
. The map
(A, B) −→tr(A
T
B) (110)
is an inner product. It satisfies also
tr(A
T
A) = |A|
2
F
. (111)
From this last expression, we remark that the Frobenius norm is also invariant
under multiplication by orthogonal matrices,
∀A ∈ R
m×n
, |OA|
F
= |A|
F
, ∀O ∈ R
m×m
satisfying O
T
O = I. (112)
The associated inner product is also invariant under multiplication by an or-
thogonal matrix,
tr
_
(OA)
T
OB
_
= tr(A
T
O
T
OB) = tr(A
T
B).
Exercise 127. Consider U = C
0
([0, 1], R). Show that the map
(f, g) −→
ˆ
1
0
f(t)g(t)dt
is an inner product.
59
U. Washington AMATH 352 - Spring 2009
The concept of orthogonality also extends to general real linear spaces.
Definition 128. Let U be a real linear space. We say that u, v ∈ U are
orthogonal for the inner product < , > when < u, v >= 0, i.e. their inner
product is 0.
We conclude by giving the definition of an inner product on a complex linear
space. The Cauchy-Schwarz inequality and the orthogonality still holds in the
complex case.
Definition 129. Consider U a complex linear space. An inner product < , >
is a map U U →C satisfying the following properties
∀u ∈ U, < u, u >≥ 0, (113a)
< u, u >= 0 if and only if u =
¯
0, (113b)
∀α ∈ R, ∀u, v ∈ U, < αu, v >= α < u, v >, (113c)
∀u, v ∈ U, < u, v >= < v, u >, (113d)
∀u, v, w ∈ U, < u + v, w >=< u, w > + < v, w > . (113e)
An inner product is a symmetric positive-definite sesquilinear form.
5.3 Errors
In order to discuss the accuracy of a numerical solution or the relative merits
of one numerical method versus another, it is necessary to choose a manner of
measuring the error. It may seem obvious what is meant by the error. But
there are often many different ways to measure the error that can sometimes
give quite different impressions as to the accuracy of an approximate solution.
Consider a problem to which the true answer is a vector x ∈ R
m
. Denote
an approximation by ˆ x. Then the error in this approximation is
e = x − ˆ x.
5.3.1 Absolute error
A natural measure of this error would be the norm of e,
|e| = |x − ˆ x| .
This is called the absolute error in the approximation.
As an example, suppose that x = 2.2 while some numerical method produced
an approximate value ˆ x = 2.20345. Then the absolute error is
[x − ˆ x[ = 0.00345 = 3.45 10
−3
.
This seems quite reasonable — we have a fairly accurate solution with three
correct digits and the absolute error is fairly small, on the order of 10
−3
. We
60
U. Washington AMATH 352 - Spring 2009
might be very pleased with an alternative method that produced an error of
10
−6
and horrified with a method that produced an error of 10
6
.
But note that our notion of what is a large error or a small error might be
thrown off completely if we were to choose a different set of units for measuring
x. For example, suppose the x discussed above were measured in meters, so
x = 2.2 meters is the correct solution. But suppose that instead we expressed
the solution (and the approximate solution) in nanometers rather than meters.
Then the true solution is x = 2.2 10
9
and the approximate solution is ˆ x =
2.20345 10
9
, giving an absolute error of
[x − ˆ x[ = 3.45 10
6
.
We have an error that seems huge and yet the solution is just as accurate as
before, with three correct digits.
Conversely, if we measured x in kilometers, then x = 2.2 10
−3
and ˆ x =
2.20345 10
−3
, so
[x − ˆ x[ = 3.45 10
−6
.
The absolute error may seem much smaller, but there are still only three correct
digits in the approximation.
5.3.2 Relative error
The above difficulties arise from a poor choice of scaling of the problem. One
way to avoid this is to consider the relative error, defined by
|x − ˆ x|
|x|
.
The size of the error is scaled by the size of the value being computed.
For the above examples, the relative error in ˆ x is equal to
[2.2 −2.20345[
[2.2[
=
¸
¸
(2.2 −2.20345) 10
9
¸
¸
[2.2 10
9
[
= 1.57 10
−3
.
regardless of what units we choose. The relative error is “dimensionless” and its
value is always independent of the units chosen to measure x, a very desirable
feature.
Moreover, in general a relative error that is approximately 10
−k
indicates
that there are roughly k correct digits in the solution, matching our intuition.
For these reasons the relative error is often a better measure of accuracy
than the absolute error. Of course if we know that our problem is “properly
scaled”, so that the solution x has a magnitude that is order 1, then it is fine
to use the absolute error, which is roughly the same as the relative error in this
case.
In fact it is generally better to insure that the problem is properly scaled
than to rely on the relative error. Poorly scaled problems can lead to other
numerical difficulties, particularly if several different scales arise in the same
61
U. Washington AMATH 352 - Spring 2009
problem so that some numbers are orders of magnitude larger than others for
nonphysical reasons. Unless otherwise noted, we will generally assume that the
problem is scaled in such a way that the absolute error is meaningful.
5.4 Conditioning
Conditioning is a fundamental issue of numerical analysis that, until now, we
have skirted. It pertains to the perturbation behavior of a mathematical prob-
lem. A well-conditioned problem is one with the property that all small pertur-
bations of the input lead to only small changes in the ouput of the problem. An
ill-conditioned problem is one with the property that some small perturbation
of the input can lead to a large change in the ouput.
For example, data errors form a source of perturbations. A mathematical
model of some real world phenomenon typically involves some parameters or
other data describing the particular situation being modeled. These values are
almost never known exactly. There may be measurement errors or there may
be values that cannot be measured in any precise way and must be “guessed” by
the modeler. Errors in the data mean that even if the model is very good and
the equation is then solved exactly it may not match reality. It is important for
the numerical analyst to evaluate the effect of these data errors.
Consider the matrix
A =
1
2
_
1 1
1 + 10
−10
1 −10
−10
_
.
Its inverse matrix A
−1
is
A
−1
=
_
1 −10
10
10
10
1 + 10
10
−10
10
_
.
The linear system
Ax =
_
1
1
_
has the solution
x =
_
1
1
_
.
The linear system
A˜ x =
_
1.1
0.9
_
has the solution
˜ x =
_
1.1 −0.2 10
10
1.1 + 0.2 10
10
_
.
Here a relatively small variation in the right hand side leads to extremely large
variations in the vector x. The problem is ill-conditioned.
In the following, we define a number called to the condition number to assess
what is the conditioning of a problem.
62
U. Washington AMATH 352 - Spring 2009
5.4.1 Error when computing Ax = b
Suppose that the vector x is perturbed to obtain the vector ˜ x. If b and
˜
b
are, respectively, satisfying Ax = b and A˜ x =
˜
b, we would like to bound the
difference between b and
˜
b.
First, we bound the difference in absolute terms. We have
_
_
_b −
˜
b
_
_
_
2
= |Ax −A˜ x|
2
_
_
_b −
˜
b
_
_
_
2
= |A(x − ˜ x)|
2
So we obtain _
_
_b −
˜
b
_
_
_
2
≤ |A|
2
|x − ˜ x|
2
. (114)
This bound gives an absolute measure of the perturbation in the right hand side
b. A small norm |A|
2
means that a small perturbation in the vector x leads to
a small perturbation in the vector b. On the other hand, a large value for |A|
2
means that the perturbation in b can be large, even when |x − ˜ x|
2
is small.
Note that the bound (114) is sharp because there exists a perturbation x − ˜ x
such that
_
_
_b −
˜
b
_
_
_
2
= |A|
2
|x − ˜ x|
2
.
To estimate the relative perturbation, we write
_
_
_b −
˜
b
_
_
_
2
|b|
2
=
|Ax −A˜ x|
2
|Ax|
2
_
_
_b −
˜
b
_
_
_
2
|b|
2
≤ |A|
2
|x − ˜ x|
2
|Ax|
2
Next, we bound the norm of Ax with the norm of A
−1
as follows
|x|
2
=
_
_
A
−1
Ax
_
_
2

_
_
A
−1
_
_
2
|Ax|
2

1
|Ax|
2

_
_
A
−1
_
_
2
|x|
2
.
So we obtain _
_
_b −
˜
b
_
_
_
2
|b|
2
≤ |A|
2
_
_
A
−1
_
_
2
|x − ˜ x|
2
|x|
2
. (115)
Definition 130. The condition number of A for the 2-norm is
κ
2
(A) = |A|
2
_
_
A
−1
_
_
2
. (116)
The bound (115) gives a relative measure of the perturbation in the right
hand side b. A small condition number means that a small perturbation in the
vector x leads to a small perturbation in the vector b. On the other hand, a
large value for κ
2
(A) means that the perturbation in b can be large, even when
|x − ˜ x|
2
is relatively small. Note that the bound (115) is sharp because there
exists a perturbation x − ˜ x such that the equality is reached.
63
U. Washington AMATH 352 - Spring 2009
Suppose that the matrix A is perturbed to obtain the matrix
˜
A. If b and
˜
b
are, respectively, satisfying Ax = b and
˜
Ax =
˜
b, we would like to bound the
difference between b and
˜
b.
First, we bound the difference in absolute terms. We have
_
_
_b −
˜
b
_
_
_
2
=
_
_
_Ax −
˜
Ax
_
_
_
2
_
_
_b −
˜
b
_
_
_
2
=
_
_
_(A−
˜
A)x
_
_
_
2
So we obtain
_
_
_b −
˜
b
_
_
_
2

_
_
_A−
˜
A
_
_
_
2
|x|
2
.
This bound gives an absolute measure of the perturbation in the right hand side
b.
To estimate the relative perturbation, we write
_
_
_b −
˜
b
_
_
_
2
|b|
2
=
_
_
_Ax −
˜
Ax
_
_
_
2
|Ax|
2
_
_
_b −
˜
b
_
_
_
2
|b|
2
≤ |x|
2
_
_
_A−
˜
A
_
_
_
2
|Ax|
2
Next, we bound the norm of Ax with the norm of A
−1
as follows
1
|Ax|
2

_
_
A
−1
_
_
2
|x|
2
.
So we obtain
_
_
_b −
˜
b
_
_
_
2
|b|
2

_
_
_A−
˜
A
_
_
_
2
_
_
A
−1
_
_
2
|x|
2
|x|
2
and, by multiplying and dividing by |A|
2
,
_
_
_b −
˜
b
_
_
_
2
|b|
2
≤ |A|
2
_
_
A
−1
_
_
2
_
_
_A−
˜
A
_
_
_
2
|A|
2
. (117)
The condition number indicates how much the vector b can be perturbed. A
small condition number means that a small perturbation in the matrix A leads
to a small perturbation in the vector b. On the other hand, a large value for
κ
2
(A) means that the perturbation in b can be large, even when
_
_
_A−
˜
A
_
_
_
2
is
relatively small.
Suppose that the matrix A and the vector x are perturbed to obtain the
matrix
˜
A and the vector ˜ x. If b and
˜
b are, respectively, satisfying Ax = b
and
˜
A˜ x =
˜
b, we bound the difference between b and
˜
b. The relative change
64
U. Washington AMATH 352 - Spring 2009
satisfies
_
_
_b −
˜
b
_
_
_
2
|b|
2
≤ |A|
2
_
_
A
−1
_
_
2
_
_
|x − ˜ x|
2
|x|
2
+
_
_
_A−
˜
A
_
_
_
2
|A|
2
+
_
_
_A−
˜
A
_
_
_
2
|A|
2
|x − ˜ x|
2
|x|
2
_
_
(118)
Again, the condition number of A is the relative change magnification factor.
Exercise 131. Prove the bound (118).
The condition number κ
2
(A) depends on the induced 2-norms of matrices
A and A
−1
. For any other induced matrix norm (like the 1-norm ||
1
or the
∞-norm ||

), a condition number can be similarly defined, for example:
κ
1
(A) = |A|
1
_
_
A
−1
_
_
1
and κ

(A) = |A|

_
_
A
−1
_
_

.
The actual numerical value of κ(A) depends on the norm being used. However,
we are usually only interested in order of magnitude estimates of the condition
number. So the particular norm is usually not very important.
Example 132. Consider the matrix
A =
1
2
_
1 1
1 + 10
−10
1 −10
−10
_
.
Its inverse matrix A
−1
is
A
−1
=
_
1 −10
10
10
10
1 + 10
10
−10
10
_
.
The condition number of A is equal to
κ
1
(A) = |A|
1
_
_
A
−1
_
_
1
=
_
1 +
10
−10
2
_

_
2 10
10
_
= 2 10
10
+ 1.
For this matrix, we have also
κ

(A) = |A|

_
_
A
−1
_
_

= 2 10
10
+ 1
and Matlab gives
κ
2
(A) = |A|
2
_
_
A
−1
_
_
2
= 2 10
10
.
Applying the vector x =
_
−1
1
_
to the matrix A results in the vector b:
b = Ax =
_
0
−10
−10
_
.
65
U. Washington AMATH 352 - Spring 2009
The perturbed vector ˜ x =
_
1
1
_
gives rise to the vector
˜
b:
˜
b = A˜ x =
_
1
1
_
.
We have the following relative changes:
|x − ˜ x|
1
|x|
1
= 1 and
_
_
_b −
˜
b
_
_
_
1
|b|
1
= 2 10
10
+ 1,
illustrating that the condition number is the magnification factor.
Note that some small perturbation of x can still result in small perturbation
of b. For example, the vectors
b = A
_
1
1
_
=
_
1
1
_
and
˜
b = A
_
1.1
0.9
_
=
_
1
1 + 10
−11
_
give the following relative changes
|x − ˜ x|
1
|x|
1
= 0.1 and
_
_
_b −
˜
b
_
_
_
1
|b|
1
=
10
−11
2
.
Remark. The induced norm of A is the largest magnification factor in a direc-
tion:
|A| = max
x=0
|Ax|
|x|
.
We can also define the smallest magnification factor:
m = min
x=0
|Ax|
|x|
.
Note that if matrix A is singular, then m = 0. The condition number is some-
times defined as the ratio
κ(A) =
|A|
m
. (119)
Expressions (116) and (119) are equal since
m = min
x=0
|Ax|
|x|
= max
x=0
|x|
|Ax|
= max
x=0
_
_
A
−1
(Ax)
_
_
|Ax|
=
_
_
A
−1
_
_
.
For any p-norm, we have
κ
p
(A) ≥ 1. (120)
In the 2-norm, orthogonal matrices are perfectly conditioned in that κ(Q) = 1
if Q is orthogonal.
66
U. Washington AMATH 352 - Spring 2009
5.4.2 Error when solving Ax = b
Given an invertible matrix A and the vector b, solving the system Ax = b is
mathematically identical to computing the vector A
−1
b. This is equivalent to
the problem just studied, except that the matrix A is replaced by the matrix
A
−1
. The condition numbers κ(A) and κ(A
−1
) are equal since
_
A
−1
_
−1
= A.
For a rectangular matrix A ∈ R
m×n
, (m ≥ n), of full rank, the condition
number is defined in terms of the pseudoinverse
κ(A) = |A|
_
_
A

_
_
= |A|
_
_
(A
T
A)
−1
A
T
_
_
.
These definitions extend also to complex matrices.
The size of the condition number is a good measure of how close to singular
the matrix is. It is finite as long as A is nonsingular, but approaches ∞ as
A approaches a singular matrix. In particular, the condition number gives a
good indication of how much accuracy we can expect in a computed solution
to Ax = b. When solving a problem Ax = b with a fixed digits of precision,
one must always expect to “lose” log
10
κ(A) digits of accuracy in computing the
solution.
Example 133. Consider the matrix and vectors
A =
_
4.1 2.8
9.7 6.6
_
x =
_
1
0
_
b =
_
4.1
9.7
_
.
Clearly, Ax = b and we have
|x|
1
= 1 and |b| = 13.8.
If the right-hand side is changed to
˜
b =
_
4.11
9.70
_
,
the solution becomes
˜ x =
_
0.34
0.97
_
.
The relative changes
_
_
_b −
˜
b
_
_
_
1
|b|
1
= 0.0007246 and
|x − ˜ x|
1
|x|
1
= 1.63.
κ(A) being the magnification factor, we have
κ
1
(A) ≥
1.63
0.0007246
= 2249.4.
For this example, we have actually an equality
κ
1
(A) = 2249.4.
67
U. Washington AMATH 352 - Spring 2009
Example 134. The Hilbert matrix H
n
= (h
ij
)
n×n
is defined by
h
ij
=
1
i + j −1
.
This matrix is often used for test purposes because of its ill-conditioned nature.
The condition number κ(H
n
) increases rapidly with n and can grow like ce
3.5n
.
5.4.3 Error when evaluating a general function
Consider the function f : R −→ R that is differentiable. The function f is
usually nonlinear.
When evaluating the function f at x, we can again compute the absolute
error on f(x) if x is slightly perturbed:
f(x + h) −f(x) = f

(ξ)h ≈ hf

(x)
where we invoked the mean-value theorem.
On the other hand, the relative error is
f(x + h) −f(x)
f(x)

hf

(x)
f(x)
=
_
xf

(x)
f(x)
_
x + h −x
x
.
So the factor xf

(x)/f(x) is the condition number for this problem.
The condition number can be extended to general functions. If the differen-
tiable function f is from a linear space U equipped with a norm into a linear
space V equipped with a norm, the absolute error satisfies
|f(u) −f(˜ u)| ≈ |J(u)(u − ˜ u)| ≤ |J(u)| |u − ˜ u| .
The relative error is
|f(u) −f(˜ u)|
|f(u)|

|J(u)(u − ˜ u)|
|f(u)|

|J(u)|
|f(u)|
|u − ˜ u| =
_
|J(u)| |u|
|f(u)|
_
|u − ˜ u|
|u|
.
The condition number becomes
|J(u)| |u|
|f(u)|
.
Example 135. Consider the problem of computing

x for x > 0. The Jacobian
of f : x →

x is the derivative f

(x) = 1/(2

x). So we have
κ =
xf

(x)
f(x)
=
1/(2

x)

x/x
=
1
2
.
This is a well-conditioned problem.
68
U. Washington AMATH 352 - Spring 2009
Example 136. Consider the problem
f :
_
x
1
x
2
_
−→x
1
−x
2
.
The Jacobian of f is
J =
_
∂f
∂x1
∂f
∂x2
_
=
_
1 −1
¸
.
Using the ∞-norm, we get
κ =
|J(x)|

|x|

[f(x)[
=
2 max([x
1
[ , [x
2
[)
[x
1
−x
2
[
.
This quantity is large when [x
1
−x
2
[ ≈ 0. So the problem is ill-conditioned
when x
1
≈ x
2
. This example highlights the hazards of cancellation error.
5.5 Useful commands in Matlab
Here are a few commands in Matlab useful for this section.
• norm(A) computes the induced 2-norm of matrix A. norm(A,inf), norm(A,1),
norm(A,p), norm(A,’fro’) computes, respectively, the ∞-norm, the 1-
norm, the p-norm, and the Frobenius norm.
• cond(A) computes the matrix condition number in 2-norm. cond(A,p) re-
turns the matrix condition number in p-norm: norm(A,p)*norm(inv(A),p).
69
U. Washington AMATH 352 - Spring 2009
6 The QR factorization
The QR algorithm is a very important algorithmic idea in numerical linear
algebra.
6.1 Reduced QR factorization
For many applications, we are interested in the column span of a matrix A and
in the knowledge of a basis for this subspace. The idea of QR factorization is
the construction of orthonormal vectors q
1
, q
2
, . . . that span the range of A.
Assume that A ∈ R
m×n
(m ≥ n) has full rank n and denote the column
vectors a
1
, a
2
, . . . , a
n
. We want to build the sequence of orthonormal vectors
q
1
, q
2
, . . . , q
n
such that
span(a
1
, . . . , a
j
) = span(q
1
, . . . , q
j
) j = 1, . . . , n. (121)
When j = 1, the relation (121) takes the form
a
1
= q
1
r
11
such that |q
1
|
2
= 1, (122)
which defines the first vector q
1
and a coefficient r
11
|a
1
|
2
= |q
1
r
11
| = [r
11
[ |q
1
| = [r
11
[ .
Arbitrarily, we may choose that r
11
> 0. Note that r
11
will not be zero because
the matrix A has full rank.
When j = 2 and the first vector q
1
is defined, the relation (121) becomes
a
2
= q
1
r
12
+q
2
r
22
such that q
T
1
q
2
= 0 and |q
2
|
2
= 1, (123)
which implies
q
T
1
a
2
= q
T
1
q
1
r
12
= r
12
and
|a
2
−q
1
r
12
|
2
= |q
2
r
22
| = [r
22
[ .
r
22
will not be zero else the column vector a
2
would be linearly dependent with
a
1
. It is impossibe because the matrix A has full rank. We can choose again
that r
22
> 0.
When j = 3 and the vectors q
1
and q
2
are defined, the relation (121) becomes
a
3
= q
1
r
13
+q
2
r
23
+q
3
r
33
such that q
T
1
q
3
= 0, q
T
2
q
3
= 0, and |q
3
|
2
= 1,
(124)
which implies
q
T
1
a
3
= q
T
1
q
1
r
13
= r
13
and q
T
2
a
3
= q
T
2
q
2
r
23
= r
23
and
|a
3
−q
1
r
13
−q
2
r
23
|
2
= |q
3
r
33
| = [r
33
[ .
70
U. Washington AMATH 352 - Spring 2009
r
33
will not be zero else the column vector a
3
would be linearly dependent with
(a
1
, a
2
). It is impossibe because the matrix A has full rank. We can choose
again that r
33
> 0.
Generally, when 1 ≤ j ≤ n and the j − 1 orthonormal vectors q
1
, . . . , q
j−1
are defined, we write
a
j
= q
1
r
1j
+ . . . +q
j
r
jj
. (125)
It is evident that an appropriate definition of the coefficients of r
ij
is
r
ij
= q
T
i
a
j
(126)
and
r
jj
=
_
_
_
_
_
a
j

j−1

i=1
q
i
(q
T
i
a
j
)
_
_
_
_
_
2
. (127)
Relation (121) can be rewritten in a matrix form
_
_
a
1
¸
¸
¸
¸
¸
¸

¸
¸
¸
¸
¸
¸
a
n
_
_
=
_
_
q
1
¸
¸
¸
¸
¸
¸

¸
¸
¸
¸
¸
¸
q
n
_
_
_
¸
¸
¸
_
r
11
r
12
r
1n
0 r
22
r
2n
.
.
.
.
.
.
.
.
.
.
.
.
0 0 r
nn
_
¸
¸
¸
_
(128)
or
A =
´
Q
´
R, (129)
where
´
Q is m n with orthonormal columns and
´
R is n n upper-triangular
matrix with positive diagonal entries. Such a factorization is called a reduced
QR factorization of A.
Theorem 137. Each matrix A ∈ R
m×n
(m ≥ n) of full rank has a unique
reduced QR factorization A =
´
Q
´
R with r
jj
> 0.
The proof is provided by the constructive method described earlier. This
method is an old idea, known as the Gram-Schmidt orthogonalization. In Figure
7, we represent graphically the reduced QR factorization.
Figure 7: Reduced QR factorization (m ≥ n)
71
U. Washington AMATH 352 - Spring 2009
Remark. Each complex matrix A ∈ C
m×n
(m ≥ n) of full rank has also a unique
reduced QR factorization A =
´
Q
´
R with r
jj
> 0.
Remark. The matrix
ˆ
Q has orthonormal columns. So we have
ˆ
Q
T
ˆ
Q = I, where
the identity matrix has dimension n n. Then we get
∀x ∈ R
n×1
,
_
_
_
ˆ
Qx
_
_
_
2
=
_
_
ˆ
Qx
_
T
ˆ
Qx =
_
x
T ˆ
Q
T ˆ
Qx =

x
T
x = |x|
2
which indicates that
_
_
_
ˆ
Q
_
_
_
2
= 1. So we obtain
|A|
2
=
_
_
_
ˆ
R
_
_
_
2
.
Example 138. Consider the matrix
A =
_
3 2
1 2
_
.
We will compute its QR factorization. The matrix A has a determinant equal
to 4 and it is of full rank. The first step constructs the vector q
1
such that
a
1
=
_
3
1
_
|a
1
|
2
=

10 =⇒q
1
=
_
3
/

10
1
/

10
_
.
Next, we write
_
2
2
_
=
_
3
1
_
r
12

10
+q
2
r
22
such that q
T
1
q
2
= 0 and |q
2
|
2
= 1,
which gives
_
3/

10
1/

10
_
T
_
2
2
_
= r
12
=⇒ r
12
=
8

10
.
So we obtain
q
2
r
22
=
_
2
2
_

8

10
_
3/

10
1/

10
_
=
_
2 −
24
10
2 −
8
10
_
=
_

4
10
12
10
_
=
_

4

160
12

160
_

160
10
=
_

1

10
3

10
_
2

10
5
.
Finally, we write
_
3 2
1 2
_
=
_
3

10
−1

10
1

10
3

10
__

10
8

10
0
2

10
5
_
.
72
U. Washington AMATH 352 - Spring 2009
Example 139. Consider the matrix
A =
_
_
1 0 0
1 3 0
1 3 6
_
_
.
We will compute its QR factorization. The matrix A is of full rank. The first
step constructs the vector q
1
such that
a
1
=
_
_
1
1
1
_
_
|a
1
|
2
=

3 =⇒q
1
=
1

3
_
_
1
1
1
_
_
.
Next, we write
_
_
0
3
3
_
_
=
_
_
1
1
1
_
_
r
12

3
+q
2
r
22
such that q
T
1
q
2
= 0 and |q
2
|
2
= 1,
which gives
_
_
1

3
_
_
1
1
1
_
_
_
_
T
_
_
0
3
3
_
_
= r
12
=⇒ r
12
=
6

3
= 2

3.
Then we have
a
2
−q
1
r
12
=
_
_
0
3
3
_
_

2

3

3
_
_
1
1
1
_
_
=
_
_
0
3
3
_
_

_
_
2
2
2
_
_
=
_
_
−2
1
1
_
_
and
r
22
=

6 and q
2
=
1

6
_
_
−2
1
1
_
_
.
For the third column, we write
_
_
0
0
6
_
_
=
_
_
1
1
1
_
_
r
13

3
+
_
_
−2
1
1
_
_
r
23

6
+q
3
r
33
such that q
T
1
q
3
= 0, q
T
2
q
3
= 0, and |q
3
|
2
= 1.
We obtain
_
_
1

3
_
_
1
1
1
_
_
_
_
T
_
_
0
0
6
_
_
= r
13
=⇒ r
13
= 2

3
and
_
_
1

6
_
_
−2
1
1
_
_
_
_
T
_
_
0
0
6
_
_
= r
23
=⇒ r
23
=

6.
73
U. Washington AMATH 352 - Spring 2009
The third column of
ˆ
Q will satisfy
q
3
r
33
=
_
_
0
0
6
_
_

_
_
2
2
2
_
_

_
_
−2
1
1
_
_
=
_
_
0
−3
3
_
_
=
_
_
0
−1
1
_
_

1

2
3

2.
Finally, we write
_
_
1 0 0
1 3 0
1 3 6
_
_
=
_
¸
_
1

3
−2

6
0
1

3
1

6
−1

2
1

3
1

6
1

2
_
¸
_
_
_

3 2

3 2

3
0

6

6
0 0 3

2
_
_
.
6.2 Gram-Schmidt orthogonalization
The steps, described in (125), (126), and (130), form the j-th iteration of the
Gram-Schmidt orthogonalization. Mathematically, it offers a simple route to
understanding and proving various properties of QR factorizations. The com-
plete algorithm is described in Algorithm 1.
Algorithm 1 Classical Gram-Schmidt
for j = 1 to n do
v = a
j
for i = 1 to j −1 do
r
ij
= q
T
i
a
j
v = v −r
ij
q
i
end for
r
jj
= |v|
2
q
j
= v/r
jj
end for
6.3 Projections
Equation (130) can be written in a matrix form
r
jj
=
_
_
(I −Q
j−1
Q
T
j−1
)a
j
_
_
2
. (130)
where the matrix Q
j−1
denote the m(j −1) matrix containing the first j −1
column vectors q
1
, . . . , q
j−1
. The matrix
P
j
= I −Q
j−1
Q
T
j−1
(131)
represents an orthogonal projection. The projection P
j
computes the compo-
nent of a
j
that is in the span of Q
j−1
and removes it. Another interpretation
is that the projector P
j
computes the component of a
j
that is not in the span
of Q
j−1
.
74
U. Washington AMATH 352 - Spring 2009
Definition 140. A projection, or projector, is a square matrix P such that
P
2
= P. It is an orthogonal projector (in contrast to an oblique projector)
when P
T
= P.
Trefethen and Bau
2
note that
“The term projector might be thought of as arising from the notion
that if one were to shine a light onto the range of P from the right
direction, then Pv would be the shadow projected by the vector v.”
Note that we have
P
2
v = Pv =⇒ P
2
v −Pv = 0 =⇒ P(Pv −v) = 0. (132)
This means that Pv −v belongs to the null space of P.
Proposition 141. Let Q be an m n matrix with orthonormal columns, i.e.
Q
T
Q = I. The matrices QQ
T
and I −QQ
T
are orthogonal projectors.
Exercise 142. Prove Proposition 141.
Proposition 143. If P is a projector, then I − P is also a projector. The
matrix I −P is called the complementary projector to P.
I −P projects exactly into the nullspace of P. If v belongs to the null space
of P, then (I −P)v = v belongs to the range of I −P, which implies
^(P) ⊂ 1(I −P).
If v belongs to range(I −P), then v = (I −P)v and
Pv = P[(I −P)v] = Pv −P
2
v = 0,
which implies that
^(P) = 1(I −P). (133)
By writing P = I −(I −P), we derive the complementary fact
^(I −P) = 1(P). (134)
We can also see that 1(P) ∩1(I −P) = ¦0¦: any vector v in both sets satisfies
Pv = v = v −Pv = 0.
Using the result (77) about dimensions, these computations show that an mm
projector P separates R
m
into two spaces,
R
m
= 1(P) ⊕1(I −P). (135)
2
L. N. Trefethen and D. Bau, Numerical linear algebra, SIAM, Philadelphia, 1997.
75
U. Washington AMATH 352 - Spring 2009
6.4 Full QR factorization
A full QR factorization of A ∈ R
m×n
(m ≥ n) goes further, appending an
additional m − n orthonormal columns to
ˆ
Q so that it becomes an m m
orthogonal matrix Q. In the process, row of zeros are appended to
ˆ
R so that it
becomes an m n matrix R, still upper-triangular. In Figure 8, we represent
graphically the reduced QR factorization.
Figure 8: Full QR factorization (m ≥ n)
When the matrix A does not have full rank, the Gram-Schmidt algorithm
can break when r
jj
is zero or v = 0. In that case, we simply choose the vector q
j
to be any arbitrary normalized vector orthogonal to the previously constructed
vectors q
1
, . . . , q
j−1
and continue the Gram-Schmidt algorithm.
Theorem 144. Every matrix A ∈ R
m×n
(m ≥ n) has a QR factorization
A = QR.
Example 145. Consider the matrix
A =
_
_
1 0 0
1 0 0
1 0 6
_
_
.
We will compute its QR factorization. The matrix A is of rank 2. The first step
constructs the vector q
1
such that
a
1
=
_
_
1
1
1
_
_
|a
1
|
2
=

3 =⇒q
1
=
1

3
_
_
1
1
1
_
_
.
Next, we write
_
_
0
0
0
_
_
=
_
_
1
1
1
_
_
r
12

3
+q
2
r
22
such that q
T
1
q
2
= 0 and |q
2
|
2
= 1,
which gives
_
_
1

3
_
_
1
1
1
_
_
_
_
T
_
_
0
0
0
_
_
= r
12
=⇒ r
12
= 0.
76
U. Washington AMATH 352 - Spring 2009
Then we have
a
2
−q
1
r
12
=
_
_
0
0
0
_
_
which implies that r
22
= 0. The vector q
2
can not be zero because it is of
norm 1. Next we simply choose the vector q
2
to be any arbitrary normalized
vector orthogonal to the previously constructed vector q
1
and continue the
Gram-Schmidt algorithm. For instance, we can choose
q
2
=
1

6
_
_
1
−2
1
_
_
.
For the third column, we write
_
_
0
0
6
_
_
=
_
_
1
1
1
_
_
r
13

3
+
_
_
1
−2
1
_
_
r
23

6
+q
3
r
33
such that q
T
1
q
3
= 0, q
T
2
q
3
= 0, and |q
3
|
2
= 1.
We obtain
_
_
1

3
_
_
1
1
1
_
_
_
_
T
_
_
0
0
6
_
_
= r
13
=⇒ r
13
= 2

3
and
_
_
1

6
_
_
1
−2
1
_
_
_
_
T
_
_
0
0
6
_
_
= r
23
=⇒ r
23
=

6.
The third column of Q will satisfy
q
3
r
33
=
_
_
0
0
6
_
_

_
_
2
2
2
_
_

_
_
1
−2
1
_
_
=
_
_
−3
0
3
_
_
=
_
_
−1
0
1
_
_

1

2
3

2.
Finally, we write
_
_
1 0 0
1 0 0
1 0 6
_
_
=
_
¸
_
1

3
1

6
−1

2
1

3
−2

6
0
1

3
1

6
1

2
_
¸
_
_
_

3 0 2

3
0 0

6
0 0 3

2
_
_
.
Note that a different choice for q
2
would result in a different factorization. For
example, setting
q
2
=
1

2
_
_
0
−1
1
_
_
,
77
U. Washington AMATH 352 - Spring 2009
we obtain
_
_
1 0 0
1 0 0
1 0 6
_
_
=
_
¸
_
1

3
0
−2

6
1

3
−1

2
1

6
1

3
1

2
1

6
_
¸
_
_
_

3 0 2

3
0 0 3

2
0 0

6
_
_
.
Remark. Each complex matrix A ∈ C
m×n
(m ≥ n) has also a full QR factor-
ization A = QR.
6.5 Solution of Ax = b by QR factorization
Suppose we wish to solve Ax = b where A is a real square nonsingular matrix.
Using the QR factorization, we can write QRx = b or
Rx = Q
T
b (136)
The right hand side is easy to compute when Q is known. The system of linear
equations in x is also easy to solve because R is triangular. This suggests the
following method for computing the solution to Ax = b:
1. Compute a QR factorization A = QR
2. Compute y = Q
T
b
3. Solve Rx = y for x
This combination is an excellent method for solving linear systems of equations.
However, it is not the standard methods for such problems. Gaussian elimina-
tion is the algorithm often used in practice, since it requires only half as many
numerical operations. The Gaussian elimination will be described later in the
notes.
6.6 Generalization of Gram-Schmidt orthogonalization
The Gram-Schmidt orthogonalization has an analogue for an abstract real linear
space. Let U be a real linear space equipped with an inner product < , > and
the associated norm || =

< , >. The complete algorithm is described in
Algorithm 2.
Exercise 146. Consider T
3
the space of polynomials of degree at most 3,
equipped with the inner product
< f, g >=
ˆ
1
−1
f(t)g(t)dt.
Starting from the basis (1, x, x
2
, x
3
), use the Gram-Schmidt construction to
build an orthonormal basis. The resulting polynomials are scalar multiples of
what are known as the Legendre polynomials, P
j
, which are conventionally
normalized so that P
j
(1) = 1. Computations with such polynomials form the
basis of spectral methods, one of the most powerful techniques for the numerical
solution of partial differential equations.
78
U. Washington AMATH 352 - Spring 2009
Algorithm 2 Classical Gram-Schmidt with a general inner-product
Require: (u
(1)
, . . . , u
(n)
), n members of U
for j = 1 to n do
v = u
(j)
for i = 1 to j −1 do
r
ij
=< q
(i)
, u
(j)
>
v = v −r
ij
q
(i)
end for
r
jj
= |v| =

< v, v >
q
(j)
= v/r
jj
end for
6.7 Useful commands in Matlab
Here are a few commands in Matlab useful for this section.
• [Q,R]=qr(A,0) computes the reduced QR factorization of A.
• [Q,R]=qr(A) computes the full QR factorization of A.
79
U. Washington AMATH 352 - Spring 2009
7 Computer arithmetic
Computer memory is partitioned into bytes consisting of 8 bits. Numbers are
usually stored in the computer using either 4 bytes (32 bits, sometimes called
single precision) or 8 bytes (64 bits, or double precision). Matlab always uses
64-bit floating point numbers (or double precision). Consequently, computers
can only represent a finite subset of real numbers. It is good to be aware of this
limitation and of potential pitfalls of computer arithmetic. Careless construction
of algorithms can have sometimes dramatic consequences as we will see in some
examples.
7.1 Digital representation limitations
Represented numbers in computer arithmetic can not be arbitrarily large or
small. For example, the IEEE double precision arithmetic permits numbers
as large as 1.79 10
308
and as small as 2.23 10
−308
. This range is usually
sufficiently large. Overflow and underflow are rarely a serious hazard.
Represented numbers have gaps between them. For example, in IEEE double
precision arithmetic, the interval [1, 2] is represented by the discrete subset
1, 1 + 2
−52
, 1 + 2 2
−52
, 1 + 3 2
−52
, . . . , 2. (137)
The interval [2, 4] is represented by the discrete subset
2, 2 + 2
−51
, 2 + 2 2
−51
, 2 + 3 2
−51
, . . . , 4,
which are the same numbers multiplied by 2. In general, the interval [2
j
, 2
j+1
] is
represented by the numbers in (137) multiplied by 2
j
. In IEEE double precision
arithmetic, the relative gap between two numbers is never larger than 2
52
≈ 2.2
10
−16
. This gap is usually negligible but ill-conditioned problems or unstable
algorithms can be affected by this finite gap.
Example 147. Consider the matrix
A
ε
=
1
2
_
1 1
1 + ε 1 −ε
_
and its inverse
A
−1
ε
=
_
1 −
1
ε
1
ε
1 +
1
ε
−1
ε
_
where 0 < ε < 1. The 1-norms for these matrices are
|A
ε
|
1
= 1 +
ε
2
_
_
A
−1
ε
_
_
1
=
2
ε
and the condition number is
κ
1
(A
ε
) = |A
ε
|
1
_
_
A
−1
ε
_
_
1
=
_
1 +
ε
2
_

2
ε
=
2
ε
+ 1.
80
U. Washington AMATH 352 - Spring 2009
Recall that the condition number gives a good indication of how much accuracy
we can expect in a computed solution to A
ε
x = b. When solving a problem
A
ε
x = b with a fixed digits of precision, one must always expect to “lose”
log
10
κ
1
(A
ε
) digits of accuracy in computing the solution. For any value of ε,
we have
A
ε
_
1
1
_
=
_
1
1
_
.
We solve the same system numerically with Matlab and compute the 1-norm
of the error
_
_
_
_
A
ε
¸
_
1
1
_

_
1
1
__
_
_
_
1
_
_
_
_
inv(A
ε
) ∗
_
1
1
_

_
1
1
__
_
_
_
1
ε = 10
−5
≈ 1.11 10
−11
≈ 1.455 10
−11
ε = 10
−8
≈ 1.11 10
−08
≈ 1.490 10
−08
ε = 10
−12
≈ 1.11 10
−04
0
Table 1: Rounding errors when solving ill-conditioned system
Note the numerical value may actually vary with your machine.
Example 148. A notorious example is the fate of the Ariane rocket launched
on June 4, 1996 (European Space Agency 1996). In the 37th second of flight, the
inertial reference system attempted to convert a 64-bit floating-point number to
a 16-bit number, but instead triggered an overflow error which was interpreted
by the guidance system as flight data, causing the rocket to veer off course and
be destroyed.
3
Example 149. The Patriot missile defense system used during the Gulf War
was also rendered ineffective due to roundoff error (Skeel 1992, U.S. GAO 1992).
The system used an integer timing register which was incremented at intervals of
0.1s. However, the integers were converted to decimal numbers by multiplying
by the binary approximation of 0.1,
0.00011001100110011001100
2
=
209715
2097152
.
As a result, after 100 hours, an error of
_
1
10

209715
2097152
_
(3600 100 10) =
5625
16384
≈ 0.3433s
had accumulated. This discrepancy caused the Patriot system to continuously
recycle itself instead of targeting properly. As a result, an Iraqi Scud missile
could not be targeted and was allowed to detonate on a barracks, killing 28
people.
3
See http://mathworld.wolfram.com/RoundoffError.html
81
U. Washington AMATH 352 - Spring 2009
7.2 Floating point numbers
Almost all processors use IEEE standard arithmetic, which is an arithmetic
system based on a floating point representation of the real numbers combined
with the binary system.
7.2.1 Binary system
Most computers deal with real numbers in binary systems, in contrast to the
decimal system that humans prefer to use. The binary system uses 2 as the
base in the same way that the decimal system uses 10. Consider the number
427.325. Then we have
427.375 = 4 10
2
+ 2 10
1
+ 7 10
0
+ 3 10
−1
+ 7 10
−2
+ 5 10
−3
.
The expression on the right contains powers of 10 and digits (here: 4, 2, 7, 3,
7, 5). If we accept the possibility of having an infinite number of digits to the
right of the decimal point, then any real number can be expressed with a sign
(+ or −) affixed to it. For example, −π is
−π = −3.14159265358979323846264338 . . .
In the binary system, only the two digits 0 and 1 are used. A number in the
binary system can be expressed in a similar manner. For example, we have
427.375 = 1 2
8
+ 1 2
7
+ 1 2
5
+ 1 2
3
+ 1 2
1
+ 1 2
0
+ 1 2
−2
+ 1 2
−3
. (138)
The base 2 number
(1001.11101)
2
= 1 2
3
+ 0 2
2
+ 0 2
1
+ 1 2
0
+ 1 2
−1
+ 1 2
−2
+ 1 2
−3
+ 0 2
−4
+ 1 2
−5
(139)
corresponds to the number 9.90625.
To compute the binary representation, we perform a series of division by 2
and reverse the order of the remainders to get the representation. For example:
• (15)
10
= (1111)
2
= 1 2
3
+ 1 2
2
+ 1 2
1
+ 1 2
0
– 15 = 2 ∗ 7 + 1 → remainder 1
– 7 = 2 ∗ 3 + 1 → remainder 1
– 3 = 2 ∗ 1 + 1 → remainder 1
– 1 = 2 ∗ 0 + 1 → remainder 1
• (11)
10
= (1011)
2
= 1 2
3
+ 0 2
2
+ 1 2
1
+ 1 2
0
– 11 = 2 ∗ 5 + 1 → remainder 1
82
U. Washington AMATH 352 - Spring 2009
– 5 = 2 ∗ 2 + 1 → remainder 1
– 2 = 2 ∗ 1 + 0 → remainder 0
– 1 = 2 ∗ 0 + 1 → remainder 1
In general, any integer β > 1 can be used as the base for a number system.
Numbers represented in base β will contain digits 0, 1, 2, . . . , β−1. The notation
(x)
β
= (−1)
s
(x
n
x
n−1
. . . x
1
x
0
.x
−1
x
−2
. . . x
−m
)
β
with x
n
,= 0 (140)
means
(x)
β
= (−1)
s
_
n

k=−m
x
k
β
k
_
= (−1)
s
(x
n
β
n
+ x
n−1
β
n−1
+ . . . + x
1
β + x
0
1
+ x
−1

1
β
+ x
−2

1
β
2
+ . . . + x
−m

1
β
m
). (141)
Computers communicate with users in the decimal system but works inter-
nally in the binary system. Conversion procedures must be executed by the
computer. We should be aware that there are two conversions involved — from
and to decimal. Errors can be involved in each conversion.
Recall that computers represent only a subset of the real numbers. For
example, a simple number like 1/10 can not be stored exactly on any binary
machine. It requires an infinite binary expression
1
10
= (0.0001 1001 1001 1001 . . .)
2
.
So the conversion of 1/10 to binary will result in a roundoff error.
7.2.2 Floating point number
Almost all processors use IEEE standard arithmetic, which is an arithmetic
system based on a floating point representation of the real numbers. In a floating
point number system, the position of the decimal (or binary) point is stored
separately from the digits. The gaps between adjacent represented numbers
scale in proportion to the size of the numbers. This is different from a fixed
point representation where the gaps are all the same.
In normalized scientific notation, any nonzero real number x can be repre-
sented as
x = ±m10
e
where the mantissa m is a number in the range
1
10
≤ m < 1 and the exponent
e is an integer (positive, negative, or zero). When x = 0, then e = 0.
In the binary system, we have
x = ±m2
e
83
U. Washington AMATH 352 - Spring 2009
where
1
2
≤ m < 1 (if x ,= 0). Both m and e are base 2 numbers.
IEEE arithmetic includes two kinds of floating point numbers: single preci-
sion (32 bits long) and double precision (64 bits long). If s, e, and m < 1 are
the 1-bit sign, 8-bit exponent, and 23-bit mantissa (or fraction) in the IEEE
single precision format, then the number represented is
(−1)
s
2
e−127
(1 + m).
On 8 bits, the exponent satisfy
0 < e < (11 111 111)
2
= 2
8
−1 = 255.
The values e = 0 and e = 255 are reserved for special cases such as 0, ±∞, and
NaN. With a 23-bit mantissa, the least significant bit in the mantissa represents
units of 2
−23
(or approximately 1.2 10
−7
). The range of positive normalized
numbers is from 2
−126
(the underflow threshold) to 2
127
(2−2
−23
) ≈ 2
128
(the
overflow threshold) or about 10
−38
to 10
38
.
If s, e, and m < 1 are the 1-bit sign, 11-bit exponent, and 52-bit mantissa
(or fraction) in the IEEE double precision format, then the number represented
is
(−1)
s
2
e−1023
(1 + m).
On 11 bits, the exponent satisfy
0 < e < (11 111 111)
2
= 2
11
−1 = 2047.
The maximum relative representation error is 2
−53
≈ 10
−16
. The range of pos-
itive normalized numbers is from 2
−1022
(the underflow threshold) to 2
1023

(2 −2
−52
) ≈ 2
1024
(the overflow threshold) or about 10
−308
to 10
308
.
7.2.3 Machine epsilon
The machine epsilon is traditionally half the distance between 1 and the next
larger floating point number, 1+2
−52
. It is also defined as the maximum relative
representation error. In a relative sense, the machine epsilon
machine
has the
property
∀x ∈ R, there exists a floating number x

, [x −x

[ ≤
machine
[x

[ .
7.2.4 Special symbols Inf and NaN
The two error values are “infinity” (often denoted “Inf”), and “NaN” (“not a
number”), which covers all other errors. Inf does not necessarily mean that the
result is actually infinite. It simply means “too large to represent”.
Both of these are encoded with the exponent field set to all 1s. (Recall
that exponent fields of all 0s or all 1s are reserved for special meanings.) The
mantissa field is set to something that can distinguish them—typically zero for
Inf and nonzero for NaN. The sign bit is meaningful for Inf, that is, floating-point
hardware distinguishes between + and −.
84
U. Washington AMATH 352 - Spring 2009
When a nonzero number is divided by zero (the divisor must be exactly
zero), a “divide by zero” event occurs, and the result is set to infinity of the
appropriate sign. In other cases in which the result’s exponent is too large to
represent, such as division of an extremely large number by an extremely small
number, an “overflow” event occurs, also producing infinity of the appropriate
sign. This is different from a divide by zero, though both produce a result of
infinity, and the distinction is usually unimportant in practice.
Floating-point hardware is generally designed to handle operands of infinity
in a reasonable way, such as
• (+Inf) + (+7) = (+Inf)
• (+Inf) × (−2) = (−Inf)
• (+Inf) × 0 = NaN — there is no meaningful thing to do
7.3 Floating point operations
7.3.1 Arithmetic
On a computer, all mathematical computations are reduced to certain elemen-
tary arithmetic operations. The classical set is +, −, , ÷. Mathematically,
these symbols represent operations in R. The corresponding computer opera-
tions involve roundoff errors.
Denote fl(x) the floating point number closest to x. Then we have
fl(x) = x(1 + ε) with [ε[ ≤
machine
.
Note that we have fl(fl(x)) = fl(x). In computer arithmetic, the four basic
operations applied to floating point numbers are exact up to a relative error
of size at most
machine
. For example, for the substraction, we have
fl(fl(x) ¸fl(y)) = (fl(x) −fl(y))(1 + ) with [[ ≤
machine
(142)
and
fl(fl(x) ¸fl(y)) = (x(1 + ε
1
) −y(1 + ε
2
))(1 + ) with [[ ≤
machine
. (143)
7.3.2 Operation counts
The work required to solve a problem on a computer is often measured in terms
of the number of floating point operations or flops needed to do the calculation.
A floating point operation is multiplication, division, addition, or subtraction
on floating point numbers.
Example 150. Computing the inner product of two vectors in R
n
,
x
T
y = x
1
y
1
+ x
2
y
2
+ . . . + x
n
y
n
,
requires n multiplications and n −1 additions for a total of 2n −1 flops.
85
U. Washington AMATH 352 - Spring 2009
Example 151. Computing y = Ax where A ∈ R
m×n
and x ∈ R
n
. The i-th
element of y is the inner product of the i-th row of A with x and requires 2n−1
flops. There are m rows. So we need to compute y
i
for i = 1, 2, . . . , m. The
total work is 2mn −m flops. When the matrix A is square (m = n), the count
becomes 2n
2
−n. For large value of n, the first term dominates and we say the
work required is O(n
2
) as n goes to +∞.
If the work required by some algorithm for a system of size n has the flop
count
W(n) = a
0
+ a
1
n + a
2
n
2
+ . . . + a
k
n
k
(where a
j
is a constant), then the last term dominates for large values of n. We
say
W(n) = O(n
k
) as n −→+∞.
More generally, if W(n) is “Big oh” of n
k
, then the ratio W(n)/n
k
remains
bounded as n −→+∞:
W(n) = O(n
k
) ⇔ 0 ≤ W(n) ≤ Cn
k
for large values of n
(where C is a constant independent on n).
Remark. When W(n) = O(n
k
), then doubling the size of the problem (from n
to 2n) will increase the work by a factor 2
k
.
Example 152. Consider A ∈ R
m×r
and B ∈ R
r×n
. Computing the matrix-
matrix product C = AB will require the computations of mn entries c
ij
. Each
entry c
ij
is the inner product of two vectors with r components, which require
2r − 1 flops. So the total work becomes 2mnr − mn flops. When m = n = r,
the matrix-matrix product requires O(n
3
) flops.
You may know how “fast” your computer processor is, e.g. 2 GHz (2 giga-
hertz, or 2 billion cycles per second). This tells something about how many
instructions the processor can execute per second. When using numerical meth-
ods we generally care most about how many floating point operations per second
or flops it can perform, where a floating point operation means adding two num-
bers together, or other basic arithmetic operation. This can vary quite a bit
depending on exactly what is being done, but generally has the same order of
magnitude as the cycle speed (a 2 GHz processor can do roughly 1 billion arith-
metic operations per second). We use the terms megaflops, gigaflops, teraflops,
etc. to talk about the speed of computers.
Currently the “fastest” computer in the world is the Roadrunner computer at
Los Alamos National Laboratory, which can perform certain benchmark linear
algebra problems at more than one petaflops (10
15
floating point operations per
second). See http://www.top500.org for a list of the top 500 machines in the
world and information about each. It should be noted, however, that these fast
machines are based on processors that are not all that fast, but are composed
of thousands of processors working together. To get this fast performance it
is necessary to develop algorithms that can use all these processors together
86
U. Washington AMATH 352 - Spring 2009
to solve a single problem. This is not easy, and one of the main challenges in
numerical methods for the future is to develop better ways of using parallel
processors.
You should know the common prefixes (the bigger ones will become more
common in the future!):
• kilo: 10
3
• mega: 10
6
• giga: 10
9
• tera: 10
12
• peta: 10
15
• exa: 10
18
7.4 Stability
Since most problems are continuous while digital computers are discrete, nu-
merical algorithms can generally not provide the exact solution. The notion of
stability is the standard way of characterizing what is possible. It corresponds
to the numerical analysts’ idea of what it means to get the “right answer”, even
if it is not exact.
Consider a mathematical problem as a function f between two linear spaces
(f : X →Y ) and an algorithm as another map
˜
f between the same two spaces.
Definition 153. An algorithm
˜
f for a problem f is accurate if we have
∀x ∈ X, 0 ≤
_
_
_
˜
f(x) −f(x)
_
_
_
|f(x)|
≤ C
machine
= O(
machine
) (144)
where C is a constant independent of x.
Accuracy is a strong statement that is difficult to attain when the problem
f is ill-conditioned. The next two definitions are less restrictive.
Definition 154. An algorithm
˜
f for a problem f is stable if we have
∀x ∈ X, ∃ ˜ x ∈ X such that
|˜ x −x|
|x|
= O(
machine
)
and
_
_
_
˜
f(x) −f(˜ x)
_
_
_
|f(˜ x)|
= O(
machine
) (145)
A stable algorithm gives nearly the right answer to nearly the right question.
Many algorithms in numerical linear algebra satisfy a condition that is both
stronger and simpler than stability.
87
U. Washington AMATH 352 - Spring 2009
Definition 155. An algorithm
˜
f for a problem f is backward stable if we have
∀x ∈ X, ∃ ˜ x ∈ X such that
|˜ x −x|
|x|
= O(
machine
) and
˜
f(x) = f(˜ x). (146)
A backward stable algorithm gives exactly the right answer to nearly the right
question.
In this course, we will not analyze algorithms from the point of view of
stability. Instead, we will discuss some examples. This is an important research
field. The book of Trefethen and Bau
4
gives a nice introduction.
7.5 Example
Consider the classical Gram-Schmidt algorithm used in the QR factorization
(see Algorithm 3)
Algorithm 3 Classical Gram-Schmidt
for j = 1 to n do
v = a
j
for i = 1 to j −1 do
r
ij
= q
T
i
a
j
v = v −r
ij
q
i
end for
r
jj
= |v|
2
q
j
= v/r
jj
end for
First, we compute the number of operations. The computation of r
ij
is an
inner product between two vectors in R
m
, which costs 2m−1 flops. The update
of v requires m multiplications and m substractions. The total work is a single
inner iteration is 4m−1 flops. All together, the number of flops required by the
algorithm is
n

j=1
__
j−1

i=1
4m−1
_
+ 2m−1 + 1
_
=
n

j=1
[(4m−1)(j −1) + 2m]
= (4m−1)
_
_
n

j=1
j
_
_
−(2m−1)n (147)
We recall the useful formula
n

k=1
k =
k(k + 1)
2
.
4
L. N. Trefethen and D. Bau, Numerical linear algebra, SIAM, Philadelphia, 1997.
88
U. Washington AMATH 352 - Spring 2009
So the total number of flops for the classical Gram-Schmidt algorithm is
(4m−1)
n(n + 1)
2
−(2m−1)n ≈ 2mn
2
= O(mn
2
).
The classical Gram-Schmidt algorithm is numerically unstable because of
rounding errors on a computer. To illustrate the unstability, consider the Hilbert
matrix
H
4
=
_
¸
¸
_
1
1
/2
1
/3
1
/4
1
/2
1
/3
1
/4
1
/5
1
/3
1
/4
1
/5
1
/6
1
/4
1
/5
1
/6
1
/7
_
¸
¸
_
.
A Matlab implementation of Algorithm 3 generates a matrix Q such that
_
_
Q
T
Q−I
_
_
1
≈ 3.6 10
−11
(148)
Fortunately, there are simple modifications that improve matters. For exam-
ple, we describe, in Algorithm 4, the modified Gram-Schmidt which requires
the same amount of flops than the classical Gram-Schmidt algorithm. For the
Algorithm 4 Modified Gram-Schmidt
for i = 1 to n do
v
i
= a
i
end for
for i = 1 to n do
r
ii
= |v
i
|
2
q
i
= v
i
/r
ii
for j = i + 1 to n do
r
ij
= q
T
i
v
j
v
j
= v
j
−r
ij
q
i
end for
end for
Hilbert matrix H
4
, the modified Gram-Schmidt algorithm will compute a matrix
Q such that
_
_
Q
T
Q−I
_
_
1
≈ 4.0 10
−13
(149)
which is better than (148).
Another possibility is to apply the classical Gram-Schmidt algorithm twice,
as described in Algorithm 5. The total number of flops is asymptotically equal
to 4mn
2
. For the Hilbert matrix H
4
, applying twice the classical Gram-Schmidt
algorithm will compute a matrix Q such that
_
_
Q
T
Q−I
_
_
1
≈ 3.6 10
−16
. (150)
Remark. The Matlab function qr uses the Householder triangularization, which
is an advanced algorithm (see Trefethen and Bau
5
). It requires, asymptotically,
2mn
2

2
3
n
3
flops.
5
L. N. Trefethen and D. Bau, Numerical linear algebra, SIAM, Philadelphia, 1997.
89
U. Washington AMATH 352 - Spring 2009
Algorithm 5 Two steps of classical Gram-Schmidt
for j = 1 to n do
v = a
j
for i = 1 to j −1 do
r
ij
= q
T
i
a
j
v = v −r
ij
q
i
end for
r
jj
= |v|
2
q
j
= v/r
jj
end for
for j = 1 to n do
v = q
j
for i = 1 to j −1 do
r
ij
= q
T
i
q
j
v = v −r
ij
q
i
end for
r
jj
= |v|
2
q
j
= v/r
jj
end for
We can compare these four algorithms for a series of Hilbert matrices. Recall
that the Hilbert matrix gets quickly ill-conditioned (see Table 2).
Condition Number
H
4
2.8 10
+04
H
5
9.4 10
+05
H
6
2.9 10
+07
H
7
9.8 10
+08
H
8
3.4 10
+10
Table 2: Condition number for Hilbert matrices using the 1-norm
Table 3 describes the resulting error when the dimension of the Hilbert
matrix is increased.
Classical G-S Modified G-S Two-Step G-S Matlab qr
H
4
3.6 10
−11
4.0 10
−13
3.6 10
−16
7.5 10
−16
H
5
1.1 10
−08
9.0 10
−12
3.2 10
−16
1.3 10
−15
H
6
1.7 10
−04
6.9 10
−10
2.5 10
−16
9.2 10
−16
H
7
5.2 10
−02
5.6 10
−09
4.9 10
−16
1.1 10
−15
H
8
1.1 10
−00
3.4 10
−07
3.9 10
−13
1.4 10
−15
Table 3: Error
_
_
Q
T
Q−I
_
_
1
with different algorithms for Hilbert matrices
90
U. Washington AMATH 352 - Spring 2009
The results highlight that the classical Gram-Schmidt algorithm is unstable.
The modified Gram-Schmidt algorithm computes an approximate orthogonal
matrix with an error proportional to the condition number. When applying
twice the classical Gram-Schmidt algorithm, the error is close to machine pre-
cision. It increases only for H
8
but a third step of Gram-Schmidt would fix the
problem. Note that the Matlab function returns an orthogonal matrix up to
machine precision.
7.6 Useful commands in Matlab
Here are a few commands in Matlab useful for this section.
• eps contains the machine epsilon.
• bin2dec converts binary string to decimal integer.
• dec2bin converts decimal integer to a binary string.
• format sets the output format. For example, format long e selects the
floating point format with 15 digits for double and 7 digits for single.
• intmax returns the largest positive integer value.
• intmin returns the smallest positive integer value.
• realmax returns the largest positive floating point number.
• double converts to double precision.
• single converts to single precision.
91
U. Washington AMATH 352 - Spring 2009
8 Linear systems of equations
A linear system of m equations in n unknowns x
1
, x
2
, . . . , x
n
has the general
form
a
11
x
1
+ a
12
x
2
+ + a
1n
x
n
= b
1
a
21
x
1
+ a
22
x
2
+ + a
2n
x
n
= b
2
.
.
.
a
m1
x
1
+ a
m2
x
2
+ + a
mn
x
n
= b
m
(151)
where the a
ij
and b
j
are some given real numbers. This can be written as
Ax = b (152)
where A is an mn matrix and b ∈ R
m
. The problem is to find a vector x ∈ R
n
so that (152) holds.
We have seen that we can view matrix-vector multiplication as a linear com-
bination of the columns of the matrix A. So another way to write this system
is
_
¸
¸
¸
_
a
11
a
21
.
.
.
a
m1
_
¸
¸
¸
_
x
1
+
_
¸
¸
¸
_
a
12
a
22
.
.
.
a
m2
_
¸
¸
¸
_
x
2
+ +
_
¸
¸
¸
_
a
1n
a
2n
.
.
.
a
mn
_
¸
¸
¸
_
x
n
=
_
¸
¸
¸
_
b
1
b
2
.
.
.
b
m
_
¸
¸
¸
_
. (153)
In other words, we are trying to find the weights x
1
, x
2
, . . . , x
n
so that the vector
b can be expressed as a linear combination of the columns of A. The objective
of this Section is to describe general algorithms to solve such linear systems.
8.1 Counting the number of solutions
When solving linear system is viewed as trying to find the weights of a linear
combination (like (153)), we see that a linear system Ax = b has at least one
solution if (and only if) the vector b lies in the subspace of R
m
spanned by the
columns of A. We recall that the range of A, 1(A), is exactly the subspace
spanned by the columns of A.
A linear system Ax = b has at least one solution if b ∈ 1(A). Might it
have more than one solution? Suppose the system has two distinct solutions,
i.e. Ax = b and also Ay = b where x ,= y. Let z be the difference between x
and y: z = x −y ∈ R
n
. Then we have
Az = A(x −y) = Ax −Ay = b −b = 0.
We have found a nontrivial linear combination (since z ,= 0) of the columns of
A that gives the zero vector. Hence the columns of A are linearly dependent.
From this, we conclude:
• If a linear system has more than one solution then rank(A) < n.
92
U. Washington AMATH 352 - Spring 2009
Conversely, suppose rank(A) < n. Then the columns of A are linearly
dependent and the null space of A, ^(A), is non trivial. So there must be a
vector z ,= 0 for which Az = 0. Now suppose b ∈ 1(A) and we have one
solution x to Ax = b. Then x +z ,= x is also a solution, since
A(x +z) = Ax +Az = b +0 = b.
Also note that if Az = 0 then A(αz) = 0 for any scalar α ∈ R, so in this case
there are infinitely many different solutions to the linear system.
We have proved the following theorem:
Theorem 156. Consider the linear system Ax = b with A ∈ R
m×n
and
b ∈ R
m
. Then:
• If b / ∈ 1(A), then the system has no solution.
• If b ∈ 1(A) and rank(A) = n, then this system has a unique solution x.
• If b ∈ 1(A) and rank(A) < n, then this system has infinitely many
solutions.
A special case that often arises in practice is a system of n equations in n
unknowns, in which case m = n in our previous discussion. All the statements
made above still apply in this case, but now we see that there are basically two
cases:
• If rank(A) = n then 1(A) = R
n
and ^(A) = ¦0¦. In this case, we say
the matrix A is nonsingular or invertible. The system Ax = b has exactly
one solution for any right hand side b ∈ R
n
.
• If rank(A) < n then the range of A is a proper subspace of R
n
(not the
full space) and so Ax = b may or may not have solutions, depending on
whether b ∈ 1(A) or not. If it does have solutions then it has infinitely
many solutions, since the null space of A has dimension n−rank(A) ≥ 1.
The matrix A is singular or noninvertible.
In this section, we will focus on the case where b ∈ 1(A). The Section on
least-squares will study the case where b / ∈ 1(A).
Example 157. The linear system
_
3 0
2 4
_ _
x
1
x
2
_
=
_
3
14
_
has exactly one solution
_
1
3
_
. We can change the right hand side of this system
to any other vector and the system will still have exactly one solution.
93
U. Washington AMATH 352 - Spring 2009
Example 158. The linear system
_
_
2 6
−5 3
1 2
_
_
_
_
x
1
x
2
x
3
_
_
=
_
_
14
1
5
_
_
has exactly one solution
_
1
2
_
. The columns of A are linearly independent
(since the second is not a scalar multiple of the first). The matrix is of full rank
and its null space is equal to ¦0¦. On the other hand, the system
_
_
2 6
−5 3
1 2
_
_
_
_
x
1
x
2
x
3
_
_
=
_
_
1
1
5
_
_
has no solution.
Example 159. The linear system
_
3 0 −1
2 4 6
_
_
_
x
1
x
2
x
3
_
_
=
_
3
14
_
has infinitely many solutions,
x =
_
_
1
3
0
_
_
,
_
_
2
−2
3
_
_
,
_
_
0
8
−3
_
_
, . . .
The difference between any two of these solutions lies in the null space of A,
which is
^(A) = span
_
_
_
_
1
−5
3
_
_
_
_
.
We also know the null space should have dimension 1 since the rank of A is 2
and so dim(^(A)) = 3 −2 = 1.
8.2 Easy-to-solve systems
We begin by looking for special types of systems that can be easily solved.
Suppose that the nn matrix A has a diagonal structure. This means that
all the nonzero elements of A are on the diagonal of A. A linear system Ax = b
is
_
¸
¸
¸
¸
¸
¸
_
a
11
0 0
0 a
22
0
.
.
. 0
.
.
.
.
.
.
.
.
.
.
.
.
.
.
. 0
0 0 a
nn
_
¸
¸
¸
¸
¸
¸
_
_
¸
¸
¸
¸
¸
¸
_
x
1
x
2
.
.
.
.
.
.
x
n
_
¸
¸
¸
¸
¸
¸
_
=
_
¸
¸
¸
¸
¸
¸
_
b
1
b
2
.
.
.
.
.
.
b
n
_
¸
¸
¸
¸
¸
¸
_
.
94
U. Washington AMATH 352 - Spring 2009
The system collapses to n simple equations and the solution is
_
¸
¸
¸
¸
¸
¸
_
x
1
x
2
.
.
.
.
.
.
x
n
_
¸
¸
¸
¸
¸
¸
_
=
_
¸
¸
¸
¸
¸
¸
_
b
1
/a
11
b
2
/a
22
.
.
.
.
.
.
b
n
/a
nn
_
¸
¸
¸
¸
¸
¸
_
.
The computation requires n floating operations. If a
ii
= 0 and b
i
= 0 for some
index i, then x
i
can be any real number. If a
ii
= 0 and b
i
,= 0 for some index i,
then no solution exists. Note that for a diagonal matrix, its determinant is
det A = a
11
a
22
a
nn
=
n

i=1
a
ii
.
Suppose that the n n matrix A has a lower triangular structure. This
means that all the nonzero elements of A are below the main diagonal of A. A
linear system Ax = b is
_
¸
¸
¸
¸
¸
¸
_
a
11
0 0
a
21
a
22
0
.
.
.
.
.
.
.
.
.
.
.
. 0
a
n1
a
n2
a
nn
_
¸
¸
¸
¸
¸
¸
_
_
¸
¸
¸
¸
¸
¸
_
x
1
x
2
.
.
.
.
.
.
x
n
_
¸
¸
¸
¸
¸
¸
_
=
_
¸
¸
¸
¸
¸
¸
_
b
1
b
2
.
.
.
.
.
.
b
n
_
¸
¸
¸
¸
¸
¸
_
.
Assume that a
ii
,= 0 for all indices i. Then we obtain x
1
from the first equation.
With the known value of x
1
substituted in the second equation, we can solve
for x
2
. We proceed in the same way obtaining x
1
, x
2
, , x
n
one at a time and
in this order. A formal algorithm for this case is called forward substitution.
The total number of operations is n(n + 2) = O(n
2
). The solution is
_
¸
¸
¸
¸
¸
¸
_
x
1
x
2
.
.
.
.
.
.
x
n
_
¸
¸
¸
¸
¸
¸
_
=
_
¸
¸
¸
¸
¸
¸
_
b
1
/a
11
(b
2
−a
21
x
1
)/a
22
.
.
.
.
.
.
(b
n

n−1
j=1
a
nj
x
j
)/a
nn
_
¸
¸
¸
¸
¸
¸
_
.
If a
ii
= 0 for some index i, then the matrix is singular. Its determinant is
det A = a
11
a
22
a
nn
=
n

i=1
a
ii
.
Example 160. The linear system
_
3 0
2 4
_ _
x
1
x
2
_
=
_
3
14
_
95
U. Washington AMATH 352 - Spring 2009
is lower triangular. So we obtain x
1
= 3/3 = 1 and 4x
2
= 14 − 2 1 = 12. So
the exact solution is
_
1
3
_
.
Suppose that the n n matrix A has an upper triangular structure. This
means that all the nonzero elements of A are below the main diagonal of A. A
linear system Ax = b is
_
¸
¸
¸
¸
¸
¸
_
a
11
a
12
a
1n
0 a
22
a
2n
.
.
. 0
.
.
.
.
.
.
.
.
.
0 0 0 a
nn
_
¸
¸
¸
¸
¸
¸
_
_
¸
¸
¸
¸
¸
¸
_
x
1
x
2
.
.
.
.
.
.
x
n
_
¸
¸
¸
¸
¸
¸
_
=
_
¸
¸
¸
¸
¸
¸
_
b
1
b
2
.
.
.
.
.
.
b
n
_
¸
¸
¸
¸
¸
¸
_
.
Assume that a
ii
,= 0 for all indices i. Then we obtain x
n
from the last equation.
With the known value of x
n
substituted in the next-to-last equation, we can
solve for x
n−1
. We proceed in the same way obtaining x
n
, x
n−1
, , x
1
one at
a time and in this order. A formal algorithm for this case is called backward
substitution. The total number of operations is still O(n
2
). If a
ii
= 0 for some
index i, then the matrix is singular. Its determinant is
det A = a
11
a
22
a
nn
=
n

i=1
a
ii
.
Another simple type of system than can be easily solved using the same
ideas, is a system obtained by permuting the equations in a triangular system.
Consider the system
_
_
a
11
a
12
0
a
21
a
22
a
23
a
31
0 0
_
_
_
_
x
1
x
2
x
3
_
_
=
_
_
b
1
b
2
b
3
_
_
.
If we simply reorder these equations, we can get a lower triangular system:
_
_
a
31
0 0
a
11
a
12
0
a
21
a
22
a
23
_
_
_
_
x
1
x
2
x
3
_
_
=
_
_
b
3
b
1
b
2
_
_
.
Then we can solve the re-ordered system in the same manner as previously. But
the remaining question is how to determine beforehand the permutation. We
will discuss a general approach in the next subsection.
8.3 Gaussian elimination
Gaussian elimination is the simplest way to solve a linear system by hand and
also the standard way for solving linear systems on computers. Gaussian elimi-
nation transforms a full linear system into an upper triangular one by applying
simple linear transformations.
96
U. Washington AMATH 352 - Spring 2009
Assume that A ∈ R
n×n
. A lower triangular matrix with diagonal entries
equal to 1 can usually be found such that L
−1
A = U, where U is an upper
triangular matrix. Thus we obtain an LU factorization of A, i.e. A = LU.
Before giving the general algorithm, we discuss some examples.
Example 161. Consider the matrix
A =
_
¸
¸
_
2 1 1 0
4 3 3 1
8 7 9 5
6 7 9 8
_
¸
¸
_
.
The first step of Gaussian elimination looks like
L
1
A =
_
¸
¸
_
1 0 0 0
−2 1 0 0
−4 0 1 0
−3 0 0 1
_
¸
¸
_
_
¸
¸
_
2 1 1 0
4 3 3 1
8 7 9 5
6 7 9 8
_
¸
¸
_
=
_
¸
¸
_
2 1 1 0
0 1 1 1
0 3 5 5
0 4 6 8
_
¸
¸
_
.
We have subtracted twice the first row from the second, four times the first row
from the third, and three times the first row from the fourth. The second step
is
L
2
L
1
A =
_
¸
¸
_
1 0 0 0
0 1 0 0
0 −3 1 0
0 −4 0 1
_
¸
¸
_
_
¸
¸
_
2 1 1 0
0 1 1 1
0 3 5 5
0 4 6 8
_
¸
¸
_
=
_
¸
¸
_
2 1 1 0
0 1 1 1
0 0 2 2
0 0 2 4
_
¸
¸
_
.
Here we have subtracted three times the second row from the third and four
times the second row from the fourth. The final step is
L
3
L
2
L
1
A =
_
¸
¸
_
1 0 0 0
0 1 0 0
0 0 1 0
0 0 −1 1
_
¸
¸
_
_
¸
¸
_
2 1 1 0
0 1 1 1
0 0 2 2
0 0 2 4
_
¸
¸
_
=
_
¸
¸
_
2 1 1 0
0 1 1 1
0 0 2 2
0 0 0 2
_
¸
¸
_
= U.
Now we need to compute the matrix (L
3
L
2
L
1
)
−1
= L
−1
1
L
−1
2
L
−1
3
. It turns out
that the inverse of L
1
is a simple modification of the matrix L
1
where the terms
below the diagonal have been negated,
L
−1
1
=
_
¸
¸
_
1 0 0 0
−2 1 0 0
−4 0 1 0
−3 0 0 1
_
¸
¸
_
−1
=
_
¸
¸
_
1 0 0 0
2 1 0 0
4 0 1 0
3 0 0 1
_
¸
¸
_
.
The inverses of L
2
and L
3
are obtained in a similar fashion. Note that this
formula for the inverse is due to the special structure of the matrices L
i
(that
are equal to the identity matrix plus an outer product with only one nonzero
97
U. Washington AMATH 352 - Spring 2009
column). The special structure of these matrices result also in simple matrix-
matrix multiplications, i.e.
L
−1
1
L
−1
2
=
_
¸
¸
_
1 0 0 0
2 1 0 0
4 0 1 0
3 0 0 1
_
¸
¸
_
_
¸
¸
_
1 0 0 0
0 1 0 0
0 3 1 0
0 4 0 1
_
¸
¸
_
=
_
¸
¸
_
1 0 0 0
2 1 0 0
4 3 1 0
3 4 0 1
_
¸
¸
_
,
namely, we simply insert the nonzero subdiagonal entries in the appropriate
places. Finally, we get
L
−1
1
L
−1
2
L
−1
3
=
_
¸
¸
_
1 0 0 0
2 1 0 0
4 3 1 0
3 4 0 1
_
¸
¸
_
_
¸
¸
_
1 0 0 0
0 1 0 0
0 0 1 0
0 0 1 1
_
¸
¸
_
=
_
¸
¸
_
1 0 0 0
2 1 0 0
4 3 1 0
3 4 1 1
_
¸
¸
_
and
A = LU =
_
¸
¸
_
1 0 0 0
2 1 0 0
4 3 1 0
3 4 1 1
_
¸
¸
_
_
¸
¸
_
2 1 1 0
0 1 1 1
0 0 2 2
0 0 0 2
_
¸
¸
_
=
_
¸
¸
_
2 1 1 0
4 3 3 1
8 7 9 5
6 7 9 8
_
¸
¸
_
.
Example 162. Consider the matrix
A =
_
_
1 4 7
2 5 8
3 6 10
_
_
.
The Gaussian elimination performs first
L
1
A =
_
_
1 0 0
−2 1 0
−3 0 1
_
_
_
_
1 4 7
2 5 8
3 6 10
_
_
=
_
_
1 4 7
0 −3 −6
0 −6 −11
_
_
.
Then the next step gives
L
2
L
1
A =
_
_
1 0 0
0 1 0
0 −2 1
_
_
_
_
1 4 7
0 −3 −6
0 −6 −11
_
_
=
_
_
1 4 7
0 −3 −6
0 0 1
_
_
.
Finally, we write
A =
_
_
1 0 0
2 1 0
3 2 1
_
_
_
_
1 4 7
0 −3 −6
0 0 1
_
_
.
98
U. Washington AMATH 352 - Spring 2009
Before describing the Gaussian elimination, we recall that linear combina-
tions of matrix rows or columns and permutations of rows and columns can be
obtained by matrix-matrix multiplication. For example, the product of matrices
_
_
1 0 0
a 1 0
b 0 1
_
_
_
_
2 3 4
5 6 7
8 9 10
_
_
=
_
_
2 3 4
5 + 2a 6 + 3a 7 + 4a
8 + 2b 9 + 3b 10 + 4b
_
_
(154)
corresponds to adding a times the first row to the second row and adding b
times the first row to the third row. The product
_
_
1 0 0
0 1 0
0 c 1
_
_
_
_
2 3 4
5 6 7
8 9 10
_
_
=
_
_
2 3 4
5 6 7
8 + 5c 9 + 6c 10 + 7c
_
_
(155)
corresponds to adding c times the second row to the third row. The product
_
_
a 0 0
0 b 0
0 0 c
_
_
_
_
2 3 4
5 6 7
8 9 10
_
_
=
_
_
2a 3a 4a
5b 6b 7b
8c 9c 10c
_
_
(156)
corresponds to row scalings. The column operations are easily obtained by
transpose. For example, the product of matrices
_
_
2 3 4
5 6 7
8 9 10
_
_
_
_
1 α β
0 1 0
0 0 1
_
_
=
_
_
2 3 + 2α 4 + 2β
5 6 + 5α 7 + 5β
8 9 + 8α 10 + 8β
_
_
(157)
corresponds to adding α times the first column to the second column and adding
β times the first column to the third column. The product
_
_
2 3 4
5 6 7
8 9 10
_
_
_
_
1 0 0
0 1 γ
0 0 1
_
_
=
_
_
2 3 4 + 3γ
5 6 7 + 6γ
8 9 10 + 9γ
_
_
(158)
corresponds to adding γ times the second column to the third column. The
product
_
_
2 3 4
5 6 7
8 9 10
_
_
_
_
α 0 0
0 β 0
0 0 γ
_
_
=
_
_
2α 3β 4γ
5α 6β 7γ
8α 9β 10γ
_
_
(159)
corresponds to column scaling.
The general Gaussian elimination goes as follows. Consider a matrix A ∈
R
n×n
. If a
11
,= 0, then we write
_
¸
¸
¸
_
1 0 0
−a
21
/a
11
1 0
.
.
.
.
.
.
.
.
.
−a
n1
/a
11
0 1
_
¸
¸
¸
_
_
¸
¸
¸
_
a
11
a
12
a
1n
a
21
a
22
a
2n
.
.
.
.
.
.
a
n1
a
n2
a
nn
_
¸
¸
¸
_
=
_
¸
¸
¸
_
a
11
a
12
a
1n
0 ˜ a
22
˜ a
2n
.
.
.
.
.
.
0 ˜ a
n2
˜ a
nn
_
¸
¸
¸
_
.
99
U. Washington AMATH 352 - Spring 2009
We have eliminated all the entries below the diagonal in the first column. If
˜ a
22
,= 0, then we can repeat the process and eliminate all the entries below the
diagonal in the second column. Suppose that we have build p−1 lower triangular
matrices with diagonal entries equal to 1 such that the matrix L
p−1
. . . L
2
L
1
A
has the first p −1 columns with zero entries under the main diagonal, i.e.
L
p−1
. . . L
2
L
1
A =
_
¸
¸
¸
¸
¸
¸
¸
¸
¸
¸
_
a
11
a
1n
0
.
.
.
.
.
.
.
.
.
.
.
.
.
.
. 0 ˆ a
pp
. . . ˆ a
pn
.
.
.
.
.
.
.
.
.
.
.
.
0 . . . 0 ˆ a
np
. . . ˆ a
nn
_
¸
¸
¸
¸
¸
¸
¸
¸
¸
¸
_
.
Then, if ˆ a
pp
,= 0, we have
_
¸
¸
¸
¸
¸
¸
¸
¸
¸
¸
¸
¸
¸
¸
¸
¸
¸
_
1 0
0
.
.
.
.
.
.
.
.
.
.
.
.
.
.
. 0 1
.
.
.
.
.
. −
ˆ ap+1,p
ˆ app
.
.
.
.
.
.
.
.
.
.
.
. 0
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
0 . . . 0 −
ˆ an,p
ˆ app
0 . . . 0 1
_
¸
¸
¸
¸
¸
¸
¸
¸
¸
¸
¸
¸
¸
¸
¸
¸
¸
_
_
¸
¸
¸
¸
¸
¸
¸
¸
¸
¸
_
a
11
a
1n
0
.
.
.
.
.
.
.
.
.
.
.
.
.
.
. 0 ˆ a
pp
. . . ˆ a
pn
.
.
.
.
.
.
.
.
.
.
.
.
0 . . . 0 ˆ a
np
. . . ˆ a
nn
_
¸
¸
¸
¸
¸
¸
¸
¸
¸
¸
_
=
_
¸
¸
¸
¸
¸
¸
¸
¸
¸
¸
¸
¸
¸
_
a
11
a
1n
0
.
.
.
.
.
.
.
.
.
.
.
.
.
.
. 0 ˆ a
pp
.
.
.
.
.
. 0 ˇ a
p+1,p+1
. . . ˇ a
p+1,p+1
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
0 . . . 0 0 ˇ a
nn
. . . ˇ a
nn
_
¸
¸
¸
¸
¸
¸
¸
¸
¸
¸
¸
¸
¸
_
By building n−1 lower triangular matrices with diagonal entries equal to 1, the
resulting matrix is upper triangular.
100
U. Washington AMATH 352 - Spring 2009
When the lower triangular matrix L
p
is
L
p
=
_
¸
¸
¸
¸
¸
¸
¸
¸
¸
¸
¸
¸
¸
¸
¸
¸
¸
_
1 0
0
.
.
.
.
.
.
.
.
.
.
.
.
.
.
. 0 1
.
.
.
.
.
. −
ˆ ap+1,p
ˆ app
.
.
.
.
.
.
.
.
.
.
.
. 0
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
0 . . . 0 −
ˆ an,p
ˆ app
0 . . . 0 1
_
¸
¸
¸
¸
¸
¸
¸
¸
¸
¸
¸
¸
¸
¸
¸
¸
¸
_
,
its inverse is
L
−1
p
=
_
¸
¸
¸
¸
¸
¸
¸
¸
¸
¸
¸
¸
¸
¸
¸
¸
¸
_
1 0
0
.
.
.
.
.
.
.
.
.
.
.
.
.
.
. 0 1
.
.
.
.
.
.
ˆ ap+1,p
ˆ app
.
.
.
.
.
.
.
.
.
.
.
. 0
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
0 . . . 0
ˆ an,p
ˆ app
0 . . . 0 1
_
¸
¸
¸
¸
¸
¸
¸
¸
¸
¸
¸
¸
¸
¸
¸
¸
¸
_
.
The product L
−1
p
L
−1
p+1
is just the unit lower triangular matrix with the entries of
both L
−1
p
and L
−1
p+1
inserted in their usual places below the diagonal. When we
take the product of all of these matrices to form L, we have the same convenient
property everywhere below the diagonal
L = L
−1
1
. . . L
−1
n−1
=
_
¸
¸
¸
¸
¸
_
1 0 0
l
21
1
l
31
l
32
1
.
.
.
.
.
.
.
.
. 0
l
m1
l
m2
. . . l
m,m−1
1
_
¸
¸
¸
¸
¸
_
.
In practical Gauss elimination, the matrices L
k
are never formed. The coef-
ficients l
jk
are computed and stored directly into L. The complete algorithm is
described in Algorithm 6.
Counting the number of operations, we have
n−1

k=1
n

i=k+1
_
_
1 +
n

j=k
2
_
_
=
n−1

k=1
n

i=k+1
(2(n −k) + 3) =
n−1

k=1
(2(n −k) + 3) (n −k).
101
U. Washington AMATH 352 - Spring 2009
Algorithm 6 Gaussian elimination without pivoting
Set U = A and L = I
for k = 1 to n −1 do
for i = k + 1 to n do
l
ik
= u
ik
/u
kk
for j = k to n do
u
i,j
= u
i,j
−l
ik
u
k,j
end for
end for
end for
After changing variables, we get
n−1

k

=1
2k
2
+ 3k

= 2
n(n + 1)(2n + 1)
6
+ 3
n(n −1)
2

4n
3
6
+O(n
2
).
The total number of floating point operations is ≈
2
3
n
3
flops.
Remark. The Gaussian elimination can also be applied to rectangular matrices
but this is rarely done in practice. So we focus on square matrices.
Remark. We described the Gaussian elimination with an emphasis on zeroing
columns. We could just as easily introduce zeros in rows. The Crout factoriza-
tion takes the matrix form
A =
˜
L
˜
U,
where
˜
L is a lower triangular matrix and
˜
U is an upper triangular matrix with
unit entries on the diagonal.
Remark. If the matrix A has an LU factorization, then we can set D = diag(U)
and M
T
= D
−1
U and we write
A = LDM
T
,
where the matrix D is diagonal and both matrices L and Mare lower triangular
matrices with unit entries on the diagonal.
Remark. If the matrix A is symmetric, the LDM factorization becomes
A = LDL
T
.
This factorization requires ≈
n
3
3
floating point operations. When all the entries
on the diagonal of D are positive, then we can set R = D
1/2
L
T
and we obtain
A = R
T
R,
which is called the Cholesky factorization. When a symmetric matrix A has a
Cholesky factorization, it is symmetric positive definitive, i.e. it satisfies
∀x ∈ R
n
¸ ¦0¦ , x
T
Ax > 0.
102
U. Washington AMATH 352 - Spring 2009
To solve a linear system Ax = b by LU factorization, the following steps
are required:
1. Compute a LU factorization A = LU (Cost ≈
2
3
n
3
= O(n
3
))
2. Compute y = L
−1
b (forward substitution) (Cost ≈ n
2
)
3. Compute x = U
−1
y (backward substitution) (Cost ≈ n
2
)
This algorithm requires less floating point operations than a QR factorization.
Note that when you want to solve several linear systems with the same matrix A
and different right hand side vectors b, the LU factorization should be computed
only once. The forward and backward substitutions require then O(n
2
) floating
point operations per right hand side.
Example 163. Determine the polynomial p(t) = c
1
+ c
2
t that satisfies:
p(1) = 2 p(3) = 5.
The linear system associated with these equations is
c
1
+ c
2
= 2
c
1
+ 3c
2
= 5
Consider the matrix
A =
_
1 1
1 3
_
.
We have
_
1 0
−1 1
_ _
1 1
1 3
_
=
_
1 1
0 2
_
.
The LU factorization of A is
A =
_
1 1
1 3
_
=
_
1 0
1 1
_ _
1 1
0 2
_
.
The coefficients satisfy
_
1 0
1 1
_ _
1 1
0 2
_ _
c
1
c
2
_
=
_
2
5
_
and
_
1 1
0 2
_ _
c
1
c
2
_
=
_
1 0
−1 1
_ _
2
5
_
=
_
2
3
_
.
Finally, we obtain
_
c
1
c
2
_
=
_
1/2
3/2
_
.
103
U. Washington AMATH 352 - Spring 2009
Example 164. We want to find a quadratic polynomial
p(t) = c
1
+ c
2
t + c
3
t
2
(160)
such that
p(2) = 0.8, p(4) = 1.6, p(5) = 3.2. (161)
We write a linear system for the coefficients c
1
, c
2
, and c
3
:
c
1
+ 2c
2
+ 4c
3
= 0.8
c
1
+ 4c
2
+ 16c
3
= 1.6
c
1
+ 5c
2
+ 25c
3
= 3.2
.
We eliminate the parameter c
1
in the last two equations to get
c
1
+ 2c
2
+ 4c
3
= 0.8
2c
2
+ 12c
3
= 0.8
3c
2
+ 21c
3
= 2.4
.
Now, we eliminate the coefficient c
2
in the last equation
c
1
+ 2c
2
+ 8c
3
= 0.8
2c
2
+ 12c
3
= 0.8
3c
3
= 1.2
.
With backward substitution, we get
c
3
= 0.4
c
2
=
1
2
(0.8 −12 0.4) =
1
2
(0.8 −4.8) = −2
c
1
= 0.8 −2 2 + 4 0.4 = 3.2
.
So the polynomial is
p(t) = 3.2 −2t + 0.4t
2
.
Instead of using the basis (1, t, t
2
), we could choose the basis (1, t − 2, (t −
2)(t −4)) and write
p(t) = d
1
+ d
2
(t −2) + d
3
(t −2)(t −4).
Using the values (161), the linear system for the coefficients d
1
, d
2
, and d
3
is
d
1
= 0.8
d
1
+ 2d
2
= 1.6
d
1
+ 3d
2
+ 3d
3
= 3.2
.
Solving by forward substitution, we get
d
1
= 0.8
d
2
=
1
2
(1.6 −0.8) = 0.4
d
3
=
1
3
(3.2 −3 0.4 −0.8) = 0.4
.
So the polynomial is
p(t) = 0.8 + 0.4(t −2) + 0.4(t −2)(t −4).
104
U. Washington AMATH 352 - Spring 2009
8.4 Gaussian elimination with pivoting
Unfortunately, Gaussian elimination as presented so far is unusable for solving
general linear systems because it is not backward stable. For certain matrices,
Gaussian elimination fails entirely because it tries to divide by zero.
Example 165. Consider the matrix
A =
_
0 1
1 1
_
.
This matrix has full rank and is well-conditioned. Nevertheless, Gaussian elim-
ination fails at the first step.
Suppose we slightly perturb the matrix
A =
_
10
−20
1
1 1
_
.
The Gaussian elimination does not fail now. The following factors are produced
L =
_
1 0
10
20
1
_
U =
_
10
−20
1
0 1 −10
20
_
.
In floating point arithmetic, with 16 digits of accuracy, the numbers will be
rounded to the nearest floating point number. The floating point matrices will
be
˜
L =
_
1 0
10
20
1
_
˜
U =
_
10
−20
1
0 −10
20
_
.
These matrices are relatively close to the exact matrices. However, the product
is
˜
L
˜
U =
_
10
−20
1
1 0
_
.
The instability can be controlled by permuting the order of rows of the
matrix being operated on. This operation is called pivoting.
Row permutations can also be written in terms of matrix multiplication. For
example, we have
_
_
0 1 0
1 0 0
0 0 1
_
_
_
_
2 3 4
5 6 7
8 9 10
_
_
=
_
_
5 6 7
2 3 4
8 9 10
_
_
. (162)
Column permutations are also possible, for example
_
_
2 3 4
5 6 7
8 9 10
_
_
_
_
0 1 0
1 0 0
0 0 1
_
_
=
_
_
3 2 4
6 5 7
9 8 10
_
_
(163)
105
U. Washington AMATH 352 - Spring 2009
Note that permutation matrices are identity matrices with permuted rows. They
are orthogonal matrices, i.e. they satisfy
_
_
0 1 0
1 0 0
0 0 1
_
_
−1
=
_
_
0 1 0
1 0 0
0 0 1
_
_
T
=
_
_
0 1 0
1 0 0
0 0 1
_
_
. (164)
We can also combine rows and columns permutations. For example, we have
_
_
0 1 0
1 0 0
0 0 1
_
_
_
_
2 3 4
5 6 7
8 9 10
_
_
_
_
0 1 0
1 0 0
0 0 1
_
_
−1
=
_
_
5 6 7
2 3 4
8 9 10
_
_
_
_
0 1 0
1 0 0
0 0 1
_
_
=
_
_
6 5 7
3 2 4
9 8 10
_
_
8.4.1 Partial pivoting
At step k of Gaussian elimination, multiples of row k are subtracted from rows
k +1, . . . , n of the working matrix X in order to introduce zeros under the diag-
onal. The entry x
kk
plays a special role and is called the pivot. Schematically,
we have
_
¸
¸
¸
¸
_

x
kk




_
¸
¸
¸
¸
_
−→
_
¸
¸
¸
¸
_

x
kk

0 ∗ ∗ ∗
0 ∗ ∗ ∗
0 ∗ ∗ ∗
_
¸
¸
¸
¸
_
.
However, there is no particular reason why the k-th row must be chosen for
the elimination. For example, we could easily introduce zeros in column k by
adding multiples of some row i with k ≤ i ≤ n,
_
¸
¸
¸
¸
_



x
ik


_
¸
¸
¸
¸
_
−→
_
¸
¸
¸
¸
_

0 ∗ ∗ ∗
0 ∗ ∗ ∗
x
ik

0 ∗ ∗ ∗
_
¸
¸
¸
¸
_
.
So we are free to choose as pivot any entry in the k-th column below the diagonal,
as long as it is nonzero. The possibility that an entry x
kk
might be zero implies
that some flexibility in choosing the pivot may sometimes be necessary. In
practice and for numerical stability, it is common to pick as pivot the largest
number among the entries below the diagonal. This strategy is called partial
pivoting (as opposed to complete pivoting described in Section 8.4.2).
106
U. Washington AMATH 352 - Spring 2009
Example 166. Consider the matrix
A =
_
¸
¸
_
2 1 1 0
4 3 3 1
8 7 9 5
6 7 9 8
_
¸
¸
_
.
With partial pivoting, the first step is to interchange the first and third row
(left-multiplication by P
1
):
_
¸
¸
_
0 0 1 0
0 1 0 0
1 0 0 0
0 0 0 1
_
¸
¸
_
_
¸
¸
_
2 1 1 0
4 3 3 1
8 7 9 5
6 7 9 8
_
¸
¸
_
=
_
¸
¸
_
8 7 9 5
4 3 3 1
2 1 1 0
6 7 9 8
_
¸
¸
_
.
Then we eliminate values in the first column (left multiplication by L
1
)
_
¸
¸
_
1 0 0 0

1
2
1 0 0

1
4
0 1 0

3
4
0 0 1
_
¸
¸
_
_
¸
¸
_
8 7 9 5
4 3 3 1
2 1 1 0
6 7 9 8
_
¸
¸
_
=
_
¸
¸
_
8 7 9 5
0 −
1
2

3
2

3
2
0 −
3
4

5
4

5
4
0
7
4
9
4
17
4
_
¸
¸
_
.
Now we permute rows 2 and 4 (left multiplication by P
2
)
_
¸
¸
_
1 0 0 0
0 0 0 1
0 0 1 0
0 1 0 0
_
¸
¸
_
_
¸
¸
_
8 7 9 5
0 −
1
2

3
2

3
2
0 −
3
4

5
4

5
4
0
7
4
9
4
17
4
_
¸
¸
_
=
_
¸
¸
_
8 7 9 5
0
7
4
9
4
17
4
0 −
3
4

5
4

5
4
0 −
1
2

3
2

3
2
_
¸
¸
_
.
We eliminate the second column entries (left multiplication by L
2
)
_
¸
¸
_
1 0 0 0
0 1 0 0
0
3
7
1 0
0
2
7
0 1
_
¸
¸
_
_
¸
¸
_
8 7 9 5
0
7
4
9
4
17
4
0 −
3
4

5
4

5
4
0 −
1
2

3
2

3
2
_
¸
¸
_
=
_
¸
¸
_
8 7 9 5
0
7
4
9
4
17
4
0 0 −
2
7
4
7
0 0 −
6
7

2
7
_
¸
¸
_
.
The third and fourth rows are now interchanged (left multiplication by P
3
)
_
¸
¸
_
1 0 0 0
0 1 0 0
0 0 0 1
0 0 1 0
_
¸
¸
_
_
¸
¸
_
8 7 9 5
0
7
4
9
4
17
4
0 0 −
2
7
4
7
0 0 −
6
7

2
7
_
¸
¸
_
=
_
¸
¸
_
8 7 9 5
0
7
4
9
4
17
4
0 0 −
6
7

2
7
0 0 −
2
7
4
7
_
¸
¸
_
.
The final step is
_
¸
¸
_
1 0 0 0
0 1 0 0
0 0 1 0
0 0 −
1
3
1
_
¸
¸
_
_
¸
¸
_
8 7 9 5
0
7
4
9
4
17
4
0 0 −
6
7

2
7
0 0 −
2
7
4
7
_
¸
¸
_
=
_
¸
¸
_
8 7 9 5
0
7
4
9
4
17
4
0 0 −
6
7

2
7
0 0 0
2
3
_
¸
¸
_
.
107
U. Washington AMATH 352 - Spring 2009
The elimination can be written in matrix form
L
3
P
3
L
2
P
2
L
1
P
1
A = U.
The multiplying matrix on the left side of Adoes not look like a lower triangular
matrix. However, we can write
L
3
P
3
L
2
P
2
L
1
P
1
= L

3
L

2
L

1
P
3
P
2
P
1
,
where we have
L

3
= L
3
, L

2
= P
3
L
2
P
−1
3
, L

1
= P
3
P
2
L
1
P
−1
2
P
−1
3
.
Indeed, we have
L
3
P
3
L
2
P
2
L
1
P
1
= L
3
P
3
L
2
(P
−1
3
P
3
)P
2
L
1
(P
−1
2
P
−1
3
P
3
P
2
)P
1
.
The matrices L

2
and L

1
satisfy
L

2
=
_
¸
¸
_
1 0 0 0
0 1 0 0
0
2
7
1 0
0
3
7
0 1
_
¸
¸
_
L

1
=
_
¸
¸
_
1 0 0 0

3
4
1 0 0

1
2
0 1 0

1
4
0 0 1
_
¸
¸
_
.
The product of the matrices L

3
L

2
L

1
is a lower triangular matrix with unit en-
tries on the diagonal and it is easily invertible by negating the subdiagonal en-
tries, just as in Gaussian elimination without pivoting. Writing L = (L

3
L

2
L

1
)
−1
and P = P
3
P
2
P
1
, we have
PA = LU
or
_
¸
¸
_
0 0 1 0
0 0 0 1
0 1 0 0
1 0 0 0
_
¸
¸
_
_
¸
¸
_
2 1 1 0
4 3 3 1
8 7 9 5
6 7 9 8
_
¸
¸
_
=
_
¸
¸
_
1 0 0 0
3
4
1 0 0
1
2

2
7
1 0
1
4

3
7

1
3
1
_
¸
¸
_
_
¸
¸
_
8 7 9 5
0
7
4
9
4
17
4
0 0 −
6
7

2
7
0 0 0
2
3
_
¸
¸
_
.
For a general n n matrix, the factorization provided by Gaussian elimina-
tion with partial pivoting can be written in the form
PA = LU, (165)
where P is a permutation matrix, L is a lower triangular matrix with unit entries
on the diagonal and lower-triangular entries smaller or equal to 1 in magnitude,
and U is upper triangular.
The complete algorithm for Gaussian elimination with partial pivoting is
described in Algorithm 7. This algorithm requires the same number of floating
point operations, ≈
2
3
n
3
, and O(n
2
) comparisons to find the pivots.
108
U. Washington AMATH 352 - Spring 2009
Algorithm 7 Gaussian elimination with partial pivoting
Set U = A, L = I, and P = I
for k = 1 to n −1 do
Select i ≥ k to maximize [u
ik
[
for r = k to n do
tmp = u
k,r
u
k,r
= u
i,r
u
i,r
= tmp
end for
for r = 1 to k −1 do
tmp = l
k,r
l
k,r
= l
i,r
l
i,r
= tmp
end for
for r = 1 to n do
tmp = p
k,r
p
k,r
= p
i,r
p
i,r
= tmp
end for
for j = k + 1 to n do
l
jk
= u
jk
/u
kk
for r = k to n do
u
j,r
= u
j,r
−l
jk
u
k,r
end for
end for
end for
109
U. Washington AMATH 352 - Spring 2009
8.4.2 Complete pivoting
In complete pivoting, the selection of pivots is choosing the largest in the whole
matrix. It requires a significant amount of time because the comparisons are
now of order O(n
3
). In practice, this is rarely done because the improvement
in stability is marginal.
In matrix form, Gaussian elimination with complete pivoting takes the form
PAQ = LU
where P and Q are permutation matrices, L is a lower triangular matrix with
unit entries on the diagonal, and U is an upper triangular matrix.
8.5 Banded systems
The Gaussian elimination requires, approximately,
2
3
n
3
floating point opera-
tions. Table 4 lists estimates of the time required to compute the LU factoriza-
tion and solve a linear system on a 100 MFlops computer.
n Flops Time
10 6.6 10
2
6.6 10
−6
s
100 6.6 10
5
6.6 10
−3
s
1000 6.6 10
8
6.6 10
0
s
10000 6.6 10
11
6.6 10
3
s ≈ 111.11 min
100000 6.6 10
14
6.6 10
6
s ≈ 1850 h ≈ 77 days
1000000 6.6 10
17
6.6 10
9
s ≈ 77000 days ≈ 210 years
Table 4: Time estimates to solve Ax = b by Gaussian elimination on a 100
MFlops computer
We assume that all the problems fit in the memory and that the initial matrix
is dense (all the entries could be nonzero). Luckily, many practical systems have
some structure or involve sparse matrices (where most elements are zero). An
example of a sparse matrix is a diagonal matrix. Solving Ax = b, when A is
a diagonal matrix, requires only O(n) flops and this takes only 0.005 seconds
rather than 210 years!
Banded systems are another examples of matrices for which the Gaussian
elimination requires only O(n) flops. A matrix is banded whenever the equations
can be ordered so that each unknown x
i
appears in only a few equations in
a “neighborhood” of the i-th equation. Formally, we say that A has upper
bandwidth q if a
ij
= 0 whenever j > i + q and lower bandwidth p if a
ij
= 0
whenever i > j + p.
Theorem 167. Suppose A ∈ R
n×n
has an LU factorization A = LU. If A
has upper bandwidth q and lower bandwidth p, then U has upper bandwidth q
and L has lower bandwidth p.
110
U. Washington AMATH 352 - Spring 2009
The proof is given in Golub and Van Loan, “Matrix Computations”, The
John Hopkins University Press, 1996. Then the Gaussian elimination can be
specialized and will involve only 2npq flops. The forward substitution requires
about 2np flops and the backward substitution about 2nq flops. Note that,
however, pivoting will destroy the band structure.
Example 168. Consider a nn symmetric, tridiagonal, positive definite matrix
A. Setting
L =
_
¸
¸
¸
¸
¸
¸
¸
_
1 . . . 0
l
1
1
.
.
.
.
.
.
.
.
.
.
.
. 1
0 . . . l
n−1
1
_
¸
¸
¸
¸
¸
¸
¸
_
and D = diag(d
1,
. . . , d
n
), the equation A = LDL
T
gives
a
11
= d
1
a
k,k−1
= l
k−1
d
k−1
a
k,k
= d
k
+ l
2
k−1
d
k−1
= d
k
+ l
k−1
a
k,k−1
The complete algorithm to compute the factorization and to solve a linear sys-
tem 8n flops.
Example 169. Consider the mass spring system in Figure 9.
Figure 9: Mass-spring system
Newton’s law f = ma applied to this system yields
m
i
¨ x
i
= k
i
(x
i−1
−x
i
) (force on mass i from spring i)
+k
i+1
(x
i+1
−x
i
) (force on mass i from spring i + 1)
+f
i
(external force on mass i)
or
M¨ x = −Kx +f
where the matrix Mis diagonal and the matrix Kis tridiagonal. Mis called the
mass matrix and K is called the stiffness matrix. Electrical engineers analyzing
circuits arrive at an analogous equation by applying Kirchoff’s and related laws
instead of Newton’s law. In this case, x represents branch currents, Mrepresents
inductances, and K represents admittances.
111
U. Washington AMATH 352 - Spring 2009
8.6 Useful commands in Matlab
Here are a few commands in Matlab useful for this section.
• inv(A) computes the inverse of matrix A when it exists.
• [L,U,P]=lu(A) returns an upper triangular matrix in U, a lower trian-
gular matrix L with a unit diagonal, and a permutation matrix P, such
that LU = PA.
• If the matrix A is square, A¸b is roughly the same as inv(A)*b, except it
is more efficiently.
– It checks whether A is diagonal, banded, triangular, a permutation
of a triangular.
– If A is symmetric, use the Cholesky or the LDL
T
factorization
– Use the LU factorization.
– If A is not square, use the QR factorization
– For further information, look at “help mldivide”.
112
U. Washington AMATH 352 - Spring 2009
9 Least squares problem
The term least squares describes a frequently used approach to solving overde-
termined or inexactly specified systems of equations in an approximate sense.
Instead of solving the equations exactly, we seek only to minimize the sum of
the squares of the residuals.
9.1 Residuals and norms
Suppose the linear system Ax = b with A ∈ R
m×n
has no solution because b
is not in the range of A. There are many situations where this arises but we
still want to find a vector x ∈ R
n
that approximately solves the system. The
residual vector r ∈ R
m
is defined by
r = b −Ax.
If x solves the system exactly then r = 0. Otherwise we can use some vector
norm to measure how small the residual is, |r| = |b −Ax|. For any particular
x ∈ R
n
, this will be a nonnegative real number.
We say that a particular vector x

∈ R
n
is the “best approximation” in some
norm if
|b −Ax

| ≤ |b −Ax| , ∀x ∈ R
n
.
The “best” solution will depend on what norm we use.
If we use the 2-norm to measure the residual then
|b −Ax|
2
=
¸
¸
¸
_
m

i=1
(b
i
−(Ax)
i
)
2
(166)
where (Ax)
i
means the i-th component of the vector Ax ∈ R
m
. In this case,
we are trying to find the vector x that minimizes the sum of the squares of
the residuals, and the vector x

that minimizes this is called the least squares
solution to the linear system.
Remark. Minimizing the sum of squares or the square root of the sum of squares
gives the same minimizer x

since the square root is an increasing function. So
we can equally well minimize
|b −Ax|
2
2
=
m

i=1
(b
i
−(Ax)
i
)
2
(167)
The vector x

that minimizes (167) will be the same as the vector that minimizes
(166).
If some observations are more important or more accurate than others, then
we might associate different weights, ω
j
, with different observations and mini-
mize
|r|
2
ω
=
m

i=1
ω
i
r
2
i
.
113
U. Washington AMATH 352 - Spring 2009
With the 1-norm, we minimize the sum of the absolute values of the residuals:
|r|
1
=
m

i=1
[r
i
[ .
This problem can be reformulated as a linear programming problem, but it is
computationally more difficult than the least squares problem. The resulting
solutions are less sensitive the presence of spurious data points or outliers.
With the ∞-norm, we minimize the largest residual:
|r|

= max
i
[r
i
[ .
This is also known as a Chebyshev fit and can be reformulated as a linear
programming problem. Chebyshev fits are frequently used in the design of
digital filtes and in the development of approximations for use in mathematical
function libraries.
We will limit ourselves to least squares problem in this class.
Remark. We will generally assume m > n and that rank(A) = n so that the
columns of A are linearly independent. In this case it can be shown that the
least squares solution is unique. (If rank(A) < n, we could add a nontrivial
element of the null space of A to x

and get another solution.)
9.2 Examples of least squares problem
First we present some examples of least squares problems. In matlab, it is
easy to solve a linear least squares problem using the backslash operator. In
this lecture, we will see what’s going on under the hood when you do this.
Example 170. Consider the matrix A =
_
_
1
2
5
_
_
and the vector b =
_
_
6
7
8
_
_
.
We want to find the value of x that minimizes the length of the vector A[x] −b,
measured in the 2-norm. Using the backslash operator from Matlab, we get
x = 2. Note that, geometrically, the value of x is the one that makes the vector
A[x] −b perpendicular to the column vector A
_
_
1
2
5
_
_
T
_
_
_
_
1
2
5
_
_
2 −
_
_
6
7
8
_
_
_
_
=
_
_
1
2
5
_
_
T
_
_
−4
−3
2
_
_
= −4 −6 + 10 = 0.
Example 171. Consider the matrix A and the vector b
A =
_
_
−1 1
0 2
1 3
_
_
and b =
_
_
−2
6
8
_
_
.
114
U. Washington AMATH 352 - Spring 2009
Using the backslash operator from Matlab, we get x =
_
3
2
_
. The residual
vector is
r = b −Ax =
_
_
−2
6
8
_
_

_
_
−1 1
0 2
1 3
_
_
_
3
2
_
=
_
_
−2
6
8
_
_

_
_
−1
4
9
_
_
=
_
_
−1
2
−1
_
_
.
Note that, now, the residual vector is orthogonal to the column vectors of A
_
_
−1 1
0 2
1 3
_
_
T
_
_
−1
2
−1
_
_
=
_
−1 0 1
1 2 3
_
_
_
−1
2
−1
_
_
=
_
1 −1
−1 + 4 −3
_
=
_
0
0
_
.
Example 172. Find the quadratic function p(t) = c
1
+ c
2
t + c
3
t
2
that gives a
least squares fit to the data (t
i,
, y
i
) from the table below
i t
i
y
i
1 0 5
2 1 4
3 2 6
4 3 9
If we try to require p(t
i
) = y
i
, we get 4 equations in the 3 unknowns c
1
, c
2
,
and c
3
, which has the form
_
¸
¸
_
1 t
1
t
2
1
1 t
2
t
2
2
1 t
3
t
2
3
1 t
4
t
2
4
_
¸
¸
_
_
_
c
1
c
2
c
3
_
_
=
_
¸
¸
_
y
1
y
2
y
3
y
4
_
¸
¸
_
.
From the data above, the matrix A and the input vector y are
y =
_
¸
¸
_
5
4
6
9
_
¸
¸
_
A =
_
¸
¸
_
1 0 0
1 1 1
1 2 4
1 3 9
_
¸
¸
_
Note that if we only required 3 of these conditions we would get a 3 3 square
system with a unique solution, corresponding to a parabola going through those
three points. But there is no single quadratic function that goes through all
four points given in the table above, and the vector y does not lie in the column
space of the matrix A. Using the backslash operator in Matlab, we get
c =
_
_
4.9
−1.6
1
_
_
.
115
U. Washington AMATH 352 - Spring 2009
Figure 10 plots the data and the least-squares fit
Figure 10: Least-squares fit with a quadratic function
9.3 Normal equations
If we want a least squares solution to a linear system Ax = b where A ∈ R
m×n
with m ≥ n and rank(A) = n (the columns of A are linearly independent),
then the unique least squares solution can be found by solving the square nn
system
A
T
Ax = A
T
b. (168)
These equations are called the normal equations. It can be shown that if
rank(A) = n then rank(A
T
A) = n as well, so this matrix is nonsingular and
the system has a unique solution.
There are several ways to derive the normal equations. The best way to
remember where these equations come from is to rewrite (169) as
A
T
(Ax −b) = 0. (169)
Recall that the vector r = b − Ax is the residual, and (169) states that the
residual vector is orthogonal to all columns of A, and hence the residual is
orthogonal to 1(A), the space spanned by the columns of A.
Example 173. Consider the 2 1 least squares problem
_
2
1
_
x =
_
3
1
_
. (170)
The right hand side b does not lie in the column space of A, which is the line
in 2-space through the origin and the point
_
2
1
_
. The solution to the least
116
U. Washington AMATH 352 - Spring 2009
squares problem is the scalar x for which
_
2x
x
_
(a point on the line) is closest
(in Euclidean distance) to the point
_
3
1
_
. It is obvious from Figure 11 that
this is the point for which the residual is orthogonal to the line.
Figure 11: Solution to the least squares problem (170)
Another way to derive the normal equations is to minimize the function
r(x) = |b −Ax|
2
2
= (b −Ax)
T
(b −Ax) .
This function is a smooth function of x and is quadratic in the components of
x. To find the minimum for r, we can set r

(x) = 0. To find r

, we recall that
the derivative is the first linear term in the Taylor expansion. So we compute
r(x +y) = (b −Ax −Ay)
T
(b −Ax −Ay)
r(x +y) = (b −Ax)
T
(b −Ax −Ay) −y
T
A
T
(b −Ax −Ay)
r(x +y) = (b −Ax)
T
(b −Ax) −(b −Ax)
T
Ay −(Ay)
T
(b −Ax) + (Ay)
T
Ay
r(x +y) = r(x) −2y
T
A
T
(b −Ax) + (Ay)
T
Ay
So we obtain
r

(x) = 2A
T
(Ax −b) = 2
_
A
T
Ax −A
T
b
_
which gives the normal equation when setting it to 0.
Note that because r(x) is quadratic in the components of x, setting the
derivatives to 0 gives a linear systems of equations. This is why minimizing the
2-norm of residuals is easy to solve using linear algebra. In other norms, it is
not so easy. For example, with the 1-norm, the function r is not smooth and
we can not find the minimum by setting the derivative equal to 0.
117
U. Washington AMATH 352 - Spring 2009
9.3.1 Classical solution
The classical way to solve least-squares problems is to solve the normal equa-
tions. We assume that the matrix A is full rank. The matrix A
T
A is square,
symmetric positive definite. The standard method of solving is by a Cholesky
factorization. It constructs a factorization A
T
A = R
T
R, where R is upper
triangular. Algorithm 8 solves the least squares problems.The number of float-
Algorithm 8 Least squares via normal equations
Form the matrix A
T
A and the vector A
T
b.
Compute the Cholesky factorization A
T
A = R
T
R.
Solve the lower triangular system R
T
w = A
T
b.
Solve the upper triangular system Rx = w.
ing point operations is, approximately, mn
2
+
n
3
3
. However, if κ denotes the
condition number of matrix A, then the matrix A
T
A has condition number κ
2
.
Thus the best we can expect from the normal equations is
|x −x
computed
|
|x|
= O(κ
2

machine
).
9.3.2 QR factorization
The modern method for solving least squares problems is based upon reduced
QR factorization. One constructs a factorization A =
ˆ
Q
ˆ
R. So we write the
normal equations A
T
Ax = A
T
b and we have
ˆ
R
T
ˆ
Q
T
ˆ
Q
ˆ
Rx =
ˆ
R
T
ˆ
Q
T
b ⇒
ˆ
R
T
ˆ
Rx =
ˆ
R
T
ˆ
Q
T
b ⇒
ˆ
Rx =
ˆ
Q
T
b.
This last equation is upper triangular and
ˆ
R is nonsingular because matrix A
is full rank. The steps for computing the solution are described in Algorithm
9.The cost is dominated by the QR factorization. Recall that the Matlab
Algorithm 9 Least squares via reduced QR factorization
Compute the reduced QR factorization A =
ˆ
Q
ˆ
R.
Compute the vector
ˆ
Q
T
b.
Solve the upper triangular system
ˆ
Rx =
ˆ
Q
T
b.
function qr requires asymptotically 2mn
2

2
3
n
3
flops.
9.3.3 SVD factorization
Another method for solving least squares problems uses the SVD decomposition.
This decomposition is a very useful result.
Theorem 174. Let A ∈ R
m×n
with m ≥ n. Then we can write
A =
ˆ
U
ˆ
ΣV
T
,
118
U. Washington AMATH 352 - Spring 2009
where
ˆ
U is of dimension mn and satisfies
ˆ
U
T
ˆ
U = I, V is of dimension nn
and satisfies V
T
V = I, and
ˆ
Σ is a n n diagonal matrix diag(σ
1
, . . . , σ
n
),
where σ
1
≥ . . . ≥ σ
n
≥ 0. The column vectors of
ˆ
U are called left singular
vectors. The column vectors of V are called right singular vectors. The scalar
σ
i
are called singular values.
For the least squares problem, we write the normal equations A
T
Ax = A
T
b
and we obtain
V
ˆ
Σ
ˆ
U
T
ˆ
U
ˆ
ΣV
T
x = V
ˆ
Σ
ˆ
U
T
b ⇒
ˆ
ΣV
T
x =
ˆ
U
T
b.
The matrix
ˆ
Σ is nonsingular when the matrix A is full rank. The steps for
computing the solution are described in Algorithm 10.The cost is dominated
Algorithm 10 Least squares via SVD decomposition
Compute the reduced SVD decomposition A =
ˆ
U
ˆ
ΣV
T
.
Compute the vector
ˆ
U
T
b.
Solve the upper triangular system
ˆ
Σw =
ˆ
U
T
b.
Set x = Vw.
by the SVD decomposition. The Matlab function svd requires asymptotically
2mn
2
+ 11n
3
flops. This method is useful when the matrix A is close to rank-
deficient.
9.4 Approximation of functions
Given some function f(t), we want to find a function g(t) from some finite
dimensional function space that approximates f(t) over some interval [a, b]. For
example, we can choose g ∈ T
k
, a polynomial of degree k for some k.
Recall that a function space of dimension n can be described by n basis
functions φ
1
(t), . . . , φ
n
(t). The problem is to find the coefficients c
1
, . . . , c
n
in
g(t) = c
1
φ
1
(t) + . . . + c
n
φ
n
(t).
The best choice of “basis functions” φ
j
(t) depends on where the function f or
the data comes from. In some cases, we may have data that we think comes
from a linear function,
g(t) = c
1
+ c
2
t
in which case we would naturally choose n = 2, φ
1
(t) = 1 and φ
2
(t) = t. Or if
we want to fit by a quadratic we would use these functions along with φ
3
(t) = t
2
.
For some kinds of approximation, this problem can reduce to a linear algebra
problem for vector
c =
_
¸
_
c
1
.
.
.
c
n
_
¸
_.
In the following, we will discuss some approaches.
119
U. Washington AMATH 352 - Spring 2009
9.4.1 Interpolation
Given a set of n data points (t
i
, y
i
), we assume that the data should approxi-
mately satisfy y = g(t) where g(t) is a function that is a linear combination of
n given functions, φ
1
(t), . . . , φ
n
(t):
g(t) = c
1
φ
1
(t) + c
2
φ
2
(t) + + c
n
φ
n
(t).
Given the “basis functions” φ
j
(t) and the data, we can solve the problem of
finding the coefficients c
j
to minimize
n

i=1
(y
i
−g(t
i
))
2
by solving the linear least squares problem Ac = y, where the matrix A and
the data vector y are
y =
_
¸
¸
¸
_
y
1
y
2
.
.
.
y
n
_
¸
¸
¸
_
A =
_
¸
¸
¸
_
φ
1
(t
1
) . . . φ
n
(t
1
)
φ
1
(t
2
) φ
n
(t
2
)
.
.
.
.
.
.
φ
1
(t
n
) . . . φ
n
(t
n
)
_
¸
¸
¸
_
.
With reasonable choice of basis functions, this system will usually be nonsingu-
lar.
Example 175. Consider n distinct points t
1
, . . . , t
n
and data y
1
, . . . , y
n
. There
exists a unique polynomial interpolant to these data in these points, that is, a
polynomial of degree at most n −1,
p(x) = c
0
+ c
1
x + . . . + c
n−1
x
n−1
,
with the property p(t
i
) = y
i
for each index i. The relationship among the points,
the data, and the coefficients can be expressed by a linear system
_
¸
¸
¸
_
1 t
1
. . . t
n−1
1
1 t
2
t
n−1
2
.
.
.
.
.
.
1 t
n
. . . t
n−1
n
_
¸
¸
¸
_
_
¸
¸
¸
_
c
1
c
2
.
.
.
c
n
_
¸
¸
¸
_
=
_
¸
¸
¸
_
y
1
y
2
.
.
.
y
n
_
¸
¸
¸
_
.
We can solve this system of equations, which is guaranteed to be nonsingular as
long as the points ¦t
i
¦ are distinct. Figure 12 presents an example of polynomial
interpolation with a polynomial of degree 10 and 11 data points.
120
U. Washington AMATH 352 - Spring 2009
Figure 12: Degree 10 polynomial interpolant to 11 data points.
The interpolant passes through all the data points. However, the fit is not
pleasing. Near the ends of the interval, p exhibits large oscillations that are an
artifact of the interpolation, not a reflection of the data. This unsatisfactory
behavior is typical of polynomial interpolation. The produced fits are often bad
and they tend to get worse if more data are utilized. Even if the fit is good, the
interpolation process may be ill-conditioned, i.e. sensitive to perturbations of
the data.
Note that the structure of the system and ease of solution depends on the
choice of basis functions.
Example 176. Find a polynomial u ∈ T
2
interpolating f(t) =

t at t
i
= 1, 4, 9.
Instead of using the canonical basis (1, t, t
2
), we select the basis
((t −4)(t −9), (t −1)(t −9), (t −1)(t −4)) .
Then the system is
_
_
24 0 0
0 −15 0
0 0 40
_
_
_
_
c
1
c
2
c
3
_
_
=
_
_
1
2
3
_
_
.
The solution is
u(t) =
1
24
(t −4)(t −9) −
2
15
(t −1)(t −9) +
3
40
(t −1)(t −4).
9.4.2 Data fitting
Given a set of m data points (t
i
, y
i
), we assume that the data should approxi-
mately satisfy y = g(t) where g(t) is a function that is a linear combination of
n given functions, φ
1
(t), . . . , φ
n
(t):
g(t) = c
1
φ
1
(t) + c
2
φ
2
(t) + + c
n
φ
n
(t).
121
U. Washington AMATH 352 - Spring 2009
Typically, we have m > n.
Given the “basis functions” φ
j
(t) and the data, we can solve the problem of
finding the coefficients c
j
to minimize
m

i=1
(y
i
−g(t
i
))
2
by solving the linear least squares problem Ac = y, where the matrix A and
the data vector y are
y =
_
¸
¸
¸
_
y
1
y
2
.
.
.
y
m
_
¸
¸
¸
_
A =
_
¸
¸
¸
_
φ
1
(t
1
) . . . φ
n
(t
1
)
φ
1
(t
2
) φ
n
(t
2
)
.
.
.
.
.
.
φ
1
(t
m
) . . . φ
n
(t
m
)
_
¸
¸
¸
_
.
Here, again, the choice of basis functions governs the ease of solving the problem.
For example, if columns of A are orthogonal to one another, then A
T
A is
diagonal.
Example 177. Consider again the 11 data points used previously for the poly-
nomial interpolation. The fit can be improved by reducing the degree of the
polynomial and solve a least-squares problem. Figure 13 plots the data and the
least-squares fit
Figure 13: Degree 7 polynomial least squares fit to 11 data point
The new polynomial does not interpolate the data but it captures their over-
all behavior much better than the polynomial of degree 10. The computation is
also less sensitive to perturbations.
Example 178. Suppose we want to fit the data
122
U. Washington AMATH 352 - Spring 2009
i t
i
y
i
1 0 2
2 1 4
3 4 2
4 9 -4
by a function of the form g(t) = c
1

t + c
2
(t −1). The linear system would
be
_
¸
¸
_
0 −1
1 0
2 3
3 8
_
¸
¸
_
_
c
1
c
2
_
=
_
¸
¸
_
2
4
2
−4
_
¸
¸
_
.
The first column is the basis function φ
1
(t) =

t evaluated at the data points
and the second column is the basis function φ
2
(t) = (t − 1) evaluated at the
data points. In this case the right hand side happens to lie in the column space
of the matrix, and there is a solution c
1
= 4, c
2
= −2 that has zero residual. In
other words the 4 data points all lie on the curve g(t) = 4

t −2(t −1). If the
data y is changed, however, this would probably not be true and then the least
squares solution would give the fit minimizing the residual. Figure 14 plots the
data and the least-squares fit
Figure 14: Least-squares fit with a polynomial and the square root function
9.4.3 Global approximation
Instead of fitting m data points, we can try to find a function g(t) that is as
close as possible to f(t) at all points in [a, b]. Now we must measure the residual
123
U. Washington AMATH 352 - Spring 2009
(difference between g(t) and f(t)) in some function norm |f −g|. For example,
the norm
|f|

= max
a≤t≤b
[f(t)[
is a natural generalization of the ∞-norm in R
n
. The inner product
< u, v >=
ˆ
b
a
u(t)v(t)dt (171)
is a natural generalization of the Euclidian inner product in R
n
. The associated
norm
|u|
2
=

< u, u > =
¸
ˆ
b
a
[u(t)[
2
dt
is a generalization to functions of the Euclidian norm in R
n
.
For example, assume that
g(t) = c
1
φ
1
(t) + c
2
φ
2
(t)
and that we are trying to minimize |f −g|
2
, i.e.
ˆ
b
a
(f(t) −c
1
φ
1
(t) −c
2
φ
2
(t))
2
dt.
We expand the equation and gather the coefficients c
1
and c
2
c
2
1
ˆ
b
a
φ
1
(t)
2
dt + 2c
1
c
2
ˆ
b
a
φ
1
(t)φ
2
(t)dt + c
2
2
ˆ
b
a
φ
2
(t)
2
dt
−2c
1
ˆ
b
a
f(t)φ
1
(t)dt −2c
2
ˆ
b
a
f(t)φ
2
(t)dt +
ˆ
b
a
f(t)
2
dt.
To compute the minimum for the norm |f −g|
2
, we look for a stationary point,
i.e. we write that the derivatives with respect to c
1
and c
2
are equal to zero
´
b
a
f(t)φ
1
(t)dt −c
1
´
b
a
φ
1
(t)
2
dt −c
2
´
b
a
φ
1
(t)φ
2
(t)dt = 0
´
b
a
f(t)φ
2
(t)dt −c
1
´
b
a
φ
1
(t)φ
2
(t)dt −c
2
´
b
a
φ
2
(t)
2
dt = 0
.
The system looks nicer if we use the inner product notation
< φ
1
, φ
1
> c
1
+ < φ
1
, φ
2
> c
2
= < f, φ
1
>
< φ
2
, φ
1
> c
1
+ < φ
2
, φ
2
> c
2
= < f, φ
2
>
.
The system has the form Bc = y where
B =
_
< φ
1
, φ
1
> < φ
1
, φ
2
>
< φ
2
, φ
1
> < φ
2
, φ
2
>
_
y =
_
< f, φ
1
>
< f, φ
2
>
_
. (172)
124
U. Washington AMATH 352 - Spring 2009
These are the normal equations. When using n basis functions, the least squares
problem becomes Bc = y, where the matrix B and the data vector y are
y =
_
¸
_
< f, φ
1
>
.
.
.
< f, φ
n
>
_
¸
_ B =
_
¸
_
< φ
1
, φ
1
> . . . < φ
1
, φ
n
>
.
.
.
.
.
.
< φ
n
, φ
1
> . . . < φ
n
, φ
n
>
_
¸
_.
Note that the system (172) becomes diagonal if < φ
1
, φ
2
>= 0 or if we choose
an orthogonal basis (orthogonal for the inner product (171)). For example, the
Legendre polynomials are a family of orthogonal polynomials on the interval
[−1, −1] for the inner product (171). The first four Legendre polynomials are
P
0
(t) = 1
P
1
(t) = t
P
2
(t) =
3
2
t
2

1
2
P
3
(t) =
5
2
t
3

3
2
t
Example 179. Consider the function f(t) = cos(t) on [−1, 1]. We look for the
best fitting in T
1
. The basis φ
1
(t) = 1 and φ
2
(t) = t is orthogonal. We have
< φ
1
, φ
1
>= 2 < f, φ
1
>= 2 sin(1) =⇒ c
1
= sin(1) ≈ 0.8415
< φ
2
, φ
2
>=
2
3
< f, φ
2
>= 0 =⇒ c
2
= 0
Figure 15 plots the function f and the best approximation in T
1
.
Figure 15: Approximation of cosine function on [−1, 1] with a constant function.
125
U. Washington AMATH 352 - Spring 2009
Next we look for the best fitting in T
2
. The basis φ
1
(t) = 1, φ
2
(t) = t, and
φ
3
(t) =
3
2
(t
2

1
3
) are orthogonal. We have
< φ
1
, φ
1
>= 2 < f, φ
1
>= 2 sin(1) =⇒ c
1
= sin(1) ≈ 0.8415
< φ
2
, φ
2
>=
2
3
< f, φ
2
>= 0 =⇒ c
2
= 0
< φ
3
, φ
3
>=
2
5
< f, φ
3
>≈ −0.124 =⇒ c
3
≈ −0.31
Figure 16 plots the function f and the best approximation in T
2
.
Figure 16: Approximation of cosine function on [−1, 1] with a quadratic func-
tion.
9.5 Useful commands in Matlab
In matlab, the normal equations could be formed and solved by the single
command
>> x = (A’*A) \ A’*b;
Computationally, however, it is both easier and better to simply set
>> x = A \ b;
This solves the least squares problem via the QR factorization.
Other useful commands in Matlab for this section include:
• [U,S,V] = svd(A,0) produces the reduced SVD decomposition. If A is
mn with m ≥ n, then svd computes only the first n columns of
ˆ
U and
Σ is n n.
• p = polyfit(x,y,n) finds the coefficients of a polynomial p(x) of degree
n that fits the data, p(x
i
) to y
i
, in a least squares sense.
126
U. Washington AMATH 352 - Spring 2009
10 Eigenvalues and eigenvectors
In talking about eigenvalues we are only interested in square matrices, A ∈
R
n×n
. Then f (u) = Au defines a mapping from R
n
to itself. If we multiply an
arbitrary vector u by A it will get mapped to a new vector Au that is typically
linearly independent from u.
Example 180. If
A =
_
3 0
0 −2
_
u =
_
4
1
_
(173)
then
Au =
_
12
−2
_
and the vector Au is not a scalar multiple of u, so the two are linearly indepen-
dent.
However, there may be particular vectors v for which Av is just a scalar
multiple of v. For the above matrix (173), we have
v =
_
1
0
_
=⇒ Av =
_
3
0
_
= 3
_
1
0
_
. (174)
Such a vector is called an eigenvector of the matrix A. Of course x = 0 always
has this property since A0 = 0, but we are interested in nonzero eigenvectors
of the matrix.
Definition 181. The vector v ,= 0 is an eigenvector of A if Av = λv for some
scalar value λ. The scalar λ is then called an eigenvalue.
If v is an eigenvector than so is αv for any scalar α, since
A(αv) = αAv = α(λv) = λ(αv).
More generally, if u and v are two eigenvectors of A corresponding to the same
eigenvalue λ, then any linear combination of u and v is also an eigenvalue
A(αu + βv) = αAu + βAv = αλu + βλv = λ(αu + βv).
So the set of all eigenvectors corresponding to a particular eigenvalue λ is a
linear subspace of R
n
.
Remark. For a linear system of equations, the matrix A can have a different
number of rows and of columns. To ask about the eigenvalues of such a matrix
A would be meaningless. Eigenvalues and eigenvectors make sense only when
the vectors x and Ax have the same number of rows or when the matrix A is
square.
127
U. Washington AMATH 352 - Spring 2009
Remark. Eigenvalues and eigenvectors are useful for algorithmic and physical
reasons. Algorithmically, eigenvalue analysis can simplify solutions of certain
problems by reducing a coupled systems of equations to a collection of scalar
problems. Physically, eigenvalue analysis can give insight into the behavior of
evolving systems governed by linear equations. The most familiar examples
include the study of resonance (of musical instruments when struck or plucked
or bowed) and of stability.
10.1 How to determine the eigenvalues of a matrix?
Recall the identity matrix I, the n n matrix with the property that Iu = u
for any vector u ∈ R
n
.
If Av = λv with v ,= 0 then we can write this as Av = λIv and rearranging
gives
(A−λI)v = 0. (175)
So if λ is an eigenvalue of A the matrix (A− λI) has a nonzero null vector v.
This can happen only if the matrix (A−λI) is singular.
So one way to find the eigenvalues of a matrix is to try to figure out the
values of λ for which (A− λI) is singular. Note that this matrix is simply the
matrix A with the value λ subtracted from each diagonal element.
We will only consider the case n = 2 in detail, so A is a 2 2 matrix and
A−λI =
_
a
11
−λ a
12
a
21
a
22
−λ
_
.
For the 2 2 case, we know that this matrix is singular only if its determinant
is zero, where
det(A) = a
11
a
22
−a
12
a
21
.
So we get
det(A−λI) = (a
11
−λ)(a
22
−λ) −a
12
a
21
(176)
det(A−λI) = λ
2
−(a
11
+ a
22
)λ −a
12
a
21
Remember that for a particular matrix we know the a
ij
values, just some num-
bers, and we are trying to find λ so that this determinant is zero. This is just
a quadratic equation in λ.
Example 182. For the matrix in (173), we get
A−λI =
_
3 −λ 0
0 −2 −λ
_
and so
det(A−λI) = −(3 −λ)(2 + λ).
In this case, the quadratic is already factored for us, and we see that this can
be zero only if λ = 3 or λ = −2. We already knew that λ = 3 was an eigenvalue
128
U. Washington AMATH 352 - Spring 2009
because of (174). We now see that there should also be vectors v for which
Av = −2v. In fact, there are
v =
_
0
1
_
=⇒ Av =
_
0
−2
_
= (−2)
_
0
1
_
(177)
or any multiple of this vector is also an eigenvector.
In the next section, we will see how to determine the eigenvectors once we
know the eigenvalues. First, some more examples of computing eigenvalues.
Example 183. Let
A =
_
3 2
4 1
_
and so A−λI =
_
3 −λ 2
4 1 −λ
_
(178)
Then det(A−λI) = (3 −λ)(1 −λ) −8 = (λ −5)(λ + 1) so the eigenvalues are
λ
1
= 5 and λ
2
= −1.
Example 184. Let
A =
_
5 −5
−2 2
_
and so A−λI =
_
5 −λ −5
−2 2 −λ
_
(179)
Then det(A−λI) = (5−λ)(2−λ) −10 = λ
2
−7λ. So the eigenvalues are λ
1
= 0
and λ
2
= 7. Note that A has an eigenvalue equal to 0. It means that A−0I is
singular, i.e. A is singular.
Any singular matrix has at least one eigenvalue equal to zero, and any null
vector of the matrix is an eigenvector associated with this eigenvalue, Av = 0v.
For the matrix (182), the vector v =
_
2
5
_
is a null vector of the matrix and
hence an eigenvector Av = 0 = 0v.
For any 2 2 matrix, computing det(A−λI) will give a quadratic equation
in λ. This quadratic equation can always be factored in the form
(λ −λ
1
)(λ −λ
2
) (180)
where λ
1
and λ
2
are two numbers, though they may be complex numbers! Note
that even matrices with real elements can have complex eigenvalues. They often
play an important role in analyzing differential equations.
For some matrices, it may happen that λ
1
= λ
2
. The eigenvalues are said to
be repeated. For example, consider
A =
_
4 0
0 4
_
. (181)
For this matrix, we have
det(A−λI) = (λ −4)
2
and so λ
1
= λ
2
= 4. Note that for this matrix any vector is an eigenvector,
Au = 4u for every u ∈ R
2
.
129
U. Washington AMATH 352 - Spring 2009
Example 185. Let
A =
_
0 1
−1 0
_
and so A−λI =
_
−λ 1
−1 −λ
_
(182)
Then det(A−λI) = λ
2
+ 1 = (λ −i)(λ + i). So the eigenvalues are λ
1
= i and
λ
2
= −i. The eigenvalues are complex. They are complex conjugates of each
other since the matrix A is real.
If n > 2 then it is harder to find the eigenvalues. We still want to find
values of λ for which (A−λI) is singular. For larger matrices, the determinant
can also be defined, and the matrix is singular if the determinant is zero. The
determinant of (A − λI) turns out to be a polynomial p(λ) of degree n in λ,
which can always be factored as
p(λ) = (λ −λ
1
) . . . (λ −λ
n
) (183)
and so a matrix of degree n always has exactly n eigenvalues (though some may
be repeated). Some may be complex (if A is real the complex ones always occur
in conjugate pairs).
Definition 186. The set of all eigenvalues of a matrix A is called the spectrum
of A, a subset of C.
We will not learn how to do this by hand for larger matrices. It is usually
very messy just to determine the polynomial, and then there is usually no way
to determine the roots except by using a numerical method. Actually numeri-
cal methods for determining eigenvalues do not determine this polynomial and
search for its roots — it is too difficult to do this way. There are many dif-
ferent algorithms that have been invented for approximating eigenvalues, too
complicated to outline here.
There are some cases where it is easy to determine the eigenvalues of a
matrix.
• If the matrix is diagonal then the n values on the diagonal are the eigen-
values of the matrix. An eigenvector for the λ
j
= a
jj
is given by the j-th
column of the identity matrix (the vector with a 1 in the j-th component
and 0 everywhere else).
• If the matrix is upper triangular or lower triangular then the eigenvalues
are again the diagonal elements, though the eigenvectors are not just the
unit vectors in this case.
Finally, we introduce a definition.
Definition 187. The algebraic multiplicity for an eigenvalue λ is the multiplic-
ity of λ as a root of the characteristic polynomial.
130
U. Washington AMATH 352 - Spring 2009
10.2 How to find the eigenvectors?
Once we know the eigenvalues of a matrix, how do we compute the eigenvectors?
Recall that if λ is an eigenvalue then (A− λI) is singular, and (175) just says
that an eigenvector v should be a null vector of this matrix.
So, the set of all eigenvectors for the eigenvalue λ is just the null space of the
matrix (A− λI). This space is called the eigenspace of the matrix associated
with this eigenvalue.
Example 188. Consider the matrix
A =
_
3 0
0 −2
_
.
We know λ
2
= −2 is an eigenvalue, for example. The associated eigenspace is
the null space of
A+ 2I =
_
5 0
0 0
_
.
The null space of this matrix is span
__
0
1
__
and this is the eigenspace.
Example 189. For the matrix
A =
_
3 2
4 1
_
,
the eigenvalues are λ
1
= 5 and λ
2
= −1. Let’s find the null space for λ
1
= 5:
A−5I =
_
−2 2
4 −4
_
.
The second column is −1 times the first, so v =
_
1
1
_
is a null vector, for
example, and is an eigenvector of A,
Av =
_
3 2
4 1
_ _
1
1
_
=
_
5
5
_
= 5v.
Example 190. For the matrix
A =
_
4 0
0 4
_
,
λ
1
= λ
2
= 4 and
A−4I =
_
0 0
0 0
_
.
The null space of this matrix is all of R
2
, so every vector is an eigenvector, as
we already knew. In this case the eigenspace is 2-dimensional. This can only
happen for repeated roots. Isolated roots have 1-dimensional eigenspaces, even
for large matrices.
131
U. Washington AMATH 352 - Spring 2009
As stated previously, the set of eigenvectors for an eigenvalue forms a sub-
space of R
n
. It is known as the eigenspace. The dimension of the eigenspace is
the number of linearly independent eigenvectors.
Definition 191. The geometric multiplicity for an eigenvalue λ is the dimension
for the eigenspace associated with the eigenvalue λ.
So far, we have seen examples where the algebraic multiplicity and the geo-
metric multiplicity are equal. In general, this is not true. The algebraic multi-
plicity is always greater or equal to the geometric multiplicity.
Example 192. Consider the matrices
A =
_
_
2 0 0
0 2 0
0 0 2
_
_
B =
_
_
2 1 0
0 2 1
0 0 2
_
_
.
Both matrices have characteristic polynomial (λ −2)
3
. 2 has an algebraic mul-
tiplicity equal to 3. For the matrix A, we can choose three independent eigen-
vectors
_
_
1
0
0
_
_
,
_
_
0
1
0
_
_
,
_
_
0
0
1
_
_
.
So the geometric multiplicity is also 3. However, for the matrix B, we can find
only one independent eigenvector
_
_
1
0
0
_
_
.
So the geometric multiplicity is only 1.
10.3 Eigenvalue decomposition
An eigenvalue decomposition of matrix A is a factorization
A = VΛV
−1
. (184)
Here the matrix V is nonsingular and Λ is a diagonal matrix. Note that such a
decomposition does not always exist.
The decomposition (184) can be rewritten
AV = VΛ (185)
or
A
_
_
v
1
¸
¸
¸
¸
¸
¸

¸
¸
¸
¸
¸
¸
v
n
_
_
=
_
_
v
1
¸
¸
¸
¸
¸
¸

¸
¸
¸
¸
¸
¸
v
n
_
_
_
¸
_
λ
1
.
.
.
λ
n
_
¸
_. (186)
132
U. Washington AMATH 352 - Spring 2009
This implies that Av
j
= λ
j
v
j
and that the j-th column of V is an eigenvector
of A associated with the eigenvalue λ
j
.
The eigenvalue decomposition (184) expresses a change of basis to eigenvec-
tor coordinates. If Ax = b and A = VΛV
−1
, we have
V
−1
b = ΛV
−1
x.
To compute Ax, we can expand x in the basis of columns of V, apply Λ, and
interpret the result as a vector of coefficients for a linear combination of the
columns of V. To solve Ax = b, we can expand b in the basis of columns
of V, solve a diagonal system with Λ, and interpret the result as a vector of
coefficients for a linear combination of the columns of V.
Definition 193. An nn matrix A is diagonalizable when it has an eigenvalue
decomposition A = VΛV
−1
.
Note that, from the eigendecomposition (184), we can not conclude that
the matrix A behaves like the diagonal matrix Λ. The statement depends on
what aspect of behavior one measures and on the matrix V. A great deal of
information can be contained in the matrix V. For example, the determinant
of A satisfies
det(A) = det(VΛV
−1
) = det(V)det(Λ)det(V)
−1
= det(Λ) =
n

i=1
λ
i
(187)
and the trace of A satisfies
tr(A) = tr(VΛV
−1
) = tr(V
−1
VΛ) = tr(Λ) =
n

i=1
λ
i
. (188)
However, the 2-norm of A satisfies
|A|
2
=
_
_
VΛV
−1
_
_
2
≤ |V|
2
|Λ|
2
_
_
V
−1
_
_
2
= κ
2
(V) |Λ|
2
. (189)
There are matrices for which this bound is sharp.
It sometimes happens that the matrix A is diagonalizable and that the n
linearly independent eigenvectors can be chosen orthogonal. In this case, the
matrix A is diagonalizable and there exists an orthogonal matrix Q such that
A = QΛQ
T
. (190)
Theorem 194. A symmetric matrix A is diagonalizable. Its eigenvalues are
real and there exists an orthogonal matrix Q such that
A = QΛQ
T
where Q
T
Q = I. (191)
One final factorization is actually the one that is most useful in numerical
analysis because all matrices can be factored in this way.
133
U. Washington AMATH 352 - Spring 2009
Theorem 195. Every square matrix A has a Schur factorization
A = QTQ

(192)
where Q is unitary and T is upper triangular.
Note that the triangular matrix T can have complex entries. Matrices A
and T have the same eigenvalues. So the eigenvalues of A necessarily appear
on the diagonal of T. Alternatively, for a real square matrix, we can write a
real reduction to an upper quasi-triangular.
Theorem 196. Every square real matrix A has a real Schur factorization:
A = Q
_
¸
¸
¸
_
R
11
R
12
. . . R
1m
0 R
22
R
2m
.
.
.
.
.
.
.
.
.
0 . . . 0 R
mm
_
¸
¸
¸
_
Q
T
(193)
where Q is orthogonal and each block R
ii
is either a 1 1 matrix or a 2 2
matrix having complex conjugate eigenvalues.
Example 197. Consider a matrix B ∈ R
m×n
. Recall that the Frobenius norm
is
|B|
F
=
¸
¸
¸
_
m

i=1
n

j=1
b
2
ij
=
_
tr (B
T
B).
The matrix B
T
B is symmetric. So it is diagonalizable, i.e. B
T
B = QΛQ
T
. So
we get
|B|
F
=
_
tr (B
T
B) =
_
tr (QΛQ
T
) =
_
tr (ΛQ
T
Q).
Since Q
T
Q = I, we get
|B|
F
=
_
tr (Λ) =
¸
¸
¸
_
n

i=1
λ
i
(B
T
B). (194)
Example 198. Consider a matrix A ∈ R
m×n
which is full rank. The matrix
A
T
A is symmetric. So it is diagonalizable, i.e. A
T
A = QΛQ
T
. Note that the
eigenvalues of A
T
A are always positive. If we have A
T
Ax = λx, then we get
x
T
A
T
Ax = λx
T
x and
|Ax|
2
2
= λ|x|
2
2
which implies that λ ≥ 0. The matrix A is full rank which implies that A
T
A is
invertible. Consequently, all the eigenvalues λ
i
are strictly positive. Denote σ
i
the positive square root of λ
i
and Σ the square diagonal matrix whose entries
are σ
1
, . . . , σ
n
. We have A
T
A = QΣ
2
Q
T
. We will show that Σ and Q are
two components of the singular value decomposition of A. Consider the matrix
134
U. Washington AMATH 352 - Spring 2009
U = AQ
T
Σ
−1
which belongs to R
m×n
. The matrix U has orthonormal column
vectors. Indeed, we have
U
T
U = Σ
−1
QA
T
AQ
T
Σ = Σ
−1
QQΣ
2
Q
T
Q
T
Σ = I.
Finally, we write
A = UΣV
T
where U is an m n matrix with orthonormal column vectors, Σ is an n n
diagonal matrix whose entries are positive, and V = Q
T
is an nn orthogonal
matrix. The scalar σ
i
are called the singular values of A. The column vectors
of V are called the right singular vectors. The column vectors of U are called
the left singular vectors. Note that we have
Av
i
= σ
i
u
i
. (195)
10.4 Case of symmetric matrices
In this section, we simplify matters by considering only matrices that are real
and symmetric. Recall that such a matrix has real eigenvalues and a complete
set of orthonormal eigenvectors. We denote λ
1
, . . . , λ
n
the set of eigenvalues
ordered in a non-decreasing way and q
1
, . . . , q
n
the associated orthonormal
eigenvectors.
10.4.1 Rayleigh quotient
Consider a real symmetric matrix A. The Rayleigh quotient of a non-zero vector
x ∈ R
n
is the scalar
ρ(x) =
x
T
Ax
x
T
x
. (196)
Notice that if x is an eigenvector, then its Rayleigh quotient is equal to the
associated eigenvalue. The Rayleigh quotient plays a big role in the computation
of eigenvalues and eigenvectors. It has some nice properties.
• The Rayleigh quotient is bounded and ranges over the interval [λ
1
, λ
n
],
i.e. for any nonzero vector x, we have
λ
1
≤ ρ(x) =
x
T
Ax
x
T
x
≤ λ
n
. (197)
• The decomposition is
Ax = ρ(x)x + (Ax −ρ(x)x)
orthogonal, i.e.
x
T
(Ax −ρ(x)x) = 0.
• The Rayleigh quotient is the scalar minimizing the norm |Ax −αx|
2
, i.e.
|Ax −ρ(x)x|
2
≤ |Ax −αx|
2
, ∀α ∈ R.
135
U. Washington AMATH 352 - Spring 2009
• The eigenvectors of A are the stationary points for the Rayleigh quotient.
Indeed, we have
∇ρ(x) =
2
x
T
x
(Ax −ρ(x)x).
The Rayleigh quotient appears in the definition of the matrix 2-norm. Recall
that the matrix 2-norm is associated with the Euclidean vector norm, namely
|B|
2
= max
x=0
|Bx|
2
|x|
2
= max
x=0
_
(Bx)
T
Bx
x
T
x
= max
x=0
_
x
T
B
T
Bx
x
T
x
. (198)
The matrix B
T
B is symmetric. So the matrix 2-norm of B is the square root of
the largest value for the Rayleigh quotient of B
T
B. Based on (197), the matrix
2-norm of B is equal to the square root of the largest eigenvalue of of B
T
B,
|B|
2
=
_
λ
max
(B
T
B). (199)
Note that the eigenvalues of B
T
B are always positive. Indeed we have
If B
T
Bx = λx, then x
T
B
T
Bx = λx
T
x ⇒ |Bx|
2
2
= λ|x|
2
2
which implies that λ ≥ 0.
10.4.2 Power iteration
Consider the following algorithm.Note that we did not specify any stopping
Algorithm 11 Power iteration
Set v
(0)
an arbitrary vector in R
n
of norm 1,
_
_
v
(0)
_
_
2
= 1.
for k = 1, 2, . . . do
w = Av
(k−1)
v
(k)
= w/ |w|
2
λ
(k)
=
_
v
(k)
_
T
Av
(k)
end for
criterion for this algorithm. In practice, termination criteria are very important.
To analyze the power iteration algorithm, notice that v
(k)
is proportional to
the vector A
k
v
(0)
. Next, we write v
(0)
as a linear combination of the orthonor-
mal eigenvectors
v
(0)
= a
1
q
1
+ . . . + a
n
q
n
.
Recall that Aq
i
= λ
i
q
i
and A
k
q
i
= λ
k
i
q
i
. So we get
v
(k)
= c
k
_
λ
k
1
a
1
q
1
+ . . . + λ
k
n
a
n
q
n
_
where c
k
is a scalar for the normalization of v
(k)
c
k
=
1
_
_
λ
k
1
a
1
q
1
+ . . . + λ
k
n
a
n
q
n
_
_
2
=
1
_
λ
2k
1
a
2
1
+ . . . + λ
2k
n
a
2
n
.
136
U. Washington AMATH 352 - Spring 2009
Factorizing byλ
k
n
, we obtain
v
(k)
=
1
_

1

n
)
2k
a
2
1
+ . . . + a
2
n
_
_
λ
1
λ
n
_
k
a
1
q
1
+
_
λ
2
λ
n
_
k
a
2
q
2
+ . . . + a
n
q
n
_
.
From here, we get the following result.
Theorem 199. Suppose the eigenvalues of A satisfy [λ
n
[ > [λ
n−1
[ ≥ . . . ≥ [λ
1
[
and q
T
n
v
(0)
= a
n
,= 0. Then the iterates for the power iteration algorithm satisfy
_
_
_v
(k)
−q
n
_
_
_
2
= O
_
¸
¸
¸
¸
λ
n−1
λ
n
¸
¸
¸
¸
k
_
,
¸
¸
¸λ
(k)
−λ
n
¸
¸
¸ = O
_
¸
¸
¸
¸
λ
n−1
λ
n
¸
¸
¸
¸
2k
_
(200)
when k →+∞.
Depending on the normalization step, the iterates can actually converge to
−q
n
. But the convergence rates do not change. Notice that the approximate
eigenvalue converges at a rate twice faster than the vector iterates. However,
power iteration is of limited use. First, it can find only the eigenvector corre-
sponding to the largest eigenvalue. Second, the convergence is linear, reducing
the error by a constant factor ≈ [λ
n−1

n
[. The convergence can be slow when
the two eigenvalues are close to each other. Combined with other ingredients,
this tool can result in a powerful algorithm.
10.5 Useful commands in Matlab
Here are a few commands in Matlab useful for this section.
• null(A) computes an orthonormal basis for the null space of A, when it
exists.
• d = eig(A) returns a vector of the eigenvalues of matrix A.
• [V, D] = eig(A) produces matrices of eigenvalues D and eigenvectors V
of matrix A, so that AV = VD.
• T = schur(A) returns the Schur matrix T.
• T = schur(A,’real’) returns the matrix T for the real Schur factoriza-
tion.
• [U, T] = schur(A) produces the Schur matrix T and a unitary matrix
U, so that AU = UT and U

U = I.
• [Q, T] = schur(A,’real’) produces the real Schur factorization: a ma-
trix T and an orthogonal matrix Q, so that AQ = QT and Q
T
Q = I.
137
U. Washington AMATH 352 - Spring 2009
A Notations
• R is the set of all real numbers (−1, π, −

2, ...).
• C is the set of all complex numbers.
• a ∈ R means “a is in R” or “a belongs to R”.
• [a, b] denotes an interval: if a ≤ x ≤ b, then x ∈ [a, b]
• (a, b) denotes an interval: if a < x < b, then x ∈ (a, b)
• [a, b) denotes an interval: if a ≤ x < b, then x ∈ [a, b)
• [a, b] ⊂ R means that [a, b] is included in R.
• R ⊃ [a, b] means that R contains [a, b].
• ∀ means “for all”.
• ∃ means “there exists”.
138
U. Washington AMATH 352 - Spring 2009
B Introduction to MATLAB
matlab is a very convenient language for testing simple algorithms, analyz-
ing data interactively, and plotting results. This language is widely used by
mathematicians and engineers.
Following are some hints on using matlab that may be particularly useful
for getting started in this class. Check out the matlab tutorials linked from
the class webpage for more hints.
• To start matlab on the PCs in the AS lab, click on the Start menu and
then Programs and then on Matlab. matlab has a nice user interface
that you should experiment with and use to learn effectively.
• Type “helpwin” at the matlab prompt to bring up some introductory
information and browse all the help pages. You might also try the demo
command in matlab.
• Use the help (or helpwin) and lookfor commands in matlab to learn
about how various commands and functions work. For example,
helpwin plot
will give you information about how to use the plot command,
lookfor axis
will give a list of all commands whose helpfile contains the string “axis” and
might help you find the command you are looking for regarding setting
axes on a plot. The which command is useful if there may be multiple
“.m” files with the same name and you are not sure which is being used.
• Temporary file storage is in the C:¸Temp directory. This should be where
matlab works by default (for example if you create a new “.m” file – also
called m-file – from matlab). When you log out, the C:¸Temp directory
will automatically be moved to C:¸oldtemp so the next user will have
an empty Temp folder. If you forget to save something on a flash drive
or another account that you need, you can log back on and recover it
from C:¸oldtemp. However, the next time you or someone else logs out,
C:¸oldtemp is overwritten and the previous version is lost forever.
If you compute something in matlab, say
>> x = sqrt(17)
matlab will do the computation and then print out the result on the screen. If
you want to do the computation but not print out the result, end the command
with a semi-colon,
>> x = sqrt(17);
This is particularly useful if you are doing computations on vectors or matrices
where the result may consist of hundreds or thousands of numbers.
If you just type
139
U. Washington AMATH 352 - Spring 2009
>> sqrt(17)
without assigning this to a variable, it will compute it and assign it to a variable
named ans. You can then use ans in a future computation. (But ans will be
redefined next time you compute something without assigning it elsewhere!)
You can have more than one command on a line if you separate them by
semicolons, or by commas if you want the result printed, for example
>> a = sqrt(17); b = 12, c = 3
the values of a, b, c will all be computed and the values of b and c will be
printed out.
In m-file scripts containing lots of commands, it is generally best to put only
one command on each line for readability. However, the examples in these notes
will sometimes put multiple command on a line to save space.
matlab views all numerical variables as matrices. A single number, e.g. x
= 2 can be thought of as a 1 1 matrix. A row vector, e.g.,
x = 1:4 which is the same as x = [1 2 3 4] or x = [1, 2, 3, 4]
is a 1 4 matrix. A column vector, e.g.
x = (1:4)’ which is the same as x = [1; 2; 3; 4]
is a 4 1 matrix. The command size(x) will tell you the size.
When using vectors only as a data structure to store a collection of numbers,
it often doesn’t matter whether they are thought of as row vectors or column
vectors. In linear algebra it is often important to distinguish between the two.
Our vectors will always be column vectors unless it is clear that a row vector
is meant. Row vectors can be transformed into column vectors and vice versa
by the transpose operator, denoted by a superscript T. In matlab, the prime
symbol is used, so for example [1; 2; 3]’ = [1, 2, 3].
In linear algebra we frequently use matrix-vector and matrix-matrix multi-
plication, which can be easily done in matlab. For example, if A is a 3 4
matrix and x is a 4 1 vector, then the command
>>b = A*x
computes the matrix-vector product (a 3 1 vector). For the multiplication
operator * to work, the matrices involved must have appropriate dimensions.
In addition to doing linear algebra, we also frequently use vectors to store
the values of a function evaluated at many points. For example, we might set
>> x = linspace(-2, 2, 100);
>> y = 2*x + 3;
in order to plot a graph of the line y = 2x + 3 for −2 ≤ x ≤ 2. The linspace
command gives a vector of 100 linearly spaced points between -2 and 2. (There’s
also a command logspace that gives logarithmically spaced points.)
140
U. Washington AMATH 352 - Spring 2009
Note: if you leave off the semicolon at the end of one of the statements
above, matlab will print the result, a vector of 100 numbers!
The linspace command results in a row vector. If you want x to be a
column vector you will have to transpose it. For plotting purposes it doesn’t
matter whether x is a row vector or a column vector. The y defined above will
be either a row or column vector depending on which we have used for x. It
is possible to plot more than one curve with a single plot command, in which
case Y could be a matrix and each column of the matrix gives the points for one
curve. In this case it does matter how it is oriented. For example:
>> x = linspace(-2, 2, 100)’;
>> Y = [2*x + 3, sin(x)];
>> plot(x,Y)
will plot the functions 2x + 3 and sin(x) on a single plot. Note that x has been
defined as a column vector here. Figure 17 contains the resulting plot
Figure 17: Plot of y = 2x + 3 and y = sin(x) over [−2, 2]
If we want to plot the parabola y = x
2
and try to set
>> y = x*x; or >> y = x^2
an error message will result because the vector x cannot be multiplied by x
in the linear algebra sense. What we want is component-wise multiplication,
resulting in a new vector with 100 components containing values of x
2
at each
point. We can type
>> y = x.*x; or >> y = x.^2
to obtain this. Figure 18 contains the resulting plot
141
U. Washington AMATH 352 - Spring 2009
Figure 18: Plot of y = x
2
over [−2, 2]
The division operator / also has a component-wise version ./
As another example, performing the above command and then setting
>> z = 2*y + y./(5 + x);
will set the ith element of z to z
i
= 2y
i
+y
i
/(6+x
i
). Note that we can multiply
by the scalar 2 using * and each element of the vector is multiplied by 2. We
can also add the scalar 5 to a vector and each element has 5 added to it.
Operators like sqrt, cos, exp apply to each element of a vector, so setting
>> z = exp(x) .* cos(y);
sets z
i
= e
xi
cos(y
i
), provided that x and y have the same size (otherwise an
error message results).
The diary command is useful for making a record of everything you type
and the results that matlab computes. If you type
>> diary(’hw1.txt’)
... some other commands ...
>> diary off
then all the other commands and responses will be stored in the file hw1.txt.
This is useful for capturing your homework solutions. You can then edit this
file, print it out, and turn it in. (Note: you must say diary off before the file
can be viewed or printed. You can later say diary on to add more to it.)
Note that this file should be in the C:¸Temp directory. If you are viewing this
directory from within the matlab window, you may have to select “All Files”
in order to see this since by default matlab displays only m-files ending with
the .m extension.
Script files: m-files are useful for storing a sequence of commands to be
executed. If you have a file called hw1a.m, for example, then typing
>> hw1a
142
U. Washington AMATH 352 - Spring 2009
will cause this set of commands to be executed. If you are using the diary com-
mand to capture things, then you may want to type “echo on” before “hw1a”
so that the commands in hw1a are echoed along with matlab’s response. Oth-
erwise the diary file will only contain the responses, not the commands.
matlab has standard control structures such as “for loops” and “while loops”.
Some examples:
x = 4;
for i=1:6
x = 0.5*(x + 19/x);
end
approximates

19 using Newton’s method. The value of x is printed out each
time it is modified.
In this loop the vector 1:6 is used to indicate that the loop should be re-
peated 6 times. Any other vector can be used in the for loop,
for n = [1, 4, 9, 16, 25]
y(n) = sqrt(n);
end
will result in y = [1 2 3 4 5].
for n = 5:-1:1
y(n) = n^2;
end
will result in y = [25 16 9 4 1].
n = 5;
while n > 0
y(n) = n^2;
n = n - 1;
end
will also result in y = [25 16 9 4 1].
These sorts of loops can be put on a single line at the prompt by using
commas. The last loop could be executed by
>> n = 5; while n>0, y(n) = n^2; n = n-1; end
This is sometimes useful for doing something quick and dirty. However, it is
much better to put the loop in a m-file with one command on each line and
proper indentation so that it is readable.
If-then-else constructs can also be used,
143
U. Washington AMATH 352 - Spring 2009
x = rand(1); % a single random number between 0 and 1
if x < 1/3
y = -1;
elseif x < 2/3
y = 0;
else
y = 1;
end
will set y to −1, 0, or 1 with equal probability. Note that the word “then” does
not appear!
The expression x < 1/3 appearing above evaluates to either 0 (representing
False) or 1 (representing True). Logical expressions can be combined using &
for “and” and | for “or”,
x = rand(1); % a single random number between 0 and 1
if (x < 1/3) |(x == 0.5)
y = -1;
elseif x < 2/3
y = 0;
else
y = 1;
end
will set y to 2 if x < 1/3 or x == 0.5. In fact this can be done more simply
(but less clearly) by
y = 2*(x<1/3 | x==0.5);
Note that == is used to test for equality, not =, which is only used to assign
values to variables.
There are several ways to print out a value or string in matlab. We have
already seen one: compute a value or just type a variable name at the prompt
without a semicolon at the end and matlab will print out the value,
>> x = 1/3;
>> x
x =
3.333333333333333e-01
(By the way, format compact has been used to get matlab to leave out the
blank lines it would normally produce around x = ).
Often we want to print out something more descriptive or make a table of
values with numbers lined up nicely. The disp command prints out a variable
without printing the name of the variable first,
>> x = 1/3;
>> disp(x)
3.333333333333333e-01
144
U. Washington AMATH 352 - Spring 2009
disp can also be used to print a string of characters. In matlab, a string of
characters is specified by putting them in single quotes,
s = ’Hello world’;
disp(s)
or simply
disp(’Hello world’)
prints out this string.
You can build a string out of pieces by putting several strings together in an
array. Here’s a useful application of this:
>> s = [’The value of x is now ’, num2str(x)];
>> disp(s)
The value of x is now 0.33333
Note that num2str converts the number x into a string that can be catenated
together with the other string to form the string s.
A more general way to make a string that includes one or more numbers
when we want to control the spacing, number of digits printed out, etc., is to
use the sprintf command, which writes formatted data to a string. This follows
the C language formatting conventions if you are familiar with C. Type help
sprintf for more details. Here is an example:
>> x = 1/3; y = 2/3;
>> s = sprintf(’ We now have x = %20.15e and y = %10.5f’, x,y);
>> disp(s)
We now have x = 3.333333333333333e-01 and y = 0.66667
Here’s an m-file script that approximates the value of the square root of a
number given an initial guess. Note the use of the input command to read in
data from the prompt:
% newtonsqrt.m
a = input(’What number do you want to find the square root of? ’);
x = input(’What is your starting guess? ’);
disp(’ ’);
disp(’iteration approximation’);
maxiter = 10;
% maximum number of iterations to allow
for i=1:maxiter
x = 0.5*(x + a/x);
disp(sprintf(’%5i %20.15e’, i,x));
if abs(x^2 - a) < 1e-12
break % breaks out of the for loop if x is good enough
end
145
U. Washington AMATH 352 - Spring 2009
end
if i==maxiter
disp(’*** Warning: may not have converged -- tolerance not satisfied’);
end
disp(’For comparison,’);
disp(sprintf(’ sqrt(%g) = %20.15e’, a, sqrt(a)));
Here is some sample output:
>> newtonsqrt
What number do you want to find the square root of? 2
What is your starting guess? 5
iteration approximation
1 2.700000000000000e+00
2 1.720370370370370e+00
3 1.441455368177650e+00
4 1.414470981367771e+00
5 1.414213585796884e+00
6 1.414213562373095e+00
For comparison,
sqrt(2) = 1.414213562373095e+00
Note that this m-file is much fancier than the simple loop we used before.
The core computation x = 0.5*(x + a/x); is the same as before, but we have
added several features along with the tabular output:
• The program tests to see if x
2
is close to a each iteration, and if it is close
enough we break out of the loop (this is a “convergence test”).
• A maximum number of iterations (10 here) is specified so that we don’t
get into an infinite loop by accident.
• If the convergence test is never satisfied then a warning message is printed
out so the user knows that the final x may not be sufficiently accurate.
Paying attention to these sorts of details is important and similar issues will
come up with most of the numerical methods we study.
There is also a fprintf command that prints directly to a file. This is useful
if you expect a lot of output and you want it to be output to a file instead of
zipping past you on the screen. For example:
% fprintf_example.m
fid = fopen(’outputfile.txt’,’w’); % open a text file for the output
for k=1:1000
frpintf(fid, ’k = %4i\n’, k); % note: \n means "new line"
end
fclose(fid);
146
U. Washington AMATH 352 - Spring 2009
produces a file named outputfile.txt containing 1000 lines
k = 1
k = 2
k = 3
k = 4
etc.
There are many other useful commands in matlab. Here are just a few
examples:
• A = zeros(m,n) makes an mn array of all zeros.
• A = ones(m,n) makes an mn array of all ones.
• size(A) returns the dimensions of A, a row vector [m n].
• B = zeros(size(A)) makes an array of all zeros the same size as A.
• A = rand(m,n) makes an m n array of random values, uniformly dis-
tributed between 0 and 1.
• A = randn(m,n) makes an m n array of random values, normally dis-
tributed (with mean 0 and variance 1).
• A = eye(n) makes an n n identity matrix.
• d = diag(A) when A is an n n matrix, creates a vector containing oly
the diagonal elements of A
• A = diag(d) when d is a vector of length n, creates an n n diagonal
matrix with d(1), . . . , d(n) on the diagonal.
• sin, cos, tan, csc, sec, cot: Trigonometric functions
• asin, acos, atan, acsc, asec, acot: Inverse trigonometric functions
• ceil, floor, round, fix: rounding real numbers to integers
• max, min, mean, median, sum, prod: act on vectors and return a scalar.
147
U. Washington AMATH 352 - Spring 2009
C Using Functions in Matlab
There are two types of m-files in matlab (files that end with the .m extension).
One type is a script, which is just a list of commands that will be executed
when you type the name of the file (without the .m) into matlab. The other
type is a function, a file called myf.m, for example, which starts with a line like
function myf(x,y)
or
function f = myf(x,y)
or
function [f,g] = myf(x,y)
Each of these would take two inputs x and y (which might be single numbers,
vectors, or matrices) and would presumably do some operations on the inputs.
The first function would not return any values (though it might print something
or produce a plot), the second form would return a single value (and f would
have to be set somewhere in the body of the function). The third form would
return two values f and g (both of which should be set somewhere in the body).
Many simple mathematical functions can be represented by a single formula
in terms of elementary operations (addition, multiplication, etc.) or standard
functions such as sin(x), cos(x), exp(x),

x, etc. An example would be
f
1
(x) = 3x
4
+ xsin(2πx)/
_
1 + x
2
. (201)
There are several possible ways to represent a function in matlab that allow us
to evaluate the function at desired points. One approach is to create an m-file
that implements the rule defining the function. For example, the function (201)
can be implemented by creating a file f1.m containing the two lines
function y = f1(x)
y = 3*x.^4 + x.*sin(2*pi*x) ./ sqrt(1 + x.^2);
Then in matlab we can evaluate this in the obvious way:
>> z = f1(1)
z =
3
>> alpha = f1(-2)
alpha =
48
>> f1(.5)
ans =
0.1875
148
U. Washington AMATH 352 - Spring 2009
Note that we have implemented the formula for f
1
(x) in a vectorized manner
using the operators .*, .ˆ , ./ rather than simply *, ˆ , /. If we only plan
to evaluate f1 at a single point, as in the above examples, then it doesn’t matter
which we use. However, by using the vector-component operators it is possible
to compute f
1
(x) simultaneously for a whole vector of x values with a single
call to f1. For example, we can compute f
1
at the three values used above by
>> f1values = f1([1 -2 .5])
f1values =
3.0000 48.0000 0.1875
Another example:
>> f1([1 2 3; 4 5 6])
ans =
3 48 243
768 1875 3888
This evaluates f
1
at a 2 3 array of points. It’s a good idea to get in the habit
of always using the vector-component operators in function definitions as it is
often convenient to be able to evaluate a function at a whole set of points.
For example, suppose we want to plot the function f
1
(x) for −1 ≤ x ≤ 1
We can do this by evaluating f
1
at a large number of points in [−1, 1] and then
using the plot command in matlab:
>> x = linspace(-1, 1, 100);
>> y = f1(x);
>> plot(x,y)
After executing these commands, x and y are each vectors of length 100. The
command y = f1(x) would result in a matlab error if we had used non-
vectorized operators in the definition, since x
4
, for example, is not defined unless
x is a square matrix (and we don’t want the matrix power). Another way to
evaluate a function at a whole set of points is to use a loop,
>> x = linspace(-1, 1, 100);
>> y = zeros(size(x));
>> for i=1:length(x), y(i) = f1(x(i)); end
In this case x(i) is a scalar value and so f1 is only called with scalar arguments.
However, this is more cumbersome than simply setting y = f1(x) and is also
typically much slower since 100 function calls are made instead of 1. For this
small number the difference would be imperceivable, but for practical problems
we will often need to evaluate functions thousands or millions of times and using
vectorized form whenever possible can greatly reduce the computational time.
Note that in this last example we set y = zeros(size(x)) before starting
the loop that correctly evaluates y(i). This preallocates storage for y of the
correct size before looping through and resetting each component of the array.
149
U. Washington AMATH 352 - Spring 2009
This is not strictly necessary for this example: matlab will automatically in-
crease the size of the array y each time through the loop if necessary as we set
each value. However, it is often more efficient to preallocate the storage since
resizing y repeatedly is time consuming (and programs that do this will run
more slowly). Another reason for preallocating the storage is that it insures
that y will be the size we expect. If y has never been used before in this mat-
lab session then we’ll get what we expect without preallocating, but if y has
already been used for some other purpose then we it may be the wrong size. For
example, suppose y was previously used to store a vector of 500 values. Then
the commands
>> x = linspace(-1, 1, 100);
>> for i=1:length(x), y(i) = f1(x(i)); end
would reset the first 100 elements of y but would not change the remaining 400
elements. The command plot(x,y) would then produce an error message since
x and y are not the same size.
C.1 In-line function definitions
For functions that are implemented by a single matlab statement, such as f1
above, there is another way to specify the function rather than creating a 2-line
m-file as done above. It is often more convenient to define it directly at the
matlab command line as an in-line function, as follows:
>> f1a = @(x) 3*x.^4 + x.*sin(2*pi*x) ./ sqrt(1 + x.^2);
The syntax indicates that f1a is a function of a single variable that is denoted
by x in the formula that follows. Once f1a has been defined in this way, it can
be evaluated in the same manner as m-file functions:
>> f1a(.5)
ans =
0.1875
The definition of a function in this manner can use other variables,
>> a = 25;
>> g = @(x) a + x;
>> g(4)
ans =
29
>> a = 100;
>> g(4)
ans =
29
When g is defined a has the value 25 and this value is then hard-wired into the
function definition. The syntax g = @(x) ... indicates that the function is
150
U. Washington AMATH 352 - Spring 2009
being defined as a function of a single variable denoted by x in the following
string, and so a is assumed to be some value already set before executing this
function definition (an error message will result if a has not already been set).
Note that later changing a without redefining g will not change the function, it
as if we had defined
>> g = @(x) 25 + x;
Functions of more than one variable can be defined by specifying more than one
argument,
>> z = 20;
>> g = @(x,y) x.*y + z*sin(y);
>> g(2,pi/2)
ans =
2.314159265358979e+01
This implements the function g(x, y) = xy + 20 sin(y). Again note that the
notation defining g indicates that it is a function of the two variables x and y
in the formula following, and so z is assumed to be a previously defined value.
C.2 Passing function names into other functions
Sometimes we will want to define or use functions that act on other functions.
For example, the built-in matlab function quad computes numerical approx-
imations to definite integrals,
´
b
a
f(x) dx. (This sort of approximation, which
we’ll study later, is sometimes called quadrature, which explains the name of
the matlab function.) To apply quad, we have to give it the function f and
also the endpoints of the interval a and b. Suppose for example we want to
approximate
´
1
0
f
1
(x) dx, where f
1
(x) is the function specified in (201). Then
we pass in to quad an implementation of the function f
1
and the endpoints. We
have seen two different ways to implement f
1
above, either as an m-file or as an
in-line function. Either can be used, but the syntax is slightly different:
>> quad(’f1’,0,1)
ans =
4.864166380749058e-01
>> quad(f1a,0,1)
ans =
4.864166380749058e-01
If we use an m-file to define f
1
, then we pass the name of the m-file to quad
as a string, and hence it is surrounded by quotes. If we use the in-line version
then f1a is a variable in matlab (of type “function”) and so we just pass in this
variable (without quotes).
151
U. Washington AMATH 352 - Spring 2009
D Plotting Functions in Matlab
Manually, when we want to plot a curve, like y = sin(x) over [0, π], we start by
taking arbitrary values for x in [0, π] and computing the corresponding value
for y. We plot the resulting points (x, y) and connect the points by a straight
line, as in “connect the dots” drawing.
• In “connect the dots” drawing, the dots are ordered by their number. Here
the chosen order is increasing (or decreasing) values of x.
• We need to specify enough points in [0, π] so that the sequence of lines
connecting the points will give a graph that is visually indistinguishable
from the plot of f(x).
In Matlab, plotting a curve follows the same procedure. We define a vector x
containing many points in the interval of interest, evaluate y = f(x) as a vector
of the corresponding function values, and then execute plot(x,y). The plot
command produces a graph by connecting the points specified by straight lines,
as in a “connect the dots” drawing.
If f(x) is a smooth function and we specify a dense enough set of points, then
this sequence of straight lines will give a graph that is visually indistinguishable
from a plot of the function f(x). But it is important to understand what
matlab is doing because it is easy to produce graphs that do a very bad job of
illustrating the true function if we are not careful, particularly if the function f
is not very smooth or has discontinuities. We will explore this with a number
of examples.
Consider y = sin(x) over [0, π].
1. Create sampling values in [0, π] and store them in a column vector x.
>> x = []; m = 25;
>> for i = 1:(m+1),
x = [x; (i-1)*pi/m];
end
>> x
x =
0
0.1257
0.2513
0.3770
.....
2. Plot and connect the dots
>> plot(x,sin(x))
152
U. Washington AMATH 352 - Spring 2009
Figure 19: Figure for plot(x,sin(x)).
Consider now y = x
2
over [0, π].
>> x = [0; 0.5; 1.0; 1.5; 2.0; 2.5; 3.0; pi];
>> plot(x, x*x)
The command result in an error because the vector x can not be multiplied by
a vector in linear algebra sense. Instead, we want a component-wise multiplica-
tion, resulting in a vector with 8 components containing values of x
2
. To get a
component-wise version, we type
>> x = [0; 0.5; 1.0; 1.5; 2.0; 2.5; 3.0; pi];
>> plot(x, x.*x)
Figure 20: Figure for plot(x, x.*x).
The same modification applies for the division operator / or the power op-
erator ^.
153
U. Washington AMATH 352 - Spring 2009
Order of points. The order of points in the sampling vector x defines the
order when connecting the dots. Figure 21 presents two plots with two different
orderings in the vector x:
1. x =
_
¸
¸
_
0
3
1
2
_
¸
¸
_
2. x =
_
¸
¸
_
0
1
2
3
_
¸
¸
_
Figure 21: Plots with different order of points in x
Parametrized curve. Consider now the curve x
2
+ 4y
2
= 1. It can be
parametrized as follows
x = cos(t), y =
sin(t)
2
, ∀t ∈ [0, 2π].
The procedure to plot this curve with Matlab goes
1. Create a vector sampling [0, 2π].
>> t = linspace(0, 2*pi, 100)’;
2. Plot the curve
>> plot(cos(t), sin(t)/2)
154
U. Washington AMATH 352 - Spring 2009
Figure 22: Parametrized curve with default view (left) and with equal axes
(right).
Suppose we want to plot
f(x) = 2 sin(x) + cos(5x) (202)
for 0 ≤ x ≤ 10 and have defined a suitable m-file or inline function f that
evaluates this function. Figure 23 shows the result of the following commands:
>> x = linspace(0, 10, 1000);
>> plot(x,f(x))
This gives a reasonable graph of this function. Note that it consists of a rapidly
varying oscillation (the cos(5x) term) superimposed on a slower oscillation (the
2 sin(x) term). With 1000 points the curve plotted looks smooth and captures
the function well. If we increased the number of points, to 5000 say, the plot
would look essentially the same. However, if we reduce the number of points
and used only 30, then we would obtain the graph of Figure 23(b), plotted using
>> x = linspace(0, 10, 30);
>> plot(x,f(x))
This still captures the essential behavior of the function, but it does not look
so smooth — it’s now clear that linear segments are plotted between specified
points. If we reduce the number of points used further, to only 10, we obtain
the plot of Figure 23(c). This looks smoother but doesn’t look much like the
proper graph. The reason why is seen in Figure 23(d), where the particular
points chosen are shown as dots along with the plot of the proper function.
The collection of straight lines plotted by the plot command can be viewed
as the the correct graph of a piecewise linear function that approximates the
function f(x) that we really intended to plot. The idea of approximating a
complicated function by a much simpler function is one of the main themes
155
U. Washington AMATH 352 - Spring 2009
Figure 23: (a) Plot of function (202) using 1000 points. (b) Plot of function
(202) using 30 points. (c) Plot of function (202) using 10 points. (d) Illustration
of where these 10 points lie on the curve y = f(x).
156
U. Washington AMATH 352 - Spring 2009
of this course and we will see many applications of this idea. matlab uses
this approximation to plot a set of points since graphics primitives exist to
connect any two points by a straight line. We will use this idea in other contexts
because linear functions are much easier to work with than arbitrary functions
in many ways. For example, we can easily integrate a linear function and so if we
approximate a function by a piecewise linear approximation we can approximate
the integral.
As another example of potential pitfalls when plotting, consider the function
f(x) = x(3 −x) + exp(−100[x −1[). (203)
Suppose we plot this function over the interval [0, 3] using the commands
>> f = @(x) x.*(3-x) + exp(-100*abs(x-1));
>> x = linspace(0,3,30);
>> plot(x,f(x))
Then we obtain the plot shown in Figure 24(a). There’s a little bump in
the curve near x = 1 but it’s barely noticeable and if we don’t think about
it we might conclude that we have a pretty good plot of f(x) using only 30
points. However, note that f(1) = 2 + exp(0) = 3. This doesn’t agree with
what is shown in the plot at all! The function has an exponentially decaying
spike centered at x = 1. To capture this we must plot using many more points.
For example, Figure 24(b) shows a plot with 1000 points.
Figure 24: (a) Plot of function (203) using 30 points. (b) Plot of function (203)
using 1000 points.
For the function f(x) in (203) it is no big deal to evaluate the function
at 1000 points to produce this plot — it takes a small fraction of a second in
matlab. However, for other practical problems it may be quite expensive to
compute each function evaluation (for example when each “function evaluation”
requires going off and solving some other problem). If so then we might want to
give more thought to where we really need to evaluate f many times and where
a more coarse representation of f suffices. In the example we’ve just considered,
157
U. Washington AMATH 352 - Spring 2009
it is only near x = 1 that we need lots of values of f in order to capture the
behavior. Away from this point we only need a few values to get a good looking
plot of the quadratic behavior. We can produce a plot that looks just as good
as Figure 24(b) but using far fewer evaluations of the function f by using an
unequally spaced set of points, e.g.,
>> x = [linspace(0,0.9,10) linspace(0.9,1.1,200) linspace(1.1,3,20)];
>> plot(x,f(x))
The vector x used here has 10 points between 0 and 0.9, followed by 200 points
between 0.9 and 1.1, and finally 20 points between 1.1 and 3, for a total of 230
points rather than 1000.
This example serves as a warning that it is good to know something about
the function you are trying to plot in order to obtain a reasonable result. This
warning applies to many of the numerical methods we will consider as well,
even if you are simply using built-in matlab commands. Do not always blindly
trust what the computer tells you. For example, suppose we want to compute
´
3
0
f(x) dx for the function of (203). We might again try the matlab function
quad and we would obtain
>> quad(f,0,3)
ans =
4.500000072864403e+00
We might conclude that 4.5000 is an accurate approximation to the integral.
However, for this particular function we can integrate exactly and find that
ˆ
3
0
x(3 −x) + exp(−100[x −1[) dx =
_
3
2
x
2

1
3
x
3
_
3
0
+
ˆ
3
0
exp(−100[x −1[) dx
= 4.5 +
ˆ
1
0
exp(−100x) dx +
ˆ
2
0
exp(−100x) dx
= 4.5 + 0.01(1 −exp(−100)) + 0.01(1 −exp(−200))
≈ 4.520000000000000000 . . .
(204)
The value 4.5000... returned by quad has an error of about 0.02 which is
quite significant. It’s easy to see where this error comes from — quad is fooled in
much the same way we might be by Figure 24. It has entirely missed the sharp
peak near x = 1 and has produced a good approximation to
´
3
0
x(3 −x) dx.
Note that we can get a much better approximation using quad if we split
the integral up into an integral from 0 to 1 and one from 1 to 3. But splitting
the interval at the point where the spike lies, we make sure that quad notices
something is going on at this point:
>> quad(f,0,1) + quad(f,1,3)
ans =
4.520000793872945e+00
158
U. Washington AMATH 352 - Spring 2009
D.1 Some other useful plotting commands
The following commands are often useful. Use the matlab help facilities for
more information on plotting.
• axis([a b c d]) Set the region shown to a ≤ x ≤ b (horizontal axis) and
c ≤ y ≤ d (vertical axis).
• hold on Hold the current plot so that the next plot command will appear
together with the current plot. Useful for plotting several curves together
or points on top of curves. Turn this off with hold off or use the clf
(clear frame) command to return to a black frame.
• figure(2) Open a second plot window, or bring this one to the top.
159

U. Washington

AMATH 352 - Spring 2009

1

Column Vectors

In geometry, the two-dimensional plane is denoted R2 and the three-dimensional space R3 . A vector is an object comprised of a magnitude and a direction. In the plane, we can draw a vector as an arrow with some length and pointing somewhere . A vector can also be thought of as a displacement. A displacement

does not depend where it starts. Consequently, the vectors are equal, even

though they start from different places, because they have equal length and equal direction. The basic idea here, combining magnitude with direction, is the key to extending to higher dimensions. In this section, we define the generalization of vectors in the two-dimensional plane and three-dimensional space. Let m be a positive integer. We denote Rm the set of all real m-tuples, i.e. the set of all sequences with m components, each of which is a real number. The standard notation for an element x of Rm is the column vector notation:  x1  .  ∀x ∈ Rm ⇐⇒ x =  .  . . xm 

(1)

It is important to remember that, in many applications of linear algebra, the elements of the vector represent something different from the three physical coordinates of ordinary space. There is often nothing unphysical about considering vectors with many more than 3 components. Example. We have  √ −1 3 ∈ R2 ,   1 7  −2/5 3  0  ∈ R , and   −3/5 √ 3 4 2    ∈ R4 . 

U. Washington

AMATH 352 - Spring 2009

1.1

Addition of column vectors
vectors x and y of Rm :    x1 + y1 y1  .  , x+y =  . . .   . . . xm + ym ym

We can define the addition of two    x1   .  x= .  y= . xm

(2)

The set Rm is closed under addition, meaning that whenever the addition is applied to vectors in Rm , we obtain another vector in the same set Rm . For example, we have         1 2 1+2 3  2  +  −4  =  2 − 4  =  −2  . 3 8 3+8 11 Exercise 1. Prove that the addition is associative: (x + y) + z = x + (y + z). Exercise 2. Prove that the addition is commutative: x + y = y + x. Exercise 3. Find the unique vector z such that x + z = x, for any vector x. z is called the zero vector and, sometimes, denoted 0. Exercise 4. Show that every vector x ∈ Rm has an additive inverse y ∈ Rm such that x + y = 0.

1.2

Scalar multiplication of a column vector

We can also define the scalar multiplication: if x ∈ Rm and α ∈ R, then the vector αx belongs to Rm and is defined by multiplying each component of x by the scalar α:     αx1 x1  .   .  (3) x =  .  α ∈ R, αx =  .  . . . xm
m

αxm

The set R is also closed under scalar multiplication. For example, we have             1 2×1 2 2 (−3) × 2 −6 2  2  =  2 × 2  =  4  , (−3)  −4  =  (−3) × (−4)  =  12  . 3 2×3 6 8 (−3) × 8 −24 Exercise 5. Prove that ∀ α, β ∈ R, α(βx) = (αβ)x. Exercise 6. Prove that ∀α ∈ R, ∀ x, y ∈ Rm , α(x + y) = αx + αy. Exercise 7. Prove that ∀α, β ∈ R, ∀ x ∈ Rm , (α + β)x = αx + βx. Exercise 8. Show that, for every vector x ∈ Rm , 1x = x. 3

for any non-zero vector x. The triangle inequality can not be an equality because we have. |xi | ≤ x It is easy to verify that · • ∀x ∈ Rm . satisfying the following conditions (which generalize important properties of the absolute value for scalars): ∀x ∈ Rm . y ∈ R . x ≥ 0. x = 0 if and only if x = 0. then every component xi will be zero. One common choice is the max-norm (or infinity-norm) denoted by x ∞ = max |xi | . 0 = x − x < x + −x = 2 x . then all the components of x are zero and. x + y ≤ x + y m (4a) (4b) (4c) (Triangle inequality).Spring 2009 1. ∀α ∈ R. ∀ i. ∀x. we have |αxi | = |α| |xi | . satisfies the required properties (4). 1 ≤ i ≤ m. • x ∞ ∞ ∞ ∞ . consequently. x to zero. • ∀α ∈ R. we use a vector norm. – If x = 0. αx ∞ = |α| x ∞ . 1≤i≤m (5) A bound on the max-norm of the error is nice because we know that every component of the error can be no greater than the max-norm. αx = |α| x . This implies that the vector x is 0. – Note that.U. x ∞ = 0. 4 . A vector norm is simply a map from vectors in Rm to nonnegative real numbers. – If x ∞ = 0. (4d) Note that a norm satisfies the following properties −x = x and x + x = 2x = 2 x . Washington AMATH 352 . for every entry i.3 Norms To measure the magnitude or the length of a vector. ≥ 0 because the absolute value is always greater or equal = 0 if and only if x = 0.

The 2-norm is frequently used. y ∈ Rm . So every entry in the vector x + y is smaller than x ∞ + y particular. the largest entry will also satisfies that. ∞. we have       1 1 1 √  2   2  = 6. there are other norms which are either more appropriate or easier to bound using our analytical tools. For example. The 1-norm is defined as follows m x 1 = i=1 |xi | . ∞  −2  4  −8 5  = 14. 2 (6) The 2-norm is often called the Euclidean norm. 3 3 3 ∞ 1 2 and   −2  4  −8  = 8. We obtain αx ∞ = |α| x ∞ • ∀x. . m x 2 = i=1 |xi | . |xi + yi | ≤ |xi | + |yi | ≤ x ∞ + y ∞ . Consequently. however. x + y ∞ ≤ x ∞ + y ∞ – The absolute value satisfies a triangle inequality.Spring 2009 We get 1≤i≤m max |αxi | = max |α| |xi | = |α| max |xi | 1≤i≤m 1≤i≤m since we are multiplying every entry by the same factor |α|.U. defined by m 1 p x p = i=1 |xi | p . (7) The 1-norm is also known as the Manhattan norm because it corresponds to the distance traveled on a grid of city streets. = 3.  2  = 14. (8) Note that the max-norm can be obtained as the limit as p → +∞ of the p-norm. for every i. 1  −2  4  −8 = 2 √ 84. Washington AMATH 352 . These norms are special cases of the general family of p-norms. we have. So we get 1≤i≤m In max |xi + yi | = x + y ∞ ≤ x ∞ + y ∞ For some problems.

6 . S1 = {x ∈ Rm | x = 1}. 2π]. Several ways are possible to draw this curve.Spring 2009 Exercise 9. The closed unit ball {x ∈ Rm | x ≤ 1} is the set of all vectors with a norm smaller than 1. 0] and y ∈ [0. Check whether the following maps. y) such that x = 1.U. Prove that the p-norm satisfies the properties (4).. defined on R3 . For example. The equation governing the unit circle for the 1-norm is x y =1 1 ⇒ |x| + |y| = 1. 0] = 1 x. the unit circle contains the vectors 1 0 . 0 1 √ . 1] Figure 2 illustrates the unit circle for the 1-norm. The equation governing the unit circle for the Euclidian norm is x y =1 2 ⇒ x2 + y 2 = 1. y ≤ 1 = 1 x ∈ [0. are norms or not • x −→ x1 + x2 + x3 • x −→ |x1 + x2 + x3 | • x −→ x4 + x4 + x4 1 2 3 Exercise 10. we can parametrize x and y as follows x = cos(θ) y = sin(θ) where θ belongs to [0.. 1] and it is composed of the following four branches: x+y x−y −x − y −x + y = 1 0 ≤ x. √ 2/2 2/2 . The curve is included in the square [−1. 1] and y ∈ [−1. (9) In R2 . for the Euclidian norm. y There exists an infinite number of points on this curve. 1] × [−1. 0] = 1 x ∈ [−1. y ∈ [−1. The unit circle (or sphere) is the set of all vectors with a norm equal to 1. The shape of this ball depends on the norm. Among them. Figure 1 illustrates the unit circle for the Euclidian norm. we can draw the unit circle as a curve composed of the points (x. Washington AMATH 352 .. 1/2 √ − 3/2 .

a weighted 2-norm can be specified as follows m x ≤ 1} corresponding to x = i=1 wi |xi | 2 (10) where the weights wi are strictly positive real numbers. Numerical linear algebra. where each component of a vector is weighted. Figure 2: Unit circle for the Euclidian norm. Washington AMATH 352 .” Other useful norms include the weighted p-norms. For example. Bau.Spring 2009 Figure 1: Unit circle for the Euclidian norm. The Danis poet Piet Hein popularized this superellipse as a pleasing shape for objects such as conference tables. Trefethen and D. Draw the unit closed ball {x ∈ R2 | the 4-norm and ∞-norm. 7 . 1 L. Philadelphia. 1997. Exercise 11. N. Sweden. SIAM. Trefethen and Bau1 note that “The Sergel plaza in Stockholm.U. has the shape of the unit ball in the 4-norm.

Find vectors with equality. consider the vector  1 x =  −8  . Exercise 13. Show that the map Draw its closed unit ball. Then we have I−1 m 2 2 2 2 x ∞ = |xI | = |xI | ≤ i=1 |xi | ≤ + |xI | + i=I+1 2 2 |xi | = x . For a given non-zero vector.  .  =  .  = 1 and  . For example. Prove that x Exercise 15. in Rm . we have √ (11) x ∞ ≤ x 2 ≤ m x ∞. all the norms are related.  . However. Show that the map unit ball.   . ≤ x 1 1 ≤m x 2 ∞. 1 = 2 √ m. Find vectors with equality. Draw its closed x1 − √ 3x2 2 2 2 + √ 3x1 + x2 2 is a norm. The second inequality comes from the fact that any component is smaller than 8. its length depends on the norm chosen to measure it. . Washington AMATH 352 .Spring 2009 Exercise 12.  . To extend this proof to Rm .  .  .   .  = 1. 1 ∞ 0 0 ∞ 2 Exercise 14. In R3 . . 2  Then we have   1 √  −8  = 8 = 82 ≤ 2 ∞  1 12 + (−8)2 + 22 =  −8  2 ≤ 82 + 82 + 82 = √   1 3  −8  2 2  (12) ∞ where the first inequality is due to the fact that we add positive numbers to 82 . we assume that the max-norm is reached at the component I. 2 |xI | + · · · + |xI | = √ m x ∞ (13) Note that these inequalities are sharp because they are lowing vectors       1 1 1  0   0       . x ≤ x 8 ≤ x 1 . |x1 | + 3 |x2 | is a norm. Prove that 1 √ m ∞ attained with the fol-   1  .U.

1]. because of the Cauchy-Schwarz inequality (16). two different inner products can give two different angle θ. q ≤ ∞. (18) cos θ = x 2 y 2 Note that. For the same two column vectors.U.4 Inner product m The Euclidean inner product of two column vectors in Rm is defined as x·y = i=1 xi yi . in R2 . The proof extends to Rm . Inner products allow the rigorous introduction of intuitive geometrical notions such as the length of a vector or the angle between two vectors. the right hand side is always in the range [−1. The cosine of the angle θ between two column vectors is expressed in terms of the inner product (14) x·y . we write |x · y| ≤ x 2 y 2 . (14) It is a map into real numbers. For any vectors x and y of Rm . We will generalize its definition later in the notes. The Euclidean norm can be defined as the square root of the inner product of x with itself x = √ m 2 x·x= i=1 |xi | . Another useful inequality is the Hölder inequality |x · y| ≤ x p y q . It is one of the most important inequalities in mathematics. For example. (17) where p and q satisfy 1/p + 1/q = 1 and 1 ≤ p.Spring 2009 1. 2 (15) The inner product of two vectors is bounded by the product of their lengths. It also provides the means of defining orthogonality between vectors (zero inner product). (16) with equality if and only if y is proportional to x. Note that x · x ≥ 0. 1 2 Then we exploit the very useful result 2αβ ≤ α2 + β 2 to obtain 2 =⇒ 2 2 2x1 x2 y1 y2 ≤ x2 y2 + x2 y1 1 2 2 2 (x · y) ≤ (x2 + x2 )(y1 + y2 ) = x 1 2 2 2 y 2 2 . Washington AMATH 352 . we have 2 2 (x · y)2 = (x1 y1 + x2 y2 )2 = x2 y1 + x2 y2 + 2x1 x2 y1 y2 . 9 . The angle θ depends on the inner product chosen. This inequality is the Cauchy-Schwarz inequality.

Spring 2009 Another important concept in linear algebra is the idea of two vectors being orthogonal to one another.0344. 10 .1) defines the vector x ∈ Rm with components equal to 1. We say that x.5 Useful commands in Matlab Here are a few commands in Matlab useful for this section. i. Washington AMATH 352 . two vectors are orthogonal if and only if the lines drawn from the origin to the points with coordinates defined by x and y are perpendicular to one another. • z = zeros(m. −1 0 . defines the vector x ∈ R3 with components 1. • x = [1. Exercise 18. 1 √ 3 . Compute the angle between the following pairs of vectors 1 0 . their inner product is 0. the angle between the two vectors is θ = arccos 1 × 2 + 3 × (−2) √ √ 12 + 32 22 + 22 = arccos √ −4 √ 10 8 = arccos −1 √ 5 = 2.U. In R2 or R3 . • x = ones(m. Example 17. and 3.1) defines the zero vector in Rm . y ∈ Rm are orthogonal for the inner product (14) when x · y = 0. x · y = 1 × (−3) + 3 × 1 = −3 + 3 = 0. The vectors x= are orthogonal. −1 0 . Consider the vectors 1 3 and 2 −2 . For the Euclidian inner product. 1 1 .2. 1/4  1 3 y= −3 1 Indeed.e. which is a generalization of “perpendicular”. 1. approximately 116 degrees. 1 0 .3]. 2. we have x · y = 1 × 4 + 3 × 1 + (−2) × 4 + 4× = 4 + 3 − 8 + 1 = 0. Example 16. So are the vectors  1  3   x=  −2  4   4  1   y=  4 .

norm(x. for i=1:1000. end 11 .1) makes a vector of length m with random values.0. norm(x.0/1000. (i-1)*5. Washington AMATH 352 .1). • x = rand(m. it is recommended to declare x with its final size as follows x = zeros(1000.3]) computes the maximum entry in the vector (here 3). respectively. the ∞-norm. end Whenever possible. normally distributed (with mean 0 and variance 1). for i=1:1000. • norm(x) computes the 2-norm of vector x.0/1000.U.1) makes a vector of length m with random values.1). • x = randn(m. the 1-norm.0].-4.inf). • max([1. uniformly distributed between 0 and 1. • The next sequence generates a vector x with 1000 components linearly distributed between 0 and 5 x = [].3]) computes the minimum entry in the vector (here -4). • dot(x. and the pnorm.p) computes.Spring 2009 • size(x) returns the dimensions of x in a row vector [m 1]. • min([1.y) computes the Euclidian inner product between vectors x and y.-4. x = [x. norm(x. x(i) = (i-1)*5.

If u. and the sum x + y is another vector in Rm . This just means that whenever these operations are applied to vectors in Rm we obtain another vector in the same set Rm . 4.e. If u. If v ∈ V and α.   . 12 . 7. v ∈ V and α ∈ R then α · (u + v) = α · u + α · v. i. We have also defined scalar multiplication: if x ∈ Rm and α ∈ R then the vector αx ∈ Rm is defined by multiplying each component of x by the scalar α. It is also possible to define a complex linear space or C-linear space where the scalars are now complex. . 0 0 5.Spring 2009 2 Linear Spaces Recall that Rm denotes the set of all real m-vectors. consists of a set of objects V along with two operations ‘+’ (addition) and ‘·’ (scalar multiplication) subject to these conditions: 1. each of which is a real number. v. . If u. or R-linear space. If v ∈ V then 1 · v = v. There is a zero vector ¯ ∈ V such that v + ¯ = v for every v ∈ V . x= . 9. w ∈ V then (u + v) + w = u + (v + w) (addition is associative). 6. β ∈ R then (αβ) · v = α · (β · v). β ∈ R then (α + β) · v = α · v + β · v.U. y ∈ Rm : we obtain the sum by adding each component. v ∈ V then u + v ∈ V (closed under addition). Verify the properties of Definition 19 for the set of column vectors Rm . The set Rm is an example of linear space. 2. Washington AMATH 352 .  y = . the set of all vectors with m components. We have also defined what we mean by addition of two vectors x. 8. If v ∈ V and α. 1. A real linear space.  ⇒x+y = . .    . xm ym xm + ym So the vector x + y is also a column vector with m rows and belongs to Rm . Every v ∈ V has an additive inverse w ∈ V such that v + w = ¯ 0. Example 20. For any vector x and any vector y in Rm . If u. we have       x1 y1 x1 + y1  . The set Rm is closed under addition and scalar multiplication. . 3. v ∈ V then u + v = v + u (addition is commutative). which can be defined more generally as follows: Definition 19. If v ∈ V and α ∈ R then α · v ∈ V (closed under scalar multiplication). 10.

 x =  . + . we have    x1 + (y1 + z1 )    . .  . ym + xm  and the m components are equal because the addition of scalar numbers is commutative. Indeed. .   . is associative. we have     x1 αx1  . the vectors x + y and y + x are     y1 + x1   . . the vector y. For any vector x and any vector equal.  ⇒ αx =  .   . .Spring 2009 2. Washington AMATH 352 . . (xm + ym ) + zm xm + (ym + zm ) The m components are equal because the addition of scalar numbers is associative. For any vector x in Rm and any scalar α in R. 3.   . −xm xm is such that x + y = 0. For any vector x in Rm . The vector y is the additive inverse of x.  x =  . . y+x=  .  . (α + β)xm αxm + βxm αxm βxm 13 . The addition of column vectors  (x1 + y1 ) + z1  . The zero vector is the vector with all its m components equal to 0.  = αx+βx. we have         (α + β)x1 αx1 + βx1 αx1 βx1      .   0  . . (α+β)x =  =  =  . . For any vector x in Rm and any scalar α and β in R. 0 5. . . . . we have  x1 + y1  . xm + ym y in Rm . 7. . 4. .  x + (y + z) =  . Indeed. defined by     −x1 x1  .  y =  . (x + y) + z =  . xm αxm So the vector αx is also a column vector with m rows and belongs to Rm .  0 =  . 6. . x+y = . . . .U. .

R). f : R −→ R.Spring 2009 8. then f + g is the function defined by (f + g)(x) = 3x2 + cos(x). if f (x) = 3x2 and g(x) = cos(x). R). we have      αx1 + αy1 α(x1 + y1 )      . R). R). we have       x1 1 × x1 x1  .      . For any vector x in Rm .  = αx+αy.  = x. we have ∀x ∈ R. . ∀x ∈ R. 2.U.  = α(βx). R) is a real linear space. 1·x=1· . 1. 3. f + g belongs to F(R. For any function f and g in F(R. in Rm and any scalar α in R. Indeed. Example 21. [(f + g) + h](x) = (f + g)(x) + h(x) = f (x) + g(x) + h(x) = f (x) + (g + h)(x) = [f + (g + h)](x). . αxm + αym α(xm + ym ) xm + ym 9. (αβ)x =   = α  . For any vector x and y  x1 + y1  . is denoted F(R. . Indeed. . The set of all possible functions mapping real numbers to real numbers.     . α(x+y) = α  .  . . xm 1 × xm xm So Rm is a real linear space. the functions f + g and g + f are equal. We will verifty that F(R. . We used the fact that the addition of scalar numbers is associative. βxm α(βxm ) (αβ)xm 10. we have       βx1 α(βx1 ) (αβ)x1  . For any vector x in Rm and any scalar α and β in R. For any function f and g in F(R. = . we have (f + g)(x) = f (x) + g(x). we have ∀x ∈ R. . Washington AMATH 352 . (f + g)(x) = f (x) + g(x) = g(x) + f (x) = (g + f )(x). . The addition of functions is associative. . So the function f + g is a function whose input is a real number and its output is also a real number. 14 . . . . = = . =  =  . We used the fact that the addition of scalar numbers is commutative. . . For instance.

U. Washington

AMATH 352 - Spring 2009

4. The zero function ¯ is the function identically 0 in R, 0 ¯ 0(x) = 0, ∀x ∈ R. (19)

5. The additive inverse of a function f is the continuous function f (−) defined by f (−) (x) = −f (x), ∀x ∈ R. (20) Indeed, we have f (x) + f (−) (x) = f (x) − f (x) = 0 for every x in R. 6. For any function f in F(R, R) and any scalar α in R, we have ∀x ∈ R, (αf )(x) = αf (x). So the function αf is a function whose input is a real number and its output is also a real number. αf belongs to F(R, R). For example, if g(x) = cos(x), 5g is the function defined by (5g)(x) = 5 cos(x), ∀x ∈ R.

7. For any function f in F(R, R) and any scalar α and β in R, we have ∀x ∈ R, [(α+β)f ](x) = (α+β)f (x) = αf (x)+βf (x) = (αf )(x)+(βf )(x). So the functions (α + β)f and αf + βf are equal. 8. For any function f and g in F(R, R) and any scalar α in R, we have ∀x ∈ R, [α(f + g)](x) = αf (x) + αg(x) = (αf )(x) + (αg)(x). So the functions α(f + g) and αf + αg are equal. 9. For any function f in F(R, R) and any scalar α and β in R, we have ∀x ∈ R, [(αβ)f ](x) = (αβ)f (x) = αβf (x) = α[βf (x)] = α(βf )(x). So the functions (αβ)f and α(βf ) are equal. 10. For any function f in F(R, R), we have ∀x ∈ R, (1 · f )(x) = 1 × f (x) = f (x). Example 22. The set of all continuous functions mapping real numbers to real numbers, f : R −→ R, is denoted C 0 (R, R). C 0 (R, R) is a real linear space. We emphasize that the zero vector takes different meanings according to the linear space V . When V = Rm , the zero vector is   0  0  ¯ =  . , 0  (21)   .  . 0 15

U. Washington

AMATH 352 - Spring 2009

the vector with m zero components. When V = C 0 (R, R), the zero vector ¯ is 0 the function identically 0 in R, ¯ 0(x) = 0, ∀x ∈ R. (22)

We will mostly study linear algebra in the context of linear spaces of vectors, but the study of other linear spaces, particularly function spaces, is extremely important in many branches of mathematics and many of the ideas introduced here carry over to other linear spaces. Exercise 23. Verify that Rm is a Q-linear space, where Q is the set of signed rational numbers. Exercise 24. Verify that C is a real linear space. Exercise 25. Verify that C is a complex linear space. Exercise 26. Show that the set Pn of all polynomials with real coefficients of degree n is a real linear space. What is the zero vector?

2.1

Subsets and subspaces

Suppose we have a linear space V and S is a subset of V , which just means every element of S is also an element of V , v ∈ S =⇒ v ∈ V. A subset might contain a finite or infinite number of elements. Example 27. The subset, S1 = 1 2 , −2.3 7 , π 1 , (24) (23)

is a subset of R2 with 3 elements. Example 28. The subset, S2 = x ∈ R3 : x2 = x2 + 3x3 , 1 is a subset of R3 with an infinite number of elements, including       0 −2 1  0 , 1 , 4 . 0 −1 1 Example 29. The subset, S3 = x ∈ R2 : x2 = 3x1 , is a subset of R2 with an infinite number of elements, including 0 0 , −2.3 −6.9 16 , π 3π . (26) (25)

U. Washington

AMATH 352 - Spring 2009

Definition 30. If S is a subset of a linear space V and S is closed under addition and scalar multiplication, then we say that S is a subspace of V . Consider the previous subsets. • The subset S1 is not a subspace of R2 because adding two vectors from S1 does not give a vector in S1 . • The subset S2 is not a subspace of R3    −2 2 1  =  −1 • The subset S3 is a subspace of R2 . Example 31. C 0 (R, R) denotes the set of functions f , f : R −→ R, that are continuous. For instance, f (x) = 3x2 + cos(x) and g(x) = |x| belong to C 0 (R, R). C 0 (R, R) is a subspace of F(R, R). Indeed, it is a subset of F(R, R), which is a real linear space. For any function f and g in C 0 (R, R), the sum function f + g is continuous. So the sum function f + g belongs to C 0 (R, R). C 0 (R, R) is closed for the addition. For any function f in C 0 (R, R) and any scalar α in R, the function αf is continuous. So the function αf belongs to C 0 (R, R). C 0 (R, R) is closed for the scalar multiplication. Consequently, C 0 (R, R) is a subspace of F(R, R). It is also a real linear space. Example 32. C 1 (R, R) denotes the set of functions f , f : R −→ R, that are continuous and differentiable and whose derivative f is also continuous. For instance, f (x) = 3x2 + cos(x) is in C 1 (R, R). On the other hand, g(x) = |x| belongs to C 0 (R, R) but not to C 1 (R, R). C 1 (R, R) is a subspace of F(R, R). Indeed, it is a subset of F(R, R), which is a real linear space. For any function f and g in C 1 (R, R), the sum function f + g is continuous. It is also differentiable and its derivative, equal to f + g , is also continuous. So the sum function f + g belongs to C 1 (R, R). C 1 (R, R) is closed for the addition. For any function f in C 1 (R, R) and any scalar α in R, the function αf is continuous. It is also differentiable and its derivative, equal to αf , is also continuous. So the function αf belongs to C 1 (R, R). C 1 (R, R) is closed for the scalar multiplication. Consequently, C 1 (R, R) is a subspace of F(R, R). It is also a real linear space. Example 33. C p (R, R) denotes the set of functions f , f : R −→ R, that are continuous and differentiable p times and whose p-th derivative f (p) is also continuous. C p (R, R) is a subspace of C 0 (R, R). 17 because  −4 2  ∈ S2 . / −2

αu ∈ S To prove that S is a real linear space. Washington AMATH 352 . R) as superset. So we just need to check the properties of being closed.U. which is a real linear space. Example 35. We need to find a superset. Use Definition 19 and check all 10 items. we will use the second approach. R). we can rewrite this as (α1 − α2 )v = ¯ But this can be true 0. The definition of S suggests C 0 (R. Find a superset U . Show that S is a subspace of U by checking the properties of being closed for the addition and the multiplication (only 2 checks): (a) ∀u. every function in S is a continuous function. To check whether f + g belongs to S. Two approaches are possible: 1. If α1 v = α2 v then by the rules of Definition 19. 18 . Why? Because if v ∈ S then αv ∈ S for any real number α (of which there are infinitely many). • The set Z = {¯ is the only subspace of V that contains a finite number of 0} elements (just 1 element). • The set Z = {¯ that contains only the zero element of the linear space V 0} is a subspace of V since ¯ + ¯ = ¯ and α · ¯ = ¯ so this set is closed under 0 0 0 0 0 these operations. R) | f (0) = 0 . ∀u ∈ S. Indeed. We would like to check that S is a real linear space. Note the following about subspaces: • The set S = V is a subspace of V (R2 is a subspace of R2 ). only if v = ¯ or if α1 − α2 = 0. we need also to compute the value of f + g at 0: (f + g)(0) = f (0) + g(0) = 0 + 0 = 0. All other subspaces contain an infinite number of elements. The sum f + g is a continuous function because f and g are continuous. Show that the set Pn of all polynomials with real coefficients of degree n is a subspace of C 0 (R. such that S ⊂ U . Let f and g be two functions in S. Show that U is a subspace of U. 2. 2. v ∈ S. 0 The fact that a subspace is also a real linear space suggests a technique to prove that a set U is a real linear space: 1.Spring 2009 Exercise 34. R) is a real linear space. If S is a subspace of V that is not all of V then it is called a proper subspace of V . u + v ∈ S (b) ∀α ∈ R. • A subspace S of a real linear space V is also a real linear space. We know that C 0 (R. Consider the set S = f ∈ C 0 (R. Find a real linear space U such that U ⊂ U.

The vectors x= 1 . 19 . Washington AMATH 352 . Two vectors are said to be linearly independent when there is no nontrivial linear combination of u and v that gives ¯ In other words. independent if the equation αu + βv = ¯ 0 (27) has only the trivial solution α = β = 0. we need also to compute the value of αf at 0: (αf )(0) = αf (0) = α0 = 0. 2α 0 The second component is zero only if α = 0.2 Linear dependence and independence If x. they are linearly 0. are scalars α. Let f be a functions in S and α ∈ R.e. two vectors u. 2.Spring 2009 So f + g belongs to S. which is zero only if β = 0. In general. So αf belongs to S. We conclude that S is a subspace of C 0 (R. for example if v = γu for some scalar γ. 2 y= 3 0 are linearly independent since neither one is a scalar multiple of the other. β ∈ R that are not both equal to zero but for which αu + βv = ¯ 0. To check whether αf belongs to S. Another way to see this is that the equation αu + βv = ¯ is 0 α + 3β 0 = . two vectors in V are linearly dependent if (and only if) one is a scalar multiple of the other. y ∈ Rm are any two vectors and α = β = 0. R) and it is also a real linear space. So (27) is satisfied only when α = β = 0. The product αf is a continuous function because f is continuous. Example 36.U. Two vectors are linearly independent when one is not a scalar multiple of the other. then αx + βy = 0x + 0y = ¯ + ¯ = ¯ 0 0 0. if there 0. Since then γu − v = ¯ (or any other linear combination with β = 0 and α = −γβ gives the 0 zero vector). In a real linear space V . v ∈ V are said to be linearly dependent if there is some nontrivial linear combination of u and v that gives ¯ i. But then the first component becomes 3β. So this trivial linear combination of x and y is always the zero vector.

q(x) = x. i.   7 x(1) = 0 . Example 39. However. 0 z= 1 1 are linearly dependent. 5 . · · · . For example. Then any nonzero value of x implies that β = 0. 0 Then we have α + 3β + γ 2α + γ = 0 0 . it is said to be linearly dependent. The polynomials p and q. ∀x ∈ R. if every nontrivial linear combination of the vectors is nonzero. When the set is not linearly independent.e. and γ = −6 results in the zero vector. u(2) . 0 x(2)   1 = 2 .Spring 2009 Example 37. Identifying each entry results in the following system of equations α + 3β + γ 2α + γ = = 0 0 These equations imply that γ = −2α and β = α/3. The set of r vectors u(1) . The vectors.U. u(r) ∈ V is linearly independent if the equation α1 u(1) + α2 u(2) + · · · + αr u(r) = ¯ 0 (28) has only the trivial solution α1 = α2 = · · · = αr = 0. the equation αp + βq = ¯ becomes 0 α + βx = 0. ∀x ∈ R. So (27) is satisfied only when α = β = 0. a linear combination with α =3. The idea of linear dependence and independence can be extended to sets of more than 2 vectors. The vectors x= 1 . Taking x = 0 gives that α must be 0. are linearly independent. α remains arbitrary. 2 y= 3 . Washington AMATH 352 . Indeed. β = 1. 0 20 x(3)   3 = 4 . Definition 38. Example 40. Consider a linear combination resulting in the zero vector 0 αx + βy + γz = . This non-trivial combination implies that the 3 vectors are linearly dependent. defined by p(x) = 1.

Exercise 41. x(2) . consider the linear combination α + βx + γx2 = 0.3 Exercise 42. x(2) = −4. x(2) =  1  . and x2 are linearly independent. x(3) = 0 . we have also βex − γe−x = 0. Check the linear dependency of the vectors       1 −2 1 x(1) = 2 . A linear combination of x(1) . If a linear combination of x(1) .1 x(1) = 2 . Let us prove this result. ∀x ∈ R (29) Taking x = 0 implies that α must be 0. x. Check the linear dependency of the vectors     1 −2. ∀x ∈ R. we obtain that α2 is also 0. If α1 = α2 = α3 = 0. ∀x ∈ R. and e−x . We will check the linear dependency of the functions 1. and x(3) is zero if and only if α1 = α2 = α3 = 0. The functions 1. 0 1. Taking a second derivative. we get βex + γe−x = 0. and x(3) is zero. then we have     7α1 + α2 + 3α3 0  =  0 . . Then taking x = 1 and x = −1 gives β+γ −β + γ = = 0 0 which implies that β = γ = 0.2 . Consider a linear combination that is equal to the zero function: α + βex + γe−x = 0. Differentiating this relationship. Plugging α = β = 0 in any equation gives that γ = 0. So the linear combination (29) is zero if and only if α = β = γ = 0. Then the first component implies that α1 is 0. x(2) . Example 44. α1 x(1) + α2 x(2) + α3 x(3) = 0 if and only if α1 = α2 = α3 = 0. we get that α3 has to be 0. 2α2 + 4α3 α1 x(1) + α2 x(2) + α3 x(3) =  5α3 0 By matching the third component. Summing the two equations results in β = 0. 21 ∀x ∈ R. Indeed.Spring 2009 are linearly independent.U. Plugging this value into the second component.5 3 Example 43. ex . 3 −6. Washington AMATH 352 . Indeeed. then α1 x(1) + α2 x(2) + α3 x(3) = 0.

Washington AMATH 352 . x2 0 1 Example 48. . the functions 1. 0 1 1 3 . Plugging this value into the equation after one differentiation. Consequently. Check the linear dependency of the functions 1. ∀x ∈ R. Since any linear combination of vectors in this set is again a linear combination of u(1) . The space S = span 1 0 2 . since any vector x ∈ R2 can be written as a linear combination of these three vectors: x1 1 0 2 = x1 + x2 +0 . Then the span of this set of vectors is the space of all linear combinations of these vectors. x2 0 1 3 22 . 1]. u(r) ∈ V be a set of r vectors. and sin(πx) on [−1. 2. ∀x ∈ R and γ = 0.Spring 2009 Summing the last two equations. which implies that β = 0. · · · . α1 . is all of R2 . span(u(1) . since any vector x ∈ R2 can be written as a linear combination of these two vectors: x1 1 0 = x1 + x2 .3 Span of a set of vectors Let V denote a real linear space and u(1) . · · · . · · · . given by (26). u(r) ) = α1 u(1) + · · · + αr u(r) . (30) This is a subspace of V . αr ∈ R . Finally. can be written as S3 = span Example 47. we have −γe−x = 0. u(r) ∈ V . · · · . 0 1 3 is all of R2 . and e−x are linearly independent. The space S = span 1 0 . Exercise 45. cos(πx).U. the first relation gives that α = 0. ex . Example 46. we obtain 2βex = 0. The subspace S3 .

· · · . in this case. 2. α1 . we require at least one vector to define this particular space. ··· . then we can find a basis consisting of fewer vectors. u(r) ) = α1 u(1) + · · · + αr u(r) . What do we mean by minimal? The subspace S3 . x2 0 1 3 Example 49. note that the last two vectors are linearly dependent. The set P2 of real polynomials of degree at most 2 is a real linear space. Washington AMATH 352 . then the set of vectors u(1) . u (1) (32a) (32b) . So we have P2 = span 1. αr ∈ R . We say that either one of these vectors alone is a basis for this particular space. Any polynomial in P2 is written as p(x) = α + βx + γx2 . It is a linear combination of functions 1. x2 . u(r) ). given by (26). x. Clearly. u(r) ∈ V form a basis for S if S = span(u(1) . x. If a set of vectors spanning the space are not linearly independent. (31) defines a subspace of V . u(r) ∈ V be a set of r vectors. · · · . 23 . there are infinitely many different ways to write an arbitrary vector x ∈ R2 as a linear combination of these three vectors.Spring 2009 Actually. we could write it as x1 1 0 2 = −x1 + (x2 − 3x1 ) + 2x1 . 2 6 . Then the span of this set of vectors. In fact it can be shown that any subspace of Rm has this form — it is the span of some set of vectors. · · · . u (r) are linearly independent. In the latter case. but specifying two vectors is redundant. A minimal set of vectors that define a space is called a basis for the space.4 Basis vectors Consider V a real linear space and u(1) . If S is a subspace of a linear space V . span(u(1) . can be written as S3 = span 1 3 = span 2 6 = span 1 3 . and x2 . · · · . · · · . More generally we make the following definition to make this idea precise: Definition 50. For example.U.

0 1 form a basis of R2 . 1 (33) There are infinitely many other bases: any set of m linearly independent vectors will do. . We say it is a 0-dimensional space. and x2 . Example 54. The space S3 .   1 0   0  . Example 58. 2. x2 ) form a basis for P2 . . . 0   0 1   0  . These functions are linearly independent. Indeed. The space S0 = {¯ consisting only of the zero vector. ··· . What is the dimension of Pn the real linear space of polynomials with real coefficients of degree less or equal to n? 24 .5 Dimension of a space There are typically infinitely many choices of basis vectors for a given space. Example 57. The number of basis vectors needed to define a space is called the dimension of the space. . . Example 56. The set of complex values C is a real linear space of dimension 2. The space P2 . Definition 53. . . Example 55. x2 ) form a basis for P2 . x. consisting of polynomials of degree at most 2.U. . special case. Exercise 59. . x. x. . It is also a complex linear space of dimension 1. 0   0 0   1  . The single vector 1 3 is one possible basis for this space. is a 1-dimensional subspace of R2 . Example 52. it can be shown that all choices of basis vectors consist of the same number of vectors. . The “standard basis” consists of the “unit vectors”. Washington AMATH 352 . The vector space Rm has dimension m. 0   0 0   0  . is a 0}. Consequently. This is the only linear space that consists of a single vector and the only linear space that has no basis. (1. Indeed they span R2 and they are linearly independent. However. The space P2 is spanned by the functions 1.Spring 2009 Example 51. (1. The vectors 1 0 . is of dimension 3. . given by (26).

verify that it is a subspace of a bigger real linear space. Then we have dimS ≤ dimV (34) where dim denotes the dimension of a real linear space. we apply these steps to the different examples. 4. all the bases are composed of the same number of elements. Write a general formula characterizing any element of S (an element can be a function. For any vector x in S and any scalar α in R.. Write a new formula characterizing any element of S as a linear combination of elements where each parameter is multiplying one element. which is a real linear space. For each space. look for a subset of these elements that still spans S and that contains linearly independent elements.Spring 2009 Exercise 60. For example. these elements form a basis for S. 1. a systematic approach to find a basis is defined in the following steps. 2. Next. Indeed it is a subset of R3 . 0 0 25 . For any vectors x and y in S. We will look for the dimension and a basis for a couple of spaces. a vector. So x + y belongs to S and S is closed for the addition. If they are linearly independent. Check whether these particular elements are linearly independent. Proposition 61. is equal to 0. If not. Consider the set S = x ∈ R3 | x3 = 0 . Example 62. Identify the independent parameters in the formula. This formula should indicate that these elements span the space S. What is the dimension of the real linear space R? Exhibit a basis. 5. S is a real linear space.). 3. we have       x1 y1 x1 + y1 x =  x2  y =  y2  ⇒ x + y =  x2 + y2  . Let V be a real linear space and S ⊂ V be a subspace. we have     x1 αx1 x =  x2  ⇒ αx =  αx2  . Washington AMATH 352 .. . S is composed of vectors with 3 rows. x3 . Note that an infinite number of bases exist. Check that the space S is indeed a real linear space. What is the dimension of R as a Q-linear space? Finally. It is also closed for the scalar multiplication. S is closed for the addition. 0 0 0 The vector x + y is a vector with 3 rows and its last component is equal to 0. we have the following important result comparing dimensions between a real linear space and one of its subspaces. whose last component.U. However.

Denote S the set to study.U. Consider any function f in S and any real number α. To find a basis for S and the dimension. So they form a basis for S. R). 0 0 To obtain a basis. then the function f + g is a function in C 1 (R. 26 . R) is a real linear space. then the function αf is a function in C 1 (R. This will prove also that S is a real linear space. we obtain that α = β = 0. we need to check whether S is closed for the multiplication. So we need to check whether S is closed for the addition and closed for the scalar multiplication. The dimension of S is 2. R). Washington AMATH 352 . 0 This general formula has two parameters x1 and x2 . So we can write       x1 1 0 x ∈ S ⇐⇒ x =  x2  = x1  0  + x2  1  . So S is a subspace of R3 and a real linear space. We notice that S is a subset of C 1 (R. we have to compute the derivative to see if it is equal to f + g: (f + g) (t) = f (t) + g (t) = f (t) + g(t) = (f (t) + g(t)) = (f + g)(t) So f + g belongs to S. Determine a basis and the dimension. So we will show that S is a subspace of C 1 (R. which is a real linear space. So αx belongs to S and S is closed for the scalar multiplication. we write a general formula for any vector in S   x1 x ∈ S ⇐⇒ x =  x2  . we need to check whether these two vectors are linearly independent. The vectors are linearly independent and they span S. R) because C 1 (R. To check whether f + g belongs to S. Consider a linear combination equal to the vector 0           0 α 0 0 1 α 0  + β  1  =  0  ⇒  β  =  0 .Spring 2009 The vector αx is a vector with 3 rows and its last component is equal to 0. Check whether the set f ∈ C 1 (R. Example 63. 0 0 0 0 0 Identifying the entries. Finally. R) is a real linear space. 0 0 0     0 1 So any vector in S is a linear combination of the two vectors  0  and  1 . 0 0 which implies that     0 1 S = span  0  . Consider any function f and g in S.  1  . R) | f (x) = f (x) is a real linear space. R) because C 1 (R.

Then we get for β = −γ − δ = 3δ − δ = 2δ. We can write a general formula for any polynomial in S. S is composed of polynomials of degree at most 3 that take the value 0 when evaluated at x = 0. 27 . So αf belongs to S. α + βx + γx2 + δx3 (x = 0) α (x = 1) β+γ+δ (x = 2) 2β + 4γ + 8δ = 0 = 0 = 0 = 0 Substracting two times the next-to-last equation from the last equation. A non-zero function is necessarily linearly independent. We conclude that S is a subspace of C 1 (R. S is a real linear space. for any α ∈ R. So any polynomial in S has the form p(x) = δ(2x − 3x2 + x3 ). Washington AMATH 352 . So (et ) is a basis for S. It is also closed for the multiplication because the values at 0 of αp is equal to αp(0) = 0 (the same thing holds for x = 1 and x = 2). which is non-zero. p + q is equal to zero because p(0) = q(0) = 0 (or p(1) = q(1) = 0 or p(2) = q(2) = 0). When evaluated at 0 (or 1 or 2). R) and it is also a real linear space.Spring 2009 To check whether αf belongs to S. and x = 2. So S is a real linear space. So we have S = span(et ). Example 64. we have to compute the derivative to see if it is equal to αf : (αf ) (t) = αf (t) = α(f (t)) = αf (t) = (αf )(t). So S is closed for the addition. The polynomial 2x − 3x2 + x3 forms a basis. The space S is spanned by the polynomial 2x − 3x2 + x3 . Any solution is proportional to et .U. Any function in S is solution to the equation f (t) = f (t). Indeed it is a subset of P3 . we get 2β + 4γ + 8δ − (2β + 2γ + 2δ) = 2γ + 6δ = 0 =⇒ γ = −3δ. A general solution is f (t) = αet . which is a real linear space. x = 1. S is closed for the addition because. Consider the set S = {p ∈ P3 | p(0) = p(1) = p(2) = 0}. for any polynomials p and q in S. No other solution exists. The dimension of S is 1. p+q is a polynomial in P3 . The dimension of S is 1.

u(2) ∈ U. for any fixed point x0 the function f (s) = g(x0 + s) − g(x0 ) is a linear function of s. and this is what is meant by an affine function more generally: a linear function shifted a constant (in this case −3). The function g(x) is properly called an affine function although people are often imprecise and call it linear.1 Linear functions from R to R The conditions (36) are very restrictive. 3. It really consists of a linear function 6x plus a translation (shifting each point downwards by 3). You can check 28 (38) (37) . 0) 0. f (αu) = αf (u) (36a) (36b) When U and V are two complex linear spaces. as illustrated in Figure 3 for two choices of a. The fact that the graph is a line helps us remember why these functions are called linear. or equivalently if. whose graph is shown in Figure 4.Spring 2009 3 Linear Functions f : U → V. The only functions satisfying these conditions are functions of the form f (x) = ax where a is some fixed real number. The function value f (u) is an element of V . (39) The graph of such a function is simply a line through the origin with slope a. We are now ready to define what we mean by a linear function. Note that f (¯ = ¯ by taking α = 0. Consider U and V two real linear spaces. (35) First. f (u(1) + u(2) ) = f (u(1) ) + f (u(2) ) ∀u ∈ U and α ∈ R. recall what we mean by the notation The function f takes an element of U as input. condition (36b) is modified such that α belongs to C. for example. It is defined for any element of U . Washington AMATH 352 . Another way to define an affine function is to say that g(x) is affine if g(x) − g(y) is a linear function of x − y. The function f : U → V is a linear function if both of the following conditions are satisfied: ∀u(1) . Definition 65. This graph is also a straight line but this is not a linear function according to the strict terms of Definition 65. Now consider the function g(x) = 6x − 3. It cannot be since g(0) = 0 and we can also easily check that g(1 + 2) = 15 while g(1) + g(2) = 12. Indeed. we have f (x) = f (x · 1) = xf (1) and f (x + y) = f (x) + f (y) = f (x · 1) + f (y · 1) = xf (1) + yf (1) = (x + y)f (1).U.

This function is said to be linear if both real functions f1 (x) and f2 (x) linear functions in the sense of Definition 65. Figure 4: Graph of the affine function g(x) = 6x − 3. so make sure you understand these basic ideas even if they seem trivial now!) 3. Washington AMATH 352 . we see that a function f : R → R2 is linear only if it has 29 . then we would say that f maps R to R2 . From what we noticed in previous section. that this is true for any function of the form g(x) = ax + b. If the outputs are real numbers and defined for any real number x.U. We could then use the symbol f (x) to denote the vector f (x) = f1 (x) f2 (x) (40) are are the the containing these two output values.5x. and these are the only affine functions for the case we are currently considering. (Right) Graph of f (x) = −0.Spring 2009 Figure 3: (Left) Graph of f (x) = 2x. (Soon these ideas will be generalized to more interesting situations.2 Linear functions from R to Rm We might have a situation where there are two different output values f1 (x) and f2 (x) that depend on all the same input value x. the simple case of a function mapping R to R.

and f 0 . 0 0 1 then we can easily find the value of f (x) for any x ∈ R3 . 30 . f 1 . am are m scalar constants. · · · . x2 . To make the notation simple.  (42) f (x) =  . 1 0 0 (45) An essential property of a linear function is the following: if we know the values of       1 0 0 (46) f 0 . am x where the values a1 . More generally. we can still talk about the function f (x) if we now think of x as the vector   x1 x = x2  . 3. (44) x3 0 0 x3 0 1 0 It is easy to show       0 0 1 f (x) = x1 f 0 + x2 f 1 + x3 f 0 . 1 . Washington AMATH 352 . Indeed. any vector of R3 is a linear combination of basis vectors       1 0 0 0 . a function f : R → Rm is linear if and only if it has the form   a1 x  . we write a linear combination in terms of the unit basis vectors               0 0 1 0 0 x1 x1 x2  =  0  + x2  +  0  = x1 0 + x2 1 + x3 0 . and 0 . and x3 .U. say x1 . (47) 0 0 1 Taking linear combinations of vectors is a fundamental operation in linear algebra.Spring 2009 form f (x) = a1 x a2 x (41) where the values a1 and a2 are two scalar constants. (43) x3 Using the standard unit basis (33).  .3 Linear functions from Rn to R Suppose f is a function that depends on several input values.

 (50) f (x) =  . You would be right! More generally. From what we noticed in the previous sections. You might guess that every linear function mapping R3 to R must have the form f (x) = a1 x1 + a2 x2 + a3 x3 (48) for some constant real numbers a1 .m. . . m. · · · . . . f is a linear function if and only if the m components f1 . This is one of the reasons why linear problems and linear equations are so important in applications.n = (aij )m×n 31 (53) am2 ··· amn . the ith function fi (x) is then given by n fi (x) = ai1 x1 + ai2 x2 + · · · + ain xn = j=1 aij xj (51) for i = 1. a2 .4 Linear functions from Rn to Rm Now suppose that the function f is a vector with m components that depends on n input values. . (49) 3. am1 A common notation is A = (aij )i=1. · · · . .  . This was recognized many years ago and so the notation of a matrix was invented. We often work with linear functions with many inputs and outputs and so it is nice to simplify the notation for describing these functions. then all we need to do is evaluate the function f for some well chosen set of n linearly independent vectors and we can evaluate f (x) for any x in all of Rn . . fm (x) According to Definition 65. an are n scalar constants. . The function is uniquely determined by mn numbers that are naturally arranged in a matrix with m rows and n columns:   a11 a12 · · · a1n  a21 a22 · · · a2n    A= . (52) . and a3 .··· . .··· . . . We can write   f1 (x)  . Washington AMATH 352 . 2. .j=1.U. fm are linear functions of x.   . a function f : Rn → R is linear if and only if it has the form f (x) = a1 x1 + a2 x2 + · · · + an xn where the values a1 .Spring 2009 It turns out that if f : Rn → R is a linear function.

That simple case is just the case m = n = 1 in which the “matrix” a is 1 × 1. αA = (αaij )m×n . we have   3 1×3+0×2+2×1 5 1 0 2   2 = = . We define the concept of matrix-vector multiplication so that this is correct: Definition 67. The set of all real matrices with m rows and n columns is denoted Rm×n . (55) The matrix-vector multiplication can   x1     x2  b =  a1 a2 · · · an   . For example.Spring 2009 For shorthand.U. For any matrices A = (aij )m×n and B = (bij )m×n in Rm×n . + xn an  where b is expressed as a linear combination of the columns aj . (54) where x is the vector of inputs with n components.  . i. This notation suggests that we multiply the m × n matrix A by the vector x to obtain the vector f (x). we might write f (x) = Ax. 32 . Next we define the addition between two matrices and the scalar multiplication.e.   xn also be displayed as follows:     = x1 a1 + x2 a2 + . For any matrix A = (aij )m×n in Rm×n and any scalar α in R. Definition 66.  .e. the fact that any linear function can be written in the form (54) (for some particular matrix A) is a nice generalization of the fact that any single linear function of a single variable has the form f (x) = ax for some single number a. If A is an m × n matrix of the form (52) and x ∈ Rn is an n-vector. Definition 69. Definition 68. the scalar multiplication αA is a matrix in Rm×n whose entries are the entries of A multiplied by α. A + B = (aij + bij )m×n . . then the product b = Ax is an m-vector (b ∈ Rm ) and the ith component of b is n bi = ai1 x1 + ai2 x2 + · · · + ain xn = j=1 aij xj . i. Washington AMATH 352 . It is a slight change of notation to highlight that x acts of A to produce b. the addition A + B is a matrix in Rm×n whose entries are the sum of the entries of A and B. . (56) (−1) × 3 + 3 × 2 + 1 × 1 4 −1 3 1 1 Note that once we have defined matrix-vector multiplication properly.

Since f and g are both linear functions. Remark. Proposition 70. Now define a new function h(x) by h(x) = f (x) + g(x). So h is a linear function and the matrix that defines it is simply the sum of the matrices A and B defining the functions f and g. Rm×n is a real linear space. But this means that h(x) is defined by the matrix-vector multiplication h(x) = Cx where C is the m × n matrix with components cij = aij + bij . by adding the corresponding elements of the two matrices. Next. we first compute the two vectors f (x) and g(x). 1. This means that. using the “vector addition” rule from Rm . To see this. so both f and g map Rn to Rm ). note that the ith component of h(x) is hi (x) = fi (x) + gi (x) n n = j=1 n aij xj + j=1 bij xj (57) = j=1 (aij + bij )xj . 33 . So the matrix A + B is also a matrix with m rows and n columns and it belongs to Rm×n . we have A = (aij )m×n B = (bij )m×n ⇒ A + B = (aij + bij )m×n . which is a complex linear space. For any matrix A and any matrix B in Rm×n . and n columns is denoted Cm×n . Washington AMATH 352 . for any vector x ∈ Rn . Suppose we have two linear functions f (x) = Ax and g(x) = Bx that are defined by two different matrices A and B (both of which are m × n. Then we add them together. to compute the value of h(x) ∈ Rm . we give a proof that Rm×n is a real linear space. The set of all matrices with complex entries.Spring 2009 An alternative definition of the addition of two matrices uses linear functions. it turns out that the function h is also a linear function.U. In other words. So the ith component of h(x) is just the ith component of f (x) added to the ith component of g(x). C=A+B where we define the sum of two matrices of the same shape in the obvious way. m rows.

U. 34 . is such that A + B = 0. 3. For any matrix A in Rm×n and any scalar α in R. The zero matrix in Rm×n is the matrix with all its mn components equal to 0. the matrix B. we have A + B = (aij + bij )m×n B + A = (bij + aij )m×n and the mn components are equal because the addition of scalar numbers is commutative. So the matrix αA is also a matrix with m rows and n columns and it belongs to Rm×n . For any matrix A in Rm×n . we have (α + β)A = ((α + β) × aij )m×n = (αaij + βaij )m×n = (αaij )m×n + (βaij )m×n = αA + βA. we have A = (aij )m×n ⇒ αA = (αaij )m×n . . 8. Indeed. 9. . we have (αβ)A = ((αβ) × aij )m×n = (αβaij )m×n = α (βaij )m×n = α (βA) . 4. we have α(A + B) = α (aij + bij )m×n = (αaij + αbij )m×n = (αaij )m×n + (αbij )m×n = αA + αB. Indeed. For any matrices A and B in Rm×n and any scalar α in R. defined by A = (aij )m×n B = (−aij )m×n . the vectors A + B and B + A are equal. The matrix B is the additive inverse of A. For any matrix A in Rm×n and any scalar α and β in R. . 0 ··· 0 5. .  0= . . Washington AMATH 352 .Spring 2009 2. The addition of matrices is associative. For any matrix A and any matrix B in Rm×n . The mn components are equal because the addition of scalar numbers is associative. For any matrix A in Rm×n and any scalar α and β in R. we have (A+B)+C = ((aij + bij ) + cij )m×n A+(B+C) = (aij + (bij + cij ))m×n . 6.   0 ··· 0  . 7.

β ∈ R. Indeed we have   Ax =    . i. . n tr(A) = a11 + a22 + · · · + ann = i=1 aii . Many of the fundamental concepts we learn about linear algebra for vectors and matrices carry over directly to functions and differential operators.Spring 2009 10. . Consider the matrices A = (aij )m×n and B = (bij )m×n . Washington AMATH 352 . we have 1 · A = 1 · (aij )m×n = (1 × aij )m×n = (aij )m×n . Exercise 72. The function. What are the entries of the matrix αA + βB? Exercise 73. m = n. n bmj xj j=1   n j=1 b1j xj   = n j=1 bmj xj n j=1 b1j xj  n j=1 (a1j n j=1 (amj . Proposition 71. So Rm×n is a real linear space. ⇒ Ax + Bx =  .e. Rm×1 denotes the set of matrices with m rows and 1 column. 3. (58) (59) The proofs simply use the associativity of real numbers to rewrite aij xj + bij xj as (aij + bij )xj . . Remark.U. Seeing these ideas in a different context now may help you to understand what “linearity” means more generally and why it is important. . + b1j )xj + bmj )xj    So we have proved something nontrivial about matrix-vector algebra using the rules of standard algebra of real numbers. The matrix-vector multiplication satisfies Ax + Bx = (A + B)x and Ax + Ay = A(x + y). Let α. n amj xj + j=1 n j=1 a1j xj    .5 Linear differential operators Recall that algebra and the idea of linearity applies to to situations other than vectors and matrices. 35 .  . Consider the square matrix A = (aij )n×n . Show that the trace function is a linear function on Rn×n .  Bx =  . It is also true for the other property of matrix-vector multiplication. Note that Rm is the set of column vectors with m rows and 1 column. (60) evaluates the trace of a square matrix. For any matrix A in Rm×n . . n j=1 amj xj  n j=1 a1j xj +  .

You are used to using this when finding the derivatives of complicated functions by splitting them up. The operator D is a linear operator because it is true that D(u + v) = D(u) + D(v). e. (61) dx dx dx d d (αu(x)) = α u(x). d2 The second derivative operator dx2 and all higher order derivative operators are also linear operators. which is just a function that takes a function as input and produces some other function as output. This gets a bit confusing so we call it an “operator” instead of a function. the operator L=8 d2 d −4 + 6I 2 dx dx (63) is a linear differential operator. since d d d 5x4 + 2 cos(6x) = 5 (x4 ) + 2 cos(6x) = 20x3 − 12 sin(6x). R). where I denotes the identity operator. and since d d d (u(x) + v(x)) = u(x) + v(x). (62) dx dx These linearity properties extend of course to arbitrary linear combinations. d The simplest differential operator is just D = dx . where U = C 1 (R. 36 .U. Washington AMATH 352 . For example. We obtain a general linear differential operator by taking a linear combination of these differential operators (and also the zero’th order derivative operator.g. Now consider the differential equation 8u (x) − 4u (x) + 6u(x) = x3 (65) (64) for 0 ≤ x ≤ 1 with u(0) = 1 and u(1) = 3. if f (x) = x3 2 then D(f ) is the function f (x) = 3x . dx dx dx Linearity was used in the first step. it satisfies the same linearity properties as in Definition 65.Spring 2009 Let us introduce the concept of a operator. Applying L to a function u(x) results in the function (Lu)(x) = 8u (x) − 4u (x) + 6u(x). The operator D is a linear operator. The problem is to find a funtion u(x) that satisfies the equation (65) everywhere in the interval and also satisfies the two boundary conditions. For example. This is a linear differential equation since it has the form (Lu)(x) = g(x) where L is the linear operator of (64) and g(x) is a given function g(x) = x3 . D(αu) = αD(u). which is just the identity operator that maps any function to itself). R) and V = C 0 (R. A differential operator computes the output function by combining various derivatives of the input function.

Exercise 75. defines the matrix A ∈ R3×2 .n) defines the matrix in Rm×n with entries equal to 1.5.3.n) makes a matrix of size m × n with random values. 37 . R) → C 0 (R.n) defines the matrix in Rm×n with entries equal to 0. 2. uniformly distributed between 0 and 1. • A*x computes the matrix-vector product. • A = rand(m. Check whether the operator O is linear. • size(A) returns the dimensions of A in a row vector [m n]. Consider the operator O : C 1 (R. R) defined by O(f )(x) = 2f (x)f (x) ∀x ∈ R. • A = zeros(m. 6]. normally distributed (with mean 0 and variance 1).n) makes a matrix of size m × n with random values. Check whether the operator O is linear. Note that commas separate the column entries on a same row and that semi-columns finish every row.Spring 2009 Exercise 74. 3. R) defined by O(f )(x) = xf (x) ∀x ∈ R.6 Useful commands in Matlab Here are a few commands in Matlab useful for this section. • A = randn(m.U. • tr(A) computes the trace of matrix A. • A = ones(m. • A = [1. R) → C 0 (R. Consider the operator O : C 0 (R. 4. Washington AMATH 352 .

U. 0 0 .2)) 38 . a21 . Washington AMATH 352 . We write a linear combination of matrices where each parameter is multiplying a matrix           1 0 0 1 0 0 0 0 0 0 A = a11 0 0 + a12 0 0 + a21 1 0 + a22 0 1 + a31 0 0 0 0 0 0 0 0 0 0 1 0   0 0 + a32 0 0 .1)) = 0 (from entry (1. ζ 0 0  0 0 .1)) = 0 (from entry (3. consider R3×2 . 0 0 0 0 0 0 0 1 0 0 0 0 These 6 matrices are also linearly independent. and a32 . where a11 . The previous section proved that Rm×n is a real linear space. a22 .2)) = 0 (from entry (3. Its dimension is equal to mn. A simple basis for Rm×n consists of the mn matrices with only one non-zero entry. For example. a12 . a31 . a31 a32 The matrix A depends on 6 parameters: a11 . a21 . 1 0 .            0 1 0 0 0 0 0 0 0 1 0 R3×2 = span 0 0 . 0 0 . a22 . a32 ∈ R. a31 . Any matrix with 3 columns and 2 rows is written as follows   a11 a12 A =  a21 a22  . 0 1 This last formula illustrates that the 6 matrices span R3×2 . 1 By identifying each entry to   α    β    γ  δ       ζ zero. Indeed.1 Matrices Space of m × n matrices The set of m × n real matrices is denoted Rm×n .1)) = 0 (from entry (2. 0 1 .2)) = 0 (from entry (2.Spring 2009 4 4. we have  1 α 0 0   0 0 0 + β 0 0 0   1 0 0 + γ 1 0 0   0 0 0 + δ 0 0 0   0 0 1 + 0 0 1    0 0 0 0 + ζ 0 0 0 0 1     α β 0 0 =  γ δ  = 0 0 . we obtain the following 6 equations = 0 (from entry (1. a12 .

Remark. The set of m × n complex matrices is denoted Cm×n . 0 0 1   0 0 0 . for example. . . c11 = a11 b11 + a12 b21 . Remark. b23 B= (68) c12 c22 c32 c42  c13 c23  . 0 0 0 0 0   0 0 1 . a32  a42  c11 c21 C= c31 c41 b11 b21 b12 b22 b13 . which will be a matrix in Rm×n .U. c33  c43 (69) where. (67) For example. if  a11 a21 A= a31 a41 then C ∈ R3×4 and  a12 a22  . a  1 0 0 basis for   0 0 0 . It is a real linear space of dimension 2mn. c12 = a11 b12 + a12 b22 . 39 . 4. Washington AMATH 352 . 0 0 0  0 0 . 1 0 . c43 = a41 b13 + a42 b23 . It is a complex linear space of dimension mn.Spring 2009 So these 6 matrices are linearly independent because the only linear combination resulting in the zero matrix is the trivial combination with α = β = γ = δ = = ζ = 0. Consequently. then we can define the product C = AB. . 0 0 0 R3×2 is given by     1 0 0 0 0 . The elements of the product matrix C are given by r cij = k=1 aik bkj .2 Matrix-matrix multiplication If A ∈ Rm×r and B ∈ Rr×n . 1 (66) The dimension of R3×2 is 6. Note that this product is only defined if the number of columns in A is equal to the number of rows in B. The set of m × n complex matrices is denoted Cm×n .

Then the products AB and BA are both defined. B ∈ Rn×n are both square and of the same size. Note. In the case where n = 1. A numerical example is depicted in Figure 6. the product is        c1 ··· cn  = A  b1 ··· bn  =  Ab1 ··· Abn  .U. Washington AMATH 352 . The diagram highlights the origins of each coefficient. The ith row of the product C = AB is the ith row of A multiplied by the matrix B. Figure 6: Example of Matrix-Matrix product (Diagram from Wikipedia). The matrix-matrix product is illustrated in Figure 5. Written in terms of columns. Yet another way to view matrix-matrix multiplication is in terms of rows. however. the matrix-matrix multiplication AB agrees with matrix-vector multiplication. Suppose A. that in general AB = BA. (70) Remark.Spring 2009 Figure 5: Illustration of Matrix-Matrix product (Diagram from Wikipedia). Each product is again an n × n matrix. B only has one column. Remark. Matrix multiplication is not commutative in general! 40 .

3 B= 4 1 5 . · · · .Spring 2009 Example 76. When the products AB and BA are equal. 0 BA = 4 1 23 .3 Range and rank of a matrix Definition 79. u ∈ Rm×1 . a(n) ).   u  v1 · · · vn =  v1 u · · · = . also called the range of A.  vn u  . · · · . . Let a(1) . 0 0 41 (75) (74) .U. . Consider the product of an m-dimensional column vector. . we say that A and B commute. • rank(A) ≤ n. u n v1 ··· un vn 4. A particular example of matrix-matrix product is the outer product. since R(A) is spanned by the n columns of A. It is denoted by R(A) = span(a(1) . The dimension of the range of A is called the rank of A: rank(A) = dim(R(A)). (73) The columns are all multiple of the same vector u. v ∈ R1×n . 0 (71) Definition 77. and similarly. then the columns are linearly dependent and there exists a vector z = ¯ such that Az = ¯ = 0m×1 . 2 (72) 1 0 2 . So a basis for R(A) has at most n vectors. B ∈ Rn×n . In this case. Example 78. with an n-dimensional row vector. The column space of A. The outer product is an m × n matrix that can be written     u v ··· u v   1 1 1 n . • if rank(A) < n. Definition 80. the columns form a basis for R(A). Washington AMATH 352 . since R(A) is a subspace of Rm . Consider A. the rows are all multiple of the same vector v. Let A= Then AB = 6 3 5 . a(n) ∈ Rm denote the columns of a matrix A ∈ Rm×n . is the subspace of Rm spanned by the columns of A. Note the following: • rank(A) ≤ m. • rank(A) = n if and only if the columns of A are linearly independent.

The two column vectors are linearly independent.U. a2 ) is a basis for R(A). We denote a1 and a2 the two columns of A. The range of A is spanned by these two column vectors: R(A) = span(a1 . 42 . Example 81. The dimension will be the number of vectors in the basis. So (a1 . a2 ) is a basis for R(A).Spring 2009 Similarly. Consider the matrix 2 A =  −5 1   6 3 . n)). The range of A is spanned by these two column vectors: R(A) = span(a1 . a matrix of full rank with m ≥ n must have n linearly independent columns. The dimension will be the number of vectors in the basis. The rank is defined similarly for complex matrices in Cm×n . To get a basis. a2 ). To get a basis. we need to find a basis. Assume there exists a linear combination of a1 and a2 equal to the zero vector: α 3 2 +β 0 4 = 0 0 ⇒ 3α = 0 2α + 4β = 0 The first equation gives that α = 0. Remark. The rank of A is 2. Consider the matrix A= 3 2 0 4 . For example. the row rank of a matrix is the dimension of of the space spanned by its rows. An m × n matrix of full rank is a matrix with the maximal possible rank (min(m. Assume there exists a linear combination of a1 and a2 equal to the zero vector:        6 0 2  2α + 6β = 0 −5α + 3β = 0 α  −5  + β  3  =  0  ⇒  2 0 α + 2β = 0 1 The last equation gives that α = −2β. we beed to find linearly independent vectors that span the range of A. Example 82. Replacing this value into the last equation gives β = 0. Row rank always equals column rank. a2 ). Consequently. So (a1 . Washington AMATH 352 . we need to find a basis. we beed to find linearly independent vectors that span the range of A. Replacing this value into the first equation gives 6β − 4β = 0. The rank of A is 2. which implies that β = 0. we have α = β = 0 and the two column vectors are linearly independent. To find the dimension of R(A). So we refer to this number simply as the rank of a matrix. 2 We denote a1 and a2 the two columns of A. To find the dimension of R(A).

Three choices are possible.U. a2 . Assume there exists a linear combination of a1 . β = 1. Compute the rank of the following matrices   2 6 1 2 1 0 2  1 3 . a3 ). and a3 the three columns of A. we need to find two vectors that are linearly independent. a2 ) is a basis for R(A). To check whether the rank is 2. Compute the range and rank of a nonzero outer product. The rank of A is 2. Replacing this value into the second and third equation gives    γ = 3α  γ = 3α 2α − 8β + 6α = 0 8α − 8β = 0 ⇒   −α − 2β + 3α = 0 2α − 2β = 0 which implies that β = α but leaves the value of α arbitrary. 0 These three vectors are linearly dependent. . The range of A is spanned by these three column vectors: R(A) = span(a1 . and a3 equal to the zero vector:          3 0 −1 0  3α − γ = 0 2α − 8β + 2γ = 0 α  2  + β  −8  + γ  2  =  0  ⇒  −1 −2 1 0 −α − 2β + γ = 0 The first equation gives that γ = 3α. Washington AMATH 352 . a2 . Here we choose a1 and a2 . For can choose α = 1. The dimension will be the number of vectors in the basis. we beed to find linearly independent vectors that span the range of A. Then we get α = β = 0 and the two column vectors are linearly independent. . Consider the matrix  3 A= 2 −1 0 −8 −2  −1 2 . Exercise 84. To get a basis. a2 . 1 We denote a1 . and γ = 3. 3 4 −1 3 1 2/3 2 Exercise 85. Assume there exists a linear combination ofa1 and a2 equal to the zero vector:        3 0 0  3α = 0 2α − 8β = 0 α  2  + β  −8  =  0  ⇒  −1 −2 0 −α − 2β = 0 which implies that α = 0. we need to find a basis. 43 . So the rank of A is smaller than 3. So (a1 . we  0 0 . Then we have          3+0−3 −1 0 3 1  2  + 1  −8  + 3  2  =  2 − 8 + 6  =  −1 − 2 + 3 1 −2 −1 example. To find the dimension of R(A).Spring 2009 Example 83.

The dimension of its null space is 0. The matrix A is full rank. A(αz + βw) = αAz + βAw = ¯ 0.U. then N (A) = {¯ . Az = ¯ . Example 88. The matrix A is full rank. Washington AMATH 352 . We have seen that the rank of this matrix is 2. Consider the matrix 2 A =  −5 1   6 3 . The rank-nullity theorem gives dimN (A) = 2 − 2 = 0.Spring 2009 4. There is no basis for this null space. Definition 86. We have N (A) = {0} . Consider the matrix A= 3 2 0 4 . 44 . Example 87. we can show dim(N (A)) = n − rank(A). then N (A) has dimension at least 1. There is no basis for this null space. In fact. 0} • if rank(A) < n. This space is called the null space of A. 2 We have seen that the rank of this matrix is 2. 0} Note the following: • if rank(A) = n. (77) (76) This result is sometimes referred to as the rank-nullity theorem or the dimension theorem. The rank-nullity theorem gives dimN (A) = 2 − 2 = 0. The dimension of its null space is 0.4 Null space of a matrix Note that the set of all vectors z for which Az = ¯ is a linear space. The null space of A ∈ Rm×n is the subspace of Rn defined by N (A) = {z ∈ Rn . Since if 0 Az = ¯ and Aw = ¯ then 0 0. We have N (A) = {0} .

So we should expect 1 parameter. For example. where z is a 1. It spans the null space. 1  z1 Assume z =  z2  is a nonzero vector belonging to the null space of A.U.  z1 nonzero vector.Spring 2009 Example 89. Replacing this value into the second and third equation gives    z3 = 3z1  z3 = 3z1 2z1 − 8z2 + 6z1 = 0 8z1 − 8z2 = 0 ⇒   −z1 − 2z2 + 3z1 = 0 2z1 − 2z2 = 0 which implies that z2 = z1 but leaves the value of z1 arbitrary. We have z3      3 0 −1 z1 3z1 − z3 Az = 0 ⇒  2 −8 2   z2  =  2z1 − 8z2 + z3  −1 −2 1 z3 −z1 − 2z2 + z3 The first equation gives that z3 = 3z1 . The dimension of its null space is  We have N (A) = span(z). Washington AMATH 352 . Consider the matrix  1 A= 1 1 1 1 1  1 1 . 3 Example 90. Consider the matrix  3 A= 2 −1 0 −8 −2  −1 2 . 1 We have seen that the rank of this matrix is 2. It is normal that z1 is arbitrary because the dimension of the null space is 1. we can choose z1 = 1. We z3 have      z1 z1 + z2 + z3 1 1 1 Az = 0 ⇒  1 1 1   z2  =  z1 + z2 + z3  1 1 1 z3 z1 + z2 + z3 45  . z2 = 1. Assume z =  z2 . Then we have        0 3+0−3 1 3 0 −1  2 −8 2   1  =  2 − 8 + 6  =  0  . The rank-nullity theorem gives dimN (A) = 3 − 2 = 1. and z3 = 3. 0 −1 − 2 + 3 3 −1 −2 1   1 This vector is nonzero. So a basis of N (A) is  1 .

So a general formula for the null space of A is    z1    .U. is the n × m matrix whose (i. ∀z1 . Compute the null space of the following matrix   2 6 1 0 2  1 3 . i) entry of A. −1 −1 Next we check whether these two vectors are linearly independent. The transpose of an m × n matrix A. . a32 (78) Definition 93. we get that the rank of A is equal to 1. When A = −AT . 3 4 −1 3 1 2/3 2 4. So the dimension of the null space is 2. For example.  1  . So these two vectors are linearly independent.  a11 A = a21 a31  a12 a22  a32 =⇒ AT = a11 a12 a21 a22 a31 . 46 . We have    1 ∀z ∈ N (A). Using the rank nullity theorem. z = z1  0  + z2  −1 So we can write  formula depends on two  0 1 . Exercise 91. the matrix A is called symmetric. We notice that this parameters z1 and z2 .5 Transpose/Adjoint of a matrix Definition 92. j) entry is the (j. the matrix A is called skew-symmetric. Consider a linear combination of these two vectors equal to the zero vector        1 0 0  α=0 β=0 α 0  + β 1  =  0  ⇒  −1 −1 0 −α − β = 0 which implies that α = β = 0. denoted AT . −1    0 1 N (A) = span  0  . z2 ∈ R . z2 N (A) =    −z1 − z2 We look for a basis for N (A). They form a basis for N (A). When A = AT . 1 2 .Spring 2009 All three equations give that z3 = −z1 − z2 . Washington AMATH 352 .

whose non-zero entries are only on the diagonal. Indeed. So.Spring 2009 Note that rank(AT ) = rank(A). A symmetric matrix must be square. Note that an hermitian matrix must be square. • a diagonal matrix. we have T T T AT A = (A) AT = AT A. the dimensions for N (A) and N (AT ) can differ. The matrix AT A is a symmetric matrix.U. since the column rank and row rank are equal. it is true that (AB)T = BT AT . 0} 3 Remark 94. When A = A∗ . then the product AT A exists and it is a square matrix of dimension n. in general. the matrix 3 0 −1 A= 2 4 6 has a null space of dimension 1 and its transpose a null space of dimension 0. obtained by negating the imaginary part of aij . The adjoint of a complex m × n matrix A. So the matrix product BT AT is defined and is an n × m matrix. then the product C = AB ∈ Rm×n exists. Washington AMATH 352 . However. Note that we have: • AT T = A. The null spaces are   1 N (A) = span  −5  N (AT ) = {¯ = {02×1 }. is the n × m matrix whose (i. When A = −A∗ . Note that in this case BT ∈ Rn×r and AT ∈ Rr×m . • the identity matrix is a symmetric matrix. (A∗ )ij = aij . i) entry of A. denoted A∗ . If A ∈ Rm×r and B ∈ Rr×n . For example. We can take the transpose of this matrix and will get an n × m matrix CT ∈ Rn×m . 47 (80) . with the order reversed. but with the order reversed! If A ∈ Rm×n is a matrix. Definition 96. the adjoint of the product of matrices is the product of the adjoint. In fact. Definition 95. the matrix A is called skew-hermitian. the matrix A is called hermitian. (79) The transpose of the product is the product of the transposes. j) entry is the conjugate (j. this is just equal to CT ∈ Rn×m . Similarly to the transpose. is symmetric.

1 2 3 x3 We have proved that. For the inner product defined in (14). Definition 100. A nonsingular or invertible matrix is a square matrix of full rank. we replace the transpose operation T with the adjoint operator ∗ . Exercise 99. if B ∈ Rm×n then multiplying B on the left by an m × m identity matrix or on the right by an n × n identity matrix leaves B unchanged. Show that A − AT is a skew-symmetric matrix. Consider x ∈ R3 . Characterize the diagonal entries of a real skew-symmetric matrix. we similarly prove that ∀x. More generally. for x ∈ R3 . Characterize the diagonal entries of a complex hermitian matrix. Show that A + AT is a symmetric matrix. √ x 2 = xT x.U. x · y = xT y. Exercise 98. (81) This relation is also true in Rm . Washington AMATH 352 . Also. y ∈ Rm . Ix = x. Prove that any square matrix is the sum of a symmetric and a skew-symmetric matrix. Ibn  =  b1 So multiplying a matrix by I leaves it unchanged. if B ∈ Rn×n .6 Matrix inverse The n × n matrix that has 1’s on the diagonal and 0’s everywhere else is called the identity matrix and denoted by I. (82) In Cm . It is called the identity matrix because for any x ∈ Rn . 4. x3  then we have  xT x = x1 x2 x3  x1  x2  = x2 + x2 + x2 . Let A be a square real matrix.Spring 2009 Exercise 97.  x1 x =  x2  . 48 .  IB =  Ib1 ···   ···  bn  = B.

The matrix Z is called the inverse matrix of A and written as A−1 . Then we have ej = Azj and combining all the vectors zj in the matrix Z. we get    e1 ··· en  = I = AZ. the following conditions are equivalent: • A has a unique inverse A−1 such that AA−1 = A−1 A = I. For A ∈ Rn×n . i. The same is true for invertible matrices.e. The matrix A−1 is the inverse for the multiplication and satisfies the relation AA−1 = I. Theorem 101. • rank(A) = n. It is the matrix version of the familiar expression aa−1 = 1 for any nonzero scalar a ∈ R.U. • R(A) = Rn . In particular. 49 (85) (84) . every basis vector ej of the standard unit basis (33) has a unique expansion in column vectors.Spring 2009 Note that a full rank n × n square matrix has columns that form a basis for the linear space Rn (or Cn for complex matrices). Therefore. Washington AMATH 352 . • N (A) = {¯ 0}. It should not be confused with the additive inverse of A that is −A. A similar result holds for complex matrices when replacing R with C. For any nonzero scalar a ∈ R. (83) Let zj denote the column vector with entries zij . AA−1 = A−1 A = I. the expression aa−1 = a−1 a = 1 holds. any vector in Rn has a unique expression as a linear combination of the column vectors. n ej = i=1 ai zij .

for any square invertible matrix A. det a c b d = ad − bc. For example. −1 = Recall.5. If A. The inverse of a 2 × 2 matrix is equal to a c if and only if ad − bc = 0. If A is invertible. for 3 × 3 50 . Indeed. A similar formula holds for inverses in the square case. Prove formula (88). The inverse matrix commutes with A and the product in either order is the identity matrix. we get A−1 Ax = x ∀x ∈ Rn . So equation (86) holds only when A and B are both invertible and. (87) Proposition 103. Since the null space of A is trivial. proving that A−1 A = I.U. Note that. we multiply on the right by A to get AA−1 A = A and A A−1 Ax − x = 0 ∀x ∈ Rn .Spring 2009 Starting from (84). Exercise 104. we have AT −1 = A−1 T . Washington AMATH 352 . Corollary 102. in particular. B ∈ Rn×n are both nonsingular. we can write B−1 A−1 AB = B−1 A−1 A B = B−1 IB = B−1 B = I and use the uniqueness of the inverse matrix. (89) b d −1 = 1 ad − bc d −b −c a (88) A 2 × 2 matrix is invertible if and only if its determinant is nonzero. that the transpose of a matrix product is the inverted product of the transposes. then the products AB and BA will also be singular and noninvertible. square. We emphasize that the inverse of a matrix is defined only for square matrices. (86) Again the order is reversed. The determinant can be defined for any square n × n matrix. If either A or B is singular. then A−1 is invertible and we have A−1 A. The scalar ad − bc is called the determinant of the matrix. then so is their product AB and −1 (AB) = B−1 A−1 . from Section 4.

Remark. This means that x is the vector of coefficients of the unique linear combination of b in the basis of columns of A.. • det(αA) = αn det(A). However. Remark. 4. The determinant satisfies the following properties: • det(I) = 1. If we knew the inverse matrix.U.Spring 2009 matrices. However. Gaussian elimination) that require less work than computing the inverse matrix and often give more accurate solutions (when the rounding errors of computer arithmetic are taken into account). we should understand x as the unique vector that satisfies the equation Ax = b. the determinant rarely finds a useful role in numerical algorithms. in practice. When writing the product x = A−1 b. Multiplication by A−1 is a change of basis operation. then we could solve any linear system Ax = b simply by multiplying b by A−1 . We give their definitions here. A square n × n real matrix O is orthogonal if OT = O−1 or OT O = I. Definition 105. there are better ways to solve the system (e.g.  a det d g  a det d g  a det d g we have  b c e f  h i  b c e f  h i  b c e f  h i = a det e f h i − b det d g f i + c det d g e h = a(ei − hf ) − b(di − gf ) + c(dh − ge) = (aei + bf g + cdh) − (ahf + bdi + cge) We can generalize that a n × n matrix is invertible if and only if its determinant is nonzero. we should not think of x = A−1 b as the result of applying A−1 to b. 51 .7 Orthogonal/Unitary matrices Orthogonal and unitary matrices play an important role in numerical algorithms. • det(AB) = det(A) det(B). where A is a square matrix of dimension n. Numerically. we rarely work with inverse matrices. • det(A−1 ) = 1/ det(A). • det(AT ) = det(A). Instead. Some of their properties will be described later in the notes. Washington AMATH 352 .

8 Useful commands in Matlab Here are a few commands in Matlab useful for this section. Let O denote an n × n real orthogonal matrix and x an n × 1 real vector. • inv(A) computes the inverse of matrix A when it exists. When A has only real entries. (91) The multiplication by an orthogonal matrix preserves the 2-norm of a vector. • rank(A) computes the rank of a matrix. 52 . • A’ computes the adjoint matrix A∗ . When n = 2. • C = A*B computes the matrix-matrix product. Washington AMATH 352 .’. 0 0 − 2/2 2/2 2 ∞ ∞ In Cm .U. • null(A) generates an orthonormal basis for the null space of A. A square n × n complex matrix Q is unitary if Q∗ = Q−1 or Q∗ Q = I. Matlab uses A. The 2-norm of Ox is equal to Ox 2 = (Ox)T Ox = √ xT OT Ox = √ xT x = x 2 . Note that they do not preserve the ∞-norm or the 1-norm. For example. the adjoint matrix A∗ is equal to the transpose matrix AT . 4. √ √ √ 2 1 2/2 √2/2 1 √ = 1 and = . • I = eye(n) creates the identity matrix in Rn×n .Spring 2009 Definition 106. the unitary matrices preserve the 2-norm of complex vectors and the inner product. Orthogonal matrices preserve also the inner product (14). To compute the matrix (without the complex conjuguate). • det(A) computes the determinant of a matrix. the orthogonal matrices are cos θ − sin θ sin θ cos θ and cos θ sin θ sin θ − cos θ (90) where θ belongs to R.

5.i bj . A norm · is a map from U → R+ . In that case. The map. we discuss norm and inner products for matrices.i 53 2 bj 2 .U. Let C = AB with entries cij and let arow. we have |cij | ≤ arow. the map  A = (aij )m×n −→  i=1 j=1 m n 1/p p |aij |  is a norm on Rm×n . (93) is called the Frobenius norm. u ≥ 0. (92d) To define a norm for matrices. αu = |α| u . we generalize the norms and the inner product defined on Rm (see Section 1. we look at the matrix just through its entries. . First. the resulting matrix norms are very similar to vector norms. u (1) +u ≤ u (1) + u (2) (Triangle inequality). ∀α ∈ R. Then we have cij = arow. Example 108. Consider U a real linear space. So. by the Cauchy-Schwarz inequality.u (2) ∈ U. In particular. u = 0 if and only if u = ¯ 0. ∀u (1) (92a) (92b) (92c) (2) .1 Norms Definition 107. Washington AMATH 352 .3) to linear spaces. several approaches are possible. Note that it is similar to the vector p-norm on Rmn . It is easy to show that this map is a norm according to Definition 107.i be the ith row of A and bj the jth column of B. Consider the linear space of real matrices U = Rm×n and the matrix A = (aij )m×n . It is similar to the 2-norm of Rmn . satisfying the following conditions : ∀u ∈ U.Spring 2009 5 Norms and Inner Products In this section. m n 2 A F = i=1 j=1 |aij | . The Frobenius norm can be used to bound the product of matrices. For example.

The 1 54 . This approach is intimately related to a vector norm. (94) The induced matrix norm is the largest value of the ratios Ax (m) / x (n) .Spring 2009 Squaring both sides and summing over i and j. 2 0 . Prove that the norm · (m. 2 0 0 1 ⇒ 1 0 0 1 = 1 1 A 1 0 0 1 = 2. Consider the linear space of real matrices Rm×n and the vector norms · (n) on the domain space Rn → R+ and · (m) on the range space Rm → R+ . the induced matrix norm compares the length of the vectors x and Ax and is equal to the largest possible ratio between these lengths. Sometimes. Note that we used the subscripts (n) and (m) to avoid any confusion with the vector p-norms. the (1.e.i 2 2 = A B . j=1 Another approach to define a matrix norm is to assess the effect of the matrix on the length of a vector.U. 2 0 0 = 0 1 1 ratio is here 1.n) satisfy the properties (92) defining a norm. for example. Washington AMATH 352 . Exercise 110.n) is the maximum gain over all directions. The resulting norm is called the induced or associated matrix norm. We have the following 0 1 relations using. the ratio Ax (m) / x (n) is called the amplification factor or gain in the direction x.n) = max x=0 Ax x (m) (n) = max x (n) =1 Ax (m) . The 1 • ⇒ = 1 1 A = 1.  bj 2 2 2 F 2 F n ≤ i=1 arow. arow. i. Consider the matrix A = • 2 0 1 = 0 1 0 ratio is here 2. we get m n AB 2 F = i=1 j=1 m n |cij | . Considering all the nonzero vectors x. It is easy to verify that · (m. The induced matrix norm is defined as A (m. the 1-norm of the vectors. So the norm A (m.1) induced matrix norm: Example 111. Definition 109.n) satisfies the properties (92).i i=1 j=1 m 2 2 2 ≤ bj   2 2 .

Since we have found one vector for which the ratio is exactly 2. Washington AMATH 352 . Therefore. Consider an arbitrary vector x. Note that aj 1 is the vector 1-norm because aj is a vector. Example 112. indeed. x 1 |x1 | + |x2 | |x1 | + |x2 | This bound shows that the ratio can not be larger than 2. where j is maximizing aj 1 . we see that the ratio depends on the vector x.1) induced matrix norm for A is equal to 2. we attain the bound. However. the unit standard basis vector of . it does not say anything about the direction of the vector. 2 1 4 1 ⇒ 1 1 2 1 = 2 1 A 1 1 2 1 = 3. Ax 1 = j=1 a j xj 1 ≤ j=1 |xj | aj 1 ≤ x max 1 1≤j≤n aj 1 . where aj is the jth colulumn vector of A. 1) induced norm is often denoted A (1. It could be less than 2 for some vectors.Spring 2009 • 2 0 1 = 0 1 1 ratio is here 3/2. at most. the case. The 1 From these examples. the (1. It seems that 2 will be the largest one.1) = A 1 . 2 0 2 = 0 1 1 ratio is here 5/3. The (1. Then we have x= and Ax = So the ratio is equal to Ax 1 2 |x1 | + |x2 | 2 |x1 | + 2 |x2 | = ≤ = 2. i=1 (95) which is equal to the maximum column sum. For any vector x ∈ Rn . Thus the matrix 1-norm is m A 1 = max aj 1≤j≤n 1 = max 1≤j≤n |aij | . Let A be an m × n matrix. This notation is not confusing with the vector 1-norm because A is a matrix.U. the induced matrix 1-norm satisfies A 1 ≤ max aj 1≤j≤n 1 . The 1 • ⇒ = 3 1 A = 5. This norm value indicates that the matrix A can multiply the length of a vector by 2. By choosing x = ej . . We can write Ax 1 ≤ A 55 1 x 1 . we have n n x1 x2 2x1 x2 x 1 = |x1 | + |x2 | Ax 1 = 2 |x1 | + |x2 | . It is.

xT x . The (2. √ 1 n x ≤ ∞ 1 x ≤ 2 1 . we remark that the 2-norm of a real matrix is invariant under multiplication by orthogonal matrices. (98) 2 x 2 Later in the notes. x ∞ For any vector y ∈ Rm . we remark that. Washington AMATH 352 . ∞) induced norm is often denoted A (∞. ∀O ∈ Rm×m satisfying OT O = I.U. As a consequence of (98). 2) induced norm is denoted A (2. We can write Ax ∞ ≤ A ∞ x ∞ . we have y ∞ ≤ y 2 ≤ √ m y ∞ . The (∞. Recall that we have √ x ∞ ≤ x 2 ≤ n x ∞. Example 114. (99) The 2-norm of a complex matrix is invariant under multiplication by unitary matrices.∞) = A ∞ . This notation is not confusing with the vector 2-norm because A is a matrix. (100) for any vector x ∈ Rn . j=1 (96) which is equal to the maximum row sum. The matrix ∞-norm is n A ∞ = max 1≤i≤m |aij | . ∀A ∈ Rm×n .Spring 2009 Example 113. all the matrix norms are related to each other. When comparing equations (95) and (96). A similar property holds for complex matrices when using the adjoint matrix A∗ . Similarly to norms in Rm . For the moment. for x = 0. for any matrix A ∈ Rm×n .2) = A 2 . This notation is not confusing with the vector ∞-norm because A is a matrix. We can write also. 56 . Let A be an m × n matrix. The matrix 2-norm is defined by A We can write Ax 2 2 = max x=0 Ax 2 = x 2 ≤ A max x=0 xT AT Ax . we have (97) A ∞ = AT 1 . just remember that the 2-norm of a matrix A is related to the matrix AT A. OA 2 = A 2 . we will introduce the spectral radius that describes precisely the 2-norm of a matrix. Let A be an m × n matrix.

Exercise 117. x = 0.U. So we obtain A 2 ≤ m A ∞ . Prove that. Note that the supremum is also the minimal upper bound. 57 . So we have ∀ x ∈ Rn . So we have ∀ x ∈ Rn . We have Ax 2 Ax ∞ ≤ ≤ x ∞ x ∞ 1 √ n √ Ax 2 ≤ n A x 2 2 (103) where we used twice the equation (100) and where the last inequality is due to the supremum. we consider the left hand side. Prove that. we have 1 A n ∞ ≤ A 1 ≤m A ∞ . Using again the supremum as the minimal upper bound. Washington AMATH 352 . we have 1 √ A n ∞ ≤ A 2 ≤ √ m A ∞ .Spring 2009 For any matrix in Rm×n . √ Ax ∞ ≤ n A x ∞ 2 . (101) Let us prove the right side of the bound. Exercise 116. We have √ Ax ∞ √ Ax ∞ √ Ax 2 (102) ≤ m ≤ m ≤ m A ∞ x 2 x 2 x ∞ where we used twice the equation (100) and where the last inequality is due to the supremum. Prove that. Exercise 115. √ Ax 2 ≤ m A x 2 √ ∞ . we have A 2 ≤ A F ≤ √ n A 2 . x = 0. Next. we obtain 1 √ A n ∞ ≤ A 2 . for any m × n matrix. we have 1 √ A m 1 ≤ A 2 ≤ √ n A 1 . for any m × n matrix. for any m × n matrix. Consider any non-zero vector x ∈ Rn .

The notions introduced in this Section can be extended to Cm×n . Show that the map x W = Wx 2 is a norm. v >=< v.U. w ∈ U. Exercise 119. 1]. Let A be an m × r real matrix and B be an r × n real matrix. · > on a real linear space U satisfies ∀u.r) B (r. Consider U = R and W a nonsingular matrix. a norm remains a map from U → R+ . v ∈ U. < αu. < u + v. Washington AMATH 352 . w > + < v. < u. u >= 0 if and only if u = ¯ 0. An inner product is a symmetric positive-definite bilinear form. R). ∀α. Remark. u >. the induced matrix norm satisfy AB In particular. · > is a map U × U → R satisfying the following properties ∀u ∈ U. ∀α ∈ R. v >= α < u. ∀u. An inner product < ·. < u. Show that the map ˆ f −→ 0 1 f (t)2 dt is a norm.r) Bx r ≤ A (m. w ∈ U. w >=< u.r) B (r.n) ≤ A (m. Consider U a real linear space. v. (104) ≤ ≤ ≤ A A A 1 2 ∞ B 1 B 2 B ∞ (105) Exercise 118. v >. Consider U = C 0 ([0. ∀u. Therefore. we have ABx m ≤ A (m. v. we have    AB 1 AB 2 AB ∞ n (m. αv + βw >= α < u. For any x ∈ Rn . (107) 58 (106a) (106b) (106c) (106d) (106e) . < u. u >≥ 0.Spring 2009 The induced matrix norm of a matrix product can also be bounded. v ∈ U.n) . When U is a complex linear space. ∀u. 5. Prove that an inner product < ·.2 Inner products Definition 120. < u. v > +β < u. β ∈ R. Exercise 121. w > .n) x n . w > .

Exercise 127. B) −→ tr(AT B) is an inner product. Consider U = Rm×n . v > Exercise 123. The map (A. v >. g) −→ 0 1 f (t)g(t)dt is an inner product. v >| ≤ < u. The map from U to R+ . Example 126. ∀A ∈ Rm×n . Given U a real linear space and an inner product < ·. Washington AMATH 352 . Consider U = C 0 ([0.U. 1]. defined by √ (109) u −→ < u. u > < v. OA F = A F . tr (OA)T OB = tr(AT OT OB) = tr(AT B). < v. u − λv >≥ 0 with λ= < u. 59 . v > . · > on U . The Cauchy-Schwarz inequality holds for any inner product. u >. R). Prove Proposition 124. The proof uses < u − λv. is a norm. we remark that the Frobenius norm is also invariant under multiplication by orthogonal matrices. (111) From this last expression. Show that the map ˆ (f. Exercise 125. we write √ √ (108) |< u. (112) The associated inner product is also invariant under multiplication by an orthogonal matrix. It satisfies also tr(AT A) = A 2 F (110) . ∀O ∈ Rm×m satisfying OT O = I.Spring 2009 Example 122. Proposition 124. with equality if and only if v is proportional to u. For any vectors u and v of U . Prove the Cauchy-Schwarz inequality (108). The Euclidean inner product of two column vectors in Rm (14) satisfy the properties (113).

3. < u. v ∈ U are orthogonal for the inner product < ·. ∀α ∈ R. < u + v. on the order of 10−3 . (113a) (113b) (113c) (113d) (113e) 5. It may seem obvious what is meant by the error. w ∈ U. u >. · > when < u. v ∈ U. v >= < v. v. ∀u. their inner product is 0. Then the error in this approximation is ˆ e = x − x. i. w > + < v. The Cauchy-Schwarz inequality and the orthogonality still holds in the complex case. Consider a problem to which the true answer is a vector x ∈ Rm .Spring 2009 The concept of orthogonality also extends to general real linear spaces. v >= α < u. w > .U. Definition 128. Denote ˆ an approximation by x. Consider U a complex linear space. An inner product < ·. Washington AMATH 352 .1 Absolute error A natural measure of this error would be the norm of e. ∀u.2 while some numerical method produced an approximate value x = 2. Let U be a real linear space. v >= 0. Then the absolute error is ˆ |x − x| = 0.3 Errors In order to discuss the accuracy of a numerical solution or the relative merits of one numerical method versus another. v >. w >=< u. it is necessary to choose a manner of measuring the error. u >= 0 if and only if u = ¯ 0. suppose that x = 2. ˆ e = x−x .20345. ˆ This seems quite reasonable — we have a fairly accurate solution with three correct digits and the absolute error is fairly small. We conclude by giving the definition of an inner product on a complex linear space. · > is a map U × U → C satisfying the following properties ∀u ∈ U.45 × 10−3 . < u. But there are often many different ways to measure the error that can sometimes give quite different impressions as to the accuracy of an approximate solution. < u.00345 = 3. u >≥ 0. ∀u. This is called the absolute error in the approximation. We say that u. v ∈ U.e. We 60 . As an example. Definition 129. An inner product is a symmetric positive-definite sesquilinear form. < αu. 5.

But suppose that instead we expressed the solution (and the approximate solution) in nanometers rather than meters.45 × 106 .2| |2. particularly if several different scales arise in the same 61 .20345| = = 1. defined by ˆ x−x . giving an absolute error of |x − x| = 3. Conversely.2 Relative error The above difficulties arise from a poor choice of scaling of the problem. suppose the x discussed above were measured in meters. the relative error in x is equal to ˆ (2.57 × 10−3 . Of course if we know that our problem is “properly scaled”.45 × 10−6 .20345 × 109 . For these reasons the relative error is often a better measure of accuracy than the absolute error. which is roughly the same as the relative error in this case.2 × 109 | regardless of what units we choose.2 × 10−3 and x = ˆ 2. ˆ The absolute error may seem much smaller. in general a relative error that is approximately 10−k indicates that there are roughly k correct digits in the solution. Then the true solution is x = 2.20345 × 10−3 . For the above examples. 5. so that the solution x has a magnitude that is order 1. so x = 2.2 − 2.Spring 2009 might be very pleased with an alternative method that produced an error of 10−6 and horrified with a method that produced an error of 106 .2 × 109 and the approximate solution is x = ˆ 2. so |x − x| = 3. Poorly scaled problems can lead to other numerical difficulties. but there are still only three correct digits in the approximation.2 meters is the correct solution.U. a very desirable feature. Washington AMATH 352 . |2.3. For example. x The size of the error is scaled by the size of the value being computed. But note that our notion of what is a large error or a small error might be thrown off completely if we were to choose a different set of units for measuring x. Moreover. One way to avoid this is to consider the relative error. if we measured x in kilometers. then x = 2. then it is fine to use the absolute error. matching our intuition.20345) × 109 |2. The relative error is “dimensionless” and its value is always independent of the units chosen to measure x.2 − 2. In fact it is generally better to insure that the problem is properly scaled than to rely on the relative error. ˆ We have an error that seems huge and yet the solution is just as accurate as before. with three correct digits.

For example. 1.2 × 1010 Here a relatively small variation in the right hand side leads to extremely large variations in the vector x. Unless otherwise noted. 1 − 1010 1 + 1010 1010 −1010 . There may be measurement errors or there may be values that cannot be measured in any precise way and must be “guessed” by the modeler. A well-conditioned problem is one with the property that all small perturbations of the input lead to only small changes in the ouput of the problem.Spring 2009 problem so that some numbers are orders of magnitude larger than others for nonphysical reasons. A mathematical model of some real world phenomenon typically involves some parameters or other data describing the particular situation being modeled. An ill-conditioned problem is one with the property that some small perturbation of the input can lead to a large change in the ouput. 62 .1 0. we will generally assume that the problem is scaled in such a way that the absolute error is meaningful. Errors in the data mean that even if the model is very good and the equation is then solved exactly it may not match reality. The problem is ill-conditioned. It is important for the numerical analyst to evaluate the effect of these data errors. until now. data errors form a source of perturbations. we have skirted.1 + 0. Washington AMATH 352 .9 . we define a number called to the condition number to assess what is the conditioning of a problem.1 − 0.4 Conditioning Conditioning is a fundamental issue of numerical analysis that.U. Consider the matrix A= Its inverse matrix A−1 is A−1 = The linear system Ax = has the solution x= The linear system A˜ = x has the solution ˜ x= 1. In the following. 1 1 1 1 . These values are almost never known exactly. It pertains to the perturbation behavior of a mathematical problem. 5.2 × 1010 1. 1 2 1 1 + 10−10 1 1 − 10−10 .

63 . To estimate the relative perturbation. respectively.4. we bound the difference in absolute terms. x ˜ difference between b and b. (114) This bound gives an absolute measure of the perturbation in the right hand side b. A small norm A 2 means that a small perturbation in the vector x leads to a small perturbation in the vector b. If b and b ˜ we would like to bound the are. On the other hand. x 2 So we obtain ˜ b−b 2 b ≤ A 2 A−1 2 2 ˜ x−x x 2 2 . even when x − x 2 is small.Spring 2009 5.U.1 Error when computing Ax = b ˜ ˜ Suppose that the vector x is perturbed to obtain the vector x. (115) Definition 130. We have ˜ b−b 2 = = 2 Ax − A˜ x ˜ A(x − x) 2 2 ˜ b−b So we obtain ˜ b−b 2 ≤ A 2 ˜ x−x 2 . Washington AMATH 352 . ˜ Note that the bound (114) is sharp because there exists a perturbation x − x ˜ = A ˜ such that b − b x−x . First. Note that the bound (115) is sharp because there ˜ exists a perturbation x − x such that the equality is reached. (116) The bound (115) gives a relative measure of the perturbation in the right hand side b. we write ˜ b−b 2 2 2 2 b = 2 2 Ax − A˜ x Ax 2 A 2 2 ˜ b−b b ≤ 2 ˜ x−x 2 Ax 2 Next. The condition number of A for the 2-norm is κ2 (A) = A 2 A−1 2 . a large value for κ2 (A) means that the perturbation in b can be large. we bound the norm of Ax with the norm of A−1 as follows x 2 = A−1 Ax 2 ≤ A−1 2 Ax 2 ⇒ 1 Ax ≤ 2 A−1 2 . satisfying Ax = b and A˜ = b. On the other hand. even when ˜ x − x 2 is relatively small. a large value for A 2 ˜ means that the perturbation in b can be large. A small condition number means that a small perturbation in the vector x leads to a small perturbation in the vector b.

a large value for ˜ κ2 (A) means that the perturbation in b can be large. Suppose that the matrix A and the vector x are perturbed to obtain the ˜ ˜ ˜ matrix A and the vector x. we bound the difference between b and b. Washington AMATH 352 . If b and b are. we bound the norm of Ax with the norm of A−1 as follows 1 Ax So we obtain ˜ b−b 2 ≤ 2 A−1 2 . If b and b ˜ ˜ are. A small condition number means that a small perturbation in the matrix A leads to a small perturbation in the vector b. The relative change and A˜ 64 . To estimate the relative perturbation. even when A − A is 2 relatively small. respectively. ˜ b−b 2 b ≤ A 2 A−1 ˜ A−A 2 2 2 A . by multiplying and dividing by A 2 . On the other hand. (117) 2 The condition number indicates how much the vector b can be perturbed. satisfying Ax = b and Ax = b.U. we bound the difference in absolute terms. x 2 b ˜ ≤ A−A 2 2 A−1 2 x x 2 2 and. we would like to bound the ˜ difference between b and b. This bound gives an absolute measure of the perturbation in the right hand side b. respectively. satisfying Ax = b ˜ ˜ ˜ x = b. We have ˜ b−b 2 = = 2 ˜ Ax − Ax 2 ˜ b−b So we obtain ˜ b−b 2 ˜ (A − A)x 2 ˜ ≤ A−A 2 x 2 . we write ˜ b−b 2 ˜ Ax − Ax = 2 b 2 2 Ax x 2 2 ˜ b−b b ≤ 2 2 ˜ A−A Ax 2 Next.Spring 2009 ˜ ˜ Suppose that the matrix A is perturbed to obtain the matrix A. First.

So the particular norm is usually not very important. The condition number κ2 (A) depends on the induced 2-norms of matrices A and A−1 . we have also κ∞ (A) = A and Matlab gives κ2 (A) = A Applying the vector x = −1 1 2 ∞ A−1 ∞ = 2 × 1010 + 1 A−1 2 = 2 × 1010 . Prove the bound (118). Example 132. The condition number of A is equal to κ1 (A) = A 1 A−1 1 = 1+ 10−10 2 × 2 × 1010 = 2 × 1010 + 1. the condition number of A is the relative change magnification factor. Consider the matrix A= Its inverse matrix A−1 is A−1 = 1 − 1010 1 + 1010 1010 −1010 . b = Ax = . 1 2 1 1 + 10−10 1 1 − 10−10 . For any other induced matrix norm (like the 1-norm · 1 or the ∞-norm · ∞ ). The actual numerical value of κ(A) depends on the norm being used. for example: κ1 (A) = A 1 A−1 1 and κ∞ (A) = A ∞ A−1 ∞ . Exercise 131. However. a condition number can be similarly defined. Washington AMATH 352 .U. to the matrix A results in the vector b: 0 −10−10 65 .Spring 2009 satisfies ˜ b−b 2  ≤ A 2 b A−1 2 2 ˜  x−x x 2 ˜ A−A 2 ˜ A−A 2 + A + 2 2 A 2 ˜ x−x x 2  2 (118) Again. we are usually only interested in order of magnitude estimates of the condition number. For this matrix.

66 .U.1 and ˜ b−b 1 b = 1 10−11 . Washington AMATH 352 . In the 2-norm. The induced norm of A is the largest magnification factor in a direction: Ax A = max . For example. We have the following relative changes: ˜ x−x x 1 1 = 1 and ˜ b−b 1 b = 2 × 1010 + 1. The condition number is sometimes defined as the ratio A κ(A) = . 2 Remark. (119) m Expressions (116) and (119) are equal since m = min x=0 A−1 (Ax) Ax x = max = max = A−1 . the vectors b=A 1 1 = 1 1 ˜ and b = A 1.9 = 1 1 + 10−11 give the following relative changes ˜ x−x x 1 1 = 0.Spring 2009 ˜ The perturbed vector x = 1 1 ˜ gives rise to the vector b: ˜ b = A˜ = x 1 1 . 1 illustrating that the condition number is the magnification factor. x=0 x We can also define the smallest magnification factor: m = min x=0 Ax . x=0 Ax x=0 x Ax (120) For any p-norm. x Note that if matrix A is singular. we have κp (A) ≥ 1. orthogonal matrices are perfectly conditioned in that κ(Q) = 1 if Q is orthogonal. Note that some small perturbation of x can still result in small perturbation of b. then m = 0.1 0.

11 9. This is equivalent to the problem just studied. It is finite as long as A is nonsingular. b = 0.0007246 and 1 ˜ x−x x 1 1 = 1.2 Error when solving Ax = b Given an invertible matrix A and the vector b. solving the system Ax = b is mathematically identical to computing the vector A−1 b. Washington AMATH 352 . Ax = b and we have x 1 = 1 and b = 13.Spring 2009 5. .34 0.4.7 . m×n For a rectangular matrix A ∈ R .63 = 2249.4. κ(A) being the magnification factor.4.8 9.7 6.6 x= 1 0 b= 4. we have κ1 (A) ≥ 1. Consider the matrix and vectors A= 4. but approaches ∞ as A approaches a singular matrix. except that the matrix A is replaced by the matrix −1 A−1 . the condition number gives a good indication of how much accuracy we can expect in a computed solution to Ax = b. The condition numbers κ(A) and κ(A−1 ) are equal since A−1 = A. we have actually an equality κ1 (A) = 2249. of full rank.1 9.70 0.97 .0007246 For this example.1 2. Example 133. (m ≥ n). The size of the condition number is a good measure of how close to singular the matrix is. Clearly. 67 . 0. In particular.63. If the right-hand side is changed to ˜ b= the solution becomes ˜ x= The relative changes ˜ b−b 1 4. These definitions extend also to complex matrices.U. the condition number is defined in terms of the pseudoinverse κ(A) = A A† = A (AT A)−1 AT . one must always expect to “lose” log10 κ(A) digits of accuracy in computing the solution. When solving a problem Ax = b with a fixed digits of precision.8.

If the differentiable function f is from a linear space U equipped with a norm into a linear space V equipped with a norm. 5. we can again compute the absolute error on f (x) if x is slightly perturbed: f (x + h) − f (x) = f (ξ)h ≈ hf (x) where we invoked the mean-value theorem. The condition number can be extended to general functions.Spring 2009 Example 134.4. Washington AMATH 352 . f (x) 2 x/x u−u = ˜ J(u) u f (u) u−u ˜ . 68 . f (x) f (x) f (x) x So the factor xf (x)/f (x) is the condition number for this problem.3 Error when evaluating a general function Consider the function f : R −→ R that is differentiable. f (u) √ Example 135. i+j−1 This matrix is often used for test purposes because of its ill-conditioned nature. the absolute error satisfies f (u) − f (˜) ≈ J(u)(u − u) ≤ J(u) u ˜ The relative error is f (u) − f (˜) u J(u)(u − u) ˜ J(u) ≈ ≤ f (u) f (u) f (u) The condition number becomes J(u) u .5n . Consider the problem of computing x for x > 0. ˜ This is a well-conditioned problem. the relative error is f (x + h) − f (x) hf (x) xf (x) x + h − x ≈ = . When evaluating the function f at x. The function f is usually nonlinear.U. On the other hand. So we have κ= √ xf (x) 1/(2 x) 1 = √ = . The Hilbert matrix Hn = (hij )n×n is defined by hij = 1 . The Jacobian √ √ of f : x → x is the derivative f (x) = 1/(2 x). u u−u . The condition number κ(Hn ) increases rapidly with n and can grow like ce3.

1). cond(A. we get κ= J(x) ∞ x |f (x)| ∞ ∂f ∂x1 ∂f ∂x2 x1 x2 −→ x1 − x2 . |x1 − x2 | This quantity is large when |x1 − x2 | ≈ 0. So the problem is ill-conditioned when x1 ≈ x2 . • cond(A) computes the matrix condition number in 2-norm. • norm(A) computes the induced 2-norm of matrix A.U. norm(A. norm(A.p)*norm(inv(A). = 1 −1 . 69 . respectively. This example highlights the hazards of cancellation error.p) returns the matrix condition number in p-norm: norm(A. and the Frobenius norm. Consider the problem f : The Jacobian of f is J= Using the ∞-norm. norm(A. the 1norm.p). the p-norm.5 Useful commands in Matlab Here are a few commands in Matlab useful for this section.Spring 2009 Example 136. 5. = 2 max(|x1 | . Washington AMATH 352 .p).inf).’fro’) computes. |x2 |) . the ∞-norm. norm(A.

. . . . Washington AMATH 352 . (124) = q3 r33 = |r33 | . Assume that A ∈ Rm×n (m ≥ n) has full rank n and denote the column vectors a1 . The idea of QR factorization is the construction of orthonormal vectors q1 . q2 . 70 . qj ) When j = 1. . . r22 will not be zero else the column vector a2 would be linearly dependent with a1 . aj ) = span(q1 . When j = 2 and the first vector q1 is defined. qn such that span(a1 . .U. (122) which defines the first vector q1 and a coefficient r11 a1 2 = q1 r11 = |r11 | q1 = |r11 | . . and q3 1 2 which implies qT a3 = qT q1 r13 = r13 and qT a3 = qT q2 r23 = r23 1 1 2 2 and a3 − q1 r13 − q2 r23 2 2 = 1. .Spring 2009 6 The QR factorization The QR algorithm is a very important algorithmic idea in numerical linear algebra. the relation (121) becomes a3 = q1 r13 + q2 r23 + q3 r33 such that qT q3 = 0. .1 Reduced QR factorization For many applications. an . qT q3 = 0. the relation (121) becomes a2 = q1 r12 + q2 r22 which implies and a2 − q1 r12 2 such that qT q2 = 0 and 1 q2 2 = 1. When j = 3 and the vectors q1 and q2 are defined. a2 . . . . . we may choose that r11 > 0. 6. It is impossibe because the matrix A has full rank. . . q2 . . n. we are interested in the column span of a matrix A and in the knowledge of a basis for this subspace. . that span the range of A. . . the relation (121) takes the form a1 = q1 r11 such that q1 2 j = 1. We want to build the sequence of orthonormal vectors q1 . Arbitrarily. (123) qT a2 = qT q1 r12 = r12 1 1 = q2 r22 = |r22 | . (121) = 1. . . . We can choose again that r22 > 0. Note that r11 will not be zero because the matrix A has full rank.

. (127) Relation (121) can be rewritten in a matrix form      r11 r12  0 r22   a1 ··· an  =  q1 ··· qn   . It is impossibe because the matrix A has full rank. . . Generally. In Figure 7. 0 r1n r2n . . + qj rjj . This method is an old idea. We can choose again that r33 > 0. ··· . qj−1 are defined. . . we represent graphically the reduced QR factorization. . . Figure 7: Reduced QR factorization (m ≥ n) 71 . a2 ). rnn      (128) (129) where Q is m × n with orthonormal columns and R is n × n upper-triangular matrix with positive diagonal entries. Each matrix A ∈ Rm×n (m ≥ n) of full rank has a unique reduced QR factorization A = QR with rjj > 0.Spring 2009 r33 will not be zero else the column vector a3 would be linearly dependent with (a1 . . . when 1 ≤ j ≤ n and the j − 1 orthonormal vectors q1 . known as the Gram-Schmidt orthogonalization.. Such a factorization is called a reduced QR factorization of A. . (125) It is evident that an appropriate definition of the coefficients of rij is rij = qT aj i and rjj = aj − i=1 j−1 (126) qi (qT aj ) i 2 .U.  . 0 ··· or A = QR. . . Theorem 137. Washington AMATH 352 . The proof is provided by the constructive method described earlier. we write aj = q1 r1j + .

The first step constructs the vector q1 such that a1 = Next. which gives 2 2 = r12 =⇒ 8 r12 = √ . Consider the matrix A= 3 1 2 2 . Each complex matrix A ∈ Cm×n (m ≥ n) of full rank has also a unique reduced QR factorization A = QR with rjj > 0. Qx = 2 ˆ Qx T ˆ Qx = ˆ ˆ xT QT Qx = √ xT x = x 2 ˆ which indicates that Q 2 = 1.Spring 2009 Remark. Washington AMATH 352 . We will compute its QR factorization. So we have QT Q = I. we write 2 2 = 3 1 r √12 + q2 r22 10 √ 3/√10 1/ 10 T 3 1 a1 2 = √ 10 =⇒ q1 = √ 3/ 10 √ 1/ 10 . Example 138. Then we get ˆ ∀x ∈ Rn×1 . such that qT q2 = 0 and 1 q2 2 = 1. we write 3 2 1 2 = √3 10 √1 10 −1 √ 10 √3 10 24 10 8 10 4 − √160 12 √ 160 2− 2− = √ 4 − 10 12 10 160 = 10 − √1 10 √3 10 √ 2 10 . 72 . The matrix Q has orthonormal columns. The matrix A has a determinant equal to 4 and it is of full rank. So we obtain A ˆ = R 2 2 . 10 So we obtain q2 r22 = 2 2 8 −√ 10 √ 3/√10 1/ 10 = = Finally. ˆ ˆ ˆ Remark.U. where the identity matrix has dimension n × n. 5 √ 10 0 √8 10 √ 2 10 5 .

and q3 1 2 We obtain  T   1 0 1  √ 1   0  = r13 3 1 6   T   −2 0 1  √ 1   0  = r23 6 1 6  73 √ r13 = 2 3 2 = 1. qT q3 = 0. .U.Spring 2009 Example 139. 6 and q2 = √ 6 1 For the third column. 6 full rank. Washington AMATH 352 . √ 6 r12 = √ = 2 3. The matrix A is of step constructs the vector q1 such that    1 √ 1 a1 =  1  a1 2 = 3 =⇒ q1 = √  3 1 Next. 1 We will compute its QR factorization. 3          √ −2 2 0 0 1 2 3 1 = 3 − 2 = 1  = 3 − √ 3 1 2 3 3 1    −2 1  1 . we write     0 1 r12  3  =  1  √ + q2 r22 3 3 1 which gives   T  0 1 1    √ 3  = r12 1 3 1 3  Then we have a2 − q1 r12 and r22 = √ =⇒ such that qT q2 = 0 and 1 q2 2 = 1. =⇒ and =⇒ r23 = √ 6. The first  1 1 . we write       0 −2 1 r13 r23  0  =  1  √ +  1  √ + q3 r33 3 6 1 6 1 such that qT q3 = 0. Consider the matrix  1 A= 1 1 0 3 3  0 0 .

The complete algorithm is described in Algorithm 1.U. 74 . described in (125). . Washington AMATH 352 . Algorithm 1 Classical Gram-Schmidt for j = 1 to n do v = aj for i = 1 to j − 1 do rij = qT aj i v = v − rij qi end for rjj = v 2 qj = v/rjj end for 6. (126). and (130). √ 0 3 2 6.2 Gram-Schmidt orthogonalization The steps. we write  1  1 1 0 3 3 will satisfy        2 −2 0 0 √ 1 2  −  1  =  −3  =  −1  × √ × 3 2. 2 2 1 3 1   0  0 = 6 1 √ 3 1 √ 3 1 √ 3 −2 √ 6 1 √ 6 1 √ 6 0 −1 √ 2 1 √ 2  √ 3  0  0 √ √  2 3 2 3 √ √ 6 6 . where the matrix Qj−1 denote the m × (j − 1) matrix containing the first j − 1 column vectors q1 . form the j-th iteration of the Gram-Schmidt orthogonalization. . The matrix Pj = I − Qj−1 QT j−1 (131) represents an orthogonal projection. The projection Pj computes the component of aj that is in the span of Qj−1 and removes it. . qj−1 . it offers a simple route to understanding and proving various properties of QR factorizations.3 Projections (130) Equation (130) can be written in a matrix form rjj = (I − Qj−1 QT )aj j−1 2 . . Another interpretation is that the projector Pj computes the component of aj that is not in the span of Qj−1 .Spring 2009 ˆ The third column of Q    0 q3 r33 =  0  −  6 Finally. Mathematically.

If v belongs to the null space of P. Bau. The matrices QQT and I − QQT are orthogonal projectors. Washington AMATH 352 . Philadelphia.U. then I − P is also a projector. Proposition 141. SIAM.Spring 2009 Definition 140. Trefethen and Bau2 note that “The term projector might be thought of as arising from the notion that if one were to shine a light onto the range of P from the right direction. Using the result (77) about dimensions. The matrix I − P is called the complementary projector to P. Prove Proposition 141. Let Q be an m × n matrix with orthonormal columns. we derive the complementary fact N (I − P) = R(P). then v = (I − P)v and Pv = P [(I − P)v] = Pv − P2 v = 0.” Note that we have P2 v = Pv =⇒ P2 v − Pv = 0 =⇒ P(Pv − v) = 0. 1997. If P is a projector. Trefethen and D. A projection. is a square matrix P such that P2 = P. Numerical linear algebra. I − P projects exactly into the nullspace of P. 2 L. Exercise 142. Proposition 143. (135) N. i. these computations show that an m×m projector P separates Rm into two spaces. which implies that N (P) = R(I − P). Rm = R(P) ⊕ R(I − P). If v belongs to range(I − P). (134) (133) (132) We can also see that R(P) ∩ R(I − P) = {0}: any vector v in both sets satisfies Pv = v = v − Pv = 0. This means that Pv − v belongs to the null space of P. 75 . which implies N (P) ⊂ R(I − P). then (I − P)v = v belongs to the range of I − P. By writing P = I − (I − P). It is an orthogonal projector (in contrast to an oblique projector) when PT = P. then Pv would be the shadow projected by the vector v.e. or projector. QT Q = I.

. a1 =  1  a1 2 = 3 =⇒ q1 = √ 3 1 1 Next.  T   1 0 1    √ 0  = r12 1 3 1 0 76 =⇒ r12 = 0. Consider the matrix  1 A= 1 1 0 0 0  0 0 . In that case. 6 We will compute its QR factorization. Every matrix A ∈ Rm×n (m ≥ n) has a QR factorization A = QR. we simply choose the vector qj to be any arbitrary normalized vector orthogonal to the previously constructed vectors q1 . row of zeros are appended to R so that it becomes an m × n matrix R. still upper-triangular.4 Full QR factorization A full QR factorization of A ∈ Rm×n (m ≥ n) goes further. appending an ˆ additional m − n orthonormal columns to Q so that it becomes an m × m ˆ orthogonal matrix Q.Spring 2009 6. . we write     0 1 r12  0  =  1  √ + q2 r22 3 0 1 which gives  such that qT q2 = 0 and 1 q2 2 = 1. Figure 8: Full QR factorization (m ≥ n) When the matrix A does not have full rank. . Theorem 144. . qj−1 and continue the Gram-Schmidt algorithm. In Figure 8. Washington AMATH 352 . In the process. we represent graphically the reduced QR factorization.U. The first step constructs the vector q1 such that     1 1 √ 1   1 . Example 145. . The matrix A is of rank 2. the Gram-Schmidt algorithm can break when rjj is zero or v = 0.

q2 = √ 2 1 77 . The vector q2 can not be zero because it is of norm 1. 2 2 1 3 1   0  0 = 6 1 √ 3 1 √ 3 1 √ 3 1 √ 6 −2 √ 6 1 √ 6 −1 √ 2 3 0  0  1 √ 0 2  √ √  0 2 3 √ 0 √ . q2 = √ 6 1 For the third column. For instance. The third column of Q    0 q3 r33 =  0  −  6 Finally.U. For example. we write      1 0 1 r23 r13   0  =  1  √ + −2  √ + q3 r33 3 6 1 6 1  such that qT q3 = 0. we can choose   1 1  −2  . and q3 1 2 We obtain T    0 1 1  √ 1   0  = r13 3 1 6  T   0 1 1  √  −2   0  = r23 6 6 1   2 = 1.Spring 2009 Then we have a2 − q1 r12  0 = 0  0  which implies that r22 = 0. we write  1  1 1 0 0 0 will satisfy        2 1 −3 −1 √ 1 2  −  −2  =  0  =  0  × √ × 3 2. qT q3 = 0. Next we simply choose the vector q2 to be any arbitrary normalized vector orthogonal to the previously constructed vector q1 and continue the Gram-Schmidt algorithm. 6 0 3 2 Note that a different choice for q2 would result in a different factorization. setting   0 1  −1  . =⇒ √ r13 = 2 3 and =⇒ r23 = √ 6. Washington AMATH 352 .

which are conventionally normalized so that Pj (1) = 1. Each complex matrix A ∈ Cm×n (m ≥ n) has also a full QR factorization A = QR. one of the most powerful techniques for the numerical solution of partial differential equations.5 Solution of Ax = b by QR factorization Suppose we wish to solve Ax = b where A is a real square nonsingular matrix.U. Compute y = QT b 3. This suggests the following method for computing the solution to Ax = b: 1. x ). · > and √ the associated norm · = < ·. However. x . −1 Starting from the basis (1. use the Gram-Schmidt construction to build an orthonormal basis. The system of linear equations in x is also easy to solve because R is triangular. Consider P3 the space of polynomials of degree at most 3. Computations with such polynomials form the basis of spectral methods. Compute a QR factorization A = QR 2. Washington AMATH 352 . it is not the standard methods for such problems. 6. we can write QRx = b or Rx = QT b (136) The right hand side is easy to compute when Q is known. · >.6 Generalization of Gram-Schmidt orthogonalization The Gram-Schmidt orthogonalization has an analogue for an abstract real linear space. The Gaussian elimination will be described later in the notes. 6. √ 0 6 Remark. since it requires only half as many numerical operations. Let U be a real linear space equipped with an inner product < ·. Pj . Gaussian elimination is the algorithm often used in practice. Using the QR factorization. g >= f (t)g(t)dt. equipped with the inner product ˆ 1 < f. The resulting polynomials are scalar multiples of what are known as the Legendre polynomials. Solve Rx = y for x This combination is an excellent method for solving linear systems of equations. 2 3 78 . Exercise 146. The complete algorithm is described in Algorithm 2.Spring 2009 we obtain  1  1 1 0 0 0   0  0 = 6 1 √ 3 1 √ 3 1 √ 3 0 −1 √ 2 1 √ 2 −2 √ 6 1 √ 6 1 √ 6  √ 3  0  0 √  0 2√3 0 3 2 . x.

• [Q. . u(n) ).Spring 2009 Algorithm 2 Classical Gram-Schmidt with a general inner-product Require: (u(1) . • [Q. v > q (j) = v/rjj end for 6.R]=qr(A) computes the full QR factorization of A. . . u(j) > v = v − rij q (i) end for √ rjj = v = < v.U.0) computes the reduced QR factorization of A. . n members of U for j = 1 to n do v = u(j) for i = 1 to j − 1 do rij =< q (i) . Washington AMATH 352 .7 Useful commands in Matlab Here are a few commands in Matlab useful for this section.R]=qr(A. 79 .

1 + 2 × 2−52 . the interval [2j . Consider the matrix Aε = and its inverse A−1 = ε 1 2 1 1 1+ε 1−ε 1− 1+ 1 ε 1 ε 1 ε −1 ε (137) where 0 < ε < 1. Example 147. 2 + 2 × 2−51 . . Consequently. 1 + 3 × 2−52 . . In general. .23 × 10−308 . 1 + 2−52 . sometimes called single precision) or 8 bytes (64 bits. the relative gap between two numbers is never larger than 252 ≈ 2. Represented numbers have gaps between them. which are the same numbers multiplied by 2. in IEEE double precision arithmetic. computers can only represent a finite subset of real numbers. . the interval [1. It is good to be aware of this limitation and of potential pitfalls of computer arithmetic. . This range is usually sufficiently large.U. In IEEE double precision arithmetic. Matlab always uses 64-bit floating point numbers (or double precision). 2 + 2−51 . 2] is represented by the discrete subset 1. 2 ε ε 80 . . This gap is usually negligible but ill-conditioned problems or unstable algorithms can be affected by this finite gap. 2j+1 ] is represented by the numbers in (137) multiplied by 2j . . the IEEE double precision arithmetic permits numbers as large as 1.79 × 10308 and as small as 2. Careless construction of algorithms can have sometimes dramatic consequences as we will see in some examples. For example. 2 + 3 × 2−51 . . 4] is represented by the discrete subset 2. 7. Numbers are usually stored in the computer using either 4 bytes (32 bits. 2.2× 10−16 . Washington AMATH 352 . 4. Overflow and underflow are rarely a serious hazard.Spring 2009 7 Computer arithmetic Computer memory is partitioned into bytes consisting of 8 bits. or double precision). For example.1 Digital representation limitations Represented numbers in computer arithmetic can not be arbitrarily large or small. The interval [2. The 1-norms for these matrices are Aε and the condition number is κ1 (Aε ) = Aε 1 1 =1+ ε 2 A−1 ε 1 = 2 ε A−1 ε 1 = 1+ ε 2 2 × = + 1.

one must always expect to “lose” log10 κ1 (Aε ) digits of accuracy in computing the solution. For any value of ε. When solving a problem Aε x = b with a fixed digits of precision.490 × 10−08 0 1 Table 1: Rounding errors when solving ill-conditioned system Note the numerical value may actually vary with your machine. we have 1 1 Aε = .Spring 2009 Recall that the condition number gives a good indication of how much accuracy we can expect in a computed solution to Aε x = b.3433s 16384 209715 . after 100 hours. an Iraqi Scud missile could not be targeted and was allowed to detonate on a barracks.000110011001100110011002 = As a result. A notorious example is the fate of the Ariane rocket launched on June 4. 0. This discrepancy caused the Patriot system to continuously recycle itself instead of targeting properly.11 × 10−08 ≈ 1.11 × 10−04 inv(Aε ) ∗ 1 1 1 − 1 1 −11 ≈ 1.1s.455 × 10 ≈ 1. 1996 (European Space Agency 1996). However. 1 1 We solve the same system numerically with Matlab and compute the 1-norm of the error Aε \ ε = 10−5 ε = 10−8 ε = 10−12 1 1 − 1 1 −11 ≈ 1. 2097152 had accumulated. The system used an integer timing register which was incremented at intervals of 0. As a result.3 Example 149. Example 148. In the 37th second of flight. 3 See http://mathworld.11 × 10 ≈ 1. causing the rocket to veer off course and be destroyed.1. an error of 1 209715 − 10 2097152 × (3600 × 100 × 10) = 5625 ≈ 0. but instead triggered an overflow error which was interpreted by the guidance system as flight data. The Patriot missile defense system used during the Gulf War was also rendered ineffective due to roundoff error (Skeel 1992. killing 28 people. GAO 1992).S. the integers were converted to decimal numbers by multiplying by the binary approximation of 0.com/RoundoffError.wolfram. U.html 81 . the inertial reference system attempted to convert a 64-bit floating-point number to a 16-bit number. Washington AMATH 352 .U.

2. If we accept the possibility of having an infinite number of digits to the right of the decimal point. 3. The binary system uses 2 as the base in the same way that the decimal system uses 10.11101)2 = 1 × 23 + 0 × 22 + 0 × 21 + 1 × 20 + 1 × 2−1 + 1 × 2−2 + 1 × 2−3 + 0 × 2−4 + 1 × 2−5 (139) corresponds to the number 9.14159265358979323846264338 . we have 427. 5). 7. For example.375 = 1 × 28 + 1 × 27 + 1 × 25 + 1 × 23 + 1 × 21 + 1 × 20 + 1 × 2−2 + 1 × 2−3 . A number in the binary system can be expressed in a similar manner. then any real number can be expressed with a sign (+ or −) affixed to it.375 = 4 × 102 + 2 × 101 + 7 × 100 + 3 × 10−1 + 7 × 10−2 + 5 × 10−3 . (138) The base 2 number (1001. Consider the number 427. . −π is −π = −3. Washington AMATH 352 . we perform a series of division by 2 and reverse the order of the remainders to get the representation.325.90625. 7. For example.U. To compute the binary representation.1 Binary system Most computers deal with real numbers in binary systems. Then we have 427. .Spring 2009 7. in contrast to the decimal system that humans prefer to use. The expression on the right contains powers of 10 and digits (here: 4.2 Floating point numbers Almost all processors use IEEE standard arithmetic.2. 7. In the binary system. only the two digits 0 and 1 are used. For example: • (15)10 = (1111)2 = 1 × 23 + 1 × 22 + 1 × 21 + 1 × 20 – 15 = 2 ∗ 7 + 1 → remainder 1 – 7 = 2 ∗ 3 + 1 → remainder 1 – 3 = 2 ∗ 1 + 1 → remainder 1 – 1 = 2 ∗ 0 + 1 → remainder 1 • (11)10 = (1011)2 = 1 × 23 + 0 × 22 + 1 × 21 + 1 × 20 – 11 = 2 ∗ 5 + 1 → remainder 1 82 . which is an arithmetic system based on a floating point representation of the real numbers combined with the binary system.

0001 1001 1001 1001 .2. 1.x−1 x−2 . . In the binary system. 2. . The gaps between adjacent represented numbers scale in proportion to the size of the numbers. x1 x0 . When x = 0. The notation (x)β = (−1)s (xn xn−1 . Conversion procedures must be executed by the computer. .U. Errors can be involved in each conversion. We should be aware that there are two conversions involved — from and to decimal. . + x−m × m ). . + x1 × β + x0 × 1 1 1 1 + x−1 × + x−2 × 2 + . 7. . we have x = ±m × 2e 83 . (141) β β β k=−m (−1)s (xn × Computers communicate with users in the decimal system but works internally in the binary system. In a floating point number system. the position of the decimal (or binary) point is stored separately from the digits. . any integer β > 1 can be used as the base for a number system. then e = 0. 10 So the conversion of 1/10 to binary will result in a roundoff error. . Recall that computers represent only a subset of the real numbers. .2 Floating point number Almost all processors use IEEE standard arithmetic. or zero). negative. any nonzero real number x can be represented as x = ±m × 10e 1 where the mantissa m is a number in the range 10 ≤ m < 1 and the exponent e is an integer (positive. a simple number like 1/10 can not be stored exactly on any binary machine. It requires an infinite binary expression 1 = (0. . which is an arithmetic system based on a floating point representation of the real numbers.)2 . Washington AMATH 352 . This is different from a fixed point representation where the gaps are all the same. . . Numbers represented in base β will contain digits 0. . . For example. x−m )β means n with xn = 0 (140) (x)β = (−1)s = xk β k β n + xn−1 × β n−1 + . β −1.Spring 2009 – 5 = 2 ∗ 2 + 1 → remainder 1 – 2 = 2 ∗ 1 + 0 → remainder 0 – 1 = 2 ∗ 0 + 1 → remainder 1 In general. In normalized scientific notation.

84 . (Recall that exponent fields of all 0s or all 1s are reserved for special meanings. |x − x | ≤ 7. 8-bit exponent. there exists a floating number x . then the number represented is (−1)s × 2e−127 × (1 + m). ±∞.Spring 2009 where 1 ≤ m < 1 (if x = 0). It simply means “too large to represent”. the least significant bit in the mantissa represents units of 2−23 (or approximately 1.2. If s. The range of positive normalized numbers is from 2−126 (the underflow threshold) to 2127 × (2 − 2−23 ) ≈ 2128 (the overflow threshold) or about 10−38 to 1038 . and 23-bit mantissa (or fraction) in the IEEE single precision format. The two error values are “infinity” (often denoted “Inf”). the exponent satisfy 0 < e < (11 111 111)2 = 211 − 1 = 2047. the machine epsilon machine has the property ∀x ∈ R. 1+2−52 . floating-point hardware distinguishes between + and −. Inf does not necessarily mean that the result is actually infinite.2 × 10−7 ).U. and m < 1 are the 1-bit sign. It is also defined as the maximum relative representation error. The values e = 0 and e = 255 are reserved for special cases such as 0. The range of positive normalized numbers is from 2−1022 (the underflow threshold) to 21023 × (2 − 2−52 ) ≈ 21024 (the overflow threshold) or about 10−308 to 10308 . and 52-bit mantissa (or fraction) in the IEEE double precision format. With a 23-bit mantissa. If s. 11-bit exponent.) The mantissa field is set to something that can distinguish them—typically zero for Inf and nonzero for NaN. that is. Both m and e are base 2 numbers. e. then the number represented is (−1)s × 2e−1023 × (1 + m).3 Machine epsilon The machine epsilon is traditionally half the distance between 1 and the next larger floating point number. and m < 1 are the 1-bit sign.4 Special symbols Inf and NaN machine |x | . and NaN. Washington AMATH 352 . and “NaN” (“not a number”). On 8 bits. The sign bit is meaningful for Inf. In a relative sense. 7. Both of these are encoded with the exponent field set to all 1s. 2 IEEE arithmetic includes two kinds of floating point numbers: single precision (32 bits long) and double precision (64 bits long).2. which covers all other errors. the exponent satisfy 0 < e < (11 111 111)2 = 28 − 1 = 255. e. On 11 bits. The maximum relative representation error is 2−53 ≈ 10−16 .

In other cases in which the result’s exponent is too large to represent.U. . ×. we have fl(fl(x) and fl(fl(x) 7. + xn yn . The corresponding computer operations involve roundoff errors. or subtraction on floating point numbers. and the result is set to infinity of the appropriate sign. In computer arithmetic.1 Floating point operations Arithmetic On a computer. all mathematical computations are reduced to certain elementary arithmetic operations. −.Spring 2009 When a nonzero number is divided by zero (the divisor must be exactly zero). xT y = x1 y1 + x2 y2 + . A floating point operation is multiplication. such as • (+Inf) + (+7) = (+Inf) • (+Inf) × (−2) = (−Inf) • (+Inf) × 0 = NaN — there is no meaningful thing to do 7. though both produce a result of infinity. for the substraction. addition.3 7. and the distinction is usually unimportant in practice. Computing the inner product of two vectors in Rn . these symbols represent operations in R. fl(y)) = (fl(x) − fl(y))(1 + ) with | |≤ machine (142) (143) Operation counts The work required to solve a problem on a computer is often measured in terms of the number of floating point operations or flops needed to do the calculation. also producing infinity of the appropriate sign. such as division of an extremely large number by an extremely small number. Example 150.3. division. Then we have fl(x) = x(1 + ε) with |ε| ≤ machine . Denote fl(x) the floating point number closest to x. This is different from a divide by zero. a “divide by zero” event occurs. the four basic operations applied to floating point numbers are exact up to a relative error of size at most machine . Note that we have fl(fl(x)) = fl(x). 85 . The classical set is +. an “overflow” event occurs. For example. Washington AMATH 352 . requires n multiplications and n − 1 additions for a total of 2n − 1 flops.3. Mathematically.2 fl(y)) = (x(1 + ε1 ) − y(1 + ε2 ))(1 + ) with | |≤ machine . Floating-point hardware is generally designed to handle operands of infinity in a reasonable way. . ÷.

etc. where a floating point operation means adding two numbers together. or 2 billion cycles per second). If the work required by some algorithm for a system of size n has the flop count W (n) = a0 + a1 n + a2 n2 + .Spring 2009 Example 151. This tells something about how many instructions the processor can execute per second. then doubling the size of the problem (from n to 2n) will increase the work by a factor 2k .top500. however.org for a list of the top 500 machines in the world and information about each. More generally. This can vary quite a bit depending on exactly what is being done. For large value of n. We say W (n) = O(nk ) as n −→ +∞. 2 GHz (2 gigahertz. Remark. that these fast machines are based on processors that are not all that fast. Consider A ∈ Rm×r and B ∈ Rr×n . or other basic arithmetic operation. . When W (n) = O(nk ). Computing y = Ax where A ∈ Rm×n and x ∈ Rn . teraflops. then the last term dominates for large values of n. to talk about the speed of computers. + ak nk (where aj is a constant). When m = n = r. which require 2r − 1 flops. When the matrix A is square (m = n). gigaflops. e. which can perform certain benchmark linear algebra problems at more than one petaflops (1015 floating point operations per second). if W (n) is “Big oh” of nk . It should be noted. . but generally has the same order of magnitude as the cycle speed (a 2 GHz processor can do roughly 1 billion arithmetic operations per second).U. Currently the “fastest” computer in the world is the Roadrunner computer at Los Alamos National Laboratory. So we need to compute yi for i = 1. then the ratio W (n)/nk remains bounded as n −→ +∞: W (n) = O(nk ) ⇔ 0 ≤ W (n) ≤ Cnk for large values of n (where C is a constant independent on n). The total work is 2mn − m flops. So the total work becomes 2mnr − mn flops. the count becomes 2n2 − n. There are m rows. the matrix-matrix product requires O(n3 ) flops. Washington AMATH 352 . but are composed of thousands of processors working together. You may know how “fast” your computer processor is. The i-th element of y is the inner product of the i-th row of A with x and requires 2n − 1 flops. the first term dominates and we say the work required is O(n2 ) as n goes to +∞. We use the terms megaflops. m. Example 152. Computing the matrixmatrix product C = AB will require the computations of mn entries cij .g. See http://www. . . . 2. When using numerical methods we generally care most about how many floating point operations per second or flops it can perform. . Each entry cij is the inner product of two vectors with r components. To get this fast performance it is necessary to develop algorithms that can use all these processors together 86 .

Accuracy is a strong statement that is difficult to attain when the problem f is ill-conditioned. 0≤ f (x) ≤C machine = O( machine ) (144) where C is a constant independent of x. and one of the main challenges in numerical methods for the future is to develop better ways of using parallel processors. The notion of stability is the standard way of characterizing what is possible. even if it is not exact. It corresponds to the numerical analysts’ idea of what it means to get the “right answer”.Spring 2009 to solve a single problem. Washington AMATH 352 .U. The next two definitions are less restrictive. ∃ x ∈ X such that ˜ x−x ˜ = O( x and machine ) ˜ f (x) − f (˜) x f (˜) x = O( machine ) (145) A stable algorithm gives nearly the right answer to nearly the right question. Many algorithms in numerical linear algebra satisfy a condition that is both stronger and simpler than stability. ˜ Definition 154. An algorithm f for a problem f is accurate if we have ˜ f (x) − f (x) ∀x ∈ X. You should know the common prefixes (the bigger ones will become more common in the future!): • kilo: 103 • mega: 106 • giga: 109 • tera: 1012 • peta: 1015 • exa: 1018 7.4 Stability Since most problems are continuous while digital computers are discrete. Consider a mathematical problem as a function f between two linear spaces ˜ (f : X → Y ) and an algorithm as another map f between the same two spaces. numerical algorithms can generally not provide the exact solution. This is not easy. An algorithm f for a problem f is stable if we have ∀x ∈ X. 87 . ˜ Definition 153.

Spring 2009 ˜ Definition 155. the number of flops required by the algorithm is n j−1 n 4m − 1 j=1 i=1 + 2m − 1 + 1 = j=1 [(4m − 1)(j − 1) + 2m]  n  j  − (2m − 1)n (147) = (4m − 1)  j=1 We recall the useful formula n k= k=1 4 L. Instead. 2 N. we will discuss some examples.5 Example Consider the classical Gram-Schmidt algorithm used in the QR factorization (see Algorithm 3) Algorithm 3 Classical Gram-Schmidt for j = 1 to n do v = aj for i = 1 to j − 1 do rij = qT aj i v = v − rij qi end for rjj = v 2 qj = v/rjj end for First. we compute the number of operations. (146) x A backward stable algorithm gives exactly the right answer to nearly the right question. Bau. The computation of rij is an inner product between two vectors in Rm . All together. Washington AMATH 352 . ∃ x ∈ X such that ˜ x−x ˜ = O( x machine ) ˜ and f (x) = f (˜). Trefethen and D. Numerical linear algebra. 88 . which costs 2m − 1 flops. Philadelphia. 7.U. In this course. This is an important research field. The total work is a single inner iteration is 4m − 1 flops. The book of Trefethen and Bau4 gives a nice introduction. 1997. The update of v requires m multiplications and m substractions. An algorithm f for a problem f is backward stable if we have ∀x ∈ X. SIAM. we will not analyze algorithms from the point of view of stability. k(k + 1) .

the modified Gram-Schmidt algorithm will compute a matrix Q such that QT Q − I 1 ≈ 4. there are simple modifications that improve matters. Numerical linear algebra. Another possibility is to apply the classical Gram-Schmidt algorithm twice. consider the Hilbert matrix   1 1/2 1/3 1/4  1/2 1/3 1/4 1/5   H4 =  1  /3 1/4 1/5 1/6  .U. (150) Remark.6 × 10−11 (148) Fortunately. Trefethen and D. 89 . in Algorithm 4. we describe. SIAM. For the Algorithm 4 Modified Gram-Schmidt for i = 1 to n do vi = ai end for for i = 1 to n do rii = vi 2 qi = vi /rii for j = i + 1 to n do rij = qT vj i vj = vj − rij qi end for end for Hilbert matrix H4 .6 × 10−16 . Philadelphia. 1997. To illustrate the unstability. 5 L. which is an advanced algorithm (see Trefethen and Bau5 ). 2 2mn2 − 3 n3 flops. For the Hilbert matrix H4 . It requires. 2 The classical Gram-Schmidt algorithm is numerically unstable because of rounding errors on a computer.Spring 2009 So the total number of flops for the classical Gram-Schmidt algorithm is n(n + 1) − (2m − 1)n ≈ 2mn2 = O(mn2 ). The total number of flops is asymptotically equal to 4mn2 . N. Bau. For example.0 × 10−13 (149) which is better than (148). as described in Algorithm 5. asymptotically. the modified Gram-Schmidt which requires the same amount of flops than the classical Gram-Schmidt algorithm. The Matlab function qr uses the Householder triangularization. applying twice the classical Gram-Schmidt algorithm will compute a matrix Q such that QT Q − I 1 ≈ 3. Washington AMATH 352 . 1/4 1/5 1/6 1/7 (4m − 1) A Matlab implementation of Algorithm 3 generates a matrix Q such that QT Q − I 1 ≈ 3.

Classical G-S 3.6 × 10−11 1.8 × 10+04 9.4 × 10−07 1 H4 H5 H6 H7 H8 Two-Step G-S 3.Spring 2009 Algorithm 5 Two steps of classical Gram-Schmidt for j = 1 to n do v = aj for i = 1 to j − 1 do rij = qT aj i v = v − rij qi end for rjj = v 2 qj = v/rjj end for for j = 1 to n do v = qj for i = 1 to j − 1 do rij = qT qj i v = v − rij qi end for rjj = v 2 qj = v/rjj end for We can compare these four algorithms for a series of Hilbert matrices.1 × 10−15 1.4 × 10+05 2.9 × 10+07 9. Condition Number 2.7 × 10−04 5.2 × 10−16 2.8 × 10+08 3.9 × 10−16 3.3 × 10−15 9.9 × 10−10 5.0 × 10−12 6.2 × 10−02 1.4 × 10+10 H4 H5 H6 H7 H8 Table 2: Condition number for Hilbert matrices using the 1-norm Table 3 describes the resulting error when the dimension of the Hilbert matrix is increased.4 × 10−15 Table 3: Error QT Q − I with different algorithms for Hilbert matrices 90 .5 × 10−16 4.1 × 10−08 1.1 × 10−00 Modified G-S 4. Recall that the Hilbert matrix gets quickly ill-conditioned (see Table 2).2 × 10−16 1.5 × 10−16 1.0 × 10−13 9.U.9 × 10−13 Matlab qr 7.6 × 10−16 3. Washington AMATH 352 .6 × 10−09 3.

7. Washington AMATH 352 . 91 . For example. • realmax returns the largest positive floating point number. • bin2dec converts binary string to decimal integer.6 Useful commands in Matlab Here are a few commands in Matlab useful for this section. • intmin returns the smallest positive integer value. the error is close to machine precision. • intmax returns the largest positive integer value. • eps contains the machine epsilon. • single converts to single precision.U. The modified Gram-Schmidt algorithm computes an approximate orthogonal matrix with an error proportional to the condition number. format long e selects the floating point format with 15 digits for double and 7 digits for single. When applying twice the classical Gram-Schmidt algorithm. • double converts to double precision. • format sets the output format. • dec2bin converts decimal integer to a binary string.Spring 2009 The results highlight that the classical Gram-Schmidt algorithm is unstable. It increases only for H8 but a third step of Gram-Schmidt would fix the problem. Note that the Matlab function returns an orthogonal matrix up to machine precision.

Let z be the difference between x and y: z = x − y ∈ Rn .e. 8. We have found a nontrivial linear combination (since z = 0) of the columns of A that gives the zero vector.U.  xn =  . A linear system Ax = b has at least one solution if b ∈ R(A).  x2 + · · · +  . is exactly the subspace spanned by the columns of A. x2 . .1 Counting the number of solutions When solving linear system is viewed as trying to find the weights of a linear combination (like (153)). .  . This can be written as Ax = b (152) where A is an m×n matrix and b ∈ Rm . xn so that the vector b can be expressed as a linear combination of the columns of A.  . 92 . am1 am2 amn bm In other words. . we see that a linear system Ax = b has at least one solution if (and only if) the vector b lies in the subspace of Rm spanned by the columns of A. we conclude: • If a linear system has more than one solution then rank(A) < n. i.   . From this. am1 x1 + am2 x2 + · · · + amn xn = bm where the aij and bj are some given real numbers. we are trying to find the weights x1 . x2 . The problem is to find a vector x ∈ Rn so that (152) holds. Hence the columns of A are linearly dependent.   .   . .  x1 +  . Washington AMATH 352 . . . We recall that the range of A. Then we have Az = A(x − y) = Ax − Ay = b − b = 0. xn has the general form a11 x1 + a12 x2 + · · · + a1n xn = b1 a21 x1 + a22 x2 + · · · + a2n xn = b2 (151) . . . R(A). We have seen that we can view matrix-vector multiplication as a linear combination of the columns of the matrix A.Spring 2009 8 Linear systems of equations A linear system of m equations in n unknowns x1 . The objective of this Section is to describe general algorithms to solve such linear systems. .  . . So another way to write this system is         b1 a1n a12 a11  b2   a2n   a22   a21          (153)  . Ax = b and also Ay = b where x = y. . . . Might it have more than one solution? Suppose the system has two distinct solutions.

then the system has no solution. All the statements made above still apply in this case. is non trivial. suppose rank(A) < n. In this section. in which case m = n in our previous discussion. Then x + z = x is also a solution. Consider the linear system Ax = b with A ∈ Rm×n and b ∈ Rm . / • If b ∈ R(A) and rank(A) = n. then this system has a unique solution x. • If rank(A) < n then the range of A is a proper subspace of Rn (not the full space) and so Ax = b may or may not have solutions. but now we see that there are basically two cases: • If rank(A) = n then R(A) = Rn and N (A) = {0}. The matrix A is singular or noninvertible. The system Ax = b has exactly one solution for any right hand side b ∈ Rn . we will focus on the case where b ∈ R(A). Now suppose b ∈ R(A) and we have one solution x to Ax = b. depending on whether b ∈ R(A) or not. The linear system 3 2 has exactly one solution 0 4 x1 x2 = 3 14 1 . / Example 157. since the null space of A has dimension n − rank(A) ≥ 1. We can change the right hand side of this system 3 to any other vector and the system will still have exactly one solution. 93 .Spring 2009 Conversely. we say the matrix A is nonsingular or invertible. A special case that often arises in practice is a system of n equations in n unknowns. N (A). since A(x + z) = Ax + Az = b + 0 = b. then this system has infinitely many solutions. so in this case there are infinitely many different solutions to the linear system. So there must be a vector z = 0 for which Az = 0. Then the columns of A are linearly dependent and the null space of A. Washington AMATH 352 . In this case.U. Then: • If b ∈ R(A). • If b ∈ R(A) and rank(A) < n. If it does have solutions then it has infinitely many solutions. We have proved the following theorem: Theorem 156. Also note that if Az = 0 then A(αz) = 0 for any scalar α ∈ R. The Section on least-squares will study the case where b ∈ R(A).

0 3 −3 The difference between any two of these solutions lies in the null space of A. . The linear system  3 2 0 −1 4 6  x1  x2  = x3 3 14 has infinitely many solutions. which is   1 N (A) = span  −5  . . The linear system      2 6 x1 14  −5 3   x2  =  1  1 2 x3 5 1 .  .  8  .2 Easy-to-solve systems We begin by looking for special types of systems that can be easily solved.  . . . On the other hand. .  .       1 2 0 x =  3  . Washington AMATH 352 .  . . 0  . This means that all the nonzero elements of A are on the diagonal of A. .. 8. ..  .  . .    . . A linear system Ax = b is      a11 0 · · · 0 x1 b1  0 a22 0   x 2   b2        . 3 We also know the null space should have dimension 1 since the rank of A is 2 and so dim(N (A)) = 3 − 2 = 1.   . 0 ··· 0 ann xn bn 94 .Spring 2009 Example 158.  .  . . . 0 . The columns of A are linearly independent 2 (since the second is not a scalar multiple of the first). The matrix is of full rank and its null space is equal to {0}. the system      2 6 x1 1  −5 3   x2  =  1  1 2 x3 5 has exactly one solution has no solution.   .  .  =  . Suppose that the n × n matrix A has a diagonal structure.   ..  −2  . .U. Example 159.   .

The solution is     b1 /a11 x1  x2    (b2 − a21 x1 )/a22     .  . .   .  .    . This means that all the nonzero elements of A are below the main diagonal of A.  .U.  .  . . . .    . Its determinant is n det A = a11 × a22 × · · · × ann = i=1 aii ..   . then the matrix is singular.   .  .    . . Then we obtain x1 from the first equation. 0  . x2 .  .  .     . The total number of operations is n(n + 2) = O(n2 ). .. If aii = 0 and bi = 0 for some index i. .Spring 2009 The system collapses to n simple  x1  x2   . then no solution exists.    .   . Suppose that the n × n matrix A has a lower triangular structure. .   . = .  =  . . we can solve for x2 . bn /ann The computation requires n floating operations. Example 160.    .  . Washington AMATH 352 . A formal algorithm for this case is called forward substitution. n−1 xn (bn − j=1 anj xj )/ann If aii = 0 for some index i. .  . . If aii = 0 and bi = 0 for some index i.  .  . = . . xn equations and the solution is    b1 /a11   b2 /a22        .  .  . an1 an2 · · · ann xn bn Assume that aii = 0 for all indices i. We proceed in the same way obtaining x1 . Note that for a diagonal matrix.  . With the known value of x1 substituted in the second equation. The linear system 3 2 0 4 x1 x2 95 = 3 14 . .       . its determinant is n det A = a11 × a22 × · · · × ann = i=1 aii . · · · . xn one at a time and in this order. . then xi can be any real number. A linear system Ax = b is      a11 0 ··· 0 x1 b1  a21 a22 0   x2   b2        .

we can solve for xn−1 . . But the remaining question is how to determine beforehand the permutation. Then we obtain xn from the last equation.. . Gaussian elimination transforms a full linear system into an upper triangular one by applying simple linear transformations.U. then the matrix is singular. xn−1 . the exact solution is 3 Suppose that the n × n matrix A has an upper triangular structure.  . With the known value of xn substituted in the next-to-last equation.     . we can get a lower triangular system:      a31 0 0 x1 b3  a11 a12 0   x2  =  b1  . Another simple type of system than can be easily ideas. 96 .   .  . We will discuss a general approach in the next subsection. So 1 . . a21 a22 a23 x3 b2 Then we can solve the re-ordered system in the same manner as previously. The total number of operations is still O(n2 ). . .3 Gaussian elimination Gaussian elimination is the simplest way to solve a linear system by hand and also the standard way for solving linear systems on computers. A formal algorithm for this case is called backward substitution. A linear system Ax = b is      b1 x1 a11 a12 · · · a1n  0 a22 a2n   x2   b2        . 8.  .Spring 2009 is lower triangular.  .  . So we obtain x1 = 3/3 = 1 and 4x2 = 14 − 2 × 1 = 12. . · · · .  =  .   .   .   . .  . We proceed in the same way obtaining xn ..  .  . Washington AMATH 352 . Its determinant is n det A = a11 × a22 × · · · × ann = i=1 aii . bn xn 0 0 ··· 0 ann Assume that aii = 0 for all indices i. 0 . solved using the same in a triangular system. If aii = 0 for some index i. x1 one at a time and in this order. This means that all the nonzero elements of A are below the main diagonal of A.   . is a system obtained by permuting the equations Consider the system     a11 a12 0 x1 b1  a21 a22 a23   x2  =  b2 a31 0 0 x3 b3 If we simply reorder these equations.

5  8 looks like 1 3 7 7 1 3 9 9   2 0 1   0 = 5   0 0 8 1 1 3 4 1 1 5 6  0 1  . 2  2 Now we need to compute the matrix (L3 L2 L1 )−1 = L−1 L−1 L−1 . and three times the first row from the fourth. A lower triangular matrix with diagonal entries equal to 1 can usually be found such that L−1 A = U. A = LU. where U is an upper triangular matrix.e. we discuss some examples. i. Washington AMATH 352 . 0  1 L−1 1 The inverses of L2 and L3 are obtained in a similar fashion.Spring 2009 Assume that A ∈ Rn×n .U. Before giving the general algorithm. Example 161. 0 0 2 4 0 4 6 8 0 −4 0 1 Here we have subtracted three times the second row from times the second row from the fourth. It turns out 1 2 3 that the inverse of L1 is a simple modification of the matrix L1 where the terms below the diagonal have been negated. The second step is      2 1 1 0 2 1 1 0 1 0 0 0  0 1 0 0  0 1 1 1   0 1 1 1      L2 L1 A =   0 −3 1 0   0 3 5 5  =  0 0 2 2  . 5  8 We have subtracted twice the first row from the second. Consider the matrix  2  4 A=  8 6 The first step of Gaussian elimination   2 1 0 0 0  −2 1 0 0   4  L1 A =   −4 0 1 0   8 6 −3 0 0 1 1 3 7 7 1 3 9 9  0 1  . Note that this formula for the inverse is due to the special structure of the matrices Li (that are equal to the identity matrix plus an outer product with only one nonzero 97 . The final step is     2 2 1 1 0 1 0 0 0  0 1 0 0  0 1 1 1   0 =   L3 L2 L1 A =  0 0 1 0  0 0 2 2   0 0 0 2 4 0 0 0 −1 1 the third and four 1 1 0 0 1 1 2 0  0 1   = U. four times the first row from the third. Thus we obtain an LU factorization of A. 1  −2 =  −4 −3  −1  0 0 0 1  2 1 0 0   =  4 0 1 0  0 0 1 3 0 1 0 0 0 0 1 0  0 0  .

The special structure of these matrix multiplications. 1 . we get   1 0 0 0  2 1 0 0   L−1 L−1 L−1 =  1 2 3  4 3 1 0  3 4 0 1 and 1  2 A = LU =   4 3  0 1 3 4 0 0 1 1  2 0 0  0  0  0 0 1 matrices result also in simple matrix0 1 3 4 0 0 1 0   1 0 0   2 = 0   4 3 1 0 1 3 4 0 0 1 0  0 0  . 1  7 −6  . 5  8 Example 162.Spring 2009 column).U. Consider the matrix  1 A= 2 3 4 5 6  7 8 . Finally. i.e. 10 The Gaussian elimination performs first     1 4 7 1 1 0 0 L1 A =  −2 1 0   2 5 8  =  0 3 6 10 0 −3 0 1 Then the next step gives  1 0 L2 L1 A =  0 1 0 −2 Finally. we simply insert the nonzero places. −11  0 1 0  0 1 0 4 −3 −6   7 1 −6  =  0 −11 0 4 −3 0  7 −6  . 0  1 subdiagonal entries in the appropriate 1 0 0 0 0 1 0 0 0 0 1 1   1 0 0   2 = 0   4 3 1 0 1 3 4 0 0 1 1  0 0   0  1 1 1 0 0 1 1 2 0   2 0 1   4 = 2   8 6 2 1 3 7 7 1 3 9 9  0 1  .   1 1 0 0 0  2 1 0 0  0 −1 −1  L1 L2 =   4 0 1 0  0 0 3 0 0 1 namely. we write 1 A= 2 3  0 1 2  0 1 0  0 1 0 98 4 −3 0 4 −3 −6  7 −6  . Washington AMATH 352 .

U. Washington

AMATH 352 - Spring 2009

Before describing the Gaussian elimination, we recall that linear combinations of matrix rows or columns and permutations of rows and columns can be obtained by matrix-matrix multiplication. For example, the product of matrices      1 0 0 2 3 4 2 3 4  a 1 0   5 6 7  =  5 + 2a 6 + 3a 7 + 4a  (154) b 0 1 8 9 10 8 + 2b 9 + 3b 10 + 4b corresponds to adding a times the first row to the   1 0 0 2  0 1 0  5 0 c 1 8 times the first row to the second row and adding b third row. The product    3 4 2 3 4  6 7 = 5 6 7 (155) 9 10 8 + 5c 9 + 6c 10 + 7c second row to the   3 4 2a 6 7  =  5b 9 10 8c third row. The product  3a 4a 6b 7b  (156) 9c 10c

corresponds to adding c times the   2 a 0 0  0 b 0  5 8 0 0 c

corresponds to row scalings. The column operations are easily obtained by transpose. For example, the product of matrices      2 3 4 1 α β 2 3 + 2α 4 + 2β  5 6 7   0 1 0  =  5 6 + 5α 7 + 5β  (157) 8 9 10 0 0 1 8 9 + 8α 10 + 8β corresponds to adding α times the first column to the second column and adding β times the first column to the third column. The product      2 3 4 + 3γ 1 0 0 2 3 4  5 6 7   0 1 γ  =  5 6 7 + 6γ  (158) 8 9 10 + 9γ 0 0 1 8 9 10 corresponds to adding product  2 3  5 6 8 9 γ times the second column to the third column. The  4 α 7  0 10 0 0 β 0   0 2α 0  =  5α γ 8α 3β 6β 9β  4γ 7γ  10γ (159)

corresponds to column scaling. The general Gaussian elimination goes as follows. Consider a matrix Rn×n . If a11 = 0, then we write     1 0 ··· 0 a11 a12 · · · a1n a11 a12 · · ·  −a21 /a11 1 · · · 0   a21 a22 · · · a2n   0 a22 · · · ˜      . .  . . = . .. . .   .  . .  . . . . . . −an1 /a11 0 1 an1 an2 ann 0 an2 ˜ 99

A∈ a1n a2n ˜ . . . ann ˜    . 

U. Washington

AMATH 352 - Spring 2009

We have eliminated all the entries below the diagonal in the first column. If a22 = 0, then we can repeat the process and eliminate all the entries below the ˜ diagonal in the second column. Suppose that we have build p−1 lower triangular matrices with diagonal entries equal to 1 such that the matrix Lp−1 . . . L2 L1 A has the first p − 1 columns with zero entries under the main diagonal, i.e.  a11 ··· .. .. . . .. . app ˆ . . . anp ˆ ... ... apn ˆ . . . ann ˆ a1n       .     

  0   .  . . Lp−1 . . . L2 L1 A =  .   .  .  .  . . 0 Then, if app = 0, we have ˆ

0 . . . ... 0

                  

1 0 . . . . . . . . . . . . . . . .. .. . . .. .

···

0

   a 11    0   .  .  .  .  .  .  .  .  .   0  ··· .. .. . . .. . app ˆ . . . anp ˆ ... ... apn ˆ . . . ann ˆ a1n            

0 . . . . . . . . .

1
a ˆ − p+1,p app ˆ

..

. .. .. . ..

. . . . . .

0 . . . 0  a11

0 . . . ... 0

0

...

0

a ˆ − an,p ˆpp

. ... .. .. . .

. 0

1 ··· a1n               

  0   .  .  .  . = .  .  .  .  .  .  . . 0

..

. app ˆ 0 . . . 0 ap+1,p+1 ˇ . . . ann ˇ ... ... ap+1,p+1 ˇ . . . ann ˇ

...

0 . . . . . . 0

By building n − 1 lower triangular matrices with diagonal entries equal to 1, the resulting matrix is upper triangular. 100

U. Washington

AMATH 352 - Spring 2009

When the lower triangular  1  .  0 ..   . ..  . .  .  .  .  . Lp =  .  .  .  .  .  .  .  .  . 0 its inverse is           =         1 0 . . . . . . . . . . . . . . . .. .. . . ...

matrix Lp is ··· .. . 1
a ˆ − p+1,p app ˆ

0

          ,        

0 . . . . . . . . .

..

. .. .. . ..

. . . . . .

0 . . . 0

0

a ˆ − an,p ˆpp

. ...

. 0

1

··· .. . 1
ap+1,p ˆ app ˆ

0

          .        

L−1 p

0 . . . . . . . . .

..

. .. .. . ..

. . . . . .

0 . . . 0

0

...

0

an,p ˆ app ˆ

. ...

. 0

1

The product L−1 L−1 is just the unit lower triangular matrix with the entries of p p+1 both L−1 and L−1 inserted in their usual places below the diagonal. When we p p+1 take the product of all of these matrices to form L, we have the same convenient property everywhere below the diagonal   1 0 ··· 0  l21  1    l31 l32  −1 −1 1 L = L1 . . . Ln−1 =  .  .  .. ..  . . . . 0  lm1 lm2 . . . lm,m−1 1 In practical Gauss elimination, the matrices Lk are never formed. The coefficients ljk are computed and stored directly into L. The complete algorithm is described in Algorithm 6. Counting the number of operations, we have  
n−1 n n n−1 n n−1

1 +
k=1 i=k+1 j=k

2 =
k=1 i=k+1

(2(n − k) + 3) =
k=1

(2(n − k) + 3) (n − k).

101

Remark. 6 2 6 2 The total number of floating point operations is ≈ 3 n3 flops. Remark. When a symmetric matrix A has a Cholesky factorization. So we focus on square matrices. This factorization requires ≈ n floating point operations.e.j end for end for end for After changing variables. it satisfies ∀x ∈ Rn \ {0} . Washington AMATH 352 . If the matrix A is symmetric.j = ui. then we can set D = diag(U) and MT = D−1 U and we write A = LDMT . i. where the matrix D is diagonal and both matrices L and M are lower triangular matrices with unit entries on the diagonal. Remark. We described the Gaussian elimination with an emphasis on zeroing columns. The Crout factorization takes the matrix form ˜˜ A = LU. then we can set R = D1/2 LT and we obtain A = RT R. we get n−1 2k 2 + 3k = 2 k =1 n(n + 1)(2n + 1) n(n − 1) 4n3 +3 ≈ + O(n2 ). which is called the Cholesky factorization.j − lik uk.U. xT Ax > 0. ˜ ˜ where L is a lower triangular matrix and U is an upper triangular matrix with unit entries on the diagonal.Spring 2009 Algorithm 6 Gaussian elimination without pivoting Set U = A and L = I for k = 1 to n − 1 do for i = k + 1 to n do lik = uik /ukk for j = k to n do ui. the LDM factorization becomes A = LDLT . We could just as easily introduce zeros in rows. If the matrix A has an LU factorization. The Gaussian elimination can also be applied to rectangular matrices but this is rarely done in practice. 102 3 . Remark. it is symmetric positive definitive. When all the entries 3 on the diagonal of D are positive.

Compute x = U−1 y (backward substitution) (Cost ≈ n2 ) This algorithm requires less floating point operations than a QR factorization.Spring 2009 To solve a linear system Ax = b by LU factorization. 0 1 1 0 1 2 c1 c2 = 2 5 1 1 1 3 = 1 1 0 1 1 0 1 2 . The linear system associated with these equations is c1 + c2 c1 + 3c2 Consider the matrix A= We have 1 −1 The LU factorization of A is A= The coefficients satisfy 1 1 and 1 0 Finally. 1 1 1 3 . Washington AMATH 352 .U. = = 2 5 103 . 1 2 c1 c2 = 1 −1 0 1 2 5 = 2 3 . we obtain c1 c2 = 1/2 3/2 . Example 163. 0 1 1 1 1 3 = 1 0 1 2 . The forward and backward substitutions require then O(n2 ) floating point operations per right hand side. Note that when you want to solve several linear systems with the same matrix A and different right hand side vectors b. Compute a LU factorization A = LU (Cost ≈ 3 n3 = O(n3 )) 2. the following steps are required: 2 1. Determine the polynomial p(t) = c1 + c2 t that satisfies: p(1) = 2 p(3) = 5. Compute y = L−1 b (forward substitution) (Cost ≈ n2 ) 3. the LU factorization should be computed only once.

4 Now.6 . p(5) = 3.4t2 .2 − 3 × 0.8 − 4.4) = 2 (0. t.8 = 1.4 − 0. c2 . the linear system for the coefficients d1 .2.8.4(t − 2)(t − 4).2 (160) (161) We eliminate the parameter c1 in the last two equations to get c1 + 2c2 + 4c3 2c2 + 12c3 3c2 + 21c3 = 0.4 1 1 2 (0. = = = 0. We want to find a quadratic polynomial p(t) = c1 + c2 t + c3 t2 such that p(2) = 0.2 p(t) = 3. 1.8) = 0. we get d1 d2 d3 So the polynomial is p(t) = 0.6 − 0.8) = −2 . 2 (1. p(4) = 1. 3. = 2. t2 ).8 − 2 × 2 + 4 × 0. Using the values (161).8 1 . we could choose the basis (1.4 1 3 (3.4 = 3.8) = 0.2 − 2t + 0. and c3 : c1 + 2c2 + 4c3 c1 + 4c2 + 16c3 c1 + 5c2 + 25c3 = = = 0.U.2 So the polynomial is Instead of using the basis (1.8 − 12 × 0. t − 2. 0. and d3 is d1 d1 + 2d2 d1 + 3d2 + 3d3 Solving by forward substitution.8 = 0. We write a linear system for the coefficients c1 . we eliminate the coefficient c2 in the last equation c1 + 2c2 + 8c3 2c2 + 12c3 3c3 With backward substitution.8 . Washington AMATH 352 .4 = 0. d2 . we get c3 c2 c1 = = = 0.6. = 3.4(t − 2) + 0.8 1.Spring 2009 Example 164.8 0.8 .6 . 104 = = = 0.2 .8 + 0. (t − 2)(t − 4)) and write p(t) = d1 + d2 (t − 2) + d3 (t − 2)(t − 4).

we have  0  1 0 Column permutations  2  5 8 1 0 0  0 2 3 0  5 6 1 8 9   4 5 7 = 2 10 8 6 3 9  7 4 . The following factors are produced L= 1 1020 0 1 U= 10−20 0 1 1 − 1020 .U. The Gaussian elimination does not fail now. This matrix has full rank and is well-conditioned.Spring 2009 8. the product is 10−20 1 ˜˜ LU = . Example 165. Gaussian elimination as presented so far is unusable for solving general linear systems because it is not backward stable.  3 4 0 1 6 7  1 0 9 10 0 0 for example   0 3 2 0 = 6 5 1 9 8  4 7  10 (163) 105 . For example. Consider the matrix A= 0 1 1 1 . For certain matrices. Gaussian elimination fails entirely because it tries to divide by zero. Gaussian elimination fails at the first step. the numbers will be rounded to the nearest floating point number. However. Nevertheless. Washington AMATH 352 . This operation is called pivoting. The floating point matrices will be 10−20 1 1 0 ˜ ˜ U= . L= 20 0 −1020 10 1 These matrices are relatively close to the exact matrices. 1 0 The instability can be controlled by permuting the order of rows of the matrix being operated on. with 16 digits of accuracy.4 Gaussian elimination with pivoting Unfortunately. In floating point arithmetic. Row permutations can also be written in terms of matrix multiplication. 10 (162) are also possible. Suppose we slightly perturb the matrix A= 10−20 1 1 1 .

n of the working matrix X in order to introduce zeros under the diagonal. we have  0  1 0 1 0 0  0 2 0  5 1 8 3 6 9  4 0 7  1 10 0 1 0 0 −1  0 5 0  = 2 1 8 6 3 9  7 4  10  6 = 3 9 0 1 0 5 2 8 1 0 0  0 0  1  7 4  10 8. it is common to pick as pivot the largest number among the entries below the diagonal.U. multiples of row k are subtracted from rows k + 1.        −→      So we are free to choose as pivot any entry in the k-th column below the diagonal. × × × ×       0 ∗ ∗ ∗  × × × ×  0 ∗ ∗ ∗ × × × × However. Washington AMATH 352 .e. we have     × × × × × × × × × ×   xkk × × ×  xkk × × ×       −→   0 ∗ ∗ ∗ . 106 . .       × × × × xik × × × × × × × × × × × × × × × ×   × × 0 0 xik 0 × × × ∗ ∗ ∗ ∗ ∗ ∗ × × × ∗ ∗ ∗    . i. we could easily introduce zeros in column k by adding multiples of some row i with k ≤ i ≤ n. In practice and for numerical stability. For example. This strategy is called partial pivoting (as opposed to complete pivoting described in Section 8. . For example. there is no particular reason why the k-th row must be chosen for the elimination. The entry xkk plays a special role and is called the pivot. they satisfy  0  1 0 1 0 0 −1  0 0 0  = 1 1 0 1 0 0 T  0 0 0  = 1 1 0 1 0 0  0 0 . . as long as it is nonzero.Spring 2009 Note that permutation matrices are identity matrices with permuted rows.4. Schematically. They are orthogonal matrices.2).4. 1 (164) We can also combine rows and columns permutations. .1 Partial pivoting At step k of Gaussian elimination. The possibility that an entry xkk might be zero implies that some flexibility in choosing the pivot may sometimes be necessary.

 0 0 0 107 .  We eliminate the second column entries (left multiplication by L2 )     8 7 9 5 8 7 9 5 1 0 0 0 7 9 17  9 17  0 7  0 1 0 0  0 4 4 4  4 4 4    4  0 3 1 0   0 −3 −5 −5  =  0 0 −2 7 4 4 4 7 7 2 1 6 3 3 2 0 7 0 1 0 −2 −2 −2 0 0 −7 −7 The third and fourth rows   1 0 0 0  0 1 0 0     0 0 0 1  0 0 1 0 The final step is  1 0 0  0 1 0   0 0 1 0 0 −1 3 are now interchanged (left   8 7 9 5 8 7 9 17   0 4 4 4 = 0 4   0 0 0 −2 7 7 6 2 0 0 −7 −7 0   .  first column (left multiplication by   8 7 9 5 7 9 5 3 3 3 1   0 −1 −2 −3 2 2 = 5 1 1 0   0 −3 −4 −5 4 4 7 9 17 7 9 8 0 4 4 4 rows 2 and 4 (left multiplication by P2 )    8 7 9 5 8 7 0 7 3 1   0 −1 −3 −2   0 2 2 4    3 5 5 = 0   0 −4 −4 −4 0 −3 4 7 9 17 0 0 0 −1 4 4 4 2 9 9 4 5 −4 3 −2 5 17 4 −5 4 −3 2   .U.Spring 2009 Example 166. 0  8 L1 )   . Consider the matrix  2  4 A=  8 6 With partial pivoting. 5  8 is to interchange the first and third row 1 3 9 9   0 8 1   4 = 5   2 8 6 7 3 1 7 9 3 1 9  5 1  . Washington AMATH 352 . 6 2 0 −7 −7  2 4 0 −7 7  8 0 0  0  0  0 1 0 7 7 4 9 9 4 6 −7 2 −7 5 17 4 −2 7 4 7  0 0 8   0 =   0 0  7 7 4 9 9 4 −6 7 5 17 4 2 −7 2 3   . the first step (left-multiplication by P1 ):   2 1 0 0 1 0  0 1 0 0  4 3    1 0 0 0  8 7 6 7 0 0 0 1 Then we eliminate  1 0  −1 1 2  1  − 0 4 −3 0 4 Now we permute  1 0 0  0 0 0   0 0 1 0 1 0 values in the  0 0 8 0 0  4  1 0  2 6 0 1 1 3 7 7 1 3 9 9  0 1  .  multiplication by P3 )  7 9 5 7 9 17  4 4 4 .

(165) where P is a permutation matrix. This algorithm requires the same number of floating point operations. 2 3 1  −3 L1 =  4  −1 2 1 −4   0 0 0 1 0 0  . 3 108 . Indeed. 3 2 3 The matrices L2 and L1 satisfy   1 0 0 0  0 1 0 0   L2 =   0 2 1 0  7 0 3 0 1 7 L2 = P3 L2 P−1 . Washington AMATH 352 . However. 3 L1 = P3 P2 L1 P−1 P−1 .Spring 2009 The elimination can be written in matrix form L3 P3 L2 P2 L1 P1 A = U. The complete algorithm for Gaussian elimination with partial pivoting is described in Algorithm 7. where we have L3 = L3 . and O(n2 ) comparisons to find the pivots. just as in Gaussian elimination without pivoting. Writing L = (L3 L2 L1 )−1 and P = P3 P2 P1 . 0 1 0  0 0 1 The product of the matrices L3 L2 L1 is a lower triangular matrix with unit entries on the diagonal and it is easily invertible by negating the subdiagonal entries. the factorization provided by Gaussian elimination with partial pivoting can be written in the form PA = LU. and U is upper triangular. we have PA = LU or  0  0   0 1 0 0 1 0 1 0 0 0  2 1 0 1  4 3  0  8 7 6 7 0 1 3 9 9   1 0 3 1   4 = 1   5 2 1 8 4 0 1 2 −7 3 −7 0 0 1 1 −3  0 8 0  0  0  0 1 0 7 7 4 9 9 4 −6 7 5 17 4 −2 7 2 3   . L is a lower triangular matrix with unit entries on the diagonal and lower-triangular entries smaller or equal to 1 in magnitude. ≈ 2 n3 .U. The multiplying matrix on the left side of A does not look like a lower triangular matrix. we can write L3 P3 L2 P2 L1 P1 = L3 L2 L1 P3 P2 P1 . we have L3 P3 L2 P2 L1 P1 = L3 P3 L2 (P−1 P3 )P2 L1 (P−1 P−1 P3 P2 )P1 .  0 0 0 For a general n × n matrix.

r = tmp end for for j = k + 1 to n do ljk = ujk /ukk for r = k to n do uj.U.r uk. L = I.r = tmp end for for r = 1 to k − 1 do tmp = lk.Spring 2009 Algorithm 7 Gaussian elimination with partial pivoting Set U = A.r end for end for end for 109 .r ui. Washington AMATH 352 .r − ljk uk. and P = I for k = 1 to n − 1 do Select i ≥ k to maximize |uik | for r = k to n do tmp = uk.r = uj.r pk.r = tmp end for for r = 1 to n do tmp = pk.r lk.r pi.r = pi.r = ui.r = li.r li.

In practice.6 × 109 s ≈ 77000 days ≈ 210 years Table 4: Time estimates to solve Ax = b by Gaussian elimination on a 100 MFlops computer We assume that all the problems fit in the memory and that the initial matrix is dense (all the entries could be nonzero). Washington AMATH 352 . Formally.6 × 105 6. we say that A has upper bandwidth q if aij = 0 whenever j > i + q and lower bandwidth p if aij = 0 whenever i > j + p.6 × 1014 6. when A is a diagonal matrix.005 seconds rather than 210 years! Banded systems are another examples of matrices for which the Gaussian elimination requires only O(n) flops. In matrix form. many practical systems have some structure or involve sparse matrices (where most elements are zero).5 Banded systems The Gaussian elimination requires.6 × 1011 6.6 × 10−3 s 6. Table 4 lists estimates of the time required to compute the LU factorization and solve a linear system on a 100 MFlops computer.Spring 2009 8. Gaussian elimination with complete pivoting takes the form PAQ = LU where P and Q are permutation matrices. If A has upper bandwidth q and lower bandwidth p.11 min 6. n 10 100 1000 10000 100000 1000000 Flops 6. requires only O(n) flops and this takes only 0. this is rarely done because the improvement in stability is marginal. 2 n3 floating point opera3 tions. Theorem 167. the selection of pivots is choosing the largest in the whole matrix.6 × 100 s 6. and U is an upper triangular matrix. Luckily. 110 .6 × 10−6 s 6.6 × 103 s ≈ 111. approximately.U. An example of a sparse matrix is a diagonal matrix. L is a lower triangular matrix with unit entries on the diagonal.6 × 102 6.6 × 108 6.4. Suppose A ∈ Rn×n has an LU factorization A = LU. 8. then U has upper bandwidth q and L has lower bandwidth p.2 Complete pivoting In complete pivoting. A matrix is banded whenever the equations can be ordered so that each unknown xi appears in only a few equations in a “neighborhood” of the i-th equation.6 × 106 s ≈ 1850 h ≈ 77 days 6.6 × 1017 Time 6. Solving Ax = b. It requires a significant amount of time because the comparisons are now of order O(n3 ).

The forward substitution requires about 2np flops and the backward substitution about 2nq flops. Example 168. Note that..     .  . . Consider a n×n symmetric. and K represents admittances. . x represents branch currents. Then the Gaussian elimination can be specialized and will involve only 2npq flops.k = = = d1 lk−1 dk−1 2 dk + lk−1 dk−1 = dk + lk−1 ak. 1996. Figure 9: Mass-spring system Newton’s law f = ma applied to this system yields mi xi ¨ = ki (xi−1 − xi ) +fi or M¨ = −Kx + f x where the matrix M is diagonal and the matrix K is tridiagonal. Electrical engineers analyzing circuits arrive at an analogous equation by applying Kirchoff’s and related laws instead of Newton’s law. Setting   1 .Spring 2009 The proof is given in Golub and Van Loan. L=  . pivoting will destroy the band structure.. In this case. the equation A = LDLT gives a11 ak.     . dn ). Consider the mass spring system in Figure 9. positive definite matrix A.k−1 The complete algorithm to compute the factorization and to solve a linear system 8n flops. 111 (force on mass i from spring i) (force on mass i from spring i + 1) +ki+1 (xi+1 − xi ) (external force on mass i) . . 0  .. Washington AMATH 352 . .U. Example 169. M is called the mass matrix and K is called the stiffness matrix.k−1 ak. tridiagonal. however. M represents inductances. “Matrix Computations”.. 1 0 . .   .. The John Hopkins University Press. ..   l1 1 . ln−1 1 and D = diag(d1. .

U. 112 .P]=lu(A) returns an upper triangular matrix in U. Washington AMATH 352 . triangular. look at “help mldivide”. use the Cholesky or the LDLT factorization – Use the LU factorization. – It checks whether A is diagonal.U. • inv(A) computes the inverse of matrix A when it exists. – If A is not square. • If the matrix A is square. and a permutation matrix P. A\b is roughly the same as inv(A)*b. except it is more efficiently. • [L. use the QR factorization – For further information. a lower triangular matrix L with a unit diagonal.Spring 2009 8. – If A is symmetric. a permutation of a triangular. banded.6 Useful commands in Matlab Here are a few commands in Matlab useful for this section. such that LU = PA.

If x solves the system exactly then r = 0. Remark. we seek only to minimize the sum of the squares of the residuals. In this case. 113 . then we might associate different weights. If we use the 2-norm to measure the residual then m b − Ax 2 = i=1 (bi − (Ax)i )2 (166) where (Ax)i means the i-th component of the vector Ax ∈ Rm . If some observations are more important or more accurate than others. ∀x ∈ Rn . Instead of solving the equations exactly. with different observations and minimize m r 2 ω = i=1 2 ωi ri . The “best” solution will depend on what norm we use. this will be a nonnegative real number. So we can equally well minimize m b − Ax 2 2 = i=1 (bi − (Ax)i )2 (167) The vector x∗ that minimizes (167) will be the same as the vector that minimizes (166). For any particular x ∈ Rn . we are trying to find the vector x that minimizes the sum of the squares of the residuals. We say that a particular vector x∗ ∈ Rn is the “best approximation” in some norm if b − Ax∗ ≤ b − Ax . r = b − Ax . Otherwise we can use some vector norm to measure how small the residual is. There are many situations where this arises but we still want to find a vector x ∈ Rn that approximately solves the system.1 Residuals and norms Suppose the linear system Ax = b with A ∈ Rm×n has no solution because b is not in the range of A. The residual vector r ∈ Rm is defined by r = b − Ax.Spring 2009 9 Least squares problem The term least squares describes a frequently used approach to solving overdetermined or inexactly specified systems of equations in an approximate sense. and the vector x∗ that minimizes this is called the least squares solution to the linear system. 9. ωj . Washington AMATH 352 .U. Minimizing the sum of squares or the square root of the sum of squares gives the same minimizer x∗ since the square root is an increasing function.

we minimize the largest residual: r ∞ = max |ri | . but it is computationally more difficult than the least squares problem.) 9. 5 5 8 5 2  Example 171. i This is also known as a Chebyshev fit and can be reformulated as a linear programming problem. we could add a nontrivial element of the null space of A to x∗ and get another solution. 1 3 8 114 . Note that. (If rank(A) < n. With the ∞-norm. Remark. we minimize the sum of the absolute values of the residuals: m r 1 = i=1 |ri | . Washington AMATH 352 . Using the backslash operator from Matlab.     1 6 Example 170. Chebyshev fits are frequently used in the design of digital filtes and in the development of approximations for use in mathematical function libraries.U. we will see what’s going on under the hood when you do this. the value of x is the one that makes the vector A[x] − b perpendicular to the column vector A T      T   1 1 6 1 −4  2   2  × 2 −  7  =  2   −3  = −4 − 6 + 10 = 0. Consider the matrix A and the vector b     −1 1 −2 A =  0 2  and b =  6  . it is easy to solve a linear least squares problem using the backslash operator. geometrically. We will limit ourselves to least squares problem in this class. We will generally assume m > n and that rank(A) = n so that the columns of A are linearly independent. In this lecture. Consider the matrix A =  2  and the vector b =  7 . This problem can be reformulated as a linear programming problem. In this case it can be shown that the least squares solution is unique.Spring 2009 With the 1-norm. we get x = 2. measured in the 2-norm. 5 8 We want to find the value of x that minimizes the length of the vector A[x] − b.2 Examples of least squares problem First we present some examples of least squares problems. The resulting solutions are less sensitive the presence of spurious data points or outliers. In matlab.

and the vector y does not lie in the column space of the matrix A. c2 .Spring 2009 Using the backslash operator from Matlab. we get   4. Using the backslash operator in Matlab. 1 115 . we get x = vector is   −2 −1 r = b − Ax =  6  −  0 8 1   1 2  3  3 2 3 2 . . Find the quadratic function p(t) = c1 + c2 t + c3 t2 that gives a least squares fit to the data (ti. the matrix A and the input    5 1 0  4   1 1   y=  6  A= 1 2 9 1 3 vector y are  0 1   4  9 Note that if we only required 3 of these conditions we would get a 3 × 3 square system with a unique solution.9 c =  −1. corresponding to a parabola going through those three points.U. which has the form      1 t1 t2  y1 1  1 t2 t2  c1   2     y2   1 t3 t2  c2 =  y3  . 8 9 −1 Note that.6  . we get 4 equations in the 3 unknowns c1 . and c3 . yi ) from the table below i 1 2 3 4 ti 0 1 2 3 yi 5 4 6 9 If we try to require p(ti ) = yi . Example 172. The residual      −2 −1 −1 =  6  −  4  =  2 . Washington AMATH 352 . now. the residual vector is orthogonal to the column vectors of A  −1  0 1 T   1 −1 2   2 = 3 −1  −1 1 0 1 2 3  −1  2 = −1 1−1 −1 + 4 − 3 0 0 = . 3 c3 1 t4 t2 y4 4 From the data above. But there is no single quadratic function that goes through all four points given in the table above.

and (169) states that the residual vector is orthogonal to all columns of A. (168) These equations are called the normal equations.U. Consider the 2 × 1 least squares problem 2 1 x= 3 1 .3 Normal equations If we want a least squares solution to a linear system Ax = b where A ∈ Rm×n with m ≥ n and rank(A) = n (the columns of A are linearly independent). It can be shown that if rank(A) = n then rank(AT A) = n as well. which is the line 2 in 2-space through the origin and the point . and hence the residual is orthogonal to R(A). Washington AMATH 352 . so this matrix is nonsingular and the system has a unique solution. There are several ways to derive the normal equations.Spring 2009 Figure 10 plots the data and the least-squares fit Figure 10: Least-squares fit with a quadratic function 9. the space spanned by the columns of A. Example 173. The solution to the least 1 116 . The best way to remember where these equations come from is to rewrite (169) as AT (Ax − b) = 0. then the unique least squares solution can be found by solving the square n × n system AT Ax = AT b. (170) The right hand side b does not lie in the column space of A. (169) Recall that the vector r = b − Ax is the residual.

Washington AMATH 352 . To find r . For example. the function r is not smooth and we can not find the minimum by setting the derivative equal to 0. Note that because r(x) is quadratic in the components of x. T This function is a smooth function of x and is quadratic in the components of x. it is not so easy. This is why minimizing the 2-norm of residuals is easy to solve using linear algebra. we can set r (x) = 0. 117 . In other norms.U. setting the derivatives to 0 gives a linear systems of equations. To find the minimum for r. Figure 11: Solution to the least squares problem (170) Another way to derive the normal equations is to minimize the function r(x) = b − Ax 2 2 = (b − Ax) (b − Ax) .Spring 2009 squares problem is the scalar x for which (in Euclidean distance) to the point 2x x (a point on the line) is closest 3 . So we compute r(x + y) r(x + y) r(x + y) r(x + y) = = = = (b − Ax − Ay)T (b − Ax − Ay) (b − Ax)T (b − Ax − Ay) − yT AT (b − Ax − Ay) (b − Ax)T (b − Ax) − (b − Ax)T Ay − (Ay)T (b − Ax) + (Ay)T Ay r(x) − 2yT AT (b − Ax) + (Ay)T Ay So we obtain r (x) = 2AT (Ax − b) = 2 AT Ax − AT b which gives the normal equation when setting it to 0. with the 1-norm. we recall that the derivative is the first linear term in the Taylor expansion. It is obvious from Figure 11 that 1 this is the point for which the residual is orthogonal to the line.

ˆ ˆ Solve the upper triangular system Rx = QT b. 9.2 QR factorization machine ). One constructs a factorization A = QR. ing point operations is.3.U. Let A ∈ Rm×n with m ≥ n. It constructs a factorization AT A = RT R. The matrix AT A is square.3. So we write the T T normal equations A Ax = A b and we have ˆ ˆ ˆˆ ˆ ˆ ˆ ˆ ˆ ˆ ˆ ˆ RT QT QRx = RT QT b ⇒ RT Rx = RT QT b ⇒ Rx = QT b.3. Compute the Cholesky factorization AT A = RT R. Then we can write ˆˆ A = UΣVT . ˆ This last equation is upper triangular and R is nonsingular because matrix A is full rank. Solve the lower triangular system RT w = AT b. The standard method of solving is by a Cholesky factorization. This decomposition is a very useful result. We assume that the matrix A is full rank. approximately. if κ denotes the 3 condition number of matrix A.Spring 2009 9. mn2 + n . 2 function qr requires asymptotically 2mn2 − 3 n3 flops.The number of floatAlgorithm 8 Least squares via normal equations Form the matrix AT A and the vector AT b. 3 The modern method for solving least squares problems is based upon reduced ˆˆ QR factorization.3 SVD factorization Another method for solving least squares problems uses the SVD decomposition. Solve the upper triangular system Rx = w. However.1 Classical solution The classical way to solve least-squares problems is to solve the normal equations.The cost is dominated by the QR factorization. Thus the best we can expect from the normal equations is x − xcomputed = O(κ2 x 9. Algorithm 8 solves the least squares problems. Recall that the Matlab Algorithm 9 Least squares via reduced QR factorization ˆˆ Compute the reduced QR factorization A = QR. Washington AMATH 352 . where R is upper triangular. then the matrix AT A has condition number κ2 . symmetric positive definite. The steps for computing the solution are described in Algorithm 9. 118 . ˆ Compute the vector QT b. Theorem 174.

cn In the following. . The scalar σi are called singular values. The column vectors of U are called left singular vectors. The steps for computing the solution are described in Algorithm 10.Spring 2009 ˆ ˆ ˆ where U is of dimension m × n and satisfies UT U = I. . we can choose g ∈ Pk . this problem can reduce to a linear algebra problem for vector   c1  . . 9. g(t) = c1 + c2 t in which case we would naturally choose n = 2. . ˆ The matrix Σ is nonsingular when the matrix A is full rank. we write the normal equations AT Ax = AT b and we obtain ˆˆ ˆˆ ˆˆ ˆ ˆ VΣUT UΣVT x = VΣUT b ⇒ ΣVT x = UT b. . we want to find a function g(t) from some finite dimensional function space that approximates f (t) over some interval [a.  c =  . .4 Approximation of functions Given some function f (t). The best choice of “basis functions” φj (t) depends on where the function f or the data comes from. . ˆ T b. . we will discuss some approaches. a polynomial of degree k for some k. The Matlab function svd requires asymptotically 2mn2 + 11n3 flops. ≥ σn ≥ 0. Set x = Vw. . For example. . This method is useful when the matrix A is close to rankdeficient. Or if we want to fit by a quadratic we would use these functions along with φ3 (t) = t2 . b]. and Σ is a n × n diagonal matrix diag(σ1 . The problem is to find the coefficients c1 . . 119 . we may have data that we think comes from a linear function. Washington AMATH 352 . V is of dimension n × n ˆ and satisfies VT V = I. The column vectors of V are called right singular vectors. Compute the vector U ˆ ˆ Solve the upper triangular system Σw = UT b. . . by the SVD decomposition. In some cases. + cn φn (t). σn ). ˆ where σ1 ≥ . . . Recall that a function space of dimension n can be described by n basis functions φ1 (t). φn (t). . cn in g(t) = c1 φ1 (t) + . φ1 (t) = 1 and φ2 (t) = t. For the least squares problem. . .The cost is dominated Algorithm 10 Least squares via SVD decomposition ˆˆ Compute the reduced SVD decomposition A = UΣVT .U. For some kinds of approximation.

Figure 12 presents an example of polynomial interpolation with a polynomial of degree 10 and 11 data points. yn    . . . . . . There exists a unique polynomial interpolant to these data in these points. with the property p(ti ) = yi for each index i.. yn . 120 . . . p(x) = c0 + c1 x + . . . where the matrix A and the data vector y are    y=  y1 y2 . We can solve this system of equations.4.. φn (t1 ) φn (t2 ) . tn−1 n      c1 c2 .1 Interpolation Given a set of n data points (ti . . which is guaranteed to be nonsingular as long as the points {ti } are distinct. yn         A=  φ1 (t1 ) φ1 (t2 ) . tn−1 1 tn−1 2 . .. . . . yi ). a polynomial of degree at most n − 1. this system will usually be nonsingular. . . φn (t): g(t) = c1 φ1 (t) + c2 φ2 (t) + · · · + cn φn (t). φn (tn )    . cn   y1 y2 . . . .. With reasonable choice of basis functions. + cn−1 xn−1 . The relationship among the points. . Consider n distinct points t1 . we assume that the data should approximately satisfy y = g(t) where g(t) is a function that is a linear combination of n given functions.  φ1 (tn ) . tn and data y1 .Spring 2009 9. and the coefficients can be expressed by a linear system      1 1 . that is. we can solve the problem of finding the coefficients cj to minimize n (yi − g(ti ))2 i=1 by solving the linear least squares problem Ac = y. . . . . Washington AMATH 352 .U. the data. φ1 (t).. Given the “basis functions” φj (t) and the data. . Example 175. 1 t1 t2 tn .. . .      =   . . . . .

t. Then the system is  24 0  0 −15 0 0 The solution is u(t) = 9. Near the ends of the interval. This unsatisfactory behavior is typical of polynomial interpolation. . not a reflection of the data. the fit is not pleasing. √ Example 176. sensitive to perturbations of the data.U. Note that the structure of the system and ease of solution depends on the choice of basis functions. we select the basis ((t − 4)(t − 9). Instead of using the canonical basis (1. (t − 1)(t − 4)) . Washington AMATH 352 . . 121 .4. 9. (t − 1)(t − 9). yi ). i. The interpolant passes through all the data points.Spring 2009 Figure 12: Degree 10 polynomial interpolant to 11 data points. φ1 (t). Find a polynomial u ∈ P2 interpolating f (t) = t at ti = 1. 4. . 24 15 40     c1 1 0 0   c2  =  2  .2 1 2 3 (t − 4)(t − 9) − (t − 1)(t − 9) + (t − 1)(t − 4). we assume that the data should approximately satisfy y = g(t) where g(t) is a function that is a linear combination of n given functions. p exhibits large oscillations that are an artifact of the interpolation. φn (t): g(t) = c1 φ1 (t) + c2 φ2 (t) + · · · + cn φn (t). . The produced fits are often bad and they tend to get worse if more data are utilized.e. the interpolation process may be ill-conditioned. t2 ). Even if the fit is good. 40 c3 3 Data fitting Given a set of m data points (ti . However.

we have m > n. The fit can be improved by reducing the degree of the polynomial and solve a least-squares problem. Here. Washington AMATH 352 . Figure 13 plots the data and the least-squares fit Figure 13: Degree 7 polynomial least squares fit to 11 data point The new polynomial does not interpolate the data but it captures their overall behavior much better than the polynomial of degree 10. .  φ1 (tm ) . where the matrix A and the data vector y are    y=  y1 y2 . . .Spring 2009 Typically. we can solve the problem of finding the coefficients cj to minimize m (yi − g(ti ))2 i=1 by solving the linear least squares problem Ac = y. .. if columns of A are orthogonal to one another.U. again. then AT A is diagonal. Example 177. φn (t1 ) φn (t2 ) . Consider again the 11 data points used previously for the polynomial interpolation. ym         A=  φ1 (t1 ) φ1 (t2 ) . For example. Example 178. Given the “basis functions” φj (t) and the data. Suppose we want to fit the data 122 . . .. the choice of basis functions governs the ease of solving the problem. . . φn (tm )    . . The computation is also less sensitive to perturbations.

c2 −4 √ The first column is the basis function φ1 (t) = t evaluated at the data points and the second column is the basis function φ2 (t) = (t − 1) evaluated at the data points. c2 = −2 that √ zero residual. this would probably not be true and then the least squares solution would give the fit minimizing the residual. Now we must measure the residual 123 . however. In has other words the 4 data points all lie on the curve g(t) = 4 t − 2(t − 1). and there is a solution c1 = 4. If the data y is changed.U. Figure 14 plots the data and the least-squares fit 0  1   2 3 be √ by a function of the form g(t) = c1 t + c2 (t − 1). Washington AMATH 352 .4.Spring 2009 i 1 2 3 4 ti 0 1 4 9 yi 2 4 2 -4  2  4  c1  =  2 . The linear system would   −1 0   3  8  Figure 14: Least-squares fit with a polynomial and the square root function 9.3 Global approximation Instead of fitting m data points. we can try to find a function g(t) that is as close as possible to f (t) at all points in [a. b]. In this case the right hand side happens to lie in the column space of the matrix.

we write that the derivatives with respect to c1 and c2 are equal to zero ´b ´b ´b f (t)φ1 (t)dt − c1 a φ1 (t)2 dt − c2 a φ1 (t)φ2 (t)dt a ´b ´b ´b f (t)φ2 (t)dt − c1 a φ1 (t)φ2 (t)dt − c2 a φ2 (t)2 dt a The system looks nicer if we use the inner product notation < φ1 . φ2 > 124 y= < f. φ2 > c2 The system has the form Bc = y where B= < φ1 . For example. φ2 > c2 < φ2 . For example. φ1 > . φ2 > < φ2 . To compute the minimum for the norm f − g 2 . i. we look for a stationary point. φ1 > < f. = < f. assume that g(t) = c1 φ1 (t) + c2 φ2 (t) and that we are trying to minimize f − g 2 . The associated norm ˆ b √ 2 |u(t)| dt u 2 = < u. The inner product ˆ < u. φ1 > < φ1 . We expand the equation and gather the coefficients c1 and c2 ˆ c2 1 a b ˆ φ1 (t) dt + 2c1 c2 a 2 b ˆ φ1 (t)φ2 (t)dt + c2 2 a b b φ2 (t)2 dt ˆ a b ˆ a ˆ f (t)φ2 (t)dt + a b − 2c1 f (t)φ1 (t)dt − 2c2 f (t)2 dt. v >= a b u(t)v(t)dt (171) is a natural generalization of the Euclidian inner product in Rn . = 0 . φ1 > c1 + < φ1 . u > = a is a generalization to functions of the Euclidian norm in Rn . φ1 > < φ2 .Spring 2009 (difference between g(t) and f (t)) in some function norm f − g . (172) = < f. the norm f ∞ = max |f (t)| a≤t≤b is a natural generalization of the ∞-norm in Rn .U.e. Washington AMATH 352 . ˆ a b (f (t) − c1 φ1 (t) − c2 φ2 (t))2 dt. φ2 > = 0 . φ1 > c1 + < φ2 .e. φ2 > . i.

 B= . 1]. For example. . . We have < φ1 . 1] with a constant function. . the least squares problem becomes Bc = y.. φ1 >= 2 2 < φ2 . Consider the function f (t) = cos(t) on [−1. Washington AMATH 352 .Spring 2009 These are the normal equations. φ1 >     . The first four Legendre polynomials are P0 (t) P1 (t) P2 (t) P3 (t) = = = = 1 t 3 2 t − 2 5 3 t − 2 1 2 3 t 2 Example 179. the Legendre polynomials are a family of orthogonal polynomials on the interval [−1. < f. φ2 >= 3 < f. φ1 >= 2 sin(1) < f.U. where the matrix B and the data vector y are     < φ1 . . . φ2 >= 0 =⇒ c1 = sin(1) ≈ 0. . φn > Note that the system (172) becomes diagonal if < φ1 . φn > < φn . < φn . φ1 > . −1] for the inner product (171). φ2 >= 0 or if we choose an orthogonal basis (orthogonal for the inner product (171)). y= . Figure 15: Approximation of cosine function on [−1.8415 =⇒ c2 = 0 Figure 15 plots the function f and the best approximation in P1 .. We look for the best fitting in P1 . The basis φ1 (t) = 1 and φ2 (t) = t is orthogonal. 125 . < φ1 . . φn > < f. When using n basis functions. φ1 > . . .

φ3 >= 2 5 < f. Washington AMATH 352 .V] = svd(A. 126 . φ2 >= 0 < f.0) produces the reduced SVD decomposition. in a least squares sense. φ2 >= 2 3 < φ3 . p(xi ) to yi . φ2 (t) = t. and φ3 (t) = 3 (t2 − 1 ) are orthogonal. This solves the least squares problem via the QR factorization.8415 c2 = 0 c3 ≈ −0. Computationally.Spring 2009 Next we look for the best fitting in P2 .S. it is both easier and better to simply set >> x = A \ b. • p = polyfit(x. Figure 16: Approximation of cosine function on [−1.y. We have 2 3 < φ1 . 1] with a quadratic function. the normal equations could be formed and solved by the single command >> x = (A’*A) \ A’*b. The basis φ1 (t) = 1. Other useful commands in Matlab for this section include: • [U. then svd computes only the first n columns of U and Σ is n × n.n) finds the coefficients of a polynomial p(x) of degree n that fits the data.124 =⇒ =⇒ =⇒ c1 = sin(1) ≈ 0. 9. φ1 >= 2 < φ2 . φ3 >≈ −0.U. φ1 >= 2 sin(1) < f.31 Figure 16 plots the function f and the best approximation in P2 .5 Useful commands in Matlab In matlab. If A is ˆ m × n with m ≥ n. however.

U. Of course x = 0 always has this property since A0 = 0. 127 . but we are interested in nonzero eigenvectors of the matrix.Spring 2009 10 Eigenvalues and eigenvectors In talking about eigenvalues we are only interested in square matrices. So the set of all eigenvectors corresponding to a particular eigenvalue λ is a linear subspace of Rn . there may be particular vectors v for which Av is just a scalar multiple of v. Eigenvalues and eigenvectors make sense only when the vectors x and Ax have the same number of rows or when the matrix A is square. Definition 181. Remark. then any linear combination of u and v is also an eigenvalue A(αu + βv) = αAu + βAv = αλu + βλv = λ(αu + βv). Then f (u) = Au defines a mapping from Rn to itself. the matrix A can have a different number of rows and of columns. we have v= 1 0 =⇒ Av = 3 0 =3× 1 0 . since A(αv) = αAv = α(λv) = λ(αv). The scalar λ is then called an eigenvalue. The vector v = 0 is an eigenvector of A if Av = λv for some scalar value λ. If we multiply an arbitrary vector u by A it will get mapped to a new vector Au that is typically linearly independent from u. For a linear system of equations. If A= then Au = 12 −2 3 0 0 −2 u= 4 1 (173) and the vector Au is not a scalar multiple of u. However. More generally. so the two are linearly independent. For the above matrix (173). if u and v are two eigenvectors of A corresponding to the same eigenvalue λ. If v is an eigenvector than so is αv for any scalar α. Washington AMATH 352 . A ∈ Rn×n . (174) Such a vector is called an eigenvector of the matrix A. To ask about the eigenvalues of such a matrix A would be meaningless. Example 180.

This can happen only if the matrix (A − λI) is singular.Spring 2009 Remark. eigenvalue analysis can simplify solutions of certain problems by reducing a coupled systems of equations to a collection of scalar problems. the quadratic is already factored for us.1 How to determine the eigenvalues of a matrix? Recall the identity matrix I. (175) So if λ is an eigenvalue of A the matrix (A − λI) has a nonzero null vector v. The most familiar examples include the study of resonance (of musical instruments when struck or plucked or bowed) and of stability. Algorithmically. So we get det(A − λI) det(A − λI) = (a11 − λ)(a22 − λ) − a12 a21 2 (176) = λ − (a11 + a22 )λ − a12 a21 Remember that for a particular matrix we know the aij values. This is just a quadratic equation in λ. we know that this matrix is singular only if its determinant is zero. 10. Note that this matrix is simply the matrix A with the value λ subtracted from each diagonal element. and we are trying to find λ so that this determinant is zero. Example 182. We will only consider the case n = 2 in detail. We already knew that λ = 3 was an eigenvalue 128 3−λ 0 0 −2 − λ . the n × n matrix with the property that Iu = u for any vector u ∈ Rn . For the matrix in (173). and we see that this can be zero only if λ = 3 or λ = −2. In this case. Eigenvalues and eigenvectors are useful for algorithmic and physical reasons. we get A − λI = and so det(A − λI) = −(3 − λ)(2 + λ). Washington AMATH 352 . If Av = λv with v = 0 then we can write this as Av = λIv and rearranging gives (A − λI)v = 0. So one way to find the eigenvalues of a matrix is to try to figure out the values of λ for which (A − λI) is singular. eigenvalue analysis can give insight into the behavior of evolving systems governed by linear equations. For the 2 × 2 case. so A is a 2 × 2 matrix and A − λI = a11 − λ a21 a12 a22 − λ . just some numbers. where det(A) = a11 a22 − a12 a21 . Physically.U.

Au = 4u for every u ∈ R2 . though they may be complex numbers! Note that even matrices with real elements can have complex eigenvalues. It means that A − 0I is singular. A is singular. 129 4 0 0 4 . Note that for this matrix any vector is an eigenvector. We now see that there should also be vectors v for which Av = −2v. we have det(A − λI) = (λ − 4)2 and so λ1 = λ2 = 4. In the next section. In fact. (181) . So the eigenvalues are λ1 = 0 and λ2 = 7. Let A= 5 −2 −5 2 and so A − λI = 5−λ −2 −5 2−λ (179) Then det(A − λI) = (5 − λ)(2 − λ) − 10 = λ2 − 7λ. Example 183. i. For any 2 × 2 matrix. The eigenvalues are said to be repeated. Any singular matrix has at least one eigenvalue equal to zero. and any null vector of the matrix is an eigenvector associated with this eigenvalue. Note that A has an eigenvalue equal to 0. For some matrices. Let A= 3 4 2 1 and so A − λI = 3−λ 4 2 1−λ (178) Then det(A − λI) = (3 − λ)(1 − λ) − 8 = (λ − 5)(λ + 1) so the eigenvalues are λ1 = 5 and λ2 = −1. First.e. we will see how to determine the eigenvectors once we know the eigenvalues. This quadratic equation can always be factored in the form (λ − λ1 )(λ − λ2 ) (180) where λ1 and λ2 are two numbers. For example. Washington AMATH 352 . They often play an important role in analyzing differential equations. consider A= For this matrix.Spring 2009 because of (174). computing det(A − λI) will give a quadratic equation in λ. 2 For the matrix (182).U. some more examples of computing eigenvalues. Example 184. the vector v = is a null vector of the matrix and 5 hence an eigenvector Av = 0 = 0v. there are v= 0 1 =⇒ Av = 0 −2 = (−2) × 0 1 (177) or any multiple of this vector is also an eigenvector. it may happen that λ1 = λ2 . Av = 0v.

and the matrix is singular if the determinant is zero. . An eigenvector for the λj = ajj is given by the j-th column of the identity matrix (the vector with a 1 in the j-th component and 0 everywhere else). There are many different algorithms that have been invented for approximating eigenvalues. a subset of C. Definition 187. For larger matrices. Some may be complex (if A is real the complex ones always occur in conjugate pairs). Finally. we introduce a definition. and then there is usually no way to determine the roots except by using a numerical method. Washington AMATH 352 . The set of all eigenvalues of a matrix A is called the spectrum of A. 130 . There are some cases where it is easy to determine the eigenvalues of a matrix. Let A= 0 −1 1 0 and so A − λI = −λ −1 1 −λ (182) Then det(A − λI) = λ2 + 1 = (λ − i)(λ + i). • If the matrix is diagonal then the n values on the diagonal are the eigenvalues of the matrix.U. the determinant can also be defined. (λ − λn ) (183) and so a matrix of degree n always has exactly n eigenvalues (though some may be repeated). They are complex conjugates of each other since the matrix A is real.Spring 2009 Example 185. . It is usually very messy just to determine the polynomial. The algebraic multiplicity for an eigenvalue λ is the multiplicity of λ as a root of the characteristic polynomial. • If the matrix is upper triangular or lower triangular then the eigenvalues are again the diagonal elements. The determinant of (A − λI) turns out to be a polynomial p(λ) of degree n in λ. We will not learn how to do this by hand for larger matrices. though the eigenvectors are not just the unit vectors in this case. The eigenvalues are complex. which can always be factored as p(λ) = (λ − λ1 ) . We still want to find values of λ for which (A − λI) is singular. So the eigenvalues are λ1 = i and λ2 = −i. If n > 2 then it is harder to find the eigenvalues. too complicated to outline here. Actually numerical methods for determining eigenvalues do not determine this polynomial and search for its roots — it is too difficult to do this way. Definition 186.

2 How to find the eigenvectors? Once we know the eigenvalues of a matrix. the eigenvalues are λ1 = 5 and λ2 = −1. for The second column is −1 times the first. Let’s find the null space for λ1 = 5: A − 5I = −2 4 2 −4 . so every vector is an eigenvector. 4 0 0 4 . For the matrix A= λ1 = λ2 = 4 and A − 4I = 0 0 0 0 . So. as we already knew. 0 1 and this is the eigenspace. Consider the matrix A= 3 0 0 −2 . We know λ2 = −2 is an eigenvalue. the set of all eigenvectors for the eigenvalue λ is just the null space of the matrix (A − λI). Av = 3 4 2 1 1 1 = 5 5 = 5v. so v = example. and (175) just says that an eigenvector v should be a null vector of this matrix.Spring 2009 10. and is an eigenvector of A. Washington AMATH 352 . The associated eigenspace is the null space of 5 0 A + 2I = . For the matrix A= 3 4 2 1 . Example 190. 0 0 The null space of this matrix is span Example 189. The null space of this matrix is all of R2 . even for large matrices. Isolated roots have 1-dimensional eigenspaces. This space is called the eigenspace of the matrix associated with this eigenvalue.U. 131 . Example 188. This can only happen for repeated roots. how do we compute the eigenvectors? Recall that if λ is an eigenvalue then (A − λI) is singular. 1 1 is a null vector. for example. In this case the eigenspace is 2-dimensional.

we have seen examples where the algebraic multiplicity and the geometric multiplicity are equal. this is not true. for the matrix B. However. Consider the  2 A= 0 0 matrices   0 0 2 2 0  B= 0 0 2 0 1 2 0  0 1 . λn 132  .U. we can find only one independent eigenvector   1  0 . 0 . 10. The geometric multiplicity for an eigenvalue λ is the dimension for the eigenspace associated with the eigenvalue λ. 2 has an algebraic multiplicity equal to 3.Spring 2009 As stated previously. The algebraic multiplicity is always greater or equal to the geometric multiplicity. Washington AMATH 352 . the set of eigenvectors for an eigenvalue forms a subspace of Rn . The decomposition (184) can be rewritten AV = VΛ or  A  v1 ···   ··· vn vn  =  v1  λ 1    . 1 . Definition 191. For the matrix A. we can choose three independent eigenvectors       1 0 0  0 .3 Eigenvalue decomposition A = VΛV−1 . So far. (186) (185) . (184) An eigenvalue decomposition of matrix A is a factorization Here the matrix V is nonsingular and Λ is a diagonal matrix. 2 Both matrices have characteristic polynomial (λ − 2)3 . In general. 0 0 1 So the geometric multiplicity is also 3. 0 So the geometric multiplicity is only 1. It is known as the eigenspace.. Example 192. The dimension of the eigenspace is the number of linearly independent eigenvectors. . Note that such a decomposition does not always exist.

solve a diagonal system with Λ. Definition 193.Spring 2009 This implies that Avj = λj vj and that the j-th column of V is an eigenvector of A associated with the eigenvalue λj . we can expand x in the basis of columns of V. and interpret the result as a vector of coefficients for a linear combination of the columns of V. the 2-norm of A satisfies A 2 = VΛV−1 2 ≤ V 2 Λ 2 V−1 2 = κ2 (V) Λ 2 . In this case. The statement depends on what aspect of behavior one measures and on the matrix V. For example. The eigenvalue decomposition (184) expresses a change of basis to eigenvector coordinates. the determinant of A satisfies n det(A) = det(VΛV−1 ) = det(V)det(Λ)det(V)−1 = det(Λ) = i=1 λi (187) and the trace of A satisfies n tr(A) = tr(VΛV−1 ) = tr(V−1 VΛ) = tr(Λ) = i=1 λi . (189) There are matrices for which this bound is sharp. (188) However. 133 . we have V−1 b = ΛV−1 x. we can expand b in the basis of columns of V. To compute Ax. To solve Ax = b. A great deal of information can be contained in the matrix V. Its eigenvalues are real and there exists an orthogonal matrix Q such that A = QΛQT where QT Q = I. (191) One final factorization is actually the one that is most useful in numerical analysis because all matrices can be factored in this way.U. An n×n matrix A is diagonalizable when it has an eigenvalue decomposition A = VΛV−1 . Washington AMATH 352 . It sometimes happens that the matrix A is diagonalizable and that the n linearly independent eigenvectors can be chosen orthogonal. apply Λ. from the eigendecomposition (184). If Ax = b and A = VΛV−1 . and interpret the result as a vector of coefficients for a linear combination of the columns of V. A symmetric matrix A is diagonalizable. we can not conclude that the matrix A behaves like the diagonal matrix Λ. Note that. (190) Theorem 194. the matrix A is diagonalizable and there exists an orthogonal matrix Q such that A = QΛQT .

. all the eigenvalues λi are strictly positive. Every square real matrix A has a real Schur factorization:   R11 R12 . Note that the eigenvalues of AT A are always positive. . we get n B F = tr (Λ) = i=1 λi (BT B).. Recall that the Frobenius norm is m n (192) B F = i=1 j=1 b2 = ij tr (BT B). Alternatively. So it is diagonalizable. So it is diagonalizable. σn . We have AT A = QΣ2 QT . (194) Example 198. Consider a matrix A ∈ Rm×n which is full rank. AT A = QΛQT . . . (193) . Denote σi the positive square root of λi and Σ the square diagonal matrix whose entries are σ1 . Consider the matrix 134 . So we get B F = tr (BT B) = tr (QΛQT ) = tr (ΛQT Q). 0 Rmm where Q is orthogonal and each block Rii is either a 1 × 1 matrix or a 2 × 2 matrix having complex conjugate eigenvalues. The matrix AT A is symmetric. Consequently.e. Example 197. The matrix BT B is symmetric. Washington AMATH 352 . Note that the triangular matrix T can have complex entries.U. .e. We will show that Σ and Q are two components of the singular value decomposition of A. Q . for a real square matrix.Spring 2009 Theorem 195. Theorem 196. R1m  0 R22 R2m   T  A = Q . i. Consider a matrix B ∈ Rm×n .   . The matrix A is full rank which implies that AT A is invertible. we can write a real reduction to an upper quasi-triangular. . . . . BT B = QΛQT . Matrices A and T have the same eigenvalues. If we have AT Ax = λx. 0 . i. Since QT Q = I. then we get xT AT Ax = λxT x and 2 2 Ax 2 = λ x 2 which implies that λ ≥ 0. Every square matrix A has a Schur factorization A = QTQ∗ where Q is unitary and T is upper triangular.. So the eigenvalues of A necessarily appear on the diagonal of T. . .

The column vectors of V are called the right singular vectors. ∀α ∈ R. i. The column vectors of U are called the left singular vectors. Σ is an n × n diagonal matrix whose entries are positive. λn the set of eigenvalues ordered in a non-decreasing way and q1 . . It has some nice properties.U. xT x (197) • The Rayleigh quotient is the scalar minimizing the norm Ax − αx 2 . for any nonzero vector x.e. 10. Washington AMATH 352 . . λn ]. Recall that such a matrix has real eigenvalues and a complete set of orthonormal eigenvectors. . . Finally. we simplify matters by considering only matrices that are real and symmetric. . We denote λ1 . x x Notice that if x is an eigenvector. we write A = UΣVT where U is an m × n matrix with orthonormal column vectors. . i.4. The Rayleigh quotient of a non-zero vector x ∈ Rn is the scalar xT Ax (196) ρ(x) = T . we have λ1 ≤ ρ(x) = • The decomposition is Ax = ρ(x)x + (Ax − ρ(x)x) orthogonal. Indeed. Ax − ρ(x)x 2 ≤ Ax − αx 135 2 .4 Case of symmetric matrices In this section.e.Spring 2009 U = AQT Σ−1 which belongs to Rm×n . The Rayleigh quotient plays a big role in the computation of eigenvalues and eigenvectors. xT Ax ≤ λn . (195) 10.1 Rayleigh quotient Consider a real symmetric matrix A. Note that we have Avi = σi ui . The scalar σi are called the singular values of A. then its Rayleigh quotient is equal to the associated eigenvalue. . . xT (Ax − ρ(x)x) = 0. i. and V = QT is an n × n orthogonal matrix. • The Rayleigh quotient is bounded and ranges over the interval [λ1 . The matrix U has orthonormal column vectors.e. qn the associated orthonormal eigenvectors. we have UT U = Σ−1 QAT AQT Σ = Σ−1 QQΣ2 QT QT Σ = I. .

To analyze the power iteration algorithm. we write v(0) as a linear combination of the orthonormal eigenvectors v(0) = a1 q1 + . . Next. . + λ2k a2 n n . Recall that Aqi = λi qi and Ak qi = λk qi .U. . Recall that the matrix 2-norm is associated with the Euclidean vector norm. v(0) for k = 1. we have 2 ρ(x) = T (Ax − ρ(x)x).Note that we did not specify any stopping Algorithm 11 Power iteration Set v(0) an arbitrary vector in Rn of norm 1. B 2 = λmax (BT B). . . 10. do w = Av(k−1) v(k) = w/ w 2 λ(k) = v(k) end for T 2 = 1. + λk an qn n 136 = 2 1 λ2k a2 1 1 + . . . x x The Rayleigh quotient appears in the definition of the matrix 2-norm. . termination criteria are very important. . So the matrix 2-norm of B is the square root of the largest value for the Rayleigh quotient of BT B. Indeed. 2. Based on (197). notice that v(k) is proportional to the vector Ak v(0) . In practice. . .2 Power iteration ⇒ Bx 2 2 =λ x 2 2 Consider the following algorithm. + an qn . .4. + λk an qn 1 n where ck is a scalar for the normalization of v(k) ck = λk a1 q1 1 1 + . then xT BT Bx = λxT x which implies that λ ≥ 0.Spring 2009 • The eigenvectors of A are the stationary points for the Rayleigh quotient. the matrix 2-norm of B is equal to the square root of the largest eigenvalue of of BT B. So we get i v(k) = ck λk a1 q1 + . (199) Note that the eigenvalues of BT B are always positive. xT x (198) The matrix BT B is symmetric. namely B 2 = max x=0 Bx 2 = max x=0 x 2 (Bx)T Bx = max x=0 xT x xT BT Bx . Indeed we have If BT Bx = λx. Av(k) criterion for this algorithm. Washington AMATH 352 .

when it exists. First. • [Q. λ(k) − λn = O λn−1 λn 2k (200) 10. Suppose the eigenvalues of A satisfy |λn | > |λn−1 | ≥ . . • T = schur(A. D] = eig(A) produces matrices of eigenvalues D and eigenvectors V of matrix A.U. power iteration is of limited use.’real’) returns the matrix T for the real Schur factorization. ≥ |λ1 | and qT v(0) = an = 0. . it can find only the eigenvector corresponding to the largest eigenvalue. .Spring 2009 Factorizing byλk . Second. reducing the error by a constant factor ≈ |λn−1 /λn |. we obtain n v(k) = 1 (λ1 /λn )2k a2 + . . Then the iterates for the power iteration algorithm satisfy n v(k) − qn when k → +∞. Combined with other ingredients. + a2 n 1 λ1 λn k a1 q1 + λ2 λn k a2 q2 + . . However. But the convergence rates do not change. the convergence is linear. Washington AMATH 352 . From here. so that AU = UT and U∗ U = I.5 Useful commands in Matlab Here are a few commands in Matlab useful for this section. T] = schur(A) produces the Schur matrix T and a unitary matrix U. Theorem 199.’real’) produces the real Schur factorization: a matrix T and an orthogonal matrix Q. 137 . • null(A) computes an orthonormal basis for the null space of A. so that AV = VD. . T] = schur(A. • [V. =O 2 λn−1 λn k . + an qn . Depending on the normalization step. this tool can result in a powerful algorithm. • [U. so that AQ = QT and QT Q = I. • T = schur(A) returns the Schur matrix T. Notice that the approximate eigenvalue converges at a rate twice faster than the vector iterates. the iterates can actually converge to −qn . The convergence can be slow when the two eigenvalues are close to each other. • d = eig(A) returns a vector of the eigenvalues of matrix A. we get the following result.

138 . b) • [a. • ∃ means “there exists”. b] denotes an interval: if a ≤ x ≤ b. then x ∈ (a. − 2. • R ⊃ [a. Washington AMATH 352 . then x ∈ [a. b) denotes an interval: if a < x < b. b] means that R contains [a. • ∀ means “for all”. b) denotes an interval: if a ≤ x < b.. b] • (a. • C is the set of all complex numbers. b) • [a.Spring 2009 A Notations √ • R is the set of all real numbers (−1. then x ∈ [a. b]. b] is included in R. . • [a.. π. • a ∈ R means “a is in R” or “a belongs to R”.).U. b] ⊂ R means that [a.

If you compute something in matlab. • Type “helpwin” at the matlab prompt to bring up some introductory information and browse all the help pages.m” file – also called m-file – from matlab). If you want to do the computation but not print out the result. you can log back on and recover it from C:\oldtemp. and plotting results. helpwin plot will give you information about how to use the plot command. For example. the next time you or someone else logs out. Check out the matlab tutorials linked from the class webpage for more hints. You might also try the demo command in matlab. When you log out. the C:\Temp directory will automatically be moved to C:\oldtemp so the next user will have an empty Temp folder. • Temporary file storage is in the C:\Temp directory. >> x = sqrt(17). Washington AMATH 352 .m” files with the same name and you are not sure which is being used. end the command with a semi-colon. If you just type 139 . lookfor axis will give a list of all commands whose helpfile contains the string “axis” and might help you find the command you are looking for regarding setting axes on a plot. say >> x = sqrt(17) matlab will do the computation and then print out the result on the screen. analyzing data interactively. This should be where matlab works by default (for example if you create a new “. Following are some hints on using matlab that may be particularly useful for getting started in this class. matlab has a nice user interface that you should experiment with and use to learn effectively. • Use the help (or helpwin) and lookfor commands in matlab to learn about how various commands and functions work.U. click on the Start menu and then Programs and then on Matlab. C:\oldtemp is overwritten and the previous version is lost forever. However. This language is widely used by mathematicians and engineers. If you forget to save something on a flash drive or another account that you need. This is particularly useful if you are doing computations on vectors or matrices where the result may consist of hundreds or thousands of numbers.Spring 2009 B Introduction to MATLAB matlab is a very convenient language for testing simple algorithms. The which command is useful if there may be multiple “. • To start matlab on the PCs in the AS lab.

A column vector. In linear algebra it is often important to distinguish between the two. we also frequently use vectors to store the values of a function evaluated at many points. we might set >> x = linspace(-2. For example. Row vectors can be transformed into column vectors and vice versa by the transpose operator. b = 12. 3]. for example >> a = sqrt(17). In matlab. it is generally best to put only one command on each line for readability.g. it will compute it and assign it to a variable named ans. (But ans will be redefined next time you compute something without assigning it elsewhere!) You can have more than one command on a line if you separate them by semicolons. matlab views all numerical variables as matrices. Our vectors will always be column vectors unless it is clear that a row vector is meant. denoted by a superscript T. c will all be computed and the values of b and c will be printed out. (There’s also a command logspace that gives logarithmically spaced points. A single number. 2. x = 2 can be thought of as a 1 × 1 matrix. You can then use ans in a future computation. x = 1:4 which is the same as x = [1 2 3 4] or x = [1. 4] is a 4 × 1 matrix. c = 3 the values of a. In linear algebra we frequently use matrix-vector and matrix-matrix multiplication.U. In addition to doing linear algebra. or by commas if you want the result printed. However. 3]’ = [1. e. the examples in these notes will sometimes put multiple command on a line to save space. 4] is a 1 × 4 matrix. >> y = 2*x + 3. 100). For the multiplication operator * to work. 3. A row vector.g. 2.) 140 . 3. then the command >>b = A*x computes the matrix-vector product (a 3 × 1 vector). so for example [1. b.g. Washington AMATH 352 . the matrices involved must have appropriate dimensions. the prime symbol is used. The linspace command gives a vector of 100 linearly spaced points between -2 and 2.. e. 2. 2. e. if A is a 3 × 4 matrix and x is a 4 × 1 vector. In m-file scripts containing lots of commands. which can be easily done in matlab. 2. The command size(x) will tell you the size. it often doesn’t matter whether they are thought of as row vectors or column vectors. in order to plot a graph of the line y = 2x + 3 for −2 ≤ x ≤ 2. When using vectors only as a data structure to store a collection of numbers.Spring 2009 >> sqrt(17) without assigning this to a variable. x = (1:4)’ which is the same as x = [1. For example.

>> Y = [2*x + 3. matlab will print the result. or >> y = x. What we want is component-wise multiplication. The y defined above will be either a row or column vector depending on which we have used for x. Figure 17 contains the resulting plot Figure 17: Plot of y = 2x + 3 and y = sin(x) over [−2. Washington AMATH 352 . 2] If we want to plot the parabola y = x2 and try to set >> y = x*x. 100)’.Y) will plot the functions 2x + 3 and sin(x) on a single plot. 2.U. resulting in a new vector with 100 components containing values of x2 at each point. For plotting purposes it doesn’t matter whether x is a row vector or a column vector. >> plot(x. For example: >> x = linspace(-2. sin(x)]. a vector of 100 numbers! The linspace command results in a row vector. It is possible to plot more than one curve with a single plot command. or >> y = x^2 an error message will result because the vector x cannot be multiplied by x in the linear algebra sense.^2 to obtain this. In this case it does matter how it is oriented. We can type >> y = x. If you want x to be a column vector you will have to transpose it.Spring 2009 Note: if you leave off the semicolon at the end of one of the statements above.*x. in which case Y could be a matrix and each column of the matrix gives the points for one curve. Note that x has been defined as a column vector here. Figure 18 contains the resulting plot 141 .

If you type >> diary(’hw1. Script files: m-files are useful for storing a sequence of commands to be executed. for example. Washington AMATH 352 . If you are viewing this directory from within the matlab window.. then typing >> hw1a 142 .txt’) . some other commands . >> diary off then all the other commands and responses will be stored in the file hw1. performing the above command and then setting >> z = 2*y + y. Operators like sqrt. cos.. will set the ith element of z to zi = 2yi + yi /(6 + xi ). and turn it in. sets zi = exi cos(yi )..U.m. exp apply to each element of a vector. You can later say diary on to add more to it. (Note: you must say diary off before the file can be viewed or printed..* cos(y). print it out.Spring 2009 Figure 18: Plot of y = x2 over [−2./ As another example. If you have a file called hw1a./(5 + x). so setting >> z = exp(x) . Note that we can multiply by the scalar 2 using * and each element of the vector is multiplied by 2. We can also add the scalar 5 to a vector and each element has 5 added to it. The diary command is useful for making a record of everything you type and the results that matlab computes.txt.) Note that this file should be in the C:\Temp directory. provided that x and y have the same size (otherwise an error message results). 2] The division operator / also has a component-wise version . you may have to select “All Files” in order to see this since by default matlab displays only m-files ending with the .m extension. This is useful for capturing your homework solutions. You can then edit this file.

5*(x + 19/x). The last loop could be executed by >> n = 5. matlab has standard control structures such as “for loops” and “while loops”. In this loop the vector 1:6 is used to indicate that the loop should be repeated 6 times. end This is sometimes useful for doing something quick and dirty. However. while n > 0 y(n) = n^2. These sorts of loops can be put on a single line at the prompt by using commas. end will result in y = [1 2 3 4 5]. for i=1:6 x = 0. 143 .Spring 2009 will cause this set of commands to be executed. end will result in y = [25 16 9 4 1]. Any other vector can be used in the for loop. y(n) = n^2. 4. n = n-1.1. 25] y(n) = sqrt(n). Washington AMATH 352 . 16. If you are using the diary command to capture things. n = n . then you may want to type “echo on” before “hw1a” so that the commands in hw1a are echoed along with matlab’s response. for n = [1. Some examples: x = 4. for n = 5:-1:1 y(n) = n^2. not the commands.U. n = 5. If-then-else constructs can also be used. The value of x is printed out each time it is modified. end will also result in y = [25 16 9 4 1]. 9. end √ approximates 19 using Newton’s method. while n>0. it is much better to put the loop in a m-file with one command on each line and proper indentation so that it is readable. Otherwise the diary file will only contain the responses.

U. Washington

AMATH 352 - Spring 2009

x = rand(1); % a single random number between 0 and 1 if x < 1/3 y = -1; elseif x < 2/3 y = 0; else y = 1; end will set y to −1, 0, or 1 with equal probability. Note that the word “then” does not appear! The expression x < 1/3 appearing above evaluates to either 0 (representing False) or 1 (representing True). Logical expressions can be combined using & for “and” and | for “or”, x = rand(1); % a single random number between 0 and 1 if (x < 1/3) |(x == 0.5) y = -1; elseif x < 2/3 y = 0; else y = 1; end will set y to 2 if x < 1/3 or x == 0.5. In fact this can be done more simply (but less clearly) by y = 2*(x<1/3 | x==0.5); Note that == is used to test for equality, not =, which is only used to assign values to variables. There are several ways to print out a value or string in matlab. We have already seen one: compute a value or just type a variable name at the prompt without a semicolon at the end and matlab will print out the value, >> x = 1/3; >> x x = 3.333333333333333e-01 (By the way, format compact has been used to get matlab to leave out the blank lines it would normally produce around x = ). Often we want to print out something more descriptive or make a table of values with numbers lined up nicely. The disp command prints out a variable without printing the name of the variable first, >> x = 1/3; >> disp(x) 3.333333333333333e-01 144

U. Washington

AMATH 352 - Spring 2009

disp can also be used to print a string of characters. In matlab, a string of characters is specified by putting them in single quotes, s = ’Hello world’; disp(s) or simply disp(’Hello world’) prints out this string. You can build a string out of pieces by putting several strings together in an array. Here’s a useful application of this: >> s = [’The value of x is now ’, num2str(x)]; >> disp(s) The value of x is now 0.33333 Note that num2str converts the number x into a string that can be catenated together with the other string to form the string s. A more general way to make a string that includes one or more numbers when we want to control the spacing, number of digits printed out, etc., is to use the sprintf command, which writes formatted data to a string. This follows the C language formatting conventions if you are familiar with C. Type help sprintf for more details. Here is an example: >> x = 1/3; y = 2/3; >> s = sprintf(’ We now have x = %20.15e and y = %10.5f’, x,y); >> disp(s) We now have x = 3.333333333333333e-01 and y = 0.66667 Here’s an m-file script that approximates the value of the square root of a number given an initial guess. Note the use of the input command to read in data from the prompt: % newtonsqrt.m a = input(’What number do you want to find the square root of? ’); x = input(’What is your starting guess? ’); disp(’ ’); disp(’iteration approximation’); maxiter = 10; % maximum number of iterations to allow for i=1:maxiter x = 0.5*(x + a/x); disp(sprintf(’%5i %20.15e’, i,x)); if abs(x^2 - a) < 1e-12 break % breaks out of the for loop if x is good enough end 145

U. Washington

AMATH 352 - Spring 2009

end if i==maxiter disp(’*** Warning: may not have converged -- tolerance not satisfied’); end disp(’For comparison,’); disp(sprintf(’ sqrt(%g) = %20.15e’, a, sqrt(a))); Here is some sample output: >> newtonsqrt What number do you want to find the square root of? 2 What is your starting guess? 5 iteration approximation 1 2.700000000000000e+00 2 1.720370370370370e+00 3 1.441455368177650e+00 4 1.414470981367771e+00 5 1.414213585796884e+00 6 1.414213562373095e+00 For comparison, sqrt(2) = 1.414213562373095e+00 Note that this m-file is much fancier than the simple loop we used before. The core computation x = 0.5*(x + a/x); is the same as before, but we have added several features along with the tabular output: • The program tests to see if x2 is close to a each iteration, and if it is close enough we break out of the loop (this is a “convergence test”). • A maximum number of iterations (10 here) is specified so that we don’t get into an infinite loop by accident. • If the convergence test is never satisfied then a warning message is printed out so the user knows that the final x may not be sufficiently accurate. Paying attention to these sorts of details is important and similar issues will come up with most of the numerical methods we study. There is also a fprintf command that prints directly to a file. This is useful if you expect a lot of output and you want it to be output to a file instead of zipping past you on the screen. For example: % fprintf_example.m fid = fopen(’outputfile.txt’,’w’); % open a text file for the output for k=1:1000 frpintf(fid, ’k = %4i\n’, k); % note: \n means "new line" end fclose(fid); 146

sec. round. tan.txt containing 1000 lines k = 1 k = 2 k = 3 k = 4 etc. normally distributed (with mean 0 and variance 1). sum. acsc. uniformly distributed between 0 and 1. . Washington AMATH 352 . creates an n × n diagonal matrix with d(1). prod: act on vectors and return a scalar. median. There are many other useful commands in matlab. • size(A) returns the dimensions of A.n) makes an m × n array of all ones. • A = rand(m. . .n) makes an m × n array of random values. atan. acos. cot: Trigonometric functions • asin. a row vector [m n]. fix: rounding real numbers to integers • max. csc. • A = eye(n) makes an n × n identity matrix. • A = ones(m. • d = diag(A) when A is an n × n matrix. cos. 147 .Spring 2009 produces a file named outputfile.n) makes an m × n array of random values. floor. asec. • A = randn(m. Here are just a few examples: • A = zeros(m. . min. creates a vector containing oly the diagonal elements of A • A = diag(d) when d is a vector of length n.U. • sin. mean. • B = zeros(size(A)) makes an array of all zeros the same size as A. d(n) on the diagonal. acot: Inverse trigonometric functions • ceil.n) makes an m × n array of all zeros.

y) Each of these would take two inputs x and y (which might be single numbers.m containing the two lines function y = f1(x) y = 3*x. The first function would not return any values (though it might print something or produce a plot). which starts with a line like function myf(x. for example. One approach is to create an m-file that implements the rule defining the function. Washington AMATH 352 .m.^4 + x. The other type is a function. etc. x. For example. vectors. One type is a script. which is just a list of commands that will be executed when you type the name of the file (without the .m extension). Many simple mathematical functions can be represented by a single formula in terms of elementary operations (addition. multiplication. the second form would return a single value (and f would have to be set somewhere in the body of the function). or matrices) and would presumably do some operations on the inputs.5) ans = 0. The third form would return two values f and g (both of which should be set somewhere in the body). (201) There are several possible ways to represent a function in matlab that allow us to evaluate the function at desired points.1875 148 . cos(x).Spring 2009 C Using Functions in Matlab There are two types of m-files in matlab (files that end with the . Then in matlab we can evaluate this in the obvious way: >> z = f1(1) z = 3 >> alpha = f1(-2) alpha = 48 >> f1(. the function (201) can be implemented by creating a file f1.g] = myf(x.*sin(2*pi*x) .) or standard √ functions such as sin(x).m) into matlab. a file called myf.U.y) or function f = myf(x. An example would be f1 (x) = 3x4 + x sin(2πx)/ 1 + x2 ./ sqrt(1 + x.y) or function [f. etc. exp(x).^2).

y) After executing these commands. 1. ˆ . is not defined unless x is a square matrix (and we don’t want the matrix power). y(i) = f1(x(i)). /. 4 5 6]) ans = 3 48 243 768 1875 3888 This evaluates f1 at a 2 × 3 array of points. However./ rather than simply *. as in the above examples. This preallocates storage for y of the correct size before looping through and resetting each component of the array. >> y = f1(x). 149 .*. For example. If we only plan to evaluate f1 at a single point. >> plot(x. for example. 100). For this small number the difference would be imperceivable.U. Washington AMATH 352 . we can compute f1 at the three values used above by >> f1values = f1([1 -2 . . Another way to evaluate a function at a whole set of points is to use a loop. Note that in this last example we set y = zeros(size(x)) before starting the loop that correctly evaluates y(i). It’s a good idea to get in the habit of always using the vector-component operators in function definitions as it is often convenient to be able to evaluate a function at a whole set of points. by using the vector-component operators it is possible to compute f1 (x) simultaneously for a whole vector of x values with a single call to f1. suppose we want to plot the function f1 (x) for −1 ≤ x ≤ 1 We can do this by evaluating f1 at a large number of points in [−1. then it doesn’t matter which we use. 1] and then using the plot command in matlab: >> x = linspace(-1. For example. . this is more cumbersome than simply setting y = f1(x) and is also typically much slower since 100 function calls are made instead of 1. However. >> for i=1:length(x).ˆ .Spring 2009 Note that we have implemented the formula for f1 (x) in a vectorized manner using the operators .1875 Another example: >> f1([1 2 3. The command y = f1(x) would result in a matlab error if we had used nonvectorized operators in the definition. >> y = zeros(size(x)).0000 0.0000 48. >> x = linspace(-1. since x4 . but for practical problems we will often need to evaluate functions thousands or millions of times and using vectorized form whenever possible can greatly reduce the computational time.5]) f1values = 3. x and y are each vectors of length 100. 1. end In this case x(i) is a scalar value and so f1 is only called with scalar arguments. 100).

>> for i=1:length(x). it is often more efficient to preallocate the storage since resizing y repeatedly is time consuming (and programs that do this will run more slowly).^2).1875 The definition of a function in this manner can use other variables. >> g = @(x) a + x. >> g(4) ans = 29 When g is defined a has the value 25 and this value is then hard-wired into the function definition. but if y has already been used for some other purpose then we it may be the wrong size. there is another way to specify the function rather than creating a 2-line m-file as done above. suppose y was previously used to store a vector of 500 values. The syntax indicates that f1a is a function of a single variable that is denoted by x in the formula that follows.^4 + x. The syntax g = @(x) . However. end would reset the first 100 elements of y but would not change the remaining 400 elements.Spring 2009 This is not strictly necessary for this example: matlab will automatically increase the size of the array y each time through the loop if necessary as we set each value. 1. If y has never been used before in this matlab session then we’ll get what we expect without preallocating. Another reason for preallocating the storage is that it insures that y will be the size we expect. >> a = 25. indicates that the function is 150 . as follows: >> f1a = @(x) 3*x. such as f1 above. Once f1a has been defined in this way. For example.y) would then produce an error message since x and y are not the same size. C.5) ans = 0..*sin(2*pi*x) . It is often more convenient to define it directly at the matlab command line as an in-line function. 100). Then the commands >> x = linspace(-1. The command plot(x. Washington AMATH 352 ./ sqrt(1 + x. >> g(4) ans = 29 >> a = 100.. y(i) = f1(x(i)).1 In-line function definitions For functions that are implemented by a single matlab statement. it can be evaluated in the same manner as m-file functions: >> f1a(.U.

1) ans = 4. Functions of more than one variable can be defined by specifying more than one argument. and so z is assumed to be a previously defined value. >> g = @(x. y) = xy + 20 sin(y).864166380749058e-01 >> quad(f1a. and hence it is surrounded by quotes. If we use the in-line version then f1a is a variable in matlab (of type “function”) and so we just pass in this variable (without quotes). Again note that the notation defining g indicates that it is a function of the two variables x and y in the formula following.314159265358979e+01 This implements the function g(x. Washington AMATH 352 .1) ans = 4.Spring 2009 being defined as a function of a single variable denoted by x in the following string. where f1 (x) is the function specified in (201). >> g(2. Suppose for example we want to ´1 approximate 0 f1 (x) dx. For example.0. is sometimes called quadrature.2 Passing function names into other functions Sometimes we will want to define or use functions that act on other functions. Then we pass in to quad an implementation of the function f1 and the endpoints. it as if we had defined >> g = @(x) 25 + x. we have to give it the function f and also the endpoints of the interval a and b. 151 .U. which we’ll study later. (This sort of approximation. which explains the name of the matlab function. either as an m-file or as an in-line function.y) x. Either can be used.0. the built-in matlab function quad computes numerical approx´b imations to definite integrals. and so a is assumed to be some value already set before executing this function definition (an error message will result if a has not already been set).*y + z*sin(y).pi/2) ans = 2. We have seen two different ways to implement f1 above.) To apply quad. C. but the syntax is slightly different: >> quad(’f1’. then we pass the name of the m-file to quad as a string. a f (x) dx. Note that later changing a without redefining g will not change the function.864166380749058e-01 If we use an m-file to define f1 . >> z = 20.

U. Washington

AMATH 352 - Spring 2009

D

Plotting Functions in Matlab

Manually, when we want to plot a curve, like y = sin(x) over [0, π], we start by taking arbitrary values for x in [0, π] and computing the corresponding value for y. We plot the resulting points (x, y) and connect the points by a straight line, as in “connect the dots” drawing. • In “connect the dots” drawing, the dots are ordered by their number. Here the chosen order is increasing (or decreasing) values of x. • We need to specify enough points in [0, π] so that the sequence of lines connecting the points will give a graph that is visually indistinguishable from the plot of f (x). In Matlab, plotting a curve follows the same procedure. We define a vector x containing many points in the interval of interest, evaluate y = f(x) as a vector of the corresponding function values, and then execute plot(x,y). The plot command produces a graph by connecting the points specified by straight lines, as in a “connect the dots” drawing. If f (x) is a smooth function and we specify a dense enough set of points, then this sequence of straight lines will give a graph that is visually indistinguishable from a plot of the function f (x). But it is important to understand what matlab is doing because it is easy to produce graphs that do a very bad job of illustrating the true function if we are not careful, particularly if the function f is not very smooth or has discontinuities. We will explore this with a number of examples. Consider y = sin(x) over [0, π]. 1. Create sampling values in [0, π] and store them in a column vector x. >> x = []; m = 25; >> for i = 1:(m+1), x = [x; (i-1)*pi/m]; end >> x x = 0 0.1257 0.2513 0.3770 ..... 2. Plot and connect the dots >> plot(x,sin(x)) 152

U. Washington

AMATH 352 - Spring 2009

Figure 19: Figure for plot(x,sin(x)). Consider now y = x2 over [0, π]. >> x = [0; 0.5; 1.0; 1.5; 2.0; 2.5; 3.0; pi]; >> plot(x, x*x) The command result in an error because the vector x can not be multiplied by a vector in linear algebra sense. Instead, we want a component-wise multiplication, resulting in a vector with 8 components containing values of x2 . To get a component-wise version, we type >> x = [0; 0.5; 1.0; 1.5; 2.0; 2.5; 3.0; pi]; >> plot(x, x.*x)

Figure 20: Figure for plot(x, x.*x). The same modification applies for the division operator / or the power operator ^. 153

U. Washington

AMATH 352 - Spring 2009

Order of points. The order of points in the sampling vector x defines the order when connecting the dots. Figure 21 presents two plots with two different orderings in the vector x:   0  3   1. x =   1  2  0  1   2. x =   2  3 

Figure 21: Plots with different order of points in x Parametrized curve. parametrized as follows Consider now the curve x2 + 4y 2 = 1. It can be sin(t) , 2

x = cos(t),

y=

∀t ∈ [0, 2π].

The procedure to plot this curve with Matlab goes 1. Create a vector sampling [0, 2π]. >> t = linspace(0, 2*pi, 100)’; 2. Plot the curve >> plot(cos(t), sin(t)/2) 154

to 5000 say.f(x)) This still captures the essential behavior of the function. to only 10. The idea of approximating a complicated function by a much simpler function is one of the main themes 155 . we obtain the plot of Figure 23(c). Note that it consists of a rapidly varying oscillation (the cos(5x) term) superimposed on a slower oscillation (the 2 sin(x) term). Figure 23 shows the result of the following commands: >> x = linspace(0.U. then we would obtain the graph of Figure 23(b). Washington AMATH 352 . 10. the plot would look essentially the same. >> plot(x. The collection of straight lines plotted by the plot command can be viewed as the the correct graph of a piecewise linear function that approximates the function f (x) that we really intended to plot. With 1000 points the curve plotted looks smooth and captures the function well. plotted using >> x = linspace(0. 1000). This looks smoother but doesn’t look much like the proper graph. If we reduce the number of points used further. 30). but it does not look so smooth — it’s now clear that linear segments are plotted between specified points.f(x)) This gives a reasonable graph of this function. If we increased the number of points. However. if we reduce the number of points and used only 30. 10. where the particular points chosen are shown as dots along with the plot of the proper function. >> plot(x. The reason why is seen in Figure 23(d). Suppose we want to plot f (x) = 2 sin(x) + cos(5x) (202) for 0 ≤ x ≤ 10 and have defined a suitable m-file or inline function f that evaluates this function.Spring 2009 Figure 22: Parametrized curve with default view (left) and with equal axes (right).

156 .U. (b) Plot of function (202) using 30 points. (d) Illustration of where these 10 points lie on the curve y = f (x).Spring 2009 Figure 23: (a) Plot of function (202) using 1000 points. Washington AMATH 352 . (c) Plot of function (202) using 10 points.

In the example we’ve just considered. (b) Plot of function (203) using 1000 points. If so then we might want to give more thought to where we really need to evaluate f many times and where a more coarse representation of f suffices. There’s a little bump in the curve near x = 1 but it’s barely noticeable and if we don’t think about it we might conclude that we have a pretty good plot of f (x) using only 30 points. For example. matlab uses this approximation to plot a set of points since graphics primitives exist to connect any two points by a straight line. (203) Figure 24: (a) Plot of function (203) using 30 points. >> plot(x. 157 . For the function f (x) in (203) it is no big deal to evaluate the function at 1000 points to produce this plot — it takes a small fraction of a second in matlab. This doesn’t agree with what is shown in the plot at all! The function has an exponentially decaying spike centered at x = 1. we can easily integrate a linear function and so if we approximate a function by a piecewise linear approximation we can approximate the integral. consider the function f (x) = x(3 − x) + exp(−100|x − 1|).Spring 2009 of this course and we will see many applications of this idea. We will use this idea in other contexts because linear functions are much easier to work with than arbitrary functions in many ways.3. However. However. for other practical problems it may be quite expensive to compute each function evaluation (for example when each “function evaluation” requires going off and solving some other problem).30).f(x)) Then we obtain the plot shown in Figure 24(a). To capture this we must plot using many more points. note that f (1) = 2 + exp(0) = 3. Figure 24(b) shows a plot with 1000 points. For example.*(3-x) + exp(-100*abs(x-1)). 3] using the commands >> f = @(x) x. As another example of potential pitfalls when plotting.U. Washington AMATH 352 . >> x = linspace(0. Suppose we plot this function over the interval [0.

for a total of 230 points rather than 1000. This warning applies to many of the numerical methods we will consider as well. We can produce a plot that looks just as good as Figure 24(b) but using far fewer evaluations of the function f by using an unequally spaced set of points.5000 is an accurate approximation to the integral. Note that we can get a much better approximation using quad if we split the integral up into an integral from 0 to 1 and one from 1 to 3. (204) The value 4.20)]. e.3) ans = 4.1) + quad(f.01(1 − exp(−200)) ≈ 4.5 + 0.1. and finally 20 points between 1.520000793872945e+00 158 .5 + 0 0 = 4.520000000000000000 . even if you are simply using built-in matlab commands. Do not always blindly trust what the computer tells you.g. However.01(1 − exp(−100)) + 0.3.Spring 2009 it is only near x = 1 that we need lots of values of f in order to capture the behavior.200) linspace(1. for this particular function we can integrate exactly and find that ˆ 0 3 ˆ 3 3 3 2 1 3 + exp(−100|x − 1|) dx x − x x(3 − x) + exp(−100|x − 1|) dx = 2 3 0 0 ˆ 1 ˆ 2 exp(−100x) dx + exp(−100x) dx = 4. returned by quad has an error of about 0. .1.1 and 3. Away from this point we only need a few values to get a good looking plot of the quadratic behavior. The vector x used here has 10 points between 0 and 0.1.9.. It’s easy to see where this error comes from — quad is fooled in much the same way we might be by Figure 24.0. But splitting the interval at the point where the spike lies.0. For example.02 which is quite significant.3) ans = 4. This example serves as a warning that it is good to know something about the function you are trying to plot in order to obtain a reasonable result.. we make sure that quad notices something is going on at this point: >> quad(f.9.500000072864403e+00 We might conclude that 4.10) >> plot(x.U.1. followed by 200 points between 0.0. It has entirely missed the sharp ´3 peak near x = 1 and has produced a good approximation to 0 x(3 − x) dx. Washington AMATH 352 .. . suppose we want to compute ´3 f (x) dx for the function of (203).9 and 1.5000.9. We might again try the matlab function 0 quad and we would obtain >> quad(f.1.f(x)) linspace(0. >> x = [linspace(0.

Spring 2009 D. • figure(2) Open a second plot window. Useful for plotting several curves together or points on top of curves. Turn this off with hold off or use the clf (clear frame) command to return to a black frame. • axis([a b c d]) Set the region shown to a ≤ x ≤ b (horizontal axis) and c ≤ y ≤ d (vertical axis). 159 . • hold on Hold the current plot so that the next plot command will appear together with the current plot. Use the matlab help facilities for more information on plotting. or bring this one to the top.U. Washington AMATH 352 .1 Some other useful plotting commands The following commands are often useful.

Sign up to vote on this title
UsefulNot useful

Master Your Semester with Scribd & The New York Times

Special offer for students: Only $4.99/month.

Master Your Semester with a Special Offer from Scribd & The New York Times

Cancel anytime.