You are on page 1of 45

TU M¨ unchen

1. Foundations of Numerics from Advanced Mathematics
Linear Algebra
Miriam Mehl: 1. Foundations of Numerics from Advanced Mathematics
Linear Algebra, October 23, 2012 1
TU M¨ unchen
1.1. Linear Algebra
Mathematical Structures
• a mathematical structure consists of one or several sets and one or several
operations defined on the set(s)
• special elements:
– neutral element (of an operation)
– inverse element (of some element x)
• a group: a structure to add and subtract
• a field: a structure to add, subtract, multiply, and divide
• a vector space: a set of vectors over a field with two operations: scalar
multiplication, addition of vectors, obeying certain axioms (which?)
• note: sometimes, the association with classical (geometric) vectors is helpful,
sometimes it is more harmful
Miriam Mehl: 1. Foundations of Numerics from Advanced Mathematics
Linear Algebra, October 23, 2012 2
TU M¨ unchen
Exercise Mathematical Structures
Show that the possible manipulations of
the Rubik’s Cube with the operation ’ex-
ecute after’ are a group.
Miriam Mehl: 1. Foundations of Numerics from Advanced Mathematics
Linear Algebra, October 23, 2012 3
TU M¨ unchen
Exercise Mathematical Structures – Solution
Show that the possible manipulations of
the Rubik’s Cube with the operation ’ex-
ecute after’ are a group.
• Closure: executing any two manipulations after one
another is a Cube minipulation, again.
• Associativity: the result of a sequence of three
manipulations is obviously always the same no matter how
you group them (the first two or the last two together).
• Identity: obviously included (just ’do nothing’).
• Invertibility: execute a manipulation in backward direction.
Miriam Mehl: 1. Foundations of Numerics from Advanced Mathematics
Linear Algebra, October 23, 2012 4
TU M¨ unchen
Exercise Mathematical Structures
Show that the rational numbers with the operations + (add) and
∗ (multiply) are a field.
Miriam Mehl: 1. Foundations of Numerics from Advanced Mathematics
Linear Algebra, October 23, 2012 5
TU M¨ unchen
Exercise Mathematical Structures – Solution
Show that the rational numbers with the operations + (add) and
∗ (multiply) are a field.
• Closure: obviously closed under + and ∗.
• Identity: 0 for +, 1 for ∗.
• Invertibility: each element q has an inverse −q under +
and
1
q
under ∗. The latter holds for all elements except from
the neutral element of +, i.e., 0.
• Associativity: well-known for both + and ∗.
• Commutativity: also known from school (a + b = b + a,
a ∗ b = b ∗ a).
• Distributivity: dito (a ∗ (b + c) = a ∗ b + a ∗ c).
Miriam Mehl: 1. Foundations of Numerics from Advanced Mathematics
Linear Algebra, October 23, 2012 6
TU M¨ unchen
Exercise Mathematical Structures
Is the set of N ×N matrices (N ∈ N) matrices with real numbers
as entries over the field of real numbers a vector space?
Miriam Mehl: 1. Foundations of Numerics from Advanced Mathematics
Linear Algebra, October 23, 2012 7
TU M¨ unchen
Exercise Mathematical Structures – Solution
Is the set of N ×N matrices (N ∈ N) matrices with real numbers
as entries over the field of real numbers a vector space?
The answer is yes. Look up the axioms and show that they hold
for the xample on your own.
Miriam Mehl: 1. Foundations of Numerics from Advanced Mathematics
Linear Algebra, October 23, 2012 8
TU M¨ unchen
Vector Spaces
• a linear combination of vectors
• linear (in)dependence of a set of vectors
• the span of a set of vectors
• a basis of a vector space
– definition?
– why do we need a basis?
– is a vector’s basis representation unique?
– is there only one basis for a vector space?
• the dimension of a vector space
• does infinite dimensionality exist?
• important applications:
– (analytic) geometry
– numerical and functional analysis: function spaces are vector spaces
(frequently named after mathematicians: Banach spaces, Hilbert spaces,
Sobolev spaces, ...)
Miriam Mehl: 1. Foundations of Numerics from Advanced Mathematics
Linear Algebra, October 23, 2012 9
TU M¨ unchen
Exercise Vector Spaces
Is the set of vectors
__
1
0
_
,
_
0
1
_
,
_
1
3
__
linearly
independent?
Miriam Mehl: 1. Foundations of Numerics from Advanced Mathematics
Linear Algebra, October 23, 2012 10
TU M¨ unchen
Exercise Vector Spaces – Solution
Is the set of vectors
__
1
0
_
,
_
0
1
_
,
_
1
3
__
linearly
independent?
The set of vectors is not linearly independent, since the third
element can easily be written as a linear combination of the first
two:
_
1
3
_
= 1 ·
_
1
0
_
+ 3 ·
_
0
1
_
.
Miriam Mehl: 1. Foundations of Numerics from Advanced Mathematics
Linear Algebra, October 23, 2012 11
TU M¨ unchen
Exercise Vector Spaces
span
__
1
0
0
_
,
_
0
0
1
__
= ?
Miriam Mehl: 1. Foundations of Numerics from Advanced Mathematics
Linear Algebra, October 23, 2012 12
TU M¨ unchen
Exercise Vector Spaces – Solution
span
__
1
0
0
_
,
_
0
0
1
__
=
__
a
0
b
_
; a, b ∈ R
_
.
Miriam Mehl: 1. Foundations of Numerics from Advanced Mathematics
Linear Algebra, October 23, 2012 13
TU M¨ unchen
Exercise Vector Spaces
Consider the set of all possible polynomials with real
coefficients as a vector space over the field of real numbers.
What’s the dimension of this space? Give a basis.
Miriam Mehl: 1. Foundations of Numerics from Advanced Mathematics
Linear Algebra, October 23, 2012 14
TU M¨ unchen
Exercise Vector Spaces – Solution
Consider the set of all possible polynomials with real
coefficients as a vector space over the field of real numbers.
What’s the dimension of this space? Give a basis.
The space is infinite dimensional, a basis is for example
_
1, x, x
2
, x
3
, . . .
_
.
Miriam Mehl: 1. Foundations of Numerics from Advanced Mathematics
Linear Algebra, October 23, 2012 15
TU M¨ unchen
Linear Mappings
• definition in the vector space context; notion of a homomorphism
• image and kernel of a homomorphism
• matrices, transposed and Hermitian of a matrix
• relations of matrices and homomorphisms
• meaning of injective, surjective, and bijective for a matrix; rank of a matrix
• meaning of the matrix columns for the underlying mapping
• matrices and systems of linear equations
• basis transformation and coordinate transformation
• mono-, epi-, iso-, endo-, and automorphisms
Miriam Mehl: 1. Foundations of Numerics from Advanced Mathematics
Linear Algebra, October 23, 2012 16
TU M¨ unchen
Exercise Linear Mappings
Is the mapping f : R
3
→R
3
,

x → 5 ·

x +
_
1
2
3
_
linear?
Miriam Mehl: 1. Foundations of Numerics from Advanced Mathematics
Linear Algebra, October 23, 2012 17
TU M¨ unchen
Exercise Linear Mappings – Solution
Is the mapping f : R
3
→R
3
,

x → 5 ·

x +
_
1
2
3
_
linear?
f is not linear, since f (α

x) = αf (

x).
Miriam Mehl: 1. Foundations of Numerics from Advanced Mathematics
Linear Algebra, October 23, 2012 18
TU M¨ unchen
Exercise Linear Mappings
What’s the linear mapping f : R
2
→R
2
corresponding to the
matrix
_
4 0
3 2
_
?
Miriam Mehl: 1. Foundations of Numerics from Advanced Mathematics
Linear Algebra, October 23, 2012 19
TU M¨ unchen
Exercise Linear Mappings – Solution
What’s the linear mapping f : R
2
→R
2
corresponding to the
matrix
_
4 0
3 2
_
?
f
__
x
y
__
=
_
4x
3x + 2y
_
.
Miriam Mehl: 1. Foundations of Numerics from Advanced Mathematics
Linear Algebra, October 23, 2012 20
TU M¨ unchen
Exercise Linear Mappings
Give the rank of the matrix
_
_
_
1 0 0 0
0 1 0 0
0 0 0 0
0 0 0 1
_
_
_
.
Is the corresponding linear mapping injective, surjective,
bijective?
Miriam Mehl: 1. Foundations of Numerics from Advanced Mathematics
Linear Algebra, October 23, 2012 21
TU M¨ unchen
Exercise Linear Mappings – Solution
Give the rank of the matrix
_
_
_
1 0 0 0
0 1 0 0
0 0 0 0
0 0 0 1
_
_
_
.
Is the corresponding linear mapping injective, surjective,
bijective?
The rank is three. Thus, the corresponding linear mapping is
neither injective, nor surjective or bijective.
Miriam Mehl: 1. Foundations of Numerics from Advanced Mathematics
Linear Algebra, October 23, 2012 22
TU M¨ unchen
Examples Linear Mappings
Monomorphism:
_
0 1
0 0
1 0
_
Epimorphism:
_
1 0 0
0 1 0
0 0 0
_
Iso-/Automorphism:
_
0 1
1 0
_
Endomorphism:
_
2 1 0
0 1 2
1 0 1
_
.
Miriam Mehl: 1. Foundations of Numerics from Advanced Mathematics
Linear Algebra, October 23, 2012 23
TU M¨ unchen
Determinants
• definition
• properties
• meaning
• occurrences
• Cramer’s rule
Miriam Mehl: 1. Foundations of Numerics from Advanced Mathematics
Linear Algebra, October 23, 2012 24
TU M¨ unchen
Determinants – Definition
det(A) =

a
1,1
a
1,2
· · · a
1,N
a
2,1
a
2,2
· · · a
2,N
.
.
.
.
.
.
.
.
.
a
N,1
a
N,N

=
a
1,1

a
2,2
· · · · · · a
2,N
.
.
.
.
.
.
.
.
.
.
.
.
a
N,2
· · · · · · a
N,N

−a
1,2

a
2,1
a
2,3
· · · a
2,N
a
3,1
a
3,N
.
.
.
.
.
.
a
N,1
a
N,3
· · · a
N,N

+ . . .
Miriam Mehl: 1. Foundations of Numerics from Advanced Mathematics
Linear Algebra, October 23, 2012 25
TU M¨ unchen
Exercise Determinants
det(A) = 0 ⇒ A defines a . . .morphism.
det(A) = 0 ⇒ A defines a . . .morphism.
Miriam Mehl: 1. Foundations of Numerics from Advanced Mathematics
Linear Algebra, October 23, 2012 26
TU M¨ unchen
Exercise Determinants – Solution
det(A) = 0 ⇒ A defines an Endomorphism.
det(A) = 0 ⇒ A defines an Automorphism.
Miriam Mehl: 1. Foundations of Numerics from Advanced Mathematics
Linear Algebra, October 23, 2012 27
TU M¨ unchen
Exercise Determinants
det(A · B) =?
det
_
A
−1
_
=?
det
_
A
T
_
=?
Miriam Mehl: 1. Foundations of Numerics from Advanced Mathematics
Linear Algebra, October 23, 2012 28
TU M¨ unchen
Exercise Determinants – Solution
det(A · B) = det(A) · det(B).
det
_
A
−1
_
= det(A)
−1
.
det
_
A
T
_
= det(A).
Miriam Mehl: 1. Foundations of Numerics from Advanced Mathematics
Linear Algebra, October 23, 2012 29
TU M¨ unchen
Exercise Determinants
Determine the solution of the linear system
2x
1
+ x
2
= 4
2x
2
+ x
3
= 0
x
1
+ x
2
+ x
3
= 3
with the help of determinants.
Miriam Mehl: 1. Foundations of Numerics from Advanced Mathematics
Linear Algebra, October 23, 2012 30
TU M¨ unchen
Exercise Determinants – Solution
Determine the solution of the linear system
2x
1
+ x
2
= 4
2x
2
+ x
3
= 0
x
1
+ x
2
+ x
3
= 3
with the help of determinants.
x
1
=

4 1 0
0 2 1
3 1 1

2 1 0
0 2 1
1 1 1

=
7
3
; x
2
=

2 4 0
0 0 1
1 3 1

2 1 0
0 2 1
1 1 1

=
−7
3
; x
3
=

2 1 4
0 2 0
1 1 3

2 1 0
0 2 1
1 1 1

=
4
3
.
Miriam Mehl: 1. Foundations of Numerics from Advanced Mathematics
Linear Algebra, October 23, 2012 31
TU M¨ unchen
Eigenvalues
• notions of eigenvalue, eigenvector, and spectrum
• similar matrices A, B:
∃S : B = SAS
−1
(i.e.: A and B as two basis representations of the same endomorphism)
• resulting objective: look for the best / cheapest representation (diagonal form)
• important: matrix A is diagonalizable iff there is a basis consisting of
eigenvectors only
• characteristic polynomial, its roots are the eigenvalues
• Jordan normal form
• important:
– spectrum characterizes a matrix
– many situations / applications where eigenvalues are crucial
Miriam Mehl: 1. Foundations of Numerics from Advanced Mathematics
Linear Algebra, October 23, 2012 32
TU M¨ unchen
Exercise Eigenvalues
Diagonalize the matrix
_
3 2
2 3
_
. Give both eigenvalues and
eigenvectors and the basis transformation matrix transforming
the given matrix in diagonal form.
Miriam Mehl: 1. Foundations of Numerics from Advanced Mathematics
Linear Algebra, October 23, 2012 33
TU M¨ unchen
Exercise Eigenvalues – Solution
Diagonalize the matrix
_
3 2
2 3
_
. Give both eigenvalues and
eigenvectors and the basis transformation matrix transforming
the given matrix in diagonal form.
Eigenvalues:

3 −λ 2
2 3 −λ

= 9 −6λ + λ
2
−4 = 5 −6λ + λ
2
⇒ λ
1,2
=


36−20
2
= 3 ±2 ⇒ λ
1
= 5, λ
2
= 1.
Miriam Mehl: 1. Foundations of Numerics from Advanced Mathematics
Linear Algebra, October 23, 2012 34
TU M¨ unchen
Exercise Eigenvalues – Solution
Diagonalize the matrix
_
3 2
2 3
_
. Give both eigenvalues and
eigenvectors and the basis transformation matrix transforming
the given matrix in diagonal form.
Eigenvector for λ
1
= 5:

−2 2
2 −2

x
y

=

0
0

⇔ x = y ⇒

x
1
=

1
1

Eigenvector for λ
2
= 1:

2 2
2 2

x
y

=

0
0

⇔ x = −y ⇒

x
2
=

1
−1

Miriam Mehl: 1. Foundations of Numerics from Advanced Mathematics
Linear Algebra, October 23, 2012 35
TU M¨ unchen
Exercise Eigenvalues – Solution
Diagonalize the matrix
_
3 2
2 3
_
. Give both eigenvalues and
eigenvectors and the basis transformation matrix transforming
the given matrix in diagonal form.
The basis transformation matrix thus is

1 1
1 −1

and results in the diagonal matrix

5 0
0 1

.
Miriam Mehl: 1. Foundations of Numerics from Advanced Mathematics
Linear Algebra, October 23, 2012 36
TU M¨ unchen
Scalar Products and Vector Norms
• notions of a linear form and a bilinear form
• scalar product: a positive-definite symmetric bilinear form
• examples of vector spaces and scalar products
• vector norms:
– definition: positivity, homogeneity, triangle inequality
– meaning of triangle inequality
– examples: Euclidean, maximum, and sum norm
• normed vector spaces
• Cauchy-Schwarz inequality
• notions of orthogonality and orthonormality
• turning a basis into an orthonormal one: Gram-Schmidt orthogonalization
Miriam Mehl: 1. Foundations of Numerics from Advanced Mathematics
Linear Algebra, October 23, 2012 37
TU M¨ unchen
Exercise Scalar Products and Vector Norms
Are the following operators scalar products in the vector space
of continuous functions on the interval [a; b]?
f , g
1
:=
_
b
a
f (x) · g(x)dx
f , g
2
:=
_
b
a
f (x) · g(x)
2
dx
f , g
3
:=
_
b
a
f
+
(x) · g(x)dx
Miriam Mehl: 1. Foundations of Numerics from Advanced Mathematics
Linear Algebra, October 23, 2012 38
TU M¨ unchen
Exercise Scalar Products and Vector Norms – Solution
Are the following operators scalar products in the vector space
of continuous functions on the interval [a; b]?
f , g
1
:=
_
b
a
f (x) · g(x)dx Yes!
f , g
2
:=
_
b
a
f (x) · g(x)
2
dx No! (not linear in g)
f , g
3
:=
_
b
a
f
+
(x) · g(x)dx No! (not positive definite)
Miriam Mehl: 1. Foundations of Numerics from Advanced Mathematics
Linear Algebra, October 23, 2012 39
TU M¨ unchen
Exercise Scalar Products and Vector Norms
Proof that a set {

x
1
,

x
2
, . . . ,

x
N
} of non-zero orthogonal vectors
in a vector space with scalar product (·, ·) always is a basis of
its span.
Miriam Mehl: 1. Foundations of Numerics from Advanced Mathematics
Linear Algebra, October 23, 2012 40
TU M¨ unchen
Exercise Scalar Products and Vector Norms – Solution
Proof that a set {

x
1
,

x
2
, . . . ,

x
N
} of non-zero orthogonal vectors
in a vector space with scalar product (·, ·) always is a basis of
its span.
Proof by contradiction:
Assume that the set is not linearly independent. Then, there is a element

x
i
taht can be
written as a linear combination

x
i
=

k∈I
α
k

x
k
of other elements, where the index set
I ⊂ {1, 2, . . . , N} does not contain i . With this, we get
0 = (

x
i
,

x
i
) =

x
i
,

k∈I
α
k

x
k

=

k∈I
α
k
(

x
i
,

x
k
) = 0.
Contradiction. Thus, the vector set is linearly independent and, thus, is a basis of its
span.
Miriam Mehl: 1. Foundations of Numerics from Advanced Mathematics
Linear Algebra, October 23, 2012 41
TU M¨ unchen
Exercise Scalar Products and Vector Norms
Transform
__
1
1
1
_
,
_
1
1
0
_
,
_
1
0
0
__
into an orthogonal basis
of R
3
.
Miriam Mehl: 1. Foundations of Numerics from Advanced Mathematics
Linear Algebra, October 23, 2012 42
TU M¨ unchen
Exercise Scalar Products and Vector Norms – Solution
Transform
__
1
1
1
_
,
_
1
1
0
_
,
_
1
0
0
__
into an orthogonal basis
of R
3
.
Gram-Schmidt orthogonalization:

x
1
=


1
1
1


,

x
2
=


1
1
0







x
1
,




1
1
0








(
x
1
,x
1
)

x
1
=


1
1
0



2
3

x
1
=



1
3
1
3

2
3



,

x
3
=


1
0
0







x
1
,




1
0
0








(
x
1
,x
1
)

x
1





x
2
,




1
0
0








(
x
2
,x
2
)

x
2
=



7
15

8
15
1
15



.
Miriam Mehl: 1. Foundations of Numerics from Advanced Mathematics
Linear Algebra, October 23, 2012 43
TU M¨ unchen
Matrix Norms
• definition:
– properties corresponding to those of vector norms
– plus sub-multiplicativity:
AB ≤ A · B
– plus consistency
Ax ≤ A · x
• matrix norms can be induced from corresponding vector norms: Euclidean,
maximum, sum
A := max
x=1
Ax
• alternative: completely new definition, for example Frobenius norm (consider
matrix as a vector, then take Euclidean norm)
Miriam Mehl: 1. Foundations of Numerics from Advanced Mathematics
Linear Algebra, October 23, 2012 44
TU M¨ unchen
Classes of Matrices
• symmetric: A = A
T
• skew-symmetric: A = −A
T
• Hermitian: A = A
H
=
¯
A
T
• s.p.d. (symmetric positive definite): x
T
Ax > 0 ∀x = 0
• orthogonal: A
−1
= A
T
(the whole spectrum has modulus 1)
• unitary: A
−1
= A
H
(the whole spectrum has modulus 1)
• normal: AA
T
= A
T
A or AA
H
= A
H
A, resp. (for those and only those matrices
there exists an orthonormal basis of eigenvectors)
Miriam Mehl: 1. Foundations of Numerics from Advanced Mathematics
Linear Algebra, October 23, 2012 45