31 views

Uploaded by Maxwell Chomas

Notes on the basics of linear algebra for graduate study.

- Study Micro
- Matrix Computation for Engineers and Scientist by Jennings
- Syllabus.341_W2016
- UT Dallas Syllabus for eco5309.501 05f taught by Daniel Obrien (obri)
- 790601 GluckReinhornRutenberg ICE
- MIMO Algebra
- Stan Reference 2.6.0
- JMathPhys_34_2968 Hojman
- Spatial Kinematic Chains Analysis Synthesis Optimization
- Lec1_27
- SGN2206LectureNew2
- The Measurement of Intellectual inﬂUence∗ Ignacio Palacios-Huerta† and Oscar
- optimization techiniques
- 151-published-Paweł CABAŁA-2010-using the AHP in evaluating decision alternatives.pdf
- MIT18 06S10 Final Answers
- Gate CSE Syllabus
- Complete Sol
- Luc TDS ING 2016 State Space Trajectories
- lecture5_partitionSpectral
- ScriptieDefinitief.pdf

You are on page 1of 28

Bai Huang

I.

Matrix Algebra

1. Vector Spaces

1.1 Real Vectors

: The set of (finite) real numbers

m : The m-dimensional Euclidean space

(Cartesian product of s: )

A Cartesian product A1 A2 Am:

All possible ordered pairs, whose i ele ent is fro

A , i = ,,

x s: The elements or components of the vector x

m: The order of the vector x

The arithmetic operations for two vectors:

Addition:

x + y = (x + y , x + y , , x + y )

a. x and y must be of the same order.

b. [commutativity] x + y = y + x

c. [associativity] (x + y) + z = x + (y + z)

Scalar multiplication: cx : = (cx1 , cx2 , , cxm ) , where c is a constant

Two vectors x and y are collinear if either x = 0 or y = 0 or y = cx.

Inner product:

< x , >= m 1 x y = x y = y x

a. Properties:

i) < , > = < , x >

ii) < , + z > = < x , > + < x , >

iii) < x , > = c < x , >

iv) <x , x> 0, with <x , x> = 0 iff x = 0

b. Norm: || x || : = < x , x >1/2

It is the geometric idea of length of the vector x.

c. A vector x is said to be normalized if ||x|| = 1

Any nonzero vector x can be normalized to x by x =

|| ||

. Since x

d. Two vectors x and y are orthogonal if <x , y> = 0. We write xy.

e. If, in addition, ||x|| = ||y|| = 1, the two vectors are said to be orthonormal.

Example: In m , the unit vectors (or elementary vectors)

e1 = ( 0 0 0) , e2 = (0 0 0) ,, em = (0 0 0 ) ,

are orthonormal.

f. Cauchy Schwarz inequality:

< , >2 ||x||2 ||y||2 with equality iff x and y are collinear.

Exercise: Prove C-S inequality using properties of inner product.

g. Triangle inequality:

||x + y|| ||x|| + ||y|| with equality iff x and y are collinear.

Exercise: Prove triangle inequality. [Hint: using C-S inequality]

x

x-y

y

By the cosine rule, ||x y||2 = ||x||2 + ||y||2 ||x||||y||cos.

After simplification, this becomes < , > = || |||| ||

, thus the angle

between x and y is determined by cos =

,

|| |||| ||

(0 <

< )

b. The projection of y onto x = ||y||cos

g

|| ||

A complex number, say u, is denoted by u = a + ib, where a and b are real

numbers and i is the imaginary unit defined by i2 = . We write Re(u) = a

and I (u) = b.

If u = a + ib, v = c + id are two complex numbers, they are said to be equal

iff a = c and b = d.

Addition: u + v = (a + c) + i(b + d)

Product: uv = (ac bd) + i(ad + bc)

The complex conjugate of u = a + ib is defined as u = a ib.

(u ) = u

(u + v) = u + v

(uv) = u v

uv u v unless uv is a real number

The modulus of u = a + ib is defined by |u| = (u u )1/2 = a2 + b 2 .

Division:

u uv* uv*

=

=

v vv* |v|2

> =

,

m

i=1

u i v*i

>1/2

1.3 Vector Spaces

A vector space

is a nonempty set of elements (called vectors) together with

two operations and a set of axioms.

Twp operations:

a. Addition: For any x, y , x + y

b. Scalar multiplication: For any x , and any real (or complex) scalar ,

x

Axioms:

a. Addition

i) x + y = y + x

ii) x + (y + z) = (x + y) + z

iii) a vector in

(denoted by 0) such that x + 0 = x for all x

iv) x , a vector in

(denoted by -x) such that x + (-x) = 0

b. Scalar multiplication

i) (x) = ()x

ii) 1x = x

c. Distributive laws

i) (x + y) = x + y

ii) ( + )x = x + x

It is the scalar rather than the vector that determines whether the space is real

or complex.

Three commonly used vector spaces:

Hilbert space

add completeness

A nonempty subset

of a vector space

is called a subspace of

if, for all

x, y , we have x + y and x for any scalar .

The intersection of two subspaces

and in a vector space , denoted by

, consists of all vectors that belong to both

and .

The union of two subspaces

and in a vector space , denoted by

,

consists of all vectors that belong to at least one of

and .

and in a vector space , denoted by

consists of all vectors of the form a + b where a

and b .

+ ,

the form 1 x1 + 2 x2 + + x

A finite set of vectors x1 , x2 ,, x (n ) is said to be linearly dependent

if there exist scalars 1 , 2 , , , not all zero, such that

1 x1 + 2 x2 + + x = 0; otherwise it is linearly independent.

are the vectors

Exercise: For which values of

( ,1, 0) , (1, ,1) , and (0,1, ) linearly dependent?

(containing possibly an infinite number

of vectors) is linearly independent if every nonempty finite subset of A is

linearly independent; otherwise it is linearly dependent.

Let A be a nonempty set of vectors from a vector space . The set

consisting of all linear combinations of vectors in A is called the subspace

spanned (or generated) by A.

Any set of n linearly independent real vectors x1 , x2 ,, x spans .

If a vector space

contains a finite set of n linearly independent vectors

x1 , x2 ,, x , but any set of n + 1 vectors is linearly dependent, then the

dimension of

will be n. We write dim ( ) = n.

In this case,

is said to be finite dimensional. If no such n exists,

is

then infinite dimensional. In particular, for = {0}, we say dim ( ) = 0.

If

is a vector space, finite or infinite, and A is a linearly independent set of

vectors from , then A is a basis of

if

is spanned by A.

Example: a. What is the dimension of when the field of scalars is ?

b. What is the dimension of when the field of scalars is ?

A complex vector space

is an inner product space if

x, y , a complex-valued function < , >, which is called the inner

product of x and y, such that

i) < , > = < , >

ii) < + , > = < , > +< , >

iii) <

, >= < , >

iv) < , > 0, with equality iff x = 0

A real vector space

is an inner product space if x, y , a real

number < , > satisfying conditions i) iv).

Consider a real-valued function (x) defined on an inner product space. It is a

norm if (x) satisfies

i) (x) = ||(x)

ii) (x) 0 with equality iff x = 0

iii) (x + y) (x) + (y)

Exercise: Show that ||x|| < , x >1/2 is a norm.

The concept of an inner product not only induces the idea of length (the norm)

||x|| < , x >1/2 , but also of distance of two vectors x and y,

d (x , y) ||x y||, which satisfies

i) d (x , x) = 0

ii) d (x , y) > 0 if x y

iii) d (x , y) = d (y , x)

iv) d (x , y) d (x , z) + d (z , y)

In any inner product space, the Cauchy-Schwarz inequality and the triangle

inequality hold. Besides, an equality, called the parallelogram theorem, is

stated as

||x + y||2 + ||x y||2 = ||x||2 + ||y||2

Example: Prove the parallelogram inequality in terms of algebra and geometry.

For two vectors x and y in an inner product space, we say that x and y are

orthogonal if < , > = 0, we write xy.

[Pythagorean Theorem]

If xy in an inner product space, ||x + y||2 = ||x||2 + ||y||2.

Exercise: Does the converse hold?

A is called an orthogonal set if each pair of vectors in A is orthogonal. If, in

addition, each vector in A has unit length, then A is called an orthonormal set.

Example: Prove that any orthonormal set is linearly independent.

Two subspaces

and of an inner product space

are said to be

orthogonal if every vector in

is orthogonal to every vector in .

If

is a subspace of an inner product space , then the space of all vectors

is called the orthogonal complement of , denoted by

.

orthogonal to

Example: Prove that

is a subspace of .

A sequence {x n }n

x if ||x x|| 0 as n .

[Continuity of inner product]

If {x n } , { yn } are two sequences in an inner product space such that

||x x|| 0, ||y y|| 0, then

i) ||x || ||x||, ||y || ||y||

ii) < x , y > < x , y >

A sequence {x n }n

||x xm || 0 as n,

( > 0, () > 0

.

. . |x xm | < ,

>

() )

Cauchy sequence in converges in the norm to an element x .

2. Matrices

2.1 Matrix Terminology

A matrix is a rectangular array of numbers, denoted

a11

a12

a1n

a21

a22

a2 n

am1 am 2

amn

The dimensions of a matrix are the numbers of its rows and columns. So the

dimension of matrix A is m n (m by n); or A is an m n matrix.

Sometimes the dimension is also called the order.

A matrix with all entries zero is called a zero matrix or null matrix.

When m = n, A becomes a square matrix.

Several types of square matrices are listed here:

A symmetric matrix is one in which aij

a ji

i, j

main diagonal.

An identity matrix is a diagonal matrix with all entries equal to 1 on the

main diagonal. It is denoted by I . For example, I 3 is a 3 3 identity

matrix.

A triangular matrix is a square matrix that has only zeros either above or

below the main diagonal. If the zeros are above the diagonal, the matrix is

said to be lower triangular. Otherwise, it is upper triangular.

An orthogonal matrix P is a square matrix that satisfies PP P P I .

Here are some concepts for square matrices: inverse, determinant, and trace.

A square matrix A is invertible if

We write B

B, s.t. An n Bn

Exercise: Prove this.

Bn n An

In .

b. A is invertible

The column vectors of

independent.

c. Example: i) A zero matrix is non-invertible

ii) The inverse of an identity matrix is itself

iii) The inverse of a diagonal matrix:

a1

a1 1

0

,

an

are linearly

an

if all ai

1

a b

c d

d

ad bc c

1

b

,

a

if ad bc

i) Definition Method

ii) Formula Method

iii) Elementary Transformation Method

e. Properties:

i) ( A 1 )

ii) ( A 1 )

iii) ( AB)

(A )

1

B 1A

iv) ( A BD 1C )

A 1B ( D CA 1 B) 1 CA

Exercise: Try to figure out the special cases for iv) when

(1) B C ?

(2) D

I, B

b, C

c?

matrix A, denoted by det (A) or |A|.

a. Definition (expansion by cofactors):

Using any row, say i, we obtain

| A|

j 1,

, n , where

j 1

Aij is the matrix obtained from A by deleting row i and column j. The

When the correct sign, ( 1)i j , is added, it becomes a cofactor, or the (i,j)

cofactor. This operation can be done using any column as well.

Obviously, it is easy to choose the row or column that has the most zeros

to make the expansion. It is unlikely, though, that you will ever calculate

any determinant over 3 3 without a computer.

b. The determinant provides important information when the matrix is that

of the coefficients of a system of linear equations. The system has a

unique solution if and only if the determinant is nonzero.

c. When the determinant corresponds to a linear transformation of a vector

space, the transformation has an inverse operation if and only if the

determinant is nonzero.

d. A is invertible

| A| 0

a square matrix with real entries:

The absolute value of the determinant gives the scale factor by which

area or volume is multiplied under the associated linear transformation,

while its sign indicates whether the transformation preserves orientation.

Example: i)

a b

c d

ad bc

a b

d

ii)

g

e

h

f

i

e

h

f

i

d

g

f

i

d

g

e

h

f. Properties:

i) Switching two rows or columns changes the determinant sign.

ii) Any determinant with two identical rows or columns has value 0.

iii) A determinant with a row or column of zeros has value 0.

iv) Adding a scalar multiple of one row (or column) to another does not

change the determinant.

n

v) | An n |

| A|

vi) | A | | A |

vii) | A 1 | | A |

viii) If A1 , A2 ,

| A1 A2

AK | | A1 || A2 |

a11

ix) | A |

| AK |

a12

a1n

a11

a22

a2 n

a21

a22

aii

i 1

ann

x) | A |

an1

A11

A12

A11

A22

A21

A22

an 2

ann

| A11 || A22 |

Exercise: Using the concepts of inverse and determinant, prove the following

properties for orthogonal matrix P :

i) P P 1

ii) | P | =

The trace of a square matrix is defined to be the sum of the entries on the

n

mail diagonal. tr ( An n )

aii .

i 1

Properties:

i) The trace is invariant under cyclic permutations, for example

tr ( ABCD )

tr ( BCDA)

tr (CDAB )

tr ( DABC )

ii) tr ( A ) tr ( A)

iii) For two matrices of the same dimensions,

tr ( A B ) tr ( AB )

tr ( BA )

tr ( B A)

aij bij

i, j

iv) tr ( A B ) tr ( A) tr ( B )

v) tr (cA)

ctr ( A)

vi)* E (tr ( X )) tr ( E ( X ))

An m n matrix can be viewed as a set of n column vectors in m , or as a set

of m rows in . Thus, associated with a matrix A are two vector spaces: the

column space and the row space.

The column space of A , denoted by colA , consists of all linear

combinations of the columns of A ,

col {x m : x = y for so e y }.

The row space of A , denoted by colA , consists of all linear combinations

of the rows of A , or the columns of A ,

col {y : y = x for so e x m }.

The column rank of A is the maximum number of linearly independent

columns it contains, namely, the dimension of the vector space that is

(col ).

spanned by its column space

The row rank of A is the maximum number of linearly independent rows it

contains, namely, the dimension of the vector space that is spanned by its row

(col ).

space

(col ) =

[The Rank Theorem] The nontrivial fact that

(col )

implies that the column rank of A is equal to its row rank. It follows that

(col ).

rk( ) =

A square n n matrix A is said to be nonsingular if rk( ) = n ;

otherwise, the matrix is singular. In fact,

A is invertible

A is nonsingular

rk( ) = n

| A| 0

rk( ) < .

be written as A BC , where Bm

and Cn

Simple properties of rank:

Let A be an m n matrix,

in ( , )

i) 0 rk( ) = rk

A is said to be of full rank if rk( ) =

A O

ii) rk( ) = 0

iii) rk( ) = n

iv) rk( ) = rk( ) if 0

in ( , ).

i) rk( + ) rk( ) + rk( )

ii) rk( ) |rk( ) rk( )|

Example: Prove i) and ii).

Rank inequalities (product):

col

i) col

ii) rk( ) in (rk( ), rk( ))

Example 1: Prove i) and ii).

Example 2: Let A be an m n matrix. If m<n, show that no m n matrix

B exists such that B A

In .

nonsingular. Show that rk(

and Cn

be

) = rk( ).

) = rk(

).

The equality of two matrices A and B : A

A (or AT )

For any matrix A :

(A )

A is a symmetric matrix

A is normal if AA

Addition:

A B [aij

A 0

A B

aij

bij ,

i, j

bij

a ji ,

i, j

AA

( A B) C

( A B)

A (B C)

B

Scalar multiplication:

cA [caij ]

Matrix product:

For an m r matrix A [aik ] and an r n matrix B [bkj ] , the product

matrix C

is the

j th column of B .

In general, AB

BA .

A and B commute if AB

BA .

A Ir

Im A

( AB )C

A( BC )

( A B )C

( AB )

AC

BC

BA

xi

(1,1,

xi2

xx

xi yi

xy

,1)( x1 , x2 ,

, xn )

yx

For two real matrices A and B of the same dimension we define the inner

product as A, B

aij bij tr ( A B ) , which induces the norm:

i, j

|| A ||

A, A

1/2

aij2

tr ( AA )

i, j

A calculation that may help to condense the notation or simplify the coding is the

Kronecker product, denoted by

.

For general matrices A of dimension m n and B of dimension p q ,

a11 B

a12 B

a1n B

a21 B

a22 B

a2 n B

am1B am 2 B

amn B

0

I

Example:

(A

B)

(A

B)

a. | A

and Bn n ,

B | | A |n | B |m

b. tr ( A

(A

B )(C

B)

tr ( A)tr ( B )

D)

( AC )

( BD )

The conjugate transpose of U, denoted by , is defined as = .

If

is real, then = .

A square matrix

is said to be Hermitian if = .

A square matrix

is said to be unitary if = .

2.3 System of Equations

Consider the set of n linear equations Ax = b, where x constitute the unknowns, A is

a know matrix of coefficients, and b is a specified vector of values. We are interested

in:

(1) whether a solution exists;

(2) if so, how to obtain it;

(3) if it does exist, then whether it is unique.

We only consider a square equation system here (i.e. those with an equal number of

equations and unknowns).

A homogeneous equation system is of the form Ax = 0.

Every homogeneous system has at least one solution, known as the zero

solution (or trivial solution).

If the system has a nonsingular matrix A, then zero is the only solution.

If the system has a singular matrix, then there is a solution set with an infinite

number of solutions. This solution set is closed under addition and scalar

multiplication.

A nonhomogeneous equation system is of the form Ax = b, where b is a

nonzero vector.

A nonhomogeneous equation system has a unique non-trivial solution

x = A 1 b if and only if A is nonsingular.

If A is singular, then the system has either no solution or an infinite number

of solutions.

m

[ : b] [ : x]

x1

x2

x3

5 .

7

3

2

5

SOLN:

1

2

3

6

1

5

1 0 0

0 1 0

3

2

0 0 1

x1

x2

x3

2 .

1

[Cramers Rule]:

Cramers Rule is an explicit formula for the solution of a system of linear

equations, with each variable given by a quotient of two determinants.

Example: Solve the equation

x1

x2

x3

5 .

7

SOLN: x1

x3

1

5

7

2

5

3

1

2

1

1

2

1

3

6

4

3

6

4

2 1 1

5 2 5

3 1 7

2 1

3

5 2

6

3 1 4

21

3 , x2

7

2

5

3

2

5

3

1

5

7

1

2

1

3

6

4

3

6

4

14

7

2,

7

1.

7

Given a vector y and a matrix X, we are interested in expressing y as a linear

combination of the columns of X. There are two possibilities.

If y lies in the column space of X, then we shall be able to find a vector b

such that y = Xb.

Suppose that y is not in the column space of X. Then there is no b such that y

= Xb holds. We can, however, write y = Xb + e, where e is the difference

between y and Xb, or residual.

We try to solve for b = ar in e e = ar in (y Xb) (y Xb).

Using the matrix calculus, b is found to be the solution to the

nonhomogenous system X y = X Xb. It follows that b = (X X) 1 X y.

Am

Cn

Bm

Dn

None of the matrices needs to be square, but A and B must have the same number

of rows, A and C must have the same number of columns, and so on.

We will mainly focus on partitioned matrices with two row blocks and two column

blocks. It can be extended to m row blocks and n column blocks case, such as

Z11

Z12

Z1n

Z 21

Z 22

Z 2n

Z m1

Zm2

Z mn

As a special case, we say that a square matrix is block-diagonal if it takes the form

Z11

Z 22

Z rr

where all diagonal blocks are square, not necessarily of the same order.

A General Principle

The main tool in obtaining the inverse, determinant, and rank of a partitioned matrix

is to write the matrix as a product of simpler matrices, that is, matrices of which one

(or two) of the four blocks is the null matrix.

Some Basic Results

Partitioned sum: Let

Z1

A1

C1

B1

D1

and Z 2

A2

C2

B2

, then Z

D2

Z1 Z 2

Z1Z 2

A1 A2 B1C2

C1 A2 D1C2

A1B2 B1 D2

.

C1 B2 D1D2

Partitioned transpose:

A B

C D

A

B

C

D

tr

A B

C D

tr ( A) tr ( D)

A1 A2

C1 C2

B1 B2

D1 D2

i)

In

Im

ii)

A B

C D

C D

A B

E O

O In

A B

C D

EA EB

C

D

Im

O

iii)

E

In

A B

C D

A EC

C

B ED

D

i)

A B

C D

O

Iq

Ip

O

ii)

A B

C D

F O

O Iq

iii)

A B

C D

Ip

Iq

B A

D C

AF

CF

B

D

A B AF

C D CF

If A is nonsingular, we have

Im

CA

Im

A 1B

In

Iq

O D CA 1 B

Similarly, if D is nonsingular, we have

Im

O

BD

In

A B

C D

Ip

O

1

D C

A BD 1C

O

In

O

.

D

Inverses

If A and D are nonsingular,

A O

O D

O

O B

C O

be orthogonal?

A

A 1 BE 1CA

E 1CA

A 1 BE

A

D 1CF

D CA 1 B are nonsingular,

A BD 1C are nonsingular,

F 1 BD

1

D 1CF 1 BD

Determinants

Let Z

A B

,

C D

If A is nonsingular, | Z | | A || D CA 1 B | .

If D is nonsingular, | Z | | D || A BD 1C |

Let A and D be square matrices, of order m and n, respectively.

a. For any m m matrix E ,

EA EB

A B

|E|

C

D

C D

b. For any n m matrix E ,

A

B

C EA D EB

Example: Will

C D

A B

A B

C D

A B

always hold?

C D

A useful set of results for analyzing a square matrix A, real or complex, arises from

the solutions to the set of equations Ax = x.

The pairs of solutions are the characteristic roots (or eigenvalues) and their

associated characteristic vectors (or eigenvectors) x.

It is easy to see that the solution set for x is closed under scalar multiplication. To

remove the indeterminancy (apart from sign), x is normalized so that x x =

(x x = when x is real).

The solution then consists of and the n-1 unknown elements in x.

Ax = x (A I)x = 0, which is a homogeneous equation system.

It has a nonzero solution if and only if the matrix (A I) is singular,

Note: For a matrix A of order n, we use 1 , 2 , , to denote its eigenvalues

a. 1 , 2 , , can be real or complex. But for a symmetric matrix A, its

eigenvalues are always real numbers.

b. If appears n > times then it is called a multiple eigenvalue and the

number n is the (algebraic) multiplicity of ; if appears only once it

is called a simple eigenvalue.

Example: i) What are the eigenvalues of a diagonal matrix?

ii) What are the eigenvalues of a triangular matrix?

Although the eigenvalues are an excellent way to characterize a matrix, they do

not characterize a matrix completely.

Two different matrices may have the same eigenvalues.

Can one eigenvector be associated with two distinct eigenvalues?

Can two distinct vectors be associated with the same eigenvalue?

Example: Prove or think of an example.

Any linear combination of eigenvectors associated with the same eigenvalue is an

eigenvector for that eigenvalue.

Exercise: Prove this statement.

The geometric multiplicity of an eigenvalue is the dimension of the space

spanned by the associated eigenvectors. This dimension cannot exceed the

algebraic multiplicity.

Example: i) Find the eigenvalues and eigenvectors for the following two matrices:

3 0

0

1 3 0

a.

0 1 0

2 1 5

b.

6

1

ii) For the matrix in a., what are the (algebraic) multiplicity and the

geometric multiplicity of each eigenvalue? What about the matrix in b.?

iii) Are the eigenvectors for each matrix linearly independent? Are they

orthogonal?

Eigenvectors associated with distinct eigenvalues are linearly independent, but

not necessarily orthogonal.

If

is an eigenvalue of A , t

eigenvector(s).

If x is an eigenvector of A , t x ( t

with the same eigenvalue.

Do A and A have the same eigenvalues? Do they have the same eigenvectors?

Either provide a proof or a counterexample.

A is nonsingular if and only if all its eigenvalues are nonzero.

Example: Prove this theorem.

scalar

0 such that A I is nonsingular.

Example: Prove this theorem.

Let

be a simple eigenvalue of a square matrix A , so that Ax

x for some

eigenvector x. If A and B commute, x is an eigenvector of B too.

Example: Prove this theorem.

Similarity

Two matrices are of the same dimension and the same rank are said to be

equivalent.

If A and B are equivalent matrices, then there exist nonsingular matrices

E and F such that B EAF .

When, in addition, A and B are square and there exists a nonsingular

matrix T such that T 1 AT B , then they are said to be similar.

Example: Prove the claim that similar matrices have the same set of eigenvalues.

Do they have the same set of eigenvectors as well?

Properties:

For an n n matrix A with eigenvalues 1 , 2 , , ,

| A|

n

i

i 1

Ak has eigenvalues

tr ( A)

k

1

k

2

k

n

n

i 1

The eigenvalues are all real

Eigenvectors associated with distinct eigenvalues are orthogonal.

(not sufficient and necessary)

The eigenvectors span

The rank is equal to the number of nonzero eigenvalues

Example:

a. Find a square matrix that does not possess this property.

b. Prove the theorem that the rank of any matrix A equals the number of

nonzero eigenvalues of A A

The matrix can be diagonalized

T 1 AT

, where

is a diagonal matrix containing the eigenvalues of A

The factorization theorems try to diagonalize the matrices. If a matrix cannot be

diagonalized, we ask how close to a diagonal representation we can get.

Because of the central role of factorization theorems, let us list them below:

If A is an m n matrix of rank r , then

A BC with Bm

EAF

and Cn

both of rank r .

[ QR Decomposition]

A QR (when r

n) with Q *Q

[Singular Value Decomposition]

A U V * with U m

m n

The diagonal entries of

are known as the singular value of A . The m

columns of U and the n columns of V are called the left-singular vectors and

right-singular vectors of A.

If A is a square matrix of order n, then

[Schur Decomposition]

P* AP M with P unitary and M upper triangular.

[Spectral Theorem]

(diagonal) with P unitary, if and only if A is normal.

P* AP

[Spectral Decomposition]

(diagonal) with P orthogonal, if A is symmetric.

P AP

(diagonal) and P BP M (diagonal) with A and

P AP

symmetric and P orthogonal, if and only if A and B commute.

[Jordan Decomposition] T 1 AT

T 1 AT

T 1 AT

diag ( A) with T unit upper nonsingular, if A upper triangular

T 1 AT

(diagonal) and T 1 BT M (diagonal) with T nonsingular, if

A has only simple eigenvalues and commutes with B .

Example:

a. [Singular Value Decomposition]

Consider the 4 5 matrix

1 0 0 0 2

A

0 0 3 0 0

0 0 0 0 0

0 4 0 0 0

I 4 and VV *

I5 .

such that

b. [Spectral Decomposition]

Consider the symmetric matrix A

|A

For

For

P AP

I | (1

)(3

) 8 (

1 , Ax

5 , Ax

2

3

1

3

1

3

2

3

2 2

2 2

1)(

x1

x2

5)

2x 2

2x1

2 2

2 2

2

3

1

3

1,

2 1

, )

3 3

1

2

,

)

3 3

1

3

2

3

1 0

0

n

xi x j aij .

i 1 j 1

x Ax , where A is a symmetric

There are some matrices, however, for which the sign of q will be determined

regardless of x. For a given matrix A,

If x Ax

If x Ax

Let A be a symmetric matrix,

a. If all eigenvalues of A are positive (negative), then A is positive definite

(negative definite).

b. If some of the roots are zero, then A is nonnegative (nonpositive) definite

if the remainder are positive (negative).

c. If A has both negative and positive roots, then A is indefinite.

Example: Use the Spectral Decomposition to explain.

Note: The if part is in fact if and only if.

If A is positive definite, then | A |

If A is positive definite, so is A 1 .

The identity matrix is positive definite.

If An

nonnegative definite.

If A is positive definite and B is a nonsingular matrix, then B AB is

positive definite.

Define d x Ax x Bx

x ( A B )x .

greater than B . We write A B . Or A B is positive definite.

a. Suppose

are those of B . If

definite.

b. If A B , then B

, i 1,

P

, n , then A B is nonnegative

A 1.

If A is nonnegative definite, then the powers can only be defined for r

0.

Cholesky Decomposition

Let A be a Hermitian, positive-definite matrix, then A can be decomposed as

A LL* , where L is a lower triangular matrix with strictly positive diagonal

entries, and L* denotes the conjugate transpose of L .

The Cholesky decomposition is of great importance in terms of numerical

computation, especially for solving the system of linear equations.

An idempotent matrix, A , is one that is equal to its square, that is, A2 A .

All of the idempotent matrices we shall encounter are symmetric, though,

idempotent matrices can be either symmetric or not.

Properties: Let A be a symmetric idempotent matrix,

a. I A is also a symmetric idempotent matrix.

b. All the eigenvalues of A are 0 or 1.

c. rk ( A) tr ( A) .

d. A is nonsingular

A

Example: Prove a. d.

A useful idempotent matrix

I.

Xn

(n

X (X X ) 1 X .

b. What is rk ( H ) ?

c. Show that I

d. What is rk ( I

H)?

Let A be a symmetric n n matrix with eigenvalues

. The

sphere, that is, x x = .

a.

ax x x = 1 obtained when x = x1

b.

in x x =

obtained when x = x

c.

ax

,,

x x =

obtained when x = x

Scalar function f ( x ) of a scalar x

A variable y is a function of another variable x , say, y

f ( x ) , if each value

respectively labeled the dependent variable and the independent variable.

Assuming that the function f ( x ) is continuous and differentiable, we obtain the

dy

,

dx

following derivatives: f ( x)

f ( x)

d2y

, and so on.

dx 2

The chain rule:

If h ( x )

(g f )x

g ( f ( x )) , then h ( x )

g ( f ( x )) f ( x ) .

Scalar function f (x) of a vector x

We can regard a function y

, xn ) as a scalar-valued function of a

f ( x1 , x2 ,

f (x) .

y

f (x)

x

Also we have

x1

x2 .

xn

f (x)

x

x1

x2

xn

For a matrix Am

f ( A)

A

f ( A)

a11

f ( A)

a12

f ( A)

a1n

f ( A)

a21

f ( A)

a22

f ( A)

a2 n .

f ( A)

am1

f ( A)

am 2

f ( A)

amn

derivative.

Commonly used derivatives

f (x)

i) f (x) x y , then

y.

x

ii) f ( B )

f ( B)

B

B is an m n matrix, then

xy .

iii) f (x)

x By , then

f (x)

x

By .

iv) f (y)

x By , then

f (x)

y

B x.

v) f (x)

matrix, then

f (x)

x

( A A )x .

Example: Let A

vi) f ( A)

that

rule.

1 3

, show that

3 4

log | A | , then

| A|

aij

f ( A)

A

( A 1)

f (x)

x

(x Ax)

x

2 Ax .

2 Ax .

f1 (x)

Let

f (x)

where

is

vector

of

order

n,

then

f m (x)

f1 (x)

x1

f (x)

x

f1 (x)

xn

.

f m (x)

x1

f m (x)

xn

f (x)

is called the Jacobian matrix.

x

The Jacobian determinant is the determinant of the Jacobian matrix if

m n.

Hessian matrix

If we take the derivative for the gradient

f (x)

, then a second derivative

x

2

f (x)

x12

f (x)

x1 x2

f (x)

x2 x1

f (x)

x22

f (x)

xn x2

f (x)

xn x1

f (x)

x1 xn

f (x)

x2 xn .

f (x)

xn2

continuous and continuously differentiable functions from Youngs theorem.)

The chain rule:

f1 (x)

Let h (x)

, then

f m (x)

h(x)

x

g (f (x))

f

f (x)

.

x

ax R = a x x Ax, where

3

a = (5 4 ) and A=

.

3

3

5

Example 2: The least squares problem:

Suppose we have y = Xb + e, where y and e are both vectors of order n, b is

a vector of order K, X is thus an n K matrix.

Solve for b = ar in e e = ar in (y Xb) (y Xb).

- Study MicroUploaded byRija Ali
- Matrix Computation for Engineers and Scientist by JenningsUploaded byAnonymous LcR6ykPBT
- Syllabus.341_W2016Uploaded bySTAurelium
- UT Dallas Syllabus for eco5309.501 05f taught by Daniel Obrien (obri)Uploaded byUT Dallas Provost's Technology Group
- 790601 GluckReinhornRutenberg ICEUploaded bySutanuBhowmick
- MIMO AlgebraUploaded byngduyhieu
- Stan Reference 2.6.0Uploaded byTerra Reynolds
- JMathPhys_34_2968 HojmanUploaded byVarov
- Spatial Kinematic Chains Analysis Synthesis OptimizationUploaded byValencia Luis Ernesto
- Lec1_27Uploaded byjuncos0729
- SGN2206LectureNew2Uploaded byssosloa
- The Measurement of Intellectual inﬂUence∗ Ignacio Palacios-Huerta† and OscarUploaded byapi-8348981
- optimization techiniquesUploaded byMuhammad Ali Khan Awan
- 151-published-Paweł CABAŁA-2010-using the AHP in evaluating decision alternatives.pdfUploaded byRahmat Adi
- MIT18 06S10 Final AnswersUploaded byandreaskailina
- Gate CSE SyllabusUploaded byprofBalamurugan
- Complete SolUploaded byDani Greentown
- Luc TDS ING 2016 State Space TrajectoriesUploaded byNitin Maurya
- lecture5_partitionSpectralUploaded byapi-3834272
- ScriptieDefinitief.pdfUploaded byJack Liu
- Tensor AlgebraUploaded byAnna Robertson
- Design of Linear Control Systems 2.pptUploaded byJos1
- 2007csccs990.UnlockedUploaded byHelperAssignment
- production controlUploaded byAbdul Nazar
- RotationUploaded byJary Lin
- Matthew G. Reuter, Tamar Seideman and Mark A. Ratner- Probing the surface-to-bulk transition: A closed-form constant-scaling algorithm for computing subsurface Green functionsUploaded byDfmso0
- kernel Hilbert spacesUploaded byrotero_pujol
- Maths All Formulae Class 9 10 11 12Uploaded byRaj
- Analysis of Multi-Domain Wave Propagation in IC EngineUploaded byhappyrockzz
- Applications of Eigen Values and VectorsUploaded bymunim matin

- Notes 354Uploaded bySarah Watkins
- No linealUploaded byAndres
- qm_lecture_notes.pdfUploaded bySimon Gikonyo
- 15.24Uploaded byMohan Rao
- PSOC-UNIT3-4-6-2012Uploaded byb4buzzi
- Linear Algebra Test 1 ReviewUploaded byjohn doe
- Comparison of Volume and Distance Constraint on Hyperspectral UnmixingUploaded byCS & IT
- What_Is_Polarization.pdfUploaded byAnil Kumar Nayak
- 233 ReviewExam1 Print Sp 2013Uploaded byL_sherry
- elecUploaded byapi-236544093
- Lect2_Fuzzy Set.pptUploaded byDavid Cesar
- physicswhitepaper.pdfUploaded byAlbertoMeroni
- Aff_Col_BA_Part_II.pdfUploaded byRamavtar Rajput
- Chapter 21 03Uploaded byaisha
- 5 2 Lin AlgebraUploaded bysevimcankaya
- sequencesUploaded byDorel Barbu
- M.Sc Chemistry syllabus, DCRUST MurthalUploaded byAnonymous 20dkHAn
- 2 Introduction to the Analytic Hierarchy Process [Matteo Brunelli 2015] (1)Uploaded byTobias Linderoth
- BCAUploaded byNitin Kumar Vats
- [J. J. Moreau, P. D. Panagiotopoulos (Eds.)] Nonsm(BookZZ.org)Uploaded bySun Shine
- Linear Vector SpacesUploaded byAhsan Shahid
- elect-sylUploaded byDeepak Kumar
- LAVectorSpacesUploaded byShree Salunkhe
- itug.pdfUploaded byHalder Subhas
- course outline matmatics.pdfUploaded bySaima BatoOl
- The Unreasonable Effectiveness of Mathematics in the Natural SciencesUploaded byAshlee Acevedo
- 9 - Multi DOF - Free VibrationsUploaded bydidiek noor
- Short Term Voltage Instability Effects on Synchronous and Induction MachinesUploaded byyourou1000
- Moving Frame MethodUploaded byTs'ung Yih
- C4 SummaryUploaded byAisha A Rehman