Linear System Theory

Dr. Vali Uddin

Hamdard University

Vali.uddin@hamdard.edu.pk

Lecture 3 1
Linear Algebra
 The above are review materials on systems
 To further study systems beyond by using Laplace and z-
transforms, we need background on linear algebra
 We shall review/study the following:
– Scalars  Vectors  Matrices
– Vector space  Representation of Vectors
– Functions on vectors  Representation of functions
– Linear algebraic equations
– Eigenvectors, Generalized eigenvectors, and Jordan-form
– Functions of a square matrix
– Norms and inner products

Lecture 3 2
Linear Spaces and Linear Operators

 Consider a set of scalars. What properties to have?

 Field F: A set of elements (scalars) and two operations
"+" (addition) & "" (multiplication) s.t. for all , , and 
 F:
– Closedness:  +   F, and     F
– Commutative:  +  =  + , and    =   
– Associative: (+)+ = +(+), and () =()
– Distributive:   ( + ) =    +   
– Unit elements:  0 and 1 s.t.  + 0 =  and   1 = 
– Inverse elements: For any ,   s.t.  +  = 0
(additive inverse). For any   0,   s.t.    = 1
(multiplicative inverse)
Lecture 3 3
Examples
 Examine the following, and see if they are fields or not:
(N, +, ), (I, +, ), (R, +, ), (C, +, )
– (N, +, ): No additive inverse, no multiplicative inverse
– (I, +, ): No multiplicative inverse
– (R, +, ): Yes
– (C, +, ): Yes
 (The set of 2 by 2 real matrices, +, )
– No multiplicative inverse
 (R(s), +, ) (the set of rational functions with real
coefficients)
– Yes

Lecture 3 4
 Linear Spaces over a Field
Q. Vector? What properties to have for a set of vectors?
 A vector space (or a linear space) over a field, denoted as
(X, F), consists of
– X ~ A set of elements (vectors);  F ~ A field
– "+" ~ Vector Addition;  "" ~ Scalar multiplication
s.t. for all x1, x2, x3  X and ,   F:
– Closedness: x1 + x2  X, and   x1  X
– Commutative: x1 + x2 = x2 + x1
– Associative: (x1 + x2) + x3 = x1 + (x2 + x3), (x) = ()x
– Distributive: (x1 + x2) = x1 + x2, (+)x1 = x1 + x1
– Unit elements:  0  X & 1  F s.t. x1+0 =x1 and 1x1 = x1
– Inverse: For all x1,  x2 s.t. x1 + x2 = 0 (additive inverse)

Lecture 3 5
Examples
 Examine the following, and see if they are fields or
not: (R2, R), (Rn, R), (Cn, C), (Cn, R), (Rn, C) with the
standard matrix addition and scalar-matrix
multiplication
 x11   x 21   x11  x 21   x11 
       
 x12   x 22   x12  x 22   x12 
x1  ,x  , x  x2  ,   x1 
 :  2  :  1  :   : 
       
 x1n   x 2n   x1n  x 2 n   x1n 

– (Rn, R), (Cn, C), and (Cn, R): Yes

– (Rn, C): No. No closedness under scalar multiplication
• The idea of a linear space goes beyond Euclidean spaces
Lecture 3 6
 What is a piecewise continuous function?
Which of the following are piecewise
continuous?

t0

lim y( t ) ~ Right hand lim it lim y( t ) ~ Left hand lim it

t t 0 ,t  t 0 t t 0 ,t  t 0

– y(t) is piecewise continuous iff y(t) is continuous

except at a finite number of points. At those points,
y(t) has well defined left and right-hand limits
– Yes. No. No.
Lecture 3 7
 Consider (Y, R), where Y is the set of all real-valued
piecewise continuous functions defined over (-, ) with
the following operations:

y1

y2

y1+y2 y1
y1

• Is (Y, R) a vector space?

• Yes. It satisfies all the requirements
Lecture 3 8
• Now define (Y, R) with Y as the set of solutions to
y  3y  2y  2

– Is this a vector space? Why or why not?

– Suppose that y1  Y and y2  Y. How about y1 + y2
and y1?
y1  3y 1  2 y1  2
y 2  3y 2  2 y 2  2

y1  y2   3y 1  y 2   2y1  y 2   4  2

y1  3y 1  2y1  2  2 except when   1

– (Y, R) is not a vector space ~ No closedness

Lecture 3 9
• How about let Y be the set of solutions to
y  3y  2y  0

– In this case, (Y, R) is a vector space

• How about (Rn[s], R), where Rn[s] is the set of
polynomials of degree less than n
n 1 n 1
i
x1   c1is , x 2   c2isi
i 0 i 0
n 1 n 1
x1  x 2   c1i  c2i s , x1   c1isi
i
i 0 i 0

– It is a vector space, meeting all the

requirements
Lecture 3 10
 How about (Rn[s], R(s)), where where Rn[s] is the
set of polynomials of degree less than n and R(s) is
the set of rational functions with real coefficients?
– No. No closedness
 How about (Rn(s), R), where Rn(s) is the set of n-
vectors of rational functions with real coefficients?
– Yes
 How about (Rn(s), R(s))?
– Yes

Lecture 3 11
Subspace
 (X, F) is a linear space, and Y  X. (Y, F) is a
subspace of (X, F) iff (Y, F) itself is a linear space
– (Y, F) is a subspace if 1y1 + 2y2  Y for all y1,
y2  Y and 1, 2  F
– Other conditions are either automatically satisfied
or are covered by this one
– Consider (R2, R). Is the set of elements satisfying
x1 - 2x2 + 1 = 0 a subspace?

Y: x1-2x2+1=0

– Not a subspace ~ 0 is not included. No closedness

Lecture 3 12
 Then how about the set of elements
satisfying x1 - 2x2 = 0?
Y: x1-2x2=0

– Yes. In fact, any straight line passing through 0

• What would be a subspace for (R3, R)?
– Any plane or straight line passing through 0
• Recall that (Rn[s], R) is a vector space, where
Rn[s] is the set of polynomials of degree < n
– (R4[s], R) is a subspace of (R5[s], R)

Lecture 3 13
 Let (X, F) be a vector space
– Given any set of vectors {xi}i=1 to n, xi X.
– Form the set of linear combinations
n 
Y    i x i , i  F 
i1 

– Then (Y, F) is a linear space, and is a subspace of

(X, F).
– It is the space spanned by {xi}i=1 to n

Lecture 3 14
Introduction to System
Theory and Linear Algebra

Lecture 3 15
 Linear Spaces and Linear
Operators
 Field F: A set of elements (scalars) and two operations
"+" (addition) & "" (multiplication) s.t. for all , , and 
 F:
– Closedness:  +   F and     F
– Commutative:  +  =  +  and    =   
– Associative: (+)+ = +(+) and () = ()
– Distributive:   ( + ) =    +   
– Unit elements:  0 and 1 s.t.  + 0 =  and   1 = 
– Inverse elements: For any ,   s.t.  +  = 0
(additive inverse). For any   0,   s.t.    = 1
(multiplicative inverse)
Lecture 3 16
Linear Spaces over a Field
 A vector space (or a linear space) over a field, denoted as
(X, F), consists of
– X ~ A set of elements (vectors);  F ~ A field
– "+" ~ Vector Addition;  "" ~ Scalar multiplication
s.t. for all x1, x2, x3  X and ,   F:
– Closedness: x1 + x2  X and   x1  X
– Commutative: x1 + x2 = x2 + x1
– Associative: (x1 + x2) + x3 = x1 + (x2 + x3), (x) =
()x
– Distributive: (x1 + x2) = x1 + x2, (+)x1 = x1 +
x1 0
Lecture 3 17
 - Unit elements:  0  X & 1  F s.t. x1+ 0 = x1 and
1x1 = x1
- Inverse: For all x1,  x2 s.t. x1 + x2 = 0 (additive
inverse)
 (X, F) is a linear space, and Y  X. (Y, F) is a
subspace of (X, F) iff (Y, F) itself is a linear space
– (Y, F) is a subspace if 1y1 + 2y2  Y for all y1, y2
 Y and 1, 2  F
– Given any set of vectors {xi}i=1 to m, xi X, form the
set of linear combinations
m 
Y    i x i , i  F 
i 1 
– Then (Y, F) is a linear space, and is a subspace of (X, F)
– It is the space spanned by {xi}i=1 to m
Lecture 3 18
Linear Independence
 A set of vectors {x1, x2, .., xm} in (X, F) is linearly
dependent iff  {1, 2, .., m} in F, not all zero, s.t.
1x1 + 2x2 + .. + nxm = 0 (*)
– If (*) holds and assume for example that 1  0, then
x1 = -[2x2 + .. + nxm]/1
i.e., x1 is a linear combination of {i}i=2 to m
 If the only set of {i}i=1 to m s.t. the above holds is
1 = 2 = .. = m = 0
then {xi}i=1 to m is said to be linearly independent
– None of xi can be expressed as a linear combination
of the rest
Lecture 3 19
 – A linearly dependent set ~ Some redundancy in the
set
Example. Consider the following vectors:
x3 x4
x1
x2
• For the following sets, are they linearly dependent
or independent?
– {x1} ~ Linearly independent
– {x1, x2} ~ Linearly dependent
– {x1, x3} ~ Linearly independent
– {x1, x3, x4} ~ Linearly dependent
Lecture 3 20
  Are the following vectors LD or LI?
 2 1  1 
x1  3, x 2  2, x 3  4 How to find it out?
     
4 5 7
– {x1, x2, .., xn} are LD iff  {1, 2, .., n}, not all
zero, s.t. 1x1 + 2x2 + .. + nxn = 0
 1   1 
   
x1 x 2 ... x n   2   0, or A  2   0, with A  x1 x 2 ... x n 
 :   : 
    Need |A| = 0 to be LD
 n  n
2 1 1
A3 2 4   10  0  These vectors are LI
4 5 7
Lecture 3 21
Example:  1   s  10 
 s5   s  5s  6 
x1   , x2   
1  1
   
 s  10   s6 

– Are they LI in (R2(s), R), where R(s) is the set of

rational functions with real coefficients?
 1 s  6  2 s  10   1  2 s  61  102 
 s  5s  6   s  5s  6 
1x1  2 x 2    
1 s  6  2 s  10  1  2 s  61  102 
   
 s  6s  10    s  6s  10  

– For the above to be zero, 1 = 2 = 0

– Consequently x1 and x2 are linearly independent

Lecture 3 22
  1   s  10 
Example (Continued)  
x1   s  5 , x 2   s  5s  6 
 
1 1
   
 s  10   s6 

• Are the same vectors LI in (R2(s), R(s))?

– If we let 1 1
1  , 2  , then
s6 s  10
 1   1 
 s  5s  6   s  5s  6 
1x1  2 x 2      0
1 1
   
 s  6s  10   s  6s  10 

– They are linearly dependent in (R2(s), R(s))

– Linearly dependence or independence depends on the
vector space under consideration
Lecture 3 23
Dimension
 For (X, R), the maximum number of LI vectors is called
the dimension of the space
x3 x4
Examples. For the following, D = ?
x1
– (R , R)
2 ~ 2
x2
– (Rn, R) ~ n
• How about (X, R), where X  the set of real-valued
piecewise continuous functions over (-, )
– Let xi = ti, i = 0, 1, 2, ...
• If  iti  0, then i  0  i
• {ti} are LI  Dimension = 
• Let Y be the set spanned by {ti}i from 0 to n-1
– (Y, R) is a subspace of (X, R) with dimension = n
Lecture 3 24
 Consider (Y, R) with Y as the set of solutions
to
y  3y  2y  0
– We know that (Y, R) is a vector space. Dimension = ?
– A linear differential equation with constant coefficient. Use
Laplace transform to solve it

s2ŷ(s)  sy(0 )  y (0 ) 3sŷ(s)  y(0 )y  2ŷ(s)  0

s2  3s  2ŷ(s)  sy(0 )  y (0 )  3y(0 )
sy(0 )  y (0 )  3y(0 )
ŷ(s)  y( t )  c1e  t  c2e  2 t
s  1s  2 

Lecture 3 25
– 2nd order system  2 natural modes and 2 ICs  D = 2
– Generally, N'th order system with N ICs, D = N

– Basis, Representation, and Orthonormalization

 Relationship among a set of vectors: Linear
dependence and linear independence
 Dimension of a linear space

• The base of a linear space: Basis

 Representations of a vector in term of a basis

 Relationship among representations for different

bases

Lecture 3 26
  Generalization of the idea of length: Norms
 A sense of orientation: Inner Product

 The concept of perpendicularity:

Orthogonality
 Gram-Schmidt Process to obtain orthonormal
vectors
 Projection and Orthogonal Projection
Theorem
– Linear Operators and Representations

Lecture 3 27
Basis and Representations
 Basis: The base, foundation, or chief
supporting factor of anything
 A set of LI vectors {e1, e2, .., en} of (X, F)
is said to be a basis of X if every vector in
X can be expressed as a unique linear
combination of them

Lecture 3 28
 • They span X
• For any x  X, then  {1, 2, .., n}
s.t.
n
x  1e1  2e2  ..  ne n   iei 1 
i 1  
x  e1 e 2 ... e n   2 
: 
x  e1 e2 ... e n     ~  
 n

– : Representation of x with respect to the basis

Lecture 3 29
Example: Consider (R3[s], R), where R3[s] is the set of
polynomials of real coefficient with degree less than 3
– D = ? An example of a basis?
– Dimension = 3
– A basis: {e1 = s2, e2 = s, e3 = 1}
– Suppose x = 5s2 + 4s + 7. What is its representation?
 5  5
 2

 
x  s s 1  4
 
    4
~ A representation
7 7
   
– A vector, such as x = 5s2 + 4s + 7, need not be in the form of
a vector
– But a vector in a finite dimensional linear space always has a
vector representation in terms of a basis
Lecture 3 30
Q. What qualifies to be a basis?
Theorem: In an n-dimensional vector space, any set of
n LI vectors qualifies as a basis
Proof:
– Let {e1, e2, .., en} be linearly independent
– For any x  X, {x, e1, e2, .., en} are linearly dependent
–  {0, 1, 2, .., n} such that

0x  1e1  2e2  ...  ne n  0

– 0  0. For otherwise, {e1, e2, .., en} are not linearly
independent. Thus
1 n
x   1e1  2e2  ...  n e n    iei
0 i 1
– Any x can be expressed as a linear combination of them

Lecture 3 31
• Is the combination unique here?
– Suppose that  another linear combination
~ n n
x   iei   i ei
i 1 i 1

 i  i  ei  0
n ~
i 1
~ ~
i  i  0 or i  i for all i ~ since {e1, e2, .., en} are LI

 What can be said now?

– The linear combination is unique
– {e1, e2, .., en} is a basis, and the proof is completed

Lecture 3 32
Example (Continued): The set of solutions to
y  3y  2y  0

– We showed that e1 = e-t and e2 = e-2t

form a basis
– Is it the unique one?
– If not, give another one

e1  e  t  e  2 t and e2  e  t  2e  2 t

– They form a basis since they are LI and span the two
dimensional space
– There are in fact an infinite number of bases
Lecture 3 33
Example (Continued): x = 5s2 + 4s + 7
– {e1 = s2, e2 = s, e3 = 1}. What would be another
basis?
e1  s2  s, e2  s  1, e3  1
– What is the representation of the above x? 5
 
x  5 s2  s  1s  1  8  1  e1 e2
 
e3    1
8
 
– When a basis is changed, the representation is
changed correspondingly
– What is the new representation? How do we formalize
this process? Why bother?
– Sometimes it is much easer to work with a particular
basis then working with others, e.g., controllable
canonical form or observable canonical form  Need
to know how to do it
Lecture 3 34
Change of Basis
 Any n LI vectors qualify as a basis
e1 e2 ... en    e1 e2 ... en   
– For a particular x, the representation is unique for each
basis
Q. Given , how to find?
– Express x in terms of e, e in terms ofe  x in terms
ofe
 11 
p  1i 
p
   
e1  e1 e2 ... en   p12 
ei  e1 e2 ... en   p1i 
 :   : 
  ~p   ~ pi
 p1n  1  p1i 
e1 e2 ... en   e1 e2 ... en  p1 p2 ... pn 
 P ~ nn
Lecture 3 35
 e1 e2 ... en   e1 e2 ... en  P
x  e1 e2 ... e n    e1 e2 ... en  P

 e1 e2 ... en      P

ith column of P: Representation of ei

w.r.t. the set of new basis

– Conversely, ith column of Q: Representation ofei

  Q w.r.t. the set of existing basis
– What is the relationship between P and Q?
  P  PQ   PQ  I, or P  Q 1

Lecture 3 36
Example (Continued) (R3[s], R)
– {e1 = s2, e2 = s, e3 = 1}
– {e1  s 2  s, e2  s  1, e3  1}

• What is P?
1
2

2  
e1  s  s  s s  1 1   1 
 1  ~ p1
 
0
2  
e2  s  s  s s  1 1  1  
  1 ~ p2
 
 0 1 0 0
 2

 
e3  1  s  s s  1 1  0 
 
P    1 1 0
 1  ~ p3  1 1 1
   
Lecture 3 37
• With {e1 = s2, e2 = s, e3 = 1} and x = 5s2 + 4s + 7:

 5
 
   4 Now {e1  s2  s, e2  s  1, e3  1}
7
 

• What is?
1 0 0  5  5 
    
  P    1 1 0   4     1
 1 1 1  7  8  ~ As expected
    

Lecture 3 38
 1  0
Example. Consider R) with e1   0 , e2   1 
(R2,
   
_  cos  
e2
e2 _ e1  cos e1  sin e2  e1 e 2   ~ q1
 e1  sin  
   sin  
e2   sin  e1  cos  e2  e1 e 2   ~ q2
e1  cos  
 cos   sin   1 1  cos  sin  
Q , P  Q  Q  
 sin  cos     sin  cos  
with | Q | cos  cos   sin  sin   cos(  )   P and   Q 
– For  =  = 45 and x = (1 1)T
 1 1  _e2
  x
  2 2 1   2  _ e1
 1 1 1  0  e2
 2 2 e1
Lecture 3 39
– Basis, Representation, and Orthonormalization
 Relationship among a set of vectors: Linear
dependence and linear independence
 Dimension of a linear space

 The base of a linear space: Basis

 Representations of a vector in term of a basis

 Relationship among representations for different

bases
• Generalization of the idea of length: Norms
 A sense of orientation: Inner Product

 The concept of perpendicularity: Orthogonality

 Gram-Schmidt Process to obtain orthonormal vectors

 Projection and Orthogonal Projection Theorem

– Linear Operators and Representations

Lecture 3 40
THE END

Lecture 3 41