Lecture2 PDF

Filtering and Identification
Lecture 2:
Random processes and Linear Least
Squares
Michel Verhaegen and Jan-Willem van Wingerden

1/35
Delft Center for Systems and Control

Delft University of Technology
Overview
• Dynamical Systems and Random
Processes
• Deterministic least squares (LS) problems
• Statistical Properties LS estimates
• Application of LS to System Identification
2/35

Dynamical Systems (Chapter 3)
v(k)
u(k) y(k)
Σ
“input” “output”
Σ : LTI (Linear Time Invariant) System with

representation:
3/35

v(k)
u(k) y(k)
Σ

representation:
P∞
• IIR: y(k) =
ℓ=−∞ g(ℓ)u(k − ℓ) + v(k)
3/35

v(k)
u(k) y(k)
Σ

representation:
P∞
• IIR: y(k) =
ℓ=−∞ g(ℓ)u(k − ℓ) + v(k)
• (Or)
( State Space Model:
x(k + 1) = Ax(k) + Bu(k)
y(k) = Cx(k) + Du(k) + v(k)
• (Or) · · · .
3/35

Signals (Chapter 4: 4.1 - 4.4)
v(k)
u(k) y(k)
Σ
4/35

v(k)
u(k) y(k)
Σ
1. The quantities u(k), y(k) are (measured) Signals (OR)

discrete time sequences (OR) sampled input-output data
sequences generally denoted as:
Given :{u(k), y(k)}N

k=1 N ∈N
4/35

v(k)
u(k) y(k)
Σ
1. The quantities u(k), y(k) are (measured) Signals (OR)

discrete time sequences (OR) sampled input-output data
sequences generally denoted as:
Given :{u(k), y(k)}N

k=1 N ∈N
2. The quanity v(k) is a stochastic process (generally

unknown) (OR) a discrete time sequence of Random
variables with a (Gaussian) probability density. In this
course we restrict to its mean and covariance function.
4/35

Stochastic Signals
e(k) Σn v(k)
In this course: A stochastic process v(k) is stationary and

“assumed” to result by filtering zero-mean white noise e(k)
with an LTI system Σn
5/35

Stationarity
Definition wide sense stationarity (WSS): A
random process x(k) ∈ R is WSS if the following
three conditions are satisfied:
1. mean is constant, mx (k) = mx
2. auto-correlation function
Rx (k, ℓ) = E[x(k)x(ℓ)] only depends on the
lag k − ℓ
3. variance E[(x(k) − mx )2 ] is finite
6/35

Stationarity
lag k − ℓ
Hence,
Rx (k, ℓ) = Rx (k − ℓ) = Rx (τ ) =
6/35

Stationarity
lag k − ℓ
Hence,
Rx (k, ℓ) = Rx (k − ℓ) = Rx (τ ) = E[x(k)x(k − τ )]
6/35

White noise e(k) ∼ (0, σe2)
A zero-mean white noise sequence (ZMWN):
The random process e(k) is a ZMWN if it has
mean zero and its auto-covariance
(auto-correlation) function equals:
(
σe2 for k = ℓ
E[e(k)e(ℓ)] =
0 otherwise
Denoted as Re (τ ) = E[e(k)e(k − τ )] = σe2 ∆(τ ),

with ∆(τ ) the unit-pulse.
7/35

RPs in the time-domain
If the (real) RPs x(k) and y(k) are wide sense
stationary (WSS), then these RPs are fully
characterized in the time-domain by their means
E[x(k)] = mx , E[y(k)] = my
and their auto-, cross-covariance functions:
h i
Cx (τ ) = E (x(k) − mx )(x(k − τ ) − mx )T
h i
Cxy (τ ) = E (x(k) − mx )(y(k − τ ) − my )T
8/35

An equivalent characterization is to replace
the auto-, cross-covariance functions
by the auto-, cross-correlation functions:
h i h i
Rx (τ ) = E x(k)x(k − τ )T = E x(k + τ )x(k)T
h i
Rxy (τ ) = E x(k)y(k − τ )T
9/35

The numerical calculation may proceed via the
assumption of ergodicity which enables to proof
relationships like:
" N
#
1 X
Pr lim x(k)x(k − τ )T = Rx (τ ) = 1
N →∞ N
k=1
10/35

Overview
• Dynamical Systems and Random Processes
11/35

Deterministic least squares problem
Given F ∈ RN ×n , y ∈ RN , the problem is:
min ǫT ǫ subject to: y = Fx + ǫ

x
The argument that minimizes this problem is the

b.
least squares solution and is denoted as, x
12/35


x
The argument that minimizes this problem is the

b.
least squares solution and is denoted as, x
For all x ∈ Rn , it satisfies,
bk22 ≤ky − F xk22
ky − F x
12/35

bk22 ≤ky − F xk22

ky − F x
y
f2
Fx
ǫ
yb
span(F ) f1
h i
where yb = F x
b = f1 f2 x
b
13/35

The classical solution
Lemma: Let the matrix F in
x
b is:
have full column rank, then the solution x
T
−1 T
b= F F
x F y
This follows from the normal equations:

T ↓
b=F T y
F Fx
14/35

Proof of the classical solution
Via the completion of squares. For all x and x̂ satisfying:
(F T F )x̂ = F T y
we can write the least squares cost function as:
ky − F xk22 = (y − F x)T (y − F x)
= y T y − xT F T y − y T F x + xT F T F x
= y T y − y T F x̂+ (x − x̂)T F T F (x − x̂)
15/35

Via the completion of squares. For all x and x̂ satisfying:
(F T F )x̂ = F T y
we can write the least squares cost function as:
ky − F xk22 = (y − F x)T (y − F x)
= y T y − xT F T y − y T F x + xT F T F x
= y T y − y T F x̂+ (x − x̂)T F T F (x − x̂)
Therefore,
arg minx ky − F xk22 = x̂
15/35

  
h i y T y −y T F 1
2
ky − F xk2 = 1 x T    
−F T y F T F x
| {z }
M
   
I −b xT yT y − yT F x
b 0 I 0
M=      ,
0 I 0 FTF −bx I
b satisfying,
for x
b = F T y.
FTFx
16/35

Proof of the classical solution (Ct’d)
  
h i y T y −y T F 1
ky − F xk22 = 1 x T    
−F T y F T F x
    
h i I −b xT yT y − yT F x
b 0 I 0 1
= 1 x T        
0 I 0 FTF −bx I x
  
h i yT y − yT F xb 0 1
= 1 (x − xb) T    
0 FTF x−xb
b satisfying F T F x
for x b = F T y.
ky − F xk22 = (y T y − y T F x b)T F T F (x − x
b) + (x − x b).
17/35

Overview
18/35

“Measurement Errors” ǫ ∼ (0, I)

x
19/35

“Measurement Errors” ǫ ∼ (0, I)

x
• F is a known full column rank matrix.

• x an unknown, deterministic vector.
• ǫ is a zero-mean random vector with
E[ǫǫT ] = I
19/35

Linear Estimators for the least squares problem
Least squares solution

T
−1
b= F F
x FTy
is a linear estimator: it is linear in y.
Definition: Linear estimator for x given y has the
form:
e = My
x
with M ∈ Rn×N
20/35

Unbiased and minimum variance
cy is unbiased if
b=M
The linear estimator x
x − x] = 0
E[b
21/35

Unbiased and minimum variance
cy is unbiased if
b=M
The linear estimator x
x − x] = 0
E[b
b=M
The linear estimator x cy is called the
minimum variance estimator if
h i h i
E (b x − x)T ≤ E (e
x − x)(b x − x)(ex − x)T
e = M y.
for all linear estimators x
21/35

The Gauss-Markov theorem
The least squares solution
T
−1 T
b= F F
x F y
is an unbiased minimum variance estimate
(UMVE) and has covariance matrix:
T T
−1
b)(x − x
E[(x − x b) ] = F F
22/35

Proof of the Gauss-Markov theorem
Linear Estimator which is UNBIASED
e = My = MF x+Mǫ ⇒ x
x e − x = (M F − I)x + M ǫ
23/35

x e − x = (M F − I)x + M ǫ
e−x
Consider the mean of x
x − x] = (M F − In )x + M E[ǫ] = (M F − I)x
E[e
The linear estimator is unbiased provided,
23/35

x e − x = (M F − I)x + M ǫ
e−x
Consider the mean of x
x − x] = (M F − In )x + M E[ǫ] = (M F − I)x
E[e
The linear estimator is unbiased provided,
x − x] = 0 ⇔ M F = I
E[e
−1
The least squares estimator M = FTF FT
clearly satisfies M F = In
23/35

Proof of Minimum Variance Property
Recall
e − x = (M F − I)x + M ǫ = M ǫ
x
Then, the covariance matrix of the Unbiased linear
e = hM y with M satisfying
estimate x i M F = I:
E (e x − x)T
x − x)(e = M E[ǫǫT ]M T
= MMT
For the least squares solution x̂(M̂ = (F T F )−1 F T ), its

covariance
h matrix equals,i
E (b x − x)T = (F T F )−1 F T F (F T F )−1
x − x)(b
= (F T F )−1
24/35

lsq_mvar.m
25/35

Overview
26/35

A simple System Identification Problem
e(k)
u(k) y(k)
Σ
With Σ given by the following (2nd order) difference equation:
y(k) + a1 y(k − 1) + a2 y(k − 2) = b1 u(k − 1) + b2 u(k − 2) +e(k)

| {z } | {z }
AR X
27/35

e(k)
u(k) y(k)
Σ

| {z } | {z }
AR X
This can be written in transfer function

 form: 
b1 q +b2 q
−1 −2
G(q) = 1+a
y(k) = G(q)u(k) + H(q)e(k)  1q
−1 +a q −2
2 
H(q) = 1+a1 q−11+a2 q−2
27/35

e(k)
u(k) y(k)
Σ

| {z } | {z }
AR X
This can be written in transfer function

 form: 
b1 q +b2 q
−1 −2
G(q) = 1+a
y(k) = G(q)u(k) + H(q)e(k)  1q
−1 +a q −2
2 
H(q) = 1+a1 q−11+a2 q−2
Then the identification problem is: {u(k), y(k)}N

k=1 → Σ̂
27/35

Solving the ARX Identification Problem
We can denote the difference equation as:  
a1
 
h i a2 
 
y(k) = −y(k − 1) −y(k − 2) u(k − 1) u(k − 2)  +e(k)
 b1 
 
b2
If we define this data relationship as the k − 2th row of the

following matrix equation (for k = 3 : N ):
y = Fx + ǫ
h i
then the ARX parameters a1 a2 b1 b2 can be found by
solving the least squares problem:
min ǫT ǫ subject to: y = F x + ǫ
x
28/35

lsq_demo.m
Consider the followig AR(MA)X model:
y(k) − 1.5y(k − 1) + 0.7y(k − 2) = u(k − 1) − u(k − 2) + e(k)

− e(k − 1) + 0.2e(k − 2)
with u(k) and e(k) independent zero-mean white noise

sequences of unit variance and length 1000. Using
{u(k), y(k)}903
k=1 estimate the parameters of a 2 nd
order ARX
model for (a) MA part zero and (b) MA as given!.
29/35

The use of the QR factorization
The QR-Theorem: Let A ∈ Rm×n (m ≥ n), then

there exists an orthogonal matrix Q ∈ Rm×m that
can be partitioned as:
h i
Q = Q1 Q2 Q1 ∈ Rm×n
such that,
" #
T R
Q A= with R ∈ Rn×n and R upper-triangular
0
[or the matrix A is factorized as Q1 R.]

30/35

QR Solution to LS problem
LSQR-Theorem: Consider the LS problem minx ky − F xk2 and
consider the following QR factorization of F and the application
of QT to y as,
     
h i R QT1 d1
F = Q1 Q2     y=  
0 QT2 d2
Consider the matrix F to have full column rank, then the LS

b and the LS residual satisfy:
solution x
b = R−1 d1
x bk2 = kd2 k2
ky − F x
31/35

SensNeq.m
32/35

Summary of Lecture 2
• Refreshment of characterization of RPs in
Time (- and Frequency) domain
• The linear least squares problem: unknown x
deterministic.
• Gauss-Markov theorem (MVUE).
• Ready for the first Home Work - download
Now!
33/35

Next Instruction session
Preparation:
Study Chapters 2(2.6-2.7) and 4 (4.1 - 4.5.2)
Get Homework 1
Next lecture:
Addressing your questions on Homework 1.
Tuesday 17-11-2015
34/35

Lecture2 PDF

Uploaded by

Document Information

Original Description:

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Lecture2 PDF

Uploaded by

Copyright:

Available Formats

Filtering and Identification

Michel Verhaegen and Jan-Willem van Wingerden

Delft Center for Systems and Control

Delft Center for Systems and Control

Σ : LTI (Linear Time Invariant) System with

Delft Center for Systems and Control

Σ : LTI (Linear Time Invariant) System with

Delft Center for Systems and Control

Σ : LTI (Linear Time Invariant) System with

Delft Center for Systems and Control

Delft Center for Systems and Control

1. The quantities u(k), y(k) are (measured) Signals (OR)

Given :{u(k), y(k)}N

Delft Center for Systems and Control

1. The quantities u(k), y(k) are (measured) Signals (OR)

Given :{u(k), y(k)}N

2. The quanity v(k) is a stochastic process (generally

Delft Center for Systems and Control

In this course: A stochastic process v(k) is stationary and

Delft Center for Systems and Control

Delft Center for Systems and Control

Delft Center for Systems and Control

Delft Center for Systems and Control

Denoted as Re (τ ) = E[e(k)e(k − τ )] = σe2 ∆(τ ),

Delft Center for Systems and Control

Delft Center for Systems and Control

Delft Center for Systems and Control

Delft Center for Systems and Control

Delft Center for Systems and Control

min ǫT ǫ subject to: y = Fx + ǫ

The argument that minimizes this problem is the

Delft Center for Systems and Control

min ǫT ǫ subject to: y = Fx + ǫ

The argument that minimizes this problem is the

Delft Center for Systems and Control

bk22 ≤ky − F xk22

Delft Center for Systems and Control

This follows from the normal equations:

Delft Center for Systems and Control

we can write the least squares cost function as:

Delft Center for Systems and Control

we can write the least squares cost function as:

Delft Center for Systems and Control

Delft Center for Systems and Control

Delft Center for Systems and Control

Delft Center for Systems and Control

Given F ∈ RN ×n , y ∈ RN , the problem is:

min ǫT ǫ subject to: y = Fx + ǫ

Delft Center for Systems and Control

Given F ∈ RN ×n , y ∈ RN , the problem is:

min ǫT ǫ subject to: y = Fx + ǫ

• F is a known full column rank matrix.

Delft Center for Systems and Control

Least squares solution

Delft Center for Systems and Control

Delft Center for Systems and Control

Delft Center for Systems and Control

Delft Center for Systems and Control

Delft Center for Systems and Control

Delft Center for Systems and Control

Delft Center for Systems and Control

For the least squares solution x̂(M̂ = (F T F )−1 F T ), its