Krylov

Methods in
MOR

Introduction Krylov Subspace Methods in

Moments &
Markov
Parameters Model Order Reduction
Krylov
Subspace

Moment Mohammad Umar Rehman

Matching

Issues with
Krylov PhD Candidate, EE Department, IIT Delhi
Methods umar.ee.iitd@gmail.com
Orthogonalization
Stopping
Criteria
March 8, 2016

1 / 20
 Outline

Krylov
Methods in
MOR
1 Introduction
Introduction

Moments & 2 Moments & Markov Parameters

Markov
Parameters

Krylov 3 Krylov Subspace

Subspace

Moment
Matching 4 Moment Matching
Issues with
Krylov
Methods
Orthogonalization
5 Issues with Krylov Methods
Stopping
Criteria Orthogonalization
Stopping Criteria

2 / 20
 Introduction
Model Reduction Problem Revisited

Krylov Given a MIMO state space model

Methods in
MOR
Eẋ = Ax + Bu
(1)
Introduction
y = Cx
Moments &
Markov where, E, A ∈ Rn×n , B ∈ Rn×m , C ∈ Rp×n ,
Parameters
u ∈ Rm , y ∈ Rp , x ∈ Rn and n is sufficiently large.
Krylov
Subspace

Moment
Matching

Issues with
Krylov
Methods
Orthogonalization
Stopping
Criteria

3 / 20
 Introduction
Model Reduction Problem Revisited

Krylov Given a MIMO state space model

Methods in
MOR
Eẋ = Ax + Bu
(1)
Introduction
y = Cx
Moments &
Markov where, E, A ∈ Rn×n , B ∈ Rn×m , C ∈ Rp×n ,
Parameters
u ∈ Rm , y ∈ Rp , x ∈ Rn and n is sufficiently large.
Krylov
Subspace It is required to obtain the following reduced order model
Moment
Matching Er ż = Ar z + Br u
(2)
Issues with y = Cr z
Krylov
Methods
Orthogonalization
Stopping
where, Er , Ar ∈ Rq×q , Br ∈ Rq×m , Cr ∈ Rp×q ,
Criteria
u ∈ Rm , y ∈ Rp , z ∈ Rq q << n
Er = WT EV, Ar = WT AV, Br = WT B, Cr T = CT V

3 / 20
 Introduction
Model Reduction Problem Revisited

Krylov Given a MIMO state space model

Methods in
MOR
Eẋ = Ax + Bu
(1)
Introduction
y = Cx
Moments &
Markov where, E, A ∈ Rn×n , B ∈ Rn×m , C ∈ Rp×n ,
Parameters
u ∈ Rm , y ∈ Rp , x ∈ Rn and n is sufficiently large.
Krylov
Subspace It is required to obtain the following reduced order model
Moment
Matching Er ż = Ar z + Br u
(2)
Issues with y = Cr z
Krylov
Methods
Orthogonalization
Stopping
where, Er , Ar ∈ Rq×q , Br ∈ Rq×m , Cr ∈ Rp×q ,
Criteria
u ∈ Rm , y ∈ Rp , z ∈ Rq q << n
Er = WT EV, Ar = WT AV, Br = WT B, Cr T = CT V
W, V are suitable Krylov subspace based projection matrices.

3 / 20
 Moments and Markov Parameters

Krylov The transfer function of the system in (1) is

Methods in
MOR
G(s) = C(sE − A)−1 B

Introduction By assuming that A is nonsingular, the Taylor series of this transfer

Moments & function around zero is:
Markov
Parameters i
G(s) = −CA−1 B − C(A−1 E)A−1 Bs − · · · − C(A−1 E) A−1 Bsi − · · ·
Krylov
Subspace

Moment
Coefficients of powers of s are known as moments
Matching i-th moment:
Issues with
Krylov M0i = C(A−1 E)i A−1 B, i = 0, 1, . . .
Methods
Orthogonalization
Stopping
Criteria
Also,
1 di G(s) 
M0i = −
i dsi s=0


is the value of subsequent derivatives of the transfer function G(s) at the

point s = 0
4 / 20
 Contd...

Krylov A different series in terms of negative powers of s is obtained when

Methods in
MOR expanded about s → ∞

G(s) = CE−1 Bs−1 +C(E−1 A)E−1 Bs−2 +· · ·+C(E−1 A)i E−1 Bs−i +· · ·
Introduction

Moments & and the coefficients are known as Markov parameters.

Markov
Parameters

Krylov Model reduction is achieved by the means of matching of Moments

Subspace
(Markov parameters)
Moment
Matching
Explicit moment matching becomes numerically cumbersome for large
Issues with
system order
Krylov
Methods
Orthogonalization
Stopping
Criteria

5 / 20
 Contd...

Krylov A different series in terms of negative powers of s is obtained when

Methods in
MOR expanded about s → ∞

G(s) = CE−1 Bs−1 +C(E−1 A)E−1 Bs−2 +· · ·+C(E−1 A)i E−1 Bs−i +· · ·
Introduction

Moments & and the coefficients are known as Markov parameters.

Markov
Parameters

Krylov Model reduction is achieved by the means of matching of Moments

Subspace
(Markov parameters)
Moment
Matching
Explicit moment matching becomes numerically cumbersome for large
Issues with
system order
Krylov Go for implicit moment matching: Krylov subspace based Projection
Methods
Orthogonalization
Stopping
Criteria

5 / 20
 Remarks 1

Krylov
Methods in
MOR
Asymptotic Waveform Evaluation (AWE) method is based
on explicit moment matching
Introduction

Moments & Matching at s = 0 is known as Padé Approximation, and

Markov
Parameters
steady state response (low frequency) is reflected in the
Krylov reduced order model.
Subspace

Moment
Matching at s → ∞ is known as Partial Realization, and
Matching the reduced order model is a good approximation of the
Issues with
Krylov
HF response.
Methods
Orthogonalization
Matching at s = s0 , i. e. at some arbitrary value of s is
Stopping
Criteria known as Rational Interpolation and is aimed at
approximating system response at specific frequency band
of interest.

6 / 20
 Defining Krylov Subspace

Krylov
Methods in
MOR Kq (A, b) = span{b, Ab, . . . , Aq−1 b},

Introduction A ∈ Rn×n and b ∈ Rn is called the starting vector. q is some given

Moments & positive integer called index of the Krylov sequence.
Markov
Parameters

Krylov
Subspace

Moment
Matching

Issues with
Krylov
Methods
Orthogonalization
Stopping
Criteria

7 / 20
 Defining Krylov Subspace

Krylov
Methods in
MOR Kq (A, b) = span{b, Ab, . . . , Aq−1 b},

Introduction A ∈ Rn×n and b ∈ Rn is called the starting vector. q is some given

Moments & positive integer called index of the Krylov sequence.
Markov
Parameters The vectors b, Ab, . . . , constructing the subspace are called basic
Krylov vectors.
Subspace

Moment
Matching

Issues with
Krylov
Methods
Orthogonalization
Stopping
Criteria

7 / 20
 Defining Krylov Subspace

Krylov
Methods in
MOR Kq (A, b) = span{b, Ab, . . . , Aq−1 b},

Introduction A ∈ Rn×n and b ∈ Rn is called the starting vector. q is some given

Moments & positive integer called index of the Krylov sequence.
Markov
Parameters The vectors b, Ab, . . . , constructing the subspace are called basic
Krylov vectors.
Subspace The Krylov subspace is also known as controllability subspace in
Moment control community.
Matching

Issues with
Krylov
Methods
Orthogonalization
Stopping
Criteria

7 / 20
 Defining Krylov Subspace

Krylov
Methods in
MOR Kq (A, b) = span{b, Ab, . . . , Aq−1 b},

Introduction A ∈ Rn×n and b ∈ Rn is called the starting vector. q is some given

Moments & positive integer called index of the Krylov sequence.
Markov
Parameters The vectors b, Ab, . . . , constructing the subspace are called basic
Krylov vectors.
Subspace The Krylov subspace is also known as controllability subspace in
Moment control community.
Matching
For each state space, there are two Krylov subspaces that are dual to
Issues with
Krylov each other, input Krylov subspace and output Krylov subspace.
Methods Either or both of subspaces can be used as projection matrices for
Orthogonalization
Stopping model reduction.
Criteria

7 / 20
 Defining Krylov Subspace

Krylov
Methods in
MOR Kq (A, b) = span{b, Ab, . . . , Aq−1 b},

Introduction A ∈ Rn×n and b ∈ Rn is called the starting vector. q is some given

Moments & positive integer called index of the Krylov sequence.
Markov
Parameters The vectors b, Ab, . . . , constructing the subspace are called basic
Krylov vectors.
Subspace The Krylov subspace is also known as controllability subspace in
Moment control community.
Matching
For each state space, there are two Krylov subspaces that are dual to
Issues with
Krylov each other, input Krylov subspace and output Krylov subspace.
Methods Either or both of subspaces can be used as projection matrices for
Orthogonalization
Stopping model reduction.
Criteria
The respective method is then called One-Sided/Two-sided

7 / 20
 Input and Output Krylov Subspaces

Krylov Input Krylov subspace

Methods in
MOR n o

q −1
Kq1 A−1 E, A−1 b = span A−1 b, . . . , A−1 E 1 A−1 b

Introduction

Moments & Output Krylov Subspace

Markov
Parameters n
q2 −1 −T o
Kq2 A−T ET , A−T c = span A−T c, . . . , A−T ET


Krylov A c
Subspace

Moment
Matching

Issues with
Krylov
Methods
Orthogonalization
Stopping
Criteria

8 / 20
 Input and Output Krylov Subspaces

Krylov Input Krylov subspace

Methods in
MOR n o

q −1
Kq1 A−1 E, A−1 b = span A−1 b, . . . , A−1 E 1 A−1 b

Introduction

Moments & Output Krylov Subspace

Markov
Parameters n
q2 −1 −T o
Kq2 A−T ET , A−T c = span A−T c, . . . , A−T ET


Krylov A c
Subspace

Moment
Matching V is any basis of Input Krylov Subspace
Issues with W is any basis of Output Krylov Subspace
Krylov
Methods
Orthogonalization
Stopping
Criteria

8 / 20
 Moment Matching: SISO

Krylov
Methods in
MOR

Introduction

Moments &
Markov Theorem If the matrix V used
in (2), is a basis of Krylov
Parameters
subspace Kq1 A−1 E, A−1 b with rank q and matrix W is
Krylov
Subspace chosen such that the matrix Ar is nonsingular, then the first q
Moment moments (around zero) of the original and reduced order
Matching

Issues with
systems match.
Krylov
Methods
Orthogonalization
Stopping
Criteria

9 / 20
 Moment Matching: SISO

Krylov
Methods in
Proof: The zero-th moment of the reduced system is
MOR
 −1
−1
mr0 = cT T
r Ar br = c V W AV
T
WT b
Introduction

Moments &
Markov
The vector A−1 b is in the Krylov subspace and it can be
Parameters written as a linear combination of the columns of the matrix V,
Krylov

∃r0 ∈ Rq : A−1 b = Vr0

Subspace

Moment
Matching

Issues with Therefore,

Krylov
Methods  −1  −1
WT AV WT b = WT AV WT AA−1 b


Orthogonalization
Stopping
Criteria

10 / 20
 Moment Matching: SISO

Krylov
Methods in
Proof: The zero-th moment of the reduced system is
MOR
 −1
−1
mr0 = cT T
r Ar br = c V W AV
T
WT b
Introduction

Moments &
Markov
The vector A−1 b is in the Krylov subspace and it can be
Parameters written as a linear combination of the columns of the matrix V,
Krylov

∃r0 ∈ Rq : A−1 b = Vr0

Subspace

Moment
Matching

Issues with Therefore,

Krylov
Methods  −1  −1
WT AV WT b = WT AV WT AA−1 b


Orthogonalization
Stopping
Criteria
 −1
= WT AV WT AVr0

10 / 20
 Moment Matching: SISO

Krylov
Methods in
Proof: The zero-th moment of the reduced system is
MOR
 −1
−1
mr0 = cT T
r Ar br = c V W AV
T
WT b
Introduction

Moments &
Markov
The vector A−1 b is in the Krylov subspace and it can be
Parameters written as a linear combination of the columns of the matrix V,
Krylov

∃r0 ∈ Rq : A−1 b = Vr0

Subspace

Moment
Matching

Issues with Therefore,

Krylov
Methods  −1  −1
WT AV WT b = WT AV WT AA−1 b


Orthogonalization
Stopping
Criteria
 −1
= WT AV WT AVr0
= r0

10 / 20
 Contd...

Krylov With this, mr0 becomes

Methods in
MOR  −1
mr0 = cT V WT AV WT b = cT Vr0 = cT A−1 b = m0
Introduction
For the next moment (first moment) consider the following result:
Moments &
Markov  −1  −1    −1
Parameters
WT AV WT EV WT AV WT b = WT AV WT EVr0
Krylov
Subspace  −1
= WT AV WT EA−1 b
Moment
Matching

Issues with and the fact that A−1 EA−1 b is also in the Krylov subspace can be
Krylov
Methods written as A−1 EA−1 b = Vr1
Orthogonalization
Stopping
Criteria

11 / 20
 Contd...

Krylov
Methods in
MOR
Thus,
 −1  −1
WT AV WT AA−1 EA−1 b = WT AV WT AVr1
Introduction


Moments &
Markov
Parameters

Krylov
Subspace

Moment
Matching

Issues with
Krylov
Methods
Orthogonalization
Stopping
Criteria

12 / 20
 Contd...

Krylov
Methods in
MOR
Thus,
 −1  −1
WT AV WT AA−1 EA−1 b = WT AV WT AVr1
Introduction


Moments &
Markov
Parameters = r1
Krylov
Subspace

Moment
Matching

Issues with
Krylov
Methods
Orthogonalization
Stopping
Criteria

12 / 20
 Contd...

Krylov
Methods in
MOR
Thus,
 −1  −1
WT AV WT AA−1 EA−1 b = WT AV WT AVr1
Introduction


Moments &
Markov
Parameters = r1
Krylov
Subspace

Moment
Matching

Issues with
 −1  −1
Krylov mr1 = cT V WT AV WT EV WT AV WT b
Methods
Orthogonalization
Stopping
Criteria

12 / 20
 Contd...

Krylov
Methods in
MOR
Thus,
 −1  −1
WT AV WT AA−1 EA−1 b = WT AV WT AVr1
Introduction


Moments &
Markov
Parameters = r1
Krylov
Subspace

Moment
Matching

Issues with
 −1  −1
Krylov mr1 = cT V WT AV WT EV WT AV WT b
Methods

= cT Vr1 = cT A−1 EA−1 b

Orthogonalization
Stopping
Criteria

12 / 20
 Contd...

Krylov
Methods in
MOR
Thus,
 −1  −1
WT AV WT AA−1 EA−1 b = WT AV WT AVr1
Introduction


Moments &
Markov
Parameters = r1
Krylov
Subspace

Moment
Matching

Issues with
 −1  −1
Krylov mr1 = cT V WT AV WT EV WT AV WT b
Methods

= cT Vr1 = cT A−1 EA−1 b

Orthogonalization
Stopping
Criteria

= m1

12 / 20
 Remarks 2

Krylov
Methods in
MOR
For the second moment, the results

2 of first moment can
Introduction
be used and the fact that A−1 E A−1 b can be written
Moments & as a linear combination of columns of matrix V
Markov
Parameters The proof can be continued by repeating these steps
Krylov (Induction) until mr(q−1) = m(q−1) i.e. q moments match.
Subspace

Moment The method discussed above was one-sided as we did not

Matching
go for computing W. Usually, W = V is chosen
Issues with
Krylov
Methods
In two-sided method W is chosen to be the basis of output
Orthogonalization Krylov subspace, then 2q moments can be matched.
Stopping
Criteria
Proof is similar for matching Markov parameters and the
MIMO case [3,4].

13 / 20
 Issues with Krylov Methods

Krylov
Methods in
MOR

Introduction

Moments &
Markov
Parameters Major issues with Krylov Subspace based MOR Methods:
Krylov
Subspace
1 Orthogonalization
Moment 2 Stopping Point of Iterative Scheme
Matching

Issues with
Krylov
Methods
Orthogonalization
Stopping
Criteria

14 / 20
 Orthogonalization

Krylov
Methods in
MOR

Introduction The Krylov vectors are known to lose independence readily

Moments & and tend to align towards the dominant vector, even for
Markov
Parameters moderate values of n and q.
Krylov
Subspace

Moment
Matching

Issues with
Krylov
Methods
Orthogonalization
Stopping
Criteria

15 / 20
 Orthogonalization

Krylov
Methods in
MOR

Introduction The Krylov vectors are known to lose independence readily

Moments & and tend to align towards the dominant vector, even for
Markov
Parameters moderate values of n and q.
Krylov
Subspace The remedy lies in constructing an orthogonal basis using
Moment Gram-Schmidt process.
Matching

Issues with
Krylov
Methods
Orthogonalization
Stopping
Criteria

15 / 20
 Orthogonalization

Krylov
Methods in
MOR

Introduction The Krylov vectors are known to lose independence readily

Moments & and tend to align towards the dominant vector, even for
Markov
Parameters moderate values of n and q.
Krylov
Subspace The remedy lies in constructing an orthogonal basis using
Moment Gram-Schmidt process.
Matching

Issues with
However, classical GS is also known to be unstable
Krylov
Methods
Orthogonalization
Stopping
Criteria

15 / 20
 Orthogonalization

Krylov
Methods in
MOR

Introduction The Krylov vectors are known to lose independence readily

Moments & and tend to align towards the dominant vector, even for
Markov
Parameters moderate values of n and q.
Krylov
Subspace The remedy lies in constructing an orthogonal basis using
Moment Gram-Schmidt process.
Matching

Issues with
However, classical GS is also known to be unstable
Krylov
Methods Go for Modified GS methods —
Orthogonalization
Stopping
Criteria

15 / 20
 Orthogonalization

Krylov
Methods in
MOR

Introduction The Krylov vectors are known to lose independence readily

Moments & and tend to align towards the dominant vector, even for
Markov
Parameters moderate values of n and q.
Krylov
Subspace The remedy lies in constructing an orthogonal basis using
Moment Gram-Schmidt process.
Matching

Issues with
However, classical GS is also known to be unstable
Krylov
Methods Go for Modified GS methods —
Orthogonalization
Stopping Arnoldi (Unsymmetric A)
Criteria

15 / 20
 Orthogonalization

Krylov
Methods in
MOR

Introduction The Krylov vectors are known to lose independence readily

Moments & and tend to align towards the dominant vector, even for
Markov
Parameters moderate values of n and q.
Krylov
Subspace The remedy lies in constructing an orthogonal basis using
Moment Gram-Schmidt process.
Matching

Issues with
However, classical GS is also known to be unstable
Krylov
Methods Go for Modified GS methods —
Orthogonalization
Stopping Arnoldi (Unsymmetric A) / Lanczos (Symmetric A)
Criteria

15 / 20
 Arnoldi Algorithm
Using Modified Gram-Schmidt Orthogonalization

Krylov
Methods in
MOR Algorithm 1 Arnoldi
b
1: Start: Choose initial starting vector b, v = kbk
Introduction
2: Calculate the next vector: v̂i = Avi−1
Moments &
Markov Orthogonalization:
Parameters
3: for j = 1 to i − 1 do
Krylov
Subspace 4: h = v̂iT vj , v̂i = v̂i − hvj
Moment Normalization:
Matching

Issues with
5: i-th column of V is vi = kv̂v̂ii k stop if v̂i = 0
Krylov 6: end for
Methods
Orthogonalization
Stopping
Criteria
Output of Arnoldi Iteration:
1 Orthonormal Projection matrix V,
2 Hessenberg Matrix H = VT AV
16 / 20
 Stopping Criterion

Krylov
Methods in
MOR
When to stop the iterative scheme?
Introduction
is another question to be answered
Moments & This also decides the size of the ROM
Markov
Parameters TU-M: Singular values based stopping criterion.1
Krylov
Subspace IIT-D: A more efficient criterion based on a index known
Moment as CNRI2 is proposed.3
Matching

Issues with
Krylov
Methods 1
B. Salimbahrami and Lohmann, B., “Stopping Criterion in Order
Orthogonalization
Stopping Reduction of Large Scale Systems Using Krylov Subspace Methods”, Proc.
Criteria
Appl. Math. Mech., 4: 682–683, 2004.
2
Coefficent of Numerical Rank Improvement
3
M. A. Bazaz, M. Nabi and S. Janardhanan. “A stopping criterion for
Krylov-subspace based model order reduction techniques”. Proc. Int.
Conf. Modelling, Identification & Control (ICMIC), pp. 921 - 925, 2012
17 / 20
 Comparison with Balanced Truncation

Krylov
Methods in
MOR
Parameter BT
Krylov

No. of Flops O n3 O q2n
Numerical Reliability for large n No Yes
Introduction
Accuracy of the reduced system More Accurate Less Accurate
Moments &
Markov Range of Applicability ∼ 103 ∼ 104 or higher
Parameters
Stability Preservation Yes No
Krylov
Subspace Iterative Method No Yes
Moment Reliable Stopping Criterion Yes No*
Matching

Issues with
Krylov
Methods
Orthogonalization
Stopping
Criteria

18 / 20
 Selected References

Krylov
Methods in 1 A. C. Antoulas, Approximation of Large Scale Dynamical
MOR
Systems, SIAM, 2005.
2 Behnam Salimbehrami, Structure Preserving Order Reduction of
Introduction Large Scale Second Order Models, PhD Thesis, TU Munich,
Moments &
Markov
2005.
Parameters 3 Rudy Eid, Time Domain Model Reduction By Moment Matching,
Krylov PhD Thesis, TU Munich, 2008.
Subspace
4 B. Salimbehrami, Boris Lohmann, Krylov Subspace Methods in
Moment
Matching Linear Model Order Reduction: Introduction and Invariance
Issues with Properties. Scientific Report, Univ. of Bremen, 2002.
Krylov 5 Zhaojun Bai, Krylov subspace techniques for reduced-order
Methods
Orthogonalization modeling of large-scale dynamical systems, Applied Numerical
Stopping
Criteria Mathematics, 43 (2002), pp 9-44.

19 / 20
 Krylov
Methods in
MOR

Introduction

Moments &
Markov
Parameters

Krylov
Subspace
Thanks!
Moment
Matching

Issues with
,
Krylov
Methods
Orthogonalization
Stopping
Criteria

20 / 20