Lecture Summary: Markov Jump Linear Systems

Vijay Gupta∗ and Richard M. Murray †

Definition: Consider a discrete time discrete state Markov process with state r(k) ∈ {1, 2, · · · , m} at time k. Denote
the transition probability Prob(r(k + 1) = j|r(k) = i) by qi j , and the resultant transition probability matrix by
Q. Also denote Prob(r(k) = j) = π j (k), with π j (0) as given. The evolution of a Markovian jump linear system
(MJLS), denoted by S1 for future reference, can be described by the following equations

x(k + 1) = Ar(k) x(k) + Br(k) u(k) + Fr(k) w(k) (1)

y(k) = Cr(k) x(k) + Gr(k) v(k),

where w(k) is zero mean white Gaussian noise with covariance Rw , v(k) is zero mean white Gaussian noise
with covariance Rv and the notation Xr(k) implies that the matrix X ∈ {X1 , X2 , · · · , Xm } with the matrix Xi being
chosen when r(k) = i. The initial state x(0) is assumed to be a zero mean Gaussian random variable with
variance Π(0). For simplicity, we will consider Fr(k) = Gr(k) ≡ I for all values of r(k). We also assume that
x(0), {w(k)}, {v(k)} and {r(k)} are mutually independent.

LQ Control: The LQR problem aims at designing the control input u(k) to minimize the finite horizon cost function
³
K £ T ¤´
T
JLQR = ∑ {r( j)} j=k+1
E K x (k)Qx(k) + u (k)Ru(k) + xT (K + 1)P(K + 1)x(K + 1),
k=1

where the expectation at time k is taken with respect to the future values of the Markov state realization,
P(K + 1), Q and R are all assumed to be positive definite, and the noises w(k) and v(k) are not present.
The controller at time k has access to control inputs {u( j)}k−1 k
j=0 , state values {x( j)} j=0 and the Markov state
de f
values {r( j)}kj=0 . Moreover, the system is said to be stabilizable if the infinite horizon cost function J∞ =
J
limK→∞ LQR
K is finite. The solution to this problem can readily be obtained through dynamic programming
arguments.

• At time k, if r(k) = i, then the optimal control input is given by

¡ ¢−1 T
u(k) = − R + BTi Pi (k + 1)Bi Bi Pi (k + 1)Ai x(k),

where for j = 1, 2, · · · , m,
m ³ ¡ ¢−1 T ´
Pj (k) = ∑ qt j Q + AtT Pt (k + 1)At − AtT Pt (k + 1)Bt R + BtT Pt (k + 1)Bt Bt Pt (k + 1)At ,
t=1

and Pj (K + 1) = P(K + 1), ∀ j = 1, 2, · · · , m.

• Assume that the Markov states reach a stationary probability distribution. A sufficient condition for
stabilizability of the system is that there exist m positive definite matrices X1 , X2 , · · · , Xm and m2 matrices
K1,1 , K1,2 , · · · , K1,m , K2,1 , · · · , Km,m such that for all j = 1, 2, · · · , m,
m ¡ ¢
X j > ∑ qi j (ATi + Ki, j BTi )Xi (ATi + Ki, j BTi )T + Q + Ki j RKiTj .
i=1
∗ Department of Electrical Engineering, University of Notre Dame, vgupta2@nd.edu
† Control and Dynamical Systems, California Institute of Technology murray@caltech.edu

1
 • A necessary condition for stabilizability is that

qi,i ρ(Ai )2 < 1, ∀i = 1, 2, · · · , m,

where ρ(Ai ) is the spectral radius of the matrix Ai that governs the dynamics of uncontrollable modes of
the process in the i-th mode.

The sufficient condition can be cast in alternate forms as linear matrix inequalities, that can be efficiently
solved. A special case of Markov jump linear systems is when the discrete states are chosen independently
from one time step to the next. For this case, consider system S1 with the additional assumption that the
Markov transition probability matrix is such that for all states i and j, qi j = qi

• At time k, if r(k) = i, then the optimal control input is given by

¡ ¢−1 T
u(k) = − R + BTi P(k + 1)Bi Bi P(k + 1)Ai x(k),

where
m ³ ¡ ¢−1 T ´
P(k) = ∑ qt Q + AtT P(k + 1)At − AtT P(k + 1)Bt T
R + Bt P(k + 1)Bt Bt P(k + 1)At .
t=1

• Assume that the Markov states reach a stationary probability distribution. A sufficient condition for
stabilizability of the system is that there exists a positive definite matrix X, and m matrices K1 , K2 , · · · ,
Km such that
m ¡ ¢
X > ∑ qi (ATi + Ki BTi )X(ATi + Ki BTi )T + Q + Ki RKiT .
i=1

• A necessary condition for stabilizability is that

qi ρ(Ai )2 < 1, ∀i = 1, 2, · · · , m,

where ρ(Ai ) is the spectral radius of the matrix Ai that governs the dynamics of uncontrollable modes of
the process in the i-th mode.

MMSE Estimation The minimum mean squared error estimate problem for the system S1 is posed by assuming
that the control ur(k) is identically zero. The objective is to identify at every time step k, an estimate x̂(k + 1)
of the state x(k + 1) that minimizes the mean squared error covariance
£ ¤
Π(k + 1) = E{w( j)},{v( j)} (x(k + 1) − x̂(k + 1))(x(k + 1) − x̂(k + 1))T ,

where the expectation is taken with respect to the process and measurement noises (but not the Markov
state realization). The estimator at time k has access to observations {y( j)}kj=0 and the Markov state val-
ues {r( j)}kj=0 . Moreover, the error covariance is said to be stable if the expected steady state error covariance
limk→∞ E{r( j)}k−1 [Π(k)] is bounded, where the expectation is taken with respect to the Markov process.
j=0

Since the estimator has access to the Markov state values till time k, the optimal estimate can be calculated
through a time-varying Kalman filter. Thus, if at time k, rk = i, the estimate evolves as

x̂(k + 1) = Ai x̂(k) + K(k) (y(k) −Ci x̂(k)) ,

where
¡ ¢−1
K(k) = Ai Π(k)CiT Ci Π(k)CiT + Rv
¡ ¢−1
Π(k + 1) = Ai Π(k)ATi + Rw − Ai Π(k)CiT Ci Π(k)CiT + Rv Ci Π(k)ATi .

2
 The error covariance Π(k) is available through the above calculations. The term E{r( j)}k−1 [Π(k)] obtained from
j=0
the optimal estimator is upper bounded1 by M(k) = ∑mj=1 M j (k) where
m ³ ¡ ¢−1 ´
M j (k) = ∑ qt j Rw + At Mt (k − 1)AtT − At Mt (k − 1)CtT Rv +Ct Mt (k − 1)CtT Ct Mt (k − 1)AtT ,
t=1

with M j (0) = Π(0) ∀ j. Moreover, if the Markov states reach a stationary probability distribution, then a
sufficient condition for stabilizability of the system is that there exist m positive definite matrices X1 , X2 , · · · ,
Xm and m2 matrices K1,1 , K1,2 , · · · , K1,m , K2,1 , · · · , Km,m such that for all j = 1, 2, · · · , m,
m ¡ ¢
X j > ∑ qi j (Ai + Ki, jCi )Xi (Ai + Ki, jCi )T + Rw + Ki j Rv KiTj .
i=1

A necessary condition for stabilizability is that

qi,i ρ(Ai )2 < 1, ∀i = 1, 2, · · · , m,
where ρ(Ai ) is the spectral radius of the matrix Ai that governs the dynamics of unobservable modes of the
process in the i-th mode.
For the case when the Markov transition probability matrix is such that for all states i and j, qi j = qi (in other
words, the states are chosen independently and identically distributed from one time step to the next). The
term E{r( j)}k−1 [Π(k)] obtained from the optimal estimator is upper bounded by M(k) where
j=0

m ³ ¡ ¢−1 ´
M(k) = ∑ qt Rw + At M(k − 1)AtT − At M(k − 1)CtT Rv +Ct M(k − 1)CtT Ct M(k − 1)AtT ,
t=1

with M(0) = Π(0). Further, a sufficient condition for stabilizability of the system is that there exists a positive
definite matrix X, and m matrices K1 , K2 , · · · , Km such that
m ¡ ¢
X > ∑ qi (Ai + KiCi )X(Ai + KiCi )T + Rw + Ki Rv KiT .
i=1

A necessary condition for stabilizability is that

qi ρ(Ai )2 < 1, ∀i = 1, 2, · · · , m,
where ρ(Ai ) is the spectral radius of the matrix Ai that governs the dynamics of unobservable modes of the
process in the i-th mode.
LQG Control The Linear Quadratic Gaussian (LQG) problem for the system S1 aims at designing the control input
u(k) to minimize the finite horizon cost function
" #
K ¡ ¢
T T T
JLQG = E ∑ x (k)Qx(k) + u (k)Ru(k) + x (K + 1)P(K + 1)x(K + 1) ,
k=1

where the expectation at time k is taken with respect to the future values of the Markov state realization,
and the measurement and process noises. Further, the matrices P(K + 1), Q and R are all assumed to be
positive definite. The controller at time k has access to control inputs {u( j)}k−1 k
j=0 , measurements {y( j)} j=0 and
the Markov state values {r( j)}kj=0 . The system is said to be stabilizable if the infinite horizon cost function
de f J
J∞ = limK→∞ LQG K is finite. The solution to this problem is provided by a separation principle and using the
optimal LQ control in conjunction with the MMSE estimation. At time k, if r(k) = i, then the optimal control
input is given by
¡ ¢−1 T
u(k) = − R + BTi Pi (k + 1)Bi Bi Pi (k + 1)Ai x̂(k),
where Pi (k) is calculated as in the optimal LQ control input and x̂(k) is calculated using a time-varying Kalman
filter. Conditions for stabilizability of the system follow from stability conditions given for LQ control and
MMSE estimation.
1 We say that A is upperbounded by B if B − A is positive semi-definite.