Kalman Filtering with Real Time

Applications 5th Edition Charles K.

Chui
Visit to download the full and correct content document:
https://textbookfull.com/product/kalman-filtering-with-real-time-applications-5th-edition
-charles-k-chui/
More products digital (pdf, epub, mobi) instant
download maybe you interests ...

''Intuitive understanding of Kalman filtering with

MATLAB'' First Edition Adjouadi

https://textbookfull.com/product/intuitive-understanding-of-
kalman-filtering-with-matlab-first-edition-adjouadi/

Digital and Kalman Filtering An Introduction to

Discrete Time Filtering and Optimum Linear Estimation
2nd Edition S. M. Bozic

https://textbookfull.com/product/digital-and-kalman-filtering-an-
introduction-to-discrete-time-filtering-and-optimum-linear-
estimation-2nd-edition-s-m-bozic/

Real Mathematical Analysis Charles Chapman Pugh

https://textbookfull.com/product/real-mathematical-analysis-
charles-chapman-pugh/

Nonlinear Filtering: Methods and Applications Kumar

Pakki Bharani Chandra

https://textbookfull.com/product/nonlinear-filtering-methods-and-
applications-kumar-pakki-bharani-chandra/
Nonlinear Filtering: Concepts and Engineering
Applications 1st Edition Jitendra R. Raol

https://textbookfull.com/product/nonlinear-filtering-concepts-
and-engineering-applications-1st-edition-jitendra-r-raol/

Learning Java An Introduction to Real World Programming

with Java 5th Edition Marc Loy

https://textbookfull.com/product/learning-java-an-introduction-
to-real-world-programming-with-java-5th-edition-marc-loy/

Statistics With R Solving Problems Using Real World

Data 1st Edition Jenine K Harris

https://textbookfull.com/product/statistics-with-r-solving-
problems-using-real-world-data-1st-edition-jenine-k-harris/

Real-Time Rendering 4th Edition Tomas Akenine-Mo■Ller

https://textbookfull.com/product/real-time-rendering-4th-edition-
tomas-akenine-mo%cc%88ller/

Real Time Digital Signal Processing from MATLAB to C

with the TMS320C6x DSPs Third Edition Morrow

https://textbookfull.com/product/real-time-digital-signal-
processing-from-matlab-to-c-with-the-tms320c6x-dsps-third-
edition-morrow/
Charles K. Chui
Guanrong Chen

Kalman Filtering
with Real-Time Applications
Fifth Edition
Kalman Filtering
Charles K. Chui Guanrong Chen
•

Kalman Filtering
with Real-Time Applications
Fifth Edition

123
Charles K. Chui Guanrong Chen
Department of Statistics Department of Electronic Engineering
Stanford University City University of Hong Kong
Stanford, CA Hong Kong
USA China

ISBN 978-3-319-47610-0 ISBN 978-3-319-47612-4 (eBook)

DOI 10.1007/978-3-319-47612-4

Library of Congress Control Number: 2017933271

Originally published as volume 17 in the series: Springer Series in Information Sciences

1st–4th edition: © 2009, 1999, 1991, 1987 Springer-Verlag Berlin Heidelberg

5th edition: © Springer International Publishing AG 2017
This work is subject to copyright. All rights are reserved by the Publisher, whether the whole or part
of the material is concerned, specifically the rights of translation, reprinting, reuse of illustrations,
recitation, broadcasting, reproduction on microfilms or in any other physical way, and transmission
or information storage and retrieval, electronic adaptation, computer software, or by similar or dissimilar
methodology now known or hereafter developed.
The use of general descriptive names, registered names, trademarks, service marks, etc. in this
publication does not imply, even in the absence of a specific statement, that such names are exempt from
the relevant protective laws and regulations and therefore free for general use.
The publisher, the authors and the editors are safe to assume that the advice and information in this
book are believed to be true and accurate at the date of publication. Neither the publisher nor the
authors or the editors give a warranty, express or implied, with respect to the material contained herein or
for any errors or omissions that may have been made. The publisher remains neutral with regard to
jurisdictional claims in published maps and institutional affiliations.

Printed on acid-free paper

This Springer imprint is published by Springer Nature

The registered company is Springer International Publishing AG
The registered company address is: Gewerbestrasse 11, 6330 Cham, Switzerland
Preface to the Fifth Edition

This new edition is dedicated to the memory of Rudolf E. Kalman (May 19, 1930–
July 2, 2016), with utmost respect and admiration. Kalman was the first to consider
the discrete-time Riccati equation for the linear stochastic dynamic model and to
derive the optimal linear feedback gain for the corresponding estimator. The
importance of this clever innovation is that it enables real-time computations of the
optimal estimator, and hence giving birth to the celebrated “Kalman Filter”, which
is the core of our discussions in this book.
Kalman also initiated the modern systems and control theories by introducing
the state-space framework and defining the basic concepts of system controllability
and observability, among his many other important contributions to science and
technology. To address the latter subject of control systems, we have written
another textbook, entitled “Linear Systems and Optimal Control”, published in
1989, also by Springer-Verlag, which may be considered as a sister treatise of the
present book on Kalman filtering.
In the preparation of this fifth edition, with the assistance from Dr. Wen Yang
and Dr. Ling Shi, we have added a new chapter, namely Chap. 12, on the study of
distributed estimation of sensor networks. Since this is a focal topic in the current
active research on real-time engineering applications of the Kalman filter, we
believe it aligns well with the other contents of the book.
We sincerely hope that the readers will find this new edition more compre-
hensive and informative, and we welcome your generous feedback.

Menlo Park, CA, USA Charles K. Chui

Hong Kong, China Guanrong Chen
August 2016

v
Preface to the Third Edition

Two modern topics in Kalman filtering are new additions to this Third Edition of
Kalman Filtering with Real-Time Applications. Interval Kalman Filtering
(Chap. 10) is added to expand the capability of Kalman filtering to uncertain
systems, and Wavelet Kalman Filtering (Chap. 11) is introduced to incorporate
efficient techniques from wavelets and splines with Kalman filtering to give more
effective computational schemes for treating problems in such areas as signal
estimation and signal decomposition. It is hoped that with the addition of these two
new chapters, the current edition gives a more complete and up-to-date treatment of
Kalman filtering for real-time applications.

College Station and Houston, TX, USA Charles K. Chui

August 1998 Guanrong Chen

vii
Preface to the Second Edition

In addition to making a number of minor corrections and updating the list of

references, we have expanded the section on “real-time system identification” in
Chap. 10 of the first edition into two sections and combined it with Chap. 8. In its
place, a very brief introduction to wavelet analysis is included in Chap. 10.
Although the pyramid algorithms for wavelet decompositions and reconstructions
are quite different from the Kalman filtering algorithms, they can also be applied to
time-domain filtering, and it is hoped that splines and wavelets can be incorporated
with Kalman filtering in the near future.

College Station and Houston, TX, USA Charles K. Chui

September 1990 Guanrong Chen

ix
Preface to the First Edition

Kalman filtering is an optimal state estimation process applied to a dynamic system

that involves random perturbations. More precisely, the Kalman filter gives a linear,
unbiased, and minimum error variance recursive algorithm to optimally estimate the
unknown state of a dynamic system from noisy data taken at discrete real-time. It
has been widely used in many areas of industrial and government applications such
as video and laser tracking systems, satellite navigation, ballistic missile trajectory
estimation, radar, and fire control. With the recent development of high-speed
computers, the Kalman filter has become more useful even for very complicated
real-time applications.
In spite of its importance, the mathematical theory of Kalman filtering and its
implications are not well understood even among many applied mathematicians and
engineers. In fact, most practitioners are just told what the filtering algorithms are
without knowing why they work so well. One of the main objectives of this text is
to disclose this mystery by presenting a fairly thorough discussion of its mathe-
matical theory and applications to various elementary real-time problems.
A very elementary derivation of the filtering equations is first presented. By
assuming that certain matrices are nonsingular, the advantage of this approach is
that the optimality of the Kalman filter can be easily understood. Of course these
assumptions can be dropped by using the more well known method of orthogonal
projection usually known as the innovations approach. This is done next, again
rigorously. This approach is extended first to take care of correlated system and
measurement noises, and then colored noise processes. Kalman filtering for non-
linear systems with an application to adaptive system identification is also discussed
in this text. In addition, the limiting or steady-state Kalman filtering theory and
efficient computational schemes such as the sequential and square-root algorithms
are included for real-time application purposes. One such application is the design
of a digital tracking filter such as the fi
fl
 and fi
fl

 trackers. Using
the limit of Kalman gains to define the fi; fl;  parameters for white noise and the
fi; fl; ;  values for colored noise processes, it is now possible to characterize this
tracking filter as a limiting or steady-state Kalman filter. The state estimation
obtained by these much more efficient prediction-correction equations is proved to
be near-optimal, in the sense that its error from the optimal estimate decays

xi
xii Preface to the First Edition

exponentially with time. Our study of this topic includes a decoupling method that
yields the filtering equations for each component of the state vector.
The style of writing in this book is intended to be informal, the mathematical
argument throughout elementary and rigorous, and in addition, easily readable by
anyone, student or professional, with a minimal knowledge of linear algebra and
system theory. In this regard, a preliminary chapter on matrix theory, determinants,
probability, and least-squares is included in an attempt to ensure that this text be
self-contained. Each chapter contains a variety of exercises for the purpose of
illustrating certain related view-points, improving the understanding of the material,
or filling in the gaps of some proofs in the text. Answers and hints are given at the
end of the text, and a collection of notes and references is included for the reader
who might be interested in further study.
This book is designed to serve three purposes. It is written not only for self-study
but also for use in a one-quarter or one-semester introductory course on Kalman
filtering theory for upper-division undergraduate or first-year graduate applied
mathematics or engineering students. In addition, it is hoped that it will become a
valuable reference to any industrial or government engineer.
The first author would like to thank the U.S. Army Research Office for con-
tinuous support and is especially indebted to Robert Green of the White Sands
Missile Range for his encouragement and many stimulating discussions. To his
wife, Margaret, he would like to express his appreciation for her understanding and
constant support. The second author is very grateful to Prof. Mingjun Chen of
Zhongshan University for introducing him to this important research area, and to
his wife Qiyun Xian for her patience and encouragement.
Among the colleagues who have made valuable suggestions, the authors would
especially like to thank Profs. Andrew Chan (Texas A&M), Thomas Huang
(Illinois), and Thomas Kailath (Stanford). Finally, the friendly cooperation and kind
assistance from Dr. Helmut Lotsch, Dr. Angela Lahee, and their editorial staff at
Springer-Verlag are greatly appreciated.

College Station, TX, USA Charles K. Chui

January 1987 Guanrong Chen
Contents

1 Preliminaries . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
1.1 Matrix and Determinant Preliminaries . . . . . . . . . . . . . . . . . . 1
1.2 Probability Preliminaries . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
1.3 Least-Squares Preliminaries . . . . . . . . . . . . . . . . . . . . . . . . . 15
Exercises. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
2 Kalman Filter: An Elementary Approach . . . . . . . . . . . . . . . . . . 19
2.1 The Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
2.2 Optimality Criterion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
2.3 Prediction-Correction Formulation . . . . . . . . . . . . . . . . . . . . . 22
2.4 Kalman Filtering Process . . . . . . . . . . . . . . . . . . . . . . . . . . . 26
Exercises. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
3 Orthogonal Projection and Kalman Filter . . . ........ . . . . . . . 33
3.1 Orthogonality Characterization of Optimal Estimates . . . . . . . . 33
3.2 Innovations Sequences. . . . . . . . . . . . . . ........ . . . . . . . 35
3.3 Minimum Variance Estimates . . . . . . . . . ........ . . . . . . . 37
3.4 Kalman Filtering Equations . . . . . . . . . . ........ . . . . . . . 38
3.5 Real-Time Tracking. . . . . . . . . . . . . . . . ........ . . . . . . . 43
Exercises. . . . . . . . . . . . . . . . . . . . . . . . . . . . ........ . . . . . . . 45
4 Correlated System and Measurement Noise Processes. . . . . . . . . . 51
4.1 The Affine Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51
4.2 Optimal Estimate Operators . . . . . . . . . . . . . . . . . . . . . . . . . 53
4.3 Effect on Optimal Estimation with Additional Data . . . . . . . . . 54
4.4 Derivation of Kalman Filtering Equations . . . . . . . . . . . . . . . 57
4.5 Real-Time Applications . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63
4.6 Linear Deterministic/Stochastic Systems. . . . . . . . . . . . . . . . . 65
Exercises. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67
5 Colored Noise Setting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 69
5.1 Outline of Procedure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70
5.2 Error Estimates . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71
5.3 Kalman Filtering Process . . . . . . . . . . . . . . . . . . . . . . . . . . . 73

xiii
xiv Contents

5.4 White System Noise . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 76

5.5 Real-Time Applications . . . . . . . . . . . . . . . . . . . . . . . . . . . . 76
Exercises. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 78
6 Limiting Kalman Filter . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81
6.1 Outline of Procedure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82
6.2 Preliminary Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 83
6.3 Geometric Convergence . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92
6.4 Real-Time Applications . . . . . . . . . . . . . . . . . . . . . . . . . . . . 98
Exercises. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 99
7 Sequential and Square-Root Algorithms. . . . . . . . . . . . . . . . . . . . 101
7.1 Sequential Algorithm. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 101
7.2 Square-Root Algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . 107
7.3 An Algorithm for Real-Time Applications . . . . . . . . . . . . . . . 110
Exercises. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 111
8 Extended Kalman Filter and System Identification . . . . . . . . . . . . 115
8.1 Extended Kalman Filter . . . . . . . . . . . . . . . . . . . . . . . . . . . . 115
8.2 Satellite Orbit Estimation . . . . . . . . . . . . . . . . . . . . . . . . . . . 118
8.3 Adaptive System Identification . . . . . . . . . . . . . . . . . . . . . . . 120
8.4 An Example of Constant Parameter Identification . . . . . . . . . . 123
8.5 Modified Extended Kalman Filter . . . . . . . . . . . . . . . . . . . . . 125
8.6 Time-Varying Parameter Identification . . . . . . . . . . . . . . . . . . 131
Exercises. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 135
9 Decoupling of Filtering Equations . . . . . . . . . . . . . . . . . . . . . . . . 139
9.1 Decoupling Formulas. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 139
9.2 Real-Time Tracking. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 142
9.3 The fi
fl
 Tracker . . . . . . . . . . . . . . . . . . . . . . . . . . . . 144
9.4 An Example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 147
Exercises. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 148
10 Kalman Filtering for Interval Systems . . . . . . . . . . . . . . . . . . . . . 151
10.1 Interval Mathematics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 152
10.1.1 Intervals and Their Properties . . . . . . . . . . . . . . . . . . 152
10.1.2 Interval Arithmetic . . . . . . . . . . . . . . . . . . . . . . . . . 153
10.1.3 Rational Interval Functions. . . . . . . . . . . . . . . . . . . . 157
10.1.4 Interval Expectation and Variance . . . . . . . . . . . . . . . 159
10.2 Interval Kalman Filtering . . . . . . . . . . . . . . . . . . . . . . . . . . . 161
10.2.1 The Interval Kalman Filtering Scheme . . . . . . . . . . . . 162
10.2.2 Suboptimal Interval Kalman Filter. . . . . . . . . . . . . . . 163
10.2.3 An Example of Target Tacking . . . . . . . . . . . . . . . . . 165
10.3 Weighted-Average Interval Kalman Filtering . . . . . . . . . . . . . 167
Exercises. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 168
Contents xv

11 Wavelet Kalman Filtering . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 171

11.1 Wavelet Preliminaries . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 171
11.1.1 Wavelet Fundamentals . . . . . . . . . . . . . . . . . . . . . . . 172
11.1.2 Discrete Wavelet Transform and Filter Banks . . . . . . . 174
11.2 Singal Estimation and Decomposition . . . . . . . . . . . . . . . . . . 177
11.2.1 Estimation and Decomposition of Random Signals . . . 177
11.2.2 An Example of Random Walk . . . . . . . . . . . . . . . . . 181
Exercises. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 182
12 Distributed Estimation on Sensor Networks . . . . . . . . . . . . . . . . . 185
12.1 Background . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 185
12.2 Problem Description . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 186
12.3 Algorithm Convergence . . . . . . . . . . . . . . . . . . . . . . . . . . . . 190
12.4 A Simulation Example. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 194
Exercises. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 195
13 Notes ......................................... . . . . 197
13.1 The Kalman Smoother . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 197
13.2 The fi
fl

 Tracker . . . . . . . . . . . . . . . . . . . . . . . . . 199
13.3 Adaptive Kalman Filtering . . . . . . . . . . . . . . . . . . . . . . . . . . 202
13.4 Adaptive Kalman Filtering Approach to Wiener Filtering . . . . . 203
13.5 The Kalman–Bucy Filter . . . . . . . . . . . . . . . . . . . . . . . . . . . 204
13.6 Stochastic Optimal Control. . . . . . . . . . . . . . . . . . . . . . . . . . 205
13.7 Square-Root Filtering and Systolic Array Implementation. . . . . 206
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 209
Answers and Hints to Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 213
Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 245
Notations

A; Ak System matrices
Ac “Square-root” of A in Cholesky factorization
Au “Square-root” of A in upper triangular decomposition
B; Bk Control input matrices
C; Ck Measurement matrices
CovðX; YÞ Covariance of random variables X and Y
EðXÞ Expectation of random variable X
EðXjY ¼ yÞ Conditional expectation
ej ; ^ej
f ðxÞ Probability density function
f ðx1 ; x2 Þ Joint probability density function
f ðx1 jx2 Þ Conditional probability density function
f k ðxk Þ Vector-valued nonlinear functions
G Limiting Kalman gain matrix
Gk Kalman gain matrix
Hk ðxk Þ Matrix-valued nonlinear function
H
In n  n Identity matrix
J Jordan canonical form of a matrix
Kk
Lðx; vÞ
MA¡ Controllability matrix
NCA Observability matrix
Onm n  m Zero matrix
P Limiting (error) covariance matrix
Pk;k Estimate (error) covariance matrix
P½i; j ði; jÞth entry of matrix P
PðXÞ Probability of random variable X
Qk Variance matrix of random vector »k
Rk Variance matrix of random vector ·k
Rn Space of column vectors x ¼ ½x1    xn T
Sk Covariance matrix of »k and ·k
tr Trace

xvii
xviii Notations

uk Deterministic control input (at the kth time instant)

Var(X) Variance of random variable X
VarðXjY ¼ yÞ Conditional variance
vk Observation (or measurement) data (at the kth time instant)
v2#
Wk Weight matrix
wj
ðWˆ f Þðb; aÞ Integral wavelet transform
xk State vector (at the kth time instant)
xk ; ^
^ xkjk Optimal filtering estimate of xk
^
xkjk
1 Optimal prediction of xk
^
xk Suboptimal estimate of xk
~
xk Near-optimal estimate of xk
x
x#, x#k
kwk “Norm” of w
hx; wi “Inner product” of x and w
Yðw0 ;    ; wr Þ “Linear span” of vectors w0 ;    ; wr
fzj g Innovations sequence of data
fi; fl; ;  Tracker parameters
fflk g; f k g White noise sequences
¡; ¡k System noise matrices
–ij Kronecker delta
†k;‘ ; †k;‘ Random (noise) vectors
·k Measurement noise (at the kth time instant)
»k System noise (at the kth time instant)
'k‘ Transition matrix
df =dA Jacobian matrix
oh=ox Jacobian matrix
Preliminaries
1

The importance of Kalman filtering in engineering applications is well known, and

its mathematical theory has been rigorously established. The main objective of this
treatise is to present a thorough discussion of the mathematical theory, computational
algorithms, and application to real-time tracking problems of the Kalman filter.
In explaining how the Kalman filtering algorithm is obtained and how well it
performs, it is necessary to use some formulas and inequalities in matrix algebra. In
addition, since only the statistical properties of both the system and measurement
noise processes in real-time applications are being considered, some knowledge of
certain basic concepts in probability theory will be helpful. This chapter is devoted
to the study of these topics.

1.1 Matrix and Determinant Preliminaries

Let Rn denote the space of all column vectors x = [x1 · · · xn ] , where x1 , . . . , xn

are real numbers. An n × n real matrix A is said to be positive definite if x Ax is a
positive number for all nonzero vectors x in Rn . It is said to be non-negative definite
if x Ax is non-negative for any x in Rn . If A and B are any two n × n matrices of
real numbers, we will use the notation

A>B

when the matrix A − B is positive definite, and

A≥B

when A − B is non-negative definite.

© Springer International Publishing AG 2017 1

C.K. Chui and G. Chen, Kalman Filtering, DOI 10.1007/978-3-319-47612-4_1
2 1 Preliminaries

We first recall the so-called Schwarz inequality:

|x y| ≤ |x| |y|, x, y ∈ Rn ,

where, as usual, the notation

|x| = (x x)1/2

is used. In addition, we recall that the above inequality becomes equality if and only
if x and y are parallel, and this, in turn, means that

x = λy or y = λx

for some scalar λ. Note, in particular, that if y = 0, then the Schwarz inequality may
be written as

x x ≥ (y x) (y y)−1 (y x).

This formulation allows us to generalize the Schwarz inequality to the matrix setting.

Lemma 1.1 (Matrix Schwarz inequality) Let P and Q be m × n and m ×  matri-

ces, respectively, such that P P is nonsingular. Then

Q Q ≥ (P Q) (P P)−1 (P Q). (1.1)

Furthermore, equality in (1.1) holds if and only if Q = PS for some n ×  matrix S.

The proof of the (vector) Schwarz inequality is simple. It amounts to observing

that the minimum of the quadratic polynomial

(x − λy) (x − λy), y = 0,

of λ is attained at

λ = (y y)−1 (y x)

and using this λ value in the above inequality. Hence, in the matrix setting, we
consider

(Q − PS) (Q − PS) ≥ 0

and choose

S = (P P)−1 (P Q),

1.1 Matrix and Determinant Preliminaries 3

so that

Q Q ≥ S  (P Q) + (P Q) S − S  (P P)S = (P Q) (P P)−1 (P Q)

as stated in (1.1). Furthermore, this inequality becomes equality if and only if

(Q − PS) (Q − PS) = 0,

or equivalently, Q = PS for some n ×  matrix S. This completes the proof of the

lemma.
We now turn to the following so-called matrix inversion lemma.

Lemma 1.2 (Matrix inversion lemma) Let

 
A11 A12
A= ,
A21 A22

where A11 and A22 are n × n and m × m nonsingular submatrices, respectively,

such that

(A11 − A12 A−1 −1

22 A21 ) and (A22 − A21 A11 A12 )

are also nonsingular. Then A is nonsingular with

A−1
 
A−1 −1 −1 −1 −1
11 + A11 A12 (A22 − A21 A11 A12 ) A21 A11 −A−1 −1
11 A12 (A22 − A21 A11 A12 )
−1
= −1 −1 −1 −1 −1
−(A22 − A21 A11 A12 ) A21 A11 (A22 − A21 A11 A12 )
 
(A11 − A12 A−1
22 A21 )
−1 −(A11 − A12 A−1 −1 −1
22 A21 ) A12 A22
= −1 −1 . (1.2)
−A22 A21 (A11 − A12 A22 A21 )−1 A22 + A22 A21 (A11 − A12 A22 A21 ) A12 A−1
−1 −1 −1 −1
22

In particular,

(A11 − A12 A−1

22 A21 )
−1

=A−1 −1 −1 −1 −1
11 + A11 A12 (A22 − A21 A11 A12 ) A21 A11 (1.3)

and

A−1 −1
11 A12 (A22 − A21 A11 A12 )
−1

=(A11 − A12 A−1 −1 −1

22 A21 ) A12 A22 . (1.4)

Furthermore,

det A = (det A11 ) det(A22 − A21 A−1

11 A12 )
= (det A22 ) det(A11 − A12 A−1
22 A21 ). (1.5)
4 1 Preliminaries

To prove this lemma, we write

  
In 0 A11 A12
A=
A21 A−1
11 Im 0 A22 − A21 A−1
11 A12

and
  
In A12 A−1 A11 − A12 A−1
22 A21 0
A= 22 .
0 Im A21 A22

Taking determinants, we obtain (1.5). In particular, we have

det A = 0,

or A is nonsingular. Now observe that

 −1
A11 A12
0 A22 − A21 A−1
11 A12
 
A−1 −A−1 −1
11 A12 (A22 − A21 A11 A12 )
−1
= 11 −1
0 (A22 − A21 A11 A12 )−1

and
 −1  
In 0 In 0
= .
A21 A−1 I
11 m −A21 A−1
11 Im

Hence, we have
 −1  −1
−1 A11 A12 In 0
A =
0 A22 − A21 A−1
11 A12
−1
A21 A11 Im

which gives the first part of (1.2). A similar proof also gives the second part of (1.2).
Finally, (1.3) and (1.4) follow by equating the appropriate blocks in (1.2).
An immediate application of Lemma 1.2 yields the following result.

Lemma 1.3 If P ≥ Q > 0, then Q−1 ≥ P−1 > 0.

Let P() = P + I where  > 0. Then P() − Q > 0. By Lemma 1.2, we have

P−1 () = [Q + (P() − Q)]−1

= Q−1 − Q−1 [(P() − Q)−1 + Q−1 ]−1 Q−1 ,

so that

Q−1 − P−1 () = Q−1 [(P() − Q)−1 + Q−1 ]−1 Q−1 > 0.
1.1 Matrix and Determinant Preliminaries 5

Letting  → 0 gives Q−1 − P−1 ≥ 0, so that

Q−1 ≥ P−1 > 0.

Now let us turn to discussing the trace of an n × n matrix A. The trace of A,

denoted by tr A, is defined as the sum of its diagonal elements, namely:


n
tr A = aii ,
i=1

where A = [aij ]. We first state some elementary properties.

Lemma 1.4 If A and B are n × n matrices, then

tr A = tr A, (1.6)
tr (A + B) = tr A + tr B, (1.7)

and

tr (λA) = λ tr A. (1.8)

If A is an n × m matrix and B is an m × n matrix, then

tr AB = tr B A = tr BA = tr A B (1.9)

and

n 
m
tr A A = aij2 . (1.10)
i=1 j=1

The proof of the above identities is immediate from the definition and we leave it
to the reader (cf. Exercise 1.1). The following result is important.

Lemma 1.5 Let A be an n × n matrix with eigenvalues λ1 , · · · , λn , multiplicities

being listed. Then

n
tr A = λi . (1.11)
i=1

To prove the lemma, we simply write A = UJU −1 where J is the Jordan canonical
form of A and U is some nonsingular matrix. Then an application of (1.9) yields


n
tr A = tr (AU)U −1 = tr U −1 (AU) = tr J = λi .
i=1

It follows from this lemma that if A > 0 then tr A > 0, and if A ≥ 0 then tr A ≥ 0.
6 1 Preliminaries

Next, we state some useful inequalities on the trace.

Lemma 1.6 Let A be an n × n matrix. Then

tr A ≤ (n tr AA )1/2 . (1.12)

We leave the proof of this inequality to the reader (cf. Exercise 1.2).

Lemma 1.7 If A and B are n × m and m ×  matrices respectively, then

tr (AB)(AB) ≤ (tr AA )(tr BB ).

Consequently, for any matrices A1 , · · · , Ap with appropriate dimensions,

tr (A1 · · · Ap )(A1 · · · Ap ) ≤ (tr A1 A 

1 ) · · · (tr Ap Ap ). (1.13)

If A = [aij ] and B = [bij ], then

 m  m 
 

tr (AB)(AB) = tr aik bkj ajk bki
k=1 k=1
⎡ ⎤
 m 2
p=1 k=1 a1k bkp ∗
⎢ .. ⎥
= tr ⎢
⎣ . ⎥
⎦
 m 2
∗ p=1 k=1 ank bkp

 m 2  

n   
n  m 
m
= aik bkp ≤ 2
aik 2
bkp
i=1 p=1 k=1 i=1 p=1 k=1 k=1
 ⎛  m ⎞

n 
m 
2 ⎝ 2 ⎠
= aik bkp = (tr AA )(tr BB ),
i=1 k=1 p=1 k=1

where the Schwarz inequality has been used. This completes the proof of the lemma.
It should be remarked that A ≥ B > 0 does not necessarily imply tr AA ≥
tr BB . An example is
 12   
0 2 −1
A= 5 and B = .
0 1 1 1

Here, it is clear that A − B ≥ 0 and B > 0, but

169
tr AA = < 7 = tr BB
25
(cf. Exercise 1.3).
For a symmetric matrix, however, we can draw the expected conclusion as follows.
1.1 Matrix and Determinant Preliminaries 7

Lemma 1.8 Let A and B be non-negative definite symmetric matrices with A ≥ B.

Then tr AA ≥ tr BB , or tr A2 ≥ tr B2 .

We leave the proof of this lemma as an exercise (cf. Exercise 1.4).

Lemma 1.9 Let B be an n × n non-negative definite symmetric matrix. Then

tr B2 ≤ (tr B)2 . (1.14)

Consequently, if A is another n × n non-negative definite symmetric matrix such

that B ≤ A, then

tr B2 ≤ (tr A)2 . (1.15)

To prove (1.14), let λ1 , · · · , λn be the eigenvalues of B. Then λ21 , · · · , λ2n are the
eigenvalues of B2 . Now, since λ1 , · · · , λn are non-negative, Lemma 1.5 gives
 2

n 
n
tr B =
2
λ2i ≤ λi = (tr B)2 .
i=1 i=1

(1.15) follows from the fact that B ≤ A implies tr B ≤ tr A.

We also have the following result which will be useful later.

Lemma 1.10 Let F be an n × n matrix with eigenvalues λ1 , · · · , λn such that

λ := max(|λ1 |, · · · , |λn |) < 1.

Then there exist a real number r satisfying 0 < r < 1 and a constant C such that

|tr F k (F k ) | ≤ Cr k

for all k = 1, 2, · · · .

Let J be the Jordan canonical form for F. Then F = UJU −1 for some nonsingular
matrix U. Hence, using (1.13), we have

|tr F k (F k ) | = |tr UJ k U −1 (U −1 ) (J k ) U  |
≤ |tr UU  tr J k (J k ) tr U −1 (U −1 ) |
≤ p(k)λ2k ,
8 1 Preliminaries

where p(k) is a polynomial in k. Now, any choice of r satisfying λ2 < r < 1 yields
the desired result, by choosing a positive constant C that satisfies
 k
λ2
p(k) ≤C
r

for all k.

1.2 Probability Preliminaries

Consider an experiment in which a fair coin is tossed such that on each toss either
the head (denoted by H) or the tail (denoted by T ) occurs. The actual result that
occurs when the experiment is performed is called an outcome of the experiment
and the set of all possible outcomes is called the sample space (denoted by S) of the
experiment. For instance, if a fair coin is tossed twice, then each result of two tosses is
an outcome, the possibilities are HH, TT , HT , TH, and the set {HH, TT , HT , TH}
is the sample space S. Furthermore, any subset of the sample space is called an event
and an event consisting of a single outcome is called a simple event.
Since there is no way to predict the outcomes, we have to assign a real number
P, between 0 and 1, to each event to indicate the probability that a certain outcome
occurs. This is specified by a real-valued function, called a random variable, defined
on the sample space. In the above example, if the random variable X = X(s), s ∈ S,
denotes the number of H’s in the outcome s, then the number P = P(X(s)) gives the
probability in percentage in the number of H’s of the outcome s. More generally,
let S be a sample space and X : S → R1 be a random variable. For each measurable
set A ⊂ R1 (and in the above example, A = {0}, {1}, or {2} indicating no H, one
H, or two H’s, respectively) define P : {events} → [0, 1], where each event is a set
{s ∈ S : X(s) ∈ A ⊂ R1 } := {X ∈ A}, subject to the following conditions:

(1) P(X ∈ A) ≥ 0 for any measurable set A ⊂ R1 ,

(2) P(X ∈ R1 ) = 1, and
(3) for any countable sequence of pairwise disjoint measurable sets Ai in R1 ,
  ∞
∞
P X ∈ ∪ Ai = P(X ∈ Ai ).
i=1
i=1

P is called the probability distribution (or probability distribution function) of

the random variable X.

If there exists an integrable function f such that


P(X ∈ A) = f (x)dx (1.16)
A
1.2 Probability Preliminaries 9

for all measurable sets A, we say that P is a continuous probability distribution

and f is called the probability density function of the random variable X. Note that
actually we could have defined f (x)dx = dλ where λ is a measure (for example, step
functions) so that the discrete case such as the example of “tossing coins” can be
included.
If the probability density function f is given by

1 − 1
(x−μ)2
f (x) = √ e 2σ 2 , σ > 0 and μ ∈ R, (1.17)
2πσ

called the Gaussian (or normal) probability density function, then P is called a normal
distribution of the random variable X, and we use the notation: X ∼ N(μ, σ 2 ). It can
be easily verified that the normaldistribution P is a probability distribution. Indeed,
(1) since f (x) > 0, P(X ∈ A) = A f (x)dx ≥ 0 for any measurable set A ⊂ R, (2) by
√
substituting y = (x − μ)/( 2σ),
 ∞  ∞
1
e−y dy = 1,
2
P(X ∈ R ) =
1
f (x)dx = √
−∞ π −∞

(cf. Exercise 1.5), and (3) since

 
f (x)dx = f (x)dx
∪ Ai i Ai
i

for any countable sequence of pairwise disjoint measurable sets Ai ⊂ R1 , we have


P(X ∈ ∪ Ai ) = P(X ∈ Ai ).
i
i

Let X be a random variable. The expectation of X indicates the mean of the values
of X, and is defined by
 ∞
E(X) = xf (x)dx. (1.18)
−∞

Note that E(X) is a real number for any random variable X with probability density
√
function f . For the normal distribution, using the substitution y = (x − μ)/( 2σ)
again, we have
 ∞
E(X) = xf (x)dx
−∞
 ∞
1 − 1 (x−μ)2
=√ xe 2σ2 dx
2πσ −∞
 ∞ √
1
( 2σy + μ)e−y dy
2
=√
π −∞
10 1 Preliminaries
 ∞
1
e−y dy
2
= μ√
π −∞
= μ. (1.19)

Note also that E(X) is the first moment of the probability density function f . The
second moment gives the variance of X defined by
 ∞
V ar (X) = E(X − E(X)) = 2
(x − E(X))2 f (x)dx. (1.20)
−∞

This number indicates the dispersion of the values of X from

√ its mean E(X). For the
normal distribution, using the substitution y = (x − μ)/( 2σ) again, we have
 ∞
V ar(X) = (x − μ)2 f (x)dx
−∞
 ∞
1 − 1 (x−μ)2
=√ (x − μ)2 e 2σ2 dx
2πσ −∞

2σ 2 ∞ 2 −y2
= √ y e dy
π −∞
= σ2 , (1.21)
∞ √
where we have used the equality −∞ y2 e−y dy = π/2 (cf. Exercise 1.6).
2

We now turn to random vectors whose components are random variables. We

denote a random n-vector X = [X1 · · · Xn ] where each Xi (s) ∈ R1 , s ∈ S.
Let P be a continuous probability distribution function of X. That is,

P(X1 ∈ A1 , · · · , Xn ∈ An )
 
= ··· f (x1 , · · · , xn )dx1 · · · dxn , (1.22)
A1 An

where A1 , · · · , An are measurable sets in R1 and f an integrable function. f is called

a joint probability density function of X and P is called a joint probability distribution
(function) of X. For each i, i = 1, · · · , n, define

fi (x) =
 ∞  ∞
··· f (x1 , · · · , xi−1 , x, xi+1 , · · · , xn )dx1 · · · dxi−1 dxi+1 · · · dxn . (1.23)
−∞ −∞
∞
Then it is clear that −∞ fi (x)dx = 1. fi is called the ith marginal probability density
function of X corresponding to the joint probability density function f (x1 , · · · , xn ).
1.2 Probability Preliminaries 11

Similarly, we define fij and fijk by deleting the integrals with respect to xi , xj and xi ,
xj , xk , respectively, etc., as in the definition of fi . If
 
1 1  −1
f (x) = exp − (x − μ) R (x − μ) , (1.24)
(2π)n/2 (det R)1/2 2

where μ is a constant n-vector and R is a symmetric positive definite matrix, we

say that f (x) is a Gaussian (or normal) probability density function of X. It can be
verified that
 ∞  ∞  ∞
f (x)dx := ··· f (x)dx1 · · · dxn = 1, (1.25)
−∞ −∞ −∞

 ∞
E(X) = xf (x)dx
−∞
⎡ ⎤
 ∞  ∞
x1
⎢ .. ⎥
:= ··· ⎣ . ⎦ f (x)dx1 · · · dxn
−∞ −∞
xn
=μ, (1.26)

and

V ar(X) = E(X − μ)(X − μ) = R. (1.27)

Indeed, since R is symmetric and positive definite, there is a unitary matrix U

such that√R = U  JU √ where J = diag[λ1 , · · · , λn ] and λ1 , · · · , λn > 0. Let y =
√1 diag[ λ1 , · · · , λn ]U(x − μ). Then
2
 ∞
f (x)dx
−∞
√ √  ∞  ∞
2n/2 λ 1 ···
λn −y12
e−yn dyn
2
= e dy1 · · ·
(2π)n/2 (λ1 · · · λn )1/2 −∞ −∞
= 1.

Equations (1.26) and (1.27) can be verified by using the same substitution as that
used for the scalar case (cf. (1.21) and Exercise 1.7).
Next, we introduce the concept of conditional probability. Consider an experiment
in which balls are drawn one at a time from an urn containing M1 white balls and
M2 black balls. What is the probability that the second ball drawn from the urn is
also black (event A2 ) under the condition that the first one is black (event A1 )? Here,
we sample without replacement; that is, the first ball is not returned to the urn after
being drawn.
12 1 Preliminaries

To solve this simple problem, we reason as follows: since the first ball drawn from
the urn is black, there remain M1 white balls and M2 − 1 black balls in the urn before
the second drawing. Hence, the probability that a black ball is drawn is now

M2 − 1
.
M1 + M2 − 1

Note that

M2 − 1 M2 M2 − 1 M2
= · ,
M1 + M2 − 1 M1 + M2 M1 + M2 − 1 M1 + M2

where M2 /(M1 + M2 ) is the probability that a black ball is picked at the first drawing,
2 −1
and (M1M+M
2
2)
· M1M+M 2 −1
is the probability that black balls are picked at both the first
and second drawings. This example motivates the following definition of conditional
probability: The conditional probability of X1 ∈ A1 given X2 ∈ A2 is defined by

P(X1 ∈ A1 , X2 ∈ A2 )
P(X1 ∈ A1 |X2 ∈ A2 ) = . (1.28)
P(X2 ∈ A2 )

Suppose that P is a continuous probability distribution function with joint proba-

bility density function f . Then (1.28) becomes
 
A1 A2 f (x1 , x2 )dx1 dx2
P(X1 ∈ A1 |X2 ∈ A2 ) =  ,
A2 f2 (x2 )dx2

where f2 defined by
 ∞
f2 (x2 ) = f (x1 , x2 )dx1
−∞

is the second marginal probability density function of f . Let f (x1 |x2 ) denote the prob-
ability density function corresponding to P(X1 ∈ A1 |X2 ∈ A2 ). f (x1 |x2 ) is called the
conditional probability density function corresponding to the conditional probability
distribution function P(X1 ∈ A1 |X2 ∈ A2 ). It is known that

f (x1 , x2 )
f (x1 |x2 ) = (1.29)
f2 (x2 )

which is called the Bayes formula (see, for example, Probability by Shiryayev
(1984)). By symmetry, the Bayes formula can be written as

f (x1 , x2 ) = f (x1 |x2 )f2 (x2 ) = f (x2 |x1 )f1 (x1 ). (1.30)

We remark that this formula also holds for random vectors X1 and X2 .
1.2 Probability Preliminaries 13

Let X and Y be random n- and m-vectors, respectively. The covariance of X and

Y is defined by the n × m matrix

Cov(X, Y ) = E[(X − E(X))(Y − E(Y )) ]. (1.31)

When Y = X, we have the variance matrix, which is sometimes called a covariance

matrix of X, V ar(X) = Cov(X, X).
It can be verified that the expectation, variance, and covariance have the following
properties:

E(AX + BY ) = AE(X) + BE(Y ) (1.32a)

  
E((AX)(BY ) ) = A(E(XY ))B (1.32b)
V ar(X) ≥ 0, (1.32c)
Cov(X, Y ) = (Cov(Y , X)) , (1.32d)

and

Cov(X, Y ) = E(XY  ) − E(X)(E(Y )) , (1.32e)

where A and B are constant matrices (cf. Exercise 1.8). X and Y are said to be
independent if f (x|y) = f1 (x) and f (y|x) = f2 (y), and X and Y are said to be uncor-
related if Cov(X, Y ) = 0. It is easy to see that if X and Y are independent then
they are uncorrelated. Indeed, if X and Y are independent then f (x, y) = f1 (x)f2 (y).
Hence,
 ∞ ∞

E(XY ) = xy f (x, y)dxdy
−∞ −∞
 ∞  ∞
= xf1 (x)dx y f2 (y)dy
−∞ −∞
= E(X)(E(Y )) ,

so that by property (1.32e) Cov(X, Y ) = 0. But the converse does not necessarily
hold, unless the probability distribution is normal. Let
    
X1 μ1
X= ∼N ,R ,
X2 μ2

where
 
R11 R12 
R= , R12 = R21 ,
R21 R22

R11 and R22 are symmetric, and R is positive definite. Then it can be shown that X1
and X2 are independent if and only if R12 = Cov(X1 , X2 ) = 0 (cf. Exercise 1.9).
14 1 Preliminaries

Let X and Y be two random vectors. Similar to the definitions of expectation and
variance, the conditional expectation of X under the condition that Y = y is defined
to be
 ∞
E(X|Y = y) = xf (x|y)dx (1.33)
−∞

and the conditional variance, which is sometimes called the conditional covariance
of X, under the condition that Y = y to be

V ar(X|Y = y)
 ∞
= [x − E(X|Y = y)][x − E(X|Y = y)] f (x|y)dx. (1.34)
−∞

Next, suppose that

       
X μx X Rxx Rxy
E = and V ar = .
Y μy Y Ryx Ryy

Then it follows from (1.24) that

 
x
f (x, y) = f
y
1
=   1/2
Rxx Rxy
(2π)n/2 det
Ryx Ryy
⎧      ⎫
⎨ 1 −1    ⎬
x μx Rxx Rxy x μx
· exp − − − .
⎩ 2 y μy Ryx Ryy y μy ⎭

It can be verified that

f (x, y)
f (x|y) =
f (y)
 
1 1  −1
= exp − (x − μ̃) R̃ (x − μ̃) , (1.35)
(2π)n/2 (det R̃)1/2 2

where
−1
μ̃ = μx + Rxy Ryy (y − μy )

and
−1
R̃ = Rxx − Rxy Ryy Ryx
1.2 Probability Preliminaries 15

(cf. Exercise 1.10). Hence, by rewriting μ̃ and R̃, we have

E(X|Y = y) = E(X) + Cov(X, Y )[V ar(Y )]−1 (y − E(Y )) (1.36)

and

V ar(X|Y = y) = V ar(X) − Cov(X, Y )[V ar(Y )]−1 Cov(Y , X). (1.37)

1.3 Least-Squares Preliminaries

Let {ξ k } be a sequence of random vectors, called a random sequence. Denote E(ξ k ) =

μk , Cov(ξ k , ξ j ) = Rkj so that V ar(ξ k ) = Rkk := Rk . A random sequence {ξ k } is
called a white noise sequence if Cov(ξ k , ξ j ) = Rkj = Rk δkj where δkj = 1 if k = j
and 0 if k = j. {ξ k } is called a sequence of Gaussian (or normal) white noise if it is
white and each ξ k is normal.
Consider the observation equation of a linear system where the observed data is
contaminated with noise, namely:

vk = Ck xk + Dk uk + ξ k ,

where, as usual, {xk } is the state sequence, {uk } the control sequence, and {vk } the
data sequence. We assume, for each k, that the q × n constant matrix Ck , q × p
constant matrix Dk , and the deterministic control p-vector uk are given. Usually,
{ξ k } is not known but will be assumed to be a sequence of zero-mean Gaussian
white noise, namely: E(ξ k ) = 0 and E(ξ k ξ 
j
) = Rkj δkj with Rk being symmetric and
positive definite, k, j = 1, 2, · · · .
Our goal is to obtain an optimal estimate ŷk of the state vector xk from the
information {vk }. If there were no noise, then it is clear that zk − Ck ŷk = 0, where
zk := vk − Dk uk , whenever this linear system has a solution; otherwise, some mea-
surement of the error zk − Ck yk must be minimized over all yk . In general, when the
data is contaminated with noise, we will minimize the quantity:

F(yk , Wk ) = E(zk − Ck yk ) Wk (zk − Ck yk )

over all n-vectors yk where Wk is a positive definite and symmetric q × q matrix,

called a weight matrix. That is, we wish to find a ŷk = ŷk (Wk ) such that

F(ŷk , Wk ) = min F(yk , Wk ).

yk
Another random document with
no related content on Scribd:
INDEX
 INDEX
Adams Presses, 50
Æthelwald, 122
Alba, the Duke of, 233
Alcuin Bible, the, described, 125–127
Alcuin, Bishop, of York, 125, 126
Aldine Press, the, at Venice, saved by intervention of Jean Grolier, 56, 238;
printing at, 206–215;
the Jenson office combined with, 214
Aldus Manutius, legend over office of, 10;
his confidence in permanence of the printed book, 11–12;
his type designs, 17;
establishes his office in Venice, 206;
his printer’s mark and slogan, 207, 208;
changes format of the book, 207;
his aims, 208;
the Greek classics of, 209;
his contributions to typography, 210;
his Hypnerotomachia Poliphili, 210–213;
Jean Grolier’s friendship with family of, 214–215
Allegro, l’, Milton’s, 93
Ambrosiana Iliad, the, 24, 25
Ambrosiana Library, the, humanistic manuscripts in, 24, 25
Angelico, Fra, 149, 290
Anglo-Saxon missionary artists, the, 125
Anne, of Brittany, Hours of, described, 149–151
Anne, Saint, 138
Antiquities of the Jews, the, described, 138–141, 146
Antonio del Cherico, Francesco d’, Book of Hours illuminated by, 111, 113,
116, 146–149, 290
Antwerp, the leading city in Europe, 223;
book manufacture in, 224;
under Spanish domination, 227;
loses her pre-eminence, 233;
purchases the Plantin office, 235;
referred to, 239
Apostrophes, Bernard Shaw’s ideas concerning, 68
Arnold, Matthew, 178
Ashburnham Collection, the, 148, 284
Augustinus, 202
Austria, the Emperor of, 105
Authors, relations between publishers and, 51, 63;
their attitude toward the physical format of their books, 67

Bandini, librarian of the Laurenziana Library, 284

Baptistery, the, at Florence, 273
Barbaro, Marco Antonio, Procurator of Saint Mark’s, 143
Bardi, the, 297
Bardi, Bardo, 297
Barlow, Sir Thomas, 85
Baskerville, John, his editions, 245;
letter from Benjamin Franklin to, 245;
his types, 245–246;
his Virgil, 246–250;
first to introduce glossy paper, 250;
Dibdin’s estimate of, 251;
referred to, 95, 244
Baynes, Ernest Harold, 104
Bedford, Anne, Duchess of, 135, 137
Bedford Book of Hours, the, described, 135–138, 146
Bedford, John, Duke of, 135, 137
Belgium, see Netherlands, the
Bellini, Giovanni, 213
Beowulf, William Morris’, 259
Berlin, library of, 196
Berry, the Duc de, the Très Riches Heures of, 116;
the Antiquities of the Jews begun for, 139
Bertieri, Raffaello, 32
Bewick, Thomas, 163, 251
Biagi, Dr. Guido, custodian of the Buonarroti and the da Vinci archives, 14,
182, 290;
defines the humanist, 15, 162;
his association with the designing of the Humanistic type, 17–33;
his comments on Bodoni, 78;
his meeting with Charles Eliot Norton, 180–183;
described, 277;
in the Laurenziana Library, 277–300;
his early ambition to become librarian of the Laurenziana, 284;
in America, 299;
his last days, 299–300;
his death, 300;
referred to, 14, 16, 17, 111
Bible, the, welfare of men and of empires based upon, 224
Biblia Amiatina, the, 287, 288
Biblia Polyglotta, Plantin’s, 227;
the story of, 227–233;
pages from, 229–231
Bibliothèque Nationale, the, Paris, 119, 127, 128, 139, 140, 141, 149, 196,
198, 220
Billfrith, Saint, 122
Bindings, 113, 288
Birmingham, England, 244
Biscioni, librarian of the Laurenziana Library, 283
Bisticci, quoted, 12
Blanche, Queen, of Castile, 129
Boccaccio, 287
Bodleian Library, the, 196
Bodoni, Giambattista, the father of modern type design, 78–82, 251;
compared with Didot, 252, 257;
referred to, 95
Bodoni Press, the revived, in Montagnola di Lugano, 79
Bodoni type, the, 78;
compared with the Didot type, 79–82;
William Morris’ dislike of, 80;
De Vinne’s admiration for, 80, 82;
estimate of, 257
Bokhara, 118
Bomberghe, Corneille, type designer, 228
Book, the, conception of early patrons of, 11;
lure of, 37;
the tangible expression of man’s intellect, 112.
See also, Illuminated book, Printed book, Written book
Bookmaking, in 1891, 42–54;
the weakness of method in, 54
Book of Hours, by Francesco d’Antonio del Cherico, 111;
described, 146–149;
referred to, 290
Books, cost of making, 58
Bookselling, inadequate methods in, 60
Boston, Howell’s comments on, 186
Boston Society of Printers, the, 99
Bourbon, Pierre II, Duc de, 140
Bourdichon, Jean, 113, 149, 150
Boyd, Henry, 26, 27
Brays, the, 292
British Museum, the, 27, 28, 117, 119, 122, 125, 132, 135, 166, 196, 298, 299
Broad Chalke, England, Maurice Hewlett’s home at, 157
Browne, Sir Thomas, 207
Budé, Guillaume, 214
Buonarroti, see Michelangelo
Buonarroti archives, the, 14, 182, 290
Burne-Jones, Sir Edward, and the Kelmscott Chaucer, 259, 261, 262;
referred to, 6
Burney, Fanny, 163
Byron, Lord, manuscript of his letters burned by John Murray III, 65
Byzantine illumination, see Illumination, Byzantine
Byzantine ink, 112

“Cabin,” the, Howell’s, 186

Cable, George W., 177
Cæsar, Elzevir’s, 240
Cambridge Immortals, the, 178
Camp, Walter, 84
Campanile, Giotto’s, at Florence, 273
Campanile, the, at Venice, 145
Carlyle, Thomas, 178
Carnegie, Andrew, 174–177
Carnegie, Mrs. Andrew, 174
Caroline minuscule, the, 126
Carolingian illumination, see Illumination, Carolingian
Carolingian School, the, in France, 125
Caslon foundry, the, 245, 246
Castiglioncello, Italy, 162
Cato, quoted, 208
Caxton, William, work of, compared with Jenson’s, 244
Cellini, Benvenuto, autobiography of, 287
Celtic illumination, see Illumination, Celtic
Censors, the, 221
Ceolfrid, Abbot of Wearmouth, 288
Ceriani, Monsignor, librarian of the Ambrosiana Library at Milan, described,
24;
his work on the Ambrosiana Iliad, 24, 25;
quoted, 25
Chantilly, the Musée Condé at, 116
Charlemagne, Emperor, 125
Charles, King, of France, son of King John, 129
Charles, King, of France, son of King Charles, 129
Charles V, emperor of the Holy Roman Empire, 216, 276
Charles VIII, of France, 282
Chaucer, the Kelmscott, the story of, 259–268
Chaucer type, the, designed by William Morris, 20
Chianti Hills, the, 171
Chiari, 297
Chinese, the, 7
Cicero, the Medicean, 283
Cicogna, Doge Pasquale, 143
Cimabue, Giovanni, 147
Clemens, Clara, 172, 174
Clemens, Samuel L., see Twain, Mark
Clemens, Mrs. Samuel L., 172, 173
Clement VII, Pope, 214, 276
Clovio, Giulio, 149, 290
Cobden-Sanderson, T. J., quoted, 96, 97;
estimate of, 96–101;
described, 98;
in Boston, 98–99;
importance of his work, 263;
his Ideal Book
quoted, 264;
his Doves Bible, 264–268;
referred to, 3, 68, 71, 95
Cockerell, Douglas, 262
Codex Argenteus, the, 119
Cole, Timothy, 106
Colonna, Francesco, 210
Colvin, Sir Sidney, 27, 28
Constable, Archibald, publisher, 66
Constantinople, might have become center of learning of XV century, 8;
destroyed by fire, 117;
the rebirth of, 118
Cosimo il Vecchio, and the Laurenziana Library, 280;
his personality and history, 280–281;
his fame as a chemist, 284
Cosimo II, and the Laurenziana Library, 283
Costs of making books, in 1891, compared with present costs, 48
Costume Antico e Moderno, Ferrario’s, 296
Country Printer, the, Howell’s, 187
Crasso, Leonardo, 210
Cremona, early printing at, 286
Curtis, George William, 178
Cuthbert, Saint, 120, 121, 122
Cyrus, King, 139

Danes, the, 120

Dante, proposed edition in Humanistic type of, 19, 32;
referred to, 158, 182;
Biagi’s work on, 182, 299;
the haunts of, 291
“Dawn and Twilight,” Michelangelo’s, 275
“Day and Night,” Michelangelo’s, 275
De Asse, Budé’s, 214
De Bure, discoverer of the Gutenberg Bible, 198
Decameron, the, manuscript copy of, 287
De Civitate Dei, Augustinus’, 202
Decorations, 116
Del Furia, librarian of the Laurenziana Library, 284
Delmonico’s, in New York City, 176
Deput, quoted on the innovations of the Elzevirs, 240
De Veritate Catholicæ Fidei, 93
De Vinne Press, the, New York, 42
De Vinne, Theodore L., 6;
his admiration for the Bodoni type, 80, 82
Dibdin, quoted on Baskerville, 251;
on Antonio Magliabecchi, 295
Didot, Firmin, the father of modern type design, 78–82;
his type discussed, 79–82, 257;
referred to, 39, 95, 257
Didot, Pierre, his Racine, 252–258
Didot Press, the, Benjamin Franklin at, 252
Didot type, the, compared with the Bodoni type, 79–82
Didots, the, in Paris, 251;
 compared with Bodoni, 252
Dobson, Austin, 158, 162–169;
his lines on Richard Garnett, 166, 167;
his ideas on fiction, 168;
his methods of work, 169;
his handwriting, 169
Dobson, Mrs. Austin, 168
Doves Bible, Cobden-Sanderson’s, described, 264–268
Doves Press, the, in London, 3, 70, 96, 263
Doves type, the, designed by Emery Walker, 18, 19;
specimen page of, 23;
in the Doves Bible, 264–268
Duneka, Frederick, 184
Dürer, Albrecht, 95
Dyck, Christoffel van, 243

Eadfrith, Bishop, 122

Earthwork Out of Tuscany, Hewlett’s, 159, 162
Eddy, Mrs. Mary Baker, 52–54
Edward VII, King, of England, 85, 140, 141
Egyptians, the, 118
Elchi, Count Angiolo Maria d’, 285
Eliot, George, in the Magliabecchian Library, 291–298;
her diary quoted, 296;
volumes consulted in writing Romola, 296–298
Elzevir, Abraham, editions of, 240;
his Terence, 241–243
Elzevir, Bonaventura, editions of, 240;
his Terence, 241–243
Elzevir, Isaac, becomes printer to the University of Leyden, 240
Elzevir, Louis, founder of the House of Elzevir, 239, 240
Elzevir, the House of, craftsmen rather than artists, 238;
in Leyden, 240;
adopt new format for the book, 240;
their editions, 240–243;
their types, 243;
their business organization, 243;
estimate of importance of their work, 243;
referred to, 95, 237, 251
England, typographical supremacy of, 194, 244–250;
second supremacy of, 258–268
English illumination, see Illumination, English
Engravings, steel, 105
Epistolæ ad Familiares, Cicero’s, 285
Ethics, in business, 65
Étienne, Henri, ruined by his Thesaurus, 56, 238;
in Geneva, 223
Étienne, Robert, becomes “printer in Greek” to François I of France, 216;
the Royal Greeks of, 219–222;
leaves France, 223;
death of, 223;
his Roman type, 222;
referred to, 252
Eugenius, Pope, 281
Evreux, Queen Jeanne d’, 129
Explicit, the, 92;
examples of, 94, 202, 204, 206, 285
Ezra, the Prophet, portrait of, 288

Famiglie del Litta, Le, 297

Felton, Cornelius Conway, President of Harvard University, 50
Ferrara, early printing at, 286
Ferrari, Dr. Luigi, librarian of the San Marco Library, Venice, 145
Ferrario, 296
Field, Eugene, described, 38;
manuscript of, 39, 41;
referred to, 38, 55
Fielding, Henry, 163
Fiesole, the heights of, 298
Firenze Antica e Moderna, Rastrelli’s, 297
Firenze Illustrata, Migliore’s, 296
Fiske, Willard, 26, 27
Flemish illumination, see Illumination, Flemish
Fletcher, Horace, friend of Eugene Field, 41;
philosophy of, 75, 82, 84;
his ideas of typography, 75;
page of his manuscript, 77;
his dinner at Graduates’ Club, New Haven, 84;
importance of his work, 85;
his friendship with William James and Henry James, 86;
letter from Henry James to, 87;
visit to Lamb House, 89
Fletcherism, 75, 83
Florence, Italy, the most fascinating city in Europe, 273;
early printing at, 286
Florence, the Grand Duke of, 295
Forest Lovers, the, Hewlett’s, 157, 158
Foucquet, Jean, 113, 138, 140, 149
France, typographical supremacy of, 194, 215–223;
loses supremacy, 223;
second supremacy of, 251–258
François I, of France, becomes patron of learning and culture, 216;
makes Robert Étienne “printer in Greek to the King,” 216;
his interest in printing, 216–221;
his relations with the censors, 221;
 referred to, 214, 216
Frankfort, 227
Franklin, Benjamin, quoted on the Baskerville editions, 245;
his letter to Baskerville, 245;
at the Didot Press, 252
French illumination, see Illumination, French
French Republic, the, 141
French School of Painting, the, 139
Fust, John, 198, 199

Gabrilowitch, Mrs. Ossip, 172

Garamond, Claude, 220
Garnett, Dr. Richard, 164, 165;
lines written by Dobson on, 166, 167;
estimate of, 166
General Theological School Library, the, New York, 196
Genesis, the Cottonian, 117
Geneva, the Étiennes at, 223
George, Saint, 137
Germany, not sufficiently developed as nation to take advantage of
Gutenberg’s discovery, 8, 9;
brief typographical supremacy of, 194–201;
loses supremacy, 201
Gherado, 149, 290
Gilder, Richard Watson, 177
Giotto, 147, 273
Golden Gospels of Saint Médard, the, described, 127–128
Golden type, the, designed by William Morris, 18
Gold leaf, 116
Gold, Oriental, 112
Goldsmith, Oliver, 163
Gothic illumination, see Illumination, Gothic
Gozzoli, Benozzo, 149, 290
Graduates’ Club, the, in New Haven, 84
Grandjon, Robert, 228
Greece, the rich humanities of, 15
Greek classics, the, first printed by Aldus, 209
Greeks, the, 7
Greek types, 56, 219–221, 238
Grimani Breviary, the, described, 141–145, 146, 149
Grimani, Cardinal Domenico, 142, 143
Grimani, Doge Antonio, 143
Grimani, Giovanni, Patriarch of Aquileia, 142
Grimani, Marino, Patriarch of Aquileia, 142
Grolier Club of New York, the, 213
Grolier, Jean, saves the Aldine Press by his intervention, 56, 238;
his friendship with family of Aldus, 214–215;
his letter to Francesco Torresani, 215
Guidobaldo, Duke of Urbino, 210
Gutenberg Bible, the, described, 194–201;
rubricator’s notes, 196, 197
Gutenberg, John; the Bible printed by, 194–201;
referred to, 7, 194, 198, 234, 237

Hadley, Pres. Arthur T., 84

Halftones, 105–107
Hammond, John Hays, 84
Hand lettering, the art of, 10.
See also, Humanistic hand lettering, Semi-uncial characters, Minuscule characters
Harkness, Mrs. Edward S., 196
Harper and Brothers, 90, 183, 184
Harper’s Magazine, 183
Harte, Bret, 184
Harvard College Library, the, 245
Harvard University, Cobden-Sanderson’s lectures at, 99
Harvey, Col. George, gives birthday dinner to Mark Twain, 176;
gives birthday dinner to William Dean Howells, 187
Hautin, 228
Hay, John, 184
Hay, Mrs. John, 184
Heidelberg, 227
Heimskringla, the, William Morris’ translation of, 260
Heinsius, letter from Deput to, 240
Henri II, of France, 136, 221
Henry IV, of England, 135
Henry VI, of England, 136
Henry VIII, of England, 216
Hewlett, Maurice, 155–162;
describes the cloister of San Lorenzo, Florence, 275
Hoar, Senator George F., makes attack on Charles Eliot Norton, 179
Hogarth, William, 163
Holbein, Hans, 95
Holland, the natural successor to Belgium in supremacy of printing, 239.
See also Netherlands, the
Holmes, Oliver Wendell, 178
Homer, the Ambrosiana, see Iliad, the Ambrosiana
Hoover, Herbert, 85
Houghton, Henry O., 6
Hours of Anne of Brittany, the, 146;
described, 149–151
Howells, William Dean, 177;
recollections and reflections on, 183–188;
the Harvey birthday dinner, 188
Humanism, Petrarch the father of, 15
Humanist, the, defined, 15, 160–162
Humanistic hand lettering, 16, 21, 24, 126
Humanistic manuscripts, the, in the Laurenziana Library, 16, 21;
in the Ambrosiana Library, 24
Humanistic movement, the, far-reaching influence of, 15;
the forerunner and essence of the Renaissance, 15;
significance of, 16, 160
Humanistic scribes, the, see Scribes, the humanistic
Humanistic type, the, first idea of design of, 17;
proposed edition of Dante in, 19, 32;
work upon, 19–24, 126, 159, 180, 181
Huntington, Henry E., library of, 196
Hypnerotomachia Poliphili, the, printed by Aldus, 210;
described, 210–213

Ideal Book, The, Cobden-Sanderson’s, 3, 99;

quoted, 264
Iliad, the Ambrosiana, 24, 25, 117
Illuminated book, the, attitude of Italian patrons toward, 11–12
Illumination, the art of, encouraged by Italian patrons in XV century, 11–13;
the underlying thought in, 112;
rich rewards in study of, 114;
various schools of, 114;
means of identifying various schools and periods, 115;
manuscripts which mark the evolution of, 116–119;
the Celtic School, 119–125;
 the Carolingian School, 125–128;
the Gothic School, 128–131;
the English School, 131–134;
the French School, 135–141, 149–151;
the Flemish School, 141–145;
the Italian School, 146–149;
cause for the decline of, 151;
opportunities for studying, 152
Illumination, Byzantine, described, 118;
referred to, 124, 125, 127
Illumination, Carolingian, 125–128
Illumination, Celtic, 119–125, 126, 129
Illumination, English, 114, 131–134
Illumination, Flemish, 114, 137, 139, 141–145, 149, 150
Illumination, French, 114, 135–141
Illumination, Gothic, 114, 128–131
Illumination, Italian Renaissance, 114, 146–149, 150
Illumination, Romanesque, 114
Illustration, 105
Imperial Library, the, in Vienna, 117
Incipit, the, 93
India, 112, 118
Ink, Byzantine gold, 112, 118;
inferior quality introduced, 238;
Didot’s, 258
Innocents Abroad, Mark Twain’s, 170
Ireland, 121
Irish monks, the, see Monks, the Irish
Irish School of Writing and Painting, the, 120
Italian illumination, see Illumination, Italian
Italic type, first used by Aldus, 17, 286;
said to be based on handwriting of Petrarch, 17, 210;
Baskerville’s, 245;
Didot’s, 257
Italy, life and customs of people of, in XV century, 8;
illumination slow in getting a hold in, 146;
typographical supremacy of, 194, 201–215;
loses supremacy, 215;
culture in the XVI century in, 223

James, Henry, Horace Fletcher’s friendship with, 86;

quoted, 86;
estimate of, 86;
letter to Horace Fletcher from, 87;
quoted, 88
James, William, Horace Fletcher’s friendship with, 86;
quoted, 86;
letter from Henry James to, 88;
his interest in printing, 90, 92
Jenson, Nicolas, type designs of, 18, 19, 22, 202, 205, 206;
the explicit in books printed by, 94, 202;
printer’s mark of, 203, 207;
sent to Germany by Charles VII of France, 204;
establishes his office in Venice, 204;
death of, 206;
his office combined with the Aldine Press, 214;
Caxton’s work compared with, 244;
referred to, 216, 286
Jenson’s Gothic type, 202, 205, 206
Jenson Roman type, the, 18, 19, 206, 268;
sample page of, 22
Joan of Arc, Mark Twain’s, 170
John of Spires, 285
Jones, George W., 32
Joseph, Saint, 138
Josephus, Flavius, 139
Justinian, the Emperor, 117
Justinian, the Pandects of, 287

Keats, John, 158

Keere, Van der, 228
Kelmscott Chaucer, the, see Chaucer, the Kelmscott
Kelmscott Press, the, 6, 55, 70, 96, 259–268
Kelmscott volumes, the, 259–268;
estimate of, 263
Koreans, the, 7

Labels, paper, 92
Lamb House, Rye, Henry James’ home, 89
Lapis lazuli, used in printing ink, 30;
in illumination, 118
Laura, see Sale, Madonna Laura de Noves de
Laurenziana Library, the, humanistic volumes at, 16;
illuminated volumes at, 119, 148, 287;
uninviting approach to, 273;
the Sala di Michelangiolo, 276;
Dr. Guido Biagi at, 277–300;
the great staircase, 278;
Vasari’s work in, 278;
the story of, 280–284;
the treasures of, 284–289;
the Hall of Tapestries, 285;
the Tribuna, 285;
the printed books in, 285;
the spell of, 300;
referred to, 14, 21, 94, 111, 182
Le Bé, Guillaume, 228
Lee, Sir Sidney, 86, 174, 175
Leigh, Maj. Frederick T., 184
Leipzig, library of, 196
Lelio, Lucrezia, 213
Leo X, Pope, 282.
See also Medici, Giovanni de’
Lettering, see Hand lettering
Letters, raised gold, 116
Lewes, George Henry, in the Magliabecchian Library, 292–298
Leyden, heroic resistance to Spanish siege, 239;
becomes the intellectual and literary center of Europe, 239;
Plantin in, 239;
the Elzevirs in, 239–240
Leyden, the University of, 239;
Plantin made printer to, 239;
Isaac Elzevir made printer to, 240
Lindisfarne Gospels, the, described, 119–125
Lippi, 296
Lippi, Fra Filippo, 28
Lipsius, the historian, 239
Lithography, 105
Little Novels of Italy, Hewlett’s, 159
Lockhart, John Gibson, 66
Longfellow, Henry Wadsworth, 178
Longmans, London publishers, 66
Lorenzo the Magnificent, see Medici, Lorenzo de’
Lorraine, the House of, and the Laurenziana Library, 283
Louis XI, of France, 140
Louis XII, of France, 150
Louis XIV, of France, 198