Textbook Abstract Methods in Information Theory 2Nd Edition Kakihara Ebook All Chapter PDF

Abstract Methods in Information Theory
2nd Edition Kakihara

Visit to download the full and correct content document:
https://textbookfull.com/product/abstract-methods-in-information-theory-2nd-edition-k
akihara/
More products digital (pdf, epub, mobi) instant
download maybe you interests ...
Quantum Information Theory 2nd Edition Mark M. Wilde
https://textbookfull.com/product/quantum-information-theory-2nd-
edition-mark-m-wilde/
Financial Markets Theory: Equilibrium, Efficiency and

Information 2nd Edition Emilio Barucci
https://textbookfull.com/product/financial-markets-theory-
equilibrium-efficiency-and-information-2nd-edition-emilio-
barucci/
Functional Numerical Methods: Applications to Abstract

Fractional Calculus 1st Edition George A. Anastassiou
https://textbookfull.com/product/functional-numerical-methods-
applications-to-abstract-fractional-calculus-1st-edition-george-
a-anastassiou/
Abstract Algebra Applications to Galois Theory

Algebraic Geometry Representation Theory and
Cryptography Second Edition Celine Carstensen-Opitz
https://textbookfull.com/product/abstract-algebra-applications-
to-galois-theory-algebraic-geometry-representation-theory-and-
cryptography-second-edition-celine-carstensen-opitz/
Prevention of Accidents and Unwanted Occurrences:
Theory, Methods, and Tools in Safety Management 2nd
Edition Albrechtsen
https://textbookfull.com/product/prevention-of-accidents-and-
unwanted-occurrences-theory-methods-and-tools-in-safety-
management-2nd-edition-albrechtsen/
Foundations of Geophysical Electromagnetic Theory and

Methods 2nd Edition Michael S. Zhdanov
https://textbookfull.com/product/foundations-of-geophysical-
electromagnetic-theory-and-methods-2nd-edition-michael-s-zhdanov/
Electronic Structure Basic Theory and Practical Methods

2nd Edition Richard M. Martin
https://textbookfull.com/product/electronic-structure-basic-
theory-and-practical-methods-2nd-edition-richard-m-martin/
A course in abstract algebra Nicholas Jackson
https://textbookfull.com/product/a-course-in-abstract-algebra-
nicholas-jackson/
Problems in Abstract Algebra 1st Edition Adrian R.

Wadsworth
https://textbookfull.com/product/problems-in-abstract-
algebra-1st-edition-adrian-r-wadsworth/
Abstract Methods in
Information Theory
9954.9789814759236.v2-tp.indd 1 19/5/16 10:32 AM

SERIES ON MULTIVARIATE ANALYSIS
Editor: M M Rao ISSN: 1793-1169
Published
Vol. 1: Martingales and Stochastic Analysis

J. Yeh
Vol. 2: Multidimensional Second Order Stochastic Processes
Y. Kakihara
Vol. 3: Mathematical Methods in Sample Surveys
H. G. Tucker
Vol. 4: Abstract Methods in Information Theory
Y. Kakihara
Vol. 5: Topics in Circular Statistics
S. R. Jammalamadaka and A. SenGupta
Vol. 6: Linear Models: An Integrated Approach
D. Sengupta and S. R. Jammalamadaka
Vol. 7: Structural Aspects in the Theory of Probability
A Primer in Probabilities on Algebraic-Topological Structures
H. Heyer
Vol. 8: Structural Aspects in the Theory of Probability (Second Edition)
H. Heyer
Vol. 9: Random and Vector Measures
M. M. Rao
Vol. 10: Abstract Methods in Information Theory (Second Edition)
Y. Kakihara
LaiFun - Abstract Methods in Information Theory.indd 1 10/5/2016 9:06:33 AM

S e r i e s o n
Multivariate
Analysis
Vol. 10
Second Edition
Abstract Methods in
Information Theory
Yûichirô Kakihara
California State University, San Bernardino, USA
World Scientific
NEW JERSEY • LONDON • SINGAPORE • BEIJING • SHANGHAI • HONG KONG • TAIPEI • CHENNAI • TOKYO
9954.9789814759236.v2-tp.indd 2 19/5/16 10:32 AM

Published by
World Scientific Publishing Co. Pte. Ltd.
5 Toh Tuck Link, Singapore 596224
USA office: 27 Warren Street, Suite 401-402, Hackensack, NJ 07601
UK office: 57 Shelton Street, Covent Garden, London WC2H 9HE
Library of Congress Cataloging-in-Publication Data

Names: Kakihara, Yūichirō.
Title: Abstract methods in information theory / by Yūichirō Kakihara
(California State University, San Bernardino, USA).
Description: Second edition. | New Jersey : World Scientific, 2016. | Series:
Series on multivariate analysis ; volume 10 | Includes bibliographical
references and index.
Identifiers: LCCN 2016015934 | ISBN 9789814759236 (hardcover : alk. paper)
Subjects: LCSH: Information theory. | Functional analysis.
Classification: LCC Q360 .K35 2016 | DDC 003/.54--dc23
LC record available at https://lccn.loc.gov/2016015934
British Library Cataloguing-in-Publication Data

A catalogue record for this book is available from the British Library.
Copyright © 2016 by World Scientific Publishing Co. Pte. Ltd.

All rights reserved. This book, or parts thereof, may not be reproduced in any form or by any means,
electronic or mechanical, including photocopying, recording or any information storage and retrieval
system now known or to be invented, without written permission from the publisher.
For photocopying of material in this volume, please pay a copying fee through the Copyright Clearance
Center, Inc., 222 Rosewood Drive, Danvers, MA 01923, USA. In this case permission to photocopy
is not required from the publisher.
ISBN 978-981-4759-23-6
Printed in Singapore
LaiFun - Abstract Methods in Information Theory.indd 2 10/5/2016 9:06:33 AM

May 17, 2016 14:21 ws-book9x6 Abstract Methods in Information Theory 9954-shrink page 1
Dedicated to my late advisor
Professor Hisaharu Umegaki

This page intentionally left blank
PREFACE TO THE FIRST EDITION
Half a century has passed since C. E. Shannon published his epoch-making paper
entitled “A mathematical theory of communication” in 1948. Thereafter, the so-
called “information theory” began to grow and has now established a firm and
broad field of study. Viewing from a mathematical angle, information theory might
be thought of having the following four parts: (1) the mathematical structure of
information sources, (2) the theory of entropy as amounts of information, (3) the
theory of information channels, and (4) the theory of coding. Probabilistic and
algebraic methods have mainly been used to develop information theory. Since the
early stage of the expansion of information theory, however, measure theoretic and
functional analysis methods have also been applied and are providing a powerful
tool to obtain rigorous results in this theory. The purpose of this book is to present
the first three parts of information theory, mentioned above, in the environment of
functional analysis, in addition to probability theory.
Here are a couple of examples in each of which functional analysis played a cru-
cial role obtaining important results in information theory. The coincidence of the
ergodic capacity Ce and the stationary capacity Cs for a certain channel was one
of the most important problems in the late 1950s. L. Breiman (1960) showed that
for a finite memory channel the equality Ce = Cs holds and, moreover, Ce is at-
tained by some ergodic input source (= measure) invoking Krein-Milman’s theorem
to the weak* compact convex set Ps (X) of all stationary input sources. Another
such example appeared in a characterization of ergodic channels. In the late 1960s,
H. Umegaki and Y. Nakamura independently proved that a stationary channel is er-
godic if and only if it is an extreme point of the convex set of all stationary channels.
Umegaki observed a one-to-one correspondence between the set of channels and a
set of certain averaging operators from the set of bounded measurable functions on
the compound space to the set of those on the input. Then a channel is identified
with an operator, called a channel operator, and hence we can make full use of func-
tional analysis in studying channels. In this book, readers will find how functional
analysis helps to describe information theory, especially the mathematical structure
of information sources and channels, in an effective way.
Here is a brief summary of this book. In Chapter I, entropy is considered as the
amount of information. Shannon’s entropy for finite schema is defined and its basic
properties are examined together with its axioms. After collecting fundamental prop-
vii
viii ABSTRACT METHODS IN INFORMATION THEORY
erties of conditional expectation and probability, Kolmogorov-Sinai’s entropy is then

obtained for a measure preserving transformation. Some fundamental properties of
the Kolmogorov-Sinai entropy are presented along with the Kolmogorov-Sinai theo-
rem. Algebraic models are introduced to describe probability measures and measure
preserving transformations. Some conjugacy problems are studied using algebraic
models. When we fix a measurable transformation and a finite partition, we can
consider Kolmogorov-Sinai’s entropy as a functional on the set of invariant (with
respect to the transformation) probability measures, called an entropy functional.
This functional is extended to be the one defined on the set of all complex valued
invariant measures, and its integral representation is obtained. Relative entropy and
Kullback-Leibler information are also studied in connection with hypothesis testing.
In Chapter II, information sources are considered. Using an alphabet message

space as a model, we describe information sources on a compact Hausdorff space.
Mean and Pointwise Ergodic Theorems are stated and proved. Ergodicity is one
of the important concepts and its characterization is presented in detail. Among
the nonstationary sources, AMS (= asymptotically mean stationary) sources are
of interest and the structure of this class is studied. Shannon-McMillan-Breiman
Theorem is then formulated for a stationary and an AMS source, which is regarded
as the ergodic theorem in information theory. Ergodic decomposition of a stationary
source is established and is applied to obtain another type of integral representation
of an entropy functional.
Chapter III, the main part of this book, is devoted to the information channels.
After defining channels, a one-to-one correspondence between the set of channels and
a set of certain averaging operators is established, as mentioned before. Strongly and
weakly mixing channels are defined as a generalization of finite dependent channels
and their basic properties are obtained. Ergodicity of stationary channels is discussed
and various necessary and sufficient conditions for it are given. For AMS channels,
absolute continuity plays a special role in characterizing ergodicity. Capacity and
transmission rate are defined for stationary channels. Coincidence of ergodic and
stationary capacities is proved under certain conditions. Finally, Shannon’s coding
theorems are stated and proved.
Special topics on channels are considered in Chapter IV. When a channel has a
noise source, some properties of such a channel are studied. If we regard a channel to
be a vector (or measure) valued function on the input space, then its measurabilities
are clarified. Some approximation problems of channels are treated. When the
output space is a (locally) compact abelian group, a harmonic analysis method
can be applied to channel theory. Some aspects of this viewpoint are presented in
detail. Finally, a noncommutative channel theory is introduced. We use a C*-algebra
approach to formulate channel operators as well as other aspects of noncommutative
extension.
PREFACE TO THE FIRST EDITION ix
Another purpose of this book is to present contributions of Professor Hisaharu

Umegaki and his school on information theory. His selected papers are published un-
der the title of “Operator Algebras and Mathematical Information Theory,” Kaigai,
Tokyo in 1985. As one of his students, the author is pleased to have a chance to
write this monograph.
In the text, III.4.5 denotes the fifth item in Section 4 of Chapter III. In a given
chapter, only the section and item number are used, and in a given section, only the
item number is used.
The author is grateful to Professor M. M. Rao at University of California, River-
side (UCR) for reading the manuscript and for the valuable suggestions. UCR has
provided the author with a very fine environment, where he could prepare this mono-
graph. He is also grateful for the hospitality of UCR.
Yûichirô Kakihara
Riverside, California
April, 1999
PREFACE TO THE SECOND EDITION
In 2001, shortly after the first edition of this book was published in 1999, the
founder of Information Theory, Claude E. Shannon, passed away. In 2012, the
author’s advisor Hisaharu Umegaki, who contributed to operator algebras and in-
formation theory, passed away. His school members have continued to research in
information theory and related topics. Information Theory is still growing in every
branch.
In this second edition, channel operators, which generalize ordinary channels, are
studied in many aspects. Gaussian channels are also considered in some detail, to-
gether with Gaussian measures on a Hilbert space. The Special Topics chapter deals
with features such as generalized capacity, channels with an intermediate noncom-
mutative system, and von Neumann algebra method for channels, in addition to the
topics of channels with a noise source, measurability and approximation of channels,
and harmonic analysis method for channels. Finally, quantum (noncommutative)
information channels are examined in an independent chapter, which may be re-
garded as an introduction to quantum information theory. Von Neumann entropy is
introduced and its generalization to a C*-algebra setting is given. Basic results on
quantum channels and entropy transmission are considered.
The author is grateful to Professor Emeritus M. M. Rao of University of California,
Riverside, who suggested writing this second edition and gave him encouragement
throughout the whole process. Special thanks are due to Ms. Lai Fun Kwong and
the Production Department of World Scientific Publishing Company for their help
and cooperation, so that we could remove most of the typographical errors.
Yûichirô Kakihara
San Bernardino, California
December, 2015
xi
CONTENTS
Preface to the First Edition vii
Preface to the Second Edition xi
Chapter I. Entropy 1
1.1. The Shannon entropy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
1.2. Conditional expectations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
1.3. The Kolmogorov-Sinai entropy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
1.4. Algebraic models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31
1.5. Entropy functionals . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43
1.6. Relative entropy and Kullback-Leibler information . . . . . . . . . . . . . . . . . . . 53
Bibliographical notes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67
Chapter II. Information Sources 69

2.1. Alphabet message spaces and information sources . . . . . . . . . . . . . . . . . . . . . 69
2.2. Ergodic theorems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74
2.3. Ergodic and mixing properties . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 78
2.4. AMS sources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 94
2.5. Shannon-McMillan-Breiman theorem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 102
2.6. Ergodic decompositions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 110
2.7. Entropy functionals, revisited . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 114
Bibliographical notes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 123
Chapter III. Information Channels 125

3.1. Information channels . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 125
3.2. Mixing channels . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 134
3.3. Semiergodic channels . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 141
3.4. Ergodic channels . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 149
3.5. AMS channels . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 155
3.6. Capacity and transmission rate . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 166
3.7. Coding theorems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 178
xiii
xiv ABSTRACT METHODS IN INFORMATION THEORY
Chapter IV. Channel Operators 190

4.1. Channel operators . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 190
4.2. Generalized channels and topological properties . . . . . . . . . . . . . . . . . . . . . . 206
4.3. Pseudo channel operators: General theory . . . . . . . . . . . . . . . . . . . . . . . . . . . . 220
4.4. Pseudo channel operators: Topological structure . . . . . . . . . . . . . . . . . . . . . . 229
Chapter V. Gaussian Channels 236

5.1. Probability measures on a Hilbert space . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 236
5.2. Gaussian measures: Equivalence and singularity . . . . . . . . . . . . . . . . . . . . . . 250
5.3. Gaussian channels . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 265
5.4. Additive Gaussian channels . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 272
Chapter VI. Special Topics 285

6.1. Channels with a noise source . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 286
6.2. Channel capacity, revisited . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 293
6.3. Measurability of channels . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 299
6.4. Approximation of channels . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 306
6.5. Harmonic analysis for channels . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 309
6.6. Channels with a noncommutative intermediate system . . . . . . . . . . . . . . . . 317
6.7. Von Neumann algebras generated by stochastic processes . . . . . . . . . . . . . 323
Chapter VII. Quantum Channels 333

7.1. Quantum entropy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 333
7.2. Quantum channels . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 344
7.3. Entropy transmission . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 351
References 359
Glossaries of Axioms 377
Indices 383
Notation index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 383
Author index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 390
Subject index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 393
CHAPTER I
ENTROPY
In this chapter, basic ideas of entropy are presented from the work of Shannon
and Kolmogorov-Sinai. The first one is defined for finite schema and the second is
for measure preserving transformations. Conjugacy between two measure preserving
transformations is considered in terms of their algebraic models. When a transfor-
mation is fixed, the entropy is defined for all transformation invariant probability
measures. In this case, it is called an entropy functional. An integral representation
of this functional is given. Relative entropy and Kullback-Leibler information are
studied in connection with sufficient statistics and hypothesis testing.
1.1. The Shannon entropy
We consider basic properties and axioms of Shannon’s entropy. Let n ∈ N (the

set of all positive integers) and X = {x1 , . . . , xn } be a finite set with a probability
Pn
distribution p = (p1 , . . . , pn ), i.e., pj = p(xj ) ≥ 0, 1 ≤ j ≤ n and pj = 1,
j=1
where p(·) denotes the probability. We usually denote this as (X, p) and call it a
complete system of events or a finite scheme. The entropy or the Shannon
entropy H(X) of a finite scheme (X, p) is defined by
Xn
H(X) = − pj log pj , (1.1)
j=1
where “log” is the natural logarithm and we regard 0 log 0 = 0 log 00 = 0. We also say
that H(X) is the uncertainty or information of the system (X, p). Justification
of these terminologies will be clarified later in this section. Since RHS (= right hand
side) of (1.1) depends only on the probability distribution p = (p1 , . . . , pn ) we may
also write
Xn
H(X) = H(p) = H(p1 , . . . , pn ) = − pj log pj .
j=1
1
2 I. ENTROPY
We need some notations. For n ∈ N, let ∆n denote the set of all n-dimensional
probability distributions p = (p1 , . . . , pn ), i.e.,
n
X
∆n = p = (p1 , . . . , pn ) : pj = 1, pj ≥ 0, 1 ≤ j ≤ n .
j=1
Let Y = {y1 , . . . , ym } be another finite set. The probability of (xj , yk ) and the
conditional probability of xj given yk are respectively denoted by p(xj , yk ) and
p(x ,y )
p(xj |yk ) = p(yj k )k if p(yk ) > 0. Then the conditional entropy H(X|Y ) of X
given Y is defined by
XX
H(X|Y ) = − p(y)p(x|y) log p(x|y). (1.2)
y∈Y x∈X
If we define H(X|y), called the conditional entropy of X given Y = y, by

X
H(X|y) = − p(x|y) log p(x|y),
x∈X
then (1.2) is interpreted as the average of these conditional entropies over Y . The
quantity I(X, Y ) defined below is called the mutual information between (X, p)
and (Y, q):
I(X, Y ) = H(X) − H(X|Y )
since we can easily verify that
I(X, Y ) = H(Y ) − H(Y |X)

= H(X) + H(Y ) − H(X, Y )
X p(x, y)
= p(x, y) log ≥ 0, (1.3)
x,y
p(x)p(y)
where X
H(X, Y ) = − p(x, y) log p(x, y)
x,y

is the entropy of the compound scheme (X, p), (Y, q) . The inequality (1.3)
will be proved in Theorem 1 below. If we consider two probability distributions
p, q ∈ ∆n of X, then the relative entropy H(p|q) of p with respect to q is
given by
Xn Xn
pj
H(p|q) = pj (log pj − log qj ) = pj log .
j=1 j=1
qj
1.1. THE SHANNON ENTROPY 3
If pj > 0 and qj = 0 for some j, then we define H(p|q) = ∞. Observe the

difference between H(X|Y ) and H(p|q). Relative entropy will be discussed in
detail in a later section.
The next two theorems give basic properties of entropies.
Theorem 1. Consider entropies on ∆n .

(1) H(p|q) ≥ 0 for p, q ∈ ∆n , and H(p|q) = 0 if and only if p = q.
(2) Let p ∈ ∆n and A = (ajk ) be an n × n doubly stochastic matrix, i.e.,
Pn P
n
ajk ≥ 0, ajk = ajk = 1 for 1 ≤ j, k ≤ n. Then, q = Ap ∈ ∆n and
j=1 k=1
H(q) ≥ H(p). The equality holds if and only if qk = pπ(k) , 1 ≤ k ≤ n for some
permutation π of {1, . . . , n}.
(3) H(X, Y ) = H(X) + H(Y |X) = H(Y ) + H(X|Y ).
(4) H(X, Y ) ≤ H(X) + H(Y ). The equality holds if and only if X and Y are
independent.
(5) H(X|Y ) ≤ H(X). The equality holds if and only if X and Y are independent.
Proof. (1) Assume H(p|q) < ∞. Using an inequality t log t ≥ t − 1 for t > 0, we
get
pj pj pj pj
log ≥ −1 or pj log ≥ pj − qj
qj qj qj qj
for j = 1, . . . , n. Hence
n
X X n
pj
H(p|q) = pj log ≥ (pj − qj ) = 0.
j=1
qj j=1
The statement about the equality follows from the fact that t log t = t − 1 if and
only if t = 1.
(2) q = Ap ∈ ∆n is clear. Since the function φ(t) = −t log t is concave
(= concave downward) for t > 0, we have
X n Xn Xn
H(q) = φ(qj ) = φ ajk pk
j=1 j=1 k=1
Xn Xn Xn
≥ ajk φ(pk ) = φ(pk ) = H(p).
j=1 k=1 k=1
P
n P
n
The equality holds if and only if φ ajk pk = φ(pk ) for 1 ≤ j ≤ n if and
k=1 k=1
only if for each j = 1, . . . , n, ajk = 1 for some k and ajk = 0 otherwise if and
only if qk = pπ(k) , 1 ≤ k ≤ n for some permutation π of {1, . . . , n}.
4 I. ENTROPY
(3) Observe the following computations:

X
H(X, Y ) = − p(x, y) log p(x, y)
x,y
X
=− p(x, y) log p(x)p(y|x)
x,y
X X
=− p(x, y) log p(x) − p(x, y) log p(y|x)
x,y x,y
= H(X) + H(Y |X),
giving the first equality and, similarly, H(X, Y ) = H(Y ) + H(X|Y ).

(4) is derived as follows:
X X
H(X) + H(Y ) = − p(x) log p(x) − p(y) log p(y)
x y
X
=− p(x, y) log p(x)p(y)
x,y
X
≥− p(x, y) log p(x, y), by (1),
x,y
= H(X, Y ).
By (1) the equality holds if and only if p(x, y) = p(x)p(y) for x ∈ X and y ∈ Y ,
i.e., X and Y are independent.
(5) is clear from (3) and (4).
Let R = (−∞, ∞), R+ = [0, ∞) and R+ = [0, ∞].

∞
Theorem 2. Let p = (pj ), q = (qj ) ∈ ∪ ∆n .
n=2
(1) (Positivity) H(p) ≥ 0.
∞
(2) (Continuity) H : ∪ ∆n → R+ is continuous.
n=2

(3) (Monotonicity) f (n) ≡ H n1 , . . . , n1 is an increasing function of n and

1 1
H(p1 , . . . , pn ) ≤ H ,... , = f (n).
n n
(4) (Extendability) H(p1 , . . . , pn ) = H(p1 , . . . , pn , 0).

(5) (Symmetry) H(p1 , . . . , pn ) = H(pπ(1) , . . . , pπ(n) ) for every permutation π
of {1, . . . , n}.
(6) (Additivity)
H(p1 q1 , . . . , p1 qm , p2 q1 , . . . , p2 qm , . . . , pn q1 , . . . , pn qm )
= H(p1 , . . . , pn ) + H(q1 , . . . , qm ).
P P P
(7) (Subadditivity) If rjk ≥ 0, rjk = 1, rjk = pj , rjk = qk , then
j,k k j
H(r11 , . . . , rnm ) ≤ H(p1 , . . . , pn ) + H(q1 , . . . , qm ).
(8) (Concavity) For p, q ∈ ∆n and α ∈ (0, 1) it holds that

H αp + (1 − α)q ≥ αH(p) + (1 − α)H(q).
Proof. (1), (2), (4) and (5) are obvious.

(3) f (n) = log n for n ≥ 1, so that f is an increasing function. As to the second
statement, without loss of generality we can assume pj > 0 for all j. Then
n
X
1 1
H(p1 , . . . , pn ) − H ,... , = − log n − pj log pj
n n j=1
n
X 1
= pj log
npj
j=1
Xn
1
≤ pj −1 , by t − 1 ≥ log t, t > 0,
j=1
npj
= 0,

or H(p1 , . . . , pn ) ≤ H n1 , . . . , n1 .
(6) follows from the following computation:
H(p1 q1 , . . . , p1 qm , p2 q1 , . . . , p2 qm , . . . , pn q1 , . . . , pn qm )
n X
X m
=− pj qk log pj qk
j=1 k=1
X X
=− pj qk log pj − pj qk log qk
j,k j,k
= H(p1 , . . . , pn ) + H(q1 , . . . , qm ).
(7) is a reformulation of Theorem 1 (4).

6 I. ENTROPY
(8) Since φ(t) = −t log t is concave for t > 0 we have

φ αpj + (1 − α)qj ≥ αφ(pj ) + (1 − α)φ(qj ), 1 ≤ j ≤ n.
Summing with respect to j we obtain the desired inequality.
We note that the relative entropy H(p|q) also has concavity. That is, for
p1 , p2 , q1 , q2 ∈ ∆n and α ∈ (0, 1) we have that

H αp1 + (1 − α)p2 αq1 + (1 − α)q2 ≤ αH(p1 |q1 ) + (1 − α)H(p2 |q2 ). (1.4)
A generalized version of this is shown in Theorem 6.3 (2). Here we give an elementary
proof to (1.4). First we prove that for ai , bi ≥ 0, 1 ≤ i ≤ n
P
n
n
X X
n ai
ai
ai log ≥ ai log i=1
P
n , (1.5)
bi
i=1 i=1 bi
i=1
where the equality holds if and only if abii = const.

bi ai
In fact, we can assume ai , bi > 0 for 1 ≤ i ≤ n. Let αi = P
n and ti = bi for
bj
j=1
P
n
1 ≤ i ≤ n. Then, since αi > 0 (1 ≤ i ≤ n) and αi = 1, Jensen’s inequality
i=1
for φ(t) = t log t, t > 0
n
X X
n
αi φ(ti ) ≥ φ α i ti
i=1 i=1
yields that
n n n
!
X bi ai a i X bi X bi ai
P
n · log ≥ P
n log P
n · .
bi bi bi
i=1 bj i=1 bj i=1 bj
j=1 j=1 j=1
Then the desired inequality (1.5) follows from the above inequality. Now, if write
pi = (pi1 , . . . , pin ) and qi = (qi1 , . . . , qin ) for i = 1, 2, then (1.5) implies that for
1≤j≤n
αp1j + (1 − α)p2j αp1j (1 − α)p2j

αp1j +(1−α)p2j log ≤ αp1j log +(1−α)p2j log .
αq1j + (1 − α)q2j αq1j (1 − α)q2j
Adding both sides with respect to j gives (1.4).

Before characterizing
the Shannon entropy we consider the function f (n) =
H n1 , . . . , n1 , n ≥ 1. f (n) standsfor the entropy or uncertainty or information
that a finite scheme X, n1 , . . . , n1 has. We impose some conditions on the func-
tion f (n). In the case where n = 1, there is no uncertainty, so that we have
1◦ ) f (1) = 0.
1

If n ≥ m, then p = n
,... , n1 has more uncertainty than q = 1
m
,... 1
,m . Hence,
2◦ ) f (n) ≥ f (m) if n ≥ m, i.e., f is nondecreasing.

If X, n1 , . . . , n1 and Y, m1
,... , m 1
are two independent schema, the com-
1 1
pound scheme is X × Y, nm , . . . , nm . In this case, the uncertainty of X × Y
should be equal to the sum of those of X and Y , i.e.,
3◦ ) f (nm) = f (n) + f (m).
Under these conditions we can characterize f as follows.
Proposition 3. Let f : N → R+ be a function satisfying conditions 1◦ ), 2◦ ) and

3◦ ) above. Then there exists some λ > 0 such that
f (n) = λ log n, n ∈ N.
Proof. This is well-known in functional equation theory. For the sake of completeness
we sketch the proof. By 3◦ ) we have f (n2 ) = 2f (n) and, in general,
f (nr ) = rf (n), n, r ∈ N, (1.6)
which can be verified by mathematical induction. Now let r, s, n ∈ N be such that

r, s ≥ 2. Choose m ∈ N so that
r m ≤ sn < r m+1 .
Then
m log r ≤ n log s < (m + 1) log r
and hence
m log s m 1
≤ < + . (1.7)
n log r n n
On the other hand, by 2◦ ) we get
f (r m ) ≤ f (sn ) < f (r m+1 )
and hence by (1.6)

mf (r) ≤ nf (s) ≤ (m + 1)f (r),
8 I. ENTROPY
so that
m f (s) m 1
≤ ≤ + . (1.8)
n f (r) n n
Thus (1.7) and (1.8) give
f (s) log s 2
− ≤ , n ≥ 1,
f (r) log r n
which implies that

f (s) f (r)
= .
log s log r
Since r, s ≥ 2 are arbitrary, it follows that for some constant λ > 0
f (n) = λ log n, n ≥ 2,
and by 1◦ ) the above equality is true for n = 1 too.
Consider a finite scheme (X, p) with p = (p1 , . . . , pn ) ∈ ∆n . If p(xj ) = pj =

1
n, then xj has log n = − log n1 as information or entropy, which is justified by
Proposition 3. This suggests that each xj has information of − log pj and H(X) =
Pn
− pj log pj is the average information that X = {x1 , . . . , xn } has, giving a good
j=1
reason to define the entropy of (X, p) by (1.1).
To characterize the Shannon entropy we consider two axioms.
The Shannon-Khinchin Axiom.

∞
(1◦ ) H : ∪ ∆n → R+ is continuous and, for every n ≥ 2,
n=2

1 1
H ,... , = max H(p) : p ∈ ∆n .
n n
(2◦ ) For every n ≥ 2 and (p1 , . . . , pn ) ∈ ∆n
H(p1 , . . . , pn , 0) = H(p1 , . . . , pn ).
m
Pj
(3◦ ) If p = (p1 , . . . , pn ) ∈ ∆n , pj = qjk , qjk ≥ 0 1 ≤ k ≤ mj , 1 ≤ j ≤ n,
k=1
H(q11 , . . . , q1m1 , . . . , qn1 , . . . , qnmn )

n
X
qj1 qjmj
= H(p1 , . . . , pn ) + pj H ,... , .
j=1
pj pj
The Faddeev Axiom.

[1◦ ] f (p) = H(p, 1 − p) : [0, 1] → R is continuous and f (p0 ) > 0 for some
p0 ∈ [0, 1].
[2◦ ] H(p1 , . . . , pn ) = H(pπ(1), . . . , pπ(n) ) for every (p1 , . . . , pn ) ∈ ∆n and per-
mutation π of {1, . . . , n}.
[3◦ ] If (p1 , . . . , pn ) ∈ ∆n and pn = q + r > 0 with q, r ≥ 0, then

q r
H(p1 , . . . , pn−1 , q, r) = H(p1 , . . . , pn ) + pn H , .
pn pn
The Faddeev Axiom is an improvement of the Shannon-Khinchin Axiom since [1◦ ]

and [3◦ ] are simpler than (1◦ ) and (3◦ ), and [2◦ ] is very natural. These two axioms are
equivalent and they imply the Shannon entropy within a positive constant multiple
as is seen in the following theorem.
Theorem 4. The following statements are equivalent to each other.

∞
(1) H(·) : ∪ ∆n → R+ satisfies the Shannon-Khinchin Axiom.
n=2
∞
(2) H(·) : ∪ ∆n → R+ satisfies the Faddeev Axiom.
n=2
(3) There is some λ > 0 such that
n
X
H(p1 , . . . , pn ) = −λ pj log pj , (p1 , . . . , pn ) ∈ ∆n , n ≥ 2. (1.9)
j=1
Proof. (1) ⇒ (2). Assume that (1) is true. [1◦ ] follows from (1◦ ).
`
[2◦ ] is derived as follows. If p1 , . . . , pn are positive rationals, then pj = mj for
some `1 , . . . , `n , m ∈ N. Hence

`1 `n
H(p1 , . . . , pn ) = H ,... ,
m m
X n
1 1 1 1 1 1
=H ,... , ,... , ,... , − pj H ,... , .
|m {z m} |m {z m} j=1
`j `j
`1 `n
Thus, for any permutation π of {1, . . . , n}, H(p1 , . . . , pn ) = H(pπ(1) , . . . , pπ(n) ).

The case where pj ’s are not necessarily rational follows from the continuity of H
((1◦ )) and the approximation by sequences of rational numbers.
[3◦ ]. It follows from (2◦ ), (3◦ ) and [2◦ ] that

1 1 1 1 1 1
H , =H , , 0, 0 = H , 0, , 0
2 2 2 2 2 2
10 I. ENTROPY

1 1 1 1
=H , + H(1, 0) + H(1, 0),
2 2 2 2
implying H(1, 0) = 0. Hence
H(p1 , . . . , pn ) = H(p1 , 0, . . . , pn−1 , 0, q, r)

n−1
X
q r
= H(p1 , . . . , pn ) + pj H(1, 0) + pn H ,
j=1
pn pn

q r
= H(p1 , . . . , pn ) + pn H , ,
pn pn
i.e., [3◦ ] holds.

(2) ⇒ (3). Assume that (2) is true. Using [3◦ ], we have for any p, q ≥ 0, r > 0
with p + q + r = 1

q r
H(p, q, r) = H(p, q + r) + (q + r)H ,
q+r q+r

p r
= H(q, p + r) + (p + r)H , .
p+r p+r
If we set f (p) = H(p, 1 − p), then the second of the above equalities becomes

q p
f (p) + (1 − p)f = f (q) + (1 − q)f . (1.10)
1−p 1−q
Letting p = 0 and 0 < q < 1, we get
f (0) = H(0, 1) = 0.
Integrating (1.10) with respect to q from 0 to 1 − p gives

Z 1 Z 1−p Z 1
2 2 f (t)
(1 − p)f (p) + (1 − p) f (t) dt = f (t) dt + p dt. (1.11)
0 0 p t3
Since f (p) is continuous and hence all terms except the first on the LHS (= left
hand side) of (1.11) are differentiable, we see that f (p) is also differentiable on
(0, 1). By differentiating (1.11) with respect to p we obtain
Z 1 Z 1
f (t) f (p)
(1 − p)f 0 (p) − f (p) − 2(1 − p) f (t) dt = −f (1 − p) + 2p 3
dt − .
0 p t p
We can simplify the above by using f (p) = f (1 − p) to get

Z 1 Z 1
f (t) f (p)
(1 − p)f 0 (p) = 2(1 − p) f (t) dt + 2p 3
dt − . (1.12)
0 p t p
It follows that f 0 (p) is also differentiable on (0, 1). By differentiating (1.12) we have
Z 1
2
f 00 (p) = − f (t) dt, 0<p<1 (1.13)
p(1 − p) 0
and integrating (1.13) twice gives
Z 1

f (p) = αp + β − 2 p log p + (1 − p) log(1 − p) f (t) dt, (1.14)
0
where α, β ∈ R are constants. Note that α = 0 since f (p) = f (1 − p) and that
(1.14) holds for 0 ≤ p ≤ 1. Thus β = 0 since f (0) = 0. Consequently, letting
R1
λ = 2 0 f (t) dt, it holds that
H(p1 , p2 ) = −λ(p1 log p1 + p2 log p2 ), (p1 , p2 ) ∈ ∆2 ,
proving (1.9) for the case n = 2.
For a general n ≥ 2 we prove (1.9) by mathematical induction. Suppose that
n
X
H(p1 , . . . , pn ) = −λ pj log pj , (p1 , . . . , pn ) ∈ ∆n
j=1
and take any (q1 , . . . , qn+1 ) ∈ ∆n+1 . We can assume that qn+1 > 0. Then observe
that
H(q1 , . . . , qn , qn+1 ) = H(q1 , . . . , qn−1 , qn + qn+1 )

qn qn+1
+ (qn + qn+1 )H ,
qn + qn+1 qn + qn+1
n−1
X
= −λ qj log qj − λ(qn + qn+1 ) log(qn + qn+1 )
j=1

qn qn+1
− λ qn log + qn+1 log
qn + qn+1 qn + qn+1
n+1
X
= −λ qj log qj .
j=1
(3) ⇒ (1). Assume that (3) is true. (1◦ ) is shown in Theorem 2 (3), (2◦ ) is clear
and (3◦ ) can be verified in a similar manner as in the proof of Theorem 2 (6).
12 I. ENTROPY
1.2. Conditional expectations
In order to study the Kolmogorov-Sinai entropy we need to work on conditional

expectations and conditional probabilities on a probability measure space. We collect
basic properties of these materials together with some proofs.
Thus, let (X, X, µ) be a probability measure space, where X is a nonempty
set, X is a σ-algebra of subsets of X and µ is a probability measure on it. A
transformation S on X into X is said to be measurable if S −1 X ⊆ X, i.e.,
S −1 A ∈ X for every A ∈ X. A measurable transformation S is said to be measure
preserving if µ ◦ S −1 = µ, i.e., µ ◦ S −1 (A) ≡ µ(S −1 A) = µ(A) for every A ∈ X.
If, in particular, a measure preserving transformation S is one-to-one and onto,
then the inverse transformation S −1 is considered. If S −1 is measurable, i.e., S is
invertible, then it satisfies S −1 X = X = SX and is also measure preserving. In
what follows we fix a quadruple (X, X, µ, S), which is called a dynamical system,
where S is measure preserving and not necessarily invertible.
We treat real-valued random variables on (X, X, µ) in this section. Denote the
L1 -space of (X, X, µ) by L1 (X), i.e., the set of all µ-integrable functions on X. If
Y is a σ-subalgebra of X and f ∈ L1 (X), let
Z
µf (A) = f dµ, A ∈ Y.
A
Then µf is a countably additive measure on Y and is absolutely continuous with

respect to µ, denoted µf µ, i.e., A ∈ X and µ(A) = 0 imply µf (A) = 0. Hence,
by the Radon-Nikodým Theorem there is a Y-measurable function g ∈ L1 (Y) such
that Z
µf (A) = g dµ, A ∈ Y.
A
The function g is unique in the µ-a.e. sense and is called the conditional expec-
tation of f relative to Y, denoted g = E(f |Y). If, in particular, f = 1A , the
indicator function of A ∈ X, then we denote E(1A |Y) = P (A|Y) and call it the
conditional probability of A relative to Y. Basic properties of conditional ex-
pectations and probabilities needed for our later work are given below. In the rest
of this section, we assume that Y is a σ-subalgebra of X, all the functions are in
L1 (X) and equalities are in the µ-a.e. sense.
(1) (Linearity) E(·|Y) : L1 (X) → L1 (Y) is a linear operator, i.e.,
E(αf + βg|Y) = αE(f |Y) + βE(g|Y), α, β ∈ R, f, g ∈ L1 (X).
In fact, the RHS is Y-measurable and the equality

Z Z

(αf + βg) dµ = αE(f |Y) + βE(g|Y) dµ
A A
1.2. CONDITIONAL EXPECTATIONS 13
holds for A ∈ Y.
(2) (Positivity) f ≥ 0 ⇒ E(f |Y) ≥ 0.
R R
In fact, 0 ≤ A f dµ = A E(f |Y) dµ for A ∈ Y and E(f |Y) is Y-measurable.

(3) E(f |Y) ≤ E |f | Y .
This follows from (2) and −|f | ≤ f ≤ |f |.
(4) (Boundedness) E(f |Y) 1
≤ kf k1 , k · k1 being the L1 -norm.
This is immediate from (3).

(5) (Idempotency) E E(f |Y) Y = E(f |Y).
For, observe that the RHS is already Y-measurable.
(6) E(·|Y) : L1 (X) → L1 (Y) is a projection of norm one.
This follows from (1), (4) and (5).
(7) If Y1 , Y2 are σ-subalgebras of X such that Y1 ⊆ Y2 , then

E E(f |Y2 ) Y1 = E(f |Y1 ).
In fact, the RHS is Y1 -measurable and

Z Z Z
f dµ = E(f |Y2) dµ = E(f |Y1 ) dµ, A ∈ Y1 .
A A A
R
(8) E(f |2) = E(f ) ≡ X f dµ, the expectation of f , where 2 = {∅, X}, the
smallest σ-subalgebra of X.
For, E(f |2) is a constant.

(9) E(f ) = E E(f |Y) .
This follows from (7).

(10) f ∈ L∞ (X), g ∈ L1 (Y) or f ∈ L1 (X), g ∈ L∞ (Y) implies E(gf |Y) =
gE(f |Y), where L∞ (X) or L∞ (Y) is the set of all µ-essentially bounded X- or
Y-measurable functions on X.
In fact, both sides are Y-measurable and the equality holds for a Y-simple func-
tion g. The general case follows from a suitable approximation by a sequence of
Y-simple functions.
(11) (Dominated Convergence Theorem) |fn | ≤ g, fn → f µ-a.e. ⇒
E(fn |Y) → E(f |Y) µ-a.e. and in L1 .
14 I. ENTROPY
For, observe that for every A ∈ Y

Z Z Z
E(fn |Y) dµ − E(f |Y) dµ ≤ E(fn |Y) − E(f |Y) dµ
A A A
Z

≤ E |fn − f | Y dµ, by (3),
ZA
= |fn − f | dµ
ZA
≤ |fn − f | dµ → 0
X
as n → ∞ by the ordinary Dominated Convergence Theorem. But then

Z Z Z Z
E(fn |Y) dµ = fn dµ → f dµ = E(f |Y) dµ
A A A A
for every A ∈ Y again by the ordinary Dominated Convergence Theorem. This

shows that E(fn |Y) → E(f |Y) µ-a.e. and in L1 .
(12) (Monotone Convergence Theorem) fn ↑ f µ-a.e. ⇒ E(fn |Y) ↑ E(f |Y)
µ-a.e.
For, we can assume without loss of generality that fn ≥ 0 µ-a.e. for n ≥ 1.
Note that lim E(fn |Y) exists and is Y-measurable. Thus for A ∈ Y we have by
n→∞
a suitable application of the ordinary Monotone Convergence Theorem twice
Z Z Z
lim E(fn |Y) dµ = lim E(fn |Y) dµ = lim fn dµ
A n→∞ n→∞ A n→∞ A
Z Z
= lim fn dµ = f dµ
n→∞
ZA A
= E(f |Y) dµ.

A
Therefore, the desired conclusion follows.

(13) (Jensen’s Inequality) If ϕ : R → R is a convex (= concave upward)
function bounded below and ϕ(f ) ∈ L1 (X) for an f ∈ L1 (X), then

ϕ E(f |Y) ≤ E ϕ(f ) Y .
In fact, note that, since ϕ is convex,

ϕ(t) = sup αt + β : α, β ∈ R are such that αs + β ≤ ϕ(s) for s ∈ R , t ∈ R.
1.2. CONDITIONAL EXPECTATIONS 15
Hence,
E ϕ(f ) Y ≥ E(αf + β|Y) = αE(f |Y) + β.
Taking the supremum with respect to α, β ∈ R on the RHS suitably, we get

E ϕ(f ) Y ≥ ϕ E(f |Y) .
(14) E(·|Y) : Lp (X) → Lp (Y) is a projection for p ≥ 1, where Lp (X) or Lp (Y)

is the set of all X- or Y-measurable functions f on X such that |f |p ∈ L1 (X).
For, since µ is a probability measure and hence Lp ⊆ L1 , E(·|Y) is defined on
all Lp for p ≥ 1. If p = ∞, then for f ∈ L∞ (X)

E(f |Y) ≤ E |f | Y ≤ kf k∞ ,
implying that kE(f |Y)k∞ ≤ kf k∞ , where k · k∞ is the µ-essential sup norm. If

1 < p < ∞, then it follows from (13) that
Z Z
p p
E(f |Y) p = E(f |Y) dµ ≤ E |f |p Y dµ
ZX X
p p
= |f | dµ = kf kp
X
p
since ϕ(t) = |t| is convex, and the desired assertion holds.
(15) E(·|Y) = PL2 (Y) , the orthogonal projection of L2 (X) onto L2 (Y).
In fact, for f ∈ L2 (X) and g ∈ L2 (Y) we have
Z Z
(f, g)2 = f g dµ = E(f g|Y) dµ
ZX X

= E(f |Y)g dµ = E(f |Y), g 2 ,
X
where (·, ·)2 is the inner product in L2 (X). This is enough to obtain the conclusion.
1 1
(16) (Hölder’s Inequality) If p + q = 1 and p, q ≥ 1, then
1 1
E |f g| Y ≤ E |f |p Y p E |g|q Y q , f ∈ Lp (X), g ∈ Lq (X).

For, we can assume 0 < E |f |p Y , E |g|q Y < ∞ µ-a.e. without loss of gen-
erality. If “p = 1 and q = ∞” or “p = ∞ and q = 1,” then the inequality is
immediate. Now suppose 1 < p, q < ∞. Then
|f g| 1 |f |p 1 |g|q
p1 q1 ≤ p E |f |p Y + q E |g|q Y
E |f |p Y E |g|q Y
Another random document with
no related content on Scribd:
Citizens of United States, not to be abridged on account of color, race, or previous
condition of servitude, (15th amendment), .. .. 20
Citizens of each State shall be entitled to the privileges and immunities of citizens in
the several States, 4 2 17
Claims, no prejudice to certain, 4 3 17
of the United States, or of the several States, not to be prejudiced by any
construction of the Constitution, 4 3 18
Coasting trade, regulations respecting, 1 9 15
Coin, Congress fix value of foreign, 1 8 15
Commerce, Congress to regulate, 1 8 15
regulations respecting, to be equal and uniform, 1 9 15
Commissions to be granted by the President, 2 3 17
Common law recognized and established, (7th amendment), .. .. 19
Congress vested with power, 1 1 13
may alter the regulations of State legislatures concerning elections of senators and
representatives, except as to place of choosing senators, 1 4 14
shall assemble once every year, 1 4 14
officers of government cannot be members of, 1 6 14
may provide for cases of removal, death, &c., of President and Vice-President, 2 1
16
may determine the time of choosing electors of President and Vice-President, 2 1
16
may invest the appointment of inferior officers in the President alone, in the
courts of law, or the heads of departments, 2 2 16
may establish courts inferior to the Supreme Court, 3 1 17
may declare the punishment of treason, 3 3 17
may prescribe the manner of proving the acts and records of each State, 4 1 17
to assent to the formation of new States, 4 3 18
may propose amendments to Constitution or call a convention, 5 1 18
to lay and collect duties, 1 8 15
to borrow money, 1 8 15
to regulate commerce, 1 8 15
to establish uniform laws of bankruptcy and naturalization, 1 8 15
to coin money, to regulate the value of coin, and fix a standard of weights and
measures, 1 8 15
to punish counterfeiting, 1 8 15
to constitute tribunals inferior to the Supreme Court, 1 8 15
to define and punish piracies, felonies on the high seas, and offenses against the
laws of nations, 1 8 15
to establish post offices and post roads, 1 8 15
to authorize patents to authors and inventors, 1 8 15
to declare war, grant letters of marque, and make rules concerning captures, 1 8
15
to raise and support armies, 1 8 15
to provide and maintain a navy, 1 8 15
to make rules for the government of the army and navy, 1 8 15
to call out the militia in certain cases, 1 8 15
to organize, arm, and discipline militia, 1 8 15
to exercise exclusive legislation over seat of government, 1 8 15
to pass laws necessary to carry the enumerated powers into effect, 1 8 15
to dispose of and make rules concerning the territory or other property of the
United States, 4 3 18
President may convene and adjourn in certain cases, 2 3 17
may enforce prohibition of slavery by appropriate legislation, (amendment), 13 2
20
Congress may, by a two-third’s vote, remove disability of persons who engaged in
rebellion, (14th amendment), 14 8 20
shall have power, by appropriate legislation, to enforce the provisions of Article
XIV, (14th amendment), 14 5 20
shall have power to enforce the provisions of Article XV, (15th amendment), 15 2
20
representation in, how apportioned, (14th amendment), .. 2 20
Constitution, how amended, 5 1 18
laws and treaties declared to be the supreme law, 6 1 18
rendered operative by the ratification of nine States, 7 1 18
Contracts, no law impairing, 1 10 16
Conventions for proposing amendments to the Constitution, 5 1 18
Counterfeiting, Congress to provide for punishment of, 1 8 15
Court, Supreme, its original and appellate jurisdiction, 3 2 17
Courts inferior to the Supreme Court may be ordained by Congress, 1 8 15
Ditto Ditto, 3 1 17
Crimes, persons accused of, fleeing from justice, may be demanded, 4 2 17
how to be tried, 3 2 17
Criminal prosecutions, proceedings in cases of, .. .. 19
Debts against the confederation to be valid, 6 1 18

Debt, public, authorized by law, shall not be questioned, (14th amendment), .. 4 20
incurred in aid of rebellion not to be assumed or paid, (14th amendment), .. 4 20
Disability of persons who engaged in rebellion (14th amendment), .. 3 20
Duties to be laid by Congress, and to be uniform, 1 8 15
further provision respecting, 1 9 15
cannot be laid by the States, 1 10 16
on exports prohibited, 1 9 15
on imports and exports imposed by States shall inure to the treasury of the United
States, 1 10 16
Elections of Senators and representative shall be prescribed by the States, 1 4 14
qualifications and returns of members of Congress to be determined by each
house, 1 5 14
Electors of President and Vice-President, how chosen, and their duties, 2 1 16
altered (see 12th amendment), .. .. 19
to vote the same day throughout the United States, 2 1 16
no senator or representative, or public officer, shall serve as, 2 1 16
Enumeration every ten years, 1 2 13
Executive power vested in the President, (see President), 2 1 16
Exports not to be taxed, 1 9 15
and imports, States prohibited from laying duties on, 1 10 16
Ex post facto law, none shall be passed, 1 9 15
prohibited to States, 1 10 16
Fines, excessive prohibited, .. .. 19

Fugitives from justice to be delivered up, 4 2 17
from service may be reclaimed, 4 2 17
Habeas corpus, writ of, can be suspended in cases of rebellion or invasion, 1 9 15

House of Representatives. (See Representatives.)
Impeachment to be brought by House of Representatives, 1 2 13

tried by the Senate, 1 3 14
Impeachment, judgment on, 1 3 14
all civil officers liable to, 2 4 17
Importation of slaves, not prohibited till 1808, 1 9 15
Judges shall hold their office during good behavior, 3 1 17

their compensation, 3 1 17
Judiciary—tribunals inferior to Supreme Court may be created, 1 8 15
Judicial power vested in a Supreme Court and courts inferior, 3 1 17
powers of the judiciary, 3 2 17
restriction as to suits against a State, .. .. 19
Judicial proceedings of each State are entitled to faith and credit in every State, 4 1
17
Jury trial secured, and shall be held in the State where the crime shall have been
committed, 3 2 17
further regulated, (6th amendment), .. .. 19
secured in suits at common law where the value of controversy shall exceed
twenty dollars, (7th amendment), .. .. 19
Law, what is declared the supreme, 6 1 18

common, recognized and established, (7th amendment), .. ..
Laws, President to see them faithfully executed, 2 3 17
Legislative powers vested in Congress. (See Congress.)
Loans, authority to make, 1 8 15
Marque and reprisal, letters of, 1 8 15

Militia to be called out, 1 8 15
to be officered by the States, 1 8 15
to be commanded by the President, 2 2 19
their right to keep and bear arms secured, (2d amendment), .. .. 19
Money shall be drawn from the treasury only by appropriation laws, 1 9 15
Congress to coin and regulate value of, 1 8 15
States cannot make, 1 10 16
Naturalization, uniform rules of, 1 8 15

Navy, Congress to provide and govern, 1 8 15
Nobility, titles of, shall not be granted by the United States, 1 9 15
nor by the States, 1 10 12
Oath of the President, 2 1 16

of the public officers, 6 1 18
Office, who prohibited from holding, (14th amendment), .. 3 20
Officers of the House of Representatives shall be chosen by the House, 1 2 13
Officers of the Senate shall be chosen by the Senate, 1 3 14
civil, may be removed by impeachment, 2 4 17
Order of one house requiring the concurrence of the other, 1 7 14
Pardons, President may grant, 2 2 16

Patents to be granted to inventors, 1 8 15
Petition, right of, .. .. 19
Persons held to service or labor, their importation or migration into the United
States may be prohibited after 1808, 1 9 15
escaping from one State to another shall be delivered up to those entitled to
service, 4 2 17
Piracy, Congress to prescribe punishment for, 1 8 15
Post offices and post roads, establishment of, 1 8 15
Powers not delegated to Congress nor prohibited to the States are reserved, (10th
amendment), .. .. 19
legislative (See Congress.)
executive (See President.)
judicial (See Judicial.)
Presents from foreign powers to public officers prohibited, 1 9 15
Press, freedom of
President of the U. S. vested with the executive power, 2 1 16
shall be chosen for four years, 2 1 16
how elected, 2 1 16
same, (12th amendment), .. .. 19
qualifications for, 2 1 16
who shall act in case of vacancy, 2 1 16
compensation of, 2 1 16
shall take an oath of office, 2 1 16
may be removed by impeachment, 2 4 17
President, commander of army, navy, and militia, 2 2 16
may require the written opinion of the heads of departments, 2 2 16
may reprieve and pardon, 2 2 16
may make treaties with consent of the Senate, 2 2 16
may appoint to office with consent of the Senate, 2 2 16
shall fill up vacancies happening during the recess of the Senate, 2 2 16
shall give information to Congress and recommend measures, 2 3 17
may convene both houses or either house, 2 3 17
may adjourn them in case of disagreement, 2 3 17
shall receive ambassadors and public ministers, 2 3 17
shall take care that the laws be faithfully executed, 2 3 17
shall commission all officers, 2 3 17
Privileges and immunities of members of Congress, 1 6 14
of citizens (See Citizens, also Rights.)
Property, Congress to provide for care of public, 4 3 18
shall not be taken for public use without just compensation, (5th amendment), .. ..
Punishments, cruel and unusual, prohibited, .. ..
Quorum for business, what shall be a, 1 5 14

of States in choosing a President by the House of Representatives, 2 1 16
Quartered, no soldier to be quartered on a citizen, .. .. 19
Rebellion, debt incurred in aid of, not to be assumed or paid, (14th amendment), .. 4
20
disability of persons who have engaged in (14th amendment), .. 3 20
Receipts and expenditures, accounts of, to be published, 1 9 15
Records, how to be authenticated, 4 1 17
Religion—no law to be made—free exercise of, .. .. 19
religious test not required, 6 .. 18
Reprieves granted by the President, 2 2
Representatives, House of, composed of members chosen every second year, 1 2 13
qualifications of voters, 1 2 13
qualifications of members, 1 2 13
apportionment of, 1 2 13
vacancies, how supplied, 1 2 13
shall choose their officers, 1 2 13
shall have the power of impeachment, 1 2 13
Representation shall be the judge of the election and qualifications of its members, 1
5 14
what shall be a quorum, 1 5 14
any number may adjourn and compel the attendance of absentees, 1 5 14
may determine the rules of proceeding, 1 5 14
may punish or expel a member, 1 5 14
shall keep a journal and publish the same, 1 5 14
shall not adjourn for more than three days nor to any other place, without the
consent of the Senate, 1 5 14
one-fifth may require the yeas and nays, 1 5 14
shall originate bills for raising revenue, 1 7 14
compensation to be ascertained by law, 1 6 14
privileged from arrest, except in certain cases, 1 6 14
Representatives shall not be questioned for speech or debate in the House, 1 6 14
shall not be appointed to office, 1 6 14
shall not serve as electors of President, 2 1 16
and direct taxes apportioned according to numbers, 1 2 13
how apportioned among the several States, (14th amendment), .. 2 20
who prohibited from being, (14th amendment), .. 3 20
of a State, vacancies in, supplied until a new election by executive authority, 1 2 13
Resolution, order, or vote, requiring the concurrence of both houses, to undergo the
formalities of bills, 1 7 14
Revenue bills to originate in the House of Representatives, 1 7 14
Rights of the citizen declared to be—
privileges of citizens of the several States, 4 2 17
liberty of conscience in matters of religion, .. .. 19
freedom of speech and of the press, .. .. 19
to assemble and petition, .. .. 19
to keep and bear arms, .. .. 19
to be exempt from the quartering of soldiers, .. .. 19
to be secure from unreasonable searches and seizures, .. .. 19
to be free from answering for a crime, unless on presentment or indictment of a
jury, .. .. 19
not to be twice jeoparded for the same offence, .. .. 19
not to be compelled to be a witness against himself, .. .. 19
not to be deprived of life, liberty, or property without due course of law, .. .. 19
private property not to be taken for public use, .. .. 19
in criminal prosecutions, shall enjoy the right of a speedy trial by jury, with all the
means necessary for his defence, .. .. 19
in civil cases trial to be by jury, and shall only be re-examined according to
common law, .. .. 19
excessive bail shall not be required, excessive fines imposed, no cruel nor unusual
punishment inflicted, .. .. 19
enumeration of certain rights shall not operate against retained rights, .. .. 19
Rules, each house shall determine its own, 1 5 14
Seat of government, exclusive legislation, 1 8 15

Searches and seizures, security against, .. .. 19
Senate, composed of two senators from each State, 1 3 14
how chosen, classed, and terms of service, 1 3 14
Senate, qualifications of senators, 1 3 14
Vice-President to be President of the, 1 3 14
shall choose their officers, 1 3 14
shall be the judge of the elections and qualifications of its members, 1 5 14
what number shall be a quorum, 1 5 14
any number may adjourn, and compel attendance of absentees, 1 5 14
may determine its rules, 1 5 14
may punish or expel a member, 1 5 14
shall keep a journal, and publish the same, except parts requiring secrecy, 1 5 14
shall not adjourn for more than three days, nor to any other place, without the
consent of the other house, 1 5 14
one-fifth may require the yeas and nays, 1 5 14
may propose amendments to bills for raising revenue, 1 7 14
shall try impeachments, 1 3 14
effect of their judgment on impeachment, 1 3 14
compensation to be ascertained by law, 1 6 14
privileged from arrest, 1 6 14
not questioned for any speech or debate, 1 6 14
shall not be appointed to office, 1 6 14
Senator, shall not be elector, 2 1 16
who prohibited from being, (14th amendment), .. 3 20
Senators and representatives, elections of, how prescribed, 1 4 14
Slaves, their importation may be prohibited after 1808, 1 9 15
escaping from one State to another may be reclaimed, 4 2 17
claims for the loss or emancipation of, to be held illegal and void, (14th
amendment), .. 4 20
Slavery, except as a punishment for crime, prohibited, amendment, 13 1 20
Congress authorized to enforce the prohibition of, (amendment), 13 2 20
Soldiers not quartered on citizens, .. .. 19
Speaker, how chosen, 1 2 13
Speech, freedom of, .. .. 19
States prohibited from—
entering into treaty, alliance, or confederation, 1 10 16
granting letters of marque, 1 10 16
coining money, 1 10 16
emitting bills of credit, 1 10 16
making anything a tender but gold and silver coin, 1 10 16
prohibited from—
passing bills of attainder, ex post facto laws, or laws impairing contracts, 1 10 16
granting titles of nobility, 1 10 16
laying duties on imports and exports, 1 10 16
laying duties on tonnage, 1 10 16
keeping troops or ships of war in time of peace, 1 10 16
entering into any agreement or contract with another State or foreign power, 1
10 16
engaging in war, 1 10 16
abridging right of United States citizens of, to vote on account of race or color,
(15th amendment), .. 1 20
States, new, may be admitted into the Union, 4 3 18
may be admitted within the jurisdiction of others, or by the junction of two or
more, with the consent of Congress and the legislatures concerned, 4 3 18
State judges bound to consider treaties, the Constitution, and the laws under it, as
supreme, 6 .. 18
State, every, guarantied a republican form of government, protected by United
States, 4 4 18
Supreme Court. (See Court and Judiciary.)
Suits at common law, proceedings in, .. .. 19
Tax, direct, according to representation, 1 2 13

shall be laid only in proportion to census, 1 9 15
Tax on exports prohibited, 1 9 15
Tender, what shall be a legal, 1 10 16
Territory or public property, Congress may make rules concerning, 4 3 18
Test, religious, shall not be required, 6 .. 18
Titles. (See Nobility.)
Title from foreign state prohibited, 1 9 15
Treason, defined, 3 3 17
two witnesses, or confession, necessary for conviction, 3 3 17
punishment of, may be prescribed by Congress, 3 3 17
Treasury, money drawn from, only by appropriation, 1 9 15
Treaties, how made, 2 2 16
the supreme law, 6 .. 18
States cannot make, 1 10 16
Vacancies happening during the recess may be filled temporarily by the President, 2
2 16
in representation in Congress, how filled, 1 2 13
Veto of the President, effect of, and proceedings on, 1 7 14
Vice-President of the U. S. to be President of the Senate, 1 3 14
how elected, 2 1 16
amendment, .. .. 19
shall, in certain cases, discharge the duties of President, 2 1 16
may be removed by impeachment, 2 4 17
Vote of one house requiring the concurrence of the other, 1 7 14
right of citizens to, not to be abridged on account of race or color, (15th
amendment), .. 1 20
War, Congress to declare, 1 8 15

Warrants for searches and seizures, when and how they shall issue (14th
Witness, in criminal cases, no one compelled to be against himself (5th
Weights and Measures, standard of, 1 8 15
Yeas and nays entered on journal, 1 6 14

BOOK V.
TABULATED HISTORY OF POLITICS.
Aggregate Issues of Paper Money in War Times, 5004
Ante-war Debts, 5015
Cabinet Officers of the Administrations, 5013

Chronological Politics, 1765–1892, 5025
Civil Officers, 5018
Customs Tariff of Great Britain, 5010
Electoral Votes for President and Vice-President, 5005

Electoral Votes; Number to which each State has been Entitled, 1789–1892, 5016
Gold; Highest and Lowest Prices of, 5024
Interest Laws of all the States and Territories of the United States, 5004
Length of Sessions of Congress, 1779–1881, 5018
National Commerce, per capita, 5023

National Debt, per capita, 5023
National Expenditures, per capita, 5023
Popular and Electoral Votes in Presidential Elections, 1789–1889, 5011

Presidents and Vice-Presidents,, 5010
President and Vice-President, Candidates for, 5016
Rebellion, Expenditures caused by, 5021
Signers of Declaration of Independence, 5015

Speakers of House of Representatives, 5020
States, when admitted, 5019
Supreme Courts, 5017
Territories, when Organized, 5019

Troops furnished by each State, 1861–65, 5020
Troops, number of called into service during the Rebellion, 5017
Value of United States Money in Foreign Gold and Silver Coin, 5003
1. Edwin Williams in Statesman’s Manual.

2. From the Statesman’s Manual, Vol. 1., by Edwin Williams.
3. Note.—See Constitution, Art. 3, Sec. 4, p. 5.
4. In the Ritual the words in parentheses are omitted. In the key to the Ritual, they are written in
figures—the alphabet used being the same as printed below. So throughout.
Key to Unlock Communications.
A B C D E F G H I J K L M
1 7 13 19 25 2 8 14 20 26 3 9 15
N O P Q R S T U V W X Y Z
21 4 10 16 22 5 11 17 23 6 12 18 24
5. Concerning what is said of cities, the key to the Ritual says: “Considered unnecessary to
decipher what is said in regard to cities.”
6. President Buchanan’s Inaugural Address.
7. From Mr. Buchanan’s Administration on the eve of the Rebellion, published by D. Appleton &
Co., 1866.
8. The Provisional Constitution adopted by the Seceded States differs from the Constitution of
the United States in several important particulars. The alterations and additions are as follows:
ALTERATIONS.
1st. The Provisional Constitution differs from the other in this: That the legislative powers of the
Provisional Government are vested in the Congress now assembled, and this body exercises all the
functions that are exercised by either or both branches of the United States Government.
2d. The Provisional President holds his office for one year, unless sooner superseded by the
establishment of a permanent Government.
3d. Each State is erected into a distinct judicial district, the judge having all the powers
heretofore vested in the district and circuit courts; and the several district judges together compose
the supreme bench—a majority of them constituting a quorum.
4th. Whenever the word “Union” occurs in the United States Constitution the word
“Confederacy” is substituted.
THE FOLLOWING ARE THE ADDITIONS.

1st. The President may veto any separate appropriation without vetoing the whole bill in which it
is contained.
2d. The African slave-trade is prohibited.
3d. Congress is empowered to prohibit the introduction of slaves from any State not a member of
this Confederacy.
4th. All appropriations must be upon the demand of the President or heads of departments.
OMISSIONS.
1st. There is no prohibition on members of Congress holding other offices of honor and
emolument under the Provisional Government.
2d. There is no provision for a neutral spot for the location of a seat of government, or for sites
for forts, arsenals, and dock-yards; consequently there is no reference made to the territorial powers
of the Provisional Government.
3d. The section in the old Constitution in reference to capitation and other direct tax is omitted;
also, the section providing that no tax or duty shall be laid on any exports.
4th. The prohibition on States keeping troops or ships of war in time of peace is omitted.
5th. The Constitution being provisional merely, no provision is made for its ratification.
AMENDMENTS.
1st. The fugitive slave clause of the old Constitution is so amended as to contain the word “slave,”
and to provide for full compensation in cases of abduction or forcible rescue on the part of the State
in which such abduction or rescue may take place.
2d. Congress, by a vote of two-thirds, may at any time alter or amend the Constitution.
TEMPORARY PROVISIONS.
1st. The Provisional Government is required to take immediate steps for the settlement of all
matters between the States forming it and their other late confederates of the United Slates in
relation to the public property and the public debt.
2d. Montgomery is made the temporary seat of government.
3d. This Constitution is to continue one year, unless altered by a two-thirds vote or superseded
by a permanent Government.
9. From Lalor’s Encyclopædia of Political Science, published by Rand & McNally. Chicago, Ill.
10. Official Journal of the Convention, pp. 9 and 10.
11. The text of Webster’s speech in reply to Hayne, now accepted as the greatest constitutional
exposition ever made by any American orator, will be found in our book devoted to Great Speeches
on Great Issues.
12. These were afterwards seized.
13. The attempted removal of these heavy guns from Allegheny Arsenal, late in December, 1860,
created intense excitement. A monster mass meeting assembled at the call of the Mayor of the city,
and citizens of all parties aided in the effort to prevent the shipment. Through the interposition of
Hon. J. K. Moorhead, Hon. R. McKnight, Judge Shaler, Judge Wilkins, Judge Shannon, and others
inquiry was instituted, and a revocation of the order obtained. The Secessionists in Congress bitterly
complained of the “mob law” which thus interfered with the routine of governmental affairs.—
McPherson’s History.
14. Resigned January 17th, 1861, and succeeded by Hon. Lot M. Morrill.
15. Did not attend.
16. Resigned and succeeded January 2d, 1861, by Hon. Stephen Coburn.
17. From the “History of Abraham Lincoln and the Overthrow of Slavery,” by Hon. Isaac N.
Arnold.
18. 1864, February 15—Repealed the above act, but provided for continuing organizations of
partisan rangers acting as regular cavalry and so to continue; and authorizing the Secretary of War to
provide for uniting all bands of partisan rangers with other organizations and bringing them under
the general discipline of the provisional army.
19. See memorandum at end of list.
20. This incident was related to the writer by Col. A. K. McClure of Philadelphia, who was in
Lancaster at the time.
21. Arnold’s “History of Abraham Lincoln.”
22. On the 23d of July, 1861, the Attorney-General, in answer to a letter from the United States
Marshal of Kansas, inquiring whether he should assist in the execution of the fugitive slave law,
wrote:
Attorney-General’s Office, July 23, 1861.
J. L. McDowell, U. S. Marshal, Kansas:
Your letter, of the 11th of July, received 19th, (under frank of Senator Lane, of Kansas,) asks
advice whether you should give your official services in the execution of the fugitive slave law.
It is the President’s constitutional duty to “take care that the laws be faithfully executed.” That
means all the laws. He has no right to discriminate, no right to execute the laws he likes, and leave
unexecuted those he dislikes. And of course you and I, his subordinates, can have no wider latitude of
discretion than he has. Missouri is a State in the Union. The insurrectionary disorders in Missouri are
but individual crimes, and do not change the legal status of the State, nor change its rights and
obligations as a member of the Union.
A refusal by a ministerial officer to execute any law which properly belongs to his office, is an
official misdemeanor, of which I have no doubt the President would take notice. Very respectfully
EDWARD BATES.
23. Republicans in Roman; Democrats in italics.

24. Democrats in italics.
25. Republicans in roman; Democrats in italics.
26. In 1860 a vote was had in the State of New York on a proposition to permit negro suffrage
without a property qualification. The result of the city was—yeas 1,640. nays 37,471. In the State—
yeas 197,505, nays 337,984. In 1864 a like proposition was defeated—yeas 85,406, nays 224,336.
In 1862, in August, a vote was had in the State of Illinois, on several propositions relating to
negroes and mulattoes, with this result:
For excluding them from the State 171,893
Against 71,306
100,587
Against granting them suffrage or right to office 21,920
For 35,649
176,271
For the enactment of laws to prohibit them from going to, or voting in, the
State 198,938
Against 44,414
154,524
—From McPherson’s History of the Great Rebellion.
27. December 23, 1862—Jefferson Davis issued a proclamation of outlawry against Major
General B. F. Butler, the last two clauses of which are:
Third. That all negro slaves captured in arms be at once delivered over to the executive
authorities of the respective States to which they belong, to be dealt with according to the laws of said
States.
Fourth. That the like orders be executed in all cases with respect to all commissioned officers of
the United States when found serving in company with said slaves in insurrection against the
authorities of the different States of this Confederacy.
28. McPherson’s History, page 317.
29. This act is in those words:
Be it enacted, &c., That hereafter every person elected or appointed to any office of honor or
profit under the government of the United States, either in the civil, military, or naval departments of
the public service, excepting the President of the United States, shall, before entering upon the duties
of such office, and before being entitled to any of the salary or other emoluments thereof, take and
subscribe the following oath or affirmation: “I, A B, do solemnly swear (or affirm) that I have never
voluntarily borne arms against the United States since I have been a citizen thereof; that I have
voluntarily given no aid, countenance, counsel, or encouragement to persons engaged in armed
hostility thereto; that I have never sought nor accepted nor attempted to exercise the functions of any
office whatever, under any authority or pretended authority, in hostility to the United States; that I
have not yielded a voluntary support to any pretended government, authority, power, or constitution
within the United States, hostile or inimical thereto; and I do further swear (or affirm) that, to the
best of my knowledge and ability, I will support and defend the Constitution of the United States
against all enemies, foreign and domestic; that I will bear true faith and allegiance to the same; that I
take this obligation freely, without any mental reservation or purpose of evasion, and that I will well
and faithfully discharge the duties of the office on which I am about to enter; so help me God;” which
said oath, so taken and signed, shall be preserved among the files of the Court, House of Congress, or
Department to which the said office may appertain. And any person who shall falsely take the said
oath shall be guilty of perjury, and on conviction, in addition to the penalties now prescribed for that
offense, shall be deprived of his office, and rendered incapable forever after, of holding any office or
place under the United States.
30. Compiled by Hon. Edward McPherson in his Hand-Book of Politics for 1868.
31. Unofficial.
32. From Greeley’s Recollections of a Busy Life, page 413.
33. From the Century of Independence by John Sully, Boston.
34. The following is a correct table of the ballots in the New York Democratic Convention:
Candidates. 1. 2. 3. 4. 5. 6. 7. 8. 9. 10. 11.
Horatio
Seymour 9
George H.
Pendleton 105 104 119½ 118½ 122 122½ 137½ 156½ 144 147½ 144½
Andrew
Johnson 65 52 34½ 32 24 21 12½ 6 5½ 6 5½
Winfield S.
Hancock 33½ 40½ 45½ 43½ 46 47 42½ 28 34½ 34 33½
Sanford E.
Church 33 33 33 33 33 33 33
Asa Packer 26 26 26 26 27 27 26 26 26½ 27½ 26
Joel Parker 13 15½ 13 13 13 13 7 7 7 7 7
James E.
English 16 12½ 7½ 7½ 7 6 6 6 6
James R.
Doolittle 13 1½ 12 12 15 12 12 12 12 12 12½
Reverdy
Johnson 8½ 8 11 8 9½
Thomas A.
Hendricks 2½ 2 9½ 11½ 19½ 30 39½ 75 80½ 82½ 88
F. P. Blair,
Jr. ½ 10½ 4½ 2 5 ½ ½ ½ ½ ½
Thomas
Ewing ½ 1 1
J. Q. Adams 1
George B.
McClellan
Salmon P.
Chase
Franklin
Pierce
John T.
Hoffman
Stephen J.
Field
Thomas H.
Seymour
Candidates. 12. 13. 14. 15. 16. 17. 18. 19. 20. 21. 22.
Horatio
Seymour 317
George H.
Pendleton 145½ 134½ 130 129½ 107½ 70½ 56½
Andrew
Johnson 4½ 4½ 5½ 5½ 6 10 5
Winfield S.
Hancock 30 48½ 56 79½ 113½ 137½ 144½ 135½ 142½ 135½
Sanford E.
Church
Asa Packer 26 26 26 22
Joel Parker 7 7 7 7 7 7 3½
James E.
English 6 16 19
James R.
Doolittle 12½ 13 13 12 12 12 12 12 12 12
Reverdy
Johnson
Thomas A.
Hendricks 89 81 84½ 82½ 70½ 80 87 107½ 121 132
F. P. Blair,
Jr. ½ ½ 13½ 13
Thomas
Ewing
J. Q. Adams
George B.
McClellan 1 ½
Salmon P.
Chase ½ ½ ½ ½ ½ 4
Franklin
Pierce 1
John T.
Hoffman 3 3
Stephen J.
Field 15 9 8
Thomas H.
Seymour 4 2
Necessary to choice 212
35. General Blair was nominated unanimously on the first ballot.
36. One Democratic elector was defeated, being cut by over 500 voters on a
local issue.
37. Commonly called “Greenbacks,” or “Legal Tender notes.”
38. Commonly called “National Bank notes.”
39. By Rand & McNally, Chicago, Ill., 1882.
40. This was partially done by the Legislature of Pennsylvania in 1881.
41. Act of March 2, 1850.
42. Act of January 19, 1866.
43. Senate doc. 181, 46th Congress.
44. Sec. 2, 258, Rev. Stat. U.S.
45. According to the affidavits of Samuel Howard and others, page 14.
46. See Report of Attorney-General United States, 1880–81.
47. Act of March 6, 1862.
48. Act of February 16, 1872.
49. Secs. 1 and 2, act of February 3, 1852.
50. See act of January 17, 1862.
51. Act of January 7, 1854, sec. 14.
52. Acts of Jan 21, 1853, and of January, 1855, sec. 29.
55. Act of January 14, 1854.
56. Sec. 106. Act March 6, 1852.
57. Enormous sums are, however, given to soldiers who were wounded during
the war, or who pretend that they were—for jobbery on an unheard of scale is
practised in connection with these pensions. It is estimated that $120,000,000
(24,000,000l.) will have to be paid during the present fiscal year, for arrears of
pension, and the number of claimants is constantly increasing, [The writer
evidently got these “facts” from sensational sources.]—Am. Pol.
58. The undeniable facts of the case were as we have briefly indicated above,
See, for example, a letter to the ‘New York Nation,’ Nov. 3, 1881.
59. Speech In New York, March 7, 1881.
60. ‘New York Tribune,’ Feb. 25, 1870.
61. Letter in New York papers, Feb. 20, 1875.
62. Mr. George William Curtis, in ‘Harper’s Magazine,’ 1870.
63. Article I. sect. vi. 2.
64. ‘Commentaries,’ I. book iii. sect. 869.
65. [These are mere traditions tinged with the spirit of some of the assaults
made in the “good old days” even against so illustrious a man as Washington.—Am.
Pol.]
66. Mr. Watterson, formerly a distinguished member of Congress, is the
author of the “tariff for revenue only” plank in the Democratic National Platform of
1880, and is now, as he has been for years, the chief editor of the Louisville Courier
Journal.
67. American, 707; scattering 989.
68. In Connecticut, the vote for Sheriff is taken. In New York, the average vote
on four of the five State officers chosen, excluding Secretary of State. In Nebraska,
Democratic and Anti-Monopoly vote combined on Judge.
69. Scattering, 106.
70. In these States the vote on Lieutenant-Governor was taken, as being from
special causes, a fairer test of party strength. In the others the principal State
officer was taken. Where State officers were not elected, the Congressional vote
was taken. In Georgia, Congressmen-at-Large was taken.
71. The vote for Chief Judge.
72. The Regular and Independent Republican vote is combined.
73. Vote of the two Democratic candidates is combined.
74. One vote lacking in each.
75. One vote lacking.
76. One vote lacking.
77. 3 votes lacking.
78. Upon the resolution of Mr. Wythe, which proposed, “That the committee
should ratify the constitution, and that whatsoever amendments might be deemed
necessary should be recommended to the consideration of the congress, which
should first assemble under the constitution, to be acted upon according to the
mode prescribed therein.”
79. In answer to an address of Governor Eustis, denouncing the conduct of the
peace party during the war, the House of Representatives of Massachusetts, in
June, 1823, say, “The change of the political sentiments evinced in the late
elections forms indeed a new era in the history of our commonwealth. It is the
triumph of reason over passion; of patriotism over party spirit. Massachusetts has
returned to her first love, and is no longer a stranger in the Union. We rejoice that
though, during the last war, such measures were adopted in this state as
occasioned double sacrifice of treasure and of life, covered the friends of the nation
with humiliation and mourning, and fixed a stain on the page of our history, a
redeeming spirit has at length arisen to take away our reproach, and restore to us
our good name, our rank among our sister states, and our just influence in the
Union.
“Though we would not renew contentions, or irritate wantonly, we believe that
there are cases when it is necessary we should ‘wound to heal.’ And we consider it
among the first duties of the friends of our national government, on this return of
power, to disavow the unwarrantable course pursued by this state, during the late
war, and to hold up the measures of that period as beacons; that the present and
succeeding generations may shun that career which must inevitably terminate in
the destruction of the individual or party who pursues it; and may learn the
important lesson, that, in all times, the path of duty is the path of safety; and that it
is never dangerous to rally around the standard of our country.”
80. 2d Dodson’s Admiralty Reports, 48. 13th Mass. Reports, 26.
81. It appears at p. 6 of the “Account” that by a vote of the House of
Representatives of Massachusetts, (260 to 290) delegates to this convention were
ordered to be appointed to consult upon the subject “of their public grievances and
concerns,” and upon “the best means of preserving their resources,” and for
procuring a revision of the constitution of the United States, “more effectually to
secure the support and attachment of all the people, by placing all upon the basis
of fair representation.”
The convention assembled at Hartford on the 15th December, 1814. On the
next day it was
Resolved, That the most inviolable secrecy shall be observed by each member
of this convention, including the secretary, as to all propositions, debates, and
proceedings thereof, until this injunction shall be suspended or altered.
On the 24th of December, the committee appointed to prepare and report a
general project of such measures as may be proper for the convention to adopt,
reported among other things,—
“1. That it was expedient to recommend to the legislatures of the states the
adoption of the most effectual and decisive measures to protect the militia of the
states from the usurpations contained in these proceedings.” [The proceedings of
Congress and the executive, in relation to the militia and the war.]
“2. That it was expedient also to prepare a statement, exhibiting the necessity
which the improvidence and inability of the general government have imposed
upon the states of providing for their own defence, and the impossibility of their
discharging this duty, and at the same time fulfilling the requisitions of the general
government, and also to recommend to the legislatures of the several states to
make provision for mutual defence, and to make an earnest application to the
government of the United States, with a view to some arrangement whereby the
state may be enabled to retain a portion of the taxes levied by Congress, for the
purpose of self-defence, and for the reimbursement of expenses already incurred
on account of the United States.
“3. That it is expedient to recommend to the several state legislatures certain
amendments to the constitution, viz.,—
“That the power to declare or make war, by the Congress of the United States,
be restricted.
“That it is expedient to attempt to make provision for restraining Congress in
the exercise of an unlimited power to make new states, and admit them into the
Union.
“That an amendment be proposed respecting slave representation and slave
taxation.”
On the 29th of December, 1814, it was proposed “that the capacity of
naturalized citizens to hold offices of trust, honor, or profit ought to be restrained,”
&c.
The subsequent proceedings are not given at large. But it seems that the report
of the committee was adopted, and also a recommendation of certain measures (of
the character of which we are not informed) to the states for their mutual defence;
and having voted that the injunction of secrecy, in regard to all the debates and
proceedings of the convention, (except so far as relates to the report finally
adopted,) be continued, the convention adjourned sine die, but as was supposed, to
meet again when circumstances should require it.
82. I refer to the authority of Chief Justice Marshall in the case of Jonathan
Robbins. I have not been able to refer to the speech, and speak from memory.
83. In this extended abstracts are given and data references omitted not
applicable to these times.
84. Speech at the Tabernacle, New York, February 10, 1843, in public debate
on this resolution:—
Resolved, That a Protective Tariff is conducive to our National Prosperity.
Affirmative: Joseph Blunt,

Horace Greeley.
Negative: Samuel J. Tilden,

Parke Godwin.
From Greeley’s “Recollections of a Busy Life.”

85. All the series were published in 1860 by Follet, Foster & Co., Columbus,
Ohio.

Textbook Abstract Methods in Information Theory 2Nd Edition Kakihara Ebook All Chapter PDF

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Textbook Abstract Methods in Information Theory 2Nd Edition Kakihara Ebook All Chapter PDF

Uploaded by

Copyright:

Available Formats

Abstract Methods in Information Theory

2nd Edition Kakihara

Quantum Information Theory 2nd Edition Mark M. Wilde

Financial Markets Theory: Equilibrium, Efficiency and

Functional Numerical Methods: Applications to Abstract

Abstract Algebra Applications to Galois Theory

Foundations of Geophysical Electromagnetic Theory and

Electronic Structure Basic Theory and Practical Methods

A course in abstract algebra Nicholas Jackson

Problems in Abstract Algebra 1st Edition Adrian R.

9954.9789814759236.v2-tp.indd 1 19/5/16 10:32 AM

Editor: M M Rao ISSN: 1793-1169

Vol. 1: Martingales and Stochastic Analysis

LaiFun - Abstract Methods in Information Theory.indd 1 10/5/2016 9:06:33 AM

9954.9789814759236.v2-tp.indd 2 19/5/16 10:32 AM

Library of Congress Cataloging-in-Publication Data

British Library Cataloguing-in-Publication Data

Copyright © 2016 by World Scientific Publishing Co. Pte. Ltd.

LaiFun - Abstract Methods in Information Theory.indd 2 10/5/2016 9:06:33 AM

Dedicated to my late advisor

Professor Hisaharu Umegaki

PREFACE TO THE FIRST EDITION

viii ABSTRACT METHODS IN INFORMATION THEORY

erties of conditional expectation and probability, Kolmogorov-Sinai’s entropy is then

In Chapter II, information sources are considered. Using an alphabet message

PREFACE TO THE FIRST EDITION ix

Another purpose of this book is to present contributions of Professor Hisaharu

PREFACE TO THE SECOND EDITION

Preface to the First Edition vii

Preface to the Second Edition xi

Chapter II. Information Sources 69

Chapter III. Information Channels 125

xiv ABSTRACT METHODS IN INFORMATION THEORY

Chapter IV. Channel Operators 190

Chapter V. Gaussian Channels 236

Chapter VI. Special Topics 285

Chapter VII. Quantum Channels 333

Glossaries of Axioms 377

1.1. The Shannon entropy

We consider basic properties and axioms of Shannon’s entropy. Let n ∈ N (the

If we define H(X|y), called the conditional entropy of X given Y = y, by

I(X, Y ) = H(Y ) − H(Y |X)

1.1. THE SHANNON ENTROPY 3

If pj > 0 and qj = 0 for some j, then we define H(p|q) = ∞. Observe the

Theorem 1. Consider entropies on ∆n .

(3) Observe the following computations:

giving the first equality and, similarly, H(X, Y ) = H(Y ) + H(X|Y ).

Let R = (−∞, ∞), R+ = [0, ∞) and R+ = [0, ∞].

(4) (Extendability) H(p1 , . . . , pn ) = H(p1 , . . . , pn , 0).

1.1. THE SHANNON ENTROPY 5

H(r11 , . . . , rnm ) ≤ H(p1 , . . . , pn ) + H(q1 , . . . , qm ).

(8) (Concavity) For p, q ∈ ∆n and α ∈ (0, 1) it holds that

Proof. (1), (2), (4) and (5) are obvious.

(7) is a reformulation of Theorem 1 (4).

(8) Since φ(t) = −t log t is concave for t > 0 we have

Summing with respect to j we obtain the desired inequality.

where the equality holds if and only if abii = const.

αp1j + (1 − α)p2j αp1j (1 − α)p2j

Adding both sides with respect to j gives (1.4).

1.1. THE SHANNON ENTROPY 7

Proposition 3. Let f : N → R+ be a function satisfying conditions 1◦ ), 2◦ ) and

f (nr ) = rf (n), n, r ∈ N, (1.6)

which can be verified by mathematical induction. Now let r, s, n ∈ N be such that