1
, with x
i
A, i 1; and A
and v A
is denoted by uv. A
nite sequence u is a prex of a nite or innite sequence w, and we
write u w, if w = uv, for some v.
The entropy H(P) of a probability distribution P = P(a), a A
is dened by the formula
H(P) =
aA
P(a) log P(a).
Here, as elsewhere in this tutorial, base two logarithms are used and
0 log 0 is dened to be 0. Random variable notation is often used in
this context. For a random variable X with values in a nite set, H(X)
denotes the entropy of the distribution of X. If Y is another random
variable, not necessarily discrete, the conditional entropy H(X[Y ) is
dened as the average, with respect to the distribution of Y , of the
entropy of the conditional distribution of X, given Y = y. The mutual
422
423
information between X and Y is dened by the formula
I(X Y ) = H(X) H(X[Y ).
If Y (as well as X) takes values in a nite set, the following alternative
formulas are also valid.
H(X[Y ) = H(X, Y ) H(Y )
I(X Y ) = H(X) +H(Y ) H(X, Y )
= H(Y ) H(Y [X).
For two distributions P and Q on A, information divergence (I
divergence) or relative entropy is dened by
D(PQ) =
aA
P(a) log
P(a)
Q(a)
.
A key property of Idivergence is that it is nonnegative and zero if and
only if P = Q. This is an instance of the logsum inequality, namely,
that for arbitrary nonnegative numbers p
1
, . . . , p
t
and q
1
, . . . , q
t
,
t
i=1
p
i
log
p
i
q
i
_
t
i=1
p
i
_
log
t
i=1
p
i
t
i=1
q
i
with equality if and only if p
i
= cq
i
, 1 i t. Here p log
p
q
is dened
to be 0 if p = 0 and + if p > q = 0.
Convergence of probability distributions, P
n
P, means point
wise convergence, that is, P
n
(a) P(a) for each a A. Topological
concepts for probability distributions, continuity, open and closed sets,
etc., are meant for the topology of pointwise convergence. Note that
the entropy H(P) is a continuous function of P, and the Idivergence
D(PQ) is a lower semicontinuous function of the pair (P, Q), contin
uous at each (P, Q) with strictly positive Q.
A code for symbols in A, with image alphabet B, is a mapping
C: A B
can
be uniquely decoded from the concatenation of the codewords of its
symbols, and each symbol can be decoded instantaneously, that is,
the receiver of any sequence w B
of which u = C(x
1
) . . . C(x
i
) is a
prex need not look at the part of w following u in order to identify u
as the code of the sequence x
1
. . . x
i
.
Of fundamental importance is the following fact.
Lemma 1.1. A function L: A N is the length function of some
prex code if and only if it satises the socalled Kraft inequality
aA
2
L(a)
1.
Proof. Given a prex code C: A B
j<i
2
L(a
j
)
can
be dyadically represented as t(i) = 0.b
1
. . . b
L(a
i
)
, and C(a
i
) = b
L(a
i
)
1
denes a prex code with length function L.
A key consequence of the lemma is Shannons noiseless coding
theorem.
Theorem 1.1. Let P be a probability distribution on A. Then each
prex code has expected length
E(L) =
aA
P(a)L(a) H(P).
Furthermore, there is a prex code with length function L(a) =
log P(a); its expected length satises
E(L) < H(P) + 1.
425
Proof. The rst assertion follows by applying the logsum inequal
ity to P(a) and 2
L(a)
in the role of p
i
and q
i
and making use of
P(a) = 1 and
2
L(a)
1. The second assertion follows since
L(a) = log P(a) obviously satises the Kraft inequality.
By the following result, even nonprex codes cannot substan
tially beat the entropy lower bound of Theorem 1.1. This justies
the practice of restricting theoretical considerations to prex codes.
Theorem 1.2. The length function of a not necessarily prex code
C: A B
satises
aA
2
L(a)
log [A[, (1.1)
and for any probability distribution P on A, the code has expected
length
E(L) =
aA
P(a)L(a) H(P) log log [A[.
Proof. It suces to prove the rst assertion, for it implies the second
assertion via the logsum inequality as in the proof of Theorem 1.1.
To this end, we may assume that for each a A and i < L(a), every
u B
i
is equal to C( a) for some a A, since otherwise C(a) can be
replaced by an u B
i
, increasing the left side of (1.1). Thus, writing
[A[ =
m
i=1
2
i
+r, m 1, 0 r < 2
m+1
,
it suces to prove (1.1) when each u B
i
, 1 i m, is a codeword,
and the remaining r codewords are of length m+1. In other words, we
have to prove that
m+r2
(m+1)
log [A[ = log(2
m+1
2 +r),
or
r2
(m+1)
log(2 + (r 2)2
m
).
426 Preliminaries
This trivially holds if r = 0 or r 2. As for the remaining case r = 1,
the inequality
2
(m+1)
log(2 2
m
)
is veried by a trite calculation for m = 1, and then it holds even more
for m > 1.
The above concepts and results extend to codes for nlength mes
sages or ncodes, that is, to mappings C: A
n
B
, B = 0, 1. In
particular, the length function L: A
n
N of an ncode is dened by
the formula C(x
n
1
) = b
L(x
n
1
)
1
, x
n
1
A
n
, and satises
x
n
1
A
n
2
L(x
n
1
)
nlog [A[;
and if C: A
n
B
x
n
1
A
n
2
L(x
n
1
)
1 .
Expected length E(L) =
x
n
1
A
n
P
n
(x
n
1
)L(x
n
1
) for a probability distribu
tion P
n
on A
n
, of a prex ncode satises
E(L) H(P
n
) ,
while
E(L) H(P
n
) log n log log [A[
holds for any ncode.
An important fact is that, for any probability distribution P
n
on
A
n
, the function L(x
n
1
) = log P
n
(x
n
1
) satises the Kraft inequality.
Hence there exists a prex ncode whose length function is L(x
n
1
) and
whose expected length satises E(L) < H(P
n
) + 1. Any such code is
called a Shannon code for P
n
.
Supposing that the limit
H = lim
n
1
n
H(P
n
)
427
exists, it follows that for any ncodes C
n
: A
n
B
P(a) =
[i: x
i
= a[
n
, a A.
A distribution P on A is called an ntype if it is the type of some
x
n
1
A
n
. The set of all x
n
1
A
n
of type P is called the type class of
the ntype P and is denoted by T
n
P
.
Lemma 2.1. The number of possible ntypes is
_
n +[A[ 1
[A[ 1
_
.
Proof. Left to the reader.
Lemma 2.2. For any ntype P
_
n +[A[ 1
[A[ 1
_
1
2
nH(P)
[T
n
P
[ 2
nH(P)
.
429
430 Large deviations, hypothesis testing
Proof. Let A = a
1
, a
2
, . . . , a
t
, where t = [A[. By the denition of
types we can write P(a
i
) = k
i
/n, i = 1, 2, . . . , t, with k
1
+k
2
+. . . +k
t
=
n, where k
i
is the number of times a
i
appears in x
n
1
for any xed
x
n
1
T
n
P
. Thus we have
[T
n
P
[ =
n!
k
1
!k
2
! k
t
!
.
Note that
n
n
= (k
1
+. . . +k
t
)
n
=
n!
j
1
! j
t
!
k
j
1
1
k
jt
t
,
where the sum is over all ttuples (j
1
, . . . , j
t
) of nonnegative integers
such that j
1
+ . . . + j
t
= n. The number of terms is
_
n +[A[ 1
[A[ 1
_
,
by Lemma 2.1, and the largest term is
n!
k
1
!k
2
! k
t
!
k
k
1
1
k
k
2
2
k
kt
t
,
for if j
r
> k
r
, j
s
< k
s
then decreasing j
r
by 1 and increasing j
s
by 1
multiplies the corresponding term by
j
r
k
r
k
s
1 +j
s
j
r
k
r
> 1.
The lemma now follows from the fact that the sum is bounded below
by its largest term and above by the largest term times the number of
terms, and noting that
n
n
k
k
1
1
k
k
2
2
k
kt
t
=
t
i=1
_
k
i
n
_
k
i
=
t
i=1
P(a
i
)
nP(a
i
)
= 2
nH(P)
.
The next result connects the theory of types with general probability
theory. For any distribution P on A, let P
n
denote the distribution of n
independent drawings from P, that is P
n
(x
n
1
) =
n
i=1
P(x
i
), x
n
1
A
n
.
Lemma 2.3. For any distribution P on A and any ntype Q
P
n
(x
n
1
)
Q
n
(x
n
1
)
= 2
nD(QP)
, if x
n
1
T
n
Q
,
_
n +[A[ 1
[A[ 1
_
1
2
nD(QP)
P
n
(T
n
Q
) 2
nD(QP)
.
2.1. Large deviations via types 431
Corollary 2.1. Let
P
n
denote the empirical distribution (type) of a
random sample of size n drawn from P. Then
Prob(D(
P
n
P) )
_
n +[A[ 1
[A[ 1
_
2
n
, > 0.
Proof. If x
n
1
T
n
Q
the number of times x
i
= a is just nQ(a), so that
P
n
(x
n
1
)
Q
n
(x
n
1
)
=
a
_
P(a)
Q(a)
_
nQ(a)
= 2
_
n
a
Q(a) log
P(a)
Q(a)
_
= 2
nD(QP)
,
that is,
P
n
(T
n
Q
) = Q
n
(T
n
Q
)2
nD(QP)
.
Here Q
n
(T
n
Q
)
_
n +[A[ 1
[A[ 1
_
1
, by Lemma 2.2 and the fact that
Q
n
(x
n
1
) = 2
nH(Q)
if x
n
1
T
n
Q
. The probability in the Corollary equals
the sum of P
n
(T
n
Q
) for all ntypes Q with D(QP) , thus Lem
mas 2.1 and 2.3 yield the claimed bound.
The empirical distribution
P
n
in the Corollary converges to P with
probability 1 as n , by the law of large numbers, or by the very
Corollary (and BorelCantelli). The next result, the nite alphabet
special case of the celebrated Sanov theorem, is useful for estimating
the (exponentially small) probability that
P
n
belongs to some set of
distributions that does not contain the true distribution P.
We use the notation D(P) = inf
Q
D(QP).
Theorem 2.1 (Sanovs Theorem.). Let be a set of distributions
on A whose closure is equal to the closure of its interior. Then for the
empirical distribution of a sample from a strictly positive distribution
P on A,
1
n
log Prob
_
P
n
_
D(P).
Proof. Let T
n
be the set of possible ntypes and let
n
= T
n
.
Lemma 2.3 implies that
Prob (
P
n
n
) = P
n
_
Qn
T
n
Q
_
432 Large deviations, hypothesis testing
is upper bounded by
_
n +[A[ 1
[A[ 1
_
2
nD(nP)
and lower bounded by
_
n +[A[ 1
[A[ 1
_
1
2
nD(nP)
.
Since D(QP) is continuous in Q, the hypothesis on implies that
D(
n
P) is arbitrarily close to D(P) if n is large. Hence the theorem
follows.
Example 2.1. Let f be a given function on A and set =
Q:
a
Q(a)f(a) > where < max
a
f(a). The set is open and
hence satises the hypothesis of Sanovs theorem. The empirical distri
bution of a random sample X
1
, ..., X
n
belongs to i (1/n)
i
f(X
i
) >
, since
a
P
n
(a)f(a) = (1/n)
i
f(X
i
). Thus we obtain the large dev
iations result
1
n
log Prob
_
1
n
n
i=1
f(X
i
) >
_
D(P).
In this case, D(P) = D(cl()P) = min D(QP), where the min
imum is over all Q for which
Q(a)f(a) . In particular, for
>
P(a)f(a) we have D(P) > 0, so that, the probability that
(1/n)
n
1
f(X
i
) > goes to 0 exponentially fast.
It is instructive to see how to calculate the exponent D(P) for the
preceding example. Consider the exponential family of distributions
P
of the form
P(a) = cP(a)2
tf(a)
, where c = (
a
P(a)2
tf(a)
)
1
. Clearly
a
P(a)f(a) is a continuous function of the parameter t and this func
tion tends to max f(a) as t . (Check!) As t = 0 gives
P = P, it
follows by the assumption
a
P(a)f(a) < < max
a
f(a)
2.1. Large deviations via types 433
that there is an element of the exponential family, with t > 0, such
that
P(a)f(a) = . Denote such a
P by Q
, so that,
Q
(a) = c
P(a)2
t
f(a)
, t
> 0,
a
Q
(a)f(a) = .
We claim that
D(P) = D(Q
P) = log c
+t
. (2.1)
To show that D(P) = D(Q
a
Q(a)f(a) > . A direct calculation gives
D(Q
P) =
a
Q
(a) log
Q
(a)
P(a)
=
a
Q
(a) [log c
+t
f(a)] = log c
+t
(2.2)
and
a
Q(a) log
Q
(a)
P(a)
=
a
Q(a) [log c
+t
+t
.
Hence
D(QP) D(Q
a
Q(a) log
Q
(a)
P(a)
= D(QQ
) > 0.
This completes the proof of (2.1).
Remark 2.1. Replacing P in (2.2) by any
P of the exponential family,
i. e.,
P(a) = cP(a)2
tf(a)
, we get that
D(Q
P) =
log
c
c
+ (t
t) = log c
+t
(log c +t).
Since D(Q
P) > 0 for
P ,= Q
, it follows that
log c +t = log
a
P(a)2
tf(a)
+t
434 Large deviations, hypothesis testing
attains its maximum at t = t
1
n
log Prob
_
1
n
n
i=1
f(X
i
) > )
__
can be represented also as
max
t0
_
log
a
P(a)2
tf(a)
+t
_
.
This latter form is the one usually found in textbooks, with the formal
dierence that logarithm and exponentiation with base e rather than
base 2 are used. Note that the restriction t 0 is not needed when
>
a
P(a)f(a), because, as just seen, the unconstrained maximum
is attained at t
a
P(a)f(a), when the exponent is equal to 0.
2.2 Hypothesis testing
Let us consider now the problem of hypothesis testing. Suppose the
statistician, observing independent drawings from an unknown distri
bution P on A, wants to test the nullhypothesis that P belongs to
a given set of distributions on A. A (nonrandomized) test of sample
size n is determined by a set C A
n
, called the critical region; the
nullhypothesis is accepted if the observed sample x
n
1
does not belong
to C. Usually the test is required to have type 1 error probability not
exceeding some > 0, that is, P
n
(C) , for all P . Subject to
this constraint, it is desirable that the type 2 error probability, that
is P(A
n
C), when P , , be small, either for a specied P ,
(testing against a simple alternative hypothesis) or, preferably, for
all P , .
Theorem 2.2. Let P
1
and P
2
be any two distributions on A, let
be a positive number, and for each n 1 suppose B
n
A
n
satises
P
n
1
(B
n
) . Then
liminf
n
1
n
log P
n
2
(B
n
) D(P
1
P
2
).
2.2. Hypothesis testing 435
The assertion of Theorem 2.2 and the special case of Theorem 2.3,
below, that there exists sets B
n
A
n
satisfying
P
n
1
(B
n
) 1,
1
n
log P
n
2
(B
n
) = D(P
1
P
2
),
are together known as Steins lemma.
Remark 2.2. On account of the logsum inequality (see Section 1),
we have for any B A
n
P
n
1
(B) log
P
n
1
(B)
P
n
2
(B)
+P
n
1
(A
n
B) log
P
n
1
(A
n
B)
P
n
2
(A
n
B)
D(P
n
1
P
n
2
)
= nD(PQ),
a special case of the lumping property in Lemma 4.1, Section 4. Since
t log t + (1 t) log(1 t) 1 for each 0 t 1, it follows that
log P
n
2
(B)
nD(PQ) + 1
P
n
1
(B)
.
Were the hypothesis P
n
1
(B
n
) of Theorem 2.2 strengthened to
P
n
1
(B
n
) 1, the assertion of that theorem would immediately follow
from the last inequality.
Proof of Theorem 2.2. With
n
=
[A[ log n
n
, say, Corollary 2.1 gives
that the empirical distribution
P
n
of a sample drawn from P
1
satis
es Prob(D(
P
n
P
1
)
n
) 0. This means that the P
n
1
probability of
the union of the type classes T
n
Q
with D(QP
1
) <
n
approaches 1 as
n . Thus the assumption P
n
1
(B
n
) implies that the intersection
of B
n
with the union of these type classes has P
n
1
probability at least
/2 when n is large, and consequently there exists ntypes Q
n
with
D(Q
n
P
1
) <
n
such that
P
n
1
(B
n
T
n
Qn
)
2
P
n
1
(T
n
Qn
).
Since samples in the same type class are equiprobable under P
n
for
each distribution P on A, the last inequality holds for P
2
in place of
P
1
. Hence, using Lemma 2.3,
P
n
2
(B
n
)
2
P
n
2
(T
n
Qn
)
2
_
n +[A[ 1
[A[ 1
_
2
nD(QnP
2
)
.
436 Large deviations, hypothesis testing
As D(Q
n
P
1
) <
n
0 implies that D(Q
n
P
2
) D(P
1
P
2
), this
completes the proof of Theorem 2.2.
Theorem 2.3. For testing the nullhypothesis that P , where
is a closed set of distributions on A, the tests with critical region
C
n
=
_
x
n
1
: inf
P
D(
P
x
n
1
P)
n
_
,
n
=
[A[ log n
n
,
have type 1 error probability not exceeding
n
, where
n
0, and for
each P
2
, , the type 2 error probability goes to 0 with exponential
rate D(P
2
).
Proof. The assertion about type 1 error follows immediately from
Corollary 2.1. To prove the remaining assertion, note that for each
P
2
, , the type 2 error probability P
2
(A
n
C
n
) equals the sum
of P
n
2
(T
n
Q
) for all ntypes Q such that inf
P
D(QP) <
n
. Denoting
the minimum of D(QP
2
) for these ntypes by
n
, it follows by Lem
mas 2.1 and 2.3, that
P
n
2
(A
n
C
n
)
_
n +[A[ 1
[A[ 1
_
2
nn
.
A simple continuity argument gives lim
n
n
= inf
P
D(PP
2
) =
D(P
2
), and hence
limsup
n
1
n
log P
n
2
(A
n
C
n
) D(P
2
).
As noted in Remark 2.3, below, the opposite inequality also holds,
hence
lim
n
1
n
log P
n
2
(A
n
C
n
) = D(P
2
),
which completes the proof of the theorem.
Remark 2.3. On account of Theorem 2.2, for any sets C
n
A
n
, such
that P
n
(C
n
) < 1, for all P , n 1, we have
liminf
n
1
n
log P
n
2
(A
n
C
n
) D(P
2
), P
2
, .
2.2. Hypothesis testing 437
Hence, the tests in Theorem 2.3 are asymptotically optimal against all
alternatives P
2
, . The assumption that is closed guarantees that
D(P
2
) > 0, whenever P
2
, . Dropping that assumption, the type
2 error probability still goes to 0 with exponential rate D(P
2
) for P
2
not in the closure of , but may not go to 0 for P
2
on the boundary of .
Finally, it should be mentioned that the criterion inf
P
D(
P
x
n
1
P)
n
dening the critical region of the tests in Theorem 2.3 is equivalent, by
Lemma 2.3, to
sup
P
P
n
(x
n
1
)
Q
n
(x
n
1
)
2
nn
= n
[A[
, Q =
P
x
n
1
.
Here the denominator is the maximum of P
n
(x
n
1
) for all distributions
P on A, thus the asymptotically optimal tests are likelihood ratio tests
in statistical terminology.
We conclude this subsection by briey discussing sequential tests. In
a sequential test, the sample size is not predetermined, rather x
1
, x
2
, . . .
are drawn sequentially until a stopping time N that depends on the
actual observations, and the null hypothesis is accepted or rejected on
the basis of the sample x
N
1
of random size N.
A stopping time N for sequentially drawn x
1
, x
2
, . . . is dened, for
our purposes, by the condition x
N
1
G, for a given set G A
of
nite sequences that satises the prex condition: no u G is a prex
of another v G; this ensures uniqueness in the denition of N. For
existence with probability 1, when x
1
, x
2
, . . . are i.i.d. drawings from a
distribution P on A, we assume that
P
(x
1
: x
1
~ u for some u G) = 1
where P
. When several
possible distributions are considered on A, as in hypothesis testing, this
condition in assumed for all of them. As P
(x
1
: x
1
~ u) = P
n
(u)
if u A
n
, the last condition equivalently means that
P
N
(u) = P
n
(u) if u G A
n
denes a probability distribution P
N
on G.
438 Large deviations, hypothesis testing
A sequential test is specied by a stopping time N or a set G A
uG
P
N
1
(u) log
P
N
1
(u)
P
N
2
(u)
= D(P
N
1
P
N
2
)
whose special case, for N equal to a constant n, appears in Remark 2.2.
Here the right hand side is the expectation of
log
P
N
1
(x
N
1
)
P
N
2
(x
N
1
)
=
N
i=1
log
P
1
(x
i
)
P
2
(x
i
)
under P
N
1
or, equivalently, under the innite product measure P
1
on
A
1
, with nite expec
tation D(P
1
P
2
), and their (random) number N is a stopping time,
Walds identity gives
D(P
N
1
P
N
2
) = E
1
(N)D(P
1
P
2
)
whenever the expectation E
1
(N) of N under P
1
(the average sample
size when hypothesis P
1
is true) is nite. It follows as in Remark 2.2
that
log
E
1
(N) D(P
1
P
2
) + 1
1
,
thus sequential tests with type 1 error probability 0 cannot have
type 2 error probability exponentially smaller than 2
E
1
(N) D(P
1
P
2
)
.
In this sense, sequential tests are not superior to tests with constant
sample size.
2.2. Hypothesis testing 439
On the other hand, sequential tests can be much superior in the
sense that one can have E
1
(N) = E
2
(N) and both error proba
bilities decreasing at the best possible exponential rates, namely with
exponents D(P
1
P
2
) and D(P
2
P
1
). This would immediately follow if
in the bound
log
1
+ (1 ) log
1
D(P
N
1
P
N
2
)
and its counterpart obtained by reversing the roles of P
1
and P
2
, the
equality could be achieved for tests with E
1
(N) = E
2
(N) .
The condition of equality in the logsum inequality gives that both
these bounds hold with the equality if and only if the probability ratio
P
N
1
(u)
P
N
2
(u)
is constant for u C and also for u G C. That condition
cannot be met exactly, in general, but it is possible to make
P
N
1
(u)
P
N
2
(u)
nearly constant on both C and GC.
Indeed, consider the sequential probability ratio test, with stopping
time N equal to the smallest n for which
c
1
<
P
n
1
(x
n
1
)
P
n
2
(x
n
1
)
< c
2
does not hold, where c
1
< 1 < c
2
are given constants, and with the
decision rule that P
1
or P
2
is accepted according as the second or the
rst inequality is violated at this stopping time. For C G A
such that
D(P
Q) = min
P
D(PQ).
In the sequel we suppose that Q(a) > 0 for all a A. The function
D(PQ) is then continuous and strictly convex in P, so that P
exists
and is unique.
The support of the distribution P is the set S(P) = a: P(a) > 0.
Since is convex, among the supports of elements of there is one
that contains all the others; this will be called the support of and
denoted by S().
Theorem 3.1. S(P
) + D(P
Q)
for all P .
440
441
Of course, if the asserted inequality holds for some P
and all
P then P
Q)] =
d
dt
D(P
t
Q) [
t=
t
,
for some
t (0, t). But
d
dt
D(P
t
Q) =
a
(P(a) P
(a)) log
P
t
(a)
Q(a)
,
and this converges (as t 0) to if P
a
(P(a) P
(a)) log
P
(a)
Q(a)
. (3.1)
It follows that the rst contingency is ruled out, proving that S(P
)
S(P), and also that the quantity (3.1) is nonnegative, proving the
claimed inequality.
Now we examine some situations in which the inequality of
Theorem 3.1 is actually an equality. For any given functions
f
1
, f
2
, . . . , f
k
on A and numbers
1
,
2
, . . . ,
k
, the set
L = P:
a
P(a)f
i
(a) =
i
, 1 i k,
if nonempty, will be called a linear family of probability distributions.
Moreover, the set c of all P such that
P(a) = cQ(a) exp
_
k
i
f
i
(a)
_
, for some
1
, . . . ,
k
,
will be called an exponential family of probability distributions; here Q
is any given distribution and
c = c(
1
, . . . ,
k
) =
_
a
Q(a) exp
_
k
i
f
i
(a)
__
1
.
442 Iprojections
We will assume that S(Q) = A; then S(P) = A for all P c. Note
that Q c. The family c depends on Q, of course, but only in a weak
manner, for any element of c could play the role of Q. If necessary to
emphasize this dependence on Q we shall write c = c
Q
.
Linear families are closed sets of distributions, exponential families
are not. Sometimes it is convenient to consider the closure cl(c) of an
exponential family c.
Theorem 3.2. The Iprojection P
) +D(P
Q), P L.
Further, if S(L) = A then Lc
Q
= P
.
Corollary 3.1. For a linear family L and exponential family c, dened
by the same functions f
1
, ..., f
k
, the intersection L cl(c) consists of a
single distribution P
, and
D(PQ) = D(PP
) +D(P
Q), P L, Q cl(c).
Proof of Theorem 3.2. By the preceding theorem, S(P
) = S(L). Hence
for every P L there is some t < 0 such that P
t
= (1 t)P
+tP L.
Therefore, we must have (d/dt)D(P
t
Q)[
t=0
= 0, that is, the quantity
(3.1) in the preceding proof is equal to 0, namely,
a
(P(a) P
(a)) log
P
(a)
Q(a)
= 0, P L. (3.2)
This proves that P
of
the subspace T of R
[A[
, spanned by the k vectors f
i
()
i
, 1 i k.
If S(L) = A then the distributions P L actually span the orthogonal
complement of T (any subspace of R
[A[
that contains a strictly positive
443
vector is spanned by the probability vectors in that subspace; the proof
is left to the reader.) Since the identity (3.2) means that the vector
log
P
()
Q()
D(P
Q)
is orthogonal to each P L, it follows that this vector belongs to
(T
c, if S(L) = A.
Next we show that any distribution P
L cl(c
Q
) satises (3.2).
Since (3.2) is equivalent to the Pythagorean identity, this will show
that L cl(c
Q
), if nonempty, consists of the single distribution equal
to the Iprojection of Q onto L. Now, let P
n
c, P
n
P
L. By the
denition of c,
log
P
n
(a)
Q(a)
= log c
n
+ (log e)
k
i=1
i,n
f
i
(a).
As P L, P
L implies
P(a)f
i
(a) =
(a)f
i
(a), i = 1, . . . , k, it
follows that
a
(P(a) P
(a)) log
P
n
(a)
Q(a)
= 0, P L.
Since P
n
P
n
denote the
Iprojection of Q onto the linear family
L
n
=
_
P:
aA
P(a)f
i
(a) =
_
1
1
n
_
i
+
1
n
aA
Q(a)f
i
(a), i = 1, . . . , k
_
.
Since (1
1
n
)P +
1
n
Q L
n
if P L, here S(L
n
) = A and therefore
P
n
c. Thus the limit of any convergent subsequence of P
n
belongs
to L cl(c).
Proof of Corollary 3.1. Only the validity of the Pythagorean identity
for Q cl(c) needs checking. Since that identity holds for Q c, taking
limits shows that the identity holds also for the limit of a sequence
Q
n
c, that is, for each Q in cl(c).
444 Iprojections
Remark 3.1. A minor modication of the proof of Theorem 3.2 shows
that the Iprojection P
(a) =
_
cQ(a) exp
_
k
1
i
f
i
(a)
_
if a B
0 otherwise.
(3.3)
This and Theorem 3.2 imply that cl(c
Q
) consists of distributions of
the form (3.3), with B = S(L) for suitable choice of the constants
1
, . . . ,
k
in the denition of L. We note without proof that also,
conversely, all such distributions belong to cl(c
Q
).
Next we show that Iprojections are relevant to maximum likelihood
estimation in exponential families.
Given a sample x
n
1
A
n
drawn from an unknown distribution sup
posed to belong to a feasible set of distributions on A, a maximum
likelihood estimate (MLE) of the unknown distribution is a maximizer
of P
n
(x
n
1
) subject to P ; if the maximum is not attained the MLE
does not exist.
Lemma 3.1. An MLE is the same as a minimizer of D(
PP) for P in
the set of feasible distributions, where
P is the empirical distribution
of the sample.
Proof. Immediate from Lemma 2.3.
In this sense, an MLE can always be regarded as a reverse I
projection. In the case when is an exponential family, the MLE
equals a proper Iprojection, though not of
P onto .
Theorem 3.3. Let the set of feasible distributions be the exponential
family
c =
_
P: P(a) = c(
1
, . . . ,
k
)Q(a) exp(
k
i=1
i
f
i
(a)), (
1
, . . . ,
k
) R
k
_
,
where S(Q) = A. Then, given a sample x
n
1
A
n
, the MLE is unique
and equals the Iprojection P
a
P(a)f
i
(a) =
1
n
n
j=1
f
i
(x
j
), 1 i k,
445
provided S(L) = A. If S(L) ,= A, the MLE does not exist, but P
will
be the MLE in that case if cl(c) rather than c is taken as the set of
feasible distributions.
Proof. The denition of L insures that
P L. Hence by Theorem 3.2
and its Corollary,
D(
PP) = D(
PP
) +D(P
P), P cl(c).
Also by Theorem 3.2, P
cl(c). Using this, the last divergence identity gives that the minimum
of D(
PP) subject to P c is uniquely attained for P = P
, if
S(L) = A, and is not attained if S(L) ,= A, while P
is always the
unique minimizer of D(
PP) subject to P cl(c). On account of
Lemma 3.1, this completes the proof of the theorem.
We conclude this subsection with a counterpart of Theorem 3.1 for
reverse Iprojections. The reader is invited to check that the theorem
below is also an analogue of one in Euclidean geometry.
Let us be given a distribution P and a closed convex set of dis
tributions on A such that S(P) S(). Then there exists Q
attaining the (nite) minimum min
Q
D(PQ); this Q
is unique if
S(P) = S(), but need not be otherwise.
Theorem 3.4. A distribution Q
).
Proof. The if part is obvious (take P
t
= P.) To prove the only if
part, S(P
t
) S(Q
t
) S(P) may be assumed else the left hand side is
innite. We claim that
aS(P)
P(a)
_
1
Q
t
(a)
Q
(a)
_
0. (3.4)
446 Iprojections
Note that (3.4) and S(P) S(P
t
) imply
aS(P
)
P
t
(a)
_
1
P(a)Q
t
(a)
P
t
(a)Q
(a)
_
0,
which, on account of log
1
t
(1 t) log e, implies in turn
aS(P
)
P
t
(a) log
P
t
(a)Q
(a)
P(a)Q
t
(a)
0.
The latter is equivalent to the inequality in the statement of the
theorem, hence it suces to prove the claim (3.4).
Now set Q
t
= (1 t)Q
+tQ
t
Q, 0 t 1. Then
0
1
t
[D(PQ
t
) D(PQ
)] =
d
dt
D(PQ
t
)
t=
t
, 0 <
t t.
With t 0 it follows that
0 lim
t0
aS(P)
P(a)
(Q
(a) Q
t
(a)) log e
(1
t)Q
(a) +
tQ
t
(a)
=
aS(P)
P(a)
Q
(a) Q
t
(a)
Q
(a)
log e.
This proves the claim (3.4) and completes the proof of Theorem 3.4.
4
fDivergence and contingency tables
Let f(t) be a convex function dened for t > 0, with f(1) = 0. The
fdivergence of a distribution P from Q is dened by
D
f
(PQ) =
a
Q(x)f
_
P(x)
Q(x)
_
.
Here we take 0f(
0
0
) = 0, f(0) = lim
t0
f(t), 0f(
a
0
) = lim
t0
tf(
a
t
) =
a lim
u
f(u)
u
.
Some examples include the following.
(1) f(t) = t log t D
f
(PQ) = D(PQ).
(2) f(t) = log t D
f
(PQ) = D(QP).
(3) f(t) = (t 1)
2
D
f
(PQ) =
a
(P(a) Q(a))
2
Q(a)
.
(4) f(t) = 1
t
D
f
(PQ) = 1
a
_
P(a)Q(a).
(5) f(t) = [t 1[ D
f
(PQ) = [P Q[ =
a
[P(a) Q(a)[.
In addition to information divergence obtained in (1), (2), the f
447
448 fDivergence and contingency tables
divergences in (3), (4), (5) are also often used in statistics. They are
called
2
divergence, Hellinger distance, and variational distance, re
spectively.
The analogue of the logsum inequality is
i
b
i
f
_
a
i
b
i
_
bf
_
a
b
_
, a =
a
i
, b =
b
i
, (4.1)
where if f is strictly convex at c = a/b, the equality holds i a
i
= cb
i
, for
all i. Using this, many of the properties of the information divergence
D(PQ) extend to general fdivergences, as shown in the next lemma.
Let B = B
1
, B
2
, . . . , B
k
be a partition of A and let P be a distribution
on A. The distribution dened on 1, 2, . . . , k by the formula
P
B
(i) =
aB
i
P(a),
is called the Blumping of P.
Lemma 4.1. D
f
(PQ) 0 and if f is strictly convex at t = 1 then
D
f
(PQ) = 0 only when P = Q. Further, D
f
(PQ) is a convex
function of the pair (P, Q), and the lumping property, D
f
(PQ)
D
f
(P
B
Q
B
) holds for any partition B of A.
Proof. The rst assertion and the lumping property obviously follow
from the analogue of the logsum inequality, (4.1). To prove convexity,
let P = P
1
+ (1 )P
2
, Q = Q
1
+ (1 )Q
2
. Then P and Q are
lumpings of distributions
P and
Q dened on the set A 1, 2 by
P(a, 1) = P
1
(a),
P(a, 2) = (1 )P
2
(a), and similarly for
Q. Hence
by the lumping property,
D
f
(PQ) D
f
(
P[[
Q) = D
f
(P
1
[[Q
1
) + (1 )D
f
(P
2
[[Q
2
).
A basic theorem about fdivergences is the following approximation
by the
2
divergence
2
(P, Q) =
(P(a) Q(a))
2
/Q(a).
Theorem 4.1. If f is twice dierentiable at t = 1 and f
tt
(1) > 0 then
for any Q with S(Q) = A and P close to Q we have
D
f
(PQ)
f
tt
(1)
2
2
(P, Q).
Formally, D
f
(PQ)/
2
(P, Q) f
tt
(1)/2 as P Q.
449
Proof. Since f(1) = 0, Taylors expansion gives
f(t) = f
t
(1)(t 1) +
f
tt
(1)
2
(t 1)
2
+(t)(t 1)
2
,
where (t) 0 as t 1. Hence
Q(a)f
_
P(a)
Q(a)
_
=
f
t
(1)(P(a) Q(a)) +
f
tt
(1)
2
(P(a) Q(a))
2
Q(a)
+
_
P(a)
Q(a)
_
(P(a) Q(a))
2
Q(a)
.
Summing over a A then establishes the theorem.
Remark 4.1. The same proof works even if Q is not xed, replacing
P Q by P Q 0, provided that no Q(a) can become arbi
trarily small. However, the theorem (the asymptotic equivalence of
fdivergences subject to the dierentiability hypotheses) does not re
main true if Q is not xed and the probabilities of Q(a) are not bounded
away from 0.
Corollary 4.1. Let f
0
= 1, f
1
, . . . , f
[A[1
be a basis for R
[A[
(regarded
as the linear space of all realvalued functions on A), orthonormal with
respect to the inner product < g, h >
Q
=
a
Q(a)g(a)h(a). Then,
under the hypotheses of Theorem 4.1,
D
f
(PQ)
f
tt
(1)
2
[A[1
i=1
_
a
P(a)f
i
(a)
_
2
,
and, for the linear family
L() = P:
a
P(a)f
i
(a) =
i
, 1 i k,
with = (
1
. . . ,
k
) approaching the zero vector,
min
PL()
D
f
(PQ)
f
tt
(1)
2
k
i=1
2
i
.
450 fDivergence and contingency tables
Proof. On account of Theorem 4.1, it suces to show that
2
(P, Q) =
[A[1
i=1
_
a
P(a)f
i
(a)
_
2
(4.2)
and, at least when = (
1
. . . ,
k
) is suciently close to the zero
vector,
min
PL()
2
(P, Q) =
k
i=1
2
i
. (4.3)
Now,
2
(P, Q) =
a
Q(a)
_
P(a)
Q(a)
1
_
2
is the squared norm of the
function g dened by g(a) =
P(a)
Q(a)
1 with respect to the given inner
product, and that equals
[A[1
i=0
< g, f
i
>
2
Q
. Here
< g, f
0
>
Q
=
a
(P(a) Q(a)) = 0
< g, f
i
>
Q
=
a
(P(a) Q(a))f
i
(a)
=
a
P(a)f
i
(a), 1 i [A[ 1,
the latter since < f
0
, f
i
>
Q
= 0 means that
a
Q(a)f
i
(a) = 0. This
proves (4.2), and (4.3) then obviously follows if some P L() sat
ises
a
P(a)f
i
(a) = 0, k + 1 i [A[ 1. Finally, the as
sumed orthonormality of 1, f
1
, . . . , f
[A[1
implies that P dened by
P(a) = Q(a)(1 +
k
i=1
i
f
i
(a)) satises the last conditions, and this
P is a distribution in L() provided it is nonnegative, which is cer
tainly the case if is suciently close to the zero vector.
One property distinguishing information divergence among f
divergences is transitivity of projections, as summarized in the following
lemma. It can, in fact, be shown that the only fdivergence for which
either of the two properties of the lemma holds is the informational
divergence.
Lemma 4.2. Let P
onto L
t
are the same.
(ii) For any translate L
t
of L, the Iprojections of Q and of P
onto
L
t
are the same, provided S(L) = A.
L
t
is called a translate of L if it is dened in terms of the same functions
f
i
, but possibly dierent
i
.
Proof. By the Pythagorean identity
D(PQ) = D(PP
) +D(P
Q), P L,
it follows that on any subset of L the minimum of D(PQ) and of
D(PP
belongs
to this exponential family. But every element of the exponential family
has the same Iprojection onto L
t
, which establishes (ii).
In the following theorem,
P
n
denotes the empirical distribution of
a random sample of size n from a distribution Q with S(Q) = A,
that is, the type of the sequence (X
1
, . . . , X
n
) where X
1
, X
2
, . . . are
independent random variables with distribution Q.
Theorem 4.2. Given real valued functions f
1
, . . . , f
k
, (1 k <
[A[ 1) on A such that f
0
= 1, f
1
, . . . , f
k
are linearly independent,
let P
n
be the Iprojection of Q onto the (random) linear family
L
n
= P:
a
P(a)f
i
(a) =
1
n
n
j=1
f
i
(X
j
), 1 i k.
Then
D(
P
n
Q) = D(
P
n
P
n
) +D(P
n
Q),
each term multiplied by
2n
log e
has a
2
limiting distribution with [A[
1, [A[ 1 k, respectively k, degrees of freedom, and the right hand
side terms are asymptotically independent.
452 fDivergence and contingency tables
The
2
distribution with k degrees of freedom is dened as the
distribution of the sum of squares of k independent random variables
having the standard normal distribution.
Proof of Theorem 4.2. The decomposition of D(
P
n
Q) is a special case
of the Pythagorean identity, see Theorem 3.2, since clearly
P
n
L
n
. To
prove the remaining assertions, assume that f
0
= 1, f
1
, . . . , f
k
are or
thonormal for the inner product dened in Corollary 4.1. This does not
restrict generality since the family L
n
depends on f
1
, . . . , f
k
through
the linear span of 1, f
1
, . . . , f
k
, only. Further, take additional functions
f
k+1
, . . . , f
[A[1
on A to obtain a basis for R
[A[
, orthonormal for the
considered inner product. Then, since
P
n
Q in probability, Corol
lary 4.1 applied to f(t) = t log t, with f
tt
(1) = log e, gives
D(
P
n
Q)
log e
2
[A[1
i=1
_
P
n
(a)f
i
(a)
_
2
=
log e
2
[A[1
i=1
1
n
n
j=1
f
i
(X
j
)
2
,
D(P
n
Q) = min
PLn
D(PQ)
log e
2
k
i=1
1
n
n
j=1
f
i
(X
j
)
2
.
Here, asymptotic equivalence of random variables means that their
ratio goes to 1 in probability, as n .
By the assumed orthonormality of f
0
= 1, f
1
, . . . , f
[A[1
, for X with
distribution Q the real valued random variables f
i
(X), 1 i [A[ 1,
have zero mean and their covariance matrix is the ([A[ 1) ([A[ 1)
identity matrix. It follows by the central limit theorem that the joint
distribution of the random variables
Z
n,i
=
1
n
n
j=1
f
i
(X
j
), 1 i [A[ 1
converges, as n , to the joint distribution of [A[ 1 independent
random variables having the standard normal distribution.
453
As the asymptotic relations established above give
2n
log e
D(
P
n
Q)
[A[1
i=1
Z
2
n,i
,
2n
log e
D(P
n
Q)
k
i=1
Z
2
n,i
,
and these imply by the Pythagorean identity that
2n
log e
D(
P
n
P
n
)
[A[1
i=k+1
Z
2
n,i
,
all the remaining claims follow.
Remark 4.2. D(
P
n
P
n
) is the preferred statistic for testing the hypo
thesis that the sample has come from a distribution in the exponential
family
c =
_
P: P(a) = cQ(a) exp
_
k
i=1
i
f
i
(a)
_
, (
1
, . . . ,
k
) R
k
_
.
Note that D(
P
n
P
n
) equals the inmum of D(
P
n
P), subject to P c,
by Corollary 3.1 in Section 3, and the test rejecting the above hypo
thesis when D(
P
n
P
n
) exceeds a threshold is a likelihood ratio test,
see Remark 2.3 in Subsection 2.2. In this context, it is relevant that
the limiting distribution of
2n
log e
D(
P
n
P
n
) is the same no matter which
member of c the sample is coming from, as any P c could play the
role of Q in Theorem 4.2.
Note also that Theorem 4.2 easily extends to further decompositions
of D(
P
n
Q). For example, taking additional functions f
k+1
, . . . , f
with
1, f
1
, . . . , f
n
be the common Iprojection
of Q and P
n
to
L
1
=
P:
a
P(a)f
i
(a) =
1
n
n
j=1
f
i
(X
j
), 1 i
.
Then
D(
P
n
Q) = D(
P
n
P
n
) +D(P
n
P
n
) +D(P
n
Q),
the right hand side terms multiplied by
2n
log e
have
2
limiting distribu
tions with degrees of freedom [A[ 1, k, k respectively, and these
terms are asymptotically independent.
454 fDivergence and contingency tables
Table 4.1 A 2dimensional contingency table
x(0, 0) x(0, 1) x(0, r
2
) x(0)
x(1, 0) x(1, 1) x(1, r
2
) x(1)
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
x(r
1
, 0) x(r
1
, 1) x(r
1
, r
2
) x(r
1
)
x(0) x(1) x(r
2
) n
Now we apply some of these ideas to the analysis of contingency
tables. A 2dimensional contingency table is indicated in Table 4.1.
The sample data have two features, with categories 0, . . . , r
1
for the
rst feature and 0, . . . , r
2
for the second feature. The cell counts
x(j
1
, j
2
), 0 j
1
r
1
, 0 j
2
r
2
are nonnegative integers; thus in the sample there were x(j
1
, j
2
) mem
bers that had category j
1
for the rst feature and j
2
for the second.
The table has two marginals with marginal counts
x(j
1
) =
r
2
j
2
=0
x(j
1
, j
2
), x(j
2
) =
r
1
j
1
=0
x(j
1
, j
2
).
The sum of all the counts is
n =
j
1
x(j
1
) =
j
2
x(j
2
) =
j
1
j
2
x(j
1
, j
2
).
The term contingency table comes from this example, the cell counts
being arranged in a table, with the marginal counts appearing at the
margins. Other forms are also commonly used, e. g., the counts are
replaced by the empirical probabilities p(j
1
, j
2
) = x(j
1
, j
2
)/n, and the
marginal counts are replaced by the marginal empirical probabilities
P(j
1
.) = x(j
1
.)/n and
P(.j
2
) = x(.j
2
)/n.
In the general case the sample has d features of interest, with the
ith feature having categories 0, 1, . . . , r
i
. The dtuples = (j
1
, . . . , j
d
)
are called cells; the corresponding cell count x() is the number of
members of the sample such that, for each i, the ith feature is in the
455
j
i
th category. The collection of possible cells will be denoted by . The
empirical distribution is dened by p() = x()/n, where n =
x()
is the sample size. By a ddimensional contingency table we mean either
the aggregate of the cell counts x(), or the empirical distribution p,
or sometimes any distribution P on (mainly when considered as a
model for the true distribution from which the sample came.)
The marginals of a contingency table are obtained by restrict
ing attention to those features i that belong to some given set
1, 2, . . . , d. Formally, for = (i
1
, . . . , i
k
) we denote by () the 
projection of = (j
1
, . . . , j
d
), that is, () = (j
i
1
, j
i
2
, . . . , j
i
k
). The
marginal of the contingency table is given by the marginal counts
x(()) =
()=()
x(
t
)
or the corresponding empirical distribution p(()) = x(())/n. In
general the marginal of any distribution P(): is dened as
the distribution P
(()) =
()=()
P(
t
).
A ddimensional contingency table has d onedimensional marginals,
d(d1)/2 twodimensional marginals, ..., corresponding to the subsets
of 1, . . . , d of one, two, ..., elements.
For contingency tables the most important linear families of distri
butions are those dened by xing certain marginals, for a family
of sets 1, . . . , d. Thus, denoting the xed marginals by
P
, ,
we consider
L = P: P
=
P
, .
The exponential family (through any given Q) that corresponds to this
linear family L consists of all distributions that can be represented in
product form as
P() = cQ()
(()). (4.4)
In particular, if L is given by xing the onedimensional marginals (i. e.,
consists of the one point subsets of 1, . . . , d) then the corresponding
456 fDivergence and contingency tables
exponential family consists of the distributions of the form
P(i
1
, . . . , i
d
) = cQ(i
1
, . . . , i
d
)a
1
(i
1
) a
d
(i
d
).
The family of all distributions of the form (4.4) is called a loglinear
family with interactions . In most applications, Q is chosen as the
uniform distribution; often the name loglinear family is restricted
to this case. Then (4.4) gives that the log of P() is equal to a sum of
terms, each representing an interaction , for it depends on =
(j
1
, . . . , j
d
) only through () = (j
i
1
, . . . , j
i
k
), where = (i
1
, . . . , i
k
).
A loglinear family is also called a loglinear model. It should be
noted that the representation (4.4) is not unique, because it corresponds
to a representation in terms of linearly dependent functions. A common
way of achieving uniqueness is to postulate a
(()) = 1 whenever at
least one component of () is equal to 0. In this manner a unique
representation of the form (4.4) is obtained, provided that with every
also the subsets of are in . Loglinear models of this form are
called hierarchical models.
Remark 4.3. The way we introduced loglinear models shows that
restricting to the hierarchical ones is more a notational than a real
restriction. Indeed, if some marginal is xed then so are the
t

marginals for all
t
.
In some cases of interest it is desirable to summarize the information
content of a contingency table by its marginals, . In such cases
it is natural to consider the linear family L consisting of those distribu
tions whose marginals equal those of the empirical distribution,
P.
If a prior guess Q is available, then we accept the Iprojection P
of Q
onto L as an estimate of the true distribution. By Theorem 3.2, this P
equals the intersection of the loglinear family (4.4), or its closure, with
the linear family L. Also, P
) = min
Pc
D(
PP)
457
is large. Unfortunately the numerical bounds obtained in
Theorem 2.3 appear too crude for most applications, and the rejection
criterion there, namely D(
PP
) [[
log n
n
, admits false acceptance
too often. A better criterion is suggested by the result in Theorem 4.2
(see also Remark 4.3) that
2n
log e
D(
PP
) has
2
limit distribution, with
specied degrees of freedom, if the null hypothesis is true. Using this
theorem, the nullhypothesis is rejected if (2n/ log e)D(
PP
) exceeds
the threshold found in the table of the
2
distribution for the selected
level of signicance. Of course, the type 1 error probability of the re
sulting test will be close to the desired one only when the sample size
n is suciently large for the distribution of the test statistic to be close
to its
2
limit. The question of how large n is needed is important but
dicult, and will not be entered here.
Now we look at the problem of outliers. A lack of t (i. e., D(
PP
)
large) may be due not to the inadequacy of the model tested, but
to outliers. A cell
0
is considered to be an outlier in the following
case: Let L be the linear family determined by the marginals (say
) of the empirical distribution
P, and let L
t
be the subfamily
of L consisting of those P L that satisfy P(
0
) =
P(
0
). Let P
be the Iprojection of P
onto L
t
. Ideally, we should consider
0
as an
outlier if D(P
P
P
) is close to D(
PP
)
then D(
PP
P
)
P(
0
) log
P(
0
)
P
(
0
)
+
_
1
P(
0
_
log
P(
0
)
P
(
0
)
,
and we declare
0
as an outlier if the righthand side of this inequality
is large, that is, after scaling by (2n/ log e), it exceeds the critical
value of
2
with one degree of freedom.
If the above method produces only a few outliers, say
0
,
1
, . . . ,
,
we consider the subset
L of L consisting of those P L that satisfy
P(
j
) =
P(
j
) for j = 0, . . . , . If the Iprojection of P
onto
L is
already close to
P, we accept the model and attribute the original
lack of t to the outliers. Then the outlier cell counts x(
j
) are
deemed unreliable and they may be adjusted to nP
(
j
).
458 fDivergence and contingency tables
Similar techniques are applicable in the case when some cell counts
are missing.
5
Iterative algorithms
In this Section we discuss iterative algorithms to compute I
projections and to minimize Idivergence between two convex sets of
distributions, as well as to estimate a distribution from incomplete
data.
5.1 Iterative scaling
The Iprojection to a linear family L is very easy to nd if L is de
termined by a partition B = (B
1
, . . . , B
k
) of A and consists of all those
distributions P whose Blumping is a given distribution (
1
, . . . ,
k
) on
1, . . . , k. Indeed, then D(PQ) D(P
B
Q
B
) =
i
log
i
/Q(B
i
)
for each P L, by the lumping property (see Lemma 4.1), and here
the equality holds for P
dened by
P
(a) = c
i
Q(a), a B
i
, where c
i
=
i
Q(B
i
)
. (5.1)
It follows that P
= P: P
= P
, where
1, . . . , d, the Iprojection P
of Q to L
is obtained by scaling Q
to adjust its marginal: P
() = Q()P
(())/Q
(()). Suppose
next that L can be represented as the intersection of families L
i
, i =
1, . . . , m, each of form as above. Then, on account of Theorem 5.1,
below, and the previous paragraph, Iprojections to L can be com
puted by iterative scaling. This applies, in particular, to Iprojections
to families dened by prescribed marginals, required in the analysis
of contingency tables: For L = P
= P
, , =
1
, . . . ,
m
,
the Iprojection of Q to L equals the limit of the sequence of dis
tributions P
(n)
dened by iterative scaling, that is, P
(0)
= Q, and
P
(n)
() = P
(n1)
()P
n
((
n
))/P
(n1)
n
((
n
)), where
1
,
2
, . . . is a
cyclic repetition of .
Suppose L
1
, . . . , L
m
, are given linear families and generate a se
quence of distributions P
n
as follows: Set P
0
= Q (any given distribu
tion with support S(Q) = A), let P
1
be the Iprojection of P
0
onto L
1
,
P
2
the Iprojection of P
1
onto L
2
, and so on, where for n > m we mean
by L
n
that L
i
for which i n (mod m); i. e., L
1
, . . . , L
m
is repeated
cyclically.
Theorem 5.1. If
m
i=1
L
i
= L ,= then P
n
P
, the Iprojection of
Q onto L.
Proof. By the Pythagorean identity, see Theorem 3.2, we have for
every P L (even for P L
n
) that
D(PP
n1
) = D(PP
n
) +D(P
n
P
n1
), n = 1, 2, . . .
Adding these equations for 1 n N we get that
D(PQ) = D(PP
0
) = D(PP
N
) +
N
n=1
D(P
n
P
n1
).
By compactness there exists a subsequence P
N
k
P
t
, say, and then
from the preceding inequality we get for N
k
that
D(PQ) = D(PP
t
) +
n=1
D(P
n
P
n1
). (5.2)
5.1. Iterative scaling 461
Since the series in (5.2) is convergent, its terms go to 0, hence also the
variational distance [P
n
P
n1
[ =
a
[P
n
(a) P
n1
(a)[ goes to 0 as
n . This implies that together with P
N
k
P
t
we also have
P
N
k
+1
P
t
, P
N
k
+2
P
t
, . . . , P
N
k
+m
P
t
.
Since by the periodic construction, among the m consecutive elements,
P
N
k
, P
N
k
+1
, . . . , P
N
k
+m1
there is one in each L
i
, i = 1, 2, . . . , m, it follows that P
t
L
i
= L.
Since P
t
L it may be substituted for P in (5.2) to yield
D(P
t
Q) =
i=1
D(P
n
P
n1
).
With this, in turn, (5.2) becomes
D(PQ) = D(PP
t
) +D(P
t
Q),
which proves that P
t
equals the Iprojection P
of Q onto L. Finally, as
P
t
was the limit of an arbitrary convergent subsequence of the sequence
P
n
, our result means that every convergent subsequence of P
n
has the
same limit P
and
completes the proof of the theorem.
In the general case when L =
m
i=1
L
i
but no explicit formulas are
available for Iprojections to the families L
i
, Theorem 5.1 need not
directly provide a practical algorithm for computing the Iprojection
to L. Still, with a twist, Theorem 5.1 does lead to an iterative algo
rithm, known as generalized iterative scaling (or the SMART algorithm)
to compute Iprojections to general linear families and, in particular,
MLEs for exponential families, see Theorem 3.3.
Generalized iterative scaling requires that the linear family
L =
_
P:
aA
P(a)f(a) =
i
, 1 i k
_
be given in terms of functions f
i
that satisfy
f
i
(a) 0,
k
i=1
f
i
(a) = 1, a A ; (5.3)
462 Iterative algorithms
accordingly, (
1
, . . . ,
k
) has to be a probability vector. This does not
restrict generality, for if L is initially represented in terms of any func
tions
f
i
, these can be replaced by f
i
= C
f
i
+D with suitable constants
C and D to make sure that f
i
0 and
k
i=1
f
i
(a) 1; if the last
inequality is strict for some a A, one can replace k by k + 1, and
introduce an additional function f
k+1
= 1
k
i=1
f
i
.
Theorem 5.2. Assuming (5.3), the nonnegative functions b
n
on A de
ned recursively by
b
0
(a) = Q(a), b
n+1
(a) = b
n
(a)
k
i=1
_
i
n,i
_
f
i
(a)
,
n,i
=
aA
b
n
(a)f
i
(a)
converge to the Iprojection P
of Q to L, that is, P
(a) =
lim
n
b
n
(a), a A.
Intuitively, in generalized iterative scaling the values b
n
(a) are updated
using all constraints in each step, via multiplications by weighted geo
metric means of the analogues
i
/
n,i
of the ratios in (5.1) that have
been used in standard iterative scaling, taking one constraint into ac
count in each step. Note that the functions b
n
need not be probability
distributions, although their limit is.
Proof of Theorem 5.2. Consider the product alphabet
A = A
1, . . . , k, the distribution
Q = Q(a)f
i
(a), (a, i)
A, and the linear
family
L of those distributions
P on
A that satisfy
P(a, i) = P(a)f
i
(a)
for some P L. Since for such
P we have D(
P
Q) = D(PQ), the
Iprojection of
Q to
L equals
P
= P
(a)f
i
(a) where P
is the I
projection of Q to L.
Note that
L =
L
1
L
2
where
L
1
is the set of all distributions
P =
L
2
=
P:
P(a, i) = P(a)f
i
(a), P any distribution on A.
It follows by Theorem 5.1 that the sequence of distributions
P
0
,
P
0
t
,
P
1
,
P
1
t
, . . . on
A dened iteratively, with
P
0
=
Q, by
P
t
n
= Iprojection to
L
1
of
P
n
,
P
n+1
= Iprojection to
L
2
of
P
n
t
5.2. Alternating divergence minimization 463
converges to
P
. In particular, writing
P
n
(a, i) = P
n
(a)f
i
(a), we have
P
n
P
P
n
by scaling, thus
P
t
n
(a, i) =
i
n,i
P
n
(a)f
i
(a),
n,i
=
aA
P
n
(a)f
i
(a) .
To nd
P
n+1
, note that for each
P = P(a)f
i
(a) in
L
2
we have,
using (5.3),
D(
P
P
t
n
) =
aA
k
i=1
P(a)f
i
(a) log
_
P(a)
P
n
(a)
_
i
n,i
_
=
aA
P(a) log
P(a)
P
n
(a)
aA
P(a)
k
i=1
f
i
(a) log
i
n,i
=
aA
P(a) log
P(a)
P
n
(a)
k
i=1
_
i
n,i
_
f
i
(a)
.
This implies, by the logsum inequality, that the minimum of D(
P
P
t
n
)
subject to
P
L
2
is attained by
P
n+1
= P
n+1
(a)f
i
(a) with
P
n+1
(a) = c
n+1
P
n
(a)
k
i=1
_
i
n,i
_
f
i
(a)
where c
n+1
is a normalizing constant. Comparing this with the recur
sion dening b
n
in the statement of the theorem, it follows by induction
that P
n
(a) = c
n
b
n
(a), n = 1, 2, . . ..
Finally, c
n
1 follows since the above formula for D(
P
P
t
n
) gives
D(
P
n+1

P
t
n
) = log c
n+1
, and D(
P
n+1

P
t
n
) 0 as in the proof of
Theorem 5.1.
5.2 Alternating divergence minimization
In this subsection we consider a very general alternating minimiza
tion algorithm which, in particular, will nd the minimum divergence
between two convex sets T and Q of distributions on a nite set A.
464 Iterative algorithms
In the general considerations below, T and Q are arbitrary sets
and D(P, Q) denotes an extended realvalued function on T Q which
satises the following conditions.
(a) < D(P, Q) +, P T, Q Q.
(b) P T, Q
= Q
).
(c) Q Q, P
= P
, Q).
A problem of interest in many situations is to determine
D
min
def
= inf
PT,QQ
D(P, Q). (5.4)
A naive attempt to solve this problem would be to start with some
Q
0
Q and recursively dene
P
n
= P
(Q
n1
), Q
n
= Q
(P
n
), n = 1, 2, . . . (5.5)
hoping that D(P
n
, Q
n
) D
min
, as n .
We show that, subject to some technical conditions, the naive iter
ation scheme (5.5) determines the inmum in (5.4). This is stated as
the following theorem.
Theorem 5.3. Suppose there is a nonnegative function (P, P
t
) de
ned on T T with the following properties:
(i) threepoints property,
(P, P
(Q)) +D(P
(P)), P
t
T, Q
t
Q.
(iii) (P
(Q), P
1
) < for Q Q with min
P1
D(P, Q) < .
Then, if min
P1
D(P, Q
0
) < , the iteration (5.5) produces (P
n
, Q
n
)
such that
lim
n
D(P
n
, Q
n
) = inf
P1,Q
D(P, Q) = D
min
. (5.6)
5.2. Alternating divergence minimization 465
Under the additional hypotheses that (iv) T is compact, (v)
D(P, Q
, where
D(P
, Q
(P
)) = D
min
; moreover, (P
, P
n
) 0 and
D(P
n+1
, Q
n
) D
min
(P
, P
n
) (P
, P
n+1
). (5.7)
Proof. We have, by the threepoints property,
(P, P
n+1
) +D(P
n+1
, Q
n
) D(P, Q
n
),
and, by the fourpoints property
D(P, Q
n
) D(P, Q) +(P, P
n
),
for all P T, Q Q. Hence
(P, P
n+1
) D(P, Q) D(P
n+1
, Q
n
) +(P, P
n
) (5.8)
We claim that the iteration (5.5) implies the basic limit result (5.6).
Indeed, since
D(P
1
, Q
0
) D(P
1
, Q
1
) D(P
2
, Q
1
) D(P
2
, Q
2
) . . .
by denition, if (5.6) were false there would exist Q and > 0 such
that D(P
n+1
, Q
n
) > D(P
(Q), P
n+1
)
(P
(Q), P
n
) , for n = 1, 2, . . ., contradicting assumption (iii) and
the nonnegativity of .
Supposing also the assumptions (iv)(vi), pick a convergent subse
quence of P
n
, say P
n
k
P
, Q
(P
)) liminf
k
D(P
n
k
, Q
n
k
) = D
min
,
and by the denition of D
min
, here the equality must hold. By (5.8)
applied to D(P, Q) = D(P
, Q
(P
)) = D
min
, it follows that
(P
, P
n+1
) D
min
D(P
n+1
, Q
n
) +(P
, P
n
),
proving (5.7). This last inequality also shows that (P
, P
n+1
)
(P
, P
n
), n = 1, 2, . . . , and, since (P
, P
n
k
) 0, by (vi), this
466 Iterative algorithms
proves that (P
, P
n
) 0. Finally, again by (vi), the latter implies
that P
n
P
.
Next we want to apply the theorem to the case when T and Q
are convex, compact sets of measures on a nite set A (in the remain
der of this subsection by a measure we mean a nonnegative, nite
valued measure, equivalently, a nonnegative, realvalued function on
A), and D(P, Q) = D(PQ) =
a
P(a) log(P(a)/Q(a)), a denition
that makes sense even if the measures do not sum to 1. The existence
of minimizers Q
(P) and P
(P
n
). To verify this, choose
a subsequence such that D(P
n
k
Q
n
k
) liminf
n
D(P
n
Q
n
) and
Q
n
k
converges to some Q
) lim
k
D(P
n
k
Q
n
k
), and the assertion follows.
Returning to the question whether assumption (iii) of Theorem 5.3
holds in our case, note that (P
(Q)P
1
) = (P
(Q)P
(Q
0
)) is nite
if the divergence D(P
(Q)P
(Q
0
)) is nite on account of the three
points property (i). Now, for each Q Q with inf
PT
D(PQ) <
whose support is contained in the support of Q
0
, the inclusions
S(P
(Q)P
(Q
0
) is nite. This
means that assumption is always satised if Q
0
has maximal support,
that is, S(Q
0
) = S(Q). Thus we have arrived at
5.2. Alternating divergence minimization 467
Corollary 5.1. Suppose T and Qare convex compact sets of measures
on a nite set A such that there exists P T with S(P) S(Q),
and let D(P, Q) = D(PQ), (P, Q) = (PQ). Then all assertions of
Theorem 5.3 are valid, provided the iteration (5.5) starts with a Q
0
Q
of maximal support.
Note that under the conditions of the corollary, there exists a unique
minimizer of D(PQ) subject to P T, unless D(PQ) = + for
every P T. There is a unique minimizer of D(PQ) subject to Q
Q if S(P) = S(Q), but not necessarily if S(P) is a proper subset
of S(Q); in particular, the sequences P
n
, Q
n
dened by the iteration
(5.5) need not be uniquely determined by the initial Q
0
Q. Still,
D(P
n
Q
n
) D
min
always holds, P
n
always converges to some P
T
with min
QQ
D(P
Q) = D
min
, and each accumulation point of Q
n
Q, then Q
n
Q
P
n
) (P
P
n+1
) =
=
aA
P
(a) log
P
n+1
(a)
P
n
(a)
+
aA
[P
n
(a) P
n+1
(a)] log e
_
max
PT
P(A)
_
max
aA
log
P
n+1
(a)
P
n
(a)
+ [P
n
(A) P
n+1
(A)] log e
where P(A)
def
=
aA
P(a); using this, the iteration can be stopped
when the last bound becomes smaller than a prescribed > 0. The
criterion becomes particularly simple if T consists of probability dis
tributions.
Corollary 5.1 can be applied, as we show below, to minimizing I
divergence when either the rst or second variable is xed and the
other variable ranges over the image of a nice set of measures on a
larger alphabet. Here nice sets of measures are those for which the
divergence minimization is easy.
468 Iterative algorithms
For a mapping T: A B and measures P on A, write P
T
for
the image of P on B, that is, P
T
(b) =
a:Ta=b
P(a). For a set T of
measures on A write T
T
= P
T
: P T.
Problem 1. Given a measure P on B and a convex set Q of measures
on A, minimize D(PQ) subject to Q Q
T
.
Problem 2. Given a measure Q on B and a convex set T of measures
on A, minimize D(PQ) subject to P T
T
.
Lemma 5.1. The minimum in Problem 1 equals D
min
= min
PT,QQ
for T = P: P
T
= P and the given Q, and if (P
, Q
) attains D
min
then Q
T
attains the minimum in Problem 1.
A similar result holds for Problem 2, with the roles of P and Q
interchanged.
Proof. The lumping property of Lemma 4.1, which also holds for ar
bitrary measures, gives
D(P
T
Q
T
) D(PQ), with equality if
P(a)
Q(a)
=
P
T
(b)
Q
T
(b)
, b = Ta.
From this it follows that if T = P: P
T
= P for a given P, then the
minimum of D(PQ) subject to P T (for Q xed) is attained for
P
= P
(Q) with
P
(a) =
Q(a)
Q
T
(b)
P(b), b = Ta (5.9)
and this minimum equals D(PQ
T
). A similar result holds also for
minimizing D(PQ) subject to Q Q (for P xed) in the case when
Q = Q: Q
T
= Q for a given Q, in which case the minimizer Q
=
Q
(P) is given by
Q
(a) =
P(a)
P
T
(b)
Q(b), b = Ta (5.10)
The assertion of the lemma follows.
Example 5.1 (Decomposition of mixtures.). Let P be a proba
bility distribution and let
1
, . . . ,
k
be arbitrary measures on a 
nite set B. The goal is to minimize D(P
k
c
i
i
) for weight vectors
5.2. Alternating divergence minimization 469
(c
1
, . . . , c
k
) with nonnegative components that sum to 1. If
1
, . . . ,
k
are probability distributions and P is the empirical distribution of a
sample drawn from the mixture
i
c
i
i
then the goal is identical to
nding the MLE of the weight vector (c
1
, . . . , c
k
).
This example ts into the framework of Problem 1, above, by set
ting A = 1, . . . , k B, T(i, b) = b, Q = Q: Q(i, b) = c
i
i
(b).
Thus we consider the iteration (5.5) as in Corollary 5.1, with T =
P:
i
P(i, b) = P(b), b B and Q above, assuming for nontriviality
that S(P)
i
S(
i
) (equivalent to the support condition in Corol
lary 5.1 in our case). As Corollary 5.1 requires starting with Q
0
Q of
maximal support, we assume Q
0
(i, b) = c
0
i
i
(b), c
0
i
> 0, i = 1, . . . , k.
To give the iteration explicitly, note that if Q
n1
(i, b) = c
n1
i
i
(b) is
already dened then P
n
is obtained, according to (5.9), as
P
n
(i, b) =
Q
n1
(i, b)
Q
T
n1
(b)
P(b) =
c
n1
i
i
(b)
j
c
n1
j
j
(b)
P(b).
To nd Q
n
Q minimizing D(P
n
Q), put P
n
(i) =
bB
P
n
(i, b) and
use Q(i, b) = c
i
i
(b) to write
D(P
n
Q) =
k
i=1
bB
P
n
(i, b) log
P
n
(i, b)
c
i
i
(b)
=
k
i=1
P
n
(i) log
P
n
(i)
c
i
+
k
i=1
bB
P
n
(i, b) log
P
n
(i, b)
P
n
(i)
i
(b)
.
This is minimized for c
n
i
= P
n
(i), hence the recursion for c
n
i
will be
c
n
i
= c
n1
i
bB
i
(b)P(b)
j
c
n1
j
j
(b)
.
Finally, we show that (c
n
1
, . . . , c
n
k
) converges to a minimizer
(c
1
, . . . , c
k
) of D(P
k
c
i
i
). Indeed, P
n
converges to a limit P
by
Corollary 5.1, hence c
n
i
= P
n
(i) also has a limit c
i
and Q
n
Q
with
Q
(i, b) = c
i
(b). By the passage following Corollary 5.1, (P
, Q
)
attains D
min
= min
PT,QQ
D(PQ), and then, by Lemma 5.1,
Q
T
=
i
c
i
attains min
QQ
T D(PQ) = D
min
.
Remark 5.1. A problem covered by Example 5.1 is that of nding
weights c
i
> 0 of sum 1 that maximize the expectation of log
i
c
i
X
i
,
470 Iterative algorithms
where X
1
, . . . , X
k
are given nonnegative random variables dened on a
nite probability space (B, P). Indeed, then
E(log
i
c
i
X
i
) = D(P
i
c
i
i
),
for
i
(b) = P(b)X
i
(b). In this case, the above iteration takes the form
c
n
i
= c
n1
i
E(
X
i
j
c
n
j
X
j
),
which is known as Covers portfolio optimization algorithm. We note
without proof that the algorithm works also for nondiscrete X
1
, . . . , X
k
.
Remark 5.2. The counterpart of the problem of Example 5.1, namely,
the minimization of D(
k
c
i
i
Q) can be solved similarly. Then the
iteration of Corollary 5.1 has to be applied to the set T consist
ing of the measures of the form P(i, b) = c
i
i
(b) and to Q =
Q:
i
Q(i, b) = Q(b), b B. Actually, the resulting iteration
is the same as that in the proof of Theorem 5.2 (assuming the
i
and Q are probability distributions), with notational dierence
that the present i, b, c
i
,
i
(b), Q(b), P
n
T, Q
n
Q correspond to
a, i, P(a), f
i
(a),
i
,
P
n
L
2
,
P
t
n
L
1
there. To see this, note that while
the even steps of the two iterations are conceptually dierent divergence
minimizations (with respect to the second, respectively, rst variable,
over the set denoted by Q or
L
1
), in fact both minimizations require
the same scaling, see (5.9), (5.10).
This observation gives additional insight into generalized iterative
scaling, discussed in the previous subsection. Note that Theorem 5.2
involves the assumption L ,= (as linear families have been dened to
be nonempty, see Section 3), and that assumption is obviously nec
essary. Still, the sequence P
n
in the proof of Theorem 5.2 is well
dened also if L = , when
L
1
and
L
2
in that proof are disjoint.
Now, the above observation and Corollary 5.1 imply that P
n
converges
to a limit P
minimizes the
Idivergence from (
1
, . . . ,
k
) of distributions (
1
, . . . ,
k
) such that
i
=
a
P(a)f
i
(a), 1 i k, for some probability distribution P on
A.
5.3. The EM algorithm 471
5.3 The EM algorithm
The expectationmaximization or EM algorithm is an iterative
method frequently used in statistics to estimate a distribution supposed
to belong to a given set Q of distributions on a set A when, instead
of a full sample x
n
1
= x
1
. . . x
n
A
n
from the unknown distribution,
only an incomplete sample y
n
1
= y
1
. . . y
n
B
n
is observable. Here
y
i
= Tx
i
for a known mapping T : A B. As elsewhere in this tutorial,
we restrict attention to the case of nite A, B.
The EM algorithm produces a sequence of distributions Q
k
Q,
regarded as consecutively improved estimates of the unknown distribu
tion, iterating the following steps E and M, starting with an arbitrary
Q
0
Q.
Step E: Calculate the conditional expectation P
k
= E
Q
k1
(
P
n
[y
n
1
)
of the empirical distribution
P
n
of the unobservable full sample, condi
tioned on the observed incomplete sample, pretending the true distri
bution equals the previous estimate Q
k1
.
Step M: Calculate the MLE of the distribution the full sample x
n
1
is coming from, pretending that the empirical distribution of x
n
1
equals
P
k
calculated in Step E. Set Q
k
equal to this MLE.
Here, motivated by Lemma 3.1, by MLE pretending the empirical
distribution equals P
k
we mean the minimizer of D(P
k
[[Q) subject to
Q Q, even if P
k
is not a possible empirical distribution (implicitly
assuming that a minimizer exists; if there are several, any one of them
may be taken). For practicality, we assume that step M is easy to
perform; as shown below, step E is always easy.
The EM algorithm is, in eect, an alternating divergence minimiza
tion, see the previous subsection. To verify this, it suces to show that
P
k
in Step E minimizes the divergence D(P[[Q
k1
) subject to P T,
for T = P : P
T
=
P
T
n
, where P
T
denotes the image of P under the
mapping T: A B. Actually, we claim that for any distribution Q on
A, the conditional expectation P = E
Q
(
P
n
[y
n
1
) attains the minimum
of D(P[[Q) subject to P
T
=
P
T
n
.
Now, writing (x, a) = 1 if x = a and 0 otherwise, we have for any
472 Iterative algorithms
a A
E
Q
(
P
n
(a)[y
n
1
) = E
Q
_
1
n
n
i=1
(x
i
, a)
y
n
1
_
=
1
n
n
i=1
E
Q
((x
i
, a)[y
i
) =
[i : y
i
= Ta[
n
Q(a)
Q
T
(Ta)
.
Indeed, under the condition that y
i
= Tx
i
is given, the conditional
expectation of (x
i
, a), that is, the conditional probability of x
i
= a,
equals Q(a)/Q
T
(Ta) if y
i
= Ta, and zero otherwise. As the empirical
distribution of y
n
1
is equal to
P
T
n
, this means that P = E
Q
(
P
n
[y
n
1
) is
given by
P(a) =
P
T
n
(Ta)
Q(a)
Q
T
(Ta)
.
Since D(P[[Q) D(P
T
[[Q
T
) for any P (by the lumping property, see
Lemma 4.1), and for P given above here clearly the equality holds, it
follows that P = E
Q
(
P
n
[y
n
1
) minimizes D(P[[Q) subject to P
T
=
P
T
n
,
as claimed.
An immediate consequence is that
D(P
1
[[Q
0
) D(P
1
[[Q
1
) D(P
2
[[Q
1
) D(P
2
[[Q
2
) . . .
In particular, as D(P
k
[[Q
k1
) = D(
P
T
n
[[Q
T
k1
), the sequence
D(
P
T
n
[[Q
T
k
) is always nonincreasing and hence converges to a limit
as k .
In the ideal case, this limit equals
min
Q
D(
P
T
n
[[Q
T
) = min
P1,Q
D(P[[Q) = D
min
where T = P : P
T
=
P
T
n
. In this ideal case, supposing some
Q in
Q
T
= Q
T
: Q Q is the unique minimizer of D(
P
T
n
[[Q
t
) subject to
Q
t
cl(Q
T
), it also holds that Q
T
k
Q. Indeed, for any convergent
subsequence of Q
T
k
, its limit Q
t
cl(Q
T
) satises
D(
P
T
n
[[Q
t
) = lim
k
D(
P
T
n
[[Q
T
k
) = D
min
,
hence Q
t
=
Q by the uniqueness assumption. Note that
Q is the MLE
of the distribution governing the elements y
i
= Tx
i
of the incomplete
sample y
n
1
, by Lemma 3.1.
5.3. The EM algorithm 473
If the set Q of feasible distributions is convex and compact, the
above ideal situation always obtains if the EM algorithm is started
with a Q
0
Q of maximal support, by Corollary 5.1 in the previous
subsection. Then by the last paragraph, supposing the minimizer of
D(
P
T
n
[[Q
t
) subject to Q
t
Q
T
is unique (for which S(
P
T
n
) = S(Q
T
) is
a sucient condition), the EM algorithm always provides a sequence
of distributions Q
k
whose Timages approach that minimizer, that is,
the MLE of the distribution underlying the incomplete sample. This
implies, in turn, that the distributions Q
k
themselves converge to a
limit, because the P
k
s obtained in the steps E as
P
k
(a) =
P
T
n
(Ta)
Q
k1
(a)
Q
T
k1
(Ta)
do converge to a limit, by Corollary 5.1.
An example of the EM algorithm with convex, compact Q is the
decomposition of mixtures in Example 5.1. It should be noted, however,
that in most situations where the EM algorithm is used in statistics, the
set Qof feasible distributions is not convex. Then Corollary 5.1 does not
apply, and the ideal case D(
P
T
n
[[Q
T
k
) D
min
need not obtain; indeed,
the iteration often gets stuck at a local optimum. A practical way to
overcome this problem is to run the algorithm with several dierent
choices of the initial Q
0
.
6
Universal coding
A Shannon code for a distribution P
n
on A
n
has the length func
tion log P
n
(x
n
1
) and produces expected length within 1 bit of the
entropy lower bound H(P
n
); it therefore provides an almost optimal
method for coding if it is known that the data x
n
1
is governed by P
n
.
In practice, however, the distribution governing the data is usually not
known, though it may be reasonable to assume that the data are com
ing from an unknown member of a known class T of processes, such
as the i.i.d. or Markov or stationary processes. Then it is desirable to
use universal codes that perform well no matter which member of
T is the true process. In this Section, we introduce criteria of good
performance of codes relative to a process. We also describe universal
codes for the classes of i.i.d. and Markov processes, and for some oth
ers, which are almost optimal in a strong sense and, in addition, are
easy to implement.
By a process with alphabet A we mean a Borel probability measure
P on A
1
: x
n
1
= a
n
1
, a
n
1
A
n
, n = 1, 2, . . . ;
474
6.1. Redundancy 475
see the Appendix for a summary of process concepts. The marginal
distribution P
n
on A
n
of a process P is dened by
P
n
(a
n
1
) = P([a
n
1
]), a
n
1
A
n
;
we also write briey P(a
n
1
) for P
n
(a
n
1
).
Simple examples are the i.i.d. processes with
P(a
n
1
) =
n
t=1
P(a
t
), a
n
1
A
n
,
and the Markov chains with
P(a
n
1
) = P
1
(a
1
)
n
t=2
P(a
t
[a
t1
), a
n
1
A
n
,
where P
1
= P
1
(a): a A is an initial distribution, and P(a[ a),
a A, a A is a transition probability matrix, that is, P([ a) is a
probability distribution on A for each a A. Stationary processes are
those that satisfy
P(x
1
: x
i+n
i+1
= a
n
1
) = P([a
n
1
]), for each i, n, and a
n
1
A
n
.
6.1 Redundancy
The ideal codelength of a message x
n
1
A
n
coming from a process
P is dened as log P(x
n
1
). For an arbitrary ncode C
n
: A
n
B
,
B = 0, 1, the dierence of its length function from the ideal will
be called the redundancy function R = R
P,Cn
:
R(x
n
1
) = L(x
n
1
) + log P(x
n
1
).
The value R(x
n
1
) for a particular x
n
1
is also called the pointwise redun
dancy.
One justication of this denition is that a Shannon code for P
n
,
with length function equal to the rounding of the ideal to the next
integer, attains the least possible expected length of a prex code
C
n
: A
n
B
, or a sequence C
n
: n = 1, 2, . . . of ncodes. The context will make
clear which possibility is being used. A code C
n
: n = 1, 2, . . . is said
to be a prex code if each C
n
is one, and strongly prex if C
m
(y
m
1
)
C
n
(x
n
1
) can hold only when y
m
1
x
n
1
.
Theorem 6.1. Given an arbitrary process P and code C
n
: n =
1, 2, . . . (not necessarily prex),
R(x
n
1
) c
n
eventually almost surely,
for any sequence of numbers c
n
with
n2
cn
< +, e.g., for c
n
=
3 log n. Moreover, if the code is strongly prex, or its length function
satises L(x
n
1
) log Q(x
n
1
) for some process Q, then
E
P
(inf
n
R(x
n
1
)) > .
Proof. Let
A
n
(c) = x
n
1
: R(x
n
1
) < c = x
n
1
: 2
L(x
n
1
)
P(x
n
1
) < 2
c
.
Then
P
n
(A
n
(c)) =
x
n
1
A
n
(c)
P(x
n
1
) < 2
c
x
n
1
An(c)
2
L(x
n
1
)
2
c
log [A
n
[
where, in the last step, we used Theorem 1.2. Hence
n=1
P
n
(A
n
(c
n
)) log [A[
n=1
n 2
cn
,
and the rst assertion follows by the BorelCantelli principle.
The second assertion will be established if we show that for codes
with either of the stated properties
P(x
1
: inf
n
R(x
n
1
) < c) < 2
c
, c > 0 ,
6.1. Redundancy 477
or in other words,
n=1
P
n
(B
n
(c)) < 2
c
where
B
n
(c) = x
n
1
: R(x
n
1
) < c, R(x
k
1
) c, k < n.
As in the proof of the rst assertion,
P
n
(B
n
(c)) < 2
c
x
n
1
Bn(c)
2
L(x
n
1
)
,
hence it suces to show that
n=1
x
n
1
Bn(c)
2
L(x
n
1
)
1.
If C
n
: n = 1, 2, . . . is a strongly prex code, the mapping
C: (
n=1
B
n
(c)) B
dened by C(x
n
1
) = C
n
(x
n
1
), x
n
1
B
n
(c), satis
es the prex condition, and the claim holds by the Kraft inequality.
If L(x
n
1
) log Q(x
n
1
), we have
x
n
1
Bn(c)
2
L(x
n
1
)
x
n
1
Bn(c)
Q(x
n
1
) = Q
n
(B
n
(c)),
and the desired inequality follows since
n=1
Q
n
(B
n
(c)) = Q(x
1
: inf
n
R(x
n
1
) < c) 1.
In the literature, dierent concepts of universality, of a code
C
n
: n = 1, 2, . . . for a given class T of processes, have been used.
A weak concept requires the convergence to 0 of the expected redun
dancy per symbol,
1
n
E
P
(R
P,Cn
) 0, for each P T; (6.1)
stronger concepts require uniform convergence to 0, for P T, of either
(1/n)E
P
(R
P,Cn
) or of (1/n) max
x
n
1
A
n
R
P,Cn
(x
n
1
).
478 Universal coding
In the context of strong universality, natural gures of merit of
a code C
n
: A
n
B
Cn
= sup
PT
max
x
n
1
A
n
R
P,Cn
(x
n
1
).
Example 6.1. For the class of i.i.d. processes, natural universal codes
are obtained by rst encoding the type of x
n
1
, and then identifying x
n
1
within its type class via enumeration. Formally, for x
n
1
of type Q, let
C
n
(x
n
1
) = C(Q)C
Q
(x
n
1
), where C: T
n
B
. Since P
n
(T
n
Q
) is maximized for P = Q, it follows that for
x
n
1
in T
Q
, the maximum pointwise redundancy of the code C
n
equals
L(Q) + log Q
n
(T
n
Q
), up to 1 bit.
Consider rst the case when the type code has xed length L(Q) =
log [T
n
[. This is asymptotically equal to ([A[ 1) log n as n ,
by Lemma 2.1 and Stirlings formula. For types Q of sequences x
n
1
in
which each a A occurs a fraction of time bounded away from 0,
one can see via Stirlings formula that log Q
n
(T
n
Q
) is asymptotically
(([A[ 1)/2) log n. Hence for such sequences, the maximum redun
dancy is asymptotically (([A[ 1)/2) log n. On the other hand, the
maximum for x
n
1
of L(Q) +log Q
n
(T
n
Q
) is attained when x
n
1
consists of
identical symbols, when Q
n
(T
n
Q
) = 1; this shows that R
Cn
is asymp
totically ([A[ 1) log n in this case.
Consider next the case when C: T
n
B
Cn
equals
log c
n
up to 2 bits. We shall see that this is essentially best possible
(Theorem 6.2), and in the present case R
Cn
= (([A[ 1)/2) log n+O(1)
(Theorem 6.3).
Note that to any prex code C
n
: A
n
B
Cn
for prex codes C
n
: A
n
B
almost equals
R
n
= min
Qn
sup
PT
x
n
1
A
n
P(x
n
1
) log
P(x
n
1
)
Q
n
(x
n
1
)
(6.2)
or
R
n
= min
Qn
sup
PT
max
x
n
1
A
n
log
P(x
n
1
)
Q
n
(x
n
1
)
. (6.3)
More exactly, each prex code C
n
: A
n
B
n
, respectively,
and a Shannon code for a Q
n
attaining the minimum in (6.2) or (6.3)
achieves this lower bound up to 1 bit. In particular, for a class T of
processes, there exist strongly universal codes with expected or max
imum redundancy per symbol converging to 0 uniformly for P T, if
and only if R
n
= o(n) or R
n
= o(n), respectively.
Our next theorem identies the minimizer in (6.3) and the value
R
n
. The related problem for R
n
will be treated in Section 7.
We use the following notation. Given a class T of processes with
alphabet A, we write
P
ML
(x
n
1
)
def
= sup
PT
P(x
n
1
), x
n
1
A
n
,
where the subscript on P
ML
emphasizes its interpretation as the maxi
mizer of P(x
n
1
) subject to P T (if it exists), that is, as the maximum
480 Universal coding
likelihood estimate of the process P T that generates x
n
1
. The nor
malized form
NML
n
(a
n
1
)
def
= P
ML
(a
n
1
)/
x
n
1
A
n
P
ML
(x
n
1
), a
n
1
A
n
,
is called the the normalized maximum likelihood distribution.
Theorem 6.2. For any class T of processes with nite alphabet A,
the minimum in (6.3) is attained for Q
n
= NML
n
, the normalized
maximum likelihood distribution, and
R
n
= log
x
n
1
A
n
P
ML
(x
n
1
).
Proof. For arbitrary Q
n
,
sup
PT
max
x
n
1
A
n
log
P(x
n
1
)
Q
n
(x
n
1
)
= log max
x
n
1
A
n
P
ML
(x
n
1
)
Q
n
(x
n
1
)
.
Here
max
x
n
1
A
n
P
ML
(x
n
1
)
Q
n
(x
n
1
)
x
n
1
A
n
Q
n
(x
n
1
)
P
ML
(x
n
1
)
Q
n
(x
n
1
)
=
x
n
1
A
n
P
ML
(x
n
1
),
with equality if Q
n
= NML
n
.
6.2 Universal codes for certain classes of processes
While Shannon codes for the distributions NML
n
, n = 1, 2, . . . are
optimal for the class T within 1 bit, with respect to the maximum
redundancy criterion, by Theorem 6.2, they are typically not practical
from the implementation point of view. We will show that for some
simple but important classes T there exist easily implementable arith
metic codes C
n
: n = 1, 2, . . . which are nearly optimal, in the sense
that
R
Cn
R
n
+ constant, n = 1, 2, . . . (6.4)
Recall that an arithmetic code (of the second kind, see equation (1.2) in
Section 1) determined by the marginals Q
n
, n = 1, 2, . . . of any process
6.2. Universal codes for certain classes of processes 481
Q, is a prex code C
n
: n = 1, 2, . . . with length function L(x
n
1
) =
log Q(x
n
1
) +1. Note that the obvious idea to take a process Q with
marginals Q
n
= NML
n
does not work, since such a process typically
does not exist (that is, the distributions NML
n
, n = 1, 2, . . ., do not
meet the consistency criteria for a process).
Below we describe suitable coding processes Q, and for the cor
responding arithmetic codes we prove upper bounds to R
Cn
. For the
class of i.i.d processes, we also determine R
n
, up to a constant, and
establish the bound (6.4) for our code. For other classes, the proof of
the claimed near optimality will be completed in the next subsection,
where we also prove near optimality in the expected redundancy sense.
In the rest of this subsection, we assume with no loss of generality
that A = 1, . . . , k.
Consider rst the case when T is the class of i.i.d. processes with
alphabet A. Let the coding process be the process Q whose marginal
distributions Q
n
= Q(x
n
1
): x
n
1
A
n
are given by
Q(x
n
1
) =
n
t=1
n(x
t
[x
t1
1
) +
1
2
t 1 +
k
2
,
where n(i[x
t1
1
) denotes the number of occurrences of the symbol i in
the past x
t1
1
. Equivalently,
Q(x
n
1
) =
k
i=1
[(n
i
1
2
)(n
i
3
2
)
1
2
]
(n 1 +
k
2
)(n 2 +
k
2
)
k
2
, (6.5)
where n
i
= n(i[x
n
1
), and (n
i
1
2
)(n
i
3
2
) . . .
1
2
= 1, by denition, if
n
i
= 0.
Note that the conditional probabilities needed for arithmetic coding
are given by the simple formula
Q(i[x
t1
1
) =
n(i[x
t1
1
) +
1
2
t 1 +
k
2
.
Intuitively, this Q(i[x
t1
1
) is an estimate of the probability of i from
the past x
t1
1
, under the assumption that the data come from an
unknown P T. The unbiased estimate n(i[x
t1
1
)/(t 1) would
be inappropriate here, since an admissible coding process requires
Q(i[x
t1
1
) > 0 for each possible x
t1
1
and i.
482 Universal coding
Remark 6.1. It is not intuitively obvious at this point in our discus
sion why exactly 1/2 is the right bias term that admits the strong
redundancy bound below. Later, in Section 7, we establish a deep con
nection between minimax expected redundancy
R
n
and mixture distri
butions with respect to priors (which is closely connected to Bayesian
ideas); the 1/2 then arises from using a specic prior. For now we note
only that replacing 1/2 by 1 in formula (6.5) leads to the coding distri
bution Q(x
n
1
) =
n
1
n
i
!
(n1+k)(n2+k)k
which equals
1
[Pn[[T
Q
[
, if x
n
1
T
Q
,
see Lemma 2.1. In this case, the length function is the same (up to 2
bits) as the rst, suboptimal, version of the twostage code in Exam
ple 6.1.
We claim that the arithmetic code determined by the process Q
satises
R
Cn
k 1
2
log n + constant. (6.6)
Since the length function is L(x
n
1
) = log Q(x
n
1
) +1, our claim will be
established if we prove
Theorem 6.3. For Q determined by (6.5), and any i.i.d. process P
with alphabet A = 1, . . . , k,
P(x
n
1
)
Q(x
n
1
)
K
0
n
k1
2
, x
n
1
A
n
,
where K
0
is a constant depending on the alphabet size k only.
Proof. We begin by noting that given x
n
1
A
n
, the i.i.d. process
with largest P(x
n
1
) is that whose onedimensional distribution equals
the empirical distribution (n
1
/n, . . . , n
k
/n) of x
n
1
, see Lemma 2.3, and
hence
P(x
n
1
) P
ML
(x
n
1
) =
k
i=1
_
n
i
n
_
n
i
.
In a moment we will use a combinatorial argument to establish the
bound
k
i=1
_
n
i
n
_
n
k
i=1
[(n
i
1
2
)(n
i
3
2
)
1
2
]
(n
1
2
)(n
3
2
)
1
2
. (6.7)
6.2. Universal codes for certain classes of processes 483
This is enough to yield the desired result, for if it is true, then we
can use P(x
n
1
)/Q(x
n
1
) P
ML
(x
n
1
)/Q(x
n
1
) and the Qformula, (6.5), to
obtain
P(x
n
1
)
Q(x
n
1
)
n
j=1
n +
k
2
j
n +
1
2
j
, x
n
1
A
n
. (6.8)
If the alphabet size k is odd, the product here simplies, and is obvi
ously of order n
k1
2
. If k is even, using
(n
1
2
)(n
3
2
)
1
2
=
(2n 1)(2n 3) 1
2
n
=
(2n)!
2
2n
n!
=
2n(2n 1) (n + 1)
2
2n
, (6.9)
the product in (6.8) can be rewritten as
(n +
k
2
1)!/(
k
2
1)!
(2n)!/2
2n
n!
,
and Stirlings formula gives that this is of order n
k1
2
. Hence, it indeed
suces to prove (6.7).
To prove (6.7), we rst use (6.9) to rewrite it as
k
i=1
_
n
i
n
_
n
k
i=1
[2n
i
(2n
i
1) (n
i
+ 1)]
2n(2n + 1) (n + 1)
, (6.10)
which we wish to establish for ktuples of nonnegative integers n
i
with
sum n. This will be done if we show that it is possible to assign to each
= 1, . . . , n in a onetoone manner, a pair (i, j), 1 i k, 1 j n,
such that
n
i
n
n
i
+j
n +
. (6.11)
Now, for any given and i, (6.11) holds i j n
i
/n. Hence the number
of those 1 j n
i
that satisfy (6.11) is greater than n
i
n
i
/n, and
the total number of pairs (i, j), 1 i k, 1 j n, satisfying (6.11)
is greater than
k
i=1
_
n
i
n
i
n
_
= n .
It follows that if we assign to = n any (i, j) satisfying (6.11) (i. e., i
may be chosen arbitrarily and j = n
i
), then recursively assign to each
484 Universal coding
= n1, n2, etc., a pair (i, j) satisfying (6.11) that was not assigned
previously, we never get stuck; at each step there will be at least one
free pair (i, j) (because the total number of pairs (i, j) satisfying
(6.11) is greater than n , the number of pairs already assigned.)
This completes the proof of the theorem.
Remark 6.2. The above proof has been preferred for it gives a sharp
bound, namely, in equation (6.7) the equality holds if x
n
1
consists of
identical symbols, and this bound could be established by a purely
combinatorial argument. An alternative proof via Stirlings formula,
however, yields both upper and lower bounds. Using equation (6.9),
the numerator in equation (6.5) can be written as
i:n
i
,=0
(2n
i
)!
2
2n
i
n!
,
which, by Stirlings formula, is bounded both above and below by con
stant times e
n
i:n
i
,=0
n
n
i
i
. The denominator in equation (6.5) can also
be expressed by factorials (trivially if k is even, and via equation (6.9)
if k is odd), and Stirlings formula shows that it is bounded both above
and below by a constant times e
n
n
n+
k1
2
. This admits the conclusion
that P
ML
(x
n
1
)/Q(x
n
1
) is bounded both above and below by a constant
times n
k1
2
, implying
Theorem 6.4. For the class of i.i.d processes,
R
n
= log
x
n
1
A
n
P
ML
(x
n
1
) =
k 1
2
log n +O(1).
Consequently, our code satisfying equation (6.6) is nearly optimal in
the sense of equation (6.4).
Next, let T be the class of Markov chains with alphabet A =
1, . . . , k. We claim that for this class, the arithmetic code determined
by the process Q below satises
R
Cn
k(k 1)
2
log n + constant. (6.12)
6.2. Universal codes for certain classes of processes 485
Let the coding process be that Q whose marginal distributions Q
n
=
Q(x
n
1
): x
n
1
A
n
are given by
Q(x
n
1
) =
1
k
k
i=1
k
j=1
[(n
ij
1/2)(n
ij
3/2) . . . 1/2]
(n
i
1 +k/2)(n
i
2 +k/2) . . . k/2
; (6.13)
here n
ij
is the number of times the pair i, j occurs in adjacent places
in x
n
1
, and n
i
=
j
n
ij
. Note that n
i
is now the number of occurrences
of i in the block x
n1
1
(rather than in x
n
1
as before). The conditional
Qprobabilities needed for arithmetic coding are given by
Q(j[x
t1
1
) =
n
t1
(i, j) +
1
2
n
t1
(i) +
k
2
, if x
t1
= i,
where n
t1
(i, j) and n
t1
(i) have similar meaning as n
ij
and n
i
above,
with x
t1
1
in the role of x
n
1
.
Similarly to the i.i.d. case, to show that the arithmetic code de
termined by Q above satises (6.12) for the class of Markov chains, it
suces to prove
Theorem 6.5. For Q determined by (6.13) and any Markov chain
with alphabet A = 1. . . . , k,
P(x
n
1
)
Q(x
n
1
)
K
1
n
k(k1)
2
, x
n
1
A
n
,
where K
1
is a constant depending on k only.
Proof. For any Markov chain, the probability of x
n
1
A
n
is of form
P(x
n
1
) = P
1
(x
1
)
n
t=2
P(x
t
[x
t1
) = P
1
(x
1
)
k
i=1
k
j=1
P(j[i)
n
ij
.
This and (6.13) imply that
P(x
n
1
)
Q(x
n
1
)
k
k
i=1
_
k
j=1
P(j[i)
n
ij
_
k
j=1
[(n
ij
1/2)(n
ij
3/2) . . . 1/2]
(n
i
1 +k/2)(n
i
2 +k/2) . . . k/2
_
.
Here, when n
i
,= 0, the square bracket is the same as the ratio in
Theorem 6.3 for a sequence x
n
i
1
A
n
i
with empirical distribution
486 Universal coding
(n
i1
/n
i
, . . . , n
ik
/n
i
), and an i.i.d. process with onedimensional distri
bution P([i). Hence, it follows from Theorem 6.3 that
P(x
n
1
)
Q(x
n
1
)
k
n
i
,=0
_
K
0
n
k1
2
i
_
(k K
k
0
)n
k(k1)
2
.
Consider next the class of Markov chains of order at most m, namely
of those processes P (with alphabet A = 1, . . . , k) for which the
probabilities P(x
n
1
), x
n
1
A
n
, n m can be represented as
P(x
n
1
) = P
m
(x
m
1
)
n
t=m+1
P(x
t
[x
t1
tm
),
where P([a
m
1
) is a probability distribution for each a
n
1
A
n
. The
Markov chains considered before correspond to m = 1. To the anal
ogy of that case we now dene a coding process Q whose marginal
distributions Q
n
, n m, are given by
Q(x
n
1
) =
1
k
m
a
m
1
A
m
k
j=1
[(n
a
m
1
j
1/2)(n
a
m
1
j
3/2) . . . 1/2]
(n
a
m
1
1 +k/2)(n
a
m
1
2 +k/2) . . . k/2
, (6.14)
where n
a
m
1
j
denotes the number of times the block a
m
1
j occurs in x
n
1
,
and n
a
m
1
=
j
n
a
m
1
j
is the number of times the block a
m
1
occurs in
x
n1
1
.
The same argument as in the proof of Theorem 6.5 gives that for
Q determined by (6.14), and any Markov chain of order m,
P(x
n
1
)
Q(x
n
1
)
K
m
n
k
m
(k1)
2
, K
m
= k
m
K
k
m
0
. (6.15)
It follows that the arithmetic code determined by Q in (6.14) is a
universal code for the class of Markov chains of order m, satisfying
R
Cn
k
m
(k 1)
2
log n + constant. (6.16)
Note that the conditional Qprobabilities needed for arithmetic coding
are now given by
Q(j[x
t1
1
) =
n
t1
(a
m
1
, j) +
1
2
n
t1
(a
m
1
) +
k
2
, if x
t1
tm
= a
m
1
,
6.2. Universal codes for certain classes of processes 487
where n
t1
(a
m
1
, j) and n
t1
(a
m
1
) are dened similarly to n
a
m
1
j
and n
a
m
1
,
with x
t1
1
in the role of x
n
1
.
A subclass of the Markov chains of order m, often used in statistical
modeling, is specied by the assumption that the transition probabil
ities P(j[a
m
1
) depend on a
m
1
through a context function f(a
m
1
) that
has less than k
m
possible values, say 1, . . . , s. For m < t n, the
tth symbol in a sequence x
n
1
A
n
is said to occur in context if
f(x
t1
tm
) = . A suitable coding process for this class, determined by
the context function f, is dened by
Q(x
m
1
) =
1
k
m
s
=1
k
j=1
[(n
,j
1/2)(n
,j
3/2) . . . 1/2]
(n
1 +k/2)(n
2 +k/2) . . . k/2
,
where n
,j
denotes the number of times j occurs in context in the
sequence x
n
1
, and n
=
k
j=1
n
,j
. The arithmetic code determined by
this process Q satises, for the present class,
R
Cn
s(k 1)
2
log n + constant, (6.17)
by the same argument as above. The conditional Qprobabilities needed
for arithmetic coding are now given by
Q(j[x
t1
1
) =
n
t1
(, j) +
1
2
n
t1
() +
k
2
, if f(x
t1
tm
) = .
Finally, let T be the class of all stationary processes with alphabet
A = 1, . . . , k. This is a large class that does not admit strong sense
universal codes, that is, the convergence in (6.1) cannot be uniform for
any code, see Example 8.3 in Section 8. We are going to show, however,
that the previous universal codes designed for Markov chains of order m
perform reasonably well also for the class T of stationary processes,
and can be used to obtain universal codes for T in the weak sense of
(6.1).
To this end, we denote by Q
(m)
the coding process dened by (6.14)
tailored to the class of Markov chains of order m (in particular, Q
(0)
is
the process dened by (6.5)), and by C
m
n
: n = 1, 2, . . . the arithmetic
code determined by the process Q
(m)
.
488 Universal coding
Theorem 6.6. Let P T have entropy rate H = lim
n
H
m
where, with X
n
denoting a representation of P, H
m
=
H(X
m+1
[X
1
, . . . , X
m
). Then
1
n
E
P
(R
P,C
m
n
) H
m
H +
k
m
(k 1)
2
log n
n
+
c
m
n
,
where c
m
is a constant depending only on m and the alphabet size k,
with c
m
= O(k
m
) as m .
Corollary 6.1. For any sequence of integers m
n
with m
n
log n, < 1/ log k, the prex code C
mn
n
: n = 1, 2, . . . satises (6.1).
Moreover, the arithmetic code determined by the mixture process
Q =
m=0
m
Q
(m)
(with
m
> 0,
m
= 1)
also satises (6.1).
Proof. Given a stationary process P, let P
(m)
denote its mth Markov
approximation, that is, the stationary Markov chain of order m with
P
(m)
(x
n
1
) = P(x
m
1
)
n
t=m+1
P(x
t
[x
t1
tm
), x
n
1
A
n
,
where
P(x
t
[x
t1
tm
) = ProbX
t
= x
t
[X
t1
tm
= x
t1
tm
.
The bound (6.15) applied to P
(m)
in the role of P gives
log
P(x
n
1
)
Q
(m)
(x
n
1
)
= log
P(x
n
1
)
P
(m)
(x
n
1
)
+ log
P
(m)
(x
n
1
)
Q
(m)
(x
n
1
)
log
P(x
n
1
)
P
(m)
(x
n
1
)
+
k
m
(k 1)
2
log n + log K
m
,
where log K
m
= O(k
m
).
Note that the expectation under P of log P(x
n
1
) equals H(P
n
),
and that of log P(x
t
[x
t1
tm
) equals H
m
. Hence for the code C
m
n
with
length function L(x
n
1
) = log Q
(m)
(x
n
1
)+1, the last bound gives that
E
P
(R
P,Cn
) < E
P
_
log
P(x
n
1
)
Q
(m)
(x
n
1
)
_
+ 2
H(P
n
) +H(P
m
) + (n m)H
m
+
k
m
(k1)
2
log n + log K
m
+ 2.
6.2. Universal codes for certain classes of processes 489
Since
H(P
n
) H(P
m
) = H(X
n
1
) H(X
m
1
) =
n1
i=m
H(X
i+1
[X
i
1
) (n m)H,
the assertion of the theorem follows.
The corollary is immediate, noting for the second assertion that
Q(x
n
1
)
m
Q
(m)
(x
m
1
) implies
log
P(x
n
1
)
Q(x
n
1
)
log
P(x
n
1
)
Q
(m)
(x
n
1
)
log
m
.
Remark 6.3. The last inequality implies that for Markov chains of
any order m, the arithmetic code determined by Q =
m=0
m
Q
(m)
per
forms eectively as well as that determined by Q
(m)
, the coding process
tailored to the class of Markov chains of order m: the increase in point
wise redundancy is bounded by a constant (depending on m). Of course,
the situation is similar for other nite or countable mixtures of coding
processes. For example, taking a mixture of coding processes tailored
to subclasses of the Markov chains of order m corresponding to dier
ent context functions, the arithmetic code determined by this mixture
will satisfy the bound (6.17) whenever the true process belongs to one
of the subclasses with s possible values of context function. Such codes
are sometimes called twice universal. Their practicality depends on how
easily the conditional probabilities of the mixture process, needed for
arithmetic coding, can be calculated. This issue is not entered here, but
we note that for the case just mentioned (with a natural restriction on
the admissible context functions) the required conditional probabilities
can be calculated via a remarkably simple context weighting algo
rithm.
7
Redundancy bounds
In this Section, we address code performance for a class of processes
with respect to the expected redundancy criterion. We also show that
the universal codes constructed for certain classes in the previous Sec
tion are optimal within a constant, both for the maximum and expected
redundancy criteria.
As noted in the previous Section, the least possible worst case ex
pected redundancy R
Cn
, attainable for a given class T of processes by
prex codes C
n
: A
n
B
n
attaining this minimum is eec
tively an optimal coding distribution for nlength messages tailored to
the class T, in the sense that a Shannon code for Q
n
attains the least
possible worst case expected redundancy within 1 bit.
Next we discuss a remarkable relationship of the expression (7.1) to
the seemingly unrelated concepts of mutual information and channel
capacity. As process concepts play no role in this discussion, we shall
simply consider some set of probability distributions on A, and its
Idivergence radius, dened as the minimum for Q of sup
P
D(PQ).
490
7.1. Iradius and channel capacity 491
Later the results will be applied to A
n
and the set of marginal distri
butions on A
n
of the processes P T, in the role of A and .
7.1 Iradius and channel capacity
The Iradius of a set of distributions on A is the minimum, for
distributions Q on A, of sup
P
D(PQ) . If the minimum is attained
by a unique Q = Q
= P
(a), a A.
If the input is selected at random according a probability measure on
, the information that the output provides for the input is measured
by the mutual information
I() = H(Q
)
_
H(P
)(d),
where Q
= Q
(a) =
_
P
(a)(d), a A.
The supremum of the mutual information I() for all probability mea
sures on is the channel capacity. A measure
0
is a capacity achiev
ing distribution if I(
0
) = sup
I().
Lemma 7.1. For arbitrary distributions Q on A and on ,
_
D(P
Q).
492 Redundancy bounds
Proof. Both sides equal +if S(Q), the support of Q, does not contain
S(P
Q)(d) =
_ _
aA
P
(a) log
P
(a)
Q(a)
_
(d)
=
_ _
aA
P
(a) log
P
(a)
Q
(a)
_
(d) +
_ _
aA
P
(a) log
Q
(a)
Q(a)
_
(d).
Using the denition of Q
Q).
Theorem 7.1. For arbitrary distributions Q on A and on ,
sup
D(P
Q) I(),
with equality if and only if is a capacity achieving distribution and
Q = Q
.
Proof. The inequality follows immediately from Lemma 7.1, as does
the necessity of the stated condition of equality. To prove suciency,
suppose on the contrary that there is a capacity achieving distribution
0
such that D(P
0
Q
0
) > I(
0
), for some
0
.
Denote by
1
the point mass at
0
and set
t
= (1 t)
0
+ t
1
,
0 < t < 1. Then by the denition of I(),
I(
t
) = H(Q
t
) (1 t)
_
H(P
)
0
(d) tH(P
0
),
so that,
d
dt
I(
t
) =
d
dt
H(Q
t
) +
_
H(P
)
0
(d) H(P
0
).
Since Q
t
= (1 t)Q
0
+tP
0
, simple calculus gives that
d
dt
H(Q
t
) =
a
(Q
0
(a) P
0
(a)) log Q
t
(a).
It follows that
lim
t0
d
dt
I(
t
) = I(
0
) +D(P
0
Q
0
) > 0,
7.1. Iradius and channel capacity 493
contradicting the assumption that
0
is capacity achieving (which im
plies that I(
t
) I(
0
)). The proof of the theorem is complete.
Note that any set of distributions on A which is a Borel subset
of R
[A[
(with the natural identication of distributions with points in
R
[A[
) has a natural parametric representation, with = and
P
)
_
H(P)(dP), Q
=
_
P(dP). (7.2)
Lemma 7.2. For any closed set of distributions on A, there exists a
probability measure
0
concentrated on a nite subset of of size m
[A[ that maximizes I(). If a parametric set of distributions P
,
is closed, there exists a capacity achieving distribution
0
concentrated
on a nite subset of of size m [A[.
Proof. If is a closed (hence compact) subset of R
[A[
, the set of all
probability measures on is compact in the usual topology of weak
convergence, where
n
means that
_
(P)
n
(dP)
_
(P)(dP)
for every continuous function on . Since I() is continuous in that
topology, its maximum is attained.
Theorem 7.1 applied with the natural parametrization of gives
that if
= Q
satises D(PQ
) I(
) for
each . Since I(
) =
_
D(PQ
) = I(
) for
=
_
(dP)
belongs to the convex hull of the set of those P that satisfy
D(PQ) = I(
=
m
i=1
i
P
i
where the distributions
P
i
satisfy D(P
i
Q
) = I(
1
, . . . , P
m
of is the
image of one concentrated on
1
, . . . ,
m
.
Corollary 7.1. Any set of probability distributions on A has an
Icentroid, that is, a unique Q
, ,
the Iradius equals the channel capacity supI(), and Q
n
converges
to the Icentroid Q
whenever I(
n
) supI().
Proof: Let denote the closure of P
n
on with I(
n
) I(
0
).
Such
n
s can be obtained as follows. Take sequences of distributions
in P
i,n
P
i
, i = 1, . . . , m.
Let
n
be the measure concentrated on
1,n
, . . . ,
m,n
, giving the same
weight to
i,n
that
0
gives to P
i
.
Finally, we establish a lower bound to channel capacity, more ex
actly, to the mutual information I() for a particular choice of , that
will be our key tool to bounding worst case expected redundancy from
below. Given a parametric set P
, of distributions on A, a
mapping
: A is regarded as a good estimator of the parameter
7.1. Iradius and channel capacity 495
if the mean square error
E


2
=
xA
P
(x)
(x)
2
is small for each . We show that if a good estimator exists, the
channel capacity cannot be too small.
Theorem 7.3. If the parameter set R
k
has Lebesgue measure
0 < () < , and an estimator
: A exists with
E


2
for each ,
then for equal to the uniform distribution on ,
I()
k
2
log
k
2e
+ log ().
To prove this theorem, we need some standard facts from informa
tion theory, stated in the next two lemmas. The dierential entropy
of a random variable X with values in R
k
that has a density f(x), is
dened as
H(X) =
_
f(x) log f(x)dx;
thus H denotes entropy as before in the discrete case, and dierential
entropy in the continuous case. The conditional dierential entropy of
X given a random variable Y with values in a nite set A (more general
cases will not be needed below), is dened similarly as
H(X[Y ) =
aA
P(a)
_
_
f(x[a) log f(x[a)dx
_
,
where P(a) is the probability of Y = a, and f(x[a) is the conditional
density of X on the condition Y = a.
Lemma 7.3. For X and Y as above, I(X Y ) = H(X) H(X[Y ).
Moreover, if Z is a function of Y then H(X[Y ) H(X[Z) H(X).
496 Redundancy bounds
Proof. By the denition of mutual information of random variables,
one with values in a nite set and the other arbitrary, see Section 1,
I(X Y ) = H(Y ) H(Y [X) = H(P)
_
H(P([x))f(x)dx,
where P([x) denotes the conditional distribution of Y on the con
dition X = x. Substituting the formula for the latter, P(a[x) =
P(a)f(x[a)/f(x), into the above equation, the claimed identity
I(X Y ) =
_
f(x) log f(x)dx +
aA
P(a)
_
f(x[a) log f(x[a)dx
follows by simple algebra.
Next, if Z is a function of Y , for each possible value c of Z let A(c)
denote the set of possible values of Y when Z = c. Then the conditional
density of X on the condition Z = c is given by
g(x[c) =
aA(c)
P(a)f(x[a)
aA(c)
P(a)
,
and Jensens inequality for the concave function t log t yields that
aA(c)
P(a)(f(x[a) log f(x[a)) (
aA(c)
P(a))(g(x[c) log g(x[c)).
Hence, by integrating and summing for all possible c, the claim
H(X[Y ) H(X[Z) follows. Finally, H(X[Z) H(X) follows simi
larly.
Lemma 7.4. A kdimensional random variable V = (V
1
, . . . , V
k
) with
EV 
2
k
2
has maximum dierential entropy if V
1
, . . . , V
k
are inde
pendent and have Gaussian distribution with mean 0 and variance
2
,
and this maximum entropy is
k
2
log(2e
2
).
Proof. The integral analogue of the logsum inequality is
_
a(x) log
a(x)
b(x)
dx a log
a
b
, a =
_
a(x)dx, b =
_
b(x)dx,
valid for any nonnegative integrable functions on R
k
. Letting a(x) be
any kdimensional density for which EV 
2
k
2
, and b(x) be the
7.2. Optimality results 497
Gaussian density
i
(2
2
)
1/2
e
(x
2
i
/2
2
)
, this inequality gives
_
a(x) log a(x)dx
_
a(x)
_
(k/2) log(2
2
) +
(x
2
i
/2
2
) log e
_
dx 0.
Here
_
a(x)(
x
2
i
)dx k
2
by assumption, hence the assertion
_
a(x) log a(x)dx (k/2) log(2e
2
)
follows, with equality if a(x) = b(x).
Proof of Theorem 7.3. Let X be a random variable uniformly dis
tributed on , and let Y be the channel output corresponding to input
X, that is, a random variable with values in A whose conditional dis
tribution on the condition X = equals P
. Further, let Z =
(Y ).
Then, using Lemma 7.3,
I() = I(X Y ) = H(X) H(X[Y )
H(X) H(X[Z) = H(X) H(X Z[Z)
H(X) H(X Z). (7.3)
The hypothesis on the estimator
implies that
EX Z
2
= E(EX Z
2
[X) =
_
E


2
(d) .
Hence, by Lemma 7.4 applied with
2
= /k,
H(X Z)
k
2
log
2e
k
.
On account of the inequality (7.3), where H(X) = log (), this com
pletes the proof of the theorem.
7.2 Optimality results
Returning to the problem of least possible worst case expected
redundancy, it follows from Corollary 7.1 that for any class T of pro
cesses with alphabet A, there exists, for each n, a unique Q
n
attaining
the minimum in (7.1). As discussed before, this Icentroid of the set
P
n
: P T of the marginals on A
n
of the processes in T is eec
tively an optimal coding distribution for nlength messages, tailored
498 Redundancy bounds
to the class T. When T is a parametric class of processes, that is,
T = P
(a
n
1
) is a measurable function of for each n and a
n
1
A
n
,
Theorem 7.1 identies the Icentroid Q
n
as
Q
n
(x
n
1
) =
_
P
,n
(x
n
1
)
n
(d), x
n
1
A
n
where
n
is a capacity achieving distribution for the channel determined
by P
,n
, provided that a capacity achieving distribution exists;
a sucient condition for the latter is the closedness of the set P
,n
,
of the marginal distributions on A
n
, see Lemma 7.2.
Typically,
n
does depend on n, and no process exists of which Q
n
would be the marginal on A
n
for n = 1, 2, . . . (a similar inconvenience
occurred also in the context of Theorem 6.2). Still, for important pro
cess classes T = P
(d) given by
Q(x
n
1
) =
_
P
,n
(x
n
1
)(d), x
n
1
A
n
, n = 1, 2, . . . (7.4)
attain the minimum of sup
D(P
,n
Q
n
) within a constant. Then Q
is a nearly optimal coding process: the arithmetic code determined
by Q attains the least possible worst case expected redundancy for T,
within a constant. Typical examples are the coding processes tailored
to the classes of i.i.d. and Markov processes, treated in the previous
subsection. We now show that these are mixture processes as in (7.4).
Their near optimality will be proved later on.
First, let T be the class of i.i.d. processes with alphabet A =
1, . . . , k, parametrized by = (p
1
, . . . , p
k1
): p
i
0,
k1
i=1
p
i
1,
with
P
(x
n
1
) =
k
i=1
p
n
i
i
, n
i
= [1 t n: x
t
= i[;
here, for = (p
1
, . . . , p
k1
), p
k
= 1 (p
1
+. . . +p
k1
).
Let be the Dirichlet distribution on with parameters
i
> 0,
7.2. Optimality results 499
i = 1, . . . , k, whose density function, with the notation above, is
f
1
,...,
k
()
def
=
(
k
i=1
i
)
k
i=1
(
i
)
k
i=1
p
i
1
i
,
where (s) =
_
0
x
s1
e
x
dx. Then (7.4) gives
Q(x
n
1
) =
_
(x
n
1
)f
1
,...,
k
()d =
(
k
i=1
i
)
k
i=1
(
i
)
_
i=1
p
n
i
+
i
1
i
d
=
(
k
i=1
i
)
k
i=1
(
i
)
i=1
(n
i
+
i
)
(
k
i=1
(n
i
+
i
))
f
n
1
+
1
,...,n
k
+
k
() d
=
k
i=1
[(n
i
+
i
1)(n
i
+
i
2) . . .
i
]
(n +
k
i=1
i
1)(n +
k
i=1
i
2) . . . (
k
i=1
i
)
,
where the last equality follows since the integral of a Dirichlet density is
1, and the function satises the functional equation (s+1) = s(s).
In particular, if
1
= . . . =
k
=
1
2
, the mixture process Q =
_
P
(d)
is exactly the coding process tailored to the i.i.d. class T, see (6.5).
Next, let T the class of Markov chains with alphabet A = 1, . . . , k,
with initial distribution equal to the uniform distribution on A, say,
parametrized by = (p
ij
)
1ik,1jk1
: p
ij
0,
k1
j=1
p
ij
1:
P
(x
n
1
) =
1
[A[
k
i=1
k
j=1
p
n
ij
ij
, n
ij
= [1 t n 1: x
t
= i, x
t+1
= j[,
where, for = (p
ij
), p
ik
= 1(p
i1
+. . . +p
ik1
). Let be the Cartesian
product of k Dirichlet (
1
2
, . . . ,
1
2
) distributions, that is, a distribution
on under which the rows of the matrix (p
ij
) are independent and
500 Redundancy bounds
Dirichlet (
1
2
, . . . ,
1
2
) distributed. The previous result implies that the
corresponding mixture process Q =
_
P
n
: A
n
E
n

2
c()
n
, , n = 1, 2, . . . .
Then, for a suitable constant K,
R
n
k
2
log n K, n = 1, 2, . . . .
Moreover, if () < +, then to any > 0 there exists a constant K
such that for each n and distribution Q
n
on A
n
( : D(P
,n
Q
n
) <
k
2
log n K) < .
Proof: It suces to prove the second assertion. Fixing 0 < (),
take C so large that
t
= , c() > C has (
t
) /2. Then, for
arbitrary
1
with (
1
) , Theorem 7.3 applied to P
,n
:
1
t
with = C/n gives
I()
k
2
log
kn
2eC
+ log (
1
t
)
where is the uniform distribution on
1
t
.
Since here (
1
t
) /2, this and Theorem 7.1 applied to
P
,n
:
1
yield
sup
1
D(P
Q
n
) I()
k
2
log
kn
2eC
+ log
2
7.2. Optimality results 501
=
k
2
log n K; K =
k
2
log
2eC
k
+ log
2
,
whenever (
1
) . This proves that the set : D(P
,n
Q
n
) <
k
2
log n K cannot have Lebesgue measure , as claimed.
Corollary 7.2. For T as above, if the expected redundancy of a prex
code C
n
: n = 1, 2, . . . satises
E
P
(R
P,Cn
)
k
2
log n , P = P
,
0
for some subset
0
of then (
0
) = 0.
Proof. Note that E
P
(R
P,Cn
)
k
2
log n implies D(PQ
n
)
k
2
log n for the distributions Q
n
associated with C
n
by Q
n
(x
n
1
) =
c2
L(x
n
1
)
. Hence it suces to show that for no
0
with (
0
) > 0
can the latter limit relation hold for each P = P
with
0
.
Now, if such
0
existed, with (
0
) = 2, say, Theorem 7.4 applied
to
0
in the role of would give (
0
, D(P
,n
Q
n
)
k
2
log n
K) > , n = 1, 2, . . . , contradicting D(P
,n
Q
n
)
k
2
log n ,
0
.
Theorem 7.5. (i) For the class of i.i.d. processes with alphabet A =
1, . . . , k,
k 1
2
log n K
1
R
n
R
n
k 1
2
log n +K
2
,
where K
1
and K
2
are constants. The worst case maximum and expected
redundancies R
Cn
and R
Cn
of the arithmetic code determined by the
coding process Q given by (6.5) are the best possible for any prex
code, up to a constant.
(ii) For the class of mth order Markov chains with alphabet A =
1, . . . , k,
(k 1)k
m
2
log n K
1
R
n
R
n
(k 1)k
m
2
log n +K
2
with suitable constants K
1
and K
2
. The arithmetic code determined
by the coding process Q given by (6.14) is nearly optimal in the sense
of (i).
502 Redundancy bounds
Proof. (i) The class T of i.i.d. processes satises the hypothesis of
Theorem 7.4, with k replaced by k 1. Suitable estimators
n
are the
natural ones: for x
n
1
A
n
with empirical distribution
P = ( p
1
, . . . , p
k
),
set
n
(x
n
1
) = ( p
1
, . . . , p
k1
). Thus the lower bound to R
n
follows from
Theorem 7.4. Combining this with the bound in (6.6) completes the
proof.
(ii) To prove the lower bound to R
n
, consider the mth order Markov
chains with uniform initial distribution, say, restricting attention to the
irreducible ones. The role of is now played by the (k 1)k
m
tuple
of transition probabilities P(j[a
m
1
), a
m
1
A
m
, j = 1, . . . , k 1. It is
not hard to see that estimating P(j[a
m
1
) from x
m
1
A
n
by the ratio
n
a
m
1
j
/n
a
m
1
(with the notation in equation (6.14)) gives rise to estimators
n
of that satisfy the hypothesis of Theorem 7.4, with (k 1)k
m
in
the role of k. Then the claimed lower bound follows, and combining it
with the bound in (6.16) completes the proof.
Remark 7.1. Analogous results hold, with similar proofs, for any sub
class of the mth order Markov chains determined by a context function,
see Subsection 6.2.
8
Redundancy and the MDL principle
Further results about redundancy for processes are discussed in
this Section, with applications to statistical inference via the minimum
description length (MDL) principle.
As in the last Sections, the term code means either an ncode
C
n
: A
n
0, 1
, or a sequence of ncodes C
n
: n = 1, 2, . . .. Codes
C
n
: n = 1, 2, . . . determined by a coding process Q will play
a distinguished role. For convenience, we will use the term Qcode
for an ideal code determined by Q, with length function L(x
n
1
) =
log Q(x
n
1
), whose redundancy function relative to a process P is
R(x
n
1
) = log
P(x
n
1
)
Q(x
n
1
)
.
The results below stated for such ideal codes are equally valid for real
(Shannon or arithmetic) codes whose length and redundancy functions
dier from those of the ideal Qcodes by less than 2 bits.
Theorem 8.1. If P and Q are mutually singular probability mea
sures on A
A
m
T
m
such that the symmetric dierence of
A and
A
m
has
measure less than ; thus,
Q(
A
m
) +
_
A\
Am
ZdP < .
Since the denition of Z
n
implies
_
Am
Z
n
dP = Q(
A
m
) for n m,
Fatous lemma gives
_
Am
ZdP liminf
n
_
Am
Z
n
dP = Q(
A
m
) .
Combining these two bounds, we obtain
E(Z) =
_
Am
ZdP +
_
A\
Am
ZdP < .
Since > 0 was arbitrary, E(Z) = 0 follows.
8.1 Codes with sublinear redundancy growth
While by Theorem 8.1 the redundancy of a Qcode relative to a
process P typically goes to innity, the next theorem gives a sucient
condition for a sublinear growth of this redundancy, that is, for the per
letter redundancy to go to zero.
8.1. Codes with sublinear redundancy growth 505
For this, we need the concept of divergence rate, dened for pro
cesses P and Q by
D(PQ) = lim
n
1
n
D(P
n
Q
n
),
provided that the limit exists. The following lemma gives a sucient
condition for the existence of divergence rate, and a divergence analogue
of the entropy theorem. For ergodicity, and other concepts used below,
see the Appendix.
Lemma 8.1. Let P be an ergodic process and Q a Markov chain of
order m with D(P
m+1
Q
m+1
) < +. Then
1
n
log
P(x
n
1
)
Q(x
n
1
)
D(PQ)
= H(P)
x
m+1
1
A
m+1
P(x
m+1
1
) log Q(x
m+1
[ x
m
1
),
both Palmost surely and in L
1
(P), with Q(x
m+1
[ x
m
1
) denoting the
transition probabilities of the Markov chain Q.
Proof: Since Q is Markov of order m,
log
P(x
n
1
)
Q(x
n
1
)
= log P(x
n
1
)log Q(x
m
1
)
nm
i=1
log Q(x
m+i
[ x
m+i1
i
), n m;
here log Q(x
m
1
) is nite with Pprobability 1, and so is log Q(x
m+1
[
x
m
1
), since D(P
m+1
Q
m+1
) < +.
By the entropy theorem, and the ergodic theorem applied to
f(x
1
) = log Q(x
m+1
[ x
m
1
), we have
1
n
log P(x
n
1
) H(P)
1
n
log
nm
i=1
log Q(x
m+i
[ x
m+i1
i
) E
P
(log Q(x
m+1
[ x
m
1
)),
both Palmost surely and in L
1
(P). The lemma follows.
Theorem 8.2. Let P be an ergodic process, and let Q =
_
U
(d)
be a mixture of processes U
) < , if
t
.
506 Redundancy and the MDL principle
Then for the process P, both the pointwise redundancy per symbol
and the expected redundancy per symbol of the Qcode go to zero as
n , the former with probability 1.
Remark 8.1. Here Q need not be the mixture of a parametric class
of processes, that is, unlike in Section 7, the index set need not be
a subset of an Euclidean space. It may be any set, endowed with a
algebra such that U
(a
n
1
) is a measurable function of for each
a
n
1
A
n
, n = 1, 2, . . ., and is any probability measure on (, ). All
subsets of we consider are supposed to belong to .
Proof of Theorem 8.2. We rst prove for the pointwise redundancy per
symbol that
1
n
R(x
n
1
) =
1
n
log
P(x
n
1
)
Q(x
n
1
)
0 , Pa.s. (8.1)
To establish this, on account of Theorem 6.1, it suces to show that
for every > 0
limsup
n
1
n
R(x
n
1
) , Pa.s..
This will be established by showing that
2
n
Q(x
n
1
)
P(x
n
1
)
+, Pa.s.
Since
Q(x
n
1
) =
_
(x
n
1
)(d)
_
(x
n
1
)(d), (8.2)
we have
2
n
Q(x
n
1
)
P(x
n
1
)
_
2
n
U
(x
n
1
)
P(x
n
1
)
(d) =
_
2
n(
1
n
log
P(x
n
1
)
U
(x
n
1
)
)
(d).
If
t
, Lemma 8.1 implies
1
n
log
P(x
n
1
)
U
(x
n
1
)
D(PU
) <
for Palmost all x
1
A
1
, ) A
t
for which the last limit
8.1. Codes with sublinear redundancy growth 507
relation does not hold, has P measure 0, and consequently for
Palmost all x
1
A
1
, the integrand in the above lower bound
to 2
n
Q(x
n
1
)/P(x
n
1
) goes to innity for almost all
t
. Hence, by
Fatous lemma, the integral itself goes to +, completing the proof of
(8.1).
To prove that also the expected redundancy per symbol
1
n
E
P
(R(x
n
1
)) goes to zero, we have to show that
1
n
E
P
(log Q(x
n
1
)) H(P).
On account of the entropy theorem, the result (8.1) is equivalent to
1
n
log Q(x
n
1
) H(P) Pa.s.,
hence it suces to show that
1
n
log Q(x
n
1
) is uniformly bounded (Pa.s.).
Since for
t
the Markov chains U
of order m satisfy
D(PU
(x
m+1
[ x
m
1
) are
bounded below by some > 0 whenever P(x
m+1
1
) > 0, see the expres
sion of D in Lemma 8.1. This implies by (8.2) that Q(x
n
1
) is bounded
below by a constant times
n
, Pa.s. The proof of Theorem 8.2 is com
plete.
Example 8.1. Let Q =
m=0
m
Q
(m)
, where
0
,
1
, . . . are positive
numbers with sum 1, and Q
(m)
denotes the process dened by equation
(6.14) (in particular, Q
(0)
and Q
(1)
are dened by (6.5) and (6.13)). This
Q satises the hypothesis of Theorem 8.2, for each ergodic process P, on
account of the mixture representation of the processes Q
(m)
established
in Subsection 7.2. Indeed, the divergence rate formula in Lemma 8.1
implies that D(PU
m=0
m
Q
(m)
is weakly universal for the class of ergodic processes,
in the sense of (6.1), and also its pointwise redundancy per symbol goes
to zero Pa.s., for each ergodic process P.
Recall that the weak universality of this Qcode has already been
established in Subsection 6.2, even for the class of all stationary pro
cesses.
Example 8.2. Let U
D(PU
) = 0 . (8.3)
Then for arbitrary numbers
> 0 with
= 1, the process
Q =
a
n
1
A
n P
a
n
1
_
1
[A[
n
a
n
1
A
n H(P
a
n
1
).
As P
a
n
1
is concentrated on the cyclic shifts of a
n
1
, implying H(P
a
n
1
)
log n, and the output distribution [A[
n
a
n
1
A
n P
a
n
1
equals the uni
form distribution on A
n
, this establishes our claim. In particular, no
strongly universal codes exist for the class T, let alone for the larger
class of all ergodic processes.
Next we consider a simple construction of a new code from a
given (nite or) countable family of codes C
, , where C
=
C
n
: A
n
B
n
(x
n
1
), with chosen de
pending on x
n
1
, preambled by a code C() of the chosen . Here
C: B
(x
n
1
), where L
(x
n
1
) and L()
denote the length functions of the codes C
[L() +L
(x
n
1
)].
If the family C
, consists of Q
, is
eectively as good as a Qcode for a mixture Q of these processes,
namely its length function satises
log Q
(1)
(x
n
1
) L(x
n
1
) log Q
(2)
(x
n
1
) + log c
2
,
where
Q
(1)
= c
1
2
L()
Q
, Q
(2)
= c
2
2
2L()
Q
,
c
1
=
2
L()
1
, c
2
=
_
2
2L()
_
1
.
510 Redundancy and the MDL principle
Proof. The Q
code C
(x
n
1
) = log Q
(x
n
1
),
hence
L(x
n
1
) = min
L
[L() log Q
(x
n
1
)] = log max
L
2
L()
Q
(x
n
1
).
Since
Q
(1)
(x
n
1
)
2
L()
Q
(x
n
1
) max
L
2
L()
Q
(x
n
1
)
2
2L()
Q
(x
n
1
) =
Q
(2)
(x
n
1
)
c
2
,
where the rst and third inequalities are implied by Krafts inequality
2
L()
1, the assertion follows.
Recalling Examples 8.1 and 8.2, it follows by Lemma 8.2 that the
list of processes Q
(m)
, m = 0, 1, . . ., with Q
(m)
dened by equation
(6.14), as well as any list of Markov processes U
, with the
property (8.3), generates a code such that for every ergodic process
P, the redundancy per symbol goes to 0 Pa.s., and also the mean
redundancy per symbol goes to 0.
8.2 The minimum description length principle
The idea of the above construction of a new code from a given (nite
or countable) family of codes underlies also the minimum description
length (MDL) principle of statistical inference that we discuss next.
MDL principle. The statistical information in data is best ex
tracted when a possibly short description of the data is found. The
statistical model best tting to the data is the one that leads to the
shortest description, taking into account that the model itself must also
be described.
Formally, in order to select a statistical model that best ts the data
x
n
1
, from a list of models indexed with elements of a (nite or) count
able set , one associates with each candidate model a code C
, with
8.2. The minimum description length principle 511
length function L
(x
n
1
), and takes a code C: B
(x
n
1
) is minimum.
For a simple model stipulating that the data are coming from a
specied process Q
is a Q
(x
n
1
) = log Q
(x
n
1
). For a composite model stipulating
that the data are coming from a process in a certain class, the associated
code C
.
To relate the MDL to other statistical principles, suppose that the
candidate models are parametric classes T
= P
of pro
cesses, with ranging over a (nite or) countable set . Suppose rst
that the code C
is chosen as a Q
code with
Q
=
_
(d), (8.4)
where
(x
n
1
) = L()log Q
(x
n
1
)
is equivalent to Bayesian inference by maximizing the posterior proba
bility (conditional probability given the data x
n
1
) of , if one assigns to
each a prior probability proportional to 2
L()
, and regards
as
a prior probability distribution on
(x
n
1
).
Suppose next that the codes C
as above are chosen to be NML codes, see Theorem 6.2, with length
functions
L
(x
n
1
) = log NML
(x
n
1
) = log P
()
ML
(x
n
1
) + log
a
n
1
A
n
P
()
ML
(a
n
1
),
where
P
()
ML
(x
n
1
) = sup
(x
n
1
) .
Then the MDL principle requires minimizing
L() +L
(x
n
1
) = log P
()
ML
(x
n
1
) +R
n
()
512 Redundancy and the MDL principle
where
R
n
() = L() + log
a
n
1
A
n
P
()
ML
(a
n
1
) .
In statistical terminology, this is an instance of penalized maximum
likelihood methods, that utilize maximization of log P
()
ML
(x
n
1
) R
n
(),
where R
n
() is a suitable penalty term.
Remark 8.2. We note without proof that, under suitable regularity
conditions, L() + L
(x
n
1
) is asymptotically equal (as n ) to
log P
()
ML
(x
n
1
) +
1
2
k
, where k
(meaning that
is
a subset of positive Lebesgue measure of R
k
). When is nite, this
admits the conclusion that MDL is asymptotically equivalent to penal
ized maximum likelihood with the socalled BIC (Bayesian information
criterion) penalty term, R
n
() =
1
2
k
, the
unique minimizer of L() log Q
(x
n
1
) is
2
L()
Q
2
L()
Q
max
\
2
L()
Q
(x
n
1
) .
8.2. The minimum description length principle 513
The hypothesis implies that Q and Q
(x
n
1
) log Q(x
n
1
) + Q
a.s.
This and the previous inequality complete the proof.
Theorem 8.4. Let T
= P
of processes, let Q
, , be
mixture processes as in equation (8.4), supposed to be mutually singu
lar, and let L() be the length function of a prex code C: B
.
Then, with possible exceptional sets N
of
measure 0, if
the true process is a nonexceptional member of either class T
, say
P = Q
(x
n
1
) is
and
. This
condition is also sucient if all processes P
1
A
for
which there exist innitely many n with
L(
) log Q
(x
n
1
) inf
\
[L() log Q
(x
n
1
)]
has Q
. By the denition of Q
, see (8.4),
this implies that the above set has P
except
possibly for in a set N
of
measure 0.
As an application of Theorem 8.4, consider the estimation of the
order of a Markov chain, with alphabet A = 1, . . . , k. As in Ex
ample 8.1, denote by Q
(m)
the coding process tailored to the class of
Markov chains of order m. According to the MDL principle, given a
sample x
n
1
A
n
from a Markov chain P of unknown order m
, take
the minimizer m = m(x
n
1
) of L(m) log Q
(m)
(x
n
1
) as an estimate of
m
.
514 Redundancy and the MDL principle
Recall that Q
(m)
equals the mixture of mth order Markov chains with
uniform initial distribution, with respect to a probability distribution
which is mutually absolutely continuous with the Lebesgue measure on
the parameter set
m
, the subset of k
m
(k 1) dimensional Euclidean
space that represents all possible transition probability matrices of m
th order Markov chains. It is not hard to see that the processes Q
(m)
,
m = 0, 1 . . . are mutually singular, hence Theorem 8.4 implies that
m(x
n
1
) = m
, or by the minimizer of
log P
(m)
ML
(x
n
1
) +
1
2
k
m
(k 1) log n .
Nevertheless, the conjecture is false, and we conclude this subsection
by a counterexample. It is unknown whether other counterexamples
also exist.
Example 8.4. Let P be the i.i.d. process with uniform distribution,
that is,
P(x
n
1
) = k
n
, x
n
1
A
n
, A = 1, . . . , k.
8.2. The minimum description length principle 515
Then m
. Here
P
(0)
ML
(x
n
1
) = sup
p
1
,...,p
k
i=1
p
n
i
i
=
k
i=1
_
n
i
n
_
n
i
is the largest probability given to x
n
1
by i.i.d. processes, with n
i
denoting
the number of times the symbol i occurs in x
n
1
, and the stated equality
holds since P
(0)
ML
(x
n
1
)/Q
(0)
(x
n
1
) is bounded both above and below by a
constant times n
k1
2
, see Remark 6.2, after Theorem 6.3.
Next, since P is i.i.d. with uniform distribution, the numbers n
i
above satisfy, as n ,
n
i
=
n
k
+O(
_
nlog log n ), eventually a.s.,
by the law of iterated logarithm. This implies
log P
(0)
ML
(x
n
1
) =
k
i=1
n
i
log
_
n
n
i
_
= nlog k +O(log log n),
since
log
n
n
i
= log k + log
_
1 +
n
kn
i
1
_
=
= log k +
_
n
kn
i
1
_
log e +O
_
n
kn
i
1
_
2
.
It follows that the left hand side of equation (8.6) equals nlog k +
k1
2
log n +O(log log n), eventually almost surely as n .
516 Redundancy and the MDL principle
Turning to the right hand side of equation (8.6), observe that if no
mblock a
m
1
A
m
occurs in x
n1
1
more than once then Q
(m)
(x
n
1
) = k
n
.
Indeed, then n
a
m
1
is nonzero for exactly n m blocks a
m
1
A
m
in the
denition (6.14) of Q
(m)
; for these, n
a
m
1
= 1 and there is exactly one
j A with n
a
m
1
j
nonzero, necessarily with n
a
m
1
j
= 1. Hence equation
(6.14) gives Q
(m)
(x
n
1
) = k
n
as claimed.
The probability that there is an mblock occurring in x
n1
1
more
than once is less than n
2
k
m
. To see this, note that for any 1 j <
< n m + 1, the conditional probability of x
j+m1
j
= x
+m1
, when
x
1
1
A
1
is xed, is k
m
, as for exactly one of the k
m
equiprobable
choices of x
+m1
A
m
will x
+m1
= x
j+m1
j
hold. Hence also the
unconditional probability of this event is k
m
, and the claim follows. In
particular, taking m
n
=
4
log k
log n, the probability that some m
n
block
occurs in x
n1
1
more than once is less than n
2
. By BorelCantelli, and
the previous observation, it follows that
log Q
(mn)
(x
n
1
) = nlog k, eventually a.s.
This, and the assumption L(m) = o(m), imply that the right hand side
of (8.6) is nlog k +o(log n), eventually almost surely, completing the
proof of equation (8.6).
APPENDIX
A
Summary of process concepts
A (stochastic) process is frequently dened as a sequence of random
variables X
n
; unless stated otherwise, we assume that each X
n
takes
values in a xed nite set A called the alphabet. The nfold joint dis
tribution of the process is the distribution P
n
on A
n
dened by the
formula
P
n
(x
n
1
) = Prob(X
i
= x
i
, 1 i n), x
n
1
A
n
.
For these distributions, the consistency conditions
P
n
(x
n
1
) =
aA
P
n+1
(x
n
1
a)
must hold. The process X
n
, indeed, any sequence of distributions
P
n
on A
n
, n = 1, 2, . . . that satises the consistency conditions, deter
mines a unique Borel probability measure P on the set A
of innite
sequences drawn from A such that each cylinder set [a
n
1
] = x
1
: x
n
1
=
a
n
1
has Pmeasure P
n
(a
n
1
); a Borel probability measure on A
is a
probability measure dened on the algebra T of Borel subsets of
A
. The probabilities P
n
(a
n
1
) = P([a
n
1
]) will be usually denoted briey
by P(a
n
1
).
A sequence of random variables X
n
whose ndimensional joint dis
tributions equal the ndimensional marginals P
n
of P, will be referred
to as a representation of the process P. Such a representation always
exists, for example the Kolmogorov representation, with X
n
dened on
the probability space (A
, T, P) by X
n
(x
1
) = x
i
, i = 1, 2, . . ..
A process P is stationary if P is invariant under the shift T, the
transformation on A
1
= x
2
. Thus P is
stationary if and only if P(T
1
A) = P(A), A T.
The entropy rate of a process P is dened as
H(P) = lim
n
1
n
H(X
1
, . . . , X
n
),
provided that the limit exists, where X
n
is a representation of the
process P. A stationary process P has entropy rate
H(P) = lim
n
H(X
n
[X
1
, . . . , X
n1
);
here the limit exists since stationarity implies that
H(X
n
[X
1
, . . . , X
n1
) = H(X
n+1
[X
2
, . . . , X
n
) H(X
n+1
[X
1
, . . . , X
n
),
and the claimed equality follows by the additivity of entropy,
H(X
1
, . . . , X
n
) = H(X
1
) +
n
i=2
H(X
i
[X
1
, . . . , X
i1
).
If P
(a
n
1
) = P
([a
n
1
])
is a measurable function of for each a
n
1
A
n
, n = 1, 2, . . . , the mixture
of the processes P
(a
n
1
)(d), a
n
1
A
n
, n = 1, 2, . . . .
520 Summary of process concepts
A process P is called ergodic if it is stationary and, in addition, no
nontrivial shiftinvariant sets exist (that is, if A T, T
1
A = A,
then P(A) = 0 or 1), or equivalently, P cannot be represented as the
mixture P = P
1
+ (1 )P
2
of two stationary processes P
1
,= P
2
(with 0 < < 1). Each stationary process is representable as a mixture
of ergodic processes (by the socalled ergodic decomposition theorem).
Other key facts about ergodic processes, needed in Section 8, are the
following:
Ergodic theorem. For an ergodic process P, and Pintegrable func
tion f on A
,
1
n
n
i=1
f(x
i
)
_
fdP,
both Palmost surely and in L
1
(P).
Entropy theorem. (ShannonMcMillanBreiman theorem) For an
ergodic process P,
1
n
log P(x
n
1
) H(P),
both Palmost surely and in L
1
(P).
For an ergodic process P, almost all innite sequences x
1
A
P(a
k
1
[x
n
1
) =
1
n k + 1
[i : x
i+k
i+1
= a
k
1
, 0 i n k[
of kblocks a
k
1
A
k
in x
n
1
approach the true probabilities P(a
k
1
) as
n , for each k 1 and a
k
1
A
k
. This follows applying the ergodic
theorem to the indicator functions of the cylinder sets [a
k
1
] in the role
of f. Finally, we note that also conversely, if Palmost all x
1
A