Professional Documents
Culture Documents
Model
{Xt X , t I}
Discrete sources
Both X and I are discrete.
Continuous sources
Discrete-time continuous sources
X is continuous; I is discrete.
Waveform sources
Both X and I are continuous.
Example 5.1
1
of size = m ( is called the quantization step-size) such that:
i i1 i
qm (X) = , if X < ,
m m m
for 1 i m.
Then the entropy of the quantized random variable qm(X) is given by
m
1 1
H(qm (X)) = log2 = log2 m (in bits).
i=1
m m
Since the entropy H(qm (X)) of the quantized version of X is a lower bound
to the entropy of X (as qm (X) is a function of X) and satises in the limit
lim H(qm (X)) = lim log2 m = ,
m m
Proof:
Therefore, given any > 0, there exists N such that for all n > N ,
b
f (x) log f (x)dx h (n)
(X) < .
X 2 X
a
Information content of continuous sources I: 5-8
where the pis are the probabilities of the dierent values of qn (X).
Information content of continuous sources I: 5-9
yielding that
b
limn [H(qn (X)) n] = a fX (x) log2 fX (x)dx. 2
Information content of continuous sources I: 5-10
Lemma 5.2 actually holds not limited for support [a, b) but for any support SX .
Theorem 5.3 [38, Theorem 1](Due to Renyi) For any real-valued random vari-
able with pdf fX , if
j
pi log2 pi
i=1
is nite, where the pis are the probabilities of the dierent values of uniformly
quantized qn (X) over support SX , then
lim [H(qn (X)) n] = fX (x) log2 fX (x)dx
n SX
This suggest that SX fX (x) log2 fX (x)dx could be a good information measure
for continuous-alphabet sources.
Dierential entropy I: 5-11
Example 5.5 A continuous random variable X with support SX = [0, 1) and pdf
fX (x) = 2x for x SX has dierential entropy equal to
1 1
x (log2 e 2 log2(2x))
2
2x log2(2x)dx =
0 2 0
1
= log2(2) = 0.278652 bits.
2 ln 2
Dierential entropy I: 5-12
Note that for a Gaussian random variable, its dierential entropy is only a
function of its variance 2 (it is independent from its mean ).
This is similar to the dierential entropy of a uniform random variable, which
only depends on dierence (b a) but not the mean (a + b)/2.
Properties of dierential entropy I: 5-16
This justies using identical notations for both I(; ) and D( ) as opposed to
the discerning notations of H() for entropy and h() for dierential entropy.
Properties of dierential entropy I: 5-20
Lemma 5.14 The following properties hold for the information measures of con-
tinuous systems.
1. Non-negativity of divergence: Let X and Y be two continuous random
variables with marginal pdfs fX and fY , respectively, such that their supports
satisfy SX SY R. Then
D(fX fY ) 0
with equality i fX (x) = fY (x) for all x SX except in a set of fX -measure
zero (i.e., X = Y almost surely).
2. Non-negativity of mutual information: For any two continuous random
variables X and Y ,
I(X; Y ) 0
with equality i X and Y are independent.
3. Conditioning never increases dierential entropy: For any two con-
tinuous random variables X and Y with joint pdf fX,Y and well-dened con-
ditional pdf fX|Y ,
h(X|Y ) h(X)
with equality i X and Y are independent.
Properties of dierential entropy I: 5-21
with equality i all the Xis are independent from each other.
8. Invariance of dierential entropy under translation: For continuous
random variables X and Y with joint pdf fX,Y and well-dened conditional
pdf fX|Y ,
h(X + c) = h(X) for any constant c R, and h(X + Y |Y ) = h(X|Y ).
The results also generalize in the multivariate case: for two continuous random
vectors X n = (X1, X2, , Xn) and Y n = (Y1, Y2, , Yn) with joint pdf
fX n,Y n and well-dened conditional pdf fX n|Y n ,
h(X n + cn ) = h(X n)
for any constant n-tuple cn = (c1, c2, , cn) Rn, and
h(X n + Y n|Y n ) = h(X n|Y n),
where the addition of two n-tuples is performed component-wise.
Properties of dierential entropy I: 5-23
10. Joint dierential entropy under linear mapping: Consider the ran-
dom (column) vector X = (X1, X2, , Xn)T with joint pdf fX n , where T
denotes transposition, and let Y = (Y1, Y2, , Yn)T be a random (column)
vector obtained from the linear transformation Y = AX, where A is an in-
vertible (non-singular) n n real-valued matrix. Then
h(Y ) = h(Y1, Y2, , Yn) = h(X1, X2, , Xn) + log2 |det(A)|,
where det(A) is the determinant of the square matrix A.
Properties of dierential entropy I: 5-24
Proof:
Without loss of generality we assume that X has a zero mean vector since its
dierential entropy is invariant under translation by Property 8 of Lemma 5.14:
h(X) = h(X );
so we assume that = 0.
Properties of dierential entropy I: 5-26
Now invoking Property 10 of Lemma 5.14 and noting that |det(A)| = 1 yield
that
h(Y1, Y2, , Yn) = h(X1, X2, , Xn) + log2 |det(A)| = h(X1, X2, , Xn).
=0
An important fact: Among all real-valued size-n random vectors (of sup-
port Rn) with identical mean vector and covariance matrix, the Gaussian
random vector has the largest dierential entropy.
The proof of this fact requires the following inequality.
with equality i K is a diagonal matrix, where Ki,i are the diagonal entries of K.
Properties of dierential entropy I: 5-30
Proof: We will present the proof in two parts: the scalar or univariate case, and
the multivariate case.
(i) Scalar case (n = 1): For a real-valued random variable with support SX = R,
mean and variance 2, let us show that
1
h(X) log2 2e 2 , (5.2.12)
2
with equality i X N (, 2).
Properties of dierential entropy I: 5-31
(ii). Multivariate case (n > 1): As in the proof of Theorem 5.18, we can use an
orthogonal square matrix A (i.e., satisfying AT = A1 and hence |det(A)| = 1)
such that AKX AT is diagonal. Therefore, the random vector generated by the
linear map
Z = AX
will have a covariance matrix given by KZ = AKX AT and hence have uncorrelated
(but not necessarily independent) components.
Properties of dierential entropy I: 5-33
Finally, equality is achieved in both (5.2.14) and (5.2.15) i the random variables
Z1 , Z2, . . ., Zn are
Gaussian and independent from each other, or equivalently i
X Nn , KX . 2
Observation 5.21 The following two results can also be shown (the proof is left
as an exercise):
1. Among all continuous random variables admitting a pdf with support the in-
terval (a, b), where b > a are real numbers, the uniformly distributed random
variable maximizes dierential entropy.
2. Among all continuous random variables admitting a pdf with support the in-
terval [0, ) and nite mean , the exponential distribution with parameter
(or rate parameter) = 1/ maximizes dierential entropy.
A systematic approach to nding distributions that maximize dierential entropy
subject to various support and moments constraints can be found in [13, 52].
AEP for continuous memoryless sources I: 5-35
The extension of AEP theorem from discrete cases to continuous cases is not
based on number counting (which is always innity for continuous sources),
but on volume measuring.
Theorem 5.22 (AEP for continuous memoryless sources) Let {Xi} i=1
be a continuous memoryless source (i.e., an innite sequence of continuous i.i.d.
random variables) with pdf fX () and dierential entropy h(X). Then
1
log fX (X1, . . . , Xn) E[ log2 fX (X)] = h(X) in probability.
n
Proof: The proof is an immediate result of the law of large numbers (e.g., see
Theorem 3.3). 2
AEP for continuous memoryless sources I: 5-36
Denition 5.23 (Typical set) For > 0 and any n given, dene the typical
set for the above continuous source as
1
Fn() xn Rn : log2 fX (X1, . . . , Xn) h(X) < .
n
Average cost constraint (t, P ) on any input n-tuple xn = (x1, x2, , xn)
transmitted over the channel by requiring that
1
n
t(xi) P, (5.4.2)
n i=1
where t() is a given non-negative real-valued function describing the cost for
transmitting an input symbol, and P is a given positive number representing
the maximal average amount of available resources per input symbol.
Proof: Fix P1 > 0 and P2 > 0. Then since C(P ) is nite for any P > 0, then by
the 3rd property in Property A.4, there exist two input distributions FX1 and FX2
such that for all > 0,
I(Xi; Yi) C(Pi) (5.4.4)
and
E[t(Xi)] Pi (5.4.5)
where Xi denotes the input with distribution FXi and Yi is the corresponding
channel output for i = 1, 2.
We do not know whether the capacity is achievable or not at the moment!
Capacity for discrete memoryless Gaussian channel I: 5-40
then the codes probability of error Pe(Cn ) is bounded away from zero for all
n suciently large.
Capacity for discrete memoryless Gaussian channel I: 5-46
Proof of the forward part: The theorem holds trivially when C(P ) = 0
because we can choose Mn = 1 for every n and have Pe(Cn) = 0. Hence, we
assume without loss of generality C(P ) > 0.
Step 0:
Take a positive satisfying < min{2, C(P )}.
Pick > 0 small enough such that 2[C(P ) C(P )] < , where the
existence of such is assured by the strictly increasing property of C(P ).
Hence, we have
C(P ) > C(P ) > 0.
2
Choose Mn to satisfy
1
C(P ) > log2 Mn > C(P ) ,
2 n
for which the choice should exist for all suciently large n.
Take = /8.
Let FX be the distribution that achieves C(P ), where C(P ) is given
by (5.4.13). In this case, FX is the Gaussian distribution with mean zero
and variance P and admits a pdf fX . Hence, E[X 2] P and
I(X; Y ) = C(P ).
Capacity for discrete memoryless Gaussian channel I: 5-47
where fX nY n (xn, y n ) is the joint input-output pdf realized when the mem-
oryless Gaussian channel (with n-fold transition pdf
n
fY n|X n (y n |xn) = fY |X (yi |xi))
i=1
where
Fn (|xn) {y n Y n : (xn, y n ) Fn()} .
Capacity for discrete memoryless Gaussian channel I: 5-51
Note that the additional term PX n (E0) in (5.4.15) is to cope with the errors
due to all-zero codeword replacement, which will be less than for all
suciently large n by the law of large numbers.
Finally, by carrying out a similar procedure as in the proof of the channel
coding theorem for discrete channels (cf. Theorem 4.10), we obtain:
E[Pe(C n)] PX n (E0) + PX n,Y n (Fnc ())
+Mn 2n(h(X,Y )+)2n(h(X)) 2n(h(Y ))
PX n (E0) + PX n,Y n (Fnc ()) + 2n(C(P )4) 2n(I(X;Y )3)
= PX n (E0) + PX n,Y n (Fnc ()) + 2n .
Accordingly, we can make the average probability of error, E[Pe(C n)], less
than 3 = 3/8 < 3/4 < for all suciently large n. 2
Capacity for discrete memoryless Gaussian channel I: 5-53
Proof of the converse part: Consider an (n, Mn) block data transmission
code satisfying the power constraint
1 2
n
c P
n i=1 i
n
sup
%
I(Xj ; Yj ) (by Theorem 2.21)
n
FX n : (1/n) i=1 E[Xi ]P
2
j=1
n
= sup %n
sup I(Xj ; Yj )
(P1 ,P2 ,...,Pn ) : (1/n) i=1 Pi =P FX n : ( i) E[Xi2 ]Pi j=1
n
sup %n
sup I(Xj ; Yj )
(P1 ,P2 ,...,Pn ) : (1/n) i=1 Pi =P j=1 FX n : ( i) E[Xi2 ]Pi
n
= sup %n
sup I(Xj ; Yj )
(P1 ,P2 ,...,Pn ) : (1/n) i=1 Pi =P j=1 FXj : E[Xj2 ]Pj
n
n
1
= sup %n
C(Pj ) = sup %n
n C(Pj )
(P1 ,P2 ,...,Pn ):(1/n) i=1 Pi =P j=1 (P1 ,P2 ,...,Pn ):(1/n) i=1 Pi =P j=1
n
1
n
sup % nC Pj (by concavity of C(P ))
n
(P1 ,P2 ,...,Pn ):(1/n) i=1 Pi =P n j=1
= nC(P ).
Capacity for discrete memoryless Gaussian channel I: 5-55
So if
1
lim inf log2 Mn > C(P ),
n n
then there exists > 0 and an integer N such that for n N ,
1
log Mn > C(P ) + .
n 2
Hence, for n N0 max{N, 2/},
C(P ) 1
Pe(Cn) 1 .
C(P ) + n(C(P ) + ) 2(C(P ) + )
2
Capacity for discrete memoryless Gaussian channel I: 5-57
Then we have
I(fXg , fY |X ) I(fXg , fYg |Xg )
fZ (y x)
= fXg (x)fZ (y x) log2 dydx
X Y fY (y)
fZg (y x)
fXg (x)fZg (y x) log2 dydx
f (y)
X Y
Yg
fZ (y x)
= fXg (x)fZ (y x) log2 dydx
X Y f Y (y)
fZg (y x)
fXg (x)fZ (y x) log2 dydx
X Y fYg (y)
fZ (y x)fYg (y)
= fXg (x)fZ (y x) log2 dydx
X Y fZg (y x)f Y (y)
fZg (y x)fY (y)
fXg (x)fZ (y x)(log2 e) 1 dydx
X Y fZ (y x)fYg (y)
fY (y)
= (log2 e) 1 fXg (x)fZg (y x)dx dy = 0,
f
Y Yg (y) X
Therefore,
1 P
log2 1 + 2 = sup I(FX , fYg |Xg )
2 FX : E[X 2 ]P
= I(fX g , fYg |Xg )
I(fX g , fY |X )
sup I(FX , fY |X )
FX : E[X 2 ]P
= C(P ).
2
The water-lling principle I: 5-60
P = P1 + P2 + P4
P2
P1 32
P4
22
12 42
The water-lling principle I: 5-61
Proof:
By denition,
C(P ) = sup I(X k ; Y k ).
%k
FX k : i=1 E[Xk ]P
2
Since the noise random variables Z1, . . . , Zk are independent from each other,
I(X k ; Y k ) =h(Y k ) h(Y k |X k )
=h(Y k ) h(Z k + X k |X k )
=h(Y k ) h(Z k |X k )
=h(Y k ) h(Z k )
k
= h(Y )
k
h(Zi)
i=1
k
k
h(Yi) h(Zi)
i=1 i=1
k
1 Pi
log 1 + 2 .
i=1
2 2 i
Equalities hold above if all the Xi inputs are independent of each other with
%k
each input Xi N (0, Pi) such that i=1 Pi = P .
The water-lling principle I: 5-62
Thus the problem is reduced to nding the power allotment that maximizes the
%
overall capacity subject to the equality constraint ki=1 Pi = P and inequality
constraints Pi 0, i = 1, . . . , k.
By using the Lagrange multiplier technique and verifying the KKT condition
(see Example B.22 in Appendix B.10), the maximizer (P1, . . . , Pk ) of
k * k +$
1 Pi
k
max log2 1 + 2 + i P i Pi P
i=1
2 i i=1 i=1
can be found by taking the derivative of the above equation (with respect to
Pi) and setting it to zero, which yields
1 1
+ = 0, if Pi > 0;
2 ln(2) Pi + i2
i = 1 1
+ 0, if Pi = 0.
2 ln(2) Pi + i2
Hence,
Pi = i2, if Pi > 0;
(equivalently, Pi = max{0, i2}),
Pi i2, if Pi = 0,
%k
where log2 e/(2) is chosen to satisfy i=1 Pi = P. 2
The water-lling principle I: 5-63
Observation 5.35
Although it seems reasonable to allocate more power to less noisy channels for
the optimization of the channel capacity according to the water-lling principle,
the Gaussian inputs required for achieving channel capacity do not t the
digital communication system in practice.
One may wonder what an optimal power allocation scheme will be when the
channel inputs are dictated to be practically discrete in values, such as
binary phase-shift keying (BPSK), quadrature phase-shift key-
ing (QPSK), or 16 quadrature-amplitude modulation (16-QAM).
Surprisingly, a dierent procedure from the water-lling principle results under
certain conditions.
Indeed, the optimal power allocation for parallel AWGN channels with inputs
constrained to be discrete is established, resulting in a new graphical power
allocation interpretation called the mercury/water-lling principle [31].
This research concludes that when the total transmission power P is small,
the strategy that maximizes the capacity follows approximately the equal
signal-to-noise ratio (SNR) principle.
The water-lling principle I: 5-64
Very recently, it was found that when the additive noise is no longer Gaussian,
the mercury adjustment fails to interpret the optimal power allocation scheme.
For this reason, the graphical interpretation of the optimal power allocation
principle is simply named two-phase water-lling principle in [48].
We end this observation by emphasizing that the true measure for a digital
communication system is of no doubt the eective transmission rate subject to
an acceptably good transmission error (e.g., an overall bit error rate 105).
In order to make the analysis tractable, researchers adopt the channel capac-
ity as the design criterion instead, in the hope of obtaining a simple reference
scheme for practical systems. Further study is thus required to examine the fea-
sibility of such an approach (even though the water-lling principle is certainly
a theoretically interesting and theoretically beautiful result).
Capacity of correlated parallel Gaussian channels I: 5-65
Proof:
In correlated parallel Gaussian channels, the input power constraint becomes
k
E[Xi2] = tr(KX ) P,
i=1
Since in each channel, the input and noise variables are independent from each
other, we have
I(X k ; Y k ) = h(Y k ) h(Y k |X k )
= h(Y k ) h(Z k + X k |X k )
= h(Y k ) h(Z k |X k )
= h(Y k ) h(Z k ).
Since h(Z k ) is not determined by the input, determining the systems capacity
reduces to maximizing h(Y k ) over all possible inputs (X1, . . . , Xk ) satisfying
the power constraint.
Now observe that the covariance matrix of Y k is equal to
KY = KX + KZ ,
which implies by Theorem 5.20 that the dierential entropy of Y k is upper
bounded by
1
h(Y ) log2 (2e)k det(KX + KZ ) ,
k
2
k
with equality i Y Gaussian. It remains to nd out whether we can nd
inputs (X1, . . . , Xk ) satisfying the power constraint which achieve the above
upper bound and maximize it.
Capacity of correlated parallel Gaussian channels I: 5-67
where i is the component of matrix locating at ith row and ith column,
which is exactly the i-th eigenvalue of KZ .
Thus, the maximum value of det(B + ) under tr(B) P is realized by a
diagonal matrix B (to achieve equality in Hadamards inequality) with
k
Bii = P and each Bii 0.
i=1
Capacity of correlated parallel Gaussian channels I: 5-69
Thus,
1 1
C(P ) = log2 (2e) det(B + ) log2 (2e)k det()
k
2 2
1k
Bii
= log2 1 + .
i=1
2 i
Z(t)
X(t) ? Y (t)
- + - H(f ) -
Waveform channel
The output waveform is given by
Y (t) = (X(t) + Z(t)) h(t), t 0,
where represents the convolution operation.1
1
Rrecall that the convolution between two signals a(t) and b(t) is defined as a(t) b(t) = a( )b(t )d .
Capacity of band-limited white Gaussian channel I: 5-73
and bandwidth W cycles per second or Hertz (Hz); i.e., its spectrum or
Fourier transform +
X(f ) F[X(t)] = X(t)ej2f tdt = 0
for all frequencies |f | > W , where j = 1 is the imaginary unit number.
Z(t) is the noise waveform of a zero-mean stationary white Gaussian process
with power spectral density N0 /2; i.e., its power spectral density PSDZ (f ),
which is the Fourier transform of the process covariance (equivalently, correla-
tion) function KZ ( ) E[Z(s)Z(s + )], s, R, is given by
+
N0
PSDZ (f ) = F[KZ (t)] = KZ (t)ej2f tdt = f.
2
h(t) is the impulse response of an ideal bandpass lter with cuto frequencies
at W Hz:
1 if W f W ,
H(f ) = F[(h(t)] =
0 otherwise.
Capacity of band-limited white Gaussian channel I: 5-74
Example 5.39 (Telephone line channel) Suppose telephone signals are band-
limited to 4 KHz. Given an SNR of 40 decibels (dB) i.e., 10 log10 NP0W = 40 dB
then from (5.8.2), we calculate that the capacity of the telephone line channel
(when modeled via the band-limited white Gaussian channel) is given by
4000 log2(1 + 10000) = 53151.4 bits/second = 51.906 Kbits/second.
Notes
We cannot use sampling theorem in this case (as we did previously) because
X(t), Y (t) and Z(t) are not band-limited but actually time-limited to
[0, T ).
Z(t) is no longer assumed white (so the noise samples are not i.i.d.)
Capacity of time-limited Gaussian channel I: 5-81
Lemma 5.42 Any real-valued function v(t) dened over [0, T ) can be decomposed
into
v(t) = vii (t),
i=1
where the real-valued functions{i(t)}
i=1 are any orthonormal set (which can span
the space with respect to v(t)) of functions on [0, T ), namely
T
1, if i = j;
i (t)j (t)dt =
0 0, if i = j,
and T
vi = i (t)v(t)dt.
0
Capacity of time-limited Gaussian channel I: 5-82
Notes
Sampling theorem employs sinc function as the orthonormal set to span the
space of bandlimited signals.
The sinc orthonormal set however cannot be used for time-limited signals.
We therefore have to introduce the Karhunen-Loeve expansion.
Capacity of time-limited Gaussian channel I: 5-83
Proof:
T T
E[Zi Zj ] = E i(t)Z(t)dt j (s)Z(s)ds
T 0 T 0
= i (t)j (s)E[Z(t)Z(s)]dtds
0 T 0 T
= i (t)j (s)KZ (s t)dtds
0 T 0 T
= i(s) KZ (s t)j (t)dt ds
0 T 0
= i(s)(j j (s))ds
0
i, if i = j;
=
0, if i = j.
2
Capacity of time-limited Gaussian channel I: 5-85
Yi = Xi + Zi , i = 1, 2, 3, . . .
T
Xi = 0 X(t)i(t)dt,
T
where Y i = 0 Y (t)i (t)dt,
- .
T
Zi = 2
0 Z(t)i (t)dt i=1 are independent Gaussian with E[Zi ] = i .
%
Since Z(t) = i=1 Zi i (t) is assumed stationary, we have
KZ ( ) = E [Z(t)Z(t + )]
= E Zi i (t) Zk k (t + )
i=1 k=1
= E [ZiZk ] i (t)k (t + )
i=1 k=1
= i i(t)i(t + ).
i=1
Capacity of time-limited Gaussian channel I: 5-86
Thus,
KZ (0) = E[Z 2 (t)] = i2i (t),
i=1
which implies
T T
1 1 1
E[Z 2 (t)] = E[Z 2(t)]dt = i 2i (t)dt = i
T 0 T i=1 0 T i=1
Consequently,
max I(X n ; Y n )
X n :tr(KX )nP
1 1
= log2 [(2e)ndet(KX + KZ )] log2 [(2e)ndet(KZ )]
2 2
1
n
Pi
= log2 1 +
i=1
2 i
KX ( ) = E [X(t)X(t + )]
= E Xii (t) Xk k (t + )
i=1 k=1
= E [XiXk ] i (t)k (t + )
i=1 k=1
= Pii (t)i(t + ).
i=1
Capacity of time-limited Gaussian channel I: 5-88
Thus,
KX (0) = E[X 2(t)] = Pi2i (t),
i=1
which implies
T T
1 1 1
P E[X 2(t)] = E[X 2(t)]dt = Pi 2i (t)dt = Pi .
T 0 T i=1 0 T i=1
Key Notes I: 5-89