You are on page 1of 8

# ENEE 627: INFORMATION THEORY

Spring 2016

Problem Set 1

Narayan

Posted: February 5, 2016, Due: February 16, 2016

Problem 1 (Axiomatic derivation of entropy)
Cover-Thomas, problem 2.46.

Problem 2
Show that a rv with values in a countable set can have infinite entropy.

Problem 3
Let X be a rv taking values in the (finite) set {1, . . . , M } with pmf PX such that
PX (1) ≤ . . . ≤ PX (M ).
Show that
[3a]
H(X) ≥ − log PX (M );
[3b]
H(X) ≥ −PX (M ) log PX (M ) − (1 − PX (M )) log(1 − PX (M )).

Problem 4
The interpretation of entropy as a measure of uncertainty of a rv suggests that “more
uniform” pmfs have larger entropy. For two pmfs P and Q on a finite set X = {1, . . . , r},
we say that P is more uniform than Q, denoted P > Q, if for the nonincreasing ordering
P
P
of probabilities p1 ≥ p2 ≥ . . . ≥ pr , q1 ≥ q2 ≥ . . . ≥ qr , it holds that ki=1 pi ≤ ki=1 qi
for every 1 ≤ k ≤ r. Show that P > Q implies H(P ) ≥ H(Q).

Problem 6 ((Exponential) Rate of smallest sets with positive probability) [6a] Let {Xt }∞ t=1 be an i. consider the -typical (n) set A ⊆ X n. Given  > 0 and a positive integer n. i ∈ X . . p1 . n for all 0 < δ < 1. [7b] Suppose next that X has a nonuniform pmf PX = (p0 .Problem 5 Let X be a rv with values in the set of positive integers {1. process with pmf PX on X . For   (n) (n) = δ > 0. 1 < µ < ∞. Show that . .d.} and with pmf P satisfying EP [X] = ∞ X xP (x) = µ. . (n) [7a] Let X be distributed uniformly on X . determine the pmf P ∗ which maximizes H(X) subject to the constraint above. Problem 7 Let {Xt }∞ t=1 be a X -valued DMS. . 2. . let ν(n. . δ) be the minimum cardinality of sets Bδ ⊆ X n with PX Bδ P (X n ∈ Bδn ) ≥ 1 − δ > 0 for all n large. Let pi = Pr{X = i}. and show that the resulting maximum entropy is given by H(P ∗ ) = µ log µ − (µ − 1) log(µ − 1). δ) = H(PX ). . Characterize explicitly the set A . where X = {0. [6b] Is the qualifier “minimum” needed in [6a]? Justify your answer. where X is a finite set. q − 1} for an integer q ≥ 2. x=1 Using Lagrange multipliers or otherwise. 1. Show that lim n 1 log ν(n. .i. pq−1 ). . . .

q−1  .

)  .

.

X n (xn ) .

.

i xn ∈ X n : .

− pi log2 pi .

≤  .

.

i ∈ X . xn ∈ X n . n i=0 ( A(n) =  where ni (xn ) = number of occurrences of the symbol i in xn . .

show in [7b] that A(n)     .[7c] For the special case q = 2.

.

 n .

.

n1 (x )  n n .

.

 .

= x ∈X : . .

− p1 .

≤ .

.

 n 1 .

 .

log2 1−p .

etc. I(X.. [8c] State the analogs of the inequalities above with a conditioning rv Z (which takes values in a finite set Z) and a map f with domain X ×Z. Y to be finite subsets of the set of real numbers. i. for all x ∈ X with PX (x) > 0. and let f be a map with domain X . there is only one possible y ∈ Y with PXY (x. p1 Problem 8 [8a] Let X and Y be finite sets.e. respectively. Problem 9 Let X and Y be rvs with values in the finite sets X and Y. (For instance. Let X and Y be rvs with values in X and Y. Verify that H(X. and determine in each case the condition for equality: H(f (X)) ≤ H(X). respectively. I(f (X) ∧ Y ) ≤ I(X ∧ Y ). H(Y |X. f (X)) = H(X). f (X)) = H(Y |X). take X . H(Y |f (X)) ≥ H(Y |X). X+Y . Z)|Z) ≤ H(X|Z). (For instance. y) > 0. H(f (X. [8b] Using [8a] above. Set Z . show that Y is a function of X.) [4d] If H(Y |X) = 0.) . prove the following inequalities. f (X) ∧ Y ) = I(X ∧ Y ).

Problem 11 Let X be a finite set. [9c] Find conditions under which H(Z) = H(X) + H(Y ). then H(Y ) ≤ H(Z) and H(X) ≤ H(Z). Problem 10 Give examples of rvs (X. W is a |X | × |X |-matrix of nonnegative elements with row and column sums equal to 1. Z) such that [10a] I(X ∧ Y |Z) < I(X ∧ Y ). if no. Thus.e. x ∈ X. give a counterexample.[9a] Show that H(Z|X) = H(Y |X). Let W = {W (x|x0 ) : x. x0 ∈ X } be a doubly-stochastic matrix. the addition of independent rvs increases uncertainty. but is just a stochastic matrix with nonnegative elements whose rows sum to 1? If yes. Argue that if X and Y are independent. [9b] Give an example (of necessarily dependent rvs) for which H(X) > H(Z) and H(Y ) > H(Z). [10b] I(X ∧ Y |Z) > I(X ∧ Y ).. [11a] Let P be a pmf on X . Y. . x0 ∈X [11b] Is the assertion of [11a] valid if W is not doubly-stochastic. i. provide a proof. Show that H(P W ) ≥ H(P ) where P W is a pmf on X given by (P W )(x) = X P (x0 )W (x|x0 ).

x∈X : f (x)=y [13a] Either prove or disprove that D(Pf ||Qf ) ≤ D(P ||Q). The mapping f induces pmfs Pf and Qf on Y corresponding to P and Q. . Xn . Now. X P (x). Xn to be mutually independent. . . . X Q(x). . [12b] Drop the assumption in [12a] of the mutual independence of X1 . x∈X : f (x)=y Qf (y) . Let X n .Problem 12 Let X and Y be finite sets. where Pf (y) . Problem 13 Let P and Q be pmfs on a finite set X . [12a] Assume X1 . show that I(X n ∧ Y n ) ≤ n X I(Xt ∧ Yt ) t=1 and deduce the necessary and sufficient condition for equality. (Y1 . . . . (X1 . . . Xn ) and Y n . [13b] Determine a condition which is both necessary and sufficient for equality above. . . Consider the mapping f : X → Y. and assume instead that PY n |X n (y n |xn ) = Πnt=1 PYt |Xt (yt |xt ) for all xn ∈ X n and y n ∈ Y n .and Y-valued rvs. respectively. . respectively. where Y is a finite set. Show that n n I(X ∧ Y ) ≥ n X I(Xt ∧ Yt ) t=1 and deduce the necessary and sufficient condition for equality. . y ∈ Y. . . . Yn ) be collections of X .

rvs with P (X1 = 1) = 1 = 1 − P (X1 = 0). 2 where V is a {0. X2 |Z1 . t≥1 where {Vt }∞ t=1 be a sequence of {0. . Z2 defined by Zt = Xt + V mod 2.i. [14b] Show that lim inf P(X n 6= Y n ) ≥ h(p) n where h(p) = −p log2 p − (1 − p) log2 (1 − p). 2 Let the sequence of rvs {Yt }∞ t=1 be defined according to Yt = Xt + V t mod 2. n ≥ 1.d.d.i. Z2 ) in terms of p. t = 1. rvs with P (V1 = 1) = p = 1 − P (V1 = 0). 1}-valued i. 1}-valued rv with P (V = 1) = p = 1 − P (V = 0). in terms of n. [14a] Compute H(X n |Y n ). p. 0<p<1 and with V independent of (X1 . Compute H(X1 . You must justify each step and show all your calculations in order to receive credit. 0<p<1 ∞ and with {Vt }∞ t=1 independent of {Xt }t=1 . 1}-valued i.Problem 14 Let {Xt }∞ t=1 be a sequence of {0. X2 ). [14c] Consider the rvs Z1 .

e. Prove that I(X ∧ Y |F1 . the finite-valued rvs X and Y with joint pmf P . K (and hence E) are finite-valued with joint pmf P. Ahmed and Praneeth are no better mutually informed than without it! Problem 16 (Shannon’s Cipher System. f1 (X)).. respectively. Here. Prove that the requirements (a) and (b) imply that H(K) ≥ H(M ). an eavesdropper observing E on a public channel cannot learn M .e. conditioned on their interactive communication. (b) M is a function of (E. Thus.. . i. F2 ) ≤ I(X ∧ Y ). a sender and a receiver share a “secret key” K.Problem 15 Suppose that Ahmed and Praneeth observe. They engage in “interactive communication. K) such that the following two requirements are met: (a) M is independent of E = e(M. Here. K). K) Sender Receiver M In the cipher system above. K).e. The rvs M. f1 and f2 are fixed mappings. K).” i. e is a fixed mapping. i. the receiver can recover M from a knowledge of K and E = e(M. Ahmed sends to Praneeth a rv F1 = f1 (X) and Praneeth responds with a rv F2 = f2 (Y.. 1949) Secret key K K Message M K E = e(M. The sender then transmits a message rv M in encrypted form as E = e(M.

Q(x) > 0.Problem 17 Let P and Q be two pmfs on a finite set X . x ∈ (n) X . define the “mismatched” -typical set B B(n) ⊆ X n by . and assume that P (x) > 0. For  > 0.

  n o.

.

.

1 n n n n .

.

= x ∈ X : .

− log Q (x ) − H(P ) + D(P ||Q) .

limn P n (B ) = 1. [17b] Show that h n oi . ≤  n (n) [17a] Prove that for every  > 0.

(n) .

.

B .

. y1 . an }. . but using a “mismatched” typical set for a pmf Q. Y ). . . Show that 2H(A) ≤ H (Axy ) + H (Axz ) + H (Ayz ) . i = 1. n. y1 ). . Z). zi ).i. the projections of a1 = (x1 . and Ayz = (Y. and n3 distinct projection on the xy−. The projections of A on the three planes are Axy = (X. in R3 . ≤ exp n H(P ) + D(P ||Q) +  . can be performed at a penalized compression rate H(P ) + D(P ||Q). Problem 18 Consider n distinct (and fixed) points ai = (xi . z1 ) on the xy−plane is the point (x1 . yi .d. Z) be distributed uniformly on {a1 . . source with “true” pmf P . Y. Hint: Let the rv A = (X. Remark: This problem shows that fixed-to-fixed length data compression of an i. respectively. n2 . (For instance. . Suppose that these points have n1 . Z). respectively. . Axz = (X. xz−. and yz−planes. .) Show that n2 ≤ n1 n2 n3 .