15 views

Uploaded by YuQin Xu

ENEE627 Problem set1

save

- Guidelines 615
- The Second Handbook on the Psychology of Mathematics Education
- Measure
- MM2 Marking Sbh
- Cryptography Presentation BY Jyoti Goyal
- CCN Lab Manual Programs
- CAPE Pure Mathematics (2016) U2 P1 Answers
- EE-SEMESTER-I-VIII-HPTU-SYLLABUS.pdf
- A Unified Block and Stream Cipher Based File Encryption 53 57
- An Elementary Course in Synthetic Projective Geometry Lehmerd1700117001-8
- Sensitivity Analysis A Multiconductor Transmission Line
- Midterm Tit 704
- Pascal's Pensees
- Note_aggregate Price Under Calvo-constraint
- Lesson 19 Ex
- Matrix and Determinants
- Math Cads
- Matlab Tut
- simulink
- Experiment No. 01
- EEC5 2011-12-2
- Linear Algebra Cheat Sheet
- MATLAB
- iit jee sllybus
- GilbertBowdenECCV06
- Ph zack
- Linear Eq Direct Method Matrices
- MIT2_72s09_lec05
- A3-R3 syllabus
- week04
- Sapiens: A Brief History of Humankind
- The Unwinding: An Inner History of the New America
- Yes Please
- Dispatches from Pluto: Lost and Found in the Mississippi Delta
- Elon Musk: Tesla, SpaceX, and the Quest for a Fantastic Future
- The Innovators: How a Group of Hackers, Geniuses, and Geeks Created the Digital Revolution
- John Adams
- Devil in the Grove: Thurgood Marshall, the Groveland Boys, and the Dawn of a New America
- A Heartbreaking Work Of Staggering Genius: A Memoir Based on a True Story
- This Changes Everything: Capitalism vs. The Climate
- Grand Pursuit: The Story of Economic Genius
- The Emperor of All Maladies: A Biography of Cancer
- The Prize: The Epic Quest for Oil, Money & Power
- Team of Rivals: The Political Genius of Abraham Lincoln
- The New Confessions of an Economic Hit Man
- The World Is Flat 3.0: A Brief History of the Twenty-first Century
- Rise of ISIS: A Threat We Can't Ignore
- Smart People Should Build Things: How to Restore Our Culture of Achievement, Build a Path for Entrepreneurs, and Create New Jobs in America
- The Hard Thing About Hard Things: Building a Business When There Are No Easy Answers
- How To Win Friends and Influence People
- Steve Jobs
- Angela's Ashes: A Memoir
- Bad Feminist: Essays
- The Light Between Oceans: A Novel
- The Silver Linings Playbook: A Novel
- Leaving Berlin: A Novel
- Extremely Loud and Incredibly Close: A Novel
- The Sympathizer: A Novel (Pulitzer Prize for Fiction)
- You Too Can Have a Body Like Mine: A Novel
- The Incarnations: A Novel
- Life of Pi
- The Love Affairs of Nathaniel P.: A Novel
- A Man Called Ove: A Novel
- Bel Canto
- The Master
- The Rosie Project: A Novel
- The Blazing World: A Novel
- We Are Not Ourselves: A Novel
- The First Bad Man: A Novel
- The Flamethrowers: A Novel
- Brooklyn: A Novel
- The Art of Racing in the Rain: A Novel
- Wolf Hall: A Novel
- The Wallcreeper
- Interpreter of Maladies
- Beautiful Ruins: A Novel
- The Kitchen House: A Novel
- The Perks of Being a Wallflower
- The Bonfire of the Vanities: A Novel
- Lovers at the Chameleon Club, Paris 1932: A Novel
- A Prayer for Owen Meany: A Novel
- The Cider House Rules
- Mislaid: A Novel

You are on page 1of 8

Spring 2016

Problem Set 1

Narayan

Posted: February 5, 2016, Due: February 16, 2016

**Problem 1 (Axiomatic derivation of entropy)
**

Cover-Thomas, problem 2.46.

Problem 2

Show that a rv with values in a countable set can have infinite entropy.

Problem 3

Let X be a rv taking values in the (finite) set {1, . . . , M } with pmf PX such that

PX (1) ≤ . . . ≤ PX (M ).

Show that

[3a]

H(X) ≥ − log PX (M );

[3b]

H(X) ≥ −PX (M ) log PX (M ) − (1 − PX (M )) log(1 − PX (M )).

Problem 4

The interpretation of entropy as a measure of uncertainty of a rv suggests that “more

uniform” pmfs have larger entropy. For two pmfs P and Q on a finite set X = {1, . . . , r},

we say that P is more uniform than Q, denoted P > Q, if for the nonincreasing ordering

P

P

of probabilities p1 ≥ p2 ≥ . . . ≥ pr , q1 ≥ q2 ≥ . . . ≥ qr , it holds that ki=1 pi ≤ ki=1 qi

for every 1 ≤ k ≤ r. Show that P > Q implies H(P ) ≥ H(Q).

Problem 6 ((Exponential) Rate of smallest sets with positive probability) [6a] Let {Xt }∞ t=1 be an i. consider the -typical (n) set A ⊆ X n. Given > 0 and a positive integer n. i ∈ X . . p1 . n for all 0 < δ < 1. [7b] Suppose next that X has a nonuniform pmf PX = (p0 .Problem 5 Let X be a rv with values in the set of positive integers {1. process with pmf PX on X . For (n) (n) = δ > 0. 1 < µ < ∞. Show that . .d.} and with pmf P satisfying EP [X] = ∞ X xP (x) = µ. . (n) [7a] Let X be distributed uniformly on X . determine the pmf P ∗ which maximizes H(X) subject to the constraint above. Problem 7 Let {Xt }∞ t=1 be a X -valued DMS. . 2. . let ν(n. . δ) be the minimum cardinality of sets Bδ ⊆ X n with PX Bδ P (X n ∈ Bδn ) ≥ 1 − δ > 0 for all n large. Let pi = Pr{X = i}. and show that the resulting maximum entropy is given by H(P ∗ ) = µ log µ − (µ − 1) log(µ − 1). δ) = H(PX ). . Characterize explicitly the set A . where X = {0. [6b] Is the qualifier “minimum” needed in [6a]? Justify your answer. where X is a finite set. q − 1} for an integer q ≥ 2. x=1 Using Lagrange multipliers or otherwise. 1. Show that lim n 1 log ν(n. .i. pq−1 ). . . .

q−1 .

) .

.

X n (xn ) .

.

i xn ∈ X n : .

− pi log2 pi .

≤ .

.

i ∈ X . xn ∈ X n . n i=0 ( A(n) = where ni (xn ) = number of occurrences of the symbol i in xn . .

show in [7b] that A(n) .[7c] For the special case q = 2.

.

n .

.

n1 (x ) n n .

.

.

= x ∈X : . .

− p1 .

≤ .

.

n 1 .

.

log2 1−p .

etc. I(X.. [8c] State the analogs of the inequalities above with a conditioning rv Z (which takes values in a finite set Z) and a map f with domain X ×Z. Y to be finite subsets of the set of real numbers. i. for all x ∈ X with PX (x) > 0. and let f be a map with domain X . there is only one possible y ∈ Y with PXY (x. p1 Problem 8 [8a] Let X and Y be finite sets.e. respectively. Problem 9 Let X and Y be rvs with values in the finite sets X and Y. (For instance. Let X and Y be rvs with values in X and Y. Verify that H(X. and determine in each case the condition for equality: H(f (X)) ≤ H(X). respectively. I(f (X) ∧ Y ) ≤ I(X ∧ Y ). H(Y |X. f (X)) = H(X). f (X)) = H(Y |X). take X . H(Y |f (X)) ≥ H(Y |X). X+Y . Z)|Z) ≤ H(X|Z). (For instance. y) > 0. H(f (X. [8b] Using [8a] above. Set Z . show that Y is a function of X.) [4d] If H(Y |X) = 0.) . prove the following inequalities. f (X) ∧ Y ) = I(X ∧ Y ).

Problem 11 Let X be a finite set. [9c] Find conditions under which H(Z) = H(X) + H(Y ). then H(Y ) ≤ H(Z) and H(X) ≤ H(Z). Problem 10 Give examples of rvs (X. W is a |X | × |X |-matrix of nonnegative elements with row and column sums equal to 1. Z) such that [10a] I(X ∧ Y |Z) < I(X ∧ Y ). if no. Thus.e. x ∈ X. give a counterexample.[9a] Show that H(Z|X) = H(Y |X). Let W = {W (x|x0 ) : x. x0 ∈ X } be a doubly-stochastic matrix. the addition of independent rvs increases uncertainty. but is just a stochastic matrix with nonnegative elements whose rows sum to 1? If yes. Argue that if X and Y are independent. [9b] Give an example (of necessarily dependent rvs) for which H(X) > H(Z) and H(Y ) > H(Z). [10b] I(X ∧ Y |Z) > I(X ∧ Y ).. [11a] Let P be a pmf on X . Y. . x0 ∈X [11b] Is the assertion of [11a] valid if W is not doubly-stochastic. i. provide a proof. Show that H(P W ) ≥ H(P ) where P W is a pmf on X given by (P W )(x) = X P (x0 )W (x|x0 ).

x∈X : f (x)=y [13a] Either prove or disprove that D(Pf ||Qf ) ≤ D(P ||Q). The mapping f induces pmfs Pf and Qf on Y corresponding to P and Q. . Xn . Now. X P (x). Xn to be mutually independent. . . . X Q(x). . [12b] Drop the assumption in [12a] of the mutual independence of X1 . x∈X : f (x)=y Qf (y) . Let X n .Problem 12 Let X and Y be finite sets. where Pf (y) . Problem 13 Let P and Q be pmfs on a finite set X . [12a] Assume X1 . show that I(X n ∧ Y n ) ≤ n X I(Xt ∧ Yt ) t=1 and deduce the necessary and sufficient condition for equality. (Y1 . . . . (X1 . . . Xn ) and Y n . [13b] Determine a condition which is both necessary and sufficient for equality above. . . Consider the mapping f : X → Y. and assume instead that PY n |X n (y n |xn ) = Πnt=1 PYt |Xt (yt |xt ) for all xn ∈ X n and y n ∈ Y n .and Y-valued rvs. respectively. . respectively. where Y is a finite set. Show that n n I(X ∧ Y ) ≥ n X I(Xt ∧ Yt ) t=1 and deduce the necessary and sufficient condition for equality. . y ∈ Y. . . . Yn ) be collections of X .

rvs with P (X1 = 1) = 1 = 1 − P (X1 = 0). 2 where V is a {0. X2 |Z1 . t≥1 where {Vt }∞ t=1 be a sequence of {0. . Z2 defined by Zt = Xt + V mod 2.i. [14b] Show that lim inf P(X n 6= Y n ) ≥ h(p) n where h(p) = −p log2 p − (1 − p) log2 (1 − p). 2 Let the sequence of rvs {Yt }∞ t=1 be defined according to Yt = Xt + V t mod 2. n ≥ 1.d.d.i. Z2 ) in terms of p. t = 1. rvs with P (V1 = 1) = p = 1 − P (V1 = 0). 1}-valued i. 1}-valued rv with P (V = 1) = p = 1 − P (V = 0). in terms of n. [14a] Compute H(X n |Y n ). p. 0<p<1 and with V independent of (X1 . Compute H(X1 . You must justify each step and show all your calculations in order to receive credit. 0<p<1 ∞ and with {Vt }∞ t=1 independent of {Xt }t=1 . 1}-valued i.Problem 14 Let {Xt }∞ t=1 be a sequence of {0. X2 ). [14c] Consider the rvs Z1 .

e. Prove that I(X ∧ Y |F1 . the finite-valued rvs X and Y with joint pmf P . K (and hence E) are finite-valued with joint pmf P. Ahmed and Praneeth are no better mutually informed than without it! Problem 16 (Shannon’s Cipher System. f1 (X)).. respectively. Here. Prove that the requirements (a) and (b) imply that H(K) ≥ H(M ). an eavesdropper observing E on a public channel cannot learn M .e. conditioned on their interactive communication. (b) M is a function of (E. Thus.. . i. F2 ) ≤ I(X ∧ Y ). a sender and a receiver share a “secret key” K.Problem 15 Suppose that Ahmed and Praneeth observe. They engage in “interactive communication. K) such that the following two requirements are met: (a) M is independent of E = e(M. Here. K). K) Sender Receiver M In the cipher system above. K).e. The rvs M. f1 and f2 are fixed mappings. K).” i. e is a fixed mapping. i. the receiver can recover M from a knowledge of K and E = e(M. Ahmed sends to Praneeth a rv F1 = f1 (X) and Praneeth responds with a rv F2 = f2 (Y.. 1949) Secret key K K Message M K E = e(M. The sender then transmits a message rv M in encrypted form as E = e(M.

Q(x) > 0.Problem 17 Let P and Q be two pmfs on a finite set X . x ∈ (n) X . define the “mismatched” -typical set B B(n) ⊆ X n by . and assume that P (x) > 0. For > 0.

n o.

.

.

1 n n n n .

.

= x ∈ X : .

− log Q (x ) − H(P ) + D(P ||Q) .

limn P n (B ) = 1. [17b] Show that h n oi . ≤ n (n) [17a] Prove that for every > 0.

(n) .

.

B .

. y1 . an }. . but using a “mismatched” typical set for a pmf Q. Y ). . . Show that 2H(A) ≤ H (Axy ) + H (Axz ) + H (Ayz ) . i = 1. n. y1 ). . Z). zi ).i. the projections of a1 = (x1 . and Ayz = (Y. and n3 distinct projection on the xy−. The projections of A on the three planes are Axy = (X. in R3 . ≤ exp n H(P ) + D(P ||Q) + . can be performed at a penalized compression rate H(P ) + D(P ||Q). Problem 18 Consider n distinct (and fixed) points ai = (xi . z1 ) on the xy−plane is the point (x1 . yi .d. Z) be distributed uniformly on {a1 . . source with “true” pmf P . Y. Hint: Let the rv A = (X. Remark: This problem shows that fixed-to-fixed length data compression of an i. respectively. n2 . (For instance. . Suppose that these points have n1 . Z). respectively. . Axz = (X. xz−. and yz−planes. .) Show that n2 ≤ n1 n2 n3 .

- Guidelines 615Uploaded byAakash Khanna
- The Second Handbook on the Psychology of Mathematics EducationUploaded byalexander reyes
- MeasureUploaded bymathematician7
- MM2 Marking SbhUploaded bySimPor
- Cryptography Presentation BY Jyoti GoyalUploaded byJyoti Goyal nitk
- CCN Lab Manual ProgramsUploaded byswapna_charan
- CAPE Pure Mathematics (2016) U2 P1 AnswersUploaded byDaveed
- EE-SEMESTER-I-VIII-HPTU-SYLLABUS.pdfUploaded byAnonymous HGD9CHZ
- A Unified Block and Stream Cipher Based File Encryption 53 57Uploaded byManish Kumar
- An Elementary Course in Synthetic Projective Geometry Lehmerd1700117001-8Uploaded bykostel.pagubossu5933
- Sensitivity Analysis A Multiconductor Transmission LineUploaded byMarioTorresCruz
- Midterm Tit 704Uploaded bymm8871
- Pascal's PenseesUploaded byStephen Hague
- Note_aggregate Price Under Calvo-constraintUploaded byTommaso Marossi
- Lesson 19 ExUploaded bySmita R. S.
- Matrix and DeterminantsUploaded byVishal Jalan
- Math CadsUploaded byRobert Gajewski
- Matlab TutUploaded bymohamed ahmed
- simulinkUploaded bydpksobs
- Experiment No. 01Uploaded byMuhammad Asghar Khan
- EEC5 2011-12-2Uploaded byNihaal Barua
- Linear Algebra Cheat SheetUploaded byMiriam Quan Johansson
- MATLABUploaded byMia Hussain
- iit jee sllybusUploaded bye
- GilbertBowdenECCV06Uploaded byLihi Alter
- Ph zackUploaded byAkbar Shakoor
- Linear Eq Direct Method MatricesUploaded bySukhwinder Singh Gill
- MIT2_72s09_lec05Uploaded byShivdayal Patel
- A3-R3 syllabusUploaded byapi-3782519
- week04Uploaded byranjan_prashant52