Information Theory and Coding (EC60332)

Indian Institute of Technology Kharagpur

Department of Electronics and Electrical Communication Engineering
Instructor: Jithin Ravi

Assignment 1 Due on: 7 February 2023

n
1X
1. For a sequence an , let bn = ai .
n
i=1

(a) It is given that lim an = a. Show that lim bn = a.

n→∞ n→∞
(b) It is given that lim bn = b. Check whether lim an = b.
n→∞ n→∞

2. Two random variables X and Y are said to be uncorrelated if E[(X − X̄)(Y − Ȳ )] = 0, where
X̄ and Ȳ denote E(X) and E(Y ), respectively.

(a) Show that, for uncorrelated random variables, the expected value of the product is equal
to the product of the expected values.
(b) Prove or disprove: “two uncorrelated random variables are always independent”.
(c) Prove or disprove: “two independent random variables are always uncorrelated”.
(d) Show that if X and Y are uncorrelated, then the variance of X + Y is equal to the
variance of X plus the variance of Y .

3. Assume that random variables X and Y are not independent. Find an example such that
E[X · Y ] = E[X] · E[Y ], and another example where E[X · Y ] ̸= E[X] · E[Y ].

4. Let X1 , X2 be two independent Bernoulli(p) random variables with p = 1/2. Let us define
Z = X1 ⊕ X2 , where ⊕ is the XOR operation of bits.

(a) Is Z independent of X1 ?
(b) Are Z, X1 , X2 independent? (Random variables A, B, C are independent if Pr(A =
a, B = b, C = c) = Pr(A = a) Pr(B = b) Pr(C = c).)
(c) Find H(X1 |Z) and H(X1 |Z, X2 ).

5. Let X be a random variable taking on a finite number of values. Find the relation (<, ≤, =
, >, ≥) between H(X) and H(Y ) if

(a) Y = log X?
(b) Y = sin X?

6. Similar to prefix of a codeword, let us consider suffix of a codeword, e.g., for 1011, the suffixes
are 1, 11 and 011. We say a code is suffix-free when no codeword is a suffix of any other
codeword.

(a) Show that suffix-free codes are uniquely decodable.

(b) Is there a code with codeword lengths (1, 2, 2) that is both prefix-free and suffix-free?
Explain.