Ain Shams University

Faculty of Engineering
Electronics and Communications Department
4th year – Spring 2022

ECE457 – Information Theory

Sheet #3
(Thursday, March 10)

1. Considering a discrete memory less source with source alphabet 𝜉 = {𝑠0 , 𝑠1 , 𝑠2 } with
respective probabilities 𝑝0 = 0.25, 𝑝1 = 0.25, 𝑝2 = 0.5 if the source being extended
a) calculate the number of symbols of the source 𝜉 2 .
b) calculate the number of symbols of the source 𝜉 3 .
c) calculate the entropy of the source 𝜉
d) calculate the entropy of the source 𝜉 2 .
e) calculate the entropy of the source 𝜉 3 .

2. Let the random variable X have to possible outcomes (a, b) with probabilities (P1, P2) find
the probabilities (P1, P2) that maximize H(X) and evaluate this maximum.

3. Let the random variable X have to possible outcomes (a, b, c) with probabilities (P1, P2,
P3) find the probabilities (P1, P2, P3) that maximize H(X) and evaluate this maximum.

4. A source has six possible outputs with probabilities as shown in the following table.
a) Which of these codes are uniquely decodable?
b) Which of these codes are instantaneous?
c) Find the average length for all uniquely decodable codes?

5. Consider the code (0, 01)

a) Is it instantaneous?
b) Is it uniquely decodable?
c) Is it Singular?

6. Which of the following codes are

a) Uniquely decodable
b) Instantaneous
C1 = {00, 01 , 0}
C2 = {00, 01, 100, 101, 11}
C3 = {0, 10, 110, 1110, ….}
C4 = {0, 00, 000, 0000}

Page | 1
Ain Shams University
Faculty of Engineering
Electronics and Communications Department
4th year – Spring 2022

ECE457 – Information Theory

Sheet #3
(Thursday, March 10)

7. Consider two information sources (S1,S2). Source (S1) sends symbols (a,b,c,d) with
probabilities (0.1,0.15,0.15,0.6) and source (S2) sends symbols (a,b,c,de,do,di) with
probabilities (0.1,0.15,0.15,0.3,0.2,0.1). What is the entropies gained by sources (S1,S2)?

Page | 2