Problem 1.

a) Let p(x,y) be the joint probability distribution of the two random variables X and Y. Define the
conditional entropy H(X|Y) in terms of the joint distribution and associated conditional
probabilities.
b) Prove the formula:
H(X,Y) = H(X|Y) + H(Y) .
Problem 2. Consider the random variable:

a) Find a binary Huffman code.

b) Find the expected code length.
c) One admits that the entropy of X is H(X) = 2.0128: is your result compatible with this value?
Explain why.
Problem 3. Consider a random variable X with four possible values x1, x2, x3, x4. Consider six possible
codes A, B, C, D, E, F as follows:

a) Which codes satisfy the Kraft inequality?

b) Which codes are prefix?
c) Which codes are uniquely decodable?

Problem 4.
A discrete memoryless source has an alphabet of three letters, xi (i = 1, 2, 3), with probabilities 0.4, 0.4,
and 0.2, respectively.
a) Find the binary Huffman code for this source and determine the average number of bits needed
for each source letter.
b) Suppose two letters at a time are encoded into a binary sequence. Find the Huffman code and the
average number of bits needed per source letter.
Problem 5. A gambler has the opportunity to bet repeatedly on the occurrence of a certain event, which is
known to occur with probability 0.6, and returns $2 for each $1 bet (for a profit of $1). Suppose that the
gambler bets a fraction b of total funds each time, where 0 ≤ b ≤ 1.
a) Write down a formula for the expected rate of growth of the gambler’s funds.
b) Find the value of b which maximizes this expected growth rate.
c) Calculate the expected growth rate in this case. Give your answers correct to three decimal
places.
1
Problem 6. Consider a horse race with four horses. Assume that each horse pays 4-for-1 if it wins. Let the
probabilities of winning of the horses be {1/2, 1/4, 1/8, 1/8}. If you started with $100 and bet optimally to
maximize your long-term growth rate, what are your optimal bets on each horse? Approximately how
much money would you have after 20 races with this strategy?
Problem 7. Find the channel capacity of the following discrete memoryless channel:

where P(Z = 0) = P(Z = a) = 1/2. The alphabet for X is {0, 1}. Y = X + Z with real addition. Assume that
Z is independent of X. Observe that the channel capacity depends on the value of a.
Problem 8. An instantaneous code has word lengths l1, l2, . . . , lm, which satisfy the strict inequality

𝐷 <1

The code alphabet is D = {0, 1, 2, . . . , D − 1}. Show that there exist arbitrarily long sequences of code
symbols in D* which can not be decoded into sequences of codewords.