Information Theory and Codes

Vittorio Giovara, Alberto Grand, Marina Lemessom
19/10/2007
Chapter 1
Entropy and Quantity of
Information
1.1 Handout
Two fair six-faced dices are thrown by Alice and only their sum is commu-
nicated to Bob. Which is the amount of information given with respect to
communicating the distinct occurrences of the two dices?
1.2 Resolution
First we calculate the entropy of the sum of the dices by filling this table, A
being the possible symbol to be sent and p
a
the probability of that symbol
to be sent.
So we compute the entropy with this formula
H(A) =
M
¸
i=1
p
i
log
1
p
i
where M is the number of different symbols and p
i
the probability associated
to the i-th symbol. The base of the logarithm is 2 because we want entropy
to be expressed in bits.
H(A) =
11
¸
i=1
p
i
log
2
1
p
i
=
1
36
log

1
1
36

+
2
36
log

1
2
36

+
3
36
log

1
3
36

+
+
4
36
log

1
4
36

+
5
36
log

1
5
36

+
6
36
log

1
6
36

+
5
36
log

1
5
36

+
+
4
36
log

1
4
36

+
3
36
log

1
3
36

+
2
36
log

1
2
36

+
1
36
log

1
1
36

=
1
A p
a
2

1
36

3

2
36

4

3
36

5

4
36

6

5
36

7

6
36

8

5
36

9

4
36

10

3
36

11

2
36

12

1
36

= 2
1
36
log(36) + 2
2
36
log

1
1
18

+ 2
3
36
log

1
1
12

+ 2
4
36
log

1
1
9

+
+ 2
5
36
log

36
5

+
6
36
log

1
1
6

=
1
18
log(36) +
1
9
log(18) +
+
1
6
log(12) +
2
9
log(9) +
5
18
log

36
5

+
1
6
log(6) =
= 3.27 bits
Then we compute the entropy of a roll of a single dice, using this B
1
alphabet.
B p
b
1

1
6

2

1
6

3

1
6

4

1
6

5

1
6

6

1
6

H(B
1
) =
6
¸
i=1
p
i
log
2
1
p
i
=
2
= 6
1
6
log

1
1
6

=
= log(6) =
= 2.58 bits
Since the symbols of the two dices have the same probability, the alpha-
bet of the second dice B
2
is equal to the first.
B
1
= B
2
Thus also the entropy of the second dice is equal to the first.
H(B
1
) = H(B
2
)
The entropy of submitting the distinct results of the two dices B
tot
is the
sum of the two entropies.
H(B
tot
) = H(B
1
) + H(B
2
) =
= 2H(B
1
) =
= 2 ∗ 2.58 =
= 5.16 bits
At last we calculate the difference of the quantity of information when
sending the sum of the dices and the single result of the roll.
H(∆) = H(B
tot
) −H(A) =
= 5.16 −3.27 =
= 1.89 bits
3
Chapter 2
Bynary Enconding
2.1 Handout
Constuct the binary Huffman tree for a four letters alphabet with probabil-
ity distribution P = (0.8, 0.1, 0.05, 0.05). Compute the average code word
length and entropy.
2.2 Resolution
We can build the binary Huffman tree by associating the alphabet A =
(a
1
, a
2
, a
3
, a
4
) which has a probability distribution of P = (0.8, 0.1, 0.05, 0.05)
to the binary alphabet B = (0, 1).
4
So we can see this association:
a
1
−→ 0
a
2
−→ 10
a
3
−→ 110
a
4
−→ 111
from which we can calculate the average code length
¯
l with this formula
¯
l =
M
¸
i=1
l
i
p
i
¯
l =
4
¸
i=1
l
i
p
i
=
= 1 ∗ 0.8 + 2 ∗ 0.1 + 3 ∗ 0.05 + 3 ∗ 0.05 =
= 0.8 + 0.2 + 0.3 =
= 1.3
And as for the entropy we use the previous formula
H(A) =
M
¸
i=1
p
i
log
1
p
i
H(B) = 0.8 log

1
0.8

+ 0.1 log

1
0.1

+ 0.1 log

1
0.05

=
= 1.022 bits
We observe that the First Shannon Theorem is respected, since
H(A) ≤
¯
l log D ≤ H(A) + log D
1.022 ≤
¯
l ≤ 2.044
5
Chapter 3
Coding Theory
3.1 Handout
Let C be a binary (11, 7, d)-code with parity-check matrix
H =

0 0 1 0 1 0 1 1 0 1 1
0 1 0 1 1 1 1 1 0 0 0
1 0 0 0 0 1 1 1 0 1 1
0 1 1 0 1 0 0 1 1 1 0
¸
¸
¸
¸
¸
1. Find a generator matrix (echelon form) G for C;
2. Find the minimum distance d;
3. Determine the weight enumerator polynomial for C;
4. Decode the following received words:
R
1
= (1000 0000 111) R
2
= (0111 0100 011)
3.2 Resolution
3.2.1 Generator Matrix Computation
It is easy to obtain the generator matrix from the parity-check matrix if we
are able to reduce the latter to an echelon form. In this computation we are
allowed to exchange and add rows.
So we swap r
1
with r
3
and r
2
with r
4
, obtaining
H =

1 0 0 0 0 1 1 1 0 1 1
0 1 1 0 1 0 0 1 1 1 0
0 0 1 0 1 0 1 1 0 1 1
0 1 0 1 1 1 1 1 0 0 0
¸
¸
¸
¸
¸
6
then we add r
3
to r
2
and this new r
2
to r
4
, obtaining
H =

1 0 0 0 0 1 1 1 0 1 1
0 1 0 0 0 0 1 0 1 0 1
0 0 1 0 1 0 1 1 0 1 1
0 0 0 1 1 1 0 1 1 0 1
¸
¸
¸
¸
¸
We have been able to write H in the form
H = (I
n−k
|A)
As a consequence, the corresponding generator matrix will have the form
G =

A
I
k

Having said that, it is trivial to compute the generator matrix
G =

0 1 1 1 0 1 1
0 0 1 0 1 0 1
1 0 1 1 0 1 1
1 1 0 1 1 0 1
1 0 0 0 0 0 0
0 1 0 0 0 0 0
0 0 1 0 0 0 0
0 0 0 1 0 0 0
0 0 0 0 1 0 0
0 0 0 0 0 1 0
0 0 0 0 0 0 1
¸
¸
¸
¸
¸
¸
¸
¸
¸
¸
¸
¸
¸
¸
¸
¸
¸
¸
¸
¸
¸
We are used to information symbols being the first k symbols of a codeword,
whereas the following (n − k) symbols are parity-check symbols. However,
this order is reversed in our case, the identity matrix being located in the
lower part of the generator matrix. The first 4 symbols of every codeword
are therefore parity-check symbols, while the last 7 symbols are information
symbols.

c
1
= x
2
+ x
3
+ x
4
+ x
6
+ x
7
c
2
= x
3
+ x
5
+ x
7
c
3
= x
1
+ x
3
+ x
4
+ x
5
+ x
7
c
4
= x
1
+ x
2
+ x
4
+ x
5
+ x
7
c
5
= x
1
c
6
= x
2
c
7
= x
3
c
8
= x
4
c
9
= x
5
c
10
= x
6
c
11
= x
7
7
3.2.2 Minimum Distance and Weight Enumerator Polyno-
mial
In order to find the minimum distance and the weight enumerator polyno-
mial for the given code, we have decided to write a simple algorithm in C.
Its purpose is to compute all possible 2
7
codewords, surveying their Ham-
ming weight and storing the results in a statistics vector. The i-th entry of
this vector contains the number of codewords having weight i and represents
therefore the i-th coefficient of the weight enumerator polynomial.
Since we are dealing with a linear code, it is trivial to find the minimum
distance between codewords. As a matter of fact
min
x,y∈C
x=y
d(x, y) = min
x,y∈C
x=y
W
H
(x −y)
but since the code is linear, c = x−y still belongs to C, the equation becomes
min
x,y∈C
x=y
d(x, y) = min
c∈C
c=0
W
H
(c)
and thus looking for the minimum distance between codewords boils down to
looking for the minimum weight over all codewords. This allows us to simply
inspect the weight enumerator polynomial coefficients of the given code: the
index of the minimum-index non-zero coefficient (disregarding coefficient A
0
,
because it corresponds to the 0 codeword) is the sought minimum distance
d.
The result provided by the algorithm are the following:
i A
i
0 1
1 0
2 0
3 13
4 26
5 24
6 24
7 26
8 13
9 0
10 0
11 1
Thus the minimum distance d is 3 and the error correction capabilty of the
given code is 1.
8
Finally the weight enumerator polynomial is
W(x, y) =
11
¸
1=0
A
i
x
i
y
11−i
=
= y
11
+ 13x
3
y
8
+ 26x
4
y
7
+ 24x
5
y
6
+ 24x
6
y
5
+ 26x
7
y
4
+ 13x
8
y
3
+ x
11
3.2.3 Words Decoding
In order to decode R
1
and R
2
, we have to compute their respective syn-
drome, being
R
1
=

1
0
0
0
0
0
0
1
1
1
1
¸
¸
¸
¸
¸
¸
¸
¸
¸
¸
¸
¸
¸
¸
¸
¸
¸
¸
¸
¸
¸
R
2
=

0
1
1
1
0
1
0
0
0
1
1
¸
¸
¸
¸
¸
¸
¸
¸
¸
¸
¸
¸
¸
¸
¸
¸
¸
¸
¸
¸
¸
and E
1
and E
2
their respective error pattern.
So we find the syndrome of the first word as
S
1
= HR
1
= HE
1
=

1
0
0
0
¸
¸
¸
¸
¸
and we can notice that the computed syndrome is equal to the first column
of the parity-check matrix.
Since the syndrome is a linear combination of the columns of the parity-
check matrix H, it is clear that the minimum weight error pattern is
ˆ
E
1
=

1
0
0
0
0
0
0
0
0
0
0
¸
¸
¸
¸
¸
¸
¸
¸
¸
¸
¸
¸
¸
¸
¸
¸
¸
¸
¸
¸
¸
9
Knowing that the error correction capability of the code is 1, we can
correct this error.
The estimated codeword for the first received vector is
ˆ
C
1
= R
1
+
ˆ
E
1
=

0
0
0
0
0
0
0
0
1
1
1
¸
¸
¸
¸
¸
¸
¸
¸
¸
¸
¸
¸
¸
¸
¸
¸
¸
¸
¸
¸
¸
and consequentely the sent message was
M
1
=

0
0
0
0
1
1
1
¸
¸
¸
¸
¸
¸
¸
¸
¸
¸
¸
¸
We now proceed to computing the syndrome for the second received
vector
S
2
= HR
2
= HE
2
=

1
0
1
1
¸
¸
¸
¸
¸
The computed syndrome is equal to the 8th column of the parity-check
matrix; therefore the minimum weight error pattern is
ˆ
E
2
=

0
0
0
0
0
0
0
1
0
0
0
¸
¸
¸
¸
¸
¸
¸
¸
¸
¸
¸
¸
¸
¸
¸
¸
¸
¸
¸
¸
¸
10
We can correct the error with the same method as before, obtaing the esti-
mated codeword
ˆ
C
2
= R
2
+
ˆ
E
2
=

0
1
1
1
0
1
0
1
0
1
1
¸
¸
¸
¸
¸
¸
¸
¸
¸
¸
¸
¸
¸
¸
¸
¸
¸
¸
¸
¸
¸
and consequentely the sent message was
M
2
=

0
1
0
1
0
1
1
¸
¸
¸
¸
¸
¸
¸
¸
¸
¸
¸
¸
11
Contents
1 Entropy and Quantity of Information 1
1.1 Handout . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
1.2 Resolution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
2 Bynary Enconding 4
2.1 Handout . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
2.2 Resolution . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
3 Coding Theory 6
3.1 Handout . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
3.2 Resolution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
3.2.1 Generator Matrix Computation . . . . . . . . . . . . . 6
3.2.2 Minimum Distance and Weight Enumerator Polynomial 8
3.2.3 Words Decoding . . . . . . . . . . . . . . . . . . . . . 9
12

Sign up to vote on this title
UsefulNot useful