This action might not be possible to undo. Are you sure you want to continue?

Welcome to Scribd! Start your free trial and access books, documents and more.Find out more

**Vittorio Giovara, Alberto Grand, Marina Lemessom
**

19/10/2007

Chapter 1

Entropy and Quantity of

Information

1.1 Handout

Two fair six-faced dices are thrown by Alice and only their sum is commu-

nicated to Bob. Which is the amount of information given with respect to

communicating the distinct occurrences of the two dices?

1.2 Resolution

First we calculate the entropy of the sum of the dices by ﬁlling this table, A

being the possible symbol to be sent and p

a

the probability of that symbol

to be sent.

So we compute the entropy with this formula

H(A) =

M

¸

i=1

p

i

log

1

p

i

where M is the number of diﬀerent symbols and p

i

the probability associated

to the i-th symbol. The base of the logarithm is 2 because we want entropy

to be expressed in bits.

H(A) =

11

¸

i=1

p

i

log

2

1

p

i

=

1

36

log

1

1

36

+

2

36

log

1

2

36

+

3

36

log

1

3

36

+

+

4

36

log

1

4

36

+

5

36

log

1

5

36

+

6

36

log

1

6

36

+

5

36

log

1

5

36

+

+

4

36

log

1

4

36

+

3

36

log

1

3

36

+

2

36

log

1

2

36

+

1

36

log

1

1

36

=

1

A p

a

2

1

36

3

2

36

4

3

36

5

4

36

6

5

36

7

6

36

8

5

36

9

4

36

10

3

36

11

2

36

12

1

36

= 2

1

36

log(36) + 2

2

36

log

1

1

18

+ 2

3

36

log

1

1

12

+ 2

4

36

log

1

1

9

+

+ 2

5

36

log

36

5

+

6

36

log

1

1

6

=

1

18

log(36) +

1

9

log(18) +

+

1

6

log(12) +

2

9

log(9) +

5

18

log

36

5

+

1

6

log(6) =

= 3.27 bits

Then we compute the entropy of a roll of a single dice, using this B

1

alphabet.

B p

b

1

1

6

2

1

6

3

1

6

4

1

6

5

1

6

6

1

6

H(B

1

) =

6

¸

i=1

p

i

log

2

1

p

i

=

2

= 6

1

6

log

1

1

6

=

= log(6) =

= 2.58 bits

Since the symbols of the two dices have the same probability, the alpha-

bet of the second dice B

2

is equal to the ﬁrst.

B

1

= B

2

Thus also the entropy of the second dice is equal to the ﬁrst.

H(B

1

) = H(B

2

)

The entropy of submitting the distinct results of the two dices B

tot

is the

sum of the two entropies.

H(B

tot

) = H(B

1

) + H(B

2

) =

= 2H(B

1

) =

= 2 ∗ 2.58 =

= 5.16 bits

At last we calculate the diﬀerence of the quantity of information when

sending the sum of the dices and the single result of the roll.

H(∆) = H(B

tot

) −H(A) =

= 5.16 −3.27 =

= 1.89 bits

3

Chapter 2

Bynary Enconding

2.1 Handout

Constuct the binary Huﬀman tree for a four letters alphabet with probabil-

ity distribution P = (0.8, 0.1, 0.05, 0.05). Compute the average code word

length and entropy.

2.2 Resolution

We can build the binary Huﬀman tree by associating the alphabet A =

(a

1

, a

2

, a

3

, a

4

) which has a probability distribution of P = (0.8, 0.1, 0.05, 0.05)

to the binary alphabet B = (0, 1).

4

So we can see this association:

a

1

−→ 0

a

2

−→ 10

a

3

−→ 110

a

4

−→ 111

from which we can calculate the average code length

¯

l with this formula

¯

l =

M

¸

i=1

l

i

p

i

¯

l =

4

¸

i=1

l

i

p

i

=

= 1 ∗ 0.8 + 2 ∗ 0.1 + 3 ∗ 0.05 + 3 ∗ 0.05 =

= 0.8 + 0.2 + 0.3 =

= 1.3

And as for the entropy we use the previous formula

H(A) =

M

¸

i=1

p

i

log

1

p

i

H(B) = 0.8 log

1

0.8

+ 0.1 log

1

0.1

+ 0.1 log

1

0.05

=

= 1.022 bits

We observe that the First Shannon Theorem is respected, since

H(A) ≤

¯

l log D ≤ H(A) + log D

1.022 ≤

¯

l ≤ 2.044

5

Chapter 3

Coding Theory

3.1 Handout

Let C be a binary (11, 7, d)-code with parity-check matrix

H =

0 0 1 0 1 0 1 1 0 1 1

0 1 0 1 1 1 1 1 0 0 0

1 0 0 0 0 1 1 1 0 1 1

0 1 1 0 1 0 0 1 1 1 0

¸

¸

¸

¸

¸

1. Find a generator matrix (echelon form) G for C;

2. Find the minimum distance d;

3. Determine the weight enumerator polynomial for C;

4. Decode the following received words:

R

1

= (1000 0000 111) R

2

= (0111 0100 011)

3.2 Resolution

3.2.1 Generator Matrix Computation

It is easy to obtain the generator matrix from the parity-check matrix if we

are able to reduce the latter to an echelon form. In this computation we are

allowed to exchange and add rows.

So we swap r

1

with r

3

and r

2

with r

4

, obtaining

H =

1 0 0 0 0 1 1 1 0 1 1

0 1 1 0 1 0 0 1 1 1 0

0 0 1 0 1 0 1 1 0 1 1

0 1 0 1 1 1 1 1 0 0 0

¸

¸

¸

¸

¸

6

then we add r

3

to r

2

and this new r

2

to r

4

, obtaining

H =

1 0 0 0 0 1 1 1 0 1 1

0 1 0 0 0 0 1 0 1 0 1

0 0 1 0 1 0 1 1 0 1 1

0 0 0 1 1 1 0 1 1 0 1

¸

¸

¸

¸

¸

We have been able to write H in the form

H = (I

n−k

|A)

As a consequence, the corresponding generator matrix will have the form

G =

A

I

k

**Having said that, it is trivial to compute the generator matrix
**

G =

0 1 1 1 0 1 1

0 0 1 0 1 0 1

1 0 1 1 0 1 1

1 1 0 1 1 0 1

1 0 0 0 0 0 0

0 1 0 0 0 0 0

0 0 1 0 0 0 0

0 0 0 1 0 0 0

0 0 0 0 1 0 0

0 0 0 0 0 1 0

0 0 0 0 0 0 1

¸

¸

¸

¸

¸

¸

¸

¸

¸

¸

¸

¸

¸

¸

¸

¸

¸

¸

¸

¸

¸

We are used to information symbols being the ﬁrst k symbols of a codeword,

whereas the following (n − k) symbols are parity-check symbols. However,

this order is reversed in our case, the identity matrix being located in the

lower part of the generator matrix. The ﬁrst 4 symbols of every codeword

are therefore parity-check symbols, while the last 7 symbols are information

symbols.

c

1

= x

2

+ x

3

+ x

4

+ x

6

+ x

7

c

2

= x

3

+ x

5

+ x

7

c

3

= x

1

+ x

3

+ x

4

+ x

5

+ x

7

c

4

= x

1

+ x

2

+ x

4

+ x

5

+ x

7

c

5

= x

1

c

6

= x

2

c

7

= x

3

c

8

= x

4

c

9

= x

5

c

10

= x

6

c

11

= x

7

7

3.2.2 Minimum Distance and Weight Enumerator Polyno-

mial

In order to ﬁnd the minimum distance and the weight enumerator polyno-

mial for the given code, we have decided to write a simple algorithm in C.

Its purpose is to compute all possible 2

7

codewords, surveying their Ham-

ming weight and storing the results in a statistics vector. The i-th entry of

this vector contains the number of codewords having weight i and represents

therefore the i-th coeﬃcient of the weight enumerator polynomial.

Since we are dealing with a linear code, it is trivial to ﬁnd the minimum

distance between codewords. As a matter of fact

min

x,y∈C

x=y

d(x, y) = min

x,y∈C

x=y

W

H

(x −y)

but since the code is linear, c = x−y still belongs to C, the equation becomes

min

x,y∈C

x=y

d(x, y) = min

c∈C

c=0

W

H

(c)

and thus looking for the minimum distance between codewords boils down to

looking for the minimum weight over all codewords. This allows us to simply

inspect the weight enumerator polynomial coeﬃcients of the given code: the

index of the minimum-index non-zero coeﬃcient (disregarding coeﬃcient A

0

,

because it corresponds to the 0 codeword) is the sought minimum distance

d.

The result provided by the algorithm are the following:

i A

i

0 1

1 0

2 0

3 13

4 26

5 24

6 24

7 26

8 13

9 0

10 0

11 1

Thus the minimum distance d is 3 and the error correction capabilty of the

given code is 1.

8

Finally the weight enumerator polynomial is

W(x, y) =

11

¸

1=0

A

i

x

i

y

11−i

=

= y

11

+ 13x

3

y

8

+ 26x

4

y

7

+ 24x

5

y

6

+ 24x

6

y

5

+ 26x

7

y

4

+ 13x

8

y

3

+ x

11

3.2.3 Words Decoding

In order to decode R

1

and R

2

, we have to compute their respective syn-

drome, being

R

1

=

1

0

0

0

0

0

0

1

1

1

1

¸

¸

¸

¸

¸

¸

¸

¸

¸

¸

¸

¸

¸

¸

¸

¸

¸

¸

¸

¸

¸

R

2

=

0

1

1

1

0

1

0

0

0

1

1

¸

¸

¸

¸

¸

¸

¸

¸

¸

¸

¸

¸

¸

¸

¸

¸

¸

¸

¸

¸

¸

and E

1

and E

2

their respective error pattern.

So we ﬁnd the syndrome of the ﬁrst word as

S

1

= HR

1

= HE

1

=

1

0

0

0

¸

¸

¸

¸

¸

and we can notice that the computed syndrome is equal to the ﬁrst column

of the parity-check matrix.

Since the syndrome is a linear combination of the columns of the parity-

check matrix H, it is clear that the minimum weight error pattern is

ˆ

E

1

=

1

0

0

0

0

0

0

0

0

0

0

¸

¸

¸

¸

¸

¸

¸

¸

¸

¸

¸

¸

¸

¸

¸

¸

¸

¸

¸

¸

¸

9

Knowing that the error correction capability of the code is 1, we can

correct this error.

The estimated codeword for the ﬁrst received vector is

ˆ

C

1

= R

1

+

ˆ

E

1

=

0

0

0

0

0

0

0

0

1

1

1

¸

¸

¸

¸

¸

¸

¸

¸

¸

¸

¸

¸

¸

¸

¸

¸

¸

¸

¸

¸

¸

and consequentely the sent message was

M

1

=

0

0

0

0

1

1

1

¸

¸

¸

¸

¸

¸

¸

¸

¸

¸

¸

¸

We now proceed to computing the syndrome for the second received

vector

S

2

= HR

2

= HE

2

=

1

0

1

1

¸

¸

¸

¸

¸

The computed syndrome is equal to the 8th column of the parity-check

matrix; therefore the minimum weight error pattern is

ˆ

E

2

=

0

0

0

0

0

0

0

1

0

0

0

¸

¸

¸

¸

¸

¸

¸

¸

¸

¸

¸

¸

¸

¸

¸

¸

¸

¸

¸

¸

¸

10

We can correct the error with the same method as before, obtaing the esti-

mated codeword

ˆ

C

2

= R

2

+

ˆ

E

2

=

0

1

1

1

0

1

0

1

0

1

1

¸

¸

¸

¸

¸

¸

¸

¸

¸

¸

¸

¸

¸

¸

¸

¸

¸

¸

¸

¸

¸

and consequentely the sent message was

M

2

=

0

1

0

1

0

1

1

¸

¸

¸

¸

¸

¸

¸

¸

¸

¸

¸

¸

11

Contents

1 Entropy and Quantity of Information 1

1.1 Handout . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1

1.2 Resolution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1

2 Bynary Enconding 4

2.1 Handout . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4

2.2 Resolution . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4

3 Coding Theory 6

3.1 Handout . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6

3.2 Resolution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6

3.2.1 Generator Matrix Computation . . . . . . . . . . . . . 6

3.2.2 Minimum Distance and Weight Enumerator Polynomial 8

3.2.3 Words Decoding . . . . . . . . . . . . . . . . . . . . . 9

12

- Tornato eSaver
- View Conference Program 2010
- [slides] Il caso Ryanair
- Il caso Ryanair
- [slides] I Mercati Geografici
- La differenziazione geografica della regolamentazione nei mercati d’accesso alla rete fissa
- Characterization of Technological Processes
- Microonde
- Metodi Numerici per Sistemi Differenziali
- Certificazione e Standard nei Sistemi di Telecomunicazioni
- Parallel and Distributed Programming on Low Latency Clusters
- Telecommunication Electronics
- [slides] Microprocessor-Based Systems - 48/32-bit division algorithm - flow chart
- Elettronica Applicata
- [slides] Parallel and Distributed Computing on Low Latency Clusters
- Sistemi Elettronici Digitali
- Introduzione all'Analisi in Tempo-Frequenza
- Controlli Automatici
- Fault Tolerant Processor Using Hybrid Hardware Redundancy
- Video Multiple Description Coding (MDC)
- Progetto di un Filtro Passa-Alto
- Esempi di Relazione di Laboratorio per il Corso di "Elettronica Applicata"
- Radio Frequency Devices - Homework Reports
- Custom Instruction on FPGA for Viterbi Algorithm
- Elettronica Analogica

Here is our solution to 3 execises we needed to resolve in our Information Theory and Codes course. Very slow steps taken for the simple part and very detailed explanation for the more complex one....

Here is our solution to 3 execises we needed to resolve in our Information Theory and Codes course. Very slow steps taken for the simple part and very detailed explanation for the more complex one. You'll find Hentropy and Quantity of Information, binary encoding through a Huffman tree and minimum distance and weight enumerator polynomial for words decoding and generator matrix.

- Blei Jordan Ba
- Properties of Logarithms
- Lesson 01 Modelling Logarithmic and Exponential Functions
- IPC4000
- Background on the solution of DSGE models
- ndx_guntoro (2)
- L.No.23
- Peoplesoft General Ledger
- Exponential and Logarithmic Functions
- Core 2 Revision Booklet
- Learning Curve PPT
- On the possible applications of some theorems concerning the Number Theory to the various mathematical aspects and sectors of String Theory I (2009)
- ASYMPTOTIC OPTIMALITY IN BAYESIAN CHANGE-POINT DETECTION PROBLEMS UNDER GLOBAL FALSE ALARM PROBABILITY CONSTRAINT
- 06176801 v
- Document
- Quick Quant Formulae
- Formulae for QA
- 160 Work Book
- Least Square Equation Solving
- 2_Consistency & Redundancy
- Linear Algebra Solutions
- 10.1.1.63.1051
- pjm-v24-n1-p16-s
- Physics 5013 Mathematical Methods of Physics
- Malaysia 97
- Log Norm
- Iterative Methods for Linear Systems
- Laboratory Report - 1
- Bayesian Estimation
- Information Theory

Are you sure?

This action might not be possible to undo. Are you sure you want to continue?

We've moved you to where you read on your other device.

Get the full title to continue

Get the full title to continue reading from where you left off, or restart the preview.

scribd