## Are you sure?

This action might not be possible to undo. Are you sure you want to continue?

**Vittorio Giovara, Alberto Grand, Marina Lemessom
**

19/10/2007

Chapter 1

Entropy and Quantity of

Information

1.1 Handout

Two fair six-faced dices are thrown by Alice and only their sum is commu-

nicated to Bob. Which is the amount of information given with respect to

communicating the distinct occurrences of the two dices?

1.2 Resolution

First we calculate the entropy of the sum of the dices by ﬁlling this table, A

being the possible symbol to be sent and p

a

the probability of that symbol

to be sent.

So we compute the entropy with this formula

H(A) =

M

¸

i=1

p

i

log

1

p

i

where M is the number of diﬀerent symbols and p

i

the probability associated

to the i-th symbol. The base of the logarithm is 2 because we want entropy

to be expressed in bits.

H(A) =

11

¸

i=1

p

i

log

2

1

p

i

=

1

36

log

1

1

36

+

2

36

log

1

2

36

+

3

36

log

1

3

36

+

+

4

36

log

1

4

36

+

5

36

log

1

5

36

+

6

36

log

1

6

36

+

5

36

log

1

5

36

+

+

4

36

log

1

4

36

+

3

36

log

1

3

36

+

2

36

log

1

2

36

+

1

36

log

1

1

36

=

1

A p

a

2

1

36

3

2

36

4

3

36

5

4

36

6

5

36

7

6

36

8

5

36

9

4

36

10

3

36

11

2

36

12

1

36

= 2

1

36

log(36) + 2

2

36

log

1

1

18

+ 2

3

36

log

1

1

12

+ 2

4

36

log

1

1

9

+

+ 2

5

36

log

36

5

+

6

36

log

1

1

6

=

1

18

log(36) +

1

9

log(18) +

+

1

6

log(12) +

2

9

log(9) +

5

18

log

36

5

+

1

6

log(6) =

= 3.27 bits

Then we compute the entropy of a roll of a single dice, using this B

1

alphabet.

B p

b

1

1

6

2

1

6

3

1

6

4

1

6

5

1

6

6

1

6

H(B

1

) =

6

¸

i=1

p

i

log

2

1

p

i

=

2

= 6

1

6

log

1

1

6

=

= log(6) =

= 2.58 bits

Since the symbols of the two dices have the same probability, the alpha-

bet of the second dice B

2

is equal to the ﬁrst.

B

1

= B

2

Thus also the entropy of the second dice is equal to the ﬁrst.

H(B

1

) = H(B

2

)

The entropy of submitting the distinct results of the two dices B

tot

is the

sum of the two entropies.

H(B

tot

) = H(B

1

) + H(B

2

) =

= 2H(B

1

) =

= 2 ∗ 2.58 =

= 5.16 bits

At last we calculate the diﬀerence of the quantity of information when

sending the sum of the dices and the single result of the roll.

H(∆) = H(B

tot

) −H(A) =

= 5.16 −3.27 =

= 1.89 bits

3

Chapter 2

Bynary Enconding

2.1 Handout

Constuct the binary Huﬀman tree for a four letters alphabet with probabil-

ity distribution P = (0.8, 0.1, 0.05, 0.05). Compute the average code word

length and entropy.

2.2 Resolution

We can build the binary Huﬀman tree by associating the alphabet A =

(a

1

, a

2

, a

3

, a

4

) which has a probability distribution of P = (0.8, 0.1, 0.05, 0.05)

to the binary alphabet B = (0, 1).

4

So we can see this association:

a

1

−→ 0

a

2

−→ 10

a

3

−→ 110

a

4

−→ 111

from which we can calculate the average code length

¯

l with this formula

¯

l =

M

¸

i=1

l

i

p

i

¯

l =

4

¸

i=1

l

i

p

i

=

= 1 ∗ 0.8 + 2 ∗ 0.1 + 3 ∗ 0.05 + 3 ∗ 0.05 =

= 0.8 + 0.2 + 0.3 =

= 1.3

And as for the entropy we use the previous formula

H(A) =

M

¸

i=1

p

i

log

1

p

i

H(B) = 0.8 log

1

0.8

+ 0.1 log

1

0.1

+ 0.1 log

1

0.05

=

= 1.022 bits

We observe that the First Shannon Theorem is respected, since

H(A) ≤

¯

l log D ≤ H(A) + log D

1.022 ≤

¯

l ≤ 2.044

5

Chapter 3

Coding Theory

3.1 Handout

Let C be a binary (11, 7, d)-code with parity-check matrix

H =

0 0 1 0 1 0 1 1 0 1 1

0 1 0 1 1 1 1 1 0 0 0

1 0 0 0 0 1 1 1 0 1 1

0 1 1 0 1 0 0 1 1 1 0

¸

¸

¸

¸

¸

1. Find a generator matrix (echelon form) G for C;

2. Find the minimum distance d;

3. Determine the weight enumerator polynomial for C;

4. Decode the following received words:

R

1

= (1000 0000 111) R

2

= (0111 0100 011)

3.2 Resolution

3.2.1 Generator Matrix Computation

It is easy to obtain the generator matrix from the parity-check matrix if we

are able to reduce the latter to an echelon form. In this computation we are

allowed to exchange and add rows.

So we swap r

1

with r

3

and r

2

with r

4

, obtaining

H =

1 0 0 0 0 1 1 1 0 1 1

0 1 1 0 1 0 0 1 1 1 0

0 0 1 0 1 0 1 1 0 1 1

0 1 0 1 1 1 1 1 0 0 0

¸

¸

¸

¸

¸

6

then we add r

3

to r

2

and this new r

2

to r

4

, obtaining

H =

1 0 0 0 0 1 1 1 0 1 1

0 1 0 0 0 0 1 0 1 0 1

0 0 1 0 1 0 1 1 0 1 1

0 0 0 1 1 1 0 1 1 0 1

¸

¸

¸

¸

¸

We have been able to write H in the form

H = (I

n−k

|A)

As a consequence, the corresponding generator matrix will have the form

G =

A

I

k

**Having said that, it is trivial to compute the generator matrix
**

G =

0 1 1 1 0 1 1

0 0 1 0 1 0 1

1 0 1 1 0 1 1

1 1 0 1 1 0 1

1 0 0 0 0 0 0

0 1 0 0 0 0 0

0 0 1 0 0 0 0

0 0 0 1 0 0 0

0 0 0 0 1 0 0

0 0 0 0 0 1 0

0 0 0 0 0 0 1

¸

¸

¸

¸

¸

¸

¸

¸

¸

¸

¸

¸

¸

¸

¸

¸

¸

¸

¸

¸

¸

We are used to information symbols being the ﬁrst k symbols of a codeword,

whereas the following (n − k) symbols are parity-check symbols. However,

this order is reversed in our case, the identity matrix being located in the

lower part of the generator matrix. The ﬁrst 4 symbols of every codeword

are therefore parity-check symbols, while the last 7 symbols are information

symbols.

c

1

= x

2

+ x

3

+ x

4

+ x

6

+ x

7

c

2

= x

3

+ x

5

+ x

7

c

3

= x

1

+ x

3

+ x

4

+ x

5

+ x

7

c

4

= x

1

+ x

2

+ x

4

+ x

5

+ x

7

c

5

= x

1

c

6

= x

2

c

7

= x

3

c

8

= x

4

c

9

= x

5

c

10

= x

6

c

11

= x

7

7

3.2.2 Minimum Distance and Weight Enumerator Polyno-

mial

In order to ﬁnd the minimum distance and the weight enumerator polyno-

mial for the given code, we have decided to write a simple algorithm in C.

Its purpose is to compute all possible 2

7

codewords, surveying their Ham-

ming weight and storing the results in a statistics vector. The i-th entry of

this vector contains the number of codewords having weight i and represents

therefore the i-th coeﬃcient of the weight enumerator polynomial.

Since we are dealing with a linear code, it is trivial to ﬁnd the minimum

distance between codewords. As a matter of fact

min

x,y∈C

x=y

d(x, y) = min

x,y∈C

x=y

W

H

(x −y)

but since the code is linear, c = x−y still belongs to C, the equation becomes

min

x,y∈C

x=y

d(x, y) = min

c∈C

c=0

W

H

(c)

and thus looking for the minimum distance between codewords boils down to

looking for the minimum weight over all codewords. This allows us to simply

inspect the weight enumerator polynomial coeﬃcients of the given code: the

index of the minimum-index non-zero coeﬃcient (disregarding coeﬃcient A

0

,

because it corresponds to the 0 codeword) is the sought minimum distance

d.

The result provided by the algorithm are the following:

i A

i

0 1

1 0

2 0

3 13

4 26

5 24

6 24

7 26

8 13

9 0

10 0

11 1

Thus the minimum distance d is 3 and the error correction capabilty of the

given code is 1.

8

Finally the weight enumerator polynomial is

W(x, y) =

11

¸

1=0

A

i

x

i

y

11−i

=

= y

11

+ 13x

3

y

8

+ 26x

4

y

7

+ 24x

5

y

6

+ 24x

6

y

5

+ 26x

7

y

4

+ 13x

8

y

3

+ x

11

3.2.3 Words Decoding

In order to decode R

1

and R

2

, we have to compute their respective syn-

drome, being

R

1

=

1

0

0

0

0

0

0

1

1

1

1

¸

¸

¸

¸

¸

¸

¸

¸

¸

¸

¸

¸

¸

¸

¸

¸

¸

¸

¸

¸

¸

R

2

=

0

1

1

1

0

1

0

0

0

1

1

¸

¸

¸

¸

¸

¸

¸

¸

¸

¸

¸

¸

¸

¸

¸

¸

¸

¸

¸

¸

¸

and E

1

and E

2

their respective error pattern.

So we ﬁnd the syndrome of the ﬁrst word as

S

1

= HR

1

= HE

1

=

1

0

0

0

¸

¸

¸

¸

¸

and we can notice that the computed syndrome is equal to the ﬁrst column

of the parity-check matrix.

Since the syndrome is a linear combination of the columns of the parity-

check matrix H, it is clear that the minimum weight error pattern is

ˆ

E

1

=

1

0

0

0

0

0

0

0

0

0

0

¸

¸

¸

¸

¸

¸

¸

¸

¸

¸

¸

¸

¸

¸

¸

¸

¸

¸

¸

¸

¸

9

Knowing that the error correction capability of the code is 1, we can

correct this error.

The estimated codeword for the ﬁrst received vector is

ˆ

C

1

= R

1

+

ˆ

E

1

=

0

0

0

0

0

0

0

0

1

1

1

¸

¸

¸

¸

¸

¸

¸

¸

¸

¸

¸

¸

¸

¸

¸

¸

¸

¸

¸

¸

¸

and consequentely the sent message was

M

1

=

0

0

0

0

1

1

1

¸

¸

¸

¸

¸

¸

¸

¸

¸

¸

¸

¸

We now proceed to computing the syndrome for the second received

vector

S

2

= HR

2

= HE

2

=

1

0

1

1

¸

¸

¸

¸

¸

The computed syndrome is equal to the 8th column of the parity-check

matrix; therefore the minimum weight error pattern is

ˆ

E

2

=

0

0

0

0

0

0

0

1

0

0

0

¸

¸

¸

¸

¸

¸

¸

¸

¸

¸

¸

¸

¸

¸

¸

¸

¸

¸

¸

¸

¸

10

We can correct the error with the same method as before, obtaing the esti-

mated codeword

ˆ

C

2

= R

2

+

ˆ

E

2

=

0

1

1

1

0

1

0

1

0

1

1

¸

¸

¸

¸

¸

¸

¸

¸

¸

¸

¸

¸

¸

¸

¸

¸

¸

¸

¸

¸

¸

and consequentely the sent message was

M

2

=

0

1

0

1

0

1

1

¸

¸

¸

¸

¸

¸

¸

¸

¸

¸

¸

¸

11

Contents

1 Entropy and Quantity of Information 1

1.1 Handout . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1

1.2 Resolution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1

2 Bynary Enconding 4

2.1 Handout . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4

2.2 Resolution . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4

3 Coding Theory 6

3.1 Handout . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6

3.2 Resolution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6

3.2.1 Generator Matrix Computation . . . . . . . . . . . . . 6

3.2.2 Minimum Distance and Weight Enumerator Polynomial 8

3.2.3 Words Decoding . . . . . . . . . . . . . . . . . . . . . 9

12

- Caller Id Fsk
- Lecture_2_
- Itc Unit III,IV,V Qns
- Information Theory and Coding
- Channel Error Coding
- ch10.ppt
- Information Theory
- Information Theory and Coding
- SML07 - Walker - Codes and Curves
- Solution to Information Theory
- information theory coding 6 sem ec notes
- Code-division multiple access
- Solution Manual of Elements of Information Theory
- Information Theory and Entropy
- Information Theory and coding
- Indices and Logarithms Explained With Worked Examples_SSZakari
- Benford's Law Exploration
- Notes Math
- SBP F4 P2 Final Exam 2007
- A12 Logarithms BP 9-22-14
- TM Matematika X IPA 8
- C2 revision
- exp_log_eqns_n
- Lossless Compression brief
- Past 3 Manual
- Form 4 Add Maths Note
- section 5 4 notes
- 129847906-Complete-Advanced-Level-Mathematics-Pure-Mathematics.pdf
- Physics Planning Practice 1
- Logs

Skip carousel

- tmp8D02
- tmpA3C6
- frbclv_wp1990-02.pdf
- wp02_08
- A Mathematical Theory of Communication
- tmpB3E1.tmp
- tmpADD4.tmp
- tmpE701.tmp
- frbrich_wp94-1.pdf
- tmpBB9E.tmp
- An Internal Intrusion Approach And Self Protection For Generating Synthetic Log File Using Forensic Techniques
- tmpDB1F.tmp
- UT Dallas Syllabus for math1314.521 05u taught by Bentley Garrett (btg032000)
- tmpC4B6
- As 1046.3-1991 Letter Symbols for Use in Electrotechnology Logarithmic Quantities and Units
- tmp685C.tmp
- Trusted Cloud Computing with Secure Aid Agency for a Couple of Clouds Collaboration
- Tmp 3970
- Technological Change, Human Capital Structure, and Multiple Growth Paths
- tmp1DD0
- tmp338E.tmp
- 66547_1980-1984
- Tmp 6460
- Money & Sustainability
- Logjam attack

- [slides] I Mercati Geografici
- Metodi Numerici per Sistemi Differenziali
- Tornato eSaver
- Elettronica Applicata
- Certificazione e Standard nei Sistemi di Telecomunicazioni
- Parallel and Distributed Programming on Low Latency Clusters
- View Conference Program 2010
- Telecommunication Electronics
- [slides] Parallel and Distributed Computing on Low Latency Clusters
- [slides] Il caso Ryanair
- Il caso Ryanair
- La differenziazione geografica della regolamentazione nei mercati d’accesso alla rete fissa
- Microonde
- [slides] Microprocessor-Based Systems - 48/32-bit division algorithm - flow chart
- Characterization of Technological Processes
- Elettronica Analogica
- Fault Tolerant Processor Using Hybrid Hardware Redundancy
- Porting serial Fortran 77 numerical computation code for OpenMP applications
- Real-Time Operating Systems Laboratory Report
- Radio Frequency Devices - Homework Reports
- Designing a custom DLX processor with Very Long Instruction Word support
- Progetto di un Filtro Passa-Alto
- Custom Instruction on FPGA for Viterbi Algorithm
- Esempi di Relazione di Laboratorio per il Corso di "Elettronica Applicata"
- Sistemi Elettronici Digitali
- Controlli Automatici
- Introduzione all'Analisi in Tempo-Frequenza
- GLE-MiPS Compiler Requisites
- Real-Time Operating Systems Homework Report
- Video Multiple Description Coding (MDC)

Sign up to vote on this title

UsefulNot usefulClose Dialog## Are you sure?

This action might not be possible to undo. Are you sure you want to continue?

Close Dialog## This title now requires a credit

Use one of your book credits to continue reading from where you left off, or restart the preview.

Loading