UNIVERSITY OF ANBAR LECTURE NOTES ON INFORMATION THEORY FOR 4th CLASS STUDENTS

COLLEGE OF ENGINEERING BY: NASER AL FALAHI ELECTRICAL ENGINEERING

WEEK 5

Shannon Coding Theorem

Consider a typical sequence S N of N messages from the source. Because the n
messages (of probability P1 , P 2 , … , P n ) occur NP 1 , NP 2 , NP n times, and because each
message is independent, the probability of occurrence of a typical sequence S N is given
by
( ) = ( ) ( ) . . . ( )

Because all possible sequences of N messages from this source have the same
composition, all the sequences (of N messages) are equiprobable, with probability
P ( S N ) . We can consider these long sequences as new messages (which are now
equiprobable). To encode one such sequence we need L N binary digits, where

1
= log
( )

By substitution:

1
= log = ( )

Shannon Coding Algorithm

1. Assign corresponding probabilities or frequency counts to each symbol so that

each symbol’s occurrence is known.
2. Sort the lists of symbols according to its probability, with the most frequently
occurring symbols at the top and the least common at the bottom, i.e.,
(descending sort).
3. Calculate the coding index (bi) that gives the necessary number of bits required
to encode the message, where:

1 1
log ≤ bi ≤ 1 + log
( ) ( )

4. Calculate the factor (Fi) by direct accumulation of the probabilities till the n
symbol.

23
UNIVERSITY OF ANBAR LECTURE NOTES ON INFORMATION THEORY FOR 4th CLASS STUDENTS
COLLEGE OF ENGINEERING BY: NASER AL FALAHI ELECTRICAL ENGINEERING

5. Convert Fi to its binary equivalent Ci :

=( )

Example: A source emits 7 symbols with probabilities { 0.3 , 0.2 , 0.15 , 0.12 , 0.1 , 0.08 ,
0.05} construct the Shannon code for this source.

Sol:

According to the agreed procedure, sort the probabilities in descending order,

First of all, we have to calculate the code index bi:

For example, b3 calculated

1 1
log ≤ b ≤ 1 + log
( ) ( )

2.73 ≤ b ≤ 3.73 , this gives b3 = 3 (taking the integer of course)

To be calculated for all symbols.

Next, we have to calculate Fi,

(Fi) is the accumulation of probabilities till (i-1), F3 = P1 +P2 = 0.3+0.2=0.5

F6 = P1 +P2+ P3+ P4+ P5=0.3+0.2+0.15+0.12+0.1=0.87, and so on…

Finally, Ci should be found, and it is calculated as follow:

For example, C4

⍟ Keep in mind that C4 must be with a 4 bits long.

0.65*2 = 1.3, so, 1 will be assigned to the code, and the rest (0.3) will be multiplied again
by 2.

0.3*2=2=0.6, so, 0 will be assigned ( as it didn't exceed 1 ) , and again

0.6*2=1.2, so, 1 will be assigned, & the rest (0.2) will be multiplied by 2

0.2*2=0.4, so, 0 will be assigned to the code.

0.4 now will be discarded (ignored), as the limit of bi was reached.

24
UNIVERSITY OF ANBAR LECTURE NOTES ON INFORMATION THEORY FOR 4th CLASS STUDENTS
COLLEGE OF ENGINEERING BY: NASER AL FALAHI ELECTRICAL ENGINEERING

The table below shows the validation of the above calculations along with all individual
codes.

Symbol probability Required bits (bi) Accumulation (Fi) Generated Code Ci

1 0.3 2 0 00
2 0.2 3 0.3 010
3 0.15 3 0.5 100
4 0.12 4 0.65 1010
5 0.1 4 0.77 1100
6 0.08 4 0.87 1101
7 0.05 5 0.95 11110

The source entropy H(m) = 2.603 bits.

The length of the generated code:

= 0.3 × 2 + 0.2 × 3 + 0.15 × 3 + 0.12 × 4 + 0.1 × 4 + 0.08 × 4 + 0.05 × 5 = 3.1

.
The coding efficiency = 3.1
∗ 100% = 83.96774 %

The redundancy =1− = 1 − 0.8396 = 0.1604

H.W. Compare between Shannon code & Fano code in term of efficiency & redundancy
for the above source.

H.W. A communication source emits 7 equiprobable symbols, Construct Shannon code.

25