Source Coding of Discrete Sources

A discrete source is a source that produces finite set of messages x1,x2, x3,
………, xn with probabilities p(x1), p(x2), p(x3),….., p(xn). The source
coder will transform these messages into a finite sequence of digits, called
the codeword of the message. If binary digits (bits) are used in this
codeword, then we obtain what is called " Binary Source Coding".
Nonbinary (such as ternary, quaternary,…etc) source coding is also
possible if the elements of this codeword are nonbinary digits.

Source of Source coder

information
x1……. p(x1) C1…….. ℓ1 codeword
x2…….. p(x2) C2 …….. ℓ2
. .
. .
xn…….. p(xn) .

H(x) Lc
bit/message Codewords
Digit/message
‫او ﺟﺪول اﻟﺘﺮﻣﯿﺰ‬ ‫ﺗﺼﻤﯿﻢ ﺟﺪول اﻟﺘﺤﻮﯾﻞ‬

The selection of codewords for different messages xi, is done according to

the following two considerations:
1- The average code length Lc must be as minimum as possible. This
average length is given by:
Lc   i  i 1  i p( xi ) digits/message
n
‫ﻣﺠﻤﻮع اﻟﻘﯿﻢ ﻋﻠﻰ اﻟﻌﺪد‬

Where li is the length of the codeword for message xi (li is in bits for binary
coding, or in digits for nonbinary coding).
2- The codewords at the receiver must be uniquely decodable. ‫ﻛﻠﻤﺔ اﻟﺴﺮ ﻟﮭﺎ ﺻﻮرة وﺣﯿﺪة ﻋﻨﺪ اﻟﻄﺮف اﻻﺧﺮ‬
To understand the above two considerations, the following example is
given:
Ex: The code table for a certain binary code is given as:

Message xi p(xi) Codeword Ci li(bits)

x1 0.2 0 1
x2 0.1 10 2
x3 0.4 110 3
x4 0.3 111 3

[1]-Find the average code length, then comment.

[2]-If the received data stream is 1 0 1 1 0 0 0 1 1 1 1 1 0…….., check if
this code is uniquely decodable or not, then comment. ‫اﻟﺠﺪول ﻣﻮﺟﻮد ﻟﺪى اﻟﻄﺮﻓﯿﻦ‬
Solution:
[1]- Lc   i  i 1  i p( xi ) =0.2+2*0.1+3*0.4+3*0.3=2.5bits/message
n

[2]- using the given code table, then:

‫ﻧﺒﺪأ ﺑﺎﻟﺪﺟﺖ اﻻول وﻧﻘﺎرن ﻣﻊ اﻟﺠﺪول ﺛﻢ ﻧﺎﺧﺬ‬
1 0 | 1 1 0 | 0 | 0 | 1 1 1 | 1 1 0 |…….., ‫اﻟﺜﺎﻧﻲ وھﻜﺬا وﻋﻨﺪ اﻟﺘﻄﺎﺑﻖ ﻧﻀﻊ ﺧﻄﻮط وھﻤﯿﺔ‬
‫ﺗﺤﻮﯾﻞ ﻣﻌﺎﻛﺲ‬.. ‫اي اﻧﮫ ﻧﺤﺼﻞ ﻋﻠﻰ ﺣﺪ وﺣﯿﺪ‬
x2 | x3 | x1| x1| x4 | x3 |……., ‫ﻣﻦ ﻛﻠﻤﺔ اﻟﺴﺮ اﻟﻰ اﻟﻤﺴﺞ‬

Hence the code is uniquely decodable, since the receiver get at only one
possible message stream.
Comments: [1]- For previous example, the code is not optimum in terms of
Lc, i.e. it could be less by another redistribution of the codewords. In fact,
we can reduce Lc by giving less li for xi with higher p(xi) such that Lc is
reduced. For example, the given code table is modified as:

Message xi p(xi) Codeword Ci li(bits)

x3 0.4 0 1
x4 0.3 10 2
x1 0.2 110 3
x2 0.1 111 3 ‫ﻗﺒﻞ اﻟﺘﻌﺪﯾﻞ‬
‫ﻛﻠﻤﺔ اﻟﺴﺮ ﻟﻠﺘﺮﺗﯿﺐ اﻻﻣﺜﻞ‬... ‫اﻟﺠﺪول ﺑﻌﺪ اﻟﺘﻌﺪﯾﻞ‬ ‫اﻟﺘﺮﺗﯿﺐ ﻋﺸﻮاﺋﻲ وﻟﯿﺲ ﺗﻨﺎزﻟﻲ‬

This gives Lc=0.4+0.3*2+0.2*3+0.1*3=1.9 bits/message which is less than

before.
[2]-: Condition for uniquely decodable code The condition for uniquely
decodable code is that if xi is given a codeword Ci of length li bits, then
these li bits must not be the beginning (from the left) of any other codeword
Cj of higher length lj for message xj. Applying this for previous example,
then if ''0'' is a codeword, then no other codeword of higher length starts
with ''0''. Also if ''10'' is a codeword, then no other codeword of higher
length starts with ''10'' and so on. ‫ ﻋﻨﺪھﺎ ﺑﻘﯿﺔ اﻟﻤﺴﺠﺎت ﯾﺠﺐ ان ﻻﺗﺒﺪأ ﺑﺘﻠﻚ اﻟﺪﺟﺖ‬.. ‫اﻓﻀﻞ ﺟﺪول ﻋﻨﺪ ﺗﺨﺼﯿﺺ ﻛﻠﻤﺔ ﺳﺮ ﻟﻤﺴﺞ ﻣﻌﯿﻦ‬

Ex: Check if the following code is uniquely decodable or not:

x1 0
x2 10
x3 101
x4 111
Solution: This code is not uniquely decodable since ''10'' is a codeword for
x2 , while the codeword for x3 starts with ''10''.
 Coding efficiency and redundancy: ‫اﻟﻜﻔﺎﺋﺔ واﻟﺘﻜﺮار‬
A code with average code length Lc digits has coding efficiency:
H ( x)

Lc log2 D where:
‫ﺗﻨﺎﺳﺐ طﺮدي ﻣﻊ اﻟﻤﻌﻠﻮﻣﺎت وﻋﻜﺴﻲ ﻣﻊ طﻮل اﻟﻜﻮد وﻋﺪد اﻟﺪﺟﺖ‬ ‫اﻟﻜﻔﺎﺋﺔ = دﺧﻞ اﻟﺠﺪول اﻟﻰ اﻟﺨﺮج ﻻن اﻗﺼﻰ ﻗﯿﻤﺔ ھﻲ واﺣﺪ‬

H(x) is source entropy in bits/message,

D=coding size ( D=2 for binary coding, D=3 for ternary coding, and so
on….).
Lc has the units of the code size, ( i.e. bits/message if D=2, ternary
digits/message if D=3, and so on…..).
H (X )
For binary coding only (D=2), then:  
Lc
(Note: If the type of the code is not given, then assume it BINARY).
The code redundancy is simply R=1-. ‫اﻟﻤﺘﻤﻤﺔ‬

Note: eata = input/output = H/Laverage = more compression (less L) = higher eata

The interpretation of code efficiency comes from the following discussion:

Source of Source coder

information
x1 C1
x2 C2
. .
. .
xn .

H(x) Lc
bit/message Digit/message

The average amount of information from the source output is H(X)

bits/message. The average amount of information at the source coder is Lc
digit/message or Lc*log2(D) bits/message. The factor log2(D) changes the
units from digits of D-sized into bits. Hence the ratio between H(X) and
Lc*log2(D) will represent the efficiency of the source coder. To more
explain the idea of source efficiency, suppose the source produces M
no. of messages, then the amount of information produced at the source output
messages
will be M*H(X) bits, while the amount of information at source encoder
output is M*Lc digits or M*Lc*log2(D) bits, so the ratio
M * H (X ) H (X )

input/output

M * Lc log2 D Lc * log2 D will represent the efficiency of the code and

relation

the redundancy will represent the amount of unwanted increase in

information due to source coding.
 Source Codes Types

1] Fixed length Codes:

This is used when the source produces almost equiprobable messages,
p(x1)p(x2)……..p(xn), then l1= l2 =…….= ln=Lc. This Lc is given for D-
sized code by:
1- Lc = logD(n) if n=Dr (r is an integer)
2- Lc= Int[logD(n)]+1 if n≠Dr . (Int=integer part)
For binary coding , then:
1-Lc=log2(n)=H(X)|max bits/message if n=2,4,8,16,….which gives
=100% .
2- L c=Int[log2(n)]+1 if n≠2r which gives less .

Ex A source produces 10 equiprobable messages. Find the efficiency of a

fixed length code used to encode the source output.

Solution Since the size of the code is not given, then we assume
D=2(binary coding). Since equiprobable, then p(xi)=0.1 for i=1,2,…10, and
Lc=Int[log2(10)]+1=4 bits/message and ‫ﻋﺪد اﻟﻤﺴﺞ ﻟﯿﺲ ﺿﻤﻦ ﻣﺠﻤﻮﻋﺔ ﺑﻮر اﻻﺛﻨﯿﻦ‬
=H(X)/Lc=log2(10)/4=83%
A straight binary code may be used here as codewords for fixed length
code:
xi p(xi) Codeword Ci
x1 0.1 0000
x2 0.1 0001
x3 0.1 0011
. . .
x10 0.1 1001

Ex: Find the efficiency of a fixed length binary code used to encode
messages obtained from throwing a fair die (a) once. (b) twice. (c) 3 times.
X XX XXX
‫ﻛﻞ رﻣﯿﺔ واﺣﺪة ﺗﻤﺜﻞ ﻣﺴﺞ ﻮاﺣﺪة و ﺗﺤﻮي ﻋﻠﻰ ﺛﻼث ﺳﻤﺒﻼت‬
Solution:
(a) n=6, possible equiprobable outputs when throwing the fair die once,
hence: Lc= Int[log2(6)]+1=3 bits/message ‫ﻧﺠﻌﻞ اﻟﺒﺴﻂ واﻟﻤﻘﺎم ﺑﻨﻔﺲ اﻟﻮﺣﺪات‬
=H(X)/Lc=log2(6)/3=86.1%
(b)For two throws, then possible messages will be 6*6=36 equiprobable,
hence: : Lc= Int[log2(36)]+1=6 bits/message ‫ﻋﻨﺪھﺎ ﻧﻌﺪل اﻟﻜﺴﺮ ﻣﻊ اﺿﺎﻓﺔ واﺣﺪ‬.. ‫ﻋﻨﺪﻣﺎ ﯾﻌﻄﻲ اﻟﻠﻮغ ﻛﺴﺮا‬
=6 bits/(2-symbols)=3bits/symbol [note that each message
consists of 2 symbols].
While H(X)=log2(6)= bits/symbol, then:
 =H(X)/Lc= log2(6)/3 = 86.1%
(c) For three throws, then possible messages will be 6*6*6=216 and :
: Lc= Int[log2(216)]+1=8 bits/message
=8 bits/(3-symbols)
=(8/3) bit/symbol [each message consists of 3 symbols], then:
=H(X)/Lc= log2(6)/(8/3) = 96.9% ‫اﻟﻤﻌﻠﻮﻣﺎت ﻣﻈﻐﻮطﺔ اﻛﺜﺮ‬

Homework: Repeat previous example for ternary coding.

Hint ; D=3; Lc = log3(n) if n=3r (r is an integer)
Or Lc= Int[log3(n)]+1 if n≠3r
H ( x)

Lc log 2 D

2]Variable Length Codes: ‫ﺛﻼث طﺮق ﻟﺘﺼﻤﯿﻢ ﺟﺪول اﻟﺘﺤﻮﯾﻞ اﻻﻣﺜﻞ‬

When message probabilities are not equal, then we use variable length
codes. These codes are sometimes called minimum redundancy codes.
Three types of these codes are given:

a)- Shannon Code:

For messages x1, x2, x3,…….,xn with prob. p(x1), p(x2), p(x3)……p(xn),
then for binary coding (D=2):
Step 1: Arrange the messages in a decreasing order of probabilities,
Step 2: Find the codeword length li such that:
1- li = -log2 p(xi) if p(xi) = (½)r =½,¼,⅛,……
2- li = Int[-log2 p(xi)]+1 if p(xi) ≠ (½)r
i 1

Step 3: Define wi   p ( x k )
‫ﻣﺟﻣوع اﻟﺎﺣﺗﻣﺎﻟﯾﺎت اﻟﺳﺎﺑﻘﺔ‬
1 > wi 0, then the codeword of xi is the
k 1
binary equivalent of wi resolved into li bits. C = Binary{ Int(2^L * W) }

Ex: Develop Shannon code for the following set of messages:

x1 x2 x3 x4 x5 x6
p(X) = [ 0.4 0.25 0.15 0.1 0.07 0.03]
then find: [a] coding efficiency and redundancy. [b]p(0) and p(1) at encoder
output. [c]amount of information produced at encoder output if the source
produces 10000 message.
Solution: ‫ﺣﺿر ﺟدول ﻓﺎرغ ﻟﻠﺣﻘول اﻟﻣﻌروﻓﺔ ﺳﺎﺑﻘﺎ‬
First, we notice that messages are already arranged in a decreasing order.
Next: l1=Int[-log2 0.4]+1=2 bits, l2= -log2 0.25=2 bits,
l3=Int[-log2(0.15)]+1=3 bits, and so on.
21

Next: w1=0, w2   p ( xk )  p ( x1 )  0.4 ,

k 1
31
w3   p ( xk )  p ( x1 )  p ( x2 )  0.4  0.25  0.65 and so on.
k 1
Next to find C1 then change w1=0 into 2 bits binary(l1 =2 bits) which is
''00'', to find C2 then change w2=0.4 into 2 bits binary (l2= 2 bits) which is
''01'' ( two times multiplication of 0.4 by 2 and take the integer part)., to
find C3, then change w3=0.65 into 3 bits (l3=3 bits), which is ''101'' ( 3 times
multiplication of 0.65 by 2 and take the integer part). Continuing to the
other messages, then we arrange the results in the following table:
C = Binary{ Int(2^L * W) }

‫ﻣﺟﻣوع اﻟﺎﺣﺗﻣﺎﻟﯾﺎت اﻟﺳﺎﺑﻘﺔ‬

xi p(xi) Li wi Ci 0i
x1 0.4 2 0 00 2
x2 0.25 2 0.4 01 1 C = Binary{ Int(2^L * W) }
= Bin{ Int(2^2 * 0.4 ) }
x3 0.15 3 0.65 101 1 = Bin { Int(1.6) }
= Bin {1} = 01
x4 0.1 4 0.8 1100 2
‫اﻟﺎﺻﻔﺎر ﻟﻠﺳﺎر ﻟﺎﺗؤﺛر‬
x5 0.07 4 0.9 1110 1
x6 0.03 6 0.97 111110 1

0i is the number of 0's in the codeword of xi.

Note that the two considerations are fulfilled ( less Li for higher p(xi) and
the code satisfies the uniquely decodable condition).
a)-To find code efficiency then H(X)=2.1918 bits/message, and: for diff. prob. H =Sum {p log p}
Lc  i1  i p( xi )  2 * 0.4  2 * 0.25  ...  2.61 bits / message
6

Then: =H(X)/Lc=2.1918/2.61=83.9%. = info. / max-inform.

 0 p( x ) i i
0.8  0.25  ..... = ‫ﻣﻌدل طول اﻟﻣﺳﺞ‬/ ‫ﻋدد اﻟﺎﺻﻔﺎر ﻟﻠﻣﺳﺞ *اﺣﺗﻣﺎﻟﯾﺔ اﻟﻣﺳﺞ‬
b)- p(0)    0.574 , and
i 1
P(0) = 0-average / bit average per message
Lc 2.61
6

1 i p ( xi )
p (1)  i 1
( 1i is no of 1's):
Lc
P(0) for x1 is 2 * 0.4 = 0.8
or simply: p(1)=1-p(0)=0.426 ‫ﯾﻣﻛن ﻓﮭم اﻟﻘﺎﻧون ﻣن اﻟوﺳطﯾن ﺑﺎﻟطرﻓﯾن‬
c)- amount of information at encoder output is simply: 10000 message ..‫اﺣﺗﻣﺎل ﺿﮭور ﺻﻔر ﺑﺎﻟﺧرج‬

10000*Lc=10000*2.61= 26.1 kbits

 Homework1: Modify Shannon coding into ternary coding. ( hint: base of
log will be 3, i.e. log3 when finding Li , also the condition for Li will be
1
( ) r and wi equation is as before, but changing it into ternary word of
3
length Li by successive multiplication by 3 taking the integer part).

Homework2:
Repeat previous example using ternary Shannon code.

b) Shannon-Fano Code ( Fano Code):

The procedure for binary Fano code is given as follows:
Step 1: arrange the messages in a decreasing order of prob.
Step 2: find out a point in the decreasing order such that the sum of
probabilities upward is almost equal to the sum of prob downward.
Step 3: assign all messages upward as ''0'' and all messages downward as
''1''.
Step 4: repeat steps 2 & 3 many times on the upward and downward parts
until all the messages are separated.

Ex: Develop Fano code for the following set of messages: ‫ﻓﺻل اﻟرﺳﺎاﺋل ﺑﺎﻟﺗﺑﺎع ﺑﺧط وھﻣﻲ ﯾﻣﺛل ﻣرﻛز اﻟﺛﻘل‬
x1 x2 x3 x4 x5 x6
p(X) = [ 0.25 0.2 0.18 0.15 0.12 0.1]
then find coding efficiency and redundancy.
Solution: messages are already given in a decreasing order of prob. To
carry out step 2, then we notice that point between x2 and x3 is the best
choice since the sum of prob upward is 0.45 while the sum of prob
downward is 0.55. we assign upward by ''0'' and downward by ''1''.
x1 0.25 0
x2 0.2 0
‫ﺧط وھﻣﻲ ﻟﻣرﻛز اﻟﺛﻘل‬
x3 0.18 1
gravity center
x4 0.15 1
x5 0.12 1
x6 0.1 1
Next we repeat the step above on the upward and downward portions,
where the upward is simply separated into two parts assigning up( message
x1) by ''0'' and down (message x2) by ''1'' where these two messages are
completely separated.
 x1 0.25 00 2
x2 0.2 01 1
x3 0.18 1
x4 0.15 1
x5 0.12 1
x6 0.1 1

The remaining downward part consists of x3, x4, x5, x6, where the best split
is between x4 and x5 (partial sum up is 0.33 and partial sum down is 0.22),
then assign up (messages x3 and x4) by ''0'' and down (messages x5 and x6)
by ''1'':

x1 0.25 00 2
x2 0.2 01
1
x3 0.18 10
x4 0.15 10 3
x5 0.12 11
x6 0.1 11
Finally x3 and x4 are directly separated by ''0'' and ''1''. And x5 and x6 are
also directly separated by ''0'' and ''1''.

‫ﻣﻊ ﻛل ﺧط ﯾﺗم ﺗﻌﯾن اﻻﺻﻔﺎر ﻟﻠرﺳﺎﺋل ﻓوق اﻟﺧط واوﺣﺎد ﻟﻠرﺳﺎﺋل ﺗﺣت اﻟﺧط‬
xi p(xi) Ci Li = Int{ -log p} +1

x1 0.25 00 2 2
x2 0.2 01 2 1
x3 0.18 100 3 4
x4 0.15 101 3 3
x5 0.12 110 3
4
x6 0.1 111 3

(numbers appear on arrows represent the order of plotting that splitting

arrow in the procedure).
Then: H(X)=2.519 bits/message and: = sum (P log P)
Lc= 2*(0.25+0.2)+3*(0.18+0.15+0.12+0.1)=2.55 bits/message = sum (l log P)
And =H(X)/Lc = 2.519/2.55=98.7%
Ex Develop Fano code for the following set of messages:
x1 x2 x3 x4 x5 x6 x7 x8
p(X)= [0.4 0.2 0.12 0.08 0.08 0.04 0.04 0.04]
 1. Shannon C= Bin Int (2^L * W)
2. Fano C= Messages Separation by Successive Center gravity.
then find coding efficiency.
Solution: using previous steps, then the following code table is obtained:
xi p(xi) Ci Li
x1 0.4 0 1
1
x2 0.2 100 3 3
x3 0.12 101 3 2
x4 0.08 1100 4 4
x5 0.08 1101 4 3
x6 0.04 1110 4 4
x7 0.04 11110 5 5
x8 0.04 11111 5
Note that less  is obtained (higher Lc) if starting arrow is made between x2
and x3 which gives the same balance of sum of prob compared with that
used in above solution. 0.4 ------- 0.6.
(numbers appear on arrows represent the order of plotting that splitting
arrow in the procedure).
H(X)= 2.5 bits/message, Lc=2.56 bits/message, then:
=H(X)/Lc = 2.5/2.56=97.6%

Ex: Modify Fano procedure for ternary coding. Separate to three groups each time use two arrows if possible
till we separat all messages

Solution: First, we find out two points(arrows) in each step that split the
sum of prob into almost three equal parts assigning them as ''0'', ''1'', and
''2''. Applying this onto previous example, then:

xi p(xi) Ci
‫ﯾﺗم اﺳﺗﺧدام ﺧطﯾن ﻟﻠﻔﺻل ﻓﻲ ﻛل ﺧطوة ﻟﻔﺻل اﻟﻣﺳﺟﺎت اﻟﻰ ﺛﻼث‬
x1 0.4 0 ‫ اذا ﻛﺎن اﻟﺑﺎﻗﻲ ﻓﻘط رﺳﺎﻟﺗﯾن ﻋﻧدھﺎ ﯾﺗم اﻟﻔﺻل اﻟﻰ ﻓﻘط‬..‫ﻛروﺑﺎت‬
x2 0.2 1 2 ‫ دون‬1 ‫ و‬0 ‫ﻛروﺑﯾن ﻓرﻋﯾﺔ واﺳﺗﺧدام اﻟﻠوﺟك‬

x3 0.12 1
x4 0.08 2
x5 0.08 2
x6 0.04 2
x7 0.04 2
x8 0.04 2

Then x1 is directly separated. Messages x2 and x3 are directly separated by

one arrow.
xi p(xi) Ci
x1 0.4 0
x2 0.2 10
 x3 0.12 11
x4 0.08 2
x5 0.08 2
x6 0.04 2
x7 0.04 2
x8 0.04 2

The remaining messages are separated by two arrows, one between x4 and
x5 and the other between x5 and x6

xi p(xi) Ci
x1 0.4 0
x2 0.2 10
x3 0.12 11
x4 0.08 20
x5 0.08 21
x6 0.04 22
x7 0.04 22
x8 0.04 22

Finally the last three messages are separated by two arrows:

xi p(xi) Ci step number;

x1 0.4 0 1
x2 0.2 10 2

x3 0.12 11 1
x4 0.08 20 3
x5 0.08 21 3
x6 0.04 220 4

x7 0.04 221 4
x8 0.04 222

(numbers appear on arrows represent the order of plotting that splitting

arrow in the procedure).
c)- Huffman code:
Procudure(Binary coding):

Step1: Arrange messages in a decreasing order of prob.

Step2: The two lowest prob messages are joined (sum their prob), assign
''0'' for one of them and ''1'' for the other.
Step3: rewrite messages once again in a decreasing order replacing the sum
of prob of step2 as a prob of one message, hence reducing number of
messages by one.
Step4:repeat steps 2 and 3 many times until you end up with total prob of
unity 1.00.
Step5: the codeword for each message is read from marked ''0'' and ''1''
following the arrows from left to right and writing codeword bits from right
to left (this is done to satisfy the condition for uniquely decodable code).

Ex: Develop binary Huffman code for the following set of messages:
x1 x2 x3 x4 x5 x6
p(X) = [ 0.4 0.25 0.15 0.1 0.07 0.03]
then find coding efficiency.

Solution The sum of the last two lowest prob is 0.1, rewrite messages and
replace these two messages by their sum of prob.

x1 0.4 0.4

x2 0.25 0.25

x3 0.15 0.15

x4 0.1 0.1

x5 0.07 0
0.1
x6 0.03 1

In the new sequence, the sum of the two lowest prob is 0.2, so we rewrite
replacing them by their sum whose position in the new sequence is between
0.25 and 0.15, then:
 x1 0.4 0.4 0.4

x2 0.25 0.25 0.25

x3 0.15 0.15 0.2

x4 0.1 0.1 0 0.15

x5 0.07 0
0.1 1
x6 0.03 1

In the new sequence, the sum of the two lowest prob is 0.35, so we rewrite
replacing them by their sum whose position in the new sequence is between
0.4 and 0.25, then:
x1 0.4 0.4 0.4 0.4

x2 0.25 0.25 0.25 0.35

x3 0.15 0.15 0.2 0 0.25

x4 0.1 0.1 0 0.15 1

x5 0.07 0
0.1 1
x6 0.03 1

Finally, in new sequence, the sum of the two lowest prob is 0.6, so we
rewrite replacing them by their sum whose position in the new sequence is
at the 1st position, then:
six messages == five transitions

x1 0.4 0.4 0.4 0.4 0.6 0

1.00
x2 0.25 0.25 0.25 0.35 0 0.4 1

x3 0.15 0.15 0.2 0 0.25 1

read codewords

x4 0.1 0.1 0 0.15 1 write codewords

correct the position in

the new sequence
x5 0.07 0
0.1 1
x6 0.03 1
two lowest prob mes.
replacing them by their sum
code table

* If the path doesn't go through this Transition * Following the arrows from the left to
Then don't mention and go to the next. the right and writing codeword bits from the right to the left
To read codewords, then the codeword for x1 is simply ''1'' following the
arrow from the left to the right. The codeword for x2 is ''01'', following the
arrows from the left to the right and writing codeword bits from the right to
the left. The codeword for x3 is ''001'' following the arrows from the left to
the right and writing codeword bits from the right to the left, and so on we
can arrange the code table as shown below:

xi p(xi) Ci Li
x1 0.4 1 1
x2 0.25 01 2
x3 0.15 001 3
x4 0.1 0000 4
x5 0.07 00010 5
x6 0.03 00011 5

This code has Lc=2.25 bits/message.

H(X)=2.1918 bits/message.
And =H(X)/Lc = 2.198/2.25=97.4%

Ex: Modify Huffman binary coding into ternary coding, then repeat
previous example using ternary Huffman coding.

Solution: For ternary Huffman coding, then we join or sum the last three
lowest prob messages. This will reduce no of messages in each step by two,
so to ensure ending up the procedure with 3 messages corresponding to
''0'',''1''', and ''2'', then we must notice the number of messages from the
beginning. If the number of messages is even, then add a dummy message
with prob of zero so that we end up with 3 messages. If the number of
messages is already odd, then leave it as it is.
To apply above on the previous example, we notice that there are 6
messages (even), then we add another message x7 with prob of 0.00, and
carry out the same previous procedure joining up 3 lowest prob messages at
a time.
 Message prob

x1 0.4 0.4 0.4 0

x2 0.25 0.25 0.35 1 1.00

x3 0.15 0.15 0 0.25 2

code table

x4 0.1 0.1 1

x5 0.07 0 0.1 2

x6 0.03 1

x7 0.00 2

this gives a code table as shown below:

xi p(xi) Ci Li (ternary digit)

x1 0.4 0 1
x2 0.25 2 1
x3 0.15 10 2
x4 0.1 11 2
x5 0.07 120 3
x6 0.03 121 3
Hence Lc=0.4+0.25+2*(0.15+0.1)+3*(0.07+0.03)=1.45 ternary
digit/message.
And =H(X)/ (log2(3)*Lc) = 2.198/(1.58*1.45)=95.6%
Note that it is less than the efficiency for binary coding. This makes us
think of what is the necessary and sufficient condition to make the coding
efficiency 100%. This will be given in the next example.

Ex: show that in any coding procedure the coding efficiency is 100% if:
1
p ( xi )  ( ) r ( r is an integer for all i=1,2,3,….n).
D
Solution:
1 r
Note that if p ( x i )  ( ) then:
D
Li = -logD(p(xi))= logD( Dr) = r which is an integer for all i.
Take for example, the source:
p(x)=[½ ¼ ⅛ 1/16 1/16], which satisfies the condition for 100% binary
coding efficiency. Using say Fano code, then:
Fano code table;
x1 0.5 0 1 step No. ;

x2 0.25 10 2
2

x3 0.125 110 3
3

x4 0.0625 1110 4
4

x5 0.0625 1111 4

Lc=0.5+2*0.25+3*0.125+4*(0.0625+0.0625)=1.875 bits/message =H(X)

Hence =H(X)/ Lc=100%

Homework: repeat previous example for the source using ternary coding:
p(X)=[1/3 1/3 1/9 1/9 1/27 1/27 1/27]

Source Extension: ‫ﺗﻤﺪﯾﺪ اﻟﻤﺼﺪر‬

Sometimes, the source produces symbols with very extreme probabilities
like p(X)=[0.9 0.1] or p(X)=[ 0.9 0.08 0.02]. ..etc. In such a case and
using any of the previous variable length coding methods, the efficiency
obtained is so small. Take for example, the source:
p(X)=[0.9 0.1]: , then using say Huffman procedure, then:

x1 0.9 0 x1 is ''0''
1.00 x2 is ''1''
x2 0.1 1

This gives Lc=1 bit/symbol, while:

H(X)=0.469 bits/symbol. =H(X)/ Lc=0.469/1=46.9%

Now if we group two symbols and regard them as one message( this
grouping is called source extension), then and assuming statistically
independent symbols, then, the prob of a group of two symbols will be the
joint prob:
Message prob
x1x1 0.81 0.81 0.81 0
1.00
x1x2 0.09 0.1 0 0.19 1

x2x1 0.09 0 0.09 1

x2x2 0.01 1
joint prob

this gives the code table:

x1x1 0.81 0 1
x1x2 0.09 11 2
x2x1 0.09 100 3
x2x2 0.01 101 3
Lc=0.81+0.18+3*0.1=1.29 bits/message
=1.29 bits/2symbols=0.645 bits/symbol
=H(X)/ Lc=0.469/0.645=72.7% which is better than one symbol/message.
Shannon showed that increasing the number of symbols/message will
increase the efficiency approaching the 100% as a limit. To check this, let
us group 3 symbols/message, then:
Message prob Message prob
x1x1x1 0.729 x1x1x1 0.729
x1x1x2 0.081 arranging in decreasing x1x2x1 0.081
x1x2x1 0.081 order x2x1x1 0.081
x1x2x2 0.009 x1x1x2 0.081
x2x1x1 0.081 x1x2x2 0.009
x2x1x2 0.009 x2x2x1 0.009
x2x2x1 0.009 x2x1x2 0.009
x2x2x2 0.001 x2x2x2 0.001

using Huffman coding then:

 0 Just one transition

x1x1x1 0.729 0.729 0.729 0.729 0.729 0.729 0.729

1.00
x1x2x1 0.081 0.081 0.081 0.081 0.081 0.190 0.271
0 1

x2x1x1 0.081 0.081 0.081 0.081 0.109 0.081

0

x1x1x2 0.081 0.081 0.081 0.081 0.081

x1x2x2 0.009 0.01 0.018 0.028

1

x2x2x1 0.009 0.009 0.01

1

x2x1x2 0.009 0.009

0 Doesn't go through this Transition
So don't mention and go to the next.
x2x2x2 0.001

this gives the code table:

x1x1x1 0.729 0 1
x1x2x1 0.081 11 2
x2x1x1 0.081 101 3
x1x1x2 0.081 1000 4
x1x2x2 0.009 100100 6
x2x2x1 0.009 100101 6
x2x1x2 0.009 100110 6
x2x2x2 0.001 100111 6

This gives Lc=0.729+0.081(2+3+4)+6*(3*0.009+0.001)=1.626 bits/messgae

=1.626 bits/3 symbols=0.542 bits/symbol
=H(X)/ Lc=0.469/0.542=86.5%, which is better than before.