INFORMATION THEORY AND

CODING
5th SEM E&C

JAYANTHDWIJESH H P M.tech (DECS)

Assistant Professor – Dept of E&C

B.G.S INSTITUTE OF TECHNOLOGY (B.G.S.I.T)

B.G Nagara, Nagamangala Tq, Mandya District- 571448
 FORMULAS FOR REFERENCE

MODULE – 2 (source coding)

 Entropy of source or Average information content of the source.

𝒒 𝟏 𝑵 𝟏
H(S) = 𝒊=𝟏 𝑷𝒊 𝐥𝐨𝐠( 𝑷 ) bits/symbol or H(S) = 𝑲=𝟏 𝑷𝑲 𝐥𝐨𝐠 𝟐 𝟏 ( 𝑷 ) bits/symbol
𝒊 𝑲

 Average length
𝒒 𝑵
L= 𝒊=𝟏 𝑷𝒊 𝒍𝒊 bits/symbol or L = 𝒊=𝟏 𝑷𝒊 𝒍𝒊 bits/symbol
 Source or code efficiency
𝑯(𝑺) 𝑯(𝑺)
𝜼𝑺 = X 𝟏𝟎𝟎% or 𝜼𝑪 = X 𝟏𝟎𝟎%
𝑳 𝑳
 Source or code redundancy
𝑯(𝑺) 𝑯(𝑺)
𝑹 𝜼𝑺 = 1- 𝜼𝑺 = (1 - 𝑳
) X 𝟏𝟎𝟎% or 𝑹 𝜼𝑪 = 1- 𝜼𝑪 = (1 - 𝑳
) X 𝟏𝟎𝟎%

 Compute the number of stages required for the encoding operation, which is
given by
𝑵−𝒓 𝒒−𝒓
𝒏=
𝒓−𝟏
or = 𝒓−𝟏
 The probability of “0”s and “1”s and “2” s in the code are found using the
formulas
𝟏 𝒒
P (0) = 𝑳 𝒊=𝟏[𝐍𝐮𝐦𝐛𝐞𝐫 𝐨𝐟 "0" s in the code for 𝑿𝒊 ] [𝒑𝒊 ] or
𝟏 𝑵
P (0) = 𝑳 𝒊=𝟏[𝐍𝐮𝐦𝐛𝐞𝐫 𝐨𝐟 "0" s in the code for 𝑿𝒊 ] [𝒑𝒊 ] .
𝟏 𝒒
P (1) = 𝑳 𝒊=𝟏[𝐍𝐮𝐦𝐛𝐞𝐫 𝐨𝐟 "1" s in the code for 𝑿𝒊 ] [𝒑𝒊 ] or
𝟏 𝑵
P (1) =
𝑳 𝒊=𝟏[𝐍𝐮𝐦𝐛𝐞𝐫 𝐨𝐟 "1" s in the code for 𝑿𝒊 ] [𝒑𝒊 ] .
𝟏 𝒒
P (2) =
𝑳 𝒊=𝟏[𝐍𝐮𝐦𝐛𝐞𝐫 𝐨𝐟 "2" s in the code for 𝑿𝒊 ] [𝒑𝒊 ] or
𝟏 𝑵
P (2) = 𝑳 𝒊=𝟏[𝐍𝐮𝐦𝐛𝐞𝐫 𝐨𝐟 "2" s in the code for 𝑿𝒊 ] [𝒑𝒊 ] .

 The variance of the word length is calculated from

Var ( 𝒍𝒊 ) = E [( 𝒍𝒊 − 𝐋 )𝟐 = 𝒒
𝒊=𝟏 𝑷 𝒊 ( 𝒍𝒊 − 𝐋 ) 𝟐
 The Smallest integer value of 𝒍𝒊 if found using
𝟏 𝟏
𝟐 𝒍𝒊  or 𝒍𝒊  𝒍𝒐𝒈𝟐 𝑷
𝑷𝒊 𝒊

 The average length 𝑳𝟐 of the 2nd extension is given by

𝒒 𝑵
𝑳𝟐 = 𝒊=𝟏 𝑷𝒊 𝒍𝒊 bits/symbol or 𝑳𝟐 = 𝒊=𝟏 𝑷𝒊 𝒍𝒊 bits/symbol
 The average length 𝑳𝟑 of the 3rd extension is given by
𝒒 𝑵
𝑳𝟑 = 𝒊=𝟏 𝑷𝒊 𝒍𝒊 bits/symbol or 𝑳𝟑 = 𝒊=𝟏 𝑷𝒊 𝒍𝒊 bits/symbol
 The entropy of the 2nd extended source is calculated as
H (𝑺𝟐 ) = 2 H(S)
 The entropy of the 3rd extended source is calculated as
H (𝑺𝟑 ) = 3H(S)
 Source or code efficiency of the 2nd extended source is
𝐇 (𝑺𝟐 ) 𝐇 (𝑺𝟐 )
𝜼(𝟐) 𝑺 = X 𝟏𝟎𝟎% or 𝜼(𝟐) 𝑪 = X 𝟏𝟎𝟎%
𝑳𝟐 𝑳𝟐

 Source or code redundancy of the 2nd extended source is

𝐇 (𝑺𝟐 ) 𝐇 (𝑺𝟐 )
𝑹(𝟐) 𝜼𝑺 = 1- 𝜼(𝟐) 𝑺 = (1 - 𝑳𝟐
) X 𝟏𝟎𝟎% or 𝑹(𝟐) 𝜼𝑪 = 𝟏 − 𝜼(𝟐) 𝑪 = (1 -
𝑳𝟐
)
𝐗 𝟏𝟎𝟎%
 Source or code efficiency of the 3rd extended source is
𝐇 (𝑺𝟑 ) 𝐇 (𝑺𝟑 )
𝜼(𝟑) 𝑺 = X 𝟏𝟎𝟎% or 𝜼(𝟑) 𝑪 = X 𝟏𝟎𝟎%
𝑳𝟑 𝟑

 Source or code redundancy of the 3rd extended source is

𝐇 (𝑺𝟑 ) 𝐇 (𝑺𝟑 )
𝑹(𝟑) 𝜼𝑺 = 1- 𝜼(𝟑) 𝑺 = (1 - 𝑳𝟐
) X 𝟏𝟎𝟎% or 𝑹(𝟑) 𝜼𝑪 = 𝟏 − 𝜼(𝟑) 𝑪 = (1 -
𝑳𝟑
)
𝐗 𝟏𝟎𝟎%
𝟑
 The average length 𝑳 of the Huffman ternary code is given by
𝑳(𝟑) = 𝒒
𝒊=𝟏 𝑷𝒊 𝒍𝒊 trinits /Msg- symbol or 𝑳
(𝟑)
= 𝑵
𝒊=𝟏 𝑷𝒊 𝒍𝒊 trinits / Msg- symbol
𝟒
 The average length 𝑳 of the Huffman quaternary code is given by
𝑳(𝟒) = 𝒒
𝒊=𝟏 𝑷𝒊 𝒍𝒊 quaternary digits /Msg- symbol or

𝑳(𝟒) = 𝑵
𝒊=𝟏 𝑷𝒊 𝒍𝒊 quaternary digits / Msg- symbol
 The entropy in ternary units/ message symbol is found by using equation
𝐇(𝐒)
𝐇𝟑 (S) = ternary units/ message symbol or
𝒍𝒐𝒈𝟐 𝟑

𝑵 𝟏
𝐇𝟑 (S) = 𝑲=𝟏 𝑷𝑲 𝐥𝐨𝐠 𝟑 𝟏 ( 𝑷 ) ternary units/ message symbol or
𝑲

𝒒 𝟏
𝐇𝟑 (S) = 𝒊=𝟏 𝑷𝒊 𝐥𝐨𝐠 𝟑 𝟏 ( 𝑷 ) ternary units/ message symbol
𝒊

 The entropy in quaternary units/ message symbol is found by using equation

𝐇(𝐒)
𝐇𝟒 (S) = quaternary units/ message symbol or
𝒍𝒐𝒈𝟐 𝟒
 𝑵 𝟏
𝐇𝟒 (S) = 𝑲=𝟏 𝑷𝑲 𝐥𝐨𝐠 𝟒 𝟏 ( ) quaternary units/ message symbol or
𝑷𝑲
𝒒 𝟏
𝐇𝟒 (S) = 𝒊=𝟏 𝑷𝒊 𝐥𝐨𝐠 𝟒 𝟏 ( ) quaternary units/ message symbol
𝑷𝒊

 Source or code efficiency of the ternary is given by

𝐇𝟑 (𝐒) 𝐇𝟑 (𝐒)
𝜼 𝒔(𝟑) = X 𝟏𝟎𝟎% or 𝜼 𝒄(𝟑) = X 𝟏𝟎𝟎% or
𝑳(𝟑) 𝑳(𝟑)
𝐇𝟑 (𝐒) 𝐇𝟑 (𝐒)
𝜼𝑺 = X 𝟏𝟎𝟎% or 𝜼𝑪 = X 𝟏𝟎𝟎%
𝑳 𝑳
 Source or code efficiency of the quaternary is given by
𝐇𝟒 (𝐒) 𝐇𝟒 (𝐒)
𝜼 𝒔(𝟒) = X 𝟏𝟎𝟎% or 𝜼 𝒄(𝟒) = X 𝟏𝟎𝟎% or
𝑳(𝟒) 𝑳(𝟒)
𝐇𝟒 (𝐒) 𝐇𝟒 (𝐒)
𝜼𝑺 = X 𝟏𝟎𝟎% or 𝜼𝑪 = X 𝟏𝟎𝟎%
𝑳 𝑳
 Source or code redundancy of the ternary is given by
𝐇𝟑 (𝐒)
𝑹 𝜼𝒔 = 1- 𝜼 𝒔(𝟑) = (1 - ) X 𝟏𝟎𝟎% or
(𝟑) 𝑳(𝟑)
𝐇𝟑 (𝐒)
𝑹 𝜼 𝒄(𝟑) = 𝟏 − 𝜼 𝒄(𝟑) = (1 - ) 𝐗 𝟏𝟎𝟎% or
𝑳(𝟑)
𝐇𝟑 (𝐒) 𝐇𝟑 (𝐒)
𝑹 𝜼𝑺 = 1- 𝜼𝑺 = (1 - 𝑳
) X 𝟏𝟎𝟎% or 𝑹 𝜼𝑪 = 1- 𝜼𝑪 = (1 - 𝑳
) X 𝟏𝟎𝟎%

 Source or code redundancy of the quaternary is given by

𝐇𝟒 (𝐒)
𝑹 𝜼𝒔 = 1- 𝜼 𝒔(𝟒) = (1 - ) X 𝟏𝟎𝟎% or
(𝟒) 𝑳(𝟒)
𝐇𝟒 (𝐒)
𝑹 𝜼 𝒄(𝟒) = 𝟏 − 𝜼 𝒄(𝟒) = (1 - ) 𝐗 𝟏𝟎𝟎% or
𝑳(𝟒)
𝐇𝟒 (𝐒) 𝐇𝟒 (𝐒)
𝑹 𝜼𝑺 = 1- 𝜼𝑺 = (1 - 𝑳
) X 𝟏𝟎𝟎% or 𝑹 𝜼𝑪 = 1- 𝜼𝑪 = (1 - 𝑳
) X 𝟏𝟎𝟎%