COMM 552

Elgazeera High Institute for

Engineering and Technology

Information Theory and Coding

Lecture – 1

Introduction

Dr. Ahmed Abouelmagd

 Introduction
Course Contents

Introduction: Uncertainty, Information, Entropy and its

properties. Source coding: Shannon coding Prefix coding, First
Shannon theorem, Huffman coding, discrete memoryless
channels, Binary symmetric channel, Mutual information and
its properties. Channel capacity, Channel coding, Second
Shannon theorem, Mutual information. Channel capacity,
Compression of information. Linear block codes, Cyclic codes,
Well-Known Block codes, Convolution codes: Code tree,
Trellis and state diagram, Maximum likelihood decoding of
convolution codes.
Prerequisites:
Course Code/Name:
COMM 350/ Signals & Systems
COMM 440/Digital Circuit Design
Textbook :
John-Proakis “Digital-Communications” 5th Ed 2015
ISBN-13: 978-0072957167
ISBN-10: 0072957166
https://www.amazon.com/Digital-Communications-John-
Proakis
References:
Digital Communications - Glover and Grant; Pearson
Ed. 2nd Ed 2008
 Student Assessment Methods

Assessment Weight Schedule

Year Work & Quizzes 20% weekly
Mid-Term Exam 20% W8
Final Exam 60% W15
Total 100% ----
 Communication Systems

• Communication systems are designed to transmit an information

generated by a source to some destination
•There exists between the source and the destination a
communicating channel affected by various disturbances
• Source: deliver Information as a sequence of source symbols.
• Source coding: provide “in average” the shortest description
of the emitted sequences ⇒ higher information rate
• Channel: generates disturbances
• Channel Coding: protect the information from errors induced
by the channel, by voluntary adding redundancy to
information ⇒ higher quality of transmission
 The designer of a communication system will be asked to:

1. Insure a high quality of transmission with an “as low as

possible” error rate in the reproduced sequence
2. Provide the higher Information rate through the channel because:
- The use of a channel is costly.
-The channel employs different limited resources (time,
frequency, power …).
Information sources are classified as:

Source definition
Analog : Emit a continuous – amplitude, continuous – time
electrical wave from.
Discrete : Emit a sequence of letters of symbols.
 Measure the information:

Concept of the amount of information:

Illustrative example : Cairo weather forecast in winter time:
Message (1): It is a cold day
Contains very little information since it is usual event that the
weather in Cairo is ‘cold’ during winter season.
Message (2): It is a cloudy day
Contains more information, since it is not often event that occurs
in Cairo during winter season.
Message (3): It is a rainy day
Contains more information, since it is not often event that occurs
in Cairo during winter season.
Message (4): Possible snow flurries
Contains even more information, since it is a rare event that
occurs in Cairo during winter season.
Therefore:
* The amount of information is related to the probability of
occurrence of the event.
* Message associated with an event ‘least likely to occur’
contains most information.
The Concept of the amount of information can now be formed
in terms of probabilities as follows:
Assume that an information source emits one of ‘q’ possible
messages: m1, m2 …… mq with p1, p2 …… pq as their probs. of
occurrence.
the information content of the kth message, can be written as
1
I(mk) α
𝑝𝑘
Also to satisfy the intuitive concept, of information.
I(mk) must be zero as pq =1
Therefore it is required that:

(1)
 - When two independent messages are received, the total
information content is the Sum of the information conveyed by
each of the messages. Thus, we have

(2)

According to (1) & (2) We can define a measure of information

as:
𝟏
𝑰 𝒎𝒌 = 𝒍𝒐𝒈 = - 𝒍𝒐𝒈 𝒑𝒌
𝒑𝒌

Unit of information measure:

Log Base - 10 : Hartley / decit
Log Base - 2 : bit
Example 2:

A source puts out one of five possible messages m1; m2; m3; m4;
m5; during each message interval. The probs. of these messages are
p1 = 1/2; p2 =1/4; p3 =1/8; p4=1/16; p5 =1/16;
Calculate the information content of these messages:
a. In bit units
b. In Hartley units

Solution 2:

𝟏
𝑰 𝒎𝒌 = 𝒍𝒐𝒈 = - 𝒍𝒐𝒈 𝒑𝒌
𝒑𝒌

Log Base - 10 : Hartley / decit

Log Base - 2 : bit
Solution 2:
𝟏
𝑰 𝒎𝒌 = 𝒍𝒐𝒈 = - 𝒍𝒐𝒈 𝒑𝒌
𝒑𝒌

a. In bit units Log Base - 2

b. (H.W.) In Hartley / decit units Log Base - 10

 Entropy
Entropy: Is the Average information content per symbol.
Suppose that:
- A source is emitting one of M possible symbols s0, s1 ….. sM in a
statically independent sequence
- Let p1, p2, …….. pM be the probabilities of occurrence of the M-
symbols respectively
- During a long period of transmission, a sequence of N symbols
have been generated.

Therefore:
s1 will occur Np1 times
s2 will occur Np2 times
::
si will occur Npi times
The information content of the ith symbol is:
𝟏
𝑰 𝒔𝒊 = 𝒍𝒐𝒈 bits
𝒑𝒊

So, piN occurrences of si contributes an information content of:

𝟏
piN 𝑰 𝒔𝒊 = piN 𝒍𝒐𝒈 bits
𝒑𝒊

Total information content of the message is = Sum of the

contribution due to each of:
𝑴 𝟏
𝑰𝒕𝒐𝒕𝒂𝒍 = 𝒊=𝟏 piN 𝒍𝒐𝒈
𝒑𝒊
bits
𝑰𝒕𝒐𝒕𝒂𝒍
Entropy H = Average information content =
𝑵
𝑴 𝟏
𝑯= 𝒊=𝟏 pi 𝒍𝒐𝒈
𝒑𝒊
bits per symbol
Entropy will attain its maximum value, when the symbol
probabilities are equal,
1
i.e., when p1 = p2 = p3 = …… = pM =
M
Example 3:
Consider a source emits two messages of probs. ‘p’ and ‘(1-p)’.
Starting from entropy definition find the source entropy
maximum value. Check your results with the max entropy
definition.

𝑴 𝟏
𝑯= 𝒊=𝟏 pi 𝒍𝒐𝒈
𝒑𝒊
bits per symbol
Solution 3:
Source entropy
𝑴 𝟏
𝑯= 𝒊=𝟏 pi 𝒍𝒐𝒈
𝒑𝒊
bits per symbol

The source emits two messages of probs. ‘p’ and ‘(1-p)’

1 1
𝐻 = p 𝑙𝑜𝑔 + (1−p) 𝑙𝑜𝑔
p (1−p)

At p = 0, H = 0 and
At p = 1, H = 0
Entropy maximum value is for equi-
probable symbols, i.e. at p=1/2

1 1
𝐻 = 𝑙𝑜𝑔2 𝟐 + 𝑙𝑜𝑔2 𝟐 =1
2 2
Check of results with the max entropy definition:

𝐻𝑚𝑎𝑥 = 𝑙𝑜𝑔2 𝟐 =1
 Information rate
If the source is emitting symbols at a fixed rate of rs symbols/sec,
the average source information rate R is defined as:
R = rs . H bits/sec
Example 4:
Consider a discrete memoryless source with a source alphabet:
A = { s0, s1, s2} with respective probs. p0 = ¼, p1 = ¼, p2 = ½. Find:
a. The entropy of the source.
b. The information rate if the source is emitting symbols at a fixed
rate of 1 symbols/sec
Solution 4:

𝑴 𝟏
𝑯= 𝒊=𝟏 pi 𝒍𝒐𝒈
𝒑𝒊
bits per symbol

R = rs . H bits/sec
Solution 4:
𝑴 𝟏
𝑯= 𝒊=𝟏 pi 𝒍𝒐𝒈
𝒑𝒊
bits per symbol

R = rs . H bits/sec

R = rs . H (A) = 1 × 1.5 = 1.5 bits/sec

Example 5:
An analog signal is band limited to B =5 Hz, sampled at the
Nyquist rate, and the samples are quantized into 4-levels. The
quantization levels Q1, Q2, Q3, and Q4 (messages) are assumed
independent and occur with probs. P1 = P4 =1/8 and P2 = P3 =3/8.
Find:
a. The entropy H and the information rate R of the source.
b. The values of H and R, if the quantities levels are so chosen
that they are equally likely to occur.
Solution 5:
𝑴 𝟏
𝑯= 𝒊=𝟏 pi 𝒍𝒐𝒈
𝒑𝒊
bits per symbol

R = rs . H bits/sec
Solution 5.a:
𝑴 𝟏
𝑯= 𝒊=𝟏 pi 𝒍𝒐𝒈
𝒑𝒊
bits per symbol

R = rs . H bits/sec

At Nyquist rate rs = 2 B symbols/sec

R = rs . H = 2B × 1.8 = 2 × 5 × 1.8 = 18 bits/sec

Solution 5.b:
𝑴 𝟏
𝑯= 𝒊=𝟏 pi 𝒍𝒐𝒈
𝒑𝒊
bits per symbol

R = rs . H bits/sec

At Nyquist rate rs = 2 B symbols/sec

R = rs . H = 2B × 2 = 2 × 5 × 2 = 20 bits/sec