Information Theory and Source Coding

Part A: Communication Theory

Dr. ir. Muhammad Nasir

Department of Electrical Engineering

The University of Lahore

October 9, 2021

Lecture 1

1
Introduction

Telecommunication
To communicate over large distance.
Transport of information/data from one point to other.
Telecommunications is the suite of technologies, devices, equipment,
facilities, networks, and applications that support communication at a
distance.

Importance of Telecommunication
Telecommunications industry is broader than it was in the past (multiple
SPs ⇒ telephone companies, cable system operators, ISPs, wireless
carriers, and satellite operators.
Telecommunications provides a technological foundation for societal
communications: e-learning, e-healthcare, e-business and much more.

Dr. ir. Muhammad Nasir (UOL) Part A: Communication Theory October 9, 2021 2 / 38
Some Telecommunication Services

Dr. ir. Muhammad Nasir (UOL) Part A: Communication Theory October 9, 2021 3 / 38
Meta-System

Dr. ir. Muhammad Nasir (UOL) Part A: Communication Theory October 9, 2021 4 / 38
Information Source: Analog Sources

Dr. ir. Muhammad Nasir (UOL) Part A: Communication Theory October 9, 2021 5 / 38
Information Source: Digital Sources

Dr. ir. Muhammad Nasir (UOL) Part A: Communication Theory October 9, 2021 6 / 38
Characteristics of a Digital Source

The alphabet from which the messages are drawn as symbols (M

possible symbols).

The source rate R [symbol/sec, baud].

The probability of occurrence of the respective symbols.

Dr. ir. Muhammad Nasir (UOL) Part A: Communication Theory October 9, 2021 7 / 38
Analog and Digital Transmission Systems

Dr. ir. Muhammad Nasir (UOL) Part A: Communication Theory October 9, 2021 8 / 38
Analog and Digital Transmission Systems Cont’d

Advantages
1 Data from different sources can be jointly transmitted (multiplexing).
2 Relatively insensitive to noise.
3 Cheap electronics.
4 error detection/correction possible with low error probability
5 easy encryption.

Disadvantages
1 More bandwidth needed than for analog signals.
2 Syncronization is difficult.

Dr. ir. Muhammad Nasir (UOL) Part A: Communication Theory October 9, 2021 9 / 38
What is Information?

The reception of a message about a certain event (i.e., experiment’s

result) brings us information. Of course the information is received only if
we do not know the content of the message in advance; this suggests
that the concept of “information” is related to the ignorance about the
result of the event.
We gain information only because of our prior uncertainty about the
event. We can say that the amount of a-priori uncertainty equals the
amount of information delivered by the subsequent knowledge of the
result of the experiment.
Information sources can be modeled by random processes, whose
statistical properties depend on the nature of the information sources
themselves. A discrete time information source X can then be
mathematically modeled by a discrete-time random process {Xi }.

10

Dr. ir. Muhammad Nasir (UOL) Part A: Communication Theory October 9, 2021 10 / 38
Definition of Information contents?

11

Dr. ir. Muhammad Nasir (UOL) Part A: Communication Theory October 9, 2021 11 / 38
Information is Additive and Source Entropy

12

Dr. ir. Muhammad Nasir (UOL) Part A: Communication Theory October 9, 2021 12 / 38
Source Entropy

Tossing a dice:
Outcomes are 1,2,3,4,5,6
Each occurs at probability 1/6
Information provided by tossing a dice is
6
P 6
P
H=− p(i)log2 p(i) = − p(i)log2 p(i)
i=1 i=1
6
1 1
P
=− 6 log2 6 = log2 6 = 2.585bits
i=1

13

Dr. ir. Muhammad Nasir (UOL) Part A: Communication Theory October 9, 2021 13 / 38
Binary Source

14

Dr. ir. Muhammad Nasir (UOL) Part A: Communication Theory October 9, 2021 14 / 38
Channel Capacity (Shannon)

15

Dr. ir. Muhammad Nasir (UOL) Part A: Communication Theory October 9, 2021 15 / 38
Example: Telephone Modem

16

Dr. ir. Muhammad Nasir (UOL) Part A: Communication Theory October 9, 2021 16 / 38
Example: Telephone Modem (Cont’d)

QUIZ!!!! 17

Dr. ir. Muhammad Nasir (UOL) Part A: Communication Theory October 9, 2021 17 / 38
Example: Telephone Modem (Cont’d)

18

Dr. ir. Muhammad Nasir (UOL) Part A: Communication Theory October 9, 2021 18 / 38
Modulation

19

Dr. ir. Muhammad Nasir (UOL) Part A: Communication Theory October 9, 2021 19 / 38
Modulation Cont’d

20

Dr. ir. Muhammad Nasir (UOL) Part A: Communication Theory October 9, 2021 20 / 38
Modulation Cont’d

21

Dr. ir. Muhammad Nasir (UOL) Part A: Communication Theory October 9, 2021 21 / 38
Modulation Cont’d

22

Dr. ir. Muhammad Nasir (UOL) Part A: Communication Theory October 9, 2021 22 / 38
Modulation Cont’d

23

Dr. ir. Muhammad Nasir (UOL) Part A: Communication Theory October 9, 2021 23 / 38
Modulation Cont’d

24

Dr. ir. Muhammad Nasir (UOL) Part A: Communication Theory October 9, 2021 24 / 38
Modulation Cont’d

25

Dr. ir. Muhammad Nasir (UOL) Part A: Communication Theory October 9, 2021 25 / 38
Basic Performance Criteria

26

Dr. ir. Muhammad Nasir (UOL) Part A: Communication Theory October 9, 2021 26 / 38
Sources of Interference and Distortion

27

Dr. ir. Muhammad Nasir (UOL) Part A: Communication Theory October 9, 2021 27 / 38
Telecommunication Engineer Goal

28

Dr. ir. Muhammad Nasir (UOL) Part A: Communication Theory October 9, 2021 28 / 38
Signals and Waveform

29

Dr. ir. Muhammad Nasir (UOL) Part A: Communication Theory October 9, 2021 29 / 38
Physical v.s. mathematical signals

30

Dr. ir. Muhammad Nasir (UOL) Part A: Communication Theory October 9, 2021 30 / 38
Mathematical and physical waveforms

31

Dr. ir. Muhammad Nasir (UOL) Part A: Communication Theory October 9, 2021 31 / 38
Signal Characteristics

32

Dr. ir. Muhammad Nasir (UOL) Part A: Communication Theory October 9, 2021 32 / 38
Signal Characteristics

33

Dr. ir. Muhammad Nasir (UOL) Part A: Communication Theory October 9, 2021 33 / 38
Decibel [dB]

34

Dr. ir. Muhammad Nasir (UOL) Part A: Communication Theory October 9, 2021 34 / 38
Decibel and power amplification

Often, the last term is neglected and this is only allowed when Rload = Rin
The dB is usually related to power.

35

Dr. ir. Muhammad Nasir (UOL) Part A: Communication Theory October 9, 2021 35 / 38
Other decibel notations

36

Dr. ir. Muhammad Nasir (UOL) Part A: Communication Theory October 9, 2021 36 / 38
Other decibel notations (Cont’d)

37

Dr. ir. Muhammad Nasir (UOL) Part A: Communication Theory October 9, 2021 37 / 38
Example decibel notations

38

Dr. ir. Muhammad Nasir (UOL) Part A: Communication Theory October 9, 2021 38 / 38