INFORMATION

THEORY AND
CODING
• Entropy
• Probabilistic information measure
 INFORMATION?

• Information - organised acts that can e interpreted as a message, idea or intelligence in

inromation theory.
• is the source of a communication system, whether it is analog or digital.

• message - electrical signal, speech/voice, text

 WHAT IS INFORMATION THEORY?

• Is a mathematical approach to study the coding of information along with the quantification
,storage and communication of information
 INFORMATION THEORY: UNCERTAINITY

•consider:
•m = {mo , m1 , m2 ..... mn }
•p = {po , p1 , p2 ....... pn } n

•total probability is given by :

i 1
pi

•when an event has a chance of occuring, then it has an uncertainity e.g p= 0.5
•if an event does not occur or is certain then then there is a lot of information and minimum probability
 INFORMATION THEORY: MEASURE OF
INFORMATION
• Measure of information
• it is the infomation content of a message

• consider source emitting independent messages with probability as follows

m = {mo , m1 , m2 ..... mn }

p = {po , p1 , p2 ....... pn }

n
total probability is given by : p
i 1
i
 INFORMATION THEORY: MEASURE OF INFORMATION

• formula :

1 base units
I k  log 2 ( ) 2 bits
pk e nat
10 decit
1
log 2 ( )
pk
Ik 
log 2
 INFORMATION THEORY: PROPERTIES
• uncertanity is directly prpotional to information

• if receiver knows what is being transmitted then information is zero

• total information is equal to sum o indepedant information

• if there are M=2N equiproable messages, then the amount o infromation carried by each message
will be N bits
 ENTROPY(H)

•  Defined as the average information per symbol

• Entropy, H =
• log() bits/symbol
 PROPERTIES

•1.   Entropy is zero if event is certain.

2. If symbols are equally probable (equiprobable) then entropy is maximum and it is given by H
=M
 SOURCE EFFICIENCY AND REDUNDANCY

•  Source efficiency, η =
• Redundancy, Re = 1 - η
 INFORMATION RATE

• The rate of transfer of a message

• R = H×r
• Where H = entropy
r = sampling rate