Professional Documents
Culture Documents
Lecture 5 Information Theory
Lecture 5 Information Theory
Baud rate =
1000 bauds per second (baud/s)
Bit rate = 1000 x 4 = 4000 bps
Example
The bit rate of a signal is 3000. If
each signal unit carries 6 bits, what is
the baud rate?
Example
23
Information Content
Hence, the elementary unit of information is a
binary unit: a bit, which can be either 1 or 0;
true or false; yes or no, black and
white, etc.
One of the basic postulates of information
theory is that information can be treated like a
measurable physical quantity, such as density
or mass.
24
Information Content
Suppose you flip a coin one million times and
write down the sequence of results. If you want
to communicate this sequence to another
person, how many bits will it take?
If it's a fair coin, the two possible outcomes,
heads and tails, occur with equal probability.
Therefore each flip requires 1 bit of information
to transmit. To send the entire sequence will
require one million bits.
25
Information Content
Suppose the coin is biased so that heads
occur only 1/4 of the time, and tails
occur 3/4. Then the entire sequence can
be sent in 811,300 bits, on average This
would seem to imply that each flip of
the coin requires just 0.8113 bits to
transmit.
How can you transmit a coin flip in less
than one bit, when the only language
available is that of zeros and ones?
26
Obviously, you can't. But if the goal is to
transmit an entire sequence of flips, and the
distribution is biased in some way, then you
can use your knowledge of the distribution to
select a more efficient code.
Another way to look at it is: a sequence of
biased coin flips contains less "information"
than a sequence of unbiased flips, so it should
take fewer bits to transmit.
Information Content
Information Theory regards information as only
those symbols that are uncertain to the receiver.
For years, people have sent telegraph messages,
leaving out non-essential words such as "a" and
"the."
In the same vein, predictable symbols can be left
out, like in the sentence, "only information
essentially to understand must be tranmitted.
Shannon made clear that uncertainty is the very
commodity of communication.
28
Information Content
Suppose we transmit a long sequence of one
million bits corresponding to the first example.
What should we do if some errors occur
during this transmission?
If the length of the sequence to be
transmitted or stored is even larger that 1
million bits, then 1 billion bits what should
we do?
29
Two main questions of
Information Theory
What to do if information gets corrupted by
errors?
How much memory does it require to store
data?
Both questions were asked and to a large
degree answered by Claude Shannon in his
1948 seminal article:
use error correction and data compression
30
Shannons basic principles of Information
Theory
Shannons theory told engineers how much
information could be transmitted over the
channels of an ideal system.
He also spelled out mathematically the principles
of data compression, which recognize what the
end of this sentence demonstrates, that only
information essentially to understand must be
transmitted.
He also showed how we could transmit
information over noisy channels at error rates we
could control.
31
Why is the Information Theory Important?
Modems
satellite communications
Data storage
Deep space communications
Wireless technology
32
Information Theory
- Quantification of information
Channels
A channel is used to get information
across:
0,1,1,0,0,1,1 Receiver
Source
binary channel
1p
A noisy, symmetric binary 0
p
0
channel applies a bit-flip p
01 with probability p: 1
1p
1
36
Channel Rate
When correcting errors, we have to be mindful of
the rate of the bits that you use to encode one bit
(in the previous example we had rate 1/3).
37
Error Correction by Shannon
Shannons basic observations:
38
A model for a Communication System
The communications systems are of a statistical
nature.
That is, the performance of the system can never
be described in a deterministic sense; it is always
given in statistical terms.
A source is a device that selects and transmits
sequences of symbols from a given alphabet.
Each selection is made at random, although this
selection may be based on some statistical rule.
39
A model for a Communication System
The channel transmits the incoming symbols
to the receiver. The performance of the
channel is also based on laws of chance.
If the source transmits a symbol A, with a
probability of P{A} and the channel lets
through the letter A with a probability
denoted by P{A|A}, then the probability of
transmitting A and receiving A is P{A}P{A|A}
40
A model for a Communication System
The channel is generally lossy: a part of the
transmitted content does not reach its
destination or it reaches the destination in a
distorted form.
41
A model for a Communication System
A very important task is the minimization of
the loss and the optimum recovery of the
original content when it is corrupted by the
effect of noise.
A method that is used to improve the
efficiency of the channel is called encoding.
An encoded message is less sensitive to noise.
42
A model for a Communication System
Decoding is employed to transform the
encoded messages into the original form,
which is acceptable to the receiver.
Encoding: F: I F(I)
Decoding: F-1: F(I) I
43
A Quantitative Measure of Information
Suppose we have to select some equipment
from a catalog which indicates n distinct
models: x1 , x2 ,..., xn
The desired amount of information I ( xk )
associated with the selection of a particular
model xk must be a function of the
probability of choosing xk :
I ( xk ) f P xk
44
A Quantitative Measure of Information
If, for simplicity, we assume that each one of
these models is selected with an equal
probability, then the desired amount of
information is only a function of n:
I1 xk f 1/ n
45
A Quantitative Measure of Information
If each piece of equipment can be ordered in
one of m different colors and the selection of
colors is also equiprobable, then the amount
of information associated with the selection of
a color c j is :
I 2 c j f P c j f 1/ m
46
A Quantitative Measure of Information
The selection may be done in two ways:
Select the equipment and then the color
independently of each other
I xk & c j I1 ( xk ) I1 (c j ) f 1/ n f 1/ m
48
A Quantitative Measure of Information
The logarithmic information measure has the
desirable property of additivity for
independent statistical experiments.
The simplest case to consider is a selection
between two eqiuprobable events. The
amount of information associated with the
selection of one out of two equiprobable
events is log2 1/ 2 log 2 2 1 and provides a
unit of information known as a bit.
49
Information Entropy
- A key measure of measure
- Expressed by the average number of
bits needed for storage or
communication
Entropy
- Quantifies the uncertainty involved in a
random variables.
Ex. Coin flip less entropy
Roll of die more entropy
Entropy units bits/second
- bits/symbol
- unit less