Professional Documents
Culture Documents
Information Theory
Father of Digital Communication
Information
Coding
Source
Communication
Channel
Destination Decoding
Information
Coding
Source
Communication
Channel
Destination Decoding
Source Channel
Data
Encoding Encoding
Channel
Source Channel
User
Decoding Decoding
Example: Disk Storage
Channel
Verify
User Unzip
CRC
In terms of Information Theory
Terminology
Source
Zip = Encoding Data Compression
= Source
Unzip Data Decompression
Decoding
Verify Channel
CRC = Decoding Error Correction
Example: VCD and DVD
MPEG RS
Moive
Encoder Encoding
CD/DVD
MPEG RS
TV
Decoder Decoding
Speech CC
Encoding Encoding
Channel
Speech CC
Decoding Decoding
GSM/CDMA
CC stands for Convolutional Code.
Example: WLAN IEEE 802.11b
CC
Data Zip
Encoding
Channel
CC
User Unzip
Decoding
IEEE 802.11b
CC stands for Convolutional Code.
Shannon Theory
1. Measurement of Information
2. Source Coding Theory
3. Channel Coding Theory
Measurement of Information
= ? bits
= ? bits
Or Lottery!?
= ? bits
Or this…
= ? bits
= ? bits
All events are probabilistic!
• Using Probability Theory, Shannon
showed that there is only one way to
measure information in terms of
number of bits:
H ( X ) p( x ) log 2 p( x )
x
6 6
H p(i ) log 2 p (i ) p(i ) log 2 p(i )
i 1 i 1
6
1 1
log 2 log 2 6 2.585 bits
i 1 6 6
Wait!
It is nonsense!
2,585,000
0.3231 32.31%
8,000,000
Let’s Do Some Test!
File Size Compression
Ratio
No 8,000,000 100%
Compression bits
Shannon 2,585,000 32.31%
bits
Winzip 2,930,736 36.63%
bits
WinRAR 2,859,336 35.74%
bits
The Winner is
Source Channel
Data
Encoding Encoding
Channel
Source Channel
User
Decoding Decoding
The Simplest Case:
Computer Network
Communications over computer network,
ex. Internet
8Mbps
2 Mbps
X4
Alice Bob
p=0.00390625
Shannon challenged:
? ?
p=0.25
Alice Bob
?
Alice 8 x (1-p)
=6 Mbps p=0 Bob
Unfortunately…
Data Audio
Compression MP3 Compression
Communicates
Faster than