KEET 3107 Information Theory & Coding

Semester 1 Session 2013/2014

Dr Norfizah Md Ali norfizah@um.edu.my norfizah@gmail.com

References:
C E Shannon, A Mathematical Theory of Communication, Bell System Technical Journal, Vol. 27, July and October 1948. W W Peterson an E J Weldon, Error-Correcting Codes, MIT Press, 1972 S Haykin, Digital Communications, Wiley 1988 J G Proakis, Communication Systems Engineering, Prentice Hall, 1995 N Abramson, Information Theory and Coding, McGraw-Hill, 1963 Any Analog and Digital Communication Text Books contain chapters on Information Theory

Introduction
Information theory- a discipline centered around common mathematical approach, to study the collection and manipulation of information. -theoretical basis for observation, measurement, data compression, data storage, communication, estimation, decision making and pattern recognition.

-guide to the development of information- transmission systems based on a study of possibilities and limitations inherent in natural law.
-a study of how the laws of probability, and of mathematics in general, describe limits on the designs of information-transmission systems, but also offer opportunities. -one may design strategies into a communication system to overcome noise and errors that may occur in the communication channel.

Introduction (Contd)
Information theory originated by Claude Shannon in his 1948 paper. Basic theory underlying the task of communication through noisy channel. He showed that each channel is characterized by channel capacity such that an arbitrarily small probability of error is achievable at any transmission rate below the channel capacity. Probability of error and rate of data transmission can be specified independently To achieve small error of probability is the method of coding the transmitted information in blocks. Information theory gives the insight into design of informationtransmission systems. By developing a clear concept of information and its transmission, a much deeper understanding of the purposes and limitations of a technique is obtained.

Channel Capacity

C= W log2(1+P/WNo) bits/s

Where P is the average transmitted power W is the channel bandwidth No/2 is the power spectral density of additive noise OR

C= W log2(1+S/N) bits/s S/N is the signal-to-noise power ratio

Information and Sources

Let E be some event which occurs with probability P(E). I(E) = log 1/P(E) units of information loga x = 1/ (logba) logb x choice of base for log determine the unit for information I(E) = log2 1/P(E) bits I(E) = ln 1/P(E) nats (natural unit) I(E) = log10 1/P(E) Hartleys In general if we use a logarithm to the base r, I(E) = logr 1/P(E) r-ary units 1 Hartley = 3.32 bits 1 nat = 1.44 bits Take note that, If P(E) = 1/2 , then I(E) = 1 bit

Source

Si,.

A source emits a sequence of symbols from a fixed finite source with alphabet S = {s1, s2., sq} are statistically independent. Such an information source is termed as zero-memory source The probabilities with which the symbols occur: P(s1), P(s2),P(sq) If symbol s1 occurs, the amount of information is: I(s1) = log 1/P(s1) bits

 The average amount of information obtained per symbol from the source is:

(s )I(si) bits
S

This quantity, the average amount of information per source symbol, is called the entropy H(S) of the zero memory source.

H(S) =

(s ) log 1/P(si) bit

H(S) is interpreted either as the average amount of information per symbol provided by the source OR as the average amount of uncertainty which the observer has before the inspection of the output of the source.

Example 1:
Consider the source S = { s1, s2 ,s3} with P(s1) = , P(s2) = P(s3) = , then H(S) = log 2 + log 4 + log 4 = 3/2 bits

For zero memory information source with q-symbol source alphabet, the maximum value of the entropy is exactly log q, and this maximum value of the entropy is achieved if, and only if, all the source symbols are equiprobable.
Binary entropy function is at maximum value for p=0.5, ie when both 1 and 0 are equally likely.

In general, the entropy of a discrete source is maximum when the letters from the source are equally probable.

Extensions of Zero-memory Source

Imagine the bits emitted from the source as being in groups of n. For example, if n=2, then the four possible symbols 00, 01, 10 and 11 The entropy per symbol of Sn is n times the entropy per symbol of S:

H(Sn) = n H(S)
Taking the second extension of the previous example : Recall the source S = { s1, s2 ,s3} with P(s1) = , P(s2) = P(s3) = , then Sn has nine symbols

Example 2:

Symbols of Sn Sequence of symbols Probability P(i)

1 s1s1

2 s1s2

3 s1s3

4 s2s1

5 s2s2

6 s2s3

7 s3s1

8 s3s 2

9 s3s3

1/4

1/8

1/8

1/8

1/16

1/16

1/8

1/16

1/16

H(S2) =

(i) log 1/P (i) bits

=1/4 log4+ 4 x 1/8 log 8 + 4 x 1/16 log 16 = 3 bits per symbol

The Markov Information Source

A more general type of information source with q symbols in which the occurrence of a source symbol si may depend on a finite number m of preceding symbols. M th-order Markov source P(si/ sji, sj2,.. sjm) for i=1,2, q; jp = 1,2.,q For an m th order markov source, the probability of emitting a given symbol is known if we know the m preceding symbols the state of the m th order Markov source, ie, qm possible state.

Example 3
Consider a second-order Markov source with the binary source alphabet S = {0,1}. Assume the conditional symbol probabilities: P(0/00) = P(1/11) =0.8 P(1/00) = P(0/11) = 0.2 P(0/01) = P(0/10) = P(1/01) = P(1/10) = 0.5 Since q is equal to 2, we assumed a second order Markov source, hence there are four states of the source 00, 01, 10, 11

State Diagram of a Second Order Markov Source. The possible states are indicated by four possible dots. The possible state transitions are indicated by arrows from state to state, with probability of a transition shown . For example: if we are in state 00 we can go to either state 01 or 00 but not to state 10 or 11. The probability of remaining in state 00 is 0.8 and the probability of going to state 01 is 0.2.