Module 5: Lecture 40

Information Theory: Channel Capacity

Dr. S .M. Zafaruddin

Associate Professor
Deptt. of EEE, BITS Pilani, Pilani Campus

Dr. Zafar (BITS Pilani) CommSys: M5L40 IT 1 / 46

Objectives of Today Lecture

Channel Capacity
Channel Capacity for AWGN

Dr. Zafar (BITS Pilani) CommSys: M5L40 IT 2 / 46

Channel Transition Matrix

Bayes’ Rules

Dr. Zafar (BITS Pilani) CommSys: M5L40 IT 3 / 46

Entropy:

1
P
Entropy: H(X) = i P (xi ) log2
P (xi )
bits per message
1
P
Entropy: H(Y ) = j P (yj ) log2 P (yj ) bits per message
P (xi , yj ) log2 P (x1i ,yj )
P P
Joint Entropy H(X, Y ) = i j

Conditional entropy: H(X|Y ) = i j P (xi , yj ) log2 P (x1i |yj )

P P

bits per message

Conditional entropy: H(Y |X) = j i P (yj , xi ) log2 P (y1j |xi )
P P

bits per message

Dr. Zafar (BITS Pilani) CommSys: M5L40 IT 4 / 46

Some Identities

H(X, Y ) = H(Y ) + H(Y |X) = H(X) + H(X|Y )

H(X, Y ) ≤ H(X) + H(Y )
H(Y |X) ≤ H(Y ) and H(X|Y ) ≤ H(X)

Dr. Zafar (BITS Pilani) CommSys: M5L40 IT 5 / 46

Transition Matrix

0 a

1 b

Dr. Zafar (BITS Pilani) CommSys: M5L40 IT 6 / 46

Transition Matrix for BSC

Dr. Zafar (BITS Pilani) CommSys: M5L40 IT 7 / 46

Mutual Information I(X; Y )

Mutual information:
I(X : Y ) = H(X) − H(X|Y ) = H(Y ) − H(Y |X)
bits/message or bits/symbol
P P P (xi ,yj )
I(X : Y ) = i j P (xi , yj ) log2 P (xi )P (yj )
P P P (yj |xi )
I(X : Y ) = i j P (xi )P (yj |xi ) log2 P P (xi )P (yj |xi )
i

Channel capacity C = maxP (xi ) I(X; Y ) bits per symbol.

Dr. Zafar (BITS Pilani) CommSys: M5L40 IT 8 / 46

Shannon’s Channel Capacity for AWGN Channel

y =x+n

Dr. Zafar (BITS Pilani) CommSys: M5L40 IT 9 / 46

Differential Entropy

R∞
Differential Entropy: H(X) = −∞ fX (x) log2 fX1(x) dx bits per
message
Differential RConditional entropy:
∞ R∞
H(X|Y ) = −∞ −∞ fXY (x, y) log2 fX|Y1(x|y) dxdy bits per
message

Dr. Zafar (BITS Pilani) CommSys: M5L40 IT 10 / 46

Mutual Information I(X; Y ): Continuous

Mutual information:
I(X : Y ) = H(X) − H(X|Y ) = H(Y ) − H(Y |X)
R∞ R∞ f (x,y)
I(X : Y ) = −∞ −∞ fXY (x, y) log2 fXX,Y
(x)fY (y)
dxdy bits per
message

Dr. Zafar (BITS Pilani) CommSys: M5L40 IT 11 / 46

Entropy: Uniform Distribution

1
f (x) = 2M in the range −M < x < M otherwise 0
R∞
H(X) = −∞ fX (x) log2 fX1(x) dx bits per message
R∞
H(X) = −∞ fX (x) log2 fX1(x) dx
RM 1
H(X) = −M 2M log2 2M
1
dx = log2 2M

Dr. Zafar (BITS Pilani) CommSys: M5L40 IT 12 / 46

Entropy: Gaussian Distribution

2 2
fX (x) = √ 1 e−x /2σ
2πσ 2
R∞
H(X) = −∞ X
f (x) log2 fX1(x) dx bits per message
Simplify log2 (.) and use the property of PDF and variance
R∞
H(x) = −∞ fX (x) log2 fX1(x) dx = 12 log2 (2πeσ 2 )

Dr. Zafar (BITS Pilani) CommSys: M5L40 IT 13 / 46

SOLUTION

Dr. Zafar (BITS Pilani) CommSys: M5L40 IT 14 / 46

Condition for Maximum Entropy

R∞
Find fX (x) that maximizes H(x) = −∞ fX (x) loge fX1(x) dx
with a constraint that
R∞
Constraint 1: −∞ fX (x)dx = 1
Any other constraint?
Input signal should have a limited power
R∞
Constraint 2: −∞ x2 fX (x)dx = σ 2
Main result: for a given mean square value, the
entropy is maximum for a Gaussian distribution and
the maximum entropy is 12 log2 (2πeσ 2 ).

Dr. Zafar (BITS Pilani) CommSys: M5L40 IT 15 / 46

Channel Capacity of Band-Limited AWGN

y =x+n
H(y|x)?
R∞ R∞ 1
H(y|x) = −∞ −∞ fXY (x, y) log2 fY |X (y|x)
dxdy bits per
message

Dr. Zafar (BITS Pilani) CommSys: M5L40 IT 16 / 46

Channel Capacity of Band-Limited AWGN
R∞ R∞ 1
H(y|x) = −∞ −∞
fXY (x, y) log2 fY |X (y|x)
dxdy
Bayes’ Rule:
R∞ R∞
H(y|x) = −∞ −∞ fX (x)fY |X (y|x) log2 fY |X1(y|x) dxdy
R∞ R∞
H(y|x) = −∞ fX (x)dx −∞ fY |X (y|x) log2 fY |X1(y|x) dy
R∞
H(y|x) = −∞ fY |X (y|x) log2 fY |X1(y|x) dy
fY |X (y|x) = fn (y − x)
R∞ 1
H(y|x) = −∞ fn (y − x) log2 fn (y−x) dy
Let z = y − x
R∞
H(y|x) = −∞ fn (z) log2 fn1(z) dz
H(y|x) = H(n)

Dr. Zafar (BITS Pilani) CommSys: M5L40 IT 17 / 46

Channel Capacity of Band-Limited AWGN

I(x; y) = H(y) − H(y|x) = H(y) − H(n)

Capacity: For a given H(n), I(x; y) is maximum when H(y)
is maximum.
y =x+n
x¯2 = S, n¯2 = N
For a given mean square value y¯2 = S + N , H(y) is maximum
when y is Gaussian.
Maximum H(y): 21 log2 [2πe(S + N )]
If n is Gaussian, y will be Gaussian only if x is Gaussian.
2
fX (x) = √2πS1
e−x /2S .

Dr. Zafar (BITS Pilani) CommSys: M5L40 IT 18 / 46

Shannon Channel Capacity of Band-Limited
AWGN

I(x; y) = H(y) − H(y|x) = H(y) − H(n)

Maximum H(y): 21 log2 [2πe(S + N )]
For Gaussian noise: H(n) = 12 log2 [2πeN ], N = N0 B.
For signal: best is Gaussian
For noise: worst is Gaussian
C = 12 log2 [2πe(S + N )] − 21 log2 [2πeN ]
C = 12 log2 [1 + NS ] bits per symbol
For a BW of B: maximum 2B symbols per second (Nyquist
ISI criteria)
C = 2B 12 log2 [1 + NS ] = B log2 (1 + S/N ) = B log2 (1 + SN R)

Dr. Zafar (BITS Pilani) CommSys: M5L40 IT 19 / 46

Shannon Channel Capacity of Band-Limited
AWGN

C = B log2 (1 + SN R)
Assumptions to get the above?
Channel: Band-limited
Noise: Additive
Noise: Gaussian distributed
Input signals: White Gaussian

Dr. Zafar (BITS Pilani) CommSys: M5L40 IT 20 / 46

Shannon Channel Capacity: Asymptotic

Capacity when transmitted signal power → ∞?

C→∞
Capacity when channel BW → ∞?
C = B log2 (1 + NS )
C = B log2 (1 + NS0 B ), where N0 : PSD, N : Power.
S
limB→∞ C = limB→∞ B log2 (1 + N0 B
)
S N0 B
limB→∞ C limB→∞ N0 [ S log2 (1 + NS0 B )]
limB→∞ C = NS0 log2 e = 1.44 NS0

Dr. Zafar (BITS Pilani) CommSys: M5L40 IT 21 / 46

Problem

Find the channel capacity for an AWGN channel with B = 10

MHz, noise PSD −120 dBW/Hz, and signal power 20 dBm. What
will be channel capacity if B → ∞.

Dr. Zafar (BITS Pilani) CommSys: M5L40 IT 22 / 46

Solution

AWGN channel capacity C = B log2 (1 + SN R): everything

in linear and SI units.
B = 107 Hz
Noise PSD: −120dBW/Hz= 10−12 W/Hz
Noise power 10−12 × 107 = 10−5 Watt
Signal Power= 20 dBm= 102 mW= 0.1 Watt.
SNR = Signal power/Noise Power= 0.1/10−5 = 106 = 60dB
C = B log2 (1 + SN R) = 107 log2 (1 + 106 ) = 1.99 × 108
bps=200 Mbps.
C∞ = 1.44 NS0 = 1011 bps= =100 Gbps

Dr. Zafar (BITS Pilani) CommSys: M5L40 IT 23 / 46

Problem 1 TB 12.4-1

A binary channel matrix is given by

 
2/3 1/3
1/10 9/10
Also Px (x1 ) = 1/3, and Px (x2 ) = 2/3. Determine H(x), H(x|y),
H(y), H(y|x) and I(x; y).

Dr. Zafar (BITS Pilani) CommSys: M5L40 IT 24 / 46

Solution : Understanding the Channel Matrix

Py/x (y1 /x1 ) = 2/3, Py/x (y2 /x1 ) = 1/3, Py/x (y1 /x2 ) = 1/10,
Py/x (y2 /x2 ) = 9/10.

Dr. Zafar (BITS Pilani) CommSys: M5L40 IT 25 / 46

Solution 1: Probability of x and y

P (x1 ) = 1/3, and P (x2 ) = 2/3

Total probability formula:
P (y1 ) = P (y1 /x1 )P (x1 ) + P (y1 /x2 )P (x2 ) = 13/45
P (y2 ) = 1 − 13/45 = 32/45

Dr. Zafar (BITS Pilani) CommSys: M5L40 IT 26 / 46

Solution 1: Entropy

H(x) = P (x1 ) log2 1/P (x1 ) + P (x2 ) log2 1/P (x2 ) = 0.918
bits/message
H(y) = P (y1 ) log2 1/P (y1 ) + P (y2 ) log2 1/P (y2 ) = 0.8673
bits/message

Dr. Zafar (BITS Pilani) CommSys: M5L40 IT 27 / 46

Solution 1: Conditional Probabilities

Bayes’rule
P (y1 |x1 )P (x1 )
P (x1 |y1 ) = P (y1 )
= 10/13
P (y2 |x1 )P (x1 )
P (x1 |y2 ) = P (y2 )
= 5/32
P (y1 |x2 )P (x2 )
P (x2 |y1 ) = P (y1 )
= 3/13
P (y2 |x2 )P (x2 )
P (x2 |y2 ) = P (y2 )
= 54/64

Dr. Zafar (BITS Pilani) CommSys: M5L40 IT 28 / 46

Solution 1: Conditional Entropy

H(x|y1 ) =
P (x1 |y1 ) log2 (1/P (x1 |y1 )) + P (x2 |y1 ) log2 (1/P (x2 |y1 )) = 0.779
H(x|y2 ) =
P (x1 |y2 ) log2 (1/P (x1 |y2 )) + P (x2 |y2 ) log2 (1/P (x2 |y2 )) = 0.624
H(x|y) = P (y1 )H(x|y1 ) + P (y2 )H(x|y2 ) = 0.6687

Dr. Zafar (BITS Pilani) CommSys: M5L40 IT 29 / 46

Solution 1: Mutual information

I(x; y) = H(x) − H(x|y) = 0.918 − 0.6687 = 0.2489.

H(y|x) = H(y) − I(x; y) = 0.618 bits/message

Dr. Zafar (BITS Pilani) CommSys: M5L40 IT 30 / 46

Channel Capacity:BSC

Dr. Zafar (BITS Pilani) CommSys: M5L40 IT 31 / 46

Solution: BSC Example 13.3 in TB

1 1
Cs = 1 − [Pe log2 Pe
+ (1 − Pe ) log2 ( 1−P e
)]

Dr. Zafar (BITS Pilani) CommSys: M5L40 IT 32 / 46

Problem

Find the channel capacity of Z channel:

Dr. Zafar (BITS Pilani) CommSys: M5L40 IT 33 / 46

Solution

Dr. Zafar (BITS Pilani) CommSys: M5L40 IT 34 / 46

Problem
In data communication using error detection code, as soon as an
error is detected, an automatic request for retransmission (ARQ)
enables retransmission of the data in error. In such a channel, the
data in error is erased. Hence, there is an erase probability p, but
the probability of error is zero. Such a channel, known as a binary
erasure channel (BEC), can be modeled as shown in Figure.
Determine H(x), H(x|y), and I(x; y) assuming the two
transmitted messages equiprobable.

Dr. Zafar (BITS Pilani) CommSys: M5L40 IT 35 / 46

Solution: Channel Matrix

x1 = 0, x2 = 1, y1 = 0, y2 = 1, y3 = E, q = 1 − p.

Dr. Zafar (BITS Pilani) CommSys: M5L40 IT 36 / 46

Solution: H(x)

Symbols are equiprobable P (x0 ) = P (x1 ) = 1/2

H(x) = 1 bits/symbol.

Dr. Zafar (BITS Pilani) CommSys: M5L40 IT 37 / 46

Solution: H(x|y)
H(X|Y ) = i j P (xi , yj ) log2 P (x1i |yj ) bits per symbol
P P

i = 2, j = 3
It is for loop
H(x) = P (x1 , y1 ) log2 P (x11 |y1 ) + P (x1 , y2 ) log2 P (x11 |y2 ) +
P (x1 , y3 ) log2 P (x11 |y3 ) + P (x2 , y1 ) log2 P (x12 |y1 ) +
P (x2 , y2 ) log2 P (x12 |y2 ) + P (x2 , y3 ) log2 P (x12 |y3 )
Calculate P (xi |yj ) and P (xi , yj ) for each combinations of i
and j
P (yi |xj )P (xi )
Use P (xi |yj ) = P (yi )
Use P (xi , yj ) = P (yi |xj )P (xi )
P (yi |xj ): given from channel matrix
P (xi ): given from input probabilities (1/2 in this case)
P (yj )? from total probability theorem
Dr. Zafar (BITS Pilani) CommSys: M5L40 IT 38 / 46
Solution: P (yj )

Total probability
P (y1 ) = P (y1 |x1 )P (x1 ) + P (y1 |x2 )P (x2 ) =
q × 1/2 + 0 × 1/2 = q/2
P (y2 ) =
P (y2 |x1 )P (x1 ) + P (y2 |x2 )P (x2 ) + 0 × 1/2 + q × 1/2 = q/2
P (y3 ) = 1 − q/2 − q/2 = 1 − q = p
P (y1 |x1 ) and P (y1 |x2 ), P (y2 |x1 ) and ,P (y2 |x2 ) are taken from
channel transition matrix.

Dr. Zafar (BITS Pilani) CommSys: M5L40 IT 39 / 46

Solution: P (xi|yj )

P (yi |xj )P (xi )

P (xi |yj ) = P (yi )
P (y1 |x1 )P (x1 )
P (x1 |y1 ) = P (y1 )
= (q × 1/2)/(q/2) = 1
Similarly
P (x2 |y1 ) = 0
P (x1 |y2 ) = 0
P (x2 |y2 ) = 1
P (x1 |y3 ) = 1/2
P (x2 |y3 ) = 1/2

Dr. Zafar (BITS Pilani) CommSys: M5L40 IT 40 / 46

Solution: Finally: P (xi, yj )

P (xi , yj ) = P (yi |xj )P (xi )

P (x1 , y1 ) = P (y1 |x1 )P (x1 ) = q × 1/2 = q/2
Similarly
P (x1 , y2 ) = 0
P (x1 , y3 ) = p/2
P (x2 , y1 ) = 0
P (x2 , y2 ) = q/2
P (x2 , y3 ) = p/2

Dr. Zafar (BITS Pilani) CommSys: M5L40 IT 41 / 46

Solution: H(x|y)

H(X|Y ) = i j P (xi , yj ) log2 P (x1i |yj ) bits per symbol

P P

i = 2, j = 3
It is for loop
H(x) = P (x1 , y1 ) log2 P (x11 |y1 ) + P (x1 , y2 ) log2 P (x11 |y2 ) +
P (x1 , y3 ) log2 P (x11 |y3 ) + P (x2 , y1 ) log2 P (x12 |y1 ) +
P (x2 , y2 ) log2 P (x12 |y2 ) + P (x2 , y3 ) log2 P (x12 |y3 )
H(x|y) = p

Dr. Zafar (BITS Pilani) CommSys: M5L40 IT 42 / 46

Solution: I(x; y)

I(x; y) = H(x) − H(x|y) = 1 − p bits per symbol.

Additional questions: How to get I(y; x), and how to get
H(y|x)
I(y; x) = I(x; y)
I(y; x) = H(y) − H(y|x)
H(y|x) = H(y) − I(y; x)
H(y) can be computed easily from probabilities of y:
P (y1 ) = q/2, P (y2 ) = q/2, and P (y3 ) = p.

Dr. Zafar (BITS Pilani) CommSys: M5L40 IT 43 / 46