Comsysf311 L39

Lecture 39: Module 6
Information Theory and Coding:
Channel Capacity:
Discrete and AWGN
Dr. S .M. Zafaruddin

Assistant Professor
Deptt. of EEE, BITS Pilani, Pilani Campus
Dr. Zafar (BITS Pilani) CommSys: L39 IT
Objectives of Today Lecture
1 Channel Capacity for DMS

2 Channel Capacity for AWGN

Module 6: Information Theory
• First part: Source coding


• Second part: Channel capacity


• Second part: Channel capacity
• Third part: Channel coding

Discrete Memory Less
Channel: Transition Matrix
• Discrete: If both input and output alphabets are
discrete.

discrete.
• Memory less: When the current output depends on
the current input and not the previous ones.

discrete.
• X = {x1 , x2 , ....xi }, Y = {y1 , y2 , ..., yj }

discrete.
• X = {x1 , x2 , ....xi }, Y = {y1 , y2 , ..., yj }
• P (yj |xi ): Transition probability or priori probability that
yj is received when xi is transmitted.

Example: BSC
• Sketch BSC and find its capacity

Entropy: Discrete
• Entropy: H(X) = 1
P
i P (xi ) log2 P (xi )
bits per message

Entropy: Discrete
P
i P (xi ) log2
P (xi )
bits per message
• Entropy: H(Y ) = 1
P
j P (yj ) log2 P (yj ) bits per message

Entropy: Discrete
P
i P (xi ) log2
P (xi )
bits per message
P
• Condition entropy:
1
P P
H(X|Y ) = i j P (xi , yj ) log2 P (xi |yj )
bits per
message

Entropy: Discrete
P
i P (xi ) log2
P (xi )
bits per message
P
1
P P
H(X|Y ) = i j P (xi , yj ) log2 P (xi |yj )
bits per
message
1
P P
H(Y |X) = j i P (yj , xi ) log2 P (yj |xi )
bits per
message

Entropy: Continuous
R∞ 1
• Entropy: H(X) = −∞
fX (x) log2 fX (x)
dx bits per
message

Entropy: Continuous
R∞
• Entropy: H(X) = −∞ fX (x) log2 fX1(x) dx bits per
message
R∞ R∞
H(X|Y ) = −∞ −∞ fXY (x, y) log2 fX|Y1(x|y) dxdy bits
per message

Mutual Information I(X; Y ):
Discrete
• Mutual information:
I(X : Y ) = H(X) − H(X|Y ) = H(Y ) − H(Y |X)

Discrete
I(X : Y ) = H(X) − H(X|Y ) = H(Y ) − H(Y |X)
• I(X : Y ) = i j P (xi , yj ) log2 PP(x(x)P
i ,yj )
P P
(y )i j

Discrete
I(X : Y ) = H(X) − H(X|Y ) = H(Y ) − H(Y |X)
• I(X : Y ) = i j P (xi , yj ) log2 PP(x(x)P
i ,yj )
P P
i (yj )
j |xi )
• I(X : Y ) = i j P (xi )P (yj /xi ) log2 P PP(x(y)P
P P
i (yj |xi )
i

Continuous
I(X : Y ) = H(X) − H(X|Y ) = H(Y ) − H(Y |X)

Continuous
I(X : Y ) = H(X) − H(X|Y ) = H(Y ) − H(Y |X)
R∞ R∞ f (x,y)
• I(X : Y ) = −∞ f (x, y) log2 fXX,Y
−∞ XY (x)fY (y)
dxdy bits
per message

Channel Capacity C
• Channel capacity C = maxP (xi ) I(X; Y ) bits per

symbol.

Channel Capacity C

symbol.
• Channel capacity C = maxfX (x) I(X; Y ) bits per
symbol.

Channel Capacity C

symbol.
symbol.
• For discrete, equiprobable messages maximizes the
capacity.

Channel Capacity C

symbol.
symbol.
• For discrete, equiprobable messages maximizes the
capacity.
• What should be condition for continuous channel?

Condition for Maximum
Entropy: Continuous
R∞
• Maximize H(x) = −∞
fX (x)log2 fX1(x) dx

Entropy: Continuous
R∞
• Maximize H(x) = −∞ fX (x)log2 fX1(x) dx
• with a constraint that

Entropy: Continuous
R∞
R∞
• Constraint 1: −∞ fX (x)dx = 1

Entropy: Continuous
R∞
R∞
• Any other constraint?

Entropy: Continuous
R∞
R∞
• Input signal should have a limited power

Entropy: Continuous
R∞
R∞
• Another constraint: mean square value of x is fixed.

Entropy: Continuous
R∞
R∞
• Another constraint: mean square value of x is fixed.
R∞ 2
• Constraint 2: −∞ x fX (x)dx = σ 2

Entropy: Continuous
• The solution of the above optimization problem:

Entropy: Continuous

• fX (x) = √ 1 2 e−x2 /2σ2
2πσ

Entropy: Continuous

• fX (x) = √ 1 2 e−x2 /2σ2
2πσ
• What is maximum entropy?

Entropy: Continuous

• fX (x) = √ 1 2 e−x2 /2σ2
2πσ
• What is maximum entropy?
R∞
• H(x) = −∞ fX (x) log2 f 1(x) dx = 21 log2 (2πeσ 2 )
X

Channel Capacity of
Band-Limited AWGN
• y(t) = x(t) + n(t)

Channel Capacity of
Band-Limited AWGN
• y(t) = x(t) + n(t)
• Sampled signal: y = x + n

Channel Capacity of
Band-Limited AWGN
• y(t) = x(t) + n(t)
• H(y|x)?

Channel Capacity of
Band-Limited AWGN
• y(t) = x(t) + n(t)
• H(y|x)?
• What is PDF of y given x

Channel Capacity of
Band-Limited AWGN
• y(t) = x(t) + n(t)
• H(y|x)?
• It is Gaussian with shifted mean x.

Channel Capacity of
Band-Limited AWGN
• y(t) = x(t) + n(t)
• H(y|x)?
• It is Gaussian with shifted mean x.
R∞ R∞
• H(y|x) = −∞ −∞ fXY (x, y) log2 fY |X1(y|x) dxdy bits per
message

Channel Capacity of
Band-Limited AWGN
R∞ R∞ 1
• H(y|x) = fXY (x, y) log2 dxdy
R−∞
∞ R−∞
∞
fY |X (y|x)
H(y|x) = −∞ −∞
fX (x)fY |X (y|x) log2 fY |X1(y|x) dxdy

Channel Capacity of
Band-Limited AWGN
R∞ R∞ 1
R−∞
∞ R−∞
∞
fY |X (y|x)
H(y|x) = −∞ −∞
R∞ R∞
• H(y|x) = −∞
fX (x)dx −∞ fY |X (y|x) log2 fY |X1(y|x) dy

Channel Capacity of
Band-Limited AWGN
R∞ R∞ 1
R−∞
∞ R−∞
∞
fY |X (y|x)
H(y|x) = −∞ −∞
R∞ R∞
• H(y|x) = −∞
R∞ 1
• H(y|x) = −∞
fY |X (y|x) log2 fY |X (y|x)
dy

Channel Capacity of
Band-Limited AWGN
R∞ R∞ 1
R−∞
∞ R−∞
∞
fY |X (y|x)
H(y|x) = −∞ −∞
R∞ R∞
• H(y|x) = −∞
R∞ 1
• H(y|x) = −∞
dy
• fY |X (y|x) = fn (y − x)

Channel Capacity of
Band-Limited AWGN
R∞ R∞ 1
R−∞
∞ R−∞
∞
fY |X (y|x)
H(y|x) = −∞ −∞
R∞ R∞
• H(y|x) = −∞
R∞ 1
• H(y|x) = −∞
dy
• fY |X (y|x) = fn (y − x)
R∞ 1
• H(y|x) = −∞ fn (y − x) log2 fn (y−x)
dy

Channel Capacity of
Band-Limited AWGN
R∞ R∞ 1
R−∞
∞ R−∞
∞
fY |X (y|x)
H(y|x) = −∞ −∞
R∞ R∞
• H(y|x) = −∞
R∞ 1
• H(y|x) = −∞
dy
• fY |X (y|x) = fn (y − x)
R∞ 1
• H(y|x) = −∞ fn (y − x) log2 fn (y−x) dy
R∞ 1
• Let z = y − x H(y|x) = −∞ fn (z) log2 fn (z)
dz

Channel Capacity of
Band-Limited AWGN
R∞ R∞ 1
R−∞
∞ R−∞
∞
fY |X (y|x)
H(y|x) = −∞ −∞
R∞ R∞
• H(y|x) = −∞
R∞ 1
• H(y|x) = −∞
dy
• fY |X (y|x) = fn (y − x)
R∞ 1
• H(y|x) = −∞ fn (y − x) log2 fn (y−x) dy
R∞ 1
• Let z = y − x H(y|x) = −∞ fn (z) log2 fn (z)
dz
• H(y|x) = H(n)

Channel Capacity of
Band-Limited AWGN
• I(x; y) = H(y) − H(y|x) = H(y) − H(n)

Channel Capacity of
Band-Limited AWGN
• I(x; y) = H(y) − H(y|x) = H(y) − H(n)
• Capacity: For a given H(n), I(x; y) is maximum when
H(y) is maximum.

Channel Capacity of
Band-Limited AWGN
• I(x; y) = H(y) − H(y|x) = H(y) − H(n)
H(y) is maximum.
• y =x+n

Channel Capacity of
Band-Limited AWGN
• I(x; y) = H(y) − H(y|x) = H(y) − H(n)
H(y) is maximum.
• y =x+n
• For a given mean square value y¯2 = S + N , H(y) is
maximum when y is Gaussian.

Channel Capacity of
Band-Limited AWGN
• I(x; y) = H(y) − H(y|x) = H(y) − H(n)
H(y) is maximum.
• y =x+n
• Maximum H(y): 12 log2 [2πe(S + N )]

Channel Capacity of
Band-Limited AWGN
• I(x; y) = H(y) − H(y|x) = H(y) − H(n)
H(y) is maximum.
• y =x+n
• Since n is Gaussian, y will be Gaussian only if x is
Gaussian.

Channel Capacity of
Band-Limited AWGN
• I(x; y) = H(y) − H(y|x) = H(y) − H(n)
H(y) is maximum.
• y =x+n
• Since n is Gaussian, y will be Gaussian only if x is
Gaussian.
• fX (x) = √ 1−x2 /2S .
2πSe

Shannon Channel Capacity of
Band-Limited AWGN
• I(x; y) = H(y) − H(y|x) = H(y) − H(n)

Band-Limited AWGN
• I(x; y) = H(y) − H(y|x) = H(y) − H(n)

Band-Limited AWGN
• I(x; y) = H(y) − H(y|x) = H(y) − H(n)
• H(n) = 12 log2 [2πe(N )]

Band-Limited AWGN
• I(x; y) = H(y) − H(y|x) = H(y) − H(n)
• H(n) = 12 log2 [2πe(N )]
• For signal: best is Gaussian

Band-Limited AWGN
• I(x; y) = H(y) − H(y|x) = H(y) − H(n)
• H(n) = 12 log2 [2πe(N )]
• For noise: worst is Gaussian

Band-Limited AWGN
• I(x; y) = H(y) − H(y|x) = H(y) − H(n)
• H(n) = 12 log2 [2πe(N )]
• C = 12 log2 [2πe(S + N )] − 12 log2 [2πe(N )]

Band-Limited AWGN
• I(x; y) = H(y) − H(y|x) = H(y) − H(n)
• H(n) = 12 log2 [2πe(N )]
• C = 12 log2 [2πe(S + N )] − 12 log2 [2πe(N )]
• C = 12 log2 [1 + NS ] bits per symbol

Band-Limited AWGN
• I(x; y) = H(y) − H(y|x) = H(y) − H(n)
• H(n) = 12 log2 [2πe(N )]
• C = 12 log2 [2πe(S + N )] − 12 log2 [2πe(N )]
• For a BW of B: 2B symbols per second

Band-Limited AWGN
• I(x; y) = H(y) − H(y|x) = H(y) − H(n)
• H(n) = 12 log2 [2πe(N )]
• C = 12 log2 [2πe(S + N )] − 12 log2 [2πe(N )]
• For a BW of B: 2B symbols per second
• C = 2B 21 log2 [1 + NS ] = B log2 (1 + S/N ) =
B log2 (1 + SN R)

Shannon Channel Capacity:
Infinite BW
• C = B log2 (1 + S
N0 B
), where N0 : PSD, N : Power.

Infinite BW
• C = B log2 (1 + S
N0 B
• limB → ∞C = limB → ∞B log2 (1 + S
N0 B
)

Infinite BW
• C = B log2 (1 + S
N0 B
N0 B
)
• limB → ∞B log2 (1 NSB ) =
0
limB → ∞ NS0 [ NS0 B log2 (1 + S
N0 B
)]

Infinite BW
• C = B log2 (1 + S
N0 B
N0 B
)
• limB → ∞B log2 (1 NSB ) =
0
limB → ∞ NS0 [ NS0 B log2 (1 + S
N0 B
)]
S
• limB → ∞B log2 (1 + N0 B
) = log2 e NS0 = 1.44 NS0

System Adaptation
• Nyquist versus Shannon

HW39
Problem 12.5-3 BP Lathi, 4th SE Edition.

Comsysf311 L39

Uploaded by

Copyright:

Available Formats

You might also like

Comsysf311 L39

Uploaded by

Document Information

Original Description:

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Comsysf311 L39

Uploaded by

Copyright:

Available Formats

Lecture 39: Module 6

Information Theory and Coding:

Dr. S .M. Zafaruddin

1 Channel Capacity for DMS

Dr. Zafar (BITS Pilani) CommSys: L39 IT

• First part: Source coding

Dr. Zafar (BITS Pilani) CommSys: L39 IT

• First part: Source coding

Dr. Zafar (BITS Pilani) CommSys: L39 IT

• First part: Source coding

Dr. Zafar (BITS Pilani) CommSys: L39 IT

Dr. Zafar (BITS Pilani) CommSys: L39 IT

Dr. Zafar (BITS Pilani) CommSys: L39 IT

Dr. Zafar (BITS Pilani) CommSys: L39 IT

Dr. Zafar (BITS Pilani) CommSys: L39 IT

• Sketch BSC and find its capacity

Dr. Zafar (BITS Pilani) CommSys: L39 IT

Dr. Zafar (BITS Pilani) CommSys: L39 IT

Dr. Zafar (BITS Pilani) CommSys: L39 IT

Dr. Zafar (BITS Pilani) CommSys: L39 IT

Dr. Zafar (BITS Pilani) CommSys: L39 IT

Dr. Zafar (BITS Pilani) CommSys: L39 IT

Dr. Zafar (BITS Pilani) CommSys: L39 IT

Dr. Zafar (BITS Pilani) CommSys: L39 IT

Dr. Zafar (BITS Pilani) CommSys: L39 IT

Dr. Zafar (BITS Pilani) CommSys: L39 IT

Dr. Zafar (BITS Pilani) CommSys: L39 IT

Dr. Zafar (BITS Pilani) CommSys: L39 IT

• Channel capacity C = maxP (xi ) I(X; Y ) bits per

Dr. Zafar (BITS Pilani) CommSys: L39 IT

• Channel capacity C = maxP (xi ) I(X; Y ) bits per

Dr. Zafar (BITS Pilani) CommSys: L39 IT

• Channel capacity C = maxP (xi ) I(X; Y ) bits per

Dr. Zafar (BITS Pilani) CommSys: L39 IT

• Channel capacity C = maxP (xi ) I(X; Y ) bits per

Dr. Zafar (BITS Pilani) CommSys: L39 IT

Dr. Zafar (BITS Pilani) CommSys: L39 IT

Dr. Zafar (BITS Pilani) CommSys: L39 IT

Dr. Zafar (BITS Pilani) CommSys: L39 IT

Dr. Zafar (BITS Pilani) CommSys: L39 IT

Dr. Zafar (BITS Pilani) CommSys: L39 IT

Dr. Zafar (BITS Pilani) CommSys: L39 IT

Dr. Zafar (BITS Pilani) CommSys: L39 IT

• The solution of the above optimization problem:

Dr. Zafar (BITS Pilani) CommSys: L39 IT

• The solution of the above optimization problem:

Dr. Zafar (BITS Pilani) CommSys: L39 IT

• The solution of the above optimization problem:

Dr. Zafar (BITS Pilani) CommSys: L39 IT

• The solution of the above optimization problem:

Dr. Zafar (BITS Pilani) CommSys: L39 IT

Dr. Zafar (BITS Pilani) CommSys: L39 IT

Dr. Zafar (BITS Pilani) CommSys: L39 IT

Dr. Zafar (BITS Pilani) CommSys: L39 IT

Dr. Zafar (BITS Pilani) CommSys: L39 IT

Dr. Zafar (BITS Pilani) CommSys: L39 IT

Dr. Zafar (BITS Pilani) CommSys: L39 IT