Information Theory: Chapter 7: Channel Capacity and Coding Theorem - Part II

Information Theory
Chapter 7 : Channel Capacity and Coding Theorem – Part II
Syed Asad Alam
Electronics Systems Division

Department of Electrical Engineering
Linköping University
May 3, 2013
asad@isy.liu.se (Linköping University) Chapter 7 May 3, 2013 1 / 39

Previous Lecture
Chapter 7 : Part I
Fano’s inequality
Channel capacity and channel models
Preview of channel coding theorem
Definitions
Jointly typical sequences

Sources
Sources
Thomas M. Cover and Joy A. Thomas, Elements of Information Theory

(2nd Edition)
Inspiration of slides from lecture slides of ECE534, University of Illinois
at Chicago
(http://www.ece.uic.edu/~devroye/courses/ECE534)
Linear Block Codes → MIT 6.02 and other online sources

Sources
Outline
1 Channel Coding Theorem
2 Zero-Error Codes
3 Converse to the Channel Coding Theorem
4 Coding Schemes
5 Feedback Capacity
6 Source-Channel Separation Theorem

Channel Coding Theorem
Proof of the basic theorem of information theory

Achievability of channel capacity (Shannonn’s second theorem)
Theorem
For a discrete memory-less channel, all rates below capacity C are achievable
Specifically, for every rate R < C, there exists a sequence of (2nR , n)

codes with maximal probably of error λn → 0
Conversely, any sequence of (2nR , n) codes with λn → 0 must have
R≤C

Key Ideas of Shannon
Arbitrary small but non-zero probability of error allowed

Large successive use of channel
Calculating average probability of error over a random choice of
codebooks

Ideas behind the Proof
Same ideas as Shannon

Random code selection
Average probability of error
...
Difference in the decoding rule
Decoding by joint typicality
Look for a codeword that is jointly typical with the received sequence
Probability that any other codeword is jointly typical is 2−nI
If fewer than 2nI → High probability that no other codewords can be
confused with the transmitted word

Proof
Generate a (2nR , n) code at random, independently, according to the

distribution
n
Y
p(xn ) = p(xi ). (1)
i=1
Codebook (C), contains 2nR codewords as rows of matrix:

 
x1 (1) x2 (1) . . . xn (1)
C =  ... .. .. ..  .

. . .  (2)
x1 (2 ) x2 (2 ) . . . xn (2nR )
nR nR
Each entry in the matrix is generated i.i.d, i.e.,

nR
2 Y
n
Y
Pr(C) = p(xi (ω)). (3)
ω=1 i=1

Proof
Random code C generated according to (3)

Code revealed to both sender and receiver
Sender and receiver know the channel transition matrix p(y|x)
A message W (ωth row of the codebook) is chosen according to a
uniform distribution
Pr(W = ω) = 2−nR , ω = 1, 2, . . . , 2nR . (4)
Send this message over n uses of the channel

Proof
Receiver declares that Ŵ was sent if:

(X n (Ŵ), Y n ) is jointly typical
(n)
There is no other index W ′ 6= Ŵ such that (X n (W ′ ), Y n ) ∈ Aǫ
If no such Ŵ exists or more than one such exist, an error is declared
If Ŵ 6= W → Decoding error
Let E be such event when decoding error occurs

Probability of Error
Let E = Ŵ(Y n ) 6= W be the error event

Calculate
X (n)
Pr(E) = Pr(C)Pe (C) (5)
C
2 nR
X 1 X
Pr(E) = Pr(C) nR λω (C) (6)
2
C w=1
2 nR
1 XX
Pr(E) = nR Pr(C)λω (C) (7)
2
w=1 C
The inner sum does not depend on ω, so assuming that the message
W = 1 was sent X
Pr(E) = Pr(C)λ1 (C) = Pr(E|W = 1) (8)
C

Probability of Error
(n)
Let Ei = {(X n (i), Y n (i) ∈ Aǫ }, i ∈ 1, 2, . . . , 2nR . Then,
Pr(E|W = 1) = P(E1c ∪ E2 ∪ · · · ∪ E2nR |W = 1) (9)

2nR
X
Pr(E|W = 1) ≤ P(E1c |W = 1) + P(Ei |W = 1) (10)
i=2
From joint typicality

P(E1c |W = 1) ≤ ǫ, for n sufficiently large
P(Ei |W = 1) ≤ 2−n(I(X;Y)−3ǫ)
2nR
X
Pr(E) = Pr(E|W = 1) ≤ P(E1c |W = 1) + P(Ei |W = 1) (11)
i=2
2 nR
X
Pr(E) ≤ ǫ + 2−n(I(X;Y)−3ǫ) ≤ 2ǫ, n is sufficiently large (12)
i=2

Random Codes
The probability of error is small, as long as R < I

If p(x) is such that it maximized I(X; Y), R < I becomes R < C
There much exist at least one code C ∗ for which we have
1 X
P(E|C ∗ ) = nR λi (C ∗ ) ≤ 2ǫ (13)
2
i
We keep half of the codewords and throw away the worst half, then the
maximal error probability will be less than 4ǫ
The codebook has now 2nR−1 codewords
1
The rate is R − n = R as n → ∞

Channel Coding Theorem – Summary
Random coding used as a proof for the theorem

Joint typicality used as a decoding rule
Theorem shows that good codes exist with arbitrarily small error
probability
Does not provide a way of constructing the best codes
If code generated at random with appropriate distribution, the code is
likely to be good but difficult to decode
Hamming codes → simplest of a class of error correcting codes
Turbo codes → Close to achieving capacity for Gaussian channels

Zero-Error Codes
Zero-Error Codes
Motivation for the proof of the converse

(n)
If Pe = 0 → R ≤ C
W is uniformly distributed, i.e., H(W) = nR, we have
nR = H(W) = H(W|Y n ) +I(W; Y n ) (14)

| {z }
=0
n
X
a n n b
≤ I(X ; Y ) ≤ I(Xi ; Yi ) (15)
i=1
≤c nC (16)
R≤C (17)

Converse to the Channel Coding Theorem
Fano’s Inequality and the Coverse to the Coding Theorem
Theorem (Fano’s Inequality)

For any estimator X̂ : X → Y → X̂, with Pe = Pr{X 6= X̂}, we have
H(Pe ) + Pe log|X | ≥ H(X|X̂) ≥ H(X|Y). (18)

H(X|Y)−1
This implies 1 + Pe log|X ≥ H(X|Y) or Pe ≥ log|X |
In this case, for a discrete memoryless channel with a codebook C and

the input message W uniformly distributed over 2nR , we have
(n)
H(W|Ŵ) ≤ 1 + Pe nR. (19)
Capacity per transmission is not increased if channel used multiple times

I(X n ; Y n ) ≤ nC for all p(xn ) (20)

Proof
I(X n ; Y n ) = H(Y n ) − H(Y n |X n ) (21)

Xn
= H(Y n ) − H(Yi |Y1 , . . . , Yi−1 , X n ) (22)
i=1
n
X
= H(Y n ) − H(Yi |Xi ) (23)
i=1
n
X n
X
≤ H(Yi ) − H(Yi |Xi ) (24)
i=1 i=1
Xn
= I(Xi ; Yi ) (25)
i=1
≤ nC (26)

The Proof of Converse
We have W → X n (W) → Y n → Ŵ
For each n, let W is uniformly distributed and
(n) P
Pr(Ŵ 6= W) = Pe = 21nR i λi
Using the results we have
nR =a H(W) (27)
b
= H(W|Ŵ) + I(W; Ŵ) (28)
(n)
≤c 1 + Pe nR + I(W; Ŵ) (29)
(n)
≤d 1 + Pe nR + I(X n ; Y n ) (30)
(n)
≤e 1 + Pe nR + nC (31)
(n) 1
R ≤ Pe R + + C (32)
n

The Proof of Converse
(n) 1
R ≤ Pe R + +C (33)
n
(n) 1 P (n)
Since Pe = 2nR i λi , Pe → 0 as n → ∞
Same with the second term, thus, R ≤ C
However, if R > C, the average probability of error is bounded away
from 0
Channel capacity : A very clear dividing point.
(n) (n)
For R ≤ C → (Pe → 0), exponentially and for R > C → (Pe → 1)

Coding Schemes
Coding Schemes
Main focus → Codes that achieve low error probability and simple to
encode/decode
Objective → Introduce redundancy such that errors can be detected and
corrected
Simplest scheme → Repition codes
Encoding → Repeat the information multiple times
Decoding → Take the majority vote
Rate goes to zero with increasing block length
Error correcting codes → Parity check codes, hamming codes
Redundant bits help in detecting/correcting errors

Coding Schemes
Linear Codes
Hamming code is an example of linear, block, error correcting codes

Any linear combination of codewords is also a code word
Blocks of bits of length (k) mapped to a code word of length n having
n − k extra bits → (n, k) code
Code rate is k/n
Codewords produced from message bits by linear transformations of the
form
c = d.G (34)
where c is an n-element row vector containing the codeword, d is a k-element
row vector containing the message and G is a k × n matrix called generator
matrix

Coding Schemes
Linear Codes
Generator matrix is given by

Gk×n = Ik×k |Ak×(n−k) (35)
There is another matrix called parity check matrix H

Rows of H are parity checks on the codewords and should satisfy
cH T = 0 (36)
Can also be derived from G

H = − PT |In−k (37)
Should satisfy
GH T = 0 (38)
Coding Schemes
Systematic Linear Codes
Mapping between message and code explicit

Split data into k-bit blocks
Add n − k parity bits to each block → n-bits long code
k n-k
Message Bits Parity Bits
n

Coding Schemes
Hamming Codes
Example in the book

 
0 0 0 1 1 1 1
H = 0 1 1 0 0 1 1 (39)
1 0 1 0 1 0 1
0000000 0100101 1000011 1100110

0001111 0101010 1001100 1101001
(40)
0010110 0110011 1010101 1110000
0011001 0111100 1011010 1111111

Coding Schemes
Hamming Codes
Formally specified as (n, k, d) codes where d is the distance

Detects and correct up to one bit error
Block Length : n = 2n−k − 1
Number of message bits : k = 2n−k − (n − k) − 1
Distance (min. number of places two codewords differ) : d = 3
Perfect codes → Achieve highest possible rate

Coding Schemes
Decoding of hamming codes
Minimum distance is 3
If codeword c is corrupted in only one place, it will be closer to c
How to discover the closest codeword without searching over all the
codewords
Let ei be a vector with a 1 in the ith position
If codeword corrupt at i, then received vector can be written can be
written as r = c + ei .
Hr = H(c + ei ) = Hc + Hei = Hei (41)
Hei is the vector corresponding to the ith column of H.

Hr provides information which bit was corrupted and reversing it to get
the correct codeword

Coding Schemes
Hamming Code (7, 4, 3)
Can be shown using a Venn diagram
P1 D1 P2
D4
D2 D3
P3

Coding Schemes
Let the message be 11012
1
0 1

Coding Schemes
Placing parity bits in each circle so that parity of each circle is even
1
0 1
1
0 1

Coding Schemes
Assume one of the bits have changed as shown

Parity constraints violated for two of the circles
Error caused by the bit at the intersection of these two circles
1
0 1
1
0 0

Coding Schemes
Other Codes
With large block lengths, more than one error possible

Multi-bit error correcting codes proposed
Reed and Solomon (1959) codes for nonbinary channels
BCH codes, which generalizes Hamming codes using Galois field theory
to construct t-error correcting codes
Examples : cd-players use error correction circuitry based on two
interleaved (32, 28, 5) and (28, 24, 5) Read-Solomon codes that allow the
decoder to correct bursts up to 4000 errors
Convolution Codes →Each output block depends on both the current input
block and some of past input blocks
LDPC and Turbo codes → Achieving capacity with high probability

Feedback Capacity
Feedback Capacity
W Xi (W, Y i−1 ) Yi Ŵ
Encoder Channel Decoder
Message p(y|x) Estimate
of
Message
Can we achieve higher rates with feedback

Proof to show we cannot
A (2nR , n) feedback code as a
Sequence of mappings xi (W, Y i−1 )
Sequence of decoding functions g : Y → 1, 2, . . . , 2nR

Feedback Capacity
Feedback Capacity
Definition
The capacity with feedback, CFB , of a discrete memoryless channel is the
supremum of all rates achievable by feedback codes
Theorem (Feedback capacity)
CFB = C = max I(X; Y) (42)

p(x)
Nonfeedback code is considered as a special case of feedback code, thus

CFB ≥ C (43)

Feedback Capacity
Proof
Proving a series of similar inequalities by using index W instead of X n
nR = H(W) = H(W|Ŵ) + I(W; Ŵ) (44)

(n)
≤1+ Pe nR + I(W; Ŵ) (45)
(n)
≤1+ Pe nR + I(W; Y n ) (46)

Feedback Capacity
Proof
I(W; Y n ) = H(Y n ) − H(Y n |W) (47)

Xn
= H(Y n ) − H(Yi |Y1 , Y2 , . . . , Yi−1 , W) (48)
i=1
n
X
= H(Y n ) − H(Yi |Y1 , Y2 , . . . , Yi−1 , W, Xi ) (49)
i=1
n
X
= H(Y n ) − H(Yi |Xi ) (50)
i=1
n
X n
X n
X
≤ H(Yi ) − H(Yi |Xi ) = I(Xi ; Yi ) (51)
i=1 i=1 i=1
≤ nC (52)

Feedback Capacity
Proof
Combining the equations
(n)
nR ≤ Pe nR + 1 + nC (53)
R ≤ C, as n → ∞ (54)
So, rates achieved with feedback is equal to one achieved without

feedback
CFB = C (55)

Source-Channel Separation Theorem
Encoder Channel Decoder
Source Channel Channel Source

Encoder Encoder
Channel Decoder Decoder
Is a two-stage method as good as any other method to transmit

information over a noisy channel?
Data compression gives us R > H and transmission gives us R < C
Is H < C necessary and sufficient for transmitting a source over a
channel?

Theorem
If
V1 , V2 , . . . , Vn is a finite alphabet stochastic process that satisfies AEP
H(V) < C
there exists a source-channel code with Pr(V̂ n 6= V n ) → 0
In converse
Theorem
For any staionary stochastic process, if
H(V) > C
the probability of error is bounded away from zero

Proof of Theorem
(n)
From AEP → |Aǫ | ≤ 2n(H(V)+ǫ)
At most contributes ǫ to probability of error
2n(H(V)+ǫ) sequences → n(H(V) + ǫ) bits for indexing
Transmit with error probability less than ǫ if R = H(V) + ǫ < C
(n) (n)
Total error of probability ≤ 2ǫ (P(V n ∈
/ Aǫ + P(g(Y n ) 6= V n |V n ∈ Aǫ )
Hence, the sequence can be constructed with low probability of error for
n sufficiently large if H(V) < C

Proof of Theorem – Converse
H(V1 , V2 , . . . , Vn )
H(V) ≤a (56)
n
H(V n )
= (57)
n
1 1
= H(V n |V̂ n ) + I(V n ; V̂ n ) (58)
n n
1 1
≤b (1 + Pr(V̂ n 6= V n )nlog|V|) + I(V n ; V̂ n ) (59)
n n
1 1
≤c (1 + Pr(V̂ n 6= V n )nlog|V|) + I(X n ; Y n ) (60)
n n
d 1 n n
≤ (1 + Pr(V̂ 6= V )nlog|V|) + C (61)
n
H(V) ≤ C as n → ∞ (62)

Summary of Theorem
Data compression and transmission tied together

Data compression → Consequence of AEP → There exists a small
subset (|2nH |) of all possible source sequences that contain most of the
probability and that we can therefore represent the source with a small
probability error using H bits per symbol
Data transmission → Based on the joint AEP → For long block lengths,
the output sequence of the channel is very likely to be jointly typical with
the input codeword, while any other codeword is jointly typical with
probability ≈ 2−nI . Hence we can use about 2nI codewords and still have
negligible error probability.
This theorem shows that we can design the source and channel code
separately and combine the results to achieve optimal performance

Thank You!!!
www.liu.se

Information Theory: Chapter 7: Channel Capacity and Coding Theorem - Part II

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Information Theory: Chapter 7: Channel Capacity and Coding Theorem - Part II

Uploaded by

Copyright:

Available Formats

Information Theory

Chapter 7 : Channel Capacity and Coding Theorem – Part II

Syed Asad Alam

Electronics Systems Division

asad@isy.liu.se (Linköping University) Chapter 7 May 3, 2013 1 / 39

asad@isy.liu.se (Linköping University) Chapter 7 May 3, 2013 2 / 39

Thomas M. Cover and Joy A. Thomas, Elements of Information Theory

asad@isy.liu.se (Linköping University) Chapter 7 May 3, 2013 3 / 39

1 Channel Coding Theorem

3 Converse to the Channel Coding Theorem

6 Source-Channel Separation Theorem

asad@isy.liu.se (Linköping University) Chapter 7 May 3, 2013 4 / 39

Channel Coding Theorem

Proof of the basic theorem of information theory

Specifically, for every rate R < C, there exists a sequence of (2nR , n)

asad@isy.liu.se (Linköping University) Chapter 7 May 3, 2013 5 / 39

Key Ideas of Shannon

Arbitrary small but non-zero probability of error allowed

asad@isy.liu.se (Linköping University) Chapter 7 May 3, 2013 6 / 39

Ideas behind the Proof

Same ideas as Shannon

asad@isy.liu.se (Linköping University) Chapter 7 May 3, 2013 7 / 39

Generate a (2nR , n) code at random, independently, according to the

Codebook (C), contains 2nR codewords as rows of matrix:

Each entry in the matrix is generated i.i.d, i.e.,

asad@isy.liu.se (Linköping University) Chapter 7 May 3, 2013 8 / 39

Random code C generated according to (3)

Send this message over n uses of the channel

asad@isy.liu.se (Linköping University) Chapter 7 May 3, 2013 9 / 39

Receiver declares that Ŵ was sent if:

asad@isy.liu.se (Linköping University) Chapter 7 May 3, 2013 10 / 39

Let E = Ŵ(Y n ) 6= W be the error event

asad@isy.liu.se (Linköping University) Chapter 7 May 3, 2013 11 / 39

Pr(E|W = 1) = P(E1c ∪ E2 ∪ · · · ∪ E2nR |W = 1) (9)

From joint typicality

asad@isy.liu.se (Linköping University) Chapter 7 May 3, 2013 12 / 39

The probability of error is small, as long as R < I

asad@isy.liu.se (Linköping University) Chapter 7 May 3, 2013 13 / 39

Channel Coding Theorem – Summary

Random coding used as a proof for the theorem

asad@isy.liu.se (Linköping University) Chapter 7 May 3, 2013 14 / 39

Motivation for the proof of the converse

nR = H(W) = H(W|Y n ) +I(W; Y n ) (14)

asad@isy.liu.se (Linköping University) Chapter 7 May 3, 2013 15 / 39

Fano’s Inequality and the Coverse to the Coding Theorem

Theorem (Fano’s Inequality)

H(Pe ) + Pe log|X | ≥ H(X|X̂) ≥ H(X|Y). (18)

In this case, for a discrete memoryless channel with a codebook C and

Capacity per transmission is not increased if channel used multiple times

asad@isy.liu.se (Linköping University) Chapter 7 May 3, 2013 16 / 39

I(X n ; Y n ) = H(Y n ) − H(Y n |X n ) (21)

asad@isy.liu.se (Linköping University) Chapter 7 May 3, 2013 17 / 39

The Proof of Converse

asad@isy.liu.se (Linköping University) Chapter 7 May 3, 2013 18 / 39

The Proof of Converse

asad@isy.liu.se (Linköping University) Chapter 7 May 3, 2013 19 / 39

asad@isy.liu.se (Linköping University) Chapter 7 May 3, 2013 20 / 39

Hamming code is an example of linear, block, error correcting codes

asad@isy.liu.se (Linköping University) Chapter 7 May 3, 2013 21 / 39

Generator matrix is given by

There is another matrix called parity check matrix H

Can also be derived from G

Systematic Linear Codes