You are on page 1of 6

Derivation of Channel Capacity for the Binary Erasure Channel (BEC) and the Z-channel

Wael I. Halbawi February 1, 2012

1

Introduction

The following document outlines the derivation of channel capacity for the Binary Erasure Channel (BEC) and the Z-channel. The derivation utilizes tools from information theory and probability and statistics. A simple Wikipedia search produces the following definition for channel capacity: ”In electrical engineering, computer science and information theory, channel capacity is the tightest upperbound on the amount of information that can be reliably transmitted over a communication channel.” In other words, for a given channel, any coding rate, that exceeds the channel capacity for that channel will result in information loss. Mathematically speaking, the capacity of a channel is defined as follows: C = max I(X; Y ) (1)

where the function I(X; Y ) represents the mutual information between the input distribution X and output distribution Y .

2

Binary Erasure Channel

Given that a bit x ∈ {0, 1} is transmitted over a BEC, characterized by parameter ǫ, the received bit at the output is y ∈ {0, 1, ?} where ? is an unknown value or simply, an ”erased bit”. The BEC can be defined in terms of transition probabilities given by p(y|x). It is trivial to notice that the channel model summarized in Table 1 that the channel does not produce erroneous bits. This model is visualized in figure 2.

Table 1: Transition probabilities for the BEC x y p(y|x) 0 0 1−ǫ 0 1 0 0 ? ǫ 1 0 0 1 1 1−ǫ 1 ? ǫ

1

H(X) = − X p(x) log(x) = −p(X = 0) log(p(X = 0)) − p(X = 1) log(p(X = 1)) = −p log(p) − (1 − p) log(1 − p) H(X) = H2 (p) (4) 2 . the source’s entropy H(X) is the binary entropy function H2 (p). We can compute this function easily using the definition of entropy. Y ) = H(X) − H(X|Y ) The distribution of the source is given by a Bernoulli random variable: PX (x) = p. x = 0.p 0 1−ǫ ǫ 0 ? ǫ 1−p 1 1−ǫ 1 Figure 1: Channel model for BEC along with transition probabilities Mutual information I(X. Figure 2 shows the binary entropy function plotted versus p(x = 0). Y ) can be expressed using source and conditional entropies H(X) and H(X|Y ). We use the logarithm function with base 2 and information is represented using bits. x = 1 (3) (2) Because of the nature of the input’s distribution. I(X. 1 − p. respectively.

we can show that . we know that the conditional entropies H(X|Y = 0) and H(X|Y = 1) are equal to zero because of the deterministic nature of these two quantities.4 0. p(Y =?) = p(X = 0)p(Y =?|x = 0) + p(X = 1)p(Y =?|X = 1) = pǫ + (1 − p)ǫ p(Y =?) = ǫ (6) p(X = 0|Y =?) = p(Y =?|X = 0)p(X = 0) p(Y =?) ǫp = ǫ p(X = 0|Y =?) = p p(X = 1|Y =?) = 1 − p 3 (7) (8) Similarly.8 0.9 0.5 0.7 0.1 0.6 H2 (p) 0.2 0.5 0.2 0. knowing that the output Y is equal to either 0 or 1 gives us no new information about X. In other words.1 0 0 0.9 p Figure 2: Binary entropy function H2 (p) The entropy of the source X conditioned on the output Y is computed using a similar manner.3 0.3 0.7 0.8 0.6 0. The entropy of the source conditioned on the output being an erasure H(X|Y =?) is derived by observing the conditional probability p(x|y) and applying Baye’s Rule.4 0. H(X|Y ) = Y 1 p(Y = y)H(X|Y = y) = p(Y = 0)H(X|Y = 0) + p(Y = 1)H(X|Y = 1) + p(?)H(X|Y =?) = p(1 − ǫ)0 + (1 − p)(1 − ǫ)0 + (pǫ + (1 − p)ǫ)H2 (p) H(X|Y ) = ǫH2 (p) (5) Using the fact the channel does not introduce any errors.1 0.

I(X. i.Table 2: Transition probabilities for the BEC x y p(y|x) 0 0 1−ǫ 0 1 ǫ 1 0 ǫ 1 1 1−ǫ Using (7) and (8). Using the results from (4) and (5) along with the definition of channel capacity in (1). we observe that the the probability distribution p(X|Y ) is a Bernoulli random variable and its corresponding entropy function is H(X|Y ) = H2 (p). characterized by parameter ǫ. we obtain the following equation for channel capacity of the BEC C = max H2 (p) − ǫH2 (p) p = max(1 − ǫ)H2 (p) p C =1−ǫ (9) We can see from (9) that the capacity of the Binary Erasure Channel is equal to the transition probability p(Y |X) = 1 − ǫ and is achieved when the input source is uniformly distributed. As with the BEC. The BSC characterized by transition probabilities in Table 2 can be illustrated using Figure 3. does not introduce any erasures in the received bits but only produces errors.e p = 1 . the BSC can be defined in terms of transition probabilities given by p(y|x). 2 3 Binary Symmetric Channel The BSC. Y ) = H(Y ) − H(Y |X) (10) 4 . we use the following expression that relates mutual information of the source and the output to the entropy of the output and the entropy of the output conditioned on the input. p 0 1−ǫ 0 ǫ ǫ 1−p 1 1−ǫ 1 Figure 3: Channel model for BSC along with transition probabilities To calculate the capacity C of this channel.

This is obtained when the argument is equal to half. (13) and (15) results in the following expression C = max {H2 (p + ǫ − 2pǫ) − H2 (ǫ)} p (15) (16) The argument is maximized when H2 (p + ǫ − 2pǫ) = 1. These probabilities are the transition probabilities of the channel listed in Table 2. We conclude that H(Y |X = 0) and H(Y |X = 1) are again Binary Entropy Functions with parameter ǫ. H(Y ) = H2 (p + ǫ − 2pǫ) The conditional entropy H(Y |X) is found using the following summation H(Y |X) = X (13) p(x)H(Y |X = x) (14) H(Y |X = 0) is found by looking at p(Y = 0|X = 0) and p(Y = 1|X = 0). we can show that the conditional entropy H(Y |X) is also a Binary Entropy Function. Using (14). respectively. p(y = 1) = X (11) p(x)P (y = 1|x) = (1 − p)(1 − ǫ) + pǫ p(y = 1) = 1 − p − ǫ + 2pǫ (12) Looking at (11) and (12). 1 2 1 p(1 − 2ǫ) + ǫ = 2 1 − 2ǫ p(1 − 2ǫ) = 2 1 p= 2 p + ǫ − 2pǫ) = (17) Therefore. H(Y |X) = H2 (ǫ) Combining (10). the capacity of the BSC is achieved when the input to the channel is a uniformly distributed binary alphabet. The entropy of the output H(Y ) is calculated as follows p(y = 0) = X p(x)P (y = 0|x) = p(x = 0)p(y = 0|x = 0) + p(x = 1)p(y = 0|x = 1) = p(1 − ǫ) + (1 − p)ǫ p(y = 0) = p + ǫ − 2pǫ Similarly. and H(Y |X = 1) is found by looking at p(Y = 0|X = 1) and p(Y = 1|X = 1). we obtain the quantities H(Y ) and H(Y |X) by studying the probability distributions PY (y) and PY (y|x).Similar to they BEC. we observe that Y is a Bernoulli random variable and therefore its entropy is a Binary Entropy Function. 5 .

3 0.5 0.3 0.8 0.6 0.6 0.2 0.7 0.5 0.C = 1 − H2 ǫ The capacity of the BSC is plotted versus ǫ ∈ [0.9 ǫ Figure 4: Capacity of BSC for various values of ǫ 1 (18) 6 .9 0.4 0.4 0.2 0.8 0.1 0.7 0. 1] 1 Capacity (bits per channel use) 0.1 0 0 0.