Chapter q
Fundamental limits in Information Theory
Problem 9.1
Amount of information gained by the occurrence of an event of probability p is
I varies with p as shown below:
8'.
O.~ 1.0
456
Problem 9.2
Let the event S=Bt, denote the emission of symbol Bt, by the source, Hence,
Bt, So sl s2 sa
PIt 0.4 0.3 0.2 0.1
I(sk) bits 1.322 1.737 2.322 3.322 Problem 9.3
Entropy of the source is
H(S) • Polog.(~ J + PI log. (:1 J + P'log.(~ J + P310g.(~ J
= ! log2(3) + ! log2(6) + ! log2(4) + ! log2(4) = 0.528 + 0.431 + 0.5 + 0.5
= 1.959 bits
Problem 9.4
Let X denote the number showing on a single roll of a die. With a die having six faces, we note that Px is 1/6. Hence, the entropy of X is
H(X) · Px log. (:x J
= .!. log2(6) = 0.431 bits 6
457
Problem 9.5
The entropy of the quantizer output is
4
H =  E P(Xk) log2 P(Xk) k=l
where Xk denotes a representation level of the quantizer. Since the quantizer input is Gaussian with zero mean, and a Gaussian density is symmetric about its mean, we find that
P(X 1) = P(X4) P(X2) = P(Xa)
The representation level Xl = 1.5 corresponds to a quantizer input +1 < Y < 00. Hence,
P(Xl) = 100 _1_ exp(rJdY
J21t 2
· ~  ~ e1~ J
= 0.1611
The representation level ~ = 0.5 corresponds to the quantizer input 0 < Y < 1. Hence,
Accordingly, the entropy of the quantizer output is
H =  2rO.1611 IOg2( 1 ) + 0.3389 IOg2(0.3389)]
l 0.1611
= 1.91 hits
458
Problem 9.6
(a) For a discrete memoryless source:
Noting that M =}(fi, we may therefore write
Ml Ml
E P(ai) = E P(Sil) P(Si2) ••• P(Si)
i=O i=O
P(SI' ) P(SI' ) ... P(Si )
1 2 n
Kl Kl Kl
= E P(Sil) E P(Si2) ••• E P(Si)
il=O i2=O in=O
= 1
(b) For k = 1,2, ... ,n, we have
For k=l, say, we may thus write
xi [J
= E Pil log2 _!:._
i=O Pil
= H(S)
Clearly, this result holds not only for k=l but also ks2, ... ,n.
459
(c)
Ml
H(S D) = E P(d" i) log2 _1_
i=O P(crt)
Ml
E
i=O
Ml + ... + E i .. O
Using the result of part (b), we thus get
H(S D) = H(S) + H(S) + •.. + H(S) = nH(S)
Problem 9.7 .
(a) The entropy of the source is
H(S) = 0.7 log2 _1_ + 0.15 log2 _1_ + 0.15 log2 _1_
0.7 0.15 0.15
= 0.258 + 0.4105 + 0.4105
= 1.079 bits
(b) The entropy of secondorder extension of the source is
H(S2) = 2 x 1.079 = 2.158 bits
460
Problem 9.8
The entropy of text is defined by the smallest number of bits that are required, on the average, to represent each letter.
According to Luckyt, English text may be represented by less than 3 bits per character, because of the redundancy built into the English language. However, the spoken equivalent of English text has much less redundancy; its entropy is correspondingly much higher than 3 bits. It follows therefore from the source coding theorem that the number of bits required to represent (store) text is much smaller than the number of bits required to represent (store) its spoken equivalent.
Problem 9.9
(a) With K equiprobable symbols, the probability of symbol BJt. is
The average codeword length is
xi
= _!_ E lk K k=O

The choice of a fixed codeword length lk=lo for all k yields the value L=lo. With K symbols in the
code, any other choice for lk yields a value for L no less than 10•
(b) Entropy of the source is
461
tR.W. Lucky, "Silicon Dreams", p.1U (St. Martin's Press, 1989).
Kl [J
H(S) = to Pk log2 :k
xi
=E
k=O
1
 log2 K = log2 K
K
The coding efficiency is therefore
11 = H(S) = log2K
L 10
For 11=1, we have
To satisfy this requirement, we choose
where 10 is an integer.
Problem 9.10
A prefix code is defined as a code in which no code work is the prefix of any other code word. By inspection, we see therefore that codes I and N are prefix codes, whereas codes II and III are not.
To draw the decision treefor a prefix code, we simply begin from some starting node, and extend branches forward until each symbol of the code is represented. We thus have:
S '\
o
463
Problem 9.11
We may construct two different Huffman codes by choosing to place a combined symbol as low or as high as possible when its probability is equal to that of another symbol.
We begin with the Huffman code generated by placing a combined symbol as low as possible:
oss
S I
a.IS
0.10
0.05



o
y
The source code is therefore
So 0
sl 11
s2 100
sa 1010
s4 1011 0.15
0.'5
0.1'5
464
o
a.S50 .. 0.55J
030~. /0.45
0.'5 _u
The average codeword length is therefore
", 0.55(1) + 0.15(2) + 0.15(3) + 0.1(4) + 0.05(4) "'1.9
The variance of L is
4
a2 ", E Pk(lk  1:)2 k=O
", 0.55(0.9)2 + 0.15(0.1)2 + 0.15(1.1)2 + 0.1(2.1)2 + 0.05(2.1)2 ", 1.29
Next placing a combined symbol as high as possible, we obtain the second Huffman code:
0,)0
o
_ .... 'fIII""" 0.45 .... 0.56,
O ~S _u
O,30Y'
0.15
__ r'~ 0,45
s~ 0,15
0.15
s 0.05
~
Correspondingly, the Huffman code is
465
The average codeword length is
L = 0.55(1) + (0.15 + 0.15 + 0.1 + 0.05) (3) = 1.9

The variance of L is
02 = 0.55( 0.9)2 + (0.15 + 0.15 + 0.1 + 0.05) (1.1)2 = 0.99
The two Huffman codes described herein have the same average codeword length but different variances.
Problem 9.12
0
S ocs ~ O·~S ,5 0.5J
a
SI o.~5 O.~5 0,'15 O.2~ 0.5
r
S 0.1.;)5 0.125 0.'25
2.
$..3 O.t~5
5"," O. \,;}S 466
The Huffman code is therefore
So 10
$1 11
s2 001
sa 010
s4 011
s5 0000
s6 0001 The average codeword length is
= 0.25(2)(2) + 0.125(3)(3) + 0.0625(4)(2) = 2.625
The entropy of the source is
H(S) = E Pk IOg2(_!_J
k=O Pk
= 0.25(2) log2 (_1_) + 0.125(3) log2 ( 1 )
lO.25 lO.125
+ 0.0625(2) log2( 1 )
lO.0625
= 2.625
The efficiency of the code is therefore
11 = H(S) = 2.625 = 1
L 2.625
We could have shown that the efficiency of the code is 100% by inspection since
467
Problem 9.13
(a)
o
0.7 0.7
0.15 ~ /0.3 J
O.15~
The Huffman code is therefore
The average codeword length is
L = 0.7(1) + 0.15(2) + 0.12(2) = 1.3
(b) For the extended source we have
Symbol 8080 8081 8082 8180 8~0 8181 8182 8281 8282
Probability 0.49 0.105 0.105 0.105 0.105 0.0225 0.0225 0.0225 0.0225 468
Applying the Huffman algorithm. to the extended source, we obtain the following source code:
soso 1
sOsl 001
sOs2 010
s1s0 011
s2s0 0000
slsl 000100
s1s2 000101
s2s1 000110
s2s2 000111 The corresponding value of the average codeword length is
L2 = 0.49(1) + 0.105(3)(3) + 0.105(4) + 0.0225(4)(4) = 2.395 bits/extended symbol
1:2 = 1.1975 bits/symbol
2
(c) The original source has entropy
H(S) = 0.7 log2(_!_) + 0.15(2) IOg2(_1_)
0.7 0.15
= 1.18
According to Eq. (lD·tS),
H(S) s Ln s H(S) + .!.
n n
This is a condition which the extended code satisfies.
Problem 9.14
Symbol Huffman Code Codeword length
A 1 1
B 011 3
C 010 3
D 001 3
E 0011 4
F 00001 5
G 00000 5 Problem 9.15
a
I
00 2
, 0
. , 
4 I

2
01 j_
e
10 .
3 I Computer code Probability Huffman Code
00 ! 0
2
11 ! 10
4
01 ! 110
8
10 ! 111
8 The number of bits used for the instructions based on the computer code, in a probabilistic sense,
is equal to 470
(1 1 1 1) .
2 _ + _ + _ + _ = 2 bits
248 8
On the other hand, the number of bits used for instructions based on the Huffman code, is equal to
1 1 117
1x_+2x_+3x_+3x_=_
2 4 8 8 4
The percentage reduction in the number of bits used for instruction, realized by adopting the Huffman code, is therefore
V4
100 x _ = 12.5%
2
Problem 9.16
Initial step
Subsequences stored: 0
Data to be parsed: 1 1 1 0 1 0 0 1 1 0 0 0 1 0 1 1 0 1 0 0 ...
Step 1
Subsequences stored: 0, 1, 11
Data to be parsed: 1 0 1 0 0 1 1 0 0 0 1 0 1 1 0 1 0 0 ..
Step 2
Subsequences stored: 0, 1, 11, 10
Data to be parsed: 1 0 0 1 1 0 0 0 1 0 1 1 0 1 0 0 ....
Step 3
Subsequences stored: 0, 1, 11, 10, 100
Data to be parsed: 1 1 0 0 0 1 0 1 1 0 1 0 0 ...
Step 4
Subsequences stored: 0, 1, 11, 10, 100, 110
Data to be parsed: 0 0 1 0 1 1 0 1 0 0 ...
471
Step 5
Subhsequences stored:
Data to be parsed:
0,1,11,10,100,110,00 10110100 ....
Step 6
Subsequences stored:
Data to be parsed:
0,1,11,10,100,110,00,101 10100 ...
Step 7
Subsequences stored:
Data to be parsed:
0,1,11,10,100,110,00,101,1010
0 ....
Now that we have gone as far as we can go with data parsing for the given sequence, we write
Numerical 1 2 3 4 5 6 7 8 9
positions
Subsequences 0, 1, 11, 10, 100, 110, 00, 101, 1010
Numerical 22, 21, 41, 31, 11, 42, 81
representations
Binary encoded 0101, 0100, 0100, 0110, 0010, 1001, 10000
blocks 472
Problem 9.17
Ip
P(Yo) = (1  p)P(XQ) + p P(Xl) = (1  p) (!) + p(!)
2 2
1
=
2
P(Yl) = p p(xo) + (1  p) p(xl) = p(!) + (1  p) (!)
2 2
1
=
2
473
Problem 9.18
P(Xo) = .!. 4
p(xI) = ~ 4
1 3
P(Yo} = (1  p) (_) + P(_)
4 4
= 1 + P
4" "2
P(Yj) = P( !) + (1  p) (!)
=~g
4 2
Problem 9.19
From Eq.(95Z)we may express the mutual information as
The joint probability P(Jt_j'Yk) has the following four possible values:
j = k = 0: P(Xo 'Yk) = Po(1p) = (1  PI) (1  p)
j = 0, k = 1: P(Xo'YI) = PoP = (1  PI)P
j = 1, k = 0: P(xI'YO) = PIP
j = k = 1: p(xI'YI) = P(1  ~)
The mutual information is therefore
474
Rearranging terms and expanding algorithms:
I(X;Y) = P log2 P + (1p) log2(1P)
 [PI(1P) + (lPI)P] IOg2[PI(1p) + (lPI)P]
 [PIP + (lPI) (1p)] IOg2[PIP + (1PI) (lp)]
Treating the transition probability P of the channel as a running parameter, we may carry out the following numerical evaluations:
p=O:
I(X;Y) =  PI log2 PI  (1  PI) log2 (1  PI) PI = 0.5, I(X;y) = 1.0
p=Ol:
I(X;Y) =  0.469  (0.1 + 0.8 PI) log2 (011 + 0.8 PI)  (0.9  0.8 PI) log2 (0.9  0.9 PI) PI = 0.5, I(X;Y) = 0.531
p=0.2:
I(X;Y) =  0.7219  (0.2 + 0.6 PI)log2 (0.2 + 0.6 PI)  (0.8  0.6 PI) log2 (0.8  0.6 PI) PI = 0.5, I(X;y) = 0.278
475
p=0.3:
I(X;Y) =  0.88129  (0.3 + 0.4 Pl) log2 (0.3 + 0.4 Pl)  (0.7  0.4 Pl) log2 (0.7  0.4 Pl) Pl = 0.5, I(X;Y) = 0.1187
p=0.5:
I(X;Y) = 0
Thus, treating the a priori probability Pt as a variable and the transition probability P as a running parameter, we get the following plots:
Problem 9.20
From the plots ofl(X;y) versus Pi for P as a running parameter, that were presented in the solution to Problem to .. IQ we observe that I(X;y) attains its maximum value at Pl=o.S for any p. Hence, evaluating the mutual information I(X;y)' at Pl=0.5, we get the channel capacity:
c = 1 + P log2 P + (1  p) log2 (1  p) = 1  H(p)
where H(p) is the entropy function ofp.
476
Problem 9.21
(a) Let
z = PI(1  p) + (1  PI) P = (1  Po) (1  p) + PoP
Hence,
PIP + (1  PI) = 1  PI (1  p)  (1  PI) P
= 1  z
Correspondingly, we may rewrite the expression for the mutual information I(X;Y) as
I(X;Y) = H(z)  H(p)
where
H(z) =  z log2 z  (1  z) log2 (1  z) H([) =  P log2 P  (1  p) log2 (1  p)
(b) Differentiating I(X;Y) with respect to Po (or PI) and setting the result equal to zero, we fmd that I(X;Y) attains its maximum value at Po = PI = 112.
(c) Setting Po = PI  112 in the expression for the variable z, we get:
z=1z=1/2
Correspondingly, we have
H(z) = 1
We thus get the following value for the channel capacity:
C = I(X;y) bo '" PI '" 1/2
= 1  H(p)
where H(p) is the entropy function of the channel transition probability p.
477
Problem 9.22
1 P
:to,
/ P
From this diagram, we obtain (by inspection)
P(Yo I XQ) = (1  p)2 + P 2 = 1  2P(1  p) P(yO I xl) = P(1  p) + (1  p)p = 2p(1  p)
Hence, the cascade of two binary symmetric channels with a transition probability p is equivalent to a single binary symmetric channel with a transition probability equal to 2P(1  p), as shown below:
Correspondingly, the channel capacity of the cascade is
C = 1  H(2P(1  p»
= 1  2P(1  p) log2 [2P(1  p)]  (1  2p + 2p 2) log2(1  2p + 2p 2)
478
Problem 9.23
1  ex
1  ex
The mutual information for the erasure channel is
1 2
• _ ~ ~ ( p(xj' Yk) )
I(X,Y)  L.JL.Jp(xj,Yk)log (x.) ( )
j=Ok=O 2 P J P Yk
(1)
The joint probabilities for the channel are
where Po + PI = 1. Substituting these values in (1), we get
Since the transition probability P = (1  u) is fixed, the mutual information I(X; Y) is maximized by choosing the a priori probability Po to maximize H(Po). This maximization occurs at Po = 112,. for which H(Po) = 1. Hence, the channel capacity C of the erasure channel is 1  o;
479
Problem 9.24
(a) When each symbol is repeated three times, we have
Messages Unused signals Channel outputs
000 001 000
010 001
011 010
100 100
101 101
110 110
111 111
We note the following: 1. The probability that no errors occur in the transmission of three Os or three Is is (1 _ p)3.
2. The probability of just one error occurring is 3p(1  p'l'.
3. The probability of two errors OC.(!(J.((";1Ijis 3p2(1 p).
4. The probability of receiving all three bits in error is p3.
With the decisionmaking based on a majority vote, it is clear that contributions 3 and 4 lead to the probability of error
(b) When each symbol is transmitted five times, we have
Messages Unused signals Channel outputs
00000 00000
00001 00001
00010 00010
00011 00011
" .
,.
,.
•
..
11110 11110
11111 11111 480 The probability of zero, one, two, three, four, or five bit errors in transmission is as follows, respectively:
(1_p)5 5p (1_p)4 1Op2(1_p)3 10p3(1_pf 5p4(Ip) p5
The last three contributions constitute the probability of error
(a) For the general case ofn=2m + 1, we note that the decisionmaking process (based on a majority vote) makes an error when m+ 1 bits or more out of the n bits of a message are received in error. The probability of i message bits being received in error is (~ f ~ 1  p)" i,
Hence, the probability of error is (in general)
n
Pe = E
i=m+l
The results derived in parts (a) and (b) for me l and m=2 are special cases of this general formula.
Problem 9.25
The differential entropy of a random variable is independent of its mean. To evaluate the differential entropy of a Gaussian vector X, consisting of the components Xl'~""'~' we may simplify our task by setting the mean of X equal to zero. Under this condition, we may express the joint probability density function of the Gaussian vector X as
481
The logarithm of fX<x) is
Hence, the differential entropy of X is
h(X) =  J J ... J fX<x) log2(fX<x») dx
= IOg2(21t)nl20102 ... 0n) J J ... J fX<x)dx
+ log2e J J ... J [x; + xi2 + ... + x~ J fX<x) dx
2~ 202 20n
We next note that
J J J fX<x) dx = 1
J J J xi2 fX<x)dx = o~
i = 1,2, ... ,n
Hence, we may simplify (1) as
482
When the individual variances are equal:
2.£"2 2 2
0'1 =02 = •• ·O'n = 0'
Correspondingly, the differential entropy of X is
Problem 9.26
(a) The differential entropy of a random variable X is
h(X) =  i: fX<x) log2 fX<x)dx
The constraint on the value x of the random variable X is
Ix Is M
of
Using the method'Lagrange multipliers, we find that h(X), subject to this constraint, is maximized
when
483
1: [ fX<x) log2 fX<x) + A fX<x)]dx
is stationary. Differentiating fX<x)log2fX<x) + AfX<X) with respect to fX<x), and then setting the result equal to zero, we get
This shows that fX<x) is independent of x. Hence, for the differential entropy h(X) under the constraints I x Is; M and 1: fX<x)dj s' to be maximum, the random variable X
must be uniformly distributed:
fX<x) = {112M,
0,
M S; x < M otherwise
(b) The maximum differential entropy of the uniformly distributed random variable X is
h(X) = (M _1_ log2(2M)dx JM 2M
= _1_ log2(2M) (M dx
2M JM
= log2(2M)
484
Problem 9.27
(a)
ioo ioo [fX(X lY)}
I(X;Y) = fx,y(X,y)log2 xdy
00 00 fx(x)
(1)
ioo ioo [fY(Y Ix) Jdxd
I(Y;X) = fx,y(x,y)log2 y
00 00 f y(y )
(2)
From Bayes' rule, we have
fx(x lY) fy(y Ix)
=~ = =~
fx(x) fy(y)
Hence, I(X;y) = I(Y;X).
(b) We note that
fx,Y(x,y) = fx(x lY)fy(y)
Therefore, we may rewrite (1) as
ioo ioo [ fx, y(x,y) }x
I(X;Y) = fx y(x,y)log2 dy
00 00 , fx(x)fy(y)
From the continuous version of the fundamental inequality, we have
ioo (fX,y(X,y) )
fx,y(X,y)log2 dxdy ~ 0
00 fx(x)fy(Y)
which is satisfied with equality if and only if
485
fx,Y(x,y) = fx(x)fy(y)
Hence,
I(X;y) ~ 0
(c) By definition, we have
h(X IY) = (00 (00 fXy(X,Y)1ogJ 1 lY Ldy
)00 )00' tfX(x ) j
(3)
By definition, we also have
h(X) = (00 fX(X)lOgJ_I_lx
)00 tfX(x) ~
Since
1: fy(y Ix)dy = 1,
we may rewrite the expression for h(X) as
h(X) = (00 fXlogJ_I_l (00 fy(y Ix)dx
)00 tfx(x) j )00
= (00 (00 fy(y Ix) fX(X)lOgJ. l_l_dy
)00 )00 tfx(x) j
But
fy(y Ix) fx(x) = fx,y(x,y)
486
Hence,
h(X) = (00 (00 f,x,Y(X,y)lOg2[_1_1
)00 )00 fx(x) j
(4)
Next, we subtract (3) from (4), obtaining
h(X)  h(X IY) = (00 (00 fXY(X,Y)[lOg2 _1_  log2 1lY YXdY
)00 )00 ' fx(x) fx(x )
foo foo [fX(X lY)}x
= fxy(x,y)log2 dy
00 00 ' fx(x)
= I(X;Y)
(d) Using the symmetric property derived in part (a), we may also write
I(Y;X) = h(Y)  h(Y IX)
Problem 9.28
By definition, we have
h(Y IX) = (00 fx,Y(X,y)lOg2[ 1", Ldy
)00 fy(y ) j
Since
fx,Y(x,y) = fy(y Ix)fx(x)
we may also write
487
hey Ix) = (00 fx(x)dx (00 fy(y lx)log2[ \ ly
)00 )00 fy(y ) r
(1)
The variable Y is related to X as
Y=X+N
Hence, the conditional probability density function fyCy I x) is identical to that of N except for a translation of x, the given value ofX. Let fN(n) denote the probability density function ofN. Then
Correspondingly, we may write
(00 fy(y lx)log2[ IIX ly = r~ flhX)lOg2[ 1 l.y
)00 fy(y ) r )00 f~y  x) r
= (00 fN(n) IOg2[ 1 l.n
)00 fN(n) r
= heN)
(2)
where, in the second line, we have used n = yx and dn = dy for a given x. Substituting ~q. (2.) in (1):
hey IX) = heN) r; fx(x) dx = heN)
488
Problem 9.29
(a) Channel bandwidth B = 3.4 kHz.
Received signaltonoise ratio SNR = 103 .= 30 dB
Hence, the channel capacity is
C = B log2(1 + SNR)
= 3.4 x 103 log2(1 + 103) = 33.9 x 103 bits/second
(b) 4800 = 3.4 x 103log2 (1 + SNR)
Solving for the unknown SNR, we get
SNR = 1.66 == 2.2 dB
Problem 9.30
With '0 distinct brightness levels with equal probability, the information in each level is log2 10 bits. With each picture frame containing 3 x 105 elements, the information content of each picture frame is 3 x 105log2 10 bits. Thus, a rate of information transmission of 30 frames per second corresponds to
30 x 3 x 105 log2._lO = 9 x 106 log2l0 bits/second
That is, the channel capacity is
C = 9 x 106 log210 bits/second
From the information capacity theorem:
489
c = B log2 (1 + SNR)
With a signaltonoise ratio SNR = 103 = 30 dB, the channel bandwidth is therefore
C
B = ::~===::
log2(1 + SNR)
9 x 106 log210
=
log11001
= 3 x 103 Hz
Problem 9.31
The information in each element is log2 10 bits.
The information in each picture is [3 log2 (10)] x 105 bits.
The transmitted information rate is [9log2 (10)] x 106 bits/second.
The channel must have this capacity. From the information capacity theorem,
c = B log2(1 + SNR).
Therefore,
[9 log2(10)] x 106 bits/second = Blog2(1 + 1000).
Solving for B, we get
6 ( log2(1O) ) 6
B = 9 x 10 Hz. = 3 x 10 Hz
log2( 1001)
490
Problem 9.32
Figure 1 shows the 64QAM constellation. Under the condition that the transmitted signal energy per symbol is maintained the same as that in Fig. 1, we get the tightly packed constellation of Fig. 2. We now find that the Euclidean distance between adjacent signal points in the tightly packed constellation of Fig. 2 is larger than that of the 64QAM constellation in Fig. 1. From Eq. (5.95) of the textbook, we recall that an increase in the minimum Euclidean distance dmin results in a
corresponding reduction in the average probability of symbol error. It follows therefore that, with the signal energy per symbol being maintained the same in the two constellations of Figs. 1 and 2, a digital communication systems using the tightly packed constellation of Fig. 2 produces a smaller probability of error than the corresponding 64QAM system represented by Fig. 1.
o
10
5
5
10
10
5
o
5
10
491
5
10
o
5
10
10
5
o
5
10
492
Problem 9.33
In the textbook, the capacity of the NEXTdominated channel is derived as
where Fa is the set of positive and negative frequencies for which S/j) > 0, where S/j) is the power spectral density of the transmitted signal.
For the NEXTdominated channel described in the question, the capacity is
c = ! flog (1 + Bf3/2 Jdf
2F 2 exp(aJJ)
a
Bf312
1 + df
( klf1l2J
exp 1
o
where B, k, 1 and fa are all constants pertaining to the transmission medium. This formula for capacity can only be evaluated numerically for prescribed values of these constants.
493
Problem 9.34
For k=1, Eq. (9./38) reduces to
(1)
Expressing Eq. (3..3:3) in decibels, we have
10 Jog2(SNR) = 6R + 10 JoglO (~ax J
(2)
The number of bits per sample R, is defined by
We thus see that Eqs. (1) and (2) are equivalent, with
494
Problem 9.35
The rate distortion function and channel capacity theorem may be summed up diagrammatically as follows:
j)o:l.o .. ljo...I'Sm. \ SS 10/\ J_ .. rfI .. f:.
'» t:t.J.,.. =rr" ~SIf>A. p',..,"/
According to the rate distortion theory, the data compression limit set by minimizing the mutual information I(X;y) lies at the extreme left of this representation; here, the symbol Y represents . the data compressed form of the source symbol X. On the other hand, according to the channel capacity theorem the data transmission limit is defined by maximizing the mutual information I(X;Y) between the channel input X and channel output Y. This latter limit lies on the extreme right of the representation shown above.
495
1 Problem 9. 3<0
Matlab codes
% Computer Problem in Chapter 9
% Figure: The minimum achievable BER as a function of
% Eb/NO for several different code rates using binary signaling. % This program calculates the Minimum required Eb/NO
% for BPSK signalling at unit power over AWGN channel
% given a rate and an allowed BER.
% Code is based on Brandon's C code.
% Ref: Brendan J. Frey, Graphical models for machine % learning and digital communications, The MIT Press. % Mathini Sellathurai
EbNo=double( [7.85168, 7.42122, 6.99319, 6.56785, 6.14714, 5.7329, 5.32711, ... 4.92926, 4.54106, 4.16568, 3.80312, 3.45317, 3.11902, 2.7981, 2.49337, 2.20617, 1.93251, 1.67587, 1.43313, 1.20671, 0.994633, 0.794801, 0.608808, 0.434862, 0.273476, 0.123322, 0.0148204, 0.144486, 0.266247, 0.374365, 0.474747, 0.5708, 0.659038, 0.736526, 0.812523, 0.878333, 0.944802, 0.996262, 1.04468, ...
1.10054, 1.14925, 1.18536, 1.22298, 1.24746, 1.27394, 1.31061, 1.34588,
1.37178, 1.37904, 1.40388, 1.42553, 1.45221, 1.43447, 1.44392, 1.46129,
1.45001, 1.50485, 1.50654, 1.50192, 1.45507, 1.60577, 1.52716, 1.54448,
1.51713, 1.54378, 1.5684]);
rate= double([9.989372e01, 9.980567e01, 9.966180e01, 9.945634e01,
9. 914587e01 , 9.868898e01, 9.804353e01, 9.722413e01, 9.619767e01,
9. 334680e01 , 9.155144e01, 8. 946454e01 , 8.715918e01, 8.459731e01,
7.881055e01, 7.565174e01, 7. 238745e01, 6. 900430e01 , 6. 556226e01 ,
6. 211661e01, 5. 866480e01 , 5. 525132e01, 5. 188620e01, 4.860017e01,
4.232136e01, 3.938277e01, 3. 653328e01, 3.382965e01, 3. 129488e01,
2.661871e01, 2.45107ge01, 2.251691e01, 2. 068837e01 , 1.894274e01,
1.733225e01, 1.588591e01, 1. 453627e01, 1.326278e01, 1.210507e01,
1.002778e01, 9. 150450e02, 8.347174e02, 7.59800ge02, 6.886473e02, 9.490156e01, 8. 1 78003e0 1,
4. 539652e01 , 2. 88979ge0 1 ,
1.101504e01, 6.266875e02, 5.698847e02, 5.188306e02, 4. 675437e02 , 4.239723e02, 3.851637e02, 3.476062e02, ...
3. 185243e02, 2.883246e02, 2.606097e02, 2.332790e02, 2. 185325e02, ... 1.941896e02, 1.764122e02, 1.586221e02, 1.444108e02, 1.314112e02]); N=66;
b=double([1e5]); % Allowed BER % Rate R (bits per channel usage)
r=double([1/32, 1/16,0.1,0.2,0.3,0.4,0.5,0.6, 0.7, 0.8,0.85,0.95]);
496
le=zeros(1,length(r»; Yo initialize buffer for Eb/NO for p=1:length(r)
c = double(r(p)*(1.0+b*log(b)+(1.0b)*log(1.0b)/log(2.0»); i=N1 ;
Yo Minimum Eb/NO calculations
~h11o ( (1~O) & (c~r~to(i» i=i1 ;
end
i=i+1 ;
if «i>O)
(i<N ) )
e =double ( EbNo (i)+(EbNo (i 1) EbNo( I ) ) * (crate (d ) / (rate (i·1) rate (L) ) le(p)=10*log10«10(e/10»)*c/r(p»;
display(le)
else
display('values out of range')
end
end
plot(10*log10(r),le, ,,) xlabel('Rate (dB)') ylabel('Minimum E_b/N_O (dB)') axis([10*log10(1/32), 0, 2 4])
497
% Computer Experiment in Chapter 9
% Program to create the figure for the minimum % Eb/NO needed for errorfree communication
% with a rate R code, over an AWGN channel
% using binary signaling
~ Thi~ prosr~ e~leul~to~ tho Minimum roquired ~b/NO % for BPSK signalling at unit power over AWGN channel % given a rate and an allowed BER.
% Code is based on Brandon's C code.
% Ref: Brendan J. Frey, Graphical models for machine % learning and digital communications, The MIT Press. % Mathini Sellathurai
EbNo= double([7.85168, 7.42122, 6.99319, 6.56785, 6.14714, 5.7329, 5.32711, ... 4.92926, 4.54106, 4.16568, 3.80312, 3.45317, 3.11902, 2.7981, 2.49337, 2.20617, 1.93251, 1.67587, 1.43313, 1.20671,0.994633, 0.794801, 0.608808, 0.434862, 0.273476, 0.123322, 0.0148204, 0.144486, 0.266247, 0.374365, 0.474747, 0.5708, 0.659038, 0.736526, 0.812523, 0.878333, 0.944802, 0.996262, 1.04468, ....
1.10054, 1.14925, 1.18536, 1.22298, 1.24746, 1.27394, 1.31061, 1.34588,
1.37178, 1.37904, 1.40388, 1.42553, 1.45221, 1.43447, 1.44392, 1.46129,
1.45001, 1.50485, 1.50654, 1.50192, 1.45507, 1.60577, 1.52716, 1.54448,
1.51713, 1.54378, 1.5684J);
rate=double( [9. 989372eOl , 9.914587eOl, 9.868898eOl, 9. 334680eOl , 9.155144eOl, 7.881055eOl, 7.565174eOl, 6.211661eOl, 5.866480eOl, 4.232136eOl, 3.938277eOl, 2.661871eOl, 2.45107geOl, 2.251691eOl, 2.068837eOl, 1.894274eOl, 1.733225eOl, 1.588591eOl, 1.453627eOl, 1.326278eOl, 1.210507eOl, 1.101504eOl, 1.002778eOl, 9. 150450e02, 8.347174e02, 7. 59800ge02 , 6.886473e02, 6.266875e02, 5.698847e02, 5. 188306e02, 4.675437e02, 4.239723e02, 3.851637e02, 3.476062e02, ...
3. 185243e02, 2. 883246e02 , 2.606097e02, 2. 332790e02 , 2. 185325e02, ... 1.941896e02, 1.764122e02, 1.586221e02, 1.444108e02, 1.314112e02J); N=66;
9.980567eOl, 9.966180eOl, 9.945634e01,
9.804353e01, 9.722413e01, 9.619767e01, 9.490156e01,
8.946454e01, 8.715918eOl, 8.459731e01, 8. 178003e01,
7.238745e01, 6. 900430e01 , 6.556226e01,
5.525132eOl, 5. 188620e0 1, 4.860017e01, 4. 539652eOl ,
3.653328eOl, 3. 382965e01 , 3. 129488e01, 2. 88979ge01 , b=double(0.5:1e5:1e5); % Allowed BER rrr=double([0.99,1/2,1/3,1/4,1/5,1/8]); % Rate R(bits/channel usage) le=zeros(l,length(b»;
for rr=1:length(rrr) r=rrr(rr) ;
for p=1:length(b)
c = double(r*(1.0+b(p)*log(b(p»+(1.0b(p»*log(1.0b(p»/log(2.0»);
498
i=N1 ;
while ( (i>=O) & (c>rate(i» ) i=i1;
end
if «i>O)
(i<N ) )
e = double(EbNo(i)+(EbNo(i1)EbNo(i»*(crate(i))/(rate(il)rate(i)) le(p)=10*log10«10(e/10»*c/r);
else
display('values out of range')
end
end
plot(le,10*log10(b), ,,) end
xlabel('E_b/N_O (dB)') ylabel('Minimum BER') axis([2 1 50 10])
499
Answer to Problem 9.36
4r.rrro
2
co
~
0
Z
:0
LU
E
:J
E
·c
~
0 1
3..
_2L L L ~
15
10
Rate (dB)
5
o
Figure 1: The minimum Eb/NO needed for errorfree communication with a rate R code, over an AWGN channel using binary signaling
500
10
20
1.5
_50L_~~~LL·L~J~
2
o
0.5
1
Figure 2: The minimum achievable BER as a function of Eb/NO for several different code rates using binary signaling
501