0 for all R,
0 (Q;P), E,(p,Q) — pR is
maximized by p = 0, yielding E,(R,Q) =
In summary, for R in the range given by (5.6.30), E,(R,Q) and R are
related by (5.6.31). For smaller values of R, £,(R,Q) and Rate related by the
linear relation (5.6.33), and for larger values, E,(R,Q) = 0. Asa function of
R, E(R.Q) is strictly decreasing and positive for all R <.A(Q:P).
Now consider the special case where 2*£,(p,Q)/@p* = 0. From (5.6.26a)
in Theorem 5.6.3, we sce that this relation must be satisfied for all p > Oif it
is satisfied for any p > 0. In this case, 2F,(p,Q)/ap is a constant and the
range over which (5.6.30) is satisfied is of zero extent (see Figure 5.6.15).
This special case is rather pathological and occurs only for noiseless channels
[for which H(¥ | ¥) = 0] and for some rather peculiar channels such as that
in Figure 5.6.16,
For the usual case, in which 0°E,(p.Q)/p* <0, the parametric equations
of (5.6.31) apply over a nonzero range of rates. From (5.6.32) and (5.6.31),
we have G°£,(R,Q)/AR? = —[0*E,(p,Q)/2p'}* > 0 and, thus, EARQ) is
strictly convex U in R over this range of R. Since 0°E,(R,Q)/2R® = 0 outside
of this range, we see that E,(R,Q) is convex U in R for all R > 0.
‘The random coding exponent, E,(R) can now be, related to E,(R,Q) by
E,(R) = max E,(R,.Q) (5.6,34)
°
This is a maximum over a set of functions that are convex U and decreasing
in R, It is easy to see (Problem 4.12) that the maximizing function is also
convex U and decreasing in R. Also, for the Q that yields capacity on the
channel, E,(R.Q) is positive for R <.4(Q;P) = C and, therefore, E,(R)
is positive for R < C. This proves the following fundamental theorem.
Theorem 5.6.4 (Noisy-Channel Coding Theorem). For any discrete
memoryless channel, the random coding exponent E,(R) {see (5.6.16) and
(5.6.18)] is a convex U, decreasing, positive function of R for 0 < R Tor that max E,(p,Q)/p
approaches C arbitrarily closely as p> oo. In either case, since £,(p,Q) is
convex in p for fixed Q, we must have E,(p.Q) = pC for some choice of
Q. From Theorem 5.6.3, this can happen iff (5.6.26a) is satisfied for that Q.
Using the same argument, these conditions are also necessary and sufficient
for R,, = C. Summarizing, the three following statements are all equivalent
(1) Ry = C; Q) R,, = C; and (3) Equation 5.6.26a is satisfied for some Q
yielding capacity.
Theorem 5.8.2 (Straight-Line Bound). For any discrete memoryless
channel for which £,,(0) < 00, let E,,(R) be the smallest linear function of
R which touches the curve £,,(R) and also satisfies E,,(0) = E.,(0). Let
Ry be the R for which £,(R) = E,,(R). Let 0,(N) and o4(W) be functions Lower Bounds to Error Probability 161
vanishing with increasing N which can be taken as
oy) = 82 4 WS + KIN (688)
y
lf ain ¥ VPUT PUTO | ie
eS yg
Villog: VN Ne Pain
In2 | Sin2 | Exl0)
+ — 5.8.10)
tnt Eun ¢ )
Then, for any positive integer N and any R, 0(N) < R < Rj, every (NLR)
code has an error probability satisfying
P. > exp (~ME LR — o/N)] + 0,(N)}) (5.8.11)
This is Theorem 4 in Shannon, Gallager, and Berlekamp (1967) and a
proof is given there. There is a slight error in the statement of that Theorem 4
in that the restriction R > 0,(N) is omitted. The proof is correct for the
theorem as stated here, however. The straight-line exponent E,,(R), com-
bined with the sphere-packing exponent, gives an upper bound to the relia-
bility E(R) for all R, 0 < R < C in the limit of large N. These bounds are
sketched in Figure 5.8.2 for the same channels as in Figure 5.8.1.
‘We now turn our attention to proving these theorems for the special case
of the BSC. We shall start with an arbitrary (N,R) code and find a lower
bound to P, which is independent of the code and thus a lower bound to P,
for all codes of the given N and R. Let x1, . . . , Xy, be the code words in the
code, M = fe], and let ¥,,..., Yy, be the decoding regions for the code
(that is, Y,, is the set of channel-output sequences that are decoded into
message m). If message m enters the encoder, x,, will be transmitted, and
correct decoding will ensue if a y € Y,, is received. Since, conditional on m,
this is an event of probability
= Pals)
the overall probability of correct decoding is
We POL 8.12)
M nmi ®F 5
Now define the Hamming distance d(y;x) between two binary sequences as
the number of positions in which the two sequences differ. For example, the
Hamming distance between (0,0,1,1,1) and (1,0,1,0,1) is 2. For a BSC with
crossover probability ¢, if d(y;x,) =m, then Ply |x,) = "(1 — "If
P. 162 ‘The Noisy-Channel Coding Theorem
109]
EelR)
E(B)
a)
5 2
© @
Figure 5.8.2. Bounds on reliability function (same channels as Figure 5.8.1).
we let A,,,, be the number of sequences y that are decoded into message m
and have distance » from x,,, then we can rewrite (5.8.12) as,
Lee
Pah SS Aner — 0 (5.8.13)
Using the binomial equality
‘ jea ee
the probability of error, P, = 1 — P, is given by
pelt 3[( ) ~ Aamo =e (5.8.14)
Mico An)
To interpret this, we notice that, if message m is transmitted, A... is the Lower Bounds to Error Probability 163
number of sequences at distance from x,, which, if received, will cause
correct decoding. Since (") is the total number of sequences at distance
n
from Xp, [() = Anim| of these sequences will cause decoding errors if
a) J
mis transmitted
We shall now find some constraints on the set of numbers {4,,,.} which
are valid for all codes with the given V and M and then minimize the right-
hand side of (5.8.14) subject to these constraints. The constraints that we
shall use are
Aum S (*)5 all nm (5.8.15)
n)
SF Say <2” (5.8.16)
The constraint (5.8.16) arises from the fact that there are 2” output sequences;
each is decoded into at most one message, and is at a unique distance from
the associated code word.
The minimum of (5.8.14), subject to these constraints, is achieved for
<4 by selecting, for all m,
dan={(n)
where k is chosen so that
(5.8.17)
Stew) & . N
wz ( ) +2 Asm 0< 3 Aya SM(7) (5.8.18)
‘The individual values Ay... are immaterial so long as their sum over m
satisfies (5.8.18). To see that this choice minimizes (5.8.14), observe that for
any other choice, we can find an n’ and ann, n’ E,(p,Q)/p has been con
sidered in detail by Jacobs and Berlekamp (1967). Their results apply to
arbitrary tree codes (that is, codes with a structure such as in Figure 6.9.2)
and to a class of sequential decoding algorithms including the Fano algorithm
described here. Let E,(p) = max E,(p,Q) and let £,(p) be the convex 0
°
hull of £,(p). That is, £,(p) is the smallest convex function greater than
or equal to E,(p); it is formed by replacing all the nonconvex © portions of
E,(p) with straight-line segments. In our notation, Jacobs and Berlekamp
show that, for any tree code of rate R > E,(p)/p,
ne")
Nov \N smo
(6.9.31)
‘The fact that higher-order moments of W, are infinite indicate that long
waiting lines will be a severe problem in sequential decoding. Jacobs and
Berlekamp show that the probability per digit that either the waiting line will
exceed a given length / or that an error will be made can decrease no faster
with / than as i-? for a code of rate E,(p)/p.
Error Probability for Sequential Decoding
In order to obtain results on error probability for sequential decoding,
we must consider codes of finite constraint length, For a code of infinite
constraint length, if 7, is finite for each m, then ftom (6.9.18) the probability
of hypothesizing a node of depth / in a tree of incorrect paths goes to zero
with increasing / and the error probability is clearly zero.
Assume that the encoder has a constraint length of L subblocks. Assume
also that the encoder, instead of accepting an infinite sequence of subblocks
from the source, accepts a finite number of subblocks Lp, where Ly is
typically much larger than L. After the Ly subblocks of data have entered the
encoder, a sequence of L — 1 subblocks of known zeros enter the encoder. In
the received value tree for such a situation, each node of depth Ly — 1 or less
will have 2’ immediate descendants and each node of depth Ly or greater
will have only one immediate descendant. Decoding proceeds according to
the rules of Figure 6.9.4, except that the decoder recognizes that nodes of
order Ly or greater have only one immediate descendant and thus rules 3
and 5 always yield backward moves on nodes of order greater than Ly. The
decoding terminates with the first Fhypothesis of a node of order Ly + L— 1
and the hypothesized source sequence at that point is taken as the
decoded sequence. For finite L and Ly the decoding must terminate
eventually. Sequential Decoding 281
If we consider the ensemble of codes defined after (6.9.10), with L finite,
it can be shown that the pair of code sequences corresponding to a given pair
of source sequences are statistically independent over the first L subblocks
starting with the first subblock in’ which the message sequences differ, but
are not independent thereafter. The probability of decoding error for this
ensemble can be upper bounded by a quantity essentially equivalent to the
random-coding bound for block codes (see Problem 6.42). Here, however,
it will be more interesting to modify the ensemble slightly and then derive a
much smaller upper bound on error probability. In the modification, we shall
make the binary convolutional encoder time varying, with the binary outputs
generated by the rule [compare with (6.9.10)].
28
=X Sandadurs 1 1
for (7.3.37).
‘Theorem 7.3.2 (Coding Theorem). Let £,(R), E,,(R), and C be defined by
(7.3.36), (7.3.37), and (7.3.1) for an arbitrary discrete-time memoryless
channel with an input constraint f(x) < &. Let R > 0 be arbitrary and let
E < max [E,(R),E,,(R)] be arbitrary. Then, for all sufficiently large N, there
exists a code of block length N with M = [e*®|code words x,,..., x,y, each
satisfying the constraint
XS) SNE,
and each satisfying .
Pug S exp (—NE) (7.3.38)
Furthermore, £,(R) > 0 for all R,0 << R min o,2). Thus the equation
& = 6(B,p) determines B as a function of p for a fixed energy &, and this
implicitly determines R(B) as a function of p (or p as a function of R and 6).
For p = 0, the resulting rate is simply the channel capacity for the given, &,
as can be seen by comparing (7.5.28) and (7.5.32) with Theorem 7.5.1. The
exponent, E(B,p), is zero for p = 0. AS p increases, for fixed energy &, B
decreases, R(B) decreases, and £(B,p) increases. As before, the slope of £
as.a function of R is —p, as seen most easly from the graphical interpretation
of Figure 5.6.4. When p increases to 1, for fixed &, B has decreased to a
critical value B,, given by
6B,)= y n=<
E,(R)
15.35)
nny 2B, = 0, (7538)
Corresponding to B,, is a critical value of rate determined by
Re= EY WinBe (7.5.36)
‘The parametric equations (7.5.28), (7.5.32), and (7.5.34)are only applicable,
for fixed &, for R in the range R,, < R < C. For R < R,,, the exponent E 350 Memoryless Channels with Diserete Time
is maximized by p = 1 with the solution for 4, and 8, as before. This leads
to an exponent £,(R) given by
EAR) = E(Bosl) + Rep — RK (1.5.37)
It should be observed that over the entire range of rates, the appropriate
distribution of energy over the set of channels is given by (7.5.27) for the
appropriate B and p. As R changes for a fixed 6, both B and p change and
the distribution of energies changes. In other words, using Theorem 7.5.1 as
an engineering guide in how to distribute energy among parallel Gaussian
noise channels is not always a wise thing, even in a system using sophisticated
coding techniques.
The coefficient [2e"*/u}* in the probability of etror expression can be
estimated by the central limit theorem as before. We have*
pe ote (7.5.38)
Vand
Choosing 4 = I/s, and using (7.5.29), we obtain
aes epVaT DE
— y (7.5.39)
wo (+ pyB
The expurgated random-coding bound for parallel additive Gaussian
noise channels is derived as an extension of the result for a single Gaussian
noise channel in the same way as the random-coding bound. Using the
ensemble of codes described by (7.5.9), we find that for any R > 0, there is
a code of M = [el] code words, each satisfying the energy constraint and
each satisfying
Paw SEXP — PR’ + YEP Yous) (75.40)
where 7
R422 (75.41)
“
Elp.XwYuss) = 2psé, + In (1 aos + £4) +2 in —288,)
2 2pa,t) "2
* This is only a good approximation if each &, is small compared to é [see Feller (1966),
Vol. I, Problem 19, p. 302}, We have also asstimed that 9 is small relative to the standard
deviation V254,2, thus assuming that the density of
LaF
constant between & — 6 and 4. In other words, (5.7.38) is close only if both ys & 1 and
if 2, « € for all n, Parallel Additive Gaussian Noise Channels 351
Making the substitutions A, = &/o,? and
= 286, + Anl(2p) (7.5.42)
we can rewrite £,(p,Xq. ¥,.8) as
Ex(AnBuop) = (1 = Bap + 4
2in [a( = =| (7.5.43)
We now maximize
B= ~pR +3 EAB) (75.44)
over A,, By, and p. The A,, are again constrained by A, > 0, 3,24, = 6.
Since E is a convex © function of the ,, a necessary and sufficient condition
‘on the maximum of E over the A,, is, for some A and all n,
OE 41 - p
24,2 |!” 298, —4,] $“*
(7.5.45)
with equality if 4,, > 0. We shall again temporarily ignore the relationship
between the f,, imposed by (7.5.42) and maximize over each £,, separately.
This yields
(7.5.46)
(1841)
(7.5.48)
Thus, defining B as 1/(42), (7.4.48) becomes
Bot > B (7.5.49)
with equality if A, > 0.
From (7.5.47), B, = 1 for A, = 0 and thus from (7.5.49) 0,2 > B. Also,
By > 1 for A, > 0, and thus, since #,0,2 = B, 0,2 < B. Again, only those
channels with 0,2 < Bare used with positive energy.
We next solve for the &,. From (7.5.46),
~ dob = 1)
Ay
2B,—1
(7.5.50) 382 Memoryless Channels with Discrete Time
For o,? < B, we have , = Bja,2, and (7.5.50) becomes
¢, ~ PRB = 6) (1551)
2B;
é= 4086 =o. (7.5.52)
meter 2B—0,¢
Equation 7.5.52 gives an implicit solution for B in terms of 6, and this in
turn determines &, from (7.5.51) and p, from (7.5.49).
For A, > 0 and fi, satisfying (7.5.46), we have calculated the value of s
satisfying (7.5.42) in (7.4.55)
s (1.5.53)
‘Thus the same value of s satisfies (7.5.42) for all n and the solution for By,
E,,, is consistent.
Next E is maximized with respect to p where 2£/@p = 0, or where
A
set Me
2(2pBy — An)
As in (7.4.47) to (7.4.51), this leads to
Yi-fet
Bat
Dying (7.5.55)
EAR) = 2% (7.5.56)
Using (7.5.49), these equations reduce to
Ra 3 Mb apa ad (7.557)
E,AR) £ (7.5.58)
Equations 7.5.52, 7.5.57, and 7.5.58 relate &, R’, and E,(R’) parametrically
in terms of p and B. They are valid only for p > 1, which for a given &, puts
an upper bound on B from (7.5.52). Comparing (7.5.52) for p = 1 with
(7.5.36), we see that this limiting B is just B.,. Thus, for a given &, the
expurgated bound is valid for
(7.5.59) Parallel Additive Gaussian Noise Channels 353,
Finally, using the approximation (7.5.38) for , choosing 6 = 1/s, and
using (7.5.53) for s, we can relate R’ to the rate R in (7.5.41) by
a [fe
RwxR42in{ = lands! 1.5.60)
Lapp" (7560)
‘The results of this section are summarized in the following theorem from
Ebert (1965).
Theorem 7.5.2. Let a set of IV parallel, discrete-time, additive Gaussian
noise channels have noise variances o,2,..., ay’. For any* B > Oand any
p. 0 mino,?
and any p> 1, there exist codes with M = fexp [R,'(B) — 2 In (2e/8]]
words, each of energy at most &,(B,p), and each with
Pom S exp [—E.(B,p)] (7.5.63)
where R,’, 6, and E,, are given by (7.5.57), (7.5.52), and (7.5.58) respectively.
For fixed & = &,(B,p), R,’(B) is strictly and continuously decreasing from
Racy to 0 and E,(B,p) strictly increasing as p increases from 1 toward 00.
For the application of these parallel channels to continuous time channels
in the next chapter, we shall see that the coefficient in (7.5.61) and the dif-
ference between R and R’ in (7.5.60) are negligible, leading to our emphasis
* In the trivial case where B < min 6,2, 6 R, and E are all zero 354 Memoryless Channels with Discrete Time
on the exponents £,(R) and E,,(R’). Ebert (1965) has also derived lower
bounds on error probability for parallel additive Gaussian noise channels
and the exponent in his lower bound agrees with E,(R) for Re, < RC.
Summary and Conclusions
In this chapter, we have extended the results of Chapters 4 and 5 to
discrete-time, memoryless channels with arbitrary input and output spaces.
The basic approach here has been to use only a finite set of letters in the
input alphabet and to partition the output alphabet in order to reduce the
channel to a discrete memoryless channel, The major difference between this
chapter and Chapters 4 and 5 is the inclusion of constraints on the input
alphabet. In Sections 7.4 and 7.5 we have applied the general results of the
first three sections to the important cases of discrete-time additive Gaussian
noise channels and parallel discrete-time additive Gaussian noise channels.
‘The importance of these channels will become apparent in Chapter 8, when
we reduce a continuous-time channel with additive Gaussian noise to a set
of parallel, discrete-time, additive Gaussian noise channels.
Historical Notes and References
‘Most of the results in Sections 7.1 to 7.3 are new except for an earlier
appearance of the analysis of input constraints (Gallager (1965). A number
of the results have been discovered independently, however, by Wagner
(1968). The random-coding bound on error probability for the discrete-time
additive Gaussian noise channel is due to Shannon (1959), who also derived
a sphere-packing lower bound to error probability. Slepian (1963) has
numerically evaluated these upper and lower bounds for a range of block
Iengths and number of code words. The results of Section 7.5 on parallel,
discrete-time, additive noise channels are due to Ebert (1965), who also
derived lower bounds to error probability for these channels. Chapter 8
WAVEFORM CHANNELS
8.1 Orthonormal Expansions of Signals and White Gaussian Noise
In this chapter, we shall discuss channels for which the input and output are
functions of time where time will now be taken as defined on the continuum
rather than at discrete instants. This immediately faces us with the conceptual
problem of the “probability” of a waveform. The probability of one of a
discrete set of events is a rather simple concept and the probability distribu-
tion function of a continuous-valued random variable is not much more
involved conceptually. Of course, we could describe a random waveform (or
random process, as it is usually called) at a particular instant of time with a
probability distribution but, in general, not even a joint probability distribu-
tion over a large number of instants of time would suffice as a complete
statistical description of the waveform. In principle, we consider a random
Process to be completely specified if we have a rule from which the joint
probability distribution over any finite set of times could be calculated. We
shall not pursue that approach in this chapter, however: instead we shall take
the approach of representing any given real or complex waveform as a series
expansion of a sct of orthonormal waveforms. We shall then characterize
random waveforms in terms of joint probability distributions over the co-
efficients in such a series expansion, While this approach may seem rather
abstract and cumbersome at first, it will turn out to be more useful, both ia
developing intuition and in proving theorems, than an approach based on
the behavior of the waveforms at various instants of time.
‘Throughout this chapter, we shall be faced with a number of mathematical
details such as the convergence of series, the interchange of orders of summa-
tion and integration, and the existence of limits. These problems cannot be
355 356 Waveform Channels
argued away on physical grounds since we are dealing here only with mathe-
matical models of physical problems. In fact, we can often obtain a great
deal of physical insight about a problem by investigating where mathematical
limits fail to exist. On the other hand, if we fuss incessantly about convergence
and limits, we shall obscure the more important issues and lose those readers
who have neither the background nor the inclination to follow intricate
limiting arguments. Many of these questions concerning convergence can be
circumvented or, at least, simplified if we restrict out attention to finite energy
functions, that is, functions 2(1) for which* {|x(|? di < 00. These functions
are often called L, functions, While they constitute quite a general class, it
should be observed that neither impulse functions nor infinite duration sine
wave have finite energy.
‘Two functions, gy(1) and (2), are called orthogonal if fy()os*(1) dr = 0
where 9,*(1) is the complex conjugate of (1). A function is said to benormal-
ized if its energy flpx(1)|E dt is equal to 1. An orthonormal set is defined as a
set of functions G4(t), Fe(0.--. each of which is normalized and each pair of
which is orthogonal; thus, for all ¢,, ¢; in the set, we have
[rior dt =o, 1)
where 5,, = I for i= jand 4,, = 0 otherwise.
Suppose now that a function x(t) can be represented in terms of an
orthonormal set by
2) = Yat (8.1.2)
Under these circumstances, the coefficients x, will satisty
*, front dt (8.1.3)
To see this, substitute (8.1.2) into (8.1.3) and integrate
(8.1.1).
Now let 2(1) be an arbitrary L, function and let x, be given by (8.1.3). We
want to investigate whether the expansion
can still be used to represent 2(1). Let x(t) be the remainder when k terms
of the expansion are used to represent «(1),
20 = 4) — S9) (8.14)
+ Throughout this chapter, any integral without specified limits is to be interpreted as
being from —<0 to =>, Orthonormal Expansions of Signals and White Gaussian Noise 357
If we multiply both sides of (8.1.4) by g,*() and integrate, we see immediately
that (1) is orthogonal to ,(1) for all 7 < k.
The energy in 2, ,(t) is given by
fisscorar =f [ro - Jenol[#o = Sern] dt
fleoyede = atfeonnoe
~Safeonwdes Yams [rconre ar
= flatartar — 3 ta (15)
To get (8.1.5), we have used (8.1.3) and its complex conjugate, 2,
Jet (DGC) de.
Since the energy of «,,. is nonnegative for all k, we have
Dial? < fle iear (6.1.6)
Steuts flewor ar 1
Equation 8.1.7 is known as Bessel’s inequality and will be used frequently
in what follows.
We can now investigate the limiting behavior of z,.(0) by considering the
difference between z,,,(t) and x,, (1) for > k. From (8.1.4), this difference is
D x9.
and has an energy el
> lal?
Bessel’s inequality shows that“
2?
is bounded and therefore "
> lel
must approach 0 as k and / increase without limit, Thus x,.4(1) approaches a
limit,* z,(1), which is orthogonal to all the (1).
2,(t) = Lim. 2, (0) (8.1.8)
* The existence of ths limit function is implied by the Riesz-Fischer Theorem. Sec, for
example, Riese and Nagy (1955). 358 Waveform Channels
‘The notation Lim. in (8.1.8) stands for limit in the mean. It means that
lim fiso- pa dt =
and does not necessarily imply that «, (1) approaches a limit for each value
of f. We can now expand 2(1) as
H0=Sd+ 20 (8.1.9)
a= [xoer@d; fr@vr@ a= 0: alli
By the infinite sum, both in (8.1.9) and throughout this chapter, we mean
Gilt)
In geometric terms, (8.1.9) states that a function 2(r) can be decomposed into
‘two terms, one in the subspace generated by the g,(t) and the other orthog-
onal to the subspace of the 9,0).
In dealing with (8.1.9), we can usually treat the infinite series just as if it
were a finite series. We can usually justify this use by means of the Schwarz
inequality, which states that, for two L functions, 2(t) and y(0),
| | a(Dy"(O dt
1
Pam 0 (See 5.6.23). Then, for all p > 0, E,(p,Q) is strictly in-
creasing and convex © as a function of p. The convexity is strict unless the
channel is noiseless in the sense that, for each pair (i,k) of inputs that are 184 ‘The Noisy-Channel Coding Theorem
Q(i) > 0, O(k) > 0), we have either P(j| KPC |).
P(j |i) forall j.
for all
‘This theorem is proved in Appendix 5B. We observe from (5.7.11) that
E,g(R) can be interpreted as the least upper bound of a set of linear functions
of R,
—pR + max E,l.Q)
a function of slope —p for each p > 1. (See Figure 5.7.1.) We observe
that, for p = 1, the sum on j can be interchanged with that on i, k in the
definition of E,(p,Q), and from this it can be seen that £,(1,Q) = £,(1,Q).
max Ex(1, Q)
@
= max Eo(l, @)
Q
Figure 5.7.1. Comparison of E,,(R) and E,(R).
‘This demonstrates the relation between E,,(R) and E,(R) as shown in
Figure 5.7.1. Since E,(p,Q) is strictly increasing with p, the above geometric
interpretation proves that, in general, £,,(R) > E,(R) for sufficiently small
R. In the limit of a very noisy channel, however, this difference becomes
arbitrarily small, as shown in Problem 5.31
In any region where E,,(R) = E,(R), we note that p = | and, thus,
i
[
na»
which is precisely the same as the uniform bound on P,,., given by (5.6.20).
The expurgated bound of (5.7.10) is thus strictly tighter than the random-
coding bound in the region where [R + (In 4)/N] is strictly less than the
smallest R for which E,,(R)
It can happen that £,,(R) is infinite for sufficiently small R. To investigate
this, we observe that the Raxis intercept of the linear function — pR + E,(p,Q)
\
Pam oxn[ nf. (R-+ #4)]} = sear af A + mgs (10) Error Probablty for an Expurgated Ensemble of Codes 155
As p-> 0, —pR + E,(p,Q) approaches a vertical line at
R= lim Elp.Q)/p
is at E,(0,Q
and £,,(R) is infinite for
R< tim £,(p,Q)ip
(In other words, for such an R, —pR + E,(p,Q) approaches 0 as p—> 0.)
Evaluating this limit by L’Hospital’s rule, we obtain
tim FD = — In | Soar. (8.7.14)
prep
[ SvPaTWeG|D 0
mm | 5 (5.7.13)
oO; otherwise
Let Ry... be the maximum value of (5.7.14) over Q,
Raao = mgx In| FOO] (5.7.16)
We have concluded that £,,(R) = 00 for R < Ry,
It can be seen from (5.7.14) that Rey = 0 if gy, = 1 for all k, i and,
otherwise, Ry, > 0. If g,; = 0 for some k and i, then those two inputs can
never be confused at the receiver, and at least one bit per channel use can be
transmitted over the channel, with no possibility of errors, by using only
those two inputs. If we choose O(k) = Q(i) = 14 for such a pair of inputs,
the right-hand side of (5.7.14) is In 2, We see from this that, if Ry... #0,
then R,,., must be at least In 2. Shannon (1956) has defined the zero-error
capacity of a channel as the largest rate at which data can be transmitted over
the channel with zero-error probability (as opposed to arbitrarily small error
probability). Since P,,,, = 0 for R < R,,.., we conclude that R,,.. is a lower
bound to the zero-error capacity of the channel, Figure 5.7.2 sketches E,,(R)
for two channels with R,,,, >0. In Problem 5.25, a simple expression is
derived for Re, .:-
The actual maximization of F,.(R) over p is quite similar to that of the
random-coding exponent. If we define
E.A{R,Q) = sup — pR’ + ExCp,Q)
then we obtain the parametric relations
+ Efp,Q) (5.7.17)
(5.7.18) 156 ‘The Noisy-Channel Coding Theorem
valid for
(5.7.19)
For larger values of R, the bound is equivalent to the straight line portion
of the random-coding bound, as discussed before. For the typical channel,
Ryn =n2
@
@
Figure 5.7.2. E,.(R) for channels with Ry, > 0-
Ry = 0, and as shown in Problem 5.24, we then have
lim E,,(R,Q) = — ¥ O(kyQUi) In [z VPGTHRTT)) 6720)
Rao
Relatively little is known about maximizing the bound over Q. The function
E,{p.Q) is not convex © in Q and can have several local maxima over Q.
Even more disturbing, if we try to optimize the expression for P,,,, in (5.7.7)
over Q(x), not restricting ourselves to a product distribution, we sometimes
find cases where a product distribution does not optimize the bound (see
Problem 5.26). In the case of any discrete memoryless channel with a binary
input, however, Jelinek (1968) has found that a product distribution always Lower Bounds to Error Probability 157
optimizes the bound, and, in fact, the optimum Q is Q(0) = Q(1) = 4
(see Problems 5.29 and 5.30).
5.8 Lower Bounds to Error Probability
In the preceding sections, we have established upper bounds on the
probability of decoding error that can be achieved on a discrete memoryless
channel in terms of the block length N and the rate R of the code. In this
section, we shall be concerned with finding the minimum probability of
error that can be achieved by any (N,R) code. We shall assume, throughout
this section, that all code words are used with equal probability. Indeed,
without some such assumption, no nonzero lower bound on error probability
would be possible, for if one of the code words were used with probability 1,
the decoder could always decode the message corresponding to that code
word and no errors would ever be made.*
For equally likely messages, the probability of decoding error for a code
of M messages is
La
Pea g 2Pem
where P,,», is the probability of error given that message m is transmitted.
We saw in the last section that for any block length N and any rate R > 0,
there exist (V,R) block codes for which both P, < exp [—NE,(R)] and
P.< exp [—NE,AR + In 4/N)]
The known lower bounds on P, for a given N and R are considerably more
tedious and subtle to derive than the upper bounds. Thus, we shall merely
state the relevant theorems here. Proofs may be found in Shannon, Gallager,
and Berlekamp (1967). Proofs for the special case of the binary symmetric
channel (BSC) will be presented here. Most of the ideas in finding lower
bounds to P, are present in this special case, but much of the detail is
avoided.
Theorem 5.8.1 (Sphere-Packing Bound). For an
memoryless channel,
(N,R) code on a discrete
P, & exp (— ME, [R — 0,(N)] + 02N)}) (5.8.1)
where
E,(R) = sup| max Ey(p.Q) — pk] (5.8.2)
polo J
and £,(p,Q) is given by (5.6.14). The quantities 0,(N) and o,() go to zero
* Itis possible, however, to lower bound the ertor probability of the worst code word in a
ccode, that is, max P,.,,, without regard to the probabilities with which code words are used
(see Problem 5.32). 158 ‘The Noisy-Channel Coding Theorem
with increasing N and can be taken as
In8 | KInN
o(N) =e +S (5.8.3)
oN (5.8.4)
where Pin is the smallest nonzero transition probability for the channel and
K is the size of the input alphabet.
It will be observed that the function E,,(R), called the sphere-packing
exponent, is defined in almost the same way as the random-coding exponent
E,(R), the only difference being in the range of p over which the maximization
is performed. As a consequence, the results of Section 5.6 are immediately
applicable to E,,(R). In particular, E,,(R) is positive for 0< R 1 and j> 1, then the corresponding field integers would
satisfy i-j = 0. This is impossible since i and j are nonzero elements of the
nal field. We have already seen that the integers modulo a prime form
a field, so this is the subfield that we are looking for. Finally, any other sub-
ficld must contain these integers of the field and the additive group of any
subfield must contain these integers as a subgroup. Thus the number of
elements in any other subfield is divisible by p and not prime. |
One immediate consequence of this theorem is that any field or subfield
with a prime number of elements is, with appropriate labelling of elements,
just the field of integers modulo the prime number. The characteristic, p,
of a Galois field is defined as the number of elements in the above prime
subfield.
If P(D) = Py + P,D + +--+ PD” is a polynomial over a field £ and
if field F is an extension of E, then we say that an element in F is a root
of P(D) if P(x) = 0; that is, if
0
yPy
As a common example, we often find complex roots of polynomials over the
real-number field. If Eis a subfield of a Galois field F then the minimal poly-
nomial f,{D) over E of an element « in F is defined 10 be the monic polynomial
over E of lowest degree for which x is a root, If the subfield is not explicitly
specified, we shall always mean the subfield with a prime number of elements
implied by Theorem 6.6.2
Theorem 6.6.3. For any subfield £ of a Galois field GF(q), and each non-
zero element « in GF(q), « has a unique minimal polynomial f(D) over E 28 Techniques for Coding and Decoding
and f,(D) is irreducible. Furthermore, for each polynomial P(D) over E,
f(D) divides P(D) iff « is a root of PD),
Proof. We have already seen that « is a root of Dr! — 1. Since D™! — 1
can be considered as a polynomial over E, this shows the existence of poly-
nomials over E for which « is a root, and thus that there is some lowest
degree monic polynomial, say f,(D), for which « is a root. If f,(D) is reducible,
then it can be expressed as the product of two lower degree monic poly-
nomials over E, say f,(D) = g(D)h( D). This, however, implies that g(a)h() =
0, and since g(x) and (2) are elements in GF(q), one or the other must be
zero, contradicting the lowest degree assumption for f,(D). Thus f,(D) is
irreducible. Now, for any P(D) over E,
P(D) = fal D)h(D) + r(D) (6.6.2)
where r(D) is a polynomial over £ of lower degree than f,(D). Since f,(2) = 0,
we have P(a) = r(x). Thus P(a) = 0 iff r(x) = 0, But since r(D) is of lower
degree than f,(D), we have r(x) = 0 iff r(D) is the zero polynomial. Thus
P(a) = 0 iff f,(D) divides P(D). Finally, if PD) is a monic polynomial of the
same degree as f,(D), P(D) = f,(D)h(D) is only satisfied for h(D) = 1,
showing that f,(D) is unique. |
In the above theorem, the statement that /,(D) is irreducible means that
there are no lower degree polynomials over E whose product is /,(D). If we
interpret f,(D) as a polynomial over GF(q), then D — « is an obvious factor
of f(D).
COROLLARY. Fora subfield £ of a Galois field GF(g), let f(D), - -. f(D)
be the distinct minimal polynomials over F for the nonzero elements of
GF(g) {that is, f,,(D), .-- + fz, ,(D) with repetitions removed). Then
pe =1= TT £0) (6.63)
Observe that (6.6.1) factors D*- — 1 into irreducible polynomials over
GF(q), and (6.6.3) factors D*! — 1 into irreducible factors over E.
Proof. Since each nonzero element in GF(q) is a root of D*1 — I, each
minimal polynomial divides D™? — 1. Since the minimal polynomials are
irreducible (over the field £),
TIA)
also divides D’-* — 1. Finally, all the nonzero elements of GF(q) are roots of
1
TA.
Hence the above product has degree at least g — I and (6.6.3) is satisfied. | Galois Fields 29
Theorem 6.6.4. Let x be a primitive element in a Galois field GF(g) of
characteristic p and let the minimal polynomial of «, f(D) [over GF(p)] have
degree . Then the number of elements in the Galois field is p" and each
clement f in the field can be represented as
he tiy (66.4)
for some choice of ig, i... i, 4 as integers of the field.
Proof. Since « is a root of f(D) = D” + fy. D4 + +++ + fo, we have
Ot fpr pt fy
(6.6.5)
Since each fis a field integer, we see that 2"
Multiplying (6.6.5) by %, we get
att YS fig (6.6.6)
‘an be represented as in (6.6.4).
Thus, since a” can be represented as in (6.6.4), a" can also, and successively
multiplying (6.6.6) by higher powers of a, we see that all powers of # can be
represented as in (6.6.4). Thus each element of GF(g) can be represented as
in (6.6.4). Now suppose that there are two different choices of the set of
field integers in (6.6.4), say ina, . «5 ig and j, 1, -.. jy that correspond to
the same element in GF(q). Then
0=(i
nade" bo + (iy = de + (iy — jn) (6.6.7)
This, however, asserts that « is a root of the polynomial (i, — j,-3)D"! +
+ (ig ~ jg), which is impossible since this polynomial has degree less
than 1. Thuseach choice ofi,_;, ... ,iycorresponds to a different field element,
and since there are p” choices for this set of field integers, ¢ = p*.
This theorem has a number of interesting consequences, First, since every
Galois field has some prime number p as characteristic, every Galois field
must have g = p" elements for some prime p and some integer n. Next, if in
(6.6.4) we replace a. with an indeterminate r, we see that the set of elements of
the field can be regarded as the set of polynomials over GF(p) of degree at
most n — | with multiplication modulo (0). In other words, the field here,
after a relabelling of elements, is the same as the ficld of polynomials over
GF(p) modulo f(0) (that is, it has the same set of elements and the same
addition and multiplication table). Two such fields, differing only in the
labelling of the elements, are said to be isomorphic, and we see from this that
every field with p” elements is isomorphic to some field of polynomials over 230 Techniques for Coding and Decoding
GF(p) modulo an 1 degree irreducible polynomial. Finally, since (6.6.3)
gives the unique factorization of a polynomial D**" — 1 over GFip) into
irreducible factors, we see that every field with p” elements has the same set
of minimal polynomials. Thus, for any field with p” elements, we can choose
a as.a root of the fixed polynomial f(D) used in Theorem 6.6.4 and represent
all clements in the field as in (6.6.4). We thus have proven
‘Theorem 6.6.5. All Galois fields with p" clements are isomorphic to a
given field of polynomials over GF(p) modulo an r-degree irreducible poly-
nomial, a
Maximal Length Codes and Hamming Codes
Suppose that h(D) is the minimal polynomial of a primitive element 2 in
GF(p"). From Theorem 6.6.5, we see that h(D) is the minimal polynomial
of a primitive element in any representation of GF(p"), and such a poly-
nomial is called a primitive polynomial of degree m. A cyclic code of block
Figure 6.6.1. Encoder for maximal-length code.
length V =p" —1 for which a primitive polynomial A(D) is the check
polynomial is known as a maximal length code. The code words of such
code can be generated by the shift register circuit of Figure 6.5.4, redrawn
in Figure 6.6.1.
One code word for this code is the code word corresponding to the generator
polynomial g(D) = [D»"-* — 1JA(D). We now show that the set of code
words consists simply of the all-zero sequence and the set of cyclic shifts of
‘g(D). Since there are m information digits in the code, there are p" code
words. Since there are p” — I cyclic shifts of g(D) (including g(D) itself), all
we need show is that the cyclic shifts of g(D) are distinct. Suppose then that
the ith and jth cyclic shifts are identical, 0 —1, and thus (by Theorem 6.6.3), h(D) cannot divide D-‘ — 1. This
is a contradiction and all cyclic shifts of g(D) ate distinct.
Since each code word is determined by its m information digits, we see
from the above argument that each cyclic shift of g(.D) has a different initial
set of m digits, and that these make up all the p™ — I nonzero combinations
of m digits from GF(p). This can be interpreted pictorially by Figure 6.6.2
which shows the code word g(D) with its ends joined together in a ring so that
the cyclic shifts of g(D) can be read off the same ring starting at different
points,
Each consecutive sequence of m digits in the ring is the information sequence
of a different cyclic shift of g(D). Furthermore, it can be seen that the informa-
tion sequence for the ith cyclic shift of g(D) is simply the contents of the
Information sequence
for a(D)
Information sequence
for Da(D)
Figure 6.6.2. Digits of g(D) arranged in ring.
shift register in Figure 6.6.1 (with the highest clockwise term on the ring being
on the left of the register) after 7 shifts of the register in the generation of
g(D).
We notice now that out of the p" —1 nonzero m-sequences, exactly
p"* begin in each of the nonzero digits of GF(p) and p"-2 — 1 begin in 0.
Since each digit in the code word corresponding to g(D) is the beginning of
‘one of these m sequences, cach nonzero code word contains exactly p"-
appearances of each nonzero digit and p"-! — 1 appearances of 0. Since the
difference of any two code words is another code word, it follows that each
pair of different code words differs in all but p"-! — 1 positions. It can be
shown (see Problem 6.24) that this is the largest number of positions in which
all pairs of code words can differ for the given block length and thus these
codes are very effective for error correction.
If the shift register in Figure 6.6.1 generates an unterminating sequence of
digits instead of stopping after p” — 1 digits, then it can be seen that the 232 Techniques for Coding and Decoding
resulting sequence is periodic with period p” — 1 and successive digits in
the sequence are found by reading clockwise around the ring of Figure 6.6.2.
Since the future output of a feedback shift register depends only on its
present contents, it can be seen that its output must repeat periodically after
its contents are identical to its contents at some previous time. Ruling out
the zero sequence, there are only p” — 1 possible contents for an m stage
register and thus the output from every m stage feedback shift register with
elements in GF(p) is periodic with a period of at most p™ — 1. Since the use
ofa primitive polynomial for the feedback connections achieves this maximum.
period, these registers are called maximal-length feedback shift registers and
the codes are called maximal length codes.
The periodic sequences from a maximal-length feedback shift register are
called pseudo noise (or p — n) sequences. If we pick a sequence of i digits
(i
You might also like
