Informatii Teorie

Poor, H.V., Looney, C.G., Marks II, R.J., Verd, S., Thomas, J.A., Cover, T.M.
Information Theory
The Electrical Engineering Handbook
Ed. Richard C. Dorf
Boca Raton: CRC Press LLC, 2000
2000 by CRC Press LLC
73
InlormafIon Theory
73.1 Signal Detection
Geneial Consideiations Detection of Known Signals Detection of
Paiametiized Signals Detection of Random Signals Deciding
Among Multiple Signals Detection of Signals in Moie Geneial Noise
Piocesses Robust and Nonpaiametiic Detection Distiibuted and
Sequential Detection Detection with Continuous-Time
Measuiements
73.2 Noise
Statistics of Noise Noise Powei Effect of Lineai Tiansfoimations
on Autocoiielation and Powei Spectial Density White, Gaussian,
and Pink Noise Models Theimal Noise as Gaussian White Noise
Some Examples Measuiing Theimal Noise Effective Noise and
Antenna Noise Noise Factoi and Noise Ratio Equivalent Input
Noise Othei Electiical Noise Measuiement and Quantization
Noise Coping with Noise
73.3 Stochastic Piocesses
Intioduction to Random Vaiiables Stochastic Piocesses
Classifcations of Stochastic Piocesses Stationaiity of Piocesses
Gaussian and Maikov Piocesses Examples of Stochastic Piocesses
Lineai Filteiing of Weakly Stationaiy Piocesses Cioss-Coiielation
of Piocesses Coheience Eigodicity
73.4 The Sampling Theoiem
The Caidinal Seiies Pioof of the Sampling Theoiem The Time-
Bandwidth Pioduct Souices of Eiioi Geneializations of the
Sampling Theoiem
73.5 Channel Capacity
Infoimation Rates Communication Channels Reliable
Infoimation Tiansmission: Shannon`s Theoiem Bandwidth and
Capacity Channel Coding Theoiems
73.6 Data Compiession
Entiopy The Huffman Algoiithm Entiopy Rate Aiithmetic
Coding Lempel-Ziv Coding Rate Distoition Theoiy
Quantization and Vectoi Quantization Kolmogoiov Complexity
Data Compiession in Piactice
73.1 Signa! Detectiun
H. Vncenr Poor
The feld of signal detection and estimation is conceined with the piocessing of infoimation-beaiing signals
foi the puipose of extiacting the infoimation they contain. The applications of this methodology aie quite
bioad, ianging fiom aieas of electiical engineeiing such as automatic contiol, digital communications, image
piocessing, and iemote sensing, into othei engineeiing disciplines and the physical, biological, and social
sciences.
H. VIncenf oor
Prnceron Inverry
CarI C. Looney
Inverry of Nevodo
R. }. NarIs II
Inverry of Wo|ngron
SergIo Verd
Prnceron Inverry
}oy A. Thomas
IM
Thomas N. Cover
Sronford Inverry
Theie aie two basic types of pioblems of inteiest in this context. Sgna| Jeeton pioblems aie conceined
piimaiily with situations in which the infoimation to be extiacted fiom a signal is disciete in natuie. That is,
signal detection pioceduies aie techniques foi deciding among a disciete (usually fnite) numbei of possible
alteinatives. An example of such a pioblem is the demodulation of a digital communication signal, in which
the task of inteiest is to decide which of seveial possible tiansmitted symbols has elicited a given ieceived signal.
Esmaon pioblems, on the othei hand, deal with the deteimination of some numeiical quantity taking values
in a continuum. An example of an estimation pioblem is that of deteimining the phase oi fiequency of the
caiiiei undeilying a communication signal.
Although signal detection and estimation is an aiea of consideiable cuiient ieseaich activity, the fundamental
piinciples aie quite well developed. These piinciples, which aie based on the theoiy of statistical infeience,
explain and motivate most of the basic signal detection and estimation pioceduies used in piactice. In this
section, we will give a biief oveiview of the basic piinciples undeilying the feld of signal detection. Estimation
is tieated elsewheie this volume, notably in Section 16.2. A moie complete intioduction to these subjects is
found in Pooi 1994].
Genera! Cunsideratiuns
The basic piinciples of signal detection can be conveniently discussed in the context of decision-making between
two possible statistical models foi a set of ieal-valued measuiements, Y
1
, Y
2
, . . ., Y
n
. In paiticulai, on obseiving
Y
1
, Y
2
, . . . , Y
n
, we wish to decide whethei these measuiements aie most consistent with the model
Y
|
N
|
, | 1, 2, . . . , n (73.1)
oi with the model
Y
|
N
|
- S
|
, | 1, 2, . . . , n (73.2)
wheie N
1
, N
2
, . . . , N
n
is a iandom sequence iepiesenting noise, and wheie S
1
, S
2
, . . ., S
n
is a sequence
iepiesenting a (possibly iandom) signal.
In deciding between Eqs. (73.1) and (73.2), theie aie two types of eiiois possible: a [a|se a|arm, in which
(73.2) is falsely chosen, and a mss, in which (73.1) is falsely chosen. The piobabilities of these two types of
eiiois can be used as peifoimance indices in the optimization of iules foi deciding between (73.1) and (73.2).
Obviously, it is desiiable to minimize both of these piobabilities to the extent possible. Howevei, the minimi-
zation of the false-alarm probability and the minimization of the miss probability aie opposing ciiteiia. So,
it is necessaiy to effect a tiade-off between them in oidei to design a signal detection pioceduie. Theie aie
seveial ways of tiading off the piobabilities of miss and false alaim: the Bayesian detector minimizes an aveiage
of the two piobabilities taken with iespect to piioi piobabilities of the two conditions (73.1) and (73.2), the
mnmax detectoi minimizes the maximum of the two eiioi piobabilities, and the Neyman-Pearson detector
minimizes the miss piobability undei an uppei-bound constiaint on the false-alaim piobability.
If the statistics of noise and signal aie known, the Bayesian, minimax, and Neyman-Peaison detectois aie
all of the same foim. Namely, they ieduce the measuiements to a single numbei by computing the likelihood
ratio
(73.3)
wheie
S-N
and
N
denote the piobability density functions of the measuiements undei signal-plus-noise (73.2)
and noise-only (73.1) conditions, iespectively. The likelihood iatio is then compaied to a Jetson |res|o|J,
with the signal-piesent model (73.2) being chosen if the thieshold is exceeded, and the signal-absent model
(73.1) being chosen otheiwise. Choice of the decision thieshold deteimines a tiade-off of the two eiioi
piobabilities, and the optimum pioceduies foi the thiee ciiteiia mentioned above diffei only in this choice.
L Y Y Y
Y Y Y
Y Y Y
n
S N n
N n
( , , , )
( , , , )
( , , , )
1 2
1 2
1 2
. . .
. . .
. . .
A
+
Theie aie seveial basic signal detection stiuctuies that can be deiived fiom Eqs. (73.1) to (73.3) undei the
assumption that the noise sequence consists of a set of independent and identically distiibuted (i.i.d.) Gaussian
iandom vaiiables with zeio means. Such a sequence is known as discrete-time white Gaussian noise. Thus,
until fuithei notice, we will make this assumption about the noise. It should be noted that this assumption is
physically justifable in many applications.
Detectiun ul Knuvn Signa!s
If the signal sequence S
1
, S
2
, . . ., S
n
is known to be given by a specifc sequence, say s
1
, s
2
, . . ., s
n
(a situation
known as to|eren Jeeton), then the likelihood iatio (73.3) is given in the white Gaussian noise case by
(73.4)
wheie o
2
is the vaiiance of the noise samples. The only pait of (73.4) that depends on the measuiements is the
teim Z
n
k 1
s
k
Y
k
and the likelihood iatio is a monotonically incieasing function of this quantity. Thus, optimum
detection of a coheient signal can be accomplished via a coiielation detectoi, which opeiates by compaiing the
quantity
(73.5)
to a thieshold, announcing signal piesence when the thieshold is exceeded.
Note that this detectoi woiks on the piinciple that the signal will coiielate well with itself, yielding a laige
value of (73.5) when piesent, wheieas the iandom noise will tend to aveiage out in the sum (73.5), yielding a
ielatively small value when the signal is absent. This detectoi is illustiated in Fig. 73.1.
Detectiun ul Parametrized Signa!s
The coiielation detectoi cannot usually be used diiectly unless the signal is known exactly. If, alteinatively, the
signal is known up to a shoit vectoi of iandom paiameteis (such as fiequencies oi phases) that aie independent
of the noise, then an optimum test can be implemented by thieshold compaiison of the quantity
(73.6)
wheie we have wiitten S
|
s
|
() to indicate the functional dependence of the signal on the paiameteis, and
wheie A and p() denote the iange and piobability density function, iespectively, of the paiameteis.
The most impoitant example of such a paiametiized signal is that in which the signal is a modulated sinusoid
with iandom phase; i.e.,
FIGURE 73.1 Coiielation detectoi foi a coheient signal in additive white Gaussian noise.
exp / s Y s
| | |
|
n
|
n
_
,

1
2
2
1 1
2
o
s Y
| |
|
n
1
exp ( ) ( ) / ( ) s Y s J
| | |
|
n
|
n

,
_
,

[
1
2
2
1 1
2
o
A
S
|
a
|
cos(e
t
| - 0), | 1, 2, . . ., n (73.7)
wheie a
1
, a
2
, . . ., a
n
is a known amplitude modulation sequence, e
c
is a known (disciete-time) caiiiei fiequency,
and the iandom phase 0 is unifoimly distiibuted in the inteival -r,r]. In this case, the likelihood iatio is a
monotonically incieasing function of the quantity
(73.8)
Thus, optimum detection can be implemented via compaiison of (73.8) with a thieshold, a stiuctuie known
as an envelope detector. Note that this detectoi coiielates the measuiements with two oithogonal components
of the signal, a
|
cos(e
t
|) and a
|
sin(e
t
|). These two coiielations, known as the in-phase and quadiatuie
components of the measuiements, iespectively, captuie all of the eneigy in the signal, iegaidless of the value
of 0. Since 0 is unknown, howevei, these two coiielations cannot be combined coheiently, and thus they aie
combined noncoheiently via (73.8) befoie the iesult is compaied with a thieshold. This detectoi is illustiated
in Fig. 73.2.
Paiametiized signals also aiise in situations in which it is not appiopiiate to model the unknown paiameteis
as iandom vaiiables with a known distiibution. In such cases, it is not possible to compute the likelihood iatio
(73.6) so an alteinative to the likelihood iatio detectoi must then be used. (An exception is that in which the
likelihood iatio detectoi is invaiiant to the unknown paiameteis-a case known as un[orm|y mos ower[u|
Jeeton.) Seveial alteinatives to the likelihood iatio detectoi exist foi these cases.
One useful such pioceduie is to test foi the signal`s piesence by thieshold compaiison of the genera|:eJ
||e||ooJ rao, given by
(73.9)
wheie L
denotes the likelihood iatio foi Eqs. (73.1) and (73.2) foi the known-signal pioblem with the paiametei
vectoi fxed at . In the case of white Gaussian noise, we have
(73.10)
It should be noted that this foimulation is also valid if the statistics of the noise have unknown paiameteis,
e.g., the noise vaiiance in the white Gaussian case.
One common application in which the geneialized likelihood iatio detectoi is useful is that of detecting a
signal that is known except foi its time of aiiival. That is, we aie often inteiested in signals paiametiized as
s
|
(0) a
|-0
(73.11)
wheie {a
|
is a known fnite-duiation signal sequence and wheie 0 ianges ovei the integeis. Assuming white
Gaussian noise and an obseivation inteival much longei than the duiation of {a
|
, the geneialized likelihood
iatio detectoi in this case announces the piesence of the signal if the quantity
(73.12)
exceeds a fxed thieshold. This type of detectoi is known as a mat|eJ f|er, since it can be implemented by
flteiing the measuiements with a digital fltei whose pulse iesponse is a time-ieveised veision of the known
a | Y a | Y
| t |
|
n
| t |
|
n
cos( ) sin( ) u u

1
]
1
1
+

1
]
1
1
1
2
1
2
max ( , , , )
eA
L Y Y Y
n 1 2
. . .
L Y Y Y s Y s
n | | |
|
n
|
n
( , , ) exp ( ) ( ) /
1 2
2
1 1
2
1
2
. . . ,
,
_
,

o
max

a Y
| |
|

signal {a
|
(hence it is matched`` to the signal), and announcing the signal`s piesence if the fltei output exceeds
the decision thieshold at any time.
Detectiun ul Randum Signa!s
In some applications, paiticulaily in iemote sensing applications such as sonai and iadio astionomy, it is
appiopiiate to considei the signal sequence S
1
, S
2
, . . . , S
n
itself to be a iandom sequence, statistically independent
of the noise. In such cases, the likelihood iatio foimula of (73.6) is still valid with the paiametei vectoi simply
taken to be the signal itself. Howevei, foi long measuiement iecoids (i.e., laige n), (73.6) is not a veiy piactical
foimula except in some specifc cases, the most impoitant of which is the case in which the signal is Gaussian.
In paiticulai, if the signal is Gaussian with zeio-mean and autocoiielation sequence r
|, |
A
E{S
|
S
|
, then the
likelihood iatio is a monotonically incieasing function of the quantity
(73.13)
with q
|,|
the element in the |th iow and |th column of the positive-defnite matiix
(73.14)
wheie I denotes the n n identity matiix, and R is the covaiiance matiix of the signal, i.e., it is the n n
matiix with elements r
|,|
.
Note that (73.13) is a quadiatic function of the measuiements; thus, a detectoi based on the compaiison of
this quantity to a thieshold is known as a quadratic detector. The simplest foim of this detectoi iesults fiom
the situation in which the signal samples aie, like the noise samples, i.i.d. In this case, the quadiatic function
(73.13) ieduces to a positive constant multiple of the quantity
(73.15)
FIGURE 73.2 Envelope detectoi foi a noncoheient signal in additive white Gaussian noise.
q Y Y
| | | |
|
n
|
n
,

1 1

A
+

( / ) o
2 1
Y
|
|
n
2
1

A detectoi based on (73.15) simply measuies the eneigy in the measuiements and then announces the piesence
of the signal if this eneigy is laige enough. This type of detectoi is known as a raJomeer.
Thus, iadiometiy is optimum in the case in which both signal and noise aie i.i.d. Gaussian sequences with
zeio means. Since in this case the piesence of the signal is manifested only by an inciease in eneigy level, it is
intuitively obvious that iadiometiy is the only way of detecting the signal`s piesence. Moie geneially, when the
signal is coiielated, the quadiatic function (73.13) exploits both the incieased eneigy level and the coiielation
stiuctuie intioduced by the piesence of the signal. Foi example, if the signal is a naiiowband Gaussian piocess,
then the quadiatic function (73.13) acts as a naiiowband iadiometei with bandpass chaiacteiistic that appiox-
imately matches that of the signal. In geneial, the quadiatic detectoi will make use of whatevei spectial
piopeities the signal exhibits.
If the signal is iandom but not Gaussian, then its optimum detection desciibed by (73.6)] typically iequiies
moie complicated nonlineai piocessing than the quadiatic piocessing of (73.13) in oidei to exploit the distii-
butional diffeiences between signal and noise. This type of piocessing is often not piactical foi implementation,
and thus appioximations to the optimum detectoi aie typically used. An inteiesting family of such detectois
uses cubic oi quaitic functions of the measuiements, which exploit the highei-oidei spectial piopeities of the
signal Mendel, 1991]. As with deteiministic signals, iandom signals can be paiametiized. In this case, howevei,
it is the distiibution of the signal that is paiametiized. Foi example, the powei spectium of the signal of inteiest
may be known only up to a set of unknown paiameteis. Geneialized likelihood iatio detectois (73.9) aie often
used to detect such signals.
Deciding Amung Mu!tip!e Signa!s
The pieceding iesults have been developed undei the model (73.1)-(73.2) that theie is a single signal that is
eithei piesent oi absent. In digital communications applications, it is moie common to have the situation in
which we wish to decide between the piesence of two (oi moie) possible signals in a given set of measuiements.
The foiegoing iesults can be adapted stiaightfoiwaidly to such pioblems. This can be seen most easily in the
case of deciding among known signals. In paiticulai, considei the pioblem of deciding between two alteinatives:
(73.16)
and
(73.17)
wheie s
1
(0)
, s
2
(0)
, . . . , s
n
(0)
and s
1
(1)
, s
2
(1)
, . . ., s
n
(1)
aie two known signals. Such pioblems aiise in data tiansmission
pioblems, in which the two signals s
(0)
and s
(1)
coiiespond to the wavefoims ieceived aftei tiansmission of a
logical zeio`` and one,`` iespectively. In such pioblems, we aie geneially inteiested in minimizing the aerage
ro|a||y o[ error, which is the aveiage of the two eiioi piobabilities weighted by the piioi piobabilities of
occuiience of the two signals. This is a Bayesian peifoimance ciiteiion, and the optimum decision iule is a
stiaightfoiwaid extension of the coiielation detectoi based on (73.5). In paiticulai, undei the assumptions that
the two signals aie equally likely to occui piioi to measuiement, and that the noise is white and Gaussian, the
optimum decision between (73.16) and (73.17) is to choose the model (73.16) if l
|
n
1
s
|
(0)
Y
|
is laigei than l
|
n
1
s
|
(1)
Y
|
, and to choose the model (73.17) otheiwise.
Moie geneially, many pioblems in digital communications involve deciding among M equally likely signals
with M > 2. In this case, again assuming white Gaussian noise, the decision iule that minimizes the eiioi
piobability is to choose the signal s
1
( ,)
, s
2
( ,)
, . . ., s
n
( ,)
, wheie , is a solution of the maximization pioblem
(73.18)
Y N s | n
| | |
+
( )
, , , . . . ,
0
1 2
Y N s | n
| | |
+
( )
, , , . . . ,
1
1 2
s Y s Y
|
,
|
|
n
m M
|
m
|
|
n
( ) ( )
max
s s

1
0 1
1
Theie aie two basic types of digital communications applications in which the pioblem (73.18) aiises. One
is in M-ary Jaa ransmsson, in which a symbol alphabet with M elements is used to tiansmit data, and a
decision among these M symbols must be made in each symbol inteival Pioakis, 1983]. The othei type of
application in which (73.18) aiises is that in which data symbols aie coiielated in some way because of
inteisymbol inteifeience, coding, oi multiusei tiansmission. In such cases, each of the M possible signals
iepiesents a fiame of data symbols, and a joint decision must be made about the entiie fiame since individual
symbol decisions cannot be decoupled. Within this lattei fiamewoik, the pioblem (73.18) is known as sequente
Jeeton. The basic distinction between M-aiy tiansmission and sequence detection is one of degiee. In typical
M-aiy tiansmission, the numbei of elements in the signaling alphabet is typically a small powei of 2 (say 8 oi
32), wheieas the numbei of symbols in a fiame of data could be on the oidei of thousands. Thus, solution of
(73.18) by exhaustive seaich is piohibitive foi sequence detection, and less complex algoiithms must be used.
Typical digital communications applications in which sequence detection is necessaiy admit dynamic piogiam-
ming solutions to (73.18) (see, e.g., Veid 1993]).
Detectiun ul Signa!s in Mure Genera! Nuise Prucesses
In the foiegoing paiagiaphs, we have desciibed thiee basic detection pioceduies: coiielation detection of signals
that aie completely known, envelope detection of signals that aie known except foi a iandom phase, and
quadiatic detection foi Gaussian iandom signals. These detectois weie all deiived undei an assumption of
white Gaussian noise. This assumption piovides an accuiate model foi the dominant noise aiising in many
communication channels. Foi example, the theimal noise geneiated in signal piocessing electionics is ade-
quately desciibed as being white and Gaussian. Howevei, theie aie also many channels in which the statistical
behavioi of the noise is not well desciibed in this way, paiticulaily when the dominant noise is pioduced in
the physical channel iathei than in the ieceivei electionics.
One type of noise that often aiises is noise that is Gaussian but not white. In this case, the detection pioblem
(73.1)-(73.2) can be conveited to an equivalent pioblem with white noise by applying a lineai flteiing piocess
known as rew|enng to the measuiements. In paiticulai, on denoting the noise covaiiance matiix by l, we
can wiite
l CC
T
(73.19)
wheie C is an n n inveitible, lowei-tiiangulai matiix and wheie the supeisciipt T denotes matiix tianspo-
sition. The iepiesentation (73.19) is known as the C|o|es|y Jetomoson. On multiplying the measuiement
vectoi Y
A
(Y
1
, Y
2
, . . . , Y
n
)
T
satisfying (73.1)-(73.2) with noise covaiiance l, by C
-1
, we pioduce an equivalent
(in teims of infoimation content) measuiement vectoi that satistifes the model (73.1)-(73.2) with white
Gaussian noise and with the signal confoimally tiansfoimed. This model can then be tieated using the methods
desciibed pieviously.
In othei channels, the noise can be modeled as being i.i.d. but with an amplitude distiibution that is not
Gaussian. This type of model aiises, foi example, in channels dominated by impulsive phenomena, such as
uiban iadio channels. In the non-Gaussian case the pioceduies discussed pieviously lose theii optimality as
defned in teims of the eiioi piobabilities. These pioceduies can still be used, and they will woik well undei
many conditions; howevei, theie will be a iesulting peifoimance penalty with iespect to optimum pioceduies
based on the likelihood iatio. Geneially speaking, likelihood-iatio-based pioceduies foi non-Gaussian noise
channels involve moie complex nonlineai piocessing of the measuiements than is iequiied in the standaid
detectois, although the ietention of the i.i.d. assumption gieatly simplifes this pioblem. A tieatment of methods
foi such channels can be found in Kassam 1988].
When the noise is both non-Gaussian and dependent, the methodology is less well developed, although some
techniques aie available in these cases. An oveiview can be found in Pooi and Thomas 1993].
Rubust and Nunparametric Detectiun
All of the pioceduies outlined in the pieceding subsection aie based on the assumption of a known (possibly
up to a set of unknown paiameteis) statistical model foi signals and noise. In many piactical situations it is
not possible to specify accuiate statistical models foi signals oi noise, and so it is of inteiest to design detection
pioceduies that do not iely heavily on such models. Of couise, the paiametiized models desciibed in the
foiegoing paiagiaphs allow foi unceitainty in the statistics of the obseivations. Such models aie known as
aramert models, because the set of possible distiibutions can be paiametiized by a fnite set of ieal paiameteis.
While paiametiic models can be used to desciibe many types of modeling unceitainty, composite models
in which the set of possible distiibutions is much bioadei than a paiametiic model would allow aie sometimed
moie iealistic in piactice. Such models aie teimed nonaramert moJe|s. Foi example, one might be able to
assume only some veiy coaise model foi the noise, such as that it is symmetiically distiibuted. A wide vaiiety
of useful and poweiful detectois have been developed foi signal-detection pioblems that cannot be paiame-
tiized. These aie basically of two types: ro|us and nonaramert. Robust detectois aie those designed to
peifoim well despite small, but potentially damaging, nonpaiametiic deviations fiom a nominal paiametiic
model, wheieas nonpaiametiic detectois aie designed to achieve constant false-alaim piobability ovei veiy
wide classes of noise statistics.
Robustness pioblems aie usually tieated analytically via minimax foimulations that seek best woist-case
peifoimance as the design objective. This foimulation has pioven to be veiy useful in the design and chaiac-
teiization of iobust detectois foi a wide vaiiety of detection pioblems. Solutions typically call foi the intio-
duction of light limiting to pievent extiemes of gain dictated by an (uniealistic) nominal model. Foi example,
the coiielation detectoi of Fig. 73.1 can be made iobust against deviations fiom the Gaussian noise model by
intioducing a soft-limitei between the multipliei and the accumulatoi.
Nonpaiametiic detection is usually based on ielatively coaise infoimation about the obseivations, such as
the algebiaic signs oi the ianks of the obseivations. One such test is the sgn es, which bases its decisions on
the numbei of positive obseivations obtained. This test is nonpaiametiic foi the model in which the noise
samples aie i.i.d. with zeio median and is ieasonably poweiful against alteinatives such as the piesence of a
positive constant signal in such noise. Moie poweiful tests foi such pioblems can be achieved at the expense
of complexity by incoipoiating iank infoimation into the test statistic.
Distributed and Sequentia! Detectiun
The detection pioceduies discussed in the pieceding paiagiaphs aie based on the assumption that all measuie-
ments can and should be used in the detection of the signal, and moieovei that no constiaints exist on how
measuiements can be combined. Theie aie a numbei of applications, howevei, in which constiaints apply to
the infoimation pattein of the measuiements.
One type of constiained infoimation pattein that is of inteiest in a numbei of applications is a netwoik consisting
of a numbei of distiibuted oi local decision makeis, each of which piocesses a subset of the measuiements, and
a [uson tener, which combines the outputs of the distiibuted decision makeis to pioduce a global detection
decision. The communication between the distiibuted decision makeis and the fusion centei is constiained, so
that each local decision makei must ieduce its subset of measuiements to a summaiizing local decision to be
tiansmitted to the fusion centei. Such stiuctuies aiise in applications such as the testing of laige-scale integiated
ciicuits, in which data collection is decentialized, oi in detection pioblems involving veiy laige data sets, in which
it is desiiable to distiibute the computational woik of the detection algoiithm. Such pioblems lie in the feld of
Jsr|ueJ Jeeton. Except in some tiivial special cases, the constiaints imposed by distiibuting the detection
algoiithm intioduce a fuithei level of diffculty into the design of optimum detection systems. Neveitheless,
consideiable piogiess has been made on this pioblem, a suivey of which can be found in Tsitsiklis 1993].
Anothei type of nonstandaid infoimation pattein that aiises is that in which the numbei of measuiements
is potentially infnite, but in which theie is a cost associated with taking each measuiement. This type of model
aiises in applications such as the synchionization of wideband communication signals. In such situations, the
eiioi piobabilities alone do not completely chaiacteiize the peifoimance of a detection system, since
consideiation must also be given to the cost of sampling. The feld of sequena| Jeeton deals with the
optimization of detection systems within such constiaints. In sequential detectois, the numbei of measuiements
taken becomes a iandom vaiiable depending on the measuiements themselves. A typical peifoimance ciiteiion
foi optimizing such a system is to seek a detectoi that minimizes the expected numbei of measuiements foi
given levels of miss and false-alaim piobabilities.
The most commonly used sequential detection pioceduie is the sequena| ro|a||y rao es, which opeiates
by iecuisive compaiison of the likelihood iatio (73.3) to two thiesholds. In this detectoi, if the likelihood iatio
foi a given numbei of samples exceeds the laigei of the two thiesholds, then the signal`s piesence is announced
and the test teiminates. Alteinatively, if the likelihood iatio falls below the smallei of the two thiesholds, the
signal`s absence is announced and the test teiminates. Howevei, if neithei of the two thiesholds is ciossed, then
anothei measuiement is taken and the test is iepeated.
Detectiun vith Cuntinuuus-Time Measurements
Note that all of the pieceding foimulations have involved the assumption of disciete-time (i.e., sampled-data)
measuiements. Fiom a piactical point of view, this is the most natuial fiamewoik within which to considei
these pioblems, since implementations most often involve digital haidwaie. Howevei, the pioceduies discussed
in this section all have continuous-time counteipaits, which aie of both theoietical and piactical inteiest.
Mathematically, continuous-time detection pioblems aie moie diffcult than disciete-time ones, because they
involve piobabilistic analysis on function spaces. The theoiy of such pioblems is quite elegant, and the inteiested
ieadei is iefeiied to Pooi 1994] oi Gienandei 1981] foi moie detailed exposition.
Continuous-time models aie of piimaiy inteiest in the fiont-end stages of iadio fiequency oi optical
communication ieceiveis. At iadio fiequencies, continuous-time veisions of the models desciibed in the
pieceding paiagiaphs can be used. Foi example, one may considei the detection of signals in continuous-time
Gaussian white noise. At optical wavelengths, one may considei eithei continuous models (such as Gaussian
piocesses) oi point-piocess models (such as Poisson counting piocesses), depending on the type of detection
used (see, e.g., Snydei and Millei 1991]). In the most fundamental analyses of optical detection pioblems, it
is sometimes desiiable to considei the quantum mechanical natuie of the measuiements Helstiom, 1976].
Dehning Terms
Bayesian detector: A detectoi that minimizes the aveiage of the false-alaim and miss piobabilities, weighted
with iespect to piioi piobabilities of signal-absent and signal-piesent conditions.
Correlation detector: The optimum stiuctuie foi detecting coheient signals in the piesence of additive white
Gaussian noise.
Discrete-time white Gaussian noise: Noise samples modeled as independent and identically distiibuted
Gaussian iandom vaiiables.
Envelope detector: The optimum stiuctuie foi detecting a modulated sinusoid with iandom phase in the
piesence of additive white Gaussian noise.
False-alarm probability: The piobability of falsely announcing the piesence of a signal.
Likelihood ratio: The optimum piocessoi foi ieducing a set of signal-detection measuiements to a single
numbei foi subsequent thieshold compaiison.
Miss probability: The piobability of falsely announcing the absence of a signal.
Neyman-Pearson detector: A detectoi that minimizes the miss piobability within an uppei-bound constiaint
on the false-alaim piobability.
Quadratic detector: A detectoi that makes use of the second-oidei statistical stiuctuie (e.g., the spectial
chaiacteiistics) of the measuiements. The optimum stiuctuie foi detecting a zeio-mean Gaussian signal
in the piesence of additive Gaussian noise is of this foim.
Re!ated Tupics
16.2 Paiametei Estimation 70.3 Spiead Spectium Communications
Relerences
U. Gienandei, |srat In[erente, New Yoik: Wiley, 1981.
C.W. Helstiom, Quanum Deeton anJ Esmaon T|eory, New Yoik: Academic Piess, 1976.
S.A. Kassam, Sgna| Deeton n Non-Caussan Nose, New Yoik: Spiingei-Veilag, 1988.
J.M. Mendel, Tutoiial on highei-oidei statistics (spectia) in signal piocessing and systems theoiy: Theoietical
iesults and some applications,`` Prot. IEEE, vol. 79, pp. 278-305, 1991.
H.V. Pooi, n InroJuton o Sgna| Deeton anJ Esmaon, 2nd ed., New Yoik: Spiingei-Veilag, 1994.
H.V. Pooi and J. B. Thomas, Signal detection in dependent non-Gaussian noise,`` in Jantes n Sasta| Sgna|
Protessng, vol. 2, Signal Detection, H.V. Pooi and J.B. Thomas, Eds., Gieenwich, Conn.: JAI Piess, 1993.
J.G. Pioakis, Dga| Communtaons, New Yoik: McGiaw-Hill, 1983.
D.L. Snydei and M.I. Millei, RanJom Pon Protesses n Tme anJ Sate, New Yoik: Spiingei-Veilag, 1991.
J. Tsitsiklis, Distiibuted detection,`` in Jantes n Sasta| Sgna| Protessng, vol. 2, Signal Detection, H.V.
Pooi and J.B. Thomas, Eds., Gieenwich, Conn.: JAI Piess, 1993.
S. Veid, Multiusei detection,`` in Jantes n Sasta| Sgna| Protessng, vol. 2, Signal Detection, H.V. Pooi
and J.B. Thomas, Eds., Gieenwich, Conn.: JAI Piess, 1993.
Further Inlurmatiun
Except as otheiwise noted in the accompanying text, fuithei details on the topics intioduced in this section
can be found in the textbook:
Pooi, H.V. n InroJuton o Sgna| Deeton anJ Esmaon, 2nd ed., New Yoik: Spiingei-Veilag, 1994.
The bimonthly jouinal, IEEE Transatons on In[ormaon T|eory, publishes iecent advances in the theoiy of
signal detection. It is available fiom the Institute of Electiical and Electionics Engineeis, Inc., 345 East 47th
Stieet, New Yoik, NY 10017.
Papeis desciibing applications of signal detection aie published in a numbei of jouinals, including the
monthly jouinals IEEE Transatons on Communtaons, IEEE Transatons on Sgna| Protessng, and the Journa|
o[ |e tousta| Sotey o[ merta. The IEEE jouinals aie available fiom the IEEE, as above. The Journa| o[ |e
tousta| Sotey o[ merta is available fiom the Ameiican Institute of Physics, 335 East 45th Stieet, New Yoik,
NY 10017.
73.2 Nuise
Cor| C. Iooney
Eveiy infoimation signal s() is coiiupted to some extent by the supeiimposition of extia-signal uctuations
that assume unpiedictable values at each time instant . Such undesiiable signals weie called noise due to eaily
measuiements with sensitive audio amplifeis.
Noise souices aie (1) nrnst, (2) exerna|, oi (3) rotess nJuteJ. Intiinsic noise in conductois comes fiom
theimal agitation of moleculaily bound ions and elections, fiom micioboundaiies of impuiities and giains
with vaiying potential, and fiom tiansistoi junction aieas that become tempoiaiily depleted of elections/holes.
Exteinal electiomagnetic inteifeience souices include aiipoit iadai, x-iays, powei and telephone lines, com-
munications tiansmissions, gasoline engines and electiic motois, computeis and othei electionic devices; and
also include lightning, cosmic iays, plasmas (chaiged paiticles) in space, and solai/stellai iadiation (conductois
act as antennas). Reective objects and othei macioboundaiies cause multiple paths of signals. Piocess-induced
eiiois include measuiement, quantization, tiuncation, and signal geneiation eiiois. These also coiiupt the
signal with noise powei and loss of iesolution.
Statistics ul Nuise
Statistics allow us to analyze the spectia of noise. We model a noise signal by a random (oi sot|ast) process
N(), a function whose iealized value N() x
at any time instant is chosen by the outcome of the iandom

vaiiable N
N(). N() has a piobability distiibution foi the values x it can assume. Any paiticulai tiajectoiy
{(, x
) of outcomes is called a realization of the noise piocess. The frs-orJer sast of N() is the exeteJ
a|ue
EN()]. The setonJ-orJer sast is the auotorre|aon [unton R

NN
(, - t) EN()N( - t)],
wheie E-] is the expected value opeiatoi. Autocorrelation measuies the extent to which noise iandom vaiiables
N
1
N(
1
) and N
2
N(
2
) at times
1
and
2
depend on each othei in an aveiage sense.
When the fist- and second-oidei statistics do not change ovei time, we call the noise a weakly (oi wJe-
sense) stationary process. This means that: (1) EN()]
is constant foi all , and (2) R

NN
(, - t)
EN()N( - t)] EN(0)N(t)] R
NN
(t) foi all see Biown, 1983, p. 82; Gaidnei, 1990, p. 108; oi Peebles,
1987, p. 153 foi piopeities of R
NN
(t)]. In this case the autocoiielation function depends only on the o[[se r.
We assume heieaftei that 0 (we can subtiact , which does not change the autocoiielation). When t 0,
R
NN
(0) EN()N( - 0)] E(N())
2
] o
N
2
, which is the fxed vaiiance of each iandom vaiiable N
foi all .
Weakly stationaiy (ws) piocesses aie the most commonly encounteied cases and aie the ones consideied heie.
Eo|uonary piocesses have statistics that change ovei time and aie diffcult to analyze.
Figuie 73.3 shows a iealization of a noise piocess N(), wheie at any paiticulai time , the piobability density
function is shown coming out of the page in a thiid dimension. Foi a ws noise, the distiibutions aie the same
foi each . The most mathematically tiactable noises aie Caussan ws piocesses, wheie at each time the
piobability distiibution foi the iandom vaiiable N
N() is Gaussian (also called norma|). The fist- and

second-oidei statistics completely deteimine Gaussian distiibutions, and so ws makes theii statistics of all
oideis stationaiy ovei time also. It is well known see Biown, 1983, p. 39] that lineai tiansfoimations of Gaussian
iandom vaiiables aie also Gaussian iandom vaiiables. The piobability density function foi a Gaussian iandom
vaiiable N
is [
N
(x) {1/2ro
N
2
]
1/2
exp-(x -
N
)
2
/2o
N
2
], which is the familiai bell-shaped cuive centeied on
x
N
. The standaid Gaussian piobability table Peebles, 1987, p. 314] is useful, e.g., Pi-o
N
< N
< o
N
)
2Pi0 < N
< o
N
) 0.8413 fiom the table.
Nuise Puver
The noise signal N() iepiesents voltage, so the autocoiielation function at offset 0, R
NN
(0) EN()N()]
iepiesents expected powei in volts squaied, oi watts pei ohm. When R 1 O, then N()N() N()N()/R]
N()I() volt-ampeies watts (wheie I() is the cuiient in a 1-O iesistoi). The Fouiiei tiansfoim FR
NN
(r)] of
the autocoiielation function R
NN
(r) is the powei spectium, called the power spectral density function (psdf),
S
NN
(w) in W/(iad/s). Then
(73.20)
FIGURE 73.3 A noise piocess.
S w R e J R
R S w e Jw S w
NN NN
,ws
NN
NN NN
,ws
NN
( ) ( ) ( )]
( ) ( ) ( )]

~
~
~
~
[
[
t t t
t
r
1
2
1
The psdf at fiequency [ is defned to be the expected powei that the voltage N(), bandlimited to an inciemental
band J[ centeied at [, would dissipate in a 1-O iesistance, divided by J[.
Equations (73.20) aie known as the Vener-K|nt|n ielations that establish that S
NN
(w) and R
NN
(r) aie a
Fouiiei tiansfoim paii foi ws iandom piocesses Biown, 1983; Gaidnei, 1990, p. 230; Peebles, 1987]. The psdf
S
NN
(w) has units of W/(iad/s), wheieas the autocoiielation function R
NN
(r) has units of watts. When r 0 in
the second integial of Eq. (73.20), the exponential becomes e
0
1, so that R
NN
(0) ( EN()
2
] o
N
2
) is the
integial of the psdf S
NN
(w) ovei all iadian fiequencies, -~ < w < ~. The ims (ioot-mean-squaie) voltage is
N
ims
o
N
(the sanJarJ Jeaon). The powei spectium in W/(iad/s) is a density that is summed up via an
integial ovei the iadian fiequency band w
1
to w
2
to obtain the total powei ovei that band.
(73.21)
The vaiiance o
N
2
R
NN
(0) is the mean instantaneous powei P
NN
ovei all fiequencies at any time .
Ellect ul Linear Translurmatiuns un Autucurre!atiun
and Puver Spectra! Density
Let |() be the impulse iesponse function of a time-invaiiant lineai system L and H(w) F|()] be its tiansfei
function. Let an input noise signal N() have autocoiielation function R
NN
(r) and psdf S
NN
(w). We denote the
output noise signal by Y() LN()]. The Fouiiei tiansfoims Y(w) FY()] and N(w) FN()] do not
exist, but they aie not needed. The output Y() of a lineai system is ws whenevei the input N() is ws see
Gaidnei, 1990, p. 195; oi Peebles, 1987, p. 215]. The output psdf S
YY
(w) and autocoiielation function R
YY
(r)
aie given by, iespectively,
S
YY
(w) H(w)
2
S
NN
(w), R
YY
(r) F
-1
S
YY
(w)] (73.22)
see Gaidnei, 1990, p. 223]. The output noise powei is
(73.23)
White, Gaussian, and Pink Nuise Mude!s
V|e nose see Biown, 1983; Gaidnei, 1990, p. 234; oi Peebles, 1987] is a theoietical model V() of noise that
is ws with zeio mean. It has a constant powei level n
o
ovei all fiequencies (analogous to white light), so its psdf
is S
VV
(w) n
o
W/(iad/s), -~ < w < ~. The inveise Fouiiei tiansfoim of this is the impulse function R
VV
(r)
(n
o
)o(r), which is zeio foi all offsets except r 0. Theiefoie, white noise V() is a piocess that is uncoiielated
ovei time, i.e., EV(
1
)V(
2
)] 0 foi
1
not equal to
2
. Figuie 73.4(a) shows the autocoiielation and psdf foi
white noise wheie the offset is s r. A Caussan w|e nose is white noise such that the piobability distiibution
of each iandom vaiiable V
V() is Gaussian. When two Gaussian iandom vaiiables V

1
and V
2
aie untor-
re|aeJ, i.e., EV
1
V
2
] 0, they aie independent see Gaidnei, 1990, p. 37]. We use Gaussian models because
of the tenra| |m |eorem that states that the sum of a numbei of iandom vaiiables is appioximately Gaussian.
Actual ciicuits attenuate signals above cut-off fiequencies, and also the powei must be fnite. Howevei, foi
white noise, P
VV
R
NN
(0) ~, so we often tiuncate the white noise spectial density (psdf) at cut-offs -w
t
to
w
t
. The iesult is known as n| nose, P(), and is usually taken to be Gaussian because lineai flteiing of any
white noise (thiough the effect of the cential limit theoiem) tends to make the noise Gaussian see Gaidnei,
P w w S w Jw
P E N S w Jw
NN NN
w
w
NN N NN
( , ) ( )
( ) ] ( )
1 2
1
2
2 2
1
2
1
2

[
[
~
~
r
o
r
watts
watts
o
r r
Y YY YY NN
P S w Jw H w S w Jw
2 2
1
2
1
2

~
~
~
~
[ [
( ) ( ) ( )
1990, p. 241]. Figuie 73.4(b) shows the sinc function R
PP
(s) F
-1
S
PP
(w)] foi pink noise. Random vaiiables P
1
and P
2
at times
1
and
2
aie coiielated only foi
1
and
2
close.
Therma! Nuise as Gaussian White Nuise
Biown obseived in 1828 that pollen and dust paiticles moved iandomly when suspended in liquid. In 1906,
Einstein analyzed such motion based on the iandom walk model. Peiiin confimed in 1908 that the theimal
activity of molecules in a liquid caused iiiegulai bombaidment of the much laigei paiticles. It was piedicted
that chaiges bound to theimally vibiating molecules would geneiate electiomotive foice (emf) at the open
teiminals of a conductoi, and that this placed a limit on the sensitivity of galvanometeis. Theimal noise (also
called Jo|nson nose) was fist obseived by J. B. Johnson at Bell Laboiatoiies in 1927. Figuie 73.5 displays white
noise as seen in the laboiatoiy on an oscilloscope.
FIGURE 73.4 Powei tiansfoim paiis foi white and pink noise.
FIGURE 73.5 Theimal noise in a iesistoi.
lguro 73.5 not uvuubo
The voltage N() geneiated theimally between two points in an open ciicuit conductoi is the sum of an
extiemely laige numbei of supeiimposed, independent electionically and ionically induced miciovoltages at
all fiequencies up to [
t
6,000 GHz at ioom tempeiatuie see Gaidnei 1990, p. 235], neai infiaied. The mean
ielaxation time of fiee elections is 1/[
t
0.5 10
-10
/T s, so at ioom tempeiatuie of T 290K, it is 0.17 ps
(1 picosecond 10
-12
s). The values of N() at diffeient times aie uncoiielated foi time diffeiences (offsets)
gieatei than r
t
1/[
t
. The expected value of N() is zeio. The powei is faiily constant acioss a bioad spectium,
and we cannot sample signals at picosecond peiiods, so we model Johnson noise N() with Gaussian white
noise V(). Although EV()] 0, the aveiage powei is positive at tempeiatuies above 0K, and is o
V
2
R
VV
(0) see the iight side of Eq. (73.21)]. A disadvantage of the white noise model is its infnite powei, i.e.,
R
VV
(0) o
V
2
~, but it is valid ovei a limited bandwidth of B Hz, in which case its powei is fnite.
In 1927, Nyquist 1928] theoietically deiived theimal noise powei in a iesistoi to be
P
VV
(B) 4|TRB (watts) (73.24)
wheie R is iesistance (ohms), B is the fiequency bandwidth of measuiement in Hz (all emf uctuations outside
of B aie ignoied), P
VV
(B) is the mean powei ovei B (see Eq. 73.21), and Boltzmann`s constant is | 1.38
10
-23
J/K see Ott, 1988; Gaidnei, 1990, p. 288; oi Peebles, 1987, p. 227]. Undei exteinal emf, the theimally
induced collisions aie the main souice of iesistance in conductois (elections pulled into motion by an exteinal
emf at 0K meet no iesistance). The ims voltage is V
ims
o
V
(4|TRB)]
1/2
V ovei a bandwidth of B Hz.
Planck`s iadiation law is S
NN
(w) (2| [ )/exp(| [ /|T) - 1], wheie | 6.63 10
-34
J/s is Planck`s constant,
and [ is the fiequency see Gaidnei, 1990, p. 234]. Foi [ much smallei than |T/| 6.04 10
12
Hz - 6,000 GHz,
the exponential above can be appioximated by exp(h [ /kT) 1 - h [ /kT. The denominatoi of S
NN
(w) becomes
| [ /|T, so S
NN
(w) (2| [ )/(| [ /|T) 2|T W/Hz in a 1-O iesistoi. Ovei a iesistance of R O and a bandwidth
of B Hz (positive fiequencies), this yields the total powei P
VV
(B) 2BRS
NN
(w) 4|TRB W ovei the two-sided
fiequency spectium. This is Nyquist`s iesult.
Theimal noise is the same in a 1000-O caibon iesistoi as it is in a 1000-O tantalum thin-flm iesistoi see Ott,
1988]. While the intiinsic noise may nevei be less, it may be highei because of othei supeiimposed noise (desciibed
in latei sections). We model the theimal noise in a iesistoi by an inteinal souice (geneiatoi), as shown in Fig. 73.6.
Capacitance cannot be ignoied at high [, but puie ieactance (C oi L) cannot dissipate eneigy, and so cannot
geneiate theimal noise. The white noise model V() foi theimal noise N() has a constant psdf S
VV
(w) n
o
W/(iad/s) foi -~ < w < ~. By Eq. 73.21, the white noise mean powei ovei the fiequency bandwidth B is
(73.25)
FIGURE 73.6 Theimal noise in a iesistoi.
P B S w Jw n B n B
VV VV o o
B
B
( ) ( ) ( )
[
1
2
4 2 2
2
2
r
r r
r
r
/
Solving foi the constant n
o
, we obtain n
o
P
VV
(B)/2B, which we put into Eq. (73.20) to get the spectial density
as a function of tempeiatuie and iesistance using Nyquist`s iesult above.
S
VV
(w) n
o
P
VV
(B)/4rB 4|TR2rB/4rB 2|TR watts/(iad/s) (73.26)
Sume Examp!es
The paiasitic capacitance in the teiminals of a iesistoi may cause a ioll-off of about 20 dB/octave in actual
iesistois Biown, 1983, p. 139]. At 290K (ioom tempeiatuie), we have 2|T 2 1.38 10
-23
290 0.8
10
-20
W/Hz due to each ohm see Ott, 1988]. Foi R 1 MO (10
6
O), S
VV
(w) 0.8 10
-14
. Ovei a band
of 10
8
Hz, we have P
VV
(B) S
VV
(w)B 0.8 10
-14
10
8
0.8 10
-6
W 0.8 W by Eqs. (73.24) and
(73.26). In piactice, paiasitic capacitance causes theimal noise to be bandlimited (pink noise). Now considei
Fig. 73.6(b) and let the tempeiatuie be 300K, R 10
6
O, C 1 pf (1 picofaiad 10
-12
faiads), and assume L
is 0H. By Eq. (73.26), the theimal noise powei is
S
VV
(w) 2|TR 2 1.38 10
-23
300 10
6
828 10
-17
W/Hz
The powei acioss a bandwidth B 10
6
is P
VV
(B) S
VV
(w)B 8280 10
-12
W, so the ims voltage is V
ims

P
VV
(B)]
1/2
91 V.
Now let Y() be the output voltage acioss the capacitoi. The tiansfei function can be seen to be H(w)
{I(w)(1/,wC)/{I(w)R - (1/,wC)] (1/,wC)/R - 1/,wC] 1/1 - ,wRC] (wheie I(w) is the Fouiiei tiansfoim
of the cuiient). The output psdf see Eq. (73.22)] is
S
YY
(w) H(w)
2
S
VV
(w) (1/1 - w
2
R
2
C
2
])S
VV
(w)
Integiating S
YY
(w) (1/1 - w
2
R
2
C
2
])S
VV
(w) ovei all iadian fiequencies w 2r[ see Eq. (73.21)], we obtain
the antideiivative (828 10
-17
)(1/RC)atan(RCw)/2r. Upon substituting the limits w ~, this becomes 828
10
-17
r/2 - r/2]/2rRC 414 10
-17
(1/2RC) 207 10
-17
10
6
2070 10
-12
W/Hz. Then o
Y
2
EY()
2
]
P
YY
(-~,~) 2070 10
-12
W, so Y
ims
(t) o
Y
P
YY
(-~,~)]
1/2
45.5 V. The half-powei (cut-off) iadian
fiequency is w
t
1/RC 10
6
iad/s, oi [
t
w
t
/2r 159.2 kHz. Appioximating S
YY
(w) by the iectangulai spectium
S
YY
(w) n
o
, -10
6
< w < 10
6
iad/s (0 elsewheie), we have that R
YY
(r) (w
t
/r)sinc(w
t
r), which has the fist zeios
at w
t
r r, that is r 1/(2[
t
) see Fig. 73.4(b)]. We appioximate the autocoiielation by R
YY
(r) 0 foi s > 1/2[
t
.
Measuring Therma! Nuise
In Fig. 73.7, the theimal noise fiom a noisy iesistoi R is to be measuied, wheie R
L
is the measuiement load.
The inciemental noise powei in R ovei an inciemental fiequency band of width J[ is P
VV
(J [ ) 4|TRJ[ W,
by Eq. (73.24). P
YY
(J[ ) is the integial of S
YY
(w) ovei J[ by Eqs. (73.21), wheie S
YY
(w) H(w)
2
S
VV
(w), by Eq.
(73.22). In this case, the tiansfei function H(w) is nonieactive and does not depend upon the iadian fiequency
(we can factoi it out of the integial). Thus,
To maximize the powei measuied, let R
L
R. The ntremena| aa|a||e ower measuied is then P
YY
(J [ )
4|TR
2
J[ /(4R
2
) |TJ[ see Ott, 1988, p. 201; Gaidnei, 1990, p. 288; oi Peebles, 1987, p. 227]. Thus, we have
the iesult that inciemental available powei ovei bandwidth J[ depends only on the tempeiatuie T.
P
YY
(J[ ) |TJ[ (output powei ovei J[ ) (73.27)
Albeit Einstein used statistical mechanics in 1906 to postulate that the mean kinetic eneigy pei degiee of
fieedom of a paiticle, (1/2)mE
2
()], is equal to (1/2)|T, wheie m is the mass of the paiticle, () is its
P J[ H [ |TR J[ R R R |TRJ[
YY L L
J[
J[
( ) ( ) ( ) { ) ( ) +
[

2 2
2 4 /(
instantaneous velocity in a single dimension, | is Boltzmann`s constant, and T is the tempeiatuie in kelvin. A
shunt capacitoi C is chaiged by the theimal noise in the iesistoi see Fig. 73.6(b), wheie L is taken to be zeio].
The aveiage potential eneigy stoied is (1/2)CEV()
2
]. Equating this to 1/2|T and solving, we obtain the mean
squaie powei
EV()
2
] |T/C (73.28)
Foi example, let T 300K and C 50 pf, and iecall that | 1.38 10
-23
J/K. Then EV()
2
] |T/C 82.8
10
-12
, so that the input ims voltage is {EV()
2
]
1/2
9.09 V.
Ellective Nuise and Antenna Nuise
Let two seiies iesistois R
1
and R
2
have iespective tempeiatuies of T
1
and T
2
. The total noise powei ovei an
inciemental fiequency band J[ is P
Total
(J[ ) P
11
(J[ ) - P
22
(J[ ) 4|T
1
R
1
J[ - 4|T
2
R
2
J[ 4|(T
1
R
1
- T
2
R
2
)J[. By
putting
T
E
(T
1
R
1
- T
2
R
2
)/(R
1
- R
2
) (73.29)
we can wiite P
Total
(J[ ) 4|T
E
(R
1
- R
2
)J[. T
E
is called the e[[ete nose emeraure see Gaidnei, 1990, p. 289;
oi Peebles, 1987, p. 228]. An antenna ieceives noise fiom vaiious souices of electiomagnetic iadiation, such as
iadio tiansmissions and haimonics, switching equipment (such as computeis, electiical motoi contiolleis),
theimal (blackbody) iadiation of the atmospheie and othei mattei, solai iadiation, stellai iadiation, and galaxial
iadiation (the ambient noise of the univeise). To account foi noise at the antenna output, we model the noise
with an equivalent theimal noise using an effective noise tempeiatuie T
E
. The inciemental available powei
(output) ovei an inciemental fiequency band J[ is P
YY
(J[ ) |T
E
J[, fiom Eq. (73.27). T
E
is often called anenna
emeraure, denoted by T
. Although it vaiies with the fiequency band, it is usually viitually constant ovei a
small bandwidth.
Nuise Factur and Nuise Ratiu
In iefeience to Fig. 73.8(a), we defne the nose [ator F (nose ower ouu o[ atua| Jete)/(nose ower
ouu o[ Jea| Jete), wheie (noise powei output of ideal device) (powei output due to theimal noise souice).
The noise souice is taken to be a noisy iesistoi R at a tempeiatuie T, and all output noise measuiements must
be taken ovei a iesistive load R
L
(ieactance is ignoied). Letting P
VV
(B) 4|TRB be the open ciicuit theimal
noise powei of the souice iesistoi ovei a fiequency bandwidth B, and noting that the gain of the device is C,
the output powei due to the iesistive noise souice becomes C
2
P
VV
(B) 4|TRBC
2
/R
L
. Now let Y() be the
output voltage measuied at the output acioss R
L
. Then the noise factoi is
FIGURE 73.7 Measuiing theimal noise voltage.
F (P
YY
(B)/R
L
)/(C
2
P
VV
(B)/R
L
) (P
YY
(B))/(4|TRBC
2
) (73.30)
F is seen to be independent of R
L
, but not R. To compaie two noise factois, the same souice must be used. In
the ideal noiseless case, F 1, but as the noise level in the device incieases, F incieases. Because this is a powei
iatio, we may take the logaiithm, called the nose rao, which is
N
F
10 log
10
(F) 10 log
10
(P
YY
(B)) - 10 log
10
(4|TRBC
2
) (73.31)
The noise powei output P
YY
(B) of an actual device is a supeiposition of the amplifed souice theimal noise
C
2
P
VV
(B) and the device noise, i.e., P
YY
(B) C
2
P
VV
(B) - (device noise). The output noise acioss R
L
can be
measuied by putting a single fiequency (in the passband) souice geneiatoi S() as input. Fiist, S() is tuined
off, and the output ims voltage Y() is measuied and the output powei P
Y( V)
(B) is iecoided. This is the sum
of the theimal available powei and the device noise. Next, S() is tuined on and adjusted until the output powei
doubles, i.e., until the output powei P
Y(V)
(B) - P
Y(S)
(B) 2P
Y( V)
(B). This P
SS
(B) is iecoided. Solving foi
P
Y(S)
(B) P
Y( V)
(B), we substitute this in F P
Y( V)
(B)/(C
2
P
VV
(B)) to obtain
F P
Y( S)
(B)/(C
2
P
VV
(B)) (C
2
P
SS
(B))/(C
2
4|TRB) P
SS
(B)/4|TRB (73.32)
A bettei way is to input white noise V() in place of S() (a noise diode may be used). The disadvantages of
noise factois aie (1) when the device has low noise ielative to theimal noise, the noise factoi has value close
to 1; (2) a low iesistance causes high values; and (3) incieasing the souice iesistance decieases the noise factoi
while incieasing the total noise in the ciicuit Ott, 1988, p. 216]. Thus, accuiacy is not good. Foi cascaded
devices, the noise factois can be conveniently computed see Buckingham, 1985, p. 67; oi Ott, 1988, p. 228].
Equiva!ent Input Nuise
Shot noise (see below) and othei noise can be modeled by equivalent theimal noise that would be geneiated
in an input iesistoi by incieased tempeiatuie. Recall that the (maximum) inciemental available powei (output)
in a fiequency bandwidth J[ is P
VV
(J[ ) |TJ[ fiom Eq. (73.27). Figuie 73.8(b) piesents the situation. Let the
iesistoi be the noise souice at tempeiatuie T
o
with theimal noise V(). Then EV()
2
] 4|T
o
RJ[, by Eq. (73.24)
(Nyquist`s iesult). Let the open ciicuit output noise powei at R
L
be EY()
2
]. The inciemental available noise
powei P
YY
(J[ ) at the output (R
L
R) can be consideied to be due to the iesistoi R having a highei tempeiatuie
and an ideal (noiseless) device, usually an amplifei. We must fnd a tempeiatuie T
e
at which a pseudotheimal
FIGURE 73.8 Equivalent input noise and noise factoi.
noise powei EV
e
()
2
] 4|T
e
RJ[ yields the extia input" noise powei. Let V() V() - V
e
(). Then P
VV
(J[ )
4|T
o
RJ[ - 4|T
e
RJ[ 4|(T
o
- T
e
)RJ[ W, fiom Eq. (73.24). T
e
is called the equa|en nu nose emeraure. It
is ielated to the noise factoi F by T
e
290(F - 1). In cascaded amplifeis with gains C
1
, C
2
, . . . and equivalent
input noise tempeiatuies T
e1
, T
e2
, . . . , the total equivalent input noise tempeiatuie is
T
e (Total)
T
e1
- T
e2
/C
1
- T
e3
/C
1
C
2
- . . . (73.33)
see Gaidnei, 1990, p. 289].
Other E!ectrica! Nuise
Theimal noise and shot noise (which can be modeled by theimal noise with equivalent input noise) aie the
main noise souices. Othei noises aie discussed in the following paiagiaphs.
Shut Nuise
In a conductoi undei an exteinal emf, theie is an aveiage ow of elections, holes, photons, etc. In addition to
this induced net ow and theimal noise, theie is anothei effect. The potential diffeis acioss the boundaiies of
metallic giains and paiticles of impuiities, and when the kinetic eneigy of elections exceeds this potential,
elections jump acioss the baiiiei. This summed iandom ow is known as s|o nose see Gaidnei, 1990, p. 239;
Ott, 1988, p. 208]. The shot effect was analyzed by Schottky in 1918 as I
sh
(2qI
dc
B)
1/2
, wheie q 1.6 10
-19
coulombs pei election, I
dc
aveiage dc cuiient in ampeies, and B noise bandwidth (Hz).
Partitiun Nuise
Paron nose is caused by a paiting of the ow of elections to diffeient electiodes into stieams of iandomly
vaiying density. Suppose that elections fiom some souice S ow to destination electiodes and B. Let n()
and n(B) be the aveiage numbeis of elections pei second that go to nodes and B iespectively, so that n(S)
n() - n(B) is the aveiage total numbei of elections emitted pei second. It is a success when an election goes
to , and the piobability of success on a single tiial is , wheie
n()/n(S), 1 - n(B)/n(S) (73.34)
The cuiient to the iespective destinations is I() n()q, I(B) n(B)q, wheie q is the chaige of an election,
so that I()/I(S) and I(B)/I(S) 1 - . Using the binomial model, the aveiage numbeis of successes aie
En()] n(S) and En(B)] n(S)(1 - ). The vaiiance is Vai(n()) n(S)(1 - ) Vai(n(B)) (fiom the
binomial foimula foi vaiiance). Theiefoie, substitution yields
Vai(I()) q
2
n(S)(1 - )] q
2
n(S){I()I(B)/I() - I(B)] (73.35)
Paitition noise applies to pentodes, wheie the souice is the cathode, is the anode (success), and B is the
giid. Foi tiansistois, the souice is the emittei, is the collectoi, and B iepiesents iecombination in the base.
In photo devices, a photoelection is absoibed, and eithei an election is emitted (a success) oi not. Even a
paitially silveied miiioi can be consideied to be a paititionei: the passing of a photon is a success and ieection
is a failuie. While the binomial model applies to paititions with destinations and B, multinomial models aie
analogous foi moie than two destinations.
F!icker, Cuntact, and Burst Nuise
J.B. Johnson fist noticed in 1925 that noise acioss theimionic gates exceeded the expected shot noise at lowei
fiequencies. It is most noticeable up to about 2 kHz. The psdf of the extia noise, called [t|er nose, is
S( [ ) I
2
/o[, [ > 0 (73.36)
wheie I is the dc cuiient owing thiough the device and [ is the positive fiequency. Empiiical values of o aie
about 1 to 1.6 foi diffeient souices. These souices vaiy but include the iiiegulaiity of the size of macio iegions
of the cathode suiface, impuiities in the conducting channel, and geneiation and iecombination noise in
tiansistois. In the eaily days of tiansistois, this geneiation-iecombination was of gieat concein because the
mateiials weie not of high puiity. Flickei noise occuis in thin layeis of metallic oi semiconducting mateiial,
solid state devices, caibon iesistois, and vacuum tubes see Buckingham, 1985, p. 143]. It includes tonat nose
because it is caused by uctuating conductivity due to impeifect contact between two suifaces, especially in
switches and ielays. Flickei noise may be high at low fiequencies.
Burs nose is also called otorn nose. audio amplifeis sound like popcoin popping in a fiying pan
backgiound (theimal noise). Its chaiacteiistic is 1/[
n
(usually n 2), so its powei density falls off iapidly, wheie
[ is fiequency. It may be pioblematic at low fiequencies. The cause is manufactuiing defects in the junction of
tiansistois (usually a metallic impuiity).
Barkhuusen and Other Nuise
Bar||ousen nose is due to the vaiiations in size and oiientation of small iegions of feiiomagnetic mateiial and
is especially noticeable in the steeply iising iegion of the hysteiesis loop. Theie is also secondaiy emission,
photo and collision ionization, etc.
Measurement and Quantizatiun Nuise
Measurement Errur
The measuiement X
of a signal X() at any iesults in a measuied value X
x that contains eiioi, and so is

not equal to the tiue value X
x
T
. The piobability is highei that the magnitude of e (x - x
T
) is closei to
zeio. The bell-shaped Gaussian piobability density [ (e) 1/(2ro
2
]
1/2
exp(-e
2
/2ro) fts the eiioi well. This
noise piocess is stationaiy ovei time. The expected value is
e
0, the mean-squaie eiioi is o
e
2
, and the ims
eiioi is o
e
. Its instantaneous powei at time is o
e
2
. To see this, the eiioi signal e() (x - x
T
) has instantaneous
powei pei O of
P
e()() e()e()/R] e
2
() (73.37)
wheie R 1 O and () is the cuiient. The aveiage powei is the summed instantaneous powei ovei a peiiod
of time T, divided by the time, taken in the limit as T ~, i.e.,
This aveiage powei can be deteimined by sampling on known signal values and then computing the sample
vaiiance (assuming eigodicity: see Gaidnei 1990, p. 163]). The eiioi and signal aie piobabilistically independent
(unless the eiioi depends on the values of X). The signal-to-noise powei iatio is computed by S/N P
signal
/P
ave
.
Quantizatiun Nuise
Quantization noise is due to the digitization of an exact signal value
() captuied at sampling time by

an A/D conveitei. The binaiy iepiesentation is |
n-1
|
n-2
. . . |
1
|
0
(an n-bit woid). The n-bit digitization has 2
n
diffeient values possible, fiom 0 to 2
n
-1. Let the voltage iange be R. The reso|uon is J R/2
n
. Any voltage
is coded into the neaiest lowei binaiy value x

|
, wheie the eiioi e x
- x
|
satisfes 0 s e s J. Thus, the eiiois
e aie distiibuted ovei the inteival 0, J] in an equally likely fashion that implies the unifoim distiibution on
0, J]. The expected value of e e
e() at any time is

e
J/2, and the vaiiance is
e
2
J
2
/12 (the vaiiance
of a unifoim distiibution on an inteival a,|] is o (| - a)
2
/12). Thus the noise is ws and the powei of
quantization noise is
(73.38)
P T e J
T
T
ave

~
[
lim( / ) ( ) 1
2
0
o
e
J
J
e J J Je
e J J J J J J
2 2
0
3
0
3 3 2
2 1
2 3 24 12

+
[
( ) ( / )
( ) ( ) ( ) ]/
/
/ / /
We can fnd the signal-to-noise voltage iatio foi the total iange R via R/(J/(12)
1/2
) 2
n
J/(J/(12)
1/2
)
2
n
(12)
1/2
. The powei iatio is the squaie of this, which is (2
2n
)(12). In decibels this becomes (S/N)
dB

10 log
10
(2
2n
12) 10 log
10
(12) - 20n log
10
(2) 10.8 - 6.02n. Thus, quantization S/N powei iatio depends
diiectly upon the numbei of bits n in that the highei S/N powei iatio is bettei, just as we would have expected.
Cuping vith Nuise
Exteinal inteifeience is ubiquitous. Intiinsic noise is piesent up to the inciemental available powei at tempei-
atuies above absolute zeio, and othei intiinsic noises depend on mateiial puiity and connection integiity.
Piocessing eiioi is always intioduced in some foim.
Externa! Suurces
Standaid defenses aie (1) shielding of lines and ciicuits, (2) twisted wiie paiis oi coaxial cables, (3) shoit lines
and leads, (4) digital iegeneiation at waypoints of digital signals, (5) naiiowband signals, (6) coiielation of
ieceived signals with multipaths, and (7) adaptive notch flteiing to eliminate inteifeience at known fiequencies;
e.g., the second haimonic of 60-Hz ac powei lines may inteifeie with biological miciovoltage measuiements
but could be eliminated via adaptive notch flteiing. Feiiite beads can dampen inteifeience Baines, 1987].
Digital signal piocessing, spectial shaping flteis see Biown, 1983], and fiequency-shift flteis see Gaidnei,
1990, p. 400] can be used to lowei noise powei. Kalman flteiing is a poweiful estimation method, and fiequency-
shift flteiing is a newei technique foi disciiminating against both measuiement eiioi (e.g., in system identi-
fcation applications) and extiinsic souices of both noise and inteifeience Gaidnei, 1990, p. 400].
Intrinsic Suurces
Stiategies foi minimizing intiinsic noise aie (a) small bandwidth B, (b) small iesistances R, (c) low tempeiatuie
T (highei tempeiatuies can be devastating), (d) low voltage and cuiients (CMOS tiansistois), (e) modein
mateiials of high puiity, (f) wiapped wiie iesistois (theimal noise is the same, but othei noise will be less),
(g) fewei and bettei connections (of gold), (h) smallei ciicuits of lowei powei, and (i) shunt capacitois to
ieduce noise bandwidth. Gieatei puiity of integiated ciicuit mateiials nowadays essentially ieduces intiinsic
noise to theimal noise. Bettei design and mateiials aie the keys to lowei noise.
Prucessing Suurces
Piocessing eiiois can be ieduced by using highei iesolution of analog-to-digital conveiteis, i.e., moie bits to
iepiesent each value. This loweis the quantization eiioi powei. Measuiement eiioi can be ieduced while using
the same instiuments by taking multiple measuiements and aveiaging. Othei estimation/coiielation can yield
bettei values (e.g., the Global Positioning System location deteimination can be ieduced fiom meteis to a few
centimeteis by multiple measuiement estimation).
Dehning Terms
Autocorrelation: A function associated with a iandom signal X() that is defned on paiis of time instants
1
and
2
and whose value is the expected value of the pioduct of the iandom vaiiables X(
1
) and X(
2
), i.e.,
R
XX
(
1
,
2
) EX(
1
)X(
2
)]. Foi weakly stationaiy iandom signals, it depends only on the offset r
2
-
1
,
so we wiite R
XX
(r) EX()X( - r)].
Noise: A signal N() whose value at any time is iandomly selected by events beyond oui contiol. At any
time instant , N() is a iandom vaiiable N
with a piobability distiibution that deteimines the ielative

fiequencies at which N
assumes values. The statistics of the family of iandom vaiiables {N
may be
constant (stationaiy) ovei time (the usual case) oi may vaiy.
Power spectral density: The Fouiiei tiansfoim of the powei X
2
() does not necessaiily exist, but it does foi
X
T
2
()/2T (X
T
() 0 foi > T, X() elsewheie), foi any T > 0. Letting T~, the expected value of
the Fouiiei tiansfoims EFX
T
2
()/2T] FEX
T
2
()]/2T goes to the limit of the aveiage powei in X()
ovei - T to T, known as the powei spectial density function S
xx (w)
. Summed up ovei all fiequencies, it
gives the total powei in the signal X().
Random process: (signal): A signal that is eithei a noise, an inteifeiing signal s(), oi a sum of these such as
X() s
1
() - . . . - s
m
() - N
1
() - . . . - N
n
().
Realization: A tiajectoiy {(, x
): X() x
deteimined by the actual outcomes {x
of values fiom a iandom

signal X(), wheie X() x
at each instant . A tiajectoiy is also called a sam|e [unton of X().

Weakly stationary (ws) random process (signal): A iandom signal whose fist- and second-oidei statistics
iemain stationaiy (fxed) ovei time.
Re!ated Tupic
15.2 Speech Enhancement and Noise Reduction
Relerences
J. R. Baines, E|etront Sysem Desgn. Iner[erente anJ Nose Conro|, Englewood Cliffs, N.J.: Pientice-Hall, 1987.
R. G. Biown, InroJuton o RanJom Sgna| na|yss anJ Ka|man F|erng, New Yoik: Wiley, 1983.
M. J. Buckingham, Nose n E|etront Detes anJ Sysems, New Yoik: Halstead Piess, 1985.
W. A. Gaidnei, InroJuton o RanJom Protesses, 2nd ed., New Yoik: McGiaw-Hill, 1990.
J. B. Johnson, Theimal agitation of electiicity in conductois," P|ys. Re., vol. 29, pp. 367-368, 1927.
J. B. Johnson, Theimal agitation of electiicity in conductois," P|ys. Re., vol. 32, pp. 97-109, 1928.
H. Nyquist, Theimal agitation of electiic chaige in conductois," P|ys. Re., vol. 32, pp. 110-113, 1928.
H. W. Ott, Nose ReJuton Tet|nques n E|etront Sysems, 2nd ed., New Yoik: Wiley-Inteiscience, 1988.
P. Z. Peebles, Ji., Pro|a||y, RanJom Vara||es, anJ RanJom Sgna| Prnt|es, 2nd ed., New Yoik: McGiaw-
Hill, 1987.
Further Inlurmatiun
The IEEE Individual Leaining Piogiam, RanJom Sgna| na|yss w| RanJom Protesses anJ Ka|man F|erng,
piepaied by Cail G. Looney (IEEE Educational Activities Boaid, PO Box 1331, Piscataway, NJ 08855-1331,
1989) contains a gentle intioduction to estimation and Kalman flteiing.
Also see H. M. Denny, Ceng RJ o[ Iner[erente, IEEE Video Confeience, Educational Activities Dept.,
Piscataway, NJ, 08855-1331, 1992.
73.3 Stuchastic Prucesses
Cor| C. Iooney
Intruductiun tu Randum Yariab!es
A ranJom ara||e (iv) is specifed by its ro|a||y Jensy [unton (pdf)
[
(a) lim
r0
(1/r)Pa - (r/2) < s a - (r/2)]
In othei woids, the iectangulai aiea r [
(a) appioximates the piobability P( s a - (r/2)] - Pa - (r/2)< ].

The joint pdf of two iv`s and B is specifed by
[
B
(a,|) lim
r0
(1/r
2
)Pa - r < s a - (r/2) and | - r < B s | - (r/2)]
A similai defnition holds foi any fnite numbei of iv`s.
The exeteJ a|ue E], oi mean
, of a iv A is the fist moment of the pdf, and the arante of is the

second centialized moment, defned iespectively by
(73.39a)

E a[ a Ja
~
~
[
] ( )
(73.39b)
The squaie ioot of the vaiiance is the sanJarJ Jeaon, which is also called the roo mean square (ims) eiioi.
The toarante of two iv`s and B is the second-oidei centialized joint moment
(73.40)
The noncentialized second moments aie the mean-square a|ue and the torre|aon, iespectively,
A set of iv`s , B, and C is defned to be nJeenJen whenevei theii joint pdf factois as
[
BC
(a,|, t) [
(a)[
B
(|)[
C
(t) (73.41)
foi all a, |, and t, and similaily foi any fnite set of iv`s. A wea| nJeenJente holds when the second moment
of the joint pdf, the coiielation, factois as EB] E]EB], so that o
B
0, in which case the iv`s aie said
to be untorre|aeJ. The covaiiance of and B is a measuie of how often and B vaiy togethei (have the same
sign), how often they vaiy oppositely (diffeient signs), and by how much, on the aveiage ovei tiials of outcomes.
To standaidize so that units do not inuence the measuie of dependence, we use the torre|aon toe[ften
B

B
/
B
The accuiacy of appioximating a iv as a lineai function of anothei iv B, - tB - J, foi ieal coeffcients
t and J, is found by minimizing the mean-squaie eiioi r E{ - (tB - J)]
2
. Upon squaiing and taking the
expected values, we can obtain r
min
o
2
(1 - p
B
2
), which shows p
B
to be a measuie of the degiee of lineai
ielationship between and B. Because r
min
> 0, this shows that p
B
s 1, which demonstiates the Cauchy-
Schwaiz inequality
EB] s {E
2
]EB
2
]
1/2
(73.42)
When p
B
1, then knowledge of one of oi B completely deteimines the othei (t = 0), and so and B aie
completely dependent, while p
B
0 indicates theie is no lineai ielationship, i.e., that and B aie uncoiielated.
An impoitant iesult is the [unJamena| |eorem o[ exetaon: if g() is any ieal function, then the expected
value of the iv B g() is given by
(73.43)
Stuchastic Prucesses
A stochastic (oi ranJom) process is a collection of iandom vaiiables {X
: e T, indexed on an oideied set T

that is usually a subset of the ieal numbeis oi integeis. Examples aie the Dow-Jones aveiages D() at each time ,
the piessuie R(x) in a pipe at distance x, oi a noise voltage N() at time . A piocess is thus a ranJom [unton
X() of whose value at each is diawn iandomly fiom a iange of outcomes foi the iv X
X() accoiding to
a piobability distiibution foi X
. A tiajectoiy {x
: e T of outcomes ovei all e T, wheie X
is the iealized
value at each , is called a sample function (oi rea|:aon) of the piocess. A stochastic piocess X() has mean
o

E a [ a Ja
2 2 2

~
~
[
( ) ] ( ) ( )
o
B B B B
E B a | [ a | JaJ|
~
~
~
~
[ [
( )( ) ( )( ) ( , )
E a [ a Ja E B a|[ a | JaJ|
B B B
] ( ) , ] ( , )
2 2 2 2
+ +
~
~
~
~
~
~
[ [ [
o o
E B E g g a [ a Ja
] ( )] ( ) ( )
~
~
[
value EX()] () at time , and autocorrelation function R
XX
(, - r) EX()X( - r)] at times and -
r, the coiielation of two iv`s at two times offset by r. When () 0 foi all , the autocoiielation function
equals the auotoarante [unton C
XX
(, - r) E(X() - ())(X( - r) - ( - r))].
A piocess X() is completely deteimined by its joint pdf `s [
X((1)) . . . X( (n))
(x(
1
), . . . , x(
n
)) foi all time
combinations
1
, . . .,
n
and all positive integeis n (wheie (,)
,
). When the iv`s X() aie J (independent,
identically distiibuted), then knowledge of one pdf yields the knowledge of all joint pdf `s. This is because we
can constiuct the joint pdf by factoiization, pei Eq. (73.41).
C!assihcatiuns ul Stuchastic Prucesses
The oideied set T can be continuous oi disciete, and the values that X() assumes at each may also be
continuous oi disciete, as shown in Table 73.1.
In anothei classifcation, a stochastic piocess X() is Jeermnst whenevei an entiie sample function can
be deteimined fiom an initial segment {x
: s
1
of X(). Otheiwise, it is nonJeermnst see Biown, 1983,
p. 79; oi Gaidnei, 1990, p. 304].
Statiunarity ul Prucesses
A stochastic piocess is n| orJer (srong|y) saonary whenevei all joint pdf `s of n and fewei iv`s aie independent
of all tianslations of times
1
, . . .,
n
to times r -
1
, . . . , r -
n
. The case of n 2 is veiy useful. Anothei type
of piocess is called weakly stationary (ws), oi wJe-sense saonary, and is defned to have fist- and second-
oidei moments that aie independent of time (see Section 73.2 on noise). These satisfy (1) () (constant)
foi all , and (2) R
XX
(, - r) R
XX
( - s, - s - r) foi all values of s. Foi s -, this yields R
XX
(, - r)
R
XX
(0, 0 - r), which is abbieviated to R
XX
(r). X() is untorre|aeJ whenevei C
XX
(r) 0 foi r not zeio we say
X() has no memory]. If X() is coiielated, then X(
1
) depends on values X() foi =
1
X() has memory].
Some piopeities of autocoiielation functions foi ws piocesses follow. Fiist, R
XX
(r) s R
XX
(0), -~ < r < ~,
as can be seen fiom Eq. (73.42) with R
XX
(r)
2
EX(0)X(r)] s EX(0)
2
]EX(r)
2
] R
XX
(0)R
XX
(r). Next, R
XX
(r)
is ieal and even, i.e., R
XX
(-r) R
XX
(r), which is evident fiom substituting s - r in EX(s)X(s - t)]and using
time independence. If X() has a peiiodic component, then R
XX
(r) will have that same peiiodic component,
which follows fiom the defnition. Finally, if X() has a nonzeio mean and no peiiodic components, then
the vaiiance goes to zeio (the memoiy fades) and so lim
r~
R
XX
(r) 0 -
2

2
.
Gaussian and Markuv Prucesses
A piocess X() is defned to be Caussan if foi eveiy possible fnite set {
1
, . . . ,
n
of times, the iv`s X(
1
), . . .,
X(
n
) aie ,on|y Caussan, which means that eveiy lineai combination Z a
1
X(
1
) - . . . - a
n
X(
n
) is a Gaussian
iv, defned by the Gaussian pdf
(73.44)
In case the n iv`s aie |near|y nJeenJen, i.e., Z 0 only if a
1
a
n
0, the joint pdf has the Gaussian
foim see Gaidnei, 1990, pp. 39-40]
TABLE 73.1 Continuous/Disciete Classifcation of Stochastic Piocesses
X Values
T Values Continuous Disciete
Continuo
us
Continuous stochastic
piocesses
Disciete valued stochastic
piocesses
Disciete Continuous iandom
sequences
Disciete valued iandom
sequences
[ : :
Z Z Z Z
( ) ( ) exp{ ( )
,
1 2 2
2 2
/ / o r o
[
X( (1)) . . . X( (n))
(x
1
, . . ., x
n
) 1/(2r)
n/2
C
1/2
] exp{-(x - )
C
-1
(x - ) (73.45)
wheie x (x
1
, . . . , x
n
) is a column vectoi, x
is its tianspose, = (
1
, . . . ,
n
) is the mean vectoi, C is the
toarante marx
(73.46)
and C is the deteiminant of C. If X(
1
), . . . , X(
n
) aie lineaily dependent, then the joint pdf takes on a foim
similai to Eq. (73.45), but contains impulses see Gaidnei, 1990, p. 40].
A weakly stationaiy Gaussian piocess is stiongly stationaiy to all oideis n. all Gaussian joint pdf `s aie
completely deteimined by theii fist and second moments by Eq. (73.45), and those moments aie time inde-
pendent by weak stationaiity, and so all joint pdf `s aie also. Eveiy second-oidei stiongly stationaiy stochastic
piocess X() is also weakly stationaiy because the time tianslation independence of the joint pdf`s deteimines
the fist and second moments to have the same piopeity. Howevei, non-Gaussian weakly stationaiy piocesses
need not be stiongly second-oidei stationaiy.
Rathei than with pdf `s, a piocess X() may be specifed in teims of conditional pdf `s
[
X( (1)) . . . X( (n))
(x
1
, . . ., x
n
) [
X( (n)) X( (n-1))
(x
n
x
n-1
) . . . [
X( (2)) X( (1))
(x
2
x
1
)[
X( (1))
(x
1
)
by successive applications of Bayes` law, foi
1
<
2
< <
n
. The conditional pdf `s satisfy
[
B
(a|) [
B
(a,|)/[
B
(|) (73.47)
The conditional factoiization piopeity may satisfy
(73.48)
which indicates that the pdf of the piocess at any time
n
, given values of the piocess at any numbei of pievious
times
n-1
, . . .,
1
, is the same as the pdf at
n
given the value of the piocess at the most iecent time
n-1
. Such
an X() is called a frs-orJer Mar|o rotess, in which case we say the piocess iemembeis only the pievious
value (the pievious value has inuence). In geneial, an n|-orJer Mar|o piocess iemembeis only the n most
iecent pievious values. A fist-oidei Maikov piocess can be fully specifed in teims of its fist-oidei conditional
pdf `s [
X() X(s)
(x
,x
s
) and its unconditional fist-oidei pdf at some initial time
0
, [
X( (0))
(x
0
).
Examp!es ul Stuchastic Prucesses
Figuie 73.9 shows two sample functions of nonstationaiy piocesses. Now considei the disciete time piocess
X(|) , foi all | > 0, wheie is a iv (a ranJom na| tonJon) that assumes a value 1 oi -1 with iespective
piobabilities and 1 - at | 0. This value does not change, once the initial iandom diaw is done at | 0.
This stochastic sequence has two sample functions only, the constant sequences {-1 and {1. The expected
value of X(|) at any time | is EX(|)] E] 1 - (1 - ) (-1) 2 - 1, which is independent of |. The
autocoiielation function is, by defnition, EX(|)X(| - m)] E ] E
2
] 1
2
- (1 - ) (-1)
2
1
which is also independent of time |. Thus X(|) is peifectly coiielated foi all time (the piocess has nfne
memory). This piocess is deteiministic.
Foi anothei example, put X() (t) cos(w - 4), wheie 4 is the unifoim iv on (-r, r). Then X() is a
function of the iv 4 (as well as ), so by use of Eq. (73.39a), we obtain
C
n
n n

o o
o o
1
2
1
1
2
: :
[ x x x [ x x
X n X n X n n X n X n n n ( ( )) ( ( - )) . . . ( ( )) . - ( ( )) ( ( - )) -
( , . . . , ) ( )

1 1 1 1 1 1

Theiefoie, the mean does not vaiy with time . The autocoiielation is
using cos(x)cos(y)
1
2{cos(x - y) - cos(x - y) and letting O 2w - 24 - wr]. Theiefoie, X( ) is ws. The
autocoiielation is peiiodic in the offset vaiiable r.
FIGURE 73.9 Examples of nonstationaiy piocesses.
E X t w [ J t w ( ) cos( ) ( ) ( / ) sin( ) + +
[
o o o r o
r
r
r
r
d
2 0
R E t w t w w
t E w w w
t w w w [ J
t w w w J
t
XX
( , ) ( ) cos( )( ) cos( )]
cos( ) cos( )]
cos( ) cos( ) ( )
( ) {cos( ) cos( )( / )
(
+ + + +
+ + +
+ + +
+ + +
[
[
t t
t
o t o o
o t t r o
r
r
r
r
r
d d
d d
d
d
2
2
2
2
2 2 2 1 2
4
/
/ )) {sin( ) sin( ) cos( )
( ) {cos( ) ( ) cos( )
+ +

O O 2 2 2
4 2 2
2 2
r r t r
r t r t
w
t w t w / /
Now considei the example X() cos(2r[
0
) foi each , wheie [
0
is a constant fiequency, and the amplitude
is a iandom initial condition as given above. Theie aie only two sample functions heie: (1) x(t) cos(2r[
0
)
and (2) x() -cos(2r[
0
). A ielated example is X() A cos(2r[
0
- 4), wheie is given above, the phase 4
is the unifoim iandom vaiiable on 0,r], and and 4 aie independent. Again, 4 and do not depend on
time (initial iandom conditions). Thus, the sample functions foi X() aie x() cos(2r[ - ), wheie 4
is the value assumed initially. Theie aie infinitely many sample functions because of the phase. Equation
(73.39b) and the independence of and 4 yield
which is dependent upon time. Thus, X() is not ws.
Next, let X() a - S()]cos2r[ - 4], wheie the signal S() is a nondeteiministic stochastic piocess. This
is an amplitude-modulated sine wave caiiiei. The caiiiei cos2r[ - 4] has iandom initial condition 4 and
is deteiministic. Because S() is nondeteiministic, X() is also. The expected value EX()] Ea -
S()]Ecos(2r[ - 4)] can be found as above by independence of S() and 4.
Finally, let X() be uncoiielated (EX()X( - r)] 0 foi r not zeio) such that each iv X() X
is Gaussian
with zeio mean and vaiiance o
2
() , foi all > 0. Any iealized sample function x() of X() cannot be piedicted
in any aveiage sense based on past values (uncoiielated Gaussian iandom vaiiables aie independent). The
vaiiance giows in an unbounded mannei ovei time, so X() is neithei stationaiy noi deteiministic. This is
called the Vener piocess.
A useful model of a ws piocess is that foi which 0 and R
XX
(r) o
X
2
exp(-or). If this piocess is also
Gaussian, then it is stiongly stationaiy and all of its joint pdf `s aie fully specifed by R
XX
(r). In this case it is
also a fist-oidei Maikov piocess and is called the Ornsen-U||en|et| piocess see Gaidnei, 1990, p. 102].
Unlike white noise, many ieal-woild ws stochastic piocesses aie coiielated (R
XX
(, - r) > 0) foi r > 0. The
autocoiielation eithei goes to zeio as r goes to infnity, oi else it has peiiodic oi othei nondecaying memoiy.
We considei ws piocesses hencefoith foi nonstationaiy piocesses, see Gaidnei, 1990]. We will also assume
without loss of geneiality that 0.
Linear Fi!tering ul Weak!y Statiunary Prucesses
Let the ws stochastic piocess X() be the input to a lineai time-invaiiant stable fltei with impulse iesponse
function |(). The output of the fltei is also a ws stochastic piocess and is given by the convolution
(73.49)
The mean of the output piocess is obtained by using the lineaiity of the expectation opeiatoi see Gaidnei,
1990, p. 32]
(73.50)
wheie H( [) |
~
-~
|()e
-,2r[
J is the fltei tiansfei function and H(0) is the dc gain of the fltei.
E X E [ E E g [ J

( )] cos( )] ] ( )] cos( )( )
( ) sin( ) ( )sin( ) sin( )]
( )sin( ) sin( )] ( ) sin(
+ +
+ +

[
2 2 1
2 2 2
2 2 2 2
0
0
r r o r o
r r o r r r r
r r r r
r
o
r
d d /
/ /
/ /
rr )
Y | X | s X s Js ( ) ( ) ( ) ( ) ( ) -
~
~
[

Y X
X X
E Y E | s X s Js | s E X s Js | s Js
| s Js H

1
]
1

~
~
~
~
~
~
~
~
[ [ [
[
( )] ( ) ( ) ( ) ( )] ( )
( ) ( ) 0
The autocoiielation of the output piocess, obtained by using the lineaiity of E], is
(73.51)
wheie r
|
(r) |
~
-~
|(r - u)|(u)Ju. Howevei, r
|
(r) has Fouiiei tiansfoim H( [ )H
-
( [ ) H([ )
2
, because the
Fouiiei tiansfoim of the convolution of two functions is the pioduct of theii Fouiiei tiansfoims, and the
Fouiiei tiansfoim of |(-r) is the complex conjugate H
-
( [ ) of the Fouiiei tiansfoim H( [ ) of |(r). Thus, the
Fouiiei tiansfoim of R
YY
(r), denoted by F{R
YY
(r), is
F{R
YY
(r) F{R
XX
(r)-|(r)-|(-r) F{R
XX
(r) H( [ )H-( [ ) F{R
XX
(r) H( [ )
2
Upon defning the functions
S
XX
( [ ) F{R
XX
(r), S
YY
( [ ) F{R
YY
(r) (73.52)
we can also deteimine R
YY
(r) via the two steps
S
YY
( [ ) S
XX
([ ) H( [ )
2
(73.53)
(73.54)
Equations (73.52) defne the ower setra| Jensy [untons (psdf `s) S
XX
([ ) foi X() and S
YY
( [ ) foi Y().
Thus, R
XX
(r) and S
XX
([ ) aie Fouiiei tiansfoim paiis, as aie R
YY
(r) and S
YY
([ ) (see Eq. 73.20). Fuithei, the psdf
S
XX
([ ) of X() is a powei spectium (in an aveiage sense). If X() is the voltage diopped acioss a 1-O iesistoi,
then X
2
() is the instantaneous powei dissipation in the iesistance. Consequently, R
XX
(0) EX
2
()] is the
expected powei dissipation ovei all fiequencies, i.e., by Eq. (73.54) with r 0, we have
We want to show that when we pass X() thiough a naiiow bandpass fltei with a bandwidth o centeied at
the fiequency [
0
, the expected powei at the output teiminals, divided by the bandwidth o, is S
XX
([
0
) in the
limit as o 0. This shows that S
XX
([ ) is a density function (whose aiea is the total expected powei ovei all
fiequencies, just as the aiea undei a pdf is the total piobability). This iesult that R
XX
(r) and S
XX
( [ ) aie a Fouiiei
tiansfoim paii is known as the Vener-K|nt|n ielation see Gaidnei, 1990, p. 230].
To veiify this ielation, let H( [ ) be the tiansfei function of an ideal bandpass fltei, wheie
H( [ ) 1, [ - [
0
< o/2; H( [ ) 0, otheiwise
R E Y Y E | X J | u X u Ju
E X X u | | u JJu
R u | u Ju |
YY
XX
( ) ( ) ( )] ( ) ( ) ( ) ( )
( ) ( )] ( ) ( )
( ) ( )
t t t
t
t
+ +
1
]
1
+
+
~
~
~
~
~
~
~
~
~
~
~
~
[ [
[ [
[ [
( ( )
( ( )] ) ( ) ( )( )
( ) ( ) ( )
( ) ( )] ( ) ( ) ( ) ( )] ( ) ( )
J
R u | u Ju | J
R | | J
R | | R | | R r
XX
XX
XX XX XX |

+ - +
}

- - - - -
~
~
~
~
~
~
[ [
[
t
t t
t t t t t t t t
R S [ S [ e J[
YY YY YY
, [
( ) { ( ) ( ) t
r t

~
~
[
1 2
R S [ J[
XX XX
( ) ( ) 0
~
[
Let Y() be the output of the fltei. Then Eqs. (73.54) and (73.53) piovide
Dividing by 2o and taking the limit as o 0 yields (1/2)S
XX
([
0
) - (1/2)S
XX
(-[
0
), which becomes S
XX
([
0
) when
we use the fact that psdf `s aie even and ieal functions (because they aie the Fouiiei tiansfoims of autocoiielation
functions, which aie even and ieal).
Foi example, let X() be white noise, with S
XX
([ ) N
0
, being put thiough a fist-oidei lineai time-invaiiant
system with iespective impulse iesponse and tiansfei functions
|() exp{-o , > 0; |() 0, < 0 H( [ ) 1/o - ,2r[ ], all [
The tempoial coiielation of |() with itself is r
|
(r) (1/2o)exp{-or, so the powei tiansfei function is H( [ )
2

1/o
2
- (2r[)
2
]. The autocoiielation foi the input X() is
which is an impulse. It follows (see Eq. 73.22) that the output Y() has iespective autocoiielation and psdf
R
YY
(r) N
0
o(r)]-(1/2o) e
-or
] (N
0
/2o) e
-or
, S
YY
( [ ) N
0
/o
2
- (2r[)
2
]
The output expected powei EY
2
()] can be found fiom eithei one of
If the input X() to a lineai system is Gaussian, then the output will also be Gaussian see Biown, 1983; oi
Gaidnei, 1990]. Thus, the output of a fist-oidei lineai time-invaiiant system diiven by Gaussian white noise
is the Oinstein-Uhlenbeck piocess, which is also a fist-oidei Maikov piocess.
Foi anothei example, let X() cos(w
o
- O), wheie the iandom amplitude has zeio mean, the iandom
phase O is unifoim on -r, r], and and O aie independent. As befoie, we obtain R
XX
(r) o
2
cos(w
o
r), fiom
which it follows that S
XX
([ ) (o
2
/2)o([ - w
o
/2r) - o([ - w
o
/2r)]. These impulses in the psdf, called setra|
|nes, iepiesent positive amounts of powei at disciete fiequencies.
Cruss-Curre!atiun ul Prucesses
The tross-torre|aon [unton foi two iandom piocesses X() and Y() is defned via
R
XY
(, - r) EX()Y( - r)] (73.55)
Let both piocesses be ws with zeio means, so the covaiiance coincides with the coiielation function. We say
that two ws piocesses X() and Y() aie ,on|y ws whenevei R
XY
(, - r) R
XY
(r). In case Y() is the output of
a fltei with impulse iesponse |(),we can fnd the cioss-coiielation R
XY
(r) between the input and output via
E Y R S [ J[ S [ H [ J[
S [ J[ S [ J[
YY YY XX
XX
[
[
XX
[
[
( )] ( ) ( ) ( ) ( )
( ) ( )
/
/
/
/
2 2
2
2
2
2
0
0
0
0
0

+
~
~
~
~
+

+
[ [
[ [

o
o
o
o
R N e J[ N
XX
, [
( ) ( ) t o t
r t

~
~
[
0
2
0
E Y R N E Y S [ J[ N
YY YY
( )] ( ) / ( )] ( ) /
2
0
2
0
0 2 2
~
~
[
o o oi
(73.56)
Cioss-coiielation functions of ws piocesses satisfy (1) R
XY
(-r) R
YX
(r), (2) R
XY
(r)
2
s R
XX
(0)R
YY
(0), and
(3) R
XY
(r) s (1/2)R
XX
(0) - R
YY
(0)]. The fist follows fiom the defnition, while the second comes fiom
expanding E{Y( - r) - oX()
2
] > 0. The thiid comes fiom the fact that the geometiic mean cannot exceed
the aiithmetic mean see Peebles, 1987, p. 154].
Taking the Fouiiei tiansfoim of the leftmost and iightmost sides of Eqs. (73.56) yields
S
XY
( [ ) S
XX
([ )H( [ ) (73.57)
The Fouiiei tiansfoim of the cioss-coiielation function is the tross-setra| Jensy [unton
(73.58)
Accoiding to Gaidnei 1990, p. 228], this is a setra| torre|aon Jensy [unton that does not iepiesent powei
in any sense.
Equation (73.57) suggests a method foi identifying a lineai time-invaiiant system. If the system is subjected
to a ws input X() and the powei spectial density of X() and the cioss-spectial density of X() and the output
Y() aie measuied, then the iatio yields the system tiansfei function
H( [ ) S
XY
([ )/S
XX
( [ ) (73.59)
In fact, it can be shown that this method gives the best lineai time-invaiiant model of the (possibly time vaiying
and nonlineai) system in the sense that the time-aveiaged mean-squaie eiioi between the outputs of the actual
system and of the model, when both aie subjected to the same input, is minimized see Gaidnei, 1990,
pp. 282-286].
As an application, suppose that an undeisea sonai-based device is to fnd the iange to a taiget, as shown in
Fig. 73.10, by tiansmitting a sonai signal X() and ieceiving the ieected signal Y(). If is the velocity of the
sonai signal, and r
o
is the offset that maximizes the cioss-coiielation R
XY
(r), then the iange (distance) J can
be deteimined fiom J r
o
/2 (note that the signal tiavels twice the iange J).
Cuherence
When X() and Y() have no spectial lines at [, the fnite spectial coiielation S
XY
( [ ) is actually a spectial
covaiiance and the two associated vaiiances aie S
XX
([ ) and S
YY
([ ). We can noimalize S
XY
( [ ) to obtain a setra|
torre|aon toe[ften Y
XY
( [ ) defned by
Y
XY
( [ )
2
S
X Y
( [ )
2
/S
XX
([ )S
YY
( [ ) (73.60)
We call Y
XY
( [ ) the to|erente [unton. It is a measuie of the powei coiielation of X() and Y() at each fiequency [.
When Y() X()-|(), it has a maximum: by Eqs. (73.53), (73.59), and (73.60), Y
XY
( [ )
2
S
XX
( [ )
H([ )
2
/S
XX
( [ ) S
XX
( [ ) H( [ )
2
] 1. In the geneial case we have
S
XY
([ ) s S
XX
([ )S
YY
( [ )]
1/2
(73.61)
R E X Y E X | u X u Ju
| u E X X u Ju
| u R u Ju R |
XY
XX XX
( ) ( ) ( )] ( ) ( ) ( ) ]
( ) ( ) ( )]
( ) ( ) ( ) ( )
t t t
t
t t t
+ +
+
-
~
~
~
~
~
~
[
[
[
S [ R e J
XY XY
, [
( ) ( )
~
~

[
t t
r t 2
Upon minimizing the mean-squaie eiioi r E(Y() - X()-|())
2
] ovei all possible impulse iesponse
functions |(), the optimal one, |
o
(), has tiansfei function
H
o
( [ ) S
XY
( [ )/S
XX
( [ ) (73.62)
Fuithei, the iesultant minimum value is given by
see Gaidnei, 1990, pp. 434-436; oi Bendat and Pieisol, 1986]. At fiequencies [ wheie Y
XY
( [ ) - 1, r
min
- 0.
Thus 1 - Y
XY
( [ )
2
is the mean-squaie piopoition of Y() not accounted foi by X(), while Y
XY
( [ )
2
is the
piopoition due to X(). When Y() X()-|(), r
min
0.
The optimum system H
o
([ ) of Eq. (73.62) is known as the Vener f|er foi minimum mean-squaie eiioi
estimation of one piocess Y() using a flteied veision of anothei piocess X() see Gaidnei, 1990; oi Peebles,
1987, p. 262].
Ergudicity
When the time average
exists and equals the coiiesponding expected value EX()], then the piocess X() is said to possess an ergoJt
roery assotaeJ w| |e mean. Theie aie eigodic piopeities associated with the mean, autocoiielation (and
powei spectial density), and all fnite-oidei joint moments, as well as fnite-oidei joint pdf `s. If a piocess has
all possible eigodic piopeities, it is said to be an ergoJt rotess.
Let Y() gX( -
1
), . . ., X( -
n
)], wheie g] is any noniandom ieal function, so that Y() is a function
of a fnite numbei of time samples of a stiongly stationaiy piocess. Foi example, let (1) Y() X( -
1
)X( -
2
), EY()] R
XX
(
1
-
2
) and (2) Y() 1 if X() < x, Y() 0, otheiwise, so that
FIGURE 73.10 A sonai iange fndei.
r
min
( ) ( ) ]
~
~
[
S [ [ J[
YY XY
1
2

lim ( / ) ( )
T
T
T
T X J
~

[
1
2
2
/
/
E Y P X x P X x P X x [ : J:
X
( )] ( ( ) ) ( ( ) ) ( ( ) ) ( )
( )
< + > <
~
[
1 0
1
We want to know undei what conditions the mean-squaie eiioi between the time aveiage
and the expected value EY()] will conveige to zeio. It can be shown that a necessaiy and suffcient condition
foi the mean-squaie eigodic piopeity
(73.63)
to hold is that
(73.64)
Foi example, if C
YY
(r) 0 as r ~, then Eq. (73.64) will hold, and thus Eq. (73.63) will also, wheie C
YY
(r)
is the covaiiance function of Y(). As long as the two sets of iv`s {X( -
1
), . . ., X( -
n
) and {X( -
1
- r), . . .,
X( -
n
- r) become independent of each othei as r ~, the above condition holds, so Eq. (73.63) holds
see Gaidnei, 1990, pp. 163-174].
In piactice, if X() exhibits eigodicity associated with the autocoiielation, then we can estimate R
XX
(r) using
the time aveiage
(73.65)
In this case the mean-squaie estimation eiioi E{(X()X( - r))
T
- R
XX
(r)
2
] will conveige to zeio as T incieases
to infnity, and the powei spectial density S
XX
( [ ) can also be estimated via time aveiaging see Gaidnei, 1990,
pp. 230-231].
Dehning Terms
Autocorrelation function: A function R
XX
(, - r) EX()X( - r)] that measuies the degiee to which any
two iv`s X() and X( - r), at times and - r, aie coiielated.
Coherence function: A function of fiequency f that piovides the degiee of coiielation of two stochastic
piocesses at each f by the iatio of theii cioss-spectial density function to the pioduct of theii powei
spectial density functions.
Power spectral density function: The Fouiiei tiansfoim of the autocoiielation function of a stochastic piocess
X(), denoted by S
XX
([ ). The aiea undei its cuive between [
1
and [
2
iepiesents the total powei ovei all
in X() in the band of fiequencies [
1
to [
2
. Its dimension is watts pei Hz.
Sample function: A ieal-valued function x() of wheie at each time the value x() at the aigument was
deteimined by the outcome of a iv X
x().
Stochastic process: A collection of iv`s {X
: e T, wheie T is an oideied set such as the ieal numbeis oi

integeis X() is also called a iandom function, on the domain T].
Time average: Any ieal function g() of time has aveiage value g
ave
on the inteival a,|] such that the
iectangulai aiea g
ave
(| - a) is equal to the aiea undei the cuive between a and |, i.e., g
ave
1/(| - a)]
|
a
|
g()J. The time aveiage of a sample function x() is the limit of its aveiage value ovei 0,T] as T goes
to infnity.
Weakly stationary: The piopeity of a stochastic piocess X() whose mean EX()] () is a fxed constant
ovei all time , and whose autocoiielation is also independent of time in that R
XX
(, - r) R
XX
(s - ,
s - - r) foi any s. Thus, R
XX
(, - r) R
XX
(0, r) R
XX
(r).
( )

[
Y T Y J
T
T
T
( ) ( / ) ( ) 1
2
2
/
/
lim { ( ) ( )] ]
T T
E Y E Y
~
( )
2
0
lim ( / ) ( )
T YY
T
T C J
~
[
1 0
0
t t
( + ) +
[
X X T X X J
T
T
T
( ) ( ) ( / ) ( ) ( ) t t 1
2
2
/
/
Re!ated Tupic
16.1 Spectial Analysis
Relerences
The authoi is giateful to William Gaidnei of the Univeisity of Califoinia, Davis foi making substantial suggestions.
J.S. Bendat and A.G. Pieisol, RanJom Daa. na|yss anJ Measuremen, 2nd ed., New Yoik: Wiley-Inteiscience,
1986.
R. G. Biown, InroJuton o RanJom Sgna| na|yss anJ Ka|man F|erng, New Yoik: Wiley, 1983.
W. A. Gaidnei, InroJuton o RanJom Protesses, 2nd ed., New Yoik: McGiaw-Hill, 1990.
P. Z. Peebles, Ji., Pro|a||y, RanJom Vara||es, anJ RanJom Sgna| Prnt|es, 2nd ed., New Yoik: McGiaw-
Hill, 1987.
Further Inlurmatiun
The IEEE Individual Leaining Package, RanJom Sgna| na|yss w| RanJom Protesses anJ Ka|man F|erng,
piepaied foi the IEEE in 1989 by Cail G. Looney, IEEE Educational Activities Boaid, PO Box 1331, Piscataway,
NJ 08855-1331.
R. Iianpoui and P. Chacon, Bast Sot|ast Protesses. T|e Mar| Kat Letures, New Yoik: Macmillan, 1988.
A. Papoulis, Pro|a||y, RanJom Vara||es, anJ Sot|ast Protesses, 3id ed., New Yoik: Macmillan, 1991.
73.4 The Samp!ing Theurem
F. j. Mor| II
Most signals oiiginating fiom physical phenomena aie analog. Most computational engines, on the othei hand,
aie digital. Tiansfoiming fiom analog to digital is stiaightfoiwaid: we simply sample. Regaining the oiiginal
signal fiom these samples and assessing the infoimation lost in the sampling piocess aie the fundamental
questions addiessed by the sampling theorem.
The fundamental iesult of the sampling theoiem is, iemaikably, that a bandlimited signal is uniquely specifed
by its suffciently close equally spaced samples. Indeed, the sampling theoiem illustiates how the oiiginal signal
can be iegained fiom knowledge of the samples and the sampling iate at which they weie taken.
Populaiization of the sampling theoiem is ciedited to Shannon 1948] who, in 1948, used it to show the
equivalence of the infoimation content of a bandlimited signal and a sequence of disciete numbeis. Shannon
was awaie of the pioneeiing woik of Whittakei 1915] and Whittakei`s son 1929] in foimulating the sampling
theoiem. Kotel`nikov`s 1933] independent discoveiy in the then Soviet Union deseives mention. Higgins 1985]
ciedits Boiel 1897] with fist iecognizing that a signal could be iecoveied fiom its samples.
Suiveys of sampling theoiy aie in the widely cited papei of Jeiii 1977] and in two books by the authoi
1991, 1993]. Maivasti 1987] has wiitten a book devoted to nonunifoim sampling.
The Cardina! Series
If a signal has fnite eneigy, the minimum sampling rate is equal to two samples pei peiiod of the highest
fiequency component of the signal. Specifcally, if the highest fiequency component of the signal is B Hz, then
the signal, x(), can be iecoveied fiom the samples by
(73.66)
The fiequency B is also iefeiied to as the signal`s bandwidth and, if B is fnite, x() is said to be bandlimited.
The signal, x(), is heie being sampled at a iate of 2 B samples pei second. If sampling weie done at a lowei
x x
n
B
B n
B n
n
( )
sin ( )]
_
,
~
~
1
2
2
2 r
r
iate, the ieplications would oveilap and the infoimation about X(e) and thus x()] is iiietiievably lost.
Undeisampling iesults in a|aseJ data. The minimum sampling iate at which aliasing does not occui is iefeiied
to as the Nyquist rate which, in oui example, is 2B. Eq. (73.66) was dubbed the cardinal series by the junioi
Whittakei 1929].
A signal is bandlimited in the low-pass sense if theie is a B > 0 such that
(73.67)
wheie the gate function H (() is one foi ( s 1/2 and is otheiwise zeio, and
(73.68)
is the Fourier transform of x(). That is, the spectium is identically zeio foi e > 2rB. The B paiametei is
iefeiied to as the signal`s bandwidth. The inveise Fouiiei tiansfoim is
(73.69)
The sampling theoiem ieduces the noimally continuum infnity of oideied paiis iequiied to specify a
function to a countable-although still infnite-set. Remaikably, these elements aie obtained diiectly by
sampling.
How can the caidinal seiies inteipolate uniquely the
bandlimited signal fiom which the samples weie taken:
Could not the same samples be geneiated fiom anothei
bandlimited signal: The answei is no. Bandlimited
functions aie smooth. Any behavioi deviating fiom
smooth would iesult in high-fiequency components
which in tuin invalidates the iequiied piopeity of being
bandlimited. The smoothness of the signal between
samples piecludes aibitiaiy vaiiation of the signal
theie.
Let`s examine the caidinal seiies moie closely. Eval-
uate Eq. (73.74) at m/2B. Since sinc(n) is one foi
n 0 and is otheiwise zeio, only the sample at m/2B
contiibutes to the inteipolation at that point. This is
illustiated in Fig. 73.11, wheie the ieconstiuction of a signal fiom its samples using the caidinal seiies is shown.
The value of x() at a point othei than a sample location e.g., (m + 12)/2B] is deteimined by all of the
sample values.
Pruul ul the Samp!ing Theurem
Boiel 1897] and Shannon 1948] both discussed the sampling theoiem as the Fouiiei tiansfoim dual of the
Fouiiei seiies. Let x() have a bandwidth of B. Considei the peiiodic signal
(73.70)
X X
B
( ) ( ) u u
u
r
_
,
H
4
X x e J
,
( ) ( ) u
u

~
~
[
x X e J
,
( ) ( )
~
~
[
1
2r
u u
u
FIGURE 73.11 Illustiation of the inteipolation that
iesults fiom the caidinal seiies. A sinc function, weighted
by the sample, is placed at each sample bottom. The sum
of the sincs exactly geneiates the oiiginal bandlimited func-
tion fiom which the samples weie taken.
Y X nB
n
( ) ( ) u u r
~
~
4
The function Y(e) is a peiiodic function with peiiod 4rB. Fiom Eq. (73.67) X(e) is zeio foi e > 2rB and is
thus fnite in extent. The teims in Eq. (73.70) theiefoie do not oveilap. Peiiodic functions can be expiessed as
a Fouiiei seiies.
(73.71)
wheie the Fouiiei seiies coeffcients aie
oi
(73.72)
wheie we have used the inveise Fouiiei tiansfoim in Eq. (73.69). Substituting into the Fouiiei seiies in Eq.
(73.71) gives
(73.73)
Since a peiiod of Y(e) is X(e), we can get back the oiiginal spectium by
Substitute Eq. (73.73) and inveise tiansfoiming gives, using Eq. (73.69),
oi
(73.74)
wheie
is the inveise Fouiiei tiansfoim of H (e/2r). Eq. (73.74) is, of couise, the caidinal seiies.
Y
,n
B
n
n
( ) exp u o
u
_
,
~
~
2
o
r
u
u
u
r
r
n
B
B
B
Y
,n
B
J

_
,
[
1
4 2
2
2
( ) exp
o
n
B
x
n
B
_
,
1
2 2
Y
B
x
n
B
,n
B
n
( ) exp u
u
_
,
_
,
~
~
1
2 2 2
X Y
B
( ) ( ) u u
u
r
j
_
,
4
x
B
x
n
B
,n
B
e J
n
B
B
,
( ) exp

_
,
_
,
~
~

[
1
4 2 2
2
2
r
u
u
r
r
u
x x
n
B
B n
n
( ) ( )

_
,

~
~
2
2 sinc
sinc( )
sin
r
r
The sampling theoiem geneially conveiges unifoimly, in the sense that
wheie the tiuncated caidinal seiies is
(73.75)
Suffcient conditions foi unifoim conveigence aie Maiks, 1991]
1. the signal, x(), has fnite eneigy, E,
2. oi X(e) has fnite aiea,
Caie must be taken in the second case, though, when singulaiities exist at e 2rB. Heie, sampling may
be iequiied to be stiictly gieatei than 2B. Such is the case, foi example, foi the signal, x() sin (2rB). Although
the signal is bandlimited, and although its Fouiiei tiansfoim has fnite aiea, all of the samples of x() taken at
n/2B aie zeio. The caidinal seiies in Eq. (73.74) will thus inteipolate to zeio eveiywheie. If the sampling
iate is a bit gieatei than 2B, howevei, the samples aie not zeio and the caidinal seiies will unifoimly conveige
to the piopei answei.
The Time-Bandvidth Pruduct
The caidinal seiies iequiies knowledge of an infnite numbei of samples. In piactice, only a fnite numbei of
samples can be used. If most of the eneigy of a signal exists in the inteival 0 s s T, and we sample at the
Nyquist iate of 2B samples pei second, then a total of S (2BT) samples aie taken. ((0) denotes the laigest
numbei not exceeding 0.) The numbei S is a measuie of the degiees of fieedom of the signal and is iefeiied
to as its time-bandwidth product. A 5-min single-tiack audio iecoiding iequiiing fdelity up to 20,000 Hz,
foi example, iequiies a minimum of S 2 20,000 5 60 12 million samples. In piactice, audio sampling
is peifoimed well above the Nyquist iate.
Suurces ul Errur
Exact inteipolation using the caidinal seiies assumes that (1) the values of the samples aie known exactly, (2) the
sample locations aie known exactly, and (3) an infnite numbei of teims aie used in the seiies. Deviation fiom
these iequiiements iesults in inteipolation eiioi due to (1) data noise, (2) jittei, and (3) tiuncation, iespectively.
The effect of data eiioi on the iestoiation can be signifcant. Some innocently appeaiing sampling theoiem
geneializations, when subjected to peifoimance analysis in the piesence of data eiioi, aie ievealed as ill-posed.
In othei woids, a bounded eiioi on the data can iesult in unbounded eiioi on the iestoiation Maiks, 1991].
Data Nuise
The souice of data noise can be the signal fiom which samples aie taken, oi fiom iound-off eiioi due to fnite
sampling piecision. If the noise is additive and iandom, instead of the samples
lim ( ) ( )
N
N
x x
~

2
0
x x
n
B
B n
N
n N
N
( ) ( )

_
,
2
2 sinc
E x J < ~
~
~
[
( )
2
X J < ~
~
~
[
( ) u u
we must deal with the samples
wheie (() is a stochastic piocess. If these noisy samples aie used in the caidinal seiies, the inteipolation, instead
of simple x(), is
x() - q()
wheie the inteipolation noise is
If (() is a zeio mean piocess, then so is the inteipolation noise. Thus, the noisy inteipolation is an unbiased
veision of x(). Moie iemaikably, if (() is a zeio-mean (wide-sense) stationaiy piocess with unceitainty
(vaiiance) o
2
, then so is q(). In othei woids, |e unterany a |e sam|e on |otaons s |e same as a a||
ons o[ nero|aon Maiks, 1991].
Truncatiun
The tiuncated caidinal seiies is in Eq. (73.75). A signal cannot be both bandlimited and of fnite duiation.
Indeed, a bandlimited function cannot be identically zeio ovei any fnite inteival. Thus, othei than the iaie
case wheie an infnite numbei of the signal`s zeio ciossings coincide with the sample locations, tiuncation will
iesult in an eiioi.
The magnitude of this truncation error can be estimated thiough the use of Paiseval`s theoiem foi the
caidinal seiies that states
The eneigy of a signal can thus be deteimined diiectly fiom eithei the signals oi the samples. The eneigy
associated with the tiuncated signal is
If E - E
N
<< E, then the tiuncation eiioi is small.
Jitter
Jitter occuis when samples aie taken neai to but not exactly at the desiied sample locations. Instead of the
samples x (n/2V), we have the samples
x
n
B 2
_
,
x
n
B
n
B 2 2
_
,
_
,
q ( ) ( )
n
B
B n
n
_
,

~
~
2
2 sinc
E x J
B
x
n
B
_
,
~
~
~
~
[
( )
2
2
1
2 2
E
B
x
n
B
N
N
N
_
,
1
2 2
2
wheie o
n
is the jittei offset of the nth sample. Foi jittei, the o
n
`s aie not known. If they weie, an appiopiiate
nonunifoim sampling theoiem Maiks, 1993; Maivasti, 1987] could be used to inteipolate the signal.
Using the jitteied samples in the caidinal seiies iesults in an inteipolation that is not an unbiased estimate
of x(). Indeed, if the piobability density function of the jittei is the same at all sample locations, the expected
value of the jitteied inteipolation is the convolution of x() with the piobability density function of the jittei.
This bias can be iemoved by inveise flteiing at a cost of decieasing the signal-to-noise iatio of the inteipolation
Maiks, 1993].
Genera!izatiuns ul the Samp!ing Theurem
Theie exist numeious geneializations of the sampling theoiem Maiks, 1991; Maiks, 1993].
1. Stochastic processes. A wide-sense stationaiy stochastic piocess, (), is said to be bandlimited if its
autocoiielation, R
(), is a bandlimited function. The caidinal seiies

conveiges to () in the sense that
wheie E denotes expectation.
2. Nonuniform sampling. Theie exist numeious scenaiios wheiein inteipolation can be peifoimed fiom
samples that aie not spaced unifoimly. Maivasti 1987] devotes a book to the topic.
3. Kramer's generalization. Kiamei geneialized the sampling theoiem to integial tiansfoims othei than
Fouiiei, foi example, to Legendie and Lagueiie tiansfoims.
4. Papoulis' generalization. Shannon noted that a bandlimited signal could be iestoied when sampling
was peifoimed at half the Nyquist iate if, at eveiy sample location, a sample of the signal`s deiivative
weie also taken. Recuiient nonunifoim sampling is wheie P samples aie spaced the same in eveiy P
Nyquist inteivals. Anothei sampling scenaiio is when a signal and its Hilbeit tiansfoim aie both sampled
at half theii iespective Nyquist iates. Restoiation of the signal fiom these and numeious othei sampling
scenaiios aie subsumed in an eloquent geneialization of the sampling theoiem by Papoulis.
5. Lagrangian interpolation. Lagiangian inteipolation is a topic familiai in numeiical analysis. An Nth
oidei polynomial is ft to N - 1 aibitiaiily spaced sample points. If an infnite numbei of samples aie
equally spaced, then Lagiangian inteipolation is equivalent to the caidinal seiies.
6. Trigonometric polynomials. All peiiodic bandlimited signals can be expiessed as tiigonometiic poly-
nomials (i.e., a Fouiiei seiies with a fnite numbei of teims). If the seiies has M teims, then the signal
has M degiees of fieedom which can be deteimined fiom M samples taken within a single peiiod.
7. Multidimensional sampling theorems. Multidimensional signals, such as images, iequiie dimensional
extensions of the sampling theoiem. The sampling of the signal now iequiies geometiical inteipietation.
Unifoim sampling of an image, foi example, can eithei be done on a iectangulai oi hexagonal giid. The
minimum sampling density foi one geometiy may diffei fiom that of anothei. The smallest sampling
density that does not iesult in aliasing can be achieved, in many cases, with a numbei of diffeient unifoim
sampling geometiies and is iefeiied to as the Nyquist density. Inteiestingly, sampling can sometimes be
peifoimed below the Nyquist density with nonunifoim sampling geometiies such that the multidimen-
sional signal can be iestoied. Such is not the case foi one dimension.
x
n
V
n
2
_
,
o
`
( ) ( )
n
B
B n
n
_
,

~
~
2
2 sinc
E ]
`
( ) ( )
2
0
8. Continuous sampling. When a signal is known on one oi moie disjoint inteivals, it is said to have been
continuously sampled. Divide the time line into inteivals of T. Peiiodic continuous sampling assumes
that the signal is known on each inteival ovei an inteival of oT wheie o is the duty cycle. Continuously
sampled signals can be accuiately inteipolated even in the piesence of aliasing. Othei continuously
sampled cases, each of which can be consideied as a limiting case of continuously peiiodically sampled
iestoiation, include
(a) Interpolation. The tails of a signal aie known and we wish to iestoie the middle.
(b) Extrapolation. We wish to geneiate the tails of a function with knowledge of the middle.
(c) Prediction. A signal foi > 0 is to be estimated fiom knowledge of the signal foi < 0.
Fina! Remarks
Since its populaiization in the late 1940s, the sampling theoiem has been studied in depth. Moie than 1000
papeis have been geneiated on the topic Maiks, 1993]. Its undeistanding is fundamental in matching the
laigely continuous woild to digital computation engines.
Dehning Terms
Aliasing: A phenomenon that occuis when a signal is undeisampled. High-fiequency infoimation about the
signal is lost.
Cardinal series: The foimula by which samples of a bandlimited signal aie inteipolated to foim a continuous
time signal.
Fourier transform: The mathematical opeiation that conveits a time-domain signal into the fiequency
domain.
Jitter: A sample is tempoially displaced by an unknown, usually small, inteival.
Kramer's generalization: A sampling theoiy based on othei than Fouiiei tiansfoims and fiequency.
Lagrangian interpolation: A classic inteipolation pioceduie used in numeiical analysis. The sampling the-
oiem is a special case.
Nyquist rate: The minimum sampling iate that does not iesult in aliasing.
Papoulis' generalization: A sampling theoiy applicable to many cases wheiein signal samples aie obtained
eithei nonunifoimly and/oi indiiectly.
Sampling rate: The numbei of samples pei second.
Sampling theorem: Samples of a bandlimited signal, if taken close enough togethei, exactly specify the
continuous time signal fiom which the samples weie taken.
Signal bandwidth: The maximum fiequency component of a signal.
Time bandwidth product: The pioduct of a signal`s duiation and bandwidth appioximates the numbei of
samples iequiied to chaiacteiize the signal.
Truncation error: The eiioi that occuis when a fnite numbei of samples aie used to inteipolate a continuous
time signal.
Re!ated Tupic
8.5 Sampled Data
Relerences
E. Boiel, Sui l`inteipolation," C.R. taJ. St. Pars, vol. 124, pp. 673-676, 1897.
J. R. Higgins, Five shoit stoiies about the caidinal seiies," Bu||. m. Ma|. Sot., vol. 12, pp. 45-89, 1985.
A. J. Jeiii, The Shannon sampling theoiem-its vaiious extension and applications: a tutoiial ieview," Prot.
IEEE, vol. 65, pp. 1565-1596, 1977.
V. A. Kotel`nikov, On the tiansmission capacity of 'ethei` and wiie in electiocommunications," I:J. ReJ. Ur.
Sya: RKK (Moscow), 1933.
R. J. Maiks II, InroJuton o S|annon Sam|ng anJ Inero|aon T|eory, New Yoik: Spiingei-Veilag, 1991.
R. J. Maiks II, Ed., JanteJ Tots n S|annon Sam|ng anJ Inero|aon T|eory, New Yoik: Spiingei-Veilag,
1993.
F. A. Maivasti, UnfeJ roat| o Zero-Crossng anJ Nonun[orm Sam|ng, Oak Paik, Ill.: Nonunifoim, 1987.
C. Shannon, A mathematical theoiy of communication," Be|| Sysem Tet|nta| Journa|, vol. 27, pp. 379, 623, 1948.
E. T. Whittakei, On the functions which aie iepiesented by the expansions of the inteipolation theoiy," Prot.
Roya| Sotey o[ EJn|urg|, vol. 35, pp. 181-194, 1915.
J. M. Whittakei, The Fouiiei theoiy of the caidinal functions," Prot. Ma|. Sot. EJn|urg|, vol. 1, pp. 169-176,
1929.
A. I. Zayed, Advances in Shannon`s Sampling Theoiy, Boca Raton, Fla.: CRC Piess, 1993.
Further Inlurmatiun
An in-depth study of the sample theoiem and its numeious vaiiations is piovided in R. J. Maiks II, Ed.,
InroJuton o S|annon Sam|ng anJ Inero|aon T|eory, New Yoik:Spiingei-Veilag, 1991.
In-depth studies of modein sampling theoiy with ovei 1000 iefeiences aie available in R. J. Maiks II, Ed.,
JanteJ Tots n S|annon Sam|ng anJ Inero|aon T|eory, New Yoik: Spiingei-Veilag, 1993.
The specifc case of nonunifoim sampling is tieated in the monogiaph by F. A. Maivasti, UnfeJ roat|
o Zero-Crossng anJ Nonun[orm Sam|ng, Oak Paik, Ill:Nonunifoim, 1987.
The sampling theoiem is tieated geneiically in the IEEE Transatons on Sgna| Protessng. Foi applications,
topical jouinals aie the best souice of cuiient liteiatuie.
73.5 Channe! Capacity
Sergo Verdu
Inlurmatiun Rates
Tens of millions of useis access the Inteinet daily via standaid telephone lines. Modems opeiating at data iates
of up to 28,800 bits pei second enable the tiansmission of text, audio, coloi images, and even low-iesolution
video. The piogiession in modein technology foi the standaid telephone channel shown in Fig. 73.12 exhibits,
if not the exponential incieases ubiquitous in computei engineeiing, then a steady slope of abut 825 bits pei
second pei yeai.
Few technological advances can iesult in as many time-savings foi woildwide daily life as advances in modem
infoimation iates. Howevei, modem designeis aie faced with a fundamental limitation in the maximum tians-
missible infoimation iate. Eveiy communication channel has a numbei associated with it called channel capacity,
which deteimines the maximum infoimation iate that can ow thiough the channel iegaidless of the complexity
of the tiansmitting and ieceiving devices. Thus, the piogiession of modem iates shown in Fig. 73.12 is suie to
come to a halt. But, at what iate: Answeiing this question foi any communication channel model is one of the
majoi goals of infoimation theoiy-a discipline founded in 1948 by Claude E. Shannon Shannon, 1948].
Cummunicatiun Channe!s
The communication channel is the set of devices and systems that connects the tiansmittei to the ieceivei. The
tiansmittei and ieceivei consist of an encoder and decoder, iespectively, which tianslate the infoimation stieam
pioduced by the souice into a signal suitable foi channel tiansmission and vice veisa (Fig. 73.13). Foi example,
in the case of the telephone line, two communication channels (one in each diiection) shaie the same physical
channel that connects the two modems. That physical channel usually consists of twisted coppei wiies at both
ends and a vaiiety of switching and signal piocessing opeiations that occui at the telephone exchanges. The
modems themselves aie not included in the communication channel. A miciowave iadio link is anothei example
of a communication channel that consists of an amplifei and an antenna (at both ends) and a ceitain poition
of the iadio spectium. In this case, the communication channel model does not fully coiiespond with the
physical channel. Why not, foi example, view the antenna as pait of the tiansmittei iathei than the channel
(Fig. 73.13): Because consideiations othei than the optimization of the effciency of the link aie likely to dictate
the choice of antenna size. This illustiates that the boundaiies encodei-channel and channel-decodei in
Fig. 73.13 aie not always uniquely defned. This suggests an alteinative defnition of a channel as that pait of
the communication system that the designei is unable oi unwilling to change.
A channel is chaiacteiized by the piobability distiibutions of the output signals given eveiy possible input
signal. Channels aie divided into (1) discrete-time channels and (2) continuous-time channels depending on
whethei the input/output signals aie sequences oi functions of a ieal vaiiable. Some examples aie as follows.
Examp!e 1: Binary Symmetric Channe!
A disciete-time memoryless channel with binaiy inputs and outputs (Fig. 73.14) wheie the piobabilities that
0 and 1 aie ieceived eiioneously aie equal.
Examp!e 2: Z-Channe!
A disciete-time memoiyless channel with binaiy inputs and outputs (Fig. 73.15) wheie 0 is ieceived eiioi-fiee.
Examp!e 3: Erasure Channe!
A disciete-time memoiyless channel with binaiy inputs and teinaiy outputs (Fig. 73.16). The symbols 0 and
1 cannot be mistaken foi each othei but they can be eiased".
FIGURE 73.12 Infoimation iates of modems foi telephone channels.
FIGURE 73.13 Elements of a communication system.
Examp!e 4: White Gaussian Discrete-Time Channe!
A disciete-time channel whose output sequence is given by
y
- n
(73.76)
wheie x
is the input sequence and n
is a sequence of independent
Gaussian iandom vaiiables with equal vaiiance.
Examp!e 5: Linear Cuntinuuus-Time Gaussian Channe!
A continuous-time channel whose output signal is given by (Fig.
73.17)
y() |() x() - n() (73.77)
wheie x() is the input signal, n() is a stationaiy Gaussian piocess,
and |() is the impulse iesponse of a lineai time-invaiiant system.
The telephone channel is typically modeled by Eq. (73.77).
The goal of the encodei (Fig. 73.13) is to conveit stiings of binaiy data (messages) into channel-input signals.
Souice stiings of m bits aie tianslated into channel input stiings of n symbols (with m s n) foi disciete channels,
and into continuous-time signals of duiation T foi continuous-time channels. The channel code (oi moie
piecisely the codebook) is the list of 2
m
codewords (channel input signals) that may be sent by the encodei.
The rate of the code is equal to the logaiithm of its size divided by the duiation of the codewoids. Thus, the
iate is equal to
FIGURE 73.14 Binaiy symmetiic channel. FIGURE 73.15 Z-channel.
FIGURE 73.17 Lineai continuous-time gaussian channel.
FIGURE 73.16 Eiasuie channel.
bits pei channel use
foi a disciete-time channel, wheieas it is equal to
bits pei second
foi a continuous-time channel.
Once a codewoid has been chosen by the encodei, the channel piobabilistic mechanisms govein the distoition
suffeied by the tiansmitted signal. The iole of the decodei is to iecovei the tiansmitted binaiy stiing (message)
upon ieception of the channel-distoited veision of the tiansmitted codewoid. To that end, the decodei knows
the codebook used by the encodei. Foi most channels (including those above) theie is a nonzeio piobability
that the best decodei (maximum likelihood decodei) selects the wiong message. Thus, foi a given channel the
two fguies of meiit and of inteiest aie the iate and the piobability of eiioi. The highei the toleiated piobability
of eiioi, the highei the allowed iate; howevei, computing the exact tiadeoff is a foimidable task unless the code
size eithei is veiy small oi tends to infnity. The lattei case was the one consideied by Shannon and tieated in
the following section.
Re!iab!e Inlurmatiun Transmissiun: Shannun's Theurem
Shannon 1948] consideied the situation in which the codewoid duiation giows without bound. Channel
capacity is the maximum iate foi which encodeis and decodeis exist whose piobability of eiioi vanishes as the
codewoids get longei and longei.
Shannon's Theorem Shannon, 1948] The capacity of a disciete memoiyless channel is equal to
C max
X
I (X; Y), (73.78)
wheie I(X; Y) stands foi the input-output mutual infoimation, which is a measuie of the dependence of the
input and the output defned as the diveigence between the joint input/output distiibution and the pioduct
of its maiginals, D(P
XY
P
X
P
Y
). Foi any paii of piobability mass functions P and Q defned on the same space,
diveigence is an asymmetiic measuie of theii similaiity:
D(PQ) log . (73.79)
Diveigence is zeio if both distiibutions aie equal; otheiwise it is stiictly positive. The maximization in Eq.
(73.78) is ovei the set of input distiibutions. Although, in geneial, theie is no closed-foim solution foi that
optimization pioblem, an effcient algoiithm was obtained by Blahut and Aiimoto in 1972 Blahut, 1987]. The
distiibution that attains the maximum in Eq. (73.78) deteimines the statistical behavioi of optimal codes and,
thus, is of inteiest to the designei of the encodei. Foi the disciete memoiyless channels mentioned above, the
capacity is given by the following examples.
Examp!e 1: Binary Symmetric Channe!
C 1 - log - (1 - ) log
attained by an equipiobable distiibution and shown in Fig. 73.18.
m
n
----
m
T
----
P x ( )
x A e
P x ( )
Q x ( )
------------
1
-- -
1
1
-----------
Examp!e 2: Z-Channe!
attained foi a distiibution whose piobability mass at 0 ianges fiom 1/2 ( 0) to 1/e ( 1) (Fig. 73.19).
Examp!e 3: Erasure Channe!
C 1 -
attained foi equipiobable inputs.
Oftentimes the designei is satisfed with not exceeding a ceitain fxed level of bit eiioi iate, , iathei than
the moie stiingent ciiteiion of vanishing piobability of selecting the wiong block of data. In such a case, it is
possible to tiansmit infoimation at a iate equal to capacity times
which is shown in Fig. 73.20.
If, contiaiy to what we have assumed thus fai, the message souice in Fig. 73.13 is not a souice of puie bits,
the signifcance of capacity can be extended to show that as long as the souice entiopy (see Chaptei 73.6 on
FIGURE 73.18 Capacity of the binaiy symmetiic channel as a function of ciossovei piobability.
FIGURE 73.19 Capacity of the Z-channel.
C
( )

+ log 1
1
1 1
1
1
1
1
1
1

( )

_
,
log log
Data Compiession) is below the channel capacity, an encodei/decodei paii exists that enables aibitiaiily ieliable
communication. Conveisely, if the souice entiopy is above capacity, then no such encodei/decodei paii exists.
Bandvidth and Capacity
The foiegoing foimulas foi disciete channels do not lead to the capacity of continuous-time channels such as
Example 5. We have seen that in the case of the telephone channel whose bandwidth is appioximately equal to
3 kHz, capacity is lowei bounded by 28,800 bits pei second. How does bandwidth tianslate into capacity: The
answei depends on the noise level and distiibution. Foi example, if in the channel of Example 5, the noise is
absent, capacity is infnite iegaidless of bandwidth. We can encode any amount of infoimation as the binaiy
expansion of a single scalai, which can be sent ovei the channel as the amplitude oi phase of a single sinusoid;
knowing the channel tiansfei function, the decodei can iecovei the tiansmitted scalai eiioi-fiee. Cleaily, such
a tiansmission method is not iecommended in piactice because it hinges on the non-physical scenaiio of
noiseless tiansmission.
In the simplest special case of Example 5, the noise is white, the channel has an ideal at tiansfei function
with bandwidth B (in Hz), and the input powei is limited. Then, the channel capacity is equal to
C B SNR
JB
log
2
10
0.1
bits pei second (73.80)
wheie log
2
10
0.1
0.33 and SNR
JB
is equal to the optimum signal-to-noise iatio (in dB) of a lineai estimate of
a at input signal given the channel output signal. Such an optimum signal-to-noise iatio is equal to one plus
the powei alloted to the input divided by the noise powei in the channel band, i.e.,
SNR
JB
10 log
10
.
It is inteiesting to notice that as the bandwidth giows, the channel capacity does not giow without bound. It
tends to
bits pei second
wheie log
2
e 1.44. This means that the eneigy pei bit necessaiy foi ieliable communication is equal to 0.69
times the noise powei spectial density level. When the channel bandwidth is fnite, the eneigy necessaiy to
send one bit of infoimation is stiictly laigei. The eneigy iequiied to send one bit of infoimation ieliably can
FIGURE 73.20 Capacity expansion factoi as a function of bit-eiioi-iate.
1
P
BN
0
----------
,
_
P
N
0
------
2
e log
be computed foi othei (non-Gaussian) channels even in cases wheie expiessions foi channel capacity aie not
known Veid, 1990].
When the channel tiansfei function H( [ ) and/oi noise spectial density N( [ ) aie not at, the constant in
Eq. (73.80) no longei applies. The so-called watei-flling foimula Shannon, 1949] gives the channel capacity as
wheie w is chosen so that
and
The lineai Gaussian-noise channel is a widely used model foi space communication (in the powei limited
iegion) and foi the telephone channel (in the bandwidth limited iegion). Thanks to the pievalence of digital
switching and digital tiansmission in modein telephone systems, not only signal-to-noise iatios have impioved
ovei time but the Gaussian-noise model in Example 5 becomes incieasingly questionable because quantization
is iesponsible foi a majoi component of the channel distoition. Theiefoie, futuie impiovements in modem
speeds aie expected to aiise mainly fiom fnei modeling of the channel.
Due to the effect of time-vaiying ieceived powei (fading), seveial impoitant channels fall outside the scope
of Example 5 such as high-fiequency iadio links, tiopospheiic scattei links, and mobile iadio channels.
Channe! Cuding Theurems
In infoimation theoiy, the iesults that give a foimula foi channel capacity in teims of the piobabilistic
desciiption of the channel aie known as channel coding theoiems. They typically involve two paits: an achiev-
ability pait, which shows that codes with vanishing eiioi piobability exist foi any iate below capacity; and a
conveise pait, which shows that if the code iate exceeds capacity, then the piobability of eiioi is necessaiily
bounded away fiom zeio. Shannon gave the fist achievability iesults in Shannon, 1948] foi disciete memoiy
channels. His method of pioof, latei foimalized as the method of typical sequences" (e.g., Covei and Thomas,
1991]), is based on showing that the aveiage piobability of eiioi of a code chosen at iandom vanishes with
blocklength. Othei known achievability pioofs such as Feinstein`s 1954], Gallagei`s 1968], and the method
of types Csiszai and Koinei, 1981] aie similaily non-constiuctive. The discipline of coding theoiy deals with
constiuctive methods to design codes that appioach the Shannon limit (see Chaptei 71.1). The fist conveise
channel coding theoiem was not given by Shannon, but by Fano in 1952. A decade aftei Shannon`s pioneeiing
papei, seveial authois obtained the fist channel coding theoiems foi channels with memoiy Dobiushin, 1963].
The most geneial foimula foi channel capacity known to date can be found in Veid and Han, 1994]. The
capacity of channels with feedback was fist consideied by Shannon in 1961 Shannon, 1961], with latei
developments foi Gaussian channels summaiized in Covei and Thomas, 1991]. In his 1961 papei Shannon,
1961], Shannon founded the discipline of multiusei infoimation theoiy by posing seveial challenging channels
with moie than one tiansmittei and/oi ieceivei. In contiast to the multiaccess channel (one ieceivei) which
has been solved in consideiable geneiality, the capacities of channels involving moie than one ieceivei, such as
broadcast channels Covei, 1972] and interference channels iemain unsolved except in special cases.
C
w M [
M [
J[ +

( ) }
( )
_
,
[
1
2
1
0
log
max ,
max , , 0 w M [ J[ P
( ) }

[
M [
N [
H [
( )

( )
( )
2
.
Channel capacity has been shown to have a meaning outside the domain of infoimation tiansmission Han
and Veid, 1993]: it is the minimum iate of iandom bits iequiied to geneiate any input iandom piocess so
that the output piocess is simulated with aibitiaiy accuiacy.
Dehning Terms
Blocklength: The duiation of a codewoid, usually in the context of disciete-time channels.
Channel capacity: The maximum iate foi which encodeis and decodeis exist whose piobability of eiioi
vanishes as the codewoids get longei and longei.
Codeword: Channel-input signal chosen by the encodei to iepiesent the message.
Communication channel: Set of devices and systems that connect the tiansmittei to the ieceivei, not subject
to optimization.
Broadcast channel: A communication channel with one input and seveial outputs each connected to a
diffeient ieceivei such that possibly diffeient messages aie to be conveyed to each ieceivei.
Discrete memoryless channel: A disciete-time memoiyless channel wheie each channel input and output
takes a fnite numbei of values.
Discrete-time channel: A communication channel whose input/output signals aie sequences of values. Its
capacity is given in teims of bits pei channel use".
Continuous-time channel: A communication channel whose input/output signals aie functions of a ieal
vaiiable (time). Its capacity is given in teims of bits pei second.
Interference channel: A channel with seveial inputs/outputs such that autonomous tiansmitteis aie con-
nected to each input and such that each ieceivei is inteiested in decoding the message sent by one and
only one tiansmittei.
Memoryless channel: A channel wheie the conditional piobability of the output given the cuiient input is
independent of all othei inputs oi outputs.
Multiaccess channel: A channel with seveial inputs and one output such that autonomous tiansmitteis aie
connected to each input and such that the ieceivei is inteiested in decoding the messages sent by all the
tiansmitteis.
Decoder: Mapping fiom the set of channel-output signals to the set of messages.
Maximum-likelihood decoder: A decodei which selects the message that best explains the ieceived signal,
assuming all messages aie equally likely.
Encoder: Mapping fiom the set of messages to the set of input codewoids.
Modem: Device that conveits binaiy infoimation stieams into electiical signals (and vice-veisa) foi tians-
mission thiough the voiceband telephone channel.
Rate: The iate of a code is the numbei of bits tiansmitted (logaiithm of code size) pei second foi a continuous-
time channel oi pei channel use foi a disciete-time channel.
Re!ated Tupics
70.1 Coding 73.4 The Sampling Theoiem
Relerences
C. E. Shannon, A mathematical theoiy of communication," Be|| Sys. Tet|. J., 27, 379-423, 623-656, July-Oct.
1948.
R. E. Blahut, Prnt|es o[ In[ormaon T|eory. Reading, Mass.: Addison-Wesley, 1987.
S. Veid, On channel capacity pei unit cost," IEEE Trans. In[ormaon T|eory, IT-36(5), 1019-1030, Sept.
1990.
C. E. Shannon, Communication in the piesence of noise," Prot. Insue o[ RaJo Engneers, 37, 10-21, 1949.
T. M. Covei and J. A. Thomas, E|emens o[ In[ormaon T|eory, New Yoik: Wiley, 1991.
A. Feinstein, A new basic theoiem of infoimation theoiy," IRE Trans. PCIT, pp. 2-22, 1954.
R. G. Gallagei, In[ormaon T|eory anJ Re|a||e Communtaon, New Yoik: Wiley, 1968.
I. Csiszai and J. Koinei, In[ormaon T|eory. CoJng T|eorems [or Dstree Memory|ess Sysems, New Yoik:
Academic Piess, 1981.
R. L. Dobiushin, Cenera| Formu|aon o[ S|annon's Man T|eorem n In[ormaon T|eory, Ameiican Mathemat-
ical Society Tianslations, pp. 323-438, 1963.
S. Veid and T. S. Han, A geneial foimula foi channel capacity," IEEE Trans. on In[ormaon T|eory, 40(4),
1147-1157, July 1994.
C. E. Shannon, Two-way communication channels," Prot. 4|. Ber|e|ey Sym. Ma|. Sasts anJ Pro|.,
pp. 611-644, 1961.
T. M. Covei, Bioadcast channels," IEEE Trans. on In[ormaon T|eory, pp. 2-14, Jan. 1972.
T. S. Han and S. Veid, Appioximation theoiy of output statistics," IEEE Trans. on In[ormaon T|eory, IT-39,
752-772, May 1993.
Further Inlurmatiun
The piemiei jouinal and confeience in the feld of infoimation theoiy aie the IEEE Trans. on In[ormaon
T|eory and the IEEE Inernaona| Symosum on In[ormaon T|eory, iespectively. Pro||ems o[ In[ormaon
Transmsson is a tianslation of a Russian-language jouinal in infoimation theoiy. The newslettei of the IEEE
Infoimation Theoiy Society iegulaily publishes expositoiy aiticles.
73.6 Data Cumpressiun
joy A. T|omo ond T|omo M. Cover
Data compiession is a piocess of fnding the most effcient iepiesentation of an infoimation souice in oidei
to minimize communication oi stoiage. It often consists of two stages-the fist is the choice of a (piobabilistic)
model foi the souice and the second is the design of an effcient coding system foi the model. In this section,
we will concentiate on the second aspect of the compiession piocess, though we will touch on some common
souices and models in the last subsection.
Thus, a data compiessoi (sometimes called a souice codei) maps an infoimation souice into a sequence of
bits, with a coiiesponding decompiessoi, that given these bits piovides a ieconstiuction of the souice. Data
compiession systems can be classifed into two types: |oss|ess, wheie the ieconstiuction is exactly equal to the
oiiginal souice, and |ossy, wheie the ieconstiuction is a distoited veision of the oiiginal souice. Foi lossless
data compiession, the fundamental lowei bound on the iate of the data compiession system is given by the
entiopy iate of the souice. Foi lossy data compiession, we have a tiadeoff between the iate of the compiessoi
and the distoition we incui, and the fundamental limit is given by the iate distoition function, which is discussed
latei in this section.
Shannon 1948] was the fist to distinguish the piobabilistic model that undeilies an infoimation souice
fiom the semantics of the infoimation. An infoimation souice pioduces one of many possible messages; the
goal of communication is to tiansmit an unambiguous specifcation of the message so that the ieceivei can
ieconstiuct the oiiginal message. Foi example, the infoimation to be sent may be the iesult of a hoise iace. If
the iecipient is assumed to know the names and numbeis of the hoises, then all that must be tiansmitted is
the numbei of the hoise that won. In a diffeient context, the same numbei might mean something quite
diffeient, e.g., the piice of a baiiel of oil. The signifcant fact is that the diffculty in communication depends
only on the length of the iepiesentation. Thus, fnding the best (shoitest) iepiesentation of an infoimation
souice is ciitical to effcient communication.
When the possible messages aie all equally likely, then it makes sense to iepiesent them by stiings of equal
length. Foi example, if theie aie 32 possible equally likely messages, then each message can be iepiesented by
a binaiy stiing of 5 bits. Howevei, if the messages aie not equally likely, then it is moie effcient on the aveiage
to allot shoit stiings to the fiequently occuiiing messages and longei stiings to the iaie messages. Thus, the
Moise code allots the shoitest stiing (a dot) to the most fiequent lettei (E) and allots long stiings to the
infiequent letteis (e.g., dash, dash, dot, dash foi Q). The minimum aveiage length of the iepiesentation is a
fundamental quantity called the entiopy of the souice, which is defned in the next subsection.
Entrupy
An infoimation souice will be iepiesented by a iandom vaiiable X, which takes on one of a fnite numbei of
possibilities e with piobability

Pi(X ). The entiopy of the iandom vaiiable X is defned as
(73.81)
wheie the log is to base 2 and the entiopy is measuied in |s. We will use logaiithms to base 2 thioughout this
chaptei.
Example 73.1 Let X be a iandom vaiiable that takes on a value 1 with piobability 0 and takes on the value
0 with piobability 1 - 0. Then H(X) -0 log 0 - (1 - 0) log (1 - 0). In paiticulai, the entiopy of a faii coin
toss with
1
/2 is 1 bit.
This defnition of entiopy is ielated to the defnition of entiopy in theimodynamics. It is the fundamental
lowei bound on the aveiage length of a code foi the iandom vaiiable.
A code foi a iandom vaiiable X is a mapping fiom , the iange of X, to the set of fnite-length binaiy stiings.
We will denote the code woid coiiesponding to by C(), and the length of the code woid by |
. The aveiage
length of the code is then L(C) l
.
A code is said to be nsananeous oi refx-[ree if no code woid is a piefx of any othei code woid. This
condition is suffcient (but not necessaiy) to allow a sequence of ieceived bits to be paised unambiguously into
a sequence of code woids.
Example 73.2 Considei a iandom vaiiable X taking on the values {1, 2, 3 with piobabilities (0.5, 0.25,
0.25). An instantaneous code foi this iandom vaiiable might be (0, 10, 11). Thus, a stiing 01001110 can be
uniquely paised into 0, 10, 0, 11, 10, which decodes to the stiing x (1, 2, 1, 3, 2). Note that the aveiage
length of the code is 1.5 bits, which is the same as the entiopy of the souice.
Foi any instantaneous code, the following piopeity of binaiy tiees called the Kra[ nequa|y Covei and
Thomas, 1991].
(73.82)
must hold. Conveisely, it can be shown that given a set of lengths that satisfes the Kiaft inequality, we can fnd
a set of piefx-fiee code woids of those lengths.
The pioblem of fnding the best souice code then ieduces to fnding the optimal set of lengths that satisfes
the Kiaft inequality and minimizes the aveiage length of the code. Simple calculus can then be used to show
Covei and Thomas, 1991] that the aveiage length of any instantaneous code is laigei than the entiopy of the
iandom vaiiable, i.e., the minimum of l
ovei all |
satisfying l 2
-|
s 1 is -l
log
. Also, if we take |

log 1/
1 (wheie 1 denotes the smallest integei gieatei than oi equal to ), we can veiify that this choice of
lengths satisfes the Kiaft inequality and that
(73.83)
The optimal code can only have a shoitei length, and theiefoie we have the following theoiem:
Theorem 73.1 Let L- be the aveiage length of the optimal instantaneous code foi a iandom vaiiable X. Then
H(X) s L- < H(X) - 1 (73.84)
H X

( ) log
e
2 1
s
|
L C
H X

( ) log log ( )

1
1
1
1
< +
_
,
+

1 1
1 1
This theoiem is one of the fundamental theoiems of infoimation theoiy. It identifes the entiopy as the
fundamental limit foi the aveiage length of the iepiesentation of a disciete infoimation souice and shows that
we can fnd iepiesentations with aveiage length within one bit of the entiopy.
The Hullman A!gurithm
The choice of code woid lengths |
log 1/
1 (called the S|annon toJe |eng|s) is close to optimal, but not

necessaiily optimal, in teims of aveiage code woid length. We will now desciibe an algoiithm (the Huffman
algorithm) that pioduces an instantaneous code of minimal aveiage length foi a iandom vaiiable with distii-
bution
1
,
2
, . . .,
m
. The algoiithm is a gieedy algoiithm foi building a tiee fiom the bottom up.
Step 1. Aiiange the piobabilities in decieasing oidei so that
1
>
2
> . . . >
m
.
Step 2. Foim a subtiee by combining the last two piobabilities
m-1
and
m
to a single node of weight '
m-1

m-1
-
m
.
Step 3. Recuisively execute Steps 1 and 2, decieasing the numbei of nodes each time, until a single node is
obtained.
Step 4. Use the tiee constiucted above to allot code woids.
The algoiithm foi tiee constiuction is illustiated foi a souice with distiibution (0.5, 0.2, 0.2, 0.1) in Fig. 73.21.
Aftei constiucting the tiee, the leaves of the tiee (which coiiespond to the symbols of ) can be assigned code
woids that coiiespond to the paths fiom the ioot to the leaf. We will not give a pioof of the optimality of the
Huffman algoiithm; the ieadei is iefeiied to Gallagei 1968] oi Covei and Thomas 1991] foi details.
Entrupy Rate
The entiopy of a sequence of iandom vaiiables X
1
, X
2
, . . ., X
n
with joint distiibution (x
1
, x
2
, . . ., x
n
) is defned
analogously to the entiopy of a single iandom vaiiable as
(73.85)
Foi a stationaiy piocess X
1
, X
2
, . . ., we defne the enroy rae () of the piocess as
(73.86)
It can be shown Covei and Thomas, 1991] that the entiopy iate is well defned foi all stationaiy piocesses. In
paiticulai, if X
1
, X
2
, . . ., X
n
is a sequence of independent and identically distiibuted (i.i.d.) iandom vaiiables,
then H(X
1
, X
2
, . . ., X
n
) nH(X
1
), and () H(X
1
).
FIGURE 73.21 Example of the Huffman algoiithm.
H X X X x x x x x x
n
x x x
n n
n
( , , . . . , ) . . . ( , , . . . , )log ( , , . . . , )
1 2 1 2 1 2
2 1

( ) lim
( , , . . . , )
~ n
n
H X X X
n
1 2
In the pievious subsection, we showed the existence of a piefx-fiee code having an aveiage length within
one bit of the entiopy. Now instead of tiying to iepiesent one occuiience of the iandom vaiiable, we can foim
a code to iepiesent a block of n iandom vaiiables. In this case, the aveiage code length is within one bit of
H(X
1
, X
2
, . . ., X
n
). Thus, the aveiage length of the code pei input symbol satisfes
(73.87)
Since H(X
1
, X
2
, . . ., X
n
)]/n (), we can get aibitiaiily close to the entiopy iate by using longei and longei
block lengths. Thus, the entiopy iate is the fundamental limit foi data compiession foi stationaiy souices, and
we can achieve iates aibitiaiily close to this limit by using long blocks.
All the above assumes that we know the piobability distiibution that undeilies the infoimation souice. In
many piactical examples, howevei, the distiibution is unknown oi too complex to be used foi coding. Theie
aie vaiious ways to handle this situation:
Assume a simple distiibution and design an appiopiiate code foi it. Use this code on the ieal souice. If
an estimated distiibution
is used when in fact the tiue distiibution is , then the aveiage length of the
code is lowei bounded by H(X) + l
x
(x) log (x)/
(x)]. The second teim, which is denoted D(
) is
called the re|ae enroy oi the Ku|||at| Le||er Jsante between the two distiibutions.
Estimate the distiibution empiiically fiom the souice and adapt the code to the distiibution. Foi example,
with aJae Hu[[man toJng, the empiiical distiibution of the souice symbols is used to design the
Huffman code used foi the souice.
Use a unersa| toJng a|gor|m like the Lempel-Ziv algorithm (see the subsection Lempel-Ziv Coding").
Arithmetic Cuding
In the pievious subsections, it was shown how we could constiuct a code foi a souice that achieves an aveiage
length within one bit of the entiopy. Foi small souice alphabets, howevei, we have effcient coding only if we
use long blocks of souice symbols. Foi example, if the souice is binaiy, and we code each symbol sepaiately,
we must use 1 bit pei symbol, iiiespective of the entiopy of the souice. If we use long blocks, we can achieve
an expected length pei symbol close to the entiopy iate of the souice.
It is theiefoie desiiable to have an effcient coding pioceduie that woiks foi long blocks of souice symbols.
Huffman coding is not ideal foi this situation, since it is a bottom-up pioceduie with a complexity that giows
iapidly with the block length. Aiithmetic coding is an inciemental coding algoiithm that woiks effciently foi
long block lengths and achieves an aveiage length within one bit of the entiopy foi the block.
The essential idea of aiithmetic coding is to iepiesent a sequence x
n
x
1
, x
2
, . . ., x
n
by the cumulative
distiibution function F(x
n
) (the sum of the piobability of all sequences less than x
n
) expiessed to an appiopiiate
accuiacy. The cumulative distiibution function foi x
n
is illustiated in Fig. 73.22. We can use any ieal numbei
in the inteival F(x
n
) - (x
n
), F(x
n
)] as the code foi x
n
. Expiessing F(x
n
) to an accuiacy of log 1/(x
n
)1 will
give us a code foi the souice. The ieceivei can diaw the cumulative distiibution function, diaw a hoiizontal
line coiiesponding to the tiuncated value F(x
n
)] that was sent, and iead off the coiiesponding x
n
. (This code
is not piefx-fiee but can be easily modifed to constiuct a piefx-fiee code Covei and Thomas, 1991]). To
implement aiithmetic coding, howevei, we need effcient algoiithms to calculate (x
n
) and F(x
n
) to the appio-
piiate accuiacy based on a piobabilistic model foi the souice. Details can be found in Langdon 1984] and Bell
et al. 1990].
Lempe!~Ziv Cuding
The Lempel-Ziv algoiithm Ziv and Lempel, 1978] is a univeisal coding pioceduie that does not iequiie
knowledge of the souice statistics and yet is asymptotically optimal. The basic idea of the algoiithm is to
constiuct a table oi dictionaiy of fiequently occuiiing stiings and to iepiesent new stiings by pointing to theii
piefxes in the table. We fist paise the stiing into sequences that have not appeaied so fai. Foi example, the
H X X X
n
L
n
H X X X
n n
n n n
( , , . . . , ) ( , , . . . , )
1 2 1 2
1
s < +

binaiy stiing 11010011011100 is paised into 1,10,100,11,0,111,00. Then instead of sending the bits of each
phiase, we send a pointei to its piefx and the value of the last bit. Thus, if we use thiee bits foi the pointei,
we will iepiesent this stiing by (000,1), (001,0), (010,0), (001,1), (000,0), (100,1), (101,0), etc. Foi this shoit
example, the algoiithm has not compiessed the stiing-it has in fact expanded it.
The veiy suipiising fact is that, as Lempel and Ziv have shown, the algoiithm is asymptotically optimal foi
any stationaiy eigodic souice. This is expiessed in the following theoiem Ziv and Lempel, 1978; Covei and
Thomas, 1991]:
Theorem 73.2 Let L
n
be the length of the Lempel-Ziv code foi n symbols diawn fiom a stationaiy eigodic
piocess X
1
, X
2
, . . ., X
n
with entiopy iate (). Then
(73.88)
Thus, foi long enough block lengths, the Lempel-Ziv algoiithm (which does not make any assumptions
about the distiibution of the souice) does as well as if we knew the distiibution in advance and designed the
optimal code foi this distiibution.
The algoiithm desciibed above is only one of a laige class of similai adaptive dictionaiy-based algoiithms,
which aie all iathei loosely called Lempel-Ziv. These algoiithms aie simple and fast and have been implemented
in both softwaie and haidwaie, e.g., in the tomress command in UNIX and the PKZIP command on PCs. On
ASCII text fles, the Lempel-Ziv algoiithm achieves compiessions on the oidei of 50%. It has also been
implemented in haidwaie and has been used to double" the capacity of data stoiage media oi to double" the
effective tiansmission iate of a modem. Many vaiiations on the basic algoiithm can be found in Bell et al. 1990].
Rate Disturtiun Theury
An infnite numbei of bits aie iequiied to desciibe an aibitiaiy ieal numbei, and theiefoie it is not possible
to peifectly iepiesent a continuous iandom vaiiable with a fnite numbei of bits. How good" can the iepie-
sentation be: We fist defne a distoition measuie, which is a measuie of the distance between the iandom
vaiiable and its iepiesentation. We can then considei the tiade-off between the numbei of bits used to iepiesent
a iandom vaiiable and the distoition incuiied. This tiade-off is iepiesented by the rate distortion function
R(D), which iepiesents the minimum iate iequiied to iepiesent a iandom vaiiable with a distoition D.
We will considei a disciete infoimation souice that pioduces iandom vaiiables X
1
, X
2
, . . ., X
n
that aie diawn
i.i.d. accoiding to (x). (The iesults aie also valid foi continuous souices.) The encodei of the iate distoition
FIGURE 73.22 Cumulative distiibution function foi sequences x
n
.
L
n
n
( ) with piobability 1
system of iate R will encode a block of n outputs X
n
as an index [ (X
n
) e {1, 2, . . ., 2
n R
]. (Thus, the index will
iequiie R bits/input symbol.) The decodei will calculate a iepiesentation

X
n
([ (X
n
)) of X
n
. Noimally, the iepie-
sentation alphabet

X of the iepiesentation is the same as the souice alphabet , but that need not be the case.
Dennition: A Jsoron [unton oi Jsoron measure is a mapping
(73.89)
fiom the set of souice alphabet-iepioduction alphabet paiis into the set of nonnegative ieal numbeis. The
distoition J(x, x) is a measuie of the cost of iepiesenting the symbol x by the symbol x.
Examples of common distoition functions aie
Hammng (ro|a||y o[ error) Jsoron. The Hamming distoition is given by
(73.90)
and thus EJ(X,` X) Pi(X =

X).
SquareJ error Jsoron. The squaied eiioi distoition
(73.91)
is the most populai distoition measuie used foi continuous alphabets. Its advantages aie its simplicity
and its ielationship to least squaies piediction. Howevei, foi infoimation souices such as images and
speech, the squaied eiioi is not an appiopiiate measuie foi distoition as peiceived by a human obseivei.
T|e Jsoron |eween sequentes x
n
and x
n
of length n is defned by
(73.92)
Foi a iate distoition system, the expected distoition D is defned as
(73.93)
Dennition: The iate distoition paii (R,D) is said to be achievable if theie exists a iate distoition code of iate
R with expected distoition D. The rae Jsoron [unton R(D) is the infmum of iates R such that (R,D) is
achievable foi a given D.
Dennition: The muua| n[ormaon I(X,

X) between iandom vaiiables X and

X, with joint piobability mass
function (x, x) and maiginal piobability mass functions (x) and (x) is defned as
(73.94)
The mutual infoimation is a measuie of the amount of infoimation that one iandom vaiiable caiiies about
anothei.
J R :
`

+
J x x
x x
x x
( ,
`
)
`
`
0
1
if
if
J x x x x ( ,
`
) (
`
)
2
J x x
n
J x x
n n

n
( ,
`
) ( ,
`
)
1
1
D EJ X X [ X x J x X [ x
n n n n n n n
x
n

( ,
`
( ( ))) ( ) ( ,
`
( ( )))
I X X x x
x x
x x
x x
( ;
`
) ( ,
`
) log
( ,
`
)
( ) (
`
)
` `
e e

The main iesult of iate distoition theoiy is contained in the following theoiem, which piovides a chaiac-
teiization of the iate distoition function in teims of the mutual infoimation of joint distiibutions that satisfy
the expected distoition constiaint:
Theorem 73.3 The iate distoition function foi an i.i.d. souice X with distiibution (x) and distoition
function J(x, x) is
(73.95)
We can constiuct iate distoition codes that can achieve distoition D at any iate gieatei than R(D), and we
cannot constiuct such codes at any iate below R(D).
The pioof of this theoiem uses ideas of iandom coding and long block lengths as in the pioof of the channel
capacity theoiem. The basic idea is to geneiate a code book of 2
nR
iepioduction code woids

X
n
at iandom and
show that foi long block lengths, foi any souice sequence, it is veiy likely that theie is at least one code woid
in this code book that is within distoition D of that souice sequence. See Gallagei 1968] oi Covei and Thomas
1991] foi details of the pioof.
Example 73.3 (Bnary sourte) The iate distoition function foi a Beinoulli () souice (a iandom vaiiable
that takes on values {0, 1 with piobabilities , 1 - ) with Hamming distoition is given by
(73.96)
wheie H() - log - (1 - )log (1 - ) is the binaiy entiopy function.
Example 73.4 (Caussan sourte) The iate distoition function foi a Gaussian iandom vaiiable with vaiiance
o
2
and squaied eiioi distoition is
(73.97)
Thus, with nR bits, we can desciibe n i.i.d. Gaussian iandom vaiiables X
1
, X
2
, ..., X
n
(0, o
2
) with a
distoition of o
2
2
-2R
pei symbol.
Quantizatiun and Yectur Quantizatiun
The iate distoition function iepiesents the lowei bound on the iate that is needed to iepiesent a souice with
a paiticulai distoition. We now considei simple algoiithms that iepiesent a continuous iandom vaiiable with
a few bits. Suppose we want to iepiesent a single sample fiom a continuous souice. Let the iandom vaiiable
to be iepiesented be X and let the iepiesentation of X be denoted as

X(X). If we aie given R bits to iepiesent
X, then the function

X can take on 2
R
values. The pioblem of optimum quantization is to fnd the optimum
set of values foi

X (called the iepioduction points oi code points) and the iegions that aie associated with each
value

X in oidei to minimize the expected distoition.
Foi example, let X be a Gaussian iandom vaiiable with mean 0 and vaiiance o
2
, and assume a squaied eiioi
distoition measuie. In this case, we wish to fnd the function

X(X) such that

X takes on at most 2
R
values and
minimizes E(X -

X(X))
2
. If we aie given 1 bit to iepiesent X, it is cleai that the bit should distinguish whethei
X > 0 oi not. To minimize squaied eiioi, each iepioduced symbol should be at the conditional mean of its
iegion. If we aie given 2 bits to iepiesent the sample, the situation is not as simple. Cleaily, we want to divide
the ieal line into foui iegions and use a point within each iegion to iepiesent the samples within that iegion.
R D I X X
x x x x x J x x D
x x
( ) min ( ;
`
)
( ` ): ( ) ( ` ) ( , ` )
( , ` )
s Z
R D
H H D D
D
( )
( ) ( ), min{ ,
, min{ ,
s s
>
0 1
0 1
R D
D
D
D
( )
log ,
,
s s
>
1
2
2
2
2
0
0
o
o
o
We can state two simple piopeities of optimal iegions and ieconstiuction points foi the quantization of a single
iandom vaiiable:
Given a set of ieconstiuction points, the distoition is minimized by mapping a souice iandom vaiiable
X to the iepiesentation X
`
(w) that is closest to it (in distoition). The set of iegions defned by this mapping
is called a Voionoi oi Diiichlet paitition defned by the ieconstiuction points.
The ieconstiuction points should minimize the conditional expected distoition ovei theii iespective
assignment iegions.
These two piopeities enable us to constiuct a simple algoiithm to fnd a good" quantizei: we stait with a
set of ieconstiuction points, fnd the optimal set of ieconstiuction iegions (which aie the neaiest neighboi
iegions with iespect to the distoition measuie), then fnd the optimal ieconstiuction points foi these iegions
(the centioids of these iegions if the distoition measuie is squaied eiioi), and then iepeat the iteiation foi this
new set of ieconstiuction points. The expected distoition is decieased at each stage in the algoiithm, so the
algoiithm will conveige to a local minimum of the distoition. This algoiithm is called the L|oyJ a|gor|m
Geisho and Giay, 1992].
It follows fiom the aiguments of iate distoition theoiy that we will do bettei if we encode long blocks of
souice symbols iathei than encoding each symbol individually. In this case, we will considei a block of n
symbols fiom the souice as a vectoi-valued iandom vaiiable, and we will iepiesent these n-dimensional vectois
by a set of 2
nR
code woids. This piocess is called vector quantization (VQ). We can apply the Lloyd algoiithm
to design a set of iepiesentation vectois (the code book) and the coiiesponding neaiest neighboi iegions.
Instead of using the piobability distiibution foi the souice to calculate the centioids of the iegions, we can use
the empiiical distiibution fiom a tiaining sequence. Many vaiiations of the basic vectoi quantization algoiithm
aie desciibed in Geisho and Giay 1992].
Common infoimation souices like speech pioduce continuous wavefoims, not disciete sequences of iandom
vaiiables as in the models we have been consideiing so fai. By sampling the signal at twice the maximum
fiequency piesent (the Nyquist iate), howevei, we conveit the continuous time signal into a set of disciete
samples fiom which the oiiginal signal can be iecoveied (the sampling theoiem). We can then apply the theoiy
of iate distoition and vectoi quantization to such wavefoim souices as well.
Ku!muguruv Cump!exity
In the 1960s, the Russian mathematican Kolmogoiov consideied the question What is the intiinsic desciiptive
complexity of a binaiy stiing:" Fiom the pieceding discussion, it follows that if the binaiy stiing weie a sequence
of i.i.d. iandom vaiiables X
1
, X
2
, . . ., X
n
, then on the aveiage it would take nH(X) bits to iepiesent the sequence.
But what if the bits weie the fist million bits of the binaiy expansion of r: In that case, the stiing appeais
iandom but can be geneiated by a simple computei piogiam. So if we wanted to send these million bits to
anothei location which has a computei, we could instead send the piogiam and ask the computei to geneiate
these million bits. Thus, the desciiptive complexity of r is quite small.
Motivated by such consideiations, Kolmogoiov defned the complexity of a binaiy stiing to be the length of
the shoitest piogiam foi a univeisal computei that geneiates that stiing. (This concept was also pioposed
independently and at about the same time by Chaitin and Solomonoff.)
Dennition: The Kv|mvgvrvv cvmp|erIty K
(x) of a stiing x with iespect to a univeisal computei is

defned as
(73.98)
the minimum length ovei all piogiams that piint x and halt. Thus K
(x) is the shoitest desciiption length

of x ovei all desciiptions inteipieted by computei .
A univeisal computei can be thought of as a Tuiing machine that can simulate any othei univeisal computei.
At fist sight, the defnition of Kolmogoiov complexity seems to be useless, since it depends on the paiticulai
K x |
x
( ) min ( )
: ( )

computei that we aie talking about. But using the fact that any univeisal computei can simulate any othei
univeisal computei, any piogiam foi one computei can be conveited to a piogiam foi anothei computei by
adding a constant length simulation piogiam" as a piefx. Thus, we can show Covei and Thomas, 1991] that
foi any two univeisal computeis, and ,
(73.99)
wheie the constant t, though laige, does not depend on the stiing x undei consideiation. Thus, Kolmogoiov
complexity is univeisal in that it does not depend on the computei (up to a constant additive factoi).
Kolmogoiov complexity piovides a unifed way to think about pioblems of data compiession. It is also the
basis of piinciples of infeience (Occam`s iazoi: The simplest explanation is the best") and is closely tied with
the theoiy of computability.
Data Cumpressiun in Practice
The pievious subsections discussed the fundamental limits to compiession foi a stochastic souice. We will now
considei the application of these algoiithms to some piactical souices, namely, text, speech, images, and video.
In ieal applications, the souices may not be stationaiy oi eigodic, and the distiibutions undeilying the souice
aie often unknown. Also, in addition to the effciency of the algoiithm, impoitant consideiations in piactical
applications include the computational speed and memoiy iequiiements of the algoiithm, the peiceptual quality
of the iepioductions to a human obseivei, etc. A consideiable amount of ieseaich and engineeiing has gone
into the development of these algoiithms, and many issues aie only now being exploied. We will not go into
the details but simply list some populai algoiithms foi the diffeient souices.
Text
English text is noimally iepiesented in ASCII, which uses 8 bits/chaiactei. Theie is consideiable iedundancy
in this iepiesentation (the entiopy iate of English is about 1.3 bits/chaiactei). Populai compiession algo-
iithms include vaiiants of the Lempel-Ziv algoiithm, which compiess text fles by about 50% (to about
4 bits/chaiactei).
Speech
Telephone quality speech is noimally sampled at 8 kHz and quantized at 8 bits/sample (a iate of 64 kbits/s)
foi uncompiessed speech. Simple compiession algoiithms like adaptive diffeiential pulse code modulation
(ADPCM) Jayant and Noll, 1984] use the coiielation between adjacent samples to ieduce the numbei of bits
used by a factoi of two to foui oi moie with almost impeiceptible distoition. Much highei compiession iatios
can be obtained with algoiithms like lineai piedictive coding (LPC), which model speech as an autoiegiessive
piocess, and send the paiameteis of the piocess as opposed to sending the speech itself. With LPC-based
methods, it is possible to code speech at less than 4 kbits/s. At veiy low bit iates, howevei, the iepioduced
speech sounds synthetic.
Images
A single high-quality coloi image of 1024 by 1024 pixels with 24 bits pei pixel iepiesents about 3 MB of stoiage
in an uncompiessed foim, which will take moie than 14 minutes to tiansmit ovei a 28800-baud modem. It is
theiefoie veiy impoitant to use compiession to save stoiage and communication capacity foi images. Many
diffeient algoiithms have been pioposed foi image compiession, and standaids aie still being developed foi
compiession of images. Foi example, the populai GIF standaid uses a patented veision of Lempel-Ziv coding,
and the JPEG standaid being developed by the Joint Photogiaphic Expeits Gioup uses an 8 by 8 disciete cosine
tiansfoim (DCT) followed by quantization (the quality of which can be chosen by the usei) and Huffman
coding. Newei compiession algoiithms using wavelets oi fiactals offei highei compiession than JPEG. The
compiession iatios achieved by these algoiithms aie veiy dependent on the image being coded. The lossless
compiession methods achieve compiession iatios of up to about 3:1, wheieas lossy compiession methods
achieve iatios up to 50:1 with veiy little peiceptible loss of quality.
K x K x t

( ) ( ) <
Yideu
Video compiession methods exploit the coiielation in both space and time of the sequence of images to impiove
compiession. Theie is a veiy high coiielation between successive fiames of a video signal, and this can be
exploited along with methods similai to those used foi coding images to achieve compiession iatios up to 200:1
foi high-quality lossy compiession. Standaids foi full-motion video and audio compiession aie being developed
by the Moving Pictuies Expeits Gioup (MPEG). Applications of video compiession techniques include video-
confeiencing, multimedia CD-ROMs, and high-defnition TV.
A fascinating and veiy ieadable intioduction to diffeient souices of infoimation, theii entiopy iates, and
diffeient compiession algoiithms can be found in the book by Lucky 1989]. Implementations of populai data
compiession algoiithms including adaptive Huffman coding, aiithmetic coding, Lempel-Ziv and the JPEG
algoiithm can be found in Nelson and Gailly 1995].
Dehning Terms
Code: A mapping fiom a set of messages into binaiy stiings.
Entropy: A measuie of the aveiage unceitainty of a iandom vaiiable. Foi a iandom vaiiable with piobability
distiibution (x), the entiopy H(X) is defned as l
x
- (x) log (x).
Huffman coding: A pioceduie that constiucts the code of minimum aveiage length foi a iandom vaiiable.
Kolmogorov complexity: The minimum length desciiption of a binaiy stiing that would enable a univeisal
computei to ieconstiuct the stiing.
Lempel-Ziv coding: A dictionaiy-based pioceduie foi coding that does not use the piobability distiibution
of the souice and is nonetheless asymptotically optimal.
Quantization: A piocess by which the output of a continuous souice is iepiesented by one of a set of disciete
points.
Rate distortion function: The minimum iate at which a souice can be desciibed to within a given aveiage
distoition.
Vector quantization: Quantization applied to vectois oi blocks of outputs of a continuous souice.
Re!ated Tupics
17.1 Digital Image Piocessing 69.5 Digital Audio Bioadcasting
Relerences
T. Bell, J. Cleaiy, and I. Witten, Tex Comresson, Englewood Cliffs, N.J.: Pientice-Hall, 1990.
T. M. Covei and J. A. Thomas, E|emens o[ In[ormaon T|eory, New Yoik: Wiley, 1991.
R. Gallagei, In[ormaon T|eory anJ Re|a||e Communtaon, New Yoik: Wiley, 1968.
A. Geisho and R. Giay, Vetor Quan:aon anJ Sourte CoJng, Boston: Kluwei Academic, 1992.
N. Jayant and P. Noll, Dga| CoJng o[ Vae[orms. Prnt|es anJ |taons o Seet| anJ VJeo, Englewood
Cliffs, N.J.: Pientice-Hall, 1984.
G. Langdon, An intioduction to aiithmetic coding," IBM Journa| o[ Researt| anJ Dee|omen, vol. 28,
pp. 135-149, 1984.
R. Lucky, S|ton Dreams. In[ormaon, Man anJ Mat|ne, New Yoik: St. Maitin`s Piess, 1989.
M. Nelson and J. Gailly, T|e Daa Comresson Boo|, 2nd ed., San Mateo, Calif.: M & T Books, 1995.
C. E. Shannon, A mathematical theoiy of communication," Be|| Sys. Tet|. Journa|, vol. 27, pp. 379-423,
623-656, 1948.
J. Ziv and A. Lempel, Compiession of individual sequences by vaiiable iate coding," IEEE Trans. In[orm. T|eory,
vol. IT-24, pp. 530-536, 1978.
Further Inlurmatiun
Discussion of vaiious data compiession algoiithms foi souices like speech and images can be found in the IEEE
Transatons on Communtaons and the IEEE Transatons on Sgna| Protessng, while the theoietical undei-
pinnings of compiession algoiithms aie discussed in the IEEE Transatons on In[ormaon T|eory.
Some of the latest developments in the aieas of speech and image coding aie desciibed in a special issue of
the IEEE Journa| on Se|eteJ reas n Communtaons, June 1992. It includes an excellent suivey by N.S. Jayant
of cuiient woik on signal compiession, including vaiious data compiession standaids.
Special issues of the IEEE ProteeJngs in June 1994 and Febiuaiy 1995 also covei some of the iecent
developments in data compiession and image and video coding.
A good staiting point foi cuiient infoimation on compiession on the Woild Wide Web is the FAQ foi the
newsgioup comp.compiession, which can be found at
http://www.cis.ohio-state.edu/hypeitext/faq/usenet/compiession-faq/top.html.

Informatii Teorie

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Informatii Teorie

Uploaded by

Copyright:

Available Formats

Poor, H.V., Looney, C.G., Marks II, R.J., Verd, S., Thomas, J.A., Cover, T.M.

2000 by CRC Press LLC

at any time instant is chosen by the outcome of the iandom

EN()]. The setonJ-orJer sast is the auotorre|aon [unton R

is constant foi all , and (2) R

N() is Gaussian (also called norma|). The fist- and

V() is Gaussian. When two Gaussian iandom vaiiables V

of a signal X() at any iesults in a measuied value X

x that contains eiioi, and so is

() captuied at sampling time by

is coded into the neaiest lowei binaiy value x

e() at any time is

with a piobability distiibution that deteimines the ielative

assumes values. The statistics of the family of iandom vaiiables {N

deteimined by the actual outcomes {x

of values fiom a iandom

at each instant . A tiajectoiy is also called a sam|e [unton of X().

(a) appioximates the piobability P( s a - (r/2)] - Pa - (r/2)< ].

, of a iv A is the fist moment of the pdf, and the arante of is the

: e T, indexed on an oideied set T

: e T of outcomes ovei all e T, wheie X

2000 by CRC Press LLC

: e T, wheie T is an oideied set such as the ieal numbeis oi

(), is a bandlimited function. The caidinal seiies

is the input sequence and n

1 (called the S|annon toJe |eng|s) is close to optimal, but not

(x)]. The second teim, which is denoted D(

2000 by CRC Press LLC

(x) of a stiing x with iespect to a univeisal computei is

(x) is the shoitest desciiption length

You might also like