Professional Documents
Culture Documents
The quantised degrees-of-freedom in a continuous signal. Why a continuous signal of finite bandwidth and duration has a fixed number of degrees-of-freedom. Diverse
illustrations of the principle that information, even in such a signal, comes in quantised,
countable, packets.
Gabor-Heisenberg-Weyl uncertainty relation. Optimal Logons. Unification of
the time-domain and the frequency-domain as endpoints of a continuous deformation. The
Uncertainty Principle and its optimal solution by Gabors expansion basis of logons.
Multi-resolution wavelet codes. Extension to images, for analysis and compression.
Kolmogorov complexity. Minimal description length. Definition of the algorithmic
complexity of a data sequence, and its relation to the entropy of the distribution from
which the data was drawn. Fractals. Minimal description length, and why this measure of
complexity is not computable.
Objectives
At the end of the course students should be able to
calculate the information content of a random variable from its probability distribution
relate the joint, conditional, and marginal entropies of variables in terms of their coupled
probabilities
define channel capacities and properties using Shannons Theorems
construct efficient codes for data on imperfect communication channels
generalise the discrete concepts to continuous signals on continuous channels
understand Fourier Transforms and the main ideas of efficient algorithms for them
describe the information resolution and compression properties of wavelets
Recommended book
* Cover, T.M. & Thomas, J.A. (1991). Elements of information theory. New York: Wiley.
Dennis GABOR (1900-1979) crystallized Hartleys insight by formulating a general Uncertainty Principle for information, expressing
the trade-off for resolution between bandwidth and time. (Signals
that are well specified in frequency content must be poorly localized
in time, and those that are well localized in time must be poorly
specified in frequency content.) He formalized the Information Diagram to describe this fundamental trade-off, and derived the continuous family of functions which optimize (minimize) the conjoint
uncertainty relation. In 1974 Gabor won the Nobel Prize in Physics
for his work in Fourier optics, including the invention of holography.
Claude SHANNON (together with Warren WEAVER) in 1949 wrote
the definitive, classic, work in information theory: Mathematical
Theory of Communication. Divided into separate treatments for
continuous-time and discrete-time signals, systems, and channels,
this book laid out all of the key concepts and relationships that define the field today. In particular, he proved the famous Source Coding Theorem and the Noisy Channel Coding Theorem, plus many
other related results about channel capacity.
S KULLBACK and R A LEIBLER (1951) defined relative entropy
(also called information for discrimination, or K-L Distance.)
E T JAYNES (since 1957) developed maximum entropy methods
for inference, hypothesis-testing, and decision-making, based on the
physics of statistical mechanics. Others have inquired whether these
principles impose fundamental physical limits to computation itself.
A N KOLMOGOROV in 1965 proposed that the complexity of a
string of data can be defined by the length of the shortest binary
program for computing the string. Thus the complexity of data
is its minimal description length, and this specifies the ultimate
compressibility of data. The Kolmogorov complexity K of a string
is approximately equal to its Shannon entropy H, thereby unifying
the theory of descriptive complexity and information theory.
4
combine our prior knowledge about physics, geology, and dairy products. Yet the frequentist definition of probability could only assign
a probability to this proposition by performing (say) a large number
of repeated trips to the moon, and tallying up the fraction of trips on
which the moon turned out to be a dairy product....
In either case, it seems sensible that the less probable an event is, the
more information is gained by noting its occurrence. (Surely discovering
that the moon IS made of green cheese would be more informative
than merely learning that it is made only of earth-like rocks.)
Probability Rules
Most of probability theory was laid down by theologians: Blaise PASCAL (1623-1662) who gave it the axiomatization that we accept today;
and Thomas BAYES (1702-1761) who expressed one of its most important and widely-applied propositions relating conditional probabilities.
Probability Theory rests upon two rules:
Product Rule:
p(A, B) = joint probability of both A and B
= p(A|B)p(B)
or equivalently,
= p(B|A)p(A)
Clearly, in case A and B are independent events, they are not conditionalized on each other and so
p(A|B) = p(A)
and p(B|A) = p(B),
in which case their joint probability is simply p(A, B) = p(A)p(B).
6
Sum Rule:
If event A is conditionalized on a number of other events B, then the
total probability of A is the sum of its joint probabilities with all B:
p(A) =
p(A, B) =
p(A|B)p(B)
From the Product Rule and the symmetry that p(A, B) = p(B, A), it
is clear that p(A|B)p(B) = p(B|A)p(A). Bayes Theorem then follows:
Bayes Theorem:
p(B|A) =
p(A|B)p(B)
p(A)
The importance of Bayes Rule is that it allows us to reverse the conditionalizing of events, and to compute p(B|A) from knowledge of p(A|B),
p(A), and p(B). Often these are expressed as prior and posterior probabilities, or as the conditionalizing of hypotheses upon data.
Worked Example:
Suppose that a dread disease affects 1/1000th of all people. If you actually have the disease, a test for it is positive 95% of the time, and
negative 5% of the time. If you dont have the disease, the test is positive 5% of the time. We wish to know how to interpret test results.
Suppose you test positive for the disease. What is the likelihood that
you actually have it?
We use the above rules, with the following substitutions of data D
and hypothesis H instead of A and B:
D = data: the test is positive
H = hypothesis: you have the disease
= the other hypothesis: you do not have the disease
H
7
Before acquiring the data, we know only that the a priori probability of having the disease is .001, which sets p(H). This is called a prior.
We also need to know p(D).
From the Sum Rule, we can calculate that the a priori probability
p(D) of testing positive, whatever the truth may actually be, is:
H)
=(.95)(.001)+(.05)(.999) = .051
p(D) = p(D|H)p(H) + p(D|H)p(
and from Bayes Rule, we can conclude that the probability that you
actually have the disease given that you tested positive for it, is much
smaller than you may have thought:
p(H|D) =
p(D|H)p(H) (.95)(.001)
=
= 0.019
p(D)
(.051)
(1)
possible solutions (or states) each having probability p n, and the other
with m possible solutions (or states) each having probability p m. Then
the number of combined states is mn, and each of these has probability
pmpn. We would like to say that the information gained by specifying
the solution to both problems is the sum of that gained from each one.
This desired property is achieved:
Imn = log2(pmpn) = log2 pm + log2 pn = Im + In
(2)
A Note on Logarithms:
In information theory we often wish to compute the base-2 logarithms
of quantities, but most calculators (and tools like xcalc) only offer
Napierian (base 2.718...) and decimal (base 10) logarithms. So the
following conversions are useful:
log2 X = 1.443 loge X = 3.322 log10 X
Henceforward we will omit the subscript; base-2 is always presumed.
Intuitive Example of the Information Measure (Eqt 1):
Suppose I choose at random one of the 26 letters of the alphabet, and we
play the game of 25 questions in which you must determine which letter I have chosen. I will only answer yes or no. What is the minimum
number of such questions that you must ask in order to guarantee finding the answer? (What form should such questions take? e.g., Is it A?
Is it B? ...or is there some more intelligent way to solve this problem?)
The answer to a Yes/No question having equal probabilities conveys
one bit worth of information. In the above example with equiprobable
states, you never need to ask more than 5 (well-phrased!) questions to
discover the answer, even though there are 26 possibilities. Appropriately, Eqt (1) tells us that the uncertainty removed as a result of solving
this problem is about -4.7 bits.
10
Entropy of Ensembles
We now move from considering the information content of a single event
or message, to that of an ensemble. An ensemble is the set of outcomes
of one or more random variables. The outcomes have probabilities attached to them. In general these probabilities are non-uniform, with
event i having probability pi, but they must sum to 1 because all possible outcomes are included; hence they form a probability distribution:
X
pi = 1
(3)
pi log pi
(4)
11
p(x = ai, y)
p(x, y)
p(x, y)
Conditional probability: From the Product Rule, we can easily see that
the conditional probability that x = ai, given that y = bj , is:
p(x = ai|y = bj ) =
p(x = ai, y = bj )
p(y = bj )
p(x, y)
p(y)
p(x, y)
p(x)
x,y
p(x, y) log
1
p(x, y)
(5)
(Note that in comparison with Eqt (4), we have replaced the sign in
front by taking the reciprocal of p inside the logarithm).
13
p(x|y = bj ) log
1
p(x|y = bj )
(6)
If we now consider the above quantity averaged over all possible outcomes that Y might have, each weighted by its probability p(y), then
we arrive at the...
Conditional entropy of an ensemble X, given an ensemble Y :
(7)
H(X|Y ) = p(y)
p(x|y) log
y
x
p(x|y)
and we know from the Sum Rule that if we move the p(y) term from
the outer summation over y, to inside the inner summation over x, the
two probability terms combine and become just p(x, y) summed over all
x, y. Hence a simpler expression for this conditional entropy is:
1
X
H(X|Y ) = p(x, y) log
(8)
x,y
p(x|y)
X
(9)
It should seem natural and intuitive that the joint entropy of a pair of
random variables is the entropy of one plus the conditional entropy of
the other (the uncertainty that it adds once its dependence on the first
one has been discounted by conditionalizing on it). You can derive the
Chain Rule from the earlier definitions of these three entropies.
Corollary to the Chain Rule:
If we have three random variables X, Y, Z, the conditionalizing of the
joint distribution of any two of them, upon the third, is also expressed
by a Chain Rule:
H(X, Y |Z) = H(X|Z) + H(Y |X, Z)
(10)
n
X
i=1
H(Xi)
(11)
Their joint entropy only reaches this upper bound if all of the random
variables are independent.
15
(13)
(14)
(15)
Fanos Inequality
We know that conditioning reduces entropy: H(X|Y ) H(X). It
is clear that if X and Y are perfectly correlated, then their conditional
entropy is 0. It should also be clear that if X is any deterministic
function of Y , then again, there remains no uncertainty about X once
Y is known and so their conditional entropy H(X|Y ) = 0.
Fanos Inequality relates the probability of error Pe in guessing X from
knowledge of Y to their conditional entropy H(X|Y ), when the number
of possible outcomes is |A| (e.g. the length of a symbol alphabet):
Pe
H(X|Y ) 1
log |A|
(18)
(19)
!"#
$
B, 3/8
D, 1/8
C, 1/8
A, 1/2
C, 1/8
A, 1/2
C, 1/8
A, 1/4
E, 1/4
B, 1/4
B, 1/4
[zW-^[,.4V\ 5W-^
I9;<321;&>9J21;& &>9V2=B4 P GJ4UT21;&05G;0525&>C254JY& 2b1;& 0AC4MU21;&80@& *m9;<*-N:*-<gAI,~052I25&
&>9V2=B4 P GNI,mA;&80(/&8*.\;125&8<3(+*-212b1;& 052IM25&4:686>A P IV9;68GWx6>I,-6>A;,mI25&8<3U=B4:C2b1;& v 4(
&8wAI25*-4:9;0^8|
E
- 5W-^,-4\5W-^
Wx:^
"
c
]M"g
Vg
o>
Symbols
Encoding
Source
encoder
{*-\:A=B&o>| iT*-056>=B&825&ECE&>CE4:=zG:,-&8050+0@4:A=B68&lI9;<)&>9;684:<&>=
mx
+ DggM
o
4V21;&>=B(+*-05&
!
(_1;&>=B& x T*-0T21;&E,mI=B\V&8052_*m9V25&8\&>=`,-&8050T21IV9EQ
+1;&cb>aV!T*-0T21;&>9Y;*-250 P &>=
05GCEY4,!H:IV9;<FIM0(&+L;9;4(D ! W!^
21;&>9DEQ +1;&+&bc68*-&>9;68G?4MU21;&/684;<*u9;\
*.0`\*-N&>9)YG|
%1;&>9'2b1;&c05GCZY[4,-0ZI=z&?&8wA;* P =B4;YIY;,-&HI9;<*.0EI P 4O(/&>=r4U+2a(/4gH oI9;<
EQ/+4V25&T21I2+*U(/&T2=BG254cIM61;*-&8N&E#D*m9 2b1;*.0`6>I05&T(/&lCEA;052_IM,.,-4;6>I25&?I2
,-&>I0524:9;&_&>9;684;<*m9;\E254ZCE4;=B&/21I94:9;&+05GCEY4,;E21;*-0tCF&>I9;021IM2(&`I=B&+*m9;6>I P IY;,-&
4UA9;*-wA;&8,-G <&8684;<*m9;\gQ
25*-,-,
(+*-21&8wA;* P =B4;YIY;,-&T05GCEY[4,-0>HYA;2(`1;&>9c*.0+9;42/I P 4(/&>=4U
2B(/4gH21;*-0/684;<*u9;\
*-0*m9;&bc68*-&>9V2>254~2=BGl2@4l4N&>=B684:CF&2b1;*.0(/&+684:9;05*-<&>=05&8wA;&>9;68&804MU05GCEY4,-0/4U,-&>9;\2b1
'I9;<684:9;05*-<&>=&>9;684:<*m9;\E&>I681 P 40@05*mY;,.&T0@&8wA;&>9;68&`4U
,-&>9;\21?'IM0IlY;,-4:68LE4U
Y;*m9I=BG
<*-\*-250>H21;&>9(/&T4:Y;2I*m9|
] ~,-4\ o
lgg'mgD
e
i
f
h
_WxT^
o>
o>
o>
o>
h/4;<&co
o
o
o>
h/4;<&Z
o>
o o>
ooo
o
oo
ooo
h/4;<&Z
%'&T684:9;0@*.<&>=&>I681684;<&T*m9 2bA=59|
oVQ++05*m9;\E21;*-0/&>9;684:<*m9;\gH[(&n9;<21I2 P =z&805&>92@&8<c(+*-21I~05&8wA;&>9;68&T,.*mL&?ooH(/&
<49;42`L:9;4((_1;&821;&>=254c<&8684;<&?I0 e f e 4:=ilhTQg+1;*-0lC?IVL&80+0A;681IE684:<&
A9;0I25*-0BUI68254:=zGQ{A=B21;&>=8H;*m9\V&>9;&>=5I,!H&8N&>9IE684;<&l(`1;&>=B&T0A;681I9ICEY;*-\:A;*-2aG
684:A;,-< Y&l=B&8054V,.NV&8<A9;*-wA;&8,.GYG,.4;4:LV*m9;\cI2Y;*-250tUA=B2b1;&>=IV1;&>I<c*m921;&T0@2=B&>IC
W!I9;< YI68L2b=5I68L*m9;\:^*-0A9;0bI25*-0BUI68254:=BGQ
RTY;05&>=BN&`21I2 U!4:=2b1;*.0/684;<&H;21;&_684;<*m9;\?=5I25&HV4:=IN&>=5IM\&_9;ACZY[&>=4UY;*.2@0 P &>=
05GCEY[4,!H*-0_\*-N&>9YVG|
o +D
b ~
(_1;*-61q*.0,.&80@0_21IV921;&l&>9V2=B4 P GQ
[Q++1;*-0T684:<&E*-0lA9;*-wA;&8,-G'<&8684;<gIY;,-&UA=B2b1;&>=+21;*-0T684;<&?*-0T*m925&>=z&80525*m9;\*m921I2
(&6>I9<&8684:<&? 8! ! gb
F2b1I2*-0_9;4?YI68L2b=5I68L*m9;\E*-0t=z&8wA;*m=B&8<
4:9;68&
(&?1IN&?2b1;&cY;*-250l4U/21;&?&>9;684;<&8<0@GCEY4V,(/&E6>I9<&8684;<&c(+*-21;4;A;2l(+I*-25*m9;\
Us4:=`CE4:=z&Ql{A=B21;&>=`21;*-0lI,-054)0I25*-0Bng&80T21;&B
l684:9;<*-25*-4:9H 21IM2_*-0l2b1;&>=B&E*.0
9;4T684:<&`(/4:=B<E(_1;*-61*-0 P =B&bngy Wx*!Q &QV0ICE&+Y;*-2 P I2525&>=@9^ 4UI`,-4:9;\&>=684;<&+(/4:=B<
Q
921;*-0`6>I05&T21;&~684:<*m9;\=5I25&`*-0_&8wAI,
25421;&T&>92b=B4 P GQ
[Q/% 1;*-,-&E21;*-0_*-0`<&b684:<gIVY;,.&W!I9;< 684;<*m9;\=@I25&l*-0`&8wAIM,
25421;&l&>9V2=B4 P GI\:I*m9^8H
4:Y;05&>=BN&T2b1I2+*-2_<4;&80T9;42_1IN&l2b1;& P =B&bngy P =B4 P &>=B2BGI9;< *-0T9;42IV9*u9;0@2I9V2I
9;&84:A;0+684;<&Q
1I99;4:9 0/n=B052+2b1;&84:=B&>CS*-0`21;& :Bb ab>V c:bB#(`1;*.681*-0>|
{g4:=I'<*-056>=z&825&CE&>CE4;=BG:,-&8050054:A=z68&(*.2b1n9;*-25&7&>92=z4 P GDJU!4:=cIV9G
W P 40@*.2@*.NV&>
^ `*-2T*-0 P 40@05*mY;,.&254c&>9;684;<&?21;&E05GCEY4,-0TI2TI9I>NV&>=5I\&l=5IM25&
EH:0bA;61q21I2>|
]S
xW {4:= P =B4:4MU05&8& 1I99;4:9 %'&>IN&>=8Q ^E+1;*-0+*-0TI,-054?054:CF&825*mCE&80_6>IM,.,-&8<)21;&Z9;4*-05&8,-&8050
684;<*u9;\E21;&84;=B&>CI0*-2<&>IM,.0/(+*-21684;<*u9;\E(+*-21;4;A;2684:9;05*-<&>=5IM25*-4:94U9;4*-05& P =B4:68&80@05&80
Wx*!Q &QgY;*-2`684:=5=5A P 2@*.4;9&8256>^8Q
+1;&/&>9V2=B4 P GTUA9;6825*-4:9E21;&>9=B& P =B&805&>9V250IUA9;<gICE&>9V2I,;,.*mCE*-2/4:9~21;&+9;ACZY[&>=4U
Y;*-250
4:9IN&>=5IM\&=z&8wA;*m=B&8<)254c=B& P =B&80@&>92+2b1;&T05GCEY[4,-0_4U21;&T054;A=B68&Q
mm
l!
2 o
3
1 0
o
x
Source
encoder
Decoder
Channel
Symbols
{*-\:A=B&?o>[|h/4;<*u9;\I9;<<&8684;<*m9;\4U05GCEY4,-0Us4:=2=@I9;0BU!&>=/4N&>=/IE61IV99;&8,sQ
%'&684:9;0@*.<&>=I9J*m9 P A;2I, P 1IY&829 3d :>>5; :8< kI9;<4:A;2 P A;2cI, P 1IY&8>
2 =
d ?V>>>5@ ?,A klI9;<=@I9;<4:CNOI=z*uIVY;,.&80 I9;<>
3
B
(_1;*-681=5I9;\&`4N&>=2b1;&805&IM, P 1IVY&8250>Q
425&/21I2/IV9;<DC 9;&8&8<9;42Y[&_21;&0ICE&lU!4:= &8yIC P ,-&`(&+C?IGE1I>NV&2b1;&_Y;*m9I=BG
*m9 P A;2IM, P 1IVY&82 d okI9;<21;&4:A;2 P A;2cI, P 1IY[&82 d o E k;H
(_1;&>=B& E =z& P =B&805&>9V250
21;&~<&8684:<&>=*-<&>925*U!G:*m9;\054;CE&T&>=5=B4:=Q+1;&l<*-056>=z&825&ZCE&>CF4:=BG;,.&80@0+61IV99;&8,6>I9q21;&>9
Y&E=B& P =B&805&>9V25&8<I0+IE05&82+4U2=5IV9;05*-25*-4:9 P =B4:YIY;*-,-*-25*-&80>|
W ?GFIHJ: ^ WB# ?GFKH : ^
QQ
QQ
QQ
QQQ
Q
Q
Q
N R
O
W ?V HJ:/< ^W ?G HJ:/< ^ >> W ?,ASHT:/< ^ W
425&T21I2/(/&T1I>NV&T21;& P =B4 P &>=B2BG?21I2/U!4:=&8N&>=BG*m9 P A;2_0@GCEY4V,sH;(/&T(+*-,-,\V&820@4:CE&b
21;*m9;\4:A;2>|
F 0
W ?GFIHJ: ^ o
W : B#
W ?GFKHT: ^! W : ^
?GF
<
WB#
?GF
W ? F HJ: ^!W :; ^
0
F 0
F 0
WB
_^ FG0 `
?3FH
<
W ?GFHJ: ^!W : ^
0 _ ^ 0 ` F
WxKa^
Z\
E!g,D
c
cedfd:>x
b
Wx:p^
0.9
0
0.1
1-p
1.0
2
p
1.0
p
1
1
1-p
1.0
999999
999999
7_
g
#
po$
HB
<
o
^ W :;rHT? F ^ ,-4\Ds : HT?3F
W
^Gt
0
? F
F 0
F 0
F 0
XW9 H B
<
<
W :
HJ?GF
?GF
^!W ?3F ^
^,-4\Ds
W : ;?GF ^[,-4\us
o
W :;r HJ? F ^
W :
o
HJ?GF
W ?GF ^
Wx:^
Source
encoder
Decoder
Correction
Observer
Correction data
{*.\;A=B&lo|ht4:=5=B&86825*-4:90@G:052@&>C
P I 050@&8<I,-4:9;\?2b1;&l684:=5=z&86825*-4:9681I99;&8,
254=B&8684:9;052=@A;682+21;&l*m9 P A;2>21;*-0`2A=59;0+4:A;2`254
Y&EW *H B^Q
%'&6>I9XUA=B2b1;&>==B&8(_=B*-25&)21;&&>92=z4 P G7UA9;6825*-4:9qW 9^?I0I'0AC4N&>=21;y
& La4*m9V2
=P B4:YIVY;*.,-*-2BG<*-052=B*mYA;25*-4:9|
<
W : ^[,-4\W-W : ^5^ o
0
Wq9^
<
<
<
<
oVQ qW )
9 _+
= ^OzDQ
401;4(#21;*-0>H(/&?9;425&E21I2/W :
*u9q21;&`&8wAI2@*.4;9?IY;4N;&|
x
W9 {_
= ^
F 0
<
HJ?GF
W : @?3F ^,-4\
s
W : ;?GF ^
W : ^ W ?GF ^
t
Wx:^
Wq9 {=_^
F 0
W :;O@? F ^ ,-4\
s
W :;HJ? F ^
W : ^
t
W ?GFKHT: ^
W : @?GF ^,-4\>}
W ?GF ^~
x 0 F 0
W=E;9E^
<
<
W!^
Wq9 _
= ^#
W : ;?GF ^[,-4\W-W : ;?GF ^@^
0 F0
+1;&lCEA;2AI,*m9VUs4:=@C?x I25*-4:9q*.021;&ZCE4;=B&9IM2A=5I,-,-G(_=B*-2525&>9|
W 9)_ _
q
= ^X
W 9c^DW =T^ 'qW 9 =_^
1 x
E!g,cDV
b
o t
+1;&>9q21;&T&>92b=B4 P GH:684;9;<*.2@*.4;9I,&>9V2=B4 P GI9;< CEA;2AI,*m9VUs4;=5C?I25*-4:9)I=B&T\*-N&>9YVG|
qW 9^ o
X
W 9 H =`^ +,-4\
W-g^ ]WBo/ g^[,.4V\WBo/ g^
x
W 9 {+
q
= ^
o P ,.4V\W-g^
Wzo+g^,-4\WBo+g^
{*-\:A=B& VI01;4(+0E21;&6>I P I68*-2BG'4UT21;&681I99;&8,+I\:I*m9;0522=5I9;05*-25*-4:9 P =z4:YIY;*-,-*.2BGQ
425&`21I2/21;&`6>I P I68*-2BGc4U 21;&_681I99;&8,
<g=z4 P 0/254j8&>=B4E(_1;&>921;&`2=5I9;05*-25*-4:9 P =z4:YI
Y;*-,-*.2BG*-0lo H;I9;<*.0+C?IMy:*mCE
* j8&8<(_1;&>921;&`2=5I9;05*-25*-4:9 P =B4:YIVY;*.,-*-2BG*-0t[:4;=+o/?*U(/&
=B&8,-*mIY;,-G'2=5IV9;0 P 40@&Z2b1;&?05GCZY[4,-0l4:9)21;&E(+*m=B&(&?I,-054)\&82T21;&?C?Iy;*mCEACICE4;A92
4U*m9VU!4:=5C?IM25*-4:921=B4:A;\;1 {*-\:A=B&TY01;4(+0/21;&T&@
&8682/4:9CEA;2AI,*m9VUs4;=5C?I25*-4:9U!4:=
I05GC?CE&82b=B*-6T*u9 P A;2TI, P 1IY[&8250>Q
a
I(X;Y)
1
I(X;Y)
0.9
0.8
0.5
0.7
0.6
0.5
0.4
0.3
0.2
0.1 0.2
0.1
0
0.1
0.2
0.3
0.8
0.9
0.3 0.4
0.5 0.6
P(transition) 0.7 0.8
0.9
0.9
0.8
0.7
0.6
0.5
0.4 P(X=0)
0.3
0.2
0.1
c"8 7Tvo!8$
CIy
@
Wq9)_`
= ^
W!o>^
c"8T
7
oVQ+f=z*j8\=B&8*m9VUs4:=z68&l&>9V250>H(/& =B&`\4*m9;\254?I<NOIV9;68&
[Q+!2 0/&>I0@GCE4=B&8684\:9;*-05&~0 P &8&8681
4O(/&8N&>=H(_1;*-,-&21;& =B&8<gA9;<gI9;68G34UT21;*-0c054;A=B68& P =B4V25&868250cI\;I*m9;052?0A;681D=5IV9;<4:C
681I=5I6825&>=&>=5=z4:=8H684:9;0@*.<&>=21;&T&>=5=z4:=/<gA;&T254cI1:AC?IV9CE*-0Ba1;&>I=B*m9;\g|
oVQ+f=z*u9;\21=B&8&ZIV9;<Us4:A= P >& 9;68&H(& =B&_\V4*m9;\c254?IE<gIV9;68&Q
[Q+!2 0/&>I0@Gc254(_=B&868LcI?9;*-68& P &>I681Q
+1;&?684;<*m9;\'9;&8&8<0Z2@4684:9;05*-<&>=~21;&?&>=5=B4;=`61I=@I6825&>=B*-0525*-680E4U/21;&?681I99;&8,I9;<7<&b
684;<&>=8HIV9;<2=BG'254'I681;*-&8N& I05*-\:9;*ng6>I9V6
2 g5<*-052bI9;68& hY&82B(/&8&>9 P ,mIA;05*mY;,-&'&>9;684;<&8<
CE&8050bI\&80>Q
x
;>m>
l
RT9;&4UT21;& 05*mC P ,-&8052)684:<&80*-0 IDB >!x!x]>Q]%'&)2IL& IJY;*m9I=BG305GCCE&82=B*-6
681I99;&8,l(+*-21IJ2=5I9;05*-25*-4:9 P =B4:YIY;*-,-*-2aG
~21;*-0 \V*.NV&80IJ681I99;&8,T6>I P I68*-2BG
o/,-4\W-g^Wzo_g^[,.4V\[Wzo_g^8Q~+1;&?9I2A=5I,Y;*m9I=BG&>9;684;<*u9;\qUs4;=`21;*-0T*.0~21;&>9
d okZ21;&T=B& P &82@*.2@*.4;9?684;<&_(*.,-,
=B& P &>I221;&80@&_<*-\*-250+I9E4;<<9:ACEY&>=4U
25*mCE&80+I9;<
P &>=aUs4:=@C C?IGBL 4:=B*-2BG?N4V25*m9;\gQ
&>9;68&+(/&+2=5I9;0bCE*-2 (o+Y;*.2@0 P &>= 0@GCEY4V,sH;I9;<?(+*-,-,g4;Y;2I*m9?I9~&>=5=B4;= * ZU o
4:=+CF4:=B&TY;*.2@0I=z&l=B&868&8*-N&8< *m9&>=@=B4:=8H;21I2*.0|
#]o Wzo+g^ % l
s
t
\ 1 0
Z
h 4:9;0@*.<&>=TIF2=5I9;05*-25*-4:9 P =B4:YIY;*-,-*-2BG 4U oVQ+1;&l681I99;&8, 6>I P I68*-2BG I0+\*-N&>9)YG
/
&8wAI25*-4:9l[o*-0Z o>VW ng\:A=B&VI^8Q
+1;&684:<&/=5I25&4MU21;&/=B& P &82@*.2@*.4;9l2@&8619;*-wA;&
I\:IM*u9;0@2+21;&l=B&805*-<gAI, P =B4:YIVY;*.,-*-2BG4U&>=5=B4:=*-0_<&>CF4:9;052=5IM25&8<*m9ng\:A=B&lVYQ
0.1
0.99
0.98
0.01
I(X;Y)(p)
I(X;Y)(0.01)
0.001
Residual error rate
Capacity
0.97
0.96
0.95
0.94
0.93
0.0001
1e-05
1e-06
1e-07
1e-08
0.92
1e-09
0.91
0.9
1e-08
Repetitiion code
H(0.01)
1e-07
1e-06
0.01
0.1
1e-10
0.01
0.1
Code rate
c"8 7~c
"
g
%'&lI=@=B*-N&lI2 1I99;4;9 0+05&8684:9;<)21;&84:=B&>CqH21;&? ;/ c ;B, Z|
{g4:=+I681I99;&8, 4U6>I P I68*-2B|
G I9;<'I054:A=B68&E4U&>9V2=B4 P GJ* U+ H
21;&>9U!4:=+I=@Y;*.2b=5I=B*-,-G 0CI,-, 8H21;&>=B&~&8y:*-052@0lIE684:<*m9;\05681;&>CE&E0A;68121IM2
21;&T054:A=B68&T*-0T=B& P =B4;<gA;68&8< (+*-21)I?=B&805*-<gAI,&>=5=B4:=+=@I25&`,.&80@0_21IV98 Q
1I99;4:9 0 P =B4:4MU4U/21;*-0T21;&84:=B&>C*-0ZIV9 &8y;*-0525&>9;68& P =z4:4U+=5I21;&>=`2b1I9ICE&>I9;0T254
684:9;052b=5A;682E0A;61J684;<&80*u9J2b1;&\&>9;&>=5I,/6>I05&QJ9 P I=B25*-6>A;,mI=E21;&681;4*-68&4UlI)\4:4;<
684;<&_*-0+<*-682I2@&8<YVGE21;&_681I=5IM6825&>=B*-0525*-680/4U
21;&`61I99;&8,
9;4*-05&Q9 P I=5IM,.,-&8,(+*-2121;&
9;4*-05&8,-&80506>I05&HY&82525&>= 684;<&80I=B&4U!25&>9lI681;*-&8N&8<?YGT684;<*m9;\ZCEA;,-25* P ,-&+*m9 P A;205GCEY[4,-0>Q
1x
)Dmg? !
b i
b i
b i
b i
i
b i
i
b i
m
m
mm
o
}
o
gW -, 4\
o
p+DW ,-4\ ^
p+
o o
^
o
W : ^ W
U
W
~
p
+1;&6>I P I68*-2BGD4MUl21;&61I99;&8,T*-0:Vp'*m9VU!4:=5C?I2@*.4;9DY;*-250 P &>=Y;*m9I=BGD<*-\*-2)4Ul21;&
681I99;&8,684;<*m9;\gQ?h+I9(/&~n9;<ICE&8681I9;*-0C254&>9;684:<&*u9VU!4:=5CI25*-4:9'Y;*-250~*u97p
681I99;&8,Y;*-250`0AIY LB&8682+254?21;&T&>=@=B4:= P =B4 P &>=B2BGc<&8056>=z*uY[&8<'IY4N&
+1;&W!p:^_ICCE*m9;\h/4;<& P =B4N:*-<&80TI , !, E! `684;<&l254 P &>=aUs4:=@C 21;*-0qI05G;0B
25&>C?IM25*-6+684:<&`*-0/4:9;&+*m9?(`1;*.68121;&+4;YN;*-4:A;0Y;*m9I=BG&>9;684:<*m9;\?4U
21;&054:A=B68&`05GCZY[4,-0
*-0 P =B&805&>9V2/*m921;&_681I99;&8,&>9;684;<&8<U!4:=5CQ{g4:=4:A=054:A=z68&_(_1;*-6>1&>CE*-250+I2&>I68125*mCE&
*m92@&>=BNOI,/o~4Ut,o E
a 05GCZY[4,-0>H(/&T2IL&T2b1;&ZY;*m9I=BG=B& P =B&805&>9V2I25*-4:9q4U21;*-0lI9;< 684 P G*-2
254Y;*-250l H
H
i IV9;< m 4MU21;&~&>9;684:<&8<Y;,-4;6L
2b1;&F=z&>C?I*m9;*m9;\ Y;*.2@0ZI=B&~\*-N&>9YG
HgIV9;
< GgVaZw t
YVG
b |
e i 3 m IV9;<
H
Z3 i 3 m
i 3 m IV9;<
H
3 i 3 m
Z3 m IV9;<
H
M 3 Z3 m
RT9=B&868& P 25*-4:9* U 21;&_Y;*m9I=BGc9;ACEY&>= 8 T21;&>9E2b1;&>=B&_*-0/9;4Z&>=@=B4:=8H&8,-05&l _%
*-0_2b1;&lY;*.2`*m9&>=5=z4:=8Q
+1;*-0 `IC?CE*m9;\J684:<&'A;05&80 JY;*-250c2@4684:=@=B&8682p]W! #o>^E&>=5=z4:= P IM2525&>=59;0
I9;<
1e-10
1e-20
1e-08
1e-07
1e-06
0.01
0.1
{*-\:A=B&Eg|+W!pM:^/_ICCE*m9;\?684:<&E=B&805*-<gAI, &>=5=B4:=/=5IM25&
%'&~6>I921;&>9684;9;05*-<&>=`21;&ZCE4;=B&T\&>9;&>=5I,6>I05&H(_1;&>=z&l(/&l1IN&l=5I9;<4:CY;*-2_&>=@=B4:=B0
A9;*Us4;=5CE,-G?<*-052=B*mYA;25&8<'Wx*!Q &Q(/&_I,-,-4(D21;& P 4050@*uY;*-,-*-2BG?4U2B(/4T4:=CE4;=B&+Y;*.2&>=5=B4:=z0 P &>=
plY;*-2+Y;,-4:68L;^8Q e \:I*m9E(/&_4;Y;2I*m9E21;& P =B4:YIY;*-,-*-2BG?4U=B&805*-<gAI,&>=5=B4;=I021;&_=B&>C?IM*u9;<&>=
4U21;&lY;*m9;4:CE*mI, 05&>=B*-&80>|
m
s p WBo+g^{m@l
t
\ 1 0
v@#X,]v_
o
05*m9;\c4:A=Us4:=@CZA;,mI~U!4:=<*-056>=B&825&~&>92b=B4 P GI9;<2bIL*m9;\?2b1;&T,.*mCE*-2>|
o
qW 9^ ,-*uC k W :/F _^ : ,-4\ } :/F :
W _^ ~
F
,-*uC k# W :/F ^[,-4\ } :/o F : ',-4\ Wq : ^ W : ^_ :I
W ^ ~
F
F
o
:
}~,.*mC k ,-4\ qW : ^ W : ^
W : ^,-4\ } :
W
^
,
~D
~
l
l
qW 9^,.*mC k ,-4\ W : ^
W!^
+1;*-0l*-0l=5IM21;&>=`(/4:=5=BG;*m9;\I0T2b1;&Z,mI2@25&>=`,-*uCF*.2~<4:&80l9;4V2T&8y:*-052>QE4O(/&8N&>=HI0T(/&EI=B&
4U!25&>9*m9V25&>=B&80525&8<7*u9721;&<* &>=z&>9;68&80FY[&82B(&8&>9 &>92=z4 P *-&80qWs*sQ &Q
*m9'2b1;&?684:9;05*-<&>=5I2@*.4;9
4UCEA;2AI,&>92=z4 P G4;=+6>I P IM68*.2BG^8H(&~<&bn9;&?21;& P =B4:Y;,-&>CI(tIGYGA;05*m9;\21;&Tn=z052
25&>=5C 4:9;,-GI0`I?CE&>I0A=z&T4UtV _B g!x !B@ v |
:
W!^
Wq9^
W : ^,-4\ } o :
W ^ ~>
l
W c G^ ~ c
(+*-21|
DW c ^
W : ^
W ? ^
DW c ^
W :
@ ? ^[,-4\
}
W :]HJ? ^
W : ^
~
n
?V
po+gMM g
W : ^ W : ^
(`1;&>=B&|e
W : ^
W!^
1;*-04 P 25*mCE*
>j I25*-4:9 P =B4:Y;,-&>C*-0/054,-N&8< A;05*m9;)
+
\
I\:=@I9;\&_CEA;,-25* P ,-*.&>=z0I9;<C?IyV
*uCF*8j *m9;\g|
- W : ^% :
> W : ^[,.4V\
W : ^- W : ^>W : ^
(_1;*-61q*.04:Y;2I*m9;&8<YVG054,-N:*m9;\g|
To+',-4\ W : ^- W : ^
W : ^ o
l
l/
p
1;&>9;68&|
o
W
^
,.4V\W! ^
W!a^
E*
7 g
921;&+6>IM05&_4MU684;925*m9;A;4:A;00@*.\;9I,-0tIV9;<c681I99;&8,-0/(&TI=B&*u9V25&>=B&80@25&8<?*m9UA9;682@*.4;9;0rWs4U
0IGE25*mCE&>^8HV61;40@&>9EU=B4:C#Ir05&824U P 40505*mY;,-&`UA9;682@*.4;9;0>HI0*m9 P A;2/254l4;A=681I99;&8,!H(+*-21
054:CF& P &>=B2A=5Y[&8<NV&>=B05*-4:94U/21;&EUA9;6825*-4:9;0?Y&8*m9;\ <&82@&86825&8<XIM2T21;&?4:A;2 P A;2Q?+1;&805&
UA9;6825*-4:9;0+21;&>92IVL&_2b1;& P ,mI68&~4U21;&T*m9 P A;2_I9;<4;A;2 P A;2+I, P 1IY&82504U2b1;&_<*-056>=B&82@&
6>I05&Q
%'&CEA;052ZI,-05421;&>97*m9;68,mA;<& 21;& P =B4:YIY;*-,-*-2BG4U_I)\*-N&>9 UA9;682@*.4;9Y&8*m9;\'*m,9 aL &8682@&8<
*m92@4Z2b1;&T61I99;&8,(_1;*-61Y=B*m9;\0`A;02@4Z2b1;&T*.<&>IF4U ,)- ?21;*-0+\&>9;&>=5IM,IV=B&>IT4U
(/4:=5LE*-0TL:9;4(_9IMX
0 EG ; aE :b@ Q
{g4;=/&8yIVC P ,-&H684:9;05*-<&>=21;&l&>9;05&>CEY;,-&80>|
oVQ
Wa^]0@*u9 W^
jI681`NOI,mA;&4U t<&bn9;&80I/<* &>=z&>92UA9;6825*-4:9HI9;<T2@4\&821;&>=(*.2b1lI P =B4;YIY;*-,.*-2BG
<*.0@2=B*mYA;25*-4:9H0IGTWq^8H(/&c1IN&I9&>9;05&>CEY;,-&Q'+4V25&?21I
2 1;&>=B&C?IG'Y&
<*.0@6>=B&825&l4;=684;925*m9;A;4:A;0c684:9;05*-<&>= P 1I05&l0b1;* U!2_L&8G;*m9;\gQ
[Q+h/4:9;05*-<&>=I705&824UZ=5I9;<4;CNOI=B*mIY;,-&80 d B;| @ o ; >>8 k'(_1;&>=B&&>I681
T2IVL&80?4:9IJ=5I9;<4:C
NOI,mA;&'I68684:=B<*m9;\254JI'}lIA;0@05*mI9<*-052=z*uYA;2@*.4;9](+*-21
052I9;<gI=z<c<&8N;*mI25*-4:9 '[21;&>9|
W d Vk a^ V 0@*u9;6W! 6 ^
*.0/21;y
& g5(_1;*-25&T9;4*-05& E
h &>9;05&>CEY;,.&HYI9;<;,-*mCE*-25&8<25
4 &>=B_2 j_I9;<(+*-21IN&>=5I\&
4P (/&>= Q
[Q/K4;=B&T\&>9;&>=5I,-,-GH(/&l1IN&TU!4:=+=5IV9;<4:C NI=B*mIY;,-&80 3d : kl2b1;&l&>9;05&>CEY;,-&E4UYI9;<;
,.*mCE*-25&8<)UA9;6825*-4:9;0>|
W 3d : k a^ : 05*m9;6W! 6^
~
xU?(/&I,-054D684;9;05*-<&>=)UA9;682@*.4;9;0,-*mCE*-25&8<#254]25*mCE&7*u9V25&>=BNI
, l
H~21;&>9(/&'4;YV
2I*m9J4:9;,-GXr 9;4;9V' j8&>=B4'684;&bc68*-&>9250I9;<D(/&6>I9J684:9;05*-<&>=21;&&>9;05&>CEY;,-&
254?Y&Z=z& P =B&805&>9V25&8<YVGIV
9 (<*mCE&>9;0@*.4;9I,_qW (G ^ P =B4:YIVY;*.,-*-2BG<*-052=B*mYA;25*-4:9
:
8
:
>
>
[
:
W
^Q
Q/K4;=B&?0 P &868* ng6>IM,.,-G;H+*U_(/&684:9;05*-<&>=E21;&&>9;05&>CEY;,-& 4U`,.*mCE*-25&8< P 4(/&>=W!YG ^8H
YI9;<;,-*mCE*-25&8<DWx254 ^I9;<c2@*uCF&b,.*mCE*-25&8<)05*-\:9I,-0ZW!9;4;9V' 8j &>=B4?4:9;,-G*u9*m9V25&> =BNI,
W! `^5^8H(&)n9;<]21I22b1;&&>9;0@&>CZY;,-&'*-0=B& P =B&805&>9V25&8<YG]I
9 ([<*uCF&>9;05*-4:9I,
=P B4:YIY;*-,-*-2aG3<*-052=B*mYA;25*-4:9(_1;*-613*-
0 8j &>=B44:A;2@05*-<&2b1;
& ([0 P 1;&>=z& =@I<*mA;
0
Q
c"8~
7 cv
o !8$
,-*mCE*-25&8<25D
4
Q
F I9;%'&)21;&>93684:9;05*-<&>=21;
& (# 0IC P ,-&80W D
< B F ^E21IM2c6>I93Y&'A;05&8<]254
681I=5I6825&>=z*.0@&lY4212b1;&T*u9 P A;2TI9;<4:A;2 P A;20@*.\;9I,-0>Q1;&>921;&l=B&8,mI25*-4:9;01;* P Y&82B(/&8&>9
21;&T*m9 P A;2TI9;< 4:A;2 P A;2+*-0_\*-N&>9YVG|
F F H/?#o >8 (
D
B F
(_1;&>=B&? F *-0`U=B4:C 21;&?YI9;<;+,.*mCE*-25&8<}lIVA;0505*mI9)<*-052=B*mYA;25*-4:9(*.2b|
1 j8&>=z4 CE&>I9I9;<
NOIV=B*mI9;68&|
k
W B H ^
W!^ ,-4\ k
&>9;68&|
W
{
B ^
W B?^6
W^
+1;&T6>I P IM68*.2BG4U21;&T681I99;&8,
*-0+2b1;&>94:Y;2I*m9;&8< YVGC?IMy:*mCE
* 8j *m9;\c2b1;*.04ONV&>=IM,.,*u9 P A;2
<*-052=B*mYA;25*-4:9;0~ 21;*-0E*-0ZI681;*-&8N&8<YVGC?IMy:*mCE
* j8*m9;\'(+*-21'=z&80 P &8682~25421;&E<*-052=B*mYA;25*-4:9
4U 0AIY BL &8682+25421;&lIN&>=5I\& P 4(/&>=/,-*mCE*-2I25*-4:9|
M F
N
o
,-4\/ W! k ^
W!^
,-4>
\ }go k
W!p^
~
o
Y;*-2;0
(_1;&>=B&
mx
W^
8
+,++
,u9 WBo*
^
(&T4;Y;2I*m9|
k ,-4\
[Q++1;*-0+*-0_4U!25&>9=B&8(_=B*-252@&>9*u9q25&>=5CE0/4U+>K O>X8 x!HbH(`1;*.681*-0_<&bn9;&8<YG
qcQ+1;&T,-*uCF*.2@*u9;\)NOI,mA;&~*.0`21;&>9|
,.4V\ l a
k
\
+1;*-0+*-0_6>I,-,-&8<21;& 1I99;4:
9 *mCE*-2>Q
Q++1;&6>I P IM68*.2BG~4Ug21;&/61IV99;&8,*.0IM61;*-&8N&8<E(_1;&>9~21;&/054:A=z68&g@,.4;4:LV0,-*mL&_9;4V*.0@&gh Q
+1;*-0?*-0c21;& YIM05*-0EUs4;=E0 P =B&>I<0 P &8682=5AC 25&8619;*-wA;&80c4MUrCE4;<gA;,mI25*-4:93I9;<*m9
P I=B25*-6>A;,mI=+h/4;<&liT*-N;*-05*-4:9)K A;,-25* P ,-& e 6868&80@0ZW!h/iTK e ^8Q
a
ck k (x)
(2)
where many possible choices are available for the expansion basis functions k (x). In the case
of Fourier expansions in one dimension, the basis functions are the complex exponentials:
k (x) = exp(ik x)
(3)
where the complex constant i = 1. A complex exponential contains both a real part and an
imaginary part, both of which are simple (real-valued) harmonic functions:
exp(i) = cos() + i sin()
(4)
which you can easily confirm by using the power-series definitions for the transcendental functions
exp, cos, and sin:
2 3
n
+
+ +
+ ,
(5)
exp() = 1 + +
1! 2!
3!
n!
cos() = 1
2 4 6
+
+ ,
2!
4!
6!
(6)
sin() =
3 5 7
+
+ ,
3!
5!
7!
(7)
Fourier Analysis computes the complex coefficients ck that yield an expansion of some function
f (x) in terms of complex exponentials:
f (x) =
n
X
ck exp(ik x)
(8)
k=n
where the parameter k corresponds to frequency and n specifies the number of terms (which
may be finite or infinite) used in the expansion.
Each Fourier coefficient ck in f (x) is computed as the (inner product) projection of the function
f (x) onto one complex exponential exp(ik x) associated with that coefficient:
1 Z +T /2
ck =
f (x) exp(ik x)dx
(9)
T T /2
where the integral is taken over one period (T ) of the function if it is periodic, or from to
+ if it is aperiodic. (An aperiodic function is regarded as a periodic one whose period is ).
For periodic functions the frequencies k used are just all multiples of the repetition frequency;
for aperiodic functions, all frequencies must be used. Note that the computed Fourier coefficients
ck are complex-valued.
38
|J,|cFJcxU94 |JF
+7?O
BDCFE"f,+/[#J^`%Z5,L!4=
% '6#&+/[#&)9;7!J% '6#&+? +- LP%k
N
BDCUE
T b p S BDN { ;7!J%3BDiCFES { % '6#KBDiCUE E
{4
/J#&)9 BDCFE7G&?ZY&+?Z- 5,#&;/4?(+7)U!$5,-.'<+-&+-.'<+7%}=D!$-}/=>5,#&;7? '<!$#@
BDCFE7k
N
p
BDCUE
T b S
BDNJ{;7!J%3BDiCFESz{|% '6#KBDiCUE E
{4
J
BDJE
S b[d|BDN p eN p E m Sd
BDN { N[{$E m ScBD { ez{$E m4
BDJE
{3x"
!J? +P?ZY,/4??ZY&+? +-(LP%O'6#[2J!J0<2&'#&\eN[]/[#&)z
/J-.+P/40<0Ov/J#&)23/J#&'<%ZY_OY&+#eN[{ TNJ{
/J#&)e { Tz{FH9K$J$zRZ7P6$ VzJR.44PJDJ,eDZ.PZ[
7&J.Z`z.J "D@$JZ[(PJ7&zg[J$VP.&<JK7$FDJ,
>
U JFXUx1Z||&J|
1.5
fseries( 1,x)
fseries( 7,x)
fseries(63,x)
rect(x)
0.5
-0.5
-1
-1.5
-1
0.15
0.2
1.5
fseries( 63,x)
fseries(163,x)
rect(x)
0.5
-0.5
-1
-1.5
-0.2
-0.15
-0.1
-0.05
0.05
'<\$5,-.+b,k'6f,f&%O8,Y&+#&!$LP+#&!&#
U
0.1
/4?}v/J#&)9bJd
H
bqH}=@
BDCUE
'*%"/40<% !;7!&#J? '6#&5&!$5&%"!$#9BDvq~ZbJdEVG?ZY&+%Z5,L!4=?ZY&+U!$5,-.'<+-&+-.'<+7%}'<%|+7,5,/40
? !@]B>CUE}/4?}/40<0
8,!J'6#[? %H
qH}NJ{g/[#&)nz{? +#&)?(!7+-.!g/4?0*+/1% ?O/4%"=/4% ?}/4%JiKH
g|qZJ
+7_O-:'*?(+5&% '6#&\ { '6#?ZY&+!$f[2$'<!$5&%}_}/^P=-.!$L?zY&+=j!$-(L5&06/4+=D!$-"%('#/[#&)9;7!J%k
@
BDCUE T N b p
S
{3x
T Nbp
T
_}'<?ZYk
B>NJ{;7!J%3BDiCUExSz{|% '6#B>iRCFE E
{
{
{ E
{
S N[{ B S b
Sz{ B b4' E
{4x
{ {
N p J b
i v { T BDN { { E [b
{ T BDNJ{S z {$E [b
!$f&% +-.2J+Jk { T {
),+#&![? +7%}?ZY&+;7!$L8&0<+7;7!$# 5&\$/4? +JG,/J#&)k
{PT bJ d l m:o @
BDCUE { r$C
p
p
B JE
B JE
_OY&+-.+
BDJE
|J,|cFJcxx1FJ}F1
"!$#
%!
( *)
!$#
+,.-0/213'145/-76*/8(9:45;<451=>45=?4:- 18@/AB$CEDFAHG$DI8@DKJML*N(O(P>QR9:45;TSVUXWMJ?L*N>Y[ZEP$3-0AHJ?L*N]\^P%QR9:45;_SUXW^J?L*NM`
ZEPIa
$b
@
B RE .y
y T
B.JE
p
&+-.;7'<% +=j!&-?ZY&+-.+/4),+-7k%ZY&!4_'6#&% +-:? '6#&\e+75,/1? '<!$#[P'#[? !+7,5,/4? '<!$#v%Z/1? '<%.U+7%
?ZY&+!$-:'*\['#,/10+7,5,/4? '<!$#&%H
*)
c!
f)
d!0# e
? *)
2g
ih
*)
2g
*)
ih
m
kjll
*)
ih
o
Hh
ll
ln
!$#
*)
4
3
saw(x)
trig.pts.8
trig.pts.64
2
1
0
-1
-2
-3
-4
-5
10
[r
X!
X!
X!
12
NJ{ T
{ T
X!
X!
X!
B.bJE
utwv
x!
dy
g4|R
X!
]| ]U[U
<z|{
}{
]
]
Y&+#&;7+/1%O@ BDCUE
v,k
S Sc
dm
T
|
bm m
?ZY&+;7!$-(-.+7%Z8,!&#&),'#&\`% '6#&+P% +-.'<+7%}'<%k
CBDd`CUE
T d % '6#B>CUES % '6# BD[CUE S % '6#K BDJCUE SX
/J#&)9/4?CgTcdxJb,k
d Tc[b S
f&% +-.2[+?ZY,/4?|!&#&+% +-.'<+7%};7!$#[2J+-.\J+7%=/4% ? +-
?ZY,/J#`?ZY&+!J?ZY&+-VG$_Y&+#`?ZY&+O24/4065&+7%O!1=?ZY&+
f,/4% '<%=>5,#&;7?('*!&#&% L/1? ;Y?ZY&+24/4065&+7%!4=]?zY&+=>5,#&;7? '<!$#/4?"?ZY&+8q+-.'<!$),'<;f,!&5,#&)U/J-.'<+7%H
[t
v/4%OC
st
JFJJ|VZJ
&?Z/J-:? '6#&\}=-.!$LX?zY&+|;7!$L8&0<+7% +-.'<+7%'6#+7,5,/4? '<!$#qG3L/[a4+ /";7Y,/J#&\J+!4=U% ;/40<+ ;7!$#&%('*),+/8,+-.'<!&),'<;=5,#&;7? '<!$# xB>CUE_'*?zY8,+-.'<!&)b
),+zR#&+9@
BDCUEPT BDC g[dE7G_OY&'<;7YY,/4%
8,+-:'*!&)ebJd
H
{ T bJ d l o @
BDCFE { r$C
o
d
T bJd l B &E { o r
B ,E T bJd l xB $E r
B.7$E
{ JdKH|+#&;7+Jk
_OY&+-.+ Tcid GF/J#&) B ,E
T
@
BDCFE T { {
{
B &E T B ,E d
$
e
e
f ?
d e
(
e *
B ,E
B.JE
e * $
twv
XG,?ZY&+#`_"+Y&!$8,+Jk
l B ,E r
l
B ,E T
BDCFE r&C
B ,E
e f $.t
e * (
e *
e
*
B.JE
B. JE
g
d e *
? (
7v
.v
J}F4JU
]
e*
[@
I
*'
g Jx|
,5,8,8,![% + @
BDCFE
B ,E7G xB>CUE
B ,E _OY,/4?=>5,#&;7?('*!&# KBDCUEY,/4%9/?z- /J#&%.=D!$- L
B ,E
T B ,E B ,E #&% +-.? '6#&\`'6#J? !`?ZY&+'6#[2J+-.% +?Z-(/J#&%.=D!$- Lk
BDCFE T bJ d l B ,E B ,E r
T bJ d lll @
B &E B JE r ,r $r
T bJ d lll @
B &E B &E r ,r $r
g 0h
"
e f
e f
g
e *
M
g
( ? ?
h
? ?
R-:!$L+75,/4?('*!&#e ,G&_"+?ZY&+#!$f&?Z/4'6#`?ZY&+zJUJ<$DJ!4=
@/J#&) k
B>CUE T l @
B $E xBDC &E r
B.JE
T @
BDCF E BDCUE
BDbJvJE
!$-7k
l @
B $E xBDCP &E r
B ,E B ,E
&'6LP'<06/J-.0<^Rk
@
B &E xB $E l B UE B UE r
c
J
>7
d X
d e
<~
_
C v G B ,ET
v Cv
N S
=
_ +L/Ja4+?ZY&+=5,#&;7? '<!$#%(^LLM+7?Z-.'<;/Jf,!&5&?Ov,k
@
BDCFE
T G B ,E|T N m bJS N m
bqH}/J5&% %('/[#`+7/JL8&0*+B% +7+U\$5,-:+$E7k
@
BDCUE T
B ,E T l r&C
T l B E r&C
@]B>CUE
Ts
"
l
d
*
F
F
(
@
M ?
BDbJJE
0.8
0.8
0.6
0.6
0.4
0.4
0.2
0.2
-2
-1.5
-1
-0.5
0.5
1.5
-2
-1.5
-1
-0.5
0.5
N G B ,E
T % '6#B ,N$E
,BDCUE
Ts m v CC &
DB b4$E
N
,N
!J?('*;7+?zY,/4?O/4%ON v,GU/[#&)!$f&% +-.2J+ B ,E ,_OY,/1?AY,/[8,8,+#&%? ! ,BDCUE
ctv
?
d [t
| eJ |FDJZ||&J
z
<t
(
1.5
[H";7!$#&;7+8&?Z5,/40<0<^
wB>CUE T s
CgTcv
C Tcv
o
bqH"?Z/Ja['#&\?ZY&+0<'LM'*?!1=+75,/1? '<!$#9bJqk
l w,BDCUE r&CTs v
v gv
u!
"
e!
c|
{
|FJF
[ s
"!$#&%('*),+-K%z/JL8&0<'6#&\ /% '<\$#,/40 xB>CUE/4?K'#[? +-.24/40<% ? !}!$f&?Z/1'#?ZY&+|),'<% ;-:+7? +"? '6LP+|% '<\$#,/10jk
BDCUE T xBDi eE w,BDCei E
BDbJJE
{
T xB>CUE w BDCFE
+LP+LPfq+-?ZY&+8,-.!$8q+-.? '<+7%g!1= ;7!$#[2J!J065&? '<!$#&%G"_ +9a&#&!4_?ZY,/4?/e8,-:!$)U5&;7?`'6#?zY&+9C
),!$L/1'#'<%9/;7!$#J2[!J065&? '<!$#'6#X?ZY&+?Z- /J#&%:=j!$-(L ),!$L/4'6#HWX'<?ZY BDCUE
B ,E`/J#&)
B ,E7k
BDCFE
B ,E T
B ,E
B ,E
B ,E w,B bJdh`E
T
y
T B P bJdh E
BDb JE
y
R-:!$L?ZY&'<%}_"+ #&!J?(+?ZY,/4?
=D!$-"/J#[^ BDCFE
B ,E?ZY&+-.+7%Z5&0<?}!4=%z/JL8&0<'6#&\ /4?"'6#[? +-.23/10*%
'#?ZY&+}C),!$L/4'6#-.+7%z5&0*?(%|'6#g/?z- /J#&%.=D!$- L
B ,EK_OY&'<;7Y'<%|?zY&+O8,+-.'<!&),'<;+7$? +#&%('*!&#
!4= B ,E_}'<?ZY8,+-.'<!&)nbJdx H
"!$#[2J+-.%(+70*^'t= BDCUE'<%P% ?Z-.'<;7? 0<^f,/J#&)&0<'6LP'<? +7)f[^
B>- /4),'6/J#&%8,+-C5,#&'<?ZE7GK?ZY,/1?'*%
B ,E'<%
7+-.!!$5&? % '<),+"?ZY&+"'6#J?(+-.23/40B. ~ E7G?ZY&+#PfJ^%z/JL8&0<'6#&\/4?K'6#J? +-:23/40<%|bJd[b
'6#?ZY&+C),!$L/4'6#k
B ,EGB. E
B ,E
T b bJd
/4%%zY&!3_#'6#U\$5,-:+n,H K+-=D!$- LP'6#&\e?ZY&+F!$5,-.'<+-?Z- /J#&%.=D!$- L !$#+75,/1? '<!$#bJqG|_"+
!$f&?Z/1'#k
B ,E
T b bJd xB bJb di E { m.o m GB. ` E
BDb JE
{
5&?_"+9a$#&!4_ B ,ET v=j!&-/10*0
"?zY&+-.+z=j!&-.+?ZY&+9%(+75&+#&;7+ xB>iR[b E
;7!$L8&0<+7? +70<^`),+zR#&+7% B ,E/J#&)gY&+#&;7+ xBDCFE7H % '6#&\P+75,/1? '<!$#gb /J#&)g8q+-=D !$- LP'6#&\P?ZY&+
'6#J2[+-.% +?z- /J#&%.=D!$- Lk
BDCUE T b[ d l B ,E r
l
T b[d b bJd xB id E { o
r
{
{o r
T b B b id E l
{
i
d
%
6
'
#
B
C
T xB E CiRdidE
BDbJJE
{
/JaJ'6#&\`?ZY&++7,/JL8&0<+P!4=
? '6LP+=D!$-"?zY&+ ),!$L/4'6#GF_"+;/J#-.+7_-.'<? +?ZY&'<%'6#?(+- LP%}!4=
?ZY&+`%Z/JL8&0<'6#&\=-.+7,5&+#&;7^ @ P#&!$- L/10*0<^5&![? +7)'6#+-.? - /4?ZY&+-?ZY,/J#- /4),'6/J#&%8,+% +7;Jk
B E T xB @ i E % '6#Kd|BDd|BD@ BD@ KeiEiRE(E
{
T xB @ i E.% '6#&;,BD@ KeiRE
BDJvJE
{
>
>
]
'
r e
*
e
X7
d e *
*
e
*
g
$F
Jv
]
$F
$e
*
[( F.^
d
. ^
]
-
>
^
!
--F
-"-F - e
]?
-
%
^]? . 10
2 - 43 R ]
. - ] % ^
_
"# %$'&
(*)+-,/
65^ 0
- c
"107'
- 8:9 <;>=
" # . ?. '
-
9 A@B]= !# DCFEG
H IJ O- "# '
K!# -F L M e _ 86 !#
$A&N(O)M+-,. !# -F
PRQSPT
@
UFVSWXY
W[
Z
8 0 q
[ 4 ? !# ]^
]( -]\ ?
# # \ ( # O^ _#
!
-F ^
5 ? - 4 $ #? ^ !# a` 0 5^? - -F !
# \ L ( -F e
- cb
'
` 0 5M?
4 ]?<
- g=]Bh a!# ^
i - ^$0i -F K^ -
!# ^
* #
T 1 3 -
-F 5M ?0 M
L -F 5^ ? #?# - !# ] \ #? #%!#
- 6 0
p
1
y = sinc(x)
0.8
0.6
0.4
0.2
-0.2
-0.4
-8
-6
-4
- gq
[? F
-2
"#
d
6
?
r6s1t6uOv*w
uOv
1.2
0.8
0.6
0.4
0.2
-W
[(
- :b
]x
=
_y
z
|{ <}'
"
@ \ 6_##?M #
-
!# g
!#
-F ^
5 ?
# O ^
- _
F %
~ ^ - ] ? @
8 H
\ - -F M
5 ? *
]?
DH
\ i ^ !# ` 0 5M?
" O^ ] !#? c
- h . -
4 ?k4 2
- ~ -Hz
?% - q
M
Ag ! #
# 4 0 ?4
^]? .A!# ^
](
^
4 (a n 2 3 ]?2
an 2 3
z
86 -
!# -F e
!#
?
- -
e ^
!
^ e
-F 5^ ? #?M#
y O
_X
!#
g4]RgA[iRg*]
# O ^
\ ]R e
# \H
_ _]? 1^ - 60 # \
^]? . ^
6 63 e \ #
^ -
g
- d ^
X
10_ e
-F 8
] ]
e
(
4 e
-
_# d8,/.]R)!+*|,/1!
"H
TQSP
ZWWRZY
- % i M
s i@ } }O*O a
5 ? F
-_z
!# (
-F
-F _# ^0
A'
- M ]]^
]( ^
J \ !
# sg@J
"#
e
^
]?
-
- - " -
-
_ s l
h
@"}Op}*O|}!C
jd @
sM
s
]
_ s
h
@"}Op}*O}_C
j:
_
-
^
! # e (
(
- - "
!# -F 7 :5^ e 8
4 ^ - ?
-F 86 6
"#
{ p
-
-
p 0 c
!#
0 ]?
-
".
. _#
( ^
0
- ]?c
- q \
'
- h
- \ !# . 8 _ #
c1R 88WS Y
sd@
TQT
{ p
-
] s
s
=
- 0
! #
a
- 6
-
F
]?
s'7s # 8
jd !#
"
?
# e - ^
- D ^5 ?
"#
s
@ jd
j
h j t
[ w @ j:t 4 w
% #?e~s ! # s 0^
-
60
s sO !#
" j: *u r s*
86 c!#
^ -
8
10
s
"8 E s
!#
@
@
-
-
]^
"# c
O+|O)MA_ O)M.+
M
1 ^ e
5
s
s
j
@
TQ
^ (
EOsc@
s h @"}Op}*O~C
"#
{ p
!#
_ s
EOsM
! s
s
_
_ ts w
s
XYORWXZY!R
jd
@
@
|
s
s
s
s
sN
sa
sN
sa
sN
{ p
h l @
}*p}O*=
|
s
s
sN
s
s
s
| { p
M
h @ { p h #
^ - 8
jd
^
jd
j
@
` e ! # d
!#
{
s
s s
-
jL
jd
(
]?
sa sN s
h @"}*p1}**[}! { p
sN
e s { sN s s h@"}*p}O*|}! { p
sN
\
]
"
A!#
- # \ !#
^ 6 ! #?
^
( F
]^
-}-
^
5
s: sN
-
!#?
C
]^
e
]?
] " 4d \
]
_ #
H( - - " - ] -
[( F
jd
"
_# ?
60F ^
C _
d
H
\ -
=h !# \H ]s
\H C
]
- ~ - *
"#
" *
[
"
]
] 4c \
]
"
d [ @ |{ p
]? i
\
[( F
"
M
jL
]
jd
M
jL
jL
\H
8!#
-
"
K ^
-
]
Stage 1
x
[( F
^
z
)..*f&
^ 4
Stage 2
Stage 3
#i
!#
_#
F
# \
!# \
. !#?
( - - 0 !#
7 !#
- 4'^
\ 4 -F !# (
_
H
] !# - - - *0 _?# 8 1 - !# A e 6 60 C
!^
$ - *
]
"R ? ! # 6 8 j:
_ # ["(
-(- O0 h n l
^
]
?
-
h -F - ! #
z 4
6 U
-F
-(- O0
] C
TQ
"#
OZY
%A
I^
-
"
sd@
"?# ]
P- O\ - 4
10
C s @
" ?#
!#? !#
!# _
_ s
h
@"}Op}*O}_C
j:
p
8
{ p
s
h
@"}Op}*O|}!C
j
{ p
L
"
_# M
z O ? 8 !# ? ] 8 ] 1^ - "
-
-
-
["
]? ?
4
-0
AA
-
q
_
84'
i
TQ
"!WZY
WZY
h H
\
~ \ #[ - !# \ ^
^ #
- e d
!# ? - - - $# " - ] - #g e - " - ] ~ - - ! # "m!#
]x e - d ? 6
\ ^0 - 9 }!E
M
uOrt6sOv &%(' ) w
*,+
p # \ a
[( -F
\H .M#1 -
p
-
)i
9
% sN
% +s
u*rt.0/ 3254 w
1
0.5
0.5
-0.5
-0.5
-0.5
- cp
[( F
\H
=
8 d
6
0
;:
g
g_
?
76
=<[
BC
C
C
C
>A
9 @
\O- 6 > @
1^ ( - z
-
!#
9*,+
9I +
9OE+
91 +
) +
9 i
!# 1 ^
-
FGFHF
FGFHF
) +
9 i
9OE+
91 +
FGFHFB9 )i+
-
8a!#
^0] D
> *N@ O
@ >QP )
+ )
@ > R
9 @ >QP 3
+ @
>A
@ >QP )S+ )
9 @ O
@
b
6
"
- !# 1
5M 860 \H #?M# -
C 8 C H
- 6_
C 8
] 7
# - ! # - !# ] ! # 7
?
4]RiRg*]
9
h
cos(x)*cos(y)
cos(y)
q
0.5
c M
5
cos(x)
"#
_# ^
@ ? e *
\ !
^ h ?
zh _#
JLK
K
K
K
M
"
!#
-
\ #
F >TP=U
+ U @X
!#
p
V
@
-
- 8
!#
;VW7V
z \ !#
-
% s
_
Y
- 1 ( * -
-F - ~ - ^
e
X
0
} @"}*p1}**O|Z} V { p
X
] ^ u10
>Q[ @
]?
>]\ @
h\H
-
!#
> *N@ ^
@ >Q[ @ > A
9 @ >]\ @
>A
@ >Q[ @
9 @ O
s
_ WaY `bDRY
WZ7aXZY!R1
0.5
0.45
0.4
DCT
DFT
Original
0.35
0.3
0.25
0.2
0.15
0.1
0.05
0
- cpp
[( F
"#
g " ^
"
4 * ^ ] n
z ?
!#
? E
[0 c
-
\ !#
e $
8R
-
4
- ^ d ? e \ #? #
=C
" #? 4F
0 !#
]^
-
- e
M
]?
-
10 ! # " "#
-
-
- 0
!?# !#
!# cCmW C
z? e h _# g " -F _# '9 @ 9 K!#
" !# - ] ^
e $ [ OEnN+po_.+ *.,h\ # -F - 1^ - 6 e \H
# O^
- z ? !# ?
# #F -F 5^ ?0
' h !#
z? e s ' -
(d4 # O^ !# ]<$^
a_#
z? :_# ?
$ # - \ -
y
[( -Fcpp # \ !# __X
-
T
QT
sqoXr!XXs!
RutXVSY0
XZY!R
BC
C
C
D
{ p
p
VyWzV
0
- zD\ #
' H -F .
'
0
JLK
{ p
p
{ p
{ p
{ p
42
{ p
p
p
M
2 $% - - ~ - >#
/
,. ^ - h
# O^
2 ]
x \ !#<!# " h ! #
\ !# _ }5{dp ` !# ]!# - ] - 6 # -F - M
-
- 4 M
A\ !# !# ] ! #
\ #
z - 8!# "
Q
@
@
|
]
ZRXV[Z(`WZY
-
H\ -F Mn g -
- !# - -
d ? e _ \H # *^ : h
zz
e
h ?c^
0i2$% - % \ !# \ ?
h
# #4 -F
-F
e
bDRZRVSWZ
h]10
-
4
] e
1^ ^
!#
1 ^ e
G&
8 e = h\ # -F !# .
# - 3
" #? 1^ ^ [H - 6
!# - R
?
$
^
}
_
E
!# @ e 9
!# ! #
!# [ ! #
4
' h
9 % 9 E h ?u -F e ] e 9 E 9 %F
- !#
-
- 60L^ -
-
4'
200
function
gradient
150
100
50
-50
-100
50
100
- cp*=
[( F
6
`
!#?
v 9 !} E
' 9 !
}E
_ ^ 6
!#
!#? 7
e
150
200
0
c X
y -F
T
250
1 ^ e
5
9 !} E R{ 9 c{ p }_E
9 }!E R{ 9 _} E { p
9s !#
s]? E
? . ^ e
\ !#
F e
300
0
{ p}Op}'}2
}
pr
-
=
9 }_E h - M - -F ? - ^
}!E}e .M #id _#
e
- 4 ^ - ? \H 4
^ - 8X yK- e
5M 6 ^ ^ !#
E
q]
\ #? #F# $ e
z? ^08
e f - !# - -F
- 7
]
? 8 y 6
- !# e
4 !# K\i n \
X
* $ ]
?
M
9000
P0903A.DAT
P0903B.DAT
8000
7000
6000
2
5000
1
4000
0
3000
-1
2000
-2
1000
-3
-4
-1000
-5
-2000
0
50
100
150
200
250
300
61
50
100
150
200
250
300
We have now encountered several theorems expressing the idea that even though a signal
is continuous and dense in time (i.e. the value of the signal is defined at each real-valued
moment in time), nevertheless a finite and countable set of discrete numbers suffices to
describe it completely, and thus to reconstruct it, provided that its frequency bandwidth is
limited.
Such theorems may seem counter-intuitive at first: How could a finite sequence of numbers, at discrete intervals, capture exhaustively the continuous and uncountable stream of
numbers that represent all the values taken by a signal over some interval of time?
In general terms, the reason is that bandlimited continuous functions are not as free to
vary as they might at first seem. Consequently, specifying their values at only certain
points, suffices to determine their values at all other points.
Three examples that we have already seen are:
Nyquists Sampling Theorem: If a signal f (x) is strictly bandlimited so that it
contains no frequency components higher than W , i.e. its Fourier Transform F (k)
satisfies the condition
F (k) = 0 for |k| > W
(1)
then f (x) is completely determined just by sampling its values at a rate of at least
2W . The signal f (x) can be exactly recovered by using each sampled value to fix the
amplitude of a sinc(x) function,
sinc(x) =
sin(x)
x
(2)
whose width is scaled by the bandwidth parameter W and whose location corresponds
to each of the sample points. The continuous signal f (x) can be perfectly recovered
from its discrete samples fn ( n
W ) just by adding all of those displaced sinc(x) functions
together, with their amplitudes equal to the samples taken:
f (x) =
X
n
fn
n
W
sin(W x n)
(W x n)
(3)
Thus we see that any signal that is limited in its bandwidth to W , during some
duration T has at most 2W T degrees-of-freedom. It can be completely specified by
just 2W T real numbers (Nyquist, 1911; R V Hartley, 1928).
Logans Theorem: If a signal f (x) is strictly bandlimited to one octave or less, so
that the highest frequency component it contains is no greater than twice the lowest
frequency component it contains
kmax 2kmin
(4)
(5)
(6)
and
and if it is also true that the signal f (x) contains no complex zeroes in common with
its Hilbert Transform (too complicated to explain here, but this constraint serves to
62
(7)
Comments:
(1) This is a very complicated, surprising, and recent result (W F Logan, 1977).
(2) Only an existence theorem has been proven. There is so far no stable constructive
algorithm for actually making this work i.e. no known procedure that can actually
recover f (x) in all cases, within a scale factor, from the mere knowledge of its zerocrossings f (x) = 0; only the existence of such algorithms is proven.
(3) The Hilbert Transform constraint (where the Hilbert Transform of a signal
is obtained by convolving it with a hyperbola, h(x) = 1/x, or equivalently by shifting
the phase of the positive frequency components of the signal f (x) by +/2 and shifting
the phase of its negative frequency components by /2), serves to exclude ensembles of signals such as a(x) sin(x) where a(x) is a purely positive function a(x) > 0.
Clearly a(x) modulates the amplitudes of such signals, but it could not change any
3
of their zero-crossings, which would always still occur at x = 0, , 2
, , ..., and so
such signals could not be uniquely represented by their zero-crossings.
(4) It is very difficult to see how to generalize Logans Theorem to two-dimensional
signals (such as images). In part this is because the zero-crossings of two-dimensional
functions are non-denumerable (uncountable): they form continuous snakes, rather
than a discrete and countable set of points. Also, it is not clear whether the one-octave
bandlimiting constraint should be isotropic (the same in all directions), in which case
the projection of the signals spectrum onto either frequency axis is really low-pass
rather than bandpass; or anisotropic, in which case the projection onto both frequency
axes may be strictly bandpass but the different directions are treated differently.
(5) Logans Theorem has been proposed as a significant part of a brain theory
by David Marr and Tomaso Poggio, for how the brains visual cortex processes and
interprets retinal image information. The zero-crossings of bandpass-filtered retinal
images constitute edge information within the image.
The Information Diagram: The Similarity Theorem of Fourier Analysis asserts
that if a function becomes narrower in one domain by a factor a, it necessarily becomes
broader by the same factor a in the other domain:
f (x) F (k)
(8)
1
k
(9)
f (ax) | |F
a
a
The Hungarian Nobel-Laureate Dennis Gabor took this principle further with great
insight and with implications that are still revolutionizing the field of signal processing
(based upon wavelets), by noting that an Information Diagram representation of signals in a plane defined by the axes of time and frequency is fundamentally quantized.
There is an irreducible, minimal, area that any signal can possibly occupy in this
plane. Its uncertainty (or spread) in frequency, times its uncertainty (or duration) in
time, has an inescapable lower bound.
63
10
10.1
If we define the effective support of a function f (x) by its normalized variance, or the
normalized second-moment:
(x)2 =
f (x)f (x)(x )2 dx
+
(10)
f (x)f (x)dx
xf (x)f (x)dx
Z +
(11)
f (x)f (x)dx
and if we similarly define the effective support of the Fourier Transform F (k) of the function
by its normalized variance in the Fourier domain:
(k) =
F (k)F (k)(k k0 )2 dk
+
(12)
F (k)F (k)dk
where k0 is the mean value, or normalized first-moment, of the Fourier transform F (k):
k0 =
kF (k)F (k)dk
Z +
(13)
F (k)F (k)dk
then it can be proven (by Schwartz Inequality arguments) that there exists a fundamental
lower bound on the product of these two spreads, regardless of the function f (x):
(x)(k)
1
4
(14)
10.2
Gabor Logons
Dennis Gabor named such minimal areas logons from the Greek word for information, or
order: logos. He thus established that the Information Diagram for any continuous signal
can only contain a fixed number of information quanta. Each such quantum constitutes
an independent datum, and their total number within a region of the Information Diagram
represents the number of independent degrees-of-freedom enjoyed by the signal.
64
The unique family of signals that actually achieve the lower bound in the Gabor-HeisenbergWeyl Uncertainty Relation are the complex exponentials multiplied by Gaussians. These
are sometimes referred to as Gabor wavelets:
f (x) = e(xx0 )
2 /a2
eik0 (xx0 )
(15)
F (k) = e(kk0 )
2 a2
eix0 (kk0 )
(16)
Note that in the case of a wavelet (or wave-packet) centered on x 0 = 0, its Fourier Transform is simply a Gaussian centered at the modulation frequency k0 , and whose size is 1/a,
the reciprocal of the wavelets space constant.
Because of the optimality of such wavelets under the Uncertainty Principle, Gabor (1946)
proposed using them as an expansion basis to represent signals. In particular, he wanted
them to be used in broadcast telecommunications for encoding continuous-time information. He called them the elementary functions for a signal. Unfortunately, because such
functions are mutually non-orthogonal, it is very difficult to obtain the actual coefficients
needed as weights on the elementary functions in order to expand a given signal in this
basis. The first constructive method for finding such Gabor coefficients was developed
in 1981 by the Dutch physicist Martin Bastiaans, using a dual basis and a complicated
non-local infinite series.
The following diagrams show the behaviour of Gabor elementary functions both as complex
wavelets, their separate real and imaginary parts, and their Fourier transforms. When a
family of such functions are parameterized to be self-similar, i.e. they are dilates and translates of each other so that they all have a common template (mother and daughter),
then they constitute a (non-orthogonal) wavelet basis. Today it is known that an infinite
class of wavelets exist which can be used as the expansion basis for signals. Because of
the self-similarity property, this amounts to representing or analyzing a signal at different
scales. This general field of investigation is called multi-resolution analysis.
65
Gabor wavelets are complex-valued functions, so for each value of x we have a phasor in
the complex plane (top row). Its phase evolves as a function of x while its magnitude grows
and decays according to a Gaussian envelope. Thus a Gabor wavelet is a kind of localised
helix. The difference between the three columns is that the wavelet has been multiplied
by a complex constant, which amounts to a phase rotation. The second row shows the
projection of its real and imaginary parts (solid and dotted curves). The third row shows
its Fourier transform for each of these phase rotations. The fourth row shows its Fourier
power spectrum which is simply a Gaussian centred at the wavelets frequency and with
width reciprocal to that of the wavelets envelope.
66
The first three rows show the real part of various Gabor wavelets. In the first column,
these all have the same Gaussian envelope, but different frequencies. In the second column,
the frequencies correspond to those of the first column but the width of each of the Gaussian
envelopes is inversely proportional to the wavelets frequency, so this set of wavelets form a
self-similar set (i.e. all are simple dilations of each other). The bottom row shows Fourier
power spectra of the corresponding complex wavelets.
67
2D Fourier Transform
Z
0.5
Z
0.5
0.5
10
Sp
Po
0.5
ati
e
gre
sit
ion 0
in
De
gre
es
-0.
5 -0.5 Po
io
sit
0 e
D
in
al
Fre 0
qu
en
10
)
PD
cy
0 cy (C
n
ue
req
lF
PD 10 -10 atia
)
Sp
(C
Figure 1: The real part of a 2-D Gabor wavelet, and its 2-D Fourier transform.
10.3
An effective strategy for extracting both coherent and incoherent image structure is the
computation of two-dimensional Gabor coefficients for the image. This family of 2-D filters
were originally proposed as a framework for understanding the orientation-selective and
spatial-frequency-selective receptive field properties of neurons in the brains visual cortex,
and as useful operators for practical image analysis problems. These 2-D filters are conjointly optimal in providing the maximum possible resolution both for information about
the spatial frequency and orientation of image structure (in a sense what), simultaneously with information about 2-D position (where). The 2-D Gabor filter family uniquely
achieves the theoretical lower bound for joint uncertainty over these four variables, as dictated by the inescapable Uncertainty Principle when generalized to four-dimensional space.
These properties are particularly useful for texture analysis because of the 2-D spectral
specificity of texture as well as its variation with 2-D spatial position. A rapid method
for obtaining the required coefficients on these elementary expansion functions for the purpose of representing any image completely by its 2-D Gabor Transform, despite the nonorthogonality of the expansion basis, is possible through the use of a relaxation neural network. A large and growing literature now exists on the efficient use of this non-orthogonal
expansion basis and its applications.
Two-dimensional Gabor filters over the image domain (x, y) have the functional form
f (x, y) = e[(xx0 )
2 /2 +(yy
0)
2 / 2
(17)
(u0 , v0 ) specify modulation, which has spatial frequency 0 = u20 + v02 and direction 0 =
arctan(v0 /u0 ). (A further degree-of-freedom not included above is the relative orientation of
the elliptic Gaussian envelope, which creates cross-terms in xy.) The 2-D Fourier transform
68
F (u, v) of a 2-D Gabor filter has exactly the same functional form, with parameters just
interchanged or inverted:
F (u, v) = e[(uu0 )
2 2 +(vv
0)
22
(18)
The real part of one member of the 2-D Gabor filter family, centered at the origin (x 0 , y0 ) =
(0, 0) and with unity aspect ratio / = 1 is shown in the figure, together with its 2-D
Fourier transform F (u, v).
2-D Gabor functions can form a complete self-similar 2-D wavelet expansion basis, with
the requirements of orthogonality and strictly compact support relaxed, by appropriate
parameterization for dilation, rotation, and translation. If we take (x, y) to be a chosen
generic 2-D Gabor wavelet, then we can generate from this one member a complete selfsimilar family of 2-D wavelets through the generating function:
mpq (x, y) = 22m (x0 , y 0 )
(19)
y 0 = 2m [x sin() + y cos()] q
(20)
(21)
It is noteworthy that as consequences of the similarity theorem, shift theorem, and modulation theorem of 2-D Fourier analysis, together with the rotation isomorphism of the 2-D
Fourier transform, all of these effects of the generating function applied to a 2-D Gabor
mother wavelet (x, y) = f (x, y) have corresponding identical or reciprocal effects on its
2-D Fourier transform F (u, v). These properties of self-similarity can be exploited when
constructing efficient, compact, multi-scale codes for image structure.
10.4
Until now we have viewed the space domain and the Fourier domain as somehow opposite, and incompatible, domains of representation. (Their variables are reciprocals; and the
Uncertainty Principle declares that improving the resolution in either domain must reduce
it in the other.) But we now can see that the Gabor domain of representation actually
embraces and unifies both of these other two domains. To compute the representation of
a signal or of data in the Gabor domain, we find its expansion in terms of elementary
functions having the form
2 /a2
(22)
The single parameter a (the space-constant in the Gaussian term) actually builds a continuous bridge between the two domains: if the parameter a is made very large, then the
second exponential above approaches 1.0, and so in the limit our expansion basis becomes
(23)
the ordinary Fourier basis. If the frequency parameter k0 and the size parameter a are
instead made very small, the Gaussian term becomes the approximation to a delta function
at location xo , and so our expansion basis implements pure space-domain sampling:
lim f (x) = (x x0 )
k0 ,a0
(24)
Hence the Gabor expansion basis contains both domains at once. It allows us to make
a continuous deformation that selects a representation lying anywhere on a one-parameter
continuum between two domains that were hitherto distinct and mutually unapproachable.
A new Entente Cordiale, indeed.
69
Reconstruction of Lena: 25, 100, 500, and 10,000 Two-Dimensional Gabor Wavelets
70
11
(25)
(26)
An interesting theorem, called the Strong Law of Large Numbers for Incompressible Sequences, asserts that the proportions of 0s and 1s in any incompressible string must be
nearly equal! Moreover, any incompressible sequence must satisfy all computable statistical
71
tests for randomness. (Otherwise, identifying the statistical test for randomness that the
string failed would reduce the descriptive complexity of the string, which contradicts its
incompressibility.) Therefore the algorithmic test for randomness is the ultimate test, since
it includes within it all other computable tests for randomness.
12
A Short Bibliography
Cover, T M and Thomas, J A (1991) Elements of Information Theory. New York: Wiley.
Blahut, R E (1987) Principles and Practice of Information Theory. New York: AddisonWesley.
Bracewell, R (1995) The Fourier Transform and its Applications. New York: McGrawHill.
Fourier analysis: besides the above, an excellent on-line tutorial is available at:
http://users.ox.ac.uk/ball0597/Fourier/
McEliece, R J (1977) The Theory of Information and Coding: A Mathematical Framework
for Communication. Addison-Wesley. Reprinted (1984) by Cambridge University Press in
Encyclopedia of Mathematics.
McMullen, C W (1968) Communication Theory Principles. New York: MacMillan.
Shannon, C and Weaver, W (1949) Mathematical Theory of Communication. Urbana:
University of Illinois Press.
Wiener, N (1948) Time Series. Cambridge: MIT Press.
72