(Chapters 9 and 10 in Papoulis)
1.5 weeks
• Linear Filtering of a Random Signals
• Power Spectrum Analysis
1
(Deterministic) Systems with Stochastic Inputs
A deterministic system1 transforms each input waveform X (t,ξ i ) into
an output waveform Y (t, ξ i ) = L[X(t, ξ i )] by operating only on the
time variable t. Thus a set of realizations at the input corresponding
to a process X(t) generates a new set of realizations {Y (t,ξ )} at the
output associated with a new process Y(t).
Y (t, ξ i )
X (t, ξ i )
X (t )
!!
!→
L[⋅]
(t )
!Y!→
!
t
t
Our goal is to study the output process statistics in terms of the input
process statistics and the system function.
1A stochastic
system on the other hand operates on both the variables t and ξ
PILLAI
2
Deterministic System (not necessarily linear)
Deterministic Systems
Memoryless Systems
Y (t ) = g[ X (t )]
Timevarying
systems
Systems with Memory
TimeInvariant
systems
Linear systems
Y (t ) = L[ X (t )]
LinearTime Invariant
(LTI) systems
X (t )
h (t )
LTI system
+∞
Y (t ) = ∫ h(t − τ ) X (τ )dτ
−∞
(61)
+∞
= ∫ h(τ ) X (t − τ )dτ . = X (t ) ∗ h(t )
−∞
3
Memoryless Systems (not necessarily linear)
The output Y(t) in this case depends only on the
present value of the input X(t). i.e., Y (t ) = g{ X .(t )}
Strictsense
stationary input
Memoryless
system
Strictsense
stationary output.
Widesense
stationary input
Memoryless
system
Need not be
stationary in
any sense.
X(t) stationary
Gaussian
Memoryless
system
Note: the filters are not random
Y(t) stationary, but
not necessarily
Gaussian
4
LTI Systems: WSS input good enough
X (t )
widesense
stationary process
LTI system
h(t)
Y (t )
widesense
stationary process.
(a)
X (t )
strictsense
stationary process
LTI system
h(t)
(b)
X (t )
Gaussian
process (also
stationary)
Linear system
Y (t )
strictsense
stationary process
Y (t )
Gaussian process
(also stationary)
(c)
5
Linear TimeInvariant (LTI) Continuous Systems
TimeInvariant System
Shift in the input results in the same shift in the output.
Y (t ) = L{X (t )} ⇒ L{X (t − t0 )} = Y (t − t0 )
(62)
Linear TimeInvariant System: linear system with timeinvariant property.
• Linear system has two important properties: superposition and
homogeneity
L{a1 X (t1 ) + a2 X (t2 )} = a1 L{ X (t1 )} + a2 L{ X (t2 )}.
∫
∞
−∞
δ (t )dt = 1
δ (t )
h(t ) = L{δ (t )}
LTI
h(t )
h (t )
(63)
Impulse
response of
the system
t
Impulse
Impulse
response
6
Linear (LTI) Filtering of a Random Signal
Y (t )
X (t )
t
X (t )
t
Y (t )
LTI
+∞
Y (t ) = ∫ − ∞ h(t − τ ) X (τ )dτ
arbitrary
input
+∞
+∞
X (t ) = ∫ − ∞ X (τ )δ (t − τ )dτ
(64a)
(61)
= ∫ − ∞ h(τ ) X (t − τ )dτ
+∞
Y (t ) = L{ X (t )} = L{∫ − ∞ X (τ )δ (t − τ )dτ }
Note: the filter
is not random.
+∞
= ∫ − ∞ L{ X (τ )δ (t − τ )dτ }
By Linearity
+∞
= ∫ − ∞ X (τ ) L{δ (t − τ )}dτ
+∞
By Timeinvariance
+∞
= ∫ − ∞ X (τ )h (t − τ )dτ = ∫ − ∞ h(τ ) X (t − τ )dτ .
(64b)
7
Mean of the Output
• Expectation and linear filtering are both linear operations
that are interchangeable.
• Not all linear operations are interchangeable (e.g., matrix
multiplication).
• The mean of the output of a LTI filter is the convolution of
the mean of the input with the the filter impulse response:
∞
$
E[Y (t)] = E & ∫ h(τ )X(t − τ )d τ ') =
% −∞
(
= E[X(t)]∗ h(t)
∫
∞
−∞
h(τ )E[X(t − τ )]d τ
(65)
• Question: suppose the input is a deterministic function s(t)
plus a zeromean random input X(t), what is the mean of
the filter output.
8
Filter Output With a Wide Sense Stationary (WSS) Input
• If the input to an LTI filter with impulse response h(t) is a WSS
process X(t), the output Y(t) has the following properties:
• Y(t) is a WSS process with expected value and autocorrelation
∞
function
(66)
µY = E[Y (t )] = µ X ∫ h(τ )dτ
−∞
∞
∫ h(u) ∫ h(v)R
RY (τ ) =
X
−∞
•
∞
(τ + v − u)dudv
(67)
−∞
X(t) and Y(t) are jointly WSS and have I/O crosscorrelation
RXY (τ ) =
∫
∞
−∞
h(u)RX (τ − u)du = RX (τ ) ⊗ h(t)
(68)
• The output autocorrelation is related to the I/O crosscorrelation
by
RY (τ ) =
∫
∞
−∞
h(−w)RXY (τ − w)dw = RXY (τ ) ⊗ h(−τ ) (69)
9
Proof:
∞
To show that
we recall that
RY (τ ) =
Y (t) =
∞
∫ h(u) ∫ h(v)R
X
−∞
(τ + v − u)dudv
−∞
(610)
∫ X(t − u)h(u)du = ∫ h(t − u)X(u)du
So,
RY (τ ) = E[Y (t + τ )Y (t)] = E #$ ∫
∫ X(t + τ − u)X(t − v)(u)h(u)h)v)du dv%&
∫ ∫ E [ X(t + τ − u)X(t − v)]h(u)h)v)dudv
= ∫ ∫ R (τ + v − u)h(u)h)v)dudv
=
(611)
X
See if you can prove the other relationships on the previous chart
on your own !. If not, see Papoulis.
10
Linear Filtering of a Discrete Time Random Signal
• If the input to a discretetime LTI filter with impulse response hn is
a WSS random sequence, Xn, the output Yn has the following
properties.
• Yn is a WSS random sequence with expected value and autocorrelation
function
µY = E[Yn ] = µ X
RY [n] =
∞
∞
∑ hn .
(612)
n = −∞
∞
∑ ∑h h R
i
j
X
[n + i − j ].
(613)
i = −∞ j = −∞
• Yn and Xn are jointly WSS with I/O crosscorrelation
RXY [n] =
∞
∑h R
i
X
[n − i ].
(614)
i = −∞
• The output autocorrelation is related to the I/O crosscorrelation by
RY [n] =
∞
∑h
−i
i = −∞
RXY [n − i ].
(615)
11
Power Spectrum Analysis
Definition: Fourier Transform
Functions g (t ) and G ( f ) are a Fourier transform pair if
∞
G ( f ) = ∫ g (t )e
−∞
− j 2πft
dt ,
∞
g (t ) = ∫ G ( f )e j 2πft df
(616)
−∞
Definition: Power Spectral Density
The power spectral density function of the WSS stochastic
process X(t) is defined as
1 $
2&
SX ( f ) ≡ lim
E % XT ( f ) '
T→∞ 2T
(617)
2
&
1 $ T
− j 2 π ft
= lim
E * ∫ X(t)e
dt +.
−T
T→∞ 2T
%
'
12
Power Spectrum Density (PSD) and the Autocorrelation
Function are a Fourier Transform Pair
If X(t) is a WSS stochastic process, RX (τ ) and SX ( f ) are the
Fourier transform pair
SX ( f ) =
∫
∞
−∞
RX (τ )e
− j 2π f τ
dτ ,
RX (τ ) =
∫
∞
−∞
SX ( f )e j 2 π f τ df
(618)
We will use the symbol ℑ to denote the Fourier Transform
This is a major result, called the WienerKhinchin Theorem.
On the next charts we will prove this important result.
13
Proof that the PSD and Autocorrelation are Fourier Pairs
For a deterministic signal x(t), the spectrum is well defined: If X (ω )
represents its Fourier transform, i.e., if
+∞
(619)
X (ω ) = ∫ −∞ x(t )e− jωt dt ,
then  X (ω ) 2 represents its energy spectrum. This follows from
Parseval s theorem since the signal energy is given by
+∞
∫ −∞
x (t )dt = 21π
2
+∞
2

X
(
ω
)

dω = E.
∫ −∞
(620)
Thus  X (ω ) 2 Δω represents the signal energy in the band (ω , ω + Δω )
(see Figure below).
2
 X (ω )
X (t )
0
t
0
Energy in (ω ,ω +Δω )
ω ω + Δω
ω
PILLAI
14
However for stochastic processes, the Fourier relationship
generates a sequence of random variables for every ω. Moreover,
for a stochastic process, E{ X(t) 2} represents the ensemble average
power (instantaneous energy) at the instant t.
To obtain the spectral distribution of power versus frequency for
stochastic processes, it is best to avoid infinite intervals to begin with,
and start with a finite interval (– T, T ) so that the Fourier Transform
of a process X(t) based over the interval (– T, T ) is given by
T
(621)
X T (ω ) = ∫ −T X (t )e− jωt dt
so that the energy per unit time, or the power spectral density (PSD)
2
(622)
 X T (ω ) 2 1 T
− jω t
=
X (t )e dt
∫
−T
2T
2T
represents the power distribution associated with that realization based
on (– T, T ). Notice that (622) represents a random variable for every
ω and its ensemble average gives, the average power distribution
based on (– T, T ). Thus the power spectral density is given by
15
#  X T (ω ) 2 $ 1 T T
− jω ( t1 − t2 )
*
PT (ω ) = E %
=
E
{
X
(
t
)
X
(
t
)}
e
dt1dt2
&
1
2
∫
− T ∫− T
' 2T
( 2T
1 T T
− jω ( t1 − t2 )
=
R
(
t
,
t
)
e
dt1dt2
(623)
∫
− T ∫− T XX 1 2
2T
represents the power distribution of X(t) based on (– T, T ). For wide
sense stationary (WSS) processes, it is possible to further simplify
(623). Thus if X(t) is assumed to be WSS, then RXX (t1 , t2 ) = RXX (t1 − t2 )
and (623) simplifies to
1
PT (ω ) =
2T
T
T
− jω ( t1 − t2 )
R
(
t
−
t
)
e
dt1dt2 .
∫ −T ∫ −T XX 1 2
(624)
Let τ = t1 − t2 and using this change of variables we have
1 2T
− jωτ
PT (ω ) =
R
(
τ
)
e
(2T −  τ )dτ
∫
− 2 T XX
2T
2T
= ∫ − 2T RXX (τ )e − jωτ (1 − 2τT )dτ ≥ 0
to be the power distribution of the WSS. process X(t) over
(– T, T ). Finally letting T → ∞ in (625), we obtain
(625)
PILLAI
16
+∞
S XX (ω ) = lim PT (ω ) = ∫−∞ RXX (τ )e− jωτ dτ ≥ 0
T →∞
(626)
to be the power spectral density of the WSS process X(t). Notice that
"↓$$ (')!!↔!!!!!!!!!!*.+!!!!!!!!┴!!↓$$ (.)≥0 (627)
i.e., the autocorrelation function and the power spectrum of a WSS
Process form a Fourier transform pair, a relation known as the
WienerKhinchin Theorem. From (627), the inverse formula gives
RXX (τ ) = 21π
+∞
jωτ
S
(
ω
)
e
dω
∫−∞ XX
(628)
and in particular for τ = 0, we an expression for the total power
1
2π
+∞
∫ −∞ S
2
(
ω
)
d
ω
=
R
(0)
=
E{
X
(t)

}= P
XX
XX
(629)
From (629), the area under S XX (ω ) represents the total power of the
process X(t), and hence S XX (ω ) truly represents the power
spectrum. (See the figure on the next chart).
PILLAI
17
S XX (ω )
S XX (ω ) Δω represents the power
in the band (ω , ω + Δω )
ω
ω ω + Δω
0
The nonnegativedefiniteness property of the autocorrelation function
translates into the nonnegative property for its Fourier
transform (power spectrum), since from (628)
n
n
n
n
∑∑ ai a RXX (ti − t j ) = ∑∑ ai a 21π
i =1 j =1
*
j
And it follows that
i =1 j =1
= 21π
*
j
+∞
∫−∞ S
+∞
∫−∞ S XX (ω )e
n
XX
(ω ) ∑i =1 ai e
jω ti
jω ( ti − t j )
2
dω
d ω ≥ 0.
RXX (τ ) nonnegative  definite ⇔ S XX (ω ) ≥ 0.
(630)
(631)
PILLAI
18
Since X(t) is a real WSS process, then RXX (τ ) = RXX (−τ ) so that
+∞
S XX (ω ) = ∫ −∞ RXX (τ )e − jωτ dτ
+∞
= ∫ −∞ RXX (τ ) cos ωτ dτ
∞
= 2 ∫ 0 RXX (τ ) cos ωτ dτ = S XX ( −ω ) ≥ 0
(632)
so that the power spectrum is an even function, (in addition to being
real and nonnegative).
PILLAI
19
Power Spectral Density Summary
• For a WSS stochastic process, X(t), the Power Spectral Density
(PSD), Sx(f), and the Autocorrelation function, RX(τ) are a Fourier
Transform pair.
• This is celebrated formula and is known as the WienerKhinchin
Theorem
• The PSD, Sx(f), is a realvalued function with the following
properties:
(a) SX ( f ) ≥ 0 for all f
∞
2
"
S
(
f
)df
=
E
X
∫ −∞ X
# (t)$% = RX (0)
(c) SX (− f ) = SX ( f )
(b)
(633)
20
Noise in Communication Systems
•
•
Thermal noise is described by a zeromean Gaussian random
process, n(t).
If the PSD is constant, then it is called white noise..
[w/Hz]
Power spectral
density
Autocorrelation
function
Probability density function
21
White Noise Process and LTI Systems
• W(t) is said to be widesense stationary (w.s.s) white noise if
E[W(t)] = constant, and
RWW (t1 , t2 ) = qδ (t1 − t2 ) = qδ (τ ).
• The Power Spectral Density of white noise is “flat” as a function of
frequency (where W"∞)
[w/Hz]
• If W(t) is also a Gaussian process, then all of its samples are
independent RVs
Colored noise
LTI
White noise
N (t ) = h (t ) ∗W (t )
h(t)
W(t)
22
Power Spectral Density of a Discrete Time Random Sequence
• The discretetime Fourier Transform (DTFT) is defined as
∞
X(ϕ ) =
∑ xn e
− j 2 πϕ n
,
∫
xn =
n=−∞
1/2
−1/2
X(ϕ )e j 2 πϕ n dϕ (634)
• The power spectral density function of the WSS random sequence Xn is
2(
% L
1
SX (ϕ ) = lim
E ' ∑ X n e− j 2 πϕ n * .
L→∞ 2L +1 '
*)
& n=−L
(635)
• The discretetime Wiener Khinchin formula is
If X n is a WSS stochastic process, RX [k ] and S X (φ ) are
a discrete  time Fourier transform pair :
S X (φ ) =
∞
∑ RX [k ]e
k = −∞
− j 2πφk
,
1/ 2
RX [k ] = ∫
−1 / 2
S X (φ )e j 2πφk dφ
(636)
23
Properties of the PSD for a Discrete Time WSS RP
For a WSS random sequence X n , the power spectral density S X (φ ) has
the following properties :
(a) S X (φ ) ≥ 0 for all φ ,
1/ 2
(b) ∫
1 / 2
[ ]
2
S X (φ )dφ = E X n = RX [0] .
(637)
(c) S X (−φ ) = S X (φ ),
(d) for any integer n, S X (φ + n) = S X (φ ).
24
Power Spectra and Linear Systems
If a WSS process X(t) with autocorrelation
h(t)
function RXX (τ ) ↔ S XX ( f ) ≥ 0 is
X(t)
Y(t)
applied to a linear system with impulse
response h(t), then the cross correlation
function RXY (τ ) and the output autocorrelation function RYY (τ ) are
given below:
RXY (τ ) = RXX (τ ) ∗ h* (−τ ), RYY (τ ) = RXX (τ ) ∗ h* ( −τ ) ∗ h(τ ). (638)
But if
f (t ) ↔ F (ω ),
g (t ) ↔ G(ω )
(639)
Then
since
f (t ) ∗ g (t ) ↔ F (ω )G(ω )
ℑ{ f (t) ∗ g(t)} =
+∞
∫ −∞
f (t) ∗ g(t)e − jωt dt
(640)
(641)
PILLAI
25
+∞
ℑ{ f (t) ∗ g(t)}= ∫ −∞
{∫
+∞
= ∫ −∞
+∞
−∞
}
f (τ )g(t − τ )d τ e − jωt dt
(642)
f (τ )e − jωτ d τ ∫ −∞ g(t − τ )e − jω (t−τ ) d (t − τ )=F (ω )G(ω ).
+∞
Using the above in (638), we get
S XY (ω ) = ℑ{RXX (ω ) ∗ h* (−τ )} = S XX (ω )H * (ω )
(643)
and since
+∞
*
∫ −∞ h (−τ )e
where
− jωτ
dτ =
+∞
(
+∞
∫ −∞ h(t )e
− jω t
dt
*
)
= H * (ω ),
H (ω ) = ∫ −∞ h(t )e− jωt dt
(644)
(645)
represents the transfer function of the system, and we find
SYY (ω ) = ℑ{RYY (τ )} = S XY (ω )H (ω ) = S XX (ω )  H (ω ) 2 .
(646)
The above (646) is very useful: the PSD of the output signal
is the product of the PSD of the input and the magnitude
squared of the transfer function.
PILLAI
26
The output power spectrum is real and nonnegative and is
related to the input spectrum and the system transfer function as in
(646), which can be used for system identification as well.
WSS White Noise Process: If W(t) is a WSS white noise process,
then
RWW (τ ) = qδ (τ ) ⇒ SWW (ω ) = q.
(647)
Thus the spectrum of a white noise process is flat, thus justifying its
name. Notice that a white noise process is unrealizable since its total
power is arbitrarily large (" ∞).
From (646), if the input to an unknown system is
a white noise process, then the output spectrum is given by
SYY (ω ) = q  H (ω ) 2
(648)
Notice that the output spectrum captures the system transfer function
characteristics entirely, and for rational systems (648) may be
used to determine the pole/zero locations of the underlying system.
PILLAI
27
The Power Spectral Density of the Output Signal
SX(f)
H(f)
SY(f) =H(f)2Sx(f)
Frequency Domain
The above relationship is very useful in system design and analysis.
28
Example: A WSS white noise process W(t) is passed
through a low pass filter (LPF) with bandwidth B/2. Find the
autocorrelation function of the output process.
Solution: Let X(t) represent the output of the LPF. Then from (648)
#q,  ω ≤ B / 2
2
S XX (ω ) = q  H (ω )  = $
.
(649)
%0,  ω > B / 2
The inverse transform gives the output autocorrelation function
B/2
B/2
RXX (τ ) = ∫ − B / 2 S XX (ω )e jωτ d ω = q ∫− B / 2 e jωτ d ω
sin( Bτ / 2)
= qB
= qB sinc( Bτ / 2)
( Bτ / 2)
R (τ )
(650)
XX
 H (ω )2
qB
1
−B / 2
(a) LPF
B/2 ω
τ
(b) Output autocorrelation
PILLAI
29
Example: Let
Y (t ) =
1
t +T
∫
2T t −T
X (τ )dτ
(651)
represent a smoothing operation using a moving window on the input
process X(t). Find the spectrum of the output Y(t) in term of that of X(t).
Solution: Consider a LTI system with impulse
response h(t) as in the figure to the right ,
then using h(t), (651) reduces to
+∞
Y (t ) = ∫ −∞ h(t − τ ) X (τ )dτ = h(t ) ∗ X (t )
so that
where
SYY (ω ) = S XX (ω )  H (ω ) 2 .
+T
H (ω ) = ∫ −T
1
2T
e− jωt dt = sinc(ωT )
h (t )
1 / 2T
−T
T
t
(652)
(653)
(654)
PILLAI
30
so that
SYY (ω ) = S XX (ω ) sinc 2 (ω T ).
(655)
sinc2 (ω T )
S XX ( ω )
ω
π
SYY (ω )
ω
ω
T
Notice that the effect of the smoothing operation in (651) is to
suppress the high frequency components in the input (beyond π / T ),
and the equivalent linear system acts as a lowpass filter (continuoustime moving average) with bandwidth 2π /T in this case.
By the way, you could also determine the power spectral density by
finding the correlation function of Y(t) directly.
PILLAI
31
Discrete – Time Processes
For discretetime w.s.s stochastic processes X(nT) with
autocorrelation sequence {rk }+∞
(proceeding as above) or formally
−∞ ,
defining a continuous time process X (t ) = ∑ n X (nT )δ (t − nT ), we get
the corresponding autocorrelation function to be
RXX (τ ) =
+∞
∑ rkδ (τ − kT ).
k =−∞
Its Fourier transform is given by
S XX (ω ) =
+∞
∑
k =−∞
rk e − jωT ≥ 0,
(656)
and it defines the power spectrum of the discretetime process X(nT).
From (656),
S XX (ω ) = S XX (ω + 2π / T )
so that S XX (ω ) is a periodic function with period
2π
2B =
.
T
(657)
(658)
PILLAI
32
This gives the inverse relation
1 B
jkωT
rk =
S
(
ω
)
e
dω
(659)
∫
− B XX
2B
and
1 B
2
(660)
r0 = E{ X (nT )  } =
S (ω )d ω
∫
− B XX
2B
represents the total power of the discretetime process X(nT). The
inputoutput relations for discretetime system h(nT) translates into
and
where
S XY (ω ) = S XX (ω ) H * (e jω )
(661)
SYY (ω ) = S XX (ω )  H (e jω ) 2
(662)
jω
H (e ) =
+∞
∑
n =−∞
h( nT ) e − jω nT
represents the discretetime system transfer function.
(663)
PILLAI
33
Example: Amplitude Modulated (AM) Signal Noise Analysis
Consider the noisy AM signal
X (t ) = m(t ) cos(ω0t + θ ) + n(t ),
(664)
Question: What is the Power Spectral Density of the AM signal?
Solution: In this case from (664), if we assume θ ~ U (0, 2π ),
then it can be shown that
1
RXX (τ ) = Rmm (τ ) cos ω 0τ + Rnn (τ )
2
so,
S XX (ω ) =
S XX (ω − ω 0 ) + S XX (ω + ω 0 )
+ S nn (ω ).
2
(a)
(666)
S XX (ω )
Smm (ω )
0
(665)
ω
−ω 0
(b)
S mm (ω − ω 0 )
ω0
ω
34
Summary: Signal Transmission with Linear Systems
Input
Output
Linear system
•
•
Deterministic signals:
Random signals:
(666)
35