EECS 225A Statistical Signal Processing Lecture 4 - 01/30/2020

Lecture 4: Spectral Representation of Random Processes

Lecturer: Jiantao Jiao Scribe: Xinyu Zhang

1 Energy Spectral Density (ESD)

Given a deterministic sequence x(n) (we sometimes also denote as xn ), its DTFT is defined as
X
X(ω) = x(n)e−jωn . (1)
n∈Z

One main property of DTFT is that if we compute the DTFT of the deterministic autocorrelation
sequence a(n) of x(n) defined as
X
a(n) = x(k)x∗ (k − n), (2)
k∈Z

we have

A(ω) = |X(ω)|2 ≥ 0. (3)

To see this, we can express the deterministic autocorrelation as a convolution of x(n) and x∗ (−n), and the
DTFT of x∗ (−n) is X ∗ (ω). The quantity A(ω) is called the Energy Spectral Density (ESD). This name is
justified through the IDTFT relation
Z π
1 X
A(ω)dω = a(0) = |x(n)|2 . (4)
2π −π
n∈Z

2 Going from deterministic to random

What if the sequence x(n) is a wide sense stationary (WSS) process? In this case we usually use capital
letter X(n) to indicate that it is a stochastic process. Can we still use the ESD to characterize the property
of X(n)?
Unfortunately, the answer is no. It is because the total energy of X(n) is in fact infinity except in trivial
cases:
X
E[ |x(n)|2 ] = ∞ (5)
n∈Z

unless E[|x(0)|2 ] = 0. Here we used that fact that X is WSS. Hence, for WSS process we need to go from
energy to power. The following computation by Wiener and Khinchin is insightful.
Suppose we truncate sequence xn from −T to T , where T is a positive integer. Then, for this finite-length
signal the total energy is finite on average. We now compute the ESD of the truncate signal. The DTFT of
the truncated signal is
T
X
XT (ω) , xn e−jωn . (6)
n=−T

1
The ESD AT (ω) of the truncated signal is

AT (ω) = |XT (ω)|2 (7)

T
X T
X
=( xn e−jωn )( x∗m ejωm ) (8)
n=−T m=−T
X
= xn x∗m e−jω(n−m) . (9)
−T ≤n,m≤T

Taking the expectation on both sides and dividing both sides by 2T + 1 (going from energy to power), we
have
1 1 X
E[AT (ω)] = RX (n − m)e−jω(n−m) (10)
2T + 1 2T + 1
−T ≤n,m≤T
2T  
X
−jωk |k|
= RX (k)e 1− (11)
2T + 1
k=−2T
P
If k∈Z |RX (k)| < ∞, then it follows from the dominated convergence theorem that
1 X
lim E[AT (ω)] = RX (k)e−jωk , (12)
T →∞ 2T + 1
k∈Z

which is the DTFT of the auto-correlation function of the stochastic process X.

We have just proved the Wiener–Khinchin theorem, which justifies the name Power Spectral Density of
the expression
X
SX (ω) = RX (k)e−jωk . (13)
k∈Z

3 Spectral representation of stochastic processes

We can interpret the IDTFT formula for deterministic signal xn
Z π
1
xn = X(ω)ejωn dω (14)
2π −π

as expressing the sequence xn as the linear combination of complex exponentials ejωn with coefficients
1
2π X(ω)dω. Can we do something similar for WSS process X(n)? The Cramer–Khinchin decomposition [1,
Theorem 4.1.3] precisely gives us this tool.
Theorem 1 (Cramer–Khinchin decomposition). Let X(n) be a complex-valued WSS stochastic process with
power spectral density SX (ω). Then there exists a unique right-continuous stochastic process F (ω), ω ∈
(−π, π] with square-integrable orthogonal increments such that
Z π
X(n) = ejωn dF (ω), (15)
−π

where for any interval [ω1 , ω2 ] ⊂ [−π, π], [ω3 , ω4 ] ⊂ [−π, π],

E[(F (ω2 − F (ω1 ))(F (ω4 ) − F (ω3 ))∗ )] = f ((ω1 , ω2 ] ∩ (ω3 , ω4 ]), (16)

where f is called the structural measure of the stochastic process F (ω), whose Radon–Nikodym derivative
with respect to the Lebesgue measure is given by SX2π(ω) .

2
 Taking ω1 = −π, we have
Z ω
dt
E|F (ω) − F (−π)|2 = SX (t) .
−π 2π

If we assume that X is zero-mean, then we have a stronger result: we can show that the process F (ω)
is centered. In other words, E[F (ω2 ) − F (ω1 )] = 0 for any ω1 , ω2 ∈ (−π, π]. One could show the version
for non-zero mean by reducing it to the zero-mean case: indeed, if X is not zero-mean with non-zero mean
µX , we can just apply the decomposition to X − µX , and then modify the representation F to add a term
µX δ(ω)dω to dF (ω).
We refer the readers to [1, Theorem 4.1.3] or [2, Theorem 4.8.2] for the proof of this statement. The
vector version of this result can be found in [2, Theorem 11.8.2]. We focus on interpreting this theorem by
computing the autocorrelation function of X(n) using this representation.
The autocorrelation function RX (k) is given by
∗
RX (k) = E[Xn Xn−k ] (17)
Z π Z π 
jω1 n −jω2 (n−k) ∗
=E e dF (ω1 ) e dF (ω2 ) (18)
−π −π
Z π
= ejωk df (ω), (19)
−π

where the last equality is the Ito isometry formula. Here f is the quadratic variation process for F (ω), and
as the Cramer–Khinchin decomposition theorem guarantees, df (ω) = SX (ω)dω2π , which is consistent with the
R π jωk dω
IDTFT formula RX (k) = −π e SX (ω) 2π .

3.1 How to interpret a stochastic integral?

The stochastic integral does not follow all the rules of the usual Lebesgue or Riemann integrals. However,
you may still understand it using the intuitions of classical integrals. We say a function f (t), 0 ≤ t ≤ T
is simple if there exist a finite subdivision 0 = t0 < t1 < . . . < tn = T of the interval [0, T ], constants
α0 , α1 , . . . , αn−1 such that
n−1
X
f (t) = αi χ(ti ,ti+1 ] (t),
i=0

where χ(ti ,ti+1 ] is the indicator function of the half-closed interval (ti , ti+1 ].
The classical Stieltjes integral shows that if a = a(t) is a nondecreasing right continuous function, then
Z ∞
χ(u,v] (t)da(t) = a(v) − a(u).
0

Then, for simple deterministic function f (t), the stochastic integral can be defined as
Z T n−1
X
f (u)dF (u) = αi (F (ti+1 ) − F (ti )).
0 i=0

3.2 Connections with the harmonic process

We first obtain the spectral representation of the harmonic process. The harmonic process X(n) is defined
PN
as X(n) = Am cos(ωm n + Φm ), where ωm , m ∈ [N ] are deterministic frequencies, Φm ∼ U [−π, π] are
m=1

3
random phases such that they are mutually independent across m, and Am , m ∈ [N ] are orthogonal zero
mean real-valued random variables independent of all the random phases Φm . Concretely, E[Am ] = 0 and
2
E[Am Am0 ] = σm δmm0 (δmm0 = 1 if m = m0 , else 0). We showed in last lecture that the PSD of X is given by
N
X 1 2
SX (ω) = σ π[δ(ω − ωm ) + δ(ω − ωm )] (20)
m=1
2 m

We first write Xn as linear combinations of ejωn for different ω. Since

ej(ωm n+φ) + e−j(ωm n+φ)
Am cos(ωm n + φ) = (21)
2
ejφ jωm n e−jφ −jωm n
= Am e + Am e , (22)
2 2
we have
N N
X ejφ jωm n X e−jφ −jωm n
X(n) = Am e + Am e (23)
m=1
2 m=1
2
N
X N
X
= Bm e jωm n
+ Cm e−jωm n , (24)
m=1 m=1
jφ −jφ
where Bm = Am e2 , Cm = Am e 2 . We have thus obtained the spectral representation of the harmonic
process X(n) in (24).
What is the function F (ω) for the harmonic process? It is in fact a piecewise right continuous function.
Concretely,


 0 ω ∈ [−π, −ωN )




 CN ω ∈ [−ωN , −ωN −1 )
∈ [−ωN −1 , −ωN −2 )




 C N + C N −1 ω
. . . ...


 PN
F (ω) = C m ω ∈ [−ω1 , ω1 ) (25)
 Pm=1
N
m=1 Cm + B1 ω ∈ [ω1 , ω2 )








 ... ...
PN PN −1
∈ [ωN −1 , ωN )




 C
m=1 m + m=1 mB ω
PN PN
m=1 Cm + m=1 Bm ω ∈ [ωN , π)
As a sanity check, we have
• Orthogonal increment:
E[Bm Bn∗ ] = 0 (26)
E[Cm Cn∗ ] =0 (27)
(28)
for all m 6= n, m ∈ [N ], n ∈ [N ] and
E[Bm Cn∗ ] = 0 (29)

for all m ∈ [N ], n ∈ [N ], where [N ] , {1, 2, . . . , N }.

• Power distribution:
Z
1 1 2
E[|Bm |2 ] = E[|Cm |2 ] = E[|Am |2 ] = σm = lim SX (ω)d(ω/2π). (30)
4 4 ∆ω→0+ ω∈[ωm −∆ω,ωm +∆ω]

4
4 Linear operations on WSS stochastic processes
Suppose two stochastic processes X(n), Y (n) are jointly WSS. We define the z-cross-spectrum as
X
SY X (z) , RY X (k)z −k , (31)
k∈Z

∗
where RY X (k) , E[Yn Xn−k ]. Then we have

SXY (z) = SY∗ X (z −∗ ) (32)

SXY (ω) = SY∗ X (ω). (33)

Here z −∗ = (1/z)∗ .
Theorem 2. [3, Lemma 6.3.1] Let Y (n) be a stochastic process obtained by passing a WSS process X(n)
through a stable LTI (Linear Time Invariant) system with impulse response h(n) (transfer function H(z)).
In other words,

Y (n) = (h ∗ X)(n) (34)

+∞
X
= h(m)X(n − m). (35)
m=−∞

Then,

SY (z) = H(z)SX (z)H ∗ (z −∗ ) (36)

SY X (z) = H(z)SX (z) (37)
∗ −∗ ∗
SXY (z) = H (z )SX (z −∗ ) (38)
2
SY (ω) = |H(ω)| SX (ω) (39)
SY X (ω) = H(ω)SX (ω) (40)
SXY (ω) = SX (ω)H ∗ (ω). (41)

Finally, if Z(n) is jointly WSS with (Y (n), X(n)) as defined, we have

SZY (z) = SZX (z)H ∗ (z −∗ ). (42)

References
[1] P. Brémaud, “Fourier analysis of stochastic processes,” in Fourier Analysis and Stochastic Processes.
Springer, 2014, pp. 119–179.
[2] P. J. Brockwell and R. A. Davis, Time series: theory and methods. Springer Science & Business Media,
1991.

[3] T. Kailath, A. H. Sayed, and B. Hassibi, Linear estimation. Prentice Hall, 2000.