ELE 324

Telecommunication Theory I

Probability Theory and Random Processes

Cenk Toker
Hacettepe University
Department of Electrical and Electronics Engineering

C. Toker ELE 324 - TELECOMMUNICATION THEORY I 1

 Random Variables
 A function whose domain is a sample space and whose range is some set of real numbers is called a
random variable if there is a probabilistic relation between the domain and range spaces.
 Consider the random variable X and and probability of the event :

where FX(x) is the cumulative distribution function (CDF) of X.

 The probability density function (pdf) of the random variable X is

 Also, note that

C. Toker ELE 324 - TELECOMMUNICATION THEORY I 2

 Multiple Random Variables
 For two random variables, X and Y, the joint CDF is FX,Y(x, y) and the joint probability function is fX,Y(x, y).
 The conditional probability density function of Y given that (X = x) is

 If X and Y are statistically independent

C. Toker ELE 324 - TELECOMMUNICATION THEORY I 3

 Statistical Averages
 The expected value or mean of X is

where E[.] is the statistical expectation operator.

C. Toker ELE 324 - TELECOMMUNICATION THEORY I 4

 Function of a Random Variable
 Consider a random variable, Y, which is a function of another random variable, X,

 If we know the PDF of Y, then the expected value of Y is

 or, equivalently

C. Toker ELE 324 - TELECOMMUNICATION THEORY I 5

 Moments
 The nth moment of the probability distribution of X is

where n = 1 gives the mean and n = 2 gives the mean-square value

 The nth central moment is

 For n = 2, we obtain the variance of X

where σX is called the standard deviation of X.

 Note that

C. Toker ELE 324 - TELECOMMUNICATION THEORY I 6

 Joint Moments
 The correlation of X and Y is

and the covariance of X and Y is

 X and Y are uncorrelated if and only if

and X and Y are orthogonal if and only if

C. Toker ELE 324 - TELECOMMUNICATION THEORY I 7

 Random Processes
 If the sample space or ensemble is a function of time, then it is called a random or stochastic process.
 For a fixed time tk, the set of numbers

forms a random variable.

 Thus, a random process is an ensemble of time functions

together with a probability rule.
 Differences between a random variable and a random
process are
◦ For a random variable, the outcome of a random experiment
is mapped into a number,
◦ For a random process, the outcome of a random experiment
is mapped into a waveform that is a function of time.

C. Toker ELE 324 - TELECOMMUNICATION THEORY I 8

 Stationarity
 If the statistical characteristics of a process is time invariant, then it is said to be stationary.
 Consider a random process X(t) and random variables X(t1), X(t2), …, X(tk). The joint CDF of the r.v.s is

 Suppose we obtain a new set of random variables by shifting the observation times, i.e. X(t1 + τ), X(t2 + τ),
…, X(tk + τ) with CDF

 X(t) is said to be stationary in the strict sense if

for all τ and t1, t2,…, tk.

C. Toker ELE 324 - TELECOMMUNICATION THEORY I 9

 Mean, Correlation and Covariance Functions
 The mean of the random process X(t) is

 If X(t) is a stationary process, then

 The autocorrelation of X(t) is

 If X(t) is stationary, then the autocorrelation depends only on the difference τ = t2 – t1

The autocovariance function is

C. Toker ELE 324 - TELECOMMUNICATION THEORY I 10

 Mean, Correlation and Covariance Functions
 If X(t) has constant mean, μX, and autocorrelation function, RX(t1, t2) = RX(t2 - t1), then X(t) is said to be a
wide-sense stationary process.

random processes

wide-sense stationary

stationary

C. Toker ELE 324 - TELECOMMUNICATION THEORY I 11

 Properties of the Autocorrelation Function
 The autocorrelation function (ACF) is

 Properties of the ACF:

◦ 1.
◦ 2. ACF is an even function, i.e.
◦ 3.

 RX(τ) gives an idea about interdependence of two random variables obtained by observing X(t) at times τ
seconds apart.

C. Toker ELE 324 - TELECOMMUNICATION THEORY I 12

 ACF
 Example
Consider

where A and fc are amplitude and frequency, and θ is the random phase angle that is uniformly distributed

The ACF of X(t) is

C. Toker ELE 324 - TELECOMMUNICATION THEORY I 13

 Cross-Correlation Functions
 Consider two random variables, X(t) and Y(t), with ACFs RX(t, u) and RY(t, u).
 The cross-correlation functions (CCFs) are

where t and u are two instants of time. Then, for w.s.s. processes, the correlation matrix is

 Note that CCF is not generally an even function, it does not have a maximum at the origin, i.e. at τ = 0, and

C. Toker ELE 324 - TELECOMMUNICATION THEORY I 14

 CCF
 Example
Consider a pair of quadrature modulated processes X1(t) and X2(t) that are related to a w.s.s. process X(t) as

where fc is the carrier frequency and θ is a uniformly distributed r.v. over [0, 2π). Moreover, θ is independent
of X(t).
Then, the CCF R12(τ) is

since θ and X(t) are independent

Note that, at τ = 0, R12(τ) = 0, hence X1(t) and X2(t) are orthogonal to each other.

C. Toker ELE 324 - TELECOMMUNICATION THEORY I 15

 Ergodicity
 The expectation of X(t), E[X(t)], is an average across the process
◦ (ensemble average)

 The time average of X(t), 𝑋ത 𝑡 , is an average along the process

◦ (sample/time average)

 The d.c. value of x(t) is defined by the time average

whose mean value (expected value) is

We say that, X(t) is ergodic in the mean, if

Remember
◦ i. E[x(t)]=μX
◦ İi.

C. Toker ELE 324 - TELECOMMUNICATION THEORY I 16

 Ergodicity
 Similarly, time averaged autocorrelation function is

 X(t) is ergodic in the autocorrelation function is

 For X(t) to be ergodic, it has to be w.s.s.

C. Toker ELE 324 - TELECOMMUNICATION THEORY I 17

 Transmission of a Random Process Through a Linear Filter
 Consider that a random process, X(t), is input to a LTI filter with impulse response, h(t), producing the
random process, Y(t), at the output

 Assume that, X(t) is a w.s.s. process, then

 The mean of Y(t) is

which is a constant for all t. 4th line follows from the fact that X(t) is w.s.s.

C. Toker ELE 324 - TELECOMMUNICATION THEORY I 18

 Transmission of a Random Process Through a Linear Filter
 The ACF of Y(t)

but, since X(t) is w.s.s., substituting τ = t - u,

meaning that Y(t) is also w.s.s.

 The mean-square value of Y(t) is

C. Toker ELE 324 - TELECOMMUNICATION THEORY I 19

 Power Spectral Density
 The frequency-domain equivalent of the mean-square value E[Y2(t)] is the following.
 Recall that

where H(f) is the transfer function of the filter. Substituting this into (*)

 Now, let τ = τ2 – τ1, then

 But, since

We have

C. Toker ELE 324 - TELECOMMUNICATION THEORY I 20

 Power Spectral Density
 The Power Spectral Density (or Power Spectrum) of X(t) is defined as

and, then

 To investigate the PSD, assume that the filter is a narrowband filter with
|H(f)|
Δf
1.0
f
 Then, if Δf is sufficiently small, we can have -fc fc

C. Toker ELE 324 - TELECOMMUNICATION THEORY I 21

Properties of the PSD
 Einstein-Wiener-Khintchine relations are

 Properties
◦ 1.

◦ 2.

◦ 3.

◦ 4. For a real valued random process

◦ 5. With (resembling a pdf.)

We may define the root-mean-squared (rms) bandwidth as

C. Toker ELE 324 - TELECOMMUNICATION THEORY I 22

 Properties of PSD
 Example (Sinusoidal wave with random phase)
 We has shown that the ACF of is

 Then, the PSD of X(t) is

C. Toker ELE 324 - TELECOMMUNICATION THEORY I 23

 Properties of PSD
 Example (Mixing of a random process with a sinusoidal process.)
 Consider a w.s.s. random process X(t) and a sinusoidal process cos(2πfct+θ), θ is a random variable
uniformly distributed in [0, 2π). X(t) and θ are independent.
 Then, the ACF of

is

 Then, the PSD of Y(t) is

C. Toker ELE 324 - TELECOMMUNICATION THEORY I 24

 Relation between the Input and Output Random Processes
 Consider an LTI system with transfer function, H(f).
 Let a random process, X(t), with PSD SX(f), be applied to the input of the system, and we observe the
random process, Y(t), with PSD, SY(f), at the output of the system.
 Then, we have

 Let , then

 Hence,

C. Toker ELE 324 - TELECOMMUNICATION THEORY I 25

 Cross-Spectral Densities
 Consider two w.s.s. random processes with CCFs, RXY(τ) and RYX(τ).
 The Cross-Spectral Densities are defined as

and
 The CCFs are obtained as

 SXY(f) and SYX(f) may be complex functions.

 Since

C. Toker ELE 324 - TELECOMMUNICATION THEORY I 26

 Cross-Spectral Densities
 Example
 Consider zero mean w.s.s. random processes X(t) and Y(t), and Z(t) = X(t) + Y(t).
 Then,

with τ = t – u

and, the PSD is

and, if X(t) and Y(t) are uncorrelated

C. Toker ELE 324 - TELECOMMUNICATION THEORY I 27

 Cross-Spectral Densities
 Example
 Consider two separate systems (filters) where the input random processes X(t) and Y(t) are w.s.s.
X(t) h1(t) V(t)

Y(t) h2(t) Z(t)

 Then,

with τ = t – u

Hence,

C. Toker ELE 324 - TELECOMMUNICATION THEORY I 28

 Gaussian Process
 The pdf of a Gaussian random variable Y has the form

where μY is the mean and σY2 is the variance of Y.

 If μY = 0, and σY2 = 1,


C. Toker ELE 324 - TELECOMMUNICATION THEORY I 29

 Central Limit Theorem
 Let Xi, i = 1, 2, …, N be a set of ranom variables and
◦ 1. Xi are statistically independent,
◦ 2. Xi have the same pdf with mean μY and variance σY2.

 Hence, {Xi} are independently and identically distributed (i.i.d.) random variables.
 Now, let
so that μY = 0 and σY2 = 1.
 Define

The Central Limit Theorem states that, the pdf of VN approaches a normalised Gaussian distribution N(0, 1) as N → ∞.

C. Toker ELE 324 - TELECOMMUNICATION THEORY I 30

 Properties of Gaussian Processes
 1. If a Gaussian process X(t) is input to a stable LTI filter, then the output Y(t) is also Gaussian.
 2. Consider samples of X(t); X(t1), X(t2), …, X(tn). If X(t) is Gaussian, then this set of samples are jointly
Gaussian with

 3. If a Gaussian process is w.s.s., then it is also stationary in the strict sense.

 4. If samples, X(t1), X(t2), …, X(tn), are uncorrelated, then these random variables are statistically
independent.

C. Toker ELE 324 - TELECOMMUNICATION THEORY I 31