Stochastic Processes

Topic 2
Jointly Gaussian Random Variables
nctuee 08f
Summary
This lecture reviews several important properties of Gaussian random vari-
ables. Specically, I will discuss:
Gaussian Random Variable
Moment Generating Function
Central Limit Theorem
Jointly Gaussian Random Variables (Multivariate Normal)
Joint Gaussian Density Function
Notation
We will use the following notation rules, unless otherwise noted, to represent
symbols during this course.
Boldface upper case letter to represent MATRIX
Boldface lower case letter to represent vector
Superscript ()
T
and ()
H
to denote transpose and hermitian (conjugate
transpose), respectively
Upper case italic letter to represent RANDOM VARIABLE
2-1
Motivation: Why a special attention to Gaussian RVs?
They are analytically tractable Preserved by linear systems
Central limit theorem Gaussian can approximate a large variety of
distributions in large samples
Useful as models of communication links
noise
channel fading eects
1 Gaussian Random Variables
Denition 1 The probability density function (pdf ) f
X
(x) for a Gaussian
random variable X with mean and variance
2
, denoted by X N(,
2
),
is given by
f
X
(x) =
1

2
2
e

(x)
2
2
2
(1)
for < x < .
Remarks:
Verify that X has mean and variance
2
If X N(,
2
), then the random variable Z =
X

has a N(0, 1)
distribution, also known as the standard normal
In general, the random variable Z
1
= aX +b for any real scalars a and
b is also Gaussian with mean a +b and variance a
2

2
A Gaussian random variable can be characterized by its rst 2 mo-
ments; i.e. E[X] and E[X
2
]
2-2
2 Moment Generating Function
Denition 2 The moment generating function (MGF) of a continuous ran-
dom variable X with pdf f
X
(x) is dened by
(t) E[e
tX
] =
_

e
tx
f
X
(x)dx,
where t is a complex variable. For a discrete random variable X with proba-
bility mass function (pmf ) p
X
(x) P
X
[X = x], the MGF is dened by
(t) E[e
tX
] =

i
e
tx
i
P
X
[X = x
i
].
Remarks:
(1) Its similar to the Laplace transform. Thus, in general, there is a one-
to-one correspondence between (t) and f
X
(x).
(2) Used for computing moments
The nth moment of X can be obtained by
E[X
n
] =
d
n
dt
n
((t))

t=0
.
(3) Solving problems involving sum of independent RVs
(4) Analytical tool to demonstrate Central Limit Theorem
Example: Let X N(
X
,
2
X
). Its MGF is given by
(t) = exp(
X
t +
2
X
t
2
/2). (2)
(1) We can compute E[X] =

(0) =
X
and E[X
2
] =

(0) =
2
X
+
2
X
.
(2) Sum of 2 indep. Gaussian
Suppose that Y N(
Y
,
2
Y
) is independent with X. Then, the MGF
for Z = X +Y is
E[e
tZ
] = E[e
tX
] E[e
tY
] = exp((
X
+
Y
)t + (
2
X
+
2
Y
)t
2
/2),
which bears the same form as (2). Therefore, from the 1-to-1 corre-
spondence between distribution of a random variable and its MGF, we
conclude that Z N(
X
+
Y
,
2
X
+
2
Y
).
2-3
3 Central Limit Theorem
Theorem 3 (CLT in Special Case) Let X
1
, X
2
, ..., X
n
be independent and
identically distributed (i.i.d.) random variables with E[X
i
] = 0 and Var[X
i
] =
1 for all i. Then,
Z
n

1

n
n

i=1
X
i
tends to the standard normal in the sense that its MGF satises
lim
n

Z
n
(t) = e
t
2
/2
,
which is the MGF of standard normal.
Remarks:
(1) For i.i.d. random variables X
1
X
n
with mean and variance
2
, the
CLT says that
1

n
n

i=1
_
X
i

_
=

n
_
X

_
tends to standard normal where X = (1/n)

n
i=1
X
i
.
(2) It happens a lot that well be asked to nd
P
_
a
n

i=1
X
i
b
_
with moderately large n for i.i.d. samples X
i
. We dont really need to
nd the pdf for the sum

i
X
i
. CLT tells us we can approximate the
normalized sum to a standard normal.
(3) For a detailed proof of CLT, see [1]
2-4
4 Jointly Gaussian Random Variables
Denition 4 (Jointly Gaussian) A collection of random variables X
1
, X
2
, , X
n
are jointly Gaussian if
n

i=1
a
i
X
i
is a Gaussian random variable for real a
i
for i = 1 n.
(1) In plain words, any linear combination of jointly Gaussian RVs is again
a Gaussian random variable
(2) Let X
1
, X
2
, , X
n
be i.i.d. Gaussian. Then, they are jointly Gaussian.
Check the MGF of

n
i=1
a
i
X
i
.
(3) Let x = [X
1
X
n
]
T
be a vector of n i.i.d. N(0, 1) random variables.
Then,
Ax +b
is a vector of jointly Gaussian random variables for all real determinis-
tic A and b
(4) JGlinear systemJG
(5) Jointly Gaussian implies Marginal Gaussian. But the converse is not
true
2-5
5 Basic Denitions
Denition 5 (Joint Density) Let x = [X
1
X
n
]
T
be a random vector
with probability distribution function F
x
(x). Then, by denition
F
x
(x) P [X
1
x
1
, , X
n
x
n
] P [x x] ,
where x = [x
1
, , x
n
]
T
is a realization of the random vector x. If the nth
partial derivative of F
x
(x) exists, the joint pdf is dened as
f
x
(x)

n
F
x
(x)
x
1
x
n
.
Denition 6 The joint moment generating function for N random variables
X
1
X
N
is dened by
(t
1
, t
2
, , t
N
) = E
_
exp
_
N

i=1
t
i
X
i
__
.
Remarks:
(1) Joint MGF uniquely determines the joint distribution of X
1
, , X
n
.
(2) X
1
, , X
n
are independent if and only if (t
1
, t
2
, , t
n
) =
X
1
(t
1
)
X
n
(t
n
).
2-6
Covariance Matrix
(a) The cross-covariance matrix K
xy
of two random vectors x and y con-
sisting of random variables X
1
, , X
n
and Y
1
, , Y
n
, respectively, is
dened by
K
xy
Cov (x, y)
E
_
(x E[x])(y E[y])
H

,
where E[x] = [E[X
1
], , E[X
n
]]
T
is the vector of expected values of
X
i
, and likewise for E[y].
(b) The covariance matrix K
xx
of a random vector x is dened by K
xx
=
E
_
(x E[x])(x E[x])
H

.
Remarks:
(1) For any matrices A and B, and vectors c and d with proper dimension,
one has Cov(Ax +c, By +d) = A Cov(x, y) B
T
(2) The diagonal terms of K
xx
, simply denoted by K
x
, are the variances

2
1
, ,
2
n
of X
1
, X
2
, , X
n
.
(3) K
x
is a Hermitian (or real symmetric) matrix. (K
x
= EE
H
)
2-7
Uncorrelated, orthogonal, and independent
Consider two real n 1 random vectors x and y. Then, we say
(a) x and y are uncorrelated if E[xy
T
] = E[x] E[y
T
].
(b) x and y are orthogonal if E[xy
T
] = 0.
(c) x and y are independent if f
xy
(x, y) = f
x
(x)f
y
(y).
Remarks:
(1) Independence implies uncorrelatedness, but the converse is NOT true.
(2) Covariance matrix of uncorrelated random variables X
1
, , X
n
is a
diagonal matrix. (O-diagonal terms are zero.)
Example:(Decorrelation of random vectors) (See 5.4-2 in textbook!)
2-8
6 Jointly Gaussian Density Function
We will determine in this section the joint density function for a jointly
Gaussian random vector. Lets see some useful facts rst.
(1) We denote x N(m, K) to indicate that x is a jointly Gaussian ran-
dom vector with mean vector m and covariance matrix K.
(2) Assume that x N(m, K), then Ax +b N(Am+b, AKA
T
). (We
have already seen that any linear transformation of JG is also JG.)
(3) The joint MGF of a jointly Gaussian random vector x N(m, K) is
given by

x
(t) = E[e
t
T
x
] = exp
_
t
T
m+
1
2
t
T
Kt
_
.
This can be veried by noting that Y t
T
x is a Gaussian random
variable from Denition 4. And, the mean and variance of Y are t
T
m
and t
T
Kt, respectively. Then,
x
(t) = E[e
Y
] =
Y
(t)

t=1
.
(4) Verify that jointly Gaussian random variables are independent if and
only if they are uncorrelated using the concept of joint MGF.
() Much easier to check E[XY ] = E[X]E[Y ] if independence.
() Suppose that x N(m, K) is a jointly Gaussian random vector
of uncorrelated Gaussian random variables X
1
X
n
, implying that K
is a diagonal matrix K = diag(
2
1

2
n
). Therefore, the joint MGF is

x
(t) = exp
_
t
T
m+
1
2
t
T
Kt
_
= exp
_
n

i=1
(t
i
m
i
+
1
2
t
2
i

2
i
)
_
=
n

i=1
exp
_
t
i
m
i
+
1
2
t
2
i

2
i
_
=
X
1
(t
1
)
X
n
(t
n
).

Important:
This is an important result, since we can determine whether jointly
Gaussian random variables are independent by simply checking its
correlation. With independence, we can easily calculate, e.g., the
joint pdf and the conditional expectation E[X|Y ] = E[X].
2-9
Example: Let X and Y be jointly Gaussian random variables with zero
mean, Var(X) =
2
X
and Var(Y ) =
2
Y
. We can nd a scalar such that
X Y and Y are independent Gaussian random variables by letting
E [(X Y )Y ] = E[X Y ]E[Y ] = 0.
From which, we have
=
E[XY ]
E[Y
2
]
=

Y
,
where
Cov(X,Y )

Y
is the correlation coecient between X and Y .
2-10
Joint Density of Two JG RVs
The joint pdf for two real jointly Gaussian random variables X and Y is
given by
f
XY
(x, y) =
1
2
X

Y
_
1
2

exp
_
1
2(1
2
)
_
_
x
X

X
_
2
2
(x
X
)(y
Y
)

Y
+
_
y
Y

Y
_
2
__
,
where
X
= E[X],
2
X
= Var[X],
Y
= E[Y ],
2
Y
= Var[Y ], and
=
E[(X
X
)(Y
Y
)]

Y
is the correlation coecient.
Proof
Lets assume
X
=
Y
= 0 for simplicity. Since X and Y are jointly Gaussian,
we know that
U = X Y and V = Y
are also jointly Gaussian random variables. From the example in the last
page, we know that U and V are independent if =

Y
. From Section 3.4
of the textbook, the joint pdf f
X,Y
(x, y) can be determined from f
U,V
(u, v)
by
f
X,Y
(x, y) =
1
|J|
f
U,V
(u, v),
where |J| = det(J) and the matrix J is given by
J =
_
X/U X/V
Y/U Y/V
_
=
_
1
X
/
Y
0 1
_
.
So, actually, we have
f
X,Y
(x, y) = f
U,V
(x y, y) = f
U
(x y)f
V
(y)
=
1
2
U

V
exp
_

(x y)
2
2
2
U
_
exp
_

y
2
2
2
V
_
,
where
2
U
= E[U
2
] = E[(X Y )
2
] = (1
2
)
2
X
and
V
=
Y
. Plugging
the results and performing some manipulations, we can show that f
X,Y
(x, y)
takes the form mentioned in the above.
2-11
Remarks:
(1) This joint pdf is commonly used to dene two jointly Gaussian random
variables. (See p. 201 in textbook.)
(2) If = 0, we have f
X,Y
(x, y) = f
X
(x)f
Y
(y), showing that uncorrelated-
ness implies independence for jointly Gaussian random variables
(3) Recall that the joint MGF for JG only depends on the mean vector
and covariance matrix. We can deduce that the joint pdf for JG is also
the case.
Let z [X Y ]
T
. The mean vector m
z
= [E[X] E[Y ]]
T
, and covariance
matrix K
z
=
_

2
X

X

Y

2
Y
_
determines the joint pdf in the form
f
z
(z) =
1
2 det(K
z
)
1/2
exp
_

1
2
(z m
z
)
T
K
1
z
(z m
z
)
_
.
(4) The contour and the surface of the joint pdf for two zero mean jointly
Gaussian X
1
and X
2
with variance 2 and correlation coecients = 0.5
are plotted respectively in Fig. 1.
2-12
Figure 1: The contour and the surface of the pdf for jointly Gaussian X
1
and X
2
with variance 2 and = 0.5.
2-13
Joint Density of n JG RVs
1. (Recall) Any jointly Gaussian random vector x can be represented by
a linear combination of the vector of i.i.d. standard normal random
variables z N(0, I
n
).
That is, if x N(m
x
, K
x
), we can write
x = K
x
1/2
z +m
x
,
where K
x
1/2
= E
1
2
E
H
with E being the matrix of orthonormal eigen-
vectors and the diagonal matrix of eigenvalues of K
x
.
2. Let z N(0, I
n
) and Ube a unitary matrix. Then, Uz has an identical
distribution as z, denoted by
z
d
=Uz.
(Justify)
a. Uz is jointly Gaussian.
b. Mean vector of Uz is a zero vector.
c. Cov(Uz,Uz) = UCov(z, z)U
T
= UU
T
= I
2-14
3. (General Expression) Let x = [X
1
, X
2
, , X
n
] be a real jointly Gaus-
sian random vector (Normal random vector) with mean vector m
x
and
covariance matrix K
x
. Then, the joint pdf f
x
(x) is given by
f
x
(x) =
1
(2)
n/2
det(K
x
)
1/2
exp
_

1
2
(x m
x
)
T
K
1
x
(x m
x
)
_
.
Remarks:
(1) Please note the dierence between random vector x and determin-
istic vector x.
(2) Assume the elements X
1
, X
2
, , X
n
of the random vector x are
uncorrelated, each with variances Var(X
i
) =
2
i
, then the joint
pdf is reduced to
f
x
(x) =
1
(2)
n/2

n
i=1

i
exp
_

1
2
n

i=1
_
x
i

i
_
2
_
.
2-15
Proof Recall that any jointly Gaussian random vector can be repre-
sented by a linear combination of a standard Gaussian random vector,
we can write
x = K
x
1
2
z +m
x
= E
1
2
E
T
z +m
x
where K
x
1
2
E
1
2
E
T
. Since E is a unitary matrix, E
T
z is also a
standard Gaussian random vector. It follows that x has an identical
distribution with E
1
2
z +m
x
.
Let y =
1
2
z. It is clear that y is also jointly Gaussian distributed with
N(0, ), and also a vector of independent Gaussian RVs. Then, the
joint pdf for y is given by
f
y
(y) =
n

i=1
1

2
1/2
i
exp
_

y
2
i
2
i
_
=
1
(2)
n/2
det()
1/2
exp
_

1
2
y
T

1
y
_
.
Once we have the joint pdf of y, we can use the concept of linear
transformation to determine the joint pdf of x from
x = Ey +m
x
.
That is,
f
x
(x) =
1
|J|
f
y
_
E
T
(x m
x
)
_
= f
y
_
E
T
(x m
x
)
_
since |J| = 1
=
1
(2)
n/2
det()
1/2
exp
_

1
2
(x m
x
)
T
E
1
E
T
(x m
x
)
_
=
1
(2)
n/2
det(K
x
)
1/2
exp
_

1
2
(x m
x
)
T
K
1
x
(x m
x
)
_
.

2-16
4. To conclude, the following 3 statements are equivalent:
Random variables X
1
, X
2
, , X
n
are jointly Gaussian.
The random variable Y =

n
i=1
a
i
X
i
is a Gaussian random vari-
able for any real a
i
.
The joint pdf for X
1
, X
2
, , X
n
is given by
f
x
(x) =
1
(2)
n/2
det(K
x
)
1/2
exp
_

1
2
(x m
x
)
T
K
1
x
(x m
x
)
_
.
References
[1] W. Feller, An Introduction to Probability Theory and Its Applications,
New York: John Wily, 2nd edition, 1971.
2-17