14022016

CL 202: Introduction to Data Analysis

Special Random Variables:
Bivariate and Multivariate
Gaussian (Normal) Distributions

Sachin C. Patawardhan and Mani Bhushan

Department of Chemical Engineering
I.I.T. Bombay

14-Feb-16

MV Gaussian

Vector of Random Variables

Automation Lab
IIT Bombay

Consider a 2 1 vector RVs

x
X 1
x 2
together with an associated probability distribution function
FX ( x1 , x2 ) Px1 x1 , x 2 x2

x21 x1

( x1 , x2 )dx1dx2

f X ( x1 , x2 ) : Bivariate probability density function

x
f
(
x
,
x
)
dx
dx
1 E x1 1 X 1 2 1 2 x1 f x1 ( x1 ).dx1

E X

2 E x 2 x2 f X ( x1 , x2 )dx1dx2 x2 f x 2 ( x2 ).dx2

14-Feb-16

MV Gaussian

14022016

Automation Lab
IIT Bombay

Covariance

CovX E ( X )( X )T

E x1 1
E x1 1 x 2 2

2
E x 2 2
E x1 1 x 2 2

E x1 1 x 2 2

1 x2 2 f X ( x1 , x2 )dx1dx2

Cov (x1 , x 2 )

i2 E x i i 2 Varx i for i 1,2

Note: Covariance is always a symmetric matrix
14-Feb-16

MV Gaussian

Automation Lab
IIT Bombay

Covariance
Thus, we can write
2
1 2
Cov ( X ) 1

22
1 2
Cov ( x1 , x 2 )

1 2

Note that

detCov( X ) 12 22 (1 2 ) 0
Since the correlation coefficient satisfies
1 1
the covariance is always
a Positive Semidefinite Matrix
14-Feb-16

MV Gaussian

14022016

Automation Lab
IIT Bombay

Bivariate Gaussian PDF

f X ( x1, x2 )

fX ( X )

1
2

1
det( )

T
exp X 1 X
2

where

x2

x1

X x

x1

: symmetric and

Property 1 :

E X and Cov[ X ]

ve definite matrix

Characterized completely by mean () and covariance ( )

14-Feb-16

MV Gaussian

Automation Lab
IIT Bombay

Bivariate Gaussian PDF

For a given vector X , the term

X T 1 X

2
1 .2

represents the square of the weighted

disctance of, X , from the mean, .

x
X X 1 1
x2 2
T

12

1 2

1 2 x1 1

22 x2 2

x 2 x 2
x1 1 x2 2
1
1
1
2
2

(1 2 ) 1 2
1 2

1
1

(2 ) det( ) 1 2 1 2
The normalization term (2 ) ensures
14-Feb-16

MV Gaussian

( x ) dx 1
6

14022016

Bivariate Gaussian PDF

Automation Lab
IIT Bombay

When the two random

variables are uncorrelated
and the distribution is
symmetric in all directions.

When the random variables

are strongly positively
correlated, the distribution
is narrow and elongated
along the diagonal.
Ref.: B. A. Ogunnaike, Random Phenomena,
CRC Press, 2010.
14-Feb-16

MV Gaussian

Bivariate Normal RV

Automation Lab
IIT Bombay

The Components x1 and x 2 can be dissimilar

physical quantites (e.g. pressure and temperature)
with drastically different ranges.
Note that the bivariate PDF is defined using
the normalized (dimensionless) quantities
x 1
x 2
and z2 2

z1 1
1
2

X T 1 X

1
z1 2 z2 2 2 z1z2
2
(1 )

Normalization brings the dissimilar quantities on equal footing.

14-Feb-16

MV Gaussian

14022016

Properties Bivariate Normal RV

Automation Lab
IIT Bombay

Property 2 : The marginal distributions of the components

of the Gaussian Random Vector are univariate Gaussian

f x1 ( x1 )

( x1 , x2 )dx2 N 1 , 12

f x 2 ( x2 )

( x1 , x2 )dx1 N 2 , 22

N i , i2

14-Feb-16

1 ( xi i )2
1
where i 1,2
exp
i2
2 i
2
MV Gaussian

Properties Bivariate Normal RV

Automation Lab
IIT Bombay

Property 3 :
If X ~ , ) is a Gaussian random vector
and A is a (2 2) known invertible matrix
and b is a (2 1) vector, then
Z AX b
is also a Gaussian distributed with
E Z A b

CovZ A A T

Linear transformations of Gaussian random variables

are also Gaussian random variables
14-Feb-16

MV Gaussian

10

14022016

Automation Lab
IIT Bombay

Multivariate Gaussian Distributions

Property 4 :

If x1 and x 2 uncorrelated, i.e. Covx1x 2 0

then x 2 and x 2 are independent ,
i.e. f x1,x 2 ( x1 , x2 ) f x1 ( x1 ) f x 2 ( x2 )
This follows from

X T 1 X x1 1
1

x 2
when Covx1x 2 0
2
2

Note :
If two arbitrary (non - Gaussian) RVs x1 and x 2
are uncorrelated, then, in general,
it does not imply that they are independent.
14-Feb-16

MV Gaussian

11

Multivariate Gaussian Distribution

Automation Lab
IIT Bombay

Consider Gaussian random variable X R n

Let R n represent mean of X and
represent ve definite covarince matrix
f X ( X ) , )

1
( 2 )

n/2

T
exp X 1 X
det( )
2

Characterized completely by mean () and covariance ( )

Normalizing factor
1
ensures
(2 )n / 2

Property 1 :

E X and Cov[ X ]

(x )dx 1

14-Feb-16

MV Gaussian

12

14022016

Automation Lab
IIT Bombay

Multivariate Gaussian Distributions

Property 2: Any portion of a Gaussian random

vector is itself Gaussian

fZ (Z )

n/2

1
T
exp Z 1 Z
det( )

XY

X
where Z , X and X

Y
XY Y
Y
1
1

1
T
fX ( X )
exp X X X1 X X
nX / 2
2
det( X )

2
f Y (Y )

1
1

T
exp Y Y Y1 Y Y
2
det( Y )

nY / 2

14-Feb-16

MV Gaussian

Properties Bivariate Normal RV

13

Automation Lab
IIT Bombay

Property 3 : The marginal distributions

of the componentsof the Gaussian Random Vector
are univariate Gaussian

f xi ( xi ) ... f X ( x1 , x2 ,..., xn )dx1..dxi 1dxi 1..dxn

N i , i2

N i , i2

14-Feb-16

for i 1,2 ,...,n

1 ( xi i ) 2
1
where i 1,2,..., n
exp
i2
2 i
2
MV Gaussian

14

14022016

Automation Lab
IIT Bombay

Multivariate Gaussian Distributions

Property 4 : if vectors X and Y are Gaussian

Random Vectors and uncorrelated,
then X and Y are independent
i.e. f X, Y ( X , Y ) f X ( X ) f Y (Y )
Note : Consider two random variable vectors X and Y.
If X and Y are uncorrelated,
i.e. CovX, Y [0]

then, in general,
it does not imply that they are independent.
14-Feb-16

MV Gaussian

Multivariate Gaussian Distributions

15

Automation Lab
IIT Bombay

Property 5: Linear transformations of Gaussian

random variables are also Gaussian random variables

If X ~ , ) is a random vector and A is a

(n n) known invertible matrix
and b is a (r 1) vector, then
Z AX b
is also a Gaussian distributed with
Z ~ A b, A ( A T )
14-Feb-16

MV Gaussian

16

14022016

Automation Lab
IIT Bombay

Summary

Gaussian Random Vectors are completely

characterized by the mean and the covariance of
the distribution.

The special properties of Gaussian RVs prove to be

extremely useful in mathematical manipulations
involving MV Gaussian densities.

Gaussian Random Vectors are very frequently

encountered in many practical applications (e.g.
measurement noise).

14-Feb-16

MV Gaussian

17

Automation Lab
IIT Bombay

Image of old German currency, Deutsche Mark, with picture of

Johann Carl Friedrich Gauss (known as prince of mathematics)
and
The univariate Gaussian distribution

14-Feb-16

MV Gaussian

18