The normal

distribution

UFC/DC
ATAII (CK0146)
PR (TIP8412)
2017.2

The normal
distribution

The multivariate
normal The normal distribution
distribution
Useful distributions
An application
PCA

Francesco Corona

Department of Computer Science

Federal University of Ceará, Fortaleza
 The normal
distribution

UFC/DC
ATAII (CK0146)
PR (TIP8412)
2017.2

The normal
distribution

The multivariate
normal
distribution

An application
PCA
The normal distribution
Useful distributions
 The normal The normal distribution
distribution

UFC/DC
ATAII (CK0146)
PR (TIP8412)
2017.2

The normal
distribution

The multivariate
normal Motivation for the normal distribution is found in the central limit theorem
distribution

An application
PCA Normal distributions provide an important family of distributions
Applications and inference

We first introduce the standard normal distribution

• Then, the general normal distribution
 The normal The normal distribution
distribution

UFC/DC
ATAII (CK0146)
PR (TIP8412)
2017.2

The normal
distribution
Consider the integral
The multivariate
normal
distribution
0.4
An application
PCA
f (z )

Z +∞  − z 2
0.2 1
I = √ exp dz (1)
−∞ 2π 2

0
−5 0 5
z
This integral exists
 The normal The normal distribution (cont.)
distribution

UFC/DC
ATAII (CK0146) Consider the relevant part of the integrand
PR (TIP8412)
2017.2 It is a positive continuous function bounded by an integrable function
The normal
distribution 3
The multivariate
normal
(2π)1/2 f (z )
distribution
2
An application  − z 2
PCA exp
1 2

0
−5 0 5
z

That is,
 − z 2
0 < exp < exp (−|z | + 1), −∞ < z < ∞
2
and Z ∞
exp (−|z | + 1)dz = 2e
−∞
 The normal The normal distribution (cont.)
distribution

UFC/DC
ATAII (CK0146)
PR (TIP8412)
2017.2

The normal
distribution

The multivariate To evaluate the integral, we note that I > 0

normal
distribution

An application We have that I 2

PCA
Z ∞  z 2 Z ∞  w 2
1 1
I2 = √ exp − dx · √ exp − dw
−∞ 2π 2 ∞ 2π 2
Z ∞ Z ∞  z 2 + w 2
1
= exp − dz dw
2π −∞ −∞ 2

The integral can be computed by changing to polar coordinates

 The normal The normal distribution (cont.)
distribution

UFC/DC
ATAII (CK0146)
PR (TIP8412)
2017.2

The normal
distribution Set z = r cos θ and w = r sin θ
The multivariate
normal Then,
distribution
Z 2π Z
An application 1 ∞ 2
PCA I2 = e −r /2
r dr dθ
2π 0 0
| {z }
2 2
xe cx dx =1/(2c)e cx =1
R

Z 2π
1
= dθ
2π 0
=1

The integrand of Equation (1) is positive on R and integrates to 1 over R

 The normal The normal distribution (cont.)
distribution

UFC/DC
ATAII (CK0146)
PR (TIP8412)
2017.2
The integrand is a PDF of a continuous random variable with support R
The normal
distribution
Denote this random variable Z , Z has the PDF
The multivariate
1  − z 2
normal
distribution
f (z ) = p exp , for − ∞ < x < ∞ (2)
(2π) 2
An application
PCA

0.4

f (z )
0.2

0
−5 0 5
z
 The normal The normal distribution (cont.)
distribution

UFC/DC
ATAII (CK0146)
PR (TIP8412)
2017.2
For t ∈ R, the MGF of Z can be derived by a completion of a square
The normal
distribution Z ∞ 1  1 
The multivariate E [exp (tZ )] = exp (tz ) √ exp − z 2 dz
normal −∞ 2π 2
distribution
1 Z ∞ 1 h 1 i
2
An application = exp t exp − (z − t)2 dz (3)
PCA 2 −∞ 2π 2
1 Z ∞ 1 h 1 i
= exp t2 √ exp − w 2 dw
2 −∞ 2π 2

For the last integral, we made a 1-to-1 change of variable w = z − t

The integral in Equation (3) has value 1, by the identity

1  − z 2
f (z ) = p exp , for − ∞ < x < ∞,
(2π) 2
 The normal The normal distribution (cont.)
distribution

UFC/DC
ATAII (CK0146)
PR (TIP8412)
2017.2

The normal
Thus, the MGF of Z
distribution
2
The multivariate MZ (t) = e (1/2t ) , for − ∞ < x < ∞ (4)
normal
distribution

An application The first two derivative of MZ (t) are

PCA
2
MZ′ (t) = te (1/2t )

2 2
MZ′′ (t) = e (1/2t )
+ t 2 e (1/2t )

Upon evaluating these derivatives at t = 0, the mean and variance of Z

E (Z ) = 0
(5)
Var(Z ) = 1
 The normal The normal distribution (cont.)
distribution

UFC/DC
ATAII (CK0146)
PR (TIP8412)
2017.2

The normal
distribution
Define the continuous random variable X = bZ + a, for b > 0
The multivariate
normal • This is a 1-to-1 transformation
distribution

An application
We want to derive the PDF of X
PCA
• The inverse of the transformation is z = b −1 (x − a)
• The Jacobian is J = b −1

Because b > 0, the PDF of X

1 1 h 1 x − a 2 i
fX (x ) = √ exp − , for − ∞ < x < ∞
2π b 2 b
 The normal The normal distribution (cont.)
distribution

UFC/DC
ATAII (CK0146)
PR (TIP8412)
2017.2

The normal
distribution

The multivariate
normal By Equation (5), we have
distribution

An application E (X ) = a
PCA
Var(X ) = b 2

The PDF of X can be written using those quantities

a can be replaced by µ = E (X )
b 2 can be replaced by σ2 = Var(X )
 The normal The normal distribution (cont.)
distribution

UFC/DC
ATAII (CK0146)
PR (TIP8412)
2017.2

The normal
distribution

The multivariate
normal
Definition 1.1
distribution Normal distribution
An application
PCA A random variable X has a normal distribution if its PDF is

1 h 1 x − µi
f (x ) = √ exp − , x ∈ (−∞, +∞) (6)
2πσ 2 σ

The parameters µ and σ2 are the mean and the variance of X

We often write that X has a N (µ, σ2 ) distribution

 The normal The normal distribution (cont.)
distribution

UFC/DC
ATAII (CK0146)
PR (TIP8412)
2017.2

The normal
distribution

The multivariate
normal
distribution Remark
An application
Consider the random variable Z with the PDF
PCA

1  − z 2
f (z ) = p exp , for − ∞ < x < ∞
(2π) 2

The RV Z is said to have a N (0, 1) distribution

• We call Z a standard normal RV
 The normal The normal distribution (cont)
distribution

UFC/DC
ATAII (CK0146)
PR (TIP8412)
2017.2

The normal
distribution

The multivariate Consider the calculation of the MGF of X

normal
distribution

An application
We use the relationship
PCA
X = σ Z+ µ
|{z} |{z}
b a

Remember that the MGF of Z

2
MZ (t) = e (1/2t ) , for − ∞ < x < ∞
 The normal The normal distribution (cont)
distribution

UFC/DC
ATAII (CK0146)
PR (TIP8412)
2017.2

The normal
distribution

The multivariate
normal
distribution
We obtain,
An application
PCA
      
E exp (tX ) = E exp t(σZ + µ) = exp (µt)E exp (tσZ )
1   1 
= exp (µt) exp σ2 t 2 = exp µt + σ2 t 2
2 2
for −∞ < t < +∞
 The normal The normal distribution (cont)
distribution

UFC/DC
ATAII (CK0146)
PR (TIP8412)
2017.2

The normal
distribution

The multivariate
normal
distribution

An application
Remark
PCA Consider the relationship between Z and X

X ∼ N (µ, σ2 ) if and only if Z = (X − µ)/σ ∼ N (0, 1)


 The normal The normal distribution (cont.)
distribution

UFC/DC
ATAII (CK0146)
PR (TIP8412)
2017.2 Example
The normal Consider the random variable X with the MGF
distribution

The multivariate
normal
distribution

An application 1.05
2
M (t) = e 2t +32t
PCA
M (t)

 1 
1 M (t) = exp µt + σ2 t 2
2

−2 0 2
t ·10−2

X has a normal distribution with µ = 2 and σ2 = 64, N (2, 54)

The random variable Z = (X − 2)/8 has a N (0, 1) distribution


 The normal The normal distribution (cont.)
distribution

UFC/DC
ATAII (CK0146)
PR (TIP8412)
2017.2
Example
The normal We can derive all the moments of the standard normal RV using its MGF
distribution

The multivariate
We can do the same for a general normal RV
normal
distribution

An application Let X ∼ N (µ, σ2 )

PCA

Hence, for all nonnegative integers k , by the binomial theorem1

k  
  X k j
E (X k ) = E (σZ + µ)k = σ E (Z j )µk−j (7)
j =0
j

All the odd moments of Z are zero

All even moments can be calculated

   
1 n n
(x + y)k =
Pn
x n−k y k =
Pn
k =0 k k =0 k
x k y n−k
 The normal The normal distribution (cont.)
distribution

UFC/DC
ATAII (CK0146)
PR (TIP8412)
2017.2

The normal f (x |µ = 3, σ2 = 1/4|1|4)

distribution

The multivariate 0.8

normal
distribution
0.6
An application 1 h 1 x − a 2 i
PCA
0.4 fX (x ) = √ exp −
2πb 2 b
0.2

0
−5 0 5 10
x

The graph is symmetric about a vertical axis at x = µ

√
The graph has its maximum of 1/(σ 2π) at x = µ
The graph has the x -axis as horizontal asymptote
The points of inflection are at x = µ ± σ
 The normal The normal distribution (cont)
distribution

UFC/DC
ATAII (CK0146)
PR (TIP8412)
2017.2

The normal
distribution

The multivariate
normal
distribution
There are many practical applications that involve normal distributions
An application
PCA
We need to be able to readily calculate probabilities

Normal PDFs contain a factor of the form exp (−s 2 )

• Antiderivatives are not in closed-form
• Numerical integration must be used
 The normal The normal distribution (cont)
distribution

UFC/DC
ATAII (CK0146)
PR (TIP8412)
2017.2

The normal
distribution

The multivariate
normal
distribution

An application We can use the relation between normal and standard normal variables
PCA

X ∼ N (µ, σ2 ) if and only if Z = (X − µ)/σ ∼ N (0, 1)

Thus, we need only calculate probabilities for standard normal RVs

 The normal The normal distribution (cont)
distribution

UFC/DC
ATAII (CK0146)
PR (TIP8412)
2017.2

The normal
distribution Consider the CDF of a standard normal random variables Z
The multivariate Z z  − w 2
normal
1
distribution
Φ(z ) = √ exp dw (8)
−∞ 2π 2
An application
PCA

Let X ∼ N (µ, σ2 )

Suppose that we want to compute FX (x ) = P (X ≤ x ) for some x

Then, for Z = (X − µ)/σ

 x − µ  x − u
FX (x ) = P (X ≤ x ) = P Z ≤ =Φ
σ σ
Thus, only integration for Φ(z ) is needed
 The normal The normal distribution
distribution

UFC/DC
ATAII (CK0146)
PR (TIP8412) Suppose we are interested in the value xp such that p = FX (xp ) for some p
2017.2
Take zp = Φ−1 (p), then xp = σzp + µ
The normal
distribution

The multivariate Consider the standard normal density

normal
distribution

An application
PCA
0.3

0.2 1  − z 2
f (z )

Φ(z ) = p exp
(2π) 2
0.1

0
−4 −2 0 2 4
z

The area to the left of zp is p

• Φ(zp ) = p
We can use an abbreviated table of probability
 The normal The normal distribution
distribution

UFC/DC
ATAII (CK0146)
PR (TIP8412)
2017.2

The normal
distribution

The multivariate
normal
distribution

An application Suppose we are interested in Φ(−z ), for some z > 0

PCA
Since the PDF is symmetric,

Φ(−z ) = 1 − Φ(z ) (9)

 The normal The normal distribution (cont.)
distribution

UFC/DC
ATAII (CK0146)
PR (TIP8412)
2017.2

The normal
distribution

The multivariate
normal
distribution

An application Consider a random variable X ∼ N (a, b)

PCA

• pnorm(x,a,b), P (X ≤ x )
• dnorm(x,a,b), the PDF of X at x
 The normal The normal distribution (cont.)
distribution

UFC/DC
ATAII (CK0146) Example
PR (TIP8412)
2017.2 Let the random variable X ∼ N (2, 25)

The normal ·10−3

distribution

f (x |µ = 2, σ2 = 25)
The multivariate 3
normal
distribution

An application 2
PCA

0
−20 0 20
x

Using abbreviated tables of probabilities

 10 − 2  0 − 2
P (0 < X < 10) = Φ −Φ
5 5
= Φ(1.6) − Φ(−0.4)
= 0.945 − (1 − 0.655) = 0.600
 The normal The normal distribution (cont.)
distribution

UFC/DC
ATAII (CK0146)
PR (TIP8412)
2017.2

The normal
distribution

The multivariate
normal
distribution

An application
 1 − 2
 − 8 − 2
PCA
P (−8 < X < 1) = Φ −Φ
5 5
= Φ(−0.2) − Φ(−2)
= (1 − 0.579) − (1 − 0.977) = 0.398


 The normal The normal distribution (cont.)
distribution

UFC/DC
ATAII (CK0146)
PR (TIP8412)
2017.2

The normal
distribution

The multivariate
normal
distribution
Example
An application Let the random variable X ∼ N (µ, σ2 )
PCA

 µ + 2σ − µ  µ − 2σ − µ
P (µ − 2σ < X < µ + 2σ) = Φ −Φ
σ σ
(10)
= Φ(2) − Φ(−2)
= 0.977 − (1 − 0.977) = 0.954


 The normal The normal distribution (cont.)
distribution

UFC/DC
ATAII (CK0146)
PR (TIP8412)
2017.2

The normal
distribution

The multivariate Theorem 1.1

normal
distribution Consider a random variable X ∼ N (µ, σ2 ) with σ2 > 0
An application
PCA
The random variable V = (X − µ)2 /σ2 is χ2 (1)

Proof

The random variable V = W 2 , with W = (X − µ)/σ ∼ N (0, 1)

The CDF G(v ) for V

√ √
G(v ) = P (W 2 ≤ v ) = P (− v ≤ W ≤ v), for v ≥ 0
 The normal The normal distribution (cont.)
distribution

UFC/DC
ATAII (CK0146)
PR (TIP8412)
2017.2

The normal
distribution
That is, √
The multivariate Z
normal
v 1 2
distribution G(v ) = 2 √ e (−w /2) dw , for 0 ≤ v
0 2π
An application
PCA Moreover,
G(v ) = 0, for v < 0

√
Consider a change of the integration variable, w = y
Z v 1 (−y/2)
G(v ) = √ √ e dy, for 0 ≤ v
0 2π y
 The normal The normal distribution (cont.)
distribution

UFC/DC
ATAII (CK0146)
PR (TIP8412)
2017.2

The normal
distribution Hence, the PDF g(v ) = G ′ (v ) of random variable V of the continuous type
The multivariate 
normal
distribution  1√ v (1/2−1) e (−v/2) , 0 < v < ∞

√
An application g(v ) = π 2

0,
PCA elsewhere

Since g(v ) is a PDF, Z ∞

g(v )dv = 1,
0

√
Then, it must be Γ(1/2) = π and therefore V ∼ χ2 (1)

 The normal The normal distribution (cont.)
distribution

UFC/DC A main property of the normal distribution is additivity under independence

ATAII (CK0146)
PR (TIP8412)
2017.2
Theorem 1.2
The normal Let X1 , . . . , Xn be independent random variables
distribution

The multivariate
normal
Suppose that Xi ∼ N (µi , σi2 ), for i = 1, . . . , n,
distribution

An application
Pn
Let Y = ai Xi , for some constants a1 , . . . , an
i=1
PCA
P P
Then, the distribution of Y is N ( ni=1 ai µi , ni=1 ai2 σi2 )

Proof

For t ∈ R, the MGF of Y

n
Y  
MY (t) = exp tai µi + (1/2)t 2 ai2 σi2
i=1
h Xn n
X i
= exp t ai µi + (1/2)t 2 ai2 σi2
i=1 i=1
P P
This is the MGF of a N ( ni=1 ai µi , ni=1 ai2 σi2 ) distribution

 The normal The normal distribution (cont.)
distribution

UFC/DC
ATAII (CK0146)
PR (TIP8412)
2017.2
Let X1 , X2 , . . . , Xn be random sample from a N (µ, σ2 )

The normal
A corollary gives the distribution of the sample mean
distribution
n
The multivariate 1X
normal X = Xi
distribution n i=1
An application
PCA

Corollary
Let X1 , . . . , Xn be IID random variables with common N (µ, σ2 ) distribution

1 Pn
Let X = i=1 Xi
n
Then, X has a N (µ, σ2 /n) distribution

Proof

In Theorem 1.2, take ai = (1/n), µi = µ and σi2 = σ2 , for i = 1, 2, . . . , n


 The normal
distribution

UFC/DC
ATAII (CK0146)
PR (TIP8412)
2017.2

The normal
distribution

The multivariate
normal
distribution

An application The multivariate normal

PCA
distribution
Useful distributions
 The normal The multivariate normal distribution
distribution

UFC/DC
ATAII (CK0146)
PR (TIP8412)
2017.2

The normal
distribution

The multivariate
normal
distribution
The multivariate normal distribution for a n-dimensional random vector
An application
PCA • Examples for the bivariate case, n = 2

The derivation is simplified by first discussing the standard case

• Then, we proceed with the general cases
 The normal The multivariate normal distribution (cont.)
distribution

UFC/DC
ATAII (CK0146)
PR (TIP8412)
2017.2

The normal
distribution

The multivariate Consider the random vector Z = (Z1 , . . . , Zn )′

normal
distribution • Z1 , . . . , Zn are IID N (0, 1) random variables
An application
PCA
Then, the density of Z
n n
Y 1 n 1 o  1 n/2 n 1X o
fZ (z) = √ exp − zi2 = exp − zi2
i=1
2π 2 2π 2 i=1
(11)
 1 n/2 n 1 o
= exp − z′ z , for z ∈ Rn
2π 2
 The normal The multivariate normal distribution (cont.)
distribution

UFC/DC
ATAII (CK0146)
PR (TIP8412)
Consider the random vector Z = (Z1 , Z2 )′
2017.2
Since n = 2, we have
The normal
distribution n n
Y 1 n 1 o  1 n/2 n 1X o
fZ (z) = √ exp − zi2 = exp − z2 ,
The multivariate
normal i=1
2π 2 2π 2 i=1 i
distribution
n 2
X o
An application 
PCA
= 1/(2π) exp − 1/2 zi2 = 1/(2π) exp − 1/2(z12 + z22 )
i=1

fZ1 ,Z2 (z1 , z2 )

0.1

4
0 2
−4 −2 0
0 2 −2
4 −4 z2
z1
 The normal The multivariate normal distribution (cont.)
distribution

UFC/DC
ATAII (CK0146)
PR (TIP8412)
2017.2

The normal
distribution

The multivariate
normal
distribution The Zi s have mean zero, unit variance and they are uncorrelated
An application
PCA
Then, the mean and covariance matrix of Z

E (Z) = 0
(12)
Cov(Z) = In

• In indicates an identity matrix of order n

 The normal The multivariate normal distribution
distribution

UFC/DC
ATAII (CK0146)
PR (TIP8412)
2017.2

The normal
distribution
2
The multivariate
normal
The MGF of each of the Zi s evaluated at ti is e (ti /2)
distribution

An application
As the Zi s are independent, the MGF of Z
PCA
n
hY i Yn
MZ (t) = E exp {ti Zi } = E [exp {ti Zi }]
i=1 i=1
n
(13)
n1X o n1 o
= exp ti2 = exp t′ t , for all t ∈ Rn
2 i=1 2
 The normal The multivariate normal distribution
distribution

UFC/DC
ATAII (CK0146) Consider the random vector Z = (Z1 , Z2 )′
PR (TIP8412)
2017.2
Since n = 2, we have
The normal n n
distribution
hY i Y
MZ (t) = E exp {ti Zi } = E [exp {ti Zi }]
The multivariate
normal i=1 i=1
distribution 2
n1X o 
An application = exp ti2 = exp 1/2(t12 + t22 ) , for all (t1 , t2 ) ∈ R2
PCA 2 i=1

1.04
MZ1 ,Z2 (t1 , t2 )

1.02

1 0.2
−0.2 0
0
0.2−0.2
t2
t1
 The normal The multivariate normal distribution (cont.)
distribution

UFC/DC
ATAII (CK0146)
PR (TIP8412)
2017.2

The normal
distribution

The multivariate
normal
distribution

An application
PCA Z is said to have a multivariate normal distribution

Z has a Nn (0, In ) distribution, mean 0, covariance matrix In

 The normal The multivariate normal distribution (cont.)
distribution

UFC/DC
ATAII (CK0146)
PR (TIP8412)
Let Σ be a n × n, symmetric and positive-definite matrix (z′ Σz > 0, ∀z)
2017.2

The normal From linear algebra, we can always decompose Σ

distribution

The multivariate Σ = Γ′ ΛΓ (14)

normal
distribution
Λ is the diagonal matrix Λ = diag(λ1 , λ2 , . . . , λn )
An application
 
PCA
λ1 0 ··· 0
0 λ2 ··· 0
 
Λ= . .. .. .. 
 .. . . . 
0 0 ··· λn
λ1 ≥ λ2 ≥ · · · ≥ λn ≥ 0 are the eigenvalues of Σ
The columns of Γ′ , v1 , v2 , · · · , vn , are the corresponding eigenvectors
 
v1,1 v2,1 ··· vn,1
 v1,2 v2,2 ··· vn,2 
 
Γ′ =  . . . 
 .. .. .. 
v1,n v2,n ··· vn,n
 The normal The multivariate normal distribution (cont.)
distribution

UFC/DC
ATAII (CK0146)
PR (TIP8412)
2017.2

The normal
distribution

The multivariate
normal
distribution

An application
PCA Σ = Γ′ ΛΓ

The factorisation is called the spectral decomposition of Σ

• Matrix Γ is orthogonal (Γ−1 = Γ′ , thus ΓΓ′ = I)
 The normal The multivariate normal distribution (cont.)
distribution

UFC/DC
ATAII (CK0146)
PR (TIP8412)
2017.2

The normal
distribution

The multivariate
Remark
normal Eigenvalues and eigenvectors
distribution

An application A (n × n) matrix A can be used to transform a n-vector x to another one y

PCA

y = Ax

Consider a scalar variable λi and a particular value of x (= ei ) such that

yei = Aei = λi ei
λi is often called an eigenvalue of A
ei is the corresponding eigenvector
 The normal The multivariate normal distribution (cont.)
distribution

UFC/DC
ATAII (CK0146)
PR (TIP8412)
2017.2

The normal
distribution Aei = λi ei
The multivariate
normal
distribution
Every (n × n) matrix has n eigenvalues and up to n eigenvectors
An application
They can be jointly specified
PCA

(λ1 In − A)e1 = 0
(λ2 In − A)e2 = 0
···
(λn In − A)en = 0

If (λi In − A)ei = 0 is satisfied, then so is (λi In − A)αei = 0 (i = 1, . . . , n)

Eigenvectors are specified up to any multiplicative constant α
 The normal The multivariate normal distribution (cont.)
distribution

UFC/DC
ATAII (CK0146)
PR (TIP8412)
2017.2
The set of n vector equations can have non-trivial solutions
The normal
distribution
The n values of λi need be solutions to the scalar equation
The multivariate
normal
distribution |λIn − A| := ∆(λ) = 0
An application
(λIn − A) is called the characteristic matrix of A
PCA

∆(λ) is the characteristic polynomial of A

The determinant gives a n-degree polynomial in λ

• It can be factored as a product of n binomials

∆(λ) = λn + cn−1 λn−1 + · · · + c1 λ + c0

= (λ − λ1 )(λ − λ2 ) · · · (λ − λn )
=0

• Each of the binomial roots is an eigenvalue of A

 The normal The multivariate normal distribution (cont.)
distribution

UFC/DC
ATAII (CK0146)
PR (TIP8412)
2017.2

The normal
distribution
If the roots are all distinct, there will be n independent eigenvectors
The multivariate
normal If the roots are repeated, there may be fewer eigenvectors
distribution

An application
PCA
For a distinct eigenvalue λi , the eigenvector ei is contained in Adj(λi In − A)
 
Adj(λi In − A) = α1 ei α2 ei · · · αn ei

Since αj are arbitrary constants, any non-zero column represents ei

Adj(λi In − A) is computed for each root (i = 1, . . . , n)

• A single eigenvector is chosen from each evaluation
 The normal The multivariate normal distribution (cont.)
distribution

UFC/DC
ATAII (CK0146)
PR (TIP8412)
2017.2

The normal
distribution

The multivariate
normal
distribution

An application Together, the n eigenvectors form the columns of the modal matrix E
PCA
 
E = e1 e2 · · · en

Here, the eigenvectors are scaled so that |ei | = 1 (i = 1, . . . , n)

 The normal The multivariate normal distribution (cont.)
distribution

UFC/DC
ATAII (CK0146)
PR (TIP8412)
2017.2
The modal matrix can be used to diagonalise A
The normal
distribution To transform A into a diagonal matrix (of eigenvalues)
The multivariate
normal
distribution  
Ae1 = λ1 e1
An application  Ae2 = λ2 e2 
PCA  
 . ..  , or AE = EΛ
 .. . 
Aen = λn en

Λ is a diagonal matrix of eigenvalues

 
λ1 0 ··· 0
0 λ2 ··· 0
 
Λ= . .. .. .. 
 .. . . . 
0 0 ··· λn
 The normal The multivariate normal distribution (cont.)
distribution

UFC/DC
ATAII (CK0146)
PR (TIP8412)
2017.2

The normal
distribution

The multivariate
normal E is non-singular because the ei are linearly independent
distribution

An application A = EΛE−1
PCA

From the diagonal matrix of eigenvalues to the original square matrix

The inverse transformation

Λ = E−1 AE
 The normal The multivariate normal distribution (cont.)
distribution

UFC/DC
ATAII (CK0146)
PR (TIP8412)
2017.2

The normal
distribution If A is a real, symmetric matrix, its eigenvalues are real
The multivariate • The eigenvectors are orthogonal to each other
normal
distribution There are n eigenvectors, even with repeated roots
An application
PCA
The eigenvectors corresponding to distinct eigenvalues
 
Adj(λi In − A) = α1 ei α2 ei · · · αn ei

The eigenvectors of eigenvalues λi with multiplicity m

d m−1    
Adj(λi In − A)  = ··· ei1 ei2 ··· eim ···
dλm−1 λ=λi


 The normal The multivariate normal distribution (cont.)
distribution

UFC/DC
ATAII (CK0146)
PR (TIP8412)
2017.2

The normal Remark

distribution
An alternative way to write the spectral decomposition
The multivariate
normal n
distribution X
Σ = Γ′ ΛΓ = λi vi vi′ (15)
An application
i=1
PCA

   
v1,1 v2,1 ··· vn,1 λ1 0 ··· 0 v1,1 v1,2 ··· v1,n
 v1,2 v2,2 ··· vn,2   0 λ2 ··· 0   v2,1 v2,2 ··· v2,n 
   
 . . .  . . .. .  . . . 
 .. .. ..   .. .. . ..   .. .. .. 
v1,n v2,n ··· vn,n 0 0 ··· λn vn,1 vn,2 ··· vn,n
| {z }| {z }
λ2 v2 ′
v2


 The normal The multivariate normal distribution (cont.)
distribution

UFC/DC
ATAII (CK0146)
PR (TIP8412)
2017.2

The normal
Because the λi s are non-negative, we can define the diagonal matrix
distribution
p p p
The multivariate Λ1/2 = diag( λ1 , λ2 , . . . , λn )
normal  1/2 
distribution λ1 0 ··· 0
An application  0
 λ2 1/2 · · · 0  
PCA =  .. .. .. .. 
 . . . . 
0 0 · · · λn 1/2

Then, the orthogonality of Γ implies

Σ = Γ′ Λ1/2 ΓΓ ′ 1/2
|{z} Λ Γ
I
| {z }
Λ
 The normal The multivariate normal distribution (cont.)
distribution

UFC/DC
ATAII (CK0146)
PR (TIP8412)
2017.2

The normal
distribution Σ = Γ′ Λ1/2 ΓΓ′ Λ1/2 Γ
|{z}
The multivariate I
normal | {z }
distribution
Λ
An application
PCA
Define the square root of the positive semi-definite matrix Σ

Σ1/2 = Γ′ Λ1/2 Γ (16)

Σ1/2 is symmetric and positive semi-definite

Suppose Σ is positive definite and all its eigenvalues are strictly positive

Then, we have
(Σ1/2 )−1 = Γ′ Λ−1 Γ = Σ−1/2 (17)
 The normal The multivariate normal distribution (cont.)
distribution

UFC/DC
ATAII (CK0146)
PR (TIP8412)
2017.2

The normal Let Z have a Nn (0, In ) distribution

distribution

The multivariate Let Σ be a positive semi-definite symmetric matrix

normal
distribution
Let µ be a n × 1 vector of constants
An application
PCA

Define the random vector X

X = Σ1/2 Z + µ (18)

By Equation (12), we have

E [X] = µ
(19)
Cov[X] = Σ1/2 Σ1/2 = Σ
 The normal The multivariate normal distribution (cont.)
distribution

UFC/DC
ATAII (CK0146)
PR (TIP8412)
2017.2

The normal
distribution

The multivariate
normal
distribution The MGF of X
An application     
PCA
MX (t) = E exp t′ X = E exp {t′ Σ1/2 Z + t′ µ}
 
′ 
= exp (t′ µ)E exp Σ1/2 t Z

′  (20)
= exp (t′ µ) exp (1/2) Σ1/2 t Σ1/2 t
 
= exp (t′ µ) exp (1/2)t′ Σt
 The normal The multivariate normal distribution (cont.)
distribution

UFC/DC
ATAII (CK0146)
PR (TIP8412)
2017.2

The normal
distribution Definition
The multivariate Multivariate normal
normal
distribution
A n-dimensional random vector X has a multivariate normal distribution
An application
PCA
if its MGF is  
MX (t) = exp t′ µ + (1/2)t′ Σt (21)
for all t ∈ Rn

• Σ is a symmetric, positive semi-definite matrix

• µ ∈ Rn

We say that X has a Nn (µ, Σ) distribution

 The normal The multivariate normal distribution (cont.)
distribution

UFC/DC
ATAII (CK0146)
PR (TIP8412)
2017.2

The normal
distribution

The multivariate
normal
distribution The definition is for positive semi-definite positive matrices Σ
An application • Σ is usually positive definite
PCA
We can get the density of X

If Σ is positive definite, then so is Σ1/2

• Its inverse is (Σ1/2 )−1 = Γ′ Λ−1 Γ = Σ−1/2
 The normal The multivariate normal distribution (cont.)
distribution

UFC/DC
ATAII (CK0146)
PR (TIP8412)
2017.2

The normal
distribution

The multivariate
normal X = Σ1/2 Z + µ
distribution

An application
The transformation between X and Z is 1-to-1
PCA

The inverse transformation

Z = Σ−1/2 (X − µ)

The Jacobian  −1/2   −1/2

Σ  =  Σ
 The normal The multivariate normal distribution (cont.)
distribution

UFC/DC
ATAII (CK0146)
PR (TIP8412)
2017.2

The normal
distribution

The multivariate
normal
distribution

An application Hence, upon simplification, the PDF of X

PCA

1 h 1 i
fX (x) = exp − (x − µ)′ Σ−1 (x − µ) , for x ∈ Rn (22)
(2π)n/2 |Σ|1/2 2
 The normal The multivariate normal distribution (cont.)
distribution

UFC/DC
ATAII (CK0146)
PR (TIP8412)
2017.2

The normal
distribution

The multivariate
normal
distribution

An application Theorem 2.1

PCA
Suppose X has a Nn (µ, Σ) distribution

Let Y = AX + b, where A is a m × n matrix and b ∈ Rm

Then, Y has a Nm (Aµ + b, AΣA′ ) distribution

 The normal The multivariate normal distribution (cont.)
distribution

UFC/DC
ATAII (CK0146)
PR (TIP8412)
2017.2

The normal Proof

distribution
 
The multivariate From MX (t) exp t′ µ + (1/2)t′ Σt , for all t ∈ Rm
normal
distribution
The MGF of Y
An application 

PCA MY (t) = E exp t′ Y
  
= E exp t′ (AX + b)
 
′ 
= exp (t′ b)E exp A′ t X

′

′

= exp (t′ b) exp A′ t µ + 1/2 A′ t Σ A′ t
 ′

′ 
= exp t Aµ + b + 1/2 t AΣA′ t

This is the MGF of a Nm (Aµ + b, AΣA′ ) distribution


 The normal The multivariate normal distribution (cont.)
distribution

UFC/DC
ATAII (CK0146)
PR (TIP8412)
2017.2

The normal
distribution

The multivariate
normal
A corollary gives us the marginal distributions of a multivariate normal RV
distribution

An application
PCA
Let X1 be any sub-vector of X of dimension m < n

We can always rearrange means and correlations

There is no loss in writing  

X1
X= (23)
X2
X2 is of dimension p = n − m
 The normal The multivariate normal distribution (cont.)
distribution

UFC/DC
ATAII (CK0146)
PR (TIP8412)
2017.2

The normal
distribution

The multivariate In the same way, we partition mean and covariance matrix of X
normal
distribution  
µ1
An application µ=
PCA
µ2
  (24)
Σ11 Σ12
Σ=
Σ21 Σ22

• Σ11 is the covariance matrix of X1

• Σ12 contains all covariances between the components of X1 and X2
 The normal The multivariate normal distribution (cont.)
distribution

UFC/DC
ATAII (CK0146)
PR (TIP8412)
2017.2

The normal
distribution
We define A to be the matrix
h  i
The multivariate 
normal A = Im  0mp
distribution
 
An application
1 0 ··· 0 0 ··· ··· 0
PCA 0 1 · · · 0 0 ··· ··· 0
 
= . . .. .. .. .. .. .. 
 .. .. . . . . . .
0 0 ··· 1 0 ··· ··· 0

Im indicates a m × m identity matrix

0mp indicates a m × p matrix of zeros

Then, X1 = AX
 The normal The multivariate normal distribution (cont.)
distribution

UFC/DC
ATAII (CK0146)
PR (TIP8412) We apply Theorem 2.1 to the transformation X1 = AX
2017.2

We have the following corollary

The normal
distribution

The multivariate
Corollary 2.1
normal
distribution Suppose X has a Nn (µ, Σ) distribution, partitioned as in Equation (23-24)
An application  
PCA µ1
µ=
µ2
 
Σ11 Σ12
Σ=
Σ21 Σ22

Then, X1 has a Nm (µ1 , Σ11 ) distribution



This is a useful result

The corollary shows that any marginal distribution of X is also normal

Mean and covariance matrix are those from the partial vector
 The normal The multivariate normal distribution (cont.)
distribution

UFC/DC
ATAII (CK0146)
PR (TIP8412)
2017.2 Example
The normal Consider the multivariate normal in the case when n = 2
distribution
The bivariate normal
The multivariate
normal
distribution We use common notation (X , Y ) rather than (X1 , X2 )
An application
PCA Suppose that (X , Y ) ∼ N2 (µ, Σ)
 
µ1
µ=
µ2
  (25)
σ22 σ12
Σ=
σ21 σ22

µ1 and σ12 are the mean and variance of X

µ2 and σ22 are the mean and variance of Y

σ12 = ρ(σ1 σ2 ) is the covariance between X and Y

ρ is the correlation coefficient between X and Y
 The normal The multivariate normal distribution (cont.)
distribution

UFC/DC
ATAII (CK0146)
PR (TIP8412)
2017.2

The normal
distribution Substituting ρσ1 σ2 for σ12 in Σ
The multivariate  2 
normal σ2 ρσ1 σ2
Σ=
distribution
ρσ2 σ1 σ22
An application
PCA

The determinant of Σ is σ12 σ22 (1 − ρ2 ) (remember that ρ2 ≤ 1)

• Suppose that ρ2 < 1, so that Σ is invertible

The inverse of Σ
 
1 σ22 −ρσ1 σ2
Σ−1 = (26)
σ12 σ22 (1 − ρ2 ) −ρσ2 σ1 σ12
 The normal The multivariate normal distribution (cont.)
distribution

UFC/DC
ATAII (CK0146)
PR (TIP8412)
2017.2

The normal
distribution

The multivariate
normal
From this expression, the PDF of (X , Y )
distribution
1
e (−q/2) ,
An application
f (x , y) = p −∞ < x , y < ∞ (27)
PCA
2πσ1 σ2 1 − ρ2

with
1 h x − µ1 2  x − µ1  x − µ2   y − µ2 2 i
q= − 2ρ + (28)
1− ρ2 σ1 σ1 σ2 σ2
 The normal The multivariate normal distribution (cont.)
distribution

UFC/DC
ATAII (CK0146)
PR (TIP8412)
2017.2

The normal
distribution

The multivariate
normal
distribution

An application
In general, if X and Y are independent RVs, correlation coefficient is zero
PCA

If they are normal, then X ∼ N (µ1 , σ12 ) and Y ∼ N (µ2 , σ22 )

• (by Corollary 2.1)
 The normal The multivariate normal distribution (cont.)
distribution

UFC/DC
ATAII (CK0146)
PR (TIP8412)
2017.2

The normal
distribution
1
The multivariate f (x , y) = p e (−q/2) , −∞ < x , y < ∞
normal
distribution
2πσ1 σ2 1 − ρ2
An application
h x − µ1 2  y − µ2 2 i
PCA q= +
σ1 σ2

From the joint PDF of (X , Y ), let the correlation coefficient be zero

• X and Y are independent
For the bivariate normal, independence corresponds to ρ = 0

The generalisation to the multivariate case also holds true


 The normal The multivariate normal distribution (cont.)
distribution

UFC/DC
ATAII (CK0146)
PR (TIP8412)
2017.2 If two random variables are independent their covariance is zero
• The converse is not necessarily true
The normal
distribution

The multivariate
Yet, it can be shown that this is true for the multivariate normal
normal
distribution

An application
PCA
Theorem 2.2
Suppose that X ∼ Nn (µ, Σ)

Suppose the partitioning

 
µ1
µ=
µ2
 
Σ11 Σ12
Σ=
Σ21 Σ22

Then, X1 and X2 are independent if and only if Σ12 = 0


 The normal The multivariate normal distribution (cont.)
distribution

UFC/DC
ATAII (CK0146)
PR (TIP8412)
2017.2

The normal
distribution
Proof
The multivariate
normal
distribution Note that Σ12 = Σ′21
An application
PCA The joint MGF of X1 and X2

Mx1 ,x2 (t1 , t2 ) =

n 1 ′ o
exp t′1 µ1 + t′2 µ2 + t1 Σ11 t1 + t′2 Σ22 t2 + t′2 Σ21 t1 + t′1 Σ12 t2 (29)
2
We used
t′ = (t′1 , t′2 )
 The normal The multivariate normal distribution (cont.)
distribution

UFC/DC
ATAII (CK0146)
PR (TIP8412)
2017.2

The normal
distribution

The multivariate
By Corollary (2.1),
normal
distribution
• X1 ∼ Nm (µ1 , Σ11 )
An application
PCA • X2 ∼ Np (µ2 , Σ22 )

The product of their marginal MGFs

n 1 ′ o
Mx1 (t1 )Mx2 (t2 ) = exp t′1 µ1 + t′2 µ2 + t1 Σ11 t1 + t′2 Σ22 t2 (30)
2

For X1 and X2 be independent, Equation (29) and (30) must be identical

 The normal The multivariate normal distribution (cont.)
distribution

UFC/DC
ATAII (CK0146)
PR (TIP8412)
2017.2

The normal
distribution

The multivariate
normal
distribution n 1 ′ o
An application
exp t′1 µ1 + t′2 µ2 + t1 Σ11 t1 + t′2 Σ22 t2 + t′2 Σ21 t1 + t′1 Σ12 t2
2
PCA

n 1 ′ o
exp t′1 µ1 + t′2 µ2 + t1 Σ11 t1 + t′2 Σ22 t2
2

Independence of X1 and X2 is verified when Σ12 = 0′ and hence Σ21 = 0

The covariances between their components are all 0


 The normal The multivariate normal distribution (cont.)
distribution

UFC/DC
ATAII (CK0146)
PR (TIP8412)
2017.2

The normal
distribution Theorem
The multivariate Suppose X has a Nn (µ, Σ) distribution, partitioned as in Equation (23-24)
normal
distribution  
µ1
An application µ=
µ2
PCA
 
Σ11 Σ12
Σ=
Σ21 Σ22

Assume that Σ is positive definite

Then, the conditional distribution of X1 |X2

h i
Nm µ1 + Σ12 Σ−1 −1
22 (X2 − µ2 ), Σ11 − Σ12 Σ22 Σ21
 The normal The multivariate normal distribution (cont.)
distribution

UFC/DC
ATAII (CK0146)
PR (TIP8412)
2017.2

The normal
distribution
Proof
The multivariate
normal
distribution Consider the joint distribution of W = X1 − Σ12 Σ−1
22 X2 and X2
An application
The joint distribution can be obtained from the transformation
PCA
    
W I −Σ12 Σ−1 X1
= m 22
X2 0 Ip X2
| {z }
A

This is a linear transformation

The joint distribution is multivariate normal (Theorem 2.1)

 The normal The multivariate normal distribution (cont.)
distribution

UFC/DC
ATAII (CK0146)
PR (TIP8412)
2017.2

The normal
distribution The means
The multivariate
normal
• E (W) = µ1 − Σ12 Σ−1
22 µ2
distribution
• E (X2 ) = µ2
An application
PCA
The covariance matrix
   
Im −Σ12 Σ−1
22 Σ11 Σ12 Im 0′
0 Ip Σ21 Σ22 −Σ−1
22 Σ21 Ip
| {z }| {z }| {z }
A Σ A′
 
Σ11 − Σ12 Σ−1
22 Σ21 0′
=
0 Σ22
 The normal The multivariate normal distribution (cont.)
distribution

UFC/DC
ATAII (CK0146)
PR (TIP8412)
2017.2

The normal
distribution

The multivariate
By Theorem 2.2, the random vectors W and X2 are independent
normal
distribution
• The conditional of W2 |X2 equals the marginal of W
An application


PCA W|X2 ∼ Nm µ1 − Σ12 Σ−1 −1
22 µ2 , Σ11 − Σ12 Σ22 Σ21

Because of this independence,

W + Σ12 Σ−1
22 X2 |X2


∼ Nm µ1 − Σ12 Σ−1 −1 −1
22 µ2 + Σ12 Σ22 X2 , Σ11 − Σ12 Σ22 Σ21


 The normal The multivariate normal distribution (cont.)
distribution

UFC/DC
ATAII (CK0146)
PR (TIP8412)
2017.2 Example
The normal Suppose that (X , Y ) ∼ N2 (µ, Σ)
distribution
 
The multivariate µ1
normal µ=
distribution
µ2
 
An application σ22 σ12
Σ=
PCA σ21 σ22

µ1 and σ12 are the mean and variance of Y (= X1 )

µ2 and σ22 are the mean and variance of X (= X2 )

Nm µ1 − Σ12 Σ−1 −1 −1
22 µ2 + Σ12 Σ22 X2 , Σ11 − Σ12 Σ22 Σ21

The expression shows the conditional distribution of Y given X = x

h σ2 i
N µ2 + ρ (x − µ1 ), σ22 (1 − ρ2 )
σ1
 The normal The multivariate normal distribution (cont.)
distribution

UFC/DC
ATAII (CK0146)
PR (TIP8412)
2017.2

The normal The coefficient of x in the conditional mean of E (Y |x ) is ρσ2 /σ1

distribution

The multivariate σ2
normal E (Y |x ) = µ2 + ρ
distribution σ1
An application
PCA The conditional mean of Y , given that X = x is linear in x

The σ1 and σ2 are the respective standard deviations

ρ is the correlation coefficient of X and Y

Remark
This follows from the fact that the coefficient of x in a linear conditional
mean E (Y |x ) is the product of correlation coefficient and σ2 /σ1
 The normal The multivariate normal distribution (cont.)
distribution

UFC/DC
ATAII (CK0146)
PR (TIP8412)
2017.2

The normal The mean of the conditional distribution of Y given X = x

distribution
• It depends upon x (unless ρ = 0)
The multivariate
normal
distribution
The variance σ22 (1 − ρ2 ) is the same for all real values of x
An application
PCA Thus, given
p that X = x , the conditional probability that Y is within (≈
2.57)σ2 (1 − ρ2 ) units of the conditional mean is 0.99, whatever is x

Most of the probability for the distribution of (X , Y ) lies within the band

σ2 p
µ2 + ρ (x − µ1 ) ± (≈ 2.57)σ2 1 − ρ2
σ1
about the plot of the linear conditional mean

 The normal The multivariate normal distribution (cont.)
distribution

UFC/DC
ATAII (CK0146)
PR (TIP8412)
2017.2

The normal
distribution
Theorem
The multivariate
normal Suppose X has a Nn (µ, Σ) distribution
distribution

An application Let Σ is positive definite

PCA
Then, the RV W = (X − µ)′ Σ(X − µ) has a χ2 (n) distribution

Proof

Write Σ = Σ1/2 Σ1/2 , with Σ1/2 = Γ′ Λ1/2 Γ

Then, Z = Σ−1/2 (X − µ) is Nn (0, In )

Pn
Let W = Z′ Z = i=1 Zi2
 The normal The multivariate normal distribution (cont.)
distribution

UFC/DC
ATAII (CK0146)
PR (TIP8412)
2017.2

The normal
distribution

The multivariate
normal
distribution Each Zi has a N (0, 1) distribution, for i = 1, 2, . . . , n
An application
From Theorem 1.1, it follows that Zi2 has a χ2 (1) distribution
PCA

Variables Z1 , . . . , Zn are independent standard normal RVs

Pn 2 2
• Thus, i=1 Zi = W has a χ (n) distribution
• (by an earlier corollary)


 The normal
distribution

UFC/DC
ATAII (CK0146)
PR (TIP8412)
2017.2

The normal
distribution

The multivariate
normal
distribution

An application
PCA
An application
The normal distribution
 The normal
distribution

UFC/DC
ATAII (CK0146)
PR (TIP8412)
2017.2

The normal
distribution

The multivariate
normal
distribution

An application
PCA
Principal components analysis
An application
 The normal Principal components analysis
distribution

UFC/DC
ATAII (CK0146)
PR (TIP8412)
2017.2

The normal
distribution

The multivariate
normal
distribution
We consider applications of the multivariate normal distribution
An application
• Principal components analysis (PCA)
PCA

It results in a linear function of a multivariate normal random vector

• The function preserves the ‘total variation in the problem’
• The random vector has independent components
 The normal Principal components analysis (cont.)
distribution

UFC/DC
ATAII (CK0146)
PR (TIP8412)
2017.2
Let the random vector X has the multivariate normal distribution Nn (µ, Σ)
The normal
distribution • Σ is positive definite
The multivariate
normal
distribution

An application
Consider the spectral decomposition of Σ = Γ′ ΛΓ
PCA The eigenvalues form the main diagonal of Λ

λ1 , λ2 , . . . , λn

The corresponding eigenvectors are the columns of Γ

v1 , v2 , . . . , vn

Assume without loss of generality that the eigenvalues are sorted

λ1 ≥ λ2 ≥ · · · ≥ λn > 0
 The normal Principal components analysis (cont.)
distribution

UFC/DC
ATAII (CK0146)
PR (TIP8412)
2017.2

The normal
distribution

The multivariate
normal
distribution
Define the random vector Y = Γ(X − µ)
An application
PCA ΓΣΓ′ = Λ, thus Y has a Nn (0, Λ) distribution (by Theorem 2.1)
The components Y1 , Y2 , . . . , Yn are independent RVs
Yi has a N (0, λi ) distribution, for i = 1, 2, . . . , n

The random vector Y is the vector of principal components

 The normal Principal components analysis (cont.)
distribution

UFC/DC
ATAII (CK0146)
PR (TIP8412)
2017.2

The normal
distribution

The multivariate
normal
distribution Total variation of a random vector sums the variances of its components
An application
PCA
For a random vector X, because Γ is an orthogonal matrix
n
X n
X
TV (X) = σi2 = Tr(Σ) = Tr(Γ′ ΛΓ) = Tr(ΛΓΓ′ ) = λi = TV (Y)
i=1 i=1

Hence, X and Y share the same total variation

 The normal Principal components analysis (cont.)
distribution

UFC/DC
ATAII (CK0146)
PR (TIP8412)
2017.2

The normal
distribution

The multivariate
normal
distribution
Consider the first component of Y
An application
PCA
Y1 = v1′ (X − µ)

• This is a linear combination of the components of X − µ

Pn
• Because of orthogonality, ||v1 ||2 = 2
j =1 v1j =1
 The normal Principal components analysis (cont.)
distribution

UFC/DC
ATAII (CK0146)
PR (TIP8412)
2017.2
Consider any other linear combination of (X − µ)
The normal
distribution
a′ (X − µ), with ||a||2 = 1
The multivariate
normal
distribution
As a ∈ Rn and because {v1 , . . . , vn } form a basis for Rn
An application
PCA We must have
n
X
a= aj vj
j =1

for some set of scalars a1 , a2 , . . . , an

Furthermore, the basis {v1 , . . . , vn } is orthonormal

n
X ′ n
X
a ′ vi = aj vj vi = aj vj′ vi = ai
j =1 j =1
 The normal Principal components analysis (cont.)
distribution

UFC/DC
ATAII (CK0146)
PR (TIP8412)
2017.2

The normal
distribution Pn
We have that Σ = Γ′ ΛΓ = i=1 λi vi vi′ and that λi > 0
The multivariate
normal
distribution
Then, the inequality
An application n
X
PCA
Var(a′ X) = a′ Cov(X)a = a′ Σa = λi (a′ vi )2
i=1
n n
(31)
X X
= λi ai2 ≤ λ1 ai2 = λ1 = Var(Y1 )
i=1 i=1

Y1 has the maximum variance of any other linear combination

Y1 is called the first principal component of X
 The normal Principal components analysis (cont.)
distribution

UFC/DC
ATAII (CK0146)
PR (TIP8412)
2017.2

The normal
distribution

The multivariate
normal
distribution

An application
Other components Y2 , Y3 , . . . , Yn share a similar property
PCA

Relative to the order of their associated eigenvalue

Second, third, ..., n-th principal component
 The normal Principal components analysis (cont.)
distribution

UFC/DC
ATAII (CK0146)
PR (TIP8412)
2017.2

The normal
distribution

The multivariate
normal
Theorem
distribution
For j = 2, . . . , n and i = 1, 2, . . . , j − 1,
An application
PCA Var[a′ X] ≤ λj = Var(Yj )

for all vectors a such that a ⊥ vi and ||a|| = 1

Proof

The proof follows the lines of that for the first principal component (⋆)