Multivariate Statistics

Statistics 2012: Module 5

1
 Canonical Correlation Analysis
• Investigate the relationships between two groups of vari-
ables x1 , . . . , xp and y 1 , . . . , y q

• Replace the original sets of variables by sets of uncorre-

lated variables u1 , . . . , up and v 1 , . . . , v q

• The new variables are linear combinations of original

variables from one group

• The new variables are obtained in order of importance

• The new variables are called the canonical variates
2
 Canonical variates
• uj = aj1 x1 + · · · + ajp xp
v j = bj1 y 1 + · · · + bjq y q
• The new variables should reflect the relations
between the two groups

• The between group relations result in correlations

• Between group correlations are difficult to
interpret due to within group correlations

3
 The first canonical variates
u1 and v 1 are linear combinations such that
• var(u1 ) = var(v 1 ) = 1
• The first canonical variates maximize the corre-
lation cor(u1 , v 1 ) among all linear combinations
that satisfy the above condition.

4
 The second canonical variates
u2 and v 2 are linear combinations such that
• var(u2 ) = var(v 2 ) = 1
• cor(u2 , u1 ) = cor(v 2 , v 1 ) = 0
• cor(u2 , v 1 ) = cor(v 2 , u1 ) = 0
• The second canonical variates maximize the cor-
relation cor(u2 , v 2 ) among all linear combina-
tions that satisfy the above conditions.

5
 Canonical variates in general
um and v m (m ≤ min(p, q)) are linear combina-
tions such that

• var(um ) = var(v m ) = 1
• cor(um , ur ) = cor(v m , v r ) = 0 for all r < m
• cor(um , v r ) = cor(v m , ur ) = 0 for all r < m
• The mth canonical variates maximize the correla-
tion cor(um , v m ) among all linear combinations
that satisfy the above conditions.

6
 Correlation matrices
Consider the following correlation matrices:

• Rxx with elements cor(xk , xl )

• Ryy with elements cor(y k , y l )

• Rxy with elements cor(xk , y l )

• Ryx = (Rxy )t

7
 Finding canonical variates
The canonical variates are given by

um = em1 x1 + em2 x2 + · · · + emp xp = Xem

v m = fm1 y 1 + fm2 y 2 + · · · + fmq y q = Y f m
−1 −1
where em are the eigenvectors of the matrix Rxx Rxy Ryy Ryx
and f m are the eigenvectors of the matrix R−1
yy R −1
yx xx Rxy
R

8
 The canonical correlations
−1
• Both Rxx Rxy R−1yy R yx and R −1
yy R −1
yx xx Rxy
R
have s = min(p, q) nonzero eigenvalues
λ1 ≥ · · · ≥ λs .
√
• cor(um , v m ) = λm for 1 ≤ m ≤ s

9
 Example: Head size
This data set contains meaurements for the heads of
the first two adult sons in 25 families.

x1 : Head length of first son

x2 : Head breadth of first son
x3 : Head length of second son
x4 : Head breadth of second son

Examine the relation between measurements of pairs

of sons

10
 Example: Head size
The first canonical variates are

u1 = 0.73x1 + 0.69x2
v 1 = 0.68x3 + 0.73x4

and their correlation is

cor(u1 , v 1 ) = 0.79

11
Example: first canonical variates
Correlation between first canonical variates

2
1
0
v1

−1
−2
−3

−3 −2 −1 0 1 2

u1

12
 Example: Head size
The second canonical variates are

u2 = 0.70x1 − 0.71x2
v 2 = 0.71x3 − 0.71x4

and their correlation is

cor(u1 , v 1 ) = 0.05

13
Example: second canonical variates
Correlation between second canonical variates

1.0
0.5
v2

0.0
−0.5

−1.0 −0.5 0.0 0.5 1.0

u2

14
 Interpretation of canonical variates
Requires examining and interpreting the coefficients
em and f m (loadings)

−→ Affected by differences in variances and within

correlations

−→ Standardize the variables

−→ Subjective: do not overstate claims!

15
 Example: Head size
• First canonical variates ≈ average of both head
sizes

→ Girth of the head

• Second canonical variates ≈ difference between

length and breadth

→ Shape of the head

16