Multivariate Statistics

Statistics 2012: Module 5

1
 Exploratory Factor Analysis
• Developed for problems that involve quantities that are
not directly measurable
→ E.g. social class, intelligence, . . .
→ These quantities are called latent variables
• Examine such quantities through measurable indicators
that are related to the quantity of interest
→ E.g. Educational background, occupation, income,
test scores, . . .
→ These indicators are called manifest variables
2
 Exploratory Factor Analysis
• Uncover the relationship between the latent and
manifest variables

• Retrieve the latent variables

• dimension reduction

3
 Factor Analysis Model
• The manifest variables are centered:
xj := xj − x̄j

• The covariances/correlations between the

manifest variables are the result of their relations
with the latent variables

• Consider a model with k factors f 1 , . . . , f k

where k < p

4
 Factor Analysis Model
A regression model of the following form is assumed

xi1 = λ11 fi1 + λ12 fi2 + · · · + λ1k fik + ui1

xi2 = λ21 fi1 + λ22 fi2 + · · · + λ2k fik + ui2
..
.

xip = λp1 fi1 + λp2 fi2 + · · · + λpk fik + uip

5
 Factor Analysis Model

or shortly xi = Λfi + ui

 
λ11 λ12 ... λ1k
 
λ21 λ22 ... λ2k 
 
where Λ =  . .. 
 .. .. 
 . . 

λp1 λp2 ... λpk

6
 Factor loadings
• λql expresses how manifest variable xq
depends on common factor f l

• λql are called the factor loadings

• Larger values relate the factor more to the
manifest variable

• Useful to infer a meaningful

interpretation/description of the factors

7
 Factor Analysis Model: assumptions
• The specific variates u1 . . . , uk are uncorrelated with
each other and with the factors f 1 . . . , f k .
→ Correlations between manifest variables arise from
their relationship with the factors

• The factors have mean zero and standard deviation 1,

and are uncorrelated.
→ λql is the covariance/correlation between xq and f l

8
 Variance decomposition
From the assumptions it follows that

∑
k
var(xj ) = λ2jl + ψj = h2j + ψj
l=1

where ψj
= var(uj ).
∑k
• hj = l=1 λ2jl is the communality of xj , that is, the
2

part of the variance related to the common factors

• ψj is the specific or unique variance of xj , that is, the

part of the variance urrelated to the common factors
9
 Covariances
The covariances equal

∑
k
cov(xj , xq ) = λjl λql
l=1

• Covariances do not dependent on the specific

variates u1 , . . . , up

• The common factors account completely for the

relations between the manifest variables

10
 Variance-covariance matrix
decomposition
The factor analysis model assumes that the
variance-covariance matrix S can be decomposed
as follows:

S = ΛΛt + ψ

where ψ = diag(ψ1 , . . . , ψp )

11
 Finding FA solutions
• Several estimation procedures have been
proposed

• The most popular ones are based on

→ Maximum likelihood, assuming normally
distributed variables

→ Principal components analysis

12
 Example: Examination marks
Children’s examination marks in the following
subjects are considered

x1 Classics
x2 French
x3 English

13
 Example: Examination marks
The data produced the following correlation matrix
x1 x2 x3
Classics 1.00 0.83 0.78

French 0.83 1.00 0.67

English 0.78 0.67 1.00

14
 Example: Factor model
We assume a single factor model

xi1 = λ1 fi + ui1
xi2 = λ2 fi + ui2
xi3 = λ3 fi + ui3

The factor f reflects ’intellectual ability’

15
 Example: Solution
The solution is

λ̂1 = 0.98, λ̂2 = 0.84, λ̂3 = 0.79

ψ̂1 = 0.03, ψ̂2 = 0.29, ψ̂2 = 0.37

−→ Factor is mostly related to classics

16
 Choosing the number of factors
• Solution completely differs if number of factors is
changed

• Too few factors: too many high loadings

• Too many factors: factors difficult to interpret

17
 Choosing the number of factors
• Sometimes suggested by the problem and data

• Use PCA rules (eigenvalues, screeplot)

• Use a formal test procedure (e.g. MLE)

18
 Example: Life expectancy
Life expectancy in years for 31 countries and for 8
categories of people

M0 Males at age 0
M25 Males at age 25
M50 Males at age 50
M75 Males at age 75
W0 Females at age 0
W25 Females at age 25
W50 Females at age 50
W75 Females at age 75

19
 Example: Life expectancy
We use the MLE test to determine the number of factors:

Test of the hypothesis that 1 factor is sufficient. The

chi square statistic is 163.11 on 20 degrees of free-
dom. The p-value is 1.88e-24
−→ 1 factor is not sufficient

20
 Example: Life expectancy

Test of the hypothesis that 2 factors are sufficient.

The chi square statistic is 45.24 on 13 degrees of
freedom. The p-value is 1.91e-05

−→ 2 factors is not sufficient

Test of the hypothesis that 3 factors are sufficient.
The chi square statistic is 6.73 on 7 degrees of free-
dom. The p-value is 0.458

−→ 3 factors is sufficient
21
 Factor rotation
• The factors are not uniquely defined!
−1
• Consider an orthogonal matrix M (M t
= M ), then
t
xi = Λfi + ui = (ΛM )(M fi ) + ui
t t
and S = (ΛM )(ΛM ) + ψ = ΛΛ + ψ

• Put fi∗ = M t fi and Λ∗ = ΛM

→ Both models are equivalent
→ One can select some ”optimal” solution!

22
 Factor rotation
Rotate factors to obtain a simple structure

• Few loadings are large and positive

• Most loadings are close to zero
• Factors are related to disjunct sets of manifest variables
• ....

−→ Many different procedures based on the chosen goal

23
 Orthogonal factor rotation
• Factors remain orthogonal

• Loadings are covariances/correlation

• Most popular methods

→ Varimax
→ Quartimax

24
 Oblique factor rotation
• Give up on orthogonality of factors
• A simpler structure of loadings is possible
• Sometimes correlated factors are expected
• Loadings are not covariances/correlation
anymore!

• Most popular methods

→ Promax
→ Oblimin

25
 Example: Life expectancy
No rotation Varimax

Factor1 Factor2 Factor3 Factor1 Factor2 Factor3

m0 0.98 -0.15 0.09 0.96 0.12 0.23

m25 0.75 0.10 0.26 0.65 0.17 0.44

m50 0.68 0.49 0.48 0.43 0.35 0.79

m75 0.38 0.70 0.29 0.08 0.52 0.66

w0 0.98 -0.14 -0.08 0.97 0.22 0.08

w25 0.94 0.31 -0.05 0.76 0.56 0.31

w50 0.80 0.58 -0.05 0.54 0.73 0.40

w75 0.46 0.77 -0.21 0.16 0.87 0.28

26
 Example: Life Expectancy
Factor interpretation

• Factor 1: Expectancy at young age

• Factor 2: Expectancy for women at older age

• Factor 3: Expectancy for men at older age

27
 Example: Life expectancy
Varimax Promax

Factor1 Factor2 Factor3 Factor1 Factor2 Factor3

m0 0.96 0.12 0.23 1.02 -0.14 0.07

m25 0.65 0.17 0.44 0.53 -0.13 0.44

m50 0.43 0.35 0.79 0.08 -0.03 0.94

m75 0.08 0.52 0.66 -0.30 0.33 0.73

w0 0.97 0.22 0.08 1.07 0.07 -0.19

w25 0.76 0.56 0.31 0.67 0.44 0.04

w50 0.54 0.73 0.40 0.32 0.66 0.15

w75 0.16 0.87 0.28 -0.09 0.98 -0.02

28
 Finding factor scores
Under the assumption of normality, one can obtain
factor scores as

ˆ −1
fi = Λ̂S xi

29
 Example: Life expectancy

Algeria

Dominican Rep
EcuadorNicaragua

2
Panama
Costa Rica Tunisia
El Salvador
Guatemala
1

Columbia
Mexico
Argentina
Grenada
Jamaica
Chile Canada
Madagascar Seychelles
Factor3

Factor2
Honduras
Trinidad(62)United States (W66)
(66)
Greenland United States (67) 4
0

South Africa(C)
United States (NW66)
Trinidad (67)
MauritiusSouth Africa(W) 3
Reunion 2
−1

Cameroon
0
−1
−2

−2
−3 −2 −1 0 1 2

Factor1

30
 FA vs PCA
PCA and FA both focus on explaining a data
structure through a small number of dimensions

• FA focuses on covariances/correlation while PCA

focuses on variances

• PCs remain the same if the number of

components k is changed while all factors in FA
change when the number of factors is changed

31
 FA vs PCA
• PCA based on covariance or correlation matrix is
completely different while FA based on
covariance and correlation matrix are essentially
equivalent

32
Confirmatory factor analysis
• Used to check some existing theory
• Incorporate constraints on loadings into
model

• Requires specialized software (Structural

Equation Modeling)

33
 Final example: Drug use
Data about the use of psychoactive substances
among school children (7th-11th grade) in 11 schools
in Los Angeles metropolitan area. The number of
times a particular substance had been used is
recorded on a 5-point scale:

1 : Never tried
2 : Only once
3 : A few times
4 : Many times
5 : Regularly

34
 Example: Drug use
Questions about the following substances were
asked
x1 : cigarettes
x2 : beer
x3 : wine
x4 : liquor
x5 : cocaine
x6 : tranquillizers
x7 : drug store medicine
x8 : heroin/opiates
x9 : marijuana
x10 : hashish
x11 : inhalants (glue, gasoline,. . . )
x12 : hallucinogenics (LSD, . . . )
x13 : amphetamine, stimulants

35
 Example: Drug use
Factor1 Factor2 Factor3 Factor4

cigarettes 0.50 0.11 0.34 0.08

beer 0.78 0.07 0.14 0.04

wine 0.78 0.06 0.05 0.04

liquor 0.72 0.15 0.19 0.10

cocaine 0.02 0.57 0.09 0.02

tranquillizers 0.12 0.54 0.15 0.31

drug store medication 0.09 0.38 0.04 0.05

heroin 0.02 0.55 0.04 0.08

marijuana 0.40 0.19 0.83 0.13

hashish 0.27 0.40 0.39 0.22

inhalants 0.19 0.47 0.14 0.16

hallucinogenics 0.03 0.51 0.07 0.32

amphetamine 0.15 0.43 0.17 0.86

36
 Example: Drug use
Factor interpretation

• Factor 1: Legal drug usage (cigarettes, beer,

wine, liquor)

• Factor 2: Hard drug usage (cocaine, heroine,

tranquilizers,. . . )

• Factor 3: Marijuana (smoking drugs)

• Factor 4: Amphetamine (illegal pills)

37