01 - Notes - Chapter 9

12/16/21
Chapter 9
Multicollinearity
Linear Regression Analysis 6E Montgomery, Peck 1

& Vining
9.1 Introduction
• Multicollinearity is a problem that plagues many regression
models. It impacts the estimates of the individual regression
coefficients.
• Uses of regression:
1. Identifying the relative effects of the regressor variables
2. Prediction and/or estimation, and
3. Selection of an appropriate set of variables for the model.

& Vining
1
12/16/21
9.1 Introduction
• If all regressors are orthogonal, then multicollinearity is not a
problem. This is a rare situation in regression analysis.
• More often than not, there are near-linear dependencies among the
regressors such that p
åt jX j = 0
j =1
is approximately true. If this sum holds exactly for a subset of

regressors, then (X’X)-1 does not exist.

& Vining
9.2 Sources of Multicollinearity
Four primary sources:

1. The data collection method employed
2. Constraints on the model or in the population
3. Model specification
4. An overdefined model

& Vining
2
12/16/21

Data collection method employed
- Occurs when only a subsample of the entire sample space has
been selected. (Soft drink delivery: number of cases and distance
tend to be correlated. That is, we may have data where only a small
number of cases are paired with short distances, large number of
cases paired with longer distances). We may be able to reduce this
multicollinearity through the sampling technique used. There is no
physical reason why you can’t sample in that area.

& Vining

& Vining
3
12/16/21
Constraints on the model or in the population.

(Electricity consumption: two variables x1 – family income
and x2 – house size). Physical constraints are present,
multicollinearity will exist regardless of collection method.

& Vining
Model Specification
Polynomial terms can cause ill-conditioning in the X’X
matrix. This is especially true if range on a regressor variable,
x, is small.

& Vining
4
12/16/21

Overdefined model
More regressor variables than observations. The best way to
counter this is to remove regressor variables.
- Recommendations:
1) Redefine the model using smaller set of regressors;
2) Do preliminary studies using subsets of regressors; or
3) Use principal components type regression methods to
remove regressors.

& Vining
9.3 Effects of Multicollinearity

Strong multicollinearity can result in large variances and covariances
for the least squares estimates of the coefficients. Recall from chapter
3, C = (X’X)-1 and
1
C jj =
1 - R 2j
Strong multicollinearity between xj and any other regressor variable
will cause Rj2 to be large, and thus Cjj to be large.
In other words, the variance of the least squares estimate of the
coefficient will be very large.

& Vining
10
5
12/16/21
9.3 Effects of Multicollinearity

Strong multicollinearity can also produce least-squares
estimates of the coefficients that are too large in absolute
value. The squared distance between the least squares
estimate and the true parameter is denoted
L12 = (bˆ - b)' (bˆ - b)
( )
E L2 = E (bˆ - b)' (bˆ - b)
1
= s 2Tr ( X' X) -1

& Vining
11

& Vining
12
6
12/16/21

& Vining
13

& Vining
14
7
12/16/21
9.4 Multicollinearity Diagnostics
• Ideal characteristics of a multicollinearity diagnostic:

1. We want the procedure to correctly indicate if
multicollinearity is present; and,
2. We want the procedure to provide some insight as to which
regressors are causing the problem.

& Vining
15
9.4.1 Examination of the Correlation Matrix

• If we scale and center the regressors in the X’X matrix, we have the
correlation matrix. The pairwise correlation between two variables xi and xj
is denoted rij. The off diagonal elements of the centered and scaled X’X
matrix (X’X matrix in correlation form) are the pairwise correlations.
• If |rij| is close to unity, then there may be an indication of multicollinearity.
But, the opposite does not always hold. That is, there may be instances when
multicollinearity is present, but the pairwise correlations do not indicate a
problem. This can happen when using pairwise correlations in a problem
with more than two variables involved.

& Vining
16
8
12/16/21

& Vining
17
The correlation matrix fails to identify the multicollinearity problem

in the Mason, Gunst & Webster data in Table 9.4, page 304.

& Vining
18
9
12/16/21
9.4.2 Variance Inflation Factors

• As discussed in Chapter 3, variance inflation factors are very
useful in determining if multicollinearity is present.
VIF j = C jj = (1 - R 2j ) -1
• VIFs > 5 to 10 are considered significant. The regressors that
have high VIFs probably have poorly estimated regression
coefficients.

& Vining
19

& Vining
20
10
12/16/21

VIFs: A Second Look and Interpretation
• The length of the normal-theory confidence interval on
the jth regression coefficient can be written as
L j = 2(C jj sˆ 2 )1 / 2 t a / 2,n- p -1

& Vining
21

• The length of the corresponding normal-theory confidence
interval based on a design with orthogonal regressors (with
same sample size, same root-mean square (rms) values) is
L* = 2sˆ ta / 2,n- p -1

& Vining
22
11
12/16/21

• Take the ratio of these two: Lj/L* = C 1jj/ 2. That is, the square
root of the jth VIF gives us a measure of how much longer the
confidence interval for the jth regression coefficient is
because of multicollinearity.
• For example, say VIF3 = 10. Then VIF3 @ 3.3 . This tells us that
that the confidence interval is 3.3 times longer than if the
regressors had been orthogonal (the best case scenario).

& Vining
23
9.4.3 Eigensystem Analysis of X’X

• The eigenvalues of X’X (denoted l1, l2, …, lp) can be used to
measure multicollinearity. Small eigenvalues are indications of
multicollinearity.
l max
The condition number of X’X is k=
l min
• This number measures the spread in the eigenvalues.
k < 100, no serious problem
100 < k < 1000, moderate to strong multicollinearity
k > 1000, strong multicollinearity.
& Vining
24
12
12/16/21
9.4.3 Eigensystem Analysis of X’X

• A large condition number indicates multicollinearity exists. It does
not tell us how many regressors are involved.
The condition indices of X’X are
l max
kj =
lj
• The number of condition indices that are large (greater than 1000)
provide a measure of the number of near linear dependencies in X’X.
• In SAS, PROC REG, in the model statement of your program, you can
use the option COLLIN; this will produce out eigenvalues, condition
indices, etc.
& Vining
25

& Vining
26
13
12/16/21

& Vining
27

& Vining
28
14
12/16/21

& Vining
29

& Vining
30
15
12/16/21

& Vining
31
9.5 Methods for Dealing with Multicollinearity
• Collect more data

• Respecify the model
• Ridge Regression and related techniques (PC regression,
LASSO, etc)

& Vining
32
16
12/16/21
• Least squares estimation gives an unbiased estimate,
E (bˆ ) = b
with minimum variance – but this variance may still be very
large, resulting in unstable estimates of the coefficients.
– Alternative: Find an estimate that is biased but with smaller variance than
the unbiased estimator.

& Vining
33
Ridge Estimator b̂ R
bˆ R = ( X' X + kI ) -1 X' y
= ( X' X + kI ) -1 X' Xβˆ
= Z βˆ k
k is a “biasing parameter” usually between 0 and 1.

& Vining
34
17
12/16/21

The effect of k on the MSE
Recall: MSE (bˆ * ) = Var (bˆ * ) + (bias ) 2
Now, MSE (bˆ *R ) = Var (bˆ *R ) + (bias ) 2
lj
= s2 å + k 2β' ( X' X + kI ) - 2 β
(l j + k ) 2
As k , Var ¯, and bias

Choose k such that the reduction in variance > increase in bias.
SS Re s = ( y - xbˆ R )' ( y - xbˆ R )

& Vining
35
• Ridge Trace
- Plots k against the coefficient estimates. If multicollinearity is
severe, the ridge trace will show it. Choose k such that b̂ R is
stable and hope the MSE is acceptable
- Ridge regression is a good alternative if the model user wants to

have all regressors in the model.

& Vining
36
18
12/16/21

& Vining
37

& Vining
38
19
12/16/21
More About Ridge Regression

• Methods for choosing k
• Relationship to other estimators
• Ridge regression and variable selection
• Generalized ridge regression (a procedure with a biasing
parameter k for each regressor

& Vining
39
Generalized Regression Techniques

& Vining
40
20
12/16/21

& Vining
41

& Vining
42
21
12/16/21

& Vining
43

& Vining
44
22
12/16/21

& Vining
45

& Vining
46
23
12/16/21

& Vining
47

& Vining
48
24
12/16/21

& Vining
49

& Vining
50
25
12/16/21

& Vining
51

& Vining
52
26
12/16/21

& Vining
53

& Vining
54
27
12/16/21

& Vining
55
9.5.4 Principal-Component Regression

& Vining
56
28
12/16/21

& Vining
57

& Vining
58
29
12/16/21
The eigenvalues suggest that a model based on 4 or 5 of the PCs

would probably be adequate.

& Vining
59

& Vining
60
30
12/16/21
Models D and E are

pretty similar

& Vining
61
31

01 - Notes - Chapter 9

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

01 - Notes - Chapter 9

Uploaded by

Copyright:

Available Formats

12/16/21

Linear Regression Analysis 6E Montgomery, Peck 1

Linear Regression Analysis 6E Montgomery, Peck 2

is approximately true. If this sum holds exactly for a subset of

Linear Regression Analysis 6E Montgomery, Peck 3

9.2 Sources of Multicollinearity

Four primary sources:

Linear Regression Analysis 6E Montgomery, Peck 4

9.2 Sources of Multicollinearity

Linear Regression Analysis 6E Montgomery, Peck 5

9.2 Sources of Multicollinearity

Linear Regression Analysis 6E Montgomery, Peck 6

9.2 Sources of Multicollinearity

Constraints on the model or in the population.

Linear Regression Analysis 6E Montgomery, Peck 7

9.2 Sources of Multicollinearity

Linear Regression Analysis 6E Montgomery, Peck 8

9.2 Sources of Multicollinearity

Linear Regression Analysis 6E Montgomery, Peck 9

9.3 Effects of Multicollinearity

Linear Regression Analysis 6E Montgomery, Peck 10

9.3 Effects of Multicollinearity

Linear Regression Analysis 6E Montgomery, Peck 11

Linear Regression Analysis 6E Montgomery, Peck 12

Linear Regression Analysis 6E Montgomery, Peck 13

Linear Regression Analysis 6E Montgomery, Peck 14

9.4 Multicollinearity Diagnostics

• Ideal characteristics of a multicollinearity diagnostic:

Linear Regression Analysis 6E Montgomery, Peck 15

9.4.1 Examination of the Correlation Matrix

Linear Regression Analysis 6E Montgomery, Peck 16

Linear Regression Analysis 6E Montgomery, Peck 17

The correlation matrix fails to identify the multicollinearity problem

Linear Regression Analysis 6E Montgomery, Peck 18

9.4.2 Variance Inflation Factors

Linear Regression Analysis 6E Montgomery, Peck 19

Linear Regression Analysis 6E Montgomery, Peck 20

9.4.2 Variance Inflation Factors

Linear Regression Analysis 6E Montgomery, Peck 21

9.4.2 Variance Inflation Factors

Linear Regression Analysis 6E Montgomery, Peck 22

9.4.2 Variance Inflation Factors

Linear Regression Analysis 6E Montgomery, Peck 23

9.4.3 Eigensystem Analysis of X’X

9.4.3 Eigensystem Analysis of X’X

Linear Regression Analysis 6E Montgomery, Peck 26

Linear Regression Analysis 6E Montgomery, Peck 27

Linear Regression Analysis 6E Montgomery, Peck 28

Linear Regression Analysis 6E Montgomery, Peck 29

Linear Regression Analysis 6E Montgomery, Peck 30

Linear Regression Analysis 6E Montgomery, Peck 31

9.5 Methods for Dealing with Multicollinearity

• Collect more data

Linear Regression Analysis 6E Montgomery, Peck 32

9.5 Methods for Dealing with Multicollinearity

• Least squares estimation gives an unbiased estimate,

Linear Regression Analysis 6E Montgomery, Peck 33

9.5 Methods for Dealing with Multicollinearity

k is a “biasing parameter” usually between 0 and 1.

Linear Regression Analysis 6E Montgomery, Peck 34

9.5 Methods for Dealing with Multicollinearity

As k ­, Var ¯, and bias ­

Linear Regression Analysis 6E Montgomery, Peck 35

As k , Var ¯, and bias