Sigma 2

Page 87
Cochran's Theorem provides the basis for the distribution theory. Under
assumption of normality of error term in the Regression model,
SSTO/sigma**2 follows a chi-square distribution with N-1 degrees of

freedom (df).
Suppose SSTO=Q1 + Q2 + . . . + Qs
sigma**2
The Q's are independent chi-square random variables if and only if the
sum of the df of the Q's add to N-1.
Distribution theory: If Q1 has q1 df and Q2 has q2 df and Q1 and Q2 are

independent then Q1/q1 divided by Q2/q2 follows an F distribution with
q1 and q2 df.
Page 88
Chapter 3
Multiple Linear Regression
Linear Regression Analysis 6E Montgomery, Peck 1

& Vining
Page 89
3.1 Multiple Regression Models

• Suppose that the yield in pounds of conversion in a chemical process
depends on temperature and the catalyst concentration. A multiple
regression model that might describe this relationship is
• This is a multiple linear regression model in two variables.

& Vining
Page 90
Figure 3.1 (a) The regression plane for the model E(y)=
50+10x1+7x2 . (b) The contour plot.

& Vining
Page 91
In general, the multiple linear regression model with k regressors is

& Vining
Page 92

& Vining
Page 93
Linear regression models may also contain interaction effects:
If we let x3 = x1x2 and β3 = β12, then the model can be written in

the form

& Vining
Page 94

& Vining
Page 95

& Vining
Page 96
3.2 Estimation of the Model Parameters

3.2.1 Least Squares Estimation of the Regression Coefficients
Notation
n – number of observations available
k – number of regressor variables
y – response or dependent variable
xij – ith observation or level of regressor j.

& Vining
Page 97
3.2.1 Least Squares Estimation of Regression Coefficients

& Vining
Page 98
The sample regression model can be written as

& Vining
Page 99
The least squares function is
The function S must be minimized with respect to the coefficients.

& Vining
Page 100
The least squares estimates of the coefficients must satisfy

& Vining
Page 101

Simplifying, we obtain the least squares normal equations:
The ordinary least squares estimators are the solutions to the normal
equations.
& Vining
Page 102

Matrix notation is typically used:
Let
where

& Vining
Page 103

& Vining
Page 104
These are the least-squares normal equations. The solution is

& Vining
Page 105

& Vining
Page 106
The n residuals can be written in matrix form as
There will be some situations where an alternative form is useful

& Vining
Page 107
Example 3-1. The Delivery

Time Data
The model of interest is

y = β0 + β1x1+ β2x2 + ε

& Vining
Page 108
Example 3-1.
The Delivery
Time Data
Figure 3.4
Scatterplot matrix
for the delivery
time data from
Example 3.1.

& Vining
Page 109
Example 3-1 The Delivery Time Data
Figure 3.5 Three-dimensional scatterplot of the delivery time data

from Example 3.1.
& Vining
Page 110
Example 3-1 The

Delivery Time Data

& Vining
Page 111

& Vining
Page 112

& Vining
Page 113

& Vining
Page 114
MINITAB Output

& Vining
Page 115
3.2.2 Geometrical Interpretation of Least Squares

& Vining
Page 116
3.2.3 Properties of Least-Squares Estimators
• Statistical Properties
E (βˆ ) = β
Cov (βˆ ) = σ 2 (X' X )−1
• Variances/Covariances
Var (βˆ j ) = σ 2 C jj
Cov(βˆ i , βˆ j ) = σ 2 C ij

& Vining
Page 117
3.2.4 Estimation of σ2
• The residual sum of squares can be shown to be:
ˆ
= y'y − β'X'y
SSRes
• The residual mean square for the model with p parameters
is:
SS Re s 2
MS Re s = = σ̂
n− p
& Vining
Page 118
3.2.4 Estimation of σ2
• Recall that the estimator of σ2 is model dependent - that
is, change the form of the model and the estimate of σ2
will invariably change.
– Note that the variance estimate is a function of the errors;
“unexplained noise about the fitted regression line”

& Vining
Page 119
Example 3.2 Delivery Time Data

& Vining
Page 120

& Vining
Page 121
3.2.5 Inadequacy of Scatter Diagrams in Multiple Regression
• Scatter diagrams of the regressor variable(s) against the

response may be of little value in multiple regression.
– These plots can actually be misleading
– If there is an interdependency between two or more regressor
variables, the true relationship between xi and y may be
masked.

& Vining
Page 122
An Illustration of the Inadequacy of Scatter Diagrams in

Multiple Regression

& Vining
Page 123
3.3 Hypothesis Testing in Multiple Linear Regression
Once we have estimated the parameters in the model, we

face two immediate questions:
1. What is the overall adequacy of the model?
2. Which specific regressors seem important?

& Vining
Page 124
3.3 Hypothesis Testing in Multiple Linear Regression
This section considers four cases:
• Test for Significance of Regression (sometimes called the global test

of model adequacy)
• Tests on Individual Regression Coefficients (or groups of
coefficients)
• Special Case of Hypothesis Testing with Orthogonal Columns in X
• Testing the General Linear Hypothesis

& Vining
Page 125
3.3.1 Test for Significance of Regression

• The test for significance is a test to determine if there is a
linear relationship between the response and any of the
regressor variables
• The hypotheses are
H0: β1 = β2 = …= βk = 0
H1: βj ≠ 0 for at least one j

& Vining
Page 126

• As in Chapter 2, the total sum of squares can be partitioned in two parts:
SST = SSR + SSRes
• This leads to an ANOVA procedure with the test (F) statistic
SS R / k MS R
F0 = =
SS Re s /(n − k − 1) MS Re s

& Vining
Page 127

• The standard ANOVA is conducted with

& Vining
Page 128
ANOVA Table:
Reject H0 if

& Vining
Page 129

& Vining
Page 130

& Vining
Page 131
To test H0: β1 = β2 = 0, we calculate the F–statistic:

& Vining
Page 132

& Vining
Page 133

• R2
– R2 is calculated exactly as in simple linear regression
– R2 can be inflated simply by adding more terms to the model (even
insignificant terms)
• Adjusted R2
– Penalizes you for added terms to the model that are not significant
2 SS Re s /(n − p )
Radj = 1−
SST /(n − 1)

& Vining
Page 134
• Adjusted R2 Example
– Say SST = 245.00, n = 15
– Suppose that for a model with three regressors, the SSres = 90.00,
then
2 90 /(15 − 4)
Radj = 1− = 0.53
245 /(15 − 1)
– Now suppose that a fourth regressor has been added, and the SSRes
= 88.00
2 88 /(15 − 5)
Radj = 1− = 0.49
245 /(15 − 1)
& Vining
Page 135
3.3.2 Tests on Individual Regression Coefficients
• Hypothesis test on any single regression coefficient:

H0 : β j = 0
• Test Statistic: H1 : β j ≠ 0
βˆ j βˆ j See Example 3.4,
t0 = =
σˆ 2 C jj se(βˆ j ) pg. 90, text
– Reject H0 if |t0| > t α / 2,n − k −1
– This is a partial or marginal test!

& Vining
Page 136
The Extra Sum of Squares method can also be used to test hypotheses
on individual model parameters or groups of parameters
Full model

& Vining
Page 137

& Vining
Page 138

& Vining
Page 139
3.3.3 Special Case of Orthogonal Columns in X
• If the columns X1 are orthogonal to the columns in X2, the

sum of squares due to β2 that is free of any dependence on
the regressors in X1.

& Vining
Page 140
Example
• Consider a dataset with four regressor variables and a single response.
• Fit the equation with all regressors and find that:
y = - 19.9 + 0.0123x1 + 27.3x2 - 0.0655x3 - 0.196x4
• Looking at the t-tests, suppose that x3 is insignificant. So it is removed. What is

the equation now?
• Generally, it is not
y = - 19.9 + 0.0123x1 + 27.3x2 - 0.196x4

& Vining
Page 141
Example
• The model must be refit with the insignificant regressors left
out of the model.
• The regression equation is
y = - 24.9 + 0.0117x1 + 31.0x2 - 0.217x4
• The refitting must be done since the coefficient estimates for
an individual regressor depend on all of the regressors, xj

& Vining
Page 142
Example
• However, if the columns are orthogonal to each other,

then there is no need to refit.
• Can you think of some situations where we would have

orthogonal columns?

& Vining
Page 143
3.4.1. Confidence Intervals on the Regression Coefficients
A 100(1-α) percent C.I. for the regression coefficient, βj is:
βˆ j − tα / 2,n − p σˆ 2C jj ≤ β j ≤ βˆ j + tα / 2,n − p σˆ 2C jj
Or,
βˆ j − tα / 2,n − p se( βˆ j ) ≤ β j ≤ βˆ j + tα / 2,n − p se( βˆ j )

& Vining
Page 144

& Vining
Page 145
3.4.2. Confidence Interval Estimation of the Mean Response
• 100(1-α) percent CI on the mean response at the point x01, x02, …, x0k is
ˆy0 − tα / 2,n − p σˆ 2 x '0 ( X'X)−1 x0 ≤ E ( y | x0 )

ˆ ˆ
≤ y0 + tα / 2,n − p σ x '0 ( X'X) x0
2 −1
• See Example 3-9 on page 95 and the discussion that follows

& Vining
Page 146

& Vining
Page 147

& Vining
Page 148

& Vining
Page 149
3.4.3. Simultaneous Confidence Intervals on Regression Coefficients
It can be shown that
(βˆ − β) ' X ' X(βˆ − β)

~ Fp ,n − p
pMS Re s
From this result, the joint confidence region for all parameters in β is
(βˆ − β) ' X ' X(βˆ − β)

≤ Fα , p ,n − p
pMS Re s
& Vining
Page 150
3.5 Prediction of New Observations
• A 100(1-α) percent prediction interval for a future

observation is
yˆ 0 − tα / 2,n − p σˆ 2 (1 + x '0 ( X'X) −1 x 0 ) ≤ y0

≤ yˆ 0 + tα / 2,n − p σˆ 2 (1 + x '0 ( X'X) −1 x 0 )

& Vining
Page 151

& Vining
Page 152
3.6 Hidden Extrapolation in Multiple Regression

• In prediction, exercise care about potentially extrapolating beyond the
region containing the original observations.
Figure 3.10 An example of

extrapolation in multiple
regression.

& Vining
Page 153
• We will define the smallest convex set containing all of the

original n data points (xi1, xi2, … xik), i = 1, 2, …, n, as the
regressor variable hull RVH.
• If a point x01, x02, …, x0k lies inside or on the boundary of the

RVH, then prediction or estimation involves interpolation, while
if this point lies outside the RVH, extrapolation is required.

& Vining
Page 154
• Diagonal elements of the matrix H = X(X’X)-1X’ can aid in

determining if hidden extrapolation exists:
• The set of points x (not necessarily data points used to fit the
model) that satisfy
is an ellipsoid enclosing all points inside the RVH.

& Vining
Page 155
Two Interesting Examples, Continued

• 3.6 A Multiple Regression Model for the Patient
Satisfaction Data
– Does adding patient age to the model improve the fit?
– Is there any indication that this model is a better predictor?
• 3.7 Does Pitching and Defense Win Baseball Games?
– Does adding number of errors to the model improve the fit?
– Is there any indication that this model is a better predictor?

& Vining
Page 156
• Let x0 be a point at which prediction or estimation is of interest.

Then
• If h00 > hmax then the point is a point of extrapolation.

& Vining
Page 157
Example 3.13
Consider prediction or
estimation at:

& Vining
Page 158
#9
d
c
a
b
Figure 3.10 Scatterplot of cases and distance for the delivery time data.
& Vining
Page 159
3.10 Standardized Regression Coefficients

• It is often difficult to directly compare regression coefficients due to
possible varying dimensions.
• It may be beneficial to work with dimensionless regression
coefficients.
• Dimensionless regression coefficients are often referred to as
standardized regression coefficients.
• Two common methods of scaling:
1. Unit normal scaling
2. Unit length scaling

& Vining
Page 160
Unit Normal Scaling

The first approach employs unit normal scaling for the
regressors and the response variable. That is,

& Vining
Page 161
Unit Normal Scaling
where

& Vining
Page 162
Unit Normal Scaling

• All of the scaled regressors and the scaled response have sample mean
equal to zero and sample variance equal to 1.
• The model becomes
• The least squares estimator:

& Vining
Page 163

Unit Length Scaling
• In unit length scaling:

& Vining
Page 164

Unit Length Scaling
• Each regressor has mean 0 and length
• The regression model becomes
• The vector of least squares regression coefficients:

& Vining
Page 165
Unit Length Scaling

• In unit length scaling, the W’W matrix is in the form of a correlation
matrix:
where rij is the simple correlation between xi and xj.

& Vining
Page 166

& Vining
Example 3.14
Page 167

& Vining
Example 3.14
Page 168

& Vining
Example 3.14
Page 169

& Vining
Example 3.14
Page 170

& Vining
Page 171
3.11 Multicollinearity
• A serious problem that may dramatically impact the usefulness of a
regression model is multicollinearity, or near-linear dependence
among the regression variables.
• Multicollinearity implies near-linear dependence among the
regressors. The regressors are the columns of the X matrix, so clearly
an exact linear dependence would result in a singular X’X.
• The presence of multicollinearity can dramatically impact the ability
to estimate regression coefficients and other uses of the regression
model.

& Vining
Page 172

& Vining
Page 173
• The main diagonal elements of the inverse of the X’X
matrix in correlation form (W’W)-1 are often called
variance inflation factors VIFs, and they are an
important multicollinearity diagnostic.
• For the soft drink delivery data,

& Vining
Page 174
• The variance inflation factors can also be written as:
where R2j is the coefficient of multiple determination

obtained from regressing xj on the other regressor variables.
• If xj is highly correlated with any other regressor variable,
then R2j will be large.

& Vining
Page 175
3.12 Why Do Regression Coefficients Have The Wrong Sign?
• Regression coefficients may have the wrong sign for the following
reasons:
1. The range of some of the regressors is too small.
2. Important regressors have not been included in the model.
3. Multicollinearity is present.
4. Computational errors have been made.

& Vining
Page 176
3.12 Why Do Regression Coefficients Have The Wrong Sign?

& Vining

Sigma 2

Uploaded by

Document Information

Original Description:

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Sigma 2

Uploaded by

Copyright:

Available Formats

Page 87

assumption of normality of error term in the Regression model,

SSTO/sigma**2 follows a chi-square distribution with N-1 degrees of

Distribution theory: If Q1 has q1 df and Q2 has q2 df and Q1 and Q2 are

Linear Regression Analysis 6E Montgomery, Peck 1

3.1 Multiple Regression Models

• This is a multiple linear regression model in two variables.

Linear Regression Analysis 6E Montgomery, Peck 2

3.1 Multiple Regression Models

Linear Regression Analysis 6E Montgomery, Peck 3

3.1 Multiple Regression Models

In general, the multiple linear regression model with k regressors is

Linear Regression Analysis 6E Montgomery, Peck 4

3.1 Multiple Regression Models

Linear Regression Analysis 6E Montgomery, Peck 5

3.1 Multiple Regression Models

Linear regression models may also contain interaction effects:

If we let x3 = x1x2 and β3 = β12, then the model can be written in

Linear Regression Analysis 6E Montgomery, Peck 6

3.1 Multiple Regression Models

Linear Regression Analysis 6E Montgomery, Peck 7

Linear Regression Analysis 6E Montgomery, Peck 8

3.2 Estimation of the Model Parameters

Linear Regression Analysis 6E Montgomery, Peck 9

3.2.1 Least Squares Estimation of Regression Coefficients

Linear Regression Analysis 6E Montgomery, Peck 10

3.2.1 Least Squares Estimation of the Regression Coefficients

The sample regression model can be written as

Linear Regression Analysis 6E Montgomery, Peck 11

3.2.1 Least Squares Estimation of the Regression Coefficients

The least squares function is

The function S must be minimized with respect to the coefficients.

Linear Regression Analysis 6E Montgomery, Peck 12

3.2.1 Least Squares Estimation of the Regression Coefficients

The least squares estimates of the coefficients must satisfy

Linear Regression Analysis 6E Montgomery, Peck 13

3.2.1 Least Squares Estimation of the Regression Coefficients

3.2.1 Least Squares Estimation of the Regression Coefficients

Linear Regression Analysis 6E Montgomery, Peck 15

3.2.1 Least Squares Estimation of the Regression Coefficients

Linear Regression Analysis 6E Montgomery, Peck 16

3.2.1 Least Squares Estimation of the Regression Coefficients

These are the least-squares normal equations. The solution is

Linear Regression Analysis 6E Montgomery, Peck 17

3.2.1 Least Squares Estimation of the Regression Coefficients

Linear Regression Analysis 6E Montgomery, Peck 18

3.2.1 Least Squares Estimation of the Regression Coefficients

The n residuals can be written in matrix form as

There will be some situations where an alternative form is useful

Linear Regression Analysis 6E Montgomery, Peck 19

Example 3-1. The Delivery

The model of interest is

Linear Regression Analysis 6E Montgomery, Peck 20

Linear Regression Analysis 6E Montgomery, Peck 21

Example 3-1 The Delivery Time Data

Figure 3.5 Three-dimensional scatterplot of the delivery time data

Example 3-1 The

Linear Regression Analysis 6E Montgomery, Peck 23

Example 3-1 The Delivery Time Data

Linear Regression Analysis 6E Montgomery, Peck 24

Example 3-1 The Delivery Time Data

Linear Regression Analysis 6E Montgomery, Peck 25