12 - Multiple Linear Regression

12
Multiple Linear Regression
CHAPTER OUTLINE
12-1 Multiple Linear Regression Model 12-3 Confidence Intervals in Multiple Linear
12-1.1 Introduction Regression
12-1.2 Least squares estimation of the 12-4.1 Use of t-tests
parameters 12-3.2 Confidence interval on the mean response
12-1.3 Matrix approach to multiple linear 12-4 Prediction of New Observations
regression 12-5 Model Adequacy Checking
12-1.4 Properties of the least squares estimators 12-5.1 Residual analysis
12-2 Hypothesis Tests in Multiple Linear 12-5.2 Influential observations
Regression 12-6 Aspects of Multiple Regression
12-2.1 Test for significance of regression Modeling
12-2.2 Tests on individual regression coefficients 12-6.1 Polynomial regression models
& subsets of coefficients 12-6.2 Categorical regressors & indicator
variables
12-6.3 Selection of variables & model building
12-6.4 Multicollinearity
1
Learning Objectives for Chapter 12
After careful study of this chapter, you should be able to do the
following:
1. Use multiple regression techniques to build empirical models to
engineering and scientific data.
2. Understand how the method of least squares extends to fitting multiple
regression models.
3. Assess regression model adequacy.
4. Test hypotheses and construct confidence intervals on the regression
coefficients.
5. Use the regression model to estimate the mean response, and to make
predictions and to construct confidence intervals and prediction
intervals.
6. Build regression models with polynomial terms.
7. Use indicator variables to model categorical regressors.
8. Use stepwise regression and other model building techniques to select
the appropriate set of variables for a regression model.
2
© John Wiley & Sons, Inc. Applied Statistics and Probability for Engineers, by Montgomery and Runger.
12-1: Multiple Linear Regression Models
12-1.1 Introduction
• Many applications of regression analysis involve
situations in which there are more than one
regressor variable.
• A regression model that contains more than one
regressor variable is called a multiple regression
model.
3
12-1.1 Introduction
• For example, suppose that the effective life of a cutting
tool depends on the cutting speed and the tool angle. A
possible multiple regression model could be
where
Y – tool life
x1 – cutting speed
x2 – tool angle
4
12-1.1 Introduction
Figure 12-1 (a) The regression plane for the model E(Y) = 50 + 10x1 + 7x2. (b) The
contour plot
5
12-1.1 Introduction
6
12-1.1 Introduction
Figure 12-2 (a) Three-

dimensional plot of the
regression model E(Y) = 50 +
10x1 + 7x2 + 5x1x2. (b) The
contour plot
7
12-1.1 Introduction
Figure 12-3 (a) Three-

dimensional plot of the
regression model E(Y) = 800 +
10x1 + 7x2 – 8.5x12 – 5x22 +
4x1x2. (b) The contour plot
8
12-1.2 Least Squares Estimation of the Parameters
9

• The least squares function is given by
• The least squares estimates must satisfy
10

• The least squares normal Equations are
• The solution to the normal Equations are the least

squares estimators of the regression coefficients.
11
Example 12-1
12
Example 12-1
13
Figure 12-4 Matrix of scatter plots (from Minitab) for the wire bond pull strength
data in Table 12-2.
14
Example 12-1
15
Example 12-1
16
Example 12-1
17
12-1.3 Matrix Approach to Multiple Linear Regression
Suppose the model relating the regressors to the

response is
In matrix notation this model can be written as
18
where
19
We wish to find the vector of least squares

estimators that minimizes:
The resulting least squares estimate is
20
21
Example 12-2
22
Example 12-2
23
Example 12-2
24
Example 12-2
25
Example 12-2
26
Example 12-2
27
28
Estimating 2
An unbiased estimator of 2 is
29
12-1.4 Properties of the Least Squares Estimators

Unbiased estimators:
Covariance Matrix:
30
12-1.4 Properties of the Least Squares Estimators

Individual variances and covariances:
In general,
31
12-2: Hypothesis Tests in Multiple Linear Regression
12-2.1 Test for Significance of Regression

The appropriate hypotheses are
The test statistic is
32
12-2.1 Test for Significance of Regression
33
Example 12-3
34
Example 12-3
35
Example 12-3
36
Example 12-3
37
R2 and Adjusted R2
The coefficient of multiple determination
• For the wire bond pull strength data, we find that R2 =

SSR/SST = 5990.7712/6105.9447 = 0.9811.
• Thus, the model accounts for about 98% of the
variability in the pull strength response.
38
R2 and Adjusted R2
The adjusted R2 is
• The adjusted R2 statistic penalizes the analyst for

adding terms to the model.
• It can help guard against overfitting (including
regressors that are not really useful)
39
12-2.2 Tests on Individual Regression Coefficients and

Subsets of Coefficients
The hypotheses for testing the significance of any

individual regression coefficient:
40
12-2.2 Tests on Individual Regression Coefficients and

Subsets of Coefficients
The test statistic is
• Reject H0 if |t0| > t/2,n-p.

• This is called a partial or marginal test
41
Example 12-4
42
Example 12-4
43
The general regression significance test or the extra

sum of squares method:
We wish to test the hypotheses:
44
A general form of the model can be written:
where X1 represents the columns of X associated with

1 and X2 represents the columns of X associated with
2
45
For the full model:
If H0 is true, the reduced model is
46
The test statistic is:
Reject H0 if f0 > f,r,n-p

The test in Equation (12-32) is often referred to as a
partial F-test
47
Example 12-6
48
Example 12-6
49
Example 12-6
50
12-3: Confidence Intervals in Multiple Linear Regression
12-3.1 Confidence Intervals on Individual

Regression Coefficients
Definition
51
Example 12-7
52
12-3.2 Confidence Interval on the Mean Response
The mean response at a point x0 is estimated by
The variance of the estimated mean response is
53
12-3.2 Confidence Interval on the Mean Response
Definition
54
Example 12-8
55
Example 12-8
56
12-4: Prediction of New Observations
A point estimate of the future observation Y0 is
A 100(1-)% prediction interval for this future

observation is
57
Figure 12-5 An example of

extrapolation in multiple regression
58
Example 12-9
59
12-5: Model Adequacy Checking
12-5.1 Residual Analysis
Example 12-10
Figure 12-6 Normal probability plot of

residuals
60
Example 12-10
61
Example 12-10
Figure 12-7 Plot of residuals
62
Example 12-10
Figure 12-8 Plot of residuals against x1.

63
Example 12-10
Figure 12-9 Plot of residuals against x2.
64
65
The variance of the ith residual is
66
67
12-5.2 Influential Observations
Figure 12-10 A point that is

remote in x-space.
68
12-5.2 Influential Observations
Cook’s distance measure
69
Example 12-11
70
Example 12-11
71
12-6: Aspects of Multiple Regression Modeling
12-6.1 Polynomial Regression Models
72
Example 12-12
73
Example 12-11
Figure 12-11 Data for Example

12-11.
74
Example 12-12
75
Example 12-12
76
12-6.2 Categorical Regressors and Indicator Variables
• Many problems may involve qualitative or

categorical variables.
• The usual method for the different levels of a
qualitative variable is to use indicator variables.
• For example, to introduce the effect of two different
operators into a regression model, we could define an
indicator variable as follows:
77
Example 12-13
78
Example 12-13
79
Example 12-13
80
Example 12-12
81
Example 12-13
82
Example 12-13
83
12-6.3 Selection of Variables and Model Building
84

All Possible Regressions – Example 12-14
85

86

Figure 12-12 A matrix of

Scatter plots from Minitab
for the Wine Quality Data.
87
88
12-6.3: Selection of Variables and Model Building - Stepwise Regression
Example 12-14
89
12-6.3: Selection of Variables and Model Building - Backward Regression
Example 12-14
90
Variance Inflation Factor (VIF)
91
The presence of multicollinearity can be detected in several
ways. Two of the more easily understood of these are:
92
Important Terms & Concepts of Chapter 12
All possible regressions Model parameters & their
Analysis of variance test in multiple interpretation in multiple regression
regression Multicollinearity
Categorical variables Multiple regression
Confidence intervals on the mean Outliers
response Polynomial regression model
Cp statistic Prediction interval on a future
Extra sum of squares method observation
Hidden extrapolation PRESS statistic
Indicator variables Residual analysis & model adequacy
Inference (test & intervals) on individual checking
model parameters Significance of regression
Influential observations Stepwise regression & related methods
Variance Inflation Factor (VIF)
93

12 - Multiple Linear Regression

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

12 - Multiple Linear Regression

Uploaded by

Copyright:

Available Formats

12

Multiple Linear Regression

Figure 12-2 (a) Three-

Figure 12-3 (a) Three-

12-1.2 Least Squares Estimation of the Parameters

12-1.2 Least Squares Estimation of the Parameters

• The least squares estimates must satisfy

12-1.2 Least Squares Estimation of the Parameters

• The solution to the normal Equations are the least

12-1.3 Matrix Approach to Multiple Linear Regression

Suppose the model relating the regressors to the

In matrix notation this model can be written as

12-1.3 Matrix Approach to Multiple Linear Regression

12-1.3 Matrix Approach to Multiple Linear Regression

We wish to find the vector of least squares

The resulting least squares estimate is

12-1.3 Matrix Approach to Multiple Linear Regression

12-1.4 Properties of the Least Squares Estimators

12-1.4 Properties of the Least Squares Estimators

12-2.1 Test for Significance of Regression

The test statistic is

12-2.1 Test for Significance of Regression

• For the wire bond pull strength data, we find that R2 =

• The adjusted R2 statistic penalizes the analyst for

12-2.2 Tests on Individual Regression Coefficients and

The hypotheses for testing the significance of any

12-2.2 Tests on Individual Regression Coefficients and

• Reject H0 if |t0| > t/2,n-p.

The general regression significance test or the extra

We wish to test the hypotheses:

A general form of the model can be written:

where X1 represents the columns of X associated with

For the full model:

If H0 is true, the reduced model is

The test statistic is:

Reject H0 if f0 > f,r,n-p

12-3.1 Confidence Intervals on Individual

12-3.2 Confidence Interval on the Mean Response

The mean response at a point x0 is estimated by

The variance of the estimated mean response is

12-3.2 Confidence Interval on the Mean Response

A point estimate of the future observation Y0 is

A 100(1-)% prediction interval for this future

Figure 12-5 An example of

Figure 12-6 Normal probability plot of

Figure 12-7 Plot of residuals

Figure 12-8 Plot of residuals against x1.

12-5.1 Residual Analysis

12-5.1 Residual Analysis

The variance of the ith residual is

12-5.1 Residual Analysis

12-5.2 Influential Observations

Figure 12-10 A point that is

12-5.2 Influential Observations

Cook’s distance measure

12-6.1 Polynomial Regression Models

Figure 12-11 Data for Example

12-6.2 Categorical Regressors and Indicator Variables

• Many problems may involve qualitative or

12-6.3 Selection of Variables and Model Building

12-6.3 Selection of Variables and Model Building

12-6.3 Selection of Variables and Model Building

12-6.3 Selection of Variables and Model Building