DoE Regression Models 3jan19 v20

Regression & Residual Analysis
Design of Experiments
1 © 2018 – Confidential
Regression Models
Agenda
1 Introduction
2 Linear Regression Models
3 Parameters Estimation
4 Examples
5 Fitting Regression Models in Designed Experiments
6 Other Topics of Interest
Introduction
o In many problems two or more variables are related, and it is of interest to model and
explore this relationship
o Example: in a chemical process the yield of product is related to the operating temperature
o The chemical engineer may want to build a model relating yield to temperature and then use
the model for:
o prediction
o process optimization
o process control
o In general, suppose that there is a single dependent variable or response y that depends on k
independent or regressor variables, for example, x1, x2, . . . , xk. The relationship between
these variables is characterized by a mathematical model called a regression model
o Regression methods are frequently used to analyze data from unplanned experiments, such
as might arise from observation of uncontrolled phenomena or historical records
o Regression methods are also very useful in designed experiments where something has “gone
wrong”
Agenda
1 Introduction
4 Examples
Linear Regression Models
o suppose that we wish to develop an empirical model relating the viscosity of a polymer to the
temperature and the catalyst feed rate. A model that might describe this relationship is:
Y: Viscosity
X1: Temperature
X2: Catalyst feed rate
o This is a multiple linear regression model with two independent variables

o We often call the independent variables predictor variables or regressors
o The term linear is used because the equation is a linear function of the unknown parameters
β0, β1, and β2
Multiple Linear Regression
o In general, the response variable y may be related to k regressor variables.
o The model is called a multiple linear regression model with k regressor variables.
o The parameters βj, j = 0, 1, . . . , k, are called the regression coefficients.
o The parameter βj represents the expected change in response y per unit change in x j when all
the remaining independent variables x i (i ≠ j) are held constant.
Approximation to Linear Regression
o Models that are more complex in appearance may often still be analyzed by
multiple linear regression techniques
o For example, consider adding an interaction term to the first-order model in two
variables, say:
o If we let x3 = x1x2 and β3 = β12, then above equation can be written as:
Another Example
o Consider the second-order response surface model in two variables:
o If we let x3 = x12, x4 = x22, x5 = x1x2, β3 = β11, β4 = β22, and β5 = β12, then this becomes:
Agenda
1 Introduction
4 Examples
Least Squares
o Least Squares is typically used to estimate the regression coefficients in a multiple
linear regression model.
o The method of least squares chooses the β’s in the equation above so that the sum
of the squares of the errors, ϵi, is minimized
Least Squares (Cont’d)
o The method of least squares chooses the β’s in the equation so that the sum of the
squares of the errors, ϵi, is minimized
o The function L is to be minimized with respect to β0, β1, . . . , βk. The least squares
estimators, must satisfy:
Least Squares (Cont’d)
o Simplifying, we get the least squares normal equations
o There are p = k+1 normal equations.

o The solution to the normal equations will be the least squares estimators of the
regression coefficients β̂0, β̂1,..., β̂k.
Matrix Notation
o In matrix notation, the model maybe written as:
Where:
o y is (n x 1) vector
o X is (n x p) matrix of levels of independent variables
o β is (p x 1) vector
o ϵ is (n x 1) vector
Matrix Notation (Cont’d)
o We wish to find the vector of least squares estimators, that minimizes
o L may be expressed as:
o The least squares estimators must satisfy:
Simplifying to:
Hence:
Matrix Notation (Cont’d)
o It is easy to see that the matrix form of the normal equations is identical to the
scalar form
o The fitted regression model is
o In scalar notation
Estimating σ2
o The difference between the actual observation yi and the corresponding fitted
value ŷi is the residual
o It is also usually necessary to estimate σ2. To develop an estimator of this

parameter, consider the sum of squares of the residuals
Estimating σ2 (cont’d)
o The last equation is called the error or the residual sum of squares. It can be
shown that
o An unbiased estimator of σ2
Agenda
1 Introduction
4 Example
Example
o Sixteen observations on the viscosity of a polymer (y) and two process variables—
reaction temperature (x1) and catalyst feed rate (x2)—are shown. We will fit a
multiple linear regression model:
Example (Cont’d)
o The X’X matrix
Example (Cont’d)
o Hence the least squares fit, with the regression coefficients reported to two
decimal places, is
o ŷ = 1566.08 + 7.62x1 + 8.58x2
Example (Cont’d)
Example (Cont’d)
Other residual measures

Out of scope ?
Example (Cont’d)
Normal probability plot of residuals
Example (Cont’d)
Residual plots
Example (Cont’d)
Residual plots
Agenda
1 Introduction
4 Examples
Regression Analysis for a 23 Factorial Design
o A chemical engineer is investigating the yield of a process. Three process variables
are of interest: temperature, pressure, and catalyst concentration. Each variable
can be run at a low and a high level, and the engineer decides to run a 2 3 design
with four center points. The design and the resulting yields are shown below,
where we have shown both the natural levels of the design factor and the +1, -1
coded variable notation normally employed in 2k factorial designs to represent the
factor levels.
o Suppose that the engineer decides to fit a main effects only model, say
Regression Analysis for a 23 Factorial Design (Cont’d)
o The 23 is an orthogonal design, and even with the added center runs it is still
orthogonal. Therefore
o Because the design is orthogonal, the X’X matrix is diagonal, the required inverse is
also diagonal, and the vector of least squares estimates of the regression
coefficients is
o The fitted regression model is
Relationship to Effect Estimates
o As we have made use of on many occasions, the regression coefficients are closely
connected to the effect estimates that would be obtained from the usual analysis
of a 23 design. For example, the effect of temperature is
o Notice the regression coefficient for x1 is
o That is, the regression coefficient is exactly one-half the usual effect estimate. This
will always be true for a 2k design. This example demonstrates that the effect
estimates from a 2k design are least squares estimates.
Relationship to Effect Estimates (Cont’d)
Also Good For…
o A 23 factorial design with a missing observation
o Inaccurate levels in design factors
Agenda
1 Introduction
4 Examples
Other Topics
o Hypothesis testing in multiple regression
o Confidence intervals
o Prediction of new response observations
o Regression model diagnostics (we will partially cover residuals analysis)
Residuals Analysis
Agenda
1 Introduction
2 ϵ Assumptions
3 Examples of Residual Plots
Introduction
o As emphasized in designed experiments, model adequacy checking is an important part of
the data analysis procedure.
o This is equally important in building regression models, the residual plots used with designed
experiments should always be examined for a regression model.
o In general, it is always necessary to:

o Examine the fitted model to ensure that it provides an adequate approximation to the
true system
o Verify that none of the least squares regression assumptions are violated.
o In addition to the residual plots, other model diagnostics are frequently useful in regression.
This is out of the scope of this presentation
o Scaled residuals
o Prediction Error Sum of Squares (PRESS)
Residuals Definition
Residual = Observed value - Predicted value
e = Y - Y’
o Properties
o ē=0
Agenda
1 Introduction
2 ϵ Assumptions
The ϵ term is assumed to be
o Normally distributed
o Independent
o Homoscedastic (the same variance at every X)
If these are true, then the observed residuals should behave in a similar fashion
ei = yi - ŷi
Agenda
1 Introduction
2 ϵ Assumptions
Residual Plots
o A residual plot is a scatter plot
Fitted Values
Residual Plots
o A residual plot is a scatter plot (Controlled design experiment)
Fitted Values
Residual Plots
o Variance increases with X. Hence the assumptions of the model are violated
o Perhaps, try weighted regression
Fitted Values
Residual Plots
o Assumptions of the model are violated
Fitted Values
Residual Plots
o Residuals are affected by time which is not accommodated by the model. Hence model violates
assumptions.
Fitted Values
Residual Plots
o Other types of residual plots. Only a does not violate the assumptions

DoE Regression Models 3jan19 v20

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

DoE Regression Models 3jan19 v20

Uploaded by

Copyright:

Available Formats

Regression & Residual Analysis

2 Linear Regression Models

5 Fitting Regression Models in Designed Experiments

6 Other Topics of Interest

2 Linear Regression Models

5 Fitting Regression Models in Designed Experiments

6 Other Topics of Interest

o This is a multiple linear regression model with two independent variables

o The parameters βj, j = 0, 1, . . . , k, are called the regression coefficients.

2 Linear Regression Models

5 Fitting Regression Models in Designed Experiments

6 Other Topics of Interest

o There are p = k+1 normal equations.

o L may be expressed as:

o The least squares estimators must satisfy:

o The fitted regression model is

o It is also usually necessary to estimate σ2. To develop an estimator of this

2 Linear Regression Models

5 Fitting Regression Models in Designed Experiments

6 Other Topics of Interest

o ŷ = 1566.08 + 7.62x1 + 8.58x2

Other residual measures

Normal probability plot of residuals

2 Linear Regression Models

5 Fitting Regression Models in Designed Experiments

6 Other Topics of Interest

o The fitted regression model is

o Notice the regression coefficient for x1 is

o Inaccurate levels in design factors

2 Linear Regression Models

5 Fitting Regression Models in Designed Experiments

6 Other Topics of Interest

o Prediction of new response observations

o Regression model diagnostics (we will partially cover residuals analysis)

3 Examples of Residual Plots

o In general, it is always necessary to:

3 Examples of Residual Plots

o Homoscedastic (the same variance at every X)

3 Examples of Residual Plots

o Perhaps, try weighted regression

You might also like