Attribution Non-Commercial (BY-NC)

1.7K views

Attribution Non-Commercial (BY-NC)

- Heteroskedasticity
- Scott and Watson Chpt 4 Solutions
- MOD1_05
- Correlations
- 3384-9919-1-PB (1).pdf
- 308.full
- Multivariate Linear Regression
- MPRA Paper 41508
- Comp Assignment d
- STOCHASTIC FRONTIER.pdf
- Summary Output Ms
- ETC1010_Final_S22015
- Regression Model for Analyzing the Dgs Structures Propagation Characterisitcs
- Smith 1940 Specific Gravity Latex
- Group 01 AoD_Assignment_Gr. 1
- Linear Regression
- Economics project on forecasting.
- Impact of Television Advertisement on Consumer Buying Behavior the Moderating Role of Religiosity in the Context of Pakistan
- amikacina2
- ACCG200_SolutionCh_03.pdf

You are on page 1of 10

11, page 1

Math 445 Chapter 11 Model Checking and Refinement

Rainfall data

In the rainfall data, we ended up leaving out case 28 (Death Valley) because it had a large residual and

its altitude was the lowest in the data set. The resulting model is therefore not applicable to such low

altitude locations. If case 28 had not been unusual, then we would not have been justified in omitting it.

Coefficientsa

Unstandardized Standardized

Coefficients Coefficients

Model B Std. Error Beta t Sig.

1 (Constant) -2.074 .525 -3.951 .001

Altitude (ft) .000725 .000241 4.647 3.012 .006

Latitude (degrees) .093924 .014285 .773 6.575 .000

Rainshadow -.431176 .059929 -.662 -7.195 .000

Altitude*Latitude -.000019 .000006 -4.620 -2.959 .007

a. Dependent Variable: Log10(Precipitation)

R2 = .80

Since there is an interaction between Altitude and Latitude, interpretation of the coefficients for these

variables becomes a little complicated. However, we can interpret the effect of the Rainshadow variable

in this model.

Chap. 11, page 2

Case 28 is an example of an outlier, a case for which the model does not fit well. Outliers have large

residuals. We are also interested in influential cases, cases whose omission changes the fitted model

substantially. Influential cases may not be outliers. Least squares is sensitive to unusual cases and an

influential case may “pull” the regression plane toward it so much that it does not have a large residual.

In simple linear regression, we could often identify influential cases simply from a scatterplot. In

multiple regression, it may not be possible to see influential cases in pairwise scatterplots and we need

additional tools.

Case-Influence statistics

Leverage:

The leverage of a case is based only on the values of the explanatory variables. It measures the distance

of the case from the mean for the explanatory variables (in multidimensional space). For one

explanatory variable, the leverage is

hi =

1 ⎡ Xi − X ⎤

2

1 (X i − X )2 + 1

⎢ ⎥ + =

(n − 1) ⎣ s X ⎦ n ∑ ( X i − X )2 n

With more than one explanatory variable, the leverage is a measure of distance in higher-dimensional

space. The distance takes into account the joint variability of the variables – see Display 11.10 on p.

316.

High-leverage cases are easy to identify visually with only one explanatory variable, but become

increasingly difficult to identify visually with more explanatory variables.

Leverages are always between 1/n and 1. The average of all the leverages in a data set is always p/n

where p is the number of explanatory variables. SPSS computes centered leverages (under the Linear

Regression…Save button), even though it calls them simply “leverages.” The centered leverage is

hi − 1 / n . Therefore, the centered leverage is between 0 and 1-1/n.

Leverage measures the potential influence of a case. High leverage cases have the potential to change

the least squares fit substantially.

Chap. 11, page 3

Studentized residuals

While the true residuals (what we called the ε i ) all have the same standard deviation σ in the regression

model, the observed residuals ei don’t. Why not?

• True residual ε i = Yi − ( β 0 + β1 X i )

• Observed residual: ei = Yi − ( βˆ0 + βˆ1 X i )

First, we already know that the size of the observed residuals tend to be smaller than the sizes of the true

residuals. That’s why we divide by n-2 when we compute the standard deviation of the observed

residuals to get an estimate of the standard deviation of the true residuals. The reason that the observed

residuals tend to be smaller is that the least squares line is the line which best fits the data so the

deviations from this line will tend to be smaller than the deviations from the true line.

What do we mean when we say that the residuals do not all have the same standard deviation? How can

a single value have a standard deviation?

What we mean is: what is the standard deviation of the residuals at each X i from many simulated sets

of data from the linear regression model with a fixed set of Xi ’s?

To carry out this simulation we would follow the following steps. The Xi ’s remain the same for every

simulation.

1. Generate a set of a set of Yi ’s where each Yi is from a normal distribution with mean β 0 + β1 X i

and standard deviation σ. That gives a set of n pairs of values ( X 1 , Y1 ), ( X 2 , Y2 ),… , ( X n , Yn ) .

2. Fit the least squares line

3. Compute the residuals.

4. Repeat steps 1-3 many times with a new set of Yi ’s each time.

Now look at the distribution of observed residuals for each X i and, in particular, compute the standard

deviation of the observed residuals at each X i . You will find that the standard deviations are different

and that the standard deviation of the residuals for X i ’s far from X (high leverage values) is smaller

than for X i ’s near X (low leverage values). In fact, it can be shown that the standard deviation of the

residual at X i is:

SD(Residuali) = σ (1 − hi )

where hi is the leverage. This formula applies to any multiple regression model, not just the simple

linear regression model.

Chap. 11, page 4

Example (simple linear regression):

Suppose Yi is normal with mean µ (Yi ) = 1 + 2 X i , i= 1,..,5, and standard deviation σ = 1, and that the

X i ’s are 1, 4, 5, 6 and 14.

Here are the Yi ’s from one simulation: 3.42, 9.86, 10.05, 12.90, 27.38. The least squares line is

Yˆ = 1.73 + 1.83 X and the residuals are: -0.145, 0.803, -0.844, 0.182, 0.004.

i i

Repeating the simulation 10,000 times, here are the mean and standard deviation of the residuals at each

Xi :

Xi 1 4 5 6 14

mean of residuals 0.008 0.002 -0.013 0.000 0.004

std. dev. of residuals 0.737 0.869 0.884 0.900 0.350

Use the formula on the previous page to calculate the standard deviation of the residuals at each X i .

How do they match the values estimated from the simulation?

Why are we so concerned about the standard deviation of the residuals at different X i ’s?

• Because a big residual is more unusual at a high leverage point than at a low-leverage point.

Therefore, standardizing the residuals by an estimate of their standard deviation is a better way to

compare residuals. Since residuals always have mean 0, this means dividing each residual by an

estimate of its standard deviation σ (1 − hi ) . Since we don’t know σ, we replace it by σˆ (the

square root of mean square residual in the ANOVA table).

• The studentized residual is

resi

studresi = .

σˆ 1 − hi

Chap. 11, page 5

• Studentized residuals are also sometimes called internally studentized residuals. In SPSS, they are

called “studentized residuals” (under the Save button on the Linear Regression window).

• A potential problem with the studentized residuals is that σ̂ may be inflated if a residual is an

outlier. Therefore, a modified version of the studentized residual is the externally studentized

residual, called the studentized deleted residual in SPSS. σ̂ is replaced by σˆ (i ) , the estimated

standard deviation of the residuals from the model fit with the ith observation omitted.

resi

studresi* =

σˆ (i ) 1 − hi

Internally and externally studentized residuals can be used in just the same way as the raw residuals: in

residual plots, normal probability plots, etc. In fact, they are preferred to the raw residuals because the

nonconstant variance of the raw residuals has been corrected for.

When examining studentized residuals, one should look for outliers. In addition, one can use the

standard normal distribution as a rough guide for identifying unusual values: e.g, we expect about 5% of

values less than -2 or greater than 2 and less than 1% to be outside the range –3 to 3.

Cook’s Distance

A more direct measure of the influence of an observation is Cook’s Distance, which measures how

much the fitted values change when each observation is omitted. For case i,

Di = ∑

n (Yˆ j (i ) − Yˆ j )

2

j =1 pσˆ 2

where p is the number of regression coefficients. The numerator of the above expression is what’s

important; the denominator just standardizes the statistic.

Yˆ j is the fitted value for case j when the whole data set is used to fit the model. Yˆ j (i ) is the fitted value

for case j when case i is omitted in fitting the model. So, for example, to calculate D1 we omit case 1,

calculate the model, and calculate the fitted values for all observations including case 1. We then

calculate the sum of the squared differences between these predicted values and the predicted values

from the model fit to all the data. A values of Cook’s D close to or greater than 1 is often considered to

be indicative of an observation with large influence.

While Cook’s D is a useful measure if the goal of the model is prediction, it is not as useful for seeing

how a particular coefficient changes when an observation is omitted. However, it can be used to

identify cases to check – omit a case with large Cook’s D and see how the coefficients of interest

change.

Chap. 11, page 6

Other measures of influence

A number of other measures of influence have been proposed. However, some of these measures are

redundant and it is not necessary to look at all of them. Two others that SPSS computes are DfFits,

which measures how much the predicted value for case i changes when case I is omitted and DfBetas,

which measures how much the omission of case i changes each of the coefficients in the model (hence,

for each case, there is a separate DfBetas value for each variable).

Chap. 11, page 7

Without case #28:

In Sec. 11.4.4, p. 320, Ramsey and Schafer suggest that if “the residual plot from a good inferential

model fails to suggest any problems, there is generally no need to examine case influence statistics at

all.” I would agree except that I would suggest that the residual plot should use the externally

studentized residuals (=studentized deleted residuals).

Chap. 11, page 8

We next examine two types of plots useful in refining models:

• Partial regression leverage plots (also called added-variable plots) are useful for visually

identifying influential and high leverage points for each regression coefficient separately. These

are not discussed in the text, but are easily available in SPSS.

• Partial residual plots (also called component-plus-residual plots) are useful for identifying

nonlinear relationships in a multiple regression model. These are discussed in the text, but are

not readily available in SPSS. They can be constructed in SPSS, but it’s rather tedious.

It might seem that simply plotting the response variable Y versus each explanatory variable would be

adequate for assessing the relationship between Y and each X variable for a multiple regression model.

However, these plots can be misleading because they do not control for the values of the other X

variables. For example, an apparently strong relationship between Y and X 1 may disappear when other

variables are included in the model. If the scatterplot of Y versus X 1 looks curved, it does not

necessarily mean that a squared term will be necessary with the other X variables in the model.

Similarly, a case that appears influential in the Y versus X 1 scatterplot may not be influential with the

other X variables in the model and a case that doesn’t appear influential may turn out to be so with the

other X variables in the model.

Plots of the residuals versus each X variable are also inadequate. They are better than Y versus X plots

because they show only the unexplained variation in Y on the y-axis. However, the X variables are not

adjusted for relationships with each other.

• A partial regression leverage plot (or added-variable plot) attempts to separate out the relationship

between the response and any explanatory variable after adjusting for the other explanatory

variables in the model.

• The steps involved creating the partial regression leverage plot for variable X 1 are:

1. Compute the residuals from the regression of Y on all the other X variables in the model except

X1 .

2. Compute the residuals from the regression of X 1 on all the other X variables in the model.

3. Plot the first set of residuals on the y-axis against the second set on the x-axis.

Steps 1-3 are repeated for all the X variables in the model.

The partial regression leverage plot for X 1 looks at the relationship between Y and X 1 after adjusting

for the other X variables. It turns out that the slope of the least squares line for this plot is exactly equal

to βˆ1 , the coefficient on X 1 in the regression model with all the X’s in it. In addition, high leverage and

influential cases for βˆ can be identified from this scatterplot. This is the primary use of the partial

1

regression leverage plots.

SPSS: Partial regression leverage plots for all X variables can be generated automatically in SPSS by

selecting “Produce all partial plots” on the Plot menu for the Regression…Linear menu.

Chap. 11, page 9

Partial residual plots

• A partial residual plot (or component-plus-residual plot) is constructed differently from a partial

regression leverage plot, but also has the property that the slope of the least squares line through

the plot is the coefficient for that variable in the multiple regression model with all the X variables

included.

• Partial residual plots are better than partial regression leverage plots for identifying nonlinear

relationships between Y and an X variable after adjusting for the other X variables in the model.

• If a clear nonlinear relationship is identified, possible solutions include adding the square of the X

variable to the model, transforming the X variable, or transforming the Y variable.

• To construct the partial residual plot for X 1 , follow the following steps. For the sake of this

example, assume there are three other X variables in the model: X 2 , X 3 , X 4 .

3. Plot the partial residuals for X 1 (on the y-axis) against X 1 (on the x-axis).

• Partial residual plots are also useful for identifying high leverage and influential cases.

• SPSS does not automatically produce partial residual plots (recall that “partial plots” in the PSSS

regression menu means “partial regression leverage plots”). It is somewhat of a hassle to produce

these plots in SPSS manually, but it can be done by following steps 1-3. It is easier to replace step

2 by the equivalent calculation:

2. pres = res + βˆ1 X 1 where res is the residual from the full model fit in step 1.

Thus, the steps are: fit the full model (step 1) and save the residuals as RES_1. Use Transform…

Compute to compute the partial residuals as RES_1+ βˆ1 X 1 where you will type in the value for

βˆ from the model fit in step 1. Plot the partial residuals versus X . Repeat for the other

1 1

variables.

A loess smooth can be added to the partial residual plot to help identify non-linear relationships.

The following page contains both partial regression leverage plots and partial residual plots for the

rainfall data where the log10(Precip) is regressed on Altitude, Latitude, and Rainshadow with no

interaction. It might be best to look for nonlinear relationships before considering interactions, but

certainly these plots can also be used for models with interactions. Case #28 has been omitted.

Coefficientsa

Unstandardized Standardized

Coefficients Coefficients

Model B Std. Error Beta t Sig.

1 (Constant) -1.137 .479 -2.372 .026

Altitude (ft) .0000139 .0000167 .089 .832 .413

Latitude (degrees) .06835 .01302 .562 5.250 .000

Rainshadow -.40686 .06795 -.625 -5.988 .000

a. Dependent Variable: Log10(Precipitation)

Chap. 11, page 10

- HeteroskedasticityUploaded byhishamsauk
- Scott and Watson Chpt 4 SolutionsUploaded byPitchou1990
- MOD1_05Uploaded bybryandown
- CorrelationsUploaded bybabybunso
- 3384-9919-1-PB (1).pdfUploaded byMano Shehzadi
- 308.fullUploaded byHaydar Kilic
- Multivariate Linear RegressionUploaded bykapurrohit4891
- MPRA Paper 41508Uploaded byNikita Chukalovskiy
- Comp Assignment dUploaded bypritish_kul2
- STOCHASTIC FRONTIER.pdfUploaded byMega Puspita
- Summary Output MsUploaded byIke Fitriyaningsih
- ETC1010_Final_S22015Uploaded byMohammad Rashman
- Regression Model for Analyzing the Dgs Structures Propagation CharacterisitcsUploaded byesatjournals
- Smith 1940 Specific Gravity LatexUploaded byNidhi Dubey
- Group 01 AoD_Assignment_Gr. 1Uploaded bySahil Jain
- Linear RegressionUploaded byGiancarlo Santos
- Economics project on forecasting.Uploaded byz.k
- Impact of Television Advertisement on Consumer Buying Behavior the Moderating Role of Religiosity in the Context of PakistanUploaded byMuhammad Umair Muneeb
- amikacina2Uploaded byVeronica Diaz Medina
- ACCG200_SolutionCh_03.pdfUploaded bylethiphuongdan
- AhmedUploaded byStacy Johnson
- R-1Uploaded bymuralidharan
- cs-ks-pendidikanbblr.fix160 new.docUploaded bynater
- 3_28.pdfUploaded byManish Kumar Singh
- Variables EnteredUploaded byKapten Kid
- OJS20120500012_14277069Uploaded byASRIANIMIPA
- Using optical NIR sensor for on-line virgin olive oils.pdfUploaded byMufik Chelsea
- LAMPIRAN.docxUploaded byAjeng Larasati
- RM Practice ExercisesUploaded byAnicar
- Case1 5In Sink EratorUploaded byNeto

- Model- vs. design-based sampling and variance estimationUploaded byFanny Sylvia C.
- ReviewChaps3-4Uploaded byFanny Sylvia C.
- SampleSizeCalcRevisitedUploaded byFanny Sylvia C.
- Hypo%26PowerLectureUploaded byFanny Sylvia C.
- ReviewChaps1-2Uploaded byFanny Sylvia C.
- Non%26ParaBootUploaded byFanny Sylvia C.
- Chapter 21Uploaded byFanny Sylvia C.
- Chapter 20Uploaded byFanny Sylvia C.
- Chapter 14Uploaded byFanny Sylvia C.
- Chapter 13Uploaded byFanny Sylvia C.
- Chapter 12Uploaded byFanny Sylvia C.
- Chapter 8Uploaded byFanny Sylvia C.
- Chapter 10Uploaded byFanny Sylvia C.
- Chapter 9Uploaded byFanny Sylvia C.
- Chapter 5Uploaded byFanny Sylvia C.
- Chapter 6Uploaded byFanny Sylvia C.
- Chapter5p2LectureUploaded byFanny Sylvia C.
- Chapter 7Uploaded byFanny Sylvia C.
- An Ova PowerUploaded byFanny Sylvia C.
- Intro BootstrapUploaded byMichalaki Xrisoula
- Good Article on Standard Error vs Standard DeviationUploaded byAshok Kumar Bharathidasan
- Data Modeling: General Linear Model &Statistical InferenceUploaded byFanny Sylvia C.
- Bio Math 94 CLUSTERING POPULATIONS BY MIXED LINEAR MODELSUploaded byFanny Sylvia C.
- GRM: Generalized Regression Model for Clustering Linear SequencesUploaded byFanny Sylvia C.
- Clustering in the Linear ModelUploaded byFanny Sylvia C.
- R Matrix TutorUploaded byFanny Sylvia C.
- The not so Short Introduction to LaTeXUploaded byoetiker
- Close Out NettingUploaded byFanny Sylvia C.

- Problems and Questions in PhysicsUploaded bytomasisla
- Structural dynamics questionsUploaded byVishnu Sivadas
- PHYSICS 237Uploaded bysky
- 10 Minutes to pandas — pandas 0.24.2 documentation.pdfUploaded byManish Kumar
- guidednotesUploaded byapi-431410685
- Heinemann Physics 12 TextUploaded byJarbyDerby
- 12615757151_2_10_ACRUploaded byPrathibha Chandrashekharappa
- Ada 360720Uploaded byswabright
- CHAPTERWISE QUESTION&ANSWER.DOCXUploaded bySuresh Babu
- 12 Effects of Aggregation on BudgetingUploaded bylaode
- difisekUploaded byMohd Zahirasri Mohd Tohir
- CBS Manual Eng 2011Uploaded byhhiep87
- Chapter 6 permutation and combination.docUploaded byYoke Kean Ng
- Topic 1- Whole NumbersUploaded bynazri
- NI Tutorial 3115 en (1)Uploaded byajith.ganesh2420
- Analysing Classroom AssessmentUploaded byHassan Basarally
- Implementation of Line of Balance Method for Scheduling Highrise BuildingUploaded byAnonymous 7VPPkWS8O
- Estimation of Stability Temperatures From Differential ThermalUploaded bybea
- Math201_F2017_SyllabusUploaded byMahabub Alam
- Raman Scattering in SolidsUploaded byxabeco
- Effect of Residual Stresses on the StrenUploaded byanhxuanqb
- 4nbt1 assessmenttask 4Uploaded byapi-280863641
- Acoustic Instrument Design BookUploaded byAtul Srivastava
- Chapter 1 FrictionUploaded bySherif Yehia Al Maraghy
- maths 12 1Uploaded byapi-230427224
- Trip.Generation.Handbook.3rd.Edition_.pdfUploaded byKobra Man
- PSV Sizing and Reaction Force Modelling Rev 1.1Uploaded byHoang-Vu Bui
- 4690-13589-1-PBUploaded byswetty19
- Determinants (4)Uploaded byAnkur Singh
- Ee.splitted and Mer 'gedUploaded byAshok Patil

## Much more than documents.

Discover everything Scribd has to offer, including books and audiobooks from major publishers.

Cancel anytime.