Multiple Regression Diagnostics 2

Sociology 8811
Copyright © 2007 by Evan Schofer
Do not copy or distribute without
permission
 Announcements
• None
 Multiple Regression Assumptions
• 3. d. Predictors (Xis) are uncorrelated with error
– This most often happens when we leave out an
important variable that is correlated with another Xi
– Example: Predicting job prestige with family wealth,
but not including education
– Omission of education will affect error term. Those
with lots of education will have large positive errors.
• Since wealth is correlated with education, it will be
correlated with that error!
– Result: coefficient for family wealth will be biased.
 Multiple Regression Assumptions
• 4. In systems of equations, error terms of
equations are uncorrelated
• Knoke, p. 256
– This is not a concern for us in this class
• Worry about that later!
 Multiple Regression Assumptions
• 5. Sample is independent, errors are random
– Not only should errors not increase with X
(heteroskedasticity), there should be no pattern at all!
• Cases that are non-independent often have
correlated error
• Things that cause patterns in error
(autocorrelation):
– Measuring data over long periods of time (e.g., every
year). Error from nearby years may be correlated.
• Called: “Serial correlation”.
 Multiple Regression Assumptions
• More things that cause patterns in error
(autocorrelation):
– Measuring data in families. All members are similar,
will have correlated error
– Measuring data in geographic space.
• Example: data on 50 US states. States in a similar region
have correlated error
• Called “spatial autocorrelation”
• There are variations of regression models to
address each kind of correlated error.
 Regression: Outliers
• Note: Even if regression assumptions are met,
slope estimates can have problems
• Example: Outliers -- cases with extreme values
that differ greatly from the rest of your sample
• More formally: “influential cases”
• Outliers can result from:
• Errors in coding or data entry
• Highly unusual cases
• Or, sometimes they reflect important “real” variation
• Even a few outliers can dramatically change
estimates of the slope, especially if N is small.
 Regression: Outliers
• Outlier Example: Extreme case that
pulls regression
4 line up

-4 -2 0 2 4
Regression line
-2 with extreme case
removed from
sample
-4
 Regression: Outliers
• Strategy for identifying outliers:
• 1. Look at regression partial plots (avplots) for
extreme values
• 2. Compute outlier diagnostic statistics
– High values indicate potential outliers
• “Leverage”
• Cook’s D
• DFFIT
• DFBETA
• residuals, standardized residuals, studentized residuals.
 Scatterplots
• Example: Study time and student achievement.
– X variable: Average # hours spent studying per day
– Y variable: Score on reading test
Case X Y Y axis
30
1 2.6 28
2 1.4 13
20
3 .65 17
4 4.1 31
10
5 .25 8
6 1.9 16 X axis
7 3.5 6 0
0 1 2 3 4
 Outliers
• Results with outlier:
Model Summaryb

Adjusted Std. Error of

Model R R Square R Square the Estimate
1 .466a .217 .060 9.1618
a. Predictors: (Constant), HRSTUDY
Coefficientsa
b. Dependent Variable: TESTSCOR
Standardi
zed
Unstandardized Coefficien
Coefficients ts
Model B Std. Error Beta t Sig.
1 (Constant) 10.662 6.402 1.665 .157
HRSTUDY 3.081 2.617 .466 1.177 .292
a. Dependent Variable: TESTSCOR
 Outlier Diagnostics
• Residuals: The numerical value of the error
• Error = distance that points falls from the line
• Cases with unusually large error may be outliers
• Note: residuals have many other uses!
• Standardized residuals
– Z-score of residuals… converts to a neutral unit
– Often, standardized residuals larger than 3 are
considered worthy of scrutiny
• But, it isn’t the best outlier diagnostic
– Studentized residuals: recalculates standarized
residual after removing case from analysis.
 Outlier Diagnostics
• Cook’s D: Identifies cases that are strongly
influencing the regression line
– SPSS calculates a value for each case
• Go to “Save” menu, click on Cook’s D
• How large of a Cook’s D is a problem?
– Rule of thumb: Values greater than: 4 / (n – k – 1)
– Example: N=7, K = 1: Cut-off = 4/5 = .80
– Cases with higher values should be examined.
 Outlier Diagnostics
• Example: Outlier/Influential Case Statistics
Hours Score Resid Std Resid Cook’s D
2.60 28 9.32 1.01 .124
1.40 13 -1.97 -.215 .006
.65 17 4.33 .473 .070
4.10 31 7.70 .841 .640
.25 8 -3.43 -.374 .082
1.90 16 -.515 -.056 .0003
3.50 6 -15.4 -1.68 .941
 Outliers
• Results with outlier removed:
Model Summaryb

Adjusted Std. Error of

Model R R Square R Square the Estimate
1 .903a .816 .770 4.2587
a. Predictors: (Constant), HRSTUDY
Coefficientsa
b. Dependent Variable: TESTSCOR
Standardi
zed
Unstandardized Coefficien
Coefficients ts
Model B Std. Error Beta t Sig.
1 (Constant) 8.428 3.019 2.791 .049
HRSTUDY 5.728 1.359 .903 4.215 .014
a. Dependent Variable: TESTSCOR
 Regression: Outliers
• Question: What should you do if you find
outliers? Drop outlier cases from the analysis?
Or leave them in?
– Obviously, you should drop cases that are incorrectly
coded or erroneous
– But, generally speaking, you should be cautious about
throwing out cases
• If you throw out enough cases, you can produce any result
that you want! So, be judicious when destroying data.
 Regression: Outliers
• Circumstances where it can be good to drop
outlier cases:
• 1. Coding errors
• 2. Single extreme outliers that radically change
results
– Your results should reflect the dataset, not one case!
• 3. If there is a theoretical reason to drop cases
– Example: In analysis of economic activity,
communist countries may be outliers
• If the study is about “capitalism”, they should be dropped.
 Regression: Outliers
• Circumstances when it is good to keep outliers
• 1. If they form meaningful cluster
– Often suggests an important subgroup in your data
• Example: Asian-Americans in a dataset on education
• In such a case, consider adding a dummy variable for them
– Unless, of course, research design is not interested in
that sub-group… then drop them!
• 2. If there are many
– Maybe they reflect a “real” pattern in your data.
 Regression: Outliers
• When in doubt: Present results both with and
without outliers
– Or present one set of results, but mention how results
differ depending on how outliers were handled
• For final projects: Check for outliers!
• At least with scatterplots
– In the text: Mention if there were outliers, how you
handled them, and the effect it had on results.
 Multicollinearity
• Another common regression problem:
Multicollinearity
• Definition: collinear = highly correlated
• Multicollinearity = inclusion of highly correlated
independent variables in a single regression model
• Recall: High correlation of X variables causes
problems for estimation of slopes (b’s)
• Recall: variable denominators approach zero, coefficients
may wrong/too large.
 Multicollinearity
• Multicollinearity symptoms:
• Unusually large standard errors and betas
• Compared to if both collinear variables aren’t included
• Betas often exceed 1.0
• Two variables have the same large effect when
included separately… but…
– When put together the effects of both variables shrink
– Or, one remains positive and the other flips sign
• Note: Not all “sign flips” are due to multicollinearity!
 Multicollinearity
• What does multicollinearity do to models?
• Note: It does not violate regression assumptions
• But, it can mess things up anyway
• 1. Multicollinearity can inflate standard error
estimates
• Large standard errors = small t-values = no rejected null
hypotheses
• Note: Only collinear variables are effected. The rest of the
model results are OK.
 Multicollinearity
• What does multicollinearity do?
• 2. It leads to instability of coefficient estimates
• Variable coefficients may fluctuate wildly when a collinear
variable is added
• These fluctuations may not be “real”, but may just reflect
amplification of “noise” and “error”
– One variable may only be slightly better at predicting Y… but
SPSS will give it a MUCH higher coefficient.
 Multicollinearity
• Diagnosing multicollinearity:
• 1. Look at correlations of all independent vars
• Correlation >.8 is is a concern
• But, sometimes problems aren’t always bivariate… and
don’t show up in bivariate correlations
– Ex: If you forget to omit a dummy variable

• 2. Watch out for the “symptoms”

• 3. Compute diagnostic statistics
• Tolerance, VIF (Variance Inflation Factor).
 Multicollinearity
• Multicollinearity diagnostic statistics:
• “Tolerance”: Easily computed in SPSS
• Low values indicate possible multicollinearity
– Start to pay attention at .3; Below .1 is very likely to be a problem
• Tolerance is computed for each independent variable by
regressing it on other independent variables
– VIF = inverse of tolerance
 Multicollinearity
• If you have 3 independent variables: X 1, X2, X3…

– Tolerance is based on doing a regression: X1 is

dependent; X2 and X3 are independent.
• Tolerance for X1 is simply 1 minus regression R-square.

• If a variable (X1) is highly correlated with all the

others (X2, X3) then they will do a good job of
predicting it in a regression
• Result: Regression r-square will be high… 1 minus r-
square will be low… indicating a problem.
 Multicollinearity
• Variance Inflation Factor (VIF) is the reciprocal
of tolerance: 1/tolerance
• High VIF indicates multicollinearity
– Gives an indication of how much the Standard Error
of a variable grows due to presence of other variables.
 Multicollinearity
• Solutions to multcollinearity
– It can be difficult if a fully specified model requires
several collinear variables
• 1. Drop unnecessary variables
• 2. If two collinear variables are really measuring
the same thing, drop one or make an index
– Example: Attitudes toward recycling; attitude toward
pollution. Perhaps they reflect “environmental views”
• 3. Advanced techniques: e.g., Ridge regression
• Uses a more efficient estimator (but not BLUE – may
introduce bias).
 “Robust” Standard Errors
• Robust / Huber / White / Sandwich standard error
– An alternative method of estimating regression SEs
• More accurate under conditions of heteroskedasticity
• Potentially more accurate under conditions of non-
independence (clustered data)
• Potentially more accurate when model is underspecified
• Stata: regress y x1 x2, robust
– Increasingly common… Some people now use them
routinely…
• But, Freedman (2006) criticizes use for underspecification:
• What use are SEs if model is underspecified and therefore
slopes are biased?
 Models and “Causality”
• Issue: People often use statistics to support
theories or claims regarding causality
– They hope to “explain” some phenomena
• What factors make kids drop out of school
• Whether or not discrimination leads to wage differences
• What factors make corporations earn higher profits
• Statistics provide information about association
• Always remember: Association (e.g.,
correlation) is not causation!
• The old aphorism is absolutely right
• Association can always be spurious
 Models and “Causality”
• How do we determine causality?
• The randomized experiment is held up as the
ideal way to determine causality
• Example: Does drug X cure cancer?
• We could look for association between receiving
drug X and cancer survival in a sample of people
• But: Association does not demonstrate causation; Effect
could be spurious
• Example: Perhaps rich people have better access to drug X;
and rich people have more skilled doctors!
• Can you think of other possible spurious processes?
 Models and “Causality”
• In a randomized experiment, people are assigned
randomly to take drug X (or not)
• Thus, taking drug X is totally random and totally
uncorrelated with any other factor (such as wealth, gender,
access to high quality doctors, etc)
• As a result, the association between drug X and
cancer survival cannot be affected by any
spurious factor
• Nor can “reverse causality” be a problem
• SO: We can make strong inferences about causality!
 Models and “Causality”
• Unfortunately, randomized experiments are
impractical (or unethical) in many cases
• Example: Consequences of high-school dropout, national
democracy, or impact of homelessness
• Plan B: Try to “control” for spurious effects:
• Option 1: Create homogenous sub-groups
– Effects of Drug X: If there is a spurious relationship
with wealth, compare people with comparable wealth
• Ex: Look at effect of drug X on cancer survivors among
people of constant wealth… eliminating spurious effect.
 Models and “Causality”
• Option 2: Use multivariate model to “control”
for spurious effects
• Examine effect of key variable “net” of other relationships
– Ex: Look at effect of Drug X, while also including a
variable for wealth
• Result: Coefficients for Drug X represent effect net of
wealth, avoiding spuriousness.
 Models and “Causality”
• Limitations of “controls” to address spuriousness
• 1. The “homogenous sub-groups” reduces N
• To control for many possible spurious effects, you’ll throw
away lots of data
• 2. You have to control for all possible spurious
effects
• If you overlook any important variable, your results could
be biased… leading to incorrect conclusions about causality
• First: It is hard to measure and control for everything
• Second: Someone can always think up another thing you
should have controlled for, undermining your causal claims.
 Models and “Causality”
• Under what conditions can a multivariate model
support statements about causality?
• In theory: A multivariate model support claims
about causality… IF:
• The sample is unbiased
• The measurement is accurate
• The model includes controls for every major possible
spurious effect
• The possibility of reverse causality can be ruled out
• And, the model is executed well: assumptions, outliers,
multicollinearity, etc. are all OK.
 Models and “Causality”
• In Practice: Scholars commonly make tentative
assertions about causality… IF:
• The data set is of high quality; sample is either random or
arguably not seriously biased
• Measures are high quality by the standards of the literature
• The model includes controls for major possible spurious
effects discussed in the prior literature
• The possibility of reverse causality is arguably unlikely
• And, the model is executed well: assumptions, outliers,
multicollinearity, etc. are all acceptable… (OR, the author
uses variants of regression necessary to address problems).
 Models and “Causality”
• In sum: Multivariate analysis is not the ideal tool
to determine causality
• If you can run an experiment, do it
• But: Multivariate models are usually the best tool that we
have!
• Advice: Multivariate models are a terrific way to
explore your data
• Don’t forget: “correlation is not causation”
• The models aren’t magic; they simply sort out correlation
• But, if used thoughtfully, they can provide hints into likely
causal processes!