Least Squares Regression

Gauss Markov Theorem

The Gauss Markov theorem tells us that if a certain set

of assumptions are met, the ordinary least squares
estimate for regression coefficients gives you the best
linear unbiased estimate (BLUE) possible.
 Gauss Markov Assumptions

• Linearity: the parameters we are estimating using the OLS method

must be themselves linear.
• Random: our data must have been randomly sampled from the
population.
• Non-Collinearity: the regressors being calculated aren’t perfectly
correlated with each other.
• Exogeneity: the regressors aren’t correlated with the error term.
• Homoscedasticity: no matter what the values of our regressors might
be, the error of the variance is constant.
 Linearity

The parameters we are estimating using the OLS method must be

themselves linear.
 Random

Random samples are used to avoid bias and other unwanted

effects.
 Non-Collinearity

A set of points is collinear if you can draw one line through

them all. The word literally means “together on a line.”
 Exogeneity

Exogeneity is a standard assumption made in regression

analysis, and when used in reference to a regression equation
tells us that the independent variables X are not dependent on
the dependent variable (Y)
Homoscedasticity
 What is a Least Squares Regression Line?

• If your data shows a linear relationship between the X and Y variables,

you will want to find the line that best fits that relationship.
• That line is called a Regression Line and has the equation ŷ= a + b x.
• The Least Squares Regression Line is the line that makes the vertical
distance from the data points to the regression line as small as
possible.
• It’s called a “least squares” because the best line of fit is one that
minimizes the variance (the sum of squares of the errors).
• This can be a bit hard to visualize but the main point is you are aiming
to find the equation that fits the points as closely as possible.
 What is Least Squares Fitting?

• Least squares fitting (also called least squares estimation) is a way to

find the best fit curve or line for a set of points.
• In this technique, the sum of the squares of the offsets (residuals) are
used to estimate the best fit curve or line instead of the absolute
values of the offsets.
• The resulting equation gives you a y-value for any x-value, not just
those x and y values plotted with points.
 Advantages of least squares fitting

• Least squares allows the residuals to be treated as a continuous

quantity where derivatives (measures of how much a function’s
output changes when an input changes) can be found. This is
invaluable, as the point of finding an equation in the first place is to
be able to predict where other points on the line (even points that are
way beyond the original points) might lie.
Disadvantages of Least Squares Fitting

• Outliers can have a disproportionate effect if you use the least

squares fitting method of finding an equation for a curve. This is
because the squares of the offsets are used instead of the absolute
value of the offsets; outliers naturally have larger offsets and will
affect the line more than points closer to the line. These
disproportionate values may be beneficial in some cases.
 Ordinary Least Squares

• Ordinary least squares regression is a way to find the line of best fit
for a set of data. It does this by creating a model that minimizes the
sum of the squared vertical distances (residuals).
• The distances are squared to avoid the problem of distances with a
negative sign. Then the problem just becomes figuring out where you
should place the line so that the distances from the points to the line
are minimized. In the following image, the best fit line A has smaller
distances from the points to the line than the randomly placed line B.
 The Residuals
• The difference between the observed and predicted values of y is the
error, or residual.