Study Notes
Ordinary Least Square
(OLS) Method
OLS Method
Ordinary Least Squares (OLS) Method
Minimises the sum of the squares of the error term.
∑ ᷇ μ 2i =∑ ¿
After minimization, you derive the normal equations.
Using the normal equations, regression coefficients can be estimated.
It is also possible to find the regression equations by using the following equation:
Y −Y =b YX ¿) or
X −X =b XY ¿)
Normal Equations
2
OLS Method
Derivation of Formulas for ^β 0 and ^β 1
3
OLS Method
4
OLS Method
Drawback of OLS Method
Gives equal weightage to all the error term.
∑ ᷇ μ 2i =∑ ( μ21 + μ22 + …. + μ2n ¿2
However, points closer to the SRF should get more weightage as compared to points
which are far away because the points which are close have contributed the most in the
construction of the SRF.
This limitation is overcome by Weighted Least Square Method.
Properties of Sample Regression Line underlying OLS Method
1. It passes through the sample means of X and Y.
∑ Y i=n β^ 0 + β^ 1 ∑ X i (from first normal equation)
Divide both sides by n,
^β =Y − ^β X
n ( )
∑Y i ^ ^ ∑ X i
= β 0+ β 1
n
Y = ^β0 + β^ 1 X ⇒ 0 1
2. Mean value of estimatedY =Y^ i is equal to the mean value of actual Y.
Y^ i= β^ 0 + β^ 1 X i
¿ ( Y − ^β 1 X ) + ^β 1 X i
¿ Y + ^β ( X −X )
1 i
∑ Y^ i ∑ Y Σ ( X i−X )
= + ^β 1
n n n
Y^ =Y [ since , ∑ ( X i −X ) =0 ] .
3. Mean value of residuals is 0.
5
OLS Method
−2 ∑(Y i − ^β 0− ^β 1 ∑ X i)=0 [from first normal equation]
∑ ^μi=0
Hence ,
^μ=0
4. The residuals are uncorrelated with the predicted Y.
Σ 1 u^ i Y^ i=Σ ^μi ( ^β 0 + ^β 1 X i )
¿ ^β Σ ^μ + ^β Σ μ^ X
0 i 1 i i
¿ ^β 0 ( 0 ) + ^β 1 ( 0 ) [from normal equations]
¿0
5. The residuals are uncorrelated with X.
Σ μ^ i X i=0 [from second normal equation]
Gauss-Markov Theorem
In the class of all linear and unbiased estimators of β, the OLS estimator ^β has the minimum
variance. In short, ^β is BLUE (best linear unbiased estimator).
It is linear, that is, a linear function of a random variable, such as the dependent variable
Y in the regression model.
It is unbiased, that is, its average or expected value, E ( ^β ), is equal to the true value, β.
It has minimum variance in the class of all such linear unbiased estimators; an unbiased
estimator with the least variance is known as an efficient estimator.
To see this, look at Precision or Standard Errors of Least Square Estimates
Precision or Standard Errors of least square estimates
6
OLS Method
7
OLS Method
8
OLS Method
9
OLS Method
Where Sxy denote the S.E. of estimate of regression equation of x on y.
Degrees of freedom
10
OLS Method
The term number of degrees of freedom means the total number of observations in the
sample (= n) less the number of independent (linear) constraints or restrictions put on them.
In other words, it is the number of independent observations out of a total of n observations.
For example, before the RSS can be computed, ᷇ β 1 and ᷇ β 2 must first be obtained.
These two estimates therefore put two restrictions on the RSS (residual sum of squares).
Therefore, there are n− 2, not n, independent observations to compute the RSS.
Following this logic, in the three-variable regression RSS will have n− 3 df, and for the k-
variable model it will have n− k df.
The general rule is this: df = (n− number of parameters estimated).
Error term 𝞵 and method of least squares
The stochastic error term is a catchall that includes all those variables that cannot be readily
quantified. It may represent variables that cannot be included in the model for lack of data
Whatever the source of the random term 𝞵, it is assumed that the average effect of the error
availability, or errors of measurement in the data, or intrinsic randomness in human behaviour.
term on the regression is marginal at best.
Refer to the example discussed above. As per the regression line Y = -7.4 + 1.64X, the
calculated values of Y will be as follows:
Error Square of
Rainfall (X) Yield (Y) Y1 (calculated) (Y-Y1) error
12 14 12.28 1.72 2.96
9 8 7.36 0.64 0.41
8 6 5.72 0.28 0.08
10 9 9 0 0
11 11 10.64 0.36 0.13
13 12 13.92 -1.92 3.69
7 3 4.08 -1.08 1.17
70 63 8.43
At these values of a and b, the sum of squares of the errors (or residuals) is minimum.
Graphically, it can be depicted as:
11
OLS Method
Regression line fitted as per The line minimizes the sum
actual data of squared errors
12
