LINEAR REGRESSION ANALYSIS

MODULE – II
Lecture - 3

Simple Linear Regression

Analysis
Dr. Shalabh
Department of Mathematics and Statistics
Indian Institute of Technology Kanpur
 2

Properties of the direct regression estimators

Unbiased property
s xy
Note that b1= and b0= y − b1 x are the linear combinations of yi (i = 1,..., n).
sxx
Therefore
n
b1 = ∑ ki yi
i =1
n n
where ki =
( xi − x ) / s xx . Note that
=i 1 =i 1
∑ ki =
0 and ∑ ki xi =
1,

n
E (b1 ) = ∑ ki E ( yi )
i =1
n
= ∑ k (β
i =1
i 0 + β1 xi )

= β1.

Thus b1 is an unbiased estimator of β1 . Next

b0 ) E [ y − b1 x ]
E (=
= E [ β 0 + β1 x − b1 x ]
=β 0 + β1 x − β1 x
= β0.

Thus b0 is an unbiased estimators of β 0 .

 3
Variances
Using the assumption that yi ' s are independently distributed, the variance of b1 is
n
=
Var (b1 )
=i 1
∑ k Var ( y ) + ∑∑ k k Cov( y , y )
i
2
i
i j ≠i
i j i j

∑ (x − x )i
2

=σ2 i
(since y1 ,..., yn are independent)
sxx2
σ 2 sxx
=
sxx2
σ2
= .
sxx

Similarly, the variance of b0 is

Var (b0 ) =
Var ( y ) + x 2 Var (b1 ) − 2 xCov( y , b1 ).
First we find that
E { y − E ( y )}{b1 − E (b1 )}
Cov( y , b1 ) =
 
= E ε (∑ ki yi − β1 ) 
 i 
1  
= E (∑ ε i )( β 0 ∑ ki + β1 ∑ ki xi + ∑ kiε i ) − β1 ∑ ε i 
n  i i i i i 
1
[ 0 + 0 + 0 + 0]
=
n
so =0
 1 x2 
(b0 ) σ 2  +
Var= .
 n s xx 
 4

Covariance

The covariance between b0 and b1 is

=
Cov (b0 , b1 ) Cov( y , b1 ) − xVar (b1 )
x 2
= − σ .
sxx
It can further be shown that the ordinary least squares estimators b0 and b1 possess the minimum variance in the class of

linear and unbiased estimators. So they are termed as the Best Linear Unbiased Estimators (BLUE). Such a property is

known as the Gauss-Markov theorem which is discussed later in multiple linear regression model.
 5
Residual sum of squares
The residual sum of squares is given as
n
SS res = ∑ εˆi2
i =1
n
= ∑ ( y − yˆ )
i =1
i i
2

n
= ∑(y −b
i =1
i 0 − b1 xi ) 2
n
= ∑ [( y − y + b x − b x )]
2
i 1 1 i
i =1
n
= ∑ [( y − y ) − b ( x − x )]
2
i 1 i
i =1
n n
n

i 1
2
=
i
=i 1 =i 1 =i 1
2
∑ ( y − y)
1
2
+b ∑ (x − x ) − 2b ∑ ( xi − x )( yi − y )

=s yy + b12 sxx − 2b12 sxx

= s yy − b12 sxx
2
s 
= s yy −  xy  sxx
 sxx 
sxy2
= s yy −
sxx
= s yy − b1sxy .
where
n
1 n
s yy =∑ ( yi − y ) , y = ∑ yi .
2

i 1= ni1
 6

Estimation of σ
2

The estimator of σ2 is obtained from residual sum of squares as follows. Assuming that Since yi is normally distributed,
so SSres has a χ2 distribution with (n - 2) degrees of freedom, so

SS res
~ χ 2 (n − 2).
σ 2

Thus using the result about the expectation of a chi-square random variable, we have

) (n − 2)σ 2 .
E ( SS res=
Thus an unbiased estimator of σ is
2

SS res
s2 = .
n−2
Note that SSres has only (n - 2) degrees of freedom. The two degrees of freedom are lost due to estimation of b0 and b1.
Since s2 depends on the estimates b0 and b1, so it is a model dependent estimate of σ2 .
 7

Estimate of variances of b0 and b1

The estimators of variances of b0 and b1 are obtained by replacing σ2 by σˆ 2 = s 2 as follows:

 21 x2 
Var=
(b0 ) s  + 
 n sxx 
and
 s2
Var (b1 ) = .
sxx
n n
It is observed that since ∑ ( yi − yˆi ) =
0, so ∑e i = 0. In the light of this property, ei can be regarded as an estimate
i =1 i =1

of unknown ε i (i = 1,..., n) . This helps in verifying the different model assumptions on the basis of the given sample

( xi , yi ), i = 1, 2,..., n.

Further, note that

n
(i) ∑xe
i =1
i i = 0,
n
(ii) ∑ yˆ e
i =1
i i = 0,
n n
(iii)
=i 1 =i 1
∑ yi = ∑ yˆi and
(iv) the fitted line always passes through ( x , y ).
 8

Centered model
Sometimes it is useful to measure the independent variable around its mean. In such a case, model yi =β 0 + β1 X i + ε i
has a centered version as follows:

yi = β 0 + β1 ( xi − x ) + β1 x + ε (i = 1, 2,..., n )
= β 0* + β1 ( xi − x ) + ε i

where β= β 0 + β1 x . The sum of squares due to error is given by

*
0

n n 2

0 ) ∑ ε = ∑  yi − β 0* − β 1 ( xi − x )  .
S (β , β =
*
1 i
2

=i 1 =i 1

Now solving
∂S ( β 0* , β1 )
=0
∂β 0*
∂S ( β 0* , β1 )
= 0,
∂β1*
we get the direct regression least squares estimates of β 0* and β1 as

b0* = y

and sxy
b1 =
sxx
respectively.
 9

Thus the form of the estimate of slope parameter β1 remains same in usual and centered model whereas the form of the
estimate of intercept term changes in the usual and centered models.

Further, the Hessian matrix of the second order partial derivatives of S ( β 0* , β1 ) with respect to β 0* and β1 is positive definite
at β 0* = b0* and β1 = b1 which ensures that S ( β 0* , β1 ) is minimized at β 0* = b0* and β1 = b1 .

E (ε i ) 0, Var
Under the assumption that = = (ε i ) σ 2 and Cov=
(ε iε j ) 0 for all=
i ≠ j 1, 2,..., n. It follows that

E (b0* ) β=
= *
0 , E (b1 ) β1 ,
σ2 σ2
=
Var (b0* ) = , Var (b1 ) .
n sxx

In this case, the fitted model of yi = β 0* + β1 ( xi − x ) + ε i is

y=y + b1 ( x − x ),

and the predicted values are

yˆi =+
y b1 ( xi − x ) (i =
1,..., n).

Note that in centered model

Cov(b0* , b1 ) = 0.
 10

No intercept term model

Sometimes in practice a model without an intercept term is used in those situations when xi =0 ⇒ yi =0 for all
i = 1, 2,..., n . A no-intercept model is
yi = β1 xi + ε i (i = 1, 2,.., n ).
For example, in analyzing the relationship between illumination of bulb (y) and electric current (X), the illumination of bulb is
zero when current is zero.

Using the data ( xi , yi ), i = 1, 2,..., n, the direct regression least squares estimate of β1 is obtained by minimizing

n n
( β1 )
S= ∑=
=i 1 =i 1
ε i2 ∑(y − β x ) i 1 i
2

and solving
∂S ( β1 )
=0
∂β1
gives the estimator of β1 as
n

∑yx i i
b =
*
1
i =1
n
.
∑x
i =1
2
i

The second order partial derivative of S ( β1 ) with respect to β1 at β1 = b1 is positive which ensures that b1 minimizes S ( β1 ).
 11

E (ε i ) 0, Var
Using the assumption that = = (ε i ) σ 2 and Cov=
(ε iε j ) 0 for all=
i ≠ j 1, 2,..., n., the properties of b1* can be
n
derived as follows: ∑ x E( y ) i i
E (b ) =
*
1
i =1
n

∑x
i =1
2
i

∑x β 2
i 1
= i =1
n

∑x
i =1
2
i

= β1.

This b1* is an unbiased estimator of β1. The variance of b1* is obtained as follows:
n

∑ x Var ( y )
2
i i
Var (b ) =
*
1
i =1
2
 n 2
 ∑ xi 
 i =1 
n

∑x 2
i
=σ2 i =1
2
 n 2
 ∑ xi 
 i =1 
σ2
= n

∑x 2
i n n
i =1
∑ yi2 − b1 ∑ yi xi
and an unbiased estimator of σ=
2
is i 1 =i 1 .
n −1