Properties of Least Squares Estimators

Simple Linear Regression

Model: Y = β0 + β1x + 

•  is the random error so Y is a random variable too.

Sample:

(x1, Y1), (x2, Y2), . . . , (xn, Yn)

Each (xi, Yi) satisfies Yi = β0 + β1xi + i

Least Squares Estimators:

Pn
(x − x)(Yi − Y)
β̂1 = Pn i
i=1
2
, β̂0 = Y − β̂1x
i=1 (xi − x)

1
 Properties of Least Squares Estimators

Assumptions on the Random Error 

• E[i] = 0

• V (i) = σ 2

Implications for the Response Variable Y

• E[Yi] = β0 + β1xi

• V (Yi) = σ 2

What can be said about:

• E[β̂1] ?

• V (β̂1) ?

• E[β̂0] ?

• V (β̂0) ?

• Cov(β̂0, β̂1) ?
2
 Properties of Least Squares Estimators

Proposition: The estimators β̂0 and β̂1 are unbiased;

that is,
E[β̂0] = β0, E[β̂1] = β1.

Proof:
Pn
(x − x)(Yi − Y)
β̂1 = Pn i
i=1
2
i=1 (xi − x)

Pn Pn
i=1 (xi − x)Yi − Y i=1(xi − x)
= Pn 2
i=1 (xi − x)

Pn
(xi − x)Yi
= Pi=1
n 2
i=1 (xi − x)

3
Thus
n
(x − x)
Pn i
X
E[β̂1] = 2
E[Yi]
i=1 i=1 (xi − x)

n
(x − x)
Pn i
X
= 2
(β0 + β1xi)
i=1 i=1 (xi − x)

Pn Pn
(xi − x) i=1 (xi − x)xi
= Pni=1 β
2 0
+ β1 P n 2
(x
i=1 i − x) i=1 (xi − x)

E[β̂0] = E[Y − β̂1x]

n
1X
= E[Yi] − E[β̂1]x
n i=1
n n
1X 1X
= (β0 + β1xi) − β1 xi
n i=1 n i=1

4
 Properties of Least Squares Estimators

Proposition: The variances of β̂0 and β̂1 are:

2
Pn 2 2
Pn 2
σ i=1 xi σ i=1 xi
V (β̂0) = n
P 2
=
(x
i=1 i − x) Sxx
and
σ2 σ2
V (β̂1) = Pn 2
= .
i=1 (xi − x) Sxx

Proof:
Pn 
i=1 (xi − x)Yi
V (β̂1) = V
Sxx
 n
2 X
1
= (xi − x)2V (Yi)
Sxx i=1

 2 Xn
!
1
= (xi − x)2 σ 2
Sxx i=1
 
1
= σ2
Sxx

5
 V (β̂0) = V (Y − β̂1x)

= V (Y ) + V (−β̂1x) + 2Cov(Y , −xβ̂1)

= V (Y ) + x2V (β̂1) − 2xCov(Y , β̂1)

σ2
 2
2 σ
= +x − 2xCov(Y , β̂1)
n Sxx

Now let’s evaluate the covariance term:

 
n n
X 1 X xj − x 
Cov(Y , β̂1) = Cov  Yi , Yj
i=1
n j=1 Sxx

n X
n
X xj − x
= Cov(Yi, Yj )
i=1 j=1
nSxx

n
X xi − x
= σ2 + 0
i=1
nSxx

= 0
6
Thus

2 2
 
σ σ
V (β̂0) = + x2
n Sxx

Sxx + nx2
= σ2
nSxx
Pn 2 2
i=1 (xi − x) + nx
= σ2
nSxx
Pn 2 2 2
i=1 (xi − 2xxi + x ) + nx
= σ2
nSxx
Pn 2
i=1 xi
= σ2
nSxx

7
 Properties of Least Squares Estimators

A Practical Matter
The variance σ 2 of the random error  is
usually not known. So it is necessary to esti-
mate it.

Proposition: The estimator

n
2 1 X 2 1
S = (Yi − Ŷi) = SSE
n − 2 i=1 n−2

is an unbiased estimator of σ 2.

8
 Properties of Least Squares Estimators

Assumptions on the Random Error 

• E[i] = 0

• V (i) = σ 2

• Each i is normally distributed.

Implications for the Estimators

• β̂1 is normally distributed with mean β1 and variance

σ2
Sxx ;

• β̂0 is Pnormally distributed with mean β0 and variance

n 2
2 i=1 xi
σ nSxx ;

(n−2)S 2
• σ2
has a χ2 distribution with n − 2 degrees of free-
dom;

• S 2 is independent of β̂0 and β̂1.

9
 Properties of Least Squares Estimators

Multiple Linear Regression

Model: Y = β0 + β1x1 + · · · + βk xk + 

Sample:

(x11, x12, . . . , x1k , Y1)

(x21, x22, . . . , x2k , Y2)
..
(xn1, xn2, . . . , xnk , Yn)

Each (xi, Yi) satisfies Yi = β0 +β1xi +· · ·+βk xk +i

Least Squares Estimators:

β̂ = (X0X)−1X0Y

10
 Properties of Least Squares Estimators

• Each β̂i is an unbiased estimator of βi: E[β̂i] = βi;

• V (β̂i) = ciiσ 2, where cii is the element in the ith row

and ith column of (X0X)−1;

• Cov(β̂i, β̂i) = cij σ 2;

• The estimator
0 0 0
2 SSE Y Y − β̂ XY
S = =
n − (k + 1) n − (k + 1)
is an unbiased estimator of σ 2.

11
 Properties of Least Squares Estimators

When  is normally distributed,

• Each β̂i is normally distributed;

• The random variable

(n − (k + 1))S 2
σ2
has a χ2 distribution with n − (k + 1) degrees of freee-
dom;

• The statistics S 2 and β̂i, i = 0, 1, . . . , k, are indepen-

dent.

12