Professional Documents
Culture Documents
In this section we will briefly discuss a matrix approach to fitting simple linear
regression models. A random sample of size n gives n equations. For the full
SLRM we have
Y1 = β0 + β1 x1 + ε1
Y2 = β0 + β1 x2 + ε2
.. ..
. .
Yn = β0 + β1 xn + εn
We can write this in matrix formulation as
Y = Xβ + ε, (2.22)
where Y is an (n×1) vector of response variables (random sample), X is an (n×
2) matrix called the design matrix, β is a (2 × 1) vector of unknown parameters
and ε is an (n × 1) vector of random errors. That is,
Y1 1 x1 ε1
Y2 1 x2 ε2
β0
Y = .. , X = .. .. , β= , ε = .. .
. . . β1 .
Yn 1 xn εn
The assumptions about the random errors let us write
ε ∼ N n 0, σ 2 I ,
that is vector ε has n-dimensional normal distribution with
ε1 E(ε1 ) 0
ε2 E(ε2 ) 0
E(ε) = E .. = .. = .. = 0
. . .
εn E(εn ) 0
and the variance-covariance matrix
var(ε1 ) cov(ε1 , ε2 ) . . . cov(ε1 , εn )
cov(ε2 , ε1) var(ε2 ) . . . cov(ε2 , εn )
Var(ε) = .. .. .. ..
. . . .
cov(εn , ε1 ) cov(εn , ε2 ) . . . var(εn )
σ2 0 . . . 0
0 σ2 . . . 0
2
= .. .. . . .. = σ I
. . . .
0 0 . . . σ2
2.8. MATRIX APPROACH TO SIMPLE LINEAR REGRESSION 49
This formulation is usually called the Linear Model (in β). All the models we
have considered so far can be written in this general form. The dimensions of
matrix X and of vector β depend on the number p of parameters in the model
and, respectively, they are n × p and p × 1. In the full SLRM we have p = 2.
Yi = β0 + εi for i = 1, . . . , n
is equivalent to
Y = 1β0 + ε
where 1 is an (n × 1) vector of 1’s.
Yi = β1 xi + εi for i = 1, . . . , n
Quadratic regression (p = 3)
Yi = β0 + β1 xi + β2 x2i + εi for i = 1, . . . , n
The normal equations obtained in the least squares method are given by
b
X T Y = X T X β.
50 CHAPTER 2. SIMPLE LINEAR REGRESSION
Note:
Let A and B be a vector and a matrix of real constants and let Z be a vector of
random variables, all of appropriate dimensions so that the addition and multipli-
cation are possible. Then
In particular,
E(Y ) = E(Xβ + ε) = Xβ
Var(Y ) = Var(Xβ + ε) = Var(ε) = σ 2 I.
These equalities let us prove the following theorem.
b of β is unbiased and its variance-
Theorem 2.7. The least squares estimator β
covariance matrix is
b = σ 2 (X T X)−1 .
Var(β)
b = Var{(X T X)−1 X T Y }
Var(β)
= (X T X)−1 X T Var(Y )X(X T X)−1
= σ 2 (X T X)−1 X T IX(X T X)−1 = σ 2 (X T X)−1 .
e = Y − Yb ,
\) = X β
where Yb = E(Y b is the vector of fitted responses µ
bi . It can be shown that
the following theorem holds.
52 CHAPTER 2. SIMPLE LINEAR REGRESSION
E(e) = 0
var[ei ] = σ 2 (1 − hii ),
where the leverage hii is the ith diagonal element of the Hat Matrix H = X(X T X)−1 X T ,
i.e.,
T
hii = xT
i (X X) xi ,
−1
where xT
i = (1, xi ) is the ith row of matrix X.
µi ) = var(xT
var(b b 2 T T 2
i β) = σ xi (X X) xi = σ hii
−1
We can easily obtain other results we have seen for the SLRM written in non-
matrix notation, now using the matrix notation, both for the full model and for a
reduced SLM (no intercept or zero slope).
b = σ 2 (X T X)−1 . Thus
Now, by Theorem 2.7, Var[β]
P 2
xi
var[βb0 ] = σ 2
nSxx
P P n o
1 x̄2
which, by writing x2 = x2 − nx̄2 + nx̄2 , can be written as σ 2 n
+ Sxx
.
Also,
−nx̄
cov(βb0 , βb1 ) = σ 2
nSxx
−σ 2 x̄
= ,
Sxx
and
σ2
var[βb1 ] = .
Sxx
The quantity hii is given by
T
hii = xTi (X X) xi
−1
P 2
1 xj −nx̄ 1
= (1 xi ) .
nSxx −nx̄ n xi
1 (xi − x̄)2
hii = + .
n Sxx
Y = Xβ0 + ε
P
where X = 1 is an (n × 1) vector of 1’s. So X T X = n, X T Y = Yi ,
which gives
1X
βb = (X T X)−1 X T Y = Yi = Ȳ = βb0 ,
n
b T −1 2 σ2
var[β] = (X X) σ = .
n
54 CHAPTER 2. SIMPLE LINEAR REGRESSION
2. No-intercept model
We saw that this example fits the General Linear Model with
x1
x2
X = .. , β = β1
.
xn
P P
So X T X = x2i and X T Y =
xi Yi , and we can calculate
P
xi Yi
βb = (X X) X Y = P 2 = βb1 ,
T −1 T
xi
2
b = σ 2 (X T X)−1 = Pσ .
Var[β]
x2i