Professional Documents
Culture Documents
Regression Analysis: The Multiple Regression Model in Matrices
Regression Analysis: The Multiple Regression Model in Matrices
Regression Analysis
(1) y = 0 + 1 x1 + + k xk + ,
(3) y = X + .
S() =
= (y X) (y X)
(4)
= y y y X X y + X X
= y y 2y X + X X.
1
D.S.G. POLLOCK: ECONOMETRICS
S
(5) = 2y X + 2 X X.
(6) X X = X y.
On the assumption that the inverse matrix exists, the equations have a unique
solution, which is the vector of ordinary least-squares estimates:
(7) = (X X)1 X y.
2
REGRESSION ANALYSIS IN MATRIX ALGEBRA
e
y
+ (X ) (y X )
(y ) (y ) = (y X ) + (X )
(9)
= (I P )y + P (y g) (I P )y + P (y g) .
The properties of the projector P , which have been used in simplifying equation
(9), indicate that
(y ) (y ) = y (I P )y + (y g) P (y g)
(10)
= e e + (X ) (X ).
3
D.S.G. POLLOCK: ECONOMETRICS
X X y P y
(11) R2 = = .
y y y y
The measure is just the square of the cosine of the angle between the vectors
and the inequality 0 R2 1 follows from the fact that the
y and P y = X ;
cosine of any angle must lie between 1 and +1.
From (13), we get the equation X1 X1 1 = X1 (y X2 2 ) which gives an ex-
pression for the leading subvector of :
On dening
4
REGRESSION ANALYSIS IN MATRIX ALGEBRA
whence
1
(20) 2 = X2 (I P1 )X2 X2 (I P1 )y.
(21) yt = + xt. + t , t = 1, . . . , T,
(22) y = + Z + .
Here, the vector = [1, 1, . . . , 1] , which consists of T units, is described alter-
natively as the dummy vector or the summation vector, whilst Z = [xtj ; t =
1, . . . T ; j = 1, . . . , k] is the matrix of the observations on the explanatory vari-
ables.
Equation (22) can be construed as a case of the partitioned regression
equation of (12). By setting X1 = and X2 = Z and by taking 1 = ,
2 = z in equations (15) and (20), we derive the following expressions for the
estimates of the parameters , z :
(23) = ( )1 (y Z z ),
1
z = Z (I P )Z Z (I P )y, with
(24) 1
P = ( )1 = .
T
To understand the eect of the operator P in this context, consider the follow-
ing expressions:
T
y= yt ,
t=1
1
(25) T
1
( ) y= yt = y,
T t=1
y = ( )1 y = [
P y = y , y, . . . , y] .
5
D.S.G. POLLOCK: ECONOMETRICS
T
T
T
(26) x (I P )x = xt (xt x
) = (xt x
)xt = (xt x
)2 .
t=1 t=1 t=1
The nal equality depends on the fact that (xt x
) (xt x
x=x ) = 0.
(29) y z + ( ).
y = [Z Z]
= y Z z
(30)
k
= y j j .
x
j=1
6
REGRESSION ANALYSIS IN MATRIX ALGEBRA
(31) y = 0 + 1 x1 + + k xk + ,
which is equation (1) again; and imagine, as before, that there are T observa-
tions on the variables. Then, these can be arrayed in the matrix form of (2)
for which the summary notation is
(32) y = X + ,
Next it is assumed that the disturbances are mutually uncorrelated and that
they have a common variance. Thus
(34)
2 , if t = s;
D() = E( ) = 2 I or, equivalently, E(t s ) =
0, if t = s.
7
D.S.G. POLLOCK: ECONOMETRICS
time will achieve the eect. The danger of miss-attributing the eects of one
variable to another is then minimised.
In an econometric context, it is often more appropriate to regard the ele-
ments of X as random variables in their own right, albeit that we are usually
reluctant to specify in detail the nature of the processes that generate the
variables. Thus, it may be declared that
In fact, for present purposes, it makes little dierence which of these assump-
tions regarding X is adopted; and, since the assumption under (35) is more
briey expressed, we shall adopt it in preference.
The rst property to be deduced from the assumptions is that
= (X X)1 X y
(39) = (X X)1 X (X + )
= + (X X)1 X .
= + (X X)1 X E()
E()
(40)
= .
Notice that, in the light of this result, equation (39) now indicates that
(41) = (X X)1 X .
E()
8
REGRESSION ANALYSIS IN MATRIX ALGEBRA
Now,
n
m
Trace(AB) = aij bji and
i=1 j=1
(46)
m
n
n
m
Trace(BA) = bj aj = aj bj .
j=1 =1 =1 j=1
But, apart from a minor change of notation, where replaces i, the expressions
on the RHS are the same. It follows that Trace(AB) = Trace(BA). The
result can be extended to cover the cyclic permutation of any number of matrix
factors. In the case of three factors A, B, C, we have
9
D.S.G. POLLOCK: ECONOMETRICS
1 2
T
2
=
e
T k t=1 t
(48)
1 (y X ).
= (y X )
T k
T
(50) E( ) = E(e2t ) = T 2 .
t=1
The nal equality follows from the fact that Trace(P ) = Trace(Ik ) = k. Putting
the results of (50) and (51) into (49), gives
(52) (y X )
E (y X ) = 2 (T k);
and, from this, the unbiasedness of the estimator in (48) follows directly.
10
REGRESSION ANALYSIS IN MATRIX ALGEBRA
= (X X)1 X (X + )
(53)
= + (X X)1 X .
The expectation is
= + (X X)1 X E()
E()
(54)
= .
D( ) = AD(y)A
= 2 (X X)1 X + G X(X X)1 + G
(57)
= 2 (X X)1 + 2 GG
= D() + 2 GG .
11
D.S.G. POLLOCK: ECONOMETRICS
V (q ) = q D()q
+ 2 q GG q
(58)
q D()q
= V (q );
We may assume that the variables in this equation are in deviation form. Let
us imagine that the columns of X1 are orthogonal to the columns of X2 such
that X1 X2 = 0. This is the same as imagining that the empirical correlation
between variables in X1 and variables in X2 is zero.
To see the eect upon the ordinary least-squares estimator, we may exam-
ine the partitioned form of the formula = (X X)1 X y. Here, there is
X1 X1 X1 X1 X2 X1 X1 0
(60) XX= [ X1 X2 ] = = ,
X2 X2 X1 X2 X2 0 X2 X2
where the nal equality follows from the condition of orthogonality. The inverse
of the partitioned form of X X in the case of X1 X2 = 0 is
1
1 X1 X1 0 (X1 X1 )1 0
(61) (X X) = = .
0 X2 X2 0 (X2 X2 )1
Ther is also
X1 X1 y
(62) Xy= y= .
X2 X2 y
12
REGRESSION ANALYSIS IN MATRIX ALGEBRA
1 = (X1 X1 )1 X1 (y X2 2 ),
(64) 1
2 = X2 (I P1 )X2 X2 (I P1 )y, P1 = X1 (X1 X1 )1 X1 .
1 = (X1 X1 )1 X1 y
(65) = (X1 X1 )1 X1 (X1 1 + X2 2 + )
= 1 + (X1 X1 )1 X1 X2 2 + (X1 X1 )1 X1 .
13
D.S.G. POLLOCK: ECONOMETRICS
L = (y X) (y X) + 2 (R r)
(67)
= y y 2y X + X X + 2 R 2 r.
On dierentiating L with respect to and setting the result to zero, we get the
following rst-order condition L/ = 0:
(68) 2 X X 2y X + 2 R = 0,
whence, after transposing the expression, eliminating the factor 2 and rearrang-
ing, we have
(69) X X + R = X y.
When these equations are compounded with the equations of the restrictions,
which are supplied by the condition L/ = 0, we get the following system:
X X R Xy
(70) = .
R 0 r
For the system to have a unique solution, that is to say, for the existence of an
estimate of , it is not necessary that the matrix X X should be invertibleit
is enough that the condition
X
(71) Rank =k
R
14
REGRESSION ANALYSIS IN MATRIX ALGEBRA
should hold, which means that the matrix should have full column rank. The
nature of this condition can be understood by considering the possibility of
estimating by applying ordinary least-squares regression to the equation
y X
(72) = + ,
r R 0
which puts the equations of the observations and the equations of the restric-
tions on an equal footing. It is clear that an estimator exits on the condition
that (X X + R R)1 exists, for which the satisfaction of the rank condition is
necessary and sucient.
Let us simplify matters by assuming that (X X)1 does exist. Then equa-
tion (68) gives an expression for in the form of
= (X X)1 X y (X X)1 R
(73)
= (X X)1 R ,
from which
15
D.S.G. POLLOCK: ECONOMETRICS
where
The expression comes from taking from both sides of (76) and from recognis-
ing that R r = R( ). It can be seen that PR is an idempotent matrix
that is subject to the conditions that
D( ) = (I PR )E{( )( ) }(I PR )
(80) = 2 (I PR )(X X)1 (I PR )
= 2 [(X X)1 (X X)1 R {R(X X)1 R }1 R(X X)1 ].
[T /2]
(81) yt = 0 + {j cos(j t) + sin(j t)} ; t = 0, 1, . . . , T 1.
j=1
16
REGRESSION ANALYSIS IN MATRIX ALGEBRA
where ctj = cos(j t) and stj = sin(j t). The vectors of the ordinates of
functions of dierent frequencies are mutually orthogonal. Therefore, amongst
these vectors, the following orthogonality conditions hold:
ci cj = si sj = 0 if i = j,
(84)
and ci sj = 0 for all i, j.
In addition, there are some sums of squares which can be taken into account
in computing the coecients of the Fourier decomposition:
c0 c0 = = T, s0 s0 = 0,
(85)
cj cj = sj sj = T /2 for j = 1, . . . , [(T 1)/2]
Th proofs ar given in a brief appendix at the end of this section. When T = 2n,
there is n = and, therefore, in view of (82), there is also
The regression formulae for the Fourier coecients can now be given.
First, there is
1
(87) 0 = (i i)1 i y = yt = y.
T t
17
D.S.G. POLLOCK: ECONOMETRICS
2
(88) j = (cj cj )1 cj y = yt cos j t,
T t
and
2
(89) j = (sj sj )1 sj y = yt sin j t.
T t
1
(90) n = (cn cn )1 cn y = (1)t yt .
T t
[T /2]
(91) yy= 02 + j2 cj cj + j2 sj sj .
j=1
T 2 T 2
n1
n
(92) (y y) (y y) = j + j2 + T n2 = .
2 j=1 2 j=1 j
18
REGRESSION ANALYSIS IN MATRIX ALGEBRA
5.4
5.2
4.8
0 25 50 75 100 125
Figure 9. The plot of 132 monthly observations on the U.S. money supply,
beginning in January 1960. A quadratic function has been interpolated
through the data.
0.015
0.01
0.005
0
0 /4 /2 3/4
Figure 10. The periodogram of the residuals of the logarithmic money-
supply data.
19
D.S.G. POLLOCK: ECONOMETRICS
This provides a simple way of characterising the growth of the money supply
over the period in question. The pattern of seasonal uctuations is remarkably
regular, as can be see from the residuals from the quadratic detrending.
The peridogram of the residual sequence is shown in Figure 10. This has a
prominent spike at the frequency value of /6 radians or 30 degrees per month,
which is the fundamental seasonal frequency. Smaller spikes are seen at 60, 90,
120, and 150 degrees, which are the harmonics of the fundamental frequency.
Their presence reects the fact that the pattern of the seasonal uctuations is
more complicated than that of a simple sinusoidal uctuation at the seasonal
frequency.
The peridodogram also shows a signicant spectral mass within the fre-
quency range [0/6]. This mass properly belongs to the trend; and, if the
trend had been adequately estimated, then its eect would not be present in
the residual, which would then show even greater regularity. In lecture 9, we
will show how a more tting trend function can be estimated.
T 1
T 1
cos(j t) = sin(j t) = 0.
t=0 t=0
Proof. We have
1 T 1
1
T
cos(j t) = {exp(ij t) + exp(ij t)}
t=0
2 t=0
T 1 T 1
1 1
= exp(i2jt/T ) + exp(i2jt/T ).
2 t=0 2 t=0
T 1
1 exp(i2j)
exp(i2jt/T ) = .
t=0
1 exp(i2j/T )
20
REGRESSION ANALYSIS IN MATRIX ALGEBRA
By similar means,
it can be show that t exp(i2j/T ) = 0; and, therefore,
it follows that t cos(j t) = 0.
An analogous proof shows that t sin(j t) = 0.
T 1
(a) cos(j t) cos(k t) = 0 ifj = k,
t=0
T 1
cos2 (j t) = T /2,
t=0
T 1
(b) sin(j t) sin(k t) = 0 ifj = k,
t=0
T 1
sin2 (j t) = T /2,
t=0
T 1
(c) cos(j t) sin(k t) = 0 ifj = k.
t=0
Proof. From the formula cos A cos B = 12 {cos(A + B) + cos(A B)}, we have
1
1
T
cos(j t) cos(k t) = {cos([j + k ]t) + cos([j k ]t)}
t=0
2
T 1
1
= {cos(2[j + k]t/T ) + cos(2[j k]t/T )} .
2 t=0
21