You are on page 1of 43

# Chapter 5: Matrix approach to simple linear regression

analysis
You need to understand matrix algebra for multiple
regression! Fox’s Section 8! contains information about
ho" to use # for matrix algebra
5\$ Matrices
%hat is a matrix&
'( matrix is a rectangular array of elements arranged in
ro"s and columns) *p \$+, of -../
0xample:
\$ ! 1
2 5 ,
1
1
¸ ]
3imension 4 Si5e of matrix: 6 ro"s × 6 columns 7 r×c
0xample: !×1
Symbolic representation of a matrix:
0xample:
\$\$ \$! \$1
!\$ !! !1
a a a
a a a
1
·
1
¸ ]
A

5\$
"here a
i:
is the ro" i and column : element of A
a
\$\$
7\$ from the abo;e example
.otice that the matrix A is in bold %hen bolding is not
possible *"riting on a piece of paper or chal<board/= the
letter is underlined > (
a
\$\$
is often called the '*\$=\$/ element) of A= a
\$!
is called
the '*\$=!/ element) of A=?
0xample: r×c matrix
\$\$ \$! \$: \$c
!\$ !! !: !c
i\$ i! i: ic
r\$ r ! r: rc
a a a a
a a a a
a a a a
a a a a
1
1
1
1
·
1
1
1
1
1
¸ ]
A
L L
L L
M M M M
L L
M M M M
L L
0xample: S@uare matrix is r×c "here r7c
0xample: AS and College BC(
5!
\$ 182
\$ !15
\$ !+8
\$ !!8
\$ \$88
1
1
1
1
·
1
1
1
1
1
¸ ]
X
M M
Dhe abo;e !8×! matrix contains the AS BC(s in the
second column
Eector 4 a r×\$ *column ;ector/ or \$×c *ro" ;ector/ matrix 4
special case of a matrix
0xample: Symbolic representation of a 1×\$ column ;ector
\$
!
1
a
a
a
1
1
·
1
1
¸ ]
A
0xample: AS and College BC(
1\$8
!18
188
!!8
\$,8
1
1
1
1
·
1
1
1
1
1
¸ ]
Y
M
51
Dhe abo;e !8×\$ ;ector contains the College BC(s
Dranspose: Fnterchange the ro"s and columns of a matrix or
;ector
0xample:
\$\$ \$! \$1
!\$ !! !1
a a a
a a a
1
·
1
¸ ]
A
and
\$\$ !\$
\$! !!
\$1 !1
a a
a a
a a
1
1

·
1
1
¸ ]
A
A is !×1 and A′ is 1×!
Dhe ′ symbol indicates a transpose= and it is said as the
"ord 'prime) Dhus= the transpose of A is '( prime)
0xample: AS and College BC(
[ ]
1\$8 !18 188 !!8 \$,8

· Y L
52
(dd or subtract the corresponding elements of matrices
"ith the same dimension
0xample:
Suppose
\$ ! 1 \$ \$8 \$
and
2 5 , 5 5 8
− −
1 1
· ·
1 1
¸ ] ¸ ]
A B
Dhen
8 \$! ! ! 8 2
and
G \$8 \$2 \$ 8 !

1 1
+ · − ·
1 1
− −
¸ ] ¸ ]
A B A B

0xample: Hsing # *basicImatrixIalgebra#/
> A<-matrix(data = c(1, 2, 3,
4, 5, 6), nrow = 2, ncol = 3, byrow =
TR!)
> cla""(A)
#1\$ %matrix%
> &<-matrix(data = c(-1, 1', -1, 5, 5, (), nrow = 2, ncol =
3, byrow = TR!)

> A)&
#,1\$ #,2\$ #,3\$
#1,\$ ' 12 2
#2,\$ * 1' 14
> A-&
#,1\$ #,2\$ #,3\$
#1,\$ 2 -( 4
#2,\$ -1 ' -2
55
.otes:
\$9e careful "ith the byrow option 9y default= this is set
to F(JS0 Dhus= the numbers "ould be entered into
the matrix by columns For example=
> matrix(data = c(1, 2, 3, 4, 5, 6), nrow = 2, ncol = 3)
#,1\$ #,2\$ #,3\$
#1,\$ 1 3 5
#2,\$ 2 4 6
!Dhe class of these ob:ects is 'matrix)
1( ;ector can be represented as a 'matrix) class type or
a type of its o"n
> y<-matrix(data = c(1,2,3), nrow = 3, ncol = 1, byrow
= TR!)
> y
#,1\$
#1,\$ 1
#2,\$ 2
#3,\$ 3
> cla""(y)
#1\$ %matrix%
> x<-c(1,2,3)
> x
#1\$ 1 2 3
> cla""(x)
#1\$ %n+m,ric%
> i"-.,ctor(x)
#1\$ TR!
Dhis can present some confusion "hen ;ectors are
multiplied "ith other ;ectors or matrices because no
specific ro" or column dimensions are gi;en More on
this shortly
5,
2( transpose of a matrix can be done using the t()
function For example=
> t(A)
#,1\$ #,2\$
#1,\$ 1 4
#2,\$ 2 5
#3,\$ 3 6
0xample: Simple linear regression model
Y
i
70*Y
i
/ K ε
i
for i7\$=?=n can be represented as
0* / · + Y Y ε
"here
\$ \$ \$
! ! !
n n n
Y 0*Y /
Y 0*Y /
= 0* / = and
Y 0*Y /
ε
1 1 1
1 1 1
ε
1 1 1
· · ·
1 1 1
1 1 1
ε
¸ ] ¸ ] ¸ ]
M M M
Y Y ε
5+
51 Matrix multiplication
Scalar > \$×\$ matrix
0xample: Matrix multiplied by a scalar
\$\$ \$! \$1
!\$ !! !1
ca ca ca
c
ca ca ca
1
·
1
¸ ]
A
"here c is a scalar
Jet
\$ ! 1
2 5 ,
1
·
1
¸ ]
A
and c7! Dhen
! 2 ,
!
8 \$8 \$!
1
·
1
¸ ]
A

Multiplying t"o matrices
Suppose you "ant to multiply the matrices A and BL ie=
A∗B or AB Fn order to do this= you need the number of
columns of A to be the same as the number of ro"s as
B For example= suppose A is !×1 and B is 1×\$8 You
can multiply these matrices Ao"e;er if B is 2×\$8
instead= these matrices could .MD be multiplied
Dhe resulting dimension of C7AB
\$Dhe number of ro"s of A is the number of ro"s of C
!Dhe number of columns of B is the number of ro"s of
C
1Fn other "ords=
" y " 5 5 y × × ×
· C A B
"here the dimension of the
matrices are sho"n belo" them
58
Ao" to multiply t"o matrices 4 an example
Suppose
1 8
\$ ! 1
and \$ !
2 5 ,
8 \$
1
1
1
· ·
1
1
¸ ]
1
¸ ]
A B
.otice that A
is !×1 and B is 1×! so C7AB can be done
1 8
\$ ! 1
\$ !
2 5 ,
8 \$
\$ 1 ! \$ 1 8 \$ 8 ! ! 1 \$
2 1 5 \$ , 8 2 8 5 ! , \$
5 +
\$+ \$,
1
1
1
· ·
1
1
¸ ]
1
¸ ]
∗ + ∗ + ∗ ∗ + ∗ + ∗
1
·
1
∗ + ∗ + ∗ ∗ + ∗ + ∗
¸ ]
1
·
1
¸ ]
C AB
Dhe 'cross product) of the ro"s of A and the columns of
B are ta<en to form C
Fn the abo;e example= D7BA≠AB "here BA is:
5G
1 8
\$ ! 1
\$ !
2 5 ,
8 \$
1 \$ 8 2 1 ! 8 5 1 1 8 ,
\$ \$ ! 2 \$ ! ! 5 \$ 1 ! ,
8 \$ \$ 2 8 ! \$ 5 8 1 \$ ,
1 , G
G \$! \$5
2 5 ,
1
1
1
·
1
1
¸ ]
1
¸ ]
∗ + ∗ ∗ + ∗ ∗ + ∗
1
1
· ∗ + ∗ ∗ + ∗ ∗ + ∗
1
1 ∗ + ∗ ∗ + ∗ ∗ + ∗
¸ ]
1
1
·
1
1
¸ ]
BA
Fn general for a !×1 matrix times a 1×! matrix:
\$\$ \$!
\$\$ \$! \$1
!\$ !!
!\$ !! !1
1\$ 1!
\$\$ \$\$ \$! !\$ \$1 1\$ \$\$ \$! \$! !! \$1 1!
!\$ \$\$ !! !\$ !1 1\$ !\$ \$! !! !! !1 1!
b b
a a a
b b
a a a
b b
a b a b a b a b a b a b
a b a b a b a b a b a b
1
1
1
· ·
1
1
¸ ]
1
¸ ]
+ + + +
1
·
1
+ + + +
¸ ]
C AB
0xample: Hsing # *basicImatrixIalgebra#/
> A<-matrix(data = c(1, 2, 3, 4, 5, 6), nrow = 2, ncol =
3, byrow = TR!)
> &<-matrix(data = c(3, ', 1, 2, ', 1), nrow = 3, ncol =
2, byrow = TR!)

5\$8
> /<-A010&
> 2<-&010A

> /
#,1\$ #,2\$
#1,\$ 5 3
#2,\$ 13 16
> 2
#,1\$ #,2\$ #,3\$
#1,\$ 3 6 *
#2,\$ * 12 15
#3,\$ 4 5 6

> 456at i" A1&7
> A1&
!rror in A 1 & 8 non-con9ormabl, array"
.otes:
\$NON is used for multiplying matrices andPor ;ectors
!O means to perform element"ise multiplications Aere
is an example "here this can be done:
> !<-A
> A1!
#,1\$ #,2\$ #,3\$
#1,\$ 1 4 *
#2,\$ 16 25 36
Dhe *i=i/ elements of each matrix are multiplied
together
1Multiplying ;ectors "ith other ;ectors or matrices can
be confusing since no ro" or column dimensions are
5\$\$
gi;en for a ;ector ob:ect For example= suppose x 7
\$
!
1
1
1
1
1
¸ ]
= a 1×\$ ;ector
> x<-c(1,2,3)
> x010x
#,1\$
#1,\$ 14
> A010x
#,1\$
#1,\$ 14
#2,\$ 32
Ao" does # <no" that "e "ant x′x *\$×\$/ instead of
xx′ *1×1/ "hen "e ha;e not told # that x is 1×\$&
Similarly= ho" does # <no" that Ax is !×\$& From the
# help for NON in the 9ase pac<age:
Multiplies t"o matrices= if they are conformable Ff
one argument is a ;ector= it "ill be promoted to
either a ro" or column matrix to ma<e the t"o
arguments conformable Ff both are ;ectors it "ill
return the inner product
(n inner product produces a scalar ;alue Ff you
"anted xx′ *1×1/= one can use the outer product NoN
> x0o0x 4o+t,r :rod+ct
#,1\$ #,2\$ #,3\$
#1,\$ 1 2 3
#2,\$ 2 4 6
5\$!
#3,\$ 3 6 *
%e "ill only need to use NON in this class
0xample: AS and College BC( *ASIcollegeIBC(Ich5#/
\$ 182
\$ !15
\$ !+8
\$ !!8
\$ \$88
1
1
1
1
·
1
1
1
1
1
¸ ]
X
M M
and
1\$8
!18
188
!!8
\$,8
1
1
1
1
·
1
1
1
1
1
¸ ]
Y
M
Find X′X= X′Y= and Y′Y
%/8<<c6ri"<<=><<?TAT(3'<</6a:t,r1<<;:a-txt%,
@?-ABA /oll,;,-ABA
1 3-'4 3-1
2 2-35 2-3
3 2-3' 3-'
4 2-'5 1-*
5 2-(3 2-5
6 4-32 3-3

> C
5\$1
#,1\$ #,2\$
#1,\$ 1 3-'4
#2,\$ 1 2-35
#3,\$ 1 2-3'
#4,\$ 1 2-'5
#5,\$ 1 2-(3
#6,\$ 1 4-32
#3,\$ 1 3-3*
#(,\$ 1 2-32
#*,\$ 1 2-6*
#1',\$ 1 '-(3
#11,\$ 1 2-3*
#12,\$ 1 3-65
#13,\$ 1 1-(5
#14,\$ 1 3-(3
#15,\$ 1 1-22
#16,\$ 1 1-4(
#13,\$ 1 2-2(
#1(,\$ 1 4-''
#1*,\$ 1 2-2(
#2',\$ 1 1-((
> E
#1\$ 3-1 2-3 3-' 1-* 2-5 3-3 3-4 2-6 2-( 1-6 2-' 2-* 2-3
3-2 1-( 1-4 2-' 3-( 2-2
#2'\$ 1-6

> t(C)010C
#,1\$ #,2\$
#1,\$ 2'-'' 51-3(''
#2,\$ 51-3( 14(-4634
> t(C)010E
#,1\$
#1,\$ 5'-1''
#2,\$ 14'-22*
> t(E)010E
#,1\$
#1,\$ 135-15

.otes:
5\$2
\$Dhe cbind() function combines items by 'c)olumns
Since \$ is only one element= it "ill replicate itself for
all elements that you are combining so that one full
matrix is formed Dhere is also a rbind() function that
combines by ro"s Dhus= rbind(a,b) forms a matrix
"ith a abo;e b
! [ ]
\$
n
!
!
\$ ! n i
i \$
n
y
y
y = y == y y
y
·
1
1
1

· ·

1
1
¸ ]
Y Y
M
1
\$
n n
i\$ i i
\$\$ !\$ n\$ !
i \$ i \$
n n
\$! !! n!
i! i i! i
i \$ i \$
n
y
x y y
x x x y
x x x
x y x y
y
· ·
· ·
1
1 1
1 ∑ ∑
1 1 1
1

· · ·
1 1 1
1
¸ ]
1 1
∑ ∑
1
¸ ] ¸ ]
¸ ]
X Y
L
L M

since x
\$\$
7?7x
n\$
7\$
2X′X
\$\$ \$!
\$\$ !\$ n\$ !\$ !!
\$! !! n!
n\$ n!
\$!
n
i!
!!
i \$
n n
!
\$! !! n!
i! i!
i \$ i \$
n!
x x
x x x x x
x x x
x x
\$ x
n x
\$ \$ \$ \$ x
x x x
x x
\$ x
·
· ·
1
1
1
1
·
1
1
¸ ]
1
¸ ]
1
1
1 ∑
1 1
1
· ·
1 1
1
¸ ]
1
∑ ∑
1
¸ ]
¸ ]
L
L M M
L
L M M
5\$5
5Aere’s another "ay to get the X matrix
> mod-9it<-lm(9orm+la = /oll,;,-ABA F @?-ABA, data =
;:a)
> mod,l-matrix(obG,ct = mod-9it)
(Hnt,rc,:t) @?-ABA
1 1 3-'4
2 1 2-35
3 1 2-3'
4 1 2-'5
5 1 2-(3
6 1 4-32
3 1 3-3*
( 1 2-32
* 1 2-6*
1' 1 '-(3
11 1 2-3*
12 1 3-65
13 1 1-(5
14 1 3-(3
15 1 1-22
16 1 1-4(
13 1 2-2(
1( 1 4-''
1* 1 2-2(
2' 1 1-((
attr(,%a""i;n%)
#1\$ ' 1
5\$,
52 Special types of matrices
Symmetric matrix: Ff A7A′= then A is symmetric
0xample:
\$ ! \$ !
=
! 1 ! 1
1 1

· ·
1 1
¸ ] ¸ ]
A A
3iagonal matrix: ( s@uare matrix "hose 'off>diagonal)
elements are 8
0xample:
\$\$
!!
11
a 8 8
8 a 8
8 8 a
1
1
·
1
1
¸ ]
A
Fdentity matrix: ( diagonal matrix "ith \$’s on the diagonal
0xample:
\$ 8 8
8 \$ 8
8 8 \$
1
1
·
1
1
¸ ]
I
.ote that 'I) *the letter I= not the number one/ usually
denotes the identity matrix
Eector and matrix of \$’s
( column ;ector of \$’s:
r \$
r \$
\$
\$
\$
×
×
1
1
1
· ·
1
1
¸ ]
1 j
M
5\$+
( matrix of \$’s:
r r
\$ \$ \$
\$ \$ \$
\$ \$ \$
×
1
1
1
·
1
1
¸ ]
J
L
L
M M O M
L
.otes:
\$
r \$ r \$
r
× ×

· j j
!
r r
r \$r \$
×
× ×

· j j J
1
r r r r r r
r
× × ×

· J J J
Eector of 8’s:
r \$
8
8
8
×
1
1
1
·
1
1
¸ ]
0
M

5\$8
55 Jinear dependence and ran< of matrix
Jet
\$ ! ,
1 2 \$!
5 , \$8
1
1
·
1
1
¸ ]
A
Dhin< of each column of A as a
;ectorL ie= A 7 QA
\$
= A
!
= A
1
R .ote that 1A
!
7A
1
Dhis
means the columns of A are 'linearly dependent)
Formally= a set of column ;ectors are linearly dependent
if there exists constants λ
\$
= λ
!
=?= λ
n
*not all 5ero/ such
that λ
\$
A
\$

!
A
!
K?Kλ
c
A
c
70 ( set of column ;ectors are
linearly independent if λ
\$
A
\$

!
A
!
K?Kλ
c
A
c
70 only for
λ
\$

!
7?7λ
c
70
Dhe ran< of a matrix is the maximum number of linearly
independent columns in the matrix
ran<*A/7!
5\$G
5, Fn;erse of a matrix
.ote that the in;erse of a scalar= say b= is b
>\$
For
example= the in;erse of b71 is 1
>\$
7\$P1 (lso= b∗b
>\$
7\$ Fn
matrix algebra= the in;erse of a matrix is another matrix
For example= the in;erse of A is A
>\$
= and AA
>\$
7A
>\$
A7I
.ote that A must be a s@uare matrix
0xample:
\$
\$ ! ! \$
=
1 2 \$5 85

1 1
· ·
1 1

¸ ] ¸ ]
A A
Chec<:
\$
\$ * !/ ! \$5 \$ \$ ! O * 85/ \$ 8
1 * !/ 2 \$5 1 \$ 2 O * 85/ 8 \$

∗ − + ∗ ∗ + −
1 1
· ·
1 1
∗ − + ∗ ∗ + −
¸ ] ¸ ]
AA
Finding the in;erse
For a general "ay= see a matrix algebra boo< *such as:
-olman= \$G88/ For a !×! matrix= there is a simple
formula Jet
\$\$ \$!
!\$ !!
a a
a a
1
·
1
¸ ]
A
Dhen
!! \$!
\$
!\$ \$\$ \$\$ !! \$! !\$
a a
\$
a a a a a a

1
·
1
− −
¸ ]
A

Eerify AA
>\$
0xample: Hsing # *basicImatrixIalgebra#/
> A<-matrix(data = c(1, 2, 3, 4), nrow = 2, ncol = 2,
byrow = TR!)
5!8
> "ol.,(A)
#,1\$ #,2\$
#1,\$ -2-' 1-'
#2,\$ 1-5 -'-5
> A010"ol.,(A)
#,1\$ #,2\$
#1,\$ 1 1-11'223,-16
#2,\$ ' 1-'''''',)''
> "ol.,(A)010A
#,1\$ #,2\$
#1,\$ 1-'''''',)'' '
#2,\$ 1-11'223,-16 1
> ro+nd("ol.,(A)010A, 2)
#,1\$ #,2\$
#1,\$ 1 '
#2,\$ ' 1
Dhe solve() function in;erts a matrix in # Dhe
solve(A,b) function can also be used to 'sol;e) for x in
Ax 7 b since A
>\$
Ax 7 A
>\$
b ⇒ Ix 7 A
>\$
b ⇒ x 7 A
>\$
b
0xample: AS and College BC( *ASIcollegeIBC(Ich5#/
#emember that
\$ 182
\$ !15
\$ !+8
\$ !!8
\$ \$88
1
1
1
1
·
1
1
1
1
1
¸ ]
X
M M
and
1\$8
!18
188
!!8
\$,8
1
1
1
1
·
1
1
1
1
1
¸ ]
Y
M
Find *X′X/
>\$
and *X′X/
>\$
X′Y
5!\$
> "ol.,(t(C)010C)
#,1\$ #,2\$
#1,\$ '-45'35(4 -'-155**3(1
#2,\$ -'-155**3( '-'6'32316
> "ol.,(t(C)010C) 010 t(C)010E
#,1\$
#1,\$ '-3'35336
#2,\$ '-6**65(4
From pre;ious output:
> mod-9it<-lm(9orm+la = /oll,;,-ABA F @?-ABA, data = ;:a)
> mod-9itDco,99ici,nt"
(Hnt,rc,:t) @?-ABA
'-3'35336 '-6**65(4
.ote that *X′X/
>\$
X′Y 7
8
\$
b
b
1
1
¸ ]
!!!
> 4Anot6,r way to ;,t (CIC)J(-1)
> "+m-9it<-"+mmary(mod-9it)
> "+m-9itDco.-+n"cal,d
(Hnt,rc,:t) @?-ABA
(Hnt,rc,:t) '-45'35(4 -'-155**3(1
@?-ABA -'-155**3( '-'6'32316
#ead 'Hses of Fn;erse Matrix) on p \$G! of -.. on your
o"n
5!!
5+ Some basic theorems of matrices
58 #andom ;ectors and matrices
( random ;ector or random matrix contains elements that
are random ;ariables
0xample: Simple linear regression model
Y
i
70*Y
i
/ K ε
i
for i7\$=?=n can be represented as
0* / · + Y Y ε
"here
\$ \$
! !
n n
Y
Y
and
Y
ε
1 1
1 1
ε
1 1
· ·
1 1
1 1
ε
¸ ] ¸ ]
M M
Y ε
are random ;ectors
0xpectation of random ;ector or matrix: Find the expected
;alue of the indi;idual elements
0xample:
\$ \$ 8 \$ \$
! ! 8 \$ !
n n 8 \$ n
Y 0*Y / S
Y 0*Y / S
0* / 0
Y 0*Y / S
¸ _ β + β
1 1 1

1 1 1
β + β

1 1 1
· · ·

1 1 1

1 1 1

β + β
¸ ] ¸ ] ¸ ]
¸ ,
Y
M M M
5!1
\$ \$
! !
n n
0* / 8
0* / 8
0* / 0
0* / 8
¸ _ ε ε
1 1 1

1 1 1
ε ε

1 1 1
· · ·

1 1 1

1 1 1

ε ε
¸ ] ¸ ] ¸ ]
¸ ,
ε
M M M
since "e assume
ε
i
T.*8=σ
!
/
Eariance>co;ariance matrix of a random ;ector
Jet U
\$
and U
!
be random ;ariables #emember that
Ear*U
\$
/70*U
\$

\$
/
!
"here 0*U
\$
/7µ
\$

Dhe co;ariance of U
\$
and U
!
is defined as Co;*U
\$
=U
!
/ 7
0Q*U
\$

\$
/*U
!

!
/R Dhe co;ariance measures the
relationship bet"een U
\$
and U
!
See p 2!, of my
Chapter 2 SD(D 188 notes for a more mathematical
explanation *http:PP"""chrisbildercomP
stat188Pschedulehtm/ .ote that the correlation
bet"een U
\$
and U
!
is
\$ !
\$ !
\$ !
Co;*U =U /
Corr*U =U /
Ear*U / Ear*U /
·

Dhe 'Cearson correlation coefficient) "hich is denoted by
r and estimates Corr*U
\$
= U
!
/
5!2
#emember that
n
i i
i \$
n n
! !
i i
i \$ i \$
*S S/*Y Y/
r
*S S/ *Y Y/
·
· ·
− −

·
1 1
− −
∑ ∑
1 1
¸ ] ¸ ]

"here S7U
\$
and Y7U
!

.otes:
 Co;*U
\$
=U
\$
/70Q*U
\$

\$
/*U
\$

\$
/R70Q*U
\$

\$
/
!
R7Ear*U
\$
/
 Ff U
\$
and U
!
are independent= then Co;*U
\$
=U
!
/78
( ;ariance>co;ariance matrix *most often :ust called the
co;ariance matrix/ is a matrix "hose elements are the
;ariances and co;ariances of random ;ariables
0xample: Jet
\$
!
U
U
1
·
1
¸ ]
Z
Dhe co;ariance matrix of Z is
Co;*Z/ 7
\$ \$ !
\$ ! !
Ear*U / Co;*U =U /
Co;*U =U / Ear*U /
1
1
¸ ]
.ote that Co;*U
\$
=U
!
/7Co;*U
!
=U
\$
/ and Co;*U
i
=U
i
/7Ear*U
i
/
-.. denote Co;*Z/ by σ
!
VZW Dhe notation that F am
using is much more pre;alent and there is less chance
for confusion *li<e Z being multiplied by σ
!
in σ
!
VZW/
0xample: Simple linear regression model
5!5
\$
!
n
Ear* / 8 8
8 Ear* / 8
Co;* /
8 8 Ear* /
ε
1
1
ε
1
·
1
1
ε
¸ ]
ε
L
L
M M O M
L
!
!
!
8 8
8 8

8 8
1
σ
1
σ
1
·
1
1
σ
¸ ]
L
L
M M O M
L
7 σ
!
I
since Co;*ε
i

i′
/78 *#emember that ε
i
T F.30C0.30.D
.*8=σ
!
//
%hat is Co;*Y/&
.ote that all co;ariance matrices are symmetric!
Some basic results
Jet W 7 AY "here Y is a random ;ector and A is a
matrix of constants *no random ;ariables in it/
Dhe follo"ing results follo":
\$0*A/ 7 A
5!,
#emember this is li<e saying 0*1/ 7 1
!0*W/ 7 0*AY/ 7 A0*Y/
(gain= this is li<e saying 0*1Y
\$
/710*Y
\$
/
1Co;*W/ 7 Co;*AY/ 7 ACo;*Y/A′
You may ha;e seen before that Ear*aY
\$
/ 7 a
!
Ear*Y
\$
/
"here a is a constant
0xample:
Jet
\$
!
\$ \$ Y
and
\$ 8 Y

1 1
· ·
1 1
¸ ] ¸ ]
A Y
Dhen
\$ !
\$
Y Y
Y

1
·
1
¸ ]
W

!

\$ \$
! !
\$ !
\$
\$ \$ Y \$ \$ Y
0* / 0 0
\$ 8 Y \$ 8 Y
0*Y / 0*Y /
0*Y /
¸ _ ¸ _ − −
1 1 1 1
· ·

1 1 1 1
¸ ] ¸ ] ¸ ] ¸ ]
¸ , ¸ ,

1
·
1
¸ ]
W
1
5!+
\$ \$
! !
\$ \$ !
\$ ! !
\$ \$ ! \$ ! !
\$ \$ !
\$ \$ Y \$ \$ Y \$ \$
Co;* / Co; Co;
\$ 8 Y \$ 8 Y \$ 8
\$ \$ Ear*Y / Co;*Y= Y / \$ \$
\$ 8 Co;*Y= Y / Ear*Y / \$ 8
Ear*Y / Co;*Y= Y / Co;*Y= Y / Ear*Y /
Ear*Y / Co;*Y= Y /

¸ _ ¸ _ − − − 1 1 1 1 1
· ·
÷ ÷
1 1 1 1 1
¸ ] ¸ ] ¸ ] ¸ ] ¸ ]
¸ , ¸ ,
− 1 1 1
·
1 1 1

¸ ] ¸ ] ¸ ]
− −
·

¸
W
\$ ! \$ ! \$ \$ !
\$ \$ ! \$
\$ \$
\$ 8
Ear*Y / Ear*Y / !Co;*Y= Y / Ear*Y / Co;*Y= Y /
Ear*Y / Co;*Y= Y / Ear*Y /
1 1
1 1

] ¸ ]
+ − − 1
·
1

¸ ]
5!8
5G Simple linear regression model in matrix terms
Y
i

8

\$
S
i

i
"here ε
i
T independent .*8=σ
!
/ and i7\$=?=n
Dhe model can be re"ritten as
Y
\$

8

\$
S
\$

\$
Y
!

8

\$
S
!

!
M
Y
n

8

\$
S
n

n
Fn matrix terms= let
\$ \$ \$
! ! 8 !
\$
n n n
Y \$ S
Y \$ S
= 7 = and
Y \$ S
ε
1 1 1
1 1 1
β ε
1
1 1 1
· · ·
1
1 1 1 β
¸ ]
1 1 1
ε
¸ ] ¸ ] ¸ ]
Y X β ε
M M M M
Dhen Y 7 Xβ K ε= "hich is
8 \$ \$ \$ \$ \$ \$
8 ! \$ ! ! ! 8 !
\$
8 n \$ n n n n
S Y \$ S
S Y \$ S
S Y \$ S
β + β + ε ε
1 1 1 1
1 1 1 1
β + β + ε β ε
1
1 1 1 1
· + ·
1
1 1 1 1 β
¸ ]
1 1 1 1
β + β + ε ε
¸ ] ¸ ] ¸ ] ¸ ]
M M M M M
.ote that 0*Y/ 7 0*Xβ K ε/ 7 0*Xβ/ K 0*ε/ 7 Xβ since 0*ε/ 7
0 and Xβ are constants
5!G
5\$8 Jeast s@uares estimation of regression parameters
From Chapter \$: Dhe least s@uares method tries to find
the b
8
and b
\$
such that SS0 7 Σ*Y>
Y
X
/
!
7 Σ*residual/
!
is
minimi5ed Formulas for b
8
and b
\$
are deri;ed using
calculus
Ft can be sho"n that b
8
and b
\$
can be found from sol;ing
the 'normal e@uations):
n n
8 \$ i i
i \$ i \$
n n n
!
8 i \$ i i i
i \$ i \$ i \$
nb b S Y
b S b S SY
· ·
· · ·
+ ·
∑ ∑
+ ·
∑ ∑ ∑
Dhe normal e@uations can be re"ritten as
′ ′
· X Xb X Y
"here
8
\$
b
b
1
·
1
¸ ]
b
Fn Section 51= it "as sho"n that
\$
n
i
!
i \$
n
\$ ! n
i i
i \$
n
Y
Y
\$ \$ \$ Y
S S S
SY
Y
·
·
1
1
1 ∑
1 1
1

· ·
1 1
1
¸ ]
1

1
¸ ]
¸ ]
X Y
L
L M
and
X′X
\$
n
i
!
i \$
n n
!
\$ ! n
i i
i \$ i \$
n
\$ S
n S
\$ \$ \$ \$ S
S S S
S S
\$ S
·
· ·
1
1
1 ∑
1 1
1
· ·
1 1
1
¸ ]
1
∑ ∑
1
¸ ]
¸ ]
L
L M M

Dhus=
′ ′
· X Xb X Y can be "ritten as
518
n n
i i
8
i \$ i \$
n n n
!
\$
i i i i
i \$ i \$ i \$
n S Y
b
b
S S SY
· ·
· · ·
1 1
∑ ∑
1 1 1
·
1 1 1
¸ ]
1 1
∑ ∑ ∑
¸ ] ¸ ]

n n
8 \$ i i
i \$ i \$
n n n
!
8 i \$ i i i
i \$ i \$ i \$
nb b S Y
b S b S SY
· ·
· · ·
1 1
+
∑ ∑
1 1
·
1 1
+ 1 1
∑ ∑ ∑
¸ ] ¸ ]
Suppose both sides of
′ ′
· X Xb X Y are multiplied by
*X′X/
>\$
Dhen
\$ \$
\$
\$
* / * /
* /
* /
− −

′ ′ ′ ′
·
′ ′
·
′ ′
·
X X X Xb X X X Y
Ib X X X Y
b X X X Y
Dherefore= "e ha;e a "ay to find b using matrix algebra!
.ote that
\$
\$
n n
i i
i \$ i \$
n n n
!
i i i i
i \$ i \$ i \$
* /
n S Y
S S SY

· ·
· · ·
′ ′
·
1 1
∑ ∑
1 1
·
1 1
1 1
∑ ∑ ∑
¸ ] ¸ ]
b X X X Y
51\$
( )
n n n
!
i i i
i \$ i \$ i \$
!
n n
n n
!
i i i
i i
i \$ i \$
i \$ i \$
S S Y
\$
S n SY
n S S
· · ·
· ·
· ·
1 1

∑ ∑ ∑
1 1
·
1 1
− 1 1
∑ ∑

∑ ∑
¸ ] ¸ ]
( ) ( ) ( ) ( )
( ) ( )
n n n n
!
i i i i i
i \$ i \$ i \$ i \$
n
n n n
!
i
i i i i
i \$
i \$ i \$ i \$
Y S S SY
\$
n *S S/
Y S n SY
· · · ·
·
· · ·
1

∑ ∑ ∑ ∑
1
·
1
1 −

− +
∑ ∑ ∑
1
¸ ]
\$
i i
!
i
Y b S
*S S/*Y Y/
*S S/
1

1
·
− −

1
1

¸ ]
0xample: AS and College BC( *ASIcollegeIBC(Ich5#/
#emember that
\$ 182
\$ !15
\$ !+8
\$ !!8
\$ \$88
1
1
1
1
·
1
1
1
1
1
¸ ]
X
M M
and
1\$8
!18
188
!!8
\$,8
1
1
1
1
·
1
1
1
1
1
¸ ]
Y
M
Find
\$
* /

′ ′
· b X X X Y %e already did this on p 5!!
> "ol.,(t(C)010C) 010 t(C)010E
#,1\$
#1,\$ '-3'35336
#2,\$ '-6**65(4
51!
> 4"in; t6, 9orm+lation o9 "ol., 9or x in Ax = b
> "ol.,(t(C)010C, t(C)010E)
#,1\$
#1,\$ '-3'35336
#2,\$ '-6**65(4
From pre;ious output:
> mod-9it<-lm(9orm+la = /oll,;,-ABA F @?-ABA, data = ;:a)
> mod-9itDco,99ici,nt"
(Hnt,rc,:t) @?-ABA
'-3'35336 '-6**65(4
511
5\$\$ Fitted ;alues and residuals
Jet
\$
n
X
Y
X
X
Y
1
1
·
1
1
¸ ]
Y M
Dhen
X
· Y Xb
since
\$ \$ 8 \$ \$
8
\$
n 8 \$ n
n
X
Y \$ S b b S
b
b
X
\$ S b b S
Y
1
+
1 1
1 1
1 1
· ·
1 1
1 1
¸ ]
1
1 1 +
¸ ] ¸ ]
¸ ]
M M M M
Aat matrix:
\$
X
* / Y

· · · Y Xb X X X X YΗY "here H7X*X′X/
>\$
X′
is the hat matrix
%hy is this called the Aat matrix&
%e "ill use this in Chapter \$8 to measure the influence
of obser;ations on the estimated regression line
#esiduals:
Jet
\$ \$ \$
n
n n
X
e Y Y
X
* /
X
e
Y Y
1

1
1
1
· · · − · − · −
1
1
1
1

¸ ]
¸ ]
e Y Y Y HY Y I H M M
Co;ariance matrix of the residuals:
Co;*e/ 7 Co;*Y*I>H// 7 *I>H/Co;*Y/*I>H/′
512
.o"= Co;*Y/ 7 Co;*Xβ K ε/ 7 Co;*ε/ since Xβ are
constants (nd= Co;*ε/ 7 σ
!
I
(lso= *I>H/ 7 *I>H/′ and *I>H/*I>H/ 7 *I>H/ *I>H/ is called a
symmetric 'idempotent) matrix You can use matrix
algebra to see this result on your o"n *replace H "ith
X*X′X/
>\$
X′ and multiply out/
Dhus= Co;*e/ 7 *I>H/σ
!
I*I>H/′ 7 σ
!
*I>H/
Dhe estimated co;ariance matrix of the residuals is then
·
!
Co;* / MS0 * / X * /= say · ∗ − · σ ∗ − e I H I H .ote that -..
"ould denote
·
Co;* / e as s
!
VeW Dhe
·
Co;* / e notation is
more predominantly used
515
5\$! (nalysis of ;ariance results
Sums of S@uares
From Chapters \$ and !:
n
!
i
i \$
SSDM *Y Y/
·
· −
∑ =
n
!
i
i \$
X
SS0 *Y Y/
·
· −
∑ = and
n
!
i
i \$
X
SS# SSDM SS0 *Y Y/
·
· − · −

Dhese can be re"ritten using matrices:
[ ] [ ]
( )
\$ \$
\$ n \$ n
n n
\$
n n n
!
i i i
i \$ i \$ i \$
n
!
n n
!
i i
i \$ i \$
n
!
i
i \$
\$
SSDM
n
Y \$ \$ Y
\$
Y Y Y Y
n
Y \$ \$ Y
Y
\$
Y Y Y
n
Y
\$
Y Y
n
*Y Y/
· · ·
· ·
·
′ ′
· −
1 1 1
1 1 1
· −
1 1 1
1 1 1
¸ ] ¸ ] ¸ ]
1
1 1
· −
∑ ∑ ∑
1 1
¸ ]
1
¸ ]
· −
∑ ∑
· −

Y Y Y JY
L
L M L M O M M
L
L M
[ ]
\$
n n
! !
\$ n i i i
i \$ i \$
n
e
X
SS0 e e e *Y Y/
e
· ·
1
1

· · · · −
∑ ∑
1
1
¸ ]
e e L M
51,
\$
SS# SSDM SS0
n
′ ′ ′
· − · − b X Y Y JY
0xample: AS and College BC( *ASIcollegeIBC(Ich5#/
Continuing the same program
> n<-l,n;t6(E) 4/an not +", row" 6,r, "inc, R do," not
Know EI" dim,n"ion
> 4/o+ld +", nrow(C) a" w,ll - C i" a nx2
matrix
> b<-"ol.,(t(C)010C) 010 t(C)010E
> E-6at<-C010b
> ,<-E-E-6at
> @<-C010"ol.,(t(C)010C)010t(C)

> L<-matrix(data = 1, nrow = n, ncol = n)
> 4=otic, t6at R will r,:,at 1 t6, corr,ct n+mb,r o9
tim," to 9ill t6, matrix
> ??TM<-t(E)010E - 1Nn010t(E)010L010E
> ??!<-t(,)010,
> O?!<-??!N(n-nrow(b))

> 4=otic, 6ow O?! +"," 1 to do t6, m+lti:lyin; "inc, it i"
a "calar- Al"o dia;(n) cr,at," an id,ntity matrix-
2ia;(c(1,2)) wo+ld cr,at, a 2x2 dia;onal matrix wit6
,l,m,nt" 1 and 2 on t6, dia;onal
> /o.-,<-O?!1(dia;(n) - @) 42o," not worKP
!rror in O?! 1 (dia;(n) - @) 8 non-con9ormabl, array"
> /o.-,<-a"-n+m,ric(O?!)1(dia;(n) - @) 4T6, a"-n+m,ric()
r,mo.," 1x1 matrix
m,anin; 9rom O?!
> ??R = ??TM - ??!

> data-9ram,(C, E, E-6at, ,)
C1 C2 E E-6at ,
51+
1 1 3-'4 3-1 2-(3453* '-26546'*1
2 1 2-35 2-3 2-351335 -'-'51334(2
3 1 2-3' 3-' 2-5*6655 '-4'334435
4 1 2-'5 1-* 2-141(33 -'-241(3331
5 1 2-(3 2-5 2-6(3611 -'-1(361'(3
6 1 4-32 3-3 3-33'1'2 -'-'3'1'1(1
3 1 3-3* 3-4 3-'3*42' '-32'5('4(
( 1 2-32 2-6 2-33'3(5 '-26*214*3
* 1 2-6* 2-( 2-5(*65* '-21'34134
1' 1 '-(3 1-6 1-2((2*4 '-3113'5*1
11 1 2-3* 2-' 2-33*361 -'-33*36115
12 1 3-65 2-* 3-261331 -'-36133'3'
13 1 1-(5 2-3 2-''1*46 '-2*('5433
14 1 3-(3 3-2 3-3(326* -'-1(326*21
15 1 1-22 1-( 1-561161 '-23((3*14
16 1 1-4( 1-4 1-343'32 -'-343'32'3
13 1 2-2( 2-' 2-3'23** -'-3'23*(33
1( 1 4-'' 3-( 3-5'6211 '-2*33(((3
1* 1 2-2( 2-2 2-3'23** -'-1'23*(33
2' 1 1-(( 1-6 2-'22*35 -'-422*353(

> data-9ram,(n, ??TM, ??!, O?!, ??R)
n ??TM ??! O?! ??R
1 2' *-64*5 1-5(3*66 '-'((22'35 (-'61534

> ro+nd(/o.-,#185, 185\$,6) 45x5 :art o9 /o.(,)
#,1\$ #,2\$ #,3\$ #,4\$ #,5\$
#1,\$ '-'(2621 -'-''3(5( -'-''4342 -'-''31'1 -'-''5'3'
#2,\$ -'-''3(5( '-'(3552 -'-''4253 -'-''5'2' -'-''41'5
#3,\$ -'-''4342 -'-''4253 '-'(3313 -'-''4'43 -'-''45*4
#4,\$ -'-''31'1 -'-''5'2' -'-''4'43 '-'(2366 -'-''36(5
#5,\$ -'-''5'3' -'-''41'5 -'-''45*4 -'-''36(5 '-'(3444
> 4Qrom :a"t worK
> data-9ram,(C = mod,l-matrix(mod-9it), E-6at =
mod-9itD9itt,d-.al+,", , = mod-9itDr,"id+al")
C--Hnt,rc,:t- C-@?-ABA E-6at ,
1 1 3-'4 2-(3453* '-26546'*1
2 1 2-35 2-351335 -'-'51334(2
3 1 2-3' 2-5*6655 '-4'334435
4 1 2-'5 2-141(33 -'-241(3331
5 1 2-(3 2-6(3611 -'-1(361'(3
6 1 4-32 3-33'1'2 -'-'3'1'1(1
518
3 1 3-3* 3-'3*42' '-32'5('4(
( 1 2-32 2-33'3(5 '-26*214*3
* 1 2-6* 2-5(*65* '-21'34134
1' 1 '-(3 1-2((2*4 '-3113'5*1
11 1 2-3* 2-33*361 -'-33*36115
12 1 3-65 3-261331 -'-36133'3'
13 1 1-(5 2-''1*46 '-2*('5433
14 1 3-(3 3-3(326* -'-1(326*21
15 1 1-22 1-561161 '-23((3*14
16 1 1-4( 1-343'32 -'-343'32'3
13 1 2-2( 2-3'23** -'-3'23*(33
1( 1 4-'' 3-5'6211 '-2*33(((3
1* 1 2-2( 2-3'23** -'-1'23*(33
2' 1 1-(( 2-'22*35 -'-422*353(
> ano.a(mod-9it)
Analy"i" o9 Rarianc, Tabl,
R,":on",8 /oll,;,-ABA
29 ?+m ?S O,an ?S Q .al+, Br(>Q)
@?-ABA 1 (-'615 (-'615 *1-3( 1-33*,-'( 111
R,"id+al" 1( 1-5((' '-'((2
---
?i;ni9- cod,"8 ' I111I '-''1 I11I '-'1 I1I '-'5 I-I '-1 I
I 1
51G
5\$1 Fnferences in #egression (nalysis
Co;ariance matrix of b:
8 8 \$
! \$
8 \$ \$
Ear*b / Co;*b =b /
Co;* / * /
Co;*b =b / Ear*b /

1

· · σ
1
¸ ]
b X X
%hy&
Co;*b/ 7 Co;Q*X′X/
>\$
X′YR
7 *X′X/
>\$
X′Co;*Y/Q*X′X/
>\$
X′R′
7 *X′X/
>\$
X′Co;*Y/X*X′X/
>\$
7 *X′X/
>\$
X′σ
!
IX*X′X/
>\$
7 σ
!
*X′X/
>\$
X′X*X′X/
>\$
7 σ
!
*X′X/
>\$
#emember that for t"o matrices C and B= *CB/′ 7
B′C′
0stimated co;ariance matrix of b:
·
· ·
· ·
8 8 \$
8 \$ \$
!
! !
i i
! !
i i
\$
Ear*b / Co;*b =b /
Co;* /
Co;*b =b / Ear*b /
\$ S S
n *S S/ *S S/
MS0
S \$
*S S/ *S S/
MS0 * /

1
· 1
1
¸ ]
1

+
1
− −
∑ ∑
1
·
1

1
− − 1 ∑ ∑
¸ ]

· ∗
b
X X
528
.ote that the
·
i
Ear*b / is from Sections !\$ and !! Dhe
·
8 \$
Co;*b =b / is ne"
0stimated ;ariance used in the CF for 0*Y
h
/:

·
!
h
\$
h h h
!
i
\$ *S S/
X
Ear*Y / MS0* * / / MS0
n *S S/

¸ _

′ ′
· · +
÷

¸ ,
X X X X
"here X
h
7 Q\$= S
h
R′
%hy&
h h
h h
! \$
h h
X
Ear*Y / Ear* /
Co;* /
* /

·

·
′ ′
· σ
X b
X b X
X X X X
Since b is a ;ector= F replaced Ear*/ "ith Co;*/ *:ust
notation/
0stimated ;ariance used in the CF for Y
h*ne"/
:

·
\$
h*ne"/ h h h
!
h
!
i
X
Ear*Y Y / MS0*\$ * / /
\$ *S S/
MS0 \$
n *S S/

′ ′
− · +
¸ _

· + +
÷

¸ ,
X X X X
0xample: AS and College BC( *ASIcollegeIBC(Ich5#/
> co.-b,ta-6at<-a"-n+m,ric(O?!)1"ol.,(t(C)010C)
> co.-b,ta-6at
#,1\$ #,2\$
#1,\$ '-'3*366'6 -'-'133621(1
52\$
#2,\$ -'-'133621( '-''5353'1*

> 4Qrom ,arli,r
> "+m-9it<-"+mmary(mod-9it)
> "+m-9itDco.-+n"cal,d 4Anot6,r way to ;,t (CIC)J(-1)
(Hnt,rc,:t) @?-ABA
(Hnt,rc,:t) '-45'35(4 -'-155**3(1
@?-ABA -'-155**3( '-'6'32316
> "+m-9itD"i;maJ2 1 "+m-9itDco.-+n"cal,d
(Hnt,rc,:t) @?-ABA
(Hnt,rc,:t) '-'3*366'6 -'-'133621(1
@?-ABA -'-'133621( '-''5353'1*
> 4Mn, mor, way to ;,t t6, co.arianc, matrix
> .co.(mod-9it)
(Hnt,rc,:t) @?-ABA
(Hnt,rc,:t) '-'3*366'6 -'-'133621(1
@?-ABA -'-'133621( '-''5353'1*
> 4Qind Rar(EJ)
> C-6<-c(1,1-(()
> a"-n+m,ric(O?!)1C-6010"ol.,(t(C)010C)010C-6
#,1\$
#1,\$ '-''6*541'3

> 4Qind Rar(E-EJ)
> a"-n+m,ric(O?!)1 (1)C-6010"ol.,(t(C)010C)010C-6)
#,1\$
#1,\$ '-'*513446

> 4Qrom @?Tcoll,;,TABATc62-R
> n<-nrow(;:a)
> C-6<-1-((
> E-6at-6<-a"-n+m,ric(mod-9itDco,99ici,nt"#1\$ )
mod-9itDco,99ici,nt"#2\$1C-6)
4a"-n+m,ric G+"t r,mo.," an +nn,,d,d nam,
> ""x<-.ar(;:aD@?-ABA)1(n-1) 4?O( (CTi - CTbar)J2 )