You are on page 1of 43

Chapter 5: Matrix approach to simple linear regression

analysis
You need to understand matrix algebra for multiple
regression! Fox’s Section 8! contains information about
ho" to use # for matrix algebra
5$ Matrices
%hat is a matrix&
'( matrix is a rectangular array of elements arranged in
ro"s and columns) *p $+, of -../
0xample:
$ ! 1
2 5 ,
1
1
¸ ]
3imension 4 Si5e of matrix: 6 ro"s × 6 columns 7 r×c
0xample: !×1
Symbolic representation of a matrix:
0xample:
$$ $! $1
!$ !! !1
a a a
a a a
1
·
1
¸ ]
A

© !8$! Christopher # 9ilder
5$
"here a
i:
is the ro" i and column : element of A
a
$$
7$ from the abo;e example
.otice that the matrix A is in bold %hen bolding is not
possible *"riting on a piece of paper or chal<board/= the
letter is underlined > (
a
$$
is often called the '*$=$/ element) of A= a
$!
is called
the '*$=!/ element) of A=?
0xample: r×c matrix
$$ $! $: $c
!$ !! !: !c
i$ i! i: ic
r$ r ! r: rc
a a a a
a a a a
a a a a
a a a a
1
1
1
1
·
1
1
1
1
1
¸ ]
A
L L
L L
M M M M
L L
M M M M
L L
0xample: S@uare matrix is r×c "here r7c
0xample: AS and College BC(
© !8$! Christopher # 9ilder
5!
$ 182
$ !15
$ !+8
$ !!8
$ $88
1
1
1
1
·
1
1
1
1
1
¸ ]
X
M M
Dhe abo;e !8×! matrix contains the AS BC(s in the
second column
Eector 4 a r×$ *column ;ector/ or $×c *ro" ;ector/ matrix 4
special case of a matrix
0xample: Symbolic representation of a 1×$ column ;ector
$
!
1
a
a
a
1
1
·
1
1
¸ ]
A
0xample: AS and College BC(
1$8
!18
188
!!8
$,8
1
1
1
1
·
1
1
1
1
1
¸ ]
Y
M
© !8$! Christopher # 9ilder
51
Dhe abo;e !8×$ ;ector contains the College BC(s
Dranspose: Fnterchange the ro"s and columns of a matrix or
;ector
0xample:
$$ $! $1
!$ !! !1
a a a
a a a
1
·
1
¸ ]
A
and
$$ !$
$! !!
$1 !1
a a
a a
a a
1
1

·
1
1
¸ ]
A
A is !×1 and A′ is 1×!
Dhe ′ symbol indicates a transpose= and it is said as the
"ord 'prime) Dhus= the transpose of A is '( prime)
0xample: AS and College BC(
[ ]
1$8 !18 188 !!8 $,8

· Y L
© !8$! Christopher # 9ilder
52
5! Matrix addition and subtraction
(dd or subtract the corresponding elements of matrices
"ith the same dimension
0xample:
Suppose
$ ! 1 $ $8 $
and
2 5 , 5 5 8
− −
1 1
· ·
1 1
¸ ] ¸ ]
A B
Dhen
8 $! ! ! 8 2
and
G $8 $2 $ 8 !

1 1
+ · − ·
1 1
− −
¸ ] ¸ ]
A B A B

0xample: Hsing # *basicImatrixIalgebra#/
> A<-matrix(data = c(1, 2, 3,
4, 5, 6), nrow = 2, ncol = 3, byrow =
TR!)
> cla""(A)
#1$ %matrix%
> &<-matrix(data = c(-1, 1', -1, 5, 5, (), nrow = 2, ncol =
3, byrow = TR!)

> A)&
#,1$ #,2$ #,3$
#1,$ ' 12 2
#2,$ * 1' 14
> A-&
#,1$ #,2$ #,3$
#1,$ 2 -( 4
#2,$ -1 ' -2
© !8$! Christopher # 9ilder
55
.otes:
$9e careful "ith the byrow option 9y default= this is set
to F(JS0 Dhus= the numbers "ould be entered into
the matrix by columns For example=
> matrix(data = c(1, 2, 3, 4, 5, 6), nrow = 2, ncol = 3)
#,1$ #,2$ #,3$
#1,$ 1 3 5
#2,$ 2 4 6
!Dhe class of these ob:ects is 'matrix)
1( ;ector can be represented as a 'matrix) class type or
a type of its o"n
> y<-matrix(data = c(1,2,3), nrow = 3, ncol = 1, byrow
= TR!)
> y
#,1$
#1,$ 1
#2,$ 2
#3,$ 3
> cla""(y)
#1$ %matrix%
> x<-c(1,2,3)
> x
#1$ 1 2 3
> cla""(x)
#1$ %n+m,ric%
> i"-.,ctor(x)
#1$ TR!
Dhis can present some confusion "hen ;ectors are
multiplied "ith other ;ectors or matrices because no
specific ro" or column dimensions are gi;en More on
this shortly
© !8$! Christopher # 9ilder
5,
2( transpose of a matrix can be done using the t()
function For example=
> t(A)
#,1$ #,2$
#1,$ 1 4
#2,$ 2 5
#3,$ 3 6
0xample: Simple linear regression model
Y
i
70*Y
i
/ K ε
i
for i7$=?=n can be represented as
0* / · + Y Y ε
"here
$ $ $
! ! !
n n n
Y 0*Y /
Y 0*Y /
= 0* / = and
Y 0*Y /
ε
1 1 1
1 1 1
ε
1 1 1
· · ·
1 1 1
1 1 1
ε
¸ ] ¸ ] ¸ ]
M M M
Y Y ε
© !8$! Christopher # 9ilder
5+
51 Matrix multiplication
Scalar > $×$ matrix
0xample: Matrix multiplied by a scalar
$$ $! $1
!$ !! !1
ca ca ca
c
ca ca ca
1
·
1
¸ ]
A
"here c is a scalar
Jet
$ ! 1
2 5 ,
1
·
1
¸ ]
A
and c7! Dhen
! 2 ,
!
8 $8 $!
1
·
1
¸ ]
A

Multiplying t"o matrices
Suppose you "ant to multiply the matrices A and BL ie=
A∗B or AB Fn order to do this= you need the number of
columns of A to be the same as the number of ro"s as
B For example= suppose A is !×1 and B is 1×$8 You
can multiply these matrices Ao"e;er if B is 2×$8
instead= these matrices could .MD be multiplied
Dhe resulting dimension of C7AB
$Dhe number of ro"s of A is the number of ro"s of C
!Dhe number of columns of B is the number of ro"s of
C
1Fn other "ords=
" y " 5 5 y × × ×
· C A B
"here the dimension of the
matrices are sho"n belo" them
© !8$! Christopher # 9ilder
58
Ao" to multiply t"o matrices 4 an example
Suppose
1 8
$ ! 1
and $ !
2 5 ,
8 $
1
1
1
· ·
1
1
¸ ]
1
¸ ]
A B
.otice that A
is !×1 and B is 1×! so C7AB can be done
1 8
$ ! 1
$ !
2 5 ,
8 $
$ 1 ! $ 1 8 $ 8 ! ! 1 $
2 1 5 $ , 8 2 8 5 ! , $
5 +
$+ $,
1
1
1
· ·
1
1
¸ ]
1
¸ ]
∗ + ∗ + ∗ ∗ + ∗ + ∗
1
·
1
∗ + ∗ + ∗ ∗ + ∗ + ∗
¸ ]
1
·
1
¸ ]
C AB
Dhe 'cross product) of the ro"s of A and the columns of
B are ta<en to form C
Fn the abo;e example= D7BA≠AB "here BA is:
© !8$! Christopher # 9ilder
5G
1 8
$ ! 1
$ !
2 5 ,
8 $
1 $ 8 2 1 ! 8 5 1 1 8 ,
$ $ ! 2 $ ! ! 5 $ 1 ! ,
8 $ $ 2 8 ! $ 5 8 1 $ ,
1 , G
G $! $5
2 5 ,
1
1
1
·
1
1
¸ ]
1
¸ ]
∗ + ∗ ∗ + ∗ ∗ + ∗
1
1
· ∗ + ∗ ∗ + ∗ ∗ + ∗
1
1 ∗ + ∗ ∗ + ∗ ∗ + ∗
¸ ]
1
1
·
1
1
¸ ]
BA
Fn general for a !×1 matrix times a 1×! matrix:
$$ $!
$$ $! $1
!$ !!
!$ !! !1
1$ 1!
$$ $$ $! !$ $1 1$ $$ $! $! !! $1 1!
!$ $$ !! !$ !1 1$ !$ $! !! !! !1 1!
b b
a a a
b b
a a a
b b
a b a b a b a b a b a b
a b a b a b a b a b a b
1
1
1
· ·
1
1
¸ ]
1
¸ ]
+ + + +
1
·
1
+ + + +
¸ ]
C AB
0xample: Hsing # *basicImatrixIalgebra#/
> A<-matrix(data = c(1, 2, 3, 4, 5, 6), nrow = 2, ncol =
3, byrow = TR!)
> &<-matrix(data = c(3, ', 1, 2, ', 1), nrow = 3, ncol =
2, byrow = TR!)

© !8$! Christopher # 9ilder
5$8
> /<-A010&
> 2<-&010A

> /
#,1$ #,2$
#1,$ 5 3
#2,$ 13 16
> 2
#,1$ #,2$ #,3$
#1,$ 3 6 *
#2,$ * 12 15
#3,$ 4 5 6

> 456at i" A1&7
> A1&
!rror in A 1 & 8 non-con9ormabl, array"
.otes:
$NON is used for multiplying matrices andPor ;ectors
!O means to perform element"ise multiplications Aere
is an example "here this can be done:
> !<-A
> A1!
#,1$ #,2$ #,3$
#1,$ 1 4 *
#2,$ 16 25 36
Dhe *i=i/ elements of each matrix are multiplied
together
1Multiplying ;ectors "ith other ;ectors or matrices can
be confusing since no ro" or column dimensions are
© !8$! Christopher # 9ilder
5$$
gi;en for a ;ector ob:ect For example= suppose x 7
$
!
1
1
1
1
1
¸ ]
= a 1×$ ;ector
> x<-c(1,2,3)
> x010x
#,1$
#1,$ 14
> A010x
#,1$
#1,$ 14
#2,$ 32
Ao" does # <no" that "e "ant x′x *$×$/ instead of
xx′ *1×1/ "hen "e ha;e not told # that x is 1×$&
Similarly= ho" does # <no" that Ax is !×$& From the
# help for NON in the 9ase pac<age:
Multiplies t"o matrices= if they are conformable Ff
one argument is a ;ector= it "ill be promoted to
either a ro" or column matrix to ma<e the t"o
arguments conformable Ff both are ;ectors it "ill
return the inner product
(n inner product produces a scalar ;alue Ff you
"anted xx′ *1×1/= one can use the outer product NoN
> x0o0x 4o+t,r :rod+ct
#,1$ #,2$ #,3$
#1,$ 1 2 3
#2,$ 2 4 6
© !8$! Christopher # 9ilder
5$!
#3,$ 3 6 *
%e "ill only need to use NON in this class
0xample: AS and College BC( *ASIcollegeIBC(Ich5#/
$ 182
$ !15
$ !+8
$ !!8
$ $88
1
1
1
1
·
1
1
1
1
1
¸ ]
X
M M
and
1$8
!18
188
!!8
$,8
1
1
1
1
·
1
1
1
1
1
¸ ]
Y
M
Find X′X= X′Y= and Y′Y
> 4R,ad in t6, data
> ;:a<-r,ad-tabl,(9il, =
%/8<<c6ri"<<=><<?TAT(3'<</6a:t,r1<<;:a-txt%,
6,ad,r=TR!, ",: = %%)
> 6,ad(;:a)
@?-ABA /oll,;,-ABA
1 3-'4 3-1
2 2-35 2-3
3 2-3' 3-'
4 2-'5 1-*
5 2-(3 2-5
6 4-32 3-3

> C<-cbind(1, ;:aD@?-ABA)
> E<-;:aD/oll,;,-ABA

> C
© !8$! Christopher # 9ilder
5$1
#,1$ #,2$
#1,$ 1 3-'4
#2,$ 1 2-35
#3,$ 1 2-3'
#4,$ 1 2-'5
#5,$ 1 2-(3
#6,$ 1 4-32
#3,$ 1 3-3*
#(,$ 1 2-32
#*,$ 1 2-6*
#1',$ 1 '-(3
#11,$ 1 2-3*
#12,$ 1 3-65
#13,$ 1 1-(5
#14,$ 1 3-(3
#15,$ 1 1-22
#16,$ 1 1-4(
#13,$ 1 2-2(
#1(,$ 1 4-''
#1*,$ 1 2-2(
#2',$ 1 1-((
> E
#1$ 3-1 2-3 3-' 1-* 2-5 3-3 3-4 2-6 2-( 1-6 2-' 2-* 2-3
3-2 1-( 1-4 2-' 3-( 2-2
#2'$ 1-6

> t(C)010C
#,1$ #,2$
#1,$ 2'-'' 51-3(''
#2,$ 51-3( 14(-4634
> t(C)010E
#,1$
#1,$ 5'-1''
#2,$ 14'-22*
> t(E)010E
#,1$
#1,$ 135-15

.otes:
© !8$! Christopher # 9ilder
5$2
$Dhe cbind() function combines items by 'c)olumns
Since $ is only one element= it "ill replicate itself for
all elements that you are combining so that one full
matrix is formed Dhere is also a rbind() function that
combines by ro"s Dhus= rbind(a,b) forms a matrix
"ith a abo;e b
! [ ]
$
n
!
!
$ ! n i
i $
n
y
y
y = y == y y
y
·
1
1
1

· ·

1
1
¸ ]
Y Y
M
1
$
n n
i$ i i
$$ !$ n$ !
i $ i $
n n
$! !! n!
i! i i! i
i $ i $
n
y
x y y
x x x y
x x x
x y x y
y
· ·
· ·
1
1 1
1 ∑ ∑
1 1 1
1

· · ·
1 1 1
1
¸ ]
1 1
∑ ∑
1
¸ ] ¸ ]
¸ ]
X Y
L
L M

since x
$$
7?7x
n$
7$
2X′X
$$ $!
$$ !$ n$ !$ !!
$! !! n!
n$ n!
$!
n
i!
!!
i $
n n
!
$! !! n!
i! i!
i $ i $
n!
x x
x x x x x
x x x
x x
$ x
n x
$ $ $ $ x
x x x
x x
$ x
·
· ·
1
1
1
1
·
1
1
¸ ]
1
¸ ]
1
1
1 ∑
1 1
1
· ·
1 1
1
¸ ]
1
∑ ∑
1
¸ ]
¸ ]
L
L M M
L
L M M
© !8$! Christopher # 9ilder
5$5
5Aere’s another "ay to get the X matrix
> mod-9it<-lm(9orm+la = /oll,;,-ABA F @?-ABA, data =
;:a)
> mod,l-matrix(obG,ct = mod-9it)
(Hnt,rc,:t) @?-ABA
1 1 3-'4
2 1 2-35
3 1 2-3'
4 1 2-'5
5 1 2-(3
6 1 4-32
3 1 3-3*
( 1 2-32
* 1 2-6*
1' 1 '-(3
11 1 2-3*
12 1 3-65
13 1 1-(5
14 1 3-(3
15 1 1-22
16 1 1-4(
13 1 2-2(
1( 1 4-''
1* 1 2-2(
2' 1 1-((
attr(,%a""i;n%)
#1$ ' 1
© !8$! Christopher # 9ilder
5$,
52 Special types of matrices
Symmetric matrix: Ff A7A′= then A is symmetric
0xample:
$ ! $ !
=
! 1 ! 1
1 1

· ·
1 1
¸ ] ¸ ]
A A
3iagonal matrix: ( s@uare matrix "hose 'off>diagonal)
elements are 8
0xample:
$$
!!
11
a 8 8
8 a 8
8 8 a
1
1
·
1
1
¸ ]
A
Fdentity matrix: ( diagonal matrix "ith $’s on the diagonal
0xample:
$ 8 8
8 $ 8
8 8 $
1
1
·
1
1
¸ ]
I
.ote that 'I) *the letter I= not the number one/ usually
denotes the identity matrix
Eector and matrix of $’s
( column ;ector of $’s:
r $
r $
$
$
$
×
×
1
1
1
· ·
1
1
¸ ]
1 j
M
© !8$! Christopher # 9ilder
5$+
( matrix of $’s:
r r
$ $ $
$ $ $
$ $ $
×
1
1
1
·
1
1
¸ ]
J
L
L
M M O M
L
.otes:
$
r $ r $
r
× ×

· j j
!
r r
r $r $
×
× ×

· j j J
1
r r r r r r
r
× × ×

· J J J
Eector of 8’s:
r $
8
8
8
×
1
1
1
·
1
1
¸ ]
0
M

© !8$! Christopher # 9ilder
5$8
55 Jinear dependence and ran< of matrix
Jet
$ ! ,
1 2 $!
5 , $8
1
1
·
1
1
¸ ]
A
Dhin< of each column of A as a
;ectorL ie= A 7 QA
$
= A
!
= A
1
R .ote that 1A
!
7A
1
Dhis
means the columns of A are 'linearly dependent)
Formally= a set of column ;ectors are linearly dependent
if there exists constants λ
$
= λ
!
=?= λ
n
*not all 5ero/ such
that λ
$
A
$

!
A
!
K?Kλ
c
A
c
70 ( set of column ;ectors are
linearly independent if λ
$
A
$

!
A
!
K?Kλ
c
A
c
70 only for
λ
$

!
7?7λ
c
70
Dhe ran< of a matrix is the maximum number of linearly
independent columns in the matrix
ran<*A/7!
© !8$! Christopher # 9ilder
5$G
5, Fn;erse of a matrix
.ote that the in;erse of a scalar= say b= is b
>$
For
example= the in;erse of b71 is 1
>$
7$P1 (lso= b∗b
>$
7$ Fn
matrix algebra= the in;erse of a matrix is another matrix
For example= the in;erse of A is A
>$
= and AA
>$
7A
>$
A7I
.ote that A must be a s@uare matrix
0xample:
$
$ ! ! $
=
1 2 $5 85


1 1
· ·
1 1

¸ ] ¸ ]
A A
Chec<:
$
$ * !/ ! $5 $ $ ! O * 85/ $ 8
1 * !/ 2 $5 1 $ 2 O * 85/ 8 $

∗ − + ∗ ∗ + −
1 1
· ·
1 1
∗ − + ∗ ∗ + −
¸ ] ¸ ]
AA
Finding the in;erse
For a general "ay= see a matrix algebra boo< *such as:
-olman= $G88/ For a !×! matrix= there is a simple
formula Jet
$$ $!
!$ !!
a a
a a
1
·
1
¸ ]
A
Dhen
!! $!
$
!$ $$ $$ !! $! !$
a a
$
a a a a a a


1
·
1
− −
¸ ]
A

Eerify AA
>$
7I on your o"n
0xample: Hsing # *basicImatrixIalgebra#/
> A<-matrix(data = c(1, 2, 3, 4), nrow = 2, ncol = 2,
byrow = TR!)
© !8$! Christopher # 9ilder
5!8
> "ol.,(A)
#,1$ #,2$
#1,$ -2-' 1-'
#2,$ 1-5 -'-5
> A010"ol.,(A)
#,1$ #,2$
#1,$ 1 1-11'223,-16
#2,$ ' 1-'''''',)''
> "ol.,(A)010A
#,1$ #,2$
#1,$ 1-'''''',)'' '
#2,$ 1-11'223,-16 1
> ro+nd("ol.,(A)010A, 2)
#,1$ #,2$
#1,$ 1 '
#2,$ ' 1
Dhe solve() function in;erts a matrix in # Dhe
solve(A,b) function can also be used to 'sol;e) for x in
Ax 7 b since A
>$
Ax 7 A
>$
b ⇒ Ix 7 A
>$
b ⇒ x 7 A
>$
b
0xample: AS and College BC( *ASIcollegeIBC(Ich5#/
#emember that
$ 182
$ !15
$ !+8
$ !!8
$ $88
1
1
1
1
·
1
1
1
1
1
¸ ]
X
M M
and
1$8
!18
188
!!8
$,8
1
1
1
1
·
1
1
1
1
1
¸ ]
Y
M
Find *X′X/
>$
and *X′X/
>$
X′Y
© !8$! Christopher # 9ilder
5!$
> "ol.,(t(C)010C)
#,1$ #,2$
#1,$ '-45'35(4 -'-155**3(1
#2,$ -'-155**3( '-'6'32316
> "ol.,(t(C)010C) 010 t(C)010E
#,1$
#1,$ '-3'35336
#2,$ '-6**65(4
From pre;ious output:
> mod-9it<-lm(9orm+la = /oll,;,-ABA F @?-ABA, data = ;:a)
> mod-9itDco,99ici,nt"
(Hnt,rc,:t) @?-ABA
'-3'35336 '-6**65(4
.ote that *X′X/
>$
X′Y 7
8
$
b
b
1
1
¸ ]
!!!
> 4Anot6,r way to ;,t (CIC)J(-1)
> "+m-9it<-"+mmary(mod-9it)
> "+m-9itDco.-+n"cal,d
(Hnt,rc,:t) @?-ABA
(Hnt,rc,:t) '-45'35(4 -'-155**3(1
@?-ABA -'-155**3( '-'6'32316
#ead 'Hses of Fn;erse Matrix) on p $G! of -.. on your
o"n
© !8$! Christopher # 9ilder
5!!
5+ Some basic theorems of matrices
#ead on your o"n
58 #andom ;ectors and matrices
( random ;ector or random matrix contains elements that
are random ;ariables
0xample: Simple linear regression model
Y
i
70*Y
i
/ K ε
i
for i7$=?=n can be represented as
0* / · + Y Y ε
"here
$ $
! !
n n
Y
Y
and
Y
ε
1 1
1 1
ε
1 1
· ·
1 1
1 1
ε
¸ ] ¸ ]
M M
Y ε
are random ;ectors
0xpectation of random ;ector or matrix: Find the expected
;alue of the indi;idual elements
0xample:
$ $ 8 $ $
! ! 8 $ !
n n 8 $ n
Y 0*Y / S
Y 0*Y / S
0* / 0
Y 0*Y / S
¸ _ β + β
1 1 1

1 1 1
β + β

1 1 1
· · ·

1 1 1

1 1 1

β + β
¸ ] ¸ ] ¸ ]
¸ ,
Y
M M M
© !8$! Christopher # 9ilder
5!1
$ $
! !
n n
0* / 8
0* / 8
0* / 0
0* / 8
¸ _ ε ε
1 1 1

1 1 1
ε ε

1 1 1
· · ·

1 1 1

1 1 1

ε ε
¸ ] ¸ ] ¸ ]
¸ ,
ε
M M M
since "e assume
ε
i
T.*8=σ
!
/
Eariance>co;ariance matrix of a random ;ector
Jet U
$
and U
!
be random ;ariables #emember that
Ear*U
$
/70*U
$

$
/
!
"here 0*U
$
/7µ
$

Dhe co;ariance of U
$
and U
!
is defined as Co;*U
$
=U
!
/ 7
0Q*U
$

$
/*U
!

!
/R Dhe co;ariance measures the
relationship bet"een U
$
and U
!
See p 2!, of my
Chapter 2 SD(D 188 notes for a more mathematical
explanation *http:PP"""chrisbildercomP
stat188Pschedulehtm/ .ote that the correlation
bet"een U
$
and U
!
is
$ !
$ !
$ !
Co;*U =U /
Corr*U =U /
Ear*U / Ear*U /
·

Dhe 'Cearson correlation coefficient) "hich is denoted by
r and estimates Corr*U
$
= U
!
/
© !8$! Christopher # 9ilder
5!2
#emember that
n
i i
i $
n n
! !
i i
i $ i $
*S S/*Y Y/
r
*S S/ *Y Y/
·
· ·
− −

·
1 1
− −
∑ ∑
1 1
¸ ] ¸ ]

"here S7U
$
and Y7U
!

.otes:
 Co;*U
$
=U
$
/70Q*U
$

$
/*U
$

$
/R70Q*U
$

$
/
!
R7Ear*U
$
/
 Ff U
$
and U
!
are independent= then Co;*U
$
=U
!
/78
( ;ariance>co;ariance matrix *most often :ust called the
co;ariance matrix/ is a matrix "hose elements are the
;ariances and co;ariances of random ;ariables
0xample: Jet
$
!
U
U
1
·
1
¸ ]
Z
Dhe co;ariance matrix of Z is
Co;*Z/ 7
$ $ !
$ ! !
Ear*U / Co;*U =U /
Co;*U =U / Ear*U /
1
1
¸ ]
.ote that Co;*U
$
=U
!
/7Co;*U
!
=U
$
/ and Co;*U
i
=U
i
/7Ear*U
i
/
-.. denote Co;*Z/ by σ
!
VZW Dhe notation that F am
using is much more pre;alent and there is less chance
for confusion *li<e Z being multiplied by σ
!
in σ
!
VZW/
0xample: Simple linear regression model
© !8$! Christopher # 9ilder
5!5
$
!
n
Ear* / 8 8
8 Ear* / 8
Co;* /
8 8 Ear* /
ε
1
1
ε
1
·
1
1
ε
¸ ]
ε
L
L
M M O M
L
!
!
!
8 8
8 8

8 8
1
σ
1
σ
1
·
1
1
σ
¸ ]
L
L
M M O M
L
7 σ
!
I
since Co;*ε
i

i′
/78 *#emember that ε
i
T F.30C0.30.D
.*8=σ
!
//
%hat is Co;*Y/&
.ote that all co;ariance matrices are symmetric!
Some basic results
Jet W 7 AY "here Y is a random ;ector and A is a
matrix of constants *no random ;ariables in it/
Dhe follo"ing results follo":
$0*A/ 7 A
© !8$! Christopher # 9ilder
5!,
#emember this is li<e saying 0*1/ 7 1
!0*W/ 7 0*AY/ 7 A0*Y/
(gain= this is li<e saying 0*1Y
$
/710*Y
$
/
1Co;*W/ 7 Co;*AY/ 7 ACo;*Y/A′
You may ha;e seen before that Ear*aY
$
/ 7 a
!
Ear*Y
$
/
"here a is a constant
0xample:
Jet
$
!
$ $ Y
and
$ 8 Y

1 1
· ·
1 1
¸ ] ¸ ]
A Y
Dhen
$ !
$
Y Y
Y

1
·
1
¸ ]
W

!

$ $
! !
$ !
$
$ $ Y $ $ Y
0* / 0 0
$ 8 Y $ 8 Y
0*Y / 0*Y /
0*Y /
¸ _ ¸ _ − −
1 1 1 1
· ·

1 1 1 1
¸ ] ¸ ] ¸ ] ¸ ]
¸ , ¸ ,

1
·
1
¸ ]
W
1
© !8$! Christopher # 9ilder
5!+
$ $
! !
$ $ !
$ ! !
$ $ ! $ ! !
$ $ !
$ $ Y $ $ Y $ $
Co;* / Co; Co;
$ 8 Y $ 8 Y $ 8
$ $ Ear*Y / Co;*Y= Y / $ $
$ 8 Co;*Y= Y / Ear*Y / $ 8
Ear*Y / Co;*Y= Y / Co;*Y= Y / Ear*Y /
Ear*Y / Co;*Y= Y /

¸ _ ¸ _ − − − 1 1 1 1 1
· ·
÷ ÷
1 1 1 1 1
¸ ] ¸ ] ¸ ] ¸ ] ¸ ]
¸ , ¸ ,
− 1 1 1
·
1 1 1

¸ ] ¸ ] ¸ ]
− −
·

¸
W
$ ! $ ! $ $ !
$ $ ! $
$ $
$ 8
Ear*Y / Ear*Y / !Co;*Y= Y / Ear*Y / Co;*Y= Y /
Ear*Y / Co;*Y= Y / Ear*Y /
1 1
1 1

] ¸ ]
+ − − 1
·
1

¸ ]
© !8$! Christopher # 9ilder
5!8
5G Simple linear regression model in matrix terms
Y
i

8

$
S
i

i
"here ε
i
T independent .*8=σ
!
/ and i7$=?=n
Dhe model can be re"ritten as
Y
$

8

$
S
$

$
Y
!

8

$
S
!

!
M
Y
n

8

$
S
n

n
Fn matrix terms= let
$ $ $
! ! 8 !
$
n n n
Y $ S
Y $ S
= 7 = and
Y $ S
ε
1 1 1
1 1 1
β ε
1
1 1 1
· · ·
1
1 1 1 β
¸ ]
1 1 1
ε
¸ ] ¸ ] ¸ ]
Y X β ε
M M M M
Dhen Y 7 Xβ K ε= "hich is
8 $ $ $ $ $ $
8 ! $ ! ! ! 8 !
$
8 n $ n n n n
S Y $ S
S Y $ S
S Y $ S
β + β + ε ε
1 1 1 1
1 1 1 1
β + β + ε β ε
1
1 1 1 1
· + ·
1
1 1 1 1 β
¸ ]
1 1 1 1
β + β + ε ε
¸ ] ¸ ] ¸ ] ¸ ]
M M M M M
.ote that 0*Y/ 7 0*Xβ K ε/ 7 0*Xβ/ K 0*ε/ 7 Xβ since 0*ε/ 7
0 and Xβ are constants
© !8$! Christopher # 9ilder
5!G
5$8 Jeast s@uares estimation of regression parameters
From Chapter $: Dhe least s@uares method tries to find
the b
8
and b
$
such that SS0 7 Σ*Y>
Y
X
/
!
7 Σ*residual/
!
is
minimi5ed Formulas for b
8
and b
$
are deri;ed using
calculus
Ft can be sho"n that b
8
and b
$
can be found from sol;ing
the 'normal e@uations):
n n
8 $ i i
i $ i $
n n n
!
8 i $ i i i
i $ i $ i $
nb b S Y
b S b S SY
· ·
· · ·
+ ·
∑ ∑
+ ·
∑ ∑ ∑
Dhe normal e@uations can be re"ritten as
′ ′
· X Xb X Y
"here
8
$
b
b
1
·
1
¸ ]
b
Fn Section 51= it "as sho"n that
$
n
i
!
i $
n
$ ! n
i i
i $
n
Y
Y
$ $ $ Y
S S S
SY
Y
·
·
1
1
1 ∑
1 1
1

· ·
1 1
1
¸ ]
1

1
¸ ]
¸ ]
X Y
L
L M
and
X′X
$
n
i
!
i $
n n
!
$ ! n
i i
i $ i $
n
$ S
n S
$ $ $ $ S
S S S
S S
$ S
·
· ·
1
1
1 ∑
1 1
1
· ·
1 1
1
¸ ]
1
∑ ∑
1
¸ ]
¸ ]
L
L M M

Dhus=
′ ′
· X Xb X Y can be "ritten as
© !8$! Christopher # 9ilder
518
n n
i i
8
i $ i $
n n n
!
$
i i i i
i $ i $ i $
n S Y
b
b
S S SY
· ·
· · ·
1 1
∑ ∑
1 1 1
·
1 1 1
¸ ]
1 1
∑ ∑ ∑
¸ ] ¸ ]

n n
8 $ i i
i $ i $
n n n
!
8 i $ i i i
i $ i $ i $
nb b S Y
b S b S SY
· ·
· · ·
1 1
+
∑ ∑
1 1
·
1 1
+ 1 1
∑ ∑ ∑
¸ ] ¸ ]
Suppose both sides of
′ ′
· X Xb X Y are multiplied by
*X′X/
>$
Dhen
$ $
$
$
* / * /
* /
* /
− −


′ ′ ′ ′
·
′ ′
·
′ ′
·
X X X Xb X X X Y
Ib X X X Y
b X X X Y
Dherefore= "e ha;e a "ay to find b using matrix algebra!
.ote that
$
$
n n
i i
i $ i $
n n n
!
i i i i
i $ i $ i $
* /
n S Y
S S SY


· ·
· · ·
′ ′
·
1 1
∑ ∑
1 1
·
1 1
1 1
∑ ∑ ∑
¸ ] ¸ ]
b X X X Y
© !8$! Christopher # 9ilder
51$
( )
n n n
!
i i i
i $ i $ i $
!
n n
n n
!
i i i
i i
i $ i $
i $ i $
S S Y
$
S n SY
n S S
· · ·
· ·
· ·
1 1

∑ ∑ ∑
1 1
·
1 1
− 1 1
∑ ∑

∑ ∑
¸ ] ¸ ]
( ) ( ) ( ) ( )
( ) ( )
n n n n
!
i i i i i
i $ i $ i $ i $
n
n n n
!
i
i i i i
i $
i $ i $ i $
Y S S SY
$
n *S S/
Y S n SY
· · · ·
·
· · ·
1

∑ ∑ ∑ ∑
1
·
1
1 −

− +
∑ ∑ ∑
1
¸ ]
$
i i
!
i
Y b S
*S S/*Y Y/
*S S/
1

1
·
− −

1
1


¸ ]
0xample: AS and College BC( *ASIcollegeIBC(Ich5#/
#emember that
$ 182
$ !15
$ !+8
$ !!8
$ $88
1
1
1
1
·
1
1
1
1
1
¸ ]
X
M M
and
1$8
!18
188
!!8
$,8
1
1
1
1
·
1
1
1
1
1
¸ ]
Y
M
Find
$
* /

′ ′
· b X X X Y %e already did this on p 5!!
> "ol.,(t(C)010C) 010 t(C)010E
#,1$
#1,$ '-3'35336
#2,$ '-6**65(4
© !8$! Christopher # 9ilder
51!
> 4"in; t6, 9orm+lation o9 "ol., 9or x in Ax = b
> "ol.,(t(C)010C, t(C)010E)
#,1$
#1,$ '-3'35336
#2,$ '-6**65(4
From pre;ious output:
> mod-9it<-lm(9orm+la = /oll,;,-ABA F @?-ABA, data = ;:a)
> mod-9itDco,99ici,nt"
(Hnt,rc,:t) @?-ABA
'-3'35336 '-6**65(4
© !8$! Christopher # 9ilder
511
5$$ Fitted ;alues and residuals
Jet
$
n
X
Y
X
X
Y
1
1
·
1
1
¸ ]
Y M
Dhen
X
· Y Xb
since
$ $ 8 $ $
8
$
n 8 $ n
n
X
Y $ S b b S
b
b
X
$ S b b S
Y
1
+
1 1
1 1
1 1
· ·
1 1
1 1
¸ ]
1
1 1 +
¸ ] ¸ ]
¸ ]
M M M M
Aat matrix:
$
X
* / Y


· · · Y Xb X X X X YΗY "here H7X*X′X/
>$
X′
is the hat matrix
%hy is this called the Aat matrix&
%e "ill use this in Chapter $8 to measure the influence
of obser;ations on the estimated regression line
#esiduals:
Jet
$ $ $
n
n n
X
e Y Y
X
* /
X
e
Y Y
1

1
1
1
· · · − · − · −
1
1
1
1

¸ ]
¸ ]
e Y Y Y HY Y I H M M
Co;ariance matrix of the residuals:
Co;*e/ 7 Co;*Y*I>H// 7 *I>H/Co;*Y/*I>H/′
© !8$! Christopher # 9ilder
512
.o"= Co;*Y/ 7 Co;*Xβ K ε/ 7 Co;*ε/ since Xβ are
constants (nd= Co;*ε/ 7 σ
!
I
(lso= *I>H/ 7 *I>H/′ and *I>H/*I>H/ 7 *I>H/ *I>H/ is called a
symmetric 'idempotent) matrix You can use matrix
algebra to see this result on your o"n *replace H "ith
X*X′X/
>$
X′ and multiply out/
Dhus= Co;*e/ 7 *I>H/σ
!
I*I>H/′ 7 σ
!
*I>H/
Dhe estimated co;ariance matrix of the residuals is then
·
!
Co;* / MS0 * / X * /= say · ∗ − · σ ∗ − e I H I H .ote that -..
"ould denote
·
Co;* / e as s
!
VeW Dhe
·
Co;* / e notation is
more predominantly used
© !8$! Christopher # 9ilder
515
5$! (nalysis of ;ariance results
Sums of S@uares
From Chapters $ and !:
n
!
i
i $
SSDM *Y Y/
·
· −
∑ =
n
!
i
i $
X
SS0 *Y Y/
·
· −
∑ = and
n
!
i
i $
X
SS# SSDM SS0 *Y Y/
·
· − · −

Dhese can be re"ritten using matrices:
[ ] [ ]
( )
$ $
$ n $ n
n n
$
n n n
!
i i i
i $ i $ i $
n
!
n n
!
i i
i $ i $
n
!
i
i $
$
SSDM
n
Y $ $ Y
$
Y Y Y Y
n
Y $ $ Y
Y
$
Y Y Y
n
Y
$
Y Y
n
*Y Y/
· · ·
· ·
·
′ ′
· −
1 1 1
1 1 1
· −
1 1 1
1 1 1
¸ ] ¸ ] ¸ ]
1
1 1
· −
∑ ∑ ∑
1 1
¸ ]
1
¸ ]
· −
∑ ∑
· −

Y Y Y JY
L
L M L M O M M
L
L M
[ ]
$
n n
! !
$ n i i i
i $ i $
n
e
X
SS0 e e e *Y Y/
e
· ·
1
1

· · · · −
∑ ∑
1
1
¸ ]
e e L M
© !8$! Christopher # 9ilder
51,
$
SS# SSDM SS0
n
′ ′ ′
· − · − b X Y Y JY
0xample: AS and College BC( *ASIcollegeIBC(Ich5#/
Continuing the same program
> n<-l,n;t6(E) 4/an not +", row" 6,r, "inc, R do," not
Know EI" dim,n"ion
> 4/o+ld +", nrow(C) a" w,ll - C i" a nx2
matrix
> b<-"ol.,(t(C)010C) 010 t(C)010E
> E-6at<-C010b
> ,<-E-E-6at
> @<-C010"ol.,(t(C)010C)010t(C)

> L<-matrix(data = 1, nrow = n, ncol = n)
> 4=otic, t6at R will r,:,at 1 t6, corr,ct n+mb,r o9
tim," to 9ill t6, matrix
> ??TM<-t(E)010E - 1Nn010t(E)010L010E
> ??!<-t(,)010,
> O?!<-??!N(n-nrow(b))

> 4=otic, 6ow O?! +"," 1 to do t6, m+lti:lyin; "inc, it i"
a "calar- Al"o dia;(n) cr,at," an id,ntity matrix-
2ia;(c(1,2)) wo+ld cr,at, a 2x2 dia;onal matrix wit6
,l,m,nt" 1 and 2 on t6, dia;onal
> /o.-,<-O?!1(dia;(n) - @) 42o," not worKP
!rror in O?! 1 (dia;(n) - @) 8 non-con9ormabl, array"
> /o.-,<-a"-n+m,ric(O?!)1(dia;(n) - @) 4T6, a"-n+m,ric()
r,mo.," 1x1 matrix
m,anin; 9rom O?!
> ??R = ??TM - ??!

> data-9ram,(C, E, E-6at, ,)
C1 C2 E E-6at ,
© !8$! Christopher # 9ilder
51+
1 1 3-'4 3-1 2-(3453* '-26546'*1
2 1 2-35 2-3 2-351335 -'-'51334(2
3 1 2-3' 3-' 2-5*6655 '-4'334435
4 1 2-'5 1-* 2-141(33 -'-241(3331
5 1 2-(3 2-5 2-6(3611 -'-1(361'(3
6 1 4-32 3-3 3-33'1'2 -'-'3'1'1(1
3 1 3-3* 3-4 3-'3*42' '-32'5('4(
( 1 2-32 2-6 2-33'3(5 '-26*214*3
* 1 2-6* 2-( 2-5(*65* '-21'34134
1' 1 '-(3 1-6 1-2((2*4 '-3113'5*1
11 1 2-3* 2-' 2-33*361 -'-33*36115
12 1 3-65 2-* 3-261331 -'-36133'3'
13 1 1-(5 2-3 2-''1*46 '-2*('5433
14 1 3-(3 3-2 3-3(326* -'-1(326*21
15 1 1-22 1-( 1-561161 '-23((3*14
16 1 1-4( 1-4 1-343'32 -'-343'32'3
13 1 2-2( 2-' 2-3'23** -'-3'23*(33
1( 1 4-'' 3-( 3-5'6211 '-2*33(((3
1* 1 2-2( 2-2 2-3'23** -'-1'23*(33
2' 1 1-(( 1-6 2-'22*35 -'-422*353(

> data-9ram,(n, ??TM, ??!, O?!, ??R)
n ??TM ??! O?! ??R
1 2' *-64*5 1-5(3*66 '-'((22'35 (-'61534

> ro+nd(/o.-,#185, 185$,6) 45x5 :art o9 /o.(,)
#,1$ #,2$ #,3$ #,4$ #,5$
#1,$ '-'(2621 -'-''3(5( -'-''4342 -'-''31'1 -'-''5'3'
#2,$ -'-''3(5( '-'(3552 -'-''4253 -'-''5'2' -'-''41'5
#3,$ -'-''4342 -'-''4253 '-'(3313 -'-''4'43 -'-''45*4
#4,$ -'-''31'1 -'-''5'2' -'-''4'43 '-'(2366 -'-''36(5
#5,$ -'-''5'3' -'-''41'5 -'-''45*4 -'-''36(5 '-'(3444
> 4Qrom :a"t worK
> data-9ram,(C = mod,l-matrix(mod-9it), E-6at =
mod-9itD9itt,d-.al+,", , = mod-9itDr,"id+al")
C--Hnt,rc,:t- C-@?-ABA E-6at ,
1 1 3-'4 2-(3453* '-26546'*1
2 1 2-35 2-351335 -'-'51334(2
3 1 2-3' 2-5*6655 '-4'334435
4 1 2-'5 2-141(33 -'-241(3331
5 1 2-(3 2-6(3611 -'-1(361'(3
6 1 4-32 3-33'1'2 -'-'3'1'1(1
© !8$! Christopher # 9ilder
518
3 1 3-3* 3-'3*42' '-32'5('4(
( 1 2-32 2-33'3(5 '-26*214*3
* 1 2-6* 2-5(*65* '-21'34134
1' 1 '-(3 1-2((2*4 '-3113'5*1
11 1 2-3* 2-33*361 -'-33*36115
12 1 3-65 3-261331 -'-36133'3'
13 1 1-(5 2-''1*46 '-2*('5433
14 1 3-(3 3-3(326* -'-1(326*21
15 1 1-22 1-561161 '-23((3*14
16 1 1-4( 1-343'32 -'-343'32'3
13 1 2-2( 2-3'23** -'-3'23*(33
1( 1 4-'' 3-5'6211 '-2*33(((3
1* 1 2-2( 2-3'23** -'-1'23*(33
2' 1 1-(( 2-'22*35 -'-422*353(
> ano.a(mod-9it)
Analy"i" o9 Rarianc, Tabl,
R,":on",8 /oll,;,-ABA
29 ?+m ?S O,an ?S Q .al+, Br(>Q)
@?-ABA 1 (-'615 (-'615 *1-3( 1-33*,-'( 111
R,"id+al" 1( 1-5((' '-'((2
---
?i;ni9- cod,"8 ' I111I '-''1 I11I '-'1 I1I '-'5 I-I '-1 I
I 1
© !8$! Christopher # 9ilder
51G
5$1 Fnferences in #egression (nalysis
Co;ariance matrix of b:
8 8 $
! $
8 $ $
Ear*b / Co;*b =b /
Co;* / * /
Co;*b =b / Ear*b /

1

· · σ
1
¸ ]
b X X
%hy&
Co;*b/ 7 Co;Q*X′X/
>$
X′YR
7 *X′X/
>$
X′Co;*Y/Q*X′X/
>$
X′R′
7 *X′X/
>$
X′Co;*Y/X*X′X/
>$
7 *X′X/
>$
X′σ
!
IX*X′X/
>$
7 σ
!
*X′X/
>$
X′X*X′X/
>$
7 σ
!
*X′X/
>$
#emember that for t"o matrices C and B= *CB/′ 7
B′C′
0stimated co;ariance matrix of b:
·
· ·
· ·
8 8 $
8 $ $
!
! !
i i
! !
i i
$
Ear*b / Co;*b =b /
Co;* /
Co;*b =b / Ear*b /
$ S S
n *S S/ *S S/
MS0
S $
*S S/ *S S/
MS0 * /

1
· 1
1
¸ ]
1

+
1
− −
∑ ∑
1
·
1

1
− − 1 ∑ ∑
¸ ]

· ∗
b
X X
© !8$! Christopher # 9ilder
528
.ote that the
·
i
Ear*b / is from Sections !$ and !! Dhe
·
8 $
Co;*b =b / is ne"
0stimated ;ariance used in the CF for 0*Y
h
/:

·
!
h
$
h h h
!
i
$ *S S/
X
Ear*Y / MS0* * / / MS0
n *S S/

¸ _

′ ′
· · +
÷


¸ ,
X X X X
"here X
h
7 Q$= S
h
R′
%hy&
h h
h h
! $
h h
X
Ear*Y / Ear* /
Co;* /
* /


·

·
′ ′
· σ
X b
X b X
X X X X
Since b is a ;ector= F replaced Ear*/ "ith Co;*/ *:ust
notation/
0stimated ;ariance used in the CF for Y
h*ne"/
:

·
$
h*ne"/ h h h
!
h
!
i
X
Ear*Y Y / MS0*$ * / /
$ *S S/
MS0 $
n *S S/

′ ′
− · +
¸ _

· + +
÷


¸ ,
X X X X
0xample: AS and College BC( *ASIcollegeIBC(Ich5#/
> co.-b,ta-6at<-a"-n+m,ric(O?!)1"ol.,(t(C)010C)
> co.-b,ta-6at
#,1$ #,2$
#1,$ '-'3*366'6 -'-'133621(1
© !8$! Christopher # 9ilder
52$
#2,$ -'-'133621( '-''5353'1*

> 4Qrom ,arli,r
> "+m-9it<-"+mmary(mod-9it)
> "+m-9itDco.-+n"cal,d 4Anot6,r way to ;,t (CIC)J(-1)
(Hnt,rc,:t) @?-ABA
(Hnt,rc,:t) '-45'35(4 -'-155**3(1
@?-ABA -'-155**3( '-'6'32316
> "+m-9itD"i;maJ2 1 "+m-9itDco.-+n"cal,d
(Hnt,rc,:t) @?-ABA
(Hnt,rc,:t) '-'3*366'6 -'-'133621(1
@?-ABA -'-'133621( '-''5353'1*
> 4Mn, mor, way to ;,t t6, co.arianc, matrix
> .co.(mod-9it)
(Hnt,rc,:t) @?-ABA
(Hnt,rc,:t) '-'3*366'6 -'-'133621(1
@?-ABA -'-'133621( '-''5353'1*
> 4Qind Rar(EJ)
> C-6<-c(1,1-(()
> a"-n+m,ric(O?!)1C-6010"ol.,(t(C)010C)010C-6
#,1$
#1,$ '-''6*541'3

> 4Qind Rar(E-EJ)
> a"-n+m,ric(O?!)1 (1)C-6010"ol.,(t(C)010C)010C-6)
#,1$
#1,$ '-'*513446

> 4Qrom @?Tcoll,;,TABATc62-R
> n<-nrow(;:a)
> C-6<-1-((
> E-6at-6<-a"-n+m,ric(mod-9itDco,99ici,nt"#1$ )
mod-9itDco,99ici,nt"#2$1C-6)
4a"-n+m,ric G+"t r,mo.," an +nn,,d,d nam,
> ""x<-.ar(;:aD@?-ABA)1(n-1) 4?O( (CTi - CTbar)J2 )
> C-bar<-m,an(;:aD@?-ABA)
> O?! 1 (1Nn ) (C-6 - C-bar)J2 N ""x) 4TaK,n 9rom t6,
/-H- 9orm+la
#,1$
© !8$! Christopher # 9ilder
52!
#1,$ '-''6*541'3
> O?! 1 (1 ) 1Nn ) (C-6 - C-bar)J2 N ""x) 4TaK,n 9rom
t6, B-H- 9orm+la
#,1$
#1,$ '-'*513446
© !8$! Christopher # 9ilder
521