Professional Documents
Culture Documents
pt
UNIVERSITY OF ZIMBABWE
May 17, 2021
de
1 Introduction
Def: A random vector is a vector whose elements are random variables.
Def: The vector E(X) = µ = (µ1 , µ2 , · · · , µp )0 is called the population mean
s
vector. The matrix E(X − µ)(X − µ)0 = V ar(X) = Σ is called the variance-
at
covariance matrix or the dispersion matrix.
Def: Let X be a p component random vector and Y be a q component
radom vector, then the covariance between X and Y is given by the following
Cov(X, Y) = E(X − µ)(Y − α)0
st
where µ = E(X)
α = E(Y)/
The correlation between the ith and j th variables is given by
σ
ρij = √σiiijσjj and the matrix ρ = [ρij ] with ρii = 1, i = 1, 2, · · · , p is called
the population correlation matrix.
uz
2 Useful results
For a random vector X with mean µ and A and B are constants, we have
z-
3. Σ = E(X0 X) − µµ0
4. V ar(a0 X) = a0 V ar(X)a =
PP
ai aj σij
1
5. V ar(AX + b) = AV ar(X)A0 since V ar(b) = 0
6. Cov(X, X) = V ar(X).
7. Cov(X, Y) = Cov(Y, X)0
8. Cov((X1 + X2 )0 Y) = Cov(X1 , Y) + Cov(X2 , Y).
pt
9. If p = q, V ar(X + Y) = V ar(X) + Cov(X, Y) + Cov(Y, X) + V ar(Y).
10. X and Y are independent then Cov(X, Y) = 0.
Note that
de
1. The measurements of the p− variables on the individuals will usually
be correlated, whilst measurements from different individuals must be
independent.
2. The violation of the assumption of the independence can have serious
s
impact on the quality of statistical inference.
at
3 The sample statistics x and S as estimators
of the respective population parameters
st
E(Sn ) = n−1
n
Σ = Σ − n1 Σ.
Proof
z-
X1 +X2 +···+Xn
E(x)=E
lsi
=E( n1 X1 ) + E( n1 X2 ) + · · · + E( n1 Xn )
= n1 µ1 + n1 µ2 + · · · + n1 µn ,
2
but since the sample is from a joint distribution, it implies that
µ1 = µ2 = · · · = µn = µ
∴ E(x) = µ
pt
h P 0 i
=E n1 Xj − µ n1
P
Xj − µ
de
h P P i
=E n12 nj=1 nl=1 (Xj − µ)(Xl − µ)0
= n12 nj=1 nl=1 E(Xj − µ)(Xl − µ)0 (Summation and Expectation are
P P
linear operators so they are commutative/their order of operation can be
interchanged.
s
for j 6= l, each entry E(Xj − µ)(Xl − µ)0 = 0
at
because covariance between component Xj and Xl is zero since these are
independent. P
∴ Cov(x) = n12 nj=1 E(Xj − µ)(Xl − µ)0 .
st
1
= n2
(Σ + Σ + · · · + Σ).
uz
1
= n2
nΣ.
= n1 Σ.
z-
Question
Show that the sample covariance matrix S is an unbiased estimator of Σ, i.e.
lsi
ES = Σ.
Soln
h Pn i
1 0
E(S) = E n−1 j=1 (X j − x)(X j − x)
hP i
1 n 0 0 0 0
= n−1 E j=1 (X j X j − X j x − xX j + xx)
3
hP i
n Pn Pn Pn
= 1
n−1
E j=1 Xj X0j − 0
j=1 Xj x −
0
j=1 xXj + j=1 xx
0
Pn
Pn Pn 0 j=1 Xj
but j=1 Xj x = ( j=1 Xj )x and n
=x
Pn Pn Pn
∴ j=1 xX0j =
Xj x = j=1 j=1 xx0 = nxx0 .
hP i
1 n 0 0
= n−1 j=1 E(Xj Xj ) − nE(xx )
pt
Since V ar(x) = E(xx0 ) − µµ0
0
⇒ E(xx ) = V ar(x) + µµ0 = n1 Σ + µµ0
de
Since V ar(Xj ) = E(Xj X0j ) − µµ0
⇒ E(Xj X0j ) = V ar(x) + µµ0 = Σ + µµ0
it then follows
hPthat i
1 n 0 1 0
E(S) = n−1 j=1 (Σ + µµ ) − n( n
Σ + µµ )
1
n(Σ + nµµ0 ) − n( n1 Σ − nµµ0 )
= n−1 s
1
= [nΣ − Σ]
at
n−1
n−1
= n−1
Σ
st
= Σ.
xi1 xi2 · · · xij · · · xin
. .. .. ..
.. . . .
lsi
4
x11 x12 · · · x1j · · ·
x1n
x21 x22 · · · x2j · · ·
x2n 1
.. .. .. ..
1
. . . .
x=
xi1 xi2 · · · xij · · · xin
..
.
.. .. .. .. 1
. . . .
pt
xp1 xp2 · · · xpj · · · xpn
i.e.
1
x= X1
n
de
where 1 is a vector of ones.
X− X110 =
.. .. ..
n
. . .
xp1 − xp xp2 − xp · · · xpn − xp
by definition
0
uz
Pn Pn Pn
i=1 (x1i − x1 )(x1i − x1 ) Pi=1 (x1i − x1 )(x2i − x2 ) · · · Pi=1 (x1i − x1 )(xpi − xp )
n (x2i − x2 )(x1i − x1 )
P n n
i=1 (x2i − x2 )(x2i − x2 ) · · · i=1 (x2i − x2 )(xpi − xp )
lsi
i=1
=
.. .. ..
Pn . Pn . Pn .
i=1 (xpi − xp )(x1i − x1 ) i=1 (xpi − xp )(x2i − x2 ) · · · i=1 (xpi − xp )(xpi − xp )
1 1
= (X − X110 )(X − X110 )0
n n
1 1
= (X − X110 )(X0 − 110 X0 )
n n
5
1 0 1
11 )(I − 110 )X0
= X(I −
n n
1 0 0
= X(I − 11 )X ,
n
since 11 = n. ∴ x = n X1 and S = X(I − n1 110 )X0
0 1
pt
5 Sample values of linear combinations of vari-
ables
Suppose that you are given two linear combinations b0 X and c0 X where
de
b0 X = b1 X1 + b2 X2 + · · · + bp Xp whose observed value on the j th trial is
b0 xj = b1 x1j + b2 x2j + · · · + bp xpj , j = 1, 2, · · · , n.
b0 x1 +b0 x2 +···+b0 xn
Sample mean = n
n−1
= b0 Sb
Sample variance of c0 X = c0 Sc
(b0 x1 −b0 x)(c0 x1 −c0 x)0 +(b0 x2 −b0 x)(c0 x2 −c0 x)0 +···+(b0 xn −b0 x)(c0 xn −c0 x)0
= n−1
6
= b0 Sc
Similarly Population mean of c0 X = E(c0 X) = c0 µ
Population variance of c0 X = c0 Σc
pt
Example
9 5 1
Consider the data matrix A = .
1 3 2
de
We have n = 3 observations on p = 2 variables X1 and X2 . Form the linear
combinations
0
X1
bX= 2 3 = 2X1 + 3X2
X2
X1
c0 X = −1 2
= −X1 + 2X2
X2 s
Calculate the sample means, variances and covariances of b0 X and c0 X
at
Solution
9+5+1
0 0
5
Sample mean of b X = b x = 2 3 3 = 2 3
1+3+2
3
2
= 10 + 6 = 16
st
0 0
5
Sample mean of c X = c x = −1 2
2
= −5 + 4 = −1
uz
16 −2
S=
−2 1
0
16 −2 2
Sample variance of b X = 2 3 = 49
z-
−2 1 3
0
16 −2 −1
Sample variance of c X = −1 2 = 28
−2 1 2
lsi
0 0
16 −2 −1
Sample covariance of b X and c X = 2 3
−2 1 2
= −28