STAT 714 MATRIX ALGEBRA REVIEW 4

REVIEW : Consider the system of equations Ax = c. If A is square and nonsingular,

then there is a unique solution to the system and it is x = A−1 c. If A is not nonsingular,
then the system can have no solution, finitely many solutions, or infinitely many solutions.

TERMINOLOGY : The linear system Ax = c is consistent if there exists an x∗ such

that Ax∗ = c; that is, if c ∈ C(A).

REMARK : We will show that

• for every m × n matrix A, there exists a n × m matrix G such that AGA = A.

• for a consistent system Ax = c, if AGA = A, then x∗ = Gc is a solution.

Result MAR4.1. Suppose that Ax = c is consistent. If G is a matrix such that

AGA = A, then x∗ = Gc is a solution to Ax = c.
Proof. Because Ax = c is consistent, there exists an x∗ such that Ax∗ = c. Note that
AGc = AGAx∗ = Ax∗ = c. Thus, x∗ = Gc is a solution. 

TERMINOLOGY : A matrix G that satisfies AGA = A is called a generalized inverse

of A and is denoted by A− . That is,

AGA = A =⇒ AA− A = A.

If A is square and nonsingular, then the generalized inverse of A is A−1 since AA− A =
AA−1 A = A.

NOTES :
• Every matrix A, regardless of its dimension, has a generalized inverse.
• Generalized inverses are not unique unless A is nonsingular.
• If A is m × n, then A− is n × m.
• A generalized inverse of A, A symmetric, is not necessarily symmetric. However, a
symmetric generalized inverse can always be found. We will thus assume that the
generalized inverse of a symmetric matrix is symmetric.
• If G is a generalized inverse of A, then G0 is a generalized inverse of A0 .
• Monahan uses Ag to denote generalized inverse, but I will use A− .

Example. Consider the matrices

   
4 1 2 1/3 −1/3 0
A= 1 1 5  and G =  −1/3 4/3 0  .
3 1 3 0 0 0

Note that r(A) = 2 because −a1 + 6a2 − a3 = 0. Thus A−1 does not exist. However, it
is easy to show that AGA = A; thus, G is a generalized inverse of A. 

PAGE 1
STAT 714 MATRIX ALGEBRA REVIEW 4

Result MAR4.2. Let A be an m × n matrix with r(A) = r. If A can be partitioned

as follows  
C D
A= ,
E F
where r(A) = r(C) = r, and Cr×r is nonsingular, then
 −1 
C 0
G=
0 0

is a generalized inverse of A. This result essentially shows that every matrix has a
generalized inverse (see Results A.10 and A.11, Monahan). Also, it gives a method to
compute it.

COMPUTATION : This is an algorithm for finding a generalized inverse A− for A, any

m × n matrix of rank r.

1. Find any r × r nonsingular submatrix C. It is not necessary that the elements of

C occupy adjacent rows and columns in A.

2. Find C−1 and (C−1 )0 .

3. Replace the elements of C by the elements of (C−1 )0 .

4. Replace all other elements of A by zeros.

5. Transpose the resulting matrix.

Result MAR4.3. Let Am×n , xn×1 , cm×1 , and In×n be matrices, and suppose that
Ax = c is consistent. Then, x∗ is a solution to Ax = c if and only if

x∗ = A− c + (I − A− A)z,

for some z ∈ Rn . Thus, we can generate all solutions by just knowing one of them; i.e.,
by knowing A− c.
Proof. (⇐=) We know that x∗ = A− c is a solution (Result MAR4.1). Suppose that
x∗ = A− c + (I − A− A)z, for some z ∈ Rn . Thus,

Ax∗ = AA− c + (A − AA− A)z

= AA− c = Ax∗ = c;

that is, x∗ = A− c + (I − A− A)z solves Ax = c. Conversely, (=⇒) suppose that x∗ is a

solution to Ax = c. Now,

x ∗ = A− c + x ∗ − A− c
= A− c + x∗ − A− Ax∗ = A− c + (I − A− A)x∗ .

Thus, x∗ = A− c + (I − A− A)z, where z = x∗ . Note that if A is nonsingular, A− = A−1

and x∗ = A−1 c + (I − A−1 A)z = A−1 c; i.e., there is just one solution. 

PAGE 2
STAT 714 MATRIX ALGEBRA REVIEW 4

NOTE : Consider the general form of the solution to Ax = c (which is assumed to be

consistent); i.e., x∗ = A− c+(I−A− A)z. We call A− c a particular solution. The term
(I − A− A)z is the general solution to the homogeneous equations Ax = 0, producing
vectors in N (A).

COMPARE : Suppose that X1 , X2 , ..., Xn is an iid sample from fX (x; θ) and let X =
(X1 , X2 , ..., Xn )0 . Suppose also that θb1 = θb1 (X) is an unbiased estimator of θ; that is,
Eθ [θb1 (X)] = θ for all θ ∈ Θ, say. The general form of an unbiased estimator for θ is

θe = θb1 + T,

where Eθ (T ) = 0 for all θ ∈ Θ.

APPLICATION : Consider the general linear model

Y = Xβ + ,

where Y is an n × 1 vector of observed responses, X is an n × p matrix of rank r < p, β is

a p × 1 vector of fixed but unknown parameters, and  is an n × 1 vector of (unobserved)
random errors. The normal equations are given by

X0 Xβ = X0 Y.

The normal equations are consistent (see below). Thus, the general form of the least
squares estimator is given by
b = (X0 X)− X0 Y + [I − (X0 X)− X0 X]z,
β

where z ∈ Rp . Of course, if r(X) = p, then (X0 X)−1 exists, and the unique solution
becomes
b = (X0 X)−1 X0 Y.
β

PROPOSITION : The normal equations X0 Xβ = X0 Y are consistent.

Proof. First, we will state and prove the following lemma.
LEMMA: For any matrix X, and for any matrices A and B,

X0 XA = X0 XB ⇐⇒ XA = XB.

Proof. The necessity part (⇐=) is obvious. For the sufficiency part (=⇒), note that

X0 XA = X0 XB =⇒ X0 XA − X0 XB = 0
=⇒ (A − B)0 (X0 XA − X0 XB) = 0
=⇒ (A − B)0 X0 (XA − XB) = 0
=⇒ (A0 X0 − B0 X0 )(XA − XB) = 0
=⇒ (XA − XB)0 (XA − XB) = 0.

PAGE 3
STAT 714 MATRIX ALGEBRA REVIEW 4

This can only be true if XA − XB = 0. Thus, the lemma is proven. Now, let (X0 X)−
denote a generalized inverse of X0 X so that X0 X(X0 X)− X0 X = X0 X. Taking A0 =
X0 X(X0 X)− and B0 = I in the lemma, we have

X0 X(X0 X)− X0 X = X0 X =⇒ X0 X(X0 X)− X0 = X0

=⇒ X0 X(X0 X)− X0 Y = X0 Y.
b = (X0 X)− X0 Y is a solution to the normal equations. Hence, the
This implies that β
normal equations are consistent. 

Example. Consider the one-way fixed effects ANOVA model

Yij = µ + αi + ij ,

for i = 1, 2 and j = 1, 2, ..., ni , where n1 = 2 and n2 = 3. It is easy to show that

 
5 2 3
0
XX=  2 2 0 .
3 0 3
One generalized inverse of X0 X is
 
0 0 0
(X0 X)−
1 =
 0 1/2 0  ,
0 0 1/3

and a solution to the normal equations (based on this generalized inverse) is

  
0 0 0 Y11 + Y12 + Y21 + Y22 + Y23
b 1 = (X0 X)− X0 Y =  0 1/2 0  
β Y11 + Y12 
1
0 0 1/3 Y21 + Y22 + Y23
   
0 0
1
=  2
(Y11 + Y12 )  =  Y 1+  .
1
3
(Y21 + Y22 + Y23 ) Y 2+
Another generalized inverse of X0 X is
 
1/3 −1/3 0
(X0 X)−
2 =
 −1/3 5/6 0  ,
0 0 0

and a solution to the normal equations (based on this generalized inverse) is

  
1/3 −1/3 0 Y11 + Y12 + Y21 + Y22 + Y23
b 2 = (X0 X)− X0 Y =  −1/3 5/6 0  
β Y11 + Y12 
2
0 0 0 Y21 + Y22 + Y23
1
   
3
(Y21 + Y22 + Y23 ) Y 2+
=  12 (Y11 + Y12 ) − 13 (Y21 + Y22 + Y23 )  =  Y 1+ − Y 2+  .
0 0

PAGE 4
STAT 714 MATRIX ALGEBRA REVIEW 4

The general solution is given by

b = (X0 X)− X0 Y + [I − (X0 X)− X0 X]z
β 1 1
    
0 1 0 0 z1
=  Y 1+  +  −1 0 0   z2 
Y 2+ −1 0 0 z3
 
z1
=  Y 1+ − z1  ,
Y 2+ − z1

where z = (z1 , z2 , z3 )0 ∈ R3 . Furthermore, we see that the first particular solution

corresponds to z1 = 0 while the second corresponds to z1 = Y 2+ . 

PAGE 5