184

Chapter 3. Vector Spaces and Subspaces

3.6 Dimensions of the Four Subspaces

The main theorem in this chapter connects rank and dimension. The rank of a matrix is the number of pivots. The dimension of a subspace is the number of vectors in a basis. We count pivots or we count basis vectors. The rank of A reveals the dimensions of all four fundamental subspaces. Here are the subspaces, including the new one. Two subspaces come directly from A, and the other two from A T : Four Fundamental Subspaces 1. The row space is C .AT /, a subspace of Rn . 2. The column space is C .A/, a subspace of Rm . 3. The nullspace is N .A/, a subspace of Rn . 4. The left nullspace is N .AT /, a subspace of Rm . This is our new space. In this book the column space and nullspace came rst. We know C .A/ and N .A/ pretty well. Now the other two subspaces come forward. The row space contains all combinations of the rows. This is the column space of A T . For the left nullspace we solve A T y D 0that system is n by m. This is the nullspace of AT . The vectors y go on the left side of A when the equation is written as y T A D 0T . The matrices A and AT are usually different. So are their column spaces and their nullspaces. But those spaces are connected in an absolutely beautiful way. Part 1 of the Fundamental Theorem nds the dimensions of the four subspaces. One fact stands out: The row space and column space have the same dimension r (the rank of the matrix). The other important fact involves the two nullspaces: N .A/ and N .AT / have dimensions n r and m r, to make up the full n and m.

Part 2 of the Fundamental Theorem will describe how the four subspaces t together (two in Rn and two in Rm /. That completes the right way to understand every Ax D b. Stay with ityou are doing real mathematics.

The Four Subspaces for R

Suppose A is reduced to its row echelon form R. For that special form, the four subspaces are easy to identify. We will nd a basis for each subspace and check its dimension. Then we watch how the subspaces change (two of them dont change!) as we look back at A. The main point is that the four dimensions are the same for A and R. As a specic 3 by 5 example, look at the four subspaces for the echelon matrix R: 2 3 mD3 1 3 5 0 7 pivot rows 1 and 2 40 0 0 1 25 nD5 r D2 0 0 0 0 0 pivot columns 1 and 4 The rank of this matrix R is r D 2 (two pivots). Take the four subspaces in order.

3.6. Dimensions of the Four Subspaces

185

1. The row space of R has dimension 2, matching the rank.

Reason: The rst two rows are a basis. The row space contains combinations of all three rows, but the third row (the zero row) adds nothing new. So rows 1 and 2 span the row space C .RT /. The pivot rows 1 and 2 are independent. That is obvious for this example, and it is always true. If we look only at the pivot columns, we see the r by r identity matrix. There is no way to combine its rows to give the zero row (except by the combination with all coefcients zero). So the r pivot rows are a basis for the row space. The dimension of the row space is the rank r. The nonzero rows of R form a basis.
2. The column space of R also has dimension r D 2.

Reason: The pivot columns 1 and 4 form a basis for C .R/. They are independent because they start with the r by r identity matrix. No combination of those pivot columns can give the zero column (except the combination with all coefcients zero). And they also span the column space. Every other (free) column is a combination of the pivot columns. Actually the combinations we need are the three special solutions !

Column 2 is 3 (column 1). Column 3 is 5 (column 1).

The special solution is . 3; 1; 0; 0; 0/. The special solution is . 5; 0; 1; 0; 0; /.

Column 5 is 9 (column 1) C 8 (column 4). That solution is . 7; 0; 0; 2; 1/. The pivot columns are independent, and they span, so they are a basis for C .R/. The dimension of the column space is the rank r. The pivot columns form a basis.
3. The nullspace has dimension n r D 5 2. There are n r D 3 free variables. Here x2 ; x3 ; x5 are free (no pivots in those columns). They yield the three special solutions to Rx D 0. Set a free variable to 1, and solve for x1 and x4 :

2 6 6 s2 D 6 6 4

3 3 17 7 07 7 05 0

2 6 6 s3 D 6 6 4

3 5 07 7 17 7 05 0

2 6 6 s5 D 6 6 4

3 7 07 7 07 7 25 1

Rx D 0 has the complete solution : x D x2 s 2 C x3 s 3 C x5 s 5

There is a special solution for each free variable. With n variables and r pivot variables, that leaves n r free variables and special solutions. N .R/ has dimension n r.

186
The nullspace has dimension n

Chapter 3. Vector Spaces and Subspaces

r. The special solutions form a basis.

The special solutions are independent, because they contain the identity matrix in rows 2, 3, 5. All solutions are combinations of special solutions, x D x 2 s2 C x3 s3 C x5 s5 , because this puts x2 , x3 and x5 in the correct positions. Then the pivot variables x 1 and x4 are totally determined by the equations Rx D 0.
4. The nullspace of R T (left nullspace of R) has dimension m

r D3

2.

Reason: The equation R T y D 0 looks for combinations of the columns of R T (the rows

of R) that produce zero. This equation R T y D 0 or y T R D 0T is y1 1; Cy2 0; Cy3 0; 0; 3; 0; 0; 0; 5; 0; 0; 0; 0; 1; 0; 0; 7 2 0 0

Left nullspace

(1)

The solutions y1 ; y2 ; y3 are pretty clear. We need y1 D 0 and y2 D 0. The variable y3 is free (it can be anything). The nullspace of R T contains all vectors y D .0; 0; y3 /. It is the line of all multiples of the basis vector .0; 0; 1/. In all cases R ends with m r zero rows. Every combination of these m r rows gives zero. These are the only combinations of the rows of R that give zero, because the pivot rows are linearly independent. The left nullspace of R contains all these solutions y D .0; ; 0; yrC1 ; ; ym / to R T y D 0. If A is m by n of rank r, its left nullspace has dimension m r.

To produce a zero combination, y must start with r zeros. This leaves dimension m r. Why is this a left nullspace? The reason is that R T y D 0 can be transposed to T y R D 0T . Now y T is a row vector to the left of R. You see the ys in equation (1) multiplying the rows. This subspace came fourth, and some linear algebra books omit itbut that misses the beauty of the whole subject. In Rn the row space and nullspace have dimensions r and n r .adding to n/: In Rm the column space and left nullspace have dimensions r and m r .total m/: So far this is proved for echelon matrices R. Figure 3.5 shows the same for A.

The Four Subspaces for A

We have a job still to do. The subspace dimensions for A are the same as for R. The job is to explain why. A is now any matrix that reduces to R D rref.A/: 2 3 1 3 5 0 7 A reduces to R A D 4 0 0 0 1 2 5 Notice C .A/ C.R/ (2) 1 3 5 1 9

3.6. Dimensions of the Four Subspaces

187

C (AT ) dim r row space all ATy column space all Ax

C (A) dim r

The big picture

Rn left nullspace ATy = 0 N (AT ) dimension m r Rm

N (A) dimension n r

nullspace Ax = 0

Figure 3.5: The dimensions of the Four Fundamental Subspaces (for R and for A). An elimination matrix takes A to R. The big picture (Figure 3.5) applies to both. The invertible matrix E is the product of the elementary matrices that reduce A to R: A to R and back
1

EA D R

and A D E

(3)

A has the same row space as R . Same dimension r and same basis.

Reason: Every row of A is a combination of the rows of R. Also every row of R is a combination of the rows of A. Elimination changes rows, but not row spaces. Since A has the same row space as R, we can choose the rst r rows of R as a basis. Or we could choose r suitable rows of the original A. They might not always be the rst r rows of A, because those could be dependent. The good r rows of A are the ones that end up as pivot rows in R.
2

The column space of A has dimension r. For every matrix this is essential: The number of independent columns equals the number of independent rows.

Wrong reason: A and R have the same column space. This is false. The columns of R often end in zeros. The columns of A dont often end in zeros. The column spaces are different, but their dimensions are the sameequal to r. Right reason: The same combinations of the columns are zero (or nonzero) for A and R. Say that another way: Ax D 0 exactly when Rx D 0. The r pivot columns (of both) are independent. Conclusion The r pivot columns of A are a basis for its column space.

188
3

Chapter 3. Vector Spaces and Subspaces

A has the same nullspace as R. Same dimension n

r and same basis.

Reason: The elimination steps dont change the solutions. The special solutions are a basis for this nullspace (as we always knew). There are n r free variables, so the dimension of the nullspace is n r. Notice that r C .n r/ equals n: .dimension of column space/ C .dimension of nullspace/ D dimension of R n :
4

The left nullspace of A (the nullspace of A T ) has dimension m

r.

Reason: AT is just as good a matrix as A. When we know the dimensions for every A, we also know them for AT . Its column space was proved to have dimension r. Since A T is n by m, the whole space is now R m . The counting rule for A was r C .n r/ D n. The counting rule for AT is r C .m r/ D m. We now have all details of the main theorem: Fundamental Theorem of Linear Algebra; Part 1 The column space and row space both have dimension r: The nullspaces have dimensions n r and m r: By concentrating on spaces of vectors, not on individual numbers or vectors, we get these clean rules. You will soon take them for grantedeventually they begin to look obvious. But if you write down an 11 by 17 matrix with 187 nonzero entries, I dont think most people would see why these facts are true: Two key facts
Example 1

dimension of C .A/ D dimension of C .A T / D rank of A dimension of C .A/ C dimension of N .A/ D 17:

A D 1 2 3 has m D 1 and n D 3 and rank r D 1.

The row space is a line in R3 . The nullspace is the plane Ax D x1 C 2x2 C 3x3 D 0. This plane has dimension 2 (which is 3 1/. The dimensions add to 1 C 2 D 3. The columns of this 1 by 3 matrix are in R 1 ! The column space is all of R1 . The left nullspace contains only the zero vector. The only solution to A T y D 0 is y D 0, no other multiple of 1 2 3 gives the zero row. Thus N .AT / is Z, the zero space with dimension 0 (which is m r/. In Rm the dimensions add to 1 C 0 D 1. 1 2 3 Example 2 A D has m D 2 with n D 3 and rank r D 1. 1 4 6 The row space is the same line through .1; 2; 3/. The nullspace must be the same plane x1 C 2x2 C 3x3 D 0. Their dimensions still add to 1 C 2 D 3. All columns are multiples of the rst column .1; 2/. Twice the rst row minus the second row is the zero row. Therefore AT y D 0 has the solution y D .2; 1/. The column space and left nullspace are perpendicular lines in R 2 . Dimensions 1 C 1 D 2. 1 2 Column space D line through Left nullspace D line through : 2 1 If A has three equal rows, its rank is . What are two of the ys in its left nullspace?

The ys in the left nullspace combine the rows to give the zero row.

3.6. Dimensions of the Four Subspaces

189

Matrices of Rank One

That last example had rank r D 1and rank one matrices are special. We can describe them all. You will see again that dimension of row space D dimension of column space. When r D 1, every row is a multiple of the same row: 2 3 2 3 1 2 3 1 6 2 6 27 4 67 7 equals 6 7 times 1 2 3 D vT : A D uvT A D 6 4 3 4 35 6 95 0 0 0 0 A column times a row (4 by 1 times 1 by 3) produces a matrix (4 by 3). All rows are multiples of the row .1; 2; 3/. All columns are multiples of the column .1; 2; 3; 0/. The row space is a line in Rn , and the column space is a line in R m . Every rank one matrix has the special form A D u v T D column times row. The columns are multiples of u. The rows are multiples of v T . The nullspace is the plane perpendicular to v. .Ax D 0 means that u.v T x/ D 0 and then v T x D 0.) It is this perpendicularity of the subspaces that will be Part 2 of the Fundamental Theorem.

REVIEW OF THE KEY IDEAS

1. The r pivot rows of R are a basis for the row spaces of R and A (same space). 2. The r pivot columns of A (!) are a basis for its column space. 3. The n r special solutions are a basis for the nullspaces of A and R (same space). r rows of I are a basis for the left nullspace of R. r rows of E are a basis for the left nullspace of A.

4. The last m 5. The last m

Note about the four subspaces The Fundamental Theorem looks like pure algebra, but it has very important applications. My favorites are the networks in Chapter 8 (often I go there for my next lecture). The equation for y in the left nullspace is A T y D 0: Flow into a node equals ow out. Kirchhoffs Current Law is the balance equation. This is (in my opinion) the most important equation in applied mathematics. All models in science and engineering and economics involve a balanceof force or heat ow or charge or momentum or money. That balance equation, plus Hookes Law or Ohms Law or some law connecting potentials to ows, gives a clear framework for applied mathematics. My textbook on Computational Science and Engineering develops that framework, together with algorithms to solve the equations: Finite differences, nite elements, spectral methods, iterative methods, and multigrid.

190

Chapter 3. Vector Spaces and Subspaces

WORKED EXAMPLES
3.6 A

Find bases and dimensions for all four fundamental subspaces if you know that 2 32 3 1 0 0 1 3 0 5 A D 4 2 1 0 5 4 0 0 1 6 5 D LU D E 1 R: 5 0 1 0 0 0 0

By changing only one of those numbers, change the dimensions of all four subspaces.
Solution

This matrix has pivots in columns 1 and 3. Its rank is r D 2. Basis .1; 3; 0; 5/ and .0; 0; 1; 6/ from R. Dimension 2. Basis .1; 2; 5/ and .0; 1; 0/ from E
1

Row space Column space Nullspace Nullspace of A T

(and A). Dimension 2. 2 D 1.

Basis . 3; 1; 0; 0/ and . 5; 0; 6; 1/ from R. Dimension 2. Basis . 5; 0; 1/ from row 3 of E. Dimension 3

We need to comment on that left nullspace N .A T /. EA D R says that the last row of E combines the three rows of A into the zero row of R. So that last row of E is a basis vector for the left nullspace. If R had two zero rows, then the last two rows of E would be a basis. (Just like elimination, y T A D 0T combines rows of A to give zero rows in R.) To change all these dimensions we need to change the rank r. The way to do that is to change an entry (any entry) in the zero row of R.
3.6 B Put four 1s into a 5 by 6 matrix of zeros, keeping the dimension of its row space as small as possible. Describe all the ways to make the dimension of its column space as small as possible. Describe all the ways to make the dimension of its nullspace as small as possible. How to make the sum of the dimensions of all four subspaces small? Solution The rank is 1 if the four 1s go into the same row, or into the same column. They can also go into two rows and two columns (so a i i D aij D aj i D ajj D 1). Since the column space and row space always have the same dimensions, this answers the rst two questions: Dimension 1. The nullspace has its smallest possible dimension 6 4 D 2 when the rank is r D 4. To achieve rank 4, the 1s must go into four different rows and columns. You cant do anything about the sum r C .n r/ C r C .m r/ D n C m. It will be 6 C 5 D 11 no matter how the 1s are placed. The sum is 11 even if there arent any 1s...

If all the other entries of A are 2s instead of 0s, how do these answers change?

Problem Set 3.6

1

(a) If a 7 by 9 matrix has rank 5, what are the dimensions of the four subspaces? What is the sum of all four dimensions?

3.6. Dimensions of the Four Subspaces

191

(b) If a 3 by 4 matrix has rank 3, what are its column space and left nullspace?
2

Find bases and dimensions for the four subspaces associated with A and B: 1 2 4 1 2 4 AD and B D : 2 4 8 2 5 8 Find a basis for each of the four subspaces associated with A: 2 3 2 32 0 1 2 3 4 1 0 0 0 1 2 40 1 2 4 65 D 41 1 05 40 0 0 AD 0 0 0 1 2 0 1 1 0 0 0

3 3 4 1 25 : 0 0

Construct a matrix with the required property or explain why this is impossible: h1i h0i (a) Column space contains 1 ; 0 , row space contains 1 ; 2 . 2 5 0 1 h3i h1i (b) Column space has basis 1 , nullspace has basis 1 .
3 1

(c) Dimension of nullspace D 1 C dimension of left nullspace. (d) Left nullspace contains
1 3

, row space contains

3 1

(e) Row space D column space, nullspace left nullspace.

5

If V is the subspace spanned by .1; 1; 1/ and .2; 1; 0/, nd a matrix A that has V as its row space. Find a matrix B that has V as its nullspace. Without elimination, nd dimensions and bases for the four subspaces for 2 3 2 3 0 3 3 3 1 A D 4 0 0 0 0 5 and B D 4 4 5 : 0 1 0 1 5 Suppose the 3 by 3 matrix A is invertible. Write down bases for the four subspaces for A, and also for the 3 by 6 matrix B D A A . What are the dimensions of the four subspaces for A; B, and C , if I is the 3 by 3 identity matrix and 0 is the 3 by 2 zero matrix? I I AD I 0 and B D T and C D 0 : 0 0T Which subspaces are the same for these matrices of different sizes? A A A A (a) A and (b) and : A A A A Prove that all three of those matrices have the same rank r.

192
10

Chapter 3. Vector Spaces and Subspaces

If the entries of a 3 by 3 matrix are chosen randomly between 0 and 1, what are the most likely dimensions of the four subspaces? What if the matrix is 3 by 5? (Important) A is an m by n matrix of rank r. Suppose there are right sides b for which Ax D b has no solution. (a) What are all inequalities (< or ) that must be true between m; n, and r? (b) How do you know that A T y D 0 has solutions other than y D 0?

11

12

Construct a matrix with .1; 0; 1/ and .1; 2; 0/ as a basis for its row space and its column space. Why cant this be a basis for the row space and nullspace? True or false (with a reason or a counterexample): (a) If m D n then the row space of A equals the column space. (b) The matrices A and A share the same four subspaces. (c) If A and B share the same four subspaces then A is a multiple of B.

13

14

Without computing A, nd bases for its four fundamental subspaces: 2 32 3 1 0 0 1 2 3 4 A D 46 1 05 40 1 2 35 : 9 8 1 0 0 1 2 If you exchange the rst two rows of A, which of the four subspaces stay the same? If v D .1; 2; 3; 4/ is in the left nullspace of A, write down a vector in the left nullspace of the new matrix. Explain why v D .1; 0; 1/ cannot be a row of A and also in the nullspace. Describe the four subspaces of R3 associated with 2 3 2 3 0 1 0 1 1 0 A D 4 0 0 1 5 and I C A D 4 0 1 1 5 : 0 0 0 0 0 1 (Left nullspace) Add the extra column b and reduce A to echelon form: 3 2 3 2 1 2 3 b1 1 2 3 b1 5: 3 6 b2 4b1 A b D 4 4 5 6 b2 5 ! 4 0 7 8 9 b3 0 0 0 b3 2b2 C b1 A combination of the rows of A has produced the zero row. What combination is it? (Look at b3 2b2 C b1 on the right side.) Which vectors are in the nullspace of A T and which are in the nullspace of A?

15

16 17

18

3.6. Dimensions of the Four Subspaces

193

19

Following the method of Problem 18, reduce A to echelon form and look at zero rows. The b column tells which combinations you have taken of the rows: 3 2 2 3 1 2 b1 1 2 b1 6 2 3 b2 7 7 (a) 4 3 4 b2 5 (b) 6 4 2 4 b3 5 4 6 b3 2 5 b4 From the b column after elimination, read off m r basis vectors in the left nullspace. Those ys are combinations of rows that give zero rows.

20

(a) Check that the solutions to Ax 2 1 0 A D 42 1 3 4

D 0 are perpendicular to the rows: 32 3 0 4 2 0 1 0 5 4 0 0 1 3 5 D ER: 1 0 0 0 0

1

(b) How many independent solutions to A T y D 0? Why is y T the last row of E

21

Suppose A is the sum of two matrices of rank one: A D uv T C wzT . (a) Which vectors span the column space of A? (b) Which vectors span the row space of A? (c) The rank is less than 2 if or if . (d) Compute A and its rank if u D z D .1; 0; 0/ and v D w D .0; 0; 1/.

22

Construct A D uvT C wzT whose column space has basis .1; 2; 4/; .2; 2; 1/ and whose row space has basis .1; 0/; .1; 1/. Write A as (3 by 2) times (2 by 2). Without multiplying matrices, nd bases for the row and column spaces of A: 2 3 1 2 3 0 3 A D 44 55 : 1 1 2 2 7 How do you know from these shapes that A cannot be invertible?

23

24

(Important) AT y D d is solvable when d is in which of the four subspaces? The solution y is unique when the contains only the zero vector. True or false (with a reason or a counterexample): (a) A and AT have the same number of pivots. (b) A and AT have the same left nullspace. (c) If the row space equals the column space then AT D A. (d) If AT D A then the row space of A equals the column space.

25

194
26

Chapter 3. Vector Spaces and Subspaces

27

(Rank of AB) If AB D C , the rows of C are combinations of the rows of . . Since B T AT D C T , the rank So the rank of C is not greater than the rank of of C is also not greater than the rank of . h i If a; b; c are given with a 0, how would you choose d so that a b has rank 1? cd Find a basis for the row space and nullspace. Show they are perpendicular! Find the ranks of the 8 by 8 checkerboard matrix B and the chess matrix C : 2 3 2 1 0 1 0 1 0 1 0 r n b q k b n 60 1 0 1 0 1 0 17 6p p p p p p p 6 7 6 four zero rows B D 6 1 0 1 0 1 0 1 0 7 and C D 6 6 7 6 4 5 4p p p p p p p 0 1 0 1 0 1 0 1 r n b q k b n 3 r p7 7 7 7 p5 r

28

The numbers r; n; b; q; k; p are all different. Find bases for the row space and left nullspace of B and C . Challenge problem: Find a basis for the nullspace of C .
29

Can tic-tac-toe be completed (5 ones and 4 zeros in A) so that rank .A/ D 2 but neither side passed up a winning move?

Challenge Problems
30

If A D uvT is a 2 by 2 matrix of rank 1, redraw Figure 3.5 to show clearly the Four Fundamental Subspaces. If B produces those same four subspaces, what is the exact relation of B to A? M is the space of 3 by 3 matrices. Multiply every matrix X in M by 2 AD4 1 1 0 0 1 1 3 1 0 5. 1 2 3 2 3 1 0 Notice: A 4 1 5 D 4 0 5 : 1 0

31

(a) Which matrices X lead to AX D zero matrix? (a) Which matrices have the form AX for some matrix X? (a) nds the nullspace of that operation AX and (b) nds the column space. What are the dimensions of those two subspaces of M? Why do the dimensions add to .n r/ C r D 9?
32

Suppose the m by n matrices A and B have the same four subspaces. If they are both in row reduced echelon form, prove that F must equal G: I F I G AD BD : 0 0 0 0