You are on page 1of 10

Working with the Basis Inverse over a Sequence of

Iterations
Robert M. Freund
February, 2004
c 2004 Massachusetts Institute of Technology.
1
_ _ ¸ ¸ ¸ ¸
¸ ¸ ¸ ¸
1 Equations Involving the Basis Matrix
At each iteration of the simplex method, we have a basis consisting of an
index of variables:
B(1), . . . , B(m) ,
from which we form the basis matrix B by collecting the columns A
B(1)
, . . . , A
B(m)
of A into a matrix:
. B := A
B(1)
¸ A
B(2)
¸ . . . ¸ A
B(m−1)
¸ A
B(m)
In order to execute the simplex method at each iteration, we need to be
able to compute:
T T
x = B
−1
r
1
and/or p = r
2
B
−1
, (1)
for iteration-specific vectors r
1
and r
2
, which is to say that we need to solve
equation systems of the type:
T
Bx = r
1
and/or p
T
B = r
2
(2)
for x and p.
2 LU Factorization
One way to solve (2) is to factor B into the product of a lower and upper
triangular matrix L, U:
B = LU ,
and then compute x and/or p as follows. To compute x, we solve the fol-
lowing two systems by back substitution:
• First solve Lv = r
1
for v
• Then solve Ux = v for x.
2
To compute p, we solve the following two systems by back substitution:
T
• First solve u
T
U = r
2
for u
• Then solve p
T
L = u
T
for p.
It is straightforward to verify that these procedures yield x and p that
satisfy (2). If we compute according to these procedures, then:
T
Bx = LUx = Lv = r
1
and p
T
B = p
T
LU = u
T
U = r
2
.
3 Updating the Basis and its Inverse
As the simplex method moves from one iteration to the next, the basis
matrix B changes by one column. Without loss of generality, assume that
the columns of A have been re-ordered so that
B := [ A
1
| . . . | A
j−1
| A
j
| A
j+1
| . . . | A
m
]
at one iteration. At the next iteration we have a new basis matrix
˜
B of the
form:
˜
B := [ A
1
| . . . | A
j−1
| A
k
| A
j+1
| . . . | A
m
] .
Here we see that column A
j
has been replaced by column A
k
in the new
basis.
Assume that at the previous iteration we have B and we have computed
an LU factorization of B that allows us to solve equations involving B
−1
. At
the current iteration, we now have
˜
B and we would like to solve equations
involving
˜
B
−1
. Although one might think that we might have to compute
˜
an LU factorization of B, that is not the case. Herein we describe how
the linear algebra of working with
˜
B
−1
is computed in practice. Before we
describe the method, we first need to digress a bit to discuss rank-1 matrices
and rank-1 updates of the inverse of a matrix.
3
_ _
3.1 Rank-1 Matrices
Consider the following matrix:
_ _
−2 2 0 −3
_
−4 4 0 −6
_
_ _
W =
_ _
−14 14 0 −21
.
10 −10 0 15
W is an example of rank-1 matrix. All rows are linearly dependent and all
columns are linearly dependent. Now define:
_ _
1
_
2
_
T
u =
_ _
and v = ( −2 2 0 −3 ) .
_
7
_
−5
If we think of u and v as n × 1 matrices, then notice that it makes sense
to write:
_
1
_ _
−2 2 0 −3
_
W = uv
T
=
_
_
_
2
7
_
_
_
× ( −2 2 0 −3 ) =
_
_
_
−4
−14
4
14
0
0
−6
−21
_
_
_
.
−5 10 −10 0 15
In fact, we can write any rank-1 matrix as uv
T
for suitable vectors u and v.
3.2 Rank-1 Update of a Matrix Inverse
Suppose we have a matrix M and we have computed its inverse M
−1
. Now
consider the matrix
˜
M := M + uv
T
M
−1
for some rank-1 matrix W = uv
T
. Then there is an exact formula for
˜
based on the data M
−1
, u, and v, which is called the Sherman-Morrison
formula:
˜
T
M
−1
Property. M is invertible if and only if v u = −1, in which case
M
−1
uv
˜
= I −
M
−1 T
M
−1
. (3)
1 + v
T
M
−1
u
4
_ _ _ _
_ _ _ _
_ _
_ _
_ _ _ _
Proof: Let
_ _
M
−1 T
uv
Q = I − M
−1
.
1 + v
T
M
−1
u
˜
Then it suffices to show that MQ = I, which we now compute:
˜
uv
MQ = M + uv
T
× I −
M
−1 T
M
−1
T
M
−1
1+v u
M
−1

M
−1
uv
T
M
−1
= M + uv
T
×
T
M
−1
1+v u
= I + uv
T
M
−1

uv
T
M
−1

uv
T
M
−1
uv
T
M
−1
T
M
−1 T
M
−1
1+v u 1+v u
1
= I + uv
T
M
−1
1 −
T
M
−1

v
T
M
−1
u
T
M
−1
1+v u 1+v u
= I
q.e.d.
˜
3.3 Solving Equations with M using M
−1
Suppose that we have a convenient way to solve equations of the form Mx =
b (for example, if we have computed an LU factorization of M ), but that
we want to solve the equation system:
˜
Mx = b .
Using (3), we can write:
˜
M
−1 T
x = M
−1
b = I −
uv
M
−1
b .
1 + v
T
M
−1
u
Now notice in this expression that we only need to work with M
−1
, which
we presume that we can do conveniently. In fact, if we let
2
x
1
= M
−1
b and x = M
−1
u ,
we can write the above as:
˜
M
−1 T 2 T T 1
1 2
x = M
−1
b = I −
uv
M
−1
b = I −
x v
x = x
1

v x
x .
T
x
2 T
x
2
1 + v
T
M
−1
u 1 + v 1 + v
5
_ _
˜
Therefore we have the following procedure for solving Mx = b:
1
• Solve the system Mx
1
= b for x
2
• Solve the system Mx
2
= u for x
v x 2
• Compute x = x
1

T 1
.
1+v
T
x
2
x
3.4 Computational Efficiency
The number of operations needed to form an LU factorization of an n × n
3
matrix M is on the order of n . Once the factorization has been computed,
the number of operations it takes to then solve Mx = b using back substi-
tution by solving Lv = b and Ux = v is on the order of n
2
. If we solve
˜ ˜
Mx = b by factorizing M and then doing back substitution, the number
2
of operations needed would therefore be n
3
+ n . However, if we use the
above rank-1 update method, the number of operations is n
2
operations for
each solve step and then 3n operations for the final step, yielding a total
operation count of 2n
2
+ 3n. This is vastly superior to n
3
+ n
2
for large n.
3.5 Application to the Simplex Method
Returning to the simplex method, recall that we presume that the current
basis is:
B := [ A
1
| . . . | A
j−1
| A
j
| A
j+1
| . . . | A
m
]
at one iteration, and at the next iteration we have a new basis matrix
˜
B of
the form:
˜
B := [ A
1
| . . . | A
j−1
| A
k
| A
j+1
| . . . | A
m
] .
Now notice that we can write:
˜
_ _
T
B = B + (A
k
− A
j
) × e
j
,
where e
j
is the j
th
unit vector (e
j
has a 1 in the j
th
component and a 0 in
every other component). This means that
˜
B is a rank-1 update of B with
j
u = (A
k
− A
j
) and v = e . (4)
6
_ _
_ _
_ _
_ _
_ _
_ _ _
_ _
_ _ _
If we wish to solve the equation system
˜
Bx = r
1
, we can apply the method
of the previous section, substituting M = B, b = r
1
, u = (A
k
− A
j
) and
v = e
j
. This works out to:
1
• Solve the system Bx
1
= r
1
for x
2
• Solve the system Bx
2
= A
k
− A
j
for x
1
(e
j
)
T
x
2
• Compute x = x
1

1+(e
j
)
T
x
2
x .
This is fine if we want to update the basis only once. In practice, how-
ever, we would like to systematically apply this method over a sequence of
iterations of the simplex method. Before we indicate how this can be done,
we need to do a bit more algebraic manipulation. Notice that using (3) and
(4) we can write:
˜
uv
B
−1
= I −
B

T
1
B

T
1
B
−1
1+v u
B
−1
(A
k
−A
j
)(e
j
)
T
B
−1
= I −
1+(e
j
)
T
B
−1
(A
k
−A
j
)
.
Now notice that because A
j
= Be
j
, it follows that B
−1
A
j
= e
j
, and substi-
tuting this in the above yields:
B
−1
I −
(B
−1
A
k
−e
j
)(e
j
)
T
B
−1
˜
=
(e
j
)
T
B
−1
A
k
˜
= EB
−1
where
e
j
_
T
˜
B
−1
A
k
− e
j
E = I − .
(e
j
)
T
B
−1
A
k
w be the solution of the system B ˜ Furthermore, if we let ˜ w = A
k
, that is,
˜ E as w = B
−1
A
k
, then we can write
˜
e
j
_
T
˜
˜
w − e
j
E = I − .
(e
j
)
T

We state this formally as:
7
_ _
_ _ _
_ _
˜
Property A. Suppose that the basis B is obtained by replacing the j
th
column of B with the new column A
k
. Let ˜ w be the solution of the system
B ˜ w = A
k
and define:
e
j
_
T
˜
˜
w − e
j
E = I − .
(e
j
)
T

Then
B
−1
˜ ˜
= EB
−1
. (5)
Once we have computed w we can easily form E. And then we have ˜
˜
from above:
˜ ˜
x = B
−1
r
1
= EB
−1
r
1
.
Using this we can construct a slightly different (but equivalent) method for
solving
˜
Bx = r
1
:
• Solve the system B ˜ w w = A
k
for ˜
w−e
j
)(e
j
)
T
• Form and save the matrix
˜
I −
( ˜
E =
(e
j
)
T

1
• Solve the system Bx
1
= r
1
for x
˜
• Compute x = Ex
1
.
Notice that
_ _
1 c˜
1
_
1 c˜
2
_
_ _
_ . . _
. .
_ . .
_
˜ _ _
E =
_ _

j
_ _
_ _
. .
_ . . _
. .

m
1
where
_
w − e
j
_
˜
c˜ = .
(e
j
)
T

˜
E is an elementary matrix, which is matrix that differs from the identity
matrix in only one column or row. To construct
˜
E we only need to solve
8
_
w = A
k
, and that the information needed to create
˜
˜ B ˜ E is the n-vector w
and the index j. Therefore the amount of memory needed to store
˜
E is just
n + 1 numbers. Also the computation of
˜
Ex
1
involves only 2n operations if
the code is written to take advantage of the very simple special structure of
˜
E.
4 Implementation over a Sequence of Iterations
Now let us look at the third iteration. Let
˜
˜
B be the basis at this iteration.
We have:
˜
B := [ A
1
| . . . | A
i−1
| A
i
| A
i+1
| . . . | A
m
]
at the second iteration, and let us suppose that at the third iteration we
replace the column A
i
with the column A
l
, and so
˜
˜
B is of the form:
˜
˜
B := [ A
1
| . . . | A
i−1
| A
l
| A
i+1
| . . . | A
m
] .
Then using Property A above, let
˜
˜ w be the solution of the system
˜
w = A
i
. Then B
˜
˜
−1
˜
˜
B
−1 ˜
˜
B = E
˜
(6)
where
_ _
_ _
˜
˜ w −e
i
e
i
_
T
˜
˜ _
E =
_
I −
˜
˜
.
(e
i
)
T
w
−1
˜
˜
B
−1
B = E
˜
= E
˜
It then follows that
˜
˜
˜
˜
EB
−1
, and so:
−1
B = E
˜
˜
˜
˜
˜
EB
−1
. (7)
B by forming E and
˜
˜
Therefore we can easily solve equations involving
˜
˜ ˜
E
and working with the original LU factorization of B.
This idea can be extended over a large sequence of pivots. We start
with a basis B and we compute and store an LU factorization of B. Let
our sequence of bases be B
0
= B, B
1
, . . . , B
k
and suppose that we have
computed matrices E
1
, . . . , E
k
with the property that
(B
l
)
−1
= E
l
E
l−1
· · · E
1
B
−1
, l = 1, . . . , k .
9
Then to work with the next basis inverse B
k+1
we compute a new matrix
E
k+1
and we write:
(B
k+1
)
−1
= E
k+1
E
k
· · · E
1
B
−1
.
This method of working with the basis inverse over a sequence of it-
erations eventually degrades due to accumulated roundoff error. In most
simplex codes this method is used for K = 50 iterations in a row, and then
the next basis is completely re-factorized from scratch. Then the process
continues for another K iterations, etc.
5 Homework Exercise
1. In Section 3.2 we considered how to compute a solution x of the equa-
˜ ˜
tion Mx = b where M = M + uv
T
and we have on hand an LU
factorization of M . Now suppose instead that we wish to compute a
˜
solution p of the equation p
T
M = c
T
for some RHS vector c. Using
the ideas in Section 3.2, develop an efficient procedure for computing
p by working only with an LU factorization of M .
2. In Section 3.5 we considered how to compute a solution x of the equa-
tion
˜
B differs from B by one column, and we have on Bx = r
1
where
˜
hand an LU factorization of B. Now suppose instead that we wish
T
to compute a solution p of the equation p
T
˜
B = r
2
for some vector
r
2
. Using the ideas in Section 3.5, develop an efficient procedure for
computing p by working only with an LU factorization of B.
10

B(m) . which is to say that we need to solve equation systems of the type: Bx = r1 for x and p. . we need to be able to compute: x = B −1 r1 and/or T pT = r2 B −1 .1 Equations Involving the Basis Matrix At each iteration of the simplex method. we solve the following two systems by back substitution: • First solve Lv = r1 for v • Then solve U x = v for x. . 2 . AB(m) of A into a matrix: B := AB(1) AB(2) . . from which we form the basis matrix B by collecting the columns AB(1) . To compute x. and/or T pT B = r 2 (2) 2 LU Factorization One way to solve (2) is to factor B into the product of a lower and upper triangular matrix L. . . . . (1) for iteration-specific vectors r1 and r2 . In order to execute the simplex method at each iteration. . we have a basis consisting of an index of variables: B(1). U : B = LU . . . and then compute x and/or p as follows. AB(m−1) AB(m) .

we solve the following two systems by back substitution: T • First solve uT U = r2 for u • Then solve pT L = uT for p. If we compute according to these procedures. | Am ] ˜ at one iteration. . . Although one might think that we might have to compute ˜ an LU factorization of B. . | Am ] . Herein we describe how ˜ the linear algebra of working with B −1 is computed in practice. we now have B and we would like to solve equations ˜ involving B −1 . | Aj−1 | Ak | Aj+1 | . that is not the case. 3 . . . Assume that at the previous iteration we have B and we have computed an LU factorization of B that allows us to solve equations involving B −1 . . then: Bx = LU x = Lv = r1 T pT B = pT LU = uT U = r2 . It is straightforward to verify that these procedures yield x and p that satisfy (2). . Without loss of generality. Here we see that column Aj has been replaced by column Ak in the new basis. At ˜ the current iteration. | Aj−1 | Aj | Aj+1 | . the basis matrix B changes by one column. At the next iteration we have a new basis matrix B of the form: ˜ B := [ A1 | . Before we describe the method. we first need to digress a bit to discuss rank-1 matrices and rank-1 updates of the inverse of a matrix.To compute p. and 3 Updating the Basis and its Inverse As the simplex method moves from one iteration to the next. . assume that the columns of A have been re-ordered so that B := [ A1 | .

and v. Now consider the matrix ˜ M := M + uv T ˜ for some rank-1 matrix W = uv T . −5 If we think of u and v as n × 1 matrices. 1 + v T M −1 u (3) 4 . Now define: 1  2  T  u=  7  and v = ( −2 2 0 −3 ) . we can write any rank-1 matrix as uv T for suitable vectors u and v. All rows are linearly dependent and all columns are linearly dependent. u. W =  −14 14 0 −21  10 −10 0 15 W is an example of rank-1 matrix.2 Rank-1 Update of a Matrix Inverse Suppose we have a matrix M and we have computed its inverse M −1 . M is invertible if and only if v T M −1 u = −1. which is called the Sherman-Morrison based on the data M formula: ˜ Property.3. Then there is an exact formula for M −1 −1 .1 Rank-1 Matrices   Consider the following matrix: −2 2 0 −3  −4  4 0 −6  . 3. then notice that it makes sense to write: 1  2 W = uv T =   7 −5     −2 2   −4 4  × ( −2 2 0 −3 ) =    −14 14 10 −10  0 0 0 0 −3 −6   . in which case ˜ M −1 = I − M −1 uv T M −1 . −21  15  In fact.

which we now compute: ˜ M Q = M + uv T × I − = M + uv T × M −1 − = I + uv T M −1 − M −1 uv T 1+v T M −1 u M −1 M −1 uv T M −1 1+v T M −1 u uv T M −1 1+v T M −1 u − uv T M −1 uv T M −1 1+v T M −1 u = I + uv T M −1 1 − =I q. 1 1+v T M −1 u − v T M −1 u 1+v T M −1 u 3. In fact. we can write the above as: ˜ x = M −1 b = I − M −1 uv T x2 v T v T x1 M −1 b = I − x1 = x1 − x2 .e. Using (3).3 ˜ Solving Equations with M using M −1 Suppose that we have a convenient way to solve equations of the form M x = b (for example. which we presume that we can do conveniently.d. if we have computed an LU factorization of M ). if we let x1 = M −1 b and x2 = M −1 u .Proof: Let Q= I− M −1 uv T M −1 . 1 + v T M −1 u Now notice in this expression that we only need to work with M −1 . we can write: ˜ x = M −1 b = I − M −1 uv T M −1 b . but that we want to solve the equation system: ˜ Mx = b . 1 + v T M −1 u 1 + v T x2 1 + v T x2 5 . 1 + v T M −1 u ˜ Then it suffices to show that M Q = I.

recall that we presume that the current basis is: B := [ A1 | .˜ Therefore we have the following procedure for solving M x = b: • Solve the system M x1 = b for x1 • Solve the system M x2 = u for x2 • Compute x = x1 − v T x1 x2 . | Am ] ˜ at one iteration. and at the next iteration we have a new basis matrix B of the form: ˜ B := [ A1 | . . the number of operations is n2 operations for each solve step and then 3n operations for the final step. This is vastly superior to n3 + n2 for large n. Once the factorization has been computed. However. 3. 1+v T x2 3. . If we solve ˜ ˜ M x = b by factorizing M and then doing back substitution. the number of operations needed would therefore be n3 + n2 . where ej is the j th unit vector (ej has a 1 in the j th component and a 0 in ˜ every other component). This means that B is a rank-1 update of B with u = (Ak − Aj ) and v = ej 6 . if we use the above rank-1 update method. | Aj−1 | Ak | Aj+1 | . (4) . the number of operations it takes to then solve M x = b using back substitution by solving Lv = b and U x = v is on the order of n2 . . | Aj−1 | Aj | Aj+1 | . Now notice that we can write: ˜ B = B + (Ak − Aj ) × ej T . | Am ] . . yielding a total operation count of 2n2 + 3n.5 Application to the Simplex Method Returning to the simplex method. . . . .4 Computational Efficiency The number of operations needed to form an LU factorization of an n × n matrix M is on the order of n3 .

if we let w be the solution of the system B w = Ak . substituting M = B.˜ If we wish to solve the equation system Bx = r1 . . Now notice that because Aj = Bej . however. we need to do a bit more algebraic manipulation. This works out to: • Solve the system Bx1 = r1 for x1 • Solve the system Bx2 = Ak − Aj for x2 • Compute x = x1 − (ej )T x1 x2 . it follows that B −1 Aj = ej . we can apply the method of the previous section. Before we indicate how this can be done. we would like to systematically apply this method over a sequence of iterations of the simplex method. In practice. Notice that using (3) and (4) we can write: ˜ B −1 = I − = I− B −1 uv T 1+v T B −1 u B −1 T 1+(ej )T B −1 (A B −1 (Ak −Aj )(ej ) k −Aj ) B −1 . u = (Ak − Aj ) and v = ej . b = r1 . ˜ ˜ ˜ ˜ w = B −1 Ak . then we can write E as w − ej ej ˜ ˜ E= I− (ej )T w ˜ We state this formally as: 7 T . 1+(ej )T x2 This is fine if we want to update the basis only once. and substituting this in the above yields: (B −1 Ak −ej )(ej ) ˜ B −1 = I − j T −1 (e ) B Ak T B −1 ˜ = EB −1 where B −1 Ak − ej ej ˜ E= I− (ej )T B −1 Ak T . that is. Furthermore.

. which is matrix that differs from the identity ˜ matrix in only one column or row. c1 ˜ c2 ˜ . Let w be the solution of the system ˜ B w = Ak and define: ˜ w − ej ej ˜ ˜ E= I− (ej )T w ˜ Then ˜ ˜ B −1 = EB −1 . (5) ˜ Once we have computed w we can easily form E.. Using this we can construct a slightly different (but equivalent) method for ˜ solving Bx = r1 : • Solve the system B w = Ak for w ˜ ˜ ˜ (w−ej )(ej ) ˜ • Form and save the matrix E = I − j T (e ) w ˜ T • Solve the system Bx1 = r1 for x1 ˜ • Compute x = Ex1 . To construct E we only need to solve 8 . 1 . Notice that      ˜= E     1 1 . . Suppose that the basis B is obtained by replacing the j th column of B with the new column Ak . . And then we have ˜ from above: ˜ ˜ x = B −1 r1 = EB −1 r1 . cj ˜ . . . ..           where c= ˜ w − ej ˜ (ej )T w ˜ ˜ E is an elementary matrix.˜ Property A. cm ˜ . T .

B1 . Then ˜ B˜ ˜˜ ˜ −1 ˜ ˜ (6) B = E B −1 where  ˜ ˜ E = I − ˜ ˜ w − ei ei T   . . . Let our sequence of bases be B0 = B. Therefore the amount of memory needed to store E is just ˜ n + 1 numbers. | Am ] at the second iteration. Ek with the property that (Bl )−1 = El El−1 · · · E1 B −1 9 . . and let us suppose that at the third iteration we ˜ ˜ replace the column Ai with the column Al . We start with a basis B and we compute and store an LU factorization of B. .˜ B w = Ak . ˜ ˜ (ei )T w ˜˜ ˜ −1 ˜˜ ˜ ˜ ˜ It then follows that B = E B −1 = E EB −1 . . l = 1. (7) ˜ ˜ ˜ ˜ ˜ Therefore we can easily solve equations involving B by forming E and E and working with the original LU factorization of B. . ˜ ˜ Then using Property A above. . 4 Implementation over a Sequence of Iterations ˜ ˜ Now let us look at the third iteration. . . . This idea can be extended over a large sequence of pivots. . and so: ˜˜ ˜ −1 ˜ ˜ B = E EB −1 . Bk and suppose that we have computed matrices E1 . Also the computation of Ex1 involves only 2n operations if the code is written to take advantage of the very simple special structure of ˜ E. | Ai−1 | Al | Ai+1 | . and so B is of the form: ˜ ˜ B := [ A1 | . . . . and that the information needed to create E is the n-vector w ˜ ˜ ˜ and the index j. We have: ˜ B := [ A1 | . Let B be the basis at this iteration. | Ai−1 | Ai | Ai+1 | . let w be the solution of the system ˜ w = Ai . . . . k . . . | Am ] . . .

etc. develop an efficient procedure for computing p by working only with an LU factorization of B. Using the ideas in Section 3.2. develop an efficient procedure for computing p by working only with an LU factorization of M .5. In Section 3. 10 . and then the next basis is completely re-factorized from scratch. and we have on hand an LU factorization of B. 5 Homework Exercise 1.5 we considered how to compute a solution x of the equa˜ ˜ tion Bx = r1 where B differs from B by one column. Now suppose instead that we wish to compute a ˜ solution p of the equation pT M = cT for some RHS vector c. Then the process continues for another K iterations. Using the ideas in Section 3. This method of working with the basis inverse over a sequence of iterations eventually degrades due to accumulated roundoff error. In most simplex codes this method is used for K = 50 iterations in a row.Then to work with the next basis inverse Bk+1 we compute a new matrix Ek+1 and we write: (Bk+1 )−1 = Ek+1 Ek · · · E1 B −1 . In Section 3. 2.2 we considered how to compute a solution x of the equa˜ ˜ tion M x = b where M = M + uv T and we have on hand an LU factorization of M . Now suppose instead that we wish T ˜ to compute a solution p of the equation pT B = r2 for some vector r2 .