The Elimination Form of the Inverse and Its Application to Linear Programming Author(s): Harry M.

Markowitz Reviewed work(s): Source: Management Science, Vol. 3, No. 3 (Apr., 1957), pp. 255-269 Published by: INFORMS Stable URL: . Accessed: 10/05/2012 02:14
Your use of the JSTOR archive indicates your acceptance of the Terms & Conditions of Use, available at . JSTOR is a not-for-profit service that helps scholars, researchers, and students discover, use, and build upon a wide range of content in a trusted digital archive. We use information technology and tools to increase productivity and facilitate new forms of scholarship. For more information about JSTOR, please contact

INFORMS is collaborating with JSTOR to digitize, preserve and extend access to Management Science.

a petroleum refinery may be indirectly related to every other. E. This "elimination form of the inverse. ***. 255 . may be expressed as the product of other matrices A. Thus the matrix describing petroleum technology has a small percentage of nonzeros.g." as we shall call it.. some problems there may be Mi which are easier to obtain and apply than A-' itself. say. any particular process uses few of these. The application of this form of inverse in linear programming is also discussed. * ReceivedOctober. MARKOWITZ' The RAND Corporation Introduction It is common for matrices in industrial applications of linear programming to have a large proportion of zero coefficients. A-'. equipment item) in. While every item (raw material. end item. The writer is indebtedto GeorgeDantzig andAlan Mannefor valuablesuggestions.. intermediate material.l-mAm. This paper discusses a particular product form of inverse which is closely related to the Gaussian elimination method of solving a set of simultaneous equations.THE ELIMINATION FORM OF THE INVERSE AND ITS APPLICATION TO LINEAR PROGRAMMING* HARRY M. If spacial or temporal distinctions are introduced into the model the percentage of non-zeros generally falls further. If we have such M1. The Elimination Form of Inverse The inverse (A-') of a matrix (A) is valuable when a number of sets of equations AX = b are to be solved using different b's and the same A. The present paper discusses a form of inverse which is especially convenient to obtain and use for matrices with a high percentage of zeros. (2) = (2) (4) = (W)(16) etc. Mm we can solve X = A-b in a series of steps: X(1= Mlb X(2) =M2X(1 X MmX(m-1) M1 is referred to as a "product form" of inverse.. like any matrix..1956. in an infinite number of ways. M. In The expression Mm . is especially valuable when A has a large number of zero coefficients.

X3 and B1) give X1.256 HARRY MARKOWITZ The elimination form of inverse can be illustrated in terms of the solution of three equations in three unknowns: (1) a11Xj + al2X2 + a13X3 = r (2) (3) a2lX1 + a22X2+ a23X3= r2 a31X1+ a32X2+ a33X3= r3 For the moment we will let the kth diagonal element be the kth pivotal elements From equation 1) we get the first equation of our back solution (Bi) X = all ri- al2X all -_ all X3 We eliminate X1 from equations 2) and 3) by adding equation to the ith ail times the first equation. Consider the transformations which occurred to our original right hand side. X3 and B2) give X2. thus obtaining = + b22X2 b23X3 (2') (3') where r2* = + b32X2 b33X3 r3* r2 r3 = = r2r3 - all i-r1 -3 ( ) all r1 Similarly we get (B2) and where r3** = X2 = b22 r2* - b22 2X3 c33X3 =r3** r3 * b2 r2 Finally (B3) X= C33 r3 (B3) gives us X3. First we formed 1 0 0 (7r*) = _a' a 1 0 ()r2 0 1 all . X2.

ELIMINATION FORM OF THE INVERSE 257 then (ri\~~ = 0 io 1 0 b32 l \*! 1 then (H) then = 1 0 0 o1 Is 0 ($ 0 /X1\ 1 _ab2 62/r2 0 1 0 b23 ~~~~~b22 0 1 Since the aibji do not depend on the rs we havre 1 a12 _?13 [10 0 0-1=al X a1 all 0 aa l 0 1 0-1 1 .

. The arithmetic operations re- . It requires the same number of multiplications and additions to apply the elimination form as it does to apply the conventional A-'.1 matrices it contains only n numbers which are not known a-priori to be 0 or 1. BmRmR.258 HARRY MARKOWITZ Similarly if A is any mXm non-singular matrix we have . where Rk is a matrix of the form 1 ~~0 0 0 Vk+l.k Vkk 0 Vmk Vkk 1 and Bk iS of the form 0 1 1 Vkk Vkk+1 Vkk _ Vkm Vkk 0 Although this elimination form of inverse consists of 2m .. A-' = B1B2 ..

..k of the -j equal zero.. The elimination form of the inverse in this case is . The kl step also gives rise to a back solution of the form = x To = 0 1 rio + E niX where at least m ....ELIMINATION FORM OF THE INVERSE 259 quired to obtain the B1 . vm oio . BmRm_i. The y vector is in the ioth column of the maAt least k trix. This step subjects the right-hand side to a transformation of the form r1 7 (r)new = o (r)old 'Yin 1 of the ey are zero.. R1 and to apply them once are exactly the same as those required to solve a set of linear equations by Gaussian elimination.. A-' = BBBmPR7n*-i . Suppose that the kth pivot is not vkkk but vi0o (whereV7 is the value of the parameter of ith equation jth variable at the kth step).R where ri = 7tl Bk: = pk .

This rearrangement changes the order of the ri exactly as does P. rimi. i. This may be so even though the usual A-' has no zeros. The next back solution is of the form Xji. thanks to Bm and P.X and similarly the new kth back solution is the one in which Xk was the jmlth component and Xj. The last back solution is of the form Xji = V rim Since rim is now the jmth component of (r). We can imagine reordering the equations so that new first back solution is the one of the form 1 k + E . Rlr give the right hand sides of the back solutions in the original equation order.. Bm. In the case of a Bk it only is necessary to record the j eliminated. . In the case of an Rk it is only necessary to record the i eliminated and the nonzero yi . is the jmth component of (r).. The transformations Rm.1 = 1 + lXjm Since. that AX = r implies X = B1*.* P*** R1 r may be seen as follows. If the matrix A has a large number of zero aqi... the non-zero nj and 1/pivotal element. transforms (r) into a vector with Xjm_1 in the j"m_jth position as well as Xjm in the jmth position.e. In recording Bk or Rk it is not necessary to write out the entire matrix.. And so on. That A-1=B1 * * P * * * R1. the elimination form of inverse may have appreciably less than n2 non zero ys and vs. Bm (as described above) will transform (r) into a vector with Xjm in the jmth position.260 HARRY MARKOWITZ 1 'Yi Rk= 1 7Ym 1 ) and P is permutation matrix such that if vijk is a pivotal element P makes the old ith component of (r) become the new jth component of r.

Suppose the *'s below represent the non-zero elements of a 5 x 5 matrix. *. The M's represent coefficients which began 43 in the matrix as zeros but ended as non-zeros. in which case the agenda cannot be followed to the end. The number oj at the bottom of each column indicates the number of elements of that column which were not already eliminated when the column was eliminated. An example agenda is presented in Table 1. * * * * * * * * * * * * * * * If all is the first pivotal element the non-zeros at step two are (barring accidental zeros) as follows * * * * * * * * * * * * * * * * But if a1nor a51is the pivotal element the pattern of non-zeros is * * * * * * * * * A table indicating zero and non-zero coefficients is a valuable aid in the choice of pivotal elements. There are. . If a pivotal element fails the test. The X's represent the original non-zero elements of the matrix. form the agenda beforehand and follow it as long as each pivotal element meets the test. One of the by-products of making an agenda beforehand is foreknowledge of all the variables which will appear in any equation and all the equations in which a variable will ever appear. Or some pivotal element may turn out to be so small that its use would adversely affect the accuracy of the results.. a new agenda can be worked out for the remaining equations and variables. This separation of the choice of the agenda and actual elimination is convenient both in hand and machine computation. The number (pi) at the right of each row indicates the number of elements of that row which were not already eliminated when the row was eliminated.. One solution to these difficulties is to have some test of the acceptability of a pivotal element.e.ELIMINATION FORM OF THE INVERSE 261 The number of non-zero v and y in an elimination form of inverse may depend on which pivotal elements are used. a complete set of pivotal elements) can be chosen before computation begins. indicate the kth pivotal element. however. two dangers attached to deciding on an agenda beforehand. Some pivotal element may accidentally become zero. From such a table an agenda (i. The numbers k = 1.

__ __ __ __ I _ I I 1 IN I8 1 1 1~ 1 I I8 1 1e 1 1' 1 I II lI l I I IR I8 I8 1 I8 1I 81 81 I I 1 1 1 1 1 1 _ _ I I~I I I I I I I I I I I I _ _ _ _ _ __ _ _ I I 1. I I I I I I I I I I I I I 1 1I I I l I I I I I I I I I II I I I I I I1 1 11__._ II I 1 1 1 1 1 1 1 1 I _ _ I I I-L 1 1 1 1 1 1 1 1 1 _ _ ?_ __ __ __ I I I I I 1 I I1 I8 I I I I __ __ _ __ _ I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I IIc 1 1 1 1 I II I I I I I I I I IC: I I I I I I II ~ I 1 1 1 1 1 I It 1 8 e' ' I 'I 'I I'1 I -- I --1-- -?1 -I __ _ _ 1.1 I 1 I I I I I I I I I I I I 1I I I I I I I I I I I I I I I I I I I I I I I I IN __ _ I I II l_ 1 I _ I I 1 1 1~ 1 I II 1 1 1 XI I 1 1 1 I 1 ___ 004 I _ I I I I I I II. III I I I I I1 I CD C4 ____I__I__I 114I Z 1l41 I IHI I cq 1 1 1 1 1 1HlIq I I I I I I 1 1 I Icq II I I I I I I I l locllli ' -C4iXvloobxclllelovIa< S IIll4llll---cllic 00 eq 00~~~~~~~6 1[ 1 I I 1 1 I 1001 1 1-1e11111111111111111111q- ~I I IIII 1 1 11 1 III It111111111111111111111126 [I I1 ItII I II8 I ~ ~ 1I~ .__.

cO l cq l - Ccq e i c - C-- co ce e C-4 c - e IC I I I I I I I I I8I 1 1 1 I>I8I 18 8 8 I I 1 1O II I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I 1 1O I I I I I I I I I I I I I I I I I I I 1 1O I I I I I I I I I I I I I I I I I I 1' 1 1 0 I I I I I I I I I I I I I I I I I I IcaI 1 _ I I I I I I I I I I I I I I IC I I I I 1 _ s I I I I I I I I I I I I I>I 1 1 1 1 1 1 1 _ I I I I I I I I I I 1 1? I I I I I I I I I _ _ I I I I I I I I I I IaZI I I I I I I I I 1 _ I I I i I I I I I I I I I I I I I I I I 1 _ I I I I I I I I I I I I I I I I1 _ II I I I I I I I I I I I I I I I I I I I I I I 1 _ __ _ I I I I I I I I I I I I I I I I I I I I1 I I I I I I I I I I I I I I I I I I I1 I_ _ I I I I I I I I I I I I I I-1 1 I 1 _ ~8 1 _ _ I I I I I I I I I I I I I I I.I -I I I1 _1 _ _ I I I I I I I I I I I I I I I I I I81 1 I _ I I I I I I I I I I I I I8 1 81 1 1~ 1 I III III I 1~~ 1-1-1-1-1-1 1 _ I>I11I 11 I I811I 1 1 11 -I I 11 I I I 1111 18811 I 111li I I 1 ~ oII IIII I ~I II ~I I -1_11_--1__11 ___ _ 1I oI I81 1 1~ 1I II I III I~ I I I I 1 11--I 118 ~ ~I _____1_ 11 I1 1I1811111 1I II I I I IIIII I I I I I I 1111111liii11 III118 1 1K1 1 I I I I I Ke II11K 263 .

1) selection principle. FIG. Figure 1 illustrates matrices with the following properties: a) all diagonal elements akk are non-zero b) if an element aik or aki in the kth columnor row is non-zero. 4 August 1953. If the kth diagonal element is used as the kth pivot. the number of non-zero elements in the elimination form of inverse is pi + E (o-j ) = p + E o-j 43 = 201.1)(oj .then all elements between ak (or akj) and akk are non-zero. The first set of equations p'A = ' or A'p = y 2 G. Dantzig. B. It is trivial to find an optimum agenda for such matrices. rj = 247 multiplications or To derive this inverse requires (i pivotal) Pij divisions and somewhat less additions or subtractions. no zero coefficient will become non-zero. William Orchard-Hays has coded the "Johniac" for the elimination form using the minimum (pi . "The Generalized Simplex Method. As compared with a possible (43)2 = 1849. 1 * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * Application to Linear Programming The simplex method for solving linear programming problems has a number of variants." RAND P-392-1. If a matrix is almost of the above form except that a few zeros are mixed in with the non-zeros. is to choose the pivot which minimizes the number of coefficients modified at each step (excluding those which are eliminated at the particular step). & Philip Wolfe. An alternative is to choose at each step the pivot which minimizes the number of zeros which become non-zero at that step. The small percentage of zeros which became non-zeros in our example is typical of cases run thus far (with approximately ten per cent non-zeros in the original matrix) with the OrchardHays code. which seems adequate generally. A recent version2 requires the solution of two sets of equations. This is equiva- Z lent to choosingthe non-zeroelementwith minimum(pi - l)(0o - 1). Alex Orden. This matrix has 197 non-zero elements. A still simpler alternative. It would be desirable to choose an agenda so as to minimize the number of zeros which become non-zero.264 HARRY MARKOWITZ The matrix which Table 1 represents was the optimum basis of a linear programming problem involving a 43-equation model of petroleum refining. Generally it is harder to find such an "optimum" agenda than to invert a matrix. . then using the kth diagonal element as the kth pivot may cause the "misplaced" zeros (and only these) to become non-zero.

. is a column vector associated with the variable X.. to be increased until some "basis" variable becomes zero. The matrix A at one iteration differs from that at the next only in that the column vector Pr. (which we will assume to be the rth column of A.) is replaced by the column vector P. Pji) Pj) (Pjl P. I. Or el A? (k = (Pj. is increased. P. first becomes zero as X._) er em ei iJ 1 Letting E = em 1 . Pi. . .. e is used to determine which basis variable X. Pr .e. The second set of equations is Ae = P8 where P.. A (k) becomes A(k+l) = (Pjl .ELIMINATION FORM OF THE INVERSE 265 yields prices p which are used to select a variable X8. currently equal to zero.

T. since this form requires less time to obtain and apply. A point is eventually reached when it is desirable to reinvert the current basis A (k. 1951.. If the product form of inverse is used. 3 See George B.266 we have HARRY MARKOWITZ A(k+)_ AkE The inverse of A(k?l) is A (k +1)E=g 1 (Kf) where (l ~~~elA e. Dantzig.) and let A (ko +k) EI= E(ko+k-1) 1 (ko)-'A (ko)-l At this point the elimination form of inverse can be of value. 358 in Activity Analysis of Production and Allocation. "Maximization of a Linear Function of Variables Subject to Linear Inequalities. "Application of the Simplex Method to a Game Theory Problem.E (k-1) R. 46. and Robert Dorfman.. April. er em 1 at each step by multiplyingout E(k) In an early variant of the simplex method the new inverse A (k+l)1 A(k) 3 was obtained In a morerecentversionthe E(k) are carried along and used as product form of inverse. New York. Koopmans. Ed. C. especially if A has a large number of zeros.VIII. . 1954. E(1)-1.E(2)1 k-l)E(k2." p. 4See George B.. 339-347." pp." in Mathematical Tables and OtherAids to Computation. Dantzig.4 If the first iteration starts with the identity matrix then A(k)-1 = . as k increases computing time required per iteration also increases. No. "The Product Form for the Inverse in the Simplex Method. e.

To see the effect of a more convenient form of inverse on the frequency of reinversion and the time required to solve a linear programming problem. Total time (T) required is n T where = Ea + fAIi + y(AIi)2 n i= 1 ZAIi = I The optimum solution must satisfy the Lagrangian equations _ i aE (a + fAIi + y(AI )2) _ _ _ _ _ _ _ _ _ _ _ X(AIi) _ _ = 0 aAIi. Hence the time (t) since the start of the last reinversion is a quadratic t = a + 3I + yI2 where a. (b) the time required for this first inversion is the same as that for any subsequent reinversion. It has been observed. Suppose it were decided that there would be n inversions (n . that.. that (a) the problem starts with a first basis to be inverted. (c) computation time per iteration is a linear function of the number of iterations since the last (re)inversion. When a more convenient form of inverse is available it may be desirable to reinvert more frequently.1 reinversions). for the moment.. 3. We will first derive some neat results under several restrictive assumptions. ry > 0. Let AIi = the number of iterations between the ith and i + ith inversion (for Let AI. with the RAND linear programming code for the IBM 701. that the number of iterations I required to solve the problem is known beforehand. Let us assume. the same percentage reduction in computing time for the whole linear programming problem as we obtain for the reinversion of A. 3+ 2-yAIi-X = 0 for all i AIi is the same for all i We can therefore rewrite the expression for T as T = n(a + fI + where I = AIi yI2) 7 n . the computing time required per iteration increases linearly with the number of iterations. Afterwards we will show computing procedures for obtaining an optimum reinversion policy under more general assumptions. Let us further assume.ELIMINATION FORM OF THE INVERSE 267 Reinverting A is only part of the operations involved in solving a linear programming problem. n-1). we must explore the question of optimum reinversion policy. for the moment. sion to the end of the problem. = the number of iterations from the nfth inveri = 1. . with stop watch as well as theory. by the use of the elimination form. We therefore cannot expect to obtain.

roughly).268 Or HARRY MARKOWITZ T -an dT dn dT dn2 + P3I+I72 n a 2- n2 2 y 2n3 oo as n -0 0 for Since -w > O all n > 0. We may as well also drop the quadratic assumption on t. Then 'the optimum z I = =1/ a T-?aV17? I 7 3I+ / (f a 7 +2rV~)I The last expression can be used for estimating the time to' be saved by using a more convenient form of inverse. If such an n is non-integral the best integral value is either that immediately above A. Thus-given our various assumptions-if a new method of inversion could produce an inverse in one-fourth the time (a) and because of its compactness it permitted the first subsequent iteration to be done in one-half the time (p. the whole linear programming problem could be done in one-half the time. We define aij = the expected value of qij where [0when I < i The (expected) time required to (re)invert the matrix at iteration i and iterate through j . or both d2T dT When dT n =4/ I Let us assume that n is integral. Let us now suppose that I is not known but has an a-priori probability distribution (derived presumably from past linear programming problems). qij = . The expected time from the beginning of the reinversion at i to the end of the problem if i I < j. or that immediately below A. and since T any n > 0 withdn d~~~~~~~~~~~~ gives a minimum value of T for all n > 0.1 without reinverting-if I > j.

is E - aulI + ai1r2 + + aIK.1IK The optimal value of Ii of I. version points can be chosen with IR so large that there is a zero probability that this iteration will occur.. Define aI2 11 = al. Expected time. **. . .. I2 We now only need to minimize E = aI2/I1 + ai2r3 + + aIK. 1(I2) + a1(I2).. could be calculated if 72 were known. + ar1I2 for various values of 12 and which can be readily computed.K-i from which we get IK and work back through IK-1(IK). etc. K-2(I1-1). to be minimized. there is no loss of generality in assuming a fixed K).ELIMINATION FORM OF THE INVERSE 269 IK (since rein- Suppose the points of reinversion are before iterations 1.1rK We repeat the process until we have E = a 1K. It is given by the function I1(I2) which minimizes a.