Handbook of Matrices H. Liitkepohl Humboldt-Universitat zu Berlin, Germany JOHN WILEY & SONS Chichester « New York « Brisbane : Toronto - SingaporeCopyright © 1996 by John Wik Baffins Lane, Chichester, West Sussex POLD IUD, England National 01243 779777 Intemational (+44) 1243 779777 e-mail (for orders and eustomer service enquiries): ex-booksi@wiley.co.uk. Visit our Home Page on http://www. wiley.co.uk of http://worw.wiley.com All Rights Reserved.No part of this book may be reproduced, stored in a retrieval system, or transmitted, in any form or by any means, electronic, ‘mechanical, photocopying, recording or otherwise, except under the terms of the Copyright. Designs and Patents Act 1988 or under the terms of a licence issued by the Copyright Licensing Agency, 90 Tottenham Court Road, London, UK WLP LIE, without the permission in writing of the publisher. Other Wiley Ednonial Offices John Wiley & Sons, tne.. 605 Third Avenue, New York, NY 10158-0012, USA Jacaranda Wiley Ltd, 33 Park Rea Queensland 4064, Australia Milton, John Wiley & Sons (Canada) Ltd, 22 Worcester Road, Rexdale, Ontario MOW 11.1, Canada John Wiley & Sons (Asia) Pte Ltd, 2 Clementi Loop 402-01, in Ning Distripark. Singapore 051.2 British Library Cataloguing in Publication Data A catalogue revord for this hook is available from the British Library ISBN 0.471 96688 6: 0 471 97018 & (pbk) Produced from camera-ready copy supplied by the authors using LaTeX Printed and bound in Great Britain by Biddles Lid, Guildford and King’s Lynn This book is printed on acid-free paper respousibly manufactured from sustainable forestation, or which at least 10 trees tre planted for each one used for paper prodketionContents Preface List. of Symbols 1 Definitions, Notation, Terminology Co Basic Notation and Terminology Operations Relating Matrices . Inequality Relations Between Matrices Operations Related to Individual Matrices Some Special Matrices... . Some ‘Terms and Quantities Related to Matrices Rules for Matrix Operations 21 2.2 2.3 24 25 2.6 Rules Related to Matrix Sums and Differences | Rules Related to Matrix Multiplication. . . Rules Related to Multiplication by a Scalar Rules for the Kronecker Product Rules for the Hadamard Product... . . Rules for Direct Sums vee Matrix Valued Functions of a Matrix . 3. 3.2 3.3 34 3.5 The Transpose The Conjugate seas The Conjugate ‘Transpose. Lees The Adjoint of a Square Matrix . ‘The Inverse of a Square Matrix 3.5.1 General Results 3.5.2. Inverses Involving Sums and Differences. . 3.5.3 Partitioned Inverses : 3.5.4 Inverses Involving Commutation, Duplication and Elimination Matrices 29CONTENTS 3.6 Generalized Inverses ...... . . 3.6.1 General Results . 7 3.6.2 The Moore -Penrose Inverse . 3.7 Matrix Powers . .e wae ee 3.8 The Absolute Value 2... 2.6... .0 00005 ‘Trace, Determinant and Rank of a Matrix . 4.1 The Trace . . oe oa cee 4.1.1 General Results... ..... Sees 4.1.2 Inequalities Involving the Trace . 4.1.3 Optimization of Functions Involving the ‘Trace 4.2. The Determinant . 4.2.1 General Results . . 42.2 Determinants of Partitioned Matrices 4.2.3. Determinants Involving Duplication Matrices 4.2.4 Determinants Involving Elimination Matrices 4.2.5 Determinants Involving Both Duplication and Elimina- tion Matrices . ne 4.2.6 Inequalities Related to Determinants we 4.2.7 Optimization of Functions Involving a Determinant. 4.3 The Rank of a Matrix 4.3.1 General Results 43.2 Matrix Decompositions Related to the Rank 4.3.3 Inequalities Related to the Rank . Eigenvalues and Singular Values 5.1 Definitions 5.2 Properties of Bigenvalues and Bigenvectors Lee 5.2.1 General Results... 0... 0..0.00-00.0.08 5.2.2 Optimization Properties of Eigenvalues .. 5.2.3 Matrix Decompositions Involving Eigenvalues... . 5.3 Eigenvalue Inequalities... 0... 0...00-0.0000- 5.3.1 Inequalities for the Eigenvalues ofa Single Matrix 5.3.2 Relations Between Eigenvalues of More Than One Matrix Results for the Spectral Radius we we Singular Values... eee 5.5.1 General Results . . 5.5.2 Inequalities go a Matrix Decompositions and Canonical Forms ....... . - 6.1 Complex Matrix Decompositions . 6.1.1 Jordan Type Decompositions . 6.1.2 Diagonal Decompositions 32 32 34 37 39 41 4l 4l 43 45 47 AT 49 51 52CONTENTS @ 6.1.3 Other Triangular Decompositions and Factorizations . . 6.1.4 Miscellaneous Decompositions . . . 6.2 Real Matrix Decompositions 6.2.1 Jordan Decompositions 6.2.2 Other Real Block Diagonal and Diagonal Decompositions 90 6.2.3 Other Triangular and Miscellaneous Reductions Vectorization Operators . - 7.1 Definitions - 72 flukes for the vec Operator 7.3 Rules for the vech Operator Vector and Matrix Norms 8.1 General Definitions . . . 8.2 Specific Norms and Inner Products 8.3 Results for General Norms and Inner Products ....... - - 8.4 Results for Matrix Norms ....- 0... 0.0005 8.4.1 General Matrix Norms 8.4.2 Induced Matrix Norms 8.5 Properties of Special Norms... . . . 8.5.1 General Results 8.5.2 Inequalities Properties of Special Matrices 9.1 Circulant Matrices . oe . 9.2 Commutation Matrices... 2... 9.2.1 General Properties 9.2.2 Kronecker Products : 9.2.3. Relations With Duplication and Elimination Matrices 9.3 Convergent Matrices . . 9.4 Diagonal Matrices... .. 9.5 Duplication Matrices 9.5.1 General Properties 92 95 95 97 99 101 101 103 104 106 106 . 108 109 . 109 iil . 113 113 , Ho Hé Wz . 1B 119 - 120 - 122 122 9.5.2 Relations With Commutation and. Elimination Matrices 123 9.5.3 Expressions With vec and vech Operators . 9.5.4 Duplication Matrices and Kronecker Products . 9.5.5 Duplication Matrices, Elimination Matrices and Kro- necker Products 9.6 Elimination Matrices wae 9.6.1 General Properties... . 123 124 126 127 127 9.6.2 Relations With Commutation and ‘Duplication Matrices 127 9.6.3 Expressions With vec and vech Operators . 9.6.4 Elimination Matrices and Kronecker Products . 128 128viii 98 9.9 gt 91 91 3 CONTENTS 9.6.5 Elimination Matrices, Duplication Matrices and Kro- necker Products Hermitian Matrices os cae 9.7.1 General Results . . Se 9.7.2 Wigenvalues of Hermitian Matrices 9.7.3. Eigenvalue Inequalities . 9.7.4 Decompositions of Hermitian Matrices Idempotent Matrices Nonuegative, Positive and Storliastic Matrices 9.9.1 Definitions 9.9.2 General Results . - 9.9.3 Results Related to the Spectral Radius Orthogonal Matrices . . 9.10.1 General Results : 9.10.2. Decompositions of Orthogonal Matrices oo... Partitioned Matrices 9.11.1 General Results 9.11.2 Determinants of Partitioned Matrices 9.11.3. Partitioned Inverses 9.11.4 Partitioned Generalized Inverses 9.11.5 Partitioned Matrices Related to Duplication Matrices Positive Definite, Negative Definite and Semidefinite Matrices 9.12.1 General Properties 9.12.2 Eigenvalue Results 9.12.3 Decomposition Theorems for Definite Matrices Symmetric Matrices 9.13.1 General Properties 9.13.2 Symmetry and Duplication Matrices 9.13.3 Figenvalues of Symmetric Matrices . 9.13.4 Figenvalue Inequalities 9.13.5 Deconipositions of Symmetric and Skew-Symmetric Matrices Triangular Matrices 9.14.1 Properties of General ‘Triangular Matrices 9.14.2 Triangularity, Elimination and Duplication Matrices 9.14.3. Properties of Strictly Triangular Matrices . . 9.15 Unitary Matrices 10 Vector and Matrix Derivatives 10.1 Notation . 10.2 Gradients and Hessian, Matrices of Real Valued Functions with Vector Arguments 10.2.1 Gradients . 147 130 131 131 133 134 137 138 139 139 140 11 142 143 144 144 . 145 146 148 . ‘49 158 159 163 164 164 165 167 167 174 174CONTENTS 10.3 10.4 10.6 10.7 10.8 10.9 10.2.2 Hessian Matrices . . Derivatives of Real Valued Functions with Matrix ‘Arguments . 10.3.1 General and Miscellaneous Rules . 10.3.2 Derivatives of the Trace . 10.3.3 Derivatives of Determinants . Jacobian Matrices of Linear Functions . . . . 10.4.1 Linear Functions with General Matrix Arguments 10.4.2 Linear Functions with Symmetric Matrix Arguments 10.4.3 Linear Functions with Triangular Matrix Arguments 10.4.4 Linear Functions of Vector and Matrix Valued Func- tions with Vector Arguments Product Rules . oe 10.5.1 Matrix Products 10.5.2 Kronecker and Hadamard Products . 10.5.3 Functions with Symmetric Matrix Arguments 10.5.4 Functions with Lower Triangular Matrix Arguments 10.5.5 Products of Matrix Valued Functions with Vector Arguments Jacobian Matrices of Functions Tnvolving Inverse Matrices 10.6.1 Matrix Products 10.6.2 Kronecker and Hadamard Products a 10.6.3 Matrix Valued Functions with Vector Arguments Chain Rules and Miscellaneous Jacobian Matrices Jacobian Determinants 10.8.1 Linear Transformations +s 10.8.2 Nonlinear Transformations ....... . Matrix Valued Functions of a Scalar Variable 11 Polynomials, Power Series and Matrices 11.1 11.2 11.3 Definitions and Notations .............4. 11.1.1 Definitions and Notation Related to Polynomials 11,1.2 Matrices Related to Polynomials .....-.....5- 11.1.3 Polynomials and Power Series Related to “Matrices Results Relating Polynomials and Matrices Polynomial Matrices 11.3.1 Definitions : 11.3.2 Results for Polynomial Matrices Appendix A Dictionary of Matrices and Related Terms References . . Index 188 195 196 198 198 199 200 202 204 204 206 208 211 . 2it Qu 213 214 215 217 217 221Preface Nowadays matrices are used in many fields of science. Accordingly they have become standard tools in statistics, econometrics, mathematics, engineering and natural sciences textbooks. In fact, many textbooks from these fields have chapters, appendices or sections on matrices. Often collections of those results used in a book are included. Of course, there are also numerous books and even journals on matrices. For my own work I have found it useful, however, to have a collection of important matrix results handy for quick reference in one source. Therefore I have started collecting matrix results which are important for my own work. Over the years, this collection has grown to an extent that it may now be a valuable source of results for other researchers and students as well. To make it useful for less advanced students I have also included many elementary results The idea is to provide a collection where special matrix results are casy to locate. Therefore there is some repetition because some results fit under different headings and are consequently listed more than once. For example, for suitable matrices A, B and C, A(B+C) = AB + AC is a result on matrix multiplication as well as on matrix sums. It is therefore listed under both headings. Although reducing search costs has been an important objective in putting together this collection of results, it is still possible that. a specific result is listed in a place where I would look for it but where not everyone else would, because too much repetition turned out to be counterproductive. Of course, this collection very much reflects my own personal preferences in this respect. Also, it is, of course, not a complete collection of results. In fact, in this respect it is again very subjective. Therefore, to make the volume more useful to others in the future, I would like to hear of any readers’ pet results that I have left out. Moreover, I would be grateful to learn about errors that readers discover. It seerns unavoidable that flaws sneak in somewhere. In this book, definitions and results are given only and no proofs. At the eud of most sections there is a note regarding proofs. Often a reference is made to one or more textbooks where proofs can be found. No attempt has been made to reference the origins of the results. Also, computational algorithmsxii are not given. These may again be found in the references. As mentioned earlier, it is hoped that this book is useful as a reference for researchers aud students. I should perhaps give a warning, however, that it requires some basic knowledge and understanding of matrix algebra. It is not meant to be a tool for self teaching at an introductory level. Before searching for specific matrix results readers may want to go over Chapter | to familiarize themselves with my notation and terminology. Definitions and brief explanations of many terms related to matrices are given in an appendix. Generally the results are stated for complex matrices (matrices of complex numbers). Of course, they also hold for real matrices because the latter may be regarded as special complex matrices. In some sections and chapters many results are formulated for real matrices only. In those instances a note to that effect appears at the beginning of the section or chapter. To make sure that a specific result holds indeed for complex matrices it is therefore recommended to check the beginning of the section and chapter where the result is found to determine possible qualifications. Finally, I would like to acknowledge the help of many people who com- mented on this volume and helped me avoiding errors. In particular, I am grateful to Alexander Benkwitz, Jorg Breitung, Maike Burda, Helmut Her- wartz, Kirstin Hubrich, Martin Moryson, and Rolf Tschernig for their care- ful scrutiny. Of course, none of them bears responsibility for any remaining errors. Financial support was provided by the Deutsche Forschungsgemiein- schaft, Sonderforschungsbereich 373. Berlin, April 1996 Helmut LiitkepohlList of Symbols General Notation = summation I product = equivalent to, by definition equal to > implies that, only if = if and only if . 3.14159... i V=T, imaginary unit complex conjugate of a complex number c In natural logarithm exp exponential function sin sine function cos cosine function max maximum min minimum sup supremum, least upper bound inf infimum, greatest lower bound d. af derivative of the function f(x) with respect to x of : . an partial derivative of the function f(-) with respect to r nt =1 n ! n nu . . (") = Rnb! binomial coefficient Sets and Spaces set ft R real numbers € complex numbersIR" IR™" c men General Matrices A= (a) A(mxn) Special Matrices Dm Im Kin ot Kin Lm Omxn 0 Matrix Operations A+B A-B AB positive integers integers m-dimensional Euclidean space, space of real (m x 1) vectors real (m x n) matrices m-dimensional complex space, space of complex (m x 1) vectors complex (m x n) matrices matrix with typical element a;; matrix with m rows and n columns matrix consisting of submatrices A, and A matrix consisting of submatrices 41,.-., An matrix consisting of submatrices A, and Ap matrix consisting of submatrices A1,..., An partitioned matrix consisting of submatrices Ains Ata, Aaa, Azz partitioned matrix consisting of submatrices Ai; (m? x $m(m + 1)) duplication matrix, 9 (m x m) identity matrix, 10 (mn x mn) commutation matrix, 9 ($m(m + 1) x m?) elimination matrix, 9 (mx n) zero or null matrix, 2 zero, null matrix or zero vector sum of matrices A and B, 3 difference between matrices 4 and B, 3 product of matrices A and B, 3xv cA, Ac product of a scalar ¢ and a matrix 4,3 A@B Kronecker product of matrices A and B. 3 AOB Hadamard or elementwise product. of matrices A and B,3 A@B direct sum of matrices A and B. 4 Matrix Transformations and Functions det A, det( A) determinant of a matrix 4, 6 tr A, te(A) trace of a matrix A, 4 All norm of a matrix A, 101 Alle Euclidean norm of a matrix A, 103 rk A, rk( A) rank of a matrix A, 12 [Alavs absolute value or modulus of a matrix 4.5 At transpose of a matrix A, 5 A conjugate of a matrix A, 5 An conjugate transpose or Hermitian adjoint of a matrix A, 5 Awe adjoint of a matrix A, 6 Av! inverse of a matrix A, 7 Ay generalized inverse of a matrix A, 7 At Moore-Penrose inverse of a matrix A, 7 Ai ith power of a matrix A, 7 AM? square root of a matrix A, 8 ay 0 dg([ais]) = . 0 anam an 0 diag(a11,..-, 4mm) = 0 mm vec, col column stacking operator, 8 rvec row stacking operator, 8 row operator that stacks the rows of a matrix in a column vector, 8 vech half-vectorization operator, 8 Matrix Inequalities A>0 all elements of A are positive real numbers. 4A>0 all elements of A are nonnegative real numbers, 4 A<0 all elements of A are negative real numbers, 4 AB each element of A is greater than the corresponding element of B, 4 ADB each element of A is greater than or equal to the corresponding element of B, 4 Other Symbols Related to Matrices minor(@i;) minor of the element aj; of a matrix A, 6 cof(a,;) cofactor of the element aj; of a matrix A, 6 MA) eigenvalue of a matrix A, 63 Amar OF Amar( A) maximum eigenvalue of a matrix A, 63 Amin OF Amin( A) minimum eigenvalue of a matrix A, 63 o(A) singular value of a matrix A, 64 Omar OF Fnax(A) maximum singular value of a matrix A, 64 Omin OF Omin( A) minimum singular value of a matrix A, 64 pA) spectral radius of the matrix A, 64 pal) characteristic polynomial of a matrix A, 63 gal) minimal polynomial of a matrix A, 2151 Definitions, Notation, Terminology 1.1 Basic Notation and Terminology An (mx n) matrix A is an array of numbers cr Qin a2, Q22 a2n as |" 2 | = ldsaye vg = feu] Ami Im? . amn Alternative notations are: A(mxn), A, A=[aj] (mx n) (m x n) m and n are positive integers denoting the row dimension and the column dimension, respectively. (m x n) is the dimension or order of A. The aij, i= 1,...,m, j= 1,...,n, are real (elements of IR) or complex numbers (elements of ©). They are the elements or entries of the matrix A and a,; is the ijth element or entry of A. The matrix A is sometimes called a real matrix if all its elements are real numbers. If some of its elements are complex numbers the matrix A is said to be a complex matrix. The set of all complex (mx n) matrices is sometimes denoted by C'™*" and the set of all real (m xn) matrices is denoted by IR™*" Further notations: An Aiz s+ Atg An Ar +++ Aggy : = [Ai] {ai}. Apt Apo + Apg Here the Ajj are (mi x nj) submatrices of A with S’,m: = m and jai) =n. A matrix written in terms of submatrices rather than individual2 HANDBOOK OF MATRICES elements is often called a partitioned matrix or a block matrix. Special cases are A= (Aj... An] = [Arie Anh where the A, are blocks of columns, and Al A=]: |, Am where the A; are blocks of rows of A. A matrix an * Gim A = (mem) |g nm having the same number of rows and columns is a square or quadratic matrix. The elements @11,422,...,@mm (aij,? = 1,...,m™) constitute its principal diagonal Im = = [ai] (mxm) |} 9 1 with ay =1 for i=1,...,m, and a; = 0 for ¢ # j is an (mx m) identity or unit matrix and 0-0 Oman =}: = [aij] 0 0 (mx n) with aj; = 0 for all i,j is an (mx n) zero or null matrix. It is sometimes simply denoted by 0. A (1 x n) matrix a=[a,,....a,] is an n-dimensional row vector or (1 x n) vector. An (m x 1) matrix b b= bm is an m-vector, m-dimensional vector or m-dimensional column vector. The set of all real m-vectors is often denoted by IR” and the set of all complex m-vectors is denoted by ©”. Many more special matrices are listed in Section 1.5 and the Appendix.DEFINITIONS, NOTATION, TERMINOLOGY 1.2 Operations Relating Matrices Addition: A = [aj] (m xn), B= [by] (m * n) A+ B= [aj + bj] (mx n) (for the rules see Section 2.1). Subtraction: A = [aj] (m xn), B = [bij] (mx n) A- B= {a,; — bij] (mx n) (for the rules see Section 2.1). Multiplication by a scalar: A= (ajj] (m x n), ¢ a number (a scalar) [caj) (m x n) Ac = [cai] (m x n) (for the rules see Section 2.3). Matrix multiplication or matrix product: A= [ay] (mx n), B= [bi] (2 xp) AB= [Sour (m x p) rat (for the rules see Section 2.2). Kronecker product, tensor product or direct product: A= [aij] (mx n), B= [bij] (Px 9) aBo- ainB A@B= : : (mp x nq) AmB -+: OmnB (for the rules see Section 2.4) Hadamard product, Schur product or elementwise product: A= [aj] (mx n), B= [bj] (mx n) A@ B & (aijbij) (m x n) (for the rules see Section 2.5)4 HANDBOOK OF MATRICES Direct sum: A(m x m), B(n xn) ape} 0 0 b| ((m +n) x (m+n)) (for the rules see Section 2.6). Hierarchy of operations: If more than two matrices are related by the foregoing operations, they are performed in the following order (i) operations in parentheses or brackets, (ii) matrix multiplication and multiplication by a scalar, (iii) Kronecker product and Hadamard product, (iv) addition and subtraction, (v) direct sum. Operations of the same hierarchical level are performed from left to right. 1.3. Inequality Relations Between Matrices {ai;], B= (bij) (7m x n) real: A>0 > aj >0,8=1,....m f=l,....n ADO = aj >0,i=1,...,.m, Lan, ASB = aj >bj,i=1,....m, fal,...n, ADB => ay >bj,i=1,....m, jfal,...n Warning: In other literature inequality signs between matrices are sometimes used to denote different relations between positive definite and semidefinite matrices. 1.4 Operations Related to Individual Matrices Trace: A {ayj) (mm x m) trAstr(A)= tet + Oram (for its properties see Section 4.1).DEFINITIONS, NOTATION, TERMINOLOGY Absolute value or modulus: A = [a,;] (m x n) lavilabe —|ar2lats ~~» |@inlabs lazilabe |@zzlabs ~~ |@2nlabs [Alas = [ {aij labs ] = : . . (mxn) lamilabs |@m2labs -*> lamnlabs where |clabs denotes the modulus of a complex number ¢ = c; + icy which is defined as [clans = \/e? + c2 = Vee. Here ¢ is the complex conjugate of ¢. Properties of the absolute value of a matrix are given in Section 3.8. Warning: In other literature |A| sometimes denotes the determinant of the matrix A Transpose: A = [a,,] (mx n) ayy) a2 an ay, aay amy : a2) 422 "Qn aj2 422 m2 As] = Gm1 Im? amn Gin An ann (nx m) that is, the rows of A are the columns of A’ (see Section 3.1 for its properties). In some other literature the transpose of a matrix A is denoted by AT Conjugate: jaij] (m x n) A= [ai]. where aj; is the complex conjugate of aj; (see Section 3.2 for its properties). Conjugate transpose or Hermitian adjoint: A = [a;j] (m x n) An 3] (nx m) (see Section 3.3 for its properties) Diagonal matrices: A = [a;;] (m x m) an 0 dg(A) = 0 @mm6 HANDBOOK OF MATRICES diag(a11,...,amm) = diag : = (see Section 9.4 for further details) Determinant: A = [ajj] (m x m) det A = det(A) = S0(-1Paui,azig X ++ X amin: where the sum is taken over all products consisting of precisely one element from each row and each column of A multiplied by —) or 1, if the permutation é1,...,ém is odd or even, respectively (see Section 4.2 for the properties of determinants) Minor of a matrix: The determinant of a submatrix of an (m x m) matrix A is a minor of A. Minor of an element of a matrix: A = (a;;] (mx m). The minor of a,; is the detertninant of the matrix obtained by deleting the ith row and jth column from A, 41 aijn1 Qi j+t . aim : aj Saja je a a1 minor(aj;) = det Mw Lin} Ginnie a Getto Ata jr Gi+ij41 * Gitim amt dm j-1 Om j+1 7 dmm Principal minor: A = {aij] (m x m) ao ae det}: : Qe ake is a principal minor of A for k = ym=1 Cofactor of an element of a matrix: A = (ajj] (m x m). The cofactor of ai; is cof(aij) El (—1)'*7 minor(a,;) Adjoint: A= [ai] (mx m),m>1, cof(aiy) cof(aim) Aw = : = [cof(aj;)]’ coflami) +++ cof(amm)DEFINITIONS, NOTATION, TERMINOLOGY Aed =1 if m= 1 (see Section 3.4 for the properties of the adjoint) Inverse: A = [a,j] (mm) with det(A) # 0. The inverse of A is the unique (m x m) inatrix AW! satisfying AA~! = A~1A = Ip, (see Section 3.5 for its properties) Generalized inverse: An (n x m) matrix AW is a generalized inverse of the (m xn) matrix A if it satisfies AA~A = A (see Section 3.6 for its properties). Moore -Penrose (generalized) inverse: The (n xm) matrix At is the Moore~Penrose (generalized) inverse of the (m x n) matrix 4 if it satisfies (i) AAPA= A, (ii) At AAt = At, (iii) (AAt)* = Aat, (iv) (At A)! = AtA {sce Section 3.6.2 for its properties). Power of a matrix: A (m x m) Ta: A= Ax: *A for positive integers i i times Bed Im for i=0 “1 (2.4) for negative integers i, if det(.A) 4 0 If A can be written as oy 0 A=U 4 ua 0 dm for some unitary matrix U’ (see Section 1.5 and Chapter 6) then the power of A is defined for any a € IR, a > 0, as follows: Ae 0 Av SU . ut, 0 d2, This definition applies for instance for Hermitian and real symmetric matrices (see Section 1.5 for the definitions of Hermitian and symmetric matrices and Section 3.7 for the properties of powers of matrices).8 HANDBOOK OF MATRICES Square root of a matrix: The (m x m) matrix Al/? is a square root of the (m x m) matrix A if A!/?.41/? = A. Elsewhere in the literature a matrix B satisfying BB = A or BB! a square root of A. A is sometimes regarded as Vectorization: 4 = {a;,] (m x n) ay a2) vec A= vee(A) = col(A)= |” (mn x 1) that is, vec stacks the columns of A in a column vector rvec( A) = [vee( A’)] that is, rvec stacks the rows of A in a row vector and row( A) = vee( A’) = rvec( Ay! that is, row stacks the rows of A in a column vector. (See Chapter 7 for the properties of vectorization operators) Half-v ctorization: 4 ja;;) (m x m) ay an Qn) ayy vech A = vech(A) = (4m(m + 1) x 1) Qmm that is, vech stacks the columns of A from the principal diagonal downwards in a column vector (see Chapter 7 for the properties of the half-vectorization operator and more details)DEFINITIONS, NOTATION, TERMINOLOGY 9 1.5 Some Special Matrices Commutation matrix: The (mn x mn) matrix Ayn is a commutation matrix if vec(A’) = Kmnvec(A) for any (m x n) matrix A. It is sometimes denoted by Aina. For example, 100000 000100 . . 010000 K32 = N32 = 000010 001000 000001 is a commutation matrix. (For the properties of commutation matrices see Section 9.2.) Diagonal matrix: An (m x m) matrix an 0 0 Qinm with a;, = 0 for i # j is a diagonal matrix. (For the properties of diagonal matrices see Section 9.4.) Duplication matrix: An (m? x gm(mn +1)) matrix Dm is a duplication matrix if vec(A) = Dj vech(A) for any symmetric (m x m) matrix A For example, 100000 010000 001000 010000 Ds=|0 00100 ooov010 001000 000010 000001 is a duplication matrix. (For the properties of duplication matrices see Section 9.5.) Elimination matrix: A (}m(m + 1) x m?) elimination matrix Lm is defined such that vech(A) = Lmvee(A) for any (m x m) matrix A10 HANDBOOK OF MATRICES For example, ui ecoosoe ecosoors coorc]e ecocoso corooco ercooc coococo ecosocoso -oocooo is an elimination inatrix. (For the properties of elimination matrices see Section 9.6.) Hermitian matrix: An (m x m) matrix 4 is Hermitian if A’ = A¥ = A (For its properties see Section 9.7.) Idempotent matrix: An (mxm) matrix A is idempotent if A? = A. (For the properties see Section 9.8.) Identity matrix: An (7 x m) matrix 1 0 In = = [aij] 0 1 with aj = 1 for i= 1,...,m and aj; = 0 for i # j is an identity or unit matrix. Nonnegative matrix: A real (m x n) matrix A = [aj] is nonnegative if ai > 0 fori=1,...,m, j= 1,...,n. (For the properties see Section 9.9.) Nonsingular matrix: An (m x m) matrix A is said to be nonsingular or invertible or regular if det(A) # 0 and thus A~! exists. (For the rules for matrix inversion see Section 3.5.) Normal matrix: Au (mm x m) matrix A is normal if A4 A = AA” Null matrix: An (m x n) matrix is a null matrix or zero matrix, denoted by Omxn or simply by 0, if all its elements are zero Orthogonal matrix: An (m x m) matrix A is orthogonal if A is nonsingular and A’ = AW. (For the properties see Section 9.10.) Positive and negative definite and semidefinite matrices: A Hermitian or real symmetric (m x m) matrix A is positive definite if 2 Ax > 0 for all (m x 1) vectors x # 0; it is positive semidefiniteDEFINITIONS, NOTATION, TERMINOLOGY M if x Ax > 0 for all (m x 1) vectors z; it is negative definite if a Az <0 for all (m x 1) vectors x # 0; it is negative semidefinite if x Ax < 0 for all (m x 1) vectors 2; it is indefinite if (m x 1) vectors x and y exist such that 2” Ax > 0 and y" Ay < 0 (see Section 9.12) Positive matrix: A real (7m xn) matrix A = [aj;] is positive if aj; > 0 for yesssm, J = 1,...,n (see Section 9.9). i= Symmetric matrix: An (m x m) matrix A = [aj;] with aj; = aj¢, i,j = 1,...,m is symmetric. In other words, A is symmetric if A’ = A (see Section 9.13) Triangular matrices: An (m x m) matrix a, 0 - 0 = [ay] m1 sot Onm with a,j = 0 for j > ¢ is lower triangular. An (m x m) matrix an Qn a5] 0 mm with a; = 0 for i > j is upper triangular (see Section 9.14) Unitary matrix: An (m xm) matrix A is unitary if it is nonsingular and A® = A~? (see Section 9.15). Note: Many more special matrices are listed in the Appendix 1.6 Some Terms and Quantities Related to Matrices Linear independence of vectors: The m-dimensional row or column vectors #),..., 2% ate linearly independent if, for complex numbers Cheeves€ks C12, +++ ++ cety = 0 implies cy = --» = cy = 0. They are linearly dependent if ¢)z, + ---+cez% = 0 holds with at least one cj # 0. In other words, 21,..., 2g are linearly dependent if a; € C exist such that for some i € {1,...,k}, 2) = ane, te-° + Qj-atinn + Ousitign tot oRrE12 HANDBOOK OF MATRICES Rank: A = [a;;] (m x n) rk A= rk(A) = maximum number of linearly independent rows or columns of A row rk A = row rk(A) = maximum number of linearly independent. rows of A col tk A = col rk(A) maximum number of linearly independent. columns of A (for rules related to the rank of a matrix see Section 4.3) Elementary operations: The following changes to a matrix are called clementary operations (i) interchanging two rows or two columns, (ii) multiplying any row ot column by a nonzero number, (iii) adding a multiple of one row to another row, (iv) adding a multiple of one column to another column Quadratic form: Given a real symmetric (»n x m) matrix A, the function Q:IR™ — IR defined by Q(z) = 2'Az is called a quadratic form. The quadratic form is called positive (semi) definite if A is positive (semi) definite. It is called negative (semi) definite if 4 is negative (semi) definite. It is indefinite if A is indefinite Hermitian form: Given a Hermitian (m x m) matrix A, the function Q:C” — IR defined by Q(z) = 2” Az is called a Hermitian form The Hermitian form is called positive (semi) definiteif A is positive (semi) definite. It is called negative (semi) definite if A is negative (semi) definite. It is indefinite if A is indefinite Characteristic polynomial: The polynomial in A given by det(Al, — A) is the characteristic polynomial of the (m xm) matrix 4 (see Section 5.1). Characteristic determinant: The determinant det(Alm — A) is the characteristic determinant of the (m xm) matrix A Characteristic equation: The equation det(Am — A) = 0 is the characteristic equation of the (m x m) matrix A Eigenvalue, characts value, characteristic root or latent root: The roots of the characteristic polynomial of an (m x m) matrix 4DEFINITIONS, NOTATION, TERMINOLOGY 13. are the eigenvalues, the characteristic values, the characteristic roots or the latent roots of A (see Chapter 5). Eigenvector or characteristic vector: An (mx 1) vector v # 0 satisfying Av = Av, where A is an eigenvalue of the (m x m) matrix A, is an eigenvector or characteristic vector of A corresponding to or associated with the eigenvalue 4 (see Chapter 5) Singular value: The singular values of an (m x n) matrix A are the nonnegative square roots of the eigenvalues of AA” if m < n, and of A# A, if m > n (see Chapter 5)2 Rules for Matrix Operations In the following all matrices are assumed to be complex matrices unless otherwise stated. All rules for complex matrices also hold for real matrices because the latter may be regarded as special complex matrices. 2.1 Rules Related to Matrix Sums and Differences (1) A,B(mxn),ceC: (a) AE B= BEA. (b) c(A + B) =(A+B)e=cA+cB (2) A,B,C(mxn): (AtB)4C=At(B4C)=ALBEC. (3) A(mxn),B,C(nxr): A(B£C)= ABEAC. (4) A,B(mxn),C(nxr): (A+ B)C = AC + BC (5) A(m xn), B,C (p xq): (a) A@(BL£C) =A@BLAOC. (b) (BEC)@A= BOALCOA (6) A,B,C (mx n): (a) AQ(BEC)=AQBLAOC. (b) (AEB) @C=AGCEBOC. (7) A,C (mxm), B,D (nxn): (A®B)£(C@D) = (ALC)@(BLD) (8) A,B (mxn): (a) |A + Blavs < |Alabs + |Blavs- (b) (A+ BY = A's B’ (c) (A+ B)4 = AM + BH, (d) AFB=A+B16 HANDBOOK OF MATRICES (9) A,B (m xm): (a) tr(.4 + B) = tr(A) + tr(B). (b) dg(A + B) = dg(A) + dg( B) (c) vech( A + B) = vech( A) + vech( B) (10) A,B (mx n): (a) tk(A + B) < rk(A) + rk(B) (b) vec( A B) = vec(A) + vec( B). (11) A (m xm): (a) A+’ is a symmetric (m x m) matrix. (b) A= A’ is a skew-symmetric (m x m) matrix, (c) A+ 44 is a Hermitian (a x m) matrix (d) 4- A is a skew-Hermitian (m x m) matrix Note: All the rules of this section follow easily from basic principles by writing down the typical elements of the matrices involved. For further details consult introductory matrix books such as Bronson (1989), Barnett (1990). Horn & Johnson (1985) and Searle (1982). 2.2 Rules Related to Matrix Multiplication (1) A(m xn), B (nx p).C(p xq): (ABYC = A(BC) = ABC. (2) A,B (mx m): AB # BA in general, (3) A(mxn),.B.C (nx p): A(BEC) = ABE AC. (4) A,B(mxn),C(n xp): (At B)C = AC + BC. (5) A(m xn), Bip xq)C (nx 0), D(q xs): a (A 2 BUC D) = AC BD. (6) A,C(mxm),B,D(nxn): (Ab BYC@ D)= AC & BD. (7) A(mxm),B(mxn): |ABlavs < |Alats|Blabs- (8) A (m xn), B (nx p): (a) (ABY = BIA (b) (AB)¥ = BA AM, (c) AB = AB. (9) A,B (im xm) nonsingular: (AB)"! = Boh A~tRULES FOR MATRIX OPERATIONS, 17 (10) A = [Ajj] (Om x n) with Aiz (me x nj),B = [By] (nm x p) with Bi (™ x Bj) AB= [= sou. E (11) A(mxn),B(n xm): tr(AB) = t(BA) (12) A,B(m xm): det(AB) = det( A) det(B). (13) A(mxn),B(n xp): tk(AB) < min{rk(A),rk(B)} (14) A(m xn) tk(AA‘) = rk(A‘A) = rk( A). (15) A(m x n),B(n xp): vec( AB) = (Jp © A)vec(B) = (B' ® Im )vec(A) = (BY & A)vec( In) (16) A (m xn), B (nx p),C (p xq): vee(ABC) = (C" 9 A)vec(B). (17) A(mxn),B(nxm): tr(AB) = vee(A’)'vec(B) = vec( B’)/vec( 4). (18) A(mxn), B(nxp),C (pxm): tr(ABC) = vec(A'y(C'2 Dvee(B). (19) A (Qn xn), B(x p).C (px q),D(q xm): tr(ABCD) = vee(D')'(C' @ A)vec( B) = vec( D(A C’)vec(B’) (20) A(m x m).B(mxn): A nonsingular > rk(AB) = rk(B) (21) A(m xn). BO xn): Buvonsingular = rk(.4B) = rk(4) (22) A (mm xm), B (on xn). C(n xn): A,C nonsingular > tk(ABC) = rk(B) (23) A (an x m) nonsingulas: AA7) = A71A = In (24) An xn) (a) Alp =ImA=A (b) AOnxp = Omxp, OpxmA = Opxn (25) A (m x n) real (a) AA’ and A‘A are symmetric positive semidefinite. (b) tk(A) =o => AA! is positive definite (c) rk(A) = = A'A is positive definite (26) A(m xn) (a) AA# and AY 4 are Hermitian positive semidefinite. (b) rk(A) =m => AA® is positive definite. (c) rk( A) =n => A* 4 is positive definite.18 HANDBOOK OF MATRICES Note: Most of the results of this section can be proven by considering typical elements of the matrices involved or follow directly from the definitions (see also introductory books such as Bronson (1989), Barnett (1990), Horn & Johnson (1985), Lancaster & Tismenetsky (1985) and Searle (1982)). The rules involving the vec operator can be found, e.g., in Magnus & Neudecker (1988) 2.3 Rules Related to Multiplication by a Scalar (1) A(mxn), ere ©: (a) er(exA) = e1 (Ata) = (ere2)A = A(erer) (b) (cr #ea)A = 1A ted. (2) A,B(mxn),c€C: cAtB)=cAtcB. (3) A(mxn),B(nxp), cce2E€€: (ce, A)(erB) = cree AB. (4) A(m xn), B(pxq), cco EC: (a) (A B) =(1.A)@ B= AG(B). (b) cA co B = (eyc2)(A @ B) (5) ABQ xn), crc. EC: (a) (A® B) =(e1.A)@ B= AO (1B). (b) cA @ c2B = (cico)(A@ B) (6) A(mxm), B(nxn)c€€: cA®B)=cAGcB. (7) A(mxn): 04 = AO = Omxn (8) A(mxn).ce: (a) [eAlabs = Iclavsl labs. (b) (cA) = eA’ (c) (cA)# = cA# (9) A(mxm),ceC: (a) tr(eA) = ¢ tr(A) (b) dg(cA) = cdg( A). (c) Ais nonsingular, ¢ #0 = (cA)7! = 2A7!. (d) det(cA) = e™det( A) (e) vech(cA) =e vech( A). ({) Ais eigenvalue of A = cA is eigenvalue of cA.RULES FOR MATRIX OPERATIONS 19 (10) A(mxn),ceC: (a) ¢#0 => rk(cA) = rk(A). (b) ¢ #0 = (cA) = Lt. (c) vee(cA) = ¢ vec(A). Note: All these rules are elementary and can be found in introductory matrix books such as Bronson (1989), Barnett (1990), Horn & Johnson (1985), Lancaster & Tismenetsky (1985) or follow directly from definitions. 2.4 Rules for the Kronecker Product (1) A(mxn),B(pxq): 4@B#¢ B® A in general (2) A(mxn), B.C(pxq): A@(BEC)=A@BEAC (3) A(m xn), B (px q),C (rx 8): A®(BQC)=(A@B)OC=AQBOC. (4) A,B (mx n), C.D (px 9) (A+ B)@(C+D)=AQC+AQD+BOC+BOD (5) A(m xn), B (px q),C (n xr), D (qx): (A® B)(C@ D) = AC@ BD. (6) A(mxn), B (px q),C (rx 8), D (nx k), E(x 0), F (sx t): (A® B@C\(D@E®F) = AD® BEQCF. (7) A(mxn),c€€: c@A=cA=AGc (8) A(mxn),B(pxq), ode € cA @ dB = cd(A@ B) = (cd A) @ B = AQ (cdB) (9) A(mxm),B(nxn),C(pxp): (A®B)@C = (A@C)O(BOC). (10) A (mx n), B(p xq): (a) (A@ BY = A’'@B’. (b) (A@ B)# = AM @ BH, () A@B=A@B (4) (A@B)* = At @ Bt20 HANDBOOK OF MATRICES {e) |A© Blavs = |Alats © |Blabs (f) rk(A @ B) = rk(A) rk(B). (11) A(mx m), B (nxn) (a) A,B nonsingular > (4 @ B)~) (b) tr(A @ B) = tr( 4) tr(B), (c) dg(A® B) = dg(A) @ dg( B) (d) det(A @ B) = (det A)"(det By” {e) A(A) and A(B) are eigenvalues of A and B, respectively, with associated eigenvectors v(A) and v(B) = (A) - A(B) is eigenvalue of 4 © B with eigenvector v(A) @ v( B). (12) c.y(mx I): 2 Sys yr = yen’. (13) e(mx I,y(nx 1): vec(zy¥) =y@r (14) A(mx_n),B(n x p).C(pxq): (C1 @ A)vec(B) = vec( ABC) (15) A (mx n), B (nx p),€ (px g).D (qx m): tr(ABCD) vee( D'(C' ® A)vec( B) = vee(D)'(A@ C'yvec( B’), vee(D')'(C’ & A)vec(B) = vec(A')'(D' © B)vee(C) vee( B')'(A' @ C)vee(D) vec(C")/(B’ © Dyvee( A). Note: A substantial collection of results on Kronecker products including many of those given here cau be found in Magnus (1988). Some results are also given in Magnus & Neudecker (1988) and Lancaster & Tismenetsky (1985). There are many more rules for Kronecker products related to special matrices, in particular to commutation, duplication and elimination matrices (see Sections 9.2, 9.5 and 9.6). 2.5 Rules for the Hadamard Product (1) A. B(mxn)ce€ (a) ABB= BWA (b) (4. B) = (cA) B = AG (cB). (2) A.B.C(mxn): (a) AG (BS C)=(A* B)SC=AG BEC.RULES FOR MATRIX OPERATIONS 21 (b) (At B)OC=AOCEBOC. (3) A,B,C, D(mxn): (A+B) G(C+D)=AOCH+AOQD+BOC+BOED. (4) A.C (mxm), B,D (nxn): (A®B)O(C#D) = (A@C)B(BED) (5) A,B (m xn): BY =A OB. BH = A4 @ BH, (c) ADB=AOB. (4) [4 Blas = |Alabs ® |Blats. (6) A(mxn): A®Om«n = Omxn © A= Oman (7) A(mxm): A® Im = Im @ A= dg(A) | (8) A(mxn)J=]i i |(mxm): A@I=A=IOA Pod (9) A,B,C (mxn): trf[A(BOC)] = te[(A’® BC). (10) A,B (mx m), f= (1..., 1)! (mx 1): tr(ABY) = j/(A® B)j. (11) 4,B,D(mxm), §=(1,....1) (rx 1) Ddiagonal > tr(ADB’D) = j'D(A © B)D)j. (12) 4,B,D(m xm): D diagonal = (DA)@ (BD) = D(A® B)D. (13) A,B (mx n): (a) vec(A # B) = diag(vee A)vec(B) = diag( vec B)vec( A). (b) vec(A % B) = (vee A) @ vee(B) (14) A,B (m x m): (a) vech(A © B) = diag(vech A)vech( B) = diag(vech B)vech(A). (b) vech(A @ B) = (vech A) ® vech( B) (15) (Schur product theorem) A,B(m xm): A,B positive (semi) definite = A % B positive (semi) definite. Note: Most results of this section follow directly from definitions. The remaining ones can be found in Magnus & Neudecker (1988). The Schur product theorem is, for example, given in Horn & Johnson (1985).2 HANDBOOK OF MATRICES 2.6 Rules for Direct Sums (1) A(mx m), B (nxn), cE€: (a) c(A@ B) = cA @cB. (b) AA B => ADBEBGA. (2) A, B(m xm), C,D(n xn): (a) (At B)®(C+ D) = (AC) + (BOD) (b) (A®C)(B@ D) = ABECD. (3) A(m xm), B (nx n),€ (px p): (a) A@(BSC)=(AGB)SC=AGBOC. (b) (ABB) @C=(A@C)G(BAC). (4) A,D(mx m), B,E (nxn), C,F (px p): (A® B&C\DGE SP) = ADS BE®CF. (5) A(mx m), B(n xn): (a) |A@ Blabs = |Alabs © |Blavs (b) (A@ BY = AO B’. (c) (A@B)# = AM BH, (d) ABB=AGB {e) (A@ B)) = A”? @ BB", if A and B are nonsingular (f) (A@ B)* = At @ Bt (6) A(mx m),B(n x n) (a) (A @ B) = tr(A) + (B) (b) dg(A ® B) = dg( A) @ dg( B) (c) tk(A @ B) = rk(A) +rk(B) ( ( d) det(A @ B) = det(A)det( B) e) MA) and A(B) are eigenvalues of A and B, respectively, with associated eigenvectors v(A) and v(B) => A(A) and A(B) are eigenvalues of A & B with eigenvectors [*s?] [atm | Note: All results of this section follow easily from the definitions (see also Horn & Johnson (1985) respectively3 Matrix Valued Functions of a Matrix In this chapter all matrices are assumed to be complex matrices unless otherwise stated. All rules for complex matrices also hold for real matrices as the latter may be regarded as special complex matrices. 3.1 The Transpose Definition: The (n x m) matrix ay 2 23 2 Ain a, ar . amt a2, a2 oo Qn 212 4220 + m2 De =| (nx m) Qmi m2 “ dmn Qn Gen “+ dmn is the transpose of the (m x n) matrix A = [a,j], (1) A, B(mxn): (At BY = Alt BY (2) A(mxn),ce€: (cA) =e’. (3) A(mxn),B(nxp): (AB) = BA (4) A(mxn),B(pxq): (AB) =4' OB (5) AB(mxn): (AQBY=A'OB’. (6) A(mxm),B(nxn): (A@BY = 4'@ BY (7) A(mxn): (a) [A’labs = [Alfne: (b) rk(A") = rk(A). (c) (AY = A.24 HANDBOOK OF MATRICES (d) (AM = (AMY = (e) A’ = A’ = Av (8) A (mx m) (a) dg(4’) = dg(). (b) tr(A’) = tr(A). (c) det(4’) = det(A) (d) (A>! = (Av), if A is nonsingular. (e) (AN? = (aed y. (f) A is eigenvalue of A > 2 is eigenvalues of A’. (9) A(mxn) real: (A’)t = (At) (10) A (m Xn), Kinn (mn x mn) commutation matrix: vee(A') = Kmnvec(A) (11) A,B(mxn): vee(B’)'vec(A) = tr(AB) (12) A(m xn), B (mx p),C (qx n), D(q x p): A By] _[a c c p\|"|B8 Dp (13) A(m xm) (a) Ais diagonal > A’ = A. (b) Ais symmetric => A’ = A. (14) A (mx m) nonsingular: A orthogonal 4 4’ = AW?. (15) A(mxn): A’A and AA’ are symmetric matrices. Note: All these results are elementary. They follow either directly from definitions or can be obtained by considering the individual elements of the matrices involved (see, ¢.g., Lancaster & ‘Tismenetsky (1985)). 3.2. The Conjugate Definitio: matrix : The (m x n) matrix A = {4,,] is the conjugate of the (m x n) [ai]. Here a,j denotes the complex conjugate of aj; (1) 4,B(mxn): AEB=A+B. (2) A(mxn),B(n xp): AB= AB (3) A(mxn),ce€ =cA.MATRIX VALUED FUNCTIONS OF A MATRIX 25 (4) A(mxn),B(pxqg): A® (5) A,B(mxn): AOB=A (6) A(mxm),B(nxn): ASB=A@B. (7) A(mxn) (a) |Alabs = |Alavs- (b) rk(A) = rk(A) (c) ASA (d) A . (ce) 44 = AH = At (f) vec(A) = vee(A). (g) Aisreal > A=A (8) A(mxm): (a) tr(A) = tr(A). (b) det(A) (c) Ast (d) Aq} (e) vech(A) = vech(A). (f) dg( A) = dg(4). (g) dis eigenvalue of A with eigenvector v = ) is eigenvalue of A with eigenvector & (9) A(mx n), B (mx p),€ (qx n), D(q x p) A B)_[A B Cc DJ Le DI (10) a1,...,¢m EC: diag@ai....,am) = diag(@1,..., am). 7, if A is nonsingular Note: The results in this section are easily obtained from basic definitions or elementwise considerations and the fact that for c),c2 € C, @@q = 41¢2 3.3 The Conjugate Transpose Definition: The conjugate transpose of the (m x n) matrix A = {a,,] is the (nx m) matrix A4 = A’ = [a,;]! (1) A,B(mxn): (A+B)! = AM + BA26 HANDBOOK OF MATRICES (2) A(mxn),B(nxp): (AB)! = BH AH (3) A(mxn)cEC: (cA)! = cA¥ (4) A(mxn), B(pxq): (A@ By" = AX @ BA. (5) A,B(mxn): (A@B)H = Ato BH (6) A(mxm),B(nxn): (ASB) = AM ge BH. (7) A(mxn): (a) [AM Jabs = [Alles (b) rk(A¥) = rk(A). (c) (AN)4 = A. (d) (4)" aMy (e) AY = AH) (1) (AME = (ate (8) vee(A") = vee("). (h) Aisreal > AY =A’, (8) A(m xm): (a) tr(A”) = tr( A) = tA). (b) det(4¥) = det(4) = det A (c) (AM)°9 = (Aad yt (d) (AM )"! = (47!)", if A is nonsingular (e) dg( A”) = dg( 4) = (dg A)! (9) A(m xn), B (mx p).C (qn). D (gx p): [3 a All Sm | c Dp} ~{ BH pH (10) a,-..,am EC: [diag(ay,...,an))# = diag(ay,...,@m). (11) A(mxm): A Hermitian => AY = 4, (12) A (mx m) nonsingular: A unitary <=> A# = An? (13) A(mxn) (a) AM A and AA" are Hermitian positive semidefinite. {b) tk(A (c) rk(A) =n => A” A is Hermitian positive definite. m => AA” is Hermitian positive definite. Note: The results in this section are easily obtained from basic definitions or clementwise considerations (see, ¢.g., Barnett (1990))MATRIX VALUED FUNCTIONS OF A MATRIX 27 3.4 The Adjoint of a Square Matrix Definition: For m > 1, the (m x m) matrix A?4 = [cof(a;,)]! is the adjoint of the (m x m) matrix A = [aj]. Here cof(aij) is the cofactor of aij. For m= 1, A*# = 1. For instance, for m = 3, det [ 422 a2g ] ae 43, 473 ] det [ ay az ] 1 a32 433 43, 433 a31 432 a2 @ ay a ay a Att =| de} 9? 8 det} OMS | det | Ot 2 432 433 43; 433 431 432 ae [2 a dex [ 2 a} ae | | 422 a3 92, a3 4 aye (1) 4,B(mxm): (AB)*4 = Bad Ad, (2) A(mxm),cEC: (cA)F =e! 404, (3) A(m xm) (a) Ae# = det(A)AM}, if A is nonsingular, (b) AAt% = A24 A = det(A) Im. (c) (yee = (Atay. (4) (AM )94 = (Aad), (e) det(A%#) = (det Ay™-! (f) rk(A) Ad (4) A(mxm),m>2: m if rk(A)=m aaa) { 1 if rk(A)=m-1 0 if rk(A) AT! exists (b) tk(A) =m = At = Ar} (c) A is diagonal dominant = A is nonsingular. (8) a,.-.,m EC La, F 0 fori peeme A= diag(ay,...,am) = Aq! = diag(a;!,....a;,') (7) Im (mx m) identity matrix: 5! (8) A (m x m) nonsingular: (a) A orthogonal <=> A“! = A’. (b) A unitary <=> 47! = A¥, (9) A (mx m) positive definite: (41/2)~! is a square root of A~! Note: Many of these results are standard rules which can be found in introductory textbooks such as Barnett (1990) and Horn & Johnson (1985) or follow inmmediately from definitions. 3.5.2 Inverses Involving Sums and Differences (1) A (mx m) with eigenvalues Ay,..., Am, [Ailabs <1, ¢ = sem:MATRIX VALUED FUNCTIONS OF A MATRIX 29 co. (a) Um + A)“ = SO(- Ay i=0 (b) Um - Ay} = “De (c) Ume +A @ APT = yay oA. i=0 (a) Um -A@ Ay = Yo Abo at (2) A(mx m), B(mxn),C (nx m),D (nxn): (A- BD“C)"! = A“ 4 A“ B(D ~ CA™!B)"'C A“! if all involved inverses exist. (3) A(mxn): det(Im+AA7) #0 = (Imt+AA")7! = Im—-AUIn +44 Ay A4. (4) A (mx m) nonsingular, B(mxm): (A+BB"), (Im+B% Aq! B) nonsingular => (A+ BB") B = A-'B(Im + BY A! BY}. (5) A,B (m x m) nonsingular: (a) AT} 4 Bol = A“ A+ BB” (b) Av? 4 Bo! nonsingular => (A714 B!)-1 = A(A + BY B= B(A+B)1A (c) A724 Bo! =(A4+B)) > ABOA= BAB (6) A.B (mx m): (a) Im + AB nonsingular => (Im + AB)"1A = A(Im + BA)7!. (b) A+B nonsingular > A—A(A+ B)"1A = B- B(A+ B)“'B. Note: Most results of this subsection may be found in Searl (1982, Chapter 5) or follow from results given there. For (1) see Section 5.4 3.5.3. Partitioned Inverses (1) A (mx m), B (mx n),C (nx m), D (nx n)30 HANDBOOK OF MATRICES AB 4 [ co |: A and (D — CA~'B) nonsingular A Bq" =[25] - Av} + A-1B(D — CA“!B)IC A“! A“ B(D — CA“! BY! —(D-CAn'By'CA7? (D-CA'By! (2) A (mx m), B (mx n).C (n x m), D (n x n) [ a 5 iF D and (A ~ BD~'C) nonsingular A BY] = le bp} = (A= BD"'C)"! —D"'C(A- BD“'C)7? (3) A (m x m) symmetric, B (m x n),C (m x p), D (n x n) symmetric. E (p x p) symmetric : -(A- BD-'C)"'BpD"! D7} + D-!C(A— BD-“!C)"1 BD"! ABC] BD Oo CoO £ F -FBD"! -FCE"! =| -D-'B'F Do! 4 D7 B'FBD-! — D-"BIFCE>? -EC'R — BOC'FBD")— B14 B-'C'FCE7! if all inverses exist and F = (A — BD~'B! ~CE~'C')", (4) Ay (mj x m,) nonsingular, i= 1,...,7 AL oy pap 0 0 Ar 0 Ay! (5) mn € IN,m> n,A(mxn),B(mx(m—n)): rk(A) = n, rk(B) = m—n, AXB =0 - AM Ay At = [A: a} = [ ee | (6) mn €IN,m [3] = [A4(AAB)!: BA(BBH)-}),MATRIX VALUED FUNCTIONS OF A MATRIX 31 Note: The inverses of the partitioned matrices in (1) - (3) are given in Magnus & Neudecker (1988, Chapter 1). The other results are straightforward consequences of the definition of the inverse. 3.5.4 Inverses Involving Commutation, Duplication and Elimination Matrices Reminder: © Kmn Or Km, denotes an (mn x mn) commutation matrix (for details, see Section 9.2). © Dm denotes the (m? x $m(m + 1)}) duplication matrix (see Section 9.5) © Lm denotes the (4m(m + 1) x m?) elimination matrix (see Section 9.6) (1) Kan = Kam (2) (Di, Dm)7! = DDH. 1 0 0 (3) (Ding Dri t= | 0 5m 0 . 0 0 (Diam? (4) (Lm big)" = Lm bby (5) A (mxm) nonsingular: [Di,(A@A)Dm)~! = Dh(A-! SA") DH. (6) A(mx m), Im © A+A® Im nonsingular (a) [D3(A ® Im)Dm)7! = 2D4Um @ A+ AG Im)“ Dm (b) (D4.Um ® A+ A® Im)Dm)7) = Dt Um © A+ AO In)! Dn (7) A,B (m x m),A@ B+ B@ A nonsingular: [D#(A @ B)Dm)"! = 2D4(A@ B+ BSA) 'Dp. (8) A (m x m) real symmetric nonsingular, c € IR : {D#[A@ A + vec(A)vec(A)']Dm}7? = Di, [teas - a1 a1 Taam Ve(Am vee A7)'| Dn (9) A, B(m x m) real, lower triangular, invertible : [Lm(A’ ® B)L},)7! = Lm((A')7! © BO YLi, (10) A (m x m) real, lower triangular, invertible, ¢ € IR {LmlA’® A +e vec(A)vec(A') JL, }-!32 HANDBOOK OF MATRICES ¢ = Ln [a eat vec(A~!)vec( A’! )!} Li, l+em Note: The results of this subsection may be found in Magnus (1988) 3.6 Generalized Inverses Definition: An (n x m) matrix Aq is a generalized inverse of the (m x n) matrix A if it satisfies AAT A = A 3.6.1 General Results (1) A(mxm): rk(A)=m ee AT = AT (2) A(mxn): (a) AX is not unique in general. (b) AAT and AWA are idempotent. (c) fm — AAW and I, — AW A are idempotent. (d) A(A A) AP is idempotent. (3) A(mxn): (a) rk(A) = rk(AW A) = rk(AAm) (b) rk(A) = tr( ATA) = tr( AAP). (c) tk(AA>) = tr(AA~) (d) rk(A7) > rk(A) (e) rk(A~) = rk(A) <= A> AAT = AT In (g) rk(A) =m <=> AAT = Ip. (4) A(mxn) (a) A(AP A) AP ASA (b) A? ACA" Ay 4M = AM (c) AC(AM A) AF is Hermitian. (d) (A) is a generalized inverse of AM. (f) rk(A) =n = - (5) Onxm is a generalized inverse of Omxn- (6) A(mxm): Ais idempotent > A is a generalized inverse of itself. (7) A (mx n).B (nx m): AM is a generalized inverse of A => A~ + B— A~ ABAA™ is a generalized inverse of AMATRI.: VALUED FUNCTIONS OF A MATRIX 33 (8) A (mx n), B,C (nx m) ; AW is a generalized inverse of A = Av +B(Im—AA™) + (In — AW A)C is a generalized inverse of A (9) A(mxn), B(mx m), C (nxn) B,C nonsingular + C-1A~ B~! is a generalized inverse of BAC (10) A(mxn), B(p xq): A> @ Bo isa generalized inverse of A © B (11) A(mxn), B(mxr), C(mxr): Generalized inverse matrices A> and C™ of A and C, respectively, exist such that AA7 BCC = B = the system of equations AXC = B can be solved for X and NX = A~ BC™ + — A> AYCC7™ is a solution for any (n x m) matrix Y Partitioned Matrices (12) A (m x m) nonsingular, B (m x n),C (r x m), D (rx n) AB CD AW Omxr 1 ig a generalized inverse of | |B = | Gh, Get ]is a generalized inverse of | 5 (13) A(mxn), rk(A) =r: B(mxm), C (nx n) are nonsingular and such that rk ]=m e=cata I, D) a»: “lea: = e| bP ] Bs 2 eoneratizea inverse of A, for any D (rx (n—r)), E ((m—r) xr), F ((m—r) x (n=). (14) A(mp xm), i= Lect Ay 0 A 0 . is a generalized inverse of 0 Ay 0 A Note: A number of books on generalized inverses exist which contain the foregoing results and more on generalized inverses (e.g., Rao & Mitra (1971), Pringle & Rayner (1971), Boullion & Odell (1971), Ben-Israel & Greville (1974). Some of these books contain also extensive lists of references34 HANDBOOK OF MATRICES 3.6.2 The Moore —Penrose Inverse Definition: The (n x m) matrix A+ is the Moore~Penrose (generalized) inverse of the (m x n) matrix A if it satisfies the following four conditions: (i) AAtA = A, (ii) A+ AAt = At, (iii) (AAt+)? = AAt, (iv) (At AF = 444 Properties (1) A(mxn): At exists and is unique (2) A(mxn): “ ps ] Vis the singular value decomposition of A > atv po oY (3) A (mx n), Uf (mx m),V (nn) UV unitary > (UAV)* = V4 atut (4) A(mx n), rk(A) =r: B(mxr), C (rx n) such that A= BC =» At =C+Bt eo! if ¢#0 0 if e=0 (6) A(mxn),c€C: (cA)t = ct At (7) A(mxn).B(rxs); (A@B)t = Ate Bt (8) A(m xm) (a) 4 is nonsingular = A* = AW). (5) cee: oe{ (b) Ais Hermitian = A?* is Hermitian. (c) A is Hermitian and idempotent = A+ = A. (9) A(mxn) (a) (At)+ = A. (b) (AN)* = (At)MATRIX VALUED FUNCTIONS OF A MATRIX 35, (c) AFAAt = AF, ({d) AtAAM = AM, (e) AM(At)4 At = At (f) At(At)4 AF = At, (g) (AM A)+ = At(At)A. (h) (AA#)* = (At)F at, (i) A(A7 A)+ AFA = AL (j) AAM (AAP HA = A. (k) At = (AM A)+ AM = AMAA) * (10) A(m xn) (a) rk(A) =m <=> AAt = In. (b) rk(A)=n <> AtA=Iy (c) rk(A) =n = At = (AMA) AH (d) rk(A) =m = At = AM(AAN)-! (e) rk(A) =n = (AAM)*+ = A(AM A)? 44 (f) rk(A) = 1 = At = [tr(AA*)]-1A# (8) A=Omxn <=> At = Onxm- (11) A(mxan): (a) rk(At) = rk(A) (b) rk(AA*) = rk(A* A) = rk(A) (c) tr(AA*) = rk(A). (12) A(m xn): (a) AA* and A+A are idempotent (b) Im — AAt and I, — A+A are idempotent. (13) A(mxn), B(nx nr): AB=Omxr > BtAt = Opxm- (14) A(mxn), B(mxr): APB =Ony, <> AtB=Onxr (15) A,B (mx n), ABY =0: (A+ B)* = At + (In — At B)[Ct + (In — C+C)M BY (At)# A(T — BC*)], where C = (Im — AA*)B and M = [In + (In — C*C)B4(At)F At BU, ~ CHC).36 HANDBOOK OF MATRICES (16) A(mxn), B(n xr), C(mx r): AM AB = AMC <=> AB = AAtC. (17) A(mxn), B(nx nr): det(BB4) ¢0 > AB(AB)+ = AAt (18) A (mx m) Hermitian idempotent, B (m x n) (a) AB = B= A ~ BB* is Hermitian idempotent with tk(A ~ BB+) = tk(A) — rk(B). (b) AB = 0 and rk(A) + rk(B) =m > A= 1, — BBY (19) 9 (m x m) Hermitian positive definite, A (m « n) AMO AAT O71 A)+ AH = AB, (20) A (mx m) Hermitian: A # 0 is eigenvalue of A with associated eigenvector r => AW! is eigenvalue of At with associated eigenvector ' Partitioned Matrices (21) Ay (ne x ny), (22) A(mxn): 0 o}*_fo 0 o aj ~[o at (23) A(mxn), B(m x p) At — A+B(Ct+ + D) : Bit = [A: B} =| Ct4+D where C = (Im ~ AA+)B and D = (lp -CtC)Ip + Up — CHC) BE (At) 4 At BU, — C*G)! xB (At) At, — BCT). (24) A(m xn), B(pxn) aqt R ] = (At -TBAt : TI, where T = E+ + (In — E+ B)A*(At+)4 BY K(1, — EE*) with E = B(Iy —A* A) and Wi = [Ip +(lp- BE+)BA*(A*)4 BY (1, ~EE*)]~!MATRIX VALUED FUNCTIONS OF A MATRIX 37 Note: These results on Moore-Penrose generalized inverses are also contained in books on generalized inverses such as Rao & Mitra (1971), Pringle & Rayner (1971), Boullion & Odell (1971), Ben-Israel & Greville (1974). A good collection of results is also contained in Magnus & Neudecker (1988), including many of those given here. Many of the results follow easily by verifying the defining properties of a Moore ~Penrose inverse. 3.7 Matrix Powers Definition: For i € Z, the ith power of the (m x m) matrix A, denoted by A’, is defined as follows: T[4 for positive integers i jal In fori =0 a («i ‘) for negative integers i, if det(.A) # 0. j=l If A can be written as A 0 A=U : ut 0 Am for some unitary matrix U (see Chapter 6) then the power of A is defined for any a € IR,a > 0, as follows Av =U . ue 0 aN This definition applies, for instance, for Hermitian matrices. The definitions are equivalent for integer values of a Properties (1) A(mxm),cECieN: (cA) actA (2) (Binomial formula) A,B(mxm)i€Ni>1: i (4+ By = S00 4% Bars... Ab Babe, jo38 HANDBOOK OF MATRICES where the second sum is taken over all k1,...,kj41 € {0,....4} with ky te +hjg1 = i-j (3) (Binomial formula) A,B(mxm): (A+B)? = A°+AB+ BA + B? (4) (Binomial formula for commuting matrices) A,B(mxm),i€ N21: AB = BA = (A+B)! =r (jaan. ‘ J Fad (5) A,B(mxm)ieIN,i>1 So ai(a- ByBts, j=0 (6) A(m xm), B(n x n),i€ IN (a) (A@ By = Ae Bi [3 5] -[0 eee (b) (ABBY =| 5 3 o BF (7) A(mxm),i€ IN: (a) |Aélavs $ |Alins- (b) (A'y = (A (c) (AH = (AMY. (4) a (e) (ANT) = (AY (f) rk(A*) < rk(A). (g) det(.A') = (det A)* (8) A(mxm),i€ INeven: —tr(Ai) = vec( Ai/?")/vee( Ai/?), (9) Im (mx m) identity matrix,i€ IN: Ti, = Im (10) FEIN #0: Obgm = Omum-MATRIX VALUED FUNCTIONS OF A MATRIX 39 (11) ie: AGATE (gh ATE Aol 0 : 0]. 0 YW (fy ginme2 or 1 0 (ria) 0 0 Aol yi 00 d o oo () (7 x2) 0 0 8. x (12) A (mx m): (a) Ais idempotent > A= A fori =1,2,.. (b) Ais nilpotent <=> A’ = Omxm for some i > 0. (c) Ais symmetric => A? is symmetric for i= 1,2,... (d) Ais Hermitian = Af is Hermitian for i= 1,2, (13) A(mxm),i€ IN vec(A?) = (Im & A®)vec( Im) = ((A')'@ Im)vec(Im) = ((A'?) @ A'?)vec(Im), if A is positive definite ((A'?Y @ AM? )vec(Im), if i is even = (A @Ajvec(A*), ifi > 2 (14) A(mx m) Al —;.00 0 <=> all eigenvalues of A have modulus less than 1. Note: Most results of this section are basic and follow immediately from definitions. The binomial formulae may be found in Johansen (1995, Section A.2) and (13) is a consequence of an important relation between Kronecker products and the vec operator (see Section 2.4). 3.8 The Absolute Value Definition: Given an (m xn) matrix / is ij) its absolute value or modulus larilabs |@r2labs ~~ (@inlabs lazilats [aaalees ~~ leznlabs |Alobs = [lasjhvs } = (m xn) lamifabs [@melaos -** l@mnfabs40 HANDBOOK OF MATRICES where the modulus of a complex number ¢ = ¢1 + itz is defined as |clavs = Ve} +c} = Vee. Here ¢ is the complex conjugate of ¢ (1) A(m xn) (a) {Alabs > Omxn (b) |Alabs = Omxn => A= Omxn (2) A(mxn),c€ ©: eAlans = Iclabs |Alabs (3) 4, B(mxn): [A+ Blavs < [Alabs + |Blabs (4) A(mxn),B(nxp): |ABlavs < [Alabs | Blats- (5) A(mxm)ieIN: |A¥lats <|Albbs- (6) A(mxn),B(pxq): |A® Blars = |Alabs ® |Blavs- (7) A(mxm),B(nxn): |A® Blars = [Alovs @ [Blavs (8) A(m xn): (a) |A‘labs = |Alabs- (b) |Alabs = [Alabs. (c) [A* las = |Aloes- (9) A (m xn), B (mx p),C (ax 2), D (qx p) [eo] Note: These results may be found in Horn & Johnson (1985, Chapter 8) or follow easily from the definition of the absolute value. _[ lAlate [Blane ate | IClavs [Plats4 Trace, Determinant and Rank of a Matrix 4.1 The Trace Definition: The trace of an (m x m) matrix A = [a,)] is defined as trA = tr(A) Say, +++ tamm = D> aii = 4.1.1 General Results (1) A,B (mx m): tr(A + B) = tr(A) + t(B). (2) A(mxm),c€C: tr(eA) =e tr(A). (3) A,B(mxm), c1.c2€C: tr(e:A + epB) = cyte( A) + eotr( B). (4) A(mx m) (a) tr(A’) = tr(A). (b) tr(A) = tr(A). (c) tr(A”) = tr(A). (5) A(mxn) (a) tr(AAt) = rk(A). (b) tr(A4 A) =0 <> A=Omxn (6) A(mxm): A idempotent = tr(A) = rk(A). (7) A(m x m) with eigenvalues A,,...,Am) tr(A) = Ay toot Am (8) A(mxn), B(n xm): (a) tr(AB) = tr(BA) (b) tr(AB) = vec( A’) vee(B) = vec( B'Y’vee( A)a HANDBOOK OF MATRICES (9) A(m xn), B (n x p),C (p x q), D (qx m) tr(ABCD) W vee( D')'(C’ @ A)vec( B) = vee(A')'(D! @ B)vee(C) vee( B')'(A’ @ C)vec( D) = vec(C’)'(B’ @ D)vec(A) (10) A,B(mxm): Bnonsingular => tr(BAB-') = tr(A). (11) A,B (m xn), je = (1. 1) (Rx 1): tr AB) = ff, (A® B)in (12) A,B, D(mxm), j= (1.02.1) (mx 1): Ddiagonal > tr(ADB'D) = j'D(A® B)Dj (13) A,B.C(mxn): tr[A(BOC)) = tf(A'o BC] (14) A(mxm): tr(A& Im) = tr(A) (15) A(mx_m),B (nxn): (a) tr(A @ B) = tr(A)tr(B). (b) tr(A @ B) = tr( A) + tr(B). (16) A (mx m). B (mx n),C (nx m),D (nxn): «| é 5 | = tr(A) + tr(D) (17) Kinm (m? x m?) commutation matrix: — tr(Aimm) (18) Dy (1? x 4m(m x 1)) duplication matrix (a) tr(Di, Din) = t(Drs Diy) = m? (b) tr(D},, Dm)7! = m(m + 3)/4. (19) Lm (4m(m + 1) x m*) elimination matrix tr(Lm Li.) = tr( Li, Lm) = $m(m + 1) (20) A = [a,,], B (m x m) real positive semidefinite, a € R,a > 0,0 #1 m tr(A®) = So af <=> Ais diagonal i=l (21) A, B #0 (1m x m) real positive semidefinite, a € R,Q B=cA for somec€ R,c > 0.TRACE, DETERMINANT AND RANK OF A MATRIX 43 (22) A,B (m x m) real positive semidefinite, @ € IR,a > 1: [tr(A + BY}? = (tr AM) + (tr BE)? <> B=cA for somec€ R,c > 0. Note: The rules involving the vec operator, the commutation, duplication and elimination matrices are given in Magnus & Neudecker (1988) and Magnus (1988). (20) ~ (22) are given in Magnus & Nendecker (1988, Chapter 11). The other rules follow from basic principles. 4.1.2 Inequalities Involving the Trace In this subsection all matrices are real unless otherwise stated. (1) A(m x n) complex: (a) [tr Alabe < tr Alans. (b) tr(A# A) = tr(AA¥) > 0. (2) A,B (mxn): (a) (Cauchy-Schwarz inequality) tr(A'B)? < tr(A’A)tr(B’B). (b) tr(A’B)? < tr(A’AB’B). (c) tr(A’B)? < tr(AA’BB’). (3) (Schur’s inequality) A(mxm): tr(A®) < tr(A’A). (4) A(m x m) positive semidefinite: (det A)'/™ < Ltr(A) (5) A(mxm): All eigenvalues of A are real => |Ltr(A)lavs < (2.tr(A?)]!/?, (6) A (m x m) symmetric with eigenvalues Ay < --- < Am, X (mx n) with XYX = In: SOA S t(XAX) SY menti iI ist (7) A= [aij] (m x m) positive semidefinite: >VR az for a>l ucasyd 2 te fi fo 1,8=a/(a—1) > tr(AB) < (tr A2)!/9(tr BE) (c) (Minkowski’s inequality) a> = (tr(A+ B)9]/* < (tr At)? + (tr BI (9) A, B (m x m) positive semidefinite: (a) aE RO (det A)'/™(det B)/™ (10) A (m x m) positive definite: Indet(A) < tr(A) — m (11) A (m x m) positive definite, B (n x m),C (m x n) BC = In = tr(C’AC) > tr( BAW By"! (12) A (m x m) positive semidefinite with maximum eigenvalue Ayer (A). B(nxm): tr( BAB’) < Amar(A)tr( BB’) (13) A (mx m) positive semidefinite with maximum eigenvalue Amar(A) tr(A) < Amaz(A)tr( A). (14) A,B(mxm): tr[(A+ BA + BY] < 2[tr(Aa’) + tr(BB’)). (15) A (n x m) positive definite, B (1m x m) positive semidefinite exp[tr(A7! B)] > ae (16) A, B (mm x m) positive semidefinite: (a) tr(AB) < tr(A & B). (b) tr(AB) < H(trA tor By. (c) tr(A© B) < F(trA+tr By? (d) trade B) > 0 (e) tr(As. B) < tr(A@ B) (17) A(m x m). B (nx n) positive semidefinite: tr(d © B) > 0. (18) A,B,C, D (m x m) positive semidefinite: C'- A, D— B positive semidefinite => tr(AB) < tr(CD). (19) A, B(m x m) symmetric: tr(AB) < Str(A? + B?). (20) A,B (mx m): tr(A® B) < $tr(A@A+ BOB),TRACE, DETERMINANT AND RANK OF A MATRIX 45 tr(A® B) < btr(A@A+ Bo B) Note: Inequalities (16) ~ (20) are from Neudecker & Shuangzhe (1993) (see also Neudecker & Shuangzhe (1995)). The other inequalities ean be found in Chapter 11 of Magnus & Neudecker (1988) 4.1.3 Optimization of Functions Involving the Trace In this subsection again all matrices are real if not otherwise stated. (1) @ (m x m) real symmetric with eigenvalues Ay < -:) < Am and associated orthonormal (m x 1) eigenvectors v,...,tm, 2 € {l...,m}: min{tr(B'QB) : Bm xn) real, B'B = In} = dv +--+ An. The minimizing matrix is B = (v1,..., Ya]. max{tr(B’QB) : B(m xn) real, B'B = In} = Am +++ + Amongst The maximizing matrix is B= [tm.....tm—ngt] (2) & (m x m) complex Hermitian with eigenvalues 4, < ++) < Am and associated orthonormal (m x 1) eigenvectors ti,....tm. ” € {1.....m}: min{tr(B’QB) : B(m xn) complex, BY B = In} = Ay ++: +An The minimizing matrix is B = max{tr(B¥QB) : B(m xn) complex, BY B= In} = Am toot Amongt: The maximizing matrix is B = [vm,...,Um—n4i) (3) 2 (m x m) positive definite with eigenvalues Ay < +--+» < Am and associated orthonocmal eigenvectors vj,...,0m.0 Apuyeh. ismor4l46 HANDBOOK OF MATRICES (4) X (m xn), Ay < ++ < An eigenvalues of X/X with associated orthonormal eigenvectors 0),...,0n, 0S r2: det(A) = ay, cof(ajy) +--+ + aim cof(aim) = ay; cof(aiyj) +--+ amj cof(am;) for any i, j € {1,...,m}. Here cof(a,;) denotes the cofacor of aj; (see Section 1.4). (3) A(mxm),cEC: det(eA) = e™det(A). (4) A,B (mx m):48 HANDBOOK OF MATRICES (a) det(AB) = det(A)det(B). (b) Bis nonsingular + det(BAB-!) = det( A) (5) Im (mm x m) identity matrix: det(Im) = L (6) A(m x m) with eigenvalues Ay... Am mE} det(A) = Ai-- Am = T] vs. 1 (7) AQ xm) (a) det(A’) = det(A). (b) det( A) = det(A). (c) det(A) = det(A). (d) det(A!) = (det A)-!. if A is nonsingular. (e) det(A%%) = (det Ayr! (f) det(A)In = AM A= AA (8) A (m xm), B (nxn) (a) det(A@ B) = (det A)"(det BY” (b) det(A fF B) = det( A)det(B). A (9) (ai) (7m x m) (a) A= diagtaii...sdyum) => det(A) = air -@nm = [TT] ai. (b) Ais triangular => det(A) = @11--+dmm = Tle ist (10) A (mx n) real: det(m + Ad’) = det(Iy + 4/4) (11) A (on xm) (a) rk(A) < m <=> det(A) = 0 (b) rk(A) =m <> det(A) ¢ 0. (c) Ais singular <=> det(.4) =0. (d) ‘The rows of A are linearly independent <=> det(A) £ 0. (e) The columns of A are linearly independent <=> det(.A) ¢ 0. (f) A has a row or column of zeros => det(A) = 0. (g) A has two identical rows or columns = det(A) = 0. (12) A.B (nx m)TRACE, DETERMINANT AND RANK OF A MATRIX 49 (a) B is obtained from A by adding to one row (column) a scalar multiple of another row (column) => det(A) = det(B) (b) B is obtained from A by interchanging two rows or columns => det(B) = —det(A). (13) (Binet Cauchy formula) A,B(mxn)m B=0. {ai)] (m x m) positive definite: det(A) = TT au > A = diag(ay...-,amm) ist (16) (Vandermonde determinant) Aye Am EC 1 Pow. 1 AL Az - Am det | = TJ -4) apot apot . Am-t 1 (17) po. maT EC: 0 4 0 0 det} 9 9 4 0 | =(-1)"Po 0 0 - 0 1 Po =PLo —Pm=2 Pai Note: Most results of this subsection are straightforward implications of the definitions and elementary matrix rules. They can be found in textbooks such as Lancaster & Tismenetsky (1985) and Barnett (1990). 4.2.2 Determinants of Partitioned Matrices (1) A (mx n), B (mx p),C = [A: B] (m x (n+ p)) (a) det(C#C) = det(A” A)det(B# (1, — AAt)B] = det(BY B)det[A4 (1, — BB*) A).50 HANDBOOK OF MATRICES (b) rk(A) =n => det(C!C) = det(A" Ajdet[B" B — BM A(AM A)-1 AM B) (c) rk(B) = p => det(C!C) = det(BY B)det[A” A — AY B(BY B)-1 BY A). (2) det [ pte | =(-1)" (3) A(m x m), B (mx n),C (n xn) | Ons é ] = det(A)det(C). (4) A(m xm), B (nx m),C (nxn): ae 4 eon | = det(A)det(C) (5) A,B,C (m xm) A m. . act a Om | = (-1)™det(B)det(C) (6) A(x m), B (mx n),C (nx m), D(n xn): 5 | = det(A)det(D — CA7'B), Anonsingular => act | Dnonsingular > ae | B CD ] = det(D)det(A — BD“) (7) A(m xm), a,b (mx 1) a A Anonsingular => det gi 0 | = —b4 42h, (8) A(x m), a,6(mx1),c€C: A nonsingular => det [ i ‘ ] = edet(A) — 6% A%% a = det(A)(c — 64 A“! a) Note: Some of these results on partitioned matrices are given in Magnus (1988). Most results are straightforward implications of the definitions and elementary matrix rules, however.TRACE, DETERMINANT AND RANK OF A MATRIX 51 4.2.3. Determinants Involving Duplication Matrices Reminder: D,, denotes the (m? x 4m(m-+1)) duplication matrix (see Section 9.5 for more details). (1) det(Di, Din) = 2" — 19/2, (2) det(Dm Di,) = 0. (3) A(mxm): (a) det(Di,(A @ A)Dm) = 27™-"/?det (Ayr! (b) det(D4£ (A @ A)DE') = 2-™™-1/2 det( Aymt!, (c) det( Dt (A ® A)Dm) = (det Ay™*!, (4) A,B (mx m): (a) det(B) = 0 => det(D£(A @ B)Dm) = 0. (b) det(B) #0 and Ay,...,Am are the eigenvalues of AB-! => det{Dh(A@ B)Dm] = 27-1)! 2det(A)(det BY” T] (Ar +A;)- Di (c) det(A) #0 and A1,...,Am are the eigenvalues of BAT! = det[Di(A @ A+ B® B)Dy) = (det Ay"*" T] (1 £ asd). 25 (5) A (m x m) with eigenvalues \1,--.,Am det[D(A & Im)Dm] = 2-7" "— det A) TT Os +5), iy det[Dh(Im @ A+ A® Im)Dm] = 2det(A)T] (Ai +s). Di (6) A= [as;), B= [6] (m x m) lower triangular: det[DE(A @ A+ B © B)Dma] = T](aieas; + 644653). a2 (7) A (mx m) real, symmetric, nonsingular, ¢ € IR: det(Di[A @ A+ cvee(A)vec(A)’]Dm) = (1+ em)det(Ay™+ (8) A (mx m) with eigenvalues A1,...,Am, 1 INji> 1: int det (>: Vue +00) = im(det A)" TT aes j=0 >tHANDBOOK OF MATRICES where { (AL A/Ae— an) if Ae HM Ber = at if X= Note: The rules presented in this subsection may be found in Magnus (1988) 4.2.4 Determinants Involving Elimination Matrices Reminder: L,, denotes the ({m(m+ 1) xm?) elimination matrix (see Section 9.6 for more details). (1) det(Lin Lin) = b (2) det( Li, Lm) = 0. (3) A= [ays]. B= [6;j] (m x m) upper (lower) triangular: det(Lm(A& B)LI.) = TT ban ist vy), B= [bij] Gm x m) lower triangular: det( L(A’ B)LI,) = I batt! (5) A= [ay]. B = [bi] (rm x m) “ det(Lm(A® BL) = TT ett anndetc at), where - ay ay bir bin Aun = . » B= My. aii bmi +. bmm (6) A (mx m) real. lower triangular, nousingular, ¢ € IR det(Li,[A © A+ ¢ vee(A)vec( AJL) = (1 + em)det(4y"*! (7) A. B(m x m) lower triangular: det (Lm( AB! © B'A)L,) det( Ln (AB’ © A'BYLI,) det(Lm (AB © BYA)Dy,) = det(Lm(AB’ © A'B')Dm) = (det Ay" *(det By" *!TRACE, DETERMINANT AND RANK OF A MATRIX 53 (8) A= [aj], B= [bij].C = [cy]. D = [diy] (m « m) lower triangular det(Lm(AB! @ C'D)Li,) = TT (eid) (aunty, det[Lm(A!@ B+C' ® D)Lin} = T] (bisa; + discs). iy (9) A= [a,j] (m x m) lower triangular, n € IN a=1 det (:. (Sur @ *) tn) = nm(det Aye! TT par. 30 RST where nn (2%, —@f)/(aee — au), if axe # an Met = | nazz, if axe = an Note: The rules presented in this subsection may be found in Magnus (1988). 4.2.5 Determinants Involving Both Duplication and Elimination Matrices (1) A= [aij], B = [bi;] (m x m) lower triangular: det(Lm(A@ B)Dm) = det(Lm(A’@ B')Dm) det(Lm(A® B')Dm) = [ban (2) A, B(m x m) 0 if det(B) = 0 _ m1 det(Em(ABBIDm) =) gee ay(det By" TT det(Ciay) if det(B) £0 ct where en Cin Cm = t Cnr Can denotes the nth principal submatrix of ABT!54 HANDBOOK OF MATRICES. cj} (m x m) lower triangular = (3) A= (aj), B= [bi], C det(Lm(A © BYC)Dya] = T] (bises)' a", int det[Lm(A BY @ C)Dm) = T] efi (aisbis "+ it (4) 4,B(m xm): det(Lm(A @ B)D#,’) 0 if det(B) = 0 _ mot FY armim—1i/2get(Ay(det BY" [TT] det(Ciny) if det( 8) #0 ni where cn. Cin Cm = Cnt Con are the principal submatrices of C= [aj] = AB™?. Note: The rules presented in this subsection are given in Magnus (1988) 4.2.6 Inequalities Related to Determinants (1) (Cauchy-Schwarz inequality) A,B(mxn): (det AY B)? < det( AM A)det( BY B). (2) (Hadamard’s inequality) ajj)(mx m): (det A)? < Ul (& st vot \jar (3) (Hadamard’s inequality) a;;] (m x m) positive semidefinite: det(A) < TT ais. (4) A= (aij](m xm): |det(A)labs < {max |asjlaps}"™m™/?. 1 (5) (Fischer's inequality) A(m x m),B(m x n),C (nxn): D= [ Ay e | positive definite + det(D) < det(A)det(C’)TRACE, DETERMINANT AND RANK OF A MATRIX 55 (6) A= [aij] (m x m) positive definite: det det (8) A = [ass] (m x m) positive definite with eigenvalues Ay <--> < Am My we OE ID: < det ia Oy)... Oke (9) A (mx m) positive definite: Indet(A) < tr(A)—m (10) A(m x m) positive semidefinite: (det A)!’ < Ltr(A). (11) A (m x m) positive definite, B (m x m) positive semidefinite: det(A + B) > det(A) (12) A, B (m x m) positive semidefinite det(A + B) > det(A) + det(B) (13) A (m x m) positive definite, B (m x m) negative semidefinite: det(A + B) < det(A) (14) (Minkowski’s inequality) A #0, B £0(m x m) positive semidefinite [det(A + B)]'/™ > (det A)/™ + (det BY (15) A, B(m x m) positive semidefinite, a € IR, 0 det(A) TJ bi. ist (19) A,B (m x m) positive definite: det(A @ B) > det( A)det( B) (20) (Ostrowski Taussky inequality) A(mx m) 3(A + A”) positive definite = det3(A +A") < |det(A)laus (21) A,B (mx m) real: det(AB) < 3[(det A)? + (det B)?] (22) A.B (mx m) real: det(A@ B) < $[det(A@ A) + det(B 2 B)). Note: Inequalities (21) and (22) are front Neudecker & Shuangzhe (1993. 1995). The other inequalities may be found in Horn & Johnson (1985) and Magnus & Neudecker (1988) 4.2.7 Optimization of Functions Involving a Determinant In this subsection all matrices are real if not stated otherwise. (1) 2 (x m) positive definite with eigenvalues Ay < Ay < 10+ < Aw and associated orthonormal (rm x 1) eigenvectors r.. : min{det(B’QB) : B(mx n), B'B = In} =Ap- dae An The minimizing niatrix is B = [v1,...,¥n] max{det(B'QB) : B(m xn), B'B = In} = Am: Am-1---Am-ntt The maxiniizing matrix is B= [Um,-.-,0m—n4i] (2) Q (m x m) complex Hermitian positive definite with cigenvalues dy S Ap S++ S Aw and associated orthonormal (m x 1) eigenvectors Prem min{det(B QB) : B (mx n) complex, BY B= In) = 1 -d2--- An The minimizing matrix is B= [r4,..., eal. max{det(B4QB) : B (m x n) complex. B¥B = I,} = Am Amat Amant ‘The maximizing matrix is B= [lmn..-..tm—nai)eTRACE, DETERMINANT AND RANK OF A MATRIX 57 (3) ¥ (mx n),X (px n),rk(X) =p: min{det[(Y — AX)(Y¥ — AX)'] : A(n x p)) = det[¥ (Un — XXX) OXY]. The minimum is attained for A = YX(XX")7?! (4) Y (mx n), X (p x n) with rk(X) = p, R(q x m) with rk(R) = q, C(qxp) min{det[(¥ — AX)(¥ — AXY'] : A (mx p),RA=C} = det{(Y — AX)(Y — AX)’]. where A = YX"(XX')7! + R(RR) NC — RY XX N')7?). (5) YX (mxn), rk(¥) = rk(X) =m, Ar < ++ < Am the eigenvalues of (XX VEN Y (YY) Y X(N)! with associated orthonormal eigenvectors v),..., 0m: min{det{((¥ — BCX)(Y — BCX)']: B(mxr),C (rx m),rk(B) = rk(C) =r} = det(¥YY’)(1 — Am) <=" = Amare) The minimum is attained for C= [Une Umar gal (XNYT and BaYX'C(CNX'CY! (6) B,C (mx m) positive definite, B diagonal, 1 << - < Dm the eigenvalues of B-!/2C;B-'/? with associated orthonormal eigenvectors v)...., Um, Lorem: min{In det(AA’ + B) + tr[(A4’ + B)-'C] : A(mxr)} = m+ Indet(C) + D> (Ai - Ind; - 1). iI The minimum is attained for Yim 0 ad A= Blum... tmorgil! . oh 0 Amar) Note: The results of this subsection may be obtained from Magnus & Neudecker (1988) and Liitkepohl (1991, Section A.14).58 HANDBOOK OF MATRICES 4.3 The Rank of a Matrix Definitions: The rank of a matrix A, denoted by rk A or rk(A), is the niaxinum number of linearly independent rows or columns of A. The column tank of a ntatrix A, denoted by col rk A of col rk(A), is the number of linearly independent columns of «1. The row rank of a matrix A, denoted by row rk A or row rk(A), is the number of linearly independent rows of A 4.3.1 General Results (1) A(m xn): row rk(.A) = col rk(A) = rk( A) (2) A(mxn).c€@: ¢#£0 = rk(ed) = rk(A) (3) A(m xn): (a) rk(.4") = rk( A). (b) rk(A) = rk(.4). fe) rk( AM) = rk(Ay (d) rk(At) = rk(A) (e) rk(AT) = rk(A) <=> AT AAT (4) A(m xm): mit ria) =4 1 if 0 if | rk(A) rk(ab’)=1 (7) AQ xn): rk(A) =r => there exist matrices B (mm x 1) and C(r xn) such that += BC. (8) A(mxn).B(n xn): Bnonsingular => rk(AB) = rk(.A). (9) A(mxn).B(mx im): B nonsingular = rk( BA) = rk( A) (10) A(m xn), B (mx m),C (nxn) B,C nonsingular. = rk(BAC) = rk(A) (11) AQn xn), B(pxq): rk(A@ B) = rk(A)rk(B). (12) A(mxm),B (nxn): rk(A& B) = rk(A) + rk(B).TRACE, DETERMINANT AND RANK OF A MATRIX 59 (13) A(mxn),B(rxm): rk(B) = 7, rk(A) =m = rk(BA) =r, (14) A(mxn) (a) tk(A) =m > At = AN(AAM) (b) tk(A) =n => At = (AMA) AM, (15) A (mx m) (a) tk(A) At = 0. (b) rk(A) = m <=> det(A) #0 (c) rk(A) = m <=> A is nonsingular. (16) A(mxn): rk(A)=0 => A=0. (17) A (m x m) idempotent: (a) tk(A)=m > A= In. (b) rk(A) = tr(A). (c) tk(Im — A) =m —rk(A) (18) rk(Im) =m (19) Kina (mn x mn) commutation matrix: — tk( Kin) = mn (20) Dm (m? x }m(m-+1)) duplication matrix: —rk( Dm (21) Ly (4m(m + 1) x m?) elimination matrix: rk(Lm) (22) A(mxm): (a) tk(Alm — A) < m <=> 2 is an eigenvalue of A {b) 0 is a simple eigenvalue of A > rk(A) = m-—1. (23) A(mxn): rk(A) =r => tk(Kinn(A’@ A)) = (24) A (mx m), Ajj (n x n) submatrix of A : rk(A) = 7, n>r > det(Ais) =0. (25) A (mx m): (a) rk(A) = r = there exist (m x m) matrices Bi with rk( Bj) = land A= By +-+- + B,. (b) rk(4‘) = rk(A+) for some i € IN = rk( A!) = rk(A?) for all ji (c) There exists an i € IN such that rk(A") = rk(A't?) (26) A (mx m) nonsingular, B (m x n),C (n x m),D (nxn): }m(m +1) Lin(m +1) 2 «[é Bla D=CA"R60 HANDBOOK OF MATRICES Note: Many of these results are elementary and follow immediately from definitions (see, e.g., textbooks such as Horn & Johnson (1985) and Lancaster & Tismenetsky (1985)). See also Magnus (1988) for some of the results. 4.3.2 Matrix Decompositions Related to the Rank (1) (Singular value decomposition) A (in xn), rk(A) =r, o1,...,0, #0 singular values of A: There exist unitary matrices U (m x m), V (nx n) such that sep [P Olyn aul ole where D = diag(o1....,¢,) and some or all of the zero submatrices disappear when r = m and/or r =n. (2) A(m xm) symmetric: rk(A) = 7 <> there exists a nonsingular (m x m) matrix 7 such that alte 0] a, set 0]? where the zero submatrices disappear when r = m. (3) (Diagonal reduction) A(mxn): rk(A) =r <=> there exists a nonsingular (m x 1m) matrix S and a nonsingular (n x n) matrix T such that —ol bb 0], ses[h 8] where some or all of the zero submatrices disappear when r = m and/or r =n. (4) A(mxim): rk(A) = rk(A2) <=> there exist nonsingular matrices P and D such that p{P 9] 5-1 4 [ 0 0 ] (5) (Rank factorization) A(mxn): tk(A) =r ¢=> there exists an (m xr) matrix B and an (r xn) matrix C with rk(B) = rk(C) = r such that 4 = BC (6) (Triangular factorization) A= [ay] (mx m).rk(A) =r det | : . #0, iTRACE, DETERMINANT AND RANK OF A MATRIX 61 => there exists a lower triangular (m x m) matrix L and an upper triangular (m x m) matrix U one of which is nonsingular such that A=LU. (7) (Polar decomposition) A(mxm): rk(A) =r <=> there exists a positive semidefinite (m x m) matrix P with tk(P) = r and a unitary (m x m) matrix U such that A= PU. Note: For these and further decomposition theorems see Chapter 6 4.3.3 Inequalities Related to the Rank (1) A(mxn): (a) tk(A) < min(m, n). (b) rk(A-) > rk(A). (2) A(mx n), B (nx r) (a) tk(AB) < min{rk(A), rk(B)}. (b) rk(A) +rk(B) < rk(AB) +n (c) AB=0 = rk(A)+rk(B) [tr(A)]?/tr( 4?) (9) (Frobenius inequality) A(mxn),B(nx1),€ (rx s): rk(AB) + rk(BC) < rk(B) + rk(ABC). Note: These results can be found in many matrix textbooks (see, e.g., Horn & Johnson (1985) or Rao (1973)) or follow easily from rules given there5 Eigenvalues and Singular Values In this chapter all matrices are complex unless otherwise stated. 5.1 Definitions For an (mx) matrix A the polynomial in 4, pa(A) = det(Alm — A), is called the characteristic polynomial of A. The roots of p4() are said to be the eigenvalues, characteristic values, characteristic roots or latent roots of A. Let A1,...,An be the distinct eigenvalues of the (m x m) matrix A. Then the characteristic polynomial can be represented as Pal) = po + Pid +--+ Pm=1 A"! + AM = (AW ALM (A= An), where the mj are positive integers with 2", m, =m. ‘The number m, is the multiplicity or algebraic multiplicity of the eigenvalue ;, § = 1,....7. An eigenvalue is called simple, if its multiplicity is 1. The geometric multiplicity of an eigenvalue A of an (m xm) matrix A is the number of blocks A 1 0 . 0 oA” 0 0 0 : sol 60... a with A on the principal diagonal, in the Jordan decomposition of A which is given in the next section (see also Section 6.1). For an (m x m) matrix A. p(A) = max{|Alabs A is an eigenvalue of A}64 HANDBOOK OF MATRICES: is the spectral radius of A. The set {1 : ) is eigenvalue of A} is the spectrum of A For an (1m x m) matrix A with eigenvalue A any (mn x 1) vector » # 0 satisfying Av = Av is said to be an eigenvector or characteristic vector of 4 corresponding to or associated with the eigenvalue A. For two (1 x m) matrices A and B a root of the polynomial pa.p(A) = det(AB — A) is sometimes called eigenvalue or characteristic value of A in the metric of B. Accordingly, for an eigenvalue \ of A in the metric of B an (mm x 1) vector v # 0 satisfying Av = Bu, is called an eigenvector or characteristic vector of A in the metric of B corresponding to or associated with A. For an (2 x n) matrix 4 the nonnegative square roots of the eigenvalues of A" A, ifm > nv, and of AA", if m < n, are said Lo be the singular values of A. 5.2 Properties of Eigenvalues and Eigenvectors Notation: (A) denotes an eigenvalue of the matrix A. Ammn(A) is the smallest eigenvalue of the matrix: A if all eigenvalues are real. Amar) is the largest eigenvalue of the matrix A if all eigenvalues are real 5.2.1 General Results (1) A(mxm): A has exactly m eigenvalues if the multiplicities of the roots are taken into account. (2) A.B(mxm): A,B have the same eigg <=> tr(A*) = (BF), k= 1... ‘alues (3) A (m x m) with eigenvalue X and associated eigenvector r : (a) dis eigenvalue of A with eigenvector t. (b) dé is cigenvalue of A’ with eigenvector v for i € IN. (c) A“! is eigenvalue of A~! with eigenvector &, if 4 is nonsingular. (4) A(m xm): (a) is eigenvalue of A <=> 2 is eigenvalue of 4’ (b) A is eigenvalue of A <=> 2 is eigenvalue of AY (c) Ais eigenvalue of A => +7 is eigenvalue of A+ Thm.EIGENVALUES AND SINGULAR VALUES 65 (5) A= [aij] (mx m): (a) A= diag(ait....,@mm) => a1iy.--1dmm are the eigenvalues of A. (b) Ais triangular > ai1,...,amm ate the eigenvalues of A (c) Ais Hermitian = all eigenvalues of A are real numbers (d) Ais real symmetric => all eigenvalues of A are real numbers (e) Ais idempotent = all eigenvalues of A are 0 or 1. (f) Ais nilpotent = all eigenvalues of A are 0. (g) Ais real orthogonal => all eigenvalues of A are 1 or —1 (h) Ais singular <=> 0 is an eigenvalue of A. (i) A is nonsingular <=> all eigenvalues of 4 are nonzero (6) A (m x m) Hermitian: (a) A is positive definite <=} all eigenvalues of A are real and greater than 0. (b) A is positive semidefinite <=> all eigenvalues of A are teal and greater than or equal to 0. (7) A(m xm) real symmetric: (a) A is positive definite <=) all eigenvalues of A are real and greater than 0 (b) A is positive semidefinite <=> all eigenvalues of A are real and greater than or equal to 0. (8) A(m xm), B (m x m) nonsingular: dis eigenvalue of A <=> A is eigenvalue of B-?AB (9) A(mxm), B(m x m) unitary: d is eigenvalue of A <> A is eigenvalue of BY AB (10) A (m x m), B (m x m) orthogonal: X is eigenvalue of A <=> ) is eigenvalue of B’AB (11) A (mx m), B(m x m) positive definite: d is eigenvalue of BA <> 1 is eigenvalue of B'/? ABM? (12) A(nxn), B(n xm), 2>m (a) 1 is eigenvalue of AB = J is eigenvalue of BA. (b) 4 #0 is eigenvalue of BA = 2 is eigenvalue of AB. (c) Arcos Am are the eigenvalues of AB = Mt... Am O0.60 are the eigenvalues of BA66 HANDBOOK OF MATRIC (13) A(mx m), ce, ¢ #0: (a) A is eigenvalue of A with eigenvector v => cA is eigenvalue of cA with eigenvector v. (b) v is eigenvector of A corresponding to eigenvalue A => cr is eigenvector of A corresponding to eigenvalue 2. (14) A (mx m),ci,e2 € Cyer or cg # 0 v1, 02 eigenvectors of A corresponding to an eigenvalue A => ¢1v) + cove is eigenvector of A corresponding to eigenvalue (15) 4 (xm) with eigenvalues 4;, A, and associated eigenvectors 0.0, £ 0: A: £A) > vj and vy are linearly independent (16) A (mx m).qo.71.-..-9p EC: Ais eigenvalue of A with eigenvector v= WF NAF- + pA? is eigenvalue of Yom +914 +20°+ AP with eigenvector v. (17) A (mx m), B (n x n), A(A),A(B) eigenvalues of A and B, respectively, with associated eigenvectors v(A) and (8). Tespectively: (a) A(.A)-A(B) is eigenvalue of (b) ACA) and A(B) are eigenvalues of 42 B with eigenvectors [ ey) ] and [ 4B) respectively (18) A (m x m) with eigenvalues \y,.... Am = TI» st (b) tr(A) = Arto + Am = Od a B with eigenvector v(A)@r(B) (a) det(A) = Ay -- An (19) A (m x m) (a) A has r nonzero eigenvalues => rk(A) > r (b) rk(Almn — A) < m—1 <=> Xis eigenvalue of A (c) 0 is a simple eigenvalue of A = rk(A) = m—1 (20) A (m x m) with eigenvalues A1,..., Am (a) |Ailabs <1, i= 1,...,m <=> det(Im — Az) #0 for |zlabs < 1, 7 EC. () [Aslane < I E, (c) JAilabs <1 som => Int A is nonsingular 1,...,m => Im2 + 4@ A is nonsingularEIGENVALUES AND SINGULAR VALUES 67 (21) (Cayley-Hamilton theorem) m A(mxm): Aty...,Am are eigenvalues of A => T](Ailm—A) = 0. a (22) A(mxm): The geometric multiplicity of an eigenvalue \ of A, that is, the number of Jordan blocks with \ on the principal diagonal is not greater than the algebraic multiplicity of A. (See Section 5.2.3 for the Jordan decomposition.) Note: Most of the results of this section can be found in Horn & Johnson (1985). The others are easy to derive from results given there. Chatelin (1993) is a reference for further results on eigenvalues and vectors including computational algorithms. 5.2.2 Optimization Properties of Eigenvalues (1) (Rayleigh-Ritz theorem) A(m x m) Hermitian: rte H Ymin(A) = min {2 Ary (mx 1), 2 ao}, rll Ar maz (A) = max { SH it (mx lee of (2) (Rayleigh-Ritz theorem for real matrices) A (m x m) real symmetric: wAr Ymin(A) = min { = = (mx) real eZ 0h, w'Ar Amar(A) = (A) max { = (3) (Courant-Fischer theorem) A (m x m) Hermitian with eigenvalues Ay <--< Am, 1 Avot rst70 HANDBOOK OF MATRICES (4) (Spectral decomposition of a Hermitian matrix) A (mx m) Hermitian with eigenvalues 1,...,Am: A = UAUA, where A = diag(Ai,...,Am) and U is the unitary (m x m) matrix whose columns are the orthonormal eigenvectors t1,....Um of A associated with A,..., Am: In other words, A= Sdn! = (5) (Spectral decomposition of a real symmetric matrix) A (mxm) teal symmetric with eigenvalues A1,...,Am > A= UAU! where A = diag(Ai,...,Am) and U is the real orthogonal (m x m) matrix whose columns are the orthonormal eigenvectors t1,..., tm of A associated with A,,...,Am. In other words, Jordan Decompositions (6) A (m x m) with distinct eigenvalues A)....,An : There exists a nonsingular (m x m) matrix T such that A = TAT~!, where AL 0 A= 0 Ag is block diagonal with blocks An, 1 0 0 0 dA 1 0 0 0 » tHl...k>n 1 o 0 An on the diagonal and {ni,...,me} = {l,...,n}, that is, the same eigenvalue may appear on the diagonal of more than one Ay. (For more details see Chapter 6.) (7) A (mx m) with m distinct eigenvalues Ay,...,Am : A = TAT", where A = diag(Ai,...,Am) and T is a nonsingular (m x m) matrix whose columns are eigenvectors of A associated with A1,..., Am:EIGENVALUES AND SINGULAR VALUES (8) (Near-diagonal Jordan decomposition) A (mx m) with distinct eigenvalues d1,...,An, € > 0a given real number: ‘There exists a nonsingular (m x m) matrix T such that A=TAT-!, where Ae) 0 A= . 0 Ag(e) is block diagonal with Me 6° 0 3 0 0%, « 0 Afe)=] 0 0 k>n i € 0 0 Dns and {m,...,e} = {1,....n} (9) (Real Jordan decomposition) A(m x m) teal with distinct real eigenvalues ,,..., Ap and distinct complex eigenvalues a, + if1,...,a +13, : There exists a real nonsingular (m x m) matrix T such that A = TAT~!, where Ay 0 Ae rY 0 nr is block diagonal with A» 1 0 0 0 A, 1 0 A=] 0 0 f= 1k DP 1HANDBOOK OF MATRICES. {pi,--.Pk} = {1.....p} and [ an | by Oo .. 0 Pin he 0 0 : he , ® | >a (tg = {hgh Note: The results of this section can be found in Horn & Johnson (1985) See Chapter 6 for further matrix decomposition results. 5.3 Eigenvalue Inequalities 5.3.1 Inequalities for the Eigenvalues of a Single Matrix (1) A (mex m) Hermitian, 2 #0 (m x 1) 1 AES Amar A) 2 Amin A) S a (2) A (mx m) Hermitian with eigenvalues 41(A),-..,Am(A) + at Ax min { 22 x(mx 1), 2F of HA < (A) < imax { 22 aly sox ezoh, i=1,._.m (3) A (mx m) Hermitian with eigenvalues Ay(A) <--> < Am(A). X (aux n): NUN Sh, > YAMA) Ip < det(X" AX) Ts A) < det(A(ny) < 1G m-n4il A) (11) (Inclusion principle) A (m x m) Hermitian with eigenvalues 41(A) < =) < Am(A), Ain) (n x n) a principal submatrix of A with eigenvalues Ai(Ainy) SS An(AGn)) Ai(A) < Ai(Afny) S Am—nyi(A), 7S Len, Amin(A) < Amin (Ain) S Amar(A(ny) S Amar(A)a4 HANDBOOK OF MATRICES 5.3.2 Relations Between Eigenvalues of More Than One Matrix (1) A Gn x m) Hermitian, B (m x m) positive semidefinite (a) Amain(A + B) > Amin( A) (b) Amar(A + B) > Amar(A)- (2) A.B (m x m) Hermitian: (a) Amin(A + B) 2 Amin(A} + Amin(B). (b) Amar(A + B) < Amar(A) + Amaz(B) (3) A.B (m x m) Hermitian, 0 < a,b ¢ IR (a) Anan(a + 6B) > Asnin( A) + bAmin( B) (b) Amar(@A + 6B) < @Amar(A) + bAmar(B) (4) A (xm) Hermitian with eigenvalues A1(A) <=» < Am(A), B (mx m) Hermitian positive sentidefinite. 44(4 + B) < < Am(4A + B) eigenvalues of A+ B M(A+B) DACA) T= 1.0m. (5) A (nxn) Hermitian with eigenvalues A,(A) <--- < Am(A). 2 (nx 1). (At rr) <-.- < Am(A), X (nx n) such that VY XN = Ayn Sm, A(NYAN) <) < An( NH AN] eigenvalues of N# : AA) < AK AX) < Amangi(A), P= 1.0.0 (8) A.B (m x m) Hermitian with eigenvalues A\(A) < --+ < Am(A) and AWB) < +++ < Am(B), respectively; Ai(4 + B) <-+- < Am(A+ B) eigenvalues of A+ B: (a) SUPA) + (BY) < SOA(A+ Bon15 EIGENVALUES AND SINGULAR VALUES (b) Aa(B) < An(A + B) —An(A) < Am(B), on am. (c) [An(A + B) = An(A)labs < max{|Aj(B)lave J = 1,....m}= p(B), n=1,....m. (d) An(A +B) < min{A;(A) +A;(B) 2 i+ j= 4m}, n=l....m S$ Am(A) and < Am(A + B) (9) (Weyl’s theorem) A, B (m x m) Hermitian with eigenvalues 4,(A) < Ai(B) < +++ < Am(B), respectively; A1(A + B) < eigenvalues of A+ B Ai(A) + 1B) din1(A) + A2(B) M(AFB)> >. da(A) + i(B) Ai(A) + Am(B) and dia(A) + Am—1(B) A(A+B) < An(A) +9(B) t=1,...,m. (10) A, B (m x m) Hermitian with eigenvalues 41(A) < --- < Am(A) and Ai(B) < --- < Am(B), respectively; A1(A4 — B) <--> < Am(A - B) am eigenvalues of A — dA) > AB), F= 1. +S Am(A), 2 (mx A(A~B)>0 => (11) A (mx m) Hermitian with eigenvalues 4,(4) < 1), ce R, o-[ 4 7] with eigenvalues 1(B) <--- < Am4i(B) Ai(B) $A(A) < Avgr(B), E=1).-.,m S$ An(A), D (pxp) (12) A (nxn) Hermitian with eigenvalues 1(A) < Hermitian, C (n x p) and AC o= [44 S]mxm with eigenvalues A\(B) < +--+ < Am(B): Ai(B) < Ai(A) S Am—n4i(B) for i=1,....076 HANDBOOK OF MATRICES : ‘The results of this section are given in Horn & Johnson (1985. Chapters 4 and 7) or follow easily from results given there. Some results. in particular those on real symmetric matrices, may also be found in Magnus & Neudecker (1988). Many of these and further results are also contained in Chatelin (1993) 5.4 Results for the Spectral Radius Reminder: A = [a,)]. B= [b,)] (m x n) real: )O <> aj >(2)OFH1L..myjal Bo = ay > (2) bi, (1) AQnxm).i€ IN: (A)! = pl A’) (2) A,B (m x m) Hermitian with eigenvalues \,(4) <->» < Am(A) and AWB) <--> < An (B), respectively; Ay(A + B) < < Am(A + B) vigenvalues of A+ B [ACA + B) = A(A)lavs S p(B) f= 1m (3) A(mxm) (a) p(A) <1 <=> AP —0asn—ox, i., A is convergent (b) pA) <1 > SO a'= (In - Ay! = (ce) A) <1 > SO(-Ay = Un $A 1=0 (d) pAD i=0 4y-! = (Ima — -A OA = (In? +A @ AP? (4) A= [ag] (om x m) real (a) 4>0 > p(A) > 0. (b) AZO, Say >0, i= 1,....m > pA) > 0 1 (c) A stochastic => p(4) > 0. (d) A doubly stochastic = p(A) > 0.EIGENVALUES AND SINGULAR VALUES 7 (e) 4>0,A' > 0 for some i>1 > p(A) > 0. (5) A,B (m x m) real (a) OS ASB = (A) < p(B) (b) 0< A< B = (A) < p(B). (6) A = [ay] (m x m) real, Ain) = principal submatrix of A: g A20 => pA) $ pA), n= (7) A= [aij] > 0 (m x m) real (a) max, ais < pl A) (b) |_min yw 0 (m x m) teal, 2 = a. 2m)! > 0(m x 1) real 1 1 (a) pin = Doon $A) S,max BO Daves. ai zz amin os 2 < pA) < , max meee (9) A (mx m) teal, A> 0,2 (mx 1) real, x > 0.a,b€ IR,a,b>0 (a) ar < Arche > acp(A) a= (A) (10) A (mx m) real, A eigenvalue of A with real eigenvector z #0 A> 0, [Alovs = p(A) = |Alabs is eigenvalue of A with eigenvector78 HANDBOOK OF MATRICES (11) A(m x m) real, A> 0 (a) p(A) is a simple eigenvalue of A, (b) p(A) is eigenvalue of A with eigenvector z > 0 (12) A (mx m) real, A> 0 p(-A) is eigenvalue of A with eigenvector z > 0,2 #0 (13) A (1m x m) teal, 4 > 0, eigenvalue of A AF (A) = [lads < p(A) (14) (Hopf’s theorem) A= [aj] (1m x m) real, A > 0,Am—1 eigenvalue of A with second largest modulus, M = max{aj; :i,j = 1,...,m},a= min{aj : ij = l.....m} VAm=ilabs . M =p p(A) Mp (15) A (mx m) real, A > 0,2 > 0 (mx 1) real eigenvector of A corresponding to eigenvalue p(A), y > 0 (mx 1) real eigenvector of A’ corresponding to eigenvalue p(A'),2"y = 1: lim [p(A)7! A] = zy! ae 0 i =1 gi jim [o( 4)" A} > 0, rk{ lim (p(A}~! AJ} = 1 ince Note: Most results of this subsection are, for instance, given in Horn & Johnson (1985) 5.5 Singular Values Notation: o(A) denotes a singular value of the matrix A. Omin( A) denotes the smallest singular value of the matrix A. Omar(A) denotes the largest singular value of the matrix 5.5.1 General Results (1) A(mxn),m>n: HAH yz omin(A) = win (“r“) sone n.e¢ah aie min{(2! AM Az)? ; 2 (n x 1), 242 = 1}EIGENVALUES AND SINGULAR VALUES 79 (2) A(mxn): 1p Cmaz(A) = mnc{ (5) -sinxry seo} = max{(2" 4" Az)/? : 2 (nx 1),242 = 1}. (3) A (mx n),r = min{m,n}, 0 > --- > oy singular values of A, l o1,....0, singular values of A” (6) A(mxn),r = min{m,n}, o1,...,0¢ singular values of A tr(A4 A) = Le (7) (Singular value decomposition) A(mxn), tk(A) = 4, o1,...,07 #0 singular values of A: There exist unitary matrices U (m x m), V (n xn) such that VTA = [ D | 0 0 where D = diag(o1,...,07) and some ot all of the zero submatrices disappear if r = m and/or r = n.80 HANDBOOK OF MATRICES. (8) (Singular value decomposition of a real matrix) A (m xn) real, rk(A) = 7, op...-,0 # 0 singular values of A There exist real orthogonal matrices {/ (mx m), V (nxn) such that UAV = [ Ho | o 0 where 2 = diag(1,...,¢r) and some or all of the zero submatrices disappear ifr = m and/or r =n (9) Atm xn) U [ ° ; | Vis the singular value decomposition of -4 0 qe ey) Oh 0] pw 2 atar[) al mtin(m.n}. OS Or. or EIR: oy... are the (10) Atm xn), of AS ate ot. ee and [m= laps singular valu zeros are the eigenvalues of 5.5.2 Inequalities (1) ALB On x nj. r = min{iyn)}, (A) > > o,(A) > 0, o(B) > > o,(B) > 0. o (A+ B) > + > o(4+ B) > O singnlar values of A.B and A+ B, respectively Opy-(A+ B) SoA) +o,(B) LSijcrwithitj > oe(A) > 0. on (B) > > 0,(B) > 0, o(ABH) > --- > o(AB") > 0 singular values of 4. B and AB", respectively: Oty AB") < o,(A)o)(B). 1S ijcrwithitj ++ > m(A) singular values of A m-ng3(AP < Do aha; < oar. n=l.2.....m ys jnEIGENVALUES AND SINGULAR VALUES 81 (5) A (mx n), B (m x (n — 1)) is obtained from A by deleting any one column of A, r = min{m,n}, o1(A) > «++ > a-(A) and o1(B) > > min(m,n-1)(B) singular values of A and B, respectively: o1(A) 2 o1(B) 2 02(A) 2 72(B) 2 + 2 on-1(B) 2 on(A) 2 0 ifm >nand o1(A) 2 01(B) 2 02(A) 2 2B) 2 + 2 om(A) 2 om(B) 2 0 ifm «+ > o,(A) and o)(B) > > mmintm-1,n)(B) singular values of A and B, respectively: ay(A) 2 o1(B) 2 2A) 2 02(B) > +++ 2 om-i(B) 2 am(A) 2 0 ifm 79(A) > 2(B) > + > on(A) > n(B) 2 0 ifm>n (7) A(mxn): omar(A) > (4tr(A4 Ay)”. Note: The results of this section can be found in Horn & Johnson (1985). Other books discussing singular values include Barnett (1990), Lancaster & ‘Tismenetsky (1985) and, in particular, Chatelin (1993) where further results may be found6 Matrix Decompositions and Canonical Forms All matrices in this chapter are complex unless otherwise specified. 6.1 Complex Matrix Decompositions 6.1.1 Jordan Type Decompositions (1) (Jordan decomposition) A (mx m) with distinct eigenvalues A1,...,4n : There exists a nonsingular (m x m) matrix T such that A = TAT~!, where Ais a Jordan form, that is, An, 1 0 0 0 An 1 0 A=] 0 0 7). Eo |, Ged kD i . ol 0 0... Yn, on the diagonal and {n,...,n4) = {1,...,n}, that is, the same eigenvalue may appear on the principal diagonal of more than onead HANDBOOK OF MATRICES A. For example, Oe 1 ° [ 0 32 | is a Jordan form of a matrix with the two distinct eigenvalues dy = 2,42 = 3.2 and 2 1:0 2 1 A=]o 21]. =P) r= [4 32 |: 00 2 . (2) (Jordan decomposition of a matrix whose eigenvalues are all distinct) A (m x m) with m distinct eigenvalues 4,,...,Am 2 A= TAT@!, where A = diag(Ay,...,Am) and T is a nonsingular (m x m) matrix whose colutnns are eigenvectors of A corresponding to Ay,..., Am (3) (Near-diagonal Jordan decomposition) A (mx m) with distinct eigenvalues Aj,....An,¢ > O a given real number: ‘There exists a nonsingular (m x a) matrix T such that A= TAT~!, where AA(e) 0 A= : 0 Ag(e) is block diagonat with An € 0 0 0 An, € 0 Mlg=] 0 0 i=l.k>n € 0 0 An, and {nj,....me} = {1,-..,n}, that is, the same eigenvalue miay appear on the principal diagonal of more than one Aj(c) Note: For proofs see, e.g.. Horn & Johnson (1985, Chapter 3) or Barnett (1990, Chapter 8).MATRIX DECOMPOSITIONS AND CANONICAL FORMS 85, 6.1.2 Diagonal Decompositions (1) (Spectral decomposition of a normal matrix) A (mx_m) normal with eigenvalues 1,...,Am: A= UAU, where A = diag(A1,...,m) and U is the unitary (m x m) matrix whose columns are the orthonormal eigenvectors t,...,¥m of A associated with Ay,...,Am. In other words, ave (2) (Spectral decomposition of a Hermitian matrix) A (m x m) Hermitian with eigenvalues 4),...,Am : A = UAU#, where A = diag(A,,...,Am) and U is the unitary (m x m) matrix whose columns are the orthonormal eigenvectors v,...,tm of A associated with Ay,...,Am. In other words, A= dnl! ii (3) (Singular value decomposition) A(mxn), rk(A) =r, o1,...,¢, #0 singular values of A: There exist unitary matrices U (mx m), V (nx n) such that -ul? lye A=U | 0 0 |h where D = diag(o,...,¢-) and some or all of the zero submatrices disappear when r = m and/or r =n. (4) (Diagonal reduction) A(mxn), rk(A) =r: There exists a nonsingular (m x m) matrix S and a nonsingular (n % n) matrix T such that o[ J, 0 aes[5 S]r where some or all of the zero submatrices disappear when r = 1m and/or r =n (5) (Congruent canonical form) A(m xm) Hermitian: There exists a nonsingular (rm x m) matrix T such that A = TAT, where A = diag(d),...,dm) and the d, are +1,—1 or 0 corresponding to the positive, negative and zero eigenvalues of A, respectively.86 HANDBOOK OF MATRICES (6) A (mx m) symmetric, rk(A) = 7: There exists a nonsingular (m x m) matrix T such that i, 0 ae r| x 0 |. where the zero submatrices disappear when r = m (7) A(mxm): There exists a nonsingular (m xm) matrix T, a unitary (mxm) matrix U and an (mxm) diagonal matrix A with nonnegative elements on the principal diagonal such that A= TUAU'T~'. Simultaneous Diagonalization of Two Matrices (8) (Simultaneous diagonalization of two Hermitian matrices) A,B (mxm) Hermitian, AB = BA: There exists a unitary (mxm) matrix U such that A = UDU* and B = UAUH, where D and A are diagonal matrices. (9) (Simultaneous diagonalization of a positive definite and a Hermitian matrix) A (mxm) Hermitian positive definite, B (mxm) Hermitian: There exists a nonsingular (m x m) matrix T such that A = TT” and B=TAT#, where A is a diagonal matrix. (10) (Simultaneous diagonalization of a positive definite and a symmetric matrix) A (mx m) positive definite, B (m x m) symmetric: There exists a nonsingular (m x m) matrix T such that A= TT and B = TAT’, where A is areal diagonal matrix with nonnegative diagonal elements. Note: The results of this subsection and further diagonal decompositions may be found in Horn & Johnson (1985, Chapters 3, 4 and 7) and Rao (1973, Chapter 1). 6.1.3 Other Triangular Decompositions and Factorizations (1) (Schur decomposition) A(mxm): There exists a unitary (m x m) matrix U and an upper triangular matrix A with the eigenvalues of A on the principal diagonal such that A= UAU#, (Choleski decomposition) A (mx m) positive definite: There exists a unique lower (upper) triangular (m x m) matrix B with real positive principal diagonal elements such that A= BB, (2MATRIX DECOMPOSITIONS AND CANONICAL FORMS. 87 (3) (4 (5) (6) (7) (8) (9) (Triangular factorization) A= [ajj] (m x m), rk(A) =r aio. Gin det nd #0, n=l,...,7: Quy. Gan There exists a lower triangular (m x m) matrix L and an upper triangular (m x m) matrix U, one of which is nonsingular, such that A=LU (Triangular factorization of a nonsingular matrix) A= [aij] (m x m) nonsingular, ay - din det |: moot #0, Gn ee Gan There exists a unique lower triangular (m x m) matrix L and a unique upper triangular (m x m) matrix {’ both having unit principal diagonal such that A = LDU, where D = diag(d),...,dm) with a). Gin Il = det yah any nn (Gram-Schmidt triangular reduction) A(mxn),m > n: There exists an (m x n) matrix U with orthogonal columns and an upper triangular (n x n) matrix R such that A= UR. (Gram-Schmidt triangular reduction of a square matrix) A(mxm): There exists a unitary (mx m) matrix {! and an upper triangular (m x m) matrix R such that A= UR. (Gram-Schmidt triangular reduction of a nonsingular matrix) A (mx m) nonsingular: There exists a unique unitary (m x m) matrix U and a unique upper triangular (m x m) matrix R with positive principal diagonal such that A = UR. A(mxm): There exist (mx m) permutation matrices P and Q.a lower triangular (m x m) matrix L and an upper triangular (m x m) matrix U so that A= PLUQ, A (mx m) nonsingular: There exists an (m x m) permutation matrix P, a lower triangular (m x m) matrix L and an upper triangular (m x m) matrix U such that A = PLU88 HANDBOOK OF MATRICES Note: For proofs and further details see, e.g., Horn & Johnson (1985, Chapters 2 and 3). 6.1.4 Miscellaneous Decompositions (1) (Rank factorization) A(mxn), rk(A) =r: There exists an (mx r) matrix B and an (rx n) matrix C with rk(B) = rk(C) = r such that A = BC. (2) (Polar decomposition) A(mxm), rk(A) =r: There exists a positive semidefinite (mx m) matrix P with rk(P) = rand a unitary (m x m) matrix U such that A= PU. (3 A(mxm) nonsingular: There exists a symmetric (m x m) matrix P and a (complex) orthogonal (m x m) matrix Q such that 4 = PQ (4 (Square root decomposition) A (m x m) positive (semi) definite: There exists a positive (semi) definite (m x m) matrix B such that A = BB, that is, B is a square root of A. (5) A(mxm) unitary: There exists a real orthogonal (m x m) matrix Q and a real symmetric (m x m) matrix S such that A = Qexp(iS) (6) A (mx m) orthogonal : There exists a real orthogonal (m x m) matrix Q and a real skew-symmetric (m x m) matrix S such that A= Qexp(iS). (7) (Echelon form) A (mx m) There exists a nonsingular (m x m) matrix B such that BA = [ej] is in echelon form, that is, for each row i, either ci; = 0,j = 1,...,m, or there exists a k € {1,...,m} such that cig = Land ci = 0 for j < k and cy = 0 for U ¢ i. (For more details on the echelon form see the Appendix.) (8 (Hermite canonical form) A(mxm): There exists a nonsingular (m x m) matrix B such that BA is an idempotent Hermite canonical form. (For the definition of a Hermite canonical form see the Appendix.) (9) A(mxm): There exists a diagonalizable (m x m) matrix D and a nilpotent (m xm) matrix N such that A= D+ N and DN = ND. Note: For results (1), (7) and (8) see Rao (1973, Chapter 1). The other results may be found in Horn & Johnson (1985, Chapter 3). Both references contain a number of further decomposition resultsMATRIX DECOMPOSITIONS AND CANONICAL FORMS 89 6.2 Real Matrix Decompositions 6.2.1 Jordan Decompositions (1) (Jordan decomposition of a real matrix with real eigenvalues) A (m x m) real with distinct eigenvalues A1,...,4n which are all real: There exists a real nonsingular (m x m) matrix T’ such that A=TAT™!, where A 0 A= 0 Ag is block diagonal with a 1 0 0 An, 1 0 0 0 : kn : : 1 0 0... An. and {ni,...,nk} = {l,...,n}, that is, the same eigenvalue may appear on the principal diagonal of more than one Ay (2) (Real Jordan decomposition) A(m x m) real with distinct real eigenvalues ,,...,Ap and distinct complex eigenvalues 01 + ift,.-.,a4 4 if, : ‘There exists a real nonsingular (m x 1m) matrix T such that A= TAT=!, where Ay 0 _ Ne A= n 0 r, is block diagonal with dp 10 0 0 Ay 0 ac] oo o kD, : : 1 0 0 a90 HANDBOOK OF MATRICES {Pi.---. Pe} = {1,..-.p}, and as, Bay I 0 0 ~ Ba, «| ° , ay, By 0 a Pa Ty 0 [a -| ‘ l= 0 0 : bh ag, Bg, G=h..82q (dig = {hg}. Note: For proofs and further details see Horn & Johnson (1985, Chapter 3) or Barnett (1990, Chapter 8) 6.2.2 Other Real Block Diagonal and Diagonal Decompositions (1) A (mx) real normal (A’A = AA’): There exists a real orthogonal (m x m) matrix Q such that Ay 0 A=Q . Q. 0 Me where the A, are real numbers or real (2 x 2) matrices of the form a; by a= [$f]. (2) (Spectral decomposition of a real symmetric matrix) A(mxm) real symmetric with eigenvalues A1,...,Am: A= QAQ’, where A = diag(Ai,...,Am) and Q is the real orthogonal (m x m) matrix whose columns are the orthonormal eigenvectors v),....Um of A associated with A1,..., Am. In other words, A= So avest iIMATRIX DECOMPOSITIONS AND CANONICAL FORMS. 91 (3) A (mx m) real skew-symmetric: ‘There exists a real orthogonal (m x m) matrix Q such that 0 0 0 Ag where the Aj are real (2 x 2) matrices of the form 0 8 we[ 8h). (4) A (mx m) real orthogonal: There exists a real orthogonal (m x m) matrix Q such that A 0 where the A; = +1 and the Aj are real (2 x 2) matrices of the form A -| cos 6; sind; sin 8; cos 6; (5) (Singular value decomposition of a real matrix) A (m x n) real, rk(A) = 7, 01,...,0¢ # 0 singular values of A ‘There exist real orthogonal matrices U (mx m), V (nxn) such that -yl? ly ase[ oY where D = diag(o;,...,0;) and some or all of the zero submatrices disappear when r = m and/or r =n Note: The results (1) ~ (4) are taken from Horn & Johnson (1985, Chapter 2) and for (5) see Chapter 7 of the same reference.92 HANDBOOK OF MATRICES 6.2.3 Other Triangular and Miscellaneous Reductions q) (2 (3) (4) (5 (6 (7) (8) (9) (Schur decomposition of a real matrix with real eigenvalues) A (mx) teal with real eigenvalues: ‘There exists a real orthogonal (m x m) matrix Q and a real upper triangular matrix A with the eigenvalues of A on the principal diagonal such that A = QAQ! (Choleski decomposition of a real matrix) A (m x m) teal positive definite : There exists a unique real lower (upper) triangular (m x m) matrix B with positive diagonal elements such that A = BB’. (Gram-Schmidt triangular reduction of a real matrix) A(mxn) real, m >: There exists a real (m x n) matrix Q with orthogonal columns and a real upper triangular (n x n) matrix R such that A = QR. A(mxn) teal, m>n: There exists a real orthogonal (m x m) matrix Q and a real upper triangular (n x n) matrix R such that. aa=[4 |: Oon-nyxn where the zero matrix disappears if n = m. (Gram-Schmidt triangular reduction of a real square matrix) A(mxm) real: ‘There exists a real orthogonal (m x m) matrix Q and a real upper triangular (m x m) matrix A such that 4 = QR. (Gram-Schmidt triangular reduction of a real nonsingular matrix) A(mxm) real nonsingular: There exists a unique real orthogonal (m xm) matrix Q and a unique upper triangular real (m x m) matrix R with positive principal diagonal such that A = QR. A(mxm) real: ‘There exists a real orthogonal (m x m) matrix Q and an upper Hessenberg matrix R such that 4 = QRQ’ (Square root decomposition) A (mx m) real positive (semi) definite: There exists a real positive (semi) definite (m x m) matrix B such that A = BB, that is, B is a square root of A. (Simultaneous diagonalization of a real positive definite and a real symmetric matrix) A (m xm) real positive definite, B (m xm) real symmetric: There exists a real nonsingular (m x m) matrix T such that A = TT" and B= TAT", where A is a real diagonal matrix. Note: These decomposition and factorization theorems can be found, for instance, in Horn & Johnson (1985) and Rao (1973). Algorithms for computingMATRIX DECOMPOSITIONS AND CANONICAL FORMS 93 many of the matrix factors discussed in this chapter are described in Golub & Van Loan (1989)7 Vectorization Operators All matrices in this chapter are complex unless otherwise specified. 7.1 Definitions vec denotes the column vectorizing operator which stacks the columns of a matrix in a column vector, that is, for an (m x n) matrix A = [aj,), ay ay mnt ay vec A =vee(A)=col(A)= |. | (mn x 1), mn Related operators are rvec(A) = [ans aia, + sim @2is + 2m Q5ty ++) 4mn] (Lx mn) [vec(A)]’96 HANDBOOK OF MATRICES which stacks the rows of A in a row vector and an a2 Gn aay row(A) = vec(A’) = rvec( A)! = (mn x 1) a2, a3) Oma which stacks the rows of A in a column vector. For example, ay aa a1 a2 as, vec | a2, a22 | = a3, 32 a2 a22 432 and a1 a2 a1 2 aay tow | a2 a22 | = a3, a9 422 1 as a3) 432 ‘The half-vectorization operator, vech, stacks only the columns from the principal diagonal of a square matrix downwards in a columin vector, that is, for an (m x m) matrix 4 = {ai;), ay a2 amt 422 vech A = vech(A) = (Jm(m +1) x 1) m2 33 GnVECTORIZATION OPERATORS 97 For example, an aa) 1 G2 413 . a3) vech | 21 azz az3 | = | & 22 a3, 432 433 asy a3 In the following sections results are given for vec and vech only. Similar results can be obtained for other vectorization operators from the results given. The following matrices are closely related to the vec and vech operators: © Kya or Kmn is an (mn x mn) commutation matrix defined such that for any (m x n) matrix A, Kmnvec(A) = vec(A’) (see Section 9.2 for its properties). @ Dm denotes the (m? x }m(m + 1)) duplication matrix defined such that vec(4) = Dmvech(A) for any symmetric (m x m) matrix A (see Section 9.5) ¢ Lm denotes the ($m(m + 1) x m?) elimination matrix defined such that vech(A) = Imvec(A) for any (mx m) matrix A (see Section 9.6) 7.2 Rules for the vec Operator (1) A,B(mxn): vec(A + B) = vee(A) + vec(B). (2) A(mxn),c€€: vec(cA) = c vee( A). (3) A(mxn): (a) vee(A’) = Kyanvec(A). (b) vec(A) = Kamvec( A’). (4) A(x mm): Lmvee(A) = vech(A) (5) A(mxn), B(n xp) yec(AB) = (Ip @ A)vec(B) = (B’ ® Im)vec( A) = (BY & A)vec( In). (6) a(mx 1),6(n x1): vec(ab’) = b@a. (7) A(mxn), B(n xr), C(r xs): vec(ABC) = (C’ ® A)vec(#). (8) A(mxn), B(r xs): (a) vec(AS B) = (In @ Kam @ 1)(vec(A) @ vee( B)). (b) vec(A’ ® B) = (Kinen @ Ir)(vec(A) ® vec( B)) (c) vec(A ® B') = (In ® Kyms)(vec( A) ® vec(B)).98 HANDBOOK OF MATRICES (d) vec(A’ ® B) = (Kinsn(In ® Kins) ® Ir)vec(A ® B) (e) vec(A ® B’) = (In @ Kyms(Kms ® Ip))vec(A @ B). (f) vec(A) @ vec(B) = (In @ Kins @ Ip)vec(A® B). (9) A, B (mx n) (a) vec(A © B) = diag(vec A)vec( B) = diag(vee Byvec( A) (b) vec(A © B) = (vee A) ® vec( B) (c) vec(A)'vee( B) = vec( BY‘ vec( A). (10) A(mxn), B(nx m): (a) vec(A’'vee( B) = tr(AB) = tr(BA). (b) vec( A’)'vec( B) = vec( B')/vec( A) (11) A(mxn), B(nx p), C(p xq), D(q xm): tl vec(D’)'(C’ ® A)vec( B) tr(ABCD) tr(DABC) tr(CDAB) tr(BCDA). (12) A(m xn), B (nx p),C (p xq), D (ax m): vee(D’)'(C’® A)vec(B) = vec(A’)'(D! @ B)vec(C) vec( B')'(A’ ® C)vec(D) vec(C")'(B" ® D)vec( A) vec( D)'(A © C")vec( B’) nou (13) A(m xn), B(nx p), C(pxq), D(q x m) vee(D')'(C' ® A)vec(B) = vec(D’)'vec(ABC) vee( A’ D')'vec( BC) vec(A’D'C’)'vec( B). uo (14) A(mxm): (a) A lower triangular => vec(A) = Li, vech(A) (b) A symmetric = Dy, vec(A) = vech(A). (c) Asymmetric > vec(A) = Dm vech(A) (15) A (mx m): (a) Df, vec(A) = vech(A + A’ — dg(A)) (b) Dé vec(A) = }vech(A + A’)VECTORIZATION OPERATORS 99 (c) Dm Di,vec(A) = vec(A + A’ — dg(A)). (A) vec(dg(A)) = Lin Lin Kin Liy Lm vec( A) (16) A(mxm), a,6(mx 1), c€C: v © Dingsvee[ § a ats |, . Di,,vec(A) c Dé, yvee [ ce ] =| Hats) Ditvec(A) (17) A,S\V (mx m): $= ASA'+V and Im2—A@ A nonsingular => vee($) = (Im2 — A@ A)" vee(V). Note: A very rich source for results on the vec operator is Magnus (1988). Magnus & Neudecker (1988) also contains many of the results given here 7.3 Rules for the vech Operator (1) A,B(mxm): vech(A + B) = vech(A) + vech(B). (2) A(mxm),c€€: vech(cA) = c vech(A). (3) A,B (m x m): (a) vech(A® B) = diag(vech A)vech(B) = diag(vech B)vech( A). (b) vech(A® B) = (vech A) © vech(B). (4) A(mxm): (a) vech(A) = Lmvec(A) (b) vech(A + A’) = 2D# vec( A). (c) vech(A + A’ — dg(A)) = Di, vec( A). (d) Di, Dmvech( A) = 2vech(A) — vech(dg(A)). (e) vech(dg(A)) = LmKmm Linvech( A). (5) A(mxm): (a) A lower triangular = L‘,,vech(A) = vec( A). (b) Asymmetric => vech(A) = Diivec(A).100 HANDBOOK OF MATRICES (c) Asymmetric = Dyvech(A) = vec(A). (6) A(mxn), B(nxp), C (px): vech(ABC) = Lin(C'® A)vec( B) (7) A, B(m xm): vech(A)! vech(B) = vech(B)' vech( A) Note: Many results for the vech operator including the nontrivial ones given here are collected in Magnus (1988).8 Vector and Matrix Norms 8.1 General Definitions A function |||] attaching a nonnegative real number ||A]| to an (mx n) matrix A is a norm if the following three conditions are satisfied for all complex (m x n) matrices A, B and complex numbers ¢ : (i) [AI] >0 if AO, (ii) lel] = lelabs [All (iii) [|A + Bll

You might also like

• Her Body and Other Parties: Stories

  

  From Everand

  Her Body and Other Parties: Stories

  Carmen Maria Machado

  Rating: 4 out of 5 stars

  4/5 (821)
• Sing, Unburied, Sing: A Novel

  

  From Everand

  Sing, Unburied, Sing: A Novel

  Jesmyn Ward

  Rating: 4 out of 5 stars

  4/5 (1104)
• The Sympathizer: A Novel (Pulitzer Prize for Fiction)

  

  From Everand

  The Sympathizer: A Novel (Pulitzer Prize for Fiction)

  Viet Thanh Nguyen

  Rating: 4.5 out of 5 stars

  4.5/5 (122)
• • Magazines
  • Podcasts
  • Sheet music
• Devil in the Grove: Thurgood Marshall, the Groveland Boys, and the Dawn of a New America

  

  From Everand

  Devil in the Grove: Thurgood Marshall, the Groveland Boys, and the Dawn of a New America

  Gilbert King

  Rating: 4.5 out of 5 stars

  4.5/5 (266)
• John Adams

  

  From Everand

  John Adams

  David McCullough

  Rating: 4.5 out of 5 stars

  4.5/5 (2409)
• The Perks of Being a Wallflower

  

  From Everand

  The Perks of Being a Wallflower

  Stephen Chbosky

  Rating: 4.5 out of 5 stars

  4.5/5 (2104)
• Brooklyn: A Novel

  

  From Everand

  Brooklyn: A Novel

  Colm Toibin

  Rating: 3.5 out of 5 stars

  3.5/5 (1946)
• The Light Between Oceans: A Novel

  

  From Everand

  The Light Between Oceans: A Novel

  M.L. Stedman

  Rating: 4.5 out of 5 stars

  4.5/5 (789)
• A Tree Grows in Brooklyn

  

  From Everand

  A Tree Grows in Brooklyn

  Betty Smith

  Rating: 4.5 out of 5 stars

  4.5/5 (1929)
• A Man Called Ove: A Novel

  

  From Everand

  A Man Called Ove: A Novel

  Fredrik Backman

  Rating: 4.5 out of 5 stars

  4.5/5 (4610)
• Fear: Trump in the White House

  

  From Everand

  Fear: Trump in the White House

  Bob Woodward

  Rating: 3.5 out of 5 stars

  3.5/5 (738)
• Rise of ISIS: A Threat We Can't Ignore

  

  From Everand

  Rise of ISIS: A Threat We Can't Ignore

  Jay Sekulow

  Rating: 3.5 out of 5 stars

  3.5/5 (137)
• Team of Rivals: The Political Genius of Abraham Lincoln

  

  From Everand

  Team of Rivals: The Political Genius of Abraham Lincoln

  Doris Kearns Goodwin

  Rating: 4.5 out of 5 stars

  4.5/5 (234)
• Wolf Hall: A Novel

  

  From Everand

  Wolf Hall: A Novel

  Hilary Mantel

  Rating: 4 out of 5 stars

  4/5 (3811)
• A Heartbreaking Work Of Staggering Genius: A Memoir Based on a True Story

  

  From Everand

  A Heartbreaking Work Of Staggering Genius: A Memoir Based on a True Story

  Dave Eggers

  Rating: 3.5 out of 5 stars

  3.5/5 (231)
• The Woman in Cabin 10

  

  From Everand

  The Woman in Cabin 10

  Ruth Ware

  Rating: 3.5 out of 5 stars

  3.5/5 (2521)
• Little Women

  

  From Everand

  Little Women

  Louisa May Alcott

  Rating: 4 out of 5 stars

  4/5 (104)
• The Art of Racing in the Rain: A Novel

  

  From Everand

  The Art of Racing in the Rain: A Novel

  Garth Stein

  Rating: 4 out of 5 stars

  4/5 (4203)
• Hidden Figures: The American Dream and the Untold Story of the Black Women Mathematicians Who Helped Win the Space Race

  

  From Everand

  Hidden Figures: The American Dream and the Untold Story of the Black Women Mathematicians Who Helped Win the Space Race

  Margot Lee Shetterly

  Rating: 4 out of 5 stars

  4/5 (897)
• The Constant Gardener: A Novel

  

  From Everand

  The Constant Gardener: A Novel

  John le Carré

  Rating: 3.5 out of 5 stars

  3.5/5 (104)
• The Outsider: A Novel

  

  From Everand

  The Outsider: A Novel

  Stephen King

  Rating: 4 out of 5 stars

  4/5 (1850)
• The Unwinding: An Inner History of the New America

  

  From Everand

  The Unwinding: An Inner History of the New America

  George Packer

  Rating: 4 out of 5 stars

  4/5 (45)
• Angela's Ashes: A Memoir

  

  From Everand

  Angela's Ashes: A Memoir

  Frank McCourt

  Rating: 4.5 out of 5 stars

  4.5/5 (440)
• The Yellow House: A Memoir (2019 National Book Award Winner)

  

  From Everand

  The Yellow House: A Memoir (2019 National Book Award Winner)

  Sarah M. Broom

  Rating: 4 out of 5 stars

  4/5 (98)
• Yes Please

  

  From Everand

  Yes Please

  Amy Poehler

  Rating: 4 out of 5 stars

  4/5 (1898)
• The Emperor of All Maladies: A Biography of Cancer

  

  From Everand

  The Emperor of All Maladies: A Biography of Cancer

  Siddhartha Mukherjee

  Rating: 4.5 out of 5 stars

  4.5/5 (271)
• The World Is Flat 3.0: A Brief History of the Twenty-first Century

  

  From Everand

  The World Is Flat 3.0: A Brief History of the Twenty-first Century

  Thomas L. Friedman

  Rating: 3.5 out of 5 stars

  3.5/5 (2259)
• Manhattan Beach: A Novel

  

  From Everand

  Manhattan Beach: A Novel

  Jennifer Egan

  Rating: 3.5 out of 5 stars

  3.5/5 (792)
• Shoe Dog: A Memoir by the Creator of Nike

  

  From Everand

  Shoe Dog: A Memoir by the Creator of Nike

  Phil Knight

  Rating: 4.5 out of 5 stars

  4.5/5 (537)
• The Little Book of Hygge: Danish Secrets to Happy Living

  

  From Everand

  The Little Book of Hygge: Danish Secrets to Happy Living

  Meik Wiking

  Rating: 3.5 out of 5 stars

  3.5/5 (401)
• The Glass Castle: A Memoir

  

  From Everand

  The Glass Castle: A Memoir

  Jeannette Walls

  Rating: 4.5 out of 5 stars

  4.5/5 (1716)
• Grit: The Power of Passion and Perseverance

  

  From Everand

  Grit: The Power of Passion and Perseverance

  Angela Duckworth

  Rating: 4 out of 5 stars

  4/5 (590)
• Elon Musk: Tesla, SpaceX, and the Quest for a Fantastic Future

  

  From Everand

  Elon Musk: Tesla, SpaceX, and the Quest for a Fantastic Future

  Ashlee Vance

  Rating: 4.5 out of 5 stars

  4.5/5 (474)
• Bad Feminist: Essays

  

  From Everand

  Bad Feminist: Essays

  Roxane Gay

  Rating: 4 out of 5 stars

  4/5 (1016)
• On Fire: The (Burning) Case for a Green New Deal

  

  From Everand

  On Fire: The (Burning) Case for a Green New Deal

  Naomi Klein

  Rating: 4 out of 5 stars

  4/5 (74)
• The Subtle Art of Not Giving a F*ck: A Counterintuitive Approach to Living a Good Life

  

  From Everand

  The Subtle Art of Not Giving a F*ck: A Counterintuitive Approach to Living a Good Life

  Mark Manson

  Rating: 4 out of 5 stars

  4/5 (5807)
• The Hard Thing About Hard Things: Building a Business When There Are No Easy Answers

  

  From Everand

  The Hard Thing About Hard Things: Building a Business When There Are No Easy Answers

  Ben Horowitz

  Rating: 4.5 out of 5 stars

  4.5/5 (346)
• The Gifts of Imperfection: Let Go of Who You Think You're Supposed to Be and Embrace Who You Are

  

  From Everand

  The Gifts of Imperfection: Let Go of Who You Think You're Supposed to Be and Embrace Who You Are

  Brené Brown

  Rating: 4 out of 5 stars

  4/5 (1091)
• Never Split the Difference: Negotiating As If Your Life Depended On It

  

  From Everand

  Never Split the Difference: Negotiating As If Your Life Depended On It

  Chris Voss

  Rating: 4.5 out of 5 stars

  4.5/5 (842)
• Principles: Life and Work

  

  From Everand

  Principles: Life and Work

  Ray Dalio

  Rating: 4 out of 5 stars

  4/5 (599)
• Steve Jobs

  

  From Everand

  Steve Jobs

  Walter Isaacson

  Rating: 4.5 out of 5 stars

  4.5/5 (806)