Simultaneous Diagonalization

John E. Mitchell

Department of Mathematical Sciences

RPI, Troy, NY 12180 USA

April 2018

Mitchell Simultaneous Diagonalization 1 / 22

 SDP formulation

Outline

1 SDP formulation

2 Simultaneous diagonalization

3 Simultaneous diagonalization of optimal solutions

4 Strict complementarity

Mitchell Simultaneous Diagonalization 2 / 22

 SDP formulation

Primal SDP
We write our standard form semidefinite program as

minX C•X
subject to Ai • X = bi , i = 1, . . . , m (1)
X ⌫ 0

where:
C is an n ⇥ n symmetric matrix
Ai is an n ⇥ n symmetric matrix for i = 1, . . . , m
bi is a scalar for i = 1, . . . , m.

The parameter matrices C and Ai need not be positive semidefinite,

although they are assumed to be symmetric. Recall that C • X
represents the Frobenius inner product between the symmetric
matrices C and X , which is equal to the trace(CX ).

Mitchell Simultaneous Diagonalization 3 / 22

 SDP formulation

Dual SDP

The dual problem is

maxy 2IR m ,S2Sn+ bT y

Pm
subject to i=1 yi Ai + S = C (2)
S ⌫ 0
Pm
Notice that the dual slack variables S = C i=1 yi Ai constitute a
symmetric positive semidefinite matrix in feasible dual solutions.

Mitchell Simultaneous Diagonalization 4 / 22

 Simultaneous diagonalization

Outline

1 SDP formulation

2 Simultaneous diagonalization

3 Simultaneous diagonalization of optimal solutions

4 Strict complementarity

Mitchell Simultaneous Diagonalization 5 / 22

 Simultaneous diagonalization

Simultaneous diagonalization

Two symmetric n ⇥ n matrices are simultaneously diagonalizable if

they have the same eigenvectors.

Lemma
If the n ⇥ n symmetric matrices M and R are simultaneously
diagonalizable then they commute.

1412=1211.

Mitchell Simultaneous Diagonalization 6 / 22

 Simultaneous diagonalization

diagonal.
Proof of lemma A: a r e
eigenvalues.
entries
din,
Let the-columns of the orthogonal matrix P consist of the eigenvectors
of the matrices, so ftp.E-ptp
O F-fr...-in)
A
M = P⇤P T , R = P⌅P T

for two diagonal matrices ⇤ and ⌅. Then

I
e .

l
-
I
MR = P⇤P T P⌅P T
= P⇤⌅P T since P is orthogonal
T
= P⌅⇤P since diagonal matrices commute
I I
= P⌅P P⇤P Tl
T
since P is orthogonal

as required.
= RM

'f";D".. -11"":*.)
Mitchell Simultaneous Diagonalization 7 / 22
 Simultaneous diagonalization

The converse
The converse also holds, so symmetric matrices commute if and only if
they are simultaneously diagonalizable.
Sketch of proof of converse:
ti¥Duu=u¥@Eu)
Assume symmetric R, M commute.
-
Assume R = UDU T , so diagonal entries of D are eigenvalues of R,
and columns of U are eigenvectors of R. Then

- RM = MR =) UDU T M = MUDU T =) DU T MU = U T MUD

I I
1 ) G )
since U orthogonal. For (i, j)-entry, Dii (UMU T )ij = (UMU T )ij Djj , so
2
⇤ 0 ... 0
3 I
where each block corresponds
-
I
6 0 ⇤ 0 7
6 7 to a different eigenvalue Dii .
UMU T = 6 .. . . .. 7
4 . . . 5 Diagonalize M by diagonalizing
0 0 ... ⇤ diagonal blocks -/ eigenspaces of R

Mitchell Simultaneous Diagonalization 8 / 22

 Simultaneous diagonalization of optimal solutions

Outline

1 SDP formulation

2 Simultaneous diagonalization

3 Simultaneous diagonalization of optimal solutions

4 Strict complementarity

Mitchell Simultaneous Diagonalization 9 / 22

 Simultaneous diagonalization of optimal solutions

Duality gap M i r C o x
m a x bly
st. Aiox--bi f t . S=C-Ey.-Ai
Xiao SEO
Let X be feasible in (1) and (y , S) be feasible in (2). The duality gap is
m
! n e
X
T
C•X b y = yi Ai + S • X bT y
i=1 i
m
X
= S•X + yi (Ai • X ) bT y

= S•X +b y = i=1
T
bT y
= S • X.

Mitchell Simultaneous Diagonalization 10 / 22

 Simultaneous diagonalization of optimal solutions

Frobenius product of X ⇤ and S ⇤

Assume (1) and (2) have optimal solutions and their values agree.
Let X ⇤ solve (1) and (y ⇤ , S ⇤ ) solve (2).
From the argument above, the Frobenius product

- trace(S ⇤ X ⇤ ) = S ⇤ • X ⇤ = 0. (3)

(5×44) i s z e r o .
of
Sun of alia, entries

Mitchell Simultaneous Diagonalization 11 / 22

 Simultaneous diagonalization of optimal solutions

Eigenvectors of optimal solutions

-oil
six#f:::
We can actually say something far stronger,
namely the# matrix product S ⇤ X ⇤ = 0, the zero matrix.
So it is not just the trace of S ⇤ X ⇤ that is equal to zero.
Theorem
Let X ⇤ solve (1) and (y ⇤ , S ⇤ ) solve (2) and assume S ⇤ • X ⇤ = 0. Then
-
S⇤X ⇤ = 0 = X ⇤S⇤.

Mitchell Simultaneous Diagonalization 12 / 22

 Simultaneous diagonalization of optimal solutions

Proof, part 1
Since S ⇤ 2 Sn+ it has an eigendecomposition,

S ⇤ = Q⇤Q T ,
indi
of S ⇤ . We now have trace
a:}
where the columns {qi , i = 1, . . . , n} of Q are the eigenvectors of S ⇤
and the diagonal entries of the diagonal matrix ⇤ are the eigenvalues
(ACE'X*Q)
=§(gAijCQTMQ7jiJ-
0 = S ⇤ • X→
⇤

= trace(Q⇤Q T X ⇤ )

EAiiCdx4QiiI@catx-
= trace(⇤Q T X ⇤ Q) from properties of trace
X n ⇣ ⌘

ealiiE.EQIu.x't.ie;
= ⇤ii Q T X ⇤ Q since ⇤ is diagonal
ii
i=1
Xn
= ⇤ii qiT X ⇤ qi
-
=§{QkiX*uQti
i=1

Mitchell Simultaneous Diagonalization 13 / 22

 Simultaneous diagonalization of optimal solutions

Proof, part 2 O= § H i i q¥Exqi

0 : X EO .
- :3¥
Since X ⇤ is positive semidefinite, we have qiT X ⇤ qi 0 8i.
Since S ⇤ is positive semidefinite, we have ⇤ii 0 8i.
Thus, we must have
O
⇤ii qiT X ⇤ qi = 0 8i.
Hence, if- qi is an eigenvector of S ⇤ with a positive eigenvalue, we must
have hii-O
l X
l#EI:
T ⇤ T T T 2
0 = qi X qi = qi LL qi = kL qi k
-

where L is the Cholesky factor of X ⇤ .

Thus,- LT qi = 0, so X ⇤ qi = 0. So qi is in the nullspace of X ⇤ ,
so it is an eigenvector of X ⇤ with eigenvalue 0.
* 0 = 0 i
L Tgi = O =) L E qi i o x gi t q
Mitchell Simultaneous Diagonalization 14 / 22
 Simultaneous diagonalization of optimal solutions

Proof, part 3

We order the columns of Q as Q = [Q1 , Q2 ],

where the columns of Q1 are eigenvectors with positive eigenvalue
and the columns of Q2 are eigenvectors with an eigenvalue of 0.
We can write the eigendecomposition as

-
⇤
⇥
S = Q1 Q2

⇤ ⇤1 0
0 0

Q1T
Q2T
=eaQt
where the diagonal entries of ⇤1 are positive.
Note that-X ⇤ Q1 = 0, since each eigenvector qi of S ⇤ with positive
eigenvalue satisfies X ⇤ qi 1 = 0.(st) 4
gr) wilt r e ran

Q,:[q,... XQ,=[Xq, Xq... Xqr)

,

Mitchell Simultaneous Diagonalization 15 / 22

 Simultaneous diagonalization of optimal solutions

Proof, part 4
I
Thus,
m  

cyan
⇤ ⇤
⇥
⇤
⇤ ⇤1 0 Q1T
X S = X Q1 Q2
- 0 0 Q2T
 
⇥ ⇤ ⇤1 0 Q1T
= 0 X ⇤ Q2
0 0 Q2T

①0 0

⇥ Q1T ⇤
=
Q2T
= 0, the n ⇥ n matrix of zeroes.

We also have
S ⇤ X ⇤ = (X ⇤ S ⇤ )T = 0T = 0,
as required. 54,#
since symmetric

Mitchell Simultaneous Diagonalization 16 / 22

 Simultaneous diagonalization of optimal solutions

Simultaneous diagonalization
Note that S ⇤ and X ⇤ commute, with both products- X ⇤ S ⇤ = S ⇤ X ⇤ = 0.
Hence, the matrices are simultaneously diagonalizable.
I n consisting of
In general, we can construct an orthonormal basis for R
three sets of vectors:
Eigenvectors of S ⇤ with positive eigenvalue that are in the
nullspace of X ⇤ . These eigenvectors comprise the columns of a
matrix Q1 .
A basis for the intersection of the nullspaces of X ⇤ and S ⇤ , which
we denote as the columns of a matrix Q̃2 , and
Eigenvectors of X ⇤ with positive eigenvalue that are in the
nullspace of S ⇤ . These eigenvectors comprise the columns of a
matrix Q̃3 .

Mitchell Simultaneous Diagonalization 17 / 22

 Simultaneous diagonalization of optimal solutions

Simultaneous diagonalization
Note that S ⇤ and X ⇤ commute, with both products X ⇤ S ⇤ = S ⇤ X ⇤ = 0.
Hence, the matrices are simultaneously diagonalizable.
I n consisting of
In general, we can construct an orthonormal basis for R
three sets of vectors:
Eigenvectors of S ⇤ with positive eigenvalue that are in the
nullspace of X ⇤ . These eigenvectors comprise the columns of a
matrix Q1 .
A basis for the intersection of the nullspaces of X ⇤ and S ⇤ , which
we denote as the columns of a matrix Q̃2 , and
Eigenvectors of X ⇤ with positive eigenvalue that are in the
nullspace of S ⇤ . These eigenvectors comprise the columns of a
matrix Q̃3 .

Mitchell Simultaneous Diagonalization 17 / 22

 Simultaneous diagonalization of optimal solutions

Simultaneous diagonalization
Note that S ⇤ and X ⇤ commute, with both products X ⇤ S ⇤ = S ⇤ X ⇤ = 0.
Hence, the matrices are simultaneously diagonalizable.
I n consisting of
In general, we can construct an orthonormal basis for R
three sets of vectors:
Eigenvectors of S ⇤ with positive eigenvalue that are in the
nullspace of X ⇤ . These eigenvectors comprise the columns of a
matrix Q1 .
A basis for the intersection of the nullspaces of X ⇤ and S ⇤ , which
we denote as the columns of a matrix Q̃2 , and
Eigenvectors of X ⇤ with positive eigenvalue that are in the
nullspace of S ⇤ . These eigenvectors comprise the columns of a
matrix Q̃3 .

Mitchell Simultaneous Diagonalization 17 / 22

 Simultaneous diagonalization of optimal solutions

Writing S ⇤ and X ⇤ using eigenvectors

We can then write

2 32 T 3
h i ⇤1 0 0 Q1
S⇤ = Q1 Q̃2 Q̃3 4 0 0 0 5 4 Q̃ T 5
2
0 0 0 Q̃3T

and 2 32 T 3
h i 0 0 0 Q1
X⇤ = Q1 Q̃2 Q̃3 4 0 0 0 5 4 Q̃ T 5
2
0 0 ⇤3 Q̃3T
where the diagonal entries in ⇤1 and ⇤3 are positive.

This is the SDP version of complementary slackness.

-

Mitchell Simultaneous Diagonalization 18 / 22

 Strict complementarity

Outline

1 SDP formulation

2 Simultaneous diagonalization

3 Simultaneous diagonalization of optimal solutions

4 Strict complementarity

Mitchell Simultaneous Diagonalization 19 / 22

 Strict complementarity

Strict complementarity for LP

In the linear programming case, there exist optimal solutions that are
strictly complementary.

The analogue of this in the SDP case is for

rank(X ⇤ ) + rank(S ⇤ ) = n,

so there are no columns in Q̃2 .

However, this does not always hold, as in the following example.

Mitchell Simultaneous Diagonalization 20 / 22

 Strict complementarity

Example I f X,,=0: XueXu--0,

The primal SDP is

[? § ¥ 0= ) X , ]= K ,
=
- 0
Xs,= X } ,
= D X ,e-O.
minX 2S3 X33
+
subject to X11 = 1.
X13 + X22 + X31 = 0←
X12 + X33 + X21 = 0
X ⌫ 0

The unique optimal primal solution is

O
2 3
1 0 0
X⇤ = 4 0 0 0 5 .
0 0 0

Mitchell Simultaneous Diagonalization 21 / 22

 Strict complementarity

Dual of examplef . c-Ey..Ai-18. %:/-g,foo!:b

The dual problem is

maxy ,S y1 2
-
y. fo,:{I-of
3
y1 y3 y2
subject to S = 4 y3 y2 0 5
y2 0 1 y3
S ⌫ 0.

The unique dual optimal solution is y ⇤ = (0, 0, 0), giving

2 3
0 0 0
S⇤ = 4 0 0 0 5 .
0 0 1

Hence rank(X ⇤ ) = rank(S ⇤ ) = 1 and rank(X ⇤ )+rank(S ⇤ ) = 2 < 3 = n.

Mitchell Simultaneous Diagonalization 22 / 22