9 views

Uploaded by Abrar Hashmi

© All Rights Reserved

- Maths Formula English Medium
- 7. Optimal Design of Water Distribution Systems
- Week 4 Solution
- 6.1 Chapter 2 Galaxy LP Problem Updated
- fcon
- Visual Math Activities
- Bertsekas convex Analysis and Optimization
- Cost Optimization of Industrial Steel Building Structures
- Wiget EPEC 2013
- Chap 002
- GAMS_Gurobi
- TB-ch02-13e-ITS
- 2631383
- Testing membership in the {0, 1/2}-closure is strongly NP-hard, even for polytopes contained in the n-dimensional 0/1-cube (2009)
- 10.1.1.154.9174.pdf
- Optimizing Hydropower Operations
- Rate Allocation
- Matlab Part 1
- Matrix Algebra Review
- 0009-2509%2874%2980070-9.pdf

You are on page 1of 54

Robert M. Freund

March, 2004

1

1 Introduction

ematical programming in the 1990s. SDP has applications in such diverse

elds as traditional convex constrained optimization, control theory, and

combinatorial optimization. Because SDP is solvable via interior point

methods, most of these applications can usually be solved very eciently

in practice as well as in theory.

LP : minimize c x

s.t. ai x = bi , i = 1, . . . , m

x n+ .

nj=1 cj xj , etc.

In fact, n+ is a closed convex cone, where K is called a closed a convex cone

if K satises the following two conditions:

K is a closed set.

2

In words, LP is the following problem:

Minimize the linear function c x, subject to the condition that x must solve

m given equations ai x = bi , i = 1, . . . , m, and that x must lie in the closed

convex cone K = n+ .

m

LD : maximize yi bi

i=1

m

s.t. yi ai + s = c

i=1

s n+ .

m

duality gap is simply cx m i=1 yi bi = (c i=1 yi ai ) x = sx 0, because

x 0 and s 0. We know from LP duality theory that so long as the pri-

mal problem LP is feasible and has bounded optimal objective value, then

the primal and the dual both attain their optima with no duality gap. That

is, there exists x and (y , s ) feasible for the primal and dual, respectively,

such that c x m

i=1 yi bi = s x = 0.

v T Xv 0 for any v n .

3

If X is an n n matrix, then X is a positive denite (pd) matrix if

n denote the set of positive denite (pd) n n symmetric matrices.

let S++

that X is symmetric and positive semidenite, and we write X Y to

denote that X Y 0. We write X 0 to denote that X is symmetric

and positive denite, etc.

n = {X S n |

X 0} is a closed convex cone in n of 2

Remark 1 S+

dimension n (n + 1)/2.

n . Pick any scalars

v T (X + W )v = v T Xv + v T W v 0,

+

n

forward to show that S+ is a closed set.

some diagonal matrix D. (Recall that Q is orthonormal means that

4

Q1 = QT , and that D is diagonal means that the o-diagonal entries

of D are all zeros.)

orthogonal eigenvectors of X, whose eigenvalues are the corresponding

diagonal entries of D.

diagonal entries of D) are all nonnegative.

diagonal entries of D) are all positive.

P v

M= ,

vT d

only if d v T P 1 v > 0.

4 Semidenite Programming

n2 components of the form (x11 , . . . , xnn ). We can also just think of X as

an object (a vector) in the space S n . All three dierent equivalent ways of

looking at X will be useful.

of X, then C(X) can be written as C X, where

5

n

n

C X := Cij Xij .

i=1 j=1

the matrix C is also symmetric. With this notation, we are now ready to

dene a semidenite program. A semidenite program (SDP ) is an opti-

mization problem of the form:

SDP : minimize C X

s.t. Ai X = bi , i = 1, . . . , m,

X 0,

Notice that in an SDP that the variable is the matrix X, but it might

be helpful to think of X as an array of n2 numbers or simply as a vector

in S n . The objective function is the linear function C X and there are m

linear equations that X must satisfy, namely Ai X = bi , i = 1, . . . , m.

The variable X also must lie in the (closed convex) cone of positive semidef-

inite symmetric matrices S+ n . Note that the data for SDP consists of the

symmetric matrix C (which is the data for the objective function) and the

m symmetric matrices A1 , . . . , Am , and the mvector b, which form the m

linear equations.

following matrices:

1 0 1 0 2 8 1 2 3

A1 = 0 3 7 , A2 = 2 6 0 , and C = 2 9 0 ,

1 7 5 8 0 4 3 0 7

6

and b1 = 11 and b2 = 19. Then the variable X will be the 3 3 symmetric

matrix:

x11 x12 x13

X = x21 x22 x23 ,

x31 x32 x33

as:

s.t.

x11 + 0x12 + 2x13 + 3x22 + 14x23 + 5x33 = 11

x11 x12 x13

X = x21 x22 x23 0.

x31 x32 x33

the standard LP constraint that x must lie in the nonnegative orthant is re-

placed by the constraint that the variable X must lie in the cone of positive

semidenite matrices. Just as x 0 states that each of the n components

7

of x must be nonnegative, it may be helpful to think of X 0 as stating

that each of the n eigenvalues of X must be nonnegative.

SDP . To see one way of doing this, suppose that (c, a1 , . . . , am , b1 , . . . , bm )

comprise the data for LP . Then dene:

ai1 0 ... 0 c1 0 ... 0

0 ai2 ... 0 0 c2 ... 0

Ai = .. .. .. .. , i = 1, . . . , m, and C = .. . .. .. .

. . . . . .. . .

0 0 . . . ain 0 0 . . . cn

SDP : minimize C X

s.t. Ai X = bi , i = 1, . . . , m,

Xij = 0, i = 1, . . . , n, j = i + 1, . . . , n,

X 0,

x1 0 ... 0

0 x2 ... 0

X = .. .. .. .. .

. . . .

0 0 ... xn

into an instance of SDP . The above construction merely shows that SDP

8

includes linear programming as a special case.

The dual problem of SDP is dened (or derived from rst principles) to be:

m

SDD : maximize yi bi

i=1

m

s.t. yi Ai + S = C

i=1

S 0.

One convenient way of thinking about this problem is as follows. Given mul-

tipliers y1 , . . . , ym , the objective is to maximize the linear function mi=1 yi bi .

The constraints of SDD state that the matrix S dened as

m

S=C yi Ai

i=1

m

C yi Ai 0.

i=1

dual problem is:

9

SDD : maximize 11y1 + 19y2

1 0 1 0 2 8 1 2 3

s.t. y1 0 3 7 + y 2 2 6 0 + S = 2 9 0

1 7 5 8 0 4 3 0 7

S 0,

s.t.

1 1y1 0y2 2 0y1 2y2 3 1y1 8y2

2 0y1 2y2 9 3y1 6y2 0 7y1 0y2 0.

3 1y1 8y2 0 7y1 0y2 7 5y1 4y2

it is presented in the format of the dual SDD, since the variables are the m

multipliers y1 , . . . , ym .

mal or dual) to any other format with great ease, and there is no loss of

generality in assuming a particular specic format for the primal or the

dual.

The following proposition states that weak duality must hold for the

primal and dual of SDP :

10

Proposition 5.1 Given a feasible solution X of SDP and a feasible solu-

tion (y, S) of SDD, the duality gap is C X m i=1 yi bi = S X 0. If

m

C X i=1 yi bi = 0, then X and (y, S) are each optimal solutions to SDP

and SDD, respectively, and furthermore, SX = 0.

trace of a matrix, dened below:

n

trace(M ) = Mjj .

j=1

identities:

trace(M N ) = trace(N M )

A B = trace(AT B)

must show that if S 0 and X 0, then S X 0. Let S = P DP T and

X = QEQT where P, Q are orthonormal matrices and D, E are nonnegative

diagonal matrices. We have:

n

= trace(DP T QEQT P ) = Djj (P T QEQT P )jj 0,

j=1

where the last inequality follows from the fact that all Djj 0 and the fact

11

that the diagonal of the symmetric positive semidenite matrix P T QEQT P

must be nonnegative.

Then from the above equalities, we have

n

Djj (P T QEQT P )jj = 0.

j=1

(P T QEQT P )jj = 0. Furthermore, the latter case implies that the j th row

of P T QEQT P is all zeros. Therefore DP T QEQT P = 0, and so SX =

P DP T QEQT = 0.

q.e.d.

Unlike the case of linear programming, we cannot assert that either SDP

or SDD will attain their respective optima, and/or that there will be no

duality gap, unless certain regularity conditions hold. One such regularity

condition which ensures that strong duality will prevail is a version of the

Slater condition, summarized in the following theorem which we will not

prove:

Theorem 5.1 Let zP and zD denote the optimal objective function values

of SDP and SDD, respectively. Suppose that there exists a feasible solution

X of SDP such that X 0, and that there exists a feasible solution (

y, S)

of SDD such that S 0. Then both SDP and SDD attain their optimal

values, and zP = zD

.

12

6 Key Properties of Linear Programming that do

not extend to SDP

programming that do not extend to SDP :

There may be a nite or innite duality gap. The primal and/or dual

may or may not attain their optima. As noted above in Theorem

5.1, both programs will attain their common optimal value if both

programs have feasible solutions in the interior of the semidenite cone.

algorithm, but it is not a nite algorithm. There is no direct analog

of a basic feasible solution for SDP .

Given rational data, the feasible region may have no rational solutions.

The optimal solution may not have rational components or rational

eigenvalues.

Given rational data whose binary encoding is size L, the norms of any

L

feasible and/or optimal solutions may exceed 22 (or worse).

Given rational data whose binary encoding is size L, the norms of any

L

feasible and/or optimal solutions may be less than 22 (or worse).

N P hard combinatorial optimization problems have convex relaxations that

are semidenite programs. In many instances, the SDP relaxation is very

tight in practice, and in certain instances in particular, the optimal solution

to the SDP relaxation can be converted to a feasible solution for the origi-

nal problem with provably good objective value. An example of the use of

SDP in combinatorial optimization is given below.

13

7.1 An SDP Relaxation of the MAX CUT Problem

Let G be an undirected graph with nodes N = {1, . . . , n}, and edge set E.

Let wij = wji be the weight on edge (i, j), for (i, j) E. We assume that

wij 0 for all (i, j) E. The MAX CUT problem is to determine a subset

S of the nodes N for which the sum of the weights of the edges that cross

from S to its complement S is maximized (where (S := N \ S) .

for j S and xj = 1 for j S. Then our formulation is:

1

n

n

M AXCU T : maximizex 4 wij (1 xi xj )

i=1 j=1

Now let

Y = xxT ,

whereby

Yij = xi xj i = 1, . . . , n, j = 1, . . . , n.

Also let W be the matrix whose (i, j)th element is wij for i = 1, . . . , n

and j = 1, . . . , n. Then MAX CUT can be equivalently formulated as:

14

1

n

n

M AXCU T : maximizeY,x 4 wij W Y

i=1 j=1

Y = xxT .

Notice in this problem that the rst set of constraints are equivalent to

Yjj = 1, j = 1, . . . , n. We therefore obtain:

1

n

n

M AXCU T : maximizeY,x 4 wij W Y

i=1 j=1

s.t. Yjj = 1, j = 1, . . . , n

Y = xxT .

Last of all, notice that the matrix Y = xxT is a symmetric rank-1 posi-

tive semidenite matrix. If we relax this condition by removing the rank-

1 restriction, we obtain the following relaxtion of MAX CUT, which is a

semidenite program:

1

n

n

RELAX : maximizeY 4 wij W Y

i=1 j=1

s.t. Yjj = 1, j = 1, . . . , n

Y 0.

CUT, i.e.,

15

M AXCU T RELAX.

As it turns out, one can also prove without too much eort that:

This is an impressive result, in that it states that the value of the semide-

nite relaxation is guaranteed to be no more than 12% higher than the value

of N P -hard problem MAX CUT.

The types of constraints that can be modeled in the SDP framework include:

linear inequalities, convex quadratic inequalities, lower bounds on matrix

norms, lower bounds on determinants of symmetric positive semidenite

matrices, lower bounds on the geometric mean of a nonnegative vector, plus

many others. Using these and other constructions, the following problems

(among many others) can be cast in the form of a semidenite program:

linear programming, optimizing a convex quadratic form subject to convex

quadratic inequality constraints, minimizing the volume of an ellipsoid that

covers a given set of points and ellipsoids, maximizing the volume of an

ellipsoid that is contained in a given polytope, plus a variety of maximum

eigenvalue and minimum eigenvalue problems. In the subsections below we

demonstrate how some important problems in convex optimization can be

re-formulated as instances of SDP .

16

8.1 SDP for Convex Quadratically Constrained Quadratic

Programming, Part I

form:

s.t. xT Qi x + qiT x + ci 0 , i = 1, . . . , m,

QCQP : minimize

x,

s.t. xT Q0 x + q0T x + c0 0

xT Qi x + qiT x + ci 0 , i = 1, . . . , m.

Qi = MiT Mi

I Mi x

0 xT Qi x + qiT x + ci 0.

xT MiT ci qiT x

17

QCQP : minimize

x,

s.t.

I M0 x

0

xT M0T c0 q0T x +

I Mi x

0 , i = 1, . . . , m.

xT MiT ci qiT x

Notice in the above formulation that the variables are and x and that

all matrix coecients are linear functions of and x.

Programming, Part II

denite program. We begin with the following elementary proposition.

1 xT

0 if and only if W xxT .

x W

alent to the following semi-denite program:

18

QCQP : minimize

x, W,

s.t.

1 T

c0 2 q0 1 xT

1 0

2 q0 Q0 x W

1 T

ci 2 qi 1 xT

1 0 , i = 1, . . . , m

2 qi Qi x W

1 xT

0.

x W

2 variables, but that

the constraints are all linear inequalities as opposed to semi-denite inequal-

ities.

in n :

19

of k points c1 , . . . , ck . We would like to nd an ellipsoid circumscribing these

k points that has minimum volume. Our problem can be written in the fol-

lowing form:

R, z

s.t. ci ER,z , i = 1, . . . , k,

M CP : minimize ln(det(R))

R, z

R 0,

and now M CP becomes:

M CP : minimize ln(det(M 2 ))

M, z

M 0.

I M ci M z

0 (ci z)T M T M (ci z) 1

(M ci M z)T 1

20

In this way we can write M CP as:

M CP : minimize 2 ln(det(M ))

M, z

I M ci M z

s.t. 0, i = 1, . . . , k,

(M ci M z)T 1

M 0.

M CP : minimize 2 ln(det(M ))

M, y

I M ci y

s.t. 0, i = 1, . . . , k,

(M ci y)T 1

M 0.

Notice that this last program involves semidenite constraints where all of

the matrix coecients are linear functions of the variables M and y. How-

ever, the objective function is not a linear function. It is possible to convert

this problem further into a genuine instance of SDP , because there is a way

to convert constraints of the form

ln(det(X))

a theoretical or a practical viewpoint, because it turns out that the function

f (X) = ln(det(X)) is extremely well-behaved and is very easy to optimize

(both in theory and in practice).

21

Finally, note that after solving the formulation of M CP above, we can

recover the matrix R and the center z of the optimal ellipsoid by computing

R = M 2 and z = M 1 y.

Recall that a given matrix R 0 and a given point z can be used to dene

an ellipsoid in n :

of k halfspaces {x | (ai )T x bi }, i = 1, . . . , k, that is,

X = {x | Ax b}

where the ith row of the matrix A consists of the entries of the vector

ai , i = 1, . . . , k. We would like to nd an ellipsoid inscribed in X of maxi-

mum volume. Our problem can be written in the following form:

R, z

s.t. ER,z X,

22

M IP : maximize det(R1 )

R, z

s.t. ER,z {x | (ai )T x bi }, i = 1, . . . , k

R 0,

M IP : maximize ln(det(R1 ))

R, z

R 0.

ith constraint is

R1 ai

x = z +

aTi R1 ai

aTi z + aTi R1 ai ,

23

M IP : maximize ln(det(R1 ))

R, z

s.t. aTi z + aTi R1 ai bi , i = 1, . . . , k

R 0.

and now M IP becomes:

M IP : maximize ln(det(M 2 ))

M, z

s.t. aTi z + aTi M T M ai bi , i = 1, . . . , k

M 0,

M IP : maximize 2 ln(det(M ))

M, z

bi aTi z 0, i = 1, . . . , k

M 0.

T 2

ai M M ai (bi ai z)

T T

(bi aTi z)I M ai

0 and

(M ai )T (bi aTi z)

bi aTi z 0.

24

In this way we can write M IP as:

M IP : minimize 2 ln(det(M ))

M, z

(bi aTi z)I M ai

s.t. 0, i = 1, . . . , k,

(M ai )T (bi aTi z)

M 0.

Notice that this last program involves semidenite constraints where all of

the matrix coecients are linear functions of the variables M and z. How-

ever, the objective function is not a linear function. It is possible, but not

necessary in practice, to convert this problem further into a genuine instance

of SDP , because there is a way to convert constraints of the form

ln(det(X))

a theoretical or a practical viewpoint, because it turns out that the function

f (X) = ln(det(X)) is extremely well-behaved and is very easy to optimize

(both in theory and in practice).

recover the matrix R of the optimal ellipsoid by computing

R = M 2 .

25

8.5 SDP for Eigenvalue Optimization

There are many types of eigenvalue optimization problems that can be for-

mualated as SDP s. A typical eigenvalue optimization problem is to create

a matrix

k

S := B wi Ai

i=1

in such a way to minimize the dierence between the largest and smallest

eigenvalue of S. This problem can be written down as:

w, S

k

s.t. S=B wi Ai ,

i=1

where min (S) and max (S) denote the smallest and the largest eigenvalue

of S, respectively. We now show how to convert this problem into an SDP .

mal matrix (i.e., Q1 = QT ) and D is a diagonal matrix consisting of the

eigenvalues of S. The conditions:

I S I

26

After premultiplying the above QT and postmultiplying Q, these conditions

become:

I D I

EOP : minimize

w, S, ,

k

s.t. S=B wi Ai

i=1

I S I.

Using constructs such as those shown above, very many other types of

eigenvalue optimization problems can be formulated as SDP s.

27

9 SDP in Control Theory

A variety of control and system problems can be cast and solved as instances

of SDP . However, this topic is beyond the scope of these notes.

a repelling force from the boundary of the feasible region. For SDP , we

need a barrier function whose values approach + as points X approach

the boundary of the semidenite cone S+n.

1 n

(possibly counting multiplicities). We can characterize the interior of the

semidenite cone as follows:

n

intS+ = {X S n | 1 (X) > 0, . . . , n (X) > 0}.

n

A natural barrier function to use to repel X from the boundary of S+

then is

n

n

ln(i (X)) = ln( i (X)) = ln(det(X)).

j=1 j=1

the positive barrier parameter :

28

BSDP () : minimize C X ln(det(X))

s.t. Ai X = bi , i = 1, . . . , m,

X 0.

Let f (X) denote the objective function of BSDP (). Then it is not too

dicult to derive:

f (X) = C X 1 , (1)

Ai X = bi , i = 1, . . . , m,

X 0, (2)

m

C X 1 = yi Ai .

i=1

can dene

S = X 1 = LT L1 ,

which implies

1 T

L SL = I,

29

and we can rewrite the Karush-Kuhn-Tucker conditions as:

Ai X = bi , i = 1, . . . , m,

X 0, X = LL

T

(3)

m

yi Ai + S = C

i=1

I 1 LT SL = 0.

From the equations of (3) it follows that if (X, y, S) is a solution of (3), then

X is feasible for SDP , (y, S) is feasible for SDD, and the resulting duality

gap is

n

n

n

n

SX = Sij Xij = (SX)jj = = n.

i=1 j=1 j=1 j=1

as 0.

However, we cannot usually solve (3) exactly, because the fourth equation

group is not linear in the variables. We will instead dene a -approximate

solution of the Karush-Kuhn-Tucker conditions (3). Before doing so, we

introduce the following norm on matrices, called the Frobenius norm:

n

n

M := M M = Mij2 .

i=1 j=1

For some important properties of the Frobenius norm, see the last subsection

30

of this section. A -approximate solution of BSDP () is dened as any

solution (X, y, S) of

Ai X = bi , i = 1, . . . , m,

X 0, X = LLT

(4)

m

yi Ai + S = C

i=1

I 1 LT SL .

S)

< 1, then X is feasible for SDP , (

y, S) is feasible for SDD, and the

duality gap satises:

m

n(1 ) C X S n(1 + ).

yi bi = X (5)

i=1

to show that S 0. To see this, dene

1 T

R=I L SL (6)

and note that R < 1. Rearranging (6), we obtain

T (I R)L

S = L 1 0

) =

S) = trace(L

trace(LL T T = trace(I R) = (n trace(R)). However,

SL)

|trace(R)| nR n, whereby we obtain

n(1 ) X

S n(1 + ).

31

q.e.d.

Based on the analysis just presented, we are motivated to develop the fol-

lowing algorithm:

(X 0 , y 0 , S 0 ) is a -approximate solution of BSDP (0 ) for some known value

of that satises < 1.

y,

Step 1. Set Current values. (X, = (X k , y k , S k ), = k .

S)

Step 2. Shrink . Set = for some (0, 1). In fact, it will be

appropriate to set

=1

+ n

the Newton step D for BSDP ( ) at X = X by factoring X =L L

T and

solving the following system of equations in the variables (D, y):

1 1 = yi Ai

m

1

C X + X DX

i=1

(7)

Ai D = 0, i = 1, . . . , m.

Denote the solution to this system by (D , y ).

+ D

X =X

32

m

S =C yi Ai

i=1

Step 5. Reset Counter and Continue. (X k+1 , y k+1 , S k+1 ) = (X , y , S ).

k+1 = . k k + 1. Go to Step 1.

=0

= 1/10

= 10

= 100

the derivation of the Newton step D and the multipliers y

33

solutions to BSDP (k ), and

Suppose that X

s.t. Ai X = bi , i = 1, . . . , m,

X 0.

f (X) = C X ln(det(X)).

1

= C X

f (X)

can be derived

and the quadratic approximation of BSDP () at X = X

as:

+ (C X

minimize f (X) + 1 X

1 ) (X X) 1 (X X) 1 (X X)

X

2

X

s.t. Ai X = bi , i = 1, . . . , m.

34

Letting D = X X,

this is equivalent to:

1 ) D + 1 X

minimize (C X 1 D X

1 D

2

D

s.t. Ai D = 0, i = 1, . . . , m.

The solution to this program will be the Newton direction. The Karush-

Kuhn-Tucker conditions for this program are necessary and sucient, and

are:

1 = yi Ai

m

1 + X

C X 1 DX

i=1

(8)

Ai D = 0, i = 1, . . . , m.

These equations are called the Normal Equations. Let D and y denote the

solution to the Normal Equations. Note in particular from the rst equation

in (8) that D must be symmetric. Suppose that (D , y ) is the (unique) so-

lution of the Normal Equations (8). We obtain the new value of the primal

variable X by taking the Newton step, i.e.,

+ D .

X =X

We can produce new values of the dual variables (y, S) by setting the new

m

value of y to be y and by setting S = C y Ai . Using (8), then, we

i=1

have that

S = X 1 D X

1 X 1 . (9)

35

We have the following very powerful convergence theorem which demon-

strates the quadratic convergence of Newtons method for this problem,

with an explicit guarantee of the range in which quadratic convergence takes

place.

Suppose that (X, y, is a -approximate solution of BSDP () and < 1.

S)

Let (D , y ) be the solution to the Normal Equations (8), and let

+ D

X =X

and

S = X 1 D X

1 X 1 .

Then (X , y , S ) is a 2 -approximate solution of BSDP ().

satises:

Proof: Our current point X

Ai X

= bi , i = 1, . . . , m, X

=L

LT 0

m

yi Ai + S = C

i=1

1 T

I L SL < 1.

Furthermore the Newton direction D and multipliers y satisfy:

Ai D = 0, i = 1, . . . , m

m

yi Ai + S = C

i=1

X =X + D = L(I +L 1 D L

T )L

T

S = X 1 D X

1 X 1 = L T (I L 1 D L

T )L

1 .

We will rst show that L 1 D L

T . It turns out that this is the

crucial fact from which everything will follow nicely. To prove this, note

that

m

m

m

yi Ai + S = C = yi Ai + S = yi Ai + L 1 D L

T (I L T )L

1 .

36

Taking the inner product with D yields:

1 D L

T L

S D = L 1 D L

T L 1 D ,

T L

1 D L

L

T SL

L 1 D L

T = I L 1 D L

T L 1 D L

T L T ,

L T 2 = I 1 L

1 D L 1 D L

L

T SL T .

Invoking the Cauchy-Schwartz inequality we obtain:

L T 2 I 1 L

1 D L 1 D L

T SLL 1 D L

T L T ,

1 D L

from which we see that L T .

+L

X = L(I 1 D L

T )L

T 0

and

S = L 1 D L

T (I L T )L

1 0,

1 D L

since L 1 D L

T < 1, which guarantees that I L T 0.

Next, factorize

1 D L

I +L T = M 2 ,

T = L (L )T

ML

X = LM

. Then note that

where we dene L = LM

1 1 T

I (L )T S L = I M L S LM

1 T T 1 D L

T )L

1 )LM 1 D L

T )M

= I ML (L (I L = I M (I L

37

1 D L

= IM M +M (L T )M = IM M +M (M M I)M = (IM M )(IM M )

1 D L

= (L 1 D L

T )(L T ).

1

1 D L 1 D L 1 D L

I (L )T S L = (L T )(L T ) (L T )2 2 .

This shows that (X , y , S ) is a 2 -approximate solution of BSDP ().

q.e.d.

S)

approximate solution of BSDP () and < 1. Let

=1

+ n

and let = . Then (X, y, is a -approximate solution of BSDP ( ).

S)

S) satises Ai X 0, and

= bi , i = 1, . . . , m, X

m

yi Ai + S = C, and so it remains to show that

i=1

1 T

L I ,

SL

= LL

where X T . We have

1 T

L I = 1 L

SL I = 1 1 L

T SL I 1 1 I

T SL

1 1 T 1

I

L SL I +

1 + n

+ n = n

= + n n = .

38

q.e.d.

-approximate solution of BSDP (0 ) and < 1. Then for all k = 1, 2, 3, ...,

(X k , y k , S k ) is a -approximate solution of BSDP (k ).

Proof: By induction, suppose that the theorem is true for iterates 0, 1, 2, ..., k.

From the Relaxation Theorem, (X k , y k , S k ) is a -approximate solution

of BSDP (k+1 ) where k+1 = k .

solution of BSDP (k+1 ).

q.e.d.

= 14 -approximate solution of BSDP (0 ). In order to obtain primal and

dual feasible solutions (X k , y k , S k ) with a duality gap of at most , one needs

to run the algorithm for at most

1.25 X 0 S 0

k = 6 n ln

0.75

iterations.

39

= 80

x^

~

x = 90

_

x = 100

40

1

1 1 1

=1 = 1 2 4 = 1 1 .

+ n 1 2+4 n 6 n

+ n 2

Therefore k

1

1

k

0 .

6 n

This implies that

m k

1

C X k

bi yik = X S n(1 + ) 1

k k k

(1.25n0 )

i=1

6 n

k

1 X 0 S0

1 (1.25n) ,

6 n 0.75n

from (5). Taking logarithms, we obtain

m

1 1.25 0

ln C X k

bi yik k ln 1 + ln X S0

i=1

6 n 0.75

k 1.25 0

+ ln X S0

6 n 0.75

1.25 X 0 S 0 1.25 0

ln + ln X S 0 = ln().

0.75 0.75

m

Therefore C X k bi yik .

i=1

q.e.d.

that is a -approximate solution of the problem BSDP (0 ). In this subsec-

tion, we show how to obtain such a starting point, given a positive denite

feasible solution X 0 of SDP .

41

We suppose that we are given a target value 0 of the barrier param-

eter, and we are given X = X 0 that is feasible for BSDP (0 ), that is,

Ai X 0 = bi , i = 1, . . . , m, and X 0 0. We will attempt to approximately

solve BSDP (0 ) starting at X = X 0 , using the Newton direction at each

iteration. The formal statement of the algorithm is as follows:

Ai X 0 = bi , i = 1, . . . , m, X 0 0.

= X k . Factor X LT .

Newton step D for BSDP (0 ) at X = X by solving the following system

of equations in the variables (D, y):

1 = yi Ai

m

0 1 0 1

C X + X DX

i=1

(10)

Ai D = 0, i = 1, . . . , m.

Denote the solution to this system by (D , y ). Set

m

S =C yi Ai .

i=1

T 1 , stop. In this

4

is a 1 -approximate solution of BSDP (0 ), along with the dual val-

case, X 4

ues (y , S ).

Step 4. Update Primal Point.

+ D

X =X

42

where

0.2

= T .

L D L

1

+ D ).

Alternatively, can be computed by a line-search of f0 (X

Step 5. Reset Counter and Continue. X k+1 X , k k + 1.

Go to Step 1.

Proposition 10.1 Suppose that (D , y ) is the solution of the Normal equa-

for the given value 0 of the barrier parameter,

tions (10) for the point X

and that

T 1 .

1 D L

L

4

1

is a -approximate solution of BSDP (0 ).

Then X 4

m

Proof: We must exhibit values (y, S) that satisfy yi Ai + S = C and

i=1

I 1 L 1 .

T SL

0 4

m

Let (D , y ) solve the Normal equations (10), and let S = C yi Ai . Then

i=1

we have from (10) that

1 T 1 T 0 T 1 T 1 T 1 1 D L

T ,

I 0

L S L = I 0 L ( (L L L L D L L ))L = L

whereby

1 T T 1 .

1 D L

I 0

L S L = L

4

q.e.d.

Step 4, then the objective function f0 (X) decreases by at least 0.0250 :

43

Proposition 10.2 Suppose that X satises Ai X = bi , i = 1, . . . , m, and

X 0. Suppose that (D , y ) is the solution of the Normal equations (10)

for a given value 0 of the barrier parameter, and that

for the point X

T > 1 .

1 D L

L

4

Then for all [0, 1),

1 D L

+ 0 L T + 2

f0 +

X 1 T D f0 (X) .

L D L

2(1 )

In particular,

0.2

0.0250 .

+

f0 X 1 T D f0 (X) (11)

L D L

In order to prove this proposition, we will need two powerful facts about the

logarithm function:

x2

ln(1 + x) x .

2(1 )

Proof: We have:

x2 x3 x4

ln(1 + x) = x 2 + 3 4 + ...

x 2 3 4 ...

x 2 2 2 ...

x2

!

= x 2 1 + |x| + |x|2 + |x|3 + . . .

x2

= x 2(1|x|)

x2

x 2(1) .

44

q.e.d.

2

ln(det(I + R)) I R .

2(1 )

n

2 . We

matrix of the eigenvalues of R. Then rst note that R = Djj

j=1

then have:

ln(det(I + R)) = ln(det(I + QDQT ))

= ln(det(I + D))

n

= ln(1 + Djj )

j=1

n 2

Djj

Djj 2(1)

j=1

R2

= I D 2(1)

2

I QT RQ 2(1)

2

= I R 2(1)

q.e.d.

= T ,

1 D L

L

1 D L

L T = .

45

Then

+

f0 X + D

T D

1 D L

L

= f0 X

+ C D 0 ln(det(L(I

= C X 1 D L

+ L T )L

T ))

0 ln(det(X))

= C X 1 D L

+ C D 0 ln(det(I + L T ))

+ C D 0 I L

f0 (X) T + 0 2

1 D L

2(1)

+ C D 0 L

= f0 (X) 1 D + 0 2

T L

2(1)

! 2

+ C 0 X

= f0 (X) 1 D + 0 .

2(1)

Now, (D , y ) solve the Normal equations:

0 1 1 = y A

m

0 1

C X + X D X

i i

i=1

(12)

Ai D = 0, i = 1, . . . , m.

Taking the inner product of both sides of the rst equation above with D

and rearranging yields:

1 D X

0 X 1 D = (C 0 X

1 ) D .

46

+

f0 X

f0 (X) 1 D + 0 2

1 D X

0 X

L T D

1 D L 2(1)

1 D L

0 L

= f0 (X) T L T + 0 2

1 D L

2(1)

0 L

= f0 (X) T 2 + 0 2

1 D L

2(1)

0 L

= f0 (X) T + 0 2

1 D L

2(1)

1 D L

+ 0 L T + 2

= f0 (X) 2(1) .

1 D L

Subsituting = 0.2 and and L T > 1

yields the nal result.

4

q.e.d.

Last of all, we prove a bound on the number of iterations that the algo-

rithm will need in order to nd a 14 -approximate solution of BSDP (0 ):

X 0 0. Let 0 be given and let f0 be the optimal objective function value

of BSDP (0 ). Then the algorithm initiated at X 0 will nd a 14 -approximate

solution of BSDP (0 ) in at most

f0 (X 0 ) f0

k=

0.0250

iterations.

Proof: This follows immediately from (11). Each iteration that is not a 14 -

approximate solution decreases the objective function f0 (X) of BSDP (0 )

by at least 0.0250 . Therefore, there cannot be more than

f0 (X 0 ) f0

0.0250

47

iterations that are not 14 -approximate solutions of BSDP (0 ).

q.e.d.

"

n

1. M = (j (M ))2 , where 1 (M ), 2 (M ), . . . , n (M ) is an enu-

j=1

meration of the n eigenvalues of M .

3. |trace(M )| nM .

4. If M < 1, then I + M 0.

diagonal matrix of the eigenvalues of M . Then

M = M M = QDQT QDQT = trace(QDQT QDQT )

n

= trace(QT QDQT QD) = trace(DD) = (j (M ))2 .

j=1

This proves the rst two assertions. To prove the third assertion, note that

n

n

= trace(D) = j (M ) n (j (M ))2 = nM .

j=1 j=1

To prove the fourth assertion, let be an eigenvalue of I + M . Then

= 1 + where is an eigenvalue of M . However, from the second asser-

tion, = 1 + 1 M > 0, and so M 0.

q.e.d.

48

Proposition 10.5 If A, B S n , then AB AB.

Proof: We have

n 2

n

n

AB = Aik Bkj

i=1 j=1 k=1

n n

n

n

2

Aik 2

Bkj

i=1 j=1 k=1 k=1

n n n n

2 2

= Aik Bkj = AB.

i=1 k=1 j=1 k=1

q.e.d.

gorithm

assume for convenience that the format of the problem is that of the dual

problem SDD. Then the feasible region of the problem can be written as:

# $

m

F = (y1 , . . . , ym ) m

|C yi Ai 0 ,

i=1

m

and the objective function is i=1 yi bi . Note that F is just a convex re-

gion in m .

Recall that at any iteration of the ellipsoid algorithm, the set of solutions

of SDD is known to lie in the current ellipsoid, and the center of the current

49

y1 , . . . , ym ). If y F , then we perform an optimality

ellipsoid is, say, y = (

m

cut of the form m i=1 i bi

y i=1 y

i bi , and use standard formulas to update

the ellipsoid and its new center. If y / F , then we perform a feasibility cut

by computing a vector h m such that h T y > h

T y for all y F .

There are four issues that must be resolved in order to implement the

above version of the ellipsoid algorithm to solve SDD:

m

i Ai . If S 0, then y F . Testing if the matrix S 0 can

i=1 y

be done by computing an exact Cholesky factorization of S, which

takes O(n3 ) operations, assuming that computing square roots can be

performed (exactly) in one operation.

puted in O(n3 ) operations. If S 0, then again assuming exact arith-

metic, we can nd an n-vector v such that vT S v < 0 in O(n3 ) oper-

ations as well. Then the feasiblity cut vector h is computed by the

formula:

i = vT Ai v,

h i = 1, . . . , m,

y F , that m

m

T T T

y h= yi v Ai v = v yi Ai v

i=1 i=1

m

m

vT C v < vT yi Ai v =

yi vT Ai v = yT h,

i=1 i=1

indeed provides a feasibility cut for F at y = y.

thereby showing h

R on the distance of some optimal solution y from the origin. This

50

cannot be done by examining the input length of the data, as is the case

in linear programming. One needs to know some special information

about the specic problem at hand in order to determine R before

solving the semidenite program.

solution of SDD. In order to prove a complexity bound on the number

of iterations needed to nd an -optimal solution, we need to know

beforehand the radius r of a Euclidean ball that is contained in the

set of -optimal solutions of SDD. The value of r also cannot be

determined by examining the input length of the data, as is the case

in linear programming. One needs to know some special information

about the specic problem at hand in order to determine r before

solving the semidenite program.

nonlinear (convex) programming, in combinatorial optimization, and in con-

trol theory. In the area of convex analysis, recent research topics include

the geometry and the boundary structure of SDP feasible regions (including

notions of degeneracy) and research related to the computational complex-

ity of SDP such as decidability questions, certicates of infeasibility, and

duality theory. In the area of combinatorial optimization, there has been

much research on the practical and the theoretical use of SDP relaxations of

hard combinatorial optimization problems. As regards interior point meth-

ods, there are a host of research issues, mostly involving the development

of dierent interior point algorithms and their properties, including rates of

convergence, performance guarantees, etc.

Because SDP has so many applications, and because interior point methods

show so much promise, perhaps the most exciting area of research on SDP

has to do with computation and implementation of interior point algorithms

51

for solving SDP . Much research has focused on the practical eciency of

interior point methods for SDP . However, in the research to date, com-

putational issues have arisen that are much more complex than those for

linear programming, and these computational issues are only beginning to

be well-understood. They probably stem from a variety of factors, including

the fact that SDP is not guaranteed to have strictly complementary optimal

solutions (as is the case in linear programming). Finally, because SDP is

such a new eld, there is no representative suite of practical problems on

which to test algorithms, i.e., there is no equivalent version of the netlib

suite of industrial linear programming problems.

http://www.zib.de/helmberg/semidef.html.

14 Exercises

n

1. For a (square) matrix M IRnn , dene trace(M ) = Mjj , and for

j=1

two matrices A, B IRkl dene

k

l

A B := Aij Bij .

i=1 j=1

Prove that:

(b) trace(M N ) = trace(N M ).

kk

2. Let S+ denote the cone of positive semi-denite symmetric matrices,

namely S+kk

= {X S kk | v T Xv

0 for all v n }. Considering

kk kk nn kk

S+ as a cone, prove that S+ = S+ , thus showing that S+

is self-dual.

52

3. Consider the problem:

(P) : minimized dT Qd

s.t. dT M d = 1 .

(S) : minimizeX QX

s.t. M X =1

X 0 .

4. Prove that

X xxT

if and only if

X x

0.

xT 1

K := {S S kk | S X 0 for all X 0} .

Prove that K = K.

6. Let min denote the smallest eigenvalue of the symmetric matrix Q.

Show that the following three optimization problems each have optimal

objective function value equal to min .

(P1) : minimized dT Qd

s.t. dT Id = 1 .

(P2) : maximize

s.t. Q I .

53

(P3) : minimizeX QX

s.t. I X =1

X0.

xT Qx + q T x + c 0

1 T

c 2 q

1 xT 1 xT

1 0 and 0.

2q Q x W x W

A B

M= ,

BT C

where A, B are symmetric matrices and A is nonsingular. Prove that

M 0 if and only if C B T A1 B 0.

54

- Maths Formula English MediumUploaded byAmolKore-Mali
- 7. Optimal Design of Water Distribution SystemsUploaded byDžana Kadrić
- Week 4 SolutionUploaded byAditya Upgade
- 6.1 Chapter 2 Galaxy LP Problem UpdatedUploaded byDharmendra Prajapati
- fconUploaded byRobin Chadha
- Visual Math ActivitiesUploaded byMSHYDERABAD4334
- Bertsekas convex Analysis and OptimizationUploaded bymahdikhd
- Cost Optimization of Industrial Steel Building StructuresUploaded bykvsamath
- Wiget EPEC 2013Uploaded byfpttmm
- Chap 002Uploaded byMar Jinita
- GAMS_GurobiUploaded byErnesto Sresearch
- TB-ch02-13e-ITSUploaded byMayuresh Kulkarni
- 2631383Uploaded byNoman Khalid
- Testing membership in the {0, 1/2}-closure is strongly NP-hard, even for polytopes contained in the n-dimensional 0/1-cube (2009)Uploaded bySebastian Pokutta
- 10.1.1.154.9174.pdfUploaded byUtsav
- Optimizing Hydropower OperationsUploaded bySujeet Kumar
- Rate AllocationUploaded byMukesh Lavan
- Matlab Part 1Uploaded byAhmadA.Alnoufi
- Matrix Algebra ReviewUploaded byrotero_pujol
- 0009-2509%2874%2980070-9.pdfUploaded byaghilif
- Upper Bound Limit Analysis by FEMUploaded byMelkamu Demewez
- Bisbos 2008 Engineering-StructuresUploaded byAndreea Nan
- Semidef Prog 2Uploaded bySérgio Monteiro
- ch02Uploaded bythemoonzone
- Modeling and Control of Co-Generation Power Plants IEEE[1]Uploaded bykruisengruven
- gloverUploaded byAnup Dugane
- 1-s2.0-S0377221708005171-main.pdfUploaded byRaju Gumma
- 67985_02Uploaded byuranub
- Petrinet And Dynamic ProgrammingUploaded byDinesh Zanwar
- 1403.4545v2sdvUploaded byAragornn Elessarr

- Course Notes Advanced DSPUploaded byAnonymous hDKqasf
- Filling-the-Missing-Gaps-in-the-Indus-Water-Treaty.pdfUploaded byAbrar Hashmi
- Indus-water-energy-text-0426.docxUploaded byAbrar Hashmi
- GAMS2Uploaded byAbrar Hashmi
- 13 Generalized Programming and Subgradient Optimization (PDF)Uploaded byAbrar Hashmi
- Learn GamsUploaded byDavid Perez
- gdxusageUploaded byAbrar Hashmi
- Guide to the Indus Basin Model Revised 1990Uploaded byAbrar Hashmi
- Lethanh2011 Coursereportsimulationofacommunicationssystemusingmatlab 130814204832 Phpapp02Uploaded byAbrar Hashmi
- matlabppt-120624045322-phpapp02Uploaded byAbrar Hashmi
- Turbo 2Uploaded bytheomen3007
- CdmaUploaded byapi-3706414
- Inter Symbol Interference (ISI) and Raised cosine filtering.pdfUploaded byAbrar Hashmi
- Filters Tutorial 2709 EnUploaded byAbrar Hashmi
- Lec1 Unconstr OptUploaded byBittu Kri
- 10 Analysis of Convex Sets (PDF)Uploaded byAbrar Hashmi
- Steepest Descent AlgorithmUploaded byশেখ ফয়সাল কবির
- Chp 1curve FittingUploaded byAbrar Hashmi

- Atomic TheoryUploaded byCardoza Ryan
- CH2 Rig Components (Calculation) 6 JuneUploaded byk yong
- Method Validation on Drug SubstancesUploaded bySrikanth Nagabiru
- CVNG 2005 May 2009 Shrivastava and Chadwick ModifiedUploaded byrajal11
- Number of Non Negative Integral Solutions of an EqautionUploaded byVamsi Karthik
- Unit III Thick Cylinders- e Notes -PVRaoUploaded bykumaravel07
- CVPR11_FaceRankOrderUploaded by.adtmmal
- Problem for the Buried Virtual Anchor Model - Intergraph CADWorx & AnalysisUploaded byChris Echavari
- Sika Lastic Fleece-120Uploaded byJulian Asis Garcia
- EXP9 CHE150 1L BenjaminUploaded byRalph Carlo Evidente
- Matrices wUploaded byNur Syahadah
- One-dimensional Consolidation - OpenSeesWikiUploaded bysipil123
- Semyon Davidovich KirlianUploaded byRishikhant
- ANSI API RECOMMENDED PRACTICE 2EQUploaded byshimataxo
- ChemistryUploaded byOyedeji
- Drying ReviewerUploaded byConrad Monterola
- Specimen QP - C1 EdexcelUploaded bydensity
- Math4_5_2018.pdfUploaded by67409b5322
- The Reality of ExistenceUploaded bygravitywriting
- Diffraction of LightUploaded bySingh Pushparaj
- Impes MethodUploaded byVishal balar
- Gas Turbine ModellingUploaded byscribdkhatn
- 11_3_YM_Wire_PracUploaded byaby251188
- Theoretical Calculation of Stiffness of Alveolar Wall(Revised)Uploaded byjohnjabaraj
- Thermo RefrigerationUploaded byKailash Nair
- Acrylamide in Food Mechanism of Formation and Influencing Factors During Heating of FoodUploaded byKatherineGlick
- QMIN13-GLM_MultivariateUploaded byEmad R. H. Issak
- Visualizing SemenUploaded byGreg McLaughlin
- is.3722.1.1983 Letter symbols and signs used in electrical technology, Part 1 General guidance on symbols and subscripts.pdfUploaded byMukesh Kumar
- Significance of Transposition for 220kV TowerUploaded bysubamanivel