Determinant maximization with linear

matrix inequality constraints

S. Boyd, L. Vandenberghe, S.-P. Wu
Information Systems Laboratory
Stanford University

SCCM Seminar 5 February 1996

1
 MAXDET problem denition

minimize cT x + log det G(x),1

subject to G(x) =
G + x G +


+ x G > 0
0 1 1 m m
F (x) =
F0 + x1F1 +


+ xmFm  0

{ x 2 Rm is variable
{ Gi = GTi 2 Rll , Fi = FiT 2 Rnn
{ F (x)  0, G(x) > 0 called linear matrix inequalities

{ looks specialized, but includes wide variety of convex

optimization problems
{ convex problem
{ tractable, in theory and practice
{ useful duality theory

2
 Outline

1. examples of MAXDET probems

2. duality theory

3. interior-point methods

3
 Special cases of MAXDET
semidenite program (SDP)
minimize cT x
subject to F (x) = F0 + x1F1 +


+ xmFm  0

,c
xopt r

F (x) 6 0 F (x)  0

LMI can represent many convex constraints

linear inequalities, convex quadratic inequalities, matrix
norm constraints, . . .
linear program
minimize cT x
subject to aTi x  bi; i = 1; : : : ; n
SDP with F (x) = diag (b , Ax)

4
analytic center of LMI
minimize log det F (x),1

F + x F +


+ x F > 0
subject to F (x) = 0 1 1 m m

{ log det F (x),1 smooth, convex on fx j F (x) > 0g

{ optimal point xac maximizes det F (x)
{ xac called analytic center of LMI F (x) > 0

r
xac

5
Minimum volume ellipsoid around points
nd min vol ellipsoid containing points x1, . . . , xK 2 Rn

ellipsoid E = fx j kAx , bk  1g
{ center A,1b
{ A = AT > 0, volume proportional to det A,1

minimize log det A,1

subject to A = AT > 0
kAxi , bk  1; i = 1; : : : ; K
convex optimization problem in A, b
(n + n(n + 1)=2 vars)
express constraints as LMI
2
I Axi , b 377
kAxi , bk  1 () 66
64 75 0
(Axi , b)T 1
6
 Maximum volume ellipsoid in polytope
nd max vol ellips. in P = fx j aTi x  bi; i = 1; : : : ; Lg

P @
R
@

s
d
E

ellipsoid E = fBy + d j kyk  1g

{ center d
{ B = B T > 0, volume proportional to det B

E  P () aTi (By + d)  bi for all kyk  1

() sup aTi By + aTi d  bi
kyk1
() kBaik + aTi d  bi; i = 1; : : : ; L
convex constraint in B and d
7
maximum volume E  P

formulation as convex problem in variables B , d:

maximize log det B
subject to B = B T > 0
kBaik + aTi d  bi; i = 1; : : : ; L

express constraints as LMI in B , d

2 3
66 (bi , ai d)I Bai 77
T
kBaik + ai d  bi () 4
T 6 75  0
(Bai) bi , ai d
T T

hence, formulation as MAXDET-problem

minimize log det B ,1
subject to B > 0
2 3
66 i
64
( b , a T d)I
i Ba i 77
75  0; i = 1; : : : ; L
(Bai) bi , ai d
T T

8
 Experiment design
estimate x from measurements
yk = aTk x + wk ; i = 1; : : : ; N
{ ak 2 fv1; : : : ; vmg, vi given test vectors
{ wk IID N (0; 1) measurement noise
{ i = fraction of ak 's equal to vi
{ N m
0 1,1
N
X T CA X N
LS estimator: xc = k=1 a a i=1 yk ak
B
@ k k
error covariance
0 1,1
E(xc , x)(xc , x)T = N B@i=1 iviviT CA = N1 E ()
1 m
X

optimal experiment design: choose i

m
i  0; i=1 i = 1;
X

that make E () `small'

{ minimize max (E ()) (E -optimality)
{ minimize Tr E () (A-optimality)
{ minimize det E () (D-optimality)
all are MAXDET problems
9
D-optimal design
0 1,1
m
X
minimize log det @
B
i=1
 v v TC
i i iA
subject to i  0; i = 1; : : : ; m
m
X
i=1
 i=1
m
X
i=1
 ivivi > 0
T

can add other convex constraints, e.g.,

{ bounds on cost or time of measurements:
cTi   bi

{ no more than 80% of the measurements is

concentrated in less than 20% of the test vectors
bm=
X5c
i=1
[i]  0:8
([i] is ith largest component of )

10
 Positive denite matrix completion

matrix A = AT
{ entries Aij , (i; j ) 2 N are xed
{ entries Aij , (i; j ) 62 N are free

positive denite completion

choose free entries such that A > 0 (if possible)

maximum entropy completion

maximize log det A
subject to A > 0
property: (A,1)ij = 0 for i; j 62 N
,1
(since @ log@AdetijA = ,(A,1 )ij )

11
 Moment problem
there exists a probability distribution on R such that
i = Eti; i = 1; : : : ; 2n
if and only if
2 3
66 1  1 : : :  n,1  n 77
66
66 1 2 : : : n n+1 77
77
H () = 66
66
.. .. .. .. 777 0
66 77
66
4
n,1 n : : : 2n,2  2n,1 7775
n n+1 : : : 2n,1 2n
LMI in variables i
hence, can solve
maximize/minimize E(c0 + c1t +


+ c2nt2n)
subject to i  Eti  i; i = 1; : : : ; 2n
over all probability distributions on R by solving SDP
maximize/minimize c0 + c11 +


+ c2n2n
subject to i  i  i; i = 1; : : : ; 2n
H (1; : : : ; 2n)  0
12
 Other applications

{ maximizing products of positive concave functions

{ minimum volume ellipsoid covering union or sum of

ellipsoids

{ maximum volume ellipsoid in intersection or sum of

ellipsoids

{ computing channel capacity in information theory

{ maximum likelihood estimation

13
 MAXDET duality theory
primal MAXDET problem
minimize cT x + log det G(x),1
subject to G(x) = G0 + x1G1 +


+ xmGm > 0
F (x) = F0 + x1F1 +


+ xmFm  0
optimal value p?

dual MAXDET problem

maximize log det W , Tr G0W , Tr F0Z + l
subject to Tr FiZ + Tr GiW = ci; i = 1; : : : ; m
W > 0; Z  0
variables W = W T 2 Rll , Z = Z T 2 Rnn
optimal value d?

properties
{ p?  d? (always)
{ p? = d? (usually)

denition
duality gap = primal objective , dual objective
14
 Example: experiment design

primal problem
0 1,1
m
X
minimize log det @
B
i=1
 v vTC
i i iA
m
X
subject to
i=1
i = 1
i  0; i = 1; : : : ; m
m
X
i=1
 ivivi > 0
T

dual problem

maximize log det W

subject to W = W T > 0
viT Wvi  1; i = 1; : : : ; m

interpretation: W determines smallest ellipsoid with

center at the origin and containing vi, i = 1; : : : ; m

15
 Central path: general

general convex optimization problem

minimize f0(x)
subject to x 2 C
f0; C convex

' is barrier function for C

{ smooth, convex
{ '(x) ! 1 as x(2 int C ) ! @C

central path
x?(t) = argmin (tf0(x) + '(x)) for t > 0
x2C

16
 Central path: MAXDET problem

f0(x) = cT x + log det G(x),1

C = fx j F (x)  0g
barrier function for LMI F (x)  0
8
< log det F (x),1 if F (x) > 0
>
'(x) = :+
> 1 otherwise

MAXDET central path: x?(t) = argmin '(t; x), with

F (x) > 0
G(x) > 0
 T 
'(t; x) = t c x + log det G(x) + log det F (x),1
,1

example: SDP
t=0 t=1


,c
 
 
 
 r




xac


cT x = p?
r

17
 Path-following for MAXDET
properties of MAXDET central path
{ from x?(t), get dual feasible Z ?(t), W ?(t)
{ corresponding duality gap is n=t
{ x?(t) ! optimal as t ! 1

path-following algorithm
given strictly feasible x, t  1
repeat
1. compute x?(t) using Newton's method
2. x := x?(t)
3. increase t
until n=t < tol

tradeo: large increase in t means

{ fast gap reduction (fewer outer iterations), but
{ many Newton steps to compute x?(t+)
(more Newton steps per outer iteration)
18
 Complexity of Newton's method
(Nesterov & Nemirovsky, late 1980s)
for self-concordant functions
denition: along a line
jf 000(t)j  Kf 00(t)3=2
Example: (K = 2)
'(t; x) = t(cT x+log det G(x),1 )+log det F (x),1 (t  1)
complexity of Newton's method
{ theorem: #Newton steps to minimize '(t; x),
starting from x(0):
#steps  10:7('(t; x(0)) , '?(t)) + 5
{ empirically: #steps  ('(t; x(0)) , '?(t)) + 3
30

25
# Newton steps

20

15

10

0
0 5 10 15 20 25 30

'(t; x (0)
) , ' (t)
?

19
 Path-following algorithm

idea: choose t+, starting point xc for Newton alg. s.t.

'(t+; xc) , '?(t+) =
(bounds # Newton steps required to compute x?(t+))

in practice: use lower bound from duality

'(t+; xc) , '?(t+)  '(t+; xc) + log det Z ,1 
+ t log det W ,1 + Tr G0W + Tr F0Z , l
= '(t+; xc) + function of W; Z

20
two extreme choices
 r 
{ xed reduction: x = x (t), t = 1 + 2 =n t
c ? +

{ predictor step along tangent of central path

x ? (t+ )
x (t)
? x? x?

xb
x?(t+ )
x?(t)

0
= 10 0
= 50
xed reduction xed reduction
10 10

−1 −1
10 10

−2 −2
10 10
duality gap

duality gap

−3 −3
10 10

−4 −4
10 10

−5 −5
10 10

10
−6
predictor 10
−6

10
−7
10
−7
predictor
−8 −8
10 10

−9 −9
10 10
0 5 10 15 20 25 30 35 40 0 5 10 15 20 25 30 35 40

Newton iterations Newton iterations

21
 Total complexity
total number of Newton steps
p
{ upper bound: O ( n log(1=))
{ practice, xed-reduction method: O ( n log(1=))
p
{ practice, with predictor steps: O (log(1=))
50

45
Newton iterations

40

35 xed reduction
30

25

20

15

10 predictor steps
5

n
0
0 5 10 15 20 25 30 35 40 45 50

50 50

45 45
Newton iterations

Newton iterations

40 40

35 35

30 30

25 25

20 20

15 15

10 10

5 5

l m
0 0
0 5 10 15 20 25 30 35 40 45 50 0 5 10 15 20 25 30 35 40 45 50

one Newton step involves a least-squares problem

2 2
minimize F~ (v) F + G~ (v) F

22
 Conclusion

MAXDET-problem
minimize cT x + log det G(x),1
subject to G(x) > 0; F (x)  0

arises in many dierent areas

{ includes SDP, LP, convex QCQP
{ geometrical problems involving ellipsoids
{ experiment design, max. likelihood estimation,
channel capacity, . . .

convex, hence can be solved very eciently

software/paper available on ftp soon (anonymous ftp to

isl.stanford.edu in /pub/boyd/maxdet)

23