Convex Optimization and System Theory

Kees Roos/A.A. Stoorvogel

e-mail: C.Roos@ewi.tudelft.nl
URL: http://www.isa.ewi.tudelft.nl/roos
WI 4218
February 8 - May 31, A.D. 2006
Optimization Group 1
Outline for week 3
Proof of the Conic Duality Theorem
Conic duality results
Pathological examples
Consequences of Conic Duality Theorem
Conic Theorem on Alternative
Robust feasibility and robust boundedness
Robust Conic Duality Theorem
Optimization Group 2
Proof of the Conic Duality Theorem (1)
(P): c
T
x min | Ax b
K
0
(D): b
T
y max | A
T
y = c, y
K

0
A. Symmetry and Weak duality were already established.
B. To prove Strong duality means to verify that
B.1. If one of the problems (P), (D) is strictly feasible and bounded, then the
other problem is solvable, and the optimal values in (P) and (D) are equal to each
other.
B.2. If both (P), (D) are strictly feasible, then both problems are solvable with
equal optimal values.
C. B.2 follows from B.1: by Weak duality, if both problems are feasible, both are bounded.
D. By symmetry, it suces to prove B.1 in the case when the strictly feasible and bounded
problem is (P).
Optimization Group 3
Proof of the Conic Duality Theorem (1)
It suces to prove that if (P) is strictly feasible and bounded then there exists a dual feasible
y

with
b
T
y

= Opt(P)
Note that Opt(P), the optimal value in (P), exists because (P) is bounded. Indeed, by Weak
duality the dual objective at a dual feasible solution is Opt(P), so that the existence of
the above y

will imply that y

is dual optimal and that Opt(P) = Opt(D).

There is nothing to prove if c = 0 then Opt(P) = 0 and y

= 0 is dual feasible,
whereas b
T
y

= 0 = Opt(P).
We therefore assume that c = 0.
Optimization Group 4
Proof of the Conic Duality Theorem (2)
1
0
. Dene
M = {y = Ax b | x R
n
, c
T
x Opt(P)} R
m
Observe that
a) M is a nonempty convex set (since c = 0 for any number z there exists x R
n
such
that c
T
x z).
b) M does not intersect the interior int K of the cone K (because c
T
x > Opt(P) for any
x int K).
2
0
. By the Separation Theorem, the non-intersecting nonempty convex sets M and int K
can be separated by a nontrivial linear form:
= 0 : sup
yM

T
y inf
uint K

T
u
Optimization Group 5
Proof of the Conic Duality Theorem (3)
M = {y = Ax b | x R
n
, c
T
x Opt(P)};
() : sup
yM

T
y inf
uint K

T
u [ = 0]
Observe that the right hand side in (*), being bounded below, is nite, and since K is a cone
must be 0, i.e.,
inf
uint K

T
u = inf
uK

T
u = 0.
Hence
T
u 0 for each u K. This means K

.
The left hand side in (*) is
sup
x:c
T
xOpt(P)
_
(A
T
)
T
x
T
b
_
;
since it is nite, A
T
should be proportional, with nonnegative coecient, to c :
0 : A
T
= c.
Summary:
= 0, 0 :

K

0;
A
T
= c;
c
T
x
T
b 0 x : c
T
x Opt(P).
Optimization Group 6
Proof of the Conic Duality Theorem (4)
(P): c
T
x min | Ax b
K
0
Summary:
= 0, 0 :

K

0;
A
T
= c;
c
T
x
T
b 0 x : c
T
x Opt(P).
3
0
. Claim: > 0. Indeed, assume = 0. Then our Summary reads
= 0 :

K

0;
A
T
= 0;

T
b 0.
Recall that (P) is strictly feasible, i.e. A x b >
K
0 for some x. We have

T
(A x b)
. .
> 0, since
0 = K

A x b int K
= (A
T
)
T
x
. .
=0

T
b
..
0
0,
which is a contradiction.
Optimization Group 7
Proof of the Conic Duality Theorem (5)
Summary:
= 0, > 0 :

K

0;
A
T
= c;
c
T
x
T
b 0 x : c
T
x Opt(P).
Setting
y

=
1
,
we get
y

0;
A
T
y

= c,
showing that y

is dual feasible! Moreover

c
T
x b
T
y

0 x : c
T
x Opt(P)

b
T
y

Opt(P)

b
T
y

= Opt(P) [by Weak duality]

Q.E.D.
Optimization Group 8
Conic duality results
(P): c
T
x min | Ax b
K
0
(D): b
T
y max | A
T
y = c, y
K

0
Corollary. Let one of the problems (P), (D) be strictly feasible and bounded. Then a pair
(x, y) of primal and dual feasible solutions is comprised of optimal solutions
if and only if the duality gap at the pair is zero:
DualityGap(x, y) c
T
x b
T
y = 0
Indeed, in the case in question Opt(P) = Opt(D), hence
c
T
x b
T
y =
_
_
c
T
x Opt(P)
. .
0
_
_
+
_
_
Opt(D) b
T
y
. .
0
_
_
.
as well as
if and only if the complementary slackness holds:
[ y
..
K

]
T
[Ax b
. .
K
] = 0
Indeed, for primal-dual feasible (x, y) we have
c
T
x b
T
y = [A
T
y]
T
x b
T
y = y
T
[Ax b].
Optimization Group 9
Pathological examples (1)
x
1
min | Ax b
_

_
x
1
x
2
1
x
1
+x
2
_

_
L
3
0

x
1
min |
_
(x
1
x
2
)
2
+1 x
1
+x
2

x
1
min | 4x
1
x
2
1, x
1
+x
2
> 0.
x1
x2
Conclusion: A conic problem can be strictly
feasible and bounded and at the same time
unsolvable!
Optimization Group 10
Pathological examples (2)
x
1
min | Ax b
_
_
x
1
x
2
1
x
1
+x
2
_
_

L
3 0
We now show that the dual problem is solvable, in agreement with the Conic Duality
Theorem.
In this example K = L
3
and
c =
_
1
0
_
, A =
_

_
1 1
0 0
1 1
_

_ , b =
_

_
0
1
0
_

_ .
Hence the dual problem is
y
2
max | A
T
y
_
y
1
+y
3
y
1
+y
3
_
=
_
1
0
_
, y
L
3
0
The linear equations imply y
1
= y
3
=
1
2
, and y
L
3
0 means
1
2

y
2
2
+
1
4
,
whence y
2
= 0. Hence the dual is solvable and Opt(P) = Opt(D) = 0.
Optimization Group 11
Pathological examples (3)
x
2
min | Ax b =
_

_
x
1
x
2
x
1
_

_
L
3
0.

x
2
min |
_
x
2
1
+x
2
2
x
1

x
2
min | x
2
= 0, x
1
0
Geometrically: We are minimizing the linear objective x
2
over the intersection of the 3D ice-
cream cone and the plane tangent to the cone along the ray {(x
1
, 0, x
1
)
T
| x
1
0}. The
intersection is the ray itself. The optimal set: the ray {(x
1
, 0)
T
| x
1
0}
x1
x3
x2
Optimization Group 12
Pathological examples (4)
x
2
min | Ax b =
_

_
x
1
x
2
x
1
_

_
L
3
0.
The ice-cream cone is self-dual:
(L
m
)

= L
m
,
so that the dual problem is
b
T
0 max | A
T

_

1
+
3

2
_
= c
_
0
1
_
,
L
3
0.

L
3
0
_

2
1
+
2
2

3

1
+
3
= 0

2
= 1
_

_
infeasible!
Conclusion: A conic problem can be solvable and possess an infeasible dual!
Optimization Group 13
Pathological examples (4)
The problem
x
2
min | Ax b
_

_
1 +x
2
0 0
0 x
1
x
2
0 x
2
0
_

_

S
3
+
0

x
2
min | x
1
0, x
2
= 0
(P)
is solvable with optimal value Opt(P) = 0. The semidenite cone S
m
+
is self-dual, and the dual problem is
y
11
max |
_

_
_

_
y
11
y
12
y
13
y
12
y
22
y
23
y
13
y
23
y
33
_

_

S
3
+
0
y
22
= 0,
y
11
+2y
23
= 1

y
11
max |
_

_
_

_
y
11
0 y
13
0 0 0
y
13
0 y
33
_

_

S
3
+
0,
y
11
= 1
(D)
(D) is solvable with optimal value Opt(D) = 1.
Conclusion: It may happen that both primal and dual conic problems are solvable and have dierent optimal
values!
Optimization Group 14
Consequences of Conic Duality Theorem (1)
Question: When does a linear vector inequality
Ax
K
b (I)
have no solutions?
Sucient condition for infeasibility: By admissible aggregation of (I) one can obtain a
contradictory scalar inequality:

K

0 : A
T
= 0,
T
b > 0. (II)
Indeed, assuming that Ax
K
b for some x, we would get
0 [
..
K

]
T
[Ax b
. .
K
]
= [A
T

. .
=0
]
T
x
T
b
=
T
b
< 0 contradiction!
Optimization Group 15
Consequences of Conic Duality Theorem (2)
Ax
K
b (I)

K

0; A
T
= 0;
T
b > 0. (II)
Conic Theorem on Alternative:
1. If (II) has a solution, then (I) has no solutions.
2. If (II) has no solutions, then (I) is almost solvable:
For every > 0, you may perturb b by no more than to get a solvable system (I):
> 0 b

: b b

< and Ax
K
b

is solvable
3. (II) has no solutions if and only if (I) is almost solvable.
Optimization Group 16
Proof of Conic Theorem on Alternative
Ax
K
b (I)

K

0; A
T
= 0;
T
b > 0. (II)
1. If (II) has a solution, then (I) has no solutions. already proved.
3. (II) has no solutions if and only if (I) is almost solvable.
if part: We should prove that if (II) has a solution

, then (I) is not almost solvable. This
is evident: for all b

close enough to b we have

A
T

= 0 and

T
b

> 0
..
by 1.
Ax
K
b

unsolvable.
only if part: We should prove that if (I) is not almost solvable, then (II) has a solution. Let
e >
K
0. Consider the conic program
t min | Ax +te
K
b.
It clearly is strictly feasible; since (I) is not almost solvable, the optimal value t

in this problem
is positive. By Conic Duality Theorem, the dual problem
b
T
max | A
T
= 0, e
T
= 1,
K

0
is solvable and has positive optimal value, Q.E.D.
Optimization Group 17
CTA versus GTA
Ax
K
b (I)

K

0; A
T
= 0;
T
b > 0. (II)
CTA: (II) has no solutions if and only if (I) is almost solvable.
CTA vs. GTA: in GTA K = R
m
+
, and here almost solvable can be strengthen to solv-
able. In the general conic case, one cannot replace almost solvable with solvable.
Optimization Group 18
CTA versus GTA: Example
The linear vector inequality
Ax b
_

_
x +1
x 1

2x
_

_
L
3
0

2x
_
(x +1)
2
+(x 1)
2

_
2x
2
+2
(I)
is unsolvable. The associated system (II) also is unsolvable!
(a)
3

_

2
1
+
2
2
_

L
3

0
_
(b)
1
+
2
+

2
3
= 0
_
A
T
= 0
_
(c)
2

1
> 0
_
b
T
> 0
_
(II)
(b) (
1

2
)
2
+
2
2
3
by (a)
..
2
2
1
+2
2
2
. .
(
1
+
2
)
2
= 2
2
3

2
= 0 contradicts (c)!
Optimization Group 19
CTA versus GTA: Example (cont.)
Ax b
_

_
x +1
x 1

2x
_

_
L
3
0 (I)
Geometrically: We are seeking for a point in the intersection of the 3D ice-cream cone and a
line which is asymptotic for a 2D hyperbolic cross-section of the cone:
x1
x3
x2
The intersection is empty, but it can be made
nonempty by arbitrarily small translations of
the line, so that (I) is almost solvable, al-
though is not solvable. Our sucient condi-
tion for unsolvability is too stupid to under-
stand this dierence.
Optimization Group 20
More on the CTA
Question: When is a scalar inequality
c
T
x d (S)
a consequence of a given vector inequality
Ax
K
b ? (V)
Answer:
1. If (S) can be obtained from (V) and the trivial inequality 0 1 by admissible linear
aggregation:

K

0 : A
T
= c,
T
b d, ()
then (S) is a consequence of (V).
2. If (S) is a consequence of (V) and (V) is strictly feasible, then (S) can be obtained from
(V) by admissible linear aggregation.
Indeed, assume that (S) is a consequence of (V) and (V) is strictly feasible.
Consider the conic problem
c
T
x min | Ax
K
b.
Since (S) is a consequence of (V), the problem is bounded with optimal value d.
By Conic Duality Theorem, the dual problem is solvable with optimal value d, so
that (*) is solvable.
Optimization Group 21
Back to duality theory for conic programming
c
T
x min | Ax b
K
0 (P)
b
T
y max | A
T
y = c y
K

0 (D)
A not strictly feasible primal-dual pair (P), (D) of conic problems may be pathological:
1. (P) may be strictly feasible, below bounded and at the same unsolvable:
_
_
_
x
1
min |
_
_
x
1
x
2
1
x
1
+x
2
_
_

L
3 0
_
_
_

_
_
_
x
1
min
4x
1
x
2
1,
x
1
+x
2
> 0
(1)
2. (P) may be solvable, while (D) may be infeasible:
_
_
_
x
2
min |
_
_
x
1
x
2
x
1
_
_

L
3 0
_
_
_

_
_
_
x
2
min
x
2
= 0,
x
1
0
(2)
3. (P) and (D) may both be solvable with distinct optimal values:
_

_
x
2
min |
_

_
1 +x
2
0 0
0 x
1
x
2
0 x
2
0
_

_

S
3
+
0
_

_
_
_
x
2
min
x
1
0,
x
2
= 0
(3)
is solvable, same as its dual, but Opt(P) = 0 = 1 = Opt(D).
Where the pathologies come from?
Optimization Group 22
Where the pathologies come from?
In all our pathological examples, the solvability status of (P) is non-robust it can be
changed by arbitrarily small perturbations of the data c, A, b.
E.g.,
in (1), passing from x
1
min to x
1
x
2
min makes the problem unbounded;
in (2), passing from
_

_
x
1
x
2
x
1
_

_
L
3
0 to
_

_
x
1
+
x
2
x
1
_

_
L
3
0 makes the problem infeasible.
in (3), passing from x
2
min to x
2
x
1
min makes the problem unbounded, and
passing from
_

_
1 +x
2
0 0
0 x
1
x
2
0 x
2
0
_

S
3
+
0 to
_

_
1 +x
2
0 0
0 x
1
x
2
0 x
2

_

S
3
+
0 makes the problem
infeasible.
From practical viewpoint, problems with non-robust feasibility status are dangerous. Un-
certainty in the data or rounding errors in the course of computations make it dicult to nd
out what is the solvability status of the problem.
Optimization Group 23
Robust feasibility and robust boundedness (1)
(CP) : d
T
min | [L b] K
(CD) : b
T
y max | y
_
L

+d
_
K

[dim = m, dim L = n]
When is the solvability status of (CP) robust?
Robust feasibility. (CP) is called robust feasible, if all problems with the same structure
(n, m, K) and data (d

, L

, b

) close enough to (d, L, b) are feasible.

(The distance between linear subspaces can be measured, e.g., as the norm of the dierence
of the orthoprojectors on the subspaces).
Robust boundedness. (CP) is called robust bounded, if all problems with the same structure
(n, m, K) and data (d

, L

, b

) close enough to (d, L, b) are bounded.

Optimization Group 24
Robust feasibility and robust boundedness (2)
(CP) : d
T
min | [L b] K
_
c
T
x min | Ax b
K
0 [Ker A = {0}]
_
(CD) : b
T
y max | y
_
L

+d
_
K

_
b
T
y max | A
T
y = c, y
K

0
_
Lemma. (CP) is robust feasible if and only if it is strictly feasible.
Indeed, it is easily seen that robust feasibility means exactly that
(i) All systems
A

x b

K
0 (1)
with (A

, b

) close to (A, b) are solvable, while strict feasibility means that

(ii) The system
Ax b >
K
0 (2)
is solvable. Implication (ii)(i) is evident: if A x b >
K
0, then A

x b

K
0 whenever (A

, b

)
is close to (A, b).
Implication (i)(ii): Let e >
K
0. If (i) takes place, there exists > 0 such that the system
Ax (b e)
K
0
is solvable; a solution to this system is a strictly feasible solution to (2).
Optimization Group 25
Robust feasibility and robust boundedness (3)
(CP) : d
T
min | [L b] K
_
c
T
x min | Ax b
K
0

(CD) : b
T
y max | y
_
L

+d

_
b
T
y max | A
T
y = c, y
K

Theorem. The following properties of a conic problem (CP) are equivalent to each other:
(i) (CP) is robust feasible and robust bounded;
(ii) (CP) is robust solvable;
(iii) (CD) is robust solvable;
(iv) (CD) is robust feasible and robust bounded;
(v) Both (CP) and (CD) are strictly feasible.
Whenever these properties take place, the optimal values in (CP) and (CD) are equal to each
other and are continuous in the data in a neighbourhood of (c, A, b).
Proof. (i)(iii): If (CP) is robust feasible and robust below bounded, (CD) is robust solvable by Conic Duality
Theorem.
(iii)(iv): evident;
(iv)(ii): symmetric to (i)(iii);
(ii)(i): evident
(i)(ii)(iii)(iv) (v): by Lemma;
(v)(i): by Lemma and Weak Duality.
Optimization Group 26