CS 726: Nonlinear Optimization 1

Lecture 2 : Background Material

Michael C. Ferris

Computer Sciences Department

University of Wisconsin-Madison

January 27 2021

Michael C. Ferris (UW-Madison) CS726:Lecture 2 Background Material 1 / 13

Announcements

In addition to the Appendix of the textbook for the course, there are
two handouts on Canvas that could be useful. They are
“MathBackground.pdf” (alternatively [Bertsekas(2015), Appendix A])
and [Bertsekas(2015), Appendix B].

Michael C. Ferris (UW-Madison) CS726:Lecture 2 Background Material 2 / 13

Continuity
A function f (x) is continuous at a point x if f (x) = lim f (t).
t!x
Define epif = {(x, )| f (x)}.
Then f is lower semi-continuous if epif is a closed set in Rn+1 . (We
called this a closed function in class but will use both interchangeably.)
Intuitively, this is saying f (x)  lim f (t).
t!x
4

2
f (x)

1
2 1 0 1 2
x

The function f (x) is lower semi-continuous if f (0) = 1 but not if f (0) = 1.

Michael C. Ferris (UW-Madison) CS726:Lecture 2 Background Material 3 / 13
Norms

The norm of x, denoted ||x||, satisfies:

1 ||x|| 0 8x and ||x|| = 0 , x = 0
2 || x|| = | | ||x|| 8 2 R
3 ||x + y ||  ||x|| + ||y || 8x, y .
Some notes:
Norms are continuous functions
Norms are convex functions (see below)
All norms are equivalent in Rn because
9c1 , c2 > 0 : c1 ⇤ ||x||b  ||x||a  c2 ⇤ ||x||b

Michael C. Ferris (UW-Madison) CS726:Lecture 2 Background Material 4 / 13

Some common norms

P
n
||x||1 = |xi |
i=1
s
P
n
||x||2 = xi2
i=1

||x||1 = max |xi |

i=1,2,...,n

Note that ||x||1 and ||x||1 can be described using a finite number of linear
constraints while ||x||2 cannot be.
We can define the norm of a matrix using a vector norm by letting:
||A|| = sup ||Ax||
||x|| .
x6=0
Using that, the condition number of A is: K (A) = ||A|| ⇤ ||A 1 ||.

Michael C. Ferris (UW-Madison) CS726:Lecture 2 Background Material 5 / 13

Polyhedral Sets

A set C = {x|Ax  b} is called polyhedral.

Sets with equality constraints are still polyhedral
We will call a bounded polyhedral set a polytope

Michael C. Ferris (UW-Madison) CS726:Lecture 2 Background Material 6 / 13

Convexity of a set
Definition
Set C is convex if 8x, y 2 C , 8↵ 2 [0, 1] : ↵x + (1 ↵)y 2 C

Prove the following statements:

1 C = {x|Ax  b} is convex.
T
2 Suppose that Ci , i 2 I are convex in Rn . Then i2I Ci is convex.
Remark: Unions of convex sets are not convex:

Unions of convex sets are not convex.

3 Suppose that C1 , C2 are convex. Then C1 + C2 is also convex.

Michael C. Ferris (UW-Madison) CS726:Lecture 2 Background Material 7 / 13
More to prove:
1 Suppose that C1 , C2 are convex. Then
C1 ⇥ C2 = {(x, y )|x 2 C1 , y 2 C2 } is also convex.
2 Suppose that A 2 Rm⇥n . Then
C 2 Rn is convex ) AC = {y |y = Ax, x 2 C } is convex in Rm

When A is [1,0], AC is the projection of C.

3 Suppose C is convex. Then the closure of C : Cl(C ) is also convex.

Michael C. Ferris (UW-Madison) CS726:Lecture 2 Background Material 8 / 13
Convex Function
Suppose f : S ⇢ Rn ! R. Extend the domain to Rn :
(
f (x), x 2 S
f¯ : Rn ! R̄ = R [ {+1} such that f¯(x) =
+1, else
In terms of convex functions, it is convenient to just consider extended
real-valued functions. For example,
min f (x) s.t. x 2 C ) min f¯(x) s.t. x 2 Rn
where (
f (x), if x 2 C
f¯(x) =
+1, if x 2
/C
or (
0, if x 2 C
f¯(x) = f (x) + IC (x), where Ic (x) =
+1, if x 2
/C
This is essentially the same as used in [Boyd and Vandenberghe(2004)]
and also in the lecture notes [Burke(2018)].
Michael C. Ferris (UW-Madison) CS726:Lecture 2 Background Material 9 / 13
Definitions
E↵ective domain: dom(f ) = {x|f (x) < +1}
Epigraph: f : Rn ! R̄, epi(f ) = {(x, µ) 2 Rn+1 |f (x)  µ} (4)
Convex function: f is convex if epi(f ) is a convex subset of Rn+1

Epigraph of a function.

Note that dom(f ) = [I 0]epi(f ) so is convex when f is.

Michael C. Ferris (UW-Madison) CS726:Lecture 2 Background Material 10 / 13
Main theorem
Theorem
f : R ! R̄. Then f is convex , 8 2 [0, 1], 8x, y

f ( x + (1 )y )  f (x) + (1 )f (y ).

Proof.
()) Suppose epi(f ) is convex. Let x, y be given and 0 < < 1. If x or
y2/ dom(f ), RHS= +1. So trivial. (Convention: positive number
⇥(+1) = +1.)
Else, suppose x, y 2 dom(f ), then (x, f (x)), (y , f (y )) 2 epi(f )
By convexity of epi(f ),

(x, f (x)) + (1 )(y , f (y ))

= ( x + (1 )y , f (x) + (1 )f (y )) 2 epi(f )
) f ( x + (1 )y )  f (x) + (1 )f (y )

Michael C. Ferris (UW-Madison) CS726:Lecture 2 Background Material 11 / 13

Cont.

Proof.
(() Suppose (x, µ), (y , ) 2 epi(f ), 0 < < 1.
RTP: (x, µ) + (1 )(y , ) 2 epi(f ).

f ( x + (1 )y )  f (x) + (1 )f (y )
 µ + (1 )
) ( x + (1 )y , µ + (1 ) ) 2 epi(f )
) (x, µ) + (1 )(y , ) 2 epi(f )

Michael C. Ferris (UW-Madison) CS726:Lecture 2 Background Material 12 / 13

Examples

1 C T x is a convex function.
Remark: Linear functions are the only functions that are both convex
and concave.
2 kxk is convex by triangular inequality.
3 e x and x 2 are convex.
4
(
1
x , if x > 0
f (x) = is convex.
+1, else

Michael C. Ferris (UW-Madison) CS726:Lecture 2 Background Material 13 / 13

 D. P. Bertsekas.
Convex Optimization Algorithms.
Athena Scientific, 2015.
S. Boyd and L. Vandenberghe.
Convex Optimization.
Cambridge University Press, 2004.
ISBN 9780521833783.
URL
http://www.informaworld.com/openurl?genre=article{&}doi=
10.1080/10556781003625177{&}magic=crossref.
J. V. Burke.
Nonlinear Optimization.
in preparation, 2018.
S. J. Wright and B. Recht.
Optimization for Data Analysis.
in proof, 2020.

Michael C. Ferris (UW-Madison) CS726:Lecture 2 Background Material 13 / 13