Lecture 9: introduction to nonlinear optimization

• Convex functions
• Convex sets
• Reminder: derivative
• Gradient descent
• Problems of gradient descent

Convex function

f
f(x)

f(b)
0

1

f(a)

f(x)
a b
x

1
Convex function

f f(x)=f(a+(1- )b)
f(a)+(1- )f(b)

f(x)

f(b)
0

1
f(a)+(1- )f(b)

f(a)

f(x)
a X= a+(1- )b b
x

Non convex function

f f(x)=f(a+(1- )b)
f(a)+(1- )f(b)

f(x)
f(x)
f(b)
f(a)+(1- )f(b) 0

1

f(a)

a X= a+(1- )b b
x

2
Concave function

f f(x)=f(a+(1- )b)
f(a)+(1- )f(b)

0

1
f(b)

f(x)

f(a)
f(x)

a X= a+(1- )b b
x

Definitions / facts, common mistakes

Definitions/facts
• If f is convex, -f is concave
• If f is concave, -f is convex

Common mistakes
• A nonconvex function is not necessarily concave
• A nonconcave function is not necessarily convex
• A function can be neither concave neither convex
• A function can be concave and convex

3
Functions not defined for all x

f
f(x)+
f(x)+

f(x)

Functions not defined for all x (convex)

f
f(x)+
f(x)+

Not defined

Not defined

f(x)

4
Functions not defined for all x (convex)

Not defined

Not defined
f(x)

Functions not defined for all x (convex)

definition
f
f(x)=+
f(x)=+

Not defined

Not defined

f(x)

5
Functions not defined for all x (not convex)

f
f(x)+
f(x)+
f(x)+

Not defined

Not defined
f(x)
x

Convex sets
0 1
Ax

x
Definition: A+(1- )B x
Convex set: for all A and B
B
in the set, if A and B are in
the set, A+(1- )B is also in
this set, for 0 1

This set is convex

6
Convex sets
0

1
Ax

x
Definition: A+(1- )B x
Convex set: for all A and B
B
in the set, if A and B are in
the set, A+(1- )B is also in
this set, for 0

1

This set is not convex

Famous convex sets

7
Famous non convex sets

Why do we use ‘convex’ for functions and sets

The epigraph (i.e. points above the graph) of a convex function is
a convex set.

Convex function Non convex function

8
Why do we use ‘convex’ for functions and sets
The epigraph (i.e. points above the graph) of a convex function is
a convex set.

Epigraph is convex Epigraph is non convex

Convex sets and functions: basic properties

Non convex function Convex function

Local minimum might not Local minimum is a global

be a global minimum minimum

9
Convex sets and functions: basic properties
Convex function

Minimum might not be unique Local minimum is a global

minimum

Convex optimization programs

Convex function

f and g are convex functions,

defined on convex sets

Convex optimization programs Local minimum is a global

are “easy” problems, compared minimum
to general optimization programs

10
Reminder: derivative

f
f(x)

Reminder: derivative

f
The derivative of a function f(x)
at a point represents its
slope

11
Gradient descent (conceptual description)
You want to find the minimum of a function, starting from a guess,
assuming that you cannot depict the graph of the function, for example
the following function

Idea:

1) Make a guess
2) Compute the derivative at this point (i.e. the slope)
3) Follow the direction of the slope (i.e. descend)
4) Stop when the slope is zero, i.e. it does not go downhill

Gradient descent (illustration)

f(x)

guess

f(m)

m x

12
Gradient descent (illustration)

f(x)

guess

next step

f(m)

m x

Gradient descent (illustration)

f(x)
new gradient
guess

next step

f(m)

m x

13
Gradient descent (illustration)

f(x)

guess

next step

f(m)

m x

Gradient descent (illustration)

f(x)

guess

stop
f(m)

m x

14
Problem 1: non convex function

f(x)
“good” guess

f(m)

m x

Problem 1: non convex function

f
“bad” guess
f(x)
“good” guess

f(m)

m x

15
Problem 2: how to stop?

What if you cannot

figure out where
the stop point is?

Problem 3: how to hit the wall?

“algebraic minimum”
f
Not defined

16
Problem 3: how to hit the wall?
“actual minimum”
f

Not defined x

17