Fundamentals of Optimization

Lecture 5

Xavier Cabezas

FCNM-ESPOL
July 20, 2020
 Outline

Convex and concave functions

Local and global optimal

Level sets

Characterization of convex functions

Additional properties of convex functions

Convex and concave functions 2

 Convex function definition

Let C be a convex set of Rn . A function f : C → R is called convex if

f (λx1 + (1 − λ)x2 ) ≤ λf (x1 ) + (1 − λ)f (x2 ), ∀x1 , x2 ∈ C, ∀λ ∈ [0, 1].

According with the expression above, f is called

• Strictly convex if the inequality is strict (<) for all x1 , x2 ∈ C with

x1 6= x2 and λ ∈ (0, 1).

• Concave if −f is convex.

• Strictly concave if −f is strictly convex.

Convex and concave functions 3

 Examples of convex and concave functions

λx1 +(1−λ)x2 λx1 +(1−λ)x2

x1 x2 x1 x2

(a) Convex (b) Concave

λx1 +(1−λ)x2
x1 x2 x1 x2

(c) Neither convex nor concave (d) Convex and concave

Convex and concave functions 4

 Linear and affine maps are convex and concave

• A linear map L : Rn → Rm is a relation such that

L(λx + y) = λL(x) + L(y), with λ ∈ R.

• A linear map can be represented by L(x) = M x, where M ∈ Rm×n .

• If L is a function, i.e., L : Rn → R, then L(x) = a0 x where a ∈ Rn .

• An affine function (or map) is a linear map moved from the origin.
La (x) = x0 + M x, with x0 ∈ Rn .
.

Convex and concave functions 5

 Linear and affine maps are convex and concave

Theorem
An affine map is both convex and concave.
.
Proof.

La (λx1 + (1 − λ)x2 ) = x0 + M (λx1 + (1 − λ)x2 )

= λx0 + (1 − λ)x0 + M (λx1 ) + M ((1 − λ)x2 ))
= λx0 + (1 − λ)x0 + λM x1 + (1 − λ)M x2
= λ(x0 + M x1 ) + (1 − λ)(x0 + λM x2 )
= λLa (x1 ) + (1 − λ)La (x2 ).

Convex and concave functions 6

 Outline

Convex and concave functions

Local and global optimal

Level sets

Characterization of convex functions

Additional properties of convex functions

Local and global optimal 7

 Local and global minimum (maximum)

For a problem P = min { f (x) | x ∈ X } (f : X → R).

• x̄ ∈ X is a local minimum if there is an ε-neighborhood

Nε (x̄) = { x | kx − x̄k ≤ ε } such that f (x̄) ≤ f (x) for each
x ∈ Nε (x̄).

• x̄ ∈ X is a global minimum if f (x̄) ≤ f (x) for all x ∈ X.

• If X is a convex set and f is a convex function, then P is called

convex problem.

For a maximization problem the given definitions are similar, ≥ instead ≤.

Local and global optimal 8

 Local and global optimal in a convex problem

Theorem
In a convex minimization problem any local minimum is a global
minimum (optimal solution).
Proof.
(by contradiction) Suppose x̄ is a local minimum but not a global
minimum. Then there is an x ∈ X such that f (x) < f (x̄).

By convexity of f we have for λ ∈ [0, 1] that

f (λx̄ + (1 − λ)x) ≤ λf (x̄) + (1 − λ)f (x)

< λf (x̄) + (1 − λ)f (x̄)
= f (x̄).

For a λ arbitrarly near to 1, λx̄ + (1 − λ)x ∈ Nε (x̄) and

f (λx̄ + (1 − λ)x) < f (x̄). This contradicts the assumption, x̄ cannot be
a local minimum.

Local and global optimal 9

 Outline

Convex and concave functions

Local and global optimal

Level sets

Characterization of convex functions

Additional properties of convex functions

Level sets 10
 Level sets

Theorem
Let f : C → R be a convex function over the convex set C ∈ Rn , the
level set levα f = { x ∈ C | f (x) ≤ α } is a convex set for all scalars α.

Proof.
If x1 , x2 ∈ levα f and λ ∈ [0, 1], then x = λx1 + (1 − λ)x2 ∈ C and
f (λx1 + (1 − λ)x2 ) ≤ λf (x1 ) + (1 − λ)f (x2 ) ≤ λα + (1 − λ)α = α,
then λx1 + (1 − λ)x2 ∈ levα f .

Important: Level sets of convex functions are convex. Is the converse

true? NO!!!.

Level sets 11
 Level sets examples
Convexity of level sets does not characterize convex functions. Level sets
characterize a wider family of functions, quasiconvex ones.

Let C be a nonempty convex set of Rn . The funtion f is said to be a

quasiconvex function if
f (λx1 + (1 − λ)x2 ) ≤ max { f (x1 ), f (x2 ) } , ∀x1 , x2 ∈ C, ∀λ ∈ [0, 1].

f α
α

levα f levα f
[ ] [ ]
x1 x2 x1 x2
(a) A convex function. (b) A Quasiconvex function.

Level sets 12
 Outline

Convex and concave functions

Local and global optimal

Level sets

Characterization of convex functions

Additional properties of convex functions

Characterization of convex functions 13

 Epigraph and hypograph

The epigraph of a function f : S → R where S ∈ Rn , is the a subset of

Rn+1 given by

epi f = { (x, y) | x ∈ S, y ∈ R, f (x) ≤ y } .

The hypograph of a function f : S → R where S ∈ Rn , is the a subset

of Rn+1 given by

hyp f = { (x, y) | x ∈ S, y ∈ R, f (x) ≥ y } .

Characterization of convex functions 14

 Epigraph and hypographs examples

epi f epi f epi f

f f
f

hyp f hyp f hyp f

(a) epi and hyp of a (b) epi and hyp of a (c) epi and hyp of a
function. convex function. concave function.

Characterization of convex functions 15

 A characterization of convex functions

Theorem
Let f : C → R be a convex function over the convex set C ∈ Rn . f is
convex if and only if epi f is a convex set.

Proof.
⇒) Let (x1 , y1 ) ∈ epi f and (x2 , y2 ) ∈ epi f , i.e., f (x1 ) ≤ y1 and
f (x2 ) ≤ y2 . Then for λ ∈ [0, 1]

λy1 + (1 − λ)y2 ≥ λf (x1 ) + (1 − λ)f (x2 ) ≥ f (λx1 + (1 − λ)x2 ),

that implies that (λx1 + (1 − λ)x2 , λy1 + (1 − λ)y2 ) ∈ epi f , then epi f
is convex.

⇐) As an exercise!

Characterization of convex functions 16

 A characterization of convex functions

• f is concave if and only if hyp f is convex.

• This characterization is an useful property, as allows us to translate

results about convex set into results about convex function.

– (As an example) Since that epi f and hyp f of a convex and

concave function f respectively are convex sets, then they have
supporting hyperplanes on their boundary.

– This lead us to the concept of subgradient.

Characterization of convex functions 17

 Subgradient and subdifferentials

Lat f : C → R be a convex function over the convex set C ∈ Rn . The

vector d ∈ Rn is a subgradient of f at a point x ∈ Rn if

f (z) ≥ f (x) + d0 (z − x), ∀z ∈ Rn .

• d is a subgradient of a concave function f at x if −d is a

subgradient of the convex function −f at x.

• The set of all subgradients of a convex (concave) function f at

x ∈ Rn is called the subdifferential of f at x. It is denoted by
∂f (x).
(Note the we use ∂ for both boundary of a set and subdifferential of
a function. In the last case the symbol is not bolded).

Characterization of convex functions 18

 Subgradients examples

f
epi f epi f
f (x)
f d0 (x − x0 )

f (x0 )

x0 x x0
(a) The subgradient at x0 . (b) Infinite subgradients at x0 .

In figure (a) f (x0 ) + d0 (x − x0 ) is the supporting hyperplane of the epi f

at x0 and the subgradient d is in this case the derivative.

Characterization of convex functions 19

 Subdifferential example

f (x) = |x| ∂f (x)

−1
0
(a) A convex function. (b) Subdifferential function.

Characterization of convex functions 20

 Another characterization of convex functions

Theorem
Let C be a convex set of Rn and f : C → R be a differentiable function
over C. The function f is convex if and only if

f (x2 ) ≥ f (x1 ) + ∇f (x1 )0 (x2 − x1 ) ∀x1 , x2 ∈ C.

• If the inequality is strict whenever x1 6= x2 , then f is strictly convex

over C

f (x2 )
f f (x1 ) + ∇f (x1 )0 (x2 − x1 )

x1 x2

Characterization of convex functions 21

 Proof: Theorem of characterization of convex functions

Proof.
⇒) Let x1 , x2 ∈ C and λ ∈ [0, 1]. Suppose that f is convex. Then
f ((1 − λ)x1 + λx2 ) ≤ (1 − λ)f (x1 ) + λf (x2 ) and rearranging terms we
have
f (x1 + λ(x2 − x1 )) − f (x1 )
≤ f (x2 ) − f (x1 ).
λ
Taking the limit when λ → 0+ we obtain the directional derivative in the
direction x2 − x1 on the left side. We know that this term is equal to
∇f (x1 )0 (x2 − x1 ). Then

∇f (x1 )0 (x2 − x1 ) ≤ f (x2 ) − f (x1 )

Characterization of convex functions 22

 Proof: Theorem of characterization of convex functions

Proof.
⇐) Suppose the inequality is true. Take z = λx1 + (1 − λ)x2 , by using
the inequality we obtain

f (x1 ) ≥ f (z) + ∇f (z)0 (x1 − z) and

f (x2 ) ≥ f (z) + ∇f (z)0 (x2 − z).

By multiplying the first inequality by λ, the second one by (1 − λ) and

adding them we have

λf (x1 ) + (1 − λ)f (x2 ) ≥ f (z) + ∇f (z)0 (λx1 + (1 − λ)x2 − z) = f (z).

Then f is convex.

Characterization of convex functions 23

 Outline

Convex and concave functions

Local and global optimal

Level sets

Characterization of convex functions

Additional properties of convex functions

Additional properties of convex functions 24

 Others important properties of convex functions
Homework

• If C is a convex set and L(x) : C → Rm , then Img(L) (image) is

convex. (The image of a convex set under a linear map is convex).

• The weighted sum of convex functions, with positive weights, is

convex. Hence, the sum of convex functions is convex.

• Any vector norm is convex.

• Let I an index set, C ∈ Rn a convex set and fi : C → R a convex

function for each i. Then the function h : C → R defined by
h(x) = supi∈I fi (x) is also convex.

• Let f : C → R be a convex function over a convex set C and let

g : R → R also a convex and non-decreasing function, then
h : C → R, defined by h(x) = g(f (x)) is convex.

Additional properties of convex functions 25