Problem Sheet 3

Optimization Techniques: MA 526

October 18, 2021 D EBDAS G HOSH

1. Which of the following functions is convex, concave, or neither? Why?

(a) f (x1 , x2 ) = 2x21 − 4x1 x2 − 8x1 + 3x2

(b) f (x1 , x2 ) = x1 e−(x1 +3x2 )
(c) f (x1 , x2 ) = −x21 − 3x22 + 4x1 x2 + 10x1 − 10x2
(d) f (x1 , x2 , x3 ) = 2x1 x2 + 2x21 + x22 + 2x23 − 5x1 x3
(e) f (x1 , x2 , x3 ) = −2x21 − 3x22 − 2x23 + 8x1 x2 + 3x1 x3 + 4x2 x3
b
2. Over what subset of {x : x > 0} is the univariate function f (x) = e−ax convex, where a > 0
and b ≥ 1?

3. Prove or disprove concavity of the following function defined over S = {(x1 , x2 ) : −1 ≤

x1 ≤ 1, −1 ≤ x2 ≤ 1}:

f (x1 , x2 ) = 10 − 3(x2 − x21 )2 .

Repeat for a convex set S ⊆ {(x1 , x2 ) : x21 ≥ x2 }.

4. Over what domain is the function f (x) = x2 (x2 − 1) convex? Is it strictly convex over the
region(s) specified? Justify your answer.

5. Show that a function f : Rn → R is affine if and only if f is both convex and concave. (A
function f is affine if it is of the form f (x) = α + c> x, where α is a scalar c is an n-vector.)
n
2 , . . . , fk : R → R be convex functions. Consider the function f defined by
6. Let f1 , fP
k
f (x) = j=1 αj fj (x), where αj > 0 for j = 1, 2, . . . , k. Show that f is convex. State and
prove a similar result for concave functions.

7. Let f1 , f2 , . . . , fk : Rn → R be convex functions. Consider the function f defined by

f (x) = max{f1 (x), f2 (x), . . . , fk (x)}. Show that f is convex. State and prove a similar
result for concave functions.

8. Let h : Rn → R be a convex function, and let g : Rn → R be a nondecreasing convex

function. Consider the composite function f : Rn → R defined by f (x) = g[h(x)]. Show
that f is convex.

9. Let S be a nonempty convex set in Rn , and let f : Rn → R be defined as follows:

1
 f (y) = inf{ky − xk : x ∈ S}.

Note that f (y) gives the distance from y to the set S and is called the distance function. Prove
that f is convex.

10. Let S = {(x1 , x2 ) : x21 + x22 ≤ 4}. Let f be the distance function defined in Problem 9. Find
the function f explicitly.

11. Let g : Rm → R be a convex function, and let h : Rn → Rm be an affine function of the

form h(x) = Ax + b, where A is an m × n matrix and b is an m × 1 vector. Then show that
the composite function f : Rn → R defined as f (x) = g[h(x)] is a convex function. Also,
assuming twice differentiability of g, derive an expression for the Hessian of f .

12. Let F be a cumulative Rdistribution function for a random variable b, that is, F (y) = Prob(b ≤
z
y). Show that φ(z) = −∞ F (y)dy is a convex function. Is φ convex for any nondecreasing
function F ?

13. Let f : S → R be defined as

(α> x)2
f (x) = β>x
,

where S is a convex subset of Rn , α and β are vectors in Rn , and where β t x > 0 for all x ∈ S.
Derive an explicit expression for the Hessian of f , and hence verify that f is convex over S.
2 2
14. Let f (x1 , x2 ) = e2x1 −x2 − 3x1 + 5x2 . Give the linear and quadratic approximations of f at
(1, 1). Are these approximations convex, concave, or neither? Why?

15. Consider the function f : R3 → R, given by f (x) = x> Ax, where

 
2 2 3
A= 1 3 1 
1 2 θ

What is the Hessian of f ? For what values of θ is f strictly convex?

16. Consider the following problem:

minimize (x1 − 4)2 + (x2 − 6)2

subject to x2 ≥ x21 and x2 ≤ 4.

Write a necessary condition for optimality and verify that it is satisfied by the point (2, 4). Is
this the optimal point? Why?

17. (a) Give an example of strictly convex function that is not bounded below and not strongly
convex.
(b) Give an example of convex function that is neither strictly convex nor affine.
(c) Give an example of strictly convex function but its strict convexity cannot be checked
by second order derivative test.

2
18. Let S be a nonempty open convex set in R, and f : S → R be infinitely many times
differentiable on S. Prove that f is strictly convex iff for each x̄ ∈ S, there exists an even n
such that f (n) (x̄) > 0, while f (j) (x̄) = 0 for any 1 < j < n.
x21
19. Show that the function f : R × R+ → R given by f (x1 , x2 ) = x2
is convex.
(
0, if x ∈ C
20. Prove that a nonempty set C ⊆ Rn is convex iff its indicator function fC (x) =
+∞, if x ∈
/C
is convex.
Note: Indicator function allows to pass constraint to objective:

min f (x) ≡ minn f (x) + IC (x).

x∈C x∈R

21. Given an arbitrary set S ⊆ Rn , not necessarily convex, the support function associated with
S is σS : Rn → R ∪ {+∞} given by σS (z) = sup x> z. Prove that σS is a convex function
x∈S
on Rn .
Note:

• Support Function is conjugate of the indicator function of S.

• Conjugate of a closed (i.e., {x ∈ domf : f (x) ≤ α} is closed ∀ α ∈ R ) and convex
function is convex.

(These two notes do not prove Problem 21.)

22. Let S be a nonempty convex set in Rn . Prove that the distance function f (x) = inf{ky − xk :
x ∈ S} is a convex function.

23. Show that the max function f (x) = max{x1 , x2 , . . . , xn } is convex on Rn .

24. Let S be a nonempty convex subset of Rn , and f : S → R. Prove that the following three are
equivalent:

(a) epi(f ) is convex,

(b) (Jensen’s inequality.) for all λi ≥ 0 with ni=1 λi = 1 and xi ∈ S, i = 1, 2, . . . , n, we
P
have !
Xn Xn
f λi xi ≤ λi f (xi )
i=1 i=1

(c) f (λx + (1 − λ)y) ≤ λf (x) + (1 − λ)f (y) ∀ x, y ∈ S, λ ∈ [0, 1].

3
 Solutions

Solution to Problem 5.

(⇐) Forward part is trivial.

(⇒) Let f is convex and concave both.

Then
f (λx1 + (1 − λ)x2 ) = λf (x1 ) + (1 − λ)f (x2 ), ∀ λ ∈ [0, 1]
Consider g(x) = f (x) − f (0). Then

g(x1 + x2 ) = f (x1 + x2 ) − f (0)

1 1
= f ( (2x1 ) + (2x2 )) − f (0)
2 2
1 1
= f (2x1 ) + f (2x2 ) − f (0).
2 2
Step 1: Claim g(αx) = αg(x) ∀ α ≥ 0
For α ≥ 1
1 1
f (x) = f ( (αx) + (1 − )0)
α α
1 1
= f (αx) + (1 − )f (0))
α α
1 1
= (g(αx) + f (0)) + (1 − )f (0)
α α
⇒ αf (x) = g(αx) + αf (0)
⇒ g(αx) = α(f (x) − f (0)) = αg(x)

for α ∈ (0, 1)

g(αx) = g(αx + (1 − α)0)

= f (αx + (1 − α)0) − f (0)
= (αf (x) + (1 − α)f (0)) − f (0)
= α(f (x) − f (0)) = αg(x)

for α = 0

g(0x) = f (0) − f (0) = 0g(x)

for α = 1

g(αx) = αg(x).

Step 2: Claim g(x1 + x2 ) = g(x1 ) + g(x2 )

Since,
g(x − x) = g(0) = 0

4
 ⇒ g(x) + g(−x) = 0
⇒ g(−x) = −g(x).
Step 3: Claim g(αx) = αg(x) ∀ α < 0.

g(αx) = g((−α)(−x)) = −αg(−x), by Step 1

= αg(x), by Step 2

Therefore, g is a linear
Pn function.
n
Take x ∈ R . x = i=1 xi ei

n
X
x= xi ei , ei → i-th unit vector
i=1
n
X
⇒ g(x) = xi g(ei ) = a1 x1 + a2 x2 + · · · + an xn , ai = g(ei )
i=1
⇒ g(x) = at x, where a = [a1 a2 · · · an ]t

Therefore,

f (x) = g(x) + f (0)

= at x + b.

Solution to Problem 6.
Let x1 , x2 ∈ Rn and λ ∈ (0, 1). Hence
k
X
h(λx1 + (1 − λ)x2 ) = αj fj (λx1 + (1 − λ)x2 )
j=1
k
X

≤ αj λfj (x1 ) + (1 − λ)fj (x2 )
j=1
k
! k
!
X X
= λ αj fj (x1 ) + (1 − λ) αj fj (x2 )
j=1 j=1
= λh(x1 ) + (1 − λ)h(x2 ).

Solution to Problem 7.
Let x1 , x2 ∈ Rn and λ ∈ (0, 1). Thus,

g(λx1 + (1 − λ)x2 ) = max {fi (λx1 + (1 − λ)x2 )}

i=1,2,...,k
≤ max {λfi (x1 ) + (1 − λ)fi (x2 )}
i=1,2,...,k
= λ max {fi (x1 )} + (1 − λ) max {fi (x2 )}
i=1,2,...,k i=1,2,...,k
= λg(x1 ) + (1 − λ)g(x2 ).

5
Solution to Problem 8.
Let x1 , x2 ∈ Rn and λ ∈ (0, 1). Thus,


(g ◦ h)(λx1 + (1 − λ)x2 ) = g h(λx1 + (1 − λ)x2 )


≤ g λh(x1 ) + (1 − λ)h(x2 )



= λg h(x1 ) + (1 − λ)g h(x2 )
= λ(g ◦ h)(x1 ) + (1 − λ)(g ◦ h)(x2 ).

6
 Solution to Problem 13.
(αt x)2
f (x) =
β tx
   αt x 2 αt x

f x1 −β (
1 βtx ) + 2( βtx
)α 1
αt x 2 αt x
 fx   −β ( ) + 2( )α

 2   2 βtx  βtx 2
∇f (x) =  ..  =  .. 
 .   .


f xn αt x 2 αt x
−βn ( β t x ) + 2( β t x )αn
 
(∇fx1 )t
 (∇fx )t 
2
H(f ) =  .. 
 
 . 
(∇fxn )t
 t t t t t 
−2β1 αβ t xx α1 β(βx−β
t x)2
1α x
+ 2α1 α1 β(βx−β 1α x
t x)2
t t t t t 
 −2β1 αβ t xx α2 β(βx−β 2α x
+ 2α1 α2 β(βx−β 2α x

t x)2 t x)2 
∇f (x1 ) =  ..
7

 
.

 
t t t t t
−2β1 αβ t xx αn β(βx−β
t x)2
nα x
+ 2α1 αn β(βx−β nα x
t x)2
 
−β1 (αt x)(α1 β t x − β1 αt x) + α1 (α1 (β t x)2 − β1 (αt x)(β t x))
t t t t 2 t t
2   −β1 (α x)(α2 β x − β2 α x) + α1 (α2 (β x) − β2 (α x)(β x)) 

= t 3 ..
(β x) 

. 
t t t t 2 t t
−β1 (α x)(αn β x − βn α x) + α1 (αn (β x) − βn (α x)(β x))
 
(β1 αt x − α1 β t x)2
t t t t
2   (β1 α x − α1 β x)(β2 α x − α2 β x) 

= t 3 ..
(β x) 

. 
t t t t
(β1 α x − α1 β x)(βn α x − αn β x)
 
(β1 αt x − α1 β t x)2 (β1 αt x − α1 β t x)(β2 αt x − α2 β t x) ··· (β1 αt x − α1 β t x)(βn αt x − αn β t x)
t t t t
2  (β2 α x − α2 β x)(β1 α x − α1 β x)
 (β2 αt x − α2 β t x)2 ··· (β2 αt x − α2 β t x)(βn αt x − αn β t x)
∴ H(f ) = t 3 

.. .. ..
(β x) 

. . ··· . 
(βn α x − αn β x)(β1 α x − α1 β x) (βn α x − αn β x)(β2 αt x − α2 β t x)
t t t t t t
··· (βn αt x − αn β t x)2
 2
=⇒ at H(f )a = {(β1 αt x − α1 β t x)2 a21 + (β2 αt x − α2 β t x)2 a22 + · · · + (βn αt x − αn β t x)2 a2n
(β t x)3
+ 2(β1 αt x − α1 β t x)(β2 αt x − α2 β t x)a1 a2
+ 2(β1 αt x − α1 β t x)(β3 αt x − α3 β t x)a1 a3
+ ···
+ 2(βn−1 αt x − αn−1 β t x)(βn αt x − αn β t x)an−1 an }

2
= (β t x)3
((β1 αt x − α1 β t x)a1 + (β2 αt x − α2 β t x)a2 + · · · + (βn αt x − αn β t x)an )2

≥ 0, ∀ a ∈ Rn

Therefore, H(f ) is positive semi definite.

8
Solution to Problem 17.

1. f (x) = − log x. f 00 (x) = 1

x2
> 0.

Solution to Problem 18.