Example.

max 2 x1 x2 2 x2 x12 2 x22

f
2 x2 2 x1
x1
f
2 x1 2 4 x2
x2

Starting from X ( 0) (0,0) , f ( X ) |X X f (0,0) (0,2)

( 0)

x1 0 t (0) 0
x2 0 t (2) 2t
f ( X ( 0) tf ( X )) f (0,2t ) 2(0)(2t ) 2(2t ) 0 2(2t ) 2 4t 8t 2
max f (t ) 4t 8t 2
t 0

Let

d
1
1
1
(4t 8t 2 ) 4 16t 0 t* , X (1) (0,0) (0,2) (0, ) .
dt
4
4
2
1
2

1
2

At (0, ) , f (0, ) (1,0)

1
2

1
2

Let X (0, ) t (1,0) (t, )

1
1
1
1
1
1
f ( X ( 2 ) tf ( X )) f (0 t, 0t ) f (t , ) 2t ( ) 2( ) t 2 2( ) 2 t t 2
2
2
2
2
2
2

Let

d
1
1 1
1 1
1
(t t 2 ) 1 2t 0 , t* , X ( 2 ) (0, ) (1,0) ( , )
dt
2
2
2 2
2 2

and so on.

132

133

6. KKT Conditions for Constrained Optimization

max f ( X )

g1 ( X ) b1
g 2 ( X ) b2

g m ( X ) bm
X {x1 , x2 ,..., xn } 0

s.t.

X * {x1*, x2 *,..., xn *} is optimal only if there exist ui , i 1,..., m , such that

1.

m
g
f
ui i 0, j 1,...n
x j i 1 x j

2. x *j (

For X X *

m
g
f
ui i ) 0, j 1,...n
x j i 1 x j

3. gi ( X *) bi 0, i 1,.., m
4. ui ( gi ( X *) bi ) 0, i 1,.., m
5. x *j 0, j 1,..., n
6. ui 0, i 1,..., m
For general constraint optimization problems, KKT conditions are necessary
conditions.
If f ( X ) is concave and all gi ( X ) are convex, then KKT conditions are necessary
and sufficient.

134

Example 1
max f ( X ) ln( x1 1) x2

s.t.

2 x1 x2 3

x1 , x2 0

It can be verified that f ( X ) is concave and g1 ( X ) 2 x1 x2 is convex. Any

points satisfying KKT conditions will be optimal.
Solution.
g
g
f
1
f
,

1, 1 2 , 1 1
x1 x1 1 x2
x1
x2

KKT Conditions:
1

m
g
f
ui i 0, j 1,...n ;
x j i 1 x j

2 x *j (

m
g
f
ui i ) 0, j 1,...n
x j i 1 x j

3 gi ( X *) bi 0, i 1,.., m ;

4 ui ( gi ( X *) bi ) 0, i 1,.., m

5 x *j 0, j 1,..., n

6 ui 0, i 1,..., m

1.1

g
f
1
u1 1
2u1 0 ,
x1
x1 x1 1

2.1

x1 (

3
5

1
2u1 ) 0
x1 1

2 x1 x2 3 0
x1 , x2 0

1.2

g
f
u1 1 1 u1 0
x2
x2

2.2

x2 (1 u1 ) 0

u1 (2 x1 x2 3) 0

u1 0

Analysis.
1
1
x1 1
1
1 u1 2u1
From 1.2: u1 1 , so
x1 1

From 5.: x1 0 , so

135

1
1
1
2u1 or
2u1 0 or
2u1 0 (*)
x1 1
x1 1
x1 1

From 2.1: x1 (

1
1
2u1 ) 0 and
2u1 0 : x1 0
x1 1
x1 1

Since u1 1 , so u1 0 . From 4:
2 x1 x2 3 0

Since x1 0 , x2 3
Since x2 3 0 and from 2.2: (1 u1 ) 0 .
So u1 1 .
Optimal solution: x1* 0 , x2* 3
and u1 1

Example 2.
max f ( X ) x1 (30 x1 ) x2 (50 2 x2 ) 3x1 5x2 10 x3

s.t.

x1 x2 x3 0

x3 17.25
x1 , x2 , x3 0
f ( X ) is concave and gi ( X ), i 1,2 are convex

f
f
f
30 2 x1 3 ,
50 4 x2 5 ,
10
x1
x2
x3
g
g
g 2
g
g1
g
1 , 1 1 , 1 1 ,
0, 2 0, 2 1
x3
x3
x1
x2
x1
x2

KKT Conditions:
1

m
g
f
ui i 0, j 1,...n ;
x j i 1 x j

3 gi ( X *) bi 0, i 1,.., m ;

2 x *j (

m
g
f
ui i ) 0, j 1,...n
x j i 1 x j

4 ui ( gi ( X *) bi ) 0, i 1,.., m

136

5 x *j 0, j 1,..., n

6 ui 0, i 1,..., m

g
g
f
(u1 1 u2 2 ) 30 2 x1 3 u1 0 (1)
x1
x1
x1
g
g
f
1.2
(u1 1 u2 2 ) 50 4 x2 5 u1 0 (2)
x2
x2
x2
g
g
f
1.3
(3)
(u1 1 u2 2 ) 10 u1 u2 0
x3
x3
x3

1.1

2.1 x1 (30 2 x1 3 u1 ) 0
2.2 x2 (50 4 x2 5 u1 ) 0
2.3 x3 (10 u1 u2 ) 0
3.1 x1 x2 x3 0
3.2 x3 17.25 0
4.1 u1 ( x1 x2 x3 ) 0
4.2 u2 ( x3 17.25) 0
5 x1 , x2 , x3 0
6. u1 , u2 0

(4)
(5)
(6)
(7)
(8)
(9)
(10)
(11)
(12)

Analysis.
CASE A. Assume that u1 u2 0 .
From (6): x3 (10 u1 u2 ) 0 10 x3 0 or x3 0 .
From (7): x1 x2 x3 0 x1 x2 0 .
So x1 x2 0 . If this is the case, then
From (2),

g
g
f
(u1 1 u2 2 ) 50 4 x2 5 u1 0
x2
x2
x2

it requires that u1 45 , this is impossible. So u1 u2 0 is impossible.

CASE B. Assume that u1 0, u2 0 .
From (10): u2 ( x3 17.25) 0 x3 17.25 0 or x3 17.25 .
From (6): x3 (10 u1 u2 ) 0 10 u2 0 or u2 10 .
This is impossible.
137

CASE C. Assume that u1 0, u2 0 .

From (9): u1 ( x1 x2 x3 ) 0 x1 x2 x3 0 or
x1 x2 x3

(13)

From (6): x3 (10 u1 u2 ) 0 , or

x3 (10 u1 ) 0 .

(14)

If x1 0 then from (1) 30 2 x1 3 u1 0

It requires u1 27 and from (3): 10 u1 u2 0 , we have 10 u1 0
or u1 10 . This is impossible.
If x2 0 , then from (2): 50 4 x2 5 u1 0 or 50 5 u1 0
and (3): 10 u1 0
it requires that u1 45 and u1 10 . This is also impossible.
So when u1 0, u2 0 , it must have x1 0 and x2 0 . From (13): x1 x2 x3
we have x3 0 .
Since x3 0 , then from (14): x3 (10 u1 ) 0 . So we have u1 10 .
Since x1 0 and x2 0 ,
from (4): x1 (30 2 x1 3 u1 ) 0 , we have: 30 2 x1 3 u1 0
From (5): x2 (50 4 x2 5 u1 ) 0 , we have: 50 4 x2 5 u1 0 .
Since u1 10 , we have:
30 2 x1 3 10 0 and 50 4 x2 5 10 0 , or
x1

30 13
50 15
8.5 , x2
8.75
4
2

x3 x1 x2 8.5 8.75 17.25

So ( x1 , x2 , x3 ) (8.5,8.75,17.25) is optimal with f ( X ) = 225.375

138

CASE D. Assume that u1 0, u 2 0 .

From (9): u1 ( x1 x2 x3 ) 0 , we have: x1 x2 x3 0
From (10), u2 ( x3 17.25) 0 , we have: x3 17.25 0 . Or x3 17.25 .
So x1 x2 17.25 . If x1 0, x2 17.25 , then from (5): x2 (50 4 x2 5 u1 ) 0
45 4 x2 u1 0 or u1 45 4 17.25 0 . This is impossible.

If x2 0, x1 17.25 , then from (4): x1 (30 2 x1 3 u1 ) 0

27 2 x1 u1 0 or u1 27 2 17.25 0 . This is also impossible.

So x1 0, x2 0 , then from (4) and (5):

27 2 x1 u1 0

(15)

45 4 x2 u1 0

Leading to 27 2 x1 45 4 x2 or 2 x2 x1 9 . Since x1 x2 17.25 , then solving

2 x 2 x1 9
results in x1 8.5 and x2 8.75 .

x1 x 2 17.25

From (15), u1 27 2 8.5 10

From (6): x3 (10 u1 u2 ) 0 . Since x3 17.25 0 , we have:
u1 u 2 10

This is impossible for u1 10 0 and u2 0

Conclusion. It is only possible when u1 10 0 and u 2 0 (in CASE C),
Example 2 problem has optimal solution with ( x1* , x2* , x3* ) (8.5,8.75,17.25) .

139

6. Convex programming for linearly constrained optimization

max f ( X )

AX B bi , i 1,..., m

xj 0

Frank-Wolfe Method
1. Starting from X (0) = ( x1( 0) , x2(0) ,..., xn(0) )
2. At X X ( 0) evaluate:

f
f
| X X ( 0 ) . Let c j
| (0)
x j
x j X X

3. Solve
n

max g ( x ) c j x j
j 1

s.t. AX B bi , i 1,..., m
xj 0

Let the optimal solution be X LP

Let X X (0) t ( X LP X ( 0) ),0 t 1
Find t* from max h(t ) f ( X (t ))
Let X (1) X (0) t * ( X LP X ( 0) )
Goto Step 2 with X (1) replacing X ( 0)
4. Stop when stopping criterion is reached

Example
max f ( X ) 5x1 x12 8x2 2 x22

s.t. 3x1 2 x2 6
x1 , x2 0

140

Solution
f
f
5 2 x1 ,
8 4 x2 . If we let 5 2 x1 0,8 4 x2 0 , it will lead to
x1
x2

x1 2.5, x2 2 . It does not satisfy the constraint 3x1 2 x2 6 .

(We can try to use KKT conditions.)

Use Frank-Wolfe method starting from ( x1( 0) , x2( 0) ) (0,0) :
Solve:

max

s.t.

f
f
|x1 0 x1
| x 0 x 2
x1
x2 2

3x1 2 x2 6
x1 , x2 0

or
max 5x1 8 x2

s.t.

3x1 2 x2 6
x1 , x2 0

The solution is: X LP ( x1 , x2 ) (0,3) . Let

( x1(1) , x2(1) ) (0,0) t[(0,3) (0,0)] (0,3t )

For f ( X ) 5x1 x12 8x2 2 x22

h(t ) f (0,3t ) 8(3t ) 2(3t ) 2 24t 18t 2
dh
2
24 36t 0, t*
dt
3
2
X (1) ( x1(1) , x2(1) ) (0,0) ((0,3) (0,0)) (0,2)
3

141

Go back to Step 2 and for

f
f
5 2 x1 ,
8 4 x2 ,
x1
x2

then at X (1) ( x1(1) , x2(1) ) (0,2) :

Solve:

f
f
|x1 0 5,
|x 2 0
x1
x2 2

max 5x1

s.t.

3x1 2 x2 6

x1 , x2 0

Solution is: X LP (2,0)

( x1( 2) , x2( 2) ) (0,2) t[(2,0) (0,2)] (2t,2 2t )
h(t ) f (2t,2 2t ) 5(2t ) (2t ) 2 8(2 2t ) 2(2 2t ) 2 8 10t 12t 2
dh
5
10 24t 0, t*
dt
12
X ( 2 ) ( x1( 2 ) , x2( 2 ) ) (0,2)

5
5 7
((2,0) (0,2)) ( , )
12
6 6

Continue.
The real optimal solution is (1, 1.5).

142