Theory of constrained

optimization
Nur Aini Masruroh
Scope

Lagrange function

Necessary and sufficient conditions for

equality constraints

Necessary and sufficient conditions for

inequality constraints

Necessary and sufficient conditions for

equality and inequality constraints
A general form for constrained
optimization problem
Minimize f(x)
Subject to h
i
(x) = i = !"#"m
g
j
(x) $ j = !"#"r
x
%
= &x
!
x
'
x
(
# x
n
) *
n
+uestion, does the theory of unconstrained
o-timization a--ly for constrained -roblem.

/n general form
minimize f(x)
subject to h
i
(x) = i = !" #" m
g
j
(x) $ j = !" #" r
x = &x
!
x
'
# x
n
)
%
/t can be con0erted into,
Minimize

L(x" 1) is called Lagrange function and 1
i
and u
j
are
called Lagrange multi-liers


+ +
m
i
r
j
j j i i
x g u x h w x f u w x L
1 1
) ( ) ( ) ( ) , , (
Lagrange function
Lagrange function: remark

2unction L transforms a constrained o-timization

into an unconstrained one

2unction L can be used to determine the first order

condition (234) necessary condition for o-timal
solutions
Necessary and suficient
conditions for equality
constraints

/n general" a -roblem 1ith equality constraints can

be ex-ressed as
Minimize f(x)
Subject to h
i
(x) = i = !"#"m
x
%
= &x
!
x
'
x
(
# x
n
)
5efine Lagrange function as

+
m
i
i i
x h w x f w x L ) ( ) ( ) , (
Necessary conditions of
optimality
m i x h
w
L
x h w x f L
i
i
m
i
i i
,..., 2 , 1 0 ) (
0 ) ( ) (

+

*emar6s,
!7 /f x8 satisfies the conditions" then L(x8" 18) = f(x8) L gi0es the same
o-timal solutions
'7 x8 is a stationary -oint for L7 /t is generally different from the stationary -oint
for f(x)
(7 1
i
can be -ositi0e or negati0e
97 As necessary conditions require first order deri0ati0es" they are also called
first order conditions (234)
Suficient conditions of
optimality

:L(x8"18) is -ositi0e definite x8 is a local minimum

:L(x8"18) is negati0e definite x8 is a local maximum

1here :L(x8"18) is second order deri0ati0es o0er x and 17
Note, as sufficient conditions require second order deri0ati0es" they are
also called second order conditions (S34)
( )
m j
n i
x
h
x
h
h w f
w x H
n m m n
T
n m
i
j
m n
i
j
n n
i i
L
,..., 2 , 1
,..., 2 , 1
0
*) *, (
) ( ) (
2 2

1
1
1
1
1
]
1

,
_

,
_

+ +

Try the folloing e!ample

!7 Min f(x) = 9x
!
'
; <x
'
'
Subject to h(x) = 'x
!
; (x
'
= > =
'7 Min f(x
!
" x
'
) = 9x
!
'
; <x
'
'
Subject to h(x
!
" x
'
) = 'x
!
; (x
'
= ? =
Necessary and suficient
conditions for inequality
constraints
/n general" a -roblem 1ith inequality constraints can
be ex-ressed as
Minimize f(x)
Subject to g
i
(x) $ i = !"#"r
x
%
= &x
!
x
'
x
(
# x
n
)
@hat are the necessary conditions for this -roblem.
%o con0ert inequality constraints into equalities" letAs
introduce slac6 0ariables Si(i=!"#"r) as follo1s
g
i
(x) ; s
i
'
=

+ +
i
i i i
s x g u x f L ) ) ( ( ) ( Let
2
A--ly the -re0ious 234 conditions on L,
r i
s x g L
s u L
x g u x f L
i i u
i i s
i
i i x
,..., 2 , 1
0 ) (
0
0 ) ( ) (
2

+

+

"elation beteen u
i
and g
i
#!$
4onsider g
i
(x) ; s
i
'
=
/f s
i
=" then g
i
(x) = g
i
(x) is a binding (acti0e) constraint
/f s
i
B" then g
i
(x) C g
i
(x) is not binding (inacti0e)
2or L(x8" u8" s8) = f(x8) u
i
&g
i
(x) ; s
i
'
) =
@hen s
i
B"
u
i
=
g
i
(x) C
u
i
g
i
(x) =
@hen s
i
= "
g
i
(x) =
u
i
B
Necessary and suficient
conditions of optimality for
general constrained problems
Deneral -roblem,
Minimize f(x)
Subject to h
i
(x) = i = !"#"m
g
i
(x) $ i = !"#"r
x
%
= &x
!
x
'
x
(
# x
n
)


+ +
m
i
r
j
j j i i
x g u x h w x f u w x L
1 1
) ( ) ( ) ( ) , , (
%eneral &arush'&uhn'Tucker
#&&T$ conditions #necessary$
!7 Linear de-endence of f" h" g
'7 4om-lementary conditions
u
i
g
j
(x) = j=!"#"r
u
j
E
(7 4onstraint feasibility
h
i
(x) = i=!"#"m
g
j
(x) $ j=!"#"r
0 ) ( ) ( ) (
1 1
+ +


r
j
j j
m
i
i i
x g u x h w x f
Sufficient Conditions of Optimality
(SOC)

:L(x8" 18" u8) is -ositi0e definite x8 is a

local minimum

:L(x8" 18" u8) is negati0e definite x8 is a

local maximum
@here :L(x8" 18" u8) is :essian matrix for L
1ith res-ect to x8" 18" and u8
Suficient conditions for the
uniqueness of a local optimum
/f f(x) is con0ex (conca0e) and the feasible
region is con0ex and a local minimum
(maximum) exists at x8"
!7 x8 is global minimum (maximum)
'7 %he FF% conditions are both necessary and
sufficient
The procedure for sol(ing &&T
equations
!7 Assume no acti0e inequality u
j
=
'7 Sol0e the equations for x" the multi-lier 1
i
of
the equalities" and the multi-lier u
j
of the
acti0e inequality (for the first iteration there
are none)
(7 /f it satisfies all the constraints" S%3G"
solution found7 3ther1ise go to ste- 97
97 Set u
j
B " then go to ste- '7
)!ample
Minimize f = x
!
'
; x
'
'
Subject to g
!
= ! = x
!
$
g
'
= x
!
= ( $
g
(
= ' = x
'
$
g
9
= x
'
= 9 $