NSW Notes

Lagrangian Optimization

We are interested in solving the constrained optimization problem

minimize

f (x)

subject to

h(x) = b

over

x X.

(P)

The variables x must belong to the constraint region X , where X is a subset of Rp .

The variables must satisfy the functional constraints h(x) = b, where h : X Rd and b Rd .
Variables that belong to the constraint region and satisfy the functional constraints are called feasible and belong
to the feasible set
X (b) = {x X : h(x) = b}.
The variables are judged according to the relative magnitude of the objective function f : X R.
A feasible solution to the optimization, x , is called optimal when f (x ) f (x) for all x X (b).
In general, an optimum x need not exist.
Inequality constraints hj (x) bj can be rewritten as equality constraints hj (x) + zj = bj , where zj 0. Then an

optimization of the form (P) can be solved over variables (x, z) X Rp+ . The variables z are called slack variables.
maxxX (b) f (x) = minxX (b) f (x), so maximizing is the same as minimizing.
The set X could incorporate the functional constraints {x : h(x) = b}, but generally we assume X is a set where the
unconstrained optimization, minxX f (x), is easier to solve. For example, X might be Rp+ and minxX f (x) might be
solved by dierentiating and nding local minima. What makes the problem dicult is the functional constraints. To
get some solution, we might penalize going over these constraints. If so, we could remove these constraints and solve a
dierent optimization
d

minimize f (x) +
j (bj hj (x)) over x X .
(L )
j=1

In the above optimization, the objective function

L(x; ) = f (x) + T (b h(x))
is called the Lagrangian of the optimization problem (P).
We call the components of the vector, Rd , Lagrange multipliers.
We use x () to notate its optimum of (L ), if it exists.
For each inequality constraint hj (x) bj , a variable zj 0 is added and the term j (bj hj (x)) is replaced by
j (bj zj hj (x)). Because we introduced a new term j zj , for a nite solution to (L ), we require j 0 and
thus, after minimizing over zj , we have zj j = 0. This equality is called complementary slackness.
If (x , z ) our optimal Lagrangian solution is also feasible then complementary slackness states j (bj hj (x )) = 0.
The unconstrained optimization (L ) is, in principle, much easier to solve and a good choice of can be used to penalize
constraints that are not satised. For example, if we solve the optimization problem (L ) with optimum x () and found
that bi > h(xi ) then, intuitively, we would make i bigger in order to make the price of overrunning constraint i higher.
Hopefully, this would then make the solution of (L ) closer to (P). The minimum of the Lagrangian is always smaller.
Lemma 2 (Weak Duality). For all Rd
min L(x; ) min f (x)

xX

Proof.
min L(x; ) min L(x; ) = min

xX

xX (b)

(9)

xX (b)

xX (b)

f (x) + T (b h(x)) = min f (x).

xX (b)
=0

In fact, if we can make the two solutions equal then we have optimality.
Theorem 1 (Lagrangian Suciency Theorem). If there exists Lagrange multipliers and a solution x ( ) to (L )
such that x ( ) is feasible for (P) then x ( ) solves the optimization problem (P).

NSW Notes

Proof. As x ( ) is feasible then certainly f (x ( )) minxX (b) f (x). Now,

f (x ( )) = f (x ( )) + T (b h(x ( )))

(as x ( ) is feasible)

(as x is optimal for (L ))

min f (x).

(by (9))

= min L(x; )
xX

xX (b)

So, f (x ( )) = minxX (b) f (x).

This result gives a simple procedure to solve an optimization:
1. Take your optimization (P) and, if necessary, add slack variables to make all inequality constraints equalities.
2. Write out the Lagrangian and solve optimization (L ) for x ().
3. By solving the constraints h(x ()) = b over , nd a so that x = x ( ) is feasible.
By Lagrangian Suciency Theorem, x is optimal.
Example 1.

minimize

x2 + 2y 2

subject to

x+y =1

over

x 0, y 0.

Answer. We do not need to add slack variables.

The Lagrangian for this optimization is: L(x, y; ) = x2 + 2y 2 + (1 x y).
So we solve Lagrangian problem: minimize x2 + 2y 2 + (1 x y) over x 0, y 0.
This is minimized when L

L
= 0 and
= 0 which implies x () =
and y () = .
x
y
2
4
For feasibility, we need x () + y () = 1. So = 4/3. Thus by the Lagrangian Suciency Theorem x = 2/3 and
y = 1/3 is the optimal solution.
Example 2.

2 x log y subject to x + y b over x 0, y 0.

Answer. Add a slack variable: minimize 2 x + log y subject to x + y + z = b over x 0, y 0, z 0.

We now have Lagrangian: L(x, y, z; ) = 2 x log y + (b x y z).
So we solve

minimize 2 x log y + (b x y z) over x 0, y 0, z 0.

For b > 0,

minimize

This is minimized when

L
=0
x

and

L
=0
y

1
which implies = 0
x

and

1
= 0.
y

So x () = 1/2 and y () = 1/. We, now, require a feasible solution x () + y () + z () = b. We may exclude the
possibility that z () > 0, because by complementary slackness z () = 0, so if z () > 0 then = 0 and so we cannot
nd a nite optimum for the Lagrangian problem. So, noting that x () = 1/2 , y () = 1/ and z () = 0, feasibility
reduces to 1/2 1/ = b, or equivalently the quadratic equation b2 + 1 = 0. Recalling that we must have 0
for a nite solution to the Lagrangian optimization, this quadratic has one negative solution, namely,

1 1 + 4b
1 + 1 + 4b
2b + 1 1 + 4b
=
, and thus y =
, x =
.
2b
2
2