Professional Documents
Culture Documents
Lagrange Multipliers and The Karush-Kuhn-Tucker Conditions: March 20, 2012
Lagrange Multipliers and The Karush-Kuhn-Tucker Conditions: March 20, 2012
Karush-Kuhn-Tucker conditions
Goal:
Want to find the maximum or minimum of a function subject to
some constraints.
Formal Statement of Problem:
Given functions f , g1 , . . . , gm and h1 , . . . , hl defined on some
domain Ω ⊂ Rn the optimization problem has the form
∇x f (x∗ ) = 0
vt (∇2 f (x∗ )) v ≥ 0, ∀v ∈ Rn
where
∂ 2 f (x) ∂ 2 f (x)
∂x2
··· ∂x1 ∂xn
1
∇2 f (x) =
.. .. ..
. . .
∂ 2 f (x) ∂ 2 f (x)
∂xn ∂x1
··· ∂x2 n
Unconstrained Maximization
Assume:
Let f : Ω → R be a continuously differentiable function.
Necessary and sufficient conditions for local maximum:
x∗ is a local maximum of f (x) if and only if
1 f has zero gradient at x∗ :
∇f (x∗ ) = 0
vt (∇2 f (x∗ )) v ≤ 0, ∀v ∈ Rn
where
∂ 2 f (x) ∂ 2 f (x)
∂x2
··· ∂x1 ∂xn
1
∇2 f (x) =
.. .. ..
. . .
∂ 2 f (x) ∂ 2 f (x)
∂xn ∂x1
··· ∂x2 n
Constrained Optimization:
Equality Constraints
Tutorial Example
Problem:
This is the constrained optimization problem we want to solve
where
x2
x1
x1 + x2 = 2
x1 + x2 = 1
x1 + x2 = 0
x1 + x2 = −1
iso-contours of f (x)
x1 + x2 = −2
f (x) = x1 + x2
Tutorial example - Feasible region
x2
x1
x1 + x2 = 2
x1 + x2 = 1
x1 + x2 = 0
x1 + x2 = −1
iso-contours of f (x)
x1 + x2 = −2
x2
feasible point xF
x1
Given a point xF on the constraint surface
x2
δx
x1
x2
−∇x f (xF )
x1
x2
δx
x2
∇x h(xF )
x1
x2
∇x h(xF ) x1
x2
x1
where µ is a scalar.
When this occurs
• If δx is orthogonal to ∇x h(xF ) then
where µ is a scalar.
When this occurs
• If δx is orthogonal to ∇x h(xF ) then
x2
critical point
x1
critical point
∇x f (x∗ ) = µ ∇x h(x∗ )
From this fact Lagrange Multipliers make sense
Problem:
Consider this constrained optimization problem
where
x2
minimum of f (x)
x1
iso-contours of f (x)
x2
x1
iso-contours of f (x)
x2
How can we recognize xF
is at a local minimum?
x1
x2
How can we recognize xF Unconstrained minimum
is at a local minimum? of f (x) lies within
the feasible region.
x1
Problem:
Our constrained optimization problem
where
Problem:
This is the constrained optimization problem we want to solve
where
x2
minimum of f (x)
x1
iso-contours of f (x)
x2
x1
iso-contours of f (x)
x2
Is xF at a local minimum?
x1
x2 Unconstrained local
How can we tell if xF minimum of f (x)
is at a local minimum? lies outside of the
feasible region.
x1
x2 Unconstrained local
How can we tell if xF minimum of f (x)
is at a local minimum? lies outside of the
feasible region.
x1
x2
x1
x1
X Is a constrained
x2 local minimum as
−∇x f (xF ) points
away from the
feasible region
x1
Given
1 ∇x L(x∗ , λ∗ ) = 0
2 λ∗ ≥ 0
3 λ∗ g(x∗ ) = 0
4 g(x∗ ) ≤ 0
5 Plus positive definite constraints on ∇xx L(x∗ , λ∗ ).
1 ∇x L(x∗ , λ∗ ) = 0
2 λ∗j ≥ 0 for j = 1, . . . , m
3 λ∗j g(x∗ ) = 0 for j = 1, . . . , m
4 gj (x∗ ) ≤ 0 for j = 1, . . . , m
5 Plus positive definite constraints on ∇xx L(x∗ , λ∗ ).
KKT for multiple equality & inequality constraints
Given the constrained optimization problem
min f (x)
x∈R2
subject to