Non-Linear Programming

2011 Daniel Kirschen and University of Washington

1
 Motivation
Method of Lagrange multipliers
Very useful insight into solutions
Analytical solution practical only for small problems
Direct application not practical for real-life problems
because these problems are too large
Difficulties when problem is non-convex
Often need to search for the solution of practical
optimization problems using:
Objective function only or
Objective function and its first derivative or
Objective function and its first and second derivatives

2011 Daniel Kirschen and University of Washington

2
 Nave One-Dimensional Search
Suppose:
That we want to find the value of xthat minimizes
f(x)
That the only thing that we can do is calculate the
value of f(x) for any value of x
We could calculatef(x)for a range of values of
xand choose the one that minimizesf(x)

2011 Daniel Kirschen and University of Washington

3
 Nave One-Dimensional Search
f(x)

x
Requires a considerable amount of
computing time if range is large and a good
accuracy is needed
2011 Daniel Kirschen and University of Washington
4
 One-Dimensional Search
f(x)

x0 x

2011 Daniel Kirschen and University of Washington 5

 One-Dimensional Search
f(x)

x0 x1 x

2011 Daniel Kirschen and University of Washington 6

 One-Dimensional Search
f(x)

Currentsearch
range

x0 x1 x2 x

Ifthefunctionisconvex,wehavebracketedtheoptimum

2011 Daniel Kirschen and University of Washington 7

 One-Dimensional Search
f(x)

x0 x1 x3 x2 x

Optimumisbetweenx1andx2
Wedonotneedtoconsiderx0anymore
2011 Daniel Kirschen and University of Washington 8
 One-Dimensional Search
f(x)

x0 x1 x4 x3 x2 x

Repeattheprocessuntiltherangeissufficientlysmall

2011 Daniel Kirschen and University of Washington 9

 One-Dimensional Search
f(x)

x0 x1 x4 x3 x2 x

Theprocedureisvalidonlyifthefunctionisconvex!

2011 Daniel Kirschen and University of Washington 10

 Multi-Dimensional Search
Unidirectional search not applicable
Nave search becomes totally impossible as
dimension of the problem increases
If we can calculate the first derivatives of the
objective function, much more efficient
searches can be developed
The gradient of a function gives the direction
in which it increases/decreases fastest

2011 Daniel Kirschen and University of Washington

11
 Steepest Ascent/Descent
Algorithm
x 2

2011 Daniel Kirschen and University of Washington

x1 12
 Steepest Ascent/Descent
Algorithm
x 2

2011 Daniel Kirschen and University of Washington

x1 13
 Unidirectional Search
Objectivefunction

2011 Daniel Kirschen and University of Washington

Gradientdirection 14
 Steepest Ascent/Descent
Algorithm
x 2

2011 Daniel Kirschen and University of Washington

x1 15
 Steepest Ascent/Descent
Algorithm
x 2

2011 Daniel Kirschen and University of Washington

x1 16
 Steepest Ascent/Descent
Algorithm
x 2

2011 Daniel Kirschen and University of Washington

x1 17
 Steepest Ascent/Descent
Algorithm
x 2

2011 Daniel Kirschen and University of Washington

x1 18
 Steepest Ascent/Descent
Algorithm
x 2

2011 Daniel Kirschen and University of Washington

x1 19
 Steepest Ascent/Descent
Algorithm
x 2

2011 Daniel Kirschen and University of Washington

x1 20
 Steepest Ascent/Descent
Algorithm
x 2

2011 Daniel Kirschen and University of Washington

x1 21
 Steepest Ascent/Descent
Algorithm
x 2

2011 Daniel Kirschen and University of Washington

x1 22
 Steepest Ascent/Descent
Algorithm
x 2

2011 Daniel Kirschen and University of Washington

x1 23
 Steepest Ascent/Descent
Algorithm
x 2

2011 Daniel Kirschen and University of Washington

x1 24
 Steepest Ascent/Descent
Algorithm
x 2

2011 Daniel Kirschen and University of Washington

x1 25
 Choosing a Direction
Direction of steepest ascent/descent is not
always the best choice
Other techniques have been used with varying
degrees of success
In particular, the direction chosen must be
consistent with the equality constraints

2011 Daniel Kirschen and University of Washington

26
 How far to go in that direction?
Unidirectional searches can be time-
consuming
Second order techniques that use information
about the second derivative of the objective
function can be used to speed up the process
Problem with the inequality constraints
There may be a lot of inequality constraints
Checking all of them every time we move in one
direction can take an enormous amount of
computing time
2011 Daniel Kirschen and University of Washington
27
 Handling of inequality constraints
x2
HowdoIknowthat
Ihavetostophere? Moveinthedirection
ofthegradient

x1

28
2011 Daniel Kirschen and University of Washington
 Handling of inequality constraints
x2
HowdoIknowthat
Ihavetostophere?

x1

29
2011 Daniel Kirschen and University of Washington
 Penalty functions

Penalty

xmin xmax
30
2011 Daniel Kirschen and University of Washington
 Penalty functions
Replace enforcement of inequality constraints
by addition of penalty terms to objective
function
Penalty

K(xxmax)2

xmin xmax
31
2011 Daniel Kirschen and University of Washington
 Problem with penalty functions
Stiffness of the penalty function must be increased
progressively to enforce the constraints tightly enough
Not very efficient method

Penalty

xmin xmax
32
2011 Daniel Kirschen and University of Washington
 Barrier functions

Barrier cost

xmin xmax
33
2011 Daniel Kirschen and University of Washington
 Barrier functions
Barrier must be made progressively closer to the limit
Works better than penalty function
Interior point methods
Barrier cost

xmin xmax
34
2011 Daniel Kirschen and University of Washington
 Non-Robustness
Differentstartingpointsmayleadtodifferentsolutionsifthe
problemisnotconvex
x2

A
D

x1
B C

X Y
2011 Daniel Kirschen and University of Washington 35
 Conclusions

Very sophisticated non-linear programming

methods have been developed
They can be difficult to use:
Different starting points may lead to different
solutions
Some problems will require a lot of iterations
They may require a lot of tuning

2011 Daniel Kirschen and University of Washington

36