Line

LineSearch
Search

G
e
o
r
gi
a
In
st
i
t
ut
eo
fT
ec
h
n
ol
o
gy
Optimization in Engineering Design
S
y
s
t
e
msR
ea
l
i
za
ti
o
nLa
b
o
r
at
or
y

1
 Line
LineSearch
Search

Line
Linesearch
searchtechniques
techniquesare
areininessence
essenceoptimization
optimizationalgorithms
algorithms
for one-dimensional minimization problems.
for one-dimensional minimization problems.
They
They areare often
often regarded
regarded as as the
the backbones
backbones ofof nonlinear
nonlinear
optimization algorithms.
optimization algorithms.
Typically,
Typically,these
thesetechniques
techniquessearch
searchaabracketed
bracketedinterval.
interval.
Often,
Often,unimodality
unimodalityisisassumed.
assumed.

a x* b

Exhaustive search requires N = (b-a)/ + 1

calculations to search the above interval, where
 is the resolution.

G
e
o
r
gi
a
In
st
i
t
ut
eo
fT
ec
h
n
ol
o
gy
Optimization in Engineering Design
S
y
s
t
e
msR
ea
l
i
za
ti
o
nLa
b
o
r
at
or
y

2
 Basic
Basicbracketing
bracketingalgorithm
algorithm

a x1 x2 b

Two point search (dichotomous search) for finding the solution to

minimizing ƒ(x):
0) assume an interval [a,b]
1) Find x1 = a + (b-a)/2 - /2 and x2 = a+(b-a)/2 + /2 where  is the
resolution.
2) Compare ƒ(x1) and ƒ(x2)
3) If ƒ(x1) < ƒ(x2) then eliminate x > x2 and set b = x2
If ƒ(x1) > ƒ(x2) then eliminate x < x1 and set a = x1
If ƒ(x1) = ƒ(x2) then pick another pair of points
4) Continue placing point pairs until interval < 2 
G
e
o
r
gi
a
In
st
i
t
ut
eo
fT
ec
h
n
ol
o
gy
Optimization in Engineering Design
S
y
s
t
e
msR
ea
l
i
za
ti
o
nLa
b
o
r
at
or
y

3
 Fibonacci
FibonacciSearch
Search

Fibonacci numbers are:

1,1,2,3,5,8,13,21,34,.. that is , the sum of the last 2 numbers

Fn = Fn-1 + Fn-2

L2 L3


a x1 x2 b
L2
L1 L1 = L2 + L3
It can be derived that
Ln = (L1 + Fn-2 ) / Fn
G
e
o
r
gi
a
In
st
i
t
ut
eo
fT
ec
h
n
ol
o
gy
Optimization in Engineering Design
S
y
s
t
e
msR
ea
l
i
za
ti
o
nLa
b
o
r
at
or
y

4
 Golden
GoldenSection
Section

b a

a-b Discard

a b

In Golden Section, you try to have b/(a-b) = a/b

which implies b*b = a*a - ab
Solving this gives a = (b ± b* sqrt(5)) / 2
a/b = -0.618 or 1.618 (Golden Section ratio)
See also 36 in your book for the derivation.
Note that 1/1.618 = 0.618

G
e
o
r
gi
a
In
st
i
t
ut
eo
fT
ec
h
n
ol
o
gy
Optimization in Engineering Design
S
y
s
t
e
msR
ea
l
i
za
ti
o
nLa
b
o
r
at
or
y

5
 Bracketing
Bracketingaa Minimum
Minimumusing
usingGolden
GoldenSection
Section
Initialize:
x1 = a + (b-a)*0.382
x2 = a + (b-a)*0.618
f1 = ƒ(x1)
a x1 x2 b
f2 = ƒ(x2)
Loop:
if f1 > f2 then
a = x1; x1 = x2; f1 = f2
x2 = a + (b-a)*0.618
f2 = ƒ(x2)
else
b = x2; x2 = x1; f2 = f1
x1 = a + (b-a)*0.382
f1 = ƒ(x1)
endif G
e
o
r
gi
a
In
st
i
t
ut
eo
fT
ec
h
n
ol
o
gy
Optimization in Engineering Design
S
y
s
t
e
msR
ea
l
i
za
ti
o
nLa
b
o
r
at
or
y

6
 Newton's
Newton'sMethods
Methods

IfIf your
your function
function isis differentiable,
differentiable, then
then you
you do
do not
not need
need to
to
evaluate two points to determine the region to be discarded.
evaluate two points to determine the region to be discarded.
Get
Get the
the slope
slope andand the
the sign
sign indicates
indicates which
which region
region to
to
discard.
discard.

Basic premise in Newton-Raphson method:

Root finding of first derivative is equivalent to finding optimum
(if function is differentiable).

Method is sometimes referred to as a line search by curve fit

because it approximates the real (unknown) objective
function to be minimized.

G
e
o
r
gi
a
In
st
i
t
ut
eo
fT
ec
h
n
ol
o
gy
Optimization in Engineering Design
S
y
s
t
e
msR
ea
l
i
za
ti
o
nLa
b
o
r
at
or
y

7
 Newton-Raphson
Newton-RaphsonMethod
Method
Used for general root finding according to
y(x0)
x1 = x 0 -
y’(x0)

Based on Taylor expansion:

y(x) = y(xk) +y’(xk)(x-xk) + 0.5 y’’(x k) (x-xk)2 + O(x3)

Discard third order term

q(x) = y(xk) +y’(xk)(x-xk) + 0.5 y’’(x k) (x-xk)2

Differentiate and you get

q’(x) = 0 +y’(xk) + y’’(x k) (x-xk)

and you want q’(x) = 0 which gives you an extreme point.

Hence, the Newton Raphson method for finding an extreme point uses
y’(xk)
xk+1 = xk -
y’’(x k)
Question: How many iterations are necessary to solve an optimization problem with a
quadratic objective function ?
Ge or
g i
a I
nstit
u t
e ofTech no l
o
gy
Optimization in Engineering Design
Sy s
tem s R eal
izati
on Lab orat
or
y

8
 False
FalsePosition
PositionMethod
Methodor
orSecant
Secant Method
Method

Second order information is expensive to calculate (for

multi-variable problems).
Thus, try to approximate second order derivative.\

Replace y''(xk) in Newton Raphson with

y' ( x k )  y' ( x k 1 )
y' ' ( x k ) 
x k  x k 1

Hence, Newton Raphson becomes

x k  x k 1
x k 1  x k  ( y' ( x k ))
y' ( x k )  y' ( x k 1 )

Main advantage is no second derivative requirement

Question: Why is this an advantage ?
G
e
o
r
gi
a
In
st
i
t
ut
eo
fT
ec
h
n
ol
o
gy
Optimization in Engineering Design
S
y
s
t
e
msR
ea
l
i
za
ti
o
nLa
b
o
r
at
or
y