Professional Documents
Culture Documents
Steps in Formulisation of An Optimisation Problem
Steps in Formulisation of An Optimisation Problem
OPTIMISATION PROBLEM
NEED IDENTIFICATION
FORMULATE CONSTRAINTS
OBTAIN SOLUTION(S)
Using Symmetry
1
Formulate the constraints
• The tensile and compressive stress must not be more
than the corresponding allowable strength
• AssumeS = S = 500MPa
yt yc
E=200 Gpa
Stress Constraints
P cos ecθ
≤ S yc
2 A1
P cot θ
≤ S yt
2 A2
P cos ecα
≤ S yt
2 A3
P
( cot θ + cot α ) ≤ S yc
2 A4
2
Stability consideration
• Buckling of the compression members AB, BD, and DE
• Euler buckling conditions
P π EA12
≤
2sin θ 1.281l 2
P π EA42
( cot θ + cot α ) ≤
2 5.76l 2
Stiffness Constraint
• Maximum vertical deflection at C
δ max = 2mm
• Thus
3
Objective Function
• Hence, Minimize
Bounds of Variables
• Set some lower and upper bounds for the
four cross sectional areas
4
Optimization Problem
Formulation
• Minimize
1.132 A1l + 2 A2l + 1.789 A3l + 1.2 A4l
subjectS to− P ≥ 0,
2 A1 sin θ
yc
P
S yt − ≥ 0,
2 A2 cot θ
P
S yt − ≥ 0,
2 A3 sin α
P
S yc − ( cot θ + cot α ) ≥ 0,
2 A4
π EA12 P
− ≥ 0,
1.281l 2sin θ
2
π EA42 P
− ( cot θ + cot α ) ≥ 0,
5.76l 2 2
Pl ⎛ 0.566 0.500 2.236 2.700 ⎞
δ max − ⎜ + + + ⎟ ≥ 0,
E ⎝ A1 A2 A3 A4 ⎠
10x10−6 ≤ A1, A2 , A3 , A4 ≤ 500 x 10−6
5
• Constraints
-Equality δ ( x) = const
-Inequality σ ( x) ≤ σ y
• Objective functions
-Maximize or Minimize
-principle of duality can be used
to convert maximize problem into minimize
problem and vice versa.
Minimize f (x)
Subject to
g j (x) ≥ 0 j= 1,2,3……J
hk ( x ) = 0 k=1,2,3…..K
xiL ≤ xi ≤ xiu i=1,2,3……N
6
4 DOF model of car
q2
q3 v
ms
kr αr k f αf
s s
q4 q1
mr u
mf u
kr t
kf t
7
• The forces F1 to F6 are calculated as follows
F1 = k f d1 ,
t
F2 = k f d 2 ,
s
F3 = α f d&2 ,
F4 = k r d 4 ,
s
F5 = αr d&4 , F6 = k r d 3
t
d 3 = q4 − f 2 (t ) , d 4 = q2 − l2 q3 − q4 .
• Constraint
say max jerk
max &q&&2 (t ) ≤ 18
• Objective function
Minimize max abs q2 (t )
A
Where A = road excitation amplitude
Limits
0 ≤ k f , k r ≤ 2kg / mm
s s
0 ≤ α f ,α r ≤ 300kg /(m / s )
Subjected to
18 − max(&q&&2 (t )) ≥ 0
8
CLASSICAL OPTIMIZATION
TECHNIQUES
• SINGLE-VARIABLE OPTIMIZATION
• MULTI-VARIABLE OPTIMIZATION
- WITH NO CONSTRAINTS
f (x )
• FUNCTION FOR THE DIFFERENT VALUES OF (x )CAN
HAVE
- RELATIVE OR LOCAL MINIMUM
- RELATIVE OR LOCAL MAXIMUM
- ABSOLUTE OR GLOBAL MINUMUM
- ABSOLUTE OR GLOBAL MAXIMUM
9
FUNCTION IS SAID TO HAVE
- RELATIVE OR LOCAL MINIMUM
• at x = x * if f ( x*) ≤ f ( x * + h)
- RELATIVE OR LOCAL MAXIMUM
• at x = x * if f ( x*) ≥ f ( x * + h)
For all sufficiently small positive and negative values of h
10
Single variable optimization
• Bracketing (exhaustive search)
• Region elimination (internal halving)
UNIMODAL FUNCTION
A unimodal function is one that has only one peak (maximum) or
valley (minimum) in a given interval.
Mathematically,
A function f (x) is unimodal if
(i) x1 < x2 < x* implies that f (x2) <f (x1), and
(ii) x2 >x1 >x* implies that f (x1) < f (x2), where x* is the minimum
point.
φ(ξ φ(ξ
) )
ξ α ξ ξ ξ β ξ
α ξ ξ ξ β
1 2 ∗ ∗ 1 2
11
• Exhaustive search algorithm (given f(x), a & b)
Step 3 Is x3 ≤ b ? If yes, go
to Step 2;
Else no minimum
exists in (a,b) or a boundary
point (a or b) is the minimum
point. a x1 x2 x3 ..... b
12
EXAMPLE
• Minimize f ( x) = x 2 + 54 in the interval (0,5).
x
Iteration continues
13
Point Estimation Methods
• Instead if using only relative values of two
points, use their magnitude also
• Typically fit a curve through a number of
known points, and find the minimum
14
Successive Quadratic Estimation
• A general quadratic function is
f ( x) = a0 + a1 ( x − x1 ) + a2 ( x − x1 )( x − x2 )
• If (f1, x1), (f2, x2) and (f3, x3) are known,
( f 2 − f1 ) 1 ⎛ ( f 3 − f1 ) ⎞
a0 = f1 a1 = ; a2 = ⎜ −a ⎟
( x2 − x1 ) ( x3 − x2 ) ⎝⎜ ( x3 − x1 ) 1 ⎠⎟
• Minima lies at
_
x1 + x2 a1
x= −
2 2 a2
Algorithm
• S1: Let x1 be the initial point, Δ be the step size,
x2 = x1 + Δ
• S2: Compute f(x1) and f(x2)
• S3: If f(x1) > f(x2) Let x3=x1+2 Δ else x3=x1- Δ,
Compute f(x3)
• S4: Determine _fmin = min(f1,f2,f3) and Xmin
• S5: Compute x _
15
Bracketing Method based on unimodal property of objective function
1) Assume an interval [a,b] with a minima in the range
2) Consider x1 and x2 within the interval.
3) Find out values of f at x=x1,x2 and Compare ƒ(x1) and ƒ(x2)
a) If ƒ(x1) < ƒ(x2) then eliminate [x2,b] and set new interval [a,x2]
b) If ƒ(x1) > ƒ(x2) then eliminate [a,x1] and set new interval [x1,b]
c) If ƒ(x1) = ƒ(x2) then eliminate [a,x1] & [x2,b] and set new interval [x1,x2]
f(x) f(x)
f2
f1 f(x) f1 f2
f1
x x
a x1 x2 b a x x b
f2 1 2
x
a x1 x2 b
FIBONACCI METHOD
APPLICATION
• To find minimum of a function of one variable even if function is
not continuous
LIMITATIONS:
• The initial interval of uncertainty i.e. range in which the optimum
lies, has to be known.
• The function being optimized has to be unimodal in the initial
interval of uncertainty.
• The exact optimum cannot be located in this method. Only an
interval known as the
Final Interval of uncertainty will be known. The final interval of
uncertainty can be made
16
Advantage of Fibonacci Series
• Fibonacci Series: F0=F1=1; Fn=Fn‐1+Fn‐1
• Hence: 1,1,2,3,5,8,13,21,34,…
• Lk‐Lk*=Lk+1*
• Hence one point
is always pre‐
calculated
Fibonacci Search Algorithm
• Step 1: L=b‐a; k = 2; Decide n;
• Step 2: L = ⎛⎜⎝ F F ⎞⎟⎠ L x = a + L x2 = b − L*k
*
k
n − k +1
n +1
1
*
k
• Step 3:
– Compute either f(x1) or f(x2) (whichever is not
computed earlier)
– Use region elimination rule
– Set new a and b
• If k=n, TERMINATE else k=k+1 and GOTO Step
2
17
THE PROCEDURE
Sequence of Fibonacci numbers is defined as:
F0=F1=1; & Fn = Fn‐1 + Fn‐2 i.e. 1,1,2,3,5,8,13,21,34,…
L2
Let initial interval L0 defined by [a, b]
x1 x2
n=total no. of experiments be known. a b
Each function evaluation is termed as an experiment
L0 = b‐a L2
Define
The first two experiment points are x1 & x2
New search interval obtained contains one previous expt. point at a distance L2* from one
end (old end) or from new end
L2
Again discard interval based on unimodal property
New interval of search obtained in L3 where
The ratio will permit us to determine ‘n’ the required number of experiments
to achieve the desired accuracy in locating the optimum point.
After conducting n‐1 experiments the remaining interval will contain one experiment point
precisely at its middle point. The nth experiment point has also to be placed there. Since no
new information can be obtained by placing point there, the nth experiment point is place
very close to the remaining valid experiment point. This enables to obtain final interval of
uncertainty to within 0.5*Ln 1 .
18
Enter N, A, B F0 = F1 = 1 FLOW CHART OF THE ALGORITHM
k=2
This is done
so that x1
Is k = N?
always lie
No left of x2
k= k+1 L2* = (Fn‐2/Fn )* (B‐A)
J=2
No Yes
Lj = B‐A Is J=N
x1 = A+L2* No Yes x1 = B‐L2*
Is L2*>(Lj/2)?
x2 = B‐L2* x2 = A‐L2*
Calculate Print A ,B
f1 = f(x1) LN = B‐A
f2= f(x2)
f1>f2 f2>f1
A =x1 Compare f1 with f2 B = x1
f1 = f2 Stop
Find new L2* = A= x1, B= x2 Find new L2* =
Fn‐j*(L1)/Fn‐(j‐2) L2* = Fn‐j*(B‐A)/Fn‐(j‐2) Fn‐j*(L1)/Fn‐(j‐2)
J=J+1
J=J+1
Golden Section Method
APPLICATION
• To find minimum of a function of one variable even if function is not continuous
The golden section method is same s the Fibonacci method except that In Fibonacci
method
1. the ratio is not constant in every iteration
2. total number of experiments to be conducted are specified in the beginning whereas in
the golden section method it is not required
The procedure is same as the Fibonacci method
L2
except that the location of the first two
x1 x2
experiments is given by L2* = L0/(phi)2 = 0.382*L0 a b
The desired accuracy can be specified to stop the L2
procedure
19
Historical background of Golden Section
The value phi has historical background.
It was believed by ancient Greeks that a building having sides d, b such that d
(d+b)/d = d/b = phi will be having most pleasing properties. b
Or 1+ (1/phi) = phi
This gives phi = (1+ 50.5)/2 = 1.618
It is also found in Euclid geometry that the division of line into two unequal parts so that
the ratio of whole to the larger part equals the ratio of larger to smaller part, being known
as the golden section or golden mean.
• In every iteration if you reduce range by a factor r
• Original range – AD; reduced to AC in 1st iteration (points B and C are already calculated)
• If the next range in iteration 2 has to be AB, AB = r*AC;
• HENCE 1‐r = r2 ‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐ SOLUTION IS GOLDEN NUMBER
1‐r
B C
D
A
r 1‐r
Golden Section Search Algorithm
• Step 1: L=b‐a; k = 1; Decide ε; map a, b to a’=0; b’=1
• Step 2: Lw=b’‐a’
– w1=a’+0.618Lw
– w2=b’‐0.618Lw
• Step 3:
– Compute either f(w1) or f(w2) (whichever is not computed
earlier)
– Use region elimination rule
– Set new a’ and b’
• If |Lw|≤ε TERMINATE else k=k+1 and GOTO Step 2
20
An Example ‐ Golden Section
Function f(x) is
0.65 –[0.75/(1+x2)] – 0.65*x*tan‐1(1/x) is minimized using golden section method with n=6
A=0 B=3
The third experiment point is placed at x3 = 0 + (x2‐x1) = 1.854 – 1.146 = 0.708.
f3 = ‐0.288943 f1 = ‐0.208
Since f3<f1 delete interval [x1,x2].
The new interval of uncertainty is [0,x1] = [0, 1.146]
An Example ‐ Golden Section
The fourth experiment point is placed at x4 = 0 + (x1‐x3) = 0.438.
f4 = ‐0.308951 f3 = ‐0.288943
Since f4<f3 delete interval [x3,x1].
The new interval of uncertainty is [0,x3] = [0,0.7080]
The fifth experiment point is placed at x5 = 0 + (x3‐x4) = 0.27.
f4 = ‐0.308951 f5 = ‐0.278
Since f4<f5 delete interval [0,x5].
The new interval of uncertainty is [x5,x3] = [0.27,0.7080]
The last experiment point is placed at x6 = x5 + (x3‐x4) = 0.54.
f4 = ‐0.308951 f6 = ‐0.308234
Since f4<f6 delete interval [x6,x3].
The new interval of uncertainty is [x5,x6] = [0.27,0.54]
Ratio of final interval to initial interval is 0.09.
21
Bounding Phase Method (for bracketing the optima)
Step 1
Choose an initial guess X0 and an increment D. Set K = 0
Step 2
If f(X0 ‐ |D|) > f(X0) > f(X0 + |D|) then D is +ve
f(X0 ‐ |D|) < f(X0) < f(X0 + |D|) then D is –ve
else goto step 1
Step3
Set XK+1 = XK + 2K*D
Step4
If f(XK+1 )<f(XK). Set K =K+1 and goto step 3
else, the minimum lies in the interval(XK‐1 ,XK+1 ) and terminate
If D is large, accuracy is poor.
Bounding Phase Method – An example
Minimize f(x) = x2 +54/x ;
Step 1 Choose an initial guess X0 = 0.6 and an increment D = 0.5. Set K = 0
Step 2 Calculate f(X0 ‐ |D|) = f(0.1)= 540.010 f(X0) = f(0.6)= 90.36
f(X0 + |D|) = f(1.1)= 50.301 We observe f(0.1)>f(0.6)>f(1.1) therefore D is +ve = 0.5
Step3 Set X1 = X0 + 20*D Î X1 = 1.1
Step4 If f(X1 ) = 50.301 < f(X0). Set K = 1 and goto step 3
Next guess X2 = X1 + 2*D = 2.1
•f(X2) = 30.124<f(X1) therefore set K=2 and goto step 3
•Next guess X3 = X2 + 22*D = 4.1
•f(X3) = 29.981<f(X2) therefore set K=3 and goto step 3
•Next guess X4 = X3 + 23*D f(X4) = 72.277>f(X3).
•Thus terminate with interval (2.1,8.1)
With D = 0.5 the bracketing is poor.
If D = 0.001, the obtained interval is (1.623,4.695).
22
Mathematical techniques and
gradient based approaches
23
Proof :
• It is given that
f ( x * + h) − f ( x*)
f ′( x ) = lim ....................1
h →0 h
Exists as a definite number, which we want to
prove to be zero. Since x* is a relative minimum,
we have
f ≤ f(x*+h)
for all values of h sufficiently close to zero. Hence
f ( x * + h) − f ( x*) h>0
≥0 if
h
f ( x * + h) − f ( x*)
≤0 if h<0
h
24
Theorem 2.2: Sufficient Condition
• Let f'’(x*) = f'’’(x*) = …………= f(n+1) (x*) ,
(x*)=0, but f(n)(x*)≠ 0
• Then f(x) is
(i) a minimum value of f(x) if f(n)(x*)>0 and n
is even ;
• (ii) a maximum value of f(x) if f(n)(x*)<0 and n
– is even ;
• (iii) neither a maximum nor a minimum if n is
odd.
h2 h n −1
f ( x * + h) = ( f ( x*) + hf ′( x*) + f ′′( x*) + ....... + f ( n −1) ( x*)
2! (n − 1)!
h n (n) 0 <θ <1
+ f ( x * +θh) for
n!
25
• As f(n)(x*)≠ 0, there exist an interval around x* for
every point of x of which the nth derivative f(n)(x*)
has the same sign, namely, that of f(n)(x*). Thus for
every point x*+h of this interval f(n)(x*+θh) has
the sign of f(n)(x*)
Thus x* will be
Relative minimum if f(n)(x*) is positive
Relative maximum if f(n)(x*) is negative
26
EXAMPLES
27
Second derivative of the function yields to ...
f ′′( x) = 240 x 3 − 630 x 2 + 240 x
The value of the function at
x = 1, f ′′( x) = −60 hence x = 1 is a relative maximum
f max | x =1 = 12
x = 2, f ′′( x) = 240 hence x = 2 is a relative minimum
f min | x = 2 = −11
28
In a two stage compressor, the working gas leaving the first
stage of compression is cooled ( by passing it trough a heat
exchanger ) before it enters the second stage of compression
to increase the efficiency. The total work input to a
compressor (W) for isentropic compression, is given by
⎡⎛ p ⎞ (k −1)/ k ⎛ p ⎞ (k −1)/ k ⎤
W = c pT1 ⎢⎜⎜ 2 ⎟⎟ + ⎜⎜ 3 ⎟⎟ − 2⎥
⎢⎣⎝ p1 ⎠ ⎝ p2 ⎠ ⎥⎦
Where cp is the specific heat of the gas at constant pressure, k
is the ratio of specific heat at constant pressure to that at
constant volume of the gas, and T1is the temperature at which
the gas enters the compressor. Find the pressure, p2, at which
inter-cooling should be done to minimize the work input to
the compressor. Also determine the minimum work done on
the compressor.
p2 = ( p1 p2 )
1/ 2
29
k −1
⎛ d 2W ⎞ 2c pT1
⎜ ⎟ = (3k −1)/ 2 k k(k +1)/ 2 k
⎜ dp 2 ⎟
⎝ 2 ⎠ p 2=( p1 p 2)1 / 2 p1 p3
d 2W p2 = ( p1 p3 )
1/ 2
2
>0 at
dp2
( k −1) / 2 k
k ⎡⎛ p3 ⎞ ⎤
Wmin = c pT1 ⎢⎜ ⎟ − 1⎥
k − 1 ⎢⎜⎝ p1 ⎟⎠ ⎥⎦
⎣
30
Gradient based Methods
• Algorithms require derivative information
• Many real world problems, difficult to obtain
information about derivatives
– Computations involved
– Nature of problem
• Still gradient methods are effective and popular
• Recommended to use in problems where derivative
information available
• Global optimum occurs where gradient is zero
• the search process terminates where gradient is zero
Overview of methods
• Methods
– Newton‐Raphson method
– Bisection method
– Secant method
– Cubic search method
31
Newton‐Raphson Method
• Goal of unconstrained optimization – reach as
small a derivative as possible
• A linear approximation of first derivative of
function at a point is expressed using Taylor’s
series expansion.
• Equate to zero to find next guess
• If the current point at iteration t is x(t), the
next iteration is governed by following
expression (t +1) (t ) f ' ( x (t ) )
x =x −
f ' ' ( x (t ) )
Newton‐Raphson Method
Algorithm
Step 1 :Choose initial guess x(1) and a small number
ε. Set k=1. compute f’(x(1))
Step 2 Compute f’’(x(k))
Step 3 Calculate
( k +1) f ' ( x(k ) )
x =x −
(k )
(k ) (1)
f ''(x )
Step 4 If | f’(x(k+1))|< ε, Terminate;
Else set k = k+1 and go to Step 2
32
Newton Raphson Method
• Convergence depends on the initial point and the
nature of objective function
• In Practice, gradients have to be computed
numerically
• At a point x(t), using central difference method:
f ( x (t ) + Δx (t ) ) − f ( x (t ) − Δx (t ) ) (2)
f ' ( x (t ) ) =
2 Δx (t )
f ( x (t ) + Δx (t ) ) − 2 f ( x (t ) ) + f ( x (t ) − Δx (t ) )
f ' ' ( x (t ) ) = (3)
( Δx (t ) ) 2
Newton Raphson Method
The parameter Δx(t) is usually taken to be
small
33
Newton‐Raphson Method
Consider minimization problem
f ( x) = x 2 + 54 / x
Step 1: We choose an initial guess x(1) = 1, a
termination factor of ε = 10‐3, and an iteration
counter k =1. computing derivative using (2). The
small increment using (4) is 0.01. The computed
derivative is ‐52.005, whereas the exact
derivative at x(1) is found to be ‐52. It is accepted
and proceeding to Step 2
Newton‐Raphson Method
Step 2 : The exact second derivative of function
x(1) = 1 is found to be 110. The second
derivative computed using (3) is 110.011,
which is close to exact value.
Step 3: We compute the next guess,
f ' ( x (1 ) )
x = x −
(2) (1 )
f ' ' ( x (1 ) )
= 1 − ( − 52 . 005 ) /(110 . 011 )
= 1 . 473
34
Newton‐Raphson Method
The derivative computed using (2) at this point is
found to be f’(x(2))=‐21.944
Step 4 : Since |f’(x(2))| not less than ε , we
increment k to 2 and go to Step 2. This completes
one iteration of the Newton‐Raphson method.
Step 2: (Second Iteration) f’’(x(2))=35.796
Step 3: (Second Iteration) The next guess x(3) =
2.086 and f’(x(3))=‐8.239
Step 4: (Second Iteration) Since |f’(x(3))| not less
than ε , we increment k to 3 and go to Step 2.
Newton‐Raphson Method
Step 2 : (Third Iteration) The second derivative at the
point f’’(x(3))=13.899
Step 3 : (Third Iteration) The new point is calculated as
f’(x(4)) =‐2.167. (Nine function evaluations)
Step 4 : (Third Iteration) Since the absolute value of this
derivative not smaller than ε, the search proceeds to
Step 2.
After three more iterations x(7)=3.0001 and f’(x(7))=‐4(10)‐
8, small enough to terminate algorithm
Since at every iteration first and second order derivatives
are evaluated, a total of three function values are
evaluated for every iteration
35
Bisection Method
• Computation of second derivative is avoided ;
only first derivative is used.
• Both function and the value and the sign of the
first derivative at two points is used to eliminate a
certain portion of the search space.
• Method similar to the region‐elimination
methods discussed.
• Algorithm assumes unimodality of the function.
Bisection Method
• Using derivative information, the minimum, is
said to be bracketed in the interval (a,b) if two
conditions – f’(a) < 0 and f’(b) >0 are satisfied.
• Requires two initial boundary points bracketing
the minimum.
• Derivatives at two boundary points and at the
middle point are calculated and compared.
• Of three points, two consecutive points with
derivatives having opposite signs are chosen for
next iteration
36
Bisection Method
Algorithm:
Step 1 : Choose two points a and b such that
f’(a)<0 and f’(b)>0. Also choose a small
number ε. Set x1 = a and x2 = b.
Step 2 : Calculate z=(x2+x1)/2 and evaluate f’(z)
Step 3 : If |f’(z)|≤ε, TERMINATE;
Else if f’(z) <0 set x1 = z and go to Step 2
Else if f’(z) >0 set x2 = z and go to Step 2
Bisection Method
• The sign of first‐derivative at the mid‐point of
the current search region is used to eliminate
half of the search region.
– If derivative –ve, minimum cannot lie in left half of
search region
– If derivative +ve, the minimum cannot lie on the
right half of the search space.
37
Bisection Method
Consider again the function:
f ( x) = x 2 + 54 / x
Step 1: Choose two points a = 2 and b = 5 such that f’(a) =
‐9.501 and f’(b) = 7.841 are of opposite sign. ε = 10‐3
Step 2 : Calculate a quantity z = (x1+x2)/2 = 3.5 and
compute f’(z)=2.591.
Step 3 : Since f’(z) > 0, the right half of the search space is
eliminated. x1 is set as 2 and x2=x=3.5 thus one
iteration completed.
At each iteration, one half of the search region is
eliminated but here the decision about which half to
eliminate depends on the derivatives at the mid point of
the interval
Bisection Method
Step 2 : (Second Iteration) z= (2+3.5)/2 =2.750 and
f’(z)=‐1.641.
Step 3 : (Second Iteration) since f’(z) < 0, x1 = 2.750
and x2 = 3.50.
Step 2 : (Third Iteration) z=3.125 and f’(z) =0.720
Step 3 : (Third Iteration) Since |f’(z)| not less than
ε, iteration continued.
At the end of 10 function evaluations intervals
(2.750,3.125), bracketing of minimum point x* =
3.0. The guess of minimum point is obtained at
interval of x = 2.938. This process continues until we
find a vanishing derivative.
38
Secant Method
• Both magnitude and sign of derivatives to create a
new point
• Derivative of function assumed to vary linearly
between two chosen boundary points.
• If two points x1 and x2, the quantity f’(x1) f’(x2) ≤ 0,
the linear approximation of the derivative x1 and x2
will have a zero derivative at a point z given by
f ' ( x2 )
z = x2 − (5)
( f ' ( x2 ) − f ' ( x1 )) /( x2 − x1 )
Secant Method
• In this method, in one iteration, more than half
the search space may be eliminated depending
on the gradient values at the chosen points.
Algorithm:
Step 1 : Algorithm same as bisection method except
that Step 2
Step 2 : Calculate new point z using (5) and
evaluate f’(z).
This algorithm also requires only one gradient
evaluation at every iteration. Thus, only two
function values are required per iteration.
39
Secant Method
Considering the problem
f ( x) = x 2 + 54 / x
Step 1: Initial points a=2 , b=5 having derivatives
f’(a) = ‐9.501 and f’(b) =7.841 with opposite
signs. Choosing ε = 10‐3 and set x1 = 2 and x2 =
5.
Step 2: Calculating new point using equation (5):
f ' (5)
z = 5− = 3.644
( f ' (5) − f ' (2)) /(5 − 2)
Secant Method
Step 3: Since f’(z) > 0, we eliminate the right part (
the region (z,b))of the original search region. The
amount of eliminated search space is (b‐z) =
1.356, which is less than half the search space (b‐
a)/2 =2.5 . Here, x1 = 2 and x2 =3.644 and proceed
with next iteration.
Step 2: (Second Iteration) z = 3.228 and f’(z) =1.127
Step 3: (Second Iteration) Right part of search space
eliminated since f’(z) > 0 . x1 = 2 and x2 = 3.228 for
next iteration
40
Secant Method
Step 2: (Third Iteration) z = 3.101 and f’(z) = 0.586
Step 3: (Third Iteration) since |f’(z)|not less than ε,
next iteration proceeded.
At end of 10 function evaluations, true minimum is
computed x=3.037. It is closer to bisection method.
Cubic Search Method
• Similar to successive quadratic point‐estimation
method except that the derivatives are used to
reduce the number of required initial points. For
ex.
f ( x) = a0 + a1 ( x − x1 ) + a2 ( x − x1 )( x − x2 ) + a3 ( x − x1 ) 2 ( x − x2 )
• a0,a1,a2,a3 be unknowns therefore requires at
least four points to determine the function.
• Function can also be determined by specifying
the function value as well as derivative at two
points
41
Cubic Search Method
• ((x1,f1,f’1), (x2,f2,f’2)) and by setting equation to
zero.
⎧ x2 , ifμ = 0
⎪ (6)
x = ⎨ x2 − μ ( x2 − x1 ), if 0 ≤ μ ≤ 1
⎪ x , ifμ > 1
⎩ 1
• Where 3( f1 − f 2 )
z= + f '1 + f ' 2
x 2 − x1
x 2 − x1
w= ( z 2 − f '1 f ' 2 )
x 2 − x1
f '2 + w − z
μ =
f ' 2 − f '1 + 2 w
Cubic Search Method
• Similar to Powell’s successive quadratic
estimation method, estimation of f(x) can be
used to estimate true minimum of objective
function.
• Estimate with earlier two points are used to find
the next estimate of the true minimum point.
• The product of their derivative of two points is
negative.
• This procedure is continued till desired accuracy
42
Cubic Search Method
Algorithm:
Step 1:
• Choose initial point x(0), a step size Δ and two
termination parameters ε1 and ε2.
• Compute f’(x(0)).
• If f’(x(0))>0, set Δ =‐Δ. Set k =0.
Step 2: Compute x(k+1)=x(k) +2kΔ
Step 3: Evaluate f’(x(k+1))
• If f’(x(k+1))f’(x(k))≤ 0 , set x1= x(k), x2= x(k+1), and go to
Step 4
• Else set k = k+1 and go to Step 2.
Cubic Search Method
Step 4: Calculate the point using (6)
Step 5: If , go to Step 6
_
f ( x) ≤ f ( x )
1
_ _ _
Else set until is achieved
x = x− (( x − x ) / 2)
1
f ( x) ≤ f ( x )
1
_ _ _
Step 6:
_ Compute f’( ). If |f’( )|≤ ε
x x 1 and |( ‐x
x 1)/
x |≤ ε2 , Terminate
_ _
Else if f’( )f’(x
x 1)<0 , set x2 = ; x
_
Else set x1 = x
Go to Step 4
43
Cubic Search Method
• Method is most effective if the exact derivative is
available
• Bracketing of minimum point achieved in first three
steps
• Bracketing algorithm similar to bounding phase
method
• Except first iteration, function value as well as first
derivative are calculated only at one new point Î Only
two new function evaluations required.
• First iteration requires repetitive execution of steps 2
and 3 to obtain bracketing points.
Cubic Search Method
_
• If new point is better than x
x 1, one of the two
points is eliminated depending on which
_
brackets the true minimum with . x
• If new point is worse than x
_ 1,the best two
among the points and will bracket.
x
• Excessive derivative computations are avoided
_
by simply modifying the point .
x
44
Cubic Search Method
Considering the function
f ( x) = x 2 + 54 / x
Step 1:
• Choosing an initial point x(0)=1, a step size Δ=
0.5 and termination parameters ε1 = ε2 =10‐3.
• The derivative of the function at x(0),f’(x(0)) =‐
52.005
• f’(x(0))<0, hence Δ= 0.5, k=0
Cubic Search Method
• x(1)=x(0)+2(0)Δ=1+1(0.5)=1.5
• f’(x(1)) = ‐21.002
• Hence, f’(x(0))f’(x(1)) not less than or equal to 0
Step 2: Hence, k =1 and
x(2) =1.5+21(0.5)=2.5
Step 3:
• Derivative f’(x(2))= ‐3.641 does not make product
negative.
• x(3) =1.5+22(0.5)=4.5 Î f’(x(3))= 6.333 , which makes
the product f’(x(2))f’(x(3)) negative
• x1 = x(2) = 2.5 and x2 = x(3) = 4.5
45
Cubic Search Method
Step 4: Estimated point is calculated
• z = ‐3.907 , w =6.190 and μ =0.735.
• x = 4.5‐0.735(4.5‐2.5) = 3.030
Step 5:
• f(x) =27.003
• since f(x) <f(x1) , go to Step 6.
Step 6:
• f’(x) =0.178 Î Termination criteria not satisfied
• f’(x)f’(x1) < 0, hence x2 = x = 3.030
• End of iteration one. and go to Step 4
Cubic Search Method
Step 4: (Second Iteration)
• x1 = 2.5 , f’(x1)=‐3.641, x2 =3.030
• f’(x2)=0.178
• x = 2.999
Step 5: (Second Iteration)
• f(x) =27.00 < f(x1)
Step 6: (Second Iteration)
• f’(x) = ‐0.007 Î x2 = 3.030 and x1 =2.999 for the next
iteration
Method is faster than Powell’s quadratic since derivatives are
employed. Powell method preferred if the derivatives are to
be evaluated numerically
46