OPTIMUM DESIGN

Dr. / Ahmed Nagib Elmekawy

Lecture 6

Fall 2017

1
 Taylor Series Expansion
Assume f(x) is:
1. Continuous function of a single variable x
2. Differentiable n times
3. x ∈ S, where S is non-empty, closed, and bounded
4. therefore x* is a possible optima

df ( x*) 1 d 2 f ( x*) 1 d 3 f ( x*)

f ( x )  f ( x*)  ( x  x*)  2
( x  x*) 2
 3
( x  x*) 3
 h.o.t.
dx 2 dx 3! dx

2
 Taylor Series Approximations
let x  x*  d , a small change or vicinit y of x * ...then
f ( x )  f ( x * d )
df ( x*) 1 d 2 f ( x*) 2 1 d 3 f ( x*) 3
f ( x )  f ( x*)  d 2
d  3
d  h.o.t.
dx 2 dx 3! dx
df ( x*)
f ( x )  f ( x*)  d First Order Approximation
dx
df ( x*) 1 d 2 f ( x*) 2
f ( x )  f ( x*)  d 2
d Second Order Approximation
dx 2 dx

3
 Taylor Series – Mult. Variables
Estimate f(x) using Taylor Series
1
f (x)  f (x*)  f (x*)( x  x*)  (x  x*)T H(x*) (x  x*)  R
T
2

Let d  (x  x*) , theref ore x  x *  d

1 T
f ( x )  f (x * d)  f (x*)  f (x*)d  d H d  R
T
2

4
 Change in function

Let’s define a change in the function with the symbol ∆

f  f (x )  f (x*)
1 T
f  f d  d H d  R
T

2
∆-Delta ∇-Del

Let’s define a first order change in the function

δf  f T δx  f • δx

5
Taylor Exp of MV

6
7
8
 Taylor Series Expansion
Assume f(x) is:
1. Continuous function of a single variable x
2. Differentiable n times
3. x ∈ S, where S is non-empty, closed, and bounded
4. therefore x* is a possible optima

df ( x*) 1 d 2 f ( x*)
f ( x )  f ( x*)  ( x  x*)  2
( x  x*)2
 ...
dx 2 dx
1 d 3 f ( x*) 1 d 4 f ( x*)
3
( x  x*)3
 4
( x  x*)4
 h.o.t.
3! dx 4! dx

9
 Taylor Series Approximations
let x  x*  d , a small change or vicinit y of x * ...then
f ( x )  f ( x * d )
df ( x*) 1 d 2 f ( x*) 2 1 d 3 f ( x*) 3
f ( x )  f ( x*)  d 2
d  3
d  h.o.t.
dx 2 dx 3! dx

df ( x*)
f ( x )  f ( x*)  d First Order Approximation
dx
df ( x*) 1 d 2 f ( x*) 2
f ( x )  f ( x*)  d 2
d Second Order Approximation
dx 2 dx

10
 Single variable minimization
Given that x* is the minimum of f(x), then any
movement away from x* is “uphill”, therefore to
guarantee that a move goes uphill
f ( x )  0
1
f ( x )  f ( x )  f ( x*)  f ( x*)d  f ( x*)d 2  ...
2
1 1
f ( x*)d 3  f iv ( x*)d 4  h.o.t.
3! 4!

First-order necessary condition

f ( x*)  0

11
Stationary point=max,min,neither
Any points satisfying f ( x*)  0

Are called “stationary” points.

Those points are a:

1. Min pt, or
2. Max pt, or
3. Neither (i.e. an inflection pt)

We need another test!

12
Second-order sufficient condition

Look at second order term

1
f ( x )  f ( x )  f ( x*)  f ( x*)d  f ( x*)d 2  ...
2
1 1
f ( x*)d 3  f iv ( x*)d 4  h.o.t.
3! 4!

Second-order sufficient condition for a minimum

f ( x*)  0

13
 Second-order condition?

What if f ( x*)  0

Then f(x*) is not a minimum of f(x*).

It is a maximum of f(x*).

14
 Single variable optimization
First-order necessary condition

f ( x*)  0

f ( x*)  0 f ( x*)  0

Second-order sufficient Second-order sufficient
condition for a minimum condition for a maximum
? ?
15
 Higher—order tests?
What if f ( x*)  0

Then the second order test fails. We need

higher order derivatives…

Min f even ( x*)  0

Max f even ( x*)  0

16
Second-order necessary conditions

Note the “= 0” possibility

f ( x*)  0 f ( x*)  0

A pt not satisfying this A pt not satisfying this

test is not a min! test is not a max!

17
 Multiple variable optimization
If x* is the minimum of f(x), then any movement
away from x* is “uphill”.

f  f (x )  f (x*)  0
1 T
f  f (x * d)  f (x*)  f (x*)d  d H d  R
T

2
1 T
f  f (x*)d  d H d (second order)
T

2
dot product triple product
scalar scalar
f  scalar
How can we guarantee that for a move in any d, we
make away from x*, we go “uphill”?
18
First-order necessary Condition
For x* to be a local minimum: f  f (x )  f (x*)  0
1 T
f  f (x*)d  d H d
T

2
 f 
 x  0
1rst order term 0
 1
 f   
  0
f (x*)  0
T
f (x*)  x2  0
  0
   
 f   
   

 n  x*
x 0

19
Second-order sufficient condition

For x* to be local minimum:

dT H d  0

That is H(x*) must be positive definite

Remember that dT H d

has “quadratic form”

20
 Quadratic form of a matrix
1 T
Given F (x )  x A x where A is symmetric, for example :
2
1 0 0  x1 
xT  x1 x2 x3  A  0 1 0 x   x2 
   x3 
0 0 1
1 0 0  x   x 
  1
x A x  x • (Ax ) Ax  0 1 0  x2    x2 
T T 1

 x  x 
0 0 1  3   3 
 x1 
x A x  x • ( Ax )  x1 x3   x2   x12  x22  x32
T T T
x2
 x3 
F (x )  x1  x22  x32 
1 2
2 21
For example, the following matrices are also associated with the same quadratic
form:

22
Dividing the coefficients equally between the off-diagonal terms, we obtain the symmetric
matrix associated with the quadratic form in E q. (a) as

23