Engineering Optimization

Concepts and Applications

Fred van Keulen

Matthijs Langelaar
CLA H21.1
A.vanKeulen@tudelft.nl

Engineering Optimization – Concepts and Applications

 Contents
● Sensitivity analysis

Engineering Optimization – Concepts and Applications

 Sensitivity analysis
● Sensitivity: derivative of response w.r.t. design variable:

 VM   VM u(s); s  

Total d VM  VM  VM u Sensitivity of system

 
derivative dsi si u si response (state variables)

u constant si constant
Partial derivatives

● Note: components of s should be independent!

Engineering Optimization – Concepts and Applications

 Sensitivity analysis (2)
● What for?

– Avoid curse of dimensionality by using higher-order

optimization algorithms (gradient-based, Newton, …)
– Examine sensitivity / robustness of optimized design
solutions (parameter sensitivity)
● When?

– Attractive when sensitivity information can be obtained

relatively cheaply

Engineering Optimization – Concepts and Applications

 Logarithmic sensitivity
1
df
● Definition: d L f d log f  f x df
  
d L x d log x  1 f dx
dx
x
d log x  1

dx x
● Advantages:

– Dimensionless, allows comparisons between parameters

– Clearly indicate the relative “strength” of the influence of

parameters:
>1: influential, important parameter
<<1: not very influential parameter
Engineering Optimization – Concepts and Applications
 Example logarithmic sensitivity
df a
f x  ax a 1

dx
a
dL f x a 1 x a 1 x
 ax  a ax  a a  a
dL x f x x
● Logarithmic sensitivity gives information on relative importance

● Always use logarithmic sensitivities when comparing sensitivity

values of different variables!

Engineering Optimization – Concepts and Applications

 Aspects of sensitivity analysis
● Implementation effort

● Efficiency

● Accuracy and consistency

Exact
Numerical
model
Response

Design variable

Engineering Optimization – Concepts and Applications

 Sensitivity analysis approaches
Implementation Efficiency
● Global finite differences Very easy Terrible*
Involves repetitive design
evaluations
● Discrete derivatives
Moderate As good
Based on differentiation
of numerical model as it gets
● Continuum derivatives
Lots of work As good
Based on differentiation
of governing equations as it gets

Engineering Optimization – Concepts and Applications

 Sensitivity analysis approaches (2)
Schematically:

x f
Model
x - df GFD
x+x f+f + dx
Model

x Governing df
Discrete
Discretization Differentiation
equations dx

x Governing df
Continuum
Differentiation Discretization
equations dx

Engineering Optimization – Concepts and Applications

 Automated differentiation
● Automatic generation of code that computes
sensitivities:

Analysis Automatic Derivative

code differentiation code

● Many different tools exist: ADIFOR, ADOL-F (Fortran),

ADIC, ADOL-C (C/C++), …
● Convenient, but generally code is several times slower
than hand-coded derivatives

Engineering Optimization – Concepts and Applications

 Finite difference derivatives
● Finite differences for sensitivity analysis (GFD):

– Simple

– Computationally inefficient (however …)

– Accuracy depends on design perturbation

● Based on Taylor series:

1 1 1
f ( x  h)  f  f ' h  f ' ' h 2  f ' ' ' h3  
1! 2! 3!
1 1
 f ( x  h)  f ( x )  f ' h  f ' ' h 2  O ( h 3 )
1! 2!
f ( x  h)  f ( x ) 1
 f '  f ' ' h  O( h 2 )
h 2!
Engineering Optimization – Concepts and Applications
 Finite difference derivatives (2)
● First order forward / backward FD:
f ( x)hf) 
( x f(hx) 1
f '  f ' ' h  O(h 2 )

h 2!
1 1 2 1
● Central FD: f ( x  h)  f  f ' h  f ' ' h  f ' ' ' h 3  O(h 4 )
1! 2! 3!
1 1 1
f ( x  h)  f  f ' h  f ' ' h  f ' ' ' h 3  O ( h 4 )
2

1! 2! 3! -
2
f ( x  h)  f ( x  h)  2 f 'h  f ' ' ' h 3  O(h 4 )
3!
f ( x  h)  f ( x  h) 1
 f '  f ' ' ' h 2  O(h 3 )
2h 3!

Engineering Optimization – Concepts and Applications

 Finite difference derivatives (3)
● Similarly:

f ( x  2hh) )22f f( x( )x  hf )( xf h( )x) 26 ( 4 ) 2 23

f '' 22
 f f ' 'h' h  O ( h ) (central)
(forward)
hh 4!3!

● Forward FD error analysis:

f ( x)  f m ( x)   m ( x) 

 f m ( x  h)  f m ( x )    m ( x  h )   m ( x )  1 2
f ' ( x)      f m ' ' h  O(h )
 h   h  2!

fm ' Condition error Truncation error

Engineering Optimization – Concepts and Applications
 FD accuracy
● Perturbation h determines error:
 f m ( x  h)  f m ( x )    m ( x  h)   m ( x )  1 2
f '( x)   
    f m '' h  O ( h )
 h   h  2!

Error

h
Engineering Optimization – Concepts and Applications
 Practical aspect: noise
● Numerical noise can spoil FD accuracy!

● Example of noise source: effect of remeshing

Normalized
stress
constraint

Hole radius

Engineering Optimization – Concepts and Applications

 Nonlinear elastic case
● Relatively cheap FD
sensitivities (exception):
– Solution technique: Load
incremental-iterative approach
Involves solution of many linear
systems, e.g. K T u  R
– FD: start the solution process for
the perturbed case from the Displacement

unperturbed solution
● Much less expensive than full analysis!
Engineering Optimization – Concepts and Applications
 Nonlinear path-independent case
● Consider: g (u(s); s)  0 (e.g. R  f ext  f int  0 )
● Solution obtained by Newton iterations:
1
 dg 
u     g, u k 1  u k  u
 du 
● For FD, solve perturbed case by iterating from nominal
solution:
g (u*; s)  ε1  0  g (u*; s  s)  ε1  δ  0

g (u(s  s); s  s)  ε 2

Engineering Optimization – Concepts and Applications

 Nonlinear path-independent case
● Pitfall: make sure to include the finite residuals in the
FD calculation!
– Consider first iteration for perturbed case:
1
 dg 
u1 ( s  s )    u k ( s ); s  s  gu k ( s ); s  s 
 du 
– For small design perturbation, this approaches:
1
 dg  Original
u1 ( s)    u k ( s); s  gu k ( s ); s 
 du  residual
– Interpretation: just an additional Newton iteration
Original residual dominates over effect of design perturbation
Engineering Optimization – Concepts and Applications
 Finite residual problem: solution
● To improve FD accuracy with finite residuals:
instead of solving
gu(s  s); s  s   0
solve
gu(s  s); s  s   gu(s); s   0

i.e. subtract original residual from new residual.

● Ok for s = 0. Original residual no longer dominates

Engineering Optimization – Concepts and Applications

 Finite difference summary

● Easy to implement, black box approach

● Inefficient, except for nonlinear path-independent and

explicitly solved transient case
● Choice of proper relative design perturbation critical

● No adjoint formulation possible: unattractive in cases

with many design variables and few responses

Engineering Optimization – Concepts and Applications

 Discrete derivatives
● Consider linear discretized equations
(e.g. linear elastic FE model):
K ( s )u( s )  f ( s )  u  K 1f

and response (e.g. equivalent stress):  Von Mises u( s ); s 

T
● Then: d  dui   du 
   
ds ui ds s u ds s
State variable vector
sensitivity

Engineering Optimization – Concepts and Applications

 State variable sensitivity
● State variable derivatives follow from differentiation of
original equation:
dK du df
K ( s )u( s )  f ( s)  uK 
ds ds ds
du df dK
 K   u
ds ds ds
du 1  df dK 
 K   u
● Decomposed K ds  ds ds 
already available (direct solver)! Pseudo-load vector

Engineering Optimization – Concepts and Applications

 State variable sensitivity (2)
du 1  df dK 
K   u
ds  ds ds 
● Nonlinear case similar:
1
du  g   g 
g (u( s); s )  0      
ds  u   s 

Already decomposed
tangent matrix
(direct solver)

Engineering Optimization – Concepts and Applications

 Semi-analytical approach
● Semi-analytical: use FD to compute pseudo-load:

du 1  df dK 
K   u
ds  ds ds 
 f ( s  s )  f ( s ) K ( s  s)  K ( s ) 
1
K   u
 s s 
● Advantages:

– Easy implementation (can be done at top level)

– Efficient computation

Engineering Optimization – Concepts and Applications

 SA: nonlinear case
● Geometrically nonlinear (history-independent) setting:

g (u( s ); s )  0
1
dg g g du du  g  g
   0    
ds s u ds ds  u  s
Already
g
● SA approach: computed using FD: decomposed
s
g gu( s ); s  s   gu( s ); s 

s s

Engineering Optimization – Concepts and Applications

 Discrete derivatives CH
EA
P!!
● Note, computation of discrete derivatives

– Only involves a linear equation, also in nonlinear case

– Allows re-use of the decomposed system matrix

 Sensitivity analysis much cheaper than analysis itself!

● Options for calculation of pseudo-load vector:

a) Analytical differentiation (lots of work)

b) Automated differentiation (code generator programs)

c) Finite difference approach

Engineering Optimization – Concepts and Applications

 SA accuracy problem
● Accuracy of semi-analytical (SA) sensitivities w.r.t.
shape variables reduces for cases with substantial
rotations (slender structures)
● Problem increases with mesh refinement!?!

SEE APPENDIX

Engineering Optimization – Concepts and Applications

 Eigenvalue sensitivities
● Important class of responses: eigenvalues
M  K v  0
● Discrete sensitivity analysis:


: M'   K 'v   ' Mv  M  K v'  0
s
T
v *: v T M '   K 'v   ' v T Mv  v T M  K v '  0
0
v T M '   K 'v
 '  T
v Mv

Engineering Optimization – Concepts and Applications

 Eigenvalue sensitivities (2)
v T M '   K 'v
● Result: ' 
v T Mv

● Note, no need to compute eigenvector sensitivities v’!

If needed, one can use Nelson’s method
(but rather expensive)
● Difficulties: eigenvalue multiplicity, mode switching ...

Engineering Optimization – Concepts and Applications

 Contents
● Sensitivity analysis:

– Brief recap discrete / SA approach

– Adjoint method

– Continuum sensitivities

● Topology optimization

● Closure

Engineering Optimization – Concepts and Applications

 Adjoint discrete sensitivities
● Discussed direct approach:
dhi hi hi du
h  h(u(s); s)   
ds j s j u ds j
and
du 
1  f K 
K (s)u(s)  f (s)  K  u
ds j  s s 
 j j 
● One backsubstitution needed for every design variable:
not attractive for many design variables
● Alternative: adjoint formulation

Engineering Optimization – Concepts and Applications

 Adjoint sensitivities
● Starting point: augmented response:
=0
h*  h(u(s); s)  λ T f (s)  K (s)u(s) 

dhi * hi hi du  f K du 

 λi  
T
    uK
ds j s j u ds j  s s ds 
 j j j 
hi  f K   hi  du
  λi
T
  
u 
T
 λi K 
s j  s s   u  ds j
 j j 
● To avoid computation of state vector derivatives,
du
choose i such that vanishes!
ds j
Engineering Optimization – Concepts and Applications
 Adjoint sensitivities (2)
● Result: dhi hi 
T  f K   hi T  du
 λ  u   λ K
ds j s j  s s   u  ds j
 j j 
T
hi T T  hi 
 λi K  0  λi  K  
u  u 
dhi hi  f K 
 λi  u
T
  
ds j s j  s s 
 j j 
● One backsubstitution per response: attractive in case of
many design variables and few responses
Engineering Optimization – Concepts and Applications
 Adjoint vs. direct
● Direct method attractive when #variables < #responses,
adjoint method attractive when #variables > #responses

● Note, adjoint method requires load vector composed of

response derivatives (specific implementation)
● Difference consists of order of computations:

dhi hi hi 1  f K 

  K  u
ds j s j u  s s 
 j j  du
λi
T ds j

Engineering Optimization – Concepts and Applications

 Sensitivities in transient case
● Transient analysis:  (ti ; s )   (ti 1 ; s )   (ti , ti 1 ; s )
d d d
  
ds ti ds ti 1 ds ti

● Sensitivities at time ti depend on sensitivities at

previous instants
– Direct method: forward time integration of sensitivities

– Adjoint method: backward time integration of sensitivities

(unattractive, storage problem)
● FD often preferred for explicitly solved transient problems

Engineering Optimization – Concepts and Applications

 Discrete derivative summary
x Governing df
Discretization Differentiation
equations dx
● Generally efficient and easy to implement, particularly
semi-analytical case (combination with FD)
● Reuse of decomposed stiffness matrix (with direct
solver – with iterative solver, reuse of preconditioner)
● Direct and adjoint versions

● SA: accuracy problems for structures under large

rotations (beams, shells)

Engineering Optimization – Concepts and Applications

 Contents
● Sensitivity analysis:

– Brief recap discrete / SA approach

– Adjoint method

– Continuum sensitivities

● Topology optimization

● Closure

Engineering Optimization – Concepts and Applications

 Continuum derivatives
x Governing df
Differentiation Discretization
equations dx

● Example: beam bending

q(x)
(Euler-Bernoulli beam)

Governing equation: I(x,s)

x
d 2 EI 
2
q  0
dx d 2w
  2 , M  EI ( x, s )
+ boundary conditions dx

Engineering Optimization – Concepts and Applications

 Continuum derivatives (2)
● Now differentiate w.r.t. s:

d 2  dI  d 2  d 
2 
E    2  EI 0 Sensitivity equation
dx  ds  dx  ds 

● Compare:

d2
2
EI   q  0 Governing equation
dx
● For nonlinear / complex problems, the continuum
sensitivity equations are often simpler
Engineering Optimization – Concepts and Applications
 Sensitivity analysis summary
● Sensitivities important in optimization:

– Efficient higher-order optimization algorithms

– Evaluation of robustness of results

● Choice of sensitivity analysis method depends on:

– Number of design variables vs. number of responses

(adjoint vs. direct)
– Type of model (cheap / expensive,
linear / nonlinear / transient)
– Implementation effort, access to source code

Engineering Optimization – Concepts and Applications

 Sensitivity analysis summary (2)

on
nti
n
tio

tte
de
nta

mo

fa
cy

cy
me

so
ien

ra

nt
ple

int
joi
cu
fic

Ad

Po
Ac
Im

Ef
Finite difference • Perturbation size critical
• Efficient for nonlinear elastic
& explicit transient case

Discrete derivatives
• Inaccurate for large rotations
Semi-analytical
• Remedies: exact / refined version

Continuum derivatives

Engineering Optimization – Concepts and Applications