Engineering de sign optimization

Lecture9: Sequential programming techniques

Husk!
If r besked om skemal gning af den sidste forelsning fraGitte og Grethe EMSDerne skal hitte p en evalueringsopgave. -Ellers gr jeg det for jer.

Jo hn Rasm ussen, Insti tu t e of M e c ha nical En gin eeri ng, A alb org Un i v ersity, 2001

Lecture 9
Seq uenti al pr ogr ammi ng t echniq ues P Basic ideas. 1st and 2nd or der methods P Seq uential Linear Progr amm i ng, SLP
< Fo r m ul atio n < Co nv e rg e n c e < M ov e li m its

Problem definition
- we sta rt the u sual way

P Convex pr ogrammi ng
< CO N LI N < M MA

I n p ut x

Model

Out pu t g i( x)

P Seq uential Quadratic Pr ogramming, SQP

< Q u a d r atic a p p r oxi mat i o ns < T h e H e ss i an < Q u a si - N e wton m et h o d s

We wish to find the design vari able vector x, which minimize s g0( x) whil e honoring the constr ai nts gi( x) # 0, i = 1.. n. As we have seen, we can onl y use numeri cal methods to sol ve the probl em, if we do not know the mathematic al struct ur e of the functi ons gi.

P T hegr and per specti ve

To read: PA rora: chapter [6 - 6.5] + 6.7

Sequential Programming
- ba sic idea
T he b asi c i dea of s eq ue nti al pr ogr a m mi ng i s to ma ke ap pr o xi mati ons of th e func ti ons gi( x) . Si nc e the a ppr o xi mati ons ar e e xpli ci t, the r es ulti ng pr obl e m c an be s ol ved ei ther a nal yti c all y or wi th a ver y effi ci ent nu meri c al method. T he a ppr o xi mated pr obl e m i s often c all ed a s ub pr obl e m. T he s ol uti on to t he s u bpr o bl e m i s us u all y o nl y an ap pr o xi mati on of th e s ol uti on to the r e al pr obl e m, s o th e pr oc edur e m ust be appl i ed it er ati v el y as a s eq uenc e of sub pr obl e ms . H enc e the t er m seq u ent ial pr ogr a m mi ng. An other c o m mon ter m i s su b pr o b lem m et h od s .

Sequential programming
Example: sequential linear p rog ramming

g i ( x) g i ( x( k) ) + g i ( x( k ) ) ( x x( k ) )
We c an make a Taylor expansion of the functions from the curren t poi nt, x(k). If we onl y i ncl ude up to line ar terms , then the resul ti ng su bprobl em is linear and can be sol ved by the Simpl ex method. This type of subproblem method is call ed sequential line ar pr ogramm i ng.

Sequential Linear Programming

- g raphical inte rpretation
W hen t he origi nal pr obl e m is c on ve x, the li n eariz ed opti mu m h as a ten denc y to fall outsi de the r eal o pti m u m. T he s ubpr obl e m i s onl y li mi te d b y it s c onstr ai nts , wher e the no nli ne ar obj ec ti v e func ti on ma y for m a n at ur al m i ni mu m. Li neari z ed pr obl e ms c an ther ef or e oft en b ec o me un bou nded or osc ill ati ng .

Move limits
- ma ke the problem bounded and limit o scillation
A n ap pr o xi mati on i s onl y vali d i n a c ert ai n r egi o n ar ou nd x(k) . W e c all i t th e tr us t r egi on. It i s a g ood i de a to c onstr ai n the s ol uti on of th e s ubpr obl e m to th e tr us t regi on. S uch c ons tr ai nts ar e c all ed mo ve li mi ts .

x (j k ) x j x j x (j k ) + x j

Jo hn Rasm ussen, Insti tu t e of M e c ha nical En gin eeri ng, A alb org Un i v ersity, 2001

Move limits
- adaptive adju stment
M o ve li mits c an be adj ust e d acc or di ng to th e pr ogr ess of the opti mi z ati on pr oc es. If th e pr oc es i s g oi ng well, t he y c a n b e e xtende d or ke pt c ons tant. If t he pr oc es s i s os cill ati ng , t hey c an be tig hten ed. W hen a desig n vari a bl e appr oac hes th e opti mu m fr o m one si de, it ful fil s th e c o nditi on: xj xj

SLP
- propertie s
P It i s bas ed on li ne ar a ppr oxi mati ons of th e obj ec ti ve func ti on a nd c o nstr ai nts. P T he s ubpr obl e ms c an be s ol ve d b y Si mpl ex. P C on ver g enc e c a nnot be m at he mati c all y pr ov ed, but i t wor ks fi ne i n mos t c as es . It w or ks best , i s the o pti m u m is f ull y c ons tr ai ne d, i .e., i s in a c or n er of d e desi g n s p ac e. P M o ve li mits ar e r eq uir e d. N o c on ver g enc e is obtai ned i f the mo ve li mi ts ar e not ad apti ve. P T he fi nal i ter ati ons ma y be m any. P It ha ndl es pr obl e ms wit h ma ny desi g n vari abl es and cons tr ai nts.

x (j k 2 ) x (j k 1) 0 x (j k 1) x (j k )

Iterations

Iterations

In thi sca se, we can relax the move limit on xj a little. Otherwi se, we tighten. Thi s way, the move limits on each variable adju st gradually to the nature of the problem.

Convex programming
- ba sic idea
Fir s t or der ap pr o xi mati ons li ke SLP ar e attr act i v e b ec aus e the y onl yr eq uir e gr adi e nts and n o hi g her d eri v ati ves . B ut SL P has the pr o bl e m th at i t r eq ui r es m ove li mi ts and i s pr one to os cill ati on. Is ther e a ny other fir s t or der ap pr o xi mati on we c o ul d us e whi c h does not hav e thi s pr obl e m? T hi s q ues ti on has cr eated the i dea of c on vex pr ogr a m mi ng .

Conservative approxi mations

- better safe than sorry
T he ap pr o xi mati on t o the feasi bl e do mai n i s s ai d to be c ons er vati ve i f it t ends to u nder- esti mate the si z e of the do mai n. T hi s is attr acti ve bec aus e i t ap pr oac hes the opti mu m t hr oug h a s eq u enc e of feasi bl e desi g ns. S LP appr o xi mati ons ar e not cons er vati ve.

The CONLIN algorithm

- by Claude Fleury
O ne of the fir st s ucc esf ul atte mpts at a c on ve x pr ogr a m mi ng al g ori th m f or g en er al nonli near pr obl e ms w as C ON LIN. It is bas ed o n a vari abl e s ubst it uti o n. So m e of t he v ari abl es of th e S LP appr oxi m ati on ar e r e pl ac ed by thei r r ec i pr oc al s.
g i (x ) g i (x ) +
( k)

The CONLIN algorithm

- continued
P M o vi ng ter ms b et wee n th e s u ms ( 1) and ( 2) will c r eat e a ppr o xi mati ons th at ar e mor e or l ess c ons er vati v e. P It i s possi bl e to s ho w th at the s c he me of p utti ng the v ari abl es with p ositi ve deri vati v es i n the s u m ( 1) and the neg ati ve ones i n t he other i s the c o mbi nati on th at i s the mos t c ons er v ati ve. P It i s als o p ossi bl e to s ol ve th e s ub pr obl e m an al ytic all y most of th e wa y by a s o- c all e d du al t ech niq ue. T hi s r educ es the n um ber of var i abl es th at ha ve to be det er mi ne d nu meri c all y ver y m uch. P C on ve x pr ogr a m mi ng c an al s o be us ed t o s ol ve i nteg er pr o bl e ms wi th man y vari abl es .

(1 )

gi (k ) (x x j ) + x j j

(2)

gi (k ) (y y j ) y j j

1 where y j = xj This can b e converted to g i ( x ) g i ( x ( k) ) +

(1 )

gi ( x x j( k ) ) + x j j

(2)

gi x j ( x j x j( k ) ) x j x j
(k )

gi where sum (1) is use d if 0, and sum ( 2) is used otherwise x j

Jo hn Rasm ussen, Insti tu t e of M e c ha nical En gin eeri ng, A alb org Un i v ersity, 2001

The Method of Moving Asymptotes, MMA

- by Kri ster Svanberg
T hi s is a c on ve x pr ogr a m mi ng metho d bas ed on a si mil ar i d ea. T he mathe mati c al for m ul ati on is slig htl y di ff er ent and slig htl y m or e c o mpl e x. It all ows f or c o nti nuous ad aptati on of the c ons er vati v enes s of the pr obl e m b y me ans of s o- c all ed as y m ptotes . T he y wor k al m ost li ke mo ve l i mits but ar e encl os e d i n the math e matic s . MM A has g ai n ed mor e pop ul ari ty tha n C ON LIN and i t wi d el y us e d for man y pur pos es.

Convex Programming
- propertie s
P T he y ar e us uall y fi rs t or d er m etho ds, s uc h as C ON LIN a nd MM A. P C on ve x pr ogr a m mi ng us es appr o xi mati o ns t hat ar e mor e c ons er vati v e tha n SLP. P T he s ol uti on tec hniq u e i s par tl y a nal yti c al an d th er efor e v er y fast . P C on ver g enc e is often t hr oug h feasi bl e s ol uti o ns. P T he y handl e pr obl ems with ma ny v ari abl es ver y well , but the y c a n h av e pr obl e ms wi th man y c onstr ai nts .

Seq. Quadratic Programming, SQP

- better app roximation s give fa ster convergence
If we us e q uadr atic a ppr o xi mati ons of the func ti ons , th en we g et a muc h mor e ac c ur ate s ubpr obl e m.

Quadratic Approximations
- have some attractive featu re s
P If H i s p osi ti ve defi ni te, the n the a ppr o xi mati on wil l c ur ve u pwar ds a nd th er efor e i ncr e ase at s o me dis ta nce fr o m the c urr e nt poi nt. T hi s mea ns t hat the pro bl e m i s auto matic all y b ounde d, an d we do not need mo ve li mi ts . P It i s possi bl e to d eri ve li near opti mali t y c o nditi ons for a q uadr ai c pr o bl e m. T hi s me ans that i t c an be s ol ved b y a n alg ori th m usi ng Si mpl ex as a s ubr outi ne.

1 gi ( x) g i ( x( k ) ) + g i ( x( k ) )( x x( k ) ) + ( x x( k ) )T H( x x(k ) ) 2
H i sthe He ssian matrix containing second derivative s:
2 gi x2 1 H= Symm gi x1 x2 2 gi x2 2
2

gi x1 xn . . 2 gi 2 xn
2

Quadratic Approxi mations

- also repre sent some p roblem s
P H c o nt ai ns s ec on d or der d eri v ati ves , and man y of the m, or d er n , alt houg h i t is s y m metri c al. R eal s ec ond or d er s ensi ti vit y anal ys is i s ver y ti me c o ns u mi ng . P S eco nd or d er al g ori th ms ar e mor e s ensi ti ve to no ns moothn ess of th e func ti ons t han l o wer or der alg ori th ms . P T he st r aig htfor war d s ol uti on of a QP pr o bl e m ofte n r eq ui r es i n ver si on or fac tori z ati on of H . T hi s c an be ti me c o ns u mi ng , if n i s l arg e.

Quasi-Newton Methods
- the solution to some of the QP problem s
P We s tar t ot with a linear appr ox i mation and a Hessian H = I. P For each step in the process, we s ave the computed gr adients of all functions. P The gradi ents of multiple desig n poi nts are used to create an overall approx i mation of H. P This approximation i mproves as more iter ations are per formed.

A cla ssof method s called quasi-Newton method s solve at least some of the se p roblem s.
Jo hn Rasm ussen, Insti tu t e of M e c ha nical En gin eeri ng, A alb org Un i v ersity, 2001

Quadr atic function approxi mated from gradients.

Quasi-Newton Methods
- continued
P Some methods can approxi mate the inverse Hessian directl y. This eli m i nates the need for inversion or factoriz ati on. P It onl y wor ks i f the functions of the real pr oblem globally c an be approximated well byq uadratic functi ons. P If the process does not converge in, say, 10 it er ations, then the overall behavior of the functi ons is probabl y not q uadratic , and it is better to reinitializ e H to I.

The Grand Perspective

Overview of method s fo r con strained optimization
C ons tra i ned Op ti mi zation Penal ty methods Exterior p e nalty Interio r p e nalty Aug men ted lag rangi an Di rect search methods Fea sible dir ections SLP Subp roble m methods C onv ex p rogrammin g SQP

Some functions ar e not approxi mated well by q uadr atic for ms .

Penalt y method s : Easy, but not ver y r obust. Feasible dir ections: R obust and ver satile, but also a little complicat ed. Can have tr oubl e wi th pr oblems w i th diff er ent scales. Re q uir es gradients onl y for directi on c hoice. Subprobl em methods: Good for larg e probl ems wi th many variabl es and sc al e dif ferenc es. R eq uire gradi ents for all it er ations.

Assign ment
SLP in ODESSY
OD ESSY s st andar d o pti miz er i s S LP wi th adapti ve mo ve li mi ts. D o wnl o ad the s a m pl e pr obl e m bu mp er. acd fr o m m y ho mepag e and st u dy the defi ni ti o n of the obj ecti ve f uncti o n an d c ons tr ai nts . R un i t as a s hape opti miz ati on i n OD ESS Y and r ec or d the c on verg enc e hi st or y. Ini ti all y, i t c o nv erg es well, but it r eq uir es a l ot of i ter ati o ns to obtai n the fi nal c on verg enc e. K nowi ng th at the s ys te m us es S LP, ex pl ai n w hy the c on verg enc e is sl o w.