Mario Zanon

Direct Methods for Optimal Control

Young Researchers Workshop on System Dynamics

and Optimal Control

22/01/2013

ESAT – Katholieke Universiteit Leuven

Outline 2 / 30

1 Direct Methods for Optimal Control

Problem formulation
Direct Single Shooting
Direct Collocation
Direct Multiple Shooting
Wrap up

2 Why Do We Want to Use Optimal Control?

3 Application Examples
Optimal Control with Collocation
MHE + MPC for Car Control

4 Conclusion, Discussion and Questions

Direct Methods for Optimal Control — Mario Zanon

Direct Methods for Optimal Control 3 / 30

1 Direct Methods for Optimal Control

Problem formulation
Direct Single Shooting
Direct Collocation
Direct Multiple Shooting
Wrap up

2 Why Do We Want to Use Optimal Control?

3 Application Examples
Optimal Control with Collocation
MHE + MPC for Car Control

4 Conclusion, Discussion and Questions

Direct Methods for Optimal Control — Mario Zanon

Direct Methods for Optimal Control - Problem formulation 4 / 30

Optimal Control Problem

Z tf
min m(x(tf )) + l(x(t), u(t))dt (1a)
x(t),u(t) t0
s.t. ẋ(t) = f (x(t), u(t)) (1b)
g (x(t), u(t)) = 0 (1c)
h(x(t), u(t)) ≥ 0 (1d)

In a more general framework:

also some parameters p are optimization variables
the model can be a DAE f (ẋ(t), x(t), z(t), u(t), p) = 0

Application to:
Control: trajectory planning and tracking
Estimation: system identification, observers
Direct Methods for Optimal Control — Mario Zanon
Direct Methods for Optimal Control - Problem formulation 5 / 30

Problem (1) is infinite dimensional!

First Discretize, Then Optimize.
Transcribe the OCP (1) into a Nonlinear Programming Problem
(NLP):
+ State-of-the-art NLP solvers available
+ Inequality constraints easy to handle
+ Commonly used due to their easy applicability
– Only approximate solution

Direct Methods for Optimal Control — Mario Zanon

Direct Methods for Optimal Control - Problem formulation 6 / 30

Three Discretization Strategies

Single Shooting: Sequential simulation and optimization
Collocation: Simultaneous simulation and optimization
Multiple Shooting: Hybrid between the 2 pervious

What is needed?
1 Parametrization of the controls u:
basis functions with local support, e.g.: piecewise constant
2 State discretization strategy: next slides

Direct Methods for Optimal Control — Mario Zanon

Direct Methods for Optimal Control - Direct Single Shooting 7 / 30

Basic idea
1 Define a time grid: t0 < t1 < . . . tN , e.g.: equidistant
2 Discretize the controls u(t), e.g.: piecewise constant on the
grid
3 Discretize the states x(t) by numerical integration

Direct Methods for Optimal Control — Mario Zanon

Direct Methods for Optimal Control - Direct Single Shooting 8 / 30

Single Shooting NLP Algorithm 1 Single shooting

N
X 1: Guess: x0 , u0 , . . . uN ;
min
x0 ,
m(xN+1 ) + ld (xk , uk ) 2: J = 0;
u0 ,...,uN k=0 3: h = []; g = [];
s.t. x0 = x(t0 ) 4: for k := 0 to N do
R
x ≤ xk ≤ x 5: xk+1 :=
R f (xk , uk );
u ≤ uk ≤ u 6: lk := l(xk , uk );
7: J += lk ;
xN+1 = x(tf )
8: h.append([xk − x; x − xk ]);
Written in a compact form: 9: h.append([uk − u; u − uk ]);
min J(x0 , U), 10: end for
x0 ,U 11: J += m(xN+1 )
s.t. g (x0 , U) = 0, 12: g = [x0 − x(t0 ); xN+1 − x(tf )];
h(x0 , U) ≥ 0, 13: Solve NLP(J,g,h)

Direct Methods for Optimal Control — Mario Zanon

Direct Methods for Optimal Control - Direct Single Shooting 9 / 30

Integrators
∂xk+1 ∂xk+1
Must provide both next state and sensitivities ∂xk , ∂uk
Different ways of computing the sensitivities:
1 External numerical differentiation
easy to implement, inaccurate, expensive

2 Variational differential equations

very accurate, reasonable cost, not 100% precise

3 Automatic Differentiation
100% precise, cheap

4 Internal numerial differentiation

very accurarte, very cheap

Direct Methods for Optimal Control — Mario Zanon

Direct Methods for Optimal Control - Direct Single Shooting 10 / 30

Pros and Cons

+ Can use state-of-the-art ODE/DAE solvers
+ Few degrees of freedom even for large state dimensions
+ Initial guess needed only for controls u
– Cannot use knowledge of the state x in initialization
– The solution can depend very nonlinearly on u
– Unstable systems difficult to treat

Direct Methods for Optimal Control — Mario Zanon

Direct Methods for Optimal Control - Direct Collocation 11 / 30

Basic idea
1 Define a time grid: t0 < t1 < . . . tN , e.g.: equidistant
2 Discretize every function f (t) as a polynomial defined for
every interval, e.g. for x(t):
D
X
xkD (t) = pj (τ )xkj
1
j=0

t =tk0 + hτ
0.5
x

D
X
0 ẋkD (t) = ṗj (τ )xkj
j=0
D
X 1 0
0 0.5 1 1.5 2 = p (τ )xkj
t h j
j=0

Direct Methods for Optimal Control — Mario Zanon

Direct Methods for Optimal Control - Direct Collocation 12 / 30

Collocation NLP
R tk+1
Integrals: tk L(x(t), u(t))dt ≈ lk (Xk , uk )

ODE: ẋ(t) − f (x(t), u(t)) ≈ ck (Xk , uk )

xk+1 −xk
e.g.: ẋ(t) − f (x(t), u(t)) ≈ tk+1 −tk − f (xk+1 , uk )

N
X
min m(xN(D+1)+N ) + lk (Xk , uk )
X0 ,...,XN+1
u0 ,...,uN k=0

s.t. x0 = x(t0 )
ck (Xk , uk ) = 0, ∀k ∈ [1, N]
x ≤ xk ≤ x
u ≤ uk ≤ u
xN(D+1)+N = x(tf )

Direct Methods for Optimal Control — Mario Zanon

Direct Methods for Optimal Control - Direct Collocation 13 / 30

Collocation NLP Algorithm 2 Collocation

N
X 1: Guess:
min m(xN(D+1)+N ) + lk x0 , . . . , xN(D+1)+N , u0 , . . . uN ;
X0 ,...,XN+1
u0 ,...,uN k=0
2: J = 0;
s.t. x0 = x(t0 ) 3: h = [];
ck (Xk , uk ) = 0 4: g = [x0 − x(t0 )];
5: for k := 0 to N do
x ≤ xk ≤ x
6: g .append(ck (Xk , uk ));
u ≤ uk ≤ u 7: J += lk (Xk , uk );
xN(D+1)+N = x(tf ) 8: h.append([Xk − x; x − Xk ]);
9: h.append([uk − u; u − uk ]);
Written in a compact form: 10: end for
min J(x0 , U), 11: J += m(xN(D+1)+N )
X ,U
12: g .append([xN(D+1)+N − x(tf )];
s.t. g (x0 , U) = 0,
13: Solve NLP(J,g,h)
h(x0 , U) ≥ 0,

Direct Methods for Optimal Control — Mario Zanon

Direct Methods for Optimal Control - Direct Collocation 14 / 30

Pros and Cons

+ Large scale but very sparse NLP
+ Can use knowledge of x in the initialization
+ Can deal well with unstable systems
– Adaptivity needs a new time grid, change in the NLP
dimensions

Example
Large use for chemical applications with long time horizons and
large state dimensions.

Direct Methods for Optimal Control — Mario Zanon

Direct Methods for Optimal Control - Direct Multiple Shooting 15 / 30

Basic idea
Something between collocation and single shooting
1 Define a time grid: t0 < t1 < . . . tN , e.g.: equidistant
2 Discretize the controls u(t), e.g.: piecewise constant on the
grid
3 The initial state of each interval xk is a variable in the NLP
4 System dynamics: close the gaps xk+1 − f (xk , uk )

Brake the problem in shorter single shooting ones + continuity

Collocation with integrators instead of polynomials

Direct Methods for Optimal Control — Mario Zanon

Direct Methods for Optimal Control - Direct Multiple Shooting 16 / 30

Multiple Shooting NLP Algorithm 3 Multiple shooting

N
X 1: Guess: x0 , . . . xN+1 , u0 , . . . uN ;
min
x0 ,...,xN+1 ,
m(x N+1 ) + l (x
d k k , u ) 2: J = 0;
u0 ,...,uN k=0 3: h = [];
s.t. x0 = x(t0 ) 4: for k := 0 to N do
R
xk+1 = fd (xk , uk ) 5:
R xk+1 − f (xk , uk );
rk+1 :=
x ≤ xk ≤ x 6: lk := l(xk , uk );
7: J += lk ;
u ≤ uk ≤ u
8: g .append([rk ]);
xN+1 = x(tf ) 9: h.append([xk − x; x − xk ]);
10: h.append([uk − u; u − uk ]);
Written in a compact form:
11: end for
min J(X , U),
X ,U 12: J += m(xN+1 )
s.t. g (X , U) = 0, 13: g .append([x0 −x(t0 ); xN+1 −x(tf )]);
14: Solve NLP(J,g,h)
h(X , U) ≥ 0,

Direct Methods for Optimal Control — Mario Zanon

Direct Methods for Optimal Control - Direct Multiple Shooting 17 / 30

Pros and Cons

+ Can use state-of-the-art adaptive ODE/DAE solvers
+ NLP still has fixed dimensions
+ Can treat unstable systems well
+ Can use knowledge of the state x in initialization
+ Easy to parallelize
– Not as sparse as collocation, not as few variables as single
shooting

Direct Methods for Optimal Control — Mario Zanon

Direct Methods for Optimal Control - Wrap up 18 / 30

What is the best choice?

Single shooting
+ less variables (smaller NLP size), easier to program
+ accuracy guaranteed by integrators
– not good at treating unstable systems
– not parallelizable
Collocation
+ very good for unstable systems
+ easy to compute the sensitivities
+/– large but sparse NLP
– no guarantee on the accuracy
Multiple shooting
+ very good for unstable systems
+ accuracy guaranteed by integrators
+ size can be reduced to the same as the single shooting NLP
– not as sparse as collocation, nor as “small” as single shooting

Direct Methods for Optimal Control — Mario Zanon

Direct Methods for Optimal Control - Wrap up 19 / 30

Are those methods plug and play?

Y Linear systems, quadratic objective (QP solution)
Y-N Mildly nonlinear, quadratic objective
N Highly nonlinear systems (NLP might be hard to solve)

NLP properties
NLP solution can depend strongly on initial guess
Depending on the problem: interior point vs SQP
Only local minimum can be found
Homotopies might be needed

System properties
Unstable dynamics: simultaneous approach
Stiff systems: implicit integration scheme needed

Direct Methods for Optimal Control — Mario Zanon

Why Do We Want to Use Optimal Control? 20 / 30

1 Direct Methods for Optimal Control

Problem formulation
Direct Single Shooting
Direct Collocation
Direct Multiple Shooting
Wrap up

2 Why Do We Want to Use Optimal Control?

3 Application Examples
Optimal Control with Collocation
MHE + MPC for Car Control

4 Conclusion, Discussion and Questions

Direct Methods for Optimal Control — Mario Zanon

Why Do We Want to Use Optimal Control? 21 / 30

Trajectory Planning
Plan an optimal
trajectory, e.g.:
Fly with
minimal fuel
Aerobatic
maneuvers
Drive while
avoiding
obstacles
Maximize
energy
production

Direct Methods for Optimal Control — Mario Zanon

Why Do We Want to Use Optimal Control? 22 / 30

Trajectory Tracking
Open Loop + Model Mismatch

1
0.5
x

−0.5

−1
0 0.5 1 1.5 2 2.5 3
t

0.5
u

−0.5

−1
0 0.5 1 1.5 2 2.5 3
t

Direct Methods for Optimal Control — Mario Zanon

Why Do We Want to Use Optimal Control? 23 / 30

Trajectory Tracking
MPC + Model Mismatch

0.04
x

0.02

0 0.5 1 1.5 2 2.5 3

t

−0.02
−0.04
u

−0.06
−0.08
−0.1
−0.12
0 0.5 1 1.5 2 2.5 3
t

Direct Methods for Optimal Control — Mario Zanon

Why Do We Want to Use Optimal Control? 24 / 30

How Does MPC Work?

Solve an OCP
Get the new state
Solve the OCP with the new starting point
Keep iterating

How to Estimate the State?

Many methods available
What about a least-squares fitting of the sensor outputs?
Z tf
min kx(t0 )kP + ky (x, u) − ỹ kQ (2a)
x,u t0
s.t. ẋ = f (x, u) (2b)
g (x, u) = 0 (2c)
h(x, u) ≥ 0 (2d)

Direct Methods for Optimal Control — Mario Zanon

Application Examples 25 / 30

1 Direct Methods for Optimal Control

Problem formulation
Direct Single Shooting
Direct Collocation
Direct Multiple Shooting
Wrap up

2 Why Do We Want to Use Optimal Control?

3 Application Examples
Optimal Control with Collocation
MHE + MPC for Car Control

4 Conclusion, Discussion and Questions

Direct Methods for Optimal Control — Mario Zanon

Application Examples - Optimal Control with Collocation 26 / 30

Airborne Wind Energy Systems

Dual airfoils Single airfoil

Node: n = 2, parent i = 1, airfoil 2 Node: n = 1, parent i = 0, airfoil 1

Tether: k = 2

Node: n = 1, parent i = 0
Tether: k = 3 Tether: k = 1

Tether: k = 1

Node: n = 3, parent i = 1, airfoil 3

Ground, Node: n = 0 Ground, Node: n = 0

Objective: maximize the extracted power

Variables: trajectory, tether length and diameter, total time

Direct Methods for Optimal Control — Mario Zanon

Application Examples - Optimal Control with Collocation 27 / 30

Parametric study
Compare the extracted power for different wing areas
Systems with up to 209 states
NLP: up to 40000 variables
Discretization:
10 control intervals, 1 to 4 collocation elements
order 4 Radau polynomials
piecewise constant controls
Optimization problem solved more than 100 times
4000
8 PW
3000
Tethers
y [m]

2000 Airfoils

1000

0
0 2000 4000 6000 8000 10000 12000 14000 16000
x [m], Pw [W/m2]

Direct Methods for Optimal Control — Mario Zanon

Application Examples - MHE + MPC for Car Control 28 / 30

Car Control
Obstacle Avoidance and Online Friction Coefficient Estimation
Y [m] 5

−5
0 20 40 60 80 100 120 140 160 180 200
X [m]
0.7

0.6

0.5
µ [-]

0.4

0.3

0.2
0 20 40 60 80 100 120 140 160 180 200
X [m]

Direct Methods for Optimal Control — Mario Zanon

Application Examples - MHE + MPC for Car Control 29 / 30

Car Control
Just to give an idea, this it the tire force:

F = µDFz sin(C arctan(Bσ − E (Bσ − arctan(Bσ))))

[B, C , D, E ] = f (pt , Fz )

1.4 1

0.9
1.2
0.8

1 0.7

0.6
0.8
F [N]

0.5

σ
0.6
0.4

0.4 0.3

0.2
0.2
0.1

0 0
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 0 0.2 0.4 0.6 0.8 1 1.2 1.4
σ F [N]

Direct Methods for Optimal Control — Mario Zanon

Conclusion, Discussion and Questions 30 / 30

Thank you for your attention.

Direct Methods for Optimal Control — Mario Zanon