13.11.

2023

Control&Automation Eng. Int. to

Optimal Control, KOM4560
Lecture #4
Instructor: Dr. Yavuz Eren

Lecture Books:
1. Optimal Control Theory: An Introduction, Donald E. Kirk.
2. Optimal Control Systems, Desienni S. Naidu.

Fall 2023

Introduction to Optimal Control, Lecture Notes #4 1

Variational Approach to Optimal Control Systems

We will deal the variational techniques and Hamiltonian function for optimal control systems.
This approach has been introduced by Pontryagin to develop the minumum principle.

Terminal Cost Problem: We consider the optimal control system where the performance index is
of general form containing a final (terminal) cost function in addition to the integral cost
function(the Bolza problem). Consider the plant, the performance index as follows

(4.1)

This problem will be solved by Pontragin principle, and we will reference the following auxiliary
result in the solution process.

(4.2)

If we use (4.2) into performance index of (4.1), we get

(4.3)

Introduction to Optimal Control, Lecture Notes #4 2

1
 13.11.2023

Note that, since the 𝑆(𝑥 𝑡! , 𝑡!) is a fixed quantity, the optimization index in (4.1) is equivalent
the index in (4.3) as 𝐽".

However, the optimal cost given by (4.1) is not similar with cost in (4.3).

We are only interested in finding the optimal control. So that, once the optimal control is
specified, the optimal cost is found using the index of 𝐽 which is given by (4.1).

Moreover, consider the following equation,

(4.4)

Now, we are ready to derive the optimal control strategy for terminal cost problem.

First of all we should impose the assumptions of the optimal condition. We assume that optimal
values of state and input values as 𝑥 ∗(𝑡) and 𝑢∗(𝑡), respectively. Then we have the optimal cost
and trajectories:

(4.5)

Introduction to Optimal Control, Lecture Notes #4 3

As a next step we should define variations of control and states as describing the perturbations:
(4.6)

So that, the state equations in (4.1) and the index (4.3) (𝐽") can be given as

(4.7)

Introduction to Optimal Control, Lecture Notes #4 4

2
 13.11.2023

Further, we need to introduce Lagrange multiplier(costate) vector 𝜆(𝑡) and using (4.4) we get
the following augmented performance index at the optimal condition:

(4.8)

Moreover, the index at perturbed condition :

(4.9)

Now, let us to define Lagrangian for each cases(optimal condition and any other condition)
given above, respectively:

(4.10)

Introduction to Optimal Control, Lecture Notes #4 5

After the definitions given above, the augmented performance index at the optimal and
perturbed condition becomes:

(4.11)

Applying the Taylor Theorem and retaining the linear terms only, we get

(4.12)

Introduction to Optimal Control, Lecture Notes #4 6

3
 13.11.2023

Now, it is time to give the first variation. Defining increment Δ𝐽, using Taylor series, extracting
the first variation 𝛿𝐽 by retaining only the first order terms, we get the first variation:

(4.13)

The equation above contains the term of 𝛿 𝑥(𝑡)̇ in the first variation so we will integrate it by
parts using the formula of ∫ 𝑢. 𝑑𝑣 = 𝑢. 𝑣 − ∫ 𝑣. 𝑑𝑢:

(4.14)

Let us to specify 𝛿𝑥 𝑡! = 0, since 𝑥 𝑡! is specified. So that, 𝛿𝐽 in (4.13) can be given as below

by considering the auxl.y result of (4.14):

(4.15)

Introduction to Optimal Control, Lecture Notes #4 7

For determining condition for extrema of the functional 𝐽, we need to apply the fundemental
theorem of calculus of variation for vanishing the first variation 𝛿𝐽.

Moreover, it should be noted that, in a typical control system(4.1), 𝛿𝑢(𝑡) is the independent
control variation and 𝛿𝑥(𝑡) is the dependent state variation.

For considering the above remark, we can choose 𝜆 𝑡 = 𝜆∗ 𝑡 which is at our disposal and
hence ℒ ∗ such that the coefficient of the dependent variation 𝛿𝑥(𝑡) in (4.15) be zero. Therefore,
the Euler-Lagrange equ.n is given as

(4.16)

where the partials are evaluated at the optimal (*) condition. Then, we know that independent
control variation 𝛿𝑢(𝑡) is arbitrary so that the coefficient of the 𝛿𝑢(𝑡) in (4.15) should be set to
zero.
(4.17)
Finally, the first variation(4.15) reduces to the following equ.n:

(4.18)

Also, we note that the system(or condition) can be written in terms of Lagrangian(3.10)
(4.19)

Introduction to Optimal Control, Lecture Notes #3 8

4
 13.11.2023

Let us deal the (4.18) and convert the 𝛿𝑥(𝑡) into an expression containing 𝛿𝑥$ . As seen from the
figüre below, we can state the slope of 𝑥̇ ∗ 𝑡 + 𝛿 𝑥(𝑡)
̇ at 𝑡$ is approximated as

(4.20)

Or,
(4.21)
and retaining only the linear terms(𝛿) in (4.21), we obtain
(4.22)

Introduction to Optimal Control, Lecture Notes #4 9

By using the last result in the boundary condition(4.18), we get the general boundary condition
in terms of Lagrangian:

(4.23)

Definition of Hamiltonian: We define Hamiltonian or also called the Pontryagin ℋ function at

the optimal condition as

(4.24)

Then, from (4.10) the Lagrangian ℒ ∗ in terms of Hamiltonian(ℋ ∗) becomes

(4.25)

Introduction to Optimal Control, Lecture Notes #4 10

10

5
 13.11.2023

Using (4.25) in (4.16), (4.17) and (4.19) and also using the terminal cost function (𝑆 =
𝑆(𝑥 𝑡 , 𝑡)), then we obtain the control, state and costate equations in terms of Hamiltonian.

Hence, for the optimal control 𝑢∗ 𝑡 becomes as follows by considering the relation of (3.17).

(3.26)

For the optimal state 𝑥 ∗ 𝑡 , the relation (4.16) becomes

which is equal to
(3.27)

And for costate 𝜆∗(𝑡)

(3.28)

Introduction to Optimal Control, Lecture Notes #4 11

11

Note that , looking the similar structure of the relation (4.27) for the optimal costate 𝜆∗(𝑡) and
(4.28) for the optimal state 𝑥 ∗ 𝑡 , it is clear that why 𝜆(𝑡) is called the costate vector.

Finall, the boundary condition(4.23) at the optimal condition can be derived by using the (4.25).

(4.29)

Different System Types:

Type a (Fixed Final Time and Fixed Final State)(Figure a): 𝑡$ and x(𝑡$ ) are fixed. Both of 𝛿𝑡$ and
𝛿𝑥$ are zero in general boundary condition of (4.29) and there is no other boundary condition.

Type b (Free Final Time and Fixed Final State)(Figure b): 𝑡$ is free and 𝛿𝑡$ is arbitrary. Since
x(𝑡$ ) is fixed, 𝛿𝑥$ is zero. Then the coefficient of the arbitrary 𝛿𝑡$ is zero in the general
boundary condition of (4.29).

(4.30)

Introduction to Optimal Control, Lecture Notes #4 12

12

6
 13.11.2023

Type c (Fixed Final Time and Free Final State)(Figure c): 𝑡$ is fixed and x(𝑡$ ) is free. So that, 𝛿𝑡$
is zero and 𝛿𝑥$ is arbitrary. So that, the coefficient of 𝛿𝑥$ in the general boundary condition of
(4.29) is zero.
(4.31)

Type d (Free Final Time and Dependent Free Final State)(Figure c): If 𝑡$ and x(𝑡$ ) are related
such that x(𝑡$ ) lies on a moving curve 𝜃(𝑡). Then,
(4.32)

Using (4.32), the boundary condition (4.29) for the optimal condition becomes

(4.33)

From the definition of the system type, 𝑡$ is free and 𝛿𝑡$ is arbitrary , we conclude that the the
coefficient of 𝛿𝑡$ in (4.33) is zero:

(4.34)

Introduction to Optimal Control, Lecture Notes #3 13

13

Type e (Free Final Time and Independent Free Final State)(Figure c): If 𝑡$ and x(𝑡$ ) are not
related then, x(𝑡$ ) and x 𝑡$ . Hence, the boundary condition (4.29) at the optimal cas becomes

(4.35)

Introduction to Optimal Control, Lecture Notes #4 14

14

7
 13.11.2023

Following figure illustrates the different types of systems considered:

Introduction to Optimal Control, Lecture Notes #4 15

15

Characterization of Extremum Points

For determinization of the character(min or max) of optimization, we need to take the second
variation and examine the sign, in other word, we have to derive sufficient condition for
extremum.

Using the augmented performance index(4.11), Lagrangian(4.25) and the boundary condition
for Type-d System(4.34), we obtain the second variation in (4.13) and using the relation
(4.25)(Lagrangian), we get

(4.36)

Introduction to Optimal Control, Lecture Notes #4 16

16

8
 13.11.2023

For the min.m, the second variation must be positive, so that it means the following matrix must
be positive definite.

(4.36)

Moreover, the important condition is that the second partial derivative of ℋ ∗ with respect to
𝑢(𝑡) must be positive.

For max.m the signs must be reserved.

Introduction to Optimal Control, Lecture Notes #4 17

17

Ex.: Given a second order system, performance index and boundary conditions, respectively

Find the optimal control, optimal state under the boundary conditions by assuming that the
control and state are unconstrained.

Introduction to Optimal Control, Lecture Notes #4 18

18

9
 13.11.2023

Summary of Calculation of Variation for Optimal Control Problems

Introduction to Optimal Control, Lecture Notes #4 19

19

10