Available online at www.sciencedirect.

com

ScienceDirect
IFAC PapersOnLine 55-27 (2022) 13–18
FOV-based
FOV-based Model
Model Predictive
Predictive Object
Object
FOV-based
FOV-based Model
Model Predictive
Predictive Object
Object
Tracking
FOV-based
Tracking
FOV-based for
Model
for
Model Quadcopters
Predictive
Quadcopters
Predictive Object
Object
Tracking
Tracking for
for Quadcopters
Quadcopters
Tracking
Tracking
∗
for
for Quadcopters
Quadcopters
∗ ∗
Fadi
Fadi Snobar
Snobar ∗ Johannes Reinhard∗ Hartwig Huber∗
∗ Johannes
∗∗ Reinhard ∗∗∗Hartwig Huber∗ ∗∗
Fadi
Marcel
Fadi Snobar
Hoffmann
Snobar ∗ Johannes
∗∗ Michael
∗ Johannes
Reinhard
Stelzig
Reinhard ∗∗∗Hartwig
Martin
∗∗∗Hartwig
Huber
Vossiek
Huber ∗ ∗∗
Marcel
Fadi
Marcel Hoffmann
Snobar
Hoffmann ∗ ∗∗ Michael
Johannes Michael Stelzig
Reinhard
Stelzig ∗ Martin
∗∗∗Hartwig
Martin Vossiek
Huber
Vossiek
∗ ∗∗
∗ ∗∗
Fadi Snobar ∗∗ Knut
Johannes Graichen
Reinhard Hartwig Huber
Marcel
Marcel Hoffmann
Hoffmann ∗∗ Knut
∗∗ Knut
Michael
Michael Stelzig
Graichen
Stelzig
∗
∗ ∗∗ Martin Vossiek∗∗
Martin Vossiek
Marcel Hoffmann Knut Michael Graichen
Stelzig∗∗ Martin Vossiek∗∗
Graichen
∗∗
∗ Knut
Knut∗∗Graichen Graichen
∗∗ ∗
∗ Chair of Automatic Control Chair of Microwaves & Photonics
∗ Chair of Automatic Control ∗∗ Chair of Microwaves & Photonics
∗ Chair of Automatic Control
Friedrich-Alexander-Universität ∗∗ Chair ofErlangen-Nürnberg
Microwaves & Photonics
∗ Chair of Automatic Control
Friedrich-Alexander-Universität ∗∗ Chair of Microwaves
Erlangen-Nürnberg & Photonics
∗ Chair {fadi.snobar,
e-mail: of Automatic Control
Friedrich-Alexander-Universität
johannes.reinhard, ∗∗ Chair of Microwaves
Erlangen-Nürnberg
hartwig.huber, & Photonics
knut.graichen,
Chair
e-mail: of Automatic Control
Friedrich-Alexander-Universität
{fadi.snobar, johannes.reinhard, Chair of Microwaves
Erlangen-Nürnberg
hartwig.huber, & Photonics
knut.graichen,
e-mail: Friedrich-Alexander-Universität
{fadi.snobar,
marcel.mh.hoffmann, johannes.reinhard,
michael.stelzig,
Friedrich-Alexander-Universität Erlangen-Nürnberg
hartwig.huber,
martin.vossiek}@fau.de
Erlangen-Nürnberg knut.graichen,
e-mail: {fadi.snobar,
marcel.mh.hoffmann,
e-mail: {fadi.snobar, johannes.reinhard,
michael.stelzig, hartwig.huber,
martin.vossiek}@fau.de knut.graichen,
e-mail: {fadi.snobar, johannes.reinhard,
marcel.mh.hoffmann,
marcel.mh.hoffmann, michael.stelzig,
johannes.reinhard,
michael.stelzig,
hartwig.huber,
martin.vossiek}@fau.de
hartwig.huber,
martin.vossiek}@fau.de
knut.graichen,
knut.graichen,
marcel.mh.hoffmann, michael.stelzig,
marcel.mh.hoffmann, michael.stelzig, martin.vossiek}@fau.de martin.vossiek}@fau.de
Abstract:
Abstract: Modern Modern developments
Modern developments
developments in in electronics,
in electronics,
electronics, electricelectric motors,
electric motors,
motors, and and automatic
and automatic
automatic control control
control have have
Abstract:
contributed
Abstract: to making
Modern quadcopters
developments in cheap,
electronics, easily accessible,
electric and
motors, with
and a wide
automatic range of
control use.have
In
have
contributed
Abstract:
contributed to
to making
Modern
making quadcopters
developments
quadcopters in cheap,
electronics,
cheap, easily
easily accessible,
electric
accessible, and
motors,
and with
and
with a wide
automatic
a wide range
range of
control
of use.
use. In
have
In
this
Abstract:
this work,
contributed
work, a
a model
Modern
to making
model predictive
developments
quadcopters
predictive controller
in
controller cheap, for
electronics,
for a quadcopter
easily
a electric
accessible,
quadcopter is
is developed
motors,
and and
with
developed a for
automatic
wide
for tracking
range
tracking a
control
of
a moving
use.have
moving In
contributed
this
object work,
contributed within a to
tothemaking
model quadcopters
predictive
field-of-view
making controller
(FOV)
quadcopters cheap,
of
cheap, a for easily
a
camera
easily accessible,
quadcopter
mounted
accessible, is
on and
the
and with
developed a
quadcopter.
with a wide
for
wide range
tracking
A of
a
mathematical
range of use.
moving
use. In
In
this
object
this work,
within
work, a
a model
the
model predictive
field-of-view
predictive controller
(FOV)
controller of a for a
camera
for a quadcopter
mounted
quadcopter is
on
is developed
the
developed quadcopter. for
for tracking
A
tracking a
mathematical
a moving
moving
object
this
object within
representation
work,
within a theof
model
the field-of-view
the quadcopter
predictive
field-of-view (FOV)is
controller
(FOV) of
presented
of a camera
for a
aa camerawith mounted
the
quadcopter
mounted help on
of
is
on the
developed
the quadcopter.
quaternions,
quadcopter. for whereasA
tracking
A mathematical
thea
mathematicaltracked
moving
representation
object within theof the
the quadcopter
field-of-view (FOV)is presented
presented
of with
camera the
mounted help of
on quaternions,
the whereas the tracked
representation
object is
representation
object is modeled
within
modeledtheof
of as
the
as
quadcopter
a car.
field-of-view
quadcopter
a car. The
The
is
FOV
(FOV)is
FOV ofis
presented
is adefinedwith
camera
definedwith the
with
mounted
the
with
help
the
help
the
of
help
on
of
help of
the
of aaquadcopter.
quaternions,pinhole
quadcopter.
quaternions,pinhole
whereas
camera
whereas
camera
A
A mathematical
the
model
mathematical
the
model
tracked
trackedand
and
representation
object
represents is
representation modeled
a of the
nonlinear
of as
the quadcopter
a car. The
constraint
quadcopter is
FOV
is presented
within is
presented defined
the with
MPC
with the
with help
the
design.
the help of
help
Not
of quaternions,
of
onlya
quaternions,pinhole
is a whereas
camera
desired
whereas pointthe
model
the tracked
in and
space
tracked
object
represents
object is
is modeled
a nonlinear
modeled as
as a
a car.
car. The
constraint
The FOV
FOVwithin is
is defined
the MPC
defined with
with the
design.
the help
Not
help of
only
of aa pinhole
is a
pinhole camera
desired
camera point model
modelin and
space
and
represents
kept
object within a
is modelednonlinear
the as a constraint
camera’s FOV
car. Thebut FOVwithin
also as the MPC
close as design.
possible Not
to only
its is
of principal aa desired
axis. point
The in
method space is
represents
kept within
within
represents
kept
a
a nonlinear
the camera’s
nonlinear
the camera’s
constraint
FOV
constraint
FOV but
but alsoisas
within
within
also
defined
asthe
the MPC
close
MPC
close
with
as
as
the help
design.
possible
design.
possible
Not
to its
Not
to
only
its a pinhole
is
principal
only is
principal
camera
aa desired
axis.
desired
axis.
point
The
point
The
modelin
method
in
method
and
space
space is
is
tested
represents
kept
tested in
within
in different
a nonlinear
the
different scenarios
camera’s
scenarios FOV of
constraint
of varying
but within
varying also complexity.
as the MPC
close
complexity. as The
design.
possible
The results
Not
resultsto are
only
its
are evaluated
is
principal
evaluated desired
axis.in
in terms
point
The
terms of
in
method
of flight
space
flightis
kept
tested
quality,
kept within
in
within the
different
constraint camera’s
scenarios
the camera’s FOV
compliance,
FOV of but
varying
and
but also as
computation close
complexity. as possible
The
as close as effort results
and to its principal
itsare
demonstrate evaluated axis.
the in The
terms
potential method
of flight
of is
the
tested
quality,
tested
quality,
in
in different
constraint
different
constraint
scenarios
compliance,
scenarios
compliance,
of
of andalso
varying
varying
and
complexity.
computation
complexity.
computation
possible
The
effort
The
effort andtodemonstrate
results
results
and
are
are
principal
demonstrate evaluated
evaluated
axis.
the
the
in
in
The
terms
potential
terms
potential
method
of
of
is
offlight
the
flight
of the
concept
tested
quality,
concept in to stabilize
different
constraint
to stabilize the
scenariosquadcopter
compliance,
the of
quadcoptervarying
and and track
complexity.
computation
and track the
the desired
The
effort
desired and object
results arewithin
demonstrate
object evaluated
within the
the camera’s
the in terms
potential
camera’s FOV.
FOV.of flight
of the
quality,
concept
quality, constraint
to stabilizecompliance,
the quadcopter and and computation
track the effort
desired and demonstrate
object within the the potential
camera’s FOV. of
of the
Copyrightconstraint
concept
concept to
to 2022 Thecompliance,
©stabilize
stabilize Authors.
the
the Thisand
quadcopter
quadcopter is ancomputation
and
and open track
trackaccess
the
the
effortunder
article
desired
desired
and demonstrate
the
object
object CCwithin
withinBY-NC-ND the
the
the potential
license
camera’s
camera’s FOV.
FOV.
the
concept to stabilize the quadcopter and track the desired object within the camera’s FOV.
INTRODUCTION/licenses/by-nc-nd/4.0/) tions,
(https://creativecommons.org
1. INTRODUCTION
1. INTRODUCTION
1. tions, e.g.
e.g. in
in Mellinger
Mellinger and
and Kumar
Kumar (2011);
(2011); Cowling
Cowling et
et al.
al.
1. INTRODUCTION tions, e.g.
(2007);
tions, e.g.
Raffo in Mellinger
in Mellinger
et al. (2010); andYang
and Kumar
Kumar et (2011);
al. (2021).
(2011); Cowling
Even
Cowling et al.
though
et al.
1. INTRODUCTION (2007);
tions,
(2007); Raffo
e.g.
Raffo in et
et al.
Mellinger
al. (2010);
(2010); and Yang
Kumar
Yang et
et al. (2021).
(2011);
al. (2021). Even
Cowling
Even though
et
though al.
1. INTRODUCTION Euler
tions,
(2007);
Euler angles
e.g.
Raffo
angles in are
et
are quite
Mellinger
al.
quite(2010); intuitive,
and Kumar
Yang
intuitive, their
et
their use
(2011);
al. use lacks
(2021).
lacks mathemat-
Cowling
Even et
though
mathemat- al.
In recent years, modern developments in the fields of (2007); Raffo et al. (2010); Yang et
In In recent
recent years,
years, modern
modern developments
developments in in the
the fields
fields of Eulersimplicity.
of ical
(2007);
Euler
ical
angles
Raffoare
angles
simplicity.
are
et On quite
al.
Onquite the
(2010);
the
intuitive,
other
Yang
intuitive,
other et al.
their
hand,
their
hand, al.the (2021).
use lacksEven
main
(2021).
use
the lacks
main Even though
mathemat-
advantages
though
mathemat-
advantages
electronics,
In recent
electronics, electric
years,
electric motors
modern
motors and
and automatic
developments
automatic incontrol
the
control opened
fields
opened of Euler
ical
of angles
simplicity.
using are
quaternions Onquite the
are intuitive,
other
that their
hand,
the use
the
gimbal-lock lacks
main mathemat-
advantages
effect inher-
In recent
electronics, years,
electric modern
motors developments
and automatic in the
control fields
opened of Euler
ical
of using angles
simplicity. are
quaternions Onquite the
are intuitive,
other
that their
hand,
the use
the
gimbal-lock lacks
main mathemat-
advantages
effect inher-
the
In
the gateway
recent
electronics,
gateway to
years,
to quadcopters
electric modern
motors
quadcopters as
and
as we
we know
developments
automatic
know them
them today.
incontrol
the
today. fields These
opened
These of entical simplicity.
of using
using
to the quaternions
Euler On the
anglesareother that
does hand,
the
not the
gimbal-lock
occur; main
and, advantages
effect
due inher-
to the
electronics,
the gateway electric
to motors
quadcopters and
as we automatic
know them control
today. opened
These ical
of
ent simplicity.
to the quaternions
Euler On the
anglesare other
that
does hand,
the
not the
gimbal-lock
occur; main
and, advantages
effect
due inher-
to the
developments
electronics,
the gateway
developments made
electric
to made quadcopters
motors
quadcopters
quadcoptersand
as we not
automatic
know
not only
them
only affordable
control
today.
affordable opened
Thesebut
but of
ent
lackusing
toof the quaternions
direct Euler angles
trigonometricare that
does the
not gimbal-lock
occur;
functions, and,
their effect
due
usage inher-
to the
offers
the gateway
developments to quadcopters
made quadcoptersas we know
not them
only today.
affordable Thesebut of
ent
lackusing
toof the quaternions
direct Euler anglesare
trigonometric that
does the
not gimbal-lock
occur;
functions, and,
their effect
due
usage inher-
to the
offers
also
the agile
gateway
developments
also agile and
and to very
made
very maneuverable
quadcopters
quadcopters
maneuverable as we unmanned
know
not them
only
unmanned areal
today.
affordable
areal vehicles
These
vehicles but ent
lack
some toof the
direct Euler
mathematical angles
trigonometric does
simplicity, not occur;
functions,
thus and,
their
consequently due
usage to the
offers
higher
developments
also agile and made
very quadcopters not only affordable but ent
lack
some to the Euler
ofmathematical
direct angles
trigonometric does
simplicity, not occur;
functions,
thus and,
their
consequently due
usage to the
offers
higher
(UAVs).
developments
also agile
(UAVs). and mademaneuverable
very quadcopters unmanned
maneuverable unmanned areal
not only affordable
areal vehicles
vehicles lack
someof
but numerical
lack direct
mathematical trigonometric
accuracy.
ofmathematical
direct simplicity,
Quadcopter
trigonometric functions,
thus
models
functions, their
consequently
using
their usage offers
higher
quaternions
usage offers
also
(UAVs).
also agile
agile and
and very
very maneuverable
maneuverable unmanned
unmanned areal
areal vehicles
vehicles some
numerical
some accuracy.
mathematical simplicity,
Quadcopter
simplicity, thus
models
thus consequently
using
consequently higher
quaternions
higher
(UAVs).
Consequently,
(UAVs). UAVs have become popular across a wide numerical
to
somedescribe
numerical accuracy.
rotations
mathematical
accuracy. Quadcopter
can be
simplicity,
Quadcopter found models
thus in
models usingetquaternions
Cariño
consequently
using quaternions
al. (2015);
higher
Consequently, UAVs
(UAVs). UAVs have have become
become popular
popular across
across aa wide to
wide Magnussen describe
numerical rotations
describeaccuracy.
to describe rotations can
Quadcopter
can be be found
be found
found in
models Cariño
using
in Cariño
Cariño et al. (2015);
etquaternions
al. (2015);
(2015);
Consequently,
area of applications
Consequently, UAVs with
have low-cost
become aerial
popular photography
across aa wideand numerical
to
Magnussen et al.
accuracy.
rotations
et al. (2013).
Quadcopter
can
(2013). models
in usinget quaternions
al.
area of applications
Consequently,
area of applications UAVs with
have
with low-cost
become
low-cost aerial
popular
aerial photography
across
photography and
wide
and to describe
Magnussen rotations
et al. can
(2013). be found in Cariño et al. (2015);
filmography
Consequently,
area of being
applications UAVs thehave
with most common
become
low-cost popular
aerial civilian
across
photography use. a Fur-
wide
and to describe
Magnussen rotations
et al. can
(2013). be found in Cariño et al. (2015);
filmography being the the most common civilian use. Fur- Fur- As far
Magnussen as the
et controller design for aa quadcopter is con-
area
area of
of applications
filmography
thermore,
filmography the being
applications ease
being ofwith low-cost
most
interfacing
with
the low-cost
most common
common
aerial
between photography
aerialcivilian
sensors
photography
civilian use.
use. aboard and As
and cerned,
Fur- As far
Magnussen
far as
as et al.
the
the al. (2013).
controller
(2013). design
controller design for
for a quadcopter
quadcopter is
is con-
con-
thermore,
filmography the ease
being of
the interfacing
most commonbetween sensors
civilian use. aboard Fur- As far
cerned, different
as the
different trajectory
controller
trajectory tracking
design
tracking for a approaches
quadcopter
approaches are
is
are pro-
con-
pro-
thermore,
the drone,
filmography
thermore, the
such
the ease
being
easeas of
the
of interfacing
cameras
most and
interfacing commonbetween
the
between UAV’ssensors
civilian
sensors main
use. aboard
aboard con-
Fur- As far
cerned, as the
different controller
trajectory design
tracking for aaapproaches
quadcopter is
are con-
pro-
the drone,
thermore, such
the easeas of cameras
interfacing and the
between UAV’ssensors main aboard con- posed
As far
cerned,
posed in
inas the the
different
the literature.
controller
trajectory
literature. For
For instance,
design
tracking
instance, for Cowling
quadcopter
approaches
Cowling et
et al.
al. (2007);
is
are con-
pro-
(2007);
the
troller,
the drone,
thermore, opens
drone, such
the
such up
easeas
new
as of cameras
possibilities
interfacing
cameras and
and the
for
between
the UAV’s
autonomous
UAV’ssensors main
main aboard con-
image-
con- cerned,
posed in different
the trajectory
literature. For tracking
instance, approaches
Cowling et al.are pro-
(2007);
troller,
the opens
drone, such up new
new
as possibilities
cameras and for
the autonomous
UAV’s main image-
con- Mellinger
cerned,
posed
Mellinger in theand
different
and Kumartrajectory
literature.
Kumar (2011)
For
(2011) use
tracking
instance,
use differential
approaches
Cowling
differential et flatness
al.are
(2007);
flatness to
pro-
to
troller,
based
the opens
localization
drone,
troller, opens such up
up asand
new possibilities
trajectory
cameras
possibilitiesand for theautonomous
planning.
for UAV’sThis
autonomous main image-
can be
con-
image- posed
Mellinger in theand literature.
Kumar For
(2011)instance,
use Cowling
differential et al. (2007);
flatness to
based localization and trajectory planning. This can be generate
posed in trajectories.
the literature. In the
For former,
instance, the trajectory
Cowling et is
al. gener-
(2007);
atroller,
based major
troller,
based
a major
opens
localization
benefit
opens
localization
benefit
up
up new
in
new
in
and
and
possibilities
numeroustrajectory
possibilities
numeroustrajectorypresent
present
for
for autonomous
planning.
or future
autonomous
planning.
or future
This
This
image-
can
scenarios,
image-
can
scenarios,
be
be
Mellinger
generate
Mellinger
generate
ated by
and
and Kumar
trajectories.
Kumar
trajectories.
minimizing the
In
In
(2011)
the
(2011)
the former,
former,
weighted
use
use sum
differential
the trajectory
differential
the oftrajectory
the
flatness
is gener-
flatness
is
velocities gener-
to
to
and
based localization and trajectory planning. This can be Mellinger
generate
ated by and Kumar
trajectories.
minimizing theIn (2011)
the former,
weighted use sum differential
the oftrajectory
the flatness
is gener-to
ae.g.,
a
major
based
a major
e.g.,
benefit
autonomous
localization
benefit
autonomous
major benefit
in
in
in
numerous
driving.
and
numerous
driving.
numerous
A
trajectory
A UAV
present
UAV
present
UAV
present
or
equipped
planning.
or
equipped
or
future
future
future
with
This
with
scenarios,
aacan
scenarios,
scenarios,
cam-
cam-be the generate
atedunderlying
generate
ated by
trajectories.
by minimizing
minimizing model
trajectories. the
the
In
In the
the former,
weighted
determines
former,
weighted sum
the
sum
the
the the velocities
oftrajectory
control
of
is
is gener-
velocities
inputs
trajectory
the velocities as
and
and
the
gener-
and
e.g.,
era
a autonomous
can
major
e.g., contribute
benefit
autonomous indriving.
an
numerous
driving. A
additional
A present
UAV equipped
adjustable
or
equippedfuture with
with a
perspective
scenarios,
a cam-
cam- the
ated
the underlying
by minimizing
underlying model
model the determines
weighted
determines the
sum
the control
of the
control inputs
velocities
inputs as
as the
and
the
era
e.g., can contribute
autonomous an
driving. additional
A UAV adjustable
equipped perspective
with a cam- net
ated
the
net thrust
by
underlying
thrust and
minimizing
and the
model
the Euler
the
Euler angle
weighted
determines
angle accelerations.
sum
the of the
control
accelerations. In the
velocities
inputs
In the latter
as and
the
latter
era
on
e.g.,
era can
complex contribute
autonomous
can scenarios
contribute an
driving.
an toadditional
anticipate
A UAV
additional adjustable
situations
equipped
adjustable perspective
with wherea
perspective cam-an the
net underlying
thrust and model
the Euler determines
angle the control
accelerations. inputs
In the as the
latter
on
era complex
can scenarios to anticipate situations where an reference,
the underlying the optimization
model is
determines done for
the the
control snap (the
inputs fourth
as the
onera
on can contribute
complex
autonomous
complex
autonomous
scenarios
vehicle
contribute
vehicle an
scenarios
an to
could
to
could
additional
anticipate
be
additional stuck.
anticipate
be stuck.
stuck.
adjustable
situations
A
adjustable
situations
perspective
where
where an
multi-perspective
perspective
A multi-perspective
multi-perspective
net
net
an derivative
thrust
reference,
thrust
reference,
net thrustthe
andoptimization
the
and
the
of
the Euler
the Euler angle
the Euler angle
optimization
anddisplacement),
is
is done
angle
accelerations.
done
with
for
for the
the snap
accelerations.
aa model
accelerations. snap
In(the
In
that
thefourth
the
(the
uses
In(the
the
latter
latter
fourth
the
latter
on complex
autonomous
approach
on complex for scenarios
vehicle
environment
scenarios to
could anticipate
be
mapping
to anticipate situations
A
with interlinked
situations where
wheresys- an
sys-
an net reference,
derivative
reference, of
the optimization
displacement),
optimization is
is done
with
done for
for the
model
the snap
that
snap uses
(the fourth
the
fourth
autonomous
approach
autonomous for vehicle
environment
vehicle could
could be stuck.
mapping
be stuck. A
with
A multi-perspective
interlinked
multi-perspective derivative
thrust
reference,
derivative of
and
the
of displacement),
the torques
optimization
displacement), as
is with
controls.
done
with a model
for
a model
Raffo
the that
snap et
that uses
al.
(the
uses the
(2010)
fourth
the
approach
tems can
autonomous
approach for
be
for environment
used
vehicle to
environment solve
could mapping
be these
stuck.
mapping with
A
with interlinked
situations. To
multi-perspective
interlinked exploitsys-
sys- net
net thrust
derivative
thrust and
of
and the
the torques
displacement),
torques as
as controls.
with
controls.a Raffo
model
Raffo et
that
et al.
uses
al. (2010)
the
(2010)
tems
approach can be
for used to
environment solve these
mapping situations.
with To exploit propose
derivative a hierarchical
of displacement), control scheme,
with where
a model model
that predic-
uses the
tems
the
approach
tems can
benefits
can
the benefits
benefits
be
for
be ofused
such toa solve
ofenvironment
used
suchto setup,
a solve
setup,
these
it
mapping
these is with interlinked
situations.
necessary
is situations.
it is necessary toTooptimize
interlinked
toTo
exploit
exploit
optimize
sys-
sys- tivenet
net thrust
propose
thrust
propose
net control
thrust
and
and the
aa hierarchical
the
hierarchical
(MPC)
and
torques
the torques
is
torques
control
control
used
as
as
asto
controls.
scheme,
controls.
scheme,
track
controls.
Raffo
where
Raffo
where
the
et
model
et
model
trajectory
Raffo
al.
et al.
(2010)
predic-
(2010)
predic-
and
al. predic-
(2010) aa
tems
the can
temsposition be
can beofusedofused
such
the to
toa
UAV solve
setup,
solveand these
it
the
these situations.
necessary
field of
situations.view toTo
To exploit
optimize
(FOV) exploit of propose
tive
propose a
control
aa H hierarchical
(MPC)
hierarchical is control
used
control to scheme,
track
scheme, where
the
where model
trajectory
model and
predic-
the
the benefits
position
benefits of such
the
such a
UAV
a setup,
and
setup, it
the
it is
is necessary
field of
necessary view to
to optimize
(FOV)
optimize of tive
proposecontrol
nonlinear (MPC)
controller
hierarchical is used
to
control to track
stabilize
scheme, the
rotational
where trajectory
model and
movements.
predic- a
the
the position
camera
benefits
position of
of the
on-the-fly
such
the UAV
a setup,
UAV and
regarding
and the
it
the field
isthe of view
necessary
field of view (FOV)
application-specific
to (FOV)
optimize of tive
of tive control
nonlinear
control
nonlinear H
H∞
∞(MPC)
controller
(MPC)
controller
is
is used
to
used
to
to
to track
stabilize
track
stabilize
the
rotational
the
rotational
trajectory
trajectory and
movements.
and
movements.
a
a
the camera
position on-the-fly
of the UAV regarding
and the the
field application-specific
of view (FOV) of An
tive MPC
control
nonlinear scheme
H ∞(MPC) is
controller also
is presented
used
to to track
stabilize by Pereida
the
rotational and
trajectory Schoellig
and
movements. a
the
the camera
interests.
camera
interests. This
position
This
on-the-fly
ofoptimization
the UAV
on-the-fly
optimization
regarding
andcan
canthe
regarding the
reach
the
reach
application-specific
from
field of solely
view (FOV) following
application-specific
from solely following of An An MPC
nonlinear
MPC
(2018),
nonlinear but
scheme
H
scheme
H∞ in
is
∞ controller
is also
also
combination
controller
presented
to
to stabilize
presentedwith
stabilize
by
by
an
Pereida
rotational
Pereida
underlying
rotational
and
and Schoellig
movements.
Schoellig
adaptive
movements.
athe
the
a
camera
interests.
camera
interests. This
pre-defined
This
pre-defined
on-the-fly
optimization
vehicle
on-the-fly
optimization
vehicle
regarding
to thecan
regarding
to thecan
the
reach
generation
the
reach
generation
application-specific
from solely
of a following
bird’s
application-specific
from solely
of a following
bird’s eye
eye
An
An MPC
(2018),
MPC
(2018),
L
but
but
controller
scheme
∞in
scheme
in to
is
is also
combination
also
combination
improve
presented
presentedwith
with
trajectory
by
an
by
an
Pereida
underlying
Pereida
underlying
tracking
and
and Schoellig
adaptive
Schoellig
adaptive
accuracy. A
interests.
aview This
pre-defined optimization
vehicle to thecan reach
generation from solely
of aauser.following
bird’s eye An
L 1 MPC
(2018), but
controller scheme
in to is also
combination
improve presentedwith
trajectory by
an Pereida
underlying
tracking and Schoellig
adaptive
accuracy. A
view map
interests.
a map around
This
pre-defined around the
optimization
vehiclethe trajectory
to thecan
trajectory reachof
generation
of a
from
a roadsolely
of
road bird’s
user. Either
following
Eithereye (2018),
L 1 but
controller
further MPC in to combination
improve
scheme is with
trajectory
used by an underlying
tracking
Mueller and adaptive
accuracy.
D’Andrea A
a
view pre-defined
map around vehiclethe to the
trajectory generation
of a of
road a bird’s
user. Eithereye (2018),
L 1
further but
controller
MPC in to combination
improve
scheme is with
trajectory
used by an underlying
tracking
Mueller and adaptive
accuracy.
D’Andrea A
way,
away,
view the
the UAV
pre-defined
map UAV around needs
vehicle
needs to
the
to apply
to modern
the modern
trajectory
apply generation
of model-based
a of auser.
road
model-based bird’scontrol
Either
control eye L 1 controller
1
further
(2013) toMPC to
generate improve
scheme referenceis trajectory
used by
trajectories tracking
Mueller by and accuracy.
D’Andrea
minimizing A
the
view
way, map
the UAV around needs the
to trajectory
apply modern of a road
model-based user. Either
control L 1 controller
further
(2013) toMPC to
generate improve
scheme referenceis trajectory
used by
trajectories tracking
Mueller by and accuracy.
D’Andrea
minimizing A
the
strategies
view
way, map
the
strategies to
UAV
to complete
around needs
complete the
to its tasks.
trajectory
apply
its modern
tasks. of a road
model-based user. Either
control further
(2013)
jerk, to
i.e. MPC generate
the scheme
third reference
derivativeis used by
trajectories
of Mueller
displacement. by and D’Andrea
minimizing
However, the
way, the UAV needs to apply modern model-based control further
(2013) toMPC generatescheme referenceis used by
trajectoriesMueller by and D’Andrea
minimizing the
strategies
way, the UAV
strategies to
to complete
needs to its apply modern model-based control jerk,
tasks. (2013) i.e.
to the third
generate derivative
reference of displacement.
trajectories by However,
minimizing the
To this
strategies
To this end,
end,to aacomplete
complete its
control-oriented
its
control-oriented
tasks.
tasks. mathematical
mathematical model
model of
of the
the
jerk,
yaw
(2013)
jerk,
yaw
i.e.
tothe
component
i.e. the
component
third
generate
third of
of
derivative
the
reference
derivative
the body
body
of
of displacement.
rates
trajectories is
displacement.
rates is not
not
However,
by considered
minimizing
However,
considered and
the
the
and
Tostrategies
this end,to a complete its
control-oriented tasks. mathematical model of the jerk,
yaw
assumed
jerk, i.e.
i.e. the
componentto
the third
be
third of
zero. derivative
the
derivative body of
of displacement.
rates is
displacement. not However,
considered
However, the
and
the
quadcopter
To this
quadcopterend, is
isa required. A
control-oriented
required. A lot
lot of
of the mathematical
mathematical
the mathematical model models
of
models the yaw
assumedcomponentto be of
zero. the body rates is not considered and
To this
quadcopter end, isa control-oriented
required. A lot ofmathematical
the mathematical model of
models the yaw
assumed
yaw component
componentto be of
zero.
of the
the body
body rates
rates is
is not
not considered
considered and
and
found
To this
quadcopter
found in in the
end,
theis
in the aliterature
required.
literature use
control-oriented
A
use Euler
lot of
Euler theangles
mathematical to describe
mathematical
angles model
to describe
describe of
models rota-
the assumed
rota- assumed to be zero. to be zero.
quadcopter
found
quadcopter is required.
isliterature
required. use A
A lot of
of the
Euler
lot the mathematical
angles to
mathematical models
models rota- assumed to be zero.
found
found in
in the
the literature
literature use
use Euler
Euler angles
angles to
to describe
describe rota-
rota-
found in the literature use Euler angles to describe rota-
2405-8963 Copyright © 2022 The Authors. This is an open access article under the CC BY-NC-ND license.
Peer review under responsibility of International Federation of Automatic Control.
10.1016/j.ifacol.2022.10.481
14 Fadi Snobar et al. / IFAC PapersOnLine 55-27 (2022) 13–18

Quadcopter
Camera 2 0
FOV xquad

lpitch
Principal axis
Moving object zquad yquad
lroll

Point to focus the camera on 1 3

Fig. 1. A quadcopter tracking a moving object.

The framework of MPC is also used in this paper due Fig. 2. Quadcopter in body frame.
to its capability to consider constraints. In particular, a Euler equations and the fundamentals of quaternions are
nonlinear MPC formulation is chosen for the quadcopter given as follows
that is extended by a pinhole camera model to keep a  
  ṗ
point in space within the FOV and centralize it whenever p 1 ∗
 Q FQ + g 
possible. d   
 ṗ  = f (x, u, t) =  m

 , (1)
dt  Q   1 
Related works include Falanga et al. (2018); Penin et al.  Qω 
(2017); Tordesillas and How (2022). In Falanga et al. ω −1
2
   Im (τu − ω × (Im ω))
(2018) MPC is used to centralize a static object within
x
the image. The so-called perception-aware MPC presented T
uses a discrete MPC formulation with inequality con- where p = [x y z] and ṗ are the position and velocity
straints for the controls and velocity. The main innovations vectors with respect to the reference frame, F is the thrust
of the study presented in the following sections are the vector generated by the rotors, m is the vehicle’s mass,
T
use of a continuous MPC formulation with inequality con- g = [0 0 g] ms−2 represents the gravity vector, ω is the
straints to keep an object within the FOV of the onboard vector of angular velocities of the vehicle with respect to
camera, and the consideration of a moving object whose the body frame, Im and τu denote the mass moment of
future trajectory has to be predicted. inertia matrix and the input torque applied to the quad-
copter in the body frame respectively. The quadcopter’s
The paper is structured as follows: the problem is de-
orientation in the reference frame is represented by the
scribed and the models of the quadcopter and the car
unit quaternion Q = (q0 , q), where q0 = Re(Q) and
are introduced in Section 2. The desired states of the
q = Im(Q). The quaternion Q∗ = (q0 , −q) is the complex
quadcopter are defined in Section 3. Section 4 presents
conjugate of Q. The thrust and input torque vectors are
the fundamentals of MPC and the respective constraints.
given by
The MPC is then evaluated by simulations in Section 5.    
0 0
F = 0 = 0 , (2)
2. SETUP DESCRIPTION
−(T0 + T1 + T2 + T3 ) Fth
   
For this study, a trajectory following controller for a lroll (−T0 + T1 + T2 − T3 ) Mtx
quadcopter is used to track a moving object, in this case, τu = lpitch (T0 − T1 + T2 − T3 ) = Mty , (3)
a car. The objective for the quadcopter is to fly in a τ 0 + τ1 − τ2 − τ3 Mtz
predefined position relative to the car and to take images respectively, where Ti are the thrusts, and τi the torques of
of the field in front of it. Figure 1 illustrates this task. each motor, lroll is the distance from the center of gravity
Note, however, that the point to be tracked can be chosen (COG) generating the rolling moment Mtx , and lpitch is the
arbitrarily w.r.t. the position and orientation of the car, for distance from the COG generating the pitching moment
instance, to monitor the car itself. The predefined position Mty , as seen in Figure 2. The thrust and torque of a motor
T can be prescribed with the help of motor parameters.
of the quadcopter is set to xset = [xset yset zset ] relative
to the car. The figure also shows the camera focusing on a Given that τi ∝ Ti , the controls of the system are chosen
point in space a certain distance in front of the car, which to be
T
ensures that the image taken is of the area directly in front u = [T0 T1 T2 T3 ] . (4)
of it. Note that for simplicity the model ignores air resistance
and the gyroscopic effect of the rotors.
2.1 Quadcopter dynamics
2.2 Moving object definition
A quadcopter is a type of UAV with four rotors, as visu-
alized in Figure 2. The model utilizes a north-east-down In this study, the moving object is assumed to be a car
(NED) reference frame (global coordinate system) and which is modeled as a single-track model. This model sim-
a front-right-down (FRD) body frame (local coordinate plification of the actual four-wheeled vehicle was chosen
system). The state-space formulation of the equations of for its simplicity and sufficiency since the dynamics of the
motion described in Cariño et al. (2015) using Newton- vehicle itself are not the focus of this study.
 Fadi Snobar et al. / IFAC PapersOnLine 55-27 (2022) 13–18 15

C ψ̇ p Quadcopter
xquad
δ δ θ
FOV xm : Penalized distance
zquad
l Principal axis
vcar
xproj x
ψ 0 b Car mext
xglobal
yglobal b xcar
O zglobal zcar
xglobal xf

Fig. 4. Quadcopter and car in the xz-plane and its projec-

Fig. 3. Single-track model of the car.
tion into the camera’s FOV.
The used single-track model is shown in Figure 3, where
l denotes the wheelbase, δ the steering angle, ψ the angle Figure 4. The principal axis is defined as a vector starting
between the car’s x-axis and the global x-axis (yaw angle), from the center of the quadcopter with a length of 1 m
and vcar the velocity of the car at the rear-axle midpoint (arbitrarily chosen) and rotated −θ around yquad . With
b. It is assumed that the car’s coordinates, velocity vcar , respect to the global coordinates, the principal axis vector
and steering angle δ are provided at each sampling instant. is given by
Due to the high data sampling rate, vcar and δ are assumed  
to be constant which further simplifies the model. cos (−θ)
∗
xm = p + Q 0 Q, (7)
As described in Laumond (1998) and by assuming com- sin (−θ)
plete grip of the tires on the ground, the kinematics of the
where p are the coordinates of the quadcopter.
single-track model shown can be described as
    In the next step, the vector xm has to be extended
ẋc vcar cos(ψ)
 ẏc   vcar sin(ψ)  (or shortened) to reach the plane on which the car
    moves. The coordinates of the extended point xmext =
ẋc =  ψ̇  =  vcar tan(δ)
l
, (5)
v̇    T
[xmext ymext zmext ] are calculated using the straight line
car 0
δ̇ 0 projection
where xc and yc are the Cartesian coordinates of the rear- xmext = p + λ (xm − p) (8)
axle midpoint (b) in the fixed global coordinate system. between p and xm . The parameter
3. OBJECT TRACKING AND CAMERA −z
λ= , (9)
ORIENTATION zm − z
is obtained by solving the equation that results from
This section defines the quadcopter’s desired car-tracking enforcing that the projection reaches the plane zglobal = 0
trajectory and camera orientation with the help of the or, respectively, zmext = 0. Finally, xmext and ymext are
models presented in Section 2. obtained from Equation (8).
3.1 Desired trajectory With xmext known, its desired position xproj can be defined
to achieve the goal of centering the camera on a point
To track the car with the quadcopter, the desired position a certain distance in front of the car. The point xproj is
T
of the quadcopter pdes = [xdes ydes zdes ] must be defined. arbitrarily chosen to have a distance of xf in front of the
As mentioned earlier, the quadcopter’s desired position car , i.e. the rear-axle midpoint b, as depicted in Figure 4.
relative to the car is xset . Given that the car travels solely Thus the desired position of the point xmext to be tracked
on the xy-plane where zglobal = 0, the desired position of yields
the quadcopter thus result in  
  xc + xf cos ψ
xc + xset cos ψ − yset sin ψ xproj = yc + xf sin ψ .
pdes = yc + xset sin ψ + yset cos ψ . (6) 0
zset

3.2 Positioning the camera 4. MPC FORMULATION

One of the main challenges faced in this study is central-
izing the field in front of the car on the image. Therefore,
the quadcopter (and thus the camera on it) must position
itself in such a way, wherein a point a certain distance in
front of the car (xproj in Figure 4) lies on the principal
axis of the camera, i.e. in the center of the image.
At first, the principal axis has to be defined. For the sake
of simplicity, the camera is assumed to be fixed to the
quadcopter and tilted by an angle θ downwards, as seen in
The control method chosen for this paper is MPC due
to its ability to control multiple-input systems states
while accounting for constraints. The core principles of
MPC are briefly reviewed and the constraints for the
quadcopter-based FOV-tracking problem are defined. For
further reading on this topic please refer to Rawlings et al.
(2017); Grüne and Pannek (2016); Camacho and Bordons
(2004); Graichen and Käpernick (2012), on which the
following is also based.
16 Fadi Snobar et al. / IFAC PapersOnLine 55-27 (2022) 13–18

4.1 Model Predictive Control Y FOV

X
X Y X
MPC aims to find an optimal input control trajectory u∗ y x2d
over a prediction horizon T > 0 that minimizes a cost C x2d x fl YZ
Z C c
functional J(u) for each time step tk = t0 + k∆t with the c Z
camera principal axis fl
sampling time ∆t > 0. A general optimal control problem
in this regard is given by center image plane
 T
min J(u) = V (x(T )) + l(x(τ ), u(τ )) dτ (10a) Fig. 5. Pinhole camera geometry based on Hartley and
u(τ ) 0 Zisserman (2000).
s.t. ẋ(τ ) = f (x(τ ), u(τ ), tk + τ ), x(0) = xk (10b)
downwards at an angle of θ. The corner points of the FOV
h(x(τ )) ≤ 0 (10c) on the image plane are given by
u(τ ) ∈ [umin , umax ], (10d)  
xCPiimage
where x(τ ) ∈ RNx are the states and u(τ ) ∈ RNu the xCPiimage =  yCPiimage  , i = 0, 1, 2, 3 . (14)
controls predicted over the MPC-internal time τ ∈ [0, T ]. 1
Equation (10b) consists of the dynamics of the quadcopter
(1) and the car (5). The cost to be minimized consists of With the FOV’s corner-points, the point xproj has to be
the integral cost function 1 translated and rotated properly to achieve a correct map-
l(x(τ ), u(τ )) = ||p(τ ) − pdes (τ )||2Qp ping. The first step is to translate the coordinates of the
point xproj accordingly. Then xproj is rotated in such a way
+ ||xmext (τ ) − xproj (τ )||2Qproj that the quadcopter’s coordinates would be rotated back
+ ||u(τ ) − udes (τ )||2Rw (11) from the global into the quadcopter coordinate system.
The last step is to rotate the coordinates by −θ around
with the desired controls udes ∈ R . The terminal cost
Nu
yglobal , because the FOV’s corner-points are on the xy-
function is defined as plane. All these steps are summed up in the following
V (x(T )) = ||p(T ) − pdes (T )||2Pp equation
+ ||xmext (T ) − xproj (T )||2Pproj . (12) xprojimage = Ry (Q(xproj − p)Q∗ ), (15)
The weighting matrices Pp , Pproj , Qp , Qproj , Rw are where Ry is a rotation matrix defined as
positive semi-definite. Moreover, (10c) accounts for the  
cos (−θ) 0 sin (−θ)
constraints defined in the next subsection. In addition, Ry = 0 1 0 . (16)
(10d) constitutes the box constraints for the control in- − sin (−θ) 0 cos (−θ)
puts u(τ ). For the sake of readability, the explicit time
dependency is omitted for the rest of the paper. Finally, the point xprojimage is mapped to the image plane
(from R3 to R2 ), for which Equation (13) is used with
4.2 Constraints fl = 1, resulting in
x 
projimage
The camera mounted on the quadcopter is modeled as   z 
xproj2d  projimage 
a pinhole camera as described in Hartley and Zisserman xproj2d = = . (17)
(2000), see Figure 5. The two-dimensional plane at a yproj2d  yprojimage 
distance of Z = fl from the camera center C is called the zprojimage
image plane. The point X in space is mapped onto the The inequality constraints (10c) for the MPC problem can
image plane where the line joining C with X crosses the then be expressed as
plane. Using the principles of similar triangles, the point    
T h0 xCP0image − xproj2d
X = [X Y Z] is mapped to h1   −yCP0
  + yproj2d 
Y T h(x(τ )) =  
h2  = −xCP3image + xproj2d  ≤ 0. (18)
image
x2d = fl XZ fl Z
(13)
on the image plane. For further detail on camera projection h3 yCP3image − yproj2d
and image formation, please refer to Hartley and Zisser-
man (2000). The box constraints (10d) are given by
umin = [0 0 0 0] N, (19)
To keep xproj within the FOV, the edges of the FOV
are handled as inequality constraints on the image plane and
(red rectangle in Figure 5) by using the principles of umax = [T0max T1max T2max T3max ] N, (20)
camera projection. The image plane is assumed to be at where Timax is the maximum thrust each rotor can pro-
Z = zglobal = 1. The camera model is defined starting duce.
from the origin of the reference frame (global coordinate
system). Thus the point xproj , i.e. the point which is to be 5. EVALUATION
kept within the FOV, must be appropriately translated
and rotated onto the image plane, since in reality the The evaluation of the MPC was done with the help
camera is mounted on the quadcopter and is pointing of simulations in different scenarios. For this purpose,
1 The squared weighted norm for a vector z ∈ Rn and weighting the MPC toolbox GRAMPC (Englert et al., 2019) for
matrix S ∈ Rn×n , is defined as ||z||2S = z T Sz. nonlinear continuous-time systems was used.
 Fadi Snobar et al. / IFAC PapersOnLine 55-27 (2022) 13–18 17

Scenario 1 Scenario 2 Scenario 3 Scenario 1 Scenario 2 Scenario 3

1
xcar0 p0
0
0.5
5

ximage
0
yglobal

10 −0.5 xproj2d (0)

xproj
15 −1
−1.5 −1 −0.5 0 0.5 1 1.5
xmext yimage
20
0 5 10 15 20 25 30
xglobal
Fig. 7. Trajectories of xproj2d on the image plane in the
Fig. 6. Trajectories of the quadcopter (solid lines) and car three test scenarios.
(dotted lines) in the three test scenarios.
5.1 Test scenario definition

Three scenarios are evaluated, in which the car takes
turns with varying radii. The plots of the car’s rear-wheel
(rear-axle midpoint) trajectories can be seen in Figure 6
as dotted lines, whereas the solid lines represent the
corresponding flight paths of the quadcopter. The initial
positions of the car and quadcopter are xcar (0) = xcar0 =
T T
[0 0 0] m and p(0) = p0 = [4 0 − 7.5] m, respectively.
The initial conditions are identical in each test scenario
to allow for better comparison. In all of the scenarios, the
following MPC parameters
T = 0.75 s, ∆t = 10 ms, Tsim = 10 s, Nhor = 30, (21)
are chosen, where Tsim is the simulation duration and Nhor
the number of discretization points within GRAMPC. Fur-
thermore, the desired position of the quadcopter relative
T
to the car is set to xset = [2.5 0 − 7.5] m, i.e. the
quadcopter is to fly 2.5 m in front of the car and 7.5 m
above it.
To follow the car’s trajectories in Figure 6 (dotted lines),
appropriate weighting matrices have to be chosen. The
following matrices
Qp = Pp = diag (60, 60, 100) m−2 , (22a)
−2
Rw = 2.5I4×4 N , (22b)
−2
Qproj = Pproj = 5I2×2 m , (22c)
have shown good results for following the trajecto-
ries, stabilizing the quadcopter and minimizing ||xmext −
xproj ||Qproj ,Pproj .
5.2 Results
The results of the quadcopter tracking the car can be seen
in Figure 6 as the solid lines. It is noticeable that, when the
car drives forward, i.e. δ = 0 deg in (5), the quadcopter’s
trajectory on the xy-plane – as to be expected – is
identical to that of the car. However, when the car takes
turns, the quadcopter turns with a larger radius than
the car, as it tries to stay 2.5 m in front of the car. For
demonstration purposes, the principal axis (dashed red
line), the intersection point of the principal axis with
the xy-plane xmext , and the point on which the camera
is to focus xproj for scenario 3 at the last time step of
the simulation are also illustrated in the Figure 6. It can
be seen that the camera is pointing towards its desired
direction, but that the point xproj is positioned slightly to
the left of the principal axis, therefore the point xprojimage
would be slightly to the left of the center of the image.
Figure 7 shows the image plane, where the red area repre-
sents the camera’s FOV. Anything beyond the edges is not
seen by the camera. Depicted are also the trajectories of
the point xprojimage . It can be seen that during the entire
simulation, the point xproj is kept within the FOV and
close to the camera center. The points drifts slightly away
from the center when the car takes sudden turns, which
the MPC cannot predict, then returns as the quadcopter
adjusts its trajectory. However, since neither trajectory
reaches the edges of the FOV, the constraints remain
inactive at any point, i.e. h(x(t)) < 0 ∀t ∈ [0, Tsim ].
A more aggressive fourth scenario considers sudden ac-
celerations and breaking events of the car. The image
plane for the fourth scenario is seen in Figure 8 with
xproj2d (0) as the starting position of both trajectories and
xproj2d (Tsim ) as the end position. The figure shows that
with the constraints ensure the visibility the point xproj
within the FOV throughout the simulation. In contrast
to this, the yellow trajectories are obtained when the
constraints are deactivated, where the visibility is partly
lost in the scenario.
A drawback of MPC is usually the high computation effort
and implementation in real-time. Table 1 summarizes the
average computation time tc per sampling interval for each
scenario. For the given hardware (standard laptop with
Intel Core i7-1165G7 processor), tc < ∆t in all scenarios
shows that the MPC scheme is real-time capable. However,
additional tuning and optimized code implementation is
still necessary to achieve real-time feasibility on an on-
board CPU.
18 Fadi Snobar et al. / IFAC PapersOnLine 55-27 (2022) 13–18
W/o constraints W/ constraints
1.5
1 Englert, T., Völz, A., Mesmer, F., Rhein, S., and Graichen,
0.5
ximage
xproj2d (Tsim )
0
−0.5 xproj2d (0)
−1
−1.5 −1 −0.5 0 0.5
yimage
Fig. 8. Trajectories of xproj2d on the image plane without
and with constraints in scenario 4.
Table 1: Average MPC computation time.
Scenario 1 Scenario 2 Scenario 3
tc 4.0 ms 4.2 ms 4.4 ms
6. CONCLUSIONS
The paper presents an MPC scheme for tracking an
object or a point relative to the object within the field-
of-view (FOV) of a quadcopter-mounted camera. First,
a quaternion-based representation of the quadcopter and
the car’s dynamics are described. The car model is used
to predict its future trajectory based on the current
steering angle and velocity. The FOV is defined by a
pinhole camera model and corresponds to four nonlinear
inequality constraints within the MPC formulation. The
trajectory tracking and desired orientation were realized
by defining the quadcopter’s desired states dependent on
the car’s coordinates and orientation. The MPC problem
is implemened using the toolbox GRAMPC. The results
show that MPC is suitable for stabilizing the quadcopter
along the predicted trajectory. Furthermore, the inequality
constraints integrated in the MPC formulation keep the
predefined point in space within the FOV of the camera.
In future studies. the controller should be tested and
validated on a real quadcopter. Although it is shown
that the MPC scheme is real-time feasible on a standard
computer, additional tuning and code optimization is
necessary to maintain real-time feasibility on the onboard
hardware of the quadcopter.
REFERENCES
Camacho, E. and Bordons, C. (2004). Model Predictive
Control. Advanced Textbooks in Control and Signal
Processing. Springer London.
Cariño, J., Abaunza, H., and Castillo, P. (2015). Quadro-
tor Quaternion Control. Proc. of IEEE International
Conference on Unmanned Aircraft Systems.
Cowling, I., Yakimenko, O., Whidborne, J., and Cooke,
A. (2007). A prototype of an autonomous controller for
a quadrotor UAV. In Proc. of IEEE European Control
Conference, 4001–4008.
K. (2019). A software framework for embedded non-
linear model predictive control using a gradient-based
augmented Lagrangian approach (GRAMPC). Opti-
mization and Engineering, 20, 769–809.
Falanga, D., Foehn, P., Lu, P., and Scaramuzza, D. (2018).
PAMPC: Perception-Aware Model Predictive Control
for Quadrotors. In Proc. of IEEE/RSJ International
Conference on Intelligent Robots and Systems, 1–8.
Graichen, K. and Käpernick, B. (2012). A real-time
gradient method for nonlinear model predictive control.
In T. Zheng (ed.), Frontiers of Model Predictive Control,
1 1.5
chapter 1. IntechOpen, Rijeka.
Grüne, L. and Pannek, J. (2016). Nonlinear Model Predic-
tive Control: Theory and Algorithms. Communications
and Control Engineering. Springer International Pub-
lishing.
Hartley, R. and Zisserman, A. (2000). Multiple View
Geometry in Computer Vision. Cambridge University
Press.
Scenario 4 Laumond, J.P. (1998). Robot Motion Planning and Con-
trol. Springer, Berlin.
4.6 ms Magnussen, O., Ottestad, M., and Hovland, G. (2013).
Experimental validation of a quaternion-based attitude
estimation with direct input to a quadcopter control
system. In Proc. of IEEE International Conference on
Unmanned Aircraft Systems, 480–485.
Mellinger, D. and Kumar, V. (2011). Minimum snap
trajectory generation and control for quadrotors. In
Proc. of IEEE International Conference on Robotics and
Automation, 2520–2525.
Mueller, M. and D’Andrea, R. (2013). A model predictive
controller for quadrocopter state interception. In Proc.
of IEEE European Control Conference, 1383–1389.
Penin, B., Spica, R., Giordano, P.R., and Chaumette, F.
(2017). Vision-based minimum-time trajectory genera-
tion for a quadrotor uav. In Prof of IEEE/RSJ Inter-
national Conference on Intelligent Robots and Systems,
6199–6206.
Pereida, K. and Schoellig, A.P. (2018). Adaptive model
predictive control for high-accuracy trajectory tracking
in changing conditions. In Proc. of IEEE/RSJ Inter-
national Conference on Intelligent Robots and Systems,
7831–7837.
Raffo, G., Ortega, M., and Rubio, F. (2010). An integral
predictive/nonlinear H∞ control structure for a quadro-
tor helicopter. Automatica, 46(1), 29–39.
Rawlings, J., Mayne, D., and Diehl, M. (2017). Model
Predictive Control: Theory, Computation, and Design.
Nob Hill Publishing.
Tordesillas, J. and How, J.P. (2022). PANTHER:
Perception-Aware Trajectory Planner in Dynamic En-
vironments. IEEE Access, 10, 22662–22677.
Yang, R., Zheng, L., Pan, J., and Cheng, H. (2021).
Learning-based predictive path following control
for nonlinear systems under uncertain disturbances.
Robotics and Automation Letters, 6(2), 2854–2861.