12th IFAC Symposium

Budapest, on Robot
Hungary, August Control
27-30, 2018
Budapest,
12th Hungary, August
IFAC Symposium 27-30,
on Robot 2018
Control
Budapest, Hungary, August 27-30, Available online at www.sciencedirect.com
2018
12th IFAC Symposium on Robot Control
Budapest, Hungary, August 27-30, 2018
ScienceDirect
IFAC PapersOnLine 51-22 (2018) 542–547
Control
Control of of a a line
line following
following robot robot based
based onon
Control of a FSM line following
estimation robot based on
Control of a FSM FSM
estimation
line following
estimation robot based on
∗
Alexia Toumpa FSM Alexandros Kouris ∗,∗∗
estimation Fotios Dimeas ∗∗
Alexia Toumpa ∗ Alexandros Kouris ∗,∗∗
∗ Fotios Dimeas
Nikos Aspragathos∗,∗∗
Alexia Toumpa ∗ Alexandros Nikos Aspragathos Kouris ∗ Fotios Dimeas ∗
∗ Nikos Aspragathos∗,∗∗ ∗
∗ Alexia Toumpa Alexandros Kouris Fotios Dimeas ∗
∗ Robotics Group, Mechanical Engineering & Aeronautics,
∗ University of
Robotics Group, Mechanical NikosEngineering
Aspragathos & Aeronautics, University of
∗Patras, Greece, {touba@ceid, dimeasf@mech, asprag@mech}.upatras.gr
Robotics
Patras,
∗∗ Group,
Greece, Mechanical dimeasf@mech,
{touba@ceid, Engineering & Aeronautics, University of
asprag@mech}.upatras.gr
∗∗ Electrical
∗Patras, and Electronic Engineering Department, Imperial College
Robotics Greece,
Electrical Group,and{touba@ceid,
Electronic dimeasf@mech,
Engineering asprag@mech}.upatras.gr
Department, Imperial College
∗∗ London, UK, a.kouris16@imperial.ac.uk University of
Mechanical Engineering & Aeronautics,
Electrical
Patras, Greece, and Electronic
London,
{touba@ceid, UK,Engineering Department,
a.kouris16@imperial.ac.uk
dimeasf@mech, Imperial College
asprag@mech}.upatras.gr
∗∗ London, UK,Engineering
a.kouris16@imperial.ac.uk
Electrical and Electronic Department, Imperial College
Abstract: Line following robots London, areUK,becoming increasingly popular both in industrial and in
a.kouris16@imperial.ac.uk
Abstract: Line following robots are becoming increasingly popular both in industrial and in
domestic applications. In these real world environments, noisy or inaccurate measurements lead
Abstract:
domestic Line following
applications. In theserobots realare becoming
world increasingly
environments, noisypopular both inmeasurements
or inaccurate industrial and leadin
to ambiguous conclusions about the robot’s state relative to the path. In this paper, a Finite
domestic
to ambiguous
Abstract: applications.
Lineconclusions
followingIn these
about
robots real
the
areworld
robot’senvironments,
becoming state relative
increasinglynoisy or inaccurate
topopular
the path.both measurements
In this
in paper, a and
industrial lead
Finite in
State Machine is introduced that detects irregularities in the measurements from an array
to ambiguous
State
domestic Machine conclusions
is introduced
applications. In about
these that theworld
real robot’s
detects state relative
irregularities
environments, intothe
noisy the
or path. In this
measurements
inaccurate paper,
from an
measurements a Finite
array
lead
of reflectance sensors. Based on the estimated state, the FSM switches between a proposed
State
of Machineconclusions
to reflectance
ambiguous is introduced
sensors. Based on that thedetects
estimatedirregularities
state, theinFSM the measurements
switches
path. In between from an
a proposedarray
variable-gain PD controllerabout for linethe robot’s
following state
and anrelative
open to loop thecontroller thishandling
for paper, a special
Finite
of reflectance
variable-gain
State Machine sensors.
PD Based
controller
is introduced foron the
line estimated
following andstate,
an the
open FSM
loop switches
controller between
for a
handling proposed
special
cases. Experimental evaluationthat detects irregularities
demonstrates that the proposedin the approach
measurements reduces from an array
considerably
variable-gain
cases.
of Experimental
reflectance PD controller Basedfor
evaluation line
thefollowing
ondemonstrates andstate,
that an
theopen
proposed loop approach
controller for handling
reduces special
considerably
the amount of sensors.errors caused by noisy estimated
measurements andtheimproves
FSM switches between
the performance aofproposed
the line
cases.
the Experimental
amount
variable-gain ofPD errors evaluation
causedfor
controller by demonstrates
noisy
line measurements
following that
and theand
an proposed
open improves
loop approach reduces
the performance
controller for considerably
handling of the line
special
following robot compared to a similar approach from the literature.
the
cases. amount
following of errors
robot
Experimental compared caused
evaluation to abysimilar
noisy measurements
demonstrates approach theand
from
that the improves
literature.
proposed the performance
approach of the line
reduces considerably
following
the amount robot compared
of(International
errors caused to abysimilar
noisy approach from and the literature.
© 2018, 1.IFAC
INTRODUCTION Federation of measurements
Automatic Control)
lenging
improves
Hosting
and
the performance
by Elsevier
requires more Ltd.complex
All rightsof reserved.
the line
algorithms than the
following robot
1. INTRODUCTION compared to a similar approach from the literature.
lenging and requires more complex algorithms than the
1. INTRODUCTION simpler
lenging approach
and requires withmorethree sensors (Borcosi et al., 2006).
simpler approach with threecomplex algorithms
sensors (Borcosi than
et al., the
2006).
Line following forms 1. a challenge
INTRODUCTION in robot competitions simpler
To
lenging approach
accurately
and with
track
requires three
the
more sensors
desired
complex (Borcosi
path, a
algorithms et
standardal.,
than 2006).
PID
the
Line following forms a challenge in robot competitions To accurately track the desired path, a standard PID
to
Linetestfollowing
the functionality
forms a and dynamics
challenge in of mobile
robot wheeled closed
competitions simpler loop
approachcontrol system
with three is usually
sensors employed.
(Borcosi et An
al., opti-
2006).
to test the functionality and dynamics of mobile wheeled To accurately tracksystem
the desired path, a standard PID
robots. Moreover, as automatic guided vehicles becom- closed
arewheeled loop control is usually employed. An opti-
to testfollowing
robots.
Line the functionality
Moreover, and dynamics
as automatic
forms a challenge guidedin of mobile
vehicles
robot are becom- mised
competitions closed
mised
To
PID
loop
PID
accurately
controller
control
controller
track
for lineisfollowing
system
for
the line usually
following
desired path,
robots
employed.
robotsa
was proposed
was
standard An opti-
proposedPID
ing increasingly
robots. Moreover, popular,
as automatic the use of line
guided following
vehicles robots in Balaji et al. (2015) to achieve higher velocity in consec-
arewheeled
becom-
ing increasingly
to spreading
test the popular,
functionality the dynamics
and use of line offollowing
mobile robots mised in Balaji
closed PID
loop etcontroller
al. (2015)
control for
to line
system following
achieve
is higher
usually robots
velocity
employed. wasinproposed
An consec-
opti-
is in commercial applications. The ability of
ing
is increasingly
spreading
robots. Moreover, popular,
in commercial
as automaticthe applications.
use of line
guided following
Theare
vehicles robots
ability
becom- of utive
in turns,
Balaji
utive
mised turns,
PID
by (2015)
considering
etcontroller
al.
by considering to line
for
a temporal
achieve higher
a following
temporal
window
velocity
window
robots was
ofinthe most
consec-
of proposed
the most
robots
is to
spreading navigate
in accurately
commercial and reliably
applications. across
The a marked
ability of recent error measurements for the integral term. A speed
robots to navigate
ing increasingly accurately
popular, the and
use reliably
offor across a marked
linecarrying
following robots utive recent
in turns,
Balaji erroret by considering
measurements
al. (2015) to afor
achievetemporal
thehigher window
integral term.
velocity ofinthe most
Aconsec-
speed
path on the ground, can be used products
robots
path
in
onto the
is warehouses
spreading navigate
ground, accurately
inorcommercial
for transferring
and
usedreliably
can be applications. across
for carryingThe aproducts
marked
ability of variation
recent
variation error
utive turns,
parameter
measurements
parameter
by
that slows
that slows
considering afor
down
the
down
temporal
the robot
integral
the
window term.
robot of
inAcurved
in speed
thecurved
most
path
in
robots on
warehouses
to the ground,
navigate or for can be and
transferring
accurately usedmaterials
for carrying
materials
reliably
between
between
across a
man-
man- paths
products
marked variation
paths
to
to
improve
parameter
improve
stability
that slows
stability
was
was
also
down
also
introduced,
the robot
introduced,
along with
inAcurved
along with
ufacturing points. Line following technology can provide recent error measurements for the integral term. speed
in warehouses
ufacturing
path on the or for
points.
ground, transferring
Line canfollowing
be used materials
technology
for between
carrying canproductsman- an
provide paths
an
opento
open loop
loop controller
improve stability
controller
towas
to
handle
also
handle
the saturation
introduced,
the saturation along ofwith
the
of the
autonomous capabilities to vehicles in environments where variation
input during parameter
sharp that
turns. slows down the robot in curved
ufacturing
autonomous
in warehouses points.
capabilities
or for Line following technology
to vehiclesmaterials
transferring in environments can
between provide
where
man-
rail infrastructure
autonomous is not
capabilities available.
to Applications
vehiclestechnology
in environments of line fol- an
where
input
pathsopenduringloopsharp
to improve controller
turns.towas
stability handle the saturation
also introduced, alongofwith the
rail infrastructure
ufacturing points. isLine
not following
available. Applications canof line fol- In
provide input
an during
real-world
open loop sharp turns.
applications
controller to and
handle robotthe competitions,
saturation com-
of the
lowing
rail robots include
infrastructure is not health care services
available. Applications (Punetha of lineet fol-
al., In real-world applications and robot competitions, com-
lowing
autonomous robotscapabilities
include health
to care services
vehicles in(Horan (Punetha
environments et al., plex
where input line
duringstructures
sharp such
turns. as crossovers, junctions, dashed
2013),
lowing manufacturing
robots include transportation
health care services et
(Punetha al., 2011),
al., In
et fol- plexreal-world
line structures applications
such asand robot competitions,
crossovers, junctions, dashed com-
2013), manufacturing
rail infrastructure nottransportation
is(Colak available. (Horan
Applications et ofal.,line
2011), lines, acute and right-angles etc may exist on thedashed path.
and
2013), entertainment
manufacturing and Yildirim,
transportation (Horan2009). et al., 2011), plex
lines, line
acute structures
and such
right-angles as crossovers,
etc may junctions,
exist on the path.
and entertainment (Colak and Yildirim, 2009). In real-world
lowing robots include health care services (Punetha et al., Moreover, inaccurate and noisy measurements may appear applications and robot competitions, com-
and
For
2013),aentertainment
robot (Colak
to accurately
manufacturing and Yildirim,
navigate
transportation along
(Horan2009).
a marked path, lines,
et al., 2011), Moreover,
plex acute
line and right-angles
inaccurate
structures and noisy
such as etc may exist
measurements
crossovers, on
junctions, may thedashed
path.
appear
For a robot to accurately navigate along a marked path, due to surface wear, and lighting variation and bouncing of
the
and deviation
aentertainment from the reference
(Colak path
and Yildirim, needs
2009). to be mea- Moreover, due
lines, toacute inaccurate
surface and wear,
right-angles noisyetc
lighting measurements
variation
may exist and on may theappear
bouncing path. of
For robot tofrom
the deviation accurately navigate
the reference pathalong a marked
needs mea- the
to be path, due
the
robot.
to surface
robot.
Moreover,
To enhance
To enhance
inaccuratewear, and the performance
lighting
thenoisy variation
performance
measurements
of the
ofand
robot,
the bouncing
robot,
may
accu-
accu-
appear of
sured. As an alternative to classical computer vision algo-
mea- rate identification and the appropriate handling
the
For deviation
sured.
rithms
As an to
a robot
(Matczak
from
alternative the reference
accurately tonavigate
and Mazurek,
classical path
2015),
needs
computer
along low-cost
to be path,
vision
a marked algo-
Infra-Red the
due robot.
rate to To enhance
identification
surface andlighting
wear, performance
appropriate variation theof
ofand
handling
such accu-
robot,
ofbouncing
cases
such cases of
sured.
rithms
the As
deviation an alternative
(Matczak from andthe to classical
Mazurek,
reference 2015),
path computer
low-cost
needs vision
to algo-
Infra-Red
be mea- should
rate be incorporated
identification and the on the
appropriate line following
handling algorithms.
(IR) sensors can be employed that can drastically reduce should
the robot. be incorporated
To enhance on performance
the line following of theof such accu-
algorithms.
robot, cases
rithms
(IR)
sured. (Matczak
sensors
As an can andemployed
be
alternative Mazurek, 2015),
that
to classical can low-cost
drastically
computer Infra-Red
reduce
the
(IR) computational
sensors can be demands.
employed Assuming
that can that vision
drastically
algo- should
the desired
reduce lowing
In
ratethis be
work, incorporated
identification we propose onathe
and appropriate line following
control
In this work, we propose a control method for a line fol-
method for
handling algorithms.
of a linecases
such fol-
the
rithmscomputational
(Matczak and demands.
Mazurek, Assuming
2015), that
low-cost the desired
Infra-Red should robot
be equipped
incorporated with
on theIR sensors,
line based
following on a
algorithms. Finite
path
the of the robot is a dark coloured line on a light coloured In this robot
lowing work, equipped
we propose witha control
IR sensors,method based forona aline fol-
Finite
(IR)computational
path of the robot
sensors is ademands.
can deviation
be dark coloured
employed Assuming
that line
can on that
a light
drasticallythecoloured
desired
reduce
can State Machine (FSM) that estimates thebased
robot’s
background,
path of the robot
background,
the computational
the is ademands.
the deviation
of the
dark coloured
of the
robot
line on
robot
Assuming
from
from
that coloured
a light
that that
the
line
line can lowing
desired State
In this robot
Machine
work, equipped
we(FSM)
propose with
that a IR sensors,
estimates
control the robot’s
method for ona position
aline
Finite
position fol-
be determined using multiple IR reflectance sensors, the relative to the (FSM) desiredthat path. The introduced FSM can
background,
be
path determined
of the theusing
robot deviation
is a multiple
dark of the
colouredIRrobot
reflectance
line from
on a that line can
sensors,
light coloured the Staterelative
lowing Machine
to the
robot desiredwith
equipped path.estimates
IR The
sensors,thebased
robot’s
introduced onFSMposition
a can
Finite
number and arrangement of which is widely studied the identify
in the relative
identify
noisy,
to theunclear,
noisy, desiredthat
unclear,
or conflicting
path.
or
IR sensor FSM
The introduced
conflicting IR sensor
measure- can
be determined
number
background, theusing
and arrangement multiple
deviation IRrobot
ofofwhich
the reflectance
is widely fromstudiedsensors,
that linein can State Machine
ments such as (FSM)
sharp turns, estimates
crossovers, the robot’smeasure-
junctions position
and line-
literature (Baharuddin et al., 2006; Ebiesuwa et al., 2013).
number
literature
be and
determined arrangement
(Baharuddin
using et ofal.,which
multiple 2006;
IR isEbiesuwa
widely studied
reflectance et in the identify
al., 2013).
sensors, ments
relative noisy,
such
to as
the unclear,
sharp
desired or
turns, conflicting
path.crossovers,
The IR sensor FSM
junctions
introduced measure-
and line-can
To map the sensor measurements into an estimation of the background colour swaps, without requiring complex sen-
literature
To map the
number and(Baharuddin
sensor
arrangement et of
measurementsal.,which
2006; into
isEbiesuwa
an estimation
widely of the ments
et al., 2013).
studied in background
identify such
noisy, as sharp
colour turns,
swaps,
unclear, or crossovers,
without
conflicting junctions
requiring
IR sensorcomplexand line-
measure-sen-
robot’s
map position relative to the desired anpath, Osorio of al. sor
etthe structures. Theswaps,
outputwithoutof the FSM switches between
To
robot’s
literature the sensor
position
(Baharuddin measurements
relative et toal.,
the into
desired
2006; estimation
path,
Ebiesuwa Osorio
etatal.,the et al. background
2013). sor
ments
a
structures.
proposed
colour
The output
such variable-gain
as sharp turns, PD
of the FSM
crossovers, requiring
controller
switches
junctions
that
complex between
and
improves
sen-
line-
the
(2006)
robot’s used reflectance
position relative sensors
to the at the
desired front
path, and Osorio rear
et al. sor
a structures.
proposed The
variable-gain output PD of the FSM
controller switches
that improvesbetween the
(2006)
Tothe used
maprobot. reflectance
the sensor sensors
measurements at the front
into anearly and
estimation at the rear
of the background
robot’s colour
performance swaps,
when without
ordinary requiring
path complex
structures sen-
are
of
(2006) used This
reflectance approach
sensors facilitates
at the front and prediction
at theetrear of a proposed
robot’s variable-gain
performance when ofPD controller
ordinary that
path improves
structures the
are
of the robot.
robot’s position This approach
relative to facilitates
the desired early
path, prediction
Osorio al. sor structures.
of confronted and The
an output
open loop the FSM
controller switches
to handle between
special
upcoming
of the robot. turnsThis and identifies
approach the robot’s
facilitates early alignment
prediction withof robot’s
confronted performance
and an open when loop ordinary
controller path structures
to handle are
special
upcoming
(2006) used turns and
reflectance identifies
sensors the
at therobot’s
front alignment
and at the with
rear a proposed
cases such as variable-gain PD controller that improves the
respect
upcoming
respect
of the
to turns
the line.
to theThis
robot. and
line.
In Bajestani and Vosoughinia (2010),
Inidentifies
Bajestani
approach theand robot’s
facilitates alignment
Vosoughinia
early prediction with
(2010), of- confronted
cases
robot’s such andloss
as
performance an of
loss open
of
line.loop
line. The
when
Thecontroller
contribution
contribution
ordinary path to handleof thisspecial
of this work
structures
work
are
an array of 13 sensors at a circular arc configuration lies in the combination of the introduced FSM for smart
respect
an arraytoof
upcoming the13line.
turns Inidentifies
sensors
and Bajestani
at a circular
theand Vosoughinia
robot’sarc alignment
configuration (2010),
with - cases
lies insuch
confronted as
and loss
the combination an of
openline. Thecontroller
ofloop
the contribution
introduced to FSM of this
handle work
forspecial
smart
enhanced by 13three extra at sensors at thearc front- was used to- filtering of IR measurements, with a motionFSM controller that
an arraytoof
enhanced
respect by
the three sensors
line. extra a circular
sensors
IninBajestani at
and configuration
theVosoughinia
front- was used(2010),to lies casesinsuch
filtering
adjusts
theof IR
the
combination
as measurements,
loss response
robot’s of line.of the
The
to
introduced
with
turns
a motion controller
contribution
and its linear
for smart
of this that
work
velocity
improve
enhanced the accuracy
byaccuracy
three extra sharp
sensors turns
at the of the path.
front- However,
used to- filtering
wasHowever, adjusts of IR
the measurements,
robot’s response towith
turnsa motion
and itsFSM controller
linearforvelocity that
improve
an array the
of 13 sensors in sharp
at a turns
circular of arc
the path.
configuration lies in the combination of the introduced smart
fusing the data of all sensors inturns
this of configuration is chal- based onthe the complexity of the path.andIn this way, velocity
the line
improve
fusing
enhanced the the accuracy
data
by three of all in sharp
sensors
extra sensors in this
at the the path.
configuration
front- wasHowever, to adjusts
is chal-
used based
filtering onofthe robot’s
IR response
complexity
measurements, towith
of the turns
path.
a motion its controller
In this linear
way, the that line
fusing
improve the thedata of all sensors
accuracy in sharpinturns this of configuration
the path. However, is chal- based adjustsonthe the complexity
robot’s response of the path.and
to turns In this way, velocity
its linear the line
Copyright
fusing the©© data
2018 IFAC of all(International
sensors in this configuration is chal- 548 based on the complexity of the path. In this way, the line
Copyright
2405-8963 2018
2018, IFAC
IFAC Federation of Automatic 548 Hosting by Elsevier Ltd. All rights reserved.
Control)
Peer review©under
Copyright responsibility of International Federation of Automatic
2018 IFAC 548Control.
10.1016/j.ifacol.2018.11.573
Copyright © 2018 IFAC 548
 IFAC SYROCO 2018
Budapest, Hungary, August 27-30, 2018 Alexia Toumpa et al. / IFAC PapersOnLine 51-22 (2018) 542–547 543

following robot can accurately track the path by achieving 
   

   
 
high speeds at straight segments and lower speeds in sharp  

  
  
turns. In an experimental evaluation with a differential    

drive robot, it is shown that the proposed controller out- 
 

 
  




performs both simple PD and PID controllers as well as a     


 


related control approach from the literature.


 
  



2. FSM-BASED STATE ESTIMATION

 
   
 
In this section, a novel method is introduced that uses 


 


 
an FSM to determine the relative position of the robot 

with respect to a marked path. A major advantage of
 
 
this approach with respect to the previously proposed
methods, is that complex paths can be identified and
handled appropriately without the need of sophisticated
or customised sensor configurations. We assume that a
collinear array of N IR-reflectance intensity sensors is
placed at the front side of a differential drive robot,
perpendicularly to the line and that the distance between
individual sensors is equal or less than the line width.
Let Vi , i = 1, ..., N denote the current normalised intensity
measurement obtained by the ith sensor of the array. At
every new sensor measurement, each value Vi is discretised
into two levels based on a predefined threshold av to
determine whether the measurement corresponds to a
line or background indication, denoted as  L and  B  lineV isible =
respectively, by defining V̄ :
L , Vi ≥ a v
V̄i =   , i = 1, ..., N, (1)
B , Vi < a v
The FSM parses V̄ and updates its current state Sj , j =
1, ..., m. The total states m that are shown in the FSM
diagram of Fig. 1 can be divided into four groups; apart
from the initial state Sinit and the state where no line
is visible by the robot SnoLine , we define the accepted
and the rejected state groups. Accepted states inform
about the robot’s position with respect to the line. States
Snormal(lef t) and Snormal(right) indicate a line detected
on the left or on the right side of the sensor array
respectively, whereas Snormal(centre) implies a well-centred
robot position on the line. Furthermore, Snormal(wideL)
and Snormal(wideR) denote a wide line which is visible
by more than M sensors (where M < N is determined
based on the number of sensors which lie above the line
in an ordinary robot-path configuration), located on the
left-most or right-most side of the IR-vector accordingly.
The case of all sensors being above the line SallLine is
considered as an accepted state, inferring that a crossover
is present. However, the detection of this state can be
linked with more complex navigation actions, such as path
planning decisions or end of path. Sinverted state indicates
a background/line colour swap which is further discussed
in Sec. 2.3. In addition, if the FSM results in a rejected
state, the robot’s PD controller uses the latest accepted
measurement until the FSM results in an accepted state.
In the case where the FSM determines a SnoLine state, the
mechanism described in Sec. 2.1 is triggered to distinguish
between sharp turns, right-angles and dashed line sections.
A comprehensive matching between FSM final states and
corresponding control reactions undertaken by the robot
to stay on-track, is listed in Table 1. Other line situation
that are not listed here, can be handled accordingly with
the appropriate number of sensors and with extra states
in the FSM.
549




 

 

Fig. 1. The proposed Finite State Machine. A transition
between states occurs if the current input X = V̄i
satisfies the condition cond. |X| indicates the number
of consecutive appearances of X on V̄ up to the
examined sensor i.
2.1 Line visibility
Primarily, the FSM determines whether the line is visible
to the robot. The lineV isible state flag declares that at
least one sensor detects a line according to:
N
(Vi ≥ av ) (2)
i=1
where denotes the logical OR operation.
We can enumerate three distinct cases where the robot
does not detect the line (lineV isible = f alse):
• Robot lost the line after a right-angle (Fig. 2a)
• Robot lost the line during a curved part (Fig.2b)
• Robot lost the line on a dashed line part (Fig. 2c)
Triggering appropriate control reactions that ensure that
the robot is kept on path, requires a clear distinction of
these cases. As the line is not visible, past measurements
should be considered to specify the latest valid robot
position. Hence “memory” is introduced to the system, by
storing information about previous sensor measurements
that are recalled with every line loss to indicate under
which circumstances the robot went off-track.
When a line is not visible the PD controller is disabled
and instead, predetermined open loop control actions are
commanded to the robot until the line becomes visible
again. In the case of a right-angle (Fig. 2a), a rotation is
performed immediately after decelerating to avoid a large
overshoot from the path. Whereas, when losing the line
along a curve (Fig. 2b), a sharp turning action is executed
without necessarily decelerating to re-align the robot with
the path. Finally, in the case of a dashed-line (Fig. 2c),
the robot keeps moving on the same direction until it re-
detects the path to continue its normal navigation.
2.2 Crossovers, junctions and noisy measurements
Sensor measurements are far from ideal, since they suffer
from noise and saturation that may lead to conflicting
decisions about the robot’s position with respect to the
path. Moreover, it is usually assumed that the robot is
continuously well-aligned with the line, which is false. As
an example, the robot may confront a right-angle under a
wide variety of different poses, because of sliding, drifting,
vibration and oscillations during its movements, which
IFAC SYROCO 2018
544
Budapest, Hungary, August 27-30, 2018 Alexia Toumpa et al. / IFAC PapersOnLine 51-22 (2018) 542–547

  

 


  Table 1. Control action depending on state
Final State Control Action
Snormal PD Controller.
 


SallLine Continue current course. Can be use to indicate

track completion etc.
SnoLine Control action based on measurement history: If
right-angle, then open loop control until line is
re-detected. Otherwise, continue current course.
Sinverted Background swap and follow new line colour.
S2lines , Reject measurement and keep current course.


StooW ideLine

   The FSM parses the discretised values V̄ and searches for

specific patterns. After all N input values are parsed, the
Fig. 2. Different cases of line loss that result to the same current state of the FSM denotes its final state, which can
FSM state and need to be distinguished. be an acceptance or a rejection state, denoting different
situations about the robot’s position relative to the line.
 


 



 For example, when a single line is visible by up to M
sensors (with M < N being defined based on the line
width and sensor configuration), it can be described by
the following type of pattern that is accepted by the FSM:

   
Snormal = B ∗ L1:M B ∗ , (3)
∗
following the notation that X denotes 0 or more rep-
Fig. 3. (a,b): Robot facing right-angle with misleading etitions of X, X + denotes 1 or more repetitions of X,
sensor measurements. (c,d): Robot obtaining similar X a:b denotes a to b repetitions of X and X c denotes
sensing pattern on crossover and background change. exactly c repetitions of X. This suggests that the following
discretised measurements, representing valid robot config-
 


 





       

Fig. 4. Sample measurement history before losing the line.
may result to misleading measurements, as shown in Fig.
3a. The robot’s latest valid measurement before losing the
line is identical to the case of Fig. 2c, indicating that it was
well-aligned on track. This will drive a decision for keeping
its current course, as the previously described approach
would suggest that a dashed line is confronted. To avoid
this confusion we propose the evaluation of the past
measurements to the current state. More specifically, after
losing the line, the state of the left-most and right-most
sensors are examined within a temporal window (of size
K) of the most recent measurements. A trigger of one of
these sensors within this temporal window would assume
the existence of a corresponding right-angle, independently
of the latest sensor measurements.
In the sample robot path of Fig. 4, the robot is about to
get off-track at instance t[k] while obtaining a straight-
line measurement. However, since at instance t[k-2] the
triggering of the left-most sensor is captured, the proposed
method will imply that a right-angle is confronted on the
track. Moreover, as this measurement is more recent than
the instance that captured the triggering of the right-most
sensor moments before t[k-K], it is prioritised for defining
the direction of the right-angle. The cases of Fig. 3b,c,d are
more difficult to handle because a conflicting measurement
is captured, suggesting that two lines are visible at the
same instance. The proposed FSM can identify all such
conflicting states and discard them.
urations using N = 8 sensors, are accepted by the FSM:
{BBBLLBBB, BLLLBBBB, BBBBBBBL, LLLBBBBB} (4)
However, more irregular cases such as a detected line that
is too wide with respect to the sensors’ configuration or
the case of multiple lines, would be rejected by the FSM
 and thus ignored by the controller to avoid confusion:
{BLBLLBBB, BLLLLLBB, LBBBBBBL, LLLLLLBB} (5)
In such cases, the FSM requests a new sensor measure-
ment, while the robot keeps moving on its previously
commanded track towards the line.
Various information about the irregularity of a rejected
state can also be inferred by having multiple rejection
states. For example the pattern:
S2Lines = B ∗ L+ B + L+ B ∗ , (6)
denotes that two different lines are visible and leads to
a rejected measurement. Also, if the sensor arrangement
suggests that a line can be captured by more than M
sensors, then the existence of a wide line is identified by
the following rejection pattern:
StooW ideLine = B ∗ LM L∗ B ∗ . (7)
2.3 Line-background colour inversion
In some cases, the colours of the line and background
may be inverted. The use of a complex sensor config-
uration to detect these cases is proposed by Bajestani
and Vosoughinia (2010). However, if examined in com-
bination with other line structures, a precise definition
of the colour swap state is required to avoid confusion.
Consequently, by defining the dominant colour as the
background, false-positives will be caused in many cases,
such as a right-angle or a junction that may result in
patterns like: BLLLLLBB, BLLBLLLB. Moreover, ex-
amining whether the colour of the left-most matches the
right-most sensor and defining that as the background
550
 IFAC SYROCO 2018
Budapest, Hungary, August 27-30, 2018 Alexia Toumpa et al. / IFAC PapersOnLine 51-22 (2018) 542–547 545

colour, invalid results are obtained in cases where the robot

Sensor array
is not well-aligned with the line. An example of such case
is shown in Fig. 3c. Desired
line position
The proposed FSM is designed to distinguish between line- e v(e)
background colour inversions and any other line structures.
Our approach is to trigger a line-background colour inver- line
sion, when a previously known as background section with
the width of an accepted line, is detected between two line ω(e)
sections. This can be described by the pattern: background
Sinverted = L+ B 1:3 L+ . (8)
In the previously discussed example of Fig. 3, the pattern
of Eq. 8 will successfully recognise the colour inversion Fig. 5. Geometrical representation of the error e.
demonstrated in case (d) and will reject the irregular mea-
surement of case (c), because the width of the background Gain tuning
is too wide to form a normal line (Eq. 3) after swapping. KP , KD Velocity tuning
When colour inversion is detected, an internal flag denot- ω v
e PD Controller
ing the background colour of the track is swapped, so that
  Robot V
B and  L values are treated conversely. ω
Open loop
controller
3. VARIABLE-GAIN PD CONTROL FOR LINE FSM
FOLLOWING Sj
Sensor aggregation
In this section, we present the proposed variable-gain PD
controller for line following of differential steering robots,
which targets to minimise the position error between the Fig. 6. Block diagram of the proposed controller.
robot and the reference path. Depending on the state of the turns of the complex path. Moreover, the control gains
FSM, the PD controller is activated as shown in the block that performed best on the complex path, cause overshoots
diagram of Fig. 6. The proposed controller is not optimised in the smoother turns of the simple path, leading to
for a specific path, but instead, it is designed to effectively oscillatory motion and motor saturation that downgrades
handle paths with varying challenges and complexity. It the performance. Hence, we conclude that a single pair
can be applied on any robot with a collinear configuration of KP and KD gains cannot be considered as the best
of reflectance sensors. pair for both simple and complex paths. Additionally,
The position error e of the robot with respect to the line most paths cannot be distinguished inherently as simple or
(Fig. 5), assuming that the line is visible by at least one complex but can incorporate characteristics of both types.
sensor, is calculated by the weighted aggregation of the So, the need of online adjusting the gains according to the
sensors’ measurement, which is a simple and straightfor- complexity of the path is emerging.
ward method: In this work, we introduce a variable-gain PD controller
N
i=1 wi Vi N with the ability to online adjust the values of KP and KD ,
e=  N
− amax , (9)
2 as a function of the measured position error of the robot
i=1 Vi
e, which can be used as an indicator of the complexity of
where amax is the maximum permissible sensor value and
the path, given a constant linear velocity v. It is assumed
wi = (i − 1)amax the weight of the ith sensor. To minimise
that in simple paths, sufficient position corrections can
the position error e, a PD controller is used to modulate
be performed by a predefined KPmin and KD min
gains,
the angular velocity ω of the robot at each control cycle:
restraining the measured position error in low values.
d Hence, as the measured position error increases, a higher
ω(e) = KP (e)e + KD (e) e, (10)
dt complexity of the line following path can be inferred.
where KP (e), KD (e) ∈ R+ are the variable gains of the
The variation function for the gains KP , KD can be
proportional and derivative terms respectively. No integral
expressed in the following form:
term is used for steady state error reduction because in  λ
complex paths it reduces the robot’s tracking performance. min |e| max min
KP,D (e) = KP,D + (KP,D − KP,D ), (11)
After experimentation with trial and error, it is observed ρ
that the best gains of the PD controller for completing where ρ indicates the absolute value of the upper bound
a line following task as soon as possible without losing of the measured position error e, the exponent λ ∈ N
the line, vary depending on the path’s complexity. To determines the degree of non-linearity, and K min and
validate this observation two paths are tested: a simple K max are the corresponding lower and upper limits for
one with smooth turns of big radius and a more complex each gain. An incremental change of the exponent λ, would
one with sharper turns (illustrated in Fig. 9a). Through dictate smaller gains near K min for a large range of e
these experiments we tune the PD gains, in order to values around 0, forming a quadratic bowl. Subsequently,
minimise the robot’s completion time on each path. The the unit power (λ=1) imposes a linear correlation between
selected gains for the simple-type path, cannot exhibit the error and gain values, while a power of zero (λ=0)
sufficient reaction to keep the robot on-track on sharp inflicts constant gains and independent from the output.

551
IFAC SYROCO 2018
546
Budapest, Hungary, August 27-30, 2018 Alexia Toumpa et al. / IFAC PapersOnLine 51-22 (2018) 542–547

To further improve the robot’s performance and accom-

plish higher velocity during straight segments of the path,
we propose a method for adjusting online the robot’s linear
velocity v. To adjusts the robot’s velocity in accordance to
the observed complexity of the path, an inversely propor-
tional relation between the current error e and the linear Fig. 7. Line following robot “Dromeas”.
velocity of the robot v(e) is proposed:
v(e) = vmin + csμ , (12)
 (|e|−ē) 
1 − ρ−ē , |e| ≥ ē
s= , (13) 

1 , |e| < ē 

   


where vmin is the selected minimum robot’s linear velocity, 

ē is the error threshold under which the robot is desired 

to move with maximum linear velocity, μ ∈ N is the power 



 


that determines the degree of the increase function and

  
hence the increase rate of the linear velocity with respect
 









































































to e, and c ∈ R+ controls the range of the velocity gain. 

When the robot navigates on a simple path (e.g. on Fig. 8. (a) FSM Evaluation Path, (b) Real Sensor Mea-
a straight line) low error values are observed, leading surements, (c) FSM final state (decision).
to maximisation of the linear velocity, and hence short
completion time. Likewise, on a complex path (e.g. with
sharp turns) greater error values are perceived, applying a
dynamic velocity adjustment on the robot, which favours
the robot’s manoeuvrability.

4. EVALUATION
 



















The proposed approach is tested on the award-winning
line following robot “Dromeas” (Fig. 7). The platform has Fig. 9. Testing tracks: (a) To determine best performing
two motors for differential steering and an array of 8 IR gain values on simple (green) and complex (red)
QTR-RC reflectance sensors. The control loop operates at paths, (b) To evaluate FSM’s identification success
a period of 3ms on the on-board Cortex-M4 processor. rate on angles, (c) To evaluate and compare the
proposed approach.
4.1 FSM-based State Estimation 

In this section the effectiveness of the proposed FSM- 
 
 


based state estimator is validated by performing experi- 

ments using various challenging line configurations. These 
compositions emulate different situations measured from 

the IR sensors caused by sensor noise, bumping of the
 

robot or even junctions, crossovers and normal lines to the 




















  










 





















path. All these irregularities can easily confuse the robot’s 
line following controller if not identified promptly. Thus, 
the proposed FSM-based approach operates as a state 
 
 



estimator that identifies and handles appropriately noisy

or contradicting sensor measurements. Fig. 8 illustrates 
the designed testing path containing a variety of challenges 
(a), the actual sensors output values recorded during the    




























































































experiment (b), as well as the FSM’s resulting state (c).  

  

  


In the conducted experiments, the proposed FSM-based
Fig. 10. Sample angles as captured by the robot’s sensors.
method correctly identifies all line configurations and suc-
cessfully discards all the irregular structures of the testing all curved and obtuse angles without the FSM, though
path, allowing the robot to stay on the desired path. acute and dashed lines cannot be distinguished without
considering previous measurements in the FSM. Moreover,
To confirm the validity of the proposed approach for iden-
the correct identification of the right-angle depends on the
tification and handling towards various types of angles,
robot’s pose. Fig. 10d demonstrates an accurate measure-
numerous experiments are performed on a testing path
ment, whereas Fig. 10e,f illustrate cases of irregular series
including curved, acute, obtuse and right-angles as well as
of measurements on a right-angle, which are successfully
a dashed line part (Fig. 9b). Fine-tuned PD gain values
handled by the proposed FSM-based approach.
are exploited to efficiently handle the curved line without
losing sight of the path. Sample sensor measurements of
4.2 Variable-gain PD control evaluation and comparison
the robot confronting some of the path challenges are
illustrated in Fig. 10. The results from multiple iterations In this section, experiments are conducted with our pro-
are presented in Table 2. The robot manages to handle posed variable-gain PD controller to evaluate and compare

552
 IFAC SYROCO 2018
Budapest, Hungary, August 27-30, 2018 Alexia Toumpa et al. / IFAC PapersOnLine 51-22 (2018) 542–547 547
Table 2. Success rate of identification
Curved Obtuse Right Acute Dashed
Simple PD 100% 100% 58.4% 0% 0%
FSM & PD 100% 100% 100% 100% 100%
Table 3. Simple/Complex Experimental Path
Path Type KP ,KD Values
Simple KP =0.38, KD =1.56 (Best for simple-type path)
Simple KP =0.70, KD =4.00 (Best for complex-type path)
Complex KP =0.38, KD =1.56 (Best for simple-type path)
Complex KP =0.70, KD =4.00 (Best for complex-type path)
Table 4. Mixed-Type Experimental Path
Controller KP KI KD
P 0.2 0.0 0.0
PD (best for simple path) 0.3 0.0 1.56
PD (best for complex path) 0.7 0.0 4.0
Variable-gain PD* [0.53, 0.62] 0.0 [2.26, 2.96]
Variable-gain PD* (with acc) [0.53, 0.62] 0.0 [2.26, 2.96] 140(+48) 2.46s
Modified PID 0.35 0.024 4.16
Modified PID (with red) 0.35 0.024 4.16
∗ ρ = 350, μ = 1, λKP = 2, λKD = 1, ē = 98
its performance along with classical P and PD approaches
and line following optimised control algorithms from the
literature. To verify the proposed approach two types of
paths are being tested: a simple and a complex one (Fig.
9a). While tuning the gains in these experiments, the
target is to minimise the time to complete the track, with
the robot’s sensors never lose sight of the line.
Initially, the best pairs of KP and KD values are being
experimentally determined for each type of path. As it is
shown in Table 3, the pairs of KP and KD for one path
type performs worse in the other. Therefore, a single pair
of KP and KD values is not considered as the optimal
pair to address a path of dynamically changing type.
Consequently, the selection of the KP and KD values
should be based on the curvature of the path, thus a
dynamic adjustment of the controller’s gain values based
on the position error is required.
Further experiments have been conducted to compare the
proposed variable-gain PD method with simple P and
PD controllers as well as the modified PID approach,
optimised for line following robots, introduced in Balaji
et al. (2015). In that paper a modified integral term that
only considers the latest 10 error values in the summation
of the classical PID controller is proposed. This targets
to counteract steady state errors, while preserving the
adaptation in curvature changes throughout the path
and avoid potential instability that can be caused by
the classical, longer range, integral term. Additionally, a
velocity reduction scheme during turns, that are identified
based on the amplitude of the integral value, was proposed.
To perform this comparison, an evaluation path consist-
ing of a simple-type followed by a complex-type path is
designed (Fig. 9c). The robot’s control algorithm is not
aware of which path type is traversed at each time. Table
4 summarises the timing results for all methods in overall
and per path type. As it is demonstrated, the proposed ap-
proach outperforms all other optimised methods in terms
of completion time. Furthermore, it is demonstrated that
the proposed acceleration scheme significantly improves
the completion time of the simple-type path with negligible
effect on the timing of the hard-type path.
5. CONCLUSION
In this paper a novel FSM for real time state estimation is
introduced to identify measurement irregularities caused
by false or noisy sensor measurements. Depending on the
states of the FSM, the robot switches between a proposed
variable-gain PD controller for accurate line following and
Completion Time a predetermined open loop controller that handles special
3.16s
3.46s cases such as when the robot sensors have lost the line.
2.76s As proved in the experiments this approach improves the
2.42s
performance of the robot by identifying and discarding
confusing sensor inputs, that could cause deviation from
the desired path. The proposed controller, outperforms
classical P and PD approaches as well as other line
v(±c) Simple Complex Overall
140 4.43s 8.02s 12.46s following control methods from the literature.
140 2.89s 5.86s 8.76s
140 3.09s 5.48s 8.57s
140 2.86s 5.24s 8.12s ACKNOWLEDGMENT
5.30s 7.77s
140 2.98 5.33s 8.33s
188(-48) 2.51s 5.36s 7.87s The authors would like to thank all members of the
Robotics Club of University of Patras for their valuable
contribution on the development of the proposed methods.
REFERENCES
Baharuddin, M.Z., Abidin, I.Z., and Mohideen, S.S.K.
(2006). Analysis of Line Sensor Configuration for the
Advanced Line Follower Robot. In Proc. of Student
Conference on Research and Development (SCOReD).
Bajestani, S.E.M. and Vosoughinia, A. (2010). Technical
report of building a line follower robot. In Electronics
and Information Engineering (ICEIE), 2010 Interna-
tional Conference On, volume 1, V1–1–V1–5. IEEE.
Balaji, V., Balaji, M., Chandrasekaran, M., Elamvazuthi,
I., et al. (2015). Optimization of pid control for high
speed line tracking robots. Procedia Computer Science,
76, 147–154.
Borcosi, I., Popescu, L., and Nebunu, D. (2006). Line
Tracking Robot. In Proc. of the 15th International
Conference on Manufacturing Systems ICMaS.
Colak, I. and Yildirim, D. (2009). Evolving a line following
robot to use in shopping centers for entertainment.
In 2009 35th Annual Conference of IEEE Industrial
Electronics, 3803–3807.
Ebiesuwa, O.O., Adekunle, Y.A., Akinyemi, L.A., and
Oyerinde, O.D. (2013). Line Follower Robot Using A
Sophisticated Sensor Approach. International Journal
of Engineering, 2(7), 1980–1982.
Horan, B., Najdovski, Z., Black, T., Nahavandi, S., and
Crothers, P. (2011). Oztug mobile robot for manufactur-
ing transportation. In Systems, Man, and Cybernetics
(SMC), 2011 IEEE International Conference on, 3554–
3560. IEEE.
Matczak, G. and Mazurek, P. (2015). History dependent
viterbi algorithm for navigation purposes of line follow-
ing robot. Image Processing & Communications, 20(4),
5–11.
Osorio, R., Romero, J.A., Peña, M., and López-Juárez,
I. (2006). Inteligent line follower mini-robot system.
International Journal of Computers, Communications
& Control, 1(2), 73–83.
Punetha, D., Kumar, N., and Mehta, V. (2013). Devel-
opment and applications of line following robot based
health care management system. International Jour-
nal of Advanced Research in Computer Engineering &
Technology (IJARCET), 2(8), 2446–2450.
553
 Available online at www.sciencedirect.com

ScienceDirect
IFAC PapersOnLine 53-2 (2020) 10162–10167

On
On the
the Influence
Influence of
of the
the Moment
Moment of of Inertia
Inertia on
on the
the Mechatronic
Mechatronic Drive
Drive Control
Control
Quality within the Exoskeleton
On the Influence of theQuality
Momentwithin the Exoskeleton
of Inertia on the Mechatronic Drive Control
QualityM.M.*.
Stebulyanin within the Exoskeleton
Ermolov I.L.**, Sukhanov A.N.**
Stebulyanin M.M.*. Ermolov I.L.**, Sukhanov A.N.** 

* Stebulyanin
Moscow StateM.M.*. Technological
Ermolov University STANKIN,A.N.**
I.L.**, Sukhanov 127055,
* Moscow State Technological
Vadkovskiy per. 3a, Moscow, Russia University
(e-mail:STANKIN, 127055,
mmsteb@rambler.ru).

Vadkovskiy
** Laboratoryper. 3a, Moscow,
of Technological Russia (e-mail:
Robotics and Mechatronics, mmsteb@rambler.ru).
Ishlinsky127055,
Institute
* Moscow State University STANKIN,
**
for Laboratory
Problems of Robotics and RAS,Mechatronics, Ishlinsky Institute
Vadkovskiy per. in3a,Mechanics
Moscow, Russia 119526
(e-mail: Prospect Vernadskogo
mmsteb@rambler.ru).
for Problems
101-1, Moscow, in Mechanics RAS, 119526 Prospect Vernadskogo
** Laboratory of Russia
Robotics (e-mail: sukhanov-artyom@yandex.ru)
and Mechatronics, Ishlinsky Institute
101-1, Moscow, Russia (e-mail: sukhanov-artyom@yandex.ru)
for Problems in Mechanics RAS, 119526 Prospect Vernadskogo
101-1, Moscow, Russia (e-mail: sukhanov-artyom@yandex.ru)
Abstract: The paper considers the interaction between human arm and electromechanical system of the
Abstract: The paper
exoskeleton device. Authors considers the interaction
proposed between human
the mathematical model arm and electromechanical
of human-exoskeleton system on
interaction of the
the
exoskeleton
example ofThe device.
an paper Authors
EMGconsiders
controlledproposed the mathematical model of human-exoskeleton interaction on the
Abstract: themechatronic
interaction betweenDC-motor. humanFeatures
arm and of electromechanical
the torque generation system withof the
example
mechatronic of an EMG
drive controlled
ofAuthors
the exoskeleton mechatronic
under DC-motor.
the conditions Features
of of of the torque of
desynchronization generation
the speeds withof the
the
exoskeleton device. proposed the mathematical model human-exoskeleton interaction on
mechatronic
operator's arm drive
and ofthetheexoskeleton
exoskeletonlink underare the conditions
shown. The of desynchronization
exoskeleton motion of the speeds
simulation results ofwith
the
example of an EMG controlled mechatronic DC-motor. Features of the torque generation with the
operator's arm andofthe
different moments exoskeleton linksystem
are shown. The exoskeleton motion simulation results with
mechatronic drive ofinertia of the drive
the exoskeleton under the areconditions
presented. of desynchronization of the speeds of the
different moments of inertia of the drive system are presented.
Copyright ©
operator's
Keywords:
arm 2020 The
Human
and the
operatorAuthors.
exoskeleton
support, is an open
ThisDesign,
link are access article
shown.
Modelling
The under the
exoskeleton
and analysis of CC
motion
HMS, BY-NC-ND
simulation license
results with
Assistive technology and
(http://creativecommons.org/
different
Keywords: moments
Human of inertia of
operator licenses/by-nc-nd/4.0)
the drive
support, system
Design, are presented.
Modelling and
rehabilitation engineering, Telepresence, Telerobotics, Mechatronics, Motion Control Systems,analysis of HMS, Assistive technology and
rehabilitation
Dynamics engineering, Telepresence, Telerobotics, Mechatronics, Motion Control Systems,
Keywords:and Humancontrol.
operator support, Design, Modelling and analysis of HMS, Assistive technology and
Dynamics and control.
rehabilitation engineering, Telepresence, Telerobotics, Mechatronics, Motion Control Systems,

Dynamics and control. 
some cases of emergency the operator may not disconnect
1. INTRODUCTION some cases of emergency the operator which may not
1. INTRODUCTION  from the control system immediately, maydisconnect
result in
from the
injury.cases control
In addition, system immediately,
during operation which
the operator may result in
One of the practical purposes of the researches in the field of some of emergency the operator may notshould sense
disconnect
One of the practical 1. INTRODUCTION
purposes of the researches in the field of injury.
external In forces
addition, during
acting on operation
the the operator
exoskeleton should
links, thussense
the
wearable robotics is the development of different exoskeleton from external
the control system immediately, which may result in
forces actingofon the exoskeleton links, thus the
wearable robotics is thelarge
development of different
objects,inexoskeleton mechanical structure duringthe exoskeleton must be equipped
systems
One of the for moving
practical purposesand of theheavyresearches taking of injury.
the fieldinto mechanical
In addition,
structure of
operation
the exoskeleton
the operator
must
should
be
sense
equipped
systems
account the for moving
imposed large and
requirements heavy objects, taking into
accuracy. with externalreversible gears. on
forces acting Self-braking
the exoskeleton transmissions
links, thus in the
wearable robotics is the developmentforof positioning
different exoskeleton with reversible gears. are Self-braking
account
Complex, the imposed requirements
systems formonotonous
moving large actions
and heavy
for require,
that positioning
objects, intaking
accuracy.
into mechanical
addition, structure
structure not abletransmissions
of the exoskeleton tomustgive in the
adequate
be equipped
Complex, monotonous actions that require, in addition, mechanical
information structure are not able directly
to give adequate
accuracy
account the in imposed
various technological
requirements operations
for positioning can be carried with
accuracy. reversible gears. Self-braking transmissions in the
about the load torque through
accuracy in various technological operations can be carried information
exoskeleton’s about
link tothetheare load torque
operator. directlyanthrough the
out by the monotonous
Complex, active exoskeleton actions device.
that At the same
require, in time, the mechanical
addition, structure not The able usetoof give exoskeleton
adequate
out by the active exoskeleton device. At the same time, the exoskeleton’s
with mechanical link to the operator. The use of an exoskeleton
operator’s
accuracy in various participation
technological in operations
decision-making can be carried when information about decoupling
the load torque from directly
the operatorthrough willthe
operator’s
manipulating participation
a heavy object in
allows decision-making
to realize the when with
flexibility complicatemechanical
exoskeleton’s the design
link to
decoupling
in
the technical
operator.
from
terms.
The use
the operator will
of an exoskeleton
out by the active exoskeleton device. At the same time, the complicate the design in technical terms.
manipulating
of technological a heavy object while
processes, allowsincreasing
to realize the the efficiency
flexibility with mechanical decoupling from the operator will
operator’s participation in decision-making when In our work we study the control system that receives the task
of technological processes, while increasing the efficiency
and productivity
manipulating a heavy object allows to realize the flexibility complicate
of operation. In
forour
thework
the
drivesweofdesign
study in technical
the control system
the exoskeleton
terms. that receives the task
from the muscular activity
and productivity of operation.
of technological processes, while increasing the efficiency for athehuman
drives of theLuca, exoskeleton from
Thisthe muscular activity
The specificity of the exoskeleton usage in the conditions of of In our work we[De
of a human
study the control
[De
(2002)].
Luca, (2002)].
system type
that
This type of
of control
receives the task
control is
is
and productivity
The specificity ofofthe
operation.
exoskeleton usage in the conditions of intuitive for operator when performing manipulative
bilateral influence of the exoskeleton’s links and operator’s intuitive for the drives of the exoskeleton from the muscular activity
bilateral influence of thespecial
exoskeleton’s links and for operator objects.when performing manipulative
limbs causes
The specificity setting
of the exoskeleton requirements
usage in the conditionsforoperator’s of operations
control, of a human with[Dedifferent
Luca, (2002)]. This type of control is
operations with different objects.
limbs
accuracy causes setting
of operations special requirements for control,
The intuitive for operator when performing manipulative
bilateral influence of theand safety of the
exoskeleton’s linksmovement.
and operator’s
accuracy
movements ofofoperations
the and safety
exoskeleton links of the ensure
should movement. the The The
safety
problems of human interaction with a robotic system that
operations withofdifferent objects. with a robotic system that
limbs causes setting special requirements for control, The problems
repeats human interaction
the biomechanics of the operator’s limbs require
movements
for of
the operator’s the exoskeleton
hand and and safetylinks
at the of should
same ensure the safety
accuracy of operations the time
movement.the controlThe repeats
careful the biomechanics
study due of theofoperator’s
to the interaction
presence a number limbs
of system require
restrictions
for the
system operator’s
should hand
implement and at
the desiredthe same
movement time the control The problems of human with a robotic that
movements of the exoskeleton links should ensureinthe response
safety careful study
on kinematics, due to the
control and ofpresence
modes of a number
of operation. of restrictions
One require
of the
system
to the should implement
disturbing effect, the desiredasmovement
represented a function in
of response
the state repeats the biomechanics the operator’s limbs
for the operator’s hand and at the same time the control on kinematics,
important aspectscontrol and modes of operation. Onehuman-
of the
to
of the
the disturbing
human effect,activity.
represented This as aespecially
function true of theforstate
the careful study due tothat
the affect
presence the ofdynamics
a numberofofthe restrictions
system shouldmusclesimplement the desired ismovement in response important
machine aspects
system that affect the dynamics
"operator-exoskeleton" is takingof into
the account
human-
of the human of
development muscles
loading activity. This isdevices
exoskeleton especially true forinthea on kinematics, control and modes of operation. One of the
operating
to the disturbing effect, represented as a function of the state both machinethe system
inertia "operator-exoskeleton"
of the is taking into account
development
wide of loading exoskeleton devices operating byinthea both
important aspects thatlinks
affect andtheactuators
dynamics of the exoskeleton
of the human-
of therange
human ofmuscles
forces. Suchactivity.systems
This is areespecially
characterized true for the inertia of the links and actuators of the exoskeleton
wide rangeofof forces.
presence external Such systems arethat
disturbances characterized
exceed by the itself,
human machineand systemthe"operator-exoskeleton"
inertia of the operator's is takinglimbs when
into account
development of loading exoskeleton devices operating in a manipulating itself, and the inertia of the operator's limbs when
theofobject.
presence
capabilities of external disturbances that exceed human both the inertia the links and actuators of the exoskeleton
wide range of[Kazerooni,
forces. Such(1993)]. systems are When developing
characterized an manipulating the object.
by the
capabilities applicable
exoskeleton [Kazerooni, loading (1993)].operations,
When developing an itself, and the inertia of the operator's limbs when
presence of external for disturbances that exceedit is necessary
human In our previous works [Gradetsky et al (2017,2018a,b)] the
exoskeleton applicable for loading operations, it is necessary manipulating the object.[Gradetsky et al (2017,2018a,b)] the
to follow the restrictions
capabilities [Kazerooni,on (1993)].the control. When developing an In our previous
simplified systemworks
"operator-exoskeleton" has been discussed
to follow the restrictions on the control. simplified system "operator-exoskeleton" has been discussed
exoskeleton applicable for loading operations, it is necessary In andouranalysed. There has been madeetanalassumption about the
Unlike the traditional control technique used in the bilateral and analysed. There has[Gradetsky previous works
been between
(2017,2018a,b)]
made an assumption about the
to follow
Unlike the the restrictions
traditional on
control the control.
technique used in the bilateral rigid connection
simplified system presented
"operator-exoskeleton" thehas master
been system,
discussed
control systems, which involves control panel device with a rigid connection presented between the master system,
control systems, system,
which involves control panelactivity
device with represented byThere
the operator's
has beenarm, andanthe slave system of the
force reflection the use of electrical of thea and analysed.
represented byrepresented
the operator's
made assumption about
Unlike
force
the traditional
reflection system,
control
the
technique
use of
used in
electrical
the bilateral
activity of the exoskeleton,
rigid connection presented by arm,
the and the slavedrive
mechatronic
between the
system
master
of the
and the
system,
muscles allows towhich
control systems, makeinvolves
the system control"operator-exoskeleton",
panel device with a exoskeleton, represented by the mechatronic drive and the
muscles allows to make the system "operator-exoskeleton", link equipped bywith handle. The analytical description
systemofofsuch
in which
force the operator
reflection system,is thedirectly
use of involved
electrical in control.
activity But in represented
of the
the operator's arm, and the slave
link equipped with handle. The analytical description of such
the
in which the operator is directly involved in control. But in exoskeleton, represented by the mechatronic drive and the
muscles allows to make the system "operator-exoskeleton",
link equipped with handle. The analytical description of such
in which the operator is directly involved in control. But in
2405-8963 Copyright © 2020 The Authors. This is an open access article under the CC BY-NC-ND license.
Peer review under responsibility of International Federation of Automatic Control.
10.1016/j.ifacol.2020.12.2743
 Stebulyanin M.M. et al. / IFAC PapersOnLine 53-2 (2020) 10162–10167 10163

system with a gap between the operator's arm and the handle, tries to form the desired movement of the object placed in the
which is referred to the technical implementation of such a hand, while forming an effort on the muscle Fdes (t ) , based on
human-machine system, has been also proposed. This article the magnitude of the misalignment between the desired angle
discusses the features of the interaction of elements in such a
 des (t ) in the elbow and the current angle  real (t ) reproduced
system, namely, the influence of the mechatronic drive
response on the change in the control signal, which in turn is by the mechatronic drive in the elbow joint of the
a reaction to the changed state of the system. exoskeleton.
Fdes (t )  K f ( des (t )   real (t )) (3)
2. MATHEMATICAL MODEL
Here K f is the proportional factor (reaction of the operator).
The main idea of this study was to organize the work of two
different subsystems taking into account their influence on Each muscle, as an element of a biomechanical system, has a
each other. In this paper we consider the exoskeleton device number of current parameters: the current non-zero length
controlled via electrical activity of the operator's muscles. l a (t ) , the current velocity of contraction Va (t ) and the current
The mathematical model of the muscle, which is used in this
developed force of contraction. In addition to the current
work, was presented earlier by Alonso et al, (2012) and
parameters, the muscle has its own boundary conditions.
Thelen (2003). The desired force that a muscle creates Fdes (t )
They include the maximum length l max , the minimum length
is expressed as follows [Zajac et al (1989a,b)]:
l min when contracting, the optimal length
l O [Silva (2003,),
Fdes (t )  a(t ) Fmax f FV (Va ) f Fl (l a ) (1)
Latash (2011)], and the maximum possible contraction force
Fmax (l a (t ), Va (t )) that a muscle can create for the current
Here f FV (Va ) represents the functional dependence of the
length and contraction rate.
contraction force from the velocity of muscle contraction,
f Fl (la )
is the dependence of contraction force from the current
length of the muscle [Romero et al (2016), Sancho-Bru et al
(2011)]. The level of activation a(t ) is a value determined by
the ratio of the contraction effort Fdes (t ) that is necessary to
create via muscle, to its maximum possible effort, which
could be developed at the current parameters of the length
l a (t ) and contraction velocity Va (t ) . Based on equation (1),
the level of muscle activation a(t ) is determined as follows:

Fdes (t ) (2)
a (t ) 
Fmax f FV (Va ) f Fl (l a )

In our simulation the level of muscle activation is the

complex variable depends on all these parameters
[Komantsev et al (2001)].
Each skeletal muscle has motor units. Each motor unit is
represented with externally excited cell surrounded by a
solution of potassium and sodium ions. The motor units are
connected to each other sequentially in threads, and the
threads themselves are collected in bundles [Zatsiorsky et al
(2012)]. A nerve impulse generated by the human brain goes
to the bundles of motor units to stimulate them. In response
to stimulation, an excitation wave passes through the
membranes of the cells of the motor units, changing the
electrical potential of the cell membranes.
The value of the activation level a(t )  [0..1] is formed by the
central nervous system of the operator on the basis of the
planned action and the current tactile and visual information
from the feedback channels. It shows the intensity of the
muscle contraction, that is, the proportion of all involved
motor units per time.
Fig. 1. General control scheme of interaction in ―operator-
Thus, the operator in the mathematical model can be
exoskeleton‖ system
represented as a task scheduler for moving the desired limb,
controlling this movement via the result of the work of the
drive system of the exoskeleton. In other words, the operator Figure 1 shows the general control scheme that represents
interaction within operator-exoskeleton system. The angle
10164 Stebulyanin M.M. et al. / IFAC PapersOnLine 53-2 (2020) 10162–10167
and inertia of the operator are simulated, and these are input
into the task scheduler. In this scheme, there is the
mathematical model of a muscle duplex, an interface block
and a model of a DC motor with current feedback loop. In
this scheme feedback is necessary for the correct operation of
the entire system. Visual position feedback allows the
operator adjusting the task to form an effort with his muscles
working in pair. Velocity feedback allows calculating the
necessary parameters in the muscle model. Also in this
scheme there is an external moment, constantly acting on the
exoskeleton. It represents the influence of the object.
The scheme also includes a database that contains numerical
values of maximum biceps brachii and triceps brachii forces
obtained from isotonic and isometric measurements. Figure 2
shows a schematic representation of the muscle duplex that
contains mathematical models of antagonist muscles, biceps
brachii and triceps brachii.
Fig. 2. Muscle duplex model (here the Deadzone is 0)
The formed value of the desired effort should be processed
via two channels – the technical channel and the biological
one (Fig.2). The technical data channel is necessary to get the
activation level obtained by the electromyogram (EMG)
sensors, which are given in the interface block. The
advantage of applying electromyography to control is
concluded in the fact that the obtained signal can be
processed before contraction of the muscle fiber completes.
The biological information channel contains the description
of the of muscle fibers’ contraction processes [DiCicco
(2004), Zaichenko (2001)]. These processes allow moving
the upper limb of the operator, forming a torque in the elbow
joint of the arm. This scheme takes into account the switching
of the muscles activity to simulate oscillating movements.
Fig. 3. Biceps brachii model
Figure 3 shows a mathematical model of the operator's biceps
brachii. This model describes the dependence of the
contraction force from the velocity of the muscle
contraction f FV (Va ) , as well as the contraction force from the
current muscle length f Fl (la ) . In this model the dependence of
the contraction force from the current muscle length is
determined as follows [Alonso (2012)]:
 la
 0, if l  1
 o
(4)
f Fl (l a )    la 1
2
  lo  l
e , if a  1
 lo
Here la is the current muscle length, calculated from the
geometry of the arm, l o is the optimal muscle length (length
at which the muscle can create maximum force).
The force-velocity factor f FV (V a ) depends on the current
contraction velocity of the muscle and is defined as [Haeufle
(2010)]:
 Stebulyanin M.M. et al. / IFAC PapersOnLine 53-2 (2020) 10162–10167 10165

 V structure formed by comparing the autonomous speeds of the

 0, if a c  1 operator's arm and the exoskeleton link (Fig.5).
 lo
 l o k CE1  Va c k CE1 V (5)
f FV (Va )   , if  1  a c  0
 k CE1l o  Va c lo
 l o k CE 2  f Vmax Va c Va c
 , if 0
 k CE 2 l o  Va c lo

Here  c is the reaction time of the muscle fibres,  , k CE1 , k CE 2

and f Vmax are factors, which calculated empirically. The
following numerical values of these factors obtained in
[Thelen (2003)] were used in the simulation process:

 c  0.1,   0.45, k CE1  0.25, k CE 2  0.06, fVmax  1.6

The model also takes into account the dynamic processes

occurring in the muscle body, namely the reaction of
synapses to the control signal coming from the central
nervous system, as well as the reaction of calcium
metabolism occurring in the membranes of muscle fiber cells,
which leads to the ion saturation of the space between actin
and myosin proteins and the subsequent contraction of motor
units. The triceps brachii model looks similarly. It has its own
length settings and coefficients [Pinchuk (2002)].
The current biopotential data is processed in the interface
block (Fig.4), which is represented by inertial first order
transfer functions, modeling the processes occurring in the
processing of biopotential data with EMG sensors. This unit
allows obtaining the triceps brachii and biceps brachii
contraction forces via the current activation level and
information taken from the database. It calculates torque in
the operator's elbow joint and uses it as task for the control
system.

Fig. 4. The interface block

This interface block simulates EMG processing unit as a
regulator that issues a control signal to the input of the
exoskeleton drive. It has configurable parameters of
integration time and ratio factor.
The formed movement velocities of the operator's arm and
the exoskeleton links may differ, so there are situations when
the operator's arm can either help to move the link of the
exoskeleton or interfere with it. This situation was considered
when simulating a DC-motor as part of the elbow joint of the
exoskeleton device in the form of a variable inertia block
Fig. 5. DC-motor with current feedback loop
The main feedback in the system uses the motor current.
Back EMF is calculated at the output of the total inertia unit.
The combination of control inputs is carried out using torques
of operator’s arm and external torque from the object.
3. SIMULATION
The simulation of the system was carried out in the
MATLAB Simulink software. Figure 6 shows the results of
10166 Stebulyanin M.M. et al. / IFAC PapersOnLine 53-2 (2020) 10162–10167

simulation of the torque engine during the movement of the
exoskeleton link in conditions of desynchronization with the
movement of the operator's hand. The desynchronization was
achieved by different model response times in the system (see
fig.3 and 4).
Thus, with a decrease in the moment of inertia of the drive
compared to the moment of inertia of the human arm, there is
a tendency of reducing the overshoot and reducing the
transition time, while the increasing in the moment of inertia
of the exoskeleton drive compared to the moment of inertia
of the human arm reduces the characteristics of the control
quality.

Fig. 6. Torques in the simulation (motor inertia here is

0.033kg*m2, arm inertia is 0.1 kg*m2)
Figure 6 shows a simulation of the process of moving a load
attached to the handle of the exoskeleton. At the initial time
moment, the torque value of the motor is maximum. Due to
the presence of inertia, the link still continues to move after
reaching the set value of the angle at the elbow (30 degrees).
Seeing this, the operator strains the triceps brachii to make
the motor run in reverse (switching). Thus, in a couple of
iterations, the system enters the steady operation mode,
which is expressed in holding a load of a certain mass.
Figure 7 shows the effect of influence of the moment of
inertia of the exoskeleton drive on the quality of control with
fixed moment of inertia of the operator's arm.

Fig. 8. Torques in the simulation with variation of motor

inertia (arm inertia is still 0.1 kg*m2)
As one can see from the figures 7 and 8, the lighter link of the
exoskeleton allows reaching the desired angle faster. While
the heavier link of the exoskeleton requires more time for this
operation.
This behaviour in the simulation can be explained by the fact
that a more inertial drive requires more time for acceleration
and braking, which in turn can reduce the safety of the
exoskeleton operating. This simulation model makes it
possible to estimate the contribution of the moment of inertia
Fig. 7. Transient processes in simulation with inertia
and other parameters in the model to overshoot during the
variation
movement of the exoskeleton link. Restrictions imposed by
 Stebulyanin M.M. et al. / IFAC PapersOnLine 53-2 (2020) 10162–10167 10167
the physiology of human joints should set limits for the
movement of exoskeleton links under different types of
control (fast or slow)
6. CONCLUSIONS
In this paper, a mathematical model of the electromechanical
drive of the exoskeleton with biofeedback was proposed. It
represents the possibility of desynchronization of the
operator's arm movement and the movement of EMG
controlled exoskeleton. The simulation showed that there
may occur such situations in which the operator can either
help to move the exoskeleton link, reducing the resistance
torque, or to interfere with the movement, creating an
additional disturbing torque. The dependence between the
moment of inertia of the mechatronic drive of the exoskeleton
and the quality of its control was also revealed.
ACKNOWLEDGMENTS
This research was supported by Basic Research Program
I.7 "New Developments in Perspective Areas of Energetics,
Mechanics and Robotics" of the Presidium of Russian
Academy of Sciences and also is supported by the Ministry
of Science and Higher Education within the framework of the
Russian State Assignment under contract No. АААА-А17-
117021310384-9
REFERENCES
Alonso, J., Romero, F., Pàmies-Vilà, R., Lugrís, U., and
Font-Llagunes, J.: A simple approach to estimate muscle
forces and orthosis actuation in powered assisted
walking of spinal cordinjured subjects, Multibody Syst.
Dyn., 28, 109–124, 2012.
De Luca C. J. Surface Electromyography: Detection and
Recording / Carlo J. De Luca, 2002 by DelSys
Incorporated, 2002. - P.10.
DiCicco, M. Comparison of Control Strategies for an EMG
Controlled Orthotic Exoskeleton for the Hand / Matthew
DiCicco, Lenny Lucas, Yoky Matsuoka // Proceedings
of the 2004 IEEE International Conference on Robotics
and Automation, New Orleans, LA, April 2004. – pp.
1622-1627.
Gradetsky, V.G., Ermolov I.L., Knyazkov M.M., and
Sukhanov A.N.. Generalized approach to bilateral
control for EMG driven exoskeleton. In Conf.Volume
113, 2017, 12th International Scientific-Technical
Conference on Electromechanics and Robotics, volume
113 of Zavalishin’s Readings, pages 1–5, 2017.
Gradetsky, V.G., Ermolov I.L., Knyazkov M.M., Semenov,
E.A., and Sukhanov A.N. The dynamic model of
operator-exoskeleton interaction. In Interactive
Collaborative Robotics, proc. of the 3rd International
Conference ICR 2018, pages 52–59. Springer Nature
Switzerland AG 2018 Printforce, the Netherlands, 2018.
Gradetsky, V.G., Ermolov I.L., Knyazkov M.M., Semenov,
E.A., and Sukhanov A.N. The influence of various
factors on the control of the exoskeleton. In Interactive
Collaborative Robotics, proc. of the 3rd International
Conference ICR 2018, pages 60–70. Springer Nature
Switzerland AG 2018 Printforce, the Netherlands, 2018.
Haeufle, D. F. B., S. Grimmer, A. Seyfarth a 2010 The role
of intrinsic muscle properties for stable hopping—
stability is achieved by the force–velocity relation.
Bioinspir. Biomim. 5, 016004. doi:10.1088/1748-
3182/5/1/016004
Kazerooni, H. Human extenders / H. Kazerooni, J. Guo //
ASME J. Dyn. Syst. Meas. Contr. 115(2B), 1993. - pp.
281–289.
Komantsev V.N., Zabolotnykh V.A. Metodicheskie osnovy
klinicheskoi elektroneiromiografii: ruk. dlia vrachei
[Methodological basis of clinical
electroneuromyography: guide for physicians]. SPb.,
Lan', 2001, 349 p. (In Russian)
Latash, M. L. (2011). Anticipatory control of Voluntary
action - Merging the ideas of equilibrium-point control
and synergic control. in Motor Control - Theories,
experiments and applications, pp. 3–26.
Pinchuk D.Iu., Dudin M.G. Biologicheskaia obratnaia sviaz'
po elektromiogramme v nevrologii i ortopedii:
spravochnoe ruk. [Biological feedback by electrical
myogram in neurology and orthopaedics: reference
manual]. SPb., Chelovek, 2002, 120 p. (In Russian)
Romero, F., & Alonso, F. J. (2016). A comparison among
different Hill-type contraction dynamics formulations for
muscle force estimation. Mechanical Sciences, 7(1), 19–
29. https://doi.org/10.5194/ms-7-19-2016
Sancho-Bru, J.L., Antonio Pérez-González, Mora, M.C.,
León, B.E., Vergara, M., Iserte, J.L., Rodríguez-
Cervantes, P.J., Morales, A.: Towards a realistic and
self-contained biomechanical model of the hand.
Theoretical biomechanics pp. 212–240 (2011),
http://www.intechopen.com/articles/show/title/towards-
a-realistic-and-self-contained-biomechanical-model-of-
the-hand
Silva, M.: Human motion analysis using multibody dynamics
and optimisation tools, PhD thesis, Instituto Superior
Técnico, Unversidade Técnica de Lisboa, Lisbon,
Portugal, 2003.
Thelen, D.: Adjustment of muscle mechanics model
parameters to simulate dynamic contractions in older
adults, J. Biomech. Eng.-T. ASME, 125, 70–77, 2003.
Zaichenko K.V., Zharinov O.O., Kulin A.N., Kulygina L.A.,
Orlov A.P.; Ed. Zaichenko K.V. S"em i obrabotka
bioelektricheskikh signalov: ucheb. posobie [The
removal and processing of bioelectric signals: manual].
SPb., 2001, 95 p. (In Russian)
Zajac, F. E. and Gordon, M. E., "Determining muscle's force
and action in multi-articular movement," Exerc Sport Sci
Rev, vol. 17, pp. 187-230, 1989.
Zajac, F. E (1989) Muscle and tendon: properties, models,
scaling, and application to biomechanics and motor
control. CRC Critical Reviews in Biomedical
Engineering 17, 359-411.
Zatsiorsky, V. and Prilutsky, B.: Biomechanics of skeletal
muscles, Human Kinetics, Human Kinetics Publisher,
Champaign, USA, 2012.
 Available online at www.sciencedirect.com
ScienceDirect
ScienceDirect
Procedia Computer Science 00 (2019) 000–000
Available online at www.sciencedirect.com
Procedia Computer Science 00 (2019) 000–000 www.elsevier.com/locate/procedia
www.elsevier.com/locate/procedia
ScienceDirect
Procedia Computer Science 169 (2020) 662–667

Postproceedings of the 10th Annual International Conference on Biologically Inspired Cognitive

Postproceedings of the 10th
Architectures, Annual
BICA 2019International Conference
(Tenth Annual Meetingonof Biologically Inspired Cognitive
the BICA Society)
Architectures, BICA 2019 (Tenth Annual Meeting of the BICA Society)
The algorithm development for operation of a computer vision
The algorithm development for operation of a computer vision
system via the OpenCV library
system via the OpenCV library
Stepan Sivkov*, Leonid Novikov, Galina Romanova, Anastasia Romanova,
Stepan Sivkov*, Leonid Novikov, Galina Romanova, Anastasia Romanova,
Denis Vaganov, Marat Valitov and Sergey Vasiliev
Denis Vaganov, Marat Valitov and Sergey Vasiliev
Technological Institute National Research Nuclear University MEPhI,
Lesnoy,
Technological Sverdlovsk
Institute Region,
National the Russian
Research NuclearFederation
University MEPhI,
Lesnoy, Sverdlovsk Region, the Russian Federation

Abstract
Abstract
The article is devoted to the effective way of getting images from the information board of the counting register for utility metering.
This method
The article is based to
is devoted onthe
theeffective
use of thewayopen sourceimages
of getting library from
OpenCV. The imageboard
the information processing algorithmregister
of the counting has beenforrepresented. The
utility metering.
method
This of calibration
method of the
is based on the use
video
of camera
the openallows
sourceuslibrary
to reduceerror
OpenCV.probability
The image in text recognition.
processing Here
algorithm has are
beensome results ofThe
represented. the
experiment.
method of calibration of the video camera allows us to reduceerror probability in text recognition. Here are some results of the
experiment.
© 2019The Authors. Published by Elsevier B.V.
©
© 2020
This
2019TheThe
is an Authors.
open Published
accessPublished
Authors. by Elsevier
article under
by Elsevier B.V.
the CC BY-NC-ND
B.V. license(https://creativecommons.org/licenses/by-nc-nd/4.0/)
This is an open access article under the CC BY-NC-ND license (http://creativecommons.org/licenses/by-nc-nd/4.0/)
Peer-review
This is an under
open responsibility
access article of the
under the CC
scientific committee
BY-NC-ND of the 9th Annual International Conference on Biologically Inspired
license(https://creativecommons.org/licenses/by-nc-nd/4.0/)
Peer-review under responsibility of the scientific committee of the 10th Annual International Conference on Biologically Inspired
Cognitive
Peer-review
Cognitive Architectures.
under responsibility of the scientific committee of the 9th Annual International Conference on Biologically Inspired
Architectures.
Cognitive Architectures.
Keywords:computer vision; computer vision system; calibration of video camera; counting registers for utility metering.
Keywords:computer vision; computer vision system; calibration of video camera; counting registers for utility metering.

1. The algorithm development for operation of a computer vision system via the OpenCV library
1. The algorithm development for operation of a computer vision system via the OpenCV library
The use of modern automated remote data collection and transmission systems from energy metering devices allow
The use of modern automated remote data collection and transmission systems from energy metering devices allow
us to:
us to:
• solve the problem of access to the object
• solve
reducethe
theproblem of access
laboriousness to theaccounting
of data object and processing
reduce the
• monitor thelaboriousness of data accounting and processing
accuracy control
monitor the
• improve the accuracy
reliabilitycontrol
of the data and significantly speed up its receipt.
• improve the reliability of the data and significantly speed up its receipt.

1877-0509© 2019 The Authors. Published by Elsevier B.V.

This is an open
1877-0509© access
2019 article under
The Authors. the CCby
Published BY-NC-ND license (https://creativecommons.org/licenses/by-nc-nd/4.0/)
Elsevier B.V.
Peer-review under
This is an open responsibility
access of the
article under thescientific committee
CC BY-NC-ND of the(https://creativecommons.org/licenses/by-nc-nd/4.0/)
license 9th Annual International Conference on Biologically Inspired Cognitive
Architectures.
Peer-review under responsibility of the scientific committee of the 9th Annual International Conference on Biologically Inspired Cognitive
Architectures.

1877-0509 © 2020 The Authors. Published by Elsevier B.V.

This is an open access article under the CC BY-NC-ND license (http://creativecommons.org/licenses/by-nc-nd/4.0/)
Peer-review under responsibility of the scientific committee of the 10th Annual International Conference on Biologically Inspired
Cognitive Architectures.
10.1016/j.procs.2020.02.193
 Stepan Sivkov et al. / Procedia Computer Science 169 (2020) 662–667 663
2 Author name / Procedia Computer Science 00 (2019) 000–000

The use of automated contactless accounting systems satisfies the requirements of cost effectiveness indicators.
One of the most promising ways to obtain information is using computer vision. The solution of computer-based
identification problems is based on the creation of artificial intelligence systems that process an image or a sequence
of images to highlight sensitive information. The computer-generated system enables to process large amounts of
information rather fast and with no loss in quality. The data can be stored and transmitted in a compressed form, thus,
the speed of data transmission to the operator is significantly increased.
The computer vision system consists of a microprocessor (the ARM architecture with reduced power consumption)
and RAM to store intermediate results and dynamically highlight structures. Low-level interaction with this block is
done by using a real-time operating system with open distributed source (Linux). This approach allows us to abstract
from the coordinating signals between the transmitter block and the video capture module.
The source data of the computer vision system is analog information from an energy metering device which enters
the analytical system from the video sensor. The video sensor matrix is illuminated depending on the light and color
rendering of the image. Thus, a video buffer is formed, where each pixel is converted into an analog signal, the
saturation of which determines the light space of RGB. It is not practical to work and process such data because of the
time spent on transferring data to a logical block. Therefore, the shots of the video stream are converted into the
YCbCr color space family. Then subsampling takes place in the form of replacing an analog signal with a digital one
by means of transmitting brightness and saturation, it reduces the video data stream. This is done by the digital signal
processor of the camcorder.
This paper presents the algorithm for operation of a computer vision system. When writing the program, we used
the library of image processing algorithms via computer vision with open source OpenCV [1].
The image is captured from a video camera over a predetermined period of time with the help of the
cvCreateCameraCapture () feature and is downloaded for further processing in the program's memory (Figure 1).

Fig. 1. The video camera image.

To highlight the area of interest in the image (tariff scale) the originally uploaded image is translated into gray
gradation (Figure 2a) using cvCvtColor () function, and then binarized (Figure 2b) with cvThreshold ().This is
necessary for the application of morphological discovery of the image.
664 Stepan Sivkov et al. / Procedia Computer Science 169 (2020) 662–667
Author name / Procedia Computer Science 00 (2019) 000–000 3

Fig. 2. (a) the grayscale image; (b) the binarized image.

Morphological discovery (Figure 3) is necessary for further highlighting the area of interest in the image. It allows
us to get rid of unwanted objects in the image that are not in this area, as well as clearly distinguish the boundaries of
the object without thin lines, protrusions, interruptions and errant dots. Morphological discovery is performed using
the cvDilate () function.

Fig. 3. Morphological discovery.

Selecting the area of interest (Figure 4) is performed using the cvSetImageROI () function. Then the area is divided
into parts with the image of individual numbers.

Fig. 4. The image area of interest.

A number is recognized by comparing its contour with the contours of previously prepared templates. The contour
search (Figure 5) is performed using the cvFindContours () function. The contours are compared in three parameters:
the area of the contour, the perimeter, and the ratio of the area of the contour to the square of the perimeter. When
these three parameters of the source image coincide within the confidence interval with the template, the digit is
 Stepan Sivkov et al. / Procedia Computer Science 169 (2020) 662–667 665
4 Author name / Procedia Computer Science 00 (2019) 000–000

considered recognized and written to the file. This algorithm is used until the entire number is recognized and
transmitted to the operator.

Fig. 5. The recognised contours on the first segment.

To carry out the process of image recognition, it is also necessary to calibrate the camera. Here is the camera
calibration technique. Camera calibration is necessary to evaluate the parameters of internal, external and lenticular
distortion of the camera. Calibrating the camera allows us to correct optical distortions, judge the distance of an object
from the camera, and also measure the size of objects in the image. The camera was calibrated in the Matlab
application package, using the example of how it was implemented in [2].
Previously prepared calibration templates with the image of a chessboard at different angles, one side of which
contains an even number of squares and the other an odd number of squares, are loaded into the Matlab Camera
Calibrator (Figure 6). A chessboard image is displayed with green circles that indicate the detected dots. A yellow
square indicates a point with coordinates (0;0).

Fig. 6. The calibration template.

For improvingthe calibration, 3 radial distortion factors and tangential distortion calculation were set because of
the fact that the lens and the image plane are not parallel.Calibration accuracy is assessed by analyzing reprojection
errors and external camera parameters.
The histogram (Figure 7) displays the reprojection error in pixels of each image, along with the overall average
error. As a rule, reprojection error of less than one pixel is acceptable.
666 Stepanname
Author Sivkov et al. / Procedia
/ Procedia ComputerComputer
Science Science 169
00 (2019) (2020) 662–667
000–000 5

Fig. 7. The histogram of the reprojection error.

The external parameters of the camera are displayed in 3D, which determines the approximate location of the
camera lens (Figure 8).

Fig. 8. Visualization of external parameters.

Our next step is to display the values of the internal and external parameters of the camera, as well as distortion
coefficients for their further use (Figure 9).
6 Author name / Procedia Computer Science 00 (2019) 000–000
Stepan Sivkov et al. / Procedia Computer Science 169 (2020) 662–667 667

Fig. 9. Camera parameters.

Based on the foregoing, the experiment has been conducted. The calibrated camera was installed on the electric
energy meter in such a way that the numerical display was placed in the center of the obtained images. Under equal
ambient lighting conditions, 50 images were obtained with different numerical values of the readings. Of these
obtained images with information on energy consumption, 49 images were correctly recognized.
Thus, the proposed method of image processing and their recognition is applicable, the reliability of the information
according to the results of the experiment is 98%. The construction of such computer vision systems is possible using
this algorithm and allows us to automate the process of collecting and transmitting data from energy meters.

References

[1] OpenCV Reference Guide. [Electronic resource]. URL: http://docs.opencv.org

[2] Lachat E. et al. (2015) “First Experiences With Kinect V2 Sensor for Close Range 3D Modeling”. International Archives of the Photogrammetry,
Remote Sensing and Spatial Information Sciences: 93–100.
[3] Merkov A. (2011) “Pattern Recognition”. Introduction to statistical training methods – URSS Editorial,– 256.
[4] Shapiro L., Stockman J. (2007) “Computer vision” – Binom. Laboratory of Knowledge, – 752.
[5] Vizilter Yu.V., Zheltov S.Yu., Bondarenko A.V. (2010) “Image Processing and Analysis in Machine Vision Problems” – M.: Fizmatkniga,. –
672.
 Nuclear Engineering and Technology 50 (2018) 613e618

Contents lists available at ScienceDirect

Nuclear Engineering and Technology

journal homepage: www.elsevier.com/locate/net

Original Article

Development of a shared remote control robot for aerial work in

nuclear power plants
Hocheol Shin*, Seung Ho Jung, You Rack Choi, ChangHoi Kim
Nuclear Robot and Diagnosis Team, Korea Atomic Energy Institute, 989-111 Daedeok-daero, Yuseong-gu, Daejeon, 305-353, Republic of Korea

a r t i c l e i n f o a b s t r a c t

Article history: We are developing a shared remote control mobile robot for aerial work in nuclear power plants (NPPs);
Received 31 January 2018 a robot consists of a mobile platform, a telescopic mast, and a dual-arm slave with a working tool. It is
Received in revised form used at a high location operating the manual operation mechanism of a fuel changer of a heavy water
10 March 2018
NPP. The robot system can cut/weld a pipe remotely in the case of an emergency or during the
Accepted 11 March 2018
Available online 23 March 2018
dismantling of the NPP. Owing to the challenging control mission considering limited human operator
cognitive capability, some remote tasks require a shared control scheme, which demands systematic
software design and integration. Therefore, we designed the architecture of the software systematically.
Keywords:
Aerial Work
© 2018 Korean Nuclear Society, Published by Elsevier Korea LLC. This is an open access article under the
Nuclear Power Plant CC BY-NC-ND license (http://creativecommons.org/licenses/by-nc-nd/4.0/).
Shared Remote Control Robot

1. Introduction Because the area of a severe accident of an NPP may be

The environment for maintaining tasks in nuclear facilities can
be hazardous. Therefore, many automation devices are being used
to inspect and repair such facilities. However, it is difficult to use
fixed automation devices for nonperiodic and infrequent mainte-
nance tasks. Therefore, remotely controlled robot systems have
been proposed.
If a fuel-handling machine, one of the major components of a
pressurized heavy-water reactor (PHWR) nuclear power plant
(NPP), ages, the machine may be stuck on the pressure tube in front
of the PHWR calandria. Although the machine has a trouble-
shooting measure of a manual drive mechanism, it is still a difficult
problem to access the manual drive mechanism. When the ma-
chine is stuck, the NPP is operated such that the radiation level is
extremely high and the machine can be located at a high position of
up to 9 m. Therefore, a human worker cannot approach the
mechanism, and the mechanism must be handled remotely. Shin
et al. developed an aerial working mobile robot for monitoring a
high radiation area and operating a manual drive mechanism of a
fuel exchange machine (MADMOFEM) [1].
Mitigation robots used during the Fukushima NPP accident and
dismantling robots for NPPs should find the exact status of the area
while moving around [2e4].
* Corresponding author.
E-mail addresses: smarthc@kaeri.re.kr (H. Shin), shjung2@kaeri.re.kr
(S.H. Jung), yrchoi@kaeri.re.kr (Y.R. Choi), chkim3@kaeri.re.kr (C. Kim).
complex with various obstacles, the dismantling robot should deal
with dismantled parts and may carry out cutting work. Therefore,
an articulated multi-arm robot system is desirable to use rather
than a robot with one task-specific device.
Using an aerial working mobile robot, we therefore designed a
remote control system for maintaining and repairing tasks in an
NPP. A dual-arm manipulator is going to be installed instead of a
single taskespecific device. We named the remote control system
to be developed MARTiN (remote control system for Maintaining
And Repairing Tasks in NPP). MARTiN can operate the manual drive
mechanism of a fuel exchange machine and cut/weld a pipe during
an emergency situation.
We developed tools for MARTiN to operate the MADMOFEM and
cut a pipe. We tested the feasibility of the prototypes.
The dual-arm manipulator as a slave robot is controlled by a
master device. It is important to reduce the working time and
operator fatigue and to increase the stability and accuracy in the
remote work by using a haptic device. In the case of complex tasks,
visual and haptic virtual guide technology is useful for reducing the
work time and operator fatigue. We tested the efficiency of con-
straining some degree of freedom (DoF) as a haptic virtual guide [5].
The target tasks and working environment vary from case to
case. Therefore, it is difficult to make the virtual guide in advance.
We used human intuition to generate the virtual guidance. The
operator sketches the path on the visual feedback image of the
remote environment. Then, the virtual guide is generated based on
the sketches [6].

https://doi.org/10.1016/j.net.2018.03.006
1738-5733/© 2018 Korean Nuclear Society, Published by Elsevier Korea LLC. This is an open access article under the CC BY-NC-ND license (http://creativecommons.org/
licenses/by-nc-nd/4.0/).
614 H. Shin et al. / Nuclear Engineering and Technology 50 (2018) 613e618

Because the user operates the master device with a limited view
of the cameras, he may not understand the situation of the slave
robot. Therefore, he may proceed with the work even though the
slave robot is approaching a poor kinematic condition such as a
singularity position. We designed a master control program to
provide helpful information to the operator to intuit the singularity
information of the slave robot [7].
The aerial working mobile robot, with a telescopic mast, can
raise its dual-arm manipulator to a height of 9 m. If the dual-arm
manipulator is operated at that height, vibration at the manipulator
base can be caused by the flexibility of the telescopic mast. It is very
difficult for a human operator to control the slave robot while
suppressing the vibration at the same time. Therefore, we propose a
shared control scheme that allows the human operator to control
the slave robot only and assign the vibration-suppressing task to
the robot system. In addition, we tested the controller using a
mockup test bed [8].
2. Target tasks
There can be many tasks in a high radiation area such as the
operation of valves and switches. First, we are going to have
MARTiN remotely operate the MADMOFEM when the machine is
stuck. In addition, we are going to develop remote control tech-
nology with which MARTiN can cut/weld a pipe under an emer-
gency situation or in the dismantling process. Fig. 1 shows an
example of the target tasks.
Fig. 2 shows the concept of an aerial task in a PHWR NPP. A dual-
arm manipulator on a telescopic mast on a mobile platform ma-
nipulates the MADMOFEM. A human worker in a safe area remotely
operates the dual-arm manipulator as a slave robot through the
master device.
3. Hardware
The mobile platform has four wheels and four flippers. Each
flipper has an active small wheel at the end. The platform is able to
pass through a gate of 0.9 m in width, to move with loads of 250 kg,
and to cross a ditch with a width of 0.75 m and a depth of 0.25 m by
using the flippers. The mobile platform changes direction using a
skid-steering method, which is easy for the platform with two omni
wheels at the rear side.
The initial height of the robot should be less than 2 m such that
the robot can pass through the gate of the shielding aisle. Therefore,

Fig. 1. Target tasks.

Fig. 2. Concept of maintenance tasks in nuclear facilities.

 H. Shin et al. / Nuclear Engineering and Technology 50 (2018) 613e618 615

Fig. 5. Cutting tool test bed.

Fig. 3. Mobile platform, slave, and master.

Fig. 6. Concept of the slave robot with the cutting tool.

Fig. 4. MADMOFEM mockup and tools.

MADMOFEM, manual drive mechanism of a fuel exchange machine.

the telescopic mast is composed of identical-shaped frames sliding

synchronously with a cable-driven mechanism; mast can reach up
to 9 m including the mobile platform.
The dual arm to be installed on the linear guide on the top of the Fig. 7. Virtual guide for a peg-in-hole task.
mast should be light with a high payload capacity and respond to
small external forces; as a result, LBR iiwa 14 R820 is chosen as the
dual-arm robot. This is a lightweight 7DoF-articulated manipulator
that weighs 29.5 kg and has a payload capacity of 14 kg; it is able to
move precisely with ±0.1 mm repeatability. It assembles parts deli-
cately and detects external forces with integrated torque sensors.
The master device is an Omega 7 of Force Dimension. Omega 7
detects the translational motion of the X/Y/Z directions and the
rotational motion of the rx/ry/rz directions. In addition, it supplies
the reflection forces, which are 12 N for the translational motion
and 8 N for grasping. Its linear accuracy is 0.01 mm. It supplies
sufficient stability with an 8 kHz refresh rate. Fig. 8. Applying a manipulability ellipsoid from the slave robot to the master device.
616 H. Shin et al. / Nuclear Engineering and Technology 50 (2018) 613e618

Fig. 9. Singularity information program GUI.

3DoF
manipulator

1DoF
linear stage

Flexible beam

Fig. 11. A shared control test bed of the manipulator on a flexible base.
DoF, degree of freedom.

We previously developed a tele-rameoperating device to drive

MADMOFEM as a one taskespecific device. The operating tool, at
3 kg, is very light compared with the tele-rameoperating device
(50 kg). Fig. 4 shows the MADMOFEM mockup, the operating tool,
and the previously developed tele-rameoperating device.
We developed a pipe-cutting tool for the slave robot arm.
Because the target is a 2-inch pipe with a 5-mm thickness, a 9-inch
Fig. 10. Shared control concept for the MARTiN System. circular saw was selected. The circular saw of 2-mm thickness can
cut the steel and stainless steel pipe with a 5-mm pitch tooth.
We cut the target steel pipe with the cutting tool test bed. It took
Fig. 3 shows the mobile platform, slave robots, and master 21 minutes to cut the target pipe. Fig. 5 shows the cutting tool test
devices.
The object to be operated and the work environment are
recognized with a depth camera such as a time of flight (ToF)
camera and stereo camera. A Kinect v2 (ToF camera) and a ZED
camera (stereo camera) were considered as remote field depth
formationegathering device. Embedded boards such as ODROID
XU4 for the Kinect v2 and Jetson TX1 for the ZED are used in
sending the remote field information to the operating room.
There are three parts to be operated at MADMOFEM: a clutch
axis, an engaging helical gear axis, and a ram driving axis. We made
a MADMOFEM mockup to be tested and an operating tool for the
slave robot.
The operating tool has three functions: operating the clutch,
engaging the helical gear, and driving the ram. The operating tool is
installed at the end of the slave robot. The slave robot can change
the position of the operating tool from the helical geareengaging
part to the clutch and the ram driving tool part so that the slave
robot can operate three parts of the MADMOFEM. Fig. 12. Vibration suppression control response.
 H. Shin et al. / Nuclear Engineering and Technology 50 (2018) 613e618
Fig. 13. Software integration diagram.
bed, and Fig. 6 shows the concept of the slave robot with the cutting
tool.
4. Shared control
Owing to the challenging control mission considering limited
human operator cognitive capability, some remote tasks require a
shared control scheme, which demands systematic software design
and integration.
We developed software modules separately for the mobile
platform, the telescopic mast, the linear guide, the slave robots, the
master devices, and the depth camera.
It would be very difficult to integrate the individual software
modules if the each module is developed without considering other
modules. Therefore, we designed the architecture of the software to
successfully integrate the software modules.
4.1. Real-time virtual guidance
To reduce the working time and operator fatigue, we adopted
virtual guide technology. However, because the environment is
unstructured, it is difficult to build the virtual guidance in advance.
Therefore, a virtual guide needs to be generated on the spot.
One of the target tasks of this research is to operate the MAD-
MOFEM, in which the slave robot arm should insert the operating
tool into the driving axes. This task is a peg-in-hole task.
First, we select the representative features of the operating tool
as a peg on the computer monitor displaying a 2D/3D image of the
spot: the center of the contour of the peg and its axis. Second, we
select the driving axis as a hole and normal to its plane. The normal
617
should be aligned with the center of the hole. Then, a guidance
fixture is created that will attract the peg to the axis of the hole.
Fig. 7 shows the virtual guide for a peg-in-hole task.
4.2. Egocentric remote control
We will provide helpful information for the operator to feel the
poor kinematic conditions of the slave robot, such as a singularity
position. To find the singularity direction, the manipulability
ellipsoid is computed from the Jacobian matrix of the slave robot.
We can apply the ellipsoid of the slave robot to the master de-
vice. We provide force feedback to the master device based on the
size and direction of the ellipsoid, resulting in preventing access to
the singularity. Fig. 8 shows the slaveemaster system, which is
composed using PHANTOM Omni as the master device and KINOVA
Jaco as the slave robot.
Fig. 9 shows a program in which the graphical user interface
(GUI) gives visual feedback information of a normalized manipu-
lability measure, singularity direction, and visual feedback of the
slaveemaster system of Fig. 8. It should be noted that when the
manipulator is far from the singularity posture, the color of the
graph is blue. On the other hand, when the robot approaches the
singularity, the color of the graph turns red. In addition, a red arrow
in the circle of the GUI indicates the direction of the singularity.
4.3. Shared control of the slave robot
It is very difficult for the tele-operator to control the slave robot
while suppressing the vibration discussed previously. For the
operator to concentrate on carrying out the target task, we applied
618 H. Shin et al. / Nuclear Engineering and Technology 50 (2018) 613e618
a shared control strategy, in which the system suppresses the vi-
bration of the flexible base automatically. Fig. 10 shows the concept
of the shared control strategy for the MARTiN system.
We built a test bed to test the shared control scheme, as shown
in Fig. 11.
Before we decompose the system dynamics, we obtain the
modeling of the system based on the EulereBernoulli theory. We
split the flexible base dynamics and the manipulator dynamics to
which the original dynamics converges as the vibration is
suppressed.
We can then build an end-effector tracking controller and a vi-
bration suppression controller separately. Fig. 12 shows the vibra-
tion suppression control response when the 3DoF manipulator
moves from a folding position to an erect position.
4.4. Integration of the control system
The different software types, developed separately, are con-
nected using TCP, USB, and CAN communications and eventually
have to be integrated into a single control system. To facilitate the
integration of such software types, we set the development envi-
ronment uniformly: Windows 7 (64bit) OS and VC 2010 for Win-
dows and Ubuntu 14.0.4 (64bit) OS and QT4/Cþþ/Python for Linux.
The control frequency or communication frequency is very
important in robot control. In addition, when integrating multiple
software types, it is especially important to set the priorities of
these frequencies. Because the shared control capability of a slave
robot and the haptic stability of a master device require a high
frequency, the communication frequencies of these components
are set at 500 Hz in the development of the MARTiN system. The
updating rate for rendering the point, cloud data of the virtual
guide and the location information of the object are set to 10 Hz. In
addition, the communication frequency of each joint state and the
joint control data are set to 100 Hz. Fig. 13 shows the communi-
cation connection and communication periods for integration of
the software described previously.
It should be noted that the user directly commands the control
system with the master device, a mouse, and a keyboard. The
control system orders the slave robot to the user-commanded po-
sition while suppressing the external disturbance, displays an
egocentric view directly from the camera, and creates an exocentric
view, virtual guidance, and the shared path on the spot; it then
feeds back the shared control information to the user through the
monitor and the master device.
5. Conclusions
We designed a remote control system, called MARTiN, for aerial
tasks in an NPP using an aerial working mobile robot, developed
control algorithms, and carried out a feasibility test. An LBR iiwa 14
R820 was chosen as the dual-arm slave robot, and an Omega 7 of
Force Dimension was chosen as the master device. We made a
MADMOFEM-operating tool, and a pipe-cutting tool for the slave
robot was created and tested.
To reduce the working time and operator fatigue and to increase
the stability and accuracy in the remote work, we adopted virtual
guidance technology and a manipulability ellipsoid to prevent the
slave robot from accessing a singularity. The shared controller was
developed to automatically suppress the vibration of the flexible
base. To integrate individual control algorithms and software, we
designed the control architecture, communication flow, and update
rate.
Evaluation indexes and experiments for the MARTiN system will
be designed to evaluate and improve the performance of the
technologies. The performance of the MARTiN system will be
evaluated through the target tasks.
Conflict of interest
There is no conflict of interest.
Acknowledgments
This work was supported by the Industrial Strategic Technology
Development Program (10060070) funded by MOTIE, Korea.
References
[1] H. Shin, C. Kim, Y. Seo, K. Jeong, Y. Choi, B. Choi, J. Moon, Aerial work robot for a
nuclear power plant with a pressurized heavy water reactor, Ann. Nucl. Energy
92 (2016) 284e288.
[2] J.W. Cho, K.M. Jeong, Remote-controlled robots used for the management of
severe accident of the Fukushima nuclear power plant, in: Trans. of the Korean
Nuclear Society Autumn Meeting, 2011, pp. 983e987.
[3] Mohamed J. Bakari, Khaled M. Zied, Derick W. Seward, Development of a multi-
arm mobile robot for nuclear decommissioning tasks, Int. J. Adv. Rob. Syst. 4 (4)
(2007) 387e406.
[4] H. Shin, C. Kim, Y. Seo, M. Kim, B. Choi, Development of a dismantling
manipulator for a nuclear reactor, in: Proc. of American Nuclear Society 2016
Topical Meeting D&RS, 2016, pp. 21e23.
[5] K.S. Choi, S.W. Ryu, G.H. Yang, Study on measure to shorten work time, through
the haptic device in teleoperation system, in: Proc. of URAI 2017, Jeju, Korea,
June 27eJuly 1, 2017, pp. 473e476.
[6] K.H. Lee, V. Pruks, J.H. Ryu, Development of shared autonomy and virtual
guidance generation system for human interactive teleoperation, in: Proc. of
URAI 2017, Jeju, Korea, June 27eJuly 1, 2017, pp. 458e461.
[7] J.W. Han, G.H. Yang, Singularity avoidance in teleoperation system through
force feedback of master device, in: Proc. of URAI 2017, Jeju, Korea, June 27eJuly
1, 2017, pp. 470e472.
[8] C. Ha, H. Kim, D. Lee, Experimental evaluation of passivity-based control of
manipulator-stage system on flexible beam, in: Proc. of URAI 2017, Jeju, Korea,
June 27eJuly 1, 2017, pp. 465e466.
 Available online at www.sciencedirect.com

ScienceDirect
Procedia Engineering 168 (2016) 1671 – 1675

30th Eurosensors Conference, EUROSENSORS 2016

Embedded control of a PMSM servo drive

without current measurements
Dino Hüllmann*, Harald Kohlhoff, Patrick Neumann
Bundesanstalt für Materialforschung und -prüfung, Unter den Eichen 87, 12205 Berlin, Germany

Abstract

A permanent-magnet synchronous motor (PMSM) servo drive for lightweight robotic platforms that have a high torque demand
at low rotational speeds has been developed. First, a current-independent torque controller is derived and cascaded with a speed
and position controller, while merely an encoder is used as a sensor device in combination with a speed estimator. Finally, the
speed estimator output is compared to gyroscope measurements and the overall functioning is verified on a real system.
© 2016
© 2016TheTheAuthors.
Authors. Published
Published by Elsevier
by Elsevier Ltd. is an open access article under the CC BY-NC-ND license
Ltd. This
(http://creativecommons.org/licenses/by-nc-nd/4.0/).
Peer-review under responsibility of the organizing committee of the 30th Eurosensors Conference.
Peer-review under responsibility of the organizing committee of the 30th Eurosensors Conference
Keywords: PMSM ; BLDC ; servo drive ; control ; optical incremental encoder ; speed estimator

1. Introduction

The permanent-magnet synchronous motor (PMSM) combines several advantages compared to classical motor
types: high power and torque density, efficiency, reliability, low inertia and high dynamics. Even though this comes
at the cost of a more complex control scheme, these characteristics make the PMSM well-suited for applications
where size and weight matter, e.g. mobile robotics. Particularly, servo drives used as actuators can benefit from it.
Modern sensorless algorithms estimate speed and position by means of current measurements, but they aim at
high-speed drives and cannot be applied to comparatively slow servo applications [1]. Hence, additional sensors are
used to measure position and speed, while current measurements are still required. In this paper, an approach is

* Corresponding author. Tel.: +49-30-8104-4790; fax: +49-30-8104-1917.

E-mail address: dino.huellmann@bam.de

1877-7058 © 2016 The Authors. Published by Elsevier Ltd. This is an open access article under the CC BY-NC-ND license
(http://creativecommons.org/licenses/by-nc-nd/4.0/).
Peer-review under responsibility of the organizing committee of the 30th Eurosensors Conference
doi:10.1016/j.proeng.2016.11.487
1672 Dino Hüllmann et al. / Procedia Engineering 168 (2016) 1671 – 1675

presented that does not depend on current measurements and, instead, derives all quantities from an optical
incremental encoder.

2. Mathematical model of the PMSM

Initially, a mathematical model of the PMSM is needed to design a suitable controller. The underlying equations
describing the system dynamics can be found in [2] or virtually all papers about field-oriented or vector control
(FOC).
The stator voltage can be expressed in the stator reference frame S, usually called the Įȕ reference frame [3], as

݀ ௌ ݀
࢛ௌ ൌ ܴௌ ‫࢏ ڄ‬ௌ ൅ ௌሻ
શ ൌ ܴௌ ‫࢏ ڄ‬ௌ ൅ ሺ‫ܮ‬ௌ ‫࢏ ڄ‬ௌ ൅ શ௠ (1)
݀‫ݐ‬ ݀‫ݐ‬

with RS being the resistance of the stator windings and iS being the 3-phase currents. શௌ is the stator flux, LS
ௌ
describes the self-inductance of the stator and શ௠ is the contribution of the rotor flux. The latter can be treated as a
ோ
constant in the rotor reference frame R, i.e. શ௠ ൌ ߰௉ெ . Usually, this frame is named dq coordinate system [3] and
can be simply transformed into the stator system S by rotating it through the electrical rotor angle ߴ:

ௌ
શ௠ ோ
ൌ ‡௝ణ ‫ ڄ‬શ௠ ൌ ‡௝ణ ‫߰ ڄ‬௉ெ with ݆ ଶ ൌ െͳ (2)

The essential control parameter, however, is the torque generated by the motor, which is given by

͵ ͵
߬ெ ൌ ‫߰ ڄ ݌ ڄ‬௉ெ ‫ ڄ‬ሾ࢏ோ ሿ ൌ ‫߰ ڄ ݌ ڄ‬௉ெ ‫ ڄ‬ൣ‡ି௝ణ ‫࢏ ڄ‬ௌ ൧ (3)
ʹ ʹ

with p being the number of pole pairs of the motor. This number also gives a relation between the electrical and the
mechanical rotor angle: ൌ ‫ߴ ڄ ݌‬୫ୣୡ୦ . At this point, a usual field-oriented controller would come into operation,
trying to adjust iS so that the desired torque is delivered. Instead, we insert (2) and (3) into (1):

݀ ௌ
࢛ௌ ൌ ܴௌ ‫࢏ ڄ‬ௌ ൅ ‫ܮ‬ௌ ࢏ ൅ ݆ ‫‡ ڄ‬௝ణ ‫ߴ ڄ‬ሶ ‫߰ ڄ‬௉ெ (4)
݀‫ݐ‬

Let the applied stator voltages be ࢛ௌ ൌ ‡௝ఝ ‫࢛  ڄ‬ ෝ , with ij being the commanded angle of the electrical field.
Further, ࢏ௌ ൌ ‡௝ఊ ‫ ڄ‬ଙƸ may denote the currents caused by these voltages. In general, the currents are out-of-phase with
the voltages and therefore the angle Ȗ differs from ij. Inserting these definitions into (4) yields

ෝ ൌ ሺܴௌ ൅ ݆ ‫ܮ‬ௌ ߛሶ ሻ݁ ௝ఊ ‫ି ݁ ڄ‬௝ణ ‫ ڄ‬ଙƸ ൅ ݆ ߴሶ߰௉ெ

‡௝ሺఝିణሻ ‫࢛ ڄ‬ (5)

ෝ െ ݆ ߴሶ߰௉ெ
݁ ௝ሺఝିణሻ ‫࢛ ڄ‬
࢏ௌ ൌ ݁ ௝ఊ ‫ ڄ‬ଙƸ ൌ ݁ ௝ణ (6)
ܴௌ ൅ ݆‫ܮ‬ௌ ߛሶ

Typically, servo PMSM are characterized by specific parameter ratios: ܴௌ ൌ ͳͶǤͶ Ÿ and ‫ܮ‬ௌ ൌ ʹ are
exemplary values for a small motor with stator dimension 8 × Ø 22 mm, 7 pole pairs and 80 windings. In case of
servo motors these parameters are optimized for high torque and not for high speed operation.
Assuming the maximum rotational speed of the motor is ߴ୫ୟ୶ ሶ ൌ ʹͶͲ”’ ൌ ͺɎ ”ƒ†Τ•, the maximum rate of
ሶ
change of the current angle Ȗ becomes ߛሶ୫ୟ୶ ൌ ‫ߴ ڄ ݌‬୫ୟ୶ ൌ ͷ͸Ɏ ”ƒ†Τ•, if short time switching effects are neglected.
Thus, ܴௌ ൌ ͳͶǤͶ Ÿ ‫ܮ ب‬ௌ ‫ߛ ڄ‬ሶ ൌ ͲǤ͵ͷ Ÿ holds and the imaginary part of the denumerator in (6) might be neglected:

͵ ‫߰ ڄ ݌‬௉ெ
߬ெ ൌ ‫ڄ‬ ൫‫ݑ‬ො ‫‹• ڄ‬ሺ߮ െ ߴሻ െ ߰௉ெ ‫ߴ ڄ‬ሶ൯ (7)
ʹ ܴௌ
 Dino Hüllmann et al. / Procedia Engineering 168 (2016) 1671 – 1675 1673

The sinus term conforms to the well-known property of synchronous machines to generate most torque if the
electrical field is 90° ahead or behind the electrical rotor position.

a) b) IMU Encoder

ࢡref Position vref Speed IJref Torque ij

Motor
Controller Controller Controller
‫ݒ‬ୣୱ୲ ൎ ߴሶ୫ୣୡ୦
ࢡmech
Speed Encoder

PMSM Load motor

Fig. 1. a) Control structure, b) The motor test bed setup used in the experiments.

3. Control structure of the servo drive

Servo drives often consist of a torque, speed and position controller. This proven concept is also applied in this
work, as shown in fig. 1a. Optical incremental encoders do not provide a speed signal themselves, therefore the
setup is augmented with an additional speed estimator, which is described in the next section.
Since friction can only decrease the acting torque and the cascaded controller is able to compensate deviations at
the torque level, we may neglect all friction terms. Moreover, we assume a constant voltage supply ‫ݑ‬ො and do a first-
order Taylor approximation of the sine term in (7) around the point ߮ െ ߴ ൌ Ͳ. Inserting these approximations
yields

߬ெ ൌ ‫ܥ‬ଵ ‫ ڄ‬ሺ߮ െ ߴሻ ൌ ‫ܥ‬ଵ ‫ ڄ‬ሺ߮ െ ‫ߴ ڄ ݌‬୫ୣୡ୦ ሻ (8)

with ‫ܥ‬ଵ ൌ ͵Τʹ ‫߰ ڄ ݌ ڄ‬௉ெ Τܴௌ ‫ݑ ڄ‬ො ൌ …‘•–Ǥ By solving (8) for the control input ij, that is the predefined angle of the
electrical field, we obtain the underlying equation of the torque controller:

߬୰ୣ୤ ሺ‫ݐ‬ሻ
߮ሺ‫ݐ‬ሻ ൌ ൅ ‫ߴ ڄ ݌‬୫ୣୡ୦ ሺ‫ݐ‬ሻ (9)
‫ܥ‬ଵ

Since the rate of change of speed, ߴሷ୫ୣୡ୦ , is directly proportional to the applied torque, a proportional-integral
controller (PI controller) is chosen to adjust the rotational velocity and to overcome steady-state errors caused by
friction and other neglected or unpredictable effects. In contrast, a simple P-controller is chosen for the positional
part.

4. Speed estimation

In [4] different velocity estimation approaches using incremental encoders are compared. One method promises
very low relative errors while having modest system requirements. Basically, a timestamp is assigned to each
detected encoder tick. Now, let ݊ሺ‫ݐ‬ሻ denote the encoder tick count at time t, then the velocity can be estimated as

݊ሺ‫ݐ‬ଶ ሻ െ ݊ሺ‫ݐ‬ଵ ሻ
‫ݒ‬ୣୱ୲ ሺ‫ݐ‬ሻ ൌ ‫ݐ‬ଶ ൐ ‫ݐ‬ଵ (10)
‫ݐ‬ଶ െ ‫ݐ‬ଵ

with ‫ݐ‬ଶ being the most recent timestamp at time t and a minimal time interval between ‫ݐ‬ଵ and ‫ݐ‬ଶ , i.e. ‫ݐ‬ଶ െ ‫ݐ‬ଵ ൒
ȟ‫ݐ‬୫୧୬ .
1674 Dino Hüllmann et al. / Procedia Engineering 168 (2016) 1671 – 1675

5. Experimental results

To test the performance of the proposed servo controller, a motor test bed was built. Its main components are the
PMSM with the specifications given above as well as an optical incremental encoder and an inertial measurement
unit (IMU) that are connected to the motor shaft (see fig. 1b). Firstly, the output of the speed estimator has to be
verified. For this purpose, trajectories are generated by turning the motor by hand, as the example in fig. 2 shows.
From the plot on the right side it can be seen that there is a systematic error portion between both signals. However,
in all recorded tests the root-mean-square error was smaller than ͲǤʹ ”ƒ†Τ• ൏ ʹ”’. This is sufficient for our
purpose.
20 0.8

0.6

Angular velocity error [rad/s]

10
Angular velocity [rad/s]

0.4

0 0.2

0
-10 -0.2

-0.4
-20
Gyroscope -0.6
Encoder-based estimate
-30 -0.8
500 1000 1500 2000 2500 3000 500 1000 1500 2000 2500 3000
t [ms] t [ms]

Fig. 2. Angular velocity measured with a gyroscope compared to the estimator using encoder measurements, ȟ‫ݐ‬୫୧୬ ൌ ͵•.

60 50

50
40
Angular velocity [rad/s]

40
30
Angle [°]

30
20
20

10
10
Reference Reference
Speed Position
0 0
750 800 850 900 950 1000 740 760 780 800 820
t [ms] t [ms]

Fig. 3. Step response of the speed controller (left) and the position controller (right).

In fig. 3, a step response of the speed and the position controller are shown. Both controllers were only slightly
tuned, so probably a better performance could be achieved using more suitable parameters. However, as the speed
step response shows, the angular velocity fluctuates around the set point. For multiple rotations it can be observed
that this fluctuation is periodic and therefore likely caused by the installed Oldham couplings or manufacturing
tolerances. On the other hand, the position step response shows a noticeable kink shortly before the set point is
reached. Most likely, this is caused by static friction occurring when the velocity falls below a specific limit.

6. Summary and conclusion

Starting from a fundamental system model of the PMSM, a current-independent torque equation was derived and
used to design a servo control setup that merely depends on encoder measurements. Experimental results have
shown the performance of both the utilized speed estimator and the controller, which were implemented on an
embedded system. However, although the cascaded controller is basically functional, there is still room for
improvements.
 Dino Hüllmann et al. / Procedia Engineering 168 (2016) 1671 – 1675 1675

Acknowledgements

This work is funded by the German Federal Ministry for Economic Affairs and Energy (BMWi), grant number
KF2201091HM4, within the ZIM program.

References

[1] J.C. Gamazo-Real, E. Vázquez-Sánchez, J. Gómez-Gil, Position and speed control of brushless DC motors using sensorless techniques and
application trends, Sensors 10 (2010) 6901–6947.
[2] P. Pillay, R. Krishnan, Modeling of permanent magnet motor drives, IEEE Transactions on Industrial Electronics 35, 4 (1988) 537–541.
[3] B Sahhary, Elektr. Antriebe mit dauermagneterregten Maschinen im dyn. sensorlosen Betrieb, Helmut-Schmidt-Universität, Hamburg (2008).
[4] R. Petrella, M. Tursini, L. Peretti, M. Zigliotto, Speed measurement algorithms for low-resolution incremental encoder equipped drives: a
comparative analysis, IEEE International Aegean Conference on Electrical Machines and Power Electronics ACEMP’07 (2007) 780–787.
 Available online at www.sciencedirect.com

ScienceDirect
Procedia CIRP 14 (2014) 488 – 493

6th CIRP International Conference on High Performance Cutting, HPC2014

Real-Time Monitoring of High-Speed Spindle Operations using Infrared

Data Transmission
K. Drödera, H.-W. Hoffmeistera, M. Luiga*, T. Tounsia, T. Blumea
a
Insitute of Machine Tools and Production Technology, Technische Universität Braunschweig, Germany
* Corresponding author. Tel.: +49-531-391-7165; fax: +49-531-391-5842. E-mail address: M.Luig@iwf.tu-braunschweig.de

Abstract

High-performance cutting is carried out with high cutting and feed speeds. Particularly, the use of heavy cutting tools
(e.g. in planing machines), it is important to monitor the clamping and balance condition of the mounted tool, as well
as the process forces. Therefore, a real-time monitoring system for high-speed operations based on the IrDA protocol
was developed. It could be shown that infrared data transmission systems allow shorter reaction times compared to
conventional wireless LAN applications. The presented monitoring system provides a reaction time of 7.14 ms at a
bandwidth of 42.5 kHz and a data rate of 4.1 MBit/s.
©2014
© 2014The
Published
Authors.byPublished
Elsevier by
B.V. Open access
Elsevier B.V. under CC BY-NC-ND license.
Selectionand
Selection andpeer-review
peer-review under
under responsibility
responsibility of the
of the International
International Scientific
Scientific Committee
Committee of theof
6ththe 6th International
CIRP CIRP International Conference
Conference on High
on High Performance
Performance Cutting. Cutting

Keywords: IrDA; real-time monitoring; process monitoring

necessary to provide high data transmission rates with a low

1. Introduction
In high performance cutting (HPC) the cutting speed and
feed speed is significantly higher than in conventional cutting
processes. This makes it more difficult to attain a sufficient
correlation between the required process parameter and the
sensor signal. Process monitoring is generally carried out
using sensor-integrated machine tool components or by
monitoring the spindle/motor current. However, the bandwidth
of the sensor system decreases with an increase in distance
between the sensor and cutting zone (Fig. 1). This is
particularly true when processing workpieces with a high
mass; the eigenfrequency of the workpiece decreases and
therefore the measurement error increases. For this reason, it
is beneficial to locate the sensor directly within the rotating
tool holder or the spindle shaft. [8]
The embedding of sensors and their electronic equipment
into high-performance machining spindles requires a fast and
reliable data transmission system in order to fulfil the
sampling theorem. When data transmission is linked to a real-
time monitoring system (e.g. safety devices), it is especially
reaction time. Frequency hopping methods, which will be
mandatory for wireless LAN applications in Europe from
2015, do not meet those requirements [1] [2] [4].
The aim of this project was to provide a minimum reaction
time for high-performance wood machining processes with a
feed speed up to 600 m/min. Planing machines are equipped
with heavy cutting tools (up to 35 kg) that are clamped within
belt-driven spindles using hollow shank taper systems. In
contrast to conventional planing machines (double-sided
bearings), this allows an increase in the rotational speed up to
12,000 rpm (cutting speed up to 100 m/s) and reduction in tool
changing time. As imbalance mass quadratically increases
with the rotational speed, a sensor-integrated machining
spindle using thin-film sensor systems was developed [3].
This spindle allows monitoring of the clamping force, the
balance condition of a mounted tool, and the cutting force. In
order to increase the bandwidth (excitation frequency: 4 kHz)
and allow to activate counter measures (e.g. due to critical
balance conditions) with a minimum reaction time, a real-time
monitoring system for high-speed spindle operations, based on

2212-8271 © 2014 Published by Elsevier B.V. Open access under CC BY-NC-ND license.
Selection and peer-review under responsibility of the International Scientific Committee of the 6th CIRP International Conference
on High Performance Cutting
doi:10.1016/j.procir.2014.03.058
 K. Dröder et al. / Procedia CIRP 14 (2014) 488 – 493
an infrared data transmission unit (IrDA standard), was
developed. The interface to the sensors is an analog-digital-
converter (ADC), which allows the use of any kind of sensor
(e.g. strain gauge, piezoelectric sensor) with the system.
Fig. 1: Comparison of process force measurements using sensor platforms
and sensor-integrated solutions
2. Infrared Data Transmission Unit
Machine spindle shafts are designed in order to achieve
maximum stiffness with a minimum mass moment of inertia.
Therefore, the major requirement of embedded data
transmission units is a high data transmission rate with a
minimum device volume on the rotating side. In Fig. 2 a
comparison of common data transmission protocols is shown.
Based on [6] and [7] the data transmission rate is plotted
against the hardware and software complexity. The field of
application of the standard ETSI EN 30028 (mandatory
frequency hopping) is also indicated [6]. As illustrated,
infrared protocols provide a higher data rate to complexity
ratio. Therefore, infrared protocols, particularly for short
distance applications (e.g. within machining spindles), are
preferable to wireless LAN applications. Furthermore, optical
communication systems are less sensitive to disruptions e.g.
generated by motors or induction coils [12]. Current
developments include ready-to-use IrDA protocol controllers
and controller cores, which provide data rates up to 1 Gbit/s
and programmable deterministic characteristics [9] [11].
The infrared data transmission unit is based on two
systems: the embedded electronics within a rotating spindle
shaft and a stationary evaluation unit (Fig. 3). On the rotating
part, the amplified sensor signals are converted using a four
channel 16 bit ADC. In order to guarantee zero data sampling
latency, a simultaneous sampling ADC (type: MAX11047)
489
Fig. 2: Comparison of common data transmission protocols for contact-less
data transmission (based on [6] and [7])
with a maximum sampling rate of 250 kSample/s was used.
The ADC is connected to an Atmel 32 bit microcontroller
(μC) with a clock rate of 66 MHz. The package type housing
the microcontroller is TQFP-64 with a maximum number of
45 GPIOs (General-Purpose Input/Output), used for the
parallel bus systems of the ADC and the IrDA protocol
controller. The IrDA protocol controller (type: IPMS_IRHSP,
Fraunhofer IPMS, [9]) is a separate integrated circuit with the
package type SSOP28. The IPMS_IRHSP supports IrPHY
VFIR (Very Fast Infrared) with a maximum data transmission
rate of 16 MBit/s and allows exclusion of level 2 (IrLAP) and
level 3 (IrLMP) of the OSI (Open Systems Interconnection)
model, in order to use it as a data streaming system. The
actual data link is carried out using two TFDU8108
transceiver modules, both of which are directly connected to
the IPMS_IRHSP. On the stationary side, a larger, and
therefore more powerful, microcontroller can be used e.g. to
run process monitoring calculations or for data transmission
using USB protocols.
Fig. 3: Setup of the infrared data transmission unit
The infrared data transmission unit was set up and tested
using two self-contained evaluation boards. In order to
analyze different components of the data transmission unit,
each component was soldered onto its own board; all main
components (ADC, μC etc.) were selected with a minimum
application volume. The rotor microcontroller evaluation
board was linked to the ADC, the IrDA protocol controller
and the transceiver modules using a base board (Fig. 4). The
490 K. Dröder et al. / Procedia CIRP 14 (2014) 488 – 493
stationary evaluation unit was set up in the same way
(excluding ADC) and linked to a personal computer using
USB HiSpeed (Fig. 5). Data transmission between both
evaluation boards is carried out by facing the transceiver
module of each evaluation unit toward each other. In order to
investigate the field of application, a simulated sensor signal
with an excitation frequency of 4 kHz was generated and
applied to the ADC on the rotating part.
Fig. 4: Evaluation board of the rotating part
Fig. 5: Evaluation board of the stationary unit
In Fig. 6 the initial sensor signal (generated by a frequency
generator) and the start of ADC conversion is shown. A
maximum sensor sampling rate of 50 kHz can be achieved
with the system. Fig. 7 shows the recorded sensor signal on
the stationary unit. With this set-up, four 16-bit sensor signals
were recorded at a time, with a bandwidth of 4 kHz and an
oversampling factor of M = 6.25 (12.5 values per sine period).
The maximum error between the initial and the recorded
sensor signal was no more than 2.17 %.
3. Real-time monitoring system for cutting processes
The schematic design of a real-time monitoring system for
cutting processes is shown in Fig. 8. The sensor unit converts
and amplifies the desired process parameter into an analog
signal, while the data transmission can be carried out using
analog or digital data. When transmitting digital data, an ADC
and a microcontroller with an internal or external protocol
controller is used within the data transmission unit (cf. section
2). The monitoring unit is the main part of the real-time
system. Within the monitoring unit, the sensor signals are
conditioned (e.g. using filters) and pattern recognition is
carried out. Appropriate algorithms allow an activation of
process based and control based measures (e.g. rejection
marking, emergency stop, actuating elements). A separate
visualization unit is used for visualization purposes while the
actual measurement signal can be recorded via the monitoring
unit and the visualization unit. Thus, a computer based or
stand alone data logger can be used.
ADC conversions
Sensor signal
Fig. 6: Initial sensor signal (frequency generated) and ADC conversion
6
Sine (4kHz) CH0
5
4
Signal Voltage [V]
3
2 Aberration:
1 max. 2.17 %
0
-1
0,6204 0,6205 0,6206 0,6207 0,6208 0,6209
Time [s]
Fig. 7: Comparison of an ideal sine formed signal (4 kHz) and the recorded
signal on the stationary unit
In order to fulfill real-time criteria within the monitoring
system, the requirement for each unit (excluding the
visualization unit) is a deterministic operation. Therefore, it is
beneficial to disable the check sum strategies of the data
transmission protocol (low bit error rate required). This
reduces the reaction time and guarantees that required for the
real-time system. Furthermore, no operating system should be
implemented within the embedded hardware of all units. The
advantage of firmware over operating systems is direct access
to the hardware. Operating systems use an interrupting
 K. Dröder et al. / Procedia CIRP 14 (2014) 488 – 493 491

The calculation time of the initial Atmel microcontroller

(AVR32) is 11 ms, which significantly decreases the potential
data transmission rate. In order to reduce the calculation time
and therefore the reaction time of the real-time monitoring
system, the AVR32 microcontroller (MCU) was compared
with a STM32 microcontroller and a ARM11 (Raspberry PI)
microprocessor (MPU). The tested processing units differ
primarily in respect to the clock rate and the internal
architecture (Table 2).
Table 2: Characteristics of the tested processing units

Description Clock Rate Type CPU FPU DSP

Fig. 8: Schematic design of a real-time monitoring system
for cutting processes MCU
AVR32 66 MHz AVR32 yes no
(32-bit)
hardware management system, which would not allow a STM32 168 MHz
MCU Cortex
yes yes
guaranteed reaction time of the real-time system. (32-bit) M4
MPU
Raspberry Pi 700 MHz ARM11 yes no
3.1. Monitoring unit (32-bit)

CPU = Central Processing Unit FPU = Floating Point Unit

The reaction time between the detection of a critical value
and the activation of a specific measure depends on the
complexity of the calculation within the monitoring unit, the
sampling rate of the ADC and the data transmission rate.
Furthermore, high net data transmission rates require a high
payload to frame size ratio. Therefore, the infrared data
transmission was carried out with a frame size of 1544 Byte.
Each frame consists of an 8 Byte header and 128 data packets
with a size of 12 Byte. The data packets consist of four sensor
signals, the actual rotation angle and a time stamp. On this
basis, the infrared data transmission rate can be calculated to
4.83 Mbit/s, equating to a frame transmission period of 2.56
ms. Within this time, receival of the frame, conditioning of
the signals and pattern recognition must be carried out within
the monitoring unit. The tested calculation routine for signal
conditioning and pattern recognition was based on the
determination of the clamping and balance condition, as well
as the process force, of a planing tool mounted within a wood
machining spindle [3]. The required quantity and complexity
of the calculation for each frame is shown in Table 1.
DSP = Digital Signal Processing Unit
In Fig. 9, the calculation time is plotted against the clock
rate of the processing units. The interpolation between the
measurement data of the AVR32 microcontroller and the
Raspberry Pi microprocessor demonstrates the similarity of
the internal architecture. The STM32 microcontroller is
equipped with a DSP (digital signal processing) unit, which
allows calculating common routines (e.g. linear calibration,
low pass filter, multiplication) with a few clock cycles only.
Therefore, the STM32 architecture provides lower calculation
time in respect to the clock rate. For example, the Raspberry
Pi with a clock rate of 700 MHz is only twice as fast as the
STM32 with a clock rate of 168 MHz. Coupled with the more
complex programming routine of the Raspberry Pi (in bare
metal), the STM32 microcontroller was used for the
monitoring unit. The STM32 reduced the calculation time to
1.82 ms with a corresponding sampling rate of 42.5 kHz.

Table 1: Calculation routine

Description Quantity Complexity

Signal Calibration 512 32 bit = 16 bit x 32 bit + 32 bit
Compensation
(temperature, 1024 32 bit = 32 bit x 32 bit + 32 bit
rotational speed)
ଵ଴଴

Low Pass Filter 512 ͵ʹܾ݅‫ ݐ‬ൌ ෍ ͵ʹ„‹– ‫–‹„ʹ͵ כ‬

௜ୀ଴

Calculations 1536 32 bit = 32 bit x 32 bit

(clamping force,
bending moment,
process forces)
2688 32 bit = 32 bit + 32 bit
896 32 bit = 32 bit / 32 bit
128 32 bit = sin(32 bit) x 32 bit Fig. 9: Comparison of calculation time of two microcontrollers (AVR32,
STM32) and one microprocessor (Raspberry Pi)
128 32 bit = cos(32 bit) x 32 bit
128 32 bit = atan(32 bit) + 32 bit
3.2. Experimental setup of the real-time monitoring system
128 32 bit = asin(32 bit / 32 bit)
128 32 bit = sqrt(32 bit x The setup of the real-time monitoring system is shown in
32 bit x 32 bit) Fig. 10. It consists of the basic components of the infrared
data transmission unit but the microcontroller on the
492 K. Dröder et al. / Procedia CIRP 14 (2014) 488 – 493

stationary side is replaced by the more powerful STM32 this manner, the monitoring system can immediately react to a
microcontroller (ARM Cortex M4 processor). This allows the critical sensor value.
carrying out of all required calculations within the
microcontroller on the stationary unit (cf. Fig. 8). The
microcontroller is further connected to a personal computer;
however, by using the personal computer for visualization
purposes only, it is possible to meet the real-time criteria.
Process and control measures can directly be activated
through the microcontroller GPIOs.
In order to investigate a continuous, contact-less data
transmission from a rotating shaft to a stationary unit, the
monitoring system was built into a test stand. The test stand
primarily consisted of a rotating shaft and a slip ring (Fig. 10).
The rotational speed of the shaft can be adjusted up to 3,000
rpm. Three transceiver modules were mounted on the shaft
using a base body with the dimension of a commonly used
belt-driven spindle shaft (type: HSK F-80). The transceiver
modules were connected to the IPMS_IRHSP protocol
controller using a slip ring. On the stationary unit, a
Fig. 11: Sequences and their duration of the real-time monitoring system
transceiver module was located within a housing built of
aluminium sheets and with a cross section of a commonly
The reaction time of the monitoring system is longest when
used motor driven machining spindle. The stationary
the critical sensor value is sampled within the first of 128
evaluation board, with its USB connection for visualisation
packets. Due to this, the test procedure was configured to
purposes, was located at the end of the real-time monitoring
manipulate the value of the first packet via pressing a user
chain.
button on the rotating part. On the stationary unit, a GPIO was
activated once the manipulated value was detected. A
maximum reaction time of 7.14 ms could then be determined
by parallel monitoring of the user button and the GPIO (Fig.
12). The minimum reaction time (critical value within the
128th packet) is 4.92 ms. In comparison to wireless LAN
protocols (e.g. 802.11 b), the transmission time for a frame of
1500 Byte is approximately 5 ms [10]. Assuming the same
time for AD conversion and the calculations that follow
(compensation, calibration, filtering, process parameters), its
reaction time would be 9.58 ms.

max. Reaction time

Fig. 10: Test stand for the evaluation of the real-time monitoring system

3.3. Determination of the maximum data transmission rate

and reaction time

The reaction time in high speed monitoring systems is the

time from ADC sampling until GPIO activation. In order to
determine the reaction time of the infrared real-time
monitoring system, the transmission sequences and their
durations were measured using an oscilloscope (Fig. 11). In
the beginning, the analog sensor values are continuously
converted into digital characters. Once 128 packets have been
sampled, the whole frame is sent to the IPMS controller. A
Fig. 12: Determination of the maximum reaction time and the sampling rate
TFDU transceiver module transmits data on the rotating part,
of the real-time system
while a TFDU transceiver module within the stationary unit
receives and forwards the frame to the IrDA controller. Once
The data transmission rate was determined by monitoring
the frame has been sent to the stationary microcontroller, each
the CONVST pin (start of ADC conversion) on the rotating
sensor value within each packet is calibrated, compensated
part (Fig. 12, optical interference). In this manner, the final
and filtered before the actual calculations are carried out. In
maximum data transmission rate was calculated from the
 K. Dröder et al. / Procedia CIRP 14 (2014) 488 – 493
frame size (1544 Byte) and the frame transmission rate (42.5
kHz / 128 data packets) to be 4.1 Mbit/s.
4. Design concept for infrared data transmission
One advantage of wireless LAN applications compared to
infrared applications is the insensitivity to dust, chips and
cooling lubricants. It is therefore important to consider these
factors during the design stage of the implementation when
using infrared data transmission systems.
Because sensor-integrated tool holders are primarily
located within the working space of a machining center, it is
beneficial to implement the infrared data transmission unit
into the machining spindle itself. In order to support an
automatic tool changing system, machining spindle shafts
generally hold a clamping system within their center. Thus, in
most cases it is not possible to place the transceiver modules
on the center line of the spindle shaft (Fig. 13, top). Due to an
internal motor, the cross section of direct driven motor
spindles is larger than in belt-driven spindles. This allows the
IrDA transceiver modules to be built into the rotating spindle
shaft in a radial orientation (Fig. 13, bottom left). To achieve
maximum optical overlapping, the stationary transceiver
module is located in the corner of the housing (cf. Fig. 10).
In belt-driven spindles with a smaller cross section (low
optical overlapping in radial orientation), it is preferable to
arrange the IrDA transceiver modules with an axial offset at
the surface shell between both bearings (Fig. 13, bottom
right). Due to the large distance between the bearings
(compared to the radial implementation space), optical
overlapping can be increased.
Fig. 13: Concepts for the implementation of IrDA data transmission
transceivers into machining spindles
5. Conclusion
High performance cutting is characterized by a high cutting
speed and a high feed speed. This leads to higher excitation
493
frequencies, which have to be considered during the selection
of an appropriate monitoring system. Particuarly when
processing workpieces with a high weight (low
eigenfrequency), it is beneficial to use sensor-integrated
machining spindles or tool holders. In order to provide fast
and reliable data transmission, a real-time monitoring system
based on infrared data transmission was developed. The
contact-less and continuously transmitting unit is based on the
IrDA VFIR protocol. Furthermore, the microcontroller within
the stationary unit of the monitoring system was optimized to
carry out all required calculations (calibration, compensation,
filter, pattern recognition). The whole system has a data
transmission rate of 4.1 Mbit/s, with a maximum reaction time
of 7.14 ms. Additionally, a sampling rate of four sensor
signals of 42.5 kHz was achieved.
Acknowledgements
The authors would like to thank the Fraunhofer Institute
for Photonic Microsystems (Dresden, Germany) for their
contribution to the presented work.
References
[1] Schildknecht, T.: Ende der Fabrikautomation mit 2,4-GHz-WLAN-
Funktechnologie?; In: open automation; 24.03.2011; http://www.
openautomation.de/2135-0-ende-der-fabrikautomation-mit-24-ghz-
wlan-funktechnologie.html (date of access: 08.10.2013)
[2] Happacher, M.: Das Deterministik-Dilemma; In: computer-automation;
04.07.2012; http://www.computer-automation.de/steuerungsebene/
fernwirken/fachwissen/article/89296/ (date of access: 08.10.2013)
[3] Hoffmeister, H.-W.; Luig, M.; Biehl, S.; Rumposch, C.: Sensor-
integrated machining spindle using thin film layers; 21st International
Wood Machining Seminar; Tsukuba; Japan; The Japan Wood Research
Society, p. 104-112; 2013.
[4] Feucht, W.: Funklösungen für die Prozessindustrie – Die Risiken im
Blick behalten; In: prozesstechnik online; 14.06.2013;
http://www.prozesstechnik-online.de/chemie/-/article/31534493/
38318703/Die-Risiken-im-Blick-behalten/art_co_INSTANCE_0000
/maximized/ (data of access: 19.11.2013)
[5] ETSI: ETSI EN 300 328 V 1.8.1 – Data Transmission Equipment
Operating in the 2,4 GHz ISM band using wide band modulation
techniques; 04.2012; http://www.etsi.org/deliver/etsi_en/300300_
300399/300328/01.08.01_30/en_300328v010801v.pdf (date of access:
19.11.2013)
[6] Latuske, R.: Bluetooth, ZigBee und IrDA – Vergleich und
Industrieanwendungen; www.ars2000.com/Drahtlose-Netzwerke.pdf
(data of access: 29.10.2013).
[7] Software Technologies Group: How does ZigBee compare with other
wireless standards?; www.stg.com/wireless/ZigBee_comp.html (data of
access: 29.10.2013).
[8] Weck, M.; Brecher, C.: Werkzeugmaschinen 3 – Mechatronische
Systeme, Vorschubantriebe, Prozessdiagnose; Vol. 6; Berlin,
Heidelberg, New York; Springer; 2006.
[9] Scholles, M.; Deicke, F.: IPMS_IRHSP – IrDA Controller; Fraunhofer
Institute for Photonic Microsystems; http://www.ipms.fraunhofer.de/
content/dam/ipms/common/products/WMS/Irda/irhsp03-e.pdf (date of
access: 20.11.2013)
[10] Lüders, C.; Winkler, M.: Pingpong – Wie das TCP/IP-Flusskontrolle
das Surf-Tempo bestimmt; c’t; Hannover; 23, p. 198-200, 2006.
[11] Scholles, M.; Deicke, F.: GigaIR Controller Core; Fraunhofer Institute
for Photonic Microsystems; http://www.ipms.fraunhofer.de/content
/dam/ipms/common/products/WMS/Irda/giga-ir-ctrl-e.pdf (date of
access: 05.03.2014)
[12] Val, T.; Peyrard, F.; Mission, M.: Study and simulation of infrared
WLAN IrDA: an alternative to the radio; Computer Communications;
26; p. 1210-1218, 2003.