electronics

Article
A Simple and Efficient Time-Sensitive Networking
Traffic Scheduling Method for Industrial Scenarios
Qing Li 1,2,3,4 , Dong Li 1,2,3, *, Xi Jin 1,2,3 , Qizhao Wang 1,2,3,4 and Peng Zeng 1,2,3
1 State Key Laboratory of Robotics, Shenyang Institute of Automation, Chinese Academy of Sciences,
Shenyang 110016, China; liqing@sia.cn (Q.L.); jinxi@sia.cn (X.J.); wangqizhao@sia.cn (Q.W.); zp@sia.cn (P.Z.)
2 Key Laboratory of Networked Control Systems, Chinese Academy of Sciences, Shenyang 110016, China
3 Institutes for Robotics and Intelligent Manufacturing, Chinese Academy of Sciences, Shenyang 110169, China
4 University of Chinese Academy of Sciences, Beijing 100049, China
* Correspondence: lidong@sia.cn




Received: 11 November 2020; Accepted: 9 December 2020; Published: 12 December 2020


Abstract: Time-Sensitive Networking (TSN) provides end-to-end data transmission with extremely
low delay and high reliability on the basis of Ethernet. It is suitable for time-sensitive applications and
will be widely used in scenarios such as autonomous driving and industrial Internet. IEEE 802.1Qbv
proposes a time-aware shaper mechanism, which enables switches to control the forwarding of traffic
in port queues according to pre-defined Gate Control List (GCL). The length of the GCL is limited, and
the previous method of scheduling cycle with a hyper period may result in a larger GCL. Based on
Satisfiability Modulo Theories (SMT), we propose a TSN scheduling method for industrial scenarios
and develops a series of scheduling constraints. Different from the previous scheduling methods,
the method proposed in this paper adopts the base period cycle to update GCL regularly, which
can effectively reduce the number of time slots in GCL and make the configuration of GCL simpler
and more efficient. In addition, compared with the traditional hyper period method, the method
proposed in this paper can calculate the scheduling results faster while ensuring low latency and
reducing the runtime effectively.

Keywords: Time-Sensitive Networking (TSN); Satisfiability Modulo Theories (SMT); scheduling;

802.1Qbv

1. Introduction
With the continuous popularization of the Industrial Internet of Things, increasingly intelligent
terminal equipment and infrastructure have formed a super-intelligent system group, and the
time-sensitive data (such as control data, fault monitoring data, etc.) generated by these systems must be
transmitted within strict time and reliable regulations [1]. The original standard Ethernet IEEE802.3 [2]
satisfies the bandwidth requirements of a wide range of application areas while maintaining scalability
and cost-effectiveness, but it is not suitable for real-time and safety-critical applications. On the other
hand, a variety of industrial Ethernet standards coexist on the market today, and the industrial Ethernet
protocols used in different industrial equipment vary from equipment supplier, so the compatibility
between various field bus is another big problem. Time-Sensitive Networking (TSN) just solves
these challenges. TSN has the following key characteristics: it provides microsecond-level precise
clock synchronization, such as 802.1AS [3], and provides multiple delay control mechanisms for
time-sensitive traffic, such as 802.1Qbv [4]. With these features, TSN can provide QoS guarantees for
the traffic it carries, and ensure real-time and deterministic data transmission. Due to the complexity
of industrial communication and automation, more exploration is needed in the practical application
of TSN. Researching and improving the performance of TSN is of great significance to the future

Electronics 2020, 9, 2131; doi:10.3390/electronics9122131 www.mdpi.com/journal/electronics

Electronics 2020, 9, x FOR PEER REVIEW 2 of 19
Electronics 2020, 9, 2131 2 of 19
transmission. Due to the complexity of industrial communication and automation, more exploration
is needed in the practical application of TSN. Researching and improving the performance of TSN is
development of industrial
of great significance communication
to the future development andofautomation control. The scheduling
industrial communication algorithm
and automation is a
control.
very important part of the network. The efficient work of the scheduling
The scheduling algorithm is a very important part of the network. The efficient work of the algorithm is closely related
to the timely
scheduling and reliably
algorithm transmission
is closely related toofthedata. Theand
timely problem
reliablyoftransmission
scheduling in ofTSN
data.networks
The problemis a
multi-objective combinatorial optimization problem, which has been proved
of scheduling in TSN networks is a multi-objective combinatorial optimization problem, which has to be NP-complete in
previous work [5].
been proved to be NP-complete in previous work [5].
The
The scheduling
scheduling in in this
this article
article is
is based
based onon the
the switch
switch with
with 802.1Qbv
802.1Qbv function,
function, and
and all
all devices
devices inin
the
the network are time-synchronized. Figure 1 describes the simplified logic of the switch port
network are time-synchronized. Figure 1 describes the simplified logic of the switch port with
with
802.1Qbv
802.1Qbv function.
function.Each Eachswitch
switch egress
egressportport
has has
8 priority queues,
8 priority and theand
queues, arriving flow enters
the arriving flowdifferent
enters
queues according to the priority. Each queue has a logical switch called GATE
different queues according to the priority. Each queue has a logical switch called GATE (G in Figure(G in Figure 1). When
the
1). value
When isthe1, itvalue
meansisthat themeans
1, it GATE thatof thethe
queue
GATE is open,
of theand the flow
queue in the corresponding
is open, and the flow in queue
the
can be transmitted,
corresponding queue and 0 means
can it is closed.
be transmitted, andGate Control
0 means it isList (GCL)
closed. GatehasControl
a cycleList
period,
(GCL)so has
GATE will
a cycle
be opened/closed
period, so GATE will cyclically according tocyclically
be opened/closed this period.according to this period.

GCL …
1 0 0 0 1 …
0 0 1 0 0 …
…

…
…
…
…
…
…
0 1 0 0 0 …

1 1 0 0 0 0 0 …
Port A 2 2 G
(ingress)
1 1 G Port C 2 2
(egress)
2 2 Port B
(ingress)
G

Figure 1. Simplified view of an 802.1Qbv-capable switch.

This research focuses on the problem of Time-Triggered (TT) (TT) traffic

traffic scheduling
scheduling in TSN, and
proposes
proposes a network architecture
architecture that can dynamically manage TSN networks for industrial industrial scenarios,
scenarios,
TSN switch and the terminal system
terminal system with time synchronization
time synchronization function can provide a real-time
guarantee for the TT flow. flow. Based on Satisfiability
Satisfiability Modulo Theories (SMT), a TSN TSN static
static scheduling
scheduling
algorithm for industrial scenarios is proposed, and a series of scheduling constraints constraints areare formulated.
formulated.
The previous researches on traffic traffic scheduling
scheduling based on 802.1Qbv 802.1Qbv have adopted hyper period for
scheduling cycles.
cycles. One
Oneofofthe
thebiggest
biggest problems
problems of of hyper
hyper period
period is that
is that the the
sizesize of hyper
of hyper period
period is
is very
very large, and an oversized hyper period results in a flow with a smaller period
large, and an oversized hyper period results in a flow with a smaller period being scheduled multiple being scheduled
multiple
times pertimes
cycle,per
so cycle,
many so many
time slotstime
are slots are required
required for scheduling
for scheduling in eachLarger
in each cycle. cycle. length
Largeroflength
GCL
of GCLrequires
tables tables requires high-performance
high-performance hardware hardware
support,support,
and most and most TSN
current current TSN switches
switches cannot
cannot support
support larger GCLs.
larger GCLs. Different
Different from from the previous
the previous 802.1Qbv-based
802.1Qbv-based schedulingmethods,
scheduling methods,the the algorithm
proposed in this paper uses the base period method to update GCL regularly, which can effectively
reduce
reduce the
the number
numberofoftimetimeslot
slotentries
entries ininthethe
GCLGCL and make
and thethe
make configuration
configuration of the
of GCL simpler
the GCL and
simpler
more efficient.
and more Additionally,
efficient. compared
Additionally, with thewith
compared traditional ways of using
the traditional wayshyper period,
of using hyperthe period,
schedulingthe
method
scheduling proposed
methodinproposed
this paperincan thiscalculate
paper can thecalculate
scheduling the result faster result
scheduling while faster
ensuring lowensuring
while latency
and
low effectively
latency andreducing thereducing
effectively runtime. the runtime.

2. Related Work
2. Related Work
Researches
Researches on
onSwitched
SwitchedEthernet hashas
Ethernet mademadeprogress in recent
progress years,years,
in recent such as Avionics
such full-duplex
as Avionics full-
Switched Etherne (AFDX) [6], TTEthernet [7], or TSN. Some articles have studied the time-predictability
duplex Switched Etherne (AFDX) [6], TTEthernet [7], or TSN. Some articles have studied the time-
predictability of the AFDX network, and analyzed some methods to realize the predictability of
Electronics 2020, 9, 2131 3 of 19

of the AFDX network, and analyzed some methods to realize the predictability of switched Ethernet [8,9].
However, the switches and end systems in the AFDX network are not synchronized with the global clock.
This paper only focuses on the transmission deterministic problem of TSN network. Many
scholars have conducted different studies in this field. The work in [10] proposes an approach based
on a Greedy Randomized Adaptive Search Program (GRASP) metaheuristic to solve the scheduling
problem. The work in [11] formulates the scheduling constraints for IEEE 802.1Qbv using the first-order
theory of arrays and solves the system’s satisfiability through the SMT solver. An SMT method is used
to minimize the number of queues used for scheduling traffic, and to ensure that the remaining queues
can be used for non-scheduled traffic in [12]. The work in [13] considers Audio-Video-Bridging (AVB)
traffic when scheduling TT traffic, which uses Time-Aware Shaper (TAS) and Credit-Based-Shaper
(CBS) scheduling mechanism, and combines routing and scheduling. The work in [14] proposes an
iterative method based on SMT that combines routing and scheduling. The work in [15] based on SMT
and 802.1Qbv presents a tool “TSNsched” to generate schedule. The above articles all use different
scheduling algorithms, but they all use hyper period cycle (HPC) as the scheduling cycle. Therefore, in
Section 5, we mainly compared the base period cycle (BPC) proposed in this paper as the scheduling
cycle with HPC method.
For the scheduling of large-scale TSN, the work in [16] proposed a segmented approach, which splits
the scheduling problem into smaller problems that can be solved independently. While maintaining
the scheduling quality, it reduces the running time of the scheduler and can schedule a larger network
in a shorter time. The work in [17] relaxes the scheduling rules, and divides the SMT problem into
multiple Optimization Modulo Theories (OMT) problems, and reduces the runtime of the solvers to an
acceptable level, and then a fast heuristic algorithm is proposed, which combines GCL and controls the
time of packet injection to eliminate scheduling conflicts.
By using the logically centralized paradigm of Software-Defined Networks (SDN), given a set
of predefined time-triggered flows, the work in [18] is the first to use Integer Linear Programming
(ILP) to formulate the integration of routing and scheduling. Subsequently, the same author proposed
incremental flow scheduling and routing algorithm in the article [19], which can dynamically add or
delete flows by scheduling one at a time. However, the author did not use the time-aware shaper
proposed in 802.1Qbv for gate control scheduling. The author proposed a concept of base period,
and then divided the base period into multiple time slots, each time slot size is enough to make an
MTU size data packets traverse the entire network, and each scheduled flow is allocated a time slot
for scheduling. One problem with this is that the time slot reserved for each flow is too large, which
causes a lot of bandwidth waste.

3. System Model

3.1. Overall Network Architecture

To meet the requirements of real-time reliability and flexible reconfiguration of the network
for edge computing services in factories, this paper proposes an in-factory network architecture for
industrial internet based on TSN technologies. The network controller is logically centralized, that is,
it has a global view in switches, topology, and traffic, which is conducive to the realization of network
control logic such as routing and scheduling. The architecture is divided into three layers, named the
field layer, data layer, and application layer, as shown in Figure 2.
Electronics 2020, 9, 2131 4 of 19
Electronics 2020, 9, x FOR PEER REVIEW 4 of 19

Application QoS QoS QoS QoS QoS QoS

Application Application Applicat ion Applicat ion Application Application
layer

Northbound Interface

TSN Controller

Data layer Southbound Interface

TSN Switch

Gateway Gateway
Gateway
Edge s erver Edge s erver
PLC
PLC
PLC
Field layer

Industrial fi eld equipment Industrial field equipment Industrial field equipment

Figure 2. Time-sensitive
Figure 2. Time-sensitive industrial
industrial architecture.
architecture.

(1)
(1) The
The field
field layer
layer includes
includes twotwo parts:
parts: the
the traditional
traditional industrial
industrial production
production network
network and and edge
edge
computing
computing service
service equipment.
equipment. The The traditional
traditional industrial
industrial production
production network
network isis composed
composed of of
industrial field equipment, including industrial control machine, intelligent
industrial field equipment, including industrial control machine, intelligent terminals, robots,terminals, robots,
programmable
programmable logic logic devices,
devices, numerical
numerical control
control machine
machine tool,
tool, etc.
etc. Edge
Edge computing
computing service
service
equipment
equipment includes
includesedge edgeservers
serversthatthat
process useruser
process requests and edge
requests anddevices that seekthat
edge devices for services.
seek for
(2) The data layer is composed of TSN switches and TSN controllers. TSN switches are used for
services.
(2) data forwarding.
The data The controller
layer is composed of TSNisswitches
responsibleandfor
TSN making the forwarding
controllers. TSN switchesrules
areofused
the for
switch,
data
calculating TSN switch traffic scheduling table and generating the bandwidth
forwarding. The controller is responsible for making the forwarding rules of the switch, allocation strategies.
(3) The application
calculating TSNlayer contains
switch trafficapplications
scheduling with different
table and QoS requirements,
generating and the applications
the bandwidth allocation
in the application layer send requests to the TSN controller. Then, the TSN controller performs
strategies.
(3) calculations,
The application layer TSN
configures contains applications
scheduling with rules,
and routing different QoSsends
and then requirements,
them to theand the
switch,
applications
and then to the in industrial
the application layer send requests to the TSN controller. Then, the TSN
field layer.
controller performs calculations, configures TSN scheduling and routing rules, and then sends
The
themscheduling
to the switch, algorithm
and then intothis
thepaper runsfield
industrial in the TSN controller for time-sensitive traffic
layer.
scheduling on the data layer.
The scheduling algorithm in this paper runs in the TSN controller for time-sensitive traffic
scheduling
3.2. on theand
Terminologies data layer.
Notations
Figure 3 shows a simple TSN network architecture model. The architecture model including end
3.2. Terminologies and Notations
systems, switches, and links. We model the topology of the TSN network as G = (E, V ). The vertex set
Figure
V refers to all3 devices
shows ainsimple TSN network
the network, architecture
including model.set
all end system The
ESarchitecture model
and all switch including
set SW. end
Therefore,
Vsystems,
= ES ∪switches,
SW. E refersand to
links. Weof
the set model thelinks
all data topology
in theofnetwork.
the TSN network as 𝐺 edge
The directed = (𝐸,from
𝑉). The
va ∈vertex
V to
vset
b ∈ V
V refers to
represents all
a devices in the
unidirectional network, including
communication linkall
fromendv asystem
to vb . set ES and
Therefore, all
the switch set
full-duplex SW.
link
Therefore, 𝑉 = ES ∪ 𝑆𝑊. 𝐸 refers to the set of all data links in the network. The directed
between the devices va and vb is represented as two edges, defined as [va , vb ] ∈ E and [vb , va ] ∈ E. edge from
𝑣 ∈notations
The 𝑉 to 𝑣 ∈used 𝑉 represents a unidirectional
in this section in Table A1.link from 𝑣 to 𝑣 . Therefore, the full-
communication
are all described
duplex link between the devices 𝑣 and 𝑣 is represented as two edges, defined as [𝑣 , 𝑣 ] ∈ 𝐸 and
[𝑣 , 𝑣 ] ∈ 𝐸. The notations used in this section are all described in Table A1.
Electronics 2020, 9, 2131 5 of 19
Electronics 2020, 9, x FOR PEER REVIEW 5 of 19

𝐸𝑆 𝐸𝑆

𝐸𝑆 𝑆𝑊 𝑆𝑊 𝐸𝑆

𝐸𝑆 𝐸𝑆

Figure 3. Time-Sensitive
Time-Sensitive Networking (TSN) network architecture model.

The entire
entirearticle
articleuses usesobject-oriented
object-oriented notationnotation to refer to torefer
specific attributes,
to specific e.g., [va , vE.g.,
attributes, [𝑣 , 𝑣 ]. 𝜃
b ]·θ represents
represents the transmission rate aof bthe link [𝑣 a, 𝑣 b]. We use [𝑣 , 𝑣 ]. 𝑐 to refer to the number
the transmission rate of the link [ v , v ] . We use [ v , v ] ·c to refer to the number of queues in the egress
of
port
queuesconnected
in the egress to vb inportva .connected to 𝑣 in 𝑣 .
We denote the set of flows in the network with S, 𝑆, flow s𝑠i ∈∈S. 𝑆. AA flow
flow can
can bebe described
described by the
tuple< 𝑇, 𝐷, 𝑟 >, denoting the period, the maximum allowed end-to-end latency (deadline)
tuplehT, D, ri, denoting the period, the maximum allowed end-to-end latency (deadline) and theandroute
the
of a flow,
route of arespectively.
flow, respectively. DuringDuring the entire thetransmission
entire transmission process,process,
the receiving end system
the receiving end must
system receive
must
the message
receive within siwithin
the message ·D. We use 𝑠 . 𝐷.v1We . . . , v𝑣n−1
, v2 , use , 𝑣, v, n…to, 𝑣represent
, 𝑣 to the nodes passing
represent the nodes through
passing thethrough
routing
path of si , then
the routing pathsi ·rof=𝑠([,vthen
1 , v2 ],𝑠[v. 2𝑟, = , . . .,,𝑣[v],
v3 ]([𝑣 [𝑣, v, n𝑣])],. …
n−1 , [𝑣 the, 𝑣research
Since ]). Since onthe
routing
research has been done very
on routing has
well
been [20],
donewe verydidwellnot propose
[20], we any did routing
not propose algorithm.
any routing We stipulate that We
algorithm. routing is generated
stipulate that routingon theis
basis of some existing algorithms.
generated on the basis of some existing algorithms.
We use p𝑝i,k, totorefer D totothe
refer thedata
Edatapacketpacketsent sentby bythe theflowflowsi𝑠inin the
thek-th𝑘-th
period
periodsi ·T.𝑠 .A
𝑇.packet
A packet cancan
be
described
be described by theby tuple ρ, φ, d,𝜌,L𝜙,, 𝑑,
the tuple< 𝐿 >, denoting
denoting the packet the size
packet in bytes, the
size in send the
bytes, time, the time,
send end-to-end delay
the end-to-
and
end the
delay transmission duration of duration
and the transmission a packet, respectively. The transmission
of a packet, respectively. duration of the
The transmission packet on
duration of the
port
packetpi,kon the port 𝑝 , [ to, packet
[va ,vb ] ·L is related ]
. 𝐿 is related
size andtotransmission
packet size and rate.transmission
Suppose thatrate.the size of a data
Suppose thatpacket
the sizeis
aofmaximum transmission unit MTU (1500 bytes) and the
a data packet is a maximum transmission unit MTU (1500 bytes) and the total overhead is 42 bytes,total overhead is 42 bytes, and the size of the
entire
and the packet
size ofisthe 1542 bytes.
entire Thenisthe
packet 1542transmission
bytes. Thenduration of packetduration
the transmission sent to 1 ofGbps
packet linksent
on the
to 1port
Gbps is
calculated
link on thein portEquation (1). in Equation (1).
is calculated
𝑝 ,[v[ ,v , ] ] . 𝜌 1542 𝑏𝑦𝑡𝑒𝑠
𝑝 ,[va[ ,vb,] ] . 𝐿 = pi,k a b ·ρ =1542 bytes = 12.336 𝜇𝑠 (1)
pi,k ·L = transmission rate= 1 𝐺𝑏𝑝𝑠 = 12.336 µs (1)
transmission rate 1 Gbps
The time period during which the queue gate in the egress port is opened and closed for each
The time period
flow is called a time duringslot window, which the queue
a time slotgate windowin the can egressbe port is opened
described by theandtuple<
closed𝜏,for 𝜔, each
𝑄 >,
flow is called a time slot window, a time slot window can be
denoting the slot start time, the slot end time, and the queue ID of the slot, respectively. In port described by the tuplehτ, ω, Qi, denoting
the
[𝑣 , slot
𝑣 ], start time,
the slot windowthe slot end time,
opening time and
of the
flowqueue ID of the slot,
𝑠 is expressed as 𝑠 respectively.
[ , ] In port
. 𝜏, and the [va , vtime
closing b ], the
of
[ v ,v ]
slot window opening time of [ , s]i is expressed
flow [ , ] as s a b ·τ, and the closing time of slot window is
slot window is expressed as 𝑠 . 𝜔. 𝑠 . 𝑄 indicates in which queue the flow will be queued,
i
expressed as s [va ,vb ] ·ω. s [va ,vb ] ·Q indicates in which queue the flow will be queued, so the upper limit of
so the upper limiti of thisi value is the number of queues of the port [𝑣 , 𝑣 ]. 𝑐. In this paper, the size
this
of the time slot window of
value is the number will queues
be taken of the
as port
input [vaand , vb ]·c.
setIn tothis paper,
a fixed the size
value. Users of can
the time
defineslot
thewindow
size of
will be taken
the time as input
slot by and set to a fixed value. Users can define the size of the time slot by themselves.
themselves.
The
The moment
momentthe thepacket
packet is received
is received by the byreceiver
the receiveris denoted by β. Then
is denoted by the𝛽 . time
Thenstamp calculation
the time stamp
formula of the packet received by the receiver
calculation formula of the packet received by the receiver is as follows: is as follows:

. 𝛽==p𝑝i,k,[v[n−1 ,vn, ] ·φ] . 𝜙

p𝑝i,k, ·β ++ [ , ]
pi,k𝑝[v, n−1 ,vn ] ·L +
. 𝐿1+ 1 (2)
The end-to-end delay of the 𝑘-th packet of flow 𝑠 is 𝑝 . 𝑑, which is calculated according to
The end-to-end delay of the k-th packet of flow si is pi,k, ·d, which is calculated according to
Equation (3).
Equation (3).
p𝑝i,k, ·d. 𝑑==p𝑝 𝛽 p−i,k𝑝[v,1 ,v[ 2 ],·φ] . 𝜙
i,k ,·β. − (3)

Because the lines between nodes in industrial scenarios are short, the link propagation delay
between nodes
between nodesininmost
mostscenarios
scenariosis is negligible.
negligible. In In
thisthis paper,
paper, the the
linklink propagation
propagation delaydelay
is setistoset to a
a fixed
fixed value of 1μs in the scheduling calculation.
value of 1µs in the scheduling calculation.
Electronics 2020, 9, x FOR PEER REVIEW 6 of 19
Electronics 2020, 9, 2131 6 of 19
3.3. Base Period
The Period
3.3. Base GCLs of all egress ports constitute a definite timetable, and the switch forwards packets
according to the timetable. Due to the periodicity of the flows, GCLs have a certain cycle time, after
Thecycle,
the first GCLsthey of allwill egress ports the
start from constitute
beginning. a definite timetable,
In all previous andscheduling
traffic the switch methods
forwardsbased packets on
according to the timetable. Due to the periodicity of the
the 802.1Qbv, GCL is cycled according to the hyper period. One of the biggest problems of hyper flows, GCLs have a certain cycle time, after
the
periodfirstiscycle,
that the they size will ofstart
hyper fromperiod the beginning.
is very large. InThen
all previous traffic scheduling
in a scheduling methods based
cycle, a particularly large
on the 802.1Qbv, GCL is cycled according to the hyper period.
number of flows must be scheduled. So, the GCL is relatively large and there are many scheduling One of the biggest problems of hyper
period
entries,iswhichthat the size of the
increases hyper period isof
complexity very large. Then
scheduling. Naresh in a scheduling
Ganesh Nayak cycle, a particularly
proposed large
the concept
number
of base periodof flows formust
the first be scheduled.
time [18], this So,article
the GCL is relatively
defines that the large and there
transmission are many
period of allscheduling
flows is an
entries,
integer multiple of the base period, and the base period is limited to be less than or equal to concept
which increases the complexity of scheduling. Naresh Ganesh Nayak proposed the the lowest of
base period for the first time [18], this article defines
transmission period that a flow in the network may have. Inspired by Naresh Ganesh Nayak, ourthat the transmission period of all flows is an
integer
transmissionmultiple of the base
scheduling wasperiod,
modeled and as thea base
cycleperiod is limited
scheduling withtoa be less than
duration of or
baseequal to the
period, thelowest
GCL
transmission
of each port cycled according to the base period. Therefore, in our scheduling algorithm, we use our
period that a flow in the network may have. Inspired by Naresh Ganesh Nayak, the
transmission
idea of the base scheduling
periodwas to modeled
carry out as scheduling
a cycle scheduling analysis with a duration
based on theof802.1Qbv
base period, the GCL
scheduling
of each port cycled
mechanism. according
Additionally, wetodefine
the base the period.
base periodTherefore,
as theingreatest
our scheduling
commonalgorithm,
divisor (GCD) we use ofthe
all
idea of the base period to carry out scheduling analysis based
periods of the flows and is represented by symbol BP. The base period is divided into multiple time on the 802.1Qbv scheduling mechanism.
Additionally,
slots, and thewenumber define the andbase sizeperiod
of time as theslotsgreatest
in each common divisor are
base period (GCD) the ofsame.
all periods
We set of the
the
flows and is represented by symbol BP. The base period is
opening/closing of the time slot in each base period to be the same, and GCL is configured according divided into multiple time slots, and the
number
to the base andperiod.
size of time All data slotspackets
in each of base
theperiod
same floware the insame.
the same We set the opening/closing
switch egress port haveofthe thesame
time
slot in each
priority andbase enterperiod
the same to bequeue,
the same, so it and
is onlyGCL is configured
necessary according
to determine thetotimethe slot
baseofperiod.
the firstAll data
packet
packets of
of each flow. the same flow in the same switch egress port have the same priority and enter the same
queue, so it with
Flows is only thenecessary
same period to determine
(the period 𝑥 ∙ 𝐵𝑃,
theistime slot𝑥of>the 1) can
firstuse
packet of each
the same flow.
time slot for different
Flows with the same period (the period is x·BP,
base period. Flows with different periods and the flow with a period of BP cannot share the x > 1) can use the same time slot for time
different
slot,
base period. Flows with different periods and the flow
because flows of different periods use same time slots will cause time slot conflicts in some with a period of BP cannot share the time slot,
because
subsequent flows baseof different
period. Figure periods4 use same time
describes slots willofcause
an example cycletime slot conflicts
scheduling usinginbasesome subsequent
period. There
are three flows, the periods of 𝑓𝑙𝑜𝑤 and 𝑓𝑙𝑜𝑤 are both 2 ∙ 𝐵𝑃, and the period of 𝑓𝑙𝑜𝑤 is 3three
base period. Figure 4 describes an example of cycle scheduling using base period. There are ∙ 𝐵𝑃,
𝑚 andthe
flows, 𝑛 periods
represent f low1 andtime
of different f lowslot
3 arein both and the𝐹𝑙𝑜𝑤
2·BP,period.
a base periodand of f 𝑓𝑙𝑜𝑤
low2 is can3·BP,usem and
same n represent
time slot
different
alternately time in slot in a base
different base period.
period, Flowe.g., 𝑓𝑙𝑜𝑤
1 and f lowis3 scheduled
can use same time slot
in time 𝑛 in BP 1,in
slot alternately and 𝑓𝑙𝑜𝑤 base
different can
period, e.g.,
be scheduled in time f low 1 is slot 𝑛 in BP 2. No flow in this example has3 the same period as 𝑓𝑙𝑜𝑤 , and 𝑓𝑙𝑜𝑤
scheduled in time slot n in BP 1, and f low can be scheduled in time slot n in BP
2. No flow ininthis
is scheduled 𝑚, so slot
slotexample has𝑚the samebe
cannot period
used as 𝑓𝑙𝑜𝑤
forf low 2 , and 𝑓𝑙𝑜𝑤
andf low 2 is scheduled
in any basein slot m,which
period, so slotcan m
be used if the period of other flows is the same as the period of 𝑓𝑙𝑜𝑤 . The opening/closing of the
cannot be used for f low 1 and f low 3 in any base period, which can be used if the period of other flows
is theslot
time same in as
each thebase
period of f low
period is to2 . be
The theopening/closing
same, so the GCL of the onlytime slot to
needs in be
each base period
configured is to be the
in accordance
same,
with the so gray
the GCL partonlyin Figureneeds4.to be configured in accordance with the gray part in Figure 4.

Transmission slot of

Transmission slot of
Time slot
Transmission slot of

n m n m n m n m … n m

BP 1 BP 2 BP 3 BP 4

Base-period Cycle of GCL

Unused time slot
Figure 4. Example of base period.
Figure 4. Example of base period.
4. Scheduling Method Based on Satisfiability Modulus Theories
4. Scheduling Method Based on Satisfiability Modulus Theories
Satisfiability Modulo Theories (SMT) solver is a tool that can find the satisfiability or unsatisfiability
Satisfiability
of a set of constraintsModulo
and areTheories
commonly (SMT) usedsolver is a toolproblems
in verifiability that can[21],
find
butthe
the satisfiability
performance or of
unsatisfiability
SMT solvers hasof a set of constraints
improved significantlyand arepast
in the commonly
few years,used in verifiability
making problems
them applicable [21],complex
to more but the
performance
problems, suchofasSMT solversproblems.
scheduling has improved significantly
If the network in the past for
is not schedulable fewa given
years,setmaking
of flowsthem
and
applicable to more complex problems, such as scheduling problems. If the network
constraints, the SMT solver will return unsatisfiability. The idea of using SMT solvers to is not schedulable
search for
for a given set of flows and constraints, the SMT solver will return unsatisfiability. The idea of using
Electronics 2020,
Electronics 2020, 9,
9, 2131
x FOR PEER REVIEW 7 of 19

SMT solvers to search for distributed system scheduling was first proposed by Steiner [22]. The
distributed
disadvantage system scheduling
of using was first
SMT solvers proposed
is that, by Steiner
for large networks,[22].itThe
maydisadvantage
take a long of using
time SMT
to solve
solvers is that, for
scheduling problems. large networks, it may take a long time to solve scheduling problems.
The
The entire
entire workflow
workflow ofof the
the scheduling
scheduling algorithm
algorithm in
in this
this paper
paper isis described
described in
in Figure
Figure5.
5.

Start

Input: topology information,

flow characteristics

Obtain first-hop switch resources

Execution scheduling constraints

Is there a free N
Obtain next-hop
time slot
switch resources
Y

Scheduling failed
N Whether to traverse
all switches

Y
Scheduling success

End

Figure 5. Workflow of Satisfiability Modulo Theories

Theories (SMT)
(SMT) algorithm.
algorithm.

Scheduling Constraints
Scheduling Constraints
(1) Periodically send constraint. All packets must be sent periodically according to the period
(1) Periodically send constraint. All packets must be sent periodically according to the period 𝑠 . 𝑇
si ·T of the flow, which means that the flow si will generate a data packet at every interval si ·T at the
of the flow, which means that the flow 𝑠 will generate a data packet at every interval 𝑠 . 𝑇 at the
sending terminal, and the sending terminal must send the packet before the next packet is generated.
sending terminal, and the sending terminal must send the packet before the next packet is generated.
The sending constraint of the sender is as follows:
The sending constraint of the sender is as follows:
∀𝑝
∀pi,k, ∈∈ s𝑠i ;;∀k
∀𝑘∈∈N;
𝑁;

si ·T ∗ (k − 1) ≤ pi,k [v1 ,v2 ] ·φ ≤ si ·T ∗ k (4)

𝑠 . 𝑇 ∗ (𝑘 − 1) ≤ 𝑝 , [[v1 ,v, ]
.𝜙 ≤ 𝑠 .𝑇 ∗ 𝑘 (4)
pi,k+1 [v1 ,v2 ] ·φ = pi,k 2 ] ·φ + s ·T
i (5)
[ , ]
𝑝 , sends
The data sending port periodically 𝜙 = 𝑝 , [so, after
.packets, ]
.𝜙 +
the𝑠 sending
.𝑇 time of the first packet p(5)
i,1
of the flow si is determined, the sending time of the subsequent generated packets is also determined.
The data sending port periodically sends packets, so after the sending time of the first packet
(2) End-to-end delay constraint. The deadline si ·D of each flow is determined according to the
𝑝 , of the flow 𝑠 is determined, the sending time of the subsequent generated packets is also
specific requirements of the scenario, and the packet of flow si must be fully received by the receiving
determined.
terminal within the cut-off time si ·D. Define the constraint as follows:
(2) End-to-end delay constraint. The deadline 𝑠 . 𝐷 of each flow is determined according to the
∀p ∈ si ; ∀k ∈ N;
specifici,krequirements of the scenario, and [the packet of flow 𝑠 must be fully received by the
pi,k va ,vb ] ·φ < pi,k ·β (6)
receiving terminal within the cut-off time 𝑠 . 𝐷. Define the constraint as follows:
∀𝑝 , ∈ 𝑠 ; ∀𝑘 ∈ 𝑁; pi,k ·d ≤ si ·D − δ (7)
[ , ]
𝑝, .𝜙 < 𝑝 , .𝛽 (6)
Electronics 2020, 9, x FOR PEER REVIEW 8 of 19
Electronics 2020, 9, 2131 8 of 19
𝑝 , .𝑑 ≤ 𝑠 .𝐷 − 𝛿 (7)
The calculationofofpi,k𝑝·d, .is𝑑shown
Thecalculation in Equation
is shown (3). This
in Equation constraint
(3). This ensures
constraint that the
ensures end-to-end
that delay
the end-to-end
delay ofi flow 𝑠 cannot be greater than the deadline. We also added the clock synchronization
of flow s cannot be greater than the deadline. We also added the clock synchronization accuracy δ to
accuracy 𝛿 to
compensate forcompensate
synchronization errors that mayerrors
for synchronization occurthat
between the two
may occur nodes.the two nodes.
between
(3)Path
(3) Pathorder
orderconstraint.
constraint.Packets
Packetsmust
must bebe
transmitted
transmittedalong thethe
along routing pathpath
routing given in advance,
given and
in advance,
the time of the packet at the next node is greater than the time of the previous
and the time of the packet at the next node is greater than the time of the previous node. node.
∀𝑝 ∈ 𝑠 ; ∀𝑘 ∈ 𝑁;
∀pi,k, ∈ si ; ∀k ∈ N;

pi,k [vx ,va ] ·φ + pi,k [vx ,va ] ·L + 1 + δ ≤ pi,k [va ,vb ] ·φ (8)
𝑝 , [ , ]. 𝜙 + 𝑝 , [ , ]. 𝐿 + 1 + 𝛿 ≤ 𝑝 , [ , ]. 𝜙 (8)
Here, vx represents the previous node of va in the routing path of flow si , and these nodes have
Here, 𝑣in advance
been stored represents the order
in the previous node of
of routing 𝑣 inand
nodes, the1routing
is the linkpath of flow 𝑠 delay.
propagation , and these nodes
have(4)been stored in advance in the order of routing nodes,
Queuing constraint. In order to prevent the nondeterministic caused by differentand 1 is the link propagation delay.
packets
queuing(4) Queuing
in the egressconstraint.
port, we In have
orderformulated
to preventcorresponding
the nondeterministic caused by This
queue constraint. different packets
constraint is
mainly for the case that packets transmitted from different nodes are forwarded to the same node atis
queuing in the egress port, we have formulated corresponding queue constraint. This constraint
mainly
the samefor the case
egress port,thatthatpackets
is, the casetransmitted
where therefromare different
the same nodes
linksare
inforwarded to the same
different routing paths.node at
the same egress port, that is, the case where there are the same
Figure 6a illustrates the nondeterministic queuing of flow. Two flows arrive at the same links in different routing paths.
switch
from different devices almost simultaneously and enter the same queue. Due to some factors,switch
Figure 6a illustrates the nondeterministic queuing of flow. Two flows arrive at the same such
from
as differentclock
inaccurate devices almost simultaneously
synchronization and enter the
between individual sameand
devices queue.frameDue to some
loss, factors, such
nondeterministic
as inaccurate
may appear when clock thesynchronization
flow is queued.between We haveindividual
two solutions devices and
for this frame loss,
situation. Thenondeterministic
first is shown in
may appear when the flow is queued. We have two solutions
Figure 6b. Two flows are allocated to the same queue, and the opening time of the queue for this situation. The first is shown
Gate of thein
Figure 6b.
previous Two
node is flows are allocated
controlled. Only after to thethe current
same queue, flow isand the opening
forwarded, timeflow
the latter of thecanqueue
enterGate of the
the queue,
previous node is controlled. Only after the current flow is forwarded,
which is achieved the time isolation between different flows and avoids interleaving. The second the latter flow can enter the
queue, which
method is shown is inachieved
Figure 6c, theallotting
time isolation two flowsbetween different
to different queues,flows and even
so that avoids interleaving.
if the arrival timeTheof
second method is shown in Figure 6c, allotting
two flows are close, the flow can be isolated and conflicts can be avoided.
two flows to different queues, so that even if the
arrival time of two flows are close, the flow can be isolated and conflicts can be avoided.
1 1 1 1
2 1 2 1 1 1 1 1
2 1 2 1 2 2 1 1 2 2 1 1
2 2 2 2 2 2
2 2

(a) (b) (c)

Figure6.6.Example
Figure Exampleofofflow
flow queuing
queuing on on port.
port. (a) Nondeterministic
(a) Nondeterministic queuing.
queuing. (b)isolation.
(b) Time Time isolation. (c)
(c) Assign
to different
Assign queues. queues.
to different

For
For flows
flowsarriving
arriving fromfromthe
thesame
samedevice,
device,the
thefirst
firstconstraint
constraint(periodically
(periodicallysend
sendconstraint)
constraint) will
will
ensure
ensurethat
thatthe thedata
datapackets
packetsofofthe
theflow
flowwill
willnot
notoverlap
overlapin inthe
thetime
timedomain.
domain. Assuming
Assuming that
thatthethedata
data
packetsp𝑝
packets i,k, and
and p j,l𝑝generated by flows si and
, generated by flows 𝑠 sand
j are𝑠forwarded to the to
are forwarded device
the device 𝑣 viavdevice
vb via device 𝑣 ,
a , and the
flow s
and the reaches v from the
i flow 𝑠 areaches 𝑣 from the device v via the link
x device 𝑣 via [v , v
x the ], and the flow s
a link [𝑣 , 𝑣 ], and reaches v from the device
j the flowa 𝑠 reaches 𝑣 from vy
via
thethe link [v
device va ].the
𝑣 y ,via Thelink
constraint
[𝑣 , 𝑣 ]. is
The as constraint
follows: is as follows:

∀pi,k, ∈∈ s𝑠i ; ;∀p

∀𝑝 ∀𝑝j,l, ∈∈s 𝑠j ; ∀k
; ∀𝑘∈ ∈N;𝑁;
∀l∀𝑙∈ ∈
N;𝑁;
     
¬ pi,k [va ,vb ] ·Q = p j,l [va ,vb ] ·Q ∨ pi,k [va ,vb ] ·φ + pi,k [va ,vb ] ·L ≤ p j,l [v y ,va ] ·φ − δ ∨ p j,l [va ,vb ] ·φ + p j,l [va ,vb ] ·L ≤ pi,k [vx ,va ] ·φ − δ
(9)
¬ 𝑝 , [ , ]. 𝑄 = 𝑝 , [ , ]. 𝑄 ⋁ 𝑝 , [ , ]. 𝜙 + 𝑝 , [ , ]. 𝐿
This constraint indicates [that for packets assigned to the same queue, the current packet can (9)
≤ 𝑝 , , ]. 𝜙 − 𝛿 ⋁ 𝑝 , [ , ]. 𝜙 + 𝑝 , [ , ]. 𝐿 ≤ 𝑝 , [ , ]. 𝜙 − 𝛿
enter the queue only after the previous packet is forwarded, or two different packets are assigned to
Thisqueues.
different constraint indicates that for packets assigned to the same queue, the current packet can
enter(5)
the queue
Link only after
congestion the previous
constraint. packet is forwarded,
This constraint ensures thator twoone
only different
packetpackets arethe
can enter assigned
link at to
a
different queues.
time on each port, that is, packets entering the same link cannot overlap in the time domain.
(5) Link
This congestionbyconstraint.
is achieved sequentialThis constraint ensures
transmission of packetsthatfrom
onlythe
onesame
packet can For
flow. enter the link
packets at a
from
time on each
different flows,port, thatenter
if they is, packets entering
the same the
queue, same link
constraint (4)cannot overlap
can ensure thatinpackets
the time domain.
entering the same
Electronics 2020, 9, 2131 9 of 19

queue will not conflicts in the time domain when they are forwarded, if they enter different queues,
constraint needs to be formulated to ensure that packets from different queues in the same port will
not collide when they enter the link. Define constraints as follows:

∀pi,k ∈ si ; ∀p j,l ∈ s j ; ∀k ∈ N; ∀l ∈ N;

pi,k−1 [va ,vb ] ·φ + pi,k−1 [va ,vb ] ·L ≤ pi,k [va ,vb ] ·φ (10)
     
pi,k [va ,vb ] ·Q = p j,l [va ,vb ] ·Q ∨ pi,k [va ,vb ] ·φ + pi,k [va ,vb ] ·L ≤ p j,l [va ,vb ] ·φ ∨ p j,l [va ,vb ] ·φ + p j,l [va ,vb ] ·L ≤ pi,k [va ,vb ] ·φ (11)

(6) Queue usage constraint. The number of queues allocated to time-critical flows cannot be greater
than the number of port queues. Define λ as the number of queues that allocated to scheduled traffic
by the port. All packets of the same flow enter the same queue. The constraints are as follows:

∀λ ∈ N; ∀c ∈ N; ∀k ∈ N; ∀l ∈ N; ∀pi,k ∈ si ; ∀pi,l ∈ si ;

0 ≤ [vb , va ]·λ ≤ [vb , va ]·c (12)

pi,k [va ,vb ] ·Q = pi,l [va ,vb ] ·Q (13)

(7) Time slot window size constraint. We will give a slot window size slotsize as input, so that we can
determine the time slot size according to the basic information of the network. This value is set as the
time for an MTU-sized packet to pass through the switch port, so that all packets of all sizes can pass
through the port. The constraint is as follows:

∀pi,k ∈ si ; ∀k ∈ N;

pi,k [va ,vb ] ·ω − pi,k [va ,vb ] ·τ = slotsize (14)

This constraint can be used to optimize the bandwidth used for scheduling traffic. If there is
no limit on the size of the time slot window, it is likely that a time slot window will occupy all the
time in a base period, so that there is no guarantee that there is remaining time to transmit the best
effort traffic. The user can decide by himself whether the constraint is required according to the actual
network scenario.
In addition, the time slot window cannot span the base period, so we limit the start time and end
time of the window to be in the same base period, and the constraints are as follows:

∀pi,k ∈ si ; ∀k ∈ N;

j pi,k [va ,vb ] ·τ k j pi,k [va ,vb ] ·ω k

= (15)
BP BP
(8) Time slot window transmission constraint. Packets must be transmitted during the corresponding
time slot window, so the start time of the time slot window of the packet pi,k on the port [va , vb ] cannot
be greater than the time when the packet starts transmission, and the end time of window cannot be
less than the time when the packet ends transmission.

∀pi,k ∈ si ; ∀k ∈ N;
   
pi,k [va ,vb ] ·τ ≤ pi,k [va ,vb ] ·φ ∧ pi,k [va ,vb ] ·φ + pi,k [va ,vb ] ·L ≤ pi,k [va ,vb ] ·ω (16)

This constraint ensures that there is enough time slot window to transmit packets, thus avoids the
situation where the time slot window is smaller than the packet transmission time and the transmission
is interrupted.
Electronics 2020, 9, 2131 10 of 19

(9) Time slot window allocation constraint. For flows with the same period in the same port, e.g.,
the period of two flows si ·T = s j ·T = n ∗ BP(n > 1), the same time slot in different base period can be
allocated. Since the scheduling entries of the same time slot in GCL have the same setting, the data
packets allocated to the same time slot should also be allocated to the same queue.

∀n ∈ N, n > 1; ∃m·[1, . . . , n − 1]; ∀pi,k ∈ si ; ∀p j,l ∈ s j ;

    
si ·T = s j ·T = n ∗ BP ∧ pi,k [va ,vb ] ·τ = p j,l [va ,vb ] ·τ + m ∗ BP · p j,l [va ,vb ] ·τ = pi,k [va ,vb ] ·τ + m ∗ BP (17)

pi,k [va ,vb ] ·Q = p j,l [va ,vb ] ·Q (18)

The same time slot cannot be shared between flows with different periods and flows with a period
equal to the BP. The constraints are as follows:

∀pi,k ∈ si ; ∀p j,l ∈ s j ; ∀k ∈ N; ∀m ∈ N;

j pi,k [va ,vb ] ·τ k

A = pi,k [va ,vb ] ·τ − ∗ BP
BP
   
si ·T , s j ·T · si ·T = s j ·T = BP ∧ m ∗ BP ≤ p j,l [va ,vb ] ·τ

≤ A − slotsize + m ∗ BP)· m ∗ BP + A + slotsize ≤ p j,l [va ,vb ] ·τ (19)
≤ (m + 1) ∗ BP))
The value of A in the formula is the start time of the slot window allocated to the flow si in the
first base period. Once the time slot of flow si is determined, the time period corresponding to the time
slot in all BP is reserved exclusively, and the flow s j cannot be used any more. Therefore, the Equation
(19) restricts the flow s j to only select the remaining unreserved time slots for scheduling in each BP.
(10) Period scheduling constraints. The scheduling time of each flow at the port is scheduled
periodically according to the period si ·T. The time when the flow si arrives at the port is represented
by pi,k [va ,vb ] ·at, and the slot window must be within the corresponding scheduling period. In other
words, after the packet of the flow si with period si ·T arrives at port [va , vb ], only the time slot window
within si ·T after the arrival time can be selected for scheduling. After the time slot of the first packet is
determined, the time slots of other packets in the same flow are also determined. Then there are the
following constraints:
∀pi,k ∈ si ; ∀k ∈ N;

pi,k [va ,vb ] ·at = pi,k [vx ,va ] ·φ + pi,k [vx ,va ] ·L + 1 (20)
   
pi,k [va ,vb ] ·at ≤ pi,k [va ,vb ] ·τ ∧ pi,k [va ,vb ] ·ω ≤ pi,k [va ,vb ] ·at + si ·T (21)

All packets of the same flow are scheduled in the same time slot of different BP, and the time
interval is si ·T. Once the corresponding time slot is determined for the first data packet of the flow, the
time slot for subsequent data packets of the flow is also determined.

∀pi,k ∈ si ; ∀k ∈ N;

pi,k+1 [va ,vb ] ·τ = pi,k [va ,vb ] ·τ + si ·T (22)

5. Evaluation
There are many different program implementations of SMT solver. In this paper, we use the most
widely used Z3 solver [19] to encode the SMT problem. The solution process of this paper based on the
SMT scheduling algorithm is as follows:
 [ , ] [ , ]
𝑝, .𝜏 = 𝑝 , .𝜏 + 𝑠 .𝑇 (22)

5. Evaluation
There
Electronics are
2020, many different program implementations of SMT solver. In this paper, we use
9, 2131 the
11 of 19
most widely used Z3 solver [19] to encode the SMT problem. The solution process of this paper based
on the SMT scheduling algorithm is as follows:
(1) Use the SMT method to construct TSN traffic scheduling problem constraints.
(1) Use the SMT method to construct TSN traffic scheduling problem constraints.
(2) According to the proposed constraints, Z3 is used for encoding.
(2) According to the proposed constraints, Z3 is used for encoding.
(3) Construct network topology and flow information.
(3) Construct network topology and flow information.
(4) A feasible solution that satisfies the constraints is calculated through Z3.
(4) A feasible solution that satisfies the constraints is calculated through Z3.
(5) Until the time slots of all flows are determined, the scheduling ends.
(5) Until the time slots of all flows are determined, the scheduling ends.
Scheduling input: We We take the network topology, flow information (including (including period,
period, routing,
routing,
size of packets, deadline) and the maximum time slot window as input.
Scheduling output: The Thesending
sendingtime
timeofofthetheflow
flowatateach
eachnode;
node; The
The arrival
arrival time
time of of each
each node
node in
in the flow path; The opening and closing time of the slot window of the of each
the flow path; The opening and closing time of the slot window of the of each switch egress port; The switch egress port;
The end-to-end
end-to-end delaydelay of each
of each flow;flow;
The The
timetime required
required to generate
to generate the schedule.
the schedule.
scheduling problem
The scheduling problem waswas solved
solved byby our code on an Intel(R) Core (TM) i7-7700 CPU @ 2.50
GHz machine
machinewith with88GB GBmemory.
memory.We Weused
usedthetheZ3Z3 v4.8.8
v4.8.8 solver
solver (64-bit)
(64-bit) [23],
[23], which,
which, in addition
in addition to
to the
the normal
normal SMTSMT function,
function, alsoalso
hashas
an an algorithm
algorithm forforsolving
solvinglinear
linearoptimization
optimizationgoalsgoals[24].
[24]. We
We use
TSNsched [15] to randomly generate network topology, and user-defined
[15] to randomly generate network topology, and user-defined network topologynetwork topology is createdis
with JAVA.
created withOur program
JAVA. and algorithm
Our program use JAVAuse
and algorithm in the linux
JAVA inenvironment, and the data
the linux environment, andprocessing
the data
uses MATLAB.
processing usesFor convenience,
MATLAB. we choose thewe
For convenience, timeline
choosegranularity
the timelineas granularity
1 µs, and theascommunication
1 μs, and the
speed of all links is 1 Gbit/s.
communication speed of all links is 1 Gbit/s.
We evaluated the time required to generate the schedule (Use “runtime” instead in the following following
articles.), the average
articles.), average end-to-end
end-to-end delay
delay ofof all flows in each test case, and the worst-case end-to-end
delay (Refers to the largest
largest end-to-end delay in this test test case.).
case.). Additionally,
Additionally, the BPC
BPC scheduling
scheduling cycle
method proposed in this paper is compared with the previous previous HPCHPC scheduling
scheduling cycle
cycle method.
method.

5.1. User-Defined
5.1. User-Defined Network
Network
This article
This article has
has created
created two
two network
network topologies
topologies as
as shown
shown in
in Figure
Figure 7,
7, which
which are
are divided into
divided into
small and medium-sized networks according to the size of the network topology, as shown
small and medium-sized networks according to the size of the network topology, as shown in Figurein
Figure 7a,b, respectively.
7a,b, respectively.

(a) (b)
Figure 7.
7. User-defined
User-definednetwork
network topology
topology example.
example. (a) Small
(a) Small network
network topologies.
topologies. (b) Medium
(b) Medium network
topologies.
network topologies.

The small network consists of 7 switches and 30 terminal systems, all the scheduling flow will be
forwarded by the SW9 , so when the number of flows is large, there will be flow conflicts and bandwidth
contention in SW9 . Using the algorithm of this paper to conduct a scheduling test, we can schedule the
number of flows with a packet size of 1500 bytes is 47 (all the flows forwarded by SW9 ). When the
number of flows is increased to 48, the program did not return a result within 50 h.
The medium network consists of 10 switches and 40 terminal systems. All of our tests, the sender
is ES0 to ES19 , and the receiver is ES20 to ES39 , so all flows will be forwarded to SW8 via SW9 . At the
 The small network consists of 7 switches and 30 terminal systems, all the scheduling flow will
be forwarded by the 𝑆𝑊 , so when the number of flows is large, there will be flow conflicts and
bandwidth contention in 𝑆𝑊 . Using the algorithm of this paper to conduct a scheduling test, we can
schedule the number of flows with a packet size of 1500 bytes is 47 (all the flows forwarded by 𝑆𝑊 ).
When the number of flows is increased to 48, the program did not return a result within 50 h.
Electronics 2020, 9, 2131 12 of 19
The medium network consists of 10 switches and 40 terminal systems. All of our tests, the sender
is 𝐸𝑆 to 𝐸𝑆 , and the receiver is 𝐸𝑆 to 𝐸𝑆 , so all flows will be forwarded to 𝑆𝑊 via 𝑆𝑊 . At
port [SW9 ,[𝑆𝑊
the port SW8,]𝑆𝑊 ], multiple
, multiple flowsflows will compete
will compete for network
for network bandwidth,
bandwidth, if not iffornot for scheduling,
scheduling, whichwhich
may
may cause
cause excessive excessive flow delay
flow delay or packet
or packet loss. We loss.
setWe
upset up multiple
multiple test for
test cases cases thefor the number
number of from
of flows flows
from
low to low
high,toand high, and performed
performed scheduling scheduling teststoin
tests in turn turn
test the to test the scheduling
scheduling quality of the quality of the
scheduling
scheduling
algorithm in algorithm in theThe
the worst case. worst case.
worst caseTheofworst case ofthat
scheduling, scheduling, that
is, there is only is, scheduled
there is only scheduled
traffic in the
traffic in and
network, the network,
there is no and there is no
best-effort best-effort
traffic and all traffic
flows and all flows are
are forwarded to forwarded
SW8 through 𝑆𝑊9 , through
to SW we can
𝑆𝑊 , we the
schedule can number
scheduleofthe number
flows with of flows with
a packet a packet
size of size is
1500 bytes of28.
1500 bytes is 28.
Wefirst
We firstset
setup
upall
allflow
flowperiods
periodsininthethetest
testcases
casestotobe
bethe
thesame,
same,then
thensetsetup upthe
theflow
flowof ofdifferent
different
periods in the test cases, and created more than 350 test cases
periods in the test cases, and created more than 350 test cases for evaluation. for evaluation.

5.1.1.
5.1.1.Flows
FlowsHave
Havethe
theSame
SamePeriod
Period
Figure
Figure88describes
describesthe theresults
resultsofofthe
thetest
testininthe
thesmall
smallnetwork,
network,increasing
increasingthe
thescheduling
schedulingflows
flowsin in
turn,
turn,and
andanalyzes
analyzesthe thechanges
changes inin runtime
runtime and
and delay
delay when
when the
the flow
flow period
period is
is set
set to
to 300 μs, 500
300 µs, 500 µs,
μs,
1000 μs,and
1000µs, and2000
2000µs.μs.Each
Eachtime
timethethe
period
periodis changed,
is changed, thethe
number
numberof flows increases
of flows fromfrom
increases 2 to 247,tothe
47,
packet size is
the packet setistoset
size 1500 bytes,
to 1500 and the
bytes, anddeadline is setistoset
the deadline 1000 µs. There
to 1000 are 184
μs. There aretest
184cases in total.
test cases in total.

Time (μs)
Time (μs)

(a) (b) (c)

Figure8.8. Small network
Figure network scheduling
scheduling results.
results. TheTheflows
flowsinin
each
eachtest case
test have
case thethe
have same period.
same (a)
period.
(a) Runtime.
Runtime. Average
(b)(b) Averageend-to- end delay
end-to-end delay. . (c) Worst-case
(c) Worst-case end-to-end
end-to-end delay.
delay.

Figure
Figure8a 8ashows
showsthe
therelationship
relationship between
between thethe
runtime
runtimeandand thethe
number
number of flows. As As
of flows. thethe
number
number of
flows increases, the runtime gradually increases. When the period is set to 500 µs,
of flows increases, the runtime gradually increases. When the period is set to 500 μs, 1000 μs, and 1000 µs, and 2000 µs,
the
2000difference between the
μs, the difference runtime
between theis runtime
not big, but it increases
is not big, but itsignificantly when the period
increases significantly whenisthe setperiod
to be
very small (e.g., 300 µs). Figure 8b,c shows that the average end-to-end delay (AVG)
is set to be very small (e.g., 300 μs). Figure 8b,c shows that the average end-to-end delay (AVG) of of all flows andall
worst-case end-to end delay (WCD) are kept within a small range, meeting the
flows and worst-case end-to end delay (WCD) are kept within a small range, meeting the deadline ofdeadline of 1000 µs.
1000Then
μs. the test is carried out in the medium-sized network, and three groups of test cases are set
up. TheTheninformation
the test isof test cases
carried out is
in shown in Table 1. The
the medium-sized transmission
network, and threeperiod of the
groups of flow in each
test cases aretest
set
group is the same. The scheduling results are shown in Figure 9.
up. The information of test cases is shown in Table 1. The transmission period of the flow in each test
group is the same. The scheduling results are shown in Figure 9.
Table 1. Test cases. Each group has 7 test cases according to the number of flows.

Number of Flows Packet Size Period Deadline

group 1 4,8,12,16,20,25,28 1500 bytes 500 µs 1000 µs
group 2 4,8,12,16,20,25,28 1500 bytes 1000 µs 1000 µs
group 3 4,8,12,16,20,25,28 1500 bytes 2000 µs 1000 µs

Figure 9a shows the relationship between the scheduling runtime and the number of flows.
With the number of flows increases, the runtime gradually increases. When the number of flows is
small, there is almost no difference in the runtime of the three groups of test cases, but with the increase
of the number of flows, the runtime of the test cases with 500 µs period increases faster than the other
two groups. Figure 9b,c shows the AVG and WCD variation. The WCD increases with the number of
flows, but all of them are within 400 µs, meeting the deadline of 1000 µs. The changes of runtime and
latency of the two groups of test cases with periods of 1000 µs and 2000 µs are basically the same.
 (a) (b) (c)
Figure 9. Medium network scheduling results. The flows in each test case have the same period. (a)
Electronics 2020, 9, 2131 13 of 19
Runtime.
Electronics 2020, 9, x(b)
FORAverage end-to-end
PEER REVIEW delay. (c) Worst-case end-to-end delay. 13 of 19

Table 1. Test cases. Each group has 7 test cases according to the number of flows.

Number of flows Packet size Period Deadline

group 1 4,8,12,16,20,25,28 1500 bytes 500 μs 1000 μs

Time (μs)

Time (μs)
group 2 4,8,12,16,20,25,28 1500 bytes 1000 μs 1000 μs
group 3 4,8,12,16,20,25,28 1500 bytes 2000 μs 1000 μs

Figure 9a shows the relationship between the scheduling runtime and the number of flows. With
the number of flows increases, the runtime gradually increases. When the number of flows is small,
there is almost no difference in the runtime of the three groups of test cases, but with the increase of
the number of flows, the runtime of the test cases with 500 μs period increases faster than the other
(a) (b) (c)
two groups. Figure 9b,c shows the AVG and WCD variation. The WCD increases with the number of
Figure
flows, but9.all
Figure 9.Medium
of themnetwork
Medium are
network scheduling
within 400 μs,results.
schedulingmeeting
results.Thethe
flows
The in each
deadline
flows oftest
in each1000case
test have
μs. The
case the same
changes
have period. (a) and
of runtime
the same period.
(a) Runtime.
Runtime. (b) Average
(b) end-to-end
Average end-to-enddelay
delay.
. (c)
(c) Worst-case
Worst-case end-to-end
end-to-end delay
delay. .
latency of the two groups of test cases with periods of 1000 μs and 2000 μs are basically the same.
5.1.2. Flows have Different Period
5.1.2. Flows Table 1. Test cases.
have Different Each group has 7 test cases according to the number of flows.
Period
The period of the flow in all test cases is arbitrarily selected from {500 µs, 1000 µs, 2000 µs, 4000 µs},
The period of the flowNumber
in all testofcases
flows Packet size
is arbitrarily selected Period Deadline
from {500 μs, 1000 μs, 2000 μs, 4000
and the size of all data packets is 1500 bytes, and the deadline is set to 1000 µs.
μs}, and the size group 1 4,8,12,16,20,25,28
of all data packets is 1500 bytes, 1500
andbytes
the deadline 500 μsis set 1000 μs μs.
to 1000
First, the scheduling test was carried out in the small network. After the flow period is set, BP is
group 2 test
First, the scheduling 4,8,12,16,20,25,28
was carried out in1500 bytes network.
the small 1000 μsAfter1000 the μs
flow period is set, BP is
calculated and the test cases with the same BP are in the same group. Additionally, 3 groups were set
calculated and the group 3 4,8,12,16,20,25,28
test cases with the same BP are 1500 bytes
in the same2000 group.μs Additionally,
1000 μs 3 groups were set
up, each group contains multiple test cases. The changes of runtime and latency are compared when
up, each group contains multiple test cases. The changes of runtime and latency are compared when
Figure
the BP size 9a shows
is 500 µs, the relationship
1000 µs, and 2000 between
µs. Thethe scheduling
test results areruntimeshown in and the number
Figure 10. of flows. With
the BP size is 500 μs, 1000 μs, and 2000 μs. The test results are shown in Figure 10.
the number of flows increases, the runtime gradually increases. When the number of flows is small,
there is almost no difference in the runtime 800 of the threeAVGgroups
(BP=500 μs)
of test cases, but with the increase of
AVG (BP=1000 μs)
the number of flows, the runtime of the 600
test cases with 500 μs
AVG (BP=2000 μs)period increases faster than the other
two groups. Figure 9b,c shows the AVG and WCD variation. The WCD increases with the number of
Time (μs)

flows, but all of them are within 400 μs,400meeting the deadline of 1000 μs. The changes of runtime and
latency of the two groups of test cases with 200
periods of 1000 μs and 2000 μs are basically the same.

5.1.2. Flows have Different Period 0

0 10 20 30 40 50
No. of flows
The period of (a)the flow in all test cases is arbitrarily (b) selected from {500 μs, 1000 μs, (c) 2000 μs, 4000
μs}, and the size of all data packets is 1500 bytes, and the deadline is set to 1000 μs.
Figure10.
10. Small
Small network test results. The period ofof thethe
flow in all testtest
cases is arbitrarily selected from
Figure
First, the scheduling test test
wasresults.
carriedThe outperiod
in the small flow
network. in all
Aftercases
the is arbitrarily
flow period selected
is set, BP is
{500 {500
from μs, 1000 μs, 2000
µs, 1000 μs, µs,
µs, 2000 40004000
μs}.µs}.(a) Runtime.
(a) Runtime. Average
(b) (b) Average end-to-end
end-to-enddelay (c) Worst-case
delay.. (c) Worst-case
calculated and the test cases with the same BP are in the same group. Additionally, 3 groups were set
end-to-end
end-to-end delay.
delay.
up, each group contains multiple test cases. The changes of runtime and latency are compared when
the BPFigure
size is 500shows
μs, 1000 μs,whenand 2000number μs. The testflows results are shown in Figure 10. BP will affect the
Figure10a10a showsthat that whenthe the numberof of flowsisisthe thesame, same,the thesize
sizeof ofthe
the BP will affect the
running
runningtime,
time,but
butthe
thedifference
differencebetween
between
800 the
thethree
threegroups
groups ofoftests
tests isisnot
notlarge.
large. AsAs the
thenumber
numberofofflows
flows
AVG (BP=500 μs)
increases, the test case with a smaller BP increases
increases, the test case with a smaller BP increasesAVGfaster. faster.
AVG Figure
(BP=1000 μs) 10b,c shows the
Figure 10b,c shows the AVG and WCD AVG and WCD
(BP=2000 μs)
600
variation.
variation.AllAllthe
theAVG
AVG is is
lower
lower than
than500500 µs.μs.In In
thethe group groupwith witha BP a ofBP500 µs, the
of 500 μs, WCD
the WCD is smaller, but
is smaller,
Time (μs)

when the number of flows increases, 400 the WCD in the test cases with a BP of 1000 µs and 2000 µs is

larger, but both are lower than 1000 µs, meeting the deadline of 1000 µs.
200
Then it was tested in the medium network. The changes of runtime and latency are compared
when the BP size is 500 µs, 1000 µs, and 0
0
2000
10
µs.
20
Figure
30
11 shows
40 50
the running results. The runtime
No. of flows
increases with the increase of the number of flows. When the same number of flows is scheduled, the
difference in the (a)runtime when the BP is 1000 µs(b) and 2000 µs respectively is small.(c) As the number of
flows increases, the runtime is smaller when the BP is 500 µs. Figure 11b,c
Figure 10. Small network test results. The period of the flow in all test cases is arbitrarily selected shows the AVG andfromWCD
variation.
{500 μs,AVG
1000and
μs, WCD
2000 μs,in 4000
all test
μs}.cases are lower
(a) Runtime. (b)than 1000 µs,
Average meeting the
end-to-end delaydeadline of 1000 µs.
. (c) Worst-case
end-to-end delay.

Figure 10a shows that when the number of flows is the same, the size of the BP will affect the
running time, but the difference between the three groups of tests is not large. As the number of flows
increases, the test case with a smaller BP increases faster. Figure 10b,c shows the AVG and WCD
variation. All the AVG is lower than 500 μs. In the group with a BP of 500 μs, the WCD is smaller,
 isbut when
larger, butthe number
both of flows
are lower thanincreases, the WCD
1000 μs, meeting theindeadline
the test cases with
of 1000 μs.a BP of 1000 μs and 2000 μs
is larger,
Then it but both
was are lower
tested in the than 1000 network.
medium μs, meeting Thethe deadline
changes of 1000 μs.
of runtime and latency are compared
when theThenBPitsize
wasistested
500 μs,in 1000
the medium network.
μs, and 2000 The changes
μs. Figure 11 shows of runtime and results.
the running latency The
are compared
runtime
when the
increases BP the
with sizeincrease
is 500 μs,of 1000 μs, andof2000
the number μs.When
flows. Figurethe 11same
shows the running
number of flowsresults. The runtime
is scheduled, the
increasesin
difference with
the the increase
runtime whenof the
the number
BP is 1000of flows.
μs andWhen2000 μstherespectively
same number of flows
is small. Asisthe
scheduled,
number ofthe
difference
Electronics 2020, 9, in the runtime when the BP is 1000 μs and 2000 μs respectively is small.
flows increases, the runtime is smaller when the BP is 500 μs. Figure 11b,c shows the AVG and WCD14
2131 As the number ofof 19
flows increases, the runtime is smaller when the BP is 500 μs. Figure 11b,c shows
variation. AVG and WCD in all test cases are lower than 1000 μs, meeting the deadline of 1000 μs. the AVG and WCD
variation. AVG and WCD in all test cases are lower than 1000 μs, meeting the deadline of 1000 μs.
2000 1000
BP=500 μs AVG (BP=500 μs) 1000
2000 BP=1000 μs 1000 AVG (BP=1000 μs)
BP=2000
BP=500μsμs 800 AVG (BP=2000
AVG μs)μs)
(BP=500 1000
1500 800
BP=1000 μs AVG (BP=1000 μs)
BP=2000 μs 800 AVG (BP=2000 μs)
1500 600 800
600
1000
600 600
400 400
1000
WCD (BP=500 μs)
500 400 400 WCD (BP=1000 μs)
200 200 WCD
WCD (BP=2000 μs)μs)
(BP=500
500 Deadline
WCD (BP=1000 μs)
200 200
0 0 0 WCD (BP=2000 μs)
0 5 10 15 20 25 30 0 5 10 15 20 25 30 0 5 10 15 Deadline
20 25 30
0 No. of flows 0 No. of flows 0 No. of flows
0 5 10 15 20 25 30 0 5 10 15 20 25 30 0 5 10 15 20 25 30
(a)
No. of flows (b)
No. of flows (c)
No. of flows
(a) (b) (c)
Figure Figure 11. Medium
11. Medium network
network testresults.
test results.TheThe period
period of ofthetheflow
flowinin allall
test cases
test casesis arbitrarily
is arbitrarily selected
selected
from {500 µs, 1000 µs, 2000 µs, 4000 µs}. (a) Runtime. (b) Average end-to-end delay. (c)Worst-
Figure
from {500 11.
μs,Medium
1000 μs,network
2000 μs,test
4000results.
μs}. The
(a) period
Runtime. of(b)theAverage
flow in end-to-end
all test cases delay
is arbitrarily
. (c) selected
Worst-case
case
from end-to-end
{500 μs, 1000 μs,. 2000 μs, 4000 μs}. (a) Runtime. (b) Average end-to-end delay. (c) Worst-
delay
end-to-end delay.
case end-to-end delay.
5.2. Randomly
5.2. Randomly Generated
Generated Network
Network Using
UsingNetwork
Network Tools
5.2. Randomly Generated Network Using Network Tools
In thisIn this section,
section, wewe evaluate
evaluate thethetraffic
trafficscheduling
scheduling in inthe
therandomly
randomly generated
generated network.network.We use We use
TSNsched
TSNsched In this
[15] to section,
[15] to generate
generate we evaluate
multiple
multiple the
testtesttraffic scheduling
caseswith
cases withdifferent
different in the randomly
network
network topology
topologygenerated
sizes
sizesand network.
andthe the We
number use of
number
TSNsched [15] to generate multiple test
of flows to evaluate the scheduling quality of our algorithm. cases with different network topology sizes and the number
flows to evaluate the scheduling quality of our algorithm.
of flows
Firstly,toconsidering
evaluate thethe scheduling
influencequality of our algorithm.
of the number of flows on the runtime, we randomly generated
Firstly, considering the influence of the number of flows on the runtime, we randomly generated
75 testFirstly,
cases inconsidering
a large network the influence
system of (20the numberand
switches of flows on the runtime,
100 terminals) with awe hop randomly
count of generated
3, and 3
75 test cases
75
groups in aset
testwere
cases large
inup, network
a large network
and flow system
period system (20(20
is 500 switches
μs, 1000 μs,and
switches and100 100 terminals)
2000 terminals) withaEach
with
μs, respectively. ahophop count
count
time the ofand
3, and
of period
3, 3 3
groups were
isgroups
changed,set
were up,
theset and flow
up, andofflow
number period
flowsperiod is 500 µs,
is 500from
increases 1000
μs, 1000 µs,
5 to μs, and
100,and 2000
the2000 µs, respectively.
μs, size
packet respectively.
is set to 1500 Each
Eachbytes, time
time theandthe period
period
the
is changed,
deadline the
is changed, number
is set the number
to 1000 of
μs.flows
of flows
The increases
increases
scheduling from
from5are
results 5toto100,
100,the
shown the packet
packet12a.
in Figure size
size isisset
The settoto1500
runtime 1500 bytes,
bytes,
increases and and
withthe the
deadline
the is set to
deadline
increase 1000
is set
of to
the1000
µs. The
μs. The
number scheduling
ofscheduling
flows. When results
results
theare
areshown
shownof
number in Figure
inflows
Figureis12a.12a. Thethere
The
small, runtime
runtime increases
is increases
basically no with
with
the increase
difference
the increase of inthethe ofruntime
numberthe number
ofbetween
flows.of When
flows.
different When
the the number
periods.
number When
of flowsthe ofnumber
isflows
small, isofthere
small,
flowsisis there
large,
basically is the
basically
smaller
no no
difference
difference
the period in
is, the
and runtime
the longer between
the runningdifferent
time periods.
is. When the
in the runtime between different periods. When the number of flows is large, the smaller the period is, number of flows is large, the smaller
and the the periodthe
longer is, and the longer
running time theis. running time is.
500 350
Periods=500 μs Periods=1000 μs
500
Periods=1000 μs 300350
400 Periods=500μsμs Periods=1000 μs
Periods=2000
Periods=1000 μs 250300
400 Periods=2000 μs
300 200250
300
200 150200

200 100150
100
50100
100
0 050
0 20 40 60 80 100 2 4 6 8 10
0 0
0 20 No.40of flows60 80 100 2 4 No. of hops
6 8 10
No.(a)of flows No.
(b)of hops
(a) (b)
Figure 12. Randomly generated network. The size of all data packets is 1500 bytes. (a) The impact of
the
Figure number
Figure
12. of scheduling
12. Randomly
Randomly generatedflows
generated on runtime
network.
network. Thesize
The size Impact
of
. (b)of dataofpackets
all data
all flow hop
packets count
isis1500 on(a)
bytes.
1500 bytes. The
runtime impact
(a) The of of
. impact
the number of scheduling flows on runtime . (b) Impact of flow hop
the number of scheduling flows on runtime. (b) Impact of flow hop count on runtime. count on runtime .
Then, considering the impact of the number of hops of the flow on the scheduling runtime, 9 test
Then,
cases with
Then, a considering
thethe
flow number
considering ofimpactofofathe
20 and
impact the
hopnumber
numberofof
number hops
2–10
hopsofwere
the
of flow on theon
randomly
the flow scheduling
generated runtime,
in 9 test
the above-
the scheduling runtime,
cases with a flow number of 20 and a hop number of 2–10 were randomly generated
9 test cases with a flow number of 20 and a hop number of 2–10 were randomly generated in the in the above-
above-mentioned large network system. The scheduling results are shown in Figure 12b. The number
of hops has a great impact on the scheduling runtime. However, the overall scheduling quality of our
scheduling algorithm is better. When the hop count is 10, the runtime of scheduling 20 flows is 339.72 s.

5.3. Comparison of Base Period and Hyper Period

In the small network, taking 5 flows as an example, we compared our BPC method with the HPC
method. The basic idea is to use flows of different periods to understand the performance difference
between the two method. The sending terminals of the 5 flows are ES0 , ES1 , ES2 , ES3 , and ES4 , and the
receiving terminals are ES15 , ES16 , ES17 , ES18 , and ES19 . A total of 8 test cases were set, the information
about the sending period, base period size, and hyper period size of each test case are shown in Table 2.
 In the small network, taking 5 flows as an example, we compared our BPC method with the HPC
method. The basic idea is to use flows of different periods to understand the performance difference
between the two method. The sending terminals of the 5 flows are 𝐸𝑆 , 𝐸𝑆 , 𝐸𝑆 , 𝐸𝑆 , and 𝐸𝑆 , and
the receiving terminals are 𝐸𝑆 , 𝐸𝑆 , 𝐸𝑆 , 𝐸𝑆 , and 𝐸𝑆 . A total of 8 test cases were set, the
information
Electronics 2020, about
9, 2131 the sending period, base period size, and hyper period size of each test case
15 ofare
19
shown in Table 2.

Table 2. Test
Table 2. Test cases
cases information
information in
in the
the small
small network.
network. Sending
Sending period
period of
of terminals,
terminals, base
base period
period size
size
(GCD), and hyper period size (LCM) per test-case, and the unit is
(GCD), and hyper period size (LCM) per test-case, and the unit is μs. µs.

ES0 𝑬𝑺𝟎 ES1𝑬𝑺𝟏 𝑬𝑺2 𝟐

ES 𝑬𝑺ES
𝟑3 𝑬𝑺𝟒 ES4GCD GCD LCM LCM
case 1 case2000
1 200020002000 2000
2000 2000
2000 200020002000 2000
2000 2000
case 2 1000
case 2 100010001000 1000
1000 1000
1000 1000 1000
1000 1000 1000 1000
case 3
case500
3 500 500500 500
500 500
500 500 500 500 500500 500
case 4 1000 1000 1000 2000 2000 1000 2000
case 5 case1000
4 100010001000 1000
1000 2000
1500 200015001000 500
2000 3000
case 6 case1000
5 100010001000 1000
1500 1500
2000 15002000 500 500
3000 6000
case 7 case1000
6 100015001000 1500
1500 2000
2000 20003000 500 500
6000 6000
case 8 case500
7 10008001500 800
1500 1000
2000 30001000 500 100
6000 4000
case 8 500 800 800 1000 1000 100 4000
Figure 13 shows the comparison results of the two methods used in test cases 1 to 8 in terms of
Figure 13 shows the comparison results of the two methods used in test cases 1 to 8 in terms of
scheduling runtime, where all results meet the delay requirement. The results show that when the
scheduling runtime, where all results meet the delay requirement. The results show that when the
periods of the flows are the same, the scheduling runtime of the two methods is similar. However,
periods of the flows are the same, the scheduling runtime of the two methods is similar. However,
when scheduling different period flows, BPC method is obviously superior to the HPC method, and
when scheduling different period flows, BPC method is obviously superior to the HPC method, and
BPC can
BPC can calculate
calculate the
the results
results at
at aa faster
faster speed.
speed.

180
160
140
120
Runtime (s)

100
80
60
40
20
0
case1 case2 case3 case4 case5 case6 case7 case8
BPC 0.42 0.43 0.42 0.61 1.64 2.89 3.2 10.68
HPC 0.42 0.41 0.43 3.07 6.65 115.15 122.1 170.47

Figure
Figure 13.
13. Comparison
Comparison of
of the
the runtime
runtime of
of the
the two
two methods
methods in
in aa small
small network.
network.

Then, in the medium network, the BPC method is compared with the HPC method, and the
runtime of the same flow period and different flow period is compared. Figure 14a shows the running
results when
when allallflow
flowperiods
periodswere werethe thesame.
same.AAtotaltotalofof9 test
9 test cases
cases were
were set,set,
thethe number
number of flows
of flows waswasset
set to and
to 12, 12, and the packet
the packet size size
is 1500is 1500 bytes,
bytes, and the andsending
the sendingperiod period increases
increases from 300fromµs300 μs toµs.
to 3000 3000 μs.
It can
It
becan
seenbefrom
seenthe from
resultsthe that
resultswhen thatallwhen
flowsall flows
have the have
same the samethere
period, period,is nothere is no significant
significant difference
difference
between the between
two methods. the twoFiguremethods. Figure the
14b shows 14brunning
shows the running
results resultsflow
of different of different
periods.flow
Theperiods.
number
The number
of flows was ofsetflows
to 8, andwasthe set packet
to 8, and sizethe is packet
1500 bytes.size is 1500
The bytes.terminals
sending The sending terminals
of the 8 flows ofarethe
ES08,
flows
ES1 , ESare
5 , 𝐸𝑆
ES6 , ,
ES 𝐸𝑆10 , ,ES𝐸𝑆
11 , , 𝐸𝑆
ES 15 ,, 𝐸𝑆
and ,
ES 𝐸𝑆
16 , a , 𝐸𝑆
total ,
of and
20 𝐸𝑆
test ,
cases a total
were of
set,20 test
the cases
informationwere set,
about the
sending period, base period size, and hyper period size of each test case are shown in Table A2.
It can be seen from the results that the BPC method has a lower runtime in these 20 test cases,
which is obviously better than the HPC method. Additionally, for 60% of the test cases, the BPC
method is significantly better than HPC method, especially for case20.
Finally, the number of GCL slot entries required by BPC method and HPC method for scheduling
is compared, and the results are shown in Figure 15. Figure 15a shows the number of GCL slot entries
required on ports [SW9 , SW3 ] when the test cases in Table 2 is scheduled. Figure 15b shows the
number of GCL slot entries required on ports [SW9 , SW8 ] when the test cases in Table A1 is scheduled.
The results shown in the figure reflect the worst-case scheduling (one slot per flow per BP, one slot per
packet per HP). It can be seen that the BPC method we proposed can significantly reduce the number
of time slot entries in GCL.
 8 150

Runti
7.5 100

Electronics 2020, 9, x FOR PEER REVIEW 50

16 of 19
7

information
Electronics the sending period, base period size,0 0and hyper
6.5
about period size of 15each test20case
16 of are
0 9, 2131
2020, 500
1000 1500 2000 2500 3000 5 10 19
shown in Table A2. Period of the flow ID of the test-cases
(a) (b)
9.5 300
BPC BPC
Figure 14. Comparison of the runtime of the two methods in the medium network. (a) The period
HPC
9 HPC 250
of the flow is the same. (b) The period of the flow is different.
8.5 200
Runtime (s)

It can be seen from the results that the BPC method has a lower runtime in these 20 test cases,
8 150
which is obviously better than the HPC method. Additionally, for 60% of the test cases, the BPC
method 7.5 is significantly better than HPC method, especially 100 for case20.
Finally,
7
the number of GCL slot entries required
50 by BPC method and HPC method for
scheduling is compared, and the results are shown in Figure 15. Figure 15a shows the number of GCL
6.5 0
slot entries 0 required
500 on ports
1000 1500[𝑆𝑊 , 𝑆𝑊2500
2000 ] when
3000the test cases0 in Table 5 2 is scheduled.
10 15Figure 15b
20 shows
the number of GCLPeriod of the flow
slot entries required on ports [𝑆𝑊 , 𝑆𝑊 ] when ID of the
the test-cases
test cases in Table A1 is
scheduled. The results shown (a) in the figure reflect the worst-case scheduling (b) (one slot per flow per
BP, Figure
one slot14.
Figure14.
per packet per HP).
Comparison
Comparisonofofthe
It can be the
theruntime
seen
runtimeofofthe two
two
that the BPC
methods
methods inin method
thethe
medium
medium
we proposed
network.
network. (a)(a)
Thecan
Thesignificantly
period
period of
reduce
the the
flow number
is the of
same. time
(b) slot
The entries
period of in
the GCL.
flow is different.
of the flow is the same. (b) The period of the flow is different.
30 60
It can be seen
BPC
from the results that the BPC method has a lower runtime in these 20BPC test cases,
which25is obviously
HPC better than the HPC method. Additionally,
50 for 60% of the test HPC
cases, the BPC
method is significantly better than HPC method, especially for case20.
20 40
Finally, the number of GCL slot entries required by BPC method and HPC method for
scheduling
15 is compared, and the results are shown in Figure
30 15. Figure 15a shows the number of GCL
slot entries required on ports [𝑆𝑊 , 𝑆𝑊 ] when the test cases in Table 2 is scheduled. Figure 15b shows
10 20
the number of GCL slot entries required on ports [𝑆𝑊 , 𝑆𝑊 ] when the test cases in Table A1 is
scheduled.
5 The results shown in the figure reflect the worst-case
10 scheduling (one slot per flow per
BP, one slot per packet per HP). It can be seen that the BPC method we proposed can significantly
0 0
reduce the 0 number
2 of time
4 slot entries
6 in8 GCL. 0 5 10 15 20
ID of the test-cases ID of the test-cases
30 (a) 60 (b)
BPC BPC
Figure
Figure
25 15. HPC
15.Comparison
Comparisonof ofthe
thenumber
numberof ofGate
GateControl 50 (GCL)
ControlList
List (GCL) time
time slot entries. (a) Result
slot entries. HPC
of
Result of port
port
[𝑆𝑊
[SW , 𝑆𝑊
9 , SW 3 ].].(b) Resultofofport
(b)Result port
[SW , 𝑆𝑊
9 , SW
[𝑆𝑊 8 ]. ].
20 40
6. Conclusions and Next Work
6. Conclusions
15 and Next Work 30
This paper studies the traffic scheduling problem in Time-Sensitive Networking. A network
This
10 paper studies the traffic scheduling problem 20 in Time-Sensitive Networking. A network
architecture that can dynamically manage TSN networks applied to the industrial networks is proposed.
architecture that can dynamically manage TSN networks applied to the industrial networks is
On the 5basis of 802.1Qbv and clock synchronization, a 10 scheduling algorithm based on base period
proposed. On the basis of 802.1Qbv and clock synchronization, a scheduling algorithm based on base
is proposed by using the Satisfiability Model Theory, which can greatly reduce the size of GCL and
period0is proposed by using the Satisfiability Model Theory, 0 which can greatly reduce the size of GCL
simplify 0the configuration
2 4 of switch
6 ports.
8 A series of scheduling
0 constraints
5 10 were formulated,
15 20 and
and simplify the configuration of switch ports. A series of scheduling constraints were formulated,
ID of the test-cases ID of the test-cases
the scheduling quality of our method was evaluated in a variety of network topologies. The results
and the scheduling quality of our method was evaluated in a variety of network topologies. The
show that the algorithm can (a) quickly calculate the scheduling results, with a(b) relatively low end-to-end
results show that the algorithm can quickly calculate the scheduling results, with a relatively low
delay,Figure
and compared
15. Comparison of the number of Gate Control List (GCL) time slot entries.results
with the previous scheduling of using hyper period, the show
(a) Result that our
of port
scheduling
[𝑆𝑊 , algorithm can calculate
𝑆𝑊 ]. (b) Result of port [𝑆𝑊the scheduling
, 𝑆𝑊 ]. results faster.
We are currently conducting research on dynamic scheduling algorithms, and intend to implement
dynamic incremental
6. Conclusions scheduling
and Next Work based on our static scheduling, mainly for scenarios where the
scheduled traffic in the network may change.
This paper studies the traffic scheduling problem in Time-Sensitive Networking. A network
architecture
Author that canConceptualization,
Contributions: dynamically manage TSN
Q.L. and X.J.;networks
methodology,applied to the
Q.L., D.L. andindustrial networks
X.J.; software, Q.L. andis
Q.W.;
proposed. On the basis of 802.1Qbv and clock synchronization, a scheduling algorithm based onQ.L.;
validation, Q.L., D.L., X.J., Q.W. and P.Z.; investigation, Q.L.; resources, D.L. and P.Z.; data curation, base
writing—original draft preparation, Q.L.; writing—review and editing, Q.L., D.L., X.J., Q.W. and P.Z.; supervision,
period is proposed by using the Satisfiability Model Theory, which can greatly reduce the size
D.L.; project administration, D.L. and P.Z.; funding acquisition, D.L. and P.Z. All authors have read and agreed to of GCL
and
the simplifyversion
published the configuration of switch ports. A series of scheduling constraints were formulated,
of the manuscript.
and the This
Funding: scheduling quality
research was fundedof our methodKey
by National wasR&Devaluated
Program ofin China
a variety of network topologies.
(2018YFB1702000), and was funded The
results
by show
Industrial that Innovation
Internet the algorithm can quicklyProject
and Development calculate theofscheduling
(Project results,
Basis Standard with a relatively
and Experimental low
Verification
for Industrial Software Defined Networking), and in part by Ningbo Science and Technology project (2018B10089).
Electronics 2020, 9, 2131 17 of 19

Conflicts of Interest: The authors declare no conflict of interest.

Appendix A

Table A1. Notations are used in this paper.

Notation Description
V Vertex set
E Link set
SW Switch set
ES End system set
S Flow set
va A single network node
si A single flow
[va , vb ]·θ Transmission rate of the link [va , vb ]
[va , vb ]·c Number of queues in egress port [va , vb ]
si ·r Routing path of flow si
si ·D Deadline of flow si
si ·T Period of flow si
pi,k Packet sent by the flow si in the k-th period si ·T
pi,k ·d End-to-end delay of packet pi,k
pi,k ·ρ Packet size
pi,k [va ,vb ] ·L Transmission duration of packet in port [va , vb ]
pi,k [va ,vb ] ·φ Send time of packet in port [va , vb ]
pi,k [va ,vb ] ·τ Time slot start time of packet in port [va , vb ]
pi,k [va ,vb ] ·ω Time slot end time of packet in port [va , vb ]
pi,k [va ,vb ] ·at Time when packet arrives at port [va , vb ]
Q The ID of the queue used by flow si
λ Number of queues that allocated to scheduled traffic
β Moment the packet is received by the receiver
δ Clock synchronization accuracy

Appendix B

Table A2. Test cases information in the medium network. Sending periodicity of terminals, base period
size (GCD) and hyper period size (LCM) per use-case, and the unit is µs.

ES0 ES1 ES5 ES6 ES10 ES11 ES15 ES16 GCD LCM
case 1 1200 1200 1200 1200 3600 3600 3600 3600 1200 3600
case 2 1000 1000 1000 1000 500 500 500 500 500 1000
case 3 1500 1500 1500 1500 3000 3000 3000 3000 1500 3000
case 4 2000 2000 2000 2000 4000 4000 4000 4000 2000 4000
case 5 1800 1800 1800 1800 2700 2700 2700 2700 900 5400
case 6 2500 2500 2500 2500 5000 5000 5000 5000 2500 5000
case 7 300 300 300 300 900 900 900 900 300 900
case 8 1000 1000 1000 1000 2000 2000 2000 2000 1000 2000
case 9 1200 3600 1200 3600 1200 1200 3600 3600 1200 3600
case 10 600 300 300 300 600 600 600 300 300 600
case 11 2000 2000 2000 2000 4000 4000 6000 6000 2000 12000
case 12 300 300 600 600 600 600 900 900 300 900
case 13 1200 1800 1200 1800 1200 1200 1800 1800 600 3600
case 14 1500 1000 1500 1000 1000 1000 1500 1500 500 3000
case 15 500 500 1500 1500 1500 1000 1000 1000 500 3000
case 16 5000 7500 5000 7500 5000 5000 7500 7500 2500 15000
case 17 3000 4500 3000 4500 3000 3000 4500 4500 1500 9000
case 18 2000 2000 2000 3000 3000 2000 3000 3000 1000 6000
case 19 1800 2700 1800 2700 1800 1800 2700 2700 900 5400
case 20 1000 1000 2000 2000 3000 3000 4000 4000 1000 12000
Electronics 2020, 9, 2131 18 of 19

References
1. Nasrallah, A.; Thyagaturu, A.S.; Alharbi, Z.; Wang, C.; Shao, X.; Reisslein, M.; ElBakoury, H. Ultra-low latency
(ULL) networks: The IEEE TSN and IETF DetNet standards and related 5G Ull research. IEEE Commun.
Surv. Tutor. 2019, 21, 88–145. [CrossRef]
2. 802.3-2015-IEEE Standard for Ethernet. Available online: https://ieeexplore.ieee.org/document/7428776
(accessed on 4 October 2020).
3. Institute of Electrical and Electronics Engineers, Inc, “802.1AS-Rev: Timing and Synchronization for Time-Sensitive
Applications”. Available online: http://www.ieee802.org/1/pages/802.1AS-rev.html (accessed on 3 September 2020).
4. Institute of Electrical and Electronics Engineers, Inc, “802.1Qbv: Enhancements for Scheduled Traffic”.
Available online: https://standards.ieee.org/standard/802_1Qbv-2015.html (accessed on 20 August 2020).
5. Lo Bello, L.; Steiner, W. A Perspective on IEEE Time-Sensitive Networking for Industrial Communication
and Automation Systems. Proc. IEEE 2019, 107, 1094–1120. [CrossRef]
6. Aeronautical Radio, Inc. ARINC 664P7: Aircraft Data Network, Part 7, Avionics Full-DupleX Switched
Ethernet Network (AFDX). 2009. Available online: https://standards.globalspec.com/std/1283307/arinc-664-p7
(accessed on 4 October 2020).
7. SAE. AS6802: Time-Triggered Ethernet; SAE International: Warrendale, PA, USA, 2011.
8. Vila-Carbo, J.; Tur-Massanet, J.; Hernandez-Orallo, E. Analysis of Switched Ethernet for Real-Time
Transmission. Tech. Fact. Autom. 2010, 221–239. [CrossRef]
9. Vila-Carbó, J.; Tur-Masanet, J.; Hernández-Orallo, E. An evaluation of switched Ethernet and Linux Traffic
Control for real-time transmission. In Proceedings of the 13th IEEE International Conference on Emerging
Technologies and Factory Automation, Hamburg, Germany, 15–18 September 2008; pp. 400–407.
10. Gavrilut, V.; Pop, P. Scheduling in time sensitive networks (TSN) for mixed-criticality industrial applications.
In Proceedings of the IEEE International Workshop on Factory Communication Systems (WFCS), Imperia,
Italy, 13–15 June 2018; pp. 1–4.
11. Serna Oliver, R.; Craciunas, S.S.; Steiner, W. IEEE 802.1Qbv gate control list synthesis using array theory
encoding. In Proceedings of the IEEE Real-Time and Embedded Technology and Applications Symposium
(RTAS), Porto, Portugal, 11–13 April 2018; pp. 13–24.
12. Craciunas, S.S.; Serna, R.; Martin, O.; Steiner, C.W. Scheduling real-time communication in IEEE 802.1Qbv
time sensitive networks. In Proceedings of the 24th International Conference on Real-Time Networks and
Systems, New York, NY, USA, 19–21 October 2016; pp. 183–192.
13. Gavrilut, V.; Zhao, L.; Raagaard, M.L.; Pop, P. AVB-Aware routing and scheduling of time-triggered traffic
for TSN. IEEE Access 2018, 6, 75229–75243. [CrossRef]
14. Mahfouzi, R.; Aminifar, A.; Samii, S.; Rezine, A.; Eles, P.; Peng, Z. Stability-aware integrated routing and
scheduling for control applications in Ethernet networks. In Proceedings of the 2018 Design, Automation
and Test in Europe Conference and Exhibition (DATE), Dresden, Germany, 19–23 March 2018; pp. 682–687.
15. Santos, A.C.T.D.; Schneider, B.; Nigam, V. TSNSCHED: Automated schedule generation for time sensitive
networking. In Proceedings of the 19th Conference on Formal Methods in Computer-Aided Design (FMCAD),
San Jose, CA, USA, 22–25 October 2019; pp. 69–77.
16. Pozo, F.; Rodriguez-Navas, G.; Hansson, H. Methods for large-scale time-triggered network scheduling.
Electronics 2019, 8, 738. [CrossRef]
17. Jin, X.; Xia, C.; Guan, N.; Xu, C.; Li, D.; Yin, Y.; Zeng, P. Real-time scheduling of massive data in time sensitive
networks with a limited number of schedule entries. IEEE Access 2020, 8, 6751–6767. [CrossRef]
18. Nayak, N.G.; Dürr, F.; Rothermel, K. Time-sensitive Software-defined Network (TSSDN) for Real-time
Applications. In Proceedings of the 24th International Conference on Real-Time Networks and Systems,
New York, NY, USA, 19–21 October 2016; pp. 193–202.
19. Nayak, N.G.; Durr, F.; Rothermel, K. Incremental flow scheduling and routing in time-sensitive
software-defined networks. IEEE Trans. Ind. Inform. 2018, 14, 2066–2075. [CrossRef]
20. Nayak, N.G.; Dürr, F.; Rothermel, K. Routing algorithms for IEEE802.1qbv networks. ACM SIGBED Rev.
2018, 15, 13–18. [CrossRef]
21. Katz, G.; Barrett, C.; Dill, D.L.; Julian, K.; Kochenderfer, M.J. Reluplex: An efficient smt solver for verifying
deep neural networks. In Proceedings of the Computer Aided Verification Conference (CAV), Heidelberg,
Germany, 24–28 July 2017.
Electronics 2020, 9, 2131 19 of 19

22. Steiner, W. An evaluation of SMT-based schedule synthesis for time-triggered multi-hop networks.
In Proceedings of the 31st IEEE Real-Time Systems Symposium, San Diego, CA, USA, 30 November–3
December 2010; pp. 375–384.
23. Z3Prover. The z3 Theorem Prover. Available online: https://github.com/Z3Prover/z3 (accessed on 4 October
2020).
24. Bjørner, N.; Phan, A.D.; Fleckenstein, L. νZ—An Optimizing SMT Solver. In Tools and Algorithms for the
Construction and Analysis of Systems; Baier, C., Tinelli, C., Eds.; TACAS: Iloilo, Philippines, 2015; pp. 194–199.

Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional
affiliations.

© 2020 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access
article distributed under the terms and conditions of the Creative Commons Attribution
(CC BY) license (http://creativecommons.org/licenses/by/4.0/).