International Journal of Production Research

ISSN: (Print) (Online) Journal homepage: www.tandfonline.com/journals/tprs20

Deep reinforcement learning for dynamic

scheduling of a flexible job shop

Renke Liu, Rajesh Piplani & Carlos Toro

To cite this article: Renke Liu, Rajesh Piplani & Carlos Toro (2022) Deep reinforcement learning
for dynamic scheduling of a flexible job shop, International Journal of Production Research,
60:13, 4049-4069, DOI: 10.1080/00207543.2022.2058432

To link to this article: https://doi.org/10.1080/00207543.2022.2058432

View supplementary material

Published online: 11 Apr 2022.

Submit your article to this journal

Article views: 7383

View related articles

View Crossmark data

Citing articles: 16 View citing articles

Full Terms & Conditions of access and use can be found at

https://www.tandfonline.com/action/journalInformation?journalCode=tprs20
INTERNATIONAL JOURNAL OF PRODUCTION RESEARCH
2022, VOL. 60, NO. 13, 4049–4069
https://doi.org/10.1080/00207543.2022.2058432

Deep reinforcement learning for dynamic scheduling of a flexible job shop

Renke Liu a , Rajesh Piplania and Carlos Torob
a School of Mechanical and Aerospace Engineering, Nanyang Technological University, Singapore, Singapore; b Vicomtech Research Centre, San
Sebastian, Spain

ABSTRACT ARTICLE HISTORY

The ability to handle unpredictable dynamic events is becoming more important in pursuing agile Received 30 March 2021
and flexible production scheduling. At the same time, the cyber-physical convergence in production Accepted 20 March 2022
system creates massive amounts of industrial data that needs to be mined and analysed in real-time. KEYWORDS
To facilitate such real-time control, this research proposes a hierarchical and distributed architecture Dynamic scheduling;
to solve the dynamic flexible job shop scheduling problem. Double Deep Q-Network algorithm is distributed multi-agent
used to train the scheduling agents, to capture the relationship between production information and systems; flexible job shop;
scheduling objectives, and make real-time scheduling decisions for a flexible job shop with constant hierarchical scheduling; deep
job arrivals. Specialised state and action representations are proposed to handle the variable speci- reinforcement learning
fication of the problem in dynamic scheduling. Additionally, a surrogate reward-shaping technique
to improve learning efficiency and scheduling effectiveness is developed. A simulation study is car-
ried out to validate the performance of the proposed approach under different scenarios. Numerical
results show that not only does the proposed approach deliver superior performance as compared to
existing scheduling strategies, its advantages persist even if the manufacturing system configuration
changes.

1. Introduction
Unpredictable real-time disruptions in manufacturing
systems lead to changes in effectiveness of scheduled
plans or activities; even minor disruptions can add up
to make the pre-developed schedule suboptimal, even
infeasible. Typical dynamic events on shop floor such as
machine breakdowns and operator unavailability must
be attended to quickly to maintain high production effi-
ciency. In addition to these internal disruptions, compa-
nies now give customers high flexibility in placing and
revising their orders to gain advantages in a customer-
oriented and highly competitive environment, making
the arrival and cancellation of orders more frequent than
ever.
Accelerating cyber-physical convergence on the shop
floor is revolutionising the availability and diversity of
industrial data. The large-scale deployment of connected
devices significantly improves the accessibility and com-
plexity of real-time industrial data (Messaoud et al.
2020). Besides, advanced computing architectures such
as cloud computing and edge computing are developed to
enhance the capability for and reduce the latency of on-
site data analytics (Chen and Ran 2019). The constantly
changing manufacturing system requires real-time and
high-volume data analytics, which exceeds the capability
of traditional shop floor management systems or human
expertise and creates a need for automatic knowledge
creation and self-adaptive control.
The manufacturing industry paradigm is shifting
from mass production to mass customisation, requir-
ing the ability to manage production with diversified
product configurations and trajectories of operations.
The scheduling problem under these circumstances is
referred to as the job shop problem (JSP), which aims
at finding the optimal sequence of jobs to be processed
at different machines. Extra complexity arises with the
deployment of ‘flexible’ machines: scheduling decisions
now consist of machine selection (routing) and job selec-
tion (sequencing); the resulting scheduling problem is
referred to as the flexible job shop problem (FJSP).
In this research, we propose a distributed and hierar-
chical approach to solve dynamic FJSP (DFJSP), realised
by Deep Reinforcement Learning (DRL). The objective
is to design an architecture in which intelligent agents
learn the mapping from shop floor information to sound
scheduling decisions through reward-driven training,
and then perform real-time control to minimise tardiness
in an environment with constant job arrivals.

CONTACT Renke Liu renke001@e.ntu.edu sg 50 Nanyang Ave, Block N3, Nanyang Ave, Singapore 639798, Singapore
Supplemental data for this article can be accessed here. https://doi.org/10.1080/00207543.2022.2058432
© 2022 Informa UK Limited, trading as Taylor & Francis Group
4050 R. LIU ET AL.
The remainder of the paper is organised as follows.
Section 2 reviews the development and latest research in
dynamic scheduling and the applications of Reinforce-
ment Learning (RL) to scheduling. Section 3 defines the
research problem and objective. Section 4 presents an
approach for solving the DFJSP. Section 5 then presents
the results of numerical experiments and the compari-
son with benchmark solutions. Finally, Section 6 high-
lights our contribution and discusses future directions for
research.
2. Literature review
2.1. Traditional approaches
Static scheduling research has created a large fam-
ily of techniques that are transferable to the dynamic
environment: the most popular techniques are priority
rules and metaheuristics. Due to the difference in tim-
ing and frequency of engagement, their applications to
dynamic scheduling problems can be categorised as com-
pletely reactive, robust-proactive and predictive-reactive
approaches (Ouelhadj and Petrovic 2008).
Priority rules build a schedule in a reactive and con-
structive manner, and are widely applied due to their ease
of implementation. The performance and comparison of
priority rules under different scenarios have been stud-
ied for a long time, and the consensus is that no single
priority rule provides strictly stronger performance along
all objectives in all scenarios (Sels, Gheysen, and Van-
houcke 2012; Xiong et al. 2017; Ðurasević and Jakobović
2018). It is also widely accepted that composite priority
rules’ performance is usually better than their building
blocks (Sels, Gheysen, and Vanhoucke 2012).
Schedules can also be produced in advance in consid-
eration of dynamic events, metaheuristics such as Tabu
Search, Simulated Annealing, Genetic Algorithm, Parti-
cle Swarm Optimisation, and Ant Colony Optimisation
are popular in developing robust-proactive schedules
(Ouelhadj and Petrovic 2008). In addition to commonly
used performance objectives such as flow time, make
span and tardiness, robustness must be included as an
extra performance metric to evaluate a schedule’s abil-
ity to absorb disruptions (Liu et al. 2017; Zhang, Song,
and Wu 2020). The development of a robust schedule
starts from a randomised solution or a population of solu-
tions; the quality of schedule is improved through the
time-consuming process of iterative evaluation (usually
by simulation of production with dynamic events) and
reproduction. The resultant high cost of schedule devel-
opment compromises metaheuristics’ ability to handle
frequent dynamic events. Most studies based on meta-
heuristic techniques focus on machine breakdowns; the
scheduling either does not or only includes a simple
rescheduling process as the impact of infrequent machine
breakdowns can be effectively minimised by incorporat-
ing robustness metrics in schedule development.
When rescheduling is needed, the developed pro-
duction schedule can be repaired by heuristics or
replaced by priority rules. Such a manner is referred
to as the predictive-reactive approach, which maintains
the highest-possible efficiency until the occurrence of
events and delivers feasible and timely scheduling deci-
sions afterwards. Recent studies also resort to hybrid
approaches to reduce the decisional latency of meta-
heuristics, thus enabling their ability to handle fre-
quent disruptions such as job arrival and cancellation.
Examples include the combination of tabu search and
genetic algorithm (Li and Gao 2020); combination of
particle swarm optimisation and genetic algorithm (Ren
et al. 2021).
2.2. Recent data-driven approaches
Traditional approaches face the dilemma of efficiency
versus quality. The high computational cost of meta-
heuristics compromises their utility in an environment
with frequent dynamic events, where the effectiveness
of optimal schedule may deteriorate quickly. On the
other hand, rule-based approaches produce near real-
time solutions at the cost of quality, due to the limited
type of information and simple mathematical operation
involved in decision-making.
Recent research aims at developing scheduling strate-
gies that use comprehensive shop floor information to
make high-quality scheduling decisions in near real-
time, by shifting the time-consuming development pro-
cess offline and applying the developed strategy to
online decision-making. Some candidate approaches are
Genetic Programming (GP), supervised learning, and
RL.
GP can be used for developing priority rules. Com-
pared to traditional manually-designed priority rules,
which utilise a limited range of job or machine infor-
mation, GP provides a way to include a variety of real-
time information in decision-making; the development
of rules is automated through iterative, population-based,
and randomised evolution.
Early studies simplify the FJSP to JSP by coupling
the sequencing rule with a universal routing rule (Ho,
Tay, and Lai 2007; Pickardt et al. 2010). Recently, GP
is also used for coevolving routing and sequencing
rules. Yska, Mei, and Zhang (2018) propose the Coop-
erative Coevolution GP (CCGP) framework to evolve
routing and sequencing rules in two separate popu-
lations. Zhang, Mei, and Zhang (2018) propose the

multi-tree GP (MTGP) architecture to exploit the inter-
action between routing and sequencing rules. There is
also research that aims to improve the performance by
feature selection (Zhang, Mei, and Zhang 2019; Zhang,
Mei et al. 2020). GP-based approaches do improve the
effectiveness of developed rules by incorporating a wider
range of features/information as input; however, their
decision-making still relies on simple computations and
logical operations.
Alternatives are techniques with complex parametri-
sation, such as supervised learning algorithms. The
key assumption in their application to scheduling is
that optimal scheduling knowledge can be learned by
parametrised algorithms such as Artificial Neural Net-
work (ANN), Support Vector Machine (SVM), and Deci-
sion Tree. Supervised learning algorithms improve the
quality of solution by analytical optimisation or gradient-
based approach, leading to faster convergence to a satis-
factory solution compared to evolutionary approaches.
ANN is primarily used for priority rule selec-
tion. Mouelhi-Chibani and Pierreval (2010) propose a
dynamic rule-selection mechanism that is triggered by
real-time events to solve the dynamic flow shop schedul-
ing problem; Guh, Shiue, and Tseng (2011) propose
an adaptive rule-selection mechanism that periodically
assigns different priority rules to various work centres in
a flexible manufacturing system.
There is also research aimed at finding the optimal
sequence of jobs. Weckman, Ganduri, and Koonce (2008)
break the JSP into a series of job-oriented classification
problems and use optimal schedules generated by GA to
train the ANN. Zang et al. (2019) also divide JSP into
several sub-problems to be solved by a hybrid network
with parallel fully-connected and convolutional layers.
Gupta, Majumder, and Laha (2019) propose an approach
to find the relative position of jobs in an optimal sched-
ule instead of developing the optimal sequence directly.
Other supervised learning applications include the use of
Decision Trees to learn the strategy from optimal sched-
ules (Olafsson and Li 2010; Jun, Lee, and Chun 2019),
and SVM as the adaptive rule-selector (Shiue 2009; Priore
et al. 2010).
2.3. Reinforcement learning-based scheduling
Compared to other supervised learning algorithms pri-
marily developed for classification tasks, DRL algorithms
can build a direct mapping from the observation of envi-
ronment to action; they also have the ability to handle
complex data due to the utilisation of artificial neural
network (ANN) as value or policy representation. These
two features made DRL a promising candidate approach
for production control in a data-intensive manufacturing
system.
INTERNATIONAL JOURNAL OF PRODUCTION RESEARCH 4051
In addition, dynamic scheduling problem can be for-
mulated as a discrete-time stochastic control process that
conforms to the definition of Markov Decision Process
(MDP) (Zhang, Xie, and Rose 2017). More specifically, a
shop that operates under a reactive policy taken by dis-
tributed scheduling agents can be modelled as a decen-
tralised Markov decision process (DEC-MDP) (Bern-
stein et al. 2002). The Markovian nature of dynamic
scheduling problem makes RL, a tool for sequential deci-
sion making in MDP, suitable for solving the scheduling
problem.
Early studies use tabular RL algorithms to perform
system-wide scheduling (Wang and Usher 2005; Aissani,
Beldjilali, and Trentesaux 2008), in which the state space
contains the abstracted system information; with action
being the selection among a set of priority rules to be
applied to all machines in the system. It is also feasi-
ble to break the scheduling problem into several sub-
problems with lower complexity, and tackle each with an
independent RL agent (Wang et al. 2020).
DRL algorithms have been incorporated into the
system-wide scheduling architecture for better ability to
handle high-dimensional input space, such as the use of
DQN (Lin et al. 2019), DDPG (Liu, Chang, and Tseng
2020) and PPO (Park et al. 2021) to solve JSP.
RL algorithms attached to each machine or work cen-
tre can also be distributed decision-makers. Gabel and
Riedmiller (2012) describe the JSP as a decentralised
Markov decision process with changing action sets and
partially ordered transition dependencies and propose a
policy-gradient approach to solve it. A similar approach
can be found in (Lang et al. 2020), where two types of
scheduling agents trained by DQN perform job sequenc-
ing and machine selection, independently, to manage a
flexible job shop.
Approaches developed to solve fixed-size scheduling
problems are hard to generalise to different classes of
tasks. There are also very few dynamic scheduling stud-
ies that do not constrain the specification of problem
to approximate a practical environment. Wang (2020)
proposes a decentralised Q-learning approach to select
scheduling rules, to tackle JSP with dynamic job arrivals.
Luo (2020) proposes a centralised double DQN approach
to solve FJSP with dynamic job arrivals, in which the
action is semi-integrated so that agents select coupled
sequencing and machine selection rules.
Some studies aim at scheduling in a specific context,
thus having specialised state and action representations.
Waschneck et al. (2018) propose a DQN approach to
schedule the movement of lots between positions in a
semiconductor manufacturing facility. Qu, Wang, and
Jasperneite (2019) propose a decentralised actor-critic
approach to allocate jobs to servers. Kuhnle, Röhrig, and
4052 R. LIU ET AL.
Lanza (2019) develop an autonomous order dispatching
system based on TRPO in a semiconductor production
facility. Shi et al. (2020) propose a DQN approach to
transfer jobs in a discrete manufacturing system. Hubbs
et al. (2020) propose a deep actor-critic approach for
scheduling in a chemical manufacturing system with
non-deterministic processing times; Malus, Kozjek, and
Vrabič (2020) propose a multi-agent approach to man-
age autonomous mobile robots on a shop floor. Yang and
Xu (2021) propose an A2C approach to manage a recon-
figurable production system with dynamic job arrival.
Luo et al. (2021) propose a PPO approach to solve the
dynamic work shop problem constrained by multiple
resources such as machine, tool and worker.
Jobs can also be the decision-making entity. Bouazza,
Sallez, and Beldjilali (2017) propose a Q-learning
approach that lets intelligent products choose a machine
selection and a job sequencing rule for each work centre
and machine they visit. Baer et al. (2019) model the flex-
ible manufacturing system as a Petri net (Zhou 2012), in
which the DQN agents attached to individual products
decide the route to different machines.
DRL-based dynamic scheduling is still a novel research
area with very few publications. This research adopts the
most general context – flexible job shop for better trans-
ferability and comparability. A formal definition of the
problem is presented in next section.
3. Problem formulation
A flexible job shop contains several work centres, each
of which may contain multiple machines. All jobs must
visit each work centre once following a predetermined
Figure 1. Hierarchical and distributed scheduling framework.
and unique sequence of operations. In a hierarchical and
distributed scheduling system, scheduling decisions are
made by routing and sequencing agents attached to each
work centre and machine, respectively; the framework is
presented in Figure 1.
Routing Agent (RA) and Sequencing Agent (SA) make
the scheduling decision by using priority rules or algo-
rithms such as DRL. The following assumptions are pro-
posed: (1) a machine can process only one operation at a
time; (2) preemption is prohibited; (3) no rework is mod-
elled; (4) no yield losses are considered; (5) buffer size is
unlimited and not explicitly modelled; (6) machine setup
and job travel time are ignored.
3.1. Notations
(1) Jobs: J = {Ji : i = 1, . . . , n} is the set of n jobs.
(2) Machines: M = {Mk : k = 1, . . . , m} is the set of m
machines.
(3) Work centres: W = {Wl : l = 1, . . . , w} is the set of
w work centres, each work centre contains several
machines, Wl = {Ma , . . . , Mz }; a machine can only
belong to one work centre.
(4) Sequence and operations: job Ji requires ni opera-
tions OJi = [O1i , . . . , Oni i ] to be served by machines
by an operation sequence Sqci = [Wi1 , . . . , Wini ].
j j
Wi denotes the work centre where the operation Oi
needs to be processed. Completion rate Pi is the ratio
of number of processed operations to ni .
(5) Processing time: ti,k denotes the processing time of Ji
j
on machine Mk . ti denotes the expected processing
 INTERNATIONAL JOURNAL OF PRODUCTION RESEARCH 4053

j Validation scenario and objective

time of Oi , and is calculated as
j 1  There are three factors that affect the production per-
ti = j
ti,k (1) formance. (1) Arrival rate of jobs: a higher arrival rate
|Wi | j
Mk ∈Wi leads to higher utilisation rate of machines and higher
congestion in the system. (2) Heterogeneity of jobs and
j machines: jobs are different with regards to their process-
PTJi = {ti : j = 1, . . . , ni } is the set of expected process-
ing time of all operations of Ji . ing time on different machines and due date tightness.
Additionally, their remaining work, slack time, and time
(6) Queue and available time of machine: J k = till due also vary over time once they start their process-
{Ja , . . . , Jz } is the set of jobs that are currently ing sequence. Machines also vary with regards to their
in queue to be processed or being processed on workload and availability. (3) Due date tightness: relaxed
machine Mk , COk denotes the time since the start TTD of a job results in lower total and average tardiness
of current operation on Mk ; AMk , the available time as jobs have more slack time (on average) to consume in
of Mk and AWl , the average available time of a work case of congestion. In addition, if the criticality of jobs
centre Wl can be calculated as in the system varies, the scheduler can take advantage of
 some jobs’ high slack time to protect more critical jobs.
AMk = ti,k − COk (2) Based on these observations, we introduce three sce-
Ji ∈J k narios for training and validation:
1 
AWl = AMk (3) (1) Expected arrival rate/ utilisation rate: we adjust
|Wl | M ∈W
k l the job arrival rate to match an expected utilisa-
(7) Time-till-due (TTD), remaining processing time tion rate of the system. Let E(t) be the expected
and slack time: NOW denotes the current time in processing time of all operations on all machines
system; Di denotes the due date of job Ji ; TTD of a and E(interval) denotes the expected time interval
job is the difference between its due date and NOW. between job arrivals. With the assumption that m
Subsequently, the slack time Si of a job is its TTDi machines are evenly distributed across all w work
less Ri , which is the sum of expected processing centres, the expected utilisation rate of the system
time of all remaining operations. These values are can be calculated as
calculated as
TTDi = Di − NOW (4) E(t) ÷ mw
E(utilization rate) = × 100%

ni
j E(interval)
Ri = ti (5)
E(t) × w
j=x = × 100%
E(interval) × m
Si = TTDi − Ri (6) (11)
where x is the index of Ji ’s imminent operation.
After the expected utilisation rate is specified, the
The value of TTDi , Ri and Si change as production
expected time between arrivals can be computed. Let us
advances.
assume that random variable X denotes the time interval
(8) Measurement of performance: Ci is the completion
between arrivals of jobs follows exponential distribution:
time of Ji , the tardiness Ti of a job is calculated as
X ∼ Exp(β), β = E(interval).
Ti = max(Ci − Di , 0) (7) It is worth mentioning that the expected utilisation
rate is calculated based on the assumption of constant
The total tardiness Tsum , maximum tardiness Tmax , and and smooth arrival, and identical jobs. In practice, some
tardy rate are calculated as idle capacity is inevitable due to stochasticity in inter-
arrival time, sequence of operations, and processing time

n
Tsum = Ti (8) on machines. The expected utilisation rate is assumed to
i=1 be 90% in this research to approximate the production in
a busy factory.
Tmax = max({Ti : i = 1, . . . , n}) (9)
|{Ji |Ti > 0}| (2) Heterogeneity of processing time: The processing
Tardy rate = (10)
n time ti,k is drawn from a uniform distribution: ti,k ∼
4054 R. LIU ET AL.
U[Low, High]. We consider two scenarios with iden-
tical average processing time: (1) High heterogene-
ity scenario: ti,k ∼ U[5, 25], where the maximum
time is five times the minimum; and (2) Low hetero-
geneity scenario: ti,k ∼ U[10, 20], where the maxi-
mum time is only twice the minimum.
(3) Due date tightness: upon creation of job Ji , it
is assigned an original TTD proportional to its
expected total processing time. The original TTD,
original slack time and due date are calculated as
original

ni
j
TTDi = αi × ti , αi ∼ U[1, High]
j=1

ni
original
Si = (αi − 1) ×
j
ti
j=1
original
Di = NOW + TTDi
The ratio αi denotes the due date tightness of job Ji .
We consider two scenarios of tightness: (1) High due date
tightness: αi ∼ U[1, 2], where the average original slack
time of jobs is half their total processing time; and (2)
Low due date tightness: αi ∼ U[1, 3], where the average
original slack time is equal to the job’s total processing
time.
Four scenario combinations are used: (1) High het-
erogeneity and high tightness: HH([5, 25]/[1, 2]); (2)
High heterogeneity and low tightness: HL([5, 25]/[1, 3]);
(3) Low heterogeneity and high tightness: LH([10, 20]/
[1, 2]); and (4) Low heterogeneity and low tightness:
LL([10, 20]/[1, 3]).
Minimising cumulative tardiness of all jobs is the pri-
mary objective in this research; additionally, the tardy
rate and maximum tardiness are also presented in the
appendix to provide a comprehensive evaluation of the
approach.
4. Proposed approach
4.1. DRL preliminaries
RL is one of the most powerful tools for solving sequen-
tial decision-making problem (van Otterlo and Wiering
2012). It can be modelled as an MDP with 5-elements
tuple representation: (S, A, P, γ , R). In MDP, an intelli-
gent agent interacts with the environment according to a
specific policy π; At each time step t, the agent observes
the current state st ∈ S and takes an action at ∈ A accord-
ing to policy π(S, A); A reward rt ∈ R is given after the
agent arrives at a new state st+1 . The goal of RL agent is
to develop a policy that maximises the cumulative reward
Gt (usually discounted).
DQN (Mnih et al. 2015) architecture improves
the performance of neural-approximated RL in high-
dimensional state space. The introduction of experience-
replay mechanism removes the correlation between
sequence of observations. The Q-network learns from a
minibatch of experience et , where et = (st , at , rt , st+1 ) is
the record of a transition in MDP. In addition, Q-network
contains action network and target network; the former
is used for producing the action and the latter to gen-
erate a stable target value for parameter optimisation.
The parameter of action network (θ A ) is updated at each
(12) training iteration, while the parameter of target network
(θ T ) synchronises with θ A at low frequency to stabilise
the learning. The parametrised target value yi for learning
(13) at ith iteration is calculated as
yi = rt + γ maxa Q(st+1 , a|θ T ) (15)
(14)
The agent can be trained by minimising the loss L(θiA )
at ith iteration. For example, the least square error is
calculated as
2  2
L(θiA ) = (yi − Q(st , at |θiA )) (16)
N
Based on DQN, Double DQN (Van Hasselt, Guez, and
Silver 2016) replaces the max operation that is used to
generate the estimation of state value by following the
action that the action network rather than target net-
work would take in the next state. To decouple the action
selection and policy evaluation and eventually reduce
over-estimation, the calculation of target value can be
done as
yi = rt + γ Q(st+1 , argmaxa Q(st+1 , a|θiA )|θ T ) (17)
Double DQN is used as the learner of the two types
of agents in the system: (1) Routing agent attached to
each work centre which performs the machine selection
for jobs upon their arrival, corresponding to the blue
dot in Figure 1, and (2) Sequencing agent attached to
each machine, selects a job to process when machine
becomes idle and finds more than one job in the queue,
corresponding to the orange box in Figure 1.
4.2. State representation
As a component of DRL algorithm, the neural network
with static architecture does not facilitate adjustment of
the size of input/ output space conditioned on a vari-
able number of queuing jobs. Many studies simply avoid
this problem by setting the research scope to a fixed size
problem with a known number of jobs. This, however,

Table 1. State features for routing agent.
Category Feature Expression
1. Information of machine Available time∗ {AMk |Mk ∈ Wl } ⎫
⎧
⎨ ⎬
Sum of processing times∗ ti,k |Mk ∈ Wl
⎩ ⎭
Ji ∈Jk
2. Information of job to be Processing time∗ {ti,k |Mk ∈ Wl }
dispatched (Ji )
Slack time Si
3. Information of arriving job Time till imminent arrival min(AT l )
Slack time min{Si |Ji ∈ AJl }
lacks practicality in a dynamic environment and cannot
be used in this research which focuses on a manufac-
turing system with a large and unknown number of job
arrivals spread over time.
Instead, we use abstracted information instead of job-
specific features to build the state space. The number of
jobs only affects the computation rather than the dimen-
sion of resulting abstracted data. Some related notation is
introduced next: (1) J NOW = {Ji |Ri > 0} is the set of jobs
that are currently in the system, and can be used to mea-
sure the congestion in the system; (2) AJ l is the set of jobs
that are currently being processed by other machines and
will arrive at the work centre Wl , the set of the arrival time
of these jobs is AT l ; and (3) for jobs in J k , the set of their
succeeding work centre is denoted as SW k .
RA’s state space consists of three categories of features:
(1) the information of machines within the work centre;
(2) the information of the job to be dispatched; and (3)
information of the job that is about to arrive. Details of
state features are given in Table 1; three types of features
(tagged with ‘∗’) are collected for each machine in the
work centre. For RA that controls work centre Wl , the
size of its state space equals |Wl | × 3 + 3.
Arrival and departure of jobs constantly change the
queue size for the SA. Thus collecting certain categories
of information for all jobs results in a state space with
variable dimension. In this work, we use abstracted and
channelled data to summarise the information of queu-
ing jobs and system to create a stable state space.
Twenty-five features are divided into six channels
according to their type and magnitude; their details are
given in Table 2.
4.3. Action representation
Available action for RA simply corresponds to the selec-
tion of machines within the work centre. For SA, how-
ever, the changing queue makes the direct selection of job
INTERNATIONAL JOURNAL OF PRODUCTION RESEARCH 4055
infeasible, hence we select four sequencing rules as the
building block of the action space to realise indirect job
selection:
(1) Shortest Processing Time (SPT): select the job with
shortest processing time of imminent operation.
(2) Work in Queue (WINQ): select the job with the
smallest sum of processing time of queuing jobs in
its succeeding stage of production. WINQ rule tends
to even the distribution of jobs in system.
(3) Critical Ratio (CR): select the job with smallest ratio
of TTD to remaining processing time. A disadvan-
tage of CR rule is its inconsistency. When available
jobs are not yet tardy, CR rule favours jobs with
smaller TTD and longer remaining processing time.
On the other hand, if available jobs are already tardy,
CR rule prioritises the job with highest overdue time
(smallest TTD) and shortest remaining processing
time.
(4) Minimum Slack (MS): select the job with smallest
slack time. This rule behaves like CR rule if jobs are
not yet overdue, favouring jobs with shorter TTD
and longer remaining processing time (shorter slack
time as a result) and remains consistent when the job
becomes tardy.
4.4. Surrogate reward-shaping
With the minimisation of tardiness as the scheduling
objective, a natural reward mechanism would be to use
the realised tardiness of job as the joint reward signal for
all RAs and SAs that the job has gone through. How-
ever, studies of multi-agent RL suggest that even though
agents can learn from the joint reward to achieve a joint
goal, they may receive spurious reward signals that orig-
inate from their teammates’ behaviour (Sunehag et al.
2017); this challenge is referred to as the multi-agent
credit assignment problem. Instead, a surrogate approach
that rewards agents for their contribution to the queuing
of jobs is developed.
An example production history of a job with three
operations is shown in Figure 2. Job arrives at the shop
floor with an initial duration of TTD that is larger than
the sum of its operation times, which leaves some slack
time as the queuing buffer. The slack time is gradually
consumed as the job queues for processing; tardiness
results if a job consumes all the slack time before its
processing is completed.
4056 R. LIU ET AL.

Table 2. State features for sequencing agent.

Channel Feature Calculation
1. Number of jobs Jobs in system |JNOW |
Jobs in queue |Jk |
|AJl |
Expected number of arriving job |W l |

2. Processing time of imminent operation of queuing Sum PT k
jobs (PT k ): PT k = {ti,k |Ji ∈ Jk } 1  k
Mean PT
|Jk |
Min min(PT ) k

3.1 Remaining processing time of queuing jobs Total remaining processing time RPT k
(RPT k ): RPT k = {Ri |Ji ∈ Jk }
1 
3.2 Available time of succeeding work centres Average remaining processing time RPT k
(ASW k ): ASW k = {AWl |Wl ∈ SW k } |Jk |
Maximum remaining processing time max(RPT k )
ASW k
Average available time of succeeding work centre
|SW k |
Minimum available time of succeeding work centre min(ASW k )
1  k
4.1 Time till due: X k = {TTDi |Ji ∈ Jk } Mean of TTD X
Min of TTD |Jk | k
min(X )
4.2 Slack time: Y k = {Si |Ji ∈ Jk } 1  k
Mean of slack time Y
|Jk |
Min of slack time min(Y k )
Min of slack time of arriving job min{Si |Ji ∈ AJl }
AMk
5. System-level information Available time share
My ∈M AMy
1 
Completion rate Pi
|JNOW | NOW
Ji ∈J
{Ji |Ji ∈JNOW ∧ TTDi <0}
Realised tardy rate |JNOW |
{Ji |Ji ∈JNOW ∧ Si <0}
Expected tardy rate |JNOW |
SPT k
6. Heterogeneity, represented by the coefficient of CV of processing time k
variation (CV) SPT
RPT k
CV of remaining processing time
RPT k
SX k
CV of TTD
Xk
SY k
CV of slack time
Yk
SASW k
CV of available time of succeeding work centres
ASW k

Figure 2. History of production.

The prevention and reduction of tardiness are equiv-
alent to the conservation of slack time at each stage of
production. However, rewarding interim target is not
always advisable as the agent might learn a strategy that
maximises the rewards and ignores the actual target
(Kuhnle, Schafer et al. 2019). Several aspects must be
considered to ensure training’s success.
Firstly, the reward function needs to balance con-
flicting objectives as pursuing different tardiness-related
objectives can lead to an undesirable outcome. For
instance, when the system is overwhelmed by a large
quantity of newly arrived jobs with tight due dates, the
SPT rule significantly reduces the sum of tardiness and
proportion of tardy jobs as compared to other rules but

at the cost of maximum tardiness, as it ‘sacrifices’ jobs
with high processing time and instead prioritises jobs
with shorter time consumption regardless of their crit-
icality. An unbiased reward must be given to improve
the performance along primary objective, which in this
research, is cumulative tardiness.
Secondly, the reward must align with system-level
performance. To avoid the credit-assignment problem,
RA and SA are trained to make decisions based on
mostly local information; the reward is also calculated
with respect to individual operations instead of eventual
tardiness. The key to training success is to ensure the
consistency between the reward and overall perfor-
mance, i.e. improve system-level performance by pursu-
ing short-term rewards.
In addition, it is very difficult to preserve the slack time
of all jobs, especially in a congested system, or when the
due dates are tight. Considering this, it is more reasonable
to quantify the likelihood of a job becoming tardy and
include this factor in the calculation of reward. For job Ji ,
we use a sigmoid function to convert its slack time Si to a
criticality factor Fi , to eliminate outliers (excessively large
values):
Si
Fi = 1 − (18)
|Si | + β
Fi ∈ (0, 2), where β is a user-specified factor to adjust
the curvature of the sigmoid function; as the value of
β increases, the sigmoid function becomes flatter and
smoother.
Based on the above considerations, we propose two
slack-driven reward functions.
Surrogate reward function for RA: RA needs to eval-
uate the availability and suitability of all machines and
assign the job to a machine that minimises long-term
cumulative tardiness. When job Ji arrives at the work
centre Wl to complete its xth operation, an estimated
slack upon the completion of operation (time step t + 1),
denoted as E(St+1
i ), can be calculated as a function of
the remaining processing time and the available time of
machines.

ni
1 
j
E(St+1
i ) = TTDi − ti − AMk (19)
j=x
|Wl | M ∈W
k l
The actual slack time St+1 is realised upon the completion
i
of the operation:

ni
St+1 = TTDi −
j
ti (20)
i
j=x+1
The reward is calculated as the gain/ loss of actual slack
time compared with the estimated slack time, adjusted by
INTERNATIONAL JOURNAL OF PRODUCTION RESEARCH 4057
the job’s criticality factor.
rt = Fi × (St+1
i − E(St+1
i )) (21)
Surrogate reward function for SA: SA needs to pri-
oritise the queuing jobs and select the one that minimises
the long-term cumulative tardiness. The reward can be
interpreted from a queuing exposure perspective: when
SA of machine Mk and its queue J k , selects a job Js , it ends
the wait of the selected job at the cost of extending queue-
ing time of other jobs, while at the same time expos-
ing the selected job to queuing at its succeeding work
centre Ws .
We term the saved or prolonged queueing time as
the gain or loss of slack time. For the selected job, the
slack time gain/ loss is defined as the average processing
time of other jobs adjusted by the criticality factor of Js ,
minus the adjusted available time of the succeeding work
centre:
{ti,k |Ji ∈ J k ∧ Ji = Js }
S1 = Fs × − δ × AWs (22)
|J k | − 1
The average slack time gain/loss of the jobs that are not
selected equals the processing time on Js adjusted by the
mean value of their criticality factors, minus the adjusted
average available time of their succeeding work centre.
Ji ∈J k ∧Ji =Js Fi
S2 = × ts,k − δ
|J k | − 1
{AWi |Wi ∈ SW k ∧ Wi = Ws }
× (23)
|J k | − 1
δ is a factor to adjust the magnitude of available time
as it is usually few times larger than the processing
time; agents would place excessive attention on succeed-
ing work centre’s workload without this adjustment. The
eventual reward is calculated as:
rt = S1 − S2 (24)
Choosing an appropriate value of β and δ for the
reward calculation is crucial for training’s success. A
cursory search leads to 40 < β < 60 and δ = 0.2; gener-
ally, agents trained with smaller β perform better in lower
heterogeneity scenarios.
4.5. Asynchronous transition
We propose an asynchronous transition approach to
align with the surrogate reward shaping. As shown in
Figure 3, the transition starts from the decision-making
point at time step t and ends upon the completion
of operation at time step t + 1. Time step t + 1 does
not necessarily coincide with the next decision-making
point.
4058 R. LIU ET AL.
Figure 3. Transition for routing and sequencing agent.
4.6. Parameters and training
Distributed scheduling architecture is essentially a multi-
agent system. We adopt parameter sharing technique and
centralised training and decentralised execution train-
ing scheme (Gupta, Egorov, and Kochenderfer 2017)
in this research to reduce the multi-agent problem to
a single-agent problem in the training stage. Agents
with similar scope and objective share the neural net-
work parameters and learn from the public experience
pool, leading to similar and cooperative behaviour. Other
DRL-related parameters are listed in Table 3.
Both types of agents are trained in a flexible job
shop consisting of three work centres, each housing
two machines. Training of RA and SA is done sepa-
rately to avoid the non-stationarity. The training process
simulates production for 100,000 units of time, during
which around 12,400 jobs arrive at the shop floor. The
process of simulation and training is shown in Figure 4.
The value of many features, especially those associ-
ated with the number of jobs and time, far exceeds the
range in which common activation functions can pro-
duce smooth and distinguishable gradient. Instance nor-
malisation technique (Ulyanov, Vedaldi, and Lempitsky
2016) is used to process the data to avoid the vanishing
gradient problem. As for the neurons, tanh (hyperbolic
tangent) function is used as the activation function as
negative values are quite common in state space. Huber
loss is used as the loss function. Learning rate decays from
10−2 to 10−3 . Stochastic gradient descent (SGD) is used
as the optimiser of ANN, the momentum is set as 0.9. The
hyperparameters of networks are listed in Table 3.
Table 3. Parameters for DRL and ANN.
Category Parameter
DRL related Discount factor (γ )
Exploration rate ( )
Minibatch size
Replay memory size
Hyperparameters of ANN Input size
Output size
Channels
Hidden layers
The architectures of ANN for RA and SA are shown
in Figure 5. In the two-branch architecture for SA, only
the data in channels 1 to 4 is processed by the instance
normalisation layer as the abstracted data in channels 5
and 6 has suitable magnitude.
The simulation model of manufacturing system is
implemented by open-source Python discrete-event sim-
ulation library SimPy (Matloff 2008). ANN is imple-
mented using Python machine learning library PyTorch
(Paszke et al. 2019). Training is carried out on a personal
computer with Intel Core i5-4210H 2.90 GHz CPU and
12 GB RAM. The records of training loss of RA and SA
are presented in Figure A-11 and A-12 respectively.
5. Numerical experiments
The experiments are divided into three stages:
(1) Independent utility test to test the independent
utility of SA and RA in designated scenarios and
qualify benchmark rules to enter the second stage of
experiment based on their performance on cumula-
tive tardiness (primary objective).
(2) Integrated DRL test that based on the result of the
first stage, wherein several sequencing and routing
rule combinations are used to validate the perfor-
mance of integrated SA and RA. Recent DRL-based
and GP-based DFJSP solutions are also compared.
(3) Scalability test that studies the performance of RA
and SA in larger-scale systems, and verifies their
scalability.
RA SA
0.8
0.3 decays to 0.1
128 64
512 256
9 25
2 4
1 6
16 × 16 × 16 × 8 × 8 48 × 36 × 36 × 24 × 24 × 12
 INTERNATIONAL JOURNAL OF PRODUCTION RESEARCH 4059

Figure 4. Simulation and training.

Figure 5. Architecture of ANN for scheduling agents.

The performance of priority rules and algorithms or algorithm in a run is calculated as:
are tested in 100 runs of simulation (each containing
multiple iterations) under each validation scenario. A
unique production incident is created in each run; the baseline − T
Tsum
rule or algorithm
sum
time interval between job arrivals, the processing time on NP = baseline
(25)
Tsum
each machine, and due date tightness of jobs are drawn
from respective statistical distributions. To make a fair
NP measures the performance gain; positive values
comparison, the production incident is then iteratively
imply better than baseline performance, higher the bet-
applied to a system controlled by DRL agents or bench-
ter. NP is presented in ‘.1’ panels in each figure below;
mark rules. Each iteration lasts for 1,000 units of time,
the mean of proposed DRL approach is drawn as green
with 124 job arrivals (on average).
dot lines.
The performance is evaluated using the following per-
formance metrics:
(2) Win rate: percentage of runs in which a rule or
(1) Normalised performance (NP): with a priority rule algorithm results in the lowest cumulative tardiness.
as the baseline, the normalised performance of rule It measures the adaptability of scheduling strategy
4060 R. LIU ET AL.

to various production incidents, and its superior- 5.1. Independent utility test
ity to others. To better present how proposed DRL
5.1.1. Routing agent
approach performs relative to benchmark rules, in
In this section, the independent utility of RA (DRL-RA) is
figures below we show the win rate with and without
tested, with six well-known routing rules listed in Table 4
the DRL agents in ‘.2’ and ‘.3’ panels, respectively.
as the benchmark. They are all coupled with First-In-
First-Out (FIFO) sequencing rule to minimise the effect
of sequencing decision; Earliest Available (EA) routing
Table 4. Benchmark routing rules.
rule is the baseline.
No. Routing Rule Description The results of numerical experiments are summarised
1 Earliest Completion Time (CT) Smallest sum of available time of in Figure 6. In each scenario, the top two benchmark rules
machine and processing time on the
machine. (except baseline) are marked with star; rules with a lower-
2 Minimum Execution Time (ET) Shortest processing time of job on the than-baseline performance are marked with cross.
machine.
3 Minimum Total Processing Least total processing time of jobs in The difference between routing strategies is decisive,
Time (TT) queue. and proposed DRL-RA exhibits strict superiority over
4 Minimum Utilisation Rate (UT) Least utilised machine, also termed as
‘longest idle time’ or ‘shortest work
all benchmark routing rules. An interesting observation
hour’ rule. is that EA rule, as the baseline with minimal active job
5 Smallest Queue Size (SQ) Fewest jobs in queue. allocation effect, delivers better performance than most
6 EA∗ Earliest available (idle) machine, also
termed as ‘shortest waiting time’ rule. routing rules, making it the second-best benchmark rule

Figure 6. Tardiness performance of RA under four validation scenarios.


in two low heterogeneity scenarios entering the stage of
the integrated DRL experiment. See Table A-6 for more
detailed results.
5.1.2. Sequencing agent
In this section, SA (DRL-SA) and other sequencing rules
are coupled with EA routing rule. In addition to the four
rules used for building the action space, we also include
in the benchmark pool nine other well-known sequenc-
ing rules designed to improve tardiness-related objectives
and six composite rules that have strong performance
in minimising tardiness. The complete list is shown in
Table 5; FIFO rule is the baseline rule. The results of
numerical experiments are shown in Figure 7. In each
scenario, the top five benchmark rules are marked with
star; rules with a lower-than-baseline performance are
marked with cross.
Ranking of rules differs in four scenarios, indicating
that the attribute of sound sequencing decision changes
in different environments; sequencing strategy utilising
limited shop floor information cannot adapt to all scenar-
ios. On the other hand, the difference in normalised per-
formance is small; even the best rule or algorithm delivers
less than 30% performance gain over baseline, resulting
in a rather even distribution of win rate. A rule with even
middling performance can win in some of the runs.
Another cross-scenario observation is that the supe-
rior performance of singular rules with more complex
parameterisation and composite rules such as MOD,
PTWINQS and DPTLWKRS is consistent.
Proposed DRL-SA outperforms most of the rules in
all the scenarios; it delivers around 4% higher perfor-
mance than the best benchmark rule in each scenario.
The comparison of win rate with and without DRL agents
shows that proposed approach does not simply imitate
Table 5. Benchmark sequencing rules.
No. Rule
1 Apparent tardiness cost (ATC)
2 Average processing time per operation (AVPRO)
3 Cost over time (COVERT)
4 CR
5 Earliest due date (EDD)
6 Least work remaining (LWKR)
7 Modified due date (MDD)
8 Modified operational due date (MOD)
9 Montagne’s heuristic (MON)
10 Slack (MS)
11 Next operation’s processing time (NPT)
12 SPT
13 WINQ
14 LWKR + SPT (LWKRSPT)
15 LWKR + MOD (LWKRMOD)
16 SPT + WINQ (PTWINQ)
17 SPT + WINQ + Slack (PTWINQS)
18 2PT + LWKR + Slack (DPTLWKRS)
19 2PT + WINQ + NPT (DPTWINQNPT)
20 FIFO∗
INTERNATIONAL JOURNAL OF PRODUCTION RESEARCH 4061
the behaviour of a specific rule, but maintains good per-
formance across all validation instances and beats the
other rules consistently. See Table A-7 for more detailed
results.
5.2. Validation of integrated DRL
In this section, trained RA and SA are integrated to con-
trol all work centres and machines to solve DFJSP. To
verify the synergistic effect between them, we also let RA
and SA run independently, in parallel.
Based on the results of the previous section, we select
top two routing rules and top five sequencing rules to cre-
ate ten benchmark combinations, with FIFO+EA being
the baseline. In addition, two novel GP-generated rule
combinations are also included, termed GP1 (Zhang,
Mei, and Zhang 2019) and GP2 (Zhang et al. 2020a),
respectively.
As it is unfair to compare with tabular RL-based
approaches given the decisive superiority of DRL algo-
rithms, the range of benchmarks is limited to the DRL-
based DFJSP solutions, which leaves very few studies to
compare against. Two peer approaches are included as the
benchmark, termed Lang 2020 (Lang et al. 2020) and Luo
2020 (Luo 2020), respectively.
Metaheuristic-based approaches are not used for com-
parison as they lack the ability to cope with dynamic job
arrival and cannot be applied to real-time scheduling.
The results are shown in Figure 8; see Table A-8 for
more detailed results.
Routing decision has more of an impact on sys-
tem performance than sequencing decision, the differ-
ence between the scheduling strategies is affected more
by the constituent routing rule than the sequencing
rule.
This phenomenon is, on one hand, a result of how
often scheduling agent gets engaged for decision-making:
under hierarchical scheduling architecture, a job must
be assigned to a machine but can be processed with-
out a sequencing decision. Additionally, routing deci-
sions can better exploit the heterogeneity of processing
times and workload. Good routing decisions can avoid
excessively long operation and queuing times, whereas
sequencing decisions only rearrange the job sequence
with the constraints of unchangeable capacity and
demand.
In four scenarios, integrated DRL outperforms all
combinations of benchmark rules, peer DRL-based and
GP-based solutions. The performance of independent
RA and SA runs in parallel also validates their utility
as a component of the integrated approach; although
SA does not provide as clear an advantage as RA, they
both contribute to the performance improvement of
4062 R. LIU ET AL.

Figure 7. Tardiness performance of SA under four validation scenarios.

 INTERNATIONAL JOURNAL OF PRODUCTION RESEARCH 4063

Figure 8. Tardiness performance of integrated DRL under four validation scenarios.

4064 R. LIU ET AL.
integrated DRL; with either one absent performance
degrades.
5.3. Scalability
In this section, proposed approach’s sensitivity to config-
uration change is tested. RA and SA run individually in
two scalability tests.
5.3.1. Higher flexibility test
For the flexibility test, the number of machines at each
work centre is increased. RA are retrained due to the
change in input/output size as the configuration of work
centres has changed; the records of training loss are
presented in Figure A-13 and A-14, respectively. For
simplicity of presentation, only the top two benchmark
rules under each validation scenario from section 5.1.1
and baseline (EA rule) are compared. The results of 3-
machine per work centre test (180 job arrivals on average)
Figure 9. Tardiness performance under higher flexibility.
are shown in Figure 9, panels (a) to (d); the results of 4-
machine per work centre test (240 job arrivals on average)
are shown in panels (e) to (h).
Flexibility amplifies the difference among the routing
rules, indicating that alternatives for job allocation can
either be an advantage or a handicap. CT rules that utilise
the job processing time and machine workload informa-
tion benefit from flexibility increase and achieve better
normalised performance and lower cumulative tardiness
even with more job arrivals. The superiority of DRL-RA
and CT rules becomes even more pronounced in the flex-
ible systems; they both reduce the cumulative tardiness
to an extremely low level with high heterogeneity of pro-
cessing time, especially when the number of machines
at a work centre reaches four. Table A-9 shows more
information on the results of cumulative tardiness.
The performance gain in flexible systems, however,
comes at the expense of a reduced win rate for the pro-
posed DRL-RA. When the exploitation of system flexibil-
ity overwhelms the high utilisation rate, the occurrence of

Figure 10. Tardiness performance under longer sequences.
tardiness is more a result of the fluctuation in job arrival
rate rather than the quality of routing decisions.
5.3.2. Longer sequence test
For the test of longer process sequence, the number of
operations (work centres) is increased to six and nine.
Due to the homogeneity of input/output size, the trained
models used in section 5.1.2 are imported and used as
the initial parameter (instead of random initialisation) to
accelerate the learning. SA is then retrained to adapt to
the new environment; the training loss records are pre-
sented in Figures A-15 and A-16, respectively. Only the
top five benchmark rules from section 5.1.2 and base-
line (FIFO rule) are compared under each validation
scenario. The results of 6-operation test are shown in
Figure 10, panels (a) to (d), while the result of 9-operation
test is shown in panels (e) to (h).
The performance gain of all sequencing rules increases
slightly when the number of operations increases;
INTERNATIONAL JOURNAL OF PRODUCTION RESEARCH 4065
SA still outperforms or at least matches the perfor-
mance of strongest benchmark rules. On the other
hand, the difference between SA and strong bench-
mark rules is insensitive to the configuration changes;
this phenomenon conforms with Holthaus and Rajen-
dran (1997), who state that the relative performance of
sequencing rules is not significantly affected by the size
of the manufacturing system. Table A-10 provides more
detailed results.
6. Conclusion
Scheduling in the factory of the future would rely heav-
ily on the ability to handle unpredictable dynamic events
with real-time responsiveness. This research proposes a
DRL architecture to solve the DFJSP as an attempt in that
direction.
Two types of scheduling agents trained with Dou-
ble DQN algorithm outperform existing scheduling
4066 R. LIU ET AL.
strategies either as independent components or when
deployed in an integrated manner; the advantage pro-
vided by scheduling agents is maintained even as the
system configuration changes. In addition to the per-
formance improvement, major contributions of this
research are:
(1) Specialised representations: Dynamic scheduling
faces the problem of ever-changing specification,
which cannot be handled by the static architecture
of existing Machine Learning (ML) algorithms. Most
current studies make assumptions to constrain the
scheduling problem to a fixed size. This research
develops abstracted state features and indirect action
representations to resolve this issue. The resulting
approach can manage scheduling problems with
long duration and large number of jobs.
(2) Knowledge-based reward-shaping: Unlike the
usual tasks where DRL has achieved great success,
such as board games and video games, most pro-
duction planning and control problems cannot be
formulated as a task to ‘win’ or ‘survive’, but instead
focus on the constrained optimisation of an objec-
tive that is complex in nature. Reward shaping based
on domain knowledge is vital to strike a balance
between solution quality and implementation cost.
This research develops a surrogate reward-shaping
technique to improve training efficiency and encour-
age cooperation among scheduling agents. It delivers
higher stability and performance than the simple
joint reward approach, enabling good reproducibil-
ity of our work.
(3) Efficient training and application: The lightweight
neural network guarantees quick decision-making
with millisecond-level latency, making real-time
control possible. Our proposed architecture also
exhibits good training efficiency: 100,000 units of
simulation time (training) take up to 20 minutes to
complete, which allows fast iteration to adapt to
the dynamic environment. Scheduling agents can
rapidly acquire new behaviour by incorporating new
experiences to replay memory or tuning parameters.
Even though the proposed architecture is demon-
strated in a small manufacturing system, its effi-
ciency is expected to persist even in larger-scale
systems with the support of stronger computation
capacity.
Research into real-time dynamic scheduling includes
the discovery of features to facilitate decision-making
and the study of operating those features. Compared
to rule-based approaches (manually designed or auto-
matically evolved), DRL algorithms possess stronger
ability to handle high-dimensional input space and
lower sensitivity to increase in feature complexity. Auto-
mated training also enables fast learning with minimal
involvement of human expertise, reducing the cost of
design and development. Most importantly, DRL is a
highly parametrised algorithm, the use of ANN pro-
vides the ability to perceive and manage complex sys-
tem dynamics. More performance improvement can
be expected by incorporating algorithmic innovations
and taking advantage of rapidly increasing computing
power.
DRL can also achieve high quality and responsiveness
simultaneously in a dynamic environment. Although
metaheuristics have long been used to produce practical
‘optimal’ schedules (given the NP-hardness of JSP and
FJSP), however, no knowledge is created along with the
schedule to facilitate the development of a new sched-
ule once disruption occurs. Learning-based approaches
overcome this disadvantage, the parametrised knowledge
can be reused with negligible cost (a forward propagation
in neural network), generalised, and transferred. These
features enable fast prototyping and deployment, and
are core capabilities of an agile production scheduling
system.
There are, however, challenges and opportunities
in transferring the DRL research into the production
domain:
Unlike the tabular or graphical state representation
in applications where DRL has achieved great success,
industrial data is messy and unstructured. The selection,
abstraction, and pre-processing of features still require
human expertise. This process can be facilitated by intro-
ducing innovative data interpretation techniques in ML
research. In addition, shop floor is a complex environ-
ment, study that considers more types of dynamic events
such as stochastic processing time and machine break-
downs would be helpful in bringing DRL closer to real-
life application.
Manual reward-shaping relies on domain expertise
and requires great effort to design and fine-tune. In addi-
tion, actions can have a lasting and widespread influence
that is hard to evaluate using a scalar reward. Incorporat-
ing causal learning and inference with RL to determine
the relationship between action and its influence is a
popular research topic and its potential for industrial
applications is promising.
The cooperative behaviour among intelligent agents
in a cyber-physical manufacturing system is not very
well-studied. Other popular techniques of cooperative
multi-agent RL, such as communication between agents
(Foerster et al. 2016) and automatic value decomposi-
tion (Sunehag et al. 2017), have also proven valuable in
improving the performance of the multi-agent system;

their application to dynamic scheduling, though, still
remains unexplored.
Data availability statement
The data that support the findings of this study are
available from the corresponding author, [Liu, R], upon
reasonable request.
Disclosure statement
No potential conflict of interest was reported by the author(s).
Notes on contributors
Renke Liu received his B.E. degree from
Hunan University, China, in 2015; and
M.S. degree from Nanyang Technolog-
ical University, Singapore, in 2017. He
is currently pursuing his Ph.D. degree
at Nanyang Technological University. His
research interests span the areas of compu-
tational intelligence’s application to pro-
duction planning and control, deep reinforcement learning,
and dynamic scheduling in complex systems.
Dr. Rajesh Piplani is an associate professor
and the director of the M.Sc. program in
Supply chain Engineering in the school of
Mechanical and Aerospace Engineering at
Nanyang Technological University, Singa-
pore. He obtained his Ph. D. from Purdue
University and M.S. from Arizona State
University. He ran the Center for Supply
Chain Management (2002-2009) at NTU, and was also the Pro-
gram Manager, Integrated Manufacturing and Service Systems
(IMSS) for Singapore research funding agency A∗STAR, man-
aging the SGD 8 Million research program for four years. Dr.
Piplani’s interests are in supply chain management of manu-
facturing enterprises, data analytics, inventory planning, and
air traffic control. Dr. Piplani has over seven years of industry
experience in India and USA.
Dr. Carlos Toro received both his Ph.D.
And M.Sc in Computer Science from the
University of the Basque Country (Spain)
and his Bachelor degree in Mechanical
Engineering (with honors) from EAFIT
University (Colombia). In 2003 he moved
to Spain and started working in applied
research focusing on Industrial and
Advanced Manufacturing. At the same time he lectured at
the University of the Basque Country. In 2007 he was invited
researcher at the University of Newcastle (Australia), return-
ing in 2011 with a Marie Curie research visitor grant funded
by the EU Commission. In 2017 he joined A∗STAR Singapore
at their Advanced Remanufacturing and Technology Centre
(ARTC) as lead architect and research scientist for the Factory
of the Future initiative which is the effort of Singapore gov-
ernment for implementing a model factory that includes the
concepts of the 4th industrial revolution. In 2021 he joined
Vicomtech as Research Line Head for Virtual Engineering &
INTERNATIONAL JOURNAL OF PRODUCTION RESEARCH 4067
Simulation where he leads the activities of Vicomtech as Digital
Twin Consortium branch coordinator for Spain.
ORCID
Renke Liu http://orcid.org/0000-0003-4375-9140
References
Aissani, N., B. Beldjilali, and D. Trentesaux. 2008. “Efficient and
effective reactive scheduling of manufacturing system using
Sarsa-multi-objective agents". MOSIM’08: 7th Conference
Internationale de Modelisation et Simulation.
Baer, Schirin, Jupiter Bakakeu, Richard Meyes, and Tobias
Meisen. 2019. Multi-Agent Reinforcement Learning for
Job Shop Scheduling in Flexible Manufacturing Systems.
Paper presented at the 2019 s International Conference
on Artificial Intelligence for Industries (AI4I), 25-27 Sept.
2019.
Bernstein, Daniel S, Robert Givan, Neil Immerman, and
Shlomo Zilberstein. 2002. “The Complexity of Decentral-
ized Control of Markov Decision Processes.” Mathematics of
Operations Research 27 (4): 819–840.
Bouazza, W., Y. Sallez, and B. Beldjilali. 2017. “A Distributed
Approach Solving Partially Flexible job-Shop Scheduling
Problem with a Q-Learning Effect.” IFAC-PapersOnLine 50
(1): 15890–15895. doi:10.1016/j.ifacol.2017.08.2354.
Chen, Jiasi, and Xukan Ran. 2019. “Deep Learning With Edge
Computing: A Review.” Proceedings of the IEEE 107 (8):
1655–1674. doi:10.1109/JPROC.2019.2921977.
Ðurasević, Marko, and Domagoj Jakobović. 2018. “A Sur-
vey of Dispatching Rules for the Dynamic Unrelated
Machines Environment.” Expert Systems with Applications
113: 555–569. doi:10.1016/j.eswa.2018.06.053.
Foerster, Jakob N., Yannis M. Assael, Nando de Freitas, and Shi-
mon Whiteson. 2016. “Learning to Communicate to Solve
Riddles with Deep Distributed Recurrent Q-Networks.”
arXiv preprint arXiv:1602.02672 .
Gabel, Thomas, and Martin Riedmiller. 2012. “Distributed Pol-
icy Search Reinforcement Learning for job-Shop Scheduling
Tasks.” International Journal of Production Research 50 (1):
41–61. doi:10.1080/00207543.2011.571443.
Guh, Ruey-Shiang, Yeou-Ren Shiue, and Tsung-Yuan Tseng.
2011. “The Study of Real Time Scheduling by an Intelligent
Multi-Controller Approach.” International Journal of Pro-
duction Research 49 (10): 2977–2997. doi:10.1080/00207541
003794884.
Gupta, Jayesh K., Maxim Egorov, and Mykel Kochenderfer.
2017. “Cooperative Multi-agent Control Using Deep Rein-
forcement Learning.” Autonomous Agents and Multiagent
Systems. AAMAS 2017. Lecture Notes in Computer Science
10642: 66–83. doi: 10.1007/978-3-319-71682-4_5.
Gupta, Jatinder N. D., Arindam Majumder, and Dipak Laha.
2019. “Flowshop Scheduling with Artificial Neural Net-
works.” Journal of the Operational Research Society 70 (10):
1–19. doi:10.1080/01605682.2019.1621220.
Ho, Nhu Binh, Joc Cing Tay, and Edmund M. K. Lai. 2007.
“An Effective Architecture for Learning and Evolving Flex-
ible job-Shop Schedules.” European Journal of Operational
Research 179 (2): 316–333. doi:10.1016/j.ejor.2006.04.007.
Holthaus, Oliver, and Chandrasekharan Rajendran. 1997. “Effi-
cient Dispatching Rules for Scheduling in a job Shop.”
4068 R. LIU ET AL.
International Journal of Production Economics 48 (1):
87–105. doi: 10.1016/S0925-5273(96)00068-0.
Hubbs, Christian D., Can Li, Nikolaos V. Sahinidis, Igna-
cio E. Grossmann, and John M. Wassick. 2020. “A Deep
Reinforcement Learning Approach for Chemical Produc-
tion Scheduling.” Computers & Chemical Engineering 141:
106982. doi:10.1016/j.compchemeng.2020.106982.
Jun, Sungbum, Seokcheon Lee, and Hyonho Chun. 2019.
“Learning Dispatching Rules Using Random Forest in Flexi-
ble job Shop Scheduling Problems.” International Journal of
Production Research 57 (10): 3290–3310. doi:10.1080/00207
543.2019.1581954.
Kuhnle, Andreas, Nicole Röhrig, and Gisela Lanza. 2019.
“Autonomous Order Dispatching in the Semiconductor
Industry Using Reinforcement Learning.” Procedia CIRP 79:
391–396. doi:10.1016/j.procir.2019.02.101.
Kuhnle, Andreas, Louis Schäfer, Nicole Stricker, and Gisela
Lanza. 2019. “Design, Implementation and Evaluation of
Reinforcement Learning for an Adaptive Order Dispatch-
ing in job Shop Manufacturing Systems.” Procedia CIRP 81:
234–239. doi:10.1016/j.procir.2019.03.041.
Lang, Sebastian, Fabian Behrendt, Nico Lanzerath, Tobias
Reggelin, and Marcel Müller. 2020. "Integration of Deep
Reinforcement Learning and Discrete-Event Simulation for
Real-Time Scheduling of a Flexible Job Shop Produc-
tion." 2020 Winter Simulation Conference (WSC):3057-68.
doi:10.1109/WSC48552.2020.9383997.
Li, Xinyu, and Liang Gao. 2020. “A Hybrid Genetic Algorithm
and Tabu Search for Multi-Objective Dynamic jsp.” In Effec-
tive Methods for Integrated Process Planning and Schedul-
ing. Engineering Applications of Computational Methods. Vol.
2, edited by Xinyu Li and Liang Gao, 377–403. Berlin, Hei-
delberg: Springer.
Lin, Chun-Cheng, Der-Jiunn Deng, Yen-Ling Chih, and Hsin-
Ting Chiu. 2019. “Smart Manufacturing Scheduling With
Edge Computing Using Multiclass Deep Q Network.” IEEE
Transactions on Industrial Informatics 15 (7): 4276–4284.
doi:10.1109/TII.2019.2908210.
Liu, Chien-Liang, Chuan-Chin Chang, and Chun-Jan Tseng.
2020. “Actor-critic Deep Reinforcement Learning for Solv-
ing job Shop Scheduling Problems.” IEEE Access 8: 71752–
71762. doi:10.1109/ACCESS.2020.2987820.
Liu, Feng, Shengbin Wang, Yong Hong, and Xiaohang Yue.
2017. “On the Robust and Stable Flowshop Schedul-
ing Under Stochastic and Dynamic Disruptions.” IEEE
Transactions on Engineering Management 64 (4): 539–553.
doi:10.1109/TEM.2017.2712611.
Luo, Shu. 2020. “Dynamic Scheduling for Flexible job Shop
with new job Insertions by Deep Reinforcement Learning.”
Applied Soft Computing 91: 106208. doi:10.1016/j.asoc.2020.
106208.
Luo, Peng Cheng, Huan Qian Xiong, Bo Wen Zhang, Jie Yang
Peng, and Zhao Feng Xiong. 2021. “Multi-resource Con-
strained Dynamic Workshop Scheduling Based on Proximal
Policy Optimisation.” International Journal of Production
Research. doi: 10.1080/00207543.2021.1975057.
Malus, Andreja, Dominik Kozjek, and Rok Vrabič. 2020. “Real-
time Order Dispatching for a Fleet of Autonomous Mobile
Robots Using Multi-Agent Reinforcement Learning.” CIRP
Annals 69 (1): 397–400. doi: 10.1016/j.cirp.2020.04.001.
Matloff, Norm. 2008. Introduction to discrete-event simulation
and the simpy language.
Messaoud, Seifeddine, Abbas Bradai, Syed Hashim Raza
Bukhari, Pham Tran Anh Quang, Olfa Ben Ahmed, and
Mohamed Atri. 2020. “A Survey on Machine Learning in
Internet of Things: Algorithms, Strategies, and Applica-
tions.” Internet of Things 12: 100314. doi:10.1016/j.iot.2020.
100314.
Mnih, Volodymyr, Koray Kavukcuoglu, David Silver, Andrei
A. Rusu, Joel Veness, Marc G. Bellemare, Alex Graves, et al.
2015. “Human-level Control Through Deep Reinforcement
Learning.” Nature 518 (7540): 529–533. doi:10.1038/nature
14236.
Mouelhi-Chibani, Wiem, and Henri Pierreval. 2010. “Train-
ing a Neural Network to Select Dispatching Rules in Real
Time.” Computers & Industrial Engineering 58 (2): 249–256.
doi:10.1016/j.cie.2009.03.008.
Olafsson, Sigurdur, and Xiaonan Li. 2010. “Learning Effec-
tive new Single Machine Dispatching Rules from Opti-
mal Scheduling Data.” International Journal of Produc-
tion Economics 128 (1): 118–126. doi:10.1016/j.ijpe.2010.
06.004.
Ouelhadj, Djamila, and Sanja Petrovic. 2008. “A Survey of
Dynamic Scheduling in Manufacturing Systems.” Journal of
Scheduling 12 (4): 417. doi:10.1007/s10951-008-0090-8.
Park, Junyoung, Jaehyeong Chun, Sang Hun Kim, Youngkook
Kim, and Jinkyoo Park. 2021. “Learning to Schedule job-
Shop Problems: Representation and Policy Learning Using
Graph Neural Network and Reinforcement Learning.” Inter-
national Journal of Production Research 59 (11): 3360–3377.
doi:10.1080/00207543.2020.1870013.
Paszke, Adam, Sam Gross, Francisco Massa, Adam Lerer, James
Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin,
Natalia Gimelshein, and Luca Antiga. 2019. Pytorch: An
imperative style, high-performance deep learning library.
Paper presented at the Advances in neural information pro-
cessing systems.
Pickardt, Christoph, Jurgen Branke, Torsten Hildebrandt, Jens
Heger, and Bernd Scholz-Reiter. 2010. Generating dispatch-
ing rules for semiconductor manufacturing to minimize
weighted tardiness. Paper presented at the 2010 Winter Sim-
ulation Conference.
Priore, Paolo, José Parreño, Raúl Pino, Alberto Gómez,
and Javier Puente. 2010. “Learning-based Scheduling of
Flexible Manufacturing Systems Using Support Vector
Machines.” Applied Artificial Intelligence 24 (3): 194–209.
doi:10.1080/08839510903549606.
Qu, Shuhui, Jie Wang, and Juergen Jasperneite. 2019. "Dynamic
scheduling in modern processing systems using expert-
guided distributed reinforcement learning." In 2019 24th
IEEE International Conference on Emerging Technologies and
Factory Automation (ETFA), 459-66.
Ren, Weibo, Yan Yan, Yaoguang Hu, and Yu Guan. 2021.
“Joint Optimisation for Dynamic Flexible job-Shop Schedul-
ing Problem with Transportation Time and Resource Con-
straints.” International Journal of Production Research. doi:
10.1080/00207543.2021.1968526..
Sels, Veronique, Nele Gheysen, and Mario Vanhoucke. 2012. “A
Comparison of Priority Rules for the job Shop Scheduling
Problem Under Different Flow Time-and Tardiness-Related
Objective Functions.” International Journal of Production
Research 50 (15): 4255–4270.
Shi, Daming, Wenhui Fan, Yingying Xiao, Tingyu Lin, and Chi
Xing. 2020. “Intelligent Scheduling of Discrete Automated

Production Line via Deep Reinforcement Learning.” Inter-
national Journal of Production Research 58 (11): 3362–3380.
doi:10.1080/00207543.2020.1717008.
Shiue, Yeou-Ren. 2009. “Data-mining-based Dynamic Dis-
patching Rule Selection Mechanism for Shop Floor Control
Systems Using a Support Vector Machine Approach.” Inter-
national Journal of Production Research 47 (13): 3669–3690.
doi:10.1080/00207540701846236.
Sunehag, Peter, Guy Lever, Audrunas Gruslys, Wojciech Mar-
ian Czarnecki, Vinicius Zambaldi, Max Jaderberg, Marc
Lanctot, et al. 2017. “Value-Decomposition Networks For
Cooperative Multi-Agent Learning.” arXiv preprint arXiv:
1706.05296.
Ulyanov, Dmitry, Andrea Vedaldi, and Victor Lempitsky. 2016.
“Instance Normalization: The Missing Ingredient for Fast
Stylization.” arXiv preprint arXiv:1607.06450.
Van Hasselt, Hado, Arthur Guez, and David Silver. 2016. Deep
reinforcement learning with double q-learning. Paper pre-
sented at the Thirtieth AAAI conference on artificial intelli-
gence.
van Otterlo, Martijn, and Marco Wiering. 2012. Reinforce-
ment Learning and Markov Decision Processes, Reinforcement
Learning: State-of-the-art. Berlin: Springer Berlin Heidel-
berg.
Wang, Yu-Fang. 2020. “Adaptive job Shop Scheduling Strategy
Based on Weighted Q-Learning Algorithm.” Journal of Intel-
ligent Manufacturing 31 (2): 417–432. doi:10.1007/s10845-
018-1454-3.
Wang, Haoxiang, Bhaba R. Sarker, Jing Li, and Jian Li. 2020.
“Adaptive Scheduling for Assembly job Shop with Uncer-
tain Assembly Times Based on Dual Q-Learning.” Interna-
tional Journal of Production Research 59 (19): 5867–5883.
doi:10.1080/00207543.2020.1794075.
Wang, Yi-Chi, and John M. Usher. 2005. “Application of Rein-
forcement Learning for Agent-Based Production Schedul-
ing.” Engineering Applications of Artificial Intelligence 18 (1):
73–82. doi:10.1016/j.engappai.2004.08.018.
Waschneck, Bernd, André Reichstaller, Lenz Belzner, Thomas
Altenmüller, Thomas Bauernhansl, Alexander Knapp, and
Andreas Kyek. 2018. “Optimization of Global Production
Scheduling with Deep Reinforcement Learning.” Procedia
CIRP 72: 1264–1269. doi:10.1016/j.procir.2018.03.212.
Weckman, Gary R., Chandrasekhar V. Ganduri, and David
A. Koonce. 2008. “A Neural Network job-Shop Sched-
uler.” Journal of Intelligent Manufacturing 19 (2): 191–201.
doi:10.1007/s10845-008-0073-9.
INTERNATIONAL JOURNAL OF PRODUCTION RESEARCH 4069
Xiong, Hegen, Huali Fan, Guozhang Jiang, and Gongfa Li.
2017. “A Simulation-Based Study of Dispatching Rules
in a Dynamic job Shop Scheduling Problem with Batch
Release and Extended Technical Precedence Constraints.”
European Journal of Operational Research 257 (1): 13–24.
doi:10.1016/j.ejor.2016.07.030.
Yang, Shengluo, and Zhigang Xu. 2021. “Intelligent Scheduling
and Reconfiguration via Deep Reinforcement Learning in
Smart Manufacturing.” International Journal of Production
Research. doi: 10.1080/00207543.2021.1943037.
Yska, Daniel, Yi Mei, and Mengjie Zhang. 2018. Genetic pro-
gramming hyper-heuristic with cooperative coevolution for
dynamic flexible job shop scheduling. Paper presented at the
European Conference on Genetic Programming, Cham.
Zang, Zelin, Wanliang Wang, Yuhang Song, Linyan Lu, Weikun
Li, Yule Wang, and Yanwei Zhao. 2019. “Hybrid Deep
Neural Network Scheduler for job-Shop Problem Based
on Convolution two-Dimensional Transformation.” Com-
putational Intelligence and Neuroscience 2019: 7172842.
doi:10.1155/2019/7172842.
Zhang, Fangfang, Yi Mei, Su Nguyen, and Mengjie Zhang.
2020. “Evolving Scheduling Heuristics via Genetic Pro-
gramming With Feature Selection in Dynamic Flexible Job-
Shop Scheduling.” IEEE Transactions on Cybernetics 51 (4):
1797–1811. doi:10.1109/TCYB.2020.3024849.
Zhang, Fangfang, Yi Mei, and Mengjie Zhang. 2018. Genetic
programming with multi-tree representation for dynamic
flexible job shop scheduling. Paper presented at the Aus-
tralasian Joint Conference on Artificial Intelligence.
Zhang, Fangfang, Yi Mei, and Mengjie Zhang. 2019. "A two-
stage genetic programming hyper-heuristic approach with
feature selection for dynamic flexible job shop scheduling."
In Proceedings of the Genetic and Evolutionary Computation
Conference, 347–55. Prague, Czech Republic: Association for
Computing Machinery.
Zhang, Rui, Shiji Song, and Cheng Wu. 2020. “Robust
Scheduling of hot Rolling Production by Local Search
Enhanced ant Colony Optimization Algorithm.” IEEE
Transactions on Industrial Informatics 16 (4): 2809–2819.
doi:10.1109/TII.2019.2944247.
Zhang, Tao, Shufang Xie, and Oliver Rose. 2017. Real-time job
shop scheduling based on simulation and Markov decision
processes. Paper presented at the 2017 Winter Simulation
Conference (WSC), Las Vegas, Nevada.
Zhou, MengChu. 2012. Petri Nets in Flexible and Agile Automa-
tion. Vol. 310. Springer Science & Business Media.