International Journal of Production Research

ISSN: (Print) (Online) Journal homepage: www.tandfonline.com/journals/tprs20

A review on reinforcement learning algorithms

and applications in supply chain management

Benjamin Rolf, Ilya Jackson, Marcel Müller, Sebastian Lang, Tobias Reggelin
& Dmitry Ivanov

To cite this article: Benjamin Rolf, Ilya Jackson, Marcel Müller, Sebastian Lang, Tobias Reggelin
& Dmitry Ivanov (2023) A review on reinforcement learning algorithms and applications in
supply chain management, International Journal of Production Research, 61:20, 7151-7179,
DOI: 10.1080/00207543.2022.2140221

To link to this article: https://doi.org/10.1080/00207543.2022.2140221

© 2022 The Author(s). Published by Informa

UK Limited, trading as Taylor & Francis
Group

Published online: 03 Nov 2022.

Submit your article to this journal

Article views: 18375

View related articles

View Crossmark data

Citing articles: 22 View citing articles

Full Terms & Conditions of access and use can be found at

https://www.tandfonline.com/action/journalInformation?journalCode=tprs20
INTERNATIONAL JOURNAL OF PRODUCTION RESEARCH
2023, VOL. 61, NO. 20, 7151–7179
https://doi.org/10.1080/00207543.2022.2140221

A review on reinforcement learning algorithms and applications in supply chain

management
Benjamin Rolf a , Ilya Jackson b , Marcel Müller a , Sebastian Lang c , Tobias Reggelin a and Dmitry Ivanovd
a Otto-von-Guericke-University Magdeburg, Magdeburg, Germany; b Massachusetts Institute of Technology, Center for Transportation &

Logistics, Cambridge, MA, USA; c Fraunhofer Institute for Factory Operation and Automation IFF, Magdeburg, Germany; d Berlin School of
Economics and Law, Berlin, Germany

ABSTRACT ARTICLE HISTORY

Decision-making in supply chains is challenged by high complexity, a combination of continuous Received 17 June 2022
and discrete processes, integrated and interdependent operations, dynamics, and adaptability. The Accepted 13 October 2022
rapidly increasing data availability, computing power and intelligent algorithms unveil new poten- KEYWORDS
tials in adaptive data-driven decision-making. Reinforcement Learning, a class of machine learning Reinforcement learning;
algorithms, is one of the data-driven methods. This semi-systematic literature review explores the supply chain management;
current state of the art of reinforcement learning in supply chain management (SCM) and proposes literature review; inventory
a classification framework. The framework classifies academic papers based on supply chain drivers, management; machine
algorithms, data sources, and industrial sectors. The conducted review revealed a few critical insights. learning; artificial
First, the classic Q-learning algorithm is still the most popular one. Second, inventory management intelligence
is the most common application of reinforcement learning in supply chains, as it is a pivotal element
of supply chain synchronisation. Last, most reviewed papers address toy-like SCM problems driven
by artificial data. Therefore, shifting to industry-scale problems will be a crucial challenge in the next
years. If this shift is successful, the vision of data-driven decision-making in real-time could become
a reality.

1. Introduction in its turn, used by plastics manufacturers to derive

Supply chains operate in increasingly complex and uncer-
tain environments. Adaptive planning and control in
such environments is of utmost importance to ensure
delivery to end customers with minimal delays and inter-
ruptions avoiding unnecessary costs and maintaining
business continuity. For implementation of adaptation-
based management principles, real-time coordination of
production scheduling, inventory control, and delivery
plans is required whereas control parameters of the sys-
tem need to be dynamically adjusted toward minimis-
ing costs, maximising revenue, satisfying a target service
level, or pursuing any other quantifiable objective with
the consideration of the dynamics, nonstationarity, and
uncertainty (Kim et al. 2005; Ivanov and Sokolov 2013;
Emerson, Zhou, and Piramuthu 2009).
The global interconnectedness and complexity may
result in a lack of visibility and risks of devastating
disruptions. An explosion at the Evonik Factory is a
notable example. In 2012, the explosion, followed by a
fire, destroyed the chemical factory in Marl, Germany.
The factory produced cyclododecatriene, used by the
chemical industry to make laurolactam. Laurolactam is,
polyamide-12, a plastic essential for strong, lightweight
components. Polyamide-12 is in the bill of materials of
any car, scattered across thousands of different parts and
manufactured by a multitude of different suppliers. The
accident created the ripple effect and threatened to dis-
rupt the entire automobile industry. Only collaboration
among competing automakers and dozens of suppliers
prevented the catastrophe (Sheffi 2020). Other exam-
ples of complexity include the semiconductor supply
chains (Khan, Mann, and Peterson 2021) and the sup-
ply chains behind vaccine production and distribution
(Sheffi 2021). Given high levels of complexity, supply
chains are prone to disruptions and suboptimal perfor-
mance caused by operational failures and information
miscoordination. The most notable examples of disrup-
tive phenomena include bullwhip and ripple effects. The
bullwhip effect, also widely known as the Forrester effect,
can be defined as the amplification of demand varia-
tion on production and order quantities as they propa-
gate downstream in supply chains (Xun Wang and Dis-
ney 2016). On the other hand, the ripple effect occurs
when a disruption, rather than being localised within

CONTACT Benjamin Rolf benjamin.rolf@ovgu.de Otto-von-Guericke-University Magdeburg, Universitätsplatz 2, Magdeburg, Germany

© 2022 The Author(s). Published by Informa UK Limited, trading as Taylor & Francis Group
This is an Open Access article distributed under the terms of the Creative Commons Attribution-NonCommercial-NoDerivatives License (http://creativecommons.
org/licenses/by-nc-nd/4.0/), which permits non-commercial re-use, distribution, and reproduction in any medium, provided the original work is properly cited, and is not
altered, transformed, or built upon in any way.
7152 B. ROLF ET AL.
one part of the supply chain, cascades downstream and
undermines the performance of the entire supply chain
(Dolgui, Ivanov, and Rozhkov 2019). Both effects cause
significant problems for supply chain managers because
they eventually give rise to over- and under-production
cycles, leading to excess inventory levels, potential stock-
outs, and suboptimal network performance. These prob-
lems may be further worsened if structural and opera-
tional vulnerabilities in the supply chain are intercon-
nected (Ivanov 2020a). Besides, the ongoing COVID-19
pandemic demonstrated a new kind of disruption, char-
acterised by the long-term presence and unpredictable
scale (Ivanov 2020b). Since the severe impact of such
disruptions cannot be easily mitigated, supply chain par-
ticipants require recovery planning and adaptation in
the presence of disruption (Ivanov 2021b, 2022a, 2022c).
In this regard, the full potential of the supply chain is
unlocked if and only if it becomes synchronised, namely,
all the critical stakeholders obtain accurate real-time
data, identify weaknesses, streamline processes, and mit-
igate risk.
Decision-making in Supply Chain Management
(SCM) is challenged by high complexity, a combina-
tion of continuous and discrete processes, integrated and
interdependent operations, as well as dynamics, result-
ing in requirements for adaptability (Ivanov, Dolgui, and
Sokolov 2012). Reinforcement Learning (RL), a class of
machine learning algorithms specialising in sequential
decision-making, is a promising candidate solution to
address these challenges. RL determines how to take
actions in an environment to maximise the reward over
time and, thus, can serve as an adaptive controller for
such complex systems. The ultimate goal of such a con-
troller (also known as an RL agent) is to learn the best
possible control actions in each possible state of the
dynamic system to maximise long-term system objec-
tives (Boute et al. 2022).
The last decade manifested itself in growing data
quantities and processing power. This trend eventually
resulted in the advent of deep learning, a machine learn-
ing approach capable of leveraging immense computa-
tional resources and capitalising on massive amounts of
data (Bengio 2016). These facts, along with the advent
of new algorithmic techniques, mature software pack-
ages, and strong interest from business, resulted in
deep reinforcement learning (DRL), the promising com-
bination of reinforcement learning and deep learning
(Krakovsky 2016). Despite its novelty, at this point, DRL
has already demonstrated remarkable performance in the
fields of road traffic navigation (Vinitsky et al. 2020),
autonomous vehicles (Isele et al. 2018), and robotics
(Gu et al. 2017). However, it is essential to empha-
sise that RL agents can not learn directly from the
physical world. Therefore all these applications require
a virtual environment or simulation, a dynamic testbed
for learning through trial and error (MacCarthy and
Ivanov 2022a, 2022b). In SCM, Supply Chain Digital
Twin seems to be the most promising candidate for
such a virtual testbed environment. The combination of
model-based and data-driven approaches by incorporat-
ing real-time data from physical and cyber sources can
ensure end-to-end visibility and permanent information
accessibility for an RL agent (Ivanov and Dolgui 2021;
Burgos and Ivanov 2021). Since such emerging digi-
tal technologies as the Internet of Things, Blockchain,
5G, cloud and edge computing make the Supply Chain
Digital Twin implementable by enabling real-time con-
nectivity and visibility at a highly granular level (Dol-
gui and Ivanov 2022; Ivanov, Dolgui, and Sokolov 2022;
Ivanov 2021a; Choi et al. 2022; Zhang, MacCarthy, and
Ivanov 2022), we can expect an increase in popularity
and faster adoption rate of the DRL applications in supply
chains.
In light of these facts, the aim of this article is to iden-
tify the main RL algorithms, and their application to SCM
using a structured review methodology. For this purpose,
the article addresses three research questions:
(1) What are the main applications of reinforcement
learning in SCM?
(2) What are the main algorithms for reinforcement
learning in SCM?
(3) How widespread is reinforcement learning for solv-
ing industrial use cases?
To answer the research questions, we first introduce
the general framework of RL and how this framework
would fit into the supply chain context. Then, we provide
a detailed breakdown of the existing studies and classify
them by the supply chain drivers, algorithms used, data
sources, and industrial sectors. Last, we provide critical
managerial insights, discuss the current challenges, and
shed light on future research avenues that may elevate the
current state of the art of RL applications for SCM. The
remainder of this article is organised as follows. Section 2
presents the review methodology and the procedure of
selecting relevant literature. Section 3 introduces the RL
paradigm and its mathematical foundations with refer-
ence to SCM. In Section 4 we perform a descriptive analy-
sis to show relevant metrics of the identified publications
such as subject areas, publication years and countries of
origin. In addition, we show the bibliometric networks of
citations and keywords to identify the most relevant pub-
lications. Section 5 presents a classification framework
that categorises the publications according to four char-
acteristics: supply chain drivers, algorithms, data sources,

Figure 1. Literature review methodology.
and industrial sectors. Section 6 provides a discussion
of the findings from the classification. Finally, Section 7
gives a critical analysis with a special focus on research
gaps and managerial insights for production managers in
the industry.
2. Review methodology
Outlining the state of knowledge for RL in SCM requires
a structured review methodology. Snyder (2019) defines
different review methodologies that depend on the
review’s objectives and the research discipline. In this
case, the semi-systematic type, a mix of quantitative and
qualitative analysis, is most suitable. A semi-systematic
literature review is beneficial when the topic is not clearly
delimited and has been studied in different research dis-
ciplines (Wong et al. 2013). It aims to present current
applications and the general state of knowledge to show
an agenda for further research (Snyder 2019).
The initial assessment revealed that the considered
topic still has very few contributions. Therefore, it is pos-
sible to include all relevant literature for the most part and
analyse it systematically. Figure 1 shows the methodology
for this literature review. After the research questions had
INTERNATIONAL JOURNAL OF PRODUCTION RESEARCH 7153
been defined, the database for the systematic part of the
literature review was selected. We only included Scopus
in the literature review as it is the most extensive database
for peer-reviewed scientific literature and highly overlaps
with other databases such as Web of Science and Google
Scholar. In addition, Scopus ensures a certain scientific
standard as it only includes peer-reviewed literature com-
pared to Google Scholar (Mongeon and Paul-Hus 2015).
The search term ‘reinforcement AND learning AND
supply AND chain’ yielded 165 publications that were
checked for violating at least one of the three exclu-
sion criteria. The first exclusion criterion removed empty
proceedings documents as the search yielded the actual
publication and the entire conference proceedings. One
publication was removed because Scopus remarked a vio-
lation of good scientific practice and recommended not
to cite the publication. Third, the authors excluded pub-
lications not relevant to the considered topic. Removal
due to thematic irrelevance required the consent of at
least two authors and was conducted after reading the
abstracts. After applying the exclusion criteria, 103 pub-
lications remained that the authors read in detail. The
number of publications (sample size n) varies slightly in
the following sections because not all publications fit all
7154 B. ROLF ET AL.

classification frameworks. We mention the sample size in

each section and explain why some publications had to
be excluded or classified into multiple categories.

3. Reinforcement learning
In contrast to such popular machine learning paradigms
as supervised and unsupervised learning, reinforcement
learning (RL) is akin to instrumental conditioning in
physiology (Sutton and Barto 2018). In instrumental con- Figure 2. MDP in the SCM context.
ditioning, animals learn associations between stimulus
and response. Such that given a stimulus (or environ-
mental state), the animal tries a response (or action). If
if there is a long trajectory of states and actions prior
the response outcome for a given stimulus is positive, the
to the received reward, it is unclear how to determine
connection between the stimulus and response is rein-
which actions should get credit for the eventual reward.
forced (Thorndike 1898). The RL paradigm is driven
Since it is often unclear whether the performed action
by the idea that intelligent systems can learn through
actually contributed to the gained reward, a standard
a similar process of trial and error (Michie and Cham-
solution is to apply n-step discounted returns, where the
bers 1968). This idea appears to be extremely powerful
cumulative rewards for the action at for n steps are expo-
when applied to the adaptive control of highly complex
nentially weighted by a discount factor γ ∈ [0, 1] as in
and dynamic systems such as supply chains (Kegenbekov
Equation (1). Nevertheless, MDP is a powerful frame-
and Jackson 2021).
work for goal-oriented autonomous learning. An adap-
Formally, RL can be defined as a Markov Decision Pro-
tive controller based on this principle can learn how to
cess (MDP). The MDP serves as the flexible framework
make optimal or nearly-optimal decisions in dynamic
for goal-directed learning that can be described as a tuple
and stochastic supply chain environments (Figure 2).
M = (S , A, P (st+1 , r | s, a), R), γ ), where:
The supply chain environment can become too com-
plex for the classic reinforced learning approach. This
• s ∈ S is a set of the possible states of the environment.
is where deep reinforcement learning (DRL) comes into
• a ∈ A is the set of possible actions that can be exe-
play. The ‘deep’ part refers to the application of an
cuted by an agent to interact with the environment.
artificial neural network to estimate possible sequences
• P (st+1 , r | s, a) denotes the transition probability to
of actions and associate them with long-term rewards,
move to the state st+1 receiving a reward r, given st ∈
increasing the manageability of solution space. Besides,
S and at ∈ A
since agents based on DRL do not store all state and value
• R ∈ R is the expected reward received from the envi-
pairs in a table, they become capable of generalising the
ronment after the agent performs action a at state s
value of states that have not been seen before or, rigor-
(Adi et al. 2020).
ously speaking, have not been encountered during the
training phase.
The sequence of states, actions, and rewards produce
a trajectory s0 , a0 , R1 , s1 , a1 , R2 , s2 , a2 , R3 , . . . , sn , where
sn stands for the terminal state. The goal of the RL agent
4. Analysis and findings
is to find the optional policy π : S × A that maps states
into actions so that the cumulative expected return over The publications considered are mostly conference
the time horizon is maximised. The expected return over papers or journal articles. Figure 3 shows the different
a finite time horizon can be defined as follows: proportions of the investigated publication in terms of
∞  document type and the subject area. Scopus assigns two
 subject areas to each document, so the analysis includes
Rt = E k
γ rt+k (1)
206 data points. The most significant proportion of doc-
k=0
uments, 34.5%, was from computer science, followed by
where γ ∈ [0, 1] is a discount factor determining how far engineering (25.7%) and mathematics (12.1%). Despite
the agent should look into the future. its small share, surprisingly large was the field of chemical
It is worth mentioning that the MDP framework is engineering (3.4%). Although SCM is an essential topic
not perfect. For example, it does not address the cru- in economics research, purely economic subject areas are
cial credit-assignment problem (Minsky 1961). Namely, underrepresented.

Figure 3. Classification and proportion of investigated publications in terms of document type and subject area. (a) Type of the
publications (n = 103) and (b) Subject area of the publications (n = 206).
Figure 4. Publications by year (n = 101).
The first publication on RL in SCM was published in
2000. Since then, the number of publications has tended
to increase. This observation corresponds to the general
trend that the number of publications is increasing in
almost all research disciplines (Ware and Mabe 2015).
Figure 4 shows the considered publications by year. Due
to the low number of publications per year, the numbers
are highly volatile. The authors can explain neither the
local increase in 2008 nor the extreme jump in published
documents from 2018 to 2019. Individual circumstances
and statistical fluctuations are therefore assumed. The
year 2022 was not considered fully, so two publications
were excluded from the figure, reducing the total number
to 101.
We used VOSviewer to investigate further the consid-
ered bibliometric network (van Eck and Waltman 2009).
The data basis was a CSV export of the 103 selected
literature sources from the Scopus database. There is a
strong linkage between occurring keywords for the con-
sidered documents. This linkage is probably due to the
relatively small number of publications on the literature’s
specific topic. The analysis considers only keywords that
appeared in at least five documents. Of 910 keywords, 46
meet this threshold. 186 keywords are mentioned at least
two times. Figure 5 shows the bibliometric network of
the 46 keywords linked with co-occurring keywords. The
size of the circles represents the number of occurrences of
INTERNATIONAL JOURNAL OF PRODUCTION RESEARCH 7155
the keyword. The keywords ‘supply chain management’,
‘supply chains’, and ‘reinforcement learning’ appear most
frequently, as they are similar to the search terms of the
review. We classify the keywords of three main clusters
presented in Figure 5 according to the approach of Ivanov
et al. (2021):
(1) Supply chain drivers and algorithms (red colour)
(a) Supply chain drivers (e.g. supply chain man-
agement, supply chains, supply chain, inventory
management, inventory control)
(b) Artificial intelligence (e.g. reinforcement learn-
ing, machine learning, deep reinforcement learn-
ing, deep learning, learning algorithms, reinforce-
ment learning techniques)
(2) Methods and data (green colour), e.g. multi agent sys-
tems, heuristic methods, computer simulation, mathe-
matical models, learning systems, production control,
education, Markov processes
(3) Applications (blue colour), e.g. commerce, sales, man-
ufacture, costs
These classifications will be used for structuring our
further analyses.
5. Classification framework
The review of publications revealed that RL models
in SCM significantly differ in their settings, assump-
tions, aims and technologies. From these differences,
we derived four criteria that are possible to map in
a classification framework and support answering the
research questions. These criteria are supply chain driver,
algorithm, data source and industrial sector. The first cri-
terion captures the supply chain setting and the aim of
the models because supply chain drivers directly affect
the supply chain performance and define the target of the
optimisation model. The second criterion assesses the RL
7156 B. ROLF ET AL.
Figure 5. Visualization of the bibliometric network of keywords with VOSviewer (n = 46).
model from a technical point of view and evaluates the
underlying algorithms and technologies. The classifica-
tion of data sources and the industrial sectors is necessary
to assess the dissemination of RL in real-world supply
chains.
The proposed classification framework is hierarchic
with three levels: criteria, classes and subclasses. The
classes originate from well-recognised academic and
industrial standard classification systems, which are the
supply chain decision-making framework (Chopra and
Meindl 2013), the OpenAI taxonomy of RL algorithms
(Achiam 2018) and the International Standard Industrial
Classification (ISIC) (UNO 2008). In the case of the Ope-
nAI taxonomy and the ISIC, the frameworks included
subclasses that could be used for this classification frame-
work. We also propose novel subclasses representing
the range of reviewed publications for the supply chain
drivers and data sources. We only include subclasses to
which at least one publication was assigned. However, it
is possible to add new subclasses in the future. The fol-
lowing subsections describe the classes and subclasses of
each criterion.
5.1. Supply chain drivers
Every supply chain aims to maximise the overall value
generated, which is the difference between the value of
the final product and the costs incurred in the sup-
ply chain (Chopra and Meindl 2013). There is not only
one way to achieve this objective but various inter-
connected supply chain drivers that impact the over-
all value generated. Chopra and Meindl (2013) define
six drivers that influence the supply chain performance:
facilities, inventory, transportation, information, sourc-
ing, and pricing. This section derives a classification
framework for RL models from these drivers. As RL
is commonly used to address optimisation problems, it
usually adjusts one or more of the drivers so that their set-
ting leads to improved supply chain performance. How-
ever, it is impossible to represent all drivers in a sin-
gle model. Instead, most identified publications consider
firmly delimited supply chain optimisation problems and
target selected drivers. Reviewing the targeted driver of
each model allows for the classification of them and
deriving of applications later on. Table 1 maps each con-
sidered publication to one of the supply chain drivers.
Almost all the publications implemented an RL soft-
ware model and presented quantitative results. There-
fore, assigning most publications to one supply chain
driver is possible considering the input variables and
performance metrics. In the rare cases where the pub-
lication considered multiple supply chain drivers, the
publication was categorised according to the most crucial
driver. The last class includes the remaining publications,
 INTERNATIONAL JOURNAL OF PRODUCTION RESEARCH 7157

Table 1. Classification of publications according to supply chain drivers.

Supply chain driver Publications
Inventory
Customer-driven replenishment Ganesan, Sundararaj, and Srinivas (2021), Meisheri et al. (2021), Zwaida, Pham, and
Beauregard (2021), Barat et al. (2020), Singi et al. (2020), Vanvuchelen, Gijsbrechts, and
Boute (2020), Barat et al. (2019), Lee and Sikora (2019), Dogan and Güner (2015), Ghorbel,
Addouche, and El Mhamedi (2015), Li et al. (2015), Yang and Zhang (2015), Mehta and
Yamparala (2014), Chodura, Dominik, and Koźlak (2011), Jiang and Sheng (2009), Xu,
Zhang, and Liu (2009), Kim, Kwon, and Baek (2008), Kim et al. (2005), Ravulapati, Rao, and
Das (2004), and Rao, Ravulapati, and Das (2003)
Supplier-driven replenishment Hachaichi, Chemingui, and Affes (2020), Tariq Afridi et al. (2020), Yang and Zhang (2015),
Zarandi, Moosavi, and Zarinbal (2013), Zhang, Xu, and Zhang (2013), Sui, Gosavi, and
Lin (2010), Xu, Zhang, and Liu (2009), Jiang (2008), Kim, Kwon, and Baek (2008), Kwon
et al. (2008), Li, Guo, and Zuo (2008), Back, Kim, and Kwon (2006), Kim et al. (2005), and Lin
and Pai (2000)
Global order management with complete information Chen et al. (2021), Kegenbekov and Jackson (2021), Perez et al. (2021), F. Wang and
Lin (2021), Alves and Mateus (2020), Bharti, Kurian, and Pillai (2020), Peng et al. (2019),
Zhou and Zhou (2019), Zhou, Purvis, and Muhammad (2016), Sun, Zhao, and Yin (2010),
Chaharsooghi, Heydari, and Zegordi (2008), Zhang and Bhattacharyya (2007), Giannoccaro
and Pontrandolfo (2002), and Pontrandolfo et al. (2002)
Global order management with incomplete information Mortazavi, Khamseh, and Azimi (2015), Saitoh and Utani (2013), Valluri, North, and
MacAl (2009), Sheremetov and Rocha-Mier (2008), Zhang and Bhattacharyya (2007), Van
Tongeren et al. (2007), Sun et al. (2006), Gang and Ruoying (2006), Sheremetov, Rocha-Mier,
and Batyrshin (2005), Sheremetov and Rocha-Mier (2004), and Bukkapatnam and Gao (2000)
Supply chain scheduling Lu et al. (2021), Lee and Sikora (2019), Marandi and Fatemi Ghomi (2019), Aissani et al. (2012),
Dahlem and Harrison (2010), Sheremetov and Rocha-Mier (2008), Li and Zhao (2006),
Sheremetov, Rocha-Mier, and Batyrshin (2005), Simsek, Albayrak, and Korth (2004),
Sheremetov and Rocha-Mier (2004), Cao, Xi, and Smith (2003), and Ren, Chai, and Liu (2002)
Transportation
Vehicle routing Gutierrez-Franco, Mejia-Argueta, and Rabelo (2021), Li et al. (2021), and Habib, Khan, and
Uddin (2017)
Vehicle scheduling Adi, Bae, and Iskandar (2021), and Bretas et al. (2019)
Transportation bidding Zhanguo (2008), and Tang and Kumara (2005)
Platooning configuration Puskás, Budai, and Bohács (2020)
Network flow routing Hwangbo and Yoo (2018)
Information
Forecasting Chien, Lin, and Lin (2020), Makridis et al. (2020), Zhang et al. (2019), Ko et al. (2011), and
Reindorp and Fu (2011)
Collaboration Xiang (2020), Li et al. (2014), Zhao, Jiang, and Feng (2010), Kaihara and Fujii (2008), and
Dangelmaier et al. (2006)
Human behaviour evaluation Ghavamipoor and Golpayegani (2020), Craven and Krejci (2017), Craven and Krejci (2016), De
Maio et al. (2016), and Valluri, North, and MacAl (2009)
Risk management Aboutorab et al. (2022), Zhao et al. (2021a), and Yang et al. (2019)
Sourcing
Supplier selection Kim, Bilsel, and Kumara (2008), and Tae, Bilsel, and Kumara (2007)
Supplier segmentation Liu (2020), Aghaie and Heidary (2019), and Du and Jiang (2019)
Pricing
Trading Hirano et al. (2021), Du and Xiao (2019), Chatzidimitriou, Symeonidis, and Mitkas (2013), and
Reeder et al. (2008)
Market simulation Liu, Howley, and Duggan (2011)
Other
Concept Filatova, El-Nouty, and Fedorenko (2021), Serrano, Mula, and Poler (2021), and Mezouar and El
Afia (2019)
Literature review Sianaki et al. (2019)

which present exclusively conceptual considerations
on RL and literature reviews.
5.1.1. Inventory
Inventory comprises decisions that impact raw materials,
work in process and finished goods, such as the man-
agement of orders and the determination of safety stock
and cycle stock (Chopra and Meindl 2013). Inventory
management or replenishment is a well-known optimi-
sation problem in SCM. It deals with the questions of
when to order and what quantity to order with the objec-
tive of finding a balance between material availability
and inventory costs. Traditional inventory models such
as the economic order quantity (EOQ) model provide
quick solutions assuming constant demand and com-
plete information (Hax and Candea 1984). However, in
real supply chains, the main challenge is dealing with
changing demands, unknown lead times, disruptions and
incomplete information. RL approaches have the advan-
tage that they can learn adaptive, situation-dependent
7158 B. ROLF ET AL.

Figure 6. Types of inventory problems. Customer-driven replenishment and supplier-driven replenishment take the view of the supplier
and have the objective to determine optimal actions for their operations. Global order management takes a global view and either aims
to find a global optimum for all participants (complete information) or observe the system behaviour if each participant takes his own
actions (incomplete information). Supply chain scheduling considers a global view and tries to find an optimal schedule in a multi-site
environment.

ordering strategies contrary to traditional inventory
models, which have fixed order quantities, safety stocks
and cycle stocks.
The inventory driver of the framework includes all
publications that aim at determining order quantities and
order times, production quantities and production times
or scheduling task sequences. We identified five major
subclasses that differ in view, driving player, number of
players and degree of information exchange. Figure 6
illustrates the five types of inventory problems. Publica-
tions that compare multiple views are assigned to multi-
ple subclasses. For example, Yang and Zhang (2015), Kim
et al. (2005), Kim, Kwon, and Baek (2008) and Xu, Zhang,
and Liu (2009) compare centralised and decentralised
supplier-customer relationships.
• Customer-driven replenishment includes replenish-
ment models in which a customer orders raw materials
from one or more independent suppliers without hav-
ing information about the supplier’s processes. The
customer, who is the driving entity of the ordering
process, takes a self-centred view and considers merely
its own performance metrics. The customer-driven
view is primarily located in supply chains based on
the pull principle. For instance, Lee and Sikora (2019)
present an agent that handles procurement, customer
sales and production scheduling simultaneously and
test it in the supply chain environment of a compe-
tition for trading agents. Zwaida, Pham, and Beau-
regard (2021) consider a hospital supply chain and
train a DRL agent to prevent drug shortages. Ganesan,
Sundararaj, and Srinivas (2021) utilise RL differently
because they do not determine each order indepen-
dently but let the agent dynamically choose from five
common inventory policies.
• Supplier-driven replenishment considers replenish-
ment models in which the supplier is the driving entity
of the ordering process and tries to fulfil a certain
service level determined by the customer. Neverthe-
less, the supplier is independent of the customer and
takes a self-centred view. A well-known example of
this subclass is vendor-managed inventory models.
The customer-driven view is mainly located in sup-
ply chains using the push principle. For instance, Tariq
Afridi et al. (2020) consider a vendor-managed inven-
tory setting in the semiconductor industry and solve
it with RL and discrete-event simulation. Hachaichi,
Chemingui, and Affes (2020) study a similar inven-
tory problem with an agent that learns to place optimal
orders.
• Global order management with complete informa-
tion includes models in which the RL agent takes a
global view and has complete information on the sta-
tus of each supply chain entity. The agent aims to
find a global optimum for the entire supply chain.
These models can be both single-agent or multi-
agent systems. Real-world examples are, for instance,
intra-company supply chains with multiple sites that
exchange information. Being one of the first and most
cited contributions to the topic, the work of Pon-
trandolfo et al. (2002), and Giannoccaro and Pon-
trandolfo (2002) belongs to this subclass. They con-
sider a supply chain model with three stages and one
agent at each stage. The agents act independently but

have information about the other stages and try to
maximise the total reward. Perez et al. (2021) com-
pare deterministic linear programming, multi-stage
stochastic linear programming and RL in a supply
chain with five stages and nine nodes. They found that
RL performs well, considering inventory balancing
but the stochastic model yields the highest profit.
• Global order management with incomplete informa-
tion considers classic supply chains comprising several
independent entities that exchange information to a
lesser extent. These models are usually multi-agent
systems in which each RL agent manages the opera-
tions of a single entity in order to find a local optimum
for the entity. For instance, Mortazavi, Khamseh, and
Azimi (2015) present a simulation-based multi-agent
RL model in a four-stage environment where each
agent tries to optimise its operations.
• Supply chain scheduling has strong similarities to clas-
sic production scheduling problems but extends the
scope to multiple sites. These models consider not
only the inventory levels of the entities but also
include restrictions such as raw material availability,
batch sizes, or sequence constraints. For instance, Ais-
sani et al. (2012) solve a multi-site flexible job shop
problem with transfer times between the sites. Lu
et al. (2021) try to minimise the total supply chain
order completion time by assigning orders to suppli-
ers.
5.1.2. Transportation
Transportation encompasses the movement of inventory
from one site to another in the supply chain (Chopra
and Meindl 2013). Transportation problems such as the
vehicle routing problem or the travelling salesman prob-
lem are widespread in academia, but they are not only
considered in the context of SCM. Hence, the structured
literature search yielded only a small excerpt of models
that explicitly consider supply chain transportation prob-
lems. Transportation problems usually aim to find the
shortest or fastest route between multiple nodes. In addi-
tion, these problems can have several restrictions such as
capacity constraints, time window constraints or pickup
and delivery restrictions. Traditionally, vehicle routing
problems are solved using computationally intensive con-
straint optimisation, heuristics, or metaheuristics from
operations research. RL is expected to provide equal or
better results in less computation time. We identified
five subclasses of transportation problems that are solved
using RL:
• Vehicle routing includes all models that solve the clas-
sic vehicle routing problem with RL. It is composed
INTERNATIONAL JOURNAL OF PRODUCTION RESEARCH 7159
of routing and route segmentation in consideration
of constraints. In the vehicle routing problem, vehi-
cles are usually not a scarce resource, but the focus
is instead on finding the best route. For instance,
Gutierrez-Franco, Mejia-Argueta, and Rabelo (2021)
consider last-mile operations in Bogota, Colombia,
and train an agent to route vehicles depending on
real-time circumstances.
• Vehicle scheduling has similar objectives as vehicle
routing but considers problems where the vehicles
are scarce resources and have to fulfil multiple tasks
during a specific period. This is often the case in
small-scale, closed systems such as the railway traffic
management problem (Bretas et al. 2019) or the inter-
terminal transportation problem (Adi, Bae, and Iskan-
dar 2021).
• Transportation bidding considers transportation prob-
lems in which minimising costs is a major part of
the problem. In a common scenario, multiple logis-
tics service providers compete for transport orders by
trying to place the lowest bid. This scenario is also
suitable for multi-agent systems because the service
providers are independent entities with individual sets
of actions. Tang and Kumara (2005) study a game-
theoretic task delegation problem in which indepen-
dent logistics service providers bid on transportation
orders. Zhanguo (2008) also consider a task delegation
problem but contrary to Tang and Kumara (2005) the
transportation agents do not compete but cooperate to
maximise the overall profit.
• Platooning configuration is a unique transportation
problem in which multiple trucks with different desti-
nations must be coupled in the form of platoons so that
costs or driven distance are minimal. Puskás, Budai,
and Bohács (2020) found out that RL is capable of con-
trolling the platooning and provides a better solution
than a heuristic.
• Network flow routing deals with transporting indivisi-
ble goods such as liquids, gases, energy, or bulk goods
that flow through stationary networks. A network of
nodes and arcs with different flow rates represents the
problem. The objective is to find the maximum flow
state of the network or the minimum distance between
the sink and the source. Hwangbo and Yoo (2018)
consider a hydrogen supply network in the Republic
of Korea. They implement an agent that manages the
supply of wind energy, solar energy, and wastewater to
the hydrogen production plants.
5.1.3. Information
Information includes measures that turn data into infor-
mation or improve the ability to make justified decisions.
7160 B. ROLF ET AL.
This class directly affects all the other supply chain drivers
because data is the basis for every decision-making pro-
cess (Chopra and Meindl 2013). The applications of
machine learning models for information retrieval and
processing are manifold, as this domain is a signifi-
cant scope of machine learning in general. However,
supervised learning algorithms are state-of-the-art in this
domain because their learning process is more straight-
forward whenever clear input-output pairs exist, which
is usually the case for applications such as forecasting.
Therefore, the prevalence of RL models is relatively low,
just like the number of models assigned to this class. Nev-
ertheless, there are valid use cases for the application of
RL:
• Forecasting deals with predicting the future from his-
torical data to support decision-making (Hyndman
and Athanasopoulos 2018). Most of the identified
RL models do not consider forecasting in the clas-
sic sense but rather address more complex problems
closely related to forecasting. For instance, the RL
agent of Chien, Lin, and Lin (2020) selects demand
forecasting methods rather than actual forecasting. Ko
et al. (2011) address a unique application of RL for pre-
dicting the location of missing products based on its
planned path and RFID tracking.
• Collaboration includes models that aim to improve the
collaboration between supply chain entities. Contrary
to global order management with complete informa-
tion, which has very similar objectives, collabora-
tion refers to models that do not consider inventory
decisions but other collaboration issues. For exam-
ple, Dangelmaier et al. (2006) train an RL agent that
learns to solve bilateral conflicts by choosing the best
actions in a conflict situation. In addition, collabo-
ration includes models that consider the negotiation
between supply chain entities such as in Kaihara and
Fujii (2008) or Li et al. (2014).
• Human behaviour evaluation focuses on the role of
the human in supply chain decision-making by eval-
uating human behaviour or considering the actual
decision-making process. Based on the models, RL
has two prominent roles in this subclass: Learning
humans’ behaviour in supply chain environments or
simulating human behaviour with multi-agent sys-
tems. For instance, De Maio et al. (2016) investigate
group decision-making processes in SCM and train
their RL model to weigh group members’ decisions to
find the best overall decision. Ghavamipoor and Gol-
payegani (2020) use RL to learn the expected profit
from customers in a webshop and adapt the service
level to it. In contrast, the RL agents of Craven and
Krejci (2017) and Craven and Krejci (2016) imitate the
behaviour of farmers and producers to derive a pricing
strategy from it.
• Risk management aims to ensure supply chain oper-
ations by taking proactive and reactive measures.
According to Chopra and Sodhi (2004), seven groups
of risks exist in SCM, which are disruptions, delays,
wrong forecasts, price fluctuations, the bankruptcy of
partners, wrong production capacities or high inven-
tory levels. RL is, for example, used to rate the impor-
tance of risks (Aboutorab et al. 2022) and propose
recovery actions after supply chain disruptions (Yang
et al. 2019).
5.1.4. Sourcing
Sourcing addresses the organisation of participants in
the supply chain, especially supplier selection (Caval-
cante et al. 2019) and segmentation, as well as outsourc-
ing and insourcing decisions (Chopra and Meindl 2013).
The reviewed publications showed that the sourcing and
inventory drivers are closely related when considering
real-world supply chain problems. Many models consider
sourcing decisions as well as inventory decisions to some
extent. Therefore, the transition between both classes is
fuzzy, and it is important to distinguish their scope for
the classification framework. Models have been assigned
to sourcing when they focus on tactical to strategic
decision-making, such as selecting suppliers or sourcing
strategies. In contrast, inventory models consider opera-
tional to tactical decisions such as order management and
scheduling. We identified two main applications of RL in
sourcing:
• Supplier selection aims to select one or more ven-
dors that supply a customer in the best way possible
concerning criteria such as costs, reliability, or qual-
ity. As described in Subsection 5.1, the use of RL in
this domain is limited because it is a typical strategic
decision. Merely, the publications of Tae, Bilsel, and
Kumara (2007) and Kim, Bilsel, and Kumara (2008)
have been identified, which study a game-theoretic
model of multiple suppliers and one manufacturer that
are modelled as agents. The manufacturer requests
orders and the suppliers can bid on the orders
and learn from the past behaviour of competing
suppliers.
• Supplier segmentation models aim to find a sourcing
strategy considering a small number of fixed, substi-
tutable suppliers. Companies commonly have primary
and backup suppliers, which help in certain situa-
tions. Du and Jiang (2019) model a system in which
an agent determines how to split orders of a manufac-
turer among two substitutable suppliers. Aghaie and
Heidary (2019) study a similar environment with a

primary supplier, a backup supplier, and a spot mar-
ket. Liu (2020) trains agents to take insourcing and
outsourcing decisions flexibly.
5.1.5. Pricing
Pricing comprises all decisions regarding costs and
prices, such as bidding, contracting, marketing strategies
and sales planning (Chopra and Meindl 2013). In SCM,
it is an essential topic for determining the delivery costs
of semi-finished products between companies and the
selling prices to consumers. Pricing is a common topic
in economics and marketing, but a few contributions
also explicitly focus on SCM and apply reinforcement
learning to it. Two applications are most important:
• Trading is the most common application of RL in
pricing. Agents are responsible for trading on mar-
kets, determining selling prices or bidding on offers
to maximise profit. For instance, Reeder et al. (2008)
train agents to participate in the market of a mul-
tiplayer online game. They conclude that trading
agents can participate in virtual economies equally
to humans, which can also be helpful for real sup-
ply chain environments. Chatzidimitriou, Symeoni-
dis, and Mitkas (2013) use an agent to manage pro-
duction and customer orders in terms of costs. Du
and Xiao (2019) consider an environment with one
supplier, two retailers and several customers and let
the agents choose between uniform and differenti-
ated pricing strategies. Hirano et al. (2021) investigate
how to prevent unintentional collusion when multiple
agents act in a supply chain environment.
• Market simulation includes models that use RL to sim-
ulate the behaviour of humans in the market, espe-
cially to evaluate the price sensitivity of consumers.
Liu, Howley, and Duggan (2011) give insights into
the relationship between market price and consumer
behaviour in a supply chain environment.
5.1.6. Other
The literature search also yielded a few publications
that do not fit the classification framework of supply
chain drivers because they do not provide quantitative
models but rather general considerations. Three publica-
tions present concepts on RL in SCM. Mezouar and El
Afia (2019) propose to extend the SCOR model with a
sixth standard process which they call ‘Learn’. The new
process shall add RL capabilities to the SCOR model and
make supply chains more adaptive. Serrano, Mula, and
Poler (2021) present a conceptual framework based on
a digital twin which shall reschedule the master produc-
tion schedule of supply chains in real-time using DRL.
INTERNATIONAL JOURNAL OF PRODUCTION RESEARCH 7161
Filatova, El-Nouty, and Fedorenko (2021) create a math-
ematical model for RL in supply chain collaboration and
sustainability.
Surprisingly, the search results contain only one litera-
ture review. The review of Sianaki et al. (2019) shows state
of the art on machine learning in smart cities, health-
care and transportation. However, no literature review
explicitly focuses on RL in SCM.
5.2. Algorithms
We modified the OpenAI taxonomy to classify papers
based on the applied RL algorithm (Achiam 2018). All
the RL algorithms can be divided into model-free and
model-based, depending on whether the agent has direct
access to a model of the environment. In this context,
an environment model refers to a function that predicts
state transitions and rewards. The most famous model-
based approach is AlphaZero, the DRL algorithm that
could master classic board games such as Shogi and
Chess through self-play (Silver et al. 2017). The com-
mon challenge is that the agent can discover and exploit
bias in the model, resulting in a policy that performs
well in the learned model, but behaves sub-optimally
in the real environment. That is why the model-based
approaches are hardly applicable if the problem is char-
acterised by stochasticity, long-term planning horizons,
partial observability, and imperfect information. Since all
the above traits are common in complex supply chains,
there is no wonder that all the reviewed studies took
advantage of model-free RL. Model-free RL, in its turn,
also constitutes two major categories: policy optimisa-
tion and value-based algorithms (see Figure 7). Table 2
contains the academic papers classified by the proposed
modification of the OpenAI taxonomy. It is worth men-
tioning that it was possible to classify RL algorithms
unambiguously only for 61 publications. The remaining
academic papers do not explicitly mention the algorithm
used.
5.2.1. Value-based algorithms
In order to find an optimal policy π(s), value-based
methods derive an approximator Qθ (s, a) parametrised
by a set of parameters θ that maps states to the corre-
sponding actions as if they are stored in a table. This
optimisation is conducted off-policy, which means that
each update utilises data collected at any point during
training. As a result, the policy π ∗ is approximated by
the optimal action-value function Q∗ (s, a), and the agent
takes action according to the following equation:
a(s) = arg max Qθ (s, a) (2)
a
7162 B. ROLF ET AL.

Figure 7. The proposed taxonomy encompasses the RL algorithms applied to supply chain problems.

Table 2. Classification of publications according to the applied algorithms.

Algorithm Publication
Value-based algorithms
Classic Q-learning Aboutorab et al. (2022), Chen et al. (2021), Hirano et al. (2021), Meisheri et al. (2021), Wang and Lin (2021), Zhao
et al. (2021b), Chien, Lin, and Lin (2020), Liu (2020), Makridis et al. (2020), Puskás, Budai, and Bohács (2020),
Aghaie and Heidary (2019), Du and Jiang (2019), Marandi and Fatemi Ghomi (2019), Mezouar and El
Afia (2019), Hwangbo and Yoo (2018), Craven and Krejci (2017), Craven and Krejci (2016), Zhou, Purvis,
and Muhammad (2016), Dogan and Güner (2015), Mortazavi, Khamseh, and Azimi (2015), Li et al. (2014),
Saitoh and Utani (2013), Sun and Zhao (2012), Reindorp and Fu (2011), Sui, Gosavi, and Lin (2010), Sun,
Zhao, and Yin (2010), Chaharsooghi, Heydari, and Zegordi (2008), Kwon et al. (2008), Reeder et al. (2008),
Zhanguo (2008), Van Tongeren et al. (2007), Zhang and Bhattacharyya (2007), Dangelmaier et al. (2006),
Tang and Kumara (2005), Ravulapati, Rao, and Das (2004), Sheremetov and Rocha-Mier (2004), and Rao,
Ravulapati, and Das (2003)
DQN Li et al. (2021), Zwaida, Pham, and Beauregard (2021), Singi et al. (2020), Tariq Afridi et al. (2020), and Nanduri
and Saavedra-Antolínez (2013)
SARSA Habib, Khan, and Uddin (2017), Aissani et al. (2012), and Dahlem and Harrison (2010)
Policy optimisation algorithms
Policy Gradient Huang and Tan (2021), Peng et al. (2019), and Mehta and Yamparala (2014)
A2C/A3C Zhu, Ke, and Wang (2021), and Barat et al. (2019)
PPO Kegenbekov and Jackson (2021), Perez et al. (2021), Alves and Mateus (2020), Hachaichi, Chemingui, and
Affes (2020), and Vanvuchelen, Gijsbrechts, and Boute (2020)
SMART Giannoccaro and Pontrandolfo (2002), and Pontrandolfo et al. (2002)
Other
Evolutionary Zhao, Jiang, and Feng (2010), and Sheremetov and Rocha-Mier (2008)
EM Govindan and Al-Ansari (2019)

Examples of value-based methods discovered during the
literature review include:
• Classic Q-learning finds an optimal policy in the sense
of maximising the expected value of the total reward
by identifying an optimal action-selection policy for
a given MDP (Watkins and Dayan 1992). For exam-
ple, Chaharsooghi, Heydari, and Zegordi (2008) pro-
posed a Q-learning-based approach to derive order-
ing policies for multiple supply chain entities in an
integrated manner to minimise total inventory costs.
Zhou, Purvis, and Muhammad (2016) combined Q-
learning with system dynamics in a multi-agent archi-
tecture to address a collaborative planning problem
in global supply chains. Dogan and Güner (2015)
used a Q-learning agent as a base complementing
it with stochastic dynamic programming and agent-
based simulations. The study analysed simultaneous
ordering and pricing decisions for retailers working in
a multi-echelon supply chain, such that retailers com-
pete for the same market where the market demand
is uncertain. Aboutorab et al. (2022) demonstrated
how Q-learning can assist risk managers in proactively
identifying the risks to the supply chain operations.
Wang and Lin (2021) proposed a multi-agent-based
collaborative replenishment model for supply chains.
Q-learning was applied to optimise supply network
topology such that the replenishment time is min-
imised. Meisheri et al. (2021) incorporated Q-learning
into the decision-making framework to solve multi-
product, multi-period inventory management prob-
lems with uncertain demand.

• Deep Q-Networks (DQN), initially invented by Deep-
Mind, are well-known for their ability to master
a wide range of computer games at a superhu-
man level. The algorithm was developed by enhanc-
ing a classic Q-learning algorithm with deep neu-
ral networks and experience replay technique (Mnih
et al. 2013). For example, Li et al. (2021) proposed
a data-driven approach based on DQN to solve
industry-scale dynamic pickup and delivery problems
in supply chains. Singi et al. (2020) compared several
DQN architectures addressing the problem of inven-
tory management in a single-product supply chain.
Zwaida, Pham, and Beauregard (2021) faced a health-
care inventory management problem. An RL agent
based on DQN was applied to approach manufac-
turing problems, supply and demand issues, and raw
material problems across the entire supply chain. The
proposed solution could automatically make a drug
refilling decision to prevent a drug shortage.
• State-action-reward-state-action (SARSA) is a modifi-
cation of the classic Q-learning algorithm. Unlike clas-
sic Q-learning, SARSA is an on-policy RL algorithm
in the sense that the action is performed by the
current policy to learn the Q-value (Rummery and
Niranjan 1994). A notable example of SARSA applica-
tion in the supply chain context includes the seminal
work by Habib, Khan, and Uddin (2017), who applied
the algorithm to an MDP that can incorporate the
design of a multi-stage supply chain network. Addi-
tionally, Dahlem and Harrison (2010) applied SARSA
to load-balancing applications and supply chain net-
work design. Aissani et al. (2012) proposed a multi-
agent approach based on SARSA to perform adaptive
scheduling in a multi-echelon supply chain.
5.2.2. Policy optimisation algorithms
Methods based on policy optimisation, on the other
hand, represent a policy explicitly as πθ (a | s). The
parameters θ are usually optimised directly by gradient
ascent on the expected return J(πθ ). This optimisation
is conducted on-policy, which means that each policy
update is performed based on data collected while util-
ising the most recent version of the policy.
Examples of methods based on policy optimisation
discovered during the literature review include:
• Policy Gradient aims to gradually increase the proba-
bilities of selecting actions that lead to higher returns
until the optimal policy is obtained. The algorith-
mic implementation of policy gradient performs by
updating policy parameters using stochastic gradi-
ent ascent on policy performance, namely θk+1 =
INTERNATIONAL JOURNAL OF PRODUCTION RESEARCH 7163
θk + α∇θ J(πθk ), where α is the learning rate (Sutton
et al. 1999). For example, in a recent paper, Huang
and Tan (2021) applied a policy gradient algorithm
to optimise the time for the supply chain order man-
agement process. Peng et al. (2019) compared the pol-
icy gradient algorithm with the (r, Q) policy, a com-
mon approach in classic operations research (Jack-
son 2020). Both policies were compared on the multi-
period capacitated supply chain optimisation prob-
lem under demand uncertainty. Mehta and Yampar-
ala (2014) formulated a general stochastic SCM prob-
lem as a multi-arm non-contextual bandit problem.
A policy gradient directly approached the problem to
find a robust policy in the new setting.
• Asynchronous Advantage Actor-Critic (A2C/A3C) are
conceptually similar to the classical actor-critic meth-
ods, where the policy πs (actor) and value estimate
Vπ (s) (critic) are trained at the same time (Sut-
ton et al. 1999). However, the core difference is that
A2C/A3C take advantage of artificial neural networks
as non-linear function approximators for both value
and policy outputs. It is worth emphasising that A2C
and A3C algorithms are mathematically equivalent.
However, they differ significantly in technical imple-
mentation. Namely, A2C can be efficiently executed
on the GPU hardware, which entails a significant com-
putational advantage compared to the typically CPU-
only A3C-based agents (Mnih et al. 2016). Examples
of A2C/A3C applications for SCM include the work
by Barat et al. (2019) who proposed an A2C-based
approach to optimise transportation flows and inven-
tory levels across the complex supply chain network
to enable the deployment of the RL agent in the real-
world system with minimal further tuning. Addition-
ally, Zhu, Ke, and Wang (2021) developed an MDP
model to depict the dynamics in ride-sourcing mar-
kets with A2C-based RL agents aiming to maximise
their individual income.
• Proximal Policy Optimisation (PPO) also utilises the
actor-critic method, such that the actor maps the
observation to action, and the critic provides an
expectation of the rewards (Schulman et al. 2017).
PPO is distinguished by simple implementation, gen-
erality, and low sample complexity (Kegenbekov and
Jackson 2021). Examples of PPO applications to sup-
ply chain problems include Hachaichi, Chemingui,
and Affes (2020) who developed an RL agent capa-
ble of placing optimal orders in the supply chain,
taking the stochastic lead times into account. Alves
and Mateus (2020) considered a multi-period four-
echelon supply chain as a sequential decision-making
problem. The PPO algorithm was applied to find a
7164 B. ROLF ET AL.
policy that minimises the total operating costs. In
a recent paper, Perez et al. (2021) compared several
DRL techniques, including PPO on a single product, a
multi-period centralised supply chain under stochas-
tic stationary consumer demand. Kegenbekov and
Jackson (2021) demonstrated how a PPO-based agent
could synchronise inbound and outbound flows and
support business continuity operating in the stochas-
tic and nonstationary environment if end-to-end visi-
bility across the entire supply chain is ensured.
• Semi-Markov Average Reward Technique (SMART) is
an extension of the temporal difference algorithm
that can be applied to a larger class of problems
for which the underlying probability structures can
not be characterised solely by Markov chains (Das
et al. 1999). Pontrandolfo et al. (2002) demonstrated
that the SMART algorithm could be applied to opti-
mise a networked production system that spans sev-
eral geographic areas. Shortly after that, it was demon-
strated that the SMART algorithm could also coor-
dinate inventory policies adopted by different sup-
ply chain participants (Giannoccaro and Pontran-
dolfo 2002).
5.2.3. Other
We also identified papers that take advantage of Bayesian
and evolutionary approaches to RL that are less common
and do not fit into the original OpenAI taxonomy. Never-
theless, these approaches represent a promising alterna-
tive to value-based and policy-optimisation methods.
• Bayesian methods in the RL setting attempt to lever-
age Bayesian inference to incorporate information
into the learning process. More specifically, this
class of methods assumes that the system designer
can add prior information about the problem in a
probabilistic distribution, and the new information
can be incorporated by standard Bayesian inference
(Ghavamzadeh et al. 2015). The Bayesian methods
applied to the supply chain domain are represented
by the expectation-maximisation algorithm (EM) ini-
tially proposed by Dayan and Hinton (1997). Govin-
dan and Al-Ansari (2019) used the EM algorithm in a
computational framework to identify and prevent fail-
ures of critical infrastructures when the global supply
chains are subject to stresses and shocks.
• Evolutionary methods include a broad class of meta-
heuristics inspired by natural evolution and the ‘sur-
vival of the fittest’ principle (Mahapatra and Pat-
naik 2018). The combination of RL and evolution-
ary methods can take different forms. Some authors
use evolutionary algorithms to search the space of
RL policies and train the agent, as it was proposed
by Moriarty, Schultz, and Grefenstette (1999). On
the other hand, it is also possible to use RL for
setting appropriate parameters of evolutionary algo-
rithms (Sakurai et al. 2010). Evolutionary methods
discovered during the literature review include varia-
tions of Genetic Algorithm-based RL (Long Zhao and
Liu 1996). For example, Zhao, Jiang, and Feng (2010)
combined classic Q-learning with a genetic algorithm
to address the multi-echelon supply chain coordina-
tion problem.
5.3. Data sources
Getting suitable data for the training of the RL agent
is often a crucial problem because the performance of
the RL model is highly dependent on the data qual-
ity. Data in a supply chain environment includes, for
instance, the supply chain layout, available suppliers, lead
times, demand scenarios, costs or additional restrictions.
Based on these data, it is possible to design the supply
chain environment and define the actions and rewards
of the agent. The reviewed publications use three pri-
mary data sources: Artificial, public, and institutional
data. Most publications created artificial data for their
RL model. The reason is usually the conceptual nature of
the papers or the lack of suitable data. However, training
agents with artificial data is usually the most straight-
forward way because data can perfectly fit the needs,
and the required amount of data is not a problem. How-
ever, some of the publications use existing data for the
training, namely public data or institutional data. Public
data comes mainly from the web or academic literature
and is freely accessible, whereas institutional data comes
from companies or the government and is confidential.
These subclasses are attractive for further review because
training an RL agent on real-world data is usually more
challenging. Especially for deploying RL in the indus-
try, it is necessary to fit the RL model to existing data.
Table 3 lists all publications that either used public or
institutional data.
5.3.1. Public data
In the following, we consider the publications in detail
that used public data:
• SCM competitions simulate representative supply
chain environments where researchers can test their
algorithms and compete directly with each other. The
most well-known competition was the yearly TAC
SCM game that took place until 2011 and was organ-
ised by Carnegie Mellon University, the University
of Minnesota and the Swedish Institute of Com-
puter Science (Collins et al. 2006). Agents had to
 INTERNATIONAL JOURNAL OF PRODUCTION RESEARCH 7165

Table 3. Classification of publications according to the data source.

Source Publications
Public
SCM competition Lee and Sikora (2019), Mehta and Yamparala (2014), and Chatzidimitriou, Symeonidis, and Mitkas (2013)
Open Data Aboutorab et al. (2022), Chen et al. (2021), Meisheri et al. (2021), Barat et al. (2020), and Barat et al. (2019)
Academic literature Kegenbekov and Jackson (2021), Perez et al. (2021), Aghaie and Heidary (2019), Valluri, North, and MacAl (2009), Chaharsooghi,
Heydari, and Zegordi (2008), Van Tongeren et al. (2007), Gang and Ruoying (2006), Sun et al. (2006), and Simsek, Albayrak,
and Korth (2004)
Online game Reeder et al. (2008)
Institutional
Industry Adi, Bae, and Iskandar (2021), Li et al. (2021), Gutierrez-Franco, Mejia-Argueta, and Rabelo (2021), Chien, Lin, and Lin (2020),
Makridis et al. (2020), Singi et al. (2020), Tariq Afridi et al. (2020), Bretas et al. (2019), Yang et al. (2019), Craven and
Krejci (2017), Craven and Krejci (2016), and Aissani et al. (2012)
Government Zhao et al. (2021a), and Hwangbo and Yoo (2018)

manage banking, production and warehousing ser-
vices to maximise their profit. Lee and Sikora (2019)
and Chatzidimitriou, Symeonidis, and Mitkas (2013)
used the data from the competition to train and test
their RL agents. Another competition that received
much attention was held by the data science platform
Kaggle (Instacart 2017). The dataset used in this com-
petition was provided by the American grocery deliv-
ery service Instacart and included 3 million customer
orders. The competition had the goal of predicting
which products would be in a customer’s next order.
Barat et al. (2019), Barat et al. (2020), and Meisheri
et al. (2021) use this dataset to model an inventory
management problem and solved it with RL. Another
competition was run by the Indian decision sciences
company Mu Sigma. Mehta and Yamparala (2014)
used the environment and the dataset from the com-
petition to implement their RL algorithm.
• Open data are other common sources that are often
suitable for training RL agents. The data science plat-
form Kaggle was already mentioned as an impor-
tant competition host. However, it is also a plat-
form where users can share datasets tailored explicitly
to data science. There exist also datasets for SCM.
Another source is the national bureaus of statistics
which provide official data on economic matters. Chen
et al. (2021) use data from the National Bureau of
Statistics of China to simulate an agri-food supply
chain and train an RL agent. Aboutorab et al. (2022)
use online news websites to identify risks and take
proactive measures with RL.
• Academic literature is the most obvious source for
data and supply chain environments. In many research
disciplines, it is common practice to publish data
and results so that other researchers can use them
to benchmark their algorithms. RL in SCM does not
have any well-known benchmark problems. One rea-
son might be the loose connection of publications in
this research domain. However, some authors use the
scenarios of former publications. For example, Simsek,
Albayrak, and Korth (2004), Gang and Ruoying (2006)
and Sun et al. (2006) all address the case study of a
research project called DispoWeb that was done by
the Technical University of Berlin (DispoWeb 2004).
The beer game is a well-known educational game in
SCM that illustrates the bullwhip effect, widespread
in logistics and production education. It was ini-
tially developed by Jay Forrester at the MIT Sloan
School of Management in the 1960s (Sterman 1992).
It is a standard environment that many researchers
tried to approach with algorithms. There are also
some approaches that try to play the beer game with
RL agents, for instance Chaharsooghi, Heydari, and
Zegordi (2008), Van Tongeren et al. (2007), and Val-
luri, North, and MacAl (2009). Furthermore, the sem-
inal work of Glasserman and Tayur (1995) inspired the
models of Kegenbekov and Jackson (2021) and Perez
et al. (2021).
• Online games provide similar environments as SCM
competitions, but they are designed for human play-
ers. However, it is possible to train RL agents for trad-
ing in multiplayer online games, as shown by Reeder
et al. (2008). Contrary to SCM competitions, the agent
competes not only with other agents but also with
human players. Based on the experience with online
games, agents can also be used in real-world supply
chain environments.
5.3.2. Institutional data
Institutional data includes all data sources that are not
public such as data from industry and government:
• Industry is an important source for data because it best
represents the current challenges and requirements
of industrial RL deployment. The publications in this
subclass either used data provided by the industry or
even collaborated with a company on the publication.
For example, Tariq Afridi et al. (2020) collaborated
with the German semiconductor manufacturer Infi-
neon or Li et al. (2021) tackled the dynamic pickup
7166 B. ROLF ET AL.

Table 4. Classification of publications according to industry.

Industry Publications
Manufacturing Chen et al. (2021), Li et al. (2021), Chien, Lin, and Lin (2020), Tariq Afridi et al. (2020), Aissani et al. (2012),
Zhang and Bhattacharyya (2007), Gang and Ruoying (2006), Sun et al. (2006), Simsek, Albayrak, and
Korth (2004), and Bukkapatnam and Gao (2000)
Wholesale and retail trade Meisheri et al. (2021), Barat et al. (2020), Makridis et al. (2020), Singi et al. (2020), Barat et al. (2019),
Craven and Krejci (2017), Craven and Krejci (2016), and Mehta and Yamparala (2014)
Transportation and storage Adi, Bae, and Iskandar (2021), and Gutierrez-Franco, Mejia-Argueta, and Rabelo (2021)
Electricity, gas, steam and air conditioning supply Xiang (2020), and Hwangbo and Yoo (2018)
Mining and quarrying Bretas et al. (2019)
Information and communication Ghavamipoor and Golpayegani (2020)
Public administration and defence Zhao et al. (2021a)
Human health and social work activities Zwaida, Pham, and Beauregard (2021)

and delivery problem of the Chinese electronics man-
ufacturer Huawei. Section 5.4 provides further clas-
sification of RL models that target certain industrial
sectors.
• Government includes all data sources from offi-
cial institutions such as government, state, province,
municipality, or the military and are not publicly avail-
able. We identified only two publications that used
government data. Zhao et al. (2021a) focus on a game
with two agents with contrary objectives and use the
case study of the United States Marine Corps logistics.
Hwangbo and Yoo (2018) used data from a province
in the Republic of Korea to model a hydrogen supply
network.
5.4. Industrial sectors
Another way to structure the publications is according to
the industrial sectors. Most publications consider generic
supply chains with generic participants such as suppliers,
producers, retailers and customers but some target spe-
cific industrial sectors. This is the case when the targeted
industry has unique requirements for its supply chain,
or the data source indicates the affiliation to an indus-
trial sector. As the publications that focus on a specific
industry are most interesting, we only consider these for
further review in this subsection. It is not always pos-
sible to classify publications to one industry, especially
manufacturing and retail, because supply chains usually
include stakeholders in both areas. In this case, we assign
them to the industry that represents the focus of the
publication. The most common classification system for
industrial sectors is the ISIC, maintained by the United
Nations (UNO 2008). It uses 21 sections that are further
classified into divisions, groups and classes. Table 4 shows
the industry affiliations according to the ISIC sections. In
the following, we describe the relevant industrial sectors
and the belonging publications:
• Manufacturing includes the transformation of mate-
rials, substances or components into new products
(UNO 2008). RL models focus on three manufactur-
ing divisions: Electronics, vehicles and food. Espe-
cially semiconductors are typical products in elec-
tronics supply chains and are often used for case
studies in academic literature. Semiconductor supply
chains are characterised by short product life cycles,
strong economies of scale, pervasive uncertainty and
a global nature (Mönch, Uzsoy, and Fowler 2018).
Tariq Afridi et al. (2020) use the example of the
German manufacturer Infineon, whereas Chien, Lin,
and Lin (2020) address a generic semiconductor sup-
ply chain. Li et al. (2021) consider the transporta-
tion of raw materials between the factories of Huawei
in China, and Bukkapatnam and Gao (2000) use a
generic computer supply chain for their inventory
management problem. Besides semiconductors, vehi-
cles are another focus in the manufacturing section.
This is mostly because of the tractor production use
case that we have already described in Subsection 5.3.1
(Simsek, Albayrak, and Korth 2004; Gang and Ruoy-
ing 2006; Sun et al. 2006). Agri-food supply chains
have to deal with perishable products and stringent
regulations regarding food safety which makes them
fundamentally different from electronics and vehicle
supply chains. For example, Chen et al. (2021) model
an entire agri-food supply chain from farmer till
consumer. Another publication considers the apparel
industry and uses the example of the Algerian com-
pany ENADITEX (Aissani et al. 2012).
• Wholesale and retail trade is another major indus-
trial sector for RL application in SCM. This section
includes trade without transformation, which is usu-
ally one of the final steps in the supply chain
(UNO 2008). All the models in this industrial sector
consider the retail of groceries or agricultural prod-
ucts. Craven and Krejci (2017) and Craven and Kre-
jci (2016) consider the trading between farmers and
producers in regional food hubs in Iowa. Makridis
et al. (2020) aim at enhancing food safety by predict-
ing the risk of food recalls based on the data of the
Greek decision analytics company Agroknow. Mehta
 INTERNATIONAL JOURNAL OF PRODUCTION RESEARCH 7167

and Yamparala (2014) develop a model for order man- (UNO 2008). Zhao et al. (2021b) simulate a mili-
agement in the food service industry. Singi et al. (2020) tary scenario in which they manage the United States
consider the case of the Indian grocery retailer DMart. Marine Corps logistics and maintenance.
Eventually, Barat et al. (2019), Barat et al. (2020), and • Human health and social activities includes health-
Meisheri et al. (2021) predict customer orders for the care and social work (UNO 2008). Zwaida, Pham,
American grocery delivery service Instacart. and Beauregard (2021) consider the drug inventory
• Transportation and storage comprises freight trans- management in a generic hospital supply chain. The
port, cargo handling and storage (UNO 2008). Two inventory management in hospital supply chains is
publications explicitly focus on the transportation different from manufacturing inventory management
industry: Adi, Bae, and Iskandar (2021) study the because shortages are more critical. Approaches must
inter-terminal truck routing in Busan port in South have a high level of awareness for inventories (Landry
Korea and Gutierrez-Franco, Mejia-Argueta, and and Beaulieu 2013).
Rabelo (2021) study the last-mile delivery in the
Columbian city of Bogota.
6. Discussion
• Electricity, gas, steam and air conditioning includes the
provision of electric power, gas, steam, or air through a 6.1. Applications
permanent infrastructure (UNO 2008). Xiang (2020)
Inventory management is the most common appli-
proposes a method that handles energy emergen-
cation of RL in SCM (see Figure 8). 61.1% of the
cies as a consequence of disruptions. Hwangbo and
RL models target inventory management and use the
Yoo (2018) consider a hydrogen supply network in
agent for controlling the material flow between mul-
which the total costs must be minimised.
tiple sites in the supply chain. The inventory mod-
• Mining and quarrying comprises the extraction of
els are almost evenly split between two-tier settings
minerals, liquids and gases (UNO 2008). Bretas
(Customer-driven and supplier-driven replenishment)
et al. (2019) approach the railway traffic management
and multi-tier settings (Global order management and
of the Hunter Valley Coal Chain Coordinator in Aus-
supply chain scheduling). A proportion of 16.5% aims
tralia with RL.
at the information driver, including forecasting, collab-
• Information and communication includes all activities
oration, human behaviour evaluation, and risk manage-
belonging to information technology, data processing
ment. Even though information retrieval and processing
and communication (UNO 2008). Ghavamipoor and
are standard domains for machine learning, the num-
Golpayegani (2020) consider an electronic services
ber of RL models in this domain is limited as supervised
supply chain that consists of different network service
learning is superior when clear input-output pairs exist.
providers providing infrastructure for e-commerce. In
RL prevalence in transportation, sourcing and pricing
this environment, they train an RL agent to adapt the
is lower than in inventory. One reason might be that
service quality adaptively based on the expected profit
transportation and pricing problems do not only exist
of the customers.
in SCM but also separate research communities focus
• Public administration and defence includes activities
on these problems. Nevertheless, both drivers are part of
carried out by the government and military activities

Figure 8. Classification and proportion of investigated publications in terms of supply chain drivers and RL algorithms. (a) Supply chain
drivers (n = 103) and (b) Algorithms (n = 61).
7168 B. ROLF ET AL.
SCM and impact performance. Although sourcing is a
key consideration in supply chains, it does not have many
contributions, possibly due to its longer decision horizon.
Generally, decision levels and prevalence of RL seem to
have coherence.
The standard definition of SCM distinguishes three
decision levels that differ in their planning horizons: Sup-
ply chain design, supply chain planning and supply chain
operation, respectively long-term, mid-term and short-
term (Chopra and Meindl 2013). Inventory management
and transportation planning are typical short-term tasks
that require frequent and fast decision-making. In con-
trast, sourcing and pricing are usually mid- to long-term
tasks with a decision horizon of a quarter to a year
(Chopra and Meindl 2013; Fleischmann, Meyr, and Wag-
ner 2008). Information has no generally valid decision
horizon because the need for information spans across
all levels (Chopra and Meindl 2013). Considering the
decision levels, most of the identified RL models address
short-term tasks and only some solve mid-term tasks.
None consider long-term tasks. A possible explanation
is the nature of decisions on the decision levels. Supply
chain design significantly impacts the success of sup-
ply chains, and decisions are taken at the managerial
level. Furthermore, these decisions are valid for several
years, and decision-making time is not an essential factor.
Hence, black-box approaches such as RL seem unsuitable
as they do not provide reasoning, and the low decision-
making time is irrelevant. Furthermore, supply chain
design is usually not a structured problem because it can-
not be captured quantitatively, and the decision-maker
has much implicit knowledge that influences his deci-
sion. For instance, the decision to choose a location for a
new production site depends not only on the mathemat-
ical location model but also on qualitative factors such as
the availability of qualified employees. Contrarily, short-
term problems such as inventory management are easier
to capture for RL models as they have a limited scope, a
short planning horizon and precise inputs and objectives.
The suitability of RL for mid-term tasks is ambiguous.
For instance, supplier selection is usually based on clear
quantitative measures, whereas policy selection regard-
ing product programs or sales markets is more difficult
to define.
Regarding the research subjects of the identified pub-
lications, the classification also revealed two general
approaches to applying RL in SCM that are independent
of supply chain drivers. Most papers directly utilise agents
for controlling operations and decision-making. These
papers aim to perform better than humans in the con-
sidered environments and improve decision-making by
optimising any performance indicator. However, a small
share of papers does not directly aim at decision-making
but uses multi-agent systems for simulation purposes.
According to the RL paradigm, each agent still tries to
find a near-optimal state in the supply chain environ-
ment. However, in this case, the aim is not to provide
direct decision support but to study the behaviour and
interaction of agents. This finding is particularly interest-
ing in environments with multiple players and substan-
tial uncertainties because it is challenging to represent
realistic human behaviour with fixed rules in traditional
simulation models. Instead, actions, states and rewards
are defined, and the system is left to its own devices
to observe the agents’ competition and collaboration.
Deriving recommendations for the supply chain problem
from the simulation is only the second step after reach-
ing an equilibrium state. The findings can, in turn, be
used to optimise one of the supply chain drivers. Pub-
lications using the second approach mainly target the
information or pricing drivers and can be found, for
instance, in human behaviour evaluation (Craven and
Krejci 2016, 2017) and market simulation (Liu, Howley,
and Duggan 2011).
6.2. Algorithms
It was possible to classify RL algorithms unambiguously
only for 61 publications (see Figure 8). The remain-
ing academic papers focus on conceptualisation and
managerial insights and do not explicitly mention the
algorithm used. Among the classified publications, the
vast majority (62%) apply Q-learning. The algorithm’s
popularity can be explained by the fact that it is model-
free, which implies that it does not require a model of the
environment, so RL agents can operate directly by sam-
pling and learning the expected rewards for an action
taken in a given state. Besides, Q-learning can handle
stochastic transitions and reward problems (Baird 1995).
Twelve publications (20%) took advantage of the DRL-
based approaches, including DQN, A2C, A3C, and PPO.
By incorporating deep artificial neural networks into the
algorithms, DRL allows agents to make decisions from
large-scale and potentially unstructured input data with-
out additional data engineering. As a result, DRL agents
can operate with substantial data inputs (e.g. annual
inventory data at the transactional level) and decide what
actions to perform to optimise the objective. We also dis-
covered a few applications of evolutionary and Bayesian
approaches, which constitute a promising alternative to
classic RL and DRL. It is important to highlight that
outside of the RL scope, Bayesian methods are gaining
 INTERNATIONAL JOURNAL OF PRODUCTION RESEARCH 7169

Figure 9. Classification and proportion of investigated publications in terms of data sources and industrial sectors. (a) Data sources
(n = 94) and (b) Industrial sectors (n = 96).

momentum in SCM applications. For example, in a recent 7. Managerial insights and future research
study, a multi-layer Bayesian network was used to simu- directions
late and quantify the impact of supply chain disruptions
The larger volume and lower latency of digital informa-
(Hosseini and Ivanov 2021).
tion produced by sources ranging from social media to
the Internet of Things have significantly extended the
6.3. Dissemination in industry scope of what can be considered measurable and quantifi-
able. Worldwide revenue for AI, Big Data, and business
The last research question considers the dissemination
analytics solutions is expected to exceed $ 274 billion
of RL in industrial SCM. By assessing the data sources
in 2022 (Brynjolfsson, Jin, and McElheran 2021). Keep-
and industrial sectors, we can indicate the relevance of
ing in mind that deep learning is still a young area of
RL in industrial applications. Figure 9 shows the propor-
research, and the efficient DRL algorithms have been dis-
tions of data sources and industrial sectors. The classi-
covered only recently (in 2013, 2016, and 2017 for DQN,
fication of data sources revealed that 66% of the publi-
A2C, and PPO, respectively), the popularity of these tech-
cations use artificial data for their implementation, and
niques is expected to increase over time. Besides the
73% consider generic supply chains. This finding indi-
recent progress in algorithmic, hardware, and data tech-
cates that the deployment of RL in SCM is still in an early
nologies, the advent of the high-level frameworks for
stage because of the mostly conceptual level in the pub-
DRL development and deployment has the potential to
lications. Artificial data simplifies the implementation
make DRL applications for SCM more affordable, widely
because it is possible to generate arbitrary data accord-
applicable, and accessible than ever before (Daniel Zhang
ing to the needs. Even if authors used non-artificial data,
et al. 2021).
these are often taken from academic literature and repre-
Although the analysed publications show that RL algo-
sent reality to a limited extent. However, there are some
rithms exhibit great potential for various SCM tasks, our
promising exemptions that have been published in the
investigations conclude that RL research for SCM appli-
last few years for the most part and consider industrial
cations is still in its infancy from a practical industrial
supply chains such as Gutierrez-Franco, Mejia-Argueta,
perspective. The reasons for this are manifold. In the fol-
and Rabelo (2021), Li et al. (2021) or Adi, Bae, and Iskan-
lowing subsections, we provide managerial insights and
dar (2021). Most generic settings are simple supply chains
derive future research directions.
with few stages and participants, little uncertainty and no
additional restrictions. The extent and uncertainties, for
instance, regarding demands and lead times, make real-
7.1. Availability of supply chain data and procedure
world supply chains far more complex than these exam-
models for the industrial implementation of
ples. Moreover, there are different types of supply chains
reinforcement learning
from which some have special requirements, for exam-
ple, regarding perishability in agri-food supply chains As mentioned in Section 5.1, most papers evaluate
or increased awareness in healthcare supply chains. If RL applications on comparatively simple problems that
authors considered specific industrial sectors, these are neglect many aspects of genuine supply chains. One of the
primarily standard manufacturing or retail supply chains main reasons for this circumstance might be that there
with few special requirements. are hardly any publicly available datasets from genuine
7170 B. ROLF ET AL.
industrial supply chains that researchers could use to
develop, and pilot RL applications. Consequently, the
vast majority of the investigated papers evaluate RL algo-
rithms on very simply structured supply chains with
randomly generated data.
In addition, some aspects currently prevent the use
of RL in real industrial SCM. The major impediment
is data acquisition, especially after the training, when
deploying an RL-trained agent for decision-making in a
real supply chain. To take advantage of the agent’s real-
time responsiveness, all necessary information from the
supply chain that the agent receives as an input state
must be continuously accessible in real-time. An obvious
problem is the required data from external companies
in the supply chain. Often companies are not eager to
make internal data available due to concerns about com-
petitive disadvantage. Even if companies agree to share
their data, implementing privacy-compliant interfaces
between the companies is necessary. From an OEM’s
perspective, the more upstream a supplier’s location is
in the supply chain, the more difficult it becomes to
obtain operational data. In practice, OEMs only know
their first-, second- and third-tier suppliers. This lim-
ited knowledge contrasts with many of the researched
RL applications in which the agent has a global view of
the supply chain. However, not only the acquisition of
external data can be a challenge. Even accessing internal
company data sometimes requires customised software
solutions, as many closed-source ERP systems do not
offer interfaces for data exchange with external software
applications. Against this background, we see a need for
detailed procedure models that describe the implemen-
tation of RL-based decision support systems from an
operations research perspective and a holistic software
engineering perspective.
Therefore, we can identify an essential need for pub-
licly available data for future research and the general
reproducibility of experiments. The Open Data paradigm
is currently gaining momentum, encouraging the free
provision of data (Braunschweig et al. 2012). SCM could
also benefit from these new data sources, especially in
disruption monitoring, as important geopolitical, ecolog-
ical, or economic events are quickly accessible on the
Internet. Furthermore, benchmark problems and open
datasets for training RL agents in supply chain environ-
ments are necessary to strengthen future research in this
domain.
7.2. Reliability and user-acceptance of
reinforcement learning decisions in SCM
Even though industrial applications are still scarce, the
benefits of RL algorithms for SCM problems are apparent.
First, agent models trained by RL allow data-driven
and autonomous decision-making in real time. Thus,
RL enables response to sudden disturbances in the sup-
ply chain by determining alternative decisions based on
the latest information from the supply chain. Second,
RL methods can compute alternative decisions in case
of sudden disturbances and adapt control strategies in
case of gradual changes in the supply chain data. By
this means, RL provides a fundament for developing
resilient decision support systems for SCM. Third, RL
methods also facilitate human-centred decision-making,
as they consider human experience when learning a deci-
sion policy. For instance, this policy can be achieved by
implementing that a human must first confirm agent-
computed decisions before they can be applied. The con-
firmation or rejection of a decision can then be used as
an additional reward and thus as a learning signal for the
agent.
A fundamental practical problem of machine learning
approaches is that the corresponding models (in particu-
lar deep neural networks) are black-box methods. Thus,
actions suggested by the agent might be difficult to com-
prehend by human decision-makers. In particular, if the
agent’s decisions deviate strongly from the intuition of
the human planner, the acceptance of the agent’s deci-
sion is diminished. A simulative validation of the agent’s
decision and meaningful visualisation of the simulation
results can help increase acceptance. Another concep-
tual problem is that the agent’s generated solution is a
data-based prediction. When assuming constant agent
parameters, the same data input will always result in
the same output. If the agent’s proposed solution is not
feasible, the agent natively has no strategies to adapt a
solution until it is feasible. One way to address this prob-
lem is to implement additional heuristic search strategies
deployed in case of an infeasible solution. The alternative
identified solution can, in turn, be used as a learning sig-
nal to adjust the agent’s decision policy. However, apply-
ing heuristic search strategies can significantly reduce
the real-time capability of the agent. For instance, Wang
et al. (2022) present a promising example of combining
RL and traditional methods of operations research.
Current RL models focus on decision-making either
with complete information or incomplete information
but they rarely assess the proposed actions regarding
resilience and human acceptance. However, these are
key characteristics for implementing RL-based decision
support systems in industrial environments. In the Post-
COVID-19 era many companies will shift to a more
resilient and transparent decision-making, as they have
encountered the consequences of neglecting these factors
(Schroeder et al. 2021). The prevalence of these factors as
objectives in decision support systems is still low. Hence,

we can identify a need for resilient and human-centred
RL in SCM. Another research gap is the application of
hybrid approaches such as the combination of RL and
metaheuristics. These approaches have been applied in
other domains but are currently limited to very few appli-
cations in the context of SCM.
8. Conclusion
In this paper, we performed a semi-systematic review to
understand algorithms, applications and practical adop-
tion of RL in SCM. Our results show that RL applications
in SCM date back to 2000. The publication dynamics
were steady until the number of publications rose dras-
tically in 2019, which can be attributed to the computa-
tional hardware, growing data quantities, and the advent
of deep learning. Most publications can be assigned to
computer science, engineering, and mathematics. The
bibliometric network consists of low linkage between the
considered publications concerning citations, but there
are many similarities in the keywords used.
As one of the major outcomes of our study, we pro-
posed a hierarchic classification framework that cate-
gorises RL applications to SCM according to four criteria
with several classes and subclasses. The criteria include
supply chain drivers, algorithms, data sources, and indus-
trial sectors.
8.1. Research question 1: what are the main
applications of reinforcement learning in SCM?
According to the supply chain driver classification, inven-
tory management problems are the most common appli-
cation in SCM, followed by information and transporta-
tion problems. The models that address inventory man-
agement problems usually use the RL agent to orches-
trate the material flow between multiple sites in the sup-
ply chain. Information is a broader class that includes
applications that aim to increase information availability,
such as forecasting, collaboration, or risk management.
In the transportation class, RL is used to address vehi-
cle routing or scheduling problems. The prevalence of RL
models in planning tasks with short decision horizons is
significantly higher than in long-term decision-making
because they mostly have a limited scope, precise inputs
and objectives, and a requirement for fast decision-
making. On the contrary, long-term decisions taken on
the managerial level require valid reasoning. However,
the decision-making time is irrelevant, and a quantita-
tive model cannot fully reflect the problem structure and
complexity. Therefore, due to the limited suitability, the
prevalence of RL in long-term decision-making is much
lower than for short-term tasks.
INTERNATIONAL JOURNAL OF PRODUCTION RESEARCH 7171
8.2. Research question 2: what are the main
algorithms of reinforcement learning in SCM?
The OpenAI taxonomy has been modified to classify
papers based on the applied RL algorithm. Among the
classified publications, most models take advantage of Q-
learning. This fact can be explained by the maturity of
the long-standing technique capable of learning stochas-
tic transitions and reward problems. Besides, Q-learning
belongs to the model-free class, implying that an RL
agent can operate directly by sampling and learning the
expected rewards. However, more recent publications use
such deep learning techniques as DQN, A2C, A3C, and
PPO. By incorporating deep artificial neural networks,
RL agents gain the capabilities to make decisions from
large-scale and potentially unstructured input data.
8.3. Research question 3: how widespread is
reinforcement learning for solving industrial use
cases?
Most RL models consider generic supply chains with
artificial data regarding the data sources and industrial
sectors. Only a few publications use public or institu-
tional data from academic literature or industry. Some
authors also address the requirements of supply chains
in specific industrial sectors such as healthcare, hydro-
gen distribution, or mining. Nevertheless, the theoretical
models show great potential for various SCM tasks. RL
models are capable of data-driven decision-making in
real-time, making them capable of responding to sudden
disturbances in the supply chain. In the ever-changing
supply chain environments, agents can adapt their con-
trol strategies in case of gradual changes in the supply
chain data. Furthermore, RL facilitates human-centric
decision-making by learning from human experience,
but it is still a black-box approach without reasoning
capabilities. The main problem in industrial deployment
is data streaming. Even trained agents require continuous
access to supply chain data in real-time, a capacity many
companies currently do not have. Furthermore, access is
usually limited to internal data, but external data from
upstream suppliers is also crucial for decision-making.
Finally, we note that RL applications can be extended
toward the novel SCM context driven by tremendous dis-
ruptions in the years of pandemic, geopolitical shocks,
component shortages, and transportation bottlenecks
and manifested in the new paradigms such as the short-
age economy (Ivanov and Dolgui 2022) and supply chain
viability (Ivanov 2020c; Ruel et al. 2021). RL applications
have high potential in the emerging paradigms of Indus-
try 5.0 (Ivanov 2022b) and reconfigurable supply chain
(Dolgui, Ivanov, and Sokolov 2020).
7172 B. ROLF ET AL.
8.4. Limitations
The applied review methodology has several limitations.
The study provides an exhaustive literature review of the
Scopus database but does not include other databases.
Other databases are expected to overlap with Scopus,
but we might have missed some relevant publications.
Regarding the search terms and keywords, we tried to
make them as general as possible to include many pub-
lications, but some relevant contributions might be miss-
ing. Especially research communities that apply RL to
supply chain problems with the emphasis on transporta-
tion, pricing, or location planning might not mention
the term ‘supply chain’, making them undetectable for
our study. Hence, we expect these domains to have more
contributions than considered in this literature review.
Disclosure statement
No potential conflict of interest was reported by the author(s).
Data availability statement
The authors confirm that the data supporting the findings of
this study are available within the article. The data were derived
from the Scopus database, available at https://www.scopus.com.
Notes on contributors
Benjamin Rolf is a researcher and PhD
candidate in the Institute of Logistics
and Material Handling Systems at Otto-
von-Guericke-University Magdeburg. He
holds a master’s degree in Industrial Engi-
neering with a specialization in Logis-
tics from Otto-von-Guericke-University
Magdeburg. His main research interest
includes the application of machine learning, simulation and
network theory in supply chain management. In his PhD thesis,
he currently works on the dynamic reconfiguration of supply
chains as a reaction to disruptions using graph-based artificial
intelligence and open data.
Dr. Ilya Jackson is a Postdoctoral Asso-
ciate at MIT Center for Transportation
& Logistics. He earned his PhD in Civil
Engineering and Transportation from the
Transport and Telecommunication Insti-
tute, where he spent one year as an assis-
tant professor shortly after that. The main
ideas of his PhD thesis had been summa-
rized in the paper ‘Neuroevolutionary approach to metamodel-
based optimization in production and logistics’, which received
the young researcher award in 2020. Dr. Ilya Jackson currently
focuses on reinforcement learning for supply chain synchro-
nization and domain-specific automated machine learning for
supply chain management and logistics.
Marcel Müller is a research fellow at the
Otto-von-Guericke-University Magdeburg.
He earned his master degree in Indus-
trial Engineering for Logistics at the Otto-
von-Guericke-University Magdeburg. His
research interests include modeling and
simulation of logistics systems and han-
dling of deadlocks. His email address is
marcel1.mueller@ovgu.de. His website is https://www.ilm.ovgu.
de/mueller.
Sebastian Lang is research fellow at the
Fraunhofer Institute for Factory Opera-
tion and Automation IFF. He holds a
master’s degree in mechanical engineer-
ing with focus on production technologies
and a master’s and bachelor’s degree in
industrial engineering and logistics. His
research interests include studying and
applying methods of machine learning and artificial intelli-
gence, simulation modeling, and mathematical optimization
for problems in production and logistics. His e-mail address
is sebastian.lang@iff.fraunhofer.de. His ResearchGate profile is
https://www.researchgate.net/profile/Sebastian_Lang5.
Tobias Reggelin is a project manager,
researcher, and lecturer at Otto von Guer-
icke University Magdeburg and Fraun-
hofer Institute for Factory Operation and
Automation IFF Magdeburg. His main
research and work interests include mod-
eling and simulation of production and
logistics systems and developing and apply-
ing logistics management games. Tobias Reggelin received a
doctoral degree in engineering from the Otto-von-Guericke-
University Magdeburg. Furthermore, he holds a master’s
degree in Engineering Management from the Rose-Hulman
Institute of Technology in Terre Haute, IN and a diploma
degree in Industrial Engineering in Logistics from the Otto-
von-Guericke-University Magdeburg. His email address is
tobias.reggelin@ovgu.de.
Dmitry Ivanov is a professor of sup-
ply chain and operations management
at Berlin School of Economics and Law
(HWR Berlin). His publication list includes
around 380 publications, including over
120 papers in international academic jour-
nals and leading textbooks Global Supply
Chain and Operations Management and
Introduction to Supply Chain Resilience. His main research
interests and results span the resilience and ripple effect in sup-
ply chains, risk analytics, and digital twins. He co-edits Inter-
national Journal of Integrated Supply Management (IJISM)
and is an associate editor of the International Journal of Pro-
duction Research (IJPR) and OMEGA. He is Chairman of
IFAC TC 5.2 ‘Manufacturing Modelling for Management and
Control’.
ORCID
Benjamin Rolf http://orcid.org/0000-0002-5454-8894

Ilya Jackson http://orcid.org/0000-0002-7457-6040
Marcel Müller http://orcid.org/0000-0001-9865-7331
Sebastian Lang http://orcid.org/0000-0003-3397-1551
Tobias Reggelin http://orcid.org/0000-0003-3001-9821
References
Aboutorab, H., O. K. Hussain, M. Saberi, and F. K. Hus-
sain. 2022. “A Reinforcement Learning-Based Framework
for Disruption Risk Identification in Supply Chains.” Future
Generation Computer Systems126: 110–122.
Achiam, J. 2018. “OpenAI Spinning up. GitHub Repository.”
Accessed 16 February 2022. https://spinningup.openai.
com/.
Adi, Taufik Nur, Hyerim Bae, and Yelita Anggiane Iskan-
dar. 2021. “Interterminal Truck Routing Optimization Using
Cooperative Multiagent Deep Reinforcement Learning.”
Processes 9 (10): 1728.
Adi, T. N., Y. A. Iskandar, H. Bae, and Y. Choi. 2020. “Reduction
of Number of Empty-Truck Trips in Inter-Terminal Trans-
portation Using Multi-agent Q-learning.” In Interconnected
Supply Chains in An Era of Innovation – Proceedings of the
8th International Conference on Information Systems, Logis-
tics and Supply Chain, ILS 2020, Austin, TX, USA, 167–
172.
Aghaie, A., and M. H. Heidary. 2019. “Simulation-Based Opti-
mization of a Stochastic Supply Chain Considering Sup-
plier Disruption: Agent-Based Modeling and Reinforcement
Learning.” Scientia Iranica26 (6): 3780–3795.
Aissani, N., A. Bekrar, D. Trentesaux, and B. Beldjilali.
2012. “Dynamic Scheduling for Multi-Site Companies: A
Decisional Approach Based on Reinforcement Multi-agent
Learning.” Journal of Intelligent Manufacturing 23 (6):
2513–2529.
Alves, J. C., and G. R. Mateus. 2020. “Deep Reinforcement
Learning and Optimization Approach for Multi-echelon
Supply Chain with Uncertain Demands.” Lecture Notes in
Computer Science (Including Subseries Lecture Notes in Arti-
ficial Intelligence and Lecture Notes in Bioinformatics) 12433
LNCS, 584–599. Berlin, Heidelberg: Springer.
Back, J.-G., C. O. Kim, and I.-H. Kwon. 2006. “An Adaptive
Inventory Control Model for a Supply Chain with Non-
stationary Customer Demands.” Lecture Notes in Computer
Science (including Subseries Lecture Notes in Artificial Intel-
ligence and Lecture Notes in Bioinformatics) 4099 LNAI,
895–900. Berlin, Heidelberg: Springer.
Baird, Leemon. 1995. “Residual Algorithms: Reinforcement
Learning with Function Approximation.” In Machine Learn-
ing Proceedings 1995, 30–37. Elsevier. doi:10.1016/b978-1-
55860-377-6.50013-x.
Barat, S., H. Khadilkar, H. Meisheri, V. Kulkarni, V. Baniwal,
P. Kumar, and M. Gajrani. 2019. “Actor Based Simulation
for Closed Loop Control of Supply Chain Using Reinforce-
ment Learning.” In Proceedings of the International Joint
Conference on Autonomous Agents and Multiagent Systems,
AAMAS, Montreal, QC, Canada, 1802–1804. Vol. 3.
Barat, S., P. Kumar, M. Gajrani, H. Khadilkar, H. Meisheri,
V. Baniwal, and V. Kulkarni. 2020. "Reinforcement Learn-
ing of Supply Chain Control Policy Using Closed Loop
Multi-agent Simulation.” Lecture Notes in Computer Science
INTERNATIONAL JOURNAL OF PRODUCTION RESEARCH 7173
(Including Subseries Lecture Notes in Artificial Intelligence
and Lecture Notes in Bioinformatics) 12025 LNAI, 26–38.
Cham: Springer.
Bengio, Yoshua. 2016. Deep Learning. Adaptive Computation
and Machine Learning Series. London, England: MIT Press.
Bharti, S., D. S. Kurian, and V. M. Pillai. 2020. “Reinforce-
ment Learning for Inventory Management.” Lecture Notes in
Mechanical Engineering, 877–885. Singapore: Springer.
Boute, Robert N., Joren Gijsbrechts, Willem van Jaarsveld, and
Nathalie Vanvuchelen. 2022. “Deep Reinforcement Learn-
ing for Inventory Control: A Roadmap.” European Journal
of Operational Research298 (2): 401–412.
Braunschweig, Katrin, Julian Eberius, Maik Thiele, and Wolf-
gang Lehner. 2012. “The State of Open Data – Limits of
Current Open Data Platforms.” In WWW ’12: Proceedings of
the 21st International Conference on World Wide Web, Lyon,
France.
Bretas, A., A. Mendes, S. Chalup, M. Jackson, R. Clement, and
C. Sanhueza. 2019. “Modelling Railway Traffic Management
Through Multi-agent Systems and Reinforcement Learn-
ing.” In 23rd International Congress on Modelling and Sim-
ulation – Supporting Evidence-Based Decision Making: The
Role of Modelling and Simulation, MODSIM 2019, Canberra,
Australia, 291–297.
Brynjolfsson, Erik, Wang Jin, and Kristina McElheran. 2021.
“The Power of Prediction: Predictive Analytics, Workplace
Complements, and Business Performance.” Business Eco-
nomics 56 (4): 217–239.
Bukkapatnam, S., and G. Gao. 2000. “Effect of Reinforcement
Learning on Coordination of Multiagent Systems.” Proceed-
ings of SPIE – The International Society for Optical Engineer-
ing 4208: 31–41.
Burgos, Diana, and Dmitry Ivanov. 2021. “Food Retail Supply
Chain Resilience and the COVID-19 Pandemic: A Digital
Twin-Based Impact Analysis and Improvement Directions.”
Transportation Research Part E: Logistics and Transportation
Review 152: 102412.
Cao, H., H. Xi, and S. F. Smith. 2003. “A Reinforcement
Learning Approach to Production Planning in the Fabri-
cation/fulfillment Manufacturing Process.” In Winter Sim-
ulation Conference Proceedings, New Orleans, LA, USA,
1417–1423. Vol. 2.
Cavalcante, Ian M., Enzo M. Frazzon, Fernando A. Forcellini,
and Dmitry Ivanov. 2019. “A Supervised Machine Learning
Approach to Data-driven Simulation of Resilient Supplier
Selection in Digital Manufacturing.” International Journal of
Information Management 49: 86–97.
Chaharsooghi, S. K., J. Heydari, and S. H. Zegordi. 2008.
“A Reinforcement Learning Model for Supply Chain Order-
ing Management: An Application to the Beer Game.” Deci-
sion Support Systems 45 (4): 949–959.
Chatzidimitriou, K. C., A. L. Symeonidis, and P. A. Mitkas.
2013. “Policy Search Through Adaptive Function Approxi-
mation for Bidding in TAC SCM.” Lecture Notes in Business
Information Processing136: 16–29.
Chen, H., Z. Chen, F. Lin, and P. Zhuang. 2021. “Effec-
tive Management for Blockchain-Based Agri-Food Supply
Chains Using Deep Reinforcement Learning.” IEEE Access
9: 36008–36018.
Chien, C.-F., Y.-S. Lin, and S.-K. Lin. 2020. “Deep Reinforce-
ment Learning for Selecting Demand Forecast Models to
7174 B. ROLF ET AL.
Empower Industry 3.5 and An Empirical Study for a Semi-
conductor Component Distributor.” International Journal of
Production Research 58 (9): 2784–2804.
Chodura, D., P. Dominik, and J. Koźlak. 2011. “Market
Strategy Choices Made by Company Using Reinforcement
Learning.” Advances in Intelligent and Soft Computing 90:
83–90.
Choi, Tsan-Ming, Alexandre Dolgui, Dmitry Ivanov, and
Erwin Pesch. 2022. “OR and Analytics for Digital, Resilient,
and Sustainable Manufacturing 4.0.” Annals of Operations
Research 310 (1): 1–6.
Chopra, Sunil, and P. Meindl. 2013. Supply Chain Management:
Strategy, Planning, and Operation. 5th ed. Boston: Pearson
Education.
Chopra, Sunil, and Man Mohan S. Sodhi. 2004. “Managing Risk
to Avoid Supply-Chain Breakdown.” MIT Sloan Manage-
ment Review 46 (1): 53–61.
Collins, John, Raghu Arunachalam, Norman Sadeh, Joakim
Eriksson, Niclas Finne, and Sverker Janson. 2006. “The Sup-
ply Chain Management Game for the 2007 Trading Agent
Competition.” http://reports-archive.adm.cs.cmu.edu/
anon/isri2007/CMU-ISRI-07-100.pdf.
Craven, T. J., and C. C. Krejci. 2016. “Assessing Manage-
ment Strategies for Intermediated Regional Food Supply
Networks.” In 2016 International Annual Conference of the
American Society for Engineering Management, ASEM 2016,
Charlotte, North Carolina, USA, 21–48.
Craven, T. J., and C. C. Krejci. 2017. “An Agent-Based Model
of Regional Food Supply Chain Disintermediation.” In ADS
’17: Proceedings of the Agent-Directed Simulation Symposium,
Virginia Beach, Virginia, USA, 83–92. Vol. 49.
Dahlem, D., and W. Harrison. 2010. “Collaborative Function
Approximation in Social Multiagent Systems.” In Proceed-
ings – 2010 IEEE/WIC/ACM International Conference on
Intelligent Agent Technology, IAT 2010, Washington, DC,
USA, 48–55. Vol. 2.
Dangelmaier, W., T. Rust, A. Döring, and B. Klöpper. 2006.
“A Reinforcement Learning Approach for Learning Coor-
dination Rules in Production Networks.” In CIMCA 2006:
International Conference on Computational Intelligence for
Modelling, Control and Automation, Jointly with IAWTIC
2006: International Conference on Intelligent Agents Web
Technologies, Sydney, Australia, 84.
Das, Tapas K., Abhijit Gosavi, Sridhar Mahadevan, and
Nicholas Marchalleck. 1999. “Solving Semi-Markov Deci-
sion Problems Using Average Reward Reinforcement Learn-
ing.” Management Science45 (4): 560–574.
Dayan, Peter, and Geoffrey E. Hinton. 1997. “Using Expecta
tion-Maximization for Reinforcement Learning.” Neural
Computation 9 (2): 271–278.
De Maio, C., G. Fenza, V. Loia, F. Orciuoli, and E. Herrera-
Viedma. 2016. “A Framework for Context-aware Hetero-
geneous Group Decision Making in Business Processes.”
Knowledge-Based Systems 102: 39–50.
DispoWeb. 2004. “Dispositive Supply Web Coordination with
Multi-Agent Systems.” https://www.aot.tu-berlin.de/index.
php?id = 1764&L = 1.
Dogan, I., and A. R. Güner. 2015. “A Reinforcement Learning
Approach to Competitive Ordering and Pricing Problem.”
Expert Systems 32 (1): 39–48.
Dolgui, Alexandre, and Dmitry Ivanov. 2022. “5G in Digi-
tal Supply Chain and Operations Management: Fostering
Flexibility, End-to-end Connectivity and Real-Time Visibil-
ity Through Internet-of-everything.” International Journal of
Production Research 60 (2): 442–451.
Dolgui, Alexandre, Dmitry Ivanov, and Maxim Rozhkov. 2019.
“Does the Ripple Effect Influence the Bullwhip Effect? An
Integrated Analysis of Structural and Operational Dynam-
ics in the Supply Chain.” International Journal of Production
Research 58 (5): 1285–1301.
Dolgui, Alexandre, Dmitry Ivanov, and Boris Sokolov. 2020.
“Reconfigurable Supply Chain: The X-network.” Inter-
national Journal of Production Research 58 (13): 4138–
4163.
Du, H., and Y. Jiang. 2019. “Backup Or Reliability Improve-
ment Strategy for a Manufacturer Facing Heterogeneous
Consumers in a Dynamic Supply Chain.” IEEE Access 7:
50419–50430.
Du, H., and T. Xiao. 2019. “Pricing Strategies for Compet-
ing Adaptive Retailers Facing Complex Consumer Behavior:
Agent-Based Model.” International Journal of Information
Technology and Decision Making 18 (6): 1909–1939.
Emerson, Denise, Wei Zhou, and Selwyn Piramuthu. 2009.
“Goodwill, Inventory Penalty, and Adaptive Supply Chain
Management.” European Journal of Operational Research 199
(1): 130–138.
Filatova, D., C. El-Nouty, and R. V. Fedorenko. 2021.
“Some Theoretical Backgrounds for Reinforcement Learn-
ing Model of Supply Chain Management Under Stochastic
Demand.” In International Conference on Information and
Digital Technologies 2021, IDT 2021, Zilina, Slovakia, 24–30.
Fleischmann, Bernhard, Herbert Meyr, and Michael Wagner.
2008. “Advanced Planning.” In Supply Chain Management
and Advanced Planning, edited by Hartmut Stadtler, and
Christoph Kilger, 81–106. Berlin, Heidelberg: Springer.
Ganesan, V. K., D. Sundararaj, and A. P. Srinivas. 2021. “Adap-
tive Inventory Replenishment for Dynamic Supply Chains
with Uncertain Market Demand.” Lecture Notes in Mechan-
ical Engineering, 325–335. Singapore: Springer.
Gang, Z., and S. Ruoying. 2006. “Policy Transition of Rein-
forcement Learning for An Agent Based SCM System.” In
2006 IEEE International Conference on Industrial Informat-
ics, INDIN’06, Singapore, 793–798.
Ghavamipoor, H., and S. A. Hashemi Golpayegani. 2020.
“A Reinforcement Learning Based Model for Adaptive Ser-
vice Quality Management in E-Commerce Websites.” Busi-
ness and Information Systems Engineering 62 (2): 159–177.
Ghavamzadeh, Mohammad, Shie Mannor, Joelle Pineau, and
Aviv Tamar. 2015. “Bayesian Reinforcement Learning: A
Survey.” Foundations and Trends® in Machine Learning 8
(5-6): 359–483. ArXiv: 1609.04436.
Ghorbel, N., S.-A. Addouche, and A. El Mhamedi. 2015.
“Forward Management of Spare Parts Stock Shortages Via
Causal Reasoning Using Reinforcement Learning.” IFAC-
PapersOnLine 48: 1061–1066.
Giannoccaro, I., and P. Pontrandolfo. 2002. “Inventory Man-
agement in Supply Chains: A Reinforcement Learning
Approach.” International Journal of Production Economics 78
(2): 153–161.
Glasserman, Paul, and Sridhar Tayur. 1995. “Sensitivity Anal-
ysis for Base-Stock Levels in Multiechelon Production-
inventory Systems.” Management Science 41 (2): 263–281.
Govindan, R., and T. Al-Ansari. 2019. “Computational Deci-
sion Framework for Enhancing Resilience of the Energy,

Water and Food Nexus in Risky Environments.” Renewable
and Sustainable Energy Reviews 112: 653–668.
Gu, Shixiang, Ethan Holly, Timothy Lillicrap, and Sergey
Levine. May 2017. “Deep Reinforcement Learning for
Robotic Manipulation with Asynchronous Off-policy
Updates.” In 2017 IEEE International Conference on Robotics
and Automation (ICRA). IEEE.
Gutierrez-Franco, E., C. Mejia-Argueta, and L. Rabelo. 2021.
“Data-driven Methodology to Support Long-lasting Logis-
tics and Decision Making for Urban Last-mile Operations.”
Sustainability (Switzerland) 13 (11): 6230.
Habib, A., M. I. Khan, and J. Uddin. 2017. “Optimal Route
Selection in Complex Multi-Stage Supply Chain Networks
Using SARSA(λ).” In 19th International Conference on Com-
puter and Information Technology, ICCIT 2016, Dhaka,
Bangladesh, 170–175.
Hachaichi, Y., Y. Chemingui, and M. Affes. 2020. “A Policy
Gradient Based Reinforcement Learning Method for Sup-
ply Chain Management.” In Proceedings of the International
Conference on Advanced Systems and Emergent Technologies,
IC_ASET 2020, Tunis, Tunisia, 135–140.
Hax, Arnoldo C., and Dan Candea. 1984. Production and Inven-
tory Management. Englewood Cliffs: Prentice-Hall.
Hirano, M., H. Matsushima, K. Izumi, and T. Mukai. 2021.
“Simulation of Unintentional Collusion Caused by Auto
Pricing in Supply Chain Markets.” Lecture Notes in Com-
puter Science (Including Subseries Lecture Notes in Artifi-
cial Intelligence and Lecture Notes in Bioinformatics) 12568
LNAI, 352–359. Cham: Springer.
Hosseini, Seyedmohsen, and Dmitry Ivanov. 2021. “A Multi-
layer Bayesian Network Method for Supply Chain Dis-
ruption Modelling in the Wake of the COVID-19 Pan-
demic.” International Journal of Production Research 60 (17):
5258–5276.
Huang, H., and X. Tan. 2021. “Application of Reinforcement
Learning Algorithm in Delivery Order System Under Sup-
ply Chain Environment.” Mobile Information Systems 2021:
1–11.
Hwangbo, S., and C. Yoo. 2018. “A Methodology of a Hybrid
Hydrogen Supply Network (HHSN) Under Alternative
Energy Resources (AERs) of Hydrogen Footprint Constraint
for Sustainable Energy Production (SEP).” Computer Aided
Chemical Engineering 43: 343–348.
Hyndman, Rob J., and George Athanasopoulos. 2018. Forecast-
ing: Principles and Practice. 2nd ed. Melbourne: OTexts.
Instacart. 2017. “Instacart Market Basket Analysis.” https://
www.kaggle.com/competitions/instacart-market-basket-
analysis/overview.
Isele, David, Reza Rahimi, Akansel Cosgun, Kaushik Subrama-
nian, and Kikuo Fujimura. May 2018. “Navigating Occluded
Intersections with Autonomous Vehicles Using Deep Rein-
forcement Learning.” In 2018 IEEE International Conference
on Robotics and Automation (ICRA). IEEE.
Ivanov, Dmitry. 2020a. “‘A Blessing in Disguise’ Or ‘as If it
Wasn’t Hard Enough Already’: Reciprocal and Aggravate
Vulnerabilities in the Supply Chain.” International Journal of
Production Research 58 (11): 3252–3262.
Ivanov, Dmitry. 2020b. “Predicting the Impacts of Epidemic
Outbreaks on Global Supply Chains: A Simulation-Based
Analysis on the Coronavirus Outbreak (COVID-19/SARS-
CoV-2) Case.” Transportation Research Part E: Logistics and
Transportation Review 136: 101922.
INTERNATIONAL JOURNAL OF PRODUCTION RESEARCH 7175
Ivanov, Dmitry. 2020c. “Viable Supply Chain Model: Integrat-
ing Agility, Resilience and Sustainability Perspectives–les
sons From and Thinking Beyond the COVID-19 Pandemic.”
Annals of Operations Research 1–21.
Ivanov, Dmitry. 2021a. “Digital Supply Chain Management and
Technology to Enhance Resilience by Building and Using
End-to-end Visibility During the COVID-19 Pandemic.”
IEEE Transactions on Engineering Management 1–11.
Ivanov, Dmitry. 2021b. Introduction to Supply Chain Resilience:
Management, Modelling, Technology. Cham: Springer Nature.
Ivanov, Dmitry. 2022a. “Blackout and Supply Chains: Cross-
Structural Ripple Effect, Performance, Resilience and Via-
bility Impact Analysis.” Annals of Operations Research
1–17.
Ivanov, Dmitry. 2022b. “The Industry 5.0 Framework: Viabi
lity-Based Integration of the Resilience, Sustainability, and
Human-centricity Perspectives.” International Journal of
Production Research.
Ivanov, Dmitry. 2022c. “Probability, Adaptability, and Time:
Some Research-Practice Paradoxes in Supply Chain Resi
lience and Viability Modeling.” International Journal of Inte-
grated Supply Management 15 (4): 454–465.
Ivanov, Dmitry, and Alexandre Dolgui. 2021. “A Digital Sup-
ply Chain Twin for Managing the Disruption Risks and
Resilience in the Era of Industry 4.0.” Production Planning
& Control 32 (9): 775–788.
Ivanov, Dmitry, and Alexandre Dolgui. 2022. “The Shortage
Economy and Its Implications for Supply Chain and Oper-
ations Management.” International Journal of Production
Research.
Ivanov, Dmitry, Alexandre Dolgui, and Boris Sokolov. 2012.
“Applicability of Optimal Control Theory to Adaptive Supply
Chain Planning and Scheduling.” Annual Reviews in Control
36 (1): 73–84.
Ivanov, Dmitry, Alexandre Dolgui, and Boris Sokolov. 2022.
“Cloud Supply Chain: Integrating Industry 4.0 and Digital
Platforms in the “Supply Chain-as-a-Service”.” Transporta-
tion Research Part E: Logistics and Transportation Review
160: 102676.
Ivanov, Dmitry, and Boris Sokolov. 2013. “Control and System-
Theoretic Identification of the Supply Chain Dynamics
Domain for Planning, Analysis and Adaptation of Perfor-
mance Under Uncertainty.” European Journal of Operational
Research 224 (2): 313–323.
Ivanov, Dmitry, Christopher S. Tang, Alexandre Dolgui, Daria
Battini, and Ajay Das. 2021. “Researchers’ Perspectives on
Industry 4.0: Multi-disciplinary Analysis and Opportuni-
ties for Operations Management.” International Journal of
Production Research 59 (7): 2055–2078.
Jackson, Ilya. 2020. “Neuroevolutionary Approach to Meta-
modeling of Production-Inventory Systems with Lost-Sales
and Markovian Demand.” In Lecture Notes in Networks and
Systems, 90–99. Cham: Springer International Publishing.
Jiang, C.. 2008. “Two-dimensional Learning Mechanisms for
Alliance Members in Multi-agent Supply Chains.” In Pro-
ceedings of the International Conference on Information Man-
agement, Innovation Management and Industrial Engineer-
ing, ICIII 2008, Taipei, Taiwan, 524–527. Vol. 2.
Jiang, C., and Z. Sheng. 2009. “Case-Based Reinforcement
Learning for Dynamic Inventory Control in a Multi-agent
Supply-chain System.” Expert Systems with Applications 36
(3 PART 2): 6520–6526.
7176 B. ROLF ET AL.
Kaihara, T., and S. Fujii. 2008. “Supply Chain Management
for Virtual Enterprises with Adaptive Multi-agent Mecha-
nism.” International Journal of Manufacturing Technology
and Management 14 (3-4): 299–310.
Kegenbekov, Z., and I. Jackson. 2021. “Adaptive Supply Chain:
Demand-Supply Synchronization Using Deep Reinforce-
ment Learning.” Algorithms 14 (8): 240.
Khan, Saif M., Alexander Mann, and Dahlia Peterson.
2021. The Semiconductor Supply Chain: Assessing National
Competitiveness. Technical Report. Center for Security
and Emerging Technology. Accessed 27 December 2021.
https://cset.georgetown.edu/publication/the-semi
conductor-supply-chain/.
Kim, T., R. U. Bilsel, and S. Kumara. 2008. “Supplier Selection in
Dynamic Competitive Environments.” International Journal
of Services Operations and Informatics 3 (3-4): 283–293.
Kim, C. O., J. Jun, J. K. Baek, R. L. Smith, and Y. D. Kim. 2005.
“Adaptive Inventory Control Models for Supply Chain Man-
agement.” International Journal of Advanced Manufacturing
Technology 26 (9-10): 1184–1192.
Kim, C. O., I.-H. Kwon, and J.-G. Baek. 2008. “Asynchronous
Action-reward Learning for Nonstationary Serial Supply
Chain Inventory Control.” Applied Intelligence 28 (1): 1–16.
Ko, J. M., C. Kwak, Y. Cho, and C. O. Kim. 2011. “Adaptive Prod-
uct Tracking in RFID-enabled Large-Scale Supply Chain.”
Expert Systems with Applications 38 (3): 1583–1590.
Krakovsky, Marina. 2016. “Reinforcement Renaissance.”
Communications of the ACM 59 (8): 12–14.
Kwon, I.-H., C. O. Kim, J. Jun, and J. H. Lee. 2008. “Case-Based
Myopic Reinforcement Learning for Satisfying Target Ser-
vice Level in Supply Chain.” Expert Systems with Applications
35 (1-2): 389–397.
Landry, Sylvain, and Martin Beaulieu. 2013. “The Challenges
of Hospital Supply Chain Management, From Central Stores
to Nursing Units.” In Handbook of Healthcare Operations
Management: Methods and Applications, edited by Brian T
Denton, 465–482. New York: Springer.
Lee, Y. S., and R. Sikora. 2019. “Application of Adaptive Strat-
egy for Supply Chain Agent.” Information Systems and E-
Business Management 17 (1): 117–157.
Li, J., P. Guo, and Z. Zuo. 2008. “Inventory Control Model
for Mobile Supply Chain Management.” In Proceedings –
The 2008 International Conference on Embedded Software
and Systems Symposia, ICESS Symposia, Chengdu, China,
459–463.
Li, X., W. Luo, M. Yuan, J. Wang, J. Lu, J. Wang, J. Lu, and J.
Zeng. 2021. “Learning to Optimize Industry-Scale Dynamic
Pickup and Delivery Problems.” In Proceedings – Inter-
national Conference on Data Engineering, Chania, Greece,
2511–2522
Li, H., T. Pang, Y. Wu, and G. Jiang. 2014. “Conflict Resolution
of Production-marketing Collaborative Planning Based on
Multi-agent Self-adaptation Negotiation.” In ICAART 2014
– Proceedings of the 6th International Conference on Agents
and Artificial Intelligence, Angers, France, 209–214. Vol. 2.
Li, Y., and J.-M. Zhao. 2006. “Applying Adaptive Multi-agent
Modeling in Agile Supply Chain Simulation.” In Proceedings
of the 2006 International Conference on Machine Learning
and Cybernetics, Dalian, China, 4191–4196.
Li, C. Y., S. H. Zhao, T. W. Zhang, and X. T. Wang. 2015. “Rein-
forcement Learning of Fuzzy Joint Replenishment Problem
in Supply Chain.” In Electronic Engineering and Information
Science – Proceedings of the 2015 International Conference on
Electronic Engineering and Information Science, ICEEIS 2015,
Harbin, China, 779–782.
Lin, F.-R., and Y.-H. Pai. 2000. “Using Multi-agent Simula-
tion and Learning to Design New Business Processes.” IEEE
Transactions on Systems, Man, and Cybernetics Part A: Sys-
tems and Humans. 30 (3): 380–384.
Liu, C. 2020. “Outsourcing Strategies for Manufacturers Fac-
ing Reputation Oriented Consumers.” In Proceedings – 2020
Chinese Automation Congress, CAC 2020, Shanghai, China,
1980–1985.
Liu, H., E. Howley, and J. Duggan. 2011. “An Agent-Based
Simulation of the Effects of Consumer Behavior on Market
Price Dynamics.” In Proceedings of the IASTED International
Conference on Applied Simulation and Modelling, ASM 2011,
Crete, Greece, 316–325.
Lu, W., H. Tan, X. Yan, and C. Lv. 2021. “Supply Chain Schedul-
ing Using Double Deep Time-Series Differential Neural
Network.” In 5th International Workshop on Advances
in Energy Science and Environment Engineering (AESEE
2021), E3S Web of Conferences 257, Xiamen, China.
MacCarthy, Bart L., and Dmitry Ivanov. 2022a. The Digital
Supply Chain. Amsterdam: Elsevier.
MacCarthy, Bart L., and Dmitry Ivanov. 2022b. “The Dig-
ital Supply Chain–emergence, Concepts, Definitions, and
Technologies.” In The Digital Supply Chain, edited by
Bart L MacCarthy and Dmitry Ivanov, 3–24. Amsterdam:
Elsevier.
Mahapatra, Bandana, and Srikanta Patnaik. 2018. “Ant Colony
Optimization.” In Advances in Swarm Intelligence for Opti-
mizing Problems in Computer Science, 79–114. 1st ed. Boca
Raton, FL: CRC Press/Taylor & Francis Group, [2019]: Chap-
man and Hall/CRC.
Makridis, G., P. Mavrepis, D. Kyriazis, I. Polychronou, and
S. Kaloudis. 2020. “Enhanced Food Safety Through Deep
Learning for Food Recalls Prediction.” Lecture Notes in Com-
puter Science (Including Subseries Lecture Notes in Artificial
Intelligence and Lecture Notes in Bioinformatics) 12323 LNAI,
566–580, Cham: Springer.
Marandi, F., and S. M. T. Fatemi Ghomi. 2019. “Network Con-
figuration Multi-Factory Scheduling with Batch Delivery:
earning-oriented Simulated Annealing Approach.” Comput-
ers and Industrial Engineering 132: 293–310.
Mehta, D., and D. Yamparala. 2014. “Policy Gradient Reinforce-
ment Learning for Solving Supply-chain Management Prob-
lems.” In I-CARE 2014: Proceedings of the 6th IBM Collab-
orative Academia Research Exchange Conference (I-CARE),
Bangalore, India, 1-4.
Meisheri, H., N. N. Sultana, M. Baranwal, V. Baniwal, S. Nath,
S. Verma, B. Ravindran, and H. Khadilkar. 2021. “Scal-
able Multi-product Inventory Control with Lead Time Con-
straints Using Reinforcement Learning.” Neural Computing
and Applications 34: 1735–1757.
Mezouar, H., and A. El Afia. 2019. “A 4-level Reference for
Self-adaptive Processes Based on SCOR and Integrating
Q-Learning.” In BDIoT’19: Proceedings of the 4th Interna-
tional Conference on Big Data and Internet of Things, Rabat,
Morocco, 1–5.
Michie, Donald, and R. A. Chambers. 1968. “BOXES: An
Experiment in Adaptive Control.” In Machine Intelligence,
edited by E. Dale, and D. Michie. Edinburgh, UK: Oliver and
Boyd.

Minsky, Marvin. 1961. “Steps Toward Artificial Intelligence.”
Proceedings of the IRE 49 (1): 8–30.
Mnih, Volodymyr, Adrià Puigdomènech Badia, Mehdi Mirza,
Alex Graves, Tim Harley, Timothy P. Lillicrap, David Sil-
ver, and Koray Kavukcuoglu. 2016. “Asynchronous Methods
for Deep Reinforcement Learning.” In Proceedings of the
33rd International Conference on International Conference
on Machine Learning – Volume 48, ICML’16, 1928–1937.
JMLR.org.
Mnih, Volodymyr, Koray Kavukcuoglu, David Silver, Alex
Graves, Ioannis Antonoglou, Daan Wierstra, and Martin
Riedmiller. 2013. “Playing Atari with Deep Reinforcement
Learning.”.
Mönch, Lars, Reha Uzsoy, and John W. Fowler. 2018. “A Survey
of Semiconductor Supply Chain Models Part I: Semicon-
ductor Supply Chains, Strategic Network Design, and Sup-
ply Chain Simulation.” International Journal of Production
Research 56 (13): 4524–4545.
Mongeon, Philippe, and Adèle Paul-Hus. 2015. “The Journal
Coverage of Web of Science and Scopus: A Comparative
Analysis.” Scientometrics 106: 213–228.
Moriarty, David E, Alan C Schultz, and John J Grefenstette.
1999. “Evolutionary Algorithms for Reinforcement Learn-
ing.” Journal of Artificial Intelligence Research 11: 241–276.
Mortazavi, A., A. Arshadi Khamseh, and P. Azimi. 2015.
“Designing of An Intelligent Self-adaptive Model for Supply
Chain Ordering Management System.” Engineering Applica-
tions of Artificial Intelligence 37: 207–220.
Nanduri, V., and I. Saavedra-Antolínez. 2013. “A Competi-
tive Markov Decision Process Model for the Energy-water-
climate Change Nexus.” Applied Energy 111: 186–198.
Peng, Z., Y. Zhang, Y. Feng, T. Zhang, Z. Wu, and H. Su. 2019.
“Deep Reinforcement Learning Approach for Capacitated
Supply Chain Optimization Under Demand Uncertainty.” In
Proceedings – 2019 Chinese Automation Congress, CAC 2019,
Hangzhou, China, 3512–3517.
Perez, H. D., C. D. Hubbs, C. Li, and I. E. Grossmann. 2021.
“Algorithmic Approaches to Inventory Management Opti-
mization.” Processes 9 (1): 1–17.
Pontrandolfo, P., A. Gosavi, O. G. Okogbaa, and T. K. Das.
2002. “Global Supply Chain Management: A Reinforcement
Learning Approach.” International Journal of Production
Research 40 (6): 1299–1317.
Puskás, E., Á. Budai, and G. Bohács. 2020. “Optimization of a
Physical Internet Based Supply Chain Using Reinforcement
Learning.” European Transport Research Review 12 (1): 47.
Rao, J. J., K. K. Ravulapati, and T. K. Das. 2003. “A Simulation-
Based Approach to Study Stochastic Inventory-planning
Games.” International Journal of Systems Science 34 (12-13):
717–730.
Ravulapati, K. K., J. Rao, and T. K. Das. 2004. “A Reinforce-
ment Learning Approach to Stochastic Business Games.”
IIE Transactions (Institute of Industrial Engineers) 36 (4):
373–385.
Reeder, J., G. Sukthankar, M. Georgiopoulos, and G. Anag-
nostopoulos. 2008. “Intelligent Trading Agents for Massively
Multi-player Game Economies.” In Proceedings of the 4th
Artificial Intelligence and Interactive Digital Entertainment
Conference, AIIDE 2008, Stanford, California, 102–107.
Reindorp, M. J., and M. C. Fu. 2011. “Dynamic Lead Time
Promising.” In IEEE SSCI 2011: Symposium Series on Com-
putational Intelligence – ADPRL 2011: 2011 IEEE Symposium
INTERNATIONAL JOURNAL OF PRODUCTION RESEARCH 7177
on Adaptive Dynamic Programming and Reinforcement
Learning, Paris, France, 176–183.
Ren, C., Y. Chai, and Y. Liu. 2002. “Agile Supply Chain Sim-
ulation Using Adaptive Multi-agent Modeling.” In Proceed-
ings of Asian Simulation Conference; System Simulation and
Scientific Computing, Shanghai, China, 752–756. Vol. 2.
Ruel, Salomée, Jamal El Baz, Dmitry Ivanov, and Ajay Das.
2021. “Supply Chain Viability: Conceptualization, Measure-
ment, and Nomological Validation.” Annals of Operations
Research1–30.
Rummery, Gavin Adrian, and Mahesan Niranjan. 1994. “On-
line Q-learning Using Connectionist Systems.”.
Saitoh, F., and A. Utani. 2013. “Coordinated Rule Acquisition of
Decision Making on Supply Chain by Exploitation-oriented
Reinforcement Learning.” Lecture Notes in Computer Sci-
ence (including Subseries Lecture Notes in Artificial Intel-
ligence and Lecture Notes in Bioinformatics) 8131 LNCS,
537–544. Berlin, Heidelberg: Springer.
Sakurai, Yoshitaka, Kouhei Takada, Takashi Kawabe, and Set-
suo Tsuruta. 2010. “A Method to Control Parameters of
Evolutionary Algorithms by Using Reinforcement Learn-
ing.” In 2010 Sixth International Conference on Signal-image
Technology and Internet Based Systems, 74–79. IEEE.
Schroeder, Meike, Birgit von See, Johannes Schnelle, and Wolf-
gang Kersten. 2021. “Impact of the Covid-19 Pandemic on
Supply Chain Management.” In Logistik in Wissenschaft und
Praxis, edited by Roy Fritzsche, Stefan Winter, and Jacob
Lohmer, 3–24. Wiesbaden: Springer Gabler.
Schulman, John, Filip Wolski, Prafulla Dhariwal, Alec Rad-
ford, and Oleg Klimov. 2017. “Proximal Policy Optimization
Algorithms.”.
Serrano, J. C., J. Mula, and R. Poler. 2021. “Digital Twin for
Supply Chain Master Planning in Zero-Defect Manufac-
turing.” IFIP Advances in Information and Communication
Technology 626: 102–111.
Sheffi, Yosef. 2020. The New (ab)normal: Reshaping Business
and Supply Chain Strategy Beyond Covid-19. Cambridge
(Mass.): MIT CTL Media. OCLC: 1258319911.
Sheffi, Yossi. 2021. A Shot in the Arm: How Science, Engineer-
ing, and Supply Chains Converged to Vaccinate the World.
Cambridge: MIT CTL Media.
Sheremetov, L., and L. Rocha-Mier. 2004. “Collective Intelli-
gence As a Framework for Supply Chain Management.” In
2004 2nd International IEEE Conference ’Intelligent Systems’
– Proceedings, Varna, Bulgaria, 417–422. Vol. 2.
Sheremetov, L., and L. Rocha-Mier. 2008. “Supply Chain Net-
work Optimization Based on Collective Intelligence and
Agent Technologies.” Human Systems Management 27 (1):
31–47.
Sheremetov, L., L. Rocha-Mier, and I. Batyrshin. 2005.
“Towards a Multi-agent Dynamic Supply Chain Simulator
for Analysis and Decision Support.” In Annual Conference
of the North American Fuzzy Information Processing Society
– NAFIPS, Detroit, Michigan, USA, 263–268.
Sianaki, O. A., A. Yousefi, A. R. Tabesh, and M. Mahdavi.
2019. “Machine Learning Applications: The Past and Cur-
rent Research Trend in Diverse Industries.” Inventions 4 (1):
8.
Silver, David, Thomas Hubert, Julian Schrittwieser, Ioannis
Antonoglou, Matthew Lai, Arthur Guez, and Marc Lanctot,
et al. 2017. “Mastering Chess and Shogi by Self-Play with a
General Reinforcement Learning Algorithm.”.
7178 B. ROLF ET AL.
Simsek, B., S. Albayrak, and A. Korth. 2004. “Reinforcement
Learning for Procurement Agents of the Factory of the
Future.” In Proceedings of the 2004 Congress on Evolutionary
Computation, CEC2004, Portland, Oregon, USA, 1331–1337.
Vol. 2.
Singi, S., S. Gopal, S. Auti, and R. Chaurasia. 2020. “Rein-
forcement Learning for Inventory Management.” In Proceed-
ings of International Conference on Intelligent Manufacturing
and Automation, Lecture Notes in Mechanical Engineering,
317–326. Singapore: Springer.
Snyder, Hannah. 2019. “Literature Review As a Research
Methodology: An Overview and Guidelines.” Journal of
Business Research 104: 333–339.
Sterman, John D. 1992. “Teaching Takes Off.” OR/MS Today 35
(3): 40–44.
Sui, Z., A. Gosavi, and L. Lin. 2010. “A Reinforcement Learning
Approach for Inventory Replenishment in Vendor-managed
Inventory Systems with Consignment Inventory.” EMJ –
Engineering Management Journal 22 (4): 44–53.
Sun, R., and G. Zhao. 2012. “Analyses About Efficiency of
Reinforcement Learning to Supply Chain Ordering Man-
agement.” In IEEE International Conference on Industrial
Informatics (INDIN), Beijing, China, 124–127.
Sun, R., G. Zhao, C. Li, and S. Tatsumi. 2006. “The Improve-
ment on Reinforcement Learning for SCM by the Agent Pol-
icy Mapping.” In IECON Proceedings (Industrial Electronics
Conference), Paris, France, 3585–3590.
Sun, R., G. Zhao, and C. Yin. 2010. “A Multi-agent Coordina-
tion of a Supply Chain Ordering Management with Multiple
Members Using Reinforcement Learning.” In IEEE Interna-
tional Conference on Industrial Informatics (INDIN), Osaka,
Japan, 612–616.
Sutton, Richard S., and Andrew G. Barto. 2018. Reinforcement
Learning: An Introduction. 2nd ed. Adaptive Computation
and Machine Learning Series. Cambridge, MA: The MIT
Press.
Sutton, Richard S., David McAllester, Satinder Singh, and
Yishay Mansour. 1999. “Policy Gradient Methods for Rein-
forcement Learning with Function Approximation.” In Pro-
ceedings of the 12th International Conference on Neural Infor-
mation Processing Systems, NIPS’99, 1057–1063. Cambridge,
MA, USA: MIT Press.
Tae, I. K., R. U. Bilsel, and S. R. T. Kumara. 2007. “A Reinforce-
ment Learning Approach for Dynamic Supplier Selection.”
In 2007 IEEE International Conference on Service Operations
and Logistics, and Informatics, SOLI, Philadelphia, Pennsyl-
vania, USA, 1-6.
Tang, K., and S. R. T. Kumara. 2005. “Cooperation in a
Multi-Stage Game for Modeling Distributed Task Delegation
in a Supply Chain Procurement Problem.” In Proceedings
of the 2005 IEEE Conference on Automation Science and
Engineering, IEEE-CASE 2005, Edmonton, Alberta, Canada,
93–98.
Tariq Afridi, M., S. Nieto-Isaza, H. Ehm, T. Ponsignon, and A.
Hamed. 2020. “A Deep Reinforcement Learning Approach
for Optimal Replenishment Policy in A Vendor Man-
aged Inventory Setting for Semiconductors.” In Proceed-
ings – Winter Simulation Conference, Orlando, Florida, USA,
1753–1764.
Thorndike, Edward L. 1898. “Animal Intelligence: An Experi-
mental Study of the Associative Processes in Animals.” The
Psychological Review: Monograph Supplements 2 (4): i–109.
UNO. 2008. “International Standard Industrial Classification of
All Economic Activities (ISIC), Rev.4.”.
Valluri, A., M. J. North, and C. M. MacAl. 2009. “Reinforcement
Learning in Supply Chains.” International Journal of Neural
Systems 19 (5): 331–344.
van Eck, Nees Jan, and Ludo Waltman. 2009. “Software Sur-
vey: VOSviewer, a Computer Program for Bibliometric Map-
ping.” Scientometrics 84 (2): 523–538.
Van Tongeren, T., U. Kaymak, D. Naso, and E. Van Asperen.
2007. “Q-learning in a Competitive Supply Chain.” In Con-
ference Proceedings – IEEE International Conference on Sys-
tems, Man and Cybernetics, Montreal, Quebec, Canada,
1211–1216.
Vanvuchelen, N., J. Gijsbrechts, and R. Boute. 2020. “Use of
Proximal Policy Optimization for the Joint Replenishment
Problem.” Computers in Industry 119:103239.
Vinitsky, Eugene, Nathan Lichtle, Kanaad Parvate, and Alexan-
dre Bayen. 2020. “Optimizing Mixed Autonomy Traffic Flow
with Decentralized Autonomous Vehicles and Multi-agent
RL.”.
Wang, Xun, and Stephen M. Disney. 2016. “The Bullwhip
Effect: Progress, Trends and Directions.” European Journal
of Operational Research 250 (3): 691–701.
Wang, F., and L. Lin. 2021. “Spare Parts Supply Chain Net-
work Modeling Based on a Novel Scale-Free Network and
Replenishment Path Optimization with Q Learning.” Com-
puters and Industrial Engineering 157: 107312.
Wang, Hao, Jiaqi Tao, Tao Peng, Alexandra Brintrup, Edward
Elson Kosasih, Yuqian Lu, Renzhong Tang, and Luoke
Hu. 2022. “Dynamic Inventory Replenishment Strategy for
Aerospace Manufacturing Supply Chain: Combining Rein-
forcement Learning and Multi-agent Simulation.” Interna-
tional Journal of Production Research 60 (13): 1–20.
Ware, Mark, and Michael Mabe. 2015. The STM Report: An
Overview of Scientific and Scholarly Journal Publishing. Tech-
nical Report. International Association of Scientific, Techni-
cal and Medical Publishers.
Watkins, Christopher J. C. H., and Peter Dayan. 1992. “Q-
learning.” Machine Learning 8 (3-4): 279–292.
Wong, Geoff, Trisha Greenhalgh, Gill Westhorp, Jeanette Buck-
ingham, and Ray Pawson. 2013. “RAMESES Publication
Standards: Meta-narrative Reviews.” BMC Medicine 11: 20.
Xiang, L. 2020. “Energy Emergency Supply Chain Collabora-
tion Optimization with Group Consensus Through Rein-
forcement Learning Considering Non-cooperative Beha
viours.” Energy 210: 118597.
Xu, J., J. Zhang, and Y. Liu. 2009. “An Adaptive Inventory
Control for a Supply Chain.” In 2009 Chinese Control and
Decision Conference, CCDC 2009, Guilin, China, 5714–5719.
Yang, S., Y. Ogawa, K. Ikeuchi, Y. Akiyama, and R. Shibasaki.
2019. “Firm-level Behavior Control After Large-Scale Urban
Flooding Using Multi-agent Deep Reinforcement Learning.”
In Proceedings of the 2nd ACM SIGSPATIAL International
Workshop on GeoSpatial Simulation, GeoSim 2019, Chicago,
Illinois, USA, 24–27.
Yang, S., and J. Zhang. 2015. “Adaptive Inventory Control
and Bullwhip Effect Analysis for Supply Chains with Non-
Stationary Demand.” In Proceedings of the 2015 27th Chi-
nese Control and Decision Conference, CCDC 2015, Qingdao,
China, 3903–3908.
Zarandi, M. H. F., S. V. Moosavi, and M. Zarinbal. 2013.
“A Fuzzy Reinforcement Learning Algorithm for Inventory

Control in Supply Chains.” International Journal of Advanced
Manufacturing Technology65 (1-4): 557–569.
Zhang, Y., and S. Bhattacharyya. 2007. “Effectiveness of Q-
learning As a Tool for Calibrating Agent-Based Supply
Network Models.” Enterprise Information Systems 1 (2):
217–233.
Zhang, Gongtao, Bart L MacCarthy, and Dmitry Ivanov. 2022.
“ The Cloud, Platforms, and Digital Twins–Enablers of the
Digital Supply Chain.” In The Digital Supply Chain, edited by
Bart L MacCarthy and Dmitry Ivanov, 77–99. Amsterdam:
Elsevier.
Zhang, Daniel, Saurabh Mishra, Erik Brynjolfsson, John
Etchemendy, Deep Ganguli, Barbara Grosz, and Terah
Lyons, et al. 2021. “The AI Index 2021 Annual Report.”.
Zhang, K., J. Xu, and J. Zhang. 2013. “A New Adaptive Inven-
tory Control Method for Supply Chains with Non-Stationary
Demand.” In 2013 25th Chinese Control and Decision Confer-
ence, CCDC 2013, Guiyang, China, 1034–1038.
Zhang, L., Y. Yin, L. Feng, and H. Fan. 2019. “Unsal-
able Risk Prediction of Fruit and Vegetable Agricultural
Products Based on Markov Decision Process.” In Pro-
ceedings of the 8th International Conference on Logistics
and Systems Engineering 2018, Changsha City, China, 206–
217.
Zhanguo, X. 2008. “Research on Refining the Distributed Sup-
ply Chain Procurement Plans Based on CRL.” In Proceedings
– International Symposium on Information Processing, ISIP
2008 and International Pacific Workshop on Web Mining
and Web-Based Application, WMWA 2008, Moscow, Russia,
119–122.
Zhao, Y., E. Hemberg, N. Derbinsky, G. Mata, and U.-M.
O’Reilly. 2021a. “Simulating a Logistics Enterprise Using An
Asymmetrical Wargame Simulation with Soar Reinforce-
ment Learning and Coevolutionary Algorithms.” In GECCO
2021 Companion – Proceedings of the 2021 Genetic and
INTERNATIONAL JOURNAL OF PRODUCTION RESEARCH 7179
Evolutionary Computation Conference Companion, Lille,
France, 1907–1915.
Zhao, H.-P., J.-D. Jiang, and Y.-C. Feng. 2010. “Coordinat-
ing Supply Chain of Stackelberg Game Model Based on
Evolutionary Game with GA-RL.” Xitong Gongcheng Lilun
yu Shijian/System Engineering Theory and Practice 30 (4):
667–672.
Zhao, Long, and Zemin Liu. 1996. “A Genetic Algorithm for
Reinforcement Learning.” In Proceedings of International
Conference on Neural Networks (ICNN’96), Washington, DC,
USA. 1056-1060.
Zhao, F., L. Zhang, J. Cao, and J. Tang. 2021b. “A Cooperative
Water Wave Optimization Algorithm with Reinforcement
Learning for the Distributed Assembly No-idle Flowshop
Scheduling Problem.” Computers and Industrial Engineering
153: 107082.
Zhou, J., M. Purvis, and Y. Muhammad. 2016. “A Combined
Modelling Approach for Multi-Agent Collaborative Plan-
ning in Global Supply Chains.” In Proceedings – 2015 8th
International Symposium on Computational Intelligence and
Design, ISCID 2015, Hangzhou, China, 592–597. Vol. 1.
Zhou, J., and X. Zhou. 2019. “Multi-Echelon Inventory Opti-
mizations for Divergent Networks by Combining Deep
Reinforcement Learning and Heuristics Improvement.” In
Proceedings – 2019 12th International Symposium on Com-
putational Intelligence and Design, ISCID 2019, Hangzhou,
China, 69–73. Vol. 1.
Zhu, Z., J. Ke, and H. Wang. 2021. “A Mean-Field Markov
Decision Process Model for Spatial-Temporal Subsidies in
Ride-Sourcing Markets.” Transportation Research Part B:
Methodological 150: 540–565.
Zwaida, T. A., C. Pham, and Y. Beauregard. 2021. “Optimiza-
tion of Inventory Management to Prevent Drug Shortages in
the Hospital Supply Chain.” Applied Sciences (Switzerland)
11 (6): 2726.