See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/360244313

Multi-agent reinforcement learning-based exploration of optimal operation

strategies of semi-batch reactors

Article in Computers & Chemical Engineering · April 2022

DOI: 10.1016/j.compchemeng.2022.107819

CITATION READS

1 138

3 authors:

Ádám Sass Alex Kummer

University of Pannonia, Veszprém University of Pannonia, Veszprém
3 PUBLICATIONS 4 CITATIONS 23 PUBLICATIONS 163 CITATIONS

SEE PROFILE SEE PROFILE

János Abonyi
University of Pannonia, Veszprém
450 PUBLICATIONS 6,526 CITATIONS

SEE PROFILE

All content following this page was uploaded by János Abonyi on 05 May 2022.

The user has requested enhancement of the downloaded file.

 Computers and Chemical Engineering 162 (2022) 107819

Contents lists available at ScienceDirect

Computers and Chemical Engineering

journal homepage: www.elsevier.com/locate/compchemeng

Multi-agent reinforcement learning-based exploration of optimal

operation strategies of semi-batch reactors
Ádám Sass a,b, Alex Kummer b,∗, János Abonyi b,c
a
KALL Ingredients Ltd., Tiszapüspöki, H-5211, Hungary
b
Institute of Chemical and Process Engineering, Department of Process Engineering, Hungary
c
MTA-PE Lendület Complex Systems Monitoring Research Group, University of Pannonia, Veszprém H-8200, Hungary

a r t i c l e i n f o a b s t r a c t

Article history: The operation of semi-batch reactors requires caution because the feeding reagents can accumulate, lead-
Received 13 November 2021 ing to hazardous situations due to the loss of control ability. This work aims to develop a method that
Revised 5 April 2022
explores the optimal operational strategy of semi-batch reactors. Since reinforcement learning (RL) is
Accepted 22 April 2022
an efficient tool to find optimal strategies, we tested the applicability of this concept. We developed a
Available online 28 April 2022
problem-specific RL-based solution for the optimal control of semi-batch reactors in different operation
Keywords: phases. The RL-controller varies the feeding rate in the feeding phase directly, while in the mixing phase,
Temperature control it works as a master in a cascade control structure. The RL-controllers were trained with different neu-
RL-controller ral network architectures to define the most suitable one. The developed RL-based controllers worked
Feeding trajectory very well and were able to keep the temperature at the desired setpoint in the investigated system. The
Cascade control results confirm the benefit of the proposed problem-specific RL-controller.
© 2022 The Authors. Published by Elsevier Ltd.
This is an open access article under the CC BY-NC-ND license
(http://creativecommons.org/licenses/by-nc-nd/4.0/)

1. Introduction oped a problem-specific RL-based solution for the optimal control

Semi-batch reactors (SBRs) are widely used in the industry to
handle highly exothermic reactions, but their operations require a
lot of attention because the reagents can accumulate, resulting in
hazardous situations (Ni et al., 2016; Bai et al., 2017). At the op-
eration of SBRs, there is a need to improve the efficiencies contin-
uously, so there is a need to define the best potential operation
strategy while the reactor always stays in the safe and control-
lable operation regime (Hall, 2018). Reinforcement learning (RL) is
an efficient tool for learning complex behaviour and strategy, and
RL agent has the potential to perform any task that requires ex-
perience gained by interacting with the process (Shin et al., 2019).
The operation of batch reactors is generally improved iteratively
based on the experience gained from the earlier operation strate-
gies. Since RL works similarly based on experiences, we think RL
can perform superbly in finding the optimal feeding strategy.
Moreover, RL is typically useful for episodic cases, so it confirms
the use of RL for the operation of batch reactors (Nian et al., 2020).
Our motivation is to explore the potential in RL-based operation of
semi-batch reactors carrying out exothermic reactions. We devel-
∗
corresponding author.
E-mail address: kummera@fmt.uni-pannon.hu (A. Kummer).
of SBRs in both the feeding and the mixing phase. Our idea is that
the operation phases can be distinguished from the aspect of the
control strategy, so two different RL-agents work independently in
the feeding and the mixing phase. The proposed RL-controllers ful-
filled our expectations, and the results confirm their benefits for
the operation of SBRs.
During the operation of SBRs, great attention must be paid to
prevent the development of thermal runaways. A tried-and-tested
method is to control the rate of reaction heat generation by the
feeding rate of the reagent. This way, the cooling system is able to
remove the reaction heat from the reactor. However, if the feed-
ing strategy is not designed correctly, the temperature may ex-
ceed the maximum allowable temperature (MAT) (Westerterp and
Molga, 20 04; 20 06; Copelli et al., 2014). One of the main risks dur-
ing the operation of the SBRs is the accumulation of the feeding
component due to poorly chosen initial conditions or feeding strat-
egy because the higher concentration may trigger a thermal run-
away resulting in a rapid and significant temperature increase (Guo
et al., 2017; Ni et al., 2017). Generally, the feeding strategy is sim-
ple, using only a constant feeding rate over the entire operation of
the reactor resulting in higher batch times compared to the oper-
ation cases with a more complex feeding strategy (Kummer et al.,
2020b).

https://doi.org/10.1016/j.compchemeng.2022.107819
0098-1354/© 2022 The Authors. Published by Elsevier Ltd. This is an open access article under the CC BY-NC-ND license (http://creativecommons.org/licenses/by-nc-nd/4.0/)
Á. Sass, A. Kummer and J. Abonyi Computers and Chemical Engineering 162 (2022) 107819

Kanavalau et al., 2019; Kummer et al., 2020a). However, there is

Notations the main disadvantage when we use NMPC to solve such a prob-
lem. Since the NMPC solves an optimization problem in every it-
a action eration step, the computational cost is high, and the computa-
c concentration of reagent, [ kmol
m3
] tional time is critical from the point of applicability. In contrast,
cp heat capacity, [ kgkJ K ] a reinforcement learning-based controller is trained offline, and it
e error does not need additional optimization in the online application.
EA activation energy, [ kmol kJ
] As a result of this, the computational time is not critical anymore
J objective function (Nian et al., 2020).
3 s The learning of the RL-based controller is similar to the con-
k reaction rate constant, [ m kmol
]
3 cept of batch-to-batch learning (Xiong and Zhang, 2005). Batch-
k0 pre-exponential factor, [ m s
kmol
] to-batch optimization uses the episodic nature of batch processes
KP proportional gain to define the optimal operating policy (Clarke-Pringle and MacGre-
KI integral gain gor, 1998), and the optimal operating conditions are defined iter-
L loss function atively in the presence of uncertainty using the previous batches
M number of episodes (Srinivasan et al., 2001).
ni mole amount of the ith component, [kmol] RL-based controllers are able to learn only from experience,
N noise and it does not require any parameter tuning; also, the RL-based
N mini batch controllers can handle non-linear systems (Ma et al., 2019). Only
NR number of reagents a few research articles can be found in the literature about RL-
OP actuator position based reactor operations. Machalek et al. optimized the operation
Q action-value function of a continuous stirred tank reactor (CSTR) by defining the opti-
PV process variable of controller mal value of the transferred heat using three different RL algo-
r reward rithms (Deep Deterministic Policy Gradient (DDPG), Twin Delayed
kmol
rR reaction rate, [ m 3 s] Deep Deterministic Policy Gradient, Proximal Policy Optimization)
R replay buffer (Machalek et al., 2020). Singh and Kodamana used Q-learning
kJ
Rg gas constant, [ kmol K
] and Deep Q-learning to control a batch polymerization process
s state of the system (Singh and Kodamana, 2020). Pan et al. proposed an oracle-assisted
SP setpoint of controller constrained Q-learning to handle the different system constraints
t time, [s] (for example, maximum process temperature, maximum concen-
T time interval, [s] tration, etc.) during the operation. Pan et al. (2021). An RL-
TJ temperature of the reactor jacket, [K] based real-time optimization framework for steady-state optimiza-
TR temperature of the reactor, [K] tion has been worked out and tested on a CSTR (Powell et al.,
TS P temperature setpoint, [K] 2020). A Monte–Carlo-based DDPG algorithm was presented in
y target Yoo et al. (2021) for the optimal control of a batch polymeriza-
u control input vector in NMPC formulation, [%] tion process. An RL-controller (Q-learning algorithm) has been ap-
UA heat transfer parameter, [ kW
K
] plied for pH process control in a laboratory-sized plant, and the
V liquid volume, [m3 ] RL-based controller provided a much smoother operation than the
x state vector in NMPC formulation conventional PID controller (Syafiie et al., 2007).
γ discount factor The motivation of our work is to investigate the potentials of
Hr kJ
reaction heat, [ kmol ] RL-based operation of SBRs carrying out exothermic reactions. Our
θ weights in the neural network RL-based operation strategy uses an RL agent both for the feeding
μ policy operator and mixing phase. The RL controller in the feeding phase is a sim-
π general policy ple single-loop controller, while it acts as a master in a cascade-
ρ kg
density, [ m 3] loop during the mixing phase. During the feeding phase, the tem-
τ trajectory perature was controlled by varying the feeding rate, and in the
τsys time constant of the system, [min] mixing phase, the temperature was controlled by manipulating the
coolant rate. A model-free RL algorithm, namely Deep Determin-
istic Policy Gradient (DDPG) method, is applied for the training,
In recent years, researchers focused on applying model-based where we investigated different neural network architectures. We
solutions to define a non-constant feeding strategy because an ad- also implemented an NMPC to control the SBR in different opera-
equate model can be used to design an appropriate feeding plan tion phases to demonstrate in which situations the RL-based con-
to maximize productivity. A model-based control approach, such troller can outperform the NMPC. The designed controllers could
as model predictive controllers (MPC), can be used to optimize the keep the reactor in a safe operation zone, and the reactor tem-
feeding strategy (Findeisen et al., 2007). Since SBRs carrying out perature did not deviate from the target value. A continuously im-
exothermic reactions have a high non-linearity in the dynamics, proving area is the application of Multi-Agent reinforcement learn-
the controller has to cope with it. Non-linear MPC (NMPC) is a ing techniques to earn a better performance in control problems
suitable tool to handle such non-linear processes (Seki et al., 2001). (Zhang et al., 2018; Chen et al., 2021). Our control strategy uses
Until today, numerous MPC-based frameworks have been de- a multi-agent setup to operate semi-batch reactors, though in this
veloped to control semi-batch reactors safely and optimally. Rossi case, the agents are decoupled from each other. Our contributions
et al. developed the simultaneous model-based optimization and to the field are as follows:
control (BSMBO&C) algorithm, which is the combination of an
NMPC and dynamic real-time optimization (DRTO) techniques • We suggested operation phase-based control structures for the
(Rossi et al., 2015; 2016; 2017). MPC-based solutions with inte- operation of SBRs using multi-agents (see Section 2)
grated runaway criteria to predict and avoid thermal runaway were • We developed problem-specific RL-controllers used in single-
also investigated carefully (Kähm and Vassiliadis, 2018a; 2018b; and cascade loops (see Section 2)

2
Á. Sass, A. Kummer and J. Abonyi
Fig. 1. The process scheme of the investigated semi-batch reactor.
• We developed a problem-specific NMPC (see Section 2)
• We demonstrate that the proposed method is successful in ex-
ploring the optimal operating strategies (see Section 3)
• We discuss the applicability and the benefits of the proposed
method in avoiding dangerous operating zones that can lead
to thermal runaway and compare its behaviour to NMPC (see
Section 3).
2. Operation phase-based control of SBRs
The operation of SBRs consists of more phases, like the feeding
and mixing phase. In the feeding phase, a reagent is fed into the
reactor, while in the mixing phase, the only goal is to maximize
the conversion at the desired temperature without any inlet feed.
The operation phases can be well separated, so different control
structures can be used to maximize their performances. Usually, in
the feeding phase, the feed rate is designed as a constant rate us-
ing a single control-loop with PID or the feed rate is varied using a
model predictive controller to predict and avoid reagent accumu-
lation. Since the reagent accumulation cannot occur in the mixing
phase, a cascade control-loop is usually used with PIDs to keep the
reactor temperature at the setpoint.
2.1. Definition of the control problem
The control goal in both operation phases is to keep the tem-
perature at the desired setpoint (SP ). In the feeding phase, the
temperature is controlled by the feed rate of a reagent by manipu-
lating the reagent feed valve (V 1), and in the mixing phase, the
temperature is controlled by manipulating the coolant flow rate
(V 2) in a cascade structure. Since the operation phases from the
control point are independent, we suggest using two different con-
trollers. Due to the potential accumulation of the feeding reagents,
high non-linearities are present in the system, which a non-linear
controller can handle. Based on these control structures and han-
dling of non-linearities, we propose an RL-based control solution to
define the optimal operation strategy for SBRs. Fig. 1 presents the
process scheme of the investigated system. An RL-controller means
an RL-agent so that a neural network can choose the right con-
trol values (actions) for the operation. The RL-controllers in both
phases use state representation (s f eeding and smixing ) to make a deci-
sion about the action values (a f eeding and amixing ). The agents inter-
act with the system through actions based on the observed states,
and the agents learn based on the received reward values.
3
Computers and Chemical Engineering 162 (2022) 107819
2.2. The applied control schemes for the operation
The applied control structures are presented in Fig. 2. In the
feeding phase, an RL-based controller leads the temperature to the
desired setpoint. The RL-based controller structure is presented in
Fig. 2a. The only feedback to the RL controller is the required state
tuple (s f eeding ) to define the appropriate actions, which is the de-
sired change of the feed valve actuator position (V 1)).
The RL-based controller was also tested in the mixing phase.
This setup uses a cascade controller with a PID controller in the
slave loop and an RL-controller in the master loop. The PID con-
troller perfectly handled the disturbances, while the RL-controller
is used to define the setpoint of the PID controller based on the
changes in the temperature of the reactor. In the mixing phase,
the reactor temperature is kept at setpoint with a cascade con-
troller, where we compared the performance of the conventional
PID controllers (see Fig. 2b) with the new proposed RL-based con-
troller (see Fig. 2c). In this case, the RL controller acts as the mas-
ter controller in the cascade algorithm, and it reacts based on the
observed states (smixing ). This cascade structure also showed that
RL-based controllers could be used in addition to being a local
level controller, as an advanced level controller too.
2.3. Setup of the reinforcement learning algorithm
We introduce in this section the applied RL algorithm (Deep
Deterministic Policy Gradient (DDPG) method). Then we present
how the RL-based training of the controllers occurred for the feed-
ing and mixing phases. We present which states are observed for
the control problem and which action is taken to control the re-
actor temperature. We also show the architectures of the neural
networks and the training algorithm.
DDPG method is an actor-critic method that was developed
based on two earlier RL methods, based on the Deep Q-network
and Deterministic Policy Gradient (DPG) (see Appendix B). Lilli-
crap et al. extended the DPG method with a replay buffer to store
tuples of historical samples. In DDPG the agent learns in mini-
batches rather than learning online. DDPG includes target networks
too, which are the copy of the actor and critic networks, and they
are used to calculate the target values (Lillicrap et al., 2015). The
DDPG method incorporates the advantages of DQN into an actor-
critic structure. The actor-critic structure with a deterministic pol-
icy gradient-based update on the actor is the first and most simple
method for continuous action space. Using DPG with actor-critic
structure in the SBR would be a relevant solution if the reactor
characteristic would not insinuate non-linearity. The main advan-
Á. Sass, A. Kummer and J. Abonyi
Fig. 2. The applied control schemes for the operation of SBRs.
Fig. 3. The applied RL algorithm.
tage of the DQN algorithm is that it uses deep neural networks
which approximate the action-value function; thus, it can exploit
the fact that neural networks can be well used for non-linear func-
tion approximation. However, using neural networks in actor-critic
structures and directly implementing Q-learning (critic) proved to
be unstable in many environments because the critic network be-
ing updated is also used in calculating the target value, the Q
update is prone to divergence (Lillicrap et al., 2015). These fac-
tors shaped our choice to go with the DDPG algorithm to solve
our continuous non-linear optimal control problem, which was the
first to solve continuous high-dimensional control tasks effectively
(Nian et al., 2020). The selected DDPG algorithm has already been
applied for the continuous control of a non-linear batch processes
(Ma et al., 2019), (Yoo et al., 2021), (Xu et al., 2020), (Yoo et al.,
2019).
The details of the utilised DDPG algorithm can be found in
Lillicrap et al. (2015). The overview of the algorithm is depicted
in Fig. 3, while Algorithm 1 presents the step by step description
of the training algorithm in the case of reactor control. In the re-
4
Computers and Chemical Engineering 162 (2022) 107819
maining part of this section, we present how this algorithm should
be interpreted.
We use target networks to calculate the target values for both
actor and critic networks, and we use soft updates for the target
networks. The weights of the target networks are updated with the
following equation:
θ ←
− τ θ + ( 1 − τ )θ  (1)
where θ  is the weight parameters of the target networks. The
exploration-exploitation dilemma is solved by adding noise (N ) to
the action at the actor network:
μ (st ) = μ(st |θtμ ) + N (2)
N can be Ornstein–Uhlenbeck process or a normal noise to ex-
plore the environment with or without noise decay.
Several hyperparameters need to be defined to use reinforce-
ment learning, such as the structure of the used deep neural
networks, the learning rate of the networks, the discount factor
(γ ), and the action noise (σ ), buffer size and minibatch size. The
Á. Sass, A. Kummer and J. Abonyi
Algorithm 1 The utilized training algorithm.
Randomly initialize critic (Q (s, a|θ Q ) and actor network (μ(s|θ μ )
with weights θ Q and θ μ
Initialize target network Q  and μ with weights θ Q  ← − θ Q and
 predicts the action with a random normal action noise (σ = 0.15)
θμ ← − θμ
Initialize replay buffer R
for episode = 1, M do
Initialize reactor model with a uniformly random reactor
temperature (TR ) and moles of the reagents (ni,0 )
Receive initial observation state s1
while t < T do
Select action at = μ(st |θ μ ) + Nt
Execute action at and observe reward rt and observe new
state
st+tsampling
Store transition(st , at , rt , st+tsampling ) in R
Sample a random minibatch of N transitions (si , ai , ri , si+1 )
from R
 
Set yi = ri + γ Q  (si+1 , μ (si+1 |θ μ )|θ Q )
Update critic by minimizing the loss:
   2
L = N1 i yi − Q si , ai |θ Q
Update the actor policy  using  the sampled policy gradient:

∇θ μ J ≈ N1 i ∇μ Q s, a|θ Q |s=si ,a=μ(st |θ μ )
Update the target networks:

θQ ← − τ θ Q + ( 1 − τ )θ Q

 
μ
θ ← μ
− τ θ + ( 1 − τ )θ μ rt =
end while
end for
most popular hyperparameter setting methods are the manual
search, the grid search, the random search, and the more so-
phisticated Bayesian optimization. Hyperparameter optimization
techniques suffer from the requirement of many learning runs to
identify good hyperparameters, so these methods highly increase
the computational costs. However, these methods can effectively
improve the performance (Paul et al., 2019). Since this research
focuses on developing the control structure, a draft sensitiv-
ity analysis of the hyperparameters was elaborated, and the
parameters were not fine-tuned by optimization.
Every state which affects the process temperature and varies in
time should be included in the state representation. In the feed-
ing phase, the observed states are presented in Eq. (3), which are
the reactor temperature (TR ), the jacket temperature (TJ ), the actua-
tor position (OP ), the concentration of the reagents, the heat trans-
fer rate (UA), and the error at the present time and at the earlier
sampling time between the reactor temperature and the setpoint
(et , et−s ). The errors must be included in the state space because,
as we experienced, the agents learn much easier and faster in this
way. It is worth noting that the knowledge earned by basic algo-
rithms (like PID) can help define the structure of RL-based con-
trollers, so, for example, it can help define an appropriate set of
states.
There is only one actuator, which is the feeding valve (V 1),
hence there is only one action: the valve’s adjustment value (OP ).
The action is limited between −10% and +10%.
s f eeding = {TR , TJ , OP, ci , . . . , cNR , UA, et , et−s } (3)
In the mixing phase, the observed states are presented in
Eq. (4), which are almost the same as s f eeding , but the setpoint
of the slave loop (SPslave ) is included. In the mixing phase, the RL
works as the master controller in the cascade loop; hence the ac-
tion is the adjustment of the setpoint of the slave loop. The action
is limited between −10 K and +10 K.
smixing = {TR , TJ , OP, SPslave , ci , . . . , cNR , UA, et , et−s } (4)
5
Computers and Chemical Engineering 162 (2022) 107819
The time interval of each action is 100 s both in the feeding and
mixing phase. This parameter must be set according to the small-
est time constant of the non-linear system to ensure that the effect
of the action can change the state of the system. The actor-network
for exploration without noise decay. The action noise provides bal-
ances the exploration and exploitation of the algorithm. If the
value is too low, then exploration is not dominant enough, so af-
ter many episodes, many actions with potentially high rewards
are still not found; therefore, the solution will be sub-optimal. If
the value is too high, then stability problems can appear. The ac-
tion value has been determined based on Lillicrap et al. (2015);
Yoo et al. (2021), and the results discussed in Section 3 show, that
the selected parameter results in good performance.
The reward function is the same for the feeding and mix-
ing phase. At the sampling times, the agent receives a reward
(r) from the environment. The reward function is formed from
three-part, as presented in Eq. (5). If the reactor temperature
is higher than MAT, then the agent receives a constant penalty.
If the reactor temperature is within an interval of ε , the agent
receives a positive reward. If the reactor temperature is lower
than the desired setpoint, the agent gets a penalty based on
the absolute difference between the setpoint and the reactor
temperature.
⎧
⎨−100 if TR,t > MAT
+100 if |TR,t − SP| ≤ ε (5)
⎩
−|TR,t − SP | otherwise
The structure of the applied neural networks is the same for
both the actor and the critic networks in all phases. A feedforward
neural network is used with fully connected layers, and we used
layer normalization on all layers of the critic and actor.
We used Adam optimizer to train the neural networks with a
learning rate of 10−4 and 10−3 for the actor and critic networks.
The target networks were updated softly with τ = 10−3 . The acti-
vation layers in the hidden layers used the Relu function. The out-
put layer of the actor used tanh function to bound the actions. The
output of the critic network did not use any activation function.
Action was included only in the last hidden layer of the critic net-
work (added to the previous layer’s outputs).
2.4. Setup of the NMPC
The performance of the RL-based controller is compared to
NMPC, which can be considered an advanced technique for opti-
mal control.
The NMPC is designed to keep the temperature of the reactor
(TR ) at the desired setpoint and at all times under MAT. The ob-
jective function considers that the process temperature follows the
setpoint, so that the error can be defined as
ek = TSP − TR,k (6)
where ek is the error between the setpoint and the temperature
of the reactor (TRk ) at kth time step. The objective function of the
open-loop optimization problem is the sum of the squared error
over the prediction horizon described in Fig. 7.
t pred

min ek 2 (7)
u j
k=0
subject to
u j+1 = u j + u j (8)
−10% ≤ u j ≤ 10% (9)
Á. Sass, A. Kummer and J. Abonyi Computers and Chemical Engineering 162 (2022) 107819

0% ≤ u j ≤ 100% (10) Table 1

Maximum average rewards at the different neural network structures.

Run no. Hidden layers Neurons Maximum average reward

TR ≤ MAT (11)
1 1 64 −540.82
where u j is the manipulated variable in jth control interval. The 2 1 128 585.57
manipulated variables are the valve opening of V1 or V2, depend- 3 1 256 1258.55
4 1 512 3591.64
ing on the operation phase of the SBR. The NMPC is formulated as 5 2 64 × 64 4250.85
velocity-based optimization thus the optimal change of the manip- 6 2 128 × 128 4492.19
ulated variables are to be found in every time step. That is limited 7 2 256 × 256 4434.51
to the same range as used in the case of the RL-based controller. 8 2 512 × 512 4626.68
The parameters of the NMPC have been tuned similarly to the
RL-based controller. The action interval, which is the time between Table 2
two control actions, was set to 100 s, which is the same value as in Dynamics of the system and PID controller parameters .
the case of the RL-controller. It is essential to define the prediction
Slave loop Master loop
horizon to be long enough to capture thermal runaway (Kähm and
Gain −0.27 1
Vassiliadis, 2018a), so we set 2500 s as the length of the prediction
τsys [min] 3.1 20.5
horizon and 10 0 0 s as the length of the control horizon, which will KP 14.8 5
add up to 10 different values of u j in one optimization step. KI 12.4 200

3. Results and discussion

As the RL agents learn from a wealth of experience that can-
not be earned on the real system, there is a need for an adequate
system model. Fortunately, there is a broad, well-established tool-
box for modelling batch systems. Although the modelling and the
validating tasks are time- and resource-intensive, they can be done
in an industrial environment, so this does not limit its applicabil-
ity. In the following, we present the applied process model and the
reactor operation results.
3.1. Process model of the reactor system
A second-order reaction is carried out in a semi-batch reac-
tor, where C product is made from reagents A and B in the A +
kmol
B → C reaction. The reaction rate (rR m 3 s ) is expressed with a
Arrhenius-type equation. The details of the process model, and the
dynamic, kinetic and thermodynamic parameters are described in
Appendix A.
3.2. Results and discussion
3.2.1. Training and testing of the RL controller in the feeding phase
The training was performed for eight different network struc-
tures to define the proper structure. The RL Agents for the con-
trollers are trained with the DDPG algorithm. A replay buffer is
used to store the historical samples, from which the samples are
chosen uniformly, and the size of the buffer is 1,0 0 0,0 0 0. The size
of the mini-batches is 64. The structure of the neural networks was
the same for both the actor and critic. One and two hidden layers
were investigated based on the 2n -principle, so with 64, 128, 256
and 512 neurons in each hidden layer.
The average rewards can be seen in Fig. 4, where the average
rewards were calculated within every 50 episodes. As it can be
seen, the reward trends saturate in the same interval, so investigat-
ing more complex network structures will not provide better solu-
tions. The actor and critic models were saved at the maximal aver-
age reward values, which can be seen in Table 1. With one hidden
layer, the controller could not learn to keep the temperature, but
with two hidden layers, the results are quite acceptable, and the
maximum rewards were obtained using 512 × 512 neurons. Figs. 5
and 6 presents how the RL controller operates using all the neural
networks with two hidden layers. In the first figure (top-left), the
temperature trajectories and the setpoint is shown. The setpoint
interval is shown with blue lines, and we can see how the differ-
ent RL controllers drive the reactor temperatures into the setpoint
interval. After ∼ 2 h, the heat generated in the reaction is no longer
enough to keep the temperature within the setpoint interval, hence
we need to change the actuator there to maintain temperature. In
the second figure (top-right), the position of the actuator (V1) is
presented. In the beginning, the RL controller opens the actuator
at maximum speed so that the reagents can be consumed in the
reaction and generate heat. As the reactor temperature gets closer
to the setpoint, the feed rate decreases to compensate or avoid
overshooting. Then the feed rate is slowly increased to keep the
temperature in the setpoint interval. In the third figure (bottom,
right), the action made by the RL controllers is presented. Although
the maximum average reward was earned with two hidden layers
and 512 × 512 neurons, the test runs show that the trained net-
work with 128 × 128 neurons performs better, for instance, there
is no overshooting at the end of the feeding phase. This contra-
diction may come from the random initial operating parameters
and the width of the averaging window. Based on the results, we
work further with the networks, including two hidden layers and
128 × 128 neurons.
3.2.2. Analysis of the cascade controller in the mixing phase
We performed the open-loop analysis of the slave and master
loop to get information about the dynamics of the system, and
we defined the gain and the time constant, which is presented in
Table 2. The time constant of the slave loop was 3.1 min, and
of the master loop, it was 20.5 min. The PID controller param-
eters (KP , KI ) were defined based on the dynamics of the loops,
which are presented in Table 2. The PID parameters were tuned
IMC-based and fine-tuned by hand.
Fig. 7 presents how the cascade controller with the PID master
works. In the top figure, the operation of the master loop is pre-
sented, and on the bottom figure, the operation of the slave loop
is shown. At the time of 2.3 h, the feeding of the reagent was
stopped so was its cooling effect. Therefore, between 2.3 h and
2.5 h, there was a need to decrease the jacket temperature to keep
the reactor temperature in the setpoint interval. As it can be seen
in Fig. 7 the reactor temperature (PVmaster ) goes to the setpoint
interval but reaches it quite slowly, and the reactor temperature
stays outside of the desired setpoint interval.
In the following, we present the control results of the cascade
with RL controller. The training was performed for four different
network structures, where the neural networks’ structure was the
same for both the actor and critic. We investigated the control re-
sults with one hidden layer and 64, 128, 256 and 512 neurons. The
average rewards are shown in Fig. 8. The reward trends saturate in
the same interval, so there is no need to investigate more complex

6
Á. Sass, A. Kummer and J. Abonyi Computers and Chemical Engineering 162 (2022) 107819

Fig. 4. Average rewards at different neural network structures.

Fig. 5. Performance of the RL controller (TR,0 = 345 K, nB,0 = 4.2 kmol).

Fig. 6. Performance of the RL controller (TR,0 = 364 K, nB,0 = 5.0 kmol).

Table 3 presented in Fig. 9, where all the four controllers are presented
Maximum average rewards at the different neural network structures .
with the different numbers of applied neurons. Using 64, 128 and
Run no. Hidden layers Neurons Maximum average reward 256 neurons, the temperature slightly exceeds the setpoint inter-
1 1 64 7119.24 vals, which is not the case using 512 neurons. Also, the smoothest
2 1 128 7052.75 action trajectory is earned using 512 neurons.
3 1 256 7002.32 As a comparison of the RL master and PID master controllers,
4 1 512 7181.75 the RL master outperformed the PID controller (the integrated er-
ror in the case of PID master was 0.669, while in the case of RL
master was 0.002).
network structures. The average rewards show that the controller
could learn and perform the control task quite well at each net- 3.2.3. Operation of the semi-batch reactor using RL-controllers
work. Each controller gained more than 70 0 0 as a maximum aver- We present how the trained RL controller in the feeding phase
age reward (see Table 3), though the highest rewards were gained and the trained RL-PID cascade controller work together during the
using 512 neurons. The control results in the mixing phase are also operation of the semi-batch reactor, which is presented in Fig. 10.

7
Á. Sass, A. Kummer and J. Abonyi Computers and Chemical Engineering 162 (2022) 107819

Fig. 7. Performance of the cascade controller with PID master.

Fig. 8. Average rewards at different neural network structures.

Fig. 9. Performance of the cascade controller with RL master.

In the top subplot, the reactor temperature is shown during the
operation, where the feeding phase lasts until 2.1 h. The controller
switch is smooth after the feeding phase, and the reactor temper-
ature stays at the setpoint. In the middle subplot, the position of
the actuator valves is presented. The position of V 2 is noisy due to
the discrete actions performed by the RL-master controller.
3.3. Control results with open-loop NMPC
The open-loop optimization problem was solved by the SQP op-
timization algorithm, where the algorithm proceeds with a mov-
ing horizon. Since we would like to compare the results of NMPC
to the RL-based control framework, we used the same 100 s as a
sample time. The number of control intervals is set to 10, and the
number of prediction intervals is set to 15 (excluding control inter-
vals). The control and prediction intervals were set quite long due
to the potential hazards of a thermal runaway because NMPC only
works if the development of the thermal runaway can be seen in
advance in the prediction horizon. Fig. 11 presents the control of
the SBR. The NMPC can solve the control task well, although no
uncertainties were considered. There is only a slight overshoot; the
reactor temperature follows the setpoint.

8
Á. Sass, A. Kummer and J. Abonyi
Fig. 10. SBR temperature control using RL-controller.
Fig. 11. SBR temperature control using NMPC.
Comparing the RL-based controllers and NMPC, we can say that
the RL-controller in the feeding phase lets in more reagents in the
first half an hour; hence the setpoint is reached earlier. On the
other hand, the feeding strategy is similar, so there is a higher
feeding rate at the beginning of the operation (until 0.6–0.7 h),
then the feeding rate is decreased to avoid high overshoot, then
the feeding rate is slightly increases continue to keep the reactor
temperature at the setpoint. The cooling flow rate continuously de-
creases in the mixing phase to follow the continuously decreasing
reaction rate.
3.4. Discussion
The application of RL-controllers show great potential in ex-
ploring the operational strategies of SBRs, but we need to de-
fine problem-specific RL-controllers. Since the different operation
phases of SBRs require different control approaches using multi-
agent techniques is a well-functioning idea. During the operation
of SBRs the agents can be decoupled from each other, so phase-
based and individual RL-controllers can be applied for the oper-
ation. Increasing the number of hidden layers and the number
of neurons leads to a better performance of the RL-controllers,
though we need to avoid unnecessarily complex neural networks.
The RL-controller is realized in a single control-loop in the feed-
ing phase, just like an ordinary control-loop using PID. Unlike a
PID controller, the RL-controller can handle the non-linearities due
to the accumulation of reagents, so developing a possible runaway
can be avoided. In the mixing phase, the RL-controller worked very
well as a master in a cascade control-loop, and it performed better
than an ordinary cascade-loop using PID controllers. The two RL-
controllers worked well in the operation of the SBR because the
9
Computers and Chemical Engineering 162 (2022) 107819
reactor temperature was led to and kept at the desired setpoint
during the whole operation.
Compared to the NMPC, the RL-based controllers followed a
similar strategy as the NMPC but reached the setpoint faster, cre-
ating a better overall yield thanks to the higher reaction rate at
higher temperatures. Contrary to the performance of the RL-based
controller, a slight overshoot appeared in the case of NMPC. Re-
garding computational requirements, in the case of NMPC, the op-
timization time in some cases was very close to the action time
interval of 100 s, which might jeopardize the applicability of the
NMPC in this case of this complex model and optimization prob-
lem. The RL-based controller was trained on the simulator “offline”
for eight hours with the same computational capacity, and the ap-
plication of the trained controller required computational capac-
ity that allowed the online application. For RL-controllers’ learning
and application, we need an adequate and reliable model over a
wide interval of operating parameters, so the used policies by the
agents become reliable and usable on the investigated real system.
The neural networks are difficult to be interpreted and understood,
and they are also difficult to be validated. Therefore, the perfor-
mance of the control needs to be monitored continuously, and if
any malfunction occurs, we may need to switch to a back-up con-
troller to keep in hand the system. A potential future research area
is to work out validation and failure analysis techniques for the
RL-controllers.
4. Conclusion
A reinforcement learning-based approach has been investigated
to operate a semi-batch reactor carrying out an exothermic re-
action optimally. We developed an RL-based control scheme to
Á. Sass, A. Kummer and J. Abonyi Computers and Chemical Engineering 162 (2022) 107819

operate SBRs, where an RL-controller is used for both the feed- Table A1
Parameters and initial conditions of the case study.
ing and mixing phase. The operation of semi-batch reactors needs
higher attention due to the potential runaway reactions. If the Parameter Value Unit
feeding reagents accumulate in a higher concentration, the reac- k0 pre-exponential factor 1.465 × 107 m3
kmols
tion may trigger non-desired, resulting in a rapid temperature in- EA
activation energy 8500 K
Rg
crease. The proposed reinforcement learning-based controller helps Hr reaction heat parameter −3.5 × 105 kJ
kmol
operate the reactor safely if the training is well-designed. MAT Maximum Allowable Temperature 373 K
kW
The RL-controller varied the feed rate in the feeding phase to UA0 initial heat transfer parameter 1.85 K
V0 initial reagent volume 0.5 m3
keep the temperature at the desired setpoint without crossing the
VJ jacket volume 0.41 m3
maximum allowable temperature. The agent was trained from ran- kJ
ρcp liquid property in reactor 4800
m3 K
dom initial reactor temperature and loaded amount of reagents, kJ
( ρ c p )J cooling agent property 4183
so the controller could define the appropriate action to take in a m3 K
wider operation interval. In the mixing phase, a cascade-type con-
troller was used to keep the temperature at the desired setpoint, Table A2
where we compared the standard cascade controller with PIDs and Operating parameters .
the RL-based cascade controller and the NMPC. The RL-based cas- Parameter Value Unit
cade controller was able to keep the temperature closer to the set-
nA, f eed feed moles of A reagent nB,0 kmol
point than the PID controller. Also, it was able to reach the setpoint nA,0 initial moles of A reagent 0 kmol
faster than the NMPC and showed zero sign of overshoot. nB,0 initial moles of B reagent U ( 4, 5 ) kmol
The reinforcement learning methods work from experience to cin,A feed concentration of A reagent 5 kmol
m3
m3
experience, just like batch-to-batch learning or run-to-run opti- Fmax maximum flow rate of feeding reagent 1.4 × 10−4 s
m3
mization. Since there is no option to perform experiments on Fj,max coolant flow rate 1.1 × 10−3 s
TR,0 initial reactor temperature U (333, 363 ) K
the real system, there is always a need for adequate and reli-
TJ,0 initial jacket temperature 298 K
able model development for learning RL-based controllers. Unfor- Tin reagent feed temperature 298 K
tunately, the modelling and simulator development of any process TJ,in coolant feed temperature 298 K
system is time and cost consuming, but for most units/systems, a
well-designed methodology for modelling is already worked out.
Also, the use of open-source programming languages can compen-
sate for the mentioned disadvantages.
dnA
The applied RL-based controller works very well in the inves- = F cin,A − V rR (A.3)
dt
tigated, simulated system, and it is worth continuing the work in
this research field. However, when implementing an RL controller
on a real system, there will be a plant-model mismatch when dnB dnC
=− = −V rR (A.4)
the controller is trained offline based is a not-perfect simulation dt dt
model. The requirement of valid information for controller tuning
implies that the RL-based controller must be fine-tuned using data
dTR F ρ c p (Tin − TR ) + (−Hr )V rR − UA(TR − TJ )
collected from the real-time operation. Although model-plant mis- = (A.5)
dt V ρ cp
match and non-observable states are not negligible in the applica-
tion of reinforcement learning, it was not analyzed in this work.
However, it will be the subject of our future investigations. dTJ FJ (ρ c p )J (TJ,in − TJ ) + UA(TR − TJ )
= (A.6)
dt (V ρ c p )J
Declaration of Competing Interest
Vdos
The authors declare that they have no known competing finan- UA = UA0 1 + (A.7)
V0
cial interests or personal relationships that could have appeared to
influence the work reported in this paper. where t is time [s], V is liquid volume [m3 ], Vdos is dosed vol-
3
ume [m3 ], F is feeding rate ms , ni is the mole amount of the
Appendix A. Process model of the reactor
ith component [kmol], TR and TJ are the reactor and jacket temper-

The reaction rate (rR kmol
m3 s
) is expressed with a Arrhenius-type atures respectively [K], ρ is density kg
m3
, c p is the heat capacity
equation:  
kJ
kgK
, Hr is reaction heat kJ
kmol
, UA is the heat transfer param-
−EA
rR = kcA cB = k0 exp cA cB (A.1) eter . kW
Rg TR K
 The considered control valves have linear characteristics, so the
m3 s following equations (Eqs. (A.8) and (A.9)) describe the inlet flow
where k is the reaction rate constant kmol
, cA and cB is the
rates.
kmol
concentration of A and B reagents , k0 and EA are the pre-
 m3  V 1 pos
exponential factor m3 s
and the activation energy kJ
respec- F = Fmax (A.8)
kmol kmol 100

kJ
tively, Rg is the gas constant kmol K
and TR is temperature of the
V 2 pos
reactor [K]. Fj = Fj,max (A.9)
100
The following differential equations describe the dynamic be-
haviour of the system. The kinetic, material and reactor geometry parameters are pre-
dV sented in Table A.4, and operating parameters are presented in
=F (A.2) Table A.5.
dt
10
Á. Sass, A. Kummer and J. Abonyi
Appendix B. Reinforcement learning
In reinforcement learning an agent interacts with the environ-
ment. The environment produces information which describes the
state of the system (st ), and the agent interacts with the environ-
ment through an action (at ) based on the observed states. The ac-
tion is performed on the environment and it transitions into the
next state, then it returns the next state and the reward (rt ) to the
agent. The information exchanged are (st , at , rt ), where t denotes
the time step, and this tuple is called an experience. A trajectory is
a sequence of experiences over an episode, τ = (s0 , a0 , r0 ), (s1 , a1 ,
r1 ), ... (sT , aT , rT ) (Graesser and Keng, 2019).
A policy (π ) of an agent is a function that maps states to ac-
tions, and the objective of an RL problem is to find the optimal
policy, where the best action is chosen at every state. The optimal
policy can be found by maximizing the sum of rewards by select-
ing the best actions. A discounted cumulative reward is used in the
objective to maximize, which can be seen in Eq. (B.1). The discount
factor (γ ) can be in the interval of [0, 1], but in most cases it is in
the interval of [0.95, 0.99].

T mann equation.
R ( τ ) = r0 + γ r1 + γ 2 r2 + . . . + γ T rT = γ t rt
t=0
The objective (J (τ )) is the expectation of returns over many tra-
jectories (see Eq. (B.2)):
 

T
J ( τ ) = E τ π [ R ( τ )] = E τ γ t rt
t=0
The agent can learn the optimal policy based on the action-
value function (Q π (s, a )) through estimating the expected return
(Eτ [R(τ )]). The optimal action-value function is the maximum of
the expected return after performing an action (a) at the current
state (s) (Graesser and Keng, 2019).
 

T
Q ∗ (s, a ) = maxEs0 =s,a0 =a,τ π γ t rt
t=0
Bellman equation defines a recursive relationship to define the
Q-function, where s and a is the subsequent state and action fol-
lowing s and a (Graesser and Keng, 2019).
Q ∗ (s, a ) = maxEs0 =s,a0 =a,τ π r (s, a ) + γ Ea π [Q π (s , a )]
If the target policy is deterministic, we can describe this
policy as a function μ : S ←− A and avoid the inner expectation
(Lillicrap et al., 2015):
Q ∗ (s, a ) = maxEs0 =s,a0 =a,τ π r (s, a ) + γ Q μ (s , μ(s ))
A commonly used policy in Q-learning is a greedy policy:
μ(s ) = argmaxa Q (s, a )
In practice, Q-function is represented with a non-linear function
approximator due to its size issues. We can use a neural network
with weights (θ ) to approximate the Q value for all possible ac-
tions in each state (Q (s, a; θ ) ≈ Q ∗ (s, a )). The neural network rep-
resenting the Q-function is called Q-network. The weights of the
Q-network are updated iteratively by minimizing the following loss
function (Ma et al., 2019; Lillicrap et al., 2015):
Loss = (r + γ maxa Q (s , a ; θ ) − Q (s, a; θ ))2
In numerous cases, continuous actions are inevitable, and policy
gradient methods help to realize it. The policy is directly modified
through its parameters to maximize the expected reward in policy
gradient methods.
max J (πθ ) = Eτ πθ [R(τ )]
θ
Computers and Chemical Engineering 162 (2022) 107819
Gradient ascent is used to update the policy parameters based
on the following equation:
θ←
− θ + α∇θ J (πθ ) (B.9)
where α is the learning rate. The term ∇θ J (πθ ) is known as the
policy gradient (Graesser and Keng, 2019).
A widespread RL algorithm family for continuous action prob-
lems is the actor-critic method. All the actor-critic algorithms
have two components learning together. One member is the actor,
which learns the parameterized policy based on the policy gradi-
ent theorem to define the desired action. The other member is the
critic, which learns the action-value function (Q (s, a )) to evaluate
the state-action pairs and provide a reinforcing signal to the ac-
tor. The actor is updated based on the following formula, derived
by Silver et al. and presented as the Deterministic Policy Gradient
(DPG) method (Silver et al., 2014).
∇θ μ J = Est ∇θ μ Q (s, a|θ Q )|s = st , a = μ(st |θ μ ) (B.10)
where μ(s|θ μ ) is the current policy by deterministically mapping
states to a specific action. The critic (Q (s, a )) learns using the Bell-
(B.1)
CRediT authorship contribution statement
Ádám Sass: Conceptualization, Methodology, Software, Formal
analysis, Investigation, Writing – original draft, Writing – review &
editing, Visualization. Alex Kummer: Conceptualization, Method-
(B.2) ology, Software, Writing – review & editing, Supervision. János
Abonyi: Conceptualization, Methodology, Writing – review & edit-
ing, Supervision.
References
Bai, W., Hao, L., Guo, Z., Liu, Y., Wang, R., Wei, H., 2017. A new criterion to identify
safe operating conditions for isoperibolic homogeneous semi-batch reactions.
Chem. Eng. J. 308, 8–17.
Chen, K., Wang, H., Valverde-Pérez, B., Zhai, S., Vezzaro, L., Wang, A., 2021. Optimal
(B.3) control towards sustainable wastewater treatment plants based on multi-agent
reinforcement learning. Chemosphere 279, 130498.
Clarke-Pringle, T.L., MacGregor, J.F., 1998. Optimization of molecular-weight distribu-
tion using batch-to-batch adjustments. Ind. Eng. Chem. Res. 37 (9), 3660–3669.
Copelli, S., Torretta, V., Pasturenzi, C., Derudi, M., Cattaneo, C.S., Rota, R., 2014. On
the divergence criterion for runaway detection: application to complex con-
trolled systems. J. Loss Prev. Process Ind. 28, 92–100.
(B.4) Findeisen, R., Allgöwer, F., Biegler, L.T., 2007. Assessment and Future Directions of
Nonlinear Model Predictive Control, vol. 358. Springer.
Graesser, L., Keng, W.L., 2019. Foundations of Deep Reinforcement Learning: Theory
and Practice in python. Addison-Wesley Professional.
Guo, Z., Chen, L., Chen, W., 2017. Development of adiabatic criterion for runaway de-
tection and safe operating condition designing in semibatch reactors. Ind. Eng.
Chem. Res. 56 (50), 14771–14780.
(B.5) Hall, B., 2018. Nonlinear Model Predictive Control and Dynamic Real-Time Optimiza-
tion of Semi-Batch Reactors-A Case Study of Expandable Polystyrene Production.
NTNU Master’s thesis.
Kähm, W., Vassiliadis, V.S., 2018. Stability criterion for the intensification of batch
(B.6) processes with model predictive control. Chem. Eng. Res. Des. 138, 292–313.
Kähm, W., Vassiliadis, V.S., 2018. Thermal stability criterion integrated in model pre-
dictive control for batch reactors. Chem. Eng. Sci. 188, 192–207.
Kanavalau, A., Masters, R., Kähm, W., Vassiliadis, V.S., 2019. Robust thermal stability
for batch process intensification with model predictive control. Comput. Chem.
Eng. 130, 106574.
Kummer, A., Nagy, L., Varga, T., 2020. Nmpc-based control scheme for a semi-batch
reactor under parameter uncertainty. Comput. Chem. Eng. 141, 106998.
Kummer, A., Varga, T., Nagy, L., 2020. Semi-batch reactor control with NMPC avoid-
ing thermal runaway. Compu. Chem. Eng. 134, 106694.
Lillicrap, T. P., Hunt, J. J., Pritzel, A., Heess, N., Erez, T., Tassa, Y., Silver, D.,
Wierstra, D., 2015. Continuous control with deep reinforcement learning.
(B.7) arXiv preprint arXiv:1509.02971
Ma, Y., Zhu, W., Benton, M.G., Romagnoli, J., 2019. Continuous control of a polymer-
ization system with deep reinforcement learning. J. Process Control 75, 40–47.
Machalek, D., Quah, T., Powell, K.M., 2020. Dynamic economic optimization of a con-
tinuously stirred tank reactor using reinforcement learning. In: 2020 American
Control Conference (ACC). IEEE, pp. 2955–2960.
Ni, L., Jiang, J., Mannan, M.S., Mebarki, A., Zhang, M., Pan, X., Pan, Y., 2017. Thermal
runaway risk of semibatch processes: esterification reaction with autocatalytic
(B.8) behavior. Ind. Eng. Chem. Res. 56 (6), 1534–1542.
11
Á. Sass, A. Kummer and J. Abonyi
Ni, L., Mebarki, A., Jiang, J., Zhang, M., Dou, Z., 2016. Semi-batch reactors: thermal
runaway risk. J. Loss Prev. Process Ind. 43, 559–566.
Nian, R., Liu, J., Huang, B., 2020. A review on reinforcement learning: introduc-
tion and applications in industrial process control. Comput. Chem. Eng. 139,
106886.
Pan, E., Petsagkourakis, P., Mowbray, M., Zhang, D., del Rio-Chanona, E.A., 2021. Con-
strained model-free reinforcement learning for process optimization. Comput.
Chem. Eng. 154, 107462.
Paul, S., Kurin, V., Whiteson, S., 2019. Fast efficient hyperparameter tuning for policy
gradient methods. Adv. Neural Inf. Process. Syst. 32, 4618–4628.
Powell, K.M., Machalek, D., Quah, T., 2020. Real-time optimization using reinforce-
ment learning. Comput. Chem. Eng. 143, 107077.
Rossi, F., Copelli, S., Colombo, A., Pirola, C., Manenti, F., 2015. Online model-based
optimization and control for the combined optimal operation and runaway pre-
diction and prevention in (fed-) batch systems. Chem. Eng. Sci. 138, 760–771.
Rossi, F., Manenti, F., Buzzi-Ferraris, G., Reklaitis, G., 2017. Combined dynamic op-
timization, optimal control and online runaway detection & prevention under
uncertainty. Chem. Eng. Trans. 57, 973–978.
Rossi, F., Reklaitis, G., Manenti, F., Buzzi-Ferraris, G., 2016. Multi-scenario robust on-
line optimization and control of fed-batch systems via dynamic model-based
scenario selection. AlChE J. 62 (9), 3264–3284.
Seki, H., Ogawa, M., Ooyama, S., Akamatsu, K., Ohshima, M., Yang, W., 2001. Indus-
trial application of a nonlinear model predictive control to polymerization reac-
tors. Control Eng. Pract. 9 (8), 819–828.
Shin, J., Badgwell, T.A., Liu, K.-H., Lee, J.H., 2019. Reinforcement learning–overview of
recent progress and implications for process control. Comput. Chem. Eng. 127,
282–294.
12
View publication stats
Computers and Chemical Engineering 162 (2022) 107819
Silver, D., Lever, G., Heess, N., Degris, T., Wierstra, D., Riedmiller, M., 2014. Determin-
istic policy gradient algorithms. In: International Conference on Machine Learn-
ing, PMLR, pp. 387–395.
Singh, V., Kodamana, H., 2020. Reinforcement learning based control of batch poly-
merisation processes. IFAC-PapersOnLine 53 (1), 667–672.
Srinivasan, B., Primus, C.J., Bonvin, D., Ricker, N.L., 2001. Run-to-run optimization via
control of generalized constraints. Control Eng. Pract. 9 (8), 911–919.
Syafiie, S., Tadeo, F., Martinez, E., 2007. Model-free learning control of neutralization
processes using reinforcement learning. Eng. Appl. Artif. Intell. 20 (6), 767–782.
Westerterp, K.R., Molga, E.J., 2004. No more runaways in fine chemical reactors. Ind.
Eng. Chem. Res. 43 (16), 4585–4594.
Westerterp, K.R., Molga, E.J., 2006. Safety and runaway prevention in batch and
semibatch reactors—A review. Chem. Eng. Res. Des. 84 (7), 543–552.
Xiong, Z., Zhang, J., 2005. A batch-to-batch iterative optimal control strategy based
on recurrent neural network models. J. Process Control 15 (1), 11–21.
Xu, X., Xie, H., Shi, J., 2020. Iterative learning control (ILC) guided reinforcement
learning control (RLC) scheme for batch processes. In: 2020 IEEE 9th Data
Driven Control and Learning Systems Conference (DDCLS). IEEE, pp. 241–246.
Yoo, H., Kim, B., Kim, J.W., Lee, J.H., 2021. Reinforcement learning based optimal
control of batch processes using Monte–Carlo deep deterministic policy gradient
with phase segmentation. Comput. Chem. Eng. 144, 107133.
Yoo, H., Kim, B., Lee, J.H., 2019. Deep deterministic policy gradient algorithm for
batch process control. Foundations of Process Analytics and Machine Learning:
FOPAM.
Zhang, K., Yang, Z., Basar, T., 2018. Networked multi-agent reinforcement learning
in continuous spaces. In: 2018 IEEE Conference on Decision and Control (CDC).
IEEE, pp. 2771–2776.