Data Driven Control of Interacting Two Tank
Hybrid System using Deep Reinforcement Learning
David Mathew Jones
M.Tech. Scholar, Dept. of Electrical Engineering
National Institute of Technology Calicut
Kerala, India
davidmathewjones@ieee.org
Abstract—This paper investigates the use of a Deep Neural
Network based Reinforcement Learning(RL) algorithm applied
to a non-linear system that is typically exhibited in the process
control industry. It aims to augment the large amounts of data
that we possess with the classical theory of dynamic systems
in control. Control systems represent a non-linear optimization
problem, and Machine Learning helps to achieve non-linear
optimization using large amounts of data.
This document demonstrates the use of Deep Deterministic
Policy Gradient (DDPG), which is an actor-critic methodology
of reinforcement learning, applied on an Interacting Two Tank
Hybrid System(ITTHS).
Index Terms—reinforcement learning, deep neural network,
actor-critic method, interacting water tank system.
I. I NTRODUCTION
We are in an era of unfathomable computational intelli-
gence, which is taking the world by storm. The huge success
of Artificial Intelligence in the usage of data to achieve
human like performance, has been motivating researchers to
implement algorithms in industrial applications ranging from
technology to healthcare. Currently most of our research
revolves around ”Narrow AI” - which is highly task specific,
and we have a long way to go before we can crack the
code to ”General AI”. In AI, the most prominent area of
focus is machine learning, which is a scientific field that
analyse statistical models and develop algorithms, in-turn
giving machines the explicit ability to self-learn tasks and
thereby eliminating rule-based programming.
We are aware that Machine Learning is composed of su-
pervised learning, unsupervised learning, and reinforcement
learning, and among these, supervised learning is currently
majorly employed across various applications. However, the
performance of supervised learning can never outperform the
supervisor or the subject matter expert since the agent only
mimics the labelling behaviour of the supervisor. Similar is
the case for unsupervised learning. Reinforcement Learning
(RL) tries to break through this performance barrier by pushing
the limits of what is currently feasible by learning the best
mapping of observations to actions (Policy) through a method
of trial-and-error search led by a scalar reward signal. Also,
it considers the feedback of the effect of actions taken on
subsequent rewards. These unique features set RL apart from
S. Kanagalakshmi
Assistant Professor, Dept. of Electrical Engineering
National Institute of Technology Calicut
Kerala, India
kanagalakshmi@nitc.ac.in
other methods of learning, pushing the current boundaries of
knowledge.
A fundamental non-linear problem across process industries
is liquid flow control and liquid level control in storage tanks
and reaction vessels. Solving non-linear problems is generally
tricky as they are less understood compared to linear
problems. Although major industries prefer conventional tank
systems for their processing, conical and spherical tanks
prove to be advantageous due to their lower cost, efficient
usage of material, ease of cleaning, and improved product
quality. Many non-linearities are introduced in the system
when such vessels are made use of in the process industry.
We can make use of modern computerized systems to control
such non-linear processes in real-time with ease.
The project aims to design a controller of Interacting Two
Tank Hybrid System, a Multi Input Multi Output Process. Here
we are given two control variables (height of each tank) and
two manipulated variables (inflow of pumps). Conventionally
such problems are handled using traditional control theory and
then designing a suitable PID, MPC, or non-linear MPC. The
process would start by writing down the equations describing
the dynamics of the systems, which would be generally non-
linear equations, and later linearize these equations. Then
linear state space control theory could be used, and we could
have a standard filter with a linear quadratic regulator or
robust control. This would work as long as we knew how
to write the equations describing the physical dynamics of the
system. However, in the real world of Industrial Processes, we
know that such systems are massively non-linear. They have
unknown dynamics, and their dimensionalities are higher, i.e.,
there may be many degrees of freedom to describe the system
state X. This high dimensionality would pose difficulties for
the system simulation. Furthermore, the physical world sys-
tems would have limited measurements and limited actuation.
A tabular comparison between PID, Model Predictive Con-
troller, and Reinforcement Learning has is shown in table I.
These fundamental challenges motivate us to use modern
data-driven or machine learning control. The method I propose
is called Data-Driven control or Machine Learning Control,
which is a modern mathematical method in machine learning
applied to control theory. It aims to augment the large amount
 TABLE I
C OMAPRISON BETWEEN PID, MPC AND RL
Attributes PID MPC RL
Controller Capability Low High High
Training Computation Cost Low Low High
Deployment Computational Cost Low High Medium
Sensitivity to Tuning High High Low
Dependency on model At all At all Only during
times times training
of data we possess with the classical theory of dynamic
systems control. Control systems represent a non-linear op-
timization problem, and Machine Learning helps achieve non-
linear optimization using large amounts of data.
There are many ways in which machine learning could be
implemented. Firstly, it can be used to write the equations us-
ing data-driven models and later apply classical control, model
predictive control, or optimal control; and this has gained the
most attention. Secondly, it could be used to learn suitable
controllers using data, i.e., explore the system, try different
control strategies, and develop the most effective controller.
The last way Machine Learning can help is in sensor and
actuator placement to get nearly optimal performance but are
very fast and easy to implement. The most effective method
among these three would be to parallelly try out methods one
and two to get the best representation model of the system
and the most efficient controller.
Here, for the Interacting Two Tank Hybrid System (ITTHS),
I have proposed a method of using what is already known
about the system, i.e., the system model, and then using
data-driven tools to decide on the best controller for the
application. This system, ITTHS, was chosen because of its
superiority. Conical and spherical tank systems have varying
cross-sectional areas, and their process variables(liquid levels)
are constantly changing and exhibits non-linear dynamic
behaviour, making control of this system a complex problem.
J. Martinez-Piazuelo and et al in his paper[1] has deployed
an algorithm which eases the training process on the control
policy. This is named as Multi actor -critic Reinforcement
Learning and unlike the traditional actor-critic algorithm eases
the value function learning task by distributing it into multiple
neural networks. The algorithm utilises the Deep Deterministic
Policy Gradient (DDPG) and has demonstrated its application
on the quadruple tank process and vertical tank process. Two
fully connected hidden layers with 20 neurons and tanh activa-
tion functions comprise the actor neural networks. Similarly,
the actors’ output layers have two neurons that represent the
two components of the system’s input, as well as a tanh
activation that keeps the control actions within [-1; 1]. On the
other hand, the critic networks are defined according to two
distinct schemes that aim to provide a fair comparison between
single-critic and multi-critic approaches. They use critics with
two hidden layers of 40 neurons in the first scheme, which
is used for multi-critic approaches.In the second scheme,
they double the size of the critic’s hidden layers to have 80
neurons. The hidden layers in both cases have tanh activation
functions, and the output layers have a single neuron with
linear activation. It is concluded that the proposed multi-critic
schemes do not alter the structure of the actor network and, as
a result, do not increase controller complexity when compared
to the standard single-critic method.
D. Machalek and et al. investigate three RL algorithms in
their paper [2]: deep deterministic policy gradient (DDPG),
twin-delayed DDPG (TD3), and proximal policy optimization.
They are graded based on how well they converge to a
stable solution and dynamically optimise the economics of
the Continuous Stirred Tank Reactor (CSTR). DDPG is the
original deep policy gradient method, and TD3 extends it by
including a second critic that operates a few steps behind
the first. The purpose of having two critics is to avoid
overestimation, because each state’s value is assumed to be the
lower of the two critics’ estimates. PPO improves its stability
by limiting the amount of policy change that can occur during
each training round.
M. Yiming and et al. completed a simulation experiment[3]
for the single tank system on the MATLAB platform with
the inclusion of disturbances. It was based on DQN algorithm
for the feedforward-feedback controller for the liquid level
control. The Deep Q-Learning Network (DQN) is a neural
network that combines Q-learning and deep learning to effec-
tively approximate the Q-learning function. In the experiment,
when compared to a PID feedback control system with the
same parameters, the feedforward-feeback controller based on
the DQN algorithm effectively overcomes the disturbance of
the control system, improves the control system’s performance,
and demonstrates the algorithm’s effectiveness.
Y. Zhang and et al. in their study[4] showed how reference
tracking as well as disturbance tracking (popularly known and
implemented widely as 2 Degree of Freedom Controller) could
be achieved by means of reinforcement learning techniques.
The validation of this method has been conducted on the Con-
tinuous Stirred Tank Reactor (CSTR) by adopting Q learning
into Adaptive Dynamic Programming(ADP) and using Kalman
filter residual generator.
It is important to mention that data driven techniques for
control have been studied much earlier. They have been
employed in Linear System Identification such as Balanced
Proper Orthogonal Decomposition(BPOD)[9], Eigensystem
Realisation Algorithm(ERA)[]and Dynamic Mode Decompo-
sition (DMD); Non - Linear System Identification such as
Nonlinear AutoRegressive Moving Average with eXogenous
input (NARMAX), Sparse Identification of Nonlinear Dynam-
ics (SINDy)[7], Genetic Programming. Also for the study of
Dynamics on measurement. Most of these methods leverage
the use of machine learning based regression. Koopman The-
ory[6] have been also looked into. For the formulation of
control laws apart from the use of Reinforcement Learning
we have SINDy with Control and DMD with control[8].
 II. R EINFORCEMENT L EARNING
Reinforcement learning(RL) is a branch of machine learning
that deals with learning control strategies from experience
to interact with a complex environment. In simple terms,
it is used to reinforce good behaviour with rewards. It has
the potential to solve complex tasks from high-dimensional,
unprocessed, sensory inputs in many areas such as playing
video games, autonomous driving, natural language, and in-
dustrial automation. In contrast to other learning methods, RL
learns from a dynamic environment, i.e., data modified due to
external conditions. One of the most famous examples of RL
implementations has been playing the board game known as
’Go’ developed by DeepMind and popularly known AlphaGo,
which defeated the world champion Lee Se-dol back in 2016.
The components of the RL framework shown in Fig. 1 com-
prise the Agent, which interacts with the Environment based
on a given Policy that is updated by an algorithm. The Agent
gets to measure its current state in the Environment and then
takes action from its previous experience, and occasionally, it
is rewarded based on the action taken.
Fig. 1. Reinforcement Learning System
The challenge here is to design the policy of what actions
to take given the current observation to maximize the chance
of getting a positive future reward. From a control system
perspective, this policy is going to be deterministic rather
than probabilistic. Therefore, the goal of the RL framework
is to optimize the policy given the reward at each state. Some
optimization techniques include - Differential Programming,
Monte Carlo, Temporal Difference, Bellman optimization,
Policy Iteration. The action taken, whether good or bad, is
evaluated using a value function. However, to guide it better,
we use Q-Learning, which is a function of state and the
action taken, assuming it does the best thing in the future
and measures the quality of being in that state and taking that
action.
A. Reinforcement Learning Workflow
There are 5 steps involved in the training of an RL agent:
1) Creation of the environment: Define the context within
which the agent can learn, including the interface be-
tween agent and environment. The environment might be
either a simulation model or an actual physical system.
Model identification is preferred for accelerated pre-
training, but it is necessary that the model is accurate
for the RL’s operating range.
2) Reward Shaping: Describe the reward signal that the
agent uses to assess its performance in relation to the
task goals, as well as how this signal is derived from
the environment. It may take a few iterations to get the
reward shaping just perfect.
3) Creation of the agent: Because the agent is made up of
the policy and the training algorithm, we must
• choose a method to represent the policy, such as
neural networks or lookup tables.
• Select the appropriate training algorithm for com-
plex problems and large state-action spaces.
4) Training and validation of the agent: Configure training
options (such as stopping criteria) and train the agent to
fine-tune the policy. Simulation is the simplest way to
validate a trained policy.
5) Deploying the policy: Deploy the trained policy repre-
sentation using generated C/C++ or CUDA code, for
example. At this point, there is no need to worry about
agents or training algorithms because the policy is a
stand-alone decision-making system.
III. D EEP D ETERMINISTIC P OLICY G RADIENT
In Deep Reinforcement Learning, we are implementing
the policy of the agent by means of deep neural networks.
The idea behind using the neural network architectures for
reinforcement learning is that we want reward signals obtained
to strengthen the connection that leads to a good policy.
Moreover, these deep neural networks are unique in their
ability to represent complex functions if we give them ample
amounts of data.
The Deep Deterministic Policy Gradient[5] formulated by
Lillicrap et al. was the first RL algorithm which was effective
in solving numerous continuous control taks of high dimen-
sionality. It is based on the Deterministic Policy Gradient
(DPG) algorithm by Silver and et al, with insights from
success of Deep Q Network (DQN) by Mnih et al. The
DDPG is a model free, off-policy, actor-critic reinforcement
learning method which can operate on continuous action and
observation spaces. It is superior to DQN in the sense that most
real world physical control tasks have real valued and high
dimensional action spaces. The DPG, also known as the actor,
is used to map states or observations to actions, whereas the
DQN, also known as the critic, is used to identify action-values
in order to update the DPG. The unification of two algorithms
have proved to reduced the problem of high bias and high
variance. In DDPG, the policy gradient methods are trained
using gradient of the critic using gradient ascent and deep
neural network function approximators are used to estimate
the action-value function.
 Here the network is trained off-policy meaning that samples based RL controller. This is considered as a Multi Input Multi
from a replay buffer are utilised so as to reduce the correlation Output(MIMO) system which takes into account the non-linear
between samples. Moreover the training is carried out sing coupled dynamics. The system is illustrated in Fig. 2 and
target Q network to give consistent targets along with batch the goal is to control the heights of tank 1 and tank 2 with
normalisation. reference to some given desired signals. The system is trained
on a 4 core i5 processor and a 1650 Nvidia GPU (1024 CUDA
Algorithm 1: DDPG algorithms cores, no tensor cores).
Initialise the critic Q(S, A) with random parameters The system parameters has been inspired by [10] and has
θQ and initialise the target critic with the same been summarised in Table II.
parameter values :θQ0 = θQ
Initialise the actor µ(S) with random parameters θµ TABLE II
and initialise the target actor with the same parameter PARAMETERS OF THE ITTHS
values :θµ0 = θµ
Initialise replay buffer R Parameters Description Values
for episode = 1,M do h1 , h2 Height of the conical and spherical tank cm
Initialise a random process N for action at any particular instant of time
H1 Total height of tank1 70 cm
exploration; H2 Total height of tank2 50 cm
Receive an initial observation S; R1 Maximum radius of tank1 30 cm
for t=1,T do R2 Maximum radius of tank2 25 cm
Select action A = µ(S) + N ; FI N 1 , FI N 2 Maximum Inflow of Tank1 & 2 200 cm3 /sec
CV1 ,CV2 Coefficient of control valve 1& 2 0 to 1
Execute action A and observe the reward R M V12 Coefficient of 0 to 1
and next observation S 0 ; interacting manual valve
Store the experience (S, A, R, S 0 )in the α1 , α2 CSA of output pipe in Tank1 & 2 1.2272cm2
experience buffer; α12 CSA of interacting pipe 1.2272cm2
g Acceleration due to gravity 981cm/sec2
Sample a random mini-batch of M experiences
(Si , Ai , Ri , Si0 )from the experience buffer;
If Si is a terminal state, set the value function
target yi to Ri , else
yi = Ri + γQ0 (Si0 , µ0 (Si0 |θµ )|θQ0 )
Update the critic parameters by minimizing the
loss L across all sampled experiences:
M
1 X
L= (yi − Q(Si , Ai |θQ ))2
M i=1
Update the actor parameters using the sampled
policy gradient:
M
1 X
∇θ µ ≈ Gai Gµi
M i=1
Gai = ∇A Q(Si , A|θQ )where, A = µ(Si |θµ )
Gµi = ∇θµ µ(Si |θµ )
Update the target network
θQ0 = τ θQ + (1 − τ )θQ0 Fig. 2. Schematics of the ITTHS

θµ0 = τ θµ + (1 − τ )θµ0
The Interacting Two Tank Hybrid System (ITTHS) com-
end prises of a conical tank and a spherical tank named as tank1
end and tank2 respectively. These tanks can hold liquid upto a
height of 70cm (H1 ) & 50cm (H2 ) and their radii are 30cm
(R1 ) & 25cm (R2 ) respectively. A restriction (HV12 ) which is
IV. A PPLICATION TO M ULTI -TANK H YBRID WATER manually operated interconnects these two tanks. We denote
S YSTEM FI N 1 and FI N 2 as the two input flow rates and FO U T 1 ,
In this work, an interacting hybrid multi-tank water system FO U T 2 as the output flow rates of the Tank1, Tank2 which
is used as the testbed for the demonstration of the DDPG flows through restriction HV12 to drain.
 Fig. 3. Simulink Model of RL-ITTHS
The liquid level of Tank1 (h1 ) & Tank2 (h2 ) are to be
controlled by varying the inlet flows of tanks. Magnetic Flow
transmitters measure the input flow FI N 1 of tank 1 and FI N 2
of tank 2 and transmit them in the form of current signals
to the Data Acquisition System. To control the motion of the
valves, current signals from the controller are transmitted to
the control valves so as to obtain the required flow and thereby
maintain the required set point level.
In the MIMO process of ITTHS, the manipulated variables
are the inflow FI N 1 and FI N 2 respectively and the measured
variables are the respective heights of the tanks h1 & h2 .
From the paper[10] by Balram Naik and et al., we get the
mathematical model of the system using the first principles.
We could use this system identification readily for our pro-
posed work.
The mass balance equation was used to develop the math-
ematical model of the conical and spherical tank. If D1 is the
maximum diameter and H1 is the conical tank’s maximum
height, then its height h1 at any given by :
dh1 1 p
= π D1 2 2 [u1 − (cv1 α1 2gh1 )] (1)
dt ( 4 )( H1 ) (h1 )
Similarly for the spherical tank the mass mathematical
model is as follows:
dh2 1 p
= [u 2 − (cv2 α2 2gh2 )] (2)
dt π(2R2 h2 − h22 )
The mathematical modelling for the Interactive Two Tank
Hybrid System has also been developed using the first princi-
ples as :
dh1 1 p
= π D1 2 2 [u1 − (cv1 α1 2gh1 ) − sign(h1 − h2 )
dt ( 4 )( H1 ) (h1 )
p It is also worth mentioning that during training the initial
M V12 α12 |2g(h1 − h2 )|] (3)
dh2 1 p
= [u2 −(cv2 α2 2gh2 )+sign(h1 −h2 )
dt π(2R2 h2 − h22 )
p
M V12 α12 |2g(h1 − h2 )|] (4)
We have made use of the equation to model the system in
the simulink. The RL agent will be trained on this system
first before physical model validation could be applied. The
complete simulink simulation modelling of the RL-ITTHS has
been shown in Fig. 3 The RL agent consists of 3 inputs namely
the Observation, the Reward and the isdone. The observation
takes in the present height of both the tanks, errors wrt to
the set point and the integral error. The Observation block is
represented in Fig. 4
Fig. 4. Agent Observation Block
The reward block is formalised taking into consideration the
fact that, the error must be within +/-0.1 range.
The isdone block makes sure that the prediction is always
within the physical constraints of the equipment setup. Here
the constraints are the height of each tank being 70cm and
50cm respectively.
The Reward block and isdone block have been represented
in Fig. 5 and Fig. 6.
height of the tank and the desired level of each tanks are
modified using a random function, so that the agent is trained
for the complete height of each tanks.
Fig. 5. Agent Reward Block
Fig. 6. Agent Stop Block
For the critic network, the observation path has a fully-
connected layer of 50 neurons and an Relu activation function
followed by a second fully-connected layer of 25 neurons.
Meanwhile the action path of the critic network has a fully
connected layer of 25 neurons. The output of the critic is
a added combination of the two sub neural networks with a
Relu activation function followed by a fully connected layer.
For the actor network, we have a fully connected layer of 3
neurons followed by a Tanh activation function and a final fully
connected layer. The learning rate of 10−4 for the actor and
10−3 for the critic have been used respectively. For updating
the actor and critic parameters we have employed smoothing
factor of 0.001. We trained with mini batches of 64 and used
replay buffer of 106 . We used Ornstein-Uhlenbeck process
with variance =0.3 and decay rate of 10−5 . The training is done
for utmost 5000 episodes, however the training can be stopped
automatically once the agent reaches an average cumulative
reward greater than 800 over 20 consecutive episodes.
V. S IMULATION R ESULTS
Although the proposed algorithm for the system was de-
signed, it was unable to complete the training due to com-
putational complexity. Typical machine learning algorithms
rely heavily on GPU to accelerate their learning process and
only minimally on CPU. Deep Reinforcement learning relies
both on GPU as well as CPU. In our case, the reinforcement
learning algorithms is taken care of by the GPU whereas the
simulation environment, reward and observation calculation,
replay buffer, part of policy network and part of rendering is
taken care by the CPU. GPU acceleration using tensor cores
will help neural network to be trained faster and a CPU with
higher cores will accommodate other factors of the simulation
environment.
The system under consideration has been designed keeping
in mind the wide range of input (desired water level and initial
water level). Also, since the system is an interactive system,
the coefficient of interaction ranging from 0 to 1 has also been
kept as a training input, which is randomly changed throughout
the course of the training. Fig. 7, shows the completed training
for just the conical tank, however for the performance study
the system inclusive of both the tanks and the complete range
of operating conditions has to be specified. When simulated,
as shown in Fig. 8, the training was able to reach upto 60
episodes due to computational constraints.
Fig. 7. RL Agent Training - Conical Tank System
Fig. 8. RL Agent Training - Overall ITTHS
 Currently modern tools for an end to end GPU accelerated
reinforcement learning is also under study such as Nvidia Isaac
Gym.
VI. C ONCLUSION
In this paper, an RL- Deep Deterministic Policy Gradient
(DDPG) Agent based controller was proposed for the Inter-
acting Two Tank Hybrid System and simulated on MATLAB
environment using system model obtained from the first prin-
ciples. The designing of the deep neural network for actor and
critic were completed. Future Score of this work will focus on
employing high end computation systems so as to complete
the training process and also run validation on physical ITTHS
setup.
R EFERENCES
[1] J. Martinez-Piazuelo, D. E. Ochoa, N. Quijano and L. F. Giraldo, ”A
Multi-Critic Reinforcement Learning Method: An Application to Multi-
Tank Water Systems,” in IEEE Access, vol. 8, pp. 173227-173238, 2020,
doi: 10.1109/ACCESS.2020.3025194.
[2] D. Machalek, T. Quah and K. M. Powell, ”Dynamic Economic Opti-
mization of a Continuously Stirred Tank Reactor Using Reinforcement
Learning,” 2020 American Control Conference (ACC), Denver, CO,
USA, 2020, pp. 2955-2960, doi: 10.23919/ACC45564.2020.9147706.
[3] M. Yiming, P. Boyu, L. Gongqing, L. Yongwen and Z. Deliang,
”Feedforward Feedback Control Based on DQN,” 2020 Chinese Control
And Decision Conference (CCDC), Hefei, China, 2020, pp. 550-554,
doi: 10.1109/CCDC49329.2020.9163990.
[4] Y. Zhang, S. X. Ding, Y. Yang and L. Li, ”Data-driven design of two-
degree-of-freedom controllers using reinforcement learning techniques,”
in IET Control Theory & Applications, vol. 9, no. 7, pp. 1011-1021, 23
4 2015, doi: 10.1049/iet-cta.2014.0156.
[5] Timothy P. Lillicrap, Jonathan J. Hunt, Alexander Pritzel, Nicolas Heess,
Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra, Continuous
control with deep reinforcement learning, CoRR abs/1509.02971 (2015).
[6] Data-driven discovery of Koopman eigenfunctions for control, Eurika
Kaiser and J. Nathan Kutz and Steven L. Brunton, 2020, 1707.01146.
[7] Sparse identification of nonlinear dynamics for model predictive control
in the low-data limit, Kaiser, E. and Kutz, J. N. and Brunton, S. L.,
Proceedings of the Royal Society A: Mathematical, Physical and Engi-
neering Sciences, vol.474, no. 2219, 2018, doi : 10.1098/rspa.2018.0335.
[8] Dynamic Mode Decomposition with Control, Joshua L. Proctor, Steven
L. Brunton, and J. Nathan Kutz SIAM Journal on Applied Dynamical
Systems 2016 15:1, 142-161.
[9] B. Moore, ”Principal component analysis in linear systems: Control-
lability, observability, and model reduction,” in IEEE Transactions on
Automatic Control, vol. 26, no. 1, pp. 17-32, February 1981, doi:
10.1109/TAC.1981.1102568.
[10] R. B. Balaram Naik and S. Kanagalakshmi, ”Mathematical Mod-
elling and Controller Design for Interacting Hybrid Two Tank System
(IHTTS),” 2020 Fourth International Conference on Inventive Sys-
tems and Control (ICISC), Coimbatore, India, 2020, pp. 297-303, doi:
10.1109/ICISC47916.2020.9171218.