IEEE TRANSACTIONS ON SMART GRID, VOL. 11, NO.
1, JANUARY 2020
Double Deep Q-Learning-Based Distributed
Operation of Battery Energy Storage
System Considering Uncertainties
Van-Hai Bui, Student Member, IEEE, Akhtar Hussain , Student Member, IEEE,
and Hak-Man Kim , Senior Member, IEEE
Abstract—Q-learning-based operation strategies are being
recently applied for optimal operation of energy storage systems,
where, a Q-table is used to store Q-values for all possible state-
action pairs. However, Q-learning faces challenges when it comes
to large state space problems, i.e., continuous state space prob-
lems or problems with environment uncertainties. In order to
address the limitations of Q-learning, this paper proposes a dis-
tributed operation strategy using double deep Q-learning method.
It is applied to managing the operation of a community battery
energy storage system (CBESS) in a microgrid system. In con-
trast to Q-learning, the proposed operation strategy is capable of
dealing with uncertainties in the system in both grid-connected
and islanded modes. This is due to the utilization of a deep neu-
ral network as a function approximator to estimate the Q-values.
Moreover, the proposed method can mitigate the overestimation
that is the major drawback of the standard deep Q-learning.
The proposed method trains the model faster by decoupling the
selection and evaluation processes. Finally, the performance of the
proposed double deep Q-learning-based operation method is eval-
uated by comparing its results with a centralized approach-based
operation.
Index Terms—Artificial intelligence, battery energy storage
system, double deep Q-learning-based operation, microgrid oper-
ation, Q-learning, optimization.
I. I NTRODUCTION
NERGY management system (EMS) have been widely
E applied for the operation of microgrids (MGs) in both
grid-connected and islanded modes. The major responsibility
of the EMS is to optimize the operation of the microgrid’s
resources with a defined objective [1], [2]. There are two fun-
damentally different approaches to the design of EMS frame-
works, which are centralized and decentralized approaches. In
the centralized approach, a centralized EMS is provided with
the relevant information of all the distributed energy resources
Manuscript received January 16, 2019; revised April 1, 2019 and May
9, 2019; accepted June 14, 2019. Date of publication June 20, 2019;
date of current version December 23, 2019. This work was supported by
the Korea Institute of Energy Technology Evaluation and Planning and
the Ministry of Trade, Industry and Energy of South Korea under Grant
20168530050030. Paper no. TSG-00082-2019. (Corresponding author: Hak-
Man Kim.)
The authors are with the Department of Electrical Engineering,
Incheon National University, Incheon 402-749, South Korea (e-mail:
hmkim@inu.ac.kr).
Color versions of one or more of the figures in this paper are available
online at http://ieeexplore.ieee.org.
Digital Object Identifier 10.1109/TSG.2019.2924025
1949-3053 c 2019 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission.
See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
Authorized licensed use limited to: Motilal Nehru National Institute of Technology. Downloaded on October 04,2020 at 07:42:10 UTC from IEEE Xplore. Restrictions apply.
457
within the MG [3]–[5]. However, with an increase in the num-
ber of control devices, the computational burden on the central
EMS and communication complexity of the network increases.
Moreover, the centralized management entails a single point
of failure, and central unit fault will cause the breakdown
of the entire system. Low flexibility/expandability is another
critical limitation of the centralized management [6]. In addi-
tion, since all entities in the system are interconnected and
may have their own operation objectives. Therefore, the
centralized EMS could have practical limitations for the
energy scheduling of MGs having diverse objectives and
ownerships [7]. Therefore, recently, decentralized energy man-
agement systems are gaining popularity due to their ability
to overcome the limitations of centralized EMSs [6], [8], [9].
A diffusion strategy-based distributed operation of MGs has
been proposed to minimize the total operation cost by using
a multiagent system [8]. A fully distributed control strategy
has been proposed based on the consensus algorithm for
the optimal resource management in an islanded MG [9].
However, in these methods, an exact mathematical model
is required for operation of the MG system. It could be
challenging to develop exact mathematical models for com-
plex systems. Moreover, these methods are the iteration-based
optimization, i.e., agents need to exchange information several
times to achieve an optimal result. This type of iterative oper-
ation could be problematic if there is any change in the system
parameters. These methods could take a long time for finding
a new optimal point and might not be suitable for real-time
optimization problems.
To overcome the disadvantages of the conventional decen-
tralized approaches, reinforcement learning (RL)-based decen-
tralized energy management has been introduced as a potential
solution for optimal operation of MGs [10]–[12]. In RL,
agents learn to achieve a given task by interacting with their
environment. Since the agents do not require any model of the
environment, they only need to know the existing states and
possible actions in each state. This method drives the learn-
ing process on the basis of penalties or rewards assessed on
a sequence of actions taken in response to the environment
dynamics [13], [14].
Q-learning being a popular method in RL, is widely used for
optimal operation of MGs [15], [16]. In Q-learning, a Q-table
is used to represent the knowledge of the agent about the envi-
ronment. The Q-value for each state and action pair reflects
 458
Fig. 1. Microgrid configuration.
the future reward associated with taking such an action in
this state. However, Q-learning-based operation methods are
only suitable for the problems with small state space. They
are not suitable for a continuous state space or for an environ-
ment with uncertainties [17], [18]. Therefore, a model which
maps the state information provided as input to Q-values
of the possible set of actions should be developed. For
this reason, a deep neural network (DNN) comes to play
the role of a function approximator, which can take state
information input, and learn to map them to Q-values for all
possible actions. Combining Q-learning with DNN leads to
a method called deep Q-learning or deep Q networks (DQN),
which enhances the performance of Q-learning for large scale
problems [19]–[21]. In DQN, same weight values are used
for both selection and evaluation of an action. It can lead
to overoptimistic value estimates [17], [22]. To mitigate this
problem, double deep Q-learning (DDQN) has been developed
to decouple the selection from the evaluation. In DDQN,
a primary network is used to choose the action and a tar-
get network is used to generate a target Q-value for that
action [22]. Therefore, a DDQN-based operation is suitable
for the agent with a large state space and the agent can avoid
being trapped in local optimization.
Due to the advantage of DDQN, a distributed operation
strategy for managing the operation of a community bat-
tery energy storage system (CBESS) in an MG system is
proposed using DDQN. The MG system is comprised of an
MG and a CBESS, where a microgrid EMS (MG-EMS) is
used for optimizing the operation of the MG and a DDQN-
based operation is used for CBESS. In grid-connected mode,
the actions of the CBESS are only dependent on the mar-
ket price signals. The effect of uncertainty of market price
signals is analyzed in this mode. On the other hand, in
islanded mode, the actions of CBESS are dependent on
both the trading price and surplus/shortage amount of the
MG. Due to the uncertainty of load and renewable resources
in the MG, the uncertainties of surplus and shortage power
of the MG are also considered in this study. In contrast to
Q-learning, the proposed operation strategy is capable of deal-
ing with uncertainties in the system in both grid-connected and
islanded modes. Moreover, the DDQN decouples the selec-
tion and evaluation of an action, thus the CBESS can avoid
the trapping into local optima. Finally, in order to analyze
the effectiveness of the proposed DDQN-based optimization
method, the operation results of the proposed method are
compared with the conventional centralized EMS results,
and other learning methods’ results. Simulation results have
Authorized licensed use limited to: Motilal Nehru National Institute of Technology. Downloaded on October 04,2020 at 07:42:10 UTC from IEEE Xplore. Restrictions apply.
IEEE TRANSACTIONS ON SMART GRID, VOL. 11, NO. 1, JANUARY 2020
proved that the proposed method can get similar results with
those of a centralized EMS, despite being a decentralized
approach. The major contributions of this study are listed
as following.
• A distributed operation strategy is proposed for CBESS
using DDQN method in both grid-connected and islanded
modes with different objectives. Due to the use of dif-
ferent DNNs, the CBESS can be continuously trained
with a new data set in real-time operation to improve
the decision accuracy.
• The proposed operation strategy is capable of handling
the system uncertainties in the environment. The charg-
ing/discharging decisions of CBESS are adjusted with
the actual data by using a target-network with optimal
parameters.
• A comparison between the proposed method and other
methods is presented to evaluate the performance of
the proposed algorithm. Simulation results have proved
that the proposed method can get similar results with
those of a centralized EMS, despite being a decentralized
approach.
• The performance of the proposed method is also tested
for a longer time span (48-hour scheduling horizon).
The results show that the proposed can perform better
in case of a longer time span. The scalability of the
proposed method is also discussed for a larger scale, e.g.,
multi-microgrid system.
II. S YSTEM M ODEL
A. System Configuration
Fig. 1 describes the system configuration, which is com-
prised of an MG and a CBESS. The MG considered in this
study consists of a controllable distributed generator (CDG),
a renewable distributed generator (RDG), a battery energy
storage system (BESS) as the energy storage device, and res-
idential loads. The CBESS is a community battery energy
storage system with different ownership from the MG [7]. The
CBESS could be owned by any electrical company in a local
electric power distribution network. The main purpose of the
CBESS is to maximize the profit in normal mode and improve
the reliability of system during emergencies. In grid-connected
mode, power can be traded among MG, CBESS, and the util-
ity grid to minimize the total operation cost. In islanded mode,
CBESS and MG can trade power to minimize load shedding in
the network. The MG is operated by an MG-EMS for minimiz-
ing its total operation cost. MG-EMS communicates with the
utility grid to get the buying/selling price signals and decides
the amount of power to be traded with the utility grid and
CBESS in grid-connected mode. However, in islanded mode,
the MG cannot trade power with the utility grid. Thus, the
MG cooperatively operates with the CBESS to minimize the
load shedding amount.
The operation of the CBESS is based on double deep
Q-learning method for both grid-connected and islanded
modes. The detailed optimization model for CBESS is
explained in the following sections.
 BUI et al.: DOUBLE DEEP Q-LEARNING-BASED DISTRIBUTED OPERATION OF BATTERY ENERGY STORAGE SYSTEM CONSIDERING UNCERTAINTIES
B. Backgrounds of Q-Learning and Deep Q-Learning
Methods
In this section, the backgrounds of Q-learning and deep
Q-network are introduced.
1) Q-Learning: Q-learning is a popular model-free rein-
forcement learning, where an agent explores the environment
and finds the optimal way to maximize the cumulative reward.
In Q-learning, the agent only needs to know its state space and
possible actions. Each state-action pair is assigned by a Q-
value, which represents the quality of the action for the given
state. When the agent carries out an action from the given state,
it receives a reward and moves to a new state. The reward is
used to update the Q-value using Bellman equation [13], [17]
as follows.
 
  
Q(s, a) ← Q(s, a) + α. r + γ . max Q s , a − Q(s, a)
a

s←s
where:
Q(s,a): Q-value for the current state s and action a pair
α: learning rate of convergence
γ: discount factor of future rewards
r: immediate reward.
The agent explores the environment for a large number
of episodes. In each episode, the agent starts from an initial
state and performs several actions until reaching the goal state
and updates its knowledge (Q-values). The choosing action is
based on the ε-greedy policy, which is a way of selecting ran-
dom actions with uniform distribution from a set of available
actions [23], [24]. Using this policy, the agent can select an
action with the current optimal policy with probability (1-ε)
or try other actions with probability ε in a hope for a bet-
ter reward resulting in improvement in Q-values and thereby,
deriving better optimal policy.
2) Deep Q-Learning: In Q-learning method, a Q-table per-
forms well in an environment where the state space is small.
However, with an increase in the state space size, Q-learning
becomes intractable [18]. Therefore, a model that maps the
state information provided as input to Q-values of the possi-
ble set of actions should be developed. For this reason, a deep
neural network (DNN) comes to play the role of a function
approximator, which can take state information input in the
form of a vector, and learn to map them to Q-values for all
possible actions. Since a DNN is being used as a function
approximator of the Q function, this process is called deep
Q-learning. The main objective is to minimize the distance
between Q(s, a) and TD-target (temporal difference target).
This objective can be expressed as minimization of the square
error loss function, as shown in (2).
 
Li (θi ) = Eα∼μ (yi − Q(s, a; θi ))2
 
   data set is randomly selected from the memory to optimize the
where yi := Eα  ∼π r + γ max Q s , a ; θi−1 |St = s, At = a
a
The TD-target yi and Q(s, a) are estimated separately by two
different neural networks, which are often called the Target and
Q-networks. The Q-values generated by this separate Target
Authorized licensed use limited to: Motilal Nehru National Institute of Technology. Downloaded on October 04,2020 at 07:42:10 UTC from IEEE Xplore. Restrictions apply.
459
network is used to compute loss after every action taken by
the agent during training time. The weight parameters θ (i − 1)
belongs to the Target-network and θ (i) belongs to the Q-
network. The actions of the agent are selected according to
the behavior policy μ(a|s). On the other side, the greedy tar-
get policy π(a|s) selects only actions a that maximize Q(s, a),
that is used to calculate the TD-target. This technique is called
fixed Q-targets. In fact, the weights are updated only once in
a while.
Important feature added to deep Q-network is experience
replay. It means that the agent can store its past experiences
and use them in batches to train DNN. Storing the experi-
ences allows the agent to randomly draw the batches and help
the network to learn from a variety of data instead of just
formalizing decisions on immediate experiences. In order to
avoid storage issues, the buffer of experience replay is fixed,
and as the new experiences get stored the old experiences get
(1) removed. For training deep neural networks, uniform batches
of random experiences are drawn from the buffer.
3) Double Deep Q-Learning: In standard DQN and
Q-learning, the “max” operator uses the same weight val-
ues both to select and to evaluate an action [13], [17]. This
makes it more likely to select overestimated values, resulting
in overoptimistic value estimates. To prevent this, the selec-
tion should be decoupled from the evaluation. This is the
idea behind double deep Q network (DDQN).
In DDQN, instead of taking the “max” over Q-values while
computing the target Q-value during training, we use a primary
network to choose the action and Target-network to gener-
ate a target Q-value for that action. By decoupling the action
choice from the target Q-value generation, it is able to reduce
the overestimation and helps to train faster and more reliably.
The TD-Target is as follows:
    
yi := Eα  ∼μ r + γ Q s , arg max Q s , a; θi ; θi−1 |St = s, At = a
a
(3)
It can be observed that the Target-network with parame-
ters θ (i − 1) evaluates the quality of the action, the action
itself is determined by the Q-network that has parameters θ (i).
The calculation of new TD-Target yi can be summarized in
the following steps: Q-network uses next state s to calculate
qualities Q(s , a) for each possible action a in state s . The
“argmax” operation applied on Q(s , a) chooses the action a*
that belongs to the highest quality. The quality Q(s , a∗ ) is
determined by the Target-network.
Deep learning techniques require large amount of train-
ing data sets and it is challenging to generate the training
data sets in many cases [13], [17]. However, in DDQN, the
agent explores the environment and makes the data set by sav-
ing its experiences in a memory. In each episode, the training
DNN and estimate the Q values. Therefore, generation of sep-
arate training data set is not required for DDQN. Moreover,
(2)
the agent always updates its memory with new experiences
and keeps on training based on the continuous feedback to
maximize the reward. In a real-time problem, the agent can
use the Target-network to estimate the Q-values while the
 460
Q-network can be continuously trained to get more accurate
decisions.
C. A Comparison Between DDQN and Other
Optimization Methods
Dynamic programming (DP) is a popular method for
optimization problems [25]. DP solves for the optimal pol-
icy or value function by recursion. It requires knowledge of
the markov decision process (MDP) or a model of the envi-
ronment so that the recursions can be carried out. Thus, it is
lumped under “planning” instead of “learning” and its objec-
tive is to determine the optimal actions with the given MDP. If
the model of the environment is not clearly known, the use of
DP could be challenging.
On the other hand, reinforcement learning (RL) doesn’t
require knowledge of the model of the environment. It is
iterative and simulation-based and learns by bootstrapping,
i.e., the value of a state or action is estimated using the values
of other states or actions. Recently, various RL methods are
also available in the literature, e.g., optimal policy RL [13],
path integral control approach to RL [26], or actor-critic (AC)
RL [13], [27].
Among these methods, AC RL is similar to DDQN, which
is used in this study. DDQN is categorized as the value
function-based algorithms. An agent learns action values and
then acts mostly greedily with respect to those values. The
policy is inherently given by the value function itself. Actor-
critic methods [27] is a type of policy gradient algorithms.
AC method implements generalized policy iteration where the
actor aims at improving the current policy and the critic eval-
uates the current policy. Two different neural networks, i.e.,
critic network and actor network are used in AC method. The
critic uses an approximation architecture and simulation to
learn a value function, which is then used to update the actor’s
policy parameters in a direction of performance improvement.
However, AC also have some limitations, especially for the
under-consideration problem in this study. There are two main
advantages of using DDQN instead of actor-critic algorithm
in this study are as following.
• Actor-critic method is useful for a continuous action
space problem [27]. However, in this paper, a BESS
model is presented with three possible actions, i.e.,
charge, discharge, and idle modes. DDQN is easier to
implement for the stated problem.
• DDQN method guarantees a global optima. However, as
the optimization of actor is a gradient-based method, it
is easy to get trapped in local optimal result [27]. It take
a lot of training time to get a global optimal result.
To evaluate the performance of the proposed method, a com-
parison between two different learning methods is presented
for operation of the CBESS in numerical results section.
D. The Proposed DDQN-Based Operation of CBESS
In this study, the uncertainties in market price signals and
in amount of surplus/shortage power are considered for opera-
tion of the CBESS. Due to these uncertainties, the environment
Authorized licensed use limited to: Motilal Nehru National Institute of Technology. Downloaded on October 04,2020 at 07:42:10 UTC from IEEE Xplore. Restrictions apply.
IEEE TRANSACTIONS ON SMART GRID, VOL. 11, NO. 1, JANUARY 2020
Algorithm 1: Reward/Penalty for CBESS in Grid-
Connected Mode
input: s = [interval, SOC, price ± u1]
select an action a based on ε-greedy policy
with probability ε select a random action
otherwise select a = argmaxa , {Q(s, a’)}
carry out action a
Pcharge_ max = min{Pmax .(SOCmax -SOC), Pcharramp_ max }
Pdischarge_ max = min{Pmax .(SOC-SOCmin ), Pdisramp_ max }
if a = “charge” then:
if SOC = SOCmax then:
r = a high penalty
else:
r = −Pcharge .price
end
else if a = “idle” then:
r=0
else:
if SOC = SOCmin then:
r = a high penalty
else:
r = Pdischarge .price
end
end
observe new state s = [interval2, SOC2 , price2 ± u1] and r
Fig. 2. Double deep Q-learning principle diagram.
could be changed any time. It leads to increase in the num-
ber of states of CBESS significantly. Therefore, a deep neural
network is used as a function approximator, which can take
state information and learn to map them to Q-values for all
possible actions. In this section, a DDQN-based distributed
operation of CBESS is introduced in detailed to estimate the
optimal action of the CBESS. The overall deep Q-learning
principle diagram for the CBESS is summarized in Fig. 2.
The CBESS agent starts from an initial state s. The CBESS
carries out several actions and receives rewards according
to the actions and learns by replay from its experiences.
CBESS can be either in charging, discharging or idle mode
in each state. Thus, SOC of CBESS can also be increased,
decreased, or kept the same with the previous state depending
on the chosen action. The rewards/penalties of performing an
action for CBESS are summarized in Algorithms 1 and 2 for
grid-connected and islanded modes, respectively.
1) Reward Function for CBESS: Algorithms 1 and 2 play
an important role in determining the reward function for the
CBESS in different operation modes. These functions define
what are the good/bad events for the CBESS. They also map
each state-action pair to a single number, a reward, indicating
the intrinsic desirability of that state [13]. The CBESS’s objec-
tive is to maximize the total reward it receives in the long run.
 BUI et al.: DOUBLE DEEP Q-LEARNING-BASED DISTRIBUTED OPERATION OF BATTERY ENERGY STORAGE SYSTEM CONSIDERING UNCERTAINTIES
Algorithm 2: Reward/Penalty for CBESS in Islanded
Mode
input: s = [interval, SOC, price±u1, surplus±u2, shortage±u3]
select an action a based on ε-greedy policy
with probability ε select a random action
otherwise select a = argmaxa ,{Q(s, a’)}
carry out action a
Pcharge_ max = min{Pmax .(SOCmax -SOC), surplus, Pchar ramp_ max }
Pdischarge_ max = min{Pmax .(SOC-SOCmin ), shortage, Pdis
ramp_ max }
if a = “charge” then:
if SOC = SOCmax then:
r = a high penalty
else:
r = −Pcharge .price
end
else if a = “idle” then:
r=0
else:
the target yj :
if SOC = SOCmin then:
r = a high penalty
else:
r = Pdischarge .price
end
end
observe new state:
s = [interval2 , SOC2 , price2 ±u1, surplus2 ±u2, shortage2 ±u3]
and r
In each given state, the CBESS determines the maximum
amount of charging/discharging power based on the current
SOC of CBESS. After an action is selected and carried out by
the CBESS, an immediate reward is returned to the CBESS
agent. The reward depends on the time interval, current SOC
and the market price signal at that interval. In charging mode,
the CBESS receives a negative reward (penalty) for buying
power. However, in case the SOC of CBESS reaches the upper
bound, it could receive a high penalty for this wrong decision
to reduce the chance to do the same in future. Contrarily, the
CBESS can receive a positive reward for discharging mode.
Similarly, it also faces a high penalty for this mode when-
ever the SOC of CBESS reaches the lower bound. Finally, in
case of idle mode, the CBESS keeps its power for future and
there is no reward for this action. After carrying out an action,
the CBESS moves to a new state with updated information.
The main differences in the modeling between grid-
connected mode and islanded mode are the number of features
of a state and price for the trading power. In grid-connected
mode, each state (input) is a vector s with three features:
interval, SOC of CBESS, and market price signal at the current
interval. In each episode, the CBESS agent always starts from
an initial state with an initial SOC and the first interval. The
CBESS is operated for a 24-hour scheduling horizon and each
time interval is set to be 1 hour. Therefore, the first interval
is usually taken as t = 1. Based on the action selection,
reward/penalty is determined considering the uncertainty of
market price signals. In islanded mode, the CBESS operation
is also constrained by the amount of surplus/shortage power.
For example, the maximum chargeable amount in islanded
mode is constrained by the amount of surplus power and sim-
ilarly, the dischargeable amount is constrained by the amount
Authorized licensed use limited to: Motilal Nehru National Institute of Technology. Downloaded on October 04,2020 at 07:42:10 UTC from IEEE Xplore. Restrictions apply.
461
Algorithm 3: DDQN-Based Operation Strategy for
CBESS
determine operation mode of CBESS
initialize replay memory D and mini-batch size
initialize action-value functions with random weights for
Q-network (θ) and target-network (θ  = θ)
for episode = 1, N do:
initialize initial state s1 :
s1 = [interval1 , SOCint , price1 ] if grid-connected mode
s1 = [interval1 , SOCint , price1 , surplus1 , shortage1 ] otherwise
for t = 1, T do:
select an action at given state st using ε-greedy policy
carry out action at observe rt and next state st+1
store transition (st , at , rt , st ) in D
sample random mini-batch of transitions (sj , aj , rj ,
sj+1 ) from D
estimate
rj if terminalsj+j
yj =
rj + γ Q(sj+1 , argmaxa (Q(sj+1 , a; θ)); θ  ) otherwise
perform a gradient decent step on(yj − Q(sj , aj ; θ ))2
with respect to θ every C steps reset θ  = θ
end
end
of shortage power. Therefore, the input s is added with two
more features, which are surplus and shortage power of the
MG. Due to the uncertainties of load and renewable resources
in the MG system, the uncertainties of the surplus/shortage
power of MG are also considered in this study.
In grid-connected mode, the price is taken from market
price signal while in islanded mode, the price is decided based
on the percentage of critical load in each interval. Thus, the
intervals having shortage power and high amount of critical
load will propose a higher buying price than the intervals hav-
ing a lower amount of critical load for reducing the shedding
amount of critical load in the system.
Moreover, due to the uncertainties (i.e., letter ui in
Algorithms 1, 2) in the MG system, the trading prices and the
amount of surplus/shortage power could be randomly changed
between lower and upper bounds of these uncertainties. This
may lead to increase in the number of states to an infinite num-
ber. Consequently, the immediate reward receiving by carrying
out an action from given state could also be changed at every
time step during an episode.
2) Algorithm for Operation of CBESS: The detailed algo-
rithm for operation of CBESS using DDQN is presented in
Algorithm 3. Firstly, the operation mode of CBESS is deter-
mined, i.e., grid-connected or islanded mode. Based on the
operation mode, the number of features for a state is deter-
mined for the CBESS. The CBESS is trained with large
number of episodes to optimize the parameters of its deep
neural networks. As mentioned earlier, there are two deep
neural networks, i.e., a Q-network and a Target-network.
These networks play an important role in the operation of
the CBESS. They help the CBESS in selection and evaluation
of an action. Therefore, optimal parameters of these networks
are necessary for operation of the CBESS. In this case, gra-
dient decent method is used for minimization of the square
 462 IEEE TRANSACTIONS ON SMART GRID, VOL. 11, NO. 1, JANUARY 2020

Fig. 3. Flowchart for operation strategy: (a) MG-EMS; (b) CBESS.

error loss function with respect to network parameters of the operation mode. The agent carries out the action and receives
Q-network (θ ). After a constant time step, all the parameters a reward and moves to a new state. This process is explaned
of the Target-network are replace by the Q-network’s param- in detail in Algorithms 1 and 2. After finishing a transition
eters. Finally, with the optimal parameters, they can guide the from s to s’, the transition information {s,a,r,s’} is stored in
CBESS for selecting an optimal action in a given state. the replay memory D. A sample random mini-batch is taken
from memory D. Then the agent estimates the actual/target
E. Operation Strategy for MG and CBESS y for the experience and performs an optimization step to
minimize the square loss function by using gradient decent
In this section, the detailed operation strategies for MG and
method with respect to Q-network’s weights (θ ). The weight
CBESS are presented, as shown in Figs. 3(a), (b), respectively.
parameters of the Q-network is updated after each training
These strategies are implemented for both grid-connected and
step and the optimal network is used to select the action of
islanded modes. In grid-connected mode, MG is connected to
CBESS. After a constant number of steps, the weight parame-
the utility grid. The hourly day-ahead market price signals,
ters of Target-network are reset by Q-network’s parameters.
generation capabilities of CDG, RDG, and BESS along with
The Target-network is used to evaluate the quality of the
load profiles of MG system are taken as inputs. The MG-
selected action. Finally, the optimal action and the amount of
EMS receives this information and performs optimization for
charging/discharging power are determined by using the DNN
all participating components to minimize the total operation
with the optimal weight parameters. The overall process of the
cost. The decision about surplus and shortage amounts is based
CBESS is summarized in Fig. 3(b).
on the generation cost of the CDG unit, market trading prices,
After gathering the information from CBESS and the util-
and local load demands of MG.
ity grid, MG-EMS decides the amount of buying/selling power
However, in the islanded mode, MG is disconnected from
from/to the utility grid and CBESS and informs the optimal
the utility grid. The main objective is to minimize the amount
results to its components. However, in islanded mode, there
of load shedding in the MG system. The amount of sur-
is no connection to the utility grid. Load shedding could be
plus/shortage power is now only based on the load profiles
implemented to maintain the power balance. After perform-
and the available generated power. Finally, The MG-EMS will
ing power sharing with the CBESS, the MG-EMS reschedules
send its information of surplus and shortage power to exter-
the operation of all the components based on the charg-
nal systems and wait for feedback information. The detailed
ing/discharging amount from CBESS. Finally, a load shedding
process is summarized in Fig. 3 (a).
scheme is implemented for maintaining the power balance in
On the CBESS side, the detail DDQN-based operation strat-
the whole system in case of having shortage power.
egy for CBESS is presented in Fig. 3(b). Firstly, the weight
parameters of deep neural networks (DNNs) (i.e., Q-network
and Target-network) are randomly initialized and the replay F. Operation of CBESS in a Multi-Microgrid System
memory is also initialized with size D. In each episode, the In a large area, there may be several MGs. They are inde-
agent starts from the initial state and selects an action based on pendently operated in normal operation. However, the uncer-
ε-greedy policy. The state of CBESS is changed based on the tainties have a high effect on the operation of a single MG. The

Authorized licensed use limited to: Motilal Nehru National Institute of Technology. Downloaded on October 04,2020 at 07:42:10 UTC from IEEE Xplore. Restrictions apply.
 BUI et al.: DOUBLE DEEP Q-LEARNING-BASED DISTRIBUTED OPERATION OF BATTERY ENERGY STORAGE SYSTEM CONSIDERING UNCERTAINTIES 463

The constraints associated with CDGs include (7)-(12).

Constraint (7) shows the upper and lower operation bounds
of CDGs. Equation (8) gives the on/off status of CDGs. The
start-up and shut-down modes are determined by using con-
straints (9), (10) based on the on/off status of CDGs. The
bounds for ramp-up/ramp-down rates of CDGs are enforced
by (11), (12), respectively.

Fig. 4. Extension to a multi-microgrid system. ui,t · Pmin

i ≤ PCDG
i,t ≤ ui,t · Pmax
i ∀i ∈ I, t ∈ T (7)

1 CDG is on
ui,t = ∀i ∈ I, t ∈ T (8)
0 CDG is off
amount of load shedding could also be high during islanded   
yi,t = max ui,t − ui,t−1 , 0 ∀i ∈ I, t ∈ T (9)
operation and may result in reduction of system reliability.   
Recently, the concept of multi-microgrid (MMG) is introduced zi,t = max ui,t−1 − ui,t , 0 ∀i ∈ I, t ∈ T (10)
as a potential solution to overcome these problems [7], [28]. PCDG
i,t − PCDG
i,t−1 ≤ RUi · (1 − yi,t ) + max{RUi , Pmin
i } · yi,t ∀i ∈ I, t ∈ T
MMG system is formed by connecting several neighboring (11)
MGs to exchange power among MGs. In case a CBESS con-
PCDG
i,t−1 − PCDG
i,t ≤ RDi · (1 − zi,t ) + max{RDi , PCDG
i,t−1 } · zi,t ∀i ∈ I, t ∈ T
nects with an MMG system, as shown in Fig. 4. Our proposed
method should be extended to optimal operation of the whole (12)
system in a distributed manner.
The power balance between the power sources and power
The internal trading in the MMG system is the main con-
demands is given by (13). The buying/selling power is the
cern for this extension. A retailer agent is introduced to solve
amount of power trading with the external systems, which is
the problem of internal trading. It provides a dynamic internal
divided into trading with the utility grid or the CBESS, as
trading prices for MMG system to maximize the total internal
given in (14), (15), respectively.
trading amount. In order to optimize the dynamic internal trad-
 Buy
ing prices, a DDQN-based decision making for the retailer
t + Pt
PPV + + Pt + PB−
WT
PCDG
i,t t
could be suitable for this issue based on the amount of i∈I
surplus/shortage power in each individual MG. Finally, the 
= Pl,t + PSell
Load
t + PB+
t ∀t ∈ T (13)
amount of surplus/shortage power is updated after performing
l∈L
internal trading among MGs, as shown in (4), (5).
PSell
t = PtSell_Grid + PSell_CBESS
t ∀t ∈ T (14)

N 
N Buy
Pt = Pt
Buy_Grid Buy_CBESS
+ Pt ∀t ∈ T (15)
Psurplus,t = Psurplus,n,t − sell,n,t ∀t ∈ T
Pint (4)
n=1 n=1 The constraints related to BESS are given by (16)-(20).

N N Constraints (16), (17) show the maximum charg-
Pshortage,t = Pshortage,n,t − buy,n,t ∀t ∈ T
Pint (5) ing/discharging power of the BESS considering the ramp
n=1 n=1 rates. SOC is updated by (18) after charging/discharging
After solving the internal trading problem, the remaining power at each interval of time. Equation (19) shows that the
surplus/shortage power can be traded with the CBESS and the value of SOC is set by initial SOC at the first interval of
utility grid. Thus, the problem becomes same with the single time (t = 1). The operation bounds of BESS are enforced
MG problem and can be solved by using the proposed method. by (20).
The optimal model for dynamic internal trading prices in  
Cap   1
0 ≤ PB+ ≤ min PB · SOCmax
B
− SOCt−1
B
· , PB+ ∀t ∈ T
MMG system is a suitable future extension of this study. In t
1 − LB+ Ramp,max
the next section, a detailed mathematical model for MG system (16)
 
are presented. Cap  
0≤ PB−
t ≤ min PB · SOCt−1
B
− SOCmin
B
· (1 − LB− ), PB−
Ramp,max ∀t ∈ T

(17)
 
G. Mathematical Model for Operation of MG 1 1
SOCtB = B
SOCt−1 − · · PB−
t − Pt · (1 − L )
B+ B+
∀t ∈ T (18)
In this section, a mixed integer linear programing (MILP)- PB
Cap 1 − LB−
based formulation of MG system is presented for both B
SOCt−1 = SOCini
B
if t = 1 (19)
grid-connected and islanded modes. In grid-connected mode, B
SOCmin ≤ SOCtB ≤ B
SOCmax ∀t ∈ T (20)
the objective function (6) is to minimize the total operation
cost associated with the fuel cost, start-up/shut-down cost of In centralized approach for CBESS, a battery energy man-
CDGs, and cost/benefit of purchasing/selling power from/to agement system (BEMS) is developed for maximizing its
the utility grid, as shown in equation (6). profit by optimal charging/discharging decisions. The objective
⎧   ⎫ function is expressed by (21).
⎪
⎨ CiCDG · PCDG + yi,t · CiSU + zi,t · CiSD ⎪
⎬


i,t
i∈I t∈T  
Min   (6) CB− CB+
· PRt − PGrid,t · PRt
Sell Buy
⎪
⎩ +
Buy
PRt · Pt
Buy
− PRSell
t · PSell
t
⎪
⎭
Max Pt (21)
t∈T t∈T t∈T

Authorized licensed use limited to: Motilal Nehru National Institute of Technology. Downloaded on October 04,2020 at 07:42:10 UTC from IEEE Xplore. Restrictions apply.
 464 IEEE TRANSACTIONS ON SMART GRID, VOL. 11, NO. 1, JANUARY 2020

In islanded mode, the system is disconnected from the

utility grid. MG system can only trade its surplus/shortage
power with CBESS. In some peak load intervals, MG and
CBESS could not fulfill the power demand in the system.
Therefore, load shedding could be performed to keep the
power balance in the system. In order to reduce the load
shedding amount, MG-EMS performs optimization for min-
imizing both the total operation cost and the load shedding
amount. The cost objective function is changed to (22) with
generation cost and penalty cost for load shedding. The power
balance of power sources and power demands is given by (23)
for islanded mode. Additionally, the objective function (22)
is also constrained by (7)-(12) and (16)-(20). Finally, the
maximum surplus power is determined by (24) for each
interval.

 
Min CiCDG · PCDG
i,t + yi,t · CiSU + zi,t · CiSD Fig. 5. Test MG system: based on the CERTS MG test-bed [33].
i∈I t∈T

 pen

+ Ct · PShort
t (22)
t∈T
shortage power.
   
t + Pt
PPV + i,t + Pt = + PB+
t + Pt
WT
PDG B−
PLoad Sur
Cap
l,t 0 ≤ PCB+
t ≤ min PCB · SOCmax
CB
−SOCt−1
CB
i∈I l∈L

− PShort ∀t ∈ T (23) 1 CB+
t
  × , PSur
, P ∀t ∈ T (28)
  1−LCB+ t Ramp,max


PSur = PPV + PWT + PDG + PB− − PLoad − PShort Cap  
max,t t t i,max t l,t t P CB −SOCCB · (1−LCB− ),
· SOCt−1
i∈I 0 ≤ PCB−
t ≤ min CB min ∀t ∈ T.
l∈L
PShort
t , PCB−
Ramp,max
− PB+
t ∀t ∈ T (24) (29)
Similarly, in centralized approach, the objective of BEMS
is to minimize the amount of load shedding in MG system. H. Scheduling and Optimal Power Flow
The objective function is presented by (25).

 The operation of a power system is generally carried out in
 two steps, i.e., optimal scheduling and optimal power flow. In
Min Short
Pt · pent
Short
(25) the first step, the commitment statuses of the components are
t∈T
determined including the power exchange with the main grid.
In this paper, a distributed operation strategy using DDQN In the second step, the status of components is shared with the
is proposed for optimal operation of CBESS. In both opera- optimal power flow problem and is allowed to deviate minutely
tion modes, the objective of CBESS is to maximize its profit from the set values. In this step, the network losses are deter-
by trading power with the MG and the utility grid. However, mined and are sent back to the first step problem. Scheduling
the charging/discharging bounds of CBESS is different in problem is again solved including the network losses. This
each operation mode. In grid-connected mode, the amount of process is repeated in an iterative way till the convergence is
charging/discharging for CBESS is totally dependent on the achieved.
market price signals. Maximum charging/discharging power However, MG is localized and small-scale power system and
of CBESS is given by (26), (27) considering the ramp rate the length of lines is short. Therefore, the resistance of the MG
constraints. lines between two buses is very small and the power line losses
  
0 ≤ PCB+
Cap
≤ min PCB · SOCmax
CB
− SOCt−1
CB
·
1
, PCB+ ∀t ∈ T
are usually negligible [29], [30]. Therefore, power losses for
t
1 − LCB+ Ramp,max power flow are not considered in this study and the network
(26) constraints, e.g., constraints about voltages and currents are
  
0≤
Cap
≤ min PCB · SOCt−1 − SOCmin · (1 − LCB− ), PCB− assumed to be fulfilled [31], [32]. The detailed analysis on
PCB− Ramp,max ∀t ∈ T
CB CB
t
the operation of CBESS considering the optimal power flow
(27) and network constraints could be a suitable extension of this
On the other hand, in islanded mode, the CBESS can only study for a larger system, e.g., multi-microgrid system.
charge surplus power from MG and discharge to MG during
intervals having shortage power. Constraints (28), (29) show
III. N UMERICAL R ESULTS
the maximum bounds for charging/discharging amount at each
interval of time considering the ramp rates. These constraints A. Input Data
also ensure that charging is only possible when MG has sur- Fig. 5 shows a test MG system based on the CERTS MG
plus power, while discharging is only possible when MG has test-bed [33] considering the uncertainty of environment, e.g.,

Authorized licensed use limited to: Motilal Nehru National Institute of Technology. Downloaded on October 04,2020 at 07:42:10 UTC from IEEE Xplore. Restrictions apply.
 BUI et al.: DOUBLE DEEP Q-LEARNING-BASED DISTRIBUTED OPERATION OF BATTERY ENERGY STORAGE SYSTEM CONSIDERING UNCERTAINTIES 465

TABLE I
T HE D ETAIL PARAMETERS OF BESS, CBESS, AND CDG

the uncertainty in power of renewable resources, load demand

profiles, and market price signals.
The test MG system is comprised of three convertor based
sources and five load banks. In this study, the three convertor
based sources are a controllable distributed generator (CDG),
a battery energy storage system (BESS), and renewable dis-
tributed generators (RDGs). The detailed parameters of CDG,
BESS, and CBESS are tabulated in Table I.
The MG is interconnected with a CBESS and the util-
ity grid. The system can be operated in both grid-connected
and islanded modes. In this study, the operation bounds of
BESS and CBESS were chosen as [10%, 90%] to elongate
the lifespan of energy storage elements, same as [34]. The
analysis is conducted for a 24-hour scheduling horizon with
a time interval of 1 hour. The DDQN-based model is coded
in Python, which uses TensorFlow library [35]. The MILP-
base model is also code in Python and solved by using the Fig. 6. Input data: (a) Market price signals considering 5% of uncertainty,
MILP solver CPLEX 12.6 [36]. The market price signals, (b) load profile; (c) The output power of RDG.
total load amount, and total output of RDGs are shown in
Figs. 6(a) [37], 6(b) [38], and 6(c) [37], respectively. The detail
numerical results are shown in the following sections for both
grid-connected and islanded modes.

B. Operation of the System in Grid-Connected Mode

This section presents the operation of MG and CBESS in
grid-connected mode. MG-EMS performs optimization to min-
imize the total operation cost of the MG system. The operation
of all components in the MG are shown in Fig. 7. The amount
of trading power is determined based on the amount of sur-
plus/shortage power in the MG system. It can be observed
that the MG always buys power with the low market price
signals and sells power with high market price signals. The
BESS also charges power during low price intervals and dis-
charges power to fulfill load amount or sell to the external
system during peak price intervals.
On the other side, CBESS is operated based on the DDQN
method. The CBESS is trained with 80000 episodes in grid-
connected mode considering two cases: without uncertainty
(case 1) and with ±5% uncertainty of market price signals
(case 2). In order to show the performance of the proposed
method, the CBESS model is also trained with actor-critic RL
method considering the same training time and uncertainty of
market price signals. After training, the optimal weight param-
eters of DNNs are used to estimate the actions of CBESS and
evaluate the quality of that action. The convergence of total
reward for CBESS using the proposed method is shown in
Fig. 8(a). It can be observed that the action of CBESS is better
when the number of episode increases. The CBESS keeps on
Fig. 7. Operation of MG in grid-connected mode.
choosing random actions with a small probability of epsilon.
Therefore, the value of the total reward is also fluctuated. In
order to show the total reward clearer, the total rewards dur-
ing last 100 episodes of training time is shown in Fig. 8(b) for
both actor-critic RL and DDQN methods. It can be observed
that the total reward of CBESS converges to optimal points
for both learning methods. However, the CBESS operation
is easily trapped in a local optima by using actor-critic RL
because the optimization of actor is a gradient-based method.
Therefore, in this study, a detailed operation of CBESS is only
analyzed by using the DDQN method. It can be observed that
the total rewards of CBESS after 80000 episodes converge
at an optimal value in cases without uncertainty. With ±5%
uncertainty of price signals, the total rewards of CBESS also
converge to optimal values based on the real values of the
market price signals in between upper and lower bounds. In
this case, the total rewards could be lower or higher than the

Authorized licensed use limited to: Motilal Nehru National Institute of Technology. Downloaded on October 04,2020 at 07:42:10 UTC from IEEE Xplore. Restrictions apply.
 466
Fig. 8. (a) Total reward for CBESS during 80000 episodes; (b) Total rewards
for CBESS during last 100 episodes.
Fig. 9. Accuracy of CBESS after 80000 episodes.
total rewards in case 1 due to the change in market price
signals (±5%).
In order to analyze the accuracy of the proposed method, the
accuracies in both cases are summarized in Fig. 9 after training
time. It can be observed that the accuracies in both cases are
acceptable (≈1) for operation of CBESS. In case 2, since the
price signals are randomly changed every step in an episode,
the state space is much bigger than case 1. Therefore, the train-
ing accuracy in case 1 converges faster than case 2. In order to
guarantee the acceptable accuracy, the CBESS could be trained
during few hours. However, the training process is off-line
training based on forecasted data for a day-ahead scheduling
of CBESS. The CBESS can use the Target-network to esti-
mate the Q-values while the Q-network can be continuously
trained to get more accurate decisions.
By using the optimal DNN, the optimal operation of CBESS
is shown in Fig. 10 by using the centralized-based approach
and the proposed method. By using the proposed method, the
optimal weight parameters of DNN are saved at the end of the
training process. Then the optimal DNN is used to estimate
the optimal actions of the CBESS. It can be observed that
the CBESS can optimize its charging/discharging decisions
with the proposed method, similar to centralized approach. The
CBESS is charged during the off-peak price intervals (intervals
2, 4, 5, and 19) and is discharged during peak price intervals
(intervals 12-15, and 20). Based on the optimal operation
Authorized licensed use limited to: Motilal Nehru National Institute of Technology. Downloaded on October 04,2020 at 07:42:10 UTC from IEEE Xplore. Restrictions apply.
IEEE TRANSACTIONS ON SMART GRID, VOL. 11, NO. 1, JANUARY 2020
Fig. 10. The operation of CBESS after 80000 episodes.
Fig. 11. Market price signals: a test case.
Fig. 12. Optimal operation of CBESS with the uncertainty of market price
signals.
of MG’s components and the optimal charging/discharging
decisions of CBESS, the total operation cost of MG and
total profit of CBESS are summarized in Table II for grid-
connected mode. The positive value represents the profit, while
the negative value represents the operation cost.
Moreover, in order to show the performance of the proposed
method for handling the uncertainty, a comparison between
a conventional EMS (planning without uncertainty) and the
proposed method considering the uncertainty of market price
signals in grid-connected mode is presented. We assumed that
the real-time data for market price signals, as shown in Fig. 11.
If the CBESS is operated following day-ahead scheduling, the
total profit could be reduced because of the error between the
forecasted and the real-time market price signals. For instance,
the CBESS could decide to charge power in a cheap interval
(interval 5) based on the forecasted data. However, the price
of this interval could not be actually cheap. By using the
proposed method, the CBESS can adjust its schedule based
on the real-time data, as shown in Fig. 12. It can be observed
that the CBESS changes the charging/discharging decisions.
Therefore, the total profit is kept at higher value. The total
profit for CBESS by following the day-ahead schedule is
15280 KRW. On the other hand, the total profit is 15720 KRW
by using the DDQN.
 BUI et al.: DOUBLE DEEP Q-LEARNING-BASED DISTRIBUTED OPERATION OF BATTERY ENERGY STORAGE SYSTEM CONSIDERING UNCERTAINTIES
TABLE II
T OTAL O PERATION C OST OF MG AND T OTAL P ROFIT
OF CBESS IN G RID -C ONNECTED M ODE
Fig. 13. Optimal operation of MG in islanded mode.
Fig. 14. Surplus/shortage power of MG considering ±10% uncertainty.
C. Operation of the System in Islanded Mode
In islanded mode, MG cannot trade power with the utility
grid. CBESS plays an important role in the operation of MG
for reducing load shedding amount. CBESS is used to shift
the surplus power to other intervals having shortage power. It
means CBESS charges the surplus power and discharges for
fulfilling the shortage power. The operation of all components
in MG system is shown in Fig. 13. The MG-EMS controls
these components to fulfill the load demand and reduces the
load shedding amount during some peak load intervals. The
detailed amount of surplus/shortage power of MG is shown in
Fig. 14 in each interval of time. The trading price signals are
shown in Fig. 15, the price is decided based the percentage of
critical load in each interval. In islanded mode, we assume that
the uncertainties of trading price signals and surplus/shortage
amount are ±5% and ±10%, respectively.
Due to increase in the number of uncertainty parameters,
the training episodes are also increased to improve the accu-
racy of training model. In this study, the model is trained for
100000 episodes for two cases: without uncertainty (case 1)
and with uncertainty (case 2). Fig. 16 shows the total reward
of CBESS during the last 100 episodes of the training time.
The total reward of CBESS after 100000 episodes converge at
an optimal value in case 1. On the other hand, in case 2, the
Authorized licensed use limited to: Motilal Nehru National Institute of Technology. Downloaded on October 04,2020 at 07:42:10 UTC from IEEE Xplore. Restrictions apply.
467
Fig. 15. Trading price considering ±5% of uncertainty.
Fig. 16. Total reward for CBESS after 100000 episodes.
Fig. 17. Accuracy for CBESS after 100000 episodes.
total reward of CBESS can also converge at optimal values
based on the real values of the trading price signals, surplus,
and shortage amount. In this case, the total reward could be
lower or higher than the total reward in case 1 due to the
changes in market price signals (±5%) and surplus/shortage
power (±10%). The accuracies in both cases are also summa-
rized in Fig. 17 after training time. It can be observed that
the accuracies in both cases are acceptable (≈1) for operation
of CBESS. In case 2, since the trading price signals and sur-
plus/shortage amount are randomly changed in each step in
between the upper and the lower bounds, the state space is
much bigger than case 1. Therefore, the training accuracy in
case 1 converges faster than case 2.
By using the optimal DDNs, the operation of CBESS is
given in Fig. 18 by the proposed method and the central-
ized approach. It can be observed that the CBESS is operated
in an optimal way by both methods. The CBESS is decided
to charge the surplus power in lower price intervals and dis-
charged to fulfill the shortage power in high price intervals.
The load shedding amount of MG system can also be signifi-
cantly reduced by using the CBESS, as shown in Fig. 19. In
case of without the CBESS, the load shedding is performed
during almost peak load intervals (intervals 7-10 and 15-19).
With the CBESS, the operation of CBESS is determined by
 468
Fig. 18. The operation of CBESS after 100000 episodes.
Fig. 19. Load shedding amount in MG system.
TABLE III
T OTAL O PERATION C OST OF MG AND T OTAL
P ROFIT OF CBESS IN I SLANDED M ODE
using the optimal DNN, the amount of load shedding is only
high in intervals 7, 15, and 19.
Similar to the grid-connected mode, thanks to the optimal
DDN, the CBESS can also adjust its decisions in this opera-
tion mode based on the real-time data of the amount of surplus
/shortage power. It leads to reduction in the load shedding
amount in MG compared with other conventional methods
(i.e., MILP, Q-learning) and improve the system reliability.
Finally, the total operation cost of MG and total profit of
CBESS are also summarized in Table III for islanded mode.
D. Operation of the CBESS System in a 48-Hour Time Span
In order to show the effectiveness of the proposed method
for a longer time span, an extension of this study is presented
for a 48-hour scheduling of CBESS using DDQN. The opera-
tion of CBESS is analyzed in both grid-connected and islanded
modes. The results are also compared with the day-ahead
scheduling (24-hour time span). The market price signals are
shown in Fig. 20(a) for 48 hours. In grid-connected mode,
the objective is to maximize the total profit of the CBESS
during the time scheduling. Therefore, the CBESS is always
discharged to the minimum operation bound at the end of the
day for 24-hour time span. However, in a 48-hour time span,
Authorized licensed use limited to: Motilal Nehru National Institute of Technology. Downloaded on October 04,2020 at 07:42:10 UTC from IEEE Xplore. Restrictions apply.
IEEE TRANSACTIONS ON SMART GRID, VOL. 11, NO. 1, JANUARY 2020
Fig. 20. (a) Market price signals; (b) Optimal operation of CBESS during
48-hour scheduling horizon.
the CBESS is trained to maximize its total profit during that
scheduling horizon, as shown in Fig. 20(b). It can be observed
that the CBESS charges power at the end of the first day with
lower price and uses for the next day to reduce the operation
cost.
Similarly, in islanded mode, the CBESS is trained for
minimizing the load shedding during the 48-hour scheduling
horizon. By using the proposed method, the amount of surplus
power during the last intervals of the first day can be charged
to the CBESS to use for fulfilling the shortage power in the
next day. Therefore, the load shedding amount could also be
reduced with a longer time scheduling.
The results for longer scheduling horizon (48 hours) are
better than those of the shorter scheduling horizon (24 hours).
However, the results are also dependent on the forecasted data
and the uncertainties in the system. The accuracy of forecasted
data could be reduced for considering a longer time period.
It leads to increase in the uncertainties in the system. That
could make worse situation for the system reliability and oper-
ation cost, as mentioned in the previous sections. Moreover,
the complexity of the system also increases in terms of training
process whenever any event occurs in the system. Therefore,
a day-ahead scheduling (24-hour time span) is proposed to
preserve the higher reliability of the system.
IV. C ONCLUSION
In this paper, a distributed operation strategy has been
developed for managing the operation of a CBESS in an
MG system using double deep Q-learning method. The MG
system is comprised of an MG and a CBESS, where an MG-
EMS was used for optimizing the operation of the MG. In
contrast to Q-learning, the proposed operation strategy was
capable of dealing with uncertainties in the system in both
grid-connected and islanded modes. Moreover, by decoupling
the selection and evaluation of an action, the proposed method
reduced the overestimation. Finally, a comparison between the
proposed strategy and other methods has been presented for
showing the effectiveness of the proposed method. The CBESS
 BUI et al.: DOUBLE DEEP Q-LEARNING-BASED DISTRIBUTED OPERATION OF BATTERY ENERGY STORAGE SYSTEM CONSIDERING UNCERTAINTIES
can optimally work with the proposed operation strategy with
a large number of episodes. The CBESS accurately determined
the optimal operation like the centralized-based method in both
grid-connected and islanded modes.
R EFERENCES
[1] C. Chen, S. Duan, T. Cai, B. Liu, and G. Hu, “Smart energy management
system for optimal microgrid economic operation,” IET Renew. Power
Gener., vol. 5, no. 3, pp. 258–267, May 2011.
[2] Q. Jiang, M. Xue, and G. Geng, “Energy management of microgrid
in grid-connected and stand-alone modes,” IEEE Trans. Power Syst.,
vol. 28, no. 3, pp. 3380–3389, Aug. 2013.
[3] F. Katiraei, R. Iravani, N. Hatziargyriou, and A. Dimeas, “Microgrid
management,” IEEE Power Energy Mag., vol. 6, no. 3, pp. 54–65,
May/Jun. 2008.
[4] W. Su and J. Wang, “Energy management systems in microgrid opera-
tions,” Elect. J., vol. 25, no. 8, pp. 45–60, Oct. 2012.
[5] D. E. Olivares, C. A. Cañizares, and M. Kazerani, “A centralized energy
management system for isolated microgrids,” IEEE Trans. Smart Grid,
vol. 5, no. 4, pp. 1864–1875, Jul. 2014.
[6] L. Meng, E. R. Sanseverino, A. Luna, T. Dragicevic, J. C. Vasquez, and
J. M. Guerrero, “Microgrid supervisory controllers and energy manage-
ment systems: A literature review,” Renew. Sustain. Energy Rev., vol. 60,
pp. 1263–1273, Jul. 2016.
[7] V.-H. Bui, A. Hussain, and H.-M. Kim, “A multiagent-based hierarchical
energy management strategy for multi-microgrids considering adjustable
power and demand response,” IEEE Trans. Smart Grid, vol. 9, no. 2,
pp. 1323–1333, Mar. 2018.
[8] V.-H. Bui, A. Hussain, and H.-M. Kim, “Diffusion strategy-based dis-
tributed operation of microgrids using multiagent system,” Energies,
vol. 10, no. 7, p. 903, Jul. 2017.
[9] Y. Xu and Z. Li, “Distributed optimal resource management based on
the consensus algorithm in a microgrid,” IEEE Trans. Ind. Electron.,
vol. 62, no. 4, pp. 2584–2592, Apr. 2015.
[10] E. C. Kara, M. Berges, B. Krogh, and S. Kar, “Using smart devices
for system-level management and control in the smart grid: A rein-
forcement learning framework,” in Proc. IEEE Smart Grid Commun.
(SmartGridComm), Nov. 2012, pp. 85–90.
[11] A. L. Dimeas and N. D. Hatziargyriou, “Multi-agent reinforcement
learning for microgrids,” in Proc. IEEE Power Energy Soc. Gen.
Meeting, Jul. 2010, pp. 1–8.
[12] G. K. Venayagamoorthy, R. K. Sharma, P. K. Gautam, and A. Ahmadi,
“Dynamic energy management system for a smart microgrid,” IEEE
Trans. Neural Netw. Learn. Syst., vol. 27, no. 8, pp. 1643–1656,
Aug. 2016.
[13] R. S. Sutton and A. G. Barto, Reinforcement Learning: An Introduction.
Cambridge, MA, USA: MIT Press, Oct. 2018.
[14] L. Panait and S. Luke, “Cooperative multi-agent learning: The state of
the art,” Auton. Agents Multi Agent Syst., vol. 11, no. 3, pp. 387–434,
Nov. 2005.
[15] E. Kuznetsova, Y.-F. Li, C. Ruiz, E. Zio, G. Ault, and K. Bell,
“Reinforcement learning for microgrid energy management,” Energy,
vol. 59, pp. 133–146, Sep. 2013.
[16] F.-D. Li, M. Wu, Y. He, and X. Chen, “Optimal control in microgrid
using multi-agent reinforcement learning,” ISA Trans., vol. 51, no. 6,
pp. 743–751, Nov. 2012.
[17] S. Dutta, Reinforcement Learning With TensorFlow: A Beginner’s Guide
to Designing Self-Learning Systems With TensorFlow and OpenAI Gym.
Birmingham, U.K.: Packt, Apr. 2018, pp. 1–336.
[18] V.-H. Bui, A. Hussain, and H.-M. Kim, “Q-learning-based operation
strategy for community battery energy storage system (CBESS) in
microgrid system,” Energies, vol. 12, no. 9, pp. 1789–1806, May 2019.
[19] V. Mnih et al., “Playing Atari with deep reinforcement learn-
ing,” arXiv preprint arXiv:1312.5602, Dec. 2013. [Online]. Available:
https://arxiv.org/pdf/1312.5602.pdf
[20] E. Mocanu et al., “On-line building energy optimization using deep
reinforcement learning,” IEEE Trans. Smart Grid, vol. 10, no. 4,
pp. 3698–3708, Jul. 2019.
[21] V. Mnih et al., “Asynchronous methods for deep reinforcement learning,”
in Proc. Int. Conf. Mach. Learn., Jun. 2016, pp. 1928–1937.
[22] H. Van Hasselt, A. Guez, and D. Silver, “Deep reinforcement learning
with double Q-learning,” in Proc. AAAI, vol. 2, Feb. 2016, p. 5.
[23] M. Wiering and M. O. Ma, Reinforcement Learning State-of-the-Art.
Berlin, Germany: Springer-Verlag, 2012.
[24] Y. Mohan, S. G. Ponnambalam, and J. I. Inayat-Hussain, “A comparative
study of policies in Q-learning for foraging tasks,” in Proc. IEEE Nat.
Biol. Inspired Comput., Dec. 2009, pp. 134–139.
Authorized licensed use limited to: Motilal Nehru National Institute of Technology. Downloaded on October 04,2020 at 07:42:10 UTC from IEEE Xplore. Restrictions apply.
469
[25] D. P. Bertsekas, Dynamic Programming and Optimal Control, vol. 1.
Belmont, MA, USA: Athena Sci., Jan. 1995.
[26] E. Theodorou, J. Buchli, and S. Schaal, “A generalized path integral con-
trol approach to reinforcement learning,” J. Mach. Learn. Res., vol. 11,
pp. 3137–3181, Nov. 2010.
[27] V. R. Konda and J. N. Tsitsiklis, “On actor-critic algorithms,” SIAM J.
Control Optim., vol. 42, no. 4, pp. 1143–1166, Apr. 2003.
[28] S. A. Arefifar, M. Ordonez, and Y. A.-R. I. Mohamed, “Energy manage-
ment in multi-microgrid systems—Development and assessment,” IEEE
Trans. Power Syst., vol. 32, no. 2, pp. 910–922, Mar. 2017.
[29] A. Hussain, V.-H. Bui, and H.-M. Kim, “Robust optimal operation
of AC/DC hybrid microgrids under market price uncertainties,” IEEE
Access, vol. 6, pp. 2654–2667, 2018.
[30] T. Dragičević, J. M. Guerrero, J. C. Vasquez, and D. Škrlec, “Supervisory
control of an adaptive-droop regulated DC microgrid with battery man-
agement capability,” IEEE Trans. Power Electron., vol. 29, no. 2,
pp. 695–706, Feb. 2014.
[31] S. Parhizi, A. Khodaei, and M. Shahidehpour, “Market-based versus
price-based microgrid optimal scheduling,” IEEE Trans. Smart Grid,
vol. 9, no. 2, pp. 615–623, Mar. 2018.
[32] A. Khodaei, “Microgrid optimal scheduling with multi-period islanding
constraints,” IEEE Trans. Power Syst., vol. 29, no. 3, pp. 1383–1392,
May 2014.
[33] R. Bayindir, E. Hossain, E. Kabalci, and K. M. M. Billah, “Investigation
on North American microgrid facility,” Int. J. Renew. Energy Res., vol. 5,
no. 2, pp. 558–574, Jun. 2015.
[34] X. Li, D. Hui, and X. Lai, “Battery energy storage station (BESS)-based
smoothing control of photovoltaic (PV) and wind power generation
fluctuations,” IEEE Trans. Sustain. Energy, vol. 4, no. 2, pp. 464–473,
Apr. 2013.
[35] M. Abadi et al., “TensorFlow: A system for large-scale machine
learning,” in Proc. OSDI, vol. 16, Nov. 2016, pp. 265–283.
[36] IBM ILOG CPLEX V12.6 User’s Manual for CPLEX 2015, CPLEX
Division, ILOG, Incline Village, NV, USA, 2015.
[37] V.-H. Bui, A. Hussain, and H.-M. Kim, “Optimal operation of
microgrids considering auto-configuration function using multiagent
system,” Energies, vol. 10, no. 10, pp. 1484–1500, Sep. 2017.
[38] A. Hussain, V.-H. Bui, and H.-M. Kim, “A resilient and privacy-
preserving energy management strategy for networked microgrids,”
IEEE Trans. Smart Grid, vol. 9, no. 3, pp. 2127–2139, May 2018.
Van-Hai Bui (S’16) received the B.S. degree in
electrical engineering from the Hanoi University
of Science and Technology, Vietnam, in 2013.
He is currently pursuing the combined master’s
and Ph.D. degrees with the Department of
Electrical Engineering, Incheon National University,
South Korea. His research interests include
microgrid operation and energy management
system.
Akhtar Hussain (S’14) received the B.E. degree
in telecommunications from the National University
of Sciences and Technology, Pakistan, in 2011,
and the M.S. degree in electrical engineering from
Myongji University, Yongin, South Korea, in 2014.
He is currently pursuing the Ph.D. degree with
Incheon National University, South Korea. He was
an Associate Engineer with SANION; IEDs devel-
opment company, South Korea, from 2014 to 2015.
His research interests are distribution automation and
protection, smart grids, and microgrid optimization.
Hak-Man Kim (SM’15) received the first
Ph.D. degree in electrical engineering from
Sungkyunkwan University, South Korea, in 1998,
and the second Ph.D. degree in information
sciences from Tohoku University, Japan, in 2011.
He was with Korea Electrotechnology Research
Institute, South Korea, from 1996 to 2008. He
is currently a Professor with the Department of
Electrical Engineering, Incheon National University,
South Korea. His research interests include
microgrid operation and control and dc power
systems.