See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/261380458

Reinforcement learning approach to dynamic activation of base station

resources in wireless networks

Conference Paper · September 2013

DOI: 10.1109/PIMRC.2013.6666710

CITATIONS READS
8 127

2 authors:

Peng-Yong Kong Dorin Panaitopol

Khalifa University Thales Communications & Security (TCS)
111 PUBLICATIONS   852 CITATIONS    28 PUBLICATIONS   99 CITATIONS   

SEE PROFILE SEE PROFILE

Some of the authors of this publication are also working on these related projects:

cognitive radio View project

All content following this page was uploaded by Peng-Yong Kong on 29 October 2014.

The user has requested enhancement of the downloaded file.

 Reinforcement Learning Approach to Dynamic
Activation of Base Station Resources in Wireless
Networks
Peng-Yong Konga and Dorin Panaitopolb
a KhalifaUniversity of Science, Technology & Research (KUSTAR), Abu Dhabi, United Arab Emirates.
b NEC Technologies (UK), FMDC Lab., ComTech Department, Nanterre, France.
E-mail: pengyong.kong@kustar.ac.ae, dorin.panaitopol@nectech.fr
Abstract—Recently, the issue of energy efficiency in wireless
networks has attracted much research attention due to the
growing concern on global warming and operator’s profitability.
We focus on energy efficiency of base stations because they
account for 80% of total energy consumed in a wireless network.
In this paper, we intend to reduce energy consumption of a base
station by dynamically activating and deactivating the modular
resources at the base station depending on the instantaneous
network traffic. We propose an online reinforcement learning
algorithm that will continuously adapt to the changing network
traffic in deciding which action to take to maximize energy saving.
As an online algorithm, the proposed scheme does not require
a separate training phase and can be deployed immediately.
Simulation results have confirmed that the proposed algorithm
can achieve more than 50% energy saving without compromising
network service quality which is measured in terms of user
blocking probability.
Index Terms—Green wireless networks, energy efficient base
station, reinforcement learning, online Q-Learning.
I. I NTRODUCTION
The increasingly popular term “green wireless networks”
refers to technological advanced wireless networks with im-
proved energy efficiency [1]. We are interested in improving
the energy efficiency to reduce carbon emissions and to lower
energy cost of the network operator ([2], [3]). Carbon emission
is an important environmental issue because increasing release
of carbon directly into the atmosphere is perceived as the cause
of global warming crisis. In addition to this environmental
aspect, escalating energy cost has etched into the profitability
of wireless network operators.
In wireless cellular network, there are various existing
efforts to improve energy efficiency at the base stations be-
cause they collectively account for about 80% of the total
energy consumption. These efforts can be broadly classified
into two categories, namely (a) Improve energy efficiency of
the base station itself, and (b) Reduce the required number
of base stations for each telecommunication network. In the
first category, the efforts involve controlling the transmission
power more optimally through parameter optimization after
taking into account coverage and capacity requirements [4],
or re-designing the base stations by using equipments and
components that are more energy efficient [5]. According
to [5], power amplifier is the most critical component in a
base station as it accounts for almost 40% of total energy
consumption, and significant improvement in energy efficiency
can be achieved by using advanced power amplifier. In the
second category, the efforts include minimizing the number
of base stations deployed by using a higher density of low
power micro and pico base stations [6]. These works in
both the categories consider the telecommunication network
to carry a specific fixed volume of traffic, which is generally
a representation of the peak hours scenario. In reality, network
traffic is dynamic as it changes temporally from time to time.
For instance, network traffic is usually high during office hours
when many business activities are carried out, and the network
traffic drops to a minimal level at night when most people are
asleep.
It is not energy efficient to maintain base stations to be fully
functional as at the peak hours while the actual network traffic
is at its minimal. In [7], dynamic network traffic characteristics
are exploited in reducing energy consumption by shutting
down some base stations during a low traffic period, or
by controlling the cell size depending on the traffic load.
However, these existing works assume each base station is
a single entity which must be controlled as a whole unit.
We envisage that the future green base stations will have
their resources organized as a collection of modular units
each with its own energy consumption profile. These modular
resource units can be radios, baseband processors, feeders,
power amplifiers, air-conditioners, etc. This modular model
is similar to the system model that has been adopted by [8].
With the modular base station, this paper intends to exploit
the dynamic nature of network traffic in reducing energy
consumption by dynamically activating and deactivating the
resource units. We propose a reinforcement learning algorithm
for the base station such that it can continuously adapt to
the ever-changing network traffic in deciding either to turn
on an additional module, to turn off an already activated
module or to maintain status quo. Reinforcement learning is a
machine learning technique that helps an agent to decide which
action to take in a given environment so as to maximize some
notions of cumulative reward [9]. In our context, the actions
are what to do with the activation of modular resource units,
the environment is about the time-varying network traffic, and
the reward is related to the amount of energy saved. To realize
reinforcement learning, we we have designed an online Q-
Learning algorithm, which does not require a separate training
phase before deployment. The rest of this paper is organized as
follows. We describe the system model in Section II. In Section
III, we propose the online Q-Learning algorithm. Section IV
presents and discusses the evaluation results. This paper ends
with concluding remarks in Section V.
II. S YSTEM M ODEL
We assume a discrete time model for the wireless network
where the time domain is divided into repetitive time slots of a
fixed duration T as illustrated in Fig. 1. In the model, system
parameters change their values only at the beginning of a time
slot.
Fig. 1. System model for dynamic activation of modular resource units at
the base station.
We consider a base station with a pool of modular resources,
such as transmitters for GSM, carriers for 3G, HSDPA and
LTE, relays for IEEE 802.16j, or pico-cells for a very dense
network. These modular resources can be activated and deac-
tivated separately from time to time similar to the assumption
adopted in [8]. Let m[n] be the number of activated resource
modules at time slot n. These activated resource modules
contributes to the total energy consumption of the base station.
The base station energy consumption Pbs [n] at time slot n is
determined as follows:
Pbs [n] = Pcnt + m[n] × Pr ,
where Pcnt is the base station’s fundamental energy consump-
tion within a time slot, and it is determined by the load
independent factors of all equipments. In the equation, Pr is
the energy consumption of each module.
Traffic load at a base station is measured in terms of the
number of active users. A user becomes active when it is
making an outgoing call or receiving an incoming call. When
the call ends, the user becomes inactive and is considered
departed from the system. Let u[n] be the number of active
users in a give time slot n. The value of u[n] depends on
u[n − 1], λ[n], which is the number of newly arrived users in
time slot n and µ[n], which is the number of newly departed
users in time slot n. As such,
u[n] = u[n − 1] + λ[n] − µ[n]. (2)
In our system model, λ[n] and µ[n] are random variables with
time-varying average values. For example, as illustrated in
Fig. 2, the average value for λ[n] is much higher at 2 PM
compared to that at 2 AM. This is a clear representation of
different network traffic conditions at different times of a day.
Fig. 2. Average number of new user arrivals in a time slot as a function
of time in a day. The horizontal axis indicates the hour with a 24-hour time
notation.
Service quality of the wireless cellular network is measured
in terms of user blocking probability. Let U be the maximum
number of users that can be supported by each resource
module. Then, a blocking occurs when a new user becomes
active, but there are already U × m[n] active users at the
base station. It is desirable to have a very low user blocking
probability, says 0.01 or lower at all times for a good network
service quality.
III. R EINFORCEMENT L EARNING A LGORITHM FOR
DYNAMIC R ESOURCE ACTIVATION
In this section, we propose a reinforcement learning algo-
rithm to dynamically control the number of activated resource
modules to reduce energy consumption without compromising
service quality. We first define a baseline scheme for perfor-
mance benchmark before describing the proposed algorithm.
A. Baseline scheme
The baseline scheme does not dynamically control the
(1) number of activated resource modules regardless of the actual
number of active users in the system. As such, m[n] = M ∀ n,
where M is the number of all available resource modules.
While the number of activated resource modules is fixed,
following (1) the energy consumption at the base station is
given below:
Pbs [n] = Pcnt + M × Pr . (3)
Notice that energy consumption for the baseline scheme is a
constant that does not change from time to time even in the
presence of dynamic load.
B. Reinforcement Learning Algorithm
Recall that a reinforcement learning algorithm helps the
base station in controlling the number of activated resource
modules given the current system state. Here, the system
state is defined by two parameters, namely the number of
active users and the number of activated resource modules.
Therefore, at time slot n, the system state is given as S[n] =
(u[n], m[n]).
At a given system state, there is a set of actions from
which the base station may choose one to take. These actions
include activating some additional resource modules, turning
off some currently activated modules, or simply do nothing.
Let an action a[n] taken in a time slot n be an integer
number such that m[n + 1] = m[n] + a[n]. Notice that
the action can be either a positive number or a negative
number. A negative a[n] gives the number of resource modules
to be deactivated. A positive a[n] indicates the number of
additional resource modules to be activated. When a[n] = 0,
there will be no change in the number of activated resource
modules. Consider a state S[n] = (u[n], m[n]), the set
of feasible actions is given as follows: A[n] = {−m[n],
−(m[n] + 1), · · · , −1, 0, +1, +2, · · · , +(M − m[n])}, where
a[n] ∈ A[n]. This set of feasible actions is formed after
observing that the base station cannot turn off more resource
modules than what are already activated, and it cannot also
turn on more resource modules than what are currently deac-
tivated. There are always M +1 feasible actions in A[n]. After
taking an action a[n] in state S[n] = (u[n], m[n]), the system
transits immediately to state S[n + 1] = (u[n + 1], m[n + 1] =
m[n] + a[n]).
The reinforcement learning algorithm aims to map for each
state, the best action such that the energy saving is maximized.
Here, the energy saving is calculated in relation to the baseline
scheme. As such, following (1) and (3), the energy saving is
given as follows:
Psaved [n] = (M − m[n]) × Pr . (4)
Reasonably, activating more resource modules will lead to a
smaller energy saving. Following (4), our goal is to maximize
the average energy saved per time slot as given below:
∑k
n=1 Psaved [n]
P̄saved = lim . (5)
k→∞ k
We further define a numerical reward function to assist the
base station in identifying the best action. In time slot n, the
reward of taking an action a[n] when the state is S[n] is given
as follows:
 Initialize Q-factors: In the initialization step, all Q-factors

M − m[n] − a[n] if u[n + 1] ≤
r(S[n], a[n]) = (m[n] + a[n]) × U (6) for state S and action a. In the literature, all Q(S, a) are


P otherwise.
As a result of taking the action, the system transitions to
state S[n + 1] and the number of activated resource modules
becomes m[n + 1] = m[n] + a[n] supporting not more than
m[n + 1] × U active users. In this new state, the number of
active users is u[n+1], which is a random variable. We do not
assume or require u[n + 1] to follow any distribution function
that is known a priori. The base station discovers the value
of u[n] by simply looking into system at time slot n. If the
number of active users does not exceed the supported limit,
the system is rewarded with the amount of energy saving as
much as M − m[n + 1]. If the number of active users is
more than what the activated resource modules can support,
user blocking occurs and the system is penalized with a large
negative reward which is P in (6). The penalty is large so that
there is no ambiguity in avoiding user blocking. In defining
the reward function, the assumption is that by always looking
for the highest reward, the base station can achieve the goal of
maximizing average energy saving per time slot while keeping
user blocking at a low level.
The reinforcement learning algorithm must discover which
action will yield the most reward by trying them and observing
their outcomes. In practice, the action taken in a time slot may
affect not only the immediate reward but also the next state
and through that, all the subsequent rewards. To implement
these processes of trying, observing and future looking, there
are various existing reinforcement learning techniques, such as
SMART, R-SMART and Q-Learning. We adopt Q-Learning
for its simplicity [10]. Compared to existing Q-Learning
algorithms that require a separate training phase before de-
ployment, we propose an online Q-Learning algorithm where
the base station does not have the benefit of a separate training
phase, and every action that is taken will incur a real reward.
Online Q-Learning is closer to a practical scenario where the
user traffic pattern is ever-changing, and there is no prior
training data.
In online Q-Learning, a quantity called Q-factor is used to
track the implied value of each state-action pair such that, for
a given state, the state-action pair with the highest Q-factor
indicates the best action. Fig. 3 shows the flow chart we use
in updating Q-factor and selecting action. In the figure, there
are four steps which are carried at the beginning of a time
slot. Notice that there is no dedicated training phase and there
is an infinite loop after initialization.
Fig. 3. Online Q-Learning for Dynamic Resource Activation.
are set to some initial values. Let Q(S, a) be the Q-factor
usually initialized to zero. We find that some state-action pairs
will lead to user blocking with very high probability. For
example, taking action to deactivate more resource modules
is not reasonable when the existing number of active users
has exceeded the supported limit. In other words, taking these
actions is likely to yield a negative reward. If we initialize
all Q-factors to zero as suggested in the literature, the Q-
Learning algorithm will learn about this eventually but the
learning process can take too long for our system with rapid
changing traffic condition. As such, we propose to initialize
the Q-factors as follows:
 IV. P ERFORMANCE E VALUATION

−1000 if u > (m + a) × U
Q(S = (u, m), a) = and a < 0 (7)


0 otherwise.
Update State: At the beginning of a time slot n, the system
advances its epoch and updates its current state. This is done
such that the new state coming out from the execution of the
selected action in the previous time slot n − 1 becomes the
current state.
Select and Take Action: In this step, ϵ-greedy approach is
used in selecting an action. Let ϵ be a very small positive real
number and ϵ ≤ 1. Then, the algorithm will select the action
with the highest Q-factor value only with probability 1 − ϵ.
With probability ϵ, the algorithm chooses equal-probably an
action from the remaining feasible actions. This is a trade-off
between exploration and exploitation, where a higher ϵ will
encourage more aggressive exploration for potentially better
but yet-to-be-known action for a given state. Once an action
a[n] is selected, it is taken immediately and the system state
also changes immediately. After executing the selected action
a[n] in time slot n, the algorithm identifies the new system
state S[n + 1] = (u[n + 1], m[n + 1]) by looking into the base
station for the actual number of active users and the number of
activated resource modules. With the information of the new
state, the reward r(S[n], a[n]) can be determined by (6).
Update Q-factors: With the current state S[n] and the new
state S[n + 1] identified, the Q-factors are updated as follows:
( 23
Q(S[n], a[n]) ← (1 − α)Q(S[n], a[n]) + α r(S[n], a[n])
) 22
+γ max Q(S[n + 1], a) , (8)
a∈A[n+1]
where α is the learning rate and γ is the discount factor. Both α
and γ are positive real numbers not larger than 1. The learning
rate affects how aggressive the algorithm is in adopting a
new reward value into its Q-factor. A higher learning rate
means the algorithm will adapt to a new environment faster.
In the literature, off-line Q-Learning in training phase requires
a diminishing α with increasing epoch. On the other hand,
online Q-Learning needs a non-decreasing α bounded away
from zero. For simplicity, we choose a fixed α with non-zero
value. The discount factor affects the presence of the sum of
all future rewards in the current time slot. When γ = 0.9, only
90% of the all the future reward earned through the next state
is considered in the current Q-factor value. A very small γ
implies the algorithm is not interested in the future because
they may be irrelevant.
The proposed online Q-Learning algorithm has a large
state space given by M × U 2 . A large state space lead to
slow learning and convergence in Q-factors. As such, we
further revise the state definition using quantization such that
S[n] = (⌈u[n]/N ⌉, m[u]), where N is a positive integer
number indicating how many active users can be grouped
together to reduce the number of all possible states.
We have performed extensive Monte Carlo simulations to
evaluate the proposed online Q-Learning algorithm. In the
simulation, each Q-Learning epoch is aligned with a time slot
and the time slot duration T = 60 seconds. The number of new
user arrival in a time slot n, i.e., λ[n] is a random integer num-
ber uniformly distributed within the range [β[n] − 5, β[n] + 5],
where β[n] is updated once every hour in the first second of
the hour. The updated value of β[n] is determined as follows:
( )
2π(960 + n)
β[n] = 20 + 8 sin , (9)
1440
where this expression tries to simulate the time-varying net-
work traffic as illustrated in Fig. 2. The number of user
departures in a time slot is a random integer number uniformly
distributed within the range [20, 30]. The following nominal
values of various simulation parameters are assumed unless
specified otherwise: M = 30, U = 50, N = 25, P = −1000,
ϵ = 0.05, γ = 0.1, and α = 0.5. Each simulation is executed
for 20,000 epochs and each simulation setting is repeated 5
times. For a setting, the average value of the 5 simulation
results is plotted as a single data point in the graphs.
26
Average energy saved (KJoule per time slot)
25
24
Learning rate = 0.1
21 Learning rate = 0.3
Learning rate = 0.5
Learning rate = 0.7
20
Learning rate = 0.9
0 5 10 15 20 25 30
Limit of action space
Fig. 4. Average energy saved (KJoule) per time slot with different learning
rates α, and limits of action space, x.
Fig. 4 shows the average energy saved per time slot at
different learning rates and limits of action space, assuming
each resource module consumes 1 unit of energy per time
slot. Limit of action space x is a constraint imposed on the
feasible action set A[n], such that A[n] ← A[n] ∩ {−x, −(x +
1), · · · , 0, +1, · · · , +(x − 1), +x}. The limit of action space
is introduced to reduce the number of feasible actions to speed
 0.0255
up online Q-Learning algorithm’s convergence. We notice that Learning rate = 0.3
Learning rate = 0.5
a higher learning rate leads to a higher energy saving. This is Learning rate = 0.7
0.025
because our network traffic changes rapidly from time to time,

Probability of user blocking

and a small learning rate cannot catch up with the change,
leading to less ideal action selections. At a sufficiently high 0.0245

learning rate, says α = 0.7, a larger action space results in a

lower energy saving. This is because a larger action space 0.024

implies a higher chance in selecting a lousy action in Q-

Learning’s exploration and more occurence of penalties when 0.0235

a bad action is selected. With a high learning rate and a small

action space, the proposed Q-Learning algorithm can achieve 0.023
0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9
a very high average energy saving. With M = 30, the energy Discount factor
saving is about 84% (25.2/30 × 100) for α = 0.9 and x = 1.
Fig. 7. User blocking probability with different learning rates α, and discount
27 factors, γ.
Average energy saved (KJoule per time slot)

Learning rate = 0.3

Learning rate = 0.5
26 Learning rate = 0.7

25
quality measured in terms of user blocking probability. In
24 general, the blocking probability can be kept low at around
23
0.025. Although this is not as low as 0.01 that we desire for,
the slight increase in blocking probability is probably the price
22
to pay for the very significant energy saving.
21
V. C ONCLUSION
20
We envisage the future green wireless base station to have
19
0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9
modular resource units that can be separately and dynamically
Discount factor activated or deactivated depending on the network traffic
load. We have proposed an online Q-Learning (reinforcement
Fig. 5. Average energy saved (KJoule) per time slot with different learning
rates α, and discount factors, γ.
learning) algorithm to perform the dynamic resource activation
in face of ever-changing network traffic. Simulation results
confirm that the proposed scheme can achieve significant
Fig. 5 shows the average energy saved per time slot at differ- energy saving as much as 80% without compromising network
ent learning rates and discount factors. The results confirm that service quality since the user blocking probability can be kept
energy saving can reach as high as 80% with proper settings. as low as 0.025.
It is desirable for the Q-Learning to have a high learning rate
and low discount factor, implying learning fast while looking R EFERENCES
not far into the future. [1] Ziaul Hasan, Hamidreza Boostanimehr and Vijay K. Bhargava, “Green
Cellular Networks: A Survey, Some Research Issues and Challenges”,
0.055 IEEE Communications Survey & Tutorials, Fourth Quarter 2011.
[2] “Energy Aware Radio and NeTwork TecHnologies (EARTH)”,
0.05
http://www.ict-earth.eu/.
[3] “Towards Real Energy-efficient Network Design (TREND)”,
Probability of user blocking

0.045
http://www.fp7-trend.eu/.
[4] Holger Claussen, Lester T. W. Ho and Florian Pivit, “Effects of Joint
Macrocell and Residential Picocell Deployment on The Network Energy
0.04
Efficiency”, IEEE PIMRC, September, 2008.
[5] Jyrki T. Louhi, “Energy Efficiency of Modern Cellular Base Stations”,
0.035
International Conference on Telecommunications Energy, pp. 475-476,
October 2007.
0.03 [6] Fred Richter, Albrecht J. Fehske and Gerhard P. Fettweis, “Energy
Learning rate = 0.1
Learning rate = 0.3
Efficiency Aspects of Base Station Deployment Strategies for Cellular
0.025 Learning rate = 0.5 Networks”, IEEE Vehicular Technology Conference, Septermber 2009.
Learning rate = 0.7 [7] Zhisheng Niu, Yiqun Wu, Jie Gong and Zexi Yang, “Cell Zooming
Learning rate = 0.9
0.02 for Cost-Efficient Green Cellular Networks”, IEEE Communications
0 5 10 15 20 25 30
Limit of action space Maganine, Vol. 48, No. 11, pp. 74-79, November 2010.
[8] Salah-Eddine Elayoubi, Louai Saker and Tijani Chahed, “Optimal Con-
Fig. 6. User blocking probability with different learning rates α, and limits trol for Base Station Sleep Mode in Energy Efficient Radio Access
of action space, x. Networks”, IEEE INFOCOM, pp. 106-110, April 2011.
[9] Richard S. Sutton and Andrew G. Barto, “Reinforcement Learning: An
Introduction”, The MIT Press, Cambridge, Massachusetts, United States
of America, 1998.
Fig. 6 and Fig. 7 show that the significant energy saving [10] Christopher J. C. H. Watkins and Peter Dayan, “Technical Notes: Q-
can be achieved without compromising the network service Learning”, Machine Learning, Vol. 8, pp. 279-292, 1992.

View publication stats