Digital Communications and Networks xxx (xxxx) xxx

Contents lists available at ScienceDirect

Digital Communications and Networks

journal homepage: www.keaipublishing.com/dcan

SMDP-based sleep policy for base stations in heterogeneous

cellular networks
Jing Wu a, Yun Li b, e, *, Hongcheng Zhuang c, Zhiwen Pan d, Guoyin Wang e, Yongju Xian a
a
College of Communication and Information Engineering, Chongqing University of Posts and Telecommunications, Chongqing, 400065, PR China
b
Chongqing Key Laboratory of Mobile Communications Technology, Chongqing University of Posts and Telecommunications, Chongqing, 400065, PR China
c
Huawei Technologies Co. Ltd., Shenzhen, 518129, PR China
d
Naitional Mobile Communications Research Laboratory, Southeast University, Nanjing, Jiangsu, 210096, PR China
e
Chongqing Key Laboratory of Computational Intelligence, Chongqing University of Posts and Telecommunications, Chongqing, 400065, PR China

A R T I C L E I N F O A B S T R A C T

Keywords: A dense heterogeneous cellular network can effectively increase the system capacity and enhance the network
Macro-femto heterogeneous network coverage. It is a key technology for the new generation of the mobile communication system. The dense
Base station sleep deployment of small base stations not only improves the quality of network service, but also brings about a
Energy consumption
significant increase in network energy consumption. This paper mainly studies the energy efficiency optimization
of the Macro-Femto heterogeneous cellular network. Considering the dynamic random changes of the access users
in the network, the sleep process of the Femto Base Stations (FBSs) is modeled as a Semi-Markov Decision Process
(SMDP) model in order to save the network energy consumption. And further, this paper gives the dynamic sleep
algorithm of the FBS based on the value iteration. The simulation results show that the proposed SMDP-based
adaptive sleep strategy of the FBS can effectively reduce the network energy consumption.

1. Introduction
The rapid growth of the number of mobile users and mobile services
has put a huge demand for the capacity of the cellular mobile commu-
nication system and the Quality of Service (QoS) of mobile users. There
are some works that focus on how to increase the QoS [1,6,8]. At the
same time, the rapid development of applications such as the Internet of
Things (IoT) and the virtual reality urgently requires the mobile
communication system to provide differentiated services. The use of a
heterogeneous cellular network structure with overlapping coverage of
the small base stations and a macro base station is an important way to
deal with the appeal issues, and it is also one of the key technologies for
the new generation mobile communication. With the increase in the size
of the mobile communication network, the energy consumption is also
growing with each passing day. The network energy efficiency has
become one of the important indicators of the network optimization.
In traditional cellular networks, many scholars have concerned
themselves with the issue of energy consumption. In Ref. [2,3], the au-
thors studied the use of green energy to power the base stations to reduce
the brown energy consumption, thereby reducing the network energy
consumption. In Ref. [4,5], the transmission power of the small base
stations was dynamically adjusted to improve the energy efficiency. In
Refs. [6], a novel QoS-aware cell selection scheme was proposed to
reduce the energy consumption. In Ref. [7], the authors proposed a radio
resource allocation algorithm to improve both the energy efficiency and
the spectrum efficiency. Base station sleep is one of the important tech-
nologies to reduce the energy consumption in a cellular network. In
Ref. [9], considering the delay caused by offloading the traffic of the
sleeping small base stations, the authors proposed a base station sleep
mechanism for a cloud computing network. In Ref. [10], an
energy-efficient maximum weighted matching algorithm was proposed
to solve the energy efficiency optimization problem. In Ref. [11], a base
station sleep strategy was proposed for a hybrid energy-powered cellular
network where the IoT applications were considered. Considering both
sleeping and clustering of the eneC Crgy-saving base stations in the
cooperative cellular network, Kim and Lee [12] proposed a joint opti-
mization algorithm for clustering and sleeping the base stations.
In recent years, how to control the sleep depth of the base stations is a
focused research area. The authors in Refs. [13] analyzed the optimal
sleep depth of the Base Stations (BSs) for reducing the energy

* Corresponding author. Chongqing Key Laboratory of Mobile Communications Technology, Chongqing University of Posts and Telecommunications, Chongqing,
400065, PR China.
E-mail address: liyun@cqupt.edu.cn (Y. Li).

https://doi.org/10.1016/j.dcan.2020.04.010
Received 23 December 2019; Received in revised form 15 March 2020; Accepted 23 April 2020
Available online xxxx
2352-8648/© 2020 Chongqing University of Posts and Telecommunications. Publishing Services by Elsevier B.V. on behalf of KeAi Communications Co. Ltd. This is an
open access article under the CC BY-NC-ND license (http://creativecommons.org/licenses/by-nc-nd/4.0/).

Please cite this article as: J. Wu et al., SMDP-based sleep policy for base stations in heterogeneous cellular networks, Digital Communications and
Networks, https://doi.org/10.1016/j.dcan.2020.04.010
J. Wu et al.
Fig. 1. Macro-Femto cellular network environment.
consumption. In Ref. [14], considering the spatio-temporal random
feature of the mobile traffic, a two-stage BS sleep scheme with the traffic
shunting was proposed to reduce the energy consumption. Based on the
adaptive cell scaling, the authors in Ref. [15] developed a small cell sleep
scheme to improve the energy efficiency by adjusting the cell coverage.
Similarly, the authors in Refs. [16] proposed a sleep control scheme that
took the capacity ratio of the SBS as the decision criterion.
Frequent switching of the base stations may cause a ping-pong effect.
Therefore, [17] adopted a double-threshold sleep mechanism. When the
number of the users served by a small base station exceeded the highest
value of a certain threshold range, the base station entered the working
mode, otherwise, it kept sleeping. In Ref. [18], the authors studied the
impact of the sleep rate of the FBSs on the system energy consumption
and analyzed the optimal sleep rate of the FBSs. In Ref. [19], the prob-
lems of the sleep of the FBSs and User Association (UA) were discussed,
and the convex optimization was used to optimize the system energy
consumption.
In [20], the authors considered the user preference and the base
station load and proposed a base station sleep strategy based on the
marginal utility. In Ref. [21], the authors analyzed the relationship be-
tween the optimal part of the spectrum reuse factor and the probability
ratio of the micro base station activity. In Ref. [22], a price based
distributed power allocation algorithm is proposed through the cooper-
ative mode of cell base station division to optimize the transmission
power of cell base station and significantly improve the energy efficiency
of the network. In Ref. [23], an adaptive energy-saving sleep algorithm of
the FBS was proposed, which would put the FBS with the lowest energy
efficiency into sleep and offload its traffic to the nearby FBSs. In
Ref. [24], the authors designed a low-complexity sleep strategy for the
base stations to improve the energy efficiency of the ultra-dense small
cell network, which enabled the base stations with the lower energy ef-
ficiency to switch to the lower energy operation mode, thereby reducing
the network energy consumption.
Considering the dynamic change of the traffic in real life, in Ref. [25],
the authors proposed a deep-nap-based dynamic sleep control
data-driven algorithm that used the deep Q-Learning networks to learn
from the high-dimensional original observations or the unquantized
system state vectors. In Refs. [26,27], they adopted the Markov decision
2
Digital Communications and Networks xxx (xxxx) xxx
process and the artificial neural network method to develop the sleep
mechanism, respectively.
Current studies only consider the energy consumption of the base
stations. In fact, when a mobile terminal communicates with a base
station, the energy consumption of the mobile terminal is also generated.
In view of the above problems, considering the energy consumption of
both the mobile terminals and the base stations, the random dynamic
change of the number of users served by the base stations, this paper
formulates the base station sleep problem in the Macro-Femto hetero-
geneous network as an SMDP-based optimization model and proposes a
base station sleep policy to enhance the energy efficiency. The main
contributions of this paper are outlined below.
1) An SMDP-based model is formulated for the base station sleep prob-
lem in the Macro-Femto heterogeneous network, which includes the
state space, the action space, the system transition probability, the
revenue function, and the discount factor.
2) The detailed transition probability and revenue function are given.
The transition probability considers the random arrival and departure
of the mobile users, and the revenue function considers both the
energy consumption of the base stations and the energy consumption
of the user terminals.
3) Further, an algorithm based on the value iteration is proposed to find
the optimal base station sleep strategy. The evaluation results show
that the proposed algorithm enhances the system energy efficiency.
The rest of this paper is organized as follows: section 2 details the
heterogeneous network structure and the system power consumption
model; section 3 models the dynamic sleep process of the FBSs as an
SMDP model; section 4 uses the dynamic programming method to obtain
the optimal dynamic sleep strategy of the FBSs; section 5 simulates the
system performance; finally, section 6 summarizes the paper.
2. System model
2.1. Network model
This paper considers a Macro-Femto’s double-layer heterogeneous
J. Wu et al. Digital Communications and Networks xxx (xxxx) xxx

Table 1 To ensure the coverage, the MBS f0 is always in the active state. This
List of notations. present study does not consider the power consumption of the MBS f0 in
Symbol Description the sleep state and the fixed power consumption of the MBS f0 in the
active state, but only considers its dynamic power consumption. The
P0 / Pi The power consumption of MBS f0 /FBSfi ði  1Þ
Pmax / Pmax The maximum transmission power of MBS f0 /FBS fi ði  1Þ dynamic power consumption of the MBS f0 is as follows:
0 i
φ0 / φi The conversion factors from DC to RF of MBS f0 /FBS fi ði  1Þ
ρ0 / ρi The load factors of MBS f0 /FBS fi ði  1Þ Pmax
0
P0 ¼ ρ0 (2)
Pu The transmission power of the user terminal u φ0
n0 ðtk Þ / The number of users being served by MBS f0 /FBS fi ði  1Þ in the tk
ni ðtk Þ timeslot Pmax represents the maximum transmission power of the MBS f0 ’s power
0
n00 ðtk Þ / The number of users being served by MBS among the potential users of
amplifier; φ0 represents the conversion factor from DC to RF of the MBS
ni0 ðtk Þ MBS f0 /FBSfi ði  1Þ in the tk timeslot
f0 , and its general range is from 0.1 to 0.2 [31]; and ρ0 represents the load
Di The event that a user in base station fi ði  0Þ ends its service
Ai The event that a user in base station fi ði  0Þ initiates its service factor of the MBS f0 . From the above, the total power consumption of the
ai ðtk Þ The action of the FBS fi ði  1Þ at the beginning of the tk timeslot base station of the system is as follows:
Ni The maximum number of users which base station fi ði  0Þ can serve
X
K
PBS ¼ P0 þ Pi (3)
i¼1
cellular network, as shown in Fig. 1. K FBSs are deployed in the coverage
In addition to the total power consumption of the base stations
area of an MBS. The MBS is always active to ensure the network coverage,
mentioned above, this paper also considers the power consumption
while the FBSs can be in the active or sleep state and use the private
model of the user terminals as follows [30]:
access mode.
The Macro-Femto double-layer heterogeneous cellular network base X
U

station is defined as F ¼ ff0 ;f1 ;f2 ;⋯;fK g, where f0 represents the MBS, f1 ;
f2 ; ⋯fK represent K FBSs, the coverage area of the entire system is S ¼ π R2
(R represents the coverage radius of the MBS), and the coverage area of
each FBS is Sf ¼ π R2f ( Rf indicates the coverage radius of the FBS), and Ui
indicates the maximum user capacity of the base station fi ði  0Þ.
Assuming that a mobile user’s service request follows the Poisson dis-
tribution with a parameter of λ, the user’s service rate follows the
exponential distribution with a parameter of μ.
When a mobile user u initiates a new service request to the FBS which
is in the active state, the mobile user u is associated with the FBS with the
S SS
probability Sf or the MBS with the probability S f . In order to save the
energy, an FBS may enter the sleep state from the active state, and users
served by the FBS will be associated with the MBS.
The notations used in this paper are listed in Table 1.
2.2. Power model
The power consumption model of the FBSfi ði  1Þ is shown in Equa-
tion (1) [17,26]:
8 the randomness of the user request and the user service time, the dy-
< PS ;
> sleep
Pi ¼ Pmax
>
: PA ¼ Pconst þ ρi i
; active
φi
Where Pmax i represents the maximum transmission power of the FBS
fi ði  1Þ ’s power amplifier; φi represents the conversion factor from DC
to RF of the FBS fi ði  1Þ, and its general range is from 0.03 to 0.05 [31];
ρi represents the load factor of the FBS fi ði  1Þ, ρi ¼ Unii ; ni indicates the
number of users served by the FBS fi ði  1Þ among the potential users of
the FBS fi ði  1Þ; Ui represents the maximum user capacity of the FBS
fi ði  1Þ. When the FBS fi ði  1Þ is in the sleep state, the power con-
sumption PS includes the power consumption PBC of the backhaul link
connection module and the static power consumption Pstatic of other
modules that need to work during the FBS fi ði  1Þ sleep, PS ¼ PBC þ
Pstatic .
When the FBS fi ði  1Þ is in the active state, its power consumption PA
includes the fixed power consumption Pconst and the dynamic power
consumption, where the fixed power consumption includes the power
consumption of the microprocessor, the backhaul link connection, the
Field Programmable Gate Array (FPGA) module, the wireless Radio
Frequency (RF) transmitter and receiver. The dynamic power consump-
tion of the FBS fi ði  1Þ is related to the load of the FBS fi ði  1Þ, and its
Pmax
calculation formula is ρi i
φi ði  1Þ.
PUE ¼ ðPCU þ ξPu Þ (4)
u¼1
U represents the user set of the system; Pu represents the transmission
power of the user terminal u; PCU represents the fixed consumption of the
user terminal u, and ξ represents the power consumption coefficient,
which is in the range of 0.6–0.95.
Further, the system power consumption of a heterogeneous cellular
network consisting of one MBS, K FBSs, and the user terminals can be
obtained as:
Ptotal ¼ ω1 PBS þ ω2 PUE (5)
ω1 ; ω2 represent the power weighting factor, and ω1 þ ω2 ¼ 1.
3. The SMDP model of dynamic sleep of the FBS
The main purpose of this paper is to reduce the energy consumption
of the heterogeneous cellular network through a reasonable base station
sleep mechanism, which requires the dynamic sleep strategy of the FBS to
be formulated according to the number of users served by the FBS. Due to
namic sleep problem of the FBS is a random decision problem. This paper
(1)
defines the number of users served by the FBS as the system state, and
discretizes the time with the change in the number of users served by the
FBS (that is, when a new user requests to arrive or the existing user
service ends). In the tkþ1 timeslot, the system state is only related to the
system state of the tk timeslot, and is independent of the system state
before the tk timeslot. At the same time, under the premise that the
mobile user’s service request obeys the Poisson distribution and the
service time obeys the exponential distribution, the transition probability
of the state in the tk timeslot to the state in the tkþ1 timeslot depends on
the time interval from the tk timeslot to the tkþ1 timeslot. Therefore,
taking the number of users served by the FBS as the system state and
reducing the energy consumption as the optimization goal, the problem
of optimally controlling the sleep of the FBS can be modeled as a semi-
Markov decision process. The standard SMDP model can be repre-
sented by a five-tuple: fS; A; P; η; Rg , where S is the system state space, A
is the decision space, P is the state transition probability matrix, and
Pðs’js; aÞ 2 P is the probability of taking action a to transit to the state s’ at
the state s, given the state s; s’ 2 S and the action a 2 A, η is the discount
factor, η 2 ð0; 1Þ, R is the reward function matrix, and Rðs’js; aÞ 2 R
represents the reward after the transition from the state s to the state s’
with the probability Pðs’js; aÞ 2 P.
Below, we focus on the SMDP model of the dynamic sleep of the FBS.

3
J. Wu et al. Digital Communications and Networks xxx (xxxx) xxx

3.1. System state space
The system state s reflects the number of users served by each base
station and the event that a user arrives or leaves. Sðtk Þ is the system state
in the tk timeslot and is expressed as:
 
n0 ðtk Þ; n1 ðtk Þ; n2 ðtk Þ; …;
Sðtk Þ ¼ ; ði  1Þ
ni ðtk Þ; …; nK ðtk Þ; e
n0 ðtk Þ represents the set of users served by the MBS f0 in the tk timeslot,
P
n0 ðtk Þ ¼ Ki¼0 ni0 ðtk Þ, n00 ðtk Þ represents the number of users being served
by the MBS f0 among the potential users of the MBS f0 in the tk timeslot,
ni0 ðtk Þði  1Þ represents the number of users being served by the MBS f0
among potential users of the FBS fi ði  1Þ in the tk timeslot; ni ðtk Þði  1Þ
represents the number of users being served by the FBS fi ði  1Þ among
the potential users of the FBS fi ði  1Þ in the tk timeslot; ni ðtk Þ 2
 changes from sleeping to working. When a user leaves the FBS fi ði  1Þ,
 0; active
; e represents an event e 2 fA0 ; ⋯; Aj ; ⋯; AK ; D0 ; ⋯; Di ; ⋯;
1; sleep
DK g that cause the system state to change, where Aj indicates that a base
station fi ði  0Þ has a new user to initiate a service request, and Di in-
dicates that a base station fi ði  0Þ has a user to end service.
3.2. Decision space
At the beginning of timeslot tk , the optimal strategy adopted for the
state Sðtk Þ is given according to the value iterative algorithm, and the
base station selects an action according to the strategy to change it from
sleeping to working or from working to sleeping, or maintains its sleep/
active state. For example, an FBS is in the active state, and the optimal
strategy for the FBS based on the value iteration is to sleep, then the FBS
shifts from the active state to the sleep state. After the state of the FBS
changes, the number of users served by the base station also changes, that
is, the system state transition occurs, and the system state after the
transition remains until the beginning of the tkþ1 time slot.
Define the action aðtk Þ as the decision to control the state of the FBS at
the beginning of the tk timeslot, including the FBS going to sleep, the FBS
going to be active, and the FBS maintaining its state, as shown in equa-
tion (7):
(6)
 
a1 ðtk Þ; a2 ðtk Þ; …;
aðtk Þ ¼ ; ði  1Þ 2 f  1; 0; 1gK (7)
ai ðtk Þ; …; aK ðtk Þ
ai ðtk Þ represents the decision to control the state of the FBS fi ði  1Þ at the
beginning of the tk timeslot, and its values are shown in Table 2:
To avoid the invalid decisions, this paper stipulates that when a new
user arrives at the FBS fi ði  1Þ, the action ai ðtk Þ of the FBS fi ði  1Þ may
be 0 or 1, which means that the FBS fi ði  1Þ maintains its current state or
the action ai ðtk Þ of the FBS fi ði  1Þ may be 0 or -1, which means that the
FBS fi ði  1Þ maintains its current state or changes from working to
sleeping.
3.3. State transition probability
In the SMDP model, when the system is in the state s 2 S, it will
remain for a period of time until its transition to the state s’ 2 S. The
period of time during which the state does not change is called the state
dwell time. Its general expression is:
τðs; aÞ ¼ Eðtkþ1  tk jsðtk Þ ¼ sj; aðtk Þ ¼ aÞ (8)
Combining the state space and the action space of the system, we get
the average dwell time of the system as shown in equation (9) [28]:

8 9
> !1 >
> K 
X  K 
X  >
>
>       >
>
>
> Nj  nj ξ nj  n0j λj þ N0  n00  1 λ0 þ nj ξ nj þ n0j μj þ ðn0 þ 1Þμ0 >
>
>
> >
>
>
> j¼1 j¼1 >
>
>
> >
>
>
> ; e ¼ Ai ; ai ¼ 0; ni 6¼ 1; i ¼ 0 >
>
>
> >
>
>
> !1 >
>
>
> K   K   >
>
>
> X       X     >
>
>
> 0 0
Nj  nj ξ nj  nj λj þ Ni  ni  ni  1 λi þ N0  n0 λ0 þ 0 0 0 0
nj ξ nj þ nj μj þ ni þ ni þ 1 μi þ n0 μ0 >
>
>
> >
>
>
> j¼1;j6¼i j¼1;j6¼i >
>
>
> >
>
>
> >
>
>
> ; e ¼ Ai ; ai ¼ 0; ni 6¼ 1; i 2 f1; ⋯; Kg >
>
>
> >
>
>
> !1 >
>
> XK   XK   >
>
>           >
>
>
>
0 0
Nj  nj ξ nj  nj λj þ Ni  ni  1 λi þ N0  n0 λ0 þ 0 0 0 0
nj ξ nj þ nj μj þ ni þ 1 μi þ n0 μ0 >
>
>
> >
>
>
> j¼1;j6¼i j¼1;j6¼i >
>
>
> >
>
>
> !1 >
>
>
> XK         XK       >
>
>
> 0 0 0 0 0 0 >
>
>
> Nj  nj ξ nj  nj λj þ Ni  ni  1 λi þ N0  n0 λ0 þ nj ξ nj þ nj μj þ ni þ 1 μi þ n0 μ0 >
>
>
> >
>
< j¼1;j6¼i j¼1;j6¼i =
τðs; aÞ ¼ ; e ¼ Ai ; ai ¼ 1; ni ¼ 1; i 2 f1; ⋯; Kg (9)
>
> >
>
>
> !1 >
>
>
> X K 
     XK 
   >
>
>
> >
>
>
> Nj  nj ξ nj  n0j λj þ N0  n00 þ 1 λ0 þ nj ξ nj þ n0j μj þ ðn0  1Þμ0 >
>
>
> >
>
>
> j¼1 j¼1 >
>
>
> >
>
>
> ; e ¼ Di ; ai ¼ 0; ni > 0; i ¼ 0 >
>
>
> >
>
>
> !1 >
>
>
> XK 
       XK 
     >
>
>
> >
>
>
>
0 0
Nj  nj ξ nj  nj λj þ Ni  ni  ni þ 1 λi þ N0  n0 λ0 þ 0 0 0 0
nj ξ nj þ nj μj þ ni þ ni  1 μi þ n0 μ0 >
>
>
> >
>
>
> j¼1;j6¼i j¼1;j6¼i >
>
>
> >
>
>
> ; e ¼ Di ; ai ¼ 0; ni > 0; i 2 f1; ⋯; Kg >
>
>
> >
>
>
> !1 >
>
>
>     >
>
>
> XK
      XK
    >
>
>
> 0 0
Nj  nj ξ nj  nj λj þ Ni  ni þ 1 λi þ N0  n0 λ0 þ 0 0 0 0
nj ξ nj þ nj μj þ ni  1 μi þ n0 μ0 >
>
>
> >
>
>
> j¼1;j6¼i j¼1;j6¼i >
>
>
> >
>
>
> >
>
>
> ; e ¼ Di ; ai ¼ 1; ni > 0; i 2 f1; ⋯; Kg >
>
>
> >
>
>
: >
;

4
J. Wu et al. Digital Communications and Networks xxx (xxxx) xxx

Table 2 6) The base station fi ði  0Þ is in the sleep state, ni ¼  1, ai ¼ 1; e’ ¼

Meaning of the action. Dj ðj 2 f0; ⋯; KgÞ.
The value of ai ðtk Þ Meaning
b. At s ¼ ðn0 ; ⋯; ni ; ⋯; nK ; Di Þði  0Þ, we have different transition prob-
 1 FBS fi ði  1Þ goes to sleep
0 FBSfi ði  1Þ remains the state abilities for the following four sub-cases.
1 FBS fi ði  1Þ goes to be active
1) The base station fi ði  0Þ is in the active state, ni > 0, ai ¼ 0; e’ ¼
Aj ðj 2 f0; ⋯; KgÞ.
 2) The base station fi ði  0Þ is in the active state, ni > 0, ai ¼ 0; e’ ¼
1; nj  0 Dj ðj 2 f0; ⋯; KgÞ.
ξðnj Þ ¼ represents the state of the base station; N0 indicates
0; nj ¼ 1
3) The base station fi ði  0Þ is in the active state, ni > 0, ai ¼ 1; e’ ¼
the maximum number of the potential users of the MBS f0 , and Ni ði  1Þ
Aj ðj 2 f0; ⋯; KgÞ.
represents the maximum number of the potential users of the FBS fi ði  1Þ
4) The base station fi ði  0Þ is in the active state, ni > 0, ai ¼ 1; e’ ¼
.
Dj ðj 2 f0; ⋯; KgÞ.
A new user requests to access to a base station fi ði  0Þ which is in the
active state and ai ¼ 0. When i ¼ 0, the user successfully gets access to
the MBS f0 . When i > 0, the user then gets successful access to the FBS 3.4. Return function
S
fi ði  1Þ with the probability Sf , and successfully accesses to the MBS f0
SS In this paper, the fundamental purpose of studying the sleep strategy
with the probability S f .
of the base station is to save the system energy consumption. Therefore,
When a new user requests to the access to the FBS fi ði  1Þ, which is in
the system energy consumption during the decision period when the
the active state and ai ¼ 0 or in the sleep state and ai ¼ 1, the user
system takes the action aðtk Þ in the current state s until it transits to
Sf
successfully gets access to the FBS fi ði  1Þ with the probability S, and another state s’ is defined as the instant return R of the system.
SSf
successfully accesses to the MBS f0 with the probability S .
According to the different events and the different decisions of the
When a new user requests to the access to the FBS fi ði  1Þ, which is in base station, the return function of the system during the dwell time can
the sleep state and ai ¼ 0, this user gets successful access to the MBS f0 . be divided into five cases:
When a base station fi ði  0Þ is in the active state, and ni > 0, ai ¼ 0,
there is a user leaving this base station, and the remaining users of the 1) e ¼ Ai ;ai ¼ 0;ni 6¼  1ði 2 f0;⋯;KgÞ, a new user requests to arrive at
base station fi ði  0Þ continue to communicate with it. the base station fi ði  0Þ, which is in the active state and ai ¼ 0. The
When the FBS fi ði  1Þ is active, ni > 0 and ai ¼  1, there is a user return function in this case is shown in equation (10):
leaving the FBS fi ði  1Þ, then the remaining users served by the FBS
fi ði  1Þ are associated with the MBS f0 .
Considering the parameters of the user arrival and departure, the

! !
K
X

8   Pmax
j    1 Sf Pmax 1 S  Sf Pmax
>
> ω1 ρj ξ nj  þ Pconst þ 1  ξ nj PS þ i
þ þ ρ0 0
þω2 PUE  τðs; aÞ
>
> φj Ui S φi U0 S φ0
>
> j¼1
>
<
Rðs; aÞ ¼ ; i ¼ f1; ⋯; Kg (10)
>
>
>
> ! !
>
> K
X max


max
>
:   Pj    1 P0
ω1 ρj ξ nj  þ Pconst þ 1  ξ nj PS þ þ ρ0 þ ω2 PUE  τðs; aÞ; i ¼ 0
j¼1
φj U0 φ0

dwell time, the sleep decision, and the user re-association, we divide the 2) e ¼ Ai ;a ¼ 0;ni ¼  1ði 2 f1;⋯;KgÞ, a new user requests to arrive at
transition probability of the system from the state s ¼ the FBS fi ði  1Þ, which is in the sleep state and ai ¼ 0, then the user is
ðn0 ; ⋯; ni ; ⋯; nK ; eÞði  1Þ to the state s’ ¼ ðn’0 ; ⋯; n’i ; ⋯; n’K ; e’Þði  1Þ associated with the MBS f0 . The return function in this case is shown
into different cases and show them as follows. The transition probabili- in equation (11):
ties for different situations are given in Appendix A.
K
X max

  Pj   
Rðs; aÞ ¼ ω1 ρj ξ nj  þ Pconst þ 1  ξ nj PS
a. At s ¼ ðn0 ; ⋯; ni ; ⋯; nK ; Ai Þði  0Þ, we have different transition prob- j¼1
φj
abilities for the following six sub-cases.
! !
1 Pmax
þ þ ρ0 0
þ ω2 PUE  τðs; aÞ (11)
1) The base station fi ði  0Þ is in the active state, ni  0, ai ¼ 0, e’ ¼ U0 φ0
Aj ðj 2 f0; ⋯; KgÞ.
2) The base station fi ði  0Þ is in the active state, ni  0, ai ¼ 0; e’ ¼
3) e ¼ Ai ;ai ¼ 1;ni ¼  1ði 2 f1;⋯;KgÞ, a new user requests to arrive at
Dj ðj 2 f0; ⋯; KgÞ.
the FBS fi ði  1Þ, which is in the sleep state and will enter the active
3) The base station fi ði  0Þ is in the sleep state, ni ¼  1, ai ¼ 0;e’ ¼
state. The return function in this case is shown in equation (12):
Aj ðj 2 f0; ⋯; KgÞ.
4) The base station fi ði  0Þ is in the sleep state, ni ¼  1, ai ¼ 0;e’ ¼
Dj ðj 2 f0; ⋯; KgÞ.
5) The base station fi ði  0Þ is in the sleep state, ni ¼  1, ai ¼ 1;e’ ¼
Aj ðj 2 f0; ⋯; KgÞ.

5
J. Wu et al. Digital Communications and Networks xxx (xxxx) xxx

0 K
0 X

1 1
  Pmax
j    1 Sf Pmax
i
ρj ξ nj  þ Pconst þ 1  ξ nj PS þ þ
B B j¼1;j6¼i φj Ui S φi C C
B B C C
Rðs; aÞ ¼ Bω1 B C þ ω2 PUE C  τðs; aÞ (12)
@ @
A A
1 S  Sf Pmax
þ ρ0 0
þ Pconst
U0 S φ0

4) e ¼ Di ; ai ¼ 0ði 2 f0; ⋯; KgÞ, the base station fi ði  0Þ is in the active

state and maintains its state, then a user leaves the base station
fi ði  0Þ. The return function in this case is shown in equation (13):

K
X max

  Pj   
Rðs; aÞ ¼ ω1 ρj ξ nj  þ Pconst þ 1  ξ nj PS
j¼1;j6¼i
φj

max ! !
1 Pmax 1 P0
þ ðni  1Þ  i þ Pconst þ þ ρ0 þ ω2 PUE  τðs; aÞ (13)
Ui φi U0 φ0

5) e ¼ Di ;ai ¼  1ði 2 f1;⋯;KgÞ, where the FBS fi ði  1Þ is in the active

state, a user leaves the FBS fi ði  1Þ and ai ¼  1, then the users
served by the FBS fi ði  1Þ will be associated with the MBS f0 . The
return function in this case is shown in equation (14):

K
X max

  Pj   
Rðs; aÞ ¼ ω1 ρj ξ nj  þ Pconst þ 1  ξ nj PS
j¼1;j6¼i
φj
! !
1 Pmax
0
þ ðn0 þ ni  1Þ  þ Pconst þ PS þ ω2 PUE  τðs; aÞ (14)
Ui φ0

4. Solution of optimal dynamic sleep strategy

5. Simulation
Through the explanation in section 3, we obtain the system state
space S, the decision space A, the state transition probability matrix P, 5.1. Simulation parameters
and the return function matrix R in the SMDP model. The SMDP model
proposed in this paper has limited state space and action space, and we This section evaluates the performance of the Optimization Sleep
can get the optimal decision based on the value iteration. Strategy (OSS) proposed in this paper and compares it with other three
Using the discount revenue model of the SMDP [28], the system’s sleep methods. The first method is the Sleep/Wake Sleep (SWS) algo-
long-term revenue model is: rithm proposed in Ref. [26]. When the load of the small base station is not
( ) very high, the macro base station will handle the traffic alone while
X providing the services to the users, then the small base station will be
VðsÞ ¼ min Pðs’js; ajÞðRðs; aÞ þ ηVðs’ÞÞ (15)
a2A
s’2S
closed. The second method is the Fixed Threshold Sleep (FTS) algorithm
proposed in Ref. [29]. Both the FBS that is in the active state with no
η is the discount revenue factor. served user sleeping and the FBS that is in the sleep state that the new
From formula (15), we get to know that in the state s 2 S of the sys- users arriving will work. The third method is the Non-Sleep Strategy
tem, the long-term expected total return VðsÞ depends entirely on the (NSS). All base stations are active and no sleep action is performed. The
values of λ, μ, and Ni . Our goal is to find an optimal strategy to obtain the simulation parameter settings of this paper are shown in Table 3.
minimum expected total return VðsÞ in each state. Therefore, when a
mobile user reaches or leaves the network, the algorithm will calculate Table 3
the return VðsÞ by taking different actions ða ¼ 1; a ¼ 0; a ¼ 1Þ in the Main simulation parameters.
current state, and choose the action that will bring the minimum cu- Parameter name Parameter value
mulative return.
MBS/FBS Number 1/2
In this paper, the detailed description of the dynamic sleep algorithm MBS/FBS Coverage radius 500 m/50 m
of the FBS based on the value iteration is shown in Algorithm 1. In step 2, MUE/FUE Maximum transmission power 30 dB m/23 dB m
Δ ¼ ½v  VðsÞ2 is defined as the square of the difference value of VðsÞ MUE/FUE Fixed power consumption 0.01 W
Power weighting factor ω1 0.9
before and after one iteration. When Δ < εð1 ηÞ
2η , the iterations are stoped User service rate μ 1
and the optimal FBS sleep strategy is returned. User arrival rate λ 0.5
MBS/FBS Maximum transmission power 10 W/1 W
( ) MBS/FBS DC to RF conversion factor 0.156/0.045
X
π ðsÞ ¼ argmin
*
Pðs’js; ajÞðRðs; aÞ þ ηVðs’ÞÞ (16) MBS/FBS Maximum user capacity 100/10
a2A MBS/FBS Maximum number of potential users 12/8
s’2S
FBS Fixed power consumption 4.8 W
FBS Hibernate fixed power 0.3 W
Algorithm 1. SMDP-based base station sleep value iterative algorithm Discount revenue factor η 0.9
Convergence threshold ε 0.1

6
J. Wu et al.
Fig. 2. Changes in system energy consumption per unit time with λ
5.2. Simulation results
Fig. 2 demonstrates the effect of the parameter λ on the system energy
consumption per unit time of the four algorithms. It can be seen in Fig. 2
that the system energy consumption per unit time of the four algorithms
decreases as λ increases. The main reason is: according to equation (9), it
can be known that an increase of λ will cause a decrease in τ, and
equations (10)-(14) show that the returning system is proportional to τ,
so the system energy consumption per unit time decreases as τ decreases.
The OSS algorithm proposed in this paper and the SWS algorithm have
better performances than the FTS algorithm and the NSS algorithm as the
OSS algorithm and the SWS algorithm adaptively make FBSs sleep or
work according to the arrival and departure of the mobile users.
Furthermore, the OSS algorithm outperforms the SWS algorithm because
the OSS algorithm considers the energy consumption of both the base
stations and the mobile users, but the SWS algorithm ignores the energy
consumption of the mobile users. The NSS algorithm does not take into
account the sleep of the base station, so its performance is the worst in
terms of the energy consumption.
Fig. 3 shows the effect of the parameter φi on the system energy
consumption per unit time of the four algorithms. It can be seen in Fig. 3
that the system energy consumption per unit time of the four algorithms
Fig. 3. Changes in system energy consumption per unit time with.φi
7
Digital Communications and Networks xxx (xxxx) xxx
Fig. 4. Changes in system energy consumption per unit time with.Ni
decreases as φi increases. The main reason is: according to formula (1), it
can be known that the transmission power of the FBS’s power amplifier
decreases as it increases, so the system energy consumption per unit time
of the four algorithms shows a downward trend. The energy consumption
per unit time of the OSS algorithm proposed in this paper is lower than
other three algorithms.
Fig. 4 presents the effect of the parameter Ni on the energy con-
sumption per unit time of the four algorithms. As can be seen in Fig. 4, the
system energy consumption per unit time of the four algorithms de-
creases as Ni increases. The main reason is: according to equation (9), it
can be known that an increase in Ni will lead to a decrease in τ. And then
it will lead to a decrease in the system energy consumption per unit time.
The energy consumption per unit time of the OSS algorithm is lower than
other three algorithms. The main reason is that the OSS algorithm
dynamically controls the active or sleep states of the FBSs according to
the arrival and departure of mobile users, so it can decrease the energy
consumption of both the base stations and the mobile users.
Fig. 5 illustrates the effect of the parameter Pmax on the energy con-
i
sumption per unit time of the four algorithms. It can be seen in Fig. 5 that
the system energy consumption per unit time of the four algorithms in-
creases as Pmax increases. The main reason is: according to equation (1), it
i
can be known that an increase in Pmax i will make the transmission power
Fig. 5. Changes in system energy consumption per unit time with.Pmax
i
J. Wu et al.
Fig. 6. Changes in system energy consumption per unit time with.Pconst
Fig. 7. Changes in system energy consumption per unit time with.Ui
of the FBS’s power amplifier increase, thereby increasing the power
consumption of the system, so it shows an upward trend. The energy
consumption per unit time of the OSS algorithm proposed in this paper is
still the lowest.
Appendix A. Transition Probability for SMDP
The transition probability is divided into the following cases:
a. At s ¼ ðn0 ; ⋯; ni ; ⋯; nK ; Ai Þði  0Þ, we have different transition probabilities for the following six sub-cases.
1) When the base station fi ði  0Þ is in the active state, ni  0, ai ¼ 0, e’ ¼ Aj ðj 2 f0; ⋯; KgÞ, the transition probability is shown in equation (A.1).
8
Digital Communications and Networks xxx (xxxx) xxx
Fig. 6 shows the effect of the parameter Pconst on the energy con-
sumption per unit time of the four algorithms. It can be seen in Fig. 6 that
the energy consumption per unit time of the four algorithms increases as
Pconst increases. The main reason is: according to formula (1), it can be
known that an increase in Pconst will make the FBS’s hardware power
consumption increase, thereby increasing the system power consump-
tion, so it shows an upward trend. The energy consumption per unit time
of the OSS algorithm proposed in this paper is the lowest.
Fig. 7 reveals the effect of the parameter Ui on the energy consump-
tion per unit time of the four algorithms. It can be seen in Fig. 7 that the
energy consumption per unit time of the OSS algorithm proposed in this
paper is the lowest. The energy consumption per unit time of the four
algorithms decreases with the increase of Ui . The main reason is: ac-
cording to equations (10)-(14), it can be known that an increase in Ui will
lead to a decrease in the returning system, thereby reducing the system
power consumption, so it shows a downward trend.
6. Summary
In this paper, we use the SMDP to model the dynamic sleep of the FBS
in the dual-layer Macro-Femto heterogeneous network. In order to
reduce the system energy consumption, this paper considers the dynamic
random changes of the number of the users served by different base
stations, and gives the detailed transition probabilities for different states
and actions. The reward functions that consider the energy consumption
of both the base stations and the mobile terminals are also presented.
Furthermore, we propose the value iteration to obtain the optimal base
station sleep strategy. Simulation results show that the dynamic sleep
mechanism of the FBSs based on the SMDP efficiently reduces the system
energy consumption of the Macro-Femto heterogeneous network.
Declaration of competing interest
We declare that we have no financialand personal relationships with
other people or organizations that can inappropriately influence our
work, there is no professional or other personal interest of any kind in any
product, service and/or company that could be construed as influencing
the position presented in, or the review of, the manuscript entitled.
Acknowledgement
This work was supported by the Program for the National Science
Foundation of China (61671096), by the Chongqing Research Program of
Basic Science and Frontier Technology (cstc2017jcyjBX0005), by
Chongqing Science and Technology Innovation Leading Talent Support
Program (CSTCCXLJRC201710), by Venture and Innovation Support
Program for Chongqing Overseas Returnee.
J. Wu et al. Digital Communications and Networks xxx (xxxx) xxx

 
Nj  n0j  1 λj τðs; aÞ; i ¼ 0; j ¼ i; n’0 ¼ n0 þ 1; n’k ¼ nk ðk 2 f1; ⋯; KgÞ
8
>
>     
>
>
>
> Nj  nj ξ nj þ n0j λj τðs; aÞ; i ¼ 0; j 6¼ i; n’0 ¼ n0 þ 1; n’k ¼ nk ðk 2 f1; ⋯; KgÞ
>
>
>
>
> 9
>
>
>
> Sf  
>
>
> N  n  n 0
 1 λ τ ðs; aÞ; n’ ¼ n ðk ¼
6 iÞ; n’ ¼ n þ 1 >
>
>
> S
j j j j k k i i >
=
>
>
>
> ; i 6¼ 0; j ¼ i
>
> >
>
>
> S  Sf   >
>
>
;
>
>
0
Nj  nj  nj  1 λj τðs; aÞ; n’0 ¼ n0 þ 1; n’k ¼ nk ðk 2 f1; ⋯; KgÞ
>
> S
>
>
< 9
Pðs’js; aÞ ¼ Sf     
>
N  n ξ n þ n 0
λ τ ðs; aÞ; n’ ¼ n ðk ¼
6 iÞ; n’ ¼ n þ 1 >
> (A.1)
> S
>
j j j j j k k i i >
=
>
>
>
> ; i 6¼ 0; j 6¼ i; j 6¼ 0
>
> >
> >
>
>
> S  Sf    
0
 >
>
;
>
> Nj  nj ξ nj þ nj λj τðs; aÞ; n’0 ¼ n0 þ 1; n’k ¼ nk ðk 2 f1; ⋯; KgÞ
>
> S
>
>
> 9
>
>
>
> Sf  
>
> N  n 0
λ τ ðs; aÞ; n’ ¼ n ðk ¼
6 iÞ; n’ ¼ n þ 1 >
>
>
>
j j j k k i i >
> S
>
=
>
> ; i 6¼ 0; j 6¼ i; j ¼ 0
>
> >
>
: S  Sf 
>
0
 >
>
>
;
Nj  nj λj τðs; aÞ; n’0 ¼ n0 þ 1; n’k ¼ nk ðk 2 f1; ⋯; KgÞ
S

2) When the base station fi ði  0Þ is in the active state, ni  0, ai ¼ 0; e’ ¼ Dj ðj 2 f0; ⋯; KgÞ, the transition probability is shown in equation (A.2).
 
n0 þ 1 μj τðs; aÞ; i ¼ 0; j ¼ i; n’0 ¼ n0 þ 1; n’k ¼ nk ðk 2 f1; ⋯; KgÞ
8 j
>
>    
>
>
>
> nj ξ nj þ n0j μj τðs; aÞ; i ¼ 0; j 6¼ i; n’0 ¼ n0 þ 1; n’k ¼ nk ðk 2 f1; ⋯; KgÞ
>
>
>
>
>
> 9
>
>
> Sf  
>
>
> nj þ n0j þ 1 μj τðs; aÞ; n’k ¼ nk ðk 6¼ iÞ; n’i ¼ ni þ 1 >
>
>
> S >
=
>
>
>
> ; i 6¼ 0; j ¼ i
>
>   >
>
>
> S  S >
>
>
> f
nj þ n0j þ 1 μj τðs; aÞ; n’0 ¼ n0 þ 1; n’k ¼ nk ðk 2 f1; ⋯; KgÞ ;
>
> S
>
>
>
< 9
Pðs’js; aÞ ¼ Sf    
>
nj ξ nj þ n0j μj τðs; aÞ; n’k ¼ nk ðk 6¼ iÞ; n’i ¼ ni þ 1 >
> (A.2)
>
> S >
=
>
>
>
> ; i 6¼ 0; j 6¼ i; j 6¼ 0
>
> >
>
>     >
>
>
> S  S >
nj ξ nj þ n0j μj τðs; aÞ; n’0 ¼ n0 þ 1; n’k ¼ nk ðk 2 f1; ⋯; KgÞ ;
f
>
>
>
> S
>
>
>
> 9
>
> Sf n0 μ τðs; aÞ; n’ ¼ n ðk 6¼ iÞ; n’ ¼ n þ 1 >
>
> >
>
>
> S j j k k i i >
=
>
>
>
> ; i 6¼ 0; j 6¼ i; j ¼ 0
>
> >
>
> >
>
: S  Sf 0 >
nj μj τðs; aÞ; n’0 ¼ n0 þ 1; n’k ¼ nk ðk 2 f1; ⋯; KgÞ ;
S

3) When the base station fi ði  0Þ is in the sleep state, ni ¼  1, ai ¼ 0; e’ ¼ Aj ðj 2 f0; ⋯; KgÞ, the transition probability is shown in equation (A.3).
8 
>
> Nj  n0j  1 λj τðs; aÞ; j ¼ i; n’0 ¼ n0 þ 1; n’k ¼ nk ðk 2 f1; ⋯; KgÞ
>
<  
Pðs’js; aÞ ¼ Nj  n0j λj τðs; aÞ; j ¼ 0; j 6¼ i; n’0 ¼ n0 þ 1; n’k ¼ nk ðk 2 f1; ⋯; KgÞ (A.3)
>
>
> 
:   
Nj  nj ξ nj  n0j λj τðs; aÞ; j 6¼ 0; j 6¼ in’0 ¼ n0 þ 1; n’k ¼ nk ðk 2 f1; ⋯; KgÞ

4) When the base station fi ði  0Þ is in the sleep state, ni ¼  1, ai ¼ 0; e’ ¼ Dj ðj 2 f0; ⋯; KgÞ, the transition probability is shown in equation (A.4).
8 
>
> n0j þ 1 μj τðs; aÞ; j ¼ i; n’0 ¼ n0 þ 1; n’k ¼ nk ðk 2 f1; ⋯; KgÞ
>
<
Pðs’js; aÞ ¼ n0j μj τðs; aÞ; j ¼ 0; j 6¼ i; n’0 ¼ n0 þ 1; n’k ¼ nk ðk 2 f1; ⋯; KgÞ (A.4)
>
>   
>
:

nj ξ nj þ n0j μj τðs; aÞ; j 6¼ 0; j 6¼ in’0 ¼ n0 þ 1; n’k ¼ nk ðk 2 f1; ⋯; KgÞ

5) When the base station fi ði  0Þ is in the sleep state, ni ¼  1, ai ¼ 1; e’ ¼ Aj ðj 2 f0; ⋯; KgÞ, the transition probability is shown in equation (A.5).

9
J. Wu et al. Digital Communications and Networks xxx (xxxx) xxx

9
8 Sf  
>
Nj  n0j  1 λj τðs; aÞ; n’k ¼ nk ðk 6¼ iÞ; n’i ¼ 1 >
>
>
> S >
=
>
>
>
> ;j ¼ i
>
> >
>
> S  Sf   >
>
>
> 0 >
;
>
> Nj  nj  1 λj τðs; aÞ; n’0 ¼ n0 þ 1; n’i ¼ 0; n’k ¼ nk ðk ¼
6 0; iÞ
>
> S
>
>
> 9
>
>
> Sf    
>
>
> Nj  nj ξ nj  n0j λj τðs; aÞ; n’k ¼ nk ðk 6¼ iÞ; n’i ¼ 1 >
>
>
< S >
=
Pðs’js; aÞ ¼ ; j 6¼ i; j 6¼ 0 (A.5)
>
> >
>
>
> S  Sf    0
 >
>
>
;
>
> Nj  nj ξ nj  nj λj τðs; aÞ; n’0 ¼ n0 þ 1; n’i ¼ 0; n’k ¼ nk ðk 6¼ 0; iÞ
>
> S
>
>
> 9
>
>
> Sf  
>
>
> Nj  n0j λj τðs; aÞ; n’k ¼ nk ðk 6¼ iÞ; n’i ¼ 1 >
>
>
> S >
=
>
>
>
> ; j 6¼ i; j ¼ 0
>
> >
: S  Sf   >
>
0 >
;
Nj  nj λj τðs; aÞ; n’0 ¼ n0 þ 1; n’i ¼ 0; n’k ¼ nk ðk 6¼ 0; iÞ
S

6) When the base station fi ði  0Þ is in the sleep state, ni ¼  1, ai ¼ 1; e’ ¼ Dj ðj 2 f0; ⋯; KgÞ, the transition probability is shown in equation (A.6).
  9
8 Sf n0 þ 1 μ τðs; aÞ; n’ ¼ n ðk 6¼ iÞ; n’ ¼ 1 >
>
> j j k k i >
>
>
> S =
>
>
>
> ;j ¼ i
>
> >
>
>
> S  Sf  0  >
>
>
;
>
> nj þ 1 μj τðs; aÞ; n’0 ¼ n0 þ 1; n’k ¼ nk ðk 2 f1; ⋯; KgÞ
>
> S
>
>
> 9
>
>
> Sf    
>
>
> nj ξ nj þ n0j μj τðs; aÞ; n’k ¼ nk ðk 6¼ iÞ; n’i ¼ 1 >
>
>
< S >
=
Pðs’js; aÞ ¼ ; j 6¼ i; j 6¼ 0 (A.6)
>
> >
>
>
>
> S  Sf    0
 >
>
;
>
> nj ξ nj þ nj μj τðs; aÞ; n’0 ¼ n0 þ 1; n’i ¼ 0; n’k ¼ nk ðk ¼ 6 0; iÞ
>
> S
>
>
>
> 9
>
> Sf 0 >
>
> n μ τðs; aÞ; n’k ¼ nk ðk 6¼ iÞ; n’i ¼ 1 >
>
>
> S j j >
=
>
>
>
> ; j 6¼ i; j ¼ 0
>
> >
: S  Sf 0 >
>
>
;
nj μj τðs; aÞ; n’0 ¼ n0 þ 1; n’i ¼ 0; n’k ¼ nk ðk 6¼ 0; iÞ
S

b. At s ¼ ðn0 ; ⋯; ni ; ⋯; nK ; Di Þði  0Þ, we have different transition probabilities for the following four sub-cases.

1) When the base station fi ði  0Þ is in the active state, ni > 0, ai ¼ 0; e’ ¼ Aj ðj 2 f0; ⋯; KgÞ, the transition probability is shown in equation (A.7).
 
8
> Nj  n0j þ 1 λj τðs; aÞ; i ¼ 0; j ¼ i; n’0 ¼ n0  1; n’k ¼ nk ðk 2 f1; ⋯; KgÞ
>
>  
>
>  
>
> 0
> Nj  nj ξ nj  nj λj τðs; aÞ; i ¼ 0; j 6¼ i; n’0 ¼ n0  1; n’k ¼ nk ðk 2 f1; ⋯; KgÞ
>
>
<  
Pðs’js; aÞ ¼ Nj  nj  n0j þ 1 λj τðs; aÞ; i 6¼ 0; j ¼ i; n’k ¼ nk ðk 6¼ iÞ; n’i ¼ ni  1 (A.7)
>
>  
>
>  
>
> Nj  nj ξ nj  n0j λj τðs; aÞ; i 6¼ 0; j 6¼ i; j 6¼ 0; n’k ¼ nk ðk 6¼ iÞ; n’i ¼ ni  1
>
>
>
>  
:
Nj  n0j λj τðs; aÞ; i 6¼ 0; j 6¼ i; j ¼ 0; n’k ¼ nk ðk 6¼ iÞ; n’i ¼ ni  1

2) When the base station fi ði  0Þ is in the active state, ni > 0, ai ¼ 0; e’ ¼ Dj ðj 2 f0; ⋯; KgÞ, the transition probability is shown in equation (A.8).
8  0 
>
> nj  1 μj τðs; aÞ; i ¼ 0; j ¼ i; n’0 ¼ n0  1; n’k ¼ nk ðk 2 f1; ⋯; KgÞ
>
>    
>
>
>
> 0
> nj ξ nj þ nj μj τ ðs; aÞ; i ¼ 0; j 6¼ i; n’0 ¼ n0  1; n’k ¼ nk ðk 2 f1; ⋯; KgÞ
>
<  
Pðs’js; aÞ ¼ nj þ n0j  1 μj τðs; aÞ; i 6¼ 0; j ¼ i; n’k ¼ nk ðk 6¼ iÞ; n’i ¼ ni  1 (A.8)
>
>
>
>    
>
>
>
> nj ξ nj þ n0j μj τðs; aÞ; i 6¼ 0; j 6¼ i; j 6¼ 0; n’k ¼ nk ðk 6¼ iÞ; n’i ¼ ni  1
>
>
:
n0j μj τðs; aÞ; i 6¼ 0; j 6¼ i; j ¼ 0; n’k ¼ nk ðk 6¼ iÞ; n’i ¼ ni  1

3) When the base station fi ði  0Þ is in the active state, ni > 0, ai ¼ 1; e’ ¼ Aj ðj 2 f0; ⋯; KgÞ, the transition probability is shown in equation (A.9).

10
J. Wu et al. Digital Communications and Networks xxx (xxxx) xxx

8  
>
> Nj  nj  n0j þ 1 λj τðs; aÞ; j ¼ i; n’i ¼ 1; n’0 ¼ n0 þ ni  1; n’k ¼ nk ðk 6¼ 0; iÞ
>
< 
 
Pðs’js; aÞ ¼ Nj  nj ξ nj  n0j λj τðs; aÞ; j 6¼ i; j 6¼ 0; n’i ¼ 1; n’0 ¼ n0 þ ni  1; n’k ¼ nk ðk 6¼ 0; iÞ (A.9)
>
>  
>
:
Nj  n0j λj τðs; aÞ; j 6¼ i; j ¼ 0; n’i ¼ 1; n’0 ¼ n0 þ ni  1; n’k ¼ nk ðk 6¼ 0; iÞ

4) When the base station fi ði  0Þ is in the active state, ni > 0, ai ¼ 1; e’ ¼ Dj ðj 2 f0; ⋯; KgÞ, the transition probability is shown in equation (A.10).
8  
>
> nj þ n0j  1 μj τðs; aÞ; j ¼ i; n’i ¼ 1; n’0 ¼ n0 þ ni  1; n’k ¼ nk ðk 6¼ 0; iÞ
>
<
   
Pðs’js; aÞ ¼ nj ξ nj þ n0j μj τðs; aÞ; j 6¼ i; j 6¼ 0; n’i ¼ 1; n’0 ¼ n0 þ ni  1; n’k ¼ nk ðk 6¼ 0; iÞ (A.10)
>
>
>
:
n0j μj τðs; aÞ; j 6¼ i; j ¼ 0; n’i ¼ 1; n’0 ¼ n0 þ ni  1; n’k ¼ nk ðk 6¼ 0; iÞ

References [16] P. Phaiwitthayaphorn, P. Boonsrimuang, P. Reangsuntea, et al., Cell throughput

based sleep control scheme for heterogeneous cellular networks, in: 2017 14th
International Conference on Electrical Engineering/Electronics, Computer,
[1] Yun Li, Shichao Xia, Mengyan Zheng, et al., Lyapunov optimization based trade-off
Telecommunications and Information Technology (ECTI-CON), 2017, pp. 584–587.
policy for mobile cloud offloading in heterogeneous wireless networks, IEEE
[17] Guanding Yu, Qimei Chen, Rui Yin, Dual-threshold sleep mode control scheme for
Transactions on Cloud Computing Early Access Article (2019), 1–1 In preparation.
small cells, IET Commun. 8 (11) (2014) 2008–2016.
[2] Xilong Li, N. Ansari, Dual-battery enabled profit driven user association in green
[18] J. Kim, W.S. Jeon, G.J. Dong, Effect of base station-sleeping ratio on energy
heterogeneous cellular networks, IEEE Transactions on Green Communications and
efficiency in densely deployed femtocell networks, IEEE Commun. Lett. 19 (4)
Networking 2 (4) (Dec. 2018) 1002–1011.
(2015) 641–644.
[3] A. Jahid, S. Hossain, Intelligent energy cooperation framework for green cellular
[19] J. Kim, W.S. Jeon, G.J. Dong, Base-station sleep management in open-access
base stations, in: 2018 International Conference on Computer, Communication,
femtocell networks, IEEE Trans. Veh. Technol. 65 (5) (2016) 3786–3791.
Chemical, Material and Electronic Engineering (IC4ME2), 2018, pp. 1–6.
[20] Yiwei Xu, Jin Chen, An optimal load-aware base station sleeping strategy in small
[4] I. Damak, H. Chaieb, R. Bouallegue, Green handoff in femtocell network, in: 2019
cell networks based on marginal utility, in: 2017 3rd International Conference on
15th International Wireless Communications Mobile Computing Conference
Big Data Computing and Communications (BIGCOM), 2017, pp. 291–296.
(IWCMC), 2019, pp. 1090–1095.
[21] Xiaomeng Chai, Zhongshan Zhang, Keping Long, Joint spectrum-sharing and base
[5] M. Lashgari, B. Maham, H. Kebriaei, Energy efficient price based power allocation
station sleep model for improving energy efficiency of heterogeneous networks,
in a small cell network by using a stackelberg game, in: 2018 IEEE International
IEEE Systems Journal 12 (1) (March 2018) 560–570.
Black Sea Conference on Communications and Networking (BlackSeaCom), 2018,
[22] Shie Wu, Zhimin Zeng, Hailun Xia, Coalition-based sleep mode and power
pp. 1–5.
allocation for energy efficiency in dense small cell networks, IET Commun. 11 (11)
[6] R. Thakur, S.N. Swain, C.S.R. Murthy, An energy efficient cell selection framework
(2017) 1662–1670.
for femtocell networks with limited backhaul link capacity, IEEE Systems Journal
[23] S. Habibi, V. Solouk, H. Kalbkhani, Adaptive sleeping technique to improve energy
12 (2) (June 2018) 1969–1980.
efficiency in ultra-dense heterogeneous networks, in: 2019 5th Conference on
[7] M. Adedoyin, O. Falowo, Joint optimization of energy efficiency and spectrum
Knowledge Based Engineering and Innovation (KBEI), 2019, pp. 782–786.
efficiency in 5G ultra-dense networks, in: 2017 European Conference on Networks
[24] Wenson Chang, Wenyen Cheng, Zhaoting Meng, et al., Energy-efficient sleep
and Communications (EuCNC), 2017, pp. 1–6.
strategy with variant sleep depths for open-access femtocell networks, IEEE
[8] Yun Li, Jie Liu, Bin Cao, et al., Joint optimization of radio and virtual machine
Commun. Lett. 23 (4) (April 2019) 708–711.
resources with uncertain user demands in mobile cloud computing, IEEE Trans.
[25] Jingchu Liu, B. Krishnamachari, Sheng Zhou, et al., DeepNap: data-driven base
Multimed. 20 (9) (2018) 2427–2438.
station sleeping operations through deep reinforcement learning, IEEE Internet of
[9] A. Alnoman, A. Anpalagan, On base station sleeping for heterogeneous cloud-fog
Things Journal 5 (6) (Dec. 2018) 4273–4282.
computing networks, in: 2018 29th Biennial Symposium on Communications (BSC),
[26] L. Saker, S.E. Elayoubi, R. Combes, et al., Optimal control of wake up mechanisms
2018, pp. 1–4.
of femtocells in heterogeneous networks, IEEE J. Sel. Area. Commun. 30 (3) (April
[10] Yun Li, Cao Liao, Chonggang Wang, et al., Energy-efficient optimal relay selection
2012) 664–672.
in cooperative cellular networks based on double auction, IEEE Trans. Wireless
[27] Jinming Hu, Wei Heng, Guodong Zhang, et al., Base station sleeping mechanism
Commun. 14 (8) (2015) 4093–4104.
based on traffic prediction in heterogeneous networks, in: Telecommunication
[11] Heng Wang, Zhenzhen Zhao, Xiaodong Cheng, et al., Base station sleeping strategy
Networks and Applications Conference, 2015, pp. 83–87.
for on-Grid energy saving in cellular networks with hybrid energy supplies in IoT
[28] M.L. Puterman, Markov Decision Processes: Discrete Stochastic Dynamic
environment, IEEE Access 6 (2018) 45578–45589.
Programming, John Wiley and Sons, Inc., 2005.
[12] J. Kim, H. Lee, S. Chong, Traffic-aware energy-saving base station sleeping and
[29] Qingmin Wang, F.R. Yu, Yi Sun, Optimal joint base station and user equipment (BS-
clustering in cooperative networks, IEEE Trans. Wireless Commun. 17 (2) (Feb.
UE) admission control for energy-efficient green wireless cellular networks, in:
2018) 1173–1186.
2012 IEEE Global Communications Conference (GLOBECOM), 2012,
[13] O. Onireti, A. Mohamed, H. Pervaiz, et al., Analytical approach to base station sleep
pp. 2119–2124.
mode power consumption and sleep depth, in: 2017 IEEE 28th Annual International
[30] Hui Ding, Haixia Zhang, Jie Tian, et al., Energy efficient user association and power
Symposium on Personal, Indoor, and Mobile Radio Communications (PIMRC),
control for dense heterogeneous networks, in: 2018 International Conference on
2017, pp. 1–7.
Computing, Networking and Communications (ICNC), 2018, pp. 741–746.
[14] Juwo Yang, Xing Zhang, Wenbo Wang, et al., Traffic adaptive base station sleeping
[31] G. Auer, V. Giannini, C. Desset, et al., How much energy is needed to run a wireless
control in inhomogeneous network, in: 2018 Annual IEEE International Systems
network? IEEE Wireless Communications 18 (5) (October 2011) 40–49.
Conference (SysCon), 2018, pp. 1–5.
[15] Xiaodong Xu, Chunjing Yuan, Wenwan Chen, et al., Adaptive cell zooming and
sleeping for green heterogeneous ultradense networks, IEEE Trans. Veh. Technol.
67 (2) (Feb. 2018) 1612–1621.

11