376 IEEE TRANSACTIONS ON MOBILE COMPUTING, VOL. 19, NO.

2, FEBRUARY 2020

Low Complexity Online Radio Access

Technology Selection Algorithm
in LTE-WiFi HetNet
Arghyadip Roy , Student Member, IEEE, Vivek Borkar , Fellow, IEEE,
Prasanna Chaporkar , Member, IEEE, and Abhay Karandikar ,, Member, IEEE

Abstract—In an offload-capable Long Term Evolution (LTE)- Wireless Fidelity (WiFi) Heterogeneous Network (HetNet), we consider
the problem of maximization of the total system throughput under voice user blocking probability constraint. The optimal policy is
threshold in nature. However, computation of optimal policy requires the knowledge of the statistics of system dynamics, viz., arrival
processes of voice and data users, which may be difficult to obtain in reality. Motivated by the Post-Decision State (PDS) framework to
learn the optimal policy under unknown statistics of system dynamics, we propose, in this paper, an online Radio Access Technology
(RAT) selection algorithm using Relative Value Iteration Algorithm (RVIA). However, the convergence speed of this algorithm can be
further improved if the underlying threshold structure of the optimal policy can be exploited. To this end, we propose a novel structure-
aware online RAT selection algorithm which reduces the feasible policy space, thereby offering lesser storage and computational
complexity and faster convergence. This algorithm provides a novel framework for designing online learning algorithms for other
problems and hence is of independent interest. We prove that both the algorithms converge to the optimal policy. Simulation results
demonstrate that the proposed algorithms converge faster than a traditional scheme. Also, the proposed schemes perform better than
other benchmark algorithms under realistic network scenarios.

Index Terms—User association, LTE-WiFi offloading, constrained MDP, threshold policy, stochastic approximation

1 INTRODUCTION

R ECENT developments in wireless communications have

witnessed a proliferation of end-user equipment, such
as tablets and smartphones, with advanced capabilities.
The ever-increasing demand for Quality of Service (QoS)
requirement of users gives rise to the standardization of var-
ious Radio Access Technologies (RATs) [2], such as Univer-
sal Mobile Telecommunications System (UMTS), Long
Term Evolution (LTE) etc. According to [3], monthly
global mobile data traffic is expected to exceed 49 exa-
bytes by 2021. One of the basic limitations of the present
RATs is the lack of support for the co-existence of multi-
ple RATs. Therefore, to address this exceptional growth
of data traffic, network providers have expressed their
interest towards efficient interworking among various
RATs in the context of upcoming Fifth Generation (5G)
[2] network. This gives rise to the notion of Heteroge-
neous Network (HetNet), where different RATs coexist
and interwork with each other. Users can be associated
with any candidate RAT and moved from one RAT to

A. Roy, V. Borkar, and P. Chaporkar are with the Department of Electrical
Engineering, Indian Institute of Technology Bombay, Mumbai, Mahara-
shtra 400076, India. E-mail: {arghyadip, borkar, chaporkar}@ee.iitb.ac.in.

A. Karandikar is with the Department of Electrical Engineering, Indian
Institute of Technology Bombay, Mumbai, Maharastra 400076, India, and
is the director, Indian Institute of Technology Kanpur, Kanpur, Uttar Pra-
desh 208016, India. E-mail: karandi@ee.iitb.ac.in, karandi@iitk.ac.in.
Manuscript received 24 Aug. 2018; revised 17 Dec. 2018; accepted 7 Jan.
2019. Date of publication 14 Jan. 2019; date of current version 6 Jan. 2020.
(Corresponding author: Arghyadip Roy.)
Digital Object Identifier no. 10.1109/TMC.2019.2892983
another in a seamless manner. Recent advances in Soft-
ware Defined Networking (SDN) [4], [5] paradigm also
facilitates these mechanisms by bringing more flexibility
in the integrated control and management of various
RATs.
In a Third Generation Partnership Project (3GPP) LTE
HetNet, interworking with IEEE 802.11 [6] Wireless Local
Area Network (WLAN) (popularly known as Wireless
Fidelity (WiFi)) operating in unlicensed band provides a
potential solution due to the complementary nature of these
RATs. The LTE Base Stations (BSs) can be deployed to pro-
vide ubiquitous coverage, whereas the WiFi Access Points
(APs) offer high bit rate solution in hot-spot regions. A user
can be associated with either LTE or WiFi in an area where
their coverage overlaps. Furthermore, according to 3GPP
Release 12 specifications [7], data users can be steered from
one RAT to another. This mechanism is known as mobile
data offloading.
Intelligent RAT selection1 and offloading decisions [8],
[9], [10], [11], [12], [13], [14], [15], [16], [17], [18] may lead to
efficient resource utilization and can be triggered either by
the user or by the network. User-initiated decisions are
taken with an objective of optimizing individual utilities
and hence, may not converge to the global optimality.
Therefore, we focus on network-initiated association and
offloading schemes which target to optimize different net-
work level system parameters.
1. The terminologies “association” and “RAT selection” are used
interchangeably throughout this paper.

1536-1233 ß 2019 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission.
See ht_tp://www.ieee.org/publications_standards/publications/rights/index.html for more information.

Authorized licensed use limited to: UNIVERSITY PUTRA MALAYSIA. Downloaded on March 10,2020 at 05:36:03 UTC from IEEE Xplore. Restrictions apply.
ROY ET AL.: LOW COMPLEXITY ONLINE RADIO ACCESS TECHNOLOGY SELECTION ALGORITHM IN LTE-WIFI HETNET
Fig. 1. LTE-WiFi HetNet architecture.
In this paper, we propose online learning algorithms to
obtain the optimal RAT selection policy for an LTE-WiFi Het-
Net. The model considered in this paper is similar to that of
[19]. As illustrated in Fig. 1, the centralized controller (for
example, an SDN controller) possesses an overall view of
both the networks and is responsible for taking network-initi-
ated RAT selection and offloading decisions. We consider
voice and data users inside the LTE-WiFi HetNet. Voice users
are always associated with LTE since voice users require QoS
guarantee which may not be provided by WiFi. We assume
that the data users can be associated with either LTE or WiFi.
Additionally, we assume that data users can be offloaded
from one RAT to another, whenever a voice user is associ-
ated, or an existing voice/data user departs.
Total system throughput is an important system parameter
from a network operator’s perspective since the throughput
experienced by the data users may have a significant impact
on the profit and customer base of the operator. From the per-
spective of maximization of total system throughput, WiFi
generally provides more throughput than that of LTE when
WiFi load is less. However, the total throughput in WiFi
decreases as the load increases [20]. Therefore, although in
low load, WiFi may be preferable to data users for association,
as the load increases, LTE may be preferred. LTE resources
are shared between voice and data users from a common pool
of resources. Throughput requirement of the LTE data users
is usually more than that of the voice users. Therefore, LTE
may reserve some of the resources, which otherwise would
not be efficiently utilized by low throughput voice users, for
data users. Thus, total system throughput maximization may
result in excessive blocking of voice users. Roy et al. [19]
addresses this inherent trade-off between the total system
throughput and blocking probability of voice users by formu-
lating this problem as a constrained average reward continu-
ous time Markov Decision Process (MDP), which maximizes
the total system throughput subject to a constraint on the voice
user blocking probability. In this paper, we propose online
learning algorithms which can be implemented without the
knowledge of statistics of arrival processes of voice and data
users to obtain the optimal RAT selection policy for an LTE-
WiFi HetNet.
One of the main advantage of learning is that it avoids
explicit estimation which may have high variance or may be
computationally prohibitive. It replaces the conditional
averaging in an iterative scheme for solving the dynamic
programming equation by an actual evaluation at an
observed transition and an incremental update which does
the conditional averaging implicitly. Also, our scheme has
the advantage that it exploits the known threshold structure
of the optimal policy to further reduce the computational
complexity, and is one of the first to do so.
Authorized licensed use limited to: UNIVERSITY PUTRA MALAYSIA. Downloaded on March 10,2020 at 05:36:03 UTC from IEEE Xplore. Restrictions apply.
377
1.1 Related Work
RAT selection and offloading techniques in HetNets can be
either user-initiated [8], [9], [10], [11], [12], [13] or network-
initiated [14], [15], [16], [17], [18], [19], [21], [22] in nature. In
[8], authors propose a user-initiated RAT selection algo-
rithm in an LTE-WiFi HetNet. The proposed algorithm
addresses the trade-off between resource utilization and
QoS. The problem where each user selfishly chooses a RAT
with an objective of maximizing its individual throughput
is considered in [10] and formulated as a non-cooperative
game. Lee et al. [11] proposes a heuristic “on-the-spot off-
loading” scheme where a data user is always steered to
WiFi, whenever it is inside the coverage area of WiFi APs.
Among the network-initiated approaches, in [15], opti-
mal client-AP association problem in a WLAN is considered
within the framework of continuous-time MDP, and user
arrival/departure is taken as a feasible decision epoch. In
[22], authors propose an optimal RAT selection algorithm
aiming at maximizing the generated revenue. However, the
proposed algorithm scales exponentially with the system
size. Authors in [16] propose distributed association algo-
rithms by formulating the association problem as a non-
cooperative game and compare the performance with the
centralized globally optimal scheme. A context-aware RAT
selection algorithm is proposed in [17]. The proposed algo-
rithm, which can be implemented on the user side, albeit
with network assistance, minimizes signaling overhead as
well as base station computations.
While the above solutions provide significant insight into
RAT selection and offloading strategies, none of them spe-
cifically focus on computational efficiency. Therefore, prac-
tical implementations of the proposed algorithms become
infeasible. There are two main driving factors behind this
issue. First, standard dynamic programming techniques
to solve optimal RAT selection and offloading problems
become computationally inefficient and thus difficult to
implement when the state space is large. Furthermore, we
need to know the statistics of arrival processes of data
and voice users which govern the transition probabilities
between different states, in order to determine the optimal
policy. In practice, this may be difficult to obtain. Recent
studies [23], [24], [25] on the characteristics of cellular traffic
reveal that although the voice traffic can be predicted accu-
rately, the state-of-art prediction schemes for data traffic are
not very satisfactory. Therefore, obtaining real-time traffic
statistics in today’s cellular network is very difficult.
In the case of unknown system statistics, we may resort to
Reinforcement Learning (RL) [26] algorithms which deter-
mine the optimal policy using a trial-and-error method.
Refs. [1], [21], [27], [28] adopt RL based schemes which can be
implemented online. Q-learning [1], [21], [27] is a traditional
RL algorithm which learns the optimal policy under
unknown system dynamics. Our preliminary work [1] under-
takes a Q-learning based approach to determine the optimal
policy. Authors in [21] aim to improve the network perfor-
mance and user experience jointly and formulate the RAT
selection problem as a Semi-Markov Decision Process
(SMDP) problem. A Q-learning based approach is also
adopted since network parameters may be difficult to obtain
in reality. Based on locally available information at users and
following a Q-learning approach, [27] undertakes distributed
traffic offloading decisions. Although the convergence to opti-
mality is guaranteed, these learning schemes need to itera-
tively learn the value functions for all state-action pairs, thus
378
possessing large memory requirement. Additionally, due to
associated exploration mechanism, their convergence rate is
slow, especially under large state space.
1.2 Our Contribution
In this paper, our primary contribution is to propose online
learning algorithms to maximize the total system through-
put subject to a constraint on the voice user blocking proba-
bility without knowing the statistics of arrival processes of
voice/data users in an LTE-WiFi HetNet. To address the
issue of slow convergence of existing learning schemes in
the literature, in this paper, we propose a Post-Decision
State (PDS) learning algorithm which speeds up the learn-
ing process by removing the action exploration. This
approach is based on reformulation of the Relative Value
Iteration Algorithm (RVIA) equation and can be imple-
mented online in the Stochastic Approximation (SA) frame-
work. Furthermore, the PDS learning algorithm has a lower
space complexity than that of Q-learning [1] because instead
of the state-action pair values, we need to store the value
functions associated with states alone. We also prove the
convergence of the PDS learning RAT selection algorithm to
the optimality.
We have shown in [19] that the optimal policy has a
threshold structure, wherein after a certain threshold on the
number of WiFi data users, data users are served using
LTE. A similar property exists for the admission of voice
users [19], where after a certain threshold on the number of
LTE data and voice users, voice users are blocked. In this
paper, we exploit the threshold properties in [19] and pro-
pose a structure-aware learning algorithm which, instead of
the entire policy space, searches the optimal policy only
from the set of threshold policies. This reduces the conver-
gence time as well as the computational and storage com-
plexity in comparison to that of the proposed PDS learning
algorithm. We prove that the threshold vector iterates in the
proposed structure-aware learning algorithm indeed con-
verge to the globally optimal solution. Note that the analyti-
cal methodologies presented in this paper to learn the
optimal threshold policy are developed independently and
can be applied to any learning problem where the optimal
policy is threshold in nature.
Although we make some simplifying assumptions to
facilitate the analysis, performance of the proposed schemes
are studied in realistic conditions without the simplifying
assumptions. Extensive simulations are conducted in ns-3
[29], a discrete event network simulator, to characterize the
performance of the proposed algorithms. It is observed
through simulations that the proposed structure-aware
RAT selection online learning algorithm outperforms the
PDS learning algorithm, providing faster convergence to
optimality. Performance comparison of the proposed algo-
rithms is made with Q-learning based RAT selection
algorithm [1]. Furthermore, we observe that the proposed
algorithms outperform other benchmark algorithms under
realistic network scenarios like presence of channel fading,
dynamic resource scheduling and user mobility. We use
3GPP recommended parameters for the simulations.
Our key contributions can be summarized as follows.

Based on the PDS paradigm, we propose an online
algorithm for RAT selection in an LTE-WiFi HetNet.
The convergence proof for the proposed algorithm is
provided.
Authorized licensed use limited to: UNIVERSITY PUTRA MALAYSIA. Downloaded on March 10,2020 at 05:36:03 UTC from IEEE Xplore. Restrictions apply.
IEEE TRANSACTIONS ON MOBILE COMPUTING, VOL. 19, NO. 2, FEBRUARY 2020

We further exploit the threshold nature of optimal
policies [19] and propose a novel structure-aware
online association algorithm. Theoretical and simula-
tion results indicate that the knowledge of threshold
property helps in achieving reductions in storage
and computational complexity as well as in conver-
gence time. We also prove that the proposed scheme
converges to the true value function.

The proposed structure-aware algorithm provides a
novel framework that can be applied for designing
online learning algorithms for other problems and
hence is of independent interest.

Performances of the proposed algorithms are com-
pared with other online algorithms in the literature [1].

Performances of the proposed algorithms are com-
pared with other benchmark RAT selection algo-
rithms under realistic scenarios.
The rest of the paper is organized as follows. Section 2
describes the system model. In Section 3, the problem for-
mulation within the framework of constrained MDP is
described. The system model and formulation adopted in our
paper is analogous to [1], [14], [19]. The developments
described after Section 3 is our point of departure. We intro-
duce the notion of PDS in Section 4. Sections 5 and 6 propose
PDS learning algorithm and structure-aware learning
algorithm, respectively, for RAT selection in an LTE-WiFi
HetNet. A comparison of computational and storage com-
plexities of the proposed and traditional algorithms is pro-
vided in Section 7. Simulation results are presented in
Section 8, followed by conclusions in Section 9. The proofs are
available in the supplemental materials file, which is available
in the IEEE Computer Society Digital Library at http://doi.
ieeecomputersociety.org/10.1109/TMC.2019.2892983.
2 SYSTEM MODEL
As demonstrated in Fig. 1, we consider a system consisting
of a WiFi AP inside the coverage area of an LTE BS, both
connected to a centralized controller using ideal lossless
links. We assume that voice and data users are present at
any geographical point in the coverage area of the LTE BS.
Data users outside the common coverage area of the LTE BS
and the WiFi AP always get associated with the LTE BS.
Therefore no decision is involved in this case. Hence with-
out loss of generality, we take into account only those data
users which are present in the dual coverage area of the
LTE BS and the WiFi AP. Data users can be associated with
either the LTE BS or the WiFi AP. We assume that in LTE,
voice and data users are allocated resources from a common
resource pool. We assume that voice and data user arrivals
are Poisson processes with means
v and
d , respectively.
Service times for voice and data users follow exponential
distributions with means m1v and m1 , respectively. Assump-
d
tions on service times follow justifications in [30]. All the
users are assumed to be stationary.
Remark 1. For brevity of notation, we have considered a
single LTE BS and a single WiFi AP. However, the system
model can be generalized to multiple LTE BSs and WiFi
APs with small modifications. When the coverage areas
of multiple APs/BSs do not overlap, we need to consider
the number of users in each AP/BS in the state space. In
case of multiple overlapping APs/BSs, we can construct a
ROY ET AL.: LOW COMPLEXITY ONLINE RADIO ACCESS TECHNOLOGY SELECTION ALGORITHM IN LTE-WIFI HETNET 379

one-to-one mapping between a user location and an TABLE 1

AP/a BS using a simple criterion such as highest aver- Transition Probability Table
age signal strength, etc. Using this criterion for multiple
overlapping APs, the problem can be reduced to single ajEl ði0 ; j0 ; k0 Þ
BS and multiple non-overlapping AP problem. The set A1 jðE1 jj . . . jjE5 Þ ði; j; kÞ
where more than one APs have equal signal strength is A2 jE1 ði þ 1; j; kÞ
non-generic in the sense that in the associated parameter A2 jE2 ði; j þ 1; kÞ
space it has Lebesgue measure 0. A3 jE2 ði; j; k þ 1Þ
A4 jE1 ði þ 1; j  1; k þ 1Þ
A5 jðE3 jjE4 Þ ði; j þ 1; k  1Þ
2.1 State & Action Space A5 jE5 ði; j  1; k þ 1Þ
The system can be modeled as a controlled continuous
time stochastic process fXðtÞgt0 . Any state s in the state
space S is symbolically represented as s ¼ ði; j; kÞ, where
i; j represent the number of voice and data users in LTE, case of departure of voice/data users, the set of all possible
respectively. The number of data users in WiFi is denoted actions is fA1 ; A5 g. However, depending on the system
by k. The arrival and departure of voice and data users are state, one or more actions may be infeasible. Note that
taken as decision epochs. Due to the Markovian nature of blocking is considered as a feasible action for voice (data)
the system, it suffices to observe the system state at each of users, only when the system is non-empty (the capacity is
the decision epochs. Note that the system state changes only reached for both LTE and WiFi).
at these decision epochs. Therefore, it is not required to con-
sider the system state at other points in time. 2.2 State Transitions
The state of the system changes whenever there is an From each state s, under an action a, the system makes a
arrival or a departure of voice/data users, referred to as transition to a different state s0 with positive probability
events. We consider five type of events, namely, an arrival pss0 ðaÞ. In state s ¼ ði; j; kÞ, let the sum of arrival and service
of a voice user in the system (E1 ), an arrival of a data user in rates of users be denoted by vði; j; kÞ. Therefore, we have
the system (E2 ), a departure of an existing voice user from
LTE (E3 ), a departure of an existing data user from LTE (E4 ) vði; j; kÞ ¼
v þ
d þ imv þ jmd þ kmd :
and a departure of an existing data user from WiFi (E5 ). At
every decision epoch, the centralized controller takes a deci- Then
sion based on the present system state. Based on the chosen 8

>
>
v
; s0 ¼ ði0 ; j0 ; k0 Þ;
action, the system moves to different states with finite > vði0 ;j0 ;k0 Þ
>
>
>
probabilities. >
>

d
vði0 ;j0 ;k0 Þ ; s0 ¼ ði0 ; j0 ; k0 Þ;
>
< 0
We assume that ði; j; kÞ 2 S if ði þ jÞ  C and k  W , im
where C is the total number of common resource blocks for pss0 ðaÞ ¼ vði0 ;j0v;k0 Þ ; s0 ¼ ði0  1; j0 ; k0 Þ;
>
>
voice and data users in LTE resource pool, and W denotes the >
> j00 m0d 0 ;
>
> s0 ¼ ði0 ; j0  1; k0 Þ;
maximum number of users in WiFi to guarantee that the aver- > vði ;j ;k Þ
> 0
>
: k md ;
age per-user throughput in WiFi is more than a certain thresh- vði0 ;j0 ;k0 Þ s0 ¼ ði0 ; j0 ; k0  1Þ:
old. Note that, the per-user throughput in WiFi monotonically
decreases with the number of WiFi data users [20]. The first Values of i0 ; j0 and k0 as a function of different actions a
condition signifies that each admitted user is provided a sin- (conditioned on events El ) are summarized in Table 1.
gle resource block, whenever resources are available.
Remark 2. For sake of simplicity, we have considered sin- 2.3 Rewards and Costs
gle resource block allocation to LTE users. Although the Let the reward and cost functions per unit time for state s
consideration of multiple resource blocks mimics the and action a be denoted by rðs; aÞ and cðs; aÞ, respectively.
practical scenario better, it complicates the system model We assume that before the data transfer starts, a Transmission
without bringing much difference in the formulation Control Protocol (TCP) connection is built between the data
developed in this paper. user and the LTE BS (WiFi AP). Let RL;V and RL;D denote the
bit rates of voice and data users in LTE, respectively. We
Let the action space, i.e., the set of all possible association assume that RL;D is the maximum data rate provided to the
and offloading strategies in case of arrival and departure of data users by the TCP pipe in LTE. In LTE network, generally
users, be denoted by A. We describe A below: voice user generates traffic at Constant Bit Rate (CBR). There-
8 fore, we assume RL;V to be a constant. RW;D ðkÞ denotes the
>
> A1 ; Block the arriving user or do nothing per-user data throughput of k users in WiFi, assuming the full
>
>
>
> during departure; buffer traffic WiFi model [20]. The calculation of RW;D ðkÞ takes
>
>
> A2 ;
> Accept voice/data user in LTE, into account factors like the contention-based medium access
<
A3 ; Accept data user in WiFi, of WiFi users, success and collision probabilities as well as
A¼
>
> A4 ; Accept voice user in LTE and offload one slot times for successful transmission, idle slots and busy slots
>
>
>
> data user to WiFi; corresponding to collisions.
>
>
>
> A ; Move one data user to a RAT (from which The reward rate for a state-action pair, which is a
: 5
departure has occurredÞ: monotone increasing function of the number of voice
and data users in LTE, is defined as the total throughput
In case of voice and data user arrivals, the sets of all possible of the system in the state under the corresponding
actions are fA1 ; A2 ; A4 g and fA1 ; A2 ; A3 g, respectively. In action. The entire description of reward rates in state

Authorized licensed use limited to: UNIVERSITY PUTRA MALAYSIA. Downloaded on March 10,2020 at 05:36:03 UTC from IEEE Xplore. Restrictions apply.
380 IEEE TRANSACTIONS ON MOBILE COMPUTING, VOL. 19, NO. 2, FEBRUARY 2020

TABLE 2 3.2 Equivalent Discrete-Time MDP and Lagrangian

Reward Rate Table Approach
To obtain the optimal policy using RVIA [32], we need to
ðajEl Þ rðs; aÞ employ the Lagrangian approach [31]. In this approach, for
ðA1 jE1 Þ iRL;V þ jRL;D þ kRW;D ðkÞ a fixed value of Lagrange Multiplier (LM) b, the reward
ðA1 jE2 Þ iRL;V þ jRL;D þ kRW;D ðkÞ function is given by
ðA1 jE3 Þ ði  1ÞRL;V þ jRL;D þ kRW;D ðkÞ
ðA1 jE4 Þ iRL;V þ ðj  1ÞRL;D þ kRW;D ðkÞ rðs; a; bÞ ¼ rðs; aÞ  bcðs; aÞ:
ðA1 jE5 Þ iRL;V þ jRL;D þ ðk  1ÞRW;D ðk  1Þ
ðA2 jE1 Þ ði þ 1ÞRL;V þ jRL;D þ kRW;D ðkÞ The dynamic programming equation described below pro-
ðA2 jE2 Þ iRL;V þ ðj þ 1ÞRL;D þ kRW;D ðkÞ vides the necessary condition for optimality in case of
ðA3 jE2 Þ iRL;V þ jRL;D þ ðk þ 1ÞRW;D ðk þ 1Þ SMDP 8s 2 S, where s0 2 S
ðA4 jE1 Þ ði þ 1ÞRL;V þ ðj  1ÞRL;D þ ðk þ 1ÞRW;D ðk þ 1Þ
X
ðA5 jE3 Þ ði  1ÞRL;V þ ðj þ 1ÞRL;D þ ðk  1ÞRW;D ðk  1Þ V ðsÞ ¼ max½rðs; a; bÞ þ pss0 ðaÞV ðs0 Þ  rtðs; aÞ;
ðA5 jE4 Þ iRL;V þ jRL;D þ ðk  1ÞRW;D ðkÞ a
s0
ðA5 jE5 Þ iRL;V þ ðj  1ÞRL;D þ kRW;D ðkÞ
where V ðsÞ; r; tðs; aÞ denote the value function of state
s 2 S, the optimal average reward and the mean transition
s ¼ ði; j; kÞ for different events and actions is provided in time from state s under the action a, respectively.
Table 2. Whenever the centralized controller blocks an Since the sojourn times are exponential, this is a special
incoming voice user, the cost rate is one unit, else it is case of continuous time controlled Markov chain for which
zero. Therefore we have
 X
1; if voice user is blocked; 0 ¼ max½rðs; a; bÞ  r þ qðs0 js; aÞV ðs0 Þ; (2)
cðs; aÞ ¼ a
s0
0; otherwise:
where qðs0 js; aÞ are controlled P transition rates satisfying
qðs0 js; aÞ  0 for s0 6¼ s and s0 qðs0 js; aÞ ¼ 0. If we scale all
3 PROBLEM DESCRIPTION
the transition rates by a positive scalar, it amounts to time
Association policy is a sequence of decision rules which scaling which scales the average reward accordingly for
describes actions to be chosen at different states and every policy including the optimal, but does not change the
decision epochs. Maximization of total system through- optimal policy. Thus, without loss of generality, we assume
put may result in blocking of voice users as the contribu- that qðsjs; aÞ 2 ð0; 1Þ 8a, implying in particular that
tion of voice users towards the total system throughput qðs0 js; aÞ 2 ½0; 1 for s0 6¼ s. Adding V ðsÞ to both sides of
is less than that of data users. Therefore, to address the Equation (2), we have
inherent trade-off between the total system throughput X
and the voice user blocking probability, we aim to obtain V ðsÞ ¼ max½rðs; a; bÞ  r þ pss0 ðaÞV ðs0 Þ; (3)
a
an association policy which maximizes the total system s0
throughput, subject to a constraint on the voice user
blocking probability. Since arrival and departure of users where pss0 ðaÞ ¼ qðs0 js; aÞ for s0 6¼ s and pss0 ðaÞ ¼ 1 þ qðs0 js; aÞ
can happen at any point in time, this problem can be for s0 ¼ s (recall that qðsjs; aÞ is negative). This equation is
formulated as a constrained continuous time MDP. It is the DP equation for a discrete time MDP (say fXn g) with
well-known that a randomized stationary optimal policy, controlled transition probabilities pss0 ðaÞ. Here onwards we
i.e., a mixture of pure policies with associated probabili- focus on discrete time setting as described in Equation (3).
ties, exists [31]. For a fixed value of b, the following equation describes
how RVIA can be used to solve the equivalent uncon-
strained maximization problem
3.1 Problem Formulation
Let M be the set of all memoryless policies. We assume that X
Vnþ1 ðsÞ ¼ max½rðs; a; bÞ þ pss0 ðaÞVn ðs0 Þ  Vn ðs Þ; (4)
the Markov chains constructed under such policies are uni- a
s0
chain and therefore have unique stationary distributions. Let
the average reward and cost of the system over infinite hori- where Vn ð:Þ is an estimate of the value function after nth
zon under the policy M 2 M be denoted by V M and BM , iteration and s is an arbitrary but fixed state. Next, we aim
respectively. Let RðtÞ and CðtÞ be the total reward and cost to determine the value of b (¼ b , say) which maximizes the
of the system up to time t, respectively. For the constrained average expected reward, subject to the cost constraint. The
MDP problem, our objective is described as follows: following gradient descent algorithm describes the rule to
update the value of b
1
Maximize: V M ¼ lim EM ½RðtÞ; 1
t!1 t bkþ1 ¼ bk þ ðBpbk  Bmax Þ;
(1) k
1
subject to: BM ¼ lim EM ½CðtÞ  Bmax ; where bk is the value of b in kth iteration, and Bpbk is the
t!1 t
voice user blocking probability at kth iteration. Once the
where EM denotes the expectation operator under policy M, value of b is determined, we obtain the optimal policy by
and Bmax denotes the constraint on the blocking probability employing a perturbation of b by a small amount  in both
of voice users. As we know that the optimal policies are sta- directions (policies pb  and pb þ , say) with associated
tionary, the limits in Equation (1) exist. costs Bb  and Bb þ , respectively. Finally, we have a

Authorized licensed use limited to: UNIVERSITY PUTRA MALAYSIA. Downloaded on March 10,2020 at 05:36:03 UTC from IEEE Xplore. Restrictions apply.
ROY ET AL.: LOW COMPLEXITY ONLINE RADIO ACCESS TECHNOLOGY SELECTION ALGORITHM IN LTE-WIFI HETNET 381

state s0 be denoted by pð^ s; s0 Þ. The post-decision Bellman

equation for the post-decision state s^ ¼ ð^x; y^; z^Þ 2 S is
X
0 0
V ð^
^ sÞ ¼ s; s Þ max½rðs ; a; bÞ þ V^ðs^0 Þ  r;
pð^ (5)
a
s0
Fig. 2. Transition among PDSs and pre-decision states.
where s^0 is the post-decision state when action a in chosen in
pre-decision state s0 . Using Equation (5), the RVIA based
randomized optimal policy where the the policies pb  and
update rule is as follows:
pb þ are chosen with probability p and ð1  pÞ, such that
X
V^nþ1 ð^
sÞ ¼ pð^ s; s0 Þ max½rðs0 ; a; bÞ þ V^n ðs^0 Þ  V^n ðs^ Þ;
pBb  þ ð1  pÞBb þ ¼ Bmax : s 0
a
(6)
We know [33] that the optimal stationary policy can be ran- V^nþ1 ðs^00 Þ ¼V^n ðs^00 Þ 8s^00 6¼ s^;
domized in at most one s 2 S where the optimal action is
randomized between two actions. where s^ is the PDS associated with the nth decision epoch
The gradient descent scheme for b converges to b . In and s^ is a fixed PDS. The idea is to update one component
Equation (1), instead of considering limiting values of point- at a time and keep the others unchanged. This idea is trans-
wise reward and cost, we consider limiting values of average lated into an online algorithm stated below which updates
reward and cost. Therefore, no relaxation of the constraint is the value function of the PDS associated with the current
performed, and the obtained solution is optimal. decision epoch.

4 POST-DECISION STATE FRAMEWORK 5 ONLINE RAT SELECTION ALGORITHM

As discussed in the previous section, the optimal policy can The system changes states based on different events, i.e., the
be determined using RVIA (Equation (4)), if the transition arrival/departure of users and various actions taken in dif-
probabilities pss0 ðaÞ are known beforehand. However, ferent states. Since we do not know the arrival rates of voice
knowledge of transition probabilities requires the knowl- and data users, and the max operator occurs outside the
edge of statistics of arrival processes of voice and data users. averaging operation with respect to the transition probabili-
In reality, it may be difficult to obtain these parameters [23], ties of the underlying Markov chain in Equation (4), online
[24], [25]. Therefore we aim to devise an online scheme implementation of the same is not feasible. However, in
which does not require the knowledge of statistics of arrival Equation (6), the expectation operation which resides out-
process and can still converge to the optimal solution. How- side the max operation can be replaced by averaging over
ever, before describing the online algorithm, we introduce time to estimate the optimal value function of the PDSs.
the notion of the PDS. Using the theory of SA [34], we can remove the expectation
A PDS is defined to be an imaginary state of the system operation in Equation (6) and still converge to optimality in
just after an action is chosen and before the unknown sys- policy by doing averaging over time.
tem dynamics (noise) adds into the system. The idea behind Let gðnÞ be a positive step-size sequence possessing the
PDS is to factor the transition from one state to another into following properties:
known and unknown components. The known component X
1 X
1
consists of the effect of the action taken in a state, whereas gðnÞ ¼ 1; ðgðnÞÞ2 < 1: (7)
the unknown component comprises the unknown random n¼1 n¼1
dynamics of the system (viz., the arrival and departure of Let hðnÞ be another step-size sequence having the same
voice and data users). Let us assume that the state of the sys- properties as that of Equation (7) along with the additional
tem is s ¼ ði; j; kÞ 2 S at some decision epoch. Based on the property
chosen action, the system moves to the post-decision state
s^ ¼ ði;
^ j; ^ 2 S. Based on the next event, the system moves
^ kÞ hðnÞ
lim ! 0: (8)
to the actual pre-decision state s0 ¼ ði0 ; j0 ; k0 Þ 2 S. Through- n!1 gðnÞ
out this paper, whenever we refer to a “state”, we always
refer to a pre-decision state. An example transition involv- The key idea is to update the value function associated with
ing pre-decision states and PDSs is illustrated in Fig. 2. one PDS at a time and keep the other PDS values unchanged.
Under action A3 , the system makes a transition from state Let Yn be the PDS P which is updated at nth iteration. Also,
s ¼ ði; j; kÞ to PDS s^ ¼ ði; j; k þ 1Þ. Under the next event E3 , s; nÞ ¼ nm¼0 If^
define gð^ s ¼ Yn g, i.e., number of times PDS s^
the system moves from the PDS s^ to the pre-decision state is updated till nth iteration. The scheme is as follows:
s0 ¼ ði  1; j; k þ 1Þ. In other words, the known information
regarding the transition from state s to s0 is incorporated in V^nþ1 ð^
sÞ ¼ ð1  gðgð^
s; nÞÞÞV^n ð^ s; nÞÞfmax½rðs0 ; a; bÞ
sÞ þ gðgð^
a
PDS s^. On the other hand, transition from PDS s^ to state s0 þ V^n ðs^0 Þ  V^n ðs^ Þg; (9)
consists only of the unknown system dynamics which is not
included in the PDS. Let V^ð^ sÞ be the value function associ- Vnþ1 ðs Þ ¼ V^n ðs^00 Þ 8s^00 6¼ s^:
^ ^00

ated with the PDS s^ 2 S. Thus, we have
sÞ ¼ Es0 ½V ðs0 Þ;
V^ð^
where the expectation Es0 is taken over all the pre-decision
states which are reachable from the post-decision state s^.
Let the transition probability from PDS s^ to pre-decision
However, the scheme (9) is a primal RVIA algorithm which
solves a dynamic programming equation for a fixed value
of LM b. To obtain optimality in b, b is to be iterated along
the timescale hðnÞ, as described below:
bnþ1 ¼ L½bn þ hðnÞðBn  Bmax Þ; (10)

Authorized licensed use limited to: UNIVERSITY PUTRA MALAYSIA. Downloaded on March 10,2020 at 05:36:03 UTC from IEEE Xplore. Restrictions apply.
382
where the projection operator L projects the value of LM to
the interval ½0; L for a large L > 0. Therefore, the primal-
dual RVIA can be described as follows.
If the system is at PDS s^ at the nth iteration, then do the
following:
V^nþ1 ð^
sÞ ¼ ð1  gðgð^
s; nÞÞÞV^n ð^ s; nÞÞfmax½rðs0 ; a; bÞ
sÞ þ gðgð^
a
þ V^n ðs^0 Þ  V^n ðs^ Þg;
V^nþ1 ðs^00 Þ ¼ V^n ðs^00 Þ 8s^00 6¼ s^;
bnþ1 ¼ L½bn þ hðnÞðBn  Bmax Þ:
The assumptions on gðnÞ and hðnÞ (Equations (7) and (8))
ensure that two quantities are updated on two different
timescales. The value of LM is updated on a slower time-
scale than that of the value function. From the slower LM
timescale point of view, V^ð^
sÞ appears to be equilibrated in
accordance with the current LM value, and from the faster
timescale view, LM appears to be almost constant. This
two-timescale scheme induces a “leader-follower” behavior.
The slow (fast) timescale iterate does not interfere in the
convergence of the fast (slow) timescale iterate.
Theorem 1. The schemes (11)-(12) converges to (V^; b ) “almost
surely”(a.s.).
Proof. Proof is provided in Section 10.1. in the supplemen-
tal material file. u
t data users using WiFi until k ¼ W , where an incom-
Based on the analysis presented above, the two timescale
PDS online learning algorithm is described in Algorithm 1.
As described in the algorithm, value functions associated
with different states, the LM and the number of iterations
are initialized at the beginning. Based on a random event
(arrival or departure of voice/data user), the system state is
initialized. When the current PDS of the system is s^, the sys-
tem chooses an action which maximizes the R.H.S expres-
sion in Equation (9). Based on the observed reward in the
current PDS s^0 , V^ð^
sÞ is updated along with the LM. This
process is repeated for every decision epoch.
Algorithm 1. PDS Learning Algorithm
1: Initialize number of iterations k 1, value function vector
V^ð^
sÞ 0; 8^s 2 S and the LM b 0.
2: while TRUE do
3: Determine the event (arrival/departure) in the current
decision epoch.
4: Choose action a which maximizes the R.H.S expression in
Equation (9).
5: Update the value function of PDS s^ using (9).
6: Update the LM according to Equation (10).
7: Update s^ s^0 and k k þ 1.
8: end while
6 STRUCTURE-AWARE ONLINE RAT SELECTION
ALGORITHM
In this section, we propose a learning algorithm exploiting
the threshold properties of the optimal policy. The PDS
learning algorithm proposed in Section 5 does not take into
account the threshold nature of the optimal policy and
hence optimizes over the entire policy space. However,
Authorized licensed use limited to: UNIVERSITY PUTRA MALAYSIA. Downloaded on March 10,2020 at 05:36:03 UTC from IEEE Xplore. Restrictions apply.
IEEE TRANSACTIONS ON MOBILE COMPUTING, VOL. 19, NO. 2, FEBRUARY 2020
utilizing the threshold nature of optimal policy, the policy
space can be reduced significantly. To this end, we propose
a structure-aware online learning algorithm which searches
the optimal policy only from the set of threshold policies,
providing faster convergence than PDS learning Algorithm.
Note that independent methodologies which are developed
in this section can be applied to any learning problem hav-
ing similar structural properties.
(11)
6.1 Gradient Based Online Algorithm
(12) Let the throughput increment in WiFi when the number
of WiFi users increases from k to ðk þ 1Þ be denoted by
~W;D ðkÞ. Therefore, R~W;D ðkÞ ¼ ðk þ 1ÞRW;D ðk þ 1Þ  kRW;D ðkÞ.
R
We assume the following.
Assumption 1. R~W;D ðkÞ is a non-increasing function of k.
This assumption is in line with the full buffer traffic
model [20].
Summary of the structural properties of the optimal pol-
icy is as follows. Detailed proofs of the structural properties
can be found in [19].
(1) Upto a threshold on the number of WiFi data users
(say kth ) serve data users in WiFi (A3 ) and then serve
them using LTE (A2 ) until LTE is full. When LTE is
full, i.e., ði þ jÞ ¼ C, the optimal policy is to serve all
ing data user is blocked.
(2) f8i; jjði þ jÞ < Cg and a voice user arrival, A4 ðA2 Þ is
better than A2 ðA4 Þ if k < kth ðk  kth Þ.
(3) f8i; jjði þ jÞ < Cg and a voice user arrival, if the
optimal action in state ði; j; kÞ is blocking, then the
optimal action in state ði þ 1; j; kÞ is blocking.
(4) f8i; jjði þ jÞ ¼ Cg and a voice user arrival, if the opti-
mal action in state ði; j; kÞ is blocking, then the opti-
mal action in state ði þ 1; j  1; kÞ is blocking.
Using the first two properties, we can eliminate a number of
suboptimal actions. In the case of data user arrival (event
E2 ) and departure of voice and data users (events E3 ; E4
and E5 ), a single decision is involved. This may provide
improved convergence because contrary to an online
algorithm without any knowledge of structural property,
we no longer need to learn optimal actions in some states.
The only event where multiple decisions are involved is the
voice user arrival (event E1 ). As stated in Property 3 and 4,
the value of the threshold on i, where the optimal action
changes to blocking, is a function of j and k. Thus, if we
have the knowledge of the values of thresholds, we can
characterize the policy completely. The idea is to optimize
over the threshold vector (say u) using an update rule, so
that the value of the threshold vector u converges to the
optimal value. Before proceeding further, we determine the
dimension of u using the analysis presented below.
Using Property 1 and 2, we can identify three regions.
(1) 0  k < kth : Using Property 1, we have j ¼ 0. For
each value of k, we need to know the value of thresh-
old which belongs to the set f0; 1; ::; Cg.
(2) k ¼ kth : Using Property 1, k ¼ kth ) j  0. Thus, it
boils down to computing a single threshold which
belongs to the set f0; 1; ::C  jg (Property 3), for each
value of j ð0  j < C). Also, we need to compute a
single threshold for ði þ jÞ ¼ C (Property 4).
ROY ET AL.: LOW COMPLEXITY ONLINE RADIO ACCESS TECHNOLOGY SELECTION ALGORITHM IN LTE-WIFI HETNET
(3) W > k > kth : Using Property 1, k > kth )
ði þ jÞ ¼ C. Thus, using Property 4, we need to
obtain the threshold of blocking for ðW  kth  1Þ
values of k.
Therefore the dimension of u ¼ ðkth þ C þ W  kth Þ =
ðC þ W Þ.
Remark 3. When the state space becomes too large, then it
becomes cumbersome to represent a policy since this
requires tabulating actions corresponding to each state.
Due to the threshold nature of the optimal policy, the
representation using the threshold vector becomes com-
putationally efficient. Instead of storing the optimal
action corresponding to each state, we just need to store
ðC þ W Þ individual thresholds.
We consider a class of threshold policies which can be
described in terms of the threshold vector u. The main idea
behind the online algorithm is to compute the gradient of the
system metric, i.e., the average reward of the system, with
respect to u and improve the policy by updating the value of
u in the direction of the gradient. Therefore, following [35],
one needs to compute the gradient of the system metric. To
express the dependence of the parameters associated with the
underlying Markov chain on u explicitly, we need to redefine
the notations. Let the transition probability associated with
the Markov chain fXn g as a function of u be given by
Pss0 ðuÞ ¼ P ðXnþ1 ¼ s0 jXn ¼ s; uÞ:
Assumption 2. We assume that for every s; s0 2 S, Pss0 ðuÞ is a
bounded, twice differentiable function, and the first and second
derivative of Pss0 ðuÞ is bounded.
Let the average reward of the Markov chain, steady state
stationary probability of state s, value function of state s (as
a function of u) be denoted by rðuÞ, pðs; uÞ and V ðs; uÞ,
respectively. The following proposition provides a closed-
form expression for the gradient of the average reward of
the system. A proof for the same can be found in [35].
Although [35] considers a generalized case where the
reward function depends on u, in our case the same proof
holds with the exception that the gradient of the reward
function is zero.
Proposition 1. Under assumptions on Pss0 ðuÞ as stated before,
we have
X X
rrðuÞ ¼ pðs; uÞ rPss0 ðuÞV ðs0 ; uÞ: (13)
s0 2S
s2S
Hence, we can compute the value of rrðuÞ (or rPss0 ðuÞ)
to construct an incremental scheme similar to a stochastic
gradient algorithm for the threshold values, of the form
unþ1 ¼ un þ hðnÞrrðun Þ;
where un represents the value of threshold vector in nth iter-
ation on the slower timescale hðnÞ. Given a threshold u, we
assume that the state transition in state s ¼ ði; j; kÞ is given
by P0 ðs0 jsÞ, if i < uðT Þ and P1 ðs0 jsÞ, otherwise, where uðT Þ
denotes the component of u which corresponds to state s.
Specifically
 
k þ j; ði þ jÞ 6¼ C;
T ¼ (15)
C þ k; ði þ jÞ ¼ C:
Authorized licensed use limited to: UNIVERSITY PUTRA MALAYSIA. Downloaded on March 10,2020 at 05:36:03 UTC from IEEE Xplore. Restrictions apply.
383
Therefore, according to the two-timescale gradient based
learning framework, on the faster timescale, we have
V nþ1 ðs; uÞ ¼ ð1  gðgðs; nÞÞÞV n ðs; uÞ þ gðgðs; nÞÞ½rðs; a; bÞ
þ V n ðs0 ; uÞ  V n ðs ; uÞ; (16)
V nþ1 ðs ; uÞ ¼ V^n ðs00 ; uÞ; 8s00 6¼ s:
00
For example, if the current state is s ¼ ði; 0; 0Þ and ði < un ð0ÞÞ,
then state transition is determined by P0 ðs0 jsÞ (accept in LTE
(A2 )), i.e., s0 ¼ ði þ 1; 0; 0Þ, else, s0 is determined by P1 ðs0 jsÞ
(blocking (A1 )), i.e., s0 ¼ ði; 0; 0Þ. However, value functions
corresponding to other states are kept unchanged.
Note that, the above scheme works for a fixed value of
threshold vector u and LM b. To obtain the optimal value
of u, u is to be iterated along the slower timescale hðnÞ.
Note that, although individual components of the thresh-
old take discrete values, we interpolate them to the contin-
uous domain to be able to apply the online update rule.
Since the threshold policy is a step function (governed by
P0 ðs0 jsÞ up to a threshold and P1 ðs0 jsÞ, thereafter) defined
at discrete points, Assumption 2 is not satisfied at every
point. Therefore we approximate the threshold policy in
state s by a randomized policy which is a function of u
(fðs; uÞ, say). We define
Pss0 ðuÞ  P1 ðs0 jsÞfðs; uÞ þ P0 ðs0 jsÞð1  fðs; uÞÞ;
ðiuðT Þ0:5Þ
where fðs; uðT ÞÞ ¼ 1þe
e
ðiuðT Þ0:5Þ in state s ¼ ði; j; kÞ, provides
a convenient approximation to the step function.
Remark 4. The rationale behind the choice of this function
is the fact that it is continuously differentiable, and the
derivative is nonzero everywhere.
While designing an online update scheme for u, instead
of rrðun Þ (See Equation (14)), we can evaluate rPss0 ðuÞ. The
steady-state stationary probabilities inside the summation
inside Equation (13) can be omitted by performing averag-
ing over time. We have
rPss0 ðuÞ ¼ ðP1 ðs0 jsÞ  P0 ðs0 jsÞÞrfðs; uÞ: (17)
In the right hand side of Equation (17), we incorporate a mul-
tiplication factor of 12 since multiplication by a constant term
does not alter the online scheme. The physical significance of
this operation is that at any iteration, we have state transi-
tions according to P0 ð:j:Þ and P1 ð:j:Þ with equal probabilities.
The update of u in the slower timescale hðnÞ is as follows:
unþ1 ðT Þ ¼ DT ½un ðT Þ þ hðnÞrfðs; un ðT ÞÞð1Þan V n ðs0 ; un Þ;
(18)
unþ1 ðT 0 Þ ¼ un ðT 0 Þ 8T 0 ¼
6 T;
where an is a random variable which takes values 0 and 1
with equal probabilities. When it takes the value 0, then s0 is
(14) determined by P1 ðs0 jsÞ, otherwise by P0 ðs0 jsÞ. The averaging
property of SA then leads to the effective drift (17). Depend-
ing on the state visited, the T th component of the vector u is
updated as shown in Equation (18). For example, if the cur-
rent state is ð1; 0; 0Þ, then un ð0Þ is updated (See Equa-
tion (15)), and other components are kept unchanged. The
projection operator DT is a function which ensures that the
iterates remain bounded in the interval ½0; MðT Þ], where
C  ðT  kth Þ; if ðkth þ CÞ  T  kth ;
MðT Þ ¼ (19)
C; else:
384
TABLE 3
Computational and Storage Complexities of Different Algorithms
Algorithm Storage complexity Computational
complexity
Q-learning [1] OðjSj jAjÞ OðjAjÞ
PDS learning OðjSjÞ ¼ OðC 2 W Þ OðjAjÞ
Structure-aware OðC 2 þ CW Þ Oð1Þ
learning
Similar to Algorithm 1, to obtain the optimal value of b, b is to
be iterated along the same timescale hðnÞ, as specified below:
bnþ1 ¼ L½bn þ hðnÞðBn  Bmax Þ;
Remark 5. The dynamics of LM and threshold vector are
not dependent on each other directly. However, both b
and u iterates depend on value functions in the faster
timescale. Therefore u is updated in the same timescale as
that of b, without requiring a third timescale.
Theorem 2. The schemes (16), (18) and (20) converge to opti-
mality a.s.
Proof. Proof is provided in Section 10.2. in the supplemen-
tal material file. u
t icy. The per-iteration computational complexity of this
Based on the analysis described above, the structure-
aware online learning algorithm is stated in Algorithm 2. As
described in the algorithm, value functions associated with
different states, the LM, the threshold vector and the number
of iterations are initialized at the beginning. When the current
state of the system is s, the system chooses the action which is
given by the current value of the threshold vector. Based on
the observed reward, V ðsÞ and u is updated along with the
LM. This process is repeated for every decision epoch.
Algorithm 2. Structure-Aware Learning Algorithm
1: Initialize number of iterations k 1, value function
V ðsÞ 0; 8s 2 S, the LM b 0 and the threshold vector
u 0.
2: while TRUE do
3: Choose action a given by the current value of threshold
vector u.
4: Update the value function of states s using (16).
5: Update the LM according to Equation (20).
6: Update threshold vector according to Equation (18).
7: Update s s0 and k k þ 1.
8: end while
7 COMPARISON OF COMPLEXITIES OF LEARNING
ALGORITHMS
In this section, we provide a comparison of storage and
computational complexities of traditional Q-learning [1], pro-
posed PDS learning and structure-aware learning algorithms.
We summarize the storage and computational complexities
of these schemes in Table 3.
Q-learning algorithm [1] stores value functions for every
state-action pair, i.e., jSj jAj values and updates the value
function of one state-action pair at a time. While updating
the value function, Q-learning evaluates jAj functions. The
PDS learning algorithm (see Equation (11)) requires storing
jSj PDS value functions and feasible actions in every state,
Authorized licensed use limited to: UNIVERSITY PUTRA MALAYSIA. Downloaded on March 10,2020 at 05:36:03 UTC from IEEE Xplore. Restrictions apply.
IEEE TRANSACTIONS ON MOBILE COMPUTING, VOL. 19, NO. 2, FEBRUARY 2020
i.e., jSj values. While updating the PDS value function, PDS
learning algorithm evaluates jAj functions, resulting in a
per-iteration complexity of jAj.
In the case of structure-aware learning algorithm, we no lon-
ger need to store jSj value functions. Rather, by virtue of the
threshold nature of optimal policy, we consider three cases.
(1) 0  k < kth : Since we have j ¼ 0, for each value of k,
we need to store ðC þ 1Þ value functions.
(2) k ¼ kth : k ¼ kth ) j  0. Thus, we need to store
ðC þ 1  jÞ value functions, for each value of
j ð0  j  C).
(3) W  k > kth : k > kth ) ði þ jÞ ¼ C. Therefore, we
need to store value functions of ðC þ 1Þ states for
(20)
each value of k.
Therefore, the total number of value functions which need
to be stored is ðC þ 1Þkth þ ðCþ1ÞðCþ2Þ
2 þ ðC þ 1ÞðW  kth Þ,
which is equal to ðCþ1ÞðCþ2Þ þ ðC þ 1ÞW . Note that, this is a
2
considerable reduction in storage complexity in comparison
to the PDS learning scheme having a storage complexity of
OðC 2 W Þ. For example, when W ¼ C, the storage complexity
reduces from OðC 3 Þ to OðC 2 Þ. Furthermore, feasible actions
corresponding to each state need not be stored separately
since the threshold vector completely characterizes the pol-
scheme (see Equation (16)) is Oð1Þ. This scheme also involves
updating a single component of the threshold vector (Equa-
tion (18)) with a computational complexity of Oð1Þ.
8 SIMULATION RESULTS
In this section, proposed PDS learning and structure-aware
learning algorithms are simulated in ns-3 to characterize
and compare their convergence behaviors. Convergence
rates of the proposed algorithms are compared with that of
the Q-learning, as proposed in our earlier work [1]. Simula-
tion results establish that the proposed PDS learning algo-
rithm provides improved convergence than Q-learning.
Furthermore, it is observed that the knowledge of structural
properties indeed reduces the convergence time.
8.1 Simulation Model and Evaluation Methodology
The simulation setup comprises a 3GPP LTE BS and an oper-
ator-deployed IEEE 802.11g WiFi AP. All users are assumed
to be stationary. Data users are distributed uniformly within
30 m radius of the WiFi AP which is approximately 50 m
away from the LTE BS. LTE and WiFi network parameters
used in simulations are chosen based on 3GPP [36], [37]
models and saturation throughput [20] IEEE 802.11g WiFi [6]
model and described in Tables 4 and 5. We consider the gen-
eration of CBR uplink traffic for voice and data users in LTE.
This is implemented in ns-3 using an application (similar to
ON/OFF application) developed by us.
For the update of the PDS value functions, threshold
vector and LM, we consider gðnÞ ¼ ðb n cþ2Þ
1
0:6 and hðnÞ ¼ n .
10
1000
8.2 Convergence Analysis
Figs. 3a and 3b illustrate how the Q-learning, PDS learning
and structure-aware learning algorithms converge with
increasing number of iterations (n). We keep
v ¼
d ¼ 1. It
is evident that both the proposed algorithms outperform
Q-learning in terms of convergence speed. Contrary to PDS
learning, even after a considerable amount of iterations,
ROY ET AL.: LOW COMPLEXITY ONLINE RADIO ACCESS TECHNOLOGY SELECTION ALGORITHM IN LTE-WIFI HETNET 385

TABLE 4 TABLE 5
LTE Network Model WiFi Network Model

Parameter Value Parameter Value

Maximum capacity 10 users Channel bit rate 54 Mbps
Voice bit rate of a single user 20 kbps UDP header 224 bits
Data bit rate of a single user 5 Mbps Packet payload 1500 bytes
Voice packet payload 50 bits Slot duration 20 ms
Data packet payload 600 bits Short inter-frame space (SIFS) 10 ms
Tx power for BS and MS 46 dBm and 23 Distributed Coordination Function IFS 50 ms
dBm (DIFS)
Noise figure for BS and MS 5 dB and 9 dB Minimum acceptable per-user 3.5 Mbps
Antenna height for BS and MS 32 m and 1.5 m throughput
Antenna parameter for BS and MS Isotropic Antenna Tx power for AP 23 dBm
Path loss (R in kms) 128:1 þ 37:6 log ðRÞ Noise figure for AP 4 dB
Antenna height for AP 2.5 m
Antenna parameter Isotropic antenna
Q-learning explores different actions with a finite probabil- Path loss (R in kms) 140:3 þ 36:7 log ðRÞ
ity, thereby reducing the convergence speed.
The knowledge of structure reduces the feasible policy
space. Therefore, the structure-aware learning algorithm and minimum values of total system throughput over this
offers faster convergence to the optimal policy. We no lon- window. We set the window size to be 100. If the ratio is
ger need to learn the optimal actions in a subset of states, more than 0.99, then we conclude that stopping criteria is
where the optimal policy is determined using structural reached, i.e., the obtained policy is in a close neighborhood
properties. As observed in Figs. 3a and 3b, the number of of the optimal policy with high probability. In Figs. 5a and
iterations before convergence reduces from 2000 to 300 and Pn
5b, total system throughput is plotted against k¼1 gðkÞ.
1000 to 300, respectively. Note that smaller number of itera- Pn
tions translates into lesser amount of time for convergence. Note that, k¼1 gðkÞ is chosen instead of n, to decouple the
Figs. 4a and 4b depict the convergence of LM as n effect of diminishing step size, while analyzing the conver-
increases. It is evident that as the number of iteration gence behavior of the proposed schemes. In other words,
increases, LM converges for both the schemes. this parameter is selected to establish that the convergence
In Table 6, we illustrate the average time taken by a single of the algorithms is indeed due to the convergence to the
iteration of Q-learning, PDS learning and structure-aware optimal policy and not due to very small values of step size
learning algorithms, respectively, in simulations. Average gðnÞ as n becomes large. We observe in Figs. 5a and 5b that
per-iteration time of an algorithm reflects the per-iteration the PDSPlearning algorithm reaches the stopping criteria
n
computational complexity. As observed from Table 6, the when k¼1 gðkÞ becomes 700, which corresponds to
time taken by the structure-aware learning algorithm is the approximately 1000 iterations. On the other hand, the
least. Also, the average per-iteration time of the PDS learning structure-aware
algorithm is slightly lower than that of Q-learning. These Pn learning algorithm reaches stopping cri-
teria when k¼1 gðkÞ becomes 300 which translates into
results corroborate the analysis illustrated in Table 3.
almost 450 iterations.
8.3 Stopping Criteria
While simulating an online learning algorithm, in practical 8.4 Consideration of Realistic Scenarios
cases, we may not want to wait till the actual convergence. While the above simulation results provide significant
Rather, we can observe the total system throughput over a insight into the convergence behavior of the proposed algo-
moving window
P of the sums of step sizes till the present rithms over traditional Q-learning, in this section, we evalu-
iteration ( nk¼1 gðkÞ) and calculate the ratio of maximum ate the performances of the proposed algorithms in realistic

Fig. 3. Plot of total system throughput versus number of iterations ðnÞ for different algorithms.

Authorized licensed use limited to: UNIVERSITY PUTRA MALAYSIA. Downloaded on March 10,2020 at 05:36:03 UTC from IEEE Xplore. Restrictions apply.
386
Fig. 4. Plot of LM versus number of iterations ðnÞ for different algorithms.
scenarios. We compare the total system throughput and
voice user blocking probability performance of the pro-
posed algorithms with that of the on-the-spot offloading
[38] and LTE-preferred schemes [19].
Although in the system model (See Section 2) we consider
single resource block allocation to LTE data users, in simula-
tions we relax this assumption and consider proportional
fair scheduling for the LTE BS which dynamically assigns
resources to the users based on user bandwidth demand.
Users randomly generate individual bandwidth demands.
However, we assume that the maximum data rate achievable
for a single data user is 5 Mbps and the bottleneck is in the
access network. Furthermore, in the previous sections, there
is no consideration of channel fading effects in LTE and
WiFi. To address that, whenever we choose an action involv-
ing offloading of a user from one RAT to another (A4 and
A5 ), the user with the worst channel is selected for offload-
ing. For example, whenever A4 is chosen and we offload a
data user from LTE to WiFi, we always choose the data user
with the lowest Signal-to-noise Ratio (SNR). Since, in general,
a user with bad channel provides bad throughput to the sys-
tem, the user with the worst channel is chosen for offloading.
We consider Extended Pedestrian A model [39] for fading in
LTE and Rayleigh fading for WiFi.
In on-the-spot offloading [38], data users always choose
WiFi unless WiFi coverage is not present. Therefore, in our
system model, on-the-spot offloading always associates data
users with WiFi until capacity is reached in WiFi. Voice users
are associated with LTE, and when LTE reaches its capacity,
voice users are blocked. In LTE-preferred scheme [19], voice
and data users are associated with LTE until LTE reaches its
capacity. When LTE reaches its capacity and a voice user
arrives, the voice user is blocked if there is no data user in
LTE. Otherwise, one existing data user is offloaded to WiFi if
capacity is available in WiFi. Upon the departure of an exist-
ing voice or data user from LTE, an existing data user in
TABLE 6
Average Per-Iteration Time of Different Algorithms
Algorithm Average per-iteration time (ms)
Q-learning [1] 49.66
PDS learning 40.07
Structure-aware learning 32.119
Authorized licensed use limited to: UNIVERSITY PUTRA MALAYSIA. Downloaded on March 10,2020 at 05:36:03 UTC from IEEE Xplore. Restrictions apply.
IEEE TRANSACTIONS ON MOBILE COMPUTING, VOL. 19, NO. 2, FEBRUARY 2020
WiFi, if any, is offloaded to LTE. While offloading, we always
choose the user with the worst channel.
8.4.1 Voice User Arrival Rate Variation
Fig. 6a depicts the blocking percentage of voice users for
on-the-spot offloading, LTE-preferred and the proposed
algorithms for varying
v . Since on-the-spot offloading
blocks voice user when LTE reaches its capacity, blocking
probability of voice users increases with
v . Since PDS
learning and structure-aware learning algorithms learn in
which states blocking is to be chosen as the optimal
action, voice user blocking probability corresponding to
these algorithms converge to the same value. Since the
proposed algorithms may block voice users even when
the LTE system does not reach its capacity, the blocking
probability values are marginally higher than that of
on-the-spot offloading. Voice users may be blocked to
save LTE resources for future data user arrivals which
have a higher throughput contribution to the system.
LTE-preferred scheme blocks a voice user when LTE sys-
tem is full and there is no data user in LTE. Therefore,
on-the-spot offloading and LTE-preferred schemes pro-
vide similar blocking probability performance.
Fig. 6b illustrates the total system throughput perfor-
mance of different algorithms under varying
v . With
increase in
v , the average number of voice users in the sys-
tem increases while the number of WiFi data users remains
the same. Therefore, in the case of on-the-spot offloading,
the total system throughput increases with
v . However,
since the throughput of voice users is small compared to
data users, the rate of increment is very small. PDS learning
and structure-aware learning algorithms learn the optimal
policy which does significant load balancing via A4 and A5 .
Also, while offloading, the proposed algorithms take chan-
nel state of users into account. Thus, these algorithms out-
perform on-the-spot offloading in terms of total system
throughput with performance improvement varying from
10.72 percent (for
v ¼ 0:13) to 28.72 percent (for
v ¼ 0:07).
With increase in
v , LTE-preferred scheme starts offloading
data users to WiFi to accommodate incoming voice users.
Under low WiFi load, total throughput of the system
increases. As WiFi load increases, the rate of increment
decreases. However, both the proposed algorithms perform
better than LTE-preferred scheme.
ROY ET AL.: LOW COMPLEXITY ONLINE RADIO ACCESS TECHNOLOGY SELECTION ALGORITHM IN LTE-WIFI HETNET 387

P
Fig. 5. Plot of total system throughput versus sum of step sizes till nth iteration ( nk¼1 gðkÞ) for different algorithms.

Fig. 6. Plot of voice user blocking fraction and total system throughput for different algorithms. (a) Voice user blocking percentage versus
v . (b) Total
system throughput versus
v (
d ¼ 1=20; mv ¼ 1=60, and md ¼ 1=10). (c) Voice user blocking percentage versus
d . (d) Total system throughput
versus
d (
v ¼ 1=6; mv ¼ 1=60, and md ¼ 1=10).

8.4.2 Data User Arrival Rate Variation
As observed in Fig. 6c, since in on-the-spot offloading,
data and voice users are served using WiFi and LTE, respec-
tively, changes in
d do not impact the blocking probability
of voice users. Performances of both PDS learning and struc-
ture-aware learning algorithms are similar to that of on-the-
spot offloading. Due to the presence of a constraint on the
voice user blocking probability, most of the voice users are
blocked when the LTE system reaches capacity. Therefore,
the proposed algorithms do most of the blocking of voice
users in the same decision epochs as that of on-the-spot off-
loading. Since LTE-preferred scheme blocks voice users only
when LTE does not have available capacity and there is no
data user in LTE, the blocking probabilities of LTE-preferred
scheme and on-the-spot offloading are same.
Since on-the-spot offloading associates data user with
WiFi, with increase in
d , the load in WiFi increases. As a
result, as
d (See Fig. 6d) increases, the effect of contention
and channel fading reduces the rate of increment of
throughput. Both the proposed algorithms perform better
than on-the-spot offloading by virtue of optimal RAT selec-
tion and load balancing actions which reduces the effect of
contention in WiFi. Also, while offloading, the proposed
algorithms take channel state of users into account.

Authorized licensed use limited to: UNIVERSITY PUTRA MALAYSIA. Downloaded on March 10,2020 at 05:36:03 UTC from IEEE Xplore. Restrictions apply.
388
Fig. 7. (a) Total system throughput versus
v (
d ¼ 1=20; mv ¼ 1=60, and md ¼ 1=10). (b) Total system throughput versus
d (
v ¼ 1=6; mv ¼ 1=60, and
md ¼ 1=10).
Therefore, the proposed algorithms outperform on-the-spot
offloading with performance improvement varying from 20
percent (for
d ¼ 0:1) to 54.6 percent (for
d ¼ 0:5). As
d
increases, LTE-preferred scheme starts offloading more data
users to WiFi. Therefore, the system throughput increases.
Under high
d , the effect of contention is lesser than that of
on-the-spot offloading, resulting in a better performance
than on-the-spot offloading. However, the proposed algo-
rithms perform better than LTE-preferred scheme.
8.5 Consideration of User Mobility
In this section, we evaluate how the proposed algorithms
perform in comparison to on-the-spot offloading and LTE-
preferred scheme in the face of user mobility. In addition to
ns-3 simulation settings described in the last section, we
also consider random waypoint model [40] for mobility of
voice and data users. As evident from Figs. 7a and 7b,
although total system throughputs provided by different
algorithms change due to mobility, comparative perfor-
mance of the proposed algorithms with respect to on-the-
spot offloading and LTE-preferred scheme remains the
same. Since mobility does not have any impact on the block-
ing probability of the voice users, the blocking probability
performances of the considered algorithms are exactly the
same as that described in Figs. 6a and 6c.
9 CONCLUSIONS
In this paper, we have proposed a PDS learning algorithm
which can be implemented online without the knowledge
of statistics of arrival processes. It has been proved that the
algorithm converges to the optimal policy. Furthermore,
another online algorithm, which exploits the threshold
structure of optimal policy, has been proposed. The knowl-
edge of threshold structure provides improvements in
computational and storage complexity and convergence
time. The proposed algorithm provides a novel framework
that can be applied for designing online learning algorithms
for any general problem and is of independent interest. We
have proved that the structure-aware learning algorithm
convergences to globally optimal threshold vector. Simula-
tion results have been presented to exhibit how the PDS
paradigm and the knowledge of structural properties pro-
vide improvement in convergence time over traditional
online association algorithms. Moreover, simulation results
Authorized licensed use limited to: UNIVERSITY PUTRA MALAYSIA. Downloaded on March 10,2020 at 05:36:03 UTC from IEEE Xplore. Restrictions apply.
IEEE TRANSACTIONS ON MOBILE COMPUTING, VOL. 19, NO. 2, FEBRUARY 2020
establish that the proposed schemes outperform on-the-spot
offloading and LTE-preferred schemes under realistic net-
work scenarios.
ACKNOWLEDGMENTS
This work has been funded by the Ministry of Electronics
and Information Technology (MeitY), Government of India,
as part of the "5G Research and Building Next Gen Solutions
for Indian Market" project.
REFERENCES
[1] A. Roy, P. Chaporkar, and A. Karandikar, “An on-line radio
access technology selection algorithm in an LTE-WiFi network,”
in Proc. IEEE Wireless Commun. Netw. Conf., 2017, pp. 1–6.
[2] Y. He, M. Chen, B. Ge, and M. Guizani, “On WiFi offloading in hetero-
geneous networks: Various incentives and trade-off strategies,” IEEE
Commun. Surveys Tuts., vol. 18, no. 4, pp. 2345–2385, Oct.–Dec. 2016.
[3] Cisco, “Cisco visual networking index: Global mobile data traffic
forecast update, 2013–2018,” White Paper, 2014.
[4] V. G. Nguyen, T. X. Do, and Y. Kim, “SDN and virtualization-
based LTE mobile network architectures: A comprehensive
survey,” Wireless Pers. Commun., vol. 86, no. 3, pp. 1401–1438, 2016.
[5] N. M. Akshatha, P. Jha, and A. Karandikar, “A centralized SDN
architecture for the 5G cellular network,” in Proc. IEEE 5G World
Forum, 2018, pp. 147–152.
[6] Wireless LAN Medium Access Control (MAC) and Physical Layer
(PHY) Specifications, IEEE Standard 802.11–2012, Part 11, 2012.
[7] 3GPP TR 37.834 v0.3.0, “Study on WLAN/3GPP Radio Inter-
working,” (2013, Jun.). [Online]. Available: http://www.3gpp.
org/DynaReport/37834.htm
[8] A. Whittier, P. Kulkarni, F. Cao, and S. Armour, “Mobile data off-
loading addressing the service quality versus resource utilisation
dilemma,” in Proc. IEEE 27th Annu. Int. Symp. Pers. Indoor Mobile
Radio Commun., 2016, pp. 1–6.
[9] F. Moety, M. Ibrahim, S. Lahoud, and K. Khawam, “Distributed heuris-
tic algorithms for RAT selection in wireless heterogeneous networks,”
in Proc. IEEE Wireless Commun. Netw. Conf., 2012, pp. 2220–2224.
[10] E. Aryafar, A. Keshavarz-Haddad, M. Wang, and M. Chiang,
“RAT selection games in HetNets,” in Proc. IEEE INFOCOM,
2013, pp. 998–1006.
[11] K. Lee, J. Lee, Y. Yi, I. Rhee, and S. Chong, “Mobile data offload-
ing: How much can WiFi deliver?” IEEE/ACM Trans. Netw.,
vol. 21, no. 2, pp. 536–550, Apr. 2013.
[12] D. Suh, H. Ko, and S. Pack, “Efficiency analysis of WiFi offloading
techniques,” IEEE Trans. Veh. Technol., vol. 65, no. 5, pp. 3813–
3817, May 2016.
[13] N. Cheng, N. Lu, N. Zhang, X. Zhang, X. S. Shen, and J. W. Mark,
“Opportunistic WiFi offloading in vehicular environment: A
game-theory approach,” IEEE Trans. Intell. Transp. Syst., vol. 17,
no. 7, pp. 1944–1955, Jul. 2016.
ROY ET AL.: LOW COMPLEXITY ONLINE RADIO ACCESS TECHNOLOGY SELECTION ALGORITHM IN LTE-WIFI HETNET
[14] A. Roy and A. Karandikar, “Optimal radio access technology
selection policy for LTE-WiFi network,” in Proc. IEEE Int. Symp.
Model. Optimization Mobile Ad Hoc Wireless Netw., 2015, pp. 291–298.
[15] G. S. Kasbekar, P. Nuggehalli, and J. Kuri, “Online client-AP asso-
ciation in WLANs,” in Proc. IEEE Int. Symp. Model. Optimization
Mobile Ad Hoc Wireless Netw., 2006, pp. 1–8.
[16] K. Khawam, S. Lahoud, M. Ibrahim, M. Yassin, S. Martin, M. El
Helou, and F. Moety, “Radio access technology selection in het-
erogeneous networks,” Phys. Commun., vol. 18, pp. 125–139, 2016.
[17] S. Barmpounakis, A. Kaloxylos, P. Spapis, and N. Alonistioti,
“Context-aware, user-driven, network-controlled RAT selection
for 5G networks,” Comput. Netw., vol. 113, pp. 124–147, 2017.
[18] B. H. Jung, N. O. Song, and D. K. Sung, “A network-assisted user-
centric WiFi-offloading model for maximizing per-user through-
put in a heterogeneous network,” IEEE Trans. Veh. Technol.,
vol. 63, no. 4, pp. 1940–1945, May 2014.
[19] A. Roy, P. Chaporkar, and A. Karandikar, “Optimal radio access
technology selection algorithm for LTE-WiFi network,” IEEE
Trans. Veh. Technol., vol. 67, no. 7, pp. 6446–6460, Jul. 2018.
[20] G. Bianchi, “Performance analysis of the IEEE 802.11 distributed
coordination function,” IEEE J. Sel. Areas Commun., vol. 18, no. 3,
pp. 535–547, Mar. 2000.
[21] M. El Helou, M. Ibrahim, S. Lahoud, K. Khawam, D. Mezher, and
B. Cousin, “A network-assisted approach for RAT selection in het-
erogeneous cellular networks,” IEEE J. Sel. Areas Commun., vol. 33,
no. 6, pp. 1055–1067, Jun. 2015.
[22] E. Khloussy, X. Gelabert, and Y. Jiang, “Investigation on MDP-
based radio access technology selection in heterogeneous wireless
networks,” Comput. Netw., vol. 91, pp. 57–67, 2015.
[23] R. Li, Z. Zhao, J. Zheng, C. Mei, Y. Cai, and H. Zhang, “The learning
and prediction of application-level traffic data in cellular networks,”
IEEE Trans. Wireless Commun., vol. 16, no. 6, pp. 3899–3912, Jun. 2017.
[24] K. Kumar, A. Gupta, R. Shah, A. Karandikar, and P. Chaporkar, “On
analyzing Indian cellular traffic characteristics for energy efficient net-
work operation,” in Proc. IEEE 21st Nat. Conf. Commun., 2015, pp. 1–6.
[25] U. Paul, A. P. Subramanian, M. M. Buddhikot, and S. R. Das,
“Understanding traffic dynamics in cellular data networks,” in
Proc. IEEE INFOCOM, 2011, pp. 882–890.
[26] R. S. Sutton and A. G. Barto, Reinforcement Learning: An Introduc-
tion. Cambridge, MA, USA: MIT Press, 1998.
[27] K. Adachi, M. Li, P. H. Tan, Y. Zhou, and S. Sun, “Q-Learning
based intelligent traffic steering in heterogeneous network,” in
Proc. IEEE Veh. Technol. Conf. Spring, 2016, pp. 1–5.
[28] S. Anbalagan, D. Kumar, D. Ghosal, G. Raja, and V. Muthuval-
liammai, “SDN-assisted learning approach for data offloading in
5G HetNets,” Mobile Netw. Appl., vol. 22, no. 4, pp. 771–782, 2017.
[29] ns-3 simulator. (2018, Dec.). [Online]. Available: http://code.
nsnam.org/ns-3-dev/
[30] T. Bonald and J. W. Roberts, “Internet and the Erlang formula,” ACM
SIGCOMM Comput. Commun. Rev., vol. 42, no. 1, pp. 23–30, 2012.
[31] E. Altman, Constrained Markov Decision Processes. Boca Raton, FL,
USA: CRC Press, 1999.
[32] M. L. Puterman, Markov Decision Processes: Discrete Stochastic
Dynamic Programming. Hoboken, NJ, USA: Wiley, 2014.
[33] F. J. Beutler and K. W. Ross, “Optimal policies for controlled Mar-
kov chains with a constraint,” J. Math. Anal. Appl., vol. 112, no. 1,
pp. 236–252, 1985.
[34] V. S. Borkar, Stochastic Approximation: A Dynamical Systems View-
point. Cambridge, U.K.: Cambridge Univ. Press, 2008.
[35] P. Marbach and J. N. Tsitsiklis, “Simulation-based optimization of
Markov reward processes,” IEEE Trans. Autom. Control, vol. 46,
no. 2, pp. 191–209, Feb. 2001.
[36] 3GPP TR 36.814 v9.0.0, “Further advancements for E-UTRA physi-
cal layer aspects,” (2010, Mar.). [Online]. Available: http://
www.3gpp.org/dynareport/36814.htm
[37] 3GPP TR 36.839 v11.1.0, “Mobility enhancements in heteroge-
neous networks,” (2012, Dec.). [Online]. Available: http://
www.3gpp.org/dynareport/36839.htm
[38] F. Mehmeti and T. Spyropoulos, “Performance analysis of “on-
the-spot” mobile data offloading,” in Proc. IEEE Global Commun.
Conf., 2013, pp. 1577–1583.
[39] 3GPP TS 36.104 V10.2.0, “Base Station (BS) Radio Transmission
and Reception,” (2011, Apr.). [Online]. Available: http://
www.3gpp.org/dynareport/36104.htm
[40] D. B. Johnson and D. A. Maltz, “Dynamic source routing in ad hoc
wireless networks,” in Mobile Computing. Berlin, Germany:
Springer, 1996, pp. 153–181.
Authorized licensed use limited to: UNIVERSITY PUTRA MALAYSIA. Downloaded on March 10,2020 at 05:36:03 UTC from IEEE Xplore. Restrictions apply.
389
[41] N. Salodkar, A. Bhorkar, A. Karandikar, and V. S. Borkar, “An on-
line learning algorithm for energy efficient delay constrained
scheduling over a fading channel,” IEEE J. Sel. Areas Commun.,
vol. 26, no. 4, pp. 732–742, May 2008.
[42] V. R. Konda and V. S. Borkar, “Actor-critic-type learning algo-
rithms for Markov decision processes,” SIAM J. Control Optimiza-
tion, vol. 38, no. 1, pp. 94–123, 1999.
[43] J. Abounadi, D. Bertsekas, and V. S. Borkar, “Learning algorithms
for Markov decision processes with average cost,” SIAM J. Control
Optimization, vol. 40, no. 3, pp. 681–698, 2001.
[44] V. S. Borkar and S. P. Meyn, “The ODE method for convergence of
stochastic approximation and reinforcement learning,” SIAM J.
Control Optimization, vol. 38, no. 2, pp. 447–469, 2000.
[45] V. S. Borkar, “An actor-critic algorithm for constrained Markov deci-
sion processes,” Syst. Control Lett., vol. 54, no. 3, pp. 207–213, 2005.
Arghyadip Roy received the BE degree from
Jadavpur University, Kolkata, India, in 2010, and
the MTech degree from IIT Kharagpur, India, in
2012. He is currently a research scholar with
the Department of Electrical Engineering, IIT
Bombay, India. He previously worked with the
Samsung R&D Institute-Bangalore, India. His
research interests include resource allocation,
optimization, and control of stochastic systems.
He is a student member of the IEEE.
Vivek Borkar received the BTech degree in elec-
trical engineering from IIT Bombay, in 1976, the
MS degree in systems and control from Case
Western Reserve University, in 1977, and the PhD
degree in electrical engineering and computer sci-
ence from the University of California at Berkeley,
in 1980. He has held positions at the TIFR Center
for Applicable Mathematics and the Indian Institute
of Science in Bengaluru, and Tata Institute of Fun-
damental Research, Mumbai, before joining IIT
Bombay, Mumbai, as institute chair professor of
electrical engineering in Aug. 2011. He has held visiting positions with the
University of Twente, MIT, the University of Maryland at College Park, and
the University of California at Berkeley. He is a fellow of the IEEE, American
Math. Society, TWAS, and the science and engineering academies in
India. His research interests include stochastic optimization and control,
covering theory, algorithms, and applications.
Prasanna Chaporkar received the MS degree
from the Faculty of Engineering, Indian Institute of
Science, Bangalore, India, in 2000, and the PhD
degree from the University of Pennsylvania,
Philadelphia, Pennsylvania, in 2006. He was a
ERCIM post-doctoral fellow with ENS, Paris,
France, and NTNU, Trondheim, Norway. Currently,
he is an associate professor with the Indian Insti-
tute of Technology, Mumbai. His research interests
include resource allocation, stochastic control,
queueing theory, and distributed systems and
algorithms. He is a member of the IEEE.
Abhay Karandikar is currently director of IIT
Kanpur (on leave from IIT Bombay). He is also a
member (part-time) of the Telecom Regulatory
Authority of India (TRAI). In IIT Bombay, he
served as institute chair professor with the Elec-
trical Engineering Department, dean (Faculty
Affairs) from 2017 to 2018, and head of the Elec-
trical Engineering Department from 2012 to 2015.
He is the founding member of the Telecom
Standards Development Society, India (TSDSI),
India’s standards body for telecom. He was the
chairman of TSDSI from 2016 to 2018. His research interests include
resource allocation in wireless networks, software defined networking,
frugal 5G, and rural broadband. He is a member of the IEEE.
" For more information on this or any other computing topic,
please visit our Digital Library at www.computer.org/csdl.