ACCEPTED FROM OPEN CALL
SDN-NFV-Aided Edge-Cloud Interplay for 5G-Envisioned Energy Internet Ecosystem
Sahil Garg, Kuljeet Kaur, Georges Kaddoum, and Song Guo
This work was supported by the
Fonds de recherche du Québec –
Nature et technologies (FRQNT)
through PBEEE via File No. 287201.
Digital Object Identifier:
10.1109/MNET.011.1900602
356
Authorized licensed use limited to: Western Sydney University. Downloaded on June 16,2021 at 00:20:14 UTC from IEEE Xplore. Restrictions apply.
GARG_LAYOUT.indd 356
Abstract
Energy Internet (also referred to as Smart Grid
2.0) is another promising application of the Indus-
trial Internet of Things (IIoT), for example, in the
way energy is being produced, traded, distributed,
and consumed. This is partly due to the lowering
of barriers (e.g., costs and Internet connectivity)
and advances in the underlying technologies, such
as smart meters, electric vehicles, and actuators.
This has also resulted in significant growth in the
volume, velocity, variety, veracity, and value of
data (i.e., the 5 Vs of big data). However, efficient-
ly and effectively handling such big data remains
challenging. One solution currently being explored
in the literature (including industry) to cope with
the increasing network traffic is to use convention-
al cloud infrastructure, but the key limitations of
such an approach include long response time and
high bandwidth consumption. Therefore, there
have been attempts to introduce next-generation
Internet of Things networks to satisfy (real-time)
network service demands and guarantee quali-
ty of service, for example, by pushing computing
capabilities closer to the users (e.g., edge of the
network). However, computation-intensive ener-
gy analytics can be challenging to perform at the
network edge by edge or fog computing devices.
Hence, there have also been attempts to utilize
software defined networking (SDN) and network
function virtualization (NFV) in order to improve
network functionality while adding programmabili-
ty and flexibility features to the network infrastruc-
ture. Specifically, NFV facilitates virtual network
function (VNF) deployment and orchestration,
while SDN controls them for specific use cases.
However, with the rise of next-generation mobile
networks (i.e., 5G), applications and services
require fast and smooth operations with greater
flexibility, efficiency, and scalability. In order to
align with 5G and leverage potential benefits of
edge computing, VNFs should possess critical pro-
cessing requirements (e.g., high throughput, low
latency, and minimal computation overheads). In
other words, virtualization plays an important role.
To date, several techniques have been introduced
in order to achieve the desired objective using
virtual machines (VMs) and containers in isolation.
However, a hybrid approach using VMs and con-
tainers is likely to bring potential benefits for the
large-scale deployment of VNFs across the het-
erogeneous edge and cloud platform. Thus, in this
article, a novel architecture for SDN integrated
with NFV, specifically for the Energy Internet eco-
system, is presented by leveraging the advantages
Sahil Garg, Kuljeet Kaur, and Georges Kaddoum are with École De Technologie Supérieure; Song Guo is with The Hong Kong Polytechnic University.
0890-8044/21/$25.00 © 2021 IEEE
of edge computing. In the considered setup, the
deployment of VNFs is achieved using a mix of
both virtualization and containerization. Findings
from our evaluation demonstrate the potential for
VNF placement across a hybrid execution setup
powered by VMs, as well as the benefits of using
containers.
Introduction to Energy Internet
Rapid advances in information and communi-
cation technologies (ICT) have accelerated the
development and application of the Industrial
Internet of Things (IIoT). Industrial automation,
for example, incorporates connected machines,
sensors, actuators, and other computing devices
to provide a seamless group of services such as
intelligent processing, improved connectivity, and
smart control abilities, as well as enhancing qual-
ity of service (QoS) and users’ quality of experi-
ence (QoE) [1]. By integrating millions to billions
of intelligent objects (e.g., physical objects, cyber
objects, and social objects) in industry applications
and settings, IIoT supports a wide range of smart
services including transportation, Energy Internet,
healthcare, and so on. Among them, Energy Inter-
net is one of the most promising IIoT applications
crucial to a country’s national and societal secu-
rity. For example, it amalgamates advanced sens-
ing, communication, and control functionalities
in order to enhance the operations of the utility
grid. This gradual shift in the utility sector is also
called Energy Internet, as it includes a great deal of
heterogeneous things like electric vehicles (EVs),
charging stations, intelligent electronic devices
(IEDs), smart meters, phasor measurement units
(PMUs), and automated revenue metering (ARM)
to facilitate a decentralized network of energy and
information, for example, in smart grids (SGs).
In Energy Internet, there is significant volume,
velocity, variety, veracity, and value (5 Vs)of data
sensed/collected, disseminated, and so on from
varying sources and utilized for business analytics
and decision making. According to an estimate
from International Data Corporation, for example,
the volume of data is doubling every two years
and exceeded 44 ZB at the end of 2020 [2].
Although the development of SGs has associated
benefits such as efficiency, reliability, and econom-
ics, big data also poses several challenges (e.g., in
data resources, transmission, storage, and analysis
[3]). Furthermore, the multi-sourced heteroge-
neous data within the Energy Internet ecosystem
is challenging to deal with in an acceptable time
frame for time-critical applications with existing
hardware and networking resources. Hence, there
IEEE Network • January/February 2021
2/2/21 6:30 PM
 have been concerns about QoS provisioning,
demand response management, customer profil-
ing, network planning, and pricing. Since data col-
lected from SGs is huge and expected to increase
further, traditional statistical models may not work
well. Consequently, this necessitates the need for
advanced data analytics and informatics (e.g., big
data analytics — BDA). A high-level overview of
Energy Internet is illustrated in Fig. 1.

N eed for Edge-Cloud Interplay for

Big Data Analytics in Energy Internet
In order to support BDA, cloud computing has
become popular over traditional computing mod-
els due to offerings like flexibility, scalability, ener-
gy efficiency, agility, and cost savings [4]. Although
the cloud provides unlimited resources along with
a wide range of services for compute-intensive
applications, delay-sensitive applications still suf-
fer from latency and bandwidth issues. Further,
the current cloud architecture also fails to fulfill
the requirements of mobility support and location
awareness. Therefore, a decentralized solution is
desired that can provide compute, data, storage,
and application services while obeying strict delay
requirements. In this context, edge computing
has emerged as a viable solution, which performs
computation closer to the proximity of end users.
The distinguishing features of edge computing
include its dense geographical distribution, high
mobility support, location awareness, heterogene-
ity, low bandwidth requirements, high throughput,
and low latency [5]. Despite several benefits, this
paradigm suffers from several issues like limited
redundancy, longer outage time, resource-con-
straints, cost effectiveness, and service hetero-
geneity. Thus, efficient interplay between cloud
and edge devices is required to overcome the
challenges they face [6]. In order to manage the
service migrations between edge and cloud data
centers, several solutions already exist including
cloudlets, fog computing, micro data centers, and
so on. However, a large number of migrations
among them may cause network congestion and
energy consumption issues. This suggests the
urgent need for a softwarized networking para-
digm that can support the interactions between
cloud and edge while offering good QoS.
SDN-Enabled Network Function Virtualization
Toward a flexible softwarized networking par-
adigm, software defined Networking (SDN)
and network function virtualization (NFV) have
emerged as promising solutions. SDN decouples
the control plane from the data plane to support
the programmability of network functions. More-
over, it performs network virtualization with specif-
ic QoS and service-level agreement requirements.
However, the dependence on physical appliances
imposes restrictions on this paradigm in deploying
new network services, such as capacity and avail-
ability. Thus, NFV has evolved, which provides
network functionality while enabling better provi-
sioning of network services [7]. More specifically,
it relates to the deployment of network functions
in specialized and dedicated hardware (running
on commodity servers in data centers) through
software virtualization techniques. However, the
interconnection of virtualized network functions
FIGURE 1. Understanding Energy Internet: a snapshot.
(VNFs) again poses challenges to the underlying
network infrastructure. Moreover, the dynamic
provisioning of services and VNF migration make
this task even more complex [8]. As a result, NFV
and SDN can be integrated in order to improve
the flexibility and simplicity of networks and ser-
vice delivery over them [9]. The amalgamation of
NFV and SDN enables protocol programmability,
flexible deployment, improved network function,
ubiquitous accessibility, centralized management,
dynamic adjustment of network resources, and
flexible provisioning in the network.
Virtualization Practices for Realizing NFV
The VNFs are usually deployed on high-capacity
servers that are present on cloud infrastructures,
unlike an edge computing setup. Hence, to cater
to the low computational requirement of the net-
work edge, virtualization is often required, which
is lightweight and scalable. Until now, the majority
of NFV deployments have focused on the utiliza-
tion of virtual machines (VMs) where the idea is
to abstract and leverage the hardware-based net-
work functions as software. However, the increas-
ing deployment of fifth generation (5G) networks,
which are mainly based on NFV/SDN technol-
ogies, pose stringent requirements in terms of
agility, latency, and support for high-end tech-
nologies such as virtual reality, machine learning,
and autonomous vehicles. Although VMs provide
complete isolation from host operating systems,
that too incurs the cost of performance penalty
with respect to virtualization. In order to cope
with this trend and accelerate the deployment of
VNFs, a new virtualization technology, contain-
ers, have emerged in order to overcome the strin-
gent hardware requirements associated with VMs.

IEEE Network • January/February 2021 357

Authorized licensed use limited to: Western Sydney University. Downloaded on June 16,2021 at 00:20:14 UTC from IEEE Xplore. Restrictions apply.

GARG_LAYOUT.indd 357 2/2/21 6:30 PM

 FIGURE 2. An illustration of the Energy Internet ecosystem.
They are lightweight and isolated Linux processes
that deploy and run distributed applications on
VNF infrastructure at the cost of reduced isola-
tion. With containerization, an application runs
quickly and reliably from one computing environ-
ment to another. Moreover, it has several benefits
in comparison to VM technology, including low
resource overhead, agility to launch new service
or upgrade existing service at runtime, resource
scalability according to demands, faster deploy-
ment and portability of new services, resiliency
against failures, and so on. Further, applications
like mobile edge computing (MEC), network slic-
ing, and cloud radio access networks (cRANs)
also demand containers for highly automated
deployment of services in the 5G network. How-
ever, their practical deployment is limited by their
support for VNF deployment on a certain set of
OSs, namely Linux, Windows, and Solaris. On the
contrary, there are numerous NFV infrastructures
that do not yet support the above mentioned OSs
currently. Thus, using container-only deployments
for VNF deployment is not feasible, particularly
for the present heterogeneous architecture sur-
faced by the edge and the cloud [10].
Motivation
The growing interconnections in the Energy Inter-
net ecosystem include networks, spanning fleets
of EVs, groups of homes, industrial sectors, build-
ings, and so on. Thus, the heterogeneous data
generated by these interconnected and complex
networks is growing at an exponential rate. Its
real-time processing and transmission to the cen-
tralized cloud infrastructure for deep analysis is an
important research direction. In this vein, here we
358
Authorized licensed use limited to: Western Sydney University. Downloaded on June 16,2021 at 00:20:14 UTC from IEEE Xplore. Restrictions apply.
GARG_LAYOUT.indd 358
present a novel networking architecture that amal-
gamates SDN and NFV paradigms while simulta-
neously extending their capabilities using an edge
computing platform. Further, it also leverages the
concept of mixed VNF deployment using VMs
and containers. Although similar approaches have
been proposed in the literature, their extension to
generic platforms/domains limit their adaptability
in the ever growing Energy Internet ecosystem.
Further, the proposed work also puts forward the
concept of mixed VNF deployment, which has
not been explored to its full potential. Thus, to
the best of our knowledge, this is the first initiative
of its kind that achieves network agility and flexi-
bility in the Energy Internet domain.
Research Contributions of This Work
In this article, we present a novel architecture for
SDN integrated with NFV specifically for the Ener-
gy Internet ecosystem by leveraging the advantag-
es of edge computing. In the considered setup,
the deployment of VNFs is achieved using both
virtualization and containerization (a lightweight
means to achieve virtualization). The key contri-
butions of the proposed work are summarized as
follows:
• The designed framework presents an
enabling network solution to andEnergy
Internet utility provider with the means of
automated provisioning, configuration, and
optimization of NFVs. In other words, the
on-demand provisioning of network func-
tions can be referred to as VNF as a service
(VNF-aaS). The proposed framework extends
the capabilities of the SDN platform along
with NFV to achieve the same.
IEEE Network • January/February 2021
2/2/21 6:30 PM
 • Additionally, the proposed solution also
leverages the benefits of VMs and containers
for VNF deployment in heterogeneous com-
puting infrastructure surfaced by the edge
and the cloud.
• In order to attain the objective of VNF
deployment across the heterogeneous vir-
tualized ecosystem, this article presents a
dedicated solution based on integer linear
programming.
• Finally, the article concludes with challenges
and opportunities associated with the pro-
posed solution.
Organization
The rest of the article is organized as follows. The
following section presents information about Ener-
gy Internet, followed by an illustrative discussion
about the system model. The proposed solution
for addressing the VNF deployment issue in a het-
erogeneous environment is presented with asso-
ciated challenges and opportunities. Finally, the
article concludes with future directions.
Layered Energy Internet Ecosystem
The considered Energy Internet ecosystem
comprises a smart power system environment
equipped with a bidirectional mode of commu-
nication and energy. It is essentially a smart grid
setup comprising different entities connected
directly or via an interface to each other. These
entities are EVs, smart homes, industrial units,
buildings, and components of the microgrid. Fig-
ure 2 illustrates the interconnection among these
entities and their respective networks using smart
meters. The considered Energy Internet infrastruc-
ture can be conceptualized as the integration of
the following three layers: customer, power and
control transmission, and power generation layer.
Their detailed information is illustrated under:
• Customer layer: This layer essentially com-
prises various networks such as home area
networks (HANs), building area networks
(BANs), industrial area networks (IANs),
micro grid area networks (MGANs), and EV
area networks (EVANs). The energy con-
sumption details of these networks are moni-
tored through smart meters.
• Power and Control Transmission Layer: This
layer is responsible for data collection from
all SMs across all networks and EV CSs.
Additionally, it is also responsible for the
transmission of energy across the customer
layer. In summary, it is the most important
layer, helping to establish long-haul com-
munication between the above-mentioned
networks and controllers for providing var-
ious high-level services including demand
response, ancillary services, renewable ener-
gy management, and so on.
• Power Generation Layer: This layer is respon-
sible for generating electricity from various
sources including renewable and non-renew-
able energy sources.
Proposed SDN-NFV Framework Using VNF-aaS
In this segment, communication infrastructure in
the Energy Internet domain using the integration
of NFV, SDN, edge, and cloud computing is pre-
sented. The key benefit of coupling SDN with NFV
IEEE Network • January/February 2021
Authorized licensed use limited to: Western Sydney University. Downloaded on June 16,2021 at 00:20:14 UTC from IEEE Xplore. Restrictions apply.
GARG_LAYOUT.indd 359
The Cloud computing plane provides unlimited data storage and computational services. Further,
it extends its services using the pay-as-you-use concept via different services such as Platform-as-a-
Service (PaaS), Software-as-a-Service (SaaS), Infrastructure-as-a-Service (IaaS), and so on.
is that it reduces capital expenditure (CAPEX) and
operational expenditure (OPEX). Further, their
amalgamation also supports fast recovery from fail-
ures and simplified configuration of new network
service by the SDN’s centralized controller.
Figure 3 provides a high-level view of the pro-
posed framework, comprising six basic planes: Energy
Internet plane, edge computing plane, cloud com-
puting plane, SDN-NFV infrastructure plane, NFV
management and orchestration plane, and virtual
infrastructure management plane. The detailed infor-
mation about these planes is illustrated as follows:
Energy Internet Plane
The Energy Internet plane comprises differ-
ent distributed networks such as HANs, BANs,
IANs, MGANs, and EVANs. These networks are
equipped with SMs, which in turn keep track of
their respective electricity consumption. The data
captured from these distributed deployed networks
is captured by the SMs and is sent to the controller
for providing different Energy Internet applications
including DR, ancillary services, frequency sup-
port, vehicle-to-grid (V2G) services, and so on. The
details of this plane are already discussed above.
Edge Computing Plane
This plane consists of a large pool of heteroge-
neous decentralized nodes with the ability to per-
form computational and storage tasks without
the intervention of third parties. The distributed
processing nodes encapsulate a wide range of
edge nodes such as nanoDC, microDCs, smart-
phones, laptops, computers, sensors, tablets, and
wireless access routers. These nodes enable data
processing near the proximity of the end user,
which involves limited latency and quick turn-
around. However, these nodes are limited by their
storage and computational processing capabilities
and thus rely on the core cloud computing plane
for execution of computationally intensive tasks.
In summary, the edge can be considered as an
extension of cloud near the end user.
Cloud Computing Plane
The cloud computing plane provides unlimited
data storage and computational services. Further,
it extends its services using the pay-as-you-use
concept via different services including platform
as a service (PaaS), software as a service (SaaS),
infrastructure as a service (IaaS), and so on. In the
considered networking framework, the aim is to
exploit the virtualized and containerized instances
of the computing layer for seamless deployment
of VNFs. This service is referred to as VNF-aaS
and would be acquired by the proposed frame-
work as and when across the edge and the cloud.
SDN-NFV Infrastructure Plane
This plane integrates the functionality of NFV
within the SDN framework.
NFV is an emerging technology that helps to
migrate certain network services on virtualized
359
2/2/21 6:30 PM
 FIGURE 3. An illustration for the proposed framework using SDN-NFV virtualization and containerization
approaches [13].
instances, which otherwise are executed on
hardware elements. NFV also helps in the rapid
deployment of network functionalities as software
components with ease of configuration facility.
Additionally, it supports elastic scaling of network
services. Essentially, NFV supports network slic-
ing by logically segregating the network and its
resources. This is achieved by virtualization on the
physical network; hence, the name NFV. Some
prominent examples of NFV functions include
TCP optimization, firewall, NAT, load balancing,
and so on [11].
On the other hand, the paradigm of SDN
is characterized by the decoupling of the con-
trol plane from the data plane. Further, it is also
notable for its ability to centralize its control
intelligence. The entity where the central control
rests is referred to as the SDN controller, that is,
centralized software to implement the network
functionality. SDN operates using the OpenFlow
protocol (OFP) for control information transmis-
sion from the SDN controller to the forwarding
elements [12].
Essentially, SDN comprises three important
planes: the data, control, and application planes.
SDN’s data plane comprises the OpenFlow
switches and routers (a.k.a., forwarding elements)
that form the backbone of the networking plat-
form. The capabilities of this plane are made avail-
able to the upper plane using the southbound
360
Authorized licensed use limited to: Western Sydney University. Downloaded on June 16,2021 at 00:20:14 UTC from IEEE Xplore. Restrictions apply.
GARG_LAYOUT.indd 360
application programming interface (API). The
control plane, on the other hand, incorporates
SDN’s controller and NFV manager (details of
the NFV manager follow). Its capabilities are
exposed to the application plane by leveraging
the northbound API. The uppermost application
plane takes care of different networking services/
applications, each having access to a certain class
of underlying resources.
NFV Management and Orchestration Plane
The roles and responsibilities of this plane are
aligned to the NFV manager, which allows it to
achieve VNF management and orchestration. It
achieves the same using three significant func-
tional blocks: NFV orchestrator (NFVO), VNF
manager (VNFM), and virtualized infrastructure
manager (VIM). Here, the NFVO helps in man-
aging the life cycle activities of network services,
while the VNFM coordinates the life cycle of
VNFs. The VIM, on the other hand, manages the
NFV infrastructure resources in order to provide
the required resources for VNF deployment and
connectivity (details follow).
It is worth mentioning here that the terms net-
work services and service chains are often inter-
changeable. However, their respective meanings
differ a lot. For instance, a network service is
essentially the final product that is delivered to the
end user and is typically characterized by its func-
IEEE Network • January/February 2021
2/2/21 6:30 PM
 tionality and performance. It is usually defined
for a virtual network with predefined source and
destination such that the traffic flowing over it is
subjected to a certain operation by a designat-
ed network service. On the contrary, a service
chain depicts the actual structure of a network
service and comprises a series of network func-
tions called VNFs. These VNFs are characterized
by the direction of traffic and a dedicated ingress
and egress point.
Virtual Infrastructure Management Plane
This plane is managed by the VIM, which plays
an essential role in controlling the physical and
virtual resources of the NFV infrastructure. In
conventional NFV infrastructures, virtualization
techniques based on standard virtual machines
(VMs) based on Xen, VMWare, Hyper-V, KVM,
and so on have been predominantly used. How-
ever, very recently a range of lightweight virtu-
alization technologies have been suggested for
NFV deployments. These lightweight virtualized
instances comprise containers, unikernels, and
minimalistic distributions of different OSs [14,
15]. Among these, the most sought-after means
is to use containers for VNF deployment. How-
ever, their practical deployment is limited by
their support for VNF deployment on a certain
set of OSs, namely Linux, Windows, and Solar-
is. On the contrary, there are numerous NFV
infrastructures that do not yet support the above
mentioned OSs currently. Thus, using contain-
er-only deployments for VNF deployment is not
feasible, particularly for the present heteroge-
neous architecture surfaced by the edge and the
cloud. Toward this end, an efficient technique
is to consider a hybrid deployment philosophy
for VNF deployment. This implies that NFV infra-
structure at the edge and cloud could contain
a mix of both VMs and containers. This calls
for designing a specialized NFV controller for
achieving optimal deployment of VNFs across
the considered virtualized instances. The details
of the proposed solution to attain the same are
discussed in the upcoming section.
The proposed Energy Internet ecosystem com-
prises a smart power system environment equipped
with a bidirectional mode of communication and
energy. It is essentially a smart grid setup compris-
ing different entities connected directly or via an
interface with each other. These entities are name-
ly EVs, smart homes, industrial units, buildings, and
components of the microgrid. Figure 2 illustrates
the interconnection among these entities and
their respective networks using smart meters. The
considered Energy Internet infrastructure can be
conceptualized as the integration of the following
three layers: customer, power and control trans-
mission, and power generation. Their detailed
information is illustrated under:
• Customer layer: This layer essentially compris-
es various networks such as HANs, BANs,
IANs, MGANs, and EVANs. The energy con-
sumption details of these networks are moni-
tored through smart meters.
• Power and Control Transmission Layer: This
layer is responsible for data collection from
all SMs across all networks and EV CSs.
Additionally, it is also responsible for the
transmission of energy across the customer
IEEE Network • January/February 2021
Authorized licensed use limited to: Western Sydney University. Downloaded on June 16,2021 at 00:20:14 UTC from IEEE Xplore. Restrictions apply.
GARG_LAYOUT.indd 361
The proposed Energy Internet ecosystem comprises a smart power system environment equipped with
a bidirectional mode of communication and energy. It is essentially a smart grid setup
comprising different entities connected directly or via an interface with each other.
layer. In summary, it is the most important
layer, helping to establish long-haul com-
munication between the above-mentioned
networks and controllers for providing var-
ious high-level services including demand
response, ancillary services, renewable ener-
gy management, and so on.
• Power Generation Layer: This layer is respon-
sible for generating electricity from various
sources including renewable and non-renew-
able energy sources.
VNF Deployment in the Proposed Setup
This section presents the VNF deployment prob-
lem across a hybrid cluster of virtualized instanc-
es in the edge-cloud setup. The details of the
designed problem are illustrated below.
The designed solution operates across two
main stages. In the first stage, the designed solu-
tion tries to find the optimal placement of a VNF
in proximity of the user (i.e., at the edge). If a
promising solution can be deduced at this stage,
a placement opportunity (either a VM or con-
tainer) can be concluded, and then the consid-
ered VNF is deployed at the edge. Otherwise,
the corresponding VNF deployment request is
forwarded to the core cloud computing plane
for further processing. However, finding the
optimal deployment policy for a group of VNFs
on virtualized instances is not straightforward as
it considers multiple objectives and constraints.
The considered problem takes into account two
main objectives, that is, maximum deployment
of VNFs across the time horizon so as to ensure
high QoS (F1(Xij)), with minimal execution time
(F 2(X ij)). These objectives are expressed as fol-
lows:
M N
F1 (X ij ) = max ∑ ∑ Xij (1)
i=1 j=1
M N
F2 (X ij ) = min ∑ ∑ Xij × Pij (2)
i=1 j=1
In the above equations, Pij denotes the processing
time of executing the ith VNF and jth virtualized
instance (either a VM or a container). Further,
variables M and N refer to the total number of
VNFs to be deployed and the available array of
VMs and containers, with i and j as their respec-
tive indices. The variable Xij is the binary decision
variable with the following definition:
X ij =
⎪⎧1 : If i th VNF is deployed on j th virtualized instance
⎨
⎪⎩0:Otherwise
In addition to the above objectives, the formu-
lated problem also considers the following set
of constraints while deploying the VNFs across
containers/VMs.
N
For instance, constraint C 1 (i.e., S j=1 X ij = 1)
denotes the VNF deployment restriction that
361
2/2/21 6:30 PM
 Thus, the overall problem consists of two

Cumulative Processing Time (s)

35
Propsed Scheme
objective functions (F1(Xij) and (F2(Xij)) and the
30
Existing Scheme following constraints:
25 F(X ij ) = ⎡⎣ −F1 (X ij ), F2 (F1 ) ⎤⎦
(3)
20 s.t. Constraints defined above hold

15 The above problem is a typical integer linear
problem with linearly defined objectives and con-
10 straints. However, it resembles the scheduling
problem and is NP-hard in nature. Thus, to obtain
5
optimal placement of VNFs in a hybrid virtual-
0 ized setup, the work exploits the advantages of
50 100 150 200 250 300 350 400 450
a popular multi-objective evolutionary algorithm,
Number of VNFs e-Non-dominated Sorting Genetic Algorithm-II
(a) (e-NSGA-II). It is well known for its ability of faster
200 convergence, reduced overall complexity, explic-
180
Propsed Scheme
Existing Scheme it diversity preservation capability, epsilon-domi-
nance, adaptive population sizing, and automatic
Number of Migration

160

140
termination.
120
Observation and Analysis
100
In this section, evaluation of the proposed
80 scheme for VNF placement in the hybrid virtual-
60 ized environment over the edge-cloud integration
40 is performed. For performance evaluation, the
20
proposed scheme has been accessed using the
widely used MOEA framework, a Java-based open
0
50 100 150 200 250 300 350 400 450 source platform for evaluating the performance of
Number of VNFs various multi-objective evolutionary algorithms.
(b) The simulation setup considered a total of 20
7500
nDCs and 1 cloud DC with variable configura-
7000
Propsed Scheme tions. In addition to this, we also assumed a total
Existing Scheme
of 150 containers and 150 VMs to be available
Aveage Overhead (ms)

6500
6000
5500
5000
4500
4000
3500
3000
2500
2000
50 100 150 200 250
Number of VNFs
(c)
FIGURE 4. An illustrative comparison of the proposed
scheme against the existing scheme: a) process-
ing time analysis; b) number of migrations;
c) overhead analysis.
allows a single VM/container to be allocated to
N
a particular VNF. Constraint C 2 (i.e., Sj=1 Xij  Pij
≤ PRi) refers to the upper limit on the processing
time of a particular VNF across different virtual-
ized instances. Here, the variable PRi refers to the
upper limit on the processing time of the ith VNF.
The rest of the constraints (C3, C4, C5) restrict the
number of resources (namely CPU cores, main
memory, and storage) allocated for the execution
M
of a particular VNF. Here, C3 = Si=1 Xij  Ci ≤ Cj,
M M
C4 = Si=1 Xij  Mi ≤ Mj, and C5 = Si=1 Xij  Si ≤ Sj;
wherein the variables Ci, M i, and S i refer to the
resource requirement of the ith VNF in terms of
CPU cores, main memory, and storage, respec-
tively. Likewise, the upper limit on these resourc-
es with respect to the jth virtualized instance is
expressed as Cj, Mj, and Sj.
in the considered edge-cloud setup with differ-
ent CPU, memory, and storage characteristics.
The number of VNFs waiting to be deployed was
assumed to be in the range of [20, 450]. Further,
the proposed scheme has been compared with
an existing scheme for VNF placement on VMs.
The obtained results are summarized as follows.
The evaluation results have been outlined on
the basis of three different evaluation criteria: pro-
cessing time, number of migrations, and overhead
300 350 400 450
analysis. The description is illustrated below.
Processing Time Analysis: It refers to the total
time for executing a particular VNF on a physical/
virtual machine in addition to the time to wait for
a free virtualized instance for execution. The total
time is expressed in terms of seconds.
Number of Migrations: This metric refers to
the number of VNF requests that need to be
migrated to the core cloud in lieu of insufficient
resources at the edge or longer waiting time for
VNF deployment. A higher number of migrations
lead to higher overall overhead.
Overhead Analysis: This refers to the addition-
al overall associated with the deployment of VNFs
on VMs and containers. It refers to the total time
required for instantiation and booting of VMs and
containers. It is expressed in milliseconds.
Figure 4a depicts the overall processing time
analysis of the proposed scheme relative to the
existing scheme. It is evident from the figure that
the processing time for both schemes increases
with increasing VNF requests for deployment.
More importantly, the processing time in the
VM-only execution setup shows enhanced dura-
tions of processing VNF requests relative to the
proposed scheme. This can be attributed to lim-

362 IEEE Network • January/February 2021

Authorized licensed use limited to: Western Sydney University. Downloaded on June 16,2021 at 00:20:14 UTC from IEEE Xplore. Restrictions apply.

GARG_LAYOUT.indd 362 2/2/21 6:30 PM

 ited resources at the edge and migration of the
incoming request to the cloud. The related results
are showcased in Fig. 4b. It is clearly noticeable
from the obtained results that the proposed
scheme incurs fewer migrations relative to the
existing scheme, although the number of VNF
requests depict a steady increase in the number of
migrations. This is primarily due to the lightweight
nature of containers, which can be deployed,
instantiated, and rebooted with reduced overhead
in comparison to conventional VMs. The associat-
ed results have been depicted using Fig. 4c.
In a nutshell, VNF placement brings greater
advantages in hybrid execution setup powered by
VMs and containers, and is thus better than the
VNF deployment strategy in a VM-only setup.
Challenges
This section presents the challenges associated
with network provisioning using NFV and SDN in
an edge-cloud setup [13].
SDN Management: The concept of NFV and
SDN are closely associated with each other and
are essentially complementary. However, this
does not imply that one is dependent on the
other. These two networking paradigms offer high
levels of dynamism and automation, which reduce
human intervention and control. Thus, tradition-
al network management approaches need sig-
nificant improvisation in order to accommodate
these paradigms. Additionally, the management
in SDN also confronts some open research ques-
tions such as the number of SDN controllers, their
deployment, and conflict management. Conse-
quently, the integration of SDN and NFV requires
significant research efforts on the above-men-
tioned issues.
Virtualization Technologies: More recently, a
number of lightweight techniques for virtualiza-
tion have been put forward. These technologies
encapsulate containers, uni-kernels, and mini-
malistic distributions of general-purpose operat-
ing systems. The associated challenges involve
constructing NFV infrastructure using them and
addressing their inherent limitations.
Separation of Control Logic: The SDN-NFV
environment offers control implementation at
two levels. The first level helps in controlling the
NFV infrastructure in order to support end-to-end
connectivity among VNFs. The second level of
control, on the other hand, helps in controlling
and configuring VNFs by exploiting SDN’s pro-
grammability features. Nevertheless, the two con-
trol logics are separate from each other. Thus, an
important research direction is to explore the fun-
damental interactions between the two in order
to optimize and scale their performance.
Security: The proposed SDN-NFV architecture
leads to resource sharing, which in turn induces
security concerns. For instance, multiple tenants
may need to coexist in a virtual or semi-virtual
environment, which may hamper its security.
Along similar lines, network slicing on the same
infrastructure can also lead to critical security
issues like one application affecting the other on a
different network slice. Thus, security implementa-
tion at the core of SDN-NFV integration is essen-
tial to support seamless network services.
Energy Efficiency: A substantial amount of
energy is consumed worldwide for data transmis-
IEEE Network • January/February 2021
Authorized licensed use limited to: Western Sydney University. Downloaded on June 16,2021 at 00:20:14 UTC from IEEE Xplore. Restrictions apply.
GARG_LAYOUT.indd 363
Although the development of the Energy Internet brings great improvement to the power grid, the
volume and complexity of data pose several challenges. To address these challenges, NFV and SDN
have emerged as strong candidates for analyzing and processing the big data generated by the Energy
Internet ecosystem, specifically in the edge-cloud environment.
sion and networking. Toward this end, the rapid
virtualization of network services brings new
opportunities and challenges for creating ener-
gy-efficient networking solutions.
Conclusion
As the world becomes increasingly connected,
the power system landscape has been undergo-
ing a major shift. Due to the low cost and easy
deployment of smart sensors, power grids are
generating big data on a real-time basis. This in
turn creates room for providing various smart
applications like efficient demand response, ancil-
lary services, vehicle-to-grid support, and smart
home management. Although the development
of the Energy Internet brings great improvement
to the power grid, the volume and complex-
ity of data pose several challenges. To address
these challenges, NFV and SDN have emerged
as strong candidates for analyzing and process-
ing the big data generated by the Energy Internet
ecosystem, specifically in the edge-cloud environ-
ment. To address these emerging challenges, this
article proposes an edge computing architecture
for SDN-NFV platform based on containerization
and virtualization. In particular, the framework
provides automated provisioning of network ser-
vices with enhanced flexibility, efficiency, and
scalability. Additionally, the article also proposes
a multi-objective VNF placement approach in a
hybrid execution environment powered by VMs
and containers. The formulated problem is based
on integer linear programming and is solved using
e-NSGA-II. The obtained results indicate superi-
or performance of SDN-NFV integration in a
hybrid execution environment in contrast to the
VM-based infrastructure.
In the future, the proposed VNF deployment
problem will be extended while taking into con-
sideration different networking parameters includ-
ing resilience, security, energy efficiency, and so
on.
References
[1] M. Chen et al., “Cognitive Information Measurements: A New
Perspective,” Info. Sciences, vol. 505, 2019, pp. 487–97.
[2] B. Li et al., “Big Data Analytics for Electric Vehicle Integration
in Green Smart Cities,” IEEE Commun. Mag., vol. 55, no. 11,
Nov. 2017, pp. 19–25.
[3] K. Wang et al., “Wireless Big Data Computing in Smart Grid,”
IEEE Wireless Commun., vol. 24, no. 2, Apr. 2017, pp. 58–64.
[4] J. Baek et al., “A Secure Cloud Computing Based Framework
for Big Data Information Management of Smart Grid,” IEEE
Trans. Cloud Computing, vol. 3, no. 2, 2015, pp. 233–44.
[5] E. Ahmed et al., “Bringing Computation Closer toward the
User Network: Is Edge Computing the Solution?” IEEE Com-
mun. Mag., vol. 55, no. 11, Nov. 2017, pp. 138–44.
[6] K. Kaur et al., “Edge Computing in the Industrial Internet
of Things Environment: Software-Defined-Networks-Based
Edge-Cloud Interplay,” IEEE Commun. Mag., vol. 56, no. 2,
Feb. 2018, pp. 44–51.
[7] R. Muñoz et al., “Integrated SDN/NFV Management and
Orchestration Architecture for Dynamic Deployment of
Virtual SDN Control Instances for Virtual Tenant Networks,”
J. Optical Commun. Networking, vol. 7, no. 11, 2015, pp.
B62–70.
363
2/2/21 6:30 PM
 [8] Q. Duan et al,. “Software-Defined Network Virtualization: An
Architectural Framework for Integrating SDN and NFV for
Service Provisioning in Future Networks,” IEEE Network, vol.
30, no. 5, Sept./Oct. 2016, pp. 10–16.
[9] J. Matias et al., “Toward an SDN-Enabled NFV Architecture,”
IEEE Commun. Mag., vol. 53, no. 4, Apr. 2015, pp. 187–93.
[10] S. Nangare. (2018, Aug.) “Virtual Machines or Containers,
Which Is Better in NFV Infrastructure?,” Aug. 2018 vmblog.
com-Virtualization Technology News and Information; http://
vmblog:com/archive/2018/08/07/virtual-machines-or-con-
tainers-which-is-better-in-nfvinfrastructure:aspx#:XC86Zc9KgV,
accessed Dec. 2018.
[11] K. Kaur et al., “An Energy-Driven Network Function Virtual-
ization for Multi-Domain Software Defined Networks,” Proc.
IEEE INFOCOM Wksps., 2019, pp. 121–26.
[12] ONF; https://www:opennetworking:org, accessed Dec.
2018.
[13] C. Bernardos et al., “Network Virtualization Research Chal-
lenges,” IETF Draft, 2018.
[14] S. Natarajan et al., “An Analysis of Lightweight Virtualiza-
tion Technologies for NFV,” IETF Draft, 2017.
[15] R. Cziva and D. P. Pezaros, “Container Network Functions:
Bringing NFV to the Network Edge,” IEEE Commun. Mag.,
vol. 55, no. 6, June 2017, pp. 24–31.
Biographies
S ahil G arg [S’15, M’18] (sahil.garg@ieee.org) received his
Ph.D. degree from the Thapar Institute of Engineering and Tech-
nology, Patiala, India, in 2018. He is currently a post-doctoral
research fellow at École de Technologie Supérieure, Université
du Québec, Montréal, Canada. He has many research con-
tributions in the area of machine learning, big data analytics,
security and privacy, the Internet of Things, and cloud comput-
ing. He has over 50 publications in high ranked Journals and
conferences, including 25+ IEEE transactions/journal papers.
He received the IEEE ICC best paper award in 2018 at Kansas
City, Missouri; and also the IEEE TCSC Award for Excellence
in Scalable Computing (Early Career Researcher) in 2020. He
serves as the Managing Editor of Springer’s Human-centric Com-
puting and Information Sciences and an Associate Editor of IEEE
Network, IEEE Transactions on Intelligent Transportation Systems,
Applied Soft Computing (Elsevier), and Wiley’s International
Journal on Communication Systems. He also serves as the Work-
shops and Symposia Officer for the IEEE ComSoc Emerging
Technology Initiative on Aerial Communications. He has guest
edited number of Special Issues in top-cited journals including
IEEE T-ITS, IEEE TII, the IEEE Internet of Things Journal, IEEE Net-
work, and Future Generation Computer Systems (Elsevier). He
serves/has served as the Workshop Chair/Publicity Co-Chair
for several IEEE/ACM conferences including IEEE INFOCOM,
IEEE GLOBECOM, IEEE ICC, ACM MobiCom, and so on. He is
a member of ACM.
Kuljeet Kaur [S’13, M’18] (kuljeet.kaur@ieee.org) received her
Ph.D. degree from the Thapar Institute of Engineering and Tech-
nology in 2018. She worked as a NSERC Postdoctoral Research
Fellow at the École de technologie supérieure (ÉTS), Université
du Québec, Montréal, Canada from 2018 till 2020. She is cur-
rently working as an Assistant Professor in the Electrical Engi-
364
Authorized licensed use limited to: Western Sydney University. Downloaded on June 16,2021 at 00:20:14 UTC from IEEE Xplore. Restrictions apply.
GARG_LAYOUT.indd 364
neering Department at ÉTS, Montreal and a Visiting Researcher
in the School of Computer Science and Engineering (SCSE) at
the Nanyang Technological University (NTU), Singapore. She
has made many research contributions in the areas of cloud
computing, energy efficiency, smart grid, frequency support, and
vehicle-to-grid. Some of her research findings are published in
top cited journals such as IEEE TII, IEEE TMM, IEEE TSG, IEEE TVT,
IEEE TNSM, the IEEE Systems Journal, IEEE TSUSC, IEEE Communi-
cations Magazine, IEEE Wireless Communications, IEEE Network,
IEEE PS, and Springer’s PPNA. She received the IEEE ICC best
paper award in 2018 at Kansas City, Missouri; and the 2019 Best
Research Paper Award from Thapar Institute of Engineering and
Technology, India. She serves as Associate Editor of Wiley’s Secu-
rity and Privacy and Springer’s Journal of Information Processing
Systems. She is a website Co-Chair of the N2Women communi-
ty, and also serves as the Vice-Chair of the IEEE Montreal Young
Professionals Affinity Group. She has also been a TPC Co-chair
for IEEE Infocom 2021, IEEE Infocom 2020 and ACM MobiCom
2020 workshops on DroneCom. She is a member of ACM.
G eorges K addoum [M’11] (georges.kaddoum@etsmtl.ca)
received his Bachelor’s degree in electrical engineering from the
École Nationale Supérieure de Techniques Avancés (ENSTA),
France, his M.Sc. degree in telecommunications and signal pro-
cessing from Telecom Bretagne (ENSTB), Brest, in 2005, and his
Ph.D. degree in signal processing and telecommunications from
the National Institute of Applied Sciences (INSA), Toulouse,
France, in 2009. He is currently an associate professor and
Tier 2 Canada Research Chair with the École de Technologie
Supérieure. His recent research activities cover wireless commu-
nication networks, resource allocations, security and space com-
munications, and navigation. He was awarded the ÉTS Research
Chair in physical-layer security for wireless networks in 2014,
and the prestigious Tier 2 Canada Research Chair in wireless
IoT networks in 2019. He has published over 150+ journal and
conference papers and has two pending patents. In addition,
he received the research excellence award of the Université du
Québec in 2018. In 2019, he received the research excellence
award from the ÉTS in recognition of his outstanding research
outcomes.
S ong G uo [M’02, SM’11, F’19] (song.guo@polyu.edu.hk)
received his Ph.D. degree in computer science from the Uni-
versity of Ottawa, Canada. He is currently a full professor in the
Department of Computing, Hong Kong Polytechnic University.
His research interests mainly include cloud and green comput-
ing, big data, and cyber-physical systems. He is the recipient of
the 2019 IEEE TCBD Best Conference Paper Award, 2018 IEEE
TCGCC Best Magazine Paper Award, 2017 IEEE Systems Journal
Annual Best Paper Award, and six other Best Paper Awards from
IEEE/ACM conferences. His work was also recognized by the
2016 Annual Best of Computing: Notable Books and Articles in
Computing in ACM Computing Reviews. He was an IEEE Com-
Soc Distinguished Lecturer (2017–2018) and served on the IEEE
ComSoc Board of Governors (2018–2019). He has also served
as General and Program Chair for numerous IEEE conferences.
He is the Editor-in-Chief of the IEEE Open Journal of the Comput-
er Society, and Associate Editor of IEEE Trans. Cloud Computing,
IEEE Transactions on Sustainable Computing and IEEE Transac-
tions on Green Communications and Networking.
IEEE Network • January/February 2021
2/2/21 6:30 PM