The Journal of Supercomputing (2018) 74:5944–5979

https://doi.org/10.1007/s11227-018-2508-1

VMDFS: virtual machine dynamic frequency scaling

framework in cloud computing

Kiamars Shojaei1 · Faramarz Safi-Esfahani2,3 · Saeed Ayat4

Published online: 4 August 2018

© Springer Science+Business Media, LLC, part of Springer Nature 2018

Abstract
Development of modern techniques, such as virtualization, underlies new solutions
to the problem of reducing energy consumption in cloud computing. However, for
the infrastructure as a service providers, it would be a difficult process to guarantee
energy saving. Analysis of the workload of applications shows that the average uti-
lization of virtual machines has many fluctuations; therefore, deciding about how to
control such fluctuations in virtual machines plays a significant role in improving the
energy consumption of datacenters. In this study, an adaptable model called virtual
machine dynamic frequency system (VMDFS) has been developed whose its innova-
tion is monitoring the average fluctuations of workloads to vary the CPU frequency of
virtual machines at runtime, dynamically. In this model, enhanced exponential moving
average method is used to predict workload fluctuations, and then after calculating a
smoothing coefficient for the utilization fluctuations, the coefficient is used to control
the CPU frequency (or computing power) of virtual machines. The proposed model
was compared with several base line approaches such as DVFS using real datasets
from CoMon project (PlanetLab). The results of experiments on VMDFS show that
besides the reduced service-level agreement violation by up to 43.22%, the overall
energy consumption is reduced by 40.16%. In addition, the overall runtime before
a host shutdown increased by 17.44% in average, while the runtime before a virtual
machine migration increased by 7.2%. This also shows an overall decrease in the
number of migrations.

Keywords Green cloud computing · Virtual machine frequency · Energy efficiency ·

Workload prediction · Self-adaptive systems

B Faramarz Safi-Esfahani
faramarz.safi@yahoo.com; fsafi@iaun.ac.ir
Extended author information available on the last page of the article

123
VMDFS: virtual machine dynamic frequency scaling framework… 5945

1 Introduction

Data centers are faced with high volumes of electric power consumption, which brings
huge operational costs to organizations. In [1], it is shown that only 10–50% of the
overall capacity of tested servers is used. Therefore, the main reason for the exten-
sive consumption of power and energy, and thus high costs of accessing information
resources, is not the expansion of computing resources in data centers, but ineffi-
cient utilization of such resources. In addition, user requests in the cloud environment
are responded by allocating virtual machines [2]. From an energy consumption per-
spective, overloaded or under-loaded servers are highly inefficient [3]. Virtualization
technology is the proposed solution to this problem. It allows every virtual machine
to define and allocate the required computing resources on behalf of a physical server.
The result is improved resource utilization and increased return on investment (ROI).
Different workload by different requesters still causes performance violation as well.
Achieving quality of service (QoS) together with assuring service-level agreement
(SLA), while energy consumption is at the minimum level, can be regarded as one of
the biggest challenges in cloud computing. CPUs capable of using dynamic voltage
frequency scaling (DVFS) will be able to decrease their consumed power by applying
different combinations of frequency and voltage [4]. DVFS dynamically scales CPU
frequency according to the overall CPU load, regardless of local workloads of virtual
machines. One solution is to predict future workload of servers so that quality features
of cloud-based data centers, e.g., energy consumption and service-level agreement
(SLA), are improved.
In [3], two methods that are based on CPU consumption threshold include median
absolute deviation (MAD) and interquartile range (IQR) that are used for detecting
overloaded hosts. These methods use historical data analysis for this purpose. A local
weight regression algorithm based on LOESS method (from the German löss short
for local regression) is also used to predict future CPU usage. The basis of this local
regression is the fitness of simple models as subsets of the main data. The steps in fact
form a proof for local regression (LR) and local regression robust (LRR) algorithms.
These two algorithms are responsible for predicting percentage of CPU usage within
next time periods and detecting the overloaded hosts. Their disadvantages include
imprecision of statistical analysis of previous CPU data in estimating the upper limit
and weakness in selecting the node with the lowest level of utilization as the under-
loaded node.
The focus on the dynamic scaling of CPU frequency and voltage in physical
machines (servers) is common in studies [5–11] and other research studies on energy
efficiency through controlling of computing power. However, the effect of changing
the features of virtual machines (like varying the CPU frequency of virtual machi-
nes—computing power) on the quality factors of cloud data centers has not been
examined in the above research studies.
Infrastructure as a service (IaaS) layer in cloud computing applies virtual machines
as virtual resources. However, enough attention has not been paid to the dynamic
scaling of the computing power of virtual machines. Previous research studies neglect
this fact that by monitoring the requests of users for resources and managing the
current and future workload of virtual machines, the management of energy in physical

123
5946 K. Shojaei et al.

machines would be possible. The novelty of this study is presenting a virtual machine
dynamic frequency system (VMDFS) framework along with algorithms to manage
current workload and predict future workload, and then estimating and setting the
CPU frequency of virtual machines, adaptively.
Accordingly, this research hypothesizes that (1) the variation of CPU frequency
of virtual machines (VMs) affects the power and energy consumption of physical
machines (PMs); (2) if we could detect future consumption pattern of virtual machines
by predicting workload applying time series approach, then we would be able to control
frequency or computing power changes in virtual machines, adaptively. It eventually
results in controlling the physical infrastructure and reducing energy consumption,
service-level agreement (SLA) violation, and the number of virtual machine migra-
tions. The enhanced exponential moving average or EEMA approach is also used to
control the fluctuation of workload in predicting the workload of virtual machines.
Controlling virtual machine’s frequency enables us to use a fraction of the computing
power of virtual machine instead of using their total computing power, entirely.
The main goal is to optimize the adaptability of a cloud-based data center as a
quality variable. In addition, contribution of this study can be highlighted as follows:
(1) presenting virtual machine dynamic frequency system (VMDFS) framework; (2)
preventing undesirable fluctuations of virtual resources (virtual machines) by present-
ing an algorithm to predict future workloads; (3) presenting an algorithm to control
the CPU frequency of virtual machines based on the prediction of future workloads,
adaptively.
The proposed model predicts the workload dynamically and then estimates the CPU
utilization of virtual machines. The prediction of future workload is difficult, when the
workload fluctuates at different time periods, stochastically. To gain more certainty
about the predicted workload, a smoothing coefficient is used in calculations. In fact,
the proposed model predicts the appropriate utilization level of virtual machines by
interacting with the properties of hosts. The results of experiments on VMDFS show
that in addition to a reduction in the violation of service-level agreement (SLA) for up
to 43.22%, the overall energy consumption, percentage of a host’s experiencing 100%
utilization, degradation of performance due to migration, number of migrations, ESV
(reduction of energy and SLA violation), and the number of shutdown hosts decreased
by 40.16, 21.77, 28.57, 27.80, 54.49, and 35.14%, respectively.
The remaining parts of the paper are organized as follows: basic concepts, study
domain, and a review of research literature are discussed in Sect. 2. Section 3 discusses
the architecture of the proposed model, provides a formal description of the proposed
model, and explains the model in detail. Model evaluation, setting up the testing
environment, evaluation metrics, and experiments are discussed in Sect. 4. Finally,
Sect. 5 closes the article with a final conclusion and future research implications.

2 Literature review and related works

In this section, the basic concepts that are used throughout the paper are first explained
and then previous studies on methods of reducing energy consumption in data centers
are discussed.

123
VMDFS: virtual machine dynamic frequency scaling framework… 5947

According to [12], the workload of cloud servers has the following features: (1)
Changes in the workload of hosts in the course of time are a stochastic process; (2)
there is a strong correlation between the workload of a host and time, which means that
past workload affects future workload; (3) in the course of a specific time period, the
workload either fluctuates or stays stable; and (4) based on the above, it is possible to
predict the servers’ workload at a specific time using basic models of time series. In this
study, first moving average (MA), exponential moving average (EMA), and enhanced
exponential moving average (EEMA) models were fitted. EEMA is an extension of
exponential moving average (EMA). At the end, it was shown that the best method
for predicting virtual machines’ workload is enhanced exponential moving average
(EEMA).

2.1 Moving average

Moving averages are forward and backward indices of observed data that regulate
the abnormal behavior of such observations. Simple moving average, smoothed mov-
ing average, and exponential moving average are different methods employed in this
model. Technical analyzers use exponential moving average generally to reduce the
delay in the curve of simple moving average. The smoothing technique dominating
these methods emphasizes the fact that the weight effect of observations decreases
exponentially as they get older. To predict a time series Y , which is recursive, EMA
uses the following:

S1  Y1 if t  1
(1)
St  α.Yt + (1 − α).St−1 if t > 1

where St is the predicted value at time t and St−1 is the predicted value at time t − 1
and Yt is the observed value at time t and α is the smoothing constant. In EMA, every
smoothed (predicted) value is the average weight of previous observations and the
previous smoothed (predicted) value. Weights are reduced exponentially relative to α.

2.2 Self-adaptive systems

A self-adaptive system evaluates its own behavior and changes them if they are not
in line with the predefined goals of the software, or if a higher level of performance
is possible. In other words, a self-adaptive system is a system that changes behavior
in response to changes in its operational environment. The manager of a self-adaptive
system consists of the following elements: monitoring, analysis, planning, execution
(MAPE), and knowledge. The monitoring element enables self-awareness and presents
the outside environment. The analysis element does the decision-making to adapt the
system to its goals. The planning and execution elements also undertake adaptive
operations in case the system deviates from its predefined goals. The operations by
these four elements are supported by knowledge [13].

123
5948 K. Shojaei et al.

Fig. 1 Mind map of this research

2.3 Related works

Figure 1 shows different methods of energy management, as well as the research

domain in this paper. Power management in two forms of hardware and software is
divided into two major categories, i.e., static power management (SPM) and dynamic
power management (DPM) [14]. The CPU consumes the most energy in servers. The
amount of the consumed energy depends on the type of that CPU. Inefficient energy
consumption in datacenters is not only due to a large number of computing resources
in large-scale datacenters, but also (and more importantly) because of inefficient use
of these resources [5]. The main objective in predicting CPU workload is to balance
the work load in servers based on the predicted CPU workload and minimize energy
consumption in hosts [6]. In energy-aware prediction techniques, workload is predicted
so that hosts can reduce the overall power consumption of the physical machine while
responding to the requests.
In [7], researchers have proposed a scheduling algorithm which is based on the
optimized level of consumption in hosts. By considering service quality, this study
has improved energy consumption. Whenever a host is at the optimum level of energy
consumption, this scheduling algorithm regulates the speed of task execution on virtual
machines allocated to physical resources of hosts. The main point highlighted in this
study is that the consumed energy, which is a result of the consumed power and time,
is specifically sensitive to the amount of virtual machine CPU consumption. The only
limitation considered for tasks in this scheduling method is the hard deadline, and
other limitations including workload at the moment of calculating the power are not
considered.

123
VMDFS: virtual machine dynamic frequency scaling framework… 5949

An attempt to reduce energy consumption in data centers through dynamic volt-

age and frequency scaling using a task-level scheduling algorithm for virtual machine
tasks can be seen in [8]. Here, a suitable scheduling for tasks and resources is pro-
posed based on the lowest and the highest consumed power by both hosts and virtual
machines. This algorithm uses the various levels defined for service-level agreement in
allocating resources to tasks. Since this study uses minimum and maximum computing
power (frequency) for virtual machines and hosts, it is unable to provide appropriate
calculations in case of different heterogeneous workloads. Only if the workload sent
to the cloud environment is uniformly dispersed, this method can be a good solution
for reducing energy consumption. In [9], a scheduling algorithm has been devel-
oped for virtual machines by means of which the cloud service provider can control
energy consumption. Based on the test results, an optimum frequency is considered
for each physical machine when allocating virtual machines to physical machines. The
researchers have shown that deadline, as a requirement in virtual machines, can be
effective in calculating energy and computing power. The ratio of performance to con-
sumed power in a physical machine is considered as the weight of that machine. Each
physical machine is then allocated to virtual machines based on its weight. Costs of vir-
tual machine migrations are not considered in this study; however, such costs directly
affect the increase in service-level agreement (SLA) violation. From the viewpoint of
cloud environment providers in [10], energy utilization dominates operational costs
ideas. Nevertheless, policies of reducing energy consumption can themselves violate
service-level agreement (SLA). Here, the focus is on improving energy utilization
in datacenters, while the rate of task completion is at its highest level. Optimization
is through choosing one part of the required CPU capacity in physical machines in
order to complete the tasks according to their lifetime. Mean decision-making between
errors of each node and the overload of virtual machine migration are also considered.
However, this study does not consider the case where there is a great workload on
virtual machines, or they are not used at all. Another thing not regarded in this study is
CPU voltage and frequency. This is notable since high workload nodes can interfere
with service-level agreement (SLA) if changes in CPU frequency are overlooked.
In order to reduce energy consumption in datacenters, some scholars have put their
efforts on the framework of workload prediction. In [11], a nonlinear regression-based
model is used for predicting power consumption in virtual machines. Linear models
cannot consider the interdependence between multiple parameters in overall workload
calculations, and thus, their predictions are not precise enough. This might be due to
the correlation between the effective measures in energy consumption. For example,
physical CPU usage greatly correlates with memory usage, and using linear models
not only reduces the precision, but also increases the complexity of predicting power
consumption. To avoid these, this study uses support vector regression (SVR). One
important feature of this study is monitoring usage of hardware properties of a host,
e.g., CPU, memory, network, and disk. However, it does not focus on virtual machines
behavior and power consumption as the main factors affecting resource allocation in
datacenters. Studies [15−17] have also pointed out the reducing of energy consump-
tion in hosts by considering the special correlation between hardware components of a
server and monitoring their energy consumption and using the models obtained from
predicting energy consumption in these components. A common feature of the above

123
5950 K. Shojaei et al.

studies is using linear multidimensional models for predicting power consumption in

hardware components effectively in reducing the overall power consumption of the
host. In [13], the focus is on the running sensitive-to-CPU activities in cloud-based
systems. The attempt is to find the best prediction method in selecting the suitable
power consumption pattern in datacenters. Using traditional time series models, i.e.,
autoregressive–moving-average (ARMA) and exponential smoothing (ES), for pre-
dicting power consumption is one the feature of this study. For new activities, they
have used segmentation and sliding window methods. It is shown in this study that ES
and segmentation outperform ARMA and sliding window, respectively. In a nutshell,
research studies [7–10] aim to optimize the power consumption of cloud of cloud
datacenters focusing on the features of virtual machines applying DVFS technique.
On the other hand, researchers in [11, 15, 16] improve the quality factors of cloud
datacenters by predicting workload.
A brief comparison of the latest related studies is shown in Table 1 in terms of
several characteristics such as adaptability with runtime environment, type of resource,
strategies and goals. In contrast to the previous related studies, this current research
proposes a framework that adapts the virtual machines’ features to the behavior of
predicted workload. Dynamically predicting future workloads, a virtual machine’s
computing power varies adaptively.

3 Proposed framework

The proposed framework in Fig. 2 as an adaptive approach with feedbacks from run-
time environment consists of five customized components of the well-known MAPE-K
[18] including monitoring, analysis, planning, execution, and knowledge components.
The five components are customized in order to vary the CPU frequency of virtual
machines based on the prediction of workloads that is a feedback from cloud runtime
environment. That is the reason why the framework is called virtual machine dynamic
frequency system (VMDFS). The monitoring element in this model is responsible
for collecting historical workload on the utilization of virtual machines in different
time intervals. In the analysis stage, the most appropriate consumption pattern is cho-
sen from workloads using enhanced exponential moving average (EEMA) prediction
method. In the planning phase, new plans are generated when the workload is predicted
according to the obtained pattern. In the execution stage, the frequency (computing
power) of virtual machines varies based the predictions. The whole information in
MAPE phases is collected in the knowledge component, afterward.
Figure 3 realizes VMDFS framework in terms of the literature of cloud comput-
ing. It includes a global manager and a local manager. The global manager collects
resource (virtual machines) information the local managers. Users’ requests are sub-
mitted to the global manager at runtime to be executed on cloud resources. The local
manager that is analogous to virtual machine monitor (VMM) includes several com-
ponents: (1) dynamic prediction interface, (2) EMA predictor, (3) VMDFS controller,
and (4) local resource manager. The main steps of VMDFS are as follows: (1) the
requests (workloads) are sent to the global manager (cloud broker) to receive comput-
ing resources (virtual machines), while they are stored in a repository to be used in the

123
 Table 1 A brief comparison of the latest related studies

Approach Adaptive Characteristics Resources Strategy Goal

[7] NO DVFS computing power CPU Regulating the speed of Reducing energy
(frequency) in PM executing tasks on VMs consumption Reducing
putting the host at an SLA violation
optimized level of energy
consumption
[8] NO Defining the minimum CPU Task scheduling algorithm Improving resource
and maximum based on DVFS utilization Reducing energy
computing power consumption and reducing
(frequency) in VM and SLA violation
host
[9] NO DVFS deadline limitation CPU, VM Extending the scheduling Reducing energy
in VMs Computing algorithm at the level of consumption and increasing
power (frequency) in virtual machine Using processing capacity
PM optimum frequency in PM,
defining weights based on
the ratio of power to
VMDFS: virtual machine dynamic frequency scaling framework…

performance in
heterogeneous PMs
[10] NO DVFS live migration CPU Task scheduling algorithm Reducing energy
Computing power based on weighted Round consumption
(frequency in PM) Robin
[11] NO S VR nonlinear regression CPU, Ram, Cache, Disk Predicting power Comparing the energy
model prediction consumption using a consumption prediction
regression-based nonlinear methods
model

123
5951
 Table 1 continued
5952

Approach Adaptive Characteristics Resources Strategy Goal

123
[12] NO Workload prediction CPU, Ram, HDD Using a linear model of
power consumption for
HPC workload and
integrating this model in
cluster monitoring systems
[22] NO “Workload prediction CPU Using ARMA and ES
methods for historical
workload and using
segmentation and sliding
window for non-historical
workload
VMDFS model (current YES DVFS, VM frequency CPU, VM Using EEMA method for
research) (computing power), predicting workload,
workload prediction smoothing predicted
average workload
fluctuations, virtual
machine frequency scaling
Optimizing energy
consumption based on
host’s multiple features
Finding the best prediction
method for choosing a
suitable power consumption
pattern
Reducing energy
consumption, SLA
violation, and the number
of migrations
K. Shojaei et al.
VMDFS: virtual machine dynamic frequency scaling framework… 5953

Fig. 2 Components of VMDFS based on MAPE-K [18]

prediction algorithms later; (2) dynamic prediction interface queries the monitoring
element to receive data necessary for prediction; (3) enhanced exponential moving
average (EEMA) prediction component smooths (fits and selects) the fluctuations of
workloads in the repository using the smoothing coefficient to be applied in the pre-
diction; (4) workload dynamic prediction interface predicts the CPU utilization of
virtual machines; (5) VMDFS controller that is installed on every node uses the result
of the workload prediction to make a right decision on VMs’ CPU utilization to adjust
VMs’ frequency (computing power), and VMs’ migration method; (6 and 7) VMDFS
controller simultaneously collects the feedbacks of varying VMs’ frequency to be con-
sidered in making decision in the next round of algorithm execution, adaptively, and
issues the commands to the local resource manager for varying the frequency of VMs,
and migrating VMs; (8) frequency tuner changes the frequency of virtual machines.
Figure 4 also illustrates the flow of information among the components of VMDFS
framework.

3.1 Formal description of the model

Table 2 introduces the parameters needed to describe VMDFS model. Each virtual
machine (VMi ) needs a part of a host’s computing power (P.mh i ). In order to have a
measure independent from the hosts’ computing power, the required computing power
VMi is considered as the number of instructions processed in 1 s. This is called MIPS
(million instructions per second).
Definition 1 In a cloud environment, a broker allocates input tasks to VMs based on
service-level agreement (SLA) requirements and scheduling policies. To respond to
Ti , VMi needs part of P.mh i , called the requested computing power (P.rvi ) in this
paper, and obtained from Eq. 2.

123
5954 K. Shojaei et al.

Fig. 3 Virtual machine dynamic frequency system (VMDFS) framework

P.r v i  U .vm i (t) × P.cv i (2)

where U .vm i (t) is a host’s CPU usage provided by vm i at time t for the requested
resources by Ti at time t, and:

123
VMDFS: virtual machine dynamic frequency scaling framework… 5955

Fig. 4 The steps of VMDFS framework


m
U .vm i (t)  U .w j (t) (3)
j1

where w j (t) is a jth workload for T j in time t.

Definition 2 The computing power allocated to VMi in order to respond to request Ti

is defined as follows:

P.r v i P.r v i ≤ P.ah i
P.av i  (4)
P.ah i o.w.

If the required processing power of VMi is less than or equal to the processing
power of the allocated host, then P.r v i calculated by Eq. 2 is considered as the power
of VMi .

Definition 3 Power consumption is generally studied for CPUs, memories, disk stor-
ages, power supplies, and cooling devices in servers available in datacenters. These
studies show that there is a strong linear relationship between power consumption and
CPU usage [3, 19]. h i usage for hosting Ti by VMi is:

P.ah i
U .h i (t)  (5)
P.mh i

that shows the usage of the host h i . In addition, p.ah i is analogous to the total pro-
cessing power of all virtual machines in the host. Therefore, Eq. 5 calculates the ratio
of p.ah i to the maximum processing power h i ( p.mh i) that is the usage of the host
hi .

123
5956 K. Shojaei et al.

Table 2 VMDFS symbol definitions

Symbol Definition

hi ith, host
P.h i Total power consumption of h i
P.mh i Maximum computing power (MIPS) of h i
P.ah i Allocate computing power (MIPS) of h i
U .h i (t) Utilization of host in time t
VMi ith, VM
P.cv i Maximum computing power (MIPS) of VMi
P.av i Allocate computing power (MIPS) of VMi
P.r v i Requested computing power (MIPS) of VMi
Ppredict .cv i (t) Predicted percentage of maximum computing
power of VMi in time t
U .vm i (t) Utilization of vmi from T i in time t
Ti ith, task
U .wi (t) Utilization of T i based on the workload in time t
w j (t) jth, workload for T j in time t
T i 100% .h i Total time during which the host ith has
experienced the utilization of 100%
T i active .h i Total time of the host ith begin active state (serving
VMs)
Cmigrate .v i Estimated cost of performance degradation of VMi

The assumption is that idle servers immediately change status to shutdown mode
[7]. Therefore, the overall consumed power in h i is defined as:
 U .h i >0
P static + P dynamic
P.h i  (6)
0 o.w.

where P static is power consumption of idle server and is defined as P static  α P max .
P max is power consumption in h i when at maximum usage and α is the ratio of static
power to maximum power (0 < α < 1), which depends on server’s physical features.
According to the linear relationship between power consumption and CPU usage, as
mentioned by Fan et al. [19], the linear model of power can be defined as follows:

 
P.h i  α Pimax + Pimax − α Pimax . U .h i (t) (7)

3.2 VMDFS model

According to [12], it is possible to predict servers’ workload at a specific time using

time series-based models. After studying time series models, enhanced exponential
moving average (EEMA) was selected for predicting workload on virtual machines.

123
VMDFS: virtual machine dynamic frequency scaling framework… 5957

The strategy employed in VMDFS consists of two phases: dynamic prediction and
VMDFS controller.

3.2.1 Phase-1: dynamic prediction

This phase that is the core of the recommended VMDFS framework receives a variety
of requests from users. After calculating the sparsity of requests and their effect on
prospective requests, the workload of each virtual machine is predicted. Therefore, the
computing power of virtual machines is tuned in a controlled fashion. Phase-1 includes
the following stages: (1) Fitting and selection: by observing current and past workload,
a pattern is extracted that can be used in the next stage to predict future workload. In
general, users’ needs for computing resources fluctuate in different time intervals. In
this study, exponential moving average (EMA) method is enhanced to deal with work-
load fluctuations that provides a dynamic pattern to predict users’ maximum required
workload in a virtual machine. This new approach is called enhanced exponential mov-
ing average (EEMA). (2) Workload prediction: the best selected prediction pattern is
used for predicting future workloads. For each time interval, maximum workload is
predicted according to the observed data in the previous time interval.

Algorithm 1: Dynamic Prediction with EEMA method

1 Input: CSV workload file Output: CSV Predicted workload file
2 data = read.csv (CSVworkloadfile)
3 Lambdas = CreateSequenceSmootingFactor(0.01, 0.9, 0.01)
4 For i=1 to length(Lambdas)
5 PrimaryPrediction = EMAmethod(data, Lambdas[i])
6 For VMs in vmList
7 PredictionWorkload = data – PrimaryPrediction
8 PredictionWorkload = PredictionWorkload [>=0]
9 PILs.add(quantile(PredictionWorkload, 0,025, 0.975))
10 EndFor
11 EndFor
12 APILs = Mean (PILs)
13 return APILs

Algorithm 1 along with the flowchart in Fig. 5 encompasses the details of EEMA to
predict the workload of all virtual machines. Moving average (MA) is a constant weight
that is used in prediction and calculated based on previous observations, while EMA
uses a specific constant weight assigned to each observation. On the other hand, the
prediction in EMA neglects previous data in prediction, while EEMA looks after the
data in specific time intervals and applies several threshold parameters. The smoothing
parameter Lambda is the value assigned to previous observations to predict future
values. It is used to avoid using constant weights for previous values and provides
new values, adaptively. It is initialized by 0.01 and is incremented by 0.01 until it
becomes equal to 0.9 as shown in line 3. Therefore, EEMA works on the collection
of historical workload for different smoothing coefficients (line 5). EEMA considers
the distance between upper and lower limits as the prediction interval. The difference
between predictions and observations is considered as prediction intervals (line 7).
From among the calculated values, positive ones are chosen (line 8). In order to obtain
the confidence interval, prediction interval lengths (PILs) are estimated based on the
percentiles 0.025 and 0.975. These percentiles determine the percentage of predictions

123
5958 K. Shojaei et al.

Fig. 5 The steps of workload prediction based on enhanced exponential moving average (EEMA) method

to be inside an interval, so that the interval can be considered the confidence interval
(line 9). This confidence interval is called the precision coefficient of EEMA. The
algorithm continues by calculating the average predicted lengths for each smoothing
coefficient lambda and for each virtual machine (line 12). This value is considered as
the average prediction interval lengths (APIL) for virtual machines and is then added
to the predicted workload as the smoothing coefficient.
Figure 6 shows the results of the above prediction method. The workload allocated to
virtual machines is shown as continuous and predicted workload for virtual machines
applying EEMA illustrated as dotted lines. According to the results, 96.91% of the
predictions are in the 95% of the precision interval.

123
VMDFS: virtual machine dynamic frequency scaling framework… 5959

Fig. 6 Real versus predicted workload

3.2.2 Phase-2: VMDFS controller

The output of dynamic prediction in Phase-1 includes the predicted percentage of

computing power required by each virtual machine in a specific time interval. Each
virtual machine has to provide such a computing power to be able to respond to the
workload in that specific time interval. The prediction results in Phase-1 as output are
given to Phase-2 as input. According to Eqs. 2–7, it is clear that there is a significant
relationship between the predefined frequency (computing power) of every virtual
machine and the consumption power of a host. Therefore, controlling the frequency
of each virtual machine affects the consumption power of the host. The output of
Phase-1 is the required computing power of each virtual machine in a specific time
interval. The predicted value for each virtual machine could hence be used instead of
the maximum power consumption of a virtual machine. Accordingly, instead of using
the maximum processing power for each virtual machine P.cvi in Eq. 2, the predicted
processing power Ppredict .cv i (t) is used.

Algorithm 2: Get Current Requested MIPS

1 Input: vmList; Output: VMs Current Requested MIPS
2 for each vm in vmList do
3 vmPredictedMips = getTotalPredictionUtilizationOfCPU(getCurrentIntervalTime(),vm)
4 vmRequestedPredectedMips = vm.getTotalMips() * vmPredictedMips
5 If vmRequestedPredectedMips > vm.getTotalMips()
vmRequestedMips = getTotalUtilizationOfCpu (getCurrentIntervalTime ()) *
6
vm.getTotalMips()
7 else
vmRequestedMips = getTotalUtilizationOfCpu(getCurrentIntervalTime ()) *
8
vmRequestedPredectedMips
currentRequestedMips = vmRequestedMips / getNumberOfPes ()
9
vmsCurrentRequestedMips.add(currentRequestedMips)
10 return vmsCurrentRequestedMips

The following is the basic process in Phase-2 of VMDFS model:

P.r v i  U .vm i (t) × (P.cv i × Ppredict .cv i (t)) (8)

To calculate P.r v i , the effect of Ppredict .cv i (t) on P.cv i for each virtual machine
is considered. Algorithm-2 shows the related process. If P.r v i , as obtained by (8), is
greater than the overall frequency of the virtual machine, the virtual machine’s pre-
defined maximum frequency will be used. This indicates that service-level agreement
(SLA) violation or VM migration is possible if the virtual machine is overloaded.

123
5960 K. Shojaei et al.

Fig. 7 The testing environment

When P.r v i is smaller than or equal to the overall VM frequency, only one portion of
the maximum VM frequency is used in calculating the requested consumption power.

4 Evaluations and testing environment setup

VMDFS framework is implemented by CloudSim simulation tool [20] that lets

researchers handle bottlenecks before deploying on real clouds. It also provides us
the possibility of doing experiments: (1) in a repeatable and controllable environment
in a controllable way; (2) with different standard workloads; and (3) in a time effec-
tive manner. In other words, it is not possible to perform experiments in real cloud
in a repeatable, dispensable, and scalable fashion. In all of the experiments, part of
the workload in CoMon project—a monitoring infrastructure for PlanetLab—is used
[21]. To analyze the workload and find the best model for workload prediction, R
open-source software is used [22]. Figure 7 shows the configuration and specification
of the testing environment and the tools used to realize VMDFS model. A case study
of VMDFS framework is presented in “Appendix”, as well.

123
VMDFS: virtual machine dynamic frequency scaling framework… 5961

Table 3 Workload specifications

Workloads Number of Mean (%) St.dev. (%) Quartile 1 Median (%) Quartile 3
VMs (%) (%)

Workload 1 1052 12.1 17.1 2 6 15

Workload 2 1078 10.56 14.4 2 6 14
Workload 3 1033 10.43 15.21 2 4 12

The target system is an IaaS environment provided by a large-scale datacenter

consisting of 800 heterogeneous physical nodes. Every physical node has a dual-
core CPU. CPU frequency of every server is considered in terms of mega instruction
per second (MIPS). Half of the servers are HP ProLiant ML110 G4 with 1860 MIPS
computing power, and the other half is HP ProLiant MA110 G5 with 2660 MIPS com-
puting power for each CPU core. Virtual machine specifications are similar to Amazon
EC2 instances, including: High-CPU Medium Instance (2500 MIPS, 0.85 GB), Extra
Large Instance (2000 MIPS, 3.75 GB), Small Instance (1000 MIPS, 1.7 GB), and
Micro Instance (500 MIPS, 613 MB) [21].
The data used in the experiments consist of workload data collected from CPUs of
more than 1000 virtual machines on the available servers at 500 different places around
the world [23]. The specifications of each of the selected workload collections for the
experiments are shown in Table 3. In addition, the time interval of measuring CPU
utilization for the workloads is 5 min. The results of investigations in [1] also show that
the average VM CPU utilization in datacenters is lower than 50%. Table 3 shows the
processor usage fluctuations of virtual machines in three workloads in terms of several
statistical parameters including: the average of resource usage (Mean), the sparsity of
using resources in virtual machines (standard deviation), median of top quartile of
data (Quartile 1), the middle point of resource usage (Median), and the median of
the last quartile of data (Quartile 3). The collected workloads for three different days
are randomly allocated to virtual machines. Also, this study uses minimum migration
time (MMT), in which the virtual machine with the lowest required migration time is
chosen for migration.

4.1 Measured metrics

According to [3], service-level agreement (SLA) violation metrics are divided into
two categories: (1) the percentage of time in which the active host has experienced
100% CPU utilization (the duration of a node being overloaded), which is specified
by SLATAH (service-level agreement time per active host) in experiments and (2)
reduced performance because of VM migrations, which is shown by PDM (perfor-
mance degradation due to migration) that are calculated as follows:

1  T i 100% .h i 1  Cmigrate .v j
n m
SLATAH  PDM  (9)
N T i active .h i M P.r v j
i1 j1

123
5962 K. Shojaei et al.

where N is the number of hosts; T i 100% .h i is the duration of h i experiencing 100%

utilization, resulting in service-level agreement (SLA) violation; T i active .h i is the dura-
tion of h i being active; M is the number of virtual machines; Cmigrate .v j is the estimate
of reduced performance due to vm j migration, which is equal to 10% of CPU utiliza-
tion throughout all vm j migrations; and P.r v j is the computing power requested by
vm j while being active.
Hence, SLAV is a measure of final service-level agreement violation (SLAV) and
is obtained via (10):

SLAV  SLATAH × PDM (10)

Another metric for estimating service-level agreement (SLA) violation, as men-

tioned in [24], is the percentage of resources that are requested but not allocated due
to lack of resources. This metric is defined in Eq. 11:
 M  N   M  N 
i1 j1 P.r v j − i1 j1 P.av j
 M  N 
i i
Overall SLA violation  (11)
i1 j1 P.r v j
i

The Eq. 11 gives the amount of requested but not allocated computing power in the
datacenter. In [3], ESV (energy and SLA violation) metric is used for describing the
balance between energy and performance, which is calculated through (12):

ESV  E × SLAV (12)

4.2 Experiments

Three categories of experiments under different conditions are conducted as follows:

(1) boundary conditions experiment; (2) experiments to choose the best prediction
method in Phase-1 of VMDFS model; and (3) experiments to show the superiority of
VMDFS model over the base research [3]. The experiments begin with an explanation
of boundary conditions experiment for the first hypothesis and then continue with
discussing the experiments that result in choosing the enhanced exponential moving
average (EEMA) prediction method in Phase-1 of VMDFS model. Finally, several
experiments are conducted on the results of enhanced exponential moving average
(EEMA) method to predict workloads in two sections.

4.2.1 Boundary conditions experiment

The boundary conditions experiment is conducted to show the effect of varying the
frequency (computing power) of a virtual machine on power and energy consumption.
The goal is to support the first hypothesis that argues that if the CPU frequency
(computing power) of virtual machines resided in a host is decreased/increased, the
energy consumption of the host will also change. Table 4 shows the characteristics of
the experimental environment for boundary conditions experiment.

123
VMDFS: virtual machine dynamic frequency scaling framework… 5963

Table 4 The experimental environment for boundary conditions experiment

MaxPower MIPS RAM host Storage BW

memory

HOST:
250 W 2500 1000 MB 1,000,000 300,000
Frequencies available for
physical CPU
59.925 69.93 79.89 89.89 100
MIPS PES number RAM Size BW

VMs:
1: {50,75,150,80} 1 128 2500 2500
2: {60,75,95,110}
3: {50,105,120,85}
4: {50,70,110,115}
Length PES Number File size Output size

Cloudlet:
{5000,4486,299,2315} 1 300 300

This experiment uses a host with the properties mentioned in Table 4. Five physical
CPU frequencies are available for this host in CloudSim. At runtime, if CPU utilization
is less than the predefined frequency, CPU frequency is adjusted to a lower value.
Otherwise, if CPU utilization is more than the predefined frequency, CPU frequency is
adjusted to a higher value. In cases where utilization reaches the minimum or maximum
predefined frequency thresholds, the CPU is regarded as overloaded or under-loaded.
The boundary condition experiment continues by applying four virtual machines with
various frequencies. The frequency of virtual machines varies according to the basic
distributions exponential, polynomial, and moving average that are illustrated in three
configurations in Fig. 8. Only the boundary condition experiment is done with limited
number of virtual machines, while other experiments will be performed by applying
three real workloads that include a large number of virtual machines. The cloudlets
according to the workloads are submitted to the host, and the simulation is repeated
four times. Power, power average consumption, and energy consumption are calculated
separately for each simulation. Table 5 shows that varying the frequency of virtual
machines results in changes in the power and energy consumption of the physical
host. The results of boundary conditions experiment applying four virtual machines
are shown in Fig. 9, while the numbers are normalized by multiplying the power sum
and power average with 0.0001 and 10 constants in order to be illustratable in one
figure.

4.2.2 Experiments to choose the best prediction method in Phase-1 of VMDFS

Table 6 shows the results of MA, EMA, and EEMA prediction methods on the work-
loads presented in Table 3. The results show that the prediction interval average length

123
5964 K. Shojaei et al.

Fig. 8 Configuration of virtual machine frequencies in boundary conditions experiment. a Config-1: expo-
nential trend line, b Config-2: polynomial trend line, c Config-3: moving average trend line

Table 5 Power, power average consumption, and energy consumption

Power sum (W) Power average (W) Energy consumption

(Wh)

1: mips  {40,65,140,65} 22,862.13 0.9085 6.3506

2: mips  {40,80,150,80} 21,245.82 0.9158 5.9016
3: mips  {40,85,140,85} 20,846.27 0.9179 5.7906
4: mips  {50,75,150,75} 19,748.43 0.9286 5.4857

Fig. 9 The impact of varying virtual machine frequency in boundary conditions experiment

(PIL) in EEMA prediction method is smaller than both EMA and MA prediction
methods. Therefore, variation of predicted and observed values (predicted values dis-
persion) in EEMA prediction method will be less than the other two methods.

123
VMDFS: virtual machine dynamic frequency scaling framework… 5965

Table 6 Prediction interval average length (PIL) in MA, EMA, and EEMA

Moving average (MA Exponential moving Enhanced exponential

%) average (EMA %) moving average (EEMA
%)

Workload 1 21.893 22.347 19.742

Workload 2 21.613 21.938 18.734
Workload 3 27.837 29.081 26.75

4.2.3 Experiments to show the superiority of VMDFS model over the base model

Experimental results evaluate VMDFS framework versus four most widely used meth-
ods for detecting overloaded hosts that are provided in [3]. Several experiments on
enhanced exponential moving average (EEMA) prediction method are conducted in
two separate sections. Each section has its own specific experiments for the workloads
presented in Table 3, and characteristics of the experiments are shown in Table 7. The
main objective of the experiments is to show the adaptability of virtual machines’
computing power with workload changes, where workload changes are predicted in a
dynamic manner.

4.2.3.1 The first part of experiments The first part of experiments studies the effects
of changes in workload (changes in CPU utilization of virtual machine) on the number
of VM migrations, the number of hosts that shut down, and also the evaluation metrics
discussed in Sect. 4.1. According to Fig. 10, VMDFS improves the methods proposed
in [3]—IQR, MAD, LRR, and LR algorithms. If a host is detected as overloaded, some
of its virtual machines must be migrated to an appropriate node based on the VM selec-
tion policy. Overloaded hosts are detected through IQR (interquartile range), MAD
(median absolute deviation), LR (local regression), and LRR (local regression robust)
algorithms [3]. To find virtual machines for migration, MMT (minimum migration
time), MM (minimization of migration), MC (maximum correlation), and RC (ran-
dom choice) algorithms are used [3,24, 25]. One disadvantage of IQR and MAD is
that for a dataset with low variability, MAD and IQR values are small. As a result,
the threshold is chosen to be close to 100%, which results in service-level agreement
(SLA) violation. In other words, MAD and IQR are not precise enough in estimating
the upper threshold for hosts. On the other hand, LR is influenced by outlier data and
may produce imprecise results.
Compared to MMT, MM, MC, and RC, the proposed VMDFS framework applying
enhanced exponential moving average (EEMA) prediction model is able to solve
the outlier and low-variety problems in the workloads that improves the results.
The smoothing coefficient in the dynamic predictions of future workload of virtual
machines along with applying the prediction technique enhanced exponential moving
average (EEMA) is the main reason of improving the results.
According to Eqs. 2–7, there is a right relationship between the predefined fre-
quency (computing power) of each virtual machine and the consumption power of
a physical host. Dynamic prediction of workload in the consequent time intervals

123
 5966

123
Table 7 Characteristics of VMDFS experiments

Part Experiment no. Workloads Host overloading VM selection policy for Experiments evaluation
detection migration metrics

Part 1 1 Workload 1 IQR, MAD, LR, LRR Minimum migration time PDM, SLATAH, SLAV,
(MMT) number of VMs
2 Workload 2 migration, energy
3 Workload 3 consumption, overall
SLA violation, ESV,
number of HOST
5 Workload 2
Shutdowns
Part 2 4 Workload 1 IQR, MAD, LR, LRR Minimum migration time Mean & StDev time
(MMT) before a host shutdown,
mean & StDev time
6 Workload 3 before a VM migration
K. Shojaei et al.
VMDFS: virtual machine dynamic frequency scaling framework… 5967

Fig. 10 According to the research article [3], system level agreements are divided into two sections: (1)
performance degradation due to migration (PDM); (2) the percent of time that active hosts experience 100%
CPU usage (SLATH) according to Eq. 9; (3) SLAV that the multiplication of PDM and SLATH according
to Eq. 10; (4) the amount of requested processing power in datacenter that is not provided according to
Eq. 11; (5) Overall SLA violation [24]; (6) ESV that is the product of the amount of energy consumption
and SLAV according to Eq. 12. Based on the mentioned parameters, the effectiveness of VMDFS versus
the algorithms in article [3] is studied. a PDM, b SLATAH, c SLAV, d number of VM migrations, e energy
consumption, f overall SLA violation, g number of host shutdowns, h ESV

enables VMDFS controller to use a tuned computing power instead of applying a

maximum computing power of each virtual machine, Eq. 8 in Sect. 3.2.2. As a result,
hosts will be less overloaded, and under-loaded hosts will be shut down before use,
which in turn will result in decreasing the number of hosts that experience 100% CPU
utilization. Consequently, both PDM (performance degradation due to migration) and
number of virtual machine migrations decrease, which is shown in Fig. 10a, d, respec-
tively. Equation 8 that is the output of the VMDFS framework is another reason for
decreasing the number of virtual machines and decreasing PDM, as well. Equation 8
illustrates that applying required computing power instead of the maximum comput-
ing power for each virtual machine prohibits hyperactive hosts according to Fig. 10a,
b.
The proposed framework has been able to reduce the number of overloaded hosts
in IQR and MAD algorithms, since unlike these two algorithms it handles outlier or
low-utilization workloads according to Fig. 10b. It is also shown that the proposed
framework is able to decrease the number of virtual machine migrations. Therefore,
SLAV—obtained from PDM and SLATAH in Eq. 8—is decreased (even for IQR and
MAD algorithms, Fig. 10c).
Figure 10f indicates that the proposed framework reduces the required computing
power of the hosts (overall SLA violation in Eq. 11). In fact, the dynamic work-
load prediction of virtual machines (virtual machines’ computing power) enables the

123
5968 K. Shojaei et al.

VMDFS controller to detect the unnecessary percentage of computing power in a

virtual machine, and this unused consumption power of CPU is free to be used in its
future use. Therefore, the processors of the hosts are able to allocate resources more
accurately.
According to [3], an overloaded host is detected by comparing the node with the
lowest computing power with other nodes. Generally, this policy increases not only
the number of migrations, but also the utilization of other active hosts by migrating
virtual machines from the overloaded node to other hosts. This policy is indeed not
suitable for the environments with a dynamic workload, since it extends the workload
among nodes. Compared to the minimum CPU utilization policy in [3], the proposed
framework makes the best decision for decreasing or increasing the frequency of vir-
tual machines by monitoring the workload of a host and predicting its future workload
(using the smoothing coefficient in enhanced exponential moving average method
(EEMA) method in Algorithm 1), which results in the best choice in detecting over-
loaded and under-loaded hosts. Using the required computing power instead of the
maximum one in virtual machines (according Algorithm 2 and Eq. 8) reduces the
number of migrations as well as energy consumption of hosts (Fig. 10e).
Based on the prior analyses, VMDFS framework uses the required computing power
of virtual machines. Applying VMDFS and the linear dependence between the power
consumption and the amount of CPU usage in Eq. 7 results in reducing the energy
consumption of hosts. According to Eq. 12 in Sect. 4.1, Fig. 10g shows the relationship
between energy consumption and SLAV metric, denoted as ESV. The decrease in
SLAV, and also in energy consumption (shown in Fig. 10c, e, respectively), shows that
ESV produces better results in all algorithms than the previous polices.
Finally, by appropriately controlling the workload on virtual machines, and thus,
decreasing the number of overloaded hosts and detecting the under-loaded ones, the
proposed model reduces the number of shutdown hosts (Fig. 10h). Figure 10d shows
the impact of the proposed framework on reducing the number of VM migrations.
This can be another reason for the reduced number of shutdown VMs. The overall
results of this part of the experiments are shown in Table 8.

4.2.3.2 The second part of experiments In the second part of experiments, the influ-
ence of the workloads in Table 3 on the runtime metrics before a host shutdown and
before a VM migration is studied. Distribution parameters such as mean and standard
deviation are considered as well. Figure 11 illustrates the effect of VMDFS framework
on the metrics of the second phase of the experiments in Table 7. The recommended
VMDFS framework controls the energy consumption of hosts so that they cannot be
hyperactive or less active, which results in not only the reduction of the number of
migrations but also improving the sleeping time of hosts. In addition, Fig. 11 shows
17.44% average increase in total time before a host shutdown. The decreased number
of shutdown hosts (as is shown in the first part of experiments, Fig. 10h) can be a
reason for the relative increase in the average total time before hosts’ shutdown. The
same behavior is seen in the standard deviation of the total time before a host shut-
down due to increase in the shutdown time of hosts (Fig. 11b). According to Fig. 11c,
d, mean and standard deviation before a virtual machine’s migration are increased in
LRR, LR, MAD, and IQR algorithms. On average, this increase is 7.2 and 8.95% for

123
VMDFS: virtual machine dynamic frequency scaling framework… 5969

Table 8 Results of the first part of experiments on VMDFS

Part Experiments PDM (%) SLATAH (%) SLAV (%) Number of

number VMs migration
(%)

Part 1 1 + 28.57 + 2.40 + 26.28 + 27.80

2 + 28.57 + 8.40 + 24.80 + 27.38
3 + 25.00 + 21.77 + 43.23 + 23.07
Part Experiments Energy Overall SLA ESV (%) Number of
number consumption violation (%) HOSTs
(%) shutdown (%)

Part 1 1 + 37.44 + 50.00 + 53.63 + 33.48

2 + 40.16 + 56.25 + 54.49 + 36.14
3 + 35.06 + 79.41 + 50.11 + 31.14

Fig. 11 The effect of VMDFS framework on algorithms in [3]. a Mean time before a host shutdown, b StDev
time before a host shutdown, c mean time before a VM migration, d StDev time before a VM migration

mean and standard deviation before a virtual machine’s migration, respectively. This
increase is due to decreasing the number of VM migrations, as well as decreasing the
number of overloaded hosts, as shown in Fig. 10d, h, respectively. The overall results
of this phase of experiments are shown in Table 9.

123
5970 K. Shojaei et al.

Table 9 Results of the second phase of experiments

Part Experiments Mean time StDev time Mean time StDev time
number before a host before a host before a VM before a VM
shutdown (%) shutdown (%) migration (%) migration (%)

Part 2 1 − 15.26 + 17.81 − 7.20 − 6.62

2 − 17.60 + 18.10 − 5.20 − 8.95
3 − 17.44 + 14.58 − 6.90 − 5.74

Table 10 VMDFS versus DVFS; average of obtained results on three workloads

PDM (%) SLATAH (%) SLAV (%) Number of VMs

migration (%)

+ 27.38 + 10.9 + 31.44 + 26.08

Energy consumption Overall SLA violation ESV (%) Number of HOSTs
(%) (%) shutdown (%)

+ 37.55 + 61.88 + 52.74 + 33.58

4.3 Experimental discussions

According to the experiments, the prediction of workload as well as controlling the

computing power of virtual machines reduces the energy consumption of cloud data
centers. The recommended VMDFS framework is able to control the power consump-
tion of virtual machines by predicting the fluctuations of workloads. The experiments
were organized in three phases and the obtained results supported the hypotheses.
The boundary condition experiment as the first phase of the experiments showed that
varying the CPU frequency of virtual machines affects the energy consumption of a
physical host. The second phase of the experiments illustrated that enhanced exponen-
tial moving average (EEMA) showed better results among several workload prediction
methods. The third phase showed that the recommended VMDFS algorithm outper-
forms DVFS as the base line approach in terms of several parameters including PDM,
SLATAH, SLAV, number of VM migrations, energy consumption, overall SLA viola-
tion, number of host shutdowns, and ESV. Table 10 illustrates applying the presented
idea on three real datasets from PlanetLab project in average.

5 Conclusion and future work

Cloud providers have integrated modern energy efficient strategies with management
infrastructures of virtual machines and physical hosts. This research paper proposed
VMDFS (virtual machine dynamic frequency system) framework that includes sev-
eral power-aware scheduling algorithms with the ability to adapt the CPU frequency

123
VMDFS: virtual machine dynamic frequency scaling framework… 5971

of virtual machines to runtime environment circumstances. A new approach is pre-

sented that predicts future workloads based on the history of workload and varies
the CPU frequency (computing power) of virtual machines, adaptively. VMDFS
framework innovated a smoothing parameter in its prediction model to control work-
load fluctuations. Energy consumption in a datacenter is reduced by estimating the
required computing power coefficient for each virtual machine. The experimental
results showed that VMDFS competes with other energy management policies in
cloud-based datacenters. The results supported that in addition to reducing viola-
tion of service-level agreement (SLA violation) for up to 43.22%, the overall energy
consumption, percentage of a host’s experiencing 100% utilization, degradation of
performance due to migration, number of migrations, ESV (reduced of Energy and
SLA violation), and the number of shutdown hosts decreased by 40.16, 21.77, 28.57,
27.80, 54.49, and 35.14%, respectively.
In addition, based on the experiences of this current research, several future works
are recommended as follows: (1) using modern approaches to estimate the confidence
coefficient in the prediction technique; (2) using new models for time series fluctuations
such as Garch and Heavy-tailed models; (3) extending the proposed model by including
multiple parameters, such as memory, bandwidth, and storage; and (4) applying novel
intelligent techniques such as deep learning in predicting workload fluctuations.

Appendix: Case study

In order to show how the proposed framework works, a practical case study is presented
for the runtime environment described in Table 11. The assumptions in this example
are as follows: there exist six hosts, ten virtual machines, and a 1-h period of workload
from PlanetLab. Table 12 shows CPU utilization in these ten virtual machines during
5-min intervals in a 1-h period. In time interval 1, the maximum computing power
(frequency) of a virtual machine is used for calculating the energy. In the consequent
intervals, VM CPU utilizations are estimated using enhanced exponential moving
average (EEMA) prediction model. These predictions affect the computing power
of each virtual machine. Table 13 shows the enhanced exponential moving average
(EEMA) prediction values for the time intervals shown in Table 12.
Power and energy calculations are performed in the first 5 mins (Time 300.1) based
on the maximum VM utilization. Before entering the second simulation interval, under-
loaded and overloaded hosts are determined. Virtual machines are then migrated from
over/under-loaded candidate hosts to appropriate target hosts using the algorithms for
selecting the best VM for migration. Consumed power and energy are calculated in
the second interval (time: 600.1). These calculations consider the effect of predicted
power consumption, as given by enhanced exponential moving average (EEMA).
Table 14 shows the calculated parameters of VMDFS evaluation metrics in the first
three intervals of the 1-h simulation period in CloudSim.

123
5972 K. Shojaei et al.

Table 11 Properties of VMs and

MIPS PEs RAM BW
hosts in the case study
VM1 2500 1 870 100,000
(100
Mbit/s)
VM2 2000 1 1740 100,000
(100
Mbit/s)
VM3 1000 1 1740 100,000
(100
Mbit/s)
VM4 500 1 613 100,000
(100
Mbit/s)
HOST1 1860 2 4096 1,000,000
(1 Gbit/s)
HOST2 2660 2 4096 1,000,000
(1 Gbit/s)

Table 12 Utilization of 10 virtual machines in 5-min intervals in a 1-h period

Mips VM0 VM1 VM2 VM3 VM4 VM5 VM6 VM7 VM8 VM9
2500 2500 2500 2000 2000 2000 1000 1000 1000 500

Time interval
1 24 23 47 14 30 4 0 7 4 11
2 34 55 4 16 31 0 2 5 2 17
3 29 19 38 14 25 2 2 6 2 13
4 26 48 26 19 26 2 0 4 13 15
5 26 35 31 15 41 4 2 4 2 15
6 21 18 29 17 18 2 1 6 0 15
7 18 17 15 15 21 4 3 7 2 17
8 25 38 37 22 24 6 2 7 2 17
9 25 14 34 19 21 6 0 7 7 18
10 20 39 3 14 26 2 2 5 8 19
11 12 13 26 22 9 2 4 3 2 11
12 40 36 27 18 27 4 5 5 8 15

123
 Table 13 Utilization prediction for 10 virtual machines using enhanced exponential moving average (EEMA) method

VM0 VM1 VM2 VM3 VM4 VM5 VM6 VM7 VM8 VM9

– – – – – – – – – –
43.74187 42.74187 66.74187 33.74187 49.74187 23.74187 19.74187 26.74187 23.74187 30.74187
52.74187 71.54187 28.04187 35.54187 50.64187 20.14187 21.54187 24.94187 21.94187 36.14187
49.14187 42.02187 54.77187 33.92187 45.33187 21.58187 21.72187 25.66187 21.76187 33.08187
46.08187 65.16987 46.64487 38.25987 45.70087 21.72587 19.93987 23.93387 31.64387 34.57587
45.77587 55.78467 50.33217 35.09367 59.23777 23.54027 21.56167 23.76107 22.73207 34.72527
41.24527 39.54615 48.9009 36.57705 39.89146 21.92171 20.82385 25.54379 20.04089 34.74021
VMDFS: virtual machine dynamic frequency scaling framework…

38.09221 37.0223 36.15778 34.92539 40.65683 23.55986 22.55007 26.62206 21.57177 36.54171
44.07691 55.66992 54.68346 41.06022 43.43337 25.52367 21.82269 26.72989 21.72486 36.72186
44.67538 35.93468 53.83603 38.97371 41.01102 25.72005 19.94995 26.74067 26.24017 37.63987
40.23522 56.46115 25.85129 34.26506 45.26879 22.13969 21.56268 24.94175 27.5917 38.63167
32.59121 35.1138 43.75281 40.99419 30.39456 21.78165 23.52395 22.96186 22.32686 31.53085

123
5973
 Table 14 VMDFS results and evaluation metrics
5974

Time: 300.1
(Host# To VM#)

123
Host0 6 9 Host1 0 3 Host2 7 Host3 1 4 Host4 8 Host5 2 5

VM0 (2500) VM1 (2500) VM2 (2500) VM3 (2500) VM4 (2500) VM5 (2500) VM6 (1000) VM7 (1000) VM8 (1000) VM9 (500)

Utilization 0.34 0.55 0.04 0.16 0.31 0.00 0.02 0.05 0.02 0.31
utilizationPredicted 0.44 0.43 0.67 0.34 0.50 0.24 0.20 0.27 0.24 0.17
utilizationPredicted * mips 1093.62 1068.79 1668.22 674.85 994.84 474.81 197.42 267.41 237.41 153.72
totalMips 849.96 1374.70 100.28 319.99 308.38 0.01 20.00 50.00 20.00 84.99
totalMipsWithPredic 371.81 587.70 66.92 107.97 308.38 0.00 3.95 13.37 4.75 26.13
totalRequestedMips 371.81 587.70 66.92 107.97 308.38 0.00 3.95 13.37 4.75 26.13
totalAllocatedMips 371.81 587.70 66.92 107.97 308.38 0.00 3.95 13.37 4.75 26.13

HOSTs Power Model:

1. PowerModelSpecPowerHpProLiantMl110G5Xeon3075
2. PowerModelSpecPowerHpProLiantMl110G5Xeon3075

Host #0 Host #1 Host #2 Host #3 Host #4 Host #5

hostTotalAllocatedMips 30.08 479.78 13.37 896.09 4.75 66.92 Data center’s energy
is 0.00 W * sec
hostTotalRequestedMips 30.08 479.78 13.37 896.09 4.75 66.92
getUtilizationOfCpu() * 100 0.81 9.02 0.36 16.84 0.13 1.26
previousUtilizationOfCpu 0.00 0.00 0.00 0.00 0.00 0.00
UtilizationOfCpu 0.01 0.09 0.00 0.17 0.00 0.01
timeFrameHostEnergy 0.00 0.00 0.00 0.00 0.00 0.00
(W * sec)
K. Shojaei et al.
 Table 14 continued
Time: 600.1
(Host# To VM#)
Host0 Shutdown Host1 0 3 9 8 Host2 Shutdown Host3 1 4 6 Host4 Shutdown Host5 2 5 7

VM0 (2500) VM1 (2500) VM2 (2500) VM3 (2000) VM4 (2000) VM5 (2000) VM6 (1000) VM7 (1000) VM8 (1000) VM9 (500)

Utilization 0.29 0.19 0.38 0.14 0.25 0.02 0.02 0.06 0.22 0.13
utilizationPredicted 0.53 0.72 0.28 0.36 0.51 0.20 0.22 0.25 0.02 0.36
utilizationPredicted * mips 1318.52 1788.30 701.27 710.83 1012.80 402.85 215.42 249.42 219.42 180.70
totalMips 724.97 475.24 949.90 280.03 500.01 8.06 19.99 59.99 20.04 65.00
totalMipsWithPredic 382.36 339.95 266.45 99.53 253.20 8.06 4.31 14.96 4.40 23.49
totalRequestedMips 382.36 266.45 99.53 8.06 339.95 14.96 4.40 23.49
totalAllocatedMips 382.36 266.45 99.53 8.06 339.95 14.96 4.40 23.49

HOSTs Power Model:

1. PowerModelSpecPowerHpProLiantMl110G5Xeon3075
2. PowerModelSpecPowerHpProLiantMl110G5Xeon3075

Host #0 Host #1 Host #2 Host #3 Host #4 Host #5

VMDFS: virtual machine dynamic frequency scaling framework…

hostTotalAllocatedMips 0.00 509.77 0.00 597.46 0.00 289.47 Data center’s energy is
79053.71 W * sec
hostTotalRequestedMips 0.00 509.77 0.00 597.46 0.00 289.47
getUtilizationOfCpu() * 100 0.00 9.58 0.00 11.23 0.00 5.44
previousUtilizationOfCpu % 0.00 9.62 0.00 16.80 0.00 2.39
UtilizationOfCpu % 0.00 9.58 0.00 11.23 0.00 5.44
timeFrameHostEnergy(W * sec) 0.00 26363.61 0.00 26836.72 0.00 25853.38

123
5975
 Table 14 continued
5976

Time: 900.1
(Host# To VM#)

123
Host0 Shutdown Host1 0 3 9 8 Host2 Shutdown Host3 1 4 6 Host4 Shutdown Host5 2 5 7

VM0 (2500) VM1 (2500) VM2 (2500) VM3 (2000) VM4 (2000) VM5 (2000) VM6 (1000) VM7 (1000) VM8 (1000) VM9 (500)

Utilization 0.26 0.48 0.26 0.19 0.26 0.02 0.00 0.04 0.13 0.15
utilizationPredicted 0.49 0.42 0.55 0.34 0.45 0.22 0.22 0.26 0.22 0.33
utilizationPredicted * mips 1228.52 1050.74 1369.23 678.47 906.64 431.64 217.21 256.61 217.65 165.41
totalMips 650.00 1199.89 650.04 379.97 520.10 40.01 0.01 40.00 129.96 75.00
totalMipsWithPredic 319.42 504.31 356.02 128.90 235.77 8.64 0.00 10.26 28.29 24.81
totalRequestedMips 319.42 504.31 356.02 128.90 235.77 8.64 0.00 10.26 28.29 24.81
totalAllocatedMips 319.42 504.31 356.02 128.90 235.77 8.64 0.00 10.26 28.29 24.81

HOSTs Power Model:

1. PowerModelSpecPowerHpProLiantMl110G5Xeon3075
2. PowerModelSpecPowerHpProLiantMl110G5Xeon3075

Host #0 Host #1 Host #2 Host #3 Host #4 Host #5

hostTotalAllocatedMips 0.00 857.44 0.00 758.98 0.00 0.00 Data center’s energy is
53,750.92 W * sec
hostTotalRequestedMips 0.00 857.44 0.00 758.98 0.00 0.00
getUtilizationOfCpu() * 100 0.00 16.20 0.00 14.27 0.00 0.00
previousUtilizationOfCpu % 0.00 14.86 0.00 12.25 0.00 0.00
UtilizationOfCpu % 0.00 16.12 0.00 14.27 0.00 0.00
timeFrameHostEnergy(W * sec) 0.00 26,996.95 0.00 26,753.97 0.00 0.00
K. Shojaei et al.
 Table 14 continued
VM0 (2500) VM1 VM2 VM3 VM4 VM5 VM6 VM7 VM8 VM9 (500) Summation
(2500) (2500) (2000) (2000) (2000) (1000) (1000) (1000)

vmTotalAllocated 795,340.45 1,062,279.50 799,169.93 375,264.79 668,639.69 42,157.44 11,331.71 41,355.03 28,429.54 83,118.38 3,907,086.46
vmTotalRequested 795,340.45 1,062,981.50 799,540.84 375,264.79 669,304.99 42,204.51 11,343.01 41,462.26 28,442.73 83,144.01 3,909,029.08
Overall SLA violation: 0.00,049,696
vmUnderAllocated 0.00 702.00 370.90 0.00 665.30 47.07 11.30 107.23 13.20 25.63 1942.62
DueToMigration
PDM: 0.000,497
SLA violation 21 Average  0.1
items
SLA Time per active if (previousAllocated < previousRequested) {…}
host: (13 items)
18.03 9.81 18.03 13.92 13.92 13.92 18.03 9.81 18.03 13.92 13.92 9.81 18.03
Total time: 4039.20 slaViolation SLATAH:
TimePerHost: 0.047
189.18
VMDFS: virtual machine dynamic frequency scaling framework…

SLA  slaTimePerActiveHost * slaDegradationDueToMigration

SLA: 0.0,000,233
timesBefore 327.94 327.94 927.94 327.94 627.94
HostShutdown:
numberOfHostShutdowns: 5
meanTimeBeforeHostShutdown: 507.94
stDevTimeBeforeHostShutdown: 268.33
timeBefore 13.92 13.92 27.84 27.84 27.84 27.84 27.84 27.84 27.84 27.84 27.84 9.81
VmMigration
meanTimeBeforeVmMigration: 24.02
stDevTimeBeforeVmMigration: 6.99

123
5977
5978 K. Shojaei et al.

References
1. Barroso LA, Hölzle U (2007) The case for energy-proportional computing. IEEE Comput 40:33–37
2. Hyukho K, Woongsup K, Yangwoo K (2010) Experimental study to improve resource utilization and
performance of cloud systems based on grid middleware. J Commun Comput 7(12):32–43
3. Beloglazov A, Buyya R (2012) Optimal online deterministic algorithms and adaptive heuristics for
energy and performance efficient dynamic consolidation of virtual machines in cloud data centers.
Concurr Comput 24:1397–1420
4. Laszewski G, Wang L, Younge A, He X (2009) Power-aware scheduling of virtual machines in DVFS-
enabled clusters. In: Proceedings of the IEEE International Conference on Cluster Computing and
Workshops
5. Minas L, Ellison B (2009) Energy efficiency for information technology: how to reduce power con-
sumption in servers and data centers. Intel Press, Santa Clara
6. Li Z, Wang C, Lv H, Xu T (2015) Research on CPU workload prediction and balancing in cloud
environment. IJHIT 8(2):159–172
7. Hosseinimotlagh S, Khunjush F (2014) Migration-less energy-aware task scheduling policies in cloud
environments. In: 28th International Conference on Advanced Information Networking and Applica-
tions Workshops (WAINA)
8. Wu C-M, Chang R-S, Chan H-Y (2013) A green energy-efficient scheduling algorithm using the DVFS
technique for cloud datacenters. FGCS 37:141–147
9. Ding Y, Qin X, Liu L, Wang T (2015) Energy efficient scheduling of virtual machines in cloud with
deadline constraint. FGCS 50:62–74
10. Alnowiser A, Aldhahri E, Alahmadi A (2014) Enhanced weighted round robin (EWRR) scheduling
with DVFS technology in cloud. In: CSCI
11. Veni T, Mary S (2016) Prediction model for virtual machine power consumption in cloud environments.
ICRTCSE 87:122–127
12. Dinda PA (1999) The statistical properties of host load. Sci Program 7(3):211–229
13. Li J, Liu X, Zhao Z, Liu J (2015) Energy consumption prediction based on time-series models for
CPU-intensive activities in the cloud. ICA3PP 9531:756–769
14. Beloglazov A, Buyya R, Lee YC, Zomaya A (2011) A taxonomy and survey of energy efficient data
centers and cloud computing systems. Adv Comput 82:47–111
15. Chen Q, Grosso P, Veldt K, Laat C (2011) Profiling energy consumption of VMs for green cloud
computing. In: IEEE Ninth International Conference DASC, pp 768–775
16. Kansal A, Zhao F, Liu J, Kothari N, Bhattacharya AA (2010) Virtual machine power metering and
provisioning. In: SoCC’10, pp 39–50
17. Bohra A, Chaudhary V (2010) VMeter: power modelling for virtualized clouds. In: Parallel Distributed
Processing, Workshops and Phd Forum (IPDPSW), 2010 IEEE International Symposium on April 2010,
pp 1–8
18. de Lemos et al. (2013) Software engineering for self-adaptive systems: a second research roadmap.
In: LNSC 7475, pp 1–32
19. Fan X, Weber WD, Barroso LA (2007) Power provisioning for a warehouse-sized computer. In: Pro-
ceedings of the 34th Annual International Symposium on Computer Architecture (ISCA), pp 13–23
20. Calheiros RN, Ranjan R, Beloglazov A, Rose C, Buyya R (2011) CloudSim: a toolkit for modeling
and simulation of cloud computing environments and evaluation of resource provisioning algorithms.
Softw Pract Exp 41(1):23–50
21. Park K, Pai VS (2006) CoMon: a mostly-scalable monitoring system for PlanetLab. ACM SIGOPS
Oper Syst Rev 40(1):65–74
22. r-project [Internet]. The R project for statistical computing. Available: https://www.r-project.org/
23. Spec.org [Internet]. Standard Performance Evaluation Corporation (SPEC); 30/12/2013. Available:
http://www.spec.org/power_ssj2008/results/
24. Beloglazov A, Buyya R (2010) Adaptive threshold-based approach for energy-efficient consolida-
tion of virtual machines in cloud data centers. In: Proceedings of the 8th International Workshop on
Middleware for Grids, Clouds and e-Science
25. Beloglazov A, Abawajy J, Buyya R (2012) Energy-aware resource allocation heuristics for efficient
management of data centers for cloud computing. FGCS 26(5):755–768

123
VMDFS: virtual machine dynamic frequency scaling framework… 5979

Affiliations

Kiamars Shojaei1 · Faramarz Safi-Esfahani2,3 · Saeed Ayat4

Kiamars Shojaei
shojaei.kumars@gmail.com
Saeed Ayat
dr.ayat@pnu.ac.ir
1 Department of Computer Engineering and Information Technology, Payame Noor University,
Qeshm International Center branch, Qeshm, Iran
2 Faculty of Computer Engineering, Najafabad Branch, Islamic Azad University, Najafabad, Iran
3 Big Data Research Center, Najafabad Branch, Islamic Azad University, Najafabad, Iran
4 Department of Computer Engineering and Information Technology, Payame Noor University,
Najafabad Branch, Najafabad, Iran

123