Polytechnic School, National University of Asunción

Doctoral Program in Computer Science

Many-Objective Resource Allocation for

Elastic Infrastructures in Overbooked Cloud
Computing Datacenters Under Uncertainty

Author

Fabio López Pires

Advisor

Prof. Benjamín Barán, D.Sc.

Submitted in total fulfillment of the requirements of the degree of

Doctor of Computer Science

July, 2017
 Abstract

Achieving an efficient resource management in cloud computing datacenters is considered a

relevant research challenge in the specialized literature. The present work focuses on resource
allocation, specifically in one of the most studied problems for resource allocation in cloud
computing datacenters: the process of selecting which virtual machines (VMs) should be
hosted at each physical machine (PM) of a cloud computing infrastructure, commonly known
in the specialized literature as Virtual Machine Placement (VMP) problem.
Based on a systematic literature review, this work presents novel taxonomies of the VMP
problem for cloud computing datacenters in order to: (1) understand different possible envir-
onments where a VMP problem could be studied, from both provider and broker perspectives
in different deployment architectures considering online as well as offline formulations, (2)
identify existing approaches for the formulation and resolution of the VMP as a combinato-
rial optimization problem and (3) present a detailed view of the VMP problem, identifying
main research opportunities to further advance in this research area.
Taking into account the large number of existing objective functions in the VMP lite-
rature, this work tackles research challenges in relation to a previously unexplored domain:
Many-Objective Virtual Machine Placement (MaVMP) problems. In particular, first formu-
lations of the following resource allocation problems were proposed: (1) MaVMP for initial
placement of VMs (static), (2) MaVMP with reconfiguration of VMs (semi-dynamic) and
(3) MaVMP for complex cloud computing environments under uncertainty (dynamic).
For MaVMP problems for initial placement of VMs, this work proposes a general many-
objective optimization framework that is able to consider as many objective functions as
needed when formulating MaVMP problems in virtualized datacenters, including a method
for effectively reducing the potentially unmanageable number of non-dominated solutions as
well as an interactive Memetic Algorithm (MA) for solving the formulated many-objective
problems. As an example of utilization of the proposed framework, for the first time a formu-
lation of a MaVMP for initial placement of VMs is proposed, considering the simultaneous
optimization of the following objective functions: (1) power consumption, (2) network traffic,
(3) economical revenue, (4) quality of service and (5) network load balancing.
Additionally, a first formulation of a MaVMP problem with reconfiguration of VMs is
presented, considering the simultaneous optimization of the following five conflicting objec-
tive functions: (1) power consumption, (2) inter-VM network traffic, (3) economical revenue,
(4) number of VM migrations and (5) network traffic overhead for VM migrations. An eva-
luation of five strategies for automatically selecting a convenient solution from a Pareto set
approximation is presented as well as an extended MA for solving the formulated problem.
Next, a first proposal of a complex Infrastructure as a Service (IaaS) environment for
VMP problems is proposed, considering service elasticity, including both vertical and hori-
zontal scaling of cloud services, as well as overbooking of physical resources, including server
and networking resources. Additionally, a first formulation of a MaVMP problem taking
into account the proposed complex IaaS environment is presented for the optimization of
the following four objective functions: (1) power consumption, (2) economical revenue, (3)
resource utilization, as well as (4) placement reconfiguration time. A two-phase optimization
scheme is considered for incremental VMP and VMP reconfiguration, combining advantages
of both online and offline VMP formulations, introducing a novel prediction-based method to
decide when to trigger a placement reconfiguration as well as a novel update-based method
to decide what to do with VMs requested during placement recalculation times.
Finally, a first scenario-based uncertainty approach for modeling the following relevant
uncertain parameters of the proposed complex IaaS environment is presented: (1) virtual
resources capacities (vertical elasticity), (2) number of VMs that compose cloud services
(horizontal elasticity), (3) utilization of server virtual resources (relevant for overbooking)
and (4) utilization of networking virtual resources (also relevant for overbooking). An expe-
rimental evaluation of the proposed two-phase optimization scheme against state-of-the-art
alternatives for VMP problems is also summarized, considering 400 different scenarios.
 To Lorena and Clotilde.

Run rabbit run

Dig that hole, forget the sun
And when at last the work is done
Don’t sit down, it’s time to dig another one
(Breathe by Pink Floyd)
 Acknowledgments

A doctoral candidature is a once-in-a-lifetime opportunity and experience. My sincere

acknowledgments to those people who helped me along my candidature. First of all, I would
like to thank my advisor, Professor Benjamín Barán, who has given me invaluable guidance
and advice throughout my work. Working with him has given me great satisfaction.
I would like to thank all my research colleagues. In particular, I thank Diego Ihara, Au-
gusto Amarilla, Leonardo Benítez, Rodrigo Ferreira, Saúl Zalimben, Sara Arévalos, Jammily
Ortigoza and Lino Chamorro. Working with them was an amazing experience.
Also, my sincere gratitude to the qualify exam committee members, Dr. Horacio Legal,
Dr. Fernando Brunetti, Dr. Juan Pane and Dr. Diego Pinto, many thanks for their con-
strictive comments and suggestions on improving my research work.
I would also like to thank all the past and current members of the Scientific and Applied
Computing Laboratory (LCCA), at the National University of Asunción. In particular, I
thank Professor Christian Schaerer, Cristhian von Lucken, Enrique Dávalos, Pedro Villalba
and Christian Cappo for their help and comments during my work.
I acknowledge all authorities and co-workers from the Itaipu Technological Park for
providing me with scholarships and invaluable support to pursue my doctoral studies. I
thank the external examiners for their reviews and suggestions on improving my work. I am
heartily thankful to my family for their support and encouragement at all times.
Contents

List of Figures 10

List of Tables 11

List of Algorithms 13

List of Acronyms 14

Nomenclature 17

1 Introduction 22
1.1 Research Problems and Objectives . . . . . . . . . . . . . . . . . . . . . . . 24
1.2 Contributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
1.3 Thesis Organization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
1.4 Additional Publications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30

I Literature Review 31

2 VMP Taxonomies for Cloud Computing 32

2.1 VMP Literature Review . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32
2.1.1 Keywords Search . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33
2.1.2 Publisher Filtering . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33
2.1.3 Reading of Abstract . . . . . . . . . . . . . . . . . . . . . . . . . . . 33
2.2 VMP Environment Taxonomy . . . . . . . . . . . . . . . . . . . . . . . . . . 34
2.2.1 Orientations: Provider-oriented or Broker-oriented . . . . . . . . . . . 35
2.2.2 Deployment Architectures . . . . . . . . . . . . . . . . . . . . . . . . 35
2.2.3 Types of Formulation: Offline or Online . . . . . . . . . . . . . . . . 37
2.3 VMP Formulation Taxonomy . . . . . . . . . . . . . . . . . . . . . . . . . . 39

5
 2.3.1 Optimization Approaches . . . . . . . . . . . . . . . . . . . . . . . . 40
2.3.2 Objective Function Groups . . . . . . . . . . . . . . . . . . . . . . . . 43
2.3.3 Solution Techniques . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48
2.4 VMP Research Opportunities . . . . . . . . . . . . . . . . . . . . . . . . . . 51
2.4.1 Unexplored Environments, Formulations and Solution Techniques . . 53
2.4.2 Broker-oriented VMP considering Online Formulations . . . . . . . . 54
2.4.3 Provider-oriented VMP considering Online Formulations . . . . . . . 55
2.4.4 Provider-oriented VMP considering PMO Optimization . . . . . . . . 56
2.4.5 Provider-oriented VMP in Distributed and Federated Clouds . . . . . 56

II MaVMP for Initial Placement of VMs (static) 58

3 MaVMP for Virtualized Datacenters 59

3.1 Many-Objective Optimization Framework . . . . . . . . . . . . . . . . . . . . 59
3.2 Part 2 - Problem Formulation . . . . . . . . . . . . . . . . . . . . . . . . . . 61
3.2.1 Part 2 - Input Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62
3.2.2 Part 2 - Output Data . . . . . . . . . . . . . . . . . . . . . . . . . . . 64
3.2.3 Part 2 - Constraints . . . . . . . . . . . . . . . . . . . . . . . . . . . 64
3.2.4 Part 2 - Objective Functions . . . . . . . . . . . . . . . . . . . . . . . 66
3.2.5 Basic Example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 69
3.3 Interactive Memetic Algorithm for MaVMP . . . . . . . . . . . . . . . . . . 70
3.3.1 Part 2 - Population Initialization . . . . . . . . . . . . . . . . . . . . 71
3.3.2 Part 2 - Infeasible Solution Reparation . . . . . . . . . . . . . . . . . 72
3.3.3 Part 2 - Local Search . . . . . . . . . . . . . . . . . . . . . . . . . . . 73
3.3.4 Part 2 - Fitness Function . . . . . . . . . . . . . . . . . . . . . . . . . 74
3.3.5 Part 2 - Variation Operators . . . . . . . . . . . . . . . . . . . . . . . 75
3.3.6 Many-Objective Considerations . . . . . . . . . . . . . . . . . . . . . 75
3.4 Part 2 - Experimental Results . . . . . . . . . . . . . . . . . . . . . . . . . . 76
3.4.1 Part 2 - Experimental Environment . . . . . . . . . . . . . . . . . . . 76
3.4.2 Experiment 1: Quality of Solutions . . . . . . . . . . . . . . . . . . . 77
 3.4.3 Experiment 2: Interactive Bounds . . . . . . . . . . . . . . . . . . . . 78
3.4.4 Experiment 3: Algorithm Scalability . . . . . . . . . . . . . . . . . . 80

III MaVMP with Reconfiguration of VMs (semi-dynamic) 82

4 MaVMP with Placement Reconfiguration 83

4.1 Part 3 - Problem Formulation . . . . . . . . . . . . . . . . . . . . . . . . . . 84
4.1.1 Part 3 - Input Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . 84
4.1.2 Part 3 - Output Data . . . . . . . . . . . . . . . . . . . . . . . . . . . 87
4.1.3 Part 3 - Constraints . . . . . . . . . . . . . . . . . . . . . . . . . . . 88
4.1.4 Part 3 - Objective Functions . . . . . . . . . . . . . . . . . . . . . . . 89
4.2 Extended Memetic Algorithm for MaVMP . . . . . . . . . . . . . . . . . . . 91
4.2.1 Part 3 - Population Initialization . . . . . . . . . . . . . . . . . . . . 93
4.2.2 Part 3 - Infeasible Solution Reparation . . . . . . . . . . . . . . . . . 93
4.2.3 Part 3 - Local Search . . . . . . . . . . . . . . . . . . . . . . . . . . . 94
4.2.4 Part 3 - Fitness Function . . . . . . . . . . . . . . . . . . . . . . . . . 95
4.2.5 Part 3 - Variation Operators . . . . . . . . . . . . . . . . . . . . . . . 96
4.3 Solution Selection Strategies . . . . . . . . . . . . . . . . . . . . . . . . . . . 96
4.3.1 Random (S1) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97
4.3.2 Preferred Solution (S2) . . . . . . . . . . . . . . . . . . . . . . . . . . 97
4.3.3 Minimum Distance to Origin (S3) . . . . . . . . . . . . . . . . . . . . 97
4.3.4 Lexicographic Order . . . . . . . . . . . . . . . . . . . . . . . . . . . 97
4.4 Part 3 - Experimental Results . . . . . . . . . . . . . . . . . . . . . . . . . . 98
4.4.1 Part 3 - Experimental Environment . . . . . . . . . . . . . . . . . . . 98
4.4.2 Experiment 4: Selection Strategy Evaluation . . . . . . . . . . . . . . 100

IV MaVMP for Cloud Computing (dynamic) 104

5 Experimental Comparison of Algorithms 105

5.1 Background and Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . 106
 5.2 Part 4.A - Problem Formulation . . . . . . . . . . . . . . . . . . . . . . . . . 107
5.2.1 Part 4.A - Input Data . . . . . . . . . . . . . . . . . . . . . . . . . . 108
5.2.2 Part 4.A - Output Data . . . . . . . . . . . . . . . . . . . . . . . . . 110
5.2.3 Part 4.A - Constraints . . . . . . . . . . . . . . . . . . . . . . . . . . 110
5.2.4 Part 4.A - Objective Functions . . . . . . . . . . . . . . . . . . . . . 111
5.2.5 Normalization and Scalarization Method . . . . . . . . . . . . . . . . 114
5.3 Part 4.A - Evaluated Algorithms . . . . . . . . . . . . . . . . . . . . . . . . 115
5.3.1 A1: First-Fit (FF) . . . . . . . . . . . . . . . . . . . . . . . . . . . . 115
5.3.2 A2: Best-Fit (BF) . . . . . . . . . . . . . . . . . . . . . . . . . . . . 115
5.3.3 A3: Worst-Fit (WF) . . . . . . . . . . . . . . . . . . . . . . . . . . . 116
5.3.4 A4: First-Fit Decreasing (FFD) . . . . . . . . . . . . . . . . . . . . . 116
5.3.5 A5: Best-Fit Decreasing (BFD) . . . . . . . . . . . . . . . . . . . . . 116
5.3.6 A6: Memetic Algorithm (MA) . . . . . . . . . . . . . . . . . . . . . . 116
5.4 Part 4.A - Experimental Evaluation . . . . . . . . . . . . . . . . . . . . . . . 117
5.4.1 Part 4.A - Experimental Environment . . . . . . . . . . . . . . . . . . 117
5.4.2 Objective Functions Weights . . . . . . . . . . . . . . . . . . . . . . . 119
5.4.3 Comparison of Algorithms . . . . . . . . . . . . . . . . . . . . . . . . 120

6 MaVMP for IaaS under Uncertainty 124

6.1 Preliminary Concepts and Research Challenges . . . . . . . . . . . . . . . . 126
6.1.1 IaaS Environments for VMP Problems . . . . . . . . . . . . . . . . . 126
6.1.2 Two-Phase Optimization Schemes for VMP problems . . . . . . . . . 128
6.1.3 Uncertainty in Cloud Computing . . . . . . . . . . . . . . . . . . . . 130
6.2 Related Works and Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . 131
6.3 Uncertain VMP Formulation . . . . . . . . . . . . . . . . . . . . . . . . . . . 136
6.3.1 Complex IaaS Environment . . . . . . . . . . . . . . . . . . . . . . . 137
6.3.2 Incremental VMP (iVMP) . . . . . . . . . . . . . . . . . . . . . . . . 141
6.3.3 VMP Reconfiguration (VMPr) . . . . . . . . . . . . . . . . . . . . . . 142
6.3.4 Part 4.B - Constraints . . . . . . . . . . . . . . . . . . . . . . . . . . 143
6.3.5 Part 4.B - Objective Functions . . . . . . . . . . . . . . . . . . . . . . 144
 6.3.6 Normalization and Scalarization Methods . . . . . . . . . . . . . . . . 148
6.3.7 Scenario-based Uncertainty Modeling . . . . . . . . . . . . . . . . . . 150
6.4 Part 4.B - Evaluated Algorithms . . . . . . . . . . . . . . . . . . . . . . . . . 151
6.4.1 Incremental VMP (iVMP) Algorithm . . . . . . . . . . . . . . . . . . 152
6.4.2 VMP Reconfiguration (VMPr) Algorithms . . . . . . . . . . . . . . . 153
6.4.3 Evaluated VMPr Triggering Methods . . . . . . . . . . . . . . . . . . 156
6.4.4 Evaluated VMPr Recovering Methods . . . . . . . . . . . . . . . . . 158
6.5 Part 4.B - Experimental Evaluation . . . . . . . . . . . . . . . . . . . . . . . 160
6.5.1 Part 4.B - Experimental Environment . . . . . . . . . . . . . . . . . . 161
6.5.2 Part 4.B - Experimental Results . . . . . . . . . . . . . . . . . . . . . 165

V Conclusions and Future Directions 168

7 Conclusions and Future Directions 169

References 175
List of Figures

2.1 VMP literature selection process. . . . . . . . . . . . . . . . . . . . . . . . . 33

2.2 Percentage of articles per publisher in the studied universe of 84 papers. . . . 34
2.3 VMP Environment Taxonomy. Example references for each environment are
presented. Unexplored environments are considered as Research Opportunities
(RO). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36

3.1 Example of placement in a virtualized datacenter infrastructure, composed by

PMs, VMs and a network topology. . . . . . . . . . . . . . . . . . . . . . . . 61
3.2 Summary of results obtained in Experiment 2 using adjusted lower and upper
bounds. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 79

4.1 Workload for Experiment 4 (10x100.vmp): number of running VMs as a func-

tion of time t. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 101
4.2 Workload for Experiment 4 (100x1000.vmp): number of running VMs as a
function of time t. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 101

5.1 Experimental Workload Traces: Number of created VMs as a function of time t.117

6.1 Two-phase optimization scheme for VMP problems considered in this work,
presenting a basic example with a placement recalculation time of β = 2 (from
t = 2 to t = 4) and a placement reconfiguration time of γ = 1 (from t = 4 to
t = 5). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 129
6.2 Temporal average cost: Average values of fs (x, t) in DC1 to DC5 per each
scenario s ∈ S. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 166

10
List of Tables

2.1 VMP Formulation Taxonomy. Example references for each environment are
presented. Unexplored environments are considered as Research Opportunities
(RO). In this table f1 (x): Energy Consumption, f2 (x): Network Traffic, f3 (x):
Economical Costs, f4 (x): Performance, f5 (x): Resource Utilization. . . . . . 40
2.2 Objective Functions - Energy Consumption Minimization Group. . . . . . . 44
2.3 Objective Functions - Network Traffic Minimization Group. . . . . . . . . . . 46
2.4 Objective Functions - Economical Costs Optimization Group. . . . . . . . . 47
2.5 Objective Functions - Performance Maximization Group. . . . . . . . . . . . 48
2.6 Objective Functions - Resource Utilization Maximization Group. . . . . . . . 48
2.7 Solution Techniques - Deterministic Algorithms . . . . . . . . . . . . . . . . 49
2.8 Solution Techniques - Heuristics . . . . . . . . . . . . . . . . . . . . . . . . . 50
2.9 Solution Techniques - Meta-Heuristics . . . . . . . . . . . . . . . . . . . . . . 51
2.10 Virtual Machine Placement Taxonomy. Elements of column % represent the
percentage of articles in the studied universe [93]. For simplicity, MAM and
PMO consider only the number of considered objective functions in column
f (x). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52

3.1 Data of network links used in basic example (see Section 3.2.5). . . . . . . . 69
3.2 Types of PMs considered in experiments. For notation see Equation (3.4). . 76
3.3 Instance types of VMs considered in experiments. For notation see Equation
(3.5). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 77
3.4 Problem instances considered in Experiments 1 to 3, all with 50% of VMs
with SLA s = 2. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 77
3.5 Summary of results obtained by the proposed algorithm in Experiment 1. . . 78
3.6 Summary of results obtained by the proposed algorithm in Experiment 3. . . 81

11
4.1 Hardware configuration of PM types considered in Experiment 4. . . . . . . 98
4.2 Instance types of VMs considered. For notation see Equation (4.3). . . . . . 99
4.3 Number of non-dominated solutions per selection strategy. . . . . . . . . . . 100
4.4 Details of Experiment 4. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 100
4.5 Selection Strategy Comparison. For selection strategy notation see Section 4.3.102

5.1 Considered Data for Generated Experimental Workload Traces (W1 to W4 ).

Details in [112]. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 118
5.2 Objective Function Costs of Evaluated Algorithms. . . . . . . . . . . . . . . 119
5.3 Objective Function Costs of Evaluated Heuristics. . . . . . . . . . . . . . . . 120
5.4 VMs Migration Overhead for MA (A6). . . . . . . . . . . . . . . . . . . . . . 123

6.1 Summary of IaaS environments and VMPr methods already studied in related
works. N/A indicates a Not Applicable criterion. . . . . . . . . . . . . . . . 132
6.2 Summary of evaluated algorithms as well as their corresponding VMPr Trig-
gering and Recovering methods. N/A indicates a Not Applicable criterion. . 150
6.3 Basic example of how the proposed VMPr Triggering method predicts values
of F (x, t) for t = 16, t = 17 and t = 18 (N̂ = 3) based on previous discrete
time instants, using Equation (6.23) with α = τ = 0.5. Predicted values are
highlighted. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 158
6.4 Types of PMs considered in simulations. For notation see Section 6.3. . . . . 160
6.5 Number of PMs per type on each considered IaaS cloud datacenter. . . . . . 161
6.6 Virtual Machine (VM) types from Amazon EC2 considered for cloud service
requests in simulations. For notation see Section 6.3. . . . . . . . . . . . . . 161
6.7 Most relevant inputs for example workload trace presented in Table 6.8. . . . 162
6.8 Example of workload trace for a complex IaaS environment. For notation see
Section 6.3. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 163
6.9 Summary of evaluation criteria in experimental results for evaluated algorithms.164
List of Algorithms

1 Interactive Memetic Algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . 72

2 Part 2 - Feasibility Verification . . . . . . . . . . . . . . . . . . . . . . . . . . 73
3 Part 2 - Infeasible Solutions Reparation . . . . . . . . . . . . . . . . . . . . . 74
4 Part 2 - Local Search . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74

5 Extended Memetic Algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . 92

6 Part 3 - Feasibility Verification . . . . . . . . . . . . . . . . . . . . . . . . . . 94
7 Part 3 - Infeasible Solutions Reparation . . . . . . . . . . . . . . . . . . . . . 95
8 Part 3 - Local Search . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 95

9 First-Fit Decreasing (FFD) for iVMP. . . . . . . . . . . . . . . . . . . . . . . 153

10 Memetic Algorithm (MA) for VMPr. . . . . . . . . . . . . . . . . . . . . . . . 154
11 Minimum Migration Time (MMT) for VMPr running at PM Hi . . . . . . . . 155
12 Update-based VMPr Recovering. . . . . . . . . . . . . . . . . . . . . . . . . . 159

13
List of Acronyms

ACO Ant Colony Optimization

AUT Adaptive Utilization Threshold
BF Best-Fit
BFD Best-Fit Decreasing
BIP Binary Integer Programming
BSP Backward Speculative Placement
CP Constraint Programming
CPU Central Process Unit
CRB Cloud Resource Brokerage
CS Cut-and-Search
CSB Cloud Service Broker
CSP Cloud Service Provider
CST Cloud Service Tenant
CWTG Cloud Workload Trace Generator
DES Double Exponential Smoothing
DNS Domain Name Service
DP Dynamic Programming
DRF Dominant Resource First
DVMA Dynamic VM Allocation
EA Evolutionary Algorithm
EC2 Elastic Compute Cloud
ECU Elastic Compute Unit
ESA Exhaustive Search Algorithm
FCFS First-Come First-Served
FF First-Fit
FFD First-Fit Decreasing
GA Genetic Algorithm
GPU Graphical Process Unit
HF Heaviest-Fit
IaaS Infrastructure as a Service
ILP Integer Linear Programming
IMSPM Improved Multidimensional Space Partition Model
iVMP Incremental Virtual Machine Placement
LMABBS Live Migration Algorithm Based on a Basic Set
LP Linear Programming
MA Memetic Algorithm
MAc Management Actions
MAM Multi-Objective Problem solved as Mono-Objective
MaOP Many-Objective Optimization Problem
MaVMP Many-Objective Virtual Machine Placement
MBFD Modified Best-Fit Decreasing
MBVMP Migration-Based Virtual Machine Placement
MF Main Finding
MILP Mixed Integer Linear Programming
MIPS Millions of Instructions per Second
MOEA Multi-Objective Evolutionary Algorithm
MOP Mono-Objective Optimization Problem
MVMP Multi-Objective Virtual Machine Placement
NIC Network Interface Card
NS Neighborhood Search
P2P Peer-to-Peer
PaaS Platform as a Service
PBO Pseudo-Boolean Optimization
PDF Probability Distribution Function
PM Physical Machine
PMO Pure Multi-Objective Optimization Problem
PSO Particle Swarm Optimization
QoE Quality of Experience
QoS Quality of Service
RO Research Opportunity
SA Simulated Annealing
SaaS Software as a Service
SARMS Self-Adaptive Resource Management System
SDN Software Defined Networking
SIP Stochastic Integer Programming
SLA Service Level Agreement
SLLC Shared Last Level Cache
SSD Solid State Drive
TPOVMP Two-Phase Online Virtual Machine Placement
TS Tabu Search
VM Virtual Machine
VMA Virtual Machine Allocation
VMC Virtual Machine Consolidation
VMcP Virtual Machine Placement Consolidation
VMP Virtual Machine Placement
VMPBBO Virtual Machine Placement Biogeography-Based Optimization
VMPr Virtual Machine Placement Reconfiguration
WAN Wide Area Network
WF Worst-Fit
Nomenclature

α Smoothing factor, where 0 ≤ α ≤ 1.

∆rk,j (t) Ratio of unsatisfied resource k at instant t where ∆rk,j (t) = 1 means no
unsatisfied resource, while ∆rk,j (t) = 0 means resource k is unsatisfied.

λk Protection factor for V rk,j ∈ [0,1]. Note that λk = 0 means full overbook-
ing while λk = 1 means no-overbooking.

φk Penalty factor for resource k, where φk ≥ 1.

τ Trend factor, where 0 ≤ τ ≤ 1.

a Network link identifier.

b Number of constraints.

bt Trend of F (x, t) at discrete time t.

C Set of chromosomes.

ci Datacenter identifier of Hi , where 1 ≤ ci ≤ cmax .

Ĉ Constant, large enough to prioritize services with larger SLA over the
ones with lower SLA.

Cla Channel capacity of link la .

Dj,k Binary variable that equals 1 if Vj and Vk are located in different PMs;
otherwise, it is 0.

Dj,k (t) Binary variable that equals 1 if Vj and Vk are located in different PMs at
instant t. otherwise, it is 0.

fi (x, t) Cost of original objective function fi (x, t).

fi (x, t)max Maximum possible cost for fi (x, t).

fi (x, t)min Minimum possible cost for fi (x, t).

f̂i (x, t) Normalized cost of objective function fi (x, t) at instant t.

F (x, t) Single objective function combining each f̂i (x, t) at instant t.

fs (x, t) Combined objective function for all discrete time instants t in scenario
s ∈ S.

fs (x, t) Temporal average of combined objective function for all discrete time in-
stants t in scenario s ∈ S.

F1 Average fs (x, t) for all scenarios s ∈ S [4].

F2 Maximum fs (x, t) considering all scenarios s ∈ S [4].

F3 Minimum fs (x, t) considering all scenarios s ∈ S.

Fa,i,i0 Binary variable that equals 1 if la ∈ mii0 ; otherwise, it is 0.

G Network topology.

H Set of available PMs.

Hi PM with identifier i.

Hcpui Processing resources of Hi .

Hhddi Storage resources of Hi .

Hrami Memory resources of Hi .

i PM identifier.

j VM identifier.

K Capacity set of the communication channels.

L Set of links in G.

Lz Lower bound of objective function z.

la Link with identifier a in L.

M Set of paths for all-to-all PM interconnections.

m Number of VMs.

mi,i0 Network path from Hi to Hi0 in M .

M Ac(Vj ) ∈ Management actions that a hypervisor must execute in order to accommo-

{0, 1, 2, 3} date the x(t + 1) placement corresponding to Vj .

M Ti,i0 Total amount of RAM memory to be migrated from PM Hi to Hi0 .

N Size of the searching universe.

n Number of PMs.
N̂ Number of prediction used in the proposed VMPr Triggering method.

P∗ Optimal Pareto set.

Pknown Pareto set approximation.

Pt Evolutionary population.

PF∗ Optimal Pareto front.

P Fknown Pareto front approximation.

P Freduced Reduced set of Pareto solutions.

P rk,i Physical resource k on Hi , where 1 ≤ k ≤ r.

p Number of decision variables.

pmaxi Maximum power consumption of Hi .

pmini Minimum power consumption of Hi . In what follows, pmini = pmaxi ∗0.6.

q Number of objective functions.

Qt Offspring population.

r Number of considered resources.

Rj Economical revenue for locating Vj .

Rj (t) Economical revenue for allocating Vj in [$].

Rrk,j (t) Economical revenue for attending V rk,j (t).

s Highest priority level of SLAs.

St Expected value of F (x, t) at discrete time t.

SLAj Service Level Agreement SLAj of a Vj . If the highest priority level is s,

then SLAj ∈ {1, . . . , s}.

T Set of network communication between VMs.

T (t) Set of network communication between VMs at instant t.

Tj,k Average network traffic between Vj and Vk [Mbps]. Note that we can
consider Tjj = 0.

T la Total amount of traffic going through link la [Mbps].

tmax Duration of a scenario in discrete time instants.

u Feasible solution.
U Set of decision vectors.

Uz Upper bound of objective function z.

U cpui Utilization ratio of processing resources used by Hi .

U rk,i (t) Utilization ratio of resource k of PM Hi at instant t.

V Set of requested VMs.

V (t) Set of requested VMs at instant t.

V cpuj Processing requirements of Vj .

V hddj Storage requirements of Vj .

V ramj Memory requirements of Vj .

V rk,j (t) Virtual resource k on Vj , where 1 ≤ k ≤ r.

Vj VM with identifier j.

v Feasible solution.

wi Weight of importance associated to fi (x).

x Placement matrix / solution.

X Decision space.

Xf Feasible decision space.

Xj Binary variable that equals 1 if Vj is located for execution on any PM;

otherwise, it is 0.

Xj (t) ∈ {0, 1} Indicates if Vj is allocated for execution on a PM (Xj (t) = 1) or not

(Xj (t) = 0) at instant t.
X̂j Indicates if Vj is allocated on the main provider (X̂j = 0) or on an altern-
ative datacenter of the cloud federation (X̂j = 0.7).

xj,i Binary variable equals 1 if Vj is located to run on Hi ; otherwise, it is 0.

xj,i (t) ∈ {0, 1} Indicates if Vj is allocated (xj,i (t) = 1) or not (xj,i (t) = 0) for execution
in a PM Hi (i.e., xj,i (t) : Vj → Hi ) at a discrete time t.

y Objective vector.

Y Objective space.

Yi Binary variable equals 1 if Hi is turned on; otherwise, it is 0.

Yi (t) ∈ {0, 1} Indicates if Hi is turned on (Yi (t) = 1) or not (Yi (t) = 0) at instant t.

Yf Feasible objective space.

z Objective function identifier.

Zt Known value of F (x, t) at discrete time t.

Z t+1 Value of F (x, t + 1) predicted at discrete time t.

Zj (t) Binary variable that equals 1 if M A(Vj ) = 2, i.e. Vj is migrated, otherwise,

it is 0 (Vj is not migrated).

21
1 Introduction
A significant number of research challenges for delivering computational resources as an
utility (like water, electricity, gas, and telephony) has already been identified [19]. In this
context, cloud computing datacenters deliver infrastructure (IaaS), platform (PaaS) and
software (SaaS) as services, available to end users in a pay-as-you-go basis [107].
Achieving an efficient resource management for IaaS service model in cloud computing
datacenters could be considered one of the most relevant challenges, including important
research topics such as: resource allocation, resource provisioning, resource mapping and
resource adaptation [104]. Admission control and proactive elasticity could be also considered
relevant research topics related to resource management in cloud computing datacenters [40].
The present work focuses on resource allocation, specifically in one of the most studied
problems for resource allocation in cloud computing datacenters: the process of selecting
which virtual machines (VMs) should be hosted at each physical machine (PM) of a cloud
computing infrastructure, commonly known as Virtual Machine Placement (VMP) problem.
Several research articles in the specialized literature demonstrated that solving the VMP
problem for efficient allocation of resources in cloud computing datacenters could significantly
improve energy-efficiency, quality of service (QoS), carbon dioxide emissions, among other
advantages; all of them with significant economical and ecological impact [6, 13, 18].
Beloglazov and Buyya proposed in [14] four different sub-problems for resource allocation
in cloud datacenters: (1) determining when a PM is overloaded, requiring migration of VMs
from this PM; (2) determining when a PM is underloaded, requiring migration of all VMs
from this PM and switching the PM to sleep mode; (3) selecting VMs to migrate from
an overloaded PM; and (4) finding a new placement of the VMs selected for migration,
considering overloaded and underloaded PMs.
A conceptual architecture considering the most studied problems related to resource
allocation in cloud computing datacenters is presented in [40], where placement of admitted
services (composed by VMs) could be solved considering different criteria and requirements.

22
 In cloud computing datacenters, there are several criteria that can be considered when
selecting a possible solution for a VMP problem, depending on management policies and
optimization objectives. These criteria can even change from one period of time to another,
which implies a variety of possible environments, formulations and different objectives to
be considered for optimization of the VMP problem. In this context, nearly 60 different
objective functions were identified in the VMP literature [93, 94, 96].
Taking into account the large number of existing objective functions and possible approa-
ches for objective function modeling, providers of cloud computing datacenters must be able
to formulate the VMP problem as a Pure Multi-Objective Optimization Problem (PMO),
optimizing more than just one objective function at a time (e.g. achieve economical rev-
enue maximization by simultaneously minimizing the total economical penalties for SLA
violations, minimizing operational costs and maximizing the profit for leasing resources). It
is important to consider that PMOs simultaneously optimizing more than three objective
functions are known as Many-Objective Optimization Problems (MaOPs), as defined in [27].
Several challenges should be considered when solving problems with more than three
objective functions [45]. These challenges are intrinsically related to the fact that as the
number of objective functions increase, the proportion of non-dominated solutions in the
population grows, being increasingly difficult to discriminate among solutions using only the
Pareto dominance relation [32]. Additionally, determining which solution to keep and which
to discard in order to converge toward the Pareto set is still a relevant issue to be addressed
[45], making more difficult to solve MaOPs. Clearly, existing difficulties in solving MaOPs
explain why it is still considered an unexplored domain in resource management of cloud
computing datacenters [58] and no many-objective formulations have already been proposed
for the VMP problem in the specialized literature before this doctoral thesis [94, 96].
The main scope of this doctoral thesis is Many-Objective Virtual Machine Placement
(MaVMP) from the perspective of cloud computing providers, focusing on single-cloud de-
ployments for the following formulations of MaVMP problems: (1) initial placement of VMs
(static), (2) reconfiguration of VMs (semi-dynamic), and (3) for cloud computing (dynamic).

23
1.1 Research Problems and Objectives

This doctoral thesis tackles research challenges in relation to Many-Objective Virtual Ma-
chine Placement (MaVMP) problems for virtualized datacenters and cloud computing en-
vironments. In particular, the following research problems are investigated:

1. How to formulate and solve MaVMP problems: The number of objective func-
tions may be rapidly increased once a complete understanding of the VMP problem is
accomplished for practical problems, where many different parameters should be con-
sidered [96]. Based on the large number of possible objective functions for the VMP,
this problem can clearly be addressed as a Many-Objective Virtual Machine Placement
(MaVMP). Considered MaVMP formulations include: (1) initial placement (static),
(2) reconfiguration of VMs (semi-dynamic), and (3) for cloud computing (dynamic).

2. How to manage solutions for a MaVMP problem: As the number of conflic-

ting objectives of a MaVMP problem formulation increases, the total number of non-
dominated solutions increases (even exponentially in some cases), being increasingly
difficult to discriminate among solutions using only the Pareto dominance relation [32].
Proposing methods to effectively reduce the number of solutions in a MaVMP problem
for initial placement of VMs can be considered a relevant research challenge.

3. How to select solutions for a MaVMP problem: In a pure multi-objective

context, a Pareto set approximation can be composed by a large number of non-
dominated solutions [143]. Automatically selecting the most convenient solution from
a set of non-dominated solutions can also be considered as a research challenge for
solving MaVMP problems with reconfiguration of VMs.

4. How to model MaVMP problems for cloud computing under uncertainty: In

cloud computing environments, particularly for IaaS service model, several parameters
could be considered as uncertain variables, taking into account its on-demand provi-
sioning model. Cloud computing datacenter managers should be able to effectively
deal with uncertain parameters for MaVMP problems.

24
 To deal with the research challenges associated to the above mentioned research problems,
the following objectives have been delineated:

1. Explore, analyze, and classify research of the Virtual Machine Placement (VMP) pro-
blem to gain a systematic understanding of the existing approaches and formulations.

2. Formulate for the first time the following resource allocation problems:

(a) MaVMP for initial placement of VMs (static);

(b) MaVMP with reconfiguration of VMs (semi-dynamic);

(c) MaVMP for cloud computing under uncertainty (dynamic).

3. Provide an effective method for reducing the potentially unmanageable number of non-
dominated solutions in the MaVMP problem for initial placement of VMs in virtualized
datacenters.

4. Develop an algorithm specially designed for solving MaVMP problems for initial place-
ment of VMs in virtualized datacenters.

5. Evaluate strategies for automatically selecting a convenient solution from a Pareto set
approximation for MaVMP problems with reconfiguration of VMs.

6. Extend the algorithm proposed in item 4, firstly designed for solving MaVMP problems
for initial placement of VMs, to solve MaVMP problems with reconfiguration of VMs.

7. Propose a complex IaaS environment for VMP problems considering the most relevant
dynamic parameters in the specialized literature.

8. Propose and experimentally evaluate novel methods related to the resolution of VMP
problems in cloud computing environments.

1.2 Contributions

The main contributions of this doctoral thesis are directly related to each of the delineated
objectives. The key contributions are:

25
1. Taxonomies of VMP problems for cloud computing (Objective 1) [10, 94, 96].

2. First formulations of the following resource allocation problems:

(a) A general many-objective optimization framework that is able to consider as

many objective functions as needed when formulating MaVMP problems for ini-
tial placement in virtualized datacenters. As a utilization example of the proposed
framework, for the first time a formulation of a MaVMP for initial placement was
proposed, considering the simultaneous optimization of the following objective
functions: (1) power consumption, (2) network traffic, (3) economical revenue,
(4) quality of service and (5) network load balancing (Objective 2(a)) [92, 95].

(b) A formulation of a MaVMP problem with reconfiguration of VMs considering the

simultaneous optimization of the following five conflicting objective functions:
(1) power consumption, (2) inter-VM network traffic, (3) economical revenue, (4)
number of VM migrations and (5) network traffic overhead for VM migrations
(Objective 2(b)) [68].

(c) A formulation of a MaVMP problem for cloud computing environments, optimi-

zing the following four objective functions: (1) power consumption, (2) economical
revenue, (3) resource utilization and (4) placement reconfiguration time. Addi-
tionally, a first scenario-based uncertainty approach for modeling the following
relevant uncertain parameters of a proposed complex IaaS environment: (1) vir-
tual resources capacities (vertical elasticity), (2) number of VMs that compose
cloud services (horizontal elasticity), (3) utilization of CPU and RAM memory
virtual resources (relevant for overbooking) as well as (4) utilization of networking
virtual resources (also relevant for overbooking) (Objective 2(c)) [99].

3. A method for reducing the potentially unmanageable number of non-dominated solu-

tions for MaVMP problems for initial placement in virtualized datacenters. The pro-
posed method is based on adjustable lower and upper bounds associated to each objec-
tive function. The effectiveness of the proposed method converging to a manageable
number of solutions was proved in different experiments (Objective 3) [92, 95].

26
 4. An interactive Memetic Algorithm (MA), designed for solving MaVMP problems for
initial placement of VMs in virtualized datacenters. An experimental evaluation of the
quality of solutions obtained by the proposed algorithm against optimal solutions in
different scenarios as well as its scalability solving instances of the problem with large
numbers of PMs and VMs (Objective 4) [95].

5. An experimental evaluation of strategies for automatically selecting a convenient solu-

tion from a Pareto set approximation for MaVMP problems with reconfiguration of
VMs. The following five selection methods were evaluate in different scenarios: (1) ran-
dom, (2) preferred solution, (3) minimum distance to origin, (4) lexicographic order
for providers and (5) lexicographic order for service (Objective 5) [68].

6. An extended MA for solving MaVMP problems with reconfiguration of VMs, including

the implementation of five evaluated selection strategies (Objective 6) [68].

7. A first proposal of a complex IaaS environment for VMP problems considering service
elasticity, including both vertical and horizontal scaling of cloud services and over-
booking of physical resources, including server (CPU and RAM) as well as networking
resources (Objective 7) [99].

8. A two-phase optimization scheme for VMP problems, combining advantages of both

online (iVMP) and offline (VMPr) VMP formulations in the proposed IaaS environ-
ment, introducing a novel prediction-based VMPr Triggering method to decide when
to trigger a placement reconfiguration as well as a novel update-based VMPr Recover-
ing method to decide what to do with VMs requested during placement recalculation
times (Objective 8) [99].

1.3 Thesis Organization

The core parts and chapters of this doctoral thesis are derived from articles published during
the doctoral candidature. The remainder of this doctoral thesis is organized as follows:

27
• Part I presents taxonomies of the VMP problem for cloud computing datacenters,
presenting a general vision of the mentioned problem and identifying main research
opportunities. This chapter is derived from:

– Fabio López-Pires and Benjamín Barán (2015), ‘A Virtual Machine Placement

Taxonomy’. In 2015 15th IEEE/ACM International Symposium on Cluster, Cloud
and Grid Computing (CCGrid), (pp. 159-168), China. IEEE [94].

– Fabio López-Pires and Benjamín Barán (2017), ‘Cloud Computing Resource

Allocation Taxonomies’. International Journal of Cloud Computing. Inderscience
Publishers [96].

– Benjamín Barán and Fabio López-Pires (2017), ‘Resource Allocation for Cloud
Infrastructures: Taxonomies and Research Challenges’. Research Advances in
Cloud Computing. S. Chaudhary, G. Somani, R. Buyya (Editors). Springer [10].

• Part II proposes a general many-objective optimization framework for MaVMP pro-

blems for initial placement of VMs, including a method for effectively reducing the
potentially unmanageable number of non-dominated solutions as well as an interactive
MA for solving the formulated problems. This chapter is derived from:

– Fabio López-Pires and Benjamín Barán (2015), ‘A Many-Objective Optimiza-

tion Framework for Virtualized Datacenters’. In 2015 5th International Conference
on Cloud Computing and Service Science (CLOSER), (pp. 439-450), Portugal.
SciTePress (Best Student Paper Award) [92].

– Fabio López-Pires and Benjamín Barán (2017), ‘Many-Objective Virtual Ma-

chine Placement’. Journal of Grid Computing. Springer (Invited Paper) [95].

• Part III presents a first formulation of a MaVMP problem with reconfiguration of VMs,
considering the simultaneous optimization of five objective functions. Additionally, an
evaluation of five different strategies for automatically selecting a convenient solution
from a Pareto set approximation as well as an extended MA for solving the formulated
problem are also presented. This chapter is derived from:

28
 – Diego Ihara, Fabio López-Pires and Benjamín Barán (2015), ‘Many-Objective
Virtual Machine Placement for Dynamic Environments’. In 2015 IEEE/ACM 8th
International Conference on Utility and Cloud Computing (UCC), (pp 203-207),
Cyprus. ACM/IEEE [68].

– Fabio López-Pires (2016), ‘Many-Objective Resource Allocation in Cloud Com-

puting Datacenters". 4th IEEE International Conference on Cloud Engineering
(IC2E). Doctoral Symposium. Germany. IEEE [88].

– Fabio López-Pires and Benjamín Barán (2017), ‘Many-Objective Optimization

for Virtual Machine Placement in Cloud Computing’. Research Advances in Cloud
Computing. S. Chaudhary, G. Somani, R. Buyya (Editors). Springer [97].

• Part IV presents a first formulation of a MaVMP problem for cloud computing con-
sidering the optimization of four objective functions in a complex IaaS environment.
Additionally, a first scenario-based uncertainty approach for modeling four uncertain
parameters of the proposed complex IaaS environment is presented, as well as novel
methods related to the resolution of the proposed problem in cloud computing data-
centers. This chapter is derived from:.

– Fabio López-Pires, Benjamín Barán, Augusto Amarilla, Leonardo Benítez,

Rodrigo Ferreira, Saúl Zalimben (2016), ‘An Experimental Comparison of Al-
gorithms for Virtual Machine Placement Considering Many Objectives". 9th IFIP
Latin America Networking Conference (LANC). Chile. ACM [98].

– Fabio López-Pires, Benjamín Barán, Leonardo Benítez, Saúl Zalimben, Au-

gusto Amarilla (2017), ‘Virtual Machine Placement for Elastic Infrastructures in
Overbooked Cloud Computing Datacenters Under Uncertainty". Future Genera-
tion Computer Systems. Elsevier [99].

• Part V summarizes the main findings of this doctoral thesis and presents several future
research directions.

29
1.4 Additional Publications

• Publications by the author in related research topics not included in this thesis:

– Sara Arévalos, Fabio López-Pires, Benjamín Barán (2015), ‘Auction-based Re-

source Provisioning in Cloud Computing. A Taxonomy’. In XLI Latin American
Computing Conference (CLEI), Perú. IEEE [8].

– Jammily Ortigoza, Fabio López-Pires, Benjamín Barán (2016), ‘A Taxonomy

on Dynamic Environments for Provider-oriented Virtual Machine Placement". In
4th International Conference on Cloud Engineering (IC2E). Germany. IEEE [112].

– Sara Arévalos, Fabio López-Pires, Benjamín Barán (2016), ‘Comparative Eva-

luation of Algorithms for Auction-based Cloud Pricing Prediction". In 4th IEEE
International Conference on Cloud Engineering (IC2E). Germany. IEEE [7].

– Lino Chamorro, Fabio López-Pires, Benjamín Barán (2016), ‘A Genetic Al-

gorithm for Dynamic Cloud Application Brokering". In 4th International Confer-
ence on Cloud Engineering (IC2E). Germany. IEEE [24].

– Jammily Ortigoza, Fabio López-Pires, Benjamín Barán (2016), ‘Workload Gen-

eration for Virtual Machine Placement in Cloud Computing Environments". In
XLII Latin American Computing Conference (CLEI). Chile. IEEE [113].

– Augusto Amarilla, Leonardo Benítez, Saúl Zalimben, Fabio López-Pires, Ben-

jamín Barán (2017), ‘Evaluating a Two-Phase Virtual Machine Placement Op-
timization Scheme for Cloud Computing Datacenters". In 12th Metaheuristics
International Conference (MIC). Spain. [5].

– Nabil Chamas, Fabio López-Pires, Benjamín Barán (2017), ‘Two-Phase Virtual

Machine Placement Algorithms for Cloud Computing. An Experimental Evalua-
tion under Uncertainty". Under Review.

30
 Part I

Literature Review

31
2 VMP Taxonomies for Cloud Computing
The VMP problem has been extensively studied in cloud computing literature and several
surveys have already been presented. Existing surveys focused on specific issues such as:
(1) energy-efficient techniques applied to the problem [13, 120], (2) particular deployment
architectures where the VMP problem is applied, as federated clouds [50], and (3) methods
for comparing performance of placement algorithms in large on-demand clouds [109].
The above mentioned surveys and research articles focused into specific issues related
to the VMP problem. Consequently, López-Pires and Barán proposed in [94, 96] a general
and extensive study of a large part of the VMP literature including 84 studied articles [93],
presenting a wide analysis of the existing approaches for the formulation and resolution of
the VMP as an optimization problem. Additionally, a novel taxonomy was proposed in [94]
for the classification of the studied articles by the main following criteria: (1) optimization
approach, (2) objective function and (3) solution technique.
The taxonomy presented in [94] was extended, including novel taxonomies and presenting
a detailed view of the existing approaches as well as several possible research opportunities to
further advance in this research area. The presented taxonomies [96] could guide interested
readers to: (1) understand different possible environments where a VMP problem could be
studied, considering both provider and broker perspectives in different deployment archi-
tectures, (2) identify existing approaches for the formulation and resolution of the VMP as
an optimization problem and (3) present a detailed view of the VMP problem, identifying
research opportunities to further advance in this research area.

2.1 VMP Literature Review

For this doctoral thesis, a selection process of current research was defined and performed
in order to study a large part of the most relevant VMP literature. This section presents a
detailed description of the literature selection process which is summarized in Figure 2.1.

32
 keyword publisher abstract selected
search filtering reading articles
(446 articles) (172 articles) (93 articles) (84 articles)

Figure 2.1: VMP literature selection process.

2.1.1 Keywords Search

The selection process of relevant articles started with a search of research articles from Google
Scholar database [scholar.google.com] with at least one of the following selected keywords
in the article title: (1) virtual machine placement, (2) vm placement, (3) virtual machine
consolidation, (4) vm consolidation or (5) server consolidation. This keywords search step
results in 446 research articles. A detailed list of the 446 resulting articles could be found in
[91], specifically in the worksheet keywords search.

2.1.2 Publisher Filtering

Considering the large number of results from keywords search step, the literature selection
process focused in research articles published by the following relevant publishers: (1) ACM,
(2) IEEE, (3) Elsevier and (4) Springer. Percentage of articles per publisher in the studied
universe is summarized in Figure 2.2. This publisher filtering step results in a reduction from
446 to 172 (38.6%) research articles of the literature. A detailed list of the 172 resulting
articles could be found in [91], specifically in the worksheet publisher filtering.

2.1.3 Reading of Abstract

Considering the 172 resulting articles from the previous step, a reading of the abstract was
performed to identify only the most relevant articles that study the VMP problem.

33
 IEEE ACM
11%
69%
5%
Elsevier
15%

Springer

Figure 2.2: Percentage of articles per publisher in the studied universe of 84 papers.

After the reading of the abstract, 93 research articles were selected from the VMP li-
terature. A detailed list of the 93 resulting articles could be found in [91], specifically in
the worksheet abstract reading. Finally, short papers (i.e. research articles with less than
6 pages) were removed from the selected literature, resulting in 84 selected articles of the
VMP literature for the detailed study presented in this doctoral thesis.

2.2 VMP Environment Taxonomy

Depending on the particular environment where a VMP problem will be studied, several
different considerations should be taken into account before considering a particular for-
mulation or technique for the resolution of the considered VMP problem. In this context,
different possible environments could be identified by classifying research articles in the VMP
literature by: (1) orientation, (2) deployment architecture and (3) type of formulation.
First, a VMP problem could be studied from both provider or broker perspectives. A
provider-oriented VMP problem could be studied considering one of the following deployment
architectures: single-cloud, distributed-cloud or federated-cloud, while a broker-oriented
VMP problem could be studied considering a multi-cloud deployment architecture. Addi-
tionally, both provider-oriented and broker-oriented VMP problems, in any of the possible
deployment architectures, could be studied considering two different types of formulation:
offline or online formulations.

34
 For a complete understanding of the possible environments where a VMP problem could
be studied, considering both provider and broker perspectives in different deployment ar-
chitectures, Figure 2.3 presents the taxonomy described in this section including example
references from the studied VMP literature [93].
The following subsections present a detailed description of the mentioned classification
criteria, including important definitions for a complete understanding of the VMP problem.

2.2.1 Orientations: Provider-oriented or Broker-oriented

Definition 2.2.1 A provider-oriented VMP problem is the process of selecting which VMs
should be hosted at each PMs of a cloud computing infrastructure.

Resource allocation in cloud computing datacenters is a main concern for Cloud Service
Providers (CSPs). According to [94], the VMP problem is mainly formulated from this
perspective. It should be mentioned that in a provider-oriented VMP, a Cloud Service Tenant
(CST) cannot decide which PMs will host the requested VMs. In the specialized literature,
this particular problem is also known as Virtual Machine Allocation (VMA) problem [69].

Definition 2.2.2 A broker-oriented VMP problem is the process of selecting which VMs
should be hosted at each Cloud Service Provider (CSP) of a cloud computing market.

Considering that the number of CSPs has been rapidly increased and nowadays there are
different pricing schemes, VM offers and features, it is difficult for CSTs to search for the most
convenient option in cloud computing markets and decide which CSP is the most convenient
to host the requested resources. Consequently, a VMP problem can also be formulated from
a broker perspective who is responsible of helping CSTs to find good allocation deals. This
particular problem is also known as Cloud Resource Brokerage (CRB) problem [140].

2.2.2 Deployment Architectures

Different deployment architectures can be considered for a VMP problem, depending on the
number of cloud computing datacenters associated to the problem instance and the type of
interconnection or relation of these datacenters.

35
 VMP Problem

Provider-oriented Broker-oriented

Single-Cloud Distributed-Cloud Federated-Cloud Multi-Cloud

Offline Online Offline Online Offline Online Offline Online

[89] [121] [26] [15] RO [39] [105] RO

Figure 2.3: VMP Environment Taxonomy. Example references for each environment are
presented. Unexplored environments are considered as Research Opportunities (RO).

A provider-oriented VMP problem could be studied considering one of the following

deployment architectures: single-cloud, distributed-cloud or federated-cloud, while a broker-
oriented VMP problem could be studied in a multi-cloud deployment architecture.

Definition 2.2.3 A provider-oriented VMP problem in a single-cloud deployment is the

process of selecting which VMs should be hosted at each PMs of a single cloud datacenter.

In scenarios considering a single cloud computing datacenter, a CSP could formulate a

VMP problem subject to commonly studied constraints such as: unique placement of VMs,
maximum capacity of resources [89] or Service Level Agreement (SLA) compliance [6, 65].
According to [94], the single-cloud deployment architecture is the most studied scenario.

Definition 2.2.4 A provider-oriented VMP problem in a distributed-cloud deployment is

the process of selecting which VMs should be hosted at each PMs of more than one cloud
computing datacenter owned by the same Cloud Service Provider (CSP).

For CSPs with a global infrastructure (e.g. Amazon), a single-cloud deployment archi-
tecture could be extended to several distributed cloud computing datacenters, where the
formulation of a VMP problem may include particular constraints and considerations.
CSPs with geo-distributed cloud computing datacenters may be interested in studying
a VMP problem for the provision of differentiated services to world-wide tenants, such as
redundancies of services (placement of VMs in different datacenters) [101] or quality and
performance of services (placement of VMs in datacenters closer to end users) [17], increasing
the complexity of the VMP problem formulation.

36
Definition 2.2.5 A provider-oriented VMP problem in a federated-cloud deployment is the
process of selecting which VMs should be hosted at each PMs of more than one cloud com-
puting datacenter owned by different Cloud Service Providers (CSPs) in a cloud federation.

In federated clouds, CSPs with excess capacity lease resources to other CSPs in need
of temporary additional resources; consequently, particular considerations associated to a
VMP problem in this deployment architecture have to be studied [50]. For example, trading
policies are generally not the same for CSPs that form part of the same cloud federation, so
a VMP problem could be studied for cost-effective selection of CSPs for workload peaks.

Definition 2.2.6 A broker-oriented VMP problem in a multi-cloud deployment is the pro-

cess of selecting which VMs should be hosted at each Cloud Service Provider (CSP) of a
cloud computing market composed by more than one CSP.

Cloud Service Brokers (CSBs) or CSTs can require VMs to be deployed in cloud com-
puting datacenters owned by different CSPs, according to particular requirements such as
disaster recovery reasons or due to legislation [41], just to cite a few.

2.2.3 Types of Formulation: Offline or Online

A VMP problem could be formulated as an offline problem considering the placement of

VMs into PMs (for provider-oriented VMP) or VMs into CSPs (for broker-oriented VMP)
for a static or semi-dynamic cloud service deployment.

Definition 2.2.7 An offline formulation of a provider-oriented VMP problem in any po-

ssible deployment architecture could be understood as the process of selecting which VMs
should be hosted at each PMs of the considered cloud computing datacenters for a static or
semi-dynamic cloud service deployment.

For a provider-oriented VMP, an offline formulation considers a previously known set of

VMs requested in static or semi-dynamic environments. It should be noted that offline for-
mulations of VMP problems are mostly appropriate for initial placement of VMs [89, 92, 100]

37
or for virtualized datacenters with deployments of VMs that rarely change its configuration
over time (i.e. reconfiguration of VMs) [68].

Definition 2.2.8 An offline formulation of a broker-oriented VMP problem could be under-

stood as the process of selecting which VMs should be hosted at each CSP of the considered
cloud computing market for a static cloud service deployment.

The broker-oriented VMP is mostly formulated as an offline problem, considering that

possible re-locations of VMs to different CSPs are still limited by interoperability factors.
A VMP problem could also be formulated as an online problem considering the placement
of VMs into PMs (provider-oriented VMP) or VMs into CSPs (broker-oriented VMP) for a
cloud service deployment with dynamic demand or parameters that change over time.

Definition 2.2.9 An online formulation of a provider-oriented VMP problem in any possible

deployment architecture could be understood as the process of selecting which VMs should
be hosted at each PMs of the considered cloud computing datacenters considering dynamic
demand or parameters that change over time.

It should be noted that an online formulation for a provider-oriented VMP problem could
be a more appropriate approach for resource allocation in cloud computing datacenters,
considering the on-demand model of cloud computing with dynamic resource provisioning
and dynamic workloads of modern cloud applications [14]. In this context, preliminary
results out of the scope of this work studied an extended taxonomy for the classification
of possible online formulations for the provider-oriented VMP, presenting an analysis of
possible challenges for providers in cloud computing environments based on the most relevant
dynamic parameters studied so far in the existing VMP literature such as: (1) resource
capacities of VMs (related to vertical elasticity), (2) number of VMs of a service (related to
horizontal elasticity) and (3) utilization of resources of VMs (related to overbooking) [112]. It
should be noted that extending the taxonomies presented in this work for specializing possible
VMP environments could be considered as a research opportunity to further advance in this
research area.

38
Definition 2.2.10 An online formulation of a broker-oriented VMP problem in any possible
deployment architecture could be understood as the process of selecting which VMs should be
hosted at each CSPs of a given cloud computing market considering demands or parameters
that change over the time.

Considering that the number of CSPs has been rapidly increased, nowadays several dy-
namic parameters could be considered for the broker-oriented VMP problem such as: (1)
dynamic pricing schemes, (2) dynamic VM offers or (3) dynamic requirements. Research on
online formulations of the broker-oriented VMP problem should advance to address existing
opportunities in emerging cloud computing markets as presented in a few articles [82, 102].

2.3 VMP Formulation Taxonomy

Considering each possible environment where a VMP problem could be studied (see Figure
2.3), several different formulations of the problem could be proposed. In this context, formu-
lations of a VMP problem may be classified by the: (1) optimization approach, (2) objective
function and (3) solution technique, as initially considered in [94].
First, a VMP problem could be formulated considering one of the following optimiza-
tion approaches: (1) mono-objective (MOP), (2) multi-objective solved as mono-objective
(MAM) or (3) pure multi-objective (PMO). Once the optimization approach is defined, for-
mulations may also be classified by the objective function(s) studied, both in minimization
and maximization contexts. These objective functions could be optimized separately or si-
multaneously, depending on the selected optimization approach. Finally, solution techniques
for solving a VMP problem could be used as a third classification criterion [94].
For a complete understanding of possible approaches for the formulation and resolution of
a VMP problem, Table 2.1 presents the taxonomy described in this section, including example
references from the studied VMP literature [93]. The following subsections present a detailed
description of the classification criteria above defined as well as special considerations for
particular formulations.

39
Table 2.1: VMP Formulation Taxonomy. Example references for each environment are
presented. Unexplored environments are considered as Research Opportunities (RO). In this
table f1 (x): Energy Consumption, f2 (x): Network Traffic, f3 (x): Economical Costs, f4 (x):
Performance, f5 (x): Resource Utilization.
Objective Functions
Technique Approach
f1 (x) f2 (x) f3 (x) f4 (x) f5 (x)
MOP [53] [114] [30] [15] [84]
Deterministic
MAM [6, 131] [6, 129] RO RO [129, 131]
Algorithms
PMO RO RO RO RO RO
MOP [39] [33] [125] [55] [84]
Heuristics MAM [37, 121] [37, 61] [29, 61] [121, 146] [20, 21]
PMO RO RO RO RO RO
MOP [135] RO [105] [142] RO
Meta-
MAM [26, 150] [26, 126] [1, 150] [126] [1, 150]
Heuristics
PMO [51, 89] [89, 100] [89, 100] RO [51]
MOP [148] RO RO RO RO
Approximation
MAM [43] RO RO RO RO
Algorithms
PMO RO RO RO RO RO

2.3.1 Optimization Approaches

This section presents the optimization approaches considered in the articles studied in [94].
The identified optimization approaches may be classified as: (1) mono-objective (MOP), (2)
multi-objective solved as mono-objective (MAM) and (3) pure multi-objective (PMO). The
mentioned optimization approaches are detailed in the following subsections.

Mono-Objective (MOP)

A mono-objective optimization approach (MOP) considers the optimization of one objective

function or the individual optimization of more than one objective function, one at a time.
Research on the VMP problem has been mainly guided by the MOP approach, identifying
almost 40 different objective functions proposed considering this optimization approach [94].
It should be noted that an objective function could be studied considering different modeling
approaches (e.g. economical revenue maximization could be achieved by minimizing the total
economical penalties for SLA violations [30], by minimizing operational costs [65, 66] or even
by maximizing the total profit for leasing resources [125]).

40
Multi-Objective solved as Mono-Objective (MAM)

The optimization of multiple objective functions combined into one objective function is
considered in this work as a multi-objective problem solved as mono-objective (MAM). This
hybrid approach enables the optimization of multiple objectives functions with the disadvan-
tage that it requires a deep knowledge of the problem domain to allow a correct combination
of the objective functions, which in most cases is not possible [11]. A growing number of
articles have proposed formulations of the VMP problem with this hybrid approach [93].
Different methods could be considered for a formulation of a VMP problem with a MAM
approach such as: weighted sum, solving the problem as mono-objective while considering
the other objective functions as constraints of the problem or even proposing fuzzy logic to
provide an efficient way for combining conflicting objectives and expert knowledge [93].

Pure Multi-Objective (PMO)

A general Pure Multi-Objective Optimization (PMO) problem includes a set of p decision

variables, q objective functions, and b constraints. Objective functions and constraints are
functions of decision variables. In a PMO formulation, x represents the decision vector (or
solution), while y represents the objective vector (or solution cost). The decision space is
denoted by X and the objective space as Y . These can be expressed as [28]:

Optimize:
y = f (x) = [f1 (x), f2 (x), ..., fq (x)] (2.1)

subject to:

e(x) = [e1 (x), e2 (x), ..., eb (x)] ≥ 0 (2.2)

where:
x = [x1 , x2 , ..., xp ] ∈ X (2.3)

y = [y1 , y2 , ..., yq ] ∈ Y (2.4)

41
 The set of constrains e(x) ≥ 0 defines the set of feasible solutions Xf ⊂ X and its
corresponding set of feasible objective vectors Yf ⊂ Y . The feasible decision space Xf is the
set of all decision vectors x in the decision space X that satisfies the constraints e(x) given
by Equation (2.2). The feasible objective space Yf is the set of the objective vectors y that
represents the image of Xf onto Y . These feasible spaces are defined as:

Xf = {x | x ∈ X ∧ e(x) ≥ 0} (2.5)

Yf = {y | y = f (x) ∀x ∈ Xf } (2.6)

To compare two solutions in a pure multi-objective context, the concept of Pareto do-
minance is used. Given two feasible solutions u, v ∈ Xf , u dominates v, denoted as u  v,
if f (u) is better or equal to f (v) in every objective function and strictly better in at least
one objective function. If neither u dominates v, nor v dominates u, u and v are said to be
non-comparable (denoted as u ∼ v).
A decision vector x is non-dominated with respect to a set U , if there is no member
of U that dominates x. The set of non-dominated solutions of the whole set of feasible
solutions Xf , is known as optimal Pareto set P ∗ . The corresponding set of objective vectors
constitutes the optimal Pareto front P F ∗ .
Taking into account the large number of existing objective functions and possible approa-
ches for objective function modeling identified in [93, 94], PMO approaches could result in
more realistic formulations of a VMP problem, optimizing more than just one objective
function at a time (e.g. achieve economical revenue maximization by simultaneously mini-
mizing the total economical penalties for SLA violations, minimizing operational costs and
maximizing the profit for leasing resources). In this context, PMOs optimizing more than
three objective functions are known as Many-Objective Optimization Problems (MaOPs), as
defined in [27].
MaOPs differ significantly from PMOs because several issues should be considered when
solving optimization problems with more than three objective functions [45]. In case of
Pareto-based algorithms, these issues are intrinsically related to the fact that as the num-
ber of objective functions increases, the proportion of non-dominated solutions grows, being

42
increasingly difficult to discriminate among solutions using only the Pareto dominance rela-
tion [32]. Additionally, determining which solution to keep and which to discard in order to
converge toward the Pareto set is still a relevant issue to be addressed [45]. As the number
of objective functions grow, the proportion of non-dominated solutions to the total number
of solutions tends to one [143], making more difficult to solve a MaOP.

2.3.2 Objective Function Groups

In cloud computing datacenters, there are several criteria that can be considered when
selecting a possible solution for a VMP problem, depending on management policies and
optimization objectives. These criteria can even change from one period of time to another,
which implies a variety of possible formulations of the problem and different objectives to
be optimized.
According to [94], the VMP literature mainly focuses on the optimization of objec-
tive functions that specifically concerns CSPs (provider-oriented VMP). Objective functions
could also be formulated considering the requirements of CSTs for allocation of a particular
service or application, often composed by more than one VM (broker-oriented VMP).
It should be mentioned that in [94], nearly 60 different objective functions were identified
for the three optimization approaches presented in Section 2.3.1. Considering the large num-
ber of proposed objective functions, identified objective functions with similar characteristics
and goals were classified into 5 objective function groups that are presented in the following
subsections (see Tables 2.2 to 2.6). Additionally, considering that MAM and PMO optimi-
zation approaches may take into account any combination of each objective function group
or even different objective functions from the same objective function group, a simplified
classification is considered in this work (see Tables 2.1 and 2.10).

f1 (x) - Energy Consumption

Energy consumption management is an important studied issue in the provider-oriented

VMP literature, with high impact in operational costs and carbon dioxide emissions for
cloud datacenter operations. According to [12], most of the time, servers operate in a very

43
low energy-efficiency possible region (i.e. between 10% and 50% of resource utilization), even
thought energy efficiency is a very important issue to address, considering its economical and
ecological impact in modern datacenters. Energy consumption is the most studied objective
function for a VMP problem including different modeling approaches [94].
The most studied alternatives for modeling energy consumption include [94]: consolida-
ting VMs on the minimum number of PMs [36] and considering a linear relationship between
power consumption and Central Process Unit (CPU) utilization [13]. Joint optimization of
energy consumption and network traffic is also very studied in the VMP literature, conside-
ring that network communication equipment represents between 10% and 20% of the total
datacenter energy consumption [43].

f2 (x) - Network Traffic

As proposed in [13], network traffic is an important objective function for optimization with
open challenges in cloud computing datacenters for the provider-oriented VMP. The main
approaches for modeling network traffic include the optimization of: network communication
costs, live migration overhead, network metrics such as: delay, data access and data transfer
time, link congestion, network performance, service response time as well as average latency,
and Wide Area Network (WAN) communication in distributed-cloud deployments [93].

Table 2.2: Objective Functions - Energy Consumption Minimization Group.

Objective Function References
Datacenter Power Consumption [76]
[21, 26, 36, 37, 39, 42, 53, 62, 64, 85, 111, 121, 126,
Energy Consumption
128, 135, 141, 144, 145, 147]
Energy Efficiency [116]
Network Power Consumption [36, 42, 43, 62, 147]
Number of PMs [6, 47, 48, 57, 60, 71, 74, 103, 117, 118, 131, 148]
Power Consumption [29, 35, 36, 42, 43, 51, 62, 89, 100, 147, 149, 150]
OSI/ISO Layer 1 Power
[76]
Consumption
OSI/ISO Layer 3 Power
[76]
Consumption

44
 A very studied approach for network traffic minimization is the placement of VMs with
high communication rate in the same PM (or at least in the same rack) to avoid the utili-
zation of network resources (or at least core network equipment). This approach includes
workload characterization and clustering techniques applied to a VMP problem to minimize
the inter-VM network traffic [93]. Additionally, modeling and quantification of live migra-
tion network overhead is an important open challenge in a provider-oriented VMP network
traffic optimization context [94].
Considering that VMs are dynamically created and destroyed in cloud computing en-
vironments, a consolidation process could require high level of flexibility where traditional
routing protocols present limitations to adjust flow paths. In [145], Wang et al. proposed net-
work traffic load balancing to improve QoS in a VMP context considering Software Defined
Networking (SDN) [106], where flow paths are determined based on network status metrics
such as low delay, low packet loss or high security, just to cite a few.

f3 (x) - Economical Costs

Economical costs optimization is an important studied issue in both provider-oriented and

broker-oriented VMP literature. For a provider-oriented VMP, economical costs optimiza-
tion is a key issue to be addressed and could be achieved by reducing operational costs.
These operational costs are mainly related to energy consumption minimization but other
formulations could also be studied, such as thermal dissipation costs [150]. Reducing penalty
costs of SLA violations is another studied approach in order to maximize economical revenue
of a CSP [93]. Finally, CSPs can maximize its economical revenue by leasing all its available
resources or at least the maximum possible [93]. In this context, VMP could be studied
jointly with admission control problem [40] and two possible scenarios could be identified:
(1) if demand for resources exceeds the current available resources, overbooking techniques
or cloud federation [40] can help CSPs to attend the requirements of the CSTs; on the other
hand, (2) idle resources can be offered in an auction-based scheme such as Amazon’s Spot
Instances [2]. Both scenarios represent open challenges for the VMP problem as well as
emerging cloud computing markets.

45
 Table 2.3: Objective Functions - Network Traffic Minimization Group.
Objective Function References
Average Traffic Latency [108]
Cloud QoE (Response Time) [61]
Cloud Service Response Time [25]
Data Access [3]
Data Transfer Time [114, 152]
End-to-end Delay [42]
Link Congestion [37]
Migration Number [20, 48]
Migration Overhead [6]
Migration Time [47]
Network Cost [72, 129, 146]
Network Performance [75]
Network Traffic [33, 37, 89, 100, 126, 145, 153]
Network Utilization [52]
Node Cost [72]
Overall Communication Cost [70]
WAN Communication [26]
Worst Case Cut Load Ratio [16]

For a broker-oriented VMP, economical costs optimization is a key issue in actual cloud
markets and CSTs look for CSPs that meet the specific requirements of a particular service,
preferably with the minimal economical costs for the required cloud infrastructure.
The most studied pricing scheme in the considered VMP literature is fixed price. How-
ever, with the recent trend of dynamic pricing of cloud resources, where the price of resources
can vary depending on the free capacity and load of the CSP, few articles have proposed
formulations of a VMP problem considering dynamic prices schemes [81].

f4 (x) - Performance

Performance optimization is an important studied issue in both provider-oriented and broker-

oriented VMP literature. Performance modeling include the optimization of: availability,
CPU demand satisfaction, deployment time, QoS, resource interference, security metrics,

46
Shared Last Level Cache (SLLC) contention and total job completion time [93]. Most of
these performance metrics may be considered for CSPs to improve the QoS in a provider-
oriented VMP or for CSTs in order to select an appropriate CSP to host their services in a
broker-oriented VMP.

f5 (x) - Resource Utilization

Cloud computing datacenters are commonly composed by different types of physical and
virtual resources such as CPU, RAM, storage, network bandwidth and even in some cases
Graphical Process Unit (GPU).
The main approaches include the maximization of resource utilization, but an important
challenge to consider is the balanced utilization of each resource [93]. Li et al. studied
the concept of elasticity, referring to how well a cloud computing datacenter can satisfy
the growth of the input VMs resource demands under limitations of PMs and network link
capacities [79].
Considering the importance of efficient utilization of resources, a relevant analysis of the
anomalies and drawbacks in some existing strategies for resource utilization is presented in
[110], proposing a novel vector-based technique that solves the considered anomalies.

Table 2.4: Objective Functions - Economical Costs Optimization Group.

Objective Function References
Economical Revenue [1, 61, 89, 100, 124, 125]
Electricity Cost [78]
Operational Cost [65, 66]
Reservation Cost [23]
Server Cost [130]
SLA Violations [29, 30, 35, 77, 116, 154]
Thermal Dissipation Cost [150]
Total Infrastructure Cost [105]

47
 Table 2.5: Objective Functions - Performance Maximization Group.
Objective Function References
Availability [15, 101, 126, 146]
CPU Demand Satisfaction [20]
Deployment Plan Time [142]
Performance [34, 55, 56, 75]
QoS [121]
Resource Interference [86]
Security Metrics [22]
Shared Last Level Cache (SLLC)
[73]
Contention
Total Job Completion Time [80]

Table 2.6: Objective Functions - Resource Utilization Maximization Group.

Objective Function References
Elasticity [79]
Maximum Average Utilization [1, 54, 84]
Resource Utilization [20, 21, 67, 74, 110, 129, 131]
Resource Wastage [1, 51, 150]

2.3.3 Solution Techniques

In the VMP literature considered in [93, 94], different techniques were considered for solving
a VMP problem. The main solution techniques include: (1) deterministic algorithms, (2)
heuristic algorithms, (3) meta-heuristic algorithms, and (4) approximation algorithms. The
four mentioned solution techniques groups are explained in the following subsections and
detailed in Tables 2.7 to 2.9.

Deterministic Algorithms

Classical deterministic techniques were considered for a VMP problem, including Constraint
Programming (CP), Linear Programming (LP), Integer Linear Programming (ILP), Mixed
Integer Linear Programming (MILP), Pseudo-Boolean Optimization (PBO) and Dynamic
Programming (DP) [94]. Most of these approaches are considered for solving novel mathe-
matical formulations of a VMP problem without any practical intention, considering that

48
obtaining the optimal solution implies a search in an universe of N possible solutions [89]:

N = (n + 1)m (2.7)

where:
N: Size of the searching universe;

n: Number of physical machines;

m: Number of virtual machines.

Heuristics

Considering that VMP is a combinatorial NP-hard problem [131], it is impracticable to

optimally solve instances of the problem for large number of PMs and VMs. Commonly,
CSPs are composed by thousands to millions of PMs and VMs, a scenario where optimal
solutions with exhaustive search algorithms can result extremely expensive. Therefore, a
trade-off between quality of solutions and computational cost has to be considered for real
world cloud management systems. Heuristics have already been extensively studied in the
literature for NP-hard problems.

Table 2.7: Solution Techniques - Deterministic Algorithms

Technique References
Binary Integer Programming (BIP) [84]
Constraint Programming (CP) [15, 30]
Convex Optimization Theory [129]
Dynamic Programming (DP) [53, 148]
Integer Linear Programming (ILP) [6]
Linear Programming (LP) [48]
Matrix Transformation Algorithm [131]
Mixed Integer Linear Programming (MILP) [76]
Primal-Dual (Hungarian) Algorithm [3]
Pseudo-Boolean Optimization (PBO) [117, 118]
Relationship Rules Framework [153]
Stochastic Integer Programming (SIP) [23]
Transverse Matrix Approach [114]

49
 Table 2.8: Solution Techniques - Heuristics
Heuristic References
Based on Best Fit Decreasing (BFD) [35, 47, 145, 154]
Based on Best Fit (BF) [37, 42]
Based on First Fit (FF) [42, 67, 125]
Based on First Fit Decreasing (FFD) [6, 74]
Dominant Resource First (DRF) [74]
First-Come, First-Served (FCFS) [111]
Greedy Algorithms [16, 36, 43, 54, 70, 75, 78, 84, 146]
Heaviest First (HF) [60]
Worst Fit-based (WF) [42]
[16, 20, 21, 22, 25, 29, 33, 34, 39, 52,
55, 56, 57, 61, 62, 64, 65, 66, 71, 71, 72,
Novel Heuristics
73, 77, 79, 80, 80, 85, 86, 101, 103, 108,
110, 116, 121, 124, 128, 130, 141, 152]

Several articles proposed heuristic-based solution techniques for a VMP problem [94].
Most of the studied articles have proposed heuristics based on well-studied algorithms such
as: First-Fit (FF), First-Fit Decreasing (FFD), Best-Fit (BF), Best-Fit Decreasing (BFD),
Worst-Fit (WF) and Heaviest-Fit (HF) [93]. Other greedy algorithms were also proposed in
addition to novel heuristics proposed for a specific VMP problem resolution.

Meta-Heuristics

As mentioned before, approximations to optimal solutions are sufficient in most of cloud

infrastructure environments. Meta-heuristics are also very useful in order to obtain good
solutions in practical time. Meta-heuristics include [94]: Memetic Algorithms (MA), Particle
Swarm Optimization (PSO), Ant Colony Optimization (ACO), Genetic Algorithm (GA),
Neighborhood Search (NS), Cut-and-Search (CS), Simulated Annealing (SA) and Tabu Search
(TS) [93]. According to the proposed taxonomy, meta-heuristics are mainly studied with
multi-objective approaches (MAM and PMO) for solving VMP problems (see Table 2.1).

50
 Table 2.9: Solution Techniques - Meta-Heuristics
Meta-Heuristic References
Ant Colony Optimization (ACO) [51]
Cut-and-Search (CS) [26]
Evolutionary Algorithm (EA) [105]
Genetic Algorithm (GA) [1, 147, 150]
Memetic Algorithm (MA) [89, 100, 135]
Neighborhood Search (NS) [126, 144]
Particle Swarm Optimization (PSO) [1, 51, 150]
Simulated Annealing (SA) [142, 149]
Tabu Search (TS) [47]

Approximation Algorithms

Heuristics and meta-heuristics provide good quality solutions, but the quality of the expected
solutions is hardly measurable. In a p-approximation algorithm, the value of a solution will
not be more (or less) than a factor p times the optimum solution. A small number of articles
proposed approximation algorithms for solving a VMP problem [94]. It is worth noting that
for cloud infrastructures, solutions obtained using heuristics or meta-heuristics techniques
are sufficiently good for most practical cases.

2.4 VMP Research Opportunities

Based on an universe of 84 studied publications systematically chosen [93], Table 2.10 su-
mmarizes the taxonomies presented in this work (see Figure 2.3 and Table 2.1). Considering
that 5 studied articles [6, 43, 47, 84, 148] proposed 2 different solution techniques for the
same VMP formulation, statistics are based on 89 different VMP formulations.
Considering a detailed analysis of the information provided by Table 2.10, this section
details research opportunities in the context of the present doctoral thesis in order to fur-
ther advance this very active research area. It should be noted that presented statistics are
defined following the studied VMP literature (of 89 different VMP formulations) and consi-
dering only as a simple base reference given that more articles may be considered and may
rapidly be published increasing this number. Additionally, different systematic reviews of the

51
Table 2.10: Virtual Machine Placement Taxonomy. Elements of column % represent the
percentage of articles in the studied universe [93]. For simplicity, MAM and PMO consider
only the number of considered objective functions in column f (x).
Deployment Optimization Solution
Oriented Formulation f (x) %
Architecture Approach Technique
Heuristic 3.37%
f1 (x)
Meta-Heuristic 2.25%
MOP
f2 (x) Heuristic 2.25%
f4 (x) Heuristic 1.12%
Offline Heuristic 2.25%
3
MAM Meta-Heuristic 3.37%
2 Deterministic 1.12%
3 Meta-Heuristic 2.25%
PMO
2 Meta-Heuristic 1.12%
Deterministic 4.49%
Heuristic 6.74%
f1 (x)
Meta-Heuristic 1.12%
Approximation 1.12%
Single-cloud Deterministic 2.25%
f2 (x)
Heuristic 5.62%
MOP
Deterministic 1.12%
f3 (x)
Heuristic 5.62%
Provider
Heuristic 7.87%
Online f4 (x)
Meta-Heuristic 1.12%
Deterministic 1.12%
f5 (x)
Heuristic 4.49%
Heuristic 3.37%
3
Meta-Heuristic 1.12%
Deterministic 3.37%
MAM
Heuristic 16.85%
2
Meta-Heuristic 1.12%
Approximation 1.12%
Offline MOP f4 (x) Deterministic 1.12%
Heuristic 1.12%
f2 (x)
Deterministic 1.12%
MOP
Distributed-cloud f3 (x) Heuristic 1.12%
Online
f5 (x) Heuristic 1.12%
3 Deterministic 1.12%
MAM
2 Meta-Heuristic 1.12%
Federated-cloud Online MOP f1 (x) Heuristic 1.12%
Deterministic 1.12%
Broker Multi-cloud Offline MOP f3 (x)
Meta-Heuristic 1.12%
Total 100%

VMP literature (e.g. considering different keywords) or extended systematic reviews (e.g.
considering additional keywords) could result in different statistics. Taking into account the
diversity of existing environments, formulations and solution techniques (see Table 2.10),
it is important to provide a general vision on the main approaches for the VMP problem,

52
enabling interested readers to easily define specific research alternatives. Consequently, the
following subsections describe research opportunities identified in this work as a main result
of the proposed VMP taxonomies (see Figure 2.3, Table 2.1 and Table 2.10).

2.4.1 Unexplored Environments, Formulations and Solution Tech-

niques

Considering the studied VMP literature [93], several unexplored VMP environments were
identified (see Figure 2.3) as described as follows.
First, provider-oriented VMP problems in federated-cloud deployments were not con-
sidered with offline formulations. Second, broker-oriented VMP problems in multi-cloud
deployments were not considered with online formulations. Consequently, no formulations
or solution techniques were studied for the mentioned unexplored VMP environments.
Additionally, unexplored formulations and solutions techniques were also identified in
this work (see Table 2.1), as briefly analyzed next.
For MOP optimization approach, f2 (x) (network traffic) and f5 (x) (resource utilization)
were not studied considering meta-heuristics as solution technique. Additionally, approxi-
mation algorithms were studied as a solution technique only for f1 (x) (energy consumption),
representing unexplored formulations with the remaining studied objective functions.
For MAM optimization approach, the resolution of VMP formulations considering f3 (x)
(economical revenue) and f4 (x) (performance) with deterministic algorithms were not stu-
died. Similar to MOP, approximation algorithms were studied as a solution technique only
for f1 (x) (energy consumption), representing unexplored formulations with the remaining
studied objective functions (see Table 2.1).
For PMO optimization approach, f4 (x) (performance) were not studied considering meta-
heuristics as solution technique, while neither deterministic algorithms, heuristics nor appro-
ximation algorithms studied PMO formulations of the VMP problem, representing unex-
plored formulations and solution techniques (see Table 2.1).
Taking into account a complete understanding of the VMP problem composed by VMP
environments and VMP formulations and solution techniques (see Table 2.10), several un-

53
explored VMP problems could also be identified. In this context, Table 2.10 could guide
interested readers to identify existing research on VMP problem according to the studied
VMP literature [93]. It should be noted that all possible VMP problems that are not presen-
ted in Table 2.10 could be considered research opportunities. As an example, MAM and
PMO optimization approaches are not presented in Table 2.10 for the following VMP en-
vironments: (1) provider-oriented VMP in distributed-clouds with offline formulations, (2)
provider-oriented VMP in federated-clouds with online formulations and (3) broker-oriented
VMP in multi-clouds with offline formulations.

2.4.2 Broker-oriented VMP considering Online Formulations

According to the proposed VMP Environment Taxonomy (see Figure 2.3), a broker-oriented
VMP problem could be studied in a multi-cloud deployment architecture considering offline
or online formulations. A broker-oriented VMP problem could also be studied considering a
mono-objective approach (MOP) or multi-objective approaches (MAM or PMO), optimizing
different objective functions (see Table 2.1).
Considering that only 2.24% of the considered VMP literature studied a broker-oriented
VMP problem as an offline optimization problem (see Table 2.10), exploring broker-oriented
VMP formulations could be considered a as research opportunity. In this context, an exten-
ded systematic review of research articles demonstrated that broker-oriented VMP problems
include several challenges, where novel dynamic multi-objective formulations were already
proposed by the author in research work not included in this doctoral thesis [24].
Several dynamic parameters could be studied for online broker-oriented VMP problems
such as: (1) pricing schemes, (2) VM offers or (3) user requirements; therefore, a detailed
survey on these parameters and formulations is suggested. It should be noted that the
implementation of possible migrations of VMs across different CSPs is limited by cloud
interoperability factors and it is still out of the scope of this work.

54
2.4.3 Provider-oriented VMP considering Online Formulations

Considering the on-demand model of cloud computing, a provider-oriented VMP problem

should be solved online to efficiently attend typical dynamic workload of modern cloud
applications. According to the studied articles, 77.53% considered this particular type of
VMP problem with several different formulations and dynamic parameters.
Research work by the author not included in this doctoral thesis [112] studied articles
of the provider-oriented VMP literature in order to understand possible challenges for CSPs
in cloud computing environments, based on the most relevant dynamic parameters studied
so far in the VMP literature. Obtained results identified that resource capacities of VMs
(associated to vertical elasticity), number of VMs of a cloud service (associated to horizontal
elasticity) and utilization of resources of VMs (related to overbooking) are the most relevant
dynamic parameters in the specialized literature. Consequently, dynamic IaaS environments
for online formulations of the provider-oriented VMP problem could be classified by one or
more of the following classification criteria: (1) elasticity and (2) overbooking.
According to results obtained so far, dynamic IaaS environments could be formulated
considering one of the following elasticity values: no elasticity, horizontal elasticity, ver-
tical elasticity or both horizontal and vertical elasticity. Additionally, identified dynamic
environments may also consider one of the following overbooking values: no overbooking,
server resources overbooking, network resources overbooking or both server and network
overbooking. Research opportunities for online formulations of a provider-oriented VMP in-
clude detailed studies on complex dynamic environments (e.g. VMP considering both types
of elasticity and both types of overbooking) in order to enable CSPs to efficiently attend
modern cloud computing applications and services.
It is important to mention that no studied article considered uncertainty for this par-
ticular VMP formulation before this doctoral thesis; therefore, modeling VMP problems for
cloud computing environments under uncertainty was considered in this thesis [99].

55
2.4.4 Provider-oriented VMP considering PMO Optimization
According to the studied articles, only 3.37% of them considered a PMO approach for an
offline formulation of provider-oriented VMP problems simultaneously optimizing 2 or 3
objective functions. PMO approaches could result in more realistic formulations of a VMP
problem, optimizing more than just one objective function at a time.
Taking into account that existing PMO formulations of a provider-oriented VMP problem
consider at most 3 objective functions and more than 60 different objective functions were
identified in [94], several formulations could still be considered, specially for PMO approa-
ches. In this context, it is important to remember that PMOs optimizing more than three
objective functions are known as MaOPs. As described in Section 2.3.1, several issues should
be considered when solving optimization problems with more than three objective functions
[45]. Considering the mentioned challenges for solving MaOPs, López-Pires and Barán have
proposed in [92] a general many-objective optimization framework that is able to consider
as many objective functions as needed when solving a VMP problem for initial placement
(static) in a PMO context. In order to converge to a treatable number of non-dominated
solutions, the authors proposed the utilization of interactive lower and upper bounds asso-
ciated to each objective function to reduce the number of possible solutions of the Pareto
set approximation Pknown , when needed by the decision maker.
Additionally, semi-dynamic (offline) formulations of a provider-oriented VMP considering
a PMO approach should be explored. For this particular type of VMP problem, PMO
approaches should include strategies for an appropriate solution selection from the Pareto
set approximation Pknown , composed by non-dominated solutions. In this context, Ihara
et al. already proposed a first formulation of a Many-Objective VMP problem (MaVMP)
with reconfiguration of VMs and considering an evaluation of several automatic selection
strategies for provider-oriented VMP problems formulated as MaOPs [68].

2.4.5 Provider-oriented VMP in Distributed and Federated Clouds

According to the studied articles, only 7.87% proposed a provider-oriented VMP problem
in a distributed-cloud deployment architecture, while only [39] studied the provider-oriented
VMP problem considering a federated-cloud deployment architecture (see Table 2.10).

56
 Resource allocation in distributed and federated cloud environments is an active research
area [59]. In a provider-oriented VMP context, several particular constraints and objective
functions may still be studied and evaluated in order to fulfill the requirements of a CSP
with geo-distributed cloud computing datacenters or different CSPs in a cloud federation.
It should be noted that a broker-oriented VMP in a distributed-cloud deployment ar-
chitecture may be adapted as a multi-cloud deployment architecture by considering each
datacenter as a different CSP in order to fulfill requirements of high-availability or legal
issues (e.g. Amazon’s us-east-1, us-west-2 or eu-west-1 datacenters)1 .
For a VMP problem in a distributed or federated-cloud deployment architecture, several
research opportunities may be still proposed considering unexplored objective functions,
novel formulations in MOP, MAM or PMO approaches or even experimental evaluation of
different solution techniques.

1
https://aws.amazon.com

57
 Part II

MaVMP for Initial Placement of VMs

(static)

58
3 MaVMP for Virtualized Datacenters
As presented in Section 2.4, provider-oriented VMP problems considering PMO optimization
represent a research challenge for the resource allocation in cloud datacenters. It is worth
remembering that no many-objective formulation had already been proposed for the VMP
problem in the specialized literature [94, 96] before our work was published in [92].
This chapter presents a general many-objective optimization framework which is able to
consider as many objective functions as needed when solving the VMP problem for initial
placement. As an example of utilization of the proposed framework, a formulation of a many-
objective VMP problem (MaVMP) is proposed, considering the simultaneous optimization
of the following five objective functions: (1) power consumption, (2) network traffic, (3)
economical revenue, (4) quality of service (QoS) and (5) network load balancing.
In the presented MaVMP formulation, a multi-level priority is associated to each VM, re-
presenting a Service Level Agreement (SLA) considered in the placement process in order to
effectively prioritize important VMs (e.g. in peaks situations where the total requested VMs
resources are higher than available PMs resources). To solve the formulated MaVMP pro-
blem, an interactive Memetic Algorithm (MA) is proposed considering particular challenges
associated to the resolution of a VMP problem in a many-objective context.

3.1 Many-Objective Optimization Framework

The general many-objective optimization framework for the VMP problem proposed in this
work considers that as the number of conflicting objectives of a MaVMP problem formulation
increases, the total number of non-dominated solutions increases (even exponentially in some
cases), being increasingly difficult to discriminate among solutions using only the dominance
relation [45]. For this reason, this work proposes the utilization of lower and upper bounds
associated to each objective function fz (x), where z ∈ {1, . . . , q} (Lz ≤ fz (x) ≤ Uz ), to
be able to reduce iteratively the number of possible non-dominated solutions of the known
approximation to the Pareto set, represented as Pknown .

59
 A MaVMP formulation, based on many objective functions and several constraints to be
detailed in Section 3.2, may be written as:
Optimize:
y = f (x) = [f1 (x), f2 (x), f3 (x), . . . , fq (x)], q>3 (3.1)

where for example:

f1 (x) = power consumption;

f2 (x) = network traffic;

f3 (x) = economical revenue;

f4 (x) = quality of service; (3.2)

f5 (x) = network load balancing;

..
.

fq (x) = any other considered objective function.

subject to constraints as:

e1 (x) : unique placement of VMs;

e2 (x) : assure provisioning of highest SLA;

e3 (x) : processing resource capacity of PMs;

e4 (x) : memory resource capacity of PMs;

e5 (x) : storage resource capacity of PMs;

e6 (x) : f1 (x) ∈ [L1 , U1 ];

(3.3)
e7 (x) : f2 (x) ∈ [L2 , U2 ];

e8 (x) : f3 (x) ∈ [L3 , U3 ];

e9 (x) : f4 (x) ∈ [L4 , U4 ];

e10 (x) : f5 (x) ∈ [L5 , U5 ];

..
.

er (x) : any other considered constraint.

60
3.2 Part 2 - Problem Formulation

A few articles proposed formulations of a pure multi-objective VMP problem (MVMP),

considering the simultaneous optimization of at most three objective functions [51, 89]. A
previous work of the authors proposed for the first time a MaVMP formulation [92]. This
work extends [92] considering the many-objective optimization framework presented in Sec-
tion 3.1. In fact, it proposes a formulation of the MaVMP problem considering the following
five objective functions to be simultaneously optimized: (1) power consumption, (2) network
traffic, (3) economical revenue, (4) quality of service and (5) network load balancing. In the
presented MaVMP formulation, a multi-level priority is associated to each VM, considered
in the placement process in order to effectively prioritize VMs. Formally, the proposed offline
(static) MaVMP problem for initial placement can be enunciated as:

Active paths of M Total traffic per path

m13 = {l1, l5, l6, l3} m13 = 4 Mbps
m14 = {l1, l5, l6, l4} m14 = 2 Mbps
m34 = {l3, l4} m34 = 4 Mbps

l5 l6

1000 Mbps 1000 Mbps

100 Mbps 100 Mbps 100 Mbps 100 Mbps

V3 l1 l2 V1 V2 l3 l4 V4

H1 H2 H3 H4

Figure 3.1: Example of placement in a virtualized datacenter infrastructure, composed by

PMs, VMs and a network topology.

61
Given a set of PMs, H = {H1 , H2 , ..., Hn }, a network topology G (as illustrated in Figure
3.1) and a set of VMs, V = {V1 , V2 , ..., Vm }, it is sought a correct placement of the set of VMs
V into the set of PMs H satisfying the b constraints of the problem and simultaneously opti-
mizing all q objective functions defined in this formulation (as energy consumption, network
traffic, economical revenue, QoS and load balancing in the network), in a pure many-objective
context.

3.2.1 Part 2 - Input Data

The proposed formulation of the MaVMP problem models a virtualized datacenter infras-
tructure, composed by PMs, VMs and a network topology that interconnects PMs. The set
of PMs is represented as a matrix H ∈ Rn×4 .
Each PM Hi is represented by processing resources of CPU (as ECU)1 , RAM memory
[GB], storage [GB] and a maximum power consumption [W] as:

Hi = [Hcpui , Hrami , Hhddi , pmaxi ], ∀i ∈ {1, ..., n} (3.4)

where:
Hcpui : Processing resources of Hi ;

Hrami : RAM memory resources of Hi ;

Hhddi : Storage resources of Hi ;

pmaxi : Maximum power consumption of Hi ;

n: Number of PMs.

It should be mentioned that the proposed notation is general enough to include additional
characteristics associated to each PM such as Graphic Processing Units (GPUs) or Network
Interface Cards (NICs) just to cite a few.
As shown in the example of Figure 3.1, a network topology of a virtualized datacenter is
represented as:
1
http://aws.amazon.com/ec2/faqs

62
 G: Network topology;

L: Set of links la in G. For simplicity, links are assumed as semi-duplex in

what follows;

M: Set of paths for all-to-all PM network interconnections;

K: Capacity set of the communication channels, typically in [Mbps].

The set of VMs requested by customers is represented as a matrix V ∈ Rm×5 . Each VM

Vj requires processing resources of CPU (as ECU)1 , RAM memory [GB] and storage [GB],
providing for them an economical revenue Rj [$] for the provider.
A SLA is also assigned to each VM to indicate its priority level. Consequently, a Vj is
represented as:

Vj = [V cpuj , V ramj , V hddj , Rj , SLAj ], ∀j ∈ {1, ..., m} (3.5)

where:
V cpuj : Processing requirements of Vj ;
V ramj : Memory requirements of Vj ;
V hddj : Storage requirements of Vj ;
Rj : Economical revenue for locating Vj ;
SLAj : Service Level Agreement SLAj of a Vj . If the highest priority level is s,
then SLAj ∈ {1, . . . , s};
m: Number of VMs.

The traffic between VMs is represented as a matrix T ∈ Rm×m . Each Vj requires net-
work communication resources [Mbps] to communicate with other VMs. The network traffic
between requested VMs is represented as:

Tj = [Tj,1 , Tj,2 , ..., Tj,m ], ∀j ∈ {1, ..., m} (3.6)

where:
Tj,k : Average network traffic between Vj and Vk [Mbps]. Note that it is con-
sidered that Tj,j = 0.

63
Figure 3.1 presents a basic example of a virtualized datacenter infrastructure, composed by
4 PMs (H = {H1 , H2 , H3 , H4 }) and a network topology considering 6 physical network links
(L = {l1 , l2 , l3 , l4 , l5 , l6 }). In this example, the set of capacity for each communication channel
is K = {100, 100, 100, 100, 1000, 1000} [Mbps] respectively. Using shortest path, a path m1,2
between H1 and H2 uses links {l1 , l2 }, i.e. m1,2 = {l1 , l2 }. Analogously, m1,3 = {l1 , l5 , l6 , l3 }
and m1,4 = {l1 , l5 , l6 , l4 }, as shown in Figure 3.1. All 4 requested VMs are correctly located
into one of the available PMs.

3.2.2 Part 2 - Output Data

A calculated solution x should indicate a complete placement of each VM Vj into the ne-
cessary PMs Hi , considering the many-objective optimization criteria applied. A placement
(or possible solution x to the formulated problem) is represented in what follows as a matrix
x = {xj,i } of dimension (m x n), where xj,i ∈ {0, 1} indicates if Vj is located (xj,i = 1) or
not (xj,i = 0) for execution on a PM Hi (i.e. xj,i : Vj → Hi ).

3.2.3 Part 2 - Constraints

Constraint 1: unique placement of VMs

A VM Vj should be located to run on a single PM Hi or alternatively, it could be not located

into any PM if the associated SLA is not the highest level of priority s (i.e. SLAj < s).
Consequently, this constraint is expressed as:

n
X
xj,i ≤ 1 ∀j ∈ {1, ..., m} (3.7)
i=1

where:

xj,i : Binary variable equals 1 if Vj is located on Hi ; otherwise, it is 0.

64
Constraint 2: assure SLA provisioning

A VM Vj with the highest level of SLA (i.e. SLAj = s) must be mandatorily located to run
on a PM Hi . Consequently, this constraint is expressed as:

n
X
xj,i = 1 ∀j such that SLAj = s (3.8)
i=1

Constraints 3-5: physical resources capacities of PMs

A PM Hi must have sufficient available resources to meet the requirements of all VMs Vj that
are located to run on Hi . In this chapter, it is not considered the overbooking of resources
[138]. Consequently, this set of constraints can be mathematically formulated as:

m
X
V cpuj × xj,i ≤ Hcpui (3.9)
j=1

m
X
V ramj × xj,i ≤ Hrami (3.10)
j=1

m
X
V hddj × xj,i ≤ Hhddi (3.11)
j=1

∀i ∈ {1, ..., n}, i.e. for all physical machine Hi .

Adjustable constraints

This work proposes the utilization of lower and upper bounds associated to each objective
function to reduce the number of possible solutions of the Pareto set approximation Pknown ,
when needed by a decision maker. Consequently, this set of adjustable bounds can be
mathematically formulated as the following constraints:

fz (x) ∈ [Lz , Uz ], ∀z ∈ {1, . . . , q} (3.12)

65
3.2.4 Part 2 - Objective Functions

A VMP problem can be defined as a many-objective optimization problem, considering the

simultaneous optimization of more than three objective functions. As a concrete example,
this chapter proposes the simultaneous optimization of the following five objective functions:

Power Consumption Minimization

Based on [13] formulation, this work also proposes the minimization of power consumption,
represented by the sum of the power consumption of each PM Hi :

n
X
f1 (x) = ((pmaxi − pmini ) × U cpui + pmini ) × Yi (3.13)
i=1

where:
f1 (x): Total power consumption of the PMs;
pmini : Minimum power consumption of Hi . It should be noted that pmini ≈
pmaxi × 0.6 according to [13];
U cpui : Utilization ratio of processing resources used by Hi ;
Yi : Binary variable equals 1 if Hi is turned on; otherwise, it is 0.

Inter-VM Network Traffic Minimization

Shrivastava et al. proposed in [127] the minimization of network traffic among VMs by
maximizing locality. Based on this approach, this work proposes Equation (3.14) to estimate
network traffic represented by the sum of average network traffic generated by each VM Vj ,
that is located to run on any PM, with other VMs Vk that are located to run on different
PMs.

m X
X m
f2 (x) = (Tj,k × Dj,k ) (3.14)
j=1 k=1

where:

66
 f2 (x): Total network traffic among VMs;
Tj,k : Average network traffic between Vj and Vk [Mbps]. Note that it is con-
sidered that Tj,j = 0;
Dj,k : Binary variable that equals 1 if Vj and Vk are located in different PMs;
otherwise, it is 0.

The traffic between two VMs Vj and Vk which are located on the same PM Hi do not
contribute to increase the total network traffic given by Equation (3.14); therefore, Dj,k = 0
if xj,i = xk,i = 1.

Economical Revenue Maximization

Based on [89], this work presents Equation (3.15) to estimate the total economical revenue
that a datacenter receives for meeting the requirements of its customers, represented by
the sum of the economical revenue obtainable by each VM Vj placement that is effectively
located for execution on any PM.

m
X
f3 (x) = (Rj × Xj ) (3.15)
j=1

where:
f3 (x): Total economical revenue for placing VMs;
Xj : Binary variable that equals 1 if Vj is located for execution on any PM;
otherwise, it is 0.

QoS Maximization

In this work, the QoS maximization proposes to locate the maximum number of VMs with
the highest level of priority associated to the SLA. This objective function is formulated in
Equation (3.16).

m
(Ĉ SLAj × SLAj × Xj )
X
f4 (x) = (3.16)
j=1

67
where:
f4 (x): Total QoS figure for a given placement;
Ĉ: Constant, large enough to prioritize services with a larger SLA over the
ones with a lower SLA.

Network Load Balancing Optimization

This work calculates the total amount of network traffic going through a semi-duplex link la
as:

 
n X
X n m X
X m
Tla = Fa,i,i0 ×  xj,i × xj 0 ,i0 × Dj,j 0 × Tj,j 0  (3.17)
i=1 i0 =1 j=1 j 0 =1

where:
T la : Total amount of traffic going through link la [Mbps];
mi,i0 : Network path between Hi and Hi0 ;
Fa,i,i0 : Binary variable that equals 1 if la ∈ mi,i0 ; otherwise, it is 0.

Inspired in the formulation presented in [38], this work calculates the Maximum Link Utili-
zation (MLU) as:

!
T la
M LU = max (3.18)
∀la ∈L Cla

where:
M LU : Maximum Link Utilization;
Cla : Channel capacity of link la [Mbps].

In this paper, the load balancing optimization of the network is formulated as the mini-
mization of the MLU, denoted as:

f5 (x) = M LU (3.19)

68
3.2.5 Basic Example

The following example details how f4 (x) and f5 (x) are calculated, considering the datacenter
infrastructure and placement presented in Figure 3.1. Interested readers can refer to [89] for
a detailed example of how f1 (x), f2 (x) and f3 (x) are calculated.
Consider the placement presented in Figure 3.1, mathematically represented by the fo-
llowing placement matrix x where, VMs V1 and V2 are executed in H3 , while V3 is executed
 
in H1 and V4 in H4 :
0 0 1 0
 
 
0 0 1 0
 
x= 


 (3.20)
1 0 0 0
 
 
 
0 0 0 1

Additionally, Ĉ = 100, SLA1 = 2, SLA2 = 2, SLA3 = 2 and SLA4 = 3. Considering

that all VMs Vj where located for its execution, as shown in Equation (3.20), all binary
variables Xj = 1. Consequently, f4 (x) is calculated as:

f4 (x) = Ĉ SLA1 × SLA1 × X1 + · · · + Ĉ SLA4 × SLA4 × X4

= 1002 × 2 × 1 + · · · + 1003 × 3 × 1 (3.21)

= 3.06x106

It is important to note that if Ĉ is not sufficiently large, f4 (x) could prefer a large number
of VMs with lower priority. As an example, if Ĉ = 1, 2 VMs with SLAj = 2 result in a
better value of f4 (x) than 1 VM with SLAj = 3, what is not correct according to what was
presented in Section 3.2.4 as the main concept for using SLA.

Table 3.1: Data of network links used in basic example (see Section 3.2.5).
la T la [Mbps] Cla [Mbps] T la /Cla [Mbps]
l1 6 100 6/100
l2 0 100 0/100
l3 8 100 8/100
l4 6 100 6/100
l5 6 1000 6/1000
l6 6 1000 6/1000
MLU 8/100

69
 Additionally, considering that each VM Vj communicates at 1 [Mbps] with any other
Vk (k 6= j), the placement x results in the values of T la and Cla presented in Table 3.1.
Consequently, f5 (x) is calculated as:

!
T l1 T l2 T l3 T l4 T l5 T l6
f5 (x) = max , , , , ,
Cl1 Cl2 Cl3 Cl4 Cl5 Cl6
6 0 8 6 6 6
 
= max , , , , , (3.22)
100 100 100 100 1000 1000
8
=
100

3.3 Interactive Memetic Algorithm for MaVMP

A Memetic Algorithm (MA) could be understood as an Evolutionary Algorithm (EA) that

in addition to the standard selection, crossover and mutation operators of most Genetic
Algorithms (GAs) includes a local optimization operator to obtain good solutions even at
early generations of an EA [9]. In the VMP context, it is valuable to obtain good quality
of solutions in short time. Consequently, a MA could be considered as a promising solution
technique for VMP problems.
This work proposes an interactive MA for solving the VMP problem in a many-objective
context, considering the proposed formulation presented in Section 3.2 to simultaneously
optimize the five objective functions presented in the previous section. The proposed al-
gorithm is extensible to consider as many objective functions as needed while only minor
modifications may be needed if the number of objective functions changes.
It was shown in [143] that many-objective optimization using Multi-Objective Evolutio-
nary Algorithms (MOEAs) is an active research area, having multiple challenges that need
to be addressed. The interactive MA presented in this section is a viable way to solve a
MaVMP problem, including desirable ranges of values for the objective functions costs in
order to interactively control the possible huge number of feasible non-dominated solution,
as described in Section 2.3.1. The proposed interactive MA (see Algorithm 1) is based on
the MA proposed in [89] and works as follows:

70
 At step 1, it is verified if the problem has at least one solution (considering only VMs
with SLAj = s) to continue with next steps. If there is no possible solution to the problem,
the algorithm returns an appropriate error message. If the problem has at least one solution,
the algorithm proceeds to step 2, which generates a set of random candidates P op0 , whose
solutions are repaired at step 3 to ensure that P op0 contains only feasible solutions. Then,
the algorithm tries to improve candidates at step 4 using local search. With the obtained
non-dominated solutions, the first Pareto set approximation Pknown is generated at step 5.
After initialization at step 6, evolution begins (iterations between steps 7 and 18). The
evolutionary process basically follows the same behavior: solutions are selected from the
union of Pknown with the evolutionary set of solutions (or population) also known as P opt
(step 8), crossover and mutation operators are applied as usual (step 9), and eventually
solutions are repaired, as there may be infeasible solutions (step 10). Improvements of
solutions of the evolutionary population P opt may be generated at step 11 using local search
(local optimization operators). At step 12, the Pareto set approximation Pknown is updated
(if applicable); while at step 13 the generation (or iteration) counter is updated. At step
15 the decision maker adjust the lower and upper bounds if it is necessary, while at step 17
a new evolutionary population P opt is selected. The evolutionary process is repeated until
the algorithm meets a stopping criterion (such as maximum number of generations), finally
returning the set of non-dominated solutions Pknown at step 19.

3.3.1 Part 2 - Population Initialization

Initially, a set of solutions P op0 is randomly generated. Each possible solution (or individual)
x is represented as a chromosome C = [C1 , C2 , . . . , Cm ] (matrix x in Section 3.2.2). The
possible values that can take each Ck for VMs with the highest value of SLAj (SLAj = s)
are in the range [1, n]. On the other hand, for VMs Vj with SLAj < s, the possible values
that can take each Ck are in the range [0, n]. Within these ranges defined by the SLAj
of each Vj , the algorithm ensures that all VMs Vj with the highest level of priority will be
located for execution on a PM Hi , while for VMs Vj with lower levels of priority SLAj , there
is always a probability larger than 0 that they may not be located for execution in any PM.

71
 Algorithm 1: Interactive Memetic Algorithm
Data: datacenter infrastructure (see Section 3.2.1)
Result: Pareto set approximation Pknown
1 check if the problem has a solution
2 initialize set of solutions P op0
3 P op00 = repair infeasible solutions of P op0
4 P op000 = apply local search to solutions of P op00
5 update set of solutions Pknown from P op000
6 t = 0; P opt = P op000
7 while stopping criterion is not met do
8 Qt = selection of solutions from P opt ∪ Pknown
9 Q0t = crossover and mutation of solutions of Qt
10 Q00t = repair infeasible solutions of Q0t
11 Q000
t = apply local search to solutions of Qt
00

12 update set of solutions Pknown from Qt 000

13 increment t
14 if interaction is needed then
15 ask for decision maker modification of (Lz and Uz )
16 end
17 P opt = non-dominated sorting from P opt ∪ Q000 t
18 end
19 return Pareto set approximation Pknown

3.3.2 Part 2 - Infeasible Solution Reparation

With a random generation at the initialization phase (step 2 of Algorithm 1) and/or solutions
generated by genetic operators (step 9 of Algorithm 1), infeasible solutions may appear,
i.e. the resources required by the VMs located for execution on certain PMs could exceed
available resources, or at least one objective function may not meet adjustable constraints.
Repairing these infeasible solutions (steps 3 and 10 of Algorithm 1) may be done in two
stages: first, in the feasibility verification process, the population is classified in two classes:
feasible or infeasible (Algorithm 2). Next, in the process of repairing infeasible solutions
(Algorithm 3), the solutions which do not meet the feasibility criteria are repaired in three
ways: (1) migrating some VMs to an available hardware, (2) turning on some PMs and then
migrating VMs to them, or (3) turning off some VMs with SLAj 6= s. Note that at step 3
of Algorithm 3, Vj migration to Hi0 can be done to other PMs, even if they are shutdown.

72
 Algorithm 2: Part 2 - Feasibility Verification
Data: set of solutions P opt
Result: set of feasible solutions P op0t
1 while there are solutions not verified do
2 feasible = true ; i = 1
3 while i ≤ n and feasible = true do
4 if solution does not satisfy constraints (3-5) then
5 feasible = false ; break
6 else
7 increment i
8 end
9 end
10 if feasible = false then
11 call Algorithm 3 (repair solution)
12 end
13 end
14 return set of feasible solutions P op0t

3.3.3 Part 2 - Local Search

Once a population contains only feasible solutions, a local search is performed (steps 4 and
11 of Algorithm 1) improving solutions found so far in the evolutionary population P opt .
The local search pseudo-code is presented in Algorithm 4.
For each individual in the evolutionary population P opt , the proposed interactive memetic
algorithm attempts to optimize a solution with a local search (step 2 of Algorithm 4). For
this purpose, with probability 21 , the algorithm tries to maximize the number of located VMs
with higher level of priority, locating all possible VMs that were not located so far, directly
increasing f3 (x) (total economical revenue) and f4 (x) (total quality of service) (steps 3 to 5
of Algorithm 4).
On the other hand, also with probability 21 , the algorithm tries to minimize the number of
PMs turned on, directly reducing f1 (x) (total power consumption) (steps 6 to 8 of Algorithm
4). With the proposed probabilistic local search method, a balanced exploitation of objective
functions (economical revenue, quality of service and power consumption) is achieved, as
experimentally verified with results presented in next section.

73
 Algorithm 3: Part 2 - Infeasible Solutions Reparation
Data: infeasible solution
Result: feasible solution
1 feasible = false ; j = 1
2 while j ≤ m and feasible = false do
3 if it is possible then
4 migrate Vj to Hi0 (i0 6= i)
5 else
6 if SLAj 6= s then
7 turn off Vj on Hi
8 else
9 replace solution with another solution from P opknown
10 end
11 end
12 end
13 return feasible solution

Algorithm 4: Part 2 - Local Search

Data: set of feasible solutions P op0t
Result: set of feasible optimized solutions P op00t
1 probability = random number between 0 and 1
2 while there are solutions not verified do
3 if probability < 0.5 then
4 Try to turn off all the possible Hi by migrating all the Vj assigned to Hi0 with
available resources (i0 6= i) and then try to turn on all the possible Vj (using SLAj
priority order) assigning them to a Hi with available resources
5 else
6 Try to turn on all the possible Vj (using SLAj priority order) assigning them to a
Hi with available resources and then try to turn off all the possible Hi by
migrating all the Vj assigned to Hi0 with available resources (i0 6= i)
7 end
8 end
9 return set of feasible optimized solutions P op00t

3.3.4 Part 2 - Fitness Function

The fitness function used in the proposed algorithm is the one presented in [31]. This fitness
value defines a non-domination rank in which a value equal to its Pareto dominance level (1
is the highest level of dominance, 2 is the next, and so on) is assigned to each individual of
the population. Between two individuals with different non-domination rank, the individual
with lower value (higher level of dominance) is considered better.
To compare individuals with the same non-domination rank, a crowding distance is used.
The basic idea in crowding distance is to find the Euclidean distance (properly normalized

74
when the objectives have different measure units) between each pair of individuals, based
on the q objectives, in a hyper-dimensional space [31]. The individual with larger crowding
distance is considered better.

3.3.5 Part 2 - Variation Operators

The proposed interactive MA uses a Binary Tournament method for selecting individuals
for crossover and mutation [28]. The crossover operator used in the presented work is the
single point cross-cut [28]. The selected individuals in the ascending population are replaced
by descendants individuals.
1
This work uses a mutation method in which each gene is mutated with a probability m
,
where m represents the number of VMs. This method offers the possibility of full uniform
gene mutation, with a very low probability (but larger than zero), which is beneficial to the
exploration of the search space, reducing the probability of stagnation in a local optimum.
The population evolution in the proposed interactive MA is based on the population evo-
lution proposed in [31]. A population P opt+1 is formed from the union of the best known
population P opt and offspring population Qt , applying non-domination rank and crowding
distance.

3.3.6 Many-Objective Considerations

Given that the number of non-dominated solutions may rapidly increase, an interactive
approach is recommended. That way, a decision maker can introduce new constraints or ad-
just existing ones, while the execution continues learning about the shape of the Pareto front
in the process. For simplicity, the present work considers lower and upper bounds associated
to each objective function in order to help the decision maker to reduce interactively the
potential huge number of solutions in the Pareto set approximation Pknown , while observing
the evolution of its corresponding Pareto front P Fknown to the region of his preference.

75
3.4 Part 2 - Experimental Results

This section summarizes experimental results obtained by the proposed algorithm in carefully
designed experiments to validate its effectiveness considering challenges associated to the
resolution of a MaVMP problem previously introduced in Section 2.3.1.
First, Experiment 1 performed an evaluation of the quality of solutions obtained by the
proposed algorithm against optimal solutions obtained with an exhaustive search algorithm
in two different scenarios. Next, Experiment 2 performed an evaluation of using lower
and upper bounds associated to each objective function f (z) (Lz ≤ fz (x) ≤ Uz ) to be
able to converge to a manageable number of solutions in the Pareto set approximation.
Finally, Experiment 3 evaluates the proposed algorithm solving instances of the problem
with large numbers of PMs and VMs. For simplicity, all experiments considered a datacenter
infrastructure composed by PMs interconnected in a simple 2-tier network topology.

3.4.1 Part 2 - Experimental Environment

Different problem instances were proposed for the above mentioned experiments considering
both homogeneous and heterogeneous hardware configurations of PMs, as well as homo-
geneous and heterogeneous VMs instance types offered by Amazon Elastic Compute Cloud
(EC2)2 . A detailed description of the hardware configuration of the PMs and VMs instance
types considered for the experiments is presented in Tables 3.2 and 3.3 respectively. Additio-
nally, a general description of the considered problem instances including its decision space
size is presented in Table 3.4, while the complete set of datacenter infrastructure input files
used for the experiments with the corresponding experimental results are available online3 .
2
http://aws.amazon.com/ec2/instance-types
3
https://github.com/flopezpires/iMaVMP

Table 3.2: Types of PMs considered in experiments. For notation see Equation (3.4).
PM Type Hcpui [ECU] Hrami [GB] Hhddi [GB] pmaxi [W]
h1.small 4 16 150 440
h1.medium 180 512 10000 1000
h1.large 350 1024 10000 1300

76
Table 3.3: Instance types of VMs considered in experiments. For notation see Equation
(3.5).
Instance Type V cpuj [ECU] V ramj [GB] V hddj [GB] Rj [$]
t2.micro 1 1 0 9
t2.small 1 2 0 18
t2.medium 2 4 0 37
m3.medium 1 4 4 50
m3.large 2 8 32 100
m3.xlarge 4 15 80 201
m3.2xlarge 8 30 160 403
c3.large 2 4 32 75
c3.xlarge 4 8 80 151
c3.2xlarge 8 15 160 302
c3.4xlarge 16 30 320 604
c3.8xlarge 32 60 640 1209
r3.large 2 15 32 126
r3.xlarge 4 30 80 252
r3.2xlarge 8 61 160 504
r3.4xlarge 16 122 0 320
r3.8xlarge 32 244 0 320

Table 3.4: Problem instances considered in Experiments 1 to 3, all with 50% of VMs with
SLA s = 2.
Experiment Input # PMs # VMs PMs and VMs (n + 1)m
1 3x5.vmp 3 5 Homogeneous 1024
1 4x8.vmp 4 8 Heterogeneous 390625
2 12x50.vmp 12 50 Heterogeneous ∼ 5x1055
3 100x1000.vmp 100 1000 Heterogeneous ∼ 2x102004

Algorithms considered in the experiments were implemented using ANSI C programming

language (gcc) and the source code is available online3 . All the presented experiments were
executed on a CentOS 6.5 Linux Operating System, with an Intel(R) Xeon(R) CPU E5530
at 2.40 GHz processor and 8 GB of RAM.

3.4.2 Experiment 1: Quality of Solutions

To compare the results obtained by the proposed interactive MA and to validate its proper
operation, an Exhaustive Search Algorithm (ESA) was also implemented for finding all
(n + 1)m possible solutions of a given instance of the VMP problem, when this alternat-
ive is computationally possible for the authors. These results were compared to the results
obtained by the proposed interactive MA.

77
 Table 3.5: Summary of results obtained by the proposed algorithm in Experiment 1.
Input P ∗ size Pknown size Execution Time (ESA) Execution Time (MA)
3x5.vmp 51 51 ∼ 1 seconds ∼ 12 seconds
4x8.vmp 30 30 ∼ 720 seconds ∼ 29 seconds

Considering that this particular experiment aims to validate the good level of exploration
in the set of feasible solutions Xf , the local search of the algorithm was disabled, strengthe-
ning its capability of exploration rather than the rapid convergence to good solutions even
in early generations of the population.
For each problem instance considered in this experiment (see Table 3.4), one run of the
exhaustive search algorithm was completed, obtaining the optimal Pareto set P ∗ and its
corresponding Pareto front P F ∗ .

Furthermore, 10 runs of the proposed algorithm were completed, after evolving popula-
tions composed by 100 individuals for 100 generations at each run. The results obtained by
the proposed algorithm for each run were combined to obtain the Pareto set approximation
Pknown and its corresponding Pareto front P Fknown .
For both considered problem instances, the proposed algorithm obtained 100% of the
solutions of P ∗ and its corresponding P F ∗ . Additionally, the proposed algorithm performed
well in execution time against the ESA, even obtaining the same optimal results in less
execution time for the 4x8.vmp scenario. A summary of the number of elements in the
corresponding Pareto sets obtained and the execution time of both algorithms are presented
in Table 3.5.

3.4.3 Experiment 2: Interactive Bounds

For the problem instance considered in this experiment (12x50.vmp), one run of the proposed
algorithm was completed, after evolving populations of 100 individuals for 300 generations.
The number of generations was incremented for this experiment from 100 to 300, taking
into account the large number of possible solutions for the particular problem considered
(see Table 3.4). An interactive adjustment of the lower or upper bounds associated to
each objective function was performed after every 100 generations in order to converge to

78
a treatable number of solutions. It is important to remark that the interactive adjustment
used in this experiment is only one of several possible ones. As an example, we may consider:
(1) automatically adjusting a % of the lower bounds associated to maximization objective
functions when the Pareto front has a defined number of elements or (2) manually adjusting
upper bounds associated to minimization objective functions until the Pareto front does not
have more than 20 elements, just to cite a pair of alternatives.
The Pareto front approximation P Fknown represents the complete set of Pareto solutions
considering unrestricted bounds (Lz = −∞ and Uz = ∞). On the other hand, Pareto front
approximation P Freduced represents the reduced set of Pareto solutions obtained by interac-
tively adjusting bounds Lz and Uz . In the first 100 generations, the proposed algorithm
obtained 251 solutions with unrestricted bounds. A decision maker evaluated the bounds
associated to f1 (x) (power consumption) and adjusted the upper bound U1 to U10 = 9000 [W],
selecting only 35 out of the 251 solutions (not considering 216 otherwise feasible solutions)
for the P Freduced as shown in Figure 3.2.

103 965
484
251
Number of solutions

102 68 63
35 36
17
101

unrestricted bounds
adjustable bounds
100 200 300
Number of generations

Figure 3.2: Summary of results obtained in Experiment 2 using adjusted lower and upper
bounds.

79
 After 200 generations, the algorithm obtained a total of 484 solutions with unrestricted
bounds. Considering instead U10 = 9000 [W], the algorithm only found 68 solutions. The
decision maker evaluated the bounds associated to f2 (x) (network traffic) and adjusted the
upper bound U2 to U20 = 115 [Mbps], selecting only 36 out of the 68 solutions (not considering
32 otherwise feasible solutions) for the P Freduced .
Finally, after 300 generations, the algorithm obtained a total of 965 solutions with un-
restricted bounds. Considering U10 = 9000 [W] and U20 = 115 [Mbps], the algorithm found 63
solutions. The decision maker evaluated the bounds associated to f3 (x) (economical revenue)
and adjusted the lower bound L3 to L03 = 13500 [$], selecting only 17 out of the 63 solutions
(not considering 46 feasible solutions) for the final P Freduced as shown in Figure 3.2. Clearly,
at the end of the iterative process, the decision maker found 17 solutions according to his
preferences instead of the unmanageable number of 965 candidate solutions.

3.4.4 Experiment 3: Algorithm Scalability

It should be noted that increasing the number of PMs and VMs in a VMP problem could
result in extremely large decision spaces, considering all (n + 1)m possible solutions (see
Table 3.4). Consequently, algorithms designed for the resolution of VMP problems should
be able to effectively solve VMP problem instances composed by large numbers of VMs and
PMs in a reasonable time.
For the problem instance considered in this experiment (100x1000.vmp), one run of the
proposed algorithm was completed, after evolving populations composed by 100, 200 and
300 individuals for 500 generations. For this particular experiment, the Pareto front appro-
ximation P Fknown represents the complete set of Pareto solutions considering unrestricted
bounds for each fz (x) (Lz = −∞ and Uz = ∞) in order to experimentally demonstrate that
large instances of the formulated MaVMP problem could result in unmanageable number
of solutions. A summary of the results obtained by the proposed algorithm is presented in
Table 3.6. The obtained results prove the capabilities of the proposed algorithm to effectively
solve instances of the proposed MaVMP problem with large numbers of PMs and VMs, as
considered in real-world scenarios.

80
 Table 3.6: Summary of results obtained by the proposed algorithm in Experiment 3.
Input # of individuals Pknown size # of generations
100x1000.vmp 100 397 100
100x1000.vmp 200 399 100
100x1000.vmp 300 509 100
100x1000.vmp 100 769 200
100x1000.vmp 200 811 200
100x1000.vmp 300 1087 200
100x1000.vmp 100 1103 300
100x1000.vmp 200 1329 300
100x1000.vmp 300 1641 300
100x1000.vmp 100 1434 400
100x1000.vmp 200 1791 400
100x1000.vmp 300 2178 400
100x1000.vmp 100 1742 500
100x1000.vmp 200 2192 500
100x1000.vmp 300 2719 500

Additionally, it could be observed that by increasing the number of individuals on popula-

tions or the number of generations, the algorithm obtained larger numbers of non-dominated
solutions with unrestricted bounds. Considering that the proposed algorithm could find more
non-dominated solutions than the obtained in this particular experiment if more computa-
tional resources for calculation are considered (or increasing the number of individuals or the
number of generations), it could be noted the importance of including additional methods
to the Pareto dominance relation (e.g. adjustable bounds) for the selection of a manageable
sub-set of Pknown in MaVMP problems.

81
 Part III

MaVMP with Reconfiguration of

VMs (semi-dynamic)

82
4 MaVMP with Placement Reconfiguration
Once an initial placement of VMs have been performed (see Chapter 3), a virtualized data-
center could be reconfigured through VM live migration in order to maintain efficiency in
operations, considering that the set of requested VMs changes over time, to study this par-
ticular semi-dynamic formulation of a VMP problem as a first approximation to dynamic
formulations of real-world Infrastructure as a Service (IaaS) environments.
According to [94, 96], the optimization of power consumption is the most studied objec-
tive function in VMP literature [13, 131]. Furthermore, network traffic [6] and economical
revenue [121, 124] are also very studied as objective functions for the VMP problem. For a
VMP problem formulation with reconfiguration of VMs, two additional objective functions
associated to migration of VMs represent challenges for CSPs: number of VM migrations
[83] and network traffic overhead for VM migrations [14].
Considering the large number of existing objective functions for VMP problems identified
in [94, 96], López-Pires and Barán have proposed in [92, 95] a many-objective optimization
framework in order to consider as many objective functions as needed when solving a MaVMP
problem in virtualized datacenters. The mentioned framework focuses on static (offline)
formulations of the MaVMP problem, proposing for the first time a MaVMP for initial
placement of VMs, simultaneously optimizing five objective functions (see Chapter 3).
To the best of the authors’ knowledge there is no published work presenting a formu-
lation of a MaVMP with reconfiguration of VMs. Consequently, this chapter extends the
formulations presented in Chapter 3 [92, 95] proposing for the first time a MaVMP with
reconfiguration of VMs, considering this time the simultaneous optimization of the following
five objective functions: (1) power consumption, (2) inter-VM network traffic, (3) economical
revenue, (4) number of VM migrations and (5) network traffic overhead for VM migrations.
To solve the formulated MaVMP problem, the interactive MA presented in Chapter 3
[92, 95] was extended to consider particular challenges associated to the resolution of a
MaVMP problem with reconfigurations of VMs, as next introduced.

83
 Several challenges need to be addressed for MaVMP formulations with reconfiguration of
VMs. In Pareto-based algorithms, the Pareto set approximation can include a large number
of non-dominated solutions. Selecting one of the non-dominated solutions can be considered
a problem for a MaVMP problem. In consequence, this work evaluates the following five
selection strategies: (1) random, (2) preferred solution, (3) minimum distance to origin, (4)
lexicographic order (provider preference) and (5) lexicographic order (service preference) to
identify convenient strategies for automatic selection of a solution for the mentioned problem.

4.1 Part 3 - Problem Formulation

To the best of the authors’ knowledge this chapter presents a first formulation of a MaVMP
with reconfiguration of VMs, considering this time the simultaneous optimization of the
following five objective functions: (1) power consumption, (2) inter-VM network traffic,
(3) economical revenue, (4) number of VM migrations and (5) network traffic overhead
for VM migrations. Formally, the proposed offline (semi-dynamic) MaVMP problem with
reconfiguration of VMs can be enunciated as:
Given the available PMs and their specifications, the requested VMs and their specifications,
the network traffic between VMs and the current placement of the VMs, it is sought a new
placement of the set of VMs in the set of PMs, satisfying the constraints of the problem while
simultaneously optimizing all defined objective functions (as power consumption, inter-VM
network traffic, economical revenue, number of VM migrations and network traffic overhead
for VM migration), in a pure many-objective context, before selecting a specific solution for
a given time instant t.

4.1.1 Part 3 - Input Data

The set of available PMs is represented as a matrix H ∈ Rn×4 :

 
 Hcpu1 Hram1 Hhdd1 pmax1 
 
 
H= 
 ... ... ... ... 
 (4.1)
 
 
Hcpun Hramn Hhddn pmaxn

84
 Each row represents a PM Hi composed by its processing resources of CPU (for example
as Elastic Compute Unit (ECU1 ) or Millions of Instructions per Second (MIPS)), RAM
memory [GB], storage [GB] and a maximum power consumption [W] as:

Hi = [Hcpui , Hrami , Hhddi , pmaxi ], ∀i ∈ {1, ..., n} (4.2)

where:
Hcpui : Processing resources of Hi ;

Hrami : RAM memory resources of Hi ;

Hhddi : Storage resources of Hi ;

pmaxi : Maximum power consumption of Hi ;

n: Number of PMs.

Accordingly, the set of VMs at instant t is represented as a matrix V (t) ∈ Rm×5 :

 
 V cpu1 V ram1 V hdd1 SLA1 R1 
 
 
V (t) = 
 ... ... ... ... ... 
 (4.3)
 
 
V cpum(t) V ramm(t) V hddm(t) SLAm(t) Rm(t)

Each Vj represents the required processing resources CPU [ECU], RAM memory [GB],
storage [GB], revenue [$] and SLA:

Vj = [V cpuj , V ramj , V hddj , SLAj , Rj ], ∀j ∈ {1, . . . , m(t)} (4.4)

where:
1
http://aws.amazon.com/ec2/faqs

85
 V cpuj : Processing requirements of Vj ;
V ramj : Memory requirements of Vj ;
V hddj : Storage requirements of Vj ;
Rj : Economical revenue for placing Vj ;
SLAj : Service Level Agreement SLAj of a Vj , where SLAj ∈ {0, 1, . . . , s} being
s the highest priority level;
m(t): Number of VMs at instant t, then m(t) ∈ {1, . . . , mmax };
mmax : Maximum number of VMs.

Once a Vj is powered off by the tenant, its resources are released, so the physical resources
can be reused. For simplicity, the index j is not considered to be reused; therefore, for this
work Vj is not a function of time.
The traffic between VMs at instant t is represented as a matrix T (t) ∈ Rm(t)×m(t) :

 
 T1,1 (t) ... T1,m(t) (t) 
 
 
T (t) = 
 ... ... ... 
 (4.5)
 
 
Tm(t),1 (t) . . . Tm(t),m(t) (t)

In Equation (4.5), Tj,k (t) represents the average communication rate in [Kbps], between
VM Vj and VM Vk at instant t. Note that we can consider Tjj (t) = 0.
The placement at instant t is represented as a matrix x(t) ∈ Rm(t)×n :

 
 x1,1 (t) ... x1,n (t) 
 
 
x(t) = 
 ... ... ... 
 (4.6)
 
 
xm(t),1 (t) . . . xm(t),n (t)

where:
xj,i (t) ∈ {0, 1} indicates if Vj is located (xj,i = 1) or not (xj,i = 0) for execution on a PM
Hi (i.e., xj,i (t) : Vj → Hi ) at instant t.

86
4.1.2 Part 3 - Output Data

A solution of the problem at each instant is a new VM placement x(t). In order to acco-
mmodate a new placement, a series of Management Actions (MAc) (i.e. VM migrations,
creation or destruction) must be performed. These are presented by the following output
data: (1) the new VM placement and (2) the list of required management actions.
The new placement at instant (t + 1) is represented as a matrix x(t + 1) of dimension
m(t + 1) × n:

 
 x1,1 ... x1,n 
 
 
x(t + 1) = 
 ... ... ... 
 (4.7)
 
 
xm(t+1),1 . . . xm(t+1),n

where: xj,i (t + 1) ∈ {0, 1} indicates if Vj is located (xj,i (t + 1) = 1) or not (xj,i (t + 1) = 0)

for execution on a PM Hi at instant t (i.e., xj,i (t + 1) : Vj → Hi ).
The set of necessary management actions in order to evolve from x(t) to x(t + 1) is
represented by:

h i
M Act→t+1 = M Ac(V1 ), . . . , M Ac(Vm(t+1) ) (4.8)

where: M Ac(Vj ) ∈ {0, 1, 2, 3} which represents the management actions that a hypervisor
must execute in order to accommodate the x(t + 1) placement corresponding to Vj .
Values returned by the M Ac(Vj ) function should be interpreted as follows:
M Ac(Vj ) = 0: no management action is necessary, i.e. xj,i (t + 1) = xj,i (t), ∀i;
M Ac(Vj ) = 1: a new VM Vj is to be placed on a PM Hi , i.e. xj,i (t + 1) = 1;
M Ac(Vj ) = 2: an existing VM Vj is to be migrated from Hi0 to another Hi , i.e.
xj,i0 (t) = 1 and xj,i (t + 1) = 1;
M Ac(Vj ) = 3: a VM Vj is shutdown, i.e. xj,i (t) = 1 but xj,i (t + 1) = 0.

87
4.1.3 Part 3 - Constraints

Constraint 1: unique placement of VMs

A VM Vj should be located to run on a single PM Hi or alternatively, it could be not located

in any PM if the associated SLAj is not the highest level of priority (in this work s = 2).
Consequently, this constraint is expressed as:

n
X
xj,i (t) ≤ 1 ∀j ∈ {1, ..., m(t)} (4.9)
i=1

where:
xj,i (t): Binary variable equals 1 if Vj is located to run on Hi at instant t; other-
wise, it is 0.

Constraint 2: assure SLA provisioning

A VM Vj with the highest level of SLA (s = 2) must necessarily be located to run on a PM

Hi . Consequently, this constraint is expressed as:

n
X
xj,i (t) = 1 ∀j such that SLAj = s
i=1 (4.10)
∀t where Vj should be active.

It should be mentioned that different levels of SLA can be considered, as presented in [92].

Constraints 3-5: physical resources capacities of PMs

A PM Hi must have sufficient available resources to meet the requirements of all VMs Vj
that are located to run on Hi at instant t. In this work, the overbooking of resources [139]
is not considered; consequently, the set of constraints can be mathematically formulated as:

m(t)
X
V cpuj × xj,i (t) ≤ Hcpui (4.11)
j=1

m(t)
X
V ramj × xj,i (t) ≤ Hrami (4.12)
j=1

88
 m(t)
X
V hddj × xj,i (t) ≤ Hhddi (4.13)
j=1

∀i ∈ {1, ..., n}, i.e. for all physical machines Hi .

4.1.4 Part 3 - Objective Functions

This section presents five objective functions that are simultaneously optimized in the pro-
posed MaVMP formulation. These objective functions are mathematically formulated as
follows:

Power Consumption Minimization

Based on Beloglazov et al. [13], this work modeled the power consumption of PMs considering
a linear relationship with the CPU utilization. The power consumption can be represented
by the sum of the power consumption of each PM Hi :

n
X
f1 (x) = ((pmaxi − pmini ) × U cpui (t) + pmini ) × Yi (4.14)
i=1

where:
f1 (x): Total power consumption of the PMs;
pmini : Minimum power consumption of Hi . As defined in [13], pmini = pmaxi ∗
0.6;
U cpui (t): Utilization ratio of processing resources used by Hi at instant t;
Yi : Binary variable equals 1 if Hi is turned on; otherwise, it is 0.

Inter-VM Network Traffic Minimization

A very studied approach for inter-VM network traffic minimization is the placement of VMs
with high communication rate in the same PM (or at least in the same rack) to avoid the
utilization of network resources (or at least core network equipment).
The minimization of network traffic among VMs, by maximizing locality, was proposed
in [127]. Based on this approach, Equation (4.15) represents the sum of average network
traffic generated between VM Vj and VM Vk when located on different PMs.

89
 m(t) m(t)
X X
f2 (x) = (Tj,k (t) × Dj,k (t)) (4.15)
j=1 k=1

where:
f2 (x): Total inter-VM network traffic;
Dj,k (t): Binary variable that equals 1 if Vj and Vk are located in different PMs
at instant t; otherwise, it is 0.
The traffic between two VMs Vj and Vk located on the same PM Hi do not contribute
to increase the total network traffic given by Equation (4.15); therefore, Dj,k (t) = 0 if
xj,i (t) = xk,i (t) = 1.

Economical Revenue Maximization

Based on [100], this work presents Equation (4.16) to estimate the total economical revenue
that a datacenter receives for meeting the requirements of its customers, represented by
the sum of the economical revenue obtainable by each VM Vj that is effectively located for
execution on any PM at instant t.

m(t)
X
f3 (x) = (Rj × Xj (t)) (4.16)
j=1

where:
f3 (x): Total economical revenue for placing VMs;
Xj (t): Binary variable that equals 1 if Vj is located for execution on any PM at
instant t; otherwise, it is 0.

Number of VM Migrations Minimization

Performance degradation may occur when migrating VMs from one PM to another [14].
Logically, it is desirable that the number of migrated VMs is kept to a minimum for better
quality of service (QoS). Therefore, Equation (4.17) represents the number of VM migrations
at time instant t:

m(t)
X
f4 (x) = Zj (t) (4.17)
j=1

90
where:
f4 (x): Number of VM migrations at instant t;
Zj (t): Binary variable that equals 1 if M A(Vj ) = 2, i.e. Vj is migrated, see
Equation (4.8); otherwise, it is 0 (Vj is not migrated).

Network Traffic Overhead for VM Migrations Minimization

As explained in [14], the overhead of VM migrations on network resources is proportional to

the memory size of the migrated VM. In this work, Equation (4.18) is proposed to minimize
the amount of RAM memory that must be copied between PMs at instant t.

m(t)
X
f5 (x) = V ramj × Zj (t) (4.18)
j=1

where:

f5 (x): Network traffic overhead for VM migrations at instant t;

It should be mentioned that there are other possible modeling approaches to estimate the
migration overhead, as presented in [133].

4.2 Extended Memetic Algorithm for MaVMP

This work extends the interactive MA proposed in [92, 95] for solving the MaVMP problem
with reconfiguration of VMs, as the one formulated in Section 4.1. The proposed algorithm
simultaneously optimizes the five objective functions presented in Section 4.1.4.
Many-objective optimization using Multi-Objective Evolutionary Algorithms (MOEAs)
is an active research area, with multiple challenges that need to be addressed regarding
scalability analysis, solutions visualization, algorithm design and experimental algorithm
evaluation as shown in [143]. At each time instant, the set of feasible placements can be
composed by a large number of non-dominated solutions. Therefore, the proposed algorithm
automatically selects one of the possible placements after each time instant according to one
of the considered selection strategies (see subsection 4.3). The proposed algorithm is based
on the one proposed in [89] and it works as follows (see Algorithm 5):

91
 Algorithm 5: Extended Memetic Algorithm
Data: datacenter infrastructure (see Section 4.1.1) and solution selection strategy
parameter
Result: solution Pselected for instant t
1 t=0
2 while there are VM requests to process do
3 check if the problem has a solution
4 Pprevious = Pselected
5 initialize set of solutions P op0
6 P op00 = repair infeasible solutions of P op0
7 P op000 = apply local search to solutions of P op00
8 update set of solutions Pknown from P op000
9 u = 0; P opu = P op000
10 while is not stopping criterion do
11 Qu = selection of solutions from P opu ∪ Pknown
12 Q0u = crossover and mutation of solutions of Qu
13 Q00u = repair infeasible solutions of Q0u
14 Q000
u = apply local search to solutions of Qu
00

15 update set of solutions Pknown from Q000 u

16 increment number of generations u
17 P opu = non-dominated sorting from P opu ∪ Q000 u
18 end
19 Pselected = selected solution (selection strategy parameter)
20 return Pselected
21 increment instant t; reset Pknown
22 end

The algorithm iterates over each set of requested VMs received at each instant t. At
step 3, the algorithm verifies if the problem has at least one solution to continue with next
steps. If there is no possible solution to the problem, the algorithm returns an appropriate
error message. If the problem has at least one solution, the algorithm proceeds to step 4 in
order to determine the current placement. After the first iteration, the current placement
is the one selected from the previous iteration. At step 5, a set P op0 of candidates is
randomly generated. These candidates are repaired at step 6 to ensure that P op0 contains
only feasible solutions. Then, the algorithm tries to improve candidates at step 7 using local
search. With the obtained non-dominated solutions, the first set Pknown is generated at step
8. After initialization in step 9, evolution begins. Evolutionary process basically follows the
same behavior: solutions are selected from the union of Pknown with the evolutionary set
of solutions also known as P opu (step 11), crossover and mutation operators are applied as
usual (step 12). Eventually solutions are repaired, as there may be infeasible ones (step 13).

92
 Improvements of solutions of the evolutionary population P opu may be generated at step
14 using local search (local optimization operators). At step 15, the Pareto set approxima-
tion Pknown is updated (if applicable); while at step 16 the generation (or iteration) counter
is updated. At step 17 a new evolutionary population P opu is selected. The evolutionary
process is repeated until the algorithm meets a stopping criterion (such as a maximum num-
ber of generations), returning one solution Pselected from the set of non-dominated solutions
Pknown in step 20, using one of the selection strategies presented in Section 4.3.

4.2.1 Part 3 - Population Initialization

Initially, a set of solutions (or population P op0 ) is randomly generated. Each solution (or
individual) is represented with a chromosome C(t) = [C1 , C2 , . . . , Cm(t) ], where m(t) ≤ mmax .
The possible values that can take each Ck for VMs with the highest value of SLAj (s = 2)
are in the range [1, n]. For VMs Vj with SLAj < s, the possible values are in the range
[0, n]. Within these ranges defined by the SLAj of each Vj , the algorithm ensures at the
initialization stage that all VMs Vj with the highest level of priority will be located for
execution on a PM Hi , while for VMs Vj with lower levels of priority SLAj , there is always
a probability larger than 0 that they may not be located for execution in any PM.

4.2.2 Part 3 - Infeasible Solution Reparation

With a random generation at the initialization phase (step 5 of Algorithm 5) and/or solutions
generated by standard genetic operators (step 12 of Algorithm 5), infeasible solutions may
appear, i.e. the resources required by the VMs located for execution on certain PMs could
exceed the available resources (see Section 4.1.3).
Repairing these infeasible solutions (steps 6 and 13 of Algorithm 5) may be done in two
stages: first, in the feasibility verification process, the population is classified in two classes:
feasible and infeasible (Algorithm 6). Later, in the process of repairing infeasible solutions
(Algorithm 7), the solutions which do not meet the feasibility criteria are repaired in three
ways: (1) migrating some VMs to an available hardware, (2) turning on some PMs and then
migrating VMs to them, or (3) turning off some VMs.

93
 Note that at step 4 of Algorithm 7, Vj migration to Hi0 can be done to other PMs, even
if they are shut down.

4.2.3 Part 3 - Local Search

Once a population contains only feasible solutions, a local search is performed (steps 7 and
14 of Algorithm 5) for improving the solutions found so far in the evolutionary population
P opu . The local search pseudo-code is presented in Algorithm 8.
For each individual in the population P opu , the proposed algorithm attempts to optimize
a solution with a local search (step 2 of Algorithm 8). For this purpose, with probability
1
2
, the algorithm tries to maximize the number of located VMs with higher level of priority,
locating all possible VMs that were not allocated so far, directly increasing economical
revenue f3 (x) (steps 3 to 5 of Algorithm 8).
On the other hand, also with probability 12 , the algorithm tries to minimize the number
of PMs turned on, directly reducing power consumption f1 (x) (steps 6 to 8 of Algorithm
8). With the proposed probabilistic local search method, a balanced exploitation of objec-
tive functions (economical revenue and power consumption) is achieved, as experimentally
verified with results presented in next section.

Algorithm 6: Part 3 - Feasibility Verification

Data: set of solutions P opu
Result: set of feasible solutions P op0u
1 while there are solutions not verified do
2 feasible = true ; i = 1
3 while i ≤ n and feasible = true do
4 if solution does not satisfy constraints (3-5) then
5 feasible = false ; break
6 else
7 increment i
8 end
9 end
10 if feasible = false then
11 call Algorithm 7 (repair solution)
12 end
13 end
14 return set of feasible solutions P op0u

94
 Algorithm 7: Part 3 - Infeasible Solutions Reparation
Data: infeasible solution
Result: feasible solution
1 feasible = false ; j = 1
2 while j ≤ m and feasible = false do
3 if it is possible then
4 migrate Vj to Hi0 (i0 6= i)
5 else
6 if SLAj 6= 2 then
7 turn off Vj on Hi
8 else
9 replace solution with another solution from Pknown
10 end
11 end
12 end
13 return feasible solution

4.2.4 Part 3 - Fitness Function

The fitness function used in the proposed algorithm is the one presented in [31]. This fitness
value defines a non-domination rank in which a value equal to its Pareto dominance level (1
is the highest level of dominance, 2 is the next, and so on) is assigned to each individual of
the population. Between two individuals with different non-domination rank, the individual
with lower value (higher level of dominance) is considered better.
To compare individuals with the same non-domination rank, a crowding distance is used.
The basic idea in crowding distance is to find the Euclidean distance (properly normalized

Algorithm 8: Part 3 - Local Search

Data: set of feasible solutions P op0u
Result: set of feasible optimized solutions P op00u
1 probability = random number between 0 and 1
2 while there are solutions not verified do
3 if probability < 0.5 then
4 Try to turn off all possible Hi by migrating all the Vj assigned to Hi0 with available
resources (i0 6= i) and then trying to turn on all the possible Vj (using SLAj
priority order) assigning them to an Hi with available resources
5 else
6 Try to turn on all the possible Vj (using SLAj priority order) assigning them to an
Hi with available resources and then trying to turn off all possible Hi by migrating
all the Vj assigned to Hi0 with available resources (i0 6= i)
7 end
8 end
9 return set of feasible optimized solutions P op00u

95
when the objectives have different measure units) between each pair of individuals, based
on the q objectives, in a hyper-dimensional space [31]. The individual with larger crowding
distance is considered better.

4.2.5 Part 3 - Variation Operators

The proposed interactive MA uses a Binary Tournament method for selecting individuals
for crossover and mutation [28]. The crossover operator used in the presented work is the
single point cross-cut [28]. The selected individuals in the ascending population are replaced
by descendants individuals.
1
This work uses a mutation method in which each gene is mutated with a probability m(t)
,
where m(t) represents the number of VMs at time instant t. This method offers the possibility
of full uniform gene mutation, with a very low probability (but larger than zero), which is
beneficial to the exploration of the search space, reducing the probability of stagnation in a
local optimum.
The population evolution in the proposed interactive MA is based on the population
evolution proposed in [31]. A population P opt+1 is formed from the union of the best known
population P opt and offspring population Qt , applying non-domination rank and crowding
distance.

4.3 Solution Selection Strategies

Several challenges need to be addressed for a MaVMP problem with reconfiguration of VMs.
In Pareto based algorithms, the Pareto set approximation can include a large number of
non-dominated solutions; therefore, selecting one of the non-dominated solutions (step 19 of
Algorithm 5) can be considered as a new difficulty for the MaVMP problems with reconfi-
guration of VMs.
This work performed an experimental evaluation of the following five selection strategies:
(1) random, (2) preferred solution, (3) minimum distance to origin, (4) lexicographic order
(provider preference) and (5) lexicographic order (service preference), as next explained.

96
4.3.1 Random (S1)

Considering that the Pareto set approximation is composed by non-dominated solutions,

randomly selecting one of the solutions could be an acceptable selection strategy.

4.3.2 Preferred Solution (S2)

A solution is defined as preferred to another non-comparable solution when it is better in

more objective functions [134]. When several solutions can be considered as preferred ones
(there is a tie), only one of these solutions is randomly selected.

4.3.3 Minimum Distance to Origin (S3)

The solution with the minimum Euclidean distance to the origin is selected, considering
all normalized objective functions in a minimization context. For this purpose, f3 (x) is
redefined as the difference between the maximum revenue possible at instant t and the
attainable revenue of each possible solution. When several solutions have equal Euclidean
distance, only one of these solution is randomly selected.

4.3.4 Lexicographic Order

Each objective function is given in an order of evaluation, similar to the ordering of letters
in a dictionary. The objective functions can be arranged in several ways in order of priority.
This work proposes two different lexicographic orders, representing the possible preferences
associated to providers (provider preference) and quality of service (service preference). Lo-
gically, different orders of priority criteria may be considered depending on each specific
context. According to Equation (2.1) there are q! possible lexicographic orders.

Provider Preference Order (S4)

The priority order is: (1) economical revenue, (2) power consumption, (3) inter-VM network
traffic, (4) number of VM migrations and (5) network traffic overhead for VM migration.

97
 Table 4.1: Hardware configuration of PM types considered in Experiment 4.
Hardware configuration Number of PMs
PM Type Hcpu [ECU] Hram [GB] Hhdd [GB] pmax [W] 10x100.vmp 100x1000.vmp
h1.small 180 512 10,000 1,000 3 30
h1.medium 260 512 10,000 1,350 3 30
h1.large 350 1,024 10,000 1,800 3 30
h2.large 400 1,024 10,000 2,000 1 10
Total PMs 10 100

Service Preference Order (S5)

The priority order is: (1) number of VM migrations, (2) network traffic overhead for VM,
(3) inter-VM network traffic, (4) power consumption and (5) economical revenue.

4.4 Part 3 - Experimental Results

This work evaluates the following five selection strategies for selecting one appropriate solu-
tion from the resulting Pareto set approximation at each time t: (1) random, (2) preferred
solution, (3) minimum distance to origin, (4) lexicographic order (provider preference) and
(5) lexicographic order (service preference). In order to evaluate the mentioned selection
strategies, several experiments were performed and the following subsections summarize the
experimental results.

4.4.1 Part 3 - Experimental Environment

The proposed Memetic Algorithm presented in Section 4.2 was implemented using the ANSI
C programming language (gcc). The source code is available online2 .
The experimental scenarios included heterogeneous PMs with hardware configurations
described in Table 4.1. Considered VMs were based on real instance types offered by Amazon
Elastic Compute Cloud (EC2) [122] as presented in Table 4.2. The experiments were per-
formed considering two different experimental scenarios (a small and a medium sized cloud
datacenter infrastructure) simulating a theoretical day (i.e. 24 hours) in a datacenter where
VMs requests are received and processed hourly as shown in Figures 4.1 and 4.2.
2
https://github.com/dihara/MaVMP

98
 In these experiments, the following configurations were considered:

• 10x100.vmp: VMP problem instance with 10 PMs initially running 100 VMs (see
workload details in Figure 4.1).

• 100x1000.vmp: VMP problem instance with 100 PMs initially running 1,000 VMs
(see workload details in Figure 4.2).

In Figures 4.1 and 4.2, the number of existing VMs to be shutdown and new VMs to
be placed are detailed for each instant t for 2 different scenarios. For simplicity, in what
follows, the traffic between VMs was considered as constant, i.e. Ti,j (t) = Ti,j . The initial
load for Experiment 1 represents 28% of total CPU resources while in Experiment 2, it is
33% of total CPU resources (see Table 4.4).
Experiments for each of the five selection strategies were repeated 10 times, given the
probabilistic nature of the proposed Memetic Algorithm. Results are analyzed in Section
4.4.2. The average number of non-dominated solutions found with each selection strategy is
shown in Table 4.3. It can be seen that in both experiments, a similar average number of
solutions with similar standard deviation were observed for all five selection strategies.

Table 4.2: Instance types of VMs considered. For notation see Equation (4.3).
Instance Type Vcpu [ECU] Vram [GB] Vhdd [GB] R [$]
t2.micro 1 1 0 9
t2.small 1 2 0 18
t2.medium 2 4 0 37
m3.medium 1 4 4 50
m3.large 2 8 32 100
m3.xlarge 4 15 80 201
m3.2xlarge 8 30 160 403
c3.large 2 4 32 75
c3.xlarge 4 8 80 151
c3.2xlarge 8 15 160 302
c3.4xlarge 16 30 320 604
c3.8xlarge 32 60 640 1209
r3.large 2 15 32 126
r3.xlarge 4 30 80 252
r3.2xlarge 8 61 160 504
r3.4xlarge 16 122 0 320
r3.8xlarge 32 244 0 320

99
 Table 4.3: Number of non-dominated solutions per selection strategy.
10x100.vmp 100x1000.vmp
Selection strategy
Average Standard Dev. Average Standard Dev.
Random 25.2 8.6 36.2 11.3
Preferred Solution 24.0 8.6 37.7 9.2
Distance to origin 21.4 8.7 30.8 10.1
Provider Preference 20.8 9.9 24.0 8.7
Service Preference 34.5 9.5 38.9 9.1

4.4.2 Experiment 4: Selection Strategy Evaluation

Table 4.5 summarizes the results obtained with both workload. As expected, when the lex-
icographic order is used, the most important objective function is the one with the best
results, i.e. the S4 strategy (provider preference) obtains the best results in power consump-
tion f1 (x), with 20% less power consumption than the worst strategy in 10x100.vmp and
2% less power consumption than the worst strategy in 100x1000.vmp. Analogously, when
service perspective is prioritized (strategy S5), the objective functions f4 (x) and f5 (x) obtain
the best results.
However, as the focus of this work is the simultaneous optimization of all five objective
functions with a PMO optimization approach, a comparison is made considering the concept
of Pareto dominance. As seen in Table 4.5 (dominance column), the S3 strategy dominates
S2 and S1 in both experiments; however, it is non-comparable with respect to S4 and S5
in both tested problem instances.

Table 4.4: Details of Experiment 4.

Parameters 10x100.vmp 100x1000.vmp
# PM 10 100
Available CPU 2,770 27,700
Initial # VM 100 1,000
VMs with SLA 0 26 325
VMs with SLA 1 38 344
VMs with SLA 2 36 331
Initial CPU load 784 (28%) 9,023 (33%)
Initial revenue 33,973 US$ 330,645 US$
Discrete time instants 24 hours 24 hours

100
m(t) 110
107 107 108 106 108
m(t)
104
100 100 98 98 99 99 99 101 102
95 93 94 96
91 90 91 92 93

0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 (t)
#
VMs
+ 11 4 2 1 5 9 8 2 7 6 12 1 5 2 11 8 10 3 9 4 1 10 6 7
cre-
ated
#
VMs
− 1 7 2 0 7 7 12 6 9 6 11 5 7 4 12 7 9 1 4 4 7 2 5 13
des-
troyed

Figure 4.1: Workload for Experiment 4 (10x100.vmp): number of running VMs as a function
of time t.

m(t) 1,000 m(t)

995 993
988 990 988 987
984 986 987 986 983 986
979 980 977 980 979 979 982 981 979
975 976 975

0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 (t)
#
VMs
+ 8 7 8 6 14 6 14 15 6 13 10 12 15 14 8 10 5 11 5 15 13 9 8 14
cre-
ated
#
VMs
− 13 9 13 15 13 11 13 7 13 10 11 5 14 15 4 12 6 15 13 11 10 10 10 7
des-
troyed

Figure 4.2: Workload for Experiment 4 (100x1000.vmp): number of running VMs as a

function of time t.

Given that S3 cannot be declared as the best strategy considering exclusively Pareto
dominance, a further comparison of selection strategies using the preference (i.e. larger
number of better objective functions) criteria [143] is presented in the corresponding column
of Table 4.5.
It may seem intuitive that the S2 strategy (that uses the preference criterion) should
be the best; however, Table 4.5 shows that strategy S3 is preferred not only to S2 but
also to S1 and to S4 in both tested problem instances. Additionally, it can be seen that
S3 is preferred to S5 in problem instance 10x100.vmp while no strategy is preferred to S3,
indicating that S3 (distance to origin) could be considered the preferred (best) strategy for
solving the proposed MaVMP problem formulation.

101
 Table 4.5: Selection Strategy Comparison. For selection strategy notation see Section 4.3.
Dominance Preference
Selection
Objective functions averages (see 4.1.4) (row  column) (row p column )
strategy
f1 (x) f2 (x) f3 (x) f4 (x) f5 (x) S1S2S3S4S5 S1 S2 S3 S4 S5 Comment
10x100.vmp
S1 9, 908 19, 981 32, 623 44 1, 526
According to dominance, S1 and
S2 9, 827 19, 991 32, 623 8 180  
S2 are dominated by S3.
S3 9, 639 19, 228 32, 623 6 124  

102
  According to preference, S3 is
S4 8, 543 21, 038 32, 623 19 520  preferable to S1, S2, S4 and S5.
S5 10, 395 21, 957 32, 623 5 150
100x1000.vmp
S1 104, 559 371, 664 325, 217 650 26, 886 According to dominance, S1 and
S2 104, 835 373, 467 325, 217 37 1, 204   S2 are dominated by S3.
S3 104, 378 370, 489 325, 217 26 804    According to preference, S3 is
S4 103, 175 374, 210 325, 217 92 3, 531  preferable to S1, S2 and S4. S5
S5 104, 860 373, 230 325, 217 20 618   is preferable to S2 and S4.
 As a consequence of the above results, for production cloud datacenters, instead of calcu-
lating all the Pareto set or a Pareto set approximation, the S3 strategy (distance to origin)
could be used to combine all considered objective functions into only one objective function,
therefore solving the studied problem considering a MAM approach (multi-objective problem
solved as mono-objective). It is important to mention that the obtained results are consistent
with the selection strategy evaluation presented in [134] for solution of a multi-cast traffic
engineering problem in computer networks.

103
 Part IV

MaVMP for Cloud Computing

(dynamic)

104
5 Experimental Comparison of Algorithms
A main concern of cloud datacenters design is to efficiently manage available resources in
order to improve performance and reduce energy consumption of a given computational
infrastructure. Most of the time, servers operate in a very low energy-efficiency region (i.e.
between 10% and 50% of resource utilization), even considering that workload peaks rarely
occur in practice [12]. Several methods were considered for addressing this issue, being the
virtualization of resources the most studied for cloud computing datacenters.
In cloud computing datacenters, resources are allocated and released dynamically trying
to serve requested demands. It is relevant to remember that the VMP problem is a NP-Hard
combinatorial optimization problem [130]. From an Infrastructure as a Service (IaaS) pro-
vider’s perspective, the VMP problem must be formulated as an online problem (considering
that requests from customers are unknown a priori) and it should be solved with short time
constraints. Consequently, solution techniques with low computational complexity are very
studied for this VMP environment [94].
The most studied heuristics in the VMP literature are: Best-Fit (BF), Best-Fit Decreas-
ing (BFD), First-Fit (FF), First-Fit Decreasing (FFD) and Worst-Fit (WF) [94].
As presented by López-Pires and Barán in [94], over 60 different objectives have been
proposed for VMP problems. The number of considered objective functions may rapidly
increase once a complete understanding of the VMP problem is accomplished for practical
problems, where many different parameters should be ideally taken into account.
This chapter focuses in online formulations of a provider-oriented VMP problem in
federated-clouds [94] to provide relevant information for design and implementation of re-
source management systems, more specifically, resource allocation algorithms for VMP pro-
blems. Many relevant objectives for IaaS providers are considered in this work. To solve
the formulated online problem, five of the most studied heuristics were compared against a
Memetic Algorithm (MA) solving an offline formulation of the problem.

105
5.1 Background and Motivation

López-Pires and Barán recently proposed in [68] and [92] offline formulations of VMP pro-
blems considering many objectives, proposing novel memetic algorithms for solving the for-
mulated problems (see Chapters 3 and 4). Considering the on-demand model of cloud
computing with dynamic resource provisioning and dynamic workloads of cloud applications
[107], the resolution of the VMP problem should be performed as fast as possible. Con-
sequently, applying only offline formulations of the VMP problem with meta-heuristics as
solution technique, may not be appropriate for these dynamic environments. Therefore, as
previously mentioned, solution techniques with low computational complexity (e.g. heurist-
ics) are very studied for solving online formulations of VMP problems [94].
To help IaaS providers in the design and implementation of resource allocation algorithms
considering many objective functions, the following research question must be answered:
Which heuristics are most suitable for solving online VMP problems in federated-clouds con-
sidering many objectives?
In this context, Fang et al. presented in [42] a validation of a proposed power-aware
algorithm against well-known heuristics such as: BF, FF and WF. Additionally, Jin et
al. studied FFD and Dominant Resource First (DRF) algorithms [74], while Anand et al.
evaluated in [6] a proposed Integer Linear Programming (ILP) formulation and an FDD
algorithm to attend common limitations of ILP algorithms for large instances of NP-Hard
optimization problems. On the contrary, this chapter does not compare novel algorithms
against well-known heuristics as presented in the above mentioned works. The main goal of
the presented experimental comparison is to analyze the online nature of the VMP problem
optimizing many different objectives, identifying advantages and disadvantages of well-known
online heuristics against offline alternatives such as the MA proposed in [68].
Experimental results presented by Ihara et al. in [68] recommend the combination of
many objective functions into a single objective for IaaS environments, given the requirement
of obtaining solutions in a short period of time (see Chapter 4).

106
 To compare the considered algorithms (see Section 5.3), a provider-oriented VMP pro-
blem formulation is proposed for the optimization of four objective functions: (1) power
consumption, (2) economical revenue, (3) quality of service and (4) resource utilization
(see Section 5.2). To combine the above mentioned objectives, a weighted sum method is
considered. Experiments were performed to define appropriate weights for each objective
function (see Section 5.4).
The remainder of this chapter is structured in the following way: Section 5.2 summarizes
the proposed provider-oriented VMP problem formulation with many objectives. Section
5.3 describes the considered algorithms to solve the formulated problem, while Section 5.4
presents experimental workloads, obtained results and main findings of the comparison.

5.2 Part 4.A - Problem Formulation

This section presents a formulation of a VMP problem considering the optimization of the
following objective functions: (1) power consumption, (2) economical revenue, (3) quality
of service and (4) resource utilization. According to the taxonomy presented in [94], this
work focuses on a provider-oriented VMP for federated-cloud deployments, considering two
formulation types: (1) online and (2) offline.
An online problem formulation is considered when an algorithm makes decisions on-the-
fly without knowing upcoming events (e.g. online heuristics for VMP problems) [14]. On
the other hand, if an algorithm has a complete knowledge of the future events of a problem
instance, the formulation is called offline (e.g. MAs for VMP problems used in [68] and [92]).
The formulation of the proposed provider-oriented VMP problem is based on [68] and
could be enunciated as:

Given a cloud infrastructure composed by a set of PMs (H), a dynamic scenario composed
by a set of VMs requested at each discrete time t (V (t)) and the current placement of VMs
into PMs (x(t)), it is incrementally sought a placement of V (t) into H for the next time
instant (x(t + 1)), satisfying the constraints and optimizing the considered objectives.

107
 In online algorithms for solving the proposed VMP problem, placement decisions are
performed at each discrete time t, without knowledge of upcoming VM requests. On the
other hand, an offline algorithm have knowledge of the complete set of VM requests in
order to decide the placement of these VMs into available PMs. Consequently, the current
placement is not necessary for offline algorithms.

5.2.1 Part 4.A - Input Data

The proposed formulation of the VMP problem models a cloud computing datacenter, com-
posed by PMs and VMs, receiving the following information as input data:

• a set of n available PMs and their specifications;

• a set of m(t) VMs at each discrete time t and their specifications;

• the current placement at each discrete time t.

The set of PMs is represented as a matrix H ∈ Rn×4 , as presented in Equation (5.1).

Each PM Hi is represented by its physical resources. This chapter considers 3 different
physical resources (P r1 -P r3 ): CPU [ECU], RAM memory [GB] and network capacity [Mbps].
It is important to mention that the proposed notation is general enough to include more
characteristics associated to physical resources. Finally, the maximum power consumption
[W] is also considered, i.e.
 
 P r1,1 P r2,1 P r3,1 pmax1 
 
 
H= 
 ... ... ... ... 
 (5.1)
 
 
P r1,n P r2,n P r3,n pmaxn

where:

P r1,i : Processing resources of Hi in [ECU];

P r2,i : Memory resources of Hi in [GB];
P r3,i : Network capacity resources of Hi in [Mbps];
pmaxi : Maximum power consumption of Hi in [W];
n: Number of PMs.

108
 The set of VMs requested by customers at each discrete time t is represented as a matrix
V (t) ∈ Rm×5 , as presented in Equation (5.2). In this work, each VM Vj requires 3 different
virtual resources (V r1 -V r3 ): CPU [ECU], RAM memory [GB] and network capacity [Mbps].
It is important to mention that the notation could represent any other set of virtual resources
such as: Block Storage or even Graphics Processing Unit (GPU). An economical revenue Rj
[$] is associated to each VM Vj as well as a priority level represented as a Service Level
Agreement (SLA). The VMs try to lease the requested virtual resources for a fixed period of
discrete time.

 
 V r1,1 V r2,1 V r3,1 SLA1 R1 
 
 
V (t) = 
 ... ... ... ... ... 
 (5.2)
 
 
V r1,m(t) V r2,m(t) V r3,m(t) SLAm(t) Rm(t)

where:

V r1,j : Processing requirements of Vj in [ECU];

V r2,j : Memory requirements of Vj in [GB];
V r3,j : Network requirements of Vj in [Mbps];
Rj : Economical revenue for attending Vj in [$];
SLAj : SLA of Vj , where SLAj ∈ {1, 2, . . . , s}, being s the highest priority level;
m(t): Number of VMs at each discrete time t, then m(t) ∈ {1, . . . , mmax };
mmax : Maximum number of acceptable VMs.

Once a VM Vj is powered-off by the customer, its virtual resources are released, so the
provider can reuse them. For simplicity, in what follows the index j is not reused; therefore,
for this work Vj is not a function of time.
The current placement of VMs into PMs (x(t)) is also considered as an input data, taking
into account that the placement of requested VMs is performed incrementally at each discrete
time t. The placement at each discrete time t is represented as a matrix x(t) ∈ Rm(t)×n ,
defined as:

109
  
 x1,1 (t) x1,2 (t) ... x1,n (t) 
 
 
x(t) = 
 ... ... ... ... 
 (5.3)
 
 
xm(t),1 (t) xm(t),2 (t) . . . xm(t),n (t)

where:

xj,i (t) ∈ {0, 1}: Indicates if Vj is allocated (xj,i (t) = 1) or not (xj,i (t) = 0) for execution
on a PM Hi (i.e., xj,i (t) : Vj → Hi ) at a discrete time t.

5.2.2 Part 4.A - Output Data

The result of the proposed VMP problem at each discrete time t is a new placement for the
next time instant (x(t + 1)). This is represented by a matrix x(t + 1) ∈ Rm(t+1)×n .
 
 x1,1 ... x1,n 
 
 
x(t + 1) = 
 ... ... ... 
 (5.4)
 
 
xm(t+1),1 . . . xm(t+1),n

where:

xj,i (t + 1) ∈ {0, 1}: Indicates if Vj is allocated (xj,i (t) = 1) or not (xj,i (t) = 0) for execution
on a PM Hi (i.e., xj,i (t) : Vj → Hi ) at a discrete time t + 1.

5.2.3 Part 4.A - Constraints

Constraint 1: Unique Placement of VMs

A VM Vj should be located to run on a single PM Hi or alternatively for each Vj such that

SLAj < s, it could not be located into any PM. Consequently, this placement constraint is
expressed as:

n
X
xj,i (t) ≤ 1 ∀j ∈ {1, . . . , m(t)} (5.5)
i=1

110
Constraint 2: Assure SLA Provisioning

A VM Vj with the highest level of SLA (i.e. SLAj = s) must be mandatorily allocated to
run on a PM Hi . Consequently, this constraint is mathematically expressed as:

n
X
xj,i (t) = 1 ∀j such that SLAj = s (5.6)
i=1

Constraints 3-5: Physical Resources of PMs

A PM Hi must have sufficient available resources to meet the requirements of all VMs Vj
that are allocated to run on Hi . Consequently, these constraints can be formulated as:

m(t)
X
V r1,j × xj,i (t) ≤ P r1,i (5.7)
j=1

m(t)
X
V r2,j × xj,i (t) ≤ P r2,i (5.8)
j=1

m(t)
X
V r3,j × xj,i (t) ≤ P r3,i (5.9)
j=1

∀i ∈ {1, ..., n}, i.e. for all PM Hi .

Physical resources are considered as resources available for VMs, without considering
resources for the PM’s hypervisor.

5.2.4 Part 4.A - Objective Functions

Each of the considered objective functions must be formulated in a single optimization con-
text (i.e. minimization or maximization) and each objective function cost must be normalized
to be comparable and combinable as a single objective. This work normalizes each objective
function cost by calculating f̂i (x, t) ∈ R, where 0 ≤ f̂i (x) ≤ 1.

Power Consumption Minimization

Based on Beloglazov et al. [13], this work models the power consumption of PMs considering
a linear relationship with the CPU utilization of PMs. This can be represented by the sum
of the power consumption of each PM Hi .

111
 n
X
min f1 (x, t) = ((pmaxi − pmini ) × U r1,i (t) + pmini ) × Yi (t) (5.10)
i=1

where:

f1 (x, t): Total power consumption of PMs;

pmini : Minimum power consumption of a PM Hi . As suggested in [13],
pmini ≈ pmaxi ∗ 0.6;
U r1,i (t): Utilization ratio of resource 1 (in this case CPU) by Hi at instant t;
Yi (t) ∈ {0, 1}: Indicates if Hi is turned on (Yi (t) = 1) or not (Yi (t) = 0) at instant t.

Economical Revenue Maximization

For IaaS customers, cloud computing resources often appear to be unlimited and can be
provisioned in any quantity at any required time t [107]. Consequently, this work considered
a federated-cloud deployment architecture, where a main provider may attend requested
resources that are not able to be provided (e.g. a workload peak) by transparently leasing
low-price resources at a federated provider [49].
In this chapter, the maximization of the total economical revenue that a provider receives
for attending the requirements of its customers is achieved by minimizing the total costs
of leasing resources from alternative datacenters of the cloud federation. Equation (5.11)
represents the mentioned leasing costs, defined by the sum of the total costs of leasing each
VM Vj that is effectively allocated for execution on any PM of an alternative datacenter of
the cloud federation.
A provider must offer its idle resources to the cloud federation at lower prices than offered
to customers in the actual cloud market. The pricing scheme may depend on the particular
agreement between providers of the cloud federation [49]. Consequently, this work considers
that the main provider may lease requested resources (that are not able to provide) from
the cloud federation at 70% (X̂j = 0.7) of its price in markets (Rj ). This objective may be
formulated as:

m(t)
X
min f2 (x, t) = (Rj × Xj (t) × X̂j ) (5.11)
j=1

112
where:

f2 (x, t): Total costs for not allocating VMs in the main provider;
Xj (t) ∈ {0, 1}: Indicates if Vj is allocated for execution on a PM (Xj (t) = 1) or not
(Xj (t) = 0) at instant t;
X̂j : Indicates if Vj is allocated on the main provider (X̂j = 0) or on an
alternative datacenter of the cloud federation (X̂j = 0.7).

Note that X̂j is not a function of time. The decision of locating a VM Vj on a federated
provider is considered only in the placement process, with no possible migrations between
providers. The value of X̂j depends on the agreement celebrated between federated providers.

Quality of Service Maximization

In this work, the quality of service (QoS) maximization proposes the location of the maximum
number of VMs with the highest level of priority associated to the SLA prior to VMs with
smaller SLA. In case the main provider allocates a VM in a federated provider the QoS is
considered as 0. In order to evaluate this objective in a minimization context, the total of
SLA violations is minimized and formulated as:

m(t)
(Ĉ SLAj × SLAj × Xj (t) × Yˆj )
X
min f3 (x, t) = (5.12)
j=1

where:

f3 (x, t): Total SLA violations figure for a given placement;

Ĉ: Constant, large enough to prioritize services with a larger SLA over
the ones with a lower SLA;
Yˆj ∈ {0, 1}: Indicates if Vj is allocated for its execution on the main provider (Yˆj = 0)
or on an alternative datacenter of the cloud federation (Yˆj = 1).

Resources Utilization Maximization

An efficient utilization of resources is a relevant management challenge to be addressed by

IaaS providers. This chapter proposes the resource utilization maximization strategy by
minimizing the average ratio of wasted resources on each PM Hi (i.e. resources that are not
allocated to any VM Vj ). This objective function is formulated in Equation (5.13).

113
 " !#
Pn U r1,i (t) + · · · + U rr,i (t)
i=1 1− × Yi (t)
r
min f4 (x, t) = Pn (5.13)
i=1 Yi (t)

where:

f4 (x, t): Average ratio of wasted resources;

U r1,i (t): Utilization ratio of resource 1 (in this case CPU) by Hi at instant t;
U rr,i (t): Utilization ratio of resource r (any resource) by Hi at instant t;
r: Number of considered resources. In this chapter 3: CPU, RAM memory and
network capacity.

The following sub-section summarizes the main considerations taken into account to
combine the four presented objective functions into a single objective.

5.2.5 Normalization and Scalarization Method

Experimental results presented in [68] recommend the combination of many normalized

objective functions into a single objective (e.g. minimum distance to origin) for IaaS envir-
onments. This chapter considers a weighted sum method to combine many objectives into a
single objective.
As previously mentioned, each objective function cost must is normalized to be compar-
able and combinable as a single objective. This work normalizes each objective function cost
by calculating f̂i (x) ∈ R, where 0 ≤ f̂i (x) ≤ 1, as defined in [119]:

fi (x) − fi (x)min
f̂i (x) = (5.14)
fi (x)max − fi (x)min
where:

f̂i (x): Normalized cost of objective function fi (x);

fi (x): Cost of objective function fi (x);
fi (x)min : Minimum possible cost for fi (x);
fi (x)max : Maximum possible cost for fi (x).

114
 Finally, the four previously presented normalized objective functions are combined into
a single objective considering a weighted sum method, expressed as follows:

q
X
min f (x) = f̂i (x) × wi (5.15)
i=1

where:

f (x): Single objective function combining each fi (x);

f̂i (x): Normalized cost of objective function fi (x);
wi : Weight of importance associated to fi (x);
q: Number of objective functions. In this case 4.

In this work, values of each weight wi associated to objective functions fi (x) were obtained
empirically by analyzing a large number of experiments to be presented in Section 5.4.

5.3 Part 4.A - Evaluated Algorithms

To analyze alternatives to solve the formulated VMP problem (see Section 5.2), an experi-
mental comparison of five different online deterministic heuristics against an offline memetic
algorithm with migration of VMs was performed. This section summarizes the six different
algorithms considered in the experimental comparison presented in this work.

5.3.1 A1: First-Fit (FF)

In the First-Fit (FF) algorithm, requested VMs Vj are allocated on the first PM Hi with
available resources (see Section 5.2.3). Interested readers can refer to [42] for details on FF,
BF and WF algorithms applied to VMP problems.

5.3.2 A2: Best-Fit (BF)

The Best-Fit (BF) algorithm, allocates requested VMs Vj on the first PM Hi with available
capacity from a sorted list of PMs in increasing order by a score associated to each PM.
The score of a PM Hi is determined as a sum of its ratios of unutilization of resources, as
detailed in [42].

115
5.3.3 A3: Worst-Fit (WF)

In the Worst-Fit (WF) algorithm, VMs Vj are allocated on the first available PM Hi of a
decreasingly ordered list of PMs based on the score of the PM, inversely to the operation of
the BF algorithm. For more details, refer to [42].

5.3.4 A4: First-Fit Decreasing (FFD)

The First-Fit Decreasing (FFD) algorithm operates similarly to the previously presented FF
algorithm. The main difference with the FF algorithm is that FFD algorithm sorts the list
of requested VMs Vj in decreasing order by requested CPU resources, as described in [47].

5.3.5 A5: Best-Fit Decreasing (BFD)

The Best-Fit Decreasing (BFD) algorithm has a similar behavior to the BF algorithm. The
main difference between the BF algorithm and the BFD one is that the BFD sorts the list of
requested VMs Vj in decreasing order by requested CPU resources. Interested readers can
refer to [35] for details on the BFD algorithm applied to VMP problems.

5.3.6 A6: Memetic Algorithm (MA)

A Memetic Algorithm (MA) is considered for an offline alternative to solve the formulated
VMP problem taking into account many objectives (see Section 5.2). This algorithm is based
on the one proposed in [68] by Ihara et al.
The considered algorithm may be classified as a meta-heuristic, following an evolutionary
process to find appropriate solutions. Basically, the considered MA follows this evolutionary
behavior: solutions are selected from an evolutionary set of solutions (or population). Cros-
sover and mutation operators are applied as usual, and eventually solutions are repaired, as
there may be unfeasible solutions. Improvements of solutions of the evolutionary population
may be generated using local optimization operators. Next, a new evolutionary popula-
tion is selected from the union of the (until that generation) best population and the set of
improved solutions. The evolutionary process is repeated until the algorithm meets a stop-

116
ping criterion (such as a maximum number of generations), returning from the evolutionary
population the solution with minimum cost of f (x).

5.4 Part 4.A - Experimental Evaluation

In this work, an experimental comparison of five of the most studied online deterministic
heuristics was performed considering several experimental workloads. The quality of solu-
tions obtained by the evaluated online heuristics was compared to the average performance
(in ten runs) of an offline MA that has complete knowledge of future VM requests and the
ability of migrating VMs between PMs.

5.4.1 Part 4.A - Experimental Environment

The six evaluated algorithms were implemented using ANSI C programming language. The
source code is available online1 , as well as all the considered experimental data. Experiments
were performed on a Linux Operating System with an Intel Core i7 2.3 GHz CPU and 12
GB of RAM memory.

1
http://github.com/DynamicVMP/vmpCompetitiveAnalysis

15 W1 : Poisson λ = 10
W2 : Poisson λ = 50
W3 : Poisson λ = 70
Number of created VMs

W4 : Uniform
12

0
0 10 20 30 40 50 60 70 80 90 100

Time (t)

Figure 5.1: Experimental Workload Traces: Number of created VMs as a function of time t.

117
Table 5.1: Considered Data for Generated Experimental Workload Traces (W1 to W4 ).
Details in [112].
Parameter Description Input Data
1. Environment No overbooking, No elasticity
2. Workload Trace Duration [t] 100
3. Number of Cloud Datacenters 1
4. Number of Cloud Services 100
5. Number of VMs per Service 1
W1 : Poisson(λ = 10)
W2 : Poisson(λ = 50)
6. VMs creation time [t]
W3 : Poisson(λ = 70)
W4 : Uniform(0,100)
7. CPU Resources [ECU] Uniform(1,8)
8. RAM Resources [GB] Uniform(1,8)
9. Network Capacity [Mbps] Uniform(10,1000)
10. Revenue of VMs (Rj ) [$] Uniform(0.1,1.5)
11. SLA Level of VMs (SLAj ) Uniform(1,5)

Physical resources (matrix H) represent an homogeneous IaaS cloud composed by 10 PMs

with the following specifications: 8 [ECU] of CPU, 10 [GB] of RAM memory, 780 [Mbps] of
network capacity and 960 [W] of maximum power consumption (see Section 5.2.1).
In this work, requested virtual resources (matrix V (t)) were considered using four different
workload traces generated using a Workload Trace Generator for provider-oriented VMP
problems [112] available for research purposes2 .
Each considered workload trace simulates customers requests to allocate a set of 100 VMs
following different Probability Distribution Functions (PDFs) for the VM request arrivals.
Three workload traces follow a Poisson PDF with different expected values (λ): (W1 ): λ =
10, (W2 ): λ = 50 and (W3 ): λ = 70. In this case, different values of λ may represent
workload peaks at different time instants t (see Figure 5.1). The fourth workload trace (W4 )
follows an Uniform PDF, representing a stable workload of VM requests. All parameters
considered for generating the experimental workload traces are presented in Table 5.1.
In order to effectively analyze the described federated-cloud deployment architecture
and provider-oriented VMP formulation considering many objectives, experimental workload
traces requested more virtual resources than the available ones in the considered main cloud
computing datacenter.
2
http://wtg.cba.com.py

118
 Table 5.2: Objective Function Costs of Evaluated Algorithms.
Algorithm
A1: FF A2: BF A3: WF A4: FFD A5: BFD A6: MA
Workload
W1 : Poisson λ = 10 3.2927 3.3098 3.5250 3.0205 3.1392 2.6096
W2 : Poisson λ = 50 2.4602 2.5112 2.4811 2.4602 2.4555 2.0001
W3 : Poisson λ = 70 1.7054 1.6458 1.7054 1.6458 1.6458 1.3588
W4 : Uniform 3.1875 3.1556 3.0489 3.0907 3.1556 2.3420
Average 2.6615 2.6556 2.6901 2.5543 2.5990 2.0776
Ranking 5th 4th 6th 2nd 3th 1st

Experiments could be summarized as: For each considered workload trace (W1 to W4 ),
one run of the following deterministic heuristics was performed in an online context: (1) FF,
(2) BF, (3) WF, (4) FFD and (5) BFD. Considering the randomness of the MA compared
against the previously mentioned heuristics, ten runs of the mentioned algorithm were per-
formed. The average values of the ten runs were considered for the experimental comparison,
as summarized in Table 5.2, where the offline MA clearly outperforms all 5 online heuristics
as expected, given that it uses complete knownledge of VM requests and migration of VMs
to optimize the objective function presented in Equation (5.15).

5.4.2 Objective Functions Weights

To determine appropriate values for the weights wi associated to each objective function
fi (x), an exploration of the VMP problem domain was performed. In this context, 1000
feasible solutions (x1 to x1000 ) were randomly generated by the MA (A6) considering: H as
described in Section 3.4.1, V (t) as presented in entorno00-1 (a benchmark available online2 ),
highest priority of VMs s = 4 and Ĉ = 1000.
Obtained values of each objective function fi (x) were normalized in f̂i (x), as described
in Section 5.2.5. Consequently, each weight wi was defined as:

1000
wi = P1000 (5.16)
k=1 f̂i (xk )

resulting:

w1 : 1.3903; w2 : 2.1379; w3 : 2.7393; w4 : 1.4586.

119
 Table 5.3: Objective Function Costs of Evaluated Heuristics.
Workload A1: FF A2: BF A3: WF A4: FFD A5: BFD Best
fi (x)
f1 (x, t) 0.9240 0.9107 0.8929 0.9255 0.9208 A3
f2 (x, t) 0.0372 0.0373 0.0391 0.0359 0.0359 A4,A5
W1

f3 (x, t) 0.6104 0.6139 0.6670 0.5101 0.5611 A4

f4 (x, t) 0.1756 0.1933 0.2556 0.1778 0.1679 A5
f1 (x, t) 0.5900 0.5756 0.5612 0.5900 0.5903 A3
f2 (x, t) 0.0404 0.0406 0.0420 0.0404 0.0404 A4,A5
W2

f3 (x, t) 0.4790 0.4975 0.4715 0.4790 0.4790 A3

f4 (x, t) 0.1652 0.1789 0.2189 0.1652 0.1618 A5
f1 (x, t) 0.4146 0.4198 0.4203 0.4198 0.4198 A1
f2 (x, t) 0.0243 0.0235 0.0245 0.0235 0.0235 A2,A4,A5
W3

f3 (x, t) 0.3155 0.3003 0.3103 0.3003 0.3003 A2,A4,A5

f4 (x, t) 0.1457 0.1296 0.1497 0.1296 0.1296 A2,A4,A5
f1 (x, t) 0.8549 0.8544 0.8694 0.8598 0.8544 A2,A5
f2 (x, t) 0.0375 0.0364 0.0356 0.0371 0.0364 A3
W4

f3 (x, t) 0.5311 0.5283 0.4884 0.4931 0.5283 A3

f4 (x, t) 0.3177 0.3034 0.2919 0.3188 0.3034 A3
f1 (x, t) 0.6959 0.6901 0.6859 0.6988 0.6963 A3
Average

f2 (x, t) 0.0349 0.0345 0.0353 0.0343 0.0341 A5

f3 (x, t) 0.4841 0.4851 0.4844 0.4457 0.4672 A4
f4 (x, t) 0.2011 0.2014 0.2291 0.1979 0.1907 A5

5.4.3 Comparison of Algorithms

This section summarizes the main findings obtained in the experimental comparison of al-
gorithms for the VMP formulation with many objectives presented in Section 5.2.
The main goal of the experimental comparison presented in this section is to define
which heuristics are most suitable for solving online formulations of provider-oriented VMP
problems in federated-clouds considering many objectives.

120
 Table 5.2 summarizes the costs of objective function f (x), that combines the four con-
sidered objectives (see Section 5.2.5), summarizing performed experiments. It is worth not-
ing that according to the experimental results presented in Table 5.2, there is no evalu-
ated heuristic that outperforms all other alternatives in all experimental workload traces.
Consequently, Table 5.2 also presents the average costs of objective function f (x) in the 4
experimental workloads as well as the corresponding ranking. It seems that FFD algorithm
is the best heuristic amount the five considered ones closely followed by BFD (logically, MA
is first one in this ranking).
To better understand the presented comparison, Table 5.3 details the average normalized
costs of each objective function f1 (x, t) to f4 (x, t), to analyze particular preferences of IaaS
providers for the optimization of the considered objectives (e.g. power consumption could
be more important in hours where the electricity price is higher).
Based on the information presented in Tables 5.2 and 5.3, the Main Findings (MF) of
the experimental comparison performed in this chapter are summarized as follows:

MF1: There is no evaluated heuristic that can clearly be considered as the best alternative
for all objective functions, considered simultaneously.

Additionally, none of the evaluated heuristics performed equally well in all 4 experimental
workloads (see Table 5.2). Consequently, an heuristic performing good enough in average
when considering different types of workloads could be sufficiently convenient.
A more detailed evaluation could be performed to obtain information for conjuntural pref-
erences of IaaS providers (see Table 5.3, where the best heuristic for each objective function
and each workload trace is highlighted in the last column).

MF2: FFD heuristic (A4) was the algorithm that outperforms all other evaluated heuristics
taking into account average results in performed experiments.

According to the average performance of each evaluated heuristic (see Table 5.2), the follo-
wing ranking was built: (1st) FDD (A4), (2nd) BFD (A5), (3th) BF (A2), (4th) FF (A1),
and (5th) WF (A3), where BFD follows very close the average performance of FDD (with a
difference of only 1.75%).

121
MF3: WF algorithm (A3) is suggested for workloads that can be considered stable (i.e. no
workload peaks).

For experimental workload trace W4 that considers an Uniform PDF for VM request arrivals,
A3 clearly presented the best performance obtaining minimum average results in Table 5.2.
It also performed as the best algorithm for 3 out of the 4 objectives, considering only W4
(see Table 5.3).

MF4: The WF (A3), BFD (A5), FFD (A4) and BFD (A5) algorithms are recommended
for f1 (x, t), f2 (x, t), f3 (x, t) and f4 (x, t) objective functions respectively, when there is a
preferred objective function (lexicographic order).

The BFD algorithm (A5) performed as the best algorithm on average for f2 (x, t) and f4 (x, t).
For f1 (x, t) the best alternative seems to be A3 while A4 could be considered as the best for
f3 (x, t) (see Table 5.3).

MF5: As expected, the offline MA (A6) outperformed all evaluated online heuristics in all
experimental workloads.

An offline MA was compared to the five evaluated heuristics to experimentally demonstrate

that online decisions made along a simulation affects the quality of solutions. Clearly, offline
algorithms such as the considered MA present a substantial advantage over online heuristics
when considering the quality of solutions for the objective function f (x) (see Table 5.2). This
advantage is presented for the following two main reasons: (1) offline algorithms have a com-
plete knowledge of future events of a problem instance and (2) only A6 considered migrations
of VMs between PMs in the comparison, reconfiguring the placement when convenient.
Having a complete knowledge of future VMs requests is considered unrealistic for IaaS envir-
onments [14]. Consequently, online algorithms, such as the heuristics evaluated in this work,
are a good alternative for VMP problems in IaaS environments. The evaluated offline MA
(A6) improves the quality of solutions with migrations of VMs between PMs, at expense of
costs associated to placement reconfigurations, as seen in Table 5.4 where the average VMs
migration overhead of each workload is presented as: (1) total number of VM migrations
and (2) total RAM memory migrated.

122
 Table 5.4: VMs Migration Overhead for MA (A6).
Migration
# of VMs Memory in [GB]
Workload
Poisson λ = 10 1493.7 4865.9
Poisson λ = 50 740.0 2775.1
Poisson λ = 70 643.6 2415.5
Uniform 1416.7 5329.8

MF6: As expected, the execution time of all online heuristics are considerably shorter than
the offline alternative.

Considering the low complexity of the evaluated heuristics, these algorithms are able to find
promising solutions in short periods of time (a main concern for IaaS providers solving VMP
problems) at the expense of not exploring solutions that can potentially result in better
quality solutions.

Clearly, any offline alternative that intelligently explore a large space of feasible solutions
will result in higher execution times, mainly considering the exponential computational com-
plexity of the VMP problem.

123
6 MaVMP for IaaS under Uncertainty
This chapter proposes a complex Infrastructure as a Service (IaaS) environment for VMP
problems, considering service elasticity and overbooking of physical resources. To the best of
the authors’ knowledge, there is no published work considering these fundamental criteria,
directly related to the most relevant dynamic parameters in the specialized literature [112].
In order to model this complex IaaS environment for VMP problems, cloud services (i.e. a
set of inter-related VMs) are considered instead of only isolated VMs.
It is worth remembering that VMP is a NP-Hard combinatorial optimization problem
[130]. From an IaaS provider perspective, the VMP problem is mostly formulated as an
online problem and must be solved with short time constraints [94].
Online decisions made along the operation of a dynamic cloud computing infrastructure
negatively affects the quality of obtained solutions in VMP problems when comparing to
offline decisions [98]. In this context, offline algorithms present a substantial advantage
over online alternatives. Unfortunately, offline formulations are not appropriate for highly
dynamic environments for real-world IaaS providers, where cloud services are requested
dynamically according to current demand.
In what follows, this chapter presents a two-phase optimization scheme, decomposing the
VMP problem into two different sub-problems, combining advantages of online and offline
VMP formulations considering a complex IaaS environment. The presented optimization
scheme for the VMP problem introduces novel methods to decide when to trigger placement
reconfigurations with migration of VMs between PMs (defined as VMPr Triggering) and
what to do with cloud services requested during placement recalculation times (defined as
VMPr Recovering).
For IaaS customers, cloud computing resources often appear to be unlimited and can
be provisioned in any quantity at any required time [107]. Consequently, this chapter con-
siders again (as in Chapter 5) a basic federated-cloud deployment architecture for the VMP
problem.

124
 It is important to consider that more than 60 different objective functions have been pro-
posed for VMP problems [94]. In this context, the number of considered objective functions
may rapidly increase once a complete understanding of the VMP problem is accomplished for
practical problems, where several different parameters should be ideally taken into account.
Consequently, a renewed formulation of the VMP problem is presented, considering the op-
timization of the following four objective functions: (1) power consumption, (2) economical
revenue, (3) resource utilization and (4) reconfiguration time.
Due to the randomness of customer requests, VMP problems should be formulated un-
der uncertainty. This work presents a scenario-based uncertainty approach for modeling
uncertain parameters, considering a two-phase optimization scheme for VMP problems in
the proposed complex IaaS environments.
An experimental evaluation against state-of-the-art alternative approaches for VMP pro-
blems was performed considering 80 different workloads in 5 different CPU load scenarios,
totalizing 400 experimental scenarios. Experimental results indicate that the proposed VMPr
Triggering and Recovering methods of the presented two-phase optimization scheme outper-
forms other evaluated alternatives, improving the quality of solutions.

In summary, the main contributions of this chapter are:

• A first proposal for a complex IaaS environment for VMP problems considering ser-
vice elasticity, including both vertical and horizontal scaling of cloud services, as well
as overbooking of physical resources, including server (CPU and RAM) as well as
networking resources [112].

• A two-phase optimization scheme for VMP problems, combining advantages of both

online and offline VMP formulations in the proposed IaaS environment, introducing
a prediction-based VMPr Triggering method to decide when to trigger a placement
reconfiguration (Research Question 1 ) as well as an update-based VMPr Recovering
method to decide what to do with VMs requested during placement recalculation times
(Research Question 2 ).

• A first scenario-based uncertainty approach for modeling the following relevant un-
certain parameters of the proposed complex IaaS environment: (1) virtual resources

125
 capacities (vertical elasticity), (2) number of VMs that compose cloud services (hori-
zontal elasticity), (3) utilization of CPU and RAM memory virtual resources (relevant
for overbooking) and (4) utilization of networking virtual resources (also relevant for
overbooking).

• A first formulation of a VMP problem considering the above mentioned contributions,

for the optimization of the following four objective functions: (1) power consumption,
(2) economical revenue, (3) resource utilization, as well as (4) placement reconfiguration
time.

• An experimental evaluation of the presented two-phase optimization scheme against

state-of-the-art alternatives for VMP problems, considering 400 different scenarios.

The remainder of this chapter is structured in the following way: preliminary concepts
and research challenges addressed in this work are introduced in Section 6.1, while related
works and motivation of this work are summarized in Section 6.2. Section 6.3 presents the
proposed uncertain VMP problem formulation considering four objectives, while Section 6.4
presents details on the design and implementation of evaluated alternatives to solve the
proposed renewed formulation of the VMP problem. Experimental results are summarized
in Section 6.5.

6.1 Preliminary Concepts and Research Challenges

The following sub-sections introduce relevant concepts related to the considered IaaS en-
vironments for VMP problems, a brief motivation for decomposing the VMP problem into
two different sub-problems in a two-phase optimization scheme as well as uncertainty issues
related to resource allocation in cloud computing. Additionally, the main challenges and
research questions addressed in this work are also briefly introduced.

6.1.1 IaaS Environments for VMP Problems

In real-world environments, IaaS providers dynamically receive requests for the placement
of cloud services with different characteristics according to different dynamic parameters.

126
In this context, preliminary results of the authors identified that the most relevant dynamic
parameters in the VMP literature are [112]: (1) resource capacities of VMs (associated to
vertical elasticity) [79], (2) number of VMs of a cloud service (associated to horizontal elasti-
city) [146] and (3) utilization of resources of VMs (relevant for overbooking) [6]. Considering
the above mentioned dynamic parameters, environments for IaaS formulations of provider-
oriented VMP problems could be classified by one or more of the following classification
criteria: (1) service elasticity and (2) overbooking of physical resources [112]. A cloud ser-
vice may represent virtual infrastructures for basic services such as Domain Name Service
(DNS), web applications or even elastic applications such as MapReduce programs [112].
An elastic cloud service could request additional resources to scale-up or scale-out the
applications’ resources to be able to support current demand, where IaaS providers should
be able to satisfy these requirements accordingly. From an IaaS provider perspective, elastic
cloud services are usually considered more important than non-elastic ones. Different IaaS
environments could be formulated considering one of the following service elasticity altern-
atives: no elasticity, horizontal elasticity, vertical elasticity or both horizontal and vertical
elasticity [112].
Additionally, resources of VMs are dynamically used, giving space to re-utilization of
idle resources that were already reserved. In this context, IaaS environments identified
in [112] may also consider one of the following overbooking alternatives: no overbooking,
server resources overbooking, network resources overbooking or both server and network
overbooking. This work formulates a VMP problem taking into account the most complex
IaaS environment identified in [112], that considers both types of service elasticity and both
types of overbooking of physical resources.
To the best of the authors’ knowledge, there is no published work considering this com-
plex dynamic environment [112]. IaaS providers efficiently solving VMP problems in this
complex dynamic environment could represent a considerable advance on this research area
and consequently cloud computing datacenters will be able to adapt according to trends of
requirements with sufficient flexibility and efficiency.

127
6.1.2 Two-Phase Optimization Schemes for VMP problems

The VMP could be formulated as both online and offline optimization problems [94]. A VMP
problem formulation is considered to be online when solution techniques (e.g. heuristics)
dynamically make decisions on-the-fly [14]. On the other hand, if solution techniques solve a
VMP problem considering a static environment where VM requests do not change over time,
the VMP problem formulation is considered to be offline [90]. Considering the on-demand
model of cloud computing with dynamic resource provisioning and dynamic workloads of
cloud applications [107], the resolution of VMP problems should be performed as fast as
possible in order to be able to support these dynamic requirements. In this context, the
VMP problem for basic IaaS environments was mostly studied in the specialized literature
considering online formulations, knowing that VM requests change according to current
demand [94].
It is important to consider that online decisions made along the operation of a cloud com-
puting infrastructure negatively affects the quality of obtained solutions of VMP problems
when comparing to offline decisions [98]. Clearly, offline algorithms present a substantial
advantage over online alternatives, when considering the quality of obtained solutions. This
advantage is reported in the literature for the following two main reasons: (1) an offline
algorithm solves a VMP problem considering a static environment where VM requests do
not change over time and (2) it considers migration of VMs between PMs, reconfiguring the
placement when convenient.
To improve the quality of solutions obtained by online algorithms, the VMP problem
could be formulated as a two-phase optimization problem, combining advantages of online
and offline formulations for IaaS environments [98].
In this context, VMP problems could be decomposed into two different sub-problems:
(1) incremental VMP (iVMP) and (2) VMP reconfiguration (VMPr). This two-phase op-
timization scheme combines both online (iVMP) and offline (VMPr) algorithms for solving
each considered VMP sub-problem (see Figure 6.1).

128
Figure 6.1: Two-phase optimization scheme for VMP problems considered in this work,
presenting a basic example with a placement recalculation time of β = 2 (from t = 2 to
t = 4) and a placement reconfiguration time of γ = 1 (from t = 4 to t = 5).

The iVMP sub-problem is considered for dynamic arriving requests, where VMs could be
created, modified and removed at runtime. Consequently, this sub-problem should be formu-
lated as an online problem and solved with short time constraints, where existing heuristics
could be reasonably appropriate. Additionally, the VMPr sub-problem is considered for im-
proving the quality of solutions obtained in the iVMP phase, reconfiguring the placement
through VM migration. This sub-problem could be formulated offline, where alternative
solution techniques could result more suitable (e.g. meta-heuristics).
The VMPr phase is triggered according to a given VMPr Triggering method. Once the
VMPr is triggered, the placement of VMs at discrete time t is recalculated during β discrete
time slots (i.e. recalculation time). In Figure 6.1, β = 2, from t = 2 to t = 4. It is
important to notice that the recalculated placement is potentially obsolete, considering the
offline nature of the VMPr phase. In fact, while the VMPr is making its calculation, the
iVMP still may receive and serve arriving requests, making obsolete the VMPr calculated
solution; therefore, the recalculated placement must be recovered accordingly using a VMPr
Recovering method, before complete reconfiguration is performed. The recovering process
as well as the migration of VMs are performed in γ discrete time slots (i.e. reconfiguration
time), where γ may vary according to the maximum amount of RAM to be migrated.

129
 Based on the literature review to be summarized in Section 6.2, the considered iVMP +
VMPr optimization scheme has been briefly studied in the VMP literature. Consequently,
several challenges for IaaS environments remain unaddressed or could be improved, consi-
dering that only basic methods have been proposed. This work identifies two main research
questions related to the considered two-phase optimization scheme, specifically for VMPr
Triggering and VMPr Recovering methods:

• Research Question 1 (RQ1): when or under which circumstances the VMPr phase
should be triggered? (VMPr Triggering method).

• Research Question 2 (RQ2): what should be done with cloud service requests
arriving during recalculation time in the VMPr phase? (VMPr Recovering method).

This work proposes novel VMPr Triggering and VMPr Recovering methods to decide
when to trigger a placement reconfiguration with migration of VMs between PMs and what
to do with cloud services requested during recalculation times. The proposed methods were
evaluated against existing state-of-the-art approaches considering 400 scenarios.

6.1.3 Uncertainty in Cloud Computing

Extensive research of uncertainty issues could be found in several fields such as: computa-
tional biology and decision making in economics, just to cite a few. Particularly, studies of
uncertainty for cloud computing are limited and uncertainty in resource allocation has not
been adequately addressed, representing several relevant research challenges [136].
According to Tchernykh et al. [136], uncertainties in cloud computing could be grouped
into: (1) parametric and (2) system uncertainties. Parametric uncertainties may represent
incomplete knowledge and variation of parameters, as presented in the considered uncertain
VMP problem. The analysis of these uncertainties may measure the effect of random para-
meters on model outputs. Research challenges in the context of this work include designing
novel resource management strategies to handle uncertainty in an effective way. IaaS pro-
viders must support requests for virtual resources in highly dynamic environments. Due
to the randomness of customer requests, algorithms for solving VMP problems should be
evaluated under uncertainty, considering several relevant uncertain parameters.

130
 This work analyzes the following uncertain parameters: (1) virtual resource capacities
(vertical elasticity), (2) number of VMs that compose cloud services (horizontal elasticity),
(3) utilization of CPU and RAM memory virtual resources (relevant for overbooking) and
(4) utilization of networking virtual resources (also relevant for overbooking).
In this work, uncertainty is modeled through a finite set of well-defined scenarios S [4]
(explained in Section 6.5.1). When parameters are uncertain, it is important to find solutions
that are acceptable for each considered scenario. Consequently, the main objective is not
to find an absolute optimum for a unique still unknown scenario but rather solutions that
behave good enough under modeled uncertainties. For this purpose, several criteria can
be applied to select among solutions such as: (1) average, (2) maximum and (3) minimum
objective costs.

6.2 Related Works and Motivation

To the best of the authors’ knowledge, there is no published work considering uncertainty
of parameters for provider-oriented VMP problem formulations. Consequently, this section
mainly focus on describing IaaS environments considered on each related work, as well as
already proposed VMPr Triggering and VMPr Recovering methods, when applicable. A
summary of considered related works is presented in Table 6.1.
Calcavecchia et al. studied in [20] a practical model of cloud service placement for a
stream (or workload) of requests where inter-related VMs are created and destroyed, con-
sidering CPU overbooking and static reservation of VMs resources. The mentioned cloud
service placement model is composed by two phases: (1) continuous deployment (or iVMP)
and (2) ongoing optimization (or VMPr). The continuous deployment is performed by
a Best-Fit Decreasing (BFD) heuristic while a Backward Speculative Placement (BSP) is
performed in the ongoing optimization phase. To improve a current placement, the ongoing
optimization is periodically triggered for the duration of the workload and canceled whenever
a new request is received.

131
 Table 6.1: Summary of IaaS environments and VMPr methods already studied in related works. N/A indicates a Not Applicable criterion.
Reference Overbooking Type Elasticity Type VMPr Triggering VMPr Recovering
[20] CPU Not Considered Periodically Cancellation
[151] Not Considered Not Considered Periodically Not Considered
[46] Not Considered Not Considered Periodically Not Considered
[87] Not Considered Not Considered Periodically Not Considered

132
[44] CPU and RAM Not Considered Periodically Not Considered
[156] Not Considered Not Considered Periodically Not Considered
[132] Not Considered Not Considered Continuously Not Considered
[13] CPU Not Considered Threshold-based N/A
[123] CPU, RAM and Network Not Considered Threshold-based N/A
[137] CPU Horizontal Threshold-based N/A
This work CPU, RAM and Network Vertical and Horizontal Prediction-based Update-based
 To ensure quick responses to VMP requests while improving energy efficiency, Yue et al.
proposed in [151] a two-phase optimization strategy, where VMs are deployed at runtime
and consolidated periodically. The placement of VMs is performed using an Improved Mul-
tidimensional Space Partition Model (IMSPM). Along with the IMSPM, a Modified Energy
Efficient Algorithm with balanced resource utilization (MEAGLE) is used to deploy VMs as
the first phase of the optimization (or iVMP). To perform the consolidation (or VMPr), a
Live Migration Algorithm Based on a Basic Set (LMABBS) is presented. The arrival of VMs
during the VMPr is not considered in this work and consequently, neither VMPr Recovering
method nor overbooking or elasticity are considered.
Considering the computational complexity of the VMP (NP-Hard problem), a decentrali-
zed decision is proposed by Feller et al. in [46], based on a Peer-to-Peer (P2P) communication
model among PMs. To accomplish a consolidation of PMs to minimize power consumption,
the mentioned work explored methods to allocate and migrate VMs in the minimum number
of PMs. An incremental allocation is performed using a First-Fit Decreasing (FFD) heuristic
to allocate requested VMs at runtime (for iVMP). A Virtual Machine Consolidation (VMC)
procedure using a cost-aware Ant Colony Optimization (ACO) is executed periodically to
consolidate VMs (for VMPr).
Li et al. proposed in [87] a hybrid approach for the VMP problem. As a first phase,
incoming VM requests are grouped into a queue of requests to gather information about
their requirements. Once the queue is full, an offline approach is considered for the VMP,
taking into account collected information. A Migration-Based Virtual Machine Placement
(MBVMP) is proposed (for VMPr), considering the migration time when planning allocation
of VMs and migration of already allocated requests to consolidate VMs and release resources.
The VMPr is executed every time the queue is full.
In [44], Farahnakian et al. proposed a Self-Adaptive Resource Management System
(SARMS) considering an Adaptive Utilization Threshold (AUT) mechanism to classify PMs
as overloaded or underloaded. To allocate incoming VM requests (during iVMP) it uses a
Best-Fit Decreasing (BFD) heuristic. The proposed SARMS triggers a VMP optimization
algorithm (or VMPr) periodically. The VMPr has two steps, migration of VMs from over-

133
loaded PMs and consolidation of VMs to release resources of underloaded PMs, switching
them to sleep mode. Since the utilization of resources is considered to define if a PM is
capable of allocating a VM request, overbooking is supported for CPU and RAM memory
resources. Requested VM resources do not change during the cloud service life-cycle.
According to Zheng et al. in [156], the VMP problem can be divided into two sub-
problems: incremental placement (VMiP) and placement consolidation (VMcP). A Best-Fit
(BF) heuristic is considered for the incremental placement phase (or iVMP). Additionally,
a VMP Biogeography-Based Optimization (VMPBBO) is proposed to optimize resources
wastage and power consumption in the consolidation process (or VMPr). The VMPr is
triggered periodically and no mention is made to VM request arrival during consolidation
process. Finally, neither overbooking nor elasticity are considered.
Svärd et al. studied in [132] a resource management system for continuous datacenter
consolidation, based on a combination of management actions like suspend / resume PMs
and VMs as well as live migration of VMs. The behavior of the proposed solution follows
a set of prioritized events: (1) VM arrival, (2) VM exit and (3) PM crash. An incremental
placement of VMs (or iVMP) considers a Best-Fit (BF) heuristic to find an appropriate PM
to host requested VMs. The VMPr could be executed in two cases: (1) after total allocation
of a list of VM requests at each discrete time or (2) when the continuous iVMP does not
find any PM to host a VM request. The arrival of VM requests during the VMPr process is
not considered; neither overbooking nor elasticity are studied.
Beloglazov et al. identified in [13] two stages for the VMP problem: (1) initial admission
of VMs and (2) optimization of the current placement. For the admission of VMs (or
iVMP) a Modified Best-Fit Decreasing (MBFD) algorithm is considered, using the CPU
utilization of VMs to sort a list of VM requests and allocate each VM into a PM that
provides the minimum increment in power consumption. Additionally, the optimization of
the current placement (or VMPr) is triggered whenever an overloaded or underloaded PM is
detected, according to well-defined CPU utilization thresholds. In this case, the VMPr runs
distributively for each overloaded or underloaded PM to migrate VMs from overloaded PMs
until each PM is appropriately loaded, consolidating VMs from underloaded PMs to decrease

134
the number of running PMs to the minimum possible number. It is important to consider
that this threshold-based triggering represents a decentralized decision process, relaxing the
computational complexity of the VMP problem. Consequently, it is not necessary to consider
the arrival of VM requests during the reconfiguration because no offline centralized decision
is performed. Considering the VMPr, a selection process is performed to determinate which
VMs should be migrated (all in case of underloaded PMs). Selected VMs are allocated by
the MBFD algorithm into PMs considering CPU overbooking.
Shi et al. proposed in [123] an online VMP formulation with a two-phase algorithm called
Two-Phase Online Virtual Machine Placement (TPOVMP). Multiple resources of PMs and
VM requests are represented as vectors. The first phase of the placement algorithm (or
iVMP), called PM Type Selection, assigns a PM type to the VM request based on a Cosine
Similarity between their vector representations. Then, the VMs are categorized by requested
resources per PM type. The second phase allocates categorized VMs into PMs of the assigned
type. In the second phase, reconfiguration of the VMs (or VMPr) is triggered by well-defined
utilization thresholds of PM resources. The VMPr does not consider handling VM requests
that arrive during a reconfiguration process. Overbooking of all resources is considered, but
no type of elasticity is taken into account.
Tighe et al. proposed in [137] an approach to jointly consider auto-scaling and dynamic
VM allocation in cloud environments. Cloud environments are modeled as workloads of cloud
service requests where CPU overbooking and horizontal elasticity are considered. A proposed
auto-scaling algorithm considers the following parameters to trigger elasticity management
actions: (1) CPU utilization of PMs, (2) requested / utilized resources of VMs and (3) SLA
metrics. The auto-scaling algorithm proposed in [137] is included as part of a Dynamic VM
Allocation (DVMA) algorithm. First, the proposed DVMA considers a Best-Fit Decreasing
(BFD) algorithm to select an appropriate PM to host an incoming VM request (iVMP).
Whenever an overloaded or underloaded PM is detected, a VMPr algorithm is triggered,
similarly to [13].

135
 In summary, most of the related works that consider IaaS environments with overbooking
are limited to CPU resources. Only [123] considered overbooking for all available resources,
as proposed in this work. Additionally, studied IaaS environments with elasticity are limited
to horizontal elasticity [137], while this work considers both vertical and horizontal elasticity.
According to the studied articles (see Table 6.1), existing VMPr Triggering methods may
be classified as: (1) periodical and (2) threshold-based. Periodically triggering the VMPr
could present disadvantages when defining a fixed reconfiguration period (e.g. every 10
minutes) because reconfigurations may be required before the established time or in certain
cases the reconfiguration may not be necessary. For threshold-based approaches, thresholds
are defined in terms of utilization of resources (e.g. CPU) without a complete knowledge of
global optimization objectives. This work proposes a prediction-based approach for a novel
VMPr Triggering method, statistically analyzing the objective function costs and proactively
detecting requirements for triggering the VMPr (see Section 6.4.3).
Additionally, most of the studied works do not consider any VMPr Recovering method,
when applicable. Only Calcavecchia et al. studied in [20] a very basic approach, canceling
the VMPr whenever a new request is received. Consequently, the VMPr is only performed
in periods with no requests, that could result unrealistic, specially for highly loaded IaaS
environments. On the other hand, this work proposes a novel VMPr Recovering method
based on updating the potentially obsolete placement recalculated in the VMPr phase with
the required cloud services created, modified and removed during the recalculation time (see
Section 6.4.4).

6.3 Uncertain VMP Formulation

This section presents a formulation of the VMP problem under uncertainty considering a
two-phase scheme for the optimization of the following objective functions: (1) power con-
sumption, (2) economical revenue, (3) resource utilization and (4) placement reconfiguration
time. According to the taxonomy presented in [94], this work focuses on a provider-oriented
VMP for federated-cloud deployments, considering a combination of two types of formula-
tions: (1) online (i.e. iVMP) and (2) offline (i.e. VMPr).

136
 An online problem formulation is considered when inputs of the problem change over time
and the algorithm does not have the entire input set available from the start (e.g. online
heuristics) [14]. On the other hand, if inputs of the problem do not change over time, the
formulation is considered offline (e.g. Memetic Algorithms (MAs) proposed in [68] and [92]).
As previously discussed in Section 6.1.2, the VMP problem could be formulated as a
two-phase optimization problem, combining advantages of online and offline formulations for
IaaS environments. In this context, VMP problems could be decomposed into two different
sub-problems: (1) incremental VMP (iVMP) and (2) VMP reconfiguration (VMPr).
The VMP problem proposed in this work takes into account a complex IaaS environ-
ment that considers service elasticity, including both vertical and horizontal scaling of cloud
services, as well as overbooking of physical resources, including both server and networking
resources, as identified in [112].
The following sub-sections summarize the complex IaaS environment for VMP problems
considered is this work, as well as formal definitions of both iVMP and VMPr sub-problems.

6.3.1 Complex IaaS Environment

The proposed formulation of the VMP problem models a complex IaaS environment, com-
posed by available PMs and VMs requested at each discrete time t, considering the following
information as input data for the proposed VMP problem:

• a set of n available PMs and their specifications (Equation (6.1));

• a set of m(t) VMs requested, at each discrete time t, and their specifications (Equation
(6.2));

• information about the utilization of resources of each active VM at each discrete time
t (Equation (6.3));

• the current placement at each discrete time t (i.e. x(t)) (Equation (6.4)).

The proposed iVMP and VMPr sub-problems consider different sub-sets of the above
mentioned input data, as presented later in Sections 6.3.2 and 6.3.3.

137
 The set of PMs owned by the IaaS provider is represented as a matrix H ∈ Rn×(r+2) , as
presented in Equation (6.1). Each PM Hi is represented by r different physical resources.
This work considers r = 3 physical resources (P r1 to P r3 ): CPU [EC2 Compute Unit
(ECU)], RAM [GB] and network capacity [Mbps]. The maximum power consumption [W]
is also considered. It is important to mention that the proposed notation is general enough
to include more characteristics associated to physical resources as Solid State Drive (SSD),
Graphical Processing Unit (GPU) or storage, just to cite a few. Finally, considering that
an IaaS provider could own more than one cloud datacenter, PMs notation also includes a
datacenter identifier ci , i.e.

 
 P r1,1 . . . P rr,1 pmax1 c1 
 
 
H= 
 ... ... ... ... ... 
 (6.1)
 
 
P r1,n . . . P rr,n pmaxn cn

where:

P rk,i : Physical resource k on Hi , where 1 ≤ k ≤ r;

pmaxi : Maximum power consumption of Hi in [W];
ci : Datacenter identifier of Hi , where 1 ≤ ci ≤ cmax ;
n: Total number of PMs.

Clearly, the set of PMs H could be modeled as a function of time t, considering PM

crashes [132], maintenance or even deployment of new hardware. The mentioned modeling
approach for PMs is out of the scope of this work and left as future work (see Chapter 7).
In peak demand situations where the IaaS provider cannot provide requested resources,
a basic cloud federation to which assign the over-demand is considered. Formulations with
sophisticated federation approaches is also left as a future work.
In the complex environment considered in this work, the IaaS provider dynamically re-
ceives requests of cloud services for placement (i.e. a set of inter-related VMs) at each discrete
time t. A cloud service Sb is composed by a set of VMs, where each VM may be located
for execution in different cloud computing datacenters according to customer preferences or
requirements (e.g. legal issues or high-availability, just to cite a few).

138
 The set of VMs requested by customers at each discrete time t is represented as a matrix
V (t) ∈ Rm(t)×(r+2) , as presented in Equation (6.2). In this work, each VM Vj requires
r = 3 different virtual resources (V r1,j (t)-V r3,j (t)): CPU [ECU], RAM memory [GB] and
network capacity [Mbps]. Additionally, a cloud service identifier bj is considered, as well as
an economical revenue Rj [$] associated to each VM Vj . As mentioned before, the proposed
notation could represent any other set of r resources.
The requested VMs try to lease the requested virtual resources for an unknown period
of discrete time.

 
 V r1,1 (t) . . . V rr,1 (t) b1 R1 (t) 
 
 
V (t) = 
 ... ... ... ... ... 
 (6.2)
 
 
V r1,m(t) (t) . . . V rr,m(t) (t) bm (t) Rm(t) (t)

where:

V rk,j (t): Virtual resource k on Vj , where 1 ≤ k ≤ r;

bj : Service identifier of Vj ;
Rj (t): Economical revenue for allocating Vj in [$];
m(t): Number of VMs at each discrete time t, where 1 ≤ m(t) ≤ mmax ;
mmax : Maximum number of VMs.

Once a VM Vj is powered-off by a customer, its virtual resources are released, so the IaaS
provider can reuse them. For simplicity, in what follows the index j is not reused.
In order to model a dynamic VMP environment taking into account both vertical and
horizontal elasticity of cloud services, the set of requested VMs V (t) may include the following
types of requests for cloud service placement at each time t:

• cloud services creation: where new cloud services Sb , composed by one or more
VMs Vj , are created. Consequently, the number of VMs at each discrete time t (i.e.
m(t)) is a function of time;

• scale-up / scale-down of VMs resources: where one or more VMs Vj of a cloud

service Sb increases (scale-up) or decreases (scale-down) its capacities of virtual re-
sources with respect to current demand (vertical elasticity). In order to model these

139
 considerations, virtual resource capacities of a VM Vj (i.e. V r1,j (t)-V r3,j (t)) are a
function of time, as well as the associated economical revenue (Rj (t));

• cloud services scale-out / scale-in: where a cloud service Sb increases (scale-out)

or decreases (scale-in) the number of associated VMs according to current demand
(horizontal elasticity). Consequently, the number of VMs Vj in a cloud service Sb at
each discrete time t, denoted as mSb (t), is a function of time;

• cloud services destruction: where virtual resources of cloud services Sb , composed

by one or more VMs Vj , are released.

In most situations, virtual resources requested by cloud services are dynamically used,
giving space to re-utilization of idle resources that were already reserved. Information about
the utilization of virtual resources at each discrete time t is required in order to model a
dynamic VMP environment where IaaS providers consider overbooking of both server and
networking physical resources.
Resource utilization of each VM Vj at each discrete time t is represented as a matrix
U (t) ∈ Rm(t)×r , as presented in Equation (6.3):

 
 U r1,1 (t) ... U rr,1 (t) 
 
 
U (t) = 
 ... ... ... 
 (6.3)
 
 
U r1,m(t) (t) . . . U rr,m(t) (t)

where:

U rk,j (t): Utilization ratio of V rk (t) in Vj at each discrete time t.

The current placement of VMs into PMs (x(t)) represents VMs requested in the previous
discrete time t − 1 and assigned to PMs; consequently, the dimension of x(t) is based on the
number of VMs m(t − 1). Formally, the placement at each discrete time t is represented as
a matrix x(t) ∈ {0, 1}m(t−1)×n , as defined in Equation (6.4):
 
 x1,1 (t) x1,2 (t) ... x1,n (t) 
 
 
x(t) = 
 ... ... ... ... 
 (6.4)
 
 
xm(t−1),1 (t) xm(t−1),2 (t) . . . xm(t−1),n (t)

140
where:

xj,i (t) ∈ {0, 1}: indicates if Vj is allocated (xj,i (t) = 1) or not (xj,i (t) = 0) for execution
in a PM Hi at a discrete time t (i.e., xj,i (t) : Vj → Hi ).

6.3.2 Incremental VMP (iVMP)

In online algorithms for solving the proposed VMP problem, placement decisions are per-
formed at each discrete time t. The formulation of the proposed iVMP (online) problem is
based on [98] and could be formally enunciated as:

Given a complex IaaS environment composed by a set of PMs (H), a set of active VMs already
requested before time t (V (t)), and the current placement of VMs into PMs (i.e. x(t)), it
is sought an incremental placement of V (t) into H for the discrete time t + 1 (x(t + 1))
without migrations, satisfying the problem constraints and optimizing the considered objective
functions.

Input Data for iVMP

The proposed formulation of the iVMP problem receives the following information as input
data:

• a set of n available PMs and their specifications (Equation (6.1));

• a dynamic set of m(t) requested VMs (already allocated VMs plus new requests) and
their specifications (Equation (6.2));

• information about the utilization of resources of each active VM at each discrete time
t (Equation (6.3));

• the current placement at each discrete time t (i.e. x(t)) (Equation (6.4)).

Output Data for iVMP

The result of the iVMP phase at each discrete time t is an incremental placement ∆x(t) for
the next time instant in such a way that x(t + 1) = x(t) + ∆x(t). Clearly, the placement at
t + 1 is represented as a matrix x(t + 1) ∈ {0, 1}m(t)×n , as defined in Equation (6.5):

141
  
 x1,1 (t + 1) x1,2 (t + 1) ... x1,n (t + 1) 
 
x(t + 1) =  (6.5)
 
 ... ... ... ... 

 
xm(t),1 (t + 1) xm(t),2 (t + 1) . . . xm(t),n (t + 1)

Formally, the placement for the next time instant x(t + 1) is a function of the current
placement x(t) and the active VMs at discrete time t, i.e.:

x(t + 1) = f [x(t), V (t)] (6.6)

6.3.3 VMP Reconfiguration (VMPr)

An offline algorithm solves a VMP problem considering a static environment where VM

requests do not change over time and considers migration of VMs between PMs. The formu-
lation of the proposed VMPr (offline) problem is based on [68, 92] and could be enunciated
as:

Given a current placement of VMs into PMs (x(t)), it is sought a placement reconfiguration
through migration of VMs between PMs for the discrete time t (i.e. x0 (t)), satisfying the
constraints and optimizing the considered objective functions.

Input Data for VMPr

The proposed formulation of the VMPr problem receives the following information as input
data:

• a set of n available PMs and their specifications (Equation (6.1));

• information about the utilization of resources of each active VM at discrete time t

(Equation (6.3));

• the current placement at discrete time t (i.e. x(t)) (Equation (6.4)).

Output Data for VMPr

The result of the VMPr problem is a placement reconfiguration through migration of VMs
between PMs for the discrete time t (i.e. x0 (t)), represented by:

142
 • a placement reconfiguration of x(t), i.e. x0 (t) (Equation (6.4));

6.3.4 Part 4.B - Constraints

Constraint 1: Unique Placement of VMs

A VM Vj should be allocated to run on a single PM Hi or alternatively located in another

federated IaaS provider. Consequently, this placement constraint is expressed as:

n
X
xj,i (t) ≤ 1 (6.7)
i=1

∀j ∈ {1, . . . , m(t)}, i.e. for all VM Vj .

where:

xj,i (t) ∈ {0, 1}: Indicates if Vj is allocated (xj,i (t) = 1) or not (xj,i (t) = 0) for execution
in a PM Hi (i.e., xj,i (t) : Vj → Hi ) at a discrete time t;
n: Total number of PMs;
m(t): Number of VMs at each discrete time t, where 1 ≤ m(t) ≤ mmax .

It should be mentioned that from an IaaS provider perspective, elastic cloud services
usually is considered more important than non-elastic ones. Consequently, resources of
elastic cloud services most of the time are allocated with higher priority over non-elastic
ones, what usually is reflected in the contracts between an IaaS provider and each customer.

Constraints 2-4: Overbooked Resources of PMs

A PM Hi must have sufficient available resources to meet the dynamic requirements of all
VMs Vj that are allocated to run on Hi . It is important to remember that resources of
VMs are dynamically used, giving space to re-utilization of idle resources that were already
reserved. Re-utilization of idle resources could represent higher risk of unsatisfied demand
in case utilization of resources increases in a short period of time. Therefore, providers need
to reserve a percentage of idle resources as a protection (defined by a protection factor λk )
in case overbooking is used. These constraints can be formulated as:

143
 m(t)  h 
X
i
xj,i (t) V rk,j (t) × U rk,j (t) + λk V rk,j (t) 1 − U rk,j (t) ≤ P rk,i (6.8)
j=1

for every time slot t, ∀i ∈ {1, . . . , n} and ∀k ∈ {1, . . . , r}, i.e. for each PM Hi and for each
of the r considered physical resource.

where:

m(t): Number of VMs at each discrete time t, where 1 ≤ m(t) ≤ mmax ;

V rk,j (t): Virtual resource k on Vj , where 1 ≤ k ≤ r;
U rk,j (t): Utilization ratio of V rk (t) in Vj at each discrete time t;
λk : Protection factor for V rk,j ∈ [0,1]. Note that λk = 0 means full
overbooking while λk = 1 means no-overbooking;
xj,i (t) ∈ {0, 1}: Indicates if Vj is allocated (xj,i (t) = 1) or not (xj,i (t) = 0) for execution
in a PM Hi (i.e., xj,i (t) : Vj → Hi ) at a discrete time t;
P rk,i : Physical resource k on Hi , where 1 ≤ k ≤ r.

Physical resources are considered as resources available for VMs, after hypervisor reser-
vation.

6.3.5 Part 4.B - Objective Functions

Although in general some objective functions can be minimized while maximizing other
objectives functions, in this chapter each of the considered objective functions are formulated
in a single optimization context (i.e. only minimization).

Power Consumption Minimization

Based on Beloglazov et al. [13], this work models the power consumption of PMs considering
a linear relationship with the CPU utilization of PMs, without taking into account PMs at
alternative datacenters of the cloud federation. The power consumption minimization can
be represented by the sum of the power consumption of each PM Hi that composes the
complex IaaS environment (see Section 6.3.1), as defined in Equation (6.9).

144
 n
X
f1 (x, t) = ((pmaxi − pmini ) × U r1,i (t) + pmini ) × Yi (t) (6.9)
i=1

where:

x: Evaluated solution of the problem;

f1 (x, t): Total power consumption of PMs at instant t;
pmaxi : Maximum power consumption of a PM Hi .
pmini : Minimum power consumption of a PM Hi . As suggested in [13],
pmini ≈ pmaxi ∗ 0.6;
U r1,i (t): Utilization ratio of resource 1 (in this case CPU) by Hi at instant t;
Yi (t) ∈ {0, 1}: Indicates if Hi is turned on (Yi (t) = 1) or not (Yi (t) = 0) at instant t.

Economical Revenue Maximization

For IaaS customers, cloud computing resources often appear to be unlimited and can be
provisioned in any quantity at any required time t [107]. Consequently, this work considers a
basic federated-cloud deployment architecture, where a main provider may support requested
resources that are not able to be provided (e.g. a workload peak) by transparently leasing
low-price resources from alternative datacenters owned by federated providers [49]. This
leasing costs should be minimized in order to maximize revenue objective function.
Equation (6.10) represents the mentioned leasing costs, defined as the sum of the total
costs of leasing each VM Vj that is effectively allocated for execution on any PM of an
alternative datacenter of the cloud federation. A provider must offer its idle resources to
the cloud federation at lower prices than offered to customers in the actual cloud market for
the federation to make sense. The pricing scheme may depend on the particular agreement
between providers of the cloud federation [49]. For simplicity, this work considers that the
main provider may lease requested resources (that are not able to be provided) from the
cloud federation at 70% (X̂j = 0.7) of its market price (Rj (t)). This Leasing Cost (LC(t))
may be formulated as:

m(t)
X
LC(t) = (Rj (t) × Xj (t) × X̂j ) (6.10)
j=1

145
where:

LC(t): Total leasing costs at instant t;

Rj (t): Economical revenue for attending Vj in [$];
Xj (t) ∈ {0, 1}: Indicates if Vj is allocated for execution on a PM (Xj (t) = 1) or not
(Xj (t) = 0) at instant t;
X̂j : Indicates if Vj is allocated on the main provider (X̂j = 0) or on an
alternative datacenter of the cloud federation (X̂j = 0.7);
m(t): Number of VMs at each discrete time t, where 1 ≤ m(t) ≤ mmax .

It is important to note that X̂j is not necessarily a function of time. The decision of
locating a VM Vj on a federated provider is considered only in the placement process, with
no possible migrations between different IaaS providers.
Additionally, overbooked resources may incur in unsatisfied demand of resources at some
periods of time, causing Quality of Service (QoS) degradation, and consequently Service Level
Agreement (SLA) violations with economical penalties. This economical penalties should be
minimized for an economical revenue maximization. Based on the workload independent QoS
metric presented in [13], formalized in SLAs, this work proposes Equation (6.11) to represent
total economical penalties for SLA violations, defined as the sum of the total proportional
penalties costs for unsatisfied demand of resources.

m(t)  r 
X X
Ex(t) = Rrk,j (t) × ∆rk,j (t) × Xj (t) × φk (6.11)
j=1 k=1

where:

Ex(t): Total economical penalties at instant t;

r: Number of considered resources. In this chapter r = 3: CPU, RAM
memory and network capacity;
Rrk,j (t): Economical revenue for attending V rk,j (t);
∆rk,j (t): Ratio of unsatisfied resource k at instant t where ∆rk,j (t) = 1 means no
unsatisfied resource. ∆rk,j (t) = 0 means resource k is unsatisfied in 100%;
Xj (t) ∈ {0, 1}: Indicates if Vj is allocated for execution on a PM (Xj (t) = 1)
or not (Xj (t) = 0) at instant t;

146
φk : Penalty factor for resource k, where φk ≥ 1;
m(t): Number of VMs at each discrete time t, where 1 ≤ m(t) ≤ mmax .

In this work, the maximization of the total economical revenue that an IaaS provider
receives is achieved by minimizing the total costs of leasing resources from alternative data-
centers of the cloud federation as well as the total economical penalties for SLA violations,
as presented in Equation (6.12), i.e.

f2 (x, t) = LC(t) + EP (t) (6.12)

where:

f2 (x, t): Total economical expediture of the main IaaS provider at instant t.

Resources Utilization Maximization

An efficient utilization of resources is a relevant resource management challenge to be ad-

dressed by IaaS providers. This work proposes a maximization of the resource utilization by
minimizing the average ratio of wasted resources on each PM Hi (i.e. resources that are not
allocated to any VM Vj ). This objective function is presented in Equation (6.13).
" Pr !#
Pn k=1 U rk,i (t)
i=1 1− × Yi (t)
r
f3 (x, t) = Pn (6.13)
i=1 Yi (t)

where:

f3 (x, t): Average ratio of wasted resources at instant t;

U rk,i (t): Utilization ratio of resource k of PM Hi at instant t;
r: Number of considered resources. In this chapter r = 3: CPU, RAM memory
and network capacity.

147
Reconfiguration Time Minimization

Performance degradation may occur when migrating VMs between PMs [14]. Logically, it is
desirable that the time of placement reconfiguration by VM migration is kept to a minimum
possible. As explained in [14], the time that a VM takes to be migrated from one PM to
another could be estimated as the ratio between the total amount of RAM memory to be
migrated and the capacity of the network channel.
Inspired in [14], once a placement reconfiguration is accepted in the VMPr phase, all
VM migrations are assumed to be performed in parallel through a management network
exclusively used for these actions, increasing 10% CPU utilization in VMs being migrated.
Consequently, the minimization of the (maximum) reconfiguration time could be achieved
by minimizing the maximum amount of memory to be migrated from one PM Hi to another
Hi0 (i 6= i0 ).
Equation (6.14) is proposed to minimize the maximum amount of RAM memory that
must be moved between PMs at instant t.

f4 (x, t) = max(M Ti,i0 ) ∀i, i0 ∈ {1, . . . , n} (6.14)

where:

f4 (x, t): Network traffic overhead for VM migrations at instant t;

M Ti,i0 : Total amount of RAM memory to be migrated from PM Hi to Hi0 .

It should be noted that there are several possible approaches to estimate the migration
overhead, as presented in [133].
The following sub-section summarizes the main considerations taken into account to com-
bine the four presented objective functions into a single objective function to be minimized
with the aim of having a single figure of merit (or optimization metric).

6.3.6 Normalization and Scalarization Methods

As a consequence of experimental results obtained in a previous work by the authors [68]

for VMP problems optimizing multiple objective functions, even in a many-objective opti-

148
mization context for cloud computing datacenters, instead of calculating a whole Pareto set
approximation, a scalarization method (e.g. minimum distance to origin) is suggested to
combine all considered objective functions into a single objective function, therefore solv-
ing the studied problem considering a Multi-Objective problem solved as Mono-Objective
(MAM) approach [94]. Consequently, each of the considered objective function must be
formulated in a single optimization context (in this case, minimization) and each objec-
tive function cost must be normalized to be comparable and combinable as a single objec-
tive. This work normalizes each objective function cost by calculating f̂i (x, t) ∈ R, where
0 ≤ f̂i (x, t) ≤ 1 for each original objective function fi (x, t).

fi (x, t) − fi (x, t)min

f̂i (x, t) = (6.15)
fi (x, t)max − fi (x, t)min

where:

f̂i (x, t): Normalized cost of objective function fi (x, t) at instant t;

fi (x, t): Cost of original objective function fi (x, t);
fi (x, t)min : Minimum possible cost for fi (x, t);
fi (x, t)max : Maximum possible cost for fi (x, t).

Finally, the presented normalized objective functions are combined into a single objective
considering a minimum Euclidean distance to the origin, expressed as:
v
u q
uX
F (x, t) = t f̂ i (x, t)
2 (6.16)
i=1

where:

F (x, t): Single objective function combining each f̂i (x, t) at instant t;
f̂i (x, t): Normalized cost of objective function fi (x, t) at instant t;
q: Number of objective functions. In this chapter q = 4.

149
6.3.7 Scenario-based Uncertainty Modeling

In this work, uncertainty is modeled through a finite set of well-defined scenarios S [4],
where the following uncertain parameters are considered: (1) virtual resources capacities
(vertical elasticity), (2) number of VMs that compose cloud services (horizontal elasticity),
(3) utilization of CPU and RAM memory virtual resources and (4) utilization of networking
virtual resources (both relevant for overbooking).
For each scenario s ∈ S, a temporal average value of the objective function F (x, t)
presented in Equation (6.16) is calculated as:

Ptmax
t=1 F (x, t)
fs (x, t) = (6.17)
tmax

where:

fs (x, t): Temporal average of combined objective function for all discrete time
instants t in scenario s ∈ S;
tmax : Duration of a scenario (or simulation) in discrete time instants.

As previously described, when parameters are uncertain, it is important to find solutions

that are acceptable for any (or most) considered scenario s ∈ S. This work considers
minimization of the following criteria to select among solutions from different evaluated
alternatives as: (1) average [4], (2) maximum [4] and (3) minimum objective function costs:

P|S|
s=1fs (x, t)
F1 = F (x, t) = (6.18)
|S|

Table 6.2: Summary of evaluated algorithms as well as their corresponding VMPr Triggering
and Recovering methods. N/A indicates a Not Applicable criterion.
Characteristics
Decision iVMP VMPr VMPr Trig- VMPr Re-
Algorithm
Approach gering covering
A7 - inspired in [20] Centralized FFD MA Periodically Cancellation
A8 - inspired in [13] Distributed FFD MMT Threshold- N/A
based
A9 - proposed in this work Centralized FFD MA Prediction- Update-based
based

150
 F2 = max(fs (x, t)) (6.19)
s∈S

F3 = min(fs (x, t)) (6.20)

s∈S

where:

F1 : Average fs (x, t) for all scenarios s ∈ S [4];

F2 : Maximum fs (x, t) considering all scenarios s ∈ S [4];
F3 : Minimum fs (x, t) considering all scenarios s ∈ S.

Although F1 and F2 are the most studied criteria in the specialized literature [4], this work
considers F3 as an additional criterion just to demonstrate that experimental conclusions do
not change when also considering minimum costs.

6.4 Part 4.B - Evaluated Algorithms

Taking into account that this work presents a novel uncertain VMP formulation considering
a complex IaaS environment (see Section 6.3), there is no published alternatives to which
we can compare the proposed algorithm. Therefore, the main goal of the experimental
evaluation to be presented in Section 6.5 is to validate that the proposed VMPr Triggering
and VMPr Recovering methods improve the quality of solutions, against adapted state-
of-the-art alternatives that originally consider only partially the proposed complex IaaS
environment.
This work evaluates three algorithms, presented in Table 6.2. Algorithm 7 (A7 ) is in-
spired in [20], considering a centralized decision approach while Algorithm 8 (A8 ) is inspired
in [13] following a distributed decision approach. Additionally, Algorithm 9 (A9 ) considers a
centralized decision approach implementing the proposed prediction-based VMPr Triggering
and update-based VMPr Recovering methods. In this context, A7 and A8 consider original
VMPr Triggering and VMPr Recovering methods proposed on each original research work
[13, 20].

151
 The following sub-sections detail additional relevant aspects related to the three al-
gorithms evaluated in this chapter.

6.4.1 Incremental VMP (iVMP) Algorithm

In experimental results previously obtained by the authors in [98], the First-Fit Decreasing
(FFD) heuristic outperformed other evaluated heuristics in average; consequently, the men-
tioned heuristic was the only one considered in this chapter for the iVMP problem in the
three evaluated algorithms (A7 to A9 ), as summarized in Table 6.2. This way, this paper
focuses in its main contribution: the VMPr phase. Further studies on alternative heuristics
for the iVMP phase is left as future work.
In the First-Fit (FF) heuristic, requested VMs Vj (t) are allocated on the first PM Hi
with available resources (see Section 6.3.4). Interested readers can refer to [42] for details on
FF algorithms for VMP problems. The considered FFD heuristic operates similarly to FF
heuristic, with the main difference that FFD heuristic sorts the list of requested VMs Vj (t)
in decreasing order by revenue Rj (t) (see details in Algorithm 9).
Taking into account the particularities of the proposed complex IaaS environment, the
FFD heuristic presents some modifications when comparing to the one presented in [98],
mainly considering the cloud service request types previously described in Section 6.3.1. In
fact, Algorithm 9 shows that cloud service destruction, scale-down of VM resources and cloud
services scale-in are processed first, in order to release resources for immediate re-utilization
(steps 1-3 of Algorithm 9). At step 4, requests from V (t) are sorted by a given criterion as
revenue (Rj (t)) in decreasing order (of course, other criterion may be considered, as CPU
[98]), where scale-up of VM resources and cloud services scale-out are firstly processed (steps
5-6), in order to consider elastic cloud services more important than non-elastic ones. Next,
unprocessed requests from Vj (t) include only cloud service creations that are allocated in
decreasing order (steps 7-18). Here, a Vj is allocated in the first Hi with available resources
(see Equation (6.8)) after considering previously sorted V (t). If no Hi has sufficient resources
to host Vj , it is allocated in another federated provider. Finally, the placement x(t + 1) is
updated and returned (steps 19-20).

152
 Algorithm 9: First-Fit Decreasing (FFD) for iVMP.
Data: H, V (t), U (t), x(t) (see notation in Section 6.3.1)
Result: Incremental Placement x(t + 1)
/* removed cloud services */
1 process cloud services destruction from V (t);
/* vertical elasticity */
2 process scale-down of VMs resources from V (t);
/* horizontal elasticity */
3 process cloud services scale-in from V (t);
/* sort VMs by revenue */
4 sort VMs by revenue (Rj (t)) in decreasing order;
/* vertical elasticity */
5 process scale-up of VMs resources from V (t);
/* horizontal elasticity */
6 process cloud services scale-out from V (t);
/* created cloud services */
7 foreach unprocessed Vj in V (t) do
8 while Vj is not allocated do
9 foreach Hi in H do
10 if Hi has enough resources to host Vj then
11 allocate Vj into Hi and break loop;
12 end
13 end
14 if Vj is still not allocated then
15 allocate Vj in another federated provider;
16 end
17 end
18 end
19 update x(t + 1) with processed requests;
20 return x(t + 1)

6.4.2 VMP Reconfiguration (VMPr) Algorithms

Previous research work by the authors focused on developing VMPr algorithms considering
centralized decisions such as the offline MAs presented in [68, 90, 92]. In this work, the
considered VMPr algorithm for centralized decision approaches (A7 and A9 ) is based on
the one presented in [68] and it works in the following way (see details in Algorithm 10):
At step 1, a set P op0 of candidate solutions is randomly generated. These candidate
solutions are repaired at step 2 to ensure that P op0 contains only feasible solutions, satisfying
constraints defined in Section 6.3.4. Then, the algorithm tries to improve candidate solutions
at step 3 using local search. With the obtained solutions, elitism is applied and the first

153
best solution x0 (t) is selected from P op000 ∪ x(t) at step 4 using objective function defined
in Equation (6.16). After an initialization in step 5, evolution begins (steps 6-12). The
evolutionary process basically follows a similar behavior: solutions are selected from the
union of the evolutionary set of solutions (or population), also known as P opu , and the
best known solution x0 (t) (step 7), crossover and mutation operators are applied as usual
(step 8), and eventually solutions are repaired, as there may be infeasible solutions (step 9).
Improvements of solutions of the evolutionary population P opu may be generated at step 10
using local search (local optimization operators). At step 11, the best known solution x0 (t)
is updated (if applicable), while at step 12 the generation (or iteration) counter is updated.
The evolutionary process is repeated until the algorithm meets a stopping criterion (such
as a maximum number of discrete time instants or iterations), returning the best known
solution x0 (t) for a placement reconfiguration. More details on the MA may be found in [68].
Additionally, a distributed decision approach is also considered in the experimental eva-
luation performed in this work (A2 ). For this purpose, the most representative related work
was considered [13]: the Minimum Migration Time (MMT) algorithm. The considered MMT
algorithm based on [13] is presented in Algorithm 11 and it work as follows:
For each time the MMT algorithm is triggered for detecting an overloaded PM Hi (step
1), all VMs Vj that are currently allocated in the considered Hi are sorted in increasing order

Algorithm 10: Memetic Algorithm (MA) for VMPr.

Data: H, U (t), x(t) (see notation in Section 6.3.1)
Result: Recalculated Placement x0 (t)
1 initialize set of candidate solutions P op0 ;
2 P op00 = repair infeasible solutions of P op0 ;
3 P op000 = apply local search to solutions of P op00 ;
4 x0 (t) = select best solution from P op000 ∪ x(t) considering Equation (6.16);
5 u = 0; P opu = P op000 ;
6 while stopping criterion is not satisfied do
7 P opu = selection of solutions from P opu ∪ x0 (t);
8 P op0u = crossover and mutation on solutions of P opu ;
9 P op00u = repair infeasible solutions of P op0u ;
10 P op000 00
u = apply local search to solutions of P opu ;
11 0 000
x (t) = select best solution from P opu considering Equation (6.16);
12 increment number of generations u;
13 end
14 return x0 (t)

154
 Algorithm 11: Minimum Migration Time (MMT) for VMPr running at PM Hi .
Data: H, U (t), x(t), Hi (see notation in Section 6.3.1)
Result: Recalculated Placement x0 (t)
/* Hi has exceeded upper threshold */
1 if Hi is overloaded then
2 sort VMs Vj allocated into Hi in increasing order by RAM;
3 while Hi is overloaded do
4 schedule migration of Vj from Hi to Hi0 using FFD;
5 end
6 end
/* Hi does not reach the lower threshold */
7 if Hi is underloaded then
8 schedule migration of all Vj from Hi to Hi0 6= Hi if possible;
9 end
10 update x0 (t) considering scheduled migrations;
11 return x0 (t)

by RAM size (step 2). This sorting is performed to be able to firstly schedule migration of Vj
with the minimum associated migration time, taking into account that the migration time
of a Vj is directly proportional to its RAM size V r2,j (t) at the considered time instant t
[13]. While Hi is still considered to be overloaded, each VM Vj is scheduled to be migrated
to another PM Hi0 with available resources (see constraint Equation (6.8)) using an FFD
heuristic (steps 3-5). On the other hand, for each time the MMT algorithm is triggered for
detecting an underloaded PM Hi (step 7), all VMs Vj currently allocated in the considered
Hi are scheduled to be migrated to another PM Hi0 with available resources (see constraint
Equation (6.8)) also using an FFD heuristic (step 8). This full migration is performed in
order to be able to shutdown (or switch to energy-saving mode) the considered Hi . Finally,
the placement is updated considering scheduled migrations (step 10), returning a recalculated
placement for reconfiguration (step 11).
The main difference between the MA (see Algorithm 10) and the MMT algorithm (see
Algorithm 11) is the considered decision approach, where the MA performs a centralized
decision that globally reconfigures the placement of VMs while the MMT algorithm performs
a distributed decision partially reconfiguring VMs allocated in only one isolated PM at a
time.

155
6.4.3 Evaluated VMPr Triggering Methods

In this work, a VMPr Triggering method defines when the VMPr phase should be triggered
in a two-phase optimization scheme for VMP problems (Research Question 1 ). Conside-
ring VMPr Triggering methods studied in Section 6.2, this work evaluated the two main
approaches: (1) periodical and (2) threshold-based. Additionally, this work proposes a
prediction-based approach for a novel VMPr Triggering method, statistically analyzing the
objective function costs and proactively detecting requirements for triggering the VMPr
phase. The following sub-sections describe the VMPr Triggering methods evaluated in this
work as part of a two-phase optimization scheme for VMP problems in the proposed complex
IaaS environment.

Periodical Triggering

As described in Section 6.2, several studied works considered to periodically triggering the
VMPr phase (see Table 6.1). This work considers the VMPr Triggering method described
in [20], triggering the VMPr phase every 10 discrete time instants.
Periodically triggering the VMPr could present disadvantages when defining a fixed re-
configuration period (e.g. every 10 time instants). For example, a reconfiguration could be
required before the established time, where optimization opportunities could be wasted or
even economical penalties could impact the cloud datacenter operation. On the other hand,
in certain cases the reconfiguration may not be necessary and triggering the VMPr could
represent profitless reconfigurations.

Threshold-based Triggering

Another very studied VMPr Triggering method considers a threshold-based approach (see
Table 6.1), where thresholds are defined in terms of utilization of PM resources (e.g. CPU).
Thresholds indicate when a PM Hi is considered to be underloaded or overloaded, and
consequently, a VMPr should be triggered. This work considers a threshold-based VMPr
Triggering method based on [13], fixing utilization thresholds for overloaded and underloaded
PM detection, for all considered resources, to 10% and 90% respectively.

156
 The above described threshold-based VMPr Triggering method makes isolated reconfigu-
ration decisions at each PM without a complete knowledge of global optimization objectives,
giving place to a distributed decision approach, as the algorithm A8 implemented in this
chapter.

Proposed Prediction-based Triggering

Considering the main identified issues related to the studied VMPr Triggering methods, this
work proposes a novel prediction-based VMPr Triggering method, statistically analyzing
the global objective function F (x, t) that is optimized (see Equation (6.16)) and proact-
ively detecting situations where a VMPr triggering is potentially required for a placement
reconfiguration.
The proposed prediction-based VMPr Triggering method considers Double Exponential
Smoothing (DES) [63] as a statistical technique for predicting values of the objective function
F (x, t), as formulated next in Equations (6.21) to (6.23):

St = α × Zt + (1 − τ )(St−1 + bt−1 ) (6.21)

bt = τ (St − St−1 ) + (1 − τ )(bt−1 ) (6.22)

Z t+1 = St + bt (6.23)

where:

α: Smoothing factor, where 0 ≤ α ≤ 1;

τ: Trend factor, where 0 ≤ τ ≤ 1;
Zt : Known value of F (x, t) at discrete time t;
St : Expected value of F (x, t) at discrete time t;
bt : Trend of F (x, t) at discrete time t;
Z t+1 : Value of F (x, t + 1) predicted at discrete time t.

At each discrete time t, the proposed VMPr Triggering method predicts the next N̂ values
of F (x, t) and effectively triggers the VMPr phase in case F (x, t) is predicted to consistently
increase, considering that F (x, t) is being minimized, as shown as follows.

157
Table 6.3: Basic example of how the proposed VMPr Triggering method predicts values of
F (x, t) for t = 16, t = 17 and t = 18 (N̂ = 3) based on previous discrete time instants, using
Equation (6.23) with α = τ = 0.5. Predicted values are highlighted.
t Zt St bt Z t+1 Comment
11 0.45 − − −
12 0.50 0.45 0.05 −
Previous values, at t = 15
13 0.55 0.53 0.06 −
14 0.60 0.59 0.07 −
15 0.65 0.65 0.06 0.66 1st prediction at t = 15, for t = 16
16 0.66 0.69 0.05 0.72 2nd prediction at t = 15, for t = 17
17 0.72 0.73 0.04 0.74 3rd prediction at t = 15, for t = 18
Trigger VMPr at t = 15 given that Z 16 < Z 17 < Z 18 with N̂ = 3

Basic Example
For a better understanding of how the proposed VMPr Triggering method works, Table
6.3 presents an example where calculated values correspond to discrete time instant t = 15,
considering parameters α = τ = 0.5. The next N̂ = 3 values of F (x, t) were calculated
based on known values of previous discrete time instants. In this basic example, the VMPr
is triggered at t = 15 considering that the 3 predicted values of F (x, t) at t = 16 to t = 18
tend to consistently increase (i.e. 0.66 < 0.72 < 0.74).

6.4.4 Evaluated VMPr Recovering Methods

It is important to consider that the placement reconfiguration calculated in the VMPr phase
is potentially obsolete, considering the offline nature of the VMPr problem formulation.
In this work, a VMPr Recovering method defines what should be done with cloud service
requests arriving during the VMPr recalculation time β (Research Question 2 ). The iVMP
may receive cloud service requests during the β discrete times that the VMPr performed the
calculation of an improved placement (see Figure 6.1). Consequently, the calculated new
placement must be recovered according to the considered VMPr Recovering method before
the reconfiguration is performed. The mentioned issue is mainly associated to centralized
decision approaches, such as the ones presented in [20, 68].
Most of the studied related works do not consider this issue for two-phase optimization
schemes for VMP problems and only a very basic method is proposed in the specialized
literature [20] (see Table 6.1). Consequently, this work proposes an update-based approach

158
for a novel VMPr Recovering method, applying operations to update the potentially obsolete
placement calculated in the VMPr phase. The following sub-sections describe the VMPr
Recovering methods evaluated in this work as part of a two-phase optimization scheme for
VMP problems in the proposed complex IaaS environment.

Canceling Reconfiguration

Calcavecchia et al. studied in [20] a very basic VMPr Recovering method, canceling the
VMPr whenever a new request is received. In this case, the VMPr is only performed in
periods with no requests, that could be considered unpractical for IaaS providers, taking
into account the highly dynamic environment of cloud computing markets and particularly
the complex IaaS environment proposed as part of this work.

Proposed Update-based Recovering

Considering the identified opportunity to improve the existing VMPr Recovering method
[20], this work proposes a novel VMPr Recovering method based on updating the placement
reconfiguration calculated in the VMPr phase, according to the changes that happened dur-
ing the placement recalculation time, applying operations to update the potentially obsolete
placement, as summarized in Algorithm 12.
The proposed update-based VMPr Recovering methods receives the placement reconfi-
guration calculated in the VMPr phase (corresponding to the discrete time t − β) and the
current placement x(t) as input data (see Algorithm 12).
Considering that any VM Vj could be destroyed, or a cloud service could be scaled-in
(horizontal elasticity) during the β discrete times where the calculation of the placement
reconfiguration was performed, these destroyed VMs are removed from x0 (t − β) (step 1).

Algorithm 12: Update-based VMPr Recovering.

Data: x(t), x0 (t − β) (see notation in Section 6.3.1)
Result: Recovered Placement x0 (t)
1 remove VMs Vj from x0 (t − β) that are no longer running in x(t)
2 adjust resources from x0 (t − β) that changed in x(t)
3 add VMs Vj from x(t) that were not considered in x0 (t − β)
4 if x0 (t − β) is better than x(t) then ;
5 return x0 (t − β);
6 else return x(t) ;

159
Next, any resource from a VM Vj could be adjusted due to a scale-up or scale-down (vertical
elasticity). Consequently, these resource adjustments are performed in x0 (t − β) (see step
2). Additionally, new VMs Vj could be created, or a cloud service could be scaled-out
(horizontal elasticity), during the calculation of x0 (t − β). In the example of Figure 6.1
cloud service S2 is created (+V2 ) and additionally a scale-out of the mentioned cloud service
was performed (+V3 ) during the recalculation time β. These VMs are added to x0 (t − β)
using an FFD heuristic (step 3), the same heuristic used in the iVMP phase. Finally, if the
partially recalculated placement x0 (t − β) is better than the current placement x(t), x0 (t − β)
is accepted (step 5) and the corresponding management actions are performed (i.e. mainly
migration of VMs between PMs). In case x0 (t − β) is not better than the current placement
x(t), no change is performed and the VMPr phase finishes without any further consequence.

6.5 Part 4.B - Experimental Evaluation

The following sub-sections summarize the experimental environment as well as the main
findings identified in the experiments performed as part of the simulations to validate the
two-phase optimization scheme for VMP problems. The quality of solutions obtained by
the evaluated algorithms in a scenario-based uncertainty model with 400 different scenarios
was compared considering the following evaluation criteria among solutions: (1) average, (2)
maximum and (3) minimum objective function costs, formally defined in Equations (6.18)
to (6.20).

Table 6.4: Types of PMs considered in simulations. For notation see Section 6.3.
P r1,i P r2,i P r3,i pmaxi
PM Type ci
[ECU] [GB] [Mbps] [W]
H.small 32 128 1000 800 1
H.medium 64 256 1000 1000 1
H.large 256 512 1000 3000 1
H.xlarge 512 1024 20000 5000 1

160
 Table 6.5: Number of PMs per type on each considered IaaS cloud datacenter.
PM Type DC1 DC2 DC3 DC4 DC5
H.small 50 30 20 15 10
H.medium 50 30 20 10 10
H.large 50 30 15 10 9
H.xlarge 30 10 8 10 8

Table 6.6: Virtual Machine (VM) types from Amazon EC2 considered for cloud service
requests in simulations. For notation see Section 6.3.
V r1,j V r2,j V r3,j Rj
ID Instance Type
[ECU] [GB] [Mbps] [$]
0 m4.large 6.5 8 450 0.12
1 c4.large 8 3.75 500 0.105
2 m4.xlarge 13 16 750 0.239
3 c4.xlarge 16 7.5 750 0.209
4 m4.2xlarge 26 32 1000 0.479
5 c4.2xlarge 31 15 1000 0.419
6 m4.4xlarge 53.5 64 2000 0.958
7 c4.4xlarge 62 30 2000 0.838
8 m4.10xlarge 124.5 160 4000 2.394
9 c4.8xlarge 132 60 4000 1.675
10 x1.32xlarge 349 1952 10000 13.338

6.5.1 Part 4.B - Experimental Environment

The three evaluated algorithms (see Table 6.2) previously presented in Section 6.4 were
implemented using Java programming language. The source code is available online1 , as
well as all the considered input data and experimental results. Experiments were performed
on a GNU Linux Operating System with an Intel(R) Xeon(R) E5530 at 2.40 GHz CPU
and 16 GB of RAM memory. The following parameters of the proposed uncertain VMP
formulation were considered for the experimental evaluation presented in this work (see
details in Section 6.3):

• Number of considered resources: r = 3;

• Recalculation time for A7 and A9: β = 2;

• Recalculation time for A2: β = 1;

• Protection factor for each resource k: λk = 0.5;

• Penalty factor for each resource k: φk = 1.

1
http://github.com/DynamicVMP/DynamicVMPFramework

161
 Table 6.7: Most relevant inputs for example workload trace presented in Table 6.8.
Parameter Input Data
Workload trace duration [t] 5
Number of IaaS datacenters 1
Number of cloud services 1
VMs creation time [t] Uniform (0,3)
Virtual resources capacities Uniform (0,10)
Number of VMs per cloud service Uniform (1,10)
Utilization of CPU and RAM memory virtual resources Poisson (0.7)
Utilization of networking virtual resources Poisson (0.7)

As input data, available PMs (see Equation (6.1)) include 4 different types of PMs, as
presented in Table 6.4. Considering the available PM types, 5 IaaS datacenters with different
number of PMs were considered (DC1 to DC5 ), as summarized in Table 6.5.
Additionally, 80 different workload traces1 of requested cloud services (V (t)) and their
specifications (see Equation (6.2)) were considered as input data as well as their utilization of
resources U (t) at each discrete time t (see Equation (6.3)). Requested VMs were considered
according to instance types offered by Amazon Elastic Compute Cloud (EC2), as summarized
in Table 6.6.
It is important to remember that in this chapter, the following parameters are con-
sidered to be uncertain: (1) virtual resources capacities (vertical elasticity), (2) number of
VMs that compose cloud services (horizontal elasticity), (3) utilization of CPU and RAM
memory virtual resources and (4) utilization of networking virtual resources (both relevant
for overbooking). Consequently, two different Probability Distribution Functions (PDFs)
were considered to represent each parameter behavior (i.e. Uniform and Poisson).
Workload traces of cloud service requests were generated using a Cloud Workload Trace
Generator (CWTG) for provider-oriented VMP problems [113], and are available online2
for research purposes. Considering parameters described in Table 6.7, Table 6.8 presents a
basic example of a workload trace considered in what follows. The duration of the presen-
ted workload trace is five discrete time instants, considering one IaaS datacenter and one
requested cloud service. Additionally, VMs are created according to an Uniform PDF on
different discrete time instants between t = 0 and t = 4.
2
http://github.com/DynamicVMP/workload-trace-generator

162
 Following inputs presented in Table 6.7, virtual resource capacities (V r1,j (t) to V r3,j (t))
of VMs are selected considering IDs associated to each instance type (from 0 to 10) (see
Table 6.6). It is important to notice that at each discrete time t, these resources may change
(vertical elasticity). Similarly to virtual resource capacities of VMs, the number of VMs
per cloud service (mSb (t)) may change uniformly between 1 and 10 at each discrete time t
(horizontal elasticity). Utilization of server as well as networking virtual resources (U r1,j (t)
to U r3,j (t)) are defined according to a Poisson PDF with expected value of 0.7 (i.e. 70%).
In the workload trace example presented in Table 6.8, cloud service S1 is composed
by two VMs at t = 0: V1 and V2 . Virtual resource capacities of both VMs represent a
c4.large instance type (see Table 6.6). Considering the high CPU utilization of both VMs
(i.e. U r1,1 (t) = 0.8 and U r1,2 (t) = 0.9), a scale-up of VM resources is performed for the next
time instant t = 1.
As can be seen at t = 1, VMs associated to S1 scaled to an instance type with more
virtual resources: m4.xlarge (vertical elasticity). At t = 1, the high CPU utilization persists,
representing a possible alarm for scaling-out the cloud service (horizontal elasticity) as can
be observed in t = 2, where S1 is composed by 3 VMs: V1 , V2 and V3 . Next, a low utilization
of CPU resources can be seen at t = 3, representing a possible alarm for scaling-in the cloud
service (horizontal elasticity).

Table 6.8: Example of workload trace for a complex IaaS environment. For notation see
Section 6.3.
V r1,j (t) V r2,j (t) V r3,j (t) Rj (t)
t Sb Vj U r1,j (t) U r2,j (t) U r3,j (t) Comment
[ECU ] [GB] [M bps] [$]
0 S1 V1 8 3.75 500 0.105 0.8 0.6 0.1
create S1
0 S1 V2 8 3.75 500 0.105 0.9 0.7 0.1
1 S1 V1 13 16 750 0.239 0.7 0.5 0.3
scale-up S1
1 S1 V2 13 16 750 0.239 0.7 0.5 0.3
2 S1 V1 13 16 750 0.239 0.6 0.3 0.2
2 S1 V2 13 16 750 0.239 0.6 0.3 0.2 scale-out S1
2 S1 V3 13 16 750 0.239 0.6 0.3 0.2
3 S1 V1 13 16 750 0.239 0.2 0.3 0.1
scale-in S1
3 S1 V2 13 16 750 0.239 0.3 0.4 0.2
4 S1 V1 8 3.75 500 0.105 0.5 0.6 0.1
scale-down S1
4 S1 V2 8 3.75 500 0.105 0.5 0.6 0.1

163
 Finally, virtual resource capacities of each VM that compose cloud service S1 are scaled-
down to a c4.large instance type (vertical elasticity) as can be observed at t = 4.
It is worth noting that server and network resources utilization (U r1,j (t) to U r3,j (t))
dynamically change from t = 0 to t = 4, representing relevant information for CSPs to apply
safe overbooking of physical resources (see example in Table 6.8). It is important to consider
that the algorithm that decides when to auto-scale a cloud service is out of the scope of this
work.
Considering the scenario-based uncertainty modeling approach presented in this work,
each evaluated scenario s ∈ S is composed by an IaaS datacenter and a workload trace of
requested cloud services, totalizing 400 different evaluated scenarios (i.e. 80 workload traces
× 5 IaaS datacenters).
For simplicity, only one cloud computing datacenter is considered in what follows, al-
though in the proposed formulation it is possible to consider more cloud computing data-
centers. Experimenting with several geo-distributed datacenters is left as a future work,
considering that additional objective functions should be taken into account for this type of
deployment (e.g. response time according to customers geographical location).
Experiments are summarized in what follows: Ten runs of the algorithms A7 and A9
were performed for the 400 considered scenarios, taking into account the randomness of the
MA considered for solving the VMPr phase when using A7 or A9. Average obtained results
are presented in Table 6.9. Additionally, the same table also shows results of one run of the
determinist A8 algorithm, performed with the same 400 scenarios.

Table 6.9: Summary of evaluation criteria in experimental results for evaluated algorithms.
Datacenter
Criterion DC1 DC2 DC3 DC4 DC5 Ranking
Algorithm
A7 0.839 0.944 1.003 1.007 1.039 2nd
F1 A8 0.841 0.946 1.002 1.003 1.037 3rd
A9 0.779 0.866 0.926 0.918 0.963 1st
A7 1.029 1.138 1.208 1.224 1.252 2nd
F2 A8 1.036 1.143 1.214 1.226 1.256 3rd
A9 1.006 1.102 1.163 1.183 1.205 1st
A7 0.691 0.798 0.830 0.806 0.839 3rd
F3 A8 0.690 0.763 0.808 0.785 0.820 2nd
A9 0.546 0.613 0.653 0.586 0.648 1st

164
 The following sub-section summarizes the main findings obtained in the experimental
evaluation of the three implemented algorithms for the proposed uncertain formulation of a
two-phase optimization scheme for VMP problems previously presented in Section 6.3.

6.5.2 Part 4.B - Experimental Results

The main goal of the presented experimental evaluation is to explore alternatives to answer
the following research questions:

• when or under what circumstances the VMPr phase should be triggered? (RQ1);

• what should be done with cloud service requests arriving during VMPr recalculation
times? (RQ2).

Table 6.9 presents values of the considered evaluation criteria, i.e. F1 , F2 and F3 costs
(see Equation (6.18) to (6.20)), summarizing results obtained in performed simulations. The
mentioned evaluation criteria are presented separately for each of the five considered IaaS
cloud datacenter. It is worth noting that the considered IaaS cloud datacenters represent
datacenters of different sizes (see Table 6.5) and consequently, the considered workload traces
represent different load of requested CPU resources (e.g. Low (≤ 30%), Medium (≤ 60%),
High (≤ 90%), Full (≤ 98%) and Saturate (≤ 120%)) workloads. The main idea of evaluating
different load of requested CPU resources is inspired in [115].
Based on the information presented in Table 6.9, the Main Findings (MFs) of the expe-
rimental evaluation performed in this chapter are summarized as follows:
MF7: Algorithm A9 that considered the proposed VMPr Triggering and VMPr Recovering
methods outperformed all other evaluated algorithms in every experiment, taking into account
the considered evaluation criteria.

In summary, A9 obtained better results (minimum cost) for the three considered evalua-
tion criteria, as presented in Table 6.9.

165
 1.25

166
0.75
A7 - inspired in
[20]
A8 - inspired in
[13]
A9 - proposed in

Temporal Average Cost fs (x, t)

this work

0.5
0 5 10 15 20 25 30 35 40 45 50 55 60 65 70 75 80

Scenario s ∈ S

Figure 6.2: Temporal average cost: Average values of fs (x, t) in DC1 to DC5 per each scenario s ∈ S.
 When considering average objective function costs (F1 ) as evaluation criterion, A9 ob-
tained between 7.7% and 9.3% better results than A7, as well as between 7.9% and 9.7%
better results than A8. Additionally, when considering maximum objective function costs
(F2 ) as evaluation criterion, the proposed A9 obtained between 2.3% and 3.9% better results
than A7, which performed as the second best algorithm in this case. When comparing to
A8, the proposed A9 algorithm obtained between 3.0% and 4.4% better results. Finally,
A9 obtained between 23.7% and 34.0% better results than A7 when considering minimum
objective function costs (F3 ) as evaluation criterion. A9 algorithm also obtained between
26.6% and 37.5% better results than A8.

To better understand the experimental evaluation summarized in Table 6.9, Figure 6.2
illustrates the temporal average cost of the single combined objective function for all scenarios
s ∈ S, denoted as fs (x, t) in Equation (6.17).

MF8: The proposed A9 outperformed other evaluated algorithms in most of the considered
scenarios, when considering average values of the single combined objective function on each
scenario s ∈ S.
As presented in Figure 6.2, A9 outperformed the other 2 algorithms in most of the
considered scenarios, while performing near to the other evaluated alternatives in the few
scenarios where it does not perform as the best alternative. A9 was the best algorithm in
80% of the 400 carefully designed and evaluated scenarios.

Summarizing, according to the performed experimental evaluation, the algorithm that

considered the proposed prediction-based VMPr Triggering and update-based VMPr Re-
covering methods (A9) is the clear alternative for solving the uncertain VMP problem in a
two-phase optimization scheme, considering the simulation results presented in this section.

167
 Part V

Conclusions and Future Directions

168
7 Conclusions and Future Directions
Based on 84 studied articles, Part I of this doctoral thesis presented general taxonomies of the
VMP problem (Objective 1, published in [10, 94, 96]) considering possible environments
where the VMP problem could be studied (see Figure 2.3) as well as different possible
formulations and techniques for the resolution of the VMP problem (see Table 2.1).
Depending on the particular environment where a VMP problem will be studied, se-
veral different considerations should be taken into account before considering a particular
formulation or technique for a VMP problem resolution. In this context, different possible
environments were identified by classifying research articles in the VMP literature by: (1)
orientation, (2) deployment architecture and (3) type of formulation [96]. A VMP problem
could be studied from both provider or broker perspectives (see Section 2.2.1). A provider-
oriented VMP problem could be studied considering one of the following deployment archi-
tectures: single-cloud, distributed-cloud or federated-cloud, while a broker-oriented VMP
problem could be studied considering a multi-cloud deployment architecture (see Section
2.2.2). Additionally, both a provider-oriented and a broker-oriented VMP problem in any
of the possible deployment architectures could be studied considering two different types of
formulation: offline or online formulations (see Section 2.2.3) [96].
Considering each environment where a VMP problem was already studied (see Figure
2.3), several possible formulations of the problem could be proposed. In this context, for-
mulations of a VMP problem was classified by: (1) optimization approach, (2) objective
function and (3) solution technique. First, a VMP problem could be formulated considering
one of the following optimization approaches: (1) mono-objective (MOP), (2) multi-objective
solved as mono-objective (MAM) or (3) pure multi-objective (PMO). Once the optimization
approach is defined, formulations were classified by the studied objective function(s), both
in minimization and maximization contexts. These objective functions could be optimized
separately or simultaneously, depending on the selected optimization approach. Finally,
solution techniques for solving a VMP problem were used as a classification criterion [96].

169
 According to Table 2.10, the VMP problem have been mostly studied as an online pro-
blem from the provider perspective, considering a single-cloud deployment architecture, re-
presenting 69.66% of the studied articles. For this particular VMP environment, 42,70%
considered a MOP approach. It could be said that online formulations of provider-oriented
VMP problems in single-cloud deployment architecture is a well studied problem and existing
literature should guide CSPs to solve VMP problems with these considerations [96].
The number of objective functions may rapidly increase once a complete understanding
of the VMP problem is accomplished for practical problems, where many different paramet-
ers should be ideally taking into account. Based on the large number of identified objective
functions for the VMP (more than 60), this problem can clearly be addressed as a Many-
Objective Optimization Problem (MaOP). Consequently, Part II of this work presented a
general many-objective optimization framework that is able to simultaneously consider as
many objective functions as needed when solving a VMP problem. As an example of utiliza-
tion of the proposed framework, a formulation of a many-objective VMP problem (MaVMP)
is proposed, considering the simultaneous optimization of the following five objective func-
tions: (1) power consumption, (2) network traffic, (3) economical revenue, (4) quality of
service and (5) network load balancing (Objective 2(a), published in [92, 95, 97]). In addi-
tion, a generalized multi-level of priorities for the SLA associated to each VM is presented.
At the same time, adjustable constraints on upper and lower limits of each objective func-
tion are also recommended as a way to interactively control a potential explosion in the num-
ber of non-dominated solutions (Objective 3, published in [92, 95]), a well known problem
of MaOPs [143]. Multiple challenges need to be considered for solving a MaOP; therefore,
an interactive Memetic Algorithm (MA) was proposed to solve the proposed many-objective
formulation (Objective 4, published in [92, 95]), validating the presented formulation and
proving that it is solvable. By no means, the authors claim that the proposed interactive
MA is the best way to solve this problem. This doctoral thesis only illustrates that it is
possible to solve the problem although the VMP problem becomes harder when increasing
the number of conflicting objective functions.

170
 To validate the algorithm presented in [92, 95], it was run with different problem in-
stances and experimental results were compared to the optimal solutions obtained using an
Exhaustive Search Algorithm (ESA) when possible. In fact, the proposed algorithm found
the complete Pareto set and its corresponding Pareto front (100%) for the problem instances
considered in Experiment 1. To demonstrate the effectiveness of the proposed algorithm re-
ducing the increasing number of non-dominated solutions obtained by the Pareto dominance
comparison, several problem instances were executed. In fact, considering the proposed
technique of interactively adjusting lower and upper bounds associated to the values of each
objective functions, the Pareto front could be reduced from 965 original candidates to a man-
ageable 17 non-dominated solutions that satisfy the decision maker preferences, as illustrated
in Experiment 2. Results obtained in Experiment 3 prove that the proposed algorithm is able
to effectively solve instances of a MaVMP problem with large numbers of PMs and VMs, as
considered in real-world virtualized datacenters. It was also proved that increasing the num-
ber of individuals or generations, the algorithm obtained larger numbers of non-dominated
solutions with unrestricted bounds, being extremely necessary to include additional methods
to the Pareto dominance relation for the selection of a manageable number of solutions.
Next, Part III of this work formulated for the first time a MaVMP with reconfiguration of
VMs, simultaneously optimizing the following five objective functions: (1) power consump-
tion, (2) inter-VM network traffic, (3) economical revenue, (4) number of VM migrations
and (5) network traffic overhead for VM migrations (Objective 2(b), published in [68]).
An extended MA was proposed to solve this semi-dynamic MaVMP (Objective 6, pub-
lished in [68]). The following five selection strategies were evaluated to automate the process
of selecting a solution from a Pareto set approximation Pknown at each instant t (Objective
5, published in [68]): (S1) random, (S2) preferred solution, (S3) minimum distance to ori-
gin, (S4) lexicographic order (provider) and (S5) lexicographic order (service). Experimental
results recommend the S3 (minimum distance to origin) strategy. As a consequence, for pro-
duction cloud datacenters instead of calculating all the Pareto set approximation, the S3
strategy should be used to combine all considered objective functions into only one objective
function, therefore solving the studied problem considering a MAM approach.

171
 To explore alternatives to formulate MaVMP problems for cloud computing, Part IV of
this doctoral thesis presented in [98] an experimental comparison of five online heuristics
for solving a proposed provider-oriented VMP problem for the optimization of the following
objectives: (1) power consumption, (2) economical revenue, (3) quality of service, and (4)
resource utilization. The quality of solutions obtained by online heuristics was compared to
the best performance of an offline MA that have complete knowledge of future VM requests
and the capacity of VM migration. A previously studied MA [68] was implemented taking
into account that obtaining optimal solutions for large instances could be impracticable [89].
Considering the evaluation of the mentioned algorithms solving several experimental
workload traces, six main findings (MF1 to MF6) were identified (see Section 5.4.3). The
main advantage of the offline MA (A6) over experimentally evaluated online heuristics (A1
to A5) is its ability to calculate better quality of solutions, as summarized in Table 5.2.
Unfortunately, offline alternatives such as A6 are not appropriate for dynamic environ-
ments of VMP problems for IaaS providers, where future VM requests are unknown and
placement should be solved in a very short time. To improve the quality of solutions ob-
tained by online algorithms such as evaluated heuristics, the VMP problem was formulated
as a two-phase optimization problem. In this context, VMP problems with many objectives
for an IaaS could be decomposed into two different sub-problems: (1) incremental VMP
(iVMP) and (2) VMP reconfiguration (VMPr) [155]. A two-phase optimization strategy
can be considered, combining both online (iVMP) and offline (VMPr) algorithms for solving
each considered VMP sub-problem. The iVMP problem is considered for dynamic arriving
requests when VMs should be created and removed at runtime. Consequently, this sub-
problem should be formulated as an online problem and solved in a short period of time,
where studied heuristics could be appropriate. On the contrary, the VMPr problem is con-
sidered for improving the quality of solutions obtained by the iVMP, considering placement
reconfigurations through migration of VMs. This sub-problem could be formulated offline,
where alternative solution techniques (e.g. meta-heuristics) could be appropriate.
Considering the main findings presented in [98], this doctoral thesis proposed a complex
IaaS environment for VMP problems considering service elasticity, including both vertical

172
and horizontal scaling of cloud services, as well as overbooking of physical resources, including
both server (CPU and RAM) and networking resources (Objective 7, published in [99]).
The proposed complex IaaS environment for VMP problems was studied in a two-phase
optimization scheme, combining advantages of both online and offline VMP formulations,
where a novel prediction-based VMPr Triggering method was proposed to decide when or
under what circumstances to trigger a placement reconfiguration (Research Question 1 ) as
well as a novel update-based VMPr Recovering method to decide what to do with VMs
requested during VMPr recalculation times (Research Question 2 ) (Objective 8, published
in [99]), as described in Section 6.4.3.
A first formulation of an uncertain MaVMP problem considering the above mentioned
contributions was also proposed, for the optimization of the following four objective func-
tions: (1) power consumption, (2) economical revenue, (3) resource utilization as well as (4)
placement reconfiguration time (see Section 6.3.5) in a context of Multi-Objective problem
solved as Mono-Objective (MAM) [68].
Additionally, a first scenario-based uncertainty approach for modeling relevant uncertain
parameters considering the proposed complex IaaS environment in the two-phase optimi-
zation scheme for VMP problems was presented (Objective 2(c), published in [99]). The
considered uncertain parameters were: (1) virtual resources capacities (vertical elasticity),
(2) number of VMs that compose cloud services (horizontal elasticity), (3) utilization of CPU
and RAM memory virtual resources and (4) utilization of networking virtual resources (both
relevant for overbooking). Each considered uncertain parameter was modeled considering
two different PDFs: (1) Uniform and (2) Poisson.
Trying to answer Research Questions 1 and 2, the proposed VMPr Triggering and VMPr
Recovering methods were experimentally evaluated against other alternatives identified in
the specialized literature (see Table 6.1), considering 400 scenarios. Experimental results
in performed simulations suggested that the best algorithm for solving the proposed uncer-
tain VMP problem in a two-phase optimization scheme is the one considering the proposed
prediction-based VMPr Triggering (answer to Research Question 1 ) and update-based VMPr
Recovering methods (answer to Research Question 2 ) used by the A9 algorithm.

173
 Several future works were also identified, mainly considering the novelty of the contribu-
tions of this work. First, a formulation of a VMP problem considering a dynamic set of PMs
H(t), to consider PM crashes, maintenance or even deployment of new generation hardware
is proposed as a future work.
Considering VMP formulations with more sophisticated cloud federation approaches is
also left as a future work, taking into account the basic cloud federation approach considered
in this work. Additionally, an experimental evaluation of alternative algorithms for both
iVMP and VMPr phase is proposed as a future work, in order to explore performance issues
with the proposed VMPr Triggering and VMPr Recovering methods.
Novel VMPr Triggering and VMPr Recovering methods could still be proposed to improve
the considered two-phase optimization scheme. A more detailed experimental evaluation of
different parameters of the proposed VMP formulation should also be considered, evaluating
different protection factors λk (see Section 6.3.4), penalty factors φk (see Section 6.3.5) or
even different scalarization methods (see Section 6.3.6).
The authors of this work also recognized the importance of jointly considering auto-
scaling algorithms with the proposed two-phase optimization scheme for VMP problems,
mainly for elastic cloud services as the considered in this work.
Experimenting with geo-distributed datacenters is also left as a future work, taking into
account that simulations presented in this work considered only one cloud datacenter.
Additionally, the authors are already working on implementing the evaluated algorithms
in IaaS middlewares (e.g. OpenStack1 ) to evaluate the proposed methods in real-world cloud
computing datacenters supporting real workloads of cloud applications.
Future directions may also include detailed studies on the VMP problem considering
dynamic pricing schemes or auction-based pricing. These type of pricing schemes represent
open challenges for both CSPs and CSBs in the VMP research and emerging cloud computing
markets.
Considering the novelty of the proposed MaVMP formulations, several future works can
be proposed considering the simultaneous optimization of most of the already presented
objective functions, studied so far in mono-objective as well as multi-objective contexts.
1
http://www.openstack.org

174
Currently, comparisons with other works are limited because this work presented first for-
mulations of different MaVMP problems; therefore, the proposed algorithms should still be
compared to other state of the art algorithms commonly used, as the ones identified in [96].

175
Bibliography
[1] Amol C Adamuthe, Rupali M Pandharpatte, and Gopakumaran T Thampi. Multiob-
jective virtual machine placement in cloud environment. In Cloud & Ubiquitous Com-
puting & Emerging Technologies (CUBE), 2013 International Conference on, pages
8–13. IEEE, 2013.

[2] Orna Agmon Ben-Yehuda, Muli Ben-Yehuda, Assaf Schuster, and Dan Tsafrir. Decon-
structing Amazon EC2 spot instance pricing. ACM Transactions on Economics and
Computation, 1(3):16, 2013.

[3] Mansoor Alicherry and TV Lakshman. Optimizing data access latencies in cloud
systems by intelligent virtual machine placement. In INFOCOM, 2013 Proceedings
IEEE, pages 647–655. IEEE, 2013.

[4] Mohamed Ali Aloulou and Federico Della Croce. Complexity of single machine
scheduling problems under scenario-based uncertainty. Operations Research Letters,
36(3):338–342, 2008.

[5] Augusto Amarilla, Saúl Zalimben, Leonardo Benítez, Fabio López-Pires, and Benjamín
Barán. Evaluating a two-phase virtual machine placement optimization scheme for
cloud computing datacenters. In 2017 Metaheuristics International Conference (MIC),
pages 99–108, 2017.

[6] Ankit Anand, J Lakshmi, and SK Nandy. Virtual machine placement optimization
supporting performance SLAs. In Cloud Computing Technology and Science (Cloud-
Com), 2013 IEEE 5th International Conference on, volume 1, pages 298–305. IEEE,
2013.

[7] S. Arévalos, Fabio López-Pires, and Benjamín Barán. A comparative evaluation of

algorithms for auction-based cloud pricing prediction. In 2016 IEEE International
Conference on Cloud Engineering (IC2E), pages 99–108, April 2016.

176
 [8] Sara Arévalos, Fabio López-Pires, and Benjamín Barán. Auction-based resource pro-
visioning in cloud computing. a taxonomy. In Computing Conference (CLEI), 2015
XLI Latin American. IEEE, 2015.

[9] M Báez, D Zárate, and Benjamín Barán. Adaptive memetic algorithms for multi-
objective optimization. In Computing Conference (CLEI), 2007 XXXIII Latin Amer-
ican, volume 2007, 2007.

[10] Benjamín Barán and Fabio López-Pires. Resource allocation for cloud infrastructures:
Taxonomies and research challenges. In Sanjay Chaudhary, Gaurav Somani, and Ra-
jkumar Buyya, editors, Research Advances in Cloud Computing, chapter 10, pages
266–290. Springer, 2017.

[11] Benjamín Barán, Christian von Lücken, and Aldo Sotelo. Multi-objective pump
scheduling optimisation using evolutionary strategies. Advances in Engineering Soft-
ware, 36(1):39–47, 2005.

[12] Luiz André Barroso and Urs Hölzle. The case for energy-proportional computing.
IEEE computer, 40(12):33–37, 2007.

[13] Anton Beloglazov, Jemal Abawajy, and Rajkumar Buyya. Energy-aware resource al-
location heuristics for efficient management of data centers for cloud computing. Future
Generation Computer Systems, 28(5):755–768, 2012.

[14] Anton Beloglazov and Rajkumar Buyya. Optimal online deterministic algorithms and
adaptive heuristics for energy and performance efficient dynamic consolidation of vir-
tual machines in cloud data centers. Concurrency and Computation: Practice and
Experience, 24(13):1397–1420, 2012.

[15] Eyal Bin, Ofer Biran, Odellia Boni, Erez Hadad, Elliot K Kolodner, Yosef Moatti, and
Dean H Lorenz. Guaranteeing high availability goals for virtual machine placement.
In Distributed Computing Systems (ICDCS), 2011 31st International Conference on,
pages 700–709. IEEE, 2011.

177
[16] Ofer Biran, Antonio Corradi, Mario Fanelli, Luca Foschini, Alexander Nus, Danny
Raz, and Ezra Silvera. A stable network-aware vm placement for cloud systems. In
Proceedings of the 2012 12th IEEE/ACM International Symposium on Cluster, Cloud
and Grid Computing (ccgrid 2012), pages 498–506. IEEE Computer Society, 2012.

[17] Kamal Bouyoucef, I Limam-Bedhiaf, and Omar Cherkaoui. Optimal allocation

approach of virtual servers in cloud computing. In Next Generation Internet (NGI),
2010 6th EURO-NF Conference on, pages 1–6. IEEE, 2010.

[18] Rajkumar Buyya, Chee Shin Yeo, and Srikumar Venugopal. Market-oriented cloud
computing: Vision, hype, and reality for delivering it services as computing utilities.
In High Performance Computing and Communications, 2008. HPCC’08. 10th IEEE
International Conference on, pages 5–13. Ieee, 2008.

[19] Rajkumar Buyya, Chee Shin Yeo, Srikumar Venugopal, James Broberg, and Ivona
Brandic. Cloud computing and emerging it platforms: Vision, hype, and reality for de-
livering computing as the 5th utility. Future Generation computer systems, 25(6):599–
616, 2009.

[20] Nicolò Maria Calcavecchia, Ofer Biran, Erez Hadad, and Yosef Moatti. Vm place-
ment strategies for cloud scenarios. In Cloud Computing (CLOUD), 2012 IEEE 5th
International Conference on, pages 852–859. IEEE, 2012.

[21] Zhibo Cao and Shoubin Dong. An energy-aware heuristic framework for virtual ma-
chine consolidation in cloud computing. The Journal of Supercomputing, pages 1–23,
2014.

[22] Eddy Caron, Anh Dung Le, Arnaud Lefray, and Christian Toinard. Definition of
security metrics for the cloud computing and security-aware virtual machine place-
ment algorithms. In Cyber-Enabled Distributed Computing and Knowledge Discovery
(CyberC), 2013 International Conference on, pages 125–131. IEEE, 2013.

[23] Sivadon Chaisiri, Bu-Sung Lee, and Dusit Niyato. Optimal virtual machine placement

178
 across multiple cloud providers. In Services Computing Conference, 2009. APSCC
2009. IEEE Asia-Pacific, pages 103–110. IEEE, 2009.

[24] Lino Chamorro, Fabio López-Pires, and Benjamín Barán. A genetic algorithm for
dynamic cloud application brokerage. IEEE International Conference on Cloud En-
gineering, 2016.

[25] Decheng Chang, Gaochao Xu, Liang Hu, and Kun Yang. A network-aware virtual ma-
chine placement algorithm in mobile cloud computing environment. In Wireless Com-
munications and Networking Conference Workshops (WCNCW), 2013 IEEE, pages
117–122. IEEE, 2013.

[26] Kuan-yin Chen, Yang Xu, Kang Xi, and H Jonathan Chao. Intelligent virtual machine
placement for cost efficiency in geo-distributed cloud systems. In Communications
(ICC), 2013 IEEE International Conference on, pages 3498–3503. IEEE, 2013.

[27] Jixiang Cheng, Gary G. Yen, and Gexiang Zhang. A many-objective evolutionary
algorithm based on directional diversity and favorable convergence. In Systems, Man
and Cybernetics (SMC), 2014 IEEE International Conference on, pages 2415–2420,
Oct 2014.

[28] Carlos Coello Coello, Gary B Lamont, and David A Van Veldhuizen. Evolutionary
algorithms for solving multi-objective problems. Springer, 2007.

[29] Aissan Dalvandi, Mohan Gurusamy, and Kee Chaing Chua. Time-aware vm-placement
and routing with bandwidth guarantees in green cloud data centers. In Cloud Com-
puting Technology and Science (CloudCom), 2013 IEEE 5th International Conference
on, volume 1, pages 212–217. IEEE, 2013.

[30] Huynh Tu Dang and Fabien Hermenier. Higher SLA satisfaction in datacenters with
continuous VM placement constraints. In Proceedings of the 9th Workshop on Hot
Topics in Dependable Systems, page 1. ACM, 2013.

[31] Kalyanmoy Deb, Amrit Pratap, Sameer Agarwal, and TAMT Meyarivan. A fast and

179
 elitist multiobjective genetic algorithm: NSGA-II. Evolutionary Computation, IEEE
Transactions on, 6(2):182–197, 2002.

[32] Kalyanmoy Deb, Ankur Sinha, and Saku Kukkonen. Multi-objective test problems,
linkages, and evolutionary methodologies. In Proceedings of the 8th annual conference
on Genetic and evolutionary computation, pages 1141–1148. ACM, 2006.

[33] Daniel S Dias and Luıs Henrique MK Costa. Online traffic-aware virtual machine place-
ment in data center networks. In Global Information Infrastructure and Networking
Symposium (GIIS), 2012, pages 1–8. IEEE, 2012.

[34] Anh Vu Do, Junliang Chen, Chen Wang, Young Choon Lee, Albert Y Zomaya, and
Bing Bing Zhou. Profiling applications for virtual machine placement in clouds. In
Cloud Computing (CLOUD), 2011 IEEE International Conference on, pages 660–667.
IEEE, 2011.

[35] Dapeng Dong and John Herbert. Energy efficient vm placement supported by data ana-
lytic service. In Cluster, Cloud and Grid Computing (CCGrid), 2013 13th IEEE/ACM
International Symposium on, pages 648–655. IEEE, 2013.

[36] Jiankang Dong, Xing Jin, Hongbo Wang, Yangyang Li, Peng Zhang, and Shiduan
Cheng. Energy-saving virtual machine placement in cloud data centers. In Cluster,
Cloud and Grid Computing (CCGrid), 2013 13th IEEE/ACM International Sym-
posium on, pages 618–624. IEEE, 2013.

[37] Jiankang Dong, Hongbo Wang, Xing Jin, Yangyang Li, Peng Zhang, and Shiduan
Cheng. Virtual machine placement for improving energy efficiency and network per-
formance in IaaS cloud. In Distributed Computing Systems Workshops (ICDCSW),
2013 IEEE 33rd International Conference on, pages 238–243. IEEE, 2013.

[38] Yezid Donoso, Ramón Fabregat, Fernando Solano, José-Luis Marzo, and Benjamín
Barán. Generalized multiobjective multitree model for dynamic multicast groups. In
Communications, 2005. ICC 2005. 2005 IEEE International Conference on, volume 1,
pages 148–152. IEEE, 2005.

180
[39] Corentin Dupont, Giovanni Giuliani, Fabien Hermenier, Thomas Schulze, and Andrey
Somov. An energy aware framework for virtual machine placement in cloud federated
data centres. In Future Energy Systems: Where Energy, Computing and Communi-
cation Meet (e-Energy), 2012 Third International Conference on, pages 1–10. IEEE,
2012.

[40] Erik Elmroth, Johan Tordsson, Francisco Hernández, Ahmed Ali-Eldin, Petter Svärd,
Mina Sedaghat, and Wubin Li. Self-management challenges for multi-cloud architec-
tures. In Towards a Service-Based Internet, pages 38–49. Springer, 2011.

[41] D. Espling, L. Larsson, W. Li, J. Tordsson, and E. Elmroth. Modeling and placement
of cloud services with internal structure. Cloud Computing, IEEE Transactions on,
PP(99):1–1, 2014.

[42] Shuo Fang, Renuga Kanagavelu, Bu-Sung Lee, Chuan Heng Foh, and Khin Mi Mi
Aung. Power-efficient virtual machine placement and migration in data centers.
In Green Computing and Communications (GreenCom), 2013 IEEE and Internet
of Things (iThings/CPSCom), IEEE International Conference on and IEEE Cyber,
Physical and Social Computing, pages 1408–1413. IEEE, 2013.

[43] Weiwei Fang, Xiangmin Liang, Shengxin Li, Luca Chiaraviglio, and Naixue Xiong.
Vmplanner: Optimizing virtual machine placement and traffic flow routing to reduce
network power costs in cloud data centers. Computer Networks, 57(1):179–196, 2013.

[44] Fahimeh Farahnakin, Rami Bahsoon, Pasi Liljeberg, and Tapio Pahikkala. Self-
adaptive resource management system in iaas clouds. In Rami Bahsoon, Pasi Lilje-
berg, and Tapio Pahikkala, editors, 9th International Conference on Cloud Computing
(IEEE CLOUD), pages 553–560. IEEE, 2016.

[45] Marco Farina and Paolo Amato. On the optimal solution definition for many-criteria
optimization problems. In Proceedings of the NAFIPS-FLINT international conference,
pages 233–238, 2002.

181
[46] Eugen Feller, Christine Morin, and Armel Esnault. A case for fully decentralized
dynamic vm consolidation in clouds. In Cloud Computing Technology and Science
(CloudCom), 2012 IEEE 4th International Conference on, pages 26–33. IEEE, 2012.

[47] Tiago Ferreto, César AF De Rose, and Hans-Ulrich Heiss. Maximum migration time
guarantees in dynamic server consolidation for virtualized data centers. In Euro-Par
2011 Parallel Processing, pages 443–454. Springer, 2011.

[48] Tiago C Ferreto, Marco AS Netto, Rodrigo N Calheiros, and César AF De Rose. Server
consolidation with migration control for virtualized data centers. Future Generation
Computer Systems, 27(8):1027–1034, 2011.

[49] M. Gahlawat and P. Sharma. Survey of virtual machine placement in federated clouds.
In Advance Computing Conference (IACC), 2014 IEEE International, pages 735–738,
Feb 2014.

[50] Monica Gahlawat and Priyanka Sharma. Survey of virtual machine placement in fed-
erated clouds. In Advance Computing Conference (IACC), 2014 IEEE International,
pages 735–738. IEEE, 2014.

[51] Yongqiang Gao, Haibing Guan, Zhengwei Qi, Yang Hou, and Liang Liu. A multi-
objective ant colony system algorithm for virtual machine placement in cloud compu-
ting. Journal of Computer and System Sciences, 79(8):1230–1242, 2013.

[52] Stefanos Georgiou, Konstantinos Tsakalozos, and Alex Delis. Exploiting network-
topology awareness for VM placement in IaaS clouds. In Cloud and Green Computing
(CGC), 2013 Third International Conference on, pages 151–158. IEEE, 2013.

[53] Hadi Goudarzi and Massoud Pedram. Energy-efficient virtual machine replication and
placement in a cloud computing system. In Cloud Computing (CLOUD), 2012 IEEE
5th International Conference on, pages 750–757. IEEE, 2012.

[54] Yang Guo, Alexander L Stolyar, and Anwar Walid. Shadow-routing based dynamic
algorithms for virtual machine placement in a network cloud. In INFOCOM, 2013
Proceedings IEEE, pages 620–628. IEEE, 2013.

182
[55] Abhishek Gupta, Laxmikant V Kalé, Dejan Milojicic, Paolo Faraboschi, and
Susanne M Balle. Hpc-aware vm placement in infrastructure clouds. In Cloud En-
gineering (IC2E), 2013 IEEE International Conference on, pages 11–20. IEEE, 2013.

[56] Abhishek Gupta, Dejan Milojicic, and Laxmikant V Kalé. Optimizing vm placement
for hpc in the cloud. In Proceedings of the 2012 workshop on Cloud services, federation,
and the 8th open cirrus summit, pages 1–6. ACM, 2012.

[57] Rohit Gupta, Sumit Kumar Bose, Srikanth Sundarrajan, Manogna Chebiyam, and
Anirban Chakrabarti. A two stage heuristic algorithm for solving the server consol-
idation problem with item-item and bin-item incompatibility constraints. In Services
Computing, 2008. SCC’08. IEEE International Conference on, volume 2, pages 39–46.
IEEE, 2008.

[58] Mateusz Guzek, Pascal Bouvry, and El-Ghazali Talbi. A survey of evolutionary com-
putation for resource management of processing in cloud computing [review article].
Computational Intelligence Magazine, IEEE, 10(2):53–67, 2015.

[59] Mohammad Mehedi Hassan, M Shamim Hossain, AM Jehad Sarkar, and Eui-Nam
Huh. Cooperative game-based distributed resource allocation in horizontal dynamic
cloud federation platform. Information Systems Frontiers, 16(4):523–542, 2014.

[60] Yufan Ho, Pangfeng Liu, and Jan-Jan Wu. Server consolidation algorithms with
bounded migration cost and performance guarantees in cloud computing. In Utility
and Cloud Computing (UCC), 2011 Fourth IEEE International Conference on, pages
154–161. IEEE, 2011.

[61] Hua-Jun Hong, De-Yu Chen, Chun-Ying Huang, Kuan-Ta Chen, and Cheng-Hsin Hsu.
Qoe-aware virtual machine placement for cloud games. In Network and Systems Sup-
port for Games (NetGames), 2013 12th Annual Workshop on, pages 1–2. IEEE, 2013.

[62] Daochao Huang, Dong Yang, Hongke Zhang, and Lei Wu. Energy-aware virtual ma-
chine placement in data centers. In Global Communications Conference (GLOBE-
COM), 2012 IEEE, pages 3243–3249. IEEE, 2012.

183
[63] J. Huang, C. Li, and J. Yu. Resource prediction based on double exponential smoothing
in cloud computing. In 2012 2nd International Conference on Consumer Electronics,
Communications and Networks (CECNet), pages 2056–2060, April 2012.

[64] Wei Huang, Xin Li, and Zhuzhong Qian. An energy efficient virtual machine place-
ment algorithm with balanced resource utilization. In Innovative Mobile and Internet
Services in Ubiquitous Computing (IMIS), 2013 Seventh International Conference on,
pages 313–319. IEEE, 2013.

[65] Zhe Huang and Danny HK Tsang. Sla guaranteed virtual machine consolidation for
computing clouds. In Communications (ICC), 2012 IEEE International Conference
on, pages 1314–1319. IEEE, 2012.

[66] Zhe Huang, Danny HK Tsang, and James She. A virtual machine consolidation frame-
work for mapreduce enabled computing clouds. In Proceedings of the 24th International
Teletraffic Congress, page 26. International Teletraffic Congress, 2012.

[67] Inkwon Hwang and Massoud Pedram. Hierarchical virtual machine consolidation in a
cloud computing system. In Cloud Computing (CLOUD), 2013 IEEE Sixth Interna-
tional Conference on, pages 196–203. IEEE, 2013.

[68] Diego Ihara, Fabio López-Pires, and Benjamín Barán. Many-objective virtual ma-
chine placement for dynamic environments. In 2015 IEEE/ACM 8th International
Conference on Utility and Cloud Computing (UCC), pages 75–79. IEEE, 2015.

[69] Ryan Jansen and Paul R Brenner. Energy efficient virtual machine allocation in the
cloud. In Green Computing Conference and Workshops (IGCC), 2011 International,
pages 1–8. IEEE, 2011.

[70] Deepal Jayasinghe, Calton Pu, Tamar Eilam, Malgorzata Steinder, Ian Whally, and
Ed Snible. Improving performance and availability of services hosted on iaas clouds
with structural constraint-aware virtual machine placement. In Services Computing
(SCC), 2011 IEEE International Conference on, pages 72–79. IEEE, 2011.

184
[71] Dailin Jiang, Peijie Huang, Piyuan Lin, and Jiacheng Jiang. Energy efficient vm
placement heuristic algorithms comparison for cloud with multidimensional resources.
In Information Computing and Applications, pages 413–420. Springer, 2012.

[72] Joe Wenjie Jiang, Tian Lan, Sangtae Ha, Minghua Chen, and Mung Chiang. Joint
vm placement and routing for data center traffic engineering. In INFOCOM, 2012
Proceedings IEEE, pages 2876–2880. IEEE, 2012.

[73] Hai Jin, Hanfeng Qin, Song Wu, and Xuerong Guo. Ccap: A cache contention-aware
virtual machine placement approach for hpc cloud. International Journal of Parallel
Programming, pages 1–18, 2013.

[74] Hao Jin, Deng Pan, Jing Xu, and Niki Pissinou. Efficient vm placement with multiple
deterministic and stochastic resources in data centers. In Global Communications
Conference (GLOBECOM), 2012 IEEE, pages 2505–2510. IEEE, 2012.

[75] Dharmesh Kakadia, Nandish Kopri, and Vasudeva Varma. Network-aware virtual
machine consolidation for large data centers. In Proceedings of the Third International
Workshop on Network-Aware Data Management, page 6. ACM, 2013.

[76] Burak Kantarci, Luca Foschini, Antonio Corradi, and Hussein T Mouftah. Inter-
and-intra data center vm-placement for energy-efficient large-scale cloud systems. In
Globecom Workshops (GC Wkshps), 2012 IEEE, pages 708–713. IEEE, 2012.

[77] Negin Kord and Hassan Haghighi. An energy-efficient approach for virtual machine
placement in cloud based data centers. In Information and Knowledge Technology
(IKT), 2013 5th Conference on, pages 44–49. IEEE, 2013.

[78] Kien Le, Ricardo Bianchini, Jingru Zhang, Yogesh Jaluria, Jiandong Meng, and Thu D
Nguyen. Reducing electricity cost through virtual machine placement in high perform-
ance computing clouds. In Proceedings of 2011 International Conference for High
Performance Computing, Networking, Storage and Analysis, page 22. ACM, 2011.

[79] Kangkang Li, Jie Wu, and Adam Blaisse. Elasticity-aware virtual machine placement

185
 for cloud datacenters. In Cloud Networking (CloudNet), 2013 IEEE 2nd International
Conference on, pages 99–107. IEEE, 2013.

[80] Kangkang Li, Huanyang Zheng, and Jie Wu. Migration-based virtual machine place-
ment in cloud systems. In Cloud Networking (CloudNet), 2013 IEEE 2nd International
Conference on, pages 83–90. IEEE, 2013.

[81] Wubin Li, Petter Svärd, Johan Tordsson, and Erik Elmroth. Cost-optimal cloud service
placement under dynamic pricing schemes. In Proceedings of the 2013 IEEE/ACM
6th International Conference on Utility and Cloud Computing, pages 187–194. IEEE
Computer Society, 2013.

[82] Wubin Li, Johan Tordsson, and Erik Elmroth. Modeling for dynamic cloud schedul-
ing via migration of virtual machines. In Cloud Computing Technology and Science
(CloudCom), 2011 IEEE Third International Conference on, pages 163–171. IEEE,
2011.

[83] Wubin Li, Johan Tordsson, and Erik Elmroth. Modeling for dynamic cloud schedul-
ing via migration of virtual machines. In Cloud Computing Technology and Science
(CloudCom), 2011 IEEE Third International Conference on, pages 163–171. IEEE,
2011.

[84] Wubin Li, Johan Tordsson, and Erik Elmroth. Virtual machine placement for predict-
able and time-constrained peak loads. In Economics of Grids, Clouds, Systems, and
Services, pages 120–134. Springer, 2012.

[85] Xin Li, Zhuzhong Qian, Sanglu Lu, and Jie Wu. Energy efficient virtual machine
placement algorithm with balanced and improved resource utilization in a data center.
Mathematical and Computer Modelling, 58(5):1222–1235, 2013.

[86] Jenn-Wei Lin and Chien-Hung Chen. Interference-aware virtual machine placement in
cloud computing systems. In Computer & Information Science (ICCIS), 2012 Inter-
national Conference on, volume 2, pages 598–603. IEEE, 2012.

186
[87] Xiao-Fang Liu, Zhi-Hui Zhan, Ke-Jing Du, and Wei-Neng Chen. Energy aware virtual
machine placement scheduling in cloud computing based on ant colony optimization
approach. In Proceedings of the 2014 conference on Genetic and evolutionary compu-
tation, pages 41–48. ACM, 2014.

[88] Fabio López-Pires. Many-objective resource allocation in cloud computing datacenters.

In 2016 IEEE International Conference on Cloud Engineering Workshop (IC2EW),
pages 213–215, April 2016.

[89] Fabio López-Pires and Benjamín Barán. Multi-objective virtual machine placement
with service level agreement: A memetic algorithm approach. In Proceedings of the
2013 IEEE/ACM 6th International Conference on Utility and Cloud Computing, pages
203–210. IEEE Computer Society, 2013.

[90] Fabio López-Pires and Benjamín Barán. Multi-objective virtual machine placement
with service level agreement: A memetic algorithm approach. In Proceedings of the
2013 IEEE/ACM 6th International Conference on Utility and Cloud Computing, pages
203–210. IEEE Computer Society, 2013.

[91] Fabio López-Pires and Benjamín Barán. Virtual machine placement literature review
(data). Technical report, Polytechnic School, National University of Asunción, June
2014.

[92] Fabio López-Pires and Benjamín Barán. A many-objective optimization framework

for virtualized datacenters. In Proceedings of the 2015 5th International Conference
on Cloud Computing and Service Science, pages 439–450, 2015.

[93] Fabio López-Pires and Benjamín Barán. Virtual machine placement literature review.
http://arxiv.org/abs/1506.01509, 2015.

[94] Fabio López-Pires and Benjamín Barán. A virtual machine placement taxonomy. In
Cluster, Cloud and Grid Computing (CCGrid), 2015 15th IEEE/ACM International
Symposium on, pages 159–168. IEEE Computer Society, May 2015.

187
 [95] Fabio López-Pires and Benjamín Barán. Many-objective virtual machine placement.
Journal of Grid Computing, 2016.

[96] Fabio López-Pires and Benjamín Barán. Cloud computing resource allocation tax-
onomies. International Journal of Cloud Computing, 2017.

[97] Fabio López-Pires and Benjamín Barán. Many-objective optimization for virtual ma-
chine placement in cloud computing. In Sanjay Chaudhary, Gaurav Somani, and
Rajkumar Buyya, editors, Research Advances in Cloud Computing, chapter 11, pages
266–290. Springer, 2017.

[98] Fabio López-Pires, Benjamín Barán, Augusto Amarilla, Leonardo Benítez, Rodrigo
Ferreira, and Saúl Zalimben. An experimental comparison of algorithms for virtual
machine placement considering many objectives. In 9th Latin America Networking
Conference (LANC), pages 75–79, 2016.

[99] Fabio López-Pires, Benjamín Barán, Leonardo Benítez, Saúl Zalimben, and Augusto
Amarilla. Virtual machine placement for elastic infrastructures in overbooked cloud
computing datacenters under uncertainty. Future Generation Computer Systems,
28(5):755–768, 2017.

[100] Fabio López-Pires, Elias Melgarejo, and Benjamín Barán. Virtual machine placement.
A multi-objective approach. In Computing Conference (CLEI), 2013 XXXIX Latin
American, pages 1–8. IEEE, 2013.

[101] Kuan Lu, Ramin Yahyapour, Philipp Wieder, Constantinos Kotsokalis, Edwin Yaqub,
and Ali Imran Jehangiri. Qos-aware vm placement in multi-domain service level agree-
ments scenarios. In Cloud Computing (CLOUD), 2013 IEEE Sixth International Con-
ference on, pages 661–668. IEEE, 2013.

[102] José Luis Lucas Simarro, Rafael Moreno-Vozmediano, Ruben S Montero, and Ig-
nacio Martín Llorente. Dynamic placement of virtual machines for cost optimization in
multi-cloud environments. In High Performance Computing and Simulation (HPCS),
2011 International Conference on, pages 1–7. IEEE, 2011.

188
[103] Fumio Machida, Masahiro Kawato, and Yoshiharu Maeno. Redundant virtual machine
placement for fault-tolerant consolidated server clusters. In Network Operations and
Management Symposium (NOMS), 2010 IEEE, pages 32–39. IEEE, 2010.

[104] Sunilkumar S Manvi and Gopal Krishna Shyam. Resource management for infras-
tructure as a service (iaas) in cloud computing: A survey. Journal of Network and
Computer Applications, 41:424–440, 2014.

[105] Ching Chuen Teck Mark, Dusit Niyato, and Tham Chen-Khong. Evolutionary optimal
virtual machine placement and demand forecaster for cloud computing. In Advanced
Information Networking and Applications (AINA), 2011 IEEE International Confer-
ence on, pages 348–355. IEEE, 2011.

[106] Nick McKeown, Tom Anderson, Hari Balakrishnan, Guru Parulkar, Larry Peterson,
Jennifer Rexford, Scott Shenker, and Jonathan Turner. Openflow: enabling innovation
in campus networks. ACM SIGCOMM Computer Communication Review, 38(2):69–
74, 2008.

[107] Peter Mell and Tim Grance. The nist definition of cloud computing. National Institute
of Standards and Technology, 53(6):50, 2009.

[108] Xiaoqiao Meng, Vasileios Pappas, and Li Zhang. Improving the scalability of data
center networks with traffic-aware virtual machine placement. In INFOCOM, 2010
Proceedings IEEE, pages 1–9. IEEE, 2010.

[109] Kevin Mills, J Filliben, and Christopher Dabrowski. Comparing vm-placement al-
gorithms for on-demand clouds. In Cloud Computing Technology and Science (Cloud-
Com), 2011 IEEE Third International Conference on, pages 91–98. IEEE, 2011.

[110] Mayank Mishra and Anirudha Sahoo. On theory of vm placement: Anomalies in

existing methodologies and their mitigation using a novel vector based approach. In
Cloud Computing (CLOUD), 2011 IEEE International Conference on, pages 275–282.
IEEE, 2011.

189
[111] Ismael Solis Moreno, Renyu Yang, Jie Xu, and Tianyu Wo. Improved energy-efficiency
in cloud datacenters with interference-aware virtual machine placement. In Autonom-
ous Decentralized Systems (ISADS), 2013 IEEE Eleventh International Symposium on,
pages 1–8. IEEE, 2013.

[112] Jammily Ortigoza, Fabio López-Pires, and Benjamín Barán. A taxonomy on dynamic
environments for provider-oriented virtual machine placement. In 2016 IEEE Interna-
tional Conference on Cloud Engineering (IC2E), pages 214–215, April 2016.

[113] Jammily Ortigoza, Fabio López-Pires, and Benjamín Barán. Workload generation
for virtual machine placement in cloud computing environments. In 2016 XLII Latin
American Computing Conference (CLEI), pages 1–9, Oct 2016.

[114] Jing Tai Piao and Jun Yan. A network-aware virtual machine placement and migration
approach in cloud computing. In Grid and Cooperative Computing (GCC), 2010 9th
International Conference on, pages 87–92. IEEE, 2010.

[115] Diego P Pinto-Roa, Carlos A Brizuela, and Benjamín Barán. Multi-objective routing
and wavelength converter allocation under uncertain traffic. Optical Switching and
Networking, 16:1–20, 2015.

[116] John J Prevost, Kranthimanoj Nagothu, Brian Kelley, and Mo Jamshidi. Optimal
update frequency model for physical machine state change and virtual machine place-
ment in the cloud. In System of Systems Engineering (SoSE), 2013 8th International
Conference on, pages 159–164. IEEE, 2013.

[117] Bruno César Ribas, Rubens Massayuki Suguimoto, Razer ANR Montano, Fabiano
Silva, and Marcos Castilho. Pbfvmc: A new pseudo-boolean formulation to virtual-
machine consolidation. In Intelligent Systems (BRACIS), 2013 Brazilian Conference
on, pages 201–206. IEEE, 2013.

[118] Bruno Cesar Ribas, Rubens Massayuki Suguimoto, Razer ANR Montano, Fabiano
Silva, Luis de Bona, and Marcos A Castilho. On modelling virtual machine consolida-

190
 tion to pseudo-boolean constraints. In Advances in Artificial Intelligence–IBERAMIA
2012, pages 361–370. Springer, 2012.

[119] Samir Ben Salem, Mourad Fakhfakh, Dorra Sellami Masmoudi, Mourad Loulou,
Patrick Loumeau, and Nouri Masmoudi. A high performances cmos ccii and high
frequency applications. Analog Integrated Circuits and Signal Processing, 49(1):71–78,
2006.

[120] Leili Salimian and Faramarz Safi. Survey of energy efficient data centers in cloud
computing. In Proceedings of the 2013 IEEE/ACM 6th International Conference on
Utility and Cloud Computing, pages 369–374. IEEE Computer Society, 2013.

[121] Katsunori Sato, Masaki Samejima, and Norihisa Komoda. Dynamic optimization of
virtual machine placement by resource usage prediction. In Industrial Informatics
(INDIN), 2013 11th IEEE International Conference on, pages 86–91. IEEE, 2013.

[122] Amazon Web Services. Amazon ec2 instances. http://aws.amazon.com/ec2/instance-

types/, June 2015.

[123] Jiyuan Shi, Fang Dong, Jinghui Zhang, Junzhou Luo, and Ding Ding. Two-phase online
virtual machine placement in heterogeneous cloud data center. In Systems, Man, and
Cybernetics (SMC), 2015 IEEE International Conference on, pages 1369–1374. IEEE,
2015.

[124] Lei Shi, Bernard Butler, Dmitri Botvich, and Brendan Jennings. Provisioning of re-
quests for virtual machine sets with placement constraints in iaas clouds. In Integrated
Network Management (IM 2013), 2013 IFIP/IEEE International Symposium on, pages
499–505. IEEE, 2013.

[125] Weiming Shi and Bo Hong. Towards profitable virtual machine placement in the
data center. In Utility and Cloud Computing (UCC), 2011 Fourth IEEE International
Conference on, pages 138–145. IEEE, 2011.

[126] Soichi Shigeta, Hiroyuki Yamashima, Tsunehisa Doi, Tsutomu Kawai, and Keisuke
Fukui. Design and implementation of a multi-objective optimization mechanism for

191
 virtual machine placement in cloud computing data center. In Cloud Computing, pages
21–31. Springer, 2013.

[127] Vivek Shrivastava, Petros Zerfos, Kang-Won Lee, Hani Jamjoom, Yew-Huey Liu, and
Suman Banerjee. Application-aware virtual machine migration in data centers. In
INFOCOM, 2011 Proceedings IEEE, pages 66–70. IEEE, 2011.

[128] Nongmaithem Ajith Singh and M Hemalatha. Reduce energy consumption through
virtual machine placement in cloud data centre. In Mining Intelligence and Knowledge
Exploration, pages 466–474. Springer, 2013.

[129] Fei Song, Daochao Huang, Huachun Zhou, Hongke Zhang, and Ilsun You. An
optimization-based scheme for efficient virtual machine placement. International
Journal of Parallel Programming, 42(5):853–872, 2014.

[130] Benjamin Speitkamp and Martin Bichler. A mathematical programming approach for
server consolidation problems in virtualized data centers. Services Computing, IEEE
Transactions on, 3(4):266–278, 2010.

[131] Meng Sun, Weidong Gu, Xinchang Zhang, Huiling Shi, and Wei Zhang. A matrix
transformation algorithm for virtual machine placement in cloud. In Trust, Security
and Privacy in Computing and Communications (TrustCom), 2013 12th IEEE Inter-
national Conference on, pages 1778–1783. IEEE, 2013.

[132] Petter Sv, Wubin Li, Eddie Wadbro, Johan Tordsson, Erik Elmroth, et al. Continu-
ous datacenter consolidation. In 2015 IEEE 7th International Conference on Cloud
Computing Technology and Science (CloudCom), pages 387–396. IEEE, 2015.

[133] Petter Svärd, Benoit Hudzia, Steve Walsh, Johan Tordsson, and Erik Elmroth. Prin-
ciples and performance characteristics of algorithms for live vm migration. ACM
SIGOPS Operating Systems Review, 49(1):142–155, 2015.

[134] Francisco Talavera, Jorge Crichigno, and Benjamín Barán. Policies for dynamical
multiobjective environment of multicast traffic engineering. In IEEE ICT, 2005.

192
[135] Maolin Tang and Shenchen Pan. A hybrid genetic algorithm for the energy-efficient
virtual machine placement problem in data centers. Neural Processing Letters, pages
1–11, 2014.

[136] Andrei Tchernykh, Uwe Schwiegelsohn, Vassil Alexandrov, and El-ghazali Talbi. To-
wards understanding uncertainty in cloud computing resource provisioning. Procedia
Computer Science, 51:1772–1781, 2015.

[137] Michael Tighe and Michael Bauer. Integrating cloud application autoscaling with dy-
namic vm allocation. In 2014 IEEE Network Operations and Management Symposium
(NOMS), pages 1–9. IEEE, 2014.

[138] Luis Tomás and Johan Tordsson. Improving cloud infrastructure utilization through
overbooking. In Proceedings of the 2013 ACM Cloud and Autonomic Computing Con-
ference, CAC ’13, pages 5:1–5:10, New York, NY, USA, 2013. ACM.

[139] Luis Tomás and Johan Tordsson. Improving cloud infrastructure utilization through
overbooking. In Proceedings of the 2013 ACM Cloud and Autonomic Computing Con-
ference, page 5. ACM, 2013.

[140] Johan Tordsson, Rubén S Montero, Rafael Moreno-Vozmediano, and Ignacio M

Llorente. Cloud brokering mechanisms for optimized placement of virtual machines
across multiple providers. Future Generation Computer Systems, 28(2):358–367, 2012.

[141] Ming-Han Tsai, Jerry Chou, and Jye Chen. Prevent vm migration in virtualized
clusters via deadline driven placement policy. In Cloud Computing Technology and
Science (CloudCom), 2013 IEEE 5th International Conference on, volume 1, pages
599–606. IEEE, 2013.

[142] Konstantinos Tsakalozos, Mema Roussopoulos, and Alex Delis. Vm placement in non-
homogeneous iaas-clouds. In Service-Oriented Computing, pages 172–187. Springer,
2011.

193
[143] Christian von Lücken, Benjamín Barán, and Carlos Brizuela. A survey on multi-
objective evolutionary algorithms for many-objective problems. Computational Opti-
mization and Applications, pages 1–50, 2014.

[144] Shangguang Wang, Zhipiao Liu, Zibin Zheng, Qibo Sun, and Fangchun Yang. Particle
swarm optimization for energy-aware virtual machine placement optimization in vir-
tualized data centers. In Parallel and Distributed Systems (ICPADS), 2013 Interna-
tional Conference on, pages 102–109. IEEE, 2013.

[145] Shao-Heng Wang, Patrick P-W Huang, Charles H-P Wen, and Li-Chun Wang. Eqvmp:
Energy-efficient and qos-aware virtual machine placement for software defined data-
center networks. In Information Networking (ICOIN), 2014 International Conference
on, pages 220–225. IEEE, 2014.

[146] Wenting Wang, Haopeng Chen, and Xi Chen. An availability-aware virtual machine
placement approach for dynamic scaling of cloud applications. In Ubiquitous Intelli-
gence & Computing and 9th International Conference on Autonomic & Trusted Com-
puting (UIC/ATC), 2012 9th International Conference on, pages 509–516. IEEE, 2012.

[147] Grant Wu, Maolin Tang, Yu-Chu Tian, and Wei Li. Energy-efficient virtual machine
placement in data centers by genetic algorithm. In Neural Information Processing,
pages 315–323. Springer, 2012.

[148] Jan-Jan Wu, Pangfeng Liu, and Jyun-Shiung Yang. Workload characteristics-aware
virtual machine consolidation algorithms. In Proceedings of the 2012 IEEE 4th Inter-
national Conference on Cloud Computing Technology and Science (CloudCom), pages
42–49. IEEE Computer Society, 2012.

[149] Yongqiang Wu, Maolin Tang, and Warren Fraser. A simulated annealing algorithm for
energy efficient virtual machine placement. In Systems, Man, and Cybernetics (SMC),
2012 IEEE International Conference on, pages 1245–1250. IEEE, 2012.

[150] Jing Xu and Jose AB Fortes. Multi-objective virtual machine placement in virtualized
data center environments. In Green Computing and Communications (GreenCom),

194
 2010 IEEE/ACM Int’l Conference on & Int’l Conference on Cyber, Physical and Social
Computing (CPSCom), pages 179–188. IEEE, 2010.

[151] Wenying Yue and Qiushuang Chen. Dynamic placement of virtual machines with
both deterministic and stochastic demands for green cloud computing. Mathematical
Problems in Engineering, 2014, 2014.

[152] Kamran Zamanifar, Nader Nasri, and M Nadimi-Shahraki. Data-aware virtual ma-
chine placement and rate allocation in cloud environment. In Advanced Computing
& Communication Technologies (ACCT), 2012 Second International Conference on,
pages 357–360. IEEE, 2012.

[153] Xiaodong Zhang, Ying Zhang, Xing Chen, Kai Liu, Gang Huang, and Jianfeng Zhan.
A relationship-based vm placement framework of cloud environment. In Proceedings
of the 2013 IEEE 37th Annual Computer Software and Applications Conference, pages
124–133. IEEE Computer Society, 2013.

[154] Xiaoqing Zhang, Qiang Yue, and Zhongtang He. Dynamic energy-efficient virtual
machine placement optimization for virtualized clouds. In Proceedings of the 2013
International Conference on Electrical and Information Technologies for Rail Trans-
portation (EITRT2013)-Volume II, pages 439–448. Springer, 2014.

[155] Qinghua Zheng, Rui Li, Xiuqi Li, Nazaraf Shah, Jianke Zhang, Feng Tian, Kuo-Ming
Chao, and Jia Li. Virtual machine consolidated placement based on multi-objective
biogeography-based optimization. Future Generation Computer Systems, 2015.

[156] Qinghua Zheng, Rui Li, Xiuqi Li, Nazaraf Shah, Jianke Zhang, Feng Tian, Kuo-Ming
Chao, and Jia Li. Virtual machine consolidated placement based on multi-objective
biogeography-based optimization. Future Generation Computer Systems, 54:95–122,
2016.

195