CLOUD

SECURITY

Practical Use of
Microservices 6

Economics of
Microservices 16

SEPTEMBER/OCTOBER 2016
www.computer.org/cloudcomputing
 Experience the Newest and Most Advanced
Thinking in Big Data Analytics

03 November 2016 | Austin, TX

Big Data: Big Hype or Big Imperative? Rock Star Speakers

BOTH.
Business departments know the promise of
big dataand they want it! But newly minted
data scientists cant yet meet expectations, and
technologies remain immature. Yes, big data is
transforming the way we doeverything. But
knowing that doesnt help you decide what steps
to take tomorrow to assure your companys future.
Thats why May 24 is your real-world answer. Kirk Borne Satyam Priyadarshy Bill Franks
Come meet the experts who are grappling with and Principal Data Chief Data Scientist, Chief Analytics
solving the problems you face in mining the value Scientist, Halliburton Officer, Teradata
of big data. You literally cant aord to miss the all Booz Allen Hamilton
new Rock Stars of Big Data 2016.

www.computer.org/bda
 Now theres
even more to
love about your
membership...

Read all your IEEE Computer Society

magazines and journals yourWAY on

NO
ADDITIONAL
FEE

ON
ON YOUR COMPUTER ON YOUR eREADER
ON YOUR SMARTPHONE ON YOUR TABLET

Introducing myCS, the digital magazine portal from IEEE Computer Society.
FinallyGo beyond static, hard-to-read PDFs. Our go-to portal makes it easy to access and customize your
favorite technical publications like Computer, IEEE Software, IEEE Security and Privacy, and more. Get started
today for state-of-the-art industry news and a fully adaptive experience.

LEARN MORE AT: mycs.computer.org

Take the
CS Library
wherever
you go!
IEEE Computer Society magazines and Transactions
are available to subscribers in the portable ePub format.

Just download the articles from the IEEE Computer

Society Digital Library, and you can read them on any device
that supports ePub, including:

Adobe Digital Editions (PC, MAC)

iBooks (iPad, iPhone, iPod touch)
Nook (Nook, PC, MAC, Android, iPad, iPhone, iPod, other devices)
EPUBReader (FireFox Add-on)
Stanza (iPad, iPhone, iPod touch)
ibis Reader (Online)
Sony Reader Library (Sony Reader devices, PC, Mac)
Aldiko (Android)
Bluere Reader (iPad, iPhone, iPod touch)
Calibre (PC, MAC, Linux)
(Can convert EPUB to MOBI format for Kindle)

www.computer.org/epub
 Editor in Chief
Mazin Yousif, T-Systems International, mazin@computer.org

Editorial Board
Pascal Bouvry, University of Luxembourg David Linthicum, Cloud Technology Partners
Ivona Brandic, Vienna University of Technology Christine Miyachi, Xerox Corporation
Christopher Crin, University of Paris 13 Omer Rana, Cardiff University
Kim-Kwang Raymond Choo, University Rajiv Ranjan, Newcastle University
of Texas at San Antonio Lutz Schubert, Ulm University
Beniamino Di Martino, Second University of Naples Alan Sill, Texas Tech University
Mianxiong Dong, Muroran Institute of Technology Zahir Tari, RMIT University
Keith G. Jeffery, Keith G. Jeffery Consultants Joe Weinman
and Cardiff University Yongwei Wu, Tsinghua University

Steering Committee
Sherman Shen, University of Waterloo (chair, Hui Lei, IBM
Communications Society liaison) V.O.K. Li, University of Hong Kong
Kirsten Ferguson-Boucher, Aberystwyth University (Communications Society liaison)
Raouf Boutaba, University of Waterloo Rolf Oppliger, eSecurity Technologies
(Communications Society Liaison) Manish Parashar, Rutgers, the State University of New Jersey
Carl Landwehr, NSF, IARPA (EIC Emeritus IEEE S&P)

Editorial Staff CS Magazine

Brian Brannon Lead Editor bbrannon@computer.org Operations Committee
Joan Taylor Content Editor Forrest Shull (chair), Brian Blake, Maria Ebling, Lieven
Eeckhout, Miguel Encarnacao, Nathan Ensmenger,
Annie Lubinsky, Keri Schreiner, Jenny Stout
Sumi Helal, San Murugesan, Shari Lawrence
Contributing Editors
Pfleeger, Yong Rui , Diomidis Spinellis, George
Carmen Garvey, Jennie Zhu-Mai Production & Design
K. Thiruvathukal, Mazin Yousif, Daniel Zeng
Robin Baldwin Senior Manager, Editorial Services
Evan Butterfield Products and Services Director
Sandy Brown Senior Business Development Manager CS Publications Board
Marian Anderson Senior Advertising Coordinator Alfredo Benso, Irena Bojanova, Greg Byrd,
Min Chen, Robert Dupuis, David S. Ebert,
Niklas Elmqvist, Davide Falessi, William Ribarsky,
Forrest Shull, Melanie Tory

IEEE Cloud Computing (ISSN 2325-6095) is published bimonthly by the IEEE
Computer Society. IEEE headquarters: Three Park Ave., 17th Floor, New York, NY
10016-5997. IEEE Computer Society Publications Office: 10662 Los Vaqueros Cir., Los
Alamitos, CA 90720; +1 714 821 8380; fax +1 714 821 4010. IEEE Computer Society
headquarters: 2001 L St., Ste. 700, Washington, DC 20036.
Subscription rates: IEEE Computer Society members get the lowest rate of US$39
per year. Go to www.computer.org/subscribe to order and for more information on
other subscription prices.
 Initial state Data loss check

22
Elastic
action
S

1-y

v v
:vs :ws $:ms

CONTENT
What will the future of cloud computing look like? What are some of the issues
professionals, practitioners, and researchers need to address when utilizing cloud
services? IEEE Cloud Computing magazine serves as a forum for the constantly
shifting cloud landscape, bringing you original research, best practices, in-depth
analysis, and timely columns from luminaries in the field.

THEmE ARTIClEs

22 Guest Editors Introduction: 44 Cryptographic Public Verication of

Cloud Security Data Integrity for Cloud Storage Systems
Peter Mueller, Chin-Tser Huang, Shui Yu, Zahir Tari, Yuan Zhang, Chunxiang Xu, Hongwei Li , and
and Ying-Dar Lin Xiaohui Liang

26 Online Analysis of Security Risks in 54 To Docker or Not to Docker:

Elastic Cloud Applications A Security Perspective
Athanasios Naskos, Anastasios Gounaris, Theo Combe, Antony Martin, and Roberto Di Pietro
Haralambos Mouratidis, and Panagiotis Katsaros

64 User-Centric Security and

34 Privacy-Preserving Access to Big Data Dependability in the Clouds-of-Clouds
in the Cloud Marc Lacoste, Markus Miettinen, Nuno Neves, Fernando
Peng Li, Song Guo, Toshiaki Miyazaki, Miao Xie, M.V. Ramos, Marko Vukolic, Fabien Charmet, Reda Yaich,
Jiankun Hu, and Weihua Zhuang Krzysztof Oborzynski, Gitesh Vernekar, and Paulo Sousa
k Data leakage check Next state External code
violation check
repositories

y z Github, Private repositories External repositories

26 54
for example (dependencies, webs
Public repositories for example)
S
1-y 1-z
Code docker build Code
github hook
v v
Dockerfile
:vs :ws $:ms
Images repositories

Private repositories
September/October 2016
Private reposito
Volume 3, Issue 5
Alternate
Public repositories registry
www.computer.org/cloudcomputing
Third-party
Public repositor
Docker repositories
hub
Official repositories

(b)
docker pull Image docker pull Image
docker hook
Development
environment

Production
environment
Columns
Docker Orchestra
(Kubernet
4 From the Editor in Chief host
76 Standards
Cont. Cont.Now
Cont. for examp
Microservices The Design and Architecture Tasks
Mazin Yousif Docker daemon
of Microservices Commands
Service
Alan Sill Host libraries

6 Cloud Tidbits Host OS

Practical Use of Microservices in Moving 81 Blue Skies Hardware
Workloads to the Cloud Opendocker run
Issues in / ps /
Scheduling Microservices in
inspect / exec ...
David S. Linthicum the Cloud
(c)
Maria Fazio, Antonio Celesti, Rajiv Ranjan, Chang Liu,

10 Cloud and the Law Lydia Chen, and Massimo Villari

Challenges in Delivering Software in the

Cloud as Microservices
Christian Esposito, Aniello Castiglione, and
Kim-Kwang Raymond Choo 52 Advertising Index
62 IEEE CS Information
16 Cloud Economics
The Economics of Microservices
Andy Singleton

Reuse Rights and Reprint Permissions: Educational or personal use of this material is permitted without fee, provided such use: 1) is not made for profit; 2)
includes this notice and a full citation to the original work on the first page of the copy; and 3) does not imply IEEE endorsement of any third-party products
or services. Authors and their companies are permitted to post the accepted version of their IEEE-copyrighted material on their own Web servers without
permission, provided that the IEEE copyright notice and a full citation to the origin al work appear on the first screen of the posted copy. An accepted manu-
script is a version which has been revised by the author to incorporate review suggestions, but not the published version with copyediting, proofreading and
formatting added by IEEE. For more information, please go to: http://www.ieee.org/publications_standards/publications/rights/paperversionpolicy.html.
Permission to reprint/republish this material for commercial, advertising, or promotional purposes or for creating new collective works for resale or redistribu-
tion must be obtained from the IEEE by writing to the IEEE Intellectual Property Rights Office, 445 Hoes Lane, Piscataway, NJ 08854-4141 or pubs-permissions
@ieee.org. Copyright 2016 IEEE. All rights reserved.
Abstracting and Library Use: Abstracting is permitted with credit to the source. Libraries are permitted to photocopy for private use of patrons, provided the
per-copy fee indicated in the code at the bottom of the first page is paid through the Copyright Clearance Center, 222 Rosewood Drive, Danvers, MA 01923.
IEEE prohibits discrimination, harassment, and bullying. For more information, visit www.ieee.org/web/aboutus/whatis/policies/p9-26.html.
 From the EditoR in Chief
Microservices
Software architectures are con-
stantly evolving. We initially had, and still
have, monolithic software architectures, where the
deployed software stack is one long string of code
that does all, including functional and nonfunc-
tional application requirements. This is sometimes
referred to as tightly coupled code. Such an ap-
proach might suffice for small applications devel-
oped by small teams as small projects. However, as
complexity (in terms of more involved functional
and nonfunctional requirements) increases, such
an approach becomes inefficient, costly, and time
consuming. Making a change is terrifying because
youll need to retest the whole application. Starting
time can be slow because you need to upload large
amounts of code. The whole thing can become a
nightmare. Another interesting twist here is that be-
cause the code is so tightly coupled, its very difficult
to incorporate new technology trends.
Modular software architectures evolved mainly
to ease the complexities around monolithic archi-
tectures. They include loosely coupled modules, la
distributed applications. Functions, services, and mi-
croservices are other examples of modular software
architectures. They differ in their size and complex-
ity, the mechanisms for interconnecting them, the
scope of integration, and whether they deliver single
Mazin Yousif
T-Systems International
mazin@computer.org
4 I E E E C l o u d C o m p u t i n g p u b l i s h e d b y t h e I E E E c o m p u t e r s o cie t y 
or multiple tasks. For example, service-oriented archi-
tectures (SOAs) bring services together through an
enterprise service bus (ESB). An ESB isnt as simple
as it might seem; it must include the glue for all the
connectivity and integration among all the services.
The key architectural advantage of modular
architectures is that they tackle the complexity of
monolithic architectures. There are other advantag-
es, however, mainly as side benefits to breaking the
code into smaller pieces:
Its easier to make changes, update, and test.
There are fewer barriers to introducing new
technology trends.
Theyre likely faster to start.
Its easier to mix and match modules with dif-
ferent profiles in terms of processor and mem-
ory needs, resulting in much better resource
utilization.
Its easier to construct applications by bringing
together modules with different functions.
The market has been positioning microservices
as the hottest new trend in software development,
and it might be. But the confusion around mi-
croservices is everywhere. When you ask N people
to define microservices or what the typical size of
a microservice is, youll likely get N + M different
definitions. Moreover, many mistakenly equate mi-
croservices with containers.
So what are microservices? Theyre programs
with a single task (or unit of work) that also in-
clude all the connectivity to the outside world as
well as the runtime requirements to run the task.
(Note that the word task is generic and refers to
the smallest function possible, but no smaller.)
Regardless, microservices inherit all the benefits
of a modular architecture. Also increasing the de-
veloper communitys interest in microservices are
containers and DevOps, which evolved around the
same time that microservices did. Containers tai-
lor themselves nicely to microservices because they
can be deployed with much less overhead than vir-
tual machines. DevOps represents an approach to
developing, testing, and running code with tighter
collaboration between developers, testers, and op-
erators. Read more about this in David Linthicums
Cloud Tidbits column.
2325-6095/16/$33.00 2016 IEEE
 New Editorial Board Member
W elcome to Christine Miyachi, the newest member
of the IEEE Cloud Computing editorial board.
Christine Miyachi has almost 30 years
of experience working for startups
and large corporations. Shes currently
a systems engineer and architect at
Xerox Corporation and holds several
patents.She works on Xeroxs Exten-
sible Interface Platform, which isa software plat-
form upon which developers can use standard
Web-based tools to create server-based applications
that can be configured for the multifunction periph-
erals touchscreen user interface. Miyachi graduated
from the University of Rochester with a BS in
electrical engineering. She holds two degrees from
the Massachusetts Institute of Technology: an MS
in technology and policy/electrical engineering and
computer science and an MS in engineering and
management. Miyachi writes a blog about software
architecture (http://abstractsoftware.blogspot.com).
For more, see www.christinemiyachi.com.

Microservices (or modular architectures in
general) are better suited for the many complex ap-
plications were building these days. This includes
enterprise applications (that is, confined within the
enterprise) as well as Web-scale applications, where
companies need to scale to reach consumers world-
wide. Microservices, specifically, work well for new
types of applications such as the Internet of Things,
where single-function sensors and actuators are de-
ployed in the field.
Given the complexities of our business environ-
ments, technologys role in the social fabric, and the
flat-world view of things, Im encouraged that tech-
nological evolutions have brought us microservices,
containers, and DevOps.
The columns in this issue address various top-
ics related to microservices. As mentioned earlier,
David Linthicum provides a generic overview of
microservices and relates them to containers and
DevOps. Cloud Economics guest author Andy Sin-
gleton addresses the costs and benefits associated
with microservices. In the Cloud and the Law col-
umn, Christian Esposito, Aniello Castiglione, and
Kim-Kwang Raymond Choo look at the possible se-
curity challenges around microservices and related
mitigation topics requiring more research. In Blue
Skies, Maria Fazio, Antonio Celesti, Rajiv Ranjan,
Lydia Chen, Chang Liu, and Massimo Villari look
at scheduling and efficient resource management for
microservices. Finally, in Standards Now, Alan Sill
explores how microservices exploit both modern and
historical standards and looks at the future of mi-
croservices development.
One last item of news is the addition of Chris-
tine Miyachi to IEEE Cloud Computings edito-
rial board (see the sidebar for a brief biography).
She currently chairs the IEEE Special Technical
Community of Cloud Computing and has worked
diligently to expand its reach to more than 10,000
subscribersan outstanding accomplishment!
Mazin Yousif is the editor in chief of IEEE Cloud
Computing. Hes the chief technology officer and vice
president of architecture for the Royal Dutch Shell
Global account at T-Systems International. Yousif has
a PhD in computer engineering from Pennsylvania
State University. Contact him at mazin@computer
.org.
Read your subscriptions through
the myCS publications portal at
http://mycs.computer.org.

Se p t e m b e r / Oc t o b e r 2 0 1 6 I EEE Clo u d Co m p u t i n g 5
 Cloud Tidbits

Practical Use of abstracted into a larger applicationfor example, a

trading systemthat remote application service has
additional value.
Note that we leverage the behavior of this remote

Microservices service more than the information it produces or

consumes. If youre a programmer, you can view
application services as subroutines or methods
something you invoke to make something happen.

in Moving The basic notion of service-oriented architecture

(SOA), and SOA using cloud computing, is to
leverage these remote services using some controlled
infrastructure that allows applications to invoke

Workloads to remote application services as if they were local to

the application. The result (or goal) is a composite
application made up of many local and remote
application services. Since theyre location and

the Cloud platform independent, they can reside on premises

What is a service, and when is a
service a microservice? Good question.
When using a service, we leverage a remote method
or behavior versus simply extracting or publishing
information to a remote system. Moreover, we
typically abstract this remote service into another
application known as a composite application, which
is usually made up of more than one service.
A good example of a service is a risk analysis
process, which runs within an enterprise to calcu-
late the risk of a financial transaction. This remote
application service is of little use by itself, but when
or within one of many cloud computing providers.
Furthermore, once services are identified and
exposed, or developed from scratch, we might have
services that can be placed on and span both on-
premises and cloud-enabled platforms. So, those are
services in general.1
Enter Microservices
Microservices is an architecture as well as a
mechanism, and is often confused with traditional
SOA-type services. Indeed, theres a great deal
of overlap. Its an architectural pattern in which
complex applications are composed of small,
independent processes that communicate with each
other using language-agnostic APIs.
This is service-oriented computing, at its
essence, decomposing the application down to
the functional primitive, and building it as sets of
services that can be leveraged by other applications,
or the application itself. This is also the foundation
of reuse, and these services are systemic to
the use of containers as well as non-container-
based applications. (See https://blog.akana.com/
the-venn-of-microservices.)

David S. The benefits of this approach include

efficiencies through reuse of microservices. As
Linthicum we rebuild applications for the cloud, we modify
them to expose services that are accessible by other
Cloud Technology Partners applications. More importantly, we can consume
david.linthicum@cloudtp.com services from the rebuilt application so we dont
have to build functionality from scratch.

6 I E E E C l o u d C o m p u t i n g p u b l i s h e d b y t h e I E E E c o m p u t e r s o cie t y  2325-6095/16/$33.00 2016 IEEE

 For instance, some programs have built-in
systems such as credit validations, mapping, and
address validation services that must be maintained.
This can cost upward of hundreds of thousands of
dollars per year. The service-based approach lets us
reach out and consume remote services that provide
this functionality and more, so you can get out of
the business of maintaining services that can be
found in other places. It also lets us expose services
for use within the enterprise by other applications,
or even sell services to other enterprises over the
open Internet.
Enter Containers
The use of containers to wrap or containerize existing
applications comes with a few advantages, including
the ability to reduce complexity by leveraging
container abstractions. The containers remove
dependencies on the underlying infrastructure
services, which reduces the complexity of dealing
with those platforms. This means we can abstract
the access to resources, such as storage, from the
application itself. This makes the application
portable, but also speeds the refactoring of the
applications, since the containers handle much of
the access to native cloud resources.
Containers also offer the ability to leverage
automation to maximize their portability, and,
with portability, their value. Through the use of
automation, were scripting a feature we could also
do manually, such as migrating containers from one
cloud to another. However, this options use cases
have proved limited. Indeed, most new applications
are built to take advantage of containers, but existing
applications are often difficult to containerize. The
objective of leveraging containers seems to be that of
a distributed architectural value versus portability,
as originally thought. However, portability is always
a byproduct of leveraging containers.
Also consider the ability to provide better
security and governance services by placing those
services around rather than within containers. In
many instances, security and governance services
are platform specific, not application specific. For
example, traditional on-premises applications tend
not to have security and governance functions
innate to the application. The ability to place
security and governance services outside of the
Se p t e m b e r / Oc t o b e r 2 0 1 6 I EEE Clo u d Co m p u t i n g
application domain provides better portability and
less complexity when refactoring. The ability to
leverage microservices in this context provides the
same advantages as well, no matter if youre using
containers or not.
Containers can provide better distributed
computing capabilities as well. A traditional
application can be divided into many different
domains, all residing within containers. These
containers can be run on any number of different
cloud platforms, including those that provide the
highest cost and performance efficiencies. So,
applications can be distributed and optimized
according to their utilization of the platform from
within the container.
For example, one could place an I/O-intensive
portion of the application on a bare-metal cloud that
provides the best performance, place a compute-
intensive portion of the application on a public
cloud that can provide the proper scaling and load
balancing, and perhaps even place a portion of the
application on traditional hardware and software.
All of these elements work together to form the
application, and the application is separated into
components that can be optimized.
Refactoring Traditional Applications for
Containers and Microservices
The process of containerizing an application and
service enabling it at the same time is more art
than science at this point. However, certain success
patterns are beginning to emerge as enterprises
begin to migrate traditional applications to the
cloud using containers and service orientation as
the architecture.
Pattern one decides quickly how the application
is to be broken into components that will be run
inside of containers in a distributed environment.
This means breaking the application down to its
functional primitives, and building it back up as
component pieces to minimize the amount of code
that needs to be changed.
Pattern two builds data access as a service for the
applications use, and has all data calls go through
those data services. This will decouple the data from
the application components (containers) and let you
change the data without breaking the application.
Moreover, youre putting data complexity into its
7
 Cloud Tidbits
own domain, which will be another container thats
accessed using data-oriented microservices.
Pattern three splurges on testing. Although
many will point to the stability of containers as a
way around black-box and white-box testing, the
application now exists in a new architecture with
new service dependencies. There could be a lot
of debugging that has to occur up front, before
deployment.
There are other sides to this as well. Lori
MacVittie, one of my advisory board members,
noted in an email that containers and microservices
seem to mix many tangentially related topics
together, but microservices have nothing to do with
When applications are put into
production, those charged with cloud
operations should take advantage of
the container architecture.
containers other than they happened to appear at
the same time.
The focus here should be on the technologies
working together, not on each technology as a
standalone. The concept is to understand that
there are some additive advantages of leveraging
both containers and microservices, which are
indeed independent. That said, if the notion is that
microservices are indeed services in the traditional
sense, I have no retort for that.
Operational Considerations
Cloud-enabled traditional applications must be
managed differently in production than they were
prior to migration. This phase is known as cloud
operations, or the operation of the application
containers in the cloud.
When applications are put into production, those
charged with cloud operations should take advantage
of the container architecture. Manage them as dis-
8 I EEE Clo u d Co m p u t i n g w w w.co m p u t er .o rg /clo u d co m p u t i n g
tributed components that function together to form
the applications, and are also separately scaled. For
instance, the container that manages the user inter-
face can be replicated across servers as the demand
for that container goes up when users log on in the
morning. This provides a handy way for cloud opera-
tions to build autoscaling features around the ap-
plication, to expand and de-expand the use of cloud
resources as needs change.
Most enterprises believe the cloud will become
the new home for applications. However, not all
applications are fit for the cloudat least, not yet.
Care must be taken to select the right applications
to make the move.
The use of containers and
microservices makes things easier.
This approach forces the application
developer charged with refactoring
the application to think about how
to best redesign the applications to
become containerized and service
oriented. In essence, youre taking a
monolithic application and turning it
into something thats more complex
and distributed. However, it should
also be more productive, agile, and cost
effective. Thats the real objective here.
Microservice Coupling
As we create microservices to serve new or existing
applications, we need to understand the benefits of
being loosely coupled. Loose coupling, as related to
microservices, has a few basic patterns.
In the location independence pattern, it doesnt
matter where the microservice exists; the other
components that need to leverage the service can
discover it within a directory and leverage it through
the late binding process. This comes in handy when
youre leveraging microservices that are consistently
changing physical and logical locations, especially
services outside of your organization that you might
not own, such as cloud-delivered resources. Your
risk calculation service might exist on premises on
Monday and within the cloud on Tuesday, and it
should make no difference to you.
Dynamic discovery is key to this concept,
meaning that calling components can locate
microservice information as needed, and without
having to bind tightly to the service. Typically these
services are private, shared, or public services as
they exist within the directory.
In the communications independence pattern,
all components can talk to each other, no matter
how they communicate at the interface or protocol
levels. Thus, we leverage enabling standards, such as
microservices, to mediate the protocol and interface
difference.
The security independence pattern is based on
the concept of mediating the difference between
security models in and between components. This
is a bit difficult to pull off, but necessary to any
service-based architecture. To enable this pattern,
you have to leverage a federated security system that
can create trust between components, no matter
what security model is local to the components. This
has been the primary force behind the number of
federated security standards that have emerged in
support of a loosely coupled model and Web services.
In the instance independence patterns, the ar-
chitecture should support component-to-component
communications using both a synchronous and an
asynchronous model, and not require that the other
component be in any particular state before receiv-
ing the request or message. Thus, if done right, all of
the services should be able to service any requesting
component asynchronously, and retain and manage
state no matter what the sequencing is.
The need for loosely coupled architecture
within your cloud computing solution is really
not the question. If you leverage cloud computing
correctly, you should have a loosely coupled
architecture, except in some rare circumstances.
Analysis and planning are also part of the mix, as
are understanding your requirements and how each
component of your architecture should leverage the
other components of your architecture. Leverage
the coupling model that works best for your
requirements.
Although this seems like a lot of
work, its really only a quick survey
and explanation of the present ser-
vices. Moreover, whereas I recommend basic
concepts and basic approaches, the needs of your
IT environment will be unique and might require
Se p t e m b e r / Oc t o b e r 2 0 1 6 I EEE Clo u d Co m p u t i n g
slightly different approaches. As long as the objec-
tive of having a complete understanding of the mi-
croservices within the problem domain are achieved,
how you do that project is up to you. Another benefit
of understanding the domain at a services level is
that you can easily leverage this work in other direc-
tions, perhaps to support the core enterprise archi-
tecture or build and/or refine your architecture.
Acknowledgments
Portions of this article were adapted from an article
I wrote for TechBeacon.2
References
1. D.S. Linthicum, Cloud Computing and SOA
Convergence in Your Enterprise: A Step-by-Step
Guide, Addison-Wesley, 2009.
2. D. Linthicum, From Containers to Microservices:
How to Modernize Legacy Applications,
TechBeacon, 19 Jan. 2016; http://techbeacon.com/
containers-microservices-how-modernize-legacy
-applications.
David S. Linthicum is senior vice president
of Cloud Technology Partners. Hes also Gigaoms
research analyst and frequently writes for InfoWorld
on deep technology subjects. His research interests
include complex distributed systems, including
cloud computing, data integration, service-oriented
architecture, Internet of Things, and big data systems.
Contact him at david@davidlinthicum.com.
Read your subscriptions through
the myCS publications portal at
http://mycs.computer.org.
9
 Cloud and the Law

Challenges the design of service-oriented soft-

ware using a set of small services.
These services are small applications that can be de-
ployed independently, with a precise and hardened

in Delivering interface, and easily integrated. Microservices are

supported by middleware for communication and a
platform for flexible and low-cost deployment.
The small applications have a high degree of in-

Software in ternal cohesion around a single task and can cope

with a simple responsibility. Here, we consider re-
sponsibility as doing (or being responsible for)
one activity only. The activity can include serving

the Cloud as or representing a particular resource. Practical ex-

amples include logging incoming messages within
a database or a file, or managing a message queue
by taking, processing, and discarding queued mes-

Microservices sages. Such a concept started within the industrial

practice of splitting large monolithic applications
into small cooperating pieces to improve their main-
tainability, scalability, and testability. Microservice
architectures are receiving increasing focus, as evi-
denced by search statistics on Google Trends.2 Aca-
Microservices, one of the latest ar- demias interest in microservices is also evidenced
chitectural trends in software engi- by the publication of the first mature academic book
neering,1 can be broadly defined as in 2015.3
The increasing popularity of microservices, de-
spite the effort required to implement them, is prob-
ably due to the benefits associated with their agility,
resilience, scalability, and maintainability.4 Specifi-
Christian Esposito cally, we could enforce separation of concerns by ap-
University of Naples
plying the single responsibility principle.
Federico II

Microservices in the Cloud

Recent advancements in container technologies and
the capability to overcome limitations in virtualiza-
Aniello Castiglione tion have, perhaps, encouraged the use of containers
University of Salerno in the cloud for software development and deploy-
ment. This might also have paved the way for the
adoption of a microservice architectural paradigm
in cloud-hosted software by lowering infrastructure
and maintenance costs.2,5,6 For example, microser-
Kim-Kwang vices support the realization of small (sized) appli-
Raymond Choo cations that are fine-grained and loosely coupled
University of Texas at and communicate through REST protocols. These
San Antonio applications are implemented using APIs provided
by the infrastructure-as-a-service (IaaS) layer for
provisioning data computing, storage, and delivery
capabilities.

10 I E E E C l o u d C o m p u t i n g p u b l i s h e d b y t h e I E E E c o m p u t e r s o cie t y  2325-6095/16/$33.00 2016 IEEE

 The growing adoption of microservices in the
cloud is motivated by the ease of deploying and
updating the software, as well as the provisioned
loose coupling provided by dynamic service dis-
covery and binding. Moreover, structuring the
software to be deployed in the cloud as a collec-
tion of microservices allows cloud service provid-
ers to offer higher scalability guarantees through
more efficient consumption of cloud resources,
and to dynamically and quickly restructure soft-
ware to accommodate growing consumer demand.
Structuring software in smaller computation units
enables optimized allocation of the application
components within proper containers in the vir-
tual machines (VMs) running on top of the host
machine provided by the cloud infrastructure, as
the example in Figure 1 illustrates. This lets us
minimize waste of resources and maximize pack-
ing of the components within a single VM. This
is possible because, despite having to realize the
same application tasks, microservices are typically
thinner than conventional software components
because they use lightweight software technolo-
gies and platforms. This model also allows a sim-
pler and faster migration of software component
instances from one VM to another to satisfy cloud
applications changing resource demands.
Partitioning monolithic applications into small
pieces of computation also allows for the segmen-
tation of application data, and thus avoids having
a single monolithic data storage. Such a solution is
required by the decentralized data governance that
a microservice architecture imposes and is known
in the literature as the sharding pattern.7 It con-
sists of dividing a data store into a set of horizon-
tal partitions, or shards, to improve scalability and
performance when storing and accessing large vol-
umes of data, as in the case of cloud-hosted applica-
tions. The different shards are kept consistent with
respect to a centralized data storage, or through a
distributed protocol.
Security Threats and Legal Ramifications
The key idea underlying the microservice architec-
ture is to distribute application complexity among
narrowly focused and independently deployable
units of computation. Such small components are
loosely coupled by means of an integration middle-
VM1 VM2 VM3
Traditional application
HM1 HM2 HM3
(a) Cloud service provider
Communication bus VM1 VM2 VM3
Microservice-based
application HM1 HM2 HM3
(b) Cloud service provider
Figure 1. Software deployment in a cloud platform using (a) conventional
and (b) microservice-based software.
ware that dynamically and seamlessly resolves the
implicit dependencies among them.
Figure 2 depicts the differences between a con-
ventional monolithic architecture and a microservice
architecture. In the first case (Figure 2a), users in-
teract with a front-end application, which redirects
user requests to multiple instances of the software
hosted within a container and implements all applica-
tion responsibility using the data stored in a database.
In the second case (Figure 2b), the software is split
into multiple components, each implementing only
its given responsibility, hosted in multiple containers,
and/or replicated at the users convenience. The de-
pendencies among the microservices and data shards
are complex and having centralized data storage can
help maintain consistency. Using existing middleware
solutions, such as those offering event notification,8
allows such architectures to cope with the complexity
of connecting multiple entities, even with the dynam-
ic discovery of components.

Se p t e m b e r / Oc t o b e r 2 0 1 6 I EEE Clo u d Co m p u t i n g 11
 Cloud and the Law
Users Database
Front-end application
and load balancer Container
(a)
Shard
Container
Shard
Users Database
Front-end application
Container
(b) and load balancer
Shard
Container
Figure 2. Architectural complexity of (a) monolithic and (b) microservice-
based software.
However, such complexity can result in secu-
rity vulnerabilities affecting one or more vectors in
the architecture,9 including single microservices,10
data shards,11 and the cooperation and orchestra-
tion among microservices.12 In the first vulner-
ability vector, having multiple components within
an architecture can enlarge the attack surface by
presenting more points of vulnerability that can be
exploited to compromise the application. In addition,
the complexity of gluing together a multitude of mi-
croservices complicates the challenge of debugging,
monitoring, and auditing the application, especially
within a cloud thats not under the application own-
ers control. Moreover, in microservice architectures,
reuse is strongly enforced by allowing the use of off-
the-shelf (OTS) components. Deployed OTS compo-
nents represent a security threat and they should be
properly validated, authenticated, and monitored.13
12 I EEE Clo u d Co m p u t i n g w w w.co m p u t er .o rg /clo u d co m p u t i n g
Trustworthiness is also an issue when dealing
with microservices. For example, an adversary could
compromise or gain control of a component, which
isnt uncommon within the public cloud context.
However, in a typical microservice architecture, the
other components assume a trusted component base;
thus, theres a real risk that attacks can be easily
propagated due to the microservices dependencies
and (blind) trust of peer components. In data shard-
ing, security threats are related to data provenance,
trustworthiness, and protection. Specifically, when
data is horizontally segmented into multiple shards,
we need to determine if the data source is verifiable
(or trustworthy) and how to protect against forging
attempts (for example, data modification and tam-
pering) during sharding.
Furthermore, data can be provided by multiple
data sources, and each should have a reputation de-
gree associated with it. This would give data consum-
ers a way to determine the trustworthiness degree
of the data prior to using it in their computations.
Data must also be kept consistent with respect to
eventual replicas (or a centralized data store), since
such a process is vulnerable to exposure, denial of
service, and alteration attacks. Even in the best-case
scenario, where we assume a single microservice has
been demonstrated to be secure and the data shard-
ing mechanism protected, the architecture presents
an additional point of vulnerability: the composition
and coordination middleware can also be targeted
and exploited. In the absence of a centralized orches-
trator, for example, microservices coordinate among
themselves through message-based communications.
These communications, if not protected, wont let us
achieve secure guarantees during data exchange and/
or receive falsified or modified coordination which is
required when undertaking complex and elaborate
functionalities.14
Should a breach or cybersecurity incident oc-
cur as a result of a successful exploitation in any of
the three vectors (single microservices, data shards,
and the cooperation and orchestration among mi-
croservices), who is responsible for users pure eco-
nomic loss (such as direct and indirect financial
loss) under the current legal frameworkthe mi-
croservice provider or the cloud service provider? If
an attacker exploits the trust in peer components
within the architecture, should the compromised
microservice provider be held responsible? What
happens if the compromised microservice provider
is found to have failed to provide reasonable indus-
try-standard security?
Because microservices are relatively new, we
dont know if there are gaps in existing legal frame-
works. For example, because there are no known
cases on which to comment, we cant effectively
assess the adequacy of existing legislation, and
challenges will arise as untested provisions are ap-
plied in civil litigation or prosecution. Thus, we
need regular dialogues between service providers
(such as microservices and cloud services), legal
practitioners, and policymakers to improve under-
standing of the nature and extent of the risks and
associated legal implications. This would, in turn,
guide further responses at the operational and stra-
tegic levels for service providers and provide the evi-
dence base to inform policymakers in governments
to design national regulatory strategies to address
the associated risks more effectively (for example,
without resulting in a floodgate of claims against
service providers).
Mitigation
Conventional solutions such as secure message-
passing middleware, physical security for cloud
infrastructure, and privacy-preserving cloud data
storage schemes might only be effective for one or
more components, but theyre unlikely to address
the security (and certainly not the legal) challenges
due to microservices unique characteristics.
Potential research to address these technical
limitations or security vulnerabilities can focus on
several areas. One area is microservice validation,
which could be conducted both in isolation and in
composition with other microservices, allowing us
to identify and mitigate potential vulnerabilities in
the software, implementation, or interaction be-
tween microservices.
In addition, we need a dynamic, and preferably
lightweight, security monitoring and management
system, responsible for monitoring and enforcing
correct behavior of microservices and other related
activities (such as cooperation). Once the behav-
ior or activities deviate from the norm, the system
should undertake corrective actions and/or coun-
termeasures to bring the application to the correct
Se p t e m b e r / Oc t o b e r 2 0 1 6 I EEE Clo u d Co m p u t i n g
state by autonomously resolving possible compro-
mises and violations.
We also need proper cryptographic schemes for
data protection at rest and in motion, allowing ac-
cess and integrity control to the exchanged messages
and stored shards while allowing privacy-preserving
processing tasks.
We need to strike a balance between
security and performance. For example,
security mitigation solutions should also provide
important nonfunctional requirements, such as
minimal overhead and tamper resistance, so they
minimally impact the systems performance and us-
ers quality of experience. Ideally, the security solu-
tions should continue to function after an attacker
has successfully taken control of the cloud infra-
structure or VMs. Since equipping microservices
with a proper security mechanism might not always
be the best option,10 offering security as a service for
microservice-based cloud applications can be an at-
tractive solution.
References
1. M. Fowler, Microservices Resource Guide,
2016; http://martinfowler.com/microservices.
2. A. Balalaie, A. Heydarnoori, and P. Jamshidi,
Microservices Architecture Enables DevOps,
IEEE Software, vol. 33, no. 3, 2016, pp. 4252.
3. S. Newman, Building Microservices: Designing
Fine-Grained Systems, OReilly Media, 2015.
4. T. Killalea, The Hidden Dividends of Microser-
vices, Comm. ACM, vol. 59, no. 8, 2016, pp.
4245.
5. F. Oliveira et al., Delivering Software with Agil-
ity and Quality in a Cloud Environment, IBM
J. Research and Development, vol. 60, nos. 23,
2016, pp. 10:1-10:11.
6. M. Villamizar et al., Infrastructure Cost Com-
parison of Running Web Applications in the
Cloud Using AWS Lambda and Monolithic and
Microservice Architectures, Proc. 16th IEEE/
ACM Intl Symp. Cluster, Cloud and Grid Com-
puting (CCGrid), 2016, pp. 179182.
7. C.H. Costa et al., Sharding by Hash Partition-
ing: A Database Scalability Pattern to Achieve
Evenly Sharded Database Clusters, Proc. 17th
13
 CLOUD AND THE LAW
Intl Conf. Enterprise Information Systems
(ICEIS), 2015.
8. C. Esposito, D. Cotroneo, and S. Russo, On Re-
liability in Publish/Subscribe Services, Comput-
er Networks, vol. 57, no. 5, 2013, pp. 13181343.
9. N. Dragoni et al., Microservices: Yesterday,
Today, and Tomorrow, 2016; https://arxiv.org/
abs/1606.04036.
10. Y. Sun, S. Nanda, and T. Jaeger, Security-as-a-
Service for Microservices-Based Cloud Applica-
tions, Proc. 7th IEEE Intl Conf. Cloud Comput-
ing Technology and Science (CloudCom), 2015,
pp. 5057.
11. F. Callegati et al., Data Security Issues in MaaS-
Enabling Platforms, Proc. Intl Forum Research
and Technologies for Society and Industry, 2016;
https://hal.inria.fr/hal-01336700.
12. S. Kim et al., High-Assurance Synthesis of
Security Services from Basic Microservices,
2017 B. Ramakrishna Rau Award
Call for Nominations
Honoring contributions to the computer microarchitecture field
New Deadline: 1 May 2017
Established in memory of Dr. B. (Bob) Ramakrishna
Rau, the award recognizes his distinguished career in
promoting and expanding the use of innovative comput-
er microarchitecture techniques, including his innovation
in complier technology, his leadership in academic and
industrial computer architecture, and his extremely high
personal and ethical standards.
WHO IS ELIGIBLE?: The candidate will have made an
outstanding innovative contribution or contributions to microarchitecture,
use of novel microarchitectural techniques or compiler/architecture
interfacing. It is hoped, but not required, that the winner will have also
contributed to the computer microarchitecture community through
teaching, mentoring, or community service.
AWARD: Certificate and a $2,000 honorarium.
PRESENTATION: Annually presented at the ACM/IEEE International
Symposium on Microarchitecture
NOMINATION SUBMISSION: This award requires 3 endorsements.
Nominations are being accepted electronically: www.computer.org/web
/awards/rau
CONTACT US: Send any award-related questions to awards@computer.org
www.computer.org/awards
14 I EEE Clo u d Co m p u t I n g
Proc. 14th Intl Symp. Software Reliability Eng.
(ISSRE), 2003, pp. 154165.
13. D.I. Savchenko, G.I. Radchenko, and O. Taipale,
Microservices Validation: Mjolnirr Platform
Case Study, Proc. 38th Intl Convention Infor-
mation and Comm. Technology, Electronics and
Microelectronics (MIPRO), 2015, pp. 235240.
14. C. Esposito and M. Ciampi, On Security in Pub-
lish/Subscribe Services: A Survey, IEEE Comm.
Surveys and Tutorials, vol. 17, no. 2, 2015, pp.
966997.
christian esposito is adjunct professor at the
University of Naples Federico II, Italy, and research
fellow and adjunct professor at the University of Saler-
no, Italy. His research interests include information
security and reliability, middleware, and distributed
systems. Esposito has a PhD in computer engineer-
ing from the University of Naples Federico II, Italy.
Contact him at christian.esposito@dia.unisa.it.
aniello castiglione is adjunct professor at
the University of Salerno, Italy, and at the University
of Naples Federico II, Italy. His research interests
include security, communication networks, informa-
tion forensics and security, and applied cryptography.
Castiglione has a PhD in computer science from the
University of Salerno, Italy. Hes member of several as-
sociations, including IEEE and ACM. Contact him at
castiglione@ieee.org.
kiM-kwang rayMond choo holds the
Cloud Technology Endowed Professorship at the Uni-
versity of Texas at San Antonio. His research interests
include cyber and information security and digital
forensics. Choo has a PhD in information security
from Queensland University of Technology, Australia.
Hes a fellow of the Australian Computer Society and
a senior member of IEEE. Contact him at raymond
.choo@fulbrightmail.org.
Read your subscriptions through
the myCS publications portal at
http://mycs.computer.org.
w w w.Co m p u t Er .o rg /Clo u d Co m p u t I n g
 IEEE Cloud Computing Call for Papers
Special Issue on
Multicloud
Submission deadline: 2 January 2017 Publication date: July/August 2017
A
s Cloud Computing evolved to a widely used
computing as a service model, limitations and intrinsic
characteristics of monolithic cloud provider offerings
emerged. Moreover, specialized computing power such as
clusters, GPUs, solid state storage, and specific applications
at different service levels can now be acquired as services
from different providers. The use of a combination of cloud
services from various providers can be performed to contour
limitations of a single provider and enhance application
execution by gathering together the necessary specific, on
demand resources for a wide range of applications.
This IEEE Cloud Computing Magazine Special Issue on
Multicloud aims to cover all aspects of connecting multiple
clouds to allow automatic, transparent, and on demand
application execution that takes advantage from the synergy
among resources of different providers. For this synergy to
become effective and efficient, connecting different providers
across their boundaries brings new, challenging efforts.
Multicloud deployment must solve challenges that include
resource management and scheduling, identity management,
trust and security issues, business models, and incentive
mechanisms in multicloud environments. We invite authors to
submit outstanding and original manuscripts on the following
topics within the context of multiclouds:
brokering mechanisms,
resource discovery and management,
security and privacy,
authentication and authorization,
applications and case studies,
auditing and accounting,
multicloud APIs,
monitoring,
data management,
performance modeling and evaluation,
www.computer.org/cloudcomputing
cloud federations,
scheduling and load balancing,
hybrid clouds,
autonomic management,
multicloud and the Internet of Things,
QoS and QoE,
economic and business models,
cross-service-level management (IaaS, PaaS, SaaS,
and XaaS),
incentive mechanisms, and
multiclouds and green computing.
Guest Editors
Dr. Luiz F. Bittencourt, University of Campinas
Dr. Rodrigo N. Calheiros, University of Melbourne
Dr. Craig A. Lee, Aerospace Corporation
Submission Information
Submissions should be 3,000 to 5,000 words long, with a
maximum of 15 references, and should follow the magazines
guidelines on style and presentation (see https://www.computer
.org/web/peer-review/magazines for full author guidelines). All
submissions will be subject to single-blind, anonymous review
in accordance with normal practice for scientific publications.
For more information, contact the guest editors at cc4-2017
@computer.org.
Authors should not assume that the audience will have
specialized experience in a particular subfield. All accepted
articles will be edited according to the IEEE Computer Society
style guide (www.computer.org/web/publications/styleguide).
Submit your papers through Manuscript Central at https://
mc.manuscriptcentral.com/ccm-cs.
 Cloud Economics
The
Economics of
Microservices
Microservices are a solutionper-
haps the only solutionto the
problem of efficiently building and
managing complex software sys-
tems. For medium-sized systems, they can deliver
cost reduction, quality improvement, agility, and
decreased time to market. For large cloud systems,
they fundamentally change the rules of the game.
Microservices are the software equivalent of Lego
bricks: they are proven to work, fit together nicely, and
can be used to rapidly construct complex solutions.
Complexity problems often arise in legacy,
monolithic systems, such as large enterprise re-
source planning or core banking systems, making
them inflexible and expensive to maintain. In many
cases, such systems are linked together so even an
apparently trivial change in a single component
can create errors in that component or others. This
therefore requires regression testing of all compo-
nents and the system as a whole. We see the same
complexity and reliability requirements in new soft-
ware and digital products. When many changes,
ranging from bug fixes to feature enhancements,
Andy Singleton
MAXOS.ai
16 I E E E C l o u d C o m p u t i n g p u b l i s h e d b y t h e I E E E c o m p u t e r s o cie t y 
occur in a monolithic application, it might never be-
come stable and reliable.
With an older monolithic architecture, its dif-
ficult or impossible to release systems over a certain
size. Capers Jones, writing in 1999 after a study of
1,000 applications, wrote that MIS applications
greater than 10,000 function points are rarely suc-
cessful.1 These problems have resulted in failed sys-
tems costing hundreds of billions of dollars over the
last 50 years. Needless to say, in the more than 15
years since, many more failures have occurred.
Fred Brooks classic book The Mythical Man-
Month described the difficulty of building and test-
ing large systems, and the decline in development
productivity that inevitably set in with increasing
system size.2 Brooks observed that adding personnel
to a late project would make it even later, and he de-
clared at the time that there is no silver bullet that
will fix the productivity problem.
Now there might well be a silver bullet: mi-
croservices architectures and continuous integra-
tion enable companies such as Google and Amazon
to build and release enormous systems such as Web
search and e-commerce engines with far more than
10,000 function points, and evolve them with great
speed. For example, Google search and applications
are built from an interlinked set of more than 5,000
services. The company tests and releases its online
systems from a single codebase with more than 2
billion lines of code, deploying about 75,000 com-
mitted changes per week.3
Some sort of service architecture is inevitable
for large systems. After all, there is a physical limit
to the amount of load one server can handleeven
a mainframe, parallel processing supercomputer
or quantum computer. There is a conceptual limit
to the number of functions one release team can
specify, test, and maintain. Beyond those limits,
functions will be placed into separate processes. Ap-
plication code will be separated from the database
tier, front-end proxy servers and load balancers will
be used to organize horizontal scaling to multiple
applications servers, and so on.
So, systems that are large, complex, and need to
scale require service architectures. But, why do we
need micro services? The answer lies in economics
and the efficiency of developing, testing, deploying,
and enhancing software. As an organization gets
2325-6095/16/$33.00 2016 IEEE
better at deploying and managing multiple services,
it will find reasons to make them smaller, more like
microservices. Moreover, microservices approaches
fit particularly well with cloud computing, enabling
the economic benefits of microservices to comple-
ment the economic benefits of cloud computing,
such as cost and user experience optimization.4
Moreover, the rapid release of microservices works
well with cloud-based online systems that dont re-
quire further distribution of software updates. What
we know as the cloud is actually a mass of microser-
vices, speaking to each other over the Internet with
the universal language of APIs. An understanding of
microservices opens the door to participating in this
huge new market and/or exploiting their benefits to
achieve competitive advantage.
Costs of Microservices
Although microservices approaches of-
fer substantial benefits, a microservices
architecture requires extra machinery,
which can impose substantial costs. It
also often requires extra code to com-
municate between services. Instead of
making simple function calls, youll de-
fine API calls or messages, and imple-
ment API calls on each end.
In addition, you need systems
such as service catalogs and messaging and queu-
ing services to discover and then route calls to the
correct service instances. When you make a call to
a microservice, youll go through a proxy or mes-
saging layer that finds a suitable instance. Because
microservices can be event-processing scripts, con-
tainers, or entire virtual machines, youll also want a
systematic way to package and deploy them.
A microservices architecture must include sys-
tems to monitor service performance and behavior
as well as special techniques to handle errors. When
a microservice isnt responding, there is no simple
way for other services to understand the error or
even see there is a problem. Youll need extra code
and monitoring to make sure problems get handled
as errors, rather than just piling up or cascading into
a catastrophe.
Finally, youll need to have new discussions
about when to include functions inside one service,
and when to break them into separate services.
Se p t e m b e r / Oc t o b e r 2 0 1 6 I EEE Clo u d Co m p u t i n g
These decisions are currently more of an art than
a science.
Benefits and Tradeoffs of Microservices
A microservices architecture has substantial costs,
but it also has substantial benefits.
System Size and Complexity
Microservices architectures are good for big sys-
tems but not for small systems. They require extra
machinery to communicate between services, route
to services, deploy services, and monitor services.
If you have a small system that wont change much
and doesnt need to scale, you can avoid this extra
machinery and save time and money by building a
I currently believe that if you have
fewer than about 60 people working
on your system, you dont need a
microservices architecture.
more monolithic system. If you have a large system
that changes frequently and does need to scale, you
benefit from a microservices architecture through
several mechanisms. Its much easier to test and
release the smaller components. You get greater
reliability through redundancy and scalability be-
cause of your ability to increase the instances of any
service thats a bottleneck. You get greater quality
through reuse of field-proven components packaged
into microservices.
Although there are, no doubt, exceptions, I
currently believe that if you have fewer than about
60 people working on your system, you dont need
a microservices architecture. Over this amount of
product complexity, youll probably benefit from a
microservices approach.
A high volume or complex system will inevitably
move to Web services and then to smaller microser-
vices. Theres a limit to the transaction volume and
throughput that a single server can handle. Beyond
17
 Cloud Economics
that point, youll have multiple servers, and routing
and load-balancing machinery typical of a service
architecture. Theres also a limit to the number of
functions that can be tested and maintained on that
server. Beyond that point, youll have multiple ser-
vices. Most modern business systems are well be-
yond these two points.
Release Frequency andAgility
Monolithic applications include a lot of functions
that need to be tested, so they take a long time to
test and releasesometimes a month or more. A
monolithic architecture works well for an installed
software product that requires several months to be
distributed, tested, and installed at customer sites.
In contrast, a microservices-centric develop-
ment team can test and release changes to smaller
A microservices architecture
provides a simple way to change
components without spreading
problems throughout the system.
components more than once per day. A company like
Amazon with thousands of services can make more
than a thousand changes per day, fixing problems
and adding new features. This type of continuous
delivery becomes a powerful tool for developers that
update cloud-based software-as-a-service (SaaS) and
online systems, which in turn means that continu-
ous delivery is also a powerful tool for business agil-
ity and competitive advantage.
Large software teams often have problems with
merging code, in addition to testing. We often see
small teams of one to seven programmers making
changes and then merging these changes with the
work of other teams. Any conflicts in the changes
from other teams need to be studied and fixed by
hand. If many groups are merging, the process be-
comes difficult and unreliable. The microservices
architecture solves this problem by skipping the
18 I EEE Clo u d Co m p u t i n g w w w.co m p u t er .o rg /clo u d co m p u t i n g
merges. Each team can run an integration test on its
code, and release it directly as a packaged service.
A microservices architecture provides a simple
way to change components without spreading prob-
lems throughout the system. If you make a signifi-
cant change to a component, youll want to know
that this doesnt cause problems in the consumers
that use the component. You dont want to test and
fix every consumer. Consumers will be reliable if
they have access to a stable API. So, microservices
maintain their old APIs and behavior, even while
they provide new features in new API calls. In fact,
this is one way to determine how to right-size a ser-
vice: making it small and simple enough that the
API can remain stable.
Microservices approaches typically utilize auto-
mated test scripts that run in a continuous integra-
tion system. Such scripts help ensure
that localized changes that shouldnt
impact the larger system in fact dont.
Continuous integration typically ex-
ploits a layer of automated testing thats
more extensive, reliable, and efficient
than the tests that run on the output of
a larger application. This drives down
the time and cost of testing.
All of these techniques add up to
the practice of continuous delivery, as
practiced by SaaS and online service
companies. Systems are chopped up
into an array of microservices. Each microservice is
assigned to a development team, which monitors it,
fixes it, and releases improvements whenever theyre
readywhether once a month, once a week, once a
day, or even more often. They maintain their APIs
and feed their services into a continuous integration
system to make sure that the whole system works
correctly. This continuous process is more adaptable
and easier to manage than the older Scrum-style ag-
ile development with its two-week cadence.
Youll benefit from a microservices architecture
if you run continuous delivery of online services, if
you do a lot of work merging code, or if you have
long test cycles or high test expenses.
ComponentSize
Bigger components (monolithic applications and
macroservices) are easier to operate and have less
code per feature because they have less interprocess
communication. Smaller microservices are easier
to build, test, mix and match, configure, and de-
ploy. You can therefore have more frequent releases,
smaller development teams, and faster onboarding
of new developers and tech leads.
Deciding how big to make a component is more
art than science. Almost any service architecture
will show a tremendous variation in service sizes,
from a 10-line script to process an event or post a
Web message, to a database server built from mil-
lions of lines of code. Not all services in a microser-
vices architecture are necessarily micro. A service
will become as big as it needs to be to provide a co-
herent, efficient, reliable function.
Build VersusBuy
Cloud vendors package their products into Web
services (for example, storage, database, and mes-
sage queues). Its easy to use these services if you
already have a microservices architecture. SaaS
vendors also make their functions available through
API calls that are easy to insert into a microservices
architecture. So, a microservices architecture will
have a higher percentage of buy versus build. This
can result in a very large reduction in build and
maintenance costs, which may create a tradeoff in
operational costs. For example, rather than incur-
ring a capital expenditure to build out compute,
storage, and network infrastructure, one can pay
for microservices on each invocation, an obvious
cost, and for data transferred to or from a cloud, a
potentially hidden cost. In a more traditional archi-
tecture, a computer systems owner is responsible
for maintenance and upgrades. In the API economy
used by microservices, its possible to buy services
for any layer in the stack, from compute and storage
up to complete SaaS functions. Vendors centrally
manage and continuously upgrade these services.
Integration
Do you have special integration needs? Microser-
vices architectures increase adaptability because
you can swap out components and include compo-
nents from different languages and platforms. You
can have different languages and platforms on dif-
ferent sides of an API call. This is important if you
do things like merger integration, where youre com-
Se p t e m b e r / Oc t o b e r 2 0 1 6 I EEE Clo u d Co m p u t i n g
bining unrelated software platforms. Its also impor-
tant when youre migrating between two platforms.
For example, if youre migrating from an old legacy
application to a new architecture, you can start with
the old architecture by inserting microservice com-
ponents to do things such as processing events. As
you switch to the new platform, you can use the old
application as a service.
One Product or ManyProducts
Microservices are very useful when youre support-
ing multiple products or developing new products
and services. You can reuse services in more than
one product. For example, most online service com-
panies use many shared components (for example,
login and authentication) for all of their products.
You might need to add only a thin layer of new ca-
pabilities and product management to launch a new
product or invade a new market. Or, you can strip
away product capabilities and sell microservices
directly. Thats what Google and Amazon do when
they sell you a cloud database instead of a search
or e-commerce app. Think of it this way: if 100 de-
velopers work for a year on a monolithic product,
theyll end up with a single product. If they divide
into 10 teams of 10 people each to develop 10 mi-
croservices, those services can theoretically be com-
bined into 210 that is, more than 1,000 different
products.
Scale of Production Services and Cost of Load
When you divide your application into microservices,
you gain more ways to increase its capacity. You can
add more instances of any microservice that comes
under heavy load. You can increase capacity more
easily and cheaply than with a monolithic system,
because youre only adding the components that are
heavily loaded, and not a whole complex app.
Your base load also declines, so you might get
more advantages from moving to an on-demand
cloud vendor. In an extreme case, you might be able
to take advantage of serverless architectures, such
as what Amazon Web Services calls Lambda func-
tions (https://aws.amazon.com/lambda), Google calls
cloud functions, (https://cloud.google.com/functions),
and Microsoft calls Azure functions (https://azure
.microsoft.com/en-us/services/functions), in which
the cloud vendor runs small scripts to process your
19
 Cloud Economics
incoming events and uses compute resources only
when such events or API calls arrive. This has ad-
vantages of simplicity and efficiency, and it can re-
duce compute costs by 90 percent or more.
DataSize, Consistency, and Complexity
When your database is small, you can provide it
to all parts of a big application, and a monolithic
architecture is practical. In the 1980s, data archi-
tects learned to normalize databases, so that each
piece of data was in one place, and they were sure
that the data was correct and not contradicted else-
where. This seems important for data like a bank
account balance. However, this central data store
propagates problems when you change the data
structure, and you need to test all of the code that
uses the data. And, as data volumes increase, it be-
comes difficult or even impossible to put the chang-
es in one place. So, the central shared database has
become outdated.
A monolithic architecture is often based on a re-
liable transaction approach typically referred to as
ACIDatomic (all or nothing), consistent, isolated
(transactions lead to identical outcomes whether ex-
ecuted in parallel or serially), and durable (perma-
nent, even in the face of a disaster). It has proven
to be impossible to run ACID systems at very large
scale. Large systems must live in a BASE world of
basic availability, soft state, and eventual consisten-
cy.5 In the BASE world, we often get data through
API calls to microservices.6
When the database is very big, or data structure
changes frequently, or data flows at high volume,
youll want a service architecture in whichdata is en-
capsulated into services, and other parts of the app
get it with API calls. In addition, the structure of the
API calls should be stable and backward compatible,
even when the underlying data or schema changes.
The architecture should also include an expandable
number of data handling nodes. Finally, the service
architecture should allow data to be replicated in an
eventually consistent way to all nodes.
A microservices architecture imposes
costs and complexity. However, it solves
some expensive problems faced by the developers of
complex systems. You should consider using a cloud-
20 I EEE Clo u d Co m p u t i n g w w w.co m p u t er .o rg /clo u d co m p u t i n g
based microservices architecture if youre dealing
with any of the following types of complexity:
large software systems with large numbers of
developers or long and expensive test cycles,
a competitive environment that requires the
rapid upgrading and release of online systems or
business services,
multiple software-based products or online
services,
migration from building and maintaining sys-
tems to buying more components that will be
continuously upgraded by vendors,
integration with systems on different platforms,
high volume of usage on cloud-based platforms,
or
large flow of data, or rapidly changing data
structures.
Your services will become cells in the API economy
of the cloud, the largest and most powerful comput-
er system ever conceived.
References
1. C. Jones, Software Assessments, Benchmarks, and
Best Practices, Addison-Wesley, 2000.
2. F.P. Brooks, Jr., The Mythical Man-Month: Es-
says on Software Engineering, Addison-Wesley,
1975.
3. R. Potvin, The Motivation for a Monolithic
Codebase: Why Google Stores Billions of Lines
of Code in a Single Repository, presentation,
@Scale conference, 2015; www.youtube.com/
watch?v=W71BTkUbdqE.
4. J. Weinman, Cloudonomics: The Business Value
of Cloud Computing, John Wiley & Sons, 2012.
5. D. Pritchett, BASE: An ACID Alternative,
ACM Queue, vol. 6, no. 3, 2008, pp. 4855.
6. E.A. Brewer, Towards Robust Distributed Sys-
tems, Principles of Distributed Computing,
2000; https://people.eecs.berkeley.edu/~brewer/
cs262b-2004/PODC-keynote.pdf.
Andy Singleton is the founder of MAXOS.ai,
Assembla, and PowerSteering Software. He writes
about continuous delivery, microservices, big systems,
and big companies. Contact him atandy@singleton.ai.
 IEEE Cloud Computing Call for Papers
Special Issue on Cloud-
Native Applications
Submission deadline: 1 March 2017 Publication date: September/October 2017
I
EEE Cloud Computing magazine seeks accessible, useful
papers for a special issue on Cloud-Native Applications
and Architecture. Many applications in enterprises are
not able to leverage the advantages of cloud computing
without a great deal of refactoring a process that is costly,
time consuming and often producing disappointing results.
However, over the last five years we have seen cloud
software architectures evolve that promote the design of
applications that, from conception to deployment, are
envisioned, prototyped and built with cloud tools and
cloud resources. These cloud-native applications are born
and run in the cloud and follow new classes of design and
maintenance patterns.
The purpose of the special issue is to urge the research
community to better define and document the cloud-native
movement. Topics of interest include but are not limited to:
Frameworks to make it easier for industry to build cloud-
native applications;
Educational approaches and community based
organizations that can promote cloud-native design
concepts;
The tooling to develop cloud-native applications;
The role of open source for building cloud-native
applications;
VM and container orchestration systems for managing
cloud-native designs;
Cloud-native applications running in hybrid cloud or
migrated from one cloud to another;
Efficient mechanisms to make legacy applications
cloud-native;
www.computer.org/cloudcomputing
Comparing applications one cloud-native and the
other not in terms of performance, security, reliability,
maintainability, scalability, etc.;
Cloud-native applications for various industry sectors
(engineering, financial, scientific, health);
Cloud-native operating systems and databases; and
New models for capacity planning and pricing inspired by
cloud-native architecture paradigms.
Special Issue Guest Editors
Roger Barga, Amazon AWS
Dennis Gannon, Indiana University
Neel Sundaresan, Microsoft Corporation
Submission Information
Submissions should be 3,000 to 5,000 words long, with a
maximum of 15 references, and should follow the magazines
guidelines on style and presentation (see https://www
.computer.org/web/peer-review/magazines for full author
guidelines). All submissions will be subject to single-blind,
anonymous review in accordance with normal practice for
scientific publications. For more information, contact the guest
editors at cc5-2017@computer.org.
Authors should not assume that the audience will have
specialized experience in a particular subfield. All accepted
articles will be edited according to the IEEE Computer Society
style guide (www.computer.org/web/publications/styleguide).
Submit your papers through Manuscript Central at https://
mc.manuscriptcentral.com/ccm-cs.Guest Editors
Guest Editors Introduction

Cloud Security
Peter Mueller, IBM Zurich Research Laboratory
Chin-Tser Huang, University of South Carolina
Shui Yu, Deakin University
Zahir Tari, RMIT University
Ying-Dar Lin, National Chiao Tung University

he cloud is becoming a major computing environment. Many

critical applications are being migrated to cloud platforms.
These include medical and finance applications, big data
applications, and applications with real-time constraints.
Gartner predicts that the bulk of future IT infrastructure
spending will be on cloud platforms and applications, and
nearly half of all large enterprises are planning cloud deployments by the end
of 2017.
However, cloud computing systems and services have become major tar-
gets for cyberattackers. Because the cloud infrastructure is to a certain de-
gree an open and shared platform, its subject to malicious attacks from both
insiders and outsiders. The high degree of virtualization in cloud systems
also makes them ideal subjects for side-channel attacks. Identity hijacking
and distribution of malicious code have become critical issues in cloud sys-
tems. Thus, organizations must carefully plan, deploy, and maintain cen-
tralized management of security properties and effectively enforce security
policies in cloud environments.
To provide strong protection of cloud platforms, infrastructure, hosted
applications, and data stored in the cloud, we need to address the security
issue from a range of perspectives. These include secure data and applica-
tion outsourcing, information leakage protection, information retrieval on
encrypted data, anonymous communication, vulnerability discovery, attack
handling, homomorphic encryption, and secure multiparty computation.
To achieve the high level of cloud security management required, we need
comprehensive vulnerability analyses and innovative security technologies in
both theory and practice.

22 I E E E C l o u d C o m p u t i n g p u b l i s h e d b y t h e I E E E c o m p u t e r s o cie t y  2325-6095/16/$33.00 2016 IEEE

 The Articles
By organizing this special issue on cloud security,
it was our intention and hope to emphasize and ad-
dress the importance of protecting and securing
cloud platforms, infrastructures, hosted applica-
tions, and data storage.
We received 42 submissions in response to our
call for papers. After two rounds of rigorous review,
we selected five high-quality papers for publication
in this special issue.
In Online Analysis of Security Risks in Elastic
Cloud Applications, Athanasios Naskos, Anasta-
sios Gounaris, Haralambos Mouratidis, and Pan-
agiotis Katsaros address security-related concerns
in elastic cloud applications stemming from the
inherent tradeoffs between security and other non-
functional requirements, such as performance. To
this end, the authors propose a solution that can
be efficiently realized by modeling the application
behavior as a Markov decision process, on top of
which they apply probabilistic model checking. The
authors show how their approach can be used to
perform online analysis and elastic decision making,
and how its runtime analysis can provide evidence
for key security-related aspects of the running appli-
cations, such as determining the probability of data
leakage in the next hour.
In Privacy-Preserving Access to Big Data in
the Cloud, Peng Li, Song Guo, Toshiaki Miyazaki,
Miao Xie, Jiankun Hu, and Weihua Zhuang focus on
the security and privacy concerns regarding third-
party cloud storage service providers, which cause
many users and companies to hesitate to move their
data to cloud storage. The authors provide a tuto-
rial and survey of oblivious RAM (ORAM), a solu-
tion designed to enable privacy-preserving access
to data stored in the cloud. Moreover, the authors
study the access load balancing problem when ap-
plying ORAM for big data in the cloud, and propose
heuristic algorithms to achieve access load balanc-
ing in both static and dynamic deployments.
In Cryptographic Public Verification of Data
Integrity for Cloud Storage Systems, Yuan Zhang,
Chunxiang Xu, Hongwei Li, and Xiaohui Liang also
deal with the security of cloud storage services but
from a different angle: the verification of data in-
tegrity. Many public verification schemes employ
a third-party auditor to verify the integrity of data
Se p t e m b e r / Oc t o b e r 2 0 1 6 I EEE Clo u d Co m p u t i n g
outsourced to cloud storage services, but theyre ef-
fective only if a strong assumption holds: that the
auditors are honest and reliable. Moreover, many
such schemes are vulnerable to an active external
adversary, who can modify the outsourced data and
tamper with the interaction messages between the
cloud server and the auditor. The authors propose
an efficient and secure public verification scheme
that uses a random masking technique to protect
against external adversaries, and requires users to
audit auditors behaviors to prevent malicious audi-
tors from forging verification results. The authors
also use Bitcoin to construct unbiased challenge
messages to thwart collusion between malicious au-
ditors and cloud servers.
In To Docker or Not to Docker: A Security
Perspective, Theo Combe, Antony Martin, and
Roberto Di Pietro address the security of cloud-
based infrastructures. They first provide a com-
prehensive overview of the container ecosystem,
which was designed for shortening development cy-
cles, providing continuous delivery, and achieving
cost savings in cloud-based infrastructures. The re-
mainder of the article is dedicated to the introduc-
tion of Docker, which is both a leading container
solution and a complete packaging and software
delivery tool. Using realistic use cases, the authors
discuss the security implications of the Docker
environment. Moreover, they define an adversary
model, point out several vulnerabilities affecting
current Docker uses, and discuss future research
directions for Docker.
In User-Centric Security and Dependability in
the Clouds of Clouds, Marc Lacoste, Markus Miet-
tinen, Nuno Neves, Fernando M.V. Ramos, Marko
Vukolic, Fabien Charmet, Reda Yaich, Krzysztof
Oborzy nski,
Gitesh Vernekar, and Paulo Sousa con-
sider the issue of lack of interoperability in a dis-
tributed environment of multiple clouds, and point
out that the complexity of management could raise
many security and dependability concerns. The au-
thors introduce secure Supercloud computing as a
new paradigm for security and dependability man-
agement of distributed clouds. Supercloud follows
a user-centric and self-managed approach to avoid
technology and vendor lock-ins. In this system, us-
ers can define U-clouds, which are isolated sets of
computation, data, and networking services run over
23
GUEST EDITORS INTRODUCTION
both private and public clouds operated by multiple
providers, with customized security requirements as
well as self-management for reducing administration
complexity. The authors present the Supercloud se-
curity architecture along with several use cases to
illustrate its practical applicability.
e hope you enjoy reading these five articles
and expect that the publication of this spe-
cial issue will both increase public awareness of the
significance of cloud security and inspire further in-
vestigation on the development and enhancement of
state-of-the-art cloud security solutions.
PETER MUELLER is a research staff member at IBM
Research. His research interests include datacenter
storage security and reliability and high-frequency
technology. Hes a senior member of IEEE and a
member of the Society for Industrial and Applied
Mathematics, the Electrochemical Society, and the
Swiss Physical Society. Contact him at pmu@zurich
.ibm.com.
CONFERENCES
in the Palm of Your Hand
Let your attendees have:
conference schedule
conference information
paper listings
and more
The conference program mobile app
works for Android devices, iPhone,
iPad, and the Kindle Fire.
For more information please contact
Conference Publishing Services (CPS) at
cps@computer.org
24 I EEE Clo u d Co m p u t I n g
CHIN-TSER HUANG is an associate professor of
computer science and engineering at the University of
South Carolina, where hes the director of the Secure
Protocol Implementation and Development Labora-
tory. His research interests include network security,
network protocol design and verification, and distrib-
uted systems. Huang has a PhD in computer science
from the University of Texas at Austin. Hes a senior
member of IEEE and ACM, and a member of Sigma
Xi and Upsilon Pi Epsilon. Contact him at huangct
@cse.sc.edu.
SHUI YU is a senior lecturer in the School of Infor-
mation Technology at Deakin University, Australia.
His research interests include networking theory, net-
work security, privacy and forensics, and mathemati-
cal modeling. Hes a senior member of IEEE. Contact
him at shui.yu@deakin.edu.au.
ZAHIR TARI is a full professor of distributed systems
at RMIT University, Australia. His research interests
include system performance (for example, webserv-
ers, peer to peer, and cloud computing) and system
security (such as SCADA systems and the cloud). Tari
has a PhD in computer science from the University
of Grenoble, France. Contact him at zahir.tari@rmit
.edu.au.
YING-DAR LIN is a distinguished professor of com-
puter science at National Chiao Tung University,
Taiwan. His research interests include design, analy-
sis, implementation, and benchmarking of network
protocols and algorithms, quality of services, network
security, deep packet inspection, wireless communi-
cations, embedded hardware/software co-design, and
software-defined networking. Lin has a PhD in com-
puter science from the University of California, Los
Angeles. Contact him at ydlin@cs.nctu.edu.tw.
Read your subscriptions through
the myCS publications portal at
http://mycs.computer.org.
W W W.Co m p u t Er .o rg /Clo u d Co m p u t I n g
 IEEE Cloud Computing Call for Papers
Intelligence in
the Cloud
Submission deadline: 1 May 2017 Publication date: November/December 2017
A
rtificial intelligence (AI), since its birth in 1950s, has
been heralded as the key to our civilizations brightest
future. To pursue the vision of AI, various machine
learning approaches (for example, deep learning, supervised
learning, unsupervised learning, reinforcement learning, and
so on) have been proposed and a few have actually been
developed and deployed in the market. The recent hype
around big data has enthusiastically renewed the call and
focus for advanced machine learning technologies to extract
knowledge from large data pools. With its rich resource
provisioning, cloud computing is widely regarded as an ideal
platform to facilitate resource-intensive machine learning so as
to enable intelligence in the cloud. Integrating intelligence into
the cloud is without doubt a promising development trend to
both cloud computing and AI.
We are still at the early stage of integrating intelligence into
the cloud. Toward this exciting future, the path still entangles
many critical challenges in different aspects.
At the application layer, cloud-based efficient and powerful AI
techniques are highly in demand that target various applications
such as natural language processing, stock analysis, medical
diagnosis, intelligent industry control, intelligent transportation,
and scientific discovery.
At the platform layer, while intelligence has been deployed
(for example, Sparks scalable machine learning MLlib and
Googles cloud machine-learning framework TensorFlow)
new machine learning engines are expected for emerging
computing frameworks (for example, the dataflow computing
model HAMR).
At the infrastructure layer, new cloud computing architecture
and resource scheduling strategies are required to support
computation-intensive and IO-intensive machine learning
algorithms. How to configure cloud computation, storage, and
networking resources for fast, efficient, and scalable machine
learning must still be addressed.
The goal of this special is to seek original articles examining
the state of the art, open research challenges, new solutions,
www.computer.org/cloudcomputing
and applications for intelligence in the cloud with special
focus on, but not limited to, the following topics:
new distributed architecture for machine learning;
new machine learning engines in the cloud;
analytics architectures, frameworks, and models for
complex intelligent systems;
intelligent cloud applications or services such as intelligent
traffic, intelligent buildings, intelligent environments,
intelligent businesses, and so on;
cloud resource allocation and optimization through
machine-learning algorithms;
machine learning for cloud resource management;
combining human and machine intelligence in the cloud; and
security and privacy issues for intelligent systems in the cloud.
Special Issue Guest Editors
Song Guo, The Hong Kong Polytechnic University,
Hong Kong
Victor Leung, University of British Columbia, Canada
Xin Yao, University of Birmingham, UK
Submission Information
Submissions should be 3,000 to 5,000 words long, with a
maximum of 15 references, and should follow the magazines
guidelines on style and presentation (see https://www
.computer.org/web/peer-review/magazines for full author
guidelines). All submissions will be subject to single-blind,
anonymous review in accordance with normal practice for
scientific publications. For more information, contact the
guest editors at ccm6-2017@computer.org.
Authors should not assume that the audience will have
specialized experience in a particular subfield. All accepted
articles will be edited according to the IEEE Computer Society
style guide (www.computer.org/web/publications/styleguide).
Submit your papers through Manuscript Central at https://
mc.manuscriptcentral.com/ccm-cs.
Cloud Security

Online Analysis
of Security Risks
in Elastic Cloud
Applications
Athanasios Naskos and Anastasios Gounaris, Aristotle University
of Thessaloniki
Haralambos Mouratidis, University of Brighton
Panagiotis Katsaros, Aristotle University of Thessaloniki

To address security- wo properties of cloud computing have made it the

main option for the deployment of Web applications
related concerns for a wide range of stakeholders, including large com-
panies, small and medium enterprises, and research
in elastic cloud institutes. First, cloud computing allows computational
applications, such resources to be released on demand, giving applica-
tion providers the option to charge in a pay-as-you-go manner. Cloud
as data loss and computing solutions therefore minimize users up-front investments
in equipment and human resources, which can then be allocated to
leakage, the authors application deployment and management. Second, providers release
resources in response to workload changes. This property, known as
propose modeling autoscaling, impacts the monetary cost of running a cloud application:
application behavior resources are used only when required.
Elastic applications can exploit these properties to provide com-
as a Markov putational resources on the fly according to current needs. Computa-
tional resources are typically provided in the form of virtual machines
decision process (VMs). Elasticity is manifested in three main forms. Horizontal scal-
ing, where either new VMs are added or existing ones are removed,
and applying provides the biggest potential for scalability and performance improve-
probabilistic model ments, because of the perceived unlimited number of VMs that can
be provided. In vertical scaling, certain properties (for example, the
checking. number of cores) of the existing VMs are modified. In the third form,

26 I E E E C l o u d C o m p u t i n g p u b l i s h e d b y t h e I E E E c o m p u t e r s o cie t y  2325-6095/16/$33.00 2016 IEEE

live migration, a VM is moved to a different physical
host while staying operational.
We focus on horizontal scaling and argue that
the goals of meeting performance requirements with
the help of horizontal scaling contradict those of se-
curity, thus calling for a risk-based security solution.
Security Concerns and Horizontal Scaling in
Public Clouds
Public and hybrid clouds, unlike private clouds, of-
fer resources to arbitrary customers, or tenants.
Tenants control neither the clouds security policy
nor the types of other tenants whose VMs are col-
located on the same physical machines. Although
this doesnt necessarily imply that public clouds are
insecure, many organizations cite it as a reason for
hesitating to migrate applications to the cloud.1
A 2013 Cloud Security Alliance report identified
data breaches due to malicious cotenants as the top
cloud-related security threat.2 These data breaches
can lead to both data leakagethe unauthorized
disclosure of data from one user to anotherand
data lossa condition where data is destroyed and
becomes unavailable. In addition, in a multitenant
environment, a lack of authorization mechanisms
for sharing physical resources increases the risk
of threats such as service traffic hijacking, which
occurs when attackers hijack cloud accounts by
stealing security credentials and eavesdropping on
activities and transactions, and side-channel at-
tacks, which use information obtained from band-
width monitoring or other similar techniques.
Moreover, when multiple tenants share an under-
lying infrastructure, the risk of threats related to
misconfiguration and uncoordinated change control
increases, allowing a malicious tenant to gain access
to another tenants resources.
Keeping the number of VMs as low as possible
is an indirect way to mitigate data leakage- and loss-
related security concerns but might entail an unac-
ceptable compromise on performance. Performance
is one of the top three most studied service-level
agreement (SLA) parameters, since critical applica-
tions require responses in a fixed, short time period.3
Therefore, a reliable cloud application must both ad-
dress security concerns and honor SLAs.
In the example in Figure 1, an elastic NoSQL
database serves user requests according to the Ya-
Se p t e m b e r / Oc t o b e r 2 0 1 6 I EEE Clo u d Co m p u t i n g
Latency distribution for different cluster sizes
(14,000 requests/second)
90
Response latency (msecs)
80
70
60
50
40
30
8 9 10 11 12 13 14 15 16 17 18
No. of VMs
Figure 1. Execution plans based on different cost
metrics. Increasing the number of virtual machines
(VMs) leads, on average, to lower response times,
subject to increased probability of a malicious tenant
being collocated.
hoo Cloud Serving Benchmark (YCSB).4 The figure
refers to a fixed rate of user requests and shows how
the average and standard deviation values of the re-
sponse latency vary with the number of VMs used.
Thus, a strict threshold on latency would force the
system to acquire additional VMs, the exact quan-
tity of which needs to be computed at runtime ac-
cording to the current workload and considering the
systems volatility. However, increasing the number
of VMs, and assuming that each VM runs on a dif-
ferent physical machine in the generic case, also
increases the probability of a malicious tenant be-
ing collocated. This probability might vary accord-
ing to the cloud provider type,5 but the important
issue is that it is not negligible. Choosing a single
trustworthy provider isnt sufficient either, given
that providers might offer VMs from other provid-
ers as well in periods of very high demand.6 Overall,
as noted earlier, adding VMs poses a threat of data
leakage and data loss. The magnitude of this threat
depends on the application. In the figure, the threat
of data loss is lower than in typical applications, be-
cause NoSQL databases are replicated at least two
or three times, making it harder for a malicious user
to destroy all copies.
27
Cloud Security
State 1 t t+1
$ $
v1
w1 m1
NO_OP
REMOVE
x1% y1% ADD
k1%
z1% VM type Data leakage
Deployment
$ cost Data loss
Figure 2. A conceptual model of an elastic application that considers both security and performance service-level agreement
(SLA) requirements. Each state represents a specific configuration of the application in terms of number of VMs, along with its
security- and performance-related properties, at a given point of time. The transition between states is due to elasticity actions.
Because of the inherent trade-offs among se-
curity and performance requirements, any solution
to analyze and enforce security-aware horizontal
scaling for cloud applications must be risk based.
It should account for the probability of data leak-
age and data loss; dynamic evolution of the external
environment and volatility of system behavior; and
potential heterogeneity of the cloud infrastructure.
We advocate the use of a formal verification ap-
proach as a means to apply mathematical reasoning
for providing security-oriented probabilistic guar-
antees for elastic cloud applicationsspecifically,
probabilistic model checking7 on top of system mod-
els in the form of Markov decision processes (MDPs),
which are instantiated on the fly. Our technique can
analyze and provide evidence for key security-related
aspects of the running applicationsfor example, to
answer questions such as, What is the probability
that there will be a data leakage in the next hour?
Moreover, it can drive elasticity decisions, taking
into account security constraints (for example, given
the current query load and the prediction for this
load in the next half hour) to decide how many VMs
to add or remove.
Using probabilistic model checking to analyze
and drive elasticity decisions has shown promising
initial results.4 Its potential in addressing security
requirements has also been demonstrated.8 Most
of the work in cloud security focuses on identifying
risks, vulnerabilities, security mechanisms, digital
signatures, access control, and manners to attain
security assurance, such as monitoring, certificates,
and auditability.9 However, the detailed investiga-
tion of security assurance during horizontal scaling,
and even more, the security-aware elasticity decision
making that we hereby enable is novel in the fields
28 I EEE Clo u d Co m p u t i n g w w w.co m p u t er .o rg /clo u d co m p u t i n g
State 2
v2
w2 m2
x2% y2%
Latency + 5 minutes k2%
violation z2%
Normal
state Added cost
of both cloud security 9,10 and dynamic resource al-
location in clouds.11,12
Modeling Elastic Applications
We advocate a model-based approach. Figure 2
shows a conceptual model of an elastic application
that considers both security and performance SLA
requirements and is deployed on a public cloud.
Each curved rectangle represents a conceptual state
at a specific time instant. We model the elastic ap-
plications evolution as a transition to a state at a
future point in time t + t through elastic actions,
such as adding or removing VMs.
For each state, we capture the features of inter-
est for the analysis and decision making:
the mixture of VM types employed, which we
assume are of two different types and/or provid-
ers, v and w;
the total deployment cost, m;
the probability of data leakage, x;
the probability of data loss, y;
the probability of performance-related SLA con-
dition violations, z; and
the probability of no security threats or perfor-
mance violations, k.
We consider all these probabilities statistically
independent, so k = (1 x)(1 y)(1 z). Further, we
can safely regard the probability of data leakage on a
single machine of type A, dlA, as independent of the
number of VMs of type A employed. So, x = 1 (1
dlv)v(1 dlw)w. Similarly, we can define y as a func-
tion of the number and type of VMs employed.
The systems evolution due to elasticity actions
refers to discrete time intervals of period t. We con-
 p0
State 1 p1
$
v1
p2
w1 m1
p3
x1% y1% p4
p5
z1% k1%
p6
p7
Figure 3. Mapping of a conceptual state to a set of Markov decision process (MDP) states. Each MDP state
corresponds to a distinct behavior type.
sider three actions: add, remove, and no_change.
In the generic case, the effects of actions at time
t might be delayed and not manifested at the next
period but after multiple time points. For example,
adding a new VM to serve a NoSQL database im-
plies that a new VM needs to be created, booted,
and configured, and it needs to receive data, which
might take more than t time. During this process,
the system should be in a transient state.
MDP Implementation of the Conceptual Model
We implement our technique using the MDP mod-
eling approach. We chose MDP because it enables
analysis and decision making and can capture non-
determinism and uncertainty in a given system.13
Both properties are essential in an elastic cloud ap-
plication. Because of horizontal scaling, at each time
point, the number of VMs can increase, remain the
same, or decrease. This gives rise to nondeterminism.
Also, at any given point, there might or might not be
a performance or security-related requirement viola-
tion. This necessitates the modeling of uncertainty.
MDPs are specified using states, actions, prob-
abilities, and rewards. The states represent system
snapshots at specific time points, which are charac-
terized by a set of system properties. The actions are
transitions between the states, which express some
change to the state properties. The probabilities re-
fer to each triple (state s)-(action a)-(state s) and rep-
resent the probabilities of transition from one state
to another due to a specific action thus quantifying
Se p t e m b e r / Oc t o b e r 2 0 1 6 I EEE Clo u d Co m p u t i n g
Data leakage
No data leakage
Performance violation
No performance violation
Data loss
No data loss
t
VM type (t)
$ Deployment cost
uncertainty. Finally, the rewards are used to perform
quantitative analysis (or solution) of MDP models.
The conceptual model needs to be implemented
as an MDP according to the analysis requirements.
Because we need to explicitly consider different
types of application behavior during analysis and
elasticity decision making, we map each conceptual
system state to multiple MDP model states, one for
each behavior type. The behavior type is defined
according to the application nonfunctional require-
ments. Let an application set three requirements:
to avoid data leakage, data loss, and latency above
a user-specified threshold. Then, each combination
of a binary variable that indicates the satisfaction of
each requirement defines a behavior type (see Fig-
ure 3). Furthermore, each MDP state is annotated
as to whether or not it refers to a transient state (not
shown in the figure).
The actions are the same as in the conceptual
model. The only difference is that, if the MDP state
is transient, only no_change is allowed, because
making further resizing decisions during instable
periods is prone to suboptimal decision making. The
next step is to define the transition probabilities.
Figure 4 gives a complementary view of Figure 3,
where each path from s to s corresponds to a MDP
model state. The probabilities p0 to p7 in Figure
3 are the product of the probabilities in Figure 4
along the corresponding path. For example, p7 = (1
x)(1 y)(1 z). In general, for a given initial state
s and action a,
29
Cloud Security
Data leakage
.LOGS
Logs Prediction No data leakage
Initial state Data loss check
Elastic y y
action
S S
1-y
v v
:vs :ws $:ms
Figure 4. Setting MDP model probabilities for state transitions based on past log entries referring to the same
mixture of VM types and prediction of future external load.
p ( s, a, s) = 1 ,
s
and multiple actions can be plausible for each state.
MDP Instantiation
To serve online analysis (which we discuss later in
detail), the model is instantiated on the fly. To this
end, the decision depth and the probabilities take
actual values. Decision depth refers to how many pe-
riods the model can account for. If the depth is too
small, the system becomes too short-sighted; if its
too big, the prediction uncertainty increases. Both
situations lead to suboptimal decisions. Clearly, the
number of MDP states grows exponentially in the
number of t periods. If a period represents 5 min-
utes of real time, a model with depth set to 4 refers
to a scenario where elasticity is reassessed every 5
minutes, and the model looks ahead for 20 minutes.
If the system evolves less rapidly, the application
manager could map each period to a longer time. In
general, setting t appropriately heavily relies on the
application environments volatility (for example, if
during night hours the workload remains roughly
stable, t can be increased).
In our approach, we derive the probabilities x,
y, and z through logs. We analyze past log entries
referring to the same mixture of VM types as the
state of interest to estimate the probabilities. For
performance-related metrics, such as latency, we
consider not only the number of VMs but also the
external load of incoming requests. This implies the
need to add a load prediction component, as Figure 4
shows. In general, for our approach to be applicable,
30 I EEE Clo u d Co m p u t i n g w w w.co m p u t er .o rg /clo u d co m p u t i n g
Performance t
Data loss VM type (t)
violation
No performance Deployment
violation No data loss $ cost
Performance
Data leakage check Next state
violation check
z
1-y 1-z
v
:vs
v
:ws $:ms
we assume that a security analysis and profiling
mechanism is in place, which is capable of deriving
attack probabilities as a function of the cluster
configuration. Such a mechanism is orthogonal to
our approach and can be even more sophisticated.
For example, upon instantiation, it could account for
whether any VM additions would involve the use of a
new physical machine rather than deploying VMs on
one thats already in use. Finally, since the models
are instantiated on the fly, the attack probabilities
can be dynamically refined.
Online Analysis and Decision Making
The analysis is based on verification of models in-
stantiated on demand. To this end, we couple the
probabilistic model with Probabilistic Computation
Tree Logic (PCTL), a probabilistic property speci-
fication language thats fed to the parallel reduced-
instruction-set multiprocessing (Prism) model
checker.14 We also show how the analysis can direct-
ly support decision making with regard to elasticity.
Prism can efficiently analyze complex models.
We have been able to solve MDP models with
9,958 states corresponding to four periods in 0.073
seconds using a machine with a quad-core CPU and
8 Gbytes of RAM, while the program is reported to
have processed models up to 1011 states on a single
machine.14
Examples of Verified Analysis Using Prism
Figure 5 gives a concise view of the analysis of two
PCTL properties. For simplicity, we grouped the
eight states of Figure 3 in two groups according to
the data leakage property. Further, we assume that
the decision depth is set to 2 and the probability of P=? [G !data_leakage] P=? [F data_leakage & steps=max_steps]
data leakage in a time interval x is 0.02. Also, in this P=0.98*0.98=0.9604 P=0.02*0.02+0.98*0.02=0.02
toy example, we allow only one action (for example,
no_change)that is, theres no nondeterminism. 0.02
? ?
Obviously, in a more complete example with nonde- ? ?
0.02
terminism including all elasticity actions, multiple
0.98
such paths are eligible.
The first PCTL property (in green) answers the 0.02
question What is the probability of having no data 0.98
leakage incident? The corresponding PCTL is ex- ? ?
? ?
pressed as P = ? [G !data_leakage], and is satisfied 0.98

by the green path in the model, while the returned t t + t1 t + t2

probability is 0.9604.
The second PCTL (in red) answers the ques-
tion What is the probability of eventually (that is, in
the final state in the verification) having data leak-
age? Its expressed as P = ? [F data_leakage & steps
= max_steps] and returns the cumulative probability
of the red transitions, which is 0.02.
Table 1 presents more complex examples of risk-
based security analysis, where any of the three ac-
tions is allowed. Thus, during analysis, we investi-
gate different sequences of actions. Formally, these
sequences are adversaries, policies, and strategies.
The results of each analysis can include multiple
adversaries. The PCTL statements are mostly self-
explanatory. In the statements, U defines which
state needs to follow the state on the left, F stands
for eventually reaching a state, G stands for a condi-
tion that needs to hold throughout the policy, and X
for the next state. We refer to the decision depth as
max_steps.
Figure 5. An example of probabilistic model checking analyzing two
properties of the application with regards to data leakage.
The examples in Table 1 fall into three catego-
ries. The first two rows refer to analyses where the
outcome is a probability of reaching certain model
states. The next two examples return a Boolean
value indicating whether a specific property holds.
The last two PCTL properties are numerical mul-
tiobjective ones, as they ask for the maximum
possible probability of reaching a state under the
condition that the probability of reaching an-
other state is bounded by a given threshold. The
objectives can refer to a single property, like data
loss in the fifth example, or multiple ones, like data
leakage, data loss, and monetary cost in the last
example.

Table 1. Additional examples of analyses enabled.

Analysis goal Probabilistic computation tree logic

(PCTL) property

What is the maximum probability (among all possible adversaries) of experiencing Pmax = ? [data leakage U !data leakage]
a data loss incident until eventually moving to a state with no data loss?

What is the maximum probability (among all possible adversaries) of moving from a Pmax = ? [data loss U F !data loss & steps
state with data leakage to a state with no data leakage? = max_steps]

Starting from any reachable state, is it always possible (that is, is there at least one filter(exists, P >= 1 [F !data leakage & steps
adversary) to eventually reach a state with no data leakage? = max_steps])

Starting from a state with no data loss, do all adversaries eventually reach a state filter(forall, P >= 1 [F data loss & steps =
with no data loss? max_steps], !data loss)

What is the maximum probability of experiencing data loss in a state that multi(P max = ? [X data loss],
immediately follows the initial state, while the probability to end up at a state with no P >= 0.9 [F !data loss & steps = max_steps])
data loss is greater than or equal to 0.9?

What is the maximum probability of having total cost of deployment less than a multi(P max = ? [F total cost <= Budget
specified budget, while the probability of experiencing any security incident does not & steps = max_steps], P <= 0.05 [G data
exceed 0.05? leakage & data loss])

Se p t e m b e r / Oc t o b e r 2 0 1 6 I EEE Clo u d Co m p u t i n g 31
Cloud Security
Adaptation of the no. of VMs to the incoming load variation
14,000
Incoming load (requests/second)
12,000
10,000
8,000
6,000
0 500 1,000 1,500 2,000 2,500
Time steps (30 seconds)
Incoming load No. of VMs (security
No. of VMs (performance) performance)
Figure 6. Example of security-aware elasticity decisions. The number of
VMs follows the incoming load. When security requirements are taken
into consideration, the elasticity actions tend to be more conservative.
Decision Making
MDPs are inherently suited for decision making as
well. To this end, each model state needs to be as-
sociated with a reward value. State rewards are
computed using functions that quantify various as-
pects of the system like performance and security
concerns or more concrete assets like the number
of active VMs or the actual deployment cost. As an
example, consider the following utility function
that uses weights to balance three aspects: the nor-
malized probability of data leakage ( p  dleak ) , data
loss ( p  dloss ) , and the latency exceeding a threshold
( p per f ) :
u ( vmw) = a p
 dleak + b p  per f , a + b + c = 1 .(1)
 dloss + c p
Note that, in general, we can prioritize threats
and objectives. We reflect this on the utility func-
tion by assigning different values to the weights. In
addition, our approach is orthogonal to any user-
defined utility function.
Our decision-making proposal is based on the
computation of the cumulative reward of every ad-
versary. The model solver examines the possible al-
ternativesthat is, all combinations of state transi-
tionsand computes the optimal cumulative state
reward along with the corresponding sequence of
actions. For example, using this utility function, the
optimal reward is the minimum one. In Prism, we
can do this with the help of a different type of PCTL
specification that asks for reward minimization rath-
32 I EEE Clo u d Co m p u t i n g w w w.co m p u t er .o rg /clo u d co m p u t i n g
er than probabilities: R{cumulative reward}min = ?
18 [F steps = max_steps].
Interestingly, several decision policies can be
16 built on top of the aforementioned verification. For
example, if multiple adversaries are returned with
No. of VMs
14 equal reward, we can use a second PCTL on other
aspects, such as the probability of security violation
12 as presented in Table 1, to choose the final strategy.
Moreover, it isnt necessary to perform all ac-
10 tions in that strategy. An elasticity decision-making
technique tailored to NoSQL databases thats pre-
8 sented elsewhere follows the steps mentioned ear-
lier.4 After deciding on the adversary at each time
point the decision mechanism is activated, our pro-
posal enacts only the first elasticity action, and then
reevaluates the whole adversary from scratch. Such
an approach is in line with a wide range of adaptive
solutions, such as model predictive control (MPC),15
which computes a sequence of adaptations but only
applies the first step. According to evaluation re-
sults published elsewhere,4 the quality of elasticity
decisions outperforms other proposals for scaling
NoSQL databases in avoiding both violations of la-
tency thresholds and overprovisioning of VMs.
Figure 6 shows how a security-aware elasticity
decision maker behaves in a setting similar to the
one described in earlier work.4 Although the load
varies (green plot), the decision maker constant-
ly reevaluates the number of VMs. The blue plot
shows the number of VMs when considering only
performance requirements, and the red plot shows
the behavior when the state rewards are computed
based on Equation 1. In the latter case, we limit
the use of VMs to mitigate the threat of data leak-
age and loss.
ur approach is of interest to both owners
of elastic applications and cloud service
providers. The outcomes of our proposal can be
used either to analyze (elastic) behavior or to make
elasticity decisions. Additionally, the analysis results
can be used to fine tune the utility function, acting
as a feedback mechanism, so that decisions are good
in practice.
Analysis and decision making in elastic appli-
cations is, by its nature, an instance of autonomic
computing problems. A key issue for autonomic
solutions is to render them dependable and endow
them with a solid formal basis. To this end, proba-
bilistic model checking not only allows for the con-
tinuous verification of system properties but is also
an effective tool for meeting both security- and
performance-oriented goals.4,8
 Although we focused on horizontal scaling with
data leakage and loss as the most prominent security
threats, our approach can also cover additional se-
curity threats affected by scaling. Moreover, its ap-
plicable to both vertical elasticity and live migration.
To this end, the envisaged models must be more
fine-grained, considering VM configuration types
and physical machines rather than only the number
of VMs. The analysis of all elasticity types in combi-
nation with additional security threats is a challeng-
ing avenue for further research.
References
1. C. Kalloniatis et al., Migrating into the Cloud:
Identifying the Major Security and Privacy
Concerns, Collaborative, Trusted and Privacy-
Aware e/m-Services, Springer, 2013, pp. 7387.
2. Cloud Security Alliance, The Notorious Nine:
Cloud Computing Top Threats in 2013, report,
2013; https://cloudsecurityalliance.org/download/
the-notorious-nine-cloud-computing-top-threats
-in-2013.
3. F. Faniyi and R. Bahsoon, A Systematic Review
of Service-Level Management in the Cloud,
ACM Computing Surveys, vol. 48, no. 3, 2015,
article 43.
4. A. Naskos et al., Dependable Horizontal Scaling
Based on Probabilistic Model Checking, Proc.
15th IEEE/ACM Intl Symp. Cluster, Cloud and
Grid Computing (CCGrid), 2015, pp. 3140.
5. S. Ramgovind, M.M. Eloff, and E. Smith, The
Management of Security in Cloud Computing,
Proc. Information Security for South Africa
(ISSA), 2010, pp. 17.
6. E. Casalicchio, D.A. Menasc, and A. Aldhalaan,
Autonomic Resource Provisioning in Cloud
Systems with Availability Goals, Proc. 2013
ACM Cloud and Autonomic Computing Conf.,
2013, article 1.
7. V. Forejt et al., Automated Verification
Techniques for Probabilistic Systems, Formal
Methods for Eternal Networked Software Systems,
LNCS 6659, 2011, pp. 53113.
8. A. Naskos et al., Security-Aware Elasticity for
NoSQL Databases, Proc. 5th Intl Conf. Model
and Data Eng. (MEDI 15), 2015, pp. 181197.
9. C.A. Ardagna et al., From Security to Assurance
in the Cloud: A Survey, ACM Computing
Surveys, vol. 48, no. 1, 2015, article no. 2.
10. H. Mouratidis et al., A Framework to Support
Selection of Cloud Providers Based on Security
and Privacy Requirements, J. Systems and
Software, vol. 86, no. 9, 2013, pp. 22762293.
Se p t e m b e r / Oc t o b e r 2 0 1 6 I EEE Clo u d Co m p u t i n g
11. Z.A. Mann, Allocation of Virtual Machines in
Cloud Data CentersA Survey of Problem Mod-
els and Optimization Algorithms, ACM Comput-
ing Surveys, vol. 48, no. 1, 2015, article 11.
12. S. Singh and I. Chana, QoS-Aware Autonomic
Resource Management in Cloud Computing: A
Systematic Review, ACM Computing Surveys,
vol. 48, no. 3, 2015, article 42.
13. M.L. Puterman, Markov Decision Processes:
Discrete Stochastic Dynamic Programming, John
Wiley & Sons, 1994.
14. M. Kwiatkowska, G. Norman, and D. Parker,
PRISM: Probabilistic Model Checking for
Performance and Reliability Analysis, ACM
SIGMETRICS Performance Evaluation Rev., vol.
36, no. 4, 2009, pp. 4045.
15. J. Maciejowski, Predictive Control with Con-
straints, Prentice Hall, 2001.
Athanasios Naskos is a PhD candidate in the
Department of Informatics of the Aristotle University
of Thessaloniki, Greece. His research interests include
cloud elasticity, distributed data management, and
model checking. Naskos has an MSc in computer sci-
ence from the Aristotle University of Thessaloniki.
Contact him at anaskos@csd.auth.gr.
Anastasios Gounaris is an assistant profes-
sor in the Department of Informatics of the Aristotle
University of Thessaloniki, Greece. His research inter-
ests include distributed data management, resource
scheduling, autonomic computing, and adaptive que-
ry processing. Gounaris has a PhD in computer sci-
ence from the University of Manchester. Contact him
at gounaria@csd.auth.gr.
Haralambos Mouratidis is professor of soft-
ware systems engineering at the School of Comput-
ing, Engineering, and Mathematics at the University
of Brighton. His research interests include secure soft-
ware systems engineering, requirements engineering,
and information systems development. Mouratidis
has a PhD in computer science from the University
of Sheffield. Contact him at H.Mouratidis@brighton
.ac.uk.
Panagiotis Katsaros is an assistant professor
in the Department of Informatics at the Aristotle Uni-
versity of Thessaloniki, Greece. His research interests
include formal analysis, model checking, and depend-
ability and security. Katsaros has a PhD in computer
science from the Aristotle University of Thessaloniki.
Contact him at katsaros@csd.auth.gr.
33
Cloud Security

Privacy-Preserving
Access to Big Data in
the Cloud
Peng Li, Song Guo, and Toshiaki Miyazaki, University of Aizu
Miao Xie and Jiankun Hu, University of New South Wales at the Australian
Defense Force Academy
Weihua Zhuang, University of Waterloo

Oblivious RAM aims to enable privacy-preserving

access to data stored in the cloud. This article
provides a tutorial about ORAM and proposes
heuristic algorithms to achieve access load balancing
in both static and dynamic deployments.

ig data has emerged in domains such as science, engineering, and com-

merce. Facebook, for example, currently stores more than 20 petabytes of
photos, and this number grows by 60 terabytes each week.1 In the big data
era, clouds become a perfect candidate for data storage by providing virtu-
ally unlimited storage that can be accessed over the Internet. By outsourc-
ing large volumes of data to cloud storage, such as Google Drive, Dropbox,

34 I E E E C l o u d C o m p u t i n g p u b l i s h e d b y t h e I E E E c o m p u t e r s o cie t y  2325-6095/16/$33.00 2016 IEEE

and Amazon Simple Storage Service, users can sim-
plify their data management and reduce data mainte-
nance costs through the pay-as-you-use model.
Due to security and privacy concerns, however,
some users and companies still hesitate to move
their data to the cloud. Although encryption can
protect data confidentiality, its insufficient because
access patterns can also leak important information.
For instance, more than 80 percent of encrypted
email queries can be identified by analyzing user ac-
cess patterns.2
In this article, we present the challenges of pre-
serving privacy in cloud storage. To address these
challenges, we apply oblivious RAM (ORAM), known
to be the most effective solution for hiding user ac-
cess patterns. We provide a tutorial about ORAM and
survey recent efforts to increase the practicability of
using it in cloud storage by reducing its overhead.
Consider distributed file systems built on
hundreds or thousands of servers in a single or
multiple geodistributed cloud sites. Applying an
ORAM-based algorithm for privacy-preserving ac-
cess can lead to serious access load imbalance
among the storage servers. Therefore, in this arti-
cle, we study a data-placement problem to achieve
a load-balanced storage system with improved avail-
ability and responsiveness. Given this problems NP-
hardness, we propose a low-complexity algorithm
that can deal with a large-scale problem with re-
spect to big data. We conduct extensive simulations
to show that the proposed algorithm finds results
close to the optimal solution, and significantly out-
performs a random data-placement algorithm.
Preserving Privacy in Cloud Storage
Cloud storage is often managed and maintained by
cloud storage providers in the form of services, and
it is comprised of logical storage pools that interact
directly with data, physical storage spanning
multiple servers, and the physical environment.
Users obtain storage capacity from providers
and operate on their data through public APIs.
Many well-known providers have begun to offer
cloud storage services in recent years. As the best
infrastructure to accommodate big data, cloud
storage has attracted attention from both industry
and academia. However, various security and
privacy concerns arising from the nature of cloud
Se p t e m b e r / Oc t o b e r 2 0 1 6 I EEE Clo u d Co m p u t i n g
storage are preventing users from subscribing to this
service. These concerns include:
Untrustiness. Users can operate on data re-
motely only through the APIs in cloud sites that
span multiple locations and belong to untrusted
third-party organizations.
Dynamic environment. Users and cloud storage
providers can dynamically change the services
requested or offered, resulting in sensitive data
moving frequently within single or around mul-
tiple organizations and increasing the possibility
of disclosing private information.
Uncensored new services. Usually, cloud storage
providers allow new services to be added with
less control than in traditional multiorganiza-
tional scenarios. A new service might introduce
security and privacy requirements at a different
level and, in some cases, additional risk is im-
posed on users as this service might need to col-
lect and store sensitive information.
The community is dedicated to improving the
security and reliability of cloud storage, and many
researchers have proposed techniques. The redun-
dant array of inexpensive disks (RAID) technique,
for example, is integrated in the high-availability and
integrity layer (HAIL),3 which manages remote file
integrity and availability across a collection of serv-
ers. Alysson Bessani and her colleagues proposed
DepSky, a distributed storage system that integrates
encryption, encoding, and replication.4 IRIS, an au-
thenticated file system for enterprises, stores data
in the cloud with resilience against potentially un-
trusted cloud providers.5 Several proposals deal with
data availability by constructing distributed storage
systems across several cloud sites. SPANStore is a
key-value storage system that exports a unified view
of storage services in geographically distributed
datacenters.6 It minimizes an application providers
cost by exploiting pricing discrepancies across pro-
viders, estimating application workload at the right
granularity, and minimizing the use of computation-
al resources.
Although security and privacy can be guaran-
teed to some extent using the aforementioned tech-
niques, protection of access privacy is still a gap to
be filled in the context of cloud storage. Mohammad
35
Cloud Security
Islam and his colleagues prove that most of the pro-
tocols proposed for privacy-preserving cloud storage
will leak access patterns due to efficiency issues.2
A deliberately designed attack that exploits access
pattern leakage can disclose a significant amount
of sensitive information, such as the identification
of encrypted email queries. The private information
retrieval (PIR) technique addresses the access pri-
vacy problem by letting users retrieve a block from
a database of N items held by a server that learns
nothing about this block.7 Unfortunately, Radu Sion
and Bogdan Carbuna showed that existing PIR tech-
niques will never be more efficient than a trivial
PIR technique of downloading the entire database.8
PIRs extremely poor performance makes it inappli-
cable in cloud storage with big data.
ORAM provides data access privacy by periodi-
cally reshuffling data blocks stored in an untrust-
ed server so user access cant be tracked. Michael
Goodrich and Michael Mitzenmacher proposed
an ORAM algorithm with O(pN) client storage to
achieve O(logN) amortized cost, that is, each oblivi-
ous read or write leads to O(logN) data access op-
erations on average.9 Elaine Shi and her colleagues
further reduced the client storage to O(1).10 Later,
Tarik Moataz and his colleagues proposed replac-
ing homomorphic eviction with a new and much
cheaper permute-and-merge eviction, so the block
size can be reduced to (log4 N) while maintaining
O(1) complexity.11
Jonathan Dautrich and his colleagues combined
PIR techniques with the most bandwidth-efficient
existing ORAM to reduce bandwidth cost.12 Chang
Liu and his colleagues developed ObliVM, a pro-
gramming framework for secure computation, and
demonstrated it on various applications (for exam-
ple, data mining, streaming algorithms, and graph
algorithms).13 Xiangyao Yu and his colleagues pro-
posed and evaluated PrORAM, a dynamic ORAM
prefetching technique.14 A heuristic compact
ORAM design, called SCORAM, is optimized for
secure computation protocols. SCORAM is almost
10 times smaller in circuit size and faster than all
other designs, so its feasible to perform secure com-
putations on gigabyte-sized datasets.15 Christopher
Fletcher and his colleagues proposed a new ORAM
structure, the PosMap lookaside buffer (PLB) and
PosMap compression techniques, that empirically
reduces the performance overhead from recursive
ORAM.16 Finally, researchers developed a novel fork
path ORAM scheme that supports redundant mem-
ory accesses by leveraging three optimization tech-
niques: path merging, ORAM request scheduling,
and merging-aware caching.17
36 I EEE Clo u d Co m p u t i n g w w w.co m p u t er .o rg /clo u d co m p u t i n g
Oblivious RAM
As Oded Goldreich and Rafail Ostrovsky originally
proposed, ORAM allows a trusted processor to use
untrusted RAM.18,19 Most existing ORAM solutions
use the basic memory structure suggested by Ostro-
vskys hierarchical scheme.19 The ORAM is arranged
in a series of progressively larger caches. Each cache
consists of a hash table of buckets. When a block
is requested, the algorithm checks a bucket at each
level of the hierarchy. If it finds the block, it con-
tinues searching for a dummy block, thus hiding
the desired blocks location. Finally, the algorithm
reinserts the block into the top-level cache. When a
cache is close to overflowing, its obliviously shuffled
into the cache below.
Recent ORAM works include optimizations of
the classic hierarchical scheme, such as the use of
cuckoo hashing and Bloom filters.20 Peter Williams
and Radu Sion proposed SR-ORAM, the first sin-
gle-round-trip polylogarithmic time ORAM, which
requires only logarithmic client storage.21 Taking
only a single round trip to perform a query, SR-
ORAM has an online communication/computation
cost of O(log n log log n). Jacob Lorch and his col-
leagues proposed Shroud, a general storage system
that hides data access patterns from the servers.22
Shroud uses many secure coprocessors acting in
parallel as client proxies in the datacenter. Circuit
ORAM, a new tree-based ORAM scheme, achieves
optimal circuit size both in theory and in practice
for realistic choices of block sizes.23
To define privacy, we denote a data access se-
quence as A = (op1, u1, data1), (op2, u2, data2), . . . ,
where opi is the read or write operation, ui is the data
address, and datai denotes the data contents. Given
two data access sequences A and A, a cloud storage
is defined to be privacy-preserved if its access pat-
terns cant be distinguished within polynomial time.
Next, we present the ORAM algorithm that we
apply to the cloud storage scenario.
We consider a client that wants to store and re-
trieve data in a cloud; the cloud is honest but curi-
ousthat is, it cant tamper with or modify the data,
but it can learn information about the data. We di-
vide the data into blocks, each of which is identified
by a unique address. For example, a typical block
size value is 64 or 256 Kbytes. Data stored on the
cloud is organized as a tree, where each node, or
bucket, stores several data blocks. Figure 1 shows an
example binary tree structure. Note that any arbi-
trary tree structure is applicable in ORAM. Follow-
ing previous work,10 we translate each read or write
operation into two primitives, ReadAndRemove and
Add, which are defined as follows:
 ReadAndRemove(u): given an address u specified
by the client, the cloud returns the correspond-
ing data block and removes it from storage.
Add(u, d): the client writes block d to address u
at the client storage.
With the two primitives, each read(u) operation
can be replaced by a ReadAndRemove(u) followed by
an Add(u, d) that writes the same data block back to
address u. Similarly, to implement a write(u, d) opera-
tion, we conduct a redundant ReadAndRemove(u) be-
fore Add(u, d). Although the number of access opera-
tions is doubled in ORAM, it prevents the untrusted
cloud from distinguishing read and write operations.
The implementation of ReadAndRemove(u) and
Add(u, d) is critical for hiding access patterns in
ORAM. When a data block is written into the cloud
storage, its always inserted into the root bucket in
level 0, as Figure 1 shows. As more data blocks are
added to the root bucket, it will eventually be full,
without residual capacity to accommodate new
blocks. To avoid overflowing, data blocks in each
nonleaf bucket are periodically evicted to its children
buckets. We assign a random number, a designator,
to each newly added data block to indicate which leaf
bucket its evicted to along the tree. Only the client
knows the mapping between the block address and
its associated designator. At each level of the tree,
the client randomly chooses several buckets to evict.
To prevent the cloud from tracking the eviction pro-
cess, dummy blocks are inserted into other children
buckets that dont receive the real data block.
To read a data block, the client first looks up
its corresponding designator in local storage, and
then reads all buckets along the path between the
root and the leaf bucket indicated by this designa-
tor. When it finds the data block, the algorithm re-
moves it from its current bucket and writes it back
to the root bucket with a new designator. This way,
the cloud cant infer which block is read because re-
peated reads for the same block will produce differ-
ent lookup paths through the tree.
Its easy to see that the algorithm guarantees an
independent and random access to the cloud stor-
age for each read or write operation. Furthermore,
the background eviction process generates a parti-
tion access sequence independent of the data access
pattern. By following an analysis similar to that of
Shi and her colleagues,10 we can prove that the algo-
rithm can preserve access privacy.
Load Balancing of ORAM Deployment
To deploy an ORAM-based storage in a distributed
system, we need to partition the corresponding tree
Se p t e m b e r / Oc t o b e r 2 0 1 6 I EEE Clo u d Co m p u t i n g
Level 0
Level 1
Level 2
Level 3
Server A Server B Server C
Figure 1. Oblivious RAM (ORAM)-based cloud storage.
structure into multiple parts, each of which is stored
in a server. For example, we consider storing the
ORAM tree shown in Figure 1 in three servers, each
of which can accommodate five buckets at most.
Figure 1 shows a partition scheme. Since the root
bucket is accessed in each read and write operation,
the server A holding the root bucket has the highest
access load. On the other hand, each read operation
involves only one leaf node, leading to the lowest
load on server B, which stores five leaf nodes in level
3. As this example illustrates, ORAM-based storage
would lead to a serious unbalanced data access load
among servers without proper bucket placement,
which motivates us to develop an algorithm to opti-
mize data placement.
Load Balancing for Static Deployment
We consider storing the data of multiple users to m
storage servers. We organize each users data as an
ORAM tree. In total, we have n buckets of size B.
The users generate a set of access requests that are
translated into a series of ReadAndRemove(u) and
Add(u, d) operations. Each bucket is the minimum
storage unit, and is associated with an access rate
ai due to the read, write, and eviction operations in
ORAM. We can estimate each buckets access rate
according to the characteristics of the ORAM algo-
rithm, such as tree structure and eviction probability.
The ith server can accommodate at most Ci buckets.
Based on the system model, our load balance prob-
lem can be described as a max-min problem.
Definition 1: The problem of load balance for deploy-
ing ORAM-based storage in clouds (LBOC). Given
a tree-based ORAM structure and a set of storage
37
Cloud Security
servers, the LBOC problem seeks a data placement
that minimizes the maximum access load among all
servers.
Since a bucket is the minimum access unit in
ORAM, we define binary variables xij to describe
bucket placement, given by
1, if the ith bucket is placed on the jth servver,
x ij
0, otherwise.
Since each bucket has to be placed at only one serv-
er, we have the following constraint:
m
x ij = 1, 1 i n. (1)
j=1
The total size of buckets deployed to a server cannot
exceed its capacity; we represent this as
n
x ij C j , 1 j m. (2)
i=1
We define another variable, yj, to denote the total
access rate to the jth server. It can be calculated by
n
yj = ai x ij , 1 j m. (3)
i=1
The maximum access rate of all servers are con-
strained by Y, giving us
y j Y , 1 j m. (4)
By summarizing these constraints, we formulate
the LBOC problem as a mixed integer linear pro-
gramming (MILP), given by
LBOC: min Y
(1), (2), (3), and (4),
x ij {0,1}, 1 i n,1 j m.
Theorem 1: The LBOC problem is NP-hard.
Proof: We can prove the NP-hardness of the LBOC
problem by reducing the well-known 2-partition
problem. We complete the proof by following a pro-
cess similar to that presented elsewhere.24
To solve the LBOC problem, we propose a fast
heuristic algorithm whose basic idea is to first solve
the MILP problem formulated by relaxing all integer
variables, and then find a feasible integer solution by
rounding the results. However, were dealing with
big data, and the corresponding formulation might
contain a large number of variables and constraints
because of too many buckets in the ORAM tree. It
would be time-consuming to solve such a large-scale
linear programming problem. The basic idea of our
proposed algorithm is to iteratively place buckets on
38 I EEE Clo u d Co m p u t i n g w w w.co m p u t er .o rg /clo u d co m p u t i n g
servers. In each iteration, we only need to deal with
a small-scale linear programming problem. Figure 2
shows the flowchart for the algorithm.
In the beginning, we initialize both the residual
capacity Cjres and current access load yjcurr of the jth
server to zero. In each iteration of the following while
loop, we conduct bucket placement by solving a lin-
ear programming problem with respect to N, current
access load, and Cjres on each server. The linear pro-
gramming problem is the same with LBOC formula-
tion except that xij is relaxed so it can be a real vari-
able between 0 and 1, as shown in (9) in Figure 2. In
addition, we consider current access load yjcurr in con-
straint (5), and constrain the capacity of each server
with Cjres in (8).
We then sort xij in descending order according
to their values after solving this linear programming
problem. For the ith bucket in set N, we place it in
the server with maximum value of xij based on the
expectation that a larger xij represents higher prob-
ability of the corresponding optimal data placement.
We finally update the values of Cjres and yjcurr; 1 j
m, to finish this iteration.
Load Balancing for Dynamic Deployment
In practice, the data access rate might change with
time. For example, some users intensively access their
data during the daytime, but rarely connect to the
cloud storage at night. Some companies retrieve their
business data for statistic computation at night to
minimize access conflict with normal business during
the day. In addition, irregular accesses exist because
of unpredictable user activities. Were motivated to
develop an online load-balancing algorithm to deal
with dynamic data accesses.
A straightforward approach is to divide the time
into discrete time slots and re-execute the algorithm
for static deployment in each time slot according to
the current access rates. Although this approach
can always guarantee load balance, it will incur fre-
quent data movement that consumes a large portion
of network bandwidth among storage servers. To
address this challenge, we propose an online algo-
rithm that dynamically adjusts data placement for a
tradeoff between load balance and data movement
among servers. In each time slot, we need to make
two decisions. First, we must decide whether data
rebalancing is needed. We define a threshold, denot-
ed by , and conduct rebalancing if the gap between
the highest and lowest load servers are greater than
the threshold . Otherwise, we keep the current data
placement. Second, we must decide how to move
data if load rebalancing is needed. Although we can
use the algorithm for static deployment, it might in-
 Start

Cjres = Cj , 1 j m

yjcurr = 0, 1 j m

WHILE
FOR
There are buckets that Put a set of unplaced buckets in set N
each xij in the order IF
havent been placed
the ith bucket isnt
Solve the following linear programming placed and Cjres > 0
min Y
yj + yjcurr Y, 1 j m; (5)
yj ai xij , 1 j m; (6)
iN
place this bucket on the jth server
m Cjres = Cjres 1;

j =1
x ij
= 1, i N; (7)
yjcurr = yjcurr + yj;
x
iN
ij
Cj , 1 j m;
res
(8)

0 xij 1, i N, 1 j m; (9)
IF
the ith bucket isnt
Sort xij in a descending order placed and Cjres > 0

END WHILE END IF

END

Figure 2. The flowchart of our proposed algorithm.

3,500
OPT
cur a large amount of data movement because the ILB
3,000 RAND
optimization process ignores the current data place-
Maximum access load

ment. Instead, we propose a simple heuristic algo- 2,500

rithm that greedily moves the highest loaded data
buckets to the server with the lowest load. The data 2,000
moving process terminates when the load gap be- 1,500
tween servers becomes less than the threshold.
1,000
Performance Evaluation
500
We conducted extensive simulations to evaluate our
proposed algorithms performance. For comparison 0
under static deployment, we also show the optimal 1 2 3 4 5 6 7 8 9 10
Instance
solution and the performance of an algorithm that
randomly allocates data buckets to servers. For dy- Figure 3. Comparison with the optimal solution in
namic deployment, we compare our proposed online 10 random instances. Our proposed algorithm, ILB,
algorithm with one that periodically updates the obtains results close to the optimal solutions.
whole deployment using the optimal solution.
We first evaluate ILBs performance by compar-
ing its results with the optimal solution. We simu- We average all results over 50 random instances
late allocating 200 data buckets to 10 servers, and with 10 servers. Each servers capacity is set as a
show the results of 10 random instances in Figure 3. Gaussian random variable with a mean of 100 and
On average, ILBs access load is 1.15 times the opti- a variance of 20. In each ILB iteration, we consider
mal solution, whereas RANDs corresponding ratio data placement for 100 buckets. As Figure 4a shows,
is 1.91. the access rate of both algorithms is an increasing
We then study the influence of the number of function of the bucket number. Moreover, the per-
buckets by changing the value from 600 to 1,000. formance gap between the two algorithms increases

Se p t e m b e r / Oc t o b e r 2 0 1 6 I EEE Clo u d Co m p u t i n g 39
Cloud Security

9,000 11,000
RAND RAND
Maximum access rate 8,000 ILB 10,000 ILB

Maximum access rate

7,000 9,000

6,000 8,000

5,000 7,000

4,000 6,000

3,000 5,000
600 700 800 900 1,000 10 20 30 40 50
Number of buckets Variance
(a) (b)

Figure 4. Maximum access rate versus (a) different numbers of buckets and (b) different variance of server
capacity. Our proposed algorithm, ILB, always outperforms the random algorithm.

12 We investigate our proposed algorithms time

ILB
ILB-S complexity by comparing its execution time with
10 the one (denoted by ILB-S) that solves the whole
Execution time (seconds)

problem using a single linear programming. As Fig-

8 ure 5 shows, ILB and ILB-S have a runtime of 0.14
seconds and 0.18 seconds, respectively, for 1,000
6 buckets and 10 servers, denoted by {1,000, 10}. In
instances with the largest scale, that is, {4,000, 40},
4 the ILB-S runs for 10.2 seconds, which is 3.4 times
the ILB runtime.
2 To evaluate our proposed online algorithms
performance, referred to as online_heuristic, for dy-
0 namic deployment, we consider a network with 100
{1,000, 10} {2,000, 20} {3,000, 30} {4,000, 40}
buckets and 10 servers, and randomly change each
Problem scale
buckets access rate. For comparison, we also show
Figure 5. Execution time under different instance the performance of an alternative online algorithm,
scales. Our proposed algorithm, ILB, is faster than online_opt, which adjusts data placement accord-
ILB-S. ing to the optimization framework. As Figure 6a
shows, we increase the rebalancing threshold from
100 to 400, and compare the maximum access load
as the number of buckets grows. For example, when of both algorithms. Although online_opt always
the number of buckets is 600, the maximum access outperforms our proposed heuristic algorithm, the
rate of RAND is 17 percent higher. The performance performance gap is small. We also compare the traf-
gap increases to 33 percent as the bucket number fic of both algorithms by increasing the value of
grows to 1,000. The results indicate that ILB can ef- from 100 to 400. As Figure 6b shows, our proposed
fectively reduce the maximum access rate. online_heuristic incurs significantly less network
We fix the mean value of server capacity at 100, traffic than online_opt during data replacement,
and then study the effect of server capacity variance because online_opt conducts global optimization
with 1,000 buckets and 10 servers. As Figure 4b for load rebalancing and ignores the current data
shows, the maximum access rate of both algorithms placement.
increases with the variance, but their performance
gap becomes smaller. We attribute this phenom-
enon to the fact that servers with small capacity RAM is promising in hiding access privacy in
quickly become full during data placement, and lots cloud storage, but there are still many open
of buckets have to be accommodated in the servers challenges in applying it in practice. We will contin-
with large capacity, leading to high access load. ue studying to reduce the complexity of the ORAM

40 I EEE Clo u d Co m p u t i n g w w w.co m p u t er .o rg /clo u d co m p u t i n g

 2,050
2,000 online_heuristic
online_opt
1,950
Maximum access load
1,900
1,850
1,800
1,750
1,700
1,650
1,600
1,550
100 200 300
The value of threshold
(a)
Figure 6. Performance comparison: (a) maximum access load under different values of threshold , and
(b) traffic of data movement under different values of threshold .
algorithm, and solving practical deployment issues
in the near future.
Acknowledgments
This work was supported by the Japan Society for
the Promotion of Science KAKENHI grant number
16K16038 and the Council for Science, Technology
and Innovation (CSTI), Cross-Ministerial Strategic
Innovation Promotion Program (SIP), Enhancement
of Societal Resiliency against Natural Disasters
(Funding agency: JST). A part of this article was pub-
lished in the proceedings of the 2014 IEEE Confer-
ence on Computer Communications Workshops.
References
1. D. Beaver et al., Finding a Needle in Haystack:
Facebooks Photo Storage, Proc. 9th USENIX
Conf. Operating Systems Design and Implemen-
tation (OSDI), 2010, pp. 4760.
2. M. Islam, M. Kuzu, and M. Kantarcioglu, Ac-
cess Pattern Disclosure on Searchable Encryp-
tion: Ramification, Attack, and Mitigation,
Proc. Network and Distributed System Security
Symp. (NDSS), 2012, pp. 115.
3. K.D. Bowers, A. Juels, and A. Oprea, Hail: A
High-Availability and Integrity Layer for Cloud
Storage, Proc. 16th ACM Conf. Computer and
Comm. Security, 2009, pp. 187198.
4. A. Bessani et al., DepSky: Dependable and Se-
cure Storage in a Cloud-of-Clouds, Proc. 6th
Conf. Computer Systems, 2011, pp. 3146.
5. E. Stefanov et al., Iris: A Scalable Cloud File
System with Efficient Integrity Checks, Proc.
28th Ann. Computer Security Applications Conf.,
2012, pp. 229238.
Se p t e m b e r / Oc t o b e r 2 0 1 6 I EEE Clo u d Co m p u t i n g
9,000
online_heuristic
8,000
online_opt
7,000
6,000
Teaffic
5,000
4,000
3,000
2,000
1,000
400 100 200 300 400
The value of threshold
(b)
6. Z. Wu et al., SPANstore: Cost-Effective Geo-
replicated Storage Spanning Multiple Cloud
Services, Proc. ACM Symp. Operating Systems
Principles, 2013, pp. 292308.
7. B. Chor et al., Private Information Retrieval, J.
ACM, vol. 45, no. 6, 1998, pp. 965981.
8. R. Sion and B. Carbunar, On the Computation-
al Practicality of Private Information Retrieval,
Proc. Network and Distributed Systems Security
Symp., 2007, pp. 110.
9. M.T. Goodrich and M. Mitzenmacher, MapRe-
duce Parallel Cuckoo Hashing and Oblivious
RAM Simulations, CoRR, vol. abs/1007.1259,
2010.
10. E. Shi et al., Oblivious RAM with O((log N)3)
Worst-Case Cost, Advances in Cryptology (ASI-
ACRYPT 11), 2011, pp. 197214.
11. T. Moataz, T. Mayberry, and E.-O. Blass, Con-
stant Communication ORAM with Small Block-
size, Proc. ACM SIGSAC Conf. Computer and
Comm. Security, 2015, pp. 862873.
12. J. Dautrich and C. Ravishankar, Combin-
ing ORAM with PIR to Minimize Bandwidth
Costs, Proc. 5th ACM Conf. Data and Applica-
tion Security and Privacy (CODASPY), 2015, pp.
289296.
13. C. Liu et al., ObliVM: A Programming Frame-
work for Secure Computation, Proc. IEEE
Symp. Security and Privacy, 2015, pp. 359376.
14. X. Yu et al., ProRAM: Dynamic Prefetcher for
Oblivious RAM, Proc. ACM/IEEE Ann. Intl
Symp. Computer Architecture (ISCA), 2015, pp.
616628.
15. X.S. Wang et al., SCORAM: Oblivious RAM
for Secure Computation, Proc. ACM SIGSAC
41
Cloud Security
Conf. Computer and Comm. Security, 2014, pp.
191202.
16. C.W. Fletcher et al., FreeCursive ORAM: [Near-
ly] Free Recursion and Integrity Verification for
Position-Based Oblivious RAM, Proc. 20th Intl
Conf. Architectural Support for Programming Lan-
guages and Operating Systems, 2015, pp. 103116.
17. X. Zhang et al., Fork Path: Improving Efficiency
of ORAM by Removing Redundant Memory Ac-
cesses, Proc. 48th Intl Symp. Microarchitecture
(MICRO), 2015, pp. 102114.
18. O. Goldreich, Towards a Theory of Software
Protection and Simulation by Oblivious RAMs,
Proc. ACM Symp. Theory of Computing, 1987,
pp. 182194.
19. R. Ostrovsky, Efficient Computation on Oblivi-
ous RAMs, Proc. ACM 22nd Ann. ACM Symp.
Theory of Computing (STOC), 1990, pp. 514523.
20. O. Goldreich and R. Ostrovsky, Software Pro-
tection and Simulation on Oblivious RAMs, J.
ACM, vol. 43, no. 3, 1996, pp. 431473.
21. P. Williams and R. Sion, Single Round Access
Privacy on Outsourced Storage, Proc. ACM
Conf. Computer and Comm. Security (CCS),
2012, pp. 293304.
22. J.R. Lorch et al., Shroud: Ensuring Private Ac-
cess to Large-Scale Data in the Data Centers,
Proc. USENIX Conf. File and Storage Technolo-
gies (FAST), 2013, pp. 199213.
23. X. Wang, H. Chan, and E. Shi, Circuit ORAM:
On Tightness of the Goldreich-Ostrovsky Lower
Bound, Proc. ACM SIGSAC Conf. Computer
and Comm. Security, 2015, pp. 850861.
24. P. Li and S. Guo, Load Balancing for Privacy-
Preserving Access to Big Data in Cloud, Proc.
IEEE Conf. Computer Comm. Workshops, 2014,
pp. 524528.
Peng Li is an associate professor in the School of
Computer Science and Engineering at the University
of Aizu, Japan. His research interests include wireless
communication and networking, specifically wireless
sensor networks, green and energy-efficient mobile
networks, cross-layer optimization for wireless net-
works, cloud computing, big data processing, and
smart grid. Li has a PhD in computer science and
engineering from the University of Aizu, Japan. Hes
a member of IEEE. Contact him at pengli@u-aizu
.ac.jp.
Song Guo is a full professor in the Department
of Computing atthe Hong Kong Polytechnic Univer-
sity. His research interests include cloud computing,
42 I EEE Clo u d Co m p u t i n g w w w.co m p u t er .o rg /clo u d co m p u t i n g
big data,wireless network,and cyberphysical systems.
Guo has a PhD in computer science from the Univer-
sity of Ottawa. Hes a senior member of IEEE, a se-
nior member of ACM, and an IEEE Communications
Society Distinguished Lecturer. Contact him at song
.guo@polyu.edu.hk.
Toshiaki Miyazaki is a professor in the School
of Computer Science and Engineering and the dean
of the Undergraduate School of Computer Science
and Engineering at the University of Aizu, Fukushima,
Japan. His research interests include reconfigurable
hardware systems, adaptive networking technologies,
and autonomous systems. Miyazaki has a PhD in elec-
tronic engineering from the Tokyo Institute of Tech-
nology. Hes a senior member of IEEE, theInstitute of
Electronics, Information, and Communication Engi-
neers, and the Information Processing Society of Ja-
pan. Contact him at miyazaki@u-aizu.ac.jp.
Miao Xie is a PhD student in the School of Engi-
neering and IT at the School of Engineering and IT,
University of New South Wales at the Australian De-
fence Force Academy. His research interests include
intrusion/anomaly detection in wireless sensor net-
works, network security, data mining, and forecasting
algorithms. Xie has a masters degree in engineering
and IT from University of New South Wales. Contact
him at m.xie@adfa.edu.au.
Jiankun Hu is a professor and research director at
the Cyber Security Lab, School of Engineering and
IT, University of New South Wales at the Australian
Defence Force Academy. His research interests in-
clude cybersecurity, including biometrics security.
Hu has a PhD in control engineering from the Harbin
Institute of Technology, China. Hes a member of the
IEEE. Contact him at j.hu@adfa.edu.au.
Weihua Zhuang is a full professor in the Depart-
ment of Electrical and Computer Engineering at the
University of Waterloo, Canada. Her research inter-
ests include multimedia wireless communications,
wireless networks, and radio positioning. Zhuang has
a PhD in electrical engineering from the University
of New Brunswick, Canada. Contact her at wzhuang
@bbcr.uwaterloo.ca.
Read your subscriptions through
the myCS publications portal at
http://mycs.computer.org.
NEW PREFERRED PLUS

MEMBERSHIP
OPTIONS TRAINING & DEVELOPMENT

FOR A
BETTER FIT. RESEARCH

BASIC

STUDENT

And a better match for your career goals.

IEEE Computer Society lets you choose your membership and
the benefits it provides to fit your specific career needs. With four
professional membership categories and one student package, you can
select the precise industry resources, offered exclusively through the
Computer Society, that will help you achieve your goals.

Learn more at www.computer.org/membership.

Cloud Security

Cryptographic Public
Verification of Data
Integrity for Cloud
Storage Systems
Yuan Zhang, Chunxiang Xu, and Hongwei Li , University of Electronic Science
and Technology of China

Xiaohui Liang, University of Massachusetts Boston

A public verification of data integrity scheme uses

a random masking technique to protect against
external adversaries. A performance analysis
demonstrates that the proposed scheme is efficient
in terms of user auditing overhead.
44 I E E E C l o u d C o m p u t i n g p u b l i s h e d b y t h e I E E E c o m p u t e r s o cie t y  2325-6095/16/$33.00 2016 IEEE
 loud storage services enable users
to outsource their data to cloud
servers and access that data re-
motely over the Internet. These
services give users an efficient and
flexible way to manage their data
without deploying and maintaining local storage de-
vices and services.14 Specifically, users can process
their data on their PCs, outsource the processed
data to cloud servers, and use the data on other de-
vices (for example, mobile phones). The great con-
venience provided by such services is leading to a
growing number of cloud storage providers.5,6
However, despite the benefits brought by cloud
storage services, critical security concerns in data
outsourcing exist. One of the most important se-
curity concerns for users is data integritythat is,
whether their data remains intact on cloud serv-
ers.7 A cloud service provider might hide data loss
incidents to maintain its reputation8 or discard data
thats rarely accessed to save storage space, while
claiming that no data loss has occurred. Moreover,
an external adversary might distort users data
on cloud servers for financial or political reasons.
Consequently, users require an efficient and secure
verification method to ensure their datas integrity.9
Some schemes rely on users to perform the veri-
ficationthat is, a user must actively and repeatedly
engage in a process that has expensive communica-
tion and computation overhead, which assumes that
the user always has a device with sufficient compu-
tation capability and Internet bandwidth to perform
integrity verification. To reduce the verification bur-
den on users, we propose a public verification para-
digm in which an external and independent auditor
periodically verifies data integrity on users behalf.
Existing public verification schemes assume the
auditor is honest and cant be corrupted. But this
is a strong assumption, since auditors can be cor-
rupted in practice. A malicious auditor can always
claim that the outsourced data is (not) retained well
in the cloud, regardless of the verification result,
so even the malicious auditor wouldnt perform the
verification. In addition, the vulnerability of existing
schemes is further exacerbated by the fact that the
malicious auditor colludes with the cloud server and
generates a biased challenging message to check the
data blocks that arent corrupted, and thus deceiving
Se p t e m b e r / Oc t o b e r 2 0 1 6 I EEE Clo u d Co m p u t i n g
the user. Constructing an efficient public verifica-
tion of data integrity for cloud storage against mali-
cious auditors is of paramount importance.10
Some public verification schemes dont protect
against external adversaries, so an active and online
adversary can intrude into the cloud server, modify
the outsourced data, tamper with the interaction
messages between the cloud server and auditor, and
pass the auditors verification. (See the Literature
Review sidebar for related work in this area.) A
common approach to resisting such adversaries is to
have the cloud server interact with the auditor using
a secure channel. However, constructing a secure
channel for each verification task is cumbersome.
We propose a public verification scheme in which
we adopt a random masking technique instead of
secure channels between cloud servers and auditors to
resist external adversaries. In addition, users are able
to examine auditors behaviors to prevent malicious
auditors from fabricating verification results.
Furthermore, we use Bitcoin to construct an unbiased
challenge message, which helps prevent malicious
auditors from colluding with cloud servers. Security
and performance analyses show that the proposed
scheme can achieve security goals with efficiency.
Public Verification of Data Integrity
Traditional cryptographic primitives for protect-
ing data security, such as message authentication
code (MAC) and signature, can ensure data integ-
rity. However, public verification schemes cant use
these methods directly since they require an exter-
nal auditor to possess the data to verify its integrity,
which creates high communication overhead. Public
key-based homomorphic linear authenticator (HLA)
offers a practical and affordable solution to enable
an auditor to verify the integrity of outsourced data
without demanding a local copy of the data.
As Figure 1 shows, the public verification
scheme consists of three entities: user, cloud server,
and auditor. The user outsources data to the cloud
server and later accesses the data as needed. After
outsourcing the data, the user delegates the data
integrity verification task to the auditor, a trusted
entity with the expertise and capabilities to perform the
verification. To verify the outsourced datas integrity,
the auditor first generates a challenge message and
issues it to the cloud server. With the challenge
45
Cloud Security
Cloud servers
Public data
Dataflow verification
Data verification delegation
User Auditor
Figure 1. System model. There are three entities in the public verification
scheme: user, cloud server (cloud service provider), and auditor.
message, the cloud server generates corresponding
proof information and sends it to the auditor. After
receiving this information, the auditor verifies the data
integrity by checking the proof informations validity. If
the verification fails, the auditor informs the user that
the data might be corrupted. Because we use HLA,
the proof information generated by the cloud server
doesnt include the data file, thus the communication
overhead between the cloud server and auditor is low.
Basic Public Verification Scheme
Because public verification aims to enable a third-
party auditor to efficiently and securely verify the
integrity of outsourced data, these schemes should
be evaluated using both systems and crypto criteria.
Systems criteria include
Efficiency. A public verification scheme should be
as efficient as possible in terms of communication
and computation overhead.
Boundless verification. A public verification
scheme should enable auditors to verify data
integrity without a priori bound on the number
of verification interactions.
Stateless auditor. Auditors should be stateless
and shouldnt need to maintain and update state
during verification.
Crypto criteria include
Soundness. Any time a cloud server passes the
auditors verification, it must possess the specified
data intact. This should be provably secure under
the security model proposed by Hovav Shacham
and Brent Waters.11
Resistance against external adversaries. A
secure public verification scheme should resist
common attacks, where an external, active,
and online adversary modifies the outsourced
46 I EEE Clo u d Co m p u t i n g w w w.co m p u t er .o rg /clo u d co m p u t i n g
data and tampers with the interaction messages
between the cloud server and auditor to pass the
verification.9
Resistance against malicious auditors. A secure
public verification should resist malicious
auditors, where a malicious auditor might
fabricate a verification result to deceive the user
and/or cloud server.10
These influencing factors arent isolated and
can be closely related; thus, public verification tech-
niques should be evaluated from both cryptography
and engineering perspectives.
Warm-up Scheme
We first review the public verification scheme pro-
posed by Shacham and Waters (SWP),11 which in-
volves a user U, cloud server C, and auditor A. SWP
consists of five algorithms.
Setup. With a security parameter , U determines the
bilinear map: e: G GGT.11 Then U chooses secret
parameters and generates the public parameters
(v = g , u1, , us), where g is the generator of the
multiplicative group G.
Store. User U transforms its data M into n blocks and
further splits each block into s sectors. User U chooses
a random element name for file naming and computes
a file tag on name and the public parameters,
which enables A to check the validity of the public
parameters used to check the data integrity. In other
words, the validity of ensures that C cant deceive A
by replacing the public parameters. Then, U gener-
ates a tag i for each data block. The tag i is based
on the BLS signature,12 which is the HLA, and allows
multiple tags to be aggregated into a single tag, where
the size of the aggregated tag is independent of the
number of tags to be aggregated, and a verifier can
confirm the validity of the aggregated tag instead of
checking the tags one by one. Finally, U outsources
the data file, file tag, and i to C.
Audit. For each verification task, A first determines I
and randomly chooses vi, i I, where I is a random
subset of set {1, , n} to determine which data
blocks should be verified, and vi is a random element
for each verification. Next, A sends the challenge
message chal = {(i, vi)}(i I) to C.
Prove. After receiving chal, C verifies , and then sends
vi
i ,j = v m ( j [1, s])
iI
i ij
iI
 Literature Review
ublic verification techniques let users outsource
data integrity verification to a third-party auditor.
GiuseppeAteniese and his colleagues proposed the
first public verification scheme.1 HovavShacham and
Brent Waters defined the first formal security model
of public verification and proposed a public verifica-
tion scheme with full proofs of security against arbi-
trary adversaries.2 Various public verification schemes
were later proposed based on this work.
Solomon Worku and his colleagues proposed
a privacy-preserving public verification scheme.3
However, they didnt consider external attacks in their
threat model, where an external adversary thats active
and online can modify the outsourced data and tam-
per with the interaction messages between the cloud
server and auditor, thus invalidating the data integrity
verification.4 A common approach to resisting such an
adversary is for the cloud server to send proof infor-
mation to the auditor using a secure channel, which
ensures the integrity and correctness of the interac-
tion messages. However, constructing a secure chan-
nel between the cloud server and auditor for each
verification task is cumbersome. Other work proposed
the first privacy-preserving public verification scheme
against external adversaries without secure channels.5
Frederik Armknecht and his colleagues, considering
that malicious auditors exist in practice, proposed the
first framework to resist malicious auditors.6 However,
their work cant achieve public verification with accept-
able communication overhead. The Secure Certifi-
cateless Public Verification scheme is the first public
verification scheme to protect against malicious audi-
tors with minimal communication overhead.7
References
1. G. Ateniese et al., Provable Data Possession at
Untrusted Stores, Proc. 2007 ACM SIGSAC Conf.
Computer and Comm. Security (CCS 07), 2007, pp.
598609.
2. H. Shacham and B. Waters, Compact Proofs of Re-
trievability, J. Cryptology, vol. 26, no. 3, 2013, pp.
442483.
3. S.G. Worku et al., Secure and Efficient Privacy-Pre-
serving Public Auditing Scheme for Cloud Storage,
Computers and Electrical Eng., vol. 40, no. 5, 2014,
pp.17031713.
4. Y. Zhang et al., Cryptanalysis of an Integrity Check-
ing Scheme for Cloud Data Sharing, J. Information
Security and Applications, vol. 23, Aug. 2015, pp.
6873.
5. C. Xu et al., An Efficient Provable Secure Public Au-
diting Scheme for Cloud Storage, KSII Trans. Inter-
net and Information Systems, vol. 8, no. 11, 2014,
pp. 42264241.
6. F. Armknecht et al., Outsourced Proofs of Retriev-
ability, Proc. 2014 ACM SIGSAC Conf. Computer
and Comm. Security (CCS 14), 2014, pp. 831843.
7. Y. Zhang et al., SCLPV: Secure Certificateless Pub-
lic Verification for Cloud-Based Cyber-Physical-So-
cial Systems Against Malicious Auditors, IEEE Trans.
Computational Social Systems, vol. 2, no. 4, 2015,
pp. 159170.

to A as the corresponding proof information. Observe where H() is a BLS hash.12

that for each verification, vi is different, ensuring the
proof informations freshness. By allowing multiple Vulnerability against External Adversaries
data blocks and signatures to be aggregated into a In SWP, the proof information generated by C is {,
short tag (that is, j and =
iI i
vi ), HLA guar- j}j[1, s]. Observe that
antees minimal communication overhead.

Verify. Upon receiving the proof information, A j = iI

v i m ij
verifies the data integrity by checking
when an external and active adversary intrudes into
s C and modifies each data block mij to m ij = m ij + lij

uj j , v ,
vi
e (, g) = e H ( i || name) for i[1, n], j [1, s]. The corresponding proof

iI j=1 information at this point is {, , j } j[1,s], where

Se p t e m b e r / Oc t o b e r 2 0 1 6 I EEE Clo u d Co m p u t i n g 47
Cloud Security
Table 1. The log file L used to verify data integrity.
Bitcoin hash Authenticator Random element
(1)
Bl t (1) Rj
(2)
Bl t (2) Rj
(k) (k)
Bl t Rj
j = iI
ij ,
vim
but still corresponds to mij rather than
ij ( i I, j [1, s ]) . To deceive A and pass the
m tablished verificationshould be checked. In the
verification, the adversary can eavesdrop the
challenge message chal, intercept the proof
information, and compute
j = j
( i, v i )chal
v ilij (1 j s).
Finally, the adversary sends the modified proof in-
formation to A, and the modified data file passes the
verification.
Vulnerability against Malicious Auditors
A corrupted auditor can deceive C and U in several
ways.
First and simplest, a malicious auditor can claim
that the outsourced data is (not) retained intact in
the cloud, no matter what the verification result is,
even the malicious auditor wont perform the verifi-
cation. Because C and U trust the auditor, they will
accept its claim without doubt.
Second, the malicious auditor can collude with
C to deceive U. In this case, the outsourced data has
been corrupted, but the auditor generates a biased
challenge message to check the data blocks, which
arent corrupted.
Third, the malicious auditor can collude with U
to circumvent C. That is, the outsourced data is re-
tained in good condition, but the auditor claims that
its been corrupted.
SWP cant protect against malicious auditors, so
it must bear a strong assumption: auditors are hon-
est and reliable. Resisting malicious auditors is thus
a worthwhile area for further study.
Protecting against Malicious Auditors and
External Adversaries
As discussed earlier, malicious auditors and external
adversaries can invalidate the SWP technique. Most
existing public verification schemes follow SWP, and
48 I EEE Clo u d Co m p u t i n g w w w.co m p u t er .o rg /clo u d co m p u t i n g
thus have the same framework and threat model.
Consequently, these schemes cant protect against ex-
Data ternal adversaries and malicious auditors.
(1)
j
(1)
An external adversary can invalidate SWP, since
(2) (2)
theres a definite linear relationship between the
j proof information (j) and the data blocks (mij). To
resist external adversaries without secure channels,
we adopt a random masking technique when com-
(k) (k)
j puting the proof information. Specifically, we use
random masking as a nonlinear disturbance code to
change the definite linear relationship between the
proof information and the data blocks to a nonlinear
relationship.
To resist malicious auditors, auditors behav-
iorthat is, whether the auditor performs the es-
enhanced scheme, the auditor is required to gener-
ate an entry for each verification task and store it
in a log file. The user audits the auditors behavior
by checking the log files validity, guaranteeing that
a malicious auditor cant fabricate a verification re-
sult to deceive the user and/or cloud server. Here, we
want to further emphasize that the periodicity of the
users audits of the auditor should be much longer
than the periodicity of the auditors verification of
the datas integrity.
However, such a paradigm cant deter malicious
auditors perfectly, since a malicious auditor can still
deceive the user by generating a biased challenge
message, where the corrupted data blocks will nev-
er be checked. For security and efficiency reasons,
its also impractical to require the user to generate
a new challenge message for each verification task.
To address this problem, we use Bitcoin to construct
the challenge message. Given a determinate time t,
if t is a past or current time, we can easily find a
Bitcoin block, which is generated in the nearest time
t; however, if t is a future time, the Bitcoin block,
which is generated in t, is unpredictable. Here, we de-
note the hash of a Bitcoin block, which is generated
in a past time t as Blt. Since Bitcoin has this property,
we can consider Bitcoin as a time-based pseudoran-
domness source. We can compute this sources out-
put when its input is a past/current time; otherwise,
the output is unpredictable.
As Figure 2 shows, the enhanced scheme consists
of six algorithms: Setup, Store, Audit, Prove, Verify,
and CheckLog. In our enhanced scheme, the first two
algorithms are the same as those in SWP.
In Audit, the auditor first acquires Blt based on
the current time t and initializes the pseudorandom
bit generator as = GetRand(Blt). Then the auditor
generates the challenging message {(i, vi)}i I on
and .
 Audit Verify Auditor
Prove Cloud server
Based on Blt, generate a
challenging message {i, vi} {i, vi }
Randomly choose r
Verify the validity of

Verify the validity of {, , j , Rj } Compute Rj , , j

Generate an entry {Blt(k),(k), t(k), Rt(k)}

Store the entry into a log file as

shown in Table I.

{M, i , }

Store Data owner

Set up Checklog Data owner

Generate secret parameters Transform the data file to

M={m11, , m1s, mns}
Generate public parameters
{v, u1, , us} Compute a file tag , and n tags i
for each mi , 1 i n
Check the validity of the log file

Figure 2. Execution steps of our scheme. Different algorithms are performed by different entities and the
details are shown in the figure.

In Prove, C randomly chooses r G as a secret blocks and generates a set of challenge messages
parameter, and computes I(B) = {{ i(1) , v (i1) }i[ I(1) ],...,{ i(b) , v (i b) }i[ I( b ) ] } , where b is
the size of the subset B. Then the user sends B to the
R j = u rj , = iI
ivi , *j = iI
v i m ij , and auditor and receives (B) , R(jB) , (jB) ( j [1, s]) , where

j = r 1 (*j + h(R j ))( j [1, s]) ,

(k)
(B) = ( i(k) ) vi .
kB iI( B)

where h() is a BLS hash. Then, C sends {, , j, Finally, the user audits the auditor by checking
Rj}j [1, s] to A.
e ( (B) , g)

s s h( R(jk ) )
In Verify, upon receiving the proof information v(i k ) (jk )
= e H ( (B) , g) (R(jk) )
u j kB , v.

{, j, Rj}j [1,s], the auditor verifies kB iI( B) kB j=1 j=1

(1)
s s

( , g) = e H(i || name) vi Rj j uj h(R j ) , v.
If Equation 1 fails, the user can consider that the
iI j=1 j=1
cloud-stored data is corrupted, and either the TPA
If the verification holds, the auditor creates an en- and/or C are malicious.
try as (Blt, , j, Rj)j[1,s]. Finally, the auditor stores the
entry in a log file L, as shown in Table 1 (in L, j [1, s]), Remark
where k denotes the index of the verification that the Unlike some previous schemes,4,7,8,13 we dont con-
auditor performs. That is, {Blt(k ) , (k ) , R(jk ) , (jk ) } is the sider privacy protection of user data against the audi-
proof information of kth verification. tor. An auditor in our scheme could be compromised
In CheckLog, to check the validity of L, the user or could collude with the cloud server, which might
first picks a random subset B of indices of Bitcoin reveal the users data to the auditor. Exploiting data

Se p t e m b e r / Oc t o b e r 2 0 1 6 I EEE Clo u d Co m p u t i n g 49
Cloud Security
encryption before outsourcing is an easy and afford-
able way to achieve such privacy protection.
Security and Performance Evaluation
We first analyze the security of the proposed scheme
by using the crypto criteria proposed earlier.
With the proposed scheme, if the cloud server
passes the auditors verification, it ensures that the
verified data is intact. This claim is provably se-
cure under the Shacham and Waters security mod-
el,11 and the formal proof is presented elsewhere.8
Therefore, our proposed scheme achieves the sound-
ness criterion.
Next, we show that the proposed scheme can
resist an external adversary. The adversary first
intrudes into the cloud server and modifies m ij to
ij = m ij + lij . In Prove, the adversary intercepts the
m the auditor with the proof information. The size of
proof information {, , j , R j } j[1,s] , where
*
j = r 1( *j + h(R j )) and j = iI
vim
Since r and r1 are unknown to the adversary, and
j = r 1 ( *j + h(R j ))
= r 1 vim ij + h(R j )
iI
= r 1 v i m ij + h(R j ) + r 1 v ilij ,
iI iI
the adversary cant compute
r 1 ( iI )
v ilij .
Thus, such an attack is computationally infeasible.
Finally, we show that the proposed scheme can
protect against a malicious auditor. Because the pro-
posed scheme meets the soundness criterion, the
cloud server and malicious auditor cant forge a proof
that passes the data integrity verification. In addi-
tion, the user will audit the auditors behavior, and
the auditor must execute the established verification.
Furthermore, because the challenge message is deter-
mined by the time-based pseudorandom source (that
is, the Bitcoin) and can be recovered by the user, the
auditor cant deceive the user by generating a biased
challenge message. In other words, even if the audi-
tor colludes with the cloud server, it cant deceive the
user by only checking the uncorrupted datas integ-
rity. Therefore, malicious auditors cant invalidate the
proposed scheme. A formal and detailed proof is pre-
sented elsewhere.10
All in all, the proposed scheme meets the crypto
criteria proposed earlier.
50 I EEE Clo u d Co m p u t i n g w w w.co m p u t er .o rg /clo u d co m p u t i n g
Next, we analyze the proposed schemes perfor-
mance in terms of communication and computation
overhead. We tested all the experiments using a
Windows 7 system with an Intel Core 2 i5 CPU run-
ning at 2.53 GHz with 2 Gbyte DDR 3 of RAM (1.74
Gbytes available). We implemented all algorithms
using C language and our code uses the MIRACL li-
brary version 5.6.1. We use an MNT elliptic curve,12
and a security level of 80 bits. The difference for
choices on s is discussed elsewhere.11 For simplicity,
we give the atomic operation analysis for the case s = 1
in the following.
We first analyze communication overhead be-
tween the cloud server and auditor. In Audit and
Prove, the auditor sends the challenge message to
the cloud server, and the cloud server responds to
the challenge message is c |i + vi|, where |i| denotes
the size of i. In our scheme, i and vi are random
ij .
numbers with 80 bits under || = 80 bits. The size of
the proof information is || + || + |j| + |Rj|, which
is approximately equal to 80 bytes under the 80-bit
security level. Therefore, the total communication
cost between the auditor and cloud server is approxi-
mately equal to 90 bytes for each verification task.
Next, we analyze our schemes computation
overhead. Because the performance analysis on the
auditor side of the proposed scheme is presented
elsewhere,8 we only analyze the additional cost on
the user side to protect against the malicious audi-
tor. In CheckLog, the user performs Equation 1 to
audit the auditors behavior, which is the only addi-
tional computational overhead for the user. Figure
3 shows the additional verification overhead on the
user side in different challenge entries. As Figure
3 shows, the user in the proposed scheme can au-
dit the auditor with high efficiency. In other word,
compared with SWP, the proposed scheme requires
higher verification costs on the user side, but this
extra cost is exactly the guarantee to resist the mali-
cious auditor, so is a worthwhile sacrifice.
e plan to focus our future research efforts in
several areas.
Most existing public verification schemes are
based on the public-key cryptosystem. In these
schemes, even the auditor is equipped with a powerful
device, so verification is a second-long (hundreds of
milliseconds) computation. For these schemes, it
would be impractical for the auditor to verify the
data integrity using a low-power device. Reducing
the computation of the operations in the public-
key cryptography to those in the symmetric-key

cryptography while retaining the benefits of public
verification is still an open issue and deserves
further investigation.
In practice, an auditor always serves multiple
cloud users and multiple cloud servers. If the audi-
tor handles multiple verification tasks from different
users and cloud servers one by one, the verification
can incur a huge delay and become a bottleneck in
applications. Therefore, enabling a single auditor to
simultaneously handle multiple verification tasks
from different users and different cloud servers with
high efficiency is also worth further study.
With the development of cryptography, many new
cryptographic primitives have been proposed, such
as program obfuscation and structure-preserving
cryptography. These new cryptographic primitives
bring new appealing features and powerful
functionalities not previously available. Combining
current public verification techniques with these
new and powerful cryptographic primitives could
provide users with more feature-rich cloud storage
services than ever before. This also remains an open
research issue that should be further explored.
Acknowledgments
The National Natural Science Foundation of
China under grants 61370203, 61472065, and
61350110238; the Science and Technology on
Communication Security Laboratory Foundation
under grant 9140C1103 01110C1103; the
International Science and Technology Cooperation
and Exchange Program of Sichuan Province,
China, under grant 2014HH0029; and the China
Postdoctoral Science Foundation under grant
2014M552336 supported this work.
References
1. S. Yu et al., Can We Beat DDos Attacks in
Clouds? IEEE Trans. Parallel and Distributed
Systems, vol. 25, no. 9, 2014, pp. 22452254.
2. S. Yu, G. Wang, and W. Zhou, Modeling Mali-
cious Activities in Cyber Space, IEEE Network,
vol. 29, no. 6, 2015, pp. 8387.
3. S. Yu, S. Guo, and I. Stojmenovic, Fool Me If
You Can: Mimicking Attacks and Anti-Attacks
in Cyberspace, IEEE Trans. Computers, vol.64,
no. 1, 2015, pp. 139151.
4. C. Wang et al., Privacy-Preserving Public Au-
diting for Secure Cloud Storage, IEEE Trans.
Computers, vol. 62, no. 2, 2013, pp. 362375.
5. H. Li et al., Enabling Fine-Grained Multi-
Keyword Search Supporting Classified Sub-
Dictionaries over Encrypted Cloud Data, IEEE
Trans. Dependable and Secure Computing, vol.
Se p t e m b e r / Oc t o b e r 2 0 1 6 I EEE Clo u d Co m p u t i n g
25
Challenge blocks number c =300
Challenge blocks number c = 460
Verification delay (seconds)
20
15
10
5
0
0 5 10 15 20 25 30 35 40 45 50
Number of entries to be audited
Figure 3. Verification delay on the user side. This
figure shows the overhead that the user audits the
auditors behavior.
13, no. 3, 2016, pp. 312325.
6. H. Li et al., Engineering Searchable Encryption
of Mobile Cloud Networks: When QoE Meets
QoP, IEEE Wireless Comm., vol. 22, no. 4, 2015,
pp. 7480.
7. C. Wang et al., Privacy-Preserving Public Au-
diting for Data Storage Security in Cloud Com-
puting, Proc. 2010 IEEE Intl Conf. Computer
Comm. (INFOCOM 10), 2010, pp. 19.
8. C. Xu et al., An Efficient Provable Secure Pub-
lic Auditing Scheme for Cloud Storage, KSII
Trans. Internet and Information Systems, vol. 8,
no. 11, 2014, pp. 42264241.
9. Y. Zhang et al., Cryptanalysis of an Integrity
Checking Scheme for Cloud Data Sharing, J.
Information Security and Applications, vol. 23,
Aug. 2015, pp. 6873.
10. Y. Zhang et al., SCLPV: Secure Certificateless
Public Verification for Cloud-Based Cyber-Phys-
ical-Social Systems Against Malicious Auditors,
IEEE Trans. Computational Social Systems, vol.
2, no. 4, 2015, pp. 159170.
11. H. Shacham and B. Waters, Compact Proofs of
Retrievability, J. Cryptology, vol. 26, no. 3, 2013,
pp. 442483.
12. D. Boneh, B. Lynn, and H. Shacham, Short Sig-
natures from the Weil Pairing, Proc. 7th Intl
Conf. Theory and Application of Cryptology and
Information Security: Advances in Cryptology
(ASIACRYPT 01), 2001, pp. 514532.
13. S. Guadie Worku et al., Secure and Efficient
Privacy-Preserving Public Auditing Scheme for
Cloud Storage, Computers and Electrical Eng.,
vol. 40, no. 5, 2014, pp.17031713.
51
Cloud Security

Yuan Zhang is a PhD student in computer sci-
ence and engineering at the University of Electronic
Science Technology of China (UESTC), Chengdu.
His research interests include cryptography, network
security, and cloud computing security. Zhang has a
BSc from UESTC. Hes a student member of IEEE.
Contact him at ZY_LoYe@126.com.
Chunxiang Xu is a professor of computer science
and technology at the University of Electronic Sci-
ence Technology of China. Her research interests in-
clude information security, cloud computing security,
and cryptography. Xu has a PhD from Xidian Univer-
sity. Shes a member of IEEE. Contact her at chxxu@
uestc.edu.cn.
software and theory from UESTC. Hes a member of
IEEE, the China Computer Federation, and the Chi-
na Association for Cryptologic Research. Contact him
at hongweili@uestc.edu.cn.
Xiaohui Liang is an assistant professor in the De-
partment of Computer Science at the University of
Massachusetts, Boston. His research interests include
applied cryptography, and security and privacy issues
for e-healthcare systems, cloud computing, mobile so-
cial networks, and smart grids. Liang has a PhD in
electrical and computer engineering from the Uni-
versity of Waterloo, Canada. Hes a member of IEEE.
Contact him at Xiaohui.Liang@umb.edu.

Hongwei Li is an associate professor in the School

of Computer Science and Engineering at the Uni-
versity of Electronic Science Technology of China Read your subscriptions through
the myCS publications portal at
(UESTC). His research interests include cryptography http://mycs.computer.org.
and the secure smart grid. Li has a PhD in computer

ADVERTISER INFORMATION

Advertising Personnel Southwest, California:

Mike Hughes
Marian Anderson: Sr. Advertising Coordinator Email: mikehughes@computer.org
Email: manderson@computer.org Phone: +1 805 529 6790
Phone: +1 714 816 2139 | Fax: +1 714 821 4010
Southeast:
Sandy Brown: Sr. Business Development Mgr. Heather Buonadies
Email sbrown@computer.org Email: h.buonadies@computer.org
Phone: +1 714 816 2144 | Fax: +1 714 821 4010 Phone: +1 973 304 4123
Fax: +1 973 585 7071
Advertising Sales Representatives (display)
Advertising Sales Representatives (Classified Line)
Central, Northwest, Far East:
Eric Kincaid Heather Buonadies
Email: e.kincaid@computer.org Email: h.buonadies@computer.org
Phone: +1 214 673 3742 Phone: +1 973 304 4123
Fax: +1 888 886 8599 Fax: +1 973 585 7071

Northeast, Midwest, Europe, Middle East: Advertising Sales Representatives (Jobs Board)
Ann & David Schissler
Email: a.schissler@computer.org, d.schissler@computer.org
Phone: +1 508 394 4026 Heather Buonadies
Fax: +1 508 394 1707 Email: h.buonadies@computer.org
Phone: +1 973 304 4123
Fax: +1 973 585 7071

52 I EEE Clo u d Co m p u t i n g w w w.co m p u t er .o rg /clo u d co m p u t i n g

 Focus on
Your Job Search
IEEE Computer Society Jobs helps you easily find
a new job in IT, software development, computer en-
gineering, research, programming, architecture, cloud
computing, consulting, databases, and many other
computer-related areas.
New feature: Find jobs recommending or requiring the
IEEE CS CSDA or CSDP certifications!

Visit www.computer.org/jobs to search technical job

openings, plus internships, from employers worldwide.

http://www.computer.org/jobs

The IEEE Computer Society is a partner in the AIP Career Network, a collection of online job sites for scientists, engineers, and com-
puting professionals. Other partners include Physics Today, the American Association of Physicists in Medicine (AAPM), American
Association of Physics Teachers (AAPT), American Physical Society (APS), AVS Science and Technology, and the Society of Physics
Students (SPS) and Sigma Pi Sigma.
Cloud Security

To Docker or Not
to Docker: A Security
Perspective
Theo Combe, Telecom Paris-Tech
Antony Martin and Roberto Di Pietro, Nokia Bell Labs

Container solutions loud computing is inherently rooted in virtualization

technologies. Recently, new lightweight virtualiza-
such as the popular tion technologies such as containers have become
increasingly popular and are nowadays an essential
Docker environment part of cloud offerings. Containers also tightly inte-
provide more grate into the host operating system, reducing the
software overhead imposed by virtual machines (VMs).1 However, this
flexibility than tighter integration also increases the attack surface, raising security
concerns.
virtual machines Existing work on container securityfocuses mainly on the relation-
ship between the host and the container.25 However, containers are
and offer near- now part of a complex ecosystem, which includes containers and vari-
native performance ous repositories and orchestrators, that is highly automated. In par-
ticular, container solutions embed automated deployment chains6that
in cloud-based are meant to speed up the code deployment processes. These chains
are often composed of third-party elements running on different plat-
infrastructures. forms provided by different providers, raising concerns about code in-
tegrity. This can cause multiple vulnerabilities that an adversary could
However, Docker exploit to penetrate the system.
and its current To the best of our knowledge, container ecosystem security has
yet to be fully investigated, despite being fundamental to container
usage scenarios adoption. Here, we address that gap and focus our investigation on
the Docker ecosystem for three reasons. First, Docker successfully be-
entail security came the reference on both the market of containers and the associ-
ated DevOps ecosystem. In particular, 92 percent of people surveyed
vulnerabilities that by ClusterHQ and DevOps.com are using or planning to use Docker in
must be addressed. a container solution.7 Second, security is the first barrier to container
adoption in the production environment,7 Docker being no exception

54 I E E E C l o u d C o m p u t i n g p u b l i s h e d b y t h e I E E E c o m p u t e r s o cie t y  2325-6095/16/$33.00 2016 IEEE

 App App App App App App App App App

Libs Libs Libs Libs Libs Libs Libs Libs Libs

Guest Guest Guest Guest Guest Guest Cont. Cont. Cont.

OS OS OS OS OS OS

VM VM VM VM VM VM Docker daemon

Hypervisor Host libraries

Hypervisor
Host OS Host OS

Hardware Hardware Hardware

(a) (b) (c)

Figure 1. Comparing various application runtime models: (a) a type 1 hypervisor, (b) a type 2 hypervisor, and (c) a container.

in this. Finally, Docker is already running in some
environments, making it possible to run experi-
ments and explore the practicality of some attacks.
Containerization and Dockerization in a
Growing Ecosystem
Cloud applications have typically leveraged virtu-
alization. However, several factorsincluding ac-
celeration of the development cycle (such as agile
methods and DevOps), an increasingly complex ap-
plication stack (mostly Web services and their frame-
works), and market pressure to densify applications
on servershave triggered the need for a fast, easy-
to-use way of pushing code into production.
Linux Containers
Figure 1 shows how virtualization hypervisors (Fig-
ures 1a and 1b) compare to a container (Figure 1c),
which provides near-bare-metal performance1 and
offers the possibility of seamlessly running mul-
tiple versions of applications on the same machine.
New instances of containers can be created quasi-
instantly to face a customer demand peak, which is
convenient for spawning applications on-demand or
quickly moving a service, such as when implement-
ing network function virtualization (NFV).
Containers have long existed in various forms
that differ by the level of isolation they provide. For
example, Berkeley Software Distribution (BSD) jails
and chroot can be considered an early form of con-
tainer technology. Recent Linux-based container so-
lutions rely on kernel supportthat is, a userspace
library to provide an interface to syscalls and front-
end applications. There are two main kernel imple-
mentations: Linux container (LXC) implementations
using cgroups and namespaces, and the OpenVZ
patch. Table 1 shows the most popular implementa-
tions and their dependences.
Containers can be integrated in a multitenant
environment, thus profiting from resource sharing
to increase average hardware use. This is achieved
by sharing the kernel with the host machine. In-
deed, unlike VMs, containers dont embed their own
kernel, but rather run directly on the host kernel.
This shortens the syscalls execution path by remov-
ing the guest kernel and the virtual hardware layer.
Additionally, containers can share software resourc-
es (such as libraries) with the host, hence avoiding
code duplication. The absence of kernel and some
system libraries make containers very lightweight
(image sizes can shrink to a few megabytes), which
enables a quick boot process.
Docker
As Figure 2 shows, the Docker ecosystem includes
various components. Docker provides a specification

Se p t e m b e r / Oc t o b e r 2 0 1 6 I EEE Clo u d Co m p u t i n g 55
Cloud Security
Table 1. Container solutions.
Base Container Library Kernel dependence
Linux Docker libcontainer cgroups + namespaces + capabilities
containers + kernel version 3.10 or above
(LXC)
LXC liblxc cgroups + namespaces + capabilities
LXD liblxc cgroups + namespaces + capabilities
Rocket AppContainer cgroups + namespaces + capabilities
+ kernel version 3.8 or above
Warden custom tools cgroups + namespaces
OpenVZ OpenVZ libCT Patched kernel
for container images and runtime, including Docker-
files that allow a reproducible building process (Figure
2a). Docker software implements this specification
using the Docker daemon, known as the Docker en-
gine. The repositories include a central repository, the
Docker hub, which lets developers upload and share
their images, along with a trademark and bindings
with third-party applications (Figure 2b). Finally, the
build process fetches code from external repositories
and holds the packages that will be embedded in the
images (Figure 2c). Docker is written in the Go lan-
guage and was first released in March 2013.
Docker specification. The specifications scope is con-
tainer images and runtime. Docker disk images are
composed of a set of layers, along with metadata in the
JavaScript Object Notation (JSON) format. The im-
ages are stored at /var/lib/docker/<driver>/,
where <driver> stands for the storage driver being
usedsuch as advanced multi-layered unification
filesystem (aufs), B-tree file system (Btrfs), virtual
filesystem switch (VFS), device mapper, or OverlayFS.
Each layer contains the filesystem modifications rela-
tive to the previous layer, starting from a base im-
age (typically, a lightweight Linux distribution). This
lightweight Linux distribution organizes the images
in trees; each image has a parent, except for the base
images, which are the roots of the trees. This struc-
ture allows Docker to ship in an image only the modi-
fications specifically related to it.
Docker can build images in two ways. It can
launch a container from an existing image (docker
run), perform modifications and installations inside
the container, and then stop the container and save
its state as a new image (docker commit). This
56 I EEE Clo u d Co m p u t i n g w w w.co m p u t er .o rg /clo u d co m p u t i n g
Other dependencies
iptables, perl, Apparmor, sqlite, Go
Go
LXC, Go
cpio, Go, squashfs, gpg
debootstrap, rake
Specific components: checkpoint/restore
in userspace (CRIU), ploop, Virtuozzo
container memory management daemon
(VCMMD)
process is close to the classical VM installation, but
must be performed at each image rebuild (such as
for updates); because the base image is standard-
ized, the sequence of commands is exactly the same.
To automate this process, Dockerfiles (Figure2a) let
users specify a base image and a sequence of com-
mands to be performed to build the image, along
with other optionssuch as exposed portsspe-
cific to the image. The image is then built with the
docker build command.
Docker internals. Docker containers create a
wrapped, controlled environment on the host ma-
chine in which arbitrary code can be (ideally) run
safely. This isolation is achieved through two main
kernel featureskernel namespaces8 and control
groups (cgroups)that were merged starting from the
Linux kernel version 2.6.24. Namespaces are used to
split the view that processes have of the system. Cur-
rently, the kernel has six different namespacesPID,
IPC, NET, MNT, UTS, and USERthat isolate vari-
ous aspects of the system. Each of these namespaces
has its own kernel internal objects related to its type,
and each gives processes a local instance of some
paths in the /proc and /sys filesystems. The Linux
namespaces isolation role is detailed elsewhere.3 The
cgroups are a kernel mechanism to restrict the re-
source usage of a process or group of processes. Their
goal is to prevent a process from taking all available
resources and starving other processes and contain-
ers on the host. Controlled resources include CPU
shares, RAM, network bandwidth, and disk I/O.
The Docker daemon. The Docker software runs as a
daemon on the host machine. It can launch contain-
ers, control their isolation level (cgroups, namespac- Developer
docker
es, capabilities restrictions, and SELinux/Apparmor machine
Cont. Cont. built
profiles), monitor them to trigger actions (such as re- Dev. Online service
environ-
start), and spawn shells into running containers for Docker
ment Physical machine
daemon
administration purposes. The software can change docker
Host libraries push Commands
iptables rules on the host and create network inter- Host OS
faces. Its also responsible for managing container Hardware
images, including pulling and pushing images on a (a)
remote registry (such as the Docker hub), building docker Image docker
push git Code Image
images from Dockerfiles, and signing images. The push
push
Dockerfile
daemon itself runs as root (with full capabilities) on
the host and is remotely controlled through a Unix External code
repositories
socket. Alternatively, the daemon can listen on a
classical TCP socket. Github, Private repositories External repositories
for example (dependencies, website,
Public repositories for example)
The Docker hub. The Docker hub online repository
lets developers upload their Docker images and lets
Code docker build Code
users download them. Developers can sign up for a github hook
Dockerfile
free account, in which all repositories are public, or
for a paid account, which lets them create private re- Images repositories
positories. Developer repositories are namespaced
that is, their name is developer/repository. Official Private repositories
Private repositories
Alternate
repositories also exist, directly provided by Docker Public repositories registry
Public repositories
Inc.; these are called repository. The Docker dae- Third-party
Docker repositories
mon, hub, and repositories are similar to a package hub
manager, with a local daemon installing software on Official repositories

both the host and the remote repositories. Some of

(b)
these repositories are official, while others are unof- docker pull docker pull
Image Image
ficial and provided by third parties. docker hook
Development
environment
Docker Security Overview
Docker security relies on three factors: isolation Production
of processes at the userspace level managed by the environment
Docker daemon, enforcement of this isolation by the Orchestrator
Docker
kernel, and network operations security. host (Kubernetes,
Cont. Cont. Cont. for example)

Isolation Tasks
Docker daemon Commands
Services
Docker containers rely exclusively on Linux kernel Host libraries
features, including namespaces, cgroups, hardening, Host OS
and capabilities. Namespace isolation and capabili- Hardware
ties drop are enabled by default, but cgroups limita- docker run / ps /
inspect / exec ...
tions arent; they must be enabled on a per-container (c)
basis through -a -c options on container launch.
The default isolation configuration is relatively Figure 2. The Docker ecosystem. (a) Docker specifies container images
strict. The only flaw is that all containers share the and runtime, including Dockerfiles that enable a reproducible building
same network bridge, enabling Address Resolution process. (b) The Docker repositories. (c) The build process. Arrows show
Protocol (ARP) poisoning attacks between contain- the code path and associated commands (docker <action>).
ers on the same host.
However, as we describe in more detail later,
Dockers global security can be lowered by options, This includes options lowering security, such as the
triggered at container launch, that give extended insecure-registry option, which disables the
access on some parts of the host to containers. Ad- Transport Layer Security (TLS) certificate check on
ditionally, security configuration can be set glob- a particular registry. Options that increase security
ally through options passed to the Docker daemon. such as the icc=false parameter, which forbids

Se p t e m b e r / Oc t o b e r 2 0 1 6 I EEE Clo u d Co m p u t i n g 57
Cloud Security
network communications between containers and
mitigates the ARP poisoning attackare available,
but they prevent multicontainer applications from
operating properly, and hence are rarely used.
Host Hardening
Host hardening through Linux kernel security mod-
ules enforces security-related limitation constraints
imposed on containers (such as compromising a
container and escaping to the host operating sys-
tem). Currently SELinux, Apparmor, and Seccomp
are supported with available default profiles. These
profiles are generic, not restrictive. The docker-
default Apparmor profile9 (https://wikitech
.wikimedia.org/wiki/Docker/apparmor), for exam-
ple, allows full access to the filesystem, network,
and all capabilities of Docker containers. Similarly,
the default SELinux policy puts all Docker objects
in the same domain. Therefore, while default hard-
ening protects the host from containers, it doesnt
protect containers from other containers. This secu-
rity aspect can be addressed by writing specific pro-
files that depend individually on the containers.
Network Security
Docker uses network resources for image distribu-
tion and remote control of the Docker daemon.
To distribute images, Docker verifies images
downloaded from a remote repository with a hash
and the connection to the registry is made over TLS
(unless explicitly specified otherwise). Moreover,
the Docker Content Trust architecture now lets de-
velopers sign their images before pushing them to
a repository.10 Content Trust relies on the update
framework (TUF),11 which was specifically designed
to address package manager flaws.12 TUF can recov-
er from a key compromise, mitigate replay attacks by
embedding expiration timestamps in signed images,
and so on. The tradeoff is complex key management;
TUF actually implements a public-key infrastruc-
ture in which each developer owns a root key (of-
fline key) that is used to sign signing keys that are
used to sign Docker images.
The Docker daemon is remote-controlled through
a socket, making it possible to perform any Docker
command from another host. By default, the sock-
et used to control the daemon is a Unix socket,
located at /var/run/docker.sock and owned
by root:docker, but it can be changed to a TCP
socket. Access to this socket lets attackers pull and
run any container in privileged mode, thereby giv-
ing them root access to the host. In case of a Unix
socket, a user member of the docker group can gain
root privileges; when a TCP socket is used, any con-
58 I EEE Clo u d Co m p u t i n g w w w.co m p u t er .o rg /clo u d co m p u t i n g
nection to this socket can give root privileges on the
host. Therefore, the connection must be secured with
TLS (tlsverify), which enables both encryption
and authentication of the two sides of the connection
(and requires additional certificate management).
Docker Usages: Security Challenges
Most of the security discussions about containers
compare them to VMs, thus assuming both tech-
nologies are equivalent in terms of design. Although
this is the aim of some container technologies (such
as OpenVZ, which is used to spawn virtual private
servers), recent lightweight container solutions such
as Docker were designed to achieve completely dif-
ferent objectives than those of VMs. Therefore, its
important to develop Dockers typical usages to dis-
cuss their security implications and how they affect
Dockers security.
Docker Usages
We can distinguish three types of Docker usages.
Recommended usages are those that Docker was
designed for, as explained in the official documenta-
tion. Docker developers recommend a microservices
approach13 that is, a container must host a single
service, in a single process or in a daemon spawning
children. Therefore, a Docker container isnt consid-
ered a VM: theres no package manager, no init pro-
cess, no sshd to manage it. All administration tasks
(container stop, restart, backups, updates, builds,
and so on) must be performed via the host machine,
which implies that the legitimate containers admin
has root access to the host.
Docker developers also recommend a repro-
ducible and automated deployment of applications.
Docker images should be built anywhere through a
generic build file (Dockerfile) which specifies the
steps to build the image from a base image. This ge-
neric way of building images makes the process and
the resulting images almost host-agnostic, depending
only on the kernel and not on the installed libraries.
Widespread usages include common usages of
Docker by application developers and system ad-
ministrators. Some system administrators or devel-
opers use Docker as a way to ship complete virtual
environments and update them regularly, turning
their containers into VMs. Although this is conve-
nient because it limits system administration tasks
to the bare minimum (such as docker pull), as
we describe later, it has several security implica-
tions. With containers embedding enough software
to run a full system (logging daemon, ssh server,
and even sometimes an init process), its tempting
to perform administration tasks from within the
container, which is completely opposed to Dockers Indirect adversaries have the same capabilities
design. Indeed, some of these administration tasks as direct ones, but they leverage the Docker ecosys-
require root access on the container, while other ad- tem (such as the code and images repositories) to
ministration actionssuch as mounting a volume reach the production environment.
in a containercould require extra capabilities that Depending on the attack phase, we identified
Docker drops by default. the following targets: containers, host operating
Platform as a service (PaaS) usages are guided system, collocated containers, code repositories,
by PaaS implementations to cope with security and images repositories, and the management network.
infrastructure integration issues. As of the end of MITREs Common Vulnerabilities and Exposures
2015, the main PaaSs integrated Docker. We focus (CVE) records illustrate that these are relevant tar-
here on Amazon Web Services and Google Container gets. Vulnerabilities found in Docker and the libcon-
Engine, the two market leaders that we experimented tainer mostly concern filesystem isolation: chroot
on. Both solutions provide similar approaches: a VM escapes (CVE-2014-9357, CVE-2015-3627), path
or cluster of VMs is created, with an orchestrator traversals (CVE-2014-6407, CVE-2014-9356, and
tool available to manage the containers
inside the VMs. In this model, the con-
tainers admin has full rights on the or-
chestrator. The microservices approach The Docker attack surface encompasses
promoted by DevOps and Docker cur-
rently requires manual configuration to the whole deployment toolchain, from
launch appropriate images on appropri-
ate nodes. This task is automated by or- the build to the execution of the images.
chestrators that manage clusters of VMs,
which themselves run on multiple physi-
cal hosts.
CVE-2014-9358), and access to special file systems
Adversary Model on the host (CVE-2015-3630). These specific vul-
Given the ecosystem and usages description, we nerabilities are all patched as of Docker 1.6.2. Be-
consider two main categories of adversaries: direct cause container processes often run with user ID 0,
and indirect. they have read and write access on the whole host
Direct adversaries can sniff, block, inject, or filesystem when they escape, which lets them over-
modify network and system communications, and write host binaries, leading to a delayed arbitrary
they directly target the production machines. Lo- code execution with root privileges.
cally or remotely, direct adversaries can compromise To subvert a Dockerized environment, we con-
several system components: sider a subset of all the potential attack vectorsthe
Docker containers, code repositories, and images re-
In-production containers. With containers from positoriesprimarily because theyre associated with
an Internet-facing container service, for exam- publicly available services and interfaces. Other at-
ple, attackers can gain root privileges on a re- tack vectors might include the host operating system,
lated container. Then, from the compromised management network, or physical access to systems.
container, they can make a denial-of-service
(DoS) attack on containers located on the same Vulnerabilities Affecting Docker Usages
host operating system. The Docker attack surface encompasses the whole
In-production host operating system. From a deployment toolchain proposed by Docker, from the
compromised container, for example, attackers build to the execution of the images, including im-
can gain access to critical host operating system age conception, the image distribution process, au-
filesthat is, launch a container escape attack. tomated builds, image signature, host configuration,
In-production Docker daemons. In this case, for and third-party components.
example, attackers might lower the default se-
curity parameters to launch Docker containers Insecure local configuration. Dockers default con-
from a compromised host operating system. figuration on local systems following recommended
The production network. From a compromised usages is relatively secure as it provides isolation
host operating system, attackers can redirect between containers and restricts containers access
network traffic and so on. to the host. Assuming these isolation mechanisms

Se p t e m b e r / Oc t o b e r 2 0 1 6 I EEE Clo u d Co m p u t i n g 59
Cloud Security
are working as expected in both the recommended
and PaaS usagesthat is, there are no implementa-
tion vulnerabilities or CVEsa privilege boundary
between the containers and the host machine has
been designed. Technical controls supporting the
boundary include the isolation of processes through
namespaces, resources management through
cgroups, and (by default) limited communication ca-
pabilities between the containers and the host.
In contrast, widespread usages take advantage
of optionsgiven either to the Docker daemon on
startup or to the command launching a container
that give containers extended access to the host.
When used with untrusted containers, these op-
tions trigger many security concerns, including the
following:
options giving extended access to the host to
containers (net=host, uts=host, privi-
leged, and so on);
the mounting of sensitive host directories in
containers;
TLS configuration of remote image registries;
permissions on the Docker control socket; and
cgroups activation (disabled by default).
For instance, when given the option net=host
at container launch, Docker doesnt place the con-
tainer into a separate NET namespace; it therefore
gives the container full access to the hosts network
stack (enabling network sniffing, reconfiguration,
and so on). The option uts=host lets the contain-
er in the same UTS namespace as the host, which
lets the container see and change the hosts name
and domain. The option cap-add=<CAP> gives
the container the specified capability, thus making
it potentially more harmful to the host. With cap-
add=SYS_ADMIN, a container can, for example, re-
mount /proc and /sys subdirectories in read/write
mode and change the hosts kernel parameters, lead-
ing to potential vulnerabilities, data leakage, or DoS.
Along with these runtime container options,
several settings on the host can influence potential
attacks. Even basic properties can at a minimum
trigger DoS. For instance, when using some storage
drivers (aufs), Docker doesnt limit containers disk
usage. A container with a storage volume can fill up
this volume and affect other containers on the same
hostor even the host itselfif the Docker storage
located at /var/lib/docker isnt mounted on a
separate partition.
As mentioned earlier, whatever the usages are,
containers are an attack vector and therefore rep-
resent a potential threat for the host. This is even
60 I EEE Clo u d Co m p u t i n g w w w.co m p u t er .o rg /clo u d co m p u t i n g
more relevant in widespread usages, where contain-
ers are used as VMs and thus have a bigger attack
surface than microservice containers. They also
have more vulnerabilities, leading to attacks such as
container escapes.
Weak local access control. Beyond the kernel
namespaces, cgroups, Docker dropping capabilities,
and mount restrictions, mandatory access control
(MAC) enforces constraints if the normal execution
flow isnt respected. This approach is visible in the
docker-default Apparmor policy. However, the
MAC profiles for containers have room for improve-
ment. In particular, Apparmor profiles typically
behave as whitelists,14 explicitly identifying which
resources any process can access while denying any
other access when the profile is in enforce mode.
However, the docker-default profile installed
with the docker.io package gives containers com-
plete access on network devices and filesystems
with a full set of capabilities, and contains a small
list of deny directives, which constitute a de facto
blacklist.
These vulnerabilities are relevant to all usages
and could lead to the attacks mentioned earlier,
such as DoS or container escapes.
Image distribution vulnerabilities. The distribution
of images through the Docker hub and other regis-
tries in the Docker ecosystem is a source of vulner-
abilities. Because these vulnerabilities are similar to
classical package managers,12 we consider only the
automated deployment pipeline perspective here.
Automated builds and webhooks proposed by
the Docker hub are key elements in the image distri-
bution process. They lead to a pipeline in which each
element has full access to the code that will end up
in production, and are increasingly hosted in the
cloud. For instance, to automate this deployment,
Docker proposes automated builds on the Docker
hub, triggered by an event from an external code re-
pository (such as github). Docker then proposes to
send an HTTP request to a Docker host reachable
on the Internet to notify it that a new image is avail-
able. This triggers an image pull and a container re-
start on the new image (through Docker hooks; see
https://docs.docker.com/docker-hub/webhooks).
In this deployment pipeline, a commit on github
will trigger a build of a new image and automatically
launch it into production. Optional test steps can be
added before production, which might themselves
be hosted at yet another provider. In this case, the
Docker hub makes a first call to a test machine that
will then pull the image, run the tests, and send re-
sults to the Docker hub using a callback URL. The
build process itself often downloads dependencies
from other third-party repositories, sometimes over
an insecure channel prone to tampering.
This setup adds several external intermediary
steps to the code path, each of which has its own
authentication and attack surface, increasing the
global attack surface.
For instance, we had the intuition that a com-
promised github account could lead to the execu-
tion of malicious code on a large set of production
machines within minutes. We therefore tested a
scenario that included a Docker hub account, a
github account, a development machine, and a
production machine. The assumption was that ad-
versaries would use the Docker ecosystem to put a
backdoored Docker container in production. More
precisely, we assumed that adversaries had success-
fully compromised some code on the code repository
(for instance, via a successful phishing attack).
Because of network restrictions (our server was
behind a corporate proxy that did not allow us to be
reached directly from the Internet on a public IP ad-
dress and a port) our servers couldnt be reached by
webhooks, so we wrote a script to monitor both our
repository on the Docker hub and downloads of new
images. Our initial intuition was confirmed: the ad-
versaries code was put in production five-and-a-half
minutes after their commit on github. Its worth not-
ing that this attack can scale to an arbitrary number
of machines watching the same Docker hub reposi-
tory. Given space limitations, we report more de-
tailed results elsewhere.15
Although compromising a code repository is
independent of Docker, automatically pushing it in
production dramatically increases the number of
compromised machines, even if the malicious code
is removed within minutes. Compromise could also
happen at the Docker hub account level with the
same consequences. Account hijacking isnt a new
problem, but it should be an increasing concern with
the multiplication of accounts at different providers.
Moreover, although the code path is usually
secured using TLS communications (and always is
with Docker), its not the case with API calls that
trigger builds and callbacks. Tampering with these
data can lead to erroneous test results, unwanted
restarts of containers, and so on. Additionally, such
a setup isnt compatible with the Content Trust
scheme, because code is processed by external enti-
ties between the developer and the production envi-
ronment. Content Trust provides an environment in
which a single entity is trusted (the person or orga-
nization that signed the images), while in this case
Se p t e m b e r / Oc t o b e r 2 0 1 6 I EEE Clo u d Co m p u t i n g
trust is split across external entities, each of which
is capable of compromising the images.
These vulnerabilities are especially relevant in
the recommended and PaaS usages, which aim at an
extensive use of automation at all layers to deliver
shorter development cycles and continuous delivery.
n orchestrator could solve some of the secu-
rity issues raised here, helping limit misuses
of Docker through higher levels of abstractionin-
cluding tasks, replication controllers, and remote
persistent storagethat completely remove host
dependence and thus enable better isolation. Or-
chestrator also brings automation as a key enabler
to repeatable, predictable, auditable security and
continuously improved security. We are currently
investigating orchestrator security issues through
experiments.
Acknowledgments
We thank the anonymous reviewers for their useful
comments.
References
1. M.G. Xavier et al., Performance Evaluation of
Container-Based Virtualization for High Per-
formance Computing Environments, Proc.
21st Euromicro Intl Conf. Parallel, Distributed,
and Network-Based Processing (PDP 13), 2013,
pp.233240.
2. T. Bui, Analysis of Docker Security, 2015,
arXiv:1501.02967v1.
3. E.Reshetova et al., Security of OS-Level Virtu-
alization Technologies, Proc. Nordic Conf. Se-
cure IT Systems, 2014, pp.7793.
4. R.Di Pietro and F.Lombardi, Security for Cloud
Computing, Artec House, 2015.
5. F. Lombardi and R. Di Pietro, Virtualization
and Cloud Security: Benefits, Caveats, and Fu-
ture Developments, Cloud Computing, Z.Mah-
mood, ed., Springer Intl Publishing, 2014,
pp.237255.
6. Docker, Automated Builds on Docker Hub,
Docker User Guide, 2016; https://docs.docker
.com/docker-hub/builds.
7. ClusterHQ and DevOps.com, The Current State
of Container Usage: Identifying and Eliminating
Barriers to Adoption, survey, June 2015; https://
clusterhq.com/assets/pdfs/state-of-container
-usage-june-2015.pdf.
8. E.W. Biederman, Multiple Instances of the
Global Linux Namespaces, Proc. Linux Symp.,
vol.1, 2006, pp.101112.
61
Cloud Security
10. Docker, Content Trust in Docker, Docker User
Guide, 2016; https://docs.docker.com/engine/
security/trust/content_trust.
11. J. Samuel et al., Survivable Key Compromise
in Software Update Systems, Proc. 17th ACM
Conf. Computer and Comm. Security (CCS 10),
2010, pp.6172.
12. J.Cappos et al., A Look in the Mirror: Attacks
on Package Managers, Proc. 15th ACM Conf.
Computer and Comm. Security, P.Ning, P.F. Sy-
verson, and S.Jha, eds., 2008, pp.565574.
13. Docker, Best Practices for Writing Dockerfiles,
Docker User Guide, 2016; https://docs.docker
.com/engine/userguide/eng-image/dockerfile
_best-practices.
14. Novell, Novell AppArmor Administration Guide,
Oct. 2007; w w w.suse.com/documentation/
apparmor/pdfdoc/ book _apparmor21_admin/
book_apparmor21_admin.pdf.
15. T.Combe, A.Martin, and R.Di Pietro, Contain-
ers: Vulnerability Analysis, tech. report, Nokia
Bell Labs; http://ricerca.mat.uniroma3.it/users/
dipietro/containers_security.pdf.
Theo Combe is a graduate student at the Ecole
Polytechnique, France, and is pursuing a double de-
gree in networks and cybersecurity at Telecom Paris-
Tech. His research interests include networked systems
PURPOSE: The IEEE Computer Society is the worlds largest association of computing
professionals and is the leading provider of technical information in the field.
MEMBERSHIP: Members receive the monthly magazine Computer, discounts, and
opportunities to serve (all activities are led by volunteer members). Membership is open to
all IEEE members, affiliate society members, and others interested in the computer field.
OMBUDSMAN: Email ombudsman@computer.org.
COMPUTER SOCIETY WEBSITE: www.computer.org
Next Board Meeting: 1314 November 2016, New Brunswick, NJ, USA
EXECUTIVE COMMITTEE
President: Roger U. Fujii
President-Elect: Jean-Luc Gaudiot; Past President: Thomas M. Conte;
Secretary: Gregory T. Byrd; Treasurer: Forrest Shull; VP, Professional and Educational
Activities: Andy T. Chen; VP, Member & Geographic Activities: Nita K. Patel;
VP, Publications: David S. Ebert; VP, Standards Activities: Mark Paulk;
VP, Technical & Conference Activities: Hausi A. Mller; 2016 IEEE Director & Delegate
Division VIII: John W. Walz; 2016 IEEE Director & Delegate Division V: Harold Javid;
2017 IEEE Director-Elect & Delegate Division VIII: Dejan S. Milojii
BOARD OF GOVERNORS
Term Expriring 2016: David A. Bader, Pierre Bourque, Dennis J. Frailey, Jill I. Gostin,
Atsuhiro Goto, Rob Reilly, Christina M. Schober
Term Expiring 2017: David Lomet, Ming C. Lin, Gregory T. Byrd, Alfredo Benso,
Forrest Shull, Fabrizio Lombardi, Hausi A. Mller
Term Expiring 2018: Ann DeMarle, Fred Douglis, Vladimir Getov, Bruce M. McMillin,
Cecilia Metra, Kunio Uchiyama, Stefano Zanero
62 I EEE Clo u d Co m p u t i n g w w w.co m p u t er .o rg /clo u d co m p u t i n g
security, telecommunications, and programming. Con-
tact him at theo-nokia@sutell.fr.
Antony Martin is a security analyst and mem-
ber of the technical staff in the security department at
Nokia Bell Labs, Nozay, France. His research interests
include network security, virtualization, cloud com-
puting, and network function virtualization. Martin
has an engineering degree in telecommunications
from Telecom Lille engineering school and holds a
number of technical certifications. Contact him at
antonymartin.pro@gmail.com.
Roberto Di Pietro is security research global
head at Nokia Bell Labs, Paris-Saclay, France, and a
part-time professor of computer science (security) at
the University of Padova, Italy. His research interests
include security, privacy, distributed systems, com-
puter forensics, and analytics. Di Pietro has a PhD
in computer science from the University of Roma La
Sapienza. Contact him at roberto.di_ pietro@nokia
-bell-labs.com.
Read your subscriptions through
the myCS publications portal at
http://mycs.computer.org.
EXECUTIVE STAFF
Executive Director: Angela R. Burgess; Director, Governance & Associate Executive
Director: Anne Marie Kelly; Director, Finance & Accounting: Sunny Hwang;
Director, Information Technology & Services: Sumit Kacker; Director, Membership
Development: Eric Berkowitz; Director, Products & Services: Evan M. Butterfield;
Director, Sales & Marketing: Chris Jensen
COMPUTER SOCIETY OFFICES
Washington, D.C.: 2001 L St., Ste. 700, Washington, D.C. 20036-4928
Phone: +1 202 371 0101 Fax: +1 202 728 9614 Email: hq.ofc@computer.org
Los Alamitos: 10662 Los Vaqueros Circle, Los Alamitos, CA 90720
Phone: +1 714 821 8380 Email: help@computer.org
MEMBERSHIP & PUBLICATION ORDERS
Phone: +1 800 272 6657 Fax: +1 714 821 4641 Email: help@computer.org
Asia/Pacific: Watanabe Building, 1-4-2 Minami-Aoyama, Minato-ku, Tokyo 107-0062,
Japan Phone: +81 3 3408 3118 Fax: +81 3 3408 3553 Email: tokyo.ofc@computer.org
IEEE BOARD OF DIRECTORS
President & CEO: Barry L. Shoop; President-Elect: Karen Bartleson; Past President:
Howard E. Michel; Secretary: Parviz Famouri; Treasurer: Jerry L. Hudgins; Director
& President, IEEE-USA: Peter Alan Eckstein; Director & President, Standards
Association: Bruce P. Kraemer; Director & VP, Educational Activities: S.K. Ramesh;
Director & VP, Membership and Geographic Activities: Wai-Choong (Lawrence) Wong;
Director & VP, Publication Services and Products: Sheila Hemami; Director & VP,
Technical Activities: Jose M.F. Moura; Director & Delegate Division V: Harold Javid;
Director & Delegate Division VIII: John W. Walz
revised 10 June 2016
IEEE Cloud Computing
Call for Papers

A
lthough cloud technologies have been advanced and adopted at an astonishing
pace, much work remains. IEEE Cloud Computing seeks to foster the evolution of
cloud computing and provide a forum for reporting original research, exchanging
experiences, and developing best practices.

IEEE Cloud Computing magazine seeks accessible, useful papers on the latest peer-reviewed
developments in cloud computing. Topics include, but arent limited to:

Cloud architectures (delivery models and deployments),

Cloud management (balancing automation and robustness with monitoring and
maintenance),
Cloud security and privacy (issues stemming from technology, process and governance,
international law, and legal frameworks),
Cloud services (cloud services drive and are driven by consumer demand; as markets
change, so do the types of services being offered),
Cloud experiences and adoption (deployment scenarios and consumer expectations),
Cloud and adjacent technology trends (exploring trends in the market and impacts on
and influences of cloud computing),
Cloud economics (direct and indirect costs of cloud computing on the consumer;
sustainable models for providers),
Cloud standardization and compliance (facilitating the standardization of cloud tech and
test suites for compliance), and
Cloud governance (transparency of processes, legal frameworks, and consumer
monitoring and reporting).

Submissions will be subject to IEEE Cloud Computing magazines peer-review process.

Articles should be at most 6,000 words, with a maximum of 15 references, and should be
understandable to a broad audience of people interested in cloud computing, big data, and
related application areas. The writing style should be down to earth, practical, and original.

All accepted articles will be edited according to the IEEE Computer Society style guide.
Submit your papers through Manuscript Central at https://mc.manuscriptcentral.com/ccm-cs.

If you have any questions, feel free to email lead editor Brian Brannon at bbrannon@computer.org.

www.computer.org/cloudcomputing
Cloud Security

User-Centric Security
and Dependability
in the Clouds-of-
Clouds
Marc Lacoste, Orange Labs
Markus Miettinen, Technische Universitt Darmstadt
Nuno Neves and Fernando M.V. Ramos, University of Lisbon
Marko Vukoli, IBM Research
Fabien Charmet and Reda Yaich, Institut Mines-Telecom
Krzysztof Oborzyski and Gitesh Vernekar, Philips Healthcare
Paulo Sousa, Maxdata Software

Secure Supercloud computing aims to provide

security and dependability management of
distributed clouds. This approach is both user-centric
and self-managed, enabling users to achieve provider
independence for security management.
64 I E E E C l o u d C o m p u t i n g p u b l i s h e d b y t h e I E E E c o m p u t e r s o cie t y  2325-6095/16/$33.00 2016 IEEE
 he high maintenance costs of
private datacenters and disaster-
recovery requirements are causing
cloud architectures to go distributed.
Virtualization is expanding out-
side a single datacenter for compute,
network, storage, and devices. Resource-specialized
clouds are becoming federated, evolving from cen-
tralized to fully distributed infrastructures across
heterogeneous resourcesa cloud-of-cloudsand
away from the datacenter to the edge.1,2
These new architecture paradigms present key
benefits:
better user performance (for example, low-
er end-to-end latency) due to fine-grained
geod istribution,
lower costs by choosing best-of-breed cloud pro-
viders in terms of pricing model,3 and
improved resilience to avoid wide-area outages
due to single points of failure.
Nevertheless, distributed cloud computing raises
several concerns,4 mainly due to these systems high
complexity and the current lack of interoperabil-
ity between heterogeneous, often proprietary, infra
structure technologies.
In practice, distributed cloud computing has re-
mained highly provider-centric, and multicloud in-
tegration remains a challenge. Adoption also suffers
from vendor lock-in, with services tightly coupled
to providers. Lack of interoperability stems mainly
from the heterogeneity of technologies (for example,
different hypervisors), and from service-resources
mappings that are incompatible across providers,
hampering, for instance, uniformity in service-level
agreements (SLAs). User control is also limited by
monolithic infrastructures, preventing fine-grained
cloud customization by the customer (for example,
hypervisors hide specific hardware capabilities).
Multicloud infrastructures also raise several
security and dependability challenges. First, infra-
structure layers, which include customer virtual
machines (VMs), provider hypervisors, and services,
are extremely vulnerable to attacks, in part due to
new virtualization technologies,5 so the infrastruc-
tures cant be trusted. Second, interoperability and
unified control of security across providers is mostly
Se p t e m b e r / Oc t o b e r 2 0 1 6 I EEE Clo u d Co m p u t i n g
absent. Policy heterogeneity among providers fa-
cilitates the introduction of more vulnerabilities be-
cause of mismatching APIs and workflows. Finally,
security administration challenges for such complex
infrastructures clearly prohibit a manual approach.
Automation of security management is required, but
still lacking, in the multicloud.
In todays provider-centric clouds, service speci
fication, security, dependability, pricing, and SLAs
are beyond users influence. To tackle the security
and dependability challenges in a multicloud, we
need new infrastructure management paradigms
that are both user-centric and self-managed. The
former means enabling self-service of cloud-of-
clouds, where customers define their own protec-
tion requirements and can avoid technology and
vendor lock-ins. The latter means reducing the ad-
ministration complexity of cloud-of-clouds through
automation techniques.
Secure Supercloud Computing
This article introduces the notion of Supercloud, a
new architectural concept that follows the vision of
user-centric distributed cloud security and depend-
ability management.6 Supercloud can be understood
as a security distribution layer, providing an end-
to-end interface between user-centric and provider-
centric views of multiple clouds.
Supercloud deploys several user-centric clouds
(or U-Clouds). A U-Cloud is a set of computation,
data storage, and communication services that lets
individual Supercloud users run their applications
and services over a distributed cloud. U-Clouds can
be implemented on top of resources from several
providers. However, strict U-Cloud is guaranteed us-
ing data encryption and dedicated U-Cloud-specific
VMs for computation.
Supercloud addresses the interoperability chal-
lenge by providing a resource abstraction layer span-
ning multiple cloud providers, decoupling resource
production by cloud providers from their consumption
by users. It also addresses the control challenge by
enabling customers to deploy clouds with self-service
security, ranging from software as a service (SaaS) to
full infrastructure as a service (IaaS), independent of
the underlying providers. In addition, it offers unified
control for automated management of security and re-
silience across different clouds.
65
Cloud Security
U-Cloud of U-Cloud of U-Cloud of
customer 1 customer 2 customer 3
Supercloud
users
Supercloud
Supercloud providers
Supercloud hypervisor
Traditional
cloud providers
Figure 1. The Supercloud concept, which includes users, Supercloud
providers, and traditional cloud providers. Cloud resource consumption
(by users) is separated from cloud resource production (by cloud
providers) thanks to the Supercloud layer, thus enabling it to overcome
vendor lock-in.
This approach has several benefits. First,
independence from the provider means lower infra-
structure operation overhead and faster service de-
ployment, but also increased homogeneity. Second,
increased customizability can also be expected, as the
customer can choose which virtualized services (such
as for security) to deploy, resulting in fully la carte
clouds. Third, it can create new business opportuni-
ties and ecosystems.7
In a nutshell, Supercloud is a provider-agnostic
distributed virtualization infrastructure for run-
ning U-Clouds, leveraging compute, data, and net-
work resources from both public cloud providers and
private cloud infrastructures (see Figure 1). This
heterogeneity impacts the level of infrastructure vis-
ibility and control that can be achieved for services
running in U-Clouds.
On one end, public clouds operated by commer-
cial cloud service providers (CSPs) give only limited
visibility and control over the hypervisor and the
network. Public CSPs are typically big players, of-
fering commercial cloud services to their customers
at massive scale, allowing them to take advantage
of cost savings and elastic resources. They provide
well-defined high-level service APIs with few pos-
sibilities for individual customers to customize the
deployment details of their cloud service instances.
On the other end, in private clouds, where the
datacenter belongs to the user, full infrastructure
66 I EEE Clo u d Co m p u t i n g w w w.co m p u t er .o rg /clo u d co m p u t i n g
access can be reached. Private CSPs are typically
entities such as a corporations IT department,
supporting tailored cloud services for its own orga-
nization. These services arent limited to a narrow
service API, but can also have lower-level control
over specific deployment details (of the computa-
tional, storage, and networking resources). This
enables the U-Cloud to provide user-centric system
services (USS), extending user control into the lower
layers of the infrastructure to enforce flexible, se-
cure, and dependable computing behaviors (for ex-
ample, firewalling, introspection, or live migration).
Supercloud is a step away from provider-centric
cloud interoperability approaches (such as hybrid/
federated clouds) that rely on business, interface,
or protocol agreements between providers. The Su-
percloud approach is closer to customer-centric
solutions (such as multicloud and broker-based ag-
gregation), but focuses on security: interoperability
is transparent to providers, using an adaptation layer
or third-party operation. (See the sidebar for a dis-
cussion of other work in this area.)
Requirements
To meet these challenges, the Supercloud architec-
ture should address the following objectives:
Self-service security: Users should be able to
specify their own protection requirements and
manage the corresponding security and privacy
policies autonomously, to control their resourc-
es security in a fine-grained manner.
Self-managed security: The architecture should
automatically and seamlessly manage the dis-
tributed clouds security over compute, storage,
and network layers, and across provider domains
to ensure compliance with user-defined security
policies.
End-to-end security: The architecture should
guarantee SLAs (for example, for isolation)
for multiple compute clouds, data protection
in a multiprovider setting, and secure network
interconnection.
Resilience: Resource management should pro-
vide robust composition of provider-agnostic
resources, leveraging primitives from multiple
providers.
This leads to the following requirements.
First, the Supercloud architecture must enable
provider independence and isolation. It should of-
fer a distributed cloud infrastructure that lets users
deploy cloud applications and services in specific
cloud instances (that is, U-Clouds) in a transpar-
 Related Work in Multicloud Security
any distributed virtualization infrastructures
(for example, microhypervisors, nested virtu-
alization, container platforms, and library operating
systems), isolation and trust management technolo-
gies, and protection automation techniques have
tackled security challenges related to multiprovider
interoperability and vulnerable software layers, but
without meeting requirements for user control, low
attack surface, interoperability, and legacy compat-
ibility. The Supercloud hybrid virtualization archi-
tecture enables flexible but efficient user-centric
tradeoffs in terms of both interoperability and secu-
rity for the multicloud.
Many solutions, such as Google Drive and
Dropbox, allow users to store their own data on the
cloud. However, most dont permit user-centric data
encryption. In current cloud-based data storage
solutions, no commercial product uses advanced
cryptographic tools for data confidentiality protec-
tion, such as those we propose to use in Supercloud.
Most major infrastructure-as-a-service providers offer
replication solutions across multiple datacenters to
support dependability; however, this remains limited
to proprietary protocols and single administrative
domains.
Software-defined networking-based virtualiza-
tion solutions allow cloud providers to offer complete
network virtualization.1 They give tenants the free-
dom to specify their network topologies and address-
ing schemes, while guaranteeing the required level of
isolation. These platforms, however, have been tar-
geting the datacenter of a single cloud provider with
full control over the infrastructure. In Supercloud, we
extend this concept, supporting the creation of virtual
networks spanning multiple datacenters that might
belong to distinct cloud providers, while including
private facilities owned by the tenant. The novelty of
our solution arises mainly from tackling the challeng-
es of using multiple clouds, including public clouds
on which we have very limited control.
Reference
1. T. Koponen et al., Network Virtualization in Mult-
itenant Datacenters, Proc. 11th USENIX Symp. Net-
worked Systems Design and Implementation (NSDI),
2014, pp. 203216.

ent and user-configurable manner. Individual U-
Clouds must be strictly separated, preventing, for
instance, misbehaving U-Clouds from impacting
other U-Clouds.
The architecture must also support interoper-
ability at the infrastructure and platform levels. It
should support a distributed cloud with flexibility and
control levels similar to those in a single-provider sce-
nariofor example, in terms of usage or migration of
resources across providers. In particular, it should en-
able the deployment of legacy applications and man-
agement tools in the distributed cloud infrastructure.
Third, it should enable user-controlled security.
It should allow users to define fine-grained security
settings to control the protection level of their cloud
resources. For instance, to meet legal requirements
that prohibit transfer of particular data types across
jurisdictional boundaries, users might need to con-
trol where their U-Cloud data is physically stored
and processed. It must also protect user privacy by
preventing cloud providers from accessing user data
without the users explicit consent.
Finally, the architecture should guarantee integ-
rity and availability of services and data. It should
allow specification and enforcement of measures
related to integrity, redundancy, and disaster recov-
ery of data resources as part of a user-provider SLA.
Performance guarantees might also be required,
namely on response times for critical accesses to
some data resources.
System Architecture
We now describe the architecture of the Super-
cloud, both statically (that is, its components) and
dynamically (that is, how these components interact
to guarantee overall security).
Static Architecture
The Supercloud architecture allows customers to
instantiate U-Clouds that run on the underlying

Se p t e m b e r / Oc t o b e r 2 0 1 6 I EEE Clo u d Co m p u t i n g 67
Cloud Security
Compute plane
management
Security
Data plane
Network plane
Network Compute Data Network Compute Data Network Compute Data
Provider 1 Provider 2 Provider n
Figure 2. High-level overview of the Supercloud architecture, including
compute, data, and network planes, and security management framework.
infrastructure. Figure 2 shows the three abstrac-
tion planes, each addressing a particular aspect of
the Supercloud system. Each plane is realized with
resources from the underlying CSPs.
The compute plane enables users to instantiate
computational nodes regardless of physical serv-
ers hosting computations. The data plane realizes
an abstract cloud data storage service transparent
to providers and data resources providing the physi-
cal storage space. The network plane provides the
connectivity between computational and storage
resources regardless of the networking infrastruc-
ture realizing physical connectivity between serv-
ers hosting computational nodes and data storage.
A security management framework provides fine-
grained control to users over protection of any com-
putational, data, and networking resources in the
abstraction planes.
This layered design minimizes interface complex-
ity between planes, clearly defining interdependences
between architectural components. Users can deploy
computational nodes and storage resources in the Su-
percloud system easily and flexibly, regardless of the
specific technical requirements of individual CSPs
resource platforms: orchestration of resources for
computation, storage and networking is handled by
respective abstraction planes.
Interplay between the three planes allows flex-
ible and efficient attack mitigation. A key risk of
federated clouds is that a malicious component be
present in a U-Cloud, considered as a service com-
position.8 Each plane provides relevant counter-
measures: enforcing VM isolation, attesting service
trustworthiness, guaranteeing data availability, or
sanitizing the network environment. Such mecha-
nisms can be orchestrated with the security man-
agement plane to prepare and enforce a relevant
security response.
68 I EEE Clo u d Co m p u t i n g w w w.co m p u t er .o rg /clo u d co m p u t i n g
Compute plane and security self-management.
Figure 3a shows a simplified computing view of two
U-Clouds: one (U-Cloud A) spans different pro-
viders, while another (U-Cloud B) is confined to a
single provider.
The virtualization infrastructure is a distribut-
ed abstraction layer for computing resources across
multiple providers. Nested virtualization is a core
U-Cloud technology because it offers interoperabil-
ity and security benefits to guarantee VM protection
despite untrusted virtualization layers.9 The pro-
vider controls the lower virtualization layer, called
L0. Public clouds usually run general-purpose hy-
pervisors (for example, Xen for Amazon). In private
clouds, more modular hypervisors enable users to
take control on a part of L0 in the form of infrastruc-
ture services for deep, fine-grained customization
of U-Cloud security. The upper virtualization layer,
called L1, provides the necessary facilities for users
to instantiate execution environments forming layer
L2, using VMs or containers that are under users
control. A horizontal orchestration component typi-
cally realizes distributed execution or migration of
L2 environments connecting multiple L1 instances.
Supercloud mainly addresses security at the in-
frastructure level, for example, to guarantee isola-
tion among system computation units such as VMs
or containers and enforce a U-Cloud boundary. Se-
curity partitioning of applications across clouds is
also important to users,10 and can be achieved on
top of the Supercloud layer as in single clouds. Pro-
vider heterogeneity is hidden within the U-Cloud,
already a secure, distributed environment for appli-
cation deployment.
The self-management infrastructure implements
autonomic configuration and management of securi-
ty aspects for the distributed cloud. Such automation
means simpler, faster, and more efficient detection
and response to threats, minimizing overall human
intervention. U-Cloud-specific components also let
users control their U-Clouds security settings. An
overall component arbitrates between such settings
and provider security requirements. The security re-
sponse to a threat is elaborated by orchestration of
multiple autonomic security loops across infrastruc-
ture layers and providers. Other services include
flexible isolation, trust management, configuration
compliance for auditability, and authorization.
Data plane. Figure 3b shows several types of stor-
age entities in the data plane. Clients represent
users of the Supercloud storage infrastructure. Or-
dinary clients interact transparently with the data
plane via storage proxies. This requires minimal
 U-Cloud A U-Cloud B

Horizontal orchestration
Overall User User User User User
security VM VM VM VM VM

management

management
Compute

Computing

Computing
self- plane

security

security
management

L1 L1 L1

Data plane
Network
plane

VM1 VM2 VM3 VM1 VM2

VM3
Public cloud

Private cloud
USS USS

Provider 1 Provider 2

(a)

Overall security self-management Client software component

User User VM User

Storage VM Storage VM Compute
server proxy plane
client DA client client

Data security
management Data plane

Network
plane

VM1 VM2 VM3 VM1 VM2 VM3

CP data Storage CPS Storage CP data CPS
node server server node

Provider 1 Provider 2

(b)
Overall security self-management

User User User User Compute

VM VM VM VM plane
OvS OvS OvS OvS

Data plane

Network security Network

management Network hypervisor plane

Network VM2 Network VM2

SDN
API proxy proxy
OvS OvS

Provider 1 Provider 2

Secure tunnel
(c)

Figure 3. Detailed view of the Supercloud architecture: (a) compute plane, (b) data plane, and (c) network
plane. Each figure shows detailed subcomponents for computation, data management, and networking, and
interplay with security self-management. (OvS: Open vSwitch; SDN: software-defined networking controller)

Se p t e m b e r / Oc t o b e r 2 0 1 6 I EEE Clo u d Co m p u t i n g 69
Cloud Security
changes to clients without installing additional li-
braries. In contrast, direct accessor clients run Su-
percloud-specific logic as a client library and can
interact and access storage servers and L1 cloud
provider services directly. Direct accessor clients
can also have certain features of storage servers
built-in. Such clients could thus also be indepen-
dent of storage servers.
Proxies, typically L2 VMs, facilitate client ac-
cess to Supercloud storage and data management
offerings, such as for encryption and secure dedu-
plication. Theyre usually stateless and can be easily
added dynamically to the system.
Servers, typically stateful L1 or L2 VMs, per-
form housekeeping of critical portions of metadata
vital to the Supercloud data planes operation, such
as metadata for storage, data integrity, or configu-
ration management. Cloud provider services (CPSs)
are L1 cloud storage services that direct accessor
clients or proxies can directly access. They expose
different APIs, notably object storage and block stor-
age. Examples include OpenStack Swift and Ama-
zons Simple Storage Service (S3) and Elastic Block
Store. Cloud provider data nodes are L1 VMs in the
distributed provider infrastructure. Complementing
CPS, they can perform computation and have locally
mounted L1 block storage for Supercloud user data.
Security self-management components allow
arbitration between provider and user data security
settings.
Network plane. Figure 3c illustrates the Supercloud
network virtualization architecture. Its main de-
sign goals are network controllability; full network
virtualization to guarantee isolation between users,
while enabling them to use their desired addressing
schemes and topologies; and VM snapshotting and
migration for availability and flexibility.
To fulfill these objectives, the architecture lever-
ages software-defined networking (SDN),11 which
provides logically centralized control over forwarding
and configuration state of the software switches run-
ning in the Supercloud VMs. OpenFlow and Open
vSwitch (OvS) technologies provide fine-grained
control of packet forwarding and of switch configu-
rations, respectively. Logical centralization of control
facilitates isolation, for example, through flow rule
redefinition at the network edge, with translation of
physical to virtual events. Availability goals extend
well-proven techniques to the multicloud setting.
For each user, a specific set of network applica-
tions that control the virtual network will run on top
the Supercloud network hypervisor that maps the
virtual and physical resources. These include an ad-
70 I EEE Clo u d Co m p u t i n g w w w.co m p u t er .o rg /clo u d co m p u t i n g
dress translator (to offer L2 and L3 address virtual-
ization), a topology abstraction module (for topology
virtualization), and a resource isolation application
(to slice network resources among tenants, such as
switch CPU and forwarding tables). The network
hypervisor controls and configures the OvS switches
that are installed in all VMs. An SDN controller will
establish secure connections with each OvS switch
to control the forwarding plane.
The network hypervisor is built as an applica-
tion that runs in the Supercloud SDN controller.
Each cloud will host a specific VM, the network
proxy, where secure tunnels are set up to all other
clouds. In a distributed configuration, each proxy
will host an instance of the SDN controller.
Security management is facilitated through the
interplay of overall security self-management and
network security management components, which
enable Supercloud users to specify user-specific
settings for network configurations inside their
U-Clouds.
Dynamic Architecture
Figure 4 illustrates two typical workflows between
some key Supercloud architecture components. User
1 interacts with its VM (u1VMx) through a set of
APIs. Providers 1 and 2 host compute (VMx), net-
working (NVMx), and storage management (DVMx)
VMs. Provider 1 also hosts a physical storage ser-
vice. Supercloud considers a nested architecture
that is, u1VMx runs inside VMx.
Supercloud users interact through four inter-
faces to deploy their applications in the cloud. The
network plane interface, typically the network hy-
pervisor, interacts with the SDN controller and
network proxies, hosted in the NVMx machines, to
handle communication and establish secure tunnels
with other clouds. The data plane interface, typi-
cally storage proxies, interacts with the providers
DVMx VMs to ensure access to the users private
data. The compute plane interface, typically the L1
hypervisor, interacts with providers VMx machines
to provide memory and CPU resources.
We describe the interfaces between Supercloud
elements in several scenarios. The first scenario re-
lates to requesting data from the cloud storage; the
second relates to establishing communication be-
tween two VMs hosted in the Supercloud. The last
example shows how the Supercloud security manage-
ment interfaces enable to deploy security services.
Access to cloud storage. In this scenario (steps
af in Figure 4), during a request to the data layer
(step a), the user VM (uVM) sends a request to the
 1

a
Network
security
U-Cloud

Data
User 1 security u1VM1 u1VM2 u1VM3

Computing
security
management

Security b 2 6
management

User p c
CSP DVM1 NVM1 5
storage DVM1 NVM2 VM2 VM
Comp d
SDN 4
Comp cont.
Provider 2
Provider 1

f
L1 API API

Hypervisor LO Hypervisor LO
SDN e
infrastructure

Figure 4. Sample Supercloud workflows. Shown are the Supercloud interfaces (computation, data,
networking, self-management) to deploy applications in several scenarios: access to cloud storage,
establishing communications between VMs, and self-management of security.

data management VM (DVM) (step b). The DVM is
aware of the resources physical location inside the
cloud infrastructure. It provides this information to
the hypervisor hosting the uVM (step c), which will
ask the network management VM (NVM) (step d) to
establish a connection between resource and uVM
through the SDN network (steps e and f).
Establishing communication between user VMs.
In this scenario (steps 16 in Figure 4), after re-
ceiving a request from User 1 (step 1), uVM1
sends a communication request through the hy-
pervisor (step 2). The hypervisor forwards the
request to the NVM (step 3), which establishes
the SDN rules for the path to communicate with
NVM2 (step 4). If the destination VM is hosted
on a different CSP, the NVM forwards the request
to the NVM of the other CSP, hence setting up
the connection. Finally, NVM2 shares the request
with VM2 (step 5), which is hosting uVM2 (step
6). Each component (VMx, DVMx, and NVMx)
is accessed independently of the provider owning
the physical resource.
Security management. Users and providers also
interact with a security management plane inter-
face (see Figures 2 and 3) to deploy, orchestrate,
enforce, and monitor security requirements. Such
requirements are specified and negotiated through
SLAs during the cloud service discovery and bro-
kering phases. This distributed protection plane is
realized through interplay of several security self-
management components spread across the Super-
cloud abstraction planes.
The resource management components are self-
management agents (SMAs) responsible for deliver-
ing atomic security services such as enforcement,
detection, reaction, and monitoring. These compo-
nents operate on a particular architecture abstrac-
tion plane and are dedicated to a specific security
service (such as intrusion detection, authorization
enforcement, or trust management). Some security
services might require multiple SMAs across mul-
tiple planes and/or providers. For instance, intru-
sion detection might require the collaboration of
multiple (cross-provider) SMAs to collect, aggregate,
and process activity logs.12
Aggregation components provide a unified and
uniform view of multiple SMAs to the orchestrator.
They abstract the heterogeneity of provider security
mechanisms, meeting platform independence and
interoperability requirements.

Se p t e m b e r / Oc t o b e r 2 0 1 6 I EEE Clo u d Co m p u t i n g 71
Cloud Security
Orchestration components are decision-making
components providing security services. Each com-
ponent is a manager for a specific security service
such as authorization and access control, intrusion
detection and prevention, and trust management.
In addition, an overall orchestrator coordinates the
actions of all security managers; a planner gener-
ates plans to reach and/or maintain security objec-
tives; and a storage manager guarantees persistence
and delivery of the knowledge needed for self-man-
agement of security. Orchestration components are
also responsible for retrieving user security require-
ments from SLAs, converting them into policies
and configurations to be enforced, and detecting
and managing conflicts between tenants, users,
and/or providers.
Use Cases
To illustrate how the Supercloud architecture can
be mapped to real-world use cases, we use examples
from the healthcare domain.
Hospital Imaging Archive
The amount of diagnostic imaging data is quickly
increasing, imposing great challenges on hospital
archive infrastructures, which must ensure high
data availability, security, and regulatory compli-
ance. A cloud-based solution can help address these
challenges.
Such a solutions architecture should minimize
the risk of security breaches and privacy violations,
including unprivileged access to data (both at rest
and during processing) with regard to defined poli-
cies. These policies might include hospital-specific
policies context, legal country boundaries, and user
groups. In terms of performance, robust data pro-
cessing with low latency is desired, especially across
different clouds.
Hospitals can store their clinical data as well as
their imaging studies in on-premises private cloud
storage. Archiving in the cloud helps simplify the
data management and hospital archive infrastruc-
tureespecially due to high-volume imaging studies
that are often as large as 1 Gbyte. Since on-premises
storage can be limited, it makes sense to store this
data securely in public cloud storage. For example, a
hospital might store data from the last six months in
the private clouds on-premises storage, while stor-
ing older data (10 years or more) in the public cloud.
Figure 5a shows a sample Supercloud implementa-
tion of such a solution.
In Figure 5a, three hospitals (A, B, and C) share
a private cloud to store and manage their clinical
data. A VM on the compute plane is dedicated to
72 I EEE Clo u d Co m p u t i n g w w w.co m p u t er .o rg /clo u d co m p u t i n g
each hospital for its operations. Whenever a hospital
VM wants to store or retrieve clinical data (for exam-
ple, MRI imaging data), it communicates with a pic-
ture archiving and communication system (PACS)
VM interfacing with the data plane. The data plane
provides an abstraction to the user VM, making all
underlying storage directly accessible (including en-
crypting stored data). Data older than six months is
stored on the public cloud, whereas recent data is
kept in the private clouds on-premises storage, pro-
viding instantaneous access to it. Here, the network
plane is responsible for handling all communica-
tions across different clouds and VMs. Hospitals can
also define their security policies, such as how other
hospitals can access their data. Components that
are dedicated to data security and security manage-
ment across the L1 hypervisor and compute VMs
will prevent any unprivileged access to data based
on security policies defined by each hospital.
Healthcare Laboratory Information System
This use-case demonstrates the impact of the Super-
cloud architecture for Maxdata Software, a healthcare
software vendor that aims to deploy its software on
the cloud as SaaS while enforcing the security re-
quirements of different healthcare institutions.
The CLINIdATALIS healthcare laboratory in-
formation system (LIS) is a cross-platform Web ap-
plication in which server components can run on any
common operating system and relational database.
The CLINIdATALIS must integrate with dozens
of other clinical and nonclinical information systems
(such as intensive care units, patient identification,
billing, and regional health portals). It includes a set
of real-time interfaces with physical electronic equip-
ment (automated analyzers). The solution consists of
three components on the server side: a stateless ap-
plication, a database engine, and database data. The
Supercloud approach allows each healthcare institu-
tion to define the U-Cloud that best fits its needs.
Concrete deployment on physical cloud providers is
then automated. The considered setting is a large
hospital cluster that employs thousands of profes-
sionals, processes tens of millions of transactions per
day, and is located in a country where personal data
protection must be guaranteed.
In a typical U-Cloud specification,
the application and database engine are repli-
cated across several VMs on the compute plane
(fault tolerance and load balancing);
data is split among different storage nodes in
the data plane (offering confidentiality, even if
one storage node is compromised);
 Comp Comp Comp Compute VM Compute VM Compute VM

Data security
Computing security Hospital A Hospital A Hospital C
Security management
management Data Data
server proxy PACS Client Client Client

Horizontal orchestration
L1 L1
Network Network
OvS security hypervisor OvS

VM VM VM VM VM VM VM

Comp Comp

Private cloud
Comp SDN Comp Comp
Public cloud

Network Data cont Network

proxy L1 proxy proxy L1 L1

API OvS OvS OvS API USS OvS USS OvS USS OvS USS OvS

Hypervisor LO Hypervisor LO

(a)

Automated Automated
analyzer analyzer

U-Cloud A

Comp Comp Compute VM Compute VM

Data security
Computing security

CLINIdATALIS CLINIdATALIS
Security
management

management Data Data

server proxy Client Client

Horizontal orchestration
L1 L1
Network Network
OvS security hypervisor OvS

VM VM VM VM VM VM VM
Comp Comp
Private cloud

Comp SDN Comp Comp

Public cloud

Network Data cont. Network

proxy L1 proxy proxy L1 L1

API OvS OvS OvS API USS OvS USS OvS USS OvS USS OvS

Hypervisor LO Hypervisor LO

(b)

Figure 5. Supercloud practical deployments: (a) high-availability storage and disaster recovery, and
(b) healthcare laboratory information system. (USS: user-centric system service)

Se p t e m b e r / Oc t o b e r 2 0 1 6 I EEE Clo u d Co m p u t i n g 73
Cloud Security
a set of networks connect application VMs to
automated analyzers running on hospital prem-
ises and to database engine VMs, which in turn
are connected to storage nodes;
VMs on the compute plane ensure confidential-
ity, integrity, and 99.99 percent availability;
storage nodes on the data plane ensure data in-
tegrity and 99.99 percent availability; and
data may be processed and stored only in a pre-
defined set of countries.
As Figure 5b shows, a Supercloud infrastructure
can then deploy the VMs on a trusted private cloud
to ensure confidentiality on the compute plane, in-
stantiate the storage nodes on a set of public cloud
providers running security mechanisms (such as
encryption and secret sharing) to ensure confiden-
tiality, and connect the different components us-
ing virtual networks provided by the network plane.
Deployments consider the locations or countries
specified by the healthcare institution. Replicated
instances of the CLINIdATALIS application run
on VMs on the compute plane. These instances then
connect to the database engine running on a differ-
ent VM linked with the data plane.
In case of regulatory, economic, or other type of
change, healthcare institutions can update U-Cloud
requirements and/or features. The Supercloud in-
frastructure automatically redeploys the solution
accordingly, enabling quick adaptation to context
changes. It also prevents vendor lock-in.
ere implementing the different compo-
nents of the Supercloud architecture to
gradually achieve integrated proof of concepts.
The solution is currently at an advanced stage of
implementation. Several results are already avail-
able (see https://Supercloud-project.eu/publications
-deliverables). However, were still integrating the
various components. Preliminary performance re-
sults have shown relatively modest overheads, giving
good indications about the potential for the solution
(such as for network virtualization13). Our next step
is to validate the approach through testbed integra-
tion. Other foreseen application domains include
network function virtualization or smart home secu-
rity. Results will be disseminated to promote open
source cloud technologies and will be contributed to
major standardization bodies.
Acknowledgments
This work is supported by the European Union Su-
percloud Project (Horizon 2020 Research and In-
74 I EEE Clo u d Co m p u t i n g w w w.co m p u t er .o rg /clo u d co m p u t i n g
novation Program, grant 644962) and by the Swiss
Secretariat for Education Research and Innovation
(contract 15.0091). It is based on contributions from
the entire Supercloud consortium.
References
1. F. Bonomi et al., Fog Computing and Its Role in
the Internet of Things, Proc. 1st Workshop Mo-
bile Cloud Computing (MCC), 2012, pp. 1316.
2. F. Manco et al., The Case for the Superfluid
Cloud, Proc. 7th USENIX Workshop Hot Topics
in Cloud Computing (HotCloud), 2015.
3. L. Zheng et al., How to Bid the Cloud, Proc.
ACM Conf. Special Interest Group on Data
Comm. (SIGCOMM), 2015, pp. 7184.
4. R. Los, D. Shackleford, and B. Sullivan, Notori-
ous Nine Cloud Computing Top Threats in 2013,
tech. report, Cloud Security Alliance, 2013.
5. D. Sgandurra and E. Lupu, Evolution of At-
tacks, Threat Models, and Solutions for Virtual-
ized Systems, ACM Computing Surveys, vol. 48,
no. 3, 2016, pp. 138.
6. D. Williams, H. Jamjoom, and H. Weatherspoon,
Plug into the Supercloud, IEEE Internet Com-
puting, vol. 17, no. 2, 2013, pp. 2834.
7. A. Ludwig and S. Schmid, Distributed Cloud
Market: Who Benefits from Specification Flex-
ibilities? ACM SIGMETRICS Performance
Evaluation Rev., vol. 43, no. 3, 2015, pp. 3841.
8. K. Bernsmed et al., Thunder in the Clouds: Se-
curity Challenges and Solutions for Federated
Clouds, Proc. IEEE 4th Intl Conf. Cloud Com-
puting Technology and Science (CloudCom),
2012, doi:10.1109/CloudCom.2012.6427547.
9. M. Ben-Yehuda et al., The Turtles Project: De-
sign and Implementation of Nested Virtualiza-
tion, Proc. 9th USENIX Conf. Operating Sys-
tems Design and Implementation (OSDI), vol. 10,
2010, pp. 423436.
10. P. Watson, Application Security through Feder-
ated Clouds, IEEE Cloud Computing, vol. 1, no.
3, 2014, pp. 7680.
11. D. Kreutz et al., Software-Defined Networking:
A Comprehensive Survey, Proc. IEEE, vol. 103,
no. 1, 2015, pp. 1476.
12. S.T. Zargar et al., DCDIDP: A Distributed, Col-
laborative, and Data-Driven Intrusion Detection
and Prevention Framework for Cloud Comput-
ing Environments, Proc. 7th Intl Conf. Col-
laborative Computing: Networking, Applications
and Worksharing (CollaborateCom), 2011, pp.
332341.
13. M. Alaluna, F. Ramos, and N. Ferreira Neves,
(Literally) above the Clouds: Virtualizing the
 Network over Multiple Clouds, Proc. IEEE
Conf. Network Softwarization (NetSoft), 2016,
pp. 112115.
Marc Lacoste is a senior research scientist in
the Security Department of Orange Labs, and tech-
nical leader of the H2020 Supercloud Project. His
main research interests include security architecture,
cloud computing security, self-protecting systems, and
open security kernels. Lacoste has a PhD in comput-
er science from the University of Grenoble, France.
Hes a member of ACM. Contact him at marc.lacoste
@orange.com.
Markus Miettinen is a researcher in the System
Security Lab at the Department of Computer Science
at Technische Universitt Darmstadt, Germany. His
research interests include contextual security, data
analysis-based security enablers, and security manage-
ment in new computation environments such as the
Internet of Things. Miettinen has an MSc in computer
science from the University of Helsinki, Finland. Con-
tact him at markus.miettinen@trust.tu-darmstadt.de.
Nuno Neves is an associate professor in and head
of the University of Lisbons Department of Computer
Science, where he leads the Navigators research group
and is on the executive board of the Large-Scale Infor-
matics Systems Laboratory (LaSIGE) research unit. His
research interests include the security and dependabil-
ity aspects of distributed systems. Neves has a PhD in
computer science from University of Illinois Urbana-
Champaign. Contact him at nuno@di.fc.ul.pt.
Fernando M.V. Ramos is an assistant professor
in the Department of Computer Science the Univer-
sity of Lisbon. His research interests include network
programmability and network virtualization. Ramos
has a PhD in computer science and engineering from
the University of Cambridge, UK. Contact him at
fvramos@ciencias.ulisboa.pt.
Marko Vukolic is a research staff member at
IBM Research, Zurich. His research interests are in
distributed algorithms and systems, including fault-
tolerance, blockchain and distributed ledgers, cloud
computing security, and distributed storage. Vukolic
has a PhD in distributed systems from Ecole Polytech-
nique Fdrale de Lausanne (EPFL), Switzerland.
Contact him at mvu@zurich.ibm.com.
Fabien Charmet is a research engineer in the
Samovar Lab (Centre National de la Recherche
Scientifique) in Tlcom SudParis, Institut Mines-
Se p t e m b e r / Oc t o b e r 2 0 1 6 I EEE Clo u d Co m p u t i n g
Tlcom. His research interests include penetration
testing, network security, and software-defined net-
working. Charmet has an MSc in enterprise architec-
ture and an MSc in network and security from the
University of Lille 1, France. Contact him at fabien
.charmet@telecom-sudparis.
Reda Yaich is a researcher in the LabSTICC
Laboratory (Centre National de la Recherche Scien-
tifique) in Tlcom Bretagne, Institut Mines-Tlcom.
His research interests include the specification and
enforcement of trust and security policies over open,
distributed, and decentralized systems. Yaich has a
PhD in computer science from Ecole des Mines of
Saint-Etienne, France. Contact him at reda.yaich
@telecom-bretagne.eu.
Krzysztof Oborzynski is a software archi-
tect at Healthcare Informatics Services and Solutions,
Clinical Platforms, Philips Healthcare. His research
interests include systems reliability, performance, and
serviceability. Oborzynski
has a PhD in computer sci-
ence at the Institute of Computing Science, Poznan
University of Technology, Poland. Contact him at
Krzysztof.Oborzynski@philips.com.
Gitesh Vernekar is a senior manager at Health-
care Informatics Services and Solutions, HealthSuite
Digital Platform, Philips Healthcare. His research in-
terests include delivering innovative and repeatable
software solutions in healthcare, banking, and high-
tech industries, and technology-enabled operations
and services. Vernekar has an international MBA in
entrepreneurship and strategy management from the
Rotterdam School Management, Erasmus University,
The Netherlands. Contact him at Gitesh.Vernekar
@philips.com.
Paulo Sousa is chief executive officer at Max-
data Software. His research interests include real-time
systems, intrusion tolerance, and security. Sousa has a
PhD in computer science from the University of Lis-
bon. Contact him at paulo.sousa@maxdata.pt.
Read your subscriptions through
the myCS publications portal at
http://mycs.computer.org.
75
 Standards Now
The Design and
Architecture of
Microservices
To explore the role of design in
software, consider two other fields
that also depend on it: art and ar-
chitecture. Like art, much of software design
can be a matter of taste. As in the art world, issues
that inspire passionate debate in the field of soft-
ware design resonate most loudly within its internal
boundaries, and dont necessarily have much of an
effect outside of those boundaries.
Design is also important in architecture, both
for aesthetic and physically important reasons. As
in the structure of buildings, architectural design
can have serious ramifications on the reliability, ro-
bustness, and suitability for use of software. Like
architects, developers are generally aware of the im-
portance of internal structural elements in software,
and study and debate the performance and business
reasons for selecting one approach over another.
One would not wish to use the plans for a per-
Alan Sill
Texas Tech University,
alan.sill@standards-now.org
76 I E E E C l o u d C o m p u t i n g p u b l i s h e d b y t h e I E E E c o m p u t e r s o cie t y 
sonal home to build a larger structure, such as a
sports arena, concert hall, or high-rise building. In
the same way, the choice of architectural design pat-
tern in software must be tuned to the desired appli-
cation, workload, and expected level of use.
Aesthetics and user reaction are important in all
of these settings. No one would argue at this stage,
in which products from all vendors tend to be beau-
tifully designed, about the need to pay significant
attention to issues of user experience and ease of
use in software design. Just as we enjoy beautifully
designed and functional buildings, software designs
are most enjoyable when theyre both useful and art-
fully built.
Microservice Architectures
Concepts related to microservices are discussed ex-
tensively elsewhere in this issue. They are, to some
degree, old wine in new bottles. The basic approach
of separating services into functions that can inter-
act via programming interfaces has been with us for
some time. Methods to implement such separation
in the framework of service-oriented architectures
(SOAs) are also not new.
Recent implementations of microservices in
cloud settings, however, take the SOA idea to new
limits that are driven by the goals of rapid, inter-
changeable, easily adapted, and easily scaled com-
ponents. This is obviously not the only way to use
clouds, but it draws well on the basic functional
features of cloud computing and is a good match to
the corresponding delivery framework. A continued
emphasis on the use of RESTful APIs as discussed
in previous Standards Now columns has also ac-
celerated the pace of change and overall utility of
cloud service delivery.
The resulting factorization of workloads and in-
crementally scalable features of microservices pro-
vide a multitude of ways by which SOA can be freed
from its previously hidebound, overly formal imple-
mentation settings and be implemented in much less
forbidding ways. One consequence of this evolution
is the development of new architectural patterns and
the corresponding emergence and use of standards.
As with art and architecture, much discretion
is left to the designer in microservice delivery. You
might be tempted to think that standards arent im-
portant, or less important, in the rapidly changing
2325-6095/16/$33.00 2016 IEEE
microservices arena, but this assumption, as I am
about to show, wouldnt be correct. To a great de-
gree, the flexibility and ease of implementation of
modern approaches to microservice architecture is
either compatible with or in fact greatly driven by
the emergence of successful design patterns that are
already in the process of becoming standards.
Microservice Delivery Using Containers
It would be equally incorrect to equate different
trends in cloud design as being equivalent. The cur-
rent tendency to implement different sets of software
in the context of software containers, for example, is
more of a coincidence than a direct consequence of
microservice design. Its true that con-
tainers can be made to isolate execution
environments from each other, and that
they lend themselves to scalability by
allowing such containers to be instanti-
ated quickly on demand.
Other features of software contain-
ers require much more work to overcome,
however, such as the need to provide
well-thought-through mechanisms for
network communications between them
and associated complications of their use
on different physical hosts or on hosts
located in different datacenters. Similar
problems crop up with regard to security, monitor-
ing, and the need to minimize the operational size of
containers. These issues require careful thought and
attention to details that arent directly related to the
SOA aspects of microservices themselves.
Despite these shortcomings, microservice de-
livery matches well in many ways to deployment in
software containers. This method is, in fact, cur-
rently the most popular way to deploy them, but to
deal effectively with the resulting complications just
described absolutely requires the use of standards.
This column has covered the emergence of such
standards in this area many times, starting with the
appc application container specification originally
developed by CoreOS, and the runC container en-
gine originally developed by Docker. Much commu-
nity work has gone into integrating the approaches
of these two specification sets and extending them
to newer, broader use cases.
Two current relevant projects of the Linux Foun-
Se p t e m b e r / Oc t o b e r 2 0 1 6 I EEE Clo u d Co m p u t i n g
dation are the Open Container Initiative (www.open-
containers.org) and the Cloud Native Computing
Foundation (CNCF, https://cncf.io). Popular image
formats include ACI, the container image format de-
fined in the appc specification, and OCI, the Open
Containers Image Format specification. Much of the
work going on within the CNCF is aimed higher up
in the software stack and addresses the large-scale
behaviors of distributed systems of microservices.
Although work is still in progress on various as-
pects of each of these standards within these orga-
nizations and on their relationship to each other, its
encouraging to see efforts of this sort emerge natu-
rally from ongoing community interests.
Just as we enjoy beautifully designed
and functional buildings, software
designs are most enjoyable when
theyre both useful and artfully built.
Data Formats and APIs
To make microservice architectures work in prac-
tice, one must get information into and out of these
services and find ways to make the information ex-
change and control-passing features take place at
component boundaries. Programmers must there-
fore address design topics dealing with data ex-
change and messaging, and must implement these
services with suitable orchestration and control.
Standards exist that provide the basis for such
data exchange. The most popular data formats in
cloud computing are JavaScript Object Notation
(JSON) and XML. JSON is documented in two
standards: Ecma Internationals ECMA-4041 and
IETFs RFC 7159.2 XML is a somewhat older but
still popular text-based format for data exchange sup-
ported by several W3C standards. Although it isnt as
human-readable as JSON, each format has particu-
lar strengths and weaknesses and both are still very
much in use.
77
 Standards Now
For Internet of Things (IoT) and sensor-oriented
settings, as discussed in the previous issue on man-
ufacturing, the Sensor Network Object Notation
(SNON, www.snon.org) is a representation based on
JSON that includes some predefined fields that are
especially useful in dealing with sensor data. In ad-
dition, the Data Distribution Service (DDS, http://
www.omg.org/spec/DDS) and DDS Data Local Re-
construction Layer (DDS-DLRL) specifications
were developed by the Object Management Group
specifically to handle data interchange tasks related
to IoT systems.
General data standards are available
to deal with the wide variety of
formats for datasets without having
to be locked into a particular format.
Unlike the other protocols Ive mentioned, DDS
can handle content-aware network routing, data pri-
oritization by transport priorities, and both unicast
and multicast communications within the methods
defined by the standard set itself.
Additionally, general data standards are avail-
able to deal with the wide variety of formats for da-
tasets without having to be locked into a particular
format. For example, working with the US National
Center for Supercomputing Applications (NCSA,
http://www.ncsa.illinois.edu) and IBM, the Open
Grid Forum has developed a language for describing
the structure of data formats without needing to re-
write them. The resulting Data Format Description
Language (DFDL, www.ogf.org/dfdl) is a flexible,
general specification set suited to a wide variety of
data input, output, and format transcription prob-
lems and is supported by both commercial and open
source software implementation tools.
Many approaches currently used in microser-
vices create custom APIs for access to specific data.
This approach is compatible with, though typically
implemented without, reference to external data for-
78 I EEE Clo u d Co m p u t i n g w w w.co m p u t er .o rg /clo u d co m p u t i n g
mat standards. As a result, current cloud microser-
vice designs are burdened with a huge variety and
multiplicity of API definitions.
In previous columns, Ive referred to the API
directory maintained, for example, by the website
ProgrammableWeb.com, which at the time of this
writing maintains a directory of more than 15,000
APIs (www.programmableweb.com/apis/directory).
This situation requires APIs to be designed to work
either in small subsets of the application arena in
which either the API is stable, or to be built to a
common self-describing or standardized pattern.
Examples of effective API standards
are the RESTful API Markup Language
(RAML, http://raml.org) and Swagger,
which has evolved into the Open API
Initiative (https://openapis.org), as dis-
cussed in previous columns.
Messaging Standards
The next step after understanding data
formats and APIs for data exchange is
to move in the direction of messaging
and application control. HTTP and its
secure variant HTTPS are the most fa-
miliar messaging standards, and are specified in a
range of IETF documents summarized at the work-
ing group website (httpwg.org/specs).
The IETF specifications underlying TCP form
the basis of a large fraction of Internet traffic. TCP
continues to receive detailed attention from the
community due to its importance in a wide variety
of settings. The most important TCP specifications
and their relationships with one another are sum-
marized in RFC 7414.3 A number of other applica-
tion, transport Internet, and link layer protocols are
also useful.4
The User Datagram Protocol (UDP) is useful for
Internet communications that can be intermittent
or dont have to be completely received at all times.5
UDP can be used to carry out IP communications
in situations in which handshaking and verification
of receipt of the individual message packets arent
necessary. The Stream Control Transmission Proto-
col (SCTP) provides an alternative to TCP and UDP
applicable to streaming use cases.6
Another example of a manufacturing-relevant
specialized transfer standard is the Constrained Ap-
plication Protocol (CoAP).7 According to its descrip-
tion, CoAP provides a request/response interaction
model between application endpoints, supports
built-in discovery of services and resources, and in-
cludes key concepts of the Web such as URIs and
Internet media types. CoAP is designed to easily
interface with HTTP for integration with the Web
while meeting specialized requirements such as
multicast support, very low overhead, and simplicity
for constrained environments.
The Extensible Messaging and Presence Protocol
(XMPP) is an XML-based communications standard
designed for message-oriented middleware communi-
cations. The core specifications for XMPP are RFCs
6120,8 6121,9 and 762210 and include a WebSocket
binding defined in RFC 7395.11 Several extensions
beyond the base specifications are supported by the
dedicated XMPP organization (see http://xmpp.org/
extensions). Beyond its applications to human-
oriented communications, XMPP is also used in
smart electrical grid applications and a variety of in-
dustrial settings. Several extensions oriented toward
use in IoT settings were published in late 2015.
Methods to handle publish/subscribe messag-
ing can have advantages compared to the protocols
when used for machine-to-machine communica-
tions at high speeds. The Message Queuing Telem-
etry Transport (MQTT, http://docs.oasis-open.org/
mqtt/mqtt/v3.1.1/os/mqtt-v3.1.1-os.html), recently
standardized by the Organization for Advanced
Structured Information Systems (OASIS), is another
example of such a method.
The Advanced Message Queuing Protocol
(AMQP) is another popular middleware messaging
standard set. It can be applied using either publish/
subscribe or point-to-point communication patterns.
OASIS published AMQP as a set of standards in
2014 (www.oasis-open.org/standards#amqpv1.0) and
adopted it as a joint International Organization for
Standardization/International Electrotechnical Com-
mission (ISO/IEC) later that year.12 AMQP has a
layered architecture, and the specification set is orga-
nized into different parts to reflect that architecture.
Networking Considerations
Networking provides the core feature that ties all
cloud services to each other. I discussed this topic
extensively in the May/June 2016 issue of this maga-
Se p t e m b e r / Oc t o b e r 2 0 1 6 I EEE Clo u d Co m p u t i n g
zine,4 stating there that the underpinnings of the
cloud consist of the ways in which otherwise discon-
nected, highly scaled, and rapidly changing collec-
tions of service components can be instantiated and
hooked together swiftly and flexibly to form the ba-
sis of a cloud service.
The need for performance is especially impor-
tant in the implementation of microservice archi-
tectures. This consideration obviously provides the
practical limit to the degree to which individual
service components can be scaled down in terms of
information exchange and functionality. Issues re-
lated to security, connectivity between microservice
components, and scalability also have considerations
that are affected by the choice of networking archi-
tecture and protocols.
A US National Institute of Standards and Tech-
nology draft publication covers this topic, with an
emphasis on security considerations.13 Although the
comment period has closed on this particular draft,
the topics general nature makes it seem to me that
the issues discussed in this document will be revis-
ited several times in the near future as the general
area of microservice delivery matures.
Special considerations that relate to networking
are also pushing some microservice frameworks in
directions that lead away from human readability of
the interchanged data and even of the on-the-wire
protocols used in the API calls and messaging. Ex-
amples that Ive discussed in previous issues contin-
ue to mature, including the recent release by Google
of its gRPC framework at version 1.0 (https://github
.com/grpc/grpc/releases/tag/v1.0.0) after an extend-
ed period of development, with multiple language
bindings now available.
The design approach underpinning gRPC makes
extensive use of protocol buffers (https://developers
.google.com/protocol-buffers/docs/reference/overview),
a design construct intended to serialize structured
data in a simpler manner than in XML but without
some of JSONs limitations, and is designed to be
compatible with HTTP/2. My personal belief is
that these developments illustrate the emergence
of new design trends for cloud services that favor
speed and responsiveness over human readability
of the exchanged data and API calls, and that might
lead to radical revisions of some of the fundamental
assumptions that govern microservices in the future.
79
 Standards Now
The discussion here has focused on
the design and architecture of mi-
croservices. Ive covered considerations related
to packaging and delivery of microservices in contain-
ers, data exchange, and data formats, messaging and
networking, focusing on some up-to-date topics on
standards related to these areas.
My next column will address topics related to
microservices orchestration, including relevant stan-
dards such as Topology and Orchestration Speci-
fication for Cloud Applications (Tosca) and Cloud
Application Management for Platforms (CAMP);
microservices control, including the Open Cloud
Computing Interface (OCCI) and Cloud Infrastruc-
ture Management Interface (CIMI) standard sets; and
serverless microservices, such as Amazon Lambda
and related concepts. Ill also take another look at the
SOA basis for microservice architectures to tie both
of these columns together.
As always, this discussion only represents my
own viewpoint. Id like to hear your opinions and
experience in this area. Im sure other readers of the
magazine would also appreciate additional informa-
tion on this topic.
Please respond with your input on this or previ-
ous columns. Please include news you think the com-
munity should know in the general areas of cloud
standards, compliance, or related topics. Im happy to
review ideas for potential submissions to the maga-
zine or for proposed guest columns. I can be reached
for this purpose at alan.sill@standards-now.org.
References
1. Ecma International, The JSON Data Interchange
Format, Ecma-404, 1st ed. 2013; www.ecma
-international.org/publications/standards/Ecma
-404.htm.
2. T. Bray, ed., The JavaScript Object Notation (JSON)
Data Interchange Format, IEEE RFC 7159, 2014;
https://www.rfc-editor.org/info/rfc7159.
3. M. Duke, et al., A Roadmap for Transmission Con-
trol Protocol (TCP) Specification Documents, IETF
RFC 7414, 2015; www.rfc-editor.org/info/rfc7414.
4. A. Sill, Standards Underlying Cloud Networking,
IEEE Cloud Computing, vol. 3, no. 3, 2016, pp. 7680.
5. J. Postel, User Datagram Protocol, IETF RFC 768,
1980; www.rfc-editor.org/info/rfc768.
6. R. Stewart, ed., Stream Control Transmission Pro-
80 I EEE Clo u d Co m p u t i n g w w w.co m p u t er .o rg /clo u d co m p u t i n g
tocol, IETF RFC 4960, 2007; www.rfc-editor.org/
info/rfc4960.
7. Z. Shelby, K. Hartke, and C. Bormann, The Con-
strained Application Protocol (CoAP), IETF RFC
7252, 2014; www.rfc-editor.org/info/rfc7252.
8. P. Saint-Andre, Extensible Messaging and Pres-
ence Protocol (XMPP): Core, IETF RFC 6120,
2011; www.rfc-editor.org/info/rfc6120.
9. P. Saint-Andre, Extensible Messaging and Presence
Protocol (XMPP): Instant Messaging and Presence,
IETF RFC 6121, 2011; www.rfc-editor.org/info/
rfc6121.
10. P. Saint-Andre, Extensible Messaging and Pres-
ence Protocol (XMPP): Address Format, IETF RFC
7622, 2015; www.rfc-editor.org/info/rfc7622.
11. L. Stout, ed., An Extensible Messaging and Pres-
ence Protocol (XMPP) Subprotocol for WebSock-
et, IETF RFC 7395, 2014; www.rfc-editor.org/
info/rfc7395.
12. Information technologyAdvanced Message
Queuing Protocol (AMQP), Intl Organization for
Standardization/Intl Electrotechnical Commis-
sion, ISO/IEC 19464, v.1.0, 2014; www.iso.org/
iso/home/store/catalogue_tc/catalogue_detail.
htm?csnumber=64955.
13. A. Karmel, R. Chadromouli, and M. Iorga, NIST
Definition of Microservices, Application Contain-
ers and System Virtual Machines, Natl Inst. of
Standards and Technology (NIST) Special Publica-
tion 800-180, 2016; http://csrc.nist.gov/publications/
drafts/800-180/sp800-180_draft.pdf.
Alan Sill directs the US National Science Founda-
tions Cloud and Autonomic Computing industry/uni-
versity cooperative research center. Hes interim senior
director of the High Performance Computing Center
and adjunct professor of physics at Texas Tech Univer-
sity, and visiting professor of distributed computing at
the University of Derby. Sill has a PhD in particle phys-
ics from American University. Hes an active member of
IEEE, the Distributed Management Task Force, and the
TeleManagement Forum, and he serves as president for
the Open Grid Forum. Hes a member of several cloud
standards working groups and national and interna-
tional standards roadmap committees, and he remains
active in particle physics and advanced computing re-
search. Contact him at alan.sill@standards-now.org.
 Blue Skies

Open Issues in Scheduling

Microservices in the Cloud
he adoption of container-based microservices architec-
tures is revolutionizing application design. By adopting
a microservices architecture, developers can engineer Maria Fazio and
Antonio Celesti
applications that are composed of multiple lightweight, University of Messina
self-contained, and portable runtime components deployed across
a large number of geodistributed servers. Rajiv Ranjan and
Chang Liu
Newcastle University
A microservices-based cloud application in- wards a DevOps culture, in
volves the interoperation of multiple microservices, which development and op- Lydia Chen
each developed separately, that can be deployed, up- erations teams work closely IBM Research
dated, and redeployed independently without com- together to support an appli-
promising the applications ecosystems integrity. cation over its lifecycle, and Massimo Villari
The ability to independently update and redeploy the go through a rapid or even University of Messina
code base of one or more microservices increases ap- continuous release cycle.
plications scalability, portability, updatability, and Microservices act as stand-
availability, but at the cost of expensive remote calls alone application subunits or
(instead of in-process calls) and increased overhead components, implementing specific communication
for cross-component synchronization. protocols for sending and receiving messages. In
The microservices-based approach is in con- microservices, data flows through smart endpoints,
trast to the traditional monolithic development which also process incoming information. Using
of applications, where each application is a single, well-defined interfaces and protocols, application
autonomous unit. For example, in a client-server developers can deploy different microservices on
application, the server is a monolithic entity that heterogeneous infrastructures without a specific
handles HTTP requests, executes logic, and re- integration framework. Generally, microservice
trieves or updates its data. The problem with such communication uses a REST approach based on
monolithic architectures is that even a small modi- HTTP and TCP protocols, XMPP, or JavaScript
fication of the applications logic requires the de- Object Notation (JSON). However, currently, there
ployment of a new running version of the entire are no widely adopted standardized protocols or
code base. A microservice architecture is light- data formats for microservice communication.1
weight and easily shipped and updated. Hence, its Microservices deployment and execution also leads
ideal for engineering applications where we can- to various networking issues. To this end, applica-
not fully anticipate functionalities in advance (for tion developers currently adopt various software-
example, the types of devices that might one day defined networking (SDN) and network function
access the application). Microservice architectures virtualization (NFV) solutions for networking
are a part of a larger shift in IT departments to- microservices.

2325-6095/16/$33.00 2016 IEEE Sep t ember /O c to ber 2016 I E E E C l o u d Co m p u t i n g  81

 Blue Skies

Guest Guest Guest Guest

Guest Guest microservice microservice
microservice microservice
processes processes
Runtime Libs Runtime Libs
Runtime Libs Runtime Libs Runtime Libs Runtime Libs Container Container
Guest OS Guest OS Container Container Container engine
VM VM Host operating system
Container engine

Hypervisor Host operating system Hypervisor

Physical cloud hardware Physical cloud hardware Physical cloud hardware

(a) (b) (c)

Figure 1. Comparison of cloud architectures: (a) hypervisor-based application deployment, (b) hypervisor-free containerized
microservice, and (c) containerized microservice within a hypervisor-managed physical cloud hardware.

Overview of Virtualization Technologies container virtualization (LCV) is the most well-

Hypervisor-based resource virtualization (such as
that used by Citrix and VMware) is a key concept in
cloud computing. Hypervisor-based virtualization
enables cloud providers to create unique virtual ma-
chines (VMs) that share a set of physical hardware re-
sources (CPU, memory, network, and disk). Each VM
executes distinct operating system instances (rang-
ing from proprietary to open source), which supports
fault-tolerant and isolated security context behavior.
Container-based virtualization can be used to
create microservices.2 A container is a collection
of operating system kernel utilities configured to
manage the physical hardware resources used by a
particular application component.3 Containeriza-
tion allows cloud providers to instantiate, relocate,
and optimize hardware resources in a more flexible
way while providing near-native performance (if de-
ployed in hypervisor-free mode). Because the con-
tainers share a single operating system kernel, they
incur lower overhead.3 However, container-based vir-
tualization leads to weaker isolation and introduces
greater security vulnerabilities than hypervisor-based
virtualization.4
From the user viewpoint, each container looks
and executes exactly like a standalone operating sys-
tem. Additionally, in a cloud computing scenario,
developers can deploy a higher density of contain-
ers (compared to VM density in hypervisor-managed
datacenters) on the same physical hardware. Linux
known container-based virtualization technology.
Popular LCV solutions include Docker, LXC, lmct-
fy, and OpenVZ.
Figure 1 shows the key architectural differenc-
es between hypervisor-based and container-based
virtualization. Figure 1a shows application compo-
nents deployed within a hypervisor-based VM that
provides abstraction for full guest operating sys-
tems (one per VM). Figure 1b shows microservice
deployment within a hypervisor-free containerized
environment. Finally, Figure 1c shows microservice
deployment within a containerized environment on
a physical hardware managed by a hypervisor-based
VM. After physical hardware (for example, a server
or appliance), a downward-facing hypervisor is more
suitable for managing infrastructure-as-a-service
(IaaS) clouds, whereas containers are more suited
for platform-as-a-service (PaaS) clouds. Having said
that, hypervisor-free containerization isnt a replace-
ment for traditional hypervisor technologies; the two
technologies complement each other and must be
carefully analyzed during the application architec-
ture design phase in terms of performance isolation,
overhead, and security requirements.
Container Engines for Microservices
Scheduling and Management
Several tools can instantiate and manage contain-
ers in clouds. Docker Swarm, for example, pro-

82 I E E E C l o u d Co m p u t i n g  w w w . c o m p u t e r . o r g / c l o u d c o m p u t i n g
vides native clustering for Docker containers. It
turns a pool of Docker hosts into a single virtual
Docker host. Because Docker Swarm serves the
standard Docker API, any tool that already com-
municates with a Docker daemon can use Swarm
to transparently scale to multiple hosts. A Docker
container manager represents the basic container-
oriented technology.
Kubernetes is an open-source technology for
automating deployment, operations, and scaling of
containerized applications. It groups the containers
making up an application into logical units for easy
management and discoveryfor example, based on
their resource requirements and other constraints.
Kubernetes also provides horizontal scaling of ap-
plications, which can be performed manually or
automatically based on CPU load. Finally, it pro-
vides automated rollouts and rollbacks and self-
healing features.
Magnum is the OpenStack API service that
makes container orchestration engines such as
Docker Swarm and Kubernetes available as first-class
resources in the OpenStack managed datacenter.
Magnum uses the Heat service to schedule an operat-
ing system image, which contains Docker and Kuber-
netes, and runs this image on either VMs or a bare
metal cluster.
The Google Container Engine provides a com-
mercial service that relies on Docker and Kuber-
netes for cluster management and orchestration.
Similarly, the Amazon Elastic Compute Cloud
(EC2) container service supports Docker containers
to be deployed on a managed cluster of Amazon EC2
instances. Rackspace is slightly behind with respect
to container-based offerings. Its beta service, Cari-
na, is based on Docker Swarm and doesnt provide
any elasticity features.
With regard to networking containerized mi-
croservices, OpenStack Neutron supports the man-
agement of virtual LANs in cloud datacenters by
creating ad hoc NFV. NFV uses virtualization tech-
nologies to manage core networking functions via
software instead of relying on hardware to handle
these functions.
Creating NFVs using Open Virtual Network
(OVN) technology guarantees an efficient and se-
cure use of the network. OVN complements existing
SDN capabilities, adding native support for virtual
Sep t ember /O c to ber 2016 I E E E C l o u d Co m p u t i n g 
network abstractions, such as virtual L1 and L2
overlays and security groups. OVN also supports the
security inspection of data transfer inside virtual
networks (for example, packet inspection); hence it
provides extra features useful for increasing custom-
er security and privacy.
Open Issues in Scheduling and Resource
Management
Despite the clear technological advances in con-
tainer and hypervisor-based virtualization technol-
ogies, we are yet to realize a standard large-scale,
performance-optimized scheduling platform for man-
aging an ecosystem of microservices networked to-
gether to create a specialized application stack, such
as a multitier Web application and Internet of Things
(IoT) application. Future efforts will focus on solv-
ing the following research challenges.
Configuration Selection and Management
A cloud application (for example, a multitier Web
application) must typically combine multiple inter-
dependent microservices that provide diverse func-
tionalitiesfor example, load balancer, webserver,
and database server. Moreover, these microservices
have both control and dataflow dependencies.
The challenges exist in dealing with heteroge-
neous configurations of microservices and cloud
datacenter resources driven by heterogeneous
performance requirements. With the increase in
microservice application functionality types (en-
cryption, compression, SQL/NSQL server, virtual
private network, and so on) and the heterogeneity
of container engines (LXC, Docker, Google, and
Amazon) and underlying cloud datacenter resourc-
es, the mapping of microservices to datacenters
demands selecting bespoke configurations from an
abundance of possibilities,5 which is impossible to
resolve manually.
Branded price calculators, available from pub-
lic cloud providers (Amazon and Azure, for ex-
ample) and academic projects (Cloudrado), allow
comparison of datacenter resource leasing costs.
However, these calculators cant recommend or
compare configurations across microservices and
datacenter resources.
We therefore need new research that focuses
on developing techniques for accurately modeling,
83
 Blue Skies
representing, and querying configurations of mi-
croservices and datacenter resources. In addition, we
need general-purpose decision-making techniques,
driven by heterogeneous performance requirements,
to automate the selection of microservice configu-
rations and their mapping to heterogeneous data-
center resources.5
Application Topology Specification and
Composition
To compose a microservices-based application topol-
ogy, you need to describe the microservices using a
well-known standard. For example, you can base mi-
croservice descriptions on the Topology and Orches-
tration Specification for Cloud Applications (Tosca)/
YAML along with the usual image representation.
Moreover, workloads pertaining to different mi-
croservices depend on each other, and changes in one
microservices execution and dataflow will influence
those of others. Overall, the topology specification
and composition needs to cover the whole life cycle
that is, deploy, patch, monitor, reconfigure, and shut-
down driven by the performance objectives of each
microservice as well as the application as a whole.
The Business Process Execution Lan-
guage (BPEL) and Web Service Choreography
Interface (WSCI) are examples of Web service com-
position (agnostic to microservices) languages used
in SOAs. The Resource and Application Description
Language (RADL) is designed for composing and
deploying VM images to different cloud providers.6
Some application topology composition and speci-
fication tools found in literature (Crane, Fig, and
Maestro, for example) cant deploy microservices
across distributed datacenter hosts.2 Although Tosca
supports topology pattern specification, it lacks sup-
port for describing data and control flow dependen-
cies between microservices, with a specific focus on
identifying event coordination and dataflow mecha-
nisms; properties of microservices in terms of work-
load features (such as data format, query rate, and
runtime I/O dependency); and performance objec-
tives and measures relevant to microservices.
Hence, an important research direction is to in-
vestigate an application-agnostic microservices com-
position framework, which will facilitate knowledge
reuse and make it simpler for application engineers
to interact with a complex computing platform.
84 I E E E C l o u d Co m p u t i n g  w w w . c o m p u t e r . o r g / c l o u d c o m p u t i n g
Performance Characterization and Isolation
In a datacenter, microservices can be deployed
inside hypervisor-based VMs or on nonvirtual-
ized physical hardware. A recent study found that
deployment within VMs imposes additional per-
formance overhead while giving no extra benefit
compared to deploying microservice containers on
nonvirtualized physical hardware.7 As noted ear-
lier, single containers, such as Docker, can sup-
port multiple and heterogeneous microservices
that provide various application-specific features
in a containerized environment. In this environ-
ment, unexpected interference and contention
can occur. For some microservices (such as a com-
pression server) storage requirements dominate,
whereas for others (for example, transactional
query processing by database server) computa-
tional requirements dominate, and for still others
(for example, a VPN server) communication re-
quirements dominate. Hence, container schedul-
ing platforms (Kubernetes, Docker Swarm, and so
on) must consider which microservices to combine
to minimize workload interference and conten-
tion. Balancing resource consumption and per-
formance is critical in deciding where to deploy
microservices.
Some recent work has investigated performance
isolation and interference detection. New hard-
ware design techniques change processor cache
architecture partitioning8 or integrate novel inser-
tion policies to pseudo-partition caches to reduce
contention.9
Hardware-based approaches add complexity
to the processor architecture and are difficult to
manage over time. Sriram Govindan and his col-
leagues developed a scheme to quantify the effects
of cache contention between consolidated work-
loads.10 However, they limit their discussion to
cache contention issues, ignoring other hardware
resource types. Ripal Nathuji and Aman Kansal
present a control theory-based consolidation ap-
proach that mitigates the effects of cache, memory,
and hardware prefetching contention of coexisting
workloads.11 However, their focus is CPU-bound or
compute-intensive applications.
Several new research topics are worthy of inves-
tigation: performance isolation and characterization
techniques when multiple microservices run in the
same container or on the same physical host; live
migration of containers to reduce interference and
contention; and tradeoffs between live migration
and restarting.
Microservice Monitoring
Guaranteed application performance requires clear
and real-time understanding of performance met-
rics across microservices and datacenter resources.
However, variations in performance metrics across
different microservices and datacenter resources
complicate this problem. For example, key perfor-
mance metrics for SDN resources are throughput
and latency; for CPU resources, theyre utilization
and throughput; and for SQL and NoSQL da-
tabase microservices, its query response time.
Therefore, how to define and formulate perfor-
mance metrics coherently across microservices to
give a holistic view of data and control flows remains
an open issue.
Monitoring tools that were popular in the grid
and cluster computing era (for example, R-GMA
and Hawkeye) were concerned only with moni-
toring performance metrics at the datacenter re-
source level (such as CPU percentage and TCP/
IP performance), but not at the microservice level
(such as end-to-end request processing latency and
communication overhead). Cluster-wide monitor-
ing frameworks (Nagios, Ganglia, Apache Ha-
doop, and Apache Spark) provide information
about hardware metrics (cluster, CPU, and mem-
ory utilization, and so on) of cluster resources
that might belong to public or private cloud data-
center.12,13 Monitoring frameworks used by the
Amazon EC2 Container Service (Amazon Cloud-
Watch) and Kubernetes (Heapster) typically moni-
tor CPU, memory, filesystem, and network usage
statistics, so they cant monitor microservice-level
performance metrics.
This leads to several new research topics, in-
cluding development of holistic techniques13 for
collecting and integrating monitoring data from all
microservices and datacenter resources so admin-
istrators or a scheduler (a computer program) can
track and understand the impact of runtime uncer-
tainities (for example, failure, load-balancing ef-
ficiency, and overloading) on performance without
understanding the whole platforms complexity.
Sep t ember /O c to ber 2016 I E E E C l o u d Co m p u t i n g 
Elastic Scheduling and Runtime Adaptation
The elastic scheduling of microservices is a com-
plex research problem due to several runtime
uncertainties.
First, its difficult to estimate microservice work-
load behavior in terms of request arrival rate,
type, and processing time distributions; I/O sys-
tem behavior; and number of users connecting to
different types and mix of microservices. The real
challenge in devising microservice-specific work-
load models is to accurately learn and fit statisti-
cal functions to the monitored distributions such
as request arrival pattern, CPU usage patterns,
memory usage patterns, I/O system behaviors, re-
quest processing time distributions, and network
usage patterns.
Without knowing the workload behaviors of
microservices, its difficult to make decisions about
the types and scale of datacenter resources to be
provisioned to microservices at any given time.
Furthermore, the availability, load, and throughput
of datacenter resources can vary in unpredictable
ways, due to failure or congestion of network links.
Kubernetes offers a microservice container re-
configuration feature, which scales by observing
CPU usage (elasticity is agnostic to the workload
behavior and performance objectives of microser-
vices). Amazons autoscaling service employs simple
threshold-based rules or scheduled actions based
on a timetable to regulate infrastructural resources
(for example, if the average CPU usage is above 40
percent, add another microservice container). Other
cloud providers have implemented similar simple
rule-based reactive runtime scheduling techniques:
Googles Cloud Platform autoscaler, Rackspaces
Auto Scale, Microsoft Azures Fabric Controller, and
IBMs Softlayer autoscale.
To the best of our knowledge, no prior work
has developed workload and resource performance
prediction models to enable reconfiguration (scal-
ing, descaling, and migration) of microservices on
cloud datacenters while ensuring microservice-
specific performance objectives. Hence, important
new research is investigating predictive workload
and performance models to forecast workload
input and performance metrics across multiple,
coexisting microservices deployed on cloud data-
center resources.
85
 Blue Skies
Storage and processing services
C1 C2
S1 S2 S3 ... Sn
VM1 VM2 VM3 VM4
IoT cloud provider
...
SA1 SA2 SA3 SAm
C1 C2 Sensing and actuating services
C3 C4 C5
Figure 2. A microservice as the enabler for the IoT application cloud.
IoT application are decomposed into collection of microservices which
are distributed across physical hardware resources available in the cloud
and on the network edge.
Evolution of Microservice-Powered Cloud
Paradigms
Wide-scale adoption of containerization technolo-
gies and microservices architectures will strongly
influence other emerging computing paradigms.
Cloud Computing and Internet of Things
The combination of cloud computing and the IoT
is presenting new opportunities for delivering new
types of application services (see Figure 2). For ex-
ample, private, public, and hybrid cloud providers
are looking to integrate their datacenters software
and hardware stacks with embedded devices (in-
cluding sensors and actuators) to provide IoT as a
service (IoTaaS).
Typically, IoT devices run customized soft-
ware developed with a particular programming
language and/or development framework. Minimal
processing and storage tasks can be performed in
IoT devices (for example, a sensor gateway or SDN
virtualization) by deploying lightweight, contain-
erized microservices.14,15 Meanwhile, the massive
data storage and processing tasks (data mining and
big data analytics) are performed in cloud datacen-
ters that exploit virtualization (both hypervisor and
container-based) to elastically scale up/down storage
and processing capabilities.
86 I E E E C l o u d Co m p u t i n g  w w w . c o m p u t e r . o r g / c l o u d c o m p u t i n g
Federated Clouds
The cloud services market has been growing in re-
cent years, a trend thats confirmed by the number
of cloud providers that have appeared on the market.
Currently, small and medium cloud providers cant
directly compete with the big players (such as Google,
Amazon, and Microsoft), so they must implement
new business strategies to penetrate the market.16,17
In particular, small and medium providers can
establish stronger partnerships to share resources
according to the rules of the cloud federation eco-
system they belong to. Small providers can federate
with large providers to gain economies of scale, op-
timize their assets, scale their capabilities, and share
resources to establish new forms of collaboration. If
a small providers cloud runs out of capacity, it can
migrate its microservices to federated datacenters to
ensure business continuity (see Figure 3).
However, federated clouds need to respond to
high heterogeneity across independent cloud systems,
efficient and secure data exchange among clouds, and
the ability to efficiently deploy resources and services
across such federated systems. Indeed, the dynamism
of a federation with incoming and outgoing providers
and variable resource availability makes microser-
vices and containers the best solution to quickly
adapt to changes in the federated system.
icroservices will simplify orchestration of
networked applications across heterogeneous
cloud datacenters and emerging microdatacenters
(on the network edge). However, the creation of
such applications (for example, smart city and smart
healthcare IoT clouds) requires new research into
scheduling and resource management algorithms
and platforms for managing highly distributed and
networked microservices.
References
1. A Sill, The Design and Architecture of Mi-
croservices, IEEE Cloud Computing, vol. 3, no.
5, 2016, pp. 7680.
2. C. Pahl and B. Lee, Containers and Clusters
for Edge Cloud Architectures: A Technology
Review, Proc. 3rd Intl Conf. Future Internet of
Things and Cloud (FiCloud), 2015, pp. 379386.
3. M. Xavier et al., Performance Evaluation of
 Cloud Home cloud services (IaaS, PaaS, SaaS)
User

Server 1 Server 2 ... Server N

Enterprise
Government

Foreign Foreign
Cloud Home cloud
cloud A cloud B
federation

Foreign cloud A Home cloud Foreign cloud B

virtual infrastructure virtual infrastructure virtual infrastructure

Home cloud
Virtual resources used by Virtual resources owned by capabilities Virtual resources used by
foreign cloud A and placed home cloud and placed in its enlargement foreign cloud B and placed
in its virtual infrastructure virtual infrastructure in its virtual infrastructure
Virtual resources Virtual resources
placed in foreign cloud A placed in foreign cloud B
and rented to home cloud and rented to home cloud

Figure 3. Microservice as the basis of federating multiple cloud datacenters as part of cohesive federation,
where datacenter providers can meet the performance requirements of client applications through optimal
placement and migration of microservices across datacenters.

Container-Based Virtualization for High Per-
formance Computing Environments, Proc.
21st Euromicro Intl Conf. Parallel, Distributed,
and Network-Based Processing (PDP), 2013, pp.
233240.
4. C. Esposito, A. Castiglione, and K.-K.R. Choo,
Challenges in Delivering Software in the Cloud
as Microservices, IEEE Cloud Computing, Vol.
3, no. 5, 2016, pp. 1014.
5. R. Ranjan et al., Cross-Layer Cloud Resource
Configuration Selection in the Big Data Era,
IEEE Cloud Computing, vol. 2, no. 3, 2015, pp.
1622.
6. M. Caballer et al., Dynamic Management of
Virtual Infrastructures, J. Grid Computing, vol.
13, Mar. 2015, pp. 5370.
7. W. Felter et al., An Updated Performance Com-
parison of Virtual Machines and Linux Contain-
ers, Proc. IEEE Intl Symp. Performance Analysis of
Systems and Software (ISPASS), 2015, pp. 171172.
8. M.K. Qureshi and Y.N. Patt, Utility-Based
Cache Partitioning: A Low-Overhead, High-
Performance, Runtime Mechanism to Partition
Shared Caches, Proc. 39th Ann. IEEE/ACM
Intl Symp. Microarchitecture (Micro 06), 2006,
pp. 423432.
9. Y. Xie and G.H. Loh, Pipp: Promotion/Inser-
tion Pseudo-Partitioning of Multi-Core Shared
Caches, Proc. 36th Ann. Intl Symp. Computer
Architecture (ISCA 09), 2009, pp. 174183.
10. S. Govindan et al., Cuanta: Quantifying Ef-
fects of Shared On-Chip Resource Interference
for Consolidated Virtual Machines, Proc. 2nd
ACM Symp. Cloud Computing (SOCC 11), 2011,
article 22.
11. R. Nathuji and A. Kansal, Q-Clouds: Manag-
ing Performance Interference Effects for QoS-
Aware Clouds, Proc. 5th European Conf. Com-
puter Systems (EuroSys 10), 2010, pp. 237250.
12. R. Ranjan, Streaming Big Data Processing in

Sep t ember /O c to ber 2016 I E E E C l o u d Co m p u t i n g  87

 Blue Skies
Datacenter Clouds, IEEE Cloud Computing,
vol. 1, no. 1, 2014, pp. 7883.
13. M. Natu et al., Holistic Performance Monitor-
ing of Hybrid Clouds: Complexities and Future
Directions, IEEE Cloud Computing, vol. 3, no.
1, 2016, pp. 7281.
14. A. Celesti et al., Exploring Container Virtu-
alization in IoT Clouds, Proc. 2016 IEEE Intl
Conf. Smart Computing (SmartComp), 2016, pp.
16.
15. M. Fazio and A. Puliafito, Cloud4sens: A Cloud-
Based Architecture for Sensor Controlling and
Monitoring, IEEE Comm, vol. 53, Mar. 2015,
pp. 4147.
16. M. Assis and L. Bittencourt, A Survey on Cloud
Federation Architectures: Identifying Function-
al and Non-functional Properties, J. Network
and Computer Applications, vol. 72, 2016, pp.
5171.
17. A. Celesti et al., Characterizing Cloud Fed-
eration in IoT, Proc. 30th Intl Conf. Advanced
Information Networking and Applications Work-
shops (WAINA), 2016, pp. 9398.
Maria Fazia is an assistant researcher of computer
science at the University of Messina. Her research in-
terests include distributed systems and wireless com-
munications, especially with regard to the design and
development of cloud solutions for IoT services and
applications. Fazia has a PhD in advanced technolo-
gies for information engineering from the University
of Messina. Contact her at mfazio@unime.it.
Antonio Celesti is a postdoctoral researcher at
University of Messina. His research interests include
distributed systems and cloud computing, with par-
ticular regard to federation, storage, security, energy
efficiency; and assistive technology. Celesti has a PhD
in advanced technology for information engineering
from the University of Messina, Italy. Contact him at
acelesti@unime.it.
Rajiv Ranjan is a reader in the School of Com-
puting Science at Newcastle University, UK; chair
professor in the School of Computer, Chinese Uni-
versity of Geosciences, Wuhan, China; and a visiting
scientist at Data61, CSIRO, Australia. His research
88 I E E E C l o u d Co m p u t i n g  w w w . c o m p u t e r . o r g / c l o u d c o m p u t i n g
interests include grid computing, peer-to-peer net-
works, cloud computing, Internet of Things, and big
data analytics. Ranjan has a PhD in computer science
and software engineering from the University of Mel-
bourne (2009). Contact him at raj.ranjan@ncl.ac.uk
or http://rajivranjan.net.
Chang Liu is a research fellow (assistant professor)
at Newcastle University, UK. His research interests in-
clude cloud computing, big data, distributed systems,
Internet of Things, and information security and pri-
vacy. Liu has a PhD in information technology from
the University of Technology, Sydney, Australia. Con-
tact him at changliu.it@gmail.com.
Lydia Y. Chen is a research staff member at the
IBM Zurich Research Lab, Zurich, Switzerland. Her
research interests include modeling, optimizing per-
formance and dependability for big data applica-
tions and highly virtualized datacenters. She received
a PhD in operations research from the Pennsylvania
State University. Contact her at yic@zurich.ibm.com.
Massimo Villari is an associate professor of
computer science at the University of Messina. His re-
search interests include cloud computing, Internet of
Things, big data analytics, and security systems. Vil-
lari has a PhD in computer engineeringfrom the Uni-
versity of Messina. Hes a member of IEEE and IARIA
boards. Contact him at mvillari@unime.it.
Read your subscriptions through
the myCS publications portal at
http://mycs.computer.org.