31/10/2022 01:56 Serverless execution of scientific workflows: Experiments with HyperFlow, AWS Lambda and Google Cloud Functions

Serverless execution of
scientific workflows:
Experiments with HyperFlow,
AWS Lambda and Google Cloud
Functions
Scientific workflows consisting of a high number of
interdependent tasks represent an important class of
complex scientific applications. Recently, a …
Ver original

Highlights
•

We evaluate the applicability of cloud functions and serverless

architectures for execution of scientific workflows.

We developed a prototype which combines HyperFlow engine

with AWS Lambda and Google Cloud Functions.

Experiments with Montage application show the good

scalability and low overheads of serverless approach.

We discuss implications of serverless architectures for resource

management.

Abstract

https://www.sciencedirect.com/science/article/abs/pii/S0167739X1730047X 1/5
31/10/2022 01:56 Serverless execution of scientific workflows: Experiments with HyperFlow, AWS Lambda and Google Cloud Functions

Scientific workflows consisting of a high number of interdependent

tasks represent an important class of complex scientific
applications. Recently, a new type of serverless infrastructures has
emerged, represented by such services as Google Cloud Functions
and AWS Lambda, also referred to as the Function-as-a-Service
model. In this paper we take a look at such serverless
infrastructures, which are designed mainly for processing
background tasks of Web and Internet of Things applications, or
event-driven stream processing. We evaluate their applicability to
more compute- and data-intensive scientific workflows and discuss
possible ways to repurpose serverless architectures for execution of
scientific workflows. We have developed prototype workflow
executor functions using AWS Lambda and Google Cloud Functions,
coupled with the HyperFlow workflow engine. These functions can
run workflow tasks in AWS and Google infrastructures, and feature
such capabilities as data staging to/from S3 or Google Cloud Storage
and execution of custom application binaries. We have successfully
deployed and executed the Montage astronomy workflow, often
used as a benchmark, and we report on initial results of its
performance evaluation. Our findings indicate that the simple mode
of operation makes this approach easy to use, although there are
costs involved in preparing portable application binaries for
execution in a remote environment.

While our solution is an early prototype, we find the presented

approach highly promising. We also discuss possible future steps
related to execution of scientific workflows in serverless
infrastructures. Finally, we perform a cost analysis and discuss
implications with regard to resource management for scientific
applications in general.

Keywords

Scientific workflows

Cloud functions

https://www.sciencedirect.com/science/article/abs/pii/S0167739X1730047X 2/5
31/10/2022 01:56 Serverless execution of scientific workflows: Experiments with HyperFlow, AWS Lambda and Google Cloud Functions

Serverless architectures

FaaS

Maciej Malawski holds a Ph.D. in Computer Science along with

an M.Sc. in Computer Science and Physics. He is an assistant
professor and researcher at the Department of Computer Science
AGH and at ACC Cyfronet AGH, Krakow, Poland. In 2011–12 he was
a postdoc and a later a visiting faculty at the Center for Research
Computing, University of Notre Dame, USA. He is the coauthor of
over 50 international publications, including journal and conference
papers and book chapters. He participated in EU ICT Cross-Grid,
ViroLab, CoreGrid, VPH-Share and PaaSage projects. His scientific
interests include parallel computing, grid and cloud systems,
resource management, and scientific applications.

Adam Gajek is a Master student at AGH University of Science and

Technology in Krakow, Poland. He received his bachelor degree
from AGH in Computer Science in 2016. In 2015 he participated in
EU ICT PaaSage project, where he worked on deployment of cloud
application and monitoring of a cloud application in collaboration
with Lufthansa Systems. His interests include functional
programming and adopting modern architectural design patterns to
business needs.

https://www.sciencedirect.com/science/article/abs/pii/S0167739X1730047X 3/5
31/10/2022 01:56 Serverless execution of scientific workflows: Experiments with HyperFlow, AWS Lambda and Google Cloud Functions

Adam Zima is a Master student at AGH University of Science and

Technology in Krakow, Poland. He received his bachelor degree in
Computer Science from AGH in 2016. He has experience as a
developer of scalable Web applications using modern software
stacks. His interests include cloud computing, functional
programming and serverless architectures.

Bartosz Balis Ph.D. in Computer Science, is an Assistant Professor

at the Department of Computer Science, AGH University of Science
and Technology, Krakow, Poland. Co-author of 80 international
publications including journal articles, conference papers, and book
chapters. His research interests include environments for eScience,
scientific workflows, grid and cloud computing. Participant of
international research projects including EU-IST CrossGrid,
CoreGRID, K-Wf Grid, ViroLab, Gredia, UrbanFlood and PaaSage.
Member of Program Committee for Conferences (e-Science 2006,
ICCS 2007–2017, ITU kaleidoscope 2013–2017, SC16 Workshops,
SIMULTECH 2016–2017).

Kamil Figiela is a Ph.D. student at AGH University of Science and

Technology in Krakow, Poland. He received his M.Sc. degree in
https://www.sciencedirect.com/science/article/abs/pii/S0167739X1730047X 4/5
31/10/2022 01:56 Serverless execution of scientific workflows: Experiments with HyperFlow, AWS Lambda and Google Cloud Functions

Computer Science in 2013. In summer 2011 he participated in

Research Experience for Undergraduates program at Center for
Research Computing, University of Notre Dame. His research
interests include cloud computing, optimization, and functional
programming.

View full text

© 2017 Elsevier B.V. All rights reserved.

https://www.sciencedirect.com/science/article/abs/pii/S0167739X1730047X 5/5