Process-as-a-Service: FaaSt Stateful Computing with Optimized Data Planes
Marcin Copik Alexandru Calotoiu
ETH Zürich ETH Zürich
Torsten Hoefler
ETH Zürich
Abstract
Fine-grained and ephemeral functions power many new
applications that benefit from elastic scaling and lower
computing costs of serverless platforms. However, they
are hampered by expensive and limited communication,
and high invocation latency. Functions cannot keep track
of state across invocations and must rely on remote stor-
age, making workflows and applications with dependen-
cies between tasks, or relying on communication be-
tween workers, difficult to implement using Function-as-
a-Service (FaaS) computing.
To continue the serverless revolution, we introduce the
concept of Process as a Service (PraaS) and show how
established operating system abstractions can be adapted
to model and implement dynamically provisioned cloud
computing workers. We present the new serverless data
plane that improves invocation performance by up to
32 times while preserving the ephemeral and elastic na-
ture of serverless workers. Finally, we build a unified
and portable communication interface for serverless, en-
abling optimized peer–to–peer communication allowing
workers to solve problems in parallel. to save costs.
1 Introduction
In less than a decade, Function-as-a-Service (FaaS), has
established itself as one of the fundamental programming
models in the cloud. In FaaS, users invoke short-running
and stateless functions and benefit from the pay–as–you–
go billing and lower costs, while cloud providers gain
more efficient resource usage and opportunities to reuse
idle hardware [1–3]. Serverless functions, the computing
model FaaS is based on, have been used in web applica-
tions, media processing, data analytics, machine learn-
ing, and scientific computing [4–9]. Even though FaaS
has achieved remarkable success in decreasing the costs
of stateless computations and microservices, the wide
adoption of serverless computing is hindered by the lim-
Rodrigo Bruno Roman Böhringer
INESC-ID, IST, ULisboa ETH Zürich
itations of its execution model [4, 10–12].
As an example of a trending workload in serverless
systems, we consider distributed machine learning so-
lutions [4, 13–16]. There, each invoked function must
compute new gradients using a subset of data. While
the stateless nature of FaaS simplifies deployment and
resource management, combining the new weights and
using them in a next round of invocations incurs major
performance overheads as in each round of updates the
output must be written and read from external cloud stor-
age, and each new round requires an invocation that goes
through the entire cloud control plane.
Without direct communication channels, a service like
serverless federated machine learning is difficult to im-
plement and inefficient. The lack of state management in
ephemeral functions [10, 12] can partially be addressed
with workarounds such as storage-based shared states,
transactions, and logs [17–20]. However, these solutions
rely on manually managed non-serverless storage in-
stances, and they negatively impact the execution times,
throughput, and cost of serverless systems [12, 20]. The
performance of such systems could be vastly improved
by taking advantage of local available memory rather
than constantly needing to access remote, slow storage
and by having the capability to bypass the control plane
for repeated invocations.
We propose a new paradigm in the serverless world:
Process-as-a-Service (PraaS) — we incorporate state
seamlessly, preserve the ephemeral nature of FaaS com-
putations, and overhaul the serverless system architec-
ture. PraaS is heavily inspired by classical Operating
System (OS) design and transfers concepts that stood the
test of time into the context of granular cloud computing.
Specifically, this new model allows us to address three of
the most significant constraints of current FaaS systems:
lack of consistent and low-latency state, a complex con-
trol plane involved in each invocation, and a lack of uni-
fied and portable communication interface.
PraaS is a new step in the evolution of cloud com-
 IaaS CaaS PraaS FaaS
Computation Unit Virtual machine Container Process Function
External Interface SSH, TCP, HTTP, RPC, RDMA SSH, TCP, HTTP, RPC HTTP, TCP HTTP
Lifetime Months Days, hours Minutes, hours Seconds
State Duration Persistent Transient Transient Ephemeral
State Location Local disk, memory Memory, cloud storage Memory, cloud storage Cloud storage
Provisioning Manual, minutes Semi-automatic, secs Automatic, msecs Automatic, msecs
Compute Resources Persistent Persistent Ephemeral Ephemeral
Billing Provisioned Provisioned Pay-as-you-go Pay-as-you-go

Figure 1: Evolution of computing platforms in the cloud - PraaS enables state persistence for ephemeral workers.

puting that provides a better balance between the per-
formance of persistent allocations and elasticity of
ephemeral workers (Fig. 1). It introduces the concept
of a session, an execution environment that retains local
state across invocations, and that exposes a communica-
tion channel so a user can send data and requests directly.
Processes open the possibility for direct warm func-
tion invocation. In traditional FaaS platforms, ev-
ery invocation traverses the centralized (and multi-
step) allocation and routing logic in the control plane
(Sec. 2.1). Moreover, serverless functions are mostly
short-running [21] and the increasing availability of fast
network transport puts further pressure on optimizing the
serverless control plane involved in every invocation. In
PraaS, communication channels exposed by sessions al-
low us to expose a direct connection to a function execut-
ing within a session. Thus, we propose a process data
plane that offers low latency warm executions over a di-
rect connection to function sandbox. Instead of apply-
ing optimizations to decrease the control overheads, we
remove the control plane overheads from the data path
entirely [22] by submitting invocation payload directly
to a process instance. The new data plane we propose
enables parallel and granular computing with serverless
functions [23, 24], where the relative influence of invo-
cation overheads is very high.
A further benefit of PraaS is the ability to move data
directly between sessions within a single process. This
allows users to deploy large-scale applications such as
distributed ML that can communicate directly by access-
ing state memory. Until now, inter-function commu-
nication has been constrained, forcing users to imple-
ment standard communication patterns on top of cloud
storage [4, 8] and introducing additional costs, non-
negligible latencies, or requiring non-serverless or even
manually managed storage services. Inspired by par-
titioned global address space (PGAS), we propose a
global memory model for serverless functions backed
by reliable and persistent storage. We provide a com-
munication model for serverless that adjusts the require-
ments of direct messaging to ephemeral functions and
enables fast communication in a portable and abstract
interface effectively replacing the functions traditionally
provided by the control plane: direct invocations using
PraaS are 32 times faster than using AWS Lambda, for
both small and large payload sizes.
To summarize, we make the following contributions:
• The new concept of serverless processes that pro-
vides an efficient model for stateful FaaS functions.
• An optimized serverless data plane with fast and
direct invocations for scalable workflows and high-
performance FaaS.
• A global memory model for serverless functions
that permits optimized function–to–function com-
munication and provides an interface portable
across systems and cloud providers.
2 Motivation
T Cloud
Gateway Controller Events
Load Balancer M
H Metadata
O W C
Sandbox
Function Server Function Server
Figure 2: FaaS control plane: each invocation includes
the management overhead ( M ).
Function-as-a-Service (FaaS) enhanced clouds with a
fine-grained and elastic programming model and FaaS
has found its way to the major cloud systems [25–28].
Functions are stateless, and invocations cannot rely on
resources and data resulting from previous executions.
Instead of using persistent and user-controlled virtual
machines and containers, function instances are placed
dynamically in cloud-managed sandboxes, e.g., contain-

2
ers and lightweight virtual machines [29]. A cold invo-
cation requires allocation of a new sandbox adding sig-
nificant overhead to the invocation latency [11, 30]. The
consecutive warm invocations can benefit from reusing
sandboxes by a preceding cold invocation, achieving
lower invocation latency. Therefore, cloud systems at-
tempt to retain virtual machines and containers after
the invocation, employing complex and sophisticated re-
tain and prewarming strategies [31–34], and trading off
higher memory consumption for faster executions. In
addition, the flexible pay-as-you-go billing is another
significant advantage of serverless systems: users are
charged only for computation time and resources used.
However, along with the benefits serverless computing
provides, it does have some disadvantages such as higher
latencies due to complex control planes, poor locality of
data due to the nonexistence of local state, and high com-
munication costs. [10, 11, 35, 36].
2.1 Serverless Control Plane
The serverless execution model requires that the func-
tion executors are allocated dynamically, and users do
not provision cloud resources. In practice, most modern
FaaS systems implement this as a centralized manage-
ment and routing system that moves user data to a func-
tion instance. Figure 2 presents a high-level overview
of the serverless control plane. It includes an abstrac-
tion of a REST API and a gateway with a persistent
network address, and uses an HTTP connection ( H ) to
hide the on-the-fly selection and allocation of function
executors. Cloud events and gateway requests trigger
( T ) function invocations. The function’s input is for-
warded to the central management ( M ) responsible for
authorization, allocation of resources, and routing execu-
tion to the selected server. In AWS Lambda, the control
logic is responsible for authorizing requests, managing
sandbox instances, and placing the execution on a cloud
server [29]. In OpenWhisk [37], the critical path of func-
tion execution is even longer. Each invocation includes
a frontend webserver, controller, database, load balancer,
and a message queue [38]. Finally, the function’s data
moves to a warm ( W ) or cold ( C ) sandbox. When the
execution is finished, the function’s output returns to the
user via the gateway ( O ).
The many steps of control logic add double-digit mil-
lisecond latency to each invocation and require copying
user payload multiple times, even though serverless sys-
tems optimize cold startups, and subsequent invocations
reuse the same warm sandbox when available. Over-
heads of the control plane can dominate the execution
time, and are much higher than the network transmis-
sion time needed to transfer input arguments [11]. The
long and complex invocation path is even more visible
3
Storage Type 1B 1 MB 10 MB
Persistent storage (AWS S3) 10.3 20.4 102.4
Key-value storage (AWS DynamoDB) 34 n/a n/a
In-memory cache (AWS ElastiCache) 0.9 24.6 84.6
Table 1: Access time [ms] to remote cloud storage from
an AWS Lambda Function with 2 GiB RAM.
in distributed applications and serverless workflows that
invoke functions many times with large payloads. The
optimization effort of serverless workflows is guided to-
wards reusing function instances and provisioning ded-
icated stateful executors [39–42], but does not offer
a solution can be generalized. Alternative approaches
include decentralized scheduling and direct allocations
[3, 39], but they are limited to systems optimized for
serverless workflows and RDMA acceleration.
Observation: Every serverless invocation incurs
overheads for authentication, resource management, and
data copying across cloud services. However, the func-
tion placement does not change in many warm invoca-
tions, making the control logic unnecessary.
2.2 Serverless State
The stateless nature of functions makes them easy to
schedule for cloud providers, but puts significant con-
straint on the programmability FaaS systems. Comput-
ing resources are allocated with ephemeral memory stor-
age that cannot be guaranteed to persist across invoca-
tions. Since many applications require retaining state be-
tween invocations, stateful functions place their state in
remote cloud storage [18–20]. While function’s state is
located in storage far away from the compute resources,
fetching and updating state adds dozens of milliseconds
of latency to the execution (Table 1), resulting in signifi-
cant performance overheads [19, 20].
Relying on remote storage to implement stateful func-
tions requires every invocation to incur into storage over-
heads, even if function sandboxes are kept warm. With
the current FaaS model, it is not possible to overcome the
remote storage access from the data path.
Observation Restricting serverless computing to
stateless functions puts unnecessary constraints on user
applications prevents data locality. Stateful functions
help mitigate some of these challenges, but their efficient
implementation is hindered by in the current FaaS model.
2.3 Serverless Communication
Communication in FaaS has always been constrained.
The message-passing paradigm cannot be applied di-
rectly to the ephemeral functions, as workers’ lifetime is
not defined, and functions require mailboxes to store and
 500
PraaS Concept Inspiration

Communication Time [ms]

2
10
Throughput [MB/s]
400 Process POSIX process model.
300 1
Function Thread in a process.
10
State POSIX process memory.
200 Session Persistence Swapping memory pages.
0
10 Communication channel Socket in a process.
100
Communication model Remote memory access, TCP
0 10
1 Network data plane in Arrakis [22],
Data plane
10
3
10
0
10
1
10
2
10
3
10
4
10
5 kernel bypass in RDMA.
Data Size [KB]
Table 2: In PraaS, we show how the major concepts of
Figure 3: Latency and throughput for peer-to-peer TCP systems design can be used to lift the process model into
communication between Lambda functions. a distributed and serverless space.
Control Plane
access messages. Unfortunately, the widely-used com-
munication pattern with persistent cloud storage adds 1 PraaS Process
Subprocess 0
non-trivial latencies and costs [10]. While direct net-
Session Controller
work communication between functions could alleviate
Session 0 Session 1
the performance issues (Fig. 3), functions do not offer
Function Function Function Function
the abstractions needed to implement communication on
2
groups of functions with dynamic group membership. Object Object Object Object Object Object
Furthermore, in typical FaaS deployments, functions op- 3 Session Memory Session Memory
erate behind NAT and cannot accept incoming connec- Communication Communication
Channel Channel
tions. UDP and TCP hole punching [43, 44] must be
applied to establish communication between function in-
stances. Not only is it a complex process, but the connec-
tions can only be created between two active functions - Figure 4: The process model in PraaS: ephemeral
the programming model of FaaS does not provide mail- functions are executed in the context of sessions with
boxes needed to send messages reliably. shared and transient state and communication channels
Observation Cheap and scalable communication is for quick user access.
necessary to build parallel and distributed applications,
but serverless programming models lack an efficient fab- PraaS functions implement the semantics of
ric and interface for communication. ephemeral workers known from FaaS systems. Our
goal is to allow quick connections and offer a simple
way to manage a function’s state. To achieve this goal,
Summary FaaS can be much more than just a platform each process function invocation is associated with a
for irregular and lightweight workloads, and the server- process session. When process sessions are allocated
less programming model aligns well with requirements within a process, they are assigned a unique identifier,
for massive parallel and granular computing [23, 24]. a communication channel, and a user-defined amount
Nevertheless, serverless systems must first overcome of memory. After the first invocation is hosted in a
critical limitations: complex control plane involved in session, the communication channel is returned to the
every invocation, lack of a fast and cheap function state, user ( 1 ). Subsequent invocations can then bypass the
and expensive, storage-based communication. control plane and use the PraaS data plane ( 2 ). This
allows functions to use data stored by previous function
invocations in the session state, and communicate
3 PraaS: Process as a Service with other concurrent functions running in the same
session( 3 ).
In this section, we present the process as a service model
starting with the core design overview and then providing
details for its features and components. A process (Fig- 3.1 Session and State
ure 4) is a natural extension of ephemeral functions with
a decoupled and transient state. A process can support In FaaS, in order to increase performance the aim is to
multiple users, isolated from each other if necessary, as- retain containers and decrease the frequency of cold star-
sociated with one tenant. Multiple sessions within PraaS tups. Similarly, in PraaS, we enable opportunistic opti-
can cooperate using globally accessible process mem- mizations with session-based direct invocations while al-
ory,using methods described in Sec. 4.1. lowing cloud providers to scale resources transparently.

4
Sessions encompass a unique identifier, the communica-
tion channel used to route invocations to already warm
containers, and the transient state memory that is avail-
able to functions. This memory is guaranteed to be lo-
cally accessible to functions invoked in the context of
the session, ensuring fast access. For this reason, ses-
sion memory is statically allocated at creation. Sessions
can dynamically allocate additional process memory on
demand (Sec. 4.1).
Isolation and Sharing Sessions allow PraaS to handle
workloads associated with different users and workflows
in a single process instance. Each session has its own
private memory that is isolated from other sessions (each
session has its own page table). When a function is in-
voked, it can only access memory within it’s session, as
shown in Figure 4 - thus functions within a session can
share memory and cooperate. The user is responsible for
avoiding data races (e.g. locking). Functions cannot ac-
cess the memory of a different session directly, but can
cooperate with functions within other sessions by using
global process memory, which we detail in Sec. 4.1.
Sessions can handle multiple invocations simultane-
ously, but are bound by the memory and compute limi-
tations of the system they run on. Cloud providers can
limit the number of simultaneous invocations an individ-
ual session can start, for example in relation to the mem-
ory the session has allocated. To allow larger workflows,
we discuss scalable management and communication of
multiple sessions within a process in Sec. 4.1.
Example We consider the widely-used pattern of us-
ing FaaS where persistent and IaaS-based applications
offload computational workload to cheap and elastic
serverless accelerators [3, 45]. A user can employ the
same session to invoke functions, and these functions
will access the same shared memory pool. The session
model in PraaS allows consecutive or concurrent invoca-
tions to always access the warm and initialized applica-
tions’ state. Furthermore, sessions enable enhanced data
sharing, e.g., reusing a large machine learning model
across many inference functions, allowing for local com-
munication and decreasing management overhead.
Swapping. Sessions are not persistent: their lifetime
is limited and they can be removed at any point by the
cloud provider. However, sessions are virtualized: their
memory is shared across invocations within a single ses-
sion and it does not perish once the sandbox - container
or a virtual machine - is removed.
Instead, sessions are swapped out to persistent cloud
storage - and the standard policy is to swap sessions after
a period of inactivity. Within a process, selecting differ-
ent sessions allow users to isolate their private data from
other users even if belonging to the same tenant.
5
Furthermore, a process is considered active when it
has active sessions, and we will not evict a process un-
til all of its sessions are swapped out. Sessions provide
a natural way of defining process lifetime and selecting
process containers for eviction. By default, when a ses-
sion is swapped out to storage by the session controller,
all current state information is written to persistent cloud
storage under a unique session ID. If a session is no
longer required, the user can also completely remove a
session along with any associated state information.
Users have the option to create user-defined check-
pointing handlers that define how a sessions’ state should
be stored into persistent and reliable cloud storage with
the session ID to minimize the storage required.
When a function is invoked, the user can include
the session identification to recover from a saved state.
When a new process is allocated, function invocations
can be queued along with their session IDs, such that
the sandbox initialization can be overlapped with down-
loading the state from storage, helping to hide the startup
latency for stateful functions.
3.2 Function Invocation Handling
PraaS functions exist in the context of sessions, and it is
the session they are associated with that determines how
they are handled. If the session is active, the user con-
nects directly to the correct communication channel, by-
passing the control plane altogether and ensuring a zero-
copy path from the user to function memory.
If the process is not running that session, the control
plane is involved: the system then checks whether state
information matching that session already exists in stor-
age. If yes, the session is swapped in and the invocation
can use all information stored in the session. Otherwise,
a new session is allocated. In both of these instances, the
communication channel information that allows fast sub-
sequent invocations is forwarded to the user. We multi-
ple functions can share a communication channel within
a session via multiplexing.
Local dispatch We allow users to specify a direct de-
pendency on the result of one or many previous invoca-
tions in each execution request. The execution request
is stalled when there is a match between specified de-
pendencies and active invocations. Co-locating invoca-
tions is an important optimization technique in server-
less workflows [41], and the local encoding of depen-
dencies avoids interactions with workflow orchestrators.
Furthermore, dependencies allow for offloading func-
tion invocations while preserving a first-in, first-out or-
der of invocations. To implement FIFO executions in the
current serverless model, users must manually wait for
 Control Plane
1 PraaS Process
Subprocess 0 2 3
Session Controller
Session 0 Function
4 Communication
Channel
Figure 5: PraaS data plane: bypassing the control plane
allows for zero-copy invocations.
the previous invocation or use a FIFO queue with func-
tion triggers. Both options incur additional latencies and
costs. Meanwhile, PraaS function dependencies allow
dispatching a pipeline of functions with minimal invo-
cation latency, keep function code integrity intact, and
avoid unnecessary fusions.
Multi-source functions Serverless functions can be
seen as unary operations, but the practice has shown that
many aggregation and workflow functions receive input
from more than one function predecessor.
In PraaS, we provide native support for functions with
a variable number of arguments. Since the session state
is decoupled from function workers, the state can effi-
ciently receive and store function arguments from multi-
ple producers. Thus, PraaS can start a multi-source func-
tion as soon as all dependencies are fulfilled. The multi-
source functions can be used in serverless workflows to
efficiently implement data flow between functions.
Example Aggregation functions are used to imple-
ment the reduce and allreduce functionalities in server-
less workloads [4, 8, 46]. Each aggregation function re-
ceives input from more than one predecessor function. In
FaaS, users must use workflow orchestrators and cloud
storage triggers to handle aggregation, as functions can-
not be invoked simultaneously. FaaS lacks support for
more complex logic, i.e., functions are invoked on each
new file in the storage, and they must manually check
if all arguments have already been placed in the storage.
In PraaS, users define an n-ary reduce function instead.
When clients invoke the reduce function through a sin-
gle session, they attach the same invocation identifica-
tion, and the process controller counts the delivered ar-
guments. The function is invoked as soon as all input
arguments arrive. The reduction is computed within a
single process, saving the costs and latencies introduced
by cloud storage.
3.3 PraaS Data Plane
PraaS helps to minimize FaaS latencies with the new
concept of a serverless data plane optimized for low-
latency and high-throughput invocations (Fig. 5). A
full serverless invocation needs to authorize requests, se-
6
lect and optionally allocate resources, and redirect users’
payload to the function executor. When many execution
requests are redirected to the same warm container, the
repeated control operations are redundant and could then
be removed from consecutive invocations. Therefore, for
as long as the authorization stays valid, the invocation
can bypass the control operations entirely and move data
directly from the user to the function executor in a pro-
cess.
In PraaS, the first time a function is invoked ( 1 ), the
control plane notifies the session controller on an avail-
able subprocess to allocate a session ( 2 ), and the server-
less system performs all configuration and authorization
tasks at the point. The user receives connection informa-
tion to be able to easily and quickly reconnect ( 3 ). Each
consecutive function in the same session is invoked via a
direct connection to the process ( 4 ), as long as the ses-
sion stays alive. Function invocation received data from
the user to the session’s state memory, and the zero-copy
approach helps to achieve high throughput on larger pay-
loads. Thus, invocation latency is bounded only by the
network fabric and the performance of function thread
allocation.
3.4 Scaling Sessions
The PraaS process is a virtual, distributed entity. To en-
sure a practical, efficient implementation we introduce
subprocesses, representing the actual containers used to
host sessions. Subprocesses are invisible to users, and
are automatically managed by the cloud provider. The
cloud provider can spawn additional subprocesses to
handle an increased load if the number of concurrent
sessions within a process is high, can remove subpro-
cesses with no active sessions, and even compact ses-
sions from multiple subprocesses unto one subprocess to
improve resource utilization. For this purpose, the pro-
cess also contains a global controller keeping track of
subprocesses along with the sessions they hosted.
Migration & Compaction. Sessions distributed
across subprocesses to allow for better load balancing
and resource retrieval by the cloud provider. Clients with
active connections to a session being migrated receive a
packet with migration details, and they can establish a
connection to the new communication channel.
4 Serverless Communication in PraaS
Serverless computing has been adopted to support dis-
tributed applications and workflows, but the limitations
of the FaaS programming model require complex and
domain-specific optimizations in scheduling and com-
munication [47–49]. Most advances in serverless com-
puting focus on optimizing north-south communication
rather than east-west communication.
FaaS applications include a few communication pat-
terns - message-passing, usually implemented as propa-
gation of a function’s output to the input of a consecutive
invocation [49]; an aggregation of the outputs from many
functions [46, 50], and using data cached in nearby in-
stances for bypassing storage and faster access [42, 51].
In contrast, PraaS provides global memory, an effi-
cient and direct communication model for serverless ap-
plications (Sec. 4.1) bypassing cloud storage but backed
up by it. Our communication model does not concentrate
on process invocations since they have a limited lifetime,
but it is focused on data movement operations, allow-
ing dynamically reshapable applications to benefit from
peer–to–peer communication.
4.1 Global Memory
In PraaS, we provide a global communication abstrac-
tion layer to enable optimized process session-to-session
data exchange and hide the complexity of serverless
communication. This communication layer is inspired
by MPI Remote Memory Access [52]. We present the
overview of the global memory system in Figure 6. Apart
from the memory sessions can privately use, which is
guaranteed to be located on the subprocess where the ses-
sion is hosted, sessions can also write to global memory.
The system provides no guarantees on which subprocess
a region of global memory is allocated. The global con-
troller stores the location of global memory objects. The
global memory is backed up by persistent cloud storage.
Memory objects are accessed through a memory inter-
face which hides data operations conducted with storage
and peer–to–peer communication.
Global controller The global memory layout imple-
ments the message-passing paradigm in serverless by re-
placing storage operations with direct communication.
We include a global controller service as an external, per-
sistent component of the system (Fig. 6), and make it re-
sponsible for monitoring global memory object location
across subprocesses and relaying requests from sessions
to local memory controllers within subprocesses.
Interface Process functions interact with the global
process memory using a simple interface that resembles
memory addressing operations in OS-level processes. In
particular, read and write operations accept an identifier
from a global namespace (think of the address space in
OS-level processes). It is also possible to allocate and
de-allocate global memory using an interface similar to
malloc and free.
7
Global Memory Controller
PraaS Process 2
Subprocess 0 Subprocess 1
Local Memory Controller Local Memory Controller
Process Memory
0
Process Memory
1 Object Object
Session 0 Session 1 3 Session 2
Function Function Function
Figure 6: PraaS Global Memory.
Communication Storage-based communication pat-
terns are slow and expensive [10, 53, 54], but the
function–to–function communication is difficult to main-
tain in FaaS due to the ephemeral nature of workers.
Sessions can communicate with other sessions by
reading and writing memory objects in the global pro-
cess memory ( 1 ). To establish such a communication
channel, a session requests the location of an object from
the global communication controller ( 2 ). The global
communication controller instructs the local memory
controller on the subprocess where the object is hosted
and the originating session to create the communication
channel ( 3 ). Therefore, any such operation does not
require sender and receiver parties to push updates to ex-
ternal services and poll for changes, respectively. Since
the global memory objects identify communication tar-
gets, we establish message-passing without identifying
and enumerating ephemeral and unreliable functions.
Furthermore, the communication replaces storage read
operations with sharing objects between process sessions
since p2p communication achieves higher bandwidths
and lower costs.
Example We consider a group of functions that com-
pute partial results, apply a global reduction to synchro-
nize the progress, and continue with the next batch of
tasks, as is the case in the distributed machine learn-
ing training [4]. In FaaS, functions must resort to stor-
age to communicate results. Furthermore, the lack of
state means that functions are invoked again for the it-
eration and users accept the penalty of reinitialization, or
functions are artificially kept alive to keep the data local-
ity and they incur costs from unnecessary waiting in the
communication round. The PraaS model provides an al-
ternative that is more affordable and more efficient at the
same time: functions store objects corresponding to their
partial results in the shared session or process memory,
allowing them to move reduction data directly to the des-
tination, and they stop incurring computing charges im-
mediately afterward. However, since session state is not
immediately evicted, functions benefit from the warm
environment and better data locality in the next iteration.
Dynamic subprocess membership In PraaS, changes
to the subprocess configuration can be triggered by the
cloud provider to scale the system up or down according
to demand. Each change must be atomic to ensure the va-
lidity of exchanged messages. The global memory con-
troller stores where global memory objects are located
across all subprocessses, and this information is updated
when the overall subprocess configuration changes. We
define this as an epoch change, and results in a two step
operation. The controller announces to all subprocesses
that a configuration change is necessary, the subpro-
cesses notify their sessions. Once all outstanding com-
munication of the last epoch is handled the sessions no-
tify their subprocesses. Then, the subprocesses notify the
controller, the configuration is changed and the subpro-
cesses and their sessions are then notified of the location
of all subprocess sessions. Thus, each change in status
begins a new epoch, and the global memory controller
distributes epoch change announcements to all partici-
pating subprocesses. The change notification allows ap-
plications to readjust messaging policies gracefully and
avoid data failures in the presence of ephemeral workers.
5 PraaS in Practice
In this section we will discuss how our proposed design
can be implemented in practice, focusing on four major
aspects: the container serving our runtime, the control
plane, how global memory works and the client library.
5.1 Control Plane
The main responsibility of the control plane is to accept
user requests and manage processes and sessions. The
outward facing part of the system is an HTTP frontend
with the following functionality:
• allocate session Creates a new session. The allo-
cation must specify a maximum amount of memory
the session can use. The session is hosted on a sub-
process with enough resources to support it;
• retrieve session Loads session state from storage;
• invoke Allocates a new session or retrieves an exist-
ing one and passes the function invocation payload;
• support for process management such as creating
a new process or deleting one.
Processes, sessions and invocation are all uniquely
identified, and have a hierarchical structure: invocations
are hosted on sessions, sessions are hosted on subpro-
cesses and a group of subprocesses forms a process.
The control plane relies on an in-memory Redis cache
to store information about which sessions belong to
which process as well as which sessions are live and
which are swapped.
Execution backend Subprocesses are managed by the
system, and scaled up and down as the load on the PraaS
8
system varies. To run and manage containers running
these subprocesses, we deploy executor servers that are
connected to the control plane. The executor server ful-
fills requests to create additional subprocesses if the sys-
tem load is high, or requests to compact sessions from
multiple subprocesses with a low load. The subprocesses
can be allocated to servers in a round-robin fashion, but
other optimized allocation policies can be implemented.
Our system does not have any restrictions in this regard,
so low-latency schedulers like the one in Lambda can be
supported [29]. The executor server can take the form of
either a VM or a serverless function.
Subprocess The subprocess runs a session controller,
that accepts requests from the control plane and man-
ages local sessions. When receiving an allocation re-
quest, it forks a new session. It also reports sessions
being swapped out or deleted to the control plane. The
subprocess also runs the local memory controller, which
we detail in Sec. 5.3.
5.2 Session
The main components of sessions are locally available
memory that can be swapped out to cloud storage (AWS
S3 in our implementation), and the data plane TCP con-
nection. Invocation payload is received directly into
memory and can be accessed by the invoked function.
To run user code, a thread pool dispatches the invoked
functions. Each session has an upper cap for the number
of active functions that can be set by the cloud provider,
for example in relation to memory used - limiting the
thread pool to N threads. Functions queued beyond this
number will wait until resources become available.
Sessions make usage metrics available to the session
controller via session memory. The controller uses these
data plane invocation metrics to decide when sessions are
no longer in use and can be swapped out. A session also
shuts down when the user closes the connection to it on
her side.
Code Deployment We propose that users define con-
tainers with both code and libraries necessary to deploy
in the PraaS system. This container is extended by our
monitoring system and runs with restricted permissions,
similar to Lambda C++ containers. Only the PraaS con-
troller runs with root privileges.
5.3 Global Memory
To implement a global view of memory physically dis-
tributed across subprocess, each subprocess runs a local
memory controller that acts as a simple server to an-
swer session’s requests for create, put, get, and delete
operations of objects. While the interface is simple, the
mechanism behind it is more involved: when a session
RTT time [ms]
requests an object, it will first request it from the lo-
cal memory controller. If the object is not on the same
subprocess, a request to the global memory controller is
made, and a NAT hole punching operation begins. The
global memory controller runs a page table with infor-
mation linking objects to subprocesses composed of tu-
ples like {Object name, Location}. The global memory
informs the local memory controller owning the object
to finalize the connection using NAT hole punching on
its side. Once the connection exists, future communi-
cation can occur without involving the global memory
controller.
5.4 Billing and Accounting
Similarly to current FaaS and IaaS platforms, PraaS al-
lows users to create sessions with a predefined amount of
memory and CPU resources. In addition specifying the
resources for local sessions, users also need to indicate
how large the process memory should be. To make this
task simpler, PraaS allows users to specify process mem-
ory increments per session so that the total process mem-
ory grows linearly with the number of sessions. Separat-
ing session memory from process memory allocation re-
sults in better resource utilization and reduces the amount
paid for unused resources [11]. Finally, to support ses-
sion swapping, PraaS also charges users for the storage
required to save swapped sessions.
Invocation accounting is performed by subprocesses,
which collect usage metrics across their lifetime for com-
putation, memory usage and I/O. We use the Docker con-
tainer metrics for this purpose on the subprocesses and
collate them across all subprocesses to allow for both re-
mote observability and granular billing.
5.5 Client Library
The client library implements the user-end of HTTP re-
quests to manage processes and sessions and binary data
is transmitted as payload over TCP connections. Clients
can serialize the data using their choice of procedure. A
further feature of the client library is the possibility for
the user to implement handlers to be executed if opera-
tions time-out or objects or sessions are not available.
5.6 Implementing PraaS
We implement PraaS as a group of extensions on top
of existing CaaS and FaaS platforms to facilitate wide
adoption in existing cloud systems. Cloud providers au-
tomatically manage a fleet of process containers running
the dedicated PraaS worker (Sec. 5.1). In particular, we
9
Warm AWS Lambda up to 32x slower.
102
101 Near-native TCP performance thanks
to bypassing control plane.
100
20 21 22 23 24 25 26 27 28 210 212 214 216 218 220221222
Message size [bytes]
AWS Lambda PraaS TCP/IP
Figure 7: PraaS vs FaaS: bypassing control plane.
demonstrate PraaS with a prototype implementation on
Knative [55], a serverless platform deployed with Ku-
bernetes [56] (Sec. 5.6). However, our design with mod-
ified containers and scheduling policies applies to other
platforms as well, and the implementation can be ported
seamlessly to other systems.
The changes to existing platforms encompass multi-
ple aspects. The scheduling must be adjusted to use
an in-memory cache that checks for process and session
IDs to make decisions. The subprocess scaling decision
must take data plane metrics provided by the sessions,
and not eliminate subprocesses with active sessions. k8s
supports scaling down by removing selected containers
through the mechanism of deletion cost. Finally, our
control plane can be deployed to provide to provide the
session-process API, and uses the kubernetes API to al-
locate containers when needed.
6 Evaluation
In this section, we evaluate PraaS. In our experiments,
we always repeat each measurement at least 100 times,
and the plot median along with the 99% non-parametric
CI. In each part of the evaluation we focus providing an-
swers to relevant questions about PraaS.
6.1 Data Plane
Does the data plane improve invocation latency?
Our approach allows fast direct invocations by using
an optimized serverless data plane. To evaluate this
claim, we compare the round-trip time for processing
an invocation with various payload sized in PraaS to
processing the same invocation on AWS Lambda. The
benchmark is executed from a t3.medium VM and in-
vokes warm AWS Lambda and PraaS functions. The
results in Fig. 7 show a consistent, significant benefit
for using PraaS, as AWS Lambda is approximately 32
times slower. AWS Lambda overheads increase with
payload size quite significantly, which suggests that mul-
 1000 Function I/O, Lambda Storage, S3
Session, PraaS
Time [ms]
100
Time [ms]
Poor storage performance for
non-trivial data movement.
10 PraaS functions are bounded by TCP
and shared memory performance.
1 0
20 21 22 23 24 25 26 27 28 210 212 214 216 218 220221222
Payload size [bytes]
Figure 8: PraaS vs FaaS: comm. between functions.
tiple copies of the payload on the critical path, which
makes large-payload invocations increasingly slow.
Furthermore, we show that PraaS has effectively no
overhead by providing the TCP baseline gathered us-
ing netperf for communicating the payload itself between
the user and the executor server. Bypassing the control
plane therefore provides a major improvement in round
trip time for function invocations.
6.2 Fast function communication
Does the use of session memory for communication
between functions improve latency?
An important concept in serverless workflows is
chaining functions to pass the output of one as input to
the other one. We claim that using session memory is
a fast way to allow for functions to communicate. We
evaluate this by launching a function with varied input
size, passing the result of to another function, which in
turn returns data to the user. For this purpose, Dynamo
DB is expensive and too limited (400 kB max element
size) so cannot be considered an option. The options
left are to either resort to accessing S3 storage or us-
ing Lambda’s function I/O. Figure 8 shows that using
PraaS session memory for function communications is
significantly faster by two orders of magnitude then us-
ing S3 storage for any data size, and about ten times
faster than Lambda function I/O. The results are not sur-
prising, and underline the need for fast, local commu-
nication rather than reliance on cloud storage solutions
for function communication. Fast communication be-
tween functions using session memory as a medium is
much faster than existing alternatives.
6.3 Session Management
Does swapping sessions incur high overheads?
To be practically useful, our system for swapping ses-
sion in and out of storage must be efficient. To evaluate
it, we measure the time needed to allocate a clean ses-
sion, restoring a swapped session or to swap out a run-
ning session.
10
Allocation, Clean Swapping, Disk
200 Restoring, Disk Swapping, S3
Restoring, S3
150
100
50
4 8 16 32 64 128 256 512 1024 2048 4096 8192
10240
Session size [bytes]
Figure 9: Overhead of session allocation and swapping.
We swap out sessions either to S3 storage or to a
gp2 instance of Elastic Block Storage attached to EC2
(a general purpose SSD). In the latter version, the cloud
provider can then swap the session to the disk attached to
the virtual machine to remove the overhead of swapping
from the critical path, and store the swapped session to
S3 storage asynchronously. S3 has very high durability,
the EBS has annual failure rate of 0.1-0.2% [57].
This has the further benefit that sessions can be re-
stored quickly when restoring from the disk. Restor-
ing a session has a lower latency than allocating a con-
tainer [58, 59] and can be done concurrently, effectively
hiding the time required to load the state from memory.
As can be seen in Fig. 9, restoring a session from disk
takes the same time as allocating a fresh session, and
even restoring from S2 is done in less then 150ms, even
for the largest session sizes. Swapping out a session also
only takes between 25 and 200 ms, depending on session
size. Therefore, session swapping does not introduce
significant overheads.
6.4 Global Memory
Does communication over global process memory im-
prove performance over using S3 storage?
We evaluate the communication between sessions us-
ing global process memory by comparing its perfor-
mance to Lambda functions communication over S3, Re-
dis or even establishing direct communication channels.
All measurements were performed on a Lambda instance
with 2 GB of memory. For PraaS, we measure a function
performing get and put operations of the same. We dis-
tinguish between the warm local scenario - data is on the
local memory controller and the warm remote - data is on
another subprocess on a different machine. The median
ping to the other machine was 350 microseconds.
The result for both bandwidth (Fig. 10a) and latency
(Fig. 10b) show clear benefits for using PraaS. The only
solution that shows similar results is the direct p2p con-
nection using Lambda functions. We expect it to be faster
than S3 and Redis as it does not involve additional ser-
vices, just one copy of the data. However, this solution
 103 Small input
2000
Large input

200
102
Send-Receive Bandwidth [MB/s]

1500

Time [ms]
150
101 1000
100

100 50 500
255.0
10.0
10 1 0 0

24

36

48

72

24

36

48

72
8

2
12

51

10

15

20

30

51

10

15

20

30
10 2
AWS AWS
PraaS PraaS

10 3
S3 PraaS, Warm Local (a) Thumbnail generation
10 4 Redis PraaS, Warm Remote
P2P Small input Large input
1500 1500
10 3 10 2 10 1 100 101 102 103 104

Time [ms]
Data Size [KB] 1000 1000

(a) Bandwidth comparison 500 500

95.5 102.3
S3 PraaS, Warm Local 0 0
Send-Receive Communication Time [ms]

Redis PraaS, Warm Remote

24

36

48

72

24

36

48

72
2

2
P2P

51

10

15

20

30

51

10

15

20

30
AWS AWS
102 PraaS PraaS

(b) Image recognition

101
Figure 11: Serverless functions in PraaS.

100 decrease cold startup frequency. While arguably such

10 1 as a limited substitute of warm and low-latency state 1 . In
10 3 10 2 10
(b) Latency comparison
Figure 10: Latency and bandwidth for communication
between Lambda functions and the global memory put-
get operations in PraaS.
Vm
AWS 3.53 · 10−3
Azure 3.87 · 10−3 ± 0.005
Table 3: Decreased costs of PraaS sessions in compari-
son to FaaS provisioned instances.
does not fit the serverless paradigm – functions must be
online at the time of transfer and they must be aware
of each other’s existence, making practical use difficult.
Furthermore we confirm the previous result that Lambda
shows significant I/O variability [11, 35]. PraaS pro-
vides fast, efficient communication even across phys-
ically separated machines.
6.5 Costs
What are the system costs?
PraaS users must be charged for keeping session state
alive and the cost should depend linearly on the life-
time of an active session. The only comparable feature
on modern commercial FaaS systems is a provisioned
function instance, known as provisioned concurrency on
AWS Lambda and premium plan for Azure Functions.
Cloud providers guarantee ready function instances to
functions are not serverless, such instances can be treated
1 100 101 102 103 104 addition to paying for using computing resources, users
Data Size [KB] are charged the active state fee Fs that depends on the
memory size and the duration of resource provisioning.
To estimate the cost of keeping PraaS sessions alive in
memory, we use the memory of other cloud services as a
proxy for the cost to the cloud operator. First, we com-
pare the cost Vm of changing from a compute-optimized
Cm Fm GV GC
to a memory-optimized virtual machine instances, and di-
4.45 · 10−3 1.5 · 10−2 76.43% 70.37%
4.45 · 10−3 1.14 · 10−2 66.05% ± 0.05 60.96%
vide it by the size of gained memory, which estimates the
additional cost of gaining one gigabyte of DRAM. We
compare c6g and x2gd instances on AWS and Fs and
Eadsv5 series on Azure, and we found that Vm almost the
same for each instance size. Then, we select the cost of
allocating additional memory when deploying PraaS on
managed container systems Cm , and we consider AWS
Fargate and Azure Container Instances. By comparing
the memory costs Vm ,Cm of PraaS deployment to the cost
of provisioned storage Fm on FaaS, we estimate the cost
decrease GV and GC of moving session state from pro-
visioned FaaS to VMs and containers, respectively. The
results presented in Table 3 prove that PraaS sessions
can be offered at a lower cost, by up to 76%, and the
estimation covers the cloud provider costs and profit in-
cluded in the price of a VM instance. Furthermore, the
memory-optimized instances come with additional SSD
storage, which could be used to implement a low-latency
tier for swapped sessions at no additional cost.
1 Provisioned concurrency instances on AWS can be recycled and
renitialized, which makes state persistence difficult, if not impossible,
to implement in practice.

11
6.6 Functions
Can FaaS functions run on PraaS?
We select the thumbnail generator as an example
of general-purpose image processing and the image-
recognition benchmark performing ResNet-50 predic-
tion as an example of integrating deep-learning inference
into applications, both from the SeBS benchmark [11].
We implement the thumbnail generation with OpenCV
4.5, and evaluate functions with two images, a 97 kB and
a 3.6 MB one. For the AWS Lambda function, we need to
submit the binary image data as a base64-encoded string
in the POST request, which adds significant overheads
due to encoding and conversions. On the other hand,
PraaS functions benefit from a payload format that is not
constrained by cloud API requirements.
The image recognition is implemented with the help of
PyTorch C++ API, using OpenCV 4.5, libtorch 1.9, and
torchvision 0.1. We evaluate functions with two inputs,
a 53 kB image and a 230 kB one. Results (Fig. 11a and
11b) prove that PraaS outperforms all AWS instances
we compared against.
6.7 Discussion
Given the features PraaS provides, it could be used as a
platform to implementing distributed logistic regression
with gradient averaging stochastic gradient descent in
both an asynchronous and synchronous fashion. The per-
formance of the system could improve compared to the
current implemention [60], due to using direct commu-
nication. The PraaS model should make it easier to ex-
press asynchronous computations. Finally, PraaS could
dynamically change the number of functions being in-
voked at a given time, while LambdaML [60] relies on a
fixed number of functions throughout the execution.
7 Related Work
Stateful Functions open the spectrum of applications
that can benefit from FaaS by allowing functions to
keep state, even if disaggregated. Researchers have
built stateful functions on top of key-value stores spe-
cialized to Serverless [18, 61], and elastic ephemeral
caches [51, 62, 63] which combine different placement
strategies to manage cost and performance. For many
applications, however, statefulness is not enough as func-
tions can fail at any time. Hence, systems such as
Beldi [19], and Boki [20] have been proposed to help de-
velopers build consistent and fault tolerant systems atop
ephemeral functions. Instead of offering yet another stor-
age option to persist function state or propose a new tech-
nique to handle faults, PraaS strikes a new balance be-
tween IaaS and FaaS by proposing the concept of ses-
12
sions that retain state. Within a process, sessions can
exchange messages. PraaS eliminates the need for state
synchronization in stateful functions by partitioning state
into ownership domains managed by sessions. Sessions
can be swapped at any time but their state is not lost.
Function Orchestration and Data Locality are also
being extensively studied. Systems such as Speedo [64]
and Nightcore [65] optimize function orchestration by
either accelerating the control plane [64] or by com-
pletely skipping it [65] for internal function invoca-
tions. Other systems have looked into how to optimize
the data path by comparing different function communi-
cation strategies and automatically adapting deployment
decisions [66], or by avoiding moving data by allowing
multiple functions to share the execution environment
over time [67].
PraaS, on the other hand, proposes that users should
be able to directly connect to sessions and thus com-
pletely avoid the control plane after the first initialization
step. In addition, by partitioning state among sessions,
process functions can exchange data directly among
them instead of relying on external storage proposed by
previous systems [7, 8]. Although taking advantage of
direct communication between functions whenever pos-
sible, PraaS offers a global memory space abstraction to
users instead of directly exposing network primitives to
developers [68].
Serverless Sandboxes utilize specialized virtualiza-
tion engines [58, 69] that drastically reduce the startup
time, and memory footprint when compared to tradi-
tional virtual machine managers. However, to continue
improving scalability and elasticity of serverless applica-
tions, soft-isolation systems [70–72] have been proposed
to co-execute multiple invocations inside the same OS
process. PraaS is, in part, inspired by such systems by
allowing multiple functions to execute concurrently in-
side a single session. By doing so, resource redundancy
is reduced and new opportunities for local communica-
tion arise. However, PraaS’s design does not preclude
other orthogonal optimization techniques such as image
pre-initialization [69, 73–75] to optimize session startup
time and memory footprint, or unikernels [76–78] to op-
timize process startup and memory footprint.
8 Conclusions
This work proposes PraaS, a new cloud computing ab-
straction that brings persistent state to ephemeral work-
ers. By taking advantage of processes and sessions, ap-
plications can now benefit from low-latency state, fast
invocations that bypass the control plane, and fast com-
munication between sessions. We showed that PraaS is
an efficient abstraction to achieve stateful functions with
direct connections and fast communication options.
References
[1] Yanqi Zhang, Íñigo Goiri, Gohar Irfan Chaudhry, Rodrigo Fon-
seca, Sameh Elnikety, Christina Delimitrou, and Ricardo Bian-
chini. Faster and cheaper serverless computing on harvested re-
sources. In Proceedings of the ACM SIGOPS 28th Symposium
on Operating Systems Principles, SOSP ’21, page 724–739, New
York, NY, USA, 2021. Association for Computing Machinery.
ISBN 9781450387095. doi: 10.1145/3477132.3483580. URL
https://doi.org/10.1145/3477132.3483580.
[2] Amoghavarsha Suresh and Anshul Gandhi. Servermore: Oppor-
tunistic execution of serverless functions in the cloud. SoCC
’21, page 570–584, New York, NY, USA, 2021. Association
for Computing Machinery. ISBN 9781450386388. doi: 10.
1145/3472883.3486979. URL https://doi.org/10.1145/
3472883.3486979.
[3] Marcin Copik, Konstantin Taranov, Alexandru Calotoiu, and
Torsten Hoefler. RFaaS: RDMA-Enabled FaaS Platform for
Serverless High-Performance Computing, 2021. URL https:
//arxiv.org/abs/2106.13859.
[4] Jiawei Jiang, Shaoduo Gan, Yue Liu, Fanlin Wang, Gustavo
Alonso, Ana Klimovic, Ankit Singla, Wentao Wu, and Ce Zhang.
Towards demystifying serverless machine learning training. In
ACM SIGMOD International Conference on Management of
Data (SIGMOD 2021), June 2021.
[5] Lixiang Ao, Liz Izhikevich, Geoffrey M. Voelker, and George
Porter. Sprocket: A serverless video processing framework.
In Proceedings of the ACM Symposium on Cloud Computing,
SoCC ’18, pages 263–274, New York, NY, USA, 2018. As-
sociation for Computing Machinery. ISBN 9781450360111.
doi: 10.1145/3267809.3267815. URL https://doi.org/10.
1145/3267809.3267815.
[6] Sadjad Fouladi, Riad S. Wahby, Brennan Shacklett,
Karthikeyan Vasuki Balasubramaniam, William Zeng, Rahul
Bhalerao, Anirudh Sivaraman, George Porter, and Keith
Winstein. Encoding, fast and slow: Low-latency video pro-
cessing using thousands of tiny threads. In 14th USENIX
Symposium on Networked Systems Design and Implemen-
tation (NSDI 17), pages 363–376, Boston, MA, March
2017. USENIX Association. ISBN 978-1-931971-37-9.
URL https://www.usenix.org/conference/nsdi17/
technical-sessions/presentation/fouladi.
[7] Matthew Perron, Raul Castro Fernandez, David DeWitt, and
Samuel Madden. Starling: A scalable query engine on cloud
functions. In Proceedings of the 2020 ACM SIGMOD Inter-
national Conference on Management of Data, SIGMOD ’20,
page 131–141, New York, NY, USA, 2020. Association for
Computing Machinery. ISBN 9781450367356. doi: 10.
1145/3318464.3380609. URL https://doi.org/10.1145/
3318464.3380609.
[8] Ingo Müller, Renato Marroquin, and Gustavo Alonso. Lambada:
Interactive data analytics on cold data using serverless cloud in-
frastructure. ArXiv, abs/1912.00937, 2019.
[9] Qifan Pu, Shivaram Venkataraman, and Ion Stoica. Shuf-
fling, fast and slow: Scalable analytics on serverless infrastruc-
ture. In 16th USENIX Symposium on Networked Systems Design
and Implementation (NSDI 19), pages 193–206, Boston, MA,
February 2019. USENIX Association. ISBN 978-1-931971-49-
2. URL https://www.usenix.org/conference/nsdi19/
presentation/pu.
13
[10] Eric Jonas, Johann Schleier-Smith, Vikram Sreekanti, Chia-che
Tsai, Anurag Khandelwal, Qifan Pu, Vaishaal Shankar, Joao Car-
reira, Karl Krauth, Neeraja Jayant Yadwadkar, Joseph E. Gonza-
lez, Raluca Ada Popa, Ion Stoica, and David A. Patterson. Cloud
programming simplified: A berkeley view on serverless comput-
ing. CoRR, abs/1902.03383, 2019. URL http://arxiv.org/
abs/1902.03383.
[11] Marcin Copik, Grzegorz Kwasniewski, Maciej Besta, Michal
Podstawski, and Torsten Hoefler. Sebs: A serverless bench-
mark suite for function-as-a-service computing. In Proceed-
ings of the 22nd International Middleware Conference, Middle-
ware ’21. Association for Computing Machinery, 2021. doi: 10.
1145/3464298.3476133. URL https://doi.org/10.1145/
3464298.3476133.
[12] Johann Schleier-Smith, Vikram Sreekanti, Anurag Khandelwal,
Joao Carreira, Neeraja J. Yadwadkar, Raluca Ada Popa, Joseph E.
Gonzalez, Ion Stoica, and David A. Patterson. What serverless
computing is and should become: The next phase of cloud com-
puting. Commun. ACM, 64(5):76–84, April 2021. ISSN 0001-
0782. doi: 10.1145/3406011. URL https://doi.org/10.
1145/3406011.
[13] L. Feng, P. Kudva, D. Da Silva, and J. Hu. Exploring server-
less computing for neural network training. In 2018 IEEE 11th
International Conference on Cloud Computing (CLOUD), pages
334–341, July 2018. doi: 10.1109/CLOUD.2018.00049.
[14] Hao Wang, Di Niu, and Baochun Li. Distributed machine learn-
ing with a serverless architecture. In IEEE INFOCOM 2019-
IEEE Conference on Computer Communications, pages 1288–
1296. IEEE, 2019.
[15] John Thorpe, Yifan Qiao, Jonathan Eyolfson, Shen Teng,
Guanzhou Hu, Zhihao Jia, Jinliang Wei, Keval Vora, Ravi Ne-
travali, Miryung Kim, et al. Dorylus: Affordable, scalable, and
accurate {GNN} training with distributed {CPU} servers and
serverless threads. In 15th USENIX Symposium on Operating
Systems Design and Implementation (OSDI 21), pages 495–514,
2021.
[16] Joao Carreira, Pedro Fonseca, Alexey Tumanov, Andrew Zhang,
and Randy Katz. Cirrus: A serverless framework for end-to-end
ml workflows. In Proceedings of the ACM Symposium on Cloud
Computing, pages 13–24, 2019.
[17] Ana Klimovic, Yawen Wang, Patrick Stuedi, Animesh Trivedi,
Jonas Pfefferle, and Christos Kozyrakis. Pocket: Elastic
ephemeral storage for serverless analytics. In Proceedings of the
12th USENIX Conference on Operating Systems Design and Im-
plementation, OSDI’18, pages 427–444, USA, 2018. USENIX
Association. ISBN 9781931971478.
[18] Vikram Sreekanti, Chenggang Wu, Xiayue Charles Lin, Johann
Schleier-Smith, Joseph E. Gonzalez, Joseph M. Hellerstein, and
Alexey Tumanov. Cloudburst: Stateful functions-as-a-service.
Proc. VLDB Endow., 13(12):2438–2452, July 2020. ISSN 2150-
8097. doi: 10.14778/3407790.3407836. URL https://doi.
org/10.14778/3407790.3407836.
[19] Haoran Zhang, Adney Cardoza, Peter Baile Chen, Sebastian An-
gel, and Vincent Liu. Fault-tolerant and transactional stateful
serverless workflows. In 14th USENIX Symposium on Operat-
ing Systems Design and Implementation (OSDI 20), pages 1187–
1204. USENIX Association, November 2020. ISBN 978-1-
939133-19-9. URL https://www.usenix.org/conference/
osdi20/presentation/zhang-haoran.
[20] Zhipeng Jia and Emmett Witchel. Boki: Stateful serverless com-
puting with shared logs. In Proceedings of the ACM SIGOPS
28th Symposium on Operating Systems Principles, SOSP ’21,
page 691–707, New York, NY, USA, 2021. Association for
Computing Machinery. ISBN 9781450387095. doi: 10.
1145/3477132.3483541. URL https://doi.org/10.1145/
3477132.3483541.
[21] Mohammad Shahrad, Rodrigo Fonseca, Inigo Goiri, Gohar
Chaudhry, Paul Batum, Jason Cooke, Eduardo Laureano, Colby
Tresness, Mark Russinovich, and Ricardo Bianchini. Server-
less in the wild: Characterizing and optimizing the server-
less workload at a large cloud provider. In 2020 USENIX
Annual Technical Conference (USENIX ATC 20), pages 205–
218. USENIX Association, July 2020. ISBN 978-1-939133-
14-4. URL https://www.usenix.org/conference/atc20/
presentation/shahrad.
[22] Simon Peter, Jialin Li, Irene Zhang, Dan R. K. Ports, Doug
Woos, Arvind Krishnamurthy, Thomas Anderson, and Timothy
Roscoe. Arrakis: The operating system is the control plane.
In 11th USENIX Symposium on Operating Systems Design and
Implementation (OSDI 14), pages 1–16, Broomfield, CO, Oc-
tober 2014. USENIX Association. ISBN 978-1-931971-16-
4. URL https://www.usenix.org/conference/osdi14/
technical-sessions/presentation/peter.
[23] Collin Lee and John Ousterhout. Granular computing. In Pro-
ceedings of the Workshop on Hot Topics in Operating Systems,
HotOS ’19, page 149–154, New York, NY, USA, 2019. As-
sociation for Computing Machinery. ISBN 9781450367271.
doi: 10.1145/3317550.3321447. URL https://doi.org/10.
1145/3317550.3321447.
[24] Pedro Garcia Lopez, Aleksander Slominski, Michael Behrendt,
and Bernard Metzler. Serverless predictions: 2021-2030, 2021.
[25] AWS Lambda. https://aws.amazon.com/lambda/, 2014.
Accessed: 2020-01-20.
[26] Azure Functions. https://azure.microsoft.com/en-us/
services/functions/, 2016. Accessed: 2020-01-20.
[27] Google Cloud Functions. https://cloud.google.com/
functions/, 2017. Accessed: 2020-01-20.
[28] IBM Cloud Functions. https://cloud.ibm.com/
functions/, 2016. Accessed: 2020-01-20.
[29] Alexandru Agache, Marc Brooker, Alexandra Iordache, Anthony
Liguori, Rolf Neugebauer, Phil Piwonka, and Diana-Maria Popa.
Firecracker: Lightweight virtualization for serverless applica-
tions. In 17th USENIX Symposium on Networked Systems Design
and Implementation (NSDI 20), pages 419–434, Santa Clara, CA,
February 2020. USENIX Association. ISBN 978-1-939133-13-
7. URL https://www.usenix.org/conference/nsdi20/
presentation/agache.
[30] Johannes Manner, Martin EndreB, Tobias Heckel, and Guido
Wirtz. Cold start influencing factors in function as a service. 2018
IEEE/ACM International Conference on Utility and Cloud Com-
puting Companion (UCC Companion), pages 181–188, 2018.
[31] Anup Mohan, Harshad Sane, Kshitij Doshi, Saikrishna Edupu-
ganti, Naren Nayak, and Vadim Sukhomlinov. Agile cold starts
for scalable serverless. In 11th USENIX Workshop on Hot
Topics in Cloud Computing (HotCloud 19), Renton, WA, July
2019. USENIX Association. URL https://www.usenix.org/
conference/hotcloud19/presentation/mohan.
14
[32] Nilanjan Daw, Umesh Bellur, and Purushottam Kulkarni.
Xanadu: Mitigating cascading cold starts in serverless function
chain deployments. In Proceedings of the 21st International
Middleware Conference, Middleware ’20, page 356–370, New
York, NY, USA, 2020. Association for Computing Machinery.
ISBN 9781450381536. doi: 10.1145/3423211.3425690. URL
https://doi.org/10.1145/3423211.3425690.
[33] Paulo Silva, Daniel Fireman, and Thiago Emmanuel Pereira. Pre-
baking functions to warm the serverless cold start. In Proceed-
ings of the 21st International Middleware Conference, Middle-
ware ’20, page 1–13, New York, NY, USA, 2020. Association
for Computing Machinery. ISBN 9781450381536. doi: 10.
1145/3423211.3425682. URL https://doi.org/10.1145/
3423211.3425682.
[34] Kun Suo, Junggab Son, Dazhao Cheng, Wei Chen, and Sabur
Baidya. Tackling cold start of serverless applications by efficient
and adaptive container runtime reusing. In 2021 IEEE Inter-
national Conference on Cluster Computing (CLUSTER), pages
433–443, 2021. doi: 10.1109/Cluster48925.2021.00018.
[35] Liang Wang, Mengyuan Li, Yinqian Zhang, Thomas Ristenpart,
and Michael Swift. Peeking behind the curtains of serverless plat-
forms. In Proceedings of the 2018 USENIX Conference on Usenix
Annual Technical Conference, USENIX ATC ’18, page 133–145,
USA, 2018. USENIX Association. ISBN 9781931971447.
[36] Adam Eivy and Joe Weinman. Be wary of the economics of
”serverless” cloud computing. IEEE Cloud Computing, 4(2):6–
12, 2017. doi: 10.1109/MCC.2017.32.
[37] Apache OpenWhisk. https://openwhisk.apache.org/,
2016. Accessed: 2020-01-20.
[38] M. Sciabarrà. Learning Apache OpenWhisk: Developing Open
Serverless Solutions. O’Reilly Media, Incorporated, 2019. ISBN
9781492046165.
[39] Benjamin Carver, Jingyuan Zhang, Ao Wang, Ali Anwar, Pan-
ruo Wu, and Yue Cheng. Wukong: A scalable and locality-
enhanced framework for serverless parallel computing. In Pro-
ceedings of the 11th ACM Symposium on Cloud Computing,
SoCC ’20, page 1–15, New York, NY, USA, 2020. Association
for Computing Machinery. ISBN 9781450381376. doi: 10.
1145/3419111.3421286. URL https://doi.org/10.1145/
3419111.3421286.
[40] Arjun Singhvi, Kevin Houck, Arjun Balasubramanian, Mo-
hammed Danish Shaikh, Shivaram Venkataraman, and Aditya
Akella. Archipelago: A scalable low-latency serverless platform,
2019.
[41] Istemi Ekin Akkus, Ruichuan Chen, Ivica Rimac, Manuel Stein,
Klaus Satzke, Andre Beck, Paarijaat Aditya, and Volker Hilt.
Sand: Towards high-performance serverless computing. In Pro-
ceedings of the 2018 USENIX Conference on Usenix Annual
Technical Conference, USENIX ATC ’18, pages 923–935, USA,
2018. USENIX Association. ISBN 9781931971447.
[42] Minchen Yu, Tingjia Cao, Wei Wang, and Ruichuan Chen. Re-
structuring serverless computing with data-centric function or-
chestration. arXiv preprint arXiv:2109.13492, 2021.
[43] Bryan Ford, Pyda Srisuresh, and Dan Kegel. Peer-to-peer com-
munication across network address translators. In Proceedings
of the Annual Conference on USENIX Annual Technical Confer-
ence, ATEC ’05, page 13, USA, April 2005. USENIX Associa-
tion.
[44] J. L. Eppinger. TCP Connections for P2P Apps: A Software Ap-
proach to Solving the NAT Problem. Carnegie Mellon University,
Technical Report, ISRI-05-104, January 2005.
[45] Jashwant Raj Gunasekaran, Prashanth Thinakaran, Mahmut Tay-
lan Kandemir, Bhuvan Urgaonkar, George Kesidis, and Chita
Das. Spock: Exploiting serverless functions for slo and cost
aware resource procurement in public cloud. In 2019 IEEE 12th
International Conference on Cloud Computing (CLOUD), pages
199–208, 2019. doi: 10.1109/CLOUD.2019.00043.
[46] V. Giménez-Alventosa, Germán Moltó, and Miguel Caballer. A
framework and a performance assessment for serverless mapre-
duce on aws lambda. Future Generation Computer Systems, 97:
259–274, 2019. ISSN 0167-739X. doi: https://doi.org/10.1016/
j.future.2019.02.057. URL https://www.sciencedirect.
com/science/article/pii/S0167739X18325172.
[47] Ahsan Ali, Riccardo Pinciroli, Feng Yan, and Evgenia Smirni.
Batch: Machine learning inference serving on serverless plat-
forms with adaptive batching. In SC20: International Confer-
ence for High Performance Computing, Networking, Storage and
Analysis, pages 1–15, 2020. doi: 10.1109/SC41405.2020.00073.
[48] Minchen Yu, Zhifeng Jiang, Hok Chun Ng, Wei Wang, Ruichuan
Chen, and Bo Li. Gillis: Serving large neural networks in server-
less functions with automatic model partitioning. In 2021 IEEE
41st International Conference on Distributed Computing Systems
(ICDCS), pages 138–148, 2021. doi: 10.1109/ICDCS51616.
2021.00022.
[49] Ashraf Mahgoub, Karthick Shankar, Subrata Mitra, Ana
Klimovic, Somali Chaterji, and Saurabh Bagchi. {SONIC}:
Application-aware data passing for chained serverless appli-
cations. In 2021 {USENIX} Annual Technical Conference
({USENIX}{ATC} 21), pages 285–301, 2021.
[50] Sadjad Fouladi, Riad S. Wahby, Brennan Shacklett,
Karthikeyan Vasuki Balasubramaniam, William Zeng, Rahul
Bhalerao, Anirudh Sivaraman, George Porter, and Keith Win-
stein. Encoding, fast and slow: Low-latency video processing
using thousands of tiny threads. In Proceedings of the 14th
USENIX Conference on Networked Systems Design and Im-
plementation, NSDI’17, pages 363–376, USA, 2017. USENIX
Association. ISBN 9781931971379.
[51] Francisco Romero, Gohar Irfan Chaudhry, Íñigo Goiri, Pragna
Gopa, Paul Batum, Neeraja J. Yadwadkar, Rodrigo Fonseca,
Christos Kozyrakis, and Ricardo Bianchini. Faa$t: A transpar-
ent auto-scaling cache for serverless applications, 2021.
[52] Torsten Hoefler, J. Dinan, Rajeev Thakur, Brian Barrett, P. Bal-
aji, William Gropp, and K. Underwood. Remote Memory Access
Programming in MPI-3. ACM Transactions on Parallel Comput-
ing (TOPC), Jan. 2015. accepted for publication on Dec. 4th.
[53] Eric Jonas, Shivaram Venkataraman, Ion Stoica, and Benjamin
Recht. Occupy the cloud: Distributed computing for the 99%.
CoRR, abs/1702.04024, 2017. URL http://arxiv.org/abs/
1702.04024.
[54] Ana Klimovic, Yawen Wang, Christos Kozyrakis, Patrick Stuedi,
Jonas Pfefferle, and Animesh Trivedi. Understanding ephemeral
storage for serverless analytics. In 2018 USENIX Annual Tech-
nical Conference (USENIX ATC 18), pages 789–794, Boston,
MA, July 2018. USENIX Association. ISBN 978-1-939133-
01-4. URL https://www.usenix.org/conference/atc18/
presentation/klimovic-serverless.
[55] Knative. https://knative.dev/, 2021. Accessed: 2021-11-
21.
15
[56] Kubernetes. https://kubernetes.io/, 2021. Accessed:
2021-11-29.
[57] AWS S3. https://aws.amazon.com/s3/, 2006. Accessed:
2021-06-07.
[58] Firecracker. https://github.com/firecracker-microvm/
firecracker, 2018. Accessed: 2020-01-20.
[59] Ethan G. Young, Pengfei Zhu, Tyler Caraza-Harter, Andrea C.
Arpaci-Dusseau, and Remzi H. Arpaci-Dusseau. The true cost
of containing: A gvisor case study. In 11th USENIX Workshop
on Hot Topics in Cloud Computing (HotCloud 19), Renton, WA,
July 2019. USENIX Association. URL https://www.usenix.
org/conference/hotcloud19/presentation/young.
[60] LambdaML. https://github.com/DS3Lab/LambdaML/
blob/master/examples/lambda/s3/linear_s3_ga_ma.
py, 2020. Accessed: 2020-01-20.
[61] Daniel Barcelona-Pons, Marc Sánchez-Artigas, Gerard Parı́s,
Pierre Sutra, and Pedro Garcı́a-López. On the faas track: Build-
ing stateful distributed applications with serverless architectures.
In Proceedings of the 20th International Middleware Confer-
ence, Middleware ’19, page 41–54, New York, NY, USA, 2019.
Association for Computing Machinery. ISBN 9781450370097.
doi: 10.1145/3361525.3361535. URL https://doi.org/10.
1145/3361525.3361535.
[62] Ana Klimovic, Yawen Wang, Patrick Stuedi, Animesh Trivedi,
Jonas Pfefferle, and Christos Kozyrakis. Pocket: Elastic
ephemeral storage for serverless analytics. In Proceedings of
the 13th USENIX Conference on Operating Systems Design and
Implementation, OSDI’18, page 427–444, USA, 2018. USENIX
Association. ISBN 9781931971478.
[63] Qifan Pu, Shivaram Venkataraman, and Ion Stoica. Shuffling,
fast and slow: Scalable analytics on serverless infrastructure. In
Proceedings of the 16th USENIX Conference on Networked Sys-
tems Design and Implementation, NSDI’19, page 193–206, USA,
2019. USENIX Association. ISBN 9781931971492.
[64] Nilanjan Daw, Umesh Bellur, and Purushottam Kulkarni.
Speedo: Fast Dispatch and Orchestration of Serverless Work-
flows, page 585–599. Association for Computing Machinery,
New York, NY, USA, 2021. ISBN 9781450386388. URL
https://doi.org/10.1145/3472883.3486982.
[65] Zhipeng Jia and Emmett Witchel. Nightcore: Efficient and
scalable serverless computing forlatency-sensitive, interactive
microservices. In Proceedings ofthe 26th ACM International
Conference on Architectural Support for Programming Lan-
guages and Operating Systems, ASPLOS ’21, New York, NY,
USA, 2021. Association for Computing Machinery. doi: 10.
1145/3445814.3446701. URL https://doi.org/10.1145/
3445814.3446701.
[66] Ashraf Mahgoub, Karthick Shankar, Subrata Mitra, Ana
Klimovic, Somali Chaterji, and Saurabh Bagchi. SONIC:
Application-aware data passing for chained serverless applica-
tions. In 2021 USENIX Annual Technical Conference (USENIX
ATC 21), pages 285–301. USENIX Association, July 2021.
ISBN 978-1-939133-23-6. URL https://www.usenix.org/
conference/atc21/presentation/mahgoub.
[67] Swaroop Kotni, Ajay Nayak, Vinod Ganapathy, and Arkaprava
Basu. Faastlane: Accelerating Function-as-a-Service work-
flows. In 2021 USENIX Annual Technical Conference (USENIX
ATC 21), pages 805–820. USENIX Association, July 2021.
ISBN 978-1-939133-23-6. URL https://www.usenix.org/
conference/atc21/presentation/kotni.
[68] Mike Wawrzoniak, Ingo Müller, Rodrigo Fraga Barcelos Association for Computing Machinery. ISBN 9781450350853.
Paulus Bruno, and Gustavo Alonso. Boxer: Data analytics on doi: 10.1145/3132747.3132763. URL https://doi.org/10.
network-enabled serverless platforms. In 11th Annual Confer- 1145/3132747.3132763.
ence on Innovative Data Systems Research (CIDR’21), 2021.
[78] Ricardo Koller and Dan Williams. Will serverless end the domi-
[69] Dong Du, Tianyi Yu, Yubin Xia, Binyu Zang, Guanglu Yan, nance of linux in the cloud? In Proceedings of the 16th Workshop
Chenggang Qin, Qixuan Wu, and Haibo Chen. Catalyzer: Sub- on Hot Topics in Operating Systems, HotOS ’17, page 169–173,
millisecond startup for serverless computing with initialization- New York, NY, USA, 2017. Association for Computing Machin-
less booting. In Proceedings of the Twenty-Fifth International ery. ISBN 9781450350686. doi: 10.1145/3102980.3103008.
Conference on Architectural Support for Programming Lan- URL https://doi.org/10.1145/3102980.3103008.
guages and Operating Systems, ASPLOS ’20, page 467–481,
New York, NY, USA, 2020. Association for Computing Machin-
ery. ISBN 9781450371025. doi: 10.1145/3373376.3378512.
URL https://doi.org/10.1145/3373376.3378512.

[70] Vojislav Dukic, Rodrigo Bruno, Ankit Singla, and Gustavo

Alonso. Photons: Lambdas on a diet. In Proceedings of the 11th
ACM Symposium on Cloud Computing, SoCC ’20, page 45–59,
New York, NY, USA, 2020. Association for Computing Machin-
ery. ISBN 9781450381376. doi: 10.1145/3419111.3421297.
URL https://doi.org/10.1145/3419111.3421297.

[71] Simon Shillaker and Peter Pietzuch. Faasm: Lightweight

isolation for efficient stateful serverless computing. In 2020
{USENIX} Annual Technical Conference ({USENIX}{ATC} 20),
pages 419–433, 2020.

[72] Sol Boucher, Anuj Kalia, David G. Andersen, and Michael

Kaminsky. Putting the ”micro” back in microservice. In
2018 USENIX Annual Technical Conference (USENIX ATC 18),
pages 645–650, Boston, MA, July 2018. USENIX Association.
ISBN 978-1-939133-01-4. URL https://www.usenix.org/
conference/atc18/presentation/boucher.

[73] Istemi Ekin Akkus, Ruichuan Chen, Ivica Rimac, Manuel Stein,
Klaus Satzke, Andre Beck, Paarijaat Aditya, and Volker Hilt.
SAND: Towards High-Performance serverless computing. In
2018 USENIX Annual Technical Conference (USENIX ATC 18),
pages 923–935, Boston, MA, July 2018. USENIX Association.
ISBN 978-1-939133-01-4. URL https://www.usenix.org/
conference/atc18/presentation/akkus.

[74] James Cadden, Thomas Unger, Yara Awad, Han Dong, Orran
Krieger, and Jonathan Appavoo. Seuss: Skip redundant paths
to make serverless fast. In Proceedings of the Fifteenth Eu-
ropean Conference on Computer Systems, EuroSys ’20, New
York, NY, USA, 2020. Association for Computing Machinery.
ISBN 9781450368827. doi: 10.1145/3342195.3392698. URL
https://doi.org/10.1145/3342195.3392698.

[75] Edward Oakes, Leon Yang, Dennis Zhou, Kevin Houck, Tyler
Harter, Andrea Arpaci-Dusseau, and Remzi Arpaci-Dusseau.
SOCK: Rapid task provisioning with serverless-optimized con-
tainers. In 2018 USENIX Annual Technical Conference (USENIX
ATC 18), pages 57–70, Boston, MA, July 2018. USENIX Associ-
ation. ISBN 978-1-931971-44-7. URL https://www.usenix.
org/conference/atc18/presentation/oakes.

[76] Yiming Zhang, Jon Crowcroft, Dongsheng Li, Chengfei Zhang,

Huiba Li, Yaozheng Wang, Kai Yu, Yongqiang Xiong, and Gui-
hai Chen. Kylinx: A dynamic library operating system for
simplified and efficient cloud virtualization. USENIX ATC
’18, page 173–185, USA, 2018. USENIX Association. ISBN
9781931971447.

[77] Filipe Manco, Costin Lupu, Florian Schmidt, Jose Mendes, Si-
mon Kuenzer, Sumit Sati, Kenichi Yasukata, Costin Raiciu, and
Felipe Huici. My vm is lighter (and safer) than your container. In
Proceedings of the 26th Symposium on Operating Systems Prin-
ciples, SOSP ’17, page 218–233, New York, NY, USA, 2017.

16