VOLUME 39, NUMBER 4 JULY/AUGUST 2019

Secure Architectures

www.computer.org/micro
 Call for 2019 Major
Awards Nominations
Deadline: 1 October 2019

Help Recognize Computing’s Most Prestigious Individuals

IEEE Computer Society awards recognize outstanding achievements and highlight
significant contributors in the teaching and R&D computing communities. All members
of the profession are invited to nominate individuals they consider most eligible to receive
international recognition through an appropriate society award.

Charles Babbage Award Computer Pioneer Award

Certificate/$1,000 Silver Medal
In recognition of significant contributions in the field Pioneering concepts and development of the
of parallel computation. computing field.

Computer Entrepreneur Award W. Wallace McDowell Award

Sterling Silver Goblet Certificate/$2,000
Vision and leadership resulting in the growth of some Recent theoretical, design, educational, practical,
segment of the computer industry. or other tangible innovative contributions.

Edward J. McCluskey Taylor L. Booth Award

Technical Achievement Award Bronze Medal/$5,000
Certificate/$2,000 Contributions to computer science and
Contributions to computer science or computer engineering education.
technology.
Computer Science & Engineering
Harry H. Goode Memorial Award Undergraduate Teaching Award
Bronze Medal/$2,000 Plaque/$2,000
Information sciences, including seminal ideas, Recognizes outstanding contributions to
algorithms, computing directions, and concepts. undergraduate education.

Hans Karlsson Award Harlan D. Mills Award

Plaque/$2,000 Plaque/$3,000
Team leadership and achievement through Contributions to the practice of software engineering
collaboration in computing standards. through the application of sound theory.

Richard E. Merwin Award

for Distinguished Service
Bronze Medal/$5,000
Nomination Deadline
Outstanding volunteer service to the
profession at large, including service Submit your nomination by
to the IEEE Computer Society.
1 October 2019 to
www.computer.org/awards
Contact us at
awards@computer.org
EDITOR-IN-CHIEF IEEE MICRO STAFF
Lizy K. John, University of Texas, Austin Journals Coordinator: Joanna Gojlik,
j.gojlik@ieee.org
EDITORIAL BOARD Peer-Review Administrator:
micro-ma@computer.org
R. Iris Bahar, Brown University
Publications Portfolio Manager: Kimberly Sperka
Mauricio Breternitz, University of Lisbon
Publisher: Robin Baldwin
David Brooks, Harvard University
Senior Advertising Coordinator: Debbie Sims
Bronis de Supinski, Lawrence Livermore
IEEE Computer Society Executive Director:
Nation al Lab
Melissa Russell
Shane Greenstein, Harvard Business School
Natalie Enright Jerger, University of Toronto IEEE PUBLISHING OPERATIONS
Hyesoon Kim, Georgia Institute of Technology
John Kim, Korea Advanced Institute of Science Senior Director, Publishing Operations:
and Technology Dawn Melley
Hsien-Hsin (Sean) Lee, Taiwan Semiconductor Director, Editorial Services: Kevin Lisankie
Manufacturing Company Director, Production Services: Peter M. Tuohy
Richard Mateosian Associate Director, Editorial Services:
Tulika Mitra, National University of Singapore Jeffrey E. Cichocki
Trevor Mudge, University of Michigan, Ann Arbor Associate Director, Information Conversion
Onur Mutlu, ETH Zurich and Editorial Support: Neelam Khinvasara
Vijaykrishnan Narayanan, The Pennsylvania Senior Art Director: Janet Dudar
State University Senior Managing Editor: Patrick Kempf
Per Stenstrom, Chalmers University of CS MAGAZINE OPERATIONS COMMITTEE
Technology
Richard H. Stern, George Washington Sumi Helal (Chair), Irena Bojanova,
University Law School Jim X. Chen, Shu-Ching Chen,
Sreenivas Subramoney, Intel Corporation Gerardo Con Diaz, David Alan Grier,
Carole-Jean Wu, Arizona State University Lizy K. John, Marc Langheinrich, Torsten Möller,
Lixin Zhang, Chinese Academy of Sciences David Nicol, Ipek Ozkaya, George Pallis,
VS Subrahmanian
ADVISORY BOARD
CS PUBLICATIONS BOARD
David H. Albonesi, Erik R. Altman, Pradip Bose,
Fabrizio Lombardi (VP for Publications),
Kemal Ebcioglu, Lieven Eeckhout,
Alfredo Benso, Cristiana Bolchini,
Michael Flynn, Ruby B. Lee, Yale Patt,
Javier Bruguera, Carl K. Chang, Fred Douglis,
James E. Smith, Marc Tremblay
Sumi Helal, Shi-Min Hu, Sy-Yen Kuo,
Subscription change of address: Avi Mendelson, Stefano Zanero, Daniel Zeng
address.change@ieee.org
COMPUTER SOCIETY OFFICE
Missing or damaged copies:
help@computer.org IEEE MICRO
c/o IEEE Computer Society
10662 Los Vaqueros Circle
Los Alamitos, CA 90720 USA +1 (714) 821-8380

IEEE Micro (ISSN 0272-1732) is published bimonthly by the IEEE Computer Society. IEEE Headquarters, Three Park Ave., 17th Floor, New York,
NY 10016-5997; IEEE Computer Society Headquarters, 2001 L St., Ste. 700, Washington, DC 20036; IEEE Computer Society Publications Office,
10662 Los Vaqueros Circle, PO Box 3014, Los Alamitos, CA 90720. Postmaster: Send address changes and undelivered copies to IEEE, Member-
ship Processing Dept., 445 Hoes Ln., Piscataway, NJ 08855. Periodicals postage is paid at New York, NY, and at additional mailing offices. Canadian
GST #125634188. Canada Post Corp. (Canadian distribution) Publications Mail Agreement #40013885. Return undeliverable Canadian addresses
to 4960-2 Walker Road; Windsor, ON N9A 6J3. Printed in USA. Reuse rights and reprint permissions: Educational or personal use of this material is
permitted without fee, provided such use: 1) is not made for profit; 2) includes this notice and a full citation to the original work on the first page of the
copy; and 3) does not imply IEEE endorsement of any third-party products or services. Author and their companies are permitted to post the accepted
version of IEEE-copyrighted material on their own webservers without permission, provided that the IEEE copyright notice and a full citation to the
original work appear on the first screen of the posted copy. An accepted manuscript is a version which has been revised by the author to incorporate
review suggestions, but not the published version with copy-editing, proofreading, and formatting added by IEEE. For more information, please go to
ieee.org/publications_standards/publications/rights/paperversionpolicy.html. Permission to reprint/republish this material for commercial, advertising,
or promotional purposes or for creating new collective works for resale or redistribution must be obtained from IEEE by writing to the IEEE Intellectual
Property Rights Office, 445 Hoes Lane, Piscataway, NJ 08854-4141 or pubs-permissions@ieee.org. ©2019 by IEEE. All rights reserved. Abstracting
and library use: Abstracting is permitted with credit to the source. Libraries are permitted to photocopy for private use of patrons, provided the per-copy
fee indicated in the code at the bottom of the first page is paid through the Copyright Clearance Center, 222 Rosewood Drive, Danvers, MA 01923.
Editorial: Unless otherwise stated, bylined articles, as well as product and service descriptions, reflect the author’s or firm’s opinion. Inclusion in IEEE
Micro does not necessarily constitute an endorsement by IEEE or the Computer Society. All submissions are subject to editing for style, clarity, and
space. IEEE prohibits discrimination, harassment, and bullying. For more information, visit ieee.org/web/aboutus/whatis/policies/p9-26.html.
 July/August 2019
Volume 39 Number 4

Special Issue
55 T
he Queuing-First
Approach for Tail
6 S Guest Editors’ Introduction
ecure Architectures Management of Interactive
Services
Simha Sethumadhavan and Mohit Tiwari Amirhossein Mirhosseini
and Thomas F. Wenisch
Published by the IEEE Computer
Society

Theme Article

8 L everaging Cache Manage-

ment Hardware for Prac-
tical Defense Against Cache
Timing Channel Attacks
Fan Yao, Hongyu Fang, MiloŠ Doroslovački,
and Guru Venkataramani

General Interest

44 R
ASSA: Resistive
Prealignment
Accelerator for Approximate
DNA Long Read Mapping
Roman Kaplan, Leonid Yavits,
and Ran Ginosar
 Image credit: ©istockphoto.com/ValeryBrozhinsky

COLUMNS AND DEPARTMENTS 35 FinalFilter: Asserting

From the Editor-in-Chief Security Properties of a
4 Secure Architectures Processor at Runtime
Lizy Kurian John Cynthia Sturton, Matthew Hicks,
Samuel T. King, and
Jonathan M. Smith
Expert Opinion
17 Toward Postquantum Micro Economics
Security for Embedded 66 The Aftermath of the
Cores Dyn DDOS Attack
Rafael Misoczki, Sean Gulley, Shane Greenstein
Vinodh Gopal, Martin G. Dixon,
Hrvoje Vrsalovic, and
Wajdi K. Feghali
27 Energy-Secure System
Architectures (ESSA):
A Workshop Report
Pradip Bose and
Saibal Mukhopadhyay
 Column: From the Editor-in-Chief

Secure Architectures
Lizy Kurian John
The University of Texas at Austin

& WELCOME TO THE July/August 2019 issue of “The Aftermath of the Dyn DDOS Attack,” Green-
IEEE Micro, which presents to you a selection stein writes about the October 2016 series of dis-
of articles on secure architectures and a few tributed denial-of-service (DDOS) attacks causing
articles on other topics. many services and platforms to be unavailable for
Computer security is becoming increasingly large segments of users in North America and
important due to the increased use of computers Europe. The attacker targeted systems operated
and the internet in our day-to-day lives. Attacks by the nameserver resolution provider Dyn, who
on computer systems are becoming very com- performs approximately 10% of the nameserver
monplace. Many recent attacks demonstrated services in the United States. Since nameserver
security vulnerabilities in commodity hardware, resolution is essential for many businesses to
pointing to the importance of secure hardware operate, the attack affected a range of businesses
architectures. This special issue including Netflix, CNBC, Twitter,
presents four articles on secure Airbnb, and Etsy. Greenstein pro-
This special issue
computer architectures. The topics vides details on the market share
presents four articles
discussed range from defense of domain name system providers
on secure computer
against cache timing channel and alerts the readers to an impor-
architectures. The
attacks to asserting security prop- tant vulnerability in today’s inter-
topics discussed range
erties of a processor at runtime. net, viz. many internet services are
from defense against
Prof. Simha Sethumadhavan of concentrated in a few providers.
cache timing channel
Columbia University and Prof. Mohit His article draws attention to the
attacks to asserting
Tiwari of the University of Texas at security properties of a lack of redundancy in internet ser-
Austin served as guest editors for the processor at runtime. vice providers.
special issue. A comprehensive article In addition to the special issue
written by Professors Sethumadha- articles on computer security,
van and Tiwari serves as an excellent introduction there are two other articles in this issue. The first
to the compendium on secure architectures. one, “RASSA: Resistive Pre-Alignment Accelerator
The four articles from the secure architecture for Approximate DNA Long Read Mapping” by
theme are accompanied very appropriately by Kaplan et al., presents an in-memory parallel archi-
a Micro Economics column related to computer tecture for similarity search for genomic sequen-
security by Shane Greenstein. In the article titled ces. As personalized medicine based on gene
mapping is emerging, hardware architectures to
support sequence searches are increasingly rele-
Digital Object Identifier 10.1109/MM.2019.2924711 vant. One challenge in mapping long sequences is
Date of current version 23 July 2019. determining the optimal mapping location of every

0272-1732 ß 2019 IEEE Published by the IEEE Computer Society IEEE Micro
4
read on to the reference sequence. In this article,
Kaplan et al. present hardware acceleration of
genomic mapping by using resistive memories
(memristors) which are elements that store infor-
mation by modulating the resistance of the nano-
scale storage elements. Memristor arrays facilitate
simultaneous compare and mapping and result in
a highly parallel compute accelerator.
The second regular channel article is “The
Queuing-First Approach for Tail Management
of Interactive Services” by Mirhosseini and
Wenisch. Cloud services are increasingly becom-
ing popular, however, service latencies in the
cloud are heavy-tailed. Some requests can take
100 times more time than the average. This arti-
cle presents two solutions to mitigate this impor-
tant problem, server pooling, and common-case
service acceleration.
I am also proud to write about a new Best
Paper award for IEEE Micro. Starting this year, the
best paper award will be given for articles pub-
lished in IEEE Micro. IEEE Computer Society has
recently started a best paper award program to
July/August 2019
acknowledge and reward the best articles in each
of the Transactions and Magazines sponsored by
the Computer Society. Articles based on a confer-
ence paper are ineligible and hence the MICRO
TopPicks articles would be ineligible for this
award. The intent of the award is to recognize
outstanding regular papers. The first award will
be announced in a couple of months.
IEEE Micro is interested in submissions on any
aspect of chip/system design or architecture.
Please consider submitting articles to IEEE
Micro and remember, all regular articles will be
eligible for the best paper award.
Hope you enjoy the secure architectures as
well as other articles presented in this issue.
Happy reading!
Lizy Kurian John is a Cullen Trust for Higher
Education Endowed Professor with the Electrical and
Computer Engineering Department, the University of
Texas at Austin, Austin, TX, USA. Contact her at
ljohn@ece.utexas.edu.
5
 Guest Editors’ Introduction

Secure Architectures
Simha Sethumadhavan Mohit Tiwari
Columbia University The University of Texas at Austin

& HARDWARE IS THE bedrock on which all com- patterns and triggers dynamic partitioning once
puting systems are built. Recently developed an anomaly is detected. The article demonstrates
hardware ideas for enhancing the software ways to integrate this combined defense in off-
security from both academia and industry hold the-shelf CPUs by using Intel’s cache partitioning
significant promise to improve software security. techniques, and shows the way for future work
At the same time, recent hardware attacks on cur- that combines prevention and detection.
rent commodity hardware have shown hardware The second article by authors from Intel is at
to be a weak foundation for building secure sys- the intersection of extremely low-level firmware
tems. As we enter Post-Moore’s Law era, there are that often forms the root of trust for secure-hard-
significant questions surrounding ware-based computing, and secu-
what would make security techni- rity against quantum-computing
ques more practical. Thus, secu- As we enter Post-
driven cryptanalysis. This article
rity is a first-order problem for Moore’s Law era, there
considers several alternatives for
computer architects today, and are significant ques-
such "post-quantum" cryptographic
this special edition highlights a few tions surrounding what
would make security schemes specifically with an eye
compelling examples across the toward low-level implementation
techniques more prac-
system stack. This issue contains
tical. Thus, security is a security issues that emerge in a
four articles on various facets of
first-order problem for firmware. As secure enclave-based
secure architectures.
computer architects computing takes off, considering
The first paper by Fan et al.
today, and this special the foundation of secure firmware is
addresses the (timely) problem of edition highlights a few an especially timely problem.
information leakage through tim- compelling examples The third article is a report on
ing, specifically through shared across the system
caches. Defenses have considered a Workshop on Energy Secure
stack.
partitioning caches across security Architectures. The workshop was
domains as well as detecting organized by Dr. Bose from IBM
attacks as they happen as two separate research and Dr. Mukhopadhyay from Georgia Tech and
directions—this article proposes to use these focused on motivating the need to secure the on-
approaches in conjunction where the detection chip energy management structures, which are
algorithm looks for anomalous cache occupancy assuming increased importance in the face
of emerging technology trends. This article
describes the challenges in designing energy
Digital Object Identifier 10.1109/MM.2019.2925152
management algorithms, attacks, and possible
solutions.
Date of current version 23 July 2019.

0272-1732 ß 2019 IEEE Published by the IEEE Computer Society IEEE Micro
6
 The final article by Sturton et al. describes an
idea called the FinalFilter. Despite the best efforts
of designers to create secure designs, unforeseen
inputs or events may cause the processor to per-
form actions that violate security. In this article,
the authors describe how the use of simple
invariants can be used to protect against such
problems. Unlike static assertions that are
removed after design time, the FinalFilter invari-
ant controls access to protected assets at all times
because they are fabricated into the hardware.
July/August 2019
Simha Sethumadhavan is an associate profes-
sor of computer science and the Chair of Cyber-
security Center, Data Science Institute, Columbia
University. He is an Alfred P. Sloan Research Fel-
low. Contact him at simha@columbia.edu.
Mohit Tiwari is an assistant professor in the
Department of Electrical and Computer Engineering,
The University of Texas at Austin. He was a
Postdoctoral Fellow at UC Berkeley (2011–2013). He
has a PhD from UC Santa Barbara. Contact him
at tiwari@austin.utexas.edu.
7
 Theme Article: Secure Architectures

Leveraging Cache
Management Hardware for
Practical Defense Against
Cache Timing Channel
Attacks
Fan Yao Milo
s Doroslova
cki
University of Central Florida George Washington University
Hongyu Fang Guru Venkataramani
George Washington University George Washington University

Abstract—Sensitive information leakage through shared hardware structures is becoming

a growing security concern. In this article, we propose a practical protection framework
against cache timing channel attacks by leveraging commercial off-the-shelf hardware
support in last level caches for cache monitoring and partitioning.

& TIMING CHANNELS ARE a form of information
leakage attacks where adversaries modulate
and (or just) observe access timing to shared
resources in order to exfiltrate secrets. Among var-
ious hardware-based information leakage attacks,
cache timing channels have become notorious,
Digital Object Identifier 10.1109/MM.2019.2920814
Date of publication 4 June 2019; date of current version 23
July 2019.
since caches presenting the largest on-chip attack
surface for adversaries to exploit combined with
high bandwidth transfers.13 Previously proposed
detection and defense techniques against
cache timing attacks either explore hardware
modifications or incur nontrivial performance
overheads.6; 12; 16; 17 For more effective system
protection and widescale deployment, it is critical
to explore ready-to-use and performance-friendly
practical protection against cache timing channel
attacks.

0272-1732 ß 2019 IEEE Published by the IEEE Computer Society IEEE Micro
8
 In this article, we propose a spy in covert channels, and victim
new framework that makes novel For more effective and spy in side channels. The spy
system protection and
use of COTS hardware to thwart infers secrets from the trojan or
widescale deployment,
cache timing channels. We observe the victim by observing the modu-
it is critical to explore
that cache block replacements by lated latency of cache accesses.2
ready-to-use and
adversaries in cache timing chan- performance-friendly To exfiltrate secrets, the spy needs
nels reveal a distinctive pattern in practical protection to determine a communication
their cache occupancy profiles, against cache timing channel. In case of covert chan-
which could be a strong indicator channel attacks. nels, the trojan and spy may alter-
for the presence of cache timing nate their accesses to the cache
channels. We leverage Intel’s temporally, while in side channels,
Cache Monitoring Technology (CMT3) available the spy has to run in parallel to the victim pro-
in recent server-class processors to perform fine- cess.17 This may vary along space dimension as
grained monitoring of LLC occupancy for individ- well (i.e., access single cache location or alternate
ual application domains. We then apply signal among multiple cache locations).
processing techniques that characterize the com- Recently, Intel’s CMT allows for uniquely
munication strength with spy processes in cache identifying each logical core with a resource
timing channels. We further leverage LLC way monitoring ID,3 and track the LLC usage for the
allocation (i.e., CAT3) and repurpose it as a mapped domains. CMT enables flexible monitor-
secure cache manager to dynamically partition ing of LLC occupancy at user-desired domain
LLC for suspicious domains and disband their granularity such as a core, a multithreaded appli-
timing channel activity. Our mechanism avoids cation, or a virtual machine. With CAT, caches
preemptively separating domains and conse- can be configured to have several different parti-
quently, does not result in high performance tions on cache ways, called classes of service
overheads to benign application domains. (CLOS), where evicting cache lines from other
In comparison to our recent work—COTS- CLOS is restricted for a given domain.
knight,19 the novel contributions of this article
are as follows:
THREAT MODEL
1) To defend against sophisticated adversaries In this article, we focus on the sophisticated
that randomize interval times between trans- form of attacker that does not rely on any prior
missions, we augment our COTSKnight memory sharing, and utilizes Prime þ Probe-
design to remove irrelevant occupancy based techniques to launch attacks on LLC simply
trace segments using time warping (see by creating conflict misses (replacement) on
the “Defense Against Advanced Adversaries” cache sets. Attacks such as Flush þ Reload require
section). shared memory blocks either through shared
2) We perform new experimental studies on vir- libraries or data sharing that may be prohibited in
tualized environments that are prone to cache practical settings. Therefore, we do not consider
timing channel attacks, and demonstrate the such forms of attacks. However, for evict þ reload
efficacy of our approach (see the “Case Study attacks, where cache replacements alter access
on Virtualized Environments” section). latencies, our design would still be applicable.
3) We identify futuristic threats (like mult- (See the “Discussion” section for details.)
iple spies and evidence tampering), and
discuss potential defense mechanisms WHY CACHE OCCUPANCY
using our proposed defense framework (see PATTERNS MATTER?
the “Discussion” section). Regardless of whether a trojan intentionally
communicates or a victim unintentionally leaks
BACKGROUND secrets to a spy, cache timing channels use one
There are typically two processes involved in of the following encoding schemes: 1) ON–OFF
cache timing channels, namely, the trojan and encoding (where spy uses timing profile of a

July/August 2019
9
 Secure Architectures
Figure 1. LLC occupancy changes for trojan/victim and spy. (a) ON-OFF: Trjn/
Victim idle. (b) ON-OFF: Trjn/Victim access. (c) Pulse-pos: Odd sets. (d) Pulse-pos:
Even sets.
13
single cache set group to infer bits/symbols ),
and 2) pulse-position encoding (where spy lever-
ages access timing of distinct cache set groups
for inferring each bit/symbol14).
Figure 1 illustrates the changes in cache occu-
pancy under the two encoding methods. In ON–OFF
encoding, when trojan/victim accesses cache and
fetches its blocks, the trojan’s cache occupancy
should first increase and then decrease during
spy’s probe when trojan/victim-owned blocks are
replaced. Similarly, the spy’s cache footprint
would first decrease due to trojan/victim’s filling
in the cache blocks and then increase when spy
probes and fills the cache with its own data. When
trojan/victim does not access the cache, neither
of the processes change their respective cache
occupancies. Under pulse-position encoding with
two distinct cache set groups used by trojan/vic-
tim (e.g., odd and even sets), we observe swing
patterns in their cache occupancies.
10
Wemakethefollowingkeyobser-
vation here: Cache timing channels
fundamentally rely on cache block
replacements that create swing pat-
terns in participating domain’s
cache occupancy regardless of the
specific timing channel protocols.
By analyzing these repetitive
swing patterns, there is a poten-
tial to uncover the communication
strength in such attacks. We note
that merely tracking cache misses
onanadversarywillnotbesufficient
as an attacker may inflate cache
misses (through issuing additional
cache loads that create self-
conflicts) on purpose to evade
detection.
SYSTEM DESIGN
Here, we first discuss CMT-
based cache occupancy monitor-
ing and trace analysis for cache
timing channel detection, and
then outline cache partitioning
strategy to prevent information
leakage.
Cache Occupancy Monitor and
Pattern Analyzer
We leverage Intel CMT10 to obtain LLC occu-
pancy data for each domain/context that allows
the system administrators to flexibly define mon-
itoring granularity, e.g., hardware threads, appli-
cations, or even VMs.
The occupancy pattern analyzer performs
the following steps to determine whether
there is a cache timing channel between two
domains.
First, the analyzer generates the time-
differentiated cache occupancy changes for
each domain. Assume that xi and yi are the
cache occupancy sample vectors obtained
within the ith window, we can then get the time-
differentiated cache occupancy traces for each
domain, denoted as Dxi;j and Dyi;j (i.e., the LLC
occupancy difference between two consecutive
samples). Figure 2(a) shows time-differentiated
LLC occupancy traces for a timing channel
IEEE Micro
that implements parallel
protocol with pulse-position
encoding.
In the second step, to
capture the unique pairwise
cache occupancy swing
pattern in timing channels,
we compute the product of
Dxi and Dyi as zi . Based on
the discussion in the “Why
Cache Occupancy Patterns
Matter?” section negative
values of zi occur when the
cache occupancy patterns
of the two processes move
in opposite directions due
to mutual cache evictions.
In the third step, our ana-
lyzer checks if z series con-
tains repeating negative
pulses that may be caused
by intentional eviction over a
longer period of time (denot- Figure 2. LLC occupancy traces, autocorrelogram, and power spectrum for a cache
ing illegal communication timing channel (with parallel pulse-position).13 (a) Parallel Protocol with pulse-position
activity). To capture the encoding. (b) Autocorrelogram (left) and power spectrum (right).
repetitive swing patterns, we
perform power spectrum
analysis in frequency domain Implementation
on ri , which is the autocorrelogram of z. We implement our framework prototype on a
Figure 2(b) illustrates the autocorrelogram real system with Intel Xeon E5-2698 v4 Proc-
and power spectrum for a (victim, spy) pair in essor. The processor comes with 16 CLOS and
timing channels.13 We can visually observe a 20 LLC slices, and each LLC slice has 20  2048
sharp peak around frequency of 290 in the 64-byte blocks. The LLC occupancy MSR reading
power spectrum, which represents a strong is sampled at 1000/s.
communication strength indicating timing chan-
nel activity (see COTSKnight19 for further
details on this algorithm).
EVALUATION
Power Spectra for Cache Timing Channels
Cache Way Allocation Manager We setup attack variants of cache timing
After the way allocation manager (allocator) is channels2; 13–15 that utilize ON–OFF and pulse-
notified of identified suspicious domains from the position encoding for spy reception and per-
analyzer, it will configure LLC using CAT to isolate form accesses to cache either serially (trojan
the suspicious pairs by heuristically assigning and spy) or in parallel (victim and spy) as
nonoverlapping cache ways to each domain based described in the “Why Cache Occupancy PAT-
on their ratio of LLC occupancy sizes during the TERNS MATTER?” section. In each case, we
last observation period. Our allocator evaluates also ran along side with at least two SPEC2006
two candidate policies, namely, Aggressive Policy, benchmarks with high LLC activity.8 The ana-
that keeps suspicious domains separated until lyzer performs power spectrum analysis based
one of them finishes execution and Jail Policy, that on time-differentiated LLC occupancy traces
partitions the two domains until a timeout period. for six combination pairs of processes. In all

July/August 2019
11
 Secure Architectures

trigger cache partitioning. Note that

we have analyzed the attack variants
with different transmission bit rates
(i.e., ranging from a few bps to several
kbps), numbers of cache sets, and
probe intervals. Our results showed
that our framework identifies all of
the trojan–spy domain pairs within
five consecutive analysis windows
after they start execution.
Figure 3. Power spectrum in attack variants including trojan/victim-spy pairs. Partition trigger rate for benign work-
(a) serial-on–off (b) para-pp loads and the corresponding perfor-
mance impact: Among all benign
workloads (each runs four SPEC2006
cases, our framework correctly identified tro- applications), only 6% of the domain pair popula-
jan/victim-spy processes since the pair consis- tion had LLC partitioning—these benchmarks cov-
tently had the highest power in the frequency ered 2% of the analysis window samples. Even
domain. In fact, our experiments show that when there are only two benign applications, it is
the attacker pair’s peak power spectrum val- worth noting that cache occupancy change pat-
ues are at least an order of magnitude higher terns are typically random. Therefore, signal power
than benign application pairs. (that captures the periodic gain-loss patterns) will
Figure 3(a) and (b) shows the analyzer’s not be any higher. Our experiment shows that the
results on representative windows for two tro- LLC partitioning only minimally impacts applica-
jan/victim-spy pairs. In the serial-on–off attack, tions that trigger partition (with less than 5% slow-
we observe a single concentrated and sharp down), and interestingly, we observe performance
peak with the power value in the frequency boost for many of them (up to 9.2%). The overall
domain, while the other data points are almost average impact on all the applications that ran
all zeros. This indicates the existence of a domi- with partitioned LLC was positive (about 1%). This
nating signal in the time domain corresponding shows that our framework can even help benign
to the repetitive gain-loss occupancy pulses due workloads while safeguarding systems against
to timing channel activity [see Figure 3(a)]. We cache timing channels.
also observe a similar isolated peak for the tro-
Runtime Overhead: Our framework implements
jan/victim-spy pair in para-pp, as shown in
the nonintrusive LLC occupancy monitoring for
Figure 3(b) where the signal power is even
only mutually distrusting domains identified by
higher compared to serial-on–off case.
the system administrator. Overall, the mechanism
We repeated several experiments with 60
incurs less than 4% CPU utilization with four
benign workload pairs with high LLC activity
active mutually distrusting domains.
time-overlapping at various random phases, and
observed the peak signal power to be less than 5
Defense Against Advanced Adversaries
about 80% of the time, and around 50 for only
In theory, advanced adversaries may use ran-
about 2% of the time. This shows that a vast
domized interval times between bit transmissi-
majority of benign workload samples do not
ons. Let us imagine a trojan and spy that setup a
exhibit isolated peaks in the frequency domain,
predetermined pseudorandom number generator
and the maximum signal power is significantly
to decide the next waiting period before bit trans-
less than any known timing channels (that have
mission. Even in such cases, our framework can
signal strengths at well over 100).
be adapted to recognize them through a signal
preprocessing procedure called time warping,5
Effectiveness of Our Framework that removes irrelevant segments from the occu-
Defeating Cache Timing Channels: We conser- pancy traces (for which Dx; Dy are close to 0 and
vatively set signal power threshold at 50 to aligns the swing patterns). After this step, the

12 IEEE Micro
periodic patterns are recon-
structed, and the cadence of
cache accesses from adver-
saries will be recovered.
Figure 4 demonstrates the
detection of this attack sce-
nario. For illustration, we
implement a prototype of
this attack by setting up the
trojan and spy as two
threads within the same pro-
cess, and configure the main
thread to control the syn-
chronization. In reality, two
separate trojan/victim and
spy need to be synchro-
nized. Figure 4(a) shows the
LLC occupancy trace for this
attack with random distan-
ces between the swing
pulses. We can see that, with
time warping, high signal
power peaks are observed Figure 4. Analysis of bit transmission at random intervals. (a) Left half shows a snippet
[see Figure 4(b)]. Addition- of original trace with random bit intervals and right half shows time-warped trace. (a)
ally, when this signal com- LLC occupancy changes for transmision with random intervals. (b) Power spectrum on
pression preprocessing step time-warped LLC occupancy trace.
is applied on benign work-
loads, we do not observe any
increase in partition trigger rate. allocation determination during the entire execu-
tion. We can see that the trojan and spy start to ini-
Case Study on Virtualized Environments tiate communication at around 188 s (when we
To evaluate the efficacy of our proposed start to observe increasing signal power). The peak
framework, we perform a case study on virtual- signal power between the trojan and spy domain
ized environment. This study is motivated by pair quickly climbs up to 126 at time 192.5 s, which
growing trend in studying timing channel attacks is when steady covert communication has begun.
in the cloud environment. We implement the para- This quickly triggers the allocator’s action that
onoff attack that works cross-VM (similar to Maur- splits the LLC ways between trojan and spy VMs.
ice et al.14) Consequently, the maximum signal power drops
We setup four KVM virtual machines where the back to nearly zero for the rest of execution, effec-
trojan and spy run on two of the VMs, and simulta- tively preventing any further timing channels. Note
neously, two other VMs corun representative that during the 1-h experiment, the peak signal
cloud benchmarks, namely video streaming power values for the other domain pairs (involving
(stream) and memcached (memcd) from Cloud- Cloudsuite applications) remained flat at values
Suite,7 both of which are highly cache-intensive. less than 3.
The trojan and spy are set to start the para-on–off
attack at a random time between 0 and 300 s.
We configure the allocator to use the Aggressive DISCUSSION
policy to demonstrate the effectiveness of LLC par- We propose a new framework that builds on
titioning. Figure 5 shows the peak signal power COTS hardware and can be augmented with a
between the trojan and spy VM pair and the way host of signal processing techniques to eliminate

July/August 2019
13
 Secure Architectures

space or just disable it (e.g.,

Google NaCl). Therefore,
clflush-based cache occupancy
deflation can be handled
easily.
Applicability of Our Tech-
nique to Other Cache Attacks:
While we mainly evaluated the
proposed technique using
Figure 5. Peak signal power values for the trojan/spy pair and the allocator’s way Prime þ Probe-based attacks,
allocation for one hour execution. our proposed framework can be
applied to other cache attacks
noise, randomness, or distortion to unveil the tim- using evictions as well. For instance, in Evict þ
ing channel activity. In this section, we discuss Reload attacks, the repetitive data loads by the
additional monitoring support and signal process- victim and subsequent evictions by the spy will
ing to detect futuristic attacks with sophisticated also introduce cache occupancy gain–loss pat-
adversaries. terns, which can be detected by our proposed
Using Multiple Spy Processes: A spy may try to framework.
evade our defense through potentially involving Current Hardware Limitations and Opportuni-
multiple processes that perform either time-multi- ties: We observe that CMT currently supports a
plexing (each process is active for a short period of minimum precision of 20 cache sets. If attackers
time iteratively) or space multiplexing (each pro- were to leverage less number of sets to carry out
cess touches a subregion of target attack, they may potentially evade COTSknight’s
sets simultaneously) for timing channels. Our pro- detection. While such attacks are possible, they
posed framework can still effectively identify such are prone to high noise. As such, the limitations
malicious activities as it essentially monitors swing mentioned above are an artifact of the current
patterns in cache occupancy usage that could CMT hardware, and not of our analysis
purposefully change cache access latencies for approach per se. That said, we note that CMT
domains as discussed in the “System Design” sec- was designed for improving performance bottle-
tion. Further, CMT þ CAT allows for dynamically necks, and not to detect cache timing channels.
defining security domains that can best isolate the Our study highlights a novel use case for LLC
capability and access boundary for each party monitoring, and we strongly believe that it
(e.g., threads and processes run by the same user would motivate processor vendors to support
belong to the same domain). The cumulative LLC improved precision and bolster system security.
occupancy pattern among all the spy’s processes
in the same domain would preserve the correlated
swing pattern that can be recognized by the RELATED WORK
analyzer. Cache-based timing channels have been widely
Using clflush to Deflate LLC Occupancy: An studied,13; 18 and hardware-based solutions have
adversary may attempt to tamper evidence of its been proposed. CC-Hunter2 detects covert timing
cache occupancy changes by compensating the channel in caches by capturing fined-grained
increase in its own cache occupancy through cache conflict miss patterns in hardware. Replay-
issuing clflush instruction. To handle such sce- Confusion17 records program’s memory accesses
narios, clflush’s usage by suspicious domains and replays them on a different machine to
may be tracked and the associated memory sizes uncover covert channels on caches. Hunger et al.9
can be accounted back to the issuing core, thus, observe the destructive read property in conten-
restoring original occupancy data for analysis. tion-based covert channel and propose a solution
Also, many system-level protections against based on anomaly detection. Demme et al.4 apply
clflush instruction have been proposed, includ- machine learning techniques on architectural-level
ing constraining clflush to only be used in kernel statistics to detect malware including side

14 IEEE Micro
channels. Fang et al.6 also discussed several futuristic threats and
We proposed a novel
use hardware pre- mechanisms to defeat such timing channels.
framework to protect
fetchers to defend caches against timing
against cache timing channel attacks ACKNOWLEDGMENT
channels. through smartly This work was supported by the U.S. National
CATalyst12 uti- leveraging COTS Science Foundation under Grant CNS-1618786,
lizes the CAT tech- support for cache
and by the Semiconductor Research Corpora-
nology to reserve monitoring and
tion Contract 2016-TS-2684. F. Yao performed
static cache parti- performance tuning.
this work as a graduate student at GWU.
tions where secure We implemented a
pages are pinned prototype of our
proposed technique on
upon request from
applications. Differ-
Intel Xeon v4 server & REFERENCES
and our experiments
ently, our pro- 1. M. Bazm, T. Sautereau, M. Lacoste, M. Sudholt,
showed that our
posed mechanism framework can and J. Menaud, “Cache-based side-channel
successfully defeat successfully thwart attacks detection through Intel Cache Monitoring
cache timing chan- several classes of Technology and Hardware Performance Counters,”
nels without appli- cache timing channels in Proc. 3rd Int. Conf. Fog Mobile Edge Comput.,
cation/user-level in both native and 2018, pp. 7–12.
inputs and parti- virtualized environment 2. J. Chen and G. Venkataramani, “CC-hunter:
tion reservation. with minimal Uncovering covert timing channels on shared
1
Bazm et al. lever- performance overhead. processor hardware,” in Proc. 47th Annu. IEEE/ACM
age cache occu- Int. Symp. Microarchit., 2014, pp. 216–228.
pancy information to detect side channels 3. Intel Corporation, Intel 64 and IA-32 Architectures
behavior in conjunction with other perfor- Software Developer’s Manual, vol. 3B, 2016.
mance counters such as cache misses. How- 4. J. Demme et al., “On the feasibility of online
ever, their proposed technique makes malware detection with performance counters,” in
anomalous behavior determination based on Proc. 40th Annu. Int. Symp. Comput. Archit., 2013,
cache footprint, which is subject to high false pp. 559–570.
positive alarms. In contrast, our framework 5. M. G. Elfeky, W. G. Aref, and A. K. Elmagarmid, “Warp:
analyzes cache occupancy gain–loss patterns Time warping for periodicity detection,” in Proc. 5th
that are shown to be the unique characteristic IEEE Int. Conf. Data Mining, 2005, pp. 138–145.
for parties involving timing channel activity,
ki, and
6. H. Fang, S. S. Dayapule, F.Yao, M. Doroslovac
which is both effective and efficient. Recently, G. Venkataramani, “Prefetch-guard: Leveraging
DAWG11 has proposed secure cache partition- hardware prefetches to defend against cache timing
ing by strictly isolating both cache hits and channels,” in Proc. IEEE Int. Symp. Hardware Oriented
misses between application domains. Secur. Trust, 2018, pp. 187–190.
7. M. Ferdman et al., “Clearing the clouds: A study of
emerging scale-out workloads on modern hardware,”
CONCLUSION ACM SIGPLAN Notices, vol. 47, pp. 37–48, 2012.
In this article, we proposed a novel framework 8. J. L. Henning, “SPEC CPU2006 benchmark
to protect caches against timing channel attacks descriptions,” ACM SIGARCH Comput. Archit. News,
through smartly leveraging COTS support for vol. 34, no. 4, pp. 1–17, 2006.
cache monitoring and performance tuning. We 9. C. Hunger, M. Kazdagli, A. Rawat, A. Dimakis,
implemented a prototype of our proposed tech- S. Vishwanath, and M. Tiwari, “Understanding
nique on Intel Xeon v4 server, and our experi- contention-based channels and using them for
ments showed that our framework can defense,” in Proc. IEEE 21st Int. Symp. High Perform.
successfully thwart several classes of cache tim- Comput. Archit., 2015, pp. 639–650.
ing channels in both native and virtualized envi- 10. Intel, “Intel-CMT-CAT Pacakage,” 2017. [Online].
ronment with minimal performance overhead. We Available: http://https://github.com/01org/intel-cmt-cat

July/August 2019
15
 Secure Architectures
11. V. Kiriansky, I. Lebedev, S. Amarasinghe, S. Devadas,
and J. Emer, “DAWG: A defense against cache timing
attacks in speculative execution processors,” in Proc.
51st Annu. IEEE/ACM Int. Symp. Microarchit., 2018,
pp. 974–987.
12. F. Liu et al., “CATalyst: Defeating last-level cache side
channel attacks in cloud computing,” in Proc. IEEE Int.
Symp. High Perform. Comput. Archit., 2016, pp. 406–418.
13. F. Liu, Y. Yarom, Q. Ge, G. Heiser, and R. B. Lee, “Last-
level cache side-channel attacks are practical,” in Proc.
IEEE Symp. Secur. Privacy, 2015, pp. 605–622.
14. C. Maurice et al., “Hello from the other side: SSH over
robust cache covert channels in the cloud,” in Proc.
Netw. Distrib. Syst. Secur. Symp., 2017, pp. 8–11.
15. T. Ristenpart, E. Tromer, H. Shacham, and S. Savage,
“Hey, you, get off of my cloud: Exploring information
leakage in third-party compute clouds,” in Proc.
16th ACM Conf. Comput. Commun. Secur., 2009,
pp. 199–212.

ki,
16. G. Venkataramani, J. Chen, and M. Doroslovac
“Detecting hardware covert timing channels,” IEEE
Micro, vol. 36, no. 5, pp. 17–27, Sep.–Oct. 2016.
17. M. Yan, Y. Shalabi, and J. Torrellas, “Replayconfusion:
Detecting cache-based covert channel attacks using
record and replay,” in Proc. 49th Annu. IEEE/ACM Int.
Symp. Microarchit., 2016, pp. 1–14.

ki, and G. Venkataramani, “Are
18. F. Yao, M. Doroslovac
coherence protocol states vulnerable to information
leakage?” in Proc. IEEE Int. Symp. High Perform.
Comput. Archit., 2018, pp. 168–179.
16

ki, and G. Venkataramani,
19. F. Yao, H. Fang, M. Doroslovac
“COTSknight: Practical defense against cache timing
channel attacks using cache monitoring and partitioning
technologies,” in Proc. HOST, 2019.
Fan Yao is an assistant professor of electrical and
computer engineering at the University of Central
Florida. His research interests include the areas
of computer architecture, hardware and system
security and cloud computing. Contact him at fan.
yao@ucf.edu.
Hongyu Fang is currently a graduate student at
George Washington University. His research interests
include signal processing, computer architecture, and
security. Contact him at hongyufang_ee@email.gwu.
edu.
Milos
Doroslovac
ki is an associate professor
of electrical and computer engineering at George
Washington University. His research area is adap-
tive signal processing with focus on communica-
tions and distributed estimation. Contact him at
doroslov@gwu.edu.
Guru Venkataramani is an associate professor
of electrical and computer engineering at George
Washington University. His research area is com-
puter architecture, security, and energy optimiza-
tion. Contact him at guruv@gwu.edu.
IEEE Micro
 Expert Opinion
Toward Postquantum
Security for Embedded
Cores
Rafael Misoczki, Sean Gulley, Vinodh Gopal,
Martin G. Dixon, Hrvoje Vrsalovic, and
Wajdi K. Feghali
Intel Corporation
& THE USE OF firmware agents—including their
use to define the functionality of embedded
cores—has proliferated on computer systems of
all scales, especially servers. The agents are
often not visible to the operating system as they
independently perform configuration, monitor-
ing, and certain control tasks. For example,
the baseboard management controller (BMC)
on server platforms runs a firmware stack
(e.g., OpenBMC (https://github.com/openbmc/
openbmc)), has a network port, some periph-
erals on external buses (e.g., I2 C’s, SPI, etc.),
and storage. The BMC is effectively another
full (but scaled-down) computing system on the
server of which it is a component. While the
BMC can directly affect the operation of a server
(e.g., by controlling its power states), it does not
interact with the OS. Its behavior is defined
completely by its firmware.
Digital Object Identifier 10.1109/MM.2019.2920203
Date of current version 23 July 2019.
July/August 2019 Published by the IEEE Computer Society
Improvements in silicon manufacturing pro-
cesses have reduced the size of these cores to
the point where they can be physically embed-
ded within silicon dies of the system compo-
nents (such as a CPU) on which they operate. In
parallel, the volume and importance of their
responsibilities have increased, beginning with
power management and escalating to security
operations that can affect functional safety.
Given this, it is critical that they run only signed
and authenticated code.
One of today’s best practices to authenticate
the firmware that runs embedded cores is public
key cryptography (digital signatures), which
relies on FIPS-140 digital signature algorithms
such as RSA and EC-DSA. However, quantum
computing will render these algorithms useless
since factorizing integers and solving the dis-
crete logarithm problem (i.e., the underlying
security problems of RSA and EC-DSA) will be
solvable in polynomial time.11 This implies that
increasing RSA/ECC key sizes will be insufficient
to defeat a quantum adversary. Prof. Michele
0272-1732 ß 2019 IEEE
17
 Expert Opinion

Mosca, from the Institute of Quan- microcontrollers in lieu of hard-

One of today’s
tum Computing of the University of wired logic. Today, the same pro-
best practices to
Waterloo, predicts that there is a authenticate the firm- cessors contain even more
50% chance that RSA-2048 public ware that runs embed- embedded agents.
key cryptosystem will be broken ded cores is public key In a typical new design from
by 2031 by quantum computers. In cryptography (digital Intel, there are microcontrollers
response to this threat, the signatures), which to handle: USB Type-C port
relies on FIPS-140 digi- switching, reconfiguration of ana-
National Institute of Standards and
tal signature algorithms log lanes, the display engine,
Technology (NIST) has started a
such as RSA and EC-
competition for standardizing post- graphics memory translations,
DSA. However, quan-
quantum cryptography (PQC). temperature compensation for
tum computing will ren-
Additionally, Grover’s algorithm 5
der these algorithms analog circuits, hardware and
challenges AES-128 and symmetric useless since factoriz- software debugging support,
cryptography algorithms of short ing integers and solv- image processing, asset adminis-
key/digest size, in general. ing the discrete tration and manageability (such
logarithm problem (i.e., as Intel’s AMT features), security,
In response, cryptographers
the underlying security packet processing, and more.
have proposed algorithms that can
problems of RSA and These microcontrollers exist
be classified into several families. EC-DSA) will be solv-
Since embedded cores cannot physically discrete in the silicon
able in polynomial time.
house large keys and are not typi- dies and execute independently
cally high-performance to handle of the execution of the primary
complex operations, one interesting question to Intel CPU(s) or governance by the operating sys-
ask is how suitable these PQC algorithms are for tem; their functionality and behavior are con-
embedded codes. In this paper, we will discuss trolled by firmware that is usually contained in
the proliferation of embedded cores, why secu- flash memory local to the microcontroller core
rity depends upon the firmware running on itself (i.e., embedded in the same silicon die).
them, provide an overview of the existing The proliferation of microcontrollers as
proposals for postquantum digital signatures, embedded agents has been enabled by multi-
discuss what approaches seem the most reason- ple factors: the availability of controllers in a
able for embedded cores, and discuss Intel’s cur- variety of sizes due to improvements in
rent direction. manufacturing processes, the desire for flexi-
bility of designs, and demand for in-the-field
updates. Microcontroller cores are available in
FIRMWARE AGENT PROLIFERATION sizes ranging from ten thousand gates to hun-
In the 1990s, a processor such as the Pentium dreds of thousands of gates. For example, an
Pro Processor from Intel began using a firmware exemplary microcontroller with 64 KB of SRAM
2
binary that defined or redefined certain opera- occupies approximately 0.1 mm on a 10-nm
tions of that CPU, which came to be known as process or around 0.1% of a client processor.
Intel Microcode (or simply microcode). As the Software tools and runtimes for popular cores
complexity of CPU and chipset designs grew, Intel (e.g., ARM-based) allow sophisticated develop-
added a separate microcontroller core (Foxton) ment flows similar to regular desktop/server
to the design of the Core i7 processor code- software; system designers can introduce new
named Nehalem. This microcontroller was functionality to a computer system through
tasked with power management, and it ran in con- these controllers using modern, high-level lan-
junction with the microcode. At the same time, guages (e.g., Rust). This late-binding flexibility
the Platform Controller Hub began to add micro- compensates for the lengthening of hardware
controllers for audio, power management, and development cycles and fabrication times,
manageability. SoC’s from the rest of the industry leading to more firmware agents working in
followed the same paradigm, embedding conjunction in these designs. On the other

18 IEEE Micro
hand, such proliferation of the embedding of
microcontrollers, their increased handling of
operation-critical tasks for the overall system,
and the easy access to sophisticated develop-
ment tools have made microcontroller cores’
firmware an attractive target for malfeasance.
For product assurance, the firmware needs
to be authenticated and protected against
tampering.
FIRMWARE SIGNATURES
To assert that the code running on a given
microcontroller originated from the expected
source (the vendor) and has not been tampered
since it was deployed, the firmware image needs
to have ways by which it can be signed and
attested; there must exist the ability to assure
that the firmware originated from a specific ven-
dor (e.g., Intel) and that the bits that comprise
the firmware binary have not been altered by a
3rd party from the time it was installed to the
time it is being loaded. This is commonly done
with digital signature verification. A simplified
example of performing this verification is to sign
a message digest (or hash) of the firmware
binary with a vendor private key and append it
to the firmware. A system that has access to the
firmware vendor’s public key can then correctly
verify the authenticity of this hash value. Any
signature verification failure signals a modified
firmware or the use of a foreign private key (i.e.,
the signature did not originate from the vendor).
Thus, this mechanism prevents attackers from
generating valid signatures of modified firmware.
In practice, firmware signature verification is
complicated by the constraints on the environ-
ment in which the verification is to take place;
since it usually occurs (at least once) very early
in the system boot process, limited memory,
and/or computational resources are available.
To provide a tangible, real-world example of
firmware signatures, we discuss Intel’s micro-
code patch.
Intel Microcode Signing
The constraints on microcode are signifi-
cant. Microcode may be loaded multiple times
from the power-on of a processor, and since it
must be verified each time, the duration to
July/August 2019
decrypt and verify is critical to system respon-
siveness. Additionally, since main memory is
not available in the earliest stages of initializa-
tion, microcode verification is done within the
processor’s cache so it must fit into a rela-
tively small footprint (e.g., 16 to 64 KB). Micro-
code as a firmware mainly differs from a
traditional microcontroller firmware image by
its target processor core: The Core and Atom
processors are multiple orders of magnitude
higher performance than a typically embedded
microcontroller core. While microcode, having
the benefit of a much faster execution via the
main CPU, could potentially adopt much more
cryptographically stronger (but more computa-
tion-resource demanding) algorithms, those
same algorithms may be unsuitable for the
more constrained embedded cores.
STATE OF ART ON POSTQUANTUM
DIGITAL SIGNATURES
There has been a recent surge in activity in
the field of PQC, with several new schemes
proposed every year and a rapidly growing
community of researchers that scrutinize the
robustness, performance and ease of use of
new and well-established postquantum crypto-
systems. This increased interest may be
related to the recently established standardiza-
tion processes on PQC. The National Institute
of Standardization and Technology started a
project to analyze possible PQC candidates for
standardization. The Internet Engineering Task
Force (IETF) has recently published Request
for Comments (RFC’s) informational docu-
ments that specify stateful postquantum digital
signatures such as the XMSS scheme.1 Experts
from the International Standards Organization
have also been working on a standing docu-
ment on PQC and, more recently, on a study-
period on stateful hash-based signatures. In
this section, we provide a brief analysis of the
maturity, robustness, and performance of the
signature schemes considered in the aforemen-
tioned PQC standardization processes.
Since most of the schemes discussed in this
section have been submitted to the NIST PQC
project, we will give some additional context
about this process. In November 2017, NIST
19
 Expert Opinion
received 19 submissions on postquantum state-
less digital signatures for the 1st round of their
PQC standardization competition. Among those,
five were lattice-based, two code-based, seven
multivariate-quadratic based, three symmetric-
crypto-based schemes, and two others that
could not be classified in any of the previous cat-
egories. In January 2019, NIST selected only nine
submissions to pass on to the second round of
their competition. Among those, three are lat-
tice-based, four are multivariate-quadratic
based, and two are symmetric-crypto-based
schemes. According to NIST, the evaluation cri-
teria used to select the second round candidates
was security (formal security proof and resis-
tance to side-channel attacks), cost and perfor-
mance (size of public key and signature,
computational efficiency, and probability of fail-
ures), and algorithm and implementation charac-
teristics (parameters flexibility, parallelism
amenability, and simplicity). NIST defined a few
security levels for their competition. Level 1
should match the postquantum security of
AES128, level 3 should match the postquantum
security of AES192, and level 5 should match
the postquantum security of AES256. Our perfor-
mance analysis focuses on parameter sets that
achieve security levels 1 and 5 (whenever possi-
ble), and the reference implementations written
by the scheme designers.
Symmetric-Crypto-Based Schemes
From a security perspective, these are the
most conservative schemes since their secu-
rity relies uniquely on the security of hash or
block ciphers, thus they do not introduce any
additional security assumptions. This category
splits into two subcategories: stateful and
stateless schemes. Stateful signature schemes
require the secure storage of some state
(data) in between signatures generation. State-
less signature schemes do not have this addi-
tional requirement. For example, RSA and EC-
DSA are stateless schemes.
In case state management is a doable task,
i.e., maintaining a piece of data securely in
between signature generation, the stateful
schemes should be considered. We remark that
code signing applications (such as firmware
20
authentication) seem to be among the most
suitable applications for stateful schemes since
the manufacturer can carefully implement state
management.
The XMSS Scheme1 is a stateful hash-based
signature that has been recently published as
an informational RFC by the IETF. It can be
regarded as an evolved version of the classical
Merkle scheme. From a security perspective,
XMSS enjoys a security proof in the standard
model based on mild security assumptions
from the hash function (e.g., pre-image and tar-
get collision resistance). From a performance
perspective, since full collision resistance is
not needed, XMSS can operate with smaller
hash digests than the Merkle scheme. This
leads to shorter signatures and faster process-
ing. For 128 bits of quantum security, XMSS
requires 64 bytes of a public key and 2.44 KB
for the signature, and verification takes about
760,000 cycles in a modern processor. For 64
bits of quantum security, XMSS requires 32
bytes of the public key and 740 bytes of signa-
tures, and verification takes about 390,000
cycles in the same machine.
In the stateless subcategory, we consider
SPHINCSþ and PICNIC. SPHINCSþ12 is a state-
less hash-based signature scheme that uses
huge Merkle trees (e.g., height of 60). It is
stateless because it selects the Merkle leaf
nodes at random, and the chance of acciden-
tally reusing the same leaf node twice (which
would void its security guarantees) is negligi-
ble given the huge number of leaf nodes (e.g.,
260). SPHINCSþ offers two sets of parameters
per security level (thus six in total), one set
optimized for speed and another optimized for
small signatures. We focus our analysis on the
speed optimized parameters using SHA-256.
For level 1, SPHINCSþ offers signatures of
16.57 KB, public keys of 32 bytes; signing takes
340 million cycles, and verification takes
14.98 million cycles (“SPHINCSþ-SHA-256-128f-
robust” parameter set). For level 5, these num-
bers grow to 48.06 KB, 64 bytes; 1,491.94 mil-
lion and 37.59 million cycles (“SPHINCSþ-SHA-
256-256f-robust” parameter set), respectively.
PICNIC8 is a signature algorithm based on a
block cipher (LowMC) and a noninteractive
IEEE Micro
proof of knowledge. LowMC has been chosen
as the underlying block cipher since it gives
smaller signature sizes. Still, the signature for
level 1 is 33.23 KB and the public key size is 32
bytes (“picnic-L1-FS” parameter set). For level
5, signature size is 129.74 KB and the public
key size is 64 bytes (“picnic-L5-FS” parameter
set). For level 1, signing takes 137.91 million
cycles and verification takes 90.63 million
cycles, while for level 5 it takes 1,112.23 million
and 736.31 million cycles, respectively.
The symmetric-crypto-based candidates
have very strong security guarantees. In case
state management is possible, stateful HBS
schemes may be the most promising approach
since they are interesting from both security and
performance perspectives. It is worth mention-
ing that digital signatures applied to verify firm-
ware authenticity of embedded cores seem one
application where state management seems pos-
sible. The fact that the signatures are generated
by manufacturers (and not end users) that can
afford a robust signing facility with state man-
agement capabilities seems to facilitate the
adoption of stateful HBS schemes in this sce-
nario. Also, the other limitation of certain state-
ful HBS schemes that can issue a limited number
of signatures does not seem a problem since
manufacturers can, most of the times, predict
how many firmware updates a device will receive
during its lifetime. On the other hand, in case
state management is not possible, stateless
schemes may be advisable; however, they are
less attractive from a performance perspective
from both size and speed metrics.
Lattice-Based Schemes
In this category, we have Falcon,3 CRYSTALS-
DILITHIUM,2 and qTesla.9 From the PQC families
that introduce additional security assumptions,
lattices are one of the most popular approaches.
From a side-channel perspective, the Gaussian
sampling process seems to offer some challenges
to be implemented in a side-channel resilient way.
Falcon is based on the Short Integer Solu-
tion problem, known in the crypto community
for some time, but it is applied to (structured)
NTRU lattices. For level 1, Falcon signatures
have 617 bytes and the public key has 897
July/August 2019
bytes, signing takes 542,000 cycles and verifi-
cation takes 88,000 cycles. For level 5, these
numbers change to 1.20 KB, 1.75 KB, 1.07 mil-
lion cycles, and 186,000 cycles, respectively.
CRYSTALS-DILITHIUM uses module lattices
problems, which can be viewed in between the
ones used in Learning-With-Errors (LWE) and
Ring-LWE problems. In other words, according
to the authors, they are just as efficient as Ring-
LWE schemes but closer to the (stronger, more
conservative) LWE underlying security problem.
For levels 1 and 3, CRYSTALS-DILITHIUM offers
signatures of size 1.99 and 3.28 KB, and public
keys of size 1.15 and 1.71 KB, respectively. The
authors did not provide parameters for level 5.
In terms of speed, for level 1, it takes 1.3 million
cycles to sign and 272,000 cycles to verify. For
level 3, it takes 1.82 million and 510,000 cycles,
respectively.
qTesla is based on the well-known Ring-LWE
problem, and it offers good performance. On
April 14, 2019, researchers presented a potential
attack against qTesla that may affect some of
their parameter sets (qTesla’s authors have yet
to respond). From a performance perspective,
for level 1, qTesla offers signatures of size 1.3 KB
and public keys of size 1.5 KB, and for level 5 sig-
natures of size 5.9 KB and public keys of size 6.4
KB. Regarding speed, for level 1, signature gener-
ation takes 492,000 cycles and verification takes
82,000 cycles, while for level 5, signature genera-
tion takes 2.1 million cycles and verification
takes 394,000 cycles.
The lattice-based cryptography field is a pop-
ular PQC approach. One point of attention is the
secure selection of parameters that still seems
to be challenging depending on the underlying
security problem. From a performance perspec-
tive, all three lattice candidates offer reasonable
performance and should be considered promis-
ing candidates.
Multivariate Quadratic (MQ)-Based Schemes
In this category, we have GeMSS,4 Rainbow,10
LUOV,6 and MQDSS.7 Several MQ schemes have
been proposed in the past and subsequently
broken. Security has become more stable in
recent years, however MQ remains the PQC fam-
ily of digital signatures whose security is the
21
 Expert Opinion

Table 1. Comparison for security level 1 or the closest. Sizes in KB, speed in millions of cycles.

Symmetric Crypto Lattices Multivariate Quadratic

CRYSTALS-
XMSS SPHINCSþ PICNIC Falcon qTesla GeMSS Rainbow LUOV MQDSS
DILITHIUM
Signature
0.72 16.57 33.23 0.60 1.99 1.37 0.03 0.06 0.30 20.36
size

Public
0.03 0.03 0.03 0.87 1.15 1.50 417.40 149.00 12.10 0.04
key size

Signing
– 340.00 137.91 0.54 1.37 0.49 690.00 0.40 5.40 26.63
speed

Verification
0.39 14.98 90.63 0.08 0.27 0.08 29.10 0.15 4.30 19.84
speed

least understood. The main benefit of MQ sch-
emes is the compact signature sizes.
GeMSS is a scheme based on the hidden
field equations underlying problem and can be
seen as a variant of Quartz, a scheme proposed
in 2001, which remains one of the fewest
unbroken MQ schemes. GeMSS signatures are
very compact: only 258 bits for level 1, and 588
bits for level 5. Public key sizes are 417 KB and
3,046.84 KB, respectively. However, GeMSS is
not speed efficient: for level 1, signing takes
6,690 million cycles and verification takes 29
million cycles, while for level 5, signing takes
25,300 million cycles and verification takes 172
million cycles.
Rainbow is a signature scheme based on the
well-known Unbalanced-Oil-and-Vinegar (UOV)
signature scheme (which itself is based on the
Oil-and-Vinegar scheme). From a practical per-
spective, for level 1, the signature is 512 bits
long and the public key is 149.00 KB long, while
signing takes 402,000 cycles and verification
takes 155,000 cycles. For level 5, the signature is
159 KB and the public key is 1,227.10 KB long,
while signing takes 3.6 million cycles and verifi-
cation takes 2.3 million cycles.
Lifted-UOV (LUOV) is a scheme also based
on the UOV scheme. The main difference from
UOV consists of some optimizations to reduce
the public key size (e.g., lifting the UOV public
key to an extension field). For level 2, signature
and public key sizes are 311 bytes and 12.1 KB,
respectively. Signing takes 5.4 million cycles
and verification takes 4.3 million cycles. For
level 5, signature and public keys are 494 bytes
and 75.5 KB, respectively, while signing takes
24 million cycles and verification takes 18 mil-
lion cycles.
MQDSS is a scheme based on the combina-
tion of the Sakumoto–Shirai–Hiwatari (SSH) iden-
tification scheme with the Fiat-Shamir transform.
This is a very innovative proposal. For level 1,
its signature and public key sizes are 20 KB and
46 bytes long, respectively, while signing takes
26 million cycles and verification takes 19 million
cycles. For level 3, its signature and public key
sizes are 42 KB and 64 bytes long, respectively,
while signing takes 85 million cycles and verifica-
tion takes 62 million cycles.
MQ-based schemes offer interesting perfor-
mance benefits, such as tiny signatures from
GeMSS or tiny public keys from MQDSS. How-
ever, the field of multivariate-quadratic schemes
would still benefit from a more comprehensive
security analysis. The PQC standardization pro-
cess may help in this process by promoting
these schemes and thus attracting an increasing
number of researchers to expand the knowledge
in this field and increase the confidence of poten-
tial users.
Tables 1 and 2 show the performance of the
second round candidates of the NIST PQC com-
petition plus the XMSS scheme published in IETF
RFC8391. We acknowledge that these numbers
(most of them obtained from the submission
packages) were collected in different platforms,
and therefore the speed numbers should be
taken as a rough approximation of the actual
performance, useful when considered from the
orders-of-magnitude perspective.

22 IEEE Micro
Table 2. Comparison for security level 5 or the closest. Sizes in KB, speed in millions of cycles.

Symmetric Crypto Lattices Multivariate Quadratic

CRYSTALS-
XMSS SPHINCSþ PICNIC Falcon qTesla GeMSS Rainbow LUOV MQDSS
DILITHIUM
Signature
2.44 48.06 129.74 1.20 3.28 5.92 0.07 1.59 0.5 42.70
size

Public
0.06 0.06 0.06 1.75 1.71 6.43 3,046.84 1,227.10 75.50 0.06
key size

Signing
– 1,491.94 1,112.23, 1.07 1.82 2.15 25,300 3.64 24.00 85.26
speed

Verification
0.74 37.59 736.31 0.18 0.51 0.39 172.00 2.39 18.00 62.30
speed

Analysis of Available PQC Solutions performance perspective, lattice-based algo-

The most important criterion for the selec-
tion of PQC schemes should be security. The
PQC transition offers considerably greater chal-
lenges than previous (symmetric) cryptographic
transitions (for example, from 3DES to AES, or
SHA-1 to SHA-2). Public key cryptosystems are
considerably more complex than symmetric
ones, and we still do not fully understand the
capabilities of a typical, future quantum adver-
sary. In this context, conservativeness seems to
be the most reasonable stance.
Schemes based on symmetric crypto building
blocks (e.g., hash or block ciphers) seem to be
an extremely promising approach, as they do
not introduce any additional security assump-
tions. In cases where state management is possi-
ble, XMSS is a strong candidate as it offers
interesting performance and security proof in
the standard model. In case state management is
not possible, SPHINCSþ may be preferable since
it does not even need zero-knowledge proofs as
seen in PICNIC. Where state management is not
possible and the performance offered by state-
less symmetric-crypto-based algorithms is not
acceptable, the competing candidates from
other PQC families should be considered. In this
case, lattice-based algorithms seem to offer an
interesting balance between security and perfor-
mance. Some of the lattice-based underlying
problems have gone through intense academic
scrutiny for several years (in some cases, deca-
des). One remark, however, is that the commu-
nity may still benefit from a better understanding
on how to choose secure parameters. From a
rithms offer decent performance. For example,
CRYSTALS-DILITHIUM has both signature and
public key size in the low single-digit kilobytes,
and speed in the low single digit millions of cycles.
Finally, multivariate-quadratic algorithms may
have interesting performance advantages, such as
the tiny signatures of 258 bits offered by GeMSS,
but they are the candidates that would benefit the
most from additional security assessments, given
the recurrent attacks against MQ digital signature
schemes throughout the history.
In summary, if state management is possible,
IETF schemes (e.g., XMSS) seem to be a very
promising approach. If state management is
not possible, the candidates of the second round
of the NIST competition (following the prioritiz-
ation described above: 1—stateless symmetric-
crypto-based, 2—lattice-based, 3—MQ) seem to
be a promising approach.
COST OF CRYPTOGRAPHY VS.
ADOPTION
Generally, technology adoption increases as
performance increases and cost decreases. We
expect postquantum cryptography to follow
suit. NIST recently published “The Economic
Impacts of the Advanced Encryption Standard,
1996-2017” which estimated a $250 billion
impact. A key factor in the success of AES has
been the phenomenal performance achieved in
modern microprocessors. Galois Counter Mode
(GCM) is a popular AES mode of operation used
in the networking space that secures the major-
ity of internet traffic. Consider the historic cost

July/August 2019
23
 Expert Opinion
of the computation of GCM, and the adoption
rate in the figure below, gathered from the ICSI
Notary (https://notary.icsi.berkeley.edu/).
In the figure above, the “performance” of run-
ning the GCM algorithm—presented as “cost in
cycles per byte” where lower cost-per-byte
indicates higher performance—corresponds to
various Intel processors that launched on
those dates, with a trend of continuously improv-
ing GCM performance. Notably, in 2013 there was
a dramatic increase in GCM performance (indi-
cated by a significant drop in the cost of cycles/
byte), which we believe contributed to a steep
rise of GCM adoption across the industry.
INTEL’S DIRECTION ON PQC AND
FIRMWARE
Intel’s strategy is to continue securing
platforms using cryptographically strong digital
signature standards that execute efficiently.
Intel will continue to be aligned with standardiza-
tion organizations and will evaluate algorithms
based on security assurance, hardware cost,
and performance—including that in embedded
cores. Until full PQC transition has completed, a
hybrid solution where two or more algorithms are
executed in parallel—in order to remove depen-
dence on a single algorithm or class of algorithms,
seems an interesting approach. This should allow
24
for more flexibility and minimize redesign time of
embedded microcontroller systems should a sin-
gle (class of) algorithms become infeasible to
deploy for their firmware authentication.
CONCLUSION
Firmware authenticity is a key factor in the
proper function and security assurance of a plat-
form built with embedded microcontroller
cores. With the need for PQC-secure crypto-
graphic algorithms to continue to assure this
authenticity, Intel is investigating possibilities
currently under consideration in various stan-
dardization processes. Our priority in this selec-
tion is and will always be security. Regarding
performance, we are specifically focusing on the
capabilities of platforms’ embedded cores to
execute PQC algorithms and looking for algo-
rithms that have parameters allowing flexibility
in fitting their execution to the capabilities of
both current and planned embedded cores.
& REFERENCES
1. A. Huelsing, D. Butin, S. Gazdag, J. Rijneveld, and
A. Mohaisen, (2018). XMSS: eXtended Merkle Signature
Scheme - Request For Comment 8391 (RFC 8391).
Internet Engineering Task Force (IETF), Retrieved: 24
June, 2019, https://tools.ietf.org/html/rfc8391
IEEE Micro
 2. V. Lyubashevsky, L. Ducas, E. Kiltz, T. Lepoint,
P. Schwabe, G. Seiler, and D. Stehle, (2017).
CRYSTALS-DILITHIUM - A Submission to the NIST
Post-Quantum Cryptography Standardization Project.
National Institute of Standards and Technology (NIST).
Retrieved: 24 June, 2019, https://pq-crystals.org/
3. T. Prest, P.-A. Fouque, J. Hoffstein, P. Kirchner,
V. Lyubashevsky, T. Pornin, and Z. Zhang, (2017).
Falcon - A Submission to the NIST Post-Quantum
Cryptography Standardization Project. National
Institute of Standards and Technology (NIST).
Retrieved: 24 June, 2019, https://falcon-sign.info/
4. A. Casanova, J.-C. Faugere, G. Macario-Rat,
J. Patarin, L. Perret, and J. Ryckeghem, (2017).
GeMSS - A Submission to the NIST Post-Quantum
Cryptography Standardization Project. National
Institute of Standards and Technology (NIST).
Retrieved: 24 June, 2019, https://www-polsys.lip6.fr/
Links/NIST/GeMSS.html
5. L. K. Grover, “A fast quantum mechanical algorithm for
database search,” in Proc. 28th Annu. ACM Symp.
Theory Comput., 1996, pp. 212–219.
6. W. Beullens, B. Preneel, A. Szepieniec,
F. Vercauteren, (2017). LUOV - A Submission to the
NIST Post-Quantum Cryptography Standardization
Project. National Institute of Standards and
Technology (NIST). Retrieved: 24 June, 2019, https://
www.esat.kuleuven.be/cosic/pqcrypto/luov/
7. S. Samardjiska, M-S. Chen, A. Hulsing, J. Rijneveld,
and P. Schwabe, (2017). MQDSS - A Submission to
the NIST Post-Quantum Cryptography Standardization
Project. National Institute of Standards and
Technology (NIST). Retrieved: 24 June, 2019, http://
mqdss.org/
8. G. Zaverucha, M. Chase, D. Derler, S. Goldfeder,
C. Orlandi, S. Ramacher, and V. Kolesnikov, (2017).
Picnic - A Submission to the NIST Post-Quantum
Cryptography Standardization Project. National Institute
of Standards and Technology (NIST). Retrieved: 24
June, 2019, https://microsoft.github.io/Picnic/
9. N. Bindel, S. Akleylek, E. Alkim, P. S. Barreto,
J. Buchmann, E. Eaton, and G. Zanon, (2017). qTesla -
A Submission to the NIST Post-Quantum Cryptography
Standardization Project. National Institute of Standards
and Technology (NIST). Retrieved: 24 June, 2019,
https://qtesla.org/
10. J. Ding, M-S. Chen, A. Petzoldt, D. Schmidt, and B-Y.
Yang, (2017). Rainbow - A Submission to the NIST
Post-Quantum Cryptography Standardization Proje\ct.
National Institute of Standards and Technology (NIST).
July/August 2019
11. P. Shor, “Algorithms for quantum computation:
Discrete logarithms and factoring,” in Proc. 35th Annu.
Symp. Foundations Comput. Sci., 1994, pp. 124–134,
Santa Fe: IEEE Comput. Soc. Press.
12. D. Bernstein, C. Dobraunig, M. Eichlseder, S. Fluhrer,
€lsing, and F. Mendel, (2017).
S. Gazdag, A. Hu
SPHINCSþ - A Submission to NIST Post-Quantum
Cryptography Standardization Project. National
Institute of Standards and Technology (NIST).
Retrieved: 24 June, 2019, https://sphincs.org/
Rafael Misoczki is a cryptographer/research
scientist at Intel Labs. His work is focused on
post-quantum cryptography and its application to
secure update, root of trust, remote attestation,
and other security flows. He has a PhD from the
University of Paris (Pierre et Marie Curie), with a
thesis on efficient constructions for post-quantum
cryptography. He also holds an MSc in electrical
engineering and a BSc in computer science from
the University of Sao Paulo. Contact him at rafael.
misoczki@intel.com.
and
Sean Gulley is a principal engineer at Intel’s Data
Center Group responsible for anticipating and
accelerating new algorithmic intensive workloads.
Since joining Intel in 2001, he has focused primarily on
cryptography and compression HW and SW solutions
for client and data center. He has a BS in computer
engineering from Tufts University and an MS in electri-
cal engineering from Stanford University. He has over
30 U.S. patents. Contact him at sean.gulley@intel.com.
Vinodh Gopal is as senior principal engineer at Intel,
working in the Data Center Group. His work includes
accelerators, instruction-set extensions for x86 and
architectural enhancements to processors, in applica-
tions such as cryptography, integrity, compression,
and analytics over a range of products. In 2019, he
won the Intel Inventor of the Year Award. Contact him
at vinodh.gopal@intel.com.
Martin G. Dixon is an Intel Fellow in the Intel Prod-
uct Assurance and Security (IPAS) group and direc-
tor of architecture at Intel Corporation. He is
responsible for guiding future research and architec-
ture decisions to secure Intel’s platforms. He has
published a dozen academic papers in the field of
computer architecture and holds 50 patents in the
field of computer architecture and cryptography.
He has a bachelor’s degree in electrical and
25
 Expert Opinion
computer engineering from Carnegie Mellon Univer-
sity. Contact him at martin.dixon@intel.com.
Hrvoje Vrsalovic has been involved in firmware and
app-to-device interface software development in one
form or another ever since being part of the original
team that created Palm’s WebOS in 2008. Since then,
he has worked on software—and sometimes hard-
ware—of many “smart” consumer products, particu-
larly wearables. He recently joined Intel’s IPAS group
as a security architect. He has a BSc in computer sci-
ence from UCSB and an MSc in electrical and com-
puter engineering from Carnegie Mellon University.
Contact him at harvey.vrsalovic@intel.com.
26
Wajdi K. Feghali is an Intel Fellow and the director
of the Security and Algorithms Center of Innovation
in the Data Center Group at Intel Corporation. He
leads the development of cryptography, compr-
ession, data integrity and data de-duplication hard-
ware and software solutions with a focus on efficient
performance across Intel products. He has been
granted more than 50 U.S. patents, with numerous
other patents pending, and is the author of several
published technical papers. He has a bachelor’s
degree in mathematics with a minor in computer sci-
ence from the University of Ottawa. Contact him at
wajdi.feghali@intel.com.
IEEE Micro
 Expert Opinion

Energy-Secure System
Architectures (ESSA):
A Workshop Report
Pradip Bose Saibal Mukhopadhyay
IBM T. J. Watson Research Center Georgia Institute of Technology

& MODERN MICROPROCESSOR CHIPS have multiple temperature) constraints. Effective parallelization
processing engines (or cores) that are architected of application codes, supported by many-core/
to solve a variety of problems in individual and many-thread hardware engines, is the established
cooperative execution modes. In the current trend in current computing. Since 96-thread
regime of commercial designs, we already see dou- POWER8 server chips have already been in the
ble-digit core counts; and if one considers the market for a few years, it is not unrealistic to
degree of hardware multithreading supported in expect around 50 cores and perhaps
each core, the number of hard-
200 hardware threads supported in
ware threads that can be sup- Ever since on-chip and a couple of generations. Of course,
ported in concurrent execution system-level power due to area pressures, one can
add up to many dozens or scores. management architec- expect to see leaner (simpler) cores
For example, IBM’s prior genera- tures have become with only modest single-thread per-
tion POWER8 processor chip routine in the industry, formance growth.
already supported up to 96 hard- concerns about reliable This technology- and market-
ware threads via its 12 cores, operation and associ- driven trend toward throughput-
each of which can execute in up ated security vulner- oriented (scale-out) designs implies
to an eight-way simultaneously abilities have been a major challenge in terms of
multithreaded mode. As present in the minds of
chip-level power and/or thermal
explained in recent ISSCC tech- both the designer and
management—in a regime where
nology trend data, while the core researcher community.
balanced performance growth (sin-
count growth has been steady, gle-thread versus throughput) at
the clock frequency has saturated around the 4- affordable power becomes a steeper challenge
GHz mark—mainly limited by power density (or over time. And, at the full system (i.e., server,
rack, or data center) level, the challenge can be
even greater. At whatever scale one is interested
Digital Object Identifier 10.1109/MM.2019.2921508 in managing such metrics (i.e., power or tempera-
Date of current version 23 July 2019. ture, or even related ones, like system reliability),

July/August 2019 Published by the IEEE Computer Society 0272-1732 ß 2019 IEEE
27
 Expert Opinion
on-chip and system-level power management
control architectures will need to be carefully
architected. These must ensure the right trade-
offs to be applied at runtime to make sure that
workload-dependent performance is maximized
(at least to the extent that customer service-level
agreements are met) while adhering to system-
imposed power consumption limits.
Ever since on-chip and system-level power
management architectures have become routine
in the industry,1-3 concerns about reliable opera-
tion and associated security vulnerabilities have
been present in the minds of both the designer
and researcher community. What if the sense-
and-actuate feedback control system(s) imple-
mented in such a design had latent bugs, wherein
a corner case workload (launched maliciously or
inadvertently) could disrupt the intended func-
tionality and cause the system to fail? Could the
chip or system incur irreparable physical dam-
age? At a minimum, could a power virus attack
result in significant performance degradation for
other (regular) customer workloads—effectively
signaling a denial of service attack? In fact, even
before power management control systems were
in vogue, research papers4 had demonstrated
physical attacks, where thermally induced mem-
ory bit-flips could enable Trojan software to take
over the full system, evading immediate detec-
tion. Later on, the Charlie Miller hacks of Apple
laptop battery control loops created quite a stir5;
and data-center level power (energy) attack vec-
tors were demonstrated.6
In light of the above motivational background,
it is not surprising that researchers from IBM
Research (led by Pradip Bose) initiated IBM inter-
nal research on a variety of power attacks and
their mitigation around the year 2010. This
research was supported by a DARPA seedling
grant in 2011–2012, and this also helped that
research team launch the ESSA workshop series
in the beginning of 2011 [in conjunction with the
International Symposium on Computer Architec-
ture (ISCA)]. An early visionary paper on the ESSA
theme was presented by Bose in 2012.14 This work-
shop series was interrupted for a few years, before
being resurrected recently in conjunction with the
Hardware Security-Focused Conference (HOST
2019). The workshop was jointly organized by Dr.
Pradip Bose from IBM Research and Prof. Saibal
Mukhopadhyay from Georgia Institute of Technol-
ogy. The key new feature of ESSA 2019 was to bring
28
circuit-level power-management experts within
the ESSA community to explore a cross-layer
approach to energy-secure processor design. In
the next section, we provide a summary descrip-
tion of ESSA 2019, which was held on May 9–10 at
Tysons Corner, VA, USA.
SUMMARY PROCEEDINGS OF
ESSA-2019
The initial years of workshop offerings around
the ESSA theme resulted in a successful spawning
off of a few academic research projects—which
was the underlying objective. Perhaps the best-
known research that has been reported in the lit-
erature since that time is the CLKSCREW attack
modality12 published by Simha Sethumadhavan’s
group at Columbia University. This work not only
demonstrates the use of software code segments
to disrupt the power management controls in a
processor, it also shows how side-channel attacks
can be orchestrated around this basic attack para-
digm. In general, the scope of “energy attacks” has
expanded to side channel attacks that exploit
energy leaks of various types. As such, in ESSA
2019 (https://www.essa-workshop.org/), the tec-
hnical scope of the workshop was expanded to
broadly cover: “the range of research being pur-
sued within industry and academia in order to
ensure robust and secure functionality while
meeting the energy-related constraints of the
green computing era.” The technical program of
the workshop (https://www.essa-workshop.org/
#program) consisted of one keynote, three visi-
onary invited talks, three ESSA-relevant special
invited talks, four contributed regular papers, and
one panel session.
Keynote
The workshop began with an informative
keynote address by John Marsh, who repre-
sented Linton Salmon, Program Manager of
the ongoing DARPA program called System
Security Integrated Through Hardware and
Firmware (SSITH).
This was a valuable readout of the most prom-
ising research projects that are currently being
pursued in the mitigation of software-assisted
hardware attacks. The list of currently active
SSITH program performers and the key innova-
tions of their approach, as quoted from Marsh’s
talk, is shown in Table 1.
IEEE Micro
Table 1. Summary description of SSITH projects.

Prime Technical area Point of contact Technical approach

Every word has metadata þ every instruction is checked
Charles Arun Thomas
H/W Architecture based on flexible security micro-Policies defined in
Draper Lab arun@draper.com
software (DSL); compartmentalization;PPASS workbench
Combination of efficient tagging,fenced,regions,
Lockheed Jim Eiche Protection,domains,per-Thread keying,and memory
H/W Architecture
Martin james.eiche@lmco.com encryption;security hardware in parallel with CPU; No
source changes (Binary analysis)
Hardware security compiler with end-to-end formal
Adam Chlipala
MIT H/W Architecture verification; generic support for tagging policies;
adamc@csail.mit.edu
compartmentalized secure enclaves.
Tags on every word of data and every instruction
SRI Robert Watson implement bounds checking and permissions;
H/W Architecture
International Robert.watson@cl.cam.ac.uk encapsulation; formal methods to verify security;security
architecture extended to DMA engines
Anti-fragility approach learns from attacks;machine
UC San Dean M Tullsen
H/W Architecture learning based on Hardware Performance Registers;
Diego tullsen@cs.used.edu
efficient X86 implementation leveraging micro-ops.
High entropy (hard to hack) Plus rapid churning (no time
University of Todd Austin
H/W Architecture to exploit) to mediate “undefined semantics”; tagging;
Michigan austin@umich.edu
encryption;relocation of memory areas.
Automated system security metrics framework and tools;
Joe Kiniry
Galois Test and Metrics objective analysis; used formal methods; GFE IP
kiniry@galois.com
development and support: RICS-V baseline for evaluation.

Visionary Talks was described. The second part of the talk

The first visionary talk was delivered by Dr. showed examples from recent literature where
Vivek De from Intel Labs. In his visionary talk existing power management circuit techniques
titled: “Attack-Resistant Energy-Efficient SoC are leveraged and re-purposed to improve resis-
Design,” Dr. De provided an in-depth perspective tance to power and electromagnetic emission
on the fundamental issues and tradeoffs in the based side-channel attack. The talk showed
design of power-performance-area (PPA) effi-
cient SoCs that can also be
architected to be resilient to
malicious attacks. Figure 1
depicts the basic features of
resilient platforms, as pre-
sented in Dr. De’s talk. The soft-
ware, firmware, and hardware
layers of abstraction in the
design stack that need to be co-
designed to factor in targeted
resiliency features, while meet-
ing critical PPA metrics were
covered. In order to bake in the
“attack resistance” shield into
the general framework of resil-
ient platforms, the architecture
of the secure roots of trust
(including the use of secure and
variation-tolerant PUF/TRNG) Figure 1. Resilient platforms.

July/August 2019
29
 Expert Opinion
Figure 2. Abstract view of energy management.
that circuit techniques can strongly impact the
energy-security tradeoff in a SoC.
The second visionary talk was delivered by
Prof. Simha Sethumadhavan from Columbia
University. In his talk, titled: “Software vecto-
red fault attacks,” Sethumadhavan recalled
the CLKSCREW attack work7,8 that his group had
pioneered, and then painted a picture of the
work that is emerging beyond that ground-
breaking prior work. The CLKSCREW research
exposed the vulnerabilities in classical DVFS-
based power management in embedded systems
(see Figure 2), and has since resulted in solu-
tions to fix such gaps in security in a class of
popular commercial mobile platforms driven
by the Android OS. Sethumadhavan presented
a wish list of future energy-secure architectures,
with a focus on hardware security features that
would need to be augmented in order to mitigate
the threat of energy-sourced side channel intru-
sions and physical attacks.
Prof. Onur Mutlu’s (ETH Zurich) visionary
talk, titled “Using commodity memory devices
to support fundamental security primitives,” was
an in-depth journey into the fundamentals of
memory system architectures, and the associated
security-related vulnerabilities. Initially, Mutlu
spoke about the solution approaches to reduce
data movement related power consumption
through architectural innovations. Subsequently,
a significant part of the talk was on ways to use
memory devices to generate true random num-
bers with low latency and high throughput. Gener-
ation and evaluation of physically unclonable
30
functions (PUFs) was another
aspect that was covered. These
capabilities were pointed up to
be crucial elements in system-
level security mechanisms. Ano-
ther item covered was quick
destruction of in-memory data
(for DRAMs). The security-cen-
tric aspects of the presentation
focused mainly on the speaker’s
work published in HPCA 2018,15
HPCA 201916 and arxiv 2019.17
Special Invited Talks
Prof. Mingoo Seok (Colum-
bia University) presented a
paper (M. Seok, A. Tang, Z.
Jiang, S. Sethumadhavan, “Blacklist core:
machine-learning-based power management tam-
pering,”) where machine-learning-based dynamic
operating performance point blacklisting is used
for mitigating software based power-management
tampering. Using CLKSCREW attack as an exam-
ple, Seok argued that static guard-banding could
mitigate such attacks but it incurs performance
degradation, power efficiency loss, and long test-
ing time. Instead, the talk introduced a detection-
then-mitigation approach, which uses a neural-net
model and detects a malicious command to put
the system on an unsafe operating performance
point. If detected, it then mitigates the attack by
ignoring the command. The algorithm and hard-
ware realization of the technique shows the ability
to detect and mitigate CLKSCREW attempts at a
reasonably small amount of overhead in power,
delay, and area. The talk illustrated that co-design
of circuit and algorithm is necessary to optimally
tradeoff the energy and security behavior of an
SoC.
Prof. Swaroop Ghosh (Penn State) spoke on:
“Security of persistent memories.” Excellent prop-
erties, such as zero leakage, high-density, scalabil-
ity, and high endurance, of emerging non-volatile
memories (NVMs) make them an attractive candi-
date for energy management in SoCs. However,
Ghosh pointed out that although NVMs can reap
energy and performance benefits they may face
new security issues that were not perceived
before. His talk discussed several potential vulner-
abilities of NVMs, and how they can be exploited
to compromise data integrity (e.g., tampering and
IEEE Micro
row-hammering) and data privacy. The talk also
explored circuit and system level methods for
sensing and inhibiting attacks on NVMs. The talk
reminded the audience that new technologies
introduced for energy management can also lead
to new security challenges.
Prof. Vijay Janapa Reddi (Harvard) spoke on:
“Closing the performance, power and reliability
gap in autonomous aerial machines.” Reddi
has been working on the topic area of “aerial
computing,” and this particular presentation
addressed the fundamental issues of power-
performance efficiency and resilience in design-
ing the embedded processor engines that power
autonomous drones. The connection between
reliability and security, as also articulated in
Sethumadhavan’s visionary talk, was re-exam-
ined briefly in Reddi’s presentation.
Contributed Regular Papers
The regular technical presentations con-
sisted of the following contributed papers:
 K. Khatamifard, L. Wang, S. Kose, A. Das,
U. Karpuzcu, “A novel class of covert chan-
nels enabled by power budget sharing.”
 A. Krishnan, P. Schaumont, “Hardware sup-
port for secure intermittent architectures.”
 I. Tochukwu and A. Ismail, “Holistic hardware
security assessment framework: a microarch-
itectural framework.”

D. Trilla, C. Hernandez, J. Abella, F. Cazorla,
“Four birds with one stone: on the use of
time randomized processors and probabilis-
tic analysis to address timing, reliability,
energy and security in critical embedded
autonomous systems.”
It is well known that runtime power manage-
ment is in charge of the optimal distribution of the
power budget—a very critical shared resource—
among system components. The paper by Toch-
ukwu et al. argued that any system-wide shared
resource can give rise to covert communication, if
not properly managed, and power budget, unfortu-
nately, does not represent an exception. The
paper presented a proof-of-concept demonstra-
tion of covert communication exploiting shared
power budget and discussed the potential design
space for countermeasures. The paper argued
that a secure power management infrastructure
must be aware of the potential threats associated
with sharing power across multiple entities.
July/August 2019
The paper by Krishnan et al. stressed the need
for a secure power transition mechanism to con-
vert the active system state into a protected non-
volatile form and back in energy harvesting based
IoT edge platforms. The paper observed that
secure checkpointings are necessary, but are
expensive to compute and require hardware-
accelerated cryptography and isolated secure
non-volatile storage. The paper defined an energy-
harvester subsystem interface that drives the
optimized execution of a secure communication
protocol such that wasted energy is eliminated
and that run-time performance is improved.
The paper by Tochukwu et al. presented the
challenging but critical need to enable a holistic
hardware security evaluation from the microarch-
itectural point of view. The paper introduced an
important step toward this direction by proposing
a framework that categorizes threat models based
on the microarchitectural components being tar-
geted and provides a generic security metric that
can be used to assess the vulnerability of compo-
nents, as well as the system as a whole.
Finally, the paper by Trilla et al. discussed that
as complexity and time-criticality of operations
being performed in the autonomous vehicles con-
tinue to increase, this creates conflicting require-
ments in designing such processors. On one hand,
a simple and predictable design of the processors
facilitate verification of functional and nonfunc-
tional metrics; but on the other hand, using
high-performance and complex processor designs
with some degrees of obfuscation can deliver
high computing performance and security. The
paper argued that time-randomized processors
(TRP), an alternative to traditional (deterministic)
designs, can address these conflicting require-
ments. TRP facilitates timing analysis via the use
of statistical/probabilistic techniques, while also
show capabilities to effectively tackle the chal-
lenges of reliability, security, and energy consump-
tion. The paper reviewed the TRP opportunities
and show that they are a natural fit to fulfill the
requirements of autonomous critical systems. The
paper showed that disruptive ideas in the macro-
and microarchitecture may be necessary to design
future energy-secure autonomous systems.
IMPACT OF THE ESSA THEME
WORKSHOPS
In this section, we will briefly examine the
ongoing impact of the ESSA theme workshop
31
 Expert Opinion

access to OS and/or power man-

agement firmware code is, of
course, a different issue.) One of
the saving factors, as assessed,
was that the IBM POWER process-
ors up to the POWER7 generation
were not subject to performance
throttling under even the highest
power workloads conceivable. In
other words, for such high-end
server-class processors, the heat
sink and cooling solution were
over-designed to make sure that
the worst-case applications would
not result in exceeding the power
limit that could be handled by the
packaging-cum-cooling solution.
Note that power or electromigra-
tion (EM)-based side channel
attack vulnerabilities were not
Figure 3. Peak temperature driven throttle points and performance effects
within the scope of this study;
for Intel Pentium IV (P4) and Pentium M (PM) class processors. Experiments done
only physical damage and denial
using gcc workload within SPEC95 and different cooling solutions: 100 gallons
of service type attacks were under
per hour (gph) is the highest fluidic cooling rate used, resembling the commercial
consideration at that time.
P4 package; 60 gph is a reduced cooling rate to represent a cheaper packaging
One should note, however, that
solution. (Experimental data: courtesy Hendrik Hamann, IBM Research). well before engaging in this
POWER7-based study, the IBM
series in the context of identifying new security- researchers had studied the performance degra-
related vulnerabilities and devising mitigation dation characteristics of Intel’s Pentium IV series
solutions thereof. So far, in terms of core security processors. Figure 3 shows the experimental
domain impact, the CLKSCREW work from characterization data that compares the temper-
Columbia 7,8 is the leading example of new ature-driven performance differences between
innovation and impact related to the ESSA theme. Intel’s Pentium IV (abbreviated here as P4) and
This work has been acknowledged by the Pentium M (PM) class processors. In this labora-
Android Security Team: (https://source.android. tory experiment (which was conducted back
com/security/overview/acknowledgements): in 2005), the packaging lid of the processor
“Adrian Tang of Columbia University (CLKSCREW was taken out and replaced by a cooling solu-
paper), CVE 2017-8252.” Qualcomm also acknowl- tion provided by a controllable (special) heat-
edged and reported the fixing of bugs associated transparent fluid flow. The thermal imaging,
with the above-quoted CVE (common vulnerabil- measurement, and calibration were conducted
ities and exposures) item. using a special infra-red camera setup.9 As
Within IBM, an in-depth research study (led Figure 3 shows, the “normal” execution time of
by Pradip Bose) was conducted, under the spon- the full gcc workload on a P4 using a commercial
sorship of a small DARPA seedling grant (during grade cooling solution is 80 s. Whereas, if the
2011–2012), to assess the threat level imposed cooling mimics a low-cost packaging solution,
by maliciously launched power/thermal viruses. the execution time degrades to about 140 s due
For commercial IBM high-end processor systems to the throttling-based dynamic temperature
(e.g., POWER7 at that time), the study was not management built into these processors. In con-
able to demonstrate any performance or func- trast, the lower power PM processor runs much
tional degradation through user-level applica- cooler, without incurring any throttling and
tion software access alone. (Supervisory mode executes the same workload in about 90 s, even

32 IEEE Micro
with the low-cost packaging solution. The experi- article presented at ISSCC 2019 by Singh et al.
ments demonstrated the possibility that throt- showed that on-chip low-dropout regulators, cou-
tling-based performance degradation (at a very pled with adaptive clocking and fine-grain DVFS
significant level) could be instigated for a given provide security against power-/EM-based side-
processor-package system product by launching channel attacks against AES engines. There are
a high-power (possibly synthetic) virus workload. many more examples in recent literature where
The literature on side-channel attack mecha- circuit level studies are being performed, and
nisms that exploit the power and/or EM monitor- techniques are being developed to enable a bot-
ing has advanced quite a lot since the inception tom-up approach to energy-secure hardware
of the first ESSA workshop in 2011. This is evi- designs. The advancements at the circuit level
denced even from some of the technical and security research showed the need for engaging
visionary talks presented at circuit community within the ESSA
ESSA 2019. The threat imposed theme, and ESSA 2019 took a posi-
by sharing of a common power Future work must con- tive step toward this goal. The
budget in a multicore chip set- nect unreliable control success is evident from the talks
ting has been described in work loops explicitly to presented at ESSA 2019 by Dr. De,
by Sasaki et al.11 and the paper vulnerabilities from a Dr. Ghosh, and Dr. Seok, all of
by Khatamifard et al. presented mainstream security
which have pointed out the need
at ESSA 2019 shows the conse- research viewpoint. In
for circuit level research in this
quence of this threat model in other words, the
domain.
guarded power man-
explicit terms.
agement principle
While at the system scale,
must be tested as a
FUTURE DIRECTIONS
ESSA themed workshop has stud- In this section, we provide our
mitigation technique
ied the potential security threats view of the future directions of
against CLKSCREW-
introduced by the power manage- research and development within
inspired attacks.
ment solutions, there have been the ESSA theme. One of the early
significant progress in recent research agenda items at IBM that
years in understanding the energy-security trade- fell out of the ESSA-theme was that of guarded
off at the circuit level. In particular, recent res- power management.10 In this solution approach,
earch threads have emerged in designing energy- the baseline power management architecture is
efficient security engines, as well as exploring on- protected through a guard mechanism. The latter
chip power-management circuits for security. A is a higher level monitor-and-control system that
specific example of this new direction has been in observes the operation of the baseline architec-
the domain of designing low-overhead techniques ture through specialized activity (performance)
for improving power and EM-based side-channel- counters. Anomalies detected in the observed
attack resistance of encryption engines. In partic- counter-based signatures can serve to trigger mit-
ular, collaborative work between Georgia Tech igation actions. The latter could include adapting
and Intel labs has demonstrated a set of studies the hardware parameters of the baseline mecha-
where on-chip integrated voltage regulators nism on-the-fly. The above-referenced work was
(IVRs) and adaptive clocking circuits, introduced pursued with robust power management in mind.
mostly for power management, have been lever- Future work must connect unreliable control
aged to improve SCA resistance. A paper pre- loops explicitly to vulnerabilities from a main-
sented by Kar et al., at the 2017 International Solid stream security research viewpoint. In other
State Circuit Conference (ISSCC) demonstrated words, the guarded power management principle
the promise of the using inductive IVRs for inhibit- must be tested as a mitigation technique against
ing power attack on AES engines. A second article CLKSCREW-inspired attacks.
from the group, authored by Singh et al. and pub- The thrust of research in support of power
lished in the IEEE JOURNAL OF SOLID STATE CIRCUITS reduction in the wake of GPU-centric high-perfor-
(JSSC) in 2019, showed inductive IVR coupled with mance compute nodes has led to techniques like
fine-grain dynamic voltage scaling and adaptive adaptive voltage guard-band management (e.g.,
clocking can inhibit power and EM-based side J. Leng et al., MICRO 2015). In future accelerator-
channel attacks on AES engines. More recently, an rich systems, the task of balancing power,

July/August 2019
33
 Expert Opinion
performance, and reliability will have to be man-
aged using systematic hardware-software manage-
ment systems. Purely software-based scheduling
heuristics will need to get supported by hardware-
based monitors. As we progress toward many-
core processor chips, old-style on-chip power
control architectures (with a single, centralized
management unit) will give way to scalable, dis-
tributed control, and management systems. An ini-
tial vision on so-called swarm power management
architectures has been portrayed in recent invited
papers (e.g., the one at DATE 201811).
ACKNOWLEDGMENTS
The ESSA 2019 workshop,13 the proceed-
ings (and anticipated impact) of which were
summarized in this article, would not have been
possible without the help of an active and
supportive program committee. The valuable
contributions of our web and publicity chair,
David Trilla, are to be noted in particular. We
are grateful to the organizing committee of the
HOST 2019 symposium for their support.
& REFERENCES
1. S. Gunther and R. Singhal, “Next generation intel
microarchitecture (nehalem) family: Architectural
insights and power management,” presented at Intel
Developer Forum, San Francisco, CA, USA, Mar. 2008.
2. M. Floyd et al., “Introducing the energy management
features of the POWER7 chip,” IEEE Micro, vol. 31,
no. 2, pp. 60–75, Mar./Apr. 2011.
3. T. Webel et al., “Robust power management in the IBM
z13,” IBM J. R&D, vol. 59, no. 4/5, pp. 16-1–16-12,
Jul./Sep. 2015.
4. S. Govindavajhala and A. W. Appel, “Using memory
errors to attack a virtual machine,” in Proc. IEEE Symp.
Secur. Privacy, 2003, pp. 154–165.
5. C. Miller, “Battery firmware hacking,” presented at
BlackHat, August 2011. [Online]. Available: https://www.
blackhat.com/html/bh-us-11/bh-us-11-briefings.html#
Miller
6. Z. Wu, M. Xie, and H. Wang, “Energy attack on server
systems,” in Proc. 5th USENIX Workshop Offensive
Technol., 2011, p. 8.
34
7. A. Tang, S. Sethumadhavan, and S. Stolfo,
“CLKSCREW: Exposing the perils of security oblivious
energy management,” in Proc. USENIX Secur. Symp.,
2017, pp. 1057–1074.
8. A. Tang, S. Sethumadhavan, and S. Stolfo, “Motivating
security-aware energy management,” IEEE Micro,
vol. 38, no. 3, pp. 98–106, May/Jun. 2018.
9. H. Hamann, A. Weger, J. Lacey, Z. Hu, and P. Bose,
“Hotspot-limited microprocessors: direct temperature
and power distribution measurements,” IEEE J. Solid
State Circ., vol. 42, no. 1, pp. 56–65, Jan. 2007.
10. N. Madan, A. Buyuktosunoglu, P. Bose, and
M. Annavaram, “A case for guarded power gating for
multi-core processors,” in Proc. 17th Int. Symp. High
Performance Comput. Arch., Feb. 2011, pp. 291–300.
11. H. Sasaki, A. Buyktosunoglu, A. Vega, and P. Bose,
“Mitigating power contention: A scheduling based
approach,” Comput. Arch. Lett., vol. 16, no. 1,
pp. 60–63, 2017.
12. A. Vega, A. Buyuktosunoglu, and P. Bose, “Energy-
secure swarm power management,” in Proc. Design
Test Eur., 2018, pp. 1652–1657.
13. ESSA-2019 Workshop. [Online]. Available: https://
www.essa-workshop.org/
14. P. Bose et al., “Power management of multi-core
chips: Challenges and pitfalls,” in Proc. Design Test
Eur., 2012, pp. 977–982.
15. J. Kim et al., “DRaNGe: Using commodity DRAM
devices to generate true random numbers with low
latency and high throughput,” in Int’l. Symp. High
Perform. Comput. Arch. (HPCA), Feb. 2018.
16. J. Kim et al., “The DRAM latency PUF: Quickly
evaluating physical unclonable functions by exploiting
the latency-reliability tradeoff in modern commodity
DRAM devices,” in Int’l. Symp. High Perform. Comput.
Arch. (HPCA), Feb. 2018.
17. L. Orosa et al., “Dataplant: in-DRAM security
mechanisms for low-cost devices,” arxiv, 2019,
[Online]. Available: https://arxiv.org/abs/1902.07344
Pradip Bose is with IBM T. J. Watson Research
Center, New York. Contact him at: pbose@us.ibm.
com.
Saibal Mukhopadhyay is with Georgia Institute of
Technology. Contact him at: saibal@ece.gatech.edu.
IEEE Micro
 Expert Opinion
FinalFilter: Asserting
Security Properties of a
Processor at Runtime
Cynthia Sturton
University of North Carolina at Chapel Hill
Matthew Hicks
Virginia Tech
& IN AN IDEAL world, it would be possible to
build a provably correct and secure processor.
However, the complexity of today’s processors
puts this ideal out of reach. The complete verifi-
cation of a modern processor remains intracta-
ble. Statically verifying even a simple security
property—for example, “hardware privilege esca-
lation never occurs”—remains beyond the state
of the art in formal verification.
Testing can complement formal verification
methods, yet testing is incomplete and bugs in
the hardware that leave it vulnerable continue
to elude test suites. Further, a crafty malicious
actor can evade typical testing coverage metrics.
Recent efforts, including that of three of the
authors, have explored the use of static analysis
on the design files (e.g., hardware description
level source code or gate-level netlists) to find
suspicious circuitry.1–3 These techniques rely on
heuristics to define patterns that indicate a likely
Digital Object Identifier 10.1109/MM.2019.2921509
Date of current version 23 July 2019.
July/August 2019 Published by the IEEE Computer Society
Samuel T. King
University of California, Davis
Jonathan M. Smith
University of Pennsylvania and DARPA
trojan and then search for instances in the
design that match the pattern. However, mali-
cious circuitry that does not match the pattern
will be missed, as will inadvertent bugs that
open vulnerabilities. By the time the weakness is
uncovered, the hardware is already in the end
user’s hands and vulnerable to attack.
In the absence of a full proof of correctness,
what is needed is a final filter: a runtime verifica-
tion technique that works—postdeployment—to
detect and respond to security property viola-
tions as they occur during execution. In this arti-
cle, we make the case for final filters using our
tool, FinalFilter, as a case study.
FINALFILTER
Prior research, including our own, has
shown that assertions hard-coded into the
design can be a cheap and effective way to
verify the correctness of any single execution
run.4; 5 Assertions can cover properties that
would be intractable to prove statically for
the current state of the art. The downside
0272-1732 ß 2019 IEEE
35
 Expert Opinion

security property will be

detected at the point of violation.
This is independent of how the
violation occurs or what the root
cause is.

THREAT MODEL
The trusted computing base
for FinalFilter includes our speci-
fication and verification process
and tools, the fabrication pro-
cess and tools, and the filter’s
current configuration.
Figure 1. Processor design flow with FinalFilter: (a) Hardware description language
implementation of the instruction set specification. (b) Vulnerability is accidentally orLifecycle Assumptions
maliciously opened in the processor. (c) FinalFilter is added to the design as the last Referring to Figure 1, we
action,6 with taps directly on the outputs of ISA state storing elements. (d) FinalFilterassume we are the last ones to
dynamically verifies the properties encoded by trusted software. FinalFilter triggers touch the processor design. We
existing repair/recovery approaches in the event of an invariant violation. FinalFilter rely on orthogonal techniques
continues to protect the repair/recovery software. to ensure that FinalFilter is not
tampered with in the supply
chain, which includes fabrica-
is that, like all execution monitors, this tion of the processor and shipping to the end
approach cannot prove that the property can user.
never be violated, only that if such a violation
occurs the monitor will catch it. As such, a Architectural Scope
final filter is a verification approach that is FinalFilter protects privileged instruction set
complementary to and should be used in architecture (ISA)-level registers. FinalFilter does
conjunction with existing testing and static not detect side-channel attacks as doing so
verification methods. requires knowledge of more than the current trace
We extend the basic idea of an assertion- of execution. The focus of this paper is the integer
based execution monitor to make it configurable core of the processor. Notably, we assume the
so that the set of properties being monitored memory hierarchy is correct.
can be updated postdeployment to reflect new
Attacker Model
information about exploitable vulnerabilities in
The attacker is free to take any action not
the design. FinalFilter is a reconfigurable, run-
precluded by our assumptions, either in hard-
time verification system that monitors the state
ware or in software. This includes an attacker
and events of the processor for invalid updates
capable of creating and exploiting a hardware
to privileged registers.
defect. An example might be a defect that causes
The mechanism of a final filter is simple
the processor to return from an exception with-
and presents a small attack surface. Yet, mak-
out restoring the privilege level.
ing it configurable does add complexity. To
minimize FinalFilter’s cost to the system’s
trustworthiness, we formally verify the cor- DESIGN
rectness properties of its component modules FinalFilter enforces properties over privileged
and of the composed system. Finally, we show ISA state and events necessary for the security of
how to verify key properties for individual software running on the processor. An example
configurations. property that we will return to is, “the processor
As a formally verified execution monitor, Final- transitions from user mode to supervisor mode if,
Filter guarantees that any trace violating a given and only if, there is an interrupt or exception.” Any

36 IEEE Micro
processor that correctly implements the specifica-
tion must satisfy this property. Proving this prop-
erty statically requires a proof across all possible
execution traces—currently an intractable task.
Yet, as an execution monitor, FinalFilter can verify
the property for every trace that is executed. Moni-
toring is done by a set of hardware-based asser-
tions over architecturally visible states and events.
FinalFilter is designed to be used in conjunc-
tion with existing software-level recovery and
repair tools. For example, BlueChip,1 a tool devel-
oped by three of the authors, can route execution
around vulnerable circuitry. FinalFilter provides
precise introspection points and can support a
variety of repair and recovery approaches.
Three aspects of the design are worth noting.
1) FinalFilter is reconfigurable after deployment
and can protect multiple security-critical
properties concurrently.
2) FinalFilter’s design is formally specified and
its implementation proven correct.
3) Execution overhead is incurred only in the
rare case that a processor violates one of the
monitored security properties.
The key insight that allowed us to make the
monitor both reconfigurable and able to handle
multiple invariants concurrently is that many
security properties can be implemented as a
Boolean combination of more simple assertions,
and these simple component assertions are usu-
ally in one of only a few forms. Users can specify
a number of simple component assertions and
combine them into one or more complex asser-
tions that monitor hardware state.
Running Example
We use security invariants (or just invariants) to
describe properties of the ISA that must be true of
a secure implementation—that if violated would
open an exploitable vulnerability. Invariants are
dynamically verified by one or more assertions
over architecturally visible state.
Consider the following component of the
privilege escalation property mentioned before:
: assertion blocks built into the monitor.
I0 ¼ A change in processor mode from low privilege
to high privilege is caused only by an exception or
a reset:
July/August 2019
Invariant I0 is a statement that the instruc-
tion set specification says must be true of the
system at all points of execution. It can be writ-
ten as a concrete assertion in terms of the ISA-
level state in the following way:
:
A0 ¼ assertðrisingEdgeðSR ½SMÞ ! ðNPC½31 : 12 ¼ 0Þ ^
risingEdgeðSR ½SMÞ ! ðNPC½7 : 0 ¼ 0Þ _
risingEdgeðSR ½SMÞ ! ðreset ¼ 1ÞÞ
where SR ½SMrepresents the supervisor mode bit
of the processor’s status register, and an exception
is indicated by the next program counter
NPCpointing to an exception vector start address.
The address will always be of the form
0x00000X00, where the “X” indicates a don’t-
care value. (This might seem as if it leaves the door
open for a processor attack that escalates privilege
while executing at an address that matches the
form 0x00000X00, but it does not. Pages in that
address range have supervisor permissions set
which implies that code executing in that address
range is already in supervisor mode. If the proces-
sor attack attempts to allow user mode execution
of supervisor mode pages, FinalFilter includes an
invariant to detect such misbehavior.)
We break A0 into three component assertions.
:
Aa ¼ assertðrisingEdgeðSR ½SMÞ ! ðNPC½31 : 12 ¼ 0ÞÞ
:
Ab ¼ assertðrisingEdgeðSR ½SMÞ ! ðNPC½7 : 0 ¼ 0ÞÞ
:
Ac ¼ assertðrisingEdgeðSR ½SMÞ ! ðreset ¼ 1ÞÞ:
Each of these individual assertions is evalu-
ated at each step of execution, and the results
are appropriately combined to form a statement
that is equivalent to A0 .
Invariant Monitor
FinalFilter reads in ISA-level state and outputs
a signal indicating whether any of the pro-
grammed invariants were violated. It works
essentially as a programmable finite state
machine. Configuration data programs the
machine with which invariants to check and ISA-
level state acts as the input to the machine. The
number of invariants it can monitor concur-
rently depends on the complexity of the associ-
ated component assertions and the number of
Using our running example, we now describe
each module in the configurable monitor, shown
in its configured state in Figure 2. In our system, we
37
 Expert Opinion
Figure 2. FinalFilter configured with assertion A0 . Starting from
the top of the figure, the components are: ISA-level state, Routing
block, Logic blocks, Assert blocks, and Merge block. The Routing
block sends ISA-level state elements to the Logic blocks; the
Logic blocks condense multibit state and constant inputs down to
a single bit output that is sent to the Assert block; the Assert block
compares the previous value of its inputs to the current value,
outputing the result as a one bit value to the Merge block; the
Merge block combines the Assertion block results to form a higher
level result that indicates if the programmed invariants still hold;
this result is tied to the processor’s exception generation logic.
refer to Aa , Ab , and Ac as component assertions,
and A0 as simply an assertion. The difference being
that Anumber is the implementation of an invariant, a
combination of component assertions, whereas
Aletter represents a component assertion corre-
sponding to one assertion block in the configura-
ble monitor.
Routing. The Routing block is responsible for
feeding the desired ISA-level state to the Logic
blocks. The configuration data determines which
state element gets routed to which Logic block.
To accommodate arbitrary outputs, each Rout-
ing block output is 32 bits wide, with zero pad-
ding as required. In our running example, SR ½SM
is output to Logic blocks 0, 2, and 4, NPC is output
to Logic blocks 1 and 3, and reset is output to
Logic block 5, as shown in Figure 2.
Logic. Each Logic block implements a compari-
son operator. Given two inputs A and B, the con-
figuration data can select one comparison
operator from the set f¼; 6¼; ; < ; ; > g. Addi-
tionally, the configuration data can choose to
mask off some portion of A or B, or both, or it
38
can substitute a constant value for the value in
B. Returning to our running example, Logic
block 1 will evaluate NPC&0xfffff000 ¼ 0 and
output the result. Logic block 3 will evaluate
NPC&0x000000ff ¼ 0 and output the result and
Logic block 5 will evaluate reset ¼ 1 and output
the result. Logic blocks 0, 2, and 4 will evaluate
SR ½SM ¼ 1 and output the result.
Assert. The Assert block implements compo-
nent assertions of the form p ! q, possibly across
several clock cycles (e.g., if p is true then three
cycles later, q is true). If it is ever the case that p is
true while q is false, the assertion is triggered and
the output of the Assert block will be high. In our
example, each of Aa , Ab , and Ac are implemented
in their own Assert block. The consequent q is
always a combinational proposition over ISA state
at a single step of execution: it is stateless and is
given by the current value sent by the Logic block.
However, the antecedent p can be stateful, possi-
bly depending on previous values sent from the
Logic block. For example, the individual asser-
tions in our example all have the antecedent
risingEdge(SR ½SM). This proposition is true at
time t if and only if SR½SMis low at time t  1 and
high at time t. The Logic block will output a signal
that is high whenever SR½SMis high and the Assert
block will determine when a rising edge of SR½SMis
seen. FinalFilter allows antecedents in one of three
forms: p 2 fTrue; :st1 ^ st ; stn g. In other words,
p can be defined as True, in which case the asser-
tion will trigger whenever q is false, or p can be
defined to be the rising edge of some ISA state s,
or p can be defined to be the value of ISA state s at
time t  n, where n is also configurable.
The Assert block uses four of the industry
standard Open Verification Library assertions:
 always(expression): expression must always
be true,
 edge(type, trigger, expression): expression
must be true when the trigger goes from 0 to
1 (type = positive),
 next(trigger, expression, cycles): expression
must be true cycles clock ticks after trigger
goes from 0 to 1,
 delta(signal, min, max): when signal changes
value, the difference must be between min
and max, inclusive.
IEEE Micro
Merge. The Merge block takes the outputs from
the Assert blocks and combines them as pre-
scribed by the configuration data. It can be
viewed as a configurable truth table. The inputs
to the truth table are the Assert block outputs—
the component assertions Aa , Ab , and Ac in our
running example. The function defining how the
component assertions combine (i.e., the out
function) is configurable at run time. The truth
table is implemented as a hierarchy of look-up
tables. For example, with 16 Assert blocks,
rather than a single lookup table with 216 rows,
the monitor would have four lookup tables with
six inputs (26 rows) each. The outputs of the
three first-level lookup tables make up the input
to a second-level lookup table, the output of
which is the output of the Merge block.
We can now complete our running example.
Let erra be the output of the Assert block for Aa ,
and let errb and errc be the output of the Assert
blocks for Ab and Ac , respectively. Remembering
that the output of each Assert block will be high
when the assert triggers, i.e., when the invariant
is violated, we combine the results of the compo-
nent assertions in the following way:
err0 ¼ ðerra jerrb Þ&errc :
As desired, err0 will be high whenever A0 is false,
i.e., whenever the A0 assertion is triggered.
Configuration Data. The configuration data
are provided by trusted software (e.g., the sys-
tem BIOS) at initialization (originally, we imagine
configuration coming from processor or mother-
board manufacturers). It is the mechanism by
which FinalFilter is configured, and portions of
the configuration data are fed into each block at
the appropriate stage.
VERIFICATION
We used the commercial model checking tool
Cadence SMV for the verification of the configura-
ble assertion fabric. For each component of Final-
Filter shown in Figure 2, we formally specified its
behavior and verified that the implementation
meets the specification.
In most cases, formally specifying a com-
ponent’s behavior involved little more than
extracting the information from the design
July/August 2019
documents. However, in two cases, the process of
formalizing the specification brought out ambigui-
ties in the design, and it was necessary to revisit
the design phase of the process. During the course
of verification, we found one implementation
error: a logical AND was used where an OR was
needed.
Ultimately, the monitor’s behavior is deter-
mined by the configuration data, and it is up to
the processor or motherboard manufacturer to
provide a correct configuration. A misconfigured
fabric could fail to provide the intended protec-
tions. We guard against misconfigurations in
three ways.
First, we protect against invalid configurations
that would result in unpredictable results. Built in
to the design of each block is a check that the
incoming configuration data are well formed. We
verify that if any of the individual components
report an invalid configuration, then FinalFilter
will not fire any assertion failures. This behavior
represents a tradeoff in the design space. On the
one hand, an accidentally misconfigured fabric,
which will never trigger an assertion, is not pro-
tecting the user. On the other hand, never firing in
the presence of misconfigured data has the benefit
of being a stable behavior— it is what exists today.
An alternative is to always fire when the fabric is
misconfigured, but this would give an attacker an
avenue for launching a denial-of-service attack
making FinalFilter a new avenue of attack, some-
thing we wish to avoid.
Second, we built a software tool to generate
the configuration data from higher level asser-
tion statements. Although only prototypical, we
hope that further developing this tool will make
generating correct configuration data relatively
easy for the user.
Third, we built a validation tool to prove
properties about individual configurations. We
prove the following sanity checks on the configu-
ration data:
 There are assertions configured.
 None of the assertions are unsatisfiable
(e.g., the following does not occur fTrue !
q ^ :qg).
 The configured assertions, as a whole, are sat-
isfiable (e.g., the following does not occur
fp ! q; p ! :qg).
39
 Expert Opinion
Figure 3. Hardware overhead with respect to the number of
assertions supported by the configurable assertion fabric,
evaluated at four optimization levels. The range in the number of
assertions represents the range in protection required by the
processors in our analyzed set from AMD. The vertical line
represents the average number of assertions required to protect
the processors in our analyzed set. As a reference point, previous
work on deployed-bug patching entails hardware overheads of up
to 200% and run time overheads of up to 100% in the common case.
 Assertions are not trivially violated (e.g., the
following does not occur fp ! :pg).
If any of these checks fail, a misconfiguration
error is reported along with information about
the offending assertion(s). The user can run this
tool before loading the configuration data into
FinalFilter. We used the z3 SMT solver as the
back end to this tool.
We note that while we formally verify the func-
tional correctness of each module in the filter, we
manually audit the connection between modules.
That is, we manually check that every module’s
output signals are appropriately tied to the next
module’s input signals. There is no logic involved
in the composition and our naming convention
made the checks straightforward. Our end-to-end
verification of the invalid configuration signals,
mentioned above, does not rely on this manual
audit.
EVALUATION
To evaluate the performance and efficacy of
FinalFilter, we implement it inside the OR1200
Processor. The OR1200 is an open source, 32-bit
RISC processor with a five-stage pipeline, sepa-
rate data and instruction caches, and MMU
40
support for virtual memory. It is popular as a
research prototype and has been used in indus-
try as well7; it is representative of what you
would see in a mid-range phone today.
We wrote a program that automatically gen-
erates the FinalFilter hardware for a given num-
ber of Assert blocks to support. Generating the
hardware programmatically makes it easy to
explore the effect of tuning different parameters,
and creates a regular naming and connection
pattern that allows us to verify the structural
connections of arbitrary filters using an induc-
tion type approach.
For a complete system capable of booting
Linux, we implemented the processor and filter
combination as the heart of a system-on-chip
that includes DD2 memory, an Ethernet control-
ler, and a UART controller. We implemented the
system-on-chip on the FPGA that comes with the
Xilinx XUP-V5 development board. We conserva-
tively clock the system at 50 MHz.
Hardware Area Overhead
Figure 3 shows how the hardware area over-
head changes as the number of assertions sup-
ported by FinalFilter increases. We built filters
with support for as little as 1 assertion to as
many as 17 assertions (the number required
to protect all AMD processors we analyzed in
our previous study on security-critical processor
bugs5).
The figure contains data at four points in the
fabric design space:
1) None. No optimization, this favors expressi-
bility over overhead.
2) One State. This optimization uses Logic
blocks with only one state input. Logic
blocks were the biggest contributor to the
area of the fabric and 83% of our security-
critical invariants used only one input to the
Logic block. This also reduces the number of
required Routing blocks by 50%.
3) Top six. This optimization replaces the Rout-
ing blocks with new Routing blocks capable
of handling the six most frequently used state
elements. We observe that 76% of invariants
require the same six ISA-level state items.
4) Both. This includes the two previous
optimizations.
IEEE Micro
USING FINALFILTER Design-time verification routines to avoid triggering a bug
Using FinalFilter requires hav- alone is insufficient; that is found postdeployment.
ing a meaningful set of properties some exploitable vul- In this article, we have not
to monitor. In prior work, we took nerabilities will make it addressed the problem of mea-
a manual approach to develop a through. FinalFilter, a suring coverage. Boule  et al. 12
set of security critical properties.5 last line of defense— add circuitry to assertions to
We studied errata documents to one that can be for- track and measure coverage. The
learn what types of exploitable mally verified—pro- question of what is a meaningful
errors can occur and we studied tects security critical
coverage metric for a set of secu-
the architecture’s specification properties of the pro-
rity properties is an open one,
documents to develop a set of cessor core.
but it is critical: such a measure
properties necessary—though can give an indication of the
not sufficient—to protect security critical state number of “unknown unknowns” that remain
of the processor. unprotected.
In subsequent work, one of the authors has
developed a semiautomated method for learning CONCLUSION
new security properties using information Design-time verification alone is insufficient;
gleaned from known exploitable bugs8; and some exploitable vulnerabilities will make it
demonstrated that properties developed for one through. FinalFilter, a last line of defense—one
RISC processor may be suitable for use, after that can be formally verified—protects security
some translation, on a second RISC processor, critical properties of the processor core. We
even across architectures.9 However, the devel- believe the idea is broadly applicable and in
opment of security-critical properties for use future work will be exploring the use of a final fil-
with FinalFilter or any property-based verifica- ter for commercial architectures and for mod-
tion method is still in its infancy and more ules outside the processor core.
research is needed.

Case Study
We configured FinalFilter with 18 assertions we
found to be critical to security in our prior work.5
We then introduced into the processor 14 vulner-
abilities from a mix of previously published ha-
rdware attacks and attacks based on exploitable
vulnerabilities from several years of AMD proces-
sor errata. For each one, we wrote a user-space
ACKNOWLEDGMENT
The authors would like to thank the editors
for their insightful comments and suggestions,
and S. Bellovin for his advice and the phrase
“final filter.”
& REFERENCES
1. M. Hicks, M. Finnicum, S. T. King, M. M. K. Martin, and

program that exploits the vulnerability and reports J. M. Smith, “Overcoming an untrusted computing

if the attack was successful. FinalFilter is expres- base: Detecting and removing malicious hardware

sive enough to implement all 18 invariants, and the automatically,” in Proc. IEEE Secur. Privacy, 2010,

configured filter detects all of the attacks. pp. 159–172.

PRIOR WORK IN DYNAMIC
VERIFICATION
FinalFilter builds on a line of research that uses
dynamic verification to catch and patch func-
tional bugs postdeployment. For example, DIVA10
is a simplified checker core that verifies the com-
putation results of the full-featured core before
the processor commits the results to the ISA level.
Narayanasamy et al.11 use instruction rewriting
2. J. Zhang, F. Yuan, L. Wei, Z. Sun, and Q. Xu,
“VeriTrust: Verification for hardware trust,” in Proc.
ACM Des. Autom. Conf., 2013, pp. 61:1–61:8.
3. A. Waksman, M. Suozzo, and S. Sethumadhavan,
“FANCI: Identification of stealthy malicious logic using
boolean functional analysis,” in Proc. ACM Conf.
Comput. Commun. Secur., 2013, pp. 697–708.
4. M. Bilzor, C. Irvine, T. Huffmire, and T. Levin, “Security
checkers: Detecting processor malicious inclusions at
runtime,” in Proc. IEEE Hardware Oriented Secur.
Trust, 2011, pp. 34–39.

July/August 2019
41
 Expert Opinion
5. M. Hicks, C. Sturton, S. T. King, and J. M. Smith,
“SPECS: A lightweight runtime mechanism for
protecting software from security-critical
processor bugs,” in Proc. ACM Conf. Architectural
Support Program. Lang. Oper. Syst., 2015,
pp. 517–529.
6. A. Waksman and S. Sethumadhavan, “Silencing
hardware backdoors,” in Proc. IEEE Symp. Secur.
Privacy, 2011, pp. 49–63.
7. R. Rubenstein, “Open Source MCU core steps in to
power third generation chip,” Jan. 2014. [Online].
Available: http://www.newelectronics.co.uk/
electronics-technology/open-source-mcu- core-steps-
in-to-power-third-generation-chip/59110/
8. R. Zhang, N. Stanley, C. Griggs, A. Chi, and C. Sturton,
“Identifying security critical properties for the dynamic
verification of a processor,” in Proc. ACM Conf.
Architectural Support Programming Lang. Operating
Syst., 2017, pp. 541–554.
9. R. Zhang, C. Deutschbein, P. Huang, and C. Sturton,
“End-to-end automated exploit generation for
diagnosing processor designs,” in Proc. IEEE/ACM
Symp. Microarchit., 2018, pp. 815–827.
10. T. M. Austin, “DIVA: a reliable substrate for deep
submicron microarchitecture design,” in Proc. ACM/
IEEE MICRO, Haifa, Israel, Nov. 1999, pp. 196–207.
[Online]. Available: http://www.eecs.umich.edu/
taustin/papers/MICRO32-diva.pdf
11. S. Narayanasamy, B. Carneal, and B. Calder, “Patching
processor design errors,” in Proc. IEEE Int. Conf. Comput.
Des., Oct. 2006, pp. 491–498. [Online]. Available: http://
cseweb.ucsd.edu/ calder/papers/ICCD-06-HWPatch.pdf
12. M. Boule, J. Chenard, and Z. Zilic, “Adding debug
enhancements to assertion checkers for hardware
emulation and silicon debug,” in Proc. Int. Conf. Comput.
Des., 2006, pp. 294–299.
Cynthia Sturton is an assistant professor and
Peter Thacher Grauer Fellow at the University of North
Carolina at Chapel Hill. She leads the Hardware Secu-
rity @ UNC research group to investigate the use of
static and dynamic analysis techniques to protect
42
against vulnerable hardware designs. Her research is
funded by several National Science Foundation
awards, the Semiconductor Research Corporation,
Intel, a Junior Faculty Development Award from the
University of North Carolina, and a Google Faculty
Research Award. She was recently awarded the
Computer Science Departmental Teaching Award
at the University of North Carolina. She has a
BSE from Arizona State University and an MS and
a PhD from the University of California, Berkeley.
Contact her at csturton@cs.unc.edu.
Matthew Hicks is an assistant professor at
Virginia Tech, working at the intersection of security,
architecture, and embedded systems, with special
emphasis on analog-domain hardware security.
Contact him at mdhicks2@VT.edu.
Samuel T. King was a professor for eight years at
the University Illinois Urbana-Champaign. He then
left his tenured position at UIUC to push himself intel-
lectually and professionally in industry. He is cur-
rently with the Computer Science Department at the
University of California Davis. He is interested in
building systems for fighting fraud and rethinking our
notion of digital identity. He has a PhD from the
University of Michigan, an MS from Stanford Univer-
sity, and a BS from UCLA. Contact him at kingst@uc-
davis.edu.
Jonathan M. Smith is currently a program manager
in the Information Innovation Office (I2O) at the Defense
Advanced Projects Research Agency (DARPA) on
leave from the University of Pennsylvania, where he
holds the Olga and Alberico Pompa Professorship of
Engineering and Applied Science and is a professor of
computer and information science. He was previously
a Member of Technical Staff at Bell Telephone Labora-
tories and Bell Communications Research, joining
Penn in 1989 after receiving his PhD from Columbia
University. He previously served as a Program Man-
ager at DARPA in 2004–2006, and was awarded the
Office of the Secretary of Defense Medal for Excep-
tional Public Service in 2006. He became an IEEE Fel-
low in 2001. Contact him at jms@cis.upenn.edu.
IEEE Micro
 General Interest

RASSA: Resistive
Prealignment Accelerator for
Approximate DNA Long
Read Mapping
Roman Kaplan, Leonid Yavits, and
Ran Ginosar
Technion—Israel Institute of Technology

Abstract—DNA read mapping is a computationally expensive bioinformatics task,

required for genome assembly and consensus polishing. It requires to find the best-
fitting location for each DNA read on a long reference sequence. A novel resistive
approximate similarity search accelerator (RASSA) exploits charge distribution and
parallel in-memory processing to reflect a mismatch count between DNA sequences.
RASSA implementation of DNA long-read prealignment outperforms the state-of-the-art
solution, minimap2, by 16–77
with comparable accuracy and provides two orders of
magnitude higher throughput than GateKeeper, a short-read prealignment hardware
architecture implemented in FPGA.

& CONSTRUCTING DNA sequence in real

HUMAN
time is paramount to development of precision
medicine1 and on-site pathogen detection of
disease outbreaks.2 Single-molecule, real-time
sequencing from Pacific Biosciences3 (PacBio) and
Oxford Nanopore Technologies4 (ONT) are new
technologies that can produce long reads within
Digital Object Identifier 10.1109/MM.2018.2890253
Date of publication 28 December 2018; date of current
version 23 July 2019.
minutes, potentially enabling real-time genomic
analysis. However, long-read DNA sequencing
poses new challenges. First, long reads contain
many thousands of base pairs (bps). Second, long
reads tend to exhibit about 15%–20% insertion,
deletion (indel), and substitution errors.3,4
To construct a complete host sequence, in
case a reference sequence exists (from a previ-
ously sequenced organism), long reads
are mapped to high-similarity locations of the
reference sequence. Determining the edit dis-
tance between every mapped read and the

0272-1732 ß 2018 IEEE Published by the IEEE Computer Society IEEE Micro
44
reference sequence requires a computationally performance and energy efficiency. Resistive
intensive local alignment procedure (e.g., Smith– approximate Hamming distance solutions exist.11
Waterman).4 Its computational time complexity is However, these do not provide the parallelism
typically OðnmÞ for two sequences with lengths n required to support a high throughput applica-
and m. Reference sequences vary from several tion, such as DNA read mapping.
millions to billions of bits per second (bps). It is In this work, we present RASSA, a resistive
therefore computationally prohibitive to perform approximate similarity search accelerator archi-
optimal alignment of every long read with the tecture for DNA long-read prealignment filtering.
entire reference sequence. RASSA is a massively parallel in-memory proces-
Read mappers (e.g., minimap6 and minimap27) sor, facilitating simultaneous comparison of
find regions of high similarity (mappings) between a long read with a reference sequence. The out-
reads or between a read and a puts of RASSA are locations on the
reference sequence, followed by RASSA is a massively reference sequence, where align-
an alignment step to determine parallel in-memory pro- ment may result in high score. The
the exact edit distance and verify cessor, facilitating key performance breakthrough of
that the mapping is correct. In simultaneous compari- RASSA is achieved by applying the
case that a prealignment algo- son of a long read with similarity search in parallel to the
rithm identifies a specific region a reference sequence. entire reference. While the complex-
in the reference suitable for map- ity of alignment is OðmnÞ, RASSA
ping, the alignment can be performed only on employs in-memory parallel computing on OðmÞ
that region, reducing alignment’s duration and memory cells to reduce the computation time to
resource requirements.8 Therefore, read OðnÞ, where m and n are read and reference
mapping can be viewed as a two-step process: pre- lengths, respectively.
alignment filtering and accurate alignment RASSA employs resistive elements, mem-
verification. The prealignment step reduces the ristors, serving at the same time as single bit
problem size for aligners by narrowing the regions storage elements and comparators. Additional
to ones with potentially high-scoring alignment. evaluation transistors translate mismatch scores
Existing prealignment hardware solu- into voltage levels, which are converted into dig-
tions9,10 target short reads (up to several hun- ital values using analog-to-digital converters
dred bps), which contain a small number of (ADC). Further processing determines the most
indel and substitution errors (less than 5%) and likely overlap candidates.
have a different error profile than that of PacBio This paper makes the following contributions.
or ONT long reads.3,4 High edit distance thresh-
old is required for mapping long but error- 1) RASSA, an in-memory processing resistive
9 approximate similarity search accelerator, is
prone reads. However, current solutions have
high false positive rates when the edit distance introduced. The parallel processing architec-
is high (i.e., greater than 15). Thus, the current ture is presented bottom-up, from the mem-
solutions for short reads are not applicable for ristor-based bitcell to base pair encoding
long reads. and up to a complete RASSA system.
Approximate computing techniques are 2) RASSA-based implementation of long read
known to trade accuracy for speed or energy prealignment filtering is developed.
efficiency. In case of long reads, multiple errors 3) Evaluation of RASSA’s prealignment filtering
are a natural part of the sequencing output. accuracy and comparative analysis of its exe-
Therefore, DNA long-read prealignment filtering cution time and throughput is conducted.
inherently tolerates the imprecision.
With the end of Dennard scaling and the
slowdown of Moore’s law, novel hardware BACKGROUND
solutions for data-intensive problems are resear- The following two sections provide concise
ched. Emerging technologies such as resistive background on the problem: DNA read mapping
memories enable new architectures with better and the memristor device technology.

July/August 2019
45
 General Interest

DNA Read Mapping

DNA sequencers out-
put fragmented regions of
DNA called reads. The
reads originate from ran-
dom locations in the
genome and may be over-
lapping. If a DNA
sequence from the same
species exists, the DNA
reads are matched
to such existing (refer-
ence) sequence [see
Figure 1(a)]. The main
assumption behind this
approach is that the refer-
ence and the reads origi-
nate from the same
species and, therefore, Figure 1. (a) Mapping of long DNA reads onto existing reference sequence.
contain small number of Red colored bps represent mismatching bps between the reference and reads.
differences (typically less (b) Single RASSA bitcell. (c) and (d) Example of two DNA bps comparison. One
than 1%). Since the main bp matches the compared pattern, preventing match line charge loss (c). The
computational effort in next bp mismatches, causing match line voltage reduction (d).
this process is placing the
reads correctly to the
reference sequence, it is called read mapping. RASSA: RESISTIVE APPROXIMATE
One common read mapping technique is indi- SIMILARITY SEARCH ACCELERATOR
vidual mapping of fixed length segments (seeds) Analyzing long-read alignments to a reference
of a read.6,7 Each seed is used as a key in a pre- sequence reveals long fragments of indel-free
calculated hash table, which values are the seed high-similarity regions [as in Figure 1(a)]. These
locations in the reference sequence. Such loca- regions usually contain tens of bps with few substi-
tions are then extended and a precise alignment tutions, and can sometimes reach hundreds of bps
3
is performed. Once all reads are placed in their (for example, in case of high-accuracy CCS reads).
positions, the new sequence is constructed from This has motivated us to use simple Hamming dis-
the overlapping regions of the reads. A disadvan- tance as a heuristic to find the overlap positions
tage of this approach is its high memory require- of long reads against a reference sequence. To
ments, long running time, and complexity. overcome indels and find high-similarity sections,
all possible overlap positions of a read against a
reference are examined. Each bp has four values,
Resistive Memories
therefore the probability of a mismatch when two
Resistive memories store information by
random bps are compared is 3=4. Comparing two
modulating the resistance of nanoscale storage
elements, called memristors. They are nonvola- random sections of equal length from DNA sequen-
tile, free of leakage power, and emerge as poten- ces leads, on average, to 75% mismatching bps.
tial alternatives to charge-based memories,12 However, when high similarity fragments are com-
including NAND flash. Memristors are two-termi- pared, the running Hamming distance may drop
nal devices, where the resistance of the device is significantly below the 75% average, thus indicat-
changed by the electrical current or voltage. The ing a potential mapping location.
resistance of the memristor is bounded by a min- RASSA is a resistive memory based massively
imum resistance RON (low resistive state) and a parallel processing-in-memory accelerator. It
maximum resistance ROFF (high resistive state). allows storing (typically, a data element per

46 IEEE Micro
memory row) and in situ processing of large data
sets. RASSA enables comparing a key pattern with
the entire data set in parallel. Every number of
mismatches (of the key pattern versus each data
element that is in each memory row) causes a spe-
cific voltage drop, allowing quantifying the num-
ber of mismatching locations (called a mismatch
score). The mismatch score is compared with a
predefined threshold value to detect the locations
that have the desired degree of similarity with the
compared pattern, indicating a viable mapping
location. The following sections describe RASSA
functionality, encoding of DNA bp, RASSA system
architecture, and hardware evaluation.
DNA Base Pair Encoding and
Mismatch Evaluation
Figure 1(b) presents the RASSA bitcell, con-
taining two transistors and one memristor
(2T1R). Each memristor serves as a single bit
storage element and a single bit comparator,
enabled by the selector transistor.
A compare operation consists of two phases: the
precharge and the evaluation. During precharge,
the match line is precharged to a certain voltage
level. At the same time, the evaluation transistor in
each bitcell is on to discharge the evaluation point
(created by the diffusion capacitances of the selec-
tor and the evaluation transistors).
During the evaluation phase, if the selector
transistor is on, a low memristor resistance
ðRON Þ allows charge to pass from the match line
to the evaluation point. The charge distribution
causes the match line voltage to drop. Sensing
the voltage of the match line compared with a
reference voltage (of zero mismatch) allows
quantifying the number of mismatches, produc-
ing the mismatch score.
Encoding. RASSA reserves four bitcells to store
a DNA bp. There are four nucleotide bases: A, C,
G, and T in each DNA bp, encoded using one-hot
encoding as “1000,” “0100,” “0010,” and “0001,”
respectively. While it is possible to encode four
nucleotide bases using two bits (for example,
“’00,” “’01,” “10,” and “11” for A, C, G, and T,
respectively), such encoding would result in dif-
ferent number of mismatching bits depending on
a specific pair (for example, two mismatching
bits in the case of A–T or C–G mismatch, or one
July/August 2019
mismatching bit in the case of A–C or A–G mis-
match), leading to ambiguous results. Since a
mismatch is signaled by reduced match line volt-
age (caused by charge redistribution), a match
should block charge flow. One hot encoding
assures that at most one mismatch may happen
in each group of four bitcells. For instance, in 60
bitcells, at most 15 mismatches may be
observed. Therefore, in this work, a memristor
in high resistive state ðROFF Þ is considered logic
“1,” while RON is considered logic “0.”
Mismatch Evaluation. During a compare
operation, the compared (key) pattern is applied
to the gates of the selector transistors of all bit-
cells. If certain groups of bitcells need to be
ignored (masked-out) during comparison, zero
is applied to the gates of the selector transistors
of such bitcells. Figure 1(c) shows a stored “A”
nucleotide symbol and a compare pattern of “A.”
The comparison results in a match, so there is
no charge redistribution path (through an ROFF
memristor). Figure 1(d) shows a mismatch,
where the stored pattern is “G” and the key pat-
tern is “T.” The mismatch results in charge redis-
tribution through an RON memristor, causing a
match line voltage drop. Figure 2(a) shows all
possible match line voltage levels during the
evaluation phase for mismatch scores of 0
through 15. The match line is sensed by an ADC
(System Architecture section). The timing of
such sensing, in addition to the per-cell transis-
tor capacitance variations, may lead to inaccura-
cies in the mismatch score. For example, for a
match line shared by up to 60 bitcells, the mis-
match score error could be 1. If the number of
bitcells sharing the match line is more than 120,
the mismatch score error could reach 3.
System Architecture
The main component of RASSA is the 2T1R
array, divided into Word Rows [see Figure 2(c)],
further divided into Sub-Words [see Figure 2(b)].
All Word Rows are connected in parallel to the
Key Pattern register [see Figure 2(d)]. The ADC is
the largest and most energy consuming compo-
nent of a Sub-Word. Therefore, in order to use
only 4-bit ADC, supporting mismatch scores of 0
through 15, the Sub-Word is limited to 60 bitcells.
There are 16 Sub-Words within a Word Row,
47
 General Interest

Figure 2. (a) Match line voltage levels for each mismatch score between zero (top curve) and 15 (bottom curve)
mismatches. Every voltage level at the sampling point is converted by the ADC. (b)–(d) Bottom-up block diagram of
RASSA. (b) Single Sub-Word, composed of 60 bitcells in NOR-like structure. (c) Single Word Row containing 16 Sub-Words,
capable of holding 240 DNA bps. (d) Complete RASSA diagram containing N Word Rows ðN ¼ 217 Þ. (e) Accumulating
mismatch scores in analog domain (Conclusions and Future Research Directions section).

amounting to 960 bitcells per word, designed for
storing and comparing up to 240 DNA bps per
cycle. In each compare operation, a compare pat-
tern is applied to all active bitcell bit lines.
The match line voltage of each Sub-Word is
sampled by the ADC and converted into a 4-bit mis-
match score [right side of Figure 2(b)]. The ADC
reference voltage and voltage level differences are
set according to the match line values for each mis-
match score, as demonstrated in Figure 2(a). The
16 Sub-Word ADC outputs are summed up to pro-
duce the mismatch score for the entire Word Row
[see Figure 2(b)]. All such scores are then com-
pared with a threshold value, in parallel, to indi-
cate the Word Rows (corresponding to sequence
locations) with the desired degree of similarity.
Timing, Power, and Area Breakdown
A Sub-Word circuit is designed, placed, and
routed using the 28-nm CMOS High-k Metal Gate
library from Global Foundries for transistor sizing,
timing, and power analysis. We perform Spectre
simulations for the FF and SS corners at 70  C and
nominal voltage. Timing analysis shows that an
operational frequency of 1 GHz is possible. For a
single Sub-Word, the precharge energy is 1.6 fJ,
while the evaluation energy (ADC and control line
switching) is 98.4 fJ. For a single Word, containing
16 Sub-Words, adders, and threshold comparator,
a single compare cycle energy is 1791 fJ.
We have manually laid out a RASSA bitcell.
The total Word Row area in 28-nm technology,
including the bitcells, ADC, adders, and compar-
ator, is 1598 mm2. Bitcell transistors occupy 4%,
adders and threshold comparator occupy 28%,
and the ADC occupies 68% of the Word Row
area, respectively. This allows placing of
131 k ð217 Þ 960-bit (240-bp) Word Rows, storing
31.5 Mb/s, on a single 209-mm die. Its worst-case
power consumption at 1 GHz is 235 W. Table 1
summarizes the RASSA system parameters.
Loading the reference sequence to RASSA is
performed on each Word Row separately and
requires two cycles per Word Row, one cycle to
write all logic “0s,” and another cycle to write all
logic “1s.” Given a reference sequence of length
L, the number of cycles required for its loading
L
equals 2  d240e.

48 IEEE Micro
Table 1. RASSA system parameters for 28-nm node
process.
Parameter Value
DNA bps per row (bits) 240ð960Þ
Words per IC 131 kð217 Þ
Memory size (DNA bps) 31:5M
Frequency 1 GHz
Single IC Power 235 W
Single IC Area 209 mm2
DNA READ PREALIGNMENT
FILTERING WITH RASSA
A single compare operation in RASSA finds
the mismatch score between the key pattern
and the contents of each Word Row. The refer-
ence sequence is stored in RASSA (contiguously,
240-bp fragment per Word Row). A fixed-size
chunk (e.g., 200 bps) of the read is fed in as a key
pattern. The mismatch score approximates the
correlation between the read chunk and the ref-
erence sequence. A long read contains multiple
chunks, therefore the compare operations are
performed multiple times, in all possible posi-
tions of a read chunk vis-a -vis a Word Row
reference fragment, sometimes involving two
neighboring Word Rows.
The number of Word Rows in RASSA defines
the number of overlap positions examined simul-
taneously. In a single cycle, dn=240e (where n is
the reference sequence length) distinct posi-
tions on the reference sequence are examined
simultaneously. To cover all possible positions,
the read chunk is shifted by one bp, and com-
pare is repeated 240 times (resembling the con-
cept of correlation).
Figure 3(a)–(d) illustrates the comparison of
a read chunk against a reference sequence in
RASSA for several cases. In these examples, a
chunk length of 200 bp is used [see Figure 3(a)].
A multicycle compare operation matches a
200 bp chunk against all its possible locations
-vis the reference sequence. In the first com-
vis-a
pare cycle [see Figure 3(b)], the first chunk of
the read is compared with all first 200 bps of
each RASSA Word Row.
Following the completion of 41(¼ 240-200 þ
1) cycles, the 200-bp chunk is compared against
July/August 2019
reference data residing in two Word Rows. Such
two-Word Row compare requires two cycles.
The even cycle mismatch score [see Figure 3(c)]
is added to the score of the following odd cycle
[of the Word Row below, Figure 3(d)], and com-
pared with the threshold [see Figure 3(d), right].
Before every even cycle, the compare pattern is
shifted by one bp to the right, shortening the
even cycle compare pattern and extending
the pattern in the odd cycle by one bp [see
Figure 3(c) and (d), right]. After 439 (¼ 41 þ 199

2) cycles, a 200-bp chunk has been compared
against all reference sequence positions. The
compare operation repeats for the rest of the
200-bp read chunks.
Figure 3(e) presents the concept of edit detec-
tion in RASSA. For simplicity, the chunk length in
this example is 30 bp. Three types of edits are
shown on the left of the figure. On the right, the
mismatch score is presented for all relative shifts
of the chunk versus the reference.
The average mismatch score in any mismatch-
ing location is 75% (the probability of an individ-
ual bp mismatch is 0.75). In the exact match case
(top of the figure), the mismatch score is 0.
A substitution results in a very low mismatch
score easily detectable by RASSA. The second
row of Figure 3(e) shows two substitution errors,
leading to the mismatch score of 2/30 ¼ 6.7%.
The third row of Figure 3(e) shows an insertion
error. The longest matching section is 18 bp to the
right of the insertion, which leads to a mismatch
score of (30 - 18)/30 ¼ 40%. With the appropriate
threshold, such scenario is detectable by RASSA
and identified as a potential mapping.
Finally, in the fourth row of Figure 3(e), a dele-
tion error is presented. Deletions are handled sim-
ilarly to insertions. In this example, the longest
matching section is also 18 bp. The mismatch
score is (30 - 18)/30 ¼ 40% as well, and is also
detectable by RASSA as a potential mapping.
While a substitution results in a much lower
mismatch score, RASSA is capable of detecting
indels just as confidently, by setting the thresh-
old accordingly.
In legitimate mismatching locations, the mismatch score is distributed
binomially. The threshold is set such that the probability of the mismatch
score to fall below it is sufficiently low. For example, P (mismatch score <
50%) ¼ 0.0008 for 30-bp chunk. For 200-bp chunk, similar probability is
reached at the threshold of 65%.
49
 General Interest

Figure 3. Illustration of a single long-read chunk examination in RASSA. (a) Long read is divided into chunks,
each 200 bp long. (b, left and right) First chunk is compared against the reference sequence in multiple locations
(simultaneously). (c) and (d) First chunk overlaps with reference sequence bps from two Word Rows. (c, left) First
part of the chunk compared with the last bps of the Word Row. (c, right) All Sub-Word mismatch scores are
summed up and stored (compare to threshold does not take place). (d, left) Second part of the chunk is
compared with the first bps of the next Word row. (d, right) All Sub-Word mismatch scores, including the previous
cycle result from the above Word Row, are summed up and compared with a threshold. Following this step, the
chunk is shifted right by one position (relative to the reference) and steps (c) and (d) are repeated. (e) Edit types
and the mismatch score found by RASSA. (f) Example of a 30-bp read chunk containing insertion, deletion and
substitution errors is compared against the reference sequence divided into 50-bp Word Rows. (g) Mismatch
score versus cycle number for first 21 cycles of comparison of the example in (f). Minimal mismatch score is
below the threshold and achieved in the 9th cycles. The threshold is determined empirically per data set.

Figure 3(f) and (g) illustrates the mismatch
score for the sliding window search of RASSA, in
presence of multiple errors. Figure 3(f) presents
an example of a reference sequence and a read
chunk containing a high-similarity region. All
possible edit types (substitution, insertion, and
deletion) exist in the chunk. Figure 3(g) illus-
trates the mismatch score as a function of the
compare cycle (the relative read chunk position).
During most cycles, the mismatch score is above
threshold. When the chunk is in its valid mapping
location [“min mismatch position” at cycle 9 in
Figure 3(f)], the mismatch score is significantly
lower than the random average 75% level. Setting
the custom threshold, for instance, at 50% allows
efficient overlap of read chunks with a number of
edits (substitutions as well as indels).
Translating the Output of RASSA to
Mapping Locations
RASSA compares every chunk of every read
against the entire reference sequence. The

50 IEEE Micro
probability of a false positive match is extremely
low. Therefore, we assume that every compare
that results in a mismatch score below the prede-
fined threshold indicates is valid mapping loca-
tion for the entire read. The output of RASSA is a
bit vector, one bit per Word Row. The index of the
Word Row, together with the iteration number
and relative position of the chunk within the read,
provides an exact coordinate of a potential map-
ping location. In most examined cases, a single
read has a single mapping location indicated by a
single compare from a single chunk.
In some other cases, a single or multiple
chunks produce multiple potential mapping
locations. In such cases, the distance between
consecutive locations is examined, starting from
the lowest coordinate. If the distance between
two consecutive locations is smaller than the
read length, the location associated with the
higher coordinate is discarded. Otherwise, both
locations are kept for further processing.
With this selection heuristic, nearby potential
mapping locations from a single or multiple
chunks are combined, while distant locations
are treated as separate mapping locations.
The mapping locations identified by RASSA
can further be verified by alignment (e.g., Smith–
Waterman algorithm),4 and used by assembly or
error correction programs.8 Unmapped reads
can either be discarded (in case of a high
sequencing coverage) or be mapped with a seed-
and-extend mapper, and then verified by an
alignment algorithm.
EVALUATION
We compare RASSA with two existing solu-
tions, minimap2,7 a state-of-the-art read mapping
tool, and GateKeeper,9 a state-of-the-art short-
read prealignment hardware accelerator.
Comparison With Minimap2
Our evaluation focuses on accuracy and
speedup. Accuracy is measured by two criteria:
1) sensitivity: correctly mapped reads; 2) false
positives: percentage of incorrect mappings out
of all mappings by RASSA.
Methodology. Minimap2 is run with the
parameters “-x map-pb” and “-x map-ont,” invok-
ing its execution for overlap detection without
July/August 2019
the alignment step. With such parameters, mini-
map2 functions as a prealignment filter. The
parameters also invoke appropriate heuristic for
PacBio and ONT reads, in addition to enabling
multithreading and SIMD extensions. To evaluate
RASSA’s prealignment filtering accuracy, we use
minimap2 as a golden reference. Speedup is cal-
culated as the ratio of minimap2 execution time,
without indexing, to RASSA execution time. The
accuracy and speedup of RASSA were obtained
using an in-house simulator. We assume that the
reference sequence has already been loaded
into RASSA prior to execution.
To find the number of incorrect output loca-
tions that might increase the total alignment time,
we have contrasted prealignment followed by
alignment with alignment without prealignment.
We have used part of the E.coli PacBio dataset,
consisting of about 1000 reads. Total prealign-
ment by RASSA took 20 ms, and the following
alignment needed to be applied to only 70-kbp
subset of the reference, taking minimap2 1490 ms.
In contrast, the same alignment applied without
the prealignment stage took 3000 ms, about twice
the time. Therefore, we decided that reads with
more than two output locations by RASSA will be
discarded and treated as incorrectly mapped.
Data Sets. We use five publicly available
datasets, three from PacBio and two from
ONT, taken from two organisms: E.coli K-12
NG1655 and Saccharomyces cerevisiae W303
(yeast). Both reference sequences are avail-
able at the NCBI (https://www.ncbi.nlm.nih.
gov/). PacBio data sets were taken from
https://github.com/PacificBiosciences/DevNet/
wiki/Datasets. Error rates, 13 including the
share of insertions, deletions, and mismatches
(I,D,M) are also presented.
E.coli
 PacBio: 100 000 reads from one SMRT cell,
5245 bps on average. Error rate: 14.2%
(I:41.7%, D:21.2%, M:37.1%)
 PacBio CCS: 260 000 high-quality CCS reads
from 16 SMRT cells, 940 bps on average.
Error rate: 1% (I:5%, D:19.5%, M:75.5%)
 ONT (from http://lab.loman.net/2016/07/30/
nanopore-r9-data-release/): 165 000 R9 1D
51
 General Interest

Table 2. Sensitivity, fraction of exact mappings, and speedup of RASSA compared to minimap2.

Data sets Large Chunk (200 bps) Small Chunk (100 bps)
False False
Sensitivity Speedup Sensitivity Speedup
Positives Positives
E.coli PacBio 79.3% 13.4% 25
83.2% 13.6% 16

E.coli PacBio
96.3% 8.9% 43
96.2% 6.9% 24

CCS
E.coli ONT 88.8% 10.5% 48
87.6% 12.4% 31

Yeast PacBio 69.8% 8.7% 77
72% 11.8% 51

Yeast ONT 85.9% 34.9% 31
85.1% 39.2% 49

minimap2 mapped only about 20% of all reads, with 50% of mappings with lower quality score than 60 (indicates a high-confidence
mapping).

reads, 9009 bps on average. Error rate: 20.2%
(I:14.5%, D:37.2%, M:48.3%)
Yeast

PacBio: 100 000 reads from one SMRT cell,
6294 bps on average. Error rate: 14% (I:5%,
D:19.5%, M:75.5%)
 ONT (ERR789757 from NCBI): 30 000 R7.3 2D
MinION reads, 11,337 bps on average. Error
rate: 13.4% (I:23.3%, D:35.7%, M:41%)
Table 2 presents the accuracy results for all
five data sets above. Chunk sizes of 200 and 100
bps and corresponding thresholds were deter-
mined empricially, trading off accuracy and per-
formance. Small changes of threshold induce
only marginal changes in accuracy. For most
data sets, 55% threshold was used on 200-bp
chunks and 45% for 100-pb chunks; for the Yeast
PacBio case, we used 45% and 40%, respectively.
Speedup. We compare RASSA execution time
with that of minimap2, executed on a server with
16-core 2-GHz Intel Xeon E5-2650 CPU and 64 GB of
RAM. Table 2 shows that RASSA achieves 16–77

speedup over minimap2.y We note that the yeast
dataset has fewer reads than E.coli, but a longer
reference sequence (11.7 Mbp versus 4.6 Mbp),
which might cause the longer execution time on
minimap2. In contrast, RASSA is insensitive to
the reference sequence length and its execution
time is determined by the length of a read chunk.
RASSA produces output (on average, one
mapping per read) at rate of 50,000–500,000
reads/s, enabling multiple simultaneously
executing instances of a typical alignment
algorithm.
Throughput Comparison With GateKeeper
We compare RASSA throughput (the number
of examined mapping locations per second) with
that of GateKeeper.9 GateKeeper was imple-
mented in a Virtex-7 FPGA using Xilinx VC709
board running at 250 MHz.
GateKeeper is designed to compare short
reads with a reference sequence. Table 3 shows
the throughput in Billions of Examined Mapping
Locations per second (BEML/s) of RASSA and
GateKeeper on two short-read data sets used in9
100-bp reads and 300-bp reads. RASSA frequency
is adjusted to that of GateKeeper. In addition, we
show the average RASSA throughput for the
200-bp reads, equivalent to the chunks lengths
used in Table 2.
RASSA outperforms GateKeeper by more
than two orders of magnitude. When applied to
the short-read mapping prealignment, RASSA
covers a read with one chunk. Consequently,
RASSA takes 1–3 cycles (for the read lengths
used in Table 3) to find the mismatch score in all
Word Rows in parallel. GakeKeeper, on the other
hand, is reported to process up to 140 (20)
Table 3. RASSA and GateKeeper9 throughput (billions of
examined mapping locations per second, BEML/s).
Read RASSA @250
GateKeeper
Lengths MHz
100 bp 1.7 BEML/s 226.8 BEML/s
200 bp - 175.2 BEML/s

300 bp 0.2 BEML/s 142.8 BEML/s

y
The overlap detection stage without alignment of minimap2.

52 IEEE Micro
mapping locations of 100-bp (300-bp) reads in
parallel, affected by the edit distance threshold.
CONCLUSIONS AND FUTURE
RESEARCH DIRECTIONS
This paper presents RASSA, an in-memory
processing parallel architecture of a resistive
approximate similarity search accelerator. We
apply RASSA to the long-read DNA mapping
problem. The length of reads, coupled with a low
read quality, poses a challenge for existing map-
pers, optimized for high-quality short reads. The
read mapping process is data and compute inten-
sive, making it a target for acceleration. RASSA
addresses the challenge by breaking long reads
into short chunks and by applying full correlation.
By allowing faster mapping on large data sets, we
potentially make a step toward real-time pathogen
or genome sequence completion.
We compared
the RASSA accu-
We compared the
racy and execution
RASSA accuracy and
time with that of
execution time with that
minimap2, a state-
of minimap2, a state-of-
of-the-art mapping the-art mapping solu-
solution, on five tion, on five long-read
long-read data sets data sets taken from
taken from two two organisms. Our
organisms. Our evaluation shows that
evaluation shows RASSA can outperform
that RASSA can minimap2 by 16–77x.
outperform mini-
map2 by 16–77
. In
addition, we compared RASSA’s throughput,
measured in examined mapping locations per
second, with that of GateKeeper, a state-of-the-
art short-read prealignment hardware accelera-
tor. We find that RASSA can outperform Gate-
Keeper by more than two orders of magnitude.
This work can be extended in several ways.
First, RASSA can be applied to read-to-read
overlap finding, which requires finding overlaps
between pairs of reads. Read-to-read overlap
finding is an important first step in de novo
genome assembly6 (constructing the host DNA
sequence without a reference sequence), a prob-
lem more computationally challenging than read
mapping. Second, a detailed design space explo-
ration needs to be performed. For example,
RASSA can further be optimized in terms of
July/August 2019
hardware cost: higher density can be achieved
by sharing ADCs among multiple Sub-Words and
by applying analog computations, as presented
in Figure 2(e). RASSA mapping and resistive CAM
alignment5 may be combined into a single high-
performance in-memory mapper/aligner. Finally,
thanks to its use of short chunks, RASSA can be
effectively applied to short reads.
& REFERENCES
1. J. L. Jameson and D. L. Longo, “Precision medicine—
Personalized, problematic, and promising,”
Obstetrical Gynecological Survey, vol. 70, no. 10,
pp. 612–614, 2015.
2. J. Quick et al., “Real-time, portable genome
sequencing for Ebola surveillance,” Nature, vol. 530,
no. 7589, pp. 228–232, 2016.
3. A. Rhoads and K. F. Auc, “PacBio sequencing and its
applications,” Genom., Proteomics Bioinform., vol. 13,
no. 5, pp. 278–289, 2015.
4. T. Laver et al., “Assessing the performance of the
oxford nanopore technologies minion,” Biomol.
Detection Quantification, vol. 3, pp. 1–8, 2015.
5. R. Kaplan, L. Yavits, R. Ginosar, and U. Weiser
“A resistive CAM processing-in-storage architecture
for DNA sequence alignment,” IEEE Micro., vol. 37,
pp. 20–28, 2017.
6. H. Li, “Minimap and Miniasm: Fast mapping and
de novo assembly for noisy long sequences,”
Bioinformatics, vol. 32, no. 14, pp. 2103–2110, 2016.
7. H. Li, “Minimap2: Pairwise alignment for nucleotide
sequences,” Bioinformatics, vol. 34, pp. 3094–3100,
2018.
8. K. Berlin, S. Koren, C. S. Chin, J. P. Drake,
J. M. Landolin, and A. M. Phillippy, “Assembling large
genomes with single-molecule sequencing and
locality-sensitive hashing,” Nature Biotechnol., vol. 33,
no. 6, pp. 623–630, 2015.
9. M. Alser, H. Hassan, H. Xin, O. Ergin, O. Mutlu, and
C. Alkan, “GateKeeper: A new hardware architecture for
accelerating pre-alignment in DNA short read mapping,”
Bioinformatics, vol. 33, no. 21, pp. 3355–3363, 2017.
10. J. S. Kim et al., “GRIM-filter: Fast seed location filtering
in DNA read mapping using processing-in-memory
technologies,” BMC Genom., vol. 19, no. 2, p. 89, 2018.
11. M. Imani, A. Rahimi, D. Kong, T. Rosing, and J. M. Rabaey,
“Exploring hyperdimensional associative memory,” in
Proc. IEEE Int. Symp. High Perform. Comput. Archit., 2017,
pp. 445–456.
53
 General Interest
12. H. Akinaga and H. Shima, “Resistive random access
memory (ReRAM) based on metal oxides,” Proc. IEEE,
vol. 98, no. 12, pp. 2237–2251, Dec. 2010.
13. J. L. Weirather et al., “Comprehensive comparison of
pacific biosciences and oxford nanopore technologies
and their applications to transcriptome analysis,”
version 2, F1000Research, 2017.
Roman Kaplan is currently a PhD student in the
Faculty of Electrical Engineering, Technion—Israel
Institute of Technology, under the supervision of
Prof. R. Ginosar. Between 2009 and 2014, he was a
software engineer. His research interests are parallel
computer architectures, in-data accelerators for
machine learning, and bioinformatics. He has a BSc
and an MSc from the Faculty of Electrical Engineer-
ing, Technion. Contact him at romankap@gmail.com.
Leonid Yavits is currently a Postdoctoral Fellow
in electrical engineering at Technion—Israel Insti-
tute of Technology. He co-founded VisionTech
where he co-designed a single chip MPEG2
codec. VisionTech was acquired by Broadcom in
54
2001. He coauthored a number of patents and
research papers on SoC and ASIC. His research
interests include non-von Neumann computer
architectures, accelerators, and processing in
memory. He has an MSc and a PhD in electrical
engineering from Technion. Contact him at leonid.
yavits@nububbles.com.
Ran Ginosar is a professor with the Department of
Electrical Engineering and serves as the head of
the VLSI Systems Research Center at the Techn-
ion—Israel Institute of Technology. He joined the
Technion faculty in 1983 and was a visiting associ-
ate professor with the University of Utah during
1989–1990, and a visiting faculty with Intel
Research Labs in 1997–1999. His research inter-
ests include VLSI architecture, manycore com-
puters, asynchronous logic and synchronization,
networks on chip, and biologic implant chips. He
has co-founded several companies in various areas
of VLSI systems. He has a BSc (summa cum laude)
from Technion and a PhD from Princeton University,
both in electrical and computer engineering.
Contact him at ran@ee.technion.ac.il.
IEEE Micro
 General Interest

The Queuing-First
Approach for Tail
Management
of Interactive Services
Amirhossein Mirhosseini and
Thomas F. Wenisch
University of Michigan

Abstract—Managing high-percentile tail latencies is key to designing user-facing cloud

services. Rare system hiccups or unusual code paths make some requests take 10
100

longer than the average. Prior work seeks to reduce tail latency by trying to address
primarily root causes of slow requests. However, often the bulk of requests comprising the
tail are not these rare slow-to-execute requests. Rather, due to head-of-line blocking, most
of the tail comprises requests enqueued behind slow-to-execute requests. Under high
disparity service distributions, queuing effects drastically magnify the impact of rare system
hiccups and can result in high tail latencies even under modest load. We demonstrate that
improving the queuing behavior of a system often yields greater benefit than mitigating the
individual system hiccups that increase service time tails. We suggest two general
directions to improve system queuing behavior–server pooling and common-case service
acceleration–and discuss circumstances where each is most beneficial.

& ONLINE DATA-INTENSIVE (OLDI) services (e.g.,
web search) traverse terabytes of data with strict
latency targets.1 Managing high-percentile tail
latencies is a key problem in designing such serv-
ices. First, to guarantee user satisfaction, services
Digital Object Identifier 10.1109/MM.2019.2897671
Date of current version 23 July 2019.
must meet strict response time service-level
objectives (SLOs), especially for tail latencies.2; 3
Second, such services typically communicate via
fan-out patterns wherein datasets are “shared”
across numerous “leaf” servers and their
responses are aggregated before responding to
the user. As such, overall latency is often dictated
by the slowest leaves (i.e., the “tail at scale”
effect4).

July/August 2019 Published by the IEEE Computer Society 0272-1732 ß 2019 IEEE
55
 General Interest

High tail latencies arise from two effects. First, behavior: server pooling, and common-case ser-
such applications’ service time distributions vice acceleration (CCSA). Server pooling is the
include outlying requests that take much longer practice of redesigning system architecture to
(10
 100
or more) than the mean.5 Some change single-server (“scale-out”) queues into
requests may require exceptional processing multiserver (“scale-up”) ones; that is, rather
time depending on their arguments (e.g., search than enqueuing requests at distinct servers/
engines1; 6) or query types (e.g., sets versus gets cores, a single queue is shared among many (i.e.,
in key-value stores5; 7). Some requests are delayed converting c G/G/1 queues into a G/G/c). Server
by system interference, such as from garbage col- pooling greatly reduces queuing delay and can
lection, page deduplication, synchronous huge- completely eliminate queueing with enough
page compaction, or network stack imped- servers (i.e., high enough c). Pooling smooths
iments.4; 8 In other cases, scheduler inefficiencies, fluctuations in both arrivals and service, making
power state transitions, suboptimal interrupt the system behave more like one with determin-
routing, poor NUMA node allocation, or virtualiza- istic interarrival and service times. Especially for
tion effects may contribute to long tail latencies.9 high disparity service time distributions (i.e.,
Finally, interference from colocated workloads rare system events/hiccups), server pooling
can cause slowdown due to contention for shared reduces the overall tail latency by breaking
caches, memory bandwidth, or global resources HoL blocking and preventing nominal requests
like network cards or switches.10; 11 from waiting behind exceptionally long ones.
A second key contributor to applications’ Even a modest degree of concurrency allows
end-to-end latency distribution are queuing many short requests to drain past stalled ones,
effects.3 Queuing arises at numer- substantially reducing weight in
ous layers causing some requests Online data-intensive
the latency distribution tail.
to wait for others4; whereas queu- (OLDI) services (e.g., CCSA improves systems’ queu-
ing also affects average perfor- web search) traverse ing behavior by deploying optimi-
mance, its effect on tail latency terabytes of data with zations that target common-case
may be catastrophic. To achieve strict latency targets. service behavior (as opposed to
performance stability, systems Managing high- optimizations that target directly
must be engineered such that the percentile tail latencies rare/slow requests or hiccups). It
overall request arrival rate is lower is a key problem in may seem counterintuitive to
than the aggregate system capacity designing such improve tail latency by optimizing
(service rate). However, as both services. typical-case request performance,
rates fluctuate, arrivals may tempo- but queuing delays are greatly
rarily outstrip service capacity, causing requests impacted by the average load, which depends
to queue. Queueing delay is most apparent more on typical-case service time than rare cases.
under high system load. However, in this paper, In single-server systems, CCSA has little
we make the case that queuing effects drastically impact when the service variance is excessively
magnify the impact of rare system events/hic- high (i.e., HoL blocking is common), as nominal
cups and can result in high tail latencies even requests queue behind rare, slow ones regard-
under modest load. Due to head-of-line (HoL) less of how fast the nominal requests are proc-
blocking, many requests are delayed by an essed. But, if there is sufficient concurrency
exceptionally slow one that stalls a server/core; (e.g., by using server pooling) that slow requests
these delayed requests account for a bulk of the rarely occupy all servers, then CCSA provides
latency distribution tail. enormous benefit by allowing nominal requests
Through stochastic queuing simulation,12 we to drain past slow ones, drastically reducing
show that improving a system’s queuing behav- wait time. Importantly, we show that, with con-
ior often yields much greater benefit than miti- currency, CCSA is more effective than reducing
gating the individual system hiccups that directly either the length or the probability of
increase service time tails. We suggest two gen- rare hiccups. Since finding and mitigating tail
eral directions for improving system queuing events is hard due to their myriad causes,13 we

56 IEEE Micro
believe this observation is encouraging—we can independent of the system’s history).14 An inter-
reduce tail latency without engaging in “whack- esting property of exponentially distributed ran-
a-mole” with rare system hiccups. dom variables is the constant ratio between their
In short, we argue that cloud system designers mean values and all of their quantiles (including
should invest optimization effort first into 1) the median and all-percentile tails), as shown in
reducing HoL blocking through higher concur-
rency and improved queuing discipline (i.e., server P ðS > aEðSÞÞ ¼ ea : (1)
pooling) and then into 2) optimizing common-
case performance to improve mean service time. Due to the memoryless property of exponen-
Both of these approaches may have greater tial distributions, M=M=c queuing systems
impact and are easier to achieve can be easily analyzed with con-
than directly pinpointing and miti- tinuous time Markov chains and
Neither inter-arrival nor
gating rare cases and hiccups; service times of inter- have closed-form solutions for
whereas server pooling smooths active cloud services many of their parameters, such
out arrival and service variability, are perfectly modeled as average waiting and sojourn
CCSA reduces the effective system by exponential (waiting plus service) times.
load. The relative impact of the distributions. But since Neither inter-arrival nor service
two approaches depends criti- requests usually times of interactive cloud services
cally on the system load and ser- originate from a large are perfectly modeled by exponen-
vice time variance. CCSA’s pool of independent tial distributions. But since
effectiveness improves as service sources (e.g., many requests usually originate from a
times become more normal and/ distinct users), they
large pool of independent sources
typically mimic Poisson
or concurrency increases. We (e.g., many distinct users), they
(memoryless) arrivals.
build a simple regression model typically mimic Poisson (memory-
on concurrency and service time less) arrivals; prior studies have
variance to estimate HoL blocking and indicate observed that interarrival time dis-
whether server pooling or CCSA is more benefi- tributions usually have small coefficients of varia-
cial in reducing tail latency. System designers tion (mostly, between 1 and 2).12 As such, inter-
can use this model to guide optimization effort arrival processes can be well approximated with
and estimate its impact. an exponential distribution (CV ¼ 1) with little
fidelity loss.15 Service time distributions, in con-
trast, may have long tails; some requests encoun-
BACKGROUND AND METHODOLOGY ter rare hiccups that increase service time by
Most interactive cloud services can be mod- 10
 100
(or even more) over the mean—
eled as A=S=c queuing systems (based on much larger than the ratio of the 99th percentile
Kendall’s notation14), where A specifies the and mean values in the exponentially distributed
request inter-arrival time distribution, S the ser- services times of M=M=c systems [ 4:6, based
vice time distribution, and c the number of concur- on (1)]. Hence, interactive cloud services are
rent servers. Regardless of the distributions, the often investigated using M=G=c queuing models
average arrival rate () must be lower than the (G stands for General).5; 16
average aggregate service rate of all servers (mc, Unfortunately, M=G=c queuing models do not
with m as the average service rate of a single have closed-form solutions for average waiting/
server); otherwise, requests queue without bound. sojourn times and the accuracy of existing
The most common queuing models used in approximations, which use only a few moments,
analytical studies are M=M=c systems (M stands is poor.17 Furthermore, to the best of our knowl-
for Markovian), where both interarrival and ser- edge, there is no widely used approximation for
vice times follow exponential distributions. It can waiting/sojourn time quantiles of these systems.
be shown that the exponential distribution is the Thus, we use stochastic queuing simulation,
only continuous distribution with the memory- based on the BigHouse methodology,12 to mea-
less property (i.e., occurrence of events is sure the tail latency of such M=G=c systems. We

July/August 2019
57
 General Interest
simulate the queuing system until we achieve
95% confidence intervals of 5% error in reported
results. We consider the first-come–first-served
(FCFS) queuing discipline as prior work9; 18
shows it to be the best non-preemptive schedul-
ing policy when tail latency is the metric of
interest.
We model “nominal” request performance by
drawing service times from an exponential distri-
bution with mean 1=mn . Then, to represent rare/
slow requests, which we call “hiccups,” with prob-
ability ph , we add an additional delay drawn from
a second exponential distribution with mean
1=mh . We vary both ph and the ratio of mn =mh in
our experiments. This hybrid model is similar to
the dual-branch Hyperexponential distribution,
which is widely used as a phase-type distribution
for approximating heavy-tailed systems.14 We
study analytical distributions as they are easier to
understand and their parameters can be tuned to
model various real scenarios.
The intent of our approach is to model the
near-memoryless nominal behavior of cloud
services and then overlay an independent distri-
bution to model hiccups. We consider hiccups
that are 1) 10
longer than average, affecting 1%
of requests, and 2) 100
longer affecting 0.1%
of requests.
1) The first one represents unusual code paths
that arise in, e.g., web search. As an example,
Microsoft observes a bimodal distribution
for Bing search,6 wherein most requests
incur latencies close to the mean but occa-
sional requests require an order of magni-
tude more processing time due to their
complicated search queries. They report a
27
ratio between the 99th percentile tail
and the median latency (which is usually
smaller than the mean). Similarly, Google
reports a 1-ms median leaf service time with
99th percentile tail latency of 10 ms.4
2) This one represents rare pauses that arise
due to system activities and interference.
As an example, Wang et al.19 studied a mul-
titier web application and identified a simi-
lar bimodal distribution incorporating
rare requests with less than 1% probability
that take 30  40
longer than the mean
due to “transient events,” such as JVM
58
garbage collection or voltage/frequency
state transitions.
THE QUEUEING-FIRST APPROACH
Requests may incur an end-to-end latency in
a high percentile tail either because the request
itself incurred a rare hiccup or due to queuing
delays. Queuing greatly magnifies the impact of
few, rare hiccups by causing nominal requests
to queue behind one with a hiccup and incur
high sojourn times. With deterministic or mem-
ory-less service times, queuing arises primarily
due to request bursts, wherein the instanta-
neous arrival rate exceeds the average service
rate. However, with high disparity service time
distributions, queuing is caused mostly by HoL
blocking, wherein the instantaneous service
rate drops temporarily well below the average
request arrival rate.
The differing nature of queuing has important
implications. First, with high disparity service,
queuing can arise even at low load; when a slow
request stalls the server for a long time, many
requests may queue behind it, even if the arrival
rate is low. Second, it increases the contribution
of nominal requests to the sojourn-time tail; while
hiccups directly impact few requests, such
requests account for a large fraction of server uti-
lization. As such, a substantial fraction of nominal
requests queue behind the exceptional ones. As
an example, in an M=G=1 queue where 0.1% of
requests incur a 100
higher than nominal ser-
vice time, the exceptional requests account for
10% server utilization. As a consequence of Pois-
son arrivals, 10% of requests arrive during such
a slow service and may also contribute to the
sojourn-time tail.
Figure 1(a) and (b) reports the normalized
99th percentile tail latency of an M=M=1 system
and its M=G=1 counterpart with the high dispar-
ity service time distribution described above
across various load levels. Figure 1(c) reports
the fraction of sojourn time spent waiting by the
1% slowest requests for both M=M=1 and
M=G=1 queues. Under low loads, wait time is
usually small in M=M=1 systems and the
sojourn-time tail is nearly the same as the ser-
vice-time tail. However, queuing accounts for
a significant fraction of tail latency when the
IEEE Micro
Figure 1. (a) Normalized service- and sojourn-time 99th percentile tail in an M/M/1 queue. (b) Normalized service- and
sojourn-time 99th percentile tail in an M/G/1 queue. (c) Average % wait time in sojourn-time tail requests. (d) % of sojourn-
time tail requests that are also in the service-time tail. The M/G/1 queue has an exponential service time distribution, but
incorporates 100
hiccups that occur in 0.1% of the requests.

service time distribution is high disparity. Fur-
thermore, since hiccups occur with a low proba-
bility (0.1%), they do not noticeably affect the
service time 99th percentile tail. However, due
to the HoL blocking, their impact on the sojourn-
time tail is large under both low and high loads.
Figure 1(d) reports the percentage of
requests in the sojourn-time tail that also con-
tribute to the service-time tail. Under both
low and high loads, the percentage is much
higher in the M=M=1 system. With high dis-
parity service times, HoL blocking in the
M=G=1 system comprises the bulk of the
tail—most sojourn-time tail requests are nomi-
nal requests that queue behind exceptionally
slow ones. Furthermore, as shown in Figure 1
(c), while the fraction of queuing delay rela-
tive to sojourn time in tail requests is higher
in the M=G=1 system, queuing still accounts
for more than half of sojourn time even in
M=M=1 systems for loads over 30%.
The takeaway is that if a service incurs either
high load or has a high disparity service time dis-
tribution, end-to-end tail latency is dominated by
queuing effects. As a result, improving system
Figure 2. Scale-out versus scale-up queuing organizations.
queuing behavior is typically more effective than
seeking to directly mitigate system hiccups that
cause heavy/long tails. Finding and mitigating
system hiccups is hard. As such, we advocate
pursuing optimizations that address queuing
behavior instead.
Server Pooling
Figure 2 contrasts two different models to
compose multiple servers. In the scale-out
model, each server has a separate request queue
and a dispatcher/load balancer steers incoming
requests into different queues such that the
request arrival rate of all servers is balanced. In
the scale-up model, instead a single request
queue is shared among all servers, which each
fetch requests from the central request queue as
they become idle. This model requires synchro-
nization of the central request queue, but
improves queuing.
It can be shown that the scale-up (M=G=c)
organization always outperforms the scale-out
organization (c  M=G=1) in principle (neglect-
ing synchronization). First, in the scale-up
organization, a server will not remain idle if
there are requests wait-
ing in the central queue.
However, in scale-out
systems, a server may
remain idle if its own
queue is empty even
while other servers have
outstanding requests.
Second, when a request
takes longer than aver-
age in a scale-out organi-
zation, all the requests
behind it suffer from

July/August 2019
59
 General Interest

Figure 3. Normalized service-time (light bars) and sojourn time (dark bars) tails of an M=G=1 queue under different
scenarios. (a) 70% load, 100
hiccups affecting 0.1% of requests. (b) 70% load, 10
hiccups affecting 1% of requests.
(c) 30% load, 100
hiccups affecting 0.1% of requests.

HoL blocking delays. In contrast, in scale-up of contemporary software systems use a

architectures, requests may be serviced by any
server; stalling at one server has 1little impact
on system-wide instantaneous service rate.
Several prior studies have observed that
scale-up queuing systems outperform scale-
out organizations.9; 5 However, a large number
scale-out queuing architecture as it is easier
to implement.9 Implementing a scale-up model
across multiple machines requires remote dis-
aggregated memory accesses or a distributed
data structure, which are difficult to imple-
ment and optimize. Even within a single

60 IEEE Micro
multicore server, implementing a scale-up
model mandates either a single synchronized
data structure or a work-stealing architecture,
which incur coherence traffic and are difficult
to scale.
We refer to the practice of consolidating
c  M=G=1 servers into a single M=G=c system
as server pooling. When service-time distribu-
tions are high disparity, HoL blocking becomes
the main source of queuing delay (and tail
latency) and the gap between the two queuing
organizations grows. We argue that server pool-
ing can play a key role in resolving HoL blocking
under such service conditions and, hence, should
be pursued despite higher implementation com-
plexity. In fact, server pooling often reduces the
tail latency more than directly mitigating the rare
hiccups that cause exceptionally long service.
Figure 3 reports the normalized service/
sojourn time tail latencies in an M=G=1 system
with different service time distributions and sys-
tem loads. The leftmost red bars represent tail
latencies in the presence of rare hiccups. The
next group of blue bars show the tail latency
where the impact (i.e., duration/probability) of
hiccups has been reduced. In particular, from left
to right, these bars represent cases where hiccup
duration is halved, their occurrence probability
is halved, and where hiccups are fully eliminated.
Finally, the cluster of green bars indicate server
pooling cases with varying number of servers c.
(We discuss the orange bars later.)
Figure 3(a) considers an exponential service
time distribution with hiccups that occur 0.1% of
the time and last 100
longer than the average
service time under 70% system load. We make
three observations: First, reducing the hiccup
probability is considerably less effective at
reducing the overall tail than reducing their
duration. The intuition is that longer hiccups
cause more requests to queue and hence exacer-
bate tails more than shorter but more frequent
hiccups. Second, pooling only two servers
reduces tail latency almost as much as halving
hiccup durations. Whereas it may be challenging
to implement high-concurrency data structures
to enable a high degree of server pooling,
sharing queues across just pairs of machines
or cores is likely easier than finding and mitigat-
ing hiccups. Finally, with greater degrees of
July/August 2019
server pooling, queuing delay vanishes and the
sojourn-time tail and service-time tail match. In
such a scenario, end-to-end tail latency is even
lower than in a system with no hiccups but with-
out pooling.
Figure 3(b) reports the same results for hic-
cups 10
longer than the average occurring in
1% of requests. Whereas the general trend
matches Figure 3(a), the gap between the ser-
vice- and sojourn-time tails is noticeably smaller
even though the total service time attributable
to hiccups is the same (10
1% ¼ 100
0:1%).
As previously observed, longer hiccups intro-
duce more severe HoL blocking and cause more
nominal requests to queue behind the excep-
tional ones (despite lower hiccup probability).
Nevertheless, in Figure 3(b), pooling across only
two servers, despite hiccups, is enough to
reduce the sojourn time tail below that of a sys-
tem without server pooling and without hiccups.
Figure 3(c) considers the same service time dis-
tribution as Figure 3(a), but under lower (30%)
system load. Here, whereas queuing delays are
typically near-negligible under low load, the high
disparity service distribution nevertheless
causes HoL blocking and a significant sojourn
time tail. Interestingly, the ratio between the
sojourn- and service-time tails is much higher
than that seen in Figure 3(b) due to longer hic-
cups and higher HoL blocking, despite lower
load. Furthermore, when HoL blocking is high
but system load is low, pooling across two serv-
ers completely eliminates queuing delay.
In summary, server pooling is highly effective
in eliminating HoL blocking and reducing queu-
ing delays that otherwise arise due to rare sys-
tem hiccups. Although pooling across many
cores/machines is often challenging, encourag-
ingly, we show that pooling across as few as two
servers is often sufficient for large tail latency
reductions.
A variety of steering and scheduling techni-
ques can enable a scale-out system to more
closely approximate scale-up system behavior.
Examples include smart load-balancing schemes
that steer requests to queues based on wait time
estimates derived from metrics like queue occu-
pancy, injecting replica requests to different
queues and then cancelling the redundant
requests,4 and various work-stealing approaches
61
 General Interest

set of orange bars in Figure 3

report service and sojourn time
tails under varying degrees of
CCSA (i.e., different speedups
of common-case service time). We
observe a large benefit in Figure 3
(b), where hiccups are relatively
short and there is little HoL block-
ing; accelerating the common-case
service time by only 20% (without
affecting its tail) reduces the
sojourn time tail almost as much
as reducing the average hiccup
Figure 4. Normalized sojourn time tail latency in an M=G=1 queue (100
hiccups length by half. Doubling the ser-
in 0.1% of requests) with various degrees of server pooling and CCSA. vice rate reduces the sojourn time
tail below that of a system with no
that migrate tasks between queues.20 While hiccups or a system with a pooling degree of two.
these techniques typically fall short of an ideal In the remaining cases [see Figure 3(a) and (c)],
M=G=c system, they still drastically reduce wait CCSA has only modest impact on the sojourn time
time and HoL blocking. Further, as shown by tail, as there is more HoL blocking and insufficient
Wierman and Zwart,18 FCFS scheduling is only concurrency for requests to avoid it
best for M=G=1 queues if the service-time dis- CCSA provides greater benefit with higher con-
tribution is light-tailed. Otherwise, variants of currency (e.g., via server pooling). Even modest
processor sharing outperform FCFS in terms concurrency is sufficient to unlock CCSA’s effec-
of tail latency. Thus, should direct implemen- tiveness; rare events are unlikely to occupy multi-
tation of server pooling prove prohibitive in a ple servers at the same time, so nominal requests
particular system, time-multiplexing machines/ nearly always bypass a stalled server. As an
cores among requests may provide an alterna- example, Figure 4 considers the scenario from
tive to address queuing due to rare hiccups. Figure 3(a) but in M=G=2 and M=G=3 systems
(i.e., with server pooling). Not only is CCSA better
Common-Case Service Acceleration than mitigating hiccups, it is also better than fur-
CCSA is another general approach to improve ther increasing server pooling. We expect that
system queuing behavior. In this approach, rather CCSA will typically be easier to implement than
than seek to mitigate the rare hiccups that cause hunting down and optimizing the underlying
high service times, instead, the system designer causes of rare performance hiccups as software
deploys optimizations that accelerate common developers are already incentivized to make the
case behavior. As such, while CCSA directly common case fast. Note that this approach is
reduces average service time, it has little effect on most beneficial for service distributions wherein,
the tail of the service distribution. Conventional despite heavy/long tails, most of the system
wisdom suggests that improving average service utilization arises from nominal requests. For
time does not improve tail latency; indeed, some example, in our modeled distributions, 10% of
prior work suggests trading off slower average system utilization is spent on hiccups. However,
performance to reign in tails.7; 6 However, reduc- in many power-law distributions, tail events con-
ing average service time increases service rate, tribute to 80%–90% of the distribution; CCSA
and hence reduces server utilization. Reduced uti- would not be as effective in such scenarios.
lization in turn reduces queuing delays.
Unlike server pooling, CCSA has little impact Discussion
when HoL blocking is high, as nominal requests We expect CCSA to be more beneficial than
enqueue behind long ones regardless of how fast server pooling, as CCSA reduces the effective sys-
nominal requests are processed. The rightmost tem load, while server pooling has no effect on

62 IEEE Micro
load/utilization. However, as we showed in the memoryless around CVdeparture ¼ 1:0, where the
previous section, CCSA is only effective in the ratio of tail-to-average cases does not decrease
absence of server HoL blocking. When HoL block- through higher concurrency [see (1)]. As a result,
ing is frequent (e.g., service time variance is high), we suggest a regression model of the form of
CCSA no longer provides benefit as nominal
requests queue behind exceptionally long ones. In CVdeparture  ðCVservice  1Þe0:8ðc1Þ þ 1 (2)
such scenarios, additional concurrency must be
introduced to unleash CCSA’s efficacy. and tune its parameter using the least squares
In single-server systems, service time variabil- method. We find its average error to be less than
ity is a good measure of HoL blocking. For exam- 13%.
ple, in Figure 3(a), where CVservice ¼ 4:2, CCSA has Using this model, we can derive CVdeparture as
negligible impact; nominal requests wait behind a proxy for the HoL blocking rate and predict
slow ones. In contrast, in Figure 3(b) where how it is affected by server pooling. Alterna-
CVservice ¼ 1:6 (near the CVservice ¼ 1:0 of M=M= tively, cloud system architects may perform
queues), CCSA is more effective than server Stochastic Queueing Simulations, similar to our
pooling. However, CVservice only approach, and directly measure
reflects HoL blocking in single- We recommend devel- CVdeparture instead of predicting it.
server systems. We suggest the opers follow a simple When the system approaches the
interdeparture time variability of optimization sequence CVdeparture ¼ 1:0 of M=M= queues,
a saturated queue (when queuing to address tail latency in blocking becomes rare; the
probability is close to 1.0) to their services: 1) intro- remaining tail of the sojourn time
measure HoL blocking in multi- duce server pooling distribution is then primarily due
server queues. In saturated until HoL blocking is suf- to service time tails or high load.
single-server queues, the inter- ficiently mitigated; 2) if Under low load, queuing delays
load is high, introduce vanish with sufficient server pool-
departure time distribution is
CCSA; 3) if end-to-end
the service time distribution ing; remaining sojourn time tails
tails remain unaccept-
(CVservice ¼ CVdeparture ). However, reflect only service tails. Under
able, only then seek to
with multiple servers, departures high load, HoL blocking will no
directly optimize rare,
interleave, reducing interdepar- high service latencies. longer be the dominant source of
ture time variability. For example, queuing delays when sufficient
in Figure 4, which is similar to concurrency has been introduced.
Figure 3(a) but with an additional 1–2 servers, the As such, with sufficient server pooling (often
CVdeparture drops (from 3.0) to 1.7 and 1.1, respec- just 2–3 servers), CCSA becomes more effective
tively. As a result, in the M=G=2 case, CCSA yields than further server pooling.
almost the same benefit as pooling. In the M=G=3 In short, we recommend developers follow a
case, where HoL blocking resembles that of an simple optimization sequence to address tail
M=M= queue with CVdeparture ¼ 1:0, CCSA yields latency in their services: 1) introduce server pool-
much better results than server pooling. ing until HoL blocking is sufficiently mitigated; 2)
We find that a simple regression model can if load is high, introduce CCSA; 3) if end-to-end
predict the CVdeparture of a saturated M=G=c queue tails remain unacceptable, only then seek to
based on its CVservice and the number of servers directly optimize rare, high service latencies.
(c). We construct the model by simulating satu-
rated queues with a set of high disparity distribu- CONCLUSION
tions with different CVservice and measure their Improving a system’s queuing behavior often
CVdeparture . We observe that small degrees of yields much greater benefit than mitigating the
server pooling quickly reduce HoL blocking. individual system hiccups that increase service
Therefore, we postulate an exponential decay time tails. We suggest two general directions
effect for the number of servers. Also, we for improving system queuing behavior—server
note that CVdeparture may not decrease below pooling, and CCSA—which synergistically address
1.0 as the interdeparture process becomes near- queuing behaviors that often drive tail latency.

July/August 2019
63
 General Interest
ACKNOWLEDGMENTS
This work was supported by the Center for
Applications Driving Architectures, one of six
centers of JUMP, a Semiconductor Research Cor-
poration program co-sponsored by DARPA.
& REFERENCES
1. L. A. Barroso, J. Dean, and U. Holzle, “Web search for
a planet: The Google cluster architecture,” IEEE Micro,
vol. 23, no. 2, pp. 22–28, Mar./Apr. 2003.
2. S. Kanev et al., “Profiling a warehouse-scale computer,”
in Proc. ACM/IEEE 42nd Annu. Int. Symp. Comput.
Archit., 2015, pp. 158–169.
3. C. Delimitrou and C. Kozyrakis, “Amdahl’s law for tail
latency,” Commun. ACM, vol. 61, no. 8, pp. 65–72, 2018.
4. J. Dean and L. A. Barroso, “The tail at scale,” Commun.
ACM, vol. 56, no. 2, pp. 74–80, 2013.
5. H. Kasture and D. Sanchez, “Tailbench: A benchmark
suite and evaluation methodology for latency-critical
applications,” in Proc. IEEE Int. Symp. Workload
Characterization., 2016, pp. 1–10.
6. M. E. Haque et al., “Few-to-many: Incremental parallelism
for reducing tail latency in interactive services,” ACM
SIGPLAN Notices, vol. 50, no. 4, pp. 161–175, 2015.
7. C.-H. Hsu et al., “Adrenaline: Pinpointing and reining in
tail queries with quick voltage boosting,” in Proc. IEEE
21st Int. Symp. High Perform. Comput. Archit., 2015,
pp. 271–282.
8. R. Kapoor, G. Porter, M. Tewari, G. M. Voelker, and
A. Vahdat, “Chronos: Predictable low latency for data
center applications,” in Proc. 3rd ACM Symp. Cloud
Comput., 2012, Art. no. 9.
9. J. Li, N. K. Sharma, D. R. Ports, and S. D. Gribble,
“Tales of the tail: Hardware, OS, and application-level
sources of tail latency,” in Proc. ACM Symp. Cloud
Comput., 2014, pp. 1–14.
10. D. Lo, L. Cheng, R. Govindaraju, P. Ranganathan, and
C. Kozyrakis, “Heracles: improving resource efficiency
at scale,” in Proc. ACM SIGARCH Comput. Archit.
News, 2015, vol. 43, no. 3, pp. 450–462.
11. A. Mirhosseini, A. Sriraman, and T. F. Wenisch,
“Enhancing server efficiency in the face of killer
microseconds,” in Proc. IEEE 25th Int. Symp. High
Perform. Comput. Archit., 2019, pp. 185–198.
12. D. Meisner, J. Wu, and T. F. Wenisch, “Bighouse: A
simulation infrastructure for data center systems,” in
Proc. IEEE Int. Symp. Perform. Anal. Syst. Softw.,
2012, pp. 35–45.
64
13. X. Yang, S. M. Blackburn, and K. S. McKinley,
“Computer performance microscopy with shim,” in
Proc. ACM/IEEE 42nd Annu. Int. Symp. Comput.
Archit., 2015, pp. 170–184.
14. M. Harchol-Balter, Performance Modeling and
Design of Computer Systems: Queueing Theory in
Action. Cambridge, U.K.: Cambridge Univ. Press,
2013.
15. D. Meisner, C. M. Sadler, L. A. Barroso, W.-D. Weber,
and T. F. Wenisch, “Power management of online
data-intensive services,” in Proc. 38th Annu. Int. Symp.
Comput. Archit., 2011, pp. 319–330.
16. D. Meisner, B. T. Gold, and T. F. Wenisch, “Powernap:
Eliminating server idle power,” ACM Sigplan Notices,
vol. 44, no. 3, pp. 205–216, 2009.
17. V. Gupta, M. Harchol-Balter, J. Dai, and B. Zwart, “On
the inapproximability of m/g/k: Why two moments of
job size distribution are not enough,” Queueing Syst.,
vol. 64, no. 1, pp. 5–48, 2010.
18. A. Wierman and B. Zwart, “Is tail-optimal scheduling
possible?” Oper. Res., vol. 60, no. 5, pp. 1249–1257, 2012.
19. Q. Wang et al., “Detecting transient bottlenecks in n-tier
applications through fine-grained analysis,” in Proc.
IEEE 33rd Int. Conf. Distrib. Comput. Syst., 2013,
pp. 31–40.
20. J. Li et al., “Work stealing for interactive services to
meet target latency,” ACM SIGPLAN Notices, vol. 51,
no. 8, p. 14, 2016.
Amirhossein Mirhosseini is currently working
toward a PhD in computer science and engineering at
the University of Michigan. His research interests cen-
ter on computer architecture with particular emphasis
on data centers, cloud computing, and microservices.
He is a student member of the IEEE and the Associa-
tion for Computing Machinery (ACM). Contact him at
miramir@umich.edu.
Thomas F. Wenisch is an associate professor of
electrical engineering and computer science and
Associate Chair of External Affairs at the University of
Michigan, Ann Arbor. His research interests center on
computer architecture with particular emphasis on
server and data center systems, memory persistency,
multiprocessor systems, and performance evaluation
methodology. He has a PhD in electrical and computer
engineering from Carnegie Mellon University. He is a
member of the IEEE and the Association for Computing
Machinery (ACM). Contact him at twenisch@umich.
edu.
IEEE Micro
 Column: Micro Economics

The Aftermath of the Dyn DDOS Attack

Shane Greenstein
Harvard Business School

& NOBODY KNOWS WHO organized the attack. tells us something about the challenges facing
It might have come from an angry gamer, or suppliers, and in this case, it tells us about a
from a rogue spy, or, perhaps, an angry rogue basic challenge in network security today. It will
spy playing games. The program hijacked many take a bit of work to appreciate the lesson, and,
cameras and home devices, and redirected them let me tip my hand, the news is not good.
to engineer a series of distributed denial of The article provides a summary of a longer
server (DDOS) attacks on a few hours apart, all study done by a group of my colleagues and
on October 21, 2016. By executing this novel and myself.1
rather clever hijack of many devices for a DDOS What happened?
attack, the attack exposed an important vulnera- Start with the basics. A website owner can set
bility in today’s internet. up its own name resolver, or hire somebody to
The attack contains one other do it for them. Many years ago most firms did it
element. It aimed at Dyn, who acts themselves. Like a lot of things
as a name resolver. Dyn enables By executing this novel on the internet, over time a set of
Internet traffic by translating the and rather clever hijack professional firms emerged,
site’s domain name (URL) into the of many devices for a while many technically sophisti-
IP address where the server behind DDOS attack, the cated firms still perform this
that domain is to be found. During attack exposed an function for themselves.
the later phases of the attack, Dyn important vulnerability What do name resolvers do?
in today’s internet. Start with the basics. It is a
servers were unable to process
users’ requests, and as a result, mouthful, but we need to under-
users lost access to web domains contracting stand it to understand what happened to Dyn.
with Dyn, such as Netflix, CNBC, and Twitter. When an application (such as a web browser)
Other well-known firms also were disabled, such wants to access a page or resource located at a
as Airbnb, Etsy, Play Station Network, and Wikia. known domain name, it can access DNS records
This article focuses on the aftermath of this and a corresponding IP address. In principle, the
event, which did not get headlines, but illus- application submits a request to a DNS “resolver”
trates an important features of the situation. asking for the IP address corresponding to a given
Specifically, how did users react? User behavior domain name. The resolver queries a root name-
server, which replies with the corresponding to
the TLD nameserver specified by the domain
Digital Object Identifier 10.1109/MM.2019.2919886 name (e.g., “.com”). The resolver then queries
Date of current version 23 July 2019. that TLD nameserver with the second component

0272-1732 ß 2019 IEEE Published by the IEEE Computer Society IEEE Micro
66
Figure 1. Market share for largest name servers.
of the domain name (e.g., “google”). The TLD
nameserver retrieves that domain’s authoritative
nameservers (e.g., “ns1.google.com”) and returns
them to the resolver. Finally, the resolver queries
one of the authoritative nameservers and receives
a usable IP address for the domain. The IP address
is passed back to the original application, which
can use it to connect to the desired host. This
entire process generally takes just milliseconds.
In practice, many users request the same thing
repeatedly, so it is possible to cache the answers
to many of the intermediate steps, and that can
speed up the resolution even more. More to the
point, caching across many servers acts as a
quasi-buffer against a DDOS attack, especially if
the attack can be rebuffed in a short time.
The emergence of numerous lessons and auto-
mated standard practices has gone hand-in-hand
with the emergence of professional firms. Such
firms know how to provide services at scale. In
turn, and as with many other professional mar-
kets, some of them became good at their service,
and that performance attracted many customers.
Thus emerged an ironic outcome. While the
internet contains many points of resiliency, the
increasing concentration of services in a small
number of providers has created concentrated
points of failure. To say it another way, Dyn per-
formed approximately 10% of nameserver serv-
ices in the United States (prior to the attack), so
bringing it down could bring down 10% of the
internet’s servers. Since name resolution also
supports a range of other communications, such
July/August 2019
as email, it could also cripple communications
at many firms. That is a big deal.
As should be obvious, the potential economic
losses could be enormous. Many businesses,
especially internet businesses, lose a lot of reve-
nue from being down for a day.
That motivates a basic question: after this
attack, what did users do? It did not take much
professional experience to understand the vul-
nerability or the lessons. Businesses were vul-
nerable to a single point of failure. A website
could construct a form of insurance by perform-
ing a simple act: maintaining multiple name serv-
ers with multihoming.
We set out to find out two things. First, how
many firms maintain multihoming? Second, how
did the use of resolvers change after the Dyn
attack? Figure 1, which is taken from Figure 7 of
our study,1 tells a big part of the answer we
found. This figure shows the market shares for
the largest providers of nameservers between
late 2011 and the middle of 2017.
What did firms do?
Figure 1 shows that, long before the Dyn attack,
name servers had embarked on a general trend
toward more concentration. The growth of three
firms—Dyn, AWS, and Cloudflare—drove this
trend. Dyn’s growth had already begun to level off
by 2014, while AWS and Cloudflare have continued
to grow unabated throughout the time period.
The figure actually does not tell the entire
story. We did a little investigation, and found
that both AWS and Cloudfare attract a high
67
 Micro Economics

fraction of the contracts from newly founded from customers of Dyn and somewhat from Neu-
firms. Older firms tend to use others. star (another large provider). No other provider
We also found that, as expected, over time saw a big change.
an increasing fraction of users contracted with You might reasonably reply that Cloudflare’s
providers instead of doing it themselves. Close to security model makes it difficult to multihome,
60% do it themselves in 2011, while close to 30% but does protect against this type of attack. Which
do it in 2017. In other words, the market is true. But what accounts for so little with all the
shares rise during a time when the number of cus- others? Most users act as if they do not care.
tomers practically doubled.
The figure also illustrates the big surprise.
CONCLUSION
The attack on Dyn had consequences for its
Let us summarize. There was a DDOS attack
commercial success. Many of its users seem to
on Dyn. It demonstrated a new vulnerability, and
have blamed Dyn for the down-
we are still vulnerable to another
time, and shifted to another pro-
The least expensive similar attack. In fact, we may be
vider. Within a couple months
insurance against more vulnerable now that bad
Dyn lost a quarter of its custom-
another attack is multi- actors have watched the prior
ers. Ouch.
homing with more than demonstration.
But wait, that is not the only one resolver. Let us be How did the user community
surprise. It is also important to clear about it: It is not act? The two big headlines are: 1)
notice what did not happen: We did expensive. It is just a a fraction of customers acted as if
not find much multihoming after minor hassle to main-
they blamed Dyn, and took precau-
the Dyn attack. Around 11% of tain multiple suppliers.
tions; and 2) all but a small frac-
existing domains multihomed
tion of non-Dyn customers did not
prior to the Dyn attack, and about 18% did so after
act as if they learned any lesson.
the attack. New domains multihomed at a rate of
I do not know about you, but I do not feel any
less than 5% prior to the attack, and after the
safer. Sigh.
attack about 8% did. In short, there was an uptick
in multihoming, but the vast majority of sites con-
tinued to use a single provider. & REFERENCE
Why does that matter? The least expensive
1. S. Bates, J. Bowers, S. Greenstein, J. Weinstock, and
insurance against another attack is multihoming
J. Zittrain, “In support of internet entropy: Mitigating an
with more than one resolver. Let us be clear
increasingly dangerous lack of redundancy in DNS
about it: It is not expensive. It is just a minor has-
resolution by major websites and services,” NBER
sle to maintain multiple suppliers. (And, more-
working paper 24317, 2018. [Online]. Available at
over, many aspects of web administration are no
SSRN: https://ssrn.com/abstract¼3241740
worse a hassle. That is the job, after all.)
Not illustrated in the figure are minutia of mar-
ket shares. In case you were wondering, most of Shane Greenstein is a professor at the Harvard
the multihoming among existing websites came Business School. Contact him at sgreenstein@hbs.edu.

68 IEEE Micro
PURPOSE: The IEEE Computer Society is the world’s largest
association of computing professionals and is the leading provider
of technical information in the field.
MEMBERSHIP: Members receive the monthly magazine
Computer, discounts, and opportunities to serve (all activities
are led by volunteer members). Membership is open to all IEEE
members, affiliate society members, and others interested in the
computer field.
COMPUTER SOCIETY WEBSITE: www.computer.org
OMBUDSMAN: Direct unresolved complaints to
ombudsman@computer.org.
CHAPTERS: Regular and student chapters worldwide provide the
opportunity to interact with colleagues, hear technical experts,
and serve the local professional community.
AVAILABLE INFORMATION: To check membership status, report
an address change, or obtain more information on any of the
following, email Customer Service at help@computer.org or call
+1 714 821 8380 (international) or our toll-free number,
+1 800 272 6657 (US):
• Membership applications
• Publications catalog
• Draft standards and order forms
• Technical committee list
• Technical committee application
• Chapter start-up procedures
• Student scholarship information
• Volunteer leaders/staff directory
• IEEE senior member grade application (requires 10 years
practice and significant performance in five of those 10)
PUBLICATIONS AND ACTIVITIES
Computer: The flagship publication of the IEEE Computer Society,
Computer, publishes peer-reviewed technical content that
covers all aspects of computer science, computer engineering,
technology, and applications.
Periodicals: The society publishes 12 magazines, 15 transactions,
and two letters. Refer to membership application or request
information as noted above.
Conference Proceedings & Books: Conference Publishing
Services publishes more than 275 titles every year.
Standards Working Groups: More than 150 groups produce IEEE
standards used throughout the world.
Technical Committees: TCs provide professional interaction in
more than 30 technical areas and directly influence computer
engineering conferences and publications.
Conferences/Education: The society holds about 200 conferences
each year and sponsors many educational activities, including
computing science accreditation.
Certifications: The society offers three software developer
credentials. For more information, visit
www.computer.org/certification.
2019 BOARD OF GOVERNORS MEETINGS
‫(ژ‬TBD) ‚ƩɋȏƨƷȵ: Teleconference
revised 1‫ ׮‬eɓǹɲ 2019
EXECUTIVE COMMITTEE
President: Cecilia Metra
President-Elect: Leila De Floriani
Past President: Hironori Kasahara
First VP: Forrest Shull; Second VP: Avi Mendelson;
Secretary: David Lomet; Treasurer: Dimitrios Serpanos;
VP, Member & Geographic Activities: Yervant Zorian;
VP, Professional & Educational Activities: Kunio Uchiyama;
VP, Publications: Fabrizio Lombardi; VP, Standards Activities:
Riccardo Mariani; VP, Technical & Conference Activities:
William D. Gropp
2018–2019 IEEE Division V Director: John W. Walz
2019 IEEE Division V Director Elect: Thomas M. Conte
2019–2020 IEEE Division VIII Director: Elizabeth L. Burd
BOARD OF GOVERNORS
¾ƷȵȂ‫ژ‬-ɱȲǠȵǠȄǒ‫ژيח׏׎אژ‬°ƌɓȵƌƨǚ‫ژ‬ƌǒƩǚǠً‫ژ‬GȵƷǒȏȵɲ‫ژِ¾ژ‬ɲȵưً‫ژ‬
%ƌɫǠư‫ژ‬°ِ‫ژ‬-ƨƷȵɋً‫ژ‬eǠǹǹ‫ژ‬Uِ‫ژ‬GȏȽɋǠȄً‫ژ‬ÞǠǹǹǠƌȂ‫ژ‬GȵȏȲȲً‫ژ‬°ɓȂǠ‫ژ‬OƷǹƌǹ
¾ƷȵȂ‫ژ‬-ɱȲǠȵǠȄǒ‫
ژي׎א׎אژ‬Ȅưɲ‫ژِ¾ژ‬ǚƷȄً‫ژ‬eȏǚȄ‫ژ‬%ِ‫ژ‬eȏǚȄȽȏȄً‫ژ‬
°ɲ‫ٮ‬äƷȄ‫ژ‬hɓȏً‫ژ‬%ƌɫǠư‫ژ‬kȏȂƷɋً‫ژ‬%ǠȂǠɋȵǠȏȽ‫ژ‬°ƷȵȲƌȄȏȽً‫ژ‬Oƌɲƌɋȏ‫ژ‬äƌȂƌȄƌ‫ژ‬
¾ƷȵȂ‫ژ‬-ɱȲǠȵǠȄǒ‫ژي׏א׎אژ‬uِ‫ژ‬ȵǠƌȄ‫ژ‬ǹƌǵƷً‫ژ‬FȵƷư‫ژ‬%ȏɓǒǹǠȽً‫ژ‬
ƌȵǹȏȽ‫ژ‬-ِ‫ژ‬eǠȂƷȄƷɼ‫ٮ‬GȏȂƷɼً‫¨ژ‬ƌȂƌǹƌɋǚƌ‫ژ‬uƌȵǠȂɓɋǚɓً‫ژ‬
-ȵǠǵ‫ژ‬eƌȄ‫ژ‬uƌȵǠȄǠȽȽƷȄً‫ژ‬hɓȄǠȏ‫ژ‬ÅƩǚǠɲƌȂƌ
EXECUTIVE STAFF
Executive Director: Melissa
ِ‫ژ‬Russell
Director, Governance & Associate Executive Director:
Anne Marie Kelly
Director, Finance & Accounting: Sunny Hwang
Director, Information Technology & Services: Sumit Kacker
Director, Marketing & Sales: Michelle Tubb
Director, Membership Development: Eric Berkowitz
COMPUTER SOCIETY OFFICES
Washington, D.C.: 2001 L St., Ste. 700, Washington, D.C.
20036-4928ٕ Phone: +1 202 371 0101ٕ Fax: +1 202 728 9614ٕ‫ژ‬
Email: hq.ofc@computer.org
Los Alamitos: 10662 Los Vaqueros Cir., Los Alamitos, CA 90720ٕ‫ژ‬
Phone: +1 714 821 8380ٕ Email: help@computer.org
Asia/Pacific: Watanabe Building, 1-4-2 Minami-Aoyama,
Minato-ku, Tokyo 107-0062, Japanٕ Phone: +81 3 3408 3118ٕ
Fax: +81 3 3408 3553ٕ Email: tokyo.ofc@computer.org
u-u-¨°OU¥‫ژۯژ‬¥ÅkU
¾U‚w‫¨‚ژ‬%-¨°‫ژ‬
¥ǚȏȄƷ‫ژٕודההژאואژ׎׎זژ׏ڹژي‬Fƌɱ‫ژٕ׏גהגژ׏אזژג׏וژ׏ڹژي‬
-ȂƌǠǹ‫ژي‬ǚƷǹȲ‫ۮ‬ƩȏȂȲɓɋƷȵِȏȵǒ
IEEE BOARD OF DIRECTORS
President & CEO: Jose M.D. Moura
President-Elect: Toshio Fukuda
Past President: James A. Jefferies
Secretary: Kathleen Kramer
Treasurer: Joseph V. Lillie
Director & President, IEEE-USA: Thomas M. Coughlin‫ژ‬
Director & President, Standards Association: Robert S. Fish‫ژ‬
Director & VP, Educational Activities: Witold M. Kinsner‫ژ‬
Director & VP, Membership and Geographic Activities:
Francis B. Grosz, Jr.
Director & VP, Publication Services & Products: Hulya Kirkici‫ژ‬
Director & VP, Technical Activities: K.J. Ray Liu