VOLUME 40, NUMBER 3 MAY/JUNE 2020

Top Picks

www.computer.org/micro
IEEE
Computer
Society Has
You Covered!
WORLD-CLASS CONFERENCES — Stay
ahead of the curve by attending one of our
200+ globally recognized conferences.

DIGITAL LIBRARY — Easily access over 780k

articles covering world-class peer-reviewed
content in the IEEE Computer Society
Digital Library.

CALLS FOR PAPERS — Discover

opportunities to write and present your
ground-breaking accomplishments.

EDUCATION — Strengthen your resume

with the IEEE Computer Society Course
Catalog and its range of offerings.

ADVANCE YOUR CAREER — Search the

new positions posted in the IEEE Computer
Society Jobs Board.

NETWORK — Make connections that count

by participating in local Region, Section,
and Chapter activities.

Explore all of the member benefits

at www.computer.org today!
EDITOR-IN-CHIEF IEEE MICRO STAFF
Lizy K. John, University of Texas, Austin Journals Production Manager: Joanna Gojlik,
j.gojlik@ieee.org
EDITORIAL BOARD Peer-Review Administrator:
micro-ma@computer.org
R. Iris Bahar, Brown University
Publications Portfolio Manager: Kimberly Sperka
Mauricio Breternitz, University of Lisbon
Publisher: Robin Baldwin
David Brooks, Harvard University
Senior Advertising Coordinator: Debbie Sims
Bronis de Supinski, Lawrence Livermore
IEEE Computer Society Executive Director:
Nation al Lab
Melissa Russell
Shane Greenstein, Harvard Business School
Natalie Enright Jerger, University of Toronto IEEE PUBLISHING OPERATIONS
Hyesoon Kim, Georgia Institute of Technology
John Kim, Korea Advanced Institute of Science Senior Director, Publishing Operations:
and Technology Dawn Melley
Hsien-Hsin (Sean) Lee, Taiwan Semiconductor Director, Editorial Services: Kevin Lisankie
Manufacturing Company Director, Production Services: Peter M. Tuohy
Richard Mateosian Associate Director, Editorial Services:
Tulika Mitra, National University of Singapore Jeffrey E. Cichocki
Trevor Mudge, University of Michigan, Ann Arbor Associate Director, Information Conversion
Onur Mutlu, ETH Zurich and Editorial Support: Neelam Khinvasara
Vijaykrishnan Narayanan, The Pennsylvania Senior Art Director: Janet Dudar
State University Senior Manager, Journals Production:
Per Stenstrom, Chalmers University of Patrick Kempf
Technology CS MAGAZINE OPERATIONS COMMITTEE
Richard H. Stern, George Washington
University Law School Sumi Helal (Chair), Irena Bojanova,
Sreenivas Subramoney, Intel Corporation Jim X. Chen, Shu-Ching Chen,
Carole-Jean Wu, Arizona State University Gerardo Con Diaz, David Alan Grier,
Lixin Zhang, Chinese Academy of Sciences Lizy K. John, Marc Langheinrich, Torsten Möller,
David Nicol, Ipek Ozkaya, George Pallis,
ADVISORY BOARD VS Subrahmanian
David H. Albonesi, Erik R. Altman, Pradip Bose, CS PUBLICATIONS BOARD
Kemal Ebcioglu, Lieven Eeckhout,
Fabrizio Lombardi (VP for Publications),
Michael Flynn, Ruby B. Lee, Yale Patt,
Alfredo Benso, Cristiana Bolchini,
James E. Smith, Marc Tremblay
Javier Bruguera, Carl K. Chang, Fred Douglis,
Subscription change of address: Sumi Helal, Shi-Min Hu, Sy-Yen Kuo,
address.change@ieee.org Avi Mendelson, Stefano Zanero, Daniel Zeng
Missing or damaged copies:
COMPUTER SOCIETY OFFICE
help@computer.org
IEEE MICRO
c/o IEEE Computer Society
10662 Los Vaqueros Circle
Los Alamitos, CA 90720 USA +1 (714) 821-8380

IEEE Micro (ISSN 0272-1732) is published bimonthly by the IEEE Computer Society. IEEE Headquarters, Three Park Ave., 17th Floor, New York,
NY 10016-5997; IEEE Computer Society Headquarters, 2001 L St., Ste. 700, Washington, DC 20036; IEEE Computer Society Publications Office,
10662 Los Vaqueros Circle, PO Box 3014, Los Alamitos, CA 90720. Postmaster: Send address changes and undelivered copies to IEEE, Member-
ship Processing Dept., 445 Hoes Ln., Piscataway, NJ 08855. Periodicals postage is paid at New York, NY, and at additional mailing offices. Canadian
GST #125634188. Canada Post Corp. (Canadian distribution) Publications Mail Agreement #40013885. Return undeliverable Canadian addresses
to 4960-2 Walker Road; Windsor, ON N9A 6J3. Printed in USA. Reuse rights and reprint permissions: Educational or personal use of this material is
permitted without fee, provided such use: 1) is not made for profit; 2) includes this notice and a full citation to the original work on the first page of the
copy; and 3) does not imply IEEE endorsement of any third-party products or services. Author and their companies are permitted to post the accepted
version of IEEE-copyrighted material on their own webservers without permission, provided that the IEEE copyright notice and a full citation to the
original work appear on the first screen of the posted copy. An accepted manuscript is a version which has been revised by the author to incorporate
review suggestions, but not the published version with copy-editing, proofreading, and formatting added by IEEE. For more information, please go to
ieee.org/publications_standards/publications/rights/paperversionpolicy.html. Permission to reprint/republish this material for commercial, advertising,
or promotional purposes or for creating new collective works for resale or redistribution must be obtained from IEEE by writing to the IEEE Intellectual
Property Rights Office, 445 Hoes Lane, Piscataway, NJ 08854-4141 or pubs-permissions@ieee.org. ©2020 by IEEE. All rights reserved. Abstracting
and library use: Abstracting is permitted with credit to the source. Libraries are permitted to photocopy for private use of patrons, provided the per-copy
fee indicated in the code at the bottom of the first page is paid through the Copyright Clearance Center, 222 Rosewood Drive, Danvers, MA 01923.
Editorial: Unless otherwise stated, bylined articles, as well as product and service descriptions, reflect the author’s or firm’s opinion. Inclusion in IEEE
Micro does not necessarily constitute an endorsement by IEEE or the Computer Society. All submissions are subject to editing for style, clarity, and
space. IEEE prohibits discrimination, harassment, and bullying. For more information, visit ieee.org/web/aboutus/whatis/policies/p9-26.html.

Published by the IEEE Computer
Society
Special Issue
Guest Editor’s Introduction
6 T he 2019 Top Picks in
Computer Architecturee
Hyesoon Kim
Theme Articles
10 U nveiling the Hardware
and Software Implica-
tions of Microservices in
Cloud and Edge Systems
Yu Gan, Yanqi Zhang, Dailun Cheng,
Ankitha Shetty, Priyal Rathi,
Nayan Katarki, Ariana Bruno, Justin Hu,
Brian Ritchken, Brendon Jackson,
Kelvin Hu, Meghna Pancholi, Yuan He,
Brett Clancy, Chris Colen, Fukang Wen,
Catherine Leung, Siyuan Wang,
Leon Zaruvinsky, Mateo Espinosa, Rick Lin,
Zhongling Liu, Jake Padilla, and
Christina Delimitrou
20 M AESTRO: A Data-Centric
Approach to Understand
Reuse, Performance, and
Hardware Cost of DNN
Mappings
Hyoukjun Kwon, Prasanth Chatarasi,
Vivek Sarkar, Tushar Krishna,
Michael Pellauer, and Angshuman Parashar
30 E nergy-Efficient Video
Processing for Virtual
Reality
Yue Leng, Jian Huang, Chi-Chun Chen,
Qiuyue Sun, and Yuhao Zhu
37 T
owards General-Purpose
Acceleration: Finding
Structure in Irregularity
Vidushi Dadu, Jian Weng, Sihao Liu,
and Tony Nowatzki
47 V arifocal Storage:
Dynamic Multiresolution
Data Storage
Yu-Ching Hu, Murtuza Lokhandwala,
Te I, and Hung-Wei Tseng
May/June 2020
Volume 40 Number 3
56 A smDB: Understanding
and Mitigating Front-End
Stalls in Warehouse-Scale
Computers
Nayana Prasad Nagendra, Grant Ayers,
David I. August, Hyoun Kyu Cho,
Svilen Kanev, Christos Kozyrakis, Trivikram
Krishnamurthy, Heiner Litz, Tipp Moseley,
and Parthasarathy Ranganathan
64 E xtending the Frontier of
Quantum Computers With
Qutrits
Pranav Gokhale, Jonathan M. Baker,
Casey Duckering, Frederic T. Chong,
Natalie C. Brown, and Kenneth R. Brown
73 A rchitecting Noisy
Intermediate-Scale
Quantum Computers:
A Real-System Study
Prakash Murali, Norbert M. Linke, Margaret
Martonosi, Ali Javadi Abhari, Nhung Hong
Nguyen, and Cinthia Huerta Alderete
81 S peculative Taint Tracking
(STT): A Comprehensive
Protection for Speculatively
Accessed Data
Jiyong Yu, Mengjia Yan, Artem Khyzha,
Adam Morrison, Josep Torrellas,
and Christopher W. Fletcher
91 M icroScope: Enabling
Microarchitectural
Replay Attacks
Dimitrios Skarlatos, Mengjia Yan,
Bhargava Gopireddy, Read Sprabery,
Josep Torrellas, and Christopher W. Fletcher
99 C reating Foundations for
Secure Microarchitec-
tures With Data-Oblivious ISA
Extensions
Jiyong Yu, Lucas Hsiung, Mohamad El Hajj,
and Christopher W. Fletcher
108 T race Wringing for Program
Trace Privacy
Deeksha Dangwal, Weilong Cui, Joseph
McMahan, and Timothy Sherwood
 Image credit: Image licensed by Ingram Publishing.

COLUMNS AND DEPARTMENTS

From the Editor-in-Chief
4 Enjoy These Top Picks,
While You Work From Home!
Lizy Kurian John

Micro Economics
118 Pandemics and the Dismal
Technology Economy
Shane Greenstein
 From the Editor-in-Chief

Enjoy These Top Picks, While

You Work From Home!
Lizy Kurian John
University of Texas at Austin

& THE NOVEL CORONAVIRUS has taken all of us to top architecture conferences during 2019 was
unchartered territories in our lives. However, as eligible to compete for the Top Picks honor. In
computer engineers and scientists, we can be total, 96 submissions were received, from which
proud of the integral contributions computers 12 articles were chosen to represent the cream
play in getting everybody connected as lock- of the crop of 2019.
downs and shelter-in-place isolations are taking Professor Hyesoon Kim of Georgia Tech
place around the world! The importance of chaired this year’s selection committee. Hyesoon
secure, low-power, and high-performance chips and 28 experts from academia and industry
and systems cannot worked hard to identify 12 Top Picks and 14
be overemphasized at The importance of Honorable Mention articles. An article recog-
this time. Microproc- secure, low-power, and nized as a Top Pick was invited to prepare a sub-
essors and microsys- high-performance mission for inclusion in this special issue. The
tems are increasingly chips and systems articles in this special issue are intended to be
relevant in everyday cannot be overempha- for a broader audience than the original confer-
lives as well as com- sized at this time. ence articles. These articles also focus more on
puting for medical Microprocessors and their potential impact. The honorable mentions
microsystems are
discoveries. are high-quality articles that unfortunately could
increasingly relevant in
While the pan- not be included in the special issue due to space
everyday lives as well
demic has taken us to as computing for constraints. They are listed in the Guest Editor’s
unfamiliar territories, medical discoveries. Introduction. Interested readers can locate them
IEEE Micro is present- in the original conference proceedings or the
ing to you the very IEEE/ACM Digital Library.
familiar IEEE Micro Top Picks issue. For more The purpose of the Top Picks issue has been
than a decade, IEEE Micro has had this tradition multifold. First and foremost, Top Picks was origi-
of evaluating articles from the previous year’s nally instituted to present the “best of the best” of
architecture conferences and selecting those the preceding year’s architecture research contri-
with the most novelty and potential for long- butions to a broader audience, including industry
term impact. IEEE Micro is upholding the tradi- and other fields. A second goal of Top Picks is to
tion this year as well. Any article published in recognize excellent research in the field and
bestow this honor on researchers who conducted
Digital Object Identifier 10.1109/MM.2020.2993184 the outstanding research that resulted in these
Date of current version 22 May 2020. articles. It is critically important for our field to

0272-1732 ß 2020 IEEE Published by the IEEE Computer Society IEEE Micro
4
honor our budding researchers and help them to
shape their careers. The Top Picks honor has been
seen to be instrumental in achieving faculty posi-
tions, leading research positions in industry, and
prestigious research grants. Third, the writing style
intended for a broader audience makes it easy for
beginner graduate students to understand the
state-of-the-art of the field as they are pondering on
topics to work for their doctorate degrees Above
all, I expect these articles to be enjoyable reads for
all the readers of IEEE Micro.
I take this opportunity to express my grati-
tude to Hyesoon and the selection committee
members, who spent countless hours during
the Christmas and New Year season to evaluate
the submissions, and conducted a face-to-face
meeting and deliberated a whole day to per-
form this important selection. Hyesoon and
the committee conducted a multistep selection
process that tried to reduce the impact of the
discussion order of the articles. I wish to
express my special thanks to Hyesoon and the
selection committee for the thoughtful process
and the hard work.
The Top Picks articles belong to four
themes: first, cloud and accelerators; second,
acceleration from understanding applications;
third, quantum computing; and fourth, security
and privacy. One may recognize that these are
highly relevant topics with or without the pan-
demic. A comprehensive article written by Hye-
soon Kim serves as an excellent introduction to
the compendium.
I want to personally congratulate all the Top
Picks authors for their fantastic work. It is a sig-
nificant honor to rise to the top in a competition
where each candidate article is already recog-
nized as an excellent piece of work. I hope that
these works have significant impact on future
computer systems.
In addition to the Top Picks articles, this issue
also features a Micro Economics column by
Shane Greenstein of Harvard Business School,
May/June 2020
titled “Pandemics and the Dismal Technology
Economy.” This article discusses the economic
crisis brought by the pandemic. The author talks
about the predictable features of such a crisis
even when the pandemic itself is unprecedented,
however, recognizes that it is difficult to make pre-
dictions on when the economy might turn around.
He also discusses the increased appeal of stream-
ing services and online communication tools.
To all the computer architects and chip
designers: while COVID19 may have altered your
daily routines, the demand for microchips and
systems is only increasing. Medical research,
data analytics to pre-
dict outbreaks, online
It is a significant honor communication and
to rise to the top in a conferencing tools,
competition where each and security and
candidate article is privacy for various
already recognized as applications, all are
an excellent piece of pointing to the need
work. I hope that these for increased resea-
works have significant
rch to design hard-
impact on future
ware and software
computer systems.
that efficiently meets
the demands of the
emerging era.
I hope this special issue is thought provoking
for our readers and helps shape the field for
many years to come. I also hope that researchers
in the field intensify their efforts to design better
chips and systems to help medical research and
ordinary daily lives. Additionally, I encourage
readers to submit to IEEE Micro. IEEE Micro is
interested in submissions on any aspect of chip/
system design or architecture.
May the Top Picks articles bring some happy
reading to you amidst this coronavirus pandemic!
Lizy Kurian John is a Cullen Trust for Higher Educa-
tion Endowed Professor in the Electrical and Computer
Engineering Department, University of Texas at Austin.
Contact her at ljohn@ece.utexas.edu.
5
 Guest Editor’s Introduction
The 2019 Top Picks in Computer
Architecture
Hyesoon Kim
Georgia Institute of Technology
& IT IS MY pleasure to introduce the “2019 Top
Picks in Computer Architecture.” This annual
publication presents 12 articles selected from
major computer architecture conferences of the
year. The 12 papers are recognized for their
importance, mainly the long-term impact and
influence on the industry and other researchers.
The selection committee members put enor-
mous effort into picking the papers. We asked
what the criteria should be for the top picks, and
then we tried to answer that question by looking
for significant improvement over previous work,
establishing a new area.
As in prior years, only 12 articles could be
selected to appear in this special issue. The
selection committee chose 14 additional high-
quality articles to be recognized as honorable
mentions. I strongly encourage you to read
these articles (see the “Honorable Mentions”
sidebar).
REVIEW PROCESS
This year’s review process built on previous
years’ selection processes. Authors submitted a
three-page document that contained a two-page
summary of the article and one page of support-
ing arguments for long-term impact and influ-
ence on other researchers and industry.
Digital Object Identifier 10.1109/MM.2020.2992834
Date of current version 22 May 2020.
0272-1732 ß 2020 IEEE Published by the IEEE Computer Society
6
Twenty eight selection committee members (see
the “Selection Committee” sidebar) read the
three-page documentations along with the origi-
nal conference papers (single-blind review pro-
cess by nature). In keeping with the successful
two-round ranking-based review process of the
past several years, the PC members first catego-
rized each article as either a top pick, an honor-
able mention, or not a top pick. They also
ranked the articles. After the first round of
reviews, all PC members participated in online
discussions to decide which articles should
move to the second round. In the first round,
all the articles were assigned at least four
reviewers, and in the second round, the articles
had at least four additional reviewers.
This year, as we expanded our research areas
into special accelerators that rely on emerging
technologies, we found it particularly challeng-
ing to ensure all reviewers understood the
underlying technologies. Because the selection
process is concerned more with the impact of
the work rather than evaluating its technical
accuracy, technical expertise is less critical than
for main conference reviews. Nonetheless, when
several papers cover similar topics, it is also
important to identify those worthy of nomina-
tion based on technical merits. To overcome the
limitations on available expertise, we increased
the number of reviewers for such emerging tech-
nology based papers.
IEEE Micro
 & SELECTION COMMITTEE
 Arka Basu, Indian Institute of Science (IISc)
 
Babak Falsafi, Ecole Polytechnique Fe  de
rale de
Lausanne (EPFL)
 Boris Grot, University of Edinburgh
 Christopher Fletcher, University of Illinois
Urbana-Champaign (UIUC)
 nez, Texas A&M University
Daniel Jime
 Dmitry Ponomarev, SUNY Binghamton
 Edward Suh, Cornell University
 Gennady Pekhimenko, University of Toronto
 Jangwoo Kim, Seoul National University
 Jayasena Nuwan, AMD
 Jishen Zhao, University of California San Diego
 John Kim, KAIST
 Jose Joao, Arm Research
PC MEETING
The in-person (now, we need to differenti-
ate in-person versus virtual!) PC meeting
occurred on January 10, 2020 on the campus of
Georgia Institute of Technology, Atlanta, GA,
USA. Of the 28 PC members from three conti-
nents, 24 attended the meeting and 4 could not
attend due to last minute emergencies. The
discussion order was loosely determined by
the articles’ overall score and rank. Articles
with similar topics were grouped together so
as to provide more consistent evaluations and
effective discussions.
The PC meeting was conducted in two phases
in order to minimize the influence of the discus-
sion order. In the first phase, we made prelimi-
nary decisions about the articles’ outcomes. In
the second phase, we adjusted the results; e.g.,
by deselecting articles from the top pick pool or
rescuing articles from the honorable mention
pool.
Because voting was often the key instrument
in deciding the outcome of the articles, we insti-
tuted a rigorous set of voting procedures. First,
only reviewers of the article voted on whether
the article was a top pick or an honorable men-
tion. If the vote was above 60% among reviewers,
the article became a candidate for top picks or
honorable mentions (unless the vote was
May/June 2020
 Josep Torrellas, University of Illinois
Urbana-Champaign (UIUC)
 Lisa Wu Wills, Duke University
 Mike O’Connor, NVIDIA/UT-Austin
 Mohit Tiwari, UT Austin
 Onur Mutlu, ETH Zurich/CMU
 Parthasarathy Ranganathan, Google
 Rajeev Balasubramonian, University of Utah
 Ravi Iyer, Intel
 Reetuparna Das, University of Michigan
 Thomas Wenisch, University of Michigan/Google
 Tor Aamodt, The University of British Columbia
 Tushar Krishna, Georgia Institute of Technology
 Ulya Karpuzcu, University of Minnesota
 Vijay Janapa Reddi, Harvard University
 Yunji Chen, Institute of Computing Technology
Chinese Academy of Sciences (ICT–CAS)
unanimous for top pick, in which case the article
was finalized as a top pick in the first phase). If
not, the vote went to all PC members excluding
those with conflicts. If the vote was above 50%,
then the article became a candidate for top picks
or honorable mentions. Since some articles had
many conflicts, for each article, we precounted
the number of votes needed to top 50%. All vote
results during the meeting were recorded and
later used to decide the order of discussions in
the second phase. This two-phase mechanism
was critical because when PC members voted,
they wanted to see all the articles before making
final decisions.
SELECTED ARTICLES
Cloud and Accelerators
The importance of cloud computing and
accelerators continues to grow in the architec-
ture community. The challenges of evaluating
cloud and edge systems are presented in
“Unveiling the Hardware and Software Implica-
tions of Microservices in Cloud and Edge Sys-
tems” by Gan et al. The article presents a new
open source benchmark suite for cloud micro-
services and evaluations on real systems to
guide future architecture designs. Accelerating
DNN is an ongoing topic in architecture
7
 Guest Editor’s Introduction

& HONORABLE MENTIONS Gray, Brucek Khailany, and Stephen W. Keckler


(MICRO 2019).
“The Accelerator Wall: Limits of Chip Special-
 “Buffets: An Efficient and Composable Storage
ization,” by Adi Fuchs and David Wentzlaff (HPCA
Idiom for Explicit Decoupled Data Orches-
2019).

tration,” by Michael Pellauer, Yakun Sophia Shao,
“TensorDIMM: A Practical Near-Memory Proc-
Jason Clemons, Neal Crago, Kartik Hegde, Ran-
essing Architecture for Embeddings and Tensor
gharajan Venkatesan, Stephen W. Keckler, Christo-
Operations in Deep Learning,” by Youngeun
pher W. Fletcher, and Joel Emer (ASPLOS 2019).
Kwon, Yunjae Lee, and Minsoo Rhu (MICRO
 “ExTensor: An Accelerator for Sparse Tensor
2019).

Algebra,” by Kartik Hegde, Hadi Asghari-Moghad-
“ComputeDRAM: In-Memory Compute Using Off-
dam, Michael Pellauer, Neal Crago, Aamer Jaleel,
the-Shelf DRAMs,” by Fei Gao, Georgios Tziant-
Edgar Solomonik, Joel Emer, and Christopher
zioulis, and David Wentzlaff (MICRO 2019).

Fletcher (MICRO 2019).
“NDA: Preventing Speculative Execution Attacks
 “A Formal Analysis of the NVIDIA PTX Memory
at Their Source,” by Ofir Weisse, Ian Neal, Kevin
Consistency Model,” by Daniel Lustig, Sameer
Loughlin, Thomas F. Wenisch, and Baris Kasikci
Sahasrabuddhe, and Olivier Giroux (ASPLOS 2019).
(MICRO 2019).
 “SpecShield: Shielding Speculative Data from
 “Janus: Optimizing Memory and Storage Support
Microarchitectural Covert Channels,” by Kristin
for Non-Volatile Memory System,” by Sihang Liu,
Barber, Anys Bacha, Li Zhou, Yinqian Zhang, and
Korakit Seemakhupt, Gennady Pekhimenko,
Radu Teodorescu (PACT 2019).
Aasheesh Kolli, and Samira Khan, (ISCA 2019).
 “Designing Vertical Processors in Monolithic
 “D-RaNGe: Using Commodity DRAM Devices to
3D,” by Bhargava Gopireddy and Josep Torrellas
Generate True Random Numbers with Low
(ISCA 2019).
Latency and High Throughput,” by Jeremie
 “CIDR: A Cost-Effective In-Line Data Reduction
S. Kim, Minesh Patel, Hasan Hassan, Lois Orosa,
System for Terabit-per-Second Scale SSD
and Onur Mutlu (HPCA 2019).

Arrays,” by Mohammadamin Ajdari, Pyeongsu
“Simba: Scaling Deep-Learning Inference with
Park, Joonsung Kim, Dongup Kwon, Jangwoo Kim
Multi-Chip-Module-Based Architecture,” by
(HPCA 2019).
Yakun Sophia Shao, Jason Clemons, Rangharajan
 “Practical Byte-Granular Memory Blacklisting
Venkatesan, Brian Zimmer, Matthew Fojtik, Nan
using Califorms,” by Hiroshi Sasaki, Miguel A.
Jiang, Ben Keller, Alicia Klinefelter, Nathaniel
Arroyo, M. Tarek Ibn Ziad, Koustubha Bhat, Kanad
Pinckney, Priyanka Raina, Stephen G. Tell, Yanq-
Sinha, and Simha Sethumadhavan (MICRO 2019).
ing Zhang, William J. Dally, Joel Emer, C. Thomas

conferences, as shown in “MAESTRO: A Data- Accelerations From Understanding

Centric Approach to Understand Reuse, Perfor-
mance, and Hardware Cost of DNN Mappings”
by Kwon et al. This article presents an analyti-
cal cost-benefit analysis framework that con-
siders the cost of DNN mapping tradeoffs in
terms of data reuse. Accelerating virtual real-
ity becomes more important, especially in
the current environment. The article “Energy-
Efficient Video Processing for Virtual Reality”
by Leng et al. seeks to improve energy effi-
ciency from the architecture support based on
the characterizations and evaluations of VR
prototypes.
Applications
Understanding target application characteris-
tics generates synergetic architectural improve-
ments. A generalized acceleration framework for
irregular workloads is presented in “Towards
General-Purpose Acceleration: Finding Structure
in Irregularity” by Dadu et al. The article
“Varifocal Storage: Dynamic Multiresolution
Data Storage” by Hu et al. proposes a dynamic
multiresolution storage system to help approxi-
mate computing. Understanding applications
can also be extended to warehouse-scale com-
puter (WSC) applications. A profiling and code

IEEE Micro
8
analysis tool to allow identification of critical
code segments and then proposing solutions to
improve the performance of WSC is presented in
“AsmDB: Understanding and Mitigating Front-
End Stalls in Warehouse-Scale Computers” by
Nagendra et al.
Quantum Computing
2019 was the year that quantum computing
architecture/compilers became one of the main-
stream computer architecture research topics.
Two articles were selected to represent the
research challenges in quantum computing. The
article “Extending the Frontier of Quantum Com-
puters With Qutrits” by Gokhale et al. presents
the use of three-level qutrits and also evaluates
the system-level impact of qutrits-based quan-
tum computing. The article also provides good
background materials for quantum computing. A
measurement-based full-stack characterization
of basic quantum computing applications on
real systems is presented in “Architecting Noisy
Intermediate-Scale Quantum Computers: A Real-
System Study” by Murali et al.
Security and Privacy
As a reflection of new computing design
requirements from security and privacy, three
articles on security and one article on privacy
are presented in this issue. The article
“Speculative Taint Tracking (STT): A Compre-
hensive Protection for Speculatively Accessed
Data” by Yu et al. provides a framework that
tracks flow of speculative instructions through
covert channels. The article “MicroScope:
Enabling Microarchitectural Replay Attacks”
by Skarlatos et al. presents a means of replay-
ing code by forcing microarchitectural reply
based on address translations. ISA extensions
design methodology to improve security while
considering performance is presented in
“Creating Foundations for Secure Microarchi-
tectures With Data-Oblivious ISA Extensions”
by Yu et al. Finally, the article “Trace Wringing
May/June 2020
for Program Trace Privacy” by Dangwal et al.
presents a compression method to generate
traces that can limit the information leak, and
memory trace writing for cache simulation is
shown as an example.
CONCLUSION
I hope you will enjoy reading these articles,
and I encourage you to explore the full confer-
ence versions of both the Top Pick and Honor-
able Mention selections. The authors made
significant efforts to write a version that can be
read by a broad audience.
ACKNOWLEDGMENTS
I would like to thank Lizy Kurian John, the
Editor-in-Chief of IEEE Micro, for providing sup-
port and guidance at every stage of issue prepa-
ration. I would also like to thank Vijay J. Redidi
for handling papers that I had conflicts with and
J. Zho for handling papers that both Vijay and I
had conflicts with. I also thank the previous
year’s guest editor, S. Dwarkadas, for her input.
Attending her Top Picks meeting last year was
very helpful in preparing for this year’s meeting.
I thank M. Qureshi for providing valuable input
during the discussions, and the submission chair
Y. Kim, R. Hadidi and the volunteer students
J. Lee, and B. Asgari who helped organize PC
meetings. I thank all the PC members who have
diligently read the articles and put enormous
effort into selecting the finalists. Finally, I thank
all the authors who have submitted their work.
Hyesoon Kim is an Associate Professor with the
School of Computer Science, Georgia Institute of
Technology. Her research areas include the intersec-
tion of computer architectures and compilers, with an
emphasis on heterogeneous architectures such as
GPUs and accelerators. She is a recipient of NSF
Career Award and is a member of Micro Hall of
Fame. She is a senior member of IEEE. Contact her
at hyesoon@cc.gatech.edu.
9
 Theme Article: Top Picks

Unveiling the Hardware

and Software Implications
of Microservices in Cloud
and Edge Systems
Yu Gan, Yanqi Zhang, Dailun Cheng,
Ankitha Shetty, Priyal Rathi, Nayan Katarki,
Ariana Bruno, Justin Hu, Brian Ritchken,
Brendon Jackson, Kelvin Hu,
Meghna Pancholi, Yuan He, Brett Clancy,
Chris Colen, Fukang Wen, Catherine Leung,
Siyuan Wang, Leon Zaruvinsky,
Mateo Espinosa, Rick Lin, Zhongling Liu,
Jake Padilla, and Christina Delimitrou
Cornell University

Abstract—Cloud services progressively shift from monolithic applications to complex graphs

of loosely-coupled microservices. This article aims at understanding the implications
microservices have across the system stack, from hardware acceleration and server design,
to operating systems and networking, cluster management, and programming frameworks.
Toward this effort, we have designed an open-sourced DeathstarBench, a benchmark suite
for interactive microservices that is both representative and extensible.

& CLOUD COMPUTING NOW powers applications
from every domain of human endeavor, which
require ever improving performance, respon-
siveness, and scalability.2,5,6,8 Many of these
Digital Object Identifier 10.1109/MM.2020.2985960
Date of publication 22 April 2020; date of current version 22
May 2020.
applications are interactive, latency critical
services that must meet strict performance
(throughput and tail latency), and availability
constraints, while also handling frequent soft-
ware updates.4–7; 12 The past five years have
seen a significant shift in the way cloud services
are designed, from large monolithic implemen-
tations, where the entire functionality of a

0272-1732 ß 2020 IEEE Published by the IEEE Computer Society IEEE Micro
10

Figure 1. Differences in the deployment of
monoliths and microservices.
service is implemented in a single binary, to
large graphs of single-concerned and loosely-
coupled microservices.1,10 This shift is becom-
ing increasingly pervasive, with large cloud pro-
viders, such as Amazon, Twitter, Netflix, Apple,
and EBay having already adopted the microser-
vices application model, and Netflix reporting
more than 200 unique microservices in their
ecosystem, as of the end of 2016.1
The increasing popularity of microservices is
justified by several reasons. First, they promote
composable software design, simplifying and
accelerating development, with each microser-
vice being responsible for a small subset of the
application’s functionality. The richer the func-
tionality of cloud services becomes, the more
the modular design of microservices helps man-
age system complexity. They similarly facilitate
deploying, scaling, and updating individual micro-
services independently, avoiding long develop-
ment cycles, and improving elasticity. For
applications that are updated on a daily basis,
modifying, recompiling, and testing a large mono-
lith is both cumbersome and prone to bugs.
Figure 1 shows the deployment differences
between a traditional monolithic service, and an
application built with microservices. While the
entire monolith is scaled out on multiple servers,
microservices allow individual components of
the end-to-end application to be elastically
scaled, with microservices of complementary
resources bin-packed on the same physical
server. Even though modularity in cloud services
was already part of the service-oriented architec-
ture (SOA) design approach, the fine granularity
of microservices, and their independent deploy-
ment create hardware and software challenges
different from those in traditional SOA workloads.
May/June 2020
Second, microservices enable programming
language and framework heterogeneity, with
each tier developed in the most suitable lan-
guage, only requiring a common API for micro-
services to communicate with each other;
typically over remote procedure calls (RPC) or a
RESTful API. In contrast, monoliths limit the lan-
guages used for development, and make fre-
quent updates cumbersome and error-prone.
Finally, microservices separate failure doma-
ins across application tiers, allowing cleaner error
isolation, and simplifying correctness and perfor-
mance debugging, unlike in monoliths, where
resolving bugs often involves troubleshooting the
entire service. This also makes them applicable to
Internet-of-Things (IoT) applications that often
host mission-critical computation.
Despite their advantages, microservices rep-
resent a significant departure from the way cloud
services are traditionally designed, and have
broad implications in both hardware and soft-
ware, changing a lot of assumptions current ware-
house-scale systems are designed with. For
example, since dependent microservices are typi-
cally placed on different physical machines, they
put a lot more pressure on high bandwidth and
low latency networking than traditional applica-
tions. Furthermore, the dependencies between
microservices introduce backpressure effects
between dependent tiers, leading to cascading
QoS violations that propagate and amplify
through the system, making performance debug-
ging expensive in both resources and time.11
Given the increasing prevalence of microser-
vices in both cloud and IoT settings, it is impera-
tive to study both their opportunities and
challenges. Unfortunately most academic work
on cloud systems is limited to the available
open-source applications; monolithic designs in
their majority. This not only prevents a wealth
of interesting research questions from being
explored, but can also lead to misdirected
research efforts whose results do not translate
to the way real cloud services are implemented.
DeathstarBench SUITE
Our article,10 presented at ASPLOS’19,
addresses the lack of representative and open-
source benchmarks built with microservices, and
11
 Top Picks
Figure 2. Graph of microservices in Social Network.
quantifies the opportunities and challenges of this
new application model across the system stack.
Benchmark Suite Design: We have designed,
implemented, and open-sourced a set of end-
to-end applications built with interactive micro-
services, representative of popular production
online services using this application model. Spe-
cifically, the benchmark suite includes a social
network, a media service, an ecommerce shop, a
hotel reservation site, a secure banking system,
and a coordination control platform for UAV
swarms. Across all applications, we adhere to the
design principles of representativeness, modular-
ity, extensibility, software heterogeneity, and end-
to-end operation.
Each service includes tens of microservices in
different languages and programming models,
including node.js, Python, C/Cþþ, Java, Java-
script, Scala, and Go, and leverages open-source
applications, such as NGINX, memcached, Mon-
goDB, Cylon, and Xapian. To create the end-to-end
services, we built custom RPC and RESTful APIs
using popular open-source frameworks like
Apache Thrift, and gRPC. Finally, to track how
user requests progress through microservices, we
have developed a lightweight and transparent to
the user distributed tracing system, similar to Dap-
per and Zipkin that tracks requests at RPC granu-
larity, associates RPCs belonging to the same end-
to-end request, and records traces in a centralized
database. We study both traffic generated by real
users of the services, and synthetic loads gener-
ated by open-loop workload generators.
Applications in DeathStarBench
Social Network: The end-to-end service imple-
ments a broadcast-style social network with
12
Figure 3. Graph of microservices in Media Service.
unidirectional follow relationships. Figure 2
shows the architecture of the end-to-end service.
Users (client) send requests over http, which
first reach a load balancer, implemented with
nginx. Once a specific webserver is selected,
also in nginx, the latter uses a php-fpm module
to talk to the microservices responsible for com-
posing and displaying posts, as well as microser-
vices for advertisements and search engines. All
messages downstream of php-fpm are Apache
Thrift RPCs. Users can create posts embedded
with text, media, links, and tags to other users.
Their posts are then broadcasted to all their fol-
lowers. Users can also read, favorite, and repost
posts, as well as reply publicly, or send a direct
message to another user. The application also
includes machine learning plugins, such as user
recommender engines, a search service using
Xapian, and microservices to record and display
user statistics, e.g., number of followers, and to
allow users to follow, unfollow, or block other
accounts. The service’s backend uses memc-
ached for caching, and MongoDB for persistent
storage for posts, profiles, media, and recom-
mendations. The service is broadly deployed at
our institution, currently servicing several hun-
dred users. We also use this deployment to
quantify the tail at scale effects of microservices.
Media Service: The application implements an
end-to-end service for browsing movie informa-
tion, as well as reviewing, rating, renting, and
streaming movies. Figure 3 shows the architec-
ture of the end-to-end service. As with the social
network, a client request hits the load balancer,
which distributes requests among multiple nginx
webservers. Users can search and browse infor-
mation about movies, including their plot,
IEEE Micro
photos, videos, cast, and review information, as
well as insert new reviews in the system for a
specific movie by logging into their account.
Users can also select to rent a movie, which
involves a payment authentication module to
verify that the user has enough funds, and a
video streaming module using nginx-hls, a pro-
duction nginx module for HTTP live streaming.
The actual movie files are stored in NFS, to avoid
the latency and complexity of accessing chunked
records from nonrelational databases, while
movie reviews are kept in memcached and Mon-
goDB instances. Movie information is main-
tained in a sharded and replicated MySQL
database. The application also includes movie
and advertisement recommenders, as well as a
couple auxiliary services for maintenance and
service discovery, which are not shown in the
figure.
E-Commerce Site: The service implements an
e-commerce site for clothing. The design draws
inspiration, and uses several components of the
open-source Sockshop application. The applica-
tion front-end in this case is a node.js service.
Clients can use the service to browse the inven-
tory using catalogue, a Go microservice that
mines the back-end memcached and MongoDB
instances holding information about products.
Users can also place orders (Go) by adding items
to their cart (Java). After they log in (Go) to their
account, they can select shipping options (Java),
process their payment (Go), and obtain an
invoice (Java) for their order. Finally, the service
includes a recommender engine for suggested
products, and microservices for creating an item
wishlist (Java), and displaying current discounts.
Hotel Reservation: The service implements a
hotel reservation site, where users can browse
information about hotels and complete reserva-
tions. The service is primarily written in Go, with
the backend tiers implemented using memc-
ached and MongoDB. Users can filter hotels
according to ratings, price, location, and avail-
ability. They also receive recommendations on
hotels they may be interested in.
Banking System: The service implements a
secure banking system that processes payments,
loan requests, and credit card transactions.
Users interface with a node.js front-end, similar
to the one in E-commerce to login to their
May/June 2020
account, search information about the bank, or
contact a representative. Once logged in, a user
can process a payment, pay their credit card bill,
browse information about loans or request one,
and obtain information about wealth manage-
ment options. Most microservices are written in
Java and Javascript. The back-end databases use
memcached and MongoDB instances.
IoT Swarm Coordination: Finally, we explore an
environment where applications run both on the
cloud and on edge devices. The service coordi-
nates the routing of a swarm of programmable
drones, which perform image recognition and
obstacle avoidance. We have designed two ver-
sion of this service. In the first, the majority of the
computation happens on the drones, including
the motion planning, image recognition, and
obstacle avoidance, with the cloud only con-
structing the initial route per-drone, and holding
persistent copies of sensor data. This architec-
ture avoids the high network latency between
cloud and edge, however, it is limited by the on-
board resources. In the second version, the cloud
is responsible for most of the computation. It per-
forms motion control, image recognition, and
obstacle avoidance for all drones, using the
ardrone-autonomy, and Cylon libraries, in
OpenCV and Javascript, respectively. The edge
devices are only responsible for collecting sensor
data and transmitting them to the cloud, as well
as recording some diagnostics using a local node.
js logging service. In this case, almost every
action suffers the cloud-edge network latency,
although services benefit from the additional
cloud resources. We use 24 programmable Parrot
AR2.0 drones, together with a backend cluster of
20 two-socket, 40-core servers.
Adoption
DeathStarBench is open-source software
under a GPL license. The project is currently in
use by several tens of research groups both in
academia and industry. In addition to the open-
source project, we have also deployed the social
network as an internal social network at Cornell
University, currently used by over 500 students,
and have used execution traces for several

https://github.com/delimitrou/DeathStarBench.
13
 Top Picks
Figure 4. Tail latency with increasing load and decreasing
frequency (RAPL) for traditional monolithic cloud applications,
and the five end-to-end DeathStarBench services. Lighter colors
(yellow) denote QoS violations.
research studies, including using ML in root
cause analysis for interactive microservices.11
HARDWARE AND SOFTWARE
IMPLICATIONS OF MICROSERVICES
We have used DeathStarBench to quantify
the implications microservices have in cloud
hardware and software, and what these implica-
tions mean for computer engineers. Below we
summarize the main findings from our study.
Server Design
We first quantified how effective current data-
center architectures are at running microservices,
as well as how datacenter hardware needs to
change to better accommodate their perfor-
mance and resource requirements. There has
been a lot of work on whether small servers can
replace high-end platforms in the cloud.3 Despite
the power benefits of simple cores, interactive
services still achieve better latency in servers
optimized for single-thread performance. Micro-
services offer an appealing target for simple
cores, given the small amount of computation
per microservice. Figure 4 (top row) shows the
change in tail latency as load increases and fre-
quency decreases using running average power
limit (RAPL) for five popular, open-source single-
tier interactive services: nginx, memcached,
MongoDB, Xapian, and Recommender. We com-
pare these against the five end-to-end services
(bottom row).
14
As expected, most interactive services are sen-
sitive to frequency scaling. Among the monolithic
workloads, MongoDB is the only one that can tol-
erate almost minimum frequency at maximum
load, due to it being I/O-bound. The other four sin-
gle-tier services experience increased latency as
frequency drops, with Xapian being the most sen-
sitive, followed by nginx, and memcached. Looking
at the same study for microservices reveals that,
perhaps counterintuitively, they are much more
sensitive to poor single-thread performance than
traditional cloud applications, despite the small
amount of per-microservice processing. The rea-
sons behind this are the strict, microsecond-level
tail latency requirements of individual microservi-
ces, which put more pressure on low and predict-
able latency, emphasizing the need for hardware
and software techniques that eliminate latency jit-
ter. Out of the five end-to-end services (we omit
Swarm-Edge, since compute happens on the edge
devices), the Social Network and E-commerce are
most sensitive to low frequency, while the Swarm
service is the least sensitive, primarily because it
is bound by the cloud-edge communication
latency, as opposed to compute speed.
Networking and OS Overheads
Microservices spend a large fraction processing
network requests of RPCs or other RESTful APIs.
While for traditional monolithic cloud services
only a small amount of time goes toward network
processing, with microservices this time increases
to 36.3% on average, and on occasion over 50% of
the end-to-end latency, causing the system’s
resource bottlenecks to change drastically. To this
end, we also explored the potential hardware accel-
eration has to address the network requirements
of microservices for low latency and high through-
put network processing. Specifically, we use a
bump-in-the-wire setup, seen in Figure 5(a), and
similar to the one given by Firestone et al.9 to off-
load the entire TCP stack on a Virtex 7 FPGA using
Vivado HLS. The FPGA is placed between the NIC
and the top of rack switch, and is connected to
both with matching transceivers, acting as a filter
on the network. We maintain the PCIe connection
between the host and the FPGA for accelerating
other services, such as the machine learning mod-
els in the recommender engines, during periods of
low network load. Figure 5(b) shows the speedup
IEEE Micro
Figure 5. (a) Overview of the FPGA configuration for RPC
acceleration, and (b) the performance benefits of acceleration in
terms of network and end-to-end tail latency.
from acceleration on network processing latency
alone, and on the end-to-end latency of each of the
services. Network processing latency improves by
1068x over native TCP, whereas end-to-end tail
latency improves by 43% and up to 2:2x. For inter-
active, latency-critical services, where even a
small improvement in tail latency is significant,
network acceleration provides a major boost in
performance.
Cluster Management
A major challenge with microservices has to
do with cluster management. Even though the
cluster manager can elastically scale out individ-
ual microservices on-demand instead of the entire
monolith, dependencies between microservices
introduce backpressure effects and cascading
QoS violations that propagate through the sys-
tem, hurting quality of service (QoS). Backpres-
sure can additionally trick the cluster manager
into penalizing or upsizing a highly utilized micro-
service, even though its saturation is the result of
backpressure from another, potentially not-satu-
rated service. Not only does this not solve the
Figure 6. Microservices graphs for three production
clouds, and our Social Network.
May/June 2020
Figure 7. Cascading QoS violations in Social
Network compared to per-microservice CPU
utilization.
performance issue, but can on occasion make it
worse, by admitting more traffic into the system.
The more complex the dependence graph
between microservices, the more pronounced
such issues become. Figure 6 shows the microser-
vices dependence graphs for three major cloud
service providers, and for one of our applications
(Social Network). The perimeter of the circle (or
sphere surface) shows the different microservi-
ces, and edges show dependencies between
them. Such dependencies are difficult for develop-
ers or users to describe, and furthermore, they
change frequently, as old microservices are
swapped out and replaced by newer services.
Figure 7 shows the impact of cascading QoS
violations in the Social Network service. Darker
colors show tail latency closer to nominal opera-
tion for a given microservice in Figure 7(a), and
low utilization in Figure 7(b). Brighter colors sig-
nify high per-microservice tail latency and high
CPU utilization. Microservices are ordered based
on the service architecture, from the back-end
services at the top, to the front-end at the bot-
tom. Figure 7(a) shows that once the back-end
service at the top experiences high tail latency,
the hotspot propagates to its upstream services,
and all the way to the front-end. Utilization in
this case can be misleading. Even though the sat-
urated back-end services have high utilization in
Figure 7(b), microservices in the middle of the
figure also have even higher utilization, without
this translating to QoS violations.
Conversely, there are microservices with
relatively low utilization and degraded perfor-
mance, for example, due to waiting on a blocking/
synchronous request from another, saturated
tier. This highlights the need for cluster manag-
ers that account for the impact dependencies
15
 Top Picks
Figure 8. (a) Microservices taking longer than
monoliths to recover from a QoS violation, even (b) in
the presence of autoscaling mechanisms.
between microservices have on end-to-end per-
formance when allocating resources.
Finally, the fact that hotspots propagate
between tiers means that once microservices
experience a QoS violation, they need longer to
recover than traditional monolithic applications,
even in the presence of autoscaling mechanisms,
which most cloud providers employ. Figure 8
shows such a case for Social Network imple-
mented with microservices, and as a monolith in
Java. In both cases, the QoS violation is detected
at the same time. However, while the cluster man-
ager can simply instantiate new copies of the
monolith and rebalance the load, autoscaling
takes longer to improve performance. This is
because, as shown in Figure 8(b), the autoscaler
simply upsizes the resources of saturated
services—seen by the progressively darker colors
of highly utilized microservices. However, serv-
ices with the highest utilization are not necessar-
ily the culprits of a QoS violation, taking the
system much longer to identify the correct source
behind the degraded performance and upsizing it.
As a result, by the time the culprit is identified,
long queues have already built up, which take
considerable time to drain.
Serverless Programming Frameworks
Microservices are often used interchangeably
with serverless compute frameworks. Serverless
enables fine-grained, short-lived cloud execution,
and is well-suited for interactive applications
with ample parallelism and intermittent activity.
We evaluated our end-to-end applications on
AWS’s serverless platform, AWS Lambda and
showed that despite avoiding the high costs of
reserved idle resources, and enabling more elas-
tic scaling in the presence of short load bursts,
serverless also results in less predictable
16
performance for interactive microservices for
three reasons. First, on current serverless plat-
forms communication between dependent func-
tions (serverless tasks) happens via remote
persistent storage, S3, in AWS’s case. This not
only introduces high, but also unpredictable
latency, as S3 is subject to long queueing delays
and rate limiting. Second, because VMs hosting
serverless tasks are terminated after a given
amount of idle time, when new tasks are spawned,
they need to reload any external dependencies,
introducing nonnegligible overheads. Finally, the
placement of serverless tasks is up to the cloud
operator’s scheduler, and hence prone to long
scheduling delays, and unpredictable perfor-
mance due to contention from external jobs shar-
ing system resources. Overall, we show that while
similar, microservices and serverless compute
each present different system challenges, and are
well suited for different application classes.
Tail at Scale Effects
Finally, we explore the system implications of
microservices at large scale. Tail at scale effects
are well-documented in warehouse-scale com-
puters,4 and refer to system (performance, avail-
ability, efficiency) issues that specifically arise due
to a system’s scale. In this article, we use the
large-scale deployment of the social network appli-
cation with hundreds of real users to the impact of
cascading performance hotspots, request skews,
and slow servers on end-to-end performance.
Figure 9(a) shows the performance impact
of dependencies between microservices on 100
EC2 instances. Microservices on the y-axis are
again ordered from the back-end in the top to
the front-end in the bottom. While initially
all microservices are behaving nominally, at
t ¼ 260 s the middle tiers, and specifically compo-
sePost, and readPost become saturated due to a
switch routing misconfiguration that overloaded
one instance of each microservice, instead of load
balancing requests across different instances. This
in turn causes their downstream services to satu-
rate, causing a similar waterfall pattern in per-tier
latency to the one in Figure 7. Toward the end of
the sampled time (t > 500 s) the back-end services
also become saturated for a similar reason, caus-
ing microservices earlier in the critical path to sat-
urate. This is especially evident for microservices
IEEE Micro
 As with the hardware and cluster management
implications above, these results again emphasize
the need for hardware and software techniques
that improve performance predictability at scale
without hurting latency and resource efficiency.

Figure 9. (a) Cascading hotspots in the large-scale

Social Network deployment, and tail at scale effects LESSONS FROM DEATHSTARBENCH
from slow servers. DeathStarBench draws attention to the need
for research in the emerging application model
in the middle of the y-axis (bright yellow), whose of microservices and highlights the research
performance was already degraded from the previ- areas with the highest potential for impact, while
ous QoS violation. To allow the system to recover, also providing a widely adopted open-source
we employed rate limiting, which constrains the infrastructure to make that research possible
admitted traffic, until hotspots dissipate. Even and reproducible.
though rate limiting is effective, it affects user Cloud systems have attracted an increasing
experience by dropping a fraction of requests. amount of work over the past five to ten years.
Finally, Figure 9(b) shows the impact of a small As cloud software evolves, the direction of such
number of slow servers on overall QoS as cluster research efforts should also evolve with it. Given
size increases. We purposely slow down a small the increasing complexity of cloud services, the
fraction of servers by enabling aggressive power switch to a fine-grained, multitier application
management, which we already saw is detrimental model, such as microservices, will continue to
to performance. For large clusters ( > 100 instan- gain traction, and requires more attention from
ces), when 1% or more of servers behave poorly, the research community in both academia and
QPS under QoS is almost zero, as industry. DeathStarBench highlights the need
these servers host at least one for systems research in this emerg-
microservice on the critical path, DeathStarBench draws ing field, and quantifies the most
degrading QoS. Even for small attention to the need for promising research directions, and
clusters (40 instances), a single research in the their potential impact.
slow server is the most the ser- emerging application DeathStarBench also highlights
vice can sustain and still achieve model of microservices the need for hardware–software
some QPS under QoS. Finally, we and highlights the codesign in cloud systems, as appli-
research areas with the
compare the impact of slow serv- cations increase in scale and com-
highest potential for
ers in clusters of equal size for plexity. Specifically, we showed that
impact, while also
the monolithic design of Social current platforms are not well-suited
providing a widely
Network. In this case, QPS is adopted open-source for the emerging cloud programming
higher even as cluster sizes grow, infrastructure to make models, which require lower latency,
since a single slow server only that research possible more elastic scaling, and more pre-
affects the instance of the mono- and reproducible. dictable responsiveness. We demon-
lith hosted on it, while the other strated that microservices come
instances operate independently. with both opportunities and chal-
The only exception are back-end databases, which lenges across the system stack, and that for sys-
even for the monolith are shared across applica- tem designers to improve QoS without sacrificing
tion instances, and sharded across machines. If resource efficiency, they need to rethink the cur-
one of the slow servers is hosting a database rent cloud stack in a vertical way, from hardware
shard, all requests directed to that instance are design and networking, to cluster management
degraded. The more complex an application’s and programming framework design. Finally, we
microservices graph, the more impactful slow also quantified the potential hardware accelera-
servers are, as the probability that a service on tion has toward addressing the performance
the critical path will be slowed-down increases. requirements of interactive microservices, and

May/June 2020
17
 Top Picks
showed that programmable acceleration can
greatly reduce one of the primary overheads of
multitier services; network processing.
As microservices continue to evolve, it is
essential for datacenter hardware, operating
and networking systems, cluster managers, and
programming frameworks to also evolve with
them, to ensure that their prevalence does not
come at a performance and/or efficiency loss.
Both DeathStarBench and the resulting study of
the system implications of microservices are a
call to action for the research community to fur-
ther explore the opportunities and challenges of
this emerging application model.
ACKNOWLEDGMENTS
We sincerely thank C. Kozyrakis, D. Sanchez,
D. Lo, as well as the academic and industrial users
of the benchmark suite, and the anonymous
reviewers for their feedback on earlier versions of
this article. This work was supported in part by
an NSF CAREER award, in part by NSF grant CNS-
1422088, in part by a Google Faculty Research
Award, in part by a Alfred P. Sloan Foundation Fel-
lowship, in part by a Facebook Faculty Research
Award, in part by a John and Norma Balen Sesqui-
centennial Faculty Fellowship, and in part by gen-
erous donations from Google Compute Engine,
Windows Azure, and Amazon EC2.
& REFERENCES
1. “The evolution of microservices,” 2016. [Online].
Available: https://www.slideshare.net/adriancockcroft/
evolution-of-microservices-craft-conference
2. L. Barroso, U. Hoelzle, and P. Ranganathan, The
Datacenter as a Computer: An Introduction to the
Design of Warehouse-Scale Machines. Morgan &
Claypool: San Rafael, CA, USA, 2018.
3. S. Chen, S. Galon, C. Delimitrou, S. Manne, and
J. F. Martinez, “Workload characterization of
interactive cloud services on big and small server
platforms,” in Proc. Int. Symp. Workload
Characterization, Oct. 2017, pp. 125–134.
4. J. Dean and L. A. Barroso, “The tail at scale,”
Commun. ACM, vol. 56 no. 2, pp. 74–80, 2013.
5. C. Delimitrou and C. Kozyrakis, “Paragon: QoS-aware
scheduling for heterogeneous datacenters,” in Proc.
18th Int. Conf. Archit. Support Program. Lang. Oper.
Syst., Houston, TX, USA, 2013, pp. 77–88.
18
6. C. Delimitrou and C. Kozyrakis, “Quasar: Resource-
efficient and QoSAware cluster management,” in
Proc. 19th Int. Conf. Archit. Support Program.
Lang. Oper. Syst., Salt Lake City, UT, USA, 2014,
pp. 127–144.
7. C. Delimitrou and C. Kozyrakis, “HCloud: Resource-
efficient provisioning in shared cloud systems,” in
Proc. 21st Int. Conf. Archit. Support Program. Lang.
Oper. Syst., Apr. 2016, pp. 473–488.
8. C. Delimitrou and C. Kozyrakis, “Bolt: I know what you
did last summer... In the cloud,” in Proc. 22nd Int. Conf.
Archit. Support Program. Lang. Oper. Syst., Apr. 2017,
pp. 599–613.
9. D. Firestone et al., “Azure accelerated networking:
Smartnics in the public cloud,” in Proc. 15th USENIX
Symp. Netw. Syst. Design Implementation, 2018,
pp. 51–66.
10. Y. Gan et al., “An open-source benchmark suite for
microservices and their hardware-software
implications for cloud and edge systems,” in Proc.
24th Int. Conf. Archit. Support Program. Lang. Oper.
Syst., Apr. 2019, pp. 3–18.
11. Y. Gan et al., “Seer: Leveraging big data to
navigate the complexity of performance debugging
in cloud microservices,” in Proc. 24th Int. Conf.
Archit. Support Program. Lang. Oper. Syst., Apr.
2019, pp. 19–33.
12. D. Lo, L. Cheng, R. Govindaraju, P. Ranganathan, and
C. Kozyrakis, “Heracles: Improving resource
efficiency at scale,” in Proc. 42nd Annu. Int. Symp.
Comput. Archit., 2015, pp. 450–462.
Yu Gan is currently working toward the Ph.D. degree
with the School of Electrical and Computer Engineer-
ing, Cornell University, where he works on cloud
computing and root cause analysis for interactive
microservices. He is a student member of IEEE and
ACM. Contact him at yg397@cornell.edu.
Yanqi Zhang is currently working toward the Ph.D.
degree with the School of Electrical and Computer
Engineering, Cornell University, where he works on
cloud systems and resource management for inter-
active microservices. He is a student member of
IEEE and ACM. Contact him at yz2297@cornell.edu.
Dailun Cheng is currently working toward the
M.Eng. degree with the School of Electrical and
Computer Engineering, Cornell University. Contact
him at dc924@cornell.edu.
IEEE Micro
Ankitha Shetty is currently working toward
the M.Eng. degree with the School of Computer
Science, Cornell University. Contact him at
aas394@cornell.edu.
Priyal Rathi is currently working toward the M.Eng.
degree with the School of Computer Science, Cornell
University. Contact him at pr348@cornell.edu.
Nayan Katarki is currently working toward the
M.Eng. degree with the School of Electrical and
Computer Engineering, Cornell University. Contact
him at nk646@cornell.edu.
Ariana Bruno is currently working toward the
M.Eng. degree with the School of Electrical and
Computer Engineering, Cornell University. Contact
him at amb633@cornell.edu.
Justin Hu is currently working toward the M.Eng.
degree with the School of Computer Science, Cornell
University. Contact him at jh2625@cornell.edu.
Brian Ritchken is currently working toward the
M.Eng. degree with the School of Electrical and
Computer Engineering, Cornell University. Contact
him at bjr96@cornell.edu.
Brendon Jackson is currently working toward the
M.Eng. degree with the School of Electrical and
Computer Engineering, Cornell University. Contact
him at btj28@cornell.edu.
Kelvin Hu is currently working toward the M.Eng.
degree with the School of Computer Science, Cornell
University. Contact him at sh2442@cornell.edu.
Meghna Pancholi is currently working toward the
B.S. degree with the School of Computer Science,
Cornell University. Contact him at mp832@cornell.edu.
Yuan He is currently working toward the M.Eng.
degree with the School of Electrical and Computer
Engineering, Cornell University. Contact him at
yh772@cornell.edu.
Brett Clancy is currently working toward the M.Eng.
degree with the School of Computer Science, Cornell
University. Contact him at bjc265@cornell.edu.
May/June 2020
Chris Colen is currently working toward the M.Eng.
degree with the School of Computer Science, Cornell
University. Contact him at cdc99@cornell.edu.
Fukang Wen is currently working toward the M.Eng.
degree with the School of Computer Science, Cornell
University. Contact him at fw224@cornell.edu.
Catherine Leung is currently working toward the
M.Eng. degree with the School of Computer Science,
Cornell University. Contact him at chl66@cornell.edu.
Siyuan Wang is currently working toward the M.Eng.
degree with the School of Computer Science, Cornell
University. Contact him at sw884@cornell.edu.
Leon Zaruvinsky is currently working toward the
M.Eng. degree with the School of Computer Science,
Cornell University. Contact him at laz37@cornell.edu.
Mateo Espinosa is currently working toward the
M.Eng. degree with the School of Computer Science,
Cornell University. Contact him at me326@cornell.edu.
Rick Lin is currently working toward the M.Eng.
degree with the School of Electrical and Computer
Engineering, Cornell University. Contact him at
cl2545@cornell.edu.
Zhongling Liu is currently working toward the
M.Eng. degree with the School of Electrical and
Computer Engineering, Cornell University. Contact
him at zl682@cornell.edu.
Jake Padilla is currently working toward the
M.Eng. degree with the School of Computer Science,
Cornell University. Contact him at jsp264@cornell.edu.
Christina Delimitrou is currently an Assistant
Professor with the School of Electrical and Computer
Engineering, Cornell University, where she works on
computer architecture and distributed systems.
Her research interests include resource-efficient data-
centers, scheduling and resource management with
quality-of-service guarantees, emerging cloud and IoT
application models, and cloud security. Delimitrou
received the Ph.D. degree in electrical engineering
from Stanford University. She is a member of IEEE and
ACM. Contact her at delimitrou@cornell.edu.
19
 Theme Article: Top Picks

MAESTRO: A Data-Centric
Approach to Understand
Reuse, Performance, and
Hardware Cost of DNN
Mappings
Hyoukjun Kwon, Prasanth Chatarasi, Michael Pellauer and Angshuman Parashar
Vivek Sarkar, and Tushar Krishna NVIDIA Corp
Georgia Tech

Abstract—The efficiency of an accelerator depends on three factors—mapping, deep

neural network (DNN) layers, and hardware—constructing extremely complicated design
space of DNN accelerators. To demystify such complicated design space and guide the
DNN accelerator design for better efficiency, we propose an analytical cost model,
MAESTRO. MAESTRO receives DNN model description and hardware resources
information as a list, and mapping described in a data-centric representation we propose
as inputs. The data-centric representation consists of three directives that enable
concise description of mappings in a compiler-friendly form. MAESTRO analyzes various
forms of data reuse in an accelerator based on inputs quickly and generates more than 20
statistics including total latency, energy, throughput, etc., as outputs. MAESTRO’s fast
analysis enables various optimization tools for DNN accelerators such as hardware design
exploration tool we present as an example.

Digital Object Identifier 10.1109/MM.2020.2985963

Date of publication 22 April 2020; date of current version 22
May 2020.

0272-1732 ß 2020 IEEE Published by the IEEE Computer Society IEEE Micro
20
Figure 1. High-level overview of mapping a high-dimensional DNN layer (CONV2D in this figure) to an
accelerator with 2-D PE array. Note that tile scheduling also needs to be done within spatial partitioning; we omit
it for simplicity. (a) An Overview of Mapping CONV2D to an Accelerator. (b) High-level Tool flow of MAESTRO.
& DEEP NEURAL NETWORK (DNN) inference accel-
erators achieve high performance by exploiting
parallelism over hundreds of processing ele-
ments (PEs) and high energy efficiency by maxi-
mizing data reuse within PEs and on-chip
scratchpads.1–4 The efficiency (performance and
energy efficiency) of a DNN accelerator depends
on three factors depicted in Figure 1: 1) the
workload (DNN layers), 2) the amount and type
of available hardware resources (hardware), and
3) the mapping strategy of a DNN layer on the
target hardware (mapping). That is, we can pre-
dict the efficiency (latency, energy, buffer
requirement, etc.) of an accelerator when we
have full parameters for those three factors,
which can guide the DNN accelerator design for
better efficiency. One critical requirement on the
efficiency estimation is that it needs to be fast
since the design space (e.g., 480 million valid
designs in our hardware DSE even if we fix the
target mapping and layer) is huge, and we need
to query the efficiency of candidate designs in
the search space when we search for an optimal
design. How do we implement such a fast effi-
ciency estimation framework that thoroughly
considers all the parameters of the three
factors that determine the efficiency of DNN
accelerators?
Such demands led to the development of an
analytical cost model instead of cycle-accurate
simulators. Analytically, modeling the complex
high-dimensional DNN accelerator design space
over the three factors (DNN layer, hardware, and
May/June 2020
mapping) is challenging because it requires
deep understanding of complex interaction of
hardware components, mapping, and DNN
layers. In particular, data reuse in scratchpad
memory hierarchy in DNN accelerators is one of
the key behaviors, which is critical for energy
efficiency, thus the prime optimization target of
DNN accelerators. Data reuse pattern is dictated
by dataflow,1 which are data/computation tile
scheduling and spatial partitioning strategies
without actual tile size as described in Figure 1
(a). To systematically and analytically model the
data reuse for DNN accelerators’ efficiency esti-
mation, we need a precise and thorough descrip-
tion of mapping and a framework to analyze data
reuse of a mapping on target hardware and the
DNN layer.
Therefore, we propose a data-centric repre-
sentation of mapping that enables precise
descriptions of all the possible mappings in a
concise and compiler-friendly manner. Leverag-
ing the compiler-friendly format, we develop
MAESTRO, a comprehensive cost-benefit analy-
sis framework based on systematic data reuse
analysis. As shown in Figure 1(b), MAESTRO
receives the three factors—DNN layer, hard-
ware, and mapping—as inputs and generates
more than 20 estimated statistics including
latency, energy, the number of buffer accesses,
buffer size requirement, etc. We validated the
performance statistics of MAESTRO against
cycle-accurate RTL simulation results5 and
reported performance in a previous work6 with
21
 Top Picks
Table 1. The taxonomy of data reuse in DNN accelerators and
implementation choices for each. We highlight implementation used
in the example reuse patterns with red texts.
the accuracy of 96.1% on average. MAESTRO pro-
vides fast cost-benefit estimation based on an
analytical model, which took 493 ms to analyze
the entire Resent50 layers7 on a 256PE NVDLA-
style2 accelerator on a laptop with i9-9980H CPU
with 16 GB of memory. MAESTRO supports
arbitrary layer sizes and a variety of layer opera-
tions from state-of-the-art DNN models, which
includes CONV1D, CONV2D, fully connected (FC)
layer, depthwise separable convolution, up-scale
convolution, etc.
DATA REUSE IN DNN
ACCELERATORS
Data reuse is the key behavior in DNN acceler-
ator that improves both latency and energy via
reducing the number of remote buffer accesses
(i.e., global buffer),1; 8 which is determined by
dataflow. Data reuse opportunities exist when
the dataflow assigns the same set of data tiles
over consecutive time on the same PE (i.e., reuse
in time) or across multiple PEs but not over con-
secutive time (i.e., reuse in space). We define
those opportunities as temporal and spatial
reuse opportunities. For example, in the example
dataflow in Figure 1, output tiles (orange tiles)
remain the same in time 0 and 1, which implies
the temporal reuse opportunities. Within time 1,
as the spatial partitioning example in Figure 1
22
shows, input tile 3 is mapped on all the PEs,
which implies the spatial reuse opportunities.
Dataflow implies data reuse opportunities,
and we can categorize data reuse in DNN acceler-
ators into four types (data reuse taxonomy),
which we summarize in Table 1. Each data reuse
type requires proper hardware support to
exploit the data reuse opportunity as actual data
reuse. We discuss those four reuse types
grouped in communication type as follows:
Spatial/Temporal Multicast. When the spa-
tial/temporal reuse opportunities are in input
tensors (i.e., filter and input activation), the
reused data can be multicasted to multiple PEs
(spatial reuse) or over time (temporal reuse).
The examples in Table 1 show such a pattern
based on fanout NoC (spatial multicast), which
delivers data to multiple PEs at the same time,
and buffer (temporal multicast).
In the spatial multicast example, tiles 1 and 2
are delivered to PE1 and PE2 at the same time
leveraging the multicast capability of fanout
hardware. Alternatively, store-and-forward style
implementation such as systolic arrays is avail-
able with tradeoff of hardware cost and latency.
In the temporal multicast example, the same
data tile appears over time in the same PE (PE1).
That is, we send the data to the future for reuse
in the future (i.e., store the data in a buffer and
read it in the future). Therefore, temporal multi-
cast, which is reading the same stored data over
time, requires a buffer, as shown in Table 1.
Spatial/Temporal Reduction. When the spa-
tial reuse opportunities are in the output activa-
tion tensor, the reuse pattern in hardware is
spatial reduction, which accumulates partial out-
puts (or, partial sums) for an output across multi-
ple PEs. The example in Table 1 shows an
example reuse pattern based on store-and-for-
ward hardware. We observe that the output tiles
1 and 2 are moving to the next PE over time,
which illustrates pipelined accumulation to the
right direction assuming that PEs are receiving
new operands from above (i.e., a row of a systolic
array). Alternatively, fanin hardware such as
reduction tree can support the spatial reduction.
In contrast, the temporal reuse opportunities
imply that we compute partial sums over time
and accumulate them within the same location.
This type of reuse requires a buffer since
IEEE Micro
Figure 2. Example CONV1D operation and mapping of the example on an accelerator. We represent the
mapping in both computation and data space, where each point corresponds to a partial sum and a data,
respectively. We use 1-based indices in this example.
intermediate results need to be stored and read
again in the future, which effectively indicates
multiple read-modify-write to a buffer. The exam-
ple in Table 1 shows such a reuse pattern, where
the output tile 1 appears at the same PE over
time.
To identify the reuse opportunities in arbi-
trary mappings, we need a precise representa-
tion of mapping and systematically infer data
reuse from the description. For those two
goals, we present a data-centric representation
of mapping, which is concise and compiler
friendly.
DESCRIBING MAPPINGS
We use a CONV1D operation described
in Figure 2 as an example operation to introduce
our mapping description. As described
in Figure 2, CONV1D operation can be under-
stood as a sliding window operation of a filter
vector on a input vector, where individual multi-
plication results within a filter window are accu-
mulated to generated one output value in the
output vector. When we project the loop indices
in the loop nest in Figure 2(a), we obtain compu-
tation space in Figure 2(b) where loop indices
are on each axis, and partial sums are projected
in the plane. We also construct data space of
each vector as shown in Figure 2(b), where the
corresponding data index is on the axis. Note
that the data index is not the same as the loop
index (e.g., the input data index x is computed
using loop indices x0 +s). Therefore, we denote
data indices using underlined index in this exam-
ple. Note that output and filter indices x0 and s
are identical to the loop indices x0 and s in this
simple example operation.
May/June 2020
We show an example of mapping on three-PE
accelerator in computation and data space
in Figure 2(b). In this example mapping, we map
three partial sum computation to each PE, and
each PE collaboratively compute partial outputs
(accumulated partial sums) on the same set of
outputs. When the PE array finishes computa-
tion in a tile (time=0 in the example), the PE
array receives the next computation tile
(time=1 in the example). The next computation
tile is in the direction of loop index x0 . We project
the same mapping on the data space as shown
in Figure 2, using the array subscripts in the
loop nest of CONV1D operation in Figure 2(a).
That is, partial sum at (x,0 s) requires weight at s,
input at x0 +s, and output at x,0 as shown in the
loop body of in Figure 2(a). In the example, we
observe that the data space explicitly shows
data reuse behavior; mapped filter values do not
move over time, which implies that the example
mapping is based on a weight-stationary style
dataflow. This implies that inferring data reuse
can be significantly simplified when we describe
mapping in the data space, which can facilitate a
fast analysis framework of DNN accelerator’s
efficiency.
Motivated by the observation, we introduce
data-centric mapping directives that directly
describe the mapping in data space.
Data-centric mapping directives
We introduce three data-centric mapping
directives in Figure 3(a). Temporal and spatial
map directives describe data mapping that
changes in time and space (PEs), respectively.
That is, temporal map corresponds to a normal
for loop in loop nest while spatial map corre-
sponds to a parallel for loop. Those two mapping
23
 Top Picks
Figure 3. Introductory example of data-centric directives. (a) Syntax of data-centric directives. (b) Semantics
of two mapping directives based on an example description process on the example CONV1D mapping
in Figure 2. (c) Capability of data-centric mapping directives that can describe a variety of mapping styles.
directives take three parameters: Mapping size,
offset, and dimension. The mapping size speci-
fies the number of data points (in tensors, map-
ping size in the target dimension since a
mapping constructs a high-dimensional volume)
mapped on each PE. The offset describes how
the mapping is updated over time on temporal
map and space on spatial map. Cluster directive
specifies the hierarchical organization of PEs,
which enables us to explore multiple parallel
dimensions in a mapping.
To understand the syntax and semantics of
data-centric mapping directives, in Figure 3, we
provide an example process to determine a cor-
responding data mapping description of the
example mapping in Figure 2(b). We omit the
input tensor because input tensor data mapping
can be easily inferred from the mapping of out-
put and filter. We first determine if the mapping
is in time or space by checking the mapped
data are the same or different (i.e., paralleliza-
tion) across PEs. Next, we check the number of
data points mapped on each PE to determine
the mapping size, which are three and one for
output and filter, respectively, in the example.
To determine the offset parameter, we check
the temporal and spatial offset on temporal and
spatial map, respectively. For example, for
24
output vector in Figure 3(b), we observe that
the starting index of mapping changes over
time 3, which implies that the temporal offset is
3. For filter vector, we observe that the starting
index of mapping for each PE changes by 1,
which implies that the spatial offset is 1. Note
that spatial map can also involve temporal
aspect as the mapping on the filter vector
in Figure 3(b); after processing all the computa-
tion that involves the first data tile on filter, the
data tile will move on to the next position. This
happens when the number of PEs is not suffi-
cient to cover entire spatially mapped dimen-
sion (also known as spatial folding), and an
implicit temporal offset of (spatial offset) 
(number of PEs) is applied. Finally, we write the
dimension on which we describe the data map-
ping, then we obtain the data-centric mapping
description of each data mapping, as shown in
the resulting data mapping description column
in Figure 3(b). To specify the entire example
mapping, we need to specify the order of
changes in data tile between output and filter
vectors. Since filter is updated in a slower man-
ner, we place the data mapping description of
filter above, and write that of output below, like
we specify the update order in loop nest (outer-
most loop index changes slower).
IEEE Micro
Capability of Mapping Directives
Using the data-centric directives, we can
describe a variety of mappings if it maps con-
secutive data points in a regular manner (i.e.,
affine loop subscripts when described in a
loop nest representation). Figure 3(c) shows
the capability of the data-centric directive by
showing the changes in the resulting mapping
when we update the base representation we
obtained in Figure 3(b). When we change the
directive order, we describe a different order
of data tile update in dimensions. This effec-
tively changes the stationary vector from
weight to output, which changes the temporal
data reuse opportunities. When we change the
spatial dimension, then we exploit the parallel-
ism in a different dimension, as the third exam-
ple in Figure 3(b) and (c) shows. Finally, if we
change the mapping size (we accordingly
update the offset to keep the description
legal), we change the amount of mapped filter
and output, as shown in Figure 3(c) and (d).
Based on the fact that data reuse is explicit
in data dimension and the capability of data-
centric directives, we implement an analytical
cost-benefit analysis framework for DNN accel-
erators, MAESTRO. We discuss a high-level
overview of MAESTRO next and discuss
insights from the case studies we performed
based on MAESTRO next.
ANALYTICAL COST MODEL
Based on the data-centric directives we dis-
cussed, we built a cost-benefit analysis frame-
work that considers all of the three factors—
DNN layers, hardware, and mapping—with pre-
cise modeling of data reuse. MAESTRO consists
of five preliminary engines: Tensor, cluster,
reuse, performance analysis, and cost analysis.
In the article, we focus on the high-level idea
without details such as edge case handling, mul-
tiple layers, and multiple level hierarchy, etc. We
present implementation details in our web page
and open-source repository. We validated
MAESTRO’s performance model against RTL sim-
ulation and reported processing delay of two
accelerators—MAERI5 and Eyeriss6 when

https://maestro.ece.gatech.edu/
May/June 2020
running VGG16 and AlexNet, respectively. The
latency estimated by MAESTRO are within 3.9%
absolute error of the cycle-accurate RTL simula-
tion and reported processing delay6 on average.
CASE STUDIES
With MAESTRO, we perform deeper case
studies about the costs-benefit tradeoff of vari-
ous mappings when applied to different DNN
operations. We evaluate five distinct mapping
styles listed in Figure 4(a) in the “Case Study I:
The Impact of Mapping Choices” section and the
preference of each mapping to different DNN
operators. For energy estimation, we multiply
activity counts with base energy values from
Cacti13 simulation (28 nm, 2 kB L1 scratchpad,
and 1 MB shared L2 buffer). We also present dis-
tinct design space of an early layer (wide and
shallow) and a late layer (narrow and deep) to
show the dramatically different hardware prefer-
ence of different DNN layers and mapping in the
“Case Study II: Hardware Design-Parameters and
Implementation Analysis” section.
Case Study I: The Impact of Mapping Choices
Figure 4(b) shows the DNN-operator granu-
larity estimation of latency and energy of each
mapping across five state-of-the-art DNN models
listed in the “Case Studies” section. Note that
this should be considered a comparison of map-
ping—not of actual designs, which can contain
several low-level implementation differences,
e.g., custom implementations of logic/memory
blocks, process technology, etc. We observe
that KC-P style mapping provides overall low
latency and energy. However, the energy effi-
ciency in VGG16 is worse than YR-P (Eyeriss1
style) mapping, and the latency is worse than
YX-P (Shidiannao14 style) mapping in UNet. This
is based on the different preference toward map-
ping of each DNN operator. YX-P provides short
latency to segmentation networks like UNet,
which has wide activation (e.g., 572  572 in the
input layer) and recovers the original activation
dimension at the end via up-scale convolution
(e.g., transposed convolutions). Such a prefer-
ence to the YX-P style is mainly based on its par-
allelization strategy: It exploits parallelism over
both of row and column dimensions in
25
 Top Picks

Figure 4. Summary of case studies. (a) List of mappings used in case study I. (b) Results of the case study I.
Top and bottom rows present latency and energy, respectively. We apply 256 PEs and 32 GBps NoC
bandwidth. We use five different DNN models; Resnet50,7 VGG16,9 ResNeXt50,10 MobileNetV2,11 and
UNet.12 The right-most column presents the average results across models for each DNN operator type and
the adaptive mapping case. We compare the number of input channels and the input activation height to
identify early and late layers (If C > Y, late layer. Else, early layer). (c) Design space of KC-P and YR-P-based
accelerators. We highlight the design space of an early and a late layer to show their significantly different
hardware preference. We apply area/power constraints based on Eyeriss6 to the DSE. The color of each data
point indicates the number of PEs. We mark the throughput- and energy-optimized designs using stars and
crosses. (d) The impact of multicast capability, bandwidth, and buffer size. Design points are selected from
the upper-most design space in (c). The name of design points refer to the differences from the throughput-
optimal reference point. Dark rows represent the efficiency of the selected design point.

activation. The energy efficiency of YR-P map-
ping in VGG16 is based on its high reuse factor
(the number of local accesses per fetch) in early
layers. The YR-P mapping has 5.8 and 15.17
higher activation and filter reuse factors, respec-
tively, in early layers. However, in late layers,
the reuse factors of YR-P and KC-P mapping are
almost similar (difference < 11%), so the KC-P
mapping provides similar energy efficiency as
YR-P in these cases. This can also be observed in
the late layer (blue) bars in Figure 4(b) bottom-
row plots.
The diverse preference to mappings of differ-
ent DNN operators motivates us to employ

IEEE Micro
26
optimal mapping for each DNN operator type.
We refer such an approach as adaptive mapping
and present the benefits in the right-most col-
umn of Figure 4(b), the average case analysis
across entire models in the DNN operator granu-
larity. By employing the adaptive approach, we
could observe a potential 37% latency and 10%
energy reduction. Such an optimization opportu-
nity can be exploited by flexible accelerators like
Flexflow15 and MAERI5 or via heterogeneous
accelerators that employ multiple subaccelera-
tors with various mapping styles in a single DNN
accelerator chip.
Case Study II: Hardware Design-Parameters
and Implementation Analysis
Using MAESTRO, we implement a hardware
design space exploration (DSE) tool that
searches four hardware parameters (the number
of PEs, L1 buffer size, L2 buffer size, and
NoC bandwidth) optimized for either energy effi-
ciency, throughput, or energy-delay-product
(EDP) within given hardware area and power
constraints. The DSE tool receives the same set
of inputs as MAESTRO with hardware area/
power constraints and the area/power of build-
ing blocks synthesized with the target technol-
ogy. For the cost of building blocks, we
implement float/fixed point multiplier and adder,
bus, bus arbiter, and global/local scratchpad in
RTL and synthesis them using 28-nm technology.
For bus and arbiter cost, we fit the costs into a
linear and quadratic model using regression
because bus cost increases linearly and arbiter
cost increases quadratically (e.g., matrix
arbiter).
Using the DSE tool, we explore the design
space of KC-P and YR-P mapping accelerators.
We set the area and power constraint as 16 mm2
and 450 mW, which is the reported chip area
and power of Eyeriss.6 We plot the entire design
space we explored in Figure 4(c). Whether an
accelerator can achieve peak throughput
depends on not only the number of PEs but also
NoC bandwidth. In particular, although an accel-
erator has sufficient number of PEs to exploit
the maximum degree of parallelism a mapping
allows, if the NoC does not provide sufficient
bandwidth, the accelerator suffers a communica-
tion bottleneck in the NoC. Such design points
May/June 2020
can be observed in the area-throughput plot
in Figure 4(c). YR-P mapping requires low NoC
bandwidth so it does not show the same behav-
ior as KC-P mapping. However, with more strin-
gent area and power constraints, YR-P mapping
will show the same behavior.
During DSE runs, MAESTRO reports buffer
requirements for each mapping and the DSE tool
places the exact amount buffers MAESTRO
reported. Contrary to intuition, larger buffer
sizes do not always provide high throughput, as
shown in buffer-throughput plots in Figure 4
(plots in the second column). The optimal points
regarding the throughput per buffer size are in
the top-left region of the buffer-throughput plots.
The existence of such points indicates that the
tiling strategy of the mapping (mapping sizes
in our directive representation) significantly
affects the efficiency of buffer use. We observe
that the throughput-optimized designs have a
moderate number of PEs and buffer sizes, imply-
ing that hardware resources need to be distrib-
uted not only to PEs but also to NoC and buffers
for high PE utilization. Likewise, we observe that
the buffer amount does not directly increase
throughput and energy efficiency. These results
imply that all the components are intertwined,
and they need to be well-balanced to obtain a
highly efficient accelerator.
We also observe the impact of hardware sup-
port for each data reuse type, discussed
in Table 1. Figure 4(d) shows such design points
found in the design space of KC-P mapping on
VGG16-conv2 layer presented in the first row of
Figure 4(c). The reference design point is the
throughput-optimized design represented as a
star in the first row of Figure 4(c). When band-
width gets smaller, the throughput significantly
drops, but energy remains similar. However, the
lack of spatial multicast or reduction support
resulted in approximately 47% energy increase,
as the third and fourth design points shows.
CONCLUSION
Fast modeling of cost-benefit space of DNN
accelerators is critical for automated optimiza-
tion tools since the design space is huge and
high dimensional based on hundreds of DNN
model, hardware, and mapping parameters. In
27
 Top Picks

this article, we presented a methodology to 2. “Nvdla deep learning accelerator,” 2017. [Online].
enable fast cost-benefit estimation of a DNN Available: http://nvdla.org.
accelerator on a given DNN model and mapping, 3. A. Parashar et al., “Scnn: An accelerator for
which consists of a compiler-friendly data-cen- compressed-sparse convolutional neural networks,”
tric representation of mappings and an analyti- in Proc. Int. Symp. Comput. Archit., 2017,
cal cost-benefit estimation framework that pp. 27–40.
exploits the explicit data reuse in data space in 4. N. P. Jouppi et al., “In-datacenter performance
data-centric repre- analysis of a tensor processing unit,” in Proc. IEEE
sentations. To ana- Using MAESTRO, we
Int. Symp. Comput. Archit., 2017, pp. 1–12.
lytically estimate the show that no single 5. H. Kwon, A. Samajdar, and T. Krishna, “Maeri:
costs and benefits, mapping and no single Enabling flexible dataflow mapping over DNN
we demystify data hardware is ideal for all accelerators via reconfigurable interconnects,” in
reuse in hardware the DNN layers, which Proc. Int. Conf. Archit. Support Program. Lang. Oper.
and required hard- implies the complexity Syst., 2018, pp. 461–475.
ware support and of the DNN accelerator 6. Y.-H. Chen, T. Krishna, J. S. Emer, and V. Sze,
apply the observa- design space. Using “Eyeriss: An energy-efficient reconfigurable
tion into the ana- hardware design accelerator for deep convolutional neural networks,”
lytical cost-benefit space exploration IEEE J. Solid-State Circuits, vol. 52, no. 1, pp. 127–
framework we
estimation frame- 138, Jan. 2017.
implemented using
work, MAESTRO. 7. K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual
MAESTRO, we also
Using MAESTRO, show that hardware
learning for image recognition,” in Proc. IEEE Conf.
we show that no sin- features can Comput. Vis. Pattern Recognit., 2016, pp. 770–778.
gle mapping and no significantly impact the 8. A. Parashar et al., “Timeloop: A systematic approach
single hardware is throughput to DNN accelerator evaluation,” in Proc. IEEE Int.
ideal for all the and energy. Symp. Perform. Anal. Syst. Softw., Mar. 2019,
DNN layers, which pp. 304–315.
implies the complex- 9. K. Simonyan and A. Zisserman, “Very deep
ity of the DNN accelerator design space. Using convolutional networks for large-scale image
hardware design space exploration framework recognition,” in Proc. Int. Conf. Learn.
we implemented using MAESTRO, we also show Representations, 2015. [Online]. Available: https://iclr.
that hardware features can significantly impact cc/archive/www/doku.php%3Fid=iclr2015:accepted-
the throughput and energy. Those cases show main.html
that the capability of MAESTRO for various anal-  r, Z. Tu, and K. He,
10. S. Xie, R. Girshick, P. Dolla
ysis problems on DNN accelerator design space. “Aggregated residual transformations for deep
In addition to the case studies we performed, neural networks,” in Proc. IEEE Conf. Comput. Vis.
MAESTRO also facilitates many other optimiza- Pattern Recognit., 2017, pp. 1492–1500.
tion (e.g., neural architecture search specialized 11. M. Sandler, A. Howard, M. Zhu, A. Zhmoginov, and L.
for a target accelerator, mapping search for a tar- Chen, “MobileNetV2: Inverted Residuals and Linear
get accelerator, etc.) frameworks based on its Bottlenecks,” in Proc. IEEE Conf. Comput. Vis. Pattern
speed and accuracy, which will lead to broad Recognit., 2018, pp. 4510–4520.
impact on various areas (DNN model design, 12. O. Ronneberger, P. Fischer, and T. Brox, “U-net:
compiler, architecture, etc.) in the DNN accelera- Convolutional networks for biomedical image
tor domain. segmentation,” in Proc. Int. Conf. Med. Image Comput.
Comput.-Assisted Intervention, 2015, pp. 234–241.
13. N. Muralimanohar, R. Balasubramonian, and N. P.
& REFERENCES Jouppi, “Cacti 6.0: A tool to model large caches,” HP
1. Y.-H. Chen, J. Emer, and V. Sze, “Eyeriss: A spatial Laboratories, vol. 27, p. 28, 2009.
architecture for energy-efficient dataflow for 14. Z. Du et al., “Shidiannao: Shifting vision processing
convolutional neural networks,” in Proc. Int. Symp. closer to the sensor,” in Proc. Int. Symp. Comput.
Comput. Archit., 2016, pp. 367–379. Archit, 2015, pp. 92–104.

IEEE Micro
28
15. W. Lu, G. Yan, J. Li, S. Gong, Y. Han, and X. Li,
“Flexflow: A flexible dataflow accelerator architecture
for convolutional neural networks,” in Proc. Int. Symp.
High Perform. Comput. Archit., 2017, pp. 553–564.
Hyoukjun Kwon is currently working toward the
Ph.D. degree in the College of Computing, Georgia
Institute of Technology. His research interest includes
communication-centric and flexible accelerator design
and modeling mappings on spatial accelerators. Kwon
received B.S. degrees in environmental materials sci-
ence and in computer science and engineering from
Seoul National University. He is a student member of
IEEE. Contact him at hyoukjun@gatech.edu.
Prasanth Chatarasi is a senior Ph.D. student
advised by Prof. Vivek Sarkar and Dr. Jun Shirako in
the School of Computer Science, Georgia Institute of
Technology. His research focuses on advancing
compiler optimizations for high-performance appli-
cations on general-purpose and domain-specific
parallel architectures. In the past, he focused on
enhancing traditional compilation techniques for
both sequential and explicitly parallel programs
for performance optimizations and debugging
on general-purpose architectures. Contact him at
cprasanth@gatech.edu.
Vivek Sarkar is a Professor and the Stephen Flem-
ing Chair for Telecommunications in the College of
Computing at Georgia Institute of Technology, where
he conducts research in multiple aspects of software
for parallel computing. He is a Fellow of ACM and
IEEE. Contact him at vsarkar@gatech.edu.
May/June 2020
Tushar Krishna is an Assistant Professor in the
School of Electrical and Computer Engineering,
Georgia Institute of Technology, where he also holds
the ON Semiconductor Junior Professorship. His
research interests include computer architecture,
on-chip interconnection networks, and deep learning
accelerators. Krishna received the Ph.D. degree in
electrical engineering and computer science from
Massachusetts Institute of Technology. He received
the NSF CRII Award in 2018. He is a member of IEEE
and ACM. Contact him at tushar@ece.gatech.edu.
Michael Pellauer is a Senior Research Scientist at
NVIDIA. His research interests are building domain
specific accelerators, with a special emphasis on
deep learning and sparse tensor algebra. Pellauer
received the Ph.D. degree from Massachusetts Insti-
tute of Technology, the Masters degree from Chalm-
ers University of Technology, and the Bachelor’s
degree from Brown University. Contact him at
mpellauer@nvida.com.
Angshuman Parashar is a Senior Research
Scientist at NVIDIA. His research interests are in
building, evaluating, and programming spatial and
data-parallel architectures, with a present focus
on automated mapping of machine learning
algorithms onto architectures based on explicit
decoupled data orchestration. Parashar received
the Ph.D. degree in computer science and engi-
neering from the Pennsylvania State University
(2007), and the B.Tech. degree in computer
science and engineering from the Indian Institute
of Technology, Delhi, India (2002). Contact him at
aparashar@nvidia.com.
29
 Theme Article: Top Picks

Energy-Efficient Video
Processing for Virtual
Reality
Yue Leng and Jian Huang Chi-Chun Chen, Qiuyue Sun, and Yuhao Zhu
University of Illinois at Urbana–Champaign University of Rochester

Abstract—Virtual reality (VR) has huge potential to enable radically new applications,
behind which spherical panoramic video processing is one of the backbone techniques.
However, current VR systems reuse the techniques designed for processing conventional
planar videos, resulting in significant energy inefficiencies. Our characterizations show
that operations that are unique to processing 360
VR content constitute 40% of the
total processing energy consumption. We present EVR, an end-to-end system for
energy-efficient VR video processing. EVR recognizes that the major contributor to the VR
tax is the projective transformation (PT) operations. EVR mitigates the overhead of PT
through two key techniques: semantic-aware streaming on the server and hardware-
accelerated rendering on the client device. Real system measurements show that EVR
reduces the energy of VR rendering by up to 58%, which translates to up to 42% energy
saving for VR devices.

& VIRTUAL (VR) has profound social

REALITY
impact in transformative ways. For instance,
immersive VR experience is shown to reduce
patient pain more effectively than traditional
medical treatments, and is seen as a promising
solution to the opioid epidemic. One of the key
use-cases of VR is 360
video processing. Unlike
Digital Object Identifier 10.1109/MM.2020.2985692
Date of publication 6 April 2020; date of current version 22
May 2020.
conventional planar videos, 360
videos embed
panoramic views of the scene. As users change
the viewing angle, the VR device renders differ-
ent parts of the scene, mostly on a head-
mounted display (HMD), providing an immersive
experience.
A major challenge in VR video processing
today is the excessive power consumption of VR
devices. Our measurements show that rendering
720p VR videos in 30 frames per second (FPS)
consistently consumes about 5 W of power,
which is twice as much power than rendering

0272-1732 ß 2020 IEEE Published by the IEEE Computer Society IEEE Micro
30
conventional planar videos and exceeds the that is specialized for PT. We implement an
thermal design point (TDP) of typical mobile EVR prototype on an Amazon AWS server
devices.4 The device power requirement instance and an NVIDA Jetson TX2 board
will only grow as users demand combined with a Xilinx Zynq-7000
higher frame-rate and resolu- A major challenge in FPGA. Real system measurements
tion, presenting a practical VR video processing show that EVR reduces the energy
challenge to the energy- and today is the excessive of VR rendering by up to 58%,
thermal-constrained mobile VR power consumption of which translates to up to 42%
devices. VR devices. Our energy saving for VR devices.
The excessive device power measurements show
is mainly attributed to the fun- that rendering 720p VR
videos in 30 frames
damental mismatch between ENERGY
per second (FPS)
today’s VR system design phi- CHARACTERIZATIONS
consistently consumes
losophy and the nature of VR A VR system involves two distinct
about 5 W of power,
videos. Today’s VR video sys- which is twice as much
stages: capture and rendering. VR
tems are designed to reuse power than rendering videos are captured by special cam-
well-established techniques conventional planar eras, which generate 360
images
designed for conventional pla- videos and exceeds that are best presented in the spheri-
nar videos.1 This strategy the thermal design cal format. The spherical images are
accelerates the deployment of point (TDP) of typical then projected to planar frames
VR videos, but causes signifi- mobile devices. through one of the spherical-to-pla-
cant energy overhead. More nar projections, such as the equirec-
specifically, VR videos are streamed and proc- tangular projection. The planar video
essed as conventional planar videos. As a result, is either directly live-streamed to client devices
once on-device, each VR frame goes through a for rendering (e.g., broadcasting a sports event),
sequence of spherical–planar projective trans- or published to a content provider, such as
formations (PT) that correctly render a user’s YouTube or Facebook, and then streamed to
current viewing area on the display. The PT client devices upon requests. Alternatively, the
operations are pure overhead uniquely associ- streamed videos can also be persisted in the
ated with processing VR videos—operations local storage on a client device for future play-
that we dub “VR tax.” Our characterizations back. This article focuses on client-side VR con-
show that “VR tax” is responsible for about 40% tent rendering, i.e., after a VR video is captured,
of the processing energy consumption, a lucra- because rendering directly impacts VR devices’
tive target for optimizations. energy efficiency.
We present EVR, an end-to-end system for Rendering VR videos consumes excessive
energy-efficient VR video processing. EVR rec- power on the VR device, which is particularly
ognizes that the major contributor to the VR problematic as VR devices are energy and ther-
tax is the PT operations. EVR mitigates the mal constrained. This section characterizes the
overhead of PT through two key techniques: energy consumption of VR devices. Although
semantic-aware streaming (SAS) on the server there are many prior studies that focused on
and hardware-accelerated rendering (HAR) on energy measurement of mobile devices such as
the client device. EVR uses SAS to reduce smartphones and smartwatches, this is the first
the chances of executing PT on VR devices by such study that specifically focuses on VR devi-
prerendering 360
frames in the cloud. ces. We show that the energy profiles between
Different from conventional prerendering tech- VR devices and traditional mobile devices are
niques, SAS exploits the key semantic informa- different.
tion inherent in VR content that is previously We conduct studies on a recently published VR
ignored. Complementary to SAS, HAR miti- video dataset, which consists of head movement
gates the energy overhead of on-device ren- traces from 59 real users viewing different 360

dering through a new hardware accelerator VR videos on YouTube.3 We replay the traces

May/June 2020
31
 Top Picks
Figure 1. Power and energy characterizations of VR device.
(a) Power distribution across the major components in a VR
device. (b) Contribution of the PT operations to the compute and
memory energy.
to mimic realistic VR viewing behaviors. We
assemble a custom VR device based on the NVIDIA
Jetson TX2 development board in order to con-
duct fine-grained power measurements on the
hardware, which are infeasible in off-the-shelf VR
devices. We refer readers to the “Evaluation
Methodology” section for a complete experimen-
tal setup.
Power and Energy Distribution
We breakdown the device power consump-
tion into five major components: display, net-
work (WiFi), storage (eMMC), memory (DRAM),
and compute (SoC). The storage system is
involved mainly for temporary caching. We
show the power distribution across the five com-
ponents for the five VR video workloads in
Figure 1(a). The power consumption is averaged
across the entire viewing period. Thus, the
power consumption of each component is pro-
portional to its energy consumption.
We make two important observations. First,
the device consistently draws a power con-
sumption of about 5 W across all five VR vid-
eos. As a comparison, the TDP of a mobile
device, i.e., the power that the cooling system
is designed to sustainably dissipate, is around
3.5 W,4 clearly indicating the need to reduce
power consumption.
Second, unlike traditional smartphone and
smartwatch applications where network,
display, and storage consume significant
energy,2,7,9 the energy consumptions of the
three components in a VR device are relatively
32
insignificant, contributing to only about 9%,
7%, and 4% of the total energy consumption,
respectively. This indicates that optimizing
network, display, and storage would lead to
marginal energy reductions. More lucrative
energy reductions come from optimizing com-
pute and memory.
Contribution of VR Operations We further
find that energy consumed by executing opera-
tions that are uniquely associated with process-
ing VR videos constitutes a significant portion of
the compute and memory energy. Such opera-
tions mainly consist of PT operations. We show
the energy contribution of the PT operations to
the total compute and memory energy as a
stacked bar chart in Figure 1(b). On average, PT
contributes to about 40% of the total compute
and memory energy, and is up to 53% in the case
of video Rhino. The PT operations exercise the
SoC more than the DRAM as is evident in their
higher contributions to compute energy than
memory energy.
Overall, our results show that the PTs would
be an ideal candidate for energy optimizations.
ENERGY-EFFICIENT VR WITH EVR
The goal of EVR is to reduce energy consump-
tion of VR devices by optimizing the core compo-
nents that contribute to the “VR tax.” We present
an end-to-end energy-efficient VR system with
optimization techniques distributed across the
cloud server and the VR client.
Semantics-Aware Streaming
Our key idea is to leverage video-inherent
semantic information that is largely ignored by
today’s VR servers. SAS specifically focuses on
one particular form of semantic information:
visual object. We show that users tend to focus
on objects in VR content, and object trajecto-
ries provide a proxy for predicting user view-
ing areas. We leverage a recently published VR
video data set,3 which consists of head move-
ment traces from 59 real users viewing differ-
ent 360
VR videos on YouTube. We further
confirm that users track the same set of
objects across frame rather than frequently
switching objects. Specifically, we measure
the time durations during which users keep
IEEE Micro

Figure 2. Cumulative distribution of tracking
durations.
tracking the movement of the same object, and
show the results in Figure 2 as a cumulative
distribution plot. On average, users spend
about 47% of time tracking an object for at
least 5 s.
The near 100% frame coverage in many vid-
eos as the number of identified objects increases
indicates that the server can effectively predict
user viewing area solely based on the visual
objects without sophisticated client-side mecha-
nisms such as using machine learning models to
predict users’ head movement.5,10 This observa-
tion frees the resource-constrained VR from per-
forming additional work and simplifies the client
design.
SAS has two major components: First, a static
and offline analysis component that extracts
objects from the VR video upon ingestion
and generates a set of FOV videos that could be
directly visualized once on a VR device; second,
a dynamic and runtime serving component that
streams FOV videos on demand to the VR device.
We augment the new FOV video with metadata
that corresponds to the head orientation for
each frame. Once the FOV video together with
its associated metadata is on the client side and
before a FOV frame is sent to the display, the VR
client compares the desired viewing area indi-
cated by the head motion sensor with the meta-
data associated with the frame. If the two match,
the client directly visualizes the frame on the dis-
play, bypassing the PT operations. Otherwise,
the client system requests the original video seg-
ment from the cloud, essentially falling back to
the normal VR rendering mode.
Note that in our current design, we make the
simplification that the client will always fall back
to the regular VR processing flow upon FOV-miss
and restart streaming FOV videos based on
May/June 2020
user’s current focus. One could also imagine
other design alternatives. For instance, the client
could send the current (desired) FOV to the
cloud service, which returns another FOV video
if there happens to be one that matches the
desired FOV. We leave it as future work to
explore the full design space of the dynamic
component.
Hardware-Accelerated Rendering
We propose a new hardware accelerator, PT
engine (PTE), that performs efficient PTs. We
design the PTE as an SoC IP block that replaces
the GPU and collaborates with other IPs such as
the Video Codec and Display Processor for VR
video rendering. Figure 3 shows how PTE fits
into a complete VR hardware architecture.
The PTE takes in frames that are decoded from
the video codec, and produces FOV frames to
the frame buffer for display. If a frame is already
prepared by the cloud server as a projected FOV
frame, the PTE sends it to the frame buffer
directly; otherwise the input frame goes through
the PTE’s datapath to generate the FOV frame.
The GPU can remain idle during VR video play-
back to save power.
The bulk of the PTE is a set of PT units (PTU)
that exploits the pixel-level parallelism. The
pixel memory (P-MEM) holds the pixel data for
the incoming input frame, and the sample mem-
ory (S-MEM) holds the pixel data for the FOV
frame that is to be sent to the frame buffer. The
PTE uses DMA to transfer the input and FOV
frame data. The PTE also provides a set of
Figure 3. Overview of the augmented hardware architecture.
33
 Top Picks
memory-mapped registers for configuration pur-
poses. The configurability allows the PTE adapt
to different popular projection methods and VR
device parameters, such as FOV size and display
resolution. The configurability ensures PTE’s
flexibility without the overhead of general-pur-
pose programmability that GPUs introduce.
The P-MEM and S-MEM must be properly
sized to minimize the DRAM traffic. Holding
the entire input frame and FOV frame would
require the P-MEM and S-MEM to match the
video resolution (e.g., 4K) and display resolu-
tion (e.g., 1440p) respectively, requiring tens
of MBs on-chip memories that are prohibitively
large in practice. Interestingly, we find that the
filtering step (the only step in the PT algorithm
that access pixel data) operates much like a
stencil operation that possesses two proper-
ties. First, PT accesses only a block of adjacent
pixels for each input. Second, the accessed pixel
blocks tend to overlap between adjacent inputs.
Thus, the P-MEM and S-MEM are designed to
hold several lines of pixels in the input and FOV
frame, which is similar to the line buffer used in
image signal processor (ISP) designs.6
EVR Implementation
Building on top of the two optimizing primi-
tives, SAS and HAR, we design EVR. EVR includes
a cloud component and a client component. The
cloud component extracts object semantics
from VR videos upon ingestion, and prerenders
a set of miniature videos that contain only the
user viewing areas and that could be directly
rendered as planar videos by leveraging the
powerful computing resources on the cloud. The
client component retrieves the miniature video
with object semantics, and leverages the special-
ized accelerator for energy-efficient on-device
rendering if the original full video is required.
For VR applications whose content comes from
panoramic videos available on the VR devices,
the HAR can accelerate the video rendering with
lower energy overhead. We implement EVR in a
prototype system, where the cloud service is
hosted on an AWS instance while the client is
deployed on a customize platform that combines
the NVIDIA TX2 and Xilinx Zynq-7000 develop-
ment boards, which can represent a typical VR
client device.
34
EVALUATION
Evaluation Methodology
Usage Scenarios. We evaluate three EVR
variants, each applies to a different use case,
to demonstrate EVR’s effectiveness and gen-
eral applicability. The three variants are as
follows.
 S: leverages SAS without HAR.
 H: uses HAR without SAS.
 SþH: combines the two techniques.
Energy Evaluation Framework Our energy
evaluation framework considers the five impor-
tant components of a VR device: network,
display, storage, memory, and compute. The
network, memory, and compute power can be
directly measured from the TX2 board through
the onboard Texas Instruments INA 3221 voltage
monitor IC. We also use a 2560  1440 AMOLED
display that is used in Samsung Gear VR and its
power is measured in our evaluation. We esti-
mate the storage energy using an empirical
eMMC energy model7 driven by the storage
traffic traces.
Baseline We compare against a baseline that is
implemented on the TX2 board and that does
not use SAS and HAR. The baseline is able to
deliver a real-time (30 FPS basis) user experi-
ence. Our goal is to show that EVR can effec-
tively reduce the energy consumption with little
loss of user experience.
Benchmark To faithfully represent real VR
user behaviors, we use a recently published
VR video data set,3 which consists of head
movement traces from 59 real users viewing
different 360
VR videos on YouTube. The vid-
eos have a 4K (3840  2160) resolution, which
is regarded as providing an immersive VR
experience. The data set is collected using the
Razer Open Source Virtual Reality HDK2 HMD
with an FOV of 110
 110
, and records users’
real-time head movement traces. We replay
the traces to emulate readings from the
IMU sensor and thereby mimic realistic VR
viewing behaviors. This trace-driven methodol-
ogy ensures the reproducibility of our results.
IEEE Micro
Results the energy efficiency of VR applications with
Energy Reductions On average, S and H cloud/client codesign.
achieve 22% and 38% compute energy savings,
respectively. SþH combines SAS and HAR and Energy Characterization of VR Devices
delivers an average 41%, and up to 58%, energy Although there are many prior studies that
saving. The compute energy savings across appli- focused on energy measurement of mobile
cations are directly proportional to the PT oper- devices, such as smartphones and smart-
ation’s contributions to the processing energy, watches, this is the first such study that specif-
as shown in Figure 1(b). For instance, Paris ically focuses on VR devices. We show that the
and Elephant have lower energy savings because energy profiles of VR devices are significantly
their PT operations contribute different from that of traditional
less to the total compute energy mobile devices. Our results sug-
consumptions. Although there are
many prior studies that gest that we must rethink the
The trend is similar for the conventional system-level power/
focused on energy
total device energy savings. SþH energy optimizations in the con-
measurement of mobile
achieves on average 29% and up text of VR processing.
devices, such as
to 42% energy reduction. The smartphones and
energy reduction increases smartwatches, this is Implication on Hardware IP Block
the VR viewing time, and also the first such study that for VR
reduces the heat dissipation specifically focuses on This article provides a case
and, thus, provides a better view- VR devices. in point for future mobile SoCs to
ing experience. integrate VR-specific and VR-opti-
mized IP blocks, and our principal idea of
User Experience Impact We
bypassing the GPU will be critical to those
also quantify user experience both quantitatively
designs. We design the PTE as a standalone IP
and qualitatively. Quantitatively, we evaluate the
block in order to enable modularity and ease dis-
percentage of FPS degradation introduced by EVR
tribution. Alternatively, the PTE logic could be
compared to the baseline. We show that the FPS
tightly integrated into either the video codec or
drop rate averaged across 59 users is only about
display processor. Indeed, many new designs of
1%. Lee et al. reported that a 5% FPS drop is
the display processor have started integrating
unlikely to affect user perception.8 We assessed
functionalities that used to be executed in GPUs,
qualitative user experience and confirmed that the
such as color space conversion. Such a tight
FPS drop is visually indistinguishable and that EVR
integration would let the display processor
delivers smooth user experiences. Although the
directly perform PT operations before scanning
goal of EVR is not to save bandwidth, EVR does
out the frame to the display, and thus reduces
reduce the network bandwidth requirement
the memory traffic induced by writing the FOV
through SAS, which transmits only the pixels that
frames from the PTE to the frame buffer.
fall within user’s sight.

Cloud/Client Codesign for VR

CONCLUSION
We anticipate that EVR will have a signifi-
cant long-term impact on VR technologies and
their applications of tomorrow. We summarize
the main contributions as follows: EVR pro-
vides the first energy characterization study
of VR devices and demonstrate their major
energy overhead; EVR develops the first hard-
ware accelerator for accelerating the critical
PT operations in VR video processing; EVR
provides a systematic approach to improve
EVR provides a cloud/client collaborative
approach to improve the energy efficiency of
VR devices. In EVR, SAS and HAR have differ-
ent tradeoffs. On one hand, HAR is applicable
regardless where the VR videos are from, but
does not completely remove the overhead of the
PT operation. SAS potentially removes the PT
operation altogether, but relies on that the VR
video is published to a cloud server first. We show
that combining the two, when applicable, achieves
the best energy efficiency.

May/June 2020
35
 Top Picks
& REFERENCES
1. WhitePaper: 360-Degree Video Rendering. [Online].
Available: https://community.arm.com/graphics/b/
blog/posts/white-paper-360-degreevideo-rendering
2. X. Chen, N. Ding, A. Jindal, C. Hu, M. Gupta, and
R. Vannithamby, “Smartphone energy drain in the wild:
Analysis and implications,” ACM SIGMETRICS
Performance Eval. Rev., vol. 43, no. 1, pp. 151–164, 2015.
3. X. Corbillon, F. Simone, and G. Simon, “360-degree
video head movement dataset,” in Proc. 8th ACM
Multimedia Syst. Conf., 2017, pp. 199–204.
4. M. Halpern, Y. Zhu, and V. Reddi, “Mobile CPU’s rise to
power: Quantifying the impact of generational mobile
CPU design trends on performance, energy, and user
satisfaction,” in Proc. Int. Symp. High-Performance
Comput. Archit., 2016, pp. 64–76.
5. B. Haynes, A. Minyaylov, M. Balazinska, L. Ceze, and
A. Cheung, “VisualCloud demonstration: A DBMS for
virtual reality,” in Proc. ACM Int. Conf. Manage. Data,
2017, pp. 1615–1618.
6. J. Hegarty et al., “Darkroom: Compiling high-level
image processing code into hardware pipelines,” in
Proc. SIGGRAPH, 2014.
7. J. Huang, A. Badam, R. Chandra, and E. Nightingale,
“WearDrive: Fast and energy-efficient storage for
wearables,” in Proc. USENIX Annu. Tech. Conf., 2015,
pp. 613–625.
8. K. Lee et al., “Outatime: Using speculation to enable
low-latency continuous interaction for mobile cloud
gaming,” in Proc. 13th Annu. Int. Conf. Mobile Syst.,
Appl., Services, 2015, pp. 151–165.
9. J. Li, A. Badam, R. Chandra, S. Swanson,
B. Worthington, and Q. Zhang, “On the energy
overhead of mobile storage systems,” in Proc. File
Storage Technol., 2014, pp. 105–118.
36
10. F. Qian, L. Ji, Bo Han, and V. Gopalakrishnan,
“Optimizing 360 video delivery over cellular networks,”
in Proc. 5th Workshop All Things Cellular: Oper., Appl.
Challenges, 2016, pp. 1–6.
Yue Leng is currently a Software Engineer with
Airbnb, San Francisco, CA, USA. Leng received the
M.S. degree in computer engineering from the Univer-
sity of Illinois at Urbana–Champaign in 2019. Contact
her at yueleng2@illinois.edu.
Jian Huang is currently an Assistant Professor with
the Electrical and Computer Engineering Depart-
ment, University of Illinois at Urbana-Champaign.
Huang received the Ph.D. degree from Georgia
Institute of Technology in 2017. Contact him at
jianh@illinois.edu.
Chi-Chun Chen is currently a Compiler Engineer
with Cray, Inc., Seattle, WA, USA. Chen received the
M.S. degree in computer science from the University
of Rochester in 2019. Contact him at cchen120@ur.
rochester.edu.
Qiuyue Sun is currently a senior undergraduate
student with the Computer Science Department,
University of Rochester. Contact her at qsun15@u.
rochester.edu.
Yuhao Zhu is currently an Assistant Professor with
the Computer Science Department, University of
Rochester. Zhu received the Ph.D. degree from
The University of Texas at Austin in 2017. He is the
corresponding author of this article. Contact him at
yzhu@rochester.edu.
IEEE Micro
 Theme Article: Top Picks

Towards General-Purpose
Acceleration: Finding
Structure in Irregularity
Vidushi Dadu, Jian Weng, Sihao Liu, and
Tony Nowatzki
University of California Los Angeles

Abstract—Programmable hardware accelerators (e.g., vector processors, GPUs) have

been extremely successful at targeting algorithms with regular control and memory
patterns to achieve order-of-magnitude performance and energy efficiency improvements.
However, they perform far under the peak on important irregular algorithms, like those
from graph processing, database querying, genomics, advanced machine learning, and
others. This work posits that the primary culprit is specific forms of irregular control flow
and memory access. By capturing the problematic behavior at a domain-agnostic level, we
propose an accelerator that is sufficiently general, matches domain-specific accelerator
performance, and significantly outperforms traditional CPUs and GPUs.

& THE SLOWING IMPROVEMENTS of technology
scaling are raising the demand for specialized
hardware accelerators, especially for increas-
ingly difficult problems. While general-purpose
data-processing hardware, like GPUs or other
vector architectures, are effective on regular
algorithms, those with irregularity in their con-
trol flow or memory access patterns suffer in
performance. As evidence, many domain-specific
Digital Object Identifier 10.1109/MM.2020.2986199
Date of publication 16 April 2020; date of current version 22
May 2020.
accelerators have been proposed for “irregular”
domains like graph processing,3,9 compressed
neural networks,4,6,10 databases12 and genomics.
Compared to such architectures, GPUs lose in
performance and/or energy efficiency by order-
of-magnitude. On the other hand, domain-agnos-
tic architectures are widely applicable, which is
valuable for economies of scale and robustness
to algorithm change. An important question
then is whether it is possible to build a program-
mable accelerator that is equally as capable as
GPUs and vector processors, but better suited
to irregular algorithms.

May/June 2020 Published by the IEEE Computer Society 0272-1732 ß 2020 IEEE
37
 Top Picks

The first step toward this goal is to recog-

nize what makes an algorithm irregular. In
computer architecture, the concept of irregu-
larity is often used informally to indicate
behavior that causes inefficiency. We argue
that to first order, the root cause of most irreg-
ularity is data dependence. Data dependence
can appear in many forms, including control,
memory address calculation, conditional mem-
ory accesses, reuse and parallelism structure.
Consider the concrete example of sorting algo-
rithms. Merge sort has regular memory and irregu-
lar control. The memory that is read or written at
each step is predetermined, but the control flow
decisions depend on the relative order of the lists
to be merged at each step. This data-dependent
control prevents speculative execution from being
effective, and it also prevents vectorization (can-
Figure 1. Restricted data-dependence forms cover
not know the operands from each lane in
many algorithms. (a) Algorithm classification.
advance). Conversely, the radix-sort algorithm
(b) Dependence forms coverage.
has regular control, but irregular memory access
(bin increment and scatter). This prevents pre-
fetching from being effective, and also prevents workload that requires both forms is triangle
vectorization with standard instructions. In gen- counting in graphs. At each vertex, AF-Indirect is
eral, data dependence interferes or complicates used to locate a neighbor node’s adjacency list,
the fundamental mechanisms that parallel pro- and stream join can be used to find common
cessors use to extract performance. neighbors (each indicating a triangle).
Insight: To address these challenges, our key Approach: Critically, we find that these
observation is that it is not necessary to handle restricted forms of data-dependence can serve as
arbitrary irregularity because data dependence abstractions, which can be exploited in hardware.
manifests in common forms across domains. This We use this insight to construct our approach to
work suggests two specific forms are critical: design a “general-purpose” accelerator. Specifi-
stream join and alias-free indirection (AF-Indirect). cally, we start with an architecture known to work
Stream join is defined by in-order processing well for regular algorithms: a systolic-style coarse-
of data, where only the relative order of con- grained reconfigurable architecture (CGRA) with
sumption and production of new data is depen- streaming memory support. We then develop
dent on control decisions. These joins are hardware and software mechanisms for our two
surprisingly common, including merge sort, restricted data-dependence forms.
database joins, and inner product sparse tensor Our design is called the sparse processing
operations. AF-Indirect is characterized by mem- unit (SPU). SPU supports fully pipelined stream
ory access with data-dependent addresses, but joins with a systolic CGRA augmented with a
where the only memory dependencies are read– novel dataflow-control model. SPU supports
modify–write. Relevant kernels include radix- high-bandwidth AF-Indirect (load/store/update)
sort, outer product sparse-tensor operations, with a banked scratchpad with aggressive reor-
hash joins, histograms, and synchronous graph dering and embedded compute units for atomic
processing (e.g., page rank). update. Data dependence complicates the sup-
These data-dependence forms are not mutu- port for finer grain data types [naive subword
ally exclusive; in fact they can be thought of single-instruction–multiple-data (SIMD) is insuffi-
as different ways to relax regular (non-data-depen- cient]. Therefore, we add support to SPU to
dent) algorithms (see Figure 1). An example of a enable decomposing the reconfigurable network

IEEE Micro
38
and wide memory access into power-of-two finer
grain resources while maintaining data-depen-
dence semantics.
Evaluation and contribution: We study machine
learning (ML) as our primary domain, and graph
processing and databases to demonstrate gener-
ality. SPU achieves between 1.8–7
speedup on
artificial intelligence (AI)/ML applications, and
SPU’s ability to retain performance on dense
algorithms led to 4.5
speedup. On graph
and database applications, SPU achieves similar
performance to domain-specific accelerators with
modest performance and power overheads.
Our primary contributions in this work are the
identification of the two common exploitable
data-dependence forms, and an ISA and hardware
mechanisms to support them. More broadly, we
believe that taking a domain-agnostic approach
can lead to novel insights and foster knowledge
transfer across domains.
EXPLOITABLE DATA-DEPENDENCE
FORMS
We observe that two restricted forms of data
dependence are sufficient to cover many algo-
rithms: stream join and AF-Indirect. In this section,
we first define these forms and give intuition on
their performance challenges for existing archi-
tectures, and then overview our proposal.
Preliminary Term—“Streams”: Both of the
dependence forms rely on the concept of stream
abstractions, so we briefly explain. Streams are
simply an ordered sequence of values. Relevant
to this work are memory streams, which are
sequences of loads or stores with a well-defined
pattern.7,12 Streams are similar to vector
accesses, but have no fixed length.
Stream Join
An interesting class of algorithms iterates
over each input (each stream) in order, but the
total order of operations (and perhaps whether
an output is produced) is data dependent. Two
relevant kernels are shown in Figure 2. Sparse
vector multiplication (a) iterates over two
sparse lists (in CSR format) where indices are
stored in sorted order, and performs the multi-
plication if there is a match. The core of the
merge kernel (b) iterates over two sorted lists,
May/June 2020
and at each step outputs the smaller item. Even
though the data structures, data types, and pur-
pose are very different, their relationship to data
dependence is the same: they both have stream
access, but the relative ordering of stream con-
sumption is data dependent (they reuse data
from some stream multiple times).
Stream-join definition: A program region that
is regular except that the reuse of stream data
and the production of outputs may depend on
the data.
Problem with CPUs/GPUs and motivation:
Because of their data-dependent nature,
stream-joins introduce branch mispredictions
for CPUs. For GPGPUs, vectorization becomes
difficult due to control divergence of single-
instruction–multiple threads (SIMT) lanes; also,
the memory pattern can diverge between lanes,
causing bank conflicts.
To visualize the problem for CPUs, see
Figure 2, which shows both the traditional data-
flow and proposed stream-join dataflow repre-
sentation for the examples above. Here, black
arrows represent data dependence, and green
arrows indicate control.
Figure 2(a) shows that the inner product
dataflow can be mapped to a dataflow-based
processor like an out-of-order core, but only at
low throughput. To explain, note that there is a
loop-carried dependence through the control-
dependent increment and memory access. This
prevents perfect pipelining, and the throughput
is limited to one instance of this computation
every n cycles, where n is the total latency of
these instructions.
Insight: Our insight is that from the perspec-
tive of the memory, the control dependence is
mostly unnecessary, as most loads at the line-
granularity will be performed anyways. There-
fore, to break the dependence, we need to sep-
arate the loads from computation (this is what
memory streams do), then expose a pipelined
mechanism for controlling the order of data
consumption. In the sparse vector example,
we would like to reuse the larger of the two
index values for consideration (data-dependent
reuse). If the comparison instruction can treat
its inputs like a queue, and specify the reuse
behavior (i.e., pop the smaller element), this
can be accomplished in a pipelined fashion.
39
 Top Picks
Figure 2. Example restricted dependence form algorithms.
Also, the multiply–accumulate operation
is performed on only matching indices, so
we should discard some of these computations/
data. Therefore, in addition to data-dependent
reuse, we also require data-dependent discard.
The merge example [see Figure 2(b)] has
a surprisingly similar form and control
dependence loop to the sparse multiplication,
where the computation is replaced by
selecting the smaller item. A similar approach
of decoupling streams and applying data-
dependent reuse and discard will break the
control dependence loop and enable high
throughput.
Our stream-join proposal: We find the desired
behavior can be accomplished with a simple and
novel control flow model for full-throughput sys-
tolic execution. In this model, each instruction
may reuse its inputs, discard the computation,
or reset a register based on a dataflow input.
Figure 2 shows the examples written in this
model.
40
To enable flexible control interpretation,
each instruction embeds a simple configurable
mapping function from the instruction output
and control input to the control operations
fðinst out; control inÞ ! reuse1; reuse2; discard; reset:
Alias-Free Indirection
Many algorithms rely on indirect read, write,
and update to memory, often showing up as
a[f(b[i])]. Figure 2 shows two examples: The
sparse-vector/sparse-matrix outer product (c)
works by performing all combinations of non-
zero multiplications, and accumulating in the
correct location in a dense output vector. Histo-
gram (d) is straightforward. Both perform a
read–modify–write access to an indirect loca-
tion. This can be viewed as two dependent
streams. Another important observation is
that there are no unknown aliases between
streams—the only dependence is between the
load and store of the indirect update.
IEEE Micro
Figure 3. SPU microarchitecture. (a) Sparse Processing Unit (SPU). (b) SPU Core. (c) Scratchpad Controller. (d) DGRA
Processing Element.
AF-Indirect Definition: A program region that
is regular (including no implicit dependencies)
except that the address of one memory stream
may depend on another, and a stream can
encode a read–modify–write operation.
Problem for CPUs/GPUs and our motivation:
On CPUs, indirect memory is possible with scat-
ter/gather, however the throughput is limited
given the limited ports to read/write vector-
length number of cache lines simultaneously.
Also, not leveraging alias-freedom means a reli-
ance on expensive load-store queues.
Although GPUs can use their banked scratch-
pads for faster indirect access, the following two
reasons limit the indirect throughput. 1. No reor-
dering of requests across subsequent vector
warp accesses.11 Doing so in a GPU would
require dependence checking of in-flight
accesses, as they cannot guarantee alias free-
dom. 2. Atomic updates to the same scratchpad
bank are not pipelined even though they access
different memory locations. The reason is that
the lock bits for atomic operations are shared
among multiple addressable locations.1 The
coarse-granularity locking is required to reduce
the locking overhead.
To visualize the inefficiency of a typical GPU
scratchpad, see Figure 2, which shows how
scratchpad vector requests (corresponding to
indirect read and atomic update, respectively) are
served on a GPU. For simplicity, we assume a warp
size of 8. As GPUs do not reorder requests across
warps, the update request vector is issued after
the completion of all read requests. For updates in
GPU, we assume one lock-bit per scratchpad bank.
Here, the three-cycle nonpipelineable operation
further worsens the overhead of bank conflicts.
May/June 2020
Insight: Our insight is that the dependence
check is not required between subsequent
requests (e.g., corresponding to different static
loads) if alias freedom is known. Further, the
atomic operations required in these algorithms
are often low latency integer arithmetic logic
unit (ALU) operations. Therefore, for depen-
dence check, the maximum number of possible
conflicting addresses is usually low (i.e., 1 less
than the atomic update latency). Hence, we
could compare with absolute addresses instead
of relying on a serializing lock bit mechanism.
Our AF-indirect proposal: We find that the
desired behavior can be accomplished by:
1) exposing alias-freedom in the hardware–
software interface to enable interleaving across
vectors: and 2) storing absolute address of pend-
ing atomic updates (maximum 2) to enable
pipelining of nonconflicting addresses. Figure 2
shows how SPU is able to reorder requests, and
also able to pipeline atomic update requests with
initiation with no bubbles. The stall is intro-
duced in the presence of “real” dependencies, for
example, see cycle-4 in AF-Indirect reordering in
Figure 2. This is limited to a maximum two-cycle
bubble.
SPARSE PROCESSING UNIT
In this section, we first overview the primary
aspects of the design, and then provide the
details of stream-join-enabled systolic-CGRA and
the banked memory exposed to knowledge of
AF-Indirect.
Figure 3(a) shows the proposed SPU archi-
tecture. SPU cores are integrated into a mesh
network-on-chip (NoC). Each core is composed
41
 Top Picks
of the specialized memory and compute
fabric: decomposable granularity reconfigura-
ble architecture (DGRA), together with a con-
trol core for coordination among streams.
Communication/synchronization: SPU pro-
vides two specialized mechanisms for communi-
cation. First, we include the multicast capability
in the network. Data can be broadcast to a sub-
set of cores, using the relative offset in the
scratchpad. As a specialization for loading main
memory, cores issue their load requests to a cen-
tralized memory stream engine, and data can be
multicast from there to relevant cores. For syn-
chronizing on data-readiness, SPU uses a data-
flow-tracker-like mechanism to wait on a count
of remote-scratchpad writes.
SPU Core
The basic operation of each core [see Figure 3
(b)] is that the control core will first configure
the DGRA for a particular dataflow computation,
and then send stream commands to the scratch-
pad controller to read data or write to the DGRA,
which itself has an input and output port inter-
face to buffer data.
Stream-join compute fabric: DGRA: We aug-
ment a systolic CGRA to support stream-join
control and dataflow computation with arbitrary
data types. Figure 3(d) shows the microarchitec-
ture of a DGRA processing element (PE) (green
color represents control).
To implement control interpretation, we
add a control lookup table (CLT) to each func-
tional unit (FU), which determines a mapping
between the control inputs and possible con-
trol operations. This mapping is configured
along with the dataflow computation graph.
During dataflow operation, CLT consumes one
of the dataflow inputs to produce control sig-
nals for the ALU (discard), associated registers
(reset), and FIFOs connected to ALU inputs
(reuse).
In the DGRA, we enable each coarse-grained
resource to be able to be decomposed to
powers-of-two fine-grain resources. For compu-
tation, the decomposable PE can split each
coarse-grained input into multiple finer-grained
inputs [16-b inputs in Figure 3(d)], which are
used to feed two separate lower granularity
42
ALUs. Correspondingly, CLT and registers are
also composable.
To route the data from PEs, the network of
the DGRA is decomposable into multiple parallel
finer-grain subnetworks (minimum 8 b). For flexi-
ble routing, we add the ability for incoming val-
ues to shift one subnetwork per switch hop.
Alias-freedom-exposed banked memory:
Because our workloads often require a mix of
linear and indirect arrays simultaneously, for
example, streaming read of indices (direct) and
associated values (indirect), we begin our design
with two logical scratchpad memories, one
highly banked and one linear. In this design,
both exist within the same address space.
Hence, memory streams may access locations in
a remote core’s scratchpad using the similar
interface for linear and indirect streams.
The role of the scratchpad controller [see
Figure 3(c)] is to generate requests for reads/
writes to the linear scratchpad, and reads/
writes/updates to the indirect scratchpad. A
control unit assigns the scratchpad streams,
and their state is maintained in either linear or
indirect stream address generation logic. The
controller should then select between any con-
current streams for address generation and
send it to the associated scratchpad to maxi-
mize expected bandwidth. The linear address
generator’s operation is simple—create wide
scratchpad requests using the linear access
pattern.
The indirect address generator creates a vec-
tor of requests by combining each element of
the stream of addresses (coming from the
compute fabric, explained in the “Exploitable
Data-Dependence Forms” section) with each ele-
ment in the parent stream (i.e., b[i] in a[f(b
[i])]). This vector of requests is sent to an arbi-
trated crossbar for distribution to banks, and a
set of queues buffer requests for each static ran-
dom access memory (SRAM) bank until they can
be serviced.
Since there are no conflicts among indirect
read/write requests, the requests are serviced
from the top of the bank queue as soon as the
scratchpad data bus becomes available. For
atomic update requests, the requests can be ser-
viced when both scratchpad read and write
buses are available, and the updated address
IEEE Micro
does not conflict with the pending updates
issued from the same bank. As the ordering of
the data returned from read requests is critical
for dataflow operations, we employ an indirect
read reorder buffer (IROB) that maintains incom-
plete requests in a circular buffer (see Figure 2).
IROB entries are deallocated in-order when a
request’s data is sent to the compute unit.
Control ISA: We leverage an open-source Figure 4. Overall performance.
stream-dataflow ISA7 for the control core’s
implementation of streams, and add support for
indirect reads/writes/updates, stream-join data- EVALUATION
flow model, and typed dataflow graph. The ISA Our evaluation broadly addresses the question
contains stream instructions for the data trans- of whether restricted data-dependence forms
fer, including reading/writing to main memory exposed to an ISA (and exploited in hardware) can
and scratchpad. help achieve general-purpose acceleration.
Comparison to general-purpose accelerators:
Figure 4 shows how SPU fairs against CPU and
METHODOLOGY GPU for workloads across ML, graph processing,
SPU: We implemented SPU’s DGRA in Chisel,
and databases.
and implemented with an industry 28-nm tech-
The workloads with a stream-join pattern—
nology. We built an SPU simulator in gem5, using
kernel support vector machines (KSVM), TPCH
a RISCV ISA for the control core.
sort heavy queries (SH), gradient boosting deci-
Architecture comparison points: Table 1 shows
sion trees (GBDT)—achieve speedup up to 10

the characteristics of the architectures we com-
speedup over CPU due to avoiding the through-
pare against, including their on-chip memory
put-limiting cyclic dependence loop and lower
sizes, FU composition, and memory bandwidth.
computational density. The GPU also suffers
We also address whether an inorder processor
from hardware underutilization as control leads
is sufficient by comparing against “SPU-inorder,”
to masking in vector lanes.
where the DGRA is replaced by an array of eight
On workloads with AF-Indirect—fully con-
inorder cores (total of 512 cores). For reference,
nected layer (FC), convolution layer (CONV),
we also compared against a dual-socket Intel
arithmetic circuits (AC), Graph, TPCH not sort-
Skylake CPU, with 24 cores.
heavy queries (N-SH)—both GPU and SPU use a
Workload implementations: We implement
histogram-based approach. However, SPU’s
SPU kernels (both dense/sparse) for each work-
aggressive reordering of indirect updates in the
load, and use a combination of libraries and
compute-enabled scratchpad far outperforms
hand-written code to compare against CPU/GPU
the limited ordering in GPU.
versions.
Finally, the ability to support both stream-
join and AF-Indirect enables the use of new com-
Table 1. Characteristics of evaluated architectures. pression techniques like run-length encoding
efficiently. These techniques effectively reduce
Characteristics GPU SPU-inorder SPU
the required memory bandwidth, thus improv-
Processor GP104 In-order SPU-core
ing performance.
Cache+Scratch 4064 kB 2560 kB 2560 kB Even though SPU-inorder can relieve
Cores 1792 512 64 SPU cores some of the vectorization overheads suffered
by GPU, it is insufficient due to lower peak
FP32 Unit 3584 2048 2432
throughput.
FP64 Unit 112 512 160 Domain accelerator comparison: Accelerators
Max Bw 243 GB/s 256 GB/s 256 GB/s for FC,4 CONV,10 and Graphs3 all employ com-
pute-enabled banked memory to achieve high

May/June 2020
43
 Top Picks
indirect throughput. SPU is able to remain within
57% of its performance. The difference is due to
other specializations, e.g., higher radix NoC in
Graph application-specified integrated circuit
(ASIC) and higher buffer access bandwidth in
CONV ASIC.
In non-sort-heavy database workloads, the
dense version of SPU performs similar to ASIC.
With efficient stream joins, SPU is able to
catch up to and surpass database ASIC, which
spends significant area resources on special-
ized sorting units.
Benefit of decomposability: In general, we
achieve datawidth-proportional speedup by
adding decomposability. In comparison to
subword-SIMD, SPU can see 2.6
speedup by
being able to vectorize run-length decoding,
which involves control-serializing computa-
tion. Similarly, branches in AC also benefit
from decomposability (3
). Overall, SPU
achieves geomean speedup of 2.12
speedup
with decomposability.
Area and power: The two major sources of
SPU’s area are the scratchpad banks and DGRA,
together occupying more than two-third of the
total; DGRA is the major contributor to power
(assuming all PEs are active).
Compared to the whole design, adding
stream-join control in the systolic CGRA
increases area by 6.6% and power by 17.5%.
Decomposability costs another 3.1% area and
7.7% power.
CONCLUSION
This work identifies two forms of data-
dependence, which are highly specializable
and are broadly applicable to a variety of
algorithms. By defining a specialized execution
model and codesigned hardware, SPU, we
enabled the efficient acceleration of a large
range of workloads. We observed up to order-
of-magnitude speedups and significant power
reductions compared to modern CPUs and
GPUs while remaining flexible.
More important than the proposed design is
how the approach of identifying and abstracting
common dependence forms can influence the
field.
Systematic understanding of irregular
accelerators: Our restricted forms of data-
44
Table 2. Analysis of related works.
Exploitable dependence
Specialized architectures
forms
TPUv1—Dense ML
GPU—Dense
No data-dependence
LSSD8—Dense
SPU
Q10012—Database
Sparse ML6—Sparse
Algebra
Stream-Join
ExTensor5—Tensor
SPU
SCNN10—DNN CONV
EIE4—DNN FC
OuterSPACE9—Sparse
AF-Indirect
Algebra
Graphicionado3—Graphs
SPU
dependence can be used to classify and
understand the fundamental capabilities of
domain-specific accelerators. Table 2 shows
the scope of several existing domain-specific
accelerators. What we see is that each gener-
ally specializes for only one form out of
stream-join, AF-Indirect or regular algorithms.
Beyond simply understanding the space, this
way of viewing algorithm’s interaction with
architecture can improve the portability of tech-
niques across domains. Consider the context of
accelerators for sparse linear algebra. SPU’s
design can join sparse lists at one element per
cycle (per PE). An idea proposed for a sparse ML
accelerator is to vectorize the join,6 so that N
elements can be joined at once from each list
(requiring N
N comparisons). The ExTensor
accelerator,5 designed for multidimensional
sparse tensor ops, goes further. It demonstrates
that a hierarchical list intersection (a form of
stream join) can be more work efficient by skip-
ping a variable number of unmatched items in a
single step. To further reduce the memory band-
width overhead of sparsity, SparTen2 proposed
a bit-vector representation of indices. Thus,
the matched indices can be found using efficient
IEEE Micro
bit-level operations. These optimizations can formulations or definitions of restricted data-
apply to SPU. More importantly, by finding struc- dependence forms, which could lead to new
ture and commonality in the dependence forms opportunities for specialization. For example, a
across domains, it becomes clear coarser grain form of data depen-
how to apply these optimizations We think it is important dence than we have explored is
to other superficially different not to forget that data-dependent parallelism (aka
problems, like database join or systems and dynamic parallelism). At the other
decision tree training. application experts are end of the spectrum could be
Impact on general-purpose pro- constantly innovating data-dependent data types, where
cessors: This work focused on new algorithms, and at a fine grain, the data-type size
reconfigurable dataflow-like pro- are now doing so with is chosen to meet the precision
cessors for implementing depen- deep knowledge of the requirements. One could imagine
underlying hardware.
dence-form specialization. While it exposing these forms as first-class
Our results support the
was convenient, other architec- primitives in the hardware/soft-
notion that a rigid
tures can equally benefit from architecture can limit
ware interface, and each could be
such specialization. certain algorithmic plausibly useful in many domains.
approaches from being Effect on algorithms: Finally,
 Indirection in GPUs: A conceiv- viable. we think it is important not to
able extension to a GPU ISA forget that systems and applica-
could enable the annotation of tion experts are constantly inno-
a program region as being alias-free indirect vating new algorithms, and are now doing so
(informed by programmer or compiler). This with deep knowledge of the underlying hard-
would allow GPU scratchpads to eliminate ware. Our results support the notion that a
memory dependence checking and enable rigid architecture can limit certain algorithmic
aggressive reordering, leading to reduced approaches from being viable. Therefore, we
impact of bank conflicts and higher through- believe that incorporating support for struc-
put. NVIDIA’s tensor core is precedence that tured irregularity into existing and new pro-
such specialization is feasible. grammable architectures can lead to
 Stream-join SIMD: Stream-join control could innovations in novel algorithms and data
be supported in a CPU, for example, through structures.
extensions to SIMD operations. An approach
could be to add specialized instructions, ACKNOWLEDGMENTS
which allow treating registers as FIFOs, and We would like to thank G. Van den Broeck
the branch instructions may control the and A. Choi for their insights and help with arith-
order of data consumption (using simple metic circuits workloads. We would also like to
finite-state machine at FIFOs). thank D. Ott and P. Subrahmanyam for their
 Hybrid FPGAs: Recent FPGAs (Xilinx Alveo) thoughtful conversations on the nature of irregu-
include neural network accelerator units, larity and data dependence. This work was
demonstrating the need for specialization supported in part by the National Science Foun-
of even reconfigurable hardware. Increasing dation under Grant CCF-1751400 and Grant CCF-
these units’ flexibility to be similar to 1937599 and in part by the gift funding from
SPU could simultaneously provide many VMware.
of the same efficiency benefits as an
ASIC while also retaining the fundamental
value proposition of FPGAs: broad work- & REFERENCES
load efficiency while retaining fine-grain 1. J. Gomez-Luna, J. M. Gonzalez-Linares, J. I. Benavides
reprogrammability. Benitez, and N. Guil Mata, “Performance modeling
of atomic additions on GPU scratchpad memory,”
Other exploitable data-dependence forms: It IEEE Trans. Parallel Distrib. Syst., vol. 24, no. 11,
is possible that there may be alternate pp. 2273–2282, Nov. 2013.

May/June 2020
45
 Top Picks
2. A. Gondimalla, N. Chesnut, M. Thottethodi, and
T. N. Vijaykumar, “SparTen: A sparse tensor accelerator
for convolutional neural networks,” in Proc. 52nd Annu.
IEEE/ACM Int. Symp. Microarchit., 2019, pp. 151–165.
3. T. J. Ham, L. Wu, N. Sundaram, N. Satish, and
M. Martonosi, “Graphicionado: A high-performance
and energy-efficient accelerator for graph analytics,”
in Proc. 49th Annu. IEEE/ACM Int. Symp. Microarchit.,
Oct. 2016, pp. 1–13.
4. S. Han et al., “EIE: Efficient inference engine on
compressed deep neural network,” in Proc. 43rd
Annu. Int. Symp. Comput. Archit., 2016, pp. 243–254.
5. K. Hegde et al., “ExTensor: An accelerator for sparse
tensor algebra,” in Proc. 52nd Annu. IEEE/ACM Int.
Symp. Microarchit., 2019, pp. 319–333.
6. A. K. Mishra, E. Nurvitadhi, G. Venkatesh, J. Pearce,
and D. Marr, “Fine-grained accelerators for sparse
machine learning workloads,” in Proc. 22nd Asia South
Pacific Design Autom. Conf., 2017, pp. 635–640.
7. T. Nowatzki, V. Gangadhar, N. Ardalani, and
K. Sankaralingam, “Stream-dataflow acceleration,” in
Proc. 44th Annu. Int. Symp. Comput. Archit., 2017,
pp. 416–429.
8. T. Nowatzki, V. Gangadhar, K. Sankaralingam, and
G. Wright, “Pushing the limits of accelerator efficiency
while retaining programmability,” in Proc. IEEE Int. Symp.
High Perform. Comput. Archit., Mar. 2016, pp. 27–39.
9. S. Pal et al., “OuterSPACE: An outer product based
sparse matrix multiplication accelerator,” in Proc. IEEE
Int. Symp. High Perform. Comput. Archit., Feb. 2018,
pp. 724–736.
10. A. Parashar et al., “SCNN: An accelerator for compressed-
sparse convolutional neural networks,” in Proc. 44th Annu.
Int. Symp. Comput. Archit., 2017, pp. 27–40.
11. NVIDIA Whitepaper, “Cuda C best practices guide,”
May 2019. [Online]. Available: https://docs.nvidia.
com/cuda/pdf/CUDA_C_Best_Practices_Guide.pdf
12. L. Wu, A. Lottarini, T. K. Paine, M. A. Kim, and K. A. Ross,
“Q100: The architecture and design of a database
processing unit,” in Proc. 19th Int. Conf. Archit. Support
Program. Lang. Oper. Syst., 2014, pp. 255–268.
46
Vidushi Dadu is currently working toward the Ph.D.
degree with the Department of Computer Science,
University of California Los Angeles. Her current
research focuses on hardware–software codesign to
enable general-purpose acceleration. Dadu received
the B.Tech. degree in electronics and communica-
tion engineering from the Indian Institute of Technol-
ogy Roorkee. She is a student member of IEEE.
Contact her at vidushi.dadu@cs.ucla.edu.
Jian Weng is currently working toward the Ph.D.
degree with the Department of Computer Science,
University of California Los Angeles. His research
interests include analyzing and designing reconfig-
urable spatial architectures along with the associ-
ated compilation techniques. Weng received the
B.Eng. degree in computer science from Shanghai
Jiao Tong University. He is a member of the Asso-
ciation of Computing Machinery. Contact him at
jian.weng@cs.ucla.edu.
Sihao Liu is currently working toward the Ph.D.
degree with the Department of Computer Science,
University of California Los Angeles. His research
interests include spatial architecture prototyping and
design space exploration. Liu received the B.Eng.
degree in electrical engineering from Xi’an Jiaotong
University. He is a student member of IEEE. Contact
him at sihao@cs.ucla.edu.
Tony Nowatzki is currently an Assistant Professor
with the Department of Computer Science, University
of California Los Angeles. His research interests
include architecture and compiler codesign and novel
hardware/software interfaces. Nowatzki received the
Ph.D. degree in computer science from the University
of Wisconsin-Madison. He is a member of IEEE.
Contact him at tjn@cs.ucla.edu.
IEEE Micro
 Theme Article: Top Picks

Varifocal Storage:
Dynamic Multiresolution
Data Storage
Yu-Ching Hu Te I
University of California, Riverside Google
Murtuza Lokhandwala Hung-Wei Tseng
North Carolina State University University of California, Riverside

Abstract—Varifocal storage (VS) presents a new architecture that coordinates

application demands, hardware accelerators, and intelligent data storage devices to
efficiently support various input resolutions of system components, but still maintain the
flexibility and quality without additional costs. Instead of faithfully shipping the raw data,
the cross-layer design of VS allows an intelligent storage device to work directly with the
running application to generate and deliver data sets in the desired resolution and quality
before going through the narrower system interconnect. In this way, VS minimizes the
bandwidth demand from the data source and allows hardware accelerators to work on
received data without additional preprocessing. Without programmers’ hints, VS achieves
1.46
speedup on a computer with approximate hardware accelerators.

& FOLLOWING THE HINTS of Amdahl’s law, com-
puter architecture/system designers always try
to “make the common case fast” and focus on
optimizing the most time-consuming compo-
nent. However, it is so easy for us to forget that
the common case changes all the time and opti-
mizing the most common case can also intro-
duce new overhead.
Digital Object Identifier 10.1109/MM.2020.2985955
Date of publication 10 April 2020; date of current version 22
May 2020.
Modern computer systems and applications
intensively rely on heterogeneous hardware accel-
erators and algorithms inspired by approximate
computing to significantly shrink the execution
time and improve energy efficiency in compute
kernels, but leave other architectural components
remaining the same as traditional, exact comput-
ing. As a result, we have now reached a point that
the side effects of approximate computing on
accelerators, moving data and adjusting data reso-
lutions (e.g., data precision levels, summarized
results, intermediate results, and sampled

May/June 2020 Published by the IEEE Computer Society 0272-1732 ß 2020 IEEE
47
 Top Picks

contexts), have overtaken compute kernels as a to a CPU and a set of DRAM modules where the
new bottleneck in many applications. computer hosts the operating system and pro-
This research presents varifocal storage (VS), vides a synchronization point for runtime data
a new architecture that coordinates application storage, the computer may incorporate other
demands, hardware accelerators, and intelligent computing units, including general-purpose
data storage devices to efficiently support vari- computing on graphics processing units, digital
ous input resolutions of system components, signal processors or tensor processing units
but still maintain the flexibility and (TPUs), to accelerate the exe-
quality without additional costs. VS cution of compute kernels of
This research presents
revisits the task allocation of workloads. These accelerators
varifocal storage (VS),
approximate computing on general- usually accept data in different
a new architecture that
purpose computers to place tasks coordinates application precisions (e.g., 32 bits in regu-
such as raw-data retrieval, data-reso- demands, hardware lar GPU cores, 16 bits in Tensor
lution adjustment, and quality con- accelerators, and intel- Cores, 8-bits in TPUs) from the
trol in the most appropriate place ligent data storage host processor architecture
within a system in a full-stack sys- devices to efficiently (e.g., 64 bit). The computer
tem design perspective. Instead of support various input also stores input/output data
faithfully shipping the raw data, the resolutions of system persistently using solid state
storage device in VS can work components, but still drives (SSDs) or storage over
directly with the running application maintain the flexibility the network through a network
and quality without interface card (NIC).
to generate and deliver data sets in
additional costs.
the desired resolution and quality These heterogeneous hard-
before going through the narrower ware components exchange data
system interconnect. In this way, VS minimizes through the system interconnect (i.e., PCIe) where
the bandwidth demand from the data source the root complex is nowadays located on the CPU.
and decreases the most latency-critical data- Due to the limited total links available from the
transfer overhead. root complex, the system usually allocates a rela-
We evaluate VS by running a wide range of tively larger amount of links to high-throughput
applications on our prototype SSD. The ideal, accelerators (e.g., 16 PCIe lanes for GPUs). The
programmer-directed VS achieves 1.52
speedup remaining components (e.g., SSDs, NICs) can only
on average over conventional approximate com- use relatively smaller amounts of PCIe links or
puting, while the automatic VS still achieves even have to share the links with other peripherals
1.46
speedup without programmers’ hints. through a PCIe switch.
As the computer may 1) use a data set for
different purposes (e.g., an application can
DEMAND OF PRESENTING DATA SETS request an image in resolutions as high as
IN DIFFERENT RESOLUTIONS 7680
4320 pixels to display or edit, but
Figure 1 illustrates the architecture of a inferencing in a machine learning application
modern heterogeneous computer. In addition only requires 1=64 of that resolution) or 2)
compute data on accelerators with different
precisions, the computer usually stores per-
sistent data with high-resolution content and
needs to generate the input data in the
desired resolution dynamically. Figure 2
explains the resulting data-processing pipeline
of running approximate-computing applica-
tions or using these hardware accelerators in
modern heterogeneous computers.
The computer first needs to issue I/O com-
Figure 1. Architecture of a modern heterogeneous computer. mands for the storage device to access raw data

IEEE Micro
48

Figure 2. Data-processing pipeline of approximate
applications using the conventional execution model.
optimized I/O library, the overhead of receiving/
preparing data sets exceeds the kernel execution
as the most critical stage in a majority of these
applications. We expect the gap in Figure 3 to grow
with the relatively fast evolution of hardware accel-
erators, but slowly improved I/O and storage
technologies.

VS SYSTEM ARCHITECTURE
Figure 4 shows VS in a heterogeneous com-
from its internal data arrays and then transfer the
puter system. VS revisits the storage-system
raw data through the underlying system intercon-
stack to allow the device to dynamically produce
nect while simultaneously serving other data-
data with different resolutions on demand. The
access requests. Once the host computer receives
VS core layer resides inside the storage device
a chunk of data, the CPU can start producing data
to change data resolutions presented to applica-
sets in lower resolutions. The compute kernel can
tions. The VS layer interacts with an existing sys-
then perform computations using the resolution-
tem I/O interfaces and provides an extended
adjusted data sets. If the kernel can leverage a
interface for resolution adjustments. The VS
hardware accelerator, the system must addition-
layer also works together with the SSD manage-
ally exchange among different components
ment layer (i.e., the flash translation layer in
through the interconnects before the accelerator
flash-based, SSDs) to locate the requested data.
can compute on the prepared data.
The host system needs an extended kernel
With these highly optimized approximate-com-
driver and API functions for the applications to
puting-based acceleration techniques but rela-
send requests, exchange data, and receive feed-
tively limited bandwidth for data exchange, the
back from the VS core layer. The host applica-
latency of retrieving and preparing data for approx-
tion interacts with the API and sends commands
imate-compute kernels becomes the most critical
specifying operators that VS should apply to the
stage in the data-processing pipeline. Figure 3 com-
raw data.
pares the latency of receiving raw data chunks
The VS core layer supports a set of operators
from a high-end NVM-Express (NVMe) storage
that are especially effective for applications that
device against the execution time of performing
contain high data-level parallelism, but are able
approximate/mixed-precision compute kernels on
to tolerate inaccuracies in data sets. The VS core
the same data chunks using an NVIDIA Tesla T4
GPU for a set of applications. Using a highly

Figure 3. Data-preparation overhead compared

against the execution time of performing compute
kernels on the same amount of data. Figure 4. System architecture of VS.

May/June 2020
49
 Top Picks
Figure 5. Data-processing pipeline of VS.
layer is also where the system performs mecha-
nisms that automatically determine the most
appropriate data resolution for quality control.
The host application can optionally enable VS’s
quality control mechanisms through VS’s API
and the kernel driver.
Figure 5 demonstrates the data processing
pipeline in VS. By using operators and quality-
control mechanisms inside the storage device,
VS allows the storage device to send adjusted
data sets to the system main memory instead of
always sending raw data as using conventional
storage devices. Then, the approximate comput-
ing components can directly work on lower reso-
lution data, without additional conversion on
the host CPU.
The proposed VS architecture allows the sys-
tem to successfully tackle the challenges of per-
formance, quality, flexibility, and cost in
accommodating various types of workloads on
modern heterogeneous computers.
Performance. VS improves performance
through 1) exploiting the richer internal band-
width and 2) reducing the total volume of data
movements in the system interconnect.
As Figure 1 depicts, the controllers found in
modern datacenter SSDs, including the control-
lers in the prototype SSD that we used for this
work, support multiple concurrent channels.
The internal bandwidth of the prototype SSD in
this article can reach up to 8 GB/s by enabling 16
channels. However, conventional architecture
and storage interfaces waste the rich internal
bandwidth of storage devices since the applica-
tion only works on the host computer and
exchanges data with the SSD using limited PCIe
bandwidth. In contrast, VS adjusts data using
the SSD controller, the mechanisms in the VS
core layer directly interact with raw data
50
without the dynamics of the system intercon-
nect bandwidth and the competitions with other
hardware components in CPU-memory bus. In
addition, VS frees up CPU resources to tackle
more useful workloads, leading to performance
gains for approximate applications on the host
side.
Since approximate computing works on
lower resolution data sets, the compute kernels
usually consume fewer bytes than exact comput-
ing ones. VS sends only adjusted data to com-
pute units so that VS reduces the size of data
going through the system interconnect. In this
way, VS improves the total latency of transfer-
ring data, mitigates the idle time in compute
units, and decreases the bandwidth demand in
the rest of the system interconnect links.
Although lossy and lossless data-compres-
sion algorithms help us to reduce data size and
save I/O bandwidth, the overhead of decom-
pressing data on the destination computing
device can easily cancel the benefit of reducing
data-transfer time.1
Quality. VS provides an additional layer of
quality control for applications by capturing
low-quality input that failed the requirement
before the data leave the storage device. With-
out VS, existing quality-control mechanisms
must request full-size, raw data from the storage
device, and compare subsets of results for exact
and approximate computation.2–6
Flexibility. In the VS architecture, the stor-
age device dynamically generates data with dif-
ferent resolutions to accommodate the demands
for diverse applications. Without an architecture
like VS, the storage system must store multiple
versions of a shared data set or provide raw
data to the host for preprocessing, hurting either
space efficiency or performance.
If the storage device stores data using lossy
algorithms to save bandwidth, the system sacri-
fices support for exact-computing. Similarly,
existing research on “approximate storage sys-
tems” proposes to store data using unreliable
memory cells, but not faithfully store raw data.
Therefore, systems that use them can neither
support exact computation nor dynamically gen-
erate data in different resolutions.7,8
IEEE Micro
 Cost. The VS core layer can leverage exist-
ing SSD controllers and minimize extra hardware
costs for the following reasons. 1) Empirical
studies,9,10 as well as our measurements in the
unmodified prototype SSD, reveal that the SSD
controller cores are mostly idle due to the rela-
tively long latency of accessing NVM devices
and the overprovisioning of processing power.
2) The critical path of the data-access pipeline is
determined by either the access time of flash
chips or the latency of the DMA stage, leaving
slacks that can be taken up by VS to apply opera-
tors without the need for additional accelera-
tors. 3) SSD controllers enjoy the benefits of
exclusive resources within the storage device
and can perform data adjustment more effi-
ciently than the host CPU.
The rest of this section will briefly describe
the current programming model, operators,
quality control mechanisms, and architectural
support of VS.
Programming Model
To prepare an application to take advantage
of the VS model, the programmer uses the VS
library to specify data resolutions and retrieve
adjusted data for the application. These library
functions help the application to set up 1) the
operators required to read data and whether
Figure 6. KMeans code sample with inserted VS
function calls.
May/June 2020
any quality control mechanism is enabled, and
2) the parameters that allow the underlying stor-
age device to adjust data as well as control varia-
bles that quality control mechanisms use to
assure the quality of the adjusted data.
Figure 6 shows the KMeans code with VS func-
tion calls inserted. The modified KMeans code ini-
tiates VS calling vs_setup to set the desired
operator, resolution, and the data format. VS
starts adjusting data only if the application calls
the vs_read function. This function resembles
the existing Linux read function except that
1) the resulting data size may be different from
the requested data size, since operators will trim
data sizes in most cases, and 2) the function will
provide feedback regarding the resolution that
VS selects. If the program calls a regular read
function to read data, VS will act as a conven-
tional data storage but not change the data
resolution.
If VS successfully adjusts the data, the appli-
cation can use a compute kernel that supports
lower resolution input (e.g., cluster_approxi-
mate) to further reduce the total execution time
of the program. Depending on the approximate
compute kernels that the application uses, the
programmer can choose different VS operators
for data adjustments when calling the vs_setup
function. In addition to the programmer’s choice
of resolutions, the programmer can optionally
enable VS’ quality control mechanisms, Autofo-
cus and iFilter. Autofocus can automatically
decide the resolution using a set of control varia-
bles for a chosen operator. The decisions that
Autofocus make are usually more conservative
than those of a programmer, but Autofocus can
nonetheless help applications adapt to data
sets. If a given application can apply multiple
versions of approximate kernels for different VS
operators, the programmer can use the iFilter
mechanism to let VS choose both the most
appropriate operators and resolutions for each
data set.
VS Operators
VS provides a set of operators to adjust data
resolutions and expose these operators through
the NVMe interface as well as the system API. VS
operators are selected under the following crite-
ria: 1) The computation overhead must match
51
 Top Picks
the processing power inside the storage device.
Therefore, VS can minimize the impact on access
latency and power consumption and avoid extra
hardware costs. 2) A wide range of applications
must be able to apply the operator, thereby
allowing for more efficient use of valuable device
resources (VS identifies the most useful opera-
tors from previous efforts11,12). 3) The operator
must allow VS to take advantage of mismatches
between external and internal bandwidths and
downsize the outgoing data. These operators
can flexibly support various resolutions and
accommodate exact computing.
The current VS framework supports the fol-
lowing categories of operators for diverse data
types.
Data Packing: The data-packing operator
trims the data set size by using fewer bytes to
express each item and by condensing the layout
in memory. Since the data-packing operator
translates raw data into a less-precise data type,
it can potentially decrease accuracy (e.g., dou-
ble!float!half or int64!int32!short!char).
Quantization: The quantization operator
rescales the raw values into a smaller value
space as well as preserves the relative order of
values. The quantization operator applicable to
the application requires a large value space.
Reduction/Tiling: The reduction operator
applies a function (e.g., average) to a group of
input values and yields a single output value. After
applying a reduction operator, VS sends only the
resulting value of each group to reduce the amount
of data passing through the system interconnect.
Sampling: The sampling operator chooses a
subset of items from the raw data and sends the
selected items to the host computer. Operators
in this category can perform uniform/random
data selection or report only the most represen-
tative data. The sampling operator can poten-
tially achieve the same effect as that of loop
perforation but without any code modification.
First-Level Quality Control in the Storage
Devices
If the input quality is way too far from the ori-
gin, the approximate computing can hardly gen-
erate meaningful results.3 Therefore, we
designed two quality control mechanisms, Auto-
focus and iFilter, both control the quality of
52
adjusted data, avoiding the cases of sending data
that fail to pass the quality control knobs to the
host before the computation occurs. In contrast,
all previous research projects censor computa-
tion results and always require at least part of
raw data to present in the host main memory as
well as being computed using exact computing.
Autofocus allows the programmer to simply
specify the desired VS-operator, letting VS
decide the most appropriate resolution that
every checked piece of the adjusted data suc-
cessfully passes through the pre-defined, opera-
tor-dependent threshold values. If low-
resolution data failed on the quality control
knobs, VS will reject low-resolution data and
gradually use higher resolutions until the quality
fits the demand and apply this resolution for the
same data set later. Utilizing another important
observation from previous research that a small
subset of input data is representative of the rest
of the input data in approximate-computing
applications that tolerate inaccuracies,3 Autofo-
cus selects the resolution using only a small por-
tion of the raw input data from a requested data
set and then monitors the quality of the adjusted
input data.
iFilter can work without programmer input
and is more effective than Autofocus for appli-
cations having compute kernels that are com-
patible with multiple VS operators. The iFilter
algorithm is similar to the Autofocus algorithm
in which it selects the most appropriate resolu-
tion for each compatible operator, except that
iFilter will keep track of the resolution and the
resulting data size for each operator. After
selecting an operator that passes all quality
control variables and generates the smallest
data size among all passing operators, iFilter
will enter the monitoring phase as in
Autofocus.
Building a VS-Compliant Storage Device
Building a VS-compliant storage device
means tackling challenges associated with 1)
providing a hardware/software interface that
allows applications to describe the resolutions
and quality of the target data, and 2) minimizing
the computational overhead/cost of adjusting
data resolutions. VS overcomes the former chal-
lenge by extending the NVMe interface; this
IEEE Micro
requires the fewest modifications to the system exact computing is 7% slower than the conven-
stack and applications. VS addresses the latter tional approximate computing.
challenge by exploiting the idle cycles available Several groups of our workloads shared the
in modern SSD controllers. same data sets, but applied different operators
and resolutions to accommodate each individual
demand. Without an architecture like VS, the
storage system must store multiple versions of a
RESULTS
shared data set or provide raw data to the host
In this article, we built a VS-compliant SSD by
for preprocessing, hurting either space effi-
extending a commercialized, datacenter-class
ciency or performance.
SSD. We attached the VS-compliant SSD to a
We also compare our mechanisms with other
high-end heterogeneous machine with a GPU.
alternatives. Autofocus outperforms a state-of-the-
The host operating system contains the
art quality control mechanism by up to 2.86

extended NVMe driver to support additional VS
because VS does not require the storage device to
NVMe commands. We measured the perfor-
deliver raw data to the host. This article also mea-
mance of the resulting system with several work-
sured that VS outperforms the best compression
loads that span a wide range of applications.
algorithm for each data set by 4.40

Figure 7 shows the relative end-
As an optimization since VS incurs zero overhead in
to-end latency of running a com-
usually comes with its decoding data on the host side.
plete workload using workloads
with the conventional approximate own overhead, we really
computing approach with GPU- need to holistically revisit
accelerated kernels as the baseline.
the interactions among CONCLUSION
different system/archi- As an optimization usually
Since VS efficiently prepares input
tectural components to comes with its own overhead, we
data sets in storage devices for
negate the introduced really need to holistically revisit
approximate computing kernels side effects to scale up the interactions among different
running on the GPU, the manual performance with new
system/architectural components
programmer-directed VS leads to a technologies.
to negate the introduced side
speedup of 1.52
for these applica-
effects to scale up performance
tions. VS also achieved an average
with new technologies. This article,
energy savings of 32% for applica-
in particular, demonstrates the case of modern
tions compared to the conventional approximate
heterogeneous computers running both exact
computing.
and approximate computing applications. By
Using Autofocus to dynamically select the
making the demand of applications visible to the
desired data resolutions, these applications
storage system and also making the computing
achieve an average speedup of 1.43
. Without
capability available for preparing data in different
any programmer intervention, iFilter can
resolutions, this article fundamentally alleviates
improve performance by 1.46
. In contrast, the
the side effects from the conventional, single-
point-optimization design—the data adjustment
and movement overhead.
The resulting architecture not only hides
the latency of data adjustment within the NVM
data access pipeline of the storage device, but
more importantly, it exploits the fact that
approximate computing only needs lower reso-
lution inputs to further reduce the data volume
flowing through the system interconnect. With-
out a full-stack, holistic design like this work,
we can never take full advantage of approxi-
Figure 7. Speedup of the end-to-end latency. mate computing.

May/June 2020
53
 Top Picks
This article leverages the success of near-
data/in-storage processing to implement the pro-
posed idea. However, in addition to “offloading
computation” that prior work focusing mainly
on, this article reveals the potential of “offering
new features” (e.g., the quality control mecha-
nisms in VS) to streamline the rest of computa-
tion. As the budget of building storage devices is
usually very limited, the processor near-data/in-
storage cannot compete with host CPUs and
hardware accelerators. We hope this work could
inspire researchers in inventing more “add-on”
features that fit the capabilities in these devices
to improve the application performance.
We also expect the outcome of this article
inspires researchers to further discover those
issues introduced by local optimizations and con-
sider the presence of heterogeneous computing
resources, or intelligent data storage and I/O
devices to achieve glocal optimizations as we
demonstrated in this article. More research on
hardware/software interfaces that do not hide
power but maintain good tradeoffs on program-
mability, simplicity, flexibility, and efficiency is
necessary for emerging computer architectures.
ACKNOWLEDGMENTS
We would like to thank the AI infrastructure
group and academic relations group from Face-
book in providing insights on datacenter work-
loads and research award funding. We gratefully
acknowledge the support of NVIDIA Corporation
with the donation of the Quadro P5000 GPU used
in the early phase of this research. We also appre-
ciate Professor Steven Swanson from the Univer-
sity of California, San Diego, in supporting the
development of our prototype SSD. This work was
sponsored by the two National Science Founda-
tion Awards 1940046 and 1940048. This work was
also supported by new faculty startupfunds from
North Carolina State University and University of
California, Riverside.
& REFERENCES
1. Y. Li et al., “A network-centric hardware/algorithm co-
design to accelerate distributed training of deep
neural networks,” in Proc. 51th Annu. IEEE/ACM Int.
Symp. Microarchit., 2018, pp. 175–188.
54
2. D. S. Khudia, B. Zamirai, M. Samadi, and S. Mahlke,
“Rumba: An online quality management system for
approximate computing,” in Proc. ACM/IEEE 42nd
Annu. Int. Symp. Comput. Archit., Jun. 2015,
pp. 554–566.
3. M. A. Laurenzano, P. Hill, M. Samadi, S. Mahlke, J.
Mars, and L. Tang, “Input responsiveness: Using
canary inputs to dynamically steer approximation,” in
Proc. 37th ACM SIGPLAN Conf. Program. Lang.
Design Implementation, 2016, pp. 161–176.
4. H. Hoffmann, S. Sidiroglou, M. Carbin, S. Misailovic, A.
Agarwal, and M. Rinard, “Dynamic knobs for
responsive power-aware computing,” in Proc. 16th Int.
Conf. Archit. Support Program. Lang. Operating Syst.,
2011, pp. 199–212.
5. A. Sampson, W. Dietl, E. Fortuna, D. Gnanapragasam,
L. Ceze, and D. Grossman, “Enerj: Approximate data
types for safe and general low-power computation,” in
Proc. 32nd ACM SIGPLAN Conf. Program. Lang.
Design Implementation, 2011, pp. 164–174.
6. X. Sui, A. Lenharth, D. S. Fussell, and K. Pingali,
“Proactive control of approximate programs,” in Proc.
21st Int. Conf. Archit. Support Program. Lang. Oper.
Syst., 2016, pp. 607–621.
7. A. Sampson, J. Nelson, K. Strauss, and L. Ceze,
“Approximate storage in solid-state memories,” in
Proc. 46th Annu. IEEE/ACM Int. Symp. Microarchit.,
2013, pp. 25–36.
8. S. Ganapathy, A. Teman, R. Giterman, A. Burg, and
G. Karakonstantis, “Approximate computing with
unreliable dynamic memories,” in Proc. IEEE 13th Int.
New Circuits Syst. Conf., Jun. 2015, pp. 1–4.
9. J. Zhang and M. Jung, “Flashabacus: A self-
governing flash-based accelerator for low-power
systems,” in Proc. 13th EuroSys Conf., 2018,
pp. 15:1–15:15.
10. G. Koo et al., “Summarizer: Trading bandwidth
with computing near storage,” in Proc. 50th
Annu. IEEE/ACM Int. Symp. Microarchit., 2017,
pp. 219–231.
11. M. Samadi, J. Lee, D. A. Jamshidi, A. Hormati, and S.
Mahlke, “Sage: Self-tuning approximation for graphics
engines,” in Proc. 46th Annu. IEEE/ACM Int. Symp.
Microarchit., 2013, pp. 13–24.
12. M. Samadi, D. A. Jamshidi, J. Lee, and S. Mahlke,
“Paraprox: Pattern-based approximation for data
parallel applications,” in Proc. 19th Int. Conf.
Archit. Support Program. Lang. Oper. Syst., 2014,
pp. 35–50.
IEEE Micro
Yu-Ching Hu is currently working toward the Ph.D.
degree with the Department of Computer Science and
Engineering, University of California, Riverside. His
research interests focus on improving the performance
of database and machine learning applications
through optimizing their interactions with heteroge-
neous computing units and storage systems. He is a
member of IEEE. Contact him at yhu130@ucr.edu.
Murtuza Lokhandwala is currently working as a
Design Verification Engineer. His interests include
system design and architecture for processors, digi-
tal design, and verification. Lokhandwala received
the master’s degree in computer engineering from
North Carolina State University. Contact him at
mlokhan@ncsu.edu.
May/June 2020
Te I is currently a Software Engineer at Google, work-
ing in the Google Translate Team. Te I received the
M.S. degree in computer science from North Carolina
State University. Contact him at tei@google.com.
Hung-Wei Tseng is an Assistant Professor with the
Department of Electrical and Computer Engineering,
University of California, Riverside. His research inter-
ests include heterogeneous computer architectures
and nonvolatile memory based storage systems as
well as their programming languages, runtime sys-
tems, compilers, and applications. Tseng received
the Ph.D. degree in computer science from the Uni-
versity of California, San Diego. Contact him at
htseng@ucr.edu.
55
 Theme Article: Top Picks

AsmDB: Understanding
and Mitigating Front-End
Stalls in Warehouse-Scale
Computers
Nayana Prasad Nagendra Christos Kozyrakis
Princeton University Stanford University
Grant Ayers Trivikram Krishnamurthy
Google Nvidia
David I. August Heiner Litz
Princeton University University of California, Santa Cruz
Hyoun Kyu Cho and Svilen Kanev Tipp Moseley and
Google Parthasarathy Ranganathan
Google

Abstract—It is well known that the datacenters hosting today’s cloud services waste
a significant number of cycles on front-end stalls. However, prior work has provided little
insights about the source of these front-end stalls and how to address them. This work
analyzes the cause of instruction cache misses at a fleet-wide scale and proposes a new
compiler-driven software code prefetching strategy to reduce instruction caches misses
by 90%.

& DUE TO THE continued growth of cloud-based
digital services, warehouse-scale computers
(WSC) are now serving billions of devices across
Digital Object Identifier 10.1109/MM.2020.2986212
Date of publication 16 April 2020; date of current version 22
May 2020.
the world. This massive growth necessitates
improving the cost and efficiency of WSCs
through microarchitectural and system software
based optimizations.
WSC workloads are characterized by deep
software stacks in which individual requests can
traverse many layers of data retrieval, data

0272-1732 ß 2020 IEEE Published by the IEEE Computer Society IEEE Micro
56
processing, communication, logging, and moni- code fragmentation and the perils of micro-optimi-
toring. As a result, the instruction working set zation; and iii) a novel software-based code pre-
sizes of WSC workloads today are often 100 fetch algorithm for reducing i-cache misses at
larger than server-class L1 instruction caches (i- fleet-wide scales.
cache)1 and are currently expanding at rates of
over 20% per year.2 As cache sizes have not
improved significantly over the last many years, AsmDB: A WSC ASSEMBLY
WSC workloads are becoming increasingly front- DATABASE
end bound. Thus, processors are no longer able To enable the necessary horizontal analysis
to sustain a high instruction fetch rate, manifest- and optimization across the server fleet, we built
ing itself in large unrealized performance gains a continuously updated assembly database
due to front-end stalls, which are dominated by (AsmDB) to collect instruction- and basic-block-
increased i-cache misses. While prior work has level information for most observed CPU cycles
identified the growing impor- across the thousands of real produc-
tance of this problem, to date, To enable the tion services executing across the
there has been little analysis of necessary horizontal Google fleet. AsmDB aggregates
the sources of these misses analysis and instruction and control-flow data col-
and of available opportunities optimization across the lected from hundreds of thousands of
to address them. server fleet, we built a machines each day and grows by mul-
We corroborate this chal- continuously updated tiple TiB each week. We have been
lenge for our WSCs on Google assembly database continuously populating AsmDB over
web search leaf servers, in (AsmDB) to collect
several years with the goal of provid-
which 13.8% of the total per- instruction- and
ing easy-to-query assembly-level infor-
formance potential is wasted basic-block-level
information for most
mation for nearly every unique
due to “front-end latency,” instruction executed in our WSCs. We
observed CPU cycles
principally caused by i-cache demonstrate several cases where
across the thousands
misses. We also measured L1 AsmDB proves invaluable for front-
of real production
i-cache miss rates of 11 misses
services executing end optimization, including spotting
per kilo-instruction, and a across the opportunities for manual optimiza-
hot steady-state instruction Google fleet. tions, finding areas for improvement
working set of approximately
in existing compiler passes, as well as
4 MiB. This is significantly
for serving as a data source for a
larger than the sizes of the L1 and L2 caches on
novel compiler-driven technique to improve i-
today’s server CPUs, but small and hot enough
cache hit rates.
to easily fit and remain in the shared L3 cache
1 AsmDB is an always-on, massive-scale fleet-
(typically 10 s of MiB).
wide performance monitoring system. It uses
To understand and improve the i-cache
hardware support to collect bursty execution
behavior of WSC applications, we focus on tools
traces, performs fleet-wide temporal and spatial
and techniques for “broad” acceleration* of thou-
sampling, and leverages sophisticated offline post-
sands of WSC workloads. At the scale of a typical
processing to construct full-program dynamic
WSC server fleet, performance improvements
control-flow graphs. Collecting and processing
of a few percentage points (and even sub-1%
improvements) lead to millions of dollars in profiling data from hundreds of thousands of
cost and energy savings, as long as they are machines is a daunting task by itself. However, we
widely applicable across workloads. To that end, have carefully designed the system architecture
our work provides three primary contributions: such that it can capture and process profiling data
i) A methodology for analyzing instruction profiles in a cost-efficient way while still processing tera-
at a fleet-wide scale; ii) detailed insights about bytes of data each week.
A fleet-wide assembly database, such as
* AsmDB, provides a scalable solution to search
“Deep” acceleration would involve focusing on a handful of workloads and
trying to recover most of the  15% performance opportunity. for performance antipatterns and opens up new

May/June 2020
57
 Top Picks
Figure 1. Fleet-wide distribution of executed
instructions, and L1- and L2-instruction misses over
unique cache lines. Like instructions, misses also
follow a long tail.
opportunities for performance and total-cost-of-
ownership optimizations. WSC servers typically
execute thousands of unique applications, so
the kernels that matter most across the fleet
(the “datacenter tax”2) may not be significant for
a single workload and are easy to overlook
in application-by-application investigations.
We leverage AsmDB’s fleet-wide data in several
case studies to understand and improve the
i-cache utilization and IPC of WSC applications.
We further correlate AsmDB with hardware
performance counter profiles collected by a
datacenter-wide profiling system—Google-wide
profiling (GWP)4—to reason about specific pat-
terns that affect front-end performance.
WSC APPLICATION ANALYSIS WITH
AsmDB
WSC applications are well-known for their long
instruction tails and flat execution profiles.2
Figure 2. Normalized execution frequency versus function size
for the top 100 hottest fleet-wide functions. memcmp is a clear
outlier.
58
Figure 1 shows that i-cache misses in WSCs have a
similar long tail. It plots the cumulative distribu-
tion of dynamic instructions, and L1-I and L2-I
misses over unique i-cache lines over a week of
execution, fleet wide. The zoomed-in view of the
graph shows that the miss cumulative distribution
function (CDF) initially has a more significant
slope than the instruction CDF, suggesting that
there exist some pointwise manual optimizations
with high potential performance gains. However,
the distribution of misses quickly tapers off. In
particular, addressing just two-thirds of dynamic
misses requires optimizations in  1M code loca-
tions, which is only conceivable leveraging auto-
mation. This points us toward exploring scalable,
automated solutions—with compiler and/or hard-
ware support and no developer intervention—to
exploit these behaviors.
EFFECTS OF CODE FRAGMENTATION
ON CACHES
Code bloat and unnecessary instruction com-
plexity, especially in frequently-executed code,
can lead to excessive i-cache pressure. We ana-
lyze code bloat in Figure 2, leveraging AsmDB-
data—it plots the normalized function hotness
(how often a particular function is called over a
fixed period) versus the function’s size in bytes
for the 100 hottest functions in our WSCs. Per-
haps unsurprisingly, it shows a loose negative
correlation: Smaller functions are called more fre-
quently. It also corroborates prior findings that
low-level library functions (“datacenter tax”2),
and specifically memcpy and memcmp, are
among the hottest in our examined workloads.
However, despite smaller functions being sig-
nificantly more frequent, they are not the major
source of i-cache misses. Overlaying miss pro-
files from GWP onto Figure 2 (shading), we
notice that most observed cache misses lie in
functions larger than 1 KiB in code size, with
over half in functions larger than 5 KiB. Most
functions of 5 KiB or larger exhibit inlined call
stacks of ten or more layers in depth.
While deep inlining is crucial for performance
in workloads with flat callgraphs, it exponen-
tially increases the amount of code loaded into
the i-cache at each inline level, of which often
only a small fraction is hot. Cold code brought
IEEE Micro
Figure 3. Fraction of hot code within a function among the 100 hottest fleet-wide functions. From the left-hand
side to right-hand side, “hot code” defined as covering 90%, 99%, and 99.9% of execution.
into the cache, in addition to the necessary hot
instructions leading to hot/cold fragmentation
and thus suboptimal utilization of the limited
cache resources.
We more formally define fragmentation to be
the fraction of code (in bytes) that is necessary
to cover the last 10%, 1%, or 0.1% of executions
of a function. Because functions are sequentially
laid out in memory, these cold bytes are very
likely to be brought into the cache by next-line
prefetchers. Intuitively, this definition measures
the fraction of i-cache capacity potentially
wasted by loading cold cache lines.
We find that intrafunction fragmentation is
especially prevalent. Even after compiling with
feedback-directed optimization, 50% of the
codes in all functions are cold, frequently
interleaved with hot code sections, and thus
practically never executed despite being likely
to be in the cache. This is true even among the
hottest and most well-optimized functions in
our server fleet.
Using AsmDB data, we calculate the measure
of fragmentation for the top 100 functions by
execution count in our server fleet. Figure 3 plots
it against the containing function size. If we con-
sider code covering the last 1% of execution as
“cold,” 66 functions out of the 100 are comprised
of more than 50% cold code. Even with a stricter
definition of cold (<0.1%), 46 functions have
more than 50% cold code. Perhaps not surpris-
ingly, there is a loose correlation with function
size—larger (more complex) functions tend to
have a larger fraction of cold code.
We attribute the intrafunction fragmentation
to the deep inlining that the compiler needs
May/June 2020
to perform when optimizing typical WSC flat
execution profiles. Hence, this suggests that
combining inlining with more aggressive hot/
cold code splitting can achieve better i-cache uti-
lization, freeing up the scarce capacity.
On a finer granularity, we find that the indi-
vidual cache lines are also often fragmented
and waste cache capacity, especially for small
functions. Unlike cold cache lines within a
function, cold bytes in a cache line are always
brought in along with the hot ones, introduc-
ing an even more significant performance
issue. This suggests that there exist opportuni-
ties to improve the basic-block layout, at link
or postlink time, when compiler profile infor-
mation is precise enough to reason about spe-
cific cache lines.
We provide a concrete example of optimizing
code bloat and fragmentation by focusing on
memcmp, one of the hottest functions contribut-
ing to cache misses. memcmp clearly stands
out of the correlation between call frequency
and function size in Figure 2. It is both extremely
frequent, and at almost 6 KiB of code, 10 larger
than memcpy, which is conceptually of similar
complexity. Examining its layout and execution
patterns (see Figure 4) suggests that it does
suffer from a high amount of fragmentation, as
we observed fleet wide in the previous section.
While covering 90% of executed instructions in
memcmp only requires two cache lines, getting
up to 99% coverage requiring 41 lines or 2.6 KiB of
cache capacity. Not only is more than 50% of the
code cold, it is also interspersed with hot regions,
increasing the likelihood to be brought in by next-
line prefetchers. Such code bloat is costly—
59
 Top Picks
Figure 4. Instruction execution profile for memcmp. 90% of
dynamic instructions are contained in 2 cache lines, covering 99%
of instructions requiring 41 i-cache lines.
performance counter data collected by GWP indi-
cate that 8.2% of all i-cache misses among the 100
hottest functions are from memcmp alone.
While conceptually simple, our version of
memcmp was highly optimized for microbench-
marks and contained many code paths for specific
input variations. We show that in WSC environ-
ments where cache capacity is especially con-
strained, it is actually better to provide a reduced
version of memcmp containing only a few paths
and that doing so improves fleet-wide performance
by up to 1%.
SOFTWARE PREFETCHING
FOR CODE
Looking into the instructions that lead to i-
cache misses, we find that, while not particularly
concentrated in specific code regions, most i-
cache misses still share common characteristics.
Specifically, missing instructions are often the
target of control-flow-changing instructions with
large jump distances.3 We find that distant
branches and calls that are not amenable to tra-
ditional cache locality or next-line prefetching
strategies account for a large fraction of cache
misses among WSC applications.
For misses at the target of a distant jump, we
propose and evaluate a profile-driven optimiza-
tion technique that intelligently injects software
prefetch instructions for code into the binary
during compilation. We outline the design of the
necessary “code prefetch” instruction, which is
similar in nature to existing data prefetch instruc-
tions, except that it fetches into the L1 i-cache
60
Figure 5. Fan-in for some misses can grow very fast
with distance, especially for library functions.
and utilizes the I-TLB instead of the D-TLB. The
implementation of such an instruction has negli-
gible hardware cost and complexity compared to
pure hardware methods and is commercially via-
ble today. While it can be implemented on top of
a wide variety of hardware front-ends, we demon-
strate its viability on a system that employs only
a next-line instruction prefetcher.
Prefetching represents a prediction problem
with a limited window of opportunity. Effective
prefetches are both accurate and timely—they
only bring in useful miss targets and do so
neither too early nor too late in order to mini-
mize early evictions and cache pollution. As a
result, an effective prefetcher would have high
overall miss coverage. Our prefetch insertion
algorithm uses profile feedback information
from AsmDB and performance counterprofiles
to ensure timely prefetches with minimal
overhead.
Some of the challenges that arise among soft-
ware prefetching techniques include—fan-in, the
number of potential paths leading to a miss
increases as the prefetch injection site is moved
backward from a missed target. Figure 5 shows
the fan-in for the top 20 i-cache misses from a web
search profile. In several cases, the number of
paths leading in to a single miss exceeds 100 even
with a lookback of only ten instructions. Our
approach leverages profiling information to only
insert helpful prefetches, increasing coverage and
minimizing fan-in. Fan-out poses another challenge
in finding the prefetch injection site as not all exe-
cution paths are likely to lead to the miss. We
address this by pruning paths that exceed a maxi-
mum fan-out threshold. Furthermore, instruction
prefetches themselves increase the code footprint
and hence need to be inserted carefully.
IEEE Micro

Figure 6. Miss coverage and performance
improvement for the best-performing configuration
for each workload.
At its core, our prefetch injection strategy
leverages the observation that the injection site
of a prefetch instruction can be freely moved
within the window of opportunity to minimize
fan-in and fan-out. We call this approach dynamic
window injection. At a high level, our prefetch
procedure first constructs the execution history
for each miss and then traverses the control
flow graph in the reverse direction until it
reaches the end of the instruction window, cal-
culated based on the application-level IPC. Next,
prefetch injection sites are searched for each
miss among each of its execution paths, which
have minimal fan-in and fan-out. Prefetch
instructions are then automatically inserted in
the selected injection sites for the correspond-
ing misses as part of the final linking steps.
We prototype the effects of our proposed
software prefetching technique on memory
traces from several WSC workloads. We evaluate
on a modified version of the zsim simulator,5 by
using the system parameters modeled against an
Intel Haswell datacenter-scale server processor.
We focus primarily on three WSC applications—
a web search leaf node, an ads matching service,
and a knowledge graph back-end. For each work-
load, we collect traces during a representative
single-machine load test, which sends realistic
loads to the server under test.
Figure 6 shows that our prefetching tech-
nique is able to eliminate 91%–96% of all i-cache
misses, with a performance improvement pro-
portional to the front-end boundedness of the
application and the gap left from NLP. In all
May/June 2020
cases, fewer than 2.5% of additional dynamic
instructions are added for code prefetches.
LONG-TERM IMPLICATIONS
With increased technological growth, WSCs
now serve billions of devices and applications
across the planet. Due to their success, we expect
an ever-greater reliance on WSCs in the near
future, providing faster, more reliable, and more
secure services to society. These increasing dem-
ands necessitate achieving higher performance for
WSCs in order to be cost- and energy-efficient for
WSC companies and their customers while simul-
taneously reducing the environmental impact on
our world.
In combination with the slowdown of Moore’s
law, improving the efficiency of existing hardware
in WSCs becomes even more critical. We analyzed
a web search binary, showing that 68% of the CPU
performance potential is lost due to pipeline stalls,
of which 13.8% are due to the front-end not being
able to deliver instructions fast enough.
This article addresses the front-end bottle-
neck on following fronts.
 First, we have built a tool that is capable of
collecting data from live datacenter applica-
tions at the granularity of instructions and at
the scale of a WSC. We have described the
architecture design decisions in detail,
enabling other WSC operators to reproduce
our system.
 Second, this article is the first work that
shows detailed characterization studies of
the processor front-end at the scale of a WSC
describing previously unreleased perfor-
mance characteristics of WSC workloads.
 Third, we have proposed and evaluated a
novel software-based code prefetch strategy
to automatically and effectively reduce i-
cache misses across large WSC workloads.
This work provides a powerful methodology to
perform further at-scale research to obtain a
detailed understanding of the microarchitectural
characteristics and the interplay between current
software and hardware. In addition, its reproduc-
ibility enables other WSC companies to perform
similar research. Overall, such research would
61
 Top Picks

enable hardware vendors to work closely with designing domain-specific accelerators becomes
software developers to better design future feasible and cost-efficient. However, while this
processors. approach has proven successful for domains
Our front-end characterization studies benefit such as deep learning, most of the fleet cycles are
the compiler and architecture communities both still executed on general-purpose processors as
in academic and industrial set- many applications are too complex
tings. Our results on micro-optimi- We developed AsmDB, and rapidly changing to render
zations, fragmentation, and code- a database for custom-designed hardware feasi-
bloat can help in fine-tuning com- instruction and basic- ble. Nevertheless, as this article
piler passes, optimizing inlining block information showed, the performance charac-
strategies, and basic block lay- across thousands of teristics of WSC applications are
outs. Similarly, our studies pro- WSC production fundamentally different from tradi-
vide valuable information to binaries, to characterize tional applications such as the
i-cache miss-working SPEC benchmark suite. WSC pro-
architecture researchers exposing
sets and miss-causing cessors may differ with capabilities
existing software loop holes that
instructions. such as our proposed instruction
can be addressed with next-gener-
ation hardware designs. prefetching mechanism, which may
Our work on software code prefetching be of little use to SPEC applications, but which
proves as a strong case study for hardware ven- delivers significant performance gains for data-
dors to provide support for a software code pre- center applications.
fetch instruction and to implement such an In summary, the evidence is strong that this
instruction in the instruction set architecture article will promote the research and develop-
(ISA). With this, compiler writers and software ment of new compiler techniques, new proces-
developers can leverage code prefetching and sor designs, and new ways of collecting and
analyzing behaviors at the warehouse scale.
its resulting performance improvements in an
automatic and scalable way.
More broadly, this article provides two CONCLUSION
insights, which we believe will have a significant This work focused on understanding and
and long-lasting impact on future research in the improving i-cache behavior, which is a critical per-
performance optimization and computer architec- formance constraint for WSC applications.
ture domain. The first insight teaches the impor- We developed AsmDB, a database for instruction
tance of enabling fleet-wide performance and basic-block information across thousands of
optimizations, which we also refer to as the WSC production binaries, to characterize i-cache
Amdahl’s law of WSC performance. Traditionally, miss-working sets and miss-causing instructions.
performance optimizations have been focused on We used these insights to motivate fine-grain lay-
individual applications. In this approach, applica- out optimizations to split hot and cold codes and
tions are profiled to determine the most compute- better utilize limited i-cache capacity. We also pro-
intensive regions, resulting in the largest perfor- posed a new feedback-driven optimization that
mance gains when optimized. However, this inserts software instructions for code prefetching
approach no longer applies to WSCs as datacen- based on the control-flow information and miss
ters run thousands of different applications profiles in AsmDB. This prefetching optimization
simultaneously. As a result, compute-intensive can cover up to 96% of i-cache misses without sig-
application-specific kernels are no longer worth nificant changes to the processor and while requir-
optimizing. Instead, performance engineers need ing only very simple front-end fetch mechanisms.
to focus on code that is shared among many appli-
cations in the fleet, representing the largest aggre-
gated percentage of compute cycles. ACKNOWLEDGMENTS
The second insight teaches the importance of This work was supported by the NSF Award
designing domain-specific general-purpose pro- CCF-1823559. Nayana Prasad Nagendra and
cessors. WSCs have grown to a size at which Grant Ayers contributed equally to this work.

IEEE Micro
62
& REFERENCES
1. G. Ayers, J. H. Ahn, C. Kozyrakis, and
P. Ranganathan, “Memory hierarchy for web search,”
in Proc. IEEE Int. Symp. High Perform. Comput.
Archit., 2018, pp. 643–656.
2. S. Kanev et al., “Profiling a warehouse-scale
computer,” in Proc. Int. Symp. Comput. Archit., 2015,
pp. 158–169.
3. R. Kumar, B. Grot, and V. Nagarajan, “Blasting through
the frontend bottleneck with shotgun,” in Proc. Archit.
Support Program. Lang. Oper. Syst., 2018, pp. 30–42.
4. G. Ren, E. Tune, T. Moseley, Y. Shi, S. Rus, and
R. Hundt, “Google-wide profiling: A continuous
profiling infrastructure for data centers,” IEEE Micro,
vol. 30, no. 4, pp. 65–79, Jul./Aug. 2010.
5. D. Sanchez and C. Kozyrakis, “ZSim: Fast and accurate
microarchitectural simulation of thousand-core systems,”
in Proc. Int. Symp. Comput. Archit., 2013, pp. 475–486.
Nayana Prasad Nagendra is currently working
toward the Ph.D. degree with the Department
of Computer Science, Princeton University. Her
research interests include performance analysis and
microarchitectural design with a focus on data
centers. This work was done while she was an intern
at Google. She is a student member of IEEE and
ACM. Contact her at nagendra@cs.princeton.edu.
Grant Ayers is currently a Software Engineer at
Google. His research interests include computer
architecture, security, and accelerators. He joined
Google after receiving the Ph.D. degree in computer
science from Stanford University. This work was
done while he was an intern at Google. Contact him
at ayers@cs.stanford.edu.
David I. August is currently a Professor with the
Department of Computer Science, Princeton Univer-
sity, where he directs the Liberty Research Group. His
research interests include compilers and computer
architectures. August received the Ph.D. degree in
electrical and computer engineering from the Univer-
sity of Illinois at Urbana–Champaign. Contact him at
august@princeton.edu.
Hyoun Kyu Cho is currently a Software Engineer at
Google. His research interests include compiler optimi-
zation, parallel computing, and performance analysis.
Cho received the Ph.D. degree in computer science
and engineering from the University of Michigan at Ann
Arbor. Contact him at netforce@google.com
May/June 2020
Svilen Kanev is currently a Software Engineer at
Google, working on translating datacenter perfor-
mance analysis insights into performance and TCO
gains. He is broadly interested in anything that strad-
dles the hardware-software interface. Kanev received
the Ph.D. degree in computer science from Harvard
University. Contact him at skanev@google.com
Christos Kozyrakis is currently a Professor of elec-
trical engineering and computer science with Stanford
University. His research interests include hardware
architectures and system software for cloud computing
and emerging workloads. Kozyrakis received the Ph.D.
degree in computer science from the University of
California Berkeley. He is a Fellow of IEEE and ACM.
Contact him at christos@cs.stanford.edu.
Trivikram Krishnamurthy is currently a Senior Engi-
neering Manager at Nvidia. Before joining Nvidia, he was
a Software Engineer at Google. Krishnamurthy received
the M.S. degree in electrical and computer engineering
from the University of California Santa Barbara. Contact
him at trivikram.krishnamurthy@gmail.com.
Heiner Litz is currently an Assistant Professor in the
Computer Science and Engineering Department, Uni-
versity of California, Santa Cruz (UCSC) and the Asso-
ciate Director of the Center for Research in Storage
Systems. His main research interests include com-
puter architecture, operating systems, and storage
with a focus on data centers. Before joining UCSC, he
was a Researcher at Google. Litz received the Ph.D.
degree from Mannheim University. He is a member of
IEEE and ACM. Contact him at hlitz@ucsc.edu.
Tipp Moseley is currently a Principal Software Engi-
neer at Google, where he works on datacenter-scale
performance analysis. His research interests include
compilers, operating systems, performance analysis,
runtime systems, fault tolerance, and optimized lock-
free data structures. Moseley received the Ph.D.
degree in computer science from the University of
Colorado at Boulder. Contact him at tipp@google.com.
Parthasarathy Ranganathan is currently a
Distinguished Engineer at Google, where he is design-
ing their next-generation systems. His research inter-
ests include systems architecture and management,
power management, and energy efficiency for servers
and datacenters. Ranganathan received the Ph.D.
degree in computer engineering from Rice University.
He is a Fellow of IEEE and ACM. Contact him at
partha.ranganathan@google.com.
63
 Theme Article: Top Picks

Extending the Frontier

of Quantum Computers
With Qutrits
Pranav Gokhale, Jonathan M. Baker, Kenneth R. Brown
Casey Duckering, and Frederic T. Chong Duke University
University of Chicago
Natalie C. Brown
Georgia Institute of Technology

Abstract—We advocate for a fundamentally different way to perform quantum computation

by using three-level qutrits instead of qubits. In particular, we substantially reduce the
resource requirements of quantum computations by exploiting a third state for temporary
variables (ancilla) in quantum circuits. Past work with qutrits has demonstrated only
constant factor improvements, owing to the log2(3) binary-to-ternary compression factor.
We present a novel technique using qutrits to achieve a logarithmic runtime decomposition
of the Generalized Toffoli gate using no ancilla—an exponential improvement over the best
qubit-only equivalent. Our approach features a 70x improvement in total two-qudit gate
count over the qubit-only decomposition. This results in improvements for important
algorithms for arithmetic and QRAM. Simulation results under realistic noise models
indicate over 90% mean reliability (fidelity) for our circuit, versus under 30% for the qubit-
only baseline. These results suggest that qutrits offer a promising path toward extending
the frontier of quantum computers.

& RECENT ADVANCES both hardware and

IN
software for quantum computation have demon-
strated significant progress toward practical
Digital Object Identifier 10.1109/MM.2020.2985976
Date of publication 16 April 2020; date of current version 22
May 2020.
outcomes. In the coming years, we expect quan-
tum computing will have important applications
in fields ranging from machine learning and
optimization to drug discovery. While early
research efforts focused on longer term systems
employing full error correction to execute large
instances of algorithms like Shor factoring and
Grover search, recent work has focused on noisy

0272-1732 ß 2020 IEEE Published by the IEEE Computer Society IEEE Micro
64
intermediate scale quantum (NISQ) computa- The net result of our work is to extend the fron-
tion. The NISQ regime considers near-term tier of what quantum computers can compute. In
machines with just tens to hundreds of quantum particular, the frontier is defined by the zone in
bits (qubits) and moderate errors. which every machine qubit is a data qubit, for
Given the severe constraints on example, a 100-qubit algorithm
quantum resources, it is critical to Given the severe running on a 100-qubit machine.
fully optimize the compilation of a constraints on quantum This is indicated by the yellow
quantum algorithm in order to have resources, it is critical region in Figure 1. In this frontier
successful computation. Prior archi- to fully optimize the zone, we do not have room for
tectural research has explored tech- compilation of a nondata workspace qubits known
niques such as mapping, scheduling, quantum algorithm as ancilla. The lack of ancilla in
and parallelism to extend the amount in order to have the frontier zone is a costly con-
of useful computation possible. In successful straint that generally leads to inef-
this article, we consider another computation. ficient circuits. For this reason,
technique: quantum trits (qutrits). typical circuits instead operate
While quantum computation is typically below the frontier zone, with many machine
expressed as a two-level binary abstraction of qubits used as ancilla. This article demonstrates
qubits, the underlying physics of quantum that ancilla can be substituted with qutrits,
systems are not intrinsically binary. Whereas enabling us to extend the ancilla-free frontier zone
classical computers operate in binary states of quantum computation.
at the physical level (e.g., clipping above
and below a threshold voltage), quantum com-
puters have natural access to an infinite BACKGROUND
spectrum of discrete energy levels. In fact, A qubit is the fundamental unit of quantum
hardware must actively suppress higher level computation. Compared to their classical coun-
states in order to achieve the two-level qubit terparts which take values of either 0 and 1,
approximation. Hence, using three-level qut- qubits may exist in a superposition of the two
rits is simply a choice of including an addi- states. We designate these two basis states as j0i
tional discrete energy level, albeit at the cost and j1i and can represent any qubit as
of more opportunities for error. jci ¼ a j0i þ b j1i with kak2 þ kbk2 ¼ 1. kak2 and
Prior work on qutrits (or more generally, kbk2 correspond to the probabilities of measur-
d-level qudits) identified only constant factor ing j0i and j1i, respectively.
gains from extending beyond qubits. In Quantum states can be acted on by quan-
general, this prior work1 has emphasized the tum gates, which preserve valid probability
information compression advantages of qut- distributions that sum to 1 and guarantee
rits. For example, N qubits can be expressed reversibility. For example, the X gate trans-
as N=log2 ð3Þ qutrits, which leads to forms a state jci ¼ a j0i þ b j1i to X jci ¼
log2 ð3Þ  1:6 constant factor improvements in b j0i þ a j1i . The X gate is also an example of a
runtimes. classical reversible operation, equivalent to the
Our approach utilizes qutrits in a novel fash- NOT operation. In quantum computation, we have
ion, essentially using the third state as tempo- a single irreversible operation called measurement
rary storage, but at the cost of higher per- that transforms a quantum state into one of the
operation error rates. Under this treatment, the two basis states with a given probability based on
runtime (i.e., circuit depth or critical path) is a and b.
asymptotically faster, and the reliability of com- In order to interact different qubits, two-qubit
putations is also improved. Moreover, our operations are used. The CNOT gate appears both
approach only applies qutrit operations in an in classical reversible computation and in quan-
intermediary stage: The input and output are tum computation. It has a control qubit and a tar-
still qubits, which is important for initialization get qubit. When the control qubit is in the j1i state,
and measurement on real devices.2; 3 the CNOT performs a NOT operation on the target.

May/June 2020
65
 Top Picks
Figure 1. Frontier of what quantum hardware can
execute is the yellow region adjacent to the 45 line.
In this region, each machine qubit is a data qubit.
Typical circuits rely on nondata ancilla qubits for
workspace and therefore operate below the frontier.
The cnot gate serves a special role in quantum
computation, allowing quantum states to become
entangled so that a pair of qubits cannot be
described as two individual qubit states. Any oper-
ation may be conditioned on one or more controls.
Many classical operations, such as AND and or
gates, are irreversible and therefore cannot
directly be executed as quantum gates. For
example, consider the output of 1 from an OR
gate with two inputs. With only this information
about the output, the value of the inputs cannot
be uniquely determined. These operations can
be made reversible by the addition of extra, tem-
porary ancilla bits initialized to j0i.
Physical systems in classical hardware are typi-
cally binary. However, in common quantum hard-
ware, such as in superconducting and trapped ion
computers, there is an infinite spectrum of discrete
energy levels. The qubit abstraction is an artificial
approximation achieved by suppressing all but the
lowest two energy levels. Instead, the hardware
may be configured to manipulate the lowest three
energy levels by operating on qutrits. In general,
such a computer could be configured to operate
on any number of d levels. As d increases, the num-
ber of opportunities for error—termed error chan-
nels—increases. Here, we focus on d ¼ 3 which is
sufficient to achieve desired improvements to the
Generalized Toffoli gate.
66
In a three-level system, we consider the
computational basis states j0i, j1i, and j2i for
qutrits. A qutrit state jci may be represented
analogously to a qubit as jci ¼ a j0i þ b j1i þ
g j2i , where a2 þ b2 þ g 2 ¼ 1. Qutrits are manipu-
lated in a similar manner to qubits; however,
there are additional gates which may be per-
formed on qutrits.
For instance, in quantum binary logic, there is
only a single X gate. In ternary, there are three X
gates denoted X01 , X02 , and X12 . Each of these Xij
can be viewed as swapping jii with jji and leaving
the third basis element unchanged. For example,
for a qutrit jci ¼ a j0i þ b j1i þ g j2i, applying X02
produces X02 jci ¼ g j0i þ b j1i þ a j2i . There are
two additional nontrivial operations on a single
trit. They are Xþ1 and X1 operations, which per-
form the addition/subtraction modulo 3.
Just as single qubit gates have qutrit analogs,
the same holds for two qutrit gates. For example,
consider the CNOT operation, where an X gate is
performed conditioned on the control being in the
j1i state. For qutrits, any of the X gates presented
above may be performed, conditioned on the con-
trol being in any of the three possible basis states.
In order to evaluate a decomposition of a
quantum circuit, we consider quantum circuit
costs. The space cost of a circuit, i.e., the num-
ber of qubits (or qutrits), is referred to as circuit
width. Requiring ancilla increases the circuit
width and, therefore, the space cost of a circuit.
The time cost for a circuit is the depth of a cir-
cuit. The depth is given as the length of the criti-
cal path from input to output.
PRIOR WORK
Qudits
Qutrits, and more generally qudits, have been
studied in past work both experimentally and
theoretically. However, in the past work, qudits
have conferred only an information compression
advantage. For example, N qubits can be com-
pressed to N=log2 ðdÞ qudits, giving only a con-
stant-factor advantage1 at the cost of greater
errors from operating qudits instead of qubits.
Ultimately, the tradeoff between information
compression and higher per-qudit errors has not
been favorable in the past work. As such, the
IEEE Micro
Table 1. Asymptotic comparison of N-controlled gate decompositions. The total gate count for all circuits scales
linearly (except for Barenco et al.,6 which scales quadratically). Our construction uses qutrits to achieve logarithmic
depth without ancilla. We benchmark our circuit construction against Gidney,4 which is the asymptotically best
ancilla-free qubit circuit.
He
This Work Gidney4
et al.5
Depth log N N log N
Ancilla 0 0 N 0
Qudit Controls are
Qubits Qubits
Types qutrits
Constants Small Large Small
past research toward building practical quantum
computers has focused on qubits.
This article introduces qutrit-based circuits,
which are asymptotically better than equivalent
qubit-only circuits. Unlike prior work, we dem-
onstrate a compelling advantage in both run-
time and reliability, thus justifying the use of
qutrits.
Generalized Toffoli Gate
The Toffoli gate itself is a simple extension of
the CNOT gate, but has two controls instead of
one control. In a Toffoli gate, the NOT is applied if
and only if both controls are j1i. Similarly, a Gener-
alized Toffoli gate has N controls and flips the tar-
get qubit if and only if all N control qubits are j1i.
The Generalized Toffoli gate is an important primi-
tive used across a wide range of quantum algo-
rithms, and it has been the focus of extensive past
optimization work. Table 1 compares past circuit
constructions for the Generalized Toffoli gate to
our construction, which is presented in full in
“Generalized Toffoli Gate” section.
Among prior work, Gidney,4 He et al.,5 and
Barenco et al.6 designs are all qubit-only. The
three circuits have varying tradeoffs. While Gid-
ney and Barenco operate at the ancilla-free fron-
tier, they have large circuit depths: Linear with a
large constant for Gidney and quadratic for Bare-
nco. While the He circuit achieves logarithmic
depth, it requires an ancilla for each data qubit,
effectively halving the effective potential of any
given quantum hardware and operating far
below the frontier. Nonetheless, in practice,
most circuit implementations use these linear-
ancilla constructions due to their small depths
and gate counts.
May/June 2020
Barenco Wang and
Lanyon et al.8
et al.6 Perkowski7
N2 N N
0 0
Controls are Target is d ¼ N-level
Qubits
qutrits qudit
Small Small Small
As in our approach, circuit constructions
from Wang and Perkowski,7 and Lanyon et al.8
have attempted to improve the ancilla-free Gen-
eralized Toffoli gate by using qudits. Wang and
Perkowski7 achieves a linear circuit depth but
by operating each control as a qutrit. The
Lanyon et al.8 construction, which has been
demonstrated experimentally, achieves linear
circuit depths by operating the target as a
d ¼ N-level qudit.
Our circuit construction, presented in the
“Generalized Toffoli Gate” section, has similar
structure to the He design, which can be rep-
resented as a binary tree of gates. However,
instead of storing temporary results with a lin-
ear number of ancilla qubits, our circuit tem-
porarily stores information directly in the
qutrit j2i state of the controls. Thus, no
ancilla are needed.
In our simulations, we benchmark our circuit
construction against the Gidney construction4
because it is the asymptotically best qubit cir-
cuit in the ancilla-free frontier zone. We label
these two benchmarks as QUTRIT and QUBIT.
CIRCUIT CONSTRUCTION
In order for quantum circuits to be executable
on hardware, they are typically decomposed into
single- and two- qudit gates. Performing efficient
low depth and low gate count decompositions is
important in both the NISQ regime and beyond.
Key Intuition
We develop the intuition for how qutrits can
be useful by considering the example of construct-
ing an AND gate. In the framework of quantum
67
 Top Picks

computing, which requires reversibility, AND is

not permitted directly. For example, consider a
two-input AND gate that outputs a 0. Given this
output value, the inputs cannot be uniquely deter-
mined since 00, 01, and 10 all yield an AND output
of 0. However, these operations can be made
reversible by the addition of an extra temporary
workspace bit initialized to 0. Using a single addi-
tional such ancilla, the AND operation can be com- Figure 2. Toffoli AND via qubits (top) versus qutrits
puted reversibly via the well-known Toffoli gate (a (bottom).
double-controlled NOT) in reversible computa-
Generalized Toffoli Gate
tion. While this approach works, it is expensive—
We now present our circuit decomposition
its decomposition into hardware-implementable
for the generalized Toffoli gate in Figure 3. The
one- and two-input gates requires at least six con-
decomposition is expressed in terms of three-
trolled-NOT gates and several single qubit gates,
qutrit gates (two controls, one target) instead of
as depicted in the top circuit of Figure 2.
single- and two- qutrit gates, because the circuit
However, if we break the qubit abstraction
can be understood purely classically at this gran-
and allow occupation of a higher qutrit energy
ularity. In actual implementation and in our sim-
level, the cost of the Toffoli AND operation is
ulation, we used a decomposition9 that requires
greatly diminished. The bottom circuit in Figure 2
six two-qutrit and seven single-qutrit physically
displays this decomposition using qutrits. The
implementable quantum gates.
goal is to elevate the jq2 ¼ 0i ancilla qubit to j1i
Our circuit decomposition is most intuitively
if and only if the top two control qubits are
understood by treating the left half of the circuit
both j1i. This is effectively an AND gate. First a
as a tree. The desired property is that the root of
j1i-controlled ðþ1 mod 3Þ is performed on q0 and
the tree q7 is j2i if and only if each of the 15 con-
q1 . This elevates q1 to j2i iff q0 and q1 were both
trols was originally in the j1i state. To verify this
j1i. Then, a j2i-controlled ðþ1 mod 2Þ gate is
property, we observe that the root q7 can only
applied to q2 . Therefore, q2 is elevated to j1i only
become j2i iff q7 was originally j1i and q3 and q11
when both q0 and q1 were j1i, as desired. The
were both previously j2i. At the next level of the
controls are restored to their original states by a
tree, we see q3 could have only been j2i if q3 was
j1i-controlled ð1 mod 3Þ gate, which resets q1 .
originally j1i and both q1 and q5 were previously
The key intuition is that temporary information
j2i, and similarly for the other triplets. At the
can be stored in the qutrit j2i state, rather
bottom level of the tree, the triplets are con-
than requiring an external ancilla. This allows
trolled on the j1i state, which are only activated
us to maximize the problem sizes that current-
when the even-index controls are all j1i. Thus, if
generation hardware can address.
any of the controls were not j1i, the j2i states
Now, notice that this AND decomposition is
would fail to propagate to the root of the tree.
actually also a Toffoli gate decomposition. In par-
The right half of the circuit performs uncomputa-
ticular, suppose that the bottom qubit is now arbi-
tion to restore the controls to their original state.
trary, rather than initialized to j0i. Then, the net
After each subsequent level of the tree struc-
effect of this circuit is to preserve the controls,
ture, the number of qubits under consideration
while flipping the bottom (target) qubit, if and
is reduced by a factor of  2. Thus, the circuit
only if q0 and q1 were both j1i. For this decomposi-
depth is logarithmic in N. Moreover, each qutrit
tion, since all inputs and outputs are binary qubits
is operated on by a constant number of gates, so
(we only occupy the j2i state temporarily during
the total number of gates is linear in N.
computation), we can insert this circuit construc-
tions into any preexisting qubit-only circuits.
Again, the key intuition in this decomposition is APPLICATION TO ALGORITHMS
that the qutrit j2i state can be used instead of The Generalized Toffoli gate is an important
ancilla to store temporary information. primitive in a broad range of quantum

IEEE Micro
68
algorithms. Here, we note two important applica-
tions of our circuit decomposition.

Arithmetic Circuits
The Generalized Toffoli is a key subcircuit
in many arithmetic circuits such as constant
addition, modular multiplication, and modular
exponentiation. The circuit for computing a
square root is also improved by a more effi-
cient Generalized Toffoli gate. As shown by
Gokhale,10 the circuit for the initial approxi-
pffiffiffi
mation to 1= x involves a sequence of stan-
dard Toffoli gates terminated by a large
OðnÞ-width Generalized Toffoli gate. Our cir-
cuit construction is directly applicable to this
terminal gate.

Quantum Machine Learning

A fundamental component of most algo-
rithms for quantum machine learning is a
quantum random access memory (QRAM).
QRAM has the classically familiar property of
mapping input index bits to output data bits.
However, unlike classical RAM, QRAM also
acts over superpositions of qubits. The initiali-
zation of QRAM is often the bottleneck for
quantum machine learning algorithms—an
expensive procedure for storing training data
into QRAM can negate any potential quantum
advantage.
However, the QRAM circuit is yet another
application that can be improved with the
qutrit-assisted Generalized Toffoli gate. In partic-
ular, the flip-flop QRAM is bottlenecked by the
application of a wide Generalized Toffoli gate for
each classical bitstring stored to the QRAM.11
Thus, an efficient Generalized Toffoli gate
reduces the cost of QRAM, relative to nonqutrit
procedures.
SIMULATOR
To simulate our circuit constructions, we
developed a qudit simulation library, built on
Google’s Cirq Python library. Cirq is a qubit-
based quantum circuit library and includes a
number of useful abstractions for quantum
states, gates, circuits, and scheduling. Our soft-
ware performs noise simulation, described in
the following.
Figure 3. Our circuit decomposition for the Generalized Toffoli
gate is shown for 15 controls and 1 target. The inputs and outputs
are both qubits, but we allow occupation of the j2i qutrit state in
between. The circuit has a tree structure and maintains the
property that the root of each subtree can only be elevated to j2i if
all of its control leaves were j1i. Thus, the U gate is only executed
if all controls are j1i. The right half of the circuit performs
uncomputation to restore the controls to their original state. This
construction applies more generally to any multiply controlled U
gate. Note that the three-input gates are decomposed into six
two-input and seven single-input gates in our actual simulation, as
based on the decomposition by Di and Wei.9
Noise Simulation
Our noise simulation procedure accounts for
both gate errors and idle errors, described
below. To determine when to apply each gate
and idle error, we use Cirq’s scheduler which
schedules each gate as early as possible, creat-
ing a sequence of Moment’s of simultaneous
gates. During each Moment, our noise simulator
applies a gate error to every qudit acted on.
Finally, the simulator applies an idle error to
every qudit. This noise simulation methodology
is consistent with previous simulation techni-
ques, which have accounted for either gate
errors or idle errors.

May/June 2020
69
 Top Picks
Figure 4. Exact circuit depths for all three benchmarked circuit
constructions for the N-controlled Generalized Toffoli up to
N ¼ 200. QUBIT scales linearly in depth and is bested by
QUTRIT’s logarithmic depth.
Gate errors arise from the imperfect applica-
tion of quantum gates. Two-qudit gates are nois-
ier than single-qudit gates, so we apply different
noise channels for the two. Idle errors arise from
the continuous decoherence of a quantum sys-
tem due to energy relaxation and interaction
with the environment.
Gate errors are reduced by performing fewer
total gates, and idle errors are reduced by
decreasing the circuit depth. Since our circuit
constructions asymptotically decrease the
depth, this means our circuit constructions
scale favorably in terms of asymptotically fewer
idle errors.
The ultimate metric of interest is the mean
fidelity, which captures the probability of overall
successful execution. We do not consider state
preparation and measurement (SPAM) errors,
Figure 5. Exact two-qudit gate counts for the two benchmarked
circuit constructions for the N-controlled Generalized Toffoli. Both
plots scale linearly; however, the QUTRIT construction has a
substantially lower linearity constant.
70
because our circuit constructions maintain
binary input and output, only occupying the
qutrit j2i states during intermediate computa-
tion. Therefore, the SPAM errors for our circuits
are identical to those for conventional qubit
circuits.
We chose noise models which represent real-
istic near-term machines. Our models encom-
passed both superconducting and trapped ion
platforms. For superconducting technology, we
simulated against four noise models: SC, SC+T1,
SC+GATES, and SC+T1+GATES. For trapped
ion technology, we simulated against three
benchmarks: TI_QUBIT, BARE_QUTRIT, and
DRESSED_QUTRIT.
RESULTS
Figure 4 plots the exact circuit depths for
the QUTRIT versus QUBIT circuit construction.
Note that the QUBIT construction is linear in
depth, with a high linearity constant. This is sig-
nificantly improved by our QUTRIT construc-
tion, which scales logarithmically in N and has a
relatively small leading coefficient.
Figure 5 plots the total number of two-qudit
gates for both circuit constructions. Our circuit
construction is not asymptotically better in total
gate count—both plots have linear scaling. How-
ever, as emphasized by the logarithmic vertical
axis, the linearity constant for our qutrit circuit
is 70 smaller than for the equivalent ancilla-
free qubit circuit.
We simulated these circuits under realistic
noise models in parallel on over 100 n1-stan-
dard-4 Google Cloud instances. These simula-
tions represent over 20 000 CPU hours, which
was sufficient to estimate mean fidelity to
an error of 2s < 0:1% for each circuit-noise
model pair.
The full results of our circuit simulations are
shown in Figure 6. All simulations are for the 14-
input (13 controls, 1 target) Generalized Toffoli
gate. We simulated both circuit constructions
against each of our noise models (when applica-
ble), yielding the 11 bars in the figure.
DISCUSSION AND FUTURE WORK
Figure 6 demonstrates that our QUTRIT
construction (orange bars) can significantly
IEEE Micro
Figure 6. Circuit simulation results for all possible pairs of circuit constructions and noise models. Each bar
represents 1000+ trials, so the error bars are all 2s < 0:1%. Our QUTRIT construction significantly
outperforms the QUBIT construction.

outperform the ancilla-free QUBIT benchmark envision other advantages to higher radix quan-
(blue bars) in fidelity (success probability) by tum computing. For example, the information-
more than 10 000. compression advantage of qudits may be particu-
For the SC, SC+T1, and SC+GATES noise mod- larly well suited to the NISQ hardware, where
els, our qutrit constructions achieve between device connectivity—and therefore diameter—is
57–83% mean fidelity, whereas the ancilla-free a bottleneck. Compressing a qubit computation
qubit constructions all have almost 0% fidelity. via qudits would allow us to reduce the graph
Only the lowest error model, SC+T1+GATES diameter.
achieves modest fidelity of 26% for the QUBIT The results presented in this article are appli-
circuit, but in this regime, the qutrit circuit is cable to quantum computing in the near term on
close to 100% fidelity. machines that are expected within the next five
The trapped ion noise models achieve similar years. The net result of this article is to extend the
results—the DRESSED_QUTRIT and frontier of what is computable by
the BARE_QUTRIT achieve approxi- quantum hardware, and hence to
mately 95% fidelity via the QUTRIT cir- Clever use of qutrits accelerate the timeline for practi-
cuit, whereas the TI_QUBIT noise offers a path to more cal quantum computing. Emphat-
model has only 45% fidelity. Between sophisticated quantum ically, our results are driven by
computation today,
the dressed and bare qutrits, the the use of qutrits for asymptoti-
without needing to wait
dressed qutrit exhibits higher fidelity cally faster ancilla-free circuits.
for better hardware. We
than the bare qutrit, as expected. Moreover, we also improve lin-
are optimistic that
Moreover, the dressed qutrit is resil- continued hardware– earity constants by two orders of
ient to leakage errors, so the simula- software codesign may magnitudes. Finally, as verified
tion results should be viewed as a further extend the frontier by our circuit simulator coupled
lower bound on its advantage over the of quantum computers. with realistic noise models, our
qubit and bare qutrit. circuits are more reliable than
Our qutrit-assisted Generalized qubit-only equivalents. In sum,
Toffoli gate has already attracted interest from clever use of qutrits offers a path to more sophisti-
both device physics and algorithms communities. cated quantum computation today, without need-
To this end, major quantum software packages ing to wait for better hardware. We are optimistic
like Cirq are now compatible with qutrit (and that continued hardware–software codesign may
qudit) simulations. We have also been working further extend the frontier of quantum computers.
with hardware groups to experimentally imple-
ment the ideas presented here. One promising
direction is to use OpenPulse, an open standard ACKNOWLEDGMENTS
for pulse-level quantum control, to experimentally We would like to thank Michel Devoret and
demonstrate a generalized Toffoli gate. We also Steven Girvin for suggesting to investigate

May/June 2020
71
 Top Picks
qutrits. We also acknowledge David Schuster for
helpful discussion on superconducting qutrits.
This work was supported in part by EPiQC, an
NSF Expedition in Computing, under Grant CCF-
1730449/1832377; in part by STAQ under Grant
NSF Phy-1818914; and in part by DOE Grants DE-
SC0020289 and DE-SC0020331. The work of Pra-
nav Gokhale was supported by the Department
of Defense through the National Defense Science
and Engineering Graduate Fellowship Program.
& REFERENCES
1. A. Pavlidis and E. Floratos, “Arithmetic circuits for
multilevel qudits based on quantum Fourier
transform,” 2017, arXiv:1707.08834.
2. J. Randall et al., “Efficient preparation and detection of
microwave dressed-state qubits and qutrits with
trapped ions,” Phys. Rev. A, vol. 91, 2015,
Art. no. 012322.
3. J. Randall, A. M. Lawrence, S. C. Webster, S. Weidt,
N. V. Vitanov, and W. K. Hensinger, “Generation of
high-fidelity quantum control methods for multilevel
systems,” Phys. Rev. A, vol. 98, 10 2018,
Art. no. 043414.
4. C. Gidney, “Constructing large controlled nots,” 2015.
5. Y. He, M.-X. Luo, E. Zhang, H.-K. Wang, and
X.-F. Wang, “Decompositions of n-qubit toffoli gates
with linear circuit complexity,” Int. J. Theor. Phys.,
vol. 56, pp. 2350–2361, Jul. 2017.
6. A. Barenco et al., “Elementary gates for quantum
computation,” Phys. Rev. A, vol. 52, pp. 3457–3467,
Nov. 1995.
7. Y. Wang and M. Perkowski, “Improved complexity of
quantum oracles for ternary grover algorithm for graph
coloring,” in Proc. 41st IEEE Int. Symp. Multiple-Valued
Logic, May 2011, pp. 294–301.
8. B. P. Lanyon et al., “Simplifying quantum logic using
higher-dimensional Hilbert spaces,” Nature Phys.,
vol. 5, pp. 134–140,, 2009.
9. Y.-M. Di and H.-R. Wei, “Elementary gates for ternary
quantum logic circuit,” 2011, arXiv:1105.5485.
72
10. P. Gokhale, “Implementation of square root function
using quantum circuits,” Undergraduate Awards,
2014.
11. D. K. Park, F. Petruccione, and J.-K. K. Rhee, “Circuit-
based quantum random access memory for classical
data,” Sci. Rep., vol. 9, no. 1, 2019, Art. no. 3949.
Pranav Gokhale is currently working toward the
Ph.D. degree with the University of Chicago. His
research focuses on breaking the abstraction
barrier between quantum hardware and software.
He is the founder of Super.tech. Contact him at
pranavgokhale@uchicago.edu.
Jonathan M. Baker is currently working toward
the Ph.D. degree with the University of Chicago. His
research is primarily focused on vertical integration
of the quantum computing hardware–software stack.
Contact him at jmbaker@uchicago.edu.
Casey Duckering is currently working toward the
Ph.D. degree with the University of Chicago, aiming
to efficiently bring together quantum algorithms
with their physical implementation on quantum com-
puters. Contact him at cduck@uchicago.edu.
Frederic T. Chong is the Seymour Goodman Pro-
fessor with the Department of Computer Science, Uni-
versity of Chicago. He is also Lead Principal
Investigator for the EPiQC Project (Enabling Practi-
cal-scale Quantum Computing), an NSF Expedition in
Computing. Contact him at chong@cs.uchicago.edu.
Natalie C. Brown is currently working toward the
Ph.D. degree with Georgia Institute of Technology.
Her research focuses on leakage error correction
and mitigation in topological surface codes. Contact
her at natalie.c.brown@duke.edu.
Kenneth R. Brown is an Associate Professor of
electrical and computer engineering with Duke Uni-
versity and the Director of the NSF Software Enabled
Architectures for Quantum co-design (STAQ) project
developing applications, software, and hardware
for ion trap quantum computers. Contact him at
kenneth.r.brown@duke.edu.
IEEE Micro
 Theme Article: Top Picks

Architecting Noisy
Intermediate-Scale
Quantum Computers:
A Real-System Study
Prakash Murali Ali Javadi Abhari
Princeton University IBM T. J. Watson Research Center
Norbert M. Linke Nhung Hong Nguyen
Joint Quantum Institute, University of Maryland University of Maryland
Margaret Martonosi Cinthia Huerta Alderete
Princeton University University of Maryland and Instituto Nacional de


Astrof
ısica, Optica
nica
y Electro

Abstract—Current quantum computers have very different qubit implementations,

instruction sets, qubit connectivity, and noise characteristics. Using real-system
evaluations on seven quantum systems from three leading vendors, our work explores
fundamental design questions concerning hardware choices, architecture, and compilation.

& QUANTUM COMPUTING (QC) is a fundamentally
new model of computation, which exploits
quantum mechanical phenomena to perform
computation. QC systems use qubits (quantum
bits) to represent information and gates
(quantum instructions) to manipulate quantum
Digital Object Identifier 10.1109/MM.2020.2985683
Date of publication 6 April 2020; date of current version 22
May 2020.
information. While the basic principles of QC have
been known since the 1980s, recent hardware
progress has ushered in the era of noisy intermedi-
ate-scale quantum (NISQ) devices. These systems
represent an important milestone toward large
scale QC, and are expected to scale to 500–1000
qubits in coming years. In spite of being too error-
prone and resource-constrained for well-known
applications like Shor’s factoring, NISQ systems
are capable of very powerful computations. Nota-
bly, Google recently demonstrated a classically

May/June 2020 Published by the IEEE Computer Society 0272-1732 ß 2020 IEEE
73
 Top Picks

intractable computation on an NISQ system with applications on seven systems from three lead-
54 qubits.1 ing vendors—IBM, Rigetti, and University of
Being early-stage, NISQ devices are highly Maryland. The systems studied represent differ-
diverse in terms of hardware and ent points in the design space,
architecture. Leading QC vendors with two leading qubit tech-
While the basic
including IBM, Rigetti, Google, principles of QC have nologies (superconducting and
IonQ, and others have adopted been known since the trapped ion qubits), different
very different approaches for build- 1980s, recent hardware connectivity topologies, pro-
ing hardware qubits. To support progress has ushered gramming interfaces, and noise
their qubit choices, vendors have in the era of noisy behavior. The diversity of sys-
also chosen different instruction intermediate-scale tems studied is important for
sets and hardware communication quantum (NISQ) understanding which aspects of
topologies. Further, QC systems devices. These systems QC design hold across different
also have variance in hardware represent an important design choices and which are
milestone toward large
noise, owing to fundamental more implementation specific.
scale QC, and are
challenges in qubit control and Our work represents the most
expected to scale to
manufacturing. While this diversity comprehensive cross-platform,
500–1000 qubits in
itself poses a challenge for efficient coming years. real-system measurements of QC
and portable application execu- prototypes ever performed.
tion, there is also a huge gap On the other hand, this design
between the QC hardware that is buildable now, space diversity also poses serious challenges for
and the resource requirements of compelling accurate comparative studies. In particular, our
real-world applications. Many interesting comparisons hinge on developing a toolflow and
applications demand large systems with several evaluation approach common to all platforms,
thousand quantum bits and high-precision oper- and yet not penalizing any particular platform
ations, but current hardware has less than while pursuing toolflow generality. Our toolflow,
100 qubits and error-prone operations. To fully TriQ, is the first top-to-bottom multivendor QC
attain practical and powerful QC, computer compiler toolflow. TriQ optimizes high-level lan-
architecture techniques and software toolchains guage programs for QC hardware by leveraging
must be employed to narrow the algorithm-to- deep but parameterized knowledge of the target
devices resource gap across a wide range of device characteristics, including the gate set,
algorithms and devices. connectivity, and noise profile. Importantly, TriQ
To this end, our article2 offers one of the avoids inefficiencies in vendor toolflows, offering
deepest explorations of cross-platform charac- up to two orders of magnitude higher reliability
teristics in QC systems, presenting a full-stack, compared to IBM’s Qiskit3 and Rigetti’s Quil4
benchmark-driven, hardware–software analysis. compiler which are the default toolchains for the
Viewing QC through the lens of computer archi- respective hardware. TriQ, therefore, allows us to
tecture, we evaluate important hardware design perform architectural analysis across diverse QC
decisions (qubit types, system size, connectivity, systems using high-level application performance
noise), the hardware–software interface (gate set measurements and is also a common compiler
choices), and software optimizations to tackle toolflow.
fundamental design questions: What instructions Our experiments with TriQ reveal several
should QC systems expose to software? Should architectural insights for QC systems. We quan-
instructions be unified in a device-independent tify the importance of gate set, ISA and connectiv-
ISA across different qubit types? How do hard- ity choices and offers design recommendations.
ware connectivity and noise characteristics We also evaluate the effects of hardware noise
impact benchmark performance? Can hardware on applications and the importance of software
limitations be overcome with a compiler? optimizations to mitigate such noise. Our results
To answer these questions, we use real- have also attracted significant academic and
system measurements to evaluate a suite of QC industry attention with vendors including IBM

IEEE Micro
74
and Rigetti incorporat-
ing our optimizations in
their compiler toolflows.
In coming years, hard-
ware and architectural
insights from our study
are likely to influence
QC.
Figure 1. Hardware qubit technology, native gate set, and software-visible gate set in
the systems used in our study. Each qubit technology lends itself to a set of native gates.
BACKGROUND
ON QC For programming, vendors expose these gates in a software-visible interface or construct
A qubit is the funda- composite gates with multiple native gates.
mental building block
a QC system. Unlike a
classical bit which is restricted to be either in
the state 0 or 1 at any instant, a qubit can exist
in a superposition state where it is a probabilistic
combination of the two basis states. This prop-
erty allows an n-bit QC system to represent 2n
basis states simultaneously, unlike classical
registers which can be in exactly one of the 2n
values at any given time. To manipulate informa-
tion, QC gates are implemented to operate on
one or more qubits, using some physical interac-
tion such as a microwave or laser pulse. Similar
to universal gates in classical systems, QC com-
putations can be expressed using a small univer-
sal set of single (1Q) and two-qubit (2Q) gates.
In particular, 2Q gates create entanglement
which is a key property exploited by algorithms.
To obtain classical output from the system,
qubits are measured or readout, collapsing the
superposition state to either 0 or 1.
QC ARCHITECTURE CHOICES
AND TRADEOFFS
NISQ systems have very diverse hardware
and architecture. While classical metrics such as
performance (time) and area are important to
evaluate these options, a key figure of merit in
the current NISQ regime is the likelihood of cor-
rect execution of applications. Owing to the
noise, a single execution of an application may
be corrupted by noise. Hence, programs are
typically run multiple times and the success rate
is measured as the fraction of trials which yields
the correct answer. Toward understanding how
system design affects success rate and perfor-
mance, we briefly discuss the key design choices
May/June 2020
here and refer the reader to our original paper
for more details.2
Figure 1 shows the different hardware qubit
technologies used in IBM, Rigetti, and UMD sys-
tems. IBM and Rigetti use superconducting
qubits, while UMD uses trapped ion qubits. On
one hand, these choices are similar to how clas-
sical computers can be realized using vacuum
tubes, relay circuits or CMOS transistors. On the
other hand, qubit technologies are very different
and do not lend themselves to abstraction simi-
lar to the ON–OFF switch abstraction in classical
technologies. For example, on IBM’s supercon-
ducting qubits, the two-qubit interactions are
achieved using the cross-resonance effect,
where one qubit is driven at the resonant fre-
quency of another qubit using a coupled hard-
ware resonator. In contrast, in UMD’s trapped
ion qubits, two-qubit interactions are achieved
using collective motional modes of an ion chain,
mediated through laser pulses.
Owing to these fundamental differences, ven-
dors implement different native gates or microop-
erations that are feasible on their platform.
Figure 1 shows these native 1Q and 2Q gates.
Even among superconducting qubits, the native
interactions may be different. For example,
Rigetti uses the controlled Z operation as the fun-
damental 2Q operation instead of the cross-reso-
nance gate in IBM. Using these native gates,
vendors choose a software-visible programming
interface which includes either native gates
themselves or composite gates which use multi-
ple native gates. These choices for software-
visible gates also differ widely across vendors.
75
 Top Picks

Finally, qubit states are

extremely fragile and difficult
to control. On current sys-
tems, typical 2Q error rates
are 1% –10%. Gate error rates
also have large spatial and
temporal variations depend-
ing on the qubit technology.
Figure 3 shows these varia-
tions for an IBM system. If a
program is executed on a sub-
set of unreliable qubits, the
success rate is greatly dimin-
Figure 2. Characteristics of the devices used in our study. Each device has different ished. In addition, quantum
qubit and gate count (higher is better), coherence time (higher is better), error rates state “decoheres” or loses
(lower is better), and topology (dense connectivity is better). reliability exponentially with
time. That is, if a qubit is ini-
tialized, there is a short win-
Furthermore, QC devices have a qubit connec- dow of coherence time within which all gates in
tion topology which determines the amount of the program must be completed. On current
communication required to perform 2Q gates. As superconducting systems, coherence time is typ-
shown in Figure 2, device topology varies across ically less than 100 ms and varies across qubits.
systems, with sparse nearest-neighbor connec- However, 2Q gates are relatively fast, requiring
tivity in IBM and Rigetti, to full all-pairs connec- hundreds of nanoseconds. On trapped ion sys-
tivity in UMD. When full connectivity is not tems, coherence time is significantly longer
available, SWAP operations are used to enable 2Q (several seconds), but 2Q gates are slower,
gates between arbitrary pairs of qubits. These requiring several hundred microseconds.
SWAPs increase program duration and more Our work focuses on how these design trade-
importantly, worsen the success rate. The choice offs influence QC computer architecture and
of connectivity is not independent of the qubit software design. Toward this, we first develop
type. Trapped ion qubits naturally support a common compiler toolflow, TriQ, that maps
full connectivity, at least at small scales, while programs onto diverse QC systems. Enabled by
superconducting qubits typically use sparse con- TriQ, we perform real-system experiments to
nectivity because of difficulties in implementing explore the architectural design space.
dense physical interconnections.

TRIQ: FULL-STACK MULTIVENDOR

Figure 3. Daily variation of error rates of four
hardware supported two-qubit controlled NOT gates in
IBMQ14. The average error rate is approximately 8%,
but there is up to 9x variation across qubits and days.
QC TOOLFLOW
Figure 4 illustrates the overall structure of
TriQ. TriQ accepts Scaffold5 programs as input.
Scaffold is a C-like quantum language which has
been used to develop large QC applications.
Using ScaffCC, Scaffold’s front-end compiler,
TriQ generates an intermediate representation
(IR) of the program and uses it as the input
for the subsequent optimization passes. TriQ
also takes hardware and system-specific features
such as gate sets, connectivity, and noise informa-
tion (from daily calibration logs for the systems)
as configurable inputs. Hardware-dependent

IEEE Micro
76
optimization using these inputs
is a distinguishing feature of TriQ
that allows it to obtain high
success rates across platforms.
As output, TriQ generates opti-
mized code in the vendor-
specified assembly code.
To compile the IR, the first
step is to map program qubits
onto distinct hardware qubits.
For example, program qubits can
be assigned to hardware qubits Figure 4. Overview of the TriQ toolflow. Inputs are high-level Scaffold programs
according to the order they are and their inputs, as well as device-specific QC system properties. Output is
used in the program. This policy optimized code in one of three vendor-specific executable formats.
can result in high communication
overhead and poor success rate
when qubits participating in 2Q gates are not Second, TriQ schedules gates in the program
mapped close together. If program qubits are in a topologically-sorted order using the IR. This
mapped onto unreliable hardware qubits, it allows maximum operations to be executed in
can further worsen the success rate. Therefore, parallel, reducing the errors due to qubit deco-
TriQ uses a noise-adaptive mapping strategy herence. For devices which do not support full
which optimizes both communication and reli- connectivity, TriQ automatically inserts the nec-
ability. TriQ chooses a set of qubits that match essary communication operations to bring
well with the communication requirements of qubits into adjacent positions before executing
the application and simultaneously, it ensures 2Q gate. To improve success rates, TriQ incorpo-
that this set of qubits has low error rates for rates noise-awareness in this step by selecting
the instruction mix of the application. TriQ the lowest error rate paths for moving qubits,
implements this policy using a satisfiability rather than any shortest distance path.
modulo theory (SMT) optimization, solved Third, TriQ translates high-level IR gates into
using Microsoft’s Z3 SMT solver. device-specific IR. Using a set of legal code trans-
To flexibly target different devices, we formations that are provided as input, TriQ
designed the SMT optimization to work with an replaces IR gates with equivalent device-specific
abstract representation of the hardware. TriQ gates, e.g., OpenQASM code for IBM systems.
preprocesses the target device’s connectivity During this pass, TriQ also applies a 1Q gate
graph and gate error rate data and converts optimization where continuous sequences of 1Q
them to a reliability matrix representation. gates are compressed into shorter sequences.
For each pair of qubits, the matrix specifies the TriQ exploits knowledge of hardware error rates
reliability of the lowest error rate path for a 2Q in this step as well. On all three vendors, single
gate between the qubits. When two hardware qubit rotations gates along the Z-axis of the
qubits are far away in the communication topol- qubit have no error.6 While compressing gate
ogy, the reliability of the best path will be low. sequences, TriQ maximizes the use of these Z
It will also be low if all paths between the two rotations, further increasing success rates.
qubits have high error rate edges. Therefore,
using the matrix, TriQ can pick communication-
and reliability-optimized mappings. Since the REAL-SYSTEM ARCHITECTURAL
core functionality of the pass operates using STUDIES USING TRIQ
this matrix abstraction, we can flexibly compute We performed real-system measurements for
good mappings for any device topology and a set of 12 benchmarks on 7 QC systems. These
noise profile simply by changing compile-time benchmarks include important QC kernels such
inputs. as the Toffoli gate and quantum Fourier transform

May/June 2020
77
 Top Picks

Figure 5. Success rate for 12 benchmarks on 7 systems. Success rates varies drastically across systems and is
influenced by error rates, qubit connectivity, and application-machine topology match. Benchmarks that are too large to be
mapped onto a machine are marked “X.” This comparison is intended to understand the impact of architectural design
choices such as gate set and connectivity on benchmark performance and is not intended to pick a winning technology,
vendor or implementation. Individual benchmark performance numbers may change over time. These measurements
represent a snapshot of the performance of these systems when we performed the experiments.

operation. To understand architectural choices,
we performed multiple experiments with each
benchmark and system, varying the level of opti-
mization and the inputs used for compilation. We
used three main variants of the compiler with
increasing levels of optimization for gate sequen-
ces, communication and for noise-adaptivity and
a fourth baseline version with no optimization.
We compared different executables in terms
of instruction count and success rate. Figure 5
shows the measured success rates using TriQ’s
full optimizations. The key insights from our
study are summarized next.
Importance of Gate Set Specificity: We studied
whether it is beneficial to expose native gates to
software, instead of abstracting them in a device-
independent gate set. When TriQ has information
about the native gate set, the gate optimization
passes offers significant benefits. TriQ expresses
several program instructions using a small num-
ber of native gates, leading to an average 50%
reduction in the instruction count and up to 26%
increase in success rate. Therefore, unlike prior
proposals for device-independent ISAs for QC sys-
tems,7; 8 our results show that such abstractions
are detrimental to high success rates. We recom-
mend that vendors make the most low-level native
gates in their devices software visible. As an anal-
ogy to classical microprocessors, this is similar to
making microoperations software visible.9
Importance of Qubit Connectivity: Our work
demonstrates that the match between applica-
tion communication requirements and device
topology significantly crucially impacts success
rates. Comparing near-neighbor versus fully-
connected systems (like IBM and UMD systems)
shows that machines with dense qubit connec-
tivity are less sensitive to application character-
istics and allow a wider variety of programs to
execute successfully. Compared to a baseline,
TriQ’s communication optimizations offer up
to 22X reduction in 2Q gate counts. For certain
programs, this means the difference between a
failed execution where noise corrupts the output
and a successful execution where the correct
answers dominate. When the architecture does
not have full connectivity, compilers like ours
can allow applications to take maximum advan-
tage of the available hardware resources.
Importance of Noise Adaptivity: Our work
shows that the noise variability in QC hardware
can be effectively mitigated by software techni-
ques. By mapping programs onto reliable regions
of the hardware and orchestrating communica-
tion along reliable hardware paths, TriQ effec-
tively shields applications from spatiotemporal
noise variations. These optimizations provide
further average success rate gains of 2.8X over
gate and communication optimizations, and
allows more applications to execute successfully.
Put together, TriQ’s optimizations offer up to 1.5-
28X higher success rates than IBM’s Qiskit,
Rigetti’s Quil compiler, and hand optimized code
from UMD. Our work is the first to show that such
optimizations are important even on trapped ion
systems which have less variability. Noise varia-
tions are likely in all near-term QC systems in the
next 5 to 10 years. Therefore, compilers like TriQ
will be crucial for reliable program executions.
TriQ’s functionality is portable across
diverse platforms while still performing full
top-to-bottom optimizations for device and

IEEE Micro
78
application characteristics provided as compile- When they are not well-matched, successful
time inputs. Leveraging microarchitecture executions are unlikely.
details such as native gate sets and noise rates Our work also breaks new ground in QC
was the key to our improvements. Therefore, benchmarking by being distinct from the exist-
QC systems are not yet ready for device-indepen- ing practices of measuring isolated hardware
dent abstraction layers that hide and obstruct characteristics or benchmarking custom-
information flow between hardware and designed applications. On one hand, vendors
software. characterize systems in terms of metrics such as
gate error rates and qubit coherence times.
These metrics are isolated measurements for
IMPACT OF OUR WORK each hardware component, and not direct meas-
Recently, tech news was dominated by dis- urements of program behavior. TriQ enables
cussions of Google’s so-called “quantum suprem- direct and accurate measurements of program
acy” announcement and reactions from other behavior across widely divergent QC platforms.
scientists and QC vendors.1 While QC systems In classical computing, this is akin to the differ-
offering high revenue streams (e.g., as cloud ence between knowing characteristics like core
accelerators) are still in the future, clearly QC is counts and clock rate, versus knowing actual
increasing in importance and has reached an benchmark performance. On the other hand,
inflection point in terms of engineering achieve- vendors have developed benchmarking applica-
ments in real implementations. This makes our tions such as quantum volume. These methods
work extremely timely, with high potential for use a family of custom generated
impact. Just this year, several aca- circuits to measure hardware qual-
demic and industry vendors have Our study features ity. Our work does not field a pre-
already adjusted their compiler systems with different ferred benchmark, but instead
toolflows and aspects of their qubit, noise, and relies on a suite of diverse applica-
exposed gate sets in response to architectural attributes tions to understand the impact of
and provides important
our work. Our optimizations are hardware on applications. This is
insights for designing
already part of IBM’s Qiskit Terra similar to the difference between
better architecture and
compiler as of version 0.8 and benchmarking supercomputers
hardware. These
Rigetti’s Quil compiler version insights will likely with LINPACK or other dedicated
1.16. TriQ, open sourced at influence future QC algorithms, and measuring the per-
https://github.com/prakashmur- ISA design. formance of real applications. We
ali/TriQ is also the first compiler believe that this approach of appli-
for trapped ion systems. cation-based benchmarking will
Our study features systems become common practice in QC, much like how
with different qubit, noise, and architectural benchmark suites such as SPEC are used for clas-
attributes and provides important insights for sical benchmarking.
designing better architecture and hardware. Most importantly, our work represents a sig-
These insights will likely influence future QC nificant advance on the way to practically viable
ISA design. Although QC applications work QC, which requires us to close a five to six order
with any universal gate set, we demonstrate of magnitude gap between algorithm needs
that shielding the natural gates for a qubit and device capabilities. Our work demonstrates
technology by abstracting them into more methods for achieving up to two orders of mag-
common gates imposes severe reliability and nitude improvements in program success rates
performance overheads on NISQ systems. and our approaches work well across vendor
Future QC ISAs need to work in tandem with implementations. In a world where increasing
the underlying qubit technology. Our work qubit count comes only with great engineering
also underscores the importance of matching effort, our work offers substantial and orthogo-
the application’s communication requirements nal advances over underlying hardware progress
and hardware topology by codesigning them. alone.

May/June 2020
79
 Top Picks
ACKNOWLEDGMENTS
This work was supported in part by EPiQC,
an NSF Expedition in Computing, under Grant
CCF-1730082. The work of Cinthia Huerta Alder-
ete was supported by CONACYT under Doctoral
Grant 455378.
& REFERENCES
1. Frank Arute et al. “Quantum supremacy using a
programmable superconducting processor,” Nature,
vol. 574, no. 7779, pp. 505–510, 2019.
2. P. Murali, N. M. Linke, M. Martonosi, A. J. Abhari,
N. H. Nguyen, and C. H. Alderete, “Full-stack, real-
system quantum computer studies: Architectural
comparisons and design insights,” in Proc. 46th Int.
Symp. Comput. Archit., 2019, pp. 527–540.
3. IBM, “IBM Qiskit,” 2018, Accessed on: Jan. 1, 2020.
[Online]. Available: https://qiskit.org/
4. Rigetti, “QuilC compiler,” 2020, Accessed on: Jan. 1,
2020. [Online]. Available: https://github.com/rigetti/quilc
5. A. J. Abhari et al., “Scaffold: Quantum programming
language,” Princeton University, Princeton, NJ, USA,
Tech. Rep. TR-934-12, 2012.
6. D. C. McKay, C. J. Wood, S. Sheldon, J. M. Chow, and
J. M. Gambetta, “Efficient z gates for quantum
computing,” Phys. Rev. A, vol. 96, Aug. 2017,
Art. no. 022330.
7. X. Fu et al., “A microarchitecture for
superconducting quantum processor,” IEEE Micro,
vol. 38, no. 3, pp. 40–47, May 2018.
8. A. W. Cross, L. S. Bishop, J. A. Smolin, and
J. M. Gambetta, “Open quantum assembly language,”
2017, arXiv:1707.03429. [Online]. Available: https://
arxiv.org/abs/1707.03429
9. M. V. Wilkes, “The best way to design an automatic
calculating machine,” in The Early British Computer
Conferences. Cambridge, MA, USA: MIT Press, 1989,
pp. 182–184.
Prakash Murali is currently a Ph.D. student in the
Computer Science Department, Princeton University.
His research focuses on accelerating the progress
toward practical quantum computation using com-
puter architecture and compilation techniques.
Contact him at pmurali@cs.princeton.edu.
80
Norbert M. Linke is currently an Assistant Professor
with the University of Maryland, College Park, and a
Fellow of the Joint Quantum Institute (JQI) working on
quantum algorithm implementations, quantum simula-
tions, and quantum networking with trapped ions. He
was a Postdoctoral Researcher and Research Scientist
with the Group of Chris Monroe, University of Maryland.
Linke received the graduate degree from the University
of Ulm, and the doctorate degree from the University of
Oxford. Contact him at linke@umd.edu.
Margaret Martonosi is currently the Hugh Trum-
bull Adams ’35 Professor of Computer Science with
Princeton University. Her research focuses on com-
puter architecture and hardware–software interface
issues in both classical and quantum systems. Mar-
tonosi received the Ph.D. degree in electrical engi-
neering from Stanford University. She is a Fellow of
IEEE and the Association for Computing Machinery
(ACM). Contact her at mrm@princeton.edu.
Ali Javadi Abhari is currently a Research Staff
Member with IBM, Armonk, NY, USA, and a Manager
of the Quantum Compiler Group. His research interests
include quantum computing software, compilation,
and architecture. Javadi Abhari received the
Ph.D. degree in electrical engineering from Princeton
University. Contact him at ali.javadi@ibm.com.
Nhung Hong Nguyen is currently a Ph.D. student
with Linke Lab, University of Maryland. She was a
Research Assistant with the Center for Quantum
Technology, Singapore, working in satellite quantum
a
key distribution. Her research focuses on digital
quantum simulation, algorithms implementation, and
error encoding on trapped ions. Nguyen received
the B.S. degree in physics from Nanyang Technolog-
ical University, Singapore, working on surface spec-
troscopy with neutral atoms. Contact her at
nhunghng@umd.edu.
Cinthia Huerta Alderete is currently a Ph.D. stu-
dent with the National Institute of Astrophysics,
Optics and Electronics (INAOE), San Andrés Chol-
ula, Mexico, currently on a research stay at the Joint
Quantum Institute, University of Maryland. Her
research is focused on, but not limited to, the simula-
tion of paraparticle oscillators in a trapped-ion sys-
tem. Aside from this topic, she had collaborated on a
few projects based on the circuit implementation of
different phenomena in quantum physics. Contact
her at aldehuer@umd.edu.
IEEE Micro
 Theme Article: Top Picks

Speculative Taint Tracking

(STT): A Comprehensive
Protection for Speculatively
Accessed Data
Jiyong Yu Adam Morrison
University of Illinois at Urbana–Champaign Tel Aviv University
Mengjia Yan Josep Torrellas and Christopher W. Fletcher
Massachusetts Institute of Technology University of Illinois at Urbana–Champaign
Artem Khyzha
Tel Aviv University

Abstract—Speculative execution attacks present an enormous security threat, capable of

reading arbitrary program data under malicious speculation, and later exfiltrating that
data over microarchitectural covert channels. This article proposes speculative taint
tracking (STT), a high-security and high-performance hardware mechanism to block these
attacks. The main idea is that it is safe to execute and selectively forward the results of
speculative instructions that read secrets, as long as we can prove that the forwarded
results do not reach potential covert channels. The technical core of the article is a new
abstraction to help identify all covert channels, and an architecture to quickly identify
when a covert channel is no longer a threat. We further conduct a detailed formal analysis
on the scheme and prove security in a companion document. When evaluated on
SPEC06 workloads, STT incurs 8.5% or 14.5% performance overhead relative to an
insecure machine.

& SPECULATIVE EXECUTION ATTACKS such as

Digital Object Identifier 10.1109/MM.2020.2985359
Date of publication 6 April 2020; date of current version 22
May 2020.
Spectre5 have opened a new chapter in hardware
security. In these attacks, malicious speculative
execution causes doomed-to-squash instruc-
tions to access and later transmit secrets over
covert channels such as the cache.9 For

May/June 2020 Published by the IEEE Computer Society 0272-1732 ß 2020 IEEE
81
 Top Picks

example, in Spectre V1 (see Figure 1) a branch To be secure and efficient, we address two key
misprediction enables the attacker to access challenges.
and leak/transmit arbitrary program data by
 First, we develop an abstraction that indi-
controlling the out-of-bounds address &array1
[off]. We refer to such data, which is brought cates how and when instructions can form
into the pipeline by a speculative instruction, as covert channels, so as to delay data forward-
secret. ing to the latest safe time.
 Second, we identify and develop a microarch-
A secure, but conservative, way to block all
speculative execution attacks— itecture to indicate exactly when data should
regardless of covert channel—is to be considered secret, so as to
delay executing all instructions that This article proposes a re-enable data forwarding at
can access a secret until such new abstraction the earliest safe time.
instructions become nonspecula- through which to view
tive. In nearly all attacks today, this covert channels on Challenge #1: New Abstractions
would imply blocking all loads until speculative microarchi- for Describing All
they are nonspeculative, which tectures, discovers Microarchitectural Covert
new points where
would be tantamount to disabling Channels
instructions can create
speculative execution. Covert channels come in dif-
covert channels, and
This article proposes a princi- ferent shapes and sizes. For
discovers a new class
pled, high-performance mechanism of covert channels example, attackers can monitor
that achieves the same security how loads interact with the
guarantee as the above conserva- cache,5 the timing of SIMD units,6
tive scheme. The key idea is that speculative execution pipeline port contention,2 branch pre-
execution is safe unless speculatively accessed dictor state,1 and more. To comprehensively
data (secrets) reaches a covert channel. In many block leakage through these different channels,
cases, speculative instructions either do not leak it is necessary to understand their common
secrets or do not form covert channels, and so characteristics.
can execute freely under speculation. For exam- To address this challenge, this article pro-
ple, the first load in Spectre V1 forms a covert poses a new abstraction through which to view
channel, but it only leaks the attacker-selected covert channels on speculative microarchitec-
address &array1[off]—not the secret data in tures, discovers new points where instructions
that address. Likewise, many instructions (e.g., can create covert channels, and discovers a new
simple arithmetic) do not form covert channels class of covert channels. We find that all covert
even if their operands are secret values. channels are one of two flavors, which we call
This article presents speculative taint track- explicit and implicit channels (which are related
ing (STT), a framework that tracks the flow of to explicit and implicit information flow8). In an
speculatively accessed data through in-flight explicit channel, data is directly passed to an
instructions (similar to dynamic information instruction whose execution creates operand-
flow tracking/DIFT7) until it is about to reach an dependent hardware resource usage, and that
instruction that may form a covert channel. STT resource usage reveals the data. For example,
then delays the forwarding of the data until how a load impacts the cache depends on the
the instruction becomes nonspeculative or load address.5 In an implicit channel, data indi-
the execution squashes due to mis-speculation. rectly influences how (or that) an instruction(s)
execute, and these changes in resource usage
reveal the data. For example, the instructions
executed after a branch reveal the branch predi-
cate.2,6 This article further defines subclasses of
the implicit channel, based on when the leakage
occurs and based on the nature of the secret-
Figure 1. Spectre variant 1. dependent condition that forms the channel.

IEEE Micro
82
 Key Advance: Safe Prediction. Through its
investigation of implicit channels, this article
makes a key advance by showing how to use
hardware predictors safely. Spectre attacks
were born from attackers mistraining predic-
tors to leak secrets. Through its abstraction
for implicit channels, STT enforces a policy
that prevents arbitrary predictor mistraining
from leaking any secret data over any covert
channel. The article shows how this enables
existing predictors to stay enabled without
leaking privacy, dramatically improving perfor-
mance. In the future, we expect the idea of
safe prediction to enable further innovation,
i.e., by enabling the design of new predictors
without fear of opening new security holes.
Indeed, our follow-on work uses this idea to
safely improve the performance of instruc-
tions that create explicit channels.11
Challenge #2: Mechanisms to Quickly
and Safely Disable Protection
Once we have mechanisms to block secret
data from reaching covert channels, the next
question is when and how to disable that protec-
tion, if speculation turns out to be correct. This is
crucial for performance, as delaying data forward-
ing longer than necessary increases the chance
that delayed instructions reach the head of the
reorder buffer (ROB) and block retirement.
STT tackles this problem with a safe but aggres-
sive approach, by re-enabling data forwarding as
soon as data becomes a function of retired register
file state. This represents the earliest safe point,
but is nontrivial to implement in hardware. For
example, a delayed instruction’s operand(s) may
be the result of a complex dependence chain
across many control flow and speculative opera-
tions. Intuitively, determining that data is a func-
tion of nonspeculative information would require
retracing a backwards slice of the program’s exe-
cution, which is costly to do quickly.
Despite the above challenges, STT proposes a
simple hardware mechanism that can disable pro-
tection/re-enable forwarding for an arbitrary
instruction in a single cycle, using hardware similar
to traditional instruction wake-up logic. The key
idea is that to determine whether data is a function
of retired state, it is sufficient to determine whether
the youngest load, whose return value influences
May/June 2020
the data, has become nonspeculative. Checking
this condition is akin to tracking a single extra
dependence for each instruction, as opposed to
performing complex backwards slice tracking.
Security Guarantees and Formal Analysis
Alongside the main paper, we formally prove
that STT enforces a novel form of noninterfer-
ence3 with respect to speculatively accessed
data. In a nutshell, we show that hardware
resource usage patterns over time are indepen-
dent of data that eventually squashes (covering
microarchitectural interference- and timing-
based attacks). We released a companion techni-
cal report12 with detailed formal analysis and a
security proof for this property.
Putting It All Together
Putting everything together, STT provides
both high security and high performance. It
does not require partitioning or flushing micro-
architectural resources, and does not require
changes to the cache/memory subsystem or the
software stack. When evaluated on SPEC06 work-
loads, STT incurs 8.5% or 14.5% performance
overhead (depending on the threat model) rela-
tive to an insecure machine.
ATTACKER MODEL AND
PROTECTION SCOPE
Attacker Model. STT assumes a powerful adver-
sary that can monitor any microarchitectural
covert channel from anywhere in the system, and
induce arbitrarily speculative execution to access
secrets and create covert channels. For example,
the attacker can monitor covert channels through
the cache/memory system,5 data-dependent arith-
metic,4 port contention,2 branch predictors,1 etc.
Scope: Protecting Speculatively Accessed Data.
We distinguish attacks based on whether the
access instruction is doomed-to-squash (tran-
sient) or bound to retire (nontransient). STT’s
goal is to block attacks involving doomed-to-
squash access instructions, shown in Figure 2.
These attacks can access data that a correct
(not miss-speculated) execution would never
access, which often results in being able to read
from any location in memory. Attacks involving
bound-to-retire access instructions are out of
83
 Top Picks
Figure 2. STT’s scope is to protect speculatively accessed data
from leaking over any microarchitectural covert channel.
Protecting values which have retired is outside of scope.
scope. They can only leak retired (or bound to
retire) register file state, not arbitrary memory,
and their leakage can be reasoned about by pro-
grammers or compilers and blocked using com-
plementary techniques (e.g., Yu et al.10).
ABSTRACTION FOR COVERT
CHANNELS
STT proposes a novel abstraction for covert
channels (see Figure 3). In our abstraction,
covert channels are broken into two classes:
explicit and implicit channels. An explicit channel,
related to explicit flow in information flow,8 is
one where data (e.g., a secret) is directly passed
to an instruction whose execution creates oper-
and-dependent hardware resource usage, and
that resource usage reveals the data. An example
is a load instruction’s changes to the cache
state. An implicit channel, related to implicit
flow,8 is one where data indirectly influences how
(or whether) an instruction or several instruc-
tions execute, and these changes in resource
usage reveal the data. An example is a branch
instruction, whose outcome determines subse-
quent instructions and thus whether some func-
tional unit is used.
We further distinguish between implicit chan-
nels, depending on when they leak secrets and
what type of branch they feature. First, we find
that implicit channels can leak at two points: when
a prediction is made (e.g., a branch prediction)
and when a resolution occurs (e.g., a branch
resolves). Second, we find that implicit channels
84
Figure 3. STT’s new classification schema for
covert channels.
can feature either an explicit or an implicit branch.
An explicit branch is a control-flow instruction,
while an implicit branch is a conceptual branch
formed in the hardware due to an optimization.
For example, store-to-load forwarding between a
store and a younger load can be viewed as an
implicit branch that checks for an address alias as
shown in Figure 4. Written this way, it is clear that
store-to-load forwarding can create a covert chan-
nel: depending on whether there is an alias, the
processor either looks up the cache or forwards
from the local store queue.
Insights From Analysis of Implicit Channels
Since it was proposed in the STT paper, the
classification for implicit channels has proven to
be a robust and useful way to represent and pin-
point the root cause of hardware security vulner-
abilities. For example, in the NetSpectre attack,6
it might be said that the attack root cause is
SIMD unit power-on time. STT’s abstraction
shows, however, that the root cause is an
explicit branch, and that “fixing” the SIMD unit
does not prevent the attack.
Even more subtly, the abstraction demon-
strates and provides cases where implicit flow
and privacy leakage do occur, despite not occur-
ring according to program semantics. Consider a
simple example where a load is control- and
data-independent of a sensitive branch, e.g., “if
(secret == rV) { rX <- rW; } load rZ <- (rY);”.
How this load executes is important, to under-
stand whether a potential cache-based covert
channel exists. Traditional software-level anal-
ysis would indicate that the execution of the
load is independent of the secret (the branch
outcome). Yet, on a speculative microarchitec-
ture, STT’s abstraction would classify this
code as a resolution-based implicit channel.
IEEE Micro
Figure 4. Rewriting a store-load pair as an implicit
branch. implIf reveals a potential covert channel as a
function of memory aliasing to the older store. This
occurs if the microarchitecture supports store-to-
load forwarding or memory-dependence
speculation.
As shown in Figure 5, if the branch mis-specu-
lates and subsequently squashes, the load
may execute either once or twice depending
on the value of secret.
Finally, the abstraction applies to a large set of
microarchitectural optimizations. For example,
the representation of store-to-load forwarding (see
Figure 4) also captures the behavior of memory-
dependence speculation with a store set predictor.
Here, the store set predictor is modeled as a predic-
tion on the implicit branch (implIf in the figure). As
we will see, being able to represent different opti-
mizations as predictions on implicit branches will
enable STT to apply a uniform mechanism to block
leakage through a variety of structures (e.g.,
branch, store set, etc., predictors).
STT: DESIGN
Framework and Concepts
STT requires that the microarchitect define
what instructions write secrets into registers
(access instructions, mainly loads), what instruc-
tions can form explicit channels (transmitters),
and what instructions form implicit channel
branch predicates (for both explicit and implicit
branches). Finally, the architect must define the
Visibility Point, after which speculation is consid-
ered safe (e.g., at the point of the oldest unre-
solved branch, or at the head of the ROB). If
the visibility point refers to an instruction
older than an access instruction, we call the
access instruction unsafe; otherwise it is consid-
ered safe.
We provide guidelines for microarchitects
on identifying access and transmit instruc-
tions. An instruction should be classified as
an access instruction if it has the potential to
read a secret. Except for loads, there are only
May/June 2020
Figure 5. Resolution-based implicit channel due to
secret-dependent pipeline squashes. When the
branch (B) resolves, it leaks the secret based on
whether a squash occurs, as this causes the
younger load to execute once or twice. There is an
analogous case when the (public) predictor state
takes the branch.
a handful of such instructions, which can be
identified manually.
An instruction should be classified as a trans-
mit instruction if its execution creates operand-
dependent resource usage that can reveal the
operand (partially or fully). Identifying implicit
branches is similar: the architect must analyze
whether the resource usage of some in-flight
instruction changes as a function of some other
instruction’s operand. This definition can be for-
malized by analyzing (offline) how information
flows in each functional unit at the SRAM-bit and
flip-flop levels to determine whether resource
usage depends on the input value, in the style of
the OISA10 or GLIFT8 formal frameworks. Auto-
matically performing such analysis is important
future work.
Taint and Untaint Propagation
Conceptually, in each clock cycle, STT
applies the following taint rules to instructions
in the ROB:
The Output Register of an Access Instruction
is tainted if and only if the access instruction
is unsafe.
The output register of a non-access instruc-
tion is tainted if and only if at least one of its
input operands is tainted.
In the implementation, taint propagation is pig-
gybacked on the existing register renaming logic
in an out-of-order core. Tainting is therefore fast.
In contrast, it is difficult to propagate “untaint” to
85
 Top Picks
all dependencies of an access instruction that
becomes safe in a single cycle. We address this
with a single-cycle implementation for untaint in
the “STT: Implementation” section.
Unlike prior DIFT schemes,7 STT does not
require tracking taint in any part of the memory
system or across store-to-load forwarding. The
reason is that because loads are access instruc-
tions, the taint of their output is determined
only based on whether they have reached the
visibility point. That is, the output of an unsafe
load is always tainted.
Blocking Covert Channels
Given STT’s rules for tainting/untainting data
and its abstraction for covert channels, STT
blocks all covert channels by applying a uniform
rule across each type.
Blocking Explicit Channels STT blocks
explicit channels by delaying the execution of
any transmit instruction whose operands are
tainted until they become untainted. This
scheme imposes relatively low overhead because
it only delays the execution of transmit instruc-
tions if they have tainted operands. For example,
a load that only reads a (potential) secret but
does not transmit one—such as the load on line
2 in Figure 1—executes without delay. The load
on line 3, however, will be delayed and eventually
squashed, thereby defeating the attack.
Blocking Implicit Channels STT blocks
implicit channels by enforcing an invariant that
the sequence of instructions fetched/executed/
squashed never depends on tainted data. That
is, STT makes the program counter independent of
tainted data. To enforce this invariant efficiently,
without needing to delay execution of instruc-
tions following a tainted branch, we introduce
two general principles to neutralize the sources
of implicit channels:
Prediction-Based Implicit Channels are
eliminated by preventing tainted data from
affecting the state of any predictor structure.
Resolution-based Implicit Channels are
eliminated by delaying the effects of branch
resolution until the branch’s predicate
becomes untainted.
86
STT’s principles can be applied to efficiently
make any hardware predictor impossible to
exploit as a covert channel for leaking specula-
tively accessed data.
Conceptually, the protection mechanism
does not need to reason about whether
an implicit channel is caused by an explicit
or implicit branch: both types have a predicate
and the policy with respect to the predicate is
the same in both cases. The implementation,
however, must identify the predicate. We illus-
trate this by showing how the STT microarchi-
tecture handles explicit branches.
Applying Principle #1 (Prediction-Based Chan-
nels). STT requires that every frontend predictor
structure be updated based only on untainted
data. This makes the execution path fetched by
the frontend unaffected by the output of unsafe
access instructions. STT passes a branch’s reso-
lution results to the direct/indirect branch pre-
dictors only after the branch’s predicate and
target address become untainted; if the branch
gets squashed before this, the predictor will not
be updated.
Figure 6(c) demonstrates the effect of
STT on a speculative execution of the code
snippet in Figure 6(a), in which the branch
B0 is mispredicted as taken. No matter how
many experiments the attacker runs, the pre-
dicted direction of the branch B will not be a
function of secret, because the branch predic-
tor is not updated when B resolves. As a
result, the execution path does not depend on
secret (top versus bottom)—it only depends
on the predicted branch direction (left versus
right).
Applying Principle #2 (Resolution-Based Chan-
nels). STT delays squashing a branch that resolves
as mispredicted until the branch’s predicate beco-
mes untainted. As a result, a doomed-to-squash
branch with a tainted predicate (such as the
branch B in Figure 6(c)) will never be squashed
and re-executed, preventing the implicit channel
leak discussed in the “Insights From Analysis of
Implicit Channels” section. As Figure 6(c) shows,
the doomed-to-squash branch B is eventually
squashed once an older (mispredicted) branch
with an untainted predicate squashes. Thus, the
squash does not leak any information about the
branch’s resolution. Importantly, it is safe to
IEEE Micro
Figure 6. STT executing the code in (a), which includes an untainted branch B0, an access instruction reading secret,
and an implicit channel (due to branch B). (a) Implicit channel formed through the squash/control dependency on B.
(b) When earlier branch B0 is predicted correctly. (c) When earlier branch B0 is predicted incorrectly (left: B predicts
taken, right: B predicts not taken).
resolve a branch as soon as its predicate becomes
untainted, even if an older branch with a tainted
predicate has not yet resolved.
STT only increases the latency of recovering
from a tainted branch misprediction. For exam-
ple, in Figure 6(b), the load does not execute
immediately after B resolves. Fortunately,
tainted branch mispredictions are only a small
fraction of overall branch mispredictions, which
are infrequent in the first place because
successful speculation requires accurate branch
prediction.
Implicit Branches. The STT paper applies the
above principles to secure several common
microarchitectural optimizations that can be for-
mulated as implicit branches, namely: store-to-
load forwarding, memory dependence specula-
tion, and memory consistency speculation. In
the process, the paper details various optimiza-
tions and cases which arise when dealing with
implicit channels. In particular: whether the
explicit/implicit branch has a prediction step,
can be resolved early or can be optimized in
some other way. For example, because store-to-
load forwarding can only result in two observ-
able outcomes (issue the load or forward from a
prior store), we hide which one occurs by
unconditionally accessing the cache.
STT: IMPLEMENTATION
We previously assumed untaint information
propagated along data dependencies instantly.
This is difficult to implement in hardware
because a word of tainted data may be a function
of complex dependence chains involving many
access instructions.
May/June 2020
A tainted register needs to be untainted once
all access instructions on which it depends
reach the visibility point, i.e., become safe.
Our key observation is that it suffices to track
only when the youngest access instruction
becomes safe, because instructions become non-
speculative in program order in the processor
ROB. We call this youngest access instruction
the youngest root of taint (YRoT).
Determining the YRoT is done through modi-
fications to rename logic in the processor front-
end. Specifically, the YRoT for an instruction X
being renamed is given by the max of 1) the
YRoT(s) of the instruction(s) producing the
arguments for X, if those instructions are not
access instructions; or 2) the ROB index of the
instruction(s) producing the arguments for X,
otherwise. (By convention, we assume the ROB
index increases from ROB head to tail.) After
rename, the YRoT is stored alongside the
instruction in its reservation station and is con-
ceptually an extra dependence for that instruc-
tion. When the visibility point changes, its new
position is broadcast to in-flight instructions,
akin to a normal writeback broadcast, and
instructions whose YRoT is less than the visibil-
ity point’s new position are allowed to execute
(assuming their other dependencies are satis-
fied). The entire architecture requires modest
changes to the frontend rename logic, storage in
reservation stations for the YRoT, and logic to
compare the YRoT to the visibility point which is
comparable to normal instruction wakeup logic.
Figure 7 shows an example. Assume the Spec-
tre attack model, i.e., that the visibility point will
be set to the ROB index of the oldest unresolved
branch. The ROB contains three unresolved
87
 Top Picks
Figure 7. Example of YRoT tracking showing a
snapshot of ROB state. Addition (add) instructions are
used to represent arithmetic (non-loads). If the YRoT
is set to “-,” that means the instruction’s youngest
dependent access instruction is a part of retired state.
branches (B1–B3) and a transmit instruction
(M3) whose operand/address r8 is a function of
the return value of two access instructions (M1
and M2). M3 is a transmit instruction (because it
is a load) and can potentially leak secrets
because mis-speculations on branches B1 and
B2 can influence the data returned by loads M1
and M2, which in turn contribute to the address
of M3 through data dependencies.
On one hand, the data dependence chain from
load M1 all the way to load M3 is quite complex.
That is, the instruction at ROB index 6 depends
Figure 8. Performance evaluation on SPEC06 and PARSEC
benchmark suites. STT outperforms the baseline secure scheme
(DelayExecute) with much smaller performance overhead, for
both Spectre and Futuristic attacker models.
88
on index 5 and index 3, index 8 depends on 6, etc.
Re-traversing this dataflow graph to propagate
untaint, akin to tracing backwards slices, would
be expensive. On the other hand, the YRoT
dependence chain is relatively simple. Each
instruction just tracks whichever is the youngest
load that contributes to its dependence chain
(e.g., load M2 for instructions 6, 8, and 9). When
branches B1 and B2 resolve, the visibility point
advances to branch B3 (ROB index 7). Since 7 is
greater than 5 (the YRoT for the transmit instruc-
tion M3), M3 is now allowed to execute. Note, the
dependence chain could have been more com-
plex, with additional branches and arithmetic
dependencies separating load M2 and load M3,
but this would not change the moment that it is
safe to execute load M3.
Importantly, the above scheme is only secure
after applying STT’s mechanisms to block both
explicit and implicit channels (see the “STT:
Design” section). That is, the scheme requires
that r8 is not a function of speculative data at
the exact moment load M2 becomes nonspecula-
tive. This requires that branch B3 not be influ-
enced by speculative data (achieved by
protections for implicit channels) and that other
intervening instructions that can cause explicit
channels not execute until they are likewise safe
(achieved by protections for explicit channels).
FORMAL ANALYSIS/SECURITY
PROOF
We formally prove12 that STT enforces a
novel notion of noninterference: at each step of
the execution, the value of a doomed register—a
register written to by a bound-to-squash access
instruction—does not influence future visible
events in the execution. This applies to all micro-
architectural timing- and interference-based
attacks. For instance, the property ensures that
the program’s completion time and hardware
resource usage—for all hardware structures
including cache, branch predictor, etc.,—is
completely independent of doomed values.
The key challenge in the analysis is how to
avoid “looking into the future” to determine if an
instruction is doomed to squash. We address
this by running the STT machine alongside a
nonspeculative in-order processor, which allows
IEEE Micro
us to verify the STT machine’s branch predic-
tions and determine whether a prediction leads
to mis-speculation.
EVALUATION RESULTS
We evaluate STT on 21 SPEC and 9 PARSEC
workloads. The results are shown in Figure 8. Rela-
tive to an insecure machine, STT adds only 13.0%/
18.2% overhead (averaged across both SPEC and
PARSEC benchmarks), depending on whether the
attack model considers only control-flow specula-
tion (Spectre) or all types of speculation (Futuris-
tic). This indicates that defending against stronger
attack models is viable with STT without sacrific-
ing much performance. Compared to the baseline
secure scheme (DelayExecute) described in the
introduction, STT reduces overhead by 4:0 in
the Spectre model and 10.5 in the Futuristic
model, on average.
ACKNOWLEDGMENTS
We thank J. Emer, S. Adve, and S. Mukherjee
for very helpful discussions. We would especially
like to thank our colleagues at Intel who contrib-
uted significant feedback throughout the project’s
development, in particular, F. Liu, M. Fernandez,
F. McKeen, and C. Rozas. This work was supported
in part by NSF under Grant CNS-1816226; in part by
Blavatnik ICRC at TAU, ISF under Grant 2005/17;
and in part by an Intel Strategic Research Alliance
Grant.
& REFERENCES
1. O. Aciicmez, J.-P. Seifert, and C. K.
“Predicting secret keys via branch prediction,”
Proc. 7th Cryptographers’ Track RSA Conf. Topics
Cryptology, 2007, pp. 225–242, doi: 10.1007/
11967668_15.
2. A. Bhattacharyya et al., “SMoTherSpectre: Exploiting
speculative execution through port contention,” in
Proc. ACM SIGSAC Conf. Comput. Commun. Secur.,
2019, pp. 785–800.
3. J. A. Goguen and J. Meseguer, “Security policies and
security models,” in Proc. IEEE Symp. Secur. Privacy,
1982, Art. no. 11.
€ dl, E. Oswald, D. Page, and M. Tunstall,
4. J. Groscha
“Side-channel analysis of cryptographic software via
early-terminating multiplications,” in Proc. Int. Conf.
Inf. Secur. Cryptology, 2009, pp. 176–192.
May/June 2020
5. P. Kocher et al., “Spectre attacks: Exploiting
speculative execution,” in Proc. IEEE Symp. Secur.
Privacy, 2019, pp. 1–19.
6. M. Schwarz, M. Schwarzl, M. Lipp, and D. Gruss,
“Netspectre: Read arbitrary memory over network,” in
Proc. Eur. Symp. Res. Comput. Secur., 2019,
pp. 279–299.
7. G. E. Suh, J. W. Lee, D. Zhang, and S. Devadas,
“Secure program execution via dynamic information
flow tracking,” in Proc. 11th Int. Conf. Archit.
Support Program. Lang. Operating Syst., 2004,
pp. 85–96.
8. M. Tiwari, H. M. Wassel, B. Mazloom, S. Mysore, F. T.
Chong, and T. Sherwood, “Complete information flow
tracking from the gates up,” in Proc. 14th Int. Conf.
Archit. Support Program. Lang. Oper. Syst., 2009,
pp. 109–120.
9. Y. Yarom and K. Falkner, “Flush+Reload: A high
resolution, low noise, L3 cache side-channel attack,”
in Proc. Usenix Secur. Symp., 2014, pp. 719–732.
10. J. Yu, L. Hsiung, M. E. Hajj, and C. W. Fletcher, “Data
oblivious ISA extensions for side channel-resistant and
high performance computing,” in Proc. 26th Netw.
Distrib. Syst. Secur. Symp. [Online]. Available: https://
eprint.iacr.org/2018/808.
11. J. Yu, N. Mantri, J. Torrellas, A. Morrison, and C. W.
Fletcher, “Speculative data-oblivious execution (SDO):
Mobilizing safe prediction for safe and efficient
speculative execution,” in Proc. Int. Symp. Comput.
Archit., 2020.
12. J. Yu, M. Yan, A. Khyzha, A. Morrison, J. Torrellas, and
C. W. Fletcher, “Speculative taint tracking (STT): A
formal analysis,” Univ. Illinois Urbana–Champaign and
Tel Aviv Univ., Tech. Rep., 2019. [Online]. Available:
Koc, http://cwfletcher.net/Content/Publications/Academics/
TechReport/stt-for mal-tr_micro19.pdf.
Jiyong Yu is currently working toward the Ph.D.
degree at the University of Illinois at Urbana–Cham-
paign. His research interests are in processor secu-
rity. Contact him at jiyongy2@illinois.edu.
Mengjia Yan is an Assistant Professor in the
Electrical Engineering and Computer Science
Department, Massachusetts Institute of Technology.
Her research interest lies in the areas of computer
architecture and hardware security, with a focus on
side channel attacks and defenses. Yan received
the Ph.D. degree from the University of Illinois
at Urbana–Champaign (UIUC). Contact her at
mengjia@csail.mit.edu.
89
 Top Picks
Artem Khyzha is a Postdoctoral Fellow in the
School of Computer Science, Tel Aviv University.
His research interests include formal methods
for software and hardware systems. Khyzha
received a joint Ph.D. degree in computer science
from Technical University of Madrid and IMDEA
Software Institute. Contact him at artkhyzha@
mail.tau.ac.il.
Adam Morrison is an Assistant Professor in the
School of Computer Science, Tel Aviv University.
His research interests include high-performance
trustworthy computer and systems architecture.
Morrison received the Ph.D. degree in computer
science from Tel Aviv University. Contact him at
mad@cs.tau.ac.il.
90
Josep Torrellas is the Saburo Muroga Professor of
Computer Science at the University of Illinois at
Urbana–Champaign (UIUC). He is the Director of the
Center for Programmable Extreme Scale Computing
and past Director of the Illinois-Intel Parallelism Center.
His research interests include computer architecture
and parallel processing. Torrellas received the Ph.D.
degree from Stanford University. Contact him at
torrellas@cs.uiuc.edu.
Christopher W. Fletcher is an Assistant Professor
in computer science at the University of Illinois at
Urbana–Champaign. He has interests ranging from
computer architecture to security to high-performance
computing (ranging from theory to practice, algorithm
to software to hardware). Fletcher received the Ph.D.
degree from Massachusetts Institute of Technology in
2016. Contact him at cwfletch@illinois.edu.
IEEE Micro
 Theme Article: Top Picks

MicroScope: Enabling
Microarchitectural
Replay Attacks
Dimitrios Skarlatos Read Sprabery
University of Illinois at Urbana–Champaign Google
Mengjia Yan Josep Torrellas and Christopher W. Fletcher
Massachusetts Institute of Technology University of Illinois at Urbana–Champaign
Bhargava Gopireddy
Nvidia

Abstract—A microarchitectural replay attack is a novel class of attack where an

adversary can denoise nearly arbitrary microarchitectural side channels in a single run of
the victim. The idea is to cause the victim to repeatedly replay by inducing pipeline
flushes. In this article, we design, implement, and demonstrate our ideas in a framework,
called MicroScope, that causes repeated pipeline flushes by inducing page faults. Our
main result shows that MicroScope can denoise the port contention channel of execution
units. Specifically, we show how MicroScope can reliably detect the presence or absence
of as few as two divide instructions in a single logical run of the victim program. We also
discuss the broader implications of microarchitectural replay attacks.

& IT IS NOW well understood that modern pro-
cessors leak secrets over microarchitectural
side and covert channels. These channels
are seemingly everywhere—from the cache1–3
to the branch predictor4 and other struc-
tures5,6—and are capable of leaking program
Digital Object Identifier 10.1109/MM.2020.2986204
Date of publication 16 April 2020; date of current version 22
May 2020.
secrets at many points in a program’s
execution.
Yet, a fundamental challenge for attackers
exploiting these channels is that the channels
are notoriously noisy. This means that multiple
measurements of the same event often return
wildly different values. This occurs, for example,
when attempting to glean secret-dependent con-
trol flow by measuring port contention inside
the pipeline.6 As a result, the attacker requires
that the victim program run many times (e.g.,

May/June 2020 Published by the IEEE Computer Society 0272-1732 ß 2020 IEEE
91
 Top Picks

thousands of times6) to increase the signal-to- secret is leaked. All the while, the victim has
noise ratio. This fact prevents attackers from logically run only once.
learning secrets in many important scenarios, This article introduces microarchitectural
such as when the victim program runs only replay attacks, and provides a complete proto-
once. type tool called MicroScope that runs the
To eliminate this limitation, this work attack on real hardware.
We investigate the
introduces microarchitectural replay attacks attack’s ability to leak secrets in straight-line
(MRAs), a new class of attacks that enables an code, branches, and loops. As a proof of con-
attacker to offset the measurement variation cept, we demonstrate how the attack can
for (i.e., to denoise) potentially any microarch- denoise the notoriously noisy side channel of
itectural side channel, even if pipeline port contention in a sin-
the victim code is executed only gle run of the victim. Finally, we
This article introduces
once. The key observation is discuss how changing different
microarchitectural
that, in modern out-of-order parameters in the attack setup
replay attacks, and
speculative cores, a dynamic yields new flavors of the attack—
provides a complete
instruction may be forced to exe- prototype tool called e.g., enabling an attack to theo-
cute multiple times due to pipe- MicroScope that runs retically bias the output of a
line squashes caused by page the attack on real hardware instruction that gener-
faults, exceptions, or other hardware. ates true random numbers.
events. By forcing the squash We released the full Micro-
and reexecution of an instruction Scope framework as a kernel mod-
multiple times, the attacker can repeatedly ule, available at https://github.com/dskarlatos/
measure the execution characteristics of such MicroScope.
instruction. We call this attack an MRA.
This work also describes and implements a
specific family of MRAs that are applicable in BRIEF BACKGROUND
the context of Intel’s Software Guard Extensions Secure enclaves,8 such as Intel’s SGX7 allow
(SGX).7 Specifically, in this environment, the sensitive user-level code to run securely on a
attacker controls the operating system (OS) and, platform alongside an untrusted supervisor
while the attacker cannot see the victim’s data (i.e., an OS and/or hypervisor). Intel’s SGX uses
directly, it controls the victim’s demand paging. the OS for translation lookaside buffer (TLB)
Now, suppose that there is an instruction I that, and page table management. Each page table
based on secret data, forms a noisy side or entry contains a present bit, which identifies if
covert channel. In addition, suppose that the the physical page is present in memory or not.
attacker finds a public-address load L that is If the bit is cleared, then the translation process
older than I and is in the reorder buffer (ROB) at fails and a page fault exception is raised. The
the same time as I. In this case, the attacker can OS is then invoked to handle it. To keep the TLB
arrange for L to page fault after a long page coherent while updating page table entries, the
walk (e.g., by clearing the present bit of the OS can selectively flush TLB entries through the
corresponding page table entry, and evicting INVLPG instruction.
the multilevel page table entries from the
cache). While the page walk is underway, I exe-
SUMMARY OF THE MICROSCOPE
cutes and the attacker observes a noisy sam-
ATTACK
ple. Then, the OS pretends to service the page
MRAs are based on the key observation that
fault but keeps the present bit cleared. As a
modern hardware allows recently executed, but
result, L will go through the page walk again
not retired, instructions to be rolled back and
and I will execute again. This process is
replayed if certain conditions are met. This
repeated many times, causing the replay of I
an arbitrary number of times until the signal-

The name MicroScope comes from the attack’s ability to peer inside nearly
to-noise ratio is reduced enough that the any microarchitectural side channel.

IEEE Micro
92
Figure 1. Timeline of a MicroScope attack. The Replayer is an untrusted OS or hypervisor process that forces
the Victim code to replay, enabling the Monitor to denoise and extract the secret information.
behavior can be exploited to mount a variety of
attacks (see the “Generalizing Microarchitec-
tural Replay Attacks” section).
An MRA attack has three actors: Replayer,
Victim, and Monitor. In a MicroScope attack,
which is a type of MRA, the Replayer is a mali-
cious OS or hypervisor that is responsible for
page table management. The Victim is an applica-
tion process that executes on some secret data
that the attacker wishes to exfiltrate. The Moni-
tor is a malicious process that performs auxiliary
operations, such as causing contention and mon-
itoring shared resources. Figure 1 shows the
timeline of the interleaved execution of the
Replayer, Victim, and Monitor for a MicroScope
attack.
Attack Setup
MicroScope is enabled by what we call a
Replay Handle. A replay handle can be any mem-
ory access instruction that occurs shortly before
one or more security-sensitive instructions in
program order.
In MicroScope, the Replayer sets up the
attack by locating the page table entries req-
uired for virtual-to-physical translation of the
replay handle. Then, it performs the following
steps, shown in the timeline  1 of Figure 1. First,
it flushes the replay handle data from the cache.
After that, it clears the present bit of the leaf
page table entry of the replay handle. After that,
it flushes the translation page table entries from
the caches. Finally, it flushes the TLB entry that
stores the virtual-to-physical translation of the
replay handle access. Together, these steps will
cause the replay handle to miss in the TLB,
induce a hardware page walk to locate the
May/June 2020
translations, and eventually suffer a page fault.
In the meantime, instructions that are younger
than the replay handle, including the sensitive
instruction(s), can execute speculatively but not
retire.
Speculative Execution in the Shadow of Page
Walks
After the attack is set up, the Replayer allows
the Victim to resume execution and issue the
replay handle, as shown in timeline  3 of Figure 1.
The replay handle access misses in the L1 TLB, L2
TLB, and page walk cache (PWC), and initiates a
page walk. The hardware page walker fetches the
necessary page table entries sequentially, start-
ing from page global directory (PGD), then page
upper directory (PUD), page middle directory
(PMD), and finally page table entry (PTE).
The Replayer can tune the duration of the
speculative execution by choosing whether vic-
tim’s page table entries are either present or
absent from the cache hierarchy and PWC (shown
in the arrows above timeline  3 of Figure 1). The
speculative instructions executing in the shadow
of the page walk may leave some state in the cache
subsystem and/or create contention for hardware
structures in the core. This allows the Monitor to
perform a noisy measurement of the secret data.
At the end of the page walk, the hardware raises a
page fault exception and squashes the speculative
instructions in the pipeline.
The Replayer is then invoked to handle the
page fault. The operation is shown in timeline  2
of Figure 1. The Replayer chooses to keep the
present bit cleared. Timeline  4 of Figure 1
shows the actions of the Victim. In this case,
after the Victim resumes and reissues the replay
93
 Top Picks
Figure 2. Simple examples of codes that present opportunities for MicroScope attacks.
(a) Single secret. (b) Loop secret. (c) Control flow secret.
handle, the whole process repeats. This process
can be repeated as many times as desired to
denoise and extract the secret information.
Monitoring Execution
The Monitor extracts the secret information
from the Victim. Depending on the side channel once in straight-line code. Furthermore, the
being exploited, the monitor can cause resource Replayer can then clear the present bit of a later
contention in parallel to the victim execution, or replay handle in the application and proceed to
prime and inspect the cache state in between monitor a later section of the code.
replays. This is shown in timeline 
5 of Figure 1,
where the Monitor executes in parallel with the
Victim’s speculative execution.
Attack Summary
The attack has the following six steps.
1) The Replayer identifies a replay handle and
prepares the attack—e.g., by priming micro-
architectural state.
2) When the Victim executes the replay handle,
it suffers a TLB miss followed by a page walk.
The time taken by this step can be over 1000
cycles, and can be tuned as per the require-
ments of the attack.
3) In the shadow of the page walk and until the
page fault is serviced, the Victim continues
to execute speculatively past the replay han-
dle into the sensitive region, potentially until
the ROB is full.
4) The Monitor can cause and measure conte-
ntion on shared hardware resources during
the Victim’s speculative execution, or inspect
the hardware state left by the Victim’s specu-
lative execution.
5) When the replay handle triggers a page fault,
the Replayer gains control and can optionally
leave the present bit cleared in the PTE
entry. This will induce another replay cycle
that the Monitor can leverage to collect more
information. Before the replay, the attacker
94
may also prime
the processor
state for the next
measurement.
6) When sufficient
measurements
have been gath-
ered, the Replayer
sets the present
bit in the PTE entry. This enables the Victim to
make forward progress.
With these steps, MicroScope can denoise a
side channel formed by, potentially, any instruc-
tion(s)—even ones that expose a secret only
SIMPLE ATTACK EXAMPLES
Figure 2 shows several examples of codes that
present opportunities for MicroScope attacks.
Each example showcases a different use case.
Single-Secret Attack
Figure 2(a) shows a simple code that has a sin-
gle secret. Line 2 accesses a public address (i.e.,
known to the OS). This access is the replay handle.
After a few other instructions, sensitive code
at Line 4 processes some secret data. We call this
computation the transmit computation of the
Victim, using terminology from prior work.9 The
transmit computation may leave some state in the
cache or may use specific functional units that cre-
ate observable contention. The goal of the adver-
sary is to extract the secret information. The
adversary can obtain it by using MicroScope to
repeatedly perform steps (2)–(5) from the “Attack
Summary” section.
Loop-Secret Attack
We now consider the scenario where we want
to monitor a given instruction in different itera-
tions of a loop. We call this case Loop Secret, and
show an example in Figure 2(b). In the code, the
loop body has a replay handle and a transmit oper-
ation. In each iteration, the transmit operation
accesses a different secret. The adversary wants
to obtain the secrets of all the iterations. The
IEEE Micro
challenging case is when the replay handle maps
to the same physical data page in all the iterations.
This scenario highlights a common problem in
side channel attacks: secret[i] and secret[i+1]
may induce similar effects, making it hard to dis-
ambiguate between the two. For example, both
secrets may be colocated in the same cache line,
or induce similar pressure on the execution units.
This fact severely impedes the ability to distin-
guish the two accesses.
MicroScope addresses this challenge by
using a second memory instruction to move
between the replay handles in different itera-
tions. This second instruction is located after
the transmit instruction in program order, and
we call it the Pivot instruction. For example, in
Figure 2(b), the instruction at Line 6 can act as
the pivot.
MicroScope uses the pivot as follows. After
the adversary infers secret[i] and is ready to pro-
ceed to extract secret[i+1], the adversary per-
forms one additional action during step 6 in the
“Attack Summary” section. Specifically, after set-
ting the present bit in the PTE entry for the replay
handle, it clears the present bit in the PTE entry
for the pivot, and resumes the Victim’s execution.
As a result, all the Victim instructions before the
pivot are retired, and a new page fault is incurred
for the pivot.
When the Replayer is invoked to handle the
pivot’s page fault, it sets the present bit for the
pivot and clears the present bit for the replay
handle. When the Victim resumes execution, it
retires all the instructions of the current iteration
and proceeds to the next iteration, suffering a
page fault in the replay handle. Steps 2–5 repeat
again, enabling the monitoring of secret[i+1].
The process is repeated for all the iterations.
Control Flow Secret Attack
A final scenario that is commonly exploited
using side channels is a secret-dependent branch
condition. We call this case Control Flow Secret,
and show an example in Figure 2(c). In the code,
the direction of the branch is determined by a
secret, which the adversary wants to extract.
As shown in the figure, the adversary uses a
replay handle before the branch, and a transmit
operation in both paths out of the branch. The
adversary can extract the direction taken by the
May/June 2020
Figure 3. Latencies measured by performing a port
contention attack. (a) Victim executes two multiply
operations. (b) Victim executes two division
operations.
branch using at least two different types of side
channels.
First, if lines 3 and 5 in Figure 2(c) access dif-
ferent cache lines, then the Monitor can perform
a cache based side-channel attack to identify the
cache line accessed, and deduce the branch
direction. A second case is when the two paths
out of the branch perform different computa-
tions. In this scenario, the Monitor can apply
pressure on the functional units and, by monitor-
ing contention, deduce the operation that the
code performs and, hence, the branch direction.
EVALUATION
We validated MRAs and MicroScope by
denoising a notoriously noisy side channel: exe-
cution unit port contention.6 For this attack, we
assume the SGX threat model. We use victim
code similar to the one in Figure 2(c), where one
side of the branch executes two division opera-
tions, and the other side executes two multipli-
cation operations. The Replayer forces the
replay of the code. Concurrently, the Monitor
executes a loop with one division operation in
each iteration. We measure the time taken by
each iteration of the Monitor loop. If the Victim
executes the code with the two multiplications,
the Monitor instructions execute fast and,
hence, no contention is measured. Figure 3(a)
shows the latency of each iteration of the Moni-
tor. We see that all but four of the samples take
less than 120 cycles, which we identify to be the
95
 Top Picks
Figure 4. Generalized MRAs.
contention threshold for our machine. If,
instead, the Victim executes the code with the
two divisions, the Monitor instructions execute
slowly. Figure 3(b) shows the latency of each
iteration. We see that 64 measurements are now
above the threshold of 120 cycles. The two cases
are clearly distinguishable. Overall, MicroScope
is able to detect the presence or absence of two
division instructions, outside of a program loop,
using replays.
We further used MicroScope to perform a
cache side-channel attack on AES.10 MicroScope
is able to extract the lines accessed in the AES
tables without noise in a single run.
GENERALIZING MRAs
Figure 4 presents a generalized overview of
MRAs. It has four parts: replay handle, replayed
code, side channel, and strategy. In the attack that
we described in the previous sections, the replay
handle is a page fault-inducing load, the replayed
code contains certain instructions that leak pri-
vacy, the side channels discussed are related to
caches and functional unit ports, and the
attacker’s strategy is to unconditionally page fault
until it has high confidence that it has extracted
the secret. We now discuss how to create different
attacks by changing some of these components.
Attacks on Program Integrity
MRAs can be used to violate program integ-
rity, if the replayed instructions are nondetermin-
istic. For example, suppose we replay the Intel
true random number generator RDRAND instruc-
tion. If the attacker can read the return value
through a side channel, it can selectively replay
RDRAND until the value matches some desired
criteria, effectively biasing the random number
generation from the victim’s perspective.y
y introduce substantial performance overhead, or
This attack does not work on Intel machines due to a technicality. We veri-
fied that modulo this technicality, it should work.
96
Attacks Using Different Replay Handles
MRAs can exploit many different sources of
replay beyond page faults. For example, they
can exploit transaction aborts in transactional
memory. Other instructions enable a limited
number of replays. They include any event that
can squash speculative execution,11 such as a
branch misprediction or a load-to-store alias.
Amplifying Physical Side Channels
MRAs may also be an effective tool to amplify
physical channels, such as power and EM.12 The
idea is that a replay attack can turn nonrepeating
code into repeating code, which power meters,
temperature sensors, and frequency analyzers
can interpret better.
FUTURE RESEARCH DIRECTIONS
AND APPLICATIONS
The work presented in this article can be
extended in numerous ways. In this section, we
describe some possible directions.
Potential Countermeasures Against MRAs
A future direction is to mitigate MRAs. We are
actively working on this area. The root cause of
MRAs is that individual dynamic instructions
may execute more than once. This can be due to
a variety of reasons (e.g., a page fault, a transac-
tion abort, or a squash due to branch mispredic-
tion). Thus, it is clear that new, general security
properties are required to comprehensively
address these vulnerabilities.
The obvious defense against these attacks
is for the hardware or the OS to insert a fence
after each pipeline flush. However, corner
cases, such as multiple instructions in close
proximity, individually causing replays, need
to be considered. In these cases, even if a
fence is introduced after every pipeline flush,
the adversary can extract information from the
resulting multiple replays.
MicroScope relies on speculative execution
to replay Victim instructions. Therefore, a
defense solution that holistically blocks side
effects caused by speculative execution can
effectively block MicroScope. However, existing
defense solutions have limited defense coverage,
require costly hardware.
IEEE Micro
 Page fault-oriented defense mechanisms
could be effective to defeat MicroScope. Unfortu-
nately, solutions that rely on Intel TSX are not
sufficient, since
TSX itself creates a
new mechanism Overall, with a good
with which to cre- interface, MicroScope
ate replays, may become a unique
debugging tool for
through transac-
sequential and parallel
tion aborts. Thus,
code.
we believe further
research is needed
before applying either of the aforementioned
defenses to any variant of MRA.
MRAs as a Speculative Defense Mechanism
While MicroScope is presented as an attack,
its operation can be used to improve security
defenses. This is because it provides a window
into what speculative execution attacks, such as
Spectre and Meltdown, can do. For example, a
Spectre attack on a given branch cannot affect
subsequent instructions that are more than
an ROB-long distance away from the branch
dynamically. MicroScope can provide this infor-
mation, and thus can determine when an instruc-
tion needs protection.
Furthermore, MicroScope can be used to per-
form black-box analysis of microarchitectural
structures, such as the ROB, load store queue
(LSQ), and others. Controlled fine-grain micro-
architectural replay capabilities can enable
the reverse engineering of hardware structures.
This approach can reveal timing, number of ports,
and interconnect information and, more impor-
tantly, uncover previously unknown behavior of
hardware units under speculative execution. Such
information is not only useful for discovering pre-
viously unknown vulnerabilities, but can further
provide a foundation for defense mechanisms.
Parallel Application Debugging
The mechanism that enables our current
MicroScope prototype, namely the capture and
reexecution of an ROB-sized set of instructions,
is a very useful primitive in software develop-
ment and debugging. Capturing these instruc-
tions can provide insights into the program
state in a way that no current tool can. Replaying
May/June 2020
these instructions deterministically may provide
a way to debug hard-to-reproduce software
bugs, such as data races. Small enhancements
may enable single-stepping the code, or the abil-
ity to change the direction of branches on the
fly. Overall, with a good interface, MicroScope
may become a unique debugging tool for sequen-
tial and parallel code.
& REFERENCES
1. F. Liu, Y. Yarom, Q. Ge, G. Heiser, and R. B. Lee,
“Last-level cache side-channel attacks are practical,”
in Proc. IEEE Symp. Secur. Privacy, May 2015,
pp. 605–622.
2. Y. Yarom, D. Genkin, and N. Heninger, “CacheBleed:
A timing attack on OpenSSL constant time RSA,” Int.
Conf. Cryptographic Hardware Embedded Syst.,
2016.
3. Y. Yarom and K. Falkner, “Flush+Reload: A high
resolution, low noise, L3 cache side-channel attack,”
in Proc. USENIX Secur. Symp., 2014.
4. D. Evtyushkin, R. Riley, N. Abu-Ghazaleh, and
D. Ponomarev, “Branchscope: A new side-channel
attack on directional branch predictor,” in Proc. Int.
Conf. Archit. Support Program. Lang. Oper. Syst.,
2018, pp. 693–707.
5. M. Yan, R. Sprabery, B. Gopireddy, C. Fletcher,
R. Campbell, and J. Torrellas, “Attack directories, not
caches: Side channel attacks in a non-inclusive
world,” in Proc. IEEE Symp. Secur. Privacy, vol. 1,
pp. 56–72, 2019, doi: 10.1109/SP.2019.00004.
6. A. C. Aldaya, B. B. Brumley, S. U. Hassan,
C. P. Garcıa, and N. Tuveri, “Port contention for fun
and profit,” IEEE Symp. Secur. Privacy, 2019,
pp. 870–887.
7. Intel, “Intel software guard extensions programming
reference,” 2013. [Online]. Available: https://software.
intel.com/sites/default/files/329298-001.pdf
8. P. Subramanyan, R. Sinha, I. Lebedev, S. Devadas,
and S. A. Seshia, “A formal foundation for secure
remote execution of enclaves,” in Proc. ACM SIGSAC
Conf. Comput. Commun. Secur., 2017, pp. 2435–2450.
9. V. Kiriansky, I. A. Lebedev, S. P. Amarasinghe,
S. Devadas, and J. Emer, “DAWG: A defense against
cache timing attacks in speculative execution
processors,” in Proc. 51st Annu. IEEE/ACM Int. Symp.
Microarchit., 2018, pp. 974–987.
97
 Top Picks
10. OpenSSL, “Open source cryptography and
SSL/TLS toolkit,” 2019. [Online]. Available: www.
openssl.org
11. C. Canella et al., “A systematic evaluation of
transient execution attacks and defenses,” in Proc.
28th USENIX Secur. Symp., 2019, pp. 249–266.
12. A. Nazari, N. Sehatbakhsh, M. Alam, A. Zajic,
and M. Prvulovic, “EDDIE: EM-based detection of
deviations in program execution,” in Proc. ACM/IEEE
44th Annu. Int. Symp. Comput. Archit., 2017,
pp. 333–346.
Dimitrios Skarlatos is a Ph.D. candidate at the
University of Illinois at Urbana–Champaign. His
research lies at the intersection of computer archi-
tecture, security, and operating systems. He builds
practical solutions that improve the performance
and bolster—or sometimes break—the security
guarantees of computing systems. Contact him at
skarlat2@illinois.edu.
Mengjia Yan is an Assistant Professor in the Electri-
cal Engineering and Computer Science Department,
Massachusetts Institute of Technology. Her research
interest lies in the areas of computer architecture and
hardware security, with a focus on side channel
attacks and defenses. Yan received the Ph.D. degree
from the University of Illinois at Urbana–Champaign
(UIUC). Contact her at mengjia@csail.mit.edu.
98
Bhargava Gopireddy is a Senior Architect at Nvi-
dia, where he works on energy-efficient GPU archi-
tectures. His research interests include energy
efficient many-core architectures and architectural
support for operating systems/security. Gopireddy
received the Ph.D. degree in computer science from
the University of Illinois at Urbana–Champaign in
2018. Contact him at bgopireddy@nvidia.com.
Read Sprabery completed his Ph.D. degree with
a focus on cloud security in 2018 at the University
of Illinois at Urbana–Champaign and now works
as a security researcher at Google. Contact him
at spraber2@illinois.edu.
Josep Torrellas is the Saburo Muroga Professor
of Computer Science at the University of Illinois at
Urbana–Champaign (UIUC). He is the Director of the
Center for Programmable Extreme Scale Computing
and past Director of the Illinois-Intel Parallelism
Center. His research interests include computer
architecture and parallel processing. Torrellas
received the Ph.D. degree from Stanford University.
Contact him at torrellas@cs.uiuc.edu.
Christopher W. Fletcher is an Assistant Professor
in computer science at the University of Illinois at
Urbana–Champaign. He has interests ranging from
computer architecture to security to high-performance
computing (ranging from theory to practice, algorithm
to software to hardware). Fletcher received the Ph.D.
degree from Massachusetts Institute of Technology in
2016. Contact him at cwfletch@illinois.edu.
IEEE Micro
 Theme Article: Top Picks

Creating Foundations
for Secure
Microarchitectures
With Data-Oblivious
ISA Extensions
Jiyong Yu Mohamad El Hajj and Christopher W. Fletcher
University of Illinois at Urbana–Champaign University of Illinois at Urbana–Champaign
Lucas Hsiung
SciFive

Abstract—It is not possible to write microarchitectural side channel-free code on

commercial processors today. Even when we try, the resulting code is low performance. This
article’s goal is to lay an ISA-level foundation, called a Data-Oblivious ISA (OISA) extension,
to address these problems. The key idea with an OISA is to explicitly but abstractly specify
security policy, so that the policy can be decoupled from the microarchitecture and even the
threat model. Analogous to a traditional ISA, this enables an OISA to serve as a portable
security-centric abstraction for software while enabling security-aware implementation and
optimization flexibility for hardware. The article starts by giving a deep-dive in OISA principles
and formal definitions underpinning OISA security. We also provide a concrete OISA built on
top of RISC-V, an implementation prototype on the RISC-V BOOM microarchitecture, a formal
analysis and security argument, and finally extensive performance evaluation on a range of
data-oblivious benchmarks.

Digital Object Identifier 10.1109/MM.2020.2985366

Date of publication 6 April 2020; date of current version 22
May 2020.

May/June 2020 Published by the IEEE Computer Society 0272-1732 ß 2020 IEEE
99
 Top Picks

& A, ARGUABLY THE, central problem in secure side channels. Depending on the microarchitec-
computer architecture today is how to reason ture, this may require closing side channels
about security amid the sea of different micro- through the cache, translation lookaside buffer
architectural side channel attacks. The prevailing (TLB), etc. That is, security is not tied to closing
approach to stop these attacks is to a specific side channel and the
block leakage stemming from one To our knowledge, this programmer works with a simple,
hardware structure at a time. For is the first proposal that portable guarantee across micro-
example, by partitioning or ran- provides a basis to architectures. On the efficiency
domizing the cache layout, we block all traditional side side, each microarchitecture can
block (or at least aggravate) cache channel and specula- choose how to implement the
timing attacks. Yet, many hardware tive execution attacks safe load operation in whatever
structures have been shown to leak on commercial-class way maximizes performance
secrets—from the cache to the microarchitectures. while preserving security (e.g.,
branch predictors,5 speculative by microcoding the load into sim-
execution,8 port contention,1 arithmetic unit tim- pler safe operations,2 or using hardware parti-
ing,2 etc. Given the many avenues to leak a secret, tioning,11 or using cryptographic techniques9).
it is paramount to explore holistic defenses that Safe loads are just one example. More gener-
provide a basis to block leakage through all hard- ally, deciding which instruction operands to des-
ware structures. ignate as safe opens a new, rich ISA design space
In this direction, the article proposes ISA which trades-off performance and hardware
design principles for what we call data-oblivious complexity.
ISAs (OISAs). The key idea with an OISA is to Beyond formulating design principles for
explicitly but abstractly specify security policy, OISAs, the article proposes a concrete OISA
so that the policy can be decoupled from the extension built on top of RISC-V, implements
microarchitecture and even the threat model. (and open sources) that OISA extension on the
Analogous to a traditional ISA, this enables an BOOM out-of-order (OoO) speculative RISC-V
OISA to serve as a portable security-centric core,3 and provides a formal analysis showing
abstraction for software while enabling security- how the OISA provides a basis to achieve nonin-
aware implementation and optimization flexibil- terference (“zero privacy leakage”) on an
ity for hardware. abstract OoO speculative machine. Crucially, the
The OISA proposed in the article annotates security analysis and principles are robust to
what data is confidential and what instruction modern attacks. Case in point, the article’s formal
operands are safe. Inspired by information flow analysis shows how the OISA soundly defeats
policies (in particular, the classic policy High Z speculative execution attacks (such as Spectre8)
Low), the hardware dynamically enforces that without introducing special case reasoning.
confidential data is never passed to unsafe oper- To our knowledge, this is the first proposal
ands, i.e., Confidential data Z Unsafe operands. that provides a basis to block all traditional side
Informally, “safe” in the article means “does not channel and speculative execution attacks on
create a microarchitectural side channel as a func- commercial-class microarchitectures.
tion of the operand” (we also provide formal defi-
nitions), but other notions of safety can be
retrofitted into the implementation without MOTIVATION: SECURE AND
changing the OISA or the programs that sit on EFFICIENT DATA-OBLIVIOUS
top of it. PROGRAMMING
OISAs enable high security, portability, and The OISA project came about by asking the
efficiency. Consider a simple example OISA following question: Is it possible today to write
instruction: a load with a safe address operand. microarchitectural side channel-free programs
Security-wise and portability-wise, the OISA on modern microarchitectures?
guarantees that when the load executes, the The answer is no. Consider the most conser-
address will not leak through microarchitectural vative approach used by practitioners, called

IEEE Micro
100
data-oblivious programming.
In a nutshell, a
data-oblivious program is one whose hardware
resource usage is independent of the program’s
inputs. To write such programs, the guidelines
are to use only simple instructions, or otherwise
ensure that complex instructions do not receive
Confidential data as operands. For example, sim-
ple bitwise math is allowed, but memory
operations/branches with Confidential data as
addresses/predicates are not (out of fear of, e.g.,
cache-based/control flow-related side channels).
Despite being extremely conservative, the
abovementioned guidelines fail in light of ISA-
invisible microarchitecture-specific optimiza-
tions. For example, on one microarchitecture, a
simple integer addition might be safe (e.g., imple-
mented as a single-cycle operation whose timing
is independent of its inputs) while on another it
might be unsafe (e.g., implemented as a bit-serial
operation that skips runs of 0s or zeros to save
time). The article describes 11 like optimizations,
which have been proposed in the literature, or
are otherwise known to be implemented already,
which break data-oblivious program security.
These include data-in-use optimizations (such as
data-dependent arithmetic) and data-at-rest opti-
mizations (such as cache compression).
In particular, the article points out for the
first time that speculative execution breaks data-
oblivious program security, by steering execu-
tion so that Confidential data is consumed by an
instruction whose execution can leak privacy.
This is nontrivial to see for realistic programs,
given the conservative guidelines used to write
data-oblivious code. For example, consider data-
oblivious decryption
1 for (i = 0; i < NUM_ROUNDS; i++)
2 state = OblDecryptRound
(state, rkey [i])
3 leak(state)
That is, perform a fixed number of decryption
rounds, where each round works on a part of the
secret key (rkey) and incrementally updates the
round state (state). Here, we assume that
OblDecryptRound, the round logic, is data obliv-
ious. leak() is a proxy for an instruction that


Data-oblivious programming goes by several other names, e.g., “constant-
time programming” and “programming in the circuit abstraction,” depending
on the community.
May/June 2020
reveals its argument over a microarchitectural
side channel.
This program is legal data-oblivious code:
The branch outcome in each iteration is public
information, the round logic is data oblivious,
and only the plaintext is meant to be revealed
after decryption is complete. Yet, unwanted pri-
vacy leaks because benign mispredictions can
cause the round logic to exit early. In this exam-
ple, an early mispredict of “not taken” allows the
attacker to see state before all rounds complete,
which allows it to perform cryptanalysis and
recover the secret key rkey.
Core Issue: No Abstraction for Security
To summarize, data-oblivious programming
today is insecure and slow. It is insecure because
of ISA-invisible microarchitecture-specific opti-
mizations. It is slow because, out of fear of leak-
ing privacy, programmers are forced into using
only the simplest of instructions.
The article sets out to address these issues
by introducing new ISA-level abstractions
for reasoning about security and enabling higher
performance. A new ISA abstraction addresses
the security problem by defining how instruc-
tions leak privacy across all compliant microarch-
itectures. It further enables higher performance
by allowing data-oblivious programs to take
advantage of higher performance instructions, as
long as those instructions are deemed safe by
the ISA, and gives microarchitects the ability to
optimize those instructions subject to the ISA-
prescribed security policies.
FORMAL DEFINITIONS FOR
MICROARCHITECTURAL SIDE
CHANNELS
To start, the article develops a security defi-
nition for microarchitectural side channel-free
execution. There are two challenges. First, how
to write the definition to account for any possi-
ble microarchitectural side channel. Second,
how to write the definition so that it sheds
insight on which instructions are “safe” from a
microarchitectural side channel perspective.
To define privacy, we adopt a trace-based
indistinguishability style definition inspired by
the oblivious RAM (ORAM)7 literature. We
101
 Top Picks
Figure 1. Changes in resource usage, as a function of data, create microarchitectural side channels. If a
latch is shaded in a given clock cycle, then it means that there is (explicit) information flow from the operands
to that latch in that cycle. Assume operands A and B are two sets of distinct data values, meant to induce
different ALU timings.
consider a program ; which takes public data x
and confidential data y as input. That program’s
execution trace, on a microarchitecture mArch,
i.e., “all the atoms in the universe that are per-
turbed as a result of running ðx; yÞ on mArch,” is
denoted mArchððx; yÞÞ. The subset of this trace
that the attacker can see (called the view) is
denoted ViewðmArchððx; yÞÞÞ. For privacy, we
require that the information in the View does
not depend on confidential information, i.e., that
ViewðmArchððx; yÞÞÞ ’ ViewðmArchððx; y0 ÞÞÞ for
all confidential data y and y0 . In this setting, ’
informally means “equal, given the capabilities
of any computationally bounded adversary.” For
example, in ORAM schemes the view is the
“memory access pattern” and ORAM seeks to
make the memory access pattern independent of
confidential data.
Next, we must define a view that captures any
possible microarchitectural side channel that an
arbitrary software-based attacker can monitor.
This is nontrivial as the attacker can monitor
many aspects of the program’s execution. For
example, its execution time, use of the cache,
arithmetic units, etc. The article makes a key
observation that all of these leakages can be mod-
eled as confidential data-dependent changes in the
program’s hardware resource usage over time. For
instance, both arithmetic units and cache sets
are hardware resources and the fact that they
are used at confidential data-dependent times is
the crux of the attacks.
Then, the question is how to determine
whether a hardware resource is currently being
“used” by a program. (Note that whether a hard-
ware resource is being “used” is independent of
the logic values currently stored in that struc-
ture.) For this, we rely on an explicit gate-level
102
information flow abstraction similar to GLIFT.10
Figure 1 shows an example using an arithmetic
logic unit (ALU) with operand-independent and
then operand-dependent timing.
First assume a single-cycle ALU (see Figure 1,
Case 1). Suppose the input arrives and is stored in
the input latches at the rising edge of cycle 1.
Using terminology from information flow, we say
the input latch is tainted in cycle 1. Now, regard-
less of the logic values of the input, the same
latches are tainted in each cycle thereafter. That
is, the output latches are tainted in cycle 2, etc.
Because which latches are tainted when is indepen-
dent of the operands, we say the single-cycle ALU
does not form a microarchitectural side channel.
Next assume an ALU with operand-dependent
timing (see Figure 1, Case 2). For example, a mul-
tiply operation that takes one or two cycles,
depending on whether an operand is 0. In this
case, depending on the input, the output latch is
either tainted in cycle 2 or cycle 3. Because
which latches are tainted when is dependent on
the operands, we say this ALU can form a micro-
architectural side channel.
Putting everything together, we model the
processor as a state machine composed of com-
binational logic and latches.** The subset of
latches that store the Confidential input are
denoted tainted at the start. Then, the View out-
puts a trace that indicates which subset of
latches are tainted in each cycle. That is, hard-
ware resource usage as a function of time. If the
microarchitecture ensures that the View is inde-
pendent of Confidential data, the microarchitec-
ture does not leak privacy. Conversely, if the
definition is not satisfied, we can pinpoint which
**W.l.o.g., we treat any state element (flip-flop, SRAM cell, etc.) as a latch.
IEEE Micro

Figure 2. Protection policies, checked before each
instruction executes.
instruction caused the problem by looking at
where the Views diverged. (To note, the article
defines taint propagation in a nonstandard way
to model only explicit information flows. This
prevents taint explosion, which would render
the definition not useful. Implicit information
flows are modeled by quantifying over all y0 .)
PRINCIPLES OF OISA DESIGN
The design principles for OISAs are twofold.
First, the OISA should expose security guaran-
tees in a microarchitecture-independent way.
That is, programs written using an OISA should
maintain the same security guarantees across all
OISA-enabled microarchitectures. Second, OISAs
should not preclude modern hardware perfor-
mance optimizations, except when those optimi-
zations have a chance to leak privacy.
To address these goals, the OISA abstraction
proposed in the article has two parts. First, the
OISA labels data to be confidential/public, to
capture whether that data is a function of user
secrets (i.e., the sensitive program inputs from
the “Formal Definitions for Microarchitectural
Side Channels” section). Second, the OISA speci-
fies, for each instruction, whether each instruc-
tion operand is safe or unsafe.
Finally, compliant microarchitectures must
monitor and take different actions based on what
data is consumed by what instruction operands
at runtime. Specifically, hardware must enforce
the following rules (shown in Figure 2).
 (Confidential Z Unsafe) When confidential
data is presented to an unsafe operand: The
hardware must delay that instruction’s exe-
cution until it is nonspeculative. If the rule
still applies when the instruction is nonspec-
ulative, the program terminates with a fault
(as continuing would constitute an informa-
tion leak).
May/June 2020
 (Confidential ! Safe) When Confidential
data is sent to a safe operand: The hardware
designer must add mechanisms to enforce
the security definition given that instruction’s
execution (see the “Formal Definitions for
Microarchitectural Side Channels” section),
for a specified view. For example, by disabling
performance optimizations, scrubbing side
effects and masking exceptions that occur as
a function of confidential operands.
 (Public ! Safe/Unsafe) When public data is
sent to safe or unsafe operands, no special
treatment is needed and execution can pro-
ceed without protection.
Despite these rules’ simplicity, they provide
both security and efficiency benefits. As we will
see in the “Formal Analysis,” they provide a uni-
form handling for both traditional- and specula-
tive-execution-based attacks.8 Case in point, the
only mention of speculation is a detail in the rule
for Confidential Z Unsafe, where we say such an
information flow delays the instruction’s execu-
tion until it is nonspeculative. This removes
false-positive violations due to benign misspecu-
lations. At the same time, the rules enable block-
ing attacks with low overhead. Case in point, the
rules encode some intuitive optimizations such
as “Public data does not need protection” and “it
is safe to compute on confidential data with safe
instructions.” The only situation where instruc-
tion execution is impeded is if confidential data
is consumed by an unsafe operand.y
Key Idea: Abstract security policies facilitate pro-
gramming simplicity, implementation flexibility,
and performance optimizations. Specifying instruc-
tion operand security policy abstractly, i.e., as
safe/unsafe, provides significant flexibility to both
the ISA and hardware designer while simplifying
programmer-level reasoning about security. At
the ISA level, an ISA designer can decide which
instructions are sufficiently important to warrant
safe operands. These choices should be made
carefully: On one hand, safe operands impose a
burden on hardware designers as the processor
must support mechanisms to uphold security viz.,
the “Formal Definitions for Microarchitectural
y
This principle directly inspired our follow-on work to block, specifically,
speculative execution attacks.12
103
 Top Picks
Figure 3. Data-oblivious ISA policy when data is
passed to an instruction operand.
Side Channels” section for those operands. On the
other hand, safe operands do not specify an imple-
mentation strategy. Hardware designers can
implement a given operation using simpler data-
oblivious instructions2 hardware partitioning11 or
cryptographic techniques9—depending on what
is efficient given public parameters and the
specific microarchitecture. In either case, pro-
grammers work with a simple guarantee: Confi-
dential values will not be at risk when consumed
by safe operands, and dynamic execution will be
terminated when violations to this policy are
detected.
Information flow policy and implementation.
The abovementioned OISA framework describes a
relatively simple security lattice (similar to {High,
Lowg),4 policy (High Z Low) and information flow
propagation rule (as written, data should be
marked Confidential if its value is interferrent with
the program’s Confidential inputs, which implies a
taint algebra similar to GLIFT10). This reflects the
article’s goal: to provide a comprehensive, but
simple, privacy guarantee for data-oblivious pro-
gramming while granting implementation flexibil-
ity to tradeoff design cost and performance. Given
other use cases these parameters, and how they
are concretely enforced by an implementation,
can be changed for a family of OISAs, a particular
OISA, or a microarchitecture that implements that
OISA. For example, to support richer security
lattices.6
DESIGN OF A CONCRETE OISA
With the principles in the “Principles for OISA
Design,” section, we now propose a baseline con-
crete OISA that can be easily implemented on top
of common existing ISAs (e.g. 86, ARM, RISC-V).
Figure 3 highlights the instructions included
in the OISA, and which operands are safe/unsafe.
Programmers that write data-oblivious code
104
will recognize this as a formalization of the
guidelines used by data-oblivious programs
today (see the “Core Issue: No Abstraction for
Security”) section. Arithmetic represents all
binary arithmetic operations with safe input
operands. Classify promotes its operand from
public to confidential. Declassify is opposite to
classify, which demotes its operand from confi-
dential to public. Branch performs a conditional
branch, but is only allowed to specify a public
destination or check a public predicate. Load and
store are only permitted to take public addresses.
An important detail is that since declassify has
the potential to make previously protected data
vulnerable, the OISA requires that the declassify
instruction be verified as corresponding to pro-
gram semantics. For example, on a speculative
microarchitecture, this would entail delaying
such an instruction until it is nonspeculative.
Extension: Memory obliviousness via safe-
address loads. A common bottleneck in existing
data-oblivious code is the inability to use confi-
dential data as a load address. Therefore, we pro-
pose a new set of instructions (an oblivious-
memory extension) that enable memory-oblivious
computation.9 Given the OISA design principles,
enabling memory-oblivious instructions is con-
ceptually simple. Instead of emulating memory
obliviousness with dummy memory operations,
we designate new load/store instructions whose
address operand is safe. This gives hardware
designers the ability to build secure and efficient
implementations, e.g., using partitioning11 or
oblivious RAM,7 for that specific operation.
Load instructions with safe address operands
are just one example of how to accelerate secure
computation with an OISA, and the article leaves
extending our concrete OISA with additional safe
instructions as future work. A key insight that
motivates this direction is that many data-oblivi-
ous codes share common kernels (e.g., sorting)
that become performance bottlenecks because
the only available safe operations are simple
instructions. By encapsulating these larger oper-
ations into new instructions with safe operands,
a future OISA can potentially achieve constant
factor or even asymptotic performance improve-
ments. For example, a sort implemented data
obliviously with simple safe instructions may
cost Oðn
log 2 nÞ operations if implemented as a
IEEE Micro

Figure 4. Microarchitectural changes needed to
support the OISA from the “Design of A Concrete
OISA” section (including the oblivious-memory
extension, denoted “omp”). Label stations check and
enforce the transition rules from Figure 2.
bitonic sort. On the other hand, if sort is speci-
fied as a single safe instruction in the OISA, an
implementation based on hardware partitioning
can achieve Oðn
log nÞ time if implemented as a
constant-time merge sort.
HARDWARE PROTOTYPE ON RISC-V
BOOM
We prototype all hardware changes needed
to support our OISA on top of the RISC-V BOOM
processor (for “Berkeley OoO Machine”).3
BOOM is the most sophisticated open RISC-V
processor, featuring modern performance opti-
mizations such as speculative and OoO execu-
tion, and is similar to commercial machines that
run data-oblivious code today.
Microarchitectural changes to support the
OISA are shown in Figure 4. The main changes are
logic at instruction issue/execute to enforce the
rules from the “Principles of OISA Design” section,
storage/logic to implement the oblivious-memory
extension, and logic to track and denote data as
confidential/public. For the latter, we implement a
hardware information flow tracking mechanism
similar to hardware dynamic information flow
tracking, but capable of checking and updating
whether data is confidential/public (the data’s
label) at any stage in the pipeline.
FORMAL ANALYSIS
In parallel to our hardware prototype, we
develop a formal analysis that models an abstract
BOOM-class processor (OoO, speculative, super-
scalar), and describe how to map the abstract
May/June 2020
BOOM to our concrete BOOM prototype. Through
this model, we prove that the OISA provides a
basis to satisfy strong security definitions such as
those we defined in the “Formal Definitions for
Microarchitectural Side Channels” section. Our
security analysis is general, and applies given any
implementation of several important processor
structures (e.g., it models the branch predictor as
an arbitrary function that takes previous branch
resolutions as input).
Importantly, we are able to prove security
while allowing high-performance hardware opti-
mizations (e.g., OoO, speculative execution) to
remain enabled in the common case and without
ever requiring hardware flushes to structures
such as the cache or branch predictors.
Security intuition. Informally, to argue secu-
rity, we need to show the following.
a) Each instruction’s resource usage/side-
effects are independent of Confidential
data.
b) The sequence of instructions that are exe-
cuted, i.e., the processor program counter
(PC), is independent of Confidential data.z
Condition (a) follows by definition by apply-
ing the rules in Figure 2 to each instruction as
it executes. A more subtle point is that condi-
tion (b) also follows from applying the same
rules. To see why, first consider a simple unpi-
pelined, in-order processor with no specula-
tion. In this case, it is clear condition (b) holds
because the only instruction type from Figure 3
that changes the PC as a function of data is a
branch, and the OISA requires that the branch
predicate and target be Public data. What hap-
pens when we consider more advanced pipe-
lines, e.g., with prediction and speculation? In
that case, microarchitectural state outside of
program semantics, e.g., the branch predictor
state, influences the PC. To extend our security
argument to these machines, we must extend
what we mean by “resource usage/side-effect”
to include these structures. Then, using induc-
tion, one can show that if conditions (a) and
(b) hold up to fetching the ith instruction, the
branch predictor state when fetching the
z
Similar requirements on “not tainting” the PC also govern prior work.10
105
 Top Picks
i þ 1th instruction is independent of Confiden-
tial data, and security follows.x
Example: Security against speculative execu-
tion attacks. The abovementioned reasoning
shows how OISAs enable security against both
nonspeculative and speculative attacks. Con-
sider the example speculative execution
attack on data-oblivious decryption from
the “Motivation: Secure and Efficient Data-
oblivious Programming” section. This attack
does not go through when using an OISA by
invoking aforementioned conditions (a) and
(b). That the branch predictor misspeculates
and executes leak() prematurely is not a func-
tion of Confidential data due to condition (b).
Furthermore, when leak() executes, it will be
unconditionally stopped by the Confidential
data Z Unsafe operands rule due to condition
(a). Extending the analysis to attacks like Spec-
tre,8 where the branch predictor is inten-
tionally mistrained, reuses the same logic.
That is, if conditions (a) and (b) hold, then the
attacker’s strategy for how to mistrain the
branch predictor cannot be a function of Confi-
dential data because the program has not
leaked Confidential data up to this point. In
that case, intentional mistraining looks the
same to the analysis as accidental misspecula-
tion, and security follows.
EVALUATION
We evaluate OISAs in terms of hardware area
and performance over a range of existing data-
oblivious programs (including linear algebra,
data structures, and graph traversal). Area-wise,
our proposal takes <5% the area of the unmodi-
fied BOOM processor. Performance-wise, the
OISA and hardware implementation provides
an 8:8/1:7 speedup on small/large datasets,
respectively, relative to data-oblivious code run-
ning on commodity machines (and with the
security and portability benefits stated before).
We also show case studies, where the OISA
speeds up constant-time AES by 4:4 and the
memory oblivious ZeroTrace9 library by 4:6 to
several orders of magnitude, depending on
parameters.
x guage and compiler support—for performing
This idea to keep the predictors a function of Public data directly inspired
the mechanism to block “implicit channels” in our follow-on work.12
106
We have open-sourced our prototype design
on the RISC-V BOOM processor at https://github.
com/cwfletcher/oisa.
DISCUSSION AND FUTURE
DIRECTIONS
OISAs can be extended in numerous direc-
tions, in particular as a way to compose exist-
ing hardware/software defensive mechanisms
and as a novel backend for the data-oblivious
stack.
Simplifying and Composing the Hardware
Trusted Computing Base (TCB)
A major impediment to progress is that many
hardware structures create side channels, and it
is not clear whether the program is “secure”
until all channels are blocked. OISAs dramati-
cally simplify this problem, enabling a new incre-
mental methodology for designing secure
hardware and software.
In their most basic deployment, an OISA might
opt to only support very basic safe instructions
(e.g., bitwise operations and basic arithmetic).
Such an OISA can likely be implemented with min-
imal changes to modern processors and already
improves the state of security today. For exam-
ple, by increasing our confidence that conserva-
tively-written codes such as constant-time codes
are really “constant time.”
Beyond this basic deployment, however,
OISAs provide a way for computer architects
to plug-and-play their high-performance
“point” defenses in a compositional way. For
example, a safe load can be implemented by
previously proposed partitioned or random-
ized cache architectures.11 Importantly, archi-
tects need only worry about how to implement
the safe load. The generic OISA rules, e.g., Con-
fidential data Z Unsafe operands, take care of
the rest.
Composing With the Data-Oblivious Stack
Beyond writing side channel-free code for
today’s hardware, there is a rich literature in
the applied cryptography community—ranging
from algorithm/data structure design to lan-
secure multiparty and encrypted computation.
IEEE Micro
 We make a key observation that the underlying
programming abstraction assumed for those
works is the same abstraction provided by an
OISA. For example, a homomorphic encryption
operation is akin to a safe instruction, just using a
different implementation suitable for a different
threat model. This enables a new, large-scale
research agenda to port insights and advances
made in the applied cryptography community to/
from the microarchitectural side channel commu-
nity. For example, we can enable high-level pro-
gramming abstractions for writing OISA-secure
code by adding a new OISA backend to existing
data-oblivious compiler frameworks. At the same
time, the notion of safe instructions provides a
new theory to explore in applied cryptography. In
particular, algorithm design in encrypted compu-
tation assumes only extremely simple safe opera-
tions (e.g., bit add or multiply). With an OISA,
however, we can choose which operations sup-
port safe operands, and co-design algorithms with
this in mind to improve performance.
& REFERENCES
1. A. C. Aldaya, B. B. Brumley, S. U. Hassan, C. P.
Garcıa, and N. Tuveri, “Port contention for fun and
profit,” in Proc. IEEE Symp. Secur. Privacy, 2019,
pp. 870–887.
2. M. Andrysco, D. Kohlbrenner, K. Mowery, R. Jhala, S.
Lerner, and H. Shacham, “On subnormal floating point
and abnormal timing,” in Proc. IEEE Symp. Secur.
Privacy, 2015, pp. 623–639.
3. C. Celio, P.-F. Chiu, B. Nikolic, D. A. Patterson, and K.
, “BOOM v2: An open-source out-of-order
Asanovic
RISC-V core,” Tech. Rep. UCB/EECS-2017-157, EECS
Dept., Univ. California, Berkeley, CA, USA, 2017.
4. D. E. Denning, “A lattice model of secure information
flow,” Commun. ACM, vol. 19, pp. 236–243, May 1976.
5. D. Evtyushkin, R. Riley, N. C. Abu-Ghazaleh, ECE, and
D. Ponomarev, “BranchScope: A new side-channel
attack on directional branch predictor,” in Proc. 23rd
Int. Conf. Archit. Support Program. Lang. Oper. Syst,
2018, pp. 693–707.
6. A. Ferraiuolo, M. Zhao, A. C. Myers, and G. E. Suh,
“HyperFlow: A processor architecture for
nonmalleable, timing-safe information flow security,” in
Proc. ACM SIGSAC Conf. Comput. Commun. Secur.,
2018, pp. 1583–1600.
May/June 2020
7. O. Goldreich and R. Ostrovsky, “Software protection
and simulation on oblivious rams,” J. ACM, vol. 43,
pp. 431–473, May 1996.
8. P. Kocher et al., “Spectre attacks: Exploiting
speculative execution,” in Proc. IEEE Symp. Secur.
Privacy, 2019, pp. 1–19.
9. S. Sasy, S. Gorbunov, and C. W. Fletcher, “ZeroTrace:
Oblivious memory primitives from Intel SGX,” in Proc.
Netw. Distrib. Syst. Secur. Symp., San Diego, CA,
USA, Feb. 18–21, 2018. Available: http://dx.doi.org/
10.14722/ndss.2018.23239
10. M. Tiwari, H. M. Wassel, B. Mazloom, S. Mysore, F. T.
Chong, and T. Sherwood, “Complete information flow
tracking from the gates up,” in Proc. 14th Int. Conf.
Archit. Support Program. Lang. Oper. Syst., 2009,
pp. 109–120.
11. Z. Wang and R. B. Lee, “New cache designs for
thwarting software cache-based side channel
attacks,” in Proc. 34th Annu. Int. Symp. Comput.
Archit., 2007, pp. 494–505.
12. J. Yu, M. Yan, A. Khyzha, A. Morrison, J. Torrellas, and
C. Fletcher, “Speculative taint tracking (STT): A
comprehensive protection for speculatively accessed
data,” in Proc. 52nd Annu. IEEE/ACM Int. Symp.
Microarchit., 2019, pp. 954–968.
Jiyong Yu is currently working toward the Ph.D.
degree at the University of Illinois at Urbana–Champaign.
His research interests are in processor security. Contact
him at jiyongy2@illinois.edu.
Lucas Hsiung received the B.S. degree from the
University of Illinois at Urbana–Champaign in 2019
and now works as a Security Verification Engineer at
SciFive. Contact him at lucas.hsiung@sifive.com.
Mohamad El Hajj is currently working toward the
M.S. degree at the University of Illinois at Urbana–
Champaign with research interests in hardware
security. Contact him at melhajj2@illinois.edu.
Christopher W. Fletcher is an Assistant Profes-
sor in computer science at the University of Illinois at
Urbana–Champaign. He has interests ranging from
computer architecture to security to high-performance
computing (ranging from theory to practice, algorithm
to software to hardware). Fletcher received the Ph.D.
degree from Massachusetts Institute of Technology in
2016. Contact him at cwfletch@illinois.edu.
107
 Theme Article: Top Picks

Trace Wringing for

Program Trace Privacy
Deeksha Dangwal Joseph McMahan
University of California, Santa Barbara University of Washington
Weilong Cui Timothy Sherwood
Google, Inc. University of California, Santa Barbara

Abstract—A quantitative approach to optimizing computer systems requires a good

understanding of how applications exercise a machine, and real program traces from
production environments lead to the clearest understanding. Unfortunately, even the
simplest program traces can leak sensitive details about users, their recent activity, or even
details of trade secret algorithms. Given the cleverness of attackers working to undo well-
intentioned, but ultimately insufficient, anonymization techniques, many organizations have
simply decided to cease making traces available. Trace wringing is a new formulation of the
problem of sharing traces where one knows a priori how much information the trace is leaking
in the worst case. The key idea is to squeeze as much information as possible out of the trace
without completely compromising its usefulness for optimization. We demonstrate the utility
of a wrung trace through cache simulation and examine the sensitivity of wrung traces to a
class of attacks on Advanced Encryption Standard (AES) encryption.

& PRIVACY IN THE digital age has become
increasingly difficult to achieve and a conten-
tious topic. As technologies that capitalize on
facial recognition, location services, and per-
sonal health tracking become mainstream,
addressing these complex privacy issues is of
foremost importance. Policy makers have put in
place regulations on data protection through the
Digital Object Identifier 10.1109/MM.2020.2986113
Date of publication 8 April 2020; date of current version 22
May 2020.
General Data Protection Regulation (GDPR) and
the California Consumer Privacy Act (CCPA).
Computer scientists and engineers must develop
systems and tools for embedding privacy into
existing and new workflows. In this article, we
describe a new approach to privacy, wringing,
with particular applicability to the problem of
sharing program traces.
When working toward application-tuned sys-
tems, developers often find themselves caught
between the need to share information (so that
partners can make intelligent design choices)
and the need to hide information (to protect

0272-1732 ß 2020 IEEE Published by the IEEE Computer Society IEEE Micro
108
proprietary methods or sensitive data). One been eliminated. While there is no known
place where this problem comes to a head is in mechanism of quantifying the amount of sensi-
the release of program traces; even the simplest tive data that remains in an arbitrary trace, we
memory access traces leak a tremendous can at least say how much total information is
amount of information. For example, we can cap- shared, which provides a useful upper bound.
ture the memory access behavior of a critical If we share only a couple thousand bits about
cryptographic function (which is known to be a a trace, we can then be certain we are not giv-
function of the secret key), a set of ing away every user’s social
lookups corresponding to the pars- The key idea, wringing,
security number by accident.
ing of a social security number, or is to squeeze as much Reconstructing a useful trace
even detailed system configuration information as possible from a few thousand bits of
parameters that are considered a out of the trace without information is hard, but inter-
trade secret. While the sharing of completely compromis- estingly we are free to use any
these traces between technology ing its utility. In the ideal public information about the
partners can lead to more robust and case, only the useful nature of these traces in help-
high-performance systems, it can structure of the trace ing us accomplish this. Com-
also leak highly sensitive informa- remains and all poten- pression, when taken to this
tion, and expose user data to secu- tially sensitive data has extreme and lossy form, con-
been eliminated.
rity vulnerabilities. Today when such nects to privacy in this unex-
traces are needed, programmers may pected way. However, as is
be asked to obfuscate the key algorithm behav- often the case in computer architecture, an
iors to hide sensitive data or provide models of important tradeoff remains between informa-
the system, which approximate the same behav- tion leaked and ability of the trace to capture
ior but omit sensitive parts. Hand-built models the program behavior.
of the system are both tedious to code and of We formalize this new approach specifically in
limited predictive power. Since there is no well- the context of memory address traces in part
defined and well-trusted approach to this prob- because we have many prior trace analysis tech-
lem, developers are often forced to resort to niques to build on.7,9,12 To expose the tradeoff
rough human-language descriptions of the inherent to this problem, we explore a new class
behavior of programs (e.g., it is 80% pointer- of memory trace synthesis techniques based on
chasing). This leads to missed opportunities, ideas from signal processing. By projecting the
frustrated optimization, and the design process address space onto a wrapped 2-D heatmap, we
ultimately suffers. Ideally, engineers would decompose memory behavior into orthogonal
access methods to eliminate any sensitive infor- set of features that can then be replayed to repro-
mation from the traces while still capturing the duce the same “visible” patterns as the traces
program behavior and its interaction with the under examination. Specifically, we use a Hough-
underlying hardware. However, the extent to transformed3 trace to find both constant and
which “sensitive” data influences program strided access patterns. We find that for memory
behavior is rarely understood by a single party, traces it is indeed possible for useful program
and even harder to argue is that it is completely behavior to be conveyed in only a few thousand
absent from a trace. bits. We demonstrate the utility of wrung traces
We present a new formulation of this prob- through cache simulation with bounded leakage,
lem of sharing traces where before release one and even examine the sensitivity of wrung traces
knows (a priori) exactly how much information to a class of attacks on AES encryption.
a trace is leaking in the worst case. The key
idea, wringing, is to squeeze as much informa- TRACE WRINGING AS A NEW GAME
tion as possible out of the trace without The program traces we look at in this article
completely compromising its utility. In the are memory access traces specifically, but more
ideal case, only the useful structure of the trace generally fall into a class of traces useful
remains and all potentially sensitive data has for application-tuning and hardware–software

May/June 2020
109
 Top Picks
Figure 1. Forcing a trace through a channel with a capacity of
only a few bits bounds the amount of sensitive data shared. While
public information such as prior non-private traces can be used in
the creation of the code, the trace to be coded must not be known
to the receiver. The objective is to minimize the number of bits
shared while maximizing the utility of the proxy trace. We measure
the utility in terms of whether or not certain utility tests are passed
by the proxy and/or how close to the original tests results they get.
We present a signal processing approach to reduce the trace to
an n-bit channel.
co-optimization (as opposed to debug traces). A
program trace can contain a tremendous
amount of information about the system under
evaluation. But, as we know, such traces are
invaluable for performance evaluation because
they demonstrate the way the system actually
behaves in the face of the workloads it must
actually handle. While the behaviors are impor-
tant at a high level, rarely are the specific ele-
ments of the trace critical. Rather it is the
relationship between those elements and the
proportions that they appear in the trace that is
often the key. This is of course not a new insight;
what we claim as new is the idea that we can for-
malize these schemes in such a way that it
bounds the amount of information leaked about
a system being traced.
Our privacy argument is simple: if we only
share n bits about a specific trace, then we can-
not leak more than n bits about that trace. In
practice, this means that if we share only a few
thousand bits of information about a trace, then
nothing beyond those bits has been leaked.
While it is not a perfect solution (some informa-
tion might be lost), it says something useful
about the maximum amount of information that
110
can be leaked. For example, it should be impossi-
ble to recover an extensive list of social security
numbers, sensitive health information, or even
an entire set of secret keys from such a trace. To
maximize privacy one wants to give away as little
data as possible about the trace. However, to
maximize utility the opposite is true. The ques-
tion is then how little can one give away from
the trace while still being useful?
Answering this question requires an analysis
across two metrics: information leaked and util-
ity, as described in Figure 1. Information is sur-
prisingly easy to quantify; it is the number of bits
from the secret trace that needs to be
“transmitted” between the full trace (which con-
tains every address) and proxy trace (which is a
stand-in for the full trace and is ready for
release). In Figure 1, Step 1 is to encode the
secret trace. Note that any information from pub-
lic traces or training data can be shared freely
and even hard-coded into the “receiver,” but in
the end, everything you wish to share about the
full trace must be represented in a single n-bit
“packet” (Step 2 in Figure 1). Quantifying utility is
harder and more use-case specific. For memory
address traces, we define a distance function
between cache miss-rates of trace vectors as one
such function (Step 3 in Figure 1), but, in general,
there are many other metrics one might use.
SIGNAL PROCESSING APPROACH
TO WRINGING
Given the above mentioned constraints, the
question is how to encode memory address
trace behavior in a general, and yet incredibly
compact, manner. Our compact representation
must also capture the structure of these traces so
that we can identify, describe, and quantify the
patterns that we care most about. We present a
signal processing pipeline for trace wringing.
Our approach describes traces as a probabilistic
grammar of generators coupled with very high
level accounting of behavior over time. The
“transmitted” bits encode both the structure
and parameters of this scheme.
To understand our approach at a high level it
is useful to start with a visual sense for the struc-
ture of such traces. We project the address trace
onto a fixed-size modulo-mapping of the memory
spaces to create a heatmap. Figure 2 shows such a
IEEE Micro
heatmap for gcc where
instruction count (time)
runs along the x-axis and
the address runs along the
y-axis. If we were to plot
this for the entire memory,
Figure 2. Phases visible in the trace generated by gcc after k-means clustering. Each of
it would clearly be too
the three colors in the bar marks a unique phase in the trace. Note, importantly, that phases
large for such a graph (the
reoccur over time.
distance between the
stack and heap would
dwarf any local behavior), so instead, we plot the Given that both strong temporal and spatial
address modulo a large power of two. Heatmaps locality features show up as lines, decomposition
such as this have the advantage of mapping into a set of line segments is a natural place to
addresses onto a more manageable space, but at start. The Hough transform can be used to then
the same time, keep the spatio–temporal struc- find the locations and orientations of certain geo-
tures that would actually impact a real cache. metric primitives, such as lines, in the given space.
Interesting and intuitive patterns emerge after We apply Hough transformation, a popular com-
looking over this graph. The flat horizontal lines puter vision technique for detecting patterns in
in the graph are patterns of repeating access to a images; for our features, we employ the Hough line
set of addresses. These are high temporal locality transform. Specifically, we use the progressive
behaviors. Sharp diagonal lines, on the other probabilistic Hough transform,5 a rendition of the
hand, are regions of high spatial locality as Hough transform algorithm that only performs
addresses are accessed one after the other in suc- voting on a subset of the input points. These input
cession. If we can concisely capture the character points are chosen based on certain features of the
of these behaviors, without transmitting the expected result, such as a threshold, the length of
addresses themselves, we can minimize the the expected line, interpolation strategies, and the
amount of information leaked. The modulo-mem- angle of the line. By interleaving the voting process
ory heatmaps exhibit hierarchical organization.
with line detection, this algorithm finds the most
Globally, there exists a recurrence of similar pat-
prevalent features first, while also minimizing the
terns in the order of a few tens of thousand
computational load. The progressive probabilistic
instructions, i.e., the presence of program phases,
Hough transform returns a set of lines, with each
and within them, we observe patterns that we
lines (x,y) coordinates in the modulo-memory
associate with the more local memory access
heatmap space. We also introduce a variable,
activity. In order to find some representative of
weight, for each line, which is a measure of dark-
the higher echelons of this hierarchy, we employ
k-means clustering for program phase analysis. 2,9 ness of the line. Some intuition about how the
Rather than encoding the entire trace monolithi- probabilistic Hough transform functions is
cally, we can encode just the k representative described in Figure 3.
clusters independently. By breaking the pattern The list of phase identifiers (the result of clus-
down into a set of simpler behaviors, we can then tering), the two ðx; yÞ coordinates of each line
tackle them one-by-one. Figure 2 shows the result segment detected by the Hough transforms, and
of running the phase detector on the memory the line’s weight in the representative phase, cre-
address trace for gcc. Each of the three colors in ate compact “information packets.” The size of the
the bar in the figure show the occurrence of three total “transmission” is n and bounds the maximum
unique phases in the memory access trace. The amount of information leaked.
technique does a good job of lining up with the After phase detection and Hough-line transfor-
repeating structures in the heatmap. With these mation, we end up with a set of lines for each rep-
phases marked, we can encode the k representa- resentative phase. We can see the decomposition
tive clusters with log2 k bits. of a phase of gcc into lines in Figure 4. Each phase

May/June 2020
111
 Top Picks
Figure 3. We capture information about lines we
observe in trace heatmaps using the Hough
transform. Here, we demonstrate its working. The
points on the test image are surveyed for parameters
in the polar coordinate space described as the
Hough transform. The intersections describe the
parameters of the detected lines. The final figure
shows the probabilistic Hough lines, the more robust
and efficient algorithm. For our heatmaps, we use the
probabilistic Hough line algorithm.
is also assigned a label indicating to which cluster
it belongs to, i.e., which representative phase
“represents” it. Since the structural information of
each phase is encoded in the Hough lines, we can
generate an “address tracelet” for each phase
using the representative’s lines. Phases from the
same cluster may occur intermittently and in dif-
ferent lengths. For all phases in the same cluster,
we generate patterns continuously in a rotating
fashion regardless of the length. Upon picking a
Hough line at time t, we generate an address
“segment” from that line based on a fixed segment
length, which captures locality at a small granular-
ity. The final proxy trace has the same length as
the original trace and captures the most salient
aspects of its behavior while at the same time leak-
ing no more than n bits.
EVALUATION AND OVERVIEW
OF RESULTS
In our pipeline, we pose the problem of shar-
ing traces between technology partners with
112
Figure 4. Producing probabilistic Hough lines5 on
top of the heatmap for a program phase in gcc. The
colors are used to indicate distinct lines produced by
the decomposition. Line length, threshold, line gap,
and line slope are input parameters that can be used
to adjust the tradeoff space between information
leakage and utility. For example, choosing a small
line length will ensure that even the smallest lines are
captured, but at the cost of transmitting information
about even the smallest features. Here, we see how
the choice of parameters can be used as a knob to
control the features we want to encode to find a
suitable tradeoff point.
two subsystems. The trace-wringing subsystem
minimizes the number of bits used to describe
the trace structure, and the generator subsys-
tem uses budgeted information to generate the
proxy trace. Trace wringing includes the genera-
tion of heatmaps from memory access traces,
phase analysis, decomposition of representative
phases into lines, and creation of packets to
transmit this information. Note that there are
actually many possible values of n and different
parameters will result in different tradeoffs
between utility and privacy.
To evaluate the effectiveness of the
approach, we take a subset of the SPEC20066
traces, wring them through our pipeline to a
target number of bits, a bit budget, and evalu-
ate the traces across a range of cache
IEEE Micro
Figure 5. (a) Heatmaps for the sensitive input trace gcc and (b) the trace-wrung proxy generated by our
pipeline. The heatmap of the trace-wrung proxy shows that both global and local features line up with the input
trace. All but the subtlest of patterns are present in the trace-wrung proxy.
configurations with regards to miss rate. We
indeed observe that as the bits of information
leakage increase, the proxy miss rate gets
closer to the ground truth miss rate, which
confirms that with more information shared,
the proxy trace we reconstruct becomes more
similar to the original trace in terms of struc-
ture. The exact nature of the tradeoff space is
explained more in the paper. In our setting, we
define utility in terms of similarity of cache
miss rates. We measure and present cache
miss rates in the article. In Figure 5, we com-
pare the proxy heatmap generated for gcc,
against the ground truth. Our wrapped address
space is of height 2048 (cache lines in the heat-
map) and each column in the heatmap corre-
sponds to 10 000 memory accesses. The figure
illustrates that our approach is able to capture
all but the subtlest of patterns.
It is also worth noting that we further explored
real attacks on the resulting traces. Specifically,
we choose to examine the trace to see if it is possi-
ble to recover an AES key using known attacks.
AES attacks based on cache sets have been well-
studied.8 We present the details of the attack in
our paper. We find that trace wrung proxies
completely stop traditional AES attacks. We per-
form this attack on a set of traces collected from
runs of AES with a random plaintext. We perform
the same attack pre- and post-wringing. Prewring-
ing, the attacker correctly guesses the upper 5
bits of all 16 key-bytes after 1838 encryptions.
This is the maximal information that can be
learned in a first-round attack with 8-byte cache
lines. Postwringing, the attack guesses wrong for
all 16 bytes of the key after 50 000 traces.
May/June 2020
CONCLUSION AND FUTURE IMPACT
Looking further out, the conflict between the
need to share information (to provide more opti-
mal performance) and hide information (for pri-
vacy) is becoming increasingly fundamental in all
of computer science. Threats to personal data pri-
vacy are emerging as a leading concern for users.
While the European GDPR and CCPA put in place
privacy and data protection requirements, the
onus of implementing tools to understand and
embed privacy into systems falls on engineers.
Computer architects must start thinking more
about privacy and provide infrastructure to
enable privacy at all levels of computing. Trace
wringing can be leveraged as one such tool.
We feel the tradeoff space exposed by trace
wringing will open the doors to future work at
the intersection of privacy and computer sys-
tems optimization. We hope to see more follow-
up work that builds on years of community expe-
rience dealing with address traces and to encode
common patterns in a general way. In many
applications, striding memory behavior is an
important component and we believe we are the
first to connect the address trace analysis prob-
lem with the Hough transform. The resulting
analysis is surprisingly robust to noise and can
capture general striding behavior. While this
approach is effective for the memory problems
we examined, there is no shortage of opportu-
nity to build on the techniques we lay out to cre-
ate more robust and higher quality trace
wringing systems. Fully leveraging the best syn-
thetic trace generation,11 trace compression,1,4
quantitative information flow,10 and statistical
modeling12 techniques and understanding what
113
 Top Picks

they each bring to possible, and perhaps more importantly, they

We feel the tradeoff
the problem is one
space exposed by
next step. Bringing trace wringing will open
the full algorithmic the doors to future work
power provided by at the intersection of
the fact that any privacy and computer
public trace data systems optimization.
can be leveraged in We hope to see more
the compression is follow-up work that
also very promising. builds on years of com-
This opportunity is munity experience
dealing with address
particularly inter-
traces and to encode
esting as it sits out-
common patterns in a
side of any past
general way.
lossy compression
or synthetic trace
scheme’s ability to exploit (i.e., minimizing total
data transferred is different than minimizing sen-
sitive data transferred).
In application-tuning and system design, one
can certainly understand how related problems
exist with storage traces, cache coherence traf-
fic, energy usage, user interaction data, and cer-
tainly location data. Clever, yet complex,
techniques have been developed to address cer-
tain anonymity problems in the past, yet the
reality is that they are often dependent on spe-
cific assumptions such as a lack of prior informa-
tion, statistical distributions governing the data,
or that number of queries can be tightly
bounded. Our wringing approach is very direct
and that comes with clarity as to what it does
and does not do. It does not guarantee anything
about how useful the resulting trace will really
be for optimization. However, it does transform
the problem of safe sharing into a measurable
systems problem subject to the myriad tools we
have at our disposal for common-case optimiza-
tion. Furthermore, it does provide a strong and
clear bound on the amount of useful information
given by the trace. For the purposes of privacy
engineering, this is exceedingly valuable. There
is a consensus on the importance of building pri-
vacy into systems that deal with information
about health, legal records and law enforcement,
transportation and location, and other sensitive
information. But, the capability of today’s
tools and methodologies is limited. Trace wring-
ing provides evidence that new methods that
bound information sharing in useful ways are
can then be improved when fed back as a system
requirement.
& REFERENCES
1. M. Burtscher, I. Ganusov, S. J. Jackson, J. Ke,
P. Ratanaworabhan, and N. B. Sam, “The VPC trace-
compression algorithms,” IEEE Trans. Comput.,
vol. 54, no. 11, pp. 1329–1344, Nov. 2005.
2. A. S. Dhodapkar and J. E. Smith, “Comparing
program phase detection techniques,” in Proc. 36th
Annu. IEEE/ACM Int. Symp. Microarchit., 2003,
pp. 217–227.
3. R. O. Duda and P. E. Hart, “Use of the Hough
transformation to detect lines and curves in pictures,”
Commun. ACM, vol. 15, no. 1, pp. 11–15, 1972.
4. E. N. Elnozahy, “Address trace compression through
loop detection and reduction,” in Proc. ACM
SIGMETRICS Perform. Eval. Rev., 1999, vol. 27,
pp. 214–215.
5. C. Galamhos, J. Matas, and J. Kittler, “Progressive
probabilistic Hough transform for line detection,” in
Proc. IEEE Comput. Soc. Conf. Comput. Vis. Pattern
Recognit., 1999, vol. 1, pp. 554–560.
6. J. L. Henning, “SPEC CPU2006 benchmark
descriptions,” ACM SIGARCH Comput. Archit. News,
vol. 34, no. 4, pp. 1–17, 2006.
7. P. Michaud, “Online compression of cache-filtered
address traces,” in Proc. IEEE Int. Symp. Perform.
Anal. Syst. Softw., 2009, pp. 185–194.
8. D. A. Osvik, A. Shamir, and E. Tromer, “Cache attacks
and countermeasures: The case of AES,” in Proc.
Cryptographers Track RSA Conf., Springer, 2006,
pp. 1–20.
9. T. Sherwood, E. Perelman, G. Hamerly, and B. Calder,
“Automatically characterizing large scale program
behavior,” ACM SIGARCH Comput. Archit. News,
vol. 30, no. 5, pp. 45–57, 2002.
10. G. Smith, “On the foundations of quantitative
information flow,” in Proc. Int. Conf. Foundations Softw.
Sci. Comput. Struct., Springer, 2009, pp. 288–302.
11. L. Van Ertvelde and L. Eeckhout, “Dispersing
proprietary applications as benchmarks through code
mutation,” ACM SIGARCH Comput. Archit. News,
ACM, vol. 36, pp. 201–210, 2008.
12. J. Weinberg and A. E. Snavely, “Accurate memory
signatures and synthetic address traces for HPC
applications,” in Proc. 22nd Annu. Int. Conf.
Supercomput., ACM, 2008, pp. 36–45.

IEEE Micro
114
Deeksha Dangwal is currently working toward the
Ph.D. degree in computer architecture with the
Department of Computer Science, University of
California, Santa Barbara. Her research interests
include computer architecture, privacy, and informa-
tion theory. She is a student member of IEEE and
ACM. She is the corresponding author of this article.
Contact her at deeksha@cs.ucsb.edu.
Weilong Cui is currently a Software Engineer with
Google. His research interests include statistical/eco-
nomic-inspired methods and programming languages
for computer architecture performance modeling, as
well as novel micro/system-architecture and its interac-
tion with software. Cui received the master’s and bach-
elor’s degrees in computer science from Peking
University and the Ph.D. degree from the Department
of Computer Science, University of California, Santa
Barbara. Contact him at cuiwl@google.com.
May/June 2020
Joseph McMahan is currently a Research Scientist
with the University of Washington, Seattle. His research
interests include computer architecture, security, formal
methods, and machine learning. McMahan received
the Ph.D. degree from the University of California Santa
Barbara. He is a member of ACM and IEEE. Contact
him at jmcmahan@cs.washington.edu.
Timothy Sherwood is currently a Professor of
computer science and the Associate Vice-Chancellor
for Research with the University of California, Santa
Barbara. He is a cofounder of the hardware security
startup Tortuga Logic and the 2016 ACM SIGARCH
Maurice Wilkes Awardee “for contributions to novel
program analysis advancing architectural modeling
and security.” Sherwood received the B.S. degree in
computer science from UC Davis, and the M.S. and
Ph.D. degrees from UC San Diego. Contact him at
sherwood@cs.ucsb.edu.
115
 !

HOST
W
O
N
TER
IS

2020
G
RE

6–9 Dec. 2020 • San Jose, CA

IEEE INTERNATIONAL SYMPOSIUM

ON HARDWARE-ORIENTED
SECURITY AND TRUST
6–9 Dec. 2020 • San Jose, CA, USA • DoubleTree by Hilton

Join dedicated professionals at the IEEE International Symposium on Hardware Oriented Security
and Trust (HOST) for an in-depth look into hardware-based security research and development.
Key Topics:
• Semiconductor design, test  • Cryptography
and failure analysis and cryptanalysis
• Computer architecture • Imaging and microscopy
• Systems security

Discover innovations from outside your sphere of influence at HOST. Learn about new research
that is critical to your future projects. Meet face-to-face with researchers and experts for
inspiration, solutions, and practical ideas you can put to use immediately.

REGISTER NOW: www.hostsymposium.org

 Department: Micro Economics

Pandemics and the Dismal

Technology Economy
Shane Greenstein
Harvard Business School

& WE SPEAK OF viruses, though not the ones forecasts are challenging to make, but less so in
that infect computers. Believe it or not, this insurance markets due to the abundance of prece-
follows a long tradition in economics. Two cen- dent. It is possible to ground a forecast in histori-
turies ago, T. Malthus first made his grim pre- cal patterns.
dictions about death from resource shortages. Forecasts become more difficult when every-
Economics has been known as the dismal sci- thing is hypothetical. For example, you may
ence ever since. have heard about “stress tests” for banks, which
Grim is the mood of the day. This is a painful were implemented after the financial mess in
moment for many stockholders, managers, and 2008. That “test”—more like an
personnel. Many readers may audit—focuses on whether the bank
not have had to think about In spite of the lack of can survive a rare and hypothetical
this topic since the financial precedent, today’s painful scenario. (Now that we are
panic of 2008, or maybe even economic crisis experiencing an actual scenario, we
the dot-com bust. broadly contains pre- will find out if the hypothetical is
Dismal as this topic is, let dictable features. Here planned appropriately.)
us understand how the pan- is why. The economy is Most of today’s dislocation realizes
one big circular flow of
demic could cause so much scenarios that had been hypothetical
expenditure in which
economic damage, and why it until a few months ago. This time
one person’s purchase
may take so long to recover. markets inherit the economic decline
is another person’s
I will keep it basic and focus sale, and that further from shutting down services in the
on the economics of this goes into somebody’s economy—more in a minute. There is
situation for the technology paycheck from which nothing in recent history like this.
economy. more purchases arise. The situation looks quite different
than it did in 2008 or 2001, where
the disruption in those instances origi-
SHORT-RUN EFFECTS nated with the mortgage lending crisis, and with
Analysts have a special set of models for sce-
the misdirected investments behind the dotcom
narios that arise with low probability and impose
boom.
high expense. You may have come across such
In spite of the lack of precedent, today’s eco-
models when buying life insurance or cata-
nomic crisis broadly contains predictable fea-
strophic medical insurance. These types of
tures. Here is why. The economy is one big
circular flow of expenditure in which one person’s
Digital Object Identifier 10.1109/MM.2020.2984182 purchase is another person’s sale, and that fur-
Date of current version 22 May 2020. ther goes into somebody’s paycheck from which

0272-1732 ß 2020 IEEE Published by the IEEE Computer Society IEEE Micro
118
more purchases arise. In addition, it just keeps
going. Virtually every part of the U.S. economy
has been finely tuned to expect this circular flow.
Until recently this flow was predictable.
The pandemic made the expectations about
flows obsolete. That introduces many disloca-
tions. Here is the chain of causality around which
all hypothetical forecasts revolve. Restaurants,
bars, schools, theaters, and sporting events have
closed in all major locations where people
“shelter in place.” As of this writing, more than a
third of the population lives in areas with such
lockdowns, and it could increase. Related parts
of the economy also have slowed considerably:
Travel on airlines, staying in hotels, vacationing
in beaches, and enjoying amusement parks.
Depending on how you count it, that inter-
rupts somewhere between 10% and 15% of the
expenditure in the US economy, and the jobs of
somewhere between 10% and 20% of the labor
force. In the third week of March over three
million people—more than 2% of the labor
force—filed new claims for unemployment. That
has never before happened in one week.
What happens next? Two additional concepts
help round out the big picture: When everybody
cuts back a little, it adds up and reinforces the
overall movements downward. Once everyone
expects the worst, as we do now, then the low
forecast becomes self-fulfilling. Economists call
these “multiplier” effects, and “self-reinforcing
expectations,” and it makes the whole worse
than the sum of its parts.
By way of analogy, system engineers might
recognize this phenomenon as secondary feed-
back effects. The sum of secondary feedbacks
reinforces the first-order effects, and makes the
system behave at a suboptimal equilibrium.
Worst of all, once it settles (at a high level of
unemployment), it does not move away easily.
The dot-com bust started from a different
place, but events there illustrate how multiplier
effects operate. In that case, the loss of financial
confidence led many of those online firms to can-
cel orders for equipment at the same time—PCs,
office networking, and application software.
That caused liquidity issues at otherwise healthy
and efficient suppliers. No funds came in for new
sales, while inventories of final products piled
up in warehouses, and the bills for last month’s
inputs and workforce needed attention.
May/June 2020
It almost goes without saying, but gloomy
forecasts about illiquidity are the mood of the
day at many firms. Related, this is also part of
the logic behind federal legislation to either
make credit available to business and cash to
households. It alleviates suffering and delays
liquidity crises.
The uncertainty about expenditure explains
some of the “wait and see” remarks from finan-
cial analysts.
WHAT THIS DOES TO FIRMS
Despite the uncertainty, what can we say?
Representative examples can illustrate.
Many firms get their flow of funds directly
from expenditures for leisure and travel. When
that expenditure drops, so too does sales at, for
example, Airbnb, Priceline, Hotel.com, Expedia,
Orbitz, and more. When restaurants receive less
visits, so too does advertising on Open Table
and Yelp and, to some extent, Google.
Related local transportation declines, such as
Uber and Lyft. Close to a third of their trips go to
airports. Additionally, many people are shelter-
ing at home, neither going to the local restaurant
nor far away on a vacation. Furthermore, while
Uber Eats might make more deliveries, it is not
enough to make up for the overall decline.
The next few months are a lousy time to sell a
consumer product in a retail outlet. Sales of
iPhones and smart phones have declined, as
consumers put off an optional purchase and
retail outlets close for health safety. Same fore-
cast for printers, home networks, and tablets.
Every parts’ supplier within those supply chains
should expect a lower sale in the near term. That
hurts sales at Apple, Samsung, Intel, Qualcomm,
and HP, as well as distributors, like CDW, Sta-
ples, and Office Max. Ouch.
The next generation of online games and
entertainment looks like a mixed bag. Online
channels help. Freemium services will tend to do
better than subscription services, if any of them
does well at all. General use will go up, especially
for established brands, such as Electronic Arts.
But not many devices will be sold, such as Xbox
or PlayStation, except through online channels.
What about Facebook and Google? Both have
seen massive increases in traffic from use, which
119
 Micro Economics

means they sell more ads with more people wireless networks, as home networks press into
online. With less shopping overall, however, each capacity utilization far above anticipated levels.
ad is not as valuable. The value of the ad-based This situation puts pressures on users to increase
business could increase or declines overall. wireless data contracts. Wireless firms also
You may say: what about online retailing? should benefit from pressure at households drop-
Amazon should be able to take advantage of ping broadband and going only with wireless
everyone shopping online instead of visiting Internet. Broadband carriers have seen home traf-
malls. And perhaps some of their third-party sup- fic increase, while pressures for cord-cutting
pliers will be fine as well (e.g., if increased (i.e., diminishing televi-
they have hand sanitizer to sell). sion contracts). The broadband
That said, the most profitable divi- Most people want to business also has always had to
sion at Amazon is AWS, and more know: how long will this manage a huge fraction of house-
broadly, nobody expects AWS, last? Forecasting the holds who do not pay their bills on
Azure, or Google Cloud to decline. recovery is especially time, which also should grow worse
The flexibility inherent in that difficult. Both biology over the next few months.
business model gives them advan- and economics plays a Finally, the financial side of
role. So does economic
tages over construction of new technology will go through a some-
policy. Many analysts
data centers or equipment pur- what predictable decline. No VC
want to know: Will any of
chases for business. will have an IPO in the near term.
the behavior exhibited
Streaming services appeal to during the period of Many startups with cash flow
all the people stuck at home. home sheltering persist issues will be sold at fire sale prices
Established services, such as into the future? to large buyers as acquisitions. As
Netflix, Hulu, YouTube, and Ama- happened in 2009, many VCs will
zon Prime video, are positioned tend to cutoff their worst perform-
to do well. This environment also gives an ing firms, but which firms and when? Good luck
opportunity to HBO Go, CBS, Sling, PlayStation with forecasting that.
Vue, and few others.
That said, new launches are always risky, but
that is especially so in this environment. For
UNANSWERED QUESTIONS
example, Quibi—with its hype and subscription Most people want to know: how long will this
price—probably would have been better off
last? Forecasting the recovery is especially diffi-
launching just a few months earlier, just as Dis- cult. Both biology and economics plays a role.
neyPlus did. Will Quibi gain traction in this envi-
So does economic policy.
ronment? Anybody’s guess. Many analysts want to know: Will any of the
Some online communication tools have got-
behavior exhibited during the period of home
ten more use too, such as WhatsApp, Zoom, and sheltering persist into the future?
Slack. Google Hangouts, Skype, and Webex have
As I write this, there is simply no precedent
seen resurgence, and lesser known services, for making an educated guess. There are too
such as Join.me, GoToMeeting, Stride, RingCen-
many unknowable unknowns.
tral, and BigMarker.
The carrier business will see a mix of experien- Shane Greenstein is a Professor at the Harvard
ces. Traffic has gone up for both wireline and Business School. Contact him at sgreenstein@ hbs.edu.

IEEE Micro
120
PURPOSE: The IEEE Computer Society is the world’s largest
association of computing professionals and is the leading provider
of technical information in the field.
MEMBERSHIP: Members receive the monthly magazine
Computer, discounts, and opportunities to serve (all activities
are led by volunteer members). Membership is open to all IEEE
members, affiliate society members, and others interested in the
computer field.
COMPUTER SOCIETY WEBSITE: www.computer.org
OMBUDSMAN: Direct unresolved complaints to
ombudsman@computer.org.
CHAPTERS: Regular and student chapters worldwide provide the
opportunity to interact with colleagues, hear technical experts,
and serve the local professional community.
AVAILABLE INFORMATION: To check membership status, report
an address change, or obtain more information on any of the
following, email Customer Service at help@computer.org or call
+1 714 821 8380 (international) or our toll-free number,
+1 800 272 6657 (US):
• Membership applications
• Publications catalog
• Draft standards and order forms
• Technical committee list
• Technical committee application
• Chapter start-up procedures
• Student scholarship information
• Volunteer leaders/staff directory
• IEEE senior member grade application (requires 10 years
practice and significant performance in five of those 10)
PUBLICATIONS AND ACTIVITIES
Computer: The flagship publication of the IEEE Computer Society,
Computer publishes peer-reviewed technical content that covers
all aspects of computer science, computer engineering,
technology, and applications.
Periodicals: The society publishes 12 magazines‫ژ‬ƌȄư‫ژז׏ژ‬DZȏɓȵȄƌǹȽ.
Refer to membership application or request information as noted
above.
Conference Proceedings & Books: Conference Publishing
Services publishes more than 275 titles every year.
Standards Working Groups: More than 150 groups produce IEEE
standards used throughout the world.
Technical Committees: TCs provide professional interaction in
more than 30 technical areas and directly influence computer
engineering conferences and publications.
Conferences/Education: The society holds about 200 conferences
each year and sponsors many educational activities, including
computing science accreditation.
Certifications: The society offers three software developer
credentials. For more information, visit
www.computer.org/certification.
BOARD OF GOVERNORS MEETING
‫ژגא‬٫‫ דאژ‬°ƷȲɋƷȂƨƷȵ‫ژ׎א׎אژ‬ǠȄ‫ژ‬uƩkƷƌȄً‫ژ‬ÝǠȵǒǠȄǠƌً‫ژ‬Å°

revised ‫ڳת‬uƌɲ‫ש׫ש׫ڳ‬
EXECUTIVE COMMITTEE
President: kƷǠǹƌ‫ژ‬%Ʒ‫ژ‬FǹȏȵǠƌȄǠ
President-Elect: FȏȵȵƷȽɋ‫ژ‬°ǚɓǹǹ
Past President: ƷƩǠǹǠƌ‫ژ‬uƷɋȵƌ
First VP: ¨ǠƩƩƌȵưȏ‫ژ‬uƌȵǠƌȄǠ; Second VP: °ɲ‫ٯ‬äƷȄ‫ژ‬hɓȏ ‫ژژژژژ‬
Secretary: %ǠȂǠɋȵǠȏȽ‫ژ‬°ƷȵȲƌȄȏȽ; Treasurer: %ƌɫǠư‫ژ‬kȏȂƷɋ
VP, MemberȽǚǠȲ & Geographic Activities: Yervant Zorian
VP, Professional & Educational Activities: °ɲ‫ٮ‬äƷȄ‫ژ‬hɓȏ ‫ژژژژژژژژژژژ‬
VP, Publications: Fabrizio Lombardi
VP, Standards Activities: Riccardo Mariani
VP, Technical & Conference Activities: William D. Gropp‫ژ‬
2019–2020 IEEE Division VIII Director: Elizabeth L. Burd‫ژ‬
‫ژ׏א׎אٮ׎א׎א‬U---‫ژ‬%ǠɫǠȽǠȏȄ‫ژ‬Ý‫ژ‬%ǠȵƷƩɋȏȵ‫¾ژي‬ǚȏȂƌȽ‫ژ‬uِ‫ژ‬ȏȄɋƷ‫ژژژژژژژژژژ‬
‫ژ׎א׎א‬U---‫ژ‬%ǠɫǠȽǠȏȄ‫ژ‬ÝUUU‫ژ‬%ǠȵƷƩɋȏȵ‫ٮ‬-ǹƷƩɋ‫ژي‬ǚȵǠȽɋǠȄƌ‫ژ‬uِ‫ژ‬°ƩǚȏƨƷȵ
BOARD OF GOVERNORS
¾ƷȵȂ‫ ژ‬-ɱȲǠȵǠȄǒ‫
ژي׎א׎א ژ‬Ȅưɲ‫ ژِ¾ ژ‬ǚƷȄً‫ ژ‬eȏǚȄ‫ ژ‬%ِ‫ ژ‬eȏǚȄȽȏȄً‫ژ‬
°ɲ‫ٮ‬äƷȄ‫ ژ‬hɓȏً‫ ژ‬%ƌɫǠư‫ ژ‬kȏȂƷɋً‫ ژ‬%ǠȂǠɋȵǠȏȽ‫ ژ‬°ƷȵȲƌȄȏȽً‫ژژژژژ‬
Oƌɲƌɋȏ‫ژ‬äƌȂƌȄƌ
¾ƷȵȂ‫ ژ‬-ɱȲǠȵǠȄǒ‫ ژي׏א׎א ژ‬uِ‫ ژ‬ȵǠƌȄ‫ ژ‬ǹƌǵƷً‫ ژ‬FȵƷư‫ ژ‬%ȏɓǒǹǠȽً‫ژ‬
ƌȵǹȏȽ‫ ژ‬-ِ‫ ژ‬eǠȂƷȄƷɼ‫ٮ‬GȏȂƷɼً‫¨ ژ‬ƌȂƌǹƌɋǚƌ‫ ژ‬uƌȵǠȂɓɋǚɓً‫ژژژژژژژژژژژ‬
-ȵǠǵ‫ ژ‬eƌȄ‫ ژ‬uƌȵǠȄǠȽȽƷȄً‫ ژ‬hɓȄǠȏ‫ ژ‬ÅƩǚǠɲƌȂƌ
¾ƷȵȂ‫ ژ‬-ɱȲǠȵǠȄǒ‫ ژيאא׎א ژ‬wǠǹȽ‫
ژ‬ȽƩǚƷȄƨȵɓƩǵً‫ژ‬
-ȵȄƷȽɋȏ‫ ژ‬ɓƌưȵȏȽ‫ٯ‬ÝƌȵǒƌȽً‫ ژ‬%ƌɫǠư‫ ژ‬°ِ‫ ژ‬-ƨƷȵɋً‫ ژ‬ÞǠǹǹǠƌȂ‫ ژ‬GȵȏȲȲً‫ژ‬
GȵƌƩƷ‫ ژ‬kƷɬǠȽً‫ ژ‬°ɋƷǑƌȄȏ‫ژ‬îƌȄƷȵȏ
EXECUTIVE STAFF
Executive Director: Melissa
ِ‫ژ‬Russell
Director, Governance & Associate Executive Director:
Anne Marie Kelly
Director, Finance & Accounting: Sunny Hwang
Director, Information Technology & Services: Sumit Kacker
Director, Marketing & Sales: Michelle Tubb
Director, Membership Development: Eric Berkowitz
COMPUTER SOCIETY OFFICES
Washington, D.C.: 2001 L St., Ste. 700, Washington, D.C.
20036-4928ٕ Phone: +1 202 371 0101ٕ Fax: +1 202 728 9614ٕ‫ژ‬
Email: ǚƷǹȲ‫ۮ‬ƩȏȂȲɓɋƷȵِȏȵǒ
Los Alamitos: 10662 Los Vaqueros Cir., Los Alamitos, CA 90720ٕ‫ژ‬
Phone: +1 714 821 8380ٕ Email: help@computer.org
u-u-¨°OU¥‫ژۯژ‬¥ÅkU
¾U‚w‫¨‚ژ‬%-¨°‫ژ‬
¥ǚȏȄƷ‫ژٕבבבגژזוהژ׎׎זژ׏ڹژي‬Fƌɱ‫ژٕ׏גהגژ׏אזژג׏וژ׏ڹژي‬
-ȂƌǠǹ‫ژي‬ǚƷǹȲ‫ۮ‬ƩȏȂȲɓɋƷȵِȏȵǒ
IEEE BOARD OF DIRECTORS
President: ¾ȏȽǚǠȏ‫ژ‬Fɓǵɓưƌ
President-Elect: °ɓȽƌȄ‫ژ‬hِ‫ٹژ‬hƌɋǚɲ‫ژٺ‬kƌȄư
Past President: eȏȽƸ‫ژ‬uِFِ‫ژ‬uȏɓȵƌ
Secretary: Kathleen
ِ‫ژ‬Kramer
Treasurer: Joseph V. Lillie
Director & President, IEEE-USA: eǠȂ‫ژ‬ȏȄȵƌư‫ژ‬
Director & President, Standards Association: Robert S. Fish‫ژ‬
Director & VP, Educational Activities: °ɋƷȲǚƷȄ‫ژ‬¥ǚǠǹǹǠȲȽ‫ژ‬
Director & VP, Membership ‫ ۯ‬Geographic Activities:‫ژژژژژژژ‬
hɓǵDZǠȄ‫ژ‬ǚɓȄ
Director & VP, Publication Services & Products: ¾ƌȲƌȄ‫ژ‬°ƌȵǵƌȵ‫ژ‬
Director & VP, Technical Activities: hƌɼɓǚǠȵȏ‫ژ‬hȏȽɓǒƷ