VOLUME 41, NUMBER 3 MAY/JUNE 2021

Top Picks From the 2020 Computer

Architecture Conferences

www.computer.org/micro
 mpute
r.org ²ƧǞƵȁ
ɈǞ
Comp ˛Ƨ
uting
nted Data
and Visual
eality Inform izatio
n
on
ce4c1.
indd Techn ation
1
ology
on Quan
tum
y Comp
uting
putin
g

Intern
et of
Ethic Thing
s s
Mach
ine L
Quan ea rning
tu
Comp m
uting
JUNE
2020

JULY 20
ww w.c 20
ompu
ter.org

ww w.c
ompu
ter.org

ce7c1.ind
d 1

Secu
rity an
Priva d
cy High-P
Auto
matio
n Comp erforman
Block uting ce
chain Hard
Digit ware
al Affect
Transf iv
ormat Comp e
ion uting
Educa
tion

MAY 20
20

w w w.c
ompu
ter.org

ce5c1.
indd
1

S
ww w.c NOVE
ompu MBER
ter.org 2020

ww w.c
ompu
ter.org

ce11c1.in
dd 1

ndd 1

ComputingEdge
Secu
riit
Priva y and
cy
Data
Intern
et
ȲɈǞ˛
ƧǞƊǶ
XȁɈƵǶǶ
ǞǐƵȁƧ
Ƶ
Your one-stop resource for industry hot topics,
technical overviews, and in-depth articles.

Cutting-edge Unique original Keeps you up to

articles from the FRQWHQWbE\ date on what you
IEEE Computer computing need to know across
Society’s portfolio thought leaders, the technology
of 12 magazines. innovators, spectrum.
DQGbH[SHUWV

Subscribe for free

www.computer.org/computingedge
EDITOR-IN-CHIEF IEEE MICRO STAFF
Journals Production Manager: Joanna Gojlik,
Lizy K. John, University of Texas, Austin, USA
j.gojlik@ieee.org
ASSOCIATE EDITOR-IN-CHIEF Cover Design: Janet Dudar
Peer Review Administrator: micro-ma@computer.org
Vijaykrishnan Narayanan, The Pennsylvania State Publications Staff Editor: Cathy Martin
University, USA Publications Operations Project Specialist: Christine
Anthony
EDITORIAL BOARD Content Quality Assurance Manager: Jennifer
Guido Araujo, University of Campinas, Brazil Carruth
Publications Portfolio Manager: Kimberly Sperka
R. Iris Bahar, Brown University, USA
Publisher: Robin Baldwin
Christopher Batten, Cornell University, USA
Executive Director: Melissa Russell
Mauricio Breternitz, University of Lisbon, Portugal
Senior Advertising Coordinator: Debbie Sims
Bronis de Supinski, Lawrence Livermore
National Laboratory, USA IEEE PUBLISHING OPERATIONS
Yasuko Eckert, AMD Research, USA Senior Director, Publishing Operations: Dawn Melley
Maya Gokhale, Lawrence Livermore National Director, Editorial Services: Kevin Lisankie
Laboratory, USA Director, Production Services: Peter M. Tuohy
Shane Greenstein, Harvard Business School, Associate Director, Editorial Services: Jeffrey E. Cichocki
USA Associate Director, Information Conversion
Russ Joseph, Northwestern University, USA and Editorial Support: Neelam Khinvasara
Hyesoon Kim, Georgia Institute of Technology, USA Senior Manager, Journals Production: Patrick Kempf
John Kim, Korea Advanced Institute of Science and
Technology, South Korea CS MAGAZINE OPERATIONS COMMITTEE
Hsien-Hsin (Sean) Lee, Taiwan Semiconductor Diomidis Spinellis (Chair), Lorena Barba,
Manufacturing Company, USA Irena Bojanova, Shu-Ching Chen, Gerardo Con Diaz,
Michael Mattioli, Goldman Sachs & Co., USA Lizy K. John, Marc Langheinrich, Torsten Möller,
Tulika Mitra, National University of Singapore, Ipek Ozkaya, George Pallis, Sean Peisert,
Singapore VS Subrahmanian, Jeff Voas
Sreenivas Subramoney, Intel Corporation, India
CS PUBLICATIONS BOARD
Carole-Jean Wu, Arizona State University, USA
Lixin Zhang, Chinese Academy of Sciences, M. Brian Blake (VP for Publications),
China David Ebert, Elena Ferrari, Hui Lei, Timothy Pinkston,
Antonio Rubio Sola, Diomidis Spinellis, Tao Xie,
ADVISORY BOARD Ex officio: Robin Baldwin, Sarah Malik, Melissa Russell,
Forrest Shull
David H. Albonesi, Cornell University, USA
Erik R. Altman, IBM, USA COMPUTER SOCIETY OFFICE
Pradip Bose, IBM T.J. Watson Research Center, USA IEEE MICRO
Kemal Ebcioglu, Global Supercomputing c/o IEEE Computer Society
Corporation, USA 10662 Los Vaqueros Circle
Lieven Eeckhout, ELIS – Ghent University, Belgium Los Alamitos, CA 90720 USA; +1 (714) 821-8380
Ruby B. Lee, Princeton University, USA
Trevor Mudge, University of Michigan, Ann Arbor, Subscription change of address:
USA address.change@ieee.org
Yale Patt, University of Texas at Austin, USA Missing or damaged copies:
James E. Smith, University of Wisconsin–Madison, USA help@computer.org

IEEE Micro (ISSN 0272-1732) is published bimonthly by the IEEE Computer Society. IEEE Headquarters: Three Park Ave., 17th Floor, New York,
NY 10016; IEEE Computer Society Headquarters: 2001 L St., Ste. 700, Washington, DC 20036; IEEE Computer Society Publications Office: 10662
Los Vaqueros Circle, PO Box 3014, Los Alamitos, CA 90720. Postmaster: Send address changes and undelivered copies to IEEE, Membership
Processing Dept., 445 Hoes Ln., Piscataway, NJ 08855. Periodicals postage is paid at New York, NY, and at additional mailing offices. Canadian
GST #125634188. Canada Post Corp. (Canadian distribution) Publications Mail Agreement #40013885. Return undeliverable Canadian addresses to
4960-2 Walker Road; Windsor, ON N9A 6J3. Printed in USA. Reuse rights and reprint permissions: Educational or personal use of this material is
permitted without fee, provided such use: 1) is not made for profit; 2) includes this notice and a full citation to the original work on the first page of the
copy; and 3) does not imply IEEE endorsement of any third-party products or services. Author and their companies are permitted to post the accepted
version of IEEE-copyrighted material on their own webservers without permission, provided that the IEEE copyright notice and a full citation to the
original work appear on the first screen of the posted copy. An accepted manuscript is a version which has been revised by the author to incorporate
review suggestions, but not the published version with copy-editing, proofreading, and formatting added by IEEE. For more information, please go to
ieee.org/publications_standards/publications/rights/paperversionpolicy.html. Permission to reprint/republish this material for commercial, advertising,
or promotional purposes or for creating new collective works for resale or redistribution must be obtained from IEEE by writing to the IEEE Intellectual
Property Rights Office, 445 Hoes Lane, Piscataway, NJ 08854 or pubs-permissions@ieee.org. ©2021 by IEEE. All rights reserved. Abstracting and
library use: Abstracting is permitted with credit to the source. Libraries are permitted to photocopy for private use of patrons, provided the per-copy
fee indicated in the code at the bottom of the first page is paid through the Copyright Clearance Center, 222 Rosewood Drive, Danvers, MA 01923.
Editorial: Unless otherwise stated, bylined articles, as well as product and service descriptions, reflect the author’s or firm’s opinion. Inclusion in IEEE
Micro does not necessarily constitute an endorsement by IEEE or the Computer Society. All submissions are subject to editing for style, clarity, and
space. IEEE prohibits discrimination, harassment, and bullying. For more information, visit ieee.org/web/aboutus/whatis/policies/p9-26.html.
 6
GUEST EDITOR’S INTRODUCTION
Top Picks From the 2020 Computer
Architecture Conferences
Daniel A. Jiménez

MAY/JUNE 2021

10 19 27
Theme Articles

The Vision Behind MLPerf: Superconductor Leaking Secrets

Understanding AI Inference Computing for
Performance Neural Networks
Vijay Janapa Reddi, Christine Cheng, Koki Ishida , Ilkwon Byun,
David Kanter, Peter Mattson, Ikki Nagaoka, Kosuke Fukumitsu,
Guenther Schmuelling, and Carole-Jean Wu Masamitsu Tanaka,
Satoshi Kawakami,
Teruo Tanimoto, Takatsugu Ono,
Jangwoo Kim, and Koji Inoue
Through Compressed
Caches
Po-An Tsai, Andres Sanchez,
Christopher W. Fletcher, and
Daniel Sanchez
 Theme Articles Continued
 34  Understanding Acceleration Opportunities at Hyperscale
Akshitha Sriraman and Abhishek Dhanotia

 42 Accelerating Genomic Data Analytics With Composable

Hardware Acceleration Framework
Tae Jun Ham, David Bruns-Smith, Brendan Sweeney, Yejin Lee, Seong Hoon Seo,
U Gyeong Song, Young H. Oh, Krste Asanovic, Jae W. Lee, and Lisa Wu Wills

50  uGEMM: Unary Computing for GEMM Applications

Di Wu, Jingjie Li, Ruokai Yin, Younghyun Kim, Joshua San Miguel, and Hsuan Hsiao

57  BabelFish: Fusing Address Translations for Containers

Dimitrios Skarlatos, Umur Darbaz, Bhargava Gopireddy, Nam Sung Kim,
and Josep Torrellas

63  Characterizing and Modeling Nonvolatile Memory Systems

Zixuan Wang, Xiao Liu, Jian Yang, Theodore Michailidis, Steven Swanson,
and Jishen Zhao

71  Temporal Computing With Superconductors

Georgios Tzimpragos, Jennifer Volk, Dilip Vasudevan, Nestan Tsiskaridze,
George Michelogiannakis, Advait Madhavan, John Shalf, and Timothy Sherwood

80  A Next-Generation Cryogenic Processor Architecture

Ilkwon Byun, Dongmoon Min, Gyuhyeon Lee, Seongmin Na, and Jangwoo Kim

87  Balancing Specialized Versus Flexible Computation in Brain–

Computer Interfaces
Ioannis Karageorgos, Karthik Sriram, Ján Veselý, Nick Lindsay, Xiayuan Wen, Michael
Wu, Marc Powell, David Borton, Rajit Manohar and
Abhishek Bhattacharjee

95  Virtual Logical Qubits: A Compact Architecture for Fault-Tolerant

Quantum Computing
Jonathan M. Baker, Casey Duckering, David I. Schuster, and Frederic T. Chong

Columns and Departments

From the Editor-in-Chief
4  Top Picks From Year 2020
Lizy Kurian John
Security
Image credit:
Image licensed
103 The Next Security Frontier: Taking the Mystery Out of the Supply Chain
by Ingram Michael Mattioli, Tom Garrison, and Baiju V. Patel
Publishing.
Micro Economics
110   R emote Work
Shane Greenstein

Also in this Issue

1   Masthead
108  IEEE Computer Society Info

www.computer.org/micro

ISSN: 0272-1732
 FROM THE EDITOR-IN-CHIEF
Top Picks From Year 2020
Lizy Kurian John, The University of Texas at Austin, TX, 78712, USA
W
e have lived more than one year in the
special circumstances created by COVID-
19. Microchips and the systems powered
by them have helped the world to survive this chal-
lenging year. At the same time, the computer architec-
ture conferences saw many interesting articles on
various aspects of computer system design. It is my
pleasure to present to you the IEEE Micro Top Picks
issue, with the very best articles from 2020.
For more than a decade, IEEE Micro has had this
tradition of evaluating articles from the previous year’s
computer architecture or computer systems confer-
ences and selecting the articles with the most novelty
and potential for long-term impact. The magazine is
upholding the tradition this year as well. Any article
published in top architecture or systems conferences
during 2020 was eligible to compete for the Top Picks
honor. In total, 120 submissions were received, from
which 12 articles were chosen to represent the cream
of the crop of 2020.
Professor Daniel Jimenez of Texas A&M University
chaired this year’s selection committee. Jimenez and
42 experts from academia and industry worked hard
to identify 12 Top Picks and 12 honorable mention
articles. An article recognized as a Top Pick was
invited to prepare a submission for inclusion in this
Special Issue. The articles in this Special Issue are
intended to be for a broader audience than the origi-
nal conference articles. These articles also focus more
on their potential impact. The honorable mentions are
high-quality articles that unfortunately could not be
included in this Special Issue due to space constraints.
They are listed in the guest editor’s introduction. Inter-
ested readers can locate them in the original confer-
ence proceedings or the IEEE/ACM Digital Library.
The purpose of the Top Picks issue has been multi-
fold. First and foremost, Top Picks was originally insti-
tuted to present the “best of the best” of the
preceding year’s architecture research contributions
to a broader audience including industry and other
0272-1732 ß 2021 IEEE
Digital Object Identifier 10.1109/MM.2021.3074759
Date of current version 25 May 2021.
4 IEEE Micro Published by the IEEE Computer Society
fields. A second goal of Top Picks is to recognize
excellent research in the field and bestow this honor
on researchers who conducted the outstanding
research that resulted in these articles. It is critically
important for our field to honor our budding research-
ers and help them to shape their careers. The Top
Picks honor has been seen to be instrumental in
achieving faculty positions, leading research positions
in industry, and prestigious research grants. Third, the
writing style intended for a broader audience makes it
easy for beginner graduate students to understand
the state of the art of the field as they are pondering
on topics to work for their PhDs. Above all, I expect
these articles to be enjoyable reads for all the readers
of IEEE Micro.
FOR MORE THAN A DECADE, IEEE
MICRO HAS HAD THIS TRADITION OF
EVALUATING ARTICLES FROM THE
PREVIOUS YEAR’S COMPUTER
ARCHITECTURE OR COMPUTER
SYSTEMS CONFERENCES AND
SELECTING THE ARTICLES WITH THE
MOST NOVELTY AND POTENTIAL FOR
LONG-TERM IMPACT. THE MAGAZINE
IS UPHOLDING THE TRADITION THIS
YEAR AS WELL.
I take this opportunity to express my gratitude to
Prof. Jimenez and the selection committee members,
who spent countless hours during the Christmas and
New Year season to evaluate the submissions, and
conducted a face-to-face meeting and deliberated a
whole day to perform this important selection. Prof.
Jimenez and the committee conducted a multistep
selection process that tried to reduce the impact of
the discussion order of the articles. I wish to express
my special thanks to them for the thoughtful process
and the hard work.
The Top Picks articles belong to the following five
themes: 1) machine learning benchmarking; 2) secu-
rity; 3) accelerators; 4) address translation; and 5) new
May/June 2021

technologies. One may recognize that these are highly
relevant topics for advancing the state of the art of
computer architecture. A comprehensive article writ-
ten by Daniel Jimenez serves as an excellent introduc-
tion to the compendium.
I want to personally congratulate all the Top Picks
authors for their fantastic work. It is a significant
honor to rise to the top in a competition where each
candidate article is already recognized as an excellent
piece of work. I hope that these works have significant
impact on future computer systems.
In addition to the Top Picks articles, this issue also
features a Micro Security article on supply chain secu-
rity from Mattioli et al., and a Micro Economics column
by Shane Greenstein of Harvard Business School,
titled “Remote Work.”
YEAR 2021 IS A SPECIAL YEAR. IT IS THE
50TH ANNIVERSARY OF THE
MICROPROCESSOR. IEEE MICRO WILL
BE PUBLISHING A SPECIAL ISSUE ON
THIS THEME IN NOVEMBER/
DECEMBER 2021.
From the Micro Security department, we present
an interesting article titled “The next security frontier:
Taking the mystery out of the supply chain,” coau-
thored by Mattioli, Garrison, and Patel. When a prod-
uct goes through several warehouses and gets
handled by many before finally being received or used
by the end client, there are many risks and vulnerabil-
ities regardless of whether the end product has been
altered from the intended one either unintentionally
or maliciously. This is true for all products, but there
are increased risks in computer system components
and systems. The authors outline some solutions,
including a centralized database or ledger, or a block-
chain-based approach, but participation will be
required from foundries, device manufacturers, and
May/June 2021
FROM THE EDITOR-IN-CHIEF
equipment manufacturers. Read this article to get an
overview of the challenges and potential paths for a
solution.
The Micro Economics column by Shane Greenstein
of Harvard Business School, titled “Remote work,” dis-
cusses how exercise during the pandemic has become
regular operation. Read this article to see how the
workplace is likely to shift even after the pandemic
and how the distinction between the workplace and
the home place is going to blur. Many thoughts on the
likely economic and societal changes are outlined in
this article.
Year 2021 is a special year. It is the 50th anniversary
of the microprocessor. IEEE Micro will be publishing a
special issue on this theme in November/December
2021. Other upcoming special issues are on FPGA com-
puting, quantum computing, commercial products, in-
memory computing, and smart agriculture.
I hope this Special Issue is thought provoking for
our readers and will help shape the field for many
years to come. I also hope that researchers in the field
intensify their efforts to design better chips and sys-
tems to help medical research and ordinary daily lives.
Additionally, I encourage readers to submit to IEEE
Micro. The magazine is interested in submissions on
any aspect of chip or system design or architecture.
Enjoy the award-winning articles presented in this
issue. All the 12 Top Picks articles can be thought of
as award-winning articles. When IEEE Micro selects its
Annual Best Paper Award at the end of each year, the
Top Picks articles are not eligible to compete for the
Best Paper Award, because they are already catego-
rized as prestigious. Hope our industry and academic
readers enjoy these articles.
Happy reading!
LIZY KURIAN JOHN is a Cullen Trust for Higher Education
Endowed Professor with the Electrical and Computer Engi-
neering Department, University of Texas at Austin. Contact
her at ljohn@ece.utexas.edu.
IEEE Micro 5
 GUEST EDITOR'S INTRODUCTION

Top Picks From the 2020 Computer

Architecture Conferences
nez, Texas A&M University, College Station, TX, 77843, USA
Daniel A. Jime

Every year, IEEE Micro selects some of the most notable articles published in the
previous year’s computer architecture conferences to highlight in this Special Issue. In
early 2021, the selection committee chose 12 articles to appear as “Top Picks” and an
additional 12 deserving an “Honorable Mention” from among the 2020 conferences.

I
t is my pleasure to introduce to you IEEE Micro’s
Top Picks from the 2020 computer architecture
conferences. Every year, IEEE Micro publishes a
special issue with the previous year’s Top Picks as
chosen by a selection committee. It was my privilege
this year to be chosen as the selection committee
chair and special issue guest editor. This year’s selec-
tions showcase a great diversity of research topics
making an impact in our community. From traditional
microarchitecture, to machine learning, nonvolatile
memory management, security, and quantum comput-
ing, our community has been very busy redefining and
reinventing computer architecture research. IEEE
Micro imposes a hard limit of 12 articles that can be
chosen as Top Picks. It was very hard to summarize all
the great research of 2020 in just a dozen articles but
the selection committee did an excellent job. The
committee chose another 12 articles as “Honorable
Mentions” described in the “Honorable Mentions”
sidebar. I encourage you to read each of the Top Picks
and Honorable Mentions.
REVIEW PROCESS
The review process for Top Picks was similar to previous
years with the exception that the selection committee
meeting was held remotely for the first time. I chose a
somewhat larger committee than previous years to
reduce individual workload, take advantage of more spe-
cialized expertise, and select some recently graduated
rising stars to give them an insight into the process. I
invited half of the previous year’s committee as well as
the program chairs of 2020’s top architecture conferen-
ces and recent Top Picks selection committee chairs.
0272-1732 ß 2021 IEEE
Digital Object Identifier 10.1109/MM.2021.3074795
Date of current version 25 May 2021.
I also invited a diverse set of other researchers who regu-
larly publish in our top conferences. The final selection
committee had 42 people.
THIS YEAR’S SELECTIONS SHOWCASE A
GREAT DIVERSITY OF RESEARCH
TOPICS MAKING AN IMPACT IN OUR
COMMUNITY. FROM TRADITIONAL
MICROARCHITECTURE, TO MACHINE
LEARNING, NONVOLATILE MEMORY
MANAGEMENT, SECURITY, AND
QUANTUM COMPUTING, OUR
COMMUNITY HAS BEEN VERY BUSY
REDEFINING AND REINVENTING
COMPUTER ARCHITECTURE
RESEARCH.
Authors were asked to prepare three-page descrip-
tions of their articles, including a citation for a hypo-
thetical “test of time” award their article could win in
ten years. Authors submitted these three-pagers
along with the original conference article. The intent
of Top Picks is to choose noteworthy articles from the
top computer architecture conferences, but I allowed
a rather broad interpretation as our community has
grown significantly. For example, this year one of the
Honorable Mentions is from OSDI and another is from
the IEEE Symposium on Security and Privacy.
The review process took place in two rounds. In the
first round, each article was assigned four reviews. After
the reviews were finished, each article was assigned a dis-
cussion lead to guide an online discussion during which
each group of reviewers decided whether the article
should proceed to the second round. Out of 120 submis-
sions, 47 went on to the second round. These articles

6 IEEE Micro Published by the IEEE Computer Society May/June 2021


were assigned an additional five reviews, so that every
article was assigned nine reviews. The vast majority of
these articles actually received nine reviews; three
received only eight due to unforeseen reviewer circum-
stances. After the second round of reviewers, there was
another online discussion period led again by discussion
leads. This time, articles were classified into categories to
help decide which articles should be discussed. During
the selection committee meeting, articles were discussed
in an order determined by an aggregate of merit scores
normalized by an algorithm that takes into account
reviewer bias. At the beginning of each discussion, those
with conflicts were moved to a Zoom break-out room
and were not able to hear or see the discussion of the
conflicted article. At the end of each discussion, a deci-
sion was made to make the article a “Top Pick,” “Honor-
able Mention,” or “Reject.” After the decision was
recorded and the shared screen was cleared of informa-
tion from the discussion, conflicts were allowed to re-
enter the main Zoom room. According to guidelines intro-
duced at the beginning of the meeting, a procedure was
in place to upgrade Honorable Mentions or Rejects if
there were fewer than 12 Top Picks or Honorable Men-
tions at the end of the discussion. I was advised by the
IEEE Micro Editor-In-Chief that 12 is the absolute limit for
Top Picks and another 12 for Honorable Mentions, and I
wanted to award as many of these distinguished awards
as possible to highlight the great research going on in the
architecture community. I am gratified but not surprised
that we filled all the slots for 12 Top Picks and an addi-
tional 12 Honorable Mentions.
SELECTED ARTICLES
This issue has an eclectic set of articles reflecting our
diverse community.
Machine Learning
Interest in machine learning has blossomed in recent
years, and architecture is on the forefront of this new
revolution. One article from a large team spanning
many companies including Google, Microsoft, Face-
book, Intel, NVIDIA, ARM, and others introduces what
has already become an industry-standard benchmark
inference suite: MLPerf, addressing the need for an
industry-wide, fair, and reproducible machine learning
benchmarking methodology. Another article from
Kyushu University, Seoul University, and Nagoya Uni-
versity proposes an optimized neural processing unit
made with single-flux quantum technology.
Security
As industry explores more complex processor designs,
inadvertent exposure of security risks remains a top
May/June 2021
GUEST EDITOR'S INTRODUCTION
concern. One article from researchers at Illinois, MIT,
EPFL, and NVIDIA gives techniques for attacking the
security of compressed caches, showing that com-
pressed caches are vulnerable to side-channel attacks.
Accelerators
One article shows the potential for accelerators to
improve performance in datacenters. This study from
Facebook and the University of Michigan character-
izes workloads at Facebook in terms of the time spent
in portions that could be offloaded to accelerators,
and then projects the performance improvement
achievable by acceleration. Unlike previous work, the
study is done in the context of increasingly common
microservices. Another article from Seoul National
University, Berkeley, Duke, and SungKyunKwan Uni-
versity looks at acceleration in the genomics domain.
It presents a framework for flexible genomic data
processing using SQL-like queries as a domain-spe-
cific language targeting FPGAs. Another fascinating
article from the University of Wisconsin and the Uni-
versity of Toronto provides a novel microarchitecture
for unary computing focused on matrix multiply for
power-constrained environments, showing a very dif-
ferent way to compute with simple circuits.
Address Translation
One article from a team at Illinois and NVIDIA gives a
new more efficient method for address translation for
containerized workloads by deduplicating page table
entries within translation-lookaside buffers and smar-
ter page table sharing techniques.
New Technologies
Several articles describe architectural aspects of new
and emerging technologies. One article from the Uni-
versity of California, San Diego, characterizes the first
widely available nonvolatile memory technology, com-
pares its behavior against previous less accurate mod-
els used in academia, and proposes modifications to
make those models more accurate. Another article
from UC Santa Barbara, Lawrence Berkeley National
Labs, NIST, and the University of Maryland shows how
to design logic and processors using superconducting
substrates, where pulses rather than voltage levels
are the fundamental physical phenomenon that carry
information. Another article from Seoul National Uni-
versity examines the design of cryogenically cooled
processors, looking at the performance benefits and
power tradeoffs and how to design a processor to bet-
ter leverage a cryogenic environment. Another article
from Yale, Rutgers, and Brown describes adapting a
traditional architecture to a new extreme domain:
IEEE Micro 7
GUEST EDITOR'S INTRODUCTION
HONORABLE MENTIONS
Article
“NeuMMU: Architectural Support for Efficient
Address Translations in Neural Processing Units”
by Hyun et al. (ASPLOS 2020).
“Hermes: A Fast, Fault-Tolerant and Linearizable
Replication Protocol” by Katsarakis et al. (ASPLOS
2020)
“DSAGEN: Synthesizing Programmable Spatial
Accelerators” by Weng et al. (ISCA 2020)
“Printed Machine Learning Classifiers” by Husnain
Mubarik et al. (MICRO 2020).
“AGAMOTTO: How Persistent is your Persistent
Memory Application?" by Neal et al. (OSDI 2020)
“Graphene: Strong yet Lightweight Row Hammer
Protection” by Park (MICRO 2020)
“Systematic Crosstalk Migitation for
Superconducting Qubits via Frequency-Aware
Compilation” by Ding et al. (MICRO 2020)
“TRRespass: Exploiting the Many Sides of Target
Row Refresh” by Frigo et al. (SP (Oakland) 2020)
“SQUARE: Strategic Quantum Ancilla Reuse for
Modular Quantum Programs via Cost-Effective
Uncomputation” by Ding et al. (ISCA 2020)
“Leaking Information Through Cache LRU States”
by Xiong and Szefer (HPCA 2020)
“Elastic Cuckoo Page Tables: Rethinking Virtual
Memory Translation for Parallelism” by Skarlatos
et al. (ASPLOS 2020)
“The Architectural Implications of Facebook’s
DNN-based Personalized Recommendation” by
Gupta et al. (HPCA 2020)
8 IEEE Micro
Summary
This article explores the design of memory
management units for emerging neural network
accelerators.
Combining insights from cache coherence and
consensus protocols, Hermes is a fault-tolerant
replication protocol that achieves high
performance and strong consistency.
This article addresses the cost of developing
specialized accelerators, proposing a framework
for programmable accelerator synthesis for a
decoupled spatial architecture.
Inkjet printed circuits are an interesting new
technology. This article gives a deep dive into in a
synergistic application domain: classifiers.
AGAMOTTO is a persistent memory debugging
framework that uses symbolic execution to test
persistent memory systems.
This article introduces a technique to prevent
DRAM Rowhammer attacks based on detecting
frequently accessed elements. The idea features
low overhead and strong security guarantees.
This article mitigates crosstalk in quantum
computing systems with a hardware–software
solution that tunes the frequency of each qubit.
This article gives a comprehensive study of Target
Row Refresh, a DRAM Rowhammer mitigation
technique, and shows that new DRAM chips
continue to be vulnerable to Rowhammer attacks.
This article shows it is possible to use ancilla (i.e.,
scratch) qubits by doing uncomputation to set the
ancilla back to the original state, allowing longer
programs to be run on a quantum computer.
This article shows how to use LRU replacement
state to create a cache covert channel between
agents sharing a cache. It works for a variety of
scenarios and pseudo-LRU designs.
This article gives a novel page table design based
on cuckoo hashing. Sequentially walked radix
tables are replaced by a set of cuckoo tables
searched in parallel. The scheme allows page
tables to scale efficiently.
This article gives a characterization of Facebook
servers running synthetic recommendation
models. It introduces open source
recommendation models representing a large
class of services important in modern datacenters.
May/June 2021

brain–computer interfaces. It gives optimizations to
make power-efficient hardware by designing a chip for
this unique application domain. Finally, an article from
the University of Chicago proposes a 2.5D virtualized
architecture for enabling the efficient implementation
of surface codes, which are error correcting codes
that enable fault-tolerant quantum computing.
THANKS
I would like to give special thanks to Elba Garza and
Gino Chacon, doctoral students at Texas A&M Uni-
versity, who were tremendously helpful running the
HotCRP and Zoom software during the selection
committee meeting. I would also like to thank Emery
Berger, ASPLOS 2021 co-PC-chair who shared with
me scripts for parsing HotCRP output to provide
files for managing conflicts of interest in Zoom;
Boris Grot and Paul Gratz who acted as selection
chair for papers with which I was conflicted; Kathryn
McKinley for helpful suggestions based on previous
SELECTION COMMITTEE
› Nael Abu-Ghazaleh, University of California
Riverside
› Abhishek Bhattacharjee, Yale University
› Marc Casas, Barcelona Supercomputing
Center
› Fred Chong, University of Chicago
› Reetuparna Das, University of Michigan
› Christina Delimitrou, Cornell University
› Ronald Dreslinski, University of Michigan
› Lieven Eeckhout, Ghent University
› Hadi Esmaeilzadeh, University of California,
San Diego
› Michael Ferdman, Stony Brook University
› Christopher Fletcher, University of Illinois-
Urbana Champaign
› Paul Gratz, Texas A&M University
› Boris Grot, University of Edinburgh
› Engin Ipek, Qualcomm
› Akanksha Jain, Arm Research
› Aamer Jaleel, NVIDIA Research
› Jose Joao, Arm Research
› Samira Khan, University of Virginia
› Martha Kim, Columbia University
› Hyesoon Kim, Georgia Tech
› John Kim, KAIST
› Gabriel Loh, Advanced Micro Devices
May/June 2021
GUEST EDITOR'S INTRODUCTION
experiences with Zoom PC meetings that made our
meeting more enjoyable; Lizy Kurian John, IEEE
Micro EIC, for asking me to be the selection chair
and guest editor, and for her helpful advice and
guidance throughout this process; and the selection
committee whose hard work over the Christmas
and New Year’s holidays made this selection pro-
cess possible. Most of all, I would like to thank the
authors of the 120 submissions, whose remarkable
contributions to research continue to make com-
puter architecture an adventure for all of us.
DANIEL A. JIMÉNEZ is a Professor in the Department of Com-
puter Science & Engineering at Texas A&M University. His
research is in microarchitecture, including microarchitectural
prediction and cache management. He invented perceptron-
based branch prediction which is used in millions of process-
ors. He is an NSF CAREER award winner, and a member of
the MICRO, HPCA, and (as of June 2021) ISCA Halls of Fame.
Contact him at djimenez@tamu.edu.
› Kathryn S McKinley, Google
› Onur Mutlu, ETH Zurich
› Prashant Nair, University of British Columbia
› Gilles A. Pokam, Intel
› Moinuddin Qureshi, Georgia Tech
› Parthasarathy Ranganathan, Google
› Adrian Sampson, Cornell University
› Daniel Sanchez, MIT
› Joshua San Miguel, University of Wisconsin-
Madison
› Simha Sethumadhavan, Columbia Univer-
sity/Chip Scan
› Sophia Shao, UC Berkeley
› Yan Solihin, University of Central Florida
› Viji Srinivasan, IBM Research
› Josep Torrellas, University of Illinois-Urbana
Champaign
› Caroline Trippel, Stanford University
› Thomas F. Wenisch, University of Michigan
and Google
› Lisa Wu Wills, Duke University
› Yuan Xie, University of California, Santa
Barbara
› Mengjia Yan, MIT
› Jishen Zhao, University of California, San
Diego
IEEE Micro 9
 THEME ARTICLE: TOP PICKS

The Vision Behind MLPerf: Understanding AI

Inference Performance
Vijay Janapa Reddi, Harvard University, Cambridge, MA, 02138, USA
Christine Cheng, Intel, Santa Clara, CA, 95054-1549, USA
David Kanter, MLCommons, Mountain View, CA, 94105, USA
Peter Mattson , Google, Mountain View, CA, 94043, USA
Guenther Schmuelling, Microsoft Corp, Redmond, WA, 98052, USA
Carole-Jean Wu , Facebook Inc., Menlo Park, CA, 342996, USA

Deep learning has sparked a renaissance in computer systems and architecture. Despite
the breakneck pace of innovation, there is a crucial issue concerning the research and
industry communities at large: how to enable neutral and useful performance
assessment for machine learning (ML) software frameworks, ML hardware accelerators,
and ML systems comprising both the software stack and the hardware. The ML field
needs systematic methods for evaluating performance that represents real-world use
cases and useful for making comparisons across different software and hardware
implementations. MLPerf answers the call. MLPerf is an ML benchmark standard driven
by academia and industry (70+ organizations). Built out of the expertise of multiple
organizations, MLPerf establishes a standard benchmark suite with proper metrics and
benchmarking methodologies to level the playing field for ML system performance
measurement of different ML inference hardware, software, and services.

I
n this article, we describe the MLPerf Inference
benchmark’s design principles. Inference spans
across the full spectrum of machine learning (ML)
systems, ranging from cloud-scale data centers to
edge devices like smartphones and endpoint systems
based on embedded hardware. We present the chal-
lenges and opportunities in developing a benchmark
that tackles the complexity, heterogeneity, and scale
of ML systems. Specifically, we describe the design of
benchmarking scenarios. These scenarios reflect real-
world ML deployment use cases to address the various
inference deployment services and the evaluation
methodologies that enable measurement results of
various optimization techniques to reflect realistic per-
formance effects from inference execution conditions.
The vision behind MLPerf11 is to provide fair and useful
benchmarks for measuring training10 and inference13 per-
formance of ML hardware, software, and services. Since
0272-1732 ß 2021 IEEE
Digital Object Identifier 10.1109/MM.2021.3066343
Date of publication 17 March 2021; date of current version
25 May 2021.
the launch of MLPerf Inference in November 2019, over
3,500 individual results from 100+ system configurations
have been submitted: the number of inference submis-
sions has tripled from 600+ to 1,800+ between the Octo-
ber 2019 and March 2021 submission rounds. Submitting
organizations include large organizations and small start-
ups, reflecting the diversity in hardware innovation
(CPUs, GPUs, FPGAs, and other accelerators) and their
respective software stacks. The performance of the slow-
est and fastest ML inference systems span over five
orders of magnitude, representing the broad spectrum of
ML deployments and use cases, as summarized in Table 1.
Although the results include various systems, the variety
of results are comparable because of the standardized
ML benchmarks, data sets, metrics, and execution rules.
MLPERF INFERENCE
Building an ML system benchmark requires us to
address three challenges: the diversity of models, the
variety of deployment scenarios, and the array of infer-
ence systems. To this end, the essential products of
MLPerf Inference are as follows.

10 IEEE Micro Published by the IEEE Computer Society May/June 2021


TABLE 1. MLPerf covers a wide range of ML deployments,
spanning from cloud datacenters to ultralow-power systems.
MLPerf Machine learning tasks
Inference
Cloud/ Image classification, object detection,
Datacenter image segmentation, speech, language
processing, recommendation,
reinforcement learning
Edge Image classification, object detection,
image segmentation, speech, language
processing
Mobile Image classification, object detection,
image segmentation, language processing
Tiny Image classification, Object detection,
Anomaly detection, Speech
MLPerf
training
Cloud/ Image classification, object detection,
Datacenter image segmentation, natural language
processing, recommendation,
reinforcement learning
HPC Climate segmentation, cosmology
parameter prediction
Identify Representative ML Inference
Workloads for Reproducibility,
Accessibility, and Fair Comparison
The ML ecosystem is rife with models. Comparing and
contrasting ML system performance is nontrivial
because implementations vary in model complexity and
execution characteristics. A name such as ResNet-50
does not uniquely or portably describe a model. Conse-
quently, understanding improvements in system perfor-
mance with such an unstable baseline is challenging.
One of the significant contributions of this work is
identifying and defining representative models and
describing an inclusive benchmarking methodology
that enables result reproducibility. Based on consen-
sus of the working groups, MLPerf inference comprises
mature models and have earned broad community
support. MLPerf models, the reference implementa-
tions, and the benchmarking infrastructure and meth-
odology are all based on open-source software and can
be easily leveraged for research and further develop-
ment (https://github.com/mlcommons/inference).
Define Different Usage Scopes for
Realistic Evaluation and Meaningful
Measurement Results
ML inference systems cover a broad spectrum, rang-
ing from deeply embedded devices to smartphones to
edge servers and data centers. These systems
May/June 2021
TOP PICKS
empower a variety of real-world ML use cases, each of
which requires scenario-specific performance metrics.
Appropriate metrics, reflecting production use cases,
are not specific to MLPerf but will benefit the ML and
systems community to make fair and neutral system
performance comparison.
MLPerf Inference consists of four evaluation sce-
narios: single-stream, multistream, server, and offline.
The scenarios capture many critical inference applica-
tions. As demonstrated in the original article,13 the
performance of an ML system can vary drastically
under these different scenarios using the correspond-
ing metrics. MLPerf Inference provides a way to emu-
late the inference system’s realistic behavior under
test; such a feature is unique amongst existing AI
benchmarks.
THE VISION BEHIND MLPERF IS TO
PROVIDE FAIR AND USEFUL
BENCHMARKS FOR MEASURING
TRAINING AND INFERENCE
PERFORMANCE OF ML HARDWARE,
SOFTWARE, AND SERVICES.
Prescribe Target Qualities and Tail-
Latency Bounds in Accordance With
Real-World Use Cases
Quality and performance are intimately connected for
all forms of ML. System architectures and their opera-
tors sometimes choose to trade off model quality to
achieve lower latency, lower total cost of ownership,
or higher throughput. The tradeoffs are application-
specific. To reflect this important aspect of real-world
deployments, benchmarks must provide representa-
tive model quality targets.
MLPerf defines machine learning model accuracy
or quality targets. For example, for accuracy-sensitive
tasks, such as recommendation systems and medical
imaging, stringent quality targets of 99% and 99.9% of
the reference implementation are specified and
enforced. We established per-model and scenario tar-
gets for inference latency and model quality. The
latency bounds and target model qualities are based
on input gathered and consensus reached from end-
users, ML systems practitioners, and the correspond-
ing working groups. As MLPerf evolves these in the
future based on industry needs, the broader research
community can track these changes to stay relevant.
IEEE Micro 11
TOP PICKS
Synthesize Rules That Enable
Benchmark Users to Showcase
Hardware and Software Capabilities
The ML community has embraced a variety of DL lan-
guages and libraries. Hence, there is a need for a
semantic-level benchmark that specifies the task to
be accomplished and the general rules to follow but
leaves implementation choices to the benchmark
submitters.
MLPerf Inference is a semantic-level benchmark
that provides flexibility for submitters to optimize the
reference models, i.e., to run the reference code we
provide through their preferred software toolchains
and measure the system performance on the hard-
ware of choice. This allows submitters to perform cus-
tom optimizations, such as use different numerical
formats and various quantization schemes. within
some restricted set of rules. MLPerf Inference pro-
vides two result divisions: closed and open. Strict rules
govern the closed division, whereas the open division
allows submitters to change the model and demon-
strate different latency and accuracy targets. The
closed division addresses the lack of a standard infer-
ence-benchmarking workflow. The open division
relaxes a subset of the benchmark rules to encourage
a wide variety of software-level innovations.
MLPERF INFERENCE FOCUSES FIRST
AND FOREMOST ON MODULAR
BENCHMARK DESIGN TO ADD NEW
MODELS AND TASKS LESS COSTLY
WHILE PRESERVING THE USAGE
SCENARIOS, TARGET QUALITIES, AND
INFRASTRUCTURE.
Present a Modular and Flexible
Inference Benchmarking Methodology
That Allows Components to Get
Updated Individually
ML applications, tasks, and models evolve quickly.
Therefore, the ultimate challenge for any ML bench-
marks is not the tasks themselves, rather the underly-
ing benchmarking methodology that can withstand
the rapid pace of change.
MLPerf Inference focuses first and foremost on
modular benchmark design to add new models and
tasks less costly while preserving the usage scenarios,
target qualities, and infrastructure. As we show in the
12 IEEE Micro
original article, our design enables the benchmark to
extend the scope to include more areas and tasks.
Between 2019 and 2020, MLPerf Inference introduced
four new inference benchmarks for data centers—
DLRM for recommendation,7,12 3D U-NET for medical
segmentation,8 RNN-T9 and BERT6 for natural lan-
guage processing, and additional inference bench-
marks for mobile—Mobile-BERT15 for natural language
processing, MobileNetEdgeTPU1 for image classifica-
tion, and DeepLabV3+5 for image segmentation.
BENCHMARK DESIGN
The ML system diversity presents a unique challenge
to deriving a robust and useful ML benchmark that
meets industry needs. MLPerf Inference adopted a set
of principles for developing a robust yet flexible
benchmark suite based on community-driven develop-
ment. Here, we describe the benchmarks, the quality
targets, and the scenarios under which the benchmark
tasks can be evaluated.
Representative, Broadly Accessible
Workloads
Designing ML benchmarks is different from designing
traditional non-ML benchmarks. MLPerf defines high-
level tasks that a machine-learning system can per-
form. For each one, we provide a canonical reference
model(s) in a few widely used frameworks. Any imple-
mentation that is mathematically equivalent to the
reference model is considered valid, and certain other
deviations (e.g., numerical formats) are allowed.
The concept of a reference model and a valid class
of equivalent implementations creates freedom for
most ML systems while still enabling relevant compar-
isons of inference systems. MLPerf provides reference
models using 32-bit floating-point weights and, for
convenience, also provides carefully implemented
equivalent models to address three formats: Tensor-
Flow, PyTorch, and ONNX.
Table 2 summarizes the vision, speech, language,
and commerce tasks in MLPerf Inference v1.0 (exclud-
ing Mobile14 and tinyMLPerf4). From 2019 to 2021,
MLPerf Inference doubled in size, adding speech, com-
merce, and medical imaging to the suite. This trend
will continue as new benchmark tasks, models and
versions become available.
Realistic End-User Scenarios
ML applications have a variety of usage models and fig-
ures of merit, which, in turn, require multiple perfor-
mance metrics. We surveyed MLPerf’s membership,
which includes customers and vendors. Based on that
May/June 2021
 TOP PICKS

TABLE 2. ML tasks in the edge- and datacenter-class for MLPerf inference v1.0. Each one reflects critical commercial and
research use cases for a large class of ML practitioners; together they also capture a broad set of computing motifs. The top
four tasks and reference models were introduced in v0.5, while the lower four were added in v0.7.

v0.5 v0.7 v1.0 Area Task Reference Data Set Quality Target (Top-1
Model Accuracy)
@ @ @ Vision Image ResNet-50 ImageNet (224x224) 99% of FP32
classification v1.5
(heavy)
@ @ @ Vision Object detection SSD- COCO (1,200x1,200) 99% of FP32
(heavy) ResNet34
@ @ @ Vision Object detection SSD- COCO (300x300) 99% of FP32
(light) MobileNet-
v1
@ Language Machine GNMT WMT16 EN-DE 99% of FP32
translation
@ @ Commerce Recommendation DLRM 1TB Click Logs 99% of FP32 and
99.9% of FP32
@ @ Language Language BERT SQuAD v1.1 (max_seq_len=384) 99% of FP32 and
processing 99.9% of FP32
@ @ Speech Speech-to-text RNN-T Librispeech dev-clean 99% of FP32
(samples < 15 seconds)
@ @ Vision Medical image 3D U-Net BraTS 2019 (224x224x160) 99% of FP32 and
segmentation 99.9% of FP32

feedback, we identified four scenarios that represent a
variety of critical inference applications: single-stream,
multistream, server, and offline. These scenarios emulate
the ML workload behavior of mobile devices, autono-
mous vehicles, robotics, and cloud-based setups.
Single-stream
This scenario represents one inference-query stream
with a query sample size of one, reflecting the many
client applications where response latency is critical.
One example is the on-device voice transcription task
on Google’s Pixel 4 smartphone. To measure perfor-
mance, we inject a single query into the inference sys-
tem; when the query is complete, we record the
completion time and inject the next query. The metric
is the query stream’s 90th-percentile latency.
Multistream
This scenario represents applications with a stream of
queries, but each query comprises multiple inferences,
reflecting a variety of industrial-automation and
remote-sensing applications. To model a concurrent
scenario, we send a new query comprising N input sam-
ples at a fixed time interval (e.g., 50 ms). The interval is
benchmark specific and also acts as a latency bound
that ranges from 50 to 100 ms. If the system is available,
it processes the incoming query. If it is still processing
the prior query in an interval, it skips the interval and
delays the remaining queries by one interval. No more
than 1% of the queries may produce one or more
skipped intervals. A query’s N input samples are contig-
uous in memory, which accurately reflects production
input pipelines and avoids penalizing systems that
would otherwise require that samples be copied to a
contiguous memory region before starting inference.
Server
This scenario represents online applications where
query arrival is random and latency is important.
Almost every consumer-facing website is a good
example, including services such as online translation
from Baidu, Google, and Microsoft. For this scenario,
queries are sent with one sample each, in accordance
with a Poisson distribution. The system under test
(SUT) responds to each query within a benchmark-
specific latency bound that varies from 15 to 250 ms.
No more than 1% of queries may exceed the latency
bound for the vision tasks and no more than 3% may
do so for translation. The scenario’s performance met-
ric is the Poisson parameter that indicates the queries
per second (QPS) achievable while meeting the quality
of service (QoS) requirement.
Offline
This scenario represents batch-processing applica-
tions where all the data are immediately available and

May/June 2021 IEEE Micro 13

TOP PICKS
latency is unconstrained. An example is identifying the
people and locations in a photo album. For the offline
scenario, we send a single query that includes all sam-
ple-data IDs to be processed, and the system is free to
process the input data in any order. Similar to the mul-
tistream scenario, neighboring samples in the query
are contiguous in memory. The metric for the offline
scenario is throughput measured in samples per
second.
For the multistream and server scenarios, latency
is a critical component of the system behavior and
constrains various performance optimizations. For
example, most inference systems require a minimum
(and architecture-specific) batch size to achieve full
utilization of the underlying computational resources.
But in a server scenario, the arrival rate of queries is
random, so systems must optimize for tail latency and
potentially process inferences with a suboptimal
batch size.
To reflect real-world inference use cases for data-
center and edge systems, each task and scenario has
specific quality and quality-of-service constraints. For
MLPerf v1.0, we require datacenter system submis-
sions to cover Server and Offline whereas Single-
stream and Offline are required for edge system sub-
missions, and multistream is optional. We additionally
require that all datacenter submissions must use ECC
for off-chip memory to reflect real-world requirements.
Robust Quality Targets
The tradeoffs between accuracy, latency, and
throughput are application specific. We require that
most implementations achieve a quality target within
1% of the FP32 reference model’s accuracy. For the
medical imaging, language processing, and recom-
mendation tasks in datacenter submissions, we addi-
tionally require achieving a quality target that is within
0.1% of the reference, reflecting realistic deployment
needs.
Statistically Confident Tail-Latency
Bounds
Each task and scenario combination requires a mini-
mum number of queries to ensure results are statisti-
cally robust and adequately capture steady-state
system behavior. That number is determined by the
tail-latency percentile, the desired margin, and the
desired confidence interval. We selected a margin that
is one-twentieth of the difference between the tail-
latency percentage and 100%. For scenarios with
latency constraints, our goal is to ensure a 99% confi-
dence interval that the constraints hold. In addition to
14 IEEE Micro
the latency constraint, each benchmark scenario
must run for at least N amount of time (e.g., 60 s in
v0.5 and 10 min in v1.0) and process additional queries
and/or samples as the scenarios requires.
INFERENCE SUBMISSION SYSTEM
An MLPerf inference submission system contains mul-
tiple components: an SUT, the Load Generator
(LoadGen), a data set, and an accuracy script. In this
section, we describe these various components. The
data set, LoadGen, and accuracy script are fixed for all
submissions and are provided by MLPerf. Submitters
have wide discretion to implement an SUT according
to requirements and engineering judgment.
System Under Test
The goal of MLPerf Inference is to measure system
performance across a wide variety of SUTs. But the
properties of realism, comparability, architecture neu-
trality, and friendliness to small submission teams
require careful tradeoffs. Therefore, we set the model-
equivalence rules to allow submitters to re-implement
models on different architectures. The rules provide a
complete list of disallowed techniques and a list of
allowed technique examples. The list illustrates the
boundaries of the blacklist while also encouraging
common and appropriate optimizations.
Load Generator
The LoadGen is a traffic generator that loads the SUT
and measures performance that is provided by MLPerf
Inference. The LoadGen produces the query traffic
according to the rules of the previously described sce-
narios (i.e., single-stream, multistream, server, and off-
line). Additionally, the LoadGen collects information
for logging, debugging, and postprocessing the data. It
records queries and responses from the SUT, and at
the end of the run, it reports statistics, summarizes
the results, and determines whether the run was valid.
The LoadGen has two primary operating modes: accu-
racy and performance. It measures the holistic perfor-
mance of the entire SUT rather than any individual
part. Finally, this condition enhances the benchmark’s
realism: inference engines typically serve as black-box
components of larger systems.
Data Set
We use standard and publicly available data sets
(Table 2) to ensure that the community can partici-
pate. Data sets are downloaded before LoadGen uses
it to run the benchmark.
May/June 2021

FIGURE 1. Performance range across systems for models in
single-stream (SS), multistream (MS), server (S), and offline
(O) scenarios.
Accuracy Checker
The LoadGen also ensures the submission system
complies with the rules. In addition, it can self-check
to determine whether its source code has been modi-
fied during the submission process. To facilitate vali-
dation, the submitter provides an experimental config
file that allows use of nondefault LoadGen features.
Audits
To provide additional verification, we introduced an
audit process in MLPerf Inference v0.7. The audits are
designed to ensure compliance with the rules and
help to understand any potential performance anoma-
lies. One submission is randomly selected for an audit,
and a second systems is selected by the submitters
collectively. During an audit, an auditor will examine,
reproduce, and experiment with a submitted system
to verify the performance.
Power Measurement
We built and introduced a set of tools to support
optional full-system power measurement for wall-pow-
ered systems. We licensed the PTDaemon software
from SPEC3 to interface with a wide variety of power
analyzers, and built a set of tools to automate the col-
lection, correlation, and verification of power con-
sumption during performance runs, thereby enabling
direct measurement of power and energy efficiency.
Power measurement also influenced our choice of the
10 minute minimum run time.
RESULT HIGHLIGHTS
In October 2019, the first version of the benchmark
(v0.5) was put to the test. We received over 600 sub-
missions in all three categories (available, preview,
and research/products under development) across
May/June 2021
TOP PICKS
FIGURE 2. Systems that perform well for throughput (offline
mode) may perform poorly under latency-bounded through-
put (server mode).
the closed and open divisions from 14 organizations.
The results are the most extensive corpus of inference
performance data available to the public, covering a
range of ML tasks and scenarios, hardware architec-
tures, and software runtimes. Here, we briefly assess
the major results on the basis of our benchmark
objectives.
Benchmark Must Capture and Reflect
the Broad Power and Performance
Range, From Mobile and Edge Devices
to Cloud Computing Systems
The performance delta between the smallest and larg-
est inference systems is four orders of magnitude, or
about 10,000x. Figure 1 shows the results across vari-
ous inference tasks and scenarios. In MobileNet-v1
single-stream scenario (SS), ResNet50 v1.5 (SS), and
SSD-MobileNet-v1 offline (O), systems exhibit a large
performance difference (100x). Because these models
have many applications, the systems that target them
cover everything from low-power embedded devices
to high-performance servers.
Accurate Modeling of Inference
Deployment Scenarios Is Key to
Produce Representative System
Performance Benchmarking Results
Figure 2 shows the throughput degradation from the
server scenario, i.e., latency-bounded throughput con-
straint that is normally ignored in evaluations. Across
all inference benchmarks (x-axis), the queries-per-sec-
ond throughput ratio between the Server and Offline
scenarios is up to 0.55 for gnmt and on average 0.8 for
ResNet50 and SSD-ResNet34. This 20% performance
difference comes from the ability of the Inference
LoadGen to accurately reflect query serving
IEEE Micro 15
TOP PICKS
scenarios—implementing query arrival patterns using
the Poisson distribution for the Server scenario. This is
important as query arrival characteristics determine
the degree of input batching by balancing queuing
effects with compute parallelism. Also, as the through-
put of AI systems increases, the effect of query arrival
patterns becomes much less pronounced.
Results Must Reflect Real-World ML
Solutions Deployed on Many Platforms,
Ranging From General-Purpose CPUs
to Programmable GPUs and DSPs,
FPGAs, and Fixed-Function
Accelerators
The MLPerf Inference results reflect this diversity.
Overall, the MLPerf Inference submissions cover most
hardware categories, as summarized in https://mlperf.
org/inference-results/. The system diversity indicates
that the inference benchmark suite and the described
methodologies are broadly applicable to evaluate any
processor architectures.
FUTURE DIRECTIONS
Demand for Expanding the Benchmark
Scope
There is a strong demand to expand the breadth and
scope of ML use cases that reflect industry diversity
and relevance. The industry has grown MLPerf Infer-
ence by adding new benchmarks, creating new bench-
mark suites, and establishing new metrics and
methodologies driven by ML use cases in less than a
year.
TO ENABLE US TO BUILD BETTER ML
MODELS THAT DRIVE SYSTEMS
ACTIVITIES, THE MLPERF COMMUNITY
HAS EXPANDED INTO CREATING
LARGE, HIGH-QUALITY, FAIR DATA
SETS TO DRIVE ML INNOVATIONS.
New Networks
MLPerf has expanded the benchmark suite with four
new inference benchmarks (DLRM, 3D U-Net, BERT,
and RNN-T). New MLPerf benchmarks go through
advisory boards to steer future benchmark construc-
tion and development. The collaboration between the
advisory boards and the MLPerf submitters produces
16 IEEE Micro
the benchmark reference code and specification.
DLRM for recommendation is the first benchmark pro-
duced using this process.16
New Benchmark Suites
Inference is deployed on a wide variety of systems,
ranging from data centers to edge servers to smart-
phones to tiny devices. Each of these domains has its
unique use-case-driven models and requirements. To
this end, driven by the demand from the industry,
MLPerf Inference has launched three new suites in
2020: 1) one for data center systems, 2) one for edge
systems, and 3) one for mobile devices such as smart-
phones (via an Android App) and laptops.14 Another
benchmark, tinyMLPerf for ultralow-power ML devi-
ces,4 is slated for release in 2021 with power
measurements.
End-to-end AI Performance
There is growing awareness that ML hardware is
developed and evaluated in a vacuum without consid-
ering the full application environment. AI applications
rely heavily on preprocessing and postprocessing
code, storage, and communication systems, which
must be considered while assessing end-to-end AI sys-
tem performance. To this end, MLPerf Inference is
maturing into measuring AI system performance by
developing an end-to-end (E2E) AI benchmarking
methodology to capture the increasing levels of
abstraction in understanding AI applications.
Stringent Metrics
MLPerf Inference has evolved quickly to not only
establish best practices for assessing ML perfor-
mance but also point the industry in the right direction
by establishing proper metrics. For instance, with
medical image segmentation (3D U-Net), MLPerf Infer-
ence increased the accuracy threshold from 99% of
FP32 accuracy to 99.9% of FP32 (0.85300 mean DICE
score). This rationale was driven by community feed-
back and input that a 1% accuracy drop could mean a
significant number of misdiagnosis in brain tumors. In
real-life, the standards are more stringent. So the com-
munity is using MLPerf Inference to set “gold stand-
ards” for what it means to achieve performance while
avoiding arbitrary numerical precision tradeoffs.
Accelerating AI Innovation
MLPerf Inference currently captures two of the three
fundamental pillars fueling ML—algorithms and sys-
tems—with the open and closed divisions. The third
pillar is the data. Data sets are the rocket fuel of
May/June 2021

machine learning. The ultimate demand for new and
improved ML systems arises out of new ML algorithms
that emerge to tackle complexity in data sets, both in
scale and diversity.
To enable us to build better ML models that drive
systems activities, the MLPerf community has expanded
into creating large, high-quality, fair data sets to drive ML
innovations. Specifically, we are spawning off new data-
set initiatives, such as People’s Speech and 1000 Words
in 1000 Languages, to fuel and accelerate language-
based advancement in ML algorithms and system devel-
opment. Taking a step further, the community is also
exploring best practices to make it easier to share and
publish ML models using our MLCube approach.
Streamlining and Fostering Research
We are beginning to see MLPerf Inference’s impact on
the experimental methodologies used to assess AI sys-
tems’ performance and efficiency. For instance, the
LoadGen we developed is being adopted as a testing
harness to evaluate ML system performance across the
scenarios. We hope to see the influence of MLPerf Infer-
ence on academic and industrial research by enabling
future works to adopt the benchmarks, metrics, quality,
and accuracy target thresholds as standard “baselines”
for systematically evaluating their research.
CONCLUSION
The explosion in ML hardware and software requires a
standard benchmark to focus on machine learning
systems with clear metrics. MLPerf enables fair
apples-to-apples comparisons—it is a neutral system
performance benchmark suite that levels the playing
field for ML systems, from programming frameworks
and compiler construction to system architectures,
and hardware implementations.
ACKNOWLEDGMENTS
In addition to the leads, MLPerf Inference Benchmark13
was carried by the following primary co-authors: Brian
Anderson (Google), Maximilien Breughe (NVIDIA),
Ramesh Chukka (Intel), Cody Coleman (Stanford), Itay
Hubara (Habana), Thomas B. Jablin (Google), Pankaj
Kanwar (Google), Anton Lokhmotov (dviditi), Francisco
Massa (Facebook), Gennady Pekhimenko (University of
Toronto & Vector Institute), Dilip Sequeira (NVIDIA),
Ashish Sirasao (Xilinx), Tom St. John (Cruise), George
Yuan (NVIDIA). MLPerf Inference is also the work of sev-
eral collaborators and submitters who helped produce
the first large set of benchmark results. To see the com-
plete list, please refer to the original MLPerf Inference
Benchmark13 and MLPerf Training Benchmark10 papers.
May/June 2021
TOP PICKS
We are also thankful for the MLCommons advisory
boards’ contribution and the colleagues for the data-
sets and model reference implementations: https://
mlcommons.org/en/credits/.
REFERENCES
1. “Introducing the next generation of on-device vision
models: MobileNetV3 and MobileNetEdgeTPU,” 2019.
[Online]. Available: https://ai.googleblog.com/2019/11/
introducing-next-generation-on-device.html
2. “ MLCommons: Machine learning innovation to benefit
everyone,” 2020. [Online]. Available: https://
mlcommons.org/en/
3. “A new version of the PTDaemon tool is now available,”
2020. [Online]. Available: https://www.spec.org/power/
docs/SPECpower-PTD-Update_Process.html
4. C. R. Banbury et al., “Benchmarking TinyML systems:
Challenges and direction,” 2020, arXiv:2003.04821.
5. L.-C. Chen, G. Papandreou, F. Schroff, and H. Adam,
“Rethinking atrous convolution for semantic image
segmentation,” 2017, arXiv:1706.05587.
6. J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert:
Pre-training of deep bidirectional transformers for
language understanding,” 2019, arXiv:1810.04805.
7. U. Gupta et al., “The architectural implications of
Facebook’s DNN-based personalized
recommendation,” in Proc. IEEE Int. Symp. High
Perform. Comput. Archit., 2020, pp. 488–501, doi:
10.1109/HPCA47549.2020.00047.
8. F. Isensee, P. F. Jaeger, S. A. A. Kohl, J. Petersen, and K.
H. Maier-Hein, “NNU-NET: A self-configuring method
for deep learning-based biomedical image
segmentation,” Nature Methods, vol. 18, no. 2, pp. 203–
211, Dec. 2020.
9. M. Johnson et al., “Google’s multilingual neural
machine translation system: Enabling zero-shot
translation,” 2017, arXiv:1611.04558.
10. P. Mattson et al., “MLPerf training benchmark,” in Proc.
3rd Conf. Mach. Learn. Syst., 2020, arXiv:1910.01500.
11. P. Mattson et al., “MLPerf: An industry standard
benchmark suite for machine learning performance,”
IEEE Micro, vol. 40, no. 2, pp. 8–16, Mar.–Apr. 2020, doi:
10.1109/MM.2020.2974843.
12. M. Naumov et al., “Deep learning recommendation
model for personalization and recommendation
systems,” 2019, arXiv:1906.00091.
13. V. J. Reddi et al., “MLPerf inference benchmark,” in
Proc. ACM/IEEE 47th Annu. Int. Symp. Comput. Archit.,
2020, pp. 446–459, doi: 10.1109/ISCA45697.2020.00045.
14. V. J. Reddi et al., “MLPerf mobile inference benchmark:
Why mobile AI benchmarking is hard and what to do
about it,” 2020, arXiv:2012.02328.
IEEE Micro 17
TOP PICKS
15. Z. Sun, H. Yu, X. Song, R. Liu, Y. Yang, and D. Zhou,
“Mobilebert: A compact task-agnostic bert for
resource-limited devices,” 2020, arXiv:2004.02984.
16. C.-J. Wu et al., “Developing a recommendation
benchmark for MLPerf training and inference,” 2020,
arXiv:2003.07336.
VIJAY JANAPA REDDI is currently an Associate Professor at
Harvard University, Cambridge, MA, USA. Before joining Har-
vard, he was an Associate Professor with the Department of
Electrical and Computer Engineering, The University of Texas
at Austin, Austin, TX, USA. His research interests include
computer architecture and runtime systems, specifically in
the context of autonomous machines and mobile and edge
computing systems. He received multiple honors and awards,
including the National Academy of Engineering (NAE) Gil-
breth Lecturer Honor and has been inducted into the MICRO
and HPCA Halls of Fame. Reddi received a B.S. degree from
Santa Clara University, an M.S. degree from the University of
Colorado at Boulder, and a Ph.D. degree in computer science
from Harvard University. Contact him at vj@ece.utexas.edu.
CHRISTINE CHENG is one of the engineering leads for deep
learning benchmarking and optimization at Intel, Santa Clara,
CA, USA. She has been involved with shaping MLPerf bench-
marks from the beginning and has also led teams at Intel to
submit results to every MLPerf training and inference round.
Before joining MLPerf/MLCommons, she was a data scientist
in sports analytics. Cheng received a B.S. degree from Cal-
tech and an M.S. degree from Stanford University. Contact
her at christine.cheng@intel.com.
DAVID KANTER is a founder and the executive director of
MLCommons, Mountain View, CA, USA, where he helps lead
the MLPerf benchmarks and other initiatives. He has more
than 16 years of experience in semiconductors, computing,
and machine learning. He founded a microprocessor and
compiler startup, was an early employee at Aster Data Sys-
tems, and has consulted for industry leaders such as Intel,
Nvidia, KLA, Applied Materials, Qualcomm, Microsoft, and
many others. Kanter received a B.S. degree with honors in
18 IEEE Micro
mathematics with a specialization in computer science and a
B.A. degeee with honors in economics from the University of
Chicago. Contact him at dkanter@gmail.com.
PETER MATTSON leads ML Metrics at Google, Mountain
View, CA, USA. He co-founded and is the president of
MLCommons, and co-founded and was the general chair of
the MLPerf Consortium that preceded it. Previously, he
founded the Programming Systems and Applications
Group at NVIDIA Research, was the VP of software infra-
structure for Stream Processors Inc. (SPI), and was a Man-
aging Engineer at Reservoir Labs. His research focuses on
understanding machine learning models and data through
quantitative metrics and analysis. Mattson received a B.S.
degree from the University of Washington and M.S. and
Ph.D. degrees from Stanford University. Contact him at
petermattson@google.com.
GUENTHER SCHMUELLING is currently an Inference Chair at
MLPerf. He is a Principal Tech Lead. He is with the Microsoft
Azure AI Infrastructure, Redmond, WA, USA. Contact him at
guschmue@microsoft.com.
CAROLE-JEAN WU is currently a Research Scientist at Face-
book AI Research, Menlo Park, CA, USA. She is an Associate
Professor at Arizona State University, Tempe, AZ, USA. Her
research lies in the domain of computer system. Her work has
pivoted into designing systems for machine learning execu-
tion at-scale and tackling system challenges to enable effi-
cient, responsible AI execution. She chairs the MLPerf
Recommendation Benchmark Advisory Board and co-chaired
MLPerf Inference. She was the recipient of the NSF CAREER
Award, the Facebook AI Infrastructure Mentorship Award, the
IEEE Young Engineer of the Year Award, the Science Founda-
tion Arizona Bisgrove Early Career Scholarship, and the Intel
PhD Fellowship, among a number of best paper awards. Wu
received a B.Sc. degree from Cornell University, and M.A. and
Ph.D. degrees from Princeton University. Contact her at
carolejeanwu@fb.com
May/June 2021
THEME ARTICLE: TOP PICKS

Superconductor Computing for Neural

Networks
Koki Ishida , Department of Advanced Information Technology, Kyushu University, Fukuoka, 464-8601, Japan
Ilkwon Byun , Department of Electrical and Computer Engineering, Seoul National University, Gwanak-gu,
08826, South Korea
Ikki Nagaoka , Department of Electronics, Nagoya University, Nagoya, 464-8601, Japan
Kosuke Fukumitsu, Department of Advanced Information Technology, Kyushu University, Fukuoka, 464-8601,
Japan
Masamitsu Tanaka , Department of Electronics, Nagoya University, Nagoya, 464-8601, Japan
Satoshi Kawakami , Teruo Tanimoto , and Takatsugu Ono , Department of Advanced Information
Technology, Kyushu University, Fukuoka, 464-8601, Japan
Jangwoo Kim, Department of Electrical and Computer Engineering, Seoul National University, Gwanak-gu,
08826, South Korea
Koji Inoue , Department of Advanced Information Technology, Kyushu University, Fukuoka, 464-8601, Japan

The superconductor single-flux-quantum (SFQ) logic family has been recognized

as a promising solution for the post-Moore era, thanks to the ultrafast and low-
power switching characteristics of superconductor devices. Researchers have
made tremendous efforts in various aspects, especially in device and circuit
design. However, there has been little progress in designing a convincing SFQ-
based architectural unit due to a lack of understanding about its potentials and
limitations at the architectural level. This article provides the design principles
for SFQ-based architectural units with an extremely high-performance neural
processing unit (NPU). To achieve our goal, we developed and validated a
simulation framework to identify critical architectural bottlenecks in designing a
performance-effective SFQ-based NPU. We propose SuperNPU, which
outperforms a conventional state-of-the-art NPU by 23 times in terms of
computing performance and 1.23 times in power efficiency even with the cooling
cost of the 4K environment.

W
e are about to enter an era where both
Moore’s law and Dennard scaling do not
hold anymore. We are running out of
effective options to improve the performance of
CMOS-based computer systems while maintaining
their power and temperature budgets.1 Therefore,
This work is licensed under a Creative Commons Attribution
4.0 License. For more information, see https://creativecom-
mons.org/licenses/by/4.0/
Digital Object Identifier 10.1109/MM.2021.3070488
Date of publication 5 April 2021; date of current version
25 May 2021.
we believe that now is the right time to exploit
emerging device technologies with significant
potential and make a serious effort to improve their
feasibility.
Among the several candidates, the superconduc-
tor single-flux-quantum (SFQ) logic family2 has
emerged as a highly promising solution for the post-
Moore era. SFQ technology, which utilizes supercon-
ductor devices operating at 4K, has significant poten-
tial for both high performance and energy efficiency.
Specifically, SFQ logic gates use low-voltage impulse-
shaped signals for their operations and achieve the
ultrafast (
1012 s) and low-energy (
1019 J)

May/June 2021 Published by the IEEE Computer Society IEEE Micro 19

TOP PICKS
switching. By focusing on these strong points, many
researchers have contributed to SFQ-related research
in various aspects, especially in device and circuit
design.
However, little research has been conducted on
SFQ-based architectures3 due to a lack of under-
standing about its architecture-level potentials and
limitations. As we show later, SFQ logic, with its
unique pulse-driven nature, requires completely dif-
ferent architecture designs from conventional
CMOS technology. In consideration of SFQ-specific
architectural tradeoffs, the following questions
must be clearly addressed. 1) Which architecture is
most promising for this technology? 2) How can we
implement various microarchitectural units in an
SFQ-friendly manner? 3) How can we fully exploit its
potential at the architecture level while considering
its limitations? 4) How do we simulate and validate
SFQ architectural units?
Our article, presented at MICRO’20, provides
straightforward answers to the above questions in the
form of SuperNPU, our design for an SFQ-based neural
processing unit (NPU). The main contributions of this
work are as follows. 1) Architecting an SFQ-based
NPU: to the best of our knowledge, this is the first
work to design an NPU, which addresses the architec-
tural tradeoffs of SFQ technology. 2) Simulation frame-
work: it is also the first work to model and validate a
simulator for SFQ-based architectures. 3) SFQ-specific
architectural optimizations: we identify critical archi-
tectural bottlenecks and optimizations, which can
cause a performance variance. 4) Significant results:
SuperNPU provides extremely high performance and
power efficiency; it outperforms a conventional design
by 23 times and 1.23 times in terms of performance
and power efficiency even with the cooling cost of the
4K environment. Our thorough architectural analysis
with a validated modeling tool clearly shows the
impact of our SFQ-specific optimization process and
the potential of SFQ computing as a post-Moore-era
solution.
BACKGROUND ON SFQ LOGIC
TECHNOLOGY
SFQ circuits utilize a small-voltage pulse as an infor-
mation carrier, which can be stored as a single mag-
netic flux quantum (SFQ) in a superconductor ring
[Figure 1(a)]. To store and transfer the SFQ, the super-
conductor ring contains superconducting devices
called Josephson junctions (JJs). Each JJ consists of a
thin insulator sandwiched by the superconductors,
20 IEEE Micro
FIGURE 1. Circuit elements and working principle of SFQ logic
technology. (a) Superconductor ring with SFQ. (b) Electrical
characteristics of JJ. (c) Circuit diagram of an SFQ-based
DFF. (d) Determination of SFQ circuits’ frequency with an
operation example of a two-input AND gate.
and it has unique electrical characteristics with which
to generate a voltage pulse [Figure 1(b)].
Figure 1(c) shows the working principle of SFQ
logic gates with an SFQ-based delay flip flop (DFF),
which consists of a single superconductor ring and a
clock line. First, when the input pulse enters the ring,

it is stored in the ring as an SFQ by switching JJ1 ( 1 ).

Next, by receiving a clock pulse ( 2 ), JJ2 is activated,
and the stored SFQ is transferred to the output as a

voltage pulse ( 3 ). In this manner, SFQ gates can rep-
resent the logical value “1” (or “0”) by the existence (or
absence) of the stored SFQ between the two adjacent
clock pulses.
The SFQ logic gates are implemented using stor-
age rings and pulse interactions in a similar way.
Figure 1(d) shows an operation example of a two-input
AND gate. The SFQ circuits’ frequency is determined by
f ¼ 1=CCT ¼ 1=ðSetupTime þ Maxðdt1 ; dt2 ÞÞ
where SetupTime is the timing constraint, and dt1 and
dt2 are the timing gaps between the input pulses and
the first clock pulse arrival. dt1 ; dt2  HoldTime must
be satisfied, where HoldTime is the other timing con-
straint. dt1 and dt2 can be shortened by delaying the
clock pulse arrival. Therefore, the timing gap between
the two inputs’ arrivals (dt3 ) is important. If there is a
large difference between the arrivals, the clock fre-
quency decreases because both inputs must arrive at
the destination gate in the same clock cycle period.
Because the timing constraints are fixed for each SFQ
gate, minimizing these two kinds of timing gaps is
essential for achieving a high clock frequency in an
SFQ design.
May/June 2021

FIGURE 2. Our baseline SFQ-based NPU with each microarchitecture unit’s design alternatives. (a) Overview of our baseline
architecture. (b) Network designs. (c) PE designs.
SFQ-FAVORABLE
ARCHITECTURAL
CHARACTERISTICS
The aim of this article is contributing to the architec-
ture community by introducing SFQ technology from
the architects’ perspective. To achieve this goal, we
describe below SFQ-favorable architectural character-
istics by fully considering the SFQ technology’s unique
features, which originate from its pulse-driven nature.
First, SFQ technology favors simple control flows
due to its deeply pipelined nature. Architects can nat-
urally apply gate-level pipelining to achieve a high fre-
quency,4,5 because all SFQ logic gates consist of
superconductor rings, i.e., all SFQ gates have a latch
functionality and, thus, can be pipelined without addi-
tional DFFs. However, the deep pipeline structure
makes it difficult to avoid data (or control) hazards
and, thus, may suffer from huge pipeline stalls. There-
fore, applications with streaming execution are more
favorable than those with a complex control flow.
Second, the SFQ technology favors sequential
memory access patterns due to its on-chip memory
implementation. There are two options for an SFQ-
based on-chip memory design: random access mem-
ory (RAM) and shift-register-based memory. However,
SFQ-based RAM suffers from poor driving capability
and scalability, mainly because of the difficulty of driv-
ing the word lines and bit lines with the small pulses.
On the other hand, a shift-register-based memory
does not have such problems, so it is a much more
practical option for on-chip memory. This means that
applications with sequential memory accesses are
much more suitable for SFQ technology.
Third, SFQ technology favors fewer off-chip mem-
ory accesses due to a lack of scalable off-chip memory
May/June 2021
TOP PICKS
technology. It has been a long-standing challenge to
implement a large-scale and high-speed off-chip mem-
ory able to operate in a 4K environment. Although
there have been a few studies on JJ-based memories
and they are currently being developed, these technol-
ogies have not been put to practical use yet. For this
reason, it is currently more practical to use CMOS
memory technology, which is slower than the JJ-based
memory but is large and reliable. Thus, computation-
oriented applications with minimal off-chip memory
accesses are suited to the current SFQ technology.
BASELINE SFQ-BASED NPU
DESIGN
After considering the presented characteristics shown
in the “SFQ-Favorable Architectural Characteristics”
section, we chose an NPU as an example of SFQ-favor-
able architecture and designed the baseline SFQ-
based NPU, as shown in Figure 2(a). Specifically, we
implemented the on-chip buffer as a shift-register-
based memory ( 1 ), the network unit (NW unit) as a 2D
systolic network ( 2 ), and the processing element (PE)
with weight-stationary dataflow ( 3 ) in an SFQ-friendly
manner.
Network Unit for Systolic Array
To design an SFQ-friendly on-chip network, we com-
pared two representative NW unit designs: fan-out
network (e.g., splitter tree) and store-and-forward
chain (e.g., systolic array), as shown in Figure 2(b). We
selected the systolic array because it is superior to
the splitter tree in both clock frequency and area. The
splitter tree significantly suffers from the low fre-
quency due to the increasing difference between the
IEEE Micro 21
TOP PICKS
FIGURE 3. Simulation framework and validation. (a) SFQ-NPU overview. (b) Validation results in terms of frequency, power con-
sumption, and area of microarchitectural and architectural units. (c) Validation setup. (d) Chip microphotograph of 2  2 PE
arrayed NPU prototype design.
arrival timings of the two PE inputs. As shown in
Figure 2(b)(1), the timing gap between input 1 and 

input 2 increases in proportion to the PE array width
due to the large difference in clock arrival at two split-
ter trees.
On the other hand, as shown in Figure 2(b)(2), the
systolic network has a smaller timing difference
 
between input 3 and input 4 , which does not scale
with the PE array width. Thus, the systolic array can
achieve a higher clock frequency than the splitter tree
can. In addition, a systolic network has a simpler struc-
ture than the splitter tree and, thus, occupies less
area. For these reasons, we decided that the systolic
array is more SFQ-friendly and chose it to be part of
our on-chip network design.
Processing Element With Weight-
Stationary Dataflow
For an SFQ-friendly PE design, we compared the
designs with two major dataflows in a systolic net-
work, i.e., weight stationary (WS), where the PE stores
weight pixels in its local register, and output station-
ary (OS), where the PE stores ofmap pixels in its local
register, as shown in Figure 2(c). We chose the PE
with WS dataflow to maximize the frequency,
because it does not include any feedback loop.
Unlike in CMOS technology, in which the clocking
scheme synchronizes all the gates, the SFQ logic
employs point-to-point (or gate-to-gate) synchroniza-
tion. SFQ circuits can achieve a higher clock fre-
quency without a feedback loop, because the data
22 IEEE Micro
propagation delay can be hidden by making the clock
pulse flow in the same direction as the data, resulting
in a small timing gap between the arrivals of the clock
and data. However, a circuit with a feedback loop
cannot apply such a frequency speed-up because the
clock and data pulses cannot flow in the same direc-
tion. Thus, we concluded that a PE design without a
feedback loop, PE with WS, is a more SFQ-friendly
choice.
SIMULATION FRAMEWORK
To enable exploration and optimization of the SFQ-
based architectures, we have developed the architec-
tural simulation framework for SFQ technology.
Figure 3(a) shows an overview of our tool targeting
SFQ-based NPU architecture, which consists of two
engines: an SFQ-NPU estimator and SFQ-NPU simula-
tor. In what follows, we describe these engines, their
implementation, and validation.
SFQ-NPU Estimator
The SFQ-NPU estimator predicts the clock frequency,
static power, access energy, and area of the target
NPU configuration. To carefully consider the unique
features of SFQ logic ranging from the device to archi-
tecture, the estimator takes a three-layer abstraction
strategy: gate-level, microarchitecture-level, and archi-
tecture-level estimations.
First, in regard to the device-level parameters,
the gate-level estimation layer derives the timing
parameters, power information, and area for all of
May/June 2021

the SFQ logic gates and wire cells. The gate models
are compatible with two SFQ technologies; rapid
single-flux-quantum (RSFQ)2 and energy-efficient
RSFQ (ERSFQ).6 For the RSFQ gates, all gate param-
eters are extracted by running JSIM7 simulations
with the RSFQ cell library.8 On the other hand, the
ERSFQ gate parameters are estimated from the
RSFQ gate parameters and ERSFQ gate features9
because of the lack of detailed fabrication
information.
Next, the microarchitecture-level layer estimates
the frequency, static power, access energy, and area
of each microarchitectural unit by utilizing the gate-
level layer’s output and the input configuration param-
eters. For an accurate frequency estimation, this layer
generates the intra-unit gate pair information consist-
ing of all source and destination gate pairs in each
unit because the two adjacent gates determine the
SFQ circuit frequency. With the intra-unit gate pair
information, this layer calculates the frequencies of all
gate pairs in the target unit and takes the minimum
value as the unit frequency. This layer also calculates
the static power, access energy, and area of each unit
on the basis of the gate count information and the
gate-level layer’s output.
Lastly, the architecture-level layer reports the esti-
mation results regarding the area, static power,
access energy, and clock frequency of the target NPU
configuration. For an accurate prediction, this layer
not only integrates the microarchitecture-level esti-
mations based on the unit counts but also considers
the inter-unit connection. For instance, it calculates
all the inter-unit communication latencies on the basis
of the interfacing gates’ timing parameters and
accounts for them when deriving the frequency of the
target NPU. The layer also calculates the area of the
wire cells required to connect each unit and includes
it in the final area result.
We thoroughly validated our SFQ-NPU estimator’s
accuracy in terms of frequency, power, and area by
comparing it with a fabricated 4-bit MAC unit mea-
sured in a 4K environment [Figure 3(c)] and in post-lay-
out characterizations of an 8-bit 8-entry shift-register-
based memory (SRmem), 8-bit NW unit, and 2-bit, and
2  2 PE-arrayed NPU. We have also fabricated a pro-
totype chip of the NPU [Figure 3(d)] and plan to mea-
sure it in detail. As Figure 3(b) shows, the SFQ-NPU
estimator accurately predicts the frequency, power,
and area of each microarchitecture unit and target
NPU. Even though our validation was conducted with
a small NPU prototype, the estimation accuracy is
rather convincing, thanks to the systolic network’s
scalable structure.
May/June 2021
TOP PICKS
SFQ-NPU Simulator
For the given SFQ-based NPU design running DNN
applications, the SFQ-NPU simulator reports the effec-
tive performance and power consumption based on
the obtained frequency and power information from
the SFQ-NPU estimator. The SFQ-NPU simulator first
analyzes all of the required weight mappings by taking
a DNN description file as an input. Next, it runs a
cycle-based simulation for each weight mapping to
derive the consumed cycles and hardware activation
ratio. Finally, it aggregates the mapping results and
reports the performance and the power results.
OPTIMIZING SFQ-BASED NPU
DESIGN
By using our validated simulation framework, we iden-
tified and resolved architectural performance bottle-
necks in our baseline SFQ-based NPU design. Then,
we devised our SFQ-optimal NPU architecture, Super-
NPU, which resolves the bottlenecks with architec-
ture-level solutions.
To make observations and SFQ-specific optimiza-
tions, we conducted performance analyses by running
six CNN workloads (i.e., Alexnet, FasterRCNN, GoogLe-
Net, MobileNet, ResNet50, and VGG16). As input infor-
mation on the fabrication process, we used the
currently available AIST 1.0-mm process in order to show
the SFQ technology’s potential conservatively. In addi-
tion, we assumed a memory bandwidth of 300 GB/s,
which is the typical value of HBM used in the recent
TPUv2.10 Note that the estimated area of the baseline
SFQ-based NPU design might be comparable to the TPU
core ( < 330 mm2 ) if JJs are equivalently scaled to 28 nm
because CMOS technology is used in the TPU design.
Architectural Bottlenecks and Design
Implications
We identified performance bottlenecks and design
implications by conducting analyses with the baseline
SFQ-based NPU design in the “Baseline SFQ-Based
NPU Design” section (called the Baseline from here on).
To show the design implications, we started from the
Baseline by following the TPU core’s architectural speci-
fications (e.g., number of PEs, on-chip memory capacity)
by focusing on their similar hardware structures.11
Our performance analyses identified two architec-
ture-level performance bottlenecks. First, we found
that the data movement overhead among different
on-chip buffers (or within a single buffer) can signifi-
cantly degrade performance. As the Baseline uses
shift-register-based on-chip buffers, it should con-
sume a huge amount of cycles corresponding to the
IEEE Micro 23
TOP PICKS
FIGURE 4. Architectural optimization summary. (a) Overview of SuperNPU with three architectural optimizations. (b) Perfor-
mance evaluation result. (d) Power efficiency evaluation result normalized to the efficiency of the TPU.
buffers’ length for moving the data from each buffer’s
tail to head. Second, we found that the PEs are signifi-
cantly underutilized because the SFQ computing units
are too fast compared with the slow off-chip memory
access. If these bottlenecks are not resolved, even our
SFQ-friendly baseline design with 52.6-GHz clock fre-
quency cannot outperform a conventional CMOS
design [Figure 4(b)]. Therefore, we should minimize the
wasteful data movements while maximizing the PE uti-
lization at the architectural level.
SuperNPU: SFQ-Optimal NPU
Architecture
In light of the above implications, we devised our SFQ-
optimal NPU architecture, SuperNPU. Figure 4(a)
shows an overview of SuperNPU, driven by three SFQ-
specific architectural optimizations: buffer optimiza-
 
tion ( 1 ), resource balancing ( 2 ), and increasing the

number of registers in PE ( 3 ). We briefly introduce our
optimization techniques and their performance
impact here; interested readers should refer to our
original paper for more details.12
First, we almost completely eliminated the data
movement overhead among the on-chip buffers by
optimizing the buffer architecture. Specifically, we
divided each on-chip buffer into small buffer chunks
(256 and 64 divisions for ofmap and ifmap buffers) and
connected them with a multiplexer and demultiplexer.
The optimized buffer architecture eliminates unneces-
sary data movements and improves buffer utilization.
As a result, the Baseline’s performance was signifi-
cantly improved (by 19 times), as shown in Figure 4

(b) 1 .
Next, we efficiently narrowed the gap between the
computation and memory speeds by increasing the on-
24 IEEE Micro
chip buffer capacity while reducing the number of PEs

[Figure 4(a) 2 ]. Our key idea for this design choice is to
reduce off-chip memory accesses by sacrificing exces-
sive computation speed without a performance loss. By
using resource balancing, we can increase each work-
load’s computational intensity (i.e., the number of MAC
operations executed with one weight data mapped on
the PE) by increasing the batch size without additional
off-chip memory accesses. As a result, we improved

performance a further 2.1 times [Figure 4(b) 2 ].
OUR SUPERNPU OUTPERFORMS THE
BASELINE BY 52 TIMES AT A 52.6-GHZ
CLOCK FREQUENCY WHILE CLEARLY
SHOWING THE RIGHT DIRECTIONS
FOR ARCHITECTURE-LEVEL
OPTIMIZATIONS.
Finally, to increase the PE utilization, we increased
the number of weight registers in each PE. With the
larger number of local weight registers, SuperNPU
achieves higher PE utilization by filling several PE pipe-
line stages with a single input data. For example, if
each PE holds two different weights from different

weight filters, as in Figure 4(a) 3 , PE can compute two
different MAC operations with one ifmap pixel. As a
result, we get an additional performance improvement

of 1.3 times [Figure 4(b) 3 ].
In summary, our SuperNPU outperforms the Baseline
by 52 times at a 52.6-GHz clock frequency while clearly
showing the right directions for architecture-level optimi-
zations. Besides, in our performance evaluation,
May/June 2021

SuperNPU outperformed TPU by 23 times, even with its
immature device technology (i.e., 1-mm niobium process).
Meanwhile, our power–efficiency evaluation [Figure 4(c)]
showed that SuperNPU with ERSFQ technology achieves
490 times higher power efficiency without considering
cooling power requirements. Even with the enormous
cost of cooling to 4K (i.e., 400 times the power consump-
tion of the device), SuperNPU attains 1.23 times higher
power efficiency compared with TPU.
IMPACTS AND PROSPECTIVE
The major impacts of our work are as follows.
› Our validated modeling framework for SFQ logic
enables architects to conduct fast and accurate
SFQ architecture design space explorations. It
can be used in other SFQ-promising domains.
The model will encourage researchers to study
SFQ architectures.
› The SFQ-based hardware design was optimized
at the microarchitecture and architecture levels.
The optimization process for NPU provided sev-
eral crucial insights, e.g., on the timing adjust-
ment of SFQ pulses at the circuit level and on
the increasing buffer and pipeline utilization.
These insights will guide the designs of follow-up
SFQ-based architectures.
› Our work shows the true potential of SFQ logic by
evaluating an SFQ-based architecture in compari-
son with its state-of-the-art CMOS counterpart.
With our model-driven analysis and optimization,
SuperNPU outperforms TPU by 23 times on aver-
age while achieving 1.23 times higher power effi-
ciency, even including the enormous cooling cost
at 4K. These significant results will motivate indus-
try and academia to work on SFQ technology to
prepare for the post-Moore era.
Meanwhile, there are critical future challenges
regarding scalability that have to be addressed before
SFQ computing platforms come into practical use.
› Scale up: Although our work shows the potential
for high performance even with immature device
technology (i.e., 1.0 mm niobium process), advances
in device integration technology are essential for
constructing sophisticated computer systems. In
addition to device shrinkage as with conventional
CMOS technology, a 3D-stacked SFQ design
would be promising due to the ultra-low-power
feature of superconductor logic devices. Besides,
manually adjusting the timing of SFQ pulses on
May/June 2021
TOP PICKS
picosecond order is a challenging problem. Thus,
placement and routing automation is indispens-
able to design large-scale SFQ circuits.
› Scale out: The superconductor transmission line
in the circuit enables low-latency lossless signal
propagation since it does not need charge/dis-
charge processes. It can be used for both on-
chip and off-chip communications, which means
that SFQ circuits will be suitable for large-scale
multichip architectures. Therefore, SFQ comput-
ing platforms have a potential for continuous
growth independently of device shrinkage.
ACKNOWLEDGMENTS
This work was supported in part by the JST-Mirai Program
under Grant JPMJMI18E1; in part by JSPS KAKENHI under
Grant JP19H01105, Grant JP18H05211, and Grant
JP18J21274; and in part by the National Research Founda-
tion of Korea (NRF) under Grant NRF-2019R1A5A1027055
and Grant NRF-2021R1A2C3014131. The circuit was
designed with the support by VDEC of the University of
Tokyo in collaboration with Cadence Design Systems,
Inc., and was fabricated in the CRAVITY of AIST.
REFERENCES
1. H. Esmaeilzadeh, E. Blem, R. St. Amant, K. Sankaralingam,
and D. Burger, “Dark silicon and the end of multicore
scaling,” in Proc. 38th Annu. Int. Symp. Comput. Archit.,
2011, pp. 365–376, doi: 10.1109/77.80745.
2. K. K. Likharev and V. K. Semenov, “RSFQ logic/memory
family: A new Josephson-junction technology for sub-
terahertz-clock-frequency digital systems,” IEEE Trans.
Appl. Supercond., vol. 1, no. 1, pp. 3–28, Mar. 1991, doi:
10.1145/2000064.2000108.
3. G. Tzimpragos et al., “A computational temporal logic
for superconducting accelerators,” in Proc. 24th Int.
Conf. Archit. Support Program. Lang. Oper. Syst., 2020,
pp. 435–448, doi: 10.1145/3373376.3378517.
4. I. Nagaoka, M. Tanaka, K. Inoue, and A. Fujimaki, “A
48GHz 5.6mW gate-level-pipelined multiplier using
single-flux quantum logic,” in Proc. IEEE Int. Solid- State
Circuits Conf., 2019, pp. 460–462, doi: 10.1109/
ISSCC.2019.8662351.
5. K. Ishida et al. “32 GHz 6.5 mW gate-level-pipelined 4-bit
processor using superconductor single-flux-quantum
logic,” in Proc. IEEE Symp. VLSI Circuits, 2020, pp. 1–2,
doi: 10.1109/VLSICircuits18222.2020.9162826.
6. D. E. Kirichenko, S. Sarwana, and A. F. Kirichenko, “Zero
static power dissipation biasing of RSFQ circuits,” IEEE
Trans. Appl. Supercond., vol. 21, no. 3, pp. 776–779, Jun. 2011,
doi: 10.1109/TASC.2010.2098432.
IEEE Micro 25
TOP PICKS
7. E. Fang and T. V. Duzer, “A Josephson integrated circuit
simulator (JSIM) for superconductive electronics
application,” in Proc. Extended Abstr. Int. Supercond.
Electron. Conf., 1989, pp. 407–410. [Online]. Available:
https://ci.nii.ac.jp/naid/10006481720/
8. Y. Yamanashi et al., “100 {GHz} demonstrations based
on the single-flux-ouantum cell library for the 10 kA/
2
cm Nb multi-layer process,” IEICE Trans. Electron., vol.
93, no. 4, pp. 440–444, 2010.
9. O. A. Mukhanov, “Energy-efficient single flux quantum
technology,” IEEE Trans. Appl. Supercond., vol. 21, no. 3,
pp. 760–769, Jun. 2011, doi: 10.1109/TASC.2010.2096792.
10. “Hot Chips 2017: A. Closer Look At Google’s TPU v2,”
[Online]. Available: https://www.tomshardware.com/
news/tpu-v2-google-machine-learning,35370.html
11. N. P. Jouppi et al., “In-datacenter performance analysis
of a tensor processing unit,” in Proc. 44th Annu. Int.
Symp. Comput. Archit., 2017, pp. 1–12, doi: 10.1145/
3079856.3080246.
12. K. Ishida et al., “SuperNPU: An extremely fast neural
processing unit using superconducting logic devices,”
in Proc. 53rd Annu. IEEE/ACM Int. Symp. Microarchit.,
2020, pp. 58–72, doi: 10.1109/MICRO50266.2020.00018.
KOKI ISHIDA is currently working at Kyushu University. His
research interests include computer system architecture using
superconductor single flux quantum logic. Ishida received a
Ph.D. degree from Kyushu University in 2021. He is one of the co-
first authors of this article. Contact him at koki.ishida@cpc.ait.
kyushu-u.ac.jp.
ILKWON BYUN is currently working toward a Ph.D. degree
with the Department of Electrical and Computer Engineering,
Seoul National University, Seoul, South Korea. His research
focuses on architecting cryogenic CMOS and superconduc-
tor-based computer systems by using computer architecture
modeling and simulation techniques. He is a Student Member
of IEEE and ACM. He is one of the co-first authors of this
article. Contact him at ik.byun@snu.ac.kr.
IKKI NAGAOKA is currently working toward a Ph.D. degree
with the Department of Electronics, Nagoya University,
Nagoya, Japan. His research interests include designing LSIs
using the superconductor single-flux quantum logic. Contact
him at nagaoka@super.nuee.nagoya-u.ac.jp.
KOSUKE FUKUMITSU is currently working toward an M.E.
degree with the Department of Advanced Information and Tech-
nology, Kyushu University, Fukuoka, Japan. His research interests
include designing the superconductor single-flux-quantum cir-
cuits. Contact him at kosuke.fukumitsu@cpc.ait.kyushu-u.ac.jp.
26 IEEE Micro
MASAMITSU TANAKA is currently an Assistant Professor with
the Department of Electronics, Graduate School of Engineering,
Nagoya University, Nagoya, Japan. His research interests
include ultra-fast/energy-efficient computing using the SFQ-
based technology and logic design methodologies. He is a
senior member of the IEEE. Contact him at masami_t@ieee.org.
SATOSHI KAWAKAMI is currently an Assistant Professor
with the Department of Advanced Information and Tech-
nology, Kyushu University, Fukuoka, Japan. His research
interests include computer architecture with emerging
devices. He is a Member of IEEE and ACM. Contact him at
satoshi.kawakami@cpc.ait.kyushu-u.ac.jp.
TERUO TANIMOTO is currently an Assistant Professor at the
Research Institute for Information Technology, Kyushu Uni-
versity, Fukuoka, Japan. His research interests include edge
computing systems, secure computer architecture, and
quantum computer system architecture. He is a Member of
IEEE and ACM. Contact him at tteruo@kyudai.jp.
TAKATSUGU ONO is currently an Associate Professor with the
Department of Advanced Information and Technology, Kyushu
University, Fukuoka, Japan. His research interests include the
areas of computer architecture with particular emphasis on
secure computing, high-performance computing, and memory
system (including nonvolatile memory). Ono received a Ph.D.
degree from Kyushu University in 2009. He is a Member of IEEE.
Contact him at takatsugu.ono@cpc.ait.kyushu-u.ac.jp.
JANGWOO KIM is currently a Professor with the Department
of Electrical and Computer Engineering, Seoul National Uni-
versity, Seoul, South Korea. His primary research interests lie
in computer architecture, server and datacenter, system
modeling, and intelligent systems. Kim received a Ph.D.
degree in electrical and computer engineering from Carnegie
Mellon University in 2008. He is a Member of IEEE and ACM.
Contact him at jangwoo@snu.ac.kr.
KOJI INOUE is a Professor with the Department of Advanced
Information and Technology, Kyushu University, Fukuoka,
Japan. He is currently the Director of System LSI Research
Center. His research interests include computer architecture
and low-power system designs. Inoue received a Ph.D. degree
from Kyushu University. He is a Member of IEEE and ACM. He
is the corresponding author of this article. Contact him at
inoue@ait.kyushu-u.ac.jp.
May/June 2021
THEME ARTICLE: TOP PICKS

Leaking Secrets Through Compressed

Caches
Po-An Tsai , NVIDIA, Santa Clara, CA, 95051, USA
Andres Sanchez

, Ecole
de
Polytechnique Fe
rale de Lausanne, 27218, Lausanne, Switzerland
Christopher W. Fletcher , University of Illinois at Urbana-Champaign, Champaign, IL,14589, USA
Daniel Sanchez , Massachusetts Institute of Technology, Cambridge, MA, 02139, USA

We offer the first security analysis of cache compression, a promising architectural

technique that is likely to appear in future mainstream processors. We find that
cache compression has novel security implications because the compressibility of a
cache line reveals information about its contents. Compressed caches introduce a
new side channel that is especially insidious, as simply storing data transmits
information about the data. We present two techniques that make attacks on
compressed caches practical. Pack+Probe allows an attacker to learn the
compressibility of victim cache lines, and Safecracker leaks secret data efficiently
by strategically changing the values of nearby data. Our evaluation on a proof-of-
concept application shows that, on a representative compressed cache
architecture, Safecracker lets an attacker compromise an 8-byte secret key in
under 10 ms. Even worse, Safecracker can be combined with latent memory safety
vulnerabilities to leak a large fraction of program memory.

O
ver the past three years, computer architec-
ture has suffered a major security crisis.
Researchers have uncovered critical security
flaws in billions of deployed processors related to specu-
lative execution, starting with Spectre1 and Meltdown,2
generating significant interest and progress in micro-
architectural side and covert channel research.
While microarchitectural side channel attacks have
been around for over a decade, speculative execution
attacks are significantly more dangerous because of
their ability to leak program data directly. In the worst
case, these attacks let the attacker construct a univer-
sal read gadget,3 capable of leaking data at attacker-
specified addresses. For example, the Spectre V1
attack—if (i < N) { B[A[i]]; }—exploits branch mispre-
diction to leak the data at address &A + i given an
attacker-controlled i.
Yet, speculative execution is only one optimization
in modern microprocessors. It is critical to ask: Are
there other microarchitectural optimizations that
0272-1732 ß 2021 IEEE
Digital Object Identifier 10.1109/MM.2021.3069158
Date of publication 26 March 2021; date of current version
25 May 2021.
enable a similarly large amount of data leakage? In
this work, we provide an answer in the affirmative by
analyzing the security of memory hierarchy compres-
sion, specifically cache compression.
Compression is an attractive technique to improve
memory performance and has received intense devel-
opment from both academia and industry. Some early
adopter processors already feature memory-hierarchy
compression, including IBM’s z15,4 Qualcomm’s Cen-
triq, and NVIDIA’s A100. As data movement becomes
increasingly critical, we expect to see general-purpose
cache compression become widely used. Nonetheless,
despite strong interest from both academia and indus-
try, prior research in this area has focused on perfor-
mance and ignored security.
We present the first security analysis of cache com-
pression. The key insight that our analysis builds on is
that the compressibility of data reveals information
about the data itself. Similar to speculative execution
attacks, we show how this allows an attacker to leak
program data directly and, in the worst case, create a
new universal read gadget that can leak large portions
of program memory. In short, we show that cache com-
pression—without speculative execution—can leak as
much program privacy as speculative execution.

May/June 2021 Published by the IEEE Computer Society IEEE Micro 27

TOP PICKS
FIGURE 1. Overview of our compressed cache attacks. (a) Simple attack on a compressed cache, where the attacker exploits
colocation with secret data to leak it. (b) Comparison between Spectre and our proposed attacks.
CACHE COMPRESSION
INTRODUCES A NEW CHANNEL
Figure 1(a) shows a simple attack on compressed
caches. The attacker seeks to steal the victim’s
encryption key and can submit encryption requests to
the victim. On each request, the victim’s encryption
function stores the key and the attacker’s plaintext
consecutively, so they fall on the same cache line.
THE KEY INSIGHT THAT OUR
ANALYSIS BUILDS ON IS THAT THE
COMPRESSIBILITY OF DATA
REVEALS INFORMATION ABOUT
THE DATA ITSELF.
Colocating secret data and attacker-controlled data
is safe with conventional caches, but it is unsafe with a
compressed cache. Suppose we run this program on a
system with a compressed cache that tries to shrink
each cache line by removing duplicate bytes. If the
attacker can observe the line’s size, it can leak all
individual bytes of the key by trying different cho-
sen plaintexts, as the compressed line’s size
28 IEEE Micro
changes when a byte of the key matches a byte of
the plaintext.
The general principle in the above mentioned exam-
ple is that when the attacker is able to colocate its own
data alongside secret data, it can learn the secret data.
Beyond cases where the victim itself facilitates coloca-
tion (e.g., by pushing arguments onto the stack),
we observe that latent security vulnerabilities related
to memory safety, such as buffer overflows, heap
spraying, and uninitialized memory, further enable the
attacker to colocate its data with secret data. Com-
bined with the new side channel in compressed
caches, these can enable a read gadget that leaks a
significant amount of data, as we will show later.
CACHE COMPRESSION ENABLES
A NEW TYPE OF ACTIVE ATTACK
All compression techniques seek to store data effi-
ciently by using a variable-length code, where the
length of the encoded message approaches the infor-
mation content, or entropy, of the data being encoded.
It trivially follows that the compressibility of a data
chunk, i.e., the compression ratio achieved, reveals
information about the data.
Hence, compressed caches introduce a new, fun-
damentally different type of side channel. As a point
May/June 2021
 TOP PICKS

FIGURE 2. Comparison of VSC and conventional caches and example showing how Pack+Probe measures compressibility on
VSC. (a) Comparison of VSC (right) versus an uncompressed set-associative cache (left). VSC divides each set of the data array
into small segments (8 bytes in this example), stores each variable-sized line as a contiguous run of segments in the data array,
modifies tags to point to the blocks’ data segments, and increases the number of tags per set relative to the uncompressed
cache (by 2
in this example) to allow tracking more, smaller lines per set. (b) Simplified example of the Pack+Probe attack on
VSC. The attacker first packs the target set to leave exactly S segments left. After the victim accesses the secret, the attacker
probes the set to see if the compressed size of the secret is larger than S segments.

of comparison, consider conventional cache-based
side channels. Conventional cache channels are
based on the presence or absence of a line in the
cache. Thus, conventional attacks make a strong
assumption, namely that the victim is written in a way
that encodes the secret as a load address.
Compressed cache attacks relax this assumption.
Compressed cache channels are based on data com-
pressibility in the cache. Data can leak regardless of how
the program is written, just based on what data is written
to memory. In this sense, our attacks on compressed
caches are more similar to Spectre and the recent Ram-
bleed5 attacks than conventional side-channel attacks.
Figure 1(b) compares Spectre and the new attacks in
this article, using an abstract view of a side-channel
attack from prior work.6 Attacker and victim reside in dif-
ferent protection domains, so the attacker resorts to a
side channel to extract the secret from the victim. To
exploit the side channel, a transmitter in the victim’s pro-
tection domain encodes the secret into the channel,
which is read and interpreted by a receiver in the attack-
er’s protection domain.
Spectre attacks allow the attacker to create differ-
ent transmitters by arranging various sequences of
mispredicted speculations. Analogously, attacks
based on cache compression allow the attacker to
create different transmitters by writing different data
into the cache. Exploiting the compressed cache side
channel requires a new receiver. In Spectre, the
receiver uses techniques like Prime+Probe to detect
the timing difference due to a line’s presence. In com-
pressed cache attacks, the receiver has to detect the
compressibility information from the channel. To this
end, we present Pack+Probe, a general technique that
leverages the timing difference due to a line’s pres-
ence to also infer its compressibility.
PACK+PROBE: MEASURING
COMPRESSIBILITY
Whereas conventional caches manage fixed-size cache
lines, compressed caches manage variable-sized lines.
Thus, compressed caches divide the data array among
variable-sized blocks and track their tags in a way that 1)
enables fast lookups and insertions; 2) allows high com-
pression ratios; and 3) avoids high tag storage overheads.
While prior work has proposed various compressed cache
architectures, Pack+Probe is general and applies broadly.
For concreteness, we explain it using a commonly used
organization, Variable-Sized Cache (VSC).7
VSC extends a set-associative design to store
compressed, variable-sized cache lines. Figure 2(a)

May/June 2021 IEEE Micro 29

TOP PICKS
FIGURE 3. Example of the Safecracker attack on BDI. The attacker fills the cache line with attacker-controlled data and brute-
forces the first 2 bytes in every 4 bytes until the guess is close to the secret value and incurs compression. The change in the
line size can be observed using Pack+Probe.
illustrates VSC and compares it with a set-associative
design. VSC divides each set of the data array into small
segments [8 bytes in Figure 2(a)]. It stores each variable-
size line as a contiguous run of segments in the data
array. Each tag includes a pointer to identify the block’s
data segments within the set, and VSC increases the
number of tags per set relative to the uncompressed
cache [e.g., by 2
in Figure 2(a)] to support smaller lines.
Pack+Probe exploits that, in compressed caches, a
victim’s access to a cache line may cause different
evictions of other lines depending on the compress-
ibility (i.e., compressed size) of the victim’s line, in
addition to the replacement policy. Since the com-
pressed cache can store variable-sized lines, whether
a newly installed line evicts other lines is determined
by the unused capacity in the compressed cache.
Figure 2(b) shows a single step of Pack+Probe. Pack
+Probe first packs the compressed cache with
attacker-controlled lines to leave exactly S segments
unused. Once the victim accesses the secret line, if its
compressed size is S segments, no evictions will hap-
pen, whereas if it is larger than S segments, at least one
of the attacker-controlled lines will be evicted. Finally,
the attacker probes (accesses) the lines it inserted
again, uses timing differences to infer which lines hit or
miss, and thus infers whether the victim’s line fits within
X bytes. Repeating these steps with a simple binary
search over values of X, the attacker can precisely
determine the compressed size of the victim’s line.
Our full paper8 shows Pack+Probe on VSC takes
less than 10 K cycles to determine the compressed
size of the victim’s line. Pack+Probe’s efficiency allows
us to further develop Safecracker, an active attack to
leak secret data.
SAFECRACKER: LEARNING
SECRETS THROUGH COLOCATION
Safecracker is named after the process used to crack
combination locks in (classic) safes, where the attacker
30 IEEE Micro
cycles through each digit of the combination and listens
for changes in the lock that signal when the digit is cor-
rect. Similarly, Safecracker provides a guess and learns
indirect outcomes (compressibility) of the guess. The
attacker then uses the outcome to guide the next guess.
In the context of compressed caches, the attacker
first makes a guess about the secret data and then builds
a data pattern that, when colocated with the secret data,
will cause the cache line to be compressed in a particular
way if the guess is correct [like in Figure 1(a)]. By measur-
ing the line’s compressibility using Pack+Probe, the
attacker knows whether the guess was correct.
Depending on the compression algorithm used, Safe-
cracker needs different search strategies. We showcase
and implement the Safecracker attack on the base-
delta-immediate (BDI) compression algorithm,9 a simple
and common baseline algorithm that relies on delta
encoding. The BDI algorithm performs intra-cache-line
compression by storing a common base value and small
deltas. For example, for eight 4-byte integer values rang-
ing from 1,280 to 1,287, BDI will store a 4-byte base of
1,280, and eight 1-byte values from 0 to 7, compressing a
32-byte cache line into a 12-byte compressed line.
Depending on the base value and the ranges of the del-
tas, BDI compresses a cache line into eight different
sizes. For example, a 4-byte base and 16 2-byte values
result in a 20-byte compressed line.
Figure 3 shows how our Safecracker attack exploits
BDI. Assume the attacker wishes to steal a 4-byte secret,
located in a cache line where all other data, 28 bytes in
this example, is attacker controlled [e.g., it is part of a
request buffer like in Figure 1(a)]. Safecracker starts by
targeting a compressed pattern of a 4-byte base and
2-bytes values (a 20-byte line). It brute-forces the first
2 bytes of the victim’s word by changing the first 2 bytes
of attacker-controlled words (i.e., trying patterns
0x00000000,0x00010000,{. . .}0xFFFF0000, at most 216 guesses),
and uses Pack+Probe to see if the cache line compresses
to the size of a 20-byte line. Since the secret in this
May/June 2021

TABLE 1. Worst-case time for Safecracker to leak secrets of different sizes.
Attack 1 bytes 2 bytes 3 bytes
Safecracker 0.11 0.30 1.05
Safecracker+ buffer overflow 0.76 1.53 2.30
example is 0x0F00BA20, when the attacker guesses
0x0F000000, the line compresses into a 20-byte line, and the
attacker records the 2 bytes it tried as part of the secret.
The attacker targets the next compressed pattern, a 4-
byte base and 1-byte values, and brute-forces the third
byte (taking at most 28 guesses), and then brute-forces
the last byte to learn all 4 bytes of the secret.
Our ASPLOS’20 paper8 details the algorithm to
steal secrets in various sizes on BDI, and we also dis-
cuss how to extend the idea to other cache compres-
sion algorithms. Safecracker on BDI can leak up to
8 bytes of secret data, which can be devastating. For
example, a 128-bit AES key is considered secure, but
leaking 64 bits degrades it to 64-bit protection, which
is insecure in many contexts.
Enhancing Safecracker With Latent
Memory-Safety Violations
The previous example assumes that the attacker-con-
trolled data (OATD) is located right next to the secret
in the same cache line. This limits the attacker to only
leak contents contiguous to attacker-controlled data.
However, if the attacker can find other latent memory-
safety violations in the victim program, then this vul-
nerability can significantly increase the amount of data
leaked and enhance the efficiency of Safecracker.
For example, combining Safecracker with buffer
overflows, the attacker can control where the attacker-
controlled OATD is located. In the worst case, if the
compression algorithm allows leaking up to X bytes per
line, memory-safety violations let the attacker leak X
bytes from every cache line. That is, if the victim has a
memory footprint of M, then Safecracker with a buffer
overflow can leak OðMÞ bytes of memory, where differ-
ent compression algorithms have different constant
factors (e.g., 8=64 ¼ 1=8th of program memory for BDI).
Moreover, buffer overflows also allow the attacker to
control how many bytes the attacker-controlled data
consists of. Leveraging this, the attacker can leak the
secret much more efficiently by making all partial
guesses at byte granularity. The attacker can allocate a
buffer that leaves only one byte of secret data in the
line. By brute-force, the attacker quickly learns this
remaining byte in the same manner as the previous
example. Once the last byte is known, the attacker
learns the second-to-last byte by allocating a smaller
May/June 2021
TOP PICKS
Time (ms) to leak secret of size
4 bytes 5 bytes 6 bytes 7 bytes 8 bytes
54.0 54.4 256 - -
3.11 3.91 4.32 5.61 6.46
buffer that does not overwrite it and brute-forcing only
this byte. This requires the victim to restore the data
over multiple invocations, which is the case with local,
stack-allocated variables. By repeating these steps, the
attacker can learn the 8 bytes of secret data even faster.
SAFECRACKER LEAKS SECRETS
EFFICIENTLY
We evaluate the effectiveness of Safecracker using
proof-of-concept workloads (PoC) and architectural sim-
ulation (simulating a compressed cache with VSC+BDI).
The first PoC has two separate processes, victim and
attacker. The victim is a login server with a vulnerability
that lets attacker-controlled input be stored next to a
secret key. The attacker can provide input to the cache
line where the key is located, without modifying the key.
The attacker can also invoke victim accesses to the
secret by issuing encryption requests that use the secret
key. This lets the attacker perform Pack+Probe.
The attacker first finds the set that holds the secret
cache line using standard Prime+Probe.10 Once the
conflicting set is found, the attacker uses Safecracker
to steal the victim’s secret key.
Table 1 reports the worst-case execution time needed
to steal different numbers of bytes. Safecracker requires
less than a second to crack a 6-byte secret value. Though
Safecracker can steal up to 8 bytes when applied to BDI,
trying to steal more than 6 bytes requires much longer
runtimes (hours for 8 bytes), because the complexity of
Safecracker on BDI grows exponentially. Nonetheless, as
we explained earlier, Safecracker can be combined with
latent memory violations to enhance its efficiency.
Buffer-Overflow-Based Attack
The second PoC builds on top of the first PoC. How-
ever, this time, the victim has a buffer-overflow vulner-
ability. The vulnerable function is as follows:
void encrypt(char *plaintext) {
char result[LINESIZE];
char data[DATASIZE]; // can be any size
char key[KEYSIZE];
memcpy(key, KEYADDR, KEYSIZE);
strcpy(result, plaintext);
{. . .}
}.
IEEE Micro 31
TOP PICKS
The buffer overflow stems from the unsafe call to
strcpy, which causes out-of-bounds writes when the
plaintext input string exceeds LINESIZE bytes. The
attacker exploits this buffer overflow to scribble over
the stack and overwrite some of the bytes of the key.
After encrypt returns, the scribbled-over line remains in
the stack, and the attacker is then able to measure its
compressibility with Pack+Probe and to run Safe-
cracker as described before.
Buffer overflows give Safecracker much higher
bandwidth by allowing it to guess a single byte on
each step. Using a buffer overflow, Safecracker steals
8 bytes of secret data in under 10 ms. Table 1 describes
that attack time with a buffer overflow grows linearly
versus exponentially as it had with the first PoC.
While Safecracker applied to BDI can steal only
8 bytes per line, the above mentioned process can be
repeated for different-sized attacker-controlled buf-
fers to steal data in other lines, e.g., 8 bytes from multi-
ple (potentially all) lines.
GENERALIZATION AND DEFENSES
Our ASPLOS’20 paper8 also discusses how to generalize
these attacks to other compressed cache architectures
and algorithms, and presents other opportunities to
colocate attacker-controlled data with secret data. We
find that 1) Pack+Probe is applicable to most com-
pressed cache architectures with a decoupled tag store
with extra tags, and a data array divided in fixed-size sets
where variable-sized blocks are laid over; 2) variants of
Safecracker can be constructed case by case for other
compression algorithms, and the better the algorithm
compresses, the more information it can leak; and 3)
there are many more ways to colocate attacker-con-
trolled data near sensitive data, both spatially (e.g., heap
spraying) and temporally (e.g., uninitialized data), and
widely used software, such as the Linux kernel, suffers
from these attack vectors.
Finally, we present multiple ways (e.g., obfuscation)
to defend against Pack+Probe, Safecracker, and other
attacks on compressed caches. We evaluate one of
them, partitioning the compressed cache, to understand
the tradeoff between security and performance. Our
analysis shows that even though it is possible to make
compressed caches secure, the straightforward solution
that partitions both the tag and data array comes at a
cost. How to limit this performance impact while mini-
mizing leakage would be interesting future work.
LESSONS LEARNED
We have presented the first security analysis of cache
compression and found that cache compression is
insecure because the compressibility of a cache line
reveals information about its contents.
32 IEEE Micro
With our proposed attacks, Pack+Probe and Safe-
cracker, we have also shown how cache compression
can potentially leak as much program privacy as spec-
ulative execution. This has significant implications for
architects. It suggests we as a community need to
revisit our vast literature and reexamine other micro-
architectural optimizations through a security lens.
Beyond the above mentioned immediate and long-
term message, our paper makes two other microarchi-
tectural side channel “firsts.”
The first work to show that memory-safety violations
can enhance microarchitectural attacks: Memory safety
vulnerabilities, e.g., buffer overflows, and microarchic-
tural attacks are both prominent vulnerability classes in
their own right. Fortunately, however, we have tradition-
ally been able to treat them as orthogonal concerns with
orthogonal sets of attacks and defenses.
In this work, we show that memory-safety violations
can enhance microarchitectural attacks. In other
words, memory safety vulnerabilities can be used to
mount (and exacerbate) microarchitectural attacks.
Worse, defenses against a given memory-safety vulner-
ability are not sufficient to block its “mirror image” in
the microarchitectural-attack world. Our original paper
illustrates this idea by showing how a “code-reuse
buffer overflow” defense (StackGuard) is insufficient
for preventing a “microarchitectural-attack buffer over-
flow” which, when combined with compressed caches,
can leak (asymptotically) all of the program memory.
WE HOPE OUR WORK CATALYZES A
LINE OF WORK IN PROACTIVE SECURITY
ANALYSIS OF MICROARCHITECTURAL
OPTIMIZATIONS.
The first data-centric, differential microarchitec-
tural attack: There is a rich heritage in the security
community for performing chosen-plaintext attacks.
In this model, the victim program P takes as input sen-
sitive data S and attacker-controlled data C, and pro-
duces an observation O, i.e., O ¼ ViewðPjS; CÞ. The
attacker more-precisely learns S by varying C and
monitoring changes in O. This style is also called a dif-
ferential attack; it is used to perform cryptanalysis and
amplify traditional side-channel, e.g., DPA, attacks.
This article presents the first data-centric differen-
tial attack in the microarchitectural attack setting.
Specifically, C is attacker-controlled data that colo-
cates with S. We show how by modulating C, the
attacker can perform a guided search to recover S by
observing changes in compressibility O. Making
May/June 2021

matters worse, depending on the cache compression
algorithm, this search can be performed in asymptoti-
cally fewer steps than brute-force guessing S.
We hope this work prevents insecure cache
compression techniques from reaching mainstream
processors. More importantly, we hope our work cata-
lyzes a line of work in proactive security analysis of
microarchitectural optimizations.
ACKNOWLEDGMENTS
We would like to thank Maleen Abeydeera, Joel Emer,
Mark Jeffrey, Anurag Mukkara, Quan Nguyen, Victor
Ying, Guowei Zhang, and the ASPLOS’20 reviewers for
their feedback. We would also like to thank Joel Emer for
his insights on the taxonomy, and Paul Kocher for shar-
ing his concerns about the security of memory compres-
sion. This work was supported in part by the National
Science Foundation under Grant CAREER-1452994 and
Grant SaTC-1816226, in part by the Google faculty
research award, and in part by an Intel ISRA grant. The
work of Andres Sanchez was supported through a MISTI
grant by the Technical University of Madrid.
REFERENCES
1. P. Kocher et al., “Spectre attacks: Exploiting
speculative execution,” in Proc. IEEE Symp. Secur.
Privacy, 2019, pp. 1–19, doi: 10.1109/SP.2019.00002.
2. M. Lipp et al., “Meltdown: Reading kernel memory from
user space,” in Proc. 27th USENIX Secur. Symp., 2018,
pp. 973–990, doi: 10.1145/3357033.
3. R. McIlroy, J. Sevcik, T. Tebbi, B. L. Titzer, and
T. Verwaest, “Spectre is here to stay: An analysis of
side-channels and speculative execution,” CoRR,
vol. abs/1902.05178, 2019. [Online]. Available: http://
arxiv.org/abs/1902.05178
4. B. Abali et al., “Data compression accelerator on IBM
power9 and z15 processors: Industrial product,” in Proc.
47th Annu. Int. Symp. Comput. Archit., 2020, pp. 1–14,
doi: 10.1109/ISCA45697.2020.00012.
5. A. Kwong, D. Genkin, D. Gruss, and Y. Yarom,
“Rambleed: Reading bits in memory without accessing
them,” in Proc. IEEE Symp. Secur. Privacy, 2020,
pp. 695–711, doi: 10.1109/SP40000.2020.00020.
6. V. Kiriansky, I. Lebedev, S. Amarasinghe, S. Devadas,
and J. Emer, “DAWG: A defense against cache timing
attacks in speculative execution processors,” in Proc.
51st Annu. IEEE/ACM Int. Symp. Microarchit., 2018,
pp. 974–987, doi: 10.1109/MICRO.2018.00083.
7. A. R. Alameldeen and D. A. Wood, “Adaptive cache
compression for high-performance processors,” in
Proc. 31st Annu. Int. Symp. Comput. Archit., 2004,
pp. 212–223, doi: 10.1109/ISCA.2004.1310776.
May/June 2021
TOP PICKS
8. P.-A. Tsai, A. Sanchez, C. W. Fletcher, and D. Sanchez,
“Safecracker: Leaking secrets through compressed
caches,” in Proc. 25th Int. Conf. Archit. Support Program.
Lang. Oper. Syst., 2020, pp. 1125–1140, doi: 10.1145/
3373376.3378453.
9. G. Pekhimenko, V. Seshadri, O. Mutlu, P. B. Gibbons,
M. A. Kozuch, and T. C. Mowry, “Base-delta-immediate
compression: Practical data compression for on-chip
caches,” in Proc. 21st Int. Conf. Parallel Archit.
Compilation Techn., 2012, pp. 377–388, doi: 10.1145/
2370816.2370870.
10. D. A. Osvik, A. Shamir, and E. Tromer, “Cache attacks
and countermeasures: The case of AES,” in Proc. 7th
Cryptographers’ Track RSA Conf. Topics Cryptology,
2006, pp. 1–20, doi: 10.1007/11605805_1.
PO-AN TSAI is currently a Research Scientist with NVIDIA,
Santa Clara, CA, USA. His current research focuses on tensor
accelerators design/modeling and memory hierarchy optimiza-
tions for domain-specific accelerators. Tsai received a B.S.
degree in electrical engineering from National Taiwan University,
Taipei, Taiwan, in 2012, and S.M. and Ph.D. degrees in computer
science from MIT, Cambridge, MA, USA, in 2015 and 2019, respec-
tively. He is a Member of IEEE. Contact him at poant@nvidia.com.
ANDRES SANCHEZ is currently working toward a graduate


degree with the Ecole Polytechnique Fe
de
rale de Lausanne,
Lausanne, Switzerland. His research topic is side-channels
detection and mitigation, security-aware systems layering,
compilation techniques, and characterization of sensitive infor-
mation. Sanchez received a B.S. degree in mathematics and
computer engineering from the Technical University of Madrid,
Madrid, Spain, in 2019. Contact him at andres.sanchez@epfl.ch.
CHRISTOPHER W. FLETCHER is currently an Assistant Pro-
fessor in computer science with the University of Illinois at
Urbana-Champaign, Champaign, IL, USA. He has interests
ranging from computer architecture to security to high-perfor-
mance computing (ranging from theory to practice, algorithm
to software to hardware). Fletcher received a Ph.D. degree
from the Massachusetts Institute of Technology, Cambridge,
MA, USA, in 2016. Contact him at cwfletch@illinois.edu.
DANIEL SANCHEZ is currently an Associate Professor with the
Department of Electrical Engineering and Computer Science,
Massachusetts Institute of Technology, Cambridge, MA, USA.
His research interests include scalable memory hierarchies,
architectural support for parallelization, and accelerators for
sparse computations. Sanchez received a Ph.D. degree in electri-
cal engineering from Stanford University, Stanford, CA, USA.
Contact him at sanchez@csail.mit.edu.
IEEE Micro 33
 THEME ARTICLE: TOP PICKS

Understanding Acceleration Opportunities

at Hyperscale
Akshitha Sriraman , University of Michigan, Ann Arbor, MI, 48109, USA
Abhishek Dhanotia , Facebook, Menlo Park, CA, 94025, USA

Modern web services run across hundreds of thousands of servers in a data center,
i.e., at hyperscale. With the end of Moore’s Law and Dennard scaling, successive
server generations running these web services exhibit diminishing performance
returns, resulting in architects adopting hardware customization. An important
question arises: Which web service software operations are worth building custom
hardware for? To answer this question, we comprehensively analyze important
Facebook production services and identify key acceleration opportunities. We then
develop an open-source analytical model, Accelerometer, to help make well-
informed hardware decisions for the acceleration opportunities we identify.

M
odern web services such as social media,
online messaging, web search, video
streaming, and online banking often sup-
port billions of users, requiring data centers that scale
to hundreds of thousands of servers, i.e., hyperscale.
Whereas hyperscale web services once had largely
monolithic software architectures, modern web serv-
ices are composed of numerous independent, special-
ized, distributed microservices (e.g., key-value serving
in a social media service).1,2 Several companies such
as Amazon, Netflix, Gilt, LinkedIn, Facebook, and
SoundCloud have adopted microservice architectures
to improve web service development and scalability.3
While at face value, hyperscale web systems seem
instantaneously available at the touch of a button, exist-
ing microservices barely meet performance require-
ments. In reality, microservices have much more
stringent performance constraints than their monolithic
counterparts, since numerous microservices must be
invoked, often serially, to serve a user’s query. For exam-
ple, a Facebook news feed service query may flow
through a pipeline of many microservices invoked via
remote procedure calls (RPCs), such as 1) Sigma: a
spam filter; 2) McRouter: a protocol router; 3) Feed: a
This work is licensed under a Creative Commons Attribution-
NonCommercial-NoDerivatives 4.0 License. For more infor-
mation, see https://creativecommons.org/licenses/by-nc-nd/
4.0/
Digital Object Identifier 10.1109/MM.2021.3066615
Date of publication 30 March 2021; date of current version
25 May 2021.
news feed stories extractor; 4) Tao: a distributed social
graph data store; and 5) MyRocks: a user database.
Complex microservice interactions place stringent per-
formance constraints on individual microservices.
As hyperscale computing grows to drive more
sophisticated applications (e.g., virtual reality and con-
versational AI), existing microservice systems will face
greater efficiency challenges due to these more com-
plex tasks. Increasingly complex microservices can be
efficiently supported if the hardware rises to meet effi-
ciency requirements. However, with the end of
Moore’s Law and Dennard scaling, a key challenge to
realizing microservice efficiency is that successive
server generations running microservices exhibit
diminishing performance returns.
To improve hardware efficiency, several architects
today work on developing numerous specialized hard-
ware accelerators for important microservice domains
[e.g. machine learning (ML) tasks]. However, large-
scale internet operators have strong economic incen-
tives to limit hardware platform diversity to: 1) main-
tain fungibility of hardware resources; 2) preserve
procurement advantages that arise from economies
of scale; and 3) limit the overhead of developing and
testing on myriad specialized hardware platforms.
Hence, an important question arises: Which microser-
vice operations consume the most CPU cycles and
are worth accelerating?
To build specialized accelerators for these key
microservice operations, it is important to first
identify which type of accelerator meets microser-
vice requirements and is worth designing and

34 IEEE Micro Published by the IEEE Computer Society May/June 2021


deploying. Deploying specialized hardware is risky
at hyperscale, as the hardware might underperform
due to performance bounds from the microservi-
ce’s software interaction with the hardware, result-
ing in high monetary losses. To make well-informed
hardware decisions, it is crucial to answer the
following question early in the design phase of
a new accelerator: How much can the accelerator
realistically improve its targeted microservice
overhead?
To answer both the above questions, our article,4
presented at ASPLOS 2020, first undertakes a compre-
hensive characterization of how microservices spend
their CPU cycles. We study seven important hyper-
scale Facebook microservices in four diverse service
domains that run across hundreds of thousands of
servers, occupying a large portion of Facebook’s global
server fleet. Our detailed breakdown of CPU cycles
consumed by various microservice operations identi-
fies key overheads and potential design optimizations.
To make well-informed hardware decisions for these
microservice overheads, we contribute Accelerometer,
an analytical model that projects realistic microser-
vice speedup for various hardware acceleration strate-
gies. We also demonstrate Accelerometer’s utility
in Facebook’s production microservices via three ret-
rospective case studies conducted when serving live
user traffic.
FIGURE 1. Breakdown of cycles spent in various leaf functions (leaf categories defined in the table to the right): Memory func-
tions consume a significant portion of total cycles.
May/June 2021
TOP PICKS
HOW DO MICROSERVICES SPEND
THEIR CPU CYCLES?
To identify microservice overheads, we comprehen-
sively characterize how seven important Facebook
production microservices spend their CPU cycles
when serving live user traffic. Very few prior works
study how cycles are spent in data centers. Kanev
et al.1 investigate the “data center tax” across Google’s
server fleet by studying cycles spent in seven types of
leaf functions invoked at the end of a call trace [e.g.,
memcpy()]. However, a leaf function study alone does not
holistically provide insight into whether acceleration
might improve a microservice functionality (e.g.,
encryption).
To analyze microservice functionalities, we must
comprehensively characterize a microservice’s entire
call stack to measure the CPU cycles spent in each
phase of the microservice’s operation after it receives
a request. Characterizing microservice functionalities
helps determine 1) whether diverse microservices exe-
cute common types of operations (e.g., compression,
serialization, and encryption) and 2) the overheads
such operations induce. Analyzing both leaf functions
and microservice functionalities helps identify key
acceleration opportunities that might inform future
software and hardware designs.
We characterize the CPU cycles spent by Face-
book’s production microservices in both leaf functions
IEEE Micro 35
TOP PICKS

and microservice functionalities. We study 1) Web: a

front-end microservice that implements PHP and
Hack; 2) Feed1 and Feed2: news feed microservices that
aggregate, rank, and display stories; 3) Ads1 and Ads2:
advertisement microservices that compute user-spe-
cific and ad-specific data; and 4) Cache1 and Cache2: large
distributed-memory object caching microservices.
FIGURE 2. Breakdown of cycles spent in the main application
logic versus orchestration work: Orchestration overheads sig-
Leaf Function Characterization nificantly dominate.
We present key leaf function breakdowns for Face-
book’s microservices in Figure 1, comparing them with
Google’s services1 and SPEC CPU2006 benchmarks.5
their performance. Cache2 also spends significant cycles
We find that many leaf function overheads are signifi-
in I/O and network interactions, and can benefit from
cant and common across microservices. We detail our
optimizations such as kernel-bypass and multiqueue
observations for dominant leaf categories.
NICs.
Memory. Most microservices spend a significant
Synchronization. Microservices such as Cache over-
fraction of cycles on memory functions that include
subscribe threads to improve throughput.8 Hence,
memory copy, free, allocation, move, set, and com-
such microservices spend significant cycles synchro-
pare. Memory copies are by far the greatest consum-
nizing frequent communication between distinct
ers of memory cycles. Data is primarily copied during
thread pools. Cache also spends a large fraction of
microservice operations such as 1) I/O pre- or
cycles in spin locks that are typically deemed perfor-
postprocessing, 2) I/O sends and receives, 3) RPC seri-
mance inefficient.9 However, Cache implements spin
alization/deserialization, and 4) microservice business
locks since it is a microsecond-scale microservice,8
logic execution (e.g., executing key-value stores in
and is hence more prone to microsecond-scale perfor-
Cache).
mance penalties that can otherwise arise from thread
We observe significant diversity in dominant service
re-scheduling, wakeups, and context switches.7
functionalities that invoke memory copies across micro-
C Libraries. We observe that Feed2, Ads1, and Ads2
services. This diversity suggests a strategy to specialize
invoke C libraries for vector operations, as they deal
copy optimizations to suit each microservice’s distinct
with large ML feature vectors. Web spends significant
needs. For example, Web can benefit from reducing copies
cycles parsing and transforming strings to process
in I/O pre- or postprocessing, whereas Cache2 can gain
queries from many URL endpoints. Interestingly, unlike
from fewer copies in network protocol stacks.
memory or kernel leaf functions, C libraries’ instruc-
Freeing memory incurs a high overhead for several
tions per cycle (IPC) scales well across CPU genera-
microservices, as the free() function does not take a
tions, as many hardware vendors primarily rely on
memory block size parameter, performing extra work
open-source SPEC benchmarks that heavily use C
to determine the size class to return the block to.
libraries to make architecture design decisions.
TCMalloc performs a hash lookup to get the size class.
Other observations. ML microservices such as Ads2
This hash tends to cache poorly, especially in the TLB,
and Feed2 spend only up to 13% of cycles on mathemati-
leading to performance losses. Although C++11 ameli-
cal operations that constitute ML inference using mul-
orates this problem by allowing compilers to invoke
tilayer perceptrons. Cache2 spends 6% of cycles in leaf
delete() with a parameter for memory block size, over-
encryption functions since it encrypts a high number of
heads still arise from 1) removing pages faulted in
queries per second (QPS). Additionally, Google’s break-
when memory was written to and 2) merging neighbor-
down for a few leaf function categories, such as mem-
ing freed blocks to produce a more valuable large free
ory or kernel, is similar to Facebook’s breakdowns. In
block. While numerous prior works optimize memory
contrast, SPEC CPU2006 benchmarks do not capture
allocations,6 very few recognize that optimizing free()
key leaf overheads faced by our microservices.
can result in significant performance wins.
Kernel. Microservices with high OS kernel overhead,
Cache1 and Cache2, invoke OS scheduler functions fre-
quently. Software/hardware optimizations that reduce Service Functionality Characterization
scheduler latency (e.g., intelligent thread switching7 We show a broad microservice functionality break-
and coalescing I/O) might considerably improve down in Figure 2.

36 IEEE Micro May/June 2021


FIGURE 3. Breakdown of CPU cycles spent in various microservice functionalities (service functionality categories defined in the
table to the right): Orchestration overheads are significant and fairly common across microservices.
We find that application logic disaggregation across
microservices has resulted in significant microservice
functionality overheads. Several microservices spend
only a small fraction of their execution time serving their
main application logic (e.g., ML-based ads recommenda-
tion or key-value serving), squandering significant cycles
facilitating the main logic via orchestration work that is
not critical to the main application logic (e.g., compres-
sion, serialization, and I/O processing). For example,
microservices that perform ML inference—Feed1, Feed2,
Ads1, and Ads2—spend as few as 33% of cycles on ML infer-
ence, consuming 42%–67% of cycles in orchestrating
inference. Hence, even if modern inference accelera-
tors10 were to offer an infinite inference speedup, the net
microservice performance would only improve by 1.49x–
2.38x. There is hence an urgent need to accelerate the
significant orchestration work that facilitates the main
application logic.
Orchestration overheads arise since a microservice,
upon receiving an RPC, must often perform operations
such as I/O processing, decompression, deserialization,
and decryption, before executing its main functionality.
Hence, many microservices face common orchestration
overheads despite great diversity in microservices’ main
application logic, as shown in Figure 3.
We make several observations about these signifi-
cant and common orchestration overheads. First, Web,
Cache1, and Cache2 spend a large portion of cycles exe-
cuting I/O, i.e., sending and receiving RPCs, and conse-
quent I/O compression and serialization overheads
dominate. Web incurs a high I/O overhead since it imple-
ments many URL endpoints and communicates with a
large back-end microservice pool. Cache1 and Cache2 are
May/June 2021
TOP PICKS
leaf microservices that support a high request rate8—
they frequently invoke RPCs to communicate with
mid-tier microservices. These microservices can bene-
fit from I/O optimizations such as kernel-bypass, multi-
queue NICs, and efficient I/O notification paradigms.
Second, Web spends only 18% of cycles in its main
web serving logic (parsing and processing client
requests), consuming 23% of cycles in reading and
updating logs. It is unusual for applications to incur
such high logging overheads; only few academic stud-
ies focus on optimizing them in hardware.
Third, Ads1, Feed1, Feed2, and Cache1 incur a high thread
pool management overhead. Intelligent thread sched-
uling and tuning7 can help these microservices.
We conclude application logic disaggregation
across microservices and the consequent increase
in inter-service communication at hyperscale has
resulted in significant and common orchestration
overheads in modern data centers. In Table 1, we
report acceleration opportunities that might inform
future software and hardware designs. We believe
that our rich overhead characterization and taxonomy
of existing optimizations will guide researchers in miti-
gating these overheads.
ACCELEROMETER: AN
ANALYTICAL MODEL FOR
HARDWARE ACCELERATION
Accelerating key common orchestration overheads in
production requires 1) designing new hardware; 2)
testing it; and 3) carefully planning capacity to provi-
sion the hardware to match projected load. Given the
IEEE Micro 37
TOP PICKS

TABLE 1. Summary of findings and suggestions for future optimizations.

Finding Acceleration opportunity

Significant orchestration overheads
Several common orchestration overheads
Poor IPC scaling for several functions
Memory copies & allocations are
significant
Memory frees are computationally expensive
High kernel overhead and low IPC
Logging overheads can dominate
High compression overhead
Cache synchronizes frequently
High event notification overhead
Software and hardware acceleration for orchestration rather than just app. logic
Accelerating common overheads (e.g., compression) can provide fleet-wide wins
Optimizations for specific leaf/service categories
Dense copies via SIMD, copying in DRAM, Intel’s I/O AT, DMA via accelerators, PIM
Faster software libraries, hardware support to remove pages
Coalesce I/O, user-space drivers, in-line accelerators, kernel-bypass
Optimizations to reduce log size or number of updates
Bit-plane compression, Buddy compression, dedicated compression hardware
Better thread pool tuning and scheduling, Intel’s TSX, coalesce I/O, vDSO
Hardware support for notifications (e.g., RDMA-style), spin versus block
hybrids

uncertainties inherent in projecting customer
demand, deploying diverse custom hardware is risky
at scale as the hardware might underperform due to
performance bounds from the microservice’s software
interactions with the hardware.
TO EASILY AND ACCURATELY MODEL
WHETHER AN ACCELERATOR IS
WORTH DESIGNING AND DEPLOYING
FOR A MICROSERVICE OPERATION,
WE DEVELOP AN ANALYTICAL
MODEL, ACCELEROMETER.
To easily identify performance bounds early in the
hardware design phase and estimate realistic gains from
hardware acceleration, there is an urgent need to
develop a simple, yet, accurate analytical model for hard-
ware acceleration. The state-of-the-art analytical model
for acceleration, LogCA,11 falls short for microservices as
it assumes that the CPU synchronously waits while the
offload operates. However, for many microservice func-
tionalities, offload is asynchronous; the processor con-
tinues doing useful work concurrent with the offload.
Capturing this concurrency-induced performance
bounds will help realistically model microservice
speedup for various hardware acceleration strategies.
To easily and accurately model whether an accelera-
tor is worth designing and deploying for a microservice
operation, we develop an analytical model, Accelerome-
ter. Accelerometer models both synchronous and
asynchronous offloads for three hardware acceleration
strategies—on-chip, off-chip, and remote.
Accelerometer assumes an abstract system with
three components: 1) host: a general-purpose CPU; 2)
accelerator: custom hardware to accelerate a kernel (or
microservice operation); and 3) interface: the
communication layer between the host and the acceler-
ator (e.g., a PCIe link). Accelerometer models both the
microservice throughput speedup (referred to as
“speedup”) and the per-request latency speedup
(referred to as “latency reduction”). Modeling both
speedup and latency reduction ensures that accelera-
tion enables a higher throughput (i.e., more QPS) without
violating microservice latency service level objectives
(SLOs). When work is offloaded to an accelerator, the
speedup and latency reduction depend on the accelera-
tion strategy and the threading design used to offload,
i.e., synchronous versus asynchronous offload.
Synchronous Offload
In a synchronous offload (Sync), the host awaits the
accelerator’s response before resuming execution
FIGURE 4. Example timeline of host and accelerator.

38 IEEE Micro May/June 2021


(see Figure 4), putting the accelerator’s operation
cycles (aC
A ) in the critical path of the host’s execution,
impacting speedup and per-request latency. The host
can consume additional cycles to 1) prepare the kernel
for offload, o0 ; 2) transfer the kernel to the accelerator,
L; and 3) wait in a queue for the accelerator to become
available, Q.
Synchronous Offload With Thread
Oversubscription
In reality, several microservices (e.g., Web and Cache)
oversubscribe threads to improve throughput by hav-
ing more threads than available cores. With synchro-
nous offload, a microservice oversubscribing threads
(Sync-OS) allows the host to schedule an available
thread to process new work, while the thread that off-
loaded work blocks awaiting the accelerator’s
response. Hence, the accelerator’s cycles do not criti-
cally affect speedup, but impact the per-request
latency. Moreover, OS overheads from the host
switching to an available thread after an offload, o1 ,
affect both speedup and latency reduction. The micro-
second-scale o1 overhead dominates in microsecond-
scale microservices (e.g., Caching), making it feasible
to incur a throughput gain at the cost of a per-request
latency slowdown. In such cases, Accelerometer can
help ensure that the microservice still meets its
latency SLO.
Asynchronous Offload
In an asynchronous offload (Async), the host does use-
ful work concurrent with the accelerator’s operation
on the offload, removing the accelerator’s cycles from
the critical path. Depending on whether the response
is picked up by the same thread that sent the request
or a different thread, OS thread switch penalties can
impact speedup and latency reduction.
Accelerometer models all these cases as well as
other nuanced scenarios to project realistic gains
early in the hardware design phase and make well-
informed hardware investments. We expect Acceler-
ometer to have the following use cases. 1) Data center
operators can estimate fleet-wide gains from optimiz-
ing key service overheads. 2) Architects can make bet-
ter accelerator design decisions and estimate realistic
gains by considering offload overheads due to micro-
service software design.
VALIDATING AND APPLYING
ACCELEROMETER
We validate Accelerometer’s utility via three retro-
spective case studies on production systems, by
May/June 2021
TOP PICKS
comparing model-estimated speedup with real micro-
service speedup determined via A/B testing. Each
study covers a distinct microservice threading sce-
nario (i.e., Sync, Sync-OS, and Async). We analyze 1) an on-
chip accelerator: a specialized hardware instruction
for encryption, AES-NI; 2) an off-chip accelerator: an
encryption device connected to the host CPU via a
PCIe link; and 3) a remote accelerator: a general-pur-
pose CPU that solely performs ML inference and is
connected to the host CPU via commodity network.
In all three studies, we show that Accelerometer
estimates the real microservice speedup with  3:7%
error.
Finally, we use Accelerometer to project speedup
for the acceleration recommendations derived from
three key common overheads identified by our charac-
terization: compression, memory copy, and memory
allocation.
LONG-TERM IMPLICATIONS
We discuss long-term implications, highlighting the
impact this work has already had.
Accelerometer in Production
As microservices evolve, Accelerometer’s generality
makes it even more suitable in determining new hard-
ware requirements early in the design phase. Since we
validated Accelerometer in production and made it
open-source,12 we are happy to report that it has been
adopted by multiple hyperscale companies (e.g., with
developing their encryption and compression acceler-
ators) to make well-informed hardware decisions. We
expect Accelerometer to trigger research in develop-
ing more complex models that account for overheads
induced by offloading to specific accelerators (e.g.,
software batching implications on FPGA memory
bandwidth versus latency).
Influence on Real Hardware Designs
In this work, we take a step back and answer the
Amdahl’s Law question of: Which overheads prevail
even after offloading a microservice’s main functional-
ity to accelerators? Our comprehensive study of real-
world microservices definitively indicates the need for
a qualitatively different approach to future accelerator
efforts. So far, data center hardware acceleration
efforts have primarily focused on the most costly
operations of a few “killer” applications (e.g., ML infer-
ence). However, accelerating orchestration overheads
can offer greater benefits as they are significant and
common across microservices.
IEEE Micro 39
TOP PICKS
As web service architectures grow more frag-
mented (e.g., deeper microservice pipelines and serv-
erless architectures), it becomes more important to
optimize the increasingly ubiquitous orchestration
overheads. However, accelerating orchestration
overheads is nontrivial as 1) orchestration libraries
are already well-optimized in software and 2) orches-
tration function invocations are frequent, involve
small data granularity, and are interspersed between
other microservice code. Hence, accelerating orche-
stration overheads will require different techniques
than those used in throughput-based specialization
blocks with coarse-grained offloads (e.g., video
processing).
ALTHOUGH ACCELEROMETER
PROVIDES THE FIRST STEP IN
DETERMINING REQUIRED
ACCELERATION STRATEGIES, WE
EXPECT SIGNIFICANT ACADEMIC AND
INDUSTRIAL INTEREST IN RETHINKING
ACCELERATORS FOR FINE-GRAINED
ORCHESTRATION OPERATIONS.
Although Accelerometer provides the first step in
determining required acceleration strategies, we
expect significant academic and industrial interest in
rethinking accelerators for fine-grained orchestration
operations. Already, a few hardware vendors have
used our study’s insights to influence hardware cus-
tomization for orchestration operations.
Characterization Approach and Tool
While it is relatively simple to measure the CPU cycles
spent in leaf functions, it is extremely difficult to cate-
gorize every path’s functionality in a microservice’s
entire call stack, as microservices have deep, complex
software stacks that are hard to parse and classify.
We developed a methodology to systematically clas-
sify each call trace path: We applied expert insights to
identify service functionality classification rules that
we then used to categorize cycles spent in various
microservice functionalities.
We integrated this characterization tool into our
fleet-wide performance monitoring infrastructure; it
currently assimilates statistics from hundreds of thou-
sands of servers from around the world to help devel-
opers visualize the performance impact of their code
changes at hyperscale. With the decline of hardware
40 IEEE Micro
performance scaling, there is a greater need for
researchers to develop such tools for performance
monitoring and optimization at all levels of the sys-
tems stack.
Industry-Academia Collaborative
Benchmarking Efforts
Many hardware vendors rely on open-source bench-
marks such as SPEC that heavily use C libraries to
make architecture decisions. Hence, in our characteri-
zation, we observe that only C libraries’ IPC scales well
across CPU generations, but the other overheads (e.g.,
memory movement and encryption) show little to no
improvement.
There is immense value in validating commonly
used benchmarks with real-world application behav-
iors. Our characterization drove a hardware vendor to
consider more representative benchmarks (in place of
traditional ones they used for decades) when evaluat-
ing hardware designs. This work has resulted in an
industry–academia joint collaborative effort to design
and open-source scale-out cloud benchmarks that
represent the hyperscale behaviors identified in our
characterization. We expect our comprehensive study
to drive continued benchmarking efforts that repre-
sent the severity of overheads in production-grade
software.
End-to-End Thinking in Accelerator
Design
Oftentimes, when designing accelerators, architects
tend to miss the end-to-end picture, i.e., overheads
that might arise from other system parts. When trying
to adopt these accelerators at hyperscale, we have
often found that they degrade performance due to
overlooked offload-induced overheads. Accelerometer
is a simple, powerful tool to help architects analytically
estimate offload-induced overheads that arise from
the end-to-end path, projecting realistic gains early in
the hardware design phase.
REFERENCES
1. S. Kanev et al., “Profiling a warehouse-scale
computer,” in Proc. Int. Symp. Comput. Archit., 2015,
pp. 158–169, doi: 10.1145/2749469.2750392.
2. A. Mirhosseini, A. Sriraman, and T. F. Wenisch,
“Enhancing server efficiency in the face of killer
microseconds,” in Proc. Int. Symp. High Perform.
Comput. Archit., 2019, pp. 185–198, doi: 10.1109/
HPCA.2019.00037.
May/June 2021

3. A. Sriraman and T. F. Wenisch, “mSuite: A benchmark
suite for microservices,” in Proc. IEEE Int. Symp.
Workload Characterization, 2018, pp. 1–12, doi: 10.1109/
IISWC.2018.8573515.
4. A. Sriraman and A. Dhanotia, “Accelerometer:
Understanding acceleration opportunities for data
center overheads at hyperscale,” in Proc. Int. Conf.
Archit. Support Program. Lang. Operating Syst., 2020,
pp. 733–750, doi: 10.1145/3373376.3378450.
5. J. L. Henning, “SPEC CPU2006 benchmark
descriptions,” ACM SIGARCH Comput. Archit. News,
vol. 34, no. 4, 2006, pp. 1–17, doi: 10.1145/
1186736.1186737.
6. S. Kanev, S. L. Xi, G.-Y. Wei and D. Brooks, “Mallacc:
Accelerating memory allocation,” in Proc. Int. Conf.
Archit. Support Program. Lang. Operating Syst., 2017,
pp. 33–45, doi: 10.1145/3037697.3037736.
7. A. Sriraman and T. F. Wenisch, “mTune: Auto-tuned
threading for OLDI microservices,” in Proc. USENIX
Conf. Operating Syst. Des. Implementation, 2018,
pp. 177–194. [Online]. Available: https://www.usenix.org/
conference/osdi18/presentation/sriraman
8. A. Sriraman, A. Dhanotia, and T. F. Wenisch, “SoftSKU:
Optimizing server architectures for microservice
diversity @scale,” in Proc. Int. Symp. Comput. Archit.,
2019, pp. 513–526, doi: 10.1145/3307650.3322227.
9. L. Luo et al., “LASER: Light, Accurate Sharing dEtection
and Repair,” in Proc. Int. Symp. High Perform. Comput.
Archit., 2016, pp. 261–273.
May/June 2021
TOP PICKS
10. N. Jouppi et al. “In-datacenter performance analysis of
a tensor processing unit,” in Proc. Int. Symp. Comput.
Archit., 2017, pp. 1–12.
11. M. S. B. Altaf, and D. A. Wood, “A high-level
performance model for hardware accelerators,” in Proc.
Int. Symp. Comput. Archit., 2017, pp. 375–388.
12. “Accelerometer,” doi: 10.5281/zenodo.3612796.
AKSHITHA SRIRAMAN is currently a Ph.D. candidate in com-
puter science and engineering at the University of Michigan,
Ann Arbor, MI, USA. Her research bridges computer architec-
ture and software systems, demonstrating the importance of
that bridge in realizing efficient hyperscale web services via
solutions that span the systems stack. Sriraman received an
M.S. degree in Embedded Systems from the University of
Pennsylvania. She is the corresponding author of this article.
Contact her at akshitha@umich.edu.
ABHISHEK DHANOTIA is currently a Performance Engineer at
Facebook, Menlo Park, CA, USA, where he works on designing
their next-generation systems and improving efficiency of data
center workloads. His research interests include computer
architecture, performance analysis, and energy efficient system
architectures for data centers. Dhanotia received an M.S. degree
in computer engineering from North Carolina State University.
Contact him at abhishekd@fb.com.
IEEE Micro 41
 THEME ARTICLE: TOP PICKS

Accelerating Genomic Data Analytics With

Composable Hardware Acceleration
Framework
Tae Jun Ham , Seoul National University, Seoul, 130-743, South Korea
David Bruns-Smith and Brendan Sweeney, University of California Berkeley, Berkeley, CA, 94720-5800, USA
Yejin Lee, Seong Hoon Seo , and U Gyeong Song, Seoul National University, Seoul, 130-743, South Korea
Young H. Oh , Sungkyunkwan University, Seoul, 03063, South Korea
Krste Asanovic , University of California Berkeley, Berkeley, CA, 94720-5800, USA
Jae W. Lee , Seoul National University, Seoul, 130-743, South Korea
Lisa Wu Wills , Duke University, Durham, NC, 27708-0187, USA

This article presents a framework, Genesis (genome analysis), to efficiently and

flexibly accelerate generic data manipulation operations that have become
performance bottlenecks in the genomic data processing pipeline utilizing FPGAs-
as-a-service. Genesis conceptualizes genomic data as a very large relational
database and uses extended SQL as a domain-specific language to construct data
manipulation queries. To accelerate the queries, we designed a Genesis hardware
library of efficient coarse-grained primitives that can be composed into a
specialized dataflow architecture. This approach explores a systematic and
scalable methodology to expedite domain-specific end-to-end accelerated system
development and deployment.

A
s the democratization of wet lab sequencing
technology drives down sequencing cost, the
cost and runtime of data analysis are becom-
ing more significant. An article published in PLoS Biol-
ogy quantitatively claimed that genomics is projected
to produce over 250 exabytes of sequence data per
year by 2025, far surpassing the current major genera-
tors of big data such as YouTube (1–2 exabytes/year)
and Twitter (1.36 petabytes/year). With the afore-
mentioned big data generation comes challenges in
genomic data acquisition, storage, distribution, and
analysis. We focus our effort on addressing the effi-
cient analysis of genomic data, in parti`cular, identify-
ing genomic variants in each individual genome, as it
is one of the most computationally demanding
pipelines.
0272-1732 ß 2021 IEEE
Digital Object Identifier 10.1109/MM.2021.3072385
Date of publication 12 April 2021; date of current version
25 May 2021.
Genomic data processing algorithms are composed
of a mixture of specific algorithms as well as generic
data manipulation operations. For example, the most
popular genome sequencing workflow, Broad Insti-
tute’s Genome Analysis ToolKit 4 (GATK4) Best Practi-
ces, consists of stages implementing specific
algorithms such as read alignment and variant calling
as well as stages performing generic data manipula-
tions such as mark duplicates and base quality score
recalibration. Thus far, most prior work focused on the
hardware acceleration of specific algorithms such as
read alignment1–3 or pair-HMM (hidden Markov model)
in variant calling. Such specialized accelerators, target-
ing a specific implementation of a particular genome
sequencing pipeline stage, have demonstrated orders
of magnitude speedups and energy-efficiency improve-
ments. With these specific algorithm accelerations in
place, the remaining unaccelerated analysis stages
that contain data manipulation operations become the
bottleneck and a large portion of the genomic analysis
execution time, making them good targets for acceler-
ation pursuant to Amdahl’s law.

42 IEEE Micro Published by the IEEE Computer Society May/June 2021


An important aspect of genomic data analysis is that
the algorithms are still being refined and special care is
needed when proposing hardware acceleration. For
example, INDEL realignment was the major performance
bottleneck in the now deprecated GATK3 and thus a
hardware accelerator targeting the stage was proposed.3
However, GATK4 does not utilize this stage with its
updated variant calling algorithms rendering the pro-
posal suitable largely for legacy pipelines. Similarly, accel-
erators targeting the pair-HMM algorithms used in the
variant calling stages of GATK4 are likely being replaced
by the DNN-based algorithm for the same stage. Noting
the rapid changes in specific algorithms, we argue that
designing accelerators for the generic data manipulation
portions of the pipelines is just as important, if not more,
than designing accelerators for the specific algorithms.
WE ARCHITECT AND EVALUATE A
FLEXIBLE ACCELERATION
FRAMEWORK THAT TARGETS GENERIC
DATA MANIPULATION OPERATIONS
COMMONLY USED IN GENOMIC DATA
PROCESSING CALLED GENESIS.
Thus, we architect and evaluate a flexible accelera-
tion framework that targets generic data manipulation
operations commonly used in genomic data processing
called Genesis. We view genomic data as traditional data
tables and use extended SQL as a domain-specific lan-
guage to process genomic analytics. Conceptualizing
the genomic data as a very large relational database
allows us to reason about the algorithms and transforms
genomic data processing stages into simple extended
SQL-style queries. Once the queries are constructed,
Genesis facilitates the translation of the queries into
hardware accelerator pipelines using the Genesis hard-
ware library that accelerates primitive operations in data-
base and genomic data processing. As a proof of
concept, we accelerate the data preprocessing phase in
GATK4 Best Practices and deploy Genesis-generated
accelerators on Amazon EC2 F1 instances. We demon-
strate that the accelerated system targeting these
queries provides a significant performance improvement
and cost savings over a commodity CPU.
GENOMIC DATA ANALYTICS
A genome is an organism’s complete set of DNAs. For a
human genome, each chromosome is represented as a
sequence of DNA base pairs expressed as a single
May/June 2021
TOP PICKS
character, A, T, C, G, representing a DNA nucleotide base.
Genomic analysis uses a DNA sequence to identify varia-
tions from a biological sample against a reference
genome. Our work focuses on genomic analysis through
the next generation sequencing (NGS) technology, the
de facto technology for the whole genome analysis. In
this process, fragmented DNA samples are read by an
NGS wet lab instrument. Raw sensor data from the
instrument are processed through an equipment-spe-
cific proprietary software (or hardware), and the instru-
ment outputs processed data called reads. Reads
contain multiple fragments from a sequence of base
pairs and a sequence of quality scores where a single
quality score represents the machine’s confidence of the
corresponding base pair measurement. This process of
postmeasurement analysis is called the primary analysis,
and the outcome of the primary analysis is an input to
the secondary analysis. Secondary analysis is a process
of identifying genomic variants. Since it is very computa-
tionally demanding, this is what most computer soft-
ware/hardware research (including ours) focuses on.
Once these genomic variants are identified, they can be
used to analyze the specific characteristics of this DNA
(e.g., disease risk).
GENESIS ACCELERATION
FRAMEWORK
Representing Genomic Data Analysis as
Relational Database Queries
Genesis conceptualizes genomic data as relational data
tables and uses SQL as a domain-specific language to
represent the target genomic analysis operations and
pipeline stages for acceleration. Genomic reads are rep-
resented as rows in a table, and attributes associated
with each read are represented as columns. We use Illu-
mina sequencer short reads (up to 151 base pairs per
read) for a specific human in our evaluated dataset. A ref-
erence sequence is fragmented into many segments and
each segment is represented as a row in the reference
table. We configure a single row in the reference table to
have about 1 M base pairs. For the efficient management
of those tables, we partition each table into multiple
tables by chromosome identifiers, and then again by the
mapped position of the reads or the reference data. For
both tables, we assign a unique partition ID to the parti-
tion. Genesis supports common SQL operations such as
Select, Where, GroupBy, Join, Limit (used to select a subset of
rows), Count, and Sum. In addition, we support two addi-
tional operations PosExplode and ReadExplode. PosExplode(COL,
INITPOS), converts an array in a single row of a single col-
umn (COL) to multiple rows with an extra POS column that
starts from the position INITPOS (POS is incremented by one
IEEE Micro 43
TOP PICKS
FIGURE 1. Example query, its execution flow, and the genesis-generated HW pipeline.
for every row that is exploded). ReadExplode converts a
read, stored as a single row in the read tables, to multiple,
separate rows where each row contains the base, the
corresponding quality score, and its position. This opera-
tion converts individual base pairs and corresponding
quality scores into separate rows utilizing its alignment
information recorded in the metadata called CIGAR
(CIGAR contains base pair alignment information such
as substitution, matching, insertion, deletion, etc.).
Finally, we support iteration over rows with the FOR Row IN
Table clause, which is similar to that of Oracle PL/SQL.
We use an example to illustrate how to construct
queries for a genomic data analysis operation and walk
through the execution of the query using a high-level
block diagram. In this example, the user wants to find
the number of bases that matches the reference for all
reads whose partition ID is equal to the constant P. In
this case, the user can represent this operation as a
sequence of SQL queries as shown in Figure 1(a), which
essentially follows the execution flow depicted in
Figure 1(b). Step 1: the set of reads and the relevant ref-
erence with the partition ID (P) are first extracted (I1).
Step 2: the relevant reference row’s base pair sequence
is expanded into multiple rows with PosExplode (I2). Step
3: for each read in the ReadPartition, its base pairs are con-
verted to a multirow table with ReadExplode (Q1). Step 4:
inner-join the ReadExplode’ed table and the subset of the
PosExplode’ed reference row table (the subset is obtained
with the LIMIT base offset clause) to obtain a joined table
44 IEEE Micro
that allows us to extract base matching information (Q2).
Step 5: the number of matching base pairs (i.e., a read’s
base pair is identical to the reference’s base pair) are
computed and inserted into the output table (Q3). Gene-
sis Hardware Library: Genesis framework lets a user
easily construct a dataflow pipeline that accelerates
the desired target query [e.g., Figure 1(a)]. The key idea
behind this framework is that a relational query can be
decomposed into a series of relational operators. For
example, it is well known that SQL queries can be easily
parsed into a tree graph where each node represents a
table (leaf node) or a relational/computational operator
(nonleaf node).4 In such a case, if there exists a set of
configurable hardware modules where each of them
can be directly mapped to each relational/computa-
tional operator, constructing a dataflow pipeline for the
query becomes rather simple. Specifically, each node in
the graph can be mapped to a Genesis hardware mod-
ule, and each edge in the graph is mapped to a hardware
queue connecting these modules.
Each Genesis hardware library module operates
with a sequence of data called streams. A stream con-
sists of many data items, each of which can contain
multiple different types of fields. Each data item is
also divided into multiple flits, where a single flit repre-
sents the atomic unit of data communication and
operation. For example, when a sequence of reads
forms a single stream, each read is a data item, and
each base pair (or multiple base pairs), which is part of
May/June 2021

FIGURE 2. Block diagrams of genesis data manipulation and computation modules.
a base pair sequence in a read, is a flit. In general, each
module consumes (or inspects) a single flit from its
input queue(s) and generates a single output flit. The
output flit is then inserted into the output queue,
which will work as an input queue for the next module.
(1) Data Manipulation and Computation
Modules
Figure 2 shows the block diagrams for four data manip-
ulation and computation modules in Genesis hardware
library. The figure visualizes the operations of each
module with the example input/output values. Detailed
explanations for each module are provided below.
Joiner merges flits from two input queues and pro-
duces a single output. For this module, a flit in an input
queue should consist of a key field and a data field.
Every cycle, this module compares keys of the flits
from two input queues and either outputs or discards
a single flit with the smaller key while leaving the other
one intact.
Filter takes input data from a single queue, checks
whether it matches the specified comparison condi-
tion (across fields or for a field and a constant), and
outputs the item if and only if the item satisfies the
specified condition.
Reducer takes a sequence of data and performs a
reduction operation (e.g., Sum, Max, Min, Count) with
a reduction tree. For this module, a reduction tree is
utilized to obtain a reduction result at a throughput of
a single flit per cycle. Note that this module can also
support reduction across multiple flits (i.e., reduction
at an item granularity).
Stream ALU takes input data from a single or two
input queues (or a single input queue and a constant
item) and performs a relatively simple unary/binary
ALU operation (e.g., NOT, ADD, SUB, CMP, AND, OR,
etc.) with data from those queues. When a single item
contains multiple values, the unary/binary operation is
performed in an elementwise manner.
May/June 2021
TOP PICKS
(2) Memory Access Modules
Memory Reader reads contiguous data from memory
and streams the read data to the next module. Given
a starting address and the total amount of data to
read from memory, it continuously sends memory
requests to memory at a memory access granularity
(e.g., 64B) as long as its internal prefetch buffer is not
full. At the same time, this module supplies the
returned data from memory to the next module at a
throughput of a single flit per cycle.
Memory Writer writes the data coming from an input
queue to memory. It takes a single flit from the previous
module per cycle and temporarily stores it in its internal
buffer. Once its internal buffer size reaches the size of
the memory access granularity (or a specific termination
condition), it sends a write request to memory starting
from the preconfigured starting address.
SPM (Scratchpad Memory) Reader simply takes an
address from the input queue and outputs the
scratchpad read result to the output queue. It can
also be configured to read all elements in the interval
when the starting address and the finishing address
are provided. This module is also used to drain all of
its content to the output queue when a drain signal is
provided.
SPM Updater takes an address and the value from
an input queue and updates the scratchpad memory.
This module supports three operating modes. First, it
can work like a memory writer, which performs
sequential writes to the SPM buffer when provided a
starting address. It can also be configured to perform
a random SPM write, which simply writes the value to
the provided address. Finally, it can be configured to
perform a read-modify-write update with the provided
modify function (e.g., add/subtract a constant).
(3) Genomic Data Processing Modules
ReadToBases supports the ReadExplode operation. This
module takes a sequence of CIGAR, POS, SEQ, and
IEEE Micro 45
TOP PICKS
optionally QUAL values from the input queues and pro-
duces a ReadExplode’ed table. Each cycle, this module
outputs the tuple of the reference position, the corre-
sponding base, and the quality score. The reference
position field can be Ins if the base is an inserted base.
Similarly, the base and quality score fields can be Del if
the base is deleted.
Constructing Hardware Accelerator
with Genesis Hardware Library
Genesis accelerates the user-provided query by con-
structing a hardware pipeline using multiple Genesis
hardware modules written in Chisel. For example, the
SQL query in Figure 1(a) is translated to the hardware
pipeline shown in Figure 1(c). The hardware pipeline
has five memory readers, and each reader reads the
data streams from READS.POS, READS.ENDPOS, READS.CIGAR,
READS.SEQ, and REFS.SEQ. Three of these memory readers
are connected to the ReadToBases module, which
generates a sequence of flits where each flit is a pair
of a base and the corresponding reference position.
This generated sequence is then provided as an input
to the Joiner. Unlike the reads data, the relevant refer-
ence data are mapped to an on-chip SPM to facilitate
data reuse. A single memory reader is connected to
the SPM Updater module so that it can initialize the
SPM with data from memory. The contents from this
SPM is retrieved with the SPM Reader, which takes
two inputs from the memory readers (the ones reading
READS.POS and READS.ENDPOS), reads the SPM contents for
the corresponding interval, and supplies the read data
(i.e., reference base pairs) to the Joiner. The Joiner
takes these two input sequences (i.e., one from the
read, another from the reference), performs an inner-
join, and passes the joined sequence to the Filter,
which compares two data fields (i.e., the base pair
from the read and the base pair from the reference),
and only outputs the matching items. Finally, the
Reducer module accumulates the number of matched
base pairs and passes the outcome to the memory
writer, which stores the outcome to memory. The con-
structed pipeline is fully pipelined and can process
one base pair per cycle. A single pipeline is often insuf-
ficient to fully utilize the available memory bandwidth
provided to the system. In order to fully utilize the
available memory bandwidth and achieve high
throughput, it is necessary to exploit abundant paral-
lelism in genomic data processing operations through
the use of multiple pipelines. Genesis treats each pipe-
line to be independent of each other except that they
share memory interfaces and the command interfa-
ces. This separation allows the utilization of different
46 IEEE Micro
hardware pipelines targeting different operations to
work together. Input/output ports of all hardware
pipelines’ memory modules are first arbitrated by a
local arbiter and then arbitrated again by one of the
global arbiters, each of which is connected to one out
of four memory channels in the system. A set of stock
Genesis modules are often enough to design accelera-
tors for data manipulation operations in the existing
gene processing pipeline. However, different genomics
data often need different treatment, and thus Genesis
is designed to support user-defined modules. Genesis
provides a standardized stream-based I/O interface
for all of its modules; a user only has to specify the
user-defined internal computation hardware in Chisel
to utilize the framework at ease.
EVALUATION
Methodology
Figure 3(a) shows the runtime breakdown of the
GATK4 Best Practices data preprocessing pipeline
with (bottom) and without (top) recently developed
hardware alignment accelerator.2 To demonstrate
Genesis’s capability to accelerate data manipulation
operations in genomic data analysis, we architected
and implemented hardware accelerators (we call each
hardware pipeline(s) constructed for a particular algo-
rithm an accelerator for the rest of this article to avoid
confusion) for three key data manipulation opera-
tions,5 namely Mark Duplicates, Metadata Update,
and the table construction phase of BQSR, which
together account for the majority of the runtime in
the data preprocessing phase of GATK4 Best Practi-
ces. Figure 3(b) shows the block diagrams for these
Genesis-generated accelerators.
We deployed Genesis-generated accelerators on
the commercial cloud using the Amazon EC2
f1.2xlarge instances. Each F1 instance contains a Xilinx
Virtex UltraScale+ VU9P FPGA card. We use a
250 MHz clock for all three accelerators. We configure
the number of pipelines to 1) the resource limit we can
fit on one FPGA card or 2) the performance limit where
an accelerator can no longer get more speedup from
parallelism due to memory or communication bottle-
necks. We used 16 pipelines for mark duplicates, 16
pipelines for metadata update, and 8 pipelines for
base quality score recalibration.
To compare our design with the software-only
implementation, we run GATK version 4.1.3 on an
Amazon EC2 r5.4xlarge instance that is memory-opti-
mized. Large memory is crucial to obtain high perfor-
mance for genomic data analysis workloads. For the
reads input dataset, we use a well-characterized
May/June 2021

FIGURE 3. GATK4 best practices runtime breakdown and block diagrams for the genesis-generated accelerators.
Illumina sequencing result of patient NA12878 obt-
ained from the Broad Institute Public Dataset, and we
use GRCh38 as the reference genome.
Performance Results
Figure 4(a) shows the speedup of three Genesis accel-
erators designed to accelerate various stages of the
GATK4 data preprocessing phase over the GATK4
software implementations run on a carefully config-
ured 8-core memory-optimized CPU instance. Genesis
achieves an overall speedup of 2 on mark duplicates
stage, 19.3 on metadata update, and 12.6 on BQSR
(covariate table construction). For metadata update
and BQSR stages, per-chromosome speedups are also
presented in Figure 4(c) and (d). Considering that
these three stages take about three and a half hours
for a single genome to execute (assuming that meta-
data update perfectly scales), Genesis reduces the
computation time to process a single person’s gene
by roughly 140 min. Figure 4(b) shows the breakdown
May/June 2021
TOP PICKS
of the Genesis framework runtime for the three stages
in the GATK4 data preprocessing phase. The figure
shows that the relatively low speedup of mark dupli-
cates stage is due to the unaccelerated software por-
tion of the stage, which is responsible for about 50%
of the baseline runtime. Furthermore, the figure indi-
cates the metadata update and BQSR speedups are
partially limited by the host-FPGA communication
(takes 53.4% and 29.5% of the runtime).
Cost Comparison
Many genomic data processing workloads exhibit a
plethora of parallelism and thus often scale relatively
well with the increased amount of resources. In such a
scenario, the cost can be a more meaningful metric
than the raw speedup itself since it considers the
amount of resources the system utilizes. We compare
the cost of running each accelerated stage in the
AWS f1 instance (1:69$/hr) with the cost of running
baseline software implementations on r5.4xlarge
IEEE Micro 47
TOP PICKS
FIGURE 4. Performance comparison of the Genesis accelerators over baseline for three GATK4 data preprocessing stages.
instance (1:29$/hr). Compared to the baseline, Genesis
reduces the cost of genomic data processing by
2.08, 15.05, and 9.84 for Mark Duplicates, Meta-
data Update, and BQSR (table construction) stages.
CONCLUSION AND IMPLICATIONS
Genesis acceleration framework explores a systematic
and scalable methodology to democratize end-to-end
accelerated system development and deployment
using a software interface that is a standardized lan-
guage as a domain-specific language and a hardware
library that is composed of efficient coarse-grained
primitives. While the work specifically showcases a set
of de facto algorithms in genomic analytics, the con-
cepts articulated and demonstrated in this work
can be applied to many other domains and inspire
accelerated systems research that offers a degree of
generality without sacrificing the efficiency brought
upon via specialization.
GENESIS HARDWARE LIBRARY
DEMONSTRATES THAT
CONSTRUCTING HARDWARE
ACCELERATORS BY UTILIZING A SET OF
COMPOSABLE HARDWARE MODULES
ENABLES A FLEXIBLE HARDWARE
ACCELERATOR DESIGN THAT CAN BE
EASILY EXTENDED OR UPDATED.
In this article, we investigate commonalities
between database and genomic analytics domains by
conceptualizing genomic data as a very large rela-
tional database and mapping genomic data analytics
algorithms into one or more relational database
48 IEEE Micro
queries. In addition, we observe that genomic process-
ing algorithms are composed of a mixture of specific
algorithms as well as generic data manipulation opera-
tions and that the generic data manipulation opera-
tions are the common primitives that can be used
across analytics domains. These commonalities allow
the sharing, reusing, and composition of hardware
modules across domains, lowering the development
costs of highly efficient accelerated systems. These
commonalities can be applied to many other big data
analytics domains such as graph analytics.
Genesis hardware library demonstrates that con-
structing hardware accelerators by utilizing a set of com-
posable hardware modules enables a flexible hardware
accelerator design that can be easily extended or
updated. We present a systematic way for accelerator
researchers to decompose the algorithms into primitive
operations and build hardware modules that directly
map to those primitives as composable hardware blocks
that form a hardware library. The algorithms are then
composed using the hardware modules and the frame-
work for ease of development. In domains where the
algorithms are constantly changing, such as genomics,
algorithm changes can be reflected quickly by simply
updating or adding new hardware library components
and recomposing the algorithm using the framework for
rapid deployment. This development methodology can
be adopted for various domains such as machine learn-
ing, having hardware library components to execute
matrix–matrix multiply, matrix–vector multiply, etc.
In the world of accelerator research, efficiently map-
ping software onto custom hardware is a challenging
problem. Researchers solve this problem by either using
high-level synthesis to generate hardware or inventing
new domain-specific languages that ease the mis-
matches between software and hardware. In this work,
we advocate to leverage an already-standardized lan-
guage as the domain-specific language and construct
May/June 2021

primitive operators that directly map software primi-
tives to hardware blocks to allow efficient mapping of
software to hardware. The effect of this approach is real-
ized in 1) the resultant accelerated systems achieving
one order of magnitude better performance and cost-
efficiency on FPGAs-in-the-cloud compared to multi-
threaded software, and 2) the ease of adoption of the
accelerated systems and the lowering of the barrier to
entry for non-hardware-savvy scientists to use the
accelerated systems, creating broader impacts.
ACKNOWLEDGMENTS
This work was supported in part by the Korean govern-
ment grant (NRF-2016M3C4A7952587). This work was
also funded in part by the Advanced Research Proj-
ects Agency-Energy (ARPA-E), U.S. Department of
Energy (Award Number DE-AR0000849), ADEPT Lab
industrial sponsor Intel, RISE Lab and APEX Lab spon-
sor Amazon Web Services, and ADEPT Lab affiliates
Google, Siemens, and SK Hynix.
REFERENCES
1. Y. Turakhia, G. Bejerano, and W. J. Dally, “Darwin: A
genomics coprocessor provides up to 15000x
acceleration on long read assembly,” in Proc. Int. Conf.
Archit. Support Program. Lang. Oper. Syst., 2018,
pp. 199–213.
2. D. Fujiki et al., “GenAX: A genome sequencing
accelerator,” in Proc. Annu. Int. Symp. Comput. Archit.,
Jun. 2018, pp. 69–82.
3. L. Wu et al., “FPGA accelerated INDEL realignment in
the cloud,” in Proc. Int. Symp. High-Perform. Comput.
Archit., 2019, pp. 277–290.
4. Oracle, “Database SQL Tuning Guide—SQL Process,”
[Online]. Avaialble: https://docs.oracle.com/database/
121/TGSQL/tgsql_interp.htm#TGSQL94618 (URL)
5. T. J. Ham et al., “Genesis: A. hardware acceleration
framework for genomic data analysis,” in Proc. Int.
Symp. Comput. Archit., Jun. 2020, pp. 254–267.
TAE JUN HAM is a Postdoctoral Researcher with Seoul
National University, Seoul, South Korea. Contact him at
taejunham@snu.ac.kr.
May/June 2021
TOP PICKS
DAVID BRUNS-SMITH is currently working toward a Ph.D.
degree with the Department of Electrical Engineering and
Computer Sciences, University of California Berkeley, Berke-
ley, CA, USA. Contact him at bruns-smith@berkeley.edu.
BRENDAN SWEENEY is currently working toward a Ph.D
degree with the Electrical and Computer Engineering Depart-
ment, The University of Texas at Austin, Austin, TX, USA. This
work was done when he was an undergraduate student at
UC Berkeley. Contact him at brs@berkeley.edu.
YEJIN LEE is currently working toward a Ph.D. degree with the
Computer Science and Engineering Department, Seoul National
University, Seoul, South Korea. Contact her at yejinlee@snu.ac.kr.
SEONG HOON SEO is currently working toward a Ph.D
degree with the Computer Science and Engineering Depart-
ment, Seoul National University, Seoul, South Korea. Contact
him at andyseo247@snu.ac.kr.
U GYEONG SONG is an undergraduate student with the
Computer Science and Engineering Department, Seoul
National University, Seoul, South Korea. Contact him at
thddnrud2010@gmail.com.
YOUNG H. OH is currently working toward a Ph.D degree with
the Semiconductor System Engineering Department, Sung-
kyunkwan University, Seoul, South Korea. Contact him at
younghwan@skku.edu.
KRSTE ASANOVIC is a Professor with the Computer Science
Division, Electrical Engineering and Computer Science Depa-
rtment, University of California Berkeley, Berkeley, CA, USA.
Contact him at krste@berkeley.edu
JAE W. LEE is an Associate Professor of computer science
and engineering (CSE) with Seoul National University (SNU),
Seoul, South Korea. Contact him at jaewlee@snu.ac.kr.
LISA WU WILLS is the Clare Boothe Luce Assistant Professor
of Computer Science and ECE with Duke University, Durham,
NC, USA. Contact her at lisa@cs.duke.edu.
IEEE Micro 49
 THEME ARTICLE: TOP PICKS

uGEMM: Unary Computing for GEMM

Applications
Di Wu , Jingjie Li , Ruokai Yin , Younghyun Kim , and Joshua San Miguel , University of
Wisconsin–Madison, Madison, WI, 53706, USA
Hsuan Hsiao, University of Toronto, Toronto, ON, M5S 1A1, Canada

General matrix multiplication (GEMM) is pervasive in various domains, such as

signal processing, computer vision, and machine learning. Conventional binary
architectures for GEMM exhibit poor scalability in area and energy efficiency, due to
the spatial nature of number representation and computing. On the contrary, unary
computing processes data in temporal domain with extremely simple logic.
However, to date, there rarely exist efficient architectures for unary GEMM. In this
work, we first present uGEMM, a hardware-efficient unary GEMM architecture
enabled by universally compatible arithmetic units, which simultaneously achieves
input-insensitivity and high output accuracy. Next, we demonstrate that the
proposed uGEMM can reliably early terminate the computation and offers dynamic
energy-accuracy scaling for real-world applications via an accuracy-aware metric.
Finally, to propel the future research for unary computing, we open source our unary
computing simulator, UnarySim.

G
eneral matrix multiplication (GEMM) is ubiqui-
tous and essential in many applications, partic-
ularly emerging deep learning. Conventional
binary GEMM implementations can leverage either hard-
ware for efficiency or software for flexibility.1 However,
as the parallelism increases, binary GEMM implementa-
tions, which compute on multiple parallel bits, suffer
from poor hardware area, power, and energy efficiency
due to exponentially growing wire congestion.
To enable efficient GEMM processing on extremely
area-, power-, or energy-constrained devices, unary
computing has been leveraged in prior works,2,3 which
employ extremely simple logic that consumes unary
data in the form of serial bit streams. Unary computing
converts computations from the spatial domain (i.e.,
traditional parallel binary computing) to the temporal
domain, enabling ultra-low-power hardware at the
cost of longer latency. There are, however, a widely
disparate set of unary computing schemes proposed
in the literature (e.g., rate-coded stochastic
0272-1732 ß 2021 IEEE
Digital Object Identifier 10.1109/MM.2021.3065369
Date of publication 11 March 2021; date of current version
25 May 2021.
computing,4 temporal-coded race logic3), with each
designed for application-specific use cases.1 Stochas-
tic computing supports both arithmetic and relational
operations but can often be inaccurate,5 while race
logic, though accurate, supports only limited arithme-
tic operations and is not applicable to GEMM.1
Such disparities and limitations pose three funda-
mental challenges to an ideal unary GEMM architec-
ture. 1) How can a designer figure out which unary
computing units would work? 2) How can a designer
ensure reliability, i.e., accuracy, while improving energy
efficiency? 3) How can a designer simulate and
explore the disparate design space? We propose the
unified unary GEMM architecture, dubbed uGEMM,
along with our UnarySim simulator to offer answers to
these questions, striving to make unary computing a
first-class citizen in resource-constrained systems.
To tackle the first challenge, we present new
mechanisms for unary arithmetic units and design uni-
versally compatible microarchitectures. These units
are highly accurate and support arbitrary input bit
streams, i.e., input-insensitivity, as indicated in the
input in Figure 1. Furthermore, they process bit
streams in a fully streaming manner without unary–
binary data interconversion, allowing for streaming
consecutive uGEMM blocks in Figure 1. The resultant

50 IEEE Micro Published by the IEEE Computer Society May/June 2021


FIGURE 1. Illustrative example of uGEMM.
uGEMM, a multiply-accumulate (MAC) array of these
novel units, naturally inherits all microarchitecture-
level benefits and outperforms counterparts. Then,
the second challenge is solved via a novel metric,
named stability, which monitors the output accuracy
in the temporal domain. Understanding the accuracy
evolution through time, we can early terminate the
computation, as shown in the output in Figure 1, to
achieve a desired efficiency reliably, i.e., in a controlla-
ble manner. The final challenge on the design space
exploration is also addressed in this work by open-
sourcing our unary computing simulator, UnarySim,
based on which the evaluations at microarchitecture,
architecture, and application levels can be
performed.1,6,7
BACKGROUND
According to the data encoding methods, we catego-
rize unary computing into rate-coded stochastic com-
puting4 and temporal-coded race logic,3 with
examples of data representations and limitations
shown in Figure 2.
Rate-Coding-Based Unary Computing
Stochastic computing adopts rate coding, whose data
value relies on the frequency of ones and zeros in the
bit stream, with each bit generated by comparing
source data with a random number generator (RNG).4
According to the polarity, bit streams can be in unipo-
lar or bipolar (unsigned or signed) formats. Given a bit
stream with the probability/frequency of ones as
PðS ¼ 1Þ 2 ½0; 1, unipolar and bipolar values are PðS ¼
FIGURE 2. Unary computing data representations and limita-
tions. .
May/June 2021
TOP PICKS
1Þ and 2  PðS ¼ 1Þ  1, respectively. Any real number
can be scaled and mapped into a bit stream.
Stochastic computing performs computation by
manipulating input bit streams statistically. In sto-
chastic computing, an adder with two inputs A and B
can be implemented by a two-input multiplexer (MUX)
whose select signal is a stochastic bit stream with
PðS ¼ 1Þ ¼ 0:5. This computes Vout ¼ ðVA þ VB Þ=2,
where a scaling factor of 2 is introduced to prevent
overflow. Besides the scaled addition using MUX, uni-
polar nonscaled addition can be done with an OR gate.
Multiplication can be achieved using an AND gate for
unipolar bit streams and an XNOR gate for bipolar bit
streams.
Although such extreme simplicity offers benefits in
hardware efficiency, stochastic computing suffers the
correlation problem5 (i.e., two bit streams are more
correlated when they are more similar), leading to
inaccuracy. For example, an AND gate only accurately
computes the product of two input bit streams when
they have zero correlation. Otherwise, if more paired
ones appear than expected, it ends up computing the
minimum function,7 as shown in Figure 2. Existing sol-
utions raise the demands for costly RNGs and
increase the latency to achieve accurate results.8,9
Temporal-Coding-Based Unary
Computing
Temporal coding is applied in race logic, which enco-
des data into the timing of a signal’s transition (or
edge), with the bit stream as a chain of ones followed
by a chain of zeros, or vice versa, and each bit gener-
ated by comparing source data with a counter out-
put.3 The polarity categorization of temporal-coded bit
streams can follow that in stochastic computing.
In race logic, an AND gate and an OR gate now per-
form minimum and maximum functions, respectively,
unlike the unipolar multiplication and unipolar non-
scaled addition in stochastic computing. As the signal
edges are deterministic, race logic can compute the
minimum and maximum accurately. However, prior to
our work, multipliers and adders have never been pro-
posed for temporal computing.
UGEMM MICROARCHITECTURE
AND ARCHITECTURE
In this section, we present novel linear functional unit
designs in Figure 3, including multiplication (uMUL),
scaled, and nonscaled additions (uSADD and
uNSADD), which are universally compatible to varying
input codings, and the integrated uGEMM architec-
ture in Figure 4.
IEEE Micro 51
TOP PICKS
FIGURE 3. uGEMM functional units. Thick line: binary signal; and thin line: unary signal.
uMUL
The proposed unipolar and bipolar uMUL are shown in
Figure 3(a) and (b). We recognize that unary multiplica-
tion using a naive AND gate [ 2 in Figure 3(a)] conditionally
produces a valid output bit when input 0 is logic one,
implying a conditional bit stream generation for input 1
[1 in Figure 3(a)]. More specifically, only when input 0 is
logic one, the RNG inside the bit stream generator for
input 1 will update, and the generator will eventually gen-
erate a new bit, i.e., input 0 is an enable signal to the bit
stream generator. As such, we thoroughly eliminate the
correlation problem and achieve high accuracy with
merely an extra enable signal. Then, for bipolar multipli-
cation using an XNOR gate, we decompose the XNOR gate
to two AND gates, with each leveraging the conditional
bit stream generation for high accuracy.
uSADD
Scaled addition calculates the average of all inputs,
where the average comes via scaling the output. For
scaled addition, the unipolar and bipolar microarchitec-
tures are identical, as in Figure 3 (c). The takeaway here
is that the input average is the mean of input-one
FIGURE 4. uGEMM architecture and its PE. Thick line: multibit stream; and thin line: single-bit stream.
52 IEEE Micro
counts. Therefore, the proposed uSADD first collects all
input bits with the parallel counter and compute the
sum in the accumulator as in block  1 . Then, the output
is set to the carry bit of the accumulator in block  2.
This means that only when there are N ones in the input,
a logic one at the output will be generated, exactly the
output scaling. Such a carry bit overflow mechanism no
longer considers the correlation between the input bit
streams, and obtains accurate results.
uNSADD
Nonscaled addition calculates the clipped sum of all
inputs, and the clipping happens when the sum over-
flows/underflows the legal unary data range. The par-
allel counter and accumulator in block  1 are the same
as in uSADD, except that now the accumulation out-
put is entirely utilized, rather than only the carry bit.
Next, a subtraction between the offset and the accu-
mulation result is performed in block  2 to retrieve the
anticipated count of output ones. The offset is of
value 0 and ðN  1Þ=2 for unipolar and bipolar bit
streams, respectively. Finally, block 3 generates the
output based on the anticipated one count and the
May/June 2021
 TOP PICKS

actual one count in the accumulator via comparison. TABLE 1. Hardware comparison of unary GEMM implemen-
When there are more anticipated ones than actual, tations.
logic ones will be output every cycle until they are
equal. Similar to uSADD, uNSADD computes based on GEMM Area Power Latency Energy
one counts and solves the correlation issue. Note that (mm2 ) (W) (m
ms) (m
mJ)
we are the first to support bipolar nonscaled addition Unipolar scaled
in unary computing. uGEMM 0.43 0.15 0.64 0.07
Gaines 1.57 0.50 0.64 0.32
Integrated uGEMM Architecture Sim 0.52 0.18 0.64 0.11
Our uGEMM can simply be built on above units, as
Jenson 0.08 0.02 163.84 3.91
they are universally compatible to varying inputs (i.e.,
input-insensitive) and directly take in/produce bit Najafi 1.26 0.46 0.64 0.29
streams as inputs and outputs (i.e., fully streaming Unipolar nonscaled
process). We use O ¼ A  B þ C as an example to illus- uGEMM 0.44 0.12 0.64 0.07
trate the uGEMM architecture, where O, A, B and C are
Gaines 1.55 0.49 0.64 0.31
of size ðm  nÞ, ðm  kÞ, ðk  nÞ, and ðm  nÞ, respec-
tively. Figure 4(a) shows the uGEMM architecture. Sim 0.50 0.15 0.64 0.09
Input A, B, and C go through the m-by-n processing Bipolar scaled
element (PE) array in the center of the figure. As
uGEMM 0.76 0.21 0.64 0.13
shown in Figure 4(b), the ði; jÞth PE performs the MAC
Gaines 1.56 0.50 0.64 0.32
of the ith row from A and the jth column from B, with k
elements for each row and column, then adds the Sim 0.53 0.18 0.64 0.12
ði; jÞth element from C as the result. Inheriting from Jenson 0.08 0.02 163.84 3.90
the features of uMUL, uSADD, and uNSADD, uGEMM
Najafi 1.25 0.45 0.64 0.29
achieves 1) high parallelism due to simple unary logic,
2) input-insensitivity and high output accuracy by Bipolar nonscaled
diminishing the correlation problem, and 3) fully uGEMM 0.77 0.20 0.64 0.13
streaming process to enable reliable early termination
for controllable energy efficiency boost. More details
on related mathematical theories and walkthrough
stability measures how early a bit stream’s value sta-
examples can be found in Wu et al.’s work.1
bilizes/converges, given a specific error budget. For a
bit stream of length L, VL represents its expected
ENERGY-ACCURACY SCALING value and Vl represents the value of the partial bit
EVALUATION stream based on the first l bits (l  L). If starting from
We list the hardware implementation result of uGEMM the lth bit, the bias DVl ¼ jVl  VL j is consistently
in Table 1. Here, we fix the GEMM shape to m ¼ k ¼ smaller than a user-defined threshold VTHD , then sta-
n ¼ 16, and evaluate uGEMM against Gaines’,4 Sim’s,10 bility is calculated as follows:
Jenson’s8 and Najafi’s9 unary schemes. Note that
uGEMM is the only one supporting bipolar nonscaled maxfljDVl > VTHD g
Stability ¼ 1  : (1)
addition and that Najafi’s design does not support L
temporal coding. We find that uGEMM consumes
small area and low power, outperforming Gaines,’ Stability ranges in [0,1], with a higher value indicat-
Sim’s and Najafi’s designs. Jenson’s unary GEMM is ing earlier termination. Usually, rate coding yields
significantly more lightweight than the others due to higher stability than temporal coding. Given a VTHD ,
its resource sharing scheme. However, Jenson’s the resultant maximum l is defined as the stable point,
design introduces long latency, resulting in higher after which the output accuracy, statistically, will
energy consumption. never drop below the threshold. More details on the
Early terminating a bit stream in the temporal usage of this metric at the microarchitecture, archi-
domain leads to a partial bit stream, which represents tecture, and application levels can be found in Wu
a less accurate value, but reduces the latency to offer et al.’s works.1,6
higher energy efficiency. As a newly proposed metric In addition to high accuracy, our unary multiplica-
for the capability of early termination in this work, tion and addition designs have consistently higher

May/June 2021 IEEE Micro 53

TOP PICKS
FIGURE 5. Progressive accuracy (curves) and stability (dots)
comparison of GEMMs. Cycle ranges from 1 to 256 in
uGEMM, Gaines,’ Sim’s and Najafi’s; and from 1 to 2562 in Jen-
son’s. RC: rate-coded input; TC: temporal-coded input.
stability than that of prior counterparts. Such high sta-
bility indicates that the maximum l is smaller for our
designs, implying that under the same error threshold,
our design can early terminate the computation earlier
than others to boost energy efficiency. By setting
VTHD ¼ 0:05, we present the accuracy evolution
through time (i.e., progressive accuracy) and the stable
point for various unary GEMM implementations in
Figure 5. We observe that uGEMM outperforms all
others in the final accuracy for both inputs, as well as
their difference, suggesting input-insensitivity. Then,
the stable points of uGEMM are also closer to the y-
axis in this plot, demonstrating that with the same error
budget, early termination in uGEMM happens earlier
than others, providing even higher energy efficiency
benefits on top of the results in Table 1. Therefore, for
each configuration and input type, uGEMM outper-
forms all other approaches in terms of how early its
accuracy can stabilize, i.e., high output stability.
Above results demonstrate that uGEMM is
more suitable than its counterparts for ultra-low-
power architectures, especially considering its reliable
energy efficiency boost via the accuracy-aware metric.
UNARYSIM: A UNARY COMPUTING
SIMULATOR
Although there exist tools for synthesizing unary cir-
cuits,11 there are still little to no publicly available tool-
chains for rapid design space exploration and
54 IEEE Micro
simulation of general unary architectures and applica-
tions. Given how diverse and specialized prior
unary works are, the gap between the demand for
ultra-low-power architecture exploration and the lack of
off-the-shelf toolchains motivated us to provide a stan-
dard means for characterizing different unary designs
and fairly quantifying their tradeoffs by open-sourcing
our unary computing simulator: UnarySim.12
IN ADDITION TO HIGH ACCURACY, OUR
UNARY MULTIPLICATION AND
ADDITION DESIGNS HAVE
CONSISTENTLY HIGHER STABILITY
THAN THAT OF PRIOR COUNTERPARTS.
UnarySim is a cycle-accurate simulator to capture
the precise behavior in a unary computing architec-
ture, as shown in Figure 6. The entire simulator con-
sists of two parts, including the software simulation
and the hardware implementation. The backbone of
the software simulation is the PyTorch deep learning
framework from Facebook; as such, UnarySim natu-
rally supports deep learning applications and is highly
scalable and extensible. UnarySim inherits the high
scalability of PyTorch by decoupling the high-level
architecture design and the low-level performance
simulation. The key simulated components are cate-
gorized into stream, kernel, and metric as follows.
1) The stream components are used to generate
the bit streams using either rate or temporal
coding and manipulate a pair of bit streams to
exhibit a specific correlation, so as to cover
major use cases in existing literature.1,4,5,7
2) The kernel components refer to the unary com-
puting units for both arithmetic (linear and non-
linear) and logical operations. The proposed
uGEMM microarchitectures and architecture
are all covered in this category, as well as some
FIGURE 6. UnarySim diagram.
May/June 2021

other general operations for deep learning and
computer vision.
3) The metric components include key unary com-
puting metrics in existing literature, like correla-
tion,5 accuracy, and our proposed stability
metric.1,6 Those metrics can be seamlessly pinned
to any node in the system during simulation and
monitor the system status over time, which can
be used to tune the system accuracy and latency.
WE INTRODUCE OUR DEDICATED
SIMULATOR FOR UNARY COMPUTING,
UNARYSIM, WHICH CAN
SIGNIFICANTLY LOWER THE LEARNING
CURVE TOWARD RAPID AND RELIABLE
UNARY COMPUTING RESEARCH.
After the system is fine-tuned, these metric compo-
nents can be removed to speed up simulation. All of
these components in software simulation are callable
PyTorch modules, which follows the PyTorch program-
ming rules, and are provided with test examples for clar-
ity. With the growing popularity of PyTorch in both the
academia and industry, we believe that UnarySim offers
a low barrier-to-entry for researchers: further extending
UnarySim to support more stream, kernel and metric
components simply requires constructing additional
PyTorch modules. In terms of the hardware implementa-
tion, we provide component-level implementation exam-
ples for the proposed and evaluated designs in this work,
as well as those in prior works.7 Note that current metric
components are merely simulated in software for perfor-
mance evaluation and do not have correspondent hard-
ware implementations. Future works for this simulator
will be the extension of the components to better sup-
port broader applications and the automation of execut-
ing arbitrary algorithms on unary hardware.
CONCLUSION
Unary computing has gained growing research atten-
tion in the past decade in the fields of signal process-
ing, computer vision, and machine learning, etc., due
to its ultra-low power consumption. In this work, to
further promote the research of unary computing into
broader disciplines, we focus on architecting GEMM,
which is ubiquitous in applications. We first present
our uGEMM microarchitecture and architecture with
benefits in input-insensitivity, output-stability, and
hardware efficiency compared to prior art. Then, we
May/June 2021
TOP PICKS
demonstrate the accuracy-aware energy-accuracy
scaling for uGEMM using our stability metric, which
offers further opportunities for energy savings via
early termination. Finally, we introduce our dedicated
simulator for unary computing, UnarySim, which can
significantly lower the learning curve toward rapid and
reliable unary computing research.
REFERENCES
1. D. Wu et al., “UGEMM: Unary computing architecture
for GEMM applications,” in Proc. 47th Annu. Int. Symp.
Comput. Archit., 2020, pp. 377–390, doi: 10.1109/
ISCA45697.2020.00040.
2. V. T. Lee, A. Alaghi, R. Pamula, V. S. Sathe, L. Ceze, and
M. Oskin, “Architecture considerations for stochastic
computing accelerators,” IEEE Trans. Comput.-Aided
Design Integr. Circuits Syst., vol. 37, no. 11, pp. 2277–
2289, Nov. 2018, doi: 10.1109/TCAD.2018.2858338.
3. A. Madhavan, T. Sherwood, and D. Strukov, “Race logic:
A hardware acceleration for dynamic programming
algorithms,” in Proc. 41st Annu. Int. Symp. Comput.
Archit., 2014, pp. 517–528, doi: 10.1109/
ISCA.2014.6853226.
4. B. R. Gaines, “Stochastic computing systems,” in Proc.
Adv. Inf. Syst. Sci., 1969, pp. 37–172, doi: 10.1007/978-1-
4899-5841-9_2.
5. A. Alaghi and John P. Hayes, “Exploiting correlation in
stochastic circuit design,” in Proc. 31st Int. Conf.
Comput. Des., 2013, pp. 39–46, doi: 10.1109/
ICCD.2013.6657023.
6. D. Wu, R. Yin, and J. SanMiguel, “Normalized stability: A
cross-level design metric for early termination in
stochastic computing,” in Proc. 26th Asia South Pacific
Des. Autom. Conf., 2021, pp. 254–259, doi: 10.1145/
3394885.3431549.
7. D. Wu and J. SanMiguel, “In-stream stochastic division
and square root via correlation,” in Proc. IEEE/ACM
56th Des. Autom. Conf., 2019, pp. 1–6, doi: 10.1145/
3316781.3317844.
8. D. Jenson and M. Riedel, “A deterministic approach to
stochastic computation,” in Proc. 35th Int. Conf.
Comput.-Aided Des., 2016, pp. 1–8, doi: 10.1145/
2966986.2966988.
9. M. H. Najafi, D. Jenson, D. J. Lilja, and M. D. Riedel,
“Performing stochastic computation deterministically,”
IEEE Trans. Very Large Scale Integration (VLSI) Syst.,
vol. 27, no. 12, pp. 2925–2938, Dec. 2019, doi: 10.1109/
TVLSI.2019.2929354.
10. H. Sim and J. Lee, “A new stochastic computing
multiplier with application to deep convolutional
neural networks,” in Proc. 54th Des. Autom. Conf.,
2017, pp. 1–6, doi: 10.1145/3061639.3062290.
IEEE Micro 55
TOP PICKS
11. K. Daruwalla, H. Zhuo, R. Shukla, and M. Lipasti, “BitSAD
v2: Compiler optimization and analysis for bitstream
computing,” ACM Trans. Archit. Code Optim., vol. 16,
no. 4, pp. 1–25, 2019, doi: 10.1145/3364999.
12. D. Wu and R. Yin, “UnarySim.” Accessed: Jan. 31, 2021.
[Online]. Available: https://github.com/diwu1990/
UnarySim
DI WU is currently a Ph.D. candidate with the Department of
Electrical and Computer Engineering, University of Wiscon-
sin–Madison, Madison, WI, USA. His research interests include
stochastic and approximate computing, as well as numerical
optimization for deep neural networks. Wu received B.S. and
M.Eng. degrees from Fudan University in 2012 and 2015,
respectively. He is a member of ACM. He is the corresponding
author of this article. Contact him at di.wu@ece.wisc.edu.
JINGJIE LI is currently a Ph.D. candidate with the Department
of Electrical and Computer Engineering, University of Wiscon-
sin–Madison, Madison, WI, USA. His research interests include
human-centered computing, Internet of Things, and embed-
ded systems. Li received a B.S. degree from Beijing Institute of
Technology and a B.Eng. (R&D) degree (Hons.) from The Aus-
tralian National University in 2017. He is a Student Member of
IEEE and ACM. Contact him at jingjie.li@wisc.edu.
RUOKAI YIN is currently a senior undergraduate at the Uni-
versity of Wisconsin-Madison, Madison, WI, USA, majored in
electrical engineering and computer science. His research
interests include design high-performance computer archi-
tecture for machine learning. Currently, he is working with
56 IEEE Micro
Prof. Joshua San Miguel on unary computing and applying
this paradigm to multiple real-world applications. Contact
him at ryin25@wisc.edu.
YOUNGHYUN KIM is currently an Assistant Professor of Elec-
trical and Computer Engineering at the University of Wiscon-
sin-Madison, Madison, WI, USA. His research interests include
energy-efficient computing, machine learning at the edge, and
cyber-physical systems. Kim received a Ph.D. degree in elec-
trical engineering and computer science from Seoul National
University in 2013. Before joining University of Wisconsin-
Madison in 2016, he was a postdoc at Purdue University,
West Lafayette, IN, USA. He is a member of IEEE and ACM.
Contact him at younghyun.kim@wisc.edu.
JOSHUA SAN MIGUEL is currently an Assistant Professor at
the University of Wisconsin-Madison, Madison, WI, USA. His
research interests include stochastic and approximate com-
puting, intermittent computing, and interconnection networks.
San Miguel received a Ph.D. degree in electrical and computer
engineering from the University of Toronto. He is a member of
IEEE and ACM. Contact him at jsanmiguel@wisc.edu.
HSUAN (JULIE) HSIAO is currently a Ph.D. candidate with the
Edward S. Rogers Sr. Department of Electrical and Computer
Engineering, University of Toronto, Toronto, ON, Canada. Her
research interests include stochastic computing and com-
puter-aided design. Hsiao received an M.A.Sc. degree in elec-
trical and computer engineering from the University of
Toronto. Contact her at julie.hsiao@mail.utoronto.ca.
May/June 2021
THEME ARTICLE: TOP PICKS

BabelFish: Fusing Address Translations for

Containers
Dimitrios Skarlatos, Umur Darbaz, Bhargava Gopireddy, Nam Sung Kim , and Josep Torrellas, University of
Illinois at Urbana-Champaign, Champaign, IL, 61820, USA

Cloud computing has begun a transformation from using virtual machines to using
containers. Containers are attractive because of their “build once, run anywhere”
computing model and their minimal performance overhead. Cloud providers
leverage the lean nature of containers to run hundreds of them or more on a few
cores. Furthermore, containers enable the serverless paradigm, which involves the
creation of short-lived processes. In this work, we identify that containerized
environments create page translations that are extensively replicated across
containers in the TLB and in page tables. The result is high TLB pressure and
redundant kernel work during page table management. To remedy this situation,
this article proposes BabelFish, a novel architecture to share page translations
across containers in the TLB and in page tables. BabelFish reduces the mean and
tail latency of containerized workloads, cold-start effects of function execution, and
container bring-up time. This work also advocates for the need to provide more
hardware support for containerized and serverless environments.

C
loud computing has been undergoing a radical
transformation with the emergence of Con-
tainers.1 Like a virtual machine (VM), a con-
tainer packages an application and all of its
dependencies, libraries, and configurations, and iso-
lates it from the system it runs on. However, while
each VM requires a guest operating system (OS), mul-
tiple containers share a single kernel. As a result, con-
tainers require significantly fewer memory resources
and have lower overheads than VMs. For these rea-
sons, cloud providers such as Google’s Compute
Engine, Amazon’s ECS, IBM’s Cloud, and Microsoft’s
Azure now provide container-based solutions. The
most prominent container solution is Docker contain-
ers. In addition, there are management frameworks,
such as Google’s Kubernetes2 and Facebook’s Twine,3
which automate the deployment, scaling, and mainte-
nance of containerized applications.
Container environments are typically oversub-
scribed, with many more containers running than
cores. Moreover, container technology has laid the
0272-1732 ß 2021 IEEE
Digital Object Identifier 10.1109/MM.2021.3073194
Date of publication 19 April 2021; date of current version
25 May 2021.
foundation for Serverless computing,4 a new cloud
computing paradigm provided by services like Ama-
zon’s Lambda, Microsoft’s Azure Functions, Google’s
Cloud Functions, and IBM’s Cloud Functions. The
most popular use of serverless computing is known as
Function-as-a-Service (FaaS). In this environment, the
user runs small code snippets called functions, which
are triggered by specified events. The cloud provider
automatically scales the number and type of functions
executed based on demand, and users are charged
only for the amount of time a function spends
computing.5,6
Our detailed analysis of containerized environ-
ments reveals that, very often, the same virtual page
number (VPN) to physical page number (PPN) transla-
tion, with the same permission bit values, is replicated
in the TLB and in page tables. One reason for this is
that containerized applications are encouraged to cre-
ate many containers, as doing so simplifies scale-out
management, load balancing, and reliability.7,8 In such
environments, applications scale with additional con-
tainers, which run the same application on different
sections of a common dataset. While each container
serves different requests and accesses different data,
a large number of the pages accessed is the same
across containers.

May/June 2021 Published by the IEEE Computer Society IEEE Micro 57

TOP PICKS
Another reason for the replication is that contain-
ers are created with forks, which replicate translations.
Further, since containers are stateless, data are usually
accessed through the mounting of directories and the
memory mapping of files, which further creates transla-
tion sharing. Also, both within and across applications,
containers often share middleware. Finally, the light-
weight nature of containers encourages cloud pro-
viders to deploy many containers in a single host.9 All
of this leads to numerous replicated page translations.
WE PROPOSE BABELFISH, A NOVEL
ARCHITECTURE TO SHARE
TRANSLATIONS ACROSS CONTAINERS
IN THE TLB AND IN PAGE TABLES,
WITHOUT SACRIFICING THE
ISOLATION PROVIDED BY THE
VIRTUAL MEMORY ABSTRACTION.
HANDLING TLB MISSES IN X86 LINUX
W hen a processor access misses in both L1 and
L2 TLBs, a page table walk begins. This is a
multistep process performed in hardware. Figure 1
shows the page walk for an address in the x86-64
architecture. The hardware reads the CR3 control
register, which contains the physical address of the
Page Global Directory (PGD) of the currently running
process. The hardware adds the 40-bit CR3 register to
bits 47-39 of the virtual address. The result is the
physical address of an entry in the PGD. The hardware
reads such address from the memory hierarchy—
accessing first the data caches and, if they declare a
miss, the main memory. The data in that address
contain the physical address of the page upper directory
(PUD), which is the next level of the translation. Such
physical address is then added to bits 38-30 of the
virtual address. The contents of the resulting address
are the physical address of the next-level table, the page
middle directory (PMD). The process is repeated using
bits 29-21 of the virtual address to reach the next table,
the Page Table (PTE). In this table, using bits 20-12 of the
virtual address, the hardware obtains the target physical
table entry (pte_t). The pte_t provides the physical page
number (PPN) and additional flags that the hardware
58 IEEE Micro
FIGURE 1. Page table walk.
Unfortunately, state-of-the-art TLB and page table
hardware and software are designed for an environ-
ment with few and diverse application processes. This
has resulted in per-process tagged TLB entries, sepa-
rate per-process page tables, and lazy page table man-
agement, where rather than updating the page
translations at process creation time, they are
updated later on demand. In containerized environ-
ments, this approach causes high TLB pressure,
redundant kernel work during page table management
and, generally, substantial overheads.
uploads into the L1 TLB to proceed with the translation
of the virtual address.
In theory, a page walk involves four cache hierarchy
accesses. In practice, a core has a translation cache
called the page walk cache (PWC) that stores a few
recently accessed entries of the first three tables (PGD,
PUD, and PMD). The hardware checks the PWC before
going to the cache hierarchy. If it hits there, it avoids a
cache hierarchy access.
When this translation process fails, a page fault
occurs and the OS is invoked. There are two relevant
types of page faults: major and minor. A major one
occurs when the page for one of these physical
addresses requested during the walk is not in memory.
In this case, the OS fetches the page from disk into
memory and resumes the translation. A minor page fault
occurs when the page is in memory, but the
corresponding entry in the tables says that the page is
not present in memory. In this case, the OS simply marks
the entry as present, and resumes the translation. This
happens, for example, when multiple processes share
the same physical page. Even though the physical page
is present in memory, a new process incurs a minor page
fault on its first access to the page.
May/June 2021

OUR PROPOSAL: BABELFISH
To remedy this problem, we propose BabelFish, a
novel architecture to share translations across con-
tainers in the TLB and in page tables, without sacrific-
ing the isolation provided by the virtual memory
abstraction. BabelFish eliminates the replication of
{VPN, PPN} translations in two ways. First, it modifies
the TLB to dynamically share identical {VPN, PPN}
pairs and permission bits across containers. Second, it
merges page table entries of different processes with
the same {VPN, PPN} translations and permission bits.
As a result, BabelFish reduces the pressure on the
TLB, reduces the cache space taken by translations,
and eliminates redundant minor page faults. In addi-
tion, it effectively prefetches shared translations into
the TLB and caches. The end result is the higher per-
formance of containerized applications and functions,
and faster container bring-up.
BabelFish has two parts. One enables TLB entry
sharing, and the other enables page table entry sharing.
ENABLING TLB ENTRY SHARING
In containerized environments, TLBs may contain mul-
tiple entries with the same {VPN, PPN} pair, the same
permission bits, and different PCID tags. Such replica-
tion can lead to TLB thrashing. To solve this problem,
BabelFish combines these entries into a single one
with the use of a new identifier called container con-
text identifier (CCID). All of the containers created by
a user for the same application are given the same
CCID. It is expected that the processes in the same
CCID group will want to share many TLB and page
table entries.
BabelFish adds a CCID field to each entry in the
TLB. Further, when the OS schedules a process, the
OS loads the process’ CCID into a register—like it cur-
rently does for the process’ PCID. Later, when the TLB
is accessed, the hardware will look for an entry with a
matching VPN tag and a matching CCID. If such an
entry is found, the translation succeeds, and the corre-
sponding PPN is read. Figure 2 shows an example for a
two-way set-associative TLB. This support allows all
the processes in the same CCID group to share
entries.
The processes of a CCID group may not want to
share some pages. In this case, a given VPN should
translate to different PPNs for different processes. To
support this case, we retain the PCID in the TLB, and
add an Ownership (O) bit in the TLB. If O is set, it indi-
cates that this page is owned rather than shared, and
a TLB hit also requires a PCID match.
May/June 2021
TOP PICKS
FIGURE 2. Two-way set-associative BabelFish TLB.
FIGURE 3. Ownership-PrivateCopy (O-PC) field. The Private-
Copy (PC) bitmask has a bit set for each process in the CCID
group that has its own private copy of the page. The ORPC bit
is the logic OR of all the bits in the PC bitmask.
We also want to support the more advanced case
where many of the processes of the CCID group want
to share the same {VPN0 , PPN0 } translation, but a few
processes of the CCID group do not, and have made
their own private copies. For example, one process
created {VPN0 , PPN1 } and another one created {VPN0 ,
PPN2 }. This situation occurs when a few of the pro-
cesses of the CCID group have written to a copy-on-
write (CoW) page and have made their own private
copy of the page, while most of the other processes
still share the original clean page. To support this
case, we integrate the ownership bit into a new TLB
field called Ownership-PrivateCopy (O-PC) (Figure 2).
Ownership-PrivateCopy Field. The O-PC field is
expanded in Figure 3. It contains the PrivateCopy (PC)
bitmask, one bit that is the logic OR of all the bits in
the PC bitmask (ORPC ), and the Ownership (O) bit. The
PC bitmask has a bit set for each process of the CCID
group that has its own private copy of this page. The
rest of the processes of the CCID group, which can be
an unlimited number, still share the clean shared page.
The BabelFish TLB is indexed as a regular TLB,
using the VPN Tag. The hardware looks for a match in
the VPN and CCID. All of the potentially matching TLB
entries will be in the same TLB set, and more than one
match may occur. On a match, the O-PC and PCID
fields are checked, and two cases are possible. First, if
the O bit is set, this is a private entry. Hence, the entry
can be used only if the process’ PCID matches the
TLB entry’s PCID field.
IEEE Micro 59
TOP PICKS
Alternately, if O is clear, this is a shared entry. In
this case, before the process can use it, the process
needs to check whether the process itself has its own
private copy of the page. To do so, the process checks
its own bit in the PC bitmask. If the bit is set, the pro-
cess cannot use this translation because the process
already has its own private copy of the page. (An entry
for such page may or may not exist in the TLB.) Other-
wise, since the process’ bit in the PC bitmask is clear,
the process can use this translation.
The O-PC information of a page is part of a TLB
entry, but only the O and ORPC bits are stored in the
page table entry. The PC bitmask is not stored in the
page table entry to avoid changing the data layout of
the page tables. Instead, it is stored in an OS software
structure, which also includes an ordered list (pid_list)
of processes that performed a CoW in the CCID group.
The order of the pids in this list encodes the mapping
of PC bitmask bits to processes. For example, the sec-
ond pid in the pid_list is the process that uses the sec-
ond bit in the PC bitmask. More details are given in
the conference paper.10
ENABLING PAGE TABLE ENTRY
SHARING
In current systems, two processes that have the same
{VPN, PPN} mapping and permission bits still need to
keep separate page table entries. This situation is com-
mon in containerized environments, where the pro-
cesses of a CCID group may share many pages (e.g., a
large library) using the same {VPN, PPN} mappings.
Keeping separate page table entries has two costs. First,
the many pte_ts requested from memory can thrash the
cache hierarchy.11 Second, every single process in the
group that accesses the page may suffer a minor page
fault, rather than only one process suffering a fault.
To solve this problem, BabelFish changes the page
table structures so that processes of the same CCID
can share one or more levels of the page tables. In the
most common case, multiple processes will share the
table in the last level of the translation. This is shown in
Figure 4. The figure shows the translation of an address
for two processes of the same CCID group that map it to
the same physical address. The two processes (one with
CR30 and the other with CR31 ) use the same last level
page (PTE). In the corresponding entries of their previous
tables (PMD), both processes place the base address of
the same PTE table. Now, both processes together suf-
fer only one minor page fault (rather than two), and
reuse the cache line that contains the target pte_t.
The default sharing level in BabelFish is a PTE table,
which maps 512 4-KB pages in x86-64. Sharing can also
60 IEEE Micro
FIGURE 4. Page table sharing in BabelFish.
occur at other levels. For example, it can occur at the
PMD level—i.e., entries in multiple PUD tables point to
the base of the same PMD table. In this case, multiple
processes can share the mapping of 512  512 4-KB
pages or 512 2-MB huge pages. Further, processes can
share a PUD table, in which case they can share even
more mappings. We always keep the first level of the
tables (PGD) private to the process.
PUTTING IT ALL TOGETHER
To understand the impact of BabelFish, we describe
an example. Consider three containers (A, B, and C)
that have the same {VPN0 , PPN0 } translation. First, A
runs on Core 0, then B runs on Core 1, and then C runs
on Core 0. Figure 5 shows the timeline of the transla-
tion process, as each container, in order, accesses
VPN0 for the first time. The top three rows of the figure
correspond to a conventional architecture, and the
lower three to BabelFish. To save space, we show the
timelines of the three containers on top of each other;
in reality, they take place in sequence.
We assume that PPN0 is in memory but not yet
marked as present in memory in any of the A, B, or C
pte_ts. We also assume that none of these transla-
tions is currently cached in the page walk cache
(PWC) of any core.
Conventional Architecture. The top three rows of
Figure 5 show the conventional process. As container
A accesses VPN0 , the access misses in the L1 and L2
TLBs, and in the PWC. Then, the page walk requires a
memory access for each level of the page table (we
assume that, once the PWC has missed, it will not be
accessed again in this page walk). First, as the entry in
the PGD is accessed, the page walker issues a cache
hierarchy request. The request misses in the L2 and L3
caches and hits in main memory. The location is read
from memory. Then, the entry in the PUD is accessed.
The process repeats for every level, until the entry in
the PTE is accessed. Since we assume that PPN0 is in
May/June 2021

FIGURE 5. Timeline of the translation process in a conventional (top) and BabelFish (bottom) architecture. In the figure, con-
tainer A runs on Core 0, then container B on Core 1, and then container C on Core 0.
memory but not marked as present, A suffers a minor
page fault as it completes the translation (see
Figure 5). Finally, A’s page table is updated and a
{VPN0 , PPN0 } translation for A is loaded into the TLB.
After that, container B running on another core
accesses VPN0 . The hardware and OS follow exactly
the same process as for A. At the end, B’s page table
is updated and a {VPN0 , PPN0 } translation for B is
loaded into the TLB.
Finally, container C running on the same core as A
accesses VPN0 . Again, the hardware and OS follow
exactly the same process. C’s page table is updated,
and a {VPN0 , PPN0 } translation for C is loaded into the
TLB. The system does not take advantage of the state
that A loaded into the TLB, PWC, or caches because
the state was for a different process.
BabelFish Architecture. The lower three rows of
Figure 5 show the behavior of BabelFish. Container A’s
access follows the same translation steps as in the
conventional architecture. After that, container B run-
ning on another core is able to perform the translation
substantially faster. Specifically, its access still misses
in the TLBs and in the PWC; this is because these are
per-core structures. However, during the page walk,
the multiple requests issued to the cache hierarchy
miss in the local L2 but hit in the shared L3 (except for
the PGD access). This is because BabelFish enables
container B to reuse the page-table entries of con-
tainer A—at any level except at the PGD level. Also,
container B does not suffer any page fault.
Finally, as C runs on the same core as A, it performs
a very fast translation. It hits in the TLB because it can
reuse the TLB translation that container A brought into
May/June 2021
TOP PICKS
the TLB. Recall that, in the x86 architecture, writes to
CR3 do not flush the TLB. This example highlights the
benefits in a scenario where multiple containers are
coscheduled on the same physical core, either in SMT
mode, or due to an oversubscribed system.
EVALUATION
We evaluate BabelFish with simulations of an 8-core pro-
cessor running a set of Docker containers in an environ-
ment with conservative container colocation. We
evaluate two types of containerized workloads (data
serving and compute) and two types of FaaS workloads
(dense and sparse). On average, under BabelFish, 53% of
the translations in containerized workloads and 93% of
the translations in FaaS workloads are shared.
Figure 6 shows the latency or time reduction
obtained by the extensions proposed by BabelFish.
BabelFish reduces the mean and tail (95th percentile)
latency of containerized data-serving workloads by
11% and 18%, respectively. It also lowers the execution
time of containerized compute workloads by 11%.
FIGURE 6. Latency or time reduction attained by BabelFish.
IEEE Micro 61
TOP PICKS
Finally, it reduces FaaS function execution time by
10%–55% and bring-up time by 8%.
FUTURE RESEARCH DIRECTIONS
AND APPLICATIONS
The core concepts of BabelFish can be extended for
other environments and resources. In this section, we
present some possible directions.
Overcommitted Environments
While container environments place substantial pres-
sure on the TLB and caches due to translations, as the
working set of applications continues to increase,
other workloads will also suffer from the same effects.
BabelFish novel design, which allows processes and
containers within a group to share replicated transla-
tions in the TLB and in the cache hierarchy, is a general
primitive that can be used in other environments to
reduce the translation-induced pressure. Furthermore,
BabelFish also supports the more advanced case where
many of the processes of a group want to share the
same translation, but a few other processes do not, and
have made their own private copies. This overall design
minimizes the context-switch overhead and effectively
prefetches shared translation in the TLB and the caches.
BABELFISH NOVEL DESIGN, WHICH
ALLOWS PROCESSES AND
CONTAINERS WITHIN A GROUP TO
SHARE REPLICATED TRANSLATIONS
IN THE TLB AND IN THE CACHE
HIERARCHY, IS A GENERAL PRIMITIVE
THAT CAN BE USED IN OTHER
ENVIRONMENTS TO REDUCE THE
TRANSLATION-INDUCED PRESSURE.
Virtualized Environments
Conventional virtualized environments are slowed
down by nested address translation. For example, a
nested address translation may require up to 24
sequential memory accesses. In such deployments,
content-aware page deduplication is a prevalent tech-
nique to reduce the memory pressure caused by mem-
ory overcommitment.
This page deduplication process creates page shar-
ing in virtualized environments, generating more repli-
cated translations. BabelFish can be extended to reduce
this translation replication in virtualized environments.
62 IEEE Micro
Container Context Identifiers (CCIDs)
CCIDs are a new way to logically group processes and
containers to enable resource sharing. CCIDs are a
useful abstraction to allow the hardware and kernel to
reason about the cooperation and isolation of execu-
tion contexts—enabling higher performance and
security guarantees. BabelFish proposes the first-class
support of CCIDs in both the kernel and in hardware,
which represents a first step toward container-aware
computing. Looking forward, it will be beneficial to
provide more hardware and systems support for con-
tainerized and serverless environments.
REFERENCES
1. Docker, “What is a container?.” [Online]. Available:
https://www.docker.com/what-container
2. Google, “Production grade container orchestration.”
[Online]. Available: https://kubernetes.io
3. C. Tang et al., “Twine: A unified cluster management
system for shared infrastructure,” in Proc. 14th USENIX
Symp. Operating Syst. Des. Implementation, USENIX
Assoc., Nov. 2020.
4. N. Savage, “Going serverless,” Commun. ACM, vol. 61,
no. 2, pp. 15–16, Jan. 2018.
5. A. Klimovic, Y. Wang, P. Stuedi, A. Trivedi, J. Pfefferle,
and C. Kozyrakis, “Pocket: Elastic ephemeral storage
for serverless analytics,” in Proc. 13th USENIX Symp.
Operating Syst. Des. Implementation, Carlsbad, CA,
USA, USENIX Assoc., 2018, pp. 427–444.
6. M. Shahrad, J. Balkind, and D. Wentzlaff, “Architectural
implications of Function-as-a-Service computing,” in
Proc. 52nd Annu. IEEE/ACM Int. Symp. Microarchit., 2019,
pp. 1063–1075.
7. B. Burns and D. Oppenheimer, “Design patterns for
container-based distributed systems,” in Proc. 8th
USENIX Workshop Hot Top. Cloud Comput., Denver,
CO, USA, Jun. 2016, pp. 108–113.
8. B. Ibryam, “Principles of container-based application
design,” Tech. Rep., Red Hat, Inc., 2017. [Online].
Available: https://www.redhat.com/cms/managed-
files/cl-cloud-native-container-design-whitepaper-
f8808kc-201710-v3-en.pdf f8808kc-201710-v3
9. IBM, “Docker at insane scale on IBM power systems.”
[Online]. Available: https://www.ibm.com/blogs/bluemix/
2015/11/dockerinsane-scale-on-ibm-power-systems
10. D. Skarlatos, U. Darbaz, B. Gopireddy, N. S. Kim, and J.
Torrellas, “Babelfish: Fusing address translations for
containers,” in Proc. ACM/IEEE 47th Annu. Int. Symp.
Comput. Archit., 2020, pp. 501–514.
11. Y. Marathe, N. Gulur, J. H. Ryoo, S. Song, and L. K. John,
“CSALT: Context switch aware large TLB,” in Proc. 50th
Annu. IEEE/ACM Int. Symp. Microarchit., 2017, pp. 449–462.
May/June 2021
THEME ARTICLE: TOP PICKS

Characterizing and Modeling Nonvolatile

Memory Systems
Zixuan Wang and Xiao Liu, University of California San Diego, La Jolla, CA, 92093, USA
Jian Yang, Google Inc., Mountain View, CA, 94043, USA
Theodore Michailidis, Steven Swanson , and Jishen Zhao , University of California San Diego, La Jolla, CA,
92093, USA

Scalable server-grade nonvolatile RAM (NVRAM) DIMMs are commercially available

with the release of Intel’s Optane DIMM. Recent studies on Optane DIMM-based
systems unveil discrepant performance characteristics, compared with what many
researchers thought before the product release. To thoroughly analyze the source
of the discrepancy and facilitate real-NVRAM-aware system design, we develop an
NVRAM microarchitecture characterization and modeling framework, consisting of
a Low-level profilEr for Non-volatile memory Systems (LENS) and a Validated cycle-
Accurate NVRAM Simulator (VANS). LENS allows users to comprehensively analyze
NVRAM performance attributes and reverse engineer NVRAM microarchitectures.
We use LENS to reverse engineer the sophisticated microarchitecture design of
Optane DIMM and generate a set of architecture implications of industrial
NVRAMs. VANS models Optane DIMM microarchitecture and is validated by
comparing with the detailed performance characteristics of Optane DIMM-
attached servers. VANS adopts a modular design that can be easily modified to
extend to other NVRAM architecture designs.

N
onvolatile RAMs (NVRAMs)1 have been envi-
sioned as a new tier of memory in server
systems. It offers byte-addressable access
through CPU load/store instructions, with comparable
performance to DRAM and durability property of stor-
age devices. Seeing these great values, a large body of
prior studies investigated how to exploit NVRAM to
benefit future computer systems.2 Yet only recently
has the first server-grade NVRAM DIMM product
come to market, namely Intel Optane DC persistent
memory (also known as Optane DIMM).3
Figure 1 illustrates an example of Optane DIMM-
based server systems. Optane DIMMs—denoted
NVRAMs—sit on the memory bus along with DRAM
DIMMs and are controlled by the processors’ inte-
grated memory controllers (iMCs). Optane DIMMs
operate in one of the two modes: memory mode or
0272-1732 ß 2021 IEEE
Digital Object Identifier 10.1109/MM.2021.3065305
Date of publication 11 March 2021; date of current version
25 May 2021.
app direct mode. In memory mode, Optane DIMMs
are configured as volatile memory with DRAM DIMMs
as data cache. In app direct mode, Optane DIMMs are
used as persistent memory, which unifies the fast-
access interface of memory with the persistence prop-
erty of storage.4 For example, programmers can
directly issue load/store instructions to access in-
memory data structures while NVRAM system soft-
ware and hardware ensure that the data structures
are recoverable in the face of system crashes and
power loss.2
The latest characterization studies5 exposed
substantial discrepant performance characteristics
compared to what we thought before the real prod-
ucts were released. For instance, Figure 2 compares
results of the widely used memory simulators,
DRAMSim26 and Ramulator,7 and our Optane
DIMM server measurement on single-thread read
and write bandwidth [see Figure 2(a)] and latency
per cache line (CL) access [see Figure 2(b)] with a
pointer chasing microbenchmark. The microbe-
nchmark randomly accesses a contiguous data
region with fixed 64 B objects. By varying the data

May/June 2021 Published by the IEEE Computer Society IEEE Micro 63

TOP PICKS
FIGURE 1. Optane DIMM-based server system configurations.
region size from 64 B to 64 KB, we observe a clear
inconsistency between the simulation and real-sys-
tem results. Other previous NVRAM emulators and
simulators also generate different performance
characteristics compared to real-machine results.
As such, previous NVRAM emulation and simulation
tools are insufficient to model modern real NVRAM
systems.
IN ORDER TO DEVELOP A MEMORY
SIMULATOR THAT MODELS THE
SOPHISTICATED PERFORMANCE
BEHAVIOR AND MICROARCHITECTURE
OF REAL NVRAM SYSTEMS, WE NEED
TO COLLECT SUFFICIENT
INFORMATION ABOUT DETAILED
PERFORMANCE CHARACTERISTICS.
In order to develop a memory simulator that models
the sophisticated performance behavior and micro-
architecture of real NVRAM systems, we need to col-
lect sufficient information about detailed performance
characteristics. This requires a comprehensive archi-
tecture-level memory system profiling. Most previous
performance profiling tools focus on investigating basic
memory system performance characteristics, such as
latency, bandwidth, and access counts. Beyond these
characteristics, DRAMA8 examines certain address
mapping schemes. However, none of the widely used
performance profiling tools allows us to analyze
detailed on-DIMM buffering and management schemes
of Optane DIMM.
Our article, presented at MICRO 2020,9 offers a set
of open-sourced NVRAM profiling and simulation
toolsa to address the above-mentioned challenges
and facilitate future research on architecture and sys-
tem designs of NVRAM-based memory systems.
a
Available at: https://github.com/TheNetAdmin/LENS-VANS
64 IEEE Micro
FIGURE 2. Comparison between memory simulators and
Optane DIMM system profiling. (a) Simulator average accu-
racy wrt. Optane DIMM load/store bandwidth (bw-ld and bw-
st) and latency (lat-ld and lat-st). (b) Comparison between
Ramulator and Optane DIMM on read latency per CL with
the pointer chasing test.
LENS: LOW-LEVEL NVRAM
PROFILER
To address the challenge of insufficient profiling tools,
this article proposes a Low-level profilEr for Non-vola-
tile memory System (LENS). LENS consists of a set of
NVRAM profiling tools and low-level microbe-
nchmarks. With detailed performance profiling, LENS
also allows researchers to reverse engineer the micro-
architecture design of NVRAM systems with on-DIMM
buffers and various control schemes. In this article, we
use LENS to profile the detailed architecture design of
Optane DIMM.
LENS Framework
LENS adopts three key components—probers—to
examine three aspects of NVRAM architecture design,
respectively. Each prober employs customized microbe-
nchmarks to trigger specific hardware behaviors, which
leads to different latency and bandwidth patterns with
different memory architecture properties. By analyzing
the performance patterns, we identify the correspond-
ing microarchitecture properties and parameters.
Buffer Prober detects the on-DIMM buffer capac-
ity, entry size, and organization, by measuring the
latency change caused by buffer overflow and read/
write amplification.
Policy Prober investigates NVRAM control poli-
cies, including data migration (e.g., for wear-leveling)
and multi-DIMM interleaving policy. Typical NVRAM
wear-leveling schemes migrate data from one NVRAM
media location to another to maintain evenly distrib-
uted wear out. The prober detects the frequency,
granularity, and latency overhead of such data migra-
tion procedures.
Performance Prober facilitates the analysis of the
other two probers to measure the device bandwidth
and latency for on-DIMM architecture components.
May/June 2021

Microbenchmarks LENS provides three microbe-
nchmarks: pointer chasing, overwrite, and stride.
Pointer chasing is a random memory access
benchmark: it divides a contiguous memory region—
referred to as a pointer chasing region (PC-Region)—
into equal-sized blocks (PC-Blocks); it reads/writes all
PC-Blocks in a PC-Region in random order, and
sequentially accesses data within each PC-Block. In
order to bypass CPU caches while still generating CL-
sized memory accesses, all pointer chasing tests are
implemented using nontemporal AVX512 load/store
instructions. Pointer chasing has three variants to
detect various buffer architecture characteristics: 1)
collecting average latency per CL with a fixed PC-
Block size across various PC-Region sizes; 2) quantify-
ing read and write amplification by using a fixed PC-
Region size, while varying the size of PC-Block; and 3)
issuing read-after-write requests, which issue writes in
a pointer chasing order, followed by reads in the same
order.
Overwrite repeatedly generates sequential writes
to a fixed memory region, and then measures the exe-
cution time of each iteration. It has two variants: 1)
collecting the execution time of each write iteration
with a fixed memory region size and 2) measuring the
frequency of long tail-latency by changing the memory
region size.
Stride sequentially reads or writes to a set of stride
CLs with a fixed striding distance. It has two variants:
1) measuring bandwidth by using a fixed striding dis-
tance and increasing the access size, and 2) charac-
terizing multi-DIMM interleaving by a fixed total
access size and a variable striding distance.
LENS Implementation. We build LENS as a Linux
kernel module to avoid the overhead of switching
between user and kernel spaces. We implement all
our microbenchmarks in assembly language to
enforce a well-controlled memory access behavior.
We also disable process preemption and hardware
prefetchers to avoid noises on probing results.
Architecture Implications of Industrial
NVRAMs
Figure 3 shows an overview of a real-machine charac-
terization with LENS. Each prober identifies certain
memory system architecture and performance char-
acteristics, as illustrated by arrows in the figure. We
identified the write-pending-queue (WPQ) size and
multi-DIMM interleaving scheme in iMC. We also iden-
tified two on-DIMM buffers, a 16 KB SRAM-based
read–modify–write (RMW) buffer and a 16 MB DRAM-
based address indirection translation (AIT) buffer, with
May/June 2021
TOP PICKS
FIGURE 3. LENS probers and Optane DIMM parameters. Yel-
low numbers are obtained from Intel documents; other num-
bers are characterized by LENS.
256 B and 4 KB access granularity, respectively. We
find that these two buffers are organized as a two-
level inclusive buffer hierarchy rather than indepen-
dent buffers. We also identified a 4 KB load-store-
queue (LSQ), which reorders the incoming requests to
perform write combining. Finally, we identified a long
tail-latency effect, which may be caused by wear-level-
ing data migration. We identified the frequency and
the block size of this data migration.
Buffer Capacity
Figure 4 shows representative profiling results of LENS
buffer prober and policy prober. Figure 4(a) shows a
pointer chasing read and write test with 64 B PC-Block
and various PC-Region sizes. The experiments mea-
sure latency per CL access. The shape of the curves
exposes the read requests overflow points at 16 KB
and 16 MB, where the read latency drastically changes.
This indicates that the Optane DIMM has two levels of
read buffers, one 16 KB and the other 16 MB. Figure 4
(a) also shows that the write curve (denoted by st) has
two overflow points at 512 B and 4 KB, respectively.
This indicates that the memory has two write buffers,
namely WPQ and LSQ, of two distinct capacities.
We refer to the 16 MB buffer as AIT buffer. Because
its read latency is approximately 100 ns, as shown in
Figure 4(a), it is likely to locate inside the on-DIMM
DRAM, where AIT resides. We believe that the 512 B
buffer is WPQ, given its small size; the 4 KB buffer is
an on-DIMM LSQ that reorders the read/write
requests.
Access Granularity
Figure 4(c) shows a read amplification test. The results
indicate that the RMW buffer and AIT buffer adopt
256 B and 4 KB access granularities, respectively.
Figure 4(d) shows a buffer write amplification test. It
demonstrates that the two write queues, WPQ and
LSQ, adopt 512 and 256 B granularities, respectively.
IEEE Micro 65
TOP PICKS
FIGURE 4. LENS profiling details. (a)–(d) Buffer prober analysis for buffer size, hierarchy, and granularity. (e)–(h) Policy prober
analysis for multi-DIMM interleaving and wear-leveling. (a) Load/store latency with 64 B PC-Block. (b) RaW latency and R+W
latency with 64 B PC-Block. (c) Read amplification score. (d) Write amplification score. (e) Execution time of sequential write
test. (f) Tail latency in overwrite test. Each iteration is one 256 B write. (g) Ratio of long tail latency with various overwrite granu-
larities. (h) L2 TLB miss per millisecond in the overwrite test of (f).
As a result, an mfence will cause the WPQ to flush in
total 512 B data; the LSQ combines 64 B writes into
256 B in order to reduce RMW operations.
Buffer Hierarchy Organization
Figure 4(b) shows a read-after-write (RaW) experiment
to characterize the buffer hierarchy organization.
Figure 4(b) indicates that RMW buffer and AIT buffer
form a two-level inclusive hierarchy—if they were
independent of each other, they would have demon-
strated a RaW latency speedup at 16 MB by fast-for-
warding data in parallel. They do not appear to be
exclusive either, because they adopt different access
granularities and entry sizes.
Figure 4(b) also demonstrates that the read-after-
write latency (denoted as RaW) is significantly higher
than the total latency of performing individual reads
and writes (denoted as R+W) for small PC-Regions.
The key reasons are small-sized RaW requests i) trig-
ger frequent memory bus redirection and ii) underuti-
lize the WPQ and LSQ capacity. Therefore, due to the
identified architectural design, small-sized requests in
Optane DIMMs tend to perform better with pure reads
or writes than with mixed read and write access pat-
terns. In addition, the high RaW latency with small PC-
Region sizes also indicates that mfence flushes the
LSQ, because i) RaW latency reduces when the PC-
Region size increases and ii) RaW latency equals the R
+W latency when the PC-Region size reaches 4 KB, the
size of the LSQ.
Multi-DIMM Interleaving Analysis
Figure 4(e) shows a multi-DIMM interleaving policy
analysis. LENS measures the execution time of
66 IEEE Micro
various sized sequential writes on interleaved and
noninterleaved DIMMs, respectively. In the inter-
leaved experiment, the first 4 KB has a similar execu-
tion time as in the noninterleaved case, indicating
that the first 4 KB is written to a single DIMM. We
also observe a repeated execution time pattern
every 4 KB, indicating that different 4 KB data writes
are directed to different DIMMs. These observations
indicate a 4 KB granularity for multi-DIMM interleav-
ing. We consider the reason for the interleaving gran-
ularity is to fully utilize the 4 KB sized LSQ and 4 KB
entries in AIT buffer.
Tail Latency Analysis
Figure 4(f) shows the overwrite experiment for tail
latency analysis. The policy prober constantly writes
data to the same 256 B memory region, and measures
the latency of each 256 B write. We observe a long tail
latency every 14;000 write iterations (i.e., every
3.4 MB for 256 B overwrite test), which incurs over
100 latency penalty on average. We do not observe
similar trends on DRAM. Moreover, L2 TLB miss rate
remains stable in the overwrite experiment, as shown
in Figure 4(h). Therefore, we consider wear-leveling as
a major contributor to the tail latency.
To study the size of the blocks that wear-leveling
tracks, we increase the size of the overwrite test
region and count the frequency of the long tail
latency. Each test writes the same amount of data to
NVRAM. As shown in Figure 4(g), the frequency of
long tail latency dramatically drops, once we overwrite
on 64 KB or larger memory regions. This indicates that
a possible block size for wear-leveling is 64 KB.
May/June 2021

FIGURE 5. VANS overview of (a) full system simulation model with VANS reconfigurable components connected to gem5 and (b)
detailed NVRAM DIMM model.
VANS: VALIDATED NVRAM
SIMULATOR
Based on our profiling observations, we build a Vali-
dated cycle-Accurate NVRAM Simulator (VANS) that
models Optane DIMM’s microarchitecture design and
its parameters. Figure 5 depicts an overview of VANS
model and example usage cases.
Figure 5(a) shows a full system simulation model
with VANS attached to gem5. The CPU cores and LLC
are provided by gem5, while VANS acts as a drop-in
replacement of gem5’s memory controller and main
memory. VANS provides a set of controller and mem-
ory components, which can be used in combination to
provide a conventional DRAM-only model or NVRAM-
involved memory mode and app direct mode. VANS’s
iMC controls multiple DRAM and NVRAM DIMMs with
various interleaving schemes. The DRAM model sup-
ports DDR4 that complies with the JEDEC standard,
and can be extended to support more DDR models
with slight modifications to the source code. The
NVRAM model supports all the hardware components
discovered by LENS and can configure them in differ-
ent combinations to simulate various possible NVRAM
architecture.
Figure 5(b) depicts a detailed NVRAM DIMM model
with an LSQ, an RMW buffer, an AIT, and NVRAM
media. The LSQ serves as the highest level storage in
the DIMM, directly queuing load/store requests from
the iMC. During each scheduling epoch, the LSQ per-
forms reordering and write combining to reduce the
number of RMW operations. The RMW Buffer receives
read and write requests from the LSQ, and accesses
the AIT buffer at certain granularity (by default 256 B
based on our Optane DIMM analysis). The AIT consists
of a translation table and a data buffer (AIT buffer).
The translation table stores the records of CPU
address to media address translation. It also stores
the media wear-leveling records in each table entry. If
one media block is wearing out, AIT stalls the inflight
May/June 2021
TOP PICKS
CPU writes to this block, migrates the data into
another media block, updates the translation record,
and then resumes the CPU write. The AIT data buffer
stores data from the media to accommodate read and
write requests from RMW buffer. We place the AIT
table and buffer in the on-DIMM DRAM, based on our
Optane DIMM profiling.
VANS is written in C++ and takes advantage of C+
+ 17 standard to improve performance and reduce
code complexity. It adopts a modularized software
design that is flexible and reconfigurable, where
each component has a consistent input and output
interface and can be configured to work with any
other component. This modular interface enables
users to configure VANS component organization
and parameters solely from a configuration file, with-
out any source code change. Figure 5(a) shows such
an example where multiple NVRAM models use vari-
ous internal components.
FIGURE 6. VANS performance validation with microbe-
nchmarks: (a)–(c) pointer chasing and (d) overwrite. (a)
Pointer chasing test (64 B PC-Block size) on noninterleaved
DIMM. (b) Pointer chasing test (64 B PC-Block size) on inter-
leaved 6DIMMs. (c) RMW buffer’s read amplification in (a). (d)
Overwrite (256 B region), each iteration is one 256 B write.
IEEE Micro 67
TOP PICKS
Verification and Validation
To verify the VANS DRAM model, we run SPEC2006
and SPEC2017 benchmarks on VANS and capture
internal DRAM command traces. Then, we employ
Micron’s DDR4 verification model with Cadence tool-
chain to test these command traces. The results dem-
onstrate that our model does not generate any illegal
DDR4 command, showing that our on-DIMM DRAM
model complies with DDR4 specification.
To validate the VANS NVRAM model, we run LENS
microbenchmarks on VANS and compare the simu-
lated load/store latency, bandwidth, and read amplifi-
cation factors with Optane DIMM-based real-machine
results. Figure 6(a) and (b) shows the pointer chasing
latency on a single DIMM and interleaved six DIMMs.
The simulated load/store latency curves match the
curves of real-system profiling. Furthermore, VANS’s
read amplification [see Figure 6(c)] and overwrite
latency [Figure 6(d)] match the Optane DIMM’s. Over-
all, VANS achieves over 86.5% accuracy across all
metrics.
LONG-TERM IMPLICATIONS
We hope to draw attention to the need for research on
architecture and systems based on sophisticated
NVRAM DIMM microarchitectures found in modern
and future devices. This work offers an open-source
toolset to make such research widely accessible.
An End-to-End Toolchain for Memory
System Research
Memory microarchitecture is becoming increasingly
complex in order to i) adapt to new memory media
and ii) embrace new computation functionality and
communication protocols.10 Yet, many of these new
memory techniques appear to be a black-box or a
grey-box and lack publicly available specifications.
Profiling tools, which can capture detailed memory
architecture designs, are critical to understanding the
memory performance characteristics and exploring
the full potential of the new techniques. The existing
profiling tools examine low-level hardware counters,11
high-level software schemes,12 or coarse-grained
architecture characteristics8 of the memory system.
Moreover, NVRAM research relies heavily on simula-
tors and emulators, while many of the existing main
memory simulators and emulators lack the flexibility
and accuracy to model the emerging nonvolatile
memory.
LENS and VANS is an end-to-end toolchain that
enables researchers to characterize and model the
new memory products without prior knowledge of its
68 IEEE Micro
microarchitecture. LENS fills in the gap with the capa-
bility of reverse engineering the detailed structures of
memory architecture components, management poli-
cies, and interfaces. VANS helps researchers to model
the emerging nonvolatile memory and communication
protocols with minimum engineering effort. This tool-
chain is beneficial to not only architecture research
but also system and software research, such as com-
piler optimizations and file system developments. Fur-
thermore, the toolchain can be easily extended to
support future research with different memory tech-
nologies and architectures.
DUE TO THE SCARCITY OF ACCESS TO
NVRAM DEVICES, NVRAM SYSTEM
RESEARCH TYPICALLY RELIES ON
PUBLICLY AVAILABLE OR OPEN-
SOURCE SIMULATION AND
EMULATION MODELS.
Need for Rethinking NVRAM System
Design
This article unveils the microarchitecture designs in
current and future NVRAM products. Conventional
DIMM architecture designs are incompatible with
novel NVRAMs, due to the discrepant performance
characteristics of NVRAM media compared with tradi-
tional DRAM devices. This requires imperative investi-
gations on NVRAM-based computer systems and
attention from both academia and industry on new
software and hardware designs.
Modeling State-of-the-Art NVRAM
Architecture
Due to the scarcity of access to NVRAM devices,
NVRAM system research typically relies on publicly
available or open-source simulation and emulation
models. Therefore, it is imperative for the research
community to have simulation models that keep up
with the advancement of NVRAM architecture
designs. VANS is verified and validated with the latest
commercialized NVRAM product. It is the first design
to offer the research community the new NVRAM
architecture design and enable NVRAM systems stud-
ies without access to the real hardware.
CONCLUSION
This article presents LENS and VANS, a low-level
performance profiler for NVRAM systems and an
May/June 2021

architecture-level memory simulator that models
the recently released Optane DIMM memory sys-
tem. Using LENS, we perform a detailed characteri-
zation of the Optane DIMM microarchitecture
design. LENS allows users to characterize the per-
formance and reverse engineer the architecture
design of real NVRAM DIMM systems. VANS allows
researchers who do not have access to Optane
DIMM physical devices to explore and evaluate new
design ideas on architecture and systems. Further-
more, both LENS and VANS are flexible to be used
with other NVRAM systems beyond the current
Optane DIMM design.
ACKNOWLEDGMENTS
This work was supported in part by the National Sci-
ence Foundation under Grant 1829524 and Grant
1817077 and in part by SRC/DARPA Center for
Research on Intelligent Storage and Processing-in-
memory.
REFERENCES
1. B. C. Lee, E. Ipek, O. Mutlu, and D. Burger, “Architecting
phase change memory as a scalable DRAM
alternative,” in Proc. Int. Symp. Comput. Archit., 2009,
pp. 2–13, doi: 10.1145/1555754.1555758.
2. J. Xu and S. Swanson, “NOVA: A log-structured
file system for hybrid volatile/non-volatile main
memories,” in Proc. USENIX Conf. File Storage Technol.,
2016, pp. 323–338, doi: 10.5555/2930583.2930608.
3. “Intel Optane Persistent Memory,” 2019. [Online].
Available: https://www.intel.com/content/www/us/en/
architecture-and-technology/optane-dc-persistent-
memory
4. J. Zhao, S. Li, D. H. Yoon, Y. Xie, and N. P. Jouppi, “Kiln:
Closing the performance gap between systems with
and without persistence support,” in Proc. Int. Symp.
Microarchit., 2013, pp. 421–432, doi: 10.1145/
2540708.2540744.
5. J. Yang, J. Kim, M. Hoseinzadeh, J. Izraelevitz, and
S. Swanson, “An empirical guide to the behavior and
use of scalable persistent memory,” in Proc. USENIX
Conf. File Storage Technol., 2020, pp. 169–182. [Online].
Available: https://www.usenix.org/conference/fast20/
presentation/yang
6. P. Rosenfeld, E. Cooper-Balis, and B. Jacob,
“DRAMSim2: A cycle accurate memory system
simulator,” IEEE Comput. Archit. Lett., vol. 10, no. 1,
pp. 16–19, Jan.–Jun. 2011, doi: 10.1109/L-CA.2011.4.
May/June 2021
TOP PICKS
7. Y. Kim, W. Yang, and O. Mutlu, “Ramulator: A fast and
extensible DRAM simulator,” IEEE Comput. Archit. Lett.,
vol. 15, no. 1, pp. 45–49, Jan.–Jun. 2016, doi: 10.1109/
LCA.2015.2414456.
8. P. Pessl, D. Gruss, C. Maurice, M. Schwarz, and
S. Mangard, “DRAMA: Exploiting DRAM addressing for
cross-CPU attacks,” in Proc. USENIX Secur. Symp., 2016,
pp. 565–581, doi: 10.5555/3241094.3241139.
9. Z. Wang, X. Liu, J. Yang, T. Michailidis, S. Swanson, and
J. Zhao, “Characterizing and modeling non-volatile
memory systems,” in Proc. Int. Symp. Microarchit., 2020,
pp. 496–508, doi: 10.1109/MICRO50266.2020.00049.
10. “Compute Express Link,” 2020. [Online]. Available:
https://www.computeexpresslink.org/
11. “Linux Perf Examples,” 2020. [Online]. Available: http://
www.brendangregg.com/perf
12. “Valgrind,” 2020. [Online]. Available: https://valgrind.
org/
ZIXUAN WANG is currently working toward a Ph.D. degree
with the Department of Computer Science and Engineering,
University of California San Diego, La Jolla, CA, USA. His
research focuses on operating system and computer archi-
tecture design for memory systems. He is the corresponding
author for this article. Contact him at zxwang@ucsd.edu.
XIAO LIU is currently working toward a Ph.D. degree with the
Department of Computer Science and Engineering, Univer-
sity of California San Diego, La Jolla, CA, USA. His research
concerns reliability support for memory hierarchy, in-memory
machine learning acceleration, persistent memory system
design, and fault tolerance in the storage system of data cen-
ters. Contact him at x1liu@ucsd.edu.
JIAN YANG is currently a software engineer at Google, Moun-
tain View, CA, USA, where he works on the host networking
team. This work was done while he was with University of Cal-
ifornia San Diego, La Jolla, CA, USA. Yang received a Ph.D.
degree in computer science from the University of California
San Diego. Contact him at jianyang@google.com.
THEODORE MICHAILIDIS is currently working toward a Ph.D.
degree with the Department of Computer Science and Engi-
neering, University of California San Diego, La Jolla, CA, USA.
His research focuses on building efficient memory and stor-
age systems through cooperative software–hardware techni-
ques. Contact him at tmichail@eng.ucsd.edu.
IEEE Micro 69
TOP PICKS
STEVEN SWANSON is currently a professor with the Depart-
ment of Computer Science and Engineering, University of
California San Diego, La Jolla, CA, USA, the Director of the
lu Chair in
Non-Volatile Systems Laboratory, and the Haliciog
Memory Systems. His research interests include the systems,
architecture, security, and reliability issues surrounding het-
erogeneous memory/storage systems, especially those that
incorporate nonvolatile, solid-state memories. He received
the NSF CAREER Award, Google Faculty Awards, and a Face-
book Faculty Award. He has been a NetApp Faculty Fellow.
He is a co-founder of the Non-Volatile Memories Workshop.
70 IEEE Micro
Swanson received a Ph.D. from the University of Washington
in 2006. Contact him at swanson@cs.ucsd.edu.
JISHEN ZHAO is currently an associate professor of com-
puter science and engineering at the University of California
San Diego, La Jolla, CA, USA. Her research interests include a
broad range of computer architecture and system topics that
bridge system software and hardware design, with an empha-
sis on memory and storage systems, and machine learning
for systems and programming. Zhao received a Ph.D. degree
in computer science and engineering from Pennsylvania
State University. Contact her at jzhao@ucsd.edu.
May/June 2021
THEME ARTICLE: TOP PICKS

Temporal Computing With Superconductors

Georgios Tzimpragos and Jennifer Volk, University of California, Santa Barbara, Santa Barbara,
CA, 93106, USA
Dilip Vasudevan , Lawrence Berkeley National Laboratory, Berkeley, CA, 94720, USA
Nestan Tsiskaridze, Stanford University, Stanford, CA, 94305, USA
George Michelogiannakis, Lawrence Berkeley National Laboratory, Berkeley, CA, 94720, USA
Advait Madhavan, University of Maryland, College Park, MD, 20742, USA
John Shalf, Lawrence Berkeley National Laboratory, Berkeley, CA, 94720, USA
Timothy Sherwood, University of California, Santa Barbara, Santa Barbara, CA, 93106, USA

Creating computing systems able to address our ever-increasing needs, especially

as we reach the end of CMOS transistor scaling, will require truly novel methods of
computing. However, the traditional logic abstractions and the digital design
patterns we understand so well have coevolved with the hardware technology that
has embodied them. As we look past CMOS, there is no reason to think that those
same abstractions best serve to encapsulate the computational potential inherent
to emerging devices. We posit that a new and radically more efficient foundation for
computing lies at the intersection of superconductor electronics and delay-coded
computation. Building on recent work in race logic, we show that superconducting
circuits can naturally compute directly over temporal relationships between pulse
arrivals; that the computational relationships between those pulse arrivals can be
formalized through a functional extension to a temporal predicate logic used in the
verification community; and that the resulting architectures can operate
asynchronously and describe real and useful computations. We verify our
hypothesis through a combination of detailed analog circuit models and layout
designs, a formal analysis of our abstractions, and an evaluation of several
superconducting temporal accelerators.

S
uperconductor electronics offer the potential
to perform computations at tremendous
speeds and energy savings—especially on large
scales.1 Unfortunately, a semantic gap exists between
the level-driven logic, which CMOS designs accept as
a foundation, and the pulse-driven logic that is natu-
rally supported by the most compelling superconduct-
ing technologies. This gap creates a variety of
challenges across the full hardware stack, from the
circuits up through the CAD and architecture levels.
0272-1732 ß 2021 IEEE
Digital Object Identifier 10.1109/MM.2021.3066377
Date of publication 17 March 2021; date of current version
25 May 2021.
A pulse, unlike a stable voltage level, will fire
through a channel for only an instant. Arranging the
network of superconducting components so that input
pulses—driven by the transfer of magnetic flux
quanta—always arrive simultaneously to logic gates to
maintain the illusion of a strictly Boolean evaluation is
a significant engineering hurdle and results in unavoid-
able overheads. If we instead think about these pulses
as the representation of nonbinary data, the natural
language for expressing computations over those data
would be one that can efficiently describe the temporal
relationships between these pulses. Here, we draw
upon two distinct lines of research, both currently dis-
connected from superconducting.
First, recent work has shown that delay-based
encoding has impressive expressive power and

May/June 2021 Published by the IEEE Computer Society IEEE Micro 71

TOP PICKS
practical utility in implementing important classes of
accelerators.2–4 The principles of the delay-coded
logic described in that prior work apply directly to
problems in superconducting. However, the fact that
its primitive operations have been so far implemented
only in CMOS under specific assumptions—e.g., edges
are used to denote event occurrences—brings up
questions about their implementability and efficiency
in the much different superconducting technology.
Second, there is a long history of work in temporal
logic. Temporal logic systems allow for the representation
and reasoning of propositions qualified in terms of time.
Although expressing and evaluating constraints on the
order of events is particularly useful for formal verification
purposes, the predicate nature of existing temporal logics
makes them insufficient for desired computing tasks. We
instead need a temporal logic with computational capa-
bilities that takes events as inputs and creates new
events as outputs based on the input relationships.
In this article, we address these issues and explore
the potential of superconducting temporal computing
by proposing a new computational temporal logic. This
computational temporal logic subsumes classical tem-
poral predicate logic and gives clear, precise, and useful
semantics to delay-based computations. Moreover, we
demonstrate how this approach enables a tradeoff
between implementation complexity and delay; show
how one can realize a functionally complete set of tem-
poral operations in superconducting circuits; and create
useful designs that effectively capture the potential of
these circuits. Finally, we identify critical timing con-
straints and develop proof-of-concept accelerator
designs that demonstrate the functional correctness
and high performance of the proposed logic scheme,
methodology, and architectures. Open-source imple-
mentations of our superconducting temporal primitives
and accelerators can be found on github.
BACKGROUND
Computing With Superconductors
Pulse-based superconductor electronics are charac-
terized by three basic features: 1) the absence of resis-
tance in static circuits at superconducting
temperatures, 2) the Josephson effect, which governs
the fundamental switching element in superconductor
circuits, the Josephson junction (JJ), and 3) the propa-
gation of single flux quanta (SFQ), appearing as ps-
duration, mV-amplitude pulses across switching JJs,
instead of static-voltage levels.
Unlike semiconductor devices, the resistance of
superconducting circuits is zero, and thus, their speed is
not limited by RC time constants. Moreover, the
72 IEEE Micro
propagation of digital data (in superconducting intercon-
nects) is near the speed of light, lossless, and consumes
no power. These speed and power benefits do not come
without challenges, though. One of the most profound is
the pulse-based nature of computation. Developing logic
using only pulses presents intellectual and application
challenges alike. Pulses cannot be sampled like voltage
levels because they do not coincide with ps precision
(because of this reason, we usually end up with synchro-
nous superconducting gates and fine-grained clocking).
Also, the relatively low integration density of JJs along
with the need for Splitters to fan out can inflate the cir-
cuit size considerably. Finally, the lack of reliable and
high-capacity superconducting memories imposes its
own distinctive constraints.
Because of these unique characteristics (and limi-
tations) of superconducting, innovation requires that
research efforts focus on computing paradigms and
architectures that depart from classic CMOS-inspired
solutions. Examples of such alternative solutions are
those that 1) use fewer JJs than transistors for the
same information throughput, 2) allow for easier
clocking, and 3) have lower memory requirements.5
We claim that temporal computing satisfies all these
“constraints” and paves a promising path forward.
Race Logic
Race logic is a prime example of computing in the time
domain.2,4 Under race logic, information is repre-
sented as a delay; and a set of basic temporal opera-
tions—FirstArrival (FA), LastArrival (LA), Delay (D),
and Inhibit (IS )—replaces the OR, AND, and NOT we
know from Boolean logic.
Figure 1 illustrates the functionality of these tem-
poral operations. As their names indicate, FA and LA
FIGURE 1. Basic operations of race logic are FirstArrival
(FA), LastArrival (LA), Delay (D), and strict Inhibit (IS ). Dt
corresponds to the delay associated with one time step.
May/June 2021

“fire” as soon as the first and last high inputs arrive,
respectively. Under the assumption that smaller
delays in rise time encode smaller magnitudes,
whereas larger magnitudes are encoded as longer
delays, FA realizes Min function and LA Max function.
The D operation delays an input event by c units of
time; in the shown example, c ¼ 2. Since the arrival
time of the rising edge is what encodes information,
delaying the 0 ! 1 transition by a fixed amount of
time is equivalent to Constant  Addition. Finally, the
IS operation, inspired by the behavior of inhibitory
postsynaptic potentials in the neurons of the neocor-
tex,3 works as a nonlinear filter that has two inputs: 1)
an inhibiting signal and 2) a data signal. If the data sig-
nal arrives first, it is allowed to pass through the gate
unchanged; otherwise, no event occurs on the output
line, which denotes 1. Prior to the next computation,
race logic-driven circuitry must be reset.
Space–Time Algebra
Space–time algebra provides a mathematical under-
pinning for temporal processing.3 Any function defined
over the domain S ¼ f0; Nþ ; 1g, where Nþ is the set of
positive natural numbers, that satisfies the properties
of invariance and causality complies with this algebra.
Interestingly, one can always make an arbitrary func-
tion satisfy these properties with the use of a refer-
ence signal r. Thus, any n-ary function defined over S
is expressible in race logic with at most n þ 1 variables.
FORMALIZATION
The transition from Boolean to race logic gives us the
freedom to set up a clean infrastructure and avoid ret-
rofitting superconductor electronics to existing
abstractions and architectures. To establish a solid
foundation and lay the groundwork for the develop-
ment of systematic methods at all levels, we formally
define the above-described temporal operations.
Computational Temporal Logic
Given the long history of temporal logics in the verifi-
cation community, we use the well-established setting
of linear temporal logic (LTL) as a starting point. The
basic LTL operators are:  sometime in the future, t u
always in the future,  next time (tomorrow), U until,
and R release. Past LTL (PLTL) extends LTL with past-
time operators, which are the temporal duals of LTLs
future-time operators, and allows one to concisely
express statements on the past time instances, such
as: ^ sometime in the past, n always in the past (his-

torically), previous time (yesterday), S since, and T
trigger.6,7
May/June 2021
TOP PICKS
TABLE 1. Semantics of PLTL.
A definition of the semantics of PLTL is provided in
Table 1. The notation hS; ti is used to signify a system
S at time step t. We say that an event f occurs at time
step t in the system S, if f holds at time step t in S,
denoted by hS; ti  f.
These operators allow for the efficient represen-
tation and reasoning of propositions qualified in
terms of time; e.g., an event in a system S has hap-
pened sometime in the past. However, they are
incapable of encapsulating when a formula f holds,
which is essential for race logic. To address this
issue, we introduce the earliest-occurrence function
E hS;ti ðf)
8
< tmin ;
> 9 tmin: tmin 2 ½½fhS;ti :hS; tmin i  f
E hS;ti ðfÞ ¼ and 8j : j 2 ½0; tmin Þ:hS; ji 6 f
>
:
1; otherwise.
This earliest occurrence function receives as
input a formula f and returns the earliest time step
tmin 2 ½½fhS;ti , where ½½fhS;ti is the scope of f at time
step t in the system S (the scope of f denotes an
interval of time steps f operates on at time step t),
such that hS; tmin i  f. If f does not hold at any
time step within ½½fhS;ti , the earliest occurrence
function returns 1, which represents an unreach-
able time step.
Race Logic Semantics
The semantics of FA, LA, D, and IS operations can be
formally defined using existing PLTL operators; as

shown in Table 2 ( c denotes the application of 
operator c times). To extract the step at which these
formulas evaluate to True for the first time in their
scope, E hS;ti ðÞ can be used. For example, E hS;ti ðFAfyÞ
will return the first time step that either f or y hold.
TABLE 2. PLTL-based semantics of race logic operations.
IEEE Micro 73
TOP PICKS
Note that an important property of race logic is that
at most one event is allowed to occur per “wire” across
the entire computation. Bounding switching activity
leads to simple and power-efficient hardware designs,
and it may even allow for an overhead-free transition
between temporal and binary domains.4 In the case of
superconducting, verifying that this property holds is
crucial to the correct operation of the circuits presented
in the section that follows. For example, if more than
one pulse appears per line, the proposed IS design will
not satisfy its specification anymore. A formal definition
of this property—represented here by A—using PLTL
operators follows:
hS; ti  Af iff nðf ! n:fÞ:
SUPERCONDUCTING TEMPORAL
ARCHITECTURES
Realizing Temporal Operations in
Superconducting
The way in which events are encoded plays a critical
role in selecting the hardware that most efficiently
implements logic operators. For example, in CMOS, in
which (under race logic) temporal events are repre-
sented by rising edges, FA and LA functions are real-
ized with a single OR and AND gate, respectively.2,4 An
FIGURE 2. Block diagrams, Mealy machine representations, and WRSPICE simulations of race logic operations implemented in
Rapid SFQ.
74 IEEE Micro
important property of edge-based event encoding is
that it automatically keeps track of the input state at
all times—a signal that has made a transition from a
“low” to a “high” state will not make a transition back
to low in the same computation. This feature breaks
down, though, when dealing with pulses—as dis-
cussed in the “Background” section, superconducting
technologies encode digital data as SFQ voltage
pulses. Pulses naturally return back to their low state,
preventing downstream nodes from implicitly knowing
the state of its predecessors.
A potential solution to this problem is to embed
the state into each gate. Interestingly, the majority
of superconducting single flux quantum (SFQ) ele-
mentary cells have both logic and storage abilities;8
thus, they can be thought of as simple state
machines. To facilitate the mapping between our
temporal operations and existing stateful SFQ ele-
ments, we draw FA, LA, D, and IS as Mealy
machines—as shown in Figure 2. Following this
representation, we build FA with a Merge and a
Destructive Read Out (DRO) cell, LA with a
C-element cell, D with a sequence of
Josephson Transmission Lines (JTL), and IS with an
Inverter cell. The length of the JTL chain depends
on the selected Dt. In contrast to the other shown
cells, JTLs do not hold state; thus, more than one
SFQ pulses can propagate through a JTL chain
May/June 2021

TABLE 3. Area and latency estimates.
Function Area (#JJs) Latency (ps)
FA 10 13
LA 6 8
D 2/JTL 5/JTL
Is 8 11
concurrently. To highlight this difference, we color
the “Init” node in the Dcf Mealy machine gray.
Area and latency results for each of these opera-
tors are provided in Table 3. The shown estimates are
based on our WRSPICE simulations using the MIT-LL
SFQ5ee 10-kA/cm2 process. Note that one of the goals
of this article is to repurpose existing superconducting
cells where possible, like those from Stony Brook.8
Self-Clocked Temporal SFQ Circuits
In contrast to more traditional superconducting
approaches, where each gate is synchronous and fine-
grained clocking is necessary, the proposed scheme
relies on asynchronous operations. That is, even in the
cases of FA and Is , where synchronous cells are used,
a data signal is routed to their “clock” port.
However, the use of synchronous components may
sometimes be beneficial. For example, although
Coincidence—which gets satisfied if and only if both f
and y arrive within the same interval—can be built
from the above-described operations,3 a more effi-
cient implementation is possible: all that is needed for
its realization is a synchronous AND gate.
To avoid costly clock trees and the clock skew
problems that come with them, we propose a data-
driven self-timing (DDST) scheme—as shown in
Figure 3. In a DDST system, timing information is car-
ried by data. More specifically, the required clock sig-
nal is generated locally by a logical OR function
between the data lines, which in the provided example
FIGURE 3. Proposed DDST scheme. The clock signal can be
locally generated from input data at each gate. If no input
pulse arrives, it is safe to assume the operator idle; thus no
clock pulse is required.
May/June 2021
TOP PICKS
is implemented by a Merger. Obviously, the shown
DDST design is not suitable for Boolean circuits; e.g., a
NOT gate must be clocked even in the absence of an
incoming pulse. However, this is not the case for tem-
poral systems, as temporal operators can be safely
considered idle for the time steps that no input pulse
arrives. To extend the “evaluation” window of a syn-
chronous logic block (defined by the delay between
data and clock pulse arrivals), JTLs can be added after
the Merger cell.
EVALUATION
For the evaluation of the proposed logic scheme and
methodology, we build, simulate, and measure the per-
formance of various superconducting temporal accel-
erators. A detailed description of our race trees4
implementation follows. Interested readers can find
additional designs and information in the original
paper9 and our github repository: https://github.com/
UCSBarchlab/Superconducting-Temporal-Logic.
Experimental Setup and Design
Principles
We perform circuit simulations and analysis in both
the open-source WRSPICE and Cadence Spectre plat-
forms using MIT-Lincoln Lab’s SFQ5ee 10-kA/cm2 pro-
cess node. The layout is completed in Cadence
Virtuoso version 16.1. For gate isolation, delay balanc-
ing, and interconnects implementation, JTLs are used.
For fan out, we use Splitters—denoted by s in the
figure that follows. Finally, we define minimum Dt in a
way that is similar to finding the cycle-time in wave-
pipelined architectures.
Proof-of-Concept Design: Race Trees
In the case of race trees (ensembles of decision trees
implemented in race logic), each tree node is consid-
ered an independent temporal threshold function and
realized with a single Inhibit operator. Figure 4(ii)
shows the block diagram of an SFQ race tree corre-
sponding to the decision tree seen in Figure 4(i). For
the construction of NOT gates, which are required for
the implementation of the label decoder, Inhibits and
an upper bound reference signal tub are used. This ref-
erence signal denotes the end of a specific time inter-
val of interest (directly related to the inputs resolution
in this case); thus, NOT will fire at t ¼ tub if and only if
the gate has received no input spikes from time refer-
ence 0 until that moment.
Figure 4(iii) and (iv) provides simulation results and
a layout diagram of this design. In the shown example,
x and y are set to 4 and 1, and tub is equal to 5. As
IEEE Micro 75
TOP PICKS

FIGURE 4. Panel (i): Decision tree with three nodes. Panel (ii): Block diagram of an SFQ race tree. Panel (iii): Simulation results for
x ¼ 4 and y ¼ 1. Panel (iv): Layout diagram (unlabeled JJs are used for interconnection, splitting, routing, and testing purposes).

TABLE 4. Estimated latency results for hardwired race trees important role in the future of high-performance com-
in both CMOS (f = 1 GHz)4 and SFQ (Dt ¼ 25 ps). puting. While there are certainly challenges remaining,
the high-speed and ultralow energy operation of
Depth Input CMOS SFQ Improvement superconductor electronics makes them a promising
res. Latency Latency candidate for this new era.
6 4 bits 17 ns 0.464 ns 37 Continuous and extended effort, mostly at the
6 8 bits 257 ns 6.464 ns 40 device level, has already carried the superconducting
8 4 bits 17 ns 0.490 ns 35
field from the first fabricated JJ in 1970 through the
development of RSFQ logic in 1985 to chips with densi-
8 8 bits 257 ns 6.490 ns 40
ties on the order of several million JJs today. With the
realization of self-shunted JJs in 2017,11 chips with
10 M JJs are now in sight. The excitement around
expected, the final outcome is Label C. The total quantum computing further drives the demand for
latency, for Dt ¼ 25 ps, is 150 ps and the design con- improvement in superconducting circuit fabrication.
sists of 164 JJs; 72 JJs for logic elements, 24 JJs for But these efforts to advance superconducting-based
Splitters, and 68 JJs for JTLs. The number of JJs in computation also highlight a fundamental mismatch
the layout is greater than 164 because of the addi- between the computational abstractions provided by
tional cost of routing and the inclusion of testing cir- these pulse-based devices and the core “logic” we
cuitry. More results and a comparison with CMOS can often assert prematurely as the basis of any comput-
be found in Table 4. ing system.
A change in the underlying technology can dis-
rupt convention and necessitate a complete ret-
CONCLUSIONS AND FUTURE hinking of computers from logic to circuits to
IMPACT architecture. While it remains to be seen if such a
CMOS scaling has been a driving factor for computing radical reconsideration (of computing) is necessary,
technology for many decades and the transistors it is clearly far more likely now than at any other
manufactured today are multiple orders of magnitude time since the creation of the first integrated cir-
more efficient, compact, and less expensive than cuits. The approach taken in this article, which
those built 30 years ago. However, as CMOS integrates ideas from languages, logic design, verifi-
approaches its thermal limits, keeping this trend going cation, circuits, and formal methods, and is info-
becomes increasingly difficult.10 Given this reality, rmed by the physical phenomena underlying the
post-Moore technologies are likely to play an operation of these novel devices, is a useful model

76 IEEE Micro May/June 2021


for exploring other approaches to computation
as well.
The versatility of this method is a broader vision for
post-Moore technological evaluation than is realized
here. However, in line with this objective, we do specif-
ically propose and evaluate the idea that a radically
more efficient foundation for computing lies at the
intersection of superconductor electronics and delay-
coded computation. In particular, we claim that the
natural language for expressing computations in
superconducting is one that can precisely and effi-
ciently describe the temporal relationships between
SFQ pulses. To support this argument, we present a
functional extension to LTL that provides the needed
abstractions to formally capture the capabilities of
computing in the time domain; we show how existing
RSFQ elementary logic cells can be repurposed to real-
ize the desired temporal operations; and we develop
architectures that can safely leverage the extremely
tight timing of superconducting circuits while avoiding
the clocks and memories that shackle more incremen-
tal approaches.
THE CONNECTION THAT THIS ARTICLE
ESTABLISHES BETWEEN PREDICATE
TEMPORAL LOGICS AND COMPUTING
PROVIDES A FRESH PERSPECTIVE ON
THE DEVELOPMENT OF
ABSTRACTIONS THAT CAN LEVERAGE
THE UNIQUE PROPERTIES OF
SUPERCONDUCTOR ELECTRONICS
AND ARE SEMANTICALLY CLOSE TO
THE THEORIES USED BY MODEL
CHECKERS AND FORMAL ANALYSIS
TOOLS.
Looking forward, we expect a growing interest in
emerging technologies, less-traditional computing
paradigms, and device-specific architectures. We see
this work as an important step toward the transition
from strictly Boolean circuits and the von Neumann
computer model to “languages” and designs that bet-
ter exploit the unique characteristics of post-Moore
electronics. Moreover, the connection that this article
establishes between predicate temporal logics and
computing provides a fresh perspective on the devel-
opment of abstractions that are semantically close to
the theories used by model checkers and formal analy-
sis tools.
May/June 2021
TOP PICKS
ACKNOWLEDGMENTS
This work was supported by the National Science Foun-
dation under Grant 1763699, Grant 1730309, Grant
1717779, and Grant 1563935. The research conducted at
LBNL was supported by the Director, Office of Science of
the U.S. Department of Energy under Contract DE-AC02-
05CH11231. We would like to thank James E. Smith, David
Donofrio, Dmitri Strukov, Alexander Wynn, Isaac Mackey,
and the anonymous reviewers for their helpful comments.
REFERENCES
1. D. S. Holmes, A. M. Kadin, and M. W. Johnson,
“Superconducting computing in large-scale hybrid
systems,” Computer, vol. 48, pp. 34–42, Dec. 2015,
doi: 10.1109/MC.2015.375.
2. A. Madhavan, T. Sherwood, and D. Strukov, “Race logic:
A hardware acceleration for dynamic programming
algorithms,” SIGARCH Comput. Archit. News, vol. 42,
pp. 517–528, Jun. 2014, doi: 10.1145/2678373.2665747.
3. J. E. Smith, “Space-time algebra: A model for
neocortical computation,” in Proc. 45th Annu. Int.
Symp. Comput. Archit., 2018, Art. no. 289–300,
doi: 10.1109/ISCA.2018.00033.
4. G. Tzimpragos, A. Madhavan, D. Vasudevan, D. Strukov,
and T. Sherwood, “Boosted race trees for low energy
classification,” in Proc. 24th Int. Conf. Archit. Support
Program. Lang. Operating Syst., 2019, pp. 215–228,
doi: 10.1145/3297858.3304036.
5. S. K. Tolpygo, “Superconductor digital electronics:
Scalability and energy efficiency issues,” Low Temp.
Phys., vol. 42, no. 5, pp. 361–379, 2016, doi: 10.1063/
1.4948618.
6. M. Benedetti and A. Cimatti, “Bounded model checking
for past LTL,” in Tools and Algorithms for the
Construction and Analysis of Systems, H. Garavel and J.
Hatcliff, Eds. Berlin, Germany: Springer, 2003, pp. 18–33,
doi: 10.1007/3-540-36577-X_3.
7. F. Laroussinie, N. Markey, and P. Schnoebelen,
“Temporal logic with forgettable past,” in Proc. 17th IEEE
Annu. Symp. Logic Comput. Sci., 2002, pp. 383–392,
doi: 10.1109/LICS.2002.1029846.
8. K. K. Likharev and V. K. Semenov, “RSFQ logic/memory
family: A new Josephson-junction technology for sub-
terahertz-clock-frequency digital systems,” IEEE Trans.
Appl. Supercond., vol. 1, no. 1, pp. 3–28, Mar. 1991,
doi: 10.1109/77.80745.
9. G. Tzimpragos et al., “A computational temporal
logic for superconducting accelerators,” in Proc.
25th Int. Conf. Archit. Support Program. Lang.
Operating Syst., 2020, pp. 435–448, doi: 10.1145/
3373376.3378517.
IEEE Micro 77
TOP PICKS
10. V. V. Zhirnov, R. K. Cavin, J. A. Hutchby, and G. I.
Bourianoff, “Limits to binary logic switch scaling—A
Gedanken model,” Proc. IEEE, vol. 91, no. 11,
pp. 1934–1939, 2003, doi: 10.1109/JPROC.2003.818324.
11. S. K. Tolpygo et al., “Developments toward a 250-nm,
fully planarized fabrication process with ten
superconducting layers and self-shunted Josephson
junctions,” in Proc. 16th Int. Supercond. Electron. Conf.,
Jun. 2017, pp. 1–3, doi: 10.1109/ISEC.2017.8314189.
GEORGIOS TZIMPRAGOS is currently working toward a Ph.D.
degree with the Department of Computer Science, University
of California, Santa Barbara, Santa Barbara, CA, USA, and is a
Research Affiliate with Lawrence Berkeley National Labora-
tory, Berkeley, CA, USA. His research interest includes com-
puter architecture, currently focusing on emerging computing
paradigms and technologies for energy-efficient processing—
ranging from tiny embedded systems to exotic supercom-
puters. Tzimpragos received an M.S. degree in electrical and
computer engineering from University of California, Davis,
Davis, CA, USA. His alma mater is the National Technical Uni-
versity of Athens, Greece. He is the corresponding author of
this article. Contact him at gtzimpragos@cs.ucsb.edu.
JENNIFER VOLK is currently working toward a Ph.D. degree
with the Department of Electrical and Computer Engineering,
University of California, Santa Barbara, Santa Barbara, CA,
USA. Her research interests include the exploitation of cir-
cuit-based quirks for architectural advancements in novel
technologies, as well as the exploration of tradeoffs between
area, energy, robustness, and sustainability. Volk received a
B.S. degree in electrical engineering from the University of
California, Santa Cruz, Santa Cruz, CA, USA. She is a Member
of IEEE and ACM. Contact her at jevolk@ucsb.edu.
DILIP VASUDEVAN is currently a Research Scientist with the
Computer Architecture Group, Lawrence Berkeley National
Laboratory, Berkeley, CA, USA. His research interests include
post-Moore architectures, reconfigurable spintronic devices,
superconductor electronics, and neuromorphic computing.
Vasudevan received a B.E. degree in electronics and communi-
cations engineering from the University of Madras, Chennai,
India, an M.S. degree in computer systems engineering from the
University of Arkansas, Fayetteville, AR, USA, and a Ph.D. degree
in informatics (computer engineering) from the University of
Edinburgh, Edinburgh, U.K. He is a Professional Member of
ACM. Contact him at dilipv@lbl.gov.
78 IEEE Micro
NESTAN TSISKARIDZE is currently a Research Engineer with
the Department of Computer Science, Stanford University,
Stanford, CA, USA. From 2011 to 2019, she was a Researcher
with Princeton University, Princeton, NJ, USA, the University of
Iowa, Iowa City, IA, USA, and the University of California, Santa
Barbara, Santa Barbara, CA, USA. Her research work focuses
on developing and adapting cutting-edge formal verification
techniques to automate the design and analysis of hardware
and software systems. Tsiskaridze received a Ph.D. degree in
computer science from the University of Manchester, Man-
chester, U.K. Contact her at nestan@stanford.edu.
GEORGE MICHELOGIANNAKIS is currently a Research Sci-
entist with Computer Architecture Group, Lawrence Berkeley
National Laboratory, Berkeley, CA, USA. He has extensive
experience in on- and off-chip networking and computer
architecture. His current research work focuses on emerging
technologies and 3-D integration in the post-Moore era, as
well as optics and architecture for HPC and datacenter net-
works. Michelogiannakis received a Ph.D. degree from Stan-
ford University, Stanford, CA, USA, in 2012. He is a Member of
IEEE and ACM. Contact him at mihelog@lbl.gov.
ADVAIT MADHAVAN is currently a Faculty Specialist with the
University of Maryland, College Park, MD, USA, affiliated with
the National Institute of Standards and Technology. His
research interests include novel encoding schemes, emerging
technologies and architectures, and CMOS VLSI aimed toward
building next generation computing systems. Madhavan
received a Ph.D. degree from the University of California, Santa
Barbara, Santa Barbara, CA, USA, in 2016. He is a Member of
IEEE. Contact him at advait.madhavan@nist.gov.
JOHN SHALF is currently with the Department Head for
Computer Science, Lawrence Berkeley National Laboratory,
Berkeley, CA, USA, and was formerly the Deputy Director of
Hardware Technology for the DOE Exascale Computing Proj-
ect and the Leader of the Green Flash Project. He is a coau-
thor of more than 90 publications in the field of parallel
computing software and HPC technology and corecipient of
three best paper awards. Before joining Lawrence Berkeley
National Laboratory in 2000, he was with the National Center
for Supercomputing Applications, University of Illinois, and
was a Visiting Scientist with the Max-Planck-Institute for
Gravitational Physics/Albert Einstein Institute, Potsdam, Ger-
many. Contact him at jshalf@lbl.gov.
May/June 2021

TIMOTHY SHERWOOD is currently a Professor of Computer
Science with the University of California, Santa Barbara,
Santa Barbara, CA, USA, specializing in the development of
processors that exploit novel technologies (e.g., supercon-
ductors and memristors), provable properties (e.g., security,
privacy, and correctness), and hardware-accelerated algo-
rithms (e.g., high-throughput scanning and new logic repre-
sentations). In 2013, he cofounded Tortuga Logic to bring
rich security analysis to hardware and embedded system
design processes. Sherwood received a B.S. degree in
May/June 2021
TOP PICKS
computer science and engineering from the University of
California, Davis, Davis, CA, USA, in 1998, and M.S. and Ph.D.
degrees in computer science and engineering from the Uni-
versity of California San Diego, San Diego, CA, USA, in 2003.
He is a recipient of the Northrop Grumman Teaching Excel-
lence Award and the ACM SIGARCH Maurice Wilkes Award,
an ACM Distinguished Scientist, and corecipient of 17 differ-
ent “best paper” and “top pick” article awards. Contact him
at sherwood@cs.ucsb.edu.
IEEE Micro 79
 THEME ARTICLE: TOP PICKS

A Next-Generation Cryogenic Processor

Architecture
Ilkwon Byun , Dongmoon Min , Gyuhyeon Lee , Seongmin Na, and Jangwoo Kim, Department of Electrical
Computer Engineering, Seoul National University, Seoul, 08826, South Korea

Cryogenic computing can achieve high performance and power efficiency by

dramatically reducing the device’s leakage power and wire resistance at low
temperatures. Recent advances in cryogenic computing focus on developing
cryogenic-optimal cache and memory devices to overcome memory capacity,
latency, and power walls. However, little research has been conducted to develop a
cryogenic-optimal core architecture even with its high potentials in performance,
power, and area efficiency. In this article, we first develop CryoCore-Model, a
cryogenic processor modeling framework that can accurately estimate the
maximum clock frequency of processor models running at 77 K. Next, driven by the
modeling tool, we design CryoCore, a 77 K-optimal core microarchitecture to
maximize the core’s performance and area efficiency while minimizing the cooling
cost. The proposed cryogenic processor architecture, in this article, achieves the
large performance improvement and power reduction and, thus, contributes to the
future of high-performance and power-efficient computer systems.

H
igh-performance computing and datacenter
industries always require the fastest and the
most power-efficient computer systems.
However, due to the prohibitively increasing CPU
power with clock frequency (i.e., power wall problem)
and the increasing performance gap between the pro-
cessors and memory devices (i.e., memory wall prob-
lem), the server performance and power efficiency
have not meaningfully improved nowadays.
Cryogenic computing, which aims to run com-
puter devices at extremely low temperatures (e.g.,
196  C; 77 K), has emerged as a highly promising
solution to resolve the power wall and memory wall
problems. Cryogenic computing allows us to increase
the server’s performance and power efficiency as fol-
lows. First, with the voltage scaling enabled by the
almost eliminated leakage current, we can increase
the clock frequency while achieving the much lower
power consumption. Second, as the wire resistivity
linearly decreases with the temperature, we can cor-
respondingly benefit from the reduced wire latency
0272-1732 ß 2021 IEEE
Digital Object Identifier 10.1109/MM.2021.3070133
Date of publication 31 March 2021; date of current version
25 May 2021.
and, thus, build much faster computing devices. For
example, the copper’s resistivity at 77 K is six times
lower than the resistivity at 300 K. As a result, cryo-
genic computing can be the breakthrough of the
power wall and memory wall problems and, thus, sig-
nificantly improve the computer’s performance.
Recent advances in cryogenic computing mainly
focus on developing cryogenic-optimal cache and mem-
ory devices to overcome the memory wall problem. For
example, our cryogenic memory research1 proposed the
cryogenic-optimal dynamic random access memory
(DRAM) design, which is 3.80 times faster than the con-
ventional DRAM with the less power consumption. In
addition, our cryogenic cache research2 proposed the
cryogenic-optimal cache design, which provides twice
higher speed with twice larger capacity while achieving
34.1% of power reduction.
Therefore, it is straightforward to develop the cryo-
genic-optimal core design as a next step to overcome
the power wall problem. However, despite its huge
potential, little research has been conducted to
develop the cryogenic-optimal processors. Note that
once the cryogenic-optimal core becomes available, it
will also take full advantage of cryogenic-optimal
cache and memory devices, which leads to a full-
cryogenic computer system.

80 IEEE Micro Published by the IEEE Computer Society May/June 2021


FIGURE 1. (a) Cryogenic processor model (i.e., CC-Model)
overview and (b) validation setup.
Our article, published at ISCA’20,3 presents the
first architecture exploration and proposes a fast,
dense, and power-efficient cryogenic-optimal proces-
sor architecture running at 77 K. To achieve the goal,
the paper tackles following three critical challenges:
1) absence of a cryogenic-core performance model;
2) unclear direction for the architecture optimization;
3) lack of the cryogenic-optimal core design.
CRYOGENIC PROCESSOR
MODELING FRAMEWORK
As the first step to enable the architecture-level study
for the cryogenic processor, we build a modeling tool
to predict the performance of processors running at
77 K. Figure 1(a) illustrates the overview of our vali-
dated cryogenic processor modeling framework, Cryo-
Core-Model (CC-Model). CC-Model takes a target
processor design and its fabrication-process informa-
tion (i.e., MOSFET model card, wire height, and width)
as inputs, and then predicts the peak clock frequency
for a wide range of temperatures including 77 K. For
this purpose, we build CC-Model with three submo-
dels: 1) MOSFET model (cryo-MOSFET), 2) wire model
(cryo-wire), and 3) processor model (cryo-pipeline).
MOSFET Model
Cryo-MOSFET predicts the major MOSFET character-
istics [i.e., on-channel current (Ion ), leakage current
May/June 2021
TOP PICKS
(Ileak )] at cryogenic temperatures by taking a MOSFET
model card as an input. The model card is a set of low-
level MOSFET variables related to the fabrication pro-
cess (e.g., gate-oxide thickness and doping concentra-
tion). Cryo-MOSFET predicts both Ion and Ileak at low
temperatures by automatically adjusting the three
most temperature-dependent MOSFET variables [i.e.,
effective carrier mobility (meff ), saturation velocity
(vsat ), and threshold voltage (Vth )].
To build the MOSFET model, we utilize our previous
model1 as a baseline and improve its accuracy at
smaller technology nodes. The model extension is
based on the industry-provided MOSFET data, which
show that the temperature dependence of meff and vsat
significantly changes with the MOSFET’s gate length.
Wire Model
Cryo-wire generates the on-chip wire characteristic
(i.e., wire resistivity) at low temperatures, based on the
given metal layer’s geometry information (i.e., wire
width and height). The wire resistivity is mainly deter-
mined by the three physical mechanisms: geometry-
independent scattering (rbulk ), grain-boundary scatter-
ing (rgb ), and surface scattering (rsf ). rbulk depends only
on the temperature, whereas rgb and rsf are primarily
determined by the wire geometry. For the accurate pre-
diction, we model both temperature and geometry
dependency based on the previous works.4,5
Processor Model
Cryo-pipeline predicts the critical-path delay of each
pipeline stage at cryogenic temperatures, by utilizing
the low-temperature MOSFET and wire properties
from cryo-MOSFET and cryo-wire, respectively. In
addition, to help the detailed reasoning and analysis,
cryo-pipeline can decompose each critical-path delay
into the transistor and the wire delay portions. There-
fore, using our modeling tool, architects can predict
the frequency speed-up at cryogenic temperatures
and analyze how the low temperatures affect the
delay of each pipeline stage in detail.
For implementing cryo-pipeline, we utilize Syn-
opsys Design Compiler Topographical Mode.6 Design
Compiler can synthesize a Verilog design based on the
logical library (i.e., transistor information) and the
physical library (i.e., metal-layer information) while
reporting the critical-path delay of each stage. By
using Design Compiler, cryo-pipeline first synthesizes
a processor layout for the target processor design
written in Verilog. With the processor layout, cryo-
pipeline then derives the critical-path delay of each
pipeline stage at 77 K by taking the logical and
IEEE Micro 81
TOP PICKS
physical libraries generated from our MOSFET and
wire models. Finally, cryo-pipeline decomposes the
critical-path delay into the transistor and wire delay
portions by applying the no-wire option to Design
Compiler.
Model Validation
We fully validate CC-Model’s accuracy by validating
each submodel. First, we validate our MOSFET model
by comparing it with the HSPICE simulation results
obtained from our industry-provided MOSFET model
card. The industry model card was prevalidated by
actual measurements for the 77–300 K temperature
range. Second, we validate our wire model by compar-
ing it with the measured data from the previous litera-
€ gl et al.7). Finally, we validate the
ture (e.g., Steinho
processor model by comparing its frequency predic-
tion with our measurement from the experimental
setup shown in Figure 1(b). We verify that all the sub-
models achieve high accuracy with low error rates (up
to 4.5%).
DESIGN PRINCIPLES FOR
CRYOGENIC PROCESSOR
Using our validated modeling framework, we reveal
the necessity of microarchitecture-level optimizations
and introduce the design principles for 77 K-optimal
core microarchitectures. Specifically, we emphasize
the importance of minimizing the dynamic power con-
sumption and maximizing the peak clock frequency at
the microarchitecture level.
For this purpose, we conduct performance, power,
and area analysis for the various core models. We
implement the target processor designs by customiz-
ing RISC-V BOOM.8 For the performance analysis, we
utilize CC-Model to analyze the maximum clock fre-
quency of processors running at 77 K. For the power
and die-area analysis, we utilize McPAT9 integrated
with cryo-MOSFET. We also include the power con-
sumption for the cryogenic cooling, as 9.65 times of
the device power consumption at 77 K.1,2,10
Principle 1: Minimize the Dynamic
Power Consumption
We first emphasize the importance of reducing the
dynamic power at the microarchitectural level. To
draw the principle, we perform a case study with the
high-performance reference core model (i.e., Intel Sky-
lake core; hp-core), which represents the high-perfor-
mance datacenter processor. Hp-core has a deep and
wide pipeline, and operates with a high voltage and
clock frequency.
82 IEEE Micro
FIGURE 2. Our analysis with two reference core models.
(a) Power consumption of hp-cores at 300 K and 77 K. (b) Peak
frequency and power consumption of lp-cores at 77 K.
Figure 2(a) shows the power consumption of hp-
cores operating at various temperatures and voltages.
In the graph, 300 K hp and 77 K hp indicate the
processors with hp-core design running at 300 K and
77 K without voltage optimization. In our analysis, we
observe that the huge amount of dynamic power (83%
in 300 K hp) remains at 77 K, which incurs the prohibi-
tively expensive cooling cost (800% in 77 K hp). To
reduce the dynamic power, we can simultaneously
decrease the Vdd and Vth levels thanks to the nearly
eliminated leakage current at 77 K. However, even
with the aggressive voltage scaling, we cannot miti-
gate the huge cooling cost while maintaining the
same performance, as shown in 77 K hp (power opt.).
That is, we cannot realize the power efficiency of cryo-
genic processors only with the voltage optimization.
Therefore, we should minimize the dynamic power at
the microarchitectural level.
Principle 2: Maximize the Clock
Frequency
We then emphasize the importance of achieving the
high frequency at the microarchitectural level. To
draw the principle, we conduct a case study with the
low-power reference core model (i.e., ARM Cortex
core; lp-core) as we highlighted the importance of the
low power consumption in the previous analysis. Lp-
core has the shallower and narrower pipeline with a
lower operation voltage and clock frequency com-
pared to hp-core.
Figure 2(b) shows the result of frequency and
power analysis for three lp-core designs running at
77 K. The three designs share the same core design
but adopt the different voltage levels to adjust their
frequencies. First, lp-core with the nominal voltage
(77 K lp) consumes less power but operates with a
May/June 2021
 TOP PICKS

TABLE 1. Hardware specifications of hp, lp, and cryocore.

Hp-core Lp-core
(i7-6700) (Cortex-A15) CryoCore

# cache load/ 4 1 1
store ports
Pipeline width 8 4 4
Load queue size 72 24 24
Store queue size 56 24 24
FIGURE 3. Optimization process for our processor designs.
Issue queue size 97 72 72
Reorder buffer 224 96 96
much lower frequency compared to 300 K hp-core. In size
other words, we should increase 77 K lp’s frequency by # physical 180 100 100
scaling its operating voltages. However, in our analy- integer registers
sis, we observe that the frequency improvement with # physical float 168 96 96
the voltage scaling is limited due to the transistor registers
speed saturation. As evidence, even though we apply Max frequency 4.0 GHz 2.5 GHz 4.0 GHz
much higher Vdd to 77 K lp (freq. opt.) and 77 K lp
Power per core 24 W 1.5 W 5.5 W
(extreme freq.), lp-core cannot achieve a higher clock (45 nm)
frequency than hp-core within the power budget. That
Core area (45 44.3 mm2 11.54 mm2 22.89
is, we cannot meaningfully improve the performance
nm) mm2
of cryogenic processors only with the voltage scaling.
Therefore, we should also maximize the clock fre- Core & L1/L2 97.51 mm2 17.51 mm2 38.89
area (45 nm) mm2
quency at the microarchitectural level.
Supply voltage 1.25 V 1.0 V 1.25 V
(Vdd )
CRYOGENIC-OPTIMAL
PROCESSOR DESIGN
Following our design principles, we present 77 K-opti- That is, we can integrate twice the number of cores
mized processor designs in terms of both performance under the same area budget.
and power efficiency. Figure 3 summarizes our optimi-
zation process in two steps: 1) microarchitecture opti-
 
mization ( 1 ) and voltage scaling ( 2 ). Vdd and Vth Scaling
Based on CryoCore, we derive two 77 K-optimal pro-
cessors by applying Vdd and Vth scaling. We explore a
Microarchitecture Optimization large number of design points with the different Vdd
We follow the two identified principles and propose and Vth levels, and obtain the power-frequency Par-
our cryogenic-optimal core microarchitecture, called eto-optimal curve, as shown in Figure 3. Among the
CryoCore. Table 1 illustrates CryoCore compared with optimal design points, we choose the two representa-
hp-core and lp-core. First, to achieve the high peak- tive designs: 1) the power-optimal design (cryogenic
frequency at the microarchitectural level, CryoCore low-power core; CLP-core) and 2) the frequency-opti-
adopts the deep pipeline and high operating voltage mal design (cryogenic high-performance core; CHP-
of hp-core. Second, to minimize the dynamic power core). CLP-core is the power-minimized processor
consumption at the microarchitectural level, we take design whose performance is the same as that of
the smaller (or fewer) microarchitecture units and, 300 K hp-core. On the other hand, CHP-core is the fre-
thus, the narrow-width pipeline of lp-core. With these quency-maximized design whose total power con-
optimizations, CryoCore running at 77 K (77 K Cryo- sumption (i.e., sum of the device and cooling power
Core) achieves the much higher peak-frequency and consumption) is the same as that of 300 K hp-core.
the lower power consumption compared to 300 K Note that we intentionally choose the same Vth value
hp-core. In addition, CryoCore consumes only 50% for both designs to implement them in a single hard-
area compared to hp-core, thanks to its narrow pipe- ware with dynamic voltage and frequency scaling
line and the reduced sizes of microarchitectural units. (DVFS). As a result, we achieve both the 1.5 times

May/June 2021 IEEE Micro 83

TOP PICKS
FIGURE 4. Multithread performance of various systems including the full-cryogenic computer system (i.e., CHP-core with 77 K
memory).
higher clock frequency (CHP-core) and the 97% lower
power consumption (CLP-core) with DVFS.
POTENTIAL OF FULL-CRYOGENIC
COMPUTER SYSTEMS
By expanding the cryogenic-computing coverage into
the CPU pipeline, our work enables the evaluation of
full cryogenic computer systems for the first time.
With our proposed cryogenic-optimal processor, we
evaluate the performance and power consumption of
the full-cryogenic computer where the CPU, cache,
and memory devices are optimized for the 77 K
operation.
Evaluation Setup
We evaluate the performance and power consumption
of full cryogenic computer systems with two represen-
tative cryogenic servers: 1) the performance-optimized
77 K server that consumes the same total power com-
pared to 300 K servers, and 2) the power-optimized
77 K server that achieves the same performance com-
pared to 300 K servers.
First, for the performance-optimized 77 K server,
we utilize CHP-core equipped with CryoCache2 and
CLL-DRAM.1 CryoCache achieves twice larger capac-
ity by using 3T-eDRAM technology at 77 K while
achieving twice faster access speed compared to
the conventional static random access memory
(SRAM-based) 300 K caches. CLL-DRAM is the
latency-optimized 77 K DRAM that is 3.80 times faster
than the conventional 300 K DRAM.
Next, for the power-optimized 77 K server, we eval-
uate the power consumption of CLP-core. As the pre-
vious works1,2 already show the significant power
reduction of cryogenic memories (i.e., 34.1% and 9.2%
reduction for the 77 K-optimal cache and DRAM,
respectively), we focus on the power reduction of the
CPU pipeline in this article.
In our evaluation, we utilize Intel i7-6700 equipped
with commodity DDR4-2400 DRAM for the baseline
design. For the simulation, we run 10 PARSEC 2.1 work-
loads11 with gem5 simulator12 and McPAT.9
84 IEEE Micro
FIGURE 5. Total power consumption of various processor
designs including CLP-core.
Performance Improvement
We first show the performance gain of the full-cryogenic
computer systems. Figure 4 shows the performance of
the full-cryogenic computer. The performance values
are the inverse of the execution time and normalized to
that of the baseline.
The full-cryogenic computer systems equipped
with CHP-core and 77 K-optimized memories (i.e.,
CHP-core with 77 K memory) achieve 2.39 times of
speed-up on average, up to 3.41 times in blackscholes.
Note that the average speed-up of the full-cryogenic
computer system (139%) is clearly higher than the
speed-up of the systems with the cryogenic-core only
(CHP-core with 300 K memory; 83%) or the cryogenic-
memory only (300 K hp-core with 77 K memory; 21%).
More importantly, the full system’s speed-up is even
much higher than the aggregated speed-up of 77 K
core only and 77 K memory only (104%), which indi-
cates the synergetic effect of the cryogenic core and
memory system together.
In summary, when designing the full-cryogenic
computer to maximize the performance, architects
can achieve up to 3.41 times of the speed-up within
the same power budget, even including the huge cool-
ing cost.
Power Reduction
Next, we evaluate the power consumption of CLP-
core. Figure 5 shows the power consumption of vari-
ous processor designs including CLP-core. The power
values are normalized to the baseline 300 K hp-core’s
power consumption. As CryoCore is twice denser
May/June 2021

than 300 K hp-core, we doubled the power consump-
tion of CryoCore and CLP-core for the conservative
comparison.
The total power consumption of CLP-core
(62.5%) is much lower than the 300 K hp-core’s
power consumption. First, by adopting our proposed
77 K-optimal microarchitecture design, CLP-core
reduces the device power consumption by 71%.
Next, with the low-power-targeted voltage scaling,
CLP-core further reduces the device power con-
sumption. As a result, CLP-core consumes 94.1%
less power than the 300 K baseline. Even including
the huge cooling cost at 77 K (i.e., 9.65 times of the
device power consumption10), CLP-core reduces the
total power consumption by 37.5% compared to the
300 K baseline.
THIS ARTICLE IS THE FIRST TO SHOW
THE POTENTIAL OF FULL-CRYOGENIC
COMPUTER SYSTEMS THAT
INTEGRATE 77 K OPTIMAL PIPELINE,
CACHE, AND DRAM DEVICES.
Note that CLP-core achieves the same single-
thread performance with doubled throughput thanks
to the twice higher core density. That is, architects
can achieve the same single-core performance and
doubled throughput with the 37.5% lower power con-
sumption by using CLP-core.
IMPACT OF OUR WORK
We believe that our work will have various long-term
impacts to initiate the active research and develop-
ment of cryogenic computer systems. We summarize
the major impacts of our work as follows:
1) the impact on the next-generation computer
systems;
2) the first microarchitectural guidelines of cryo-
genic computing;
3) the first widely applicable and accurate model-
ing framework for cryogenic computing.
Impact on the Next-Generation
Computer Systems
This article is the first to show the potential of full-
cryogenic computer systems that integrate 77 K-opti-
mal pipeline, cache, and DRAM devices. The full-
cryogenic computer achieves the 3.41 times higher
May/June 2021
TOP PICKS
performance or the 37.5% of power reduction, which
has not been seen in other computer-architectural
studies for the general-purpose computing. Thanks
to the large potential, we believe that cryogenic
computing will be considered as a practical and
promising candidate for future computer systems,
especially for the performance-critical and power-
hungry HPC and datacenter industries. We are also
convinced that the significant potential will moti-
vate architects to engage in the cryogenic comput-
ing research area.
First Architectural Guidelines of
Cryogenic Computing
This article is also the first to provide fundamental
guidelines to build cryogenic hardware and architecture.
To run the CPU core with the highest performance and
power efficiency, we propose the 77 K-targeted design
guidelines, which are generally applicable to any types
of hardware components. Our architecture guidelines
will inspire follow-up studies to design and develop vari-
ous cryogenic-optimal computer devices, especially
compute-intensive hardware, such as CPU, GPU, and
accelerators.
First Widely Applicable and Accurate
Modeling Framework
Finally, this article is also the first to propose an accu-
rate and widely applicable modeling framework for
cryogenic computing. As our framework is developed
on top of the commercial-grade digital design tool (i.e.,
Design Compiler of Synopsys), it can model any type
of computer device written in Verilog. In addition, as
we carefully validate the tool with the actual measure-
ments and industry data, the framework guarantees
high accuracy. We believe that our modeling frame-
work will contribute to future cryogenic computer
architecture exploration.
ACKNOWLEDGMENTS
This work was supported in part by the Samsung Elec-
tronics, SK Hynix, National Research Foundation of
Korea (NRF) funded by the Korean Government under
Grant NRF-2019R1A5A1027055, NRF-2019-Global Ph.D.
Fellowship Program, and Grant NRF-2021R1A2C3014131;
in part by the Institute of Information & Communica-
tions Technology Planning & Evaluation funded by the
Korean Government under Grant 1711080972; in part by
the Creative Pioneering Researchers Program through
Seoul National University; and in part by the Automation
and Systems Research Institute (ASRI) and Inter-univer-
sity Semiconductor Research Center at Seoul National
IEEE Micro 85
TOP PICKS
University. Ilkwon Byun and Dongmoon Min contributed
equally to this work.
REFERENCES
1. G.-H. Lee, D. Min, I. Byun, and J. Kim, “Cryogenic
computer architecture modeling with memory-side
case studies,” in Proc. 46th Int. Symp. Comput. Archit.,
2019, pp. 774–787, doi: 10.1145/3307650.3322219.
2. D. Min, I. Byun, G.-H. Lee, S. Na, and J. Kim, “CryoCache:
A fast, large, and cost-effective cache architecture for
cryogenic computing,” in Proc. 25th Int. Conf. Archit.
Support Programm. Lang. Oper. Syst., 2020,
pp. 449–464, doi: 10.1145/3373376.3378513.
3. I. Byun, D. Min, G.-H. Lee, S. Na, and J. Kim, “CryoCore: A
fast and dense processor architecture for cryogenic
computing,” in Proc. ACM/IEEE 47th Annu. Int. Symp.
Comput. Archit., 2020, pp. 335–348, doi: 10.1109/
ISCA45697.2020.00037.
4. R. A. Matula, “Electrical resistivity of copper, gold,
palladium, and silver,” J. Phys. Chem. Reference Data,
vol. 8, no. 4, pp. 1147–1298, 1979, doi: 10.1063/1.555614.
5. C.-K. Hu et al., “Future on-chip interconnect
metallization and electromigration,” in Proc. IEEE Int.
Rel. Phys. Symp., 2018, pp. 4F.1-1–4F.1-6, doi: 10.1109/
IRPS.2018.8353597.
6. Synopsys, “Synopsys DC ultra,” 2019. [Online]. Available:
https://www.synopsys.com/implementation-and-
signoff/rtl-synthesis-test/dc-ultra.html
€ gl, G. Schindler, G. Steinlesberger,
7. W. Steinho
M. Traving, and M. Engelhardt, “Comprehensive study
of the resistivity of copper wires with lateral
dimensions of 100 nm and smaller,” J. Appl. Phys., vol.
97, no. 2, 2005, Art. no. 023706, doi: 10.1063/1.1834982.
8. C. Celio, D. A. Patterson, and K. Asanovic, “The Berkeley
out-of-order machine (BOOM): An industry-
competitive, synthesizable, parameterized RISC-V
processor,” EECS Dept., Univ. California, Berkeley, CA,
USA, Tech. Rep. UCB/EECS-2015-167, 2015.
9. S. Li, J. H. Ahn, R. D. Strong, J. B. Brockman,
D. M. Tullsen, and N. P. Jouppi, “McPAT: An integrated
power, area, and timing modeling framework for
multicore and manycore architectures,” in Proc.
42nd Annu. IEEE/ACM Int. Symp. Microarchit., 2009,
pp. 469–480, doi: 10.1145/1669112.1669172.
10. Y. Iwasa, Case Studies in Superconducting Magnets:
Design and Operational Issues. Berlin, Germany:
Springer, 2009.
11. C. Bienia, S. Kumar, J. P. Singh, and K. Li, “The PARSEC
benchmark suite: Characterization and architectural
implications,” in Proc. 17th Int. Conf. Parallel Archit.
Compilation Techn., 2008, pp. 72–81, doi: 10.1145/
1454115.1454128.
86 IEEE Micro
12. N. Binkert et al., “The gem5 simulator,” ACM SIGARCH
Comput. Archit. News, vol. 39, no. 2, pp. 1–7, 2011, doi:
10.1145/2024716.2024718.
ILKWON BYUN is currently working toward a Ph.D. degree
with the Department of Electrical and Computer Engineer-
ing, Seoul National University, Seoul, South Korea. His
research interests include architecting cryogenic CMOS and
superconductor-based computer systems by using com-
puter architecture modeling and simulation techniques. He
is a Student Member of IEEE and ACM. Contact him at
ik.byun@snu.ac.kr.
DONGMOON MIN is currently working toward a Ph.D. degree
with the Department of Electrical and Computer Engineering,
Seoul National University, Seoul, South Korea. His current
research interests include cryogenic computing for both
CMOS-based and superconductor-based computer systems.
He is a Student Member of IEEE and ACM. Contact him at
dongmoon.min@snu.ac.kr.
GYUHYEON LEE is currently working toward a Ph.D. degree
with the Department of Electrical and Computer Engineering,
Seoul National University, Seoul, South Korea. His research
interests include performance modeling methodologies for
both conventional and cryogenic computer architectures. He
is a Student Member of IEEE and ACM. Contact him at
guhylee@snu.ac.kr.
SEONGMIN NA is currently working toward a Ph.D. degree
with the Department of Electrical and Computer Engineering,
Seoul National University, Seoul, South Korea. His current
research interests include hardware optimization techniques
in various areas including cryogenic computing and deep
learning acceleration. He is a Student Member of IEEE and
ACM. Contact him at seongmin.na@snu.ac.kr.
JANGWOO KIM is currently a Professor with the Department
of Electrical and Computer Engineering, Seoul National Uni-
versity, Seoul, South Korea. His primary research interests
include computer architecture, server and datacenter, sys-
tem modeling, and intelligent systems. Kim received a Ph.D.
degree in electrical and computer engineering from Carnegie
Mellon University, Pittsburgh, PA, USA, in 2008. He is a Mem-
ber of IEEE and ACM. He is the corresponding author of this
article. Contact him at jangwoo@snu.ac.kr.
May/June 2021
THEME ARTICLE: TOP PICKS

Balancing Specialized Versus Flexible

Computation in Brain–Computer Interfaces
Ioannis Karageorgos and Karthik Sriram , Yale University, New Haven, CT, 06520, USA

n Vesel
y , Rutgers—The State University of New Jersey, New Brunswick, NJ, 08901-8554, USA, and Yale
Ja
University, New Haven, CT, 06520, USA
Nick Lindsay and Xiayuan Wen, Yale University, New Haven, CT, 06520, USA
Michael Wu, Rutgers—The State University of New Jersey, New Brunswick, NJ, 08901-8554, USA
Marc Powell, University of Pittsburgh, Pittsburgh, PA, USA
David Borton, Brown University, Providence, RI, 02912, USA
Rajit Manohar and Abhishek Bhattacharjee, Yale University, New Haven, CT, 06520, USA

We are building HALO, a flexible ultralow-power processing architecture for

implantable brain– computer interfaces (BCIs) that directly communicate with
biological neurons in real time. This article discusses the rigid power, performance,
and flexibility tradeoffs that BCI designers must balance, and how we overcome
them via HALO’s palette of domain-specific hardware accelerators, general-
purpose microcontroller, and configurable interconnect. Our evaluations using
neuronal data collected in vivo from a nonhuman primate, along with full-stack
algorithm to chip codesign, show that HALO achieves flexibility and superior
performance per watt versus existing implantable BCIs.

B
y enabling direct brain-computer communica-
tion, brain–computer interfaces (BCIs) can
accelerate the process of scientific discovery,
restore sensory capabilities, mitigate symptoms of
movement disorders like Parkinson’s disease, treat
pharmacologically resistant depression and anxiety,
and even restore motor capabilities for spinal cord
injury, brain strokes, and amyotropic lateral sclero-
sis.1–3 BCIs interrogate biological neurons and decode
pathological behavior or the user’s intent, guiding
stimulation of the brain to mitigate seizures, control
prostheses, actuate assistive devices, and more. BCIs
have even been shown to augment human capabili-
ties; e.g., enhancing short-term memory capacity,
monitoring attention and mental state to enhance
performance, navigating augmented realities via sig-
nals from the motor cortex, and reading signals from
the visual cortex to infer words, pictures, and videos.4
Consequently, Facebook and Microsoft are competing
0272-1732 ß 2021 IEEE
Digital Object Identifier 10.1109/MM.2021.3065455
Date of publication 11 March 2021; date of current version
25 May 2021.
with Neuralink, Kernel, Neuropace, and Medtronic to
build BCIs that read/stimulate an ever-increasing
number of biological neurons with high signal fidelity.5
Modern BCIs designs are of two types. While
some are noninvasive in the form of headsets or
other external devices,1 invasive BCIs surgically
implanted on, around, and in the brain tissue are able
to record and stimulate large numbers of neurons
with higher signal fidelity, spatial resolution, and
tighter real-time characteristics.6 Low-power hard-
ware for onboard processing is critical to the success
of implantable BCIs, especially because elevating tis-
sue temperature by just 1
can damage the brain’s
cellular structure.7
CHALLENGES OF BCI DESIGN
BCI applications must read the electrophysiological activ-
ity of as many biological neurons as possible with high spa-
tial and temporal resolution to be useful. Modern BCIs
extract neuronal activity at data rates of 10–50 Mbps, with
Neuralink demonstrating even orders of magnitude higher
data rates,5 and DARPA’s NESD program targeting com-
munication with millions of neurons.8 These large volumes
of data may need to be processed in real time. For

May/June 2021 Published by the IEEE Computer Society IEEE Micro 87

TOP PICKS

TABLE 1. Existing commercial and research BCIs meet target power budgets by either restricting their scope to a single use
case, or by dropping brain–computer communication bandwidth. HALO is the first flexible implantable BCI architecture to
overcome this tradeoff.

Medtronic Neuropace Aziz Kassiri Neuralink NURIP HALO

2 2 10 2 5 11

Tasks supported
Spike detection • • • • • • @
Compression • • @ • • • @
Seizure prediction • @ • @ • @ @
Movement intent @ • • • • • @
Encryption • • • • • • @
Technical capabilities
Programmable @ Limited • @ • Limited @
Read channels 4 8 256 24 3072 32 96
Data rate (Mbps) 0.01 0.02 9.76 1.32 545 0.13 46
Safety ( < 15 mW) @ @ @ @ • @ @

example, BCIs that treat seizures must read the activity of
biological neurons, process it to detect signs of a seizure
or its imminent arrival, predict the movement of the sei-
zure through different brain regions, determine where to
apply electrical stimulus (and for how long) to mitigate sei-
zure symptoms, and then stimulate brain tissue.9 All of
this must be done accurately, necessitating significant sig-
nal processing of many neuronal channels of data, and
quickly, within milliseconds of seizure detection.9 At the
same time, BCIs may not exceed 15 mW for safe chronic
implantation, a target that is notoriously challenging given
the high data rates that BCIs must support.
BCI APPLICATIONS MUST READ THE
ELECTROPHYSIOLOGICAL ACTIVITY
OF AS MANY BIOLOGICAL NEURONS
AS POSSIBLE WITH HIGH SPATIAL
AND TEMPORAL RESOLUTION TO BE
USEFUL.
Designers have responded by building BCIs that
either achieve power efficiency via specialization for a
restricted set of applications/treatments for specific
disorders in specific brain regions, or more flexible
multiuse designs that achieve power efficiency but
only by restricting the number of neurons they read/
stimulate. Consequently, the modern BCI ecosystem
is fragmented, with many different single-use devices,
and lacks standardization of computational capabili-
ties. Table 1 captures this predicament by
summarizing the limitations of the current state-of-
the-art commercial and research BCIs.
OUR APPROACH: THE HALO
PROJECT
An ideal BCI must be flexible as its operation may
need to be personalized, there may be multiple neuro-
logical conditions to treat, and several brain–com-
puter interactions to support. In response, we are
building HALO, a high-performance, ultralow-power,
and flexible BCI processing architecture. HALO is a
full-stack design effort that uses electrophysiological
data collected in vivo from a nonhuman primate’s
motor cortex (specifically, from the regions responsi-
ble for arm and leg movement) to evaluate a BCI archi-
tecture that balances a palette of power-efficient
accelerators with configurable dataflow to support
frequently used neural processing kernels. We are
realizing HALO via several tape-outs, with Figure 1(a)
illustrating a chip diagram of our HALO architecture in
a 12-nm technology, after augmenting over the 28-nm
technology from the original paper.3 Furthermore,
Figure 1(b) shows how the processing architecture
integrates with the remainder of a typical implantable
BCI device.
In realizing HALO, we make several research contri-
butions. First, we systematically map the design space of
BCI applications to identify a list of target capabilities to
support. Because commercial BCIs are generally single-
use devices, identification of a canonical set of applica-
tions that more flexible BCIs should strive to support has
hitherto remained unanswered. This list includes disease

88 IEEE Micro May/June 2021


FIGURE 1. Chip diagram on the left shows our HALO tape-out in a 12-nm technology. The block diagram on the right shows other key
components of implantable BCIs, including the sensors, which consists of conductive needles that penetrate millimeters of cortical tissue,
analog components, a radio, and power sources. Implantable BCIs are packaged in a hermetically fused silica capsule or titanium capsule.
treatment, signal processing, and secure transmission of
neuronal data (e.g., compression and encryption of extra-
cellular voltage streams). As BCIs are an active area of
research, this list is nonexhaustive. Nevertheless, it iden-
tifies a broader set of tasks needed for a flexible BCI plat-
form as a starting point, while also offering a viable path
to integrate these tasks.
Second, we navigate a large design space of architec-
ture and integration options to realize this list of BCI capa-
bilities by using principled hardware–software codesign.
Standard low-power design dictates that we realize one
accelerator per BCI task in the form of a dedicated ASIC.
We refer to this as a monolithic ASIC design, and find that
they often exceed the 15 mW power budget. In response,
we refactor the underlying algorithm of the original BCI
tasks into distinct pieces that realize different phases of
the algorithm. We refer to these pieces as kernels, and
show that they facilitate design of ultralow-power hard-
ware processing elements (PEs) via novel hardware–soft-
ware codesign approaches. We round out the design with
a low-power RISC-V microcontroller to configure PEs into
processing pipelines and support computation for which
there are currently no PEs. The result is an unconventional
TABLE 2. Overview of hardware–software codesign
techniques used to realize HALO.
Technique Direction
Kernel PE decomposition SW! HW
PE reuse generalization SW! HW
PE locality refactoring SW HW
Spatial reprogramming SW HW
Counter saturation SW$ HW
NoC route selection SW! HW
May/June 2021
TOP PICKS
style of heterogeneity, where a family of accelerator PEs,
each of which is identified in our chip tape-out diagram in
Figure 1(a), operates in unrelated clock domains with low-
power asynchronous circuit-switched communication.
Third, we devise several hardware–software codesign
techniques that raise the level of abstraction of BCI
design from “bits and wires” to architectural choices that
take inspiration from the world of software engineering.
Table 2 summarizes these techniques, which we
discuss in the next section. These approaches
enable HALO to achieve 4–57 and 2 lower power
dissipation than software alternatives and mono-
lithic ASICs, respectively.
COMPUTATIONAL TASKS
SUPPORTED BY HALO
Figure 2 presents an overview of the HALO architec-
ture. The block diagram on the left shows the PEs in
our design and the configurable interconnect used to
assemble PEs to realize the task pipelines shown on
the right. HALO supports BCI tasks ranging from those
that require real-time closed-loop support for treat-
ment of neurological disorders to those that exfiltrate
neural recordings to external systems for postprocess-
ing and batch analysis. The first category consists of
support for seizure treatment and amelioration of
movement disorders. Seizure prediction/stimulation
pipelines that break neuronal feedback loops are
responsible for seizure severity present cutting-edge
capabilities of FDA-approved clinical BCIs. So do algo-
rithms to detect/stimulate the brain to counteract
movement disorders associated with essential tremor
and Parkinson’s disease. HALO supports FFTs, cross-
correlation, and bandpass filters over linear models to
support closed-loop treatment of these neurological
disorders.
IEEE Micro 89
TOP PICKS
FIGURE 2. HALO consists of low-power hardware PEs and a RISC-V microcontroller. The PEs are configured into pipelines to
realize tasks ranging from compression (in blue) to spike detection (in green). Optional PEs (e.g., AES encryption) are shown in
square brackets. PEs operating in parallel (e.g., FFT, XCOR, and BBF for seizure prediction) are shown in curly brackets.
The second workload category includes compres-
sion to reduce radio transmission bandwidth. Apart
from some specific and well-understood forms of lossy
compression—such as spike sorting—BCIs generally
require lossless compression. HALO supports spike
detection via the near-energy operator (NEO) PE, and
also implements several lossless compression var-
iants as their effectiveness can change across brain
regions and patient activity. We support lossless LZ4
and LZMA compression, as well as discrete wavelet
transform (DWT) compression. Compression ratios
vary by as much as 40% depending on compression
algorithm and target brain region.3
Finally, although state-of-the-art BCIs do not cur-
rently support encryption, we foresee it as necessary
in future BCIs for secure data exfiltration. HIPAA,
NIST, and NSA require using AES with an encryption
key of at least 128 bits.
HALO ARCHITECTURE
HALO supports five tasks, and can set up two of them
in multiple ways, leading to a total of eight distinct
pipelines configurable by a doctor or technician. The
RISC-V microcontroller is used to configure these pipe-
lines via the programmable switches. With the conven-
tional monolithic ASIC approach, this means that we
would implement eight ASICs. However, we decom-
pose these pipelines into the PEs of Figure 2.
Decomposing BCI Tasks Into PEs
Kernel PE Decomposition: Some BCI tasks consist of
distinct computational kernels naturally amenable to
PE decomposition. For example, seizure prediction
combines kernels for FFT, cross-correlation (XCOR),
Butterworth bandpass filtering (BBF), and a support
vector machine (SVM). We realize each as a PE, as
shown in Figure 2. As FFT, XCOR, and BBF have no
data dependencies, they can operate in parallel. This
approach saves power because XCOR contains com-
plex computation (e.g., divisions and square roots)
90 IEEE Micro
that scales quadratically with channel count. In con-
trast, BBF is a simple filter with minimal arithmetic
that scales linearly with channel count. Separating
XCOR and BBF into separate PEs ensures that BBF’s
filtering logic is clocked over an order of magnitude
slower than the logic for cross-correlation.
PE Reuse Generalization: Many BCI tasks use
computational kernels slightly differently. We develop
configurable PEs that can be shared among BCI tasks.
Consider movement intent, which can be decom-
posed into FFT, followed by logic that checks whether
the FFT output is in a particular spectral range. We
create a threshold PE (THR) to determine when a PE’s
output is within a specified numerical range and
enable sharing of the FFT between movement intent
and seizure prediction tasks. The FFT PE is configura-
ble because movement intent requires 14–25-point
FFTs to detect drops in signal power, while seizure pre-
diction requires 1024-point FFTs.
Algorithm 1. LZMA pseudocode
1: Function LZMA_compress_blockinput
2: output ¼ listðlzma headerÞ;
3: while data ¼ input:getðÞdo
4: best match ¼ find best matchðdataÞ;
5: Probmatch ¼ countðtablematch ; best matchÞ
6: =count totalðtablematch Þ;
7: r1 ¼ range encodeðProbmatch Þ;
8: output:push backðr1Þ;
9: increment counterðtablematch ; best matchÞ;
10: end while
11: Return output;
12: end function
Major Refactoring: PE decomposition can require
significant refactoring of the original algorithm. Con-
sider LZMA and DWTMA compression. Both algo-
rithms use Markov (MA) chains to calculate the
probability of the current input value based on
observed history, which is used to pick more efficient
encoding of the input signal. We found that using the
May/June 2021

combined MA PE overshoots the 15-mW power bud-
get. To solve this problem, we refactored the original
algorithm to make it more amenable for PE decompo-
sition. To separate algorithmic phases, we realize that
data locality (i.e., following routines that manipulate
major data structures) is a good indicator of kernel
boundaries within programs. This observation is tied
to the fact that PEs in HALO have only local memories
and cannot share large amounts of data. Locality
refactoring highlights how design decisions about the
architecture (i.e., use of PE-local memories) guided
refactoring of our algorithms.
Algorithm 1 demonstrates how we use this insight
to change LZMA. The second half of this algorithm
can be separated into probability calculations and fre-
quency information updates centered around the
maintenance of the core MA data structure, the fre-
quency table (in green), as well as efficient encoding
(in blue). This refactoring permits bringing together
phases that operate on the same data structures
within the hardware, allowing us to separate the PEs
since they can now operate independently with mini-
mal data movement. This permits clocking each com-
ponent at significantly lower frequency, leading to
power savings of 2.
PE Optimizations
Unchanged PE Output: Some of the PEs (e.g., XCOR
and LZ) process data in blocks instead of samples and
wait for all inputs in the block to arrive, before comput-
ing in a bursty manner. Bursty computation is problem-
atic as it requires either large buffers to sink the bursts
or high PE frequency to meet data rates while sustain-
ing periods of bursty activity. Neither is ideal from the
perspective of saving power. To achieve power
improvements, we spatially reprogram the original
algorithm and codesign it with the hardware. Consider
the XCOR PE. The original algorithm performs compu-
tation at the end once all data have been filled into the
block. We refactor the algorithm to process inputs as
early as they are available. The final form in Algorithm 2
reduces the amount of computation needed in the final
step, as well as the number of buffers needed to store
the inputs. This translates to a power savings of 2.2
over the original algorithm. This technique also extends
to other PEs like LZ to achieve 1.5 power reduction.
Finally, LZ and MA PEs require initialization of data
structures at the beginning of every compressed
block. We found that dedicated circuits are necessary
to meet the 15 mW power budget. These circuits use
only combinational logic and reduce PE power con-
sumption by 1:8.
May/June 2021
TOP PICKS
Algorithm 2. XCOR spatial programming refactoring
1: function XCORinput; output
2: // channel[][] stores input in appropriate channel location
3: channel½channel num½sample num ¼ input
4: // data[] stores sums of input received so far
5: data½countþ ¼ input
6: // data_lag[] stores sums of input till LAG
7: If count 2 ¼¼ LAGthen
8: data lag½count ¼ data½count
9: end if
10: // Finish correlation computation
11: if channel:filledðÞthen
12: for eachi; j 2 channels do
13: avg i ¼ data½i=SIZE
14: avg j ¼ ðdata½j  data lag½jÞ=SIZE
15: output:push backðavg i; avg jÞ
16: end for
17: return output
18: end if
19: end function
Modified PE Output: Although initialization circuits
decrease the direct power/performance cost of start-
ing a new compression block, there is also an indirect
cost of using uninitialized internal structures, which
leads to lower compression rates. This presents a prob-
lem with respect to the choice of block size. Large
block sizes lead to better estimates of frequencies, but
small block sizes allow the use of smaller data types
and reduce the memory footprint and power of the MA
PE. One might balance power/compression ratio for an
ideal design, but such an approach does not find a
design point that fits within the constrained power
budget. Instead, we observe that the frequencies of
values within a block remain largely unchanged after
they have stabilized. Consequently, we allow the fre-
quency counters to saturate and set block size inde-
pendently of counter bit width. Overall, counter
saturation modification allows HALO to benefit both
from reduced memory footprint of 16 bit counters, and
better compression ratio of larger blocks.
On-Chip Network
Each PE operates at the lowest frequency needed for
data processing rates, and synthesize with established
synchronous design flows. While running PEs in sepa-
rate clock domains saves power, it can potentially com-
plicate inter-PE communication. Prior work on globally
asynchronous locally synchronous (GALS) architec-
tures12 encountered these issues for packet-switched
on-chip networks. Unfortunately, we cannot repurpose
their solutions as our analysis with the DSENT tool
IEEE Micro 91
TOP PICKS

estimates that a simple packet-switched mesh net-

work consumes over 50 mW, well over our 15-mW
power budget. Instead, we codesign inter-PE communi-
cation with the BCI algorithms. The decomposition of
BCI tasks into kernels creates static and well-defined
data-flows between PEs. NoC route selection allows
replacement of a packet-switched network to a far
lower power circuit-switched network built on an
asynchronous communication fabric.

FIFO Interfaces
Since the publication of our original paper on HALO,3
one challenge that we encountered is that our GALS-
based approach requires careful data rate matching
between PEs in separate clock domains. We use per-PE
FIFO buffers to transfer data from the network into the
form expected by the PE and to perform this rate
matching. Consider PEs f and g. If their computation is
regular—i.e., the functions produce and consume data
in a perfectly periodic fashion—then a simple interface
between the two for clock domain conversion suffices.
However, if f produces bursty data or g consumes data
in bursts, then a FIFO is needed between f and g to
smooth out producer–consumer patterns. The size of
this FIFO is determined by the computational proper-
ties of f and g, and the frequencies at which they oper-
ate. Increasing the frequency of g beyond the
minimum operating point to meet data throughput
needs would reduce the FIFO size required. We have
found that balancing FIFO size with PE frequency is key
to meeting the 15-mW power budget.
EVALUATION
Our 15-mW target power budget includes the HALO
chip, sensors, ADC, amplifier, and radio technologies.
We assume a microelectrode array with 96 channels,
each of which records each sample encoded in 16 bits
at a frequency of 30 kHz, yielding a data rate of
46 Mbps. After accounting for all analog components,
HALO’s processing pipelines (including the radio)
must consume no more than 12 mW. All results pre-
sented use a commercial 28-nm fully depleted silicon-
on-insulator (FD-SOI) CMOS process except when
noted otherwise. Synthesis and power analysis is per-
formed using Cadence synthesis tools with standard
cell libraries from STMicroelectronics.
We use electrophysiological data collected from the
brain of a non-human primate. Microelectrode arrays
were implanted in two locations in the motor cortex,
corresponding to the left upper and lower limbs. We use
recordings of brain activity while the animal performed
tasks such as walking on a treadmill, reaching for a treat,
FIGURE 3. Power (in log-scale) of PEs, control logic, and
radios for HALO versus RISC-V and monolithic ASICs. To
meet the 15-mW device power budget, these components
(without ADCs and amplifiers) need to be under 12 mW (the
red line). We compare HALO against the lowest power RISC-
V and HALO-no-NoC, which shows how much power would
be saved if HALO’s configurability were sacrificed.
and overcoming a moving styrofoam obstacle. All
research protocols were approved and monitored by
Brown University’s Institutional Animal Care and Use
Committee, and all research was performed in accor-
dance with relevant guidelines and regulations.
HALO PRESENTS A WET LAB-TO-CHIP
DESIGN PROJECT THAT EXPLORES
THE QUESTION OF HOW TO BUILD A
FLEXIBLE ULTRALOW-POWER
PROCESSING ARCHITECTURE FOR
NEXT-GENERATION BCIS.
Figure 3 compares HALO’s power versus ASICs
and software alternatives on RISC-V. Software tasks
can execute on microcontroller cores in both single-
core and multicore designs, where we divide the 96
channel data streams and operate on them in paral-
lel. We study 1–64 RISC-V core counts and report the
best configuration per task. We also show an ideal-
ized version of HALO where the on-chip interconnect
is removed to quantify the power penalty for the con-
figurability that the network offers. Both HALO var-
iants use the optimizations from the ones described
in the “PE Optimization” section. HALO uses less
power than monolithic ASICs and RISC-V approaches.

92 IEEE Micro May/June 2021


Finally, as we have been extending our chip design
efforts, we have discovered the crucial impact of FIFO
design on total power. For each PE, we have evaluated
the power utilized for various configurations with fre-
quency and input and output FIFO buffers. For each
frequency, we select the lowest FIFO size required for
the design, and report its power. For example, for the
LIC PE, we have found that the lowest power configu-
ration is achieved at 24 MHz, with an 8-entry input
FIFO and no output FIFO. We also note that power
consumed can vary by as much as 1 mW depending on
these configuration options.
CONCLUSION
HALO presents a wet lab-to-chip design project that
explores the question of how to build a flexible ultra-
low-power processing architecture for next-genera-
tion BCIs. While this work performs an initial explora-
tion of workloads that are important for neuroscience,
the list of tasks can be expanded. Future BCIs will
implement other workloads, with different pipelines
targeting different research and medical objectives.
Because of its modular design, HALO will be able to
support such workloads seamlessly.
REFERENCES
1. A. L. S. Ferreira, L. C. D. Miranda, and E. E. Cunha de
Miranda, “A survey of interactive systems based on
brain-computer interfaces,” SBC J. Interactive Syst.,
vol. 4, no. 1, pp. 3–13, 2013, doi: 10.5753/jis.2013.623.
2. H. Kassiri et al., “Closed-loop neurostimulators: A
survey and a seizure-predicting design example for
intractable epilepsy treatment,” IEEE Trans. Biomed.
Circuits Syst., vol. 11, no. 5, pp. 1026–1040, Oct. 2017, doi:
10.1109/TBCAS.2017.2694638.
3. I. K. et al., “Hardware-software codesign for brain-
computer interfaces,” in Proc. 47th Annu. Int. Symp.
Comput. Archit., 2020, pp. 391–404, doi: 10.1109/
ISCA45697.2020.00041.
4. C. Cinel, D. Valeriani, and R. Poli, “Neurotechnologies
for human cognitive augmentation: Current state of
the art and future prospects,” Front., Hum. Neurosci.,
vol. 13, 2019, Art. no. 13, doi: 10.3389/fnhum.2019.00013.
5. E. Musk and Neuralink, “An integrated brain-machine
interface platform with thousands of channels,” J. Med.
Internet Res., vol. 21, no. 10, 2019, Paper e16194, doi:
10.1101/703801.
6. I. Stevenson and K. Kording, “How advances in neural
recording affect data analysis,” Nature Neurosci.,
vol. 14, pp. 139–42, 2011, doi: 10.1038/nn.2731.
May/June 2021
TOP PICKS
7. P. D. Wolf, “Thermal considerations for the design of an
implanted cortical brain-machine interface (BMI),”
Indwelling Neural Implants: Strategies for Contending
With the in Vivo Environment, W. M. Reichert, Eds. Boca
Raton (FL): CRC Press/Taylor & Francis, 2008, ch. 3, doi:
10.1201/9781420009309-11.
8. DARPA, Bridging the Bio-Electronic Divide Accessed:
Aug. 10, 2019. [Online]. Available: https://www.darpa.
mil/news-events/2015-01-19.
9. S. Li, W. Zhou, Q. Yuan, and Y. Liu, “Seizure
prediction using spike rate of intracranial EEG,” IEEE
Trans. Neural Syst., vol. 21, no. 6, pp. 880–886, Nov.
2013, doi: 10.1109/TNSRE.2013.2282153.
10. J. N. Y. Aziz et al., “256-Channel neural recording and
delta compression microsystem with 3D electrodes,”
IEEE J. Solid-State Circuits, vol. 44, no. 3, pp. 995–1005,
Mar. 2009, doi: 10.1109/JSSC.2008.2010997.
11. G. O’Leary et al., “NURIP: Neural interface processor
for brain-state classification and programmable-
waveform neurostimulation,” IEEE J. Solid-State
Circuits, vol. 53, no. 11, pp. 3150–3162, Nov.
2018, doi: 10.1109/JSSC.2018.2869579.
12. M. Krstic and E. Grass, “New GALS technique for
datapath architectures,” in Proc. Int. Workshop
Power Timing Modeling, Optim. Simulation, 2003,
pp. 161–170, doi: 10.1007/978-3-540-39762-5_18.
IOANNIS KARAGEORGOS is currently an associate research
scientist at Yale University, New Haven, CT, USA. His primary
research interests focuses on the general area of VLSI,
including GALS architectures, logical and physical SoC/ASIC
design, and DTCO. Karageorgos received a Ph.D. degree in
electrical engineering from KU Leuven and IMEC, Belgium.
Contact him at ikarageo@aya.yale.edu.
KARTHIK SRIRAM is currently a Ph.D. student at Yale Univer-
sity, New Haven, CT, USA. His research interests include com-
puter systems and architecture, hardware-software codesign,
specially in the design of brain–computer interfaces. Sriram
received a B.S. degree in computer science from Rutgers Uni-
versity. He is the corresponding author of this article. Contact
him at karthik.sriram@yale.edu.

VESELY
JAN
is currently a software engineer with Nvidia.
His interests span the areas of architecture, operating
systems, and compiler techniques for accelerators. Vesel
y
contributed to this work while he was a visiting student at
Yale. Vesel
y graduated in 2021 from Rutgers University. His
thesis focused on hardware and software methods of inte-
grating accelerators into heterogeneous systems. Contact
him at jan.vesely@rutgers.edu.
IEEE Micro 93
TOP PICKS
NICK LINDSAY is currently a graduate student at Yale Uni-
versity, New Haven, CT, USA. His interests include building
secure, safe, and high-performance heterogeneous systems.
Lindsay received a B.Eng. degree in electrical engineering
degree from the University of Glasgow. Contact him at
Nick.Lindsay@yale.edu.
XIAYUAN WEN is currently a Ph.D. student at Yale University,
New Haven, CT, USA. Her research interests include com-
puter architecture and circuit design. Wen received the B.S.
degree from Nanjing University and an M.S. degree from Yale
University. Contact her at xiayuan.wen@yale.edu.
MICHAEL WU is an incoming Ph.D. student at Yale University,
New Haven, CT, USA. His research focuses on the applications
of machine learning in computer systems. Wu received a bache-
lor’s degree in computer science from Rutgers University, New
Brunswick, NJ, USA. Contact him at mw811@cs.rutgers.edu.
MARC POWELL is currently a postdoctoral associate with
the Department of Neurological Surgery, University of
Pittsburgh. His research focuses on the development of
advanced neural technologies designed to provide unprec-
edented access to the nervous system and apply these
tools to the treatment of neurological dysfunction caused
by injury or disease. A major goal of his work is to facili-
tate the clinical translation of these devices and ensure
that they are implemented safely and reliably. Powell
received a bachelor’s degree in biomedical engineering
from Georgia Institute of Technology in 2014 and a Ph.D.
degree in biomedical engineering from Brown University in
2021. Contact him at marc_powell@pitt.edu.
94 IEEE Micro
DAVID BORTON is currently an assistant professor of bio-
medical engineering at the Brown University School of Engi-
neering, the Carney Institute for Brain Science, and is also a
biomedical engineer at the Providence Veterans Affairs Cen-
ter for Neurorestoration and Neurotechnology, New Haven,
CT, USA. He leads an interdisciplinary team of researchers
focused on the design, development, and deployment of novel
neural recording and stimulation technologies. His team lever-
ages engineering principles to untangle the underpinnings of
sensorimotor and neuropsychiatric disease and injury. Borton
received a B.S. degree in biomedical engineering from Wash-
ington University in St. Louis in 2006 and a Ph.D. degree in
bioengineering from Brown University in 2012. He was a Marie
Curie Postdoctoral Fellow at the Ecole Polytechnique Frale de
Lausanne. Contact him at david_borton@brown.edu.
RAJIT MANOHAR is currently a John C. Malone Professor of
Electrical Engineering and a professor of computer science at
Yale University, New Haven, CT, USA. His research focuses on
the design and implementation of asynchronous circuits and
systems. Manohar received a Ph.D. degree in computer sci-
ence from Caltech. Contact him at rajit.manohar@yale.edu.
ABHISHEK BHATTACHARJEE is currently an associate pro-
fessor of computer science at Yale University, New Haven,
CT, USA. His research focuses on computer architecture and
systems at all scales of computing, ranging from server sys-
tems for large-scale data centers to embedded systems for
implantable brain– computer interfaces. Bhattacharjee
received a bachelor’s degree in engineering from McGill Uni-
versity in 2005 and a Ph.D. from Princeton University in 2010.
Contact him at abhishek@cs.yale.edu.
May/June 2021
THEME ARTICLE: TOP PICKS

Virtual Logical Qubits: A Compact

Architecture for Fault-Tolerant Quantum
Computing
Jonathan M. Baker , Casey Duckering , David I. Schuster , and Frederic T. Chong , University of Chicago,
Chicago, IL, 60637, USA

Fault-tolerant quantum computing is required to execute many of the most

promising quantum applications. In recent years, numerous error correcting codes,
such as the surface code, have emerged which are well suited for current and future
limited connectivity 2-D devices. We find quantum memory, particularly resonant
cavities with transmon qubits arranged in a 2.5-D architecture, can efficiently
implement surface codes with around 20
fewer transmons via this work. We
virtualize 2-D memory addresses by storing the code in layers of qubit memories
connected to each transmon. Distributing logical qubits across many memories has
minimal impact on fault tolerance and results in substantially more efficient logical
operations. Virtualized logical qubit (VLQ) systems can achieve fault tolerance
comparable to conventional 2-D transmon-only architectures while putting within
reach a proof-of-concept experimental demonstration of around ten logical qubits,
requiring only 11 transmons and 9 attached cavities.

Q
uantum devices have improved significantly
in the last several years both in terms of
physical error rates and number of usable
quantum bits (qubits). Concurrently, great progress
has been made at the software level such as improved
compilation procedures reducing required overhead
for program execution. These efforts are directed at
enabling noisy intermediate-scale quantum (NISQ)1
algorithms to demonstrate the power of quantum
computing and are expected to run some important
programs.
Despite this, these machines will be too small
for error correction and unable to run large-scale
programs due to unreliable qubits. The ultimate
goal is to construct fault-tolerant machines capable
of executing thousands of gates and in the long
term to execute large-scale algorithms with speed-
ups over classical algorithms. There are a number
of promising error correction schemes which have
been proposed such as the surface code.2,3 The
0272-1732 ß 2021 IEEE
Digital Object Identifier 10.1109/MM.2021.3072789
Date of publication 13 April 2021; date of current version
25 May 2021.
surface code is a particularly appealing candidate
because of its low overhead, high error threshold,
and its reliance on few nearest-neighbor interac-
tions in a 2-D array of qubits, a common feature of
currently popular hardware like superconducting
transmon qubits.
Current architectures for both NISQ and fault-tol-
erant quantum computers make no distinction
between the memory and processing of quantum
information. While currently viable, as larger devices
are built, the engineering challenges of scaling to hun-
dreds of qubits become readily apparent. For trans-
mon technology, some of these issues include
fabrication consistency and crosstalk during parallel
operations. Every qubit needs dedicated control wires
and signal generators, which fill the refrigerator the
device runs in. To scale to the millions of qubits
needed for useful fault-tolerant machines,4 we need
to adopt a memory-based architecture to decouple
qubit count from transmon count.
We use a recently realized qubit memory technol-
ogy, which stores qubits in a superconducting cavity.5
Stored in cavity, qubits have a significantly longer life-
time (coherence time), but must be loaded into a
transmon for computation. We design and evaluate a
system-level organization of these components within

May/June 2021 Published by the IEEE Computer Society IEEE Micro 95

TOP PICKS
FIGURE 1. Our fault-tolerant architecture with random-
access memory local to each transmon. On top is the typical
2-D grid of transmon qubits. Attached below each data trans-
mon is a resonant cavity storing error-prone data qubits
(shown as black circles). This pattern is tiled in 2-D to obtain
a 2.5-D array of logical qubits. Our key innovation here is stor-
ing the qubits that make up each logical qubit (shown as
checkerboards) spread across many cavities to enable effi-
cient computation.
the context of a novel surface code embedding and
fault-tolerant quantum operations.
Our proposed 2.5-D memory-based design is a typi-
cal 2-D grid of transmons with memory added, Figure 1.
This can be compared with the traditional 2-D error
correction implementation in Figure 2, where the
checkerboards represent error-corrected logical
qubits. The logical qubits in this system are stored at
unique virtual addresses in memory cavities when not
in use. They are loaded to a physical address in the
transmons and made accessible for computation on
request and are periodically loaded to correct errors,
similar to DRAM refresh. This design allows for more
efficient operations such as the transversal CNOT
between logical qubits sharing the same physical
address, i.e., colocated in the same cavities. This is not
possible on the surface code in 2-D, which requires
methods such as braiding or lattice surgery for a
CNOT operation.
We develop an embedding from the standard
representation to this new architecture, which
reduces the required number of physical transmon
qubits by a factor of approximately k, the number of
resonant modes per cavity which is expected to be at
least 10 and is expected to improve over time. We also
develop a Compact variant saving an additional pffiffiffiffiffi 2
.
This means we can obtain a code distance 2k times
greater or use hardware with only 1=ð2kÞ the required
96 IEEE Micro
FIGURE 2. Typical 2-D superconducting qubit architecture.
The dots are transmon qubits where black are used as data
and gray are used as ancilla for error correction. The lines
indicate physical connections between qubits that allow
operations between them. Four logical qubits, each consist-
ing of nine error-prone data qubits are shown here in the
rotated surface code with distance 3. Z parity checks are
shaded yellow (light) and X parity checks are shaded blue
(dark) where checks on only two data are drawn as half
circles.
physical transmons for a given algorithm. In the near-
to-intermediate term, when qubits are a highly con-
strained resource, this will accelerate a path toward
fault-tolerant computation. In fact, the smallest
instance of Compact requires only 11 transmons and 9
cavities for about ten logical qubits. Via simulation, we
determine the error correction threshold rates for
each and find they are all close to the baseline thresh-
old meaning the additional error sources do not signif-
icantly impact the performance.
BACKGROUND
Superconducting Qubit Architectures
In contrast to other leading qubit technologies such
as trapped ion devices with one or more fully con-
nected qubit chains, superconducting qubits are typi-
cally connected in nearest neighbor topologies, often
a 2-D mesh on a regular square grid. This limitation
makes engineering these devices easier but results in
high communication costs, increasing the chance of
errors on NISQ devices, and communication conges-
tion for error corrected operations. More background
on superconducting hardware can be found by Krantz
et al.6
May/June 2021

Qubit Memory Technology
Recently, studies have demonstrated random access
memory for quantum information.5 Qubit states can
be stored in the resonant modes of physical supercon-
ducting cavities attached to a transmon qubit and
depicted as the individual cylinders in Figure 1. Cur-
rently demonstrated error rates are promising, and
there is nothing fundamental preventing this technol-
ogy from becoming competitive with other transmon
devices as it matures. We expect operation error rates
to improve, cavity sizes and coherence times to
increase and in general expect performance to
improve as it has with other quantum technologies.
Local memory is not free. Stored qubits cannot be
operated directly. Instead, operations are mediated
through the transmon. To operate on qubits stored in
memory, we first load the qubit from memory. Then,
we perform the desired operation on the transmons,
and store the qubit back in its original location. A two-
qubit operation such as a CNOT can also be per-
formed directly between the transmon and a qubit in
its connected cavity by manipulating higher states of
the transmon. Qubits stored in the same cavity cannot
be operated on in parallel. There are two primary bene-
fits of this technology. First, we are able to quickly per-
form two-qubit interactions between any pair of
qubits stored in the same cavity. Second, qubits
stored in the cavity are expected to have longer coher-
ence times by about one order of magnitude.
Surface Codes
The surface code2 is one of the most promising quan-
tum error correction protocols because it requires
only nearest neighbor connectivity between physical
qubits and improvements continue to be made.7 The
surface code is implemented on a 2-D array of physical
qubits shown in Figure 2. These qubits are either data,
where the state of the logical qubit is stored, or ancilla
used for syndrome extraction (parity checks). These
ancilla qubits are measured to stabilize the entangled
state of the data. These ancilla fall into two categories,
measure-Z and measure-X for Z syndromes and X syn-
dromes designed to detect bit and phase errors,
respectively.
Each X (Z) plaquette corresponds to a single mea-
sure-X (Z) qubit and the four data, which it interacts
with. The corners of each plaquette are the data
qubits. For the baseline, we use standard Z and X syn-
drome extraction (parity measurement) circuits where
the qubits of this circuit are physical qubits. The Z-syn-
drome measures the bit-parity of its corner qubits and
the X-syndrome measures their phase parity. By
May/June 2021
TOP PICKS
repeatedly performing syndrome extraction and
detecting parity changes, we are able to locate errors.
This repeated syndrome extraction collapses any error
to a correctable Pauli error and forces the data to
remain in what is called the code state. We may detect
errors which occur as changes in measurement out-
comes of the parity checks.
There are two primary ways to manipulate the logi-
cal qubits of the surface code to perform desired logi-
cal operations–braiding and lattice surgery. In this
article, we will primarily consider lattice surgery, which
has been shown to have some advantages over braid-
ing like using fewer physical qubits. For a more thor-
ough introduction to lattice surgery, we refer the
reader to Horsman et al.3,8,9 In our proposed scheme,
all primitive lattice surgery operations can be used
such as split and merge, which together perform a log-
ical CNOT. For universal quantum computation in sur-
face codes, we allow for the creation and use of magic
states such as jTi or jCCZi.
VIRTUALIZED LOGICAL QUBITS
Our proposed architecture is an embedding of the sur-
face code, which virtualizes logical qubits, saving in
required number of transmons. This takes advantage
of quantum resonant cavity memory technology to
store logical qubits, in the form of surface code
patches, in memory local to the computational
transmons.
Natural Surface Code Embedding
Our embedding slices the plane of surface code tiles
into many pieces, storing them flat in memory to
enable them to stitch together on-demand. This
embedding enables the fast transversal CNOT and
high connectivity.
For every transmon in this architecture (the com-
pute qubits in the top layer of Figure 1), there is a cav-
ity attached with a fixed number of resonant modes k.
Each cavity can store k qubits, one per mode. Each
transmon can load and store qubits from its attached
cavity. All transmons can be operated on in parallel as
is the case in most superconducting hardware. We
expect this technology to allow cavity size k on the
order of 10 to 100 qubits.
Consider the rotated surface code of Figure 2 and
the high level view of this architecture in Figure 1. We
map each of the physical qubits of this logical qubit to
the same mode z of each cavity in this memory archi-
tecture. Another logical qubit can be mapped to a dif-
ferent mode of the same set of cavities. We view this
as stacking the surface code patches, the logical
IEEE Micro 97
TOP PICKS

qubits, together under the same set of transmon two logical qubits. To do this, one of the qubits must
qubits. The transmons themselves are only used for be moved to the same 2-D address as the other using
logical operations and error correction cycles per- a move operation described by Litinski.8 Once the two
formed on the patches. qubits are in the same 2-D address, the transversal
In this memory architecture, we are unable to CNOT can be applied.
operate on qubits stored in the same cavity in parallel,
however we are permitted to operate on qubits stored
Compact Surface Code Embedding
in different cavities in parallel. In order to detect mea-
In the previous scheme, half of the transmons did not
surement errors, we require d, the distance of the
have attached cavities (or they did not make use of
code, rounds of syndrome extraction before we per-
them). An ancilla and data qubit could share a trans-
form our decoding algorithm and correct errors. We
mon because the data are stored in the cavity the
can load a logical qubit (meaning load all data in paral-
majority of the time and the ancilla are reset every
lel to each transmon), perform all d rounds of extrac-
cycle. This leads to a more efficient, Compact embed-
tion, then store the qubit, this is our All-at-once
ding which halves the required number of transmons.
strategy. Alternatively, we can interleave the extrac-
This comes at the cost of additional loads and stores
tion cycles by loading the logical qubit in index 0, per-
from memory due to contention during error correc-
forming one syndrome extraction step, then storing.
tion, effectively trading some error and time for signifi-
We execute this same procedure for every logical
cant space savings.
qubit in the stack and repeat d times.
This mapping results in plaquettes, which resemble
Up to k logical qubits share the same set of trans-
triangles rather than squares, where the center of the
mons thereby more efficiently storing these qubits
hypotenuse of each triangle corresponds to both the
than on a single large surface. To interact logical
ancilla qubit and the data qubit, stored “beneath” in its
qubits in different stacks, we load them in parallel to
cavity. Every data qubit is still mapped to the same
the transmons then interact them via lattice surgery
index. We illustrate this transformation from our
operation. In these cases, all of the other stacks’ trans-
undistorted Natural surface code patch to Compact in
mons between the interacting logical qubits act as a
Figure 3.
single logical ancilla. Furthermore, physical operations
This new mapping also requires a new syndrome
between qubits in the same cavity enable our system
extraction procedure because data cannot be loaded
to perform fast transversal two-qubit interactions if
while a transmon is in use as an ancilla. A single round
the logical qubits are colocated in the same stack.
of syndrome extraction can be executed by dividing
the plaquettes into four groups, with each group con-
taining noninterfering plaquettes. Two plaquettes are
Transversal CNOT
noninterfering if they do not share their ancilla with
A major advantage of this 2.5-D architecture is the
any data qubits of the other plaquette. It is imperative
ability to do two-qubit operations transversely using
this process use both the minimum number of loads
the third dimension. The logical operation is per-
and stores and keep data qubits loaded for as short a
formed directly by doing the same physical gate to
time as possible as the error incurred during this cir-
every data qubit and correcting any resulting errors.
cuit directly impacts the error threshold for the code.
For typical 2-D error correcting codes like the surface
Error correction can be performed interleaved or all-
code, transversal two-qubits operations are not possi-
at-once just as with natural.
ble because the corresponding data qubits of two logi-
cal patches cannot be made adjacent. However, with
memory, it is possible to load one patch into the trans- Beyond the Surface Code
mons and apply two-qubit gates mediated by each The surface code is an appealing choice for currently
transmon onto the data qubits for a second qubit available superconducting devices because of its rela-
stored in one mode of the cavities. The transversal tively low overhead and it requires only limited nearest
CNOT can be performed in a single round of d error neighbor connectivity. For this new memory-based
correction cycles while the lattice surgery CNOT takes architecture, there is a fortuitous match with the sur-
six rounds. This can translate to major savings in run- face code, making it an even more appealing candi-
time for algorithms. date. However, there are many other candidate error
The transversal CNOT is not limited to logical correction codes such as the color code.
qubits currently stored in the same 2-D address. With One particularly relevant class of codes for this
an extra step, it is possible to transversely interact any architecture are Bosonic codes.10 The surface and

98 IEEE Micro May/June 2021

 TOP PICKS

FIGURE 3. Transformation from natural to compact. (a) Natural embedding: Only data have attached cavities (not shown). (b)
The transformation: Z ancilla (over yellow/light areas) merge with the upper-right data transmon and X ancilla (over blue/dark
areas) merge with the lower left data transmon. The opposite parings are key to keeping 4-way grid connectivity. (c) Compact
embedding: All ancilla transmons without attached cavities have been removed. All remaining transmons have cavities and are
used as both data and ancilla.

color codes protect quantum information by using
many physical qubits to construct a single logical
qubit. For Bosonic codes, instead we can create
redundancy by using many modes of a single physical
system. The modes of the cavities in the underlying
hardware of our systems can be used to implement
Bosonic codes directly. Unfortunately, Bosonic codes
in practice only approximately correct errors and do
not have the property that as you scale the size of the
code you can obtain an exponential reduction in the
logical error rate of the system. Bosonic codes can be
used effectively for error mitigation.
It is unclear, given an architecture, what the best
error correction scheme is. Ideally, we want a code
which takes full advantage of the high connectivity
between information stored in the same cavity. For
practical demonstrations, we also want a code which
requires a small number of transmons. Bosonic codes
are viable options for this architecture and have a
somewhat natural fit. It is yet to be seen what is the
best choice, however we have shown the surface
code is a fortuitous match with this new memory-
based architecture.
EVALUATION
Error Threshold Results
We detail our threshold results in Figure 4. We study
five different code distances in order to obtain the
physical error threshold value. The threshold value
indicates at which point increasing the code distance,
d improves the logical error rate instead of hurting it.
This threshold is a function of both the physical sys-
tem model, the chosen syndrome extraction circuit,
and the specific decoding procedure. The major differ-
ence in each procedure is the additional error sources
and different syndrome extraction procedures. The
slopes for each code distance compared across the
various schemes is stable, indicating each scheme
improves at a similar rate, post error threshold, and

FIGURE 4. Error thresholds for the baseline 2-D architecture and natural and compact variants of our 2.5-D architecture. The
thresholds are comparable to the baseline indicating the space savings obtained in our system does not substantially reduce
the error thresholds. The slopes of the lines in this figure indicate, postthreshold, how much improvement in physical error rates
improve logical error rate. Except for the baseline, all use a cavity size of 10.

May/June 2021 IEEE Micro 99

TOP PICKS
FIGURE 5. Sensitivity of logical error rate to various error sources in compact (Interleaved). The logical error rates are most sen-
sitive to physical error of loads/stores and SC-SC (transmon–transmon) gates. The logical error rate is less sensitive to transmon
and cavity coherence times (not shown) and mostly insensitive to effects of cavity size.
showing that the logical error rate decays exponen-
tially with d as desired. This is significant because it
means we will be able to save on total number of
transmons without major degradation of the error
threshold.
Error Sensitivity Results
Different system-level details affect the threshold of
the code. Here we focus on compact, interleaved as
the most efficient physical qubit mapping and subject
to a wide variety of errors. The results of these sensi-
tivity studies are found in Figure 5. The logical error
rate is sensitive to a particular error source’s probabil-
ity if the slope of the line is pronounced at the marked
reference value. The logical error rate for compact,
interleaved is sensitive to all changes in system-level
details to some degree. The gate error rates show the
highest sensitivity, indicating improvement in these
will give the greatest benefit. Coherence times (plots
not shown) are not quite as sensitive but the slightly
over 10
offset between the cavity and transmon
plots shows that there is no benefit in transmon T1
being longer than 1=10 cavity T1 when the cavity size is
10. The lines taper off, indicating other errors sources
eventually dominate. Initially, we expected the cavity
size to have a large impact on the logical error rate.
However, when coherence times are high and gate
error rates are fairly low below the threshold, the logi-
cal error rate does increase proportional to the length
of the cavity but the effect is very minor. Given cavi-
ties with good coherence times, this indicates our pro-
posed system will be able to scale smoothly into the
future as cavity sizes increase.
While larger cavity sizes will make this architecture
even more advantageous, there will be a point at
which it has a vanishing benefit because the delay
between error correction becomes too long and
100 IEEE Micro
decoherence error dominates. For the error rates used
in the evaluation, we find that cavity decoherence
error starts dominating after cavity size k  150. After
this point, it would be more beneficial to improve the
cavity coherence time.
CONCLUSION
Current NISQ machines are powerful demonstrations,
but fall short of many serious applications without
error correction. This article makes an error-corrected
machine 20
easier to build by exploiting quantum
memory with a codesigned architecture to enable
medium-scale quantum machines (100–1000 trans-
mons) and allow the industry to realize the long-term
potential of scalable quantum machines.
Current quantum computers are noisy and they
are incapable of running sizable programs accurately.
There is currently a major gap between what is avail-
able on the market and what is required to execute
the famed quantum algorithms with quantum error
correction. Currently, approaches toward bridging this
gap fall into a few categories: either push the error
rates of current devices down below the error thresh-
old of known codes, or design new codes. Our work
bridges this gap in a completely different way by
exploring the use of new technology, resonant cavi-
ties, for the design of new architectures which better
support error correction codes already available.
Locally accessible quantum memory can be used
to create a 2.5-D architecture better suited for the
code than traditional approaches. Our architecture
enables the execution of the surface codes with lat-
tice surgery operations with significantly lower physi-
cal requirements resulting in higher distance codes
with fewer total transmon qubits. What does this
mean? Given a fixed physical error rate below the
threshold, we can use larger codes to obtain strictly
May/June 2021
 TOP PICKS

better logical error rates. This enables the execution of JONATHAN M. BAKER is currently working toward a
longer input programs. Conversely, for a fixed desired Ph.D. degree with the University of Chicago, IL, USA. His
logical error rate, determined by the application, we
research is primarily focused on vertical integration of the
can run on hardware with worse error rates which will
quantum computing hardware-software stack. Contact
be available years sooner. This demonstrates the ben-
him at jmbaker@uchicago.edu.
efit of codesigning quantum architectures alongside
the applications and technologies.
CASEY DUCKERING is currently working toward a Ph.D.
degree with the University of Chicago, Chicago, IL, USA, aim-
REFERENCES ing to efficiently bring together quantum algorithms and error
1. J. Preskill, “Quantum computing in the NISQ era and
correction with their physical implementations on quantum
beyond,” Quantum, vol. 2 , 2018, Art. no. 79.
computers. Contact him at cduck@uchicago.edu.
2. A. G. Fowler, M. Mariantoni, J. M. Martinis, and
A. N. Cleland, “Surface codes: Towards practical large-
scale quantum computation,” Phys. Rev. A, vol. 86,
no. 3, 2012, Art. no. 032324. DAVID I. SCHUSTER is an Associate Professor with the
3. C. Horsman, A. G. Fowler, S. Devitt, and R. Van Meter, Department of Physics, University of Chicago, Chicago, IL,
“Surface code quantum computing by lattice surgery,” USA. His work is currently centered on superconducting
New J. Phys., vol. 14, no. 12, 2012, Art. no. 123011. quantum circuits to make quantum memories, perform error

4. C. Gidney and M. Ekera, “How to factor 2048 bit RSA
correction, and create topologically protected qubits. Schus-
integers in 8 hours using 20 million noisy qubits,”
ter received a Ph.D. degree from Yale University in 2007. Con-
Quantum, vol. 5, p. 433, Apr. 2021.
5. R. Naik et al., “Random access quantum information tact him at david.schuster@uchicago.edu.
processors using multimode circuit quantum
electrodynamics,” Nature Commun., vol. 8, no. 1, 2017,
Art. no. 1904. FREDERIC T. CHONG is the Seymour Goodman Professor
6. P. Krantz, M. Kjaergaard, F. Yan, T. P. Orlando, with the Department of Computer Science, University of Chi-
S. Gustavsson, and W. D. Oliver, “A quantum engineer’s
cago, Chicago, IL, USA. He is also Lead Principal Investigator
guide to superconducting qubits,” Appl. Phys. Rev.,
for the EPiQC Project (Enabling Practical-scale Quantum
vol. 6, no. 2, 2019, Art. no. 021318.
Computing), an NSF Expedition in Computing. He was a fac-
7. J. P. Bonilla-Ataides, D. K. Tuckett, S. D. Bartlett,
S. T. Flammia, and B. J. Brown, “The XZZX surface code,” ulty member and Chancellor’s fellow at UC Davis from 1997
2020, arXiv:2009.07851. to 2005. He was also a Professor of Computer Science, the
8. D. Litinski, “A game of surface codes: Large-scale Director of Computer Engineering, and the Director of the
quantum computing with lattice surgery,” Quantum, Greenscale Center for Energy-Efficient Computing at UCSB
vol. 3, 2019, Art. no. 128. from 2005 to 2015. Chong received a Ph.D. degree from MIT
9. L. Lao et al., “Mapping of lattice surgery-based
in 1996. He is a recipient of the NSF CAREER Award, the Intel
quantum circuits on surface code architectures,”
Outstanding Researcher Award, and ten best paper awards.
Quantum Sci. Technol., vol. 4, no. 1, 2018, Art. no.
He currently serves on the National Quantum Initiative Advi-
015005.
10. W. Cai, Y. Ma, W. Wang, C.-L. Zou, and L. Sun, “Bosonic sory Committee and is the Chief Scientist and Co-Founder of
quantum error correction codes in superconducting Super.tech, a quantum software company. Contact him at
quantum circuits,” Fundam. Res., vol. 1, pp. 50–67, 2021. chong@cs.uchicago.edu.

May/June 2021 IEEE Micro 101

 IEEE COMPUTER SOCIETY JOBS BOARD

Evolving Career
Opportunities
Need Your Skills
Explore new options—upload your resume today

Changes in the marketplace shift demands for vital skills

www.computer.org/jobs and talent. The IEEE Computer Society Jobs Board is a
valuable resource tool to keep job seekers up to date on
the dynamic career opportunities offered by employers.
Take advantage of these special resources for job seekers:

JOB ALERTS TEMPLATES WEBINARS

CAREER RESUMES VIEWED

ADVICE BY TOP EMPLOYERS

No matter what your career level, the IEEE Computer

Society Jobs Board keeps you connected to
workplace trends and exciting career prospects.
EDITOR: Michael Mattioli, michael.mattioli@gs.com

DEPARTMENT: SECURITY

The Next Security Frontier: Taking the

Mystery Out of the Supply Chain
Michael Mattioli , Goldman Sachs & Co., New York, NY, 10282, USA
Tom Garrison and Baiju V. Patel, Intel Corporation, Mountain View, CA, 94040, USA

The modern technology supply chain is highly complex. Throughout the various
stages of design, manufacture, assembly, transport, and operation, a compute
device is subject to tampering (malicious or otherwise). This exposes end users and
consumers of compute devices to a variety of risks at varying levels of impact. Two
potential technology solutions, which aim to provide transparency and insight into
the systems in use, are proposed. The success and adoption of these choices, or any
other solutions developed, is highly dependent on the participation of ecosystem
partners such as foundries, original device manufacturers (ODMs), and original
equipment manufacturers (OEMs).

B
y the time a computing system (e.g., laptop,
desktop, server, smart watch, tablet, etc.) is
delivered to its intended end user, the sum of
its parts has traveled through a highly complex
supply chain. This supply chain, illustrated in Figure 1,
includes diverse component suppliers, subsystem
manufacturers, integrators, and original equipment
manufacturers (OEMs).
The final product may go through several ware-
houses and may be transported/handled via several
shipping companies before finally being received,
unboxed, and deployed/used.
Considering ever-increasing threats to this supply
chain; end users have a growing need to know that
the final product they received is indeed the product
they ordered. Unintentional mistakes/errors, poor han-
dling, or intentional fraud are key risks at play. Addi-
tional risks may come from malicious actors, including
nation states and well-funded criminal organizations
who are motivated to tamper with systems in the sup-
ply chain. The consequences of these risks may
include financial or reputational harm.
Frankly, many treat an arbitrary computing system
as a “black box” and blindly trust the supplier(s) and
0272-1732 ß 2021 IEEE
Digital Object Identifier 10.1109/MM.2021.3072174
Date of current version 25 May 2021.
transport methods. However, a growing portion of
the population, such as financial services or gov-
ernment institutions and even consumers are
keen to understand and gain insight into the sys-
tems they use to store and process sensitive infor-
mation as well as conduct high-stakes transactions
(e.g., financial, health care, education, etc.). They
are actively taking steps to ensure that the com-
puting system, as delivered, meets their risk profile
and can fulfill their compliance, security, and per-
formance requirements.
It is important for the modern technology supply
chain ecosystem to take measures to ensure a grow-
ing portion of the population can trust the supply
chain with increased accuracy and decreased cost.
Improving transparency in the supply chain will help
meet the need for security and quality assurance
among broader portions of the population as both
awareness and risks continue to grow.
KEY RISKS TO SECURITY
Any typical component or system changes hands doz-
ens of times from inception to deployment and, ulti-
mately, retirement. The supply chain is a continuous,
evolving process and, as illustrated in Figure 2, may
not end even when it leaves the end user’s hands (e.g.,
recycle or donate).
Participants in the supply chain, such as the OEMs,
also treat the role they play as intellectual property

May/June 2021 Published by the IEEE Computer Society IEEE Micro 103
SECURITY
FIGURE 1. High-level, simplified depiction of the modern technology supply chain.
(IP) or some sort of “business secret,” which makes it
even more challenging to evaluate and manage risk.
Design
The first window of opportunity to insert risk into a
computing system is by attacking the design and
manufacturing of the individual components that will
eventually comprise the system. Schematics can be
altered, design tools can be compromised, and collab-
oration solutions can be manipulated to alter a com-
ponent from its intended composition. Subsequent
risks are introduced when end users work with the
supplier, such as an OEM, to design and/or configure a
platform (e.g., build-to-order systems). During such
engagements, there is room for unintentional human
error, such as misinterpretation of phone calls and
emails. Risks might also include intentional malicious
action that can cause harm in the process. Behind the
scenes, the vendor also works with their respective
suppliers, such as original design manufacturers
(ODMs), to do similar functions such as source sub-
components and/or assemble components; this fur-
ther exacerbates the potential for human error.
Assembly and Manufacturing
Further down the line, during assembly and manufactur-
ing, there are more opportunities for malicious actions
FIGURE 2. High-level depiction of the supply chain lifecycle.
104 IEEE Micro
such as circuit design modification (e.g., hardware tro-
jans, scan chain attacks, etc.), firmware modifications,
and even counterfeiting.1 A disruption to the supply
chain, such as a factory fire (local) or the 2020 COVID-19
pandemic (global), may also force OEMs to substitute
parts to ensure timely delivery of computing systems to
end users. The many levels and layers of the supply
chain make it nearly impossible to keep a watchful eye
on every single party involved and ensure that no harm,
either intentional or unintentional, was done.
Shipment
After a system has left the factory, there are many
opportunities for tampering, modifications, or changes
within the hardware, firmware, and software. Once a
device is received and deployed, it may be serviced
(e.g., components replaced) in different locations or
by different parties throughout its lifecycle. It is practi-
cally impossible to have complete confidence that a
system has not been tampered with in some manner.
The potential for tampering applies to each of the
functional components of a computing system. For
example, a solid-state drive (SSD) shipped to an ODM
for integration into a computing system could be tem-
pered with by having the firmware within the drive
replaced with a malicious version.
Operation
Transparency in the supply chain helps the end user
ensure that they received exactly what was ordered.
This is a significant step in maintaining continuous
trust in the computing ecosystem. Since most com-
puting systems are frequently upgraded with the lat-
est functional and security updates, it is extremely
important to maintain the current state of the system
throughout its entire lifecycle. Ideally, one should have
the complete and latest information (e.g., firmware
version updates) about each critical component of
the computing system, and should be able to validate
the version updates and configurations against the
May/June 2021

expected state in real-time with reasonable assurance
that nothing malicious has modified the system to an
unacceptable configuration/state. Granted, this might
be more feasible for large-scale commercial entities
but is rather complex and dizzying for consumers; in
either case, it is difficult to achieve.
Impact
Just as causes of unintended changes to the platform
vary, the impact to the end user can vary from benign
to more serious consequences rooted in malicious
intent, such as the following.
› Financial—Stealing sensitive information can
lead to financial gain through access to nonpub-
lic information.
› Reputational—Exposing and fabricating informa-
tion, or interrupting operations, can cause severe
reputational damage to a person/company.
› Revenge—Personal attacks against a company
or organization by a disgruntled employee or end
user can cause potential damages ranging from
physical alterations, compromise of IP, to
destruction of reputation.
› Chaotic—Some individuals and criminal organi-
zations simply want to “watch the world burn.”
Different entities, private and public, are exposed
to these and other threats at varying levels. It is imper-
ative that supply chain transparency is designed to
scale to not just different size businesses but to con-
sumers as well over time.
POWER OF TRANSPARENCY
As might be expected, transparency has a cost.
Beyond the obvious, managing the supply chain
means keeping track of each component through
manufacturing, warehousing, and delivery. This, in
turn, limits the flexibility of the manufacturer.
For example, a manufacturer may wish to have
multiple sources for a keyboard and would want to be
flexible in selecting the keyboard to be integrated into
a laptop based on the availability of parts and the cost
at any given time. However, one may require or specify
a keyboard from a specific supplier, which may impact
a timely delivery and also introduce challenges and
additional cost due to inventory management.
Inventory may also be impacted due to unex-
pected natural calamities such as the 2020 COVID-19
pandemic. In order to deal with such disruptions, as
well as those caused by smaller unforeseen events, it
May/June 2021
SECURITY
is typical to include multiple sources for every signifi-
cant component to support business continuity.
At the same time, suppliers also need to protect
their confidential business information including com-
plex business relationships, procurement processes,
and inventory management. This type of information
in the wrong hands can have significant impact to the
business and can lead to compliance issues. There-
fore, the ecosystem needs to align on the right tech-
nology as well as processes to minimize misuse of
transparency while meeting end user requirements.
Additionally, transparency needs to maintain a bal-
ance between the needs of supply chain and the
needs of the end user and consider a baseline that
may be available to everyone with additional levels of
detail requiring different agreements.
TRUST REQUIRES INDUSTRY-WIDE
PARTICIPATION
A computing system is commonly composed of vari-
ous components. Many of these components can be
classified as “smart”; these are components that typi-
cally contain their own firmware. Firmware has signifi-
cant functional capabilities including the ability to
access, and potentially manipulate, critical or sensitive
(e.g., personal, mission-critical, etc.) data.
The smart components are typically manufactured
by different suppliers and, in turn, are composed of both
active and passive subcomponents. It is important that
one has full transparency of active components due to
their significant capabilities to access and modify data.
Passive components, such as resistors, capacitors, phys-
ical packaging, printed circuit boards (PCBs), or displays
can also pose a risk to one’s data if not implemented
correctly. For example, if PCB routing had certain wires
close to the surface or implemented sockets for expan-
sion or options, an adversary might be able to easily sub-
stitute or even add a component without the knowledge
of the end user.
While it may not be practical to have full transpar-
ency at each passive component level, it is important,
at a minimum, to have transparency at the active
component level and to establish trust in the entire
supply chain; this requires an industry-wide initiative
and agreement on key technical underpinnings in
order for the initiative to be successful. With the ulti-
mate goal of achieving full transparency, we recom-
mend starting small yet critical components of the
computing system first and then develop technology
and processes to progress ahead.
While we do not attempt to specify a comprehensive
list of smart components for initial implementation,
IEEE Micro 105
SECURITY
some examples that the industry should consider as
starting points are the CPU, SSD, wireless (e.g., Blue-
tooth, WiFi, etc.) interfaces, motherboard, and any/all
embedded controllers (ECs). It is not enough to know
the manufacturer of these components or subsystems,
one needs to know details such as the firmware running
on these components, the date and location of manu-
facture, and even the design revision.
PROPOSED SOLUTIONS
Technology choices we make to establish transpar-
ency will have long-term impact on the industry both
from longevity and cost perspective. We propose two
models which are 1) use of a ledger or database and 2)
a method of self-reporting.
Ledger or Database
As a computing system goes through assembly and
delivery in the supply chain, each entity makes an entry
in a ledger, thus, creating a record that can be retrieved
later by the end user to meet their need for transpar-
ency. Once a computing system is delivered to the end
user, the same process can be used to record changes,
such as a change of ownership, to maintain a continu-
ous record for the entire lifespan of the computing sys-
tem. Two fundamental approaches exist here.
Centralized Approach
A trusted third-party maintains the ledger or database
of all the transactions and manages updates to and
retrieval of information per agreement between
trusted third party and members of the supply chain.
The fundamental trust model is based on the idea that
without collaboration between multiple member com-
panies in the supply chain, any inconsistency (inten-
tional or otherwise) might not be detected. For
example, if a system manufacturer entry shows they
shipped 1M computing systems with a specific SSD
from a specific supplier, a corresponding entry by the
supplier in their database is necessary to corroborate.
Blockchain Approach
Application of blockchain technology to ingredient sup-
ply chains has been widely discussed. A 2019 Fortune
article outlined Bumble Bee Tuna’s use of blockchain to
create an electronic and distributed ledger chronicling a
fish’s capture, processing, and travel history to ensure
freshness and product quality2. Beyond this ability to
track ingredients, the distributed ledger can be expanded
to keep a running and growing record over the opera-
tional lifecycle for each system. Tightly controlled and
permissioned ledger entries can continue for updates,
106 IEEE Micro
changes, and ownership transitions that occur as the
system travels through distribution and integration and
then, ultimately, provisioning, operation, and modifica-
tion by the end user or owner. The private and public
cryptographic controls inherent in existing blockchain
infrastructures can be applied to balance the simulta-
neous needs of all ecosystem participants. Component
suppliers can contribute product information into the
blockchain without the worry of sharing sensitive prod-
uct details to competitors. Similarly, system manufac-
turers can create system-level ledgers from component
and subsystem vendors in private transactions shared
only with those whose access permissions have been
cryptographically proven. Ownership can be transferred
to operators who can manage ledger updates to track
key information regarding location, application, upgrades,
updates, and usage statistics. The usefulness of this led-
ger can extend from the resale of the device into the sec-
ondary market; the ledger can provide sellers with better
insights into possible IP loss based on the usage and
application while providing buyers a better understand-
ing of the provenance of the device.
Self-Reporting
Tracking the components in the supply chain while the
device is manufactured and readied to be delivered to
the end user helps ensure that intended components
are used in the device. Once the device is in the hands
of the end user, the device can provide a cryptographic
report (i.e., attestation) of the current configuration of all
of the smart components in the device. The configura-
tion can include current firmware version, security ver-
sion, hardware ID, etc. Each report may further include
information about nonactive or passive components
that are part of the smart component or even include a
report-out of smart subcomponents as well as revision
or change log for additional transparency.
In order to establish trust into the report, the smart
component needs to include a cryptographic key and
associated certificate, as depicted in Figure 3, and,
when queried, produces a signed report along with the
certificate so that the end user can validate the
authenticity and trust in certificate and subsequently
trust the report.
Finally, the entire computing system can either col-
lect report-out from each of the smart components
and may produce a comprehensive report for the
entire system or may leave it up to the end user to
query each smart component of interest. The benefit
of a system-level report-out is that the end user gets
the full picture without necessarily having to under-
stand how to query each of the components (as the
May/June 2021

FIGURE 3. Simplified illustration of computer components
with a certificate of authenticity.
manufacturer of the system has the best understand-
ing). This would require the system to also have a
trusted smart component that cannot only collect all
the information but is able to produce a cryptographi-
cally signed comprehensive report.
The shortcoming of this type of approach is that
each component has an additional cost of obtaining a
certificate and complexity of providing a report-out.
Additionally, the end user does not have the benefit of
transparency from any component that does not
implement this capability.
GOVERNANCE
There are two potential paths toward solution adop-
tion. The first is a self-regulated or market-driven
approach where each participant in the supply chain
may choose their own degree of participation in trans-
parency. End users, based on their purchasing prefer-
ence, will drive supply chain to adopt the “right” level
of transparency to meet market needs. The other is by
means of an industry group or consortium in which
computing system supply chain participants form a
consortium to establish technical direction and a gov-
ernance model to provide consistent and sufficient
transparency to meet broad industry needs.
RECOMMENDATION
With increasing reliance on information technology
for business-critical and personal data, having trans-
parency in the supply chain, as well as information on
the current state of the computing systems in use, is
vital to the economic health of the information tech-
nology ecosystem. Transparency by itself is of the
limited value without the information needed to
assess risk or compliance. Beyond the ability to verify
that what the end user received is indeed what they
ordered, there is an additional value of transparency
May/June 2021
SECURITY
post deployment as well. For example, if a security
risk is identified (e.g., a publication in the common
vulnerabilities and exposures system), the end user
can use transparency to precisely pinpoint impacted
systems and take appropriate remediation steps.
Due to the diverse and complex nature of the ecosys-
tem, no one company or institution can single-hand-
edly solve this problem. The choice of technologies
and collaboration will have long-term implication on
evolution of capabilities to meet current and future
needs as well as the ability to scale and manage
costs. We strongly recommended that industry lead-
ers come together and take a comprehensive long-
term approach to address this important problem.
REFERENCES
1. M. Mattioli, “Consumer exposure to counterfeit
hardware,” IEEE Consum. Electron. Mag., to be
published, doi: 10.1109/MCE.2020.3023873.
2. Fortune, “Bumble bee foods aims to put all its fish on
a blockchain. It’s starting with ‘fair trade’ tuna,” Mar.
2010. Accessed: Feb. 17, 2021. [Online]. Available:
https://fortune.com/2019/03/08/tuna-blockchain-
bumble-bee-sap/
MICHAEL MATTIOLI currently leads the Hardware Engineer-
ing team within Goldman Sachs, New York, NY, USA. He is
responsible for the design and engineering of the firm’s digital
experiences and technologies. He is also responsible for the
overall strategy and execution of hardware innovation both
within the firm and within the broader technology industry.
Contact him at michael.mattioli@gs.com.
TOM GARRISON is currently the Vice President and General
Manager of Security Strategy and Initiatives within the Client
Computing Group, Intel Corporation, Mountain View, CA,
USA. He leads efforts to help customers and manufacturers
deploy tooling and processes for greater security assurance,
supply chain transparency, and cybersecurity innovation. He
also launches industry-wide initiatives and research with eco-
system partners and academia to accelerate cybersecurity
product assurance. Contact him at tom.garrison@intel.com.
BAIJU V. PATEL is currently an Intel Fellow within the Client
Computing Group, Intel Corporation, Mountain View, CA,
USA. He is responsible for setting the technical direction for
the Client Computing Group and Intel’s security technolo-
gies. Contact him at baiju.v.patel@intel.com.
IEEE Micro 107
PURPOSE: The IEEE Computer Society is the world’s largest
association of computing professionals and is the leading provider
of technical information in the field.
MEMBERSHIP: Members receive the monthly magazine
Computer, discounts, and opportunities to serve (all activities
are led by volunteer members). Membership is open to all IEEE
members, affiliate society members, and others interested in the
computer field.
COMPUTER SOCIETY WEBSITE: www.computer.org
OMBUDSMAN: Direct unresolved complaints to
ombudsman@computer.org.
CHAPTERS: Regular and student chapters worldwide provide the
opportunity to interact with colleagues, hear technical experts,
and serve the local professional community.
AVAILABLE INFORMATION: To check membership status, report
an address change, or obtain more information on any of the
following, email Customer Service at help@computer.org or call
+1 714 821 8380 (international) or our toll-free number,
+1 800 272 6657 (US):
• Membership applications
• Publications catalog
• Draft standards and order forms
• Technical committee list
• Technical committee application
• Chapter start-up procedures
• Student scholarship information
• Volunteer leaders/staff directory
• IEEE senior member grade application (requires 10 years
practice and significant performance in five of those 10)
PUBLICATIONS AND ACTIVITIES
Computer: The flagship publication of the IEEE Computer Society,
Computer publishes peer-reviewed technical content that covers
all aspects of computer science, computer engineering,
technology, and applications.
Periodicals: The society publishes 12 magazines and 17 journals.
Refer to membership application or request information as noted
above.
Conference Proceedings & Books: Conference Publishing
Services publishes more than 275 titles every year.
Standards Working Groups: More than 150 groups produce IEEE
standards used throughout the world.
Technical Committees: TCs provide professional interaction in
more than 30 technical areas and directly influence computer
engineering conferences and publications.
Conferences/Education: The society holds about 200 conferences
each year and sponsors many educational activities, including
computing science accreditation.
Certifications: The society offers three software developer
credentials. For more information, visit
www.computer.org/certification.
BOARD OF GOVERNORS MEETING
14, 15, 16, and 18 June 2021, virtual
revised 2 April 2021
EXECUTIVE COMMITTEE
President: Forrest Shull
President-Elect: William D. Gropp
Past President: Leila De Floriani
First VP: Riccardo Mariani; Second VP: Fabrizio Lombardi
Secretary: Ramalatha Marimuthu; Treasurer: David Lomet
VP, Membership & Geographic Activities: Andre Oboler
VP, Professional & Educational Activities: Hironori Washizaki
VP, Publications: M. Brian Blake
VP, Standards Activities: Riccardo Mariani
VP, Technical & Conference Activities: Grace Lewis
2021–2022 IEEE Division VIII Director: Christina M. Schober
2020-2021 IEEE Division V Director: Thomas M. Conte
2021 IEEE Division V Director-Elect: Cecilia Metra
BOARD OF GOVERNORS
Term Expiring 2021: M. Brian Blake, Fred Douglis,
Carlos E. Jimenez-Gomez, Ramalatha Marimuthu,
Erik Jan Marinissen, Kunio Uchiyama
Term Expiring 2022: Nils Aschenbruck,
Ernesto Cuadros‐Vargas, David S. Ebert, Grace Lewis,
Hironori Washizaki, Stefano Zanero
Term Expiring 2023: Jyotika Athavale, Terry Benzel,
Takako Hashimoto, Irene Pazos Viana, Annette Reilly,
Deborah Silver
EXECUTIVE STAFF
Executive Director: Melissa A. Russell
Director, Governance & Associate Executive Director:
Anne Marie Kelly
Director, Conference Operations: Silvia Ceballos
Director, Finance & Accounting: Sunny Hwang
Director, Information Technology & Services: Sumit Kacker
Director, Marketing & Sales: Michelle Tubb
Director, Membership & Education: Eric Berkowitz
COMPUTER SOCIETY OFFICES
Washington, D.C.: 2001 L St., Ste. 700, Washington, D.C.
20036-4928; Phone: +1 202 371 0101; Fax: +1 202 728 9614;
Email: help@computer.org
Los Alamitos: 10662 Los Vaqueros Cir., Los Alamitos, CA 90720;
Phone: +1 714 821 8380; Email: help@computer.org
MEMBERSHIP & PUBLICATION ORDERS
Phone: +1 800 678 4333; Fax: +1 714 821 4641;
Email: help@computer.org
IEEE BOARD OF DIRECTORS
President: Susan K. “Kathy” Land
President-Elect: K.J. Ray Liu
Past President: Toshio Fukuda
Secretary: Kathleen A. Kramer
Treasurer: Mary Ellen Randall
Director & President, IEEE-USA: Katherine J. Duncan Director
& President, Standards Association: James Matthews Director
& VP, Educational Activities: Stephen Phillips Director & VP,
Membership & Geographic Activities:
Maike Luiken
Director & VP, Publication Services & Products: Lawrence Hall
Director & VP, Technical Activities: Roger U. Fujii
 IEEE Computer Society
Volunteer Service Awards
Nominations accepted throughout the year.

T. Michael Elliott Distinguished Outstanding Contribution Certificate 

Service Certificate  Third highest level service certificate for a
Highest service award in recognition for specific achievement of major value to the
distinguished service to the IEEE Computer IEEE Computer Society, i.e., launching a major
Society at a level of dedication rarely conference series, a specific publication,
demonstrated. i.e., initiating a Society program standards and model curricula.
or conference, continuing officership, or long-
term and active service on Society committees.
Continuous Service Certificate
Recognize and encourage ongoing involvement
Meritorious Service Certificate  of volunteers in IEEE Computer Society
Second highest level service certificate for programs. The initial certificate may be awarded
meritorious service to an IEEE Computer after three years of continuous service.
Society-sponsored activity. i.e., significant as
an editorship, committee, Computer Society
Certificate of Appreciation
officer, or conference general or program chair. Areas of contribution would include service with
a conference organizing or program committee.
May be given to subcommittee members in lieu
of a letter of appreciation.

Nominations
Submit your nomination at
http://bit.ly/computersocietyawards
Contact us at
awards@computer.org
 DEPARTMENT: MICRO ECONOMICS
Remote Work
Shane Greenstein , Harvard Business School, Boston, MA, 02163, USA
W
e are living through a monumental change
in work. Experiments during the pandemic
turned into regular operations, and have
had a profound impact on perceptions of the viability of
remote work. I do not lightly use the word “profound.” The
trend reverses more than two centuries of separating the
location of work and residences. That separation has
deep roots, and reflects something fundamental about
the gains to society from separating work from other
activities. All developed countries have this feature.
The pandemic brought about an acceleration of
remote work. To appreciate how suddenly this arrived,
consider that two years ago less than 25% of the U.S.
workforce participated in some sort of remote work at
home, and it was typically less than half their work
hours. During the height of the pandemic, however,
approximately 40% of the U.S. labor force participated
in remote work, and it was full time at home.
Surveys of the U.S. experience indicate the shift to
remote work started with some initial variance and
hesitance in March, 2020, but by the end of April, this
hesitance had evaporated. The practice emerged
quickly, with anybody working remotely who could do
so. Once it started, few went back, as firms put in
place expensive investments to support remote work,
and established routines. A large amount of travel got
cancelled, and managers explored how much they
gained or lost from substituting to online meetings.
Those investments and drastic behavioral changes
would never have been done in normal times.
Now, the learning has been learned, and nobody
will forget the experience, nor tear out the software
that supports it, which raises the question: How much
of this will persist after the pandemic? Today’s column
reviews some of the latest research on this question.
BEYOND PUBLIC DISCUSSION
Two extremes examples tend to receive attention in
public conversation. On the one hand, some remote
work took place out of necessity, and, therefore,
0272-1732 ß 2021 IEEE
Digital Object Identifier 10.1109/MM.2021.3073433
Date of current version 25 May 2021.
110 IEEE Micro Published by the IEEE Computer Society
reluctantly. On the other hand, many employees and
their employers discovered unexpected benefits from
remote employment, and have considered continuing
the practice.
Start with reluctant remote work. Nobody expects
a reluctant employee to continue to work at home
once the pandemic declines. The reluctance has many
origins. For example, many daily work experiences can
best be performed face-to-face with coworkers and in
person, and no zoom call can replicate that. In addi-
tion, remote education does not work well for the vast
majority of K-12 students, and many parents unsuc-
cessfully had to navigate their work and child’s travails
with virtual school. All would happily return to the tra-
ditional arrangements.
The source of gains also varied, but two circum-
stances dominated. For one, less commute time led to
real gains in time devoted to work, and that arose
across a wide variety of jobs. Second, some firms
could get out of real-estate contracts. Less real-estate
expense leads to real savings in the budget.
To be sure, these gains arose unevenly, with sur-
veys showing massive differences across different
occupations and product markets. Skilled employees
at technology firms tended to be among the biggest
users of remote work, for example, while most services
tended not to be. Something subtle also emerged,
namely, a shifting of norms, especially with respect to
travel. Many business trips did not occur. Expenses
and time were saved, albeit, many sales also did not
occur that a more personal touch could have pushed
over the line.
Just in case there was any doubt, productivity
gains at the individual level did not make up for the
decline in the macro economy. Only in rare instances
did a firm find itself in a better place, and most of
those experiences arose in technology companies
whose online businesses faced explosive demand.
MAKE A FORECAST
To get clues on the likelihood and direction of the
anticipated shift to remote work after the pandemic,
forecasters needed to get beyond anecdote and char-
acterize broad patterns. Two professors from the Uni-
versity of Chicago, Jonathan Dingell and Brent
May/June 2021

Neiman, took a clever approach. They made an edu-
cated guess at the start of the pandemic about how
remote work would affect some occupations and not
others. Their speculation turned out to be close to
actual experience.1
They reasoned that it was feasible for many to
work from home, and social distancing might push the
reluctant to try it. In spring 2020, they estimated the
maximum percentage of the workforce that could
work from home, landing on an estimate of 37% of the
workforce.
TO SAY IT BLUNTLY, WITH THE
EXCEPTION OF “ESSENTIAL
WORKERS” SUCH AS DOCTORS AND
OTHER MEDICAL PERSONNEL, THE
BEST PAID WORKERS IN THE
ECONOMY WERE MOST ABLE TO
ADJUST TO THE PANDEMIC, WHILE
THE WORST PAID WORKERS IN THE
ECONOMY WERE REQUIRED TO FACE
THE HEALTH RISKS ASSOCIATED WITH
REPORTING TO WORK AT A
TRADITIONAL LOCATION.
This estimate arose out of a close reading of the
Occupational Information Network, a database main-
tained by the Bureau of Labor Statistics. It provides
detailed descriptions of over one thousand pre-pan-
demic occupations in the U.S. economy. Dingell and
Neiman classified occupations by whether an
employee performed physical activity, used heavy
machinery at work, required constant communica-
tions with others, and several other criteria that pre-
cluded remote work. Based on surveys of the
prevalence of those occupations, they estimated what
fraction, at most, could work from home.
This forecast yielded novel insights that differed
from pre-pandemic experiments with remote work.
Prior studies tended to stress the compatibility of
remote work with individualistic and asynchronistic
activity, such as that found in contract programming.
As it turned out, however, much more was feasible
across a much wider range of design, engineering, and
financial activities than anybody thought possible.
In addition, this approach yielded several addi-
tional observations. Dingell and Neiman calculated
that this workforce accounted for 46% of the wages in
the economy. To say it bluntly, with the exception of
“essential workers” such as doctors and other medical
May/June 2021
MICRO ECONOMICS
personnel, the best paid workers in the economy were
most able to adjust to the pandemic, while the worst
paid workers in the economy were required to face
the health risks associated with reporting to work at a
traditional location.
Most remarkably, this simple prediction turned
out to be a reasonable estimate for many other
countries. In other words, focusing on the feasibility
of moving occupations to remote work helps
explain why countries with high income and difficul-
ties managing the pandemic shared a similar eco-
nomic experience.
The broad trend hides a great deal of variance in
experience, to be sure. For example, numerous sur-
veys of the U.S. labor experience found that profes-
sional women with school-age children had a more
difficult time with remote work, regardless of the
occupation. The type of employer also shaped the
experience, with larger firms managing it better in gen-
eral, with the notable exceptions of retailing and
entertainment.2
What does this mean for the economy when it
does go back to normal? Stated simply, remote
work is not a possibility in the entire economy, but
it has become a strong probability in the some of
the highest paid occupations, and especially, where
it is possible to separate childcare from work at
home.
VARIANCE
Forecasters have started to think hard about the dis-
tinction between the transient and permanent
changes wrought by remote work. Again, it involves
considerable guess work, since nobody can fully antic-
ipate what part of remote work will persist in the long
run.
Some forecasts are easy to make. Some jobs, such
as event planners, are devoted to administrative
efforts to support conventions and face-to-face meet-
ings. Another fraction of jobs goes into supporting
commuters to downtown areas—e.g., attendants of
vehicles for mobile lunches, and security guards of
parking lots. The number of these jobs will decline.
More difficult to forecast, new jobs will emerge to
meet the new demands from remote work. For exam-
ple, it will become somebody’s jobs to find ways to
facilitate the rotation of desks and meeting rooms
among those showing up every few days, or among
those visiting a remote location where another branch
for the firm resides. Some of these staff will coordinate
new types of meetings among workers from the same
firm, who merely want to meet once a month, but do
IEEE Micro 111
MICRO ECONOMICS
so in a bigger room, or a setting that accommodates
larger teams. Some of these jobs are difficult to imag-
ine because many firms have never experienced this
activity at such scale.
STATED SIMPLY, REMOTE WORK IS
NOT A POSSIBILITY IN THE ENTIRE
ECONOMY, BUT IT HAS BECOME A
STRONG PROBABILITY IN THE SOME
OF THE HIGHEST PAID OCCUPATIONS,
AND ESPECIALLY, WHERE IT IS
POSSIBLE TO SEPARATE CHILDCARE
FROM WORK AT HOME.
The rise of remote work also will enable some
workers to become more of a contractor and less of
an employee, accelerating a trend that already has
gotten a foothold in technology labor markets. This
will generate a new type of job, namely, extending IT
services into the fraction of the remote workforce
that becomes closer to a contractor. Some of these
staff will take on the administrative tasks just like IT
staff within large organizations, but with a more dis-
persed clientele.
HR and many managerial tasks also will be rede-
fined by this shift. Especially difficult to predict is
how organizational cultures will change with a high
112 IEEE Micro
fraction of employment in remote work. Employees
must develop enough trust to transact with one
another by sharing the mission and a list of com-
mon experiences. Deep organizational cultures,
however, need to get beyond transactional trust. It
is essential to share a mission in order to share les-
sons and informal knowledge, and coordinate long-
term plans. In other words, if a large fraction of a
company’s employees alter where they perform
their job, then that also will lead to changes in the
daily tasks of many of that company’s other
employees.
REFERENCES
1. J. I. Dingell and B. Neiman, “How many jobs can be done
at home?,” J. Public Econ., vol. 189, Sep. 2020,
Art. no. 104235. [Online]. Available: https://www.
sciencedirect.com/science/article/abs/pii/
S0047272720300992
2. N. Bloom, R. S. Fletcher, and E. Yeh, “The impact of
COVID-19 on US firms,” National Bureau of Economic
Research, Cambridge, MA, USA, Working Paper 28314.
[Online]. Available: https://www.nber.org/papers/
w28314
SHANE GREENSTEIN is a Professor at Harvard Business
School. Contact him at sgreenstein@hbs.edu.
May/June 2021
Get Published in the IEEE Open
Journal of the
$PNQVUFS
4PDJFUZ

Submit a paper today to the

premier open access journal
in DPNQVUJOH
BOE
JOGPSNBUJPO

UFDIOPMPHZ.

Your research will benefit from

the IEEE marketing launch and
5 million unique monthly users
of the IEEE Xplore® Digital Library.
Plus, this journal is fully open
and compliant with funder
mandates, including Plan S.

Submit your paper today!

Visit XXXDPNQVUFSPSHPK to learn more.
IEEE
Computer
Society Has
You Covered!
WORLD-CLASS CONFERENCES — Stay
ahead of the curve by attending one of our
215+ globally recognized conferences.

DIGITAL LIBRARY — Easily access over 800k

articles covering world-class peer-reviewed
content in the IEEE Computer Society
Digital Library.

CALLS FOR PAPERS — Discover

opportunities to write and present your
ground-breaking accomplishments.

EDUCATION — Strengthen your resume

with the IEEE Computer Society Course
Catalog and its range of offerings.

ADVANCE YOUR CAREER — Search the

new positions posted in the IEEE Computer
Society Jobs Board.

NETWORK — Make connections that count

by participating in local Region, Section,
and Chapter activities.

Explore all of the member benefits

at www.computer.org today!