VOLUME 41, NUMBER 2 MARCH/APRIL 2021

Hot Chips

www.computer.org/micro
 mpute
r.org ²ƧǞƵȁ
ɈǞ
Comp ˛Ƨ
uting
nted Data
and Visual
eality Inform izatio
n
on
ce4c1.
indd Techn ation
1
ology
on Quan
tum
y Comp
uting
putin
g

Intern
et of
Ethic Thing
s s
Mach
ine L
Quan ea rning
tu
Comp m
uting
JUNE
2020

JULY 20
ww w.c 20
ompu
ter.org

ww w.c
ompu
ter.org

ce7c1.ind
d 1

Secu
rity an
Priva d
cy High-P
Auto
matio
n Comp erforman
Block uting ce
chain Hard
Digit ware
al Affect
Transf iv
ormat Comp e
ion uting
Educa
tion

MAY 20
20

w w w.c
ompu
ter.org

ce5c1.
indd
1

S
ww w.c NOVE
ompu MBER
ter.org 2020

ww w.c
ompu
ter.org

ce11c1.in
dd 1

ndd 1

ComputingEdge
Secu
riit
Priva y and
cy
Data
Intern
et
ȲɈǞ˛
ƧǞƊǶ
XȁɈƵǶǶ
ǞǐƵȁƧ
Ƶ
Your one-stop resource for industry hot topics,
technical overviews, and in-depth articles.

Cutting-edge Unique original Keeps you up to

articles from the FRQWHQWbE\ date on what you
IEEE Computer computing need to know across
Society’s portfolio thought leaders, the technology
of 12 magazines. innovators, spectrum.
DQGbH[SHUWV

Subscribe for free

www.computer.org/computingedge
EDITOR-IN-CHIEF IEEE MICRO STAFF
Lizy K. John, University of Texas, Austin, USA Journals Production Manager: Joanna Gojlik,
j.gojlik@ieee.org
ASSOCIATE EDITOR-IN-CHIEF
Peer-Review Administrator:
Vijaykrishnan Narayanan, The Pennsylvania State micro-ma@computer.org
University, USA Publications Portfolio Manager: Kimberly Sperka
EDITORIAL BOARD Publisher: Robin Baldwin
Senior Advertising Coordinator: Debbie Sims
Guido Araujo, University of Campinas, Brazil IEEE Computer Society Executive Director: Melissa
R. Iris Bahar, Brown University, USA Russell
Christopher Batten, Cornell University, USA
Mauricio Breternitz, University of Lisbon, Portugal IEEE PUBLISHING OPERATIONS
Bronis de Supinski, Lawrence Livermore
Senior Director, Publishing Operations: Dawn Melley
National Laboratory, USA
Director, Editorial Services: Kevin Lisankie
Yasuko Eckert, AMD Research, USA
Director, Production Services: Peter M. Tuohy
Maya Gokhale, Lawrence Livermore National
Associate Director, Editorial Services: Jeffrey E. Cichocki
Laboratory, USA
Associate Director, Information Conversion
Shane Greenstein, Harvard Business School, USA
and Editorial Support: Neelam Khinvasara
Hyesoon Kim, Georgia Institute of Technology, USA
Senior Art Director: Janet Dudar
John Kim, Korea Advanced Institute of Science and
Senior Manager, Journals Production: Patrick Kempf
Technology, South Korea
Hsien-Hsin (Sean) Lee, Taiwan Semiconductor CS MAGAZINE OPERATIONS COMMITTEE
Manufacturing Company, USA
Richard Mateosian, USA Diomidis Spinellis (Chair), Lorena Barba,
Michael Mattioli, Goldman Sachs & Co., USA Irena Bojanova, Shu-Ching Chen, Gerardo Con Diaz,
Tulika Mitra, National University of Singapore, Lizy K. John, Marc Langheinrich, Torsten Möller,
Singapore Ipek Ozkaya, George Pallis, Sean Peisert,
Trevor Mudge, University of Michigan, Ann Arbor, USA VS Subrahmanian, Jeff Voas
Richard H. Stern, George Washington University CS PUBLICATIONS BOARD
Law School, USA
Sreenivas Subramoney, Intel Corporation, India M. Brian Blake (VP for Publications),
Carole-Jean Wu, Arizona State University, USA Hui Lei, Antonio Rubio, Diomidis Spinellis,
Lixin Zhang, Chinese Academy of Sciences, China Ex officio: Robin Baldwin, Melissa Russell, Forrest
Shull
ADVISORY BOARD
David H. Albonesi, Cornell University, USA COMPUTER SOCIETY OFFICE
Erik R. Altman, IBM, USA IEEE MICRO
Pradip Bose, IBM T.J. Watson Research Center, USA c/o IEEE Computer Society
Kemal Ebcioglu, Global Supercomputing 10662 Los Vaqueros Circle
Corporation, USA Los Alamitos, CA 90720 USA; +1 (714) 821-8380
Lieven Eeckhout, ELIS – Ghent University, Belgium
Michael Flynn, Stanford University, USA
Ruby B. Lee, Princeton University, USA Subscription change of address:
Yale Patt, University of Texas at Austin, USA address.change@ieee.org
James E. Smith, University of Wisconsin–Madison, USA Missing or damaged copies:
Marc Tremblay, Sun Microsystems, USA help@computer.org

IEEE Micro (ISSN 0272-1732) is published bimonthly by the IEEE Computer Society. IEEE Headquarters: Three Park Ave., 17th Floor, New York,
NY 10016; IEEE Computer Society Headquarters: 2001 L St., Ste. 700, Washington, DC 20036; IEEE Computer Society Publications Office: 10662
Los Vaqueros Circle, PO Box 3014, Los Alamitos, CA 90720. Postmaster: Send address changes and undelivered copies to IEEE, Membership
Processing Dept., 445 Hoes Ln., Piscataway, NJ 08855. Periodicals postage is paid at New York, NY, and at additional mailing offices. Canadian
GST #125634188. Canada Post Corp. (Canadian distribution) Publications Mail Agreement #40013885. Return undeliverable Canadian addresses to
4960-2 Walker Road; Windsor, ON N9A 6J3. Printed in USA. Reuse rights and reprint permissions: Educational or personal use of this material is
permitted without fee, provided such use: 1) is not made for profit; 2) includes this notice and a full citation to the original work on the first page of the
copy; and 3) does not imply IEEE endorsement of any third-party products or services. Author and their companies are permitted to post the accepted
version of IEEE-copyrighted material on their own webservers without permission, provided that the IEEE copyright notice and a full citation to the
original work appear on the first screen of the posted copy. An accepted manuscript is a version which has been revised by the author to incorporate
review suggestions, but not the published version with copy-editing, proofreading, and formatting added by IEEE. For more information, please go to
ieee.org/publications_standards/publications/rights/paperversionpolicy.html. Permission to reprint/republish this material for commercial, advertising,
or promotional purposes or for creating new collective works for resale or redistribution must be obtained from IEEE by writing to the IEEE Intellectual
Property Rights Office, 445 Hoes Lane, Piscataway, NJ 08854 or pubs-permissions@ieee.org. ©2021 by IEEE. All rights reserved. Abstracting and
library use: Abstracting is permitted with credit to the source. Libraries are permitted to photocopy for private use of patrons, provided the per-copy
fee indicated in the code at the bottom of the first page is paid through the Copyright Clearance Center, 222 Rosewood Drive, Danvers, MA 01923.
Editorial: Unless otherwise stated, bylined articles, as well as product and service descriptions, reflect the author’s or firm’s opinion. Inclusion in IEEE
Micro does not necessarily constitute an endorsement by IEEE or the Computer Society. All submissions are subject to editing for style, clarity, and
space. IEEE prohibits discrimination, harassment, and bullying. For more information, visit ieee.org/web/aboutus/whatis/policies/p9-26.html.
 6
GUEST EDITORS’ INTRODUCTION
Best Papers From Hot Chips 32
Priyanka Raina and Cliff Young

MARCH/APRIL 2021

7 15 22
Theme Articles

IBM’s POWER10 Processor Marvell ThunderX3: The Xbox Series

William J. Starke, Next-Generation X System
Brian W. Thompto, Jeff A. Stuecheli, Arm-Based Server Architecture
and José E. Moreira
Processor Mark Grossman and Jeff rey
Andrews
Rabin Sugumar, Mehul Shah, and
Ricardo Ramirez
 Theme Articles Continued
29 NVIDIA A100 Tensor Core GPU: Performance and Innovation
Jack Choquette, Wishwesh Gandhi, Olivier Giroux, Nick Stam, and Ronny Krashinsky

36 Manticore: A 4096-Core RISC-V Chiplet Architecture for  

 Ultraefficient Floating-Point Computing
Florian Zaruba, Fabian Schuiki, and Luca Benini

43 Pensando Distributed Services Architecture

Michael Galles and Francis Matus

50 Compute Substrate for Software 2.0

Jasmina Vasiljevic, Ljubisa Bajic, Davor Capalija, Stanislav Sokorac, Dragoljub
Ignjatovic, Lejla Bajic, Milos Trajkovic, Ivan Hamer, Ivan Matosevic, Aleksandar Cejkov,
Utku Aydonat, Tony Zhou, Syed Zohaib Gilani, Armond Paiva, Joseph Chu, Djordje
Maksimovic, Stephen Alexander Chin, Zahi Moudallal, Akhmed Rakhmati, Sean
Nij ar, Almeet Bhullar, Boris Drazic, Charles Lee, James Sun, Kei-Ming Kwong, James
Connolly, Miles Dooley, Hassan Farooq, Joy Yu Ting Chen, Matthew Walker, Keivan
Dabiri, Kyle Mabee, Rakesh Shaji Lal, Namal Rajatheva, Renjith Retnamma, Shripad
Karodi, Daniel Rosen, Emilio Munoz, Andrew Lewycky, Aleksandar Knezevic, Raymond
Kim, Allan Rui, Alexander Drouillard, and David Thompson

General Interest
56 The Design Process for Google’s Training Chips: TPUv2 and TPUv3
  Thomas Norrie, Nishant Patil, Doe Hyun Yoon, George Kurian, Sheng Li, James Laudon,  
  Cliff Young, Norman Jouppi, and David Patterson

64   Klessydra-T: Designing Vector Coprocessors for Multithreaded

Edge-Computing Cores
Abdallah Cheikh, Stefano Sordillo, Antonio Mastrandrea, Francesco Menichelli,
Giuseppe Scotti, and Mauro Olivieri

Columns and Departments

From the Editor-in-Chief
4 CPUs, GPUs, and More From Hot Chips‘32
Lizy Kurian John
Security
  72 Hidden Potential Within Video Game Consoles
Michael Mattioli and Atte Lahtiranta
Awards
  78 A Brief History of Warehouse-Scale Computing: Reflections Upon Receiving the
2020 Eckert-Mauchly Award
Luiz André Barroso
Micro Economics
 86   The Economics of Confrontational Conversation
Shane Greenstein

Also in this Issue

1  Masthead
  84 IEEE Computer Society Info

www.computer.org/micro

ISSN: 0272-1732
 FROM THE EDITOR-IN-CHIEF
CPUs, GPUs, and More From Hot
Chips‘32
Lizy Kurian John , The University of Texas, Austin, TX, 78712, USA
W
elcome to the March/April 2021 issue of
IEEE Micro. This Issue features selected
articles from the Hot Chips ’32 Symposium,
held virtually in August 2020. COVID-19 forced Hot
Chips’32 to be a virtual event, however the chips are
more interesting and powerful than ever! Whether it is
graphics acceleration or sheer increase in traditional
compute capability, the chip design arena has become
hotter than ever. A lot of money is pouring into designing
both special purpose and general purpose chips. IEEE
Micro is pleased to present seven selected articles
based on the presentations at the Hot Chips Sympo-
sium for our readers. Priyanka Raina of Stanford Univer-
sity and Cliff Young of Google served as guest editors
for this special issue. They have compiled an excellent
selection of articles on emerging chips and systems
from the Symposium, including articles on IBM Power10,
Marvell Thunder X3, Xbox Series X, NVIDIA A100, Manti-
core 4096 core chiplet, Pensando Distributed Service
Architecture, and TensTorrent Compute substrate for
Software 2.0. Please read the Guest Editors’ Introduc-
tion to get a preview of the seven articles. Thanks to the
editors, authors, and reviewers who worked hard to put
this issue together.
In addition to the aforementioned seven Hot Chips
articles, there are two General Interest articles. The
first one is on the design process for Google’s TPUv2
and TPUv3 training chips, written by Norrie et al. from
Google. This was also originally presented at Hot
Chips, however because the authors included one of
the guest editors, this was considered as a General
Interest article, and the review process was separately
coordinated. The article describes Google’s approach
to machine learning (ML) hardware, and provides
details on the scalar computation unit, the vector
computation unit, the matrix computation units, the
memory system, the interconnect, and the floor plan
of TPU v2. The enhancements in TPU v3 compared to
0272-1732 ß 2021 IEEE
Digital Object Identifier 10.1109/MM.2021.3064094
Date of current version 26 March 2021.
4 IEEE Micro Published by the IEEE Computer Society
TPU v2 are discussed next and the performance is
compared using roofline plots.
The second General Interest article is “Klessydra-T:
Designing Vector Coprocessors for Multithreaded
Edge-Computing Cores” by Cheikh et al. of Sapienza
University of Rome. This work addresses the introduc-
tion of coprocessor acceleration in interleaved multi-
threading (IMT) cores for extreme edge computing.
Specifically, it explores possible alternatives to imple-
ment vector coprocessing units in RISC-V cores,
showing the synergy between IMT and data-level
parallelism in edge-computing applications.
WHETHER IT IS GRAPHICS
ACCELERATION OR SHEER INCREASE
IN TRADITIONAL COMPUTE
CAPABILITY, THE CHIP DESIGN ARENA
HAS BECOME HOTTER THAN EVER.
A LOT OF MONEY IS POURING INTO
DESIGNING BOTH SPECIAL PURPOSE
AND GENERAL PURPOSE CHIPS.
This issue also features three department articles.
Michael Mattioli of Goldman Sachs who joined the
IEEE Micro Editorial Board as a new Department Editor
has coauthored an article with Atte Lahtiranta on the
hidden capabilities of video game consoles. The
authors describe nonvideo game capabilities of video
game consoles such as web browsing, video confer-
encing, and audio/video/document content creation.
They posit that video game consoles are ideal for
enterprise deployment, with security features and
defense against a wide range of threats. The Microsoft
Xbox Series X architecture is described with emphasis
on the security processor. The security processor is
housed in the security complex with crypto engine,
random number generator, secure RAM, secure ROM,
security fuse bank, and side-channel monitors. Please
read the article to appreciate the less-discussed fea-
tures of video game consoles.
March/April 2021

This issue additionally includes an Award article
from Luiz Andre Barroso of Google who received the
2020 Eckert-Mauchly Award. In his article, the author
provides a brief history of warehouse-scale computing.
He describes the progression of datacenter computing
as it evolved during the last two decades, and also
describes his personal journey as a computer engineer.
He concludes the article acknowledging how the pan-
demic has made many to realize the importance of com-
puting technology and cloud-based services, and how
these have allowed us to continue to work and live.
Another article presented in this issue is a Micro
Economics column by Shane Greenstein, “The Econom-
ics of Confrontational Conversation,” discussing how
confrontational conversations are commonplace on the
internet. Greenstein focuses on the economics relevant
to such confrontational conversations. One economic
factor is that it is inexpensive to host terabytes of data.
Additionally simple focal platforms attract more users,
which then attracts more apps and content. The third
relevant economic fact is that the mechanisms to
address confrontation, whether human moderation or
algorithmic processes, will be expensive.
MANY THEMATIC SPECIAL ISSUES
ARE PLANNED FOR THE REMAINDER
OF 2021. THEMES INCLUDE QUANTUM
COMPUTING, FPGA COMPUTING,
IN-MEMORY COMPUTING, AND SMART
AGRICULTURE.
There have been some additions/enhancements
to the IEEE Micro Editorial Board this month. Dr.
Vijaykrishnan Narayanan of Penn State has been pro-
moted to the Associate Editor-in-Chief of IEEE Micro.
Michael Mattioli of Goldman Sachs will serve as a
Department Editor for Security and Product Reviews.
Prof. Guido Araujo of the University of Campinas
(Brazil) joins the Editorial Board this month as an
Associate Editor. I look forward to working with all of
them and bringing to you an even more interesting
reading experience.
March/April 2021
FROM THE EDITOR-IN-CHIEF
Let me also provide an overview of what to expect
in upcoming issues. The May/June issue will be the
popular “Top Picks” Special Issue which presents the
best of the best from articles in computer architecture
conferences in 2020. Prof. Daniel Jimenez of Texas
A&M University and a selection committee from
industry and academia have selected 12 papers from
more than 100 articles that were submitted in
response to the Top Picks call for papers. Readers can
look forward to an amazing collection of excellent
articles in May/June.
Many thematic special issues are planned for the
remainder of 2021. Themes include quantum comput-
ing, FPGA computing, in-memory computing, and
smart agriculture. The July/August issue will be a Spe-
cial Issue on Quantum Computing, guest edited by
Ulya Karpuzcu of the University of Minnesota. The
FPGA Computing Special Issue will be guest edited by
Maya Gokhale of Lawrence Livermore Laboratories
and Lesley Shannon of Simon Fraser University, Can-
ada. The In-Memory Computing Special Issue will be
guest edited by Reetuparna Das of the University of
Michigan. We also have an open call for smart agricul-
ture, focusing on the use of artificial intelligence and
IoTs in agriculture, guest edited by Neeraj Kumar of
Thapar University and Sudip Misra of IIT Kharagpur.
We invite readers to submit to these Special
Issues. Please find the open calls at:
› https://www.computer.org/digital-library/
magazines/mi/call-for-papers-special-issue-
on-processing-in-memory.
› https://www.computer.org/digital-library/
magazines/mi/call-for-papers-special-issue-on-
ai-edge-and-iot-for-smart-agriculture.
IEEE Micro is interested in submissions on any
aspect of chip/system design or architecture.
Hope you enjoy the articles presented in this issue.
Happy reading!
LIZY KURIAN JOHN is a Cullen Trust for Higher Education
Endowed Professor with the Electrical and Computer Engi-
neering Department, University of Texas at Austin, Austin,
TX, USA. Contact her at ljohn@ece.utexas.edu.
IEEE Micro 5
 GUEST EDITORS' INTRODUCTION
Best Papers From Hot Chips 32
Priyanka Raina , Stanford University, Stanford, CA, 94305, USA
Cliff Young , Google, Palo Alto, CA, 94306, USA
W
elcome to our special issue of IEEE Micro,
which highlights the best presentations
from Hot Chips 32, held virtually on
August 16–18, 2020. Like many things in 2020, Hot
Chips was unprecedented, going virtual for the first
time in its history. Presentations were done by video,
with a small production team working in a studio and
a virtual conference supplemented by Zoom chat-
rooms. Despite the switch to the virtual format and
the challenges to the global economy, attendance
was the highest ever, and the technical program was
robust, with strong representation from traditional
CPU, GPU, and FPGA manufacturers, and strong offer-
ings from startups in both communication networks
and neural networks. This issue collects articles
derived from the best talks, chosen after the confer-
ence by the Program Committee. It is a great time to
work in computer architecture, where the combina-
tion of approaching limits to Moore’s Law and new
transformative application areas mean that both
incumbent computer architectures and new startups
have large contributions to make.
Two of the articles focus on server processors, one
from a long-time mainframe manufacturer and one from
a potential disruptor of the server business. “IBM’s
Power10 Processor” describes the newest instance of
the POWER architecture from the seminal computing
company. Power10 was designed for general-purpose
enterprise computing with interconnection across
16 chips in a multiprocessor, with 1-TB/s memory band-
width per CPU and high-bandwidth links to accelerators
including GPUs. By contrast, “Marvell ThunderX3: Next-
Generation Arm-Based Server Processor” represents
the new wave of ARM-based server-class chips that aim
to change the performance and price/performance of
the datacenter server market.
Two articles describe chips with significant
graphics capability. “The Xbox Series X System Archi-
tecture” details the system-on-a-chip that powers
0272-1732 ß 2021 IEEE
Digital Object Identifier 10.1109/MM.2021.3060294
Date of current version 26 March 2021.
6 IEEE Micro Published by the IEEE Computer Society
Microsoft’s latest gaming console, dedicating over
two-thirds of its die to the GPU that delivers 4K at 120
frames per second. While graphics remain central to
the mission of the article titled “NVIDIA A100 Tensor
Core GPU: Performance and Innovation,” GPUs have
become the default for high performance and program-
mability in one solution, supporting a huge variety of
scientific computing workloads and powering both
neural network training and inference.
Bridging between the general-purpose floating-
point power of GPUs and the specialized application
focus of neural network accelerators, “Manticore: A
4096-Core RISC-V Chiplet Architecture for Ultraeffi-
cient Floating-Point Computing” uses the extensibility
of the RISC-V ISA to reduce control overheads and
energy costs for neural-network workloads.
Our final two articles come from startups, both
with networking in their roots. “Pensando Distributed
Services Architecture” describes their domain-specific
architecture (including chips and programmable soft-
ware) for building new applications and services within
a datacenter network. “Compute Substrate for Soft-
ware 2.0” explains startup TensTorrent’s unique archi-
tecture for neural network acceleration, which takes a
packet-network-inspired approach to control and flex-
ibility, allowing better support for sparsity, varying
numerical precision, and compression than older,
more monolithic neural network accelerators.
All of the talks from Hot Chips 32 (https://www.
hotchips.org/archives/hc32/) are available at the Hot
Chips website. Hot Chips are run by a great set of
volunteers, including a sophisticated logistical and mar-
keting team in the Organizing Committee and the won-
derful set of academic and professional computer
architects on our Program Committee. No other confer-
ence has the same focus on production computer sys-
tems, presented by their designers, sharing how and why
they built their chips. We hope you find this issue, and
future Hot Chips, as informative and fun as we have.
March/April 2021
THEME ARTICLE: HOT CHIPS

IBM’s POWER10 Processor

William J. Starke , Brian W. Thompto  E. Moreira
, Jeff A. Stuecheli, Jose , International Business Machines
Corporation, Armonk, NY, 10504, USA

The IBM POWER10 processor represents the 10th generation of the POWER family of
enterprise computing engines. It is built on a balance of computation and
bandwidth, delivered by powerful processing cores and intrachip interconnect,
respectively. Multiple system interconnect infrastructures support configurations
with up to 16 processor chips and up to 1920 simultaneous threads of execution, as
well as an expansive memory system with up to 2 Petabytes of addressing space.
Cross-system memory sharing and coherent accelerator attach are also supported.
The POWER10 processing core has been significantly enhanced over its POWER9
predecessor, including the addition of an all-new matrix math engine. Throughput
gains from POWER9 to POWER10 average 30% at the core level and three-fold at the
socket level. Those gains can reach ten- or twenty-fold at the socket level for
matrix-intensive computations.

T
he IBM POWER10 processor delivers significant
gains in capacity and capability over its immedi-
ate POWER9 predecessor1,2: an average 20%
single-thread performance boost, and 30% gain in
core throughput over a wide range of applications.
Combined with a two-and-a-half increase in the num-
ber of cores per package, these improvements result
in three times or better per socket throughput on pop-
ular integer, floating-point, and commercial workloads,
and 2–4 times increased memory bandwidth, depend-
ing on memory technology. For matrix math, the gains
in performance can reach 10 or 20 times through a
new computational engine.
Additional breakthroughs include: a new Power-
AXON system interconnect with 1 TB/s of bandwidth
per POWER10 chip and support for cross-system mem-
ory clustering; a new Open Memory Interface (OMI) that
supports multiple industry-standard memory technolo-
gies on the same processor chip; a modular building
block die that enables systems with up to 1920 simulta-
neous threads of execution; hardware-enforced security
to protect sensitive code and data from attacks; and AI-
optimized machine instructions to address the
increased computing demands of modern machine
learning/deep learning business applications.
0272-1732 ß 2021 IEEE
Digital Object Identifier 10.1109/MM.2021.3058632
Date of publication 10 February 2021; date of current version
26 March 2021.
POWER10 PROCESSOR CHIP
A POWER10 processor die (see Figure 1) consists of 18
billion transistors in 602 mm2 of silicon, compared to 8
billion transistors in POWER9, and is built in Samsung’s
7-nm technology with 18 metal layers. The central part
of the die, approximately 300 mm2, is occupied by 16
enterprise-grade cores, each capable of running eight
simultaneous threads of execution (SMT8), and their
associated 2- and 8-MB levels 2 and 3 cache regions,
respectively. To better match the supply and demand
of processor chips with the maximum number of cores,
we cap the number of active cores in a die to 15, keep-
ing one core as a manufacturing spare. This results in
up to 120 simultaneous threads of execution, backed
by 120 MB of level-3 cache.
The remaining half of the POWER10 processor chip
area is dedicated to the system interconnect, includ-
ing the two protocol spines to the left and right of the
core/cache complex, supporting the various intercon-
nect protocols for memory, multiple processors, accel-
erators, clusters, and I/O. The periphery of the die is
filled with high bandwidth, power efficient signaling
circuits that implement the PowerAXON,3 OMI,3 and
PCI Gen5 I/O infrastructures.
Not visible in Figure 1 are the large numbers of
communication trunk lines, which run horizontally
over the two L3 hemispheres and vertically over the
protocol spines. The placement of the L3 hemi-
spheres, the protocol spines, the trunk lines, the inter-
cept of these vertical and horizontal trunk lines, and
the location of the protocol spines next to the

March/April 2021 Published by the IEEE Computer Society IEEE Micro 7

HOT CHIPS

FIGURE 1. POWER10 processor chip. Approximately half the die area is dedicated to cores and caches. The other half is for the
various system interconnects, including memory interfaces, SMP, accelerators, clustering, and I/O.

signaling infrastructure are the result of rearchitecting
the chip floorplan around a computation-to-band-
width balance.
POWER10 processor chips can be packaged in either
single- or dual-chip modules (SCM/DCM). The SCM con-
figuration is optimized for scale-up systems and maxi-
mizes power, interconnect bandwidth, and memory
capacity delivered to each core. It also supports more
flexible topologies, allowing configurations with up to 16
processor chips. The DCM configuration is optimized for
scale-out systems and maximizes computational and I/
O density while trading off the power and memory per
core compared with the SCM. It limits configurations to
a maximum of four DCMs (eight processor chips).
The POWER10 chip introduces new security fea-
tures for cloud paradigms that extend trusted virtuali-
zation environments to include protected containers
and include in-line memory encryption and application
level protections against attacks.
POWER10 SYSTEMS
POWER10 systems are built around the three intercon-
nect infrastructures shown in Figure 2: the OMI, for
connecting processors to memory; the PowerAXON
interface, for interconnecting processor chips to other
processors and accelerators and for implementing
cross-system memory clustering; and PCI Gen 5 for I/
O and other system interconnect.
PowerAXON and OMI signaling runs at rates of up
to 32 GT/s. With a combined 256 bidirectional lanes,
this results in up to 2 TB/s of total bandwidth on a pro-
cessor chip, with 128 lanes and 1 TB/s for each inter-
face. These are shown to the left and right of the
POWER10 chip in Figure 2, respectively. (See Figure 1
for physical placement of the interfaces. Each Power-
AXON corner has 32 lanes plus 4 spares.)
THE POWER10 CHIP INTRODUCES
NEW SECURITY FEATURES FOR
CLOUD PARADIGMS THAT EXTEND
TRUSTED VIRTUALIZATION
ENVIRONMENTS TO INCLUDE
PROTECTED CONTAINERS AND
INCLUDE IN-LINE MEMORY
ENCRYPTION AND APPLICATION
LEVEL PROTECTIONS AGAINST
ATTACKS.

8 IEEE Micro March/April 2021


FIGURE 2. POWER10 system interconnect. OMI is used for attaching memory to the processor. PowerAXON provides the SMP,
clustering, and accelerator interfaces. PCIe Gen5 is used for I/O and other interconnect.
OMI is a technology agnostic memory interface
based on open standards. Memory is attached to the
processor chip through an OMI-compliant buffer chip,
which encapsulates technology specific requirements
as first introduced in IBM’s POWER8 processor and its
companion Centaur memory buffer chip.4 POWER10
systems will initially use DDR4 memory, through a
buffer chip built by Microchip.3 This buffer chip imple-
ments 25.6-GT/s signaling over an 8-bit interface,
which matches its 8-byte DDR4 channel operating at
3.2 GT/s. With 16 channels in use, up to 410 GB/s of
peak DDR4 bandwidth can be achieved per POWER10
chip, with a latency that is only 10 ns over traditional
DDR4 DIMMs. DDR5 memory DIMMs can be sup-
ported later through a future buffer chip.
Alternative memory technologies can also be
deployed with POWER10 processors using OMI. As
shown in the right half of Figure 2, both high-band-
width GDDR memory and high-capacity nonvolatile
storage-class memory can be connected to the same
OMI channels through corresponding buffer chips and
using standard OMI DIMM form factors. A fully popu-
lated 16-channel GDDR configuration would achieve
over 800 GB/s of memory bandwidth to a single
POWER10 processor. This approaches the bandwidth
achieved with high-bandwidth memory (HBM), but at
higher capacities and lower cost. Alternatively, a stor-
age-class (nonvolatile) memory solution could achieve
capacities in the Terabytes per DIMM range.
The PowerAXON infrastructure is used for system
scaling, including multiprocessor interconnect, device
March/April 2021
HOT CHIPS
attach, and memory clustering. The largest single-sys-
tem configuration consists of 16 SCMs, intercon-
nected as shown in the lower left corner of Figure 2.
Each module is at most two hops away from any other
module. A system with this configuration can run up
to 1920 simultaneous threads of execution and con-
tain up to 256 OMI DIMMs (16 DIMMs attached to
each of 16 SCMs), with a maximum capacity of 1 Peta-
byte of memory. (POWER10 processors have a 2-Peta-
byte physical memory address space.)
PowerAXON can also be used to support Open-
CAPI, an open, asymmetric protocol for coherently
attaching compute accelerators, memory devices,
network interfaces, and storage controllers, either in a
device slot or a cabled external enclosure. Since its
introduction with POWER9, a variety of vendors have
provided OpenCAPI-attached devices that expand
and enhance the functionality of POWER systems.
POWER10 OpenCAPI provides a new level of perfor-
mance and functionality over the prior version.
The third and final functionality of PowerAXON in
POWER10 that we discuss in this article is memory clus-
tering, shown in the top left corner of Figure 2. This new
feature of POWER10, which is called memory inception,
delivers the long-sought functionality of server disaggre-
gation. Memory inception enables systems to directly
share each other’s main memory. The latency through
memory inception is 50–100 ns over that of a remote (2-
hop) socket within a server, and it is still low enough to
be used as main memory. The protocol for memory
inception is built on top of the OpenCAPI protocol and
IEEE Micro 9
HOT CHIPS
different from the SMP protocol used to build (up to) 16-
socket systems. Memory inception does not implement
a cache coherence scheme and is not meant to enable
larger single system image configurations. Rather, the
goal is to allow one server to map its address space to
the physical memory of another server.
As a scenario for using memory inception, con-
sider the case of a cluster of homogeneous servers,
each with enough memory for the average workload.
By borrowing memory from other machines, a host-
ing system can run large memory workloads that go
beyond the capacity of any single server. Another
scenario is a hub-and-spoke configuration, in which
a very large central server has a big pool of memory,
distributed as needed across a large set of much
smaller machines. This combines the cost efficiency
of small machines with the memory capacity of a
much larger server.
Memory inception can also be used as the mes-
sage layer for a large cluster of POWER10 servers.
Combined with the processor’s 2-Petabyte address
space, memory inception can use the address trans-
lation facilities in each server to create a multihop
interconnect, with messages delivered simply by
writing to the target memory. A robust, fully hard-
ware managed end-to-end message capability is
possible in clusters with thousands of nodes, deliv-
ering high bandwidth, low latency, and flexible
topologies.
The final component of the POWER10 system inter-
connect, shown in the bottom of Figure 2, are the PCIe
Gen 5 interfaces. PCIe is central to the I/O infrastruc-
ture of POWER10 systems, and up to 64 lanes are
available in a DCM (32 lanes per chip). With a signaling
rate of 32 GT/s, a single DCM can achieve 252 GB/s of
I/O bandwidth in each direction.
POWER10 CORE
The POWER10 core is the processing engine that runs
both system and user software, responsible for the
computational capacity and capability of POWER10
systems. There are two focus areas in the design of
the POWER10 core: performance strength and power
efficiency. A 30% average increase in core throughput
while cutting power consumption in half combine to
deliver a 2.6-fold average increase in energy efficiency
for computations. The increased energy efficiency has
allowed the implementation of DCMs with up to 30
SMT8 cores, and up to a three-fold throughput over
current POWER9 modules with similar power con-
sumption. The POWER10 core retains the modular
architecture from POWER9 that provides a second
10 IEEE Micro
variant of the chip with twice as many SMT4 cores per
chip (up to 60 per DCM).
The microarchitecture of the POWER10 core,
together with key factors affecting its performance
and power efficiency, are shown in Figure 3. The block
diagram shows those microarchitecture resources
available for the execution of 1 to 4 simultaneous
threads, corresponding to half of the total resources
in an SMT8 core. POWER10 core components colored
in green were somewhat improved in capacity over
the predecessor POWER9 core. Those colored in blue
had their capacity at least doubled and those in red
had their capacity at least quadrupled. These addi-
tional resources along with various other improve-
ments in latency and microarchitecture are
responsible for the 30% average increase in core
throughput and a much higher boost in performance
in some cases.
Each POWER10 SMT8 core has an associated 2
MiB L2 cache that provides both instructions and data
and is four times the capacity of POWER9. For each
half of the core, instructions are fetched at a sus-
tained rate of up 32-bytes per cycle and predecoded
before being installed in a 48-KiB instruction cache
(50% more capacity than POWER9). During the prede-
code stage, select pairs of instructions can be identi-
fied for fusion into a single internal operation of the
microarchitecture, which leads to a faster and more
efficient execution of those instructions. The new 64-
bit prefix instructions in Power ISA 3.15 are also identi-
fied in that stage. POWER10 then decodes and dis-
patches to the execution slices up to 8 instructions
per cycle per thread, or 16 instructions per cycle per
SMT8 core. This represents a 33% increase in dispatch
rate when compared to POWER9. Over a thousand
instructions can be in-flight, from dispatch to commit,
in a POWER10 SMT8 core, representing a doubling of
the out-of-order execution capabilities over POWER9.
The translation lookaside buffer (TLB) has been
increased four-fold with 8192 entries per SMT8 core,
while at the same time reducing the latency and
increasing throughput over POWER9.
The four execution slices of POWER9 have been
widened to 128 bits each. This has resulted in a dou-
bling of the general SIMD rate of execution, to a
maximum of four SIMD instructions per cycle per
thread or up to 8 SIMD instruction per cycle per
SMT8 core. Crypto processing in the execution sli-
ces have also been enhanced, with an overall four-
fold gain in throughput from POWER9 to POWER10
core.
A single-thread of execution can load up to two
32-byte data chunks per cycle from the L1 cache,
March/April 2021

FIGURE 3. POWER10 core microarchitecture. The boxes on the left show the improvements over POWER9 on performance and
power efficiency, respectively. The latency numbers include both absolute values and improvements over POWER9.
with a total SMT8 core load bandwidth of 128 bytes
per cycle. (The same bandwidth can also be achieved
from the L2 cache.) A single thread of execution can
store up to four instructions per cycle by gathering
from up to two store queue entries when each entry
includes a fused store operation. Stores always tar-
get the L2 cache and maximum bandwidth is 32
bytes per cycle per thread or 64 bytes per cycle per
SMT8 core.
Complementing the four general purpose execu-
tion slices, POWER10 introduces a new matrix math
accelerator (MMA) unit, optimized for the execution
of new matrix instructions in Power ISA 3.1. The
instructions perform BLAS2- and BLAS3-class opera-
tions on eight 512-bit accumulator registers that are
added to the architecture. The instructions use either
two or three 128-bit vector-scalar registers to perform
rank-1, -2, -4 or -8 updates on either a 4  2 or 4  4
matrix stored in an accumulator. Each input vector-
scalar register contains either a 2  1 vector of dou-
ble-precision elements, a 4  1 vector of single-preci-
sion elements, a 4  2 matrix of 16-bit elements (half-
precision floating-point,6 bfloat16,7 or signed integer),
a 4  4 matrix of 8-bit elements (signed/unsigned inte-
ger), or a 4  8 matrix of 4-bit elements (signed
integer).
The MMA microarchitecture reduces data
switching by storing the accumulators locally in the
March/April 2021
HOT CHIPS
unit itself, significantly reducing the total data move-
ment (bits  distance) when compared to an equiva-
lent 512-bit SIMD operation. The result is improved
power efficiency and higher frequency, enabling the
POWER10 core to achieve a four-fold increase in
matrix math throughput compared to the POWER9
core.
In addition, a focus on power efficiency domi-
nated many other elements of the POWER10 core
microarchitecture and design. When compared to
the POWER9 core, there is more use of clock-gating
and an emphasis on reducing data switching. The
branch prediction accuracy has been improved,
which results in less wasted work and improves
thread latency. Instruction fusion also helps with
both performance and power efficiency, by combin-
ing multiple instructions in fewer operations.
POWER10 supports both independent and dependent
forms of fusion. Dependent fusion combines the exe-
cution of two instructions that share a register
dependence into a single operation (with no depen-
dent latency) or a latency optimized pair of opera-
tions, whereas independent fusion enables the
combining of loads or stores to adjacent memory
locations into a single wider access reducing
resource consumption and conflicts.
The register file for the general-purpose and vec-
tor-scalar registers requires four times fewer write-
IEEE Micro 11
HOT CHIPS
FIGURE 4. POWER10 core speeds and feeds. Load/store and SIMD bandwidth have been doubled over POWER9, matching SIMD
and load throughputs. The matrix math unit offers increased throughput in computational-intensive operations.
ports per entry, when compared to POWER9, without
compromising performance. The L1 data and instruc-
tion caches have been converted to use effective-
address tags, which means that address translation
from effective- to real-address only needs to be per-
formed on L1 cache misses, as opposed to on every
load or store, as with POWER9. Cache latency has also
been optimized reducing time in flight and pipeline
hazards including the addition of zero-cycle store for-
warding latency for loads and reduced latency for L2
and L3 caches.
The L1 data cache is a write-through cache and
stores commit to L2, which is physical-address
indexed and tagged, and inclusive of L1. L3 is a victim
cache for evictions from its local L2 and is also physi-
cal-address indexed and tagged. The L3 region associ-
ated with a core can also be populated with cast-outs
from other cores depending on the state of dynamic
sharing policies. A miss from a load or store by a core
can be sourced by any L2 or L3, either on the same or
another chip in the system.
The speeds and feeds of the POWER10 core are
summarized in Figure 4. The SMT8 core attaches to a
multilayered memory hierarchy, which provides it with
up to 32 bytes/cycle of simultaneous read and write
bandwidth to main memory. Read and write band-
width to the local 8 MiB L3 cache is 64 bytes/cycle for
an SMT8 core. Read and write bandwidth to the level-
2 and level-1 cache is 128 and 64 bytes/cycle for an
SMT8 core, respectively.
12 IEEE Micro
THE IBM POWER10 PROCESSOR
COMBINES INNOVATIONS IN SILICON
TECHNOLOGY, SYSTEM
INTERCONNECTS, MEMORY SYSTEMS,
AND PROCESSING CORE TO DELIVER
A COMPUTE ENGINE THAT IS
OPTIMIZED FOR THE MODERN DAY
DEMANDS OF THE ENTERPRISE ON
PREMISES AND IN THE CLOUD.
The table in the lower right corner of Figure 4
shows the computation rates that can be achieved
with the MMA instructions. The columns show the
maximum rate of operations per cycle for a single
instruction, a single-thread (which can issue up to two
instructions per cycle) and per SMT8 core. The opera-
tion rate increases as the input data type gets shorter,
and varies from a maximum of 64 (floating-point) oper-
ations per cycle per SMT8 core when the inputs are
double-precision floating-point elements to a maxi-
mum of 1024 (integer) operations per cycle per SMT8
core when the inputs are 4-bit integers.
POWER10 PERFORMANCE
Comparisons of performance between POWER9 and
POWER10 are shown in Figure 5. The comparisons are
March/April 2021
 HOT CHIPS

FIGURE 5. POWER10 general purpose socket performance gains. The three-fold improvement in performance comes from a
combination of increased number of cores and more powerful cores. DDR5 will double memory bandwidth in the future.

derived from presilicon simulations and have been cor-
related against first-pass silicon. We do not yet have
the final version of the chips and the results reflect the
projected frequency of operation for production
POWER10 parts. The figures are for a dual-socket
POWER10 system relative to a dual-socket POWER9
S924 server. We observe a three-fold improvement in
performance across integer (SPECint2017_rate), float-
ing-point (SPECfp2017_rate), and commercial bench-
marks. For memory streaming benchmarks, the
POWER10 gains over POWER9 range from two- to four-
fold, using DDR4 and DDR5 memory, respectively.
For computations that are heavy on matrix math,
the gains from POWER9 to POWER10 are even more
substantial, as shown in Figure 6. LINPACK is expected
to run ten times faster in POWER10 than POWER9,
when compared socket-to-socket. The same is
expected for single-precision floating-point implemen-
tation of the Resnet-50 benchmark. When some of the
new mixed-precision math features of POWER10 are
taken into account, our evaluation shows that Resnet-
50 will execute up to 15 (with bfloat16 data type) or 20
(with 8-bit integer data type) times faster than the stan-
dard single-precision Resnet-50 in POWER9.

FIGURE 6. POWER10 SIMD/AI socket performance gains. The matrix math accelerator delivers four times the throughput of
POWER9 SIMD. Combined with two-and-a-half times the number of cores, it results in a ten-fold improvement in socket through-
put for computational-intensive operations. Further gains are possible with the new reduced-precision operations.

March/April 2021 IEEE Micro 13

HOT CHIPS
CONCLUSION
The IBM POWER10 processor combines innovations in
silicon technology, system interconnects, memory sys-
tems, and processing core to deliver a compute
engine that is optimized for the modern day demands
of the Enterprise on premises and in the cloud. Innova-
tive memory interfaces and cross-system memory
functionality enable new architectures for cloud com-
puting, while improved performance and power effi-
ciency of the computational cores support a three-
fold improvement in comparable system throughput
over a broad spectrum of applications. For matrix-
intensive computations in the fields of machine- and
deep-learning, the gains can be more than ten-fold.
REFERENCES
1. S. K. Sadasivam, B. W. Thompto, R. Kalla, and W. J.
Starke, “IBM power9 processor architecture,” IEEE
Micro, vol. 37, no. 2, pp. 40–51, Mar./Apr. 2017.
2. W. A. Hanson, “The CORAL supercomputer systems,”
IBM J. Res. Develop., vol. 64, no. 3/4, pp. 1:1–1:10,
May/Jul. 2020.
3. J. Stuecheli, S. Willenborg, and W. Starke, “IBM’s next
generation POWER processor,” in Proc. IEEE Hot Chips
31 Symp., Cupertino, CA, USA, 2019, pp. 1–19.
4. W. J. Starke et al., “The cache and memory subsystems
of the IBM POWER8 processor,” IBM J. Res. Develop.,
vol. 59, no. 1, pp. 3:1–3:13, Jan./Feb. 2015.
5. IBM Corporation, “Power ISA version 3.1,” May 2020.
6. “IEEE Standard for Floating-Point Arithmetic—Redline,"
in IEEE Std 754-2019 (Revision of IEEE 754-2008),
pp. 1–148, Jul. 2019.
7. G. Tagliavini, S. Mach, D. Rossi, A. Marongiu, and
L. Benin, “A transprecision floating-point platform for
ultra-low power computing,” Proc. Des., Autom. Test
Eur. Conf. Exhib., Dresden, Germany, 2018,
pp. 1051–1056, doi: 10.23919/DATE.2018.8342167.
WILLIAM J. STARKE is an IBM Distinguished Engineer and
the Chief Architect and Engineer for the IBM POWER10
14 IEEE Micro
processor. He is responsible for shaping the processor cache
hierarchy, symmetric multiprocessor (SMP) interconnect,
cache coherence, memory, and I/O controllers, accelerators,
and logical system structures for Power Systems. He is an IBM
Master Inventor, authoring roughly 300 patents. Starke
received a B.S. in computer science from Michigan Technolog-
ical University. Contact him at wstarke@us.ibm.com.
BRIAN W. THOMPTO is an IBM Distinguished Engineer and
the Chief Architect for the IBM POWER10 core. He has led
global development teams across ten generations of IBM
POWER and IBM System z processors. He has been recog-
nized by two IBM corporate awards and as an IBM Master
Inventor. Thompto received a B.S. in electrical engineering
and computer science from the University of Wisconsin Madi-
son. Contact him at bthompto@us.ibm.com.
JEFF A. STUECHELI is a Senior Technical Staff Member in the
IBM Power Systems Development Group. He works on server
hardware architecture. His most recent work includes
advanced memory architectures, cache coherence, and
accelerator design. He has contributed to the development
of numerous IBM products in the POWER architecture family,
most recently the POWER10 design. He has been appointed
an IBM Master Inventor, authoring over 190 patents. Stue-
cheli received a B.S., an M.S., and a Ph.D. in electrical engi-
neering from The University of Texas Austin. Contact him
at jeffas@us.ibm.com.
 E. MOREIRA is a Distinguished Research Staff Member
JOSE
with the IBM Thomas J. Watson Research Center. Moreira
received a B.S. in physics and a B.S. and an M.S. in electrical
engineering from the University of Sao Paulo and a Ph.D. in
electrical engineering from the University of Illinois at Urbana-
Champaign. He is a Fellow of IEEE and a Distinguished Scien-
tist of ACM. Contact him at jmoreira@us.ibm.com.
March/April 2021
THEME ARTICLE: HOT CHIPS

Marvell ThunderX3: Next-Generation

Arm-Based Server Processor
Rabin Sugumar , Mehul Shah , and Ricardo Ramirez , Marvell Semiconductor Inc., Sunnyvale, CA, 475644,
USA

Marvell ThunderX3 is the third-generation Arm-based server processor from

Marvell. This article describes the architecture of ThunderX3, going into details of
core microarchitecture. ThunderX3 is distinctive among mainstream processors in
supporting four SMT threads. The article describes the threading microarchitecture
of ThunderX3 and highlights threading benefits. The article also touches on the
on-chip interconnect and L3-cache and the power management subsystem. Initial
benchmark results from silicon show 30%þ single thread and up to 3X socket level
performance gains over the prior generation resulting in industry-leading single
thread and socket level performance.

T
hunderX3 is the latest server chip from Marvell
based on the Arm instruction set architecture
(ISA) manufactured in TSMC 7 nm.
Server processors constitute a segment of the
overall processor market. They are typically deployed
in racks in temperature-controlled warehouse settings
called datacenters and are accessed from client devi-
ces for a variety of purposes including checking email,
web search, and cloud services. Historically, several
ISAs have been used in server processors including
IBM mainframe ISAs, SPARC, PA RISC, and Itanium.
While there are still important residual islands that
use these other architectures, most servers today run
on the x86 ISA created by Intel. The shift to x86 was
driven by the low cost of x86 servers and the availabil-
ity of Linux, a mature, open source operating system.
The low cost of x86 servers was enabled by the econo-
mies of scale resulting from the use of the x86 archi-
tecture in PCs. However, processor shipment volumes
have now shifted from PCs to cell phones where the
Arm ISA is almost universally used. Marvell and other
companies are riding the mobile wave to enter the
server market with the Arm ISA.
The design of server processors is focused on
reducing the total cost of ownership (TCO) at the
datacenter—that is, to provide the required service at
the lowest total dollar cost, including both the cost to
0272-1732 ß 2021 IEEE
Digital Object Identifier 10.1109/MM.2021.3055451
Date of publication 29 January 2021; date of current version
26 March 2021.
buy and the cost to run. From the perspective of pro-
cessor design, TCO optimization translates to optimiz-
ing performance per dollar and performance per watt
at the platform level. TCO optimization is a multidi-
mensional problem and different datacenters take dif-
ferent approaches. Typically, there is a maximum
power [referred to as thermal design power (TDP)]
that datacenters specify for a processor—the data-
center design requires that the processor stay under
this TDP under all circumstances. The design goal for
a server processor is then to achieve the maximum
possible performance while staying under this TDP.
There are die area limits as well to manage the
manufacturing cost of the die. The workload that is
often used as an initial performance gate is the CPU
benchmark from the SPEC organization.1 Both single
thread and rate performance of the CPU benchmark
are important—single thread as a measure of the
responsiveness of the system and rate as a measure
of the throughput capability. The CPU benchmark
from SPEC is a good overall measure of the capability
of a server chip but may not always correlate directly
to performance seen on customer applications.
Focus areas of server processor design, thus,
include optimizing single thread performance as mea-
sured by the Integer CPU single thread benchmark
from SPEC, optimizing socket level performance as
measured by the Integer CPU rate benchmark from
SPEC, optimizing performance on workloads com-
monly run in the datacenter such as web servers,
databases, web search, Java application servers, and
other such workloads, optimizing die area and power,

March/April 2021 Published by the IEEE Computer Society IEEE Micro 15

HOT CHIPS

obtaining performance from more cores running

slower is more efficient with respect to power dissipa-
tion than performance from fewer cores running faster
owing to the higher voltage required for faster opera-
tion and the quadratic impact of voltage on power.
Each of the 60 cores has four hardware threads. x86
systems from Intel and AMD support two threads—
the ThunderX line uses four threads since in the server
space cache miss rates are so high that there is still
significant pipeline slack with two threads and good
additional performance gains are seen at four threads.
The cores implement Armv8.3 with select v8.4 and
v8.5 features pulled in.2

FIGURE 1. ThunderX3 overview of key features.

IN THIS ARTICLE, WE WILL START
providing as much memory capacity and bandwidth, WITH AN OVERVIEW OF THUNDERX3
and I/O bandwidth as possible within the constraints of AND THEN WE WILL DESCRIBE THE
die area, providing an interconnect for multisocket ARCHITECTURE OF THUNDERX3
deployments, good virtualization support for cloud GOING INTO SOME DETAIL ON THE
deployments, and providing secure boot, monitoring
CORE, THREADING, AND MEMORY
and management facilities for large-scale deployments.
SUBSYSTEM ARCHITECTURE.
ThunderX3 is the third-generation Arm-based server
chip from Marvell. The chip prior to ThunderX3 was
ThunderX2. ThunderX2 is a 32 core Armv8.1
processor—each core had four hardware threads—so a
ThunderX3 has eight 64-bit wide DDR4 SDRAM
single ThunderX2 die has 128 CPUs exposed to the OS.
channels with a max bit transfer rate of 3.2 Gbits/s on
ThunderX2 has several firsts. It featured in the first Arm-
the data pins. The peak DDR bandwidth per ThunderX3
based system that entered the Supercomputer Top500,
chip works out to 204 GB/s and efficiency around 80%
it is the first non-x86 system to be deployed in Microsoft
of peak is reasonable. More memory bandwidth will be
Azure, it had industry-leading performance in particular
helpful to feed the 60 cores on certain applications, but
on memory bandwidth-intense benchmarks at the time
there are cost-benefit tradeoffs and eight seems the
it was released and is the most widely deployed Arm
optimal design point. ThunderX3 has 64 PCIe lanes sup-
server processor. In the last two years, ThunderX2 has
porting up to Gen4 speeds, which is 16 giga-transfers/s
gone through qualification cycles at datacenter vendors
per lane. ThunderX3 does not include any on-die acceler-
and a variety of issues have been addressed that are
ators. While accelerators are widely used in the datacen-
required for at scale deployment in datacenters. Thun-
ter, different customers have their own preferred
derX3 builds on this experience.
external accelerators and there is no industry standard
In this article, we will start with an overview of
accelerator that may be usefully integrated on the die.
ThunderX3 and then we will describe the architecture
This is one reason custom server designs that are opti-
of ThunderX3 going into some detail on the core,
mized for each customer is expected to gain traction in
threading, and memory subsystem architecture. The
future generations. ThunderX3 also integrates USB and
focus will be on the features that made a difference
SATA ports for local storage and miscellaneous device
on the performance with some insight provided on
support, avoiding the need for external chips to provide
why the features help. We also present performance
such support on a platform.
data from ThunderX3 silicon.
ThunderX3 includes a sophisticated on-die power
monitoring and management system that has access
THUNDERX3 OVERVIEW to a variety of sensors and performance counters on
Figure 1 has an overview of the key features of Thun- the chip and the platform and is able to control fre-
derX3. On a single die, there are 60 cores. High core quency individually of the cores and the memory inter-
count works well on server processors since the work- connect as well as control voltage, power state, and
loads often have inherent concurrency—for instance, performance configuration of cores. It periodically
serving web pages for multiple clients. In addition, monitors the status of different sensors and

16 IEEE Micro March/April 2021


FIGURE 2. ThunderX3 core block diagram.
performance counters, and in conjunction with input
from software (OS or Hypervisor) makes decisions on
frequency and voltage settings and configuration to
optimize chip performance and power.
THUNDERX3 CORE
The ThunderX3 core is a deep out of order core with
support for four-way multithreading. On single thread
performance, it is competitive with the highest end
cores from x86 competitors, and the support for four-
way threading allows it to outperform high-end com-
petitor cores on a wide variety of workloads.
Figure 2 has the core block diagram. Up to eight
instructions per cycle are fetched from the instruc-
tion cache—way prediction is used at the instruc-
tion cache to simplify the design and to reduce
power. Concurrent with instruction cache access,
various branch prediction structures are also read to
decide the next bundle of instructions to be fetched.
There are separate prediction structures for condi-
tional branches, indirect branches, and returns. Sup-
port for decoupled fetch was added in ThunderX3—
this is a mechanism to keep fetching following an
instruction cache miss, using BTBs and branch pre-
dictors to predict future PCs—it is quite effective
showing 1.5 to 2 gain on datacenter codes such
as webservers and databases that tend to have a
large instruction footprint and predictable basic
block sequences. Fetch bundles break on a 64B
cache line boundary and on a taken branch. Front-
end inefficiencies introduced by fetch breaks and
branch resteer latencies are a major challenge in
high-end CPU design since IPCs on many cache resi-
dent codes may reach 3 or more, and even a single
cycle additional delay on a frequently executed
branch resteer could lower IPC by a high percentage.
March/April 2021
HOT CHIPS
ThunderX3 implements
resteer at different lev-
els to reduce these pen-
alties and does bundle
merging to smooth out
the instruction stream.
The eight instructions
that are fetched go into
the decoder. The
decoder maps instruc-
tions to micro-ops—
most instructions map to
a single micro-op, but
there are a few that map
to multiple micro-ops.
Micro-op expansion was
reduced significantly going from ThunderX2 to Thun-
derX3. ThunderX2 was derived from an earlier MIPS-
based architecture, and during the transition to Arm,
most instructions that did not map to a MIPS instruc-
tion were broken into micro-ops. In ThunderX3,
instructions such as loads and stores with register
plus register addressing that see widespread use by
Arm compilers were mapped to a single micro-op.
Reducing micro-op expansion results in additional
complexity in the execution unit to execute these
more complex operations, but the performance gain
was worth it. Decode also fuses certain instruction
pairs such as simple integer instructions and branches
into a single micro-op. On average across the SPEC
integer suite about 0.95 micro-ops are output by
decode per instruction.
Decoded micro-ops go into a skid buffer—the skid
buffer is the separation point between the front end
of the pipe and the back end of the pipe. The skid
buffer is one of the few structures in the core that is
statically partitioned among the threads—to simplify
thread arbitration at dispatch. The renaming unit picks
up micro-ops from the skid buffer, does renaming, allo-
cates various backend structures for the micro-op,
and writes the micro-op into the scheduler, the reor-
der buffer and other queues such as the load and
store queue as needed. Renaming is done at four
micro-ops per cycle and there are optimizations
around merging to reduce instruction delivery ineffi-
ciencies created by breaks in fetch bundles. Note that
renaming width is four while fetch and decode width
are eight—it may seem that the pipeline is out of bal-
ance, but in practice most workloads do not run any-
where close to peak rates in a sustained fashion.
Performance and design studies showed that increas-
ing rename width had low performance benefit at the
current design point, but high design cost.
IEEE Micro 17
HOT CHIPS

TABLE 1. Sources of single thread gains on ThunderX3 over

ThunderX2.

Feature Rough Pct gain

(SPECInt)
Size Icache Size 0.5%
512 KB L2-cache 2.5%
Larger out-of-order 5%
structures
Width Wider decode 2%
Additional ALU port 1.5%
Two branches per 1.5%
cycle
Algorithm Branch prediction 3%
enhancements
FIGURE 3. ThunderX3 measured gains over ThunderX2.
Front-end resteer 1%
enhancements
Reduce micro-op 6% 30% higher than ThunderX2 at the same frequency,
expansion
and on the Floating-point version (SPECFp), it is 35%
D-cache bank 0.5% higher. Obtaining 30%–35% on single thread on a
conflict reduction
broad benchmark suite such as cpu2017 requires
Reduce FP 1%
improvements on several fronts. Table 1 provides
structural hazards
some color on where the gains come from. The
Prefetch 1.5%
enhancements improvements may be classified along four dimen-
sions: increase in size, increase in width, algorithmic
Latency FP latency reduction 0.5%
improvements, and latency improvements. On size,
the L1 instruction cache and L2-cache sizes were
The next step is for micro-ops to be issued and doubled and various out of order structures were
executed. This is done from a 70-entry unified issue made larger such as the reorder buffer, the scheduler,
queue and micro-ops may issue to one of seven and the load and store queues. On width, decode was
ports: four ALUs or four FP/Neon units, two load/ made wider, an ALU was added, and support for
store address pipes, and a store data pipe. Since branch execution on an additional ALU was also
there is just a single store data port sustained added. Even though the size and width get the most
throughput on stores is limited to one per cycle. Max- visibility when cores are discussed, it is seen from
imum transfer width between the L1 D-cache and the Table 1 that the major source of gains was algorith-
pipeline is two 16B blocks. mic. Reduction in micro-op expansion from Thun-
The data cache is 32 KB, eight-way. Both the data derX2 improved pipeline efficiency significantly. In
cache and the instruction cache are backed up by a addition, branch prediction and prefetch algorithms
unified 512-KB L2-cache. The hardware prefetcher pre- were enhanced and other algorithmic tweaks at vari-
fetches into the level 2 cache—it keys off stride pat- ous points in the pipeline got incremental gains. The
terns and in addition prefetches aggressively around last dimension is latency—floating point latencies
cache lines where a miss is seen to perform region were reduced—this bought a little bit of performance
prefetching. There are small level 1 TLBs adjacent to on SPECInt and even more on SPECFp.
both the instruction cache and the data cache and a Figure 3 shows some performance data from sili-
large level 2 TLB. Four-way threading support is dis- con. On single thread SPECInt performance gain is
tributed throughout the core and is discussed in greater than 1.5 from a combination of architecture
greater detail later in this article. performance improvement and frequency increase.
On SPECInt, the rate performance gain is around
2.5 from a combination of architectural gains,
THUNDERX3 PERFORMANCE increased core count, frequency increase, and the
ThunderX3’s single thread performance on the Inte- larger L3-cache and greater DDR bandwidth.
ger version of the cpu2017 benchmark (SPECInt) is On SPECFp single thread, the gains are higher than

18 IEEE Micro March/April 2021


FIGURE 4. Thread arbitration.
1.5 and on datacenter codes we are seeing 2–3
on a variety of codes.
THUNDERX3 THREADING
There are four hardware threads per core on ThunderX3.
Each hardware thread has the full Arm CPU state—so
the OS sees four CPUs per core. The four hardware
threads share almost all core resources including
caches, execution units, pipeline stages, structures
such as the reorder buffer and the scheduler as well as
interfaces to the external world. The core was designed
with four threads from the beginning—there was never
a single thread variant of the core. A back of the enve-
lope analysis of area penalty of threading was done and
the area penalty came out to around 5%. The gain from
threading on a variety of codes is well over 5%—so
threading is quite area and power efficient.
Figure 4 describes thread arbitration, which provides
a flavor of how threading is implemented in the core.
There are four points of arbitration. Once at fetch, once
at rename, once in the scheduler, and once at retire. At
fetch, the goal of the arbitration algorithm is to treat the
four threads uniformly while achieving the maximum
execution rate. On each cycle, the algorithm looks at
the active threads and picks the thread that has the
fewest instructions/micro-ops in flight further down the
pipe. That ensures that among active threads, pipeline
utilization is balanced to some extent. Similarly, at
rename, among the threads that have at least one
micro-op in the skid buffer, arbitration picks the thread
that has the fewest micro-ops further down the pipeline.
The scheduler is threading agnostic—it picks the oldest
FIGURE 5. Threading performance gains.
March/April 2021
HOT CHIPS
micro-op in the scheduler for each issue port on each
cycle. The execution units are threading agnostic as
well, executing whatever micro-op is delivered to them
by the scheduler. Finally, the retire unit picks the thread
that has the most micro-ops to retire, with a mechanism
to prevent starvation when a single micro-op in a thread
has not retired for a while. With this arbitration scheme,
when running similar workloads on all threads, perfor-
mance is uniform among the threads. Of course, when
the threads are running workloads with vastly different
profiles, the notion of what is fair itself is not clear, but
results are reasonable with no thread experiencing dra-
matic slowdowns.
Figure 5 shows single core performance gains from
threading—threading gains often correlate with inst-
ructions per clock (IPC) that a single thread is able to
achieve, which is a measure of how efficiently the
thread is using the pipeline—the more slack there is in
execution, the more opportunity there is for threading
gains. But cache pressure is a factor in some cases.
On MySQL, which runs at a low IPC, we see more than
2 gain from going to four threads. For a benchmark
such as leela (a go playing code from cpu2017), which
achieves mid-range IPC, the gain from threading is still
good at 1.7 to 1.8 at four threads. For x264, which is
video encoder also from cpu2017, the IPC is high but
there are still decent gains.
THUNDERX3 SOC
Figure 6 shows the overall SOC—it is a switched ring
with a switch at the top and bottom. Each leg is bidi-
rectional, and the traffic is routed through the
switches to the different legs. During the
design of ThunderX3, a mesh was consid-
ered for the interconnect but given that
DDR bandwidth was not increasing dra-
matically from ThunderX2 to ThunderX3
a mesh did not seem necessary, and the
switched ring architecture was adopted
to minimize design changes.
Cores are grouped in four core clus-
ters along with an interconnect slice,
L3-cache and coherence control logic
IEEE Micro 19
HOT CHIPS
FIGURE 6. ThunderX3 SOC block diagram.
FIGURE 7. Socket level performance.
to form tiles. Tiles are replicated to form the Core
and L3-cache cluster in the middle of the die. DDR,
PCIe, and other I/O tap into the rings and are on
the periphery. Support logic such as the interrupt
controller, the system management unit, and debug
assist also tap into the rings and are in the
periphery.
The L3-cache is 90 MB (1.5 MB per core) on Thun-
derX3 versus 32 MB (1 MB per core) on ThunderX2.
Physical addresses are mapped statically to L3-cache
tiles and the location of the requesting core has no
impact on tile selection. The large, shared L3-cache
allows large datasets and instructions of highly mul-
tithreaded datacenter codes to be held in the
20 IEEE Micro
L3-cache efficiently without duplication. The L3-
cache is mostly exclusive—i.e., it is filled on evict
from the L2-cache—an exclusive cache architecture
increases the number of distinct cache lines that are
held in on-die SRAM. For many datacenter codes, L3-
cache bandwidth is the primary scaling limiter and
much work was done during the design to optimize
L3-cache bandwidth. Running the STREAM bench-
mark3 out of L3-cache, the per core L3-cache band-
width on ThunderX3 is greater than ThunderX2
despite the almost 2 increase in core count on
ThunderX3 over ThunderX2.
Figure 7 shows socket level performance from
threading on a MySQL benchmark4—there is a linear
region where cores are kicking in, and then there is
a threading region where the additional threads on
each core kick in and finally it flattens out—at the
socket level, we see 89 over a single thread from a
combination of cores and threads, which is excellent
scaling.
CONCLUSION
ThunderX3 is a major upgrade over ThunderX2 achiev-
ing up to 3X performance in the same performance
envelope in single die configurations. Single thread
performance is industry leading. Four-way multi-
threading provides a significant competitive advan-
tage over other systems.
ACKNOWLEDGMENTS
The authors would like to thank their colleagues,
hundreds of engineers over the years, who worked
days, nights, and weekends, sometimes under
extremely challenging circumstances, to build Thun-
derX3 and its predecessors. They would also like to
thank the anonymous reviewers for the helpful
feedback.
REFERENCES
1. 2017. [Online]. Available: https://www.spec.org/cpu2017/
Docs/overview.html
2. Arm Developer. Understanding the Armv8.x extensions.
2019. [Online]. Available: https://developer.arm.com/
architectures/learn-the-architecture/understanding-
the-armv8-x-extensions/single-page
3. 2016. [Online]. Available: http://www.cs.virginia.edu/
stream/ref.html
4. 2020. [Online]. Available: https://github.com/akopytov/
sysbench
March/April 2021

RABIN SUGUMAR is a Chief Architect with Marvell, Sunnyvale,
CA, USA and leads the architecture group for the ThunderX
server processor line. In his role, he participates in a variety of
aspects of ThunderX development and productization. He
joined Marvell from Cavium, which Marvell acquired in 2018.
Prior to this, he was with Broadcom, where he was one of the
lead architects on the server processor that later
became ThunderX2 when the team from Broadcom moved to
Cavium. During his career, he has worked on architecture and
design of vector processors at Cray Research; early multi-
threaded and out-of-order SPARC processors with Sun Micro-
systems; and Arm-based processors with Broadcom, Cavium,
and now, Marvell. Sugumar received a Ph.D. in computer sci-
ence and engineering from the University of Michigan, Ann
Arbor, MI, USA. Contact him at rsugumar@marvell.com.
MEHUL SHAH is a Principal Architect involved in the design
of server processors with Marvell, Sunnyvale, CA, USA. He
came to Marvell through the acquisition of Cavium. Prior to
that, he was one of the designers of the server processors
March/April 2021
HOT CHIPS
with Broadcom that later became ThunderX2. Before Broad-
com, he was involved in architecture and verification of high-
performance embedded processors at two startups, PA Semi
and SiByte. Shah received a B.S. in electrical engineering
and computer science from UC Berkeley, Berkeley, CA, USA
and an M.S. in electrical engineering from UCLA, Los Angeles,
CA, USA. Contact him at mehuls@marvell.com.
RICARDO RAMIREZ is a Principal Engineer with Marvell, Santa
Clara, CA, USA and is the lead designer on the ThunderX CPU
logic design team. He joined Marvell with the Cavium acquisi-
tion in 2018. Before Cavium, he was with Broadcom as the lead
designer on the server processor that became ThunderX2
when the team moved to Cavium. Throughout his career, he
has been involved in developing many high-performance CPU
cores, which include Intel’s Itanium processor and Broadcom’s
XLR and XLP line of MIPS-based processor. Ramirez received
an M.S.E.E. from Stanford University, Stanford, CA, USA. Con-
tact him at rramirez@marvell.com.
IEEE Micro 21
 THEME ARTICLE: HOT CHIPS

The Xbox Series X System Architecture

Mark Grossman and Jeffrey Andrews, Microsoft, Redmond, WA, 98052, USA

The Xbox Series X console, released in November 2020, contains a System on Chip
(SoC) created in partnership with AMD. This article describes its architecture
including the intelligence for input processing, rendering game graphics and audio,
managing storage, user services, and security, all under a tiered operating system.

KEY FEATURES As listed above, there are multiple custom on-die

T
he Xbox 2020 console generation features
numerous upgrades to the previous architec-
ture that powered console products from 2013
to 2017, including Xbox One, Xbox One S, and Xbox
One X. Some of the key architectural features of the
Series X1 are listed in Table 1.
subsystems for multimedia, security, and I/O. Media-
related blocks are fed by the Media Hub interface, while
security and audio blocks are fed by the System Hub for
best quality of service. The primary I/O interface is the 8-
lane Gen4 PCI Express interface, which connects to a
South Bridge chip, internal and (optional) external NVMe
flash SSDs, and gigabit ethernet NIC.

THE DIE AND PACKAGE MEMORY CONFIGURATION AND

Figure 1 shows a physical view of the SoC die. Note
that the bulk of the die—more than 75%—is dedicated
to the GPU. The bandwidth of the wide DRAM inter-
face is sized for the GPU efficiency. The die is 360.4
mm2 with 15.3 billion transistors. It is fabricated in
TSMC’s N7 process node and packaged on a 12-layer
substrate in a 52.5-mm2 2963-ball BGA package, with
0.80-mm minimum ball pitch.
Figure 2 shows the system block diagram. The
eight simultaneous-multithreading CPUs are orga-
nized in two clusters, or complexes, of four cores.
Each core has its own first-level instruction and data
caches, as well as its own 512-kB second-level cache.
Each cluster has its own 4-MB third-level cache,
which is populated by cast-outs from the CPU L2s.
The LLC is effectively 4 L2s þ L3 capacity, thus 6 MB
per cluster. The CPU caches are coherent with the
GPU—i.e., they can be snooped by all CPU cores and
by the GPU. The CPU, GPU, and communication hubs
share access to the Scalable Data Fabric for DRAM
access. There are 20 16-bit-wide GDDR6 DRAM chan-
nels, each with its own interface containing queues
and scheduler. The DRAM interfaces are placed on
three sides of the die; the fourth side is left free for
power delivery.
0272-1732 ß 2021 IEEE
Digital Object Identifier 10.1109/MM.2021.3058629
Date of publication 15 February 2021; date of current version
26 March 2021.
PERFORMANCE
The 20-channel DRAM memory subsystem balances
high bandwidth for the GPU and low latency for the
CPU cores. Because the memory capacity (16 GB) is
not an integer multiple of 20, memory space is seg-
mented into a high-bandwidth 10-GB partition spread
across all channels, and a 6-GB partition spread
across 12 of the channels. The system places lower
bandwidth resources in the latter partition.
At 14 Gb/s raw pin bandwidth, the full memory sub-
system can achieve a theoretical peak of 560 GB/s, all
of which may be needed by the GPU. As with every
SDRAM-based system, this is derated due to row
refresh, read-write turnaround, and new-page access
overhead. The hierarchical caches, write-gathering,
and memory interface logic optimize the batching of
requests to maximize utilization. Each channel has 16
banks, giving a total of 320, which helps hide overhead
and helps lower the average CPU memory latency
despite the high GPU demand.
COST/PERFORMANCE TRENDS
Moore’s Law has been a reliable basis for increasing the
number of transistors per square die millimeter. That has
more than tripled between 2013 (Xbox One) and 2020
(Series X). It has been a less reliable benefit to power
consumption, and especially die cost, in the most recent
silicon technology nodes. The die area for successive
Xbox console SOCs has remained roughly constant, but
the SOC cost has increased considerably, due to
increased silicon fabrication process layer count,

22 IEEE Micro Published by the IEEE Computer Society March/April 2021


TABLE 1. Key features.
Area Feature
Compute Eight 3.8-GHz “Zen 2” CPU cores
Graphics Sampler Feedback Streaming (SFS)
DirectX Raytracing acceleration (DXR)
Variable Rate Shading (VRS)
Machine Learning acceleration
Direct3D Mesh Shading
Memory 320-bit 14-Gbit/s GDDR6 DRAM, 560-GB/s
peak
Storage Xbox Velocity Architecture – 2.4 GB/s (Raw),
4.8 GB/s (compressed), encrypted assets from
NVMe SSD
Audio Opus Audio Decode
High quality sample rate converter
Project Acoustics (3D audio) acceleration
Display 8K UHDTV support
120-Hz TV support
HDMI 2.1 with 10-Gbit/s FRL, ALLM, and DSC
Variable Refresh Rate (VRR)
Linear light display processing
Security Host Security Processor (HSP) root of trust
Media Security Processor (MSP)
SHACK elevation of privilege protection
additional complex processing steps, and the cost to
develop new architectural structures of a growing het-
erogeneous system. The primary mitigating factors that
keep overall product cost from increasing are as follows:
› Architectural improvements on existing IP
blocks such as the CPU and GPU.
› New custom subsystems to offload more gen-
eral-purpose programmable CPU and GPU
workloads.
FIGURE 1. The SoC die.
March/April 2021
HOT CHIPS
THE CPUs
The high-end out-of-order CPU cores bring high-effi-
ciency compute to demanding 120-frames/s work-
loads. They run at 3.8 GHz with one thread enabled, or
3.6 GHz with two threads. One Xbox addition is Secu-
rity Privilege Level Execution and Access Protection
(SPLEAP), which is a hardware function to prevent
Escalation of Privilege attacks. Xbox One systems
used an earlier version of SPLEAP.
The design also incorporates 2x floating point pipe-
lines. Using AVX256 instructions gives a peak of 32 sin-
gle-precision floating-point operations per clock cycle
or 972 GFLOPS for eight CPU cores. This is a valuable
compute resource for the most demanding use of
these common operations.
DISPLAY ENGINE
Display processing is the most difficult real-time func-
tion in the Xbox console due to the high bandwidth for
all the display planes—game, video, and user inter-
face. It is very important to keep display processing
off the GPU’s unified shader engine so more GPU per-
formance is available for rendering.
Xbox Series X performs full linear-light processing,
enabling extremely high quality resizing and composi-
tion of input display planes. In the early days of high
dynamic range2 and wide color gamut content, some
operations were performed on gamma coded pixel
values, which can cause color shifts, banding, and
other artifacts. Linear values require higher preci-
sion than gamma-coded values to preserve both
dynamic range and precision but were chosen for
the sake of fidelity. The addition of a hardware 3-D
lookup table for dynamic range conversion also off-
loads GPU work.
Three features that Xbox had been trying to bring
to the HDMI standard since the beginning of HDMI 2.0
efforts are now fully implemented in Xbox Series X via
HDMI 2.1.3 These are auto low latency mode, variable
rate refresh, and display stream compression, which
improve latency and image quality without increasing
bandwidth.
AUDIO UNITS
Overall, the three hardware audio engines in Xbox
Series X have more peak single-precision floating-
point performance than all eight of the Xbox One X
CPU cores running at 2.3 GHz. The CPFU2 is a new
engine focused on efficient audio convolution, FFTs,
reverb, and complex arithmetic for audio algorithms. It
enables new realistic audio experiences such as in
Project Acoustics,4 where environments are modeled,
IEEE Micro 23
HOT CHIPS
FIGURE 2. SoC block diagram.
and 3-D audio sources are simulated in real time.
MOVAD is a hyper real-time Opus5 audio decoder,
with a throughput-matched high-quality sample rate
converter. It can process more than 300 real-time
channels of Opus. Because of its optimal quality and
compression ratio, the Opus CODEC was chosen to
implement in hardware. Opus allows different SILK
voice CODEC and CELT music CODEC mixes per audio
frame. The SRC engine in MOVAD has >100 dB signal-
to-noise ratio across game use cases, which are much
more difficult and varied than traditional music/voice
audio.
FIGURE 3. GPU block diagram.
24 IEEE Micro
PLUTON HSP AND MSP
The integrated Hardware Security Platform6 (HSP)
raises Xbox’s already robust hardware security thresh-
old with the addition of Secure Hardware Crypto Keys
(SHACK). The HSP orchestrates security operations
without other software or firmware involvement.
The Media Security Platform unit (MSP) offloads
simultaneous cryptographic and compression-related
processing of streams of storage data to/from the
NVMe SSD. The MSP performs LZ lossless decompres-
sion and custom, texture optimized BCPack decompres-
sion, giving an average 2:1 space and bandwidth boost.
March/April 2021

XBOX VELOCITY ARCHITECTURE
(XVA)
New Xbox hardware and software innovations contrib-
ute to storage acceleration.
DRAM technology has for many years failed to shrink
capacitor size significantly, making it challenging for con-
sole systems to grow capacity. Fortunately, flash mem-
ory has continued to cost-reduce very well. From 2012–
2020, DRAM prices have reduced by about 5% year over
year, compared with approximately 23% for flash.
This reduction has allowed the hard disk drive
(HDD) to be replaced with a flash drive. This approach
allows the multiple NAND flash chip instances to
deliver the needed bandwidth for filling a DRAM-based
cache acting as a dynamic game art content cache
instead of a static buffer. The SSD acts as a read-only
backing store, resulting in effective DRAM art content
storage to be many times larger. An added value of
the SSD to users is that HDDs were doomed to have
increasingly poorer load times, due to the inherent
square term density versus linear bit-rate problem.
These are the main components of XVA7:
› The SSD itself, which has a customized NVMe
controller to deliver a guaranteed 2.4 GB/s. This
is more than 40 times the bandwidth of Xbox
One.
› The MSP described above.
› A new DirectStorage API, which bypasses the
traditional system file I/O protocols to give appli-
cations simplified, driver-supported control over
streaming-oriented accesses.
With these coordinated elements, along with Sam-
pler Feedback Streaming detailed later, games can
achieve an overall 2.5 improvement in RAM utiliza-
tion. This is largely since the GPU often accesses only
a fraction of a given texture, especially at the finest
levels of detail.
THE GPU
The goal for the GPU was to create a console-class
design that significantly advances gamers' sense of
immersion in realistic worlds. A generational increase in
raw graphics operations per second is natural, but
because of the cost considerations described previ-
ously, little real estate could be devoted to brand new
GPU functions, so enhancements were added judi-
ciously. Figure 3 shows the overall RDNA-based GPU
structure. It fully supports Direct3D12 feature level 12_2.
There are four Shader Arrays, each with 6 or 7 dual
CUs for 26 total active. Each Shader Array has its own
March/April 2021
HOT CHIPS
color, depth, rasterization, and primitive assembly
units. Frame buffers are tiled so that color and depth
units work in spatially balanced subspaces. A single
unified Geometry Engine handles primitives including
mesh shading and generates higher order surfaces. A
new three-level cache hierarchy (shown in blue) opti-
mizes data latency and sharing among related threads
of execution. It can snoop the CPU caches (but CPUs
cannot snoop the GPUs). Translation from virtual to
physical addressing is likewise handled via a three-
level TLB hierarchy. The entire shader complex is fed
by a multicore command processor with custom firm-
ware to handle two independent command streams.8
Dual CU
The workhorse of the GPU is the dual compute unit,
also known as the Work Group Processor. It has four
32-wide SIMD units arranged to execute either 32-wide
or 64-wide scalar thread groups and to allow sharing of
local data held in the LDS. Each ALU lane can perform
a fused multiply-add per clock, performing full-rate 32-
bit math and double-rate 16-bit floating-point math.
The faster single-cycle instruction issue rate reduces
stalls. The sequencer coissues up to seven instructions
of four types per cycle per CU, or a total of 14 instruc-
tions per cycle for the dual unit. There are three types
of level zero caches—one for SIMD texture and SIMD
load/store operations, one for instructions, and one for
scalar data, shared among all SIMD units in the WGP.
Architecturally, these CUs have substantially better
performance per clock cycle on average graphics work-
loads relative to the CUs in AMD’s GCN generation.
GPU Evolution
The normalized graph in Figure 4 shows that process
and design advances have allowed raw Xbox shader
TFLOPS (trillion float operations per second) to
FIGURE 4. Evolution of performance and capacity.
IEEE Micro 25
HOT CHIPS
increase about ninefold since 2013. This growth has
enabled developers to create ever more stunning visu-
als. Note that memory space and bandwidth have
grown much more slowly–only 2-3.
The brown line tracks the number of TV screen pix-
els that must be filled. Because consumers have been
able to increase their TV resolution from FHD
(1920  1080) to 4K UHD at 120 frames/s, the pixel count
has gone up almost as fast as shader power. Taking the
average of GPU compute and memory capability, the
useable graphics performance increase is in the 4–6
range depending on the game title.
By these metrics, the GPU is falling behind the per-
formance-per-pixel curve; but developers and players
alike want ever nicer more realistic pixels. A big part of
the solution for Xbox Series X involves architectural
enhancements that amplify the raw performance. The
actual increase depends on several factors, including
adoption of new techniques by game programmers
and the specific content.
OVER MANY YEARS, GAME ENGINES
HAVE TRIED MULTIPLE APPROACHES
TO LOADING ONLY ENOUGH TEXTURE
DETAIL TO SATISFY THE DEMANDS OF
THE FRAMES ABOUT TO BE RENDERED.
IN SERIES X THERE ARE TWO NEW
STRUCTURES IN THE GPU TO ASSIST
WITH TILE-BY-TILE MANAGEMENT OF A
MODEST TEXTURE WORKING SET
PLACED IN RAM JUST BEFORE OR JUST
AFTER NEEDED.
Variable Rate Shading
Traditionally, GPUs had to run a shader thread on
every rendered pixel, or fragment, to generate a color
value—and with antialiasing, multiple fragments
depth-buffered and shaded per screen pixel. We
observed that for most scenes that amount of unique
work is overkill, since there is not high spatial fre-
quency variation everywhere. A single fragment color
can serve for multiple subpixel samples or multiple
pixels without noticeable degradation. Proposed in
2018 by Microsoft, VRS is now widespread.
Figure 5 shows an analysis of a typical rendered
frame—in this case from the Sponza scene model.
The black areas are where shading can occur at quar-
ter resolution, i.e., one fragment for a two-by-two pixel
area. Turquoise areas are shaded once per two-by-one
pixel area; yellow per one-by-two. Red areas are
shaded at the normal 11 pixel rate.
26 IEEE Micro
This shows that, on average, we only need to shade
every other pixel; but we need to be judicious about
the distribution of fragments to not lose visual detail.
VRS addresses this using a set of bias controls. The
rate can be determined based on knowing which
objects have high detail, which primitives within
objects, or based on individual 8x8-pixel screen tiles.
For instance, since the game title performs multiple
rendering passes, it is possible to predict which areas
of the screen that might ordinarily have high detail will
be blurred in later passes.
A programmable combination of the different
types of rates is supported, which increases or
decreases the nominal rate, which is limited in fine-
ness to the global antialiasing level set for a given ren-
dering pass.
Object edge detail is preserved, and VRS can be
used alongside other resolution enhancing techni-
ques, including temporal antialiasing, super resolu-
tion, and even checkerboarding. The actual amount
of dedicated hardware for this feature is tiny but can
have a payoff of around 10%–30% in improved perfor-
mance, allowing higher frame rates and more math
per pixel.
Sampler Feedback Streaming
Over many years, game engines have tried multiple
approaches to loading only enough texture detail to
satisfy the demands of the frames about to be ren-
dered. In Series X there are two new structures in the
GPU to assist with tile-by-tile management of a mod-
est texture working set placed in RAM just before or
just after needed. There is a residency map per texture
that clamps the level of detail (LOD) of each tile, and a
request map that records the finest MIPMAP level
that was requested for each tile since it was last reset.
The tile size is flexible. The sequence of SFS opera-
tions is as follows:
1. The first step is to allocate virtual memory space
for the entire texture. Then, the title loads all
coarsest mipmap levels—in this case, from the
coarsest level up to level 2, which is 1=4 by 1=4 the
dimensions of the finest level, requiring just 6.7%
of all pixels to be resident in memory. Finer levels
are divided into tiles.
2. The next step is to render. Closer-up portions of
a texture require more detail. With SFS, the
shader executes a single sample macroinstruc-
tion that combines residency map lookup to
determine the current detail level with the fetch
of the actual texture data. Since the shader
March/April 2021
 HOT CHIPS

FIGURE 5. Typical shading rates, sponza.

sample instruction, as in the past, has already bilinear filtering on the left, fractional LOD values
calculated which LOD tiles should have been greater than zero leak over the boundary into the
fetched, those values are captured in the sepa- lower detail tiles, meaning that pixels sampled in those
rate request map. The closer tiles need more regions would need nonresident LOD 0 pixels to be
detail and the farther tiles may need less than blended in, causing visual errors. With a new biased fil-
what is resident. The latter are candidates for ter function shown on the right, the transition zones
eviction if the cache is full. are moved toward the coarser LOD map texel, so that
3. After rendering, the application reads back the the nonresidency problem is avoided.
request map, compares with its saved copy of the Overall, with a very small incremental hardware
residency map, and uses the XVA API to bring in cost, Sample Feedback Streaming gives the same or
the higher detail tiles it decides are needed from better level of visual detail with up to 60% savings in I/
flash. For each fine LOD tile, the corresponding O and memory footprint costs.
region of the coarser LODs is also loaded to pro-
vide the right detail everywhere in that region. Ray Tracing
4. Finally, the residency map is updated to reflect The Series X GPU supports DirectX ray tracing accel-
the current state of the tile cache and uploaded eration, allowing the most physically realistic render-
to the texture unit. The next time that texture is ing techniques to be used in real time. But in this
accessed the finer detail ready to use. console generation, developers still want to use tradi-
tional rendering techniques evolved over decades
There is a second mode of the request map to sup-
port texture-space shading, which saves GPU work by
deferring rendering passes that generate texture tiles
until it is known which are needed. In this mode,
instead of one detail value per tile for an entire tex-
ture, accesses are tracked using a single bit per tile for
every MIPMAP level.
Ideally these new tile maps, treated just like other
textures, stay on-die for low latency access, so they
are designed to be as small as possible—meaning tiles
should be as large as practicable. But we also do not
want to see seams between tiles with different levels
of available detail. In the example in Figure 6, the red FIGURE 6. Xbox series X texture LOD filtering.
tile has LOD 0 resident, orange is LOD 1, etc. With

March/April 2021 IEEE Micro 27

HOT CHIPS
without a performance penalty. They can apply ray
tracing selectively where materials and environments
demand. This means the GPU needs a good balance
of die resources dedicated to the two techniques.
We have added hardware embedded in the com-
pute units to perform intersections of rays with accel-
eration structures that represent the scene geometry
hierarchy. That task is a sizeable fraction of the overall
specialized ray tracing workload. The rest can be per-
formed with the baseline shader design and memory
hierarchy with good real-time quality. The hardware
performs up to 380 billion ray-box intersections or 95
billion ray-triangle intersections per second, or any
mix of the two. The overall speedup varies with scene
complexity and lighting characteristics, but for the
intersection task it can be up to ten times the perfor-
mance of a pure shader-based implementation.
Machine Learning Support
Game engines increasingly make use of machine learn-
ing inference for a variety of game-related tasks, from
character behavior and animation to super resolution
detail enhancement. Xbox Series X includes a small
increment for small-integer operations in the compute
units for inference,9 accelerating some tasks up to 10.
CONCLUSION
The Series X SoC, in conjunction with software innova-
tions provided by the software teams at Microsoft and
game development companies, help “get technology
out of the way” and deliver developers’ vision to
gamers worldwide.
REFERENCES
1. 2020. [Online]. Available: https://www.xbox.com/en-US/
consoles/xbox-series-x
2. 2021. [Online]. Available: https://en.wikipedia.org/wiki/
High-dynamic-range_video
28 IEEE Micro
3. 2017. [Online]. Available: HDMI 2.1 specs and features:
Everything you need to know TechHive.com
4. 2019. [Online]. Available: https://docs.microsoft.com/en-
us/gaming/acoustics/what-is-acoustics
5. 2020. [Online]. Available: https://opus-codec.org/
6. 2020. [Online]. Available: https://www.microsoft.com/
security/blog/2020/11/17/meet-the-microsoft-pluton-
processor-the-security-chip-designed-for-the-future-of-
windows-pcs/
7. 2020. [Online]. Available: https://news.xbox.com/en-us/
2020/07/14/a-closer-look-at-xbox-velocity-architecture
8. 2015. [Online]. Available: https://gamingbolt.com/xbox-
one-gpu-has-8-graphics-contexts-uses-multiple-gpu-
command-streams-to-reduce-cpu-gpu-latency
9. 2020. [Online]. Available: Inside Xbox Series X: the full
specs, Eurogamer.net
MARK GROSSMAN is a Principal Architect with Microsoft,
working on GPUs and display processing for multiple genera-
tions of consoles and headsets. His main interests are high-
performance graphics systems and hardware-accelerated
algorithms. He is a Founder of Silicon Graphics. Grossman
received a B.A. in information and computer science from the
University of California, Santa Cruz, CA, USA. Contact him at
mark.grossman@microsoft.com.
JEFFREY ANDREWS is a Distinguished Engineer with Micro-
soft, directing a silicon IP architecture team within Azure.
His team’s focus areas include: silicon security, machine
learning, storage, graphics, audio, display, image process-
ing, and embedded CPUs. He has worked on every Xbox
console, plus four game consoles, and four startups before
Microsoft. Andrews received a B.Sc. in computer architec-
ture from the University of Illinois, Urbana-Champaign, IL,
USA. Contact him at jeffrey.andrews@microsoft.com.
March/April 2021
THEME ARTICLE: HOT CHIPS

NVIDIA A100 Tensor Core GPU: Performance

and Innovation
Jack Choquette , Wishwesh Gandhi, Olivier Giroux, Nick Stam, and Ronny Krashinsky, NVIDIA,
Singapore, 609927

NVIDIA A100 Tensor Core GPU is NVIDIA’s latest flagship GPU. It has been designed
with many new innovative features to provide performance and capabilities for
HPC, AI, and data analytics workloads. Feature enhancements include a Third-
Generation Tensor Core, new asynchronous data movement and programming
model, enhanced L2 cache, HBM2 DRAM, and third-generation NVIDIA NVLink I/O.

NVIDIA A100 TENSOR CORE GPU faster than V100 on important HPC applications within
OVERVIEW molecular dynamics, physics, engineering ,and geo sci-

T
he diversity of compute-intensive applications
in modern cloud data centers has driven the
explosion of GPU-accelerated cloud computing.
Such applications include AI deep learning (DL) train-
ing and inference, data analytics, scientific computing,
genomics, edge video analytics and 5G services,
graphics rendering, and cloud gaming. The NVIDIA
Ampere architecture-based A100 GPU brings record-
setting scale and novel capabilities to these work-
loads. The A100 GPU is 54 billion transistors built on
TSMC’s 7-nm process. It has 108 streaming microproc-
essors (SMs) with 6912 CUDA cores, 40 MB of L2
cache, 600 GB/s of NVIDIA NVLink interconnect band-
width, and 1.6 TB/s of HBM2 memory bandwidth. It
also has new elastic GPU capabilities including scale-
out support with multi-instance GPU (MIG) virtuali-
zation, and scale-up support with a third-generation
50-Gb/s NVLink I/O interface connecting multiple
A100s directly or through NVIDIA’s NVSwitch. Inside
the A100 SM are new third-generation tensor cores
with support for fine-grain sparsity and new BFloat16
(BF16), TensorFloat-32 (TF32), and FP64 datatypes.
The SM also adds new asynchronous data movement
instructions and barriers which work together to effi-
ciently stream data in a programmer-friendly way.
A comparison shows that A100 provides dramati-
cally higher performance than currently available com-
mercial designs and NVIDIA’s previous generation
V100 GPU.1 A100 runs approximately 1.5
to over 2

0272-1732 ß 2021 IEEE
Digital Object Identifier 10.1109/MM.2021.3061394
Date of publication 23 February 2021; date of current version
26 March 2021.
ence areas (see Figure 1). For DL workloads, DGX
super-pods with A100 have set records for the MLPerf
benchmark,2 handily surpassing all other commercially
available systems, including Google’s TPUv3 and Hua-
wei’s Ascend systems (see Figure 2). The benchmark
also demonstrates A100’s breadth of support for AI
networks, the only system able to run all benchmarks,
and run them with high performance.
Many new and innovative features in A100 contrib-
ute to its high performance and capabilities. This arti-
cle will cover a few of those improvements: top to
bottom features to support strong scaling, elastic
GPU capabilities to support scale up and scale out,
and asynchronous programming features to enable
efficiency and programmer productivity.
STRONG SCALING: TOP TO
BOTTOM
A typical deep neural network consists of long chains
of interconnected layers (see Figure 3). While there is
a massive amount of compute in a network, the paral-
lelism is broken up into layers of smaller, sequentially
dependent chunks of work. Each layer takes an input
activation tensor and weight tensor, performs an oper-
ation similar to a matrix multiplication, and outputs an
activation tensor. To leverage the large amount of
compute available in a GPU, the output tensor is bro-
ken down into smaller tiles which are distributed
across the different SMs.
A100’s Tensor Core throughput is 2.5 times higher
on dense FP16 data than the previous generation V100
GPU. In weak scaling, both the network size and paral-
lelism must grow to match the increased throughput.
In strong scaling, both the network size and available

March/April 2021 Published by the IEEE Computer Society IEEE Micro 29

HOT CHIPS

FIGURE 1. NVIDIA A100 HPC performance HPC relative to NVIDIA V100, normalized to per chip.

FIGURE 2. MLPerf v0.7 performance for NVIDIA A100 and other commercially available DL systems, relative to NVIDIA V100 &
normalized to per chip.

FIGURE 3. DL network mapping to GPU.

30 IEEE Micro March/April 2021


FIGURE 4. NVIDIA A100 & V100 tensor core formats and performance.
parallelism remain fixed, and A100 runs the network
faster.
To achieve strong scaling, A100 needed to scale
performance at all levels: from the tensor cores,
through the cache hierarchy, through DRAM, and chip
interconnect.
SM Core
A100’s new tensor cores have increased the process-
ing of dense FP16 data by 2x per SM and 2.5
per GPU
over V100 (see Figure 4). The tensor cores added sup-
port for additional formats, including the TensorFloat-
32 operation that improves the processing of FP32
data by 10
per GPU. A100’s tensor cores also support
fine-grain sparsity, which doubles the throughput
when processing sparse data.
The 2
increase in FP16 math throughput per SM
for dense data requires 2
more data bandwidth, and
the effective 4
increase for sparse data requires 3

more data bandwidth.
In the SM core, multiple improvements in data
delivery provide this increase in data bandwidth (see
Figure 5). Inside the V100 and A100 SM, the network
layer tile is further broken down into four smaller tiles
which are each processed by a 32-thread warp. V100’s
tensor cores were designed to work at 8-thread granu-
larity, and required the tiles to be further broken down
to four smaller tiles per warp. Each of these tiles loads
the tensor data from shared memory (SMEM), which
in aggregate requires all data to be loaded four times.
In A100, the tensor cores were reorganized and
enhanced to work at 32-thread granularity. This
March/April 2021
HOT CHIPS
reduces the number of times data needs to be loaded
and reduces SMEM bandwidth by half.
Data transfer efficiency from the memory system
was also improved. In V100, data must first be loaded
into the register file and then stored into SMEM. In
A100, a new asynchronous combined load-global-
store-shared instruction was added which transfers
data directly into SMEM, bypassing the register file
and increasing efficiency.
Combined with the Tensor Core organization
changes, A100 reduces 6 L1þSMEM accesses (L1 read,
SMEM write, 4 SMEM reads) down to only two SMEM
reads. The asynchronous capability of the transfer of
memory also helps the SM to continuously stream data,
improving utilization throughout the memory system.
A100 L2 Cache
The A100 L2 cache is a shared resource for the SMs.
Additional bandwidth from the L2 cache is necessary
to achieve strong scaling, as the number of SMs is
increased in A100 and each SM processes data at a
faster rate. A100 delivers a 2.3x L2 bandwidth increase
over V100, supported by a new structure to efficiently
move this data.
The L2 cache is divided into two partitions to enable
higher bandwidth and lower latency memory access
(Figure 6). Each L2 partition localizes and caches data
for memory accesses from SMs directly connected to
the partition. Hardware cache-coherence maintains
the CUDA programming model across the full GPU,
and applications will automatically leverage the band-
width and latency benefits of A100’s new L2 cache.
IEEE Micro 31
HOT CHIPS
FIGURE 5. Data delivery for V100 and A100.
L2 Cache Residency Controls
A100 gives applications the capability to influence the
persistence of data in the L2 cache, allowing higher
bandwidth and lower latency accesses to the global
memory. The so-called persistent accesses control
the replacement policy to effectively set-aside a por-
tion of L2 cache. Normal or streaming accesses to
FIGURE 6. NVIDIA A100 die.
32 IEEE Micro
global memory can only utilize this portion of L2 when
it is unused by persistent accesses.
There are two primary mechanisms that allow per-
sistent data to be resident in the L2. In the first, an
address-based window is specified where all the read/
write accesses are persistently cached in the L2. Indi-
vidual accesses are not tagged in this scheme. Alter-
natively, controls can be specified at a finer-grained,
per-memory-operation basis.
A100 DRAM Bandwidth
The A100 GPU includes 40 GB of fast HBM2 DRAM
memory on its SXM4-style circuit board. The memory
is organized as five active HBM2 stacks with eight
memory dies per stack. With a 1215 MHz (DDR) data
rate the A100 HBM2 delivers 1555-GB/s memory band-
width, which is more than 1.7x higher than Tesla V100
memory bandwidth.
A100 Compute Data Compression
To boost DRAM bandwidth and L2 capacity, A100
implements a data-compression feature for sparsity
and other compressible data patterns. Compression
in L2 provides up to 4
improvement to DRAM read/
write bandwidth, up to 4
improvement in L2 read
bandwidth, and up to 2
improvement in L2 capacity.
Compression is initiated by marking buffers avail-
able for compressibility using new APIs in CUDA 11.
Data written to these buffers is examined inside the
L2 by the hardware. For specific data patterns that
match the hardware algorithms, the data are com-
pressed and written back to the L2. Any subsequent
March/April 2021

FIGURE 7. NVIDIA DGX-A100 system block diagram.
access will take advantage of the compressed data
bandwidth. Compression helps both the write and
read accesses to DRAM and increases the effective
DRAM bandwidth available.
A100 Gen 3 NVLink
The third generation of NVIDIA’s high-speed NVLink
interconnect is implemented in A100 and the new
NVSwitch. NVLink is a reliable, high bandwidth, low-
latency memory interconnect, and includes resiliency
features such as link-level error detection and packet
replay mechanisms to guarantee successful transmis-
sion of data. The new NVLink has a data rate of 50 Gb/s
per signal pair, nearly doubling the 25.78-Gb/s rate in
Tesla V100.3 Each link uses four differential signal pairs
(four lanes) in each direction compared to eight signal
pairs (eight lanes) in V100. A single link provides 25-GB/
s bandwidth in each direction similar to V100 GPUs, but
uses only half the signals compared to V100. The total
number of NVLink links is increased to 12 in A100, ver-
sus six in V100, yielding 600-GB/s total bandwidth for an
entire A100 versus 300 GB/s for Tesla V100. The twelve
NVLink links in each A100 allow a variety of configura-
tions with high-speed connections to other GPUs and
switches.
All writes in the third-generation NVLink now
require an acknowledgement from the destination.
This allows synchronization to be performed at the
requester, and error attribution to be returned to a
specific execution context. Writes are allowed to be
pipeline to the destination while the requestor is wait-
ing on a response. New features to improve the
March/April 2021
HOT CHIPS
efficiency of small payload writes and data-less
responses were also added.
ELASTIC GPU: MULTI-GPU
SCALE UP
The twelve NVLink links in each A100 allow a variety of
configurations with high-speed connections to other
GPUs and switches. To meet the growing computa-
tional demands of larger and more complex DNNs and
HPC simulations, the new NVIDIA DGX A100 system
(Figure 7) includes eight A100 GPUs connected by the
new NVLink-enabled NVSwitch.
Multiple DGX A100 systems can be connected via
a networking fabric like NVIDIA Mellanox InfiniBand
and Mellanox Ethernet to scale out data centers, cre-
ating powerful supercomputer-class systems. More
powerful NVIDIA DGX POD and NVIDIA DGX Super-
POD systems will include multiple DGX A100 systems
to provide much greater compute power with strong
scaling.
ELASTIC GPU: MULTI-INSTANCE
GPU (MIG)
While many data center workloads continue to scale,
both in size and complexity, some acceleration tasks
are not as demanding, such as early-stage develop-
ment or inference on simple models at low batch
sizes. Data center managers aim to keep resource uti-
lization high, so an ideal data center accelerator not
only needs to efficiently handle a large workload—it
also efficiently accelerates many smaller workloads.
IEEE Micro 33
HOT CHIPS
FIGURE 8. Traditional CUDA Cþþ program with exposed latency.
The new MIG feature can partition each A100 into
as many as seven GPU Instances for optimal utiliza-
tion, effectively expanding access to every user and
application. The A100 GPU’s new MIG capability can
divide a single GPU into multiple GPU partitions called
GPU Instances. Each instance’s SMs have separate
and isolated paths through the entire memory sys-
tem—the on-chip crossbar ports, L2 cache banks,
memory controllers, and DRAM address busses are all
assigned uniquely to an individual instance. This
ensures that an individual user’s workload can run
with predictable throughput and latency, with the
same L2 cache allocation and DRAM bandwidth even
if other tasks are thrashing their own caches or satu-
rating their DRAM interface. Using this capability, MIG
can partition available GPU compute resources to pro-
vide a defined quality of service (QoS) with fault isola-
tion for different clients (such as VMs, containers,
processes, and so on). It enables multiple GPU Instan-
ces to run in parallel on a single, physical A100 GPU.
MIG also keeps the CUDA programming model
unchanged to minimize programming effort.
MIG enables users to see and schedule jobs on vir-
tual GPU Instances as if they were physical GPUs.
MIG works with Linux operating systems and their
hypervisors.
PRODUCTIVITY: ASYNCHRONOUS
PROGRAMMING
The goal of CUDA is to compile intuitive Cþþ sources
into high-performance executable programs for the
GPU. In this pursuit, a compiler’s goals are in tension:
to maximize operation-level parallelism, but never to
alter the program’s semantics. NVIDIA’s work on joint
hardware & programming system codesign directly
addresses this tension.
34 IEEE Micro
Updates to CUDA for A100 expand the expressive-
ness of Cþþ for data and computation pipelining,
which we identified as a growing source of difficulty
for CUDA programmers. The goal of pipelining in soft-
ware is the same as in hardware: to keep execution
resources busy by overlapping the latency of different
phases of computation. This is difficult to express in
Cþþ due to its conservative requirements around
memory consistency.
The example program in Figure 8 shows that oper-
ation-level parallelism is prevented by synchronization
semantics. Software pipelining depends on having the
opportunity to execute independent work while syn-
chronization is resolving at the boundaries of stages.
The first programming model innovation borrows
from asynchronous programming: separate the arrival
and waiting steps, as in phasers.4 Early in develop-
ment, NVIDIA recognized this innovation would bene-
fit programmers beyond CUDA, so NVIDIA offered
both specifications and implementations to the com-
munity; it is now part of ISO Cþþ 205 and is available
in LLVM (libcxx) today.
The second innovation extends this foundation
with asynchronous data movement capabilities. This
example program in Figure 9 combines both asynchro-
nous barrier and data movement operations, resulting
in a remarkably precise expression of programming
intent. By leveraging the relaxed semantics of asyn-
chronous operations, there is a net reduction in the
difficulty of compiling the program and performance is
both higher and more predictable.
CONCLUSION
NVIDIA’s A100 GPU is the largest and most advanced
GPU developed by NVIDIA, and builds on the ground-
work laid by previous generations of NVIDIA GPUs. It
March/April 2021

FIGURE 9. Asynchronous data movement and computation.
is the result of the work of thousands of engineers
that worked together from transistors to standards
and everything in between such as system integration
and programming model design. A100 provides
unprecedented acceleration at every scale, and adds
powerful new features which deliver dramatically
faster performance for HPC, AI, and data analytics
workloads.
REFERENCES
1. J. Choquette, O. Giroux, and D. Foley, “Volta:
Performance and programmability,” IEEE Micro, vol. 38,
no. 2, pp. 42–52, Mar./Apr. 2018.
2. P. Mattson et al., “MLPerf training benchmark,” 2019,
arXiv:1910.01500.
3. A. Ishii et al., “NVSwitch and DGX-2: NVLink-Switching
chip and scale-up compute server,” Hot Chips, 2018.
4. J. Shirako, D. M. Peixotto, V. Sarkar, and W. N. Scherer,
“Phasers: A unified deadlock-free construct for
collective and point-to-point synchronization,” in Proc.
22nd Annu. Int. Conf. Supercomput., 2008, pp. 277–288.
5. International Standard ISO/IEC 14882:2020 – Program
ming Language Cþþ.
JACK CHOQUETTE is a Senior Distinguished Engineer with
NVIDIA, where he has led the architecture development of
NVIDIA’s GPGPU Streaming Microprocessors for multiple
generations. He has been leading CPU and system designs
for over 25 years. Choquette received an M.S. in computer
engineering from the University of Illinois Urbana-Champaign,
Champaign, IL, USA. Contact him at jchoquette@nvidia.com.
March/April 2021
HOT CHIPS
WISHWESH GANDHI is a Senior Director of architecture at
NVIDIA, Singapore. He has led the architecture development
of the GPU memory system for multiple generations. He
has been working with Integrated and Discrete GPU mem-
ory architecture for more than 20 years. Contact him at
wgandhi@nvidia.com.
OLIVIER GIROUX is a Distinguished Architect at NVIDIA, and
the ISO Cþþ Concurrency and Parallelism Chair. He has
worked on ten GPU and six SM architecture generations,
with a focus on clarifying the programming model of GPU
threads. Giroux received an M.S. in computer science from
McGill University, Montreal, QC, Canada. Contact him at
ogiroux@nvidia.com.
NICK STAM is a Senior Technical Marketing Director with
NVIDIA. His team provides tech support to press, and also
produces our GPU white papers. Before NVIDIA, he worked
at PC Magazine USA, and cofounded the ExtremeTech web-
site. Stam received an M.S. in computer science from SUNY
Binghamton, NY, USA. Contact him at nstam@nvidia.com.
RONNY KRASHINSKY is a Distinguished Engineer with NVI-
DIA where he has architected GPUs for 11 years. He began
his NVIDIA career in Research, and later joined the Streaming
Multiprocessor team. He now focuses on deep-learning com-
pute architecture. Krashinsky received a Ph.D. in electrical
engineering and computer science from Massachusetts Insti-
tute of Technology (MIT), Cambridge, MA, USA. Contact him
at rkrashinsky@nvidia.com.
IEEE Micro 35
 THEME ARTICLE: HOT CHIPS

Manticore: A 4096-Core RISC-V Chiplet

Architecture for Ultraefficient Floating-Point
Computing
Florian Zaruba and Fabian Schuiki € rich, 8092, Zu
, Integrated Systems Laboratory, ETH Zu € rich, Switzerland
Luca Benini , Department of Electrical, Electronic and Information Engineering, University of Bologna, 40126,
Bologna, Italy, and Integrated Systems Laboratory, ETH Zürich, Zürich, 8092, Switzerland

Data-parallel problems demand ever growing floating-point (FP) operations per

second under tight area- and energy-efficiency constraints. In this work, we present
Manticore, a general-purpose, ultraefficient chiplet-based architecture for data-
parallel FP workloads. We have manufactured a prototype of the chiplet’s
computational core in Globalfoundries 22FDX process and demonstrate more than 5x
improvement in energy efficiency on FP intensive workloads compared to CPUs and
GPUs. The compute capability at high energy and area efficiency is provided in “Snitch:
A tiny pseudo dual-issue processor for area and energy efficient execution of floating-
point intensive workloads,” IEEE Trans. Comput., containing eight small integer cores,
each controlling a large floating-point unit (FPU). The core supports two custom ISA
extensions: The SSRs extension elides explicit load and store instructions by encoding
them as register reads and writes (“Stream semantic registers: A lightweight RISC-V
ISA extension achieving full compute utilization in single-issue cores,” IEEE Trans.
Comput.). The floating-point repetition extension decouples the integer core from the
FPU allowing floating-point instructions to be issued independently. These two
extensions allow the single-issue core to minimize its instruction fetch bandwidth and
saturate the instruction bandwidth of the FPU, achieving FPU utilization above 90%,
with more than 40% of core area dedicated to the FPU.

D
omains such as data analytics, machine
learning, and scientific computing are depen-
dent on increasing compute resources.3
Increasing technology node densities result in sys-
tems that are mainly limited by thermal design power
and the most feasible way to increase the amount of
active compute units is to design more energy-effi-
cient architectures. While many emerging architec-
tures,4 especially in the machine learning domain,
tradeoff floating-point (FP) precision for higher
throughput and efficiency, algorithms such as stencils
and linear differential equations require higher preci-
sion arithmetic. Domain-specific accelerators are a
0272-1732 ß 2020 IEEE
Digital Object Identifier 10.1109/MM.2020.3045564
Date of publication 17 December 2020; date of current
version 26 March 2021.
prominent example for how to leverage specializa-
tion.5 Unfortunately, they are hard to adjust to algo-
rithmic changes and tied to a specific application
domain.
The trend in leading-edge general-purpose com-
puter architectures paints a similar picture on the
importance of increasing energy efficiency. Two prom-
inent examples of recent high-performance architec-
tures are Fujitsu’s A64FX6 and NVIDIA’s A100.7 Both
systems strive to control their 32 (A64FX) and 16
(A100) wide multilane SP (sp) data-path with as few
instructions as possible.
With the proposed Manticore system, we pursue a
similar goal. We achieve this goal by pairing a simple,
in-order, 32-bit RISC-V integer core with a large float-
ing-point unit (FPU). Two instruction set architecture
(ISA) extensions: SSRs and floating-point repetition
(FREP) make it possible for the single-issue integer

36 IEEE Micro Published by the IEEE Computer Society March/April 2021


FIGURE 1. Conceptual floorplan of the package. Arrangement
of the chiplets and HBM on the interposer. Each chiplet has
its own, private, 8 GB HBM. Chiplets interconnect via die-to-
die serial links8.
core to saturate the bandwidth of its FPU, achieving
utilization higher than 90% for compute-bound kernels.
CHIPLET ARCHITECTURE
The proposed Manticore architecture consists of four
222-mm2 (14.9 x 14.9 mm2 ) 22-nm chiplet dies on an inter-
poser. Using chiplets improves yield and reduces cost.
Each die has three short-range, multichannel, in-package
chip-to-chip links,8 one to each sibling. They are used for
interdie synchronization and chiplet-to-chiplet nonuni-
form memory access. Furthermore, each chiplet has
access to a private 8-GB high-bandwidth memory (HBM).
The conceptual floorplan is depicted in Figure 1.
FIGURE 2. Conceptual floorplan of an individual chiplet.
Arrangement of individual cluster quadrants, interconnects,
L2 memory, HBM2 controller, PCIe controller, and quad-core
Ariane RV64GC system.
March/April 2021
HOT CHIPS
FIGURE 3. Memory hierachy of the Manticore concept. Four
cluster form a quadrant and share an uplink into the next stage.
Four S1 quadrants form an S2 quadrant which share an uplink
to the next stage. Two S2 quadrant form an S3 quadrant. Four
S3 quadrants per chiplet share access to the HBM memory.
The chiplet (see Figure 2) contains four quadrants,
consisting of 32 clusters with eight cores each, which
results in 1024 cores for all four quadrants on a chiplet.
Furthermore, each chiplet contains four Ariane RV64GC
management cores9 an HBM (256 GB/s) controller, a
27 MB of L2 memory, and a 16x PCIe endpoint (31.5 GB/s)
for host communication, as shown in Figure 2.
The four Ariane management cores run a general-
purpose operating system such as Linux and manage
the Snitch clusters and program off-loading. The Man-
ticore chiplet has enough silicon area to support
27-MB on-chip shared L2 memory for critical data stor-
age such as neural network weights or stencil kernels.
Memory Hierarchy
Each quadranty (see Figure 3) is further subdivided into
multiple stages, in a tree-structure using an interconnect
tuned for burst-based direct memory transfer (DMA)
accesses. Four clusters share an instruction cache and
an uplink into the next stage. These four clusters have a
high aggregate bandwidth of 64 TB/s among each other
and can perform low-latency, high-bandwidth intraclus-
ter data transfers. As shown in Figure 3, clusters share
y
The term quadrant is somewhat generic and does not neces-
sarily imply four members (cores or lower stage quadrants),
as the number of members can be adjusted to match the
available bandwidth into the next stage, as for example, in
stage three of our system
IEEE Micro 37
HOT CHIPS
FIGURE 4. Simplified block diagram of a Snitch-based compute
cluster. The core complex (CC) contains the integer core and
the FPU as well as necessary hardware for the SSRs and FREP.
The cluster contains eight core corplices, which share an
instruction cache and a tightly coupled data memory. A DMA
engines is used for efficient, bulk, data movement.
the uplink into the next higher stage, the bandwidth to
the other S1 quadrants becomes smaller. Bandwidth is
subsequently thinned as four S1 quadrants share an
instruction cache and an uplink into the S2 quadrant
and two S2 quadrants share an uplink into the S3 quad-
rant. In the last stage of hierarchy 16 S3 quadrants,
distributed over four chiplets (nonuniform memory
access), share four HBMs with an aggregated peak
bandwidth of 1 TB/s. This bandwidth thinning scheme
allows us to have a very low diameter, low latency inter-
connect topology, which can sustainably saturate the
HBM bandwidth while being benign to floorplanning
and physical design. The interconnect also allows for a
very high cluster-to-cluster internal bandwidth,
through multiple stages, which by far exceeds the
bandwidth into the memory. With this model, we effi-
ciently support cluster-to-cluster traffic, while, at the
same time, fully loading the memory system.
Compute Cluster
The compute cluster consists of eight small, 22 kGE,
single-stage, 32-bit RISC-V processor cores1 (see
Figure 4). Each Snitch core contains a double-preci-
sion (DP) FPU, which can be used to compute one DP
fused multiply–add (FMA) operation or two SP FMA
per cycle. When running at 1 GHz, a cluster with eight
Snitch cores is able to compute 16 DP or 32 SP flop,
resulting in 4 TDPflop/s for the entire Manticore sys-
tem. All eight cores have elementwise, low latency,
access into 128-KiB tightly coupled and shared
scratchpad memory. Moreover, a DMA engine is in
charge of moving blocks of data into the scratchpad
memory over a 512-bit data bus. The cores are clocked
38 IEEE Micro
FIGURE 5. Effect of SSRs and frep on the hot loop of a dot prod-
uct kernel. (a) Left: baseline simplified RISC-V implementation,
with address calculation and pointer increment omitted for
brevity. Right: SSRs implementation with memory loads
encoded as reads from stream registers; additional stream con-
figuration instructions required ahead of the loop. (b) Left:
implementation with loop bookkeeping using baseline RISC-V
instructions. Right: implementation with an frep hardware loop,
with all bookkeeping to occur implicitly in the hardware.
at 1 GHz, thus delivering more than 4 TDPflop/s peak
compute per chiplet.
With this architecture, we achieve a very high com-
pute/control ratio: 44% of the system consisting of
compute units, another 44% spent on the L1 memory
and just 12% of the area are spent on the control parts.
PROGRAMMING
We leverage two custom RISC-V ISA extensions to
achieve extremely high fp utilization and efficiency:
Xssr and Xfrep.
Stream Semantic Registers (Xssr)
SSRs2 offer a means to elide explicit load/store instruc-
tions in a program. This is achieved by giving a subset
of the processor core’s registers stream semantics.
When enabled, a read from such an SSRs is translated
in hardware into a load from memory, and conversely, a
register write becomes a store to memory. Since an in-
order single-issue core can only execute a single
instruction every cycle, the presence of loads and
stores in a hot loop of the program diminishes FPU utili-
zation significantly. For example, consider a dot prod-
uct, which has to issue two loads from memory for
every FMA operation, as shown in Figure 5(a). In this
scenario, even if the loop is fully unrolled, we achieve at
most 33% FPU utilization. In theory, this allows the FPU
to be 100% utilized, and even a simple processor can
achieve >90% utilization in many relevant kernels with-
out resorting to complex and energy-inefficient wide
issue superscalar or very long instruction word (VLIW)
architectures.2 SSRs offer a way to elide memory
March/April 2021
 HOT CHIPS

FIGURE 6. Typical execution of a matrix-vector multiplication implementation leveraging the SSRs and frep extensions. The 16
instructions are fetched and decoded once by the integer pipeline of the processor core (b), and expanded to 204 executed
instructions in the fpu (c). (a) Reference C implementation with a square matrix A of fixed size 48. (b) Resulting assembly imple-
mentation as stored in the binary and fetched/decoded by the processor core. (c) Execution traces of the integer pipeline (left)
and the fp pipeline (right).

accesses and address computation in hot loops, which
in many cases leaves no integer instructions in the
loop body.
Floating-Point Repetition (Xfrep)
The frep1 extension implements a FPU-only hardware
loop. Consider a dot product utilizing SSRs for example,
as shown in Figure 5(b). Besides the essential FMA oper-
ation running on the FPU, the loop only consists of a trip
count increment (addi) and a back branch (bne). This
loop can be replaced by a FREP instruction, which loops
a range of subsequent FP instructions (one in this case)
for a configurable number of times. The RISC-V ISA
makes the integration of such an extension very
straightforward as most instructions either operate
entirely on integer or entirely on FP registers. Only a
handful, such as comparisons or moves between integer
and FP domains, exchange information from one
domain to the other. We leverage this separation and
insert a microloop sequence buffer of 16 instructions
between the Snitch core and the FPU. FREP instructions
configure this buffer to emit a range of buffered instruc-
tions multiple times into the FPU, which essentially
implements the hardware loop. Since this happens
entirely in the FPU subsystem outside of the Snitch
core, the core’s integer pipeline can run in parallel,
enabling a pseudo-dual-issue mode of operation that
would not be achievable with a traditional hardware
loop. This allows the core to perform nontrivial book-
keeping and address calculation while the FPU is run-
ning, without incurring a reduction of the FPU utilization.
Typical SSR/FREP Execution
As a concrete example, let us consider the matrix-vec-
tor multiplication operation shown in Figure 6(a). A
typical implementation leveraging Manticore’s SSRs
and FREP extensions is shown in Figure 6(b). The
address computation and memory accesses of A and
x are entirely performed by the SSRs ft0 and ft1. The
inner loop is implemented using an FREP instruction
and unrolled to compute four results in parallel in
order to avoid pipeline stalls due to FPU latency. The
outer loop is executed by the integer core. It stores
the results (fsd), implements loop bookkeeping (addi,
bltu), and initializes a (fmv.d).
As shown in Figure 6(c), the 16 instructions of the
assembly implementation are fetched and decoded
once by the integer pipeline of the processor core and
expand to 204 executed instructions in the FPU through
the use of FREP. This leaves 188 cycles for the integer
pipeline for other tasks, such as preparing the next loop
iteration or coordination of data movement. In case no
other work is required, the 16 instructions fetched over
204 cycles of execution amounts to roughly one instruc-
tion every 13 cycles, mitigating the von Neumann bottle-
neck by reducing instruction fetch bandwidth by more
than one order of magnitude. Since the FPU can execute
the loop iterations back-to-back and of the 204 instruc-
tions, 192 perform actual computation, this kernel can
achieve up to 94% FPU utilization.
Compilers can leverage these new instructions
through scalar evolution and loop analysis to detect
loops with the appropriate structure, and matching

March/April 2021 IEEE Micro 39

HOT CHIPS

FIGURE 7. Floorplan of the prototype silicon. The two Ariane cores as well as the Snitch cluster have been designed hierar-
chically. The core’s follow a star-shaped layout around the shared instruction cache.

against amenable address calculation, load and use

patterns as a peephole optimization. These facilities
are commonly provided as part of the existing com-
piler infrastructure in both GCC and LLVM.

PROTOTYPE
A 3 x 3-mm2 prototype containing the logic core of the
chiplet architecture was manufactured and character-
ized using the Globalfoundries 22FDX technology. The
prototype in Figure 7 contains three Snitch clusters
with eight cores (each configured with 8-KiB L1
instruction cache and 128-KiB L1 data memory orga-
nized in 32 banks), a dual-core Ariane (with 16-KiB L1
instruction cache and 32-KB data cache), 1.25-MiB L2
memory, and a 400-MHz, double data-rate, 2.56-GB/s,
digital-only chip-to-chip link.

SILICON PERFORMANCE FIGURE 8. Compute performance, energy efficiency, speed,

Efficiency
We measured the speed and power consumption of the
prototype silicon under representative workloads and
operating conditions. Figure 8 shows the DP per-
and power consumption for different operating voltages.
Measured on the prototype silicon across eight sample dies.
Cores performing matrix multiplications, at 90% FPU utiliza-
tion. Performance and efficiency doubles across range.

40 IEEE Micro March/April 2021


FIGURE 9. Performance roofline plot of DNN training work-
loads. We group convolutions and linear/pooling layers to
indicate performance in the compute- and memory-bound
regions, respectively. The Manticore architecture is very effi-
cient at tracking the performance and bandwidth roofline,
with a detachment down to 5% for low-intensity and 14% for
high-intensity optimized kernels.
formance and energy efficiency achieved when execut-
ing parallel workloads on the 24 cores of our prototype
22-nm silicon, for different operating voltages. The chips
offer a wide range of operating points and choices for
performance/efficiency tradeoffs, which we leverage
through dynamic voltage and frequency scaling based
on the current workload’s operational intensity. This
allows us to essentially adjust the roofline of the system
to match the current workload. In high-performance
mode running over 1 GHz at 0.9V VDD, our architecture
achieves a peak performance of 54 GDPflop/s across 24
cores and a compute density of up to 20 GDPflop/s mm2 ,
which translates to 9.2 TDPflop/s across a full 4096
cores. In max-efficiency mode running at 0.5 GHz at 0.6 V
VDD, our architecture achieves an industry-leading effi-
ciency of 188 GDPflop/sW, while still delivering a respect-
able 25 GDPflop/s across 24 cores, which translates to
4.3 TDPflop/s across a full 4096 cores.
Roofline
To assess the performance of the manufactured sili-
con, we analyzed workloads from training steps of a
set of deep neural networks (DNNs). Figure 9 shows
the roofline plot of our architecture across a full train-
ing step. We estimate full-system performance based
on cycle-accurate simulation of a smaller instantiation
of the hardware, combined with an architectural
model of the full system and measured performance
characteristics of the prototype silicon. The compute-
bound convolution layers in the workload reach >80%
of the system’s peak performance, and the proximity
to the point of inflection of the roofline indicates a bal-
anced utilization of the hardware capabilities. The
memory-bound linear and pooling layers reach >90%
March/April 2021
HOT CHIPS
FIGURE 10. Top: Estimated sp energy efficiency of a full DNN
training step, overall and specifically on the convolution
layers for the conceptual Manticore chiplet architecture. Bot-
tom: dp energy efficiency on linear algebra (assuming 90% of
peak performance); Manticore shown for maximum perfor-
mance and maximum efficiency operating points.
of the system’s peak bandwidth. Since DNN workloads
tend to be dominated by the convolution layers, the
overall performance, which considers all layers, is
almost identical to the convolution performance.
Overall we observe that the Manticore architecture is
very efficient at tracking the performance and band-
width roofline, with a detachment down to 5% for low
intensity and 14% for high-intensity optimized kernels.
The worst-case detachment of 34% from the roofline
is encountered in the intermediate region around the
point of inflection, where intuitively, the aggregate
bandwidth pressure on the L1 TCDM is highest due to
the DMA and the compute units both operating at
capacity and banking conflicts more frequently stall L1
memory accesses.
Applications
Figure 10 shows the SP energy efficiency achieved in a
DNN training step overall, and on the compute-bound
convolutions specifically, across a variety of networks,
and the industry-leading dp efficiency on linear alge-
bra. On sp DNN training workloads, Manticore’s actual
efficiency is competitive with the V100 GPU’s peak
efficiency and outperforms the Core i9-9900K CPU by
2 and the Neoverse N110 by 3. On dp workloads,
Manticore outperforms a V100 GPU’s peak efficiency
by 6, the N1 by 7, the Celerity RISC-V CPU by 9,
and the Core i9-9900K CPU by 15. Our architecture
IEEE Micro 41
HOT CHIPS
achieves this despite these chips having a substantial
technology advantage due to their 7-, 12-, and 14-nm
FinFET processes. Regarding the A100 GPU, our initial
estimates based on data published by Nvidia7 suggest
that it achieves a 25% improvement on SP and DP
over the V100 in terms of speed at similar power con-
sumption. This indicates that Manticore has just 25%
lower efficiency on SP than A100, but outperforms it
on DP by 5, despite the A100’s significant 7-nm Fin-
FET technology advantage. Manticore delivers signifi-
cantly higher peak FP performance than comparable
RISC-V architectures11 in 16 nm.
OVERALL WE OBSERVE THAT THE
MANTICORE ARCHITECTURE IS VERY
EFFICIENT AT TRACKING THE
PERFORMANCE AND BANDWIDTH
ROOFLINE, WITH A DETACHMENT
DOWN TO 5% FOR LOW INTENSITY
AND 14% FOR HIGH-INTENSITY
OPTIMIZED KERNELS.
ACKNOWLEDGMENTS
This work was supported by the European Union’s
H2020 program under Grant 826647 (European Pro-
cessor Initiative-EPI) and Grant 732631 (Open Trans-
precision Computing - “OPRECOMP”).
REFERENCES
1. F. Zaruba, F. Schuiki, T. Hoefler, and L. Benini, “Snitch: A
tiny pseudo dual-issue processor for area and energy
efficient execution of floating-point intensive
workloads,” IEEE Trans. Comput., to be published.
2. F. Schuiki, F. Zaruba, T. Hoefler, and L. Benini, “Stream
semantic registers: A lightweight RISC-V ISA extension
achieving full compute utilization in single-issue cores,”
IEEE Trans. Comput., to be published.
3. “AI and compute,” 2020. Accessed: Oct. 5, 2020. [Online].
Available: https://openai.com/blog/ai-and-compute/
4. N. P. Jouppi et al., “A domain-specific supercomputer for
training deep neural networks,” Commun. ACM, 2020.
42 IEEE Micro
5. A. Yang, “Deep learning training at scale spring crest
deep learning accelerator,” in Proc. Symp. High
Performance Chips, vol. 31, 2019.
6. T. Yoshida, “Fujitsu high performance CPU for the Post-
K Computer,” in Proc. Symp. High Performance Chips,
vol. 30, 2018.
7. Nvidia, “NVIDIA Ampere GA102 GPU Architecture - The
Ultimate Play,” 2020.
8. P. Vivet et al., “A 220GOPS 96-core processor with 6
chiplets 3D-stacked on an active interposer offering
0.6ns/mm latency, 3TBit/s/mm2 inter-chiplet
interconnects and 156mW/mm2@82% Peak-Efficiency
DC-DC Converters,” in Proc. IEEE Int. Conf. Solid-State
Circuits, 2020.
9. F. Zaruba and L. Benini, “The cost of application-class
processing: Energy and performance analysis of a
Linux-ready 1.7-GHz 64-bit RISC-V core in 22-nm FDSOI
Technology,” IEEE Trans. Very Large Scale Integr. (VLSI)
Syst., vol. 27, no. 11, pp. 2629–2640, Nov. 2019.
10. R. Christy et al., “A 3GHz Arm Neoverse N1 CPU in 7nm
FinFet for infrastructure applications,” in Proc. IEEE Int.
Conf. Solid-State Circuits, 2020.
11. S. Davidson et al., “The Celerity open-source 511-core
RISC-V tiered accelerator fabric: Fast architectures and
design methodologies for fast chips,” IEEE Micro, vol.
38, no. 2, pp. 30–41, Mar./Apr. 2018.
FLORIAN ZARUBA received a B.Sc. from TU Wien, Vienna, Aus-
tria, in 2014 and an M.Sc. in 2017 from the Swiss Federal Institute
€ rich, Zu
of Technology Zu € rich, Switzerland, where he is currently
working toward a Ph.D. with the Digital Circuits and Systems
group of Luca Benini. Contact him at zarubaf@iis.ee.ethz.ch.
FABIAN SCHUIKI received a B.Sc. and an M.Sc. in electrical
engineering in 2014 and 2016, respectively, from ETH
€ rich, Zu
Zu € rich, Switzerland, where he is currently working
toward a Ph.D. with the Digital Circuits and Systems group of
Luca Benini. Contact him at fschuiki@iis.ee.ethz.ch.
LUCA BENINI holds the Chair of Digital Circuits and Systems,
ETHZ and is Full Professor with the Universita di Bologna. He is
a Fellow of the ACM and a member of the Academia Europaea.
Contact him at lbenini@iis.ee.ethz.ch.
March/April 2021
THEME ARTICLE: HOT CHIPS

Pensando Distributed Services Architecture

Michael Galles and Francis Matus , Pensando Systems, Inc., Milpitas, CA, 95035, USA

The Pensando distributed services architecture is designed to provide

programmable security, network, storage, and visibility services to a data center
landscape growing in scale and complexity. This domain-specific architecture
combines P4 packet processing pipelines with embedded ARM CPUs and hardware
offload engines in a tightly coupled, cache coherent interconnect. Millions of flows
are tracked by hardware and distributed to a combination of P4 programs, ARM
processes, custom hardware engines, and PCIe based hosts and devices. There are
currently 16- and 7-nm ASIC implementations of this distributed services
architecture, future generations are in design.

D
ata center growth in scale, bandwidth, appli-
cation diversity, and security requirements
has stretched traditional networking and
storage IO solutions beyond their design targets.2
Server connectivity speeds have moved to 100 Gb/s
and beyond while the number of interconnected serv-
ers in a data center surges. Virtualization, storage
disaggregation, and service mesh architectures are
driving the amount of east–west (intradata center)
traffic up along with the need for more sophisticated
network segmentation, overlays, and telemetry.
Security requirements, including encryption for data-
in-flight and data-at-rest, and stateful firewalls with
connection tracking further increases the complexity
and computational load on data center networking
services.
To address these challenges, the Pensando
distributed services architecture uses P41 domain-
specific processing coupled with general purpose
CPUs to deliver networking, security, and storage
services in a scalable and flexible solution. Distrib-
uted services cards (DSCs) are deployed at the
edge of compute and storage resources in the data
center and managed by a central Policy services
manager (PSM). This article will focus on the archi-
tecture and performance of the first two genera-
tions of Pensando ASICs, implemented in 16- and
7-nm processes.
0272-1732 ß 2021 IEEE
Digital Object Identifier 10.1109/MM.2021.3058560
Date of publication 10 February 2021; date of current version
26 March 2021.
ARCHITECTURE
The distributed services ASIC connects directly to the
data center Ethernet network near the compute edge.
It can operate as a standalone, bump-in-the-wire
device or as an endpoint controller, connecting servers
or endpoint devices to the data center network via
PCIe ports. Inside the ASIC, network traffic flows
through multiple P4 pipelines interconnected via a cen-
tral packet buffer. P4 programs process packets in-line
and may divert packets or flows to on-chip offload
engines and CPUs as well as to PCIe attached hosts or
target devices. In the ASIC block diagram (Figure 1), the
blue blocks (lower half) primarily handle packet-based
traffic and services, while the green blocks (upper half)
are primarily memory transaction based, providing off-
loads and data stream processing to memory buffers.
THIS ARTICLE WILL FOCUS ON THE
ARCHITECTURE AND PERFORMANCE
OF THE FIRST TWO GENERATIONS OF
PENSANDO ASICS, IMPLEMENTED IN
16- AND 7-NM PROCESSES.
Life of a Packet
Packets received through the network enter on one of
the Ethernet MAC interfaces and are placed in the
packet buffer according to L2 or L3 class of service
(COS). The packet buffer provides pause absorption
buffering and COS aware arbitration for the P4 ingress
pipeline. Consistent with the P4 model, packets pass
through the P4 ingress pipeline for flow classification,
firewall, tunnel endpoint processing, and other ingress
services; return to the packet buffer in case of

March/April 2021 Published by the IEEE Computer Society IEEE Micro 43

HOT CHIPS
FIGURE 1. Asic block diagram.
multicast or span replication; then enter the P4
egress pipeline to apply, for example, network
address translation, policers, and telemetry. Having
completed network processing tasks, the packet
can be sent back out an Ethernet MAC port or it
can be passed up toward the ARM subsystem or
local host.
If the packet is sent to the embedded ARM or local
host, it must be processed according to the needs of the
kernel or user level driver running on that CPU. The
packet passes through a third P4 pipeline, where it runs
another program in accordance with the protocol type
of the packet, i.e., Ethernet, RDMA, NVMe, or a new pro-
tocol. A significant advantage of running protocol-spe-
cific P4 programs associated with a targeted kernel or
user driver is the ability to customize descriptor formats,
completion events, interrupt schemes, and data formats
to the needs of the targeted driver.
In summary, network packets enter the ASIC, pass
through P4 ingress, and P4 egress pipelines to apply
security and network services, are associated with a
target interface and device type, and pass through P4
driver interface programs. The final packet or data
buffer are delivered to the attached host, embedded
ARM, or integrated offload engine. Packets sent by
the host enter through the PCIe MAC to be processed
by the P4 driver interface programs, then by the P4
ingress and egress pipelines to apply security and net-
work services before being sent out through the Ether-
net MAC. The following section takes a closer look at
the P4 pipeline internals.
P4 Pipelines
Pensando ASICs include multiple P43 pipelines (Figure 2)
as the centerpiece of a domain-specific processing
architecture. Each P4 pipeline starts with a programma-
ble parser. At its core, the programmable parser
44 IEEE Micro
FIGURE 2. Single P4 pipeline with 8 stages.
includes multiple engines designed to extract header
fields and populate a packet header vector (PHV) at
rates of 100 million packets per second or more. Follow-
ing the parser are six to eight match action stages, each
stage able to load multiple tables per packet and per-
form complex actions which modify the PHV and update
memory data structures. A single P4 pipeline includes
local TCAM and SRAM resources where high bandwidth
tables can be stored. Large scale tables as well as
tables shared between multiple pipelines are stored in
attached DRAM or host memory. The table base
address steers table requests to SRAM or to DRAM,
with per-pipeline caches reducing DRAM traffic for
commonly accessed table entries. The P4 pipeline
concludes with a deparser to reassemble the packet
from the updated PHV fields. Alternatively, the pipe-
line can conclude with a DMA engine to directly
copy packet header, payload, and meta data struc-
tures to and from local or host memory for CPU or
offload engine processing.
P4 Stages
Each P4 stage (Figure 3) within a pipeline begins
with a table engine (TE). The TE extracts bits and
bytes from the PHV in any combination to build a
table key. Table keys constructed from multiple
header fields can be up to 512 bits in width, or keys
can be chained together to create extra wide keys
up to 2048 bits in width. The resulting key may be
hashed or used as a direct index, then matched
against any data structure in TCAM, SRAM, DRAM,
or attached host memory.
The table data and lookup result (match or no
match) are then forwarded to a match processing unit
March/April 2021

FIGURE 3. Single P4 stage with 4 MPUs.
(MPU), along with the original key and meta data from
the PHV. Architecturally, an important role of the TE is
to provide a latency tolerance mechanism to protect
the MPUs from stalls. The TE logic processes multiple
PHVs in advance of the MPUs, issuing high latency
table reads and moving to the next PHV before earlier
table reads complete. As table responses return to the
TE, lookup results and action entry vectors are passed
to the MPUs for immediate execution, the MPUs never
stall waiting for table results. This latency tolerance
feature of the TE allows tables to be stored in high
latency DRAM memory, enabling large scale tables to
be processed by the MPUs at high packet rates.
Multiple MPUs in one stage can work on the same
PHV at the same time, provided each MPU is access-
ing a different table result. MPUs have a dedicated
write path to the stage data buffer where PHVs are
kept, and writes are merged at the bit level to support
updating arbitrary header formats and alignments.
After the final write completes, the PHV graduates to
the next P4 stage.
MPUs
The Pensando ASICs include four MPUs per P4 stage,
although this number is an implementation detail
which can be scaled without changing pipeline pro-
gramming. As the TE completes a table fetch and
match operation, results are delivered to the next
available MPU. Subsequent table fetches to the same
entry which have serial update dependencies are
locked by the TE to support single flow processing and
data ordering. Ordered table lookup results are deliv-
ered to an MPU along with an entry program counter
(PC), calculated from the current table configuration
and a programmable offset which may be stored in a
table entry.
Starting from the entry PC, the MPU (Figure 4)
executes a run-to-completion program stored in
DRAM. Instructions are fetched and cached in the
stage, which experience a high hit rate due to
March/April 2021
HOT CHIPS
functional code locality as P4 divides the pipeline
into discrete functional stages.
The MPU implements a domain-specific instruction
set architecture with an emphasis on bit field manipula-
tion and fast header updates. In particular, the MPU ISA
focuses on rapid field extraction from multiple sources
(tables, registers, or PHV) and forwards those fields
directly to an ALU or straight to the PHV. MPU instruc-
tion types include register and field comparisons,
branches, Boolean, and arithmetic ALU operations, and
memory load and store operations. In addition to these
familiar CPU instructions, the MPU includes PHV
write operations, packet sequence number comparison
and modification, queue state reduction, leading 0 and
1 detection, and other special protocol acceleration
instructions. A general purpose, 64-bit-wide register file
can be used to hold intermediate computational values,
plus a domain-specific, 512-bit-wide table entry vector
and a 512-bit-wide header field vector provide operands
directly to ALU instructions.
seq c1, p.tcp_valid, 1
phvwr.c1 p.daddr, d.lb_addr
tbladd.c1 d.byte_cnt, k.pkt_sz
The above MPU code snippet is a portion of
a simple TCP flow based stateful load balancer.
The seq (set if equal) instruction sets a conditional
flag c1 if the parser has marked the packet as TCP.
The second instruction will overwrite the PHV desti-
nation IP address with the load balancer address, a
field which was returned by stage’s TE table read.
This overwrite is conditional on the c1 flag being set
(TCP traffic only), as is the final instruction which
adds the packet size to the byte count to track
the load balancing destination traffic via the tbladd
(table add) instruction.
In addition to efficient packet header manipula-
tion, MPUs are used to interpret and create memory
descriptors which drive DMA operations based on
packet content, flow state, or processor work queue
entries. For example, received TCP packets are parsed,
classified to a connection or flow, sequence checked,
and DMA’ed directly to associated data stream buf-
fers. These data stream buffers can then be passed to
crypto offload processing for P4 TLS support, higher
layer applications running on general purpose host/
ARM CPUs, or P4 proxy programs to attach the data
stream to a new connection. Processing memory
descriptors in the P4 pipeline has the additional
advantage of adapting to customized descriptor for-
mats, allowing P4 programs to read and write packets
directly from/to Linux mbufs with associated meta-
data in the format required by DPDK4 or other drivers.
IEEE Micro 45
HOT CHIPS
FIGURE 4. Single MPU block diagram.
64-bit MPU Instructions
Internally, the MPU instruction fetch, branch, and
load/store blocks are similar to existing CPU designs.
The MPU differs from most CPU designs in which its
instruction set architecture is based on 8-byte-wide
instructions, allowing it to process networking
domain jobs with higher efficiency due to richer
encoding available from wider instructions. For exam-
ple, the MPU ALU operand load aligner is based on bit
fields instead of byte fields, allowing any header field
pair of arbitrary length and offset to be loaded with a
single instruction, whereas a traditional CPU would
need to mask and shift header fields for alignment.
MPU ALU instructions also accept header fields
directly as ALU inputs, allowing ADD, XOR, COM-
PARE, and similar instructions to specify operands
directly from header fields, whereas a traditional CPU
must load general purpose registers before operating
with its ALU. For updating the PHV, the MPU has a
PHV Write unit to process custom instructions capa-
ble of updating or appending to the PHV. Again, one
or more header fields can be transferred directly
from table data to the PHV using a single instruction,
with no alignment restrictions on source or destina-
tion fields.
Hardware Queue Management
Packets which enter the P4 pipeline form the wire or
which originate from internal events are placed
in hardware queues. Hardware queues are used to
manage interfaces, flows, connections, and any other
objects which require ordered tracking and sched-
uling. Software can configure up to 16 million hard-
ware queues (Figure 5). Each queue stores its current
state in a DRAM-based qstate record, which includes
a count of enqueued objects, pointers to arrays or
linked lists of enqueued objects, connection state
46 IEEE Micro
FIGURE 5. Sixteen million queues.
and peer information, and a process ID used for
memory protection.
Events which enqueue new objects are signaled
with a doorbell mechanism, allowing one or more
objects to be added to any queue with a single door-
bell action. Events which require processing in the
future, such as TCP timers or activity checks, set
hardware time markers for future scheduling. The
queue scheduler organizes all 16 million queues into a
DRAM array, tracking scheduling groups, interface
groups, and COS groups. Priority scheduling, min/max
data rates, and deficit weighted round robin schedul-
ing are available across scheduling groups. When a
queue is scheduled, a PHV token with the scheduled
queue ID and COS is inserted into the assigned
P4 pipeline for processing. The TE then accesses
the qstate array, launching programs depending on
queue type and state, which in turn fetch queued
objects and descriptors as entries of tables in the
P4 program.
Central Packet Buffer
The multiple independent P4 pipelines are intercon-
nected via a center packet buffer. The packet buffer
design is a shared memory switch, with virtual out-
put queueing for multiple classes of service and
enhanced transmission selection5 scheduling at the
outputs. Multicast replication, SPAN (switch port
analyzer) queues, and network pause buffering are
provided by the central packet buffer. In addition, an
ingress packet burst overflow feature allows short
term bursts to be written to overflow regions in
DRAM memory. The central packet buffer operates
in the packet domain, whereas the system-on-chip
(SOC) operates in the memory transaction domain.
The P4 DMA engines bridge these domains, convert-
ing packets to memory transactions.
March/April 2021

SOC, NOC, ARM Processors
A full SOC subsystem with multicore ARM A-72 CPUs
is connected to a hardware coherent, shared cache,
and memory hierarchy. Processors, memories, P4
pipelines, offload engines, and external devices com-
municate through a high-speed network-on-chip
(NOC). The P4 pipelines are tightly coupled to the
ARM processors, supporting direct delivery of packets,
headers, and/or metadata to the ARM L2 caches. This
tight coupling allows packet and flow processing to be
passed back and forth between P4 pipelines, offload
engines, and ARM processing services with low over-
head. Tasks which are well suited to the P4 domain-
specific processing are performed in the P4 pipelines
and offload engines, including packet parsing, encap-
sulation, cryptography, compression, classification,
stateful firewalls, telemetry, and data stream segmen-
tation and reassembly. Higher level tasks better suited
to general purpose processing are performed by the
ARM cores or host CPUs, such as exception handling,
application level processing, and storage volume man-
agement. P4 programming determines which portions
of data buffers, header buffers, and completion infor-
mation to deliver to the DRAM, SOC last level cache,
or ARM L2 on a per-application, per-packet basis. For
chained offload operations, which may include encryp-
tion, compression, and data integrity operations mixed
in with P4 control functions, a 4-MB chaining buffer is
attached directly to the NOC to support high band-
width, multihop offload chaining.
Root of Trust and Cryptography
Pensando ASICs provide a Root of Trust based on a
physical unclonable function and an ARM TrustZone
core running from protected memory. Boot loaders
and operating systems are authenticated before
execution, providing a secure root of trust that is
not dependent on or vulnerable to attached hosts.
Within the data center, central controller software
securely communicates with trusted agents running
on the ARMs at each DSC to enforce data center
wide policies.
Secure protocols including IPsec and TLS proxy can
be initialized by ARM processes, then P4 programming
controls packet-by-packet flow processing using keys
stored in secure on-chip memory. Cryptography opera-
tions are performed by internal hardware offload
engines under the direction of P4-created descriptor
lists. Cryptography hardware includes a hardware
entropy source driving a NIST SP 800-90 compliant
deterministic random bit generator and public key
exchange hardware to assist with connection setup.
March/April 2021
HOT CHIPS
Cryptography offloads for AES-GCM, AES-XTS, AES-
CBC, AES-CCM, ChaChaPoly, HMAC, SHA3-512, and
SHA2-256 are available to provide high performance
security processing.
Storage Offloads
Storage virtualization and management is provided
with a combination of P4 programs and ARM code.
Individual packet and transaction processing is han-
dled by P4, while ARM programs handle connection
establishment and volume management. P4 device
driver programs present standard NVMe device inter-
faces to the host and DMA descriptors and data to
the ASIC for offload processing or transport. Storage
offload engines are available to P4 and ARM process-
ors, and can be mapped to external hosts as a PCIe-
based accelerator endpoint. Dedicated storage off-
load engines include compression and decompression
at 100 Gb/s using LZRW1A, GZIP, or Deflate algo-
rithms. Reed–Solomon Erasure coding engines with up
to 12 data and 4 parity blocks operate at 100 Gb/s.
Data integrity engines operate at 200 Gb/s for multiple
algorithms, including CRC64, M-Adler32, CRC32C, and
Alder-32. Deduplication support is based on SHA2 and
SHA3 engines.
PCIe Ports and Virtualization
PCIe virtualization hardware allows software running
on the SOC to configure the number and type of devi-
ces presented to the host, including configuration
space, BAR (base address register) ranges, and virtual
function counts. This allows multiple network, storage,
and host visible offload devices to be presented and
attached to different processes or guest OS instances.
PCIe lanes can be bifurcated into multiple ports, each
port can operate as a virtual endpoint switch or a root
complex. If multiple virtual switch ports are configured,
multiple hosts can attach over separate PCIe lanes
and share networking services as if they were local to
each host. Internal ASIC COS buffering resources
enforce fairness between hosts, preventing noisy
neighbors from impacting adjacent hosts. Alterna-
tively, if a PCIe port is configured as a root complex,
storage devices such as NVMe drives are controlled by
the SOC and virtualized to local and remote hosts.
Other PCIe devices including GPUs and machine learn-
ing accelerators can also be controlled by the ASIC.
P4-16 COMPILER
Pensando has developed a compiler which accepts
P4-16 imperative code and generates parser sequence
commands, TE key generation, and table match
IEEE Micro 47
HOT CHIPS
TABLE 1. ASIC implementations.
parameters, as well as MPU action instruction sequen-
ces. In order to quantitatively measure the compiler
capability of generating optimal code for the P4 pipe-
lines, we compared the performance of two implemen-
tations of the same network functionality: termination
of a VXLAN overlay with flow-based packet routing. A
production-grade implementation consists of hand
written MPU assembly code. The compiler implemen-
tation is coded in P4. Several performance tests were
run with various traffic patterns presented to the ASIC
using both implementations, and the number of for-
warded packets per second was measured. Overall,
the P4 implementation achieves 85% of the perfor-
mance of the hand-written assembly implementation.
This demonstrates that the domain-specific compiler
is an effective and efficient tool to turn P4 programs
into machine code for Pensando’s packet forwarding
pipelines.
As generations of the Pensando domain-specific
architecture progress, hardware features are added to
improve capabilities, performance, and efficiency.
These improvements can be in the MPU instruction
set, conditions causing pipeline hazards or stalls, pro-
grammable parser control, hash table options, or other
areas. The P4 compiler abstracts these implementa-
tion-related changes away, allowing imperative, user-
created P4 code to take advantage of new ASIC gen-
erations without source code changes.
48 IEEE Micro
TABLE 2. Performance results.
ASIC IMPLEMENTATIONS
Pensando has completed the design of two genera-
tions of ASIC implementations and is currently archi-
tecting a third generation. All ASICs are software
compatible and features are forward compatible.
Table 1 summarizes implementation details of the first
ASIC generation, codenamed Capri, and the second
ASIC generation, codenamed Elba. Architectural fidel-
ity is preserved across generations, so many improve-
ments are focused on increased scale and
performance. In particular, the move from HBM mem-
ory in Capri to DDR memory in Elba was done to signif-
icantly increase the number and size of P4 tables
supported, while the reduced memory bandwidth was
compensated for with larger caches and more latency
tolerance built into the P4 pipelines.
PERFORMANCE
The performance results in Table 2 apply to a “bump in
the wire” full service forwarding application, which
includes a stateful firewall per connection, telemetry
collection on every flow, and connection establish-
ment/teardown managed by ARM CPUs while datapath
operations are handled in the P4 pipeline. Latency and
jitter were measured on the wire by lab testing devices.
Comparing performance with available solutions
is difficult as few publish results with identical service
features, but solutions offering a subset of these
features show a fraction of the delivered MPPS6
as compared to the DSC. Connection per second
establishment and tracking performance of the DCS is
one order of magnitude higher than available solutions7
and similar to large, multiport service appliances.8
CONCLUSION/
ACKNOWLEDGMENTS
Modern data centers are facing rapidly evolving needs in
security, networking, telemetry, storage, and scale which
are not addressed by current equipment. Pensando has
March/April 2021

developed a distributed services architecture to address
these data center needs while providing user program-
mability based on a domain-specific, open source,
permissively licensed P4 language. The first two genera-
tions of this architecture are realized in ASICs fabricated
in 16- and 7-nm processes.
MODERN DATA CENTERS ARE FACING
RAPIDLY EVOLVING NEEDS IN
SECURITY, NETWORKING, TELEMETRY,
STORAGE, AND SCALE WHICH ARE NOT
ADDRESSED BY CURRENT
EQUIPMENT. PENSANDO HAS
DEVELOPED A DISTRIBUTED SERVICES
ARCHITECTURE TO ADDRESS THESE
DATA CENTER NEEDS WHILE
PROVIDING USER PROGRAMMABILITY
BASED ON A DOMAIN-SPECIFIC, OPEN
SOURCE, PERMISSIVELY LICENSED P4
LANGUAGE.
The authors would like to recognize the talented
and dedicated team of engineers who created the
Pensando Distributed Architecture, hardware and
ASIC implementations, compiler design, software
March/April 2021
HOT CHIPS
infrastructure, testing and support which make these
products possible. We look forward to delivering
future generations of this architecture to meet the
evolving data center scale, security, and innovation
challenges.
REFERENCES
1. “P4: Programming protocol-independent packet
processors,” Jul. 2014. Accessed: Apr. 7, 2015.
2. “Azure accelerated networking: Smartnics in the public
cloud,” NSDI, 2018.
3. “P4-16 specification,” P4 Official Website, p4.org.
4. Data Plane Development Kit. [Online]. Available: dpdk.org
5. IEEE 802.1Qaz – Enhanced Transmission Selection.
6. L. Csikor et al., “Transition to SDN is HARMLESS: Hybrid
architecture for migrating legacy ethernet switches to
SDN,” IEEE/ACM Trans. Netw., vol. 28, no. 1, pp. 275–288,
Feb. 2020.
7. Cisco Firepower 2100 Series Data Sheet, Jan. 2021.
[Online]. Available: https://www.cisco.com/c/en/us/
products/collateral/security/firepower-2100-series/
datasheet-c78-742473.html
8. F5 Big-IP iSeries Appliance datasheets. [Online].
Available: https://www.f5.com/pdf/products/big-ip-
platforms-datasheet.pdf
IEEE Micro 49
 THEME ARTICLE: HOT CHIPS

Compute Substrate for Software 2.0

Jasmina Vasiljevic , Ljubisa Bajic, Davor Capalija, Stanislav Sokorac, Dragoljub Ignjatovic , Lejla Bajic,
Milos Trajkovic, Ivan Hamer, Ivan Matosevic, Aleksandar Cejkov, Utku Aydonat , Tony Zhou, Syed Zohaib Gilani,
Armond Paiva, Joseph Chu, Djordje Maksimovic, Stephen Alexander Chin, Zahi Moudallal, Akhmed Rakhmati,
Sean Nijjar, Almeet Bhullar, Boris Drazic, Charles Lee, James Sun, Kei-Ming Kwong, James Connolly,
Miles Dooley, Hassan Farooq, Joy Yu Ting Chen, Matthew Walker, Keivan Dabiri, Kyle Mabee, Rakesh Shaji Lal,
Namal Rajatheva, Renjith Retnamma, Shripad Karodi, Daniel Rosen, Emilio Munoz, Andrew Lewycky,
Aleksandar Knezevic, Raymond Kim, Allan Rui, Alexander Drouillard, and David Thompson, Tenstorrent Inc.,
Toronto, ON, M3C 2G9, Canada

The rapidly growing compute demands of AI necessitate the creation of new

computing architectures and approaches. Tenstorrent designed its architecture
(embodied in Grayskull and Wormhole devices) to tackle this challenge via two
fundamental and synergistic approaches. The first is via compute-on-packets fabric
that is built from ground up for massive scaleout. The second is the ability to
execute dynamic computation, built into the compiler, runtime software and
hardware architecture. By combining these approaches, TensTorrent will enable
continued scaling of AI workloads.

C
ompute demand of AI is skyrocketing at a rate
that far outpaces the compute density
improvements that can be gained by Moore’s
Law alone3,4 and approaches based on monolithic
shared memory models. We have chosen to attack
this challenge via two approaches, dynamic computa-
tion and massive scaleout. We design dynamic com-
putation to enable a wide range of techniques that
intelligently “forgo unnecessary computation” or
“compute only what is relevant to input,” akin to what
our brains do.
Large clusters are already the norm for training of
AI models, while inference for some large models also
requires multi-device execution. The shared memory
paradigm cannot enable the required scale, which
necessitates a paradigm shift to a multicore private-
memory model, a foundation in our scaleout architec-
ture. On top of this, we build a push-based data move-
ment in which data transfers are explicitly planned
and controlled.
These two approaches can be synergistically com-
bined to take the current steep slope of increasing AI
This work is licensed under a Creative Commons Attribution
4.0 License. For more information, see https://creativecom-
mons.org/licenses/by/4.0/
Digital Object Identifier 10.1109/MM.2021.3061912
Date of publication 9 March 2021; date of current version 26
March 2021.
compute and storage requirements and reduce it down
to something much more compatible with Moore’s Law.
In the “Hardware” section, we present the details
of our hardware, with primary focus on the Grayskull
device. The “Software” section presents our software
stack. The “Full Stack Performance Optimizations”
section deep dives into several full stack performance
optimizations enabled by our hardware and software.
The “Dynamic Execution” section summarizes various
dynamic execution approaches enabled by our archi-
tecture. Performance results are presented in the
“Results” section. Finally, the “Conclusion” section
concludes this article.
HARDWARE
Devices
Over the last four years, Tenstorrent designed three
chips, shown in Figure 2 and summorized in Table 1.
Jawbridge is a small 14-nm test-chip, containing six
first-generation Tensix cores.
Grayskull, shown in Figure 1, is our first production
chip in 12-nm technology, and is currently in evalua-
tion with multiple customers. It is first incarnation of
our large cluster-on-a-chip multicore architecture and
is composed of 120 compute cores. The physical area
of the 10x12 grid of cores is 477 mm2. Each core oper-
ates independently; it has its own unique instruction
queue and progresses at its own pace, in contrast to
monolithic chip-scale SIMD, VLIW or single-kernel

50 IEEE Micro Published by the IEEE Computer Society March/April 2021

 HOT CHIPS

TABLE 1. TensTorrent devices.

Jawbridge Grayskull Wormhole

Manufactured 2019 2020 2021
Technology 14 nm 12 nm 12 nm
Compute grid 2x3 grid of 10x12 grid 10x8 grid of
cores of cores cores
On-chip 6 MB 120 MB 120 MB
SRAM
Off-chip IO 1 ch. 8 ch. 16 ports of
LPDDR4, LPDDR4, 100G Ethernet,
PCIe PCIe 6 ch. GDDR6,
FIGURE 1. Grayskull 75W PCIe board. Gen4x4 Gen4x16 PCIe Gen4x16
CPUs 4 core 4 core 4 core OoO
OoO ARC OoO ARC ARC
GPU models. A network-on-chip is used to transfer
data between the cores and to synchronize cores at
various points during the program execution. The Wormhole device (12 nm) contains 16 ports of
The Network-on-Chip (NoC) is a 2-D, bi-directional 100-Gb Ethernet, an integrated network switch, and six
torus, with a bandwidth of 192 GB/s per node. The channels of GDDR6. The compute fabric is composed of
NoC was architected internally alongside the Tensix Tensix cores similar to Grayskull. This communication-
core, and it has been optimized for ML workloads, oriented architecture realizes our vision of converged
described in more detail in the “Block Sparsity” sec- networking and accelerated AI compute on a single
tion. The NoC connects all the cores, as well as off- device. The large number of dedicated communication
chip communication and memory controller blocks. links on Wormhole enable many-device scaleout, by con-
As a result, each of the 120 compute cores has access necting Wormhole devices directly to each other, with-
to the PCIe and to the DRAM off-chip memory. out a central host CPU processor or Ethernet switches.
The chip includes a Gen4x16 PCIe block for com-
munication with the host processor and other Gray- Tensix Core
skull devices. Eight channels of LPDDR4 are located TensTorrent architecture operates on the basis of pack-
along the north and south edges of the compute grid. ets. The data units moved between the memories on the
In terms of numerical precision, Grayskull NoC are packets. Also, compute is executed directly on
supports: packets, shown in Figure 4. A single Tensix core contains
a packet processor, a packet manager, SRAM, and five
1. full fp16/BFLOAT at 92 TFLOPs; RISC processors. The RISC processors execute the run-
2. reduced precision mode of fp16/BFLOAT, at 122 time software which dispatches instructions to
TFLOPs; the packet processor and to the packet manager. On
3. block-based 8-bit floating point at 368 FLOPs. Grayskull, the SRAM has a capacity of 1 MB, with a

FIGURE 2. TensTorrent devices. (a) Jawbridge (2019). (b) Grayskull (2020). (c) Wormhole (2021).

March/April 2021 IEEE Micro 51

HOT CHIPS
FIGURE 3. Single Tensix core.
384-GB/s read/write bandwidth. The memory space is
primarily used by the local core, but it is directly accessi-
ble by remote cores as well.
Packet Compute Engine
The packet compute is a SIMD-based matrix and vec-
tor engine with a high degree of flexibility and
programmability. Peak compute density is 3 TOPs at
8-bit precision, or 0.75 TFLOPs at 16-bit floating-point
precision. The packet compute engine is software-
programmable via the associated RISC cores, which
execute kernels written in standard C language and
issue matrix and vector instructions to the engine. A
large number of PyTorch and TensorFlow deep learn-
ing instructions are supported. The matrix and vector
engine support operations on a range of integer and
floating-point formats. Most importantly it natively
handles sparse computation to achieve speedup,
reduce power, and memory footprint.
The packet compute engine does not have a global
view of execution across the multicore system. Its
operation is driven primarily by the packet manager:
incoming packets from the packet manager are com-
puted and returned to the packet manager for storage
or data transfer.
Packet Manager Engine
The packet manager, depicted in Figure 3, is com-
posed of data transfer engine, router, and tensor
manipulation engine.
The data transfer engine is responsible for execut-
ing all data movement and synchronization among the
compute engines, as well as between on-chip SRAM,
off-chip memory and I/O. The packet manager and
compute engine each receive their own unique instruc-
tion queues from the compiler, and they execute con-
currently. The packet manager completely de-burdens
52 IEEE Micro
FIGURE 4. Packets flowing between compute operations. (a)
Tensor decomposed into packets. (b) Packets flowing
between compute operations.
the compute engine from the complexity of data move-
ment and multicore synchronization. These features of
the packet manager realize the push-based transfer
model, which maximizes the overlap between compute
and data transfers.
The router moves packets across the NoC. It pro-
vides guaranteed ordering, manages flow control and
backpressure, and has deadlock-free operation. It is
also optimized for the way AI workloads are parallel-
ized across our multicore architecture and has effi-
cient multicast and gather capabilities.
Finally, the tensor manipulation engine can per-
form dynamic packet compression; the smaller mem-
ory footprint enabled by compression results in faster
data transfers and an increase in data locality. Fur-
thermore, tensor manipulation instructions can be
executed by this engine, described in the “Optimiza-
tion of Tensor Manipulation Instructions” section.
SOFTWARE
The software is composed of three main pieces.
1. The machine learning framework integration and
plugin software.
2. The runtime software, executing on the RISC
processors.
3. The ahead-of-time graph compiler.
Framework Integration
The Tenstorrent compiler and runtime have been
natively integrated into PyTorch, and support both
inference and training flows. The user can target execu-
tion on a single device, or multidevice cluster. In either
case, the hardware is visible to the user as a single
device. The multidevice scheduling and parallelization
are orchestrated behind the scenes by the software
stack. In addition, the software stack can execute
March/April 2021

ONNX networks as well, enabling a funnel from the
frameworks that export into the ONNX format.
Graph Compiler
The graph compiler is composed of three main compo-
nents, the front end, optimizer, and back end. The primary
role of the front end is to lower a wide range of instruc-
tions to a smaller number of optimized instructions sup-
ported by the hardware. Instructions are parallelized and
scheduled onto the device cores by the optimizer, which
maximizes performance by balancing compute, data local-
ity, and data movement. The back-end translates the com-
piled graph down into instruction queues for each core.
The packet managers and NoC connecting the
cores is are visible to the software and the data move-
ment and synchronization are both controlled explicitly
by the compiler. To schedule the data movement, the
compiler packetizes each tensor by splitting it into
“mini-tensors,” and each mini-tensor is combined with
a packet header. Each packet header contains a unique
packet ID, and all data is referenced via unique packet
IDs. The header also contains routing information,
enabling the packet manager to perform the desired
data transfers between the cores across the NoC.
Runtime Software
The runtime software runs concurrently on RISC pro-
cessors within every core. The compiled executable
contains instruction queues for the packet processor
and the packet manager of every core. The runtime
software manages the queues, and dispatches instruc-
tions to the packet compute and the packet manager.
Buffers containing packets are dynamically allo-
cated and de-allocated during runtime. The runtime
software works in tight collaboration with the packet
manager to store packets into the allocated buffers.
The runtime also controls the storage target, allowing
for buffers that do not fit into a core’s local SRAM to
spill to either remote SRAM, or to an off-chip memory.
The architecture also supports various types of
conditional executions such as if-statements, and for
and while loops. The runtime software interprets the
instruction queues generated for each core and can
execute jumps to a specific instruction in the instruc-
tion queues to reflect control flow decisions.
FULL STACK PERFORMANCE
OPTIMIZATIONS
Optimization of Data Transfers: The
Push-Based Model
Traditional multicore devices operate on a pull-based
data transfer model. For example, when a compute
March/April 2021
HOT CHIPS
core is ready to begin computing, as a first step it
issues a request to copy remotely stored data (from
another core’s cache, or from DRAM) into its local
memory or cache. After the copy has been completed,
the compute core starts computing. The read request
latency combined with a data transfer through a
potentially congested NoC or memory port, could
result in the consumer core being idle while waiting
for its data to arrive.
In contrast, our architecture operates on a push-
based data transfer model. A core that produces an
output buffer is aware of the consumer core that
needs to receive it. Instead of waiting for the con-
sumer core to issue a remote read request, the pro-
ducer core proactively copies the buffer to the
consumer core. This approach minimizes the idle time
of the consumer core.
The data transfer engine executes all the required
flow control for the push-based data transfer model. It
receives an instruction queue from the graph compiler
containing information about the producer-consumer
connectivity. The instructions enable the data transfer
engine to execute data transfers using a number of
multicore synchronization instructions, including
exchange of data transfer status, such as data-ready
or memory-space-ready.
Optimization of Tensor Manipulation
Instructions
Instructions that make up deep neural networks fall
into two main categories: 1) math instructions, and 2)
tensor manipulation (TM) instructions. TM instruc-
tions do not modify the data inside the tensor, but
simply reshuffle the tensor contents. Common TM
instructions in NLP networks include reshape, trans-
pose, flatten, and permute.
The TM reshuffling is performed on the intermedi-
ate activation data, and hence must be executed dur-
ing runtime. One implementation approach is to issue
and execute each TM instruction independently to
hardware during runtime. This typically involves a spe-
cific read/write pattern from/to memory, where the
patterns match the particular TM to be implemented.
GPUs execute TMs using this approach, shown in
Figure 5(c), which idles compute cores while perform-
ing potentially complex memory access.
In contrast, our architecture overlaps the execu-
tion of math instructions performed by the compute
engine and the TM instructions performed by the ten-
sor manipulation engine. This process is facilitated by
the graph compiler. The execution trace in Figure 5(b)
shows the overlapping of compute and TMs. The
IEEE Micro 53
HOT CHIPS
FIGURE 5. Tensor manipulation instructions executed on
Grayskull and a GPU. (a) Application graph received from
PyTorch. (b) Execution trace on a Grayskull Tensix core. (c)
Execution trace on a GPU.
MM_1 compute instruction and the Reshape and
Transpose instructions execute concurrently, in a
pipelined fashion.
The tensor manipulation engine is programmable –
it receives its own unique instruction queue from the
compiler. TM instructions are executed using a combi-
nation of two methods. First, it contains a small stor-
age that it uses as a scratch pad to load a packet and
reshuffle it in place. Second, it can execute complex
memory read/write patterns. Using a combination of
these approaches, any tensor manipulation instruc-
tion can be implemented inline as the packets are
being streamed out of the packet compute engine
and being written into local SRAM.
Flexible Scheduling and Parallelization
The TensTorrent architecture unlocks a tremendous
amount of concurrency. All building blocks receive
their own unique instruction queues from the compiler
and can progress at their own pace. As a result, the
overlap between compute and data transfers is
maximized.
However, any single parallelization approach even-
tually plateaus, hence the desire to support flexible
parallelization approaches along all available dimen-
sions for any given compute layer. Each individual
deep learning operation can be parallelized across a
variable number of cores, combining a number of par-
allelization approaches. In addition, operations can be
run in parallel, be pipelined, or sequential, across the
many cores of a device, shown in Figure 6.
54 IEEE Micro
FIGURE 6. Flexible scheduling and parallelization. (a) Pipeline
parallelism. (b) Model parallelism.
DYNAMIC EXECUTION
Dynamic execution is an umbrella term representing
various approaches that reduce the computational
complexity of a network at runtime. Some approaches
can be represented within the topology of network
itself, such as Mixture-of-Experts (MoE), while others
can be used to augment the network execution during
runtime. Four approaches enabled by the Tenstorrent
architecture are described next.
Block Sparsity
Tensors feeding into math operations within networks
contain a variable amount of sparsity within them.
Certain models have been tuned to take advantage of
sparsity of trained parameters,2 or take advantage of
block sparsity in model parameters.5 However, these
approaches do not tap into a large potential of spar-
sity in activations, which can be inherent or induced
at runtime.7 To fully utilize this potential, in addition to
model parameter sparsity, our architecture supports
block sparsity of activations, which enables quadratic
gains from run-time activation sparsity.
Dynamic Precision
Similar to scientific computing applications, numerical
precision can be traded off for an increase in perfor-
mance and a reduction in power. The Tenstorrent
architecture enables a numerical precision to be set
at a fine grain level, per each packet in the neural net-
work. The setting can be specified both ahead-of-time
by the compiler, as well as during runtime.
March/April 2021

Runtime Compression
Tensors can be compressed and decompressed dur-
ing runtime via dedicated hardware blocks. Parame-
ters can be compressed at compile time, and de-
compressed at runtime by the hardware. Similarly, dur-
ing runtime, the output activations of a math layer can
be compressed inline as they are being produced.
Both approaches result in reduced memory footprint,
in addition to power savings from smaller and faster
data transfers.
Conditional Computation
MoE1 involves the use of conditional computations,
where only parts of the network are computed, on a
per-input basis. This allows a significant increase in
model capacity, without a proportional increase in
computational complexity. A host CPU can easily
implement such conditional computation, however
that approach involves unnecessary data transfers
between host and the AI processor and is impractical
for large scaleout implementations. The Tenstorrent
architecture is the first architecture, to the best of our
knowledge, that tightly couples native conditional
computation together with the dense compute within
an AI processor fabric.
RESULTS
We measured a baseline result for BERT-base in
BFLOAT16 precision at 2830 sequences/s. Significant
speedup is achievable by applying two optimizations
on top of this baseline: dynamic activation sparsity
and use of 8-bit floating point. In our experiments we
observe that 75% sparsity in activations (induced
dynamically at runtime) results in 4x speedup on BERT
layers. Similarly, we observe that using block-based
8-bit floating point precision provides a factor of two.
The two optimizations can be combined synergisti-
cally—they both reduce the activation memory foot-
print linearly for a total of 8x reduction, and 8-bit floats
reduce the model parameter memory footprint by 2x.
This enables the majority of the model parameters to
fit on chip allowing the sparsified layers to be fed from
local SRAM. Realizing this on an entire BERT-base is
work in progress and we project a score of 23 345
sequences/s.
CONCLUSION
We solve the private memory parallel computing prob-
lem and tensor manipulations in a way that removes
communication, synchronization, and data shuffle bot-
tlenecks and enables keeping all the compute units
highly utilized.
March/April 2021
HOT CHIPS
In comparison to alternative and competing archi-
tectures, Tenstorrent’s architecture is the only one
which has all the features of the next-generation com-
puting architecture, one that allows fusion of Software
2.0 (“neural nets”)8 and Software 1.0 (“classical
programs”):
› Same programming model for single-chip and mul-
tichip scaleout. Parallelization for private memory
combined with direct compute-on-packets.
› Flexible parallelization across all tensor dimensions
and approaches, including the time dimension.
› Native ability to intermingle dense-math-heavy
nodes of a neural net with sparse nodes, proce-
dural nodes (i.e., “classical programs”), and vari-
ous dynamic and conditional execution
techniques. These are all critical building blocks
for a compute substrate that allows fusion of
Software 1.0 and Software 2.0., entirely removing
the need for host CPU fallback.
› Fully programmable: kernels and runtime system
(firmware) are written in standard C language.
› Scaleout capability and flexibility: scaleout via
standard Ethernet that seamlessly integrates
into our NoC.
REFERENCES
1. N. Shazeer et al., “Outrageouly large neural networks:
The sparsley-gated mixture-of-experts layer,” in Proc.
Int. Conf. Learn. Representations, 2017. [Online].
Available: https://arxiv.org/abs/1701.06538
2. V. Sanh et al., “Movement pruning: Adaptive sparsity by
fine-tuning,” 2020. [Online]. Available: https://arxiv.org/
abs/2005.07683
3. OpenAI, “AI and compute,” 2018. [Online]. Available:
https://openai.com/blog/ai-and-compute/
4. Stanford HAI. “Artificial intelligence index report 2019,”
2019. [Online]. Available: https://hai.stanford.edu/sites/
default/files/ai_index_2019_report.pdf
5. T. Gale et al., “Sparse GPU kernels for deep learning,” 2020.
[Online]. Available: https://arxiv.org/pdf/2006.10901.pdf
6. “Fast block sparse matricies for pytorch,” 2020. [Online].
Available: https://github.com/huggingface/pytorch_
block_sparse
7. Z. Chen et al., “You look twice: Gaternet for dynamic
filter selection in CNNs,” 2019. [Online]. Available:
https://arxiv.org/pdf/1811.11205.pdf
8. A. Karpathy, “Software 2.0,” 2017. [Online]. Available:
https://medium.com/@karpathy/software-2-0-
a64152b37c35
IEEE Micro 55
 GENERAL INTEREST
The Design Process for Google’s Training
Chips: TPUv2 and TPUv3
Thomas Norrie , Nishant Patil , Doe Hyun Yoon , George Kurian , Sheng Li , James Laudon ,
Cliff Young , and Norman Jouppi , Google, Mountain View, CA, 94043, USA
David Patterson , Google, Mountain View, CA, 94043, USA and UC Berkeley, Berkeley, CA, 94720, USA
F
ive years ago, few would have predicted that a
software company like Google would build its
own chips. Nevertheless, Google has been
deploying chips for machine learning (ML) training since
2017, powering key Google services. These tensor proc-
essing units (TPUs) are composed of chips, systems, and
software, all codesigned in-house. This article details the
circumstances that led to this outcome, the challenges
and opportunities observed, the approach taken for the
chips, a quick review of performance, and finally a retro-
spective on the results. A companion paper describes
the supercomputers built from these chips, the compiler,
and its performance in detail.6
FORCES PUSHING TOWARDS
CUSTOM ML HARDWARE
In 2013, only one year after AlexNet swept the Image-
Net competition,4 Google leaders predicted ML would
soon be ready to attack difficult problems like produc-
tion versions of image and speech recognition. Alas,
these ML models were so computationally expensive
that a sustainable service was nearly impossible, as
Internet service costs scale by the number of users
who grow as the service improves. The motivation to
slash ML inference serving costs led to the TPUv1
inference system, deployed successfully in 2015.5
TPUv1 exposed the next bottleneck: ML training.
The team quickly pivoted into building the TPUv2
training system. Two years later, TPUv2 powered key
Google services with fast and cost-effective training.
CHALLENGES AND OPPORTUNITIES
OF BUILDING ML HARDWARE
ML training brings challenges relative to ML inference:
› More computation. More means both the types of
computation—for example, backpropagation
0272-1732 ß 2021 IEEE
Digital Object Identifier 10.1109/MM.2021.3058217
Date of publication 9 February 2021; date of current version
26 March 2021.
56 IEEE Micro Published by the IEEE Computer Society
requires matrix transposition and loss functions—
and the amount of computation. An inference cal-
culation typically executes on the order of 109
floating point operations, but Google’s production
training applications require 1019–1022; more than
10 orders of magnitude larger!
› More memory. During training, temporary data
are kept longer for use during backpropagation.
With inference, there is no backpropagation, so
data are more ephemeral.
› Wider operands. Inference can tolerate int8
numerics relatively easily, but training is more
sensitive to dynamic range due to the accumula-
tion of small gradients during weight update.
› More programmability. Much of training is
experimentation, meaning unstable workload
targets such as new model architectures or
optimizers. The operating mode for handling a
long tail of training workloads can be quite
different from a heavily optimized inference
system.
› Harder parallelization. For inference, one chip can
hit most latency targets. Beyond that, chips can be
scaled out for greater throughput. In contrast, exa-
flops-scale training runs need to produce a single,
consistent set of parameters across the full sys-
tem, which is easily bottlenecked by off-chip
communication.
These problems felt daunting initially, plus we had
constraints on time and staffing. Time matters
because each day saved during development acceler-
ates our production training pipeline a day. And as for
staffing: while Google is teeming with engineers, they
are not all available for our project. Ultimately, the
TPU team had only a cup from Google’s large engi-
neering pool. Thus, we had to be ambitious to over-
come the complexities of training, but the time and
staffing budget set constraints.
To prioritize, we sorted activities into two buckets:
those where we must do a great job, and those that
we only have to do good enough. The first bucket
included the following:
March/April 2021

FIGURE 1. Transforming the TPUv1 datapath into the TPUv2 datapath.
1 Build quickly.
2 Achieve high chip performance.
3 Scale efficiently to numerous chips.
4 Work for new workloads out of the box.
5 Be cost effective.
Everything else was in the second bucket. While
tempting to brush these second bucket issues aside
as minor embarrassments, the reality is building and
delivering a good system is as much about what you
decide not to do as what you decide to do. In retro-
spect, these decisions are not embarrassing after all!
We refer to the relevant goals using the circled
numbers (e.g.,
2 ) throughout the discussion to high- AVAILABLE FOR OUR PROJECT.
light “first bucket” design decisions.
OUR APPROACH TO ML
HARDWARE
TPUv1 provided a familiar starting point for our train-
ing chip [see Figure 1(a)]. The high-bandwidth loop
(red) identifies the important part of TPUv1: the core
data and computation loop that crunches neural net-
work layers quickly. DDR3 DRAM feeds the loop at
much lower bandwidth with model parameters. The
PCIe connection to the host CPU exchanges model
inputs and outputs at even lower bandwidth.
Figure 1 shows five piecewise edits that turn TPUv1
into a training chip. First, splitting on-chip SRAM
makes sense when buffering data between sequential
March/April 2021
GENERAL INTEREST
fixed-function units, but that is bad for training as on-
chip memory requires more flexibility. The first edit
merges these into a single vector memory [see Figure 1
(b) and (c)]. For the activation pipeline, we moved
away from the fixed-function datapath (containing
pooling units or hard-coded activation functions) and
built a more programmable vector unit [see Figure 1
(d)]
4 . The matrix multiply unit (MXU) attaches to the
vector unit as coprocessor [Figure 1(e)].
WHILE GOOGLE IS TEEMING WITH
ENGINEERS, THEY ARE NOT ALL
ULTIMATELY, THE TPU TEAM HAD
ONLY A CUP FROM GOOGLE’S LARGE
ENGINEERING POOL. THUS, WE HAD
TO BE AMBITIOUS TO OVERCOME THE
COMPLEXITIES OF TRAINING, BUT THE
TIME AND STAFFING BUDGET SET
CONSTRAINTS.
Loading the read-only parameters into the MXU for
inference does not work for training. Training writes
those parameters, and it needs significant buffer
space for temporary per-step variables. Hence, DDR3
moves behind Vector Memory so that the pair form a
memory hierarchy [also in Figure 1(e)]. Adopting in-
IEEE Micro 57
GENERAL INTEREST
FIGURE 2. TPUv2 datapath in more detail, showing both cores, which appear as a single PCIe device.
package HBM DRAM instead of DDR3 upgrades band-
width twentyfold, critical to utilizing the core
2. bra.6) In these discussions, we landed on two impor-
Last is scale. These humongous computations are
much bigger than any one chip. We connect the mem-
ory system to a custom interconnect fabric (ICI for
InterChip Interconnect) for multichip training [see
1
2 . Second, we could ensure gener-
Figure 1(f)]
3 . And with that final edit, we have a train- ality by architecting within the principled language of
ing chip!
4 . That meant focusing on the compu-
Figure 2 provides a cleaner diagram, showing the
two-core configuration. The TPUv2 core datapath is
blue, HBM is green, host connectivity is purple, and
the interconnect router and links are yellow. The
TPUv2 Core contains the building blocks of linear alge-
bra: scalar, vector, and matrix computation.
Why two cores? The simpler answer is that we
could fit a second core into a reasonably sized chip
2
5 . But then why not build a single, bigger core? Wire
latencies grow as the core gets bigger, and the two-
core configuration hits the right balance between rea-
sonable pipeline latency and additional per-chip com-
putation capacity. We stopped at two bigger cores
because they are easier to program as they allow the
developer to reason about one unified instruction
stream operating on big chunks of data
4 , rather than and six immediates.
exposing lots of tiny cores that need to be lashed
together. We took advantage of training fundamen-
tally being a big-data problem.
The following sections dive deeper into key chip
components.
1 . It is not the flashiest solution,
TPUV2 CORE
The TPUv2 Core was codesigned closely with the com-
piler team to ensure programmability
4 . (The tion happens. At the bottom right is the DMA port into
58 IEEE Micro
compiler’s name is XLA, for accelerated linear alge-
tant themes. First, a VLIW architecture was the
simplest way for the hardware to express instruction
level parallelism and allowed us to utilize known com-
piler techniques
linear algebra
tational primitives for scalar, vector, and matrix data
types.
Scalar Computation Unit
The scalar unit is where computation originates. It
fetches complete VLIW bundles from a local instruc-
tion memory, executes the scalar operations slots
locally, and then forwards decoded instructions on to
the vector and matrix units where execution happens
later, decoupled from scalar execution. The VLIW bun-
dle is 322 bits and is composed of two scalar slots,
four vector slots (two used for vector load/store), two
matrix slots (a push and a pop), one miscellaneous
slot (a simple example would be a delay instruction),
Figure 3(a) shows a diagram of the scalar unit. At
the top left is the instruction bundle memory. While an
instruction cache backed by HBM would have been
nice, a DMA target for software-managed instruction
overlays was easier
but remember that this part just needs to be “good
enough.” The greatness in this system lies elsewhere.
Down the left of Figure 3(a), we perform scalar
decode and issue, and on the right is where scalar execu-
March/April 2021

FIGURE 3. (a) TPUv2 core scalar unit and (b) single vector lane. The vector unit contains 128 vector lanes.
the memory system, primarily to HBM. That feeds into
local Scalar Memory SRAM to issue loads and stores
against. They feed a 32-deep register file containing 32-
bit words, which finally feeds into a dual-issue ALU at
2.
the top right. Execution interlock is managed by stan-
dard hold conditions on the instructions, while synchro-
nization flags provide a way to interlock against
software-managed DMA operations. The memory hierar-
chy is under the control of the compiler, which simplifies
the hardware design while delivering high performance
1
2
5.
Vector Computation Unit
2 . While it performs most of
After scalar execution, the instruction bundle and up to
three scalar register values are forwarded to the vector
5.
unit. Figure 3(b) shows a single vector lane, and the
entire vector unit contains 128 such vector lanes. Each
lane contains an additional 8-way execution dimension
called the sublane. Each sublane contains a dual issue
32-bit ALU connected into a 32 deep register file. All
together, the vector computation unit allows operation
on eight sets of 128-wide vectors per clock cycle. Sub-
2
5.
lanes let TPUv2 increase the vector versus matrix com-
pute ratio, which is useful for batch normalization.6
Each lane’s register files perform loads and stores
against the lane’s local slice of vector memory, and
that memory connects into the DMA system (primarily
providing access to HBM).
On the right of Figure 3(b) is the connectivity into
the matrix units. The push instruction slots send data
vectors out to the matrix units. A result FIFO captures
any returning result vectors from the matrix units, and
these can be popped into vector memory using the
pop instruction slots. The result FIFO lets us avoid
2
4,
strict execution schedule constraints for the long-
latency matrix operations and shorten register life-
times, simplifying the compiler
1. bfloat16.
March/April 2021
GENERAL INTEREST
Matrix Computation Units
The matrix multiply unit is the computational heart of
the TPU. It is a 128 x 128 systolic array of multipliers
and adders, delivering 32,768 operations per cycle
The inputs are two matrices: a left hand side matrix
and a right hand side matrix. The left hand side matrix
streams over a preloaded right hand side matrix to
create a streaming result matrix, which is sent directly
to the vector unit’s result FIFO. The right hand side
matrix can be optionally transposed when loaded into
the matrix unit.
Critically, the systolic array structure provides high
computational density
the FLOPS per second, it is not the largest contributor
to chip area (see Figure 4)
Numerics shape computation density. Multiplica-
tions use bfloat16: this is a 16-bit format that has the
same exponent range as float32 but with fewer bits of
mantissa. The accumulation happens in full 32-bit
floating point.
Bfloat16 works seamlessly for almost all ML train-
ing, while reducing hardware and energy costs
Our estimate for the more recent 7 nm is that
bfloat16 has a 1.5 energy advantage over the IEEE
16-bit float: 0.11 versus 0.16 pJ for add and 0.21 versus
0.31 pJ for multiply. Moreover, bfloat16 is easier for
ML software to use than fp16, since developers need
to perform loss scaling to get fp16 to work.9 Many are
willing to do the extra programming to go faster, but
some are not. For example, all but 1 of the 200 ML
experts at the Vector Institute rely on the slower 32-
bit floating point in GPUs.8 We know of no downsides
for bfloat16 versus fp16 for ML. As it takes less area
and energy and is easier for ML software to use
bfloat16 is catching on. TPUv2 was the first, but ARM,
Intel, and NVIDIA have subsequently embraced
IEEE Micro 59
GENERAL INTEREST

Matrix multiplication is not the only important connect into a supercomputer, with many chips work-
matrix transformation, so another set of units effi- ing together to train a model.
ciently perform other matrix operations like trans- The chip includes four off-chip links and two on-
poses, row reductions, or column permutations. chip links connected with an on-chip router. These
four links enable the 2-D torus system interconnect,
TPUV2 MEMORY SYSTEM which matches many common ML communication
patterns, like all-reduce. The four external links deliver
As discussed earlier, the TPUv2 Core has a variety of
500 Gb/s and the two internal links are twice as fast.
SRAM-based scratchpad memories. These are archi-
Some torus links are within the TPU tray, and the rest
tecturally visible to software and accessed using loads
are through cables across trays and racks.
and stores. This approach gives the compiler predict-
An important property of interconnect design is ease
able execution control over both computation and
of use 4 : DMAs to other chips look just like DMAs to
SRAM and simplifies the hardware 2 4.
local HBM, albeit with a push-only restriction for simplic-
But the memory story cannot stop with SRAM,
ity 1 . This common DMA interface allows the compiler
because most models would not fit. Figure 2 shows
to treat the full system’s memory space consistently.
that high-bandwidth, in-package HBM backs up the
The dedicated TPU network enables scalable syn-
SRAM. The compiler moves data between HBM and
chronous training across TPUs. Asynchronous training
SRAM using asynchronous DMAs. Since HBM holds
was the state-of-the-art previously, but our studies
vectors and matrices, DMAs can stride through mem-
showed synchronous converges better 4 —async pri-
ory to allow slicing off subdimensions of larger dimen-
marily allowed broader scaling when networking was
sional structures to reduce the DMA descriptor
limited.
overhead. When a DMA finishes, a completion notifi-
We connect the TPU pod to storage over the data-
cation lands in the core’s sync flags, allowing the pro-
center network to feed the input data for the model
gram to stall until data arrives.
via a PCIe-connected CPU host. The system balance
Taking a step back, this approach provides a clean
across CPU, network, and storage is critical to
partitioning of concerns. The core (blue in Figure 2)
achieve end-to-end performance at scale 3 . The
provides predictable, tightly scheduled execution 2.
PCIe straw is tiny (16 GB/s per chip) the in-package
The memory system (in green) handles asynchronous
bandwidth is huge (700 GB/s per chip), and the inter-
prefetch DMA execution from the larger HBM memory
connect links are somewhere in the middle (4 60
space 5 . The hand-off between the regimes is man-
GB/s). Building TPUv2 to be flexible and programma-
aged with sync flags.
ble allows operation on all data locally instead of
Ultimately, the primary goal of the memory system
moving data back to the host through a bandwidth-
is to keep the datapath fed 2 , so HBM’s high band-
constricted PCIe bus 2.
width is critical. At 700 GB/s per chip, the HBM of
TPUv2 provides 20 the bandwidth of the pair of DDR3
channels in TPUv1. This increase allows full computa- TPUV2 FLOOR PLAN
tion utilization at much lower data reuse rates 4. The floorplan in Figure 4 is not stylish, but it was good
Zooming out to the chip-level memory system, enough and allowed us to focus on more important
Figure 2 shows each core is connected to half of the things 1 . Most area is for the computation cores in

chip’s HBM. The split HBM memory space might be a
bit surprising, but the reason is simple: we wanted to
build this chip quickly
1 , this approach simplified the and bottom, with the interconnect router sitting in the
design, and it was good enough.
Each core also has a set of PCIe queues to exchange
data with the host. Between the two cores is the inter-
connect router that also connects to the off-chip links.
5.
TPUV2 INTERCONNECT
A dedicated interconnect is the foundation of the
TPUv2 supercomputer (“pod”). TPUv1 was a single-
chip system built as a coprocessor, which works for
inference. Training Google production models would
require months on a single chip. Hence, TPUv2s can
blue, with noticeable area for the memory system and
interconnect. The two cores are split across the top
donut hole in the middle. The white areas are not
empty, but filled with wiring.
The two matrix multiply units are at the top center
and bottom center. The FLOP/second delivered in
such a small area demonstrates the computation den-
sity available with the bfloat16 systolic array
TPUV3
We hoped to avoid the second system effect2; we did not
want to blow everything we worked hard for in TPUv2 by
building the kitchen sink into TPUv3. TPUv3 is a “mid-life

60 IEEE Micro March/April 2021

 GENERAL INTEREST

FIGURE 4. TPUv2 floor plan.

kicker” that leveraged what we already built—both use 16 It seems quaint now, but we thought the
nm technology—but let us pick the low-hanging fruit left 256-chip TPUv2 system was huge. ML’s voracious
behind by the quick development of TPUv2 1 . The most appetite continues,1 so moving to 1024-chip sys-
important enhancements were the following. tems was critical
3.

› Doubling the matrix multiply units to get double

maximum FLOPS/second 2 (high leverage due PERFORMANCE SYNOPSIS
to the matrix multiply unit computation density). Figure 5 shows a roofline model of TPUv2/v3 and the
› Increasing clock frequency from 700 to 940 MHz 2 NVIDIA V100 GPU. It identifies programs as memory-
(30% performance from tuning pipelines). bound (under the slanted roofline) or compute-
› Bumping HBM performance by 30% using a bound (under the flat roofline) using operational
higher bus speed 2. intensity (operations per DRAM byte accessed) to
› Doubling HBM capacity enabling bigger models place applications appropriately. The vertical lines in
and bigger batch sizes [see Figure 5(a)]
4. Figure 5(a) show the performance improvements
› Raising interconnect link bandwidth by 30% to from TPUv2 to TPUv3. CNN0 is actually above its
650 Gb/s per link 2. fp32 roofline in Figure 5(b)! This model includes
› Raising the maximum scale to a 1024-chip sys- many 33 convolutions and V100 uses Winograd
tem, up from the 256-chip limit of TPUv2 3. Transform that does fewer operations than a

March/April 2021 IEEE Micro 61

GENERAL INTEREST

FIGURE 5. (a) Roofline Models11 for TPUv2 and TPUv3 (left). (b) Roofline Models of V100 for fp16 and fp32 (right). RNN0/RNN1
move up and to the right in (a) since the larger HBM capacity of TPUv3 enabled bigger batch sizes; all others use the same batch
size. MLP0/MLP1 could not run on Volta in (b) because the embedding tables were too big.

convolution. However, Figure 5(b) assumes the same unwilling to do the extra work of loss scaling for
computation for comparison, lifting CNN0 through fp16 9 4.

its roofline. › The peak computation gain of TPUv3 over

TPUv2 is 2.7, but the improvements in mem-
ory bandwidth, ICI bandwidth, and clock rate
AS DEEP LEARNING CONTINUES TO
are only 1.3. Were the extra MXUs in TPUv3
EVOLVE, AND AS WE UNDERSTAND underutilized due to bottlenecks elsewhere?
THE WORKLOADS AND SCALING The geometric mean of performance gains of
LIMITS BETTER, OPPORTUNITIES TPUv3 over TPUv2 were 1.8 for both MLPerf
CONTINUE FOR FURTHER CODESIGN 0.6 benchmarks and Google production appli-
ACROSS ML MODELS, SOFTWARE, cations 2 . MXUs are a small part of the chip

AND HARDWARE TO IMPROVE NEXT (see Figure 4), so doubling the MXUs worked
GENERATION TPUS AND OTHER well 5.

DOMAIN-SPECIFIC ARCHITECTURES. › Google applications scaled to 96%–99% of per-

fect linear speedup 3 ; applications for general-

purpose supercomputers are rarely that high.

› 3 allows TPUv3s to scale to supercomputer

size and performance. The GigaFLOPS/Watt of

A companion paper6 measures performance in a TPUv3 supercomputer running a Google
detail, summarized here. application was 50 higher than a general-pur-
pose supercomputer running the Linpack
› Although TPUv3 is a smaller chip (<700 versus benchmark6 5.
812 mm2) in an older semiconductor technology
than Volta (16 nm versus 12 nm), the geometric
mean of performance per chip is the same for CONCLUSION
the MLPerf 0.6 benchmarks 5 . GPUs are larger
TPUv2/v3 met our “first bucket” goals:
in part because they also perform graphics
operations.
1 Build quickly.
› Using Google production applications, TPUv3 is Our cross-team co-design philosophy found sim-
on average five times faster. Like the Vector pler-to-design hardware solutions that also gave
Institute, Googlers use 32-bit floating point oper- more predictable software control, such DMA to
ations on V100 [see Figure 5(b)] as they were main memory (HBM) and compiler-controlled on
chip memory instead of caches. Along the way,

we made difficult tradeoffs to preserve the devel-
MLPerf 0.7 previewed the next generation TPU chip (TPUv4),
which matched the performance of NVIDIA’s next generation opment schedule, such as splitting the HBM
GPU chip (Ampere), but that’s a story for another time. between the two cores, tolerating an inefficient

62 IEEE Micro March/April 2021


chip layout, and using FIFOs to simplify XLA com-
piler scheduling.
2 Achieve high performance.
Matrix computation density, high HBM bandwidth,
and XLA compiler optimizations deliver excellent
performance.
3 Scale up.
Our system-first approach and simple-to-use inter-
connect let TPUv3 scale natively to 1024 chips and
deliver nearly linear speedup for production
applications.
4 Work for new workloads out of the box.
To support the deluge of training workloads, we
built a core grounded in linear algebra that works
well with the XLA compiler, and HBM ensures we
have enough capacity and bandwidth to keep
pace with growing models.
5 Be cost effective.
The matrix units are efficient, the design was sim-
ple without gratuitous bells and whistles, and we
got our money’s worth in performance.
As deep learning continues to evolve, and as we
understand the workloads and scaling limits better,
opportunities continue for further codesign across ML
models, software, and hardware to improve next gener-
ation TPUs and other domain-specific architectures.
ACKNOWLEDGMENTS
TPUv2/v3 exist thanks to the creativity and hard work
of the TPU team, from chips to systems to software.
The author list represents a small slice, and we are
grateful for the opportunity to work together. Thanks
go to Paul Barham, Eli Bendersky, Dehao Chen, Clifford
Chao, Chiachen Chou, Jeff Dean, Brian Fosco, Ben
Gelb, Jesse Guss, Peter Hawkins, Blake Hechtman,
Mark Heffernan, Richard Ho, Robert Hundt, Michael
Isard, Terry Kang, Fritz Kruger, Naveen Kumar, Sameer
Kumar, Steve Lacy, Chris Leary, Hyouk-Joong Lee,
David Majnemer, Lifeng Nai, Rahul Nagarajan, Tayo
March/April 2021
GENERAL INTEREST
Oguntebi, Andy Phelps, Paul Rodman, Bjarke Roune,
Brennan Saeta, Amir Salek, Julian Schrittwieser, Dan
Steinberg, Andy Swing, Horia Toma, Shibo Wang, Tao
Wang, Yujing Zhang, and many more.
REFERENCES
1. D. Amodei and D. Hernandez, “AI and compute,” 2018.
[Online]. Available: blog.openai.com/aiand-compute
2. F. P. Brooks Jr, The Mythical Man-Month: Essays on
Software Engineering. London, U.K.: Pearson Education,
1975.
3. J. J. Dongarra et al.., “The LINPACK benchmark: Past,
present and future,” Concurrency Comput., Pract.
Experience, vol. 15, no. 9, pp. 803–820, 2003.
4. A. Krizhevsky et al., “Imagenet classification with deep
convolutional neural networks,” in Proc. Neural Inf.
Process. Syst., 2012, pp. 1097–1105.
5. N. P. Jouppi et al., “In-datacenter performance analysis
of a tensor processing unit,” in Proc. 44th Annu. Int.
Symp. Comput. Archit., 2017, pp. 1–12.
6. N. P. Jouppi et al., “A domain-specific supercomputer
for training deep neural networks,” Commun. ACM,
vol. 18, 63, no. 7, pp. 67–78, 2020.
7. N. Kumar, “Google breaks AI performance records in
MLPerf with world’s fastest training supercomputer,”
2020. [Online]. Available: cloud.google.com/blog/
products/ai-machine-learning/google-breaks-ai-
performance-records-in-mlperf-with-worlds-fastest-
training-supercomputer
8. J. Lin, X. Li, and G. Pekhimenko, “Multi-node Bert-
pretraining: Cost-efficient approach,” 2020,
arXiv:2008.00177.
9. P. Micikevicius et al., “Mixed precision training,” 2017,
arXiv:1710.03740.
10. D. Silver et al., “A general reinforcement learning
algorithm that masters chess, shogi, and go through
self-play,” Science, vol. 362, no. 6419, pp. 140–1144,
2018.
11. S. Williams, A. Waterman, and D. Patterson, “Roofline:
An insightful visual performance model for multicore
architectures,” Commun. ACM, vol. 52, no. 4, pp. 65–76,
2009.
IEEE Micro 63
 GENERAL INTEREST

Klessydra-T: Designing Vector Coprocessors

for Multithreaded Edge-Computing Cores
Abdallah Cheikh , Stefano Sordillo, Antonio Mastrandrea, Francesco Menichelli, Giuseppe Scotti , and
Mauro Olivieri , Sapienza University of Rome, 00185, Rome, Italy

Computation-intensive kernels, such as convolutions, matrix multiplication, and

Fourier transform, are fundamental to edge-computing AI, signal processing, and
cryptographic applications. Interleaved-multithreading (IMT) processor cores are
interesting to pursue energy efficiency and low hardware cost for edge computing,
yet they need hardware acceleration schemes to run heavy computational
workloads. Following a vector approach to accelerate computations, this article
explores possible alternatives to implement vector coprocessing units in RISC-V
cores, showing the synergy between IMT and data-level parallelism in the target
workloads.

I
nterleaved multithreading (IMT) or barrel-process-
ing is a simple and widely known program execution
paradigm that alternates instructions belonging to
different execution threads in the stages of a single-
issue in-order processor pipeline.1,2,3 In this scheme,
while the throughput is limited to one instruction per
cycle (IPC), pipeline stalls due to interinstruction
dependence are avoided without any hardware over-
head for dependence management. As long as the
application workload can be programmed as multiple
threads, the IMT approach can sustain IPC ¼ 1 with rel-
atively high clock frequency and high energy efficiency,
thanks to the hardware simplicity, which is a desirable
goal in embedded edge-computing processors.
Nonetheless, to execute computationally heavy
applications on the extreme edge, any processor core
needs hardware acceleration support. Two broad clas-
ses of hardware acceleration exist: hardware units that
autonomously execute entire computation kernels
upon memory-mapped commands from the processor
core, and instruction acceleration units, sometimes
referred to as coprocessors, that take over complex
instructions and thus are directly sequenced by the
core instruction stream. Coprocessors imply less com-
munication overhead, yet they can be efficiently
exploited only within instruction set architectures
0272-1732 ß 2021 IEEE
Digital Object Identifier 10.1109/MM.2021.3050962
Date of publication 12 January 2021; date of current version
26 March 2021.
(ISA) that allow extensions dedicated to particular
computation domains, such as RISC-V.4
Edge computing devices regard energy efficiency
as the prime concern. This article addresses the intro-
duction of vector coprocessor acceleration in IMT
cores for extreme edge computing, showing that an
IMT processor has an architectural design advantage
over other cores with similar IPC, which allows exploit-
ing hardware acceleration with higher energy effi-
ciency and speed.
In this context, we specifically address supporting
accelerated vector operations, to execute ubiquitous
computation kernels in edge computing applications:
› 2-D convolution, covering the broad area of deep
neural network applications5;
› fast Fourier transform (FFT), typical of signal
processing applications, for example, in 5G IoT
devices6;
› matrix multiplication (MatMul) used in a variety
of fields, predominantly in cryptography.
A typical scenario is to run homogenous workloads
on all the threads applying the same algorithm on dif-
ferent input data, e.g., convoluting multiple image
frames. Otherwise, one can take advantage of the mul-
tiple contexts provided in an IMT core and run a com-
posite workload running different algorithms, e.g.,
transmitting an encrypted stream of a preprocessed
video/audio, by convoluting an image while analyzing
an audio stream via FFT then encrypting the processed
data using an algorithm that heavily relies on MatMul.

64 IEEE Micro Published by the IEEE Computer Society March/April 2021


In this article, we design, implement, and evaluate a
whole taxonomy of coprocessor acceleration schemes
for IMT cores, analyzing them for performance, area,
and energy efficiency on the above application cases.
The contributions of this work are as follows:
› We provide designers with a quantitative
comparison between different coprocessing
schemes referring to different computation
kernels.
› Specifically, we identify the optimal balance
between thread level parallelism (TLP) and data
level parallelism (DLP) in the addressed
scenarios.
› We demonstrate the performance and energy
efficiency of the IMT approach in the target
application contexts by comparing it with pro-
cessor cores in the same complexity range.
› We show the potentials of an open hardware
design based on the RISCV instruction set along
with its open programming environment.
THIS ARTICLE ADDRESSES THE
INTRODUCTION OF VECTOR
COPROCESSOR ACCELERATION IN
IMT CORES FOR EXTREME EDGE
COMPUTING, SHOWING THAT AN IMT
PROCESSOR HAS AN ARCHITECTURAL
DESIGN ADVANTAGE OVER OTHER
CORES WITH SIMILAR IPC.
BACKGROUND
Many previous works reported the design of hardware
accelerated cores in edge-computing applications.
Lim et al.7 report the design details of a low-volt-
age microcontroller with subword-SIMD support. This
article is more general in investigating various SISD-
SIMD-MIMD combinations in coprocessor design. The
work by Botman et al.8 is similar and investigates ad
hoc ISA encoding and pipeline stage balancing for
power efficiency and introduces a dedicated copro-
cessor interface. Yet, the authors do not elaborate on
coprocessor architectures and performance. Our work
further differs from Gautschi et al.7 and Luo and
Zhang8 in targeting RISC-V compliance.
Gautschi et al.9 describe a RISC-V processor with
DSP hardware support, targeting near-threshold volt-
age operation, and in the Diet-SODA design,10 a SIMD-
oriented DSP accelerator also runs in near-threshold
March/April 2021
GENERAL INTEREST
regime. Our study is agnostic about supply or bias
voltage tuning, purely addressing DLP and TLP balanc-
ing for energy efficiency in any physical implementa-
tion, including soft-cores on FPGA, as shown in our
results.
A hardware convolution engine for image process-
ing is presented by Conti and Benini,11 focusing on the
optimal buffer design to store selected portions of the
input image. Chen et al.12 and Du et al.13 also present
convolution accelerators, based on parallel hardware
units and local data reuse. Our study adopts a different
approach, based on multipurpose vector coprocessors
equipped with scratchpad memories, coupled with an
IMT processor, to hide memory latency.
This article builds on the activity reported by
Cheikh et al.14 which was an initial effort into design-
ing a mathematical accelerator for a RISC-V core, and
by Cheikh et al.,3 who addressed the best performing
pipeline organization for an IMT RISC-V core.
THE KLESSYDRA-T IMT
ARCHITECTURE
The processing core discussed in this article, named
Klessydra-T13, is a parametric design implementing an
IMT four-stage-pipeline RISC-V processor. It supports
the RV32IMA instruction set,4 augmented by a custom
extension composed of a small subset of mathemati-
cal vector instructions. The Klessydra-T13 core (see
Figure 1) realizes a pure IMT paradigm as defined by
the following points:
› thread context switch at each clock cycle;
› in-order, single issue instruction execution;
› feed-forward pipeline (no hardware support for
branching-hazard and data-hazard handling);
› bare metal execution (RISCV M mode).
The core interleaves three hardware threads (harts 4)
in the instruction pipeline. The register file, program
counter, and CSR unit are replicated per hart. A hard-
ware context counter (harc) switches between the
hart program counters on a rotation basis to fetch
instructions from the program memory. The three harts
in the four pipeline stages provide a register file access
fence, so that it never possible for any two instructions
to manifest a dependence hazard in the pipeline.
The T13 core includes multiple units in the execu-
tion stage, namely a load/store unit (LSU), a scalar
execution unit (EXEC), and a vector-oriented multipur-
pose functional unit (MFU), which implements the
coprocessing features. The LSU works in parallel with
other units when executing store instructions, which
IEEE Micro 65
GENERAL INTEREST
FIGURE 1. Klessydra T13 block organization.
cannot cause a write-back conflict on the register file.
The MFU is allowed to read operands from the register
file, but can write results only to local scratchpad
memories (SPMs). The LSU manages data transfers
to/from the data memory from/to the SPMs via dedi-
cated instructions.
The MFU executes vector arithmetic instructions,
whose latency is proportional to the vector length. A
hart requesting access to the busy MFU executes a
self-referencing jump until the MFU becomes free,
avoiding unnecessary stalls of other harts in the pipe-
line that are independent from the MFU being busy.
The custom instruction extension supported by
the MFU and LSU is summarized in Table 1. The
instructions implement vector operations without
relying on a vector register file, but rather on a mem-
ory space mapped on the local SPMs, for maximum
flexibility. The programmer can move vector data at
any point of the SPM address space with no constraint
except the total capacity of the SPMs, which in turn is
a parameter of the microarchitecture design.
The coprocessor instructions are exposed to the
programmer as very simple intrinsic functions, fully
integrated into the RISC-V GCC compiler toolchain.
HARDWARE ACCELERATION
SCHEMES
The MFU and SPMs are accessed through a
scratchpad-memory interface (SPMI). The user can
66 IEEE Micro
configure the number of parallel lanes D in the
MFU, the number of MFUs F, the SPM capacity, the
number of SPMs N, the number of SPMIs M, and
the sharing scheme of MFUs, and SPMI among
harts. The MFU is the engine that accelerates vec-
tor computations. It can operate on different inte-
ger data element widths (8, 16, 32bits) in subword-
SIMD fashion, and also in element-SIMD fashion
when D is configured to multiply the execution
lanes for DLP. A typical vector arithmetic operation
has an initial latency between 4 and 8 cycles to
access the SPM.
Each SPM has one read and one write port. The
parameter D that defines the MFU lanes also corre-
sponds to the number of SPM banks; all the banks of
an SPM are accessed together as a single SPM line.
When the MFU executes a vector operation, it
fetches an entire SPM data line in every clock cycle,
composed of multiple vector elements. A bank read
rotator aligns the source operands coming from the
SPM line, and a bank write rotator aligns the destina-
tion data to the correct banks in an SPM line. When
the LSU fills the SPM banks with data from the 32-bit
data memory port, a bank interleaver switches
between the banks. The reader may refer to the work
by Botman et al.14 for internal details of the units
inside the MFU and SPMs.
Furthermore, the coprocessor can be configured
to implement the following sharing schemes among
harts.
March/April 2021
 GENERAL INTEREST

TABLE 1. Custom vector instruction extension. coprocessor instructions, provided they use different
internal functional units of the MFU (e.g., adder, multi-
Assembly syntax – (r) denotes Short description plier). Harts requesting a busy internal unit in the MFU
memory addressing via register r
kmemld (rd),(rs1),(rs2) load vector into get stalled until the contended unit becomes free.
scratchpad region This scheme can exploit DLP by multilane SIMD exe-
kmemstr (rd),(rs1),(rs2) store vector into main cution, and also TLP in the form of a heterogeneous
memory
MIMD execution.
kaddv (rd),(rs1),(rs2) adds vectors in
scratchpad region The explored design parameters and correspond-
ksubv (rd),(rs1),(rs2) subtract vectors in ing configurations, for reference in reporting perfor-
scratchpad region mance results, are the following:
kvmul (rd),(rs1),(rs2) multiply vectors in
scratchpad region
kvred (rd),(rs1) reduce vector by › M ¼ 1, F ¼ 1, D ¼ 1: SISD
addition › M ¼ 1, F ¼ 1, D ¼ 2,4,8: Pure SIMD
kdotp (rd),(rs1),(rs2) vector dot product into
register
› M ¼ 3, F ¼ 3, D ¼ 1: Symmetric MIMD
ksvaddsc (rd),(rs1),(rs2) add vector þ scalar › M ¼ 3, F ¼ 3, D ¼ 2,4,8: Symmetric MIMD þ SIMD
into scratchpad › M ¼ 3, F ¼ 1, D ¼ 1: Heterogenous MIMD
ksvaddrf (rd),(rs1),rs2 add vector þ scalar › M ¼ 3, F ¼ 1, D ¼ 2,4,8: Heterogenous MIMD þ SIMD
into register
ksvmulsc (rd),(rs1),(rs2) multiply vector þ
scalar into scratchpad We use N ¼ 3 in MatMul and convolutions, and
ksvmulrf (rd),(rs1),rs2 multiply vector þ N ¼ 4 in FFT.
scalar into register Finally, we refer to the T13 microarchitecture con-
kdotpps (rd),(rs1),(rs2) vector dot product and figured with no hardware acceleration as Klessydra
post scaling
ksrlv (rd),(rs1),rs2 vector logic shift within T03.
scratchpad
ksrav (rd),(rs1),rs2 vector arithmetic shift
within scratchpad PERFORMANCE RESULTS
krelu (rd),(rs1) vector ReLu within
scratchpad We run a set of test programs composed of 2-D convo-
kvslt (rd),(rs1),(rs2) compare vectors and lution, FFT, and MatMul kernels. We adopted the
create mask vector widely used 3  3 filter size on matrix sizes of 4  4, 8
ksvslt (rd),(rs1),rs2 compare vector-scalar
and create mask
 8, 16  16, and 32  32 elements for convolutions.
kvcp (rd),(rs1) copy vector within FFT was run on 256 samples, and MatMul on 64  64
scratchpad region element matrices. The element width was kept 32 bit
in fixed-point representation. The tests were organized
as follows:
Shared coprocessor: All the harts share a single
MFU/SPM subsystem. In the case of busy MFU, any › homogeneous workload, running multiple instan-
hart wanting to access it is stalled until the MFU ces of the same kernel on multiple harts, on dif-
becomes free. In this scheme, parallel execution may ferent input data;
occur between coprocessor and noncoprocessor › composite workload, running convolutions, FFTs,
instructions. Yet, the MFU/SPM may exploit pure DLP and MatMul repeatedly on three respective
acceleration, by multilane SIMD execution. harts.
Thread-dedicated coprocessors: A complete
MFU/SPM subsystem is appointed to each hart, elimi- The performance was measured by taking the
nating coprocessor contention. Stalls can only happen average cycle count to execute one computation ker-
if two instructions of the same hart request MFU oper- nel. Table 2 summarizes the results, which are dis-
ation. This scheme can exploit DLP by multilane SIMD cussed below.
execution and TLP by fully symmetric MIMD execu- Cycle count: With small matrix convolutions, the
tion, allowing multiple vector instructions to execute accelerated core reached up to 3  cycle count
in parallel. speed-up over a nonaccelerated IMT core (Klessydra
Thread-dedicated SPMI/shared MFU: A dedicated T03), and 2 speed-up over single-threaded, DSP-
SPM address space is kept for each hart, while the extended core (RI5CY9).
harts share one MFU at the functional unit level. This As expected, large matrix convolutions and Mat-
scheme still allows interhart parallel execution of Mul obtain more considerable advantage from vector-

March/April 2021 IEEE Micro 67

GENERAL INTEREST
TABLE 2. Summary of performance results and synthesis results. Green ¼ best case; red ¼ worst case.
accelerated cores, quantified in 13 cycle count
speed-up relative to Klessydra T03, 9 relative to the
RI5CY core, and 19 relative to ZeroRiscy. In contrast,
FFT takes benefit from TLP and reduced data memory
accesses rather than from DLP.
Figure 2 quantifies the contribution of DLP and TLP
for convolutions on different matrix sizes. For small
vectors, TLP inherently exhibits better contributions
to speed-up than DLP, while as the vector size grows,
the DLP boost dominates. Implementations exploiting
both TLP and DLP performed much better than pure
DLP also with large matrices. A key outcome is that a
single core IMT processor can exploit both DLP and
TLP and follow the gray curve, while a single-threaded
core exploiting only DLP acceleration follows the blue
curve.
Notably, the heterogeneous MIMD coprocessor,
which has three times less functional units than the
fully symmetric MIMD, employed only 1% – 7% more
cycles than the latter.
FIGURE 2. DLP and TLP cycle-count boost in 2-D convolu-
tions for different matrix sizes.
68 IEEE Micro
Maximum clock frequency: All the cores under
analysis were implemented as FPGA soft-cores. The
clock speed exhibited the sharpest drops as the TLP
grew larger: in the heterogeneous MIMD scheme, the
crossbar mapping the SPMI output data on the shared
MFU units became the critical path for D ¼ 4,8. Pipe-
lining the crossbar to reduce the critical path introdu-
ces hardware overhead, compromising the area
advantage of the heterogeneous MIMD configuration.
Absolute execution time: The cycle count and
the operating frequency allow calculating the total
execution time. Figure 3 compares the actual execu-
tion time speed-up relative to the ZeroRiscy core
taken as the reference when each core operates at
its maximum frequency. In pure SIMD configurations,
the speed-up grows linearly with the DLP for the
explored DLP range. Yet, exploiting TLP, by going
from a SISD/SIMD to symmetric and heterogenous
MIMD, improved the speedup in all cases, despite
the frequency drop associated with the MIMD copro-
cessor. Thanks to exploiting both TLP and DLP, the
symmetric MIMDþSIMD schemes exhibit the lowest
execution times, reaching up to 17 speed-up over Zer-
oriscy for Convolution 32  32 and up to 13 speed-up
for the composite workload. Notably, the heteroge-
neous MIMD configurations maintain an almost perfect
overlap with the symmetric MIMD.
The nonaccelerated Klessydra-T03, while employ-
ing a higher cycle count than RI5CY due to the
absence of DSP and hardware-loop extensions, exhib-
its an absolute performance advantage over RI5CY
thanks to a more than double frequency attained by
the pure IMT microarchitecture. When compared to
ZeroRiscy, T03 exhibits both lower cycle count and
higher frequency.
March/April 2021

FIGURE 3. Execution time speed-up with respect to Zeroriscy
core, taken as reference. For the composite test the average
kernel speed-up is reported.
Hardware Resource Utilization: In cost-con-
strained applications, it is crucial to find an optimal
balance between speed-up and area overhead. The
heterogenous MIMD þ SIMD scheme with D ¼ 2
resulted to be a possible best choice with all test
programs.
The nonaccelerated T03 exhibits only a slightly
more significant footprint than the tiny ZeroRiscy
core, despite the replicated register file to support
multithreading, thanks to the LUT-RAM implementa-
tion of the registers.
Energy Efficiency: The average energy per algo-
rithmic operation (multiplications and additions) is
a general measure of the energy efficiency attained
by a processor core in implementing an algorithm
computation. Figure 4 reports the outcome of this
analysis, referring to the soft-core implementations.
TABLE 3. Higher order filter evaluation results for cycle count, total time at max frequency, and total energy. Green ¼ best case;
red ¼ worst case.
March/April 2021
GENERAL INTEREST
FIGURE 4. Average energy per algorithmic operation, normal-
ized to the case of the Zeroriscy soft-core, taken as
reference.
The results are presented as the reduction in nJ/op
relative to Zeroriscy, taken as reference, which
exhibited 4.24 nJ/op as the best case in the ana-
lyzed workloads.
The most energy efficient designs resulted to be
the symmetric MIMD and heterogenous MIMD
schemes, again exhibiting an almost complete overlap
and reaching over 85% energy saving related to the
reference Zeroriscy. Despite having the smallest area
footprint, the pure SIMD schemes resulted in a larger
energy consumption, due to low exploitation of TLP.
Larger Filters: Convolutional neural networks pri-
marily employ 3 3 filters (VGG16) but also larger ones
(e.g., 11  11 in Alexnet, 5  5 in Googlenet). Large
masks such as 7  7 are used in Sobel, Gaussian
smoothing, median filtering. We evaluated the vector
coprocessor schemes with filters ranging from 5  5
IEEE Micro 69
GENERAL INTEREST
to 11  11, on 32  32 element matrices. Table 3 shows
the speed-up and energy efficiency trends continue as
the filter dimensions grow larger, favoring higher DLP.
The improvement referring to ZeroRiscy grows up to
35  when using 11 11 filters.
The symmetric and heterogeneous MIMDþSIMD
schemes, with D ¼ 2, maintain similar performance
and energy results throughout the analyzed cases.
The results confirm that an IMT core capable of MIMD
acceleration increasingly performs better than a sin-
gle-thread SIMD acceleration.
CONCLUSION
The scientific outcome of this article can be summa-
rized in the following list of evidence:
› The MIMD-SIMD vector coprocessor schemes
enable tuning the TLP and DLP contribution and
obtain the best results in absolute performance
and energy efficiency, reaching >15 speed-up
and -85% energy per operation.
› Kernels that are less effectively vectorizable can
still benefit from acceleration through SPMs and
TLP, in an IMT core, reaching 2  -3  speed-up.
› Fully symmetric and heterogeneous MIMD give
very similar results, showing that coprocessor
contention can be effectively mitigated by func-
tional unit heterogeneity, allowing hardware
resource saving. From the same observation, we
can state that functional unit contention is less
impacting than SPM contention, in all the
kernels.
› Pure DLP acceleration always gives inferior
results than a balanced TLP/DLP acceleration.
An IMT microarchitecture can benefit from TLP
and DLP acceleration in a single core.
› In the absence of hardware acceleration, IMT
still exhibits an absolute performance advantage
over single-thread execution thanks to the sim-
plified hardware structure.
The Klessydra-T parametric cores are available as
open source designs on GitHub at https://perma.cc/
6FYD-AF68.
REFERENCES
1. C. Bechara et al., “A small footprint interleaved
multithreaded processor for embedded systems,” in
Proc. 18th IEEE Int. Conf. Electron., Circuits, Syst., 2011,
pp. 685–690.
70 IEEE Micro
2. A. Cheikh et al., “The microarchitecture of a multi-
threaded RISC-V compliant processing core family for
IoT end-nodes,” in Proc. Int. Conf. Appl. Electron.
Pervading Ind., Environ. Soc., 2017, pp. 89–97.
3. M. Olivieri et al., “Investigation on the optimal
pipeline organization in RISC-V multi-threaded soft
processor cores,” in Proc IEEE New Gener. CAS, 2017,
pp. 45–48.
4. RISC-V Unprivileged Instruction Set specifications,
December, 2019. [Online]. Available: https://riscv.org/
specifications/
5. F. Samie, L. Bauer, and J. Henkel, “From cloud down to
things: An overview of machine learning in Internet of
Things,” IEEE Internet Things J., vol. 6, no. 3,
pp. 4921–4934, Jun. 2019.
6. F. Luo and C. J. Zhang, Signal Processing for 5G:
Algorithms and Implementations. New York, NY, USA:
Wiley, 2016.
7. K. Lim, S. Jeong, Y. Kim, and H. S. Yang, “CalmRISCTM : A
low power microcontroller with efficient coprocessor
interface,” Microprocessors Microsyst., vol. 25,
pp. 247–261, 2001.
8. F. Botman, J. deVos, S. Bernard, F. Stas, J. Legat,
and D. Bol, “Bellevue: A 50MHz variable-width SIMD
32bit microcontroller at 0.37V for processing-
intensive wireless sensor nodes,” in Proc. IEEE Int.
Symp. Circuits Syst., Melbourne, Australia, 2014,
pp. 1207–1210.
9. M. Gautschi et al., “Near-threshold RISC-V core with
DSP extensions for scalable IoT endpoint devices,” IEEE
Trans. Very Large Scale Integr. (VLSI) Syst., vol. 25,
no. 10, pp. 2700–2713, Oct. 2017.
10. S. Seo et al., “Diet SODA: A power-efficient processor
for digital cameras,” in Proc. 16th ACM/IEEE Int. Symp.
Low Power Electron. Des., 2010, pp. 79–84.
11. F. Conti and L. Benini, “A ultra-low-energy convolution
engine for fast brain-inspired vision in multicore
clusters,” in Proc. IEEE Des., Autom. Test Eur. Conf.
Exhib., 2015, pp. 683–688.
12. Y. H. Chen, T. Krishna, J. S. Emer, and V. Sze,
“Eyeriss: An energy-efficient reconfigurable
accelerator for deep convolutional neural networks,”
IEEE J. Solid-State Circuits, vol. 52, no. 1, pp. 127–138,
Jan. 2017.
13. L. Du et al., “A reconfigurable streaming deep
convolutional neural network accelerator for Internet
of Things,” IEEE Trans. Circuits Syst. I, Reg. Papers,
vol. 65, no. 1, pp. 198–208, Jan. 2018.
14. A. Cheikh et al., “Efficient mathematical accelerator
design coupled with an interleaved multi-threading
RISC-V microprocessor,” in Proc. Int. Conf. Appl.
Electron. Pervading Ind., Environ. Soc., 2019,
pp. 529–539.
March/April 2021

ABDALLAH CHEIKH is currently a Postdoctoral Researcher
with Sapienza University of Rome, Italy. His interest in com-
puter organization, and design drove him to pursue a PhD fel-
lowship at Sapienza University of Rome majoring in computer
architecture. His research activities cover design, implemen-
tation, and verification of a wide range of microprocessor
architectures, vector accelerators, and morphing processors.
Cheikh received a B.S. and an M.S. in electrical engineering
from Rafik Hariri University, Lebanon, in 2014 and 2016,
respectively, and a Ph.D. in 2020. Contact him at abdallah.
cheikh@uniroma1.it
STEFANO SORDILLO has been a Doctorate student with the
Sapienza University of Rome, Italy, since 2019. His research
activities cover microprocessor core design, hardware acceler-
ators, and neural network algorithms for IoT devices. Sordillo
received an M.S. (Laurea) “cum laude” in electronics engineer-
ing from Sapienza University of Rome, Italy, in 2019. Contact
him at stefano.sordillo@uniroma1.it.
ANTONIO MASTRANDREA is a Research Assistant with the
Department of Information Engineering, Electronics and Tele-
communications, Sapienza University of Rome, Italy. His cur-
rent research interests include digital system-on-chip
architectures and nano-CMOS circuits oriented to high-speed
computation. Mastrandrea received an M.S. (Laurea) (cum
laude) in electronics engineering and a Ph.D. from the Sapi-
enza University of Rome, in 2010 and 2014, respectively. Con-
tact him at antonio.mastrandrea@uniroma1.it.
FRANCESCO MENICHELLI is currently an Assistant Profes-
sor with Sapienza University of Rome, Italy. His scientific
interests focus on low-power digital design and, in particular,
in system level and architectural level techniques for low-
power consumption, power modeling, and cosimulation of
digital system platforms. Menichelli received an M.S. in
March/April 2021
GENERAL INTEREST
electronic engineering “cum laude” in 2001 and a Ph.D. in
electronic engineering from the University of Rome “La Sapi-
enza” in 2005. He coauthored more than 40 publications in
international journals and conference proceedings. Contact
him at francesco.menichelli@uniroma1.it.
GIUSEPPE SCOTTI became a Researcher (Assistant Profes-
sor) with the DIET Department, University of Rome ”La Sapi-
enza,” Italy, in 2010 and became an Associate Professor with
the same department in 2015. His research activity was
mainly concerned with integrated circuits design and
focused on design methodologies able to guarantee robust-
ness with respect to parameter variations in both analog cir-
cuits and digital VLSI circuits. Scotti received an M.S. and a
Ph.D. in electronic engineering from the University of Rome
“La Sapienza,” in 1999 and 2003, respectively. He has coau-
thored about 50 publications in international journals and
more than 70 contributions in conference proceedings. Con-
tact him at giuseppe.scotti@uniroma1.it.
MAURO OLIVIERI joined Sapienza University of Rome, Italy,
as an Associate Professor in 1998. He is a Visiting
Researcher with the Barcelona Supercomputing Center
within the European Processor Initiative Project. His
research interests include microprocessor core design and
digital nano-scale circuits. He authored more than 100
papers and a textbook in three volumes on digital VLSI
design. Olivieri received an M.S. (Laurea) “cum laude” in 1991
and a Ph.D. in electronics and computer engineering from
the University of Genoa, Italy, in 1994, where he was an
Assistant Professor from 1995 to 1998. He has been a mem-
ber of the TPC of IEEE DATE and was General Co-Chair of
IEEE/ACM ISLPED’15. He is an Evaluator for the European
Commission in the ECSEL Joint Undertaking. He is a Senior
Member of IEEE. Contact him at mauro.olivieri@uniroma1.it.
IEEE Micro 71
 EDITOR: Michael Mattioli, michael.mattioli@gs.com

DEPARTMENT: SECURITY

Hidden Potential Within Video Game

Consoles
Michael Mattioli and Atte Lahtiranta, Goldman Sachs & Co., New York , NY, 10282, USA

Video game consoles share many of the characteristics of an ideal device for use in
enterprise deployments. In comparison to many desktop and notebook PCs
available in the market, modern video game consoles are actually quite powerful
and capable. They provide an excellent user experience with simple and intuitive
setup and operation. At the heart of the design of many modern video game
consoles is security; they are remarkably resilient against very sophisticated
hardware and software attacks. They are also rather cost-effective in comparison
to modern PCs.

V
ideo game consoles are ideal devices for
enterprise deployments; they are powerful,
versatile, easy to use, cost-effective, and
extremely secure. Systems suitable for use in an enter-
prise must be able to handle a variety of workloads to
ensure that users remain productive; such workloads
include, but are not limited to, video conferencing,
web browsing, content creation (e.g., spreadsheets,
presentations, documents, audio, video, etc.), and
audio/video streaming. Security is equally as impor-
tant, if not more important, than user productivity;
enterprise systems routinely process sensitive infor-
mation, which is critical to the organization’s success.
Of course, all enterprise systems strive to reduce over-
all total cost of ownership (TCO); this can be achieved
by simply reducing the cost of the hardware itself as
well as reducing the operational expenses (OpEx)
incurred by managing and maintaining the systems
(e.g., patching, updates, user training/support, etc.).
The purpose of this article is to describe the ideal
characteristics of a device suitable for use in enter-
prise deployments and demonstrate how video game
consoles are designed with these characteristics and
traits in mind, which, therefore, makes them an excel-
lent fit. It is extremely important to understand that
while this article may use specific consoles as
0272-1732 ß 2021 IEEE
Digital Object Identifier 10.1109/MM.2021.3055681
Date of current version 26 March 2021.
examples of such devices many of the points being
made apply to almost all modern video game consoles
including, but not limited to, the Nintendo Switch, the
Sony PlayStation 4, the Sony PlayStation 5, and the
Microsoft Xbox Series S.
PERFORMANCE
Relative to desktop and notebook PCs available at
their time of release, modern video game consoles
are quite powerful in comparison. As described in
Table 1, video game consoles feature the latest tech-
nologies available at their respective times of
release. Unsurprisingly, performance is critical when
considering devices for use in the enterprise. It is
fairly common for enterprise users to be working
across multiple applications and contexts at once;
high-performance devices aid user workflow and
productivity.
THE PURPOSE OF THIS ARTICLE IS TO
DESCRIBE THE IDEAL
CHARACTERISTICS OF A DEVICE
SUITABLE FOR USE IN ENTERPRISE
DEPLOYMENTS AND DEMONSTRATE
HOW VIDEO GAME CONSOLES ARE
DESIGNED WITH THESE
CHARACTERISTICS AND TRAITS IN
MIND, WHICH, THEREFORE, MAKES
THEM AN EXCELLENT FIT.

72 IEEE Micro Published by the IEEE Computer Society March/April 2021

 SECURITY

TABLE 1. Hardware details of various video game consoles released over several years.

System Release Date CPU Memory Process CPU Microarch. GPU Microarch.
Cores Tech.
Microsoft Xbox One November 2013 8x x86-64 8 GB DDR3 TSMC 28 nm AMD Jaguar AMD GCN 2
Sony PlayStation 4 November 2013 8x x86-64 8 GB GDDR5 TSMC 28 nm AMD Jaguar AMD GCN 2
Microsoft Xbox One S August 8x x86-64 8 GB DDR3 TSMC 28 nm AMD Jaguar AMD GCN 2
2016
Sony PlayStation 4 November 2016 8x x86-64 8 GB GDDR5 TSMC 28 nm AMD Jaguar AMD GCN 4
Pro
Nintendo Switch March 2017 4x ARMv8 4 GB TSMC 20 nm ARM Cortex- NVIDIA
LPDDR4 A57 Maxwell
Microsoft Xbox One X November 8x x86-64 12 GB GDDR5 TSMC 16FF+ AMD Jaguar AMD GCN 4
2017
Microsoft Xbox November 8x x86-64 16 GB TSMC N7e AMD Zen 2 AMD RDNA 2
Series X 2020 GDDR6
Microsoft Xbox November 8x x86-64 10 GB TSMC N7e AMD Zen 2 AMD RDNA 2
Series S 2020 GDDR6
Sony PlayStation 5 November 8x x86-64 16 GB TSMC N7e AMD Zen 2 AMD RDNA 2
2020 GDDR6

The Microsoft Xbox One X features a semicustom
system on a chip (SoC) developed in partnership with
Microsoft and advanced microdevices (AMD).1 The
SoC is implemented using Taiwan Semiconductor
Manufacturing Company’s (TSMC) 16-nm Fin Field-
effect transistor (FinFET) Plus (16FF+) technology; it
FIGURE 1. Microsoft Xbox Series X video game console and
controller.3
features a CPU composed of 8 64-bit x86 cores oper-
ating at 2.3 GHz and a GPU composed of 40 com-
pute units operating at 1.172 GHz. The SoC uses a
unified memory pool, shared by both the CPU and the
GPU, which consists of 12 GB of GDDR5 DRAM; the
total memory bandwidth is 326.4 GB/s. The console
supports HDMI 2.0b display output with high-band-
width digital content protection (HDCP) 2.2, 10-bit
HDR, and a resolution of 3840 2160 at 60 Hz. The
GPU is further optimized for a version of Microsoft’s
DirectX 12 graphics API specific to the system.
The Microsoft Xbox Series X, shown in Figure 1, fea-
tures a semicustom (SoC) developed in partnership
with Microsoft and AMD.2
The SoC is implemented using TSMC’s 7 nm FinFET
Enhanced (N7e) technology; it features a CPU com-
posed of 8 64-bit x86 cores operating at 3.8 GHz, or
3.6 GHz with simultaneous multithreading (SMT)
enabled, and a GPU composed of 52 compute units
operating at 1.825 GHz. The SoC uses a unified memory
pool, shared by both the CPU and the GPU, which con-
sists of 16 GB of GDDR6 DRAM; 10 GB, reserved for the
GPU, operate at 560 GB/s and 6 GB, reserved for the
CPU, operate at 336 GB/s. The console supports HDMI
2.1 display output with the same features as the Xbox
One X SoC in addition to fixed rate link (FRL), variable
refresh rate (VRR), display stream compression (DSC),
4:4:4 chrome subsampling, and a resolution of either
3840  2160 at 120 Hz or 7680  4320 at 60 Hz. The GPU
is further optimized for a version of Microsoft’s DirectX
12 Ultimate graphics API specific to the system.

March/April 2021 IEEE Micro 73

SECURITY
For comparison, the Intel Core i5-8600 T, an Intel
eighth-generation desktop SoC released several
months after the Xbox One X, features a CPU com-
posed of 6 64-bit x86 cores operating at a base fre-
quency of 2.3 GHz and a GPU operating at 1.15 GHz;
this SoC is implemented using Intel’s third-generation
14 nm++ technology.4 It supports HDMI display output
with a resolution of 4096  2304 at 24 Hz; it also sup-
ports the mainstream version of Microsoft’s DirectX
12 graphics API.
It is important to understand that while these SoCs
differ drastically in regards to cost, intended use, pro-
cess technology, thermal design power (TDP), instruc-
tions per cycle (IPC), and various other aspects of SoC
and CPU and GPU design, the intent here is to high-
light that video game consoles are powerful and fea-
ture-rich in comparison to mainstream compute
devices available at the time of their release.
USER EXPERIENCE
Video game consoles provide an elegant and engaging
experience for all ages and skill levels. This character-
istic is ideal for enterprises, because it enables all of
its users to be functional and productive without
requiring additional training, learning, etc.
Ease of Use
Content (games) aside, the systems are designed for
both children and adults with limited experience or
understanding of technology. Only two cables are
required to use the system: power and display (usually
HDMI). User input and haptic feedback is performed
through an ergonomically designed controller; how-
ever, modern systems also support traditional key-
board and mouse input.5 System and software (game)
updates and purchases are obtained through a single
source (e.g., Microsoft Store, Nintendo eShop, Sony
PlayStation Store, etc.), which is tightly integrated into
the system making it easy and intuitive to find, pur-
chase, and download software.
Versatility
The systems are also rather versatile; developers can
write and release a variety of different software titles.
Aside from the obvious (games), this includes, but is
not limited to, video streaming applications, music
streaming applications, video conferencing applica-
tions, web browsers, and cloud storage applications.
For example, Hulu is available on the Nintendo Switch
via the Nintendo eShop and Spotify is available on the
PlayStation consoles via the Sony PlayStation Store.
Enterprises have the ability to write their own custom
74 IEEE Micro
internal applications and independent software ven-
dors (ISVs) can also develop and distribute applica-
tions for public consumption.
SECURITY
At the heart of their design is thoughtful and practical
defense against a wide range of threats. Without ques-
tion, security is paramount in enterprise contexts. It is
common for users to access and store sensitive infor-
mation, which is crucial to the success and well-being
of the organization and its stakeholders (e.g., employ-
ees, customers, clients, shareholders, etc.).
Identity and Access Management
Centralized identity and access management (IAM) is
used throughout the entire ecosystem. The Microsoft
consoles require users authenticate using a Microsoft
account; similarly, the Sony consoles require a Sony
account and the Nintendo consoles require a Nin-
tendo account. These identities are then used for
access and privilege management. Identity is required
to associate and maintain licenses for software
(games and applications) and subscription services. It
is also used to control communication and interaction
with other users; access to user information (e.g.,
online status, currently running software, etc.) and
interactions (e.g., text messages, voice messages, in-
game chat, etc.) can be explicitly granted to or
revoked from other users.
Patching and Updates
The systems are designed such that only fully patched
and updated systems and software can access pro-
tected resources (e.g., Xbox LIVE, Nintendo Online,
game servers, etc.) and interact with other compliant
(patched and updated) systems and users.
Upon boot, the system attempts to connect to sin-
gle, trusted authority (e.g., Xbox LIVE, PlayStation Net-
work, etc.). Upon connection, it then checks for any
system updates and prompts the user to download
and install them. If the user chooses to skip/decline
any pending updates or if a connection to the trusted
authority cannot be made (for whatever reason), they
can continue to use the system offline and use soft-
ware (games) that is already installed. Simply, the sys-
tem will not allow a user to connect to the trusted
authority (e.g., to play games, to chat with friends,
etc.) unless the system is fully patched and updated.
If the system is fully patched and updated but the
software (game) the user wishes to launch is not fully
updated, the user is prompted to download and install
any pending updates. If the user chooses to skip/
March/April 2021
 SECURITY

decline any pending updates, they can continue to use

the software offline; in other words, they cannot inter-
act (play) with other users online using that software
unless it is fully patched and updated.

Hardware Security
Video game consoles are designed to be remarkably
resilient against various hardware and software
attacks. The entire business model of a modern video
game console is centered around software sales, not
hardware sales. They are designed around the premise
that the end-user cannot be trusted; an end-user’s
motivation is to play games for free (piracy) and/or
modify the game to achieve an unfair advantage over FIGURE 3. High-level depiction of the Xbox One X SoC
other players (cheat). Therefore, extreme measures architecture.6
must be taken to prevent physical attacks against the
system. However, the end-user is not the only
untrusted entity. Xbox One X and NVMe in the Xbox Series X). Keys
The Xbox One X is an excellent example of such used to decrypt information are fed into the SCE
design;6 many of these security features have been through a dedicated hardware pin connecting it to the
carried forward to the Xbox Series X. Quite literally, Crypto Engine inside of another custom-designed ele-
the only trusted entity of the entire Xbox One X sys- ment within the SoC referred to as the Security Com-
tem is the SoC itself; the internal storage, DRAM, opti- plex; this ensures that the keys are never exposed to
cal drive, etc., are considered untrusted. Therefore, all software at any point in time. This Security Complex
information which leaves the SoC must be encrypted also closely monitors the system clock, voltage, tem-
and all information which enters the SoC must be perature, and reset; these are commonly manipulated
decrypted and integrity checked. to attack a system.
All data are stored in nonvolatile memory using a One of the core tenets of the console’s security
format known as the Xbox Virtual Disk (XVD). As illus- design is defense in depth; in other words, an attacker
trated in Figure 2, all data are stored in an NT File Sys- must break through multiple layers of security. In addi-
tem (NTFS) virtual disk and then encrypted and tion to encrypting and integrity checking all informa-
hashed (for both confidentiality and integrity); finally, tion, which passes through the SoC, the system uses a
the root digest of the hash tree is signed using Micro- three-OS architecture,7 as illustrated in Figure 4.
soft RSA (for integrity of the hash tree itself).
The system SoC, illustrated in Figure 3, features a
custom-designed element referred to as the Stream-
ing Crypto Engine (SCE), which is able to decrypt infor-
mation loaded from the internal storage as fast as it
can be read from the underlying I/O bus (SATA III in

FIGURE 2. High-level depiction of an Xbox Virtual Disk (XVD) FIGURE 4. High-level depiction of the Xbox One X three-OS
structure.6 architecture.6

March/April 2021 IEEE Micro 75

SECURITY

TABLE 2. Cost comparison of various video game consoles at For comparison, USD 499 spent today can buy a
their time of release. Lenovo ThinkCentre M720q, which includes 128 GB
of internal storage, an Intel Pentium Gold G5400T
System Release Internal Cost SoC, and 4 GB of DDR4 DRAM.8 Clearly, the consoles
Date Storage (USD) are rather competitively priced compared to modern
Microsoft Xbox November 500 GB 499 PCs. However, the true cost of any hardware deploy-
One 2013 ment in the enterprise extends far beyond the
Sony PlayStation 4 November 500 GB 399 device itself. One must consider patching, mainte-
2013 nance, and management of the device throughout
Microsoft Xbox August 500 GB 299 its entire lifecycle. Considering that patches and
One S 2016 software are released and distributed directly
Sony PlayStation 4 November 1 TB 399 through the trusted authority (e.g., Microsoft via
Pro 2016 Xbox LIVE, Nintendo via Nintendo Online, etc.), there
Nintendo Switch March 32 GB 299 is less operational overhead for an enterprise which
2017 would otherwise have to build its own infrastructure
Microsoft Xbox November 500 GB 499 to do so.
One X 2017
Microsoft Xbox November 1 TB 499
Series X 2020 THE NOTION OF USING A VIDEO GAME
CONSOLE IN THE ENTERPRISE MAY
Microsoft Xbox November 512 GB 299
Series S 2020 SEEM LAUGHABLE AT FIRST GLANCE.
HOWEVER, AS DISCUSSED, VIDEO
Sony PlayStation 5 November 825 GB 399
w/o optical drive 2020 GAME CONSOLES ACTUALLY EMBODY
MANY OF THE CHARACTERISTICS OF
Sony PlayStation 5 November 825 GB 499
w/optical drive 2020 AN IDEAL DEVICE FOR USE IN THE
ENTERPRISE.

The Host Operating System (OS) contains all of the

drivers and necessary components for interacting
with and communicating with the underlying hard-
ware. The Software OS contains the game or applica-
tion that is running in the foreground. The System OS
is responsible for everything outside of the software
title itself such as networking communication and
drawing any user interface elements outside of the
software. As illustrated in Figure 4, the majority of the
system memory is allocated to the Software OS.
The Hypervisor is responsible for enforcement of the
signed code. Using the stage-2 CPU MMU, it will verify
that every page of memory, which is marked for execu-
tion, is signed by Microsoft; if it is not signed, it will not
be executed. This architecture not only allows devel-
opers to create more engaging experiences, it further
strengthens the overall security of the system by
layering and separating software components.
COST
As described in Table 2, video game consoles, at
the time of their respective times of release, have
been priced at or below USD 499 for the last sev-
eral years.
CONCLUSION
The notion of using a video game console in the enter-
prise may seem laughable at first glance. However, as
discussed, video game consoles actually embody
many of the characteristics of an ideal device for use
in the enterprise: high performance, excellent user
experience, versatility, cost-effective, and secure at its
core. Technologies from video game consoles are
already trickling into the enterprise. Security technolo-
gies found in some of the latest AMD CPUs such as
Secure Memory Encryption (SME) and Secure
Encrypted Virtualization (SEV) originated in the design
of the Xbox One X.9 Hypervisor-Protected Code Integ-
rity (HVCI), a feature now generally available in Hyper-
V for Windows, originated in the design of the hypervi-
sor signed code enforcement technology previously
discussed. Microsoft’s recently announced Pluton
security processor originated in the SoC previously
discussed.10 Other recently released consoles such as
the Microsoft Xbox Series S and the Sony PlayStation
5 share the same characteristics and carry these traits
forward; performance has improved with adoption of
NVMe internal storage11 and semicustom variants of

76 IEEE Micro March/April 2021


AMD’s Zen 2 CPU cores and RDNA 2 GPU cores while
security has improved simply because threats (piracy
and cheating) and attacks only become more
advanced over time.
REFERENCES
1. J. Sell, “The xbox one x scorpio engine,” IEEE Micro,
vol. 38, no. 2, pp. 53–60, Mar./Apr. 2018.
2. J. Andrews and M. Grossman, “Xbox Series X System
Architecture,” Aug. 2020. Accessed: Jan. 24, 2021.
[Online]. Available: https://hotchips.org/assets/program/
conference/day1/HotChips2020_GPU_Microsoft_Jeff
_Andrews_v2.pdf
3. W. Commons, “File:xbox Series X 2.jpg,” Dec. 2020.
Accessed: Jan. 24, 2021. [Online]. Available: https://
commons.wikimedia.org/wiki/File:Xbox_Series_X_2.jpg
4. Intel, “Intel Core i5-8600t Processor,” 2020. Accessed:
Sep. 27, 2020. [Online]. Available: https://ark.intel.com/
content/www/us/en/ark/products/129938/intel-core-
i5-8600t-processor-9m-cache-up-to-3-70-ghz.html
5. Microsoft, “Mouse and Keyboard Support on Xbox
One,” 2020. Accessed: Sep. 27, 2020. [Online]. Available:
https://support.xbox.com/help/hardware-network/
accessories/mouse-keyboard
6. T. Chen, “Guarding against physical attacks: The Xbox
one story,” Oct. 2019. Accessed: Sep. 27, 2020. [Online].
Available: https://www.platformsecuritysummit.com/
2019/speaker/chen/
7. E. Brechner, “Getting diagonal on the Xbox one
trifecta,” Computer, vol. 46, no. 8, pp. 77–78, 2013.
March/April 2021
SECURITY
8. Newegg, “Lenovo Thinkcentre m720q 10t8sdj900
Desktop Computer - Pentium Gold g5400t -
4 Gb Ram - 128 Gb Ssd - Tiny - Black,” 2020. Accessed:
Sep. 26, 2020. [Online]. Available: https://www.newegg.
com/p/N82E16883994324
9. CRN, “Amd’s Xbox, Playstation Work Led to a Big
Security Feature in Epyc,” Aug. 2019. Accessed: Aug.
2019. [Online]. Available: https://www.crn.com/news/
components-peripherals/amd-s-xbox-playstation-work-
led-to-a-big-security-feature-in-epyc
10. Microsoft, “Meet the Microsoft Pluton Processor—The
Security Chip Designed for the Future of Windows Pcs,”
Nov. 2020. Accessed: Nov. 23, 2020. [Online]. Available:
https://www.microsoft.com/security/blog/2020/11/17/
meet-the-microsoft-pluton-processor-the-security-
chip-designed-for-the-future-of-windows-pcs/
11. M. Mattioli, “History of video game distribution,” IEEE
Consum. Electron. Mag., vol. 12, no. 3, pp. 312–322,
Sep. 2020.
MICHAEL MATTIOLI leads the Hardware Engineering team
within Goldman Sachs, New York, NY, USA. He is responsible
for the design and engineering of the firm’s digital experien-
ces and technologies. He is also responsible for the overall
strategy and execution of hardware innovation both within
the firm and within the broader technology industry. Contact
him at michael.mattioli@gs.com.
ATTE LAHTIRANTA is currently a Chief Technology Officer
and head of Core Engineering with Goldman Sachs, New
York, NY, USA. Contact him at atte.lahtiranta@gs.com.
IEEE Micro 77
 DEPARTMENT: AWARDS
A Brief History of Warehouse-Scale
Computing
Reflections Upon Receiving the 2020 Eckert-Mauchly Award

Barroso
Luiz Andre , Google, Mountain View, CA, 94043, USA
R
eceiving the 2020 ACM-IEEE Eckert-Mauchly
Award this past June was among the most
rewarding experiences of my career. I am
grateful to IEEE Micro for giving me the opportunity to
share here the story behind the work that led to this
award, a short version of my professional journey so
far, as well as a few things I learned along the way.

THE PRACTICE OF COMPUTER
SCIENCE
For many of us our earliest models of professional-
ism come from observing our parents’ approach to
their work. That was the case for me observing my
father, a surgeon working in public hospitals in Rio
de Janeiro. Throughout his career, he was continu-
ally investigating new treatments, collecting case
studies, participating and publishing in medical con-
ferences, despite never having held an academic or
research position. He was dedicated to the practice
of medicine but always made time to help advance
knowledge in his area of expertise.
Without really being aware of it, I ended up following
my father’s path and became a practitioner myself. As a
practitioner, my list of peer-reviewed publications is
notably shorter than most of the previous winners of
this award, but every time I had something valuable to
share with the academic community, I felt welcomed by
our top research conferences, and those articles tended
to be well received. Practitioners like myself tend to pub-
lish papers in the past tense, reporting on ideas that


Administered jointly by ACM and the IEEE Computer Soci-
ety, the award is given for contributions to computer and digi-
tal systems. In 2020, my award was given for pioneering the
design of warehouse-scale computing and driving it from
concept to industry.
0272-1732 ß 2021 IEEE
Digital Object Identifier 10.1109/MM.2021.3055379
Date of current version 26 March 2021.
78 IEEE Micro Published by the IEEE Computer Society
have been implemented and launched as products.
Practitioners can contribute to our community by look-
ing back and showing us how those ideas played out (or
not) in practical applications. Commercial success or
the lack thereof can be an objective judge of the merits
of research ideas; even if cruelly so at times. In giving
me this award, the IEEE Computer Society and ACM are
highlighting the role of practitioners in our field.
Now, as this award is about the practice of ware-
house-scale computing, I should get to that with no
further delay.
A BRIEF HISTORY OF
WAREHOUSE-SCALE COMPUTING
If it is indeed true that “great poets imitate and
improve,”1 poetry and computing may have something
in common after all. Warehouse-scale computers (the
name we eventually gave to the computers we began to
design at Google in the early 2000s) are the technical
descendents of numerous distributed computing sys-
tems that aimed to make multiple independent com-
puters behave as a single unit. That family begins with
VAXclusters2 in the 1980s, a networked collection of
VAX computers with a distributed lock manager that
attempted to present itself as a single system to the
user. In the 1990s, the concept of computing clusters
began to be explored using lower end or desktop com-
puters and local area networks with systems such as
NASA’s Beowulf clusters3 and UC Berkeley’s NOW
project.4
FOR MANY OF US OUR EARLIEST
MODELS OF PROFESSIONALISM
COME FROM OBSERVING OUR
PARENTS’ APPROACH TO THEIR
WORK. THAT WAS THE CASE FOR ME
OBSERVING MY FATHER, A SURGEON
WORKING IN PUBLIC HOSPITALS IN
RIO DE JANEIRO.
March/April 2021

When I arrived at Google, in 2001, I found a company
of brilliant programmers that was short on cash but not
on confidence as they had already committed to a
strategy of systems built from inexpensive desktop-
class components. Cheap might be a fairer characteri-
zation of those early systems than inexpensive. The
first generation of those computer racks, tenderly nick-
named “corkboards” consisted of desktop mother-
boards loosely resting on sheets of cork that isolated
the printed circuit boards from the metal tray, with disk
drives themselves loosely resting on top of DIMMs.
Despite my hardware background,y I had joined Goo-
gle to try to become a software engineer. In my early
years, I was not involved in building computers but
instead I worked developing our index searching soft-
ware and related software infrastructure components
such as load balancers and remote procedure call librar-
ies. Three years later, Urs Ho€ lzle asked me to build a
hardware team capable not only of building sound
server-class systems but to invent new technologies in
the datacenter space. The years I had spent in software
development turned out to be extremely useful in this
new role since my first-hand understanding of Google’s
software stack was essential to architecting the
machinery needed to run it. We published some of those
early insights into the architectural requirements for
Google-class workloads in an IEEE Micro paper in 2003.6
OUR TEAM’S LACK OF EXPERIENCE IN
DATACENTER DESIGN MAY HAVE
BEEN AN ASSET AS WE SET OUT TO
QUESTION NEARLY EVERY ASPECT OF
HOW THESE FACILITIES WERE
DESIGNED
In our earliest days as a hardware team we focused
primarily on designing servers and datacenter net-
working, but quickly realized that we would need to
design the datacenters themselves. Up until that point
internet companies deployed computing machinery in
third-party colocation facilities (businesses that provi-
sioned space, power, cooling, and internet connectiv-
ity for large scale computing gear), and Google was no
exception. As the scale of our deployments grew, the
minimum footprint required for a Google cluster was
beginning to be larger than the total size of existing
y
My Ph.D. and the earlier phase of my career had been in
computer architecture, particularly in microprocessor and
memory system design.
March/April 2021
AWARDS
co-location facilities, so we had to build our own facili-
ties in order to continue to grow our services.
At that point, it became evident to us how much
room for improvement there was in the design of
datacenters. As a third-party hosting business, data-
centers were put together by groups of disjoint
engineering crafts that knew little of each other’s dis-
ciplines; civil engineers built the building, mechanical
engineers provisioned cooling, electrical engineers dis-
tributed power, hardware designers built servers, soft-
ware engineers wrote internet services. The lack of
cross-disciplinary coordination resulted in facilities
that were both expensive and incredibly energy ineffi-
cient. Our team’s lack of experience in datacenter
design may have been an asset as we set out to ques-
tion nearly every aspect of how these facilities were
designed. Perhaps most importantly we had the
chance to look at the entire system design, from cool-
ing towers to compilers, and that perspective quickly
revealed significant opportunities for improvement.
Speed of deployment was also a critical factor in
those days as we were often running dangerously close
to exhausting our computing capacity as our traffic
grew, so our initial approach was to prefabricate ready-
to-deploy computer rooms inside forty foot shipping
containers. Containers gave us a datacenter floor
where we could isolate the hot (exhaust) plenum from
the cold aisle and shortened the total distance the air
needed to be moved; both were factors that improved
cooling efficiency. All that the container needed to
function was power, cold water and networking, and
we had a 1200-server machine room ready to deploy.
That original container-based deployment also
introduced other innovations that led to cost, perfor-
mance, and energy efficiency improvements. Here are
some of the most notable ones:
› Higher temperature air cooling: We determined
through field experiments that contrary to
common wisdom the electronic components
believed to be most affected by air temperature
were still quite reliable at reasonably high tem-
peratures (think 70F instead of 60F).8 This made
it possible to run many facilities using evapora-
tive cooling and improved cooling efficiency.
› Distributed uninterruptible power supplies (UPS):
Typical datacenters were built with a UPS room
(a room full of batteries) in order to store enough
energy to ride electrical grid glitches. As such ac
voltage was rectified to power the UPS and then
inverted to distribute to the machine room only
then to be rectified again by per-server power
supplies, incurring losses at each transformation
IEEE Micro 79
AWARDS
FIGURE 1. A Google warehouse-scale computer in Belgium.
step. We instead eliminated the UPS room and
introduced per tray (and later per rack) batteries.
That way power entering the building only
needed to be rectified once per machine.
› Single-voltage rail power supplies: Every server
used to be outfitted with a power supply that con-
verted ac voltage into a number of dc voltage rails
(12 V, 5 V 3.3 V, etc.) based on old standards
for electronic components. By 2005, most elec-
tronic components did not use any of the standard
dc rails so yet another set of dc/dc conversions
needed to happen onboard. The allocation of
power among multiple rails also lowered power
supply efficiency sometimes below 70%. We intro-
duced a single-rail power supply that reached 90%
efficiency and created on-board only the voltages
actually used by components.
› 1000-port GigE Ethernet switch: Datacenter net-
working bandwidth was beginning to become a
bottleneck for many warehouse-scale applica-
tions, but enterprise-grade switches were not only
very expensive but also lacked offerings for large
numbers of high bandwidth endpoints. Using a col-
lection of inexpensive edge switches configured
as a multistage network, our team created the first
of a family of distributed datacenter networking
products (codenamed Firehose) that could deliver
80 IEEE Micro
a gigabit of nonoversubscribed bandwidth to up
to a thousand servers.
Although our adventure with shipping containers
lasted only that one generation and soon after we
found ways to obtain the same efficiencies with more
traditional building shells, the innovations from that
first program have continued to evolve into industry-
leading solutions over generations of warehouse-scale
machines. Figure 1 shows a birds-eye view of a modern
Warehouse-scale computer.
MY JOURNEY
I knew I wanted to be an electrical engineer when I was
8 years old and got to help my grandfather work on his
HAM radio equipment. Putting aside the fact that eight-
year-olds should not be making career choices, I find it
difficult to question that decision to this date. Although
I had always been a good student, I struggled a bit dur-
ing my Ph.D. and graduated late. I did have a few things
going for me: an ability to focus, stamina for hard work,
and a lot of luck. As an example, after a 24-year drought
the Brazilian men’s national soccer team chose to win a
World Cup, during my hardest year in graduate school,
delivering a degree of joy that was badly needed to get
me to the finish line. Less than a year after that World
Cup I was working in my grad student office on a
March/April 2021

Saturday afternoon when I got a call from Norm Jouppi
inviting me to interview for a research job at Digital
Equipment’s Western Research Lab (WRL). At the time
Norm was already one of the most highly respected
computer architects in the world and perhaps nothing
in my career since has compared to the feeling I had
that day—Norm Jouppi knew who I was!
I KNEW I WANTED TO BE AN
ELECTRICAL ENGINEER WHEN I WAS
8 YEARS OLD AND GOT TO HELP MY
GRANDFATHER WORK ON HIS HAM
RADIO EQUIPMENT. PUTTING ASIDE
THE FACT THAT EIGHT-YEAR- OLDS
SHOULD NOT BE MAKING CAREER
CHOICES, I FIND IT DIFFICULT TO
QUESTION THAT DECISION TO THIS
DATE.
I joined DEC WRL and had the chance to learn from
top researchers like Kourosh Gharachorloo and collab-
orate with leading computer architects such as Sarita
Adve, Susan Eggers, Mateo Valero, and Josep Lariba-
Pey. During that time, I also met Mark Hill who would
become a friend and a mentor. Later, at Google I
would also have the chance to coauthor papers with
other leading figures in our field such as Tom Wenisch,
Wolf Weber, David Patterson, and Christos Kozyrakis.
Perhaps nothing summarizes the impact that friends
and luck can have in your life more than the story of how
I came to join Google. As I was trying to make a decision
between two options, Jeff Dean asked me whether the
other company I was considering had also served me
creme bru
e during my interviews. I thanked Jeff and
^ le
accepted the Google offer the very next morning.
The brilliance and generosity of countless people at
Google have been essential to the work that led to this
award, but I must highlight three here: Urs Ho € lzle who
has been a close collaborator and possibly the single
person most to blame for Google’s overall systems
infrastructure successes; Bart Sano who managed the
Platforms team that built out the infrastructure we
have today (I was the technical lead for for Bart’s team
for many years); and Partha Ranganathan who is our
computing technical lead today and is taking Google’s
architectural innovation into the future.
One part of my career I have no hesitation to brag
about is the quality of the students I have had a
chance to host as interns at DEC and Google. They
were (to date) Partha Rahganathan, Rob Stets, Jack
March/April 2021
AWARDS
Lo, Sujay Parekh, Ed Bugnion, Alex Ramirez, Gautham
Thambidorai, Karthik Sankaranarayanan, David Meis-
ner, and David Lo. We worked together on many fun
projects and I hope for more in the future. Although
my dad is no longer with us, I am also fortunate to
count on the love and support of my family. My mom
Cecilia, my godmother Margarida, my siblings Paula,
Tina, and Carlos and their families, and my wife
Catherine Warner who is the award life gives me every
single day.
PERHAPS NOTHING SUMMARIZES
THE IMPACT THAT FRIENDS AND
LUCK CAN HAVE IN YOUR LIFE MORE
THAN THE STORY OF HOW I CAME TO
JOIN GOOGLE. AS I WAS TRYING TO
MAKE A DECISION BETWEEN TWO
OPTIONS, JEFF DEAN ASKED ME
WHETHER THE OTHER COMPANY I
WAS CONSIDERING HAD ALSO

SERVED ME CREME ^ EE
BRUL
DURING
MY INTERVIEWS. I THANKED JEFF
AND ACCEPTED THE GOOGLE OFFER
THE VERY NEXT MORNING.
THREE LESSONS
I will finish this essay by sharing with you three lessons
I have learned in this first half of my career, in the hope
that they may be useful to engineers who are at an
earlier stage in their journey.
Consider the Winding Road
As an engineer you stand on a foundation of knowl-
edge that enables you to branch into many different
kinds of work. Although there is always risk when you
take on something new, the upside of being adventur-
ous with your career can be amazingly rewarding. I for
one never let my complete ignorance about a new
field stop me from giving it a go.
As a result, I have worked in areas ranging from
chip design to datacenter design; from writing soft-
ware for web search to witnessing my team launch
satellites into space; from writing software for Google
Scholar to using ML to automatically update Google
Maps; from research in compiler optimizations to
deploying exposure notification technology to curb
the spread of Covid-19.8
It seems a bit crazy, but not going in a straight line
has worked out really well for me and resulted in a rich
IEEE Micro 81
AWARDS

set of professional experiences. Whatever the out-
come, you will be inoculated from boredom.
Develop Respect for the Obvious
The surest way to waste a career is to work on unim-
portant things. I have found that big, important prob-
lems have one feature in common: they tend to be
straightforward to grasp even if they are hard to solve.
Those problems stare you right in the face. They are
obvious and they deserve your attention.
Let me give you some examples by listing some of
my more well-cited papers next to the formulation of
the problems address:
revisit my position on technical matters that I had
invested significant time and effort on, especially
when the original position had a track record of suc-
cess. I will present just one illustrative example.
I JOINED GOOGLE AFTER A FAILED
MULTIYEAR CHIP DESIGN PROJECT
AND AS SUCH I IMMEDIATELY
EMBRACED GOOGLE’S DESIGN
PHILOSOPHY OF STAYING AWAY
FROM SILICON DESIGN OURSELVES.

Publication Problem addressed I joined Google after a failed multiyear chip design
ISCA'98: “Memory “High-end project and as such I immediately embraced Google’s
System Characterization microprocessors are design philosophy of staying away from silicon design
of Commercial being sold to run ourselves. Later as the technical lead of Google’s data-
Workloads”10 commercial workloads, so center infrastructure, I consistently avoided using
with Kourosh why are we designing
exotic or specialized silicon even when they could
Gharachorloo and them for number
Edouard Bugnion crunching?” demonstrate performance of efficiency improvements
for some workloads, since betting on the low cost
ISCA'00: “Piranha: A “Thread-level parallelism
Scalable Architecture is easy. Instruction level base of general purpose components consistently
Based on Single-Chip parallelism is hard. Should proved to be the winning choice. Year after year, bet-
Multiprocessing”5 we bet on thread-level ting on general purpose solutions proved successful.
with Kourosh parallelism then?” Then, deep learning acceleration for large ML mod-
Gharachorloo, Robert els arose as the first opportunity in my career to build
McNamara, Andreas
Nowatzyk, specialized components that would have both broad
Shaz Qadeer, Barton applicability and dramatic efficiency advantages when
Sano, Scott Smith, Robert compared to general purpose designs. Our estimates
Stets, and Ben Verghese indicated that large fractions of Google’s emerging AI
CACM '17: “The Attack of “If datacenter-wide events workloads could be executed in these specialized
the Killer Microsecond”11 run at microsecond accelerators with as much as a 40 cost/efficiency
with Mike Marty, Dave speeds, why do we only advantage over general purpose computing.
Patterson, and Partha optimize for millisecond
That was a time to ignore the past successes of bet-
Ranganathan and nanosecond
latencies?” ting on general purpose off-the-shelf components and
invest heavily on the design and deployment of our own
CACM '13: “The Tail at “Large scale services
Scale”12 should be resilient to silicon to accelerate ML workloads. Coming full circle,
with Jeff Dean performance hiccups in this meant that it was now my time to call Norm Jouppi
any of their and ask him to join us to become the lead architect for
subcomponents” what was to become our TPU accelerators program.
IEEE Computer '07: “A “Shouldn’t servers use
Case for Energy- little energy when they
proportional are doing little work?” CONCLUDING
Computing”13
Before the onset of the current pandemic, some of us
€ lzle
with Urs Ho
may have underappreciated how important computing
If it takes you much more than a couple of sentences technology and cloud-based services have become to
to explain the problem you are trying to solve, you our society. In this last year, these technologies have
should seriously consider the possibility of it not being allowed many of us to continue to work, to connect
that important to be solved. with loved ones, and to support each other. I am grate-
ful to all of those at Google and everywhere in our
Even Successes Have a “Sell-By” Date industry who have built such essential technologies,
Some of the most intellectually stimulating moments and I am inspired to be working in a field with still so
in my career have come about when I was forced to much potential to improve people’s lives.

82 IEEE Micro March/April 2021

 AWARDS

REFERENCES 9. Google & Apple Exposure Notification technology.

2020. [Online]. Available: g.co/ENS
1. W. H. Davenport Adams, “Imitators and plagiarists,” The
10. L. A. Barroso, K. Gharachorloo, and E. Bugnion,
Gentleman’s Magazine, Jan. 1892
“Memory system characterization of commercial
2. N. P. Kronenberg, H. M. Levy, and W. D. Strecker,
workloads,” SIGARCH Comput. Archit. News, vol. 26,
“VAXcluster: A closely-coupled distributed system,”
no. 3, pp. 3–14, Jun. 1998.
ACM Trans. Comput. Syst., vol. 4, May 1986, Art. no. 130.
11. L. Barroso, M. Marty, D. Patterson, and P. Ranganathan,
[Online]. Available: https://doi.org/10.1145/
“Attack of the killer microseconds,” Commun. ACM,
214419.214421
vol. 60, no. 4, pp. 48–54, Apr. 2017.
3. T. Sterling, D. Becker, M. Warren, T. Cwik, J. Salmon, and
12. J. Dean and L. A. Barroso, “The tail at scale,” Commun.
B. Nitzberg, “An assessment of Beowulf class computing
ACM, vol. 56, no. 2, pp. 74–80, Feb. 2013.
for NASA requirements: Initial findings from the first
13. L. A. Barroso and U. Ho€ lzle, “The case for energy-
NASA workshop on Beowulf-class clustered computing,”
proportional computing,” Computer, vol. 40, no. 12,
in Proc. IEEE Aerosp. Conf., 1998, pp. 367–381.
pp. 33–37, Dec. 2007.
4. T. E. Anderson, D. E. Culler, and D. Patterson, “A case
for NOW (Networks of Workstations),” IEEE Micro,

BARROSO is a Google Fellow and a former VP of
LUIZ ANDRE
vol. 15, no. 1, pp. 54–64, Feb. 1995.
5. L. A. Barroso et al., “Piranha: A scalable architecture Engineering at Google. His technical interests include
based on single-chip multiprocessing,” in Proc. 27th machine learning infrastructure, privacy, and the design and
Annu. Int. Symp. Comput. Archit., 2000, pp. 282–293. programming of warehouse-scale computers. He has pub-
6. L. A. Barroso, J. Dean, and U. Holzle, “Web search for a lished several technical papers and has co-authored the
planet: The Google cluster architecture,” IEEE Micro,
book The Datacenter as a Computer, now in its 3rd edition.
vol. 23, no. 2, pp. 22–28, Mar./Apr. 2003.
He is a Fellow of the ACM and the AAAS and he is a member
7. A. Singh et al., “Jupiter rising: A decade of Clos
of the National Academy of Engineering. Barroso received a
topologies and centralized control in Google’s
datacenter network,” SIGCOMM Comput. Commun. B.S. and an M.S. in electrical engineering from the Pontifícia
Rev., vol. 45, no. 4, pp. 183–197, Oct. 2015. Universidade Católica of Rio de Janeiro, Rio de Janeiro, Brazil,
8. E. Pinheiro, W. Weber, and L. Barroso, “Failure trends in and a Ph.D. in computer engineering from the University
a large disk drive population,” in Proc. 5th USENIX Conf. of Southern California, Los Angeles, CA, USA. He is the
File Storage Technol., Feb. 2007, pp. 17–29. recipient of the 2020 Eckert-Mauchly award. Contact him at
luiz@barroso.org.

March/April 2021 IEEE Micro 83

PURPOSE: The IEEE Computer Society is the world’s largest
association of computing professionals and is the leading provider
of technical information in the field.
MEMBERSHIP: Members receive the monthly magazine
Computer, discounts, and opportunities to serve (all activities
are led by volunteer members). Membership is open to all IEEE
members, affiliate society members, and others interested in the
computer field.
COMPUTER SOCIETY WEBSITE: www.computer.org
OMBUDSMAN: Direct unresolved complaints to
ombudsman@computer.org.
CHAPTERS: Regular and student chapters worldwide provide the
opportunity to interact with colleagues, hear technical experts,
and serve the local professional community.
AVAILABLE INFORMATION: To check membership status, report
an address change, or obtain more information on any of the
following, email Customer Service at help@computer.org or call
+1 714 821 8380 (international) or our toll-free number,
+1 800 272 6657 (US):
• Membership applications
• Publications catalog
• Draft standards and order forms
• Technical committee list
• Technical committee application
• Chapter start-up procedures
• Student scholarship information
• Volunteer leaders/staff directory
• IEEE senior member grade application (requires 10 years
practice and significant performance in five of those 10)
PUBLICATIONS AND ACTIVITIES
Computer: The flagship publication of the IEEE Computer Society,
Computer publishes peer-reviewed technical content that covers
all aspects of computer science, computer engineering,
technology, and applications.
Periodicals: The society publishes 12 magazines‫ژ‬ƌȄư‫ژו׏ژ‬DZȏɓȵȄƌǹȽ.
Refer to membership application or request information as noted
above.
Conference Proceedings & Books: Conference Publishing
Services publishes more than 275 titles every year.
Standards Working Groups: More than 150 groups produce IEEE
standards used throughout the world.
Technical Committees: TCs provide professional interaction in
more than 30 technical areas and directly influence computer
engineering conferences and publications.
Conferences/Education: The society holds about 200 conferences
each year and sponsors many educational activities, including
computing science accreditation.
Certifications: The society offers three software developer
credentials. For more information, visit
www.computer.org/certification.
BOARD OF GOVERNORS MEETING
‫
ژ׏א‬ȲȵǠǹ‫ژً׏א׎אژ‬ɫǠȵɋɓƌǹ
revised ‫ڳ׮‬uƌȵƩǚ‫ת׫ש׫ڳ‬
EXECUTIVE COMMITTEE
President: FȏȵȵƷȽɋ‫ژ‬°ǚɓǹǹ
President-Elect: ÞǠǹǹǠƌȂ‫ژ‬%ِ‫ژ‬GȵȏȲȲ
Past President: kƷǠǹƌ‫ژ‬%Ʒ‫ژ‬FǹȏȵǠƌȄǠ
First VP: ¨ǠƩƩƌȵưȏ‫ژ‬uƌȵǠƌȄǠ; Second VP: FƌƨȵǠɼǠȏ‫ژ‬kȏȂƨƌȵưǠ ‫ژژژژژ‬
Secretary: ¨ƌȂƌǹƌɋǚƌ‫ژ‬uƌȵǠȂɓɋǚɓ; Treasurer: %ƌɫǠư‫ژ‬kȏȂƷɋ
VP, MemberȽǚǠȲ & Geographic Activities:
ȄưȵƷ‫‚ژ‬ƨȏǹƷȵ
VP, Professional & Educational Activities: OǠȵȏȄȏȵǠ‫ژ‬ÞƌȽǚǠɼƌǵǠ ‫ژژژژژژژژژژژ‬
VP, Publications: uِ‫ژ‬ȵǠƌȄ‫ژ‬ǹƌǵƷ
VP, Standards Activities: Riccardo Mariani
VP, Technical & Conference Activities: GȵƌƩƷ‫ژ‬kƷɬǠȽ‫ژژژژژژژژژژژ‬
20‫–׏א‬202‫ א‬IEEE Division VIII Director: ǚȵǠȽɋǠȄƌ‫ژ‬uِ‫ژ‬°ƩǚȏƨƷȵ‫ژ‬
‫ژ׏א׎אٮ׎א׎א‬U---‫ژ‬%ǠɫǠȽǠȏȄ‫ژ‬Ý‫ژ‬%ǠȵƷƩɋȏȵ‫¾ژي‬ǚȏȂƌȽ‫ژ‬uِ‫ژ‬ȏȄɋƷ‫ژژژژژژژژژژ‬
‫ژ׏א׎א‬U---‫ژ‬%ǠɫǠȽǠȏȄ‫ژ‬Ý‫ژ‬%ǠȵƷƩɋȏȵ‫ٮ‬-ǹƷƩɋ‫ژي‬ƷƩǠǹǠƌ‫ژ‬uƷɋȵƌ
BOARD OF GOVERNORS
¾ƷȵȂ‫ژ‬-ɱȲǠȵǠȄǒ‫ژي׏א׎אژ‬uِ‫ژ‬ȵǠƌȄ‫ژ‬ǹƌǵƷً‫ژ‬FȵƷư‫ژ‬%ȏɓǒǹǠȽً
ƌȵǹȏȽ‫ژ‬-ِ‫ژ‬eǠȂƷȄƷɼ‫ٮ‬GȏȂƷɼً‫¨ژ‬ƌȂƌǹƌɋǚƌ‫ژ‬uƌȵǠȂɓɋǚɓً
-ȵǠǵ‫ژ‬eƌȄ‫ژ‬uƌȵǠȄǠȽȽƷȄً‫ژ‬hɓȄǠȏ‫ژ‬ÅƩǚǠɲƌȂƌ
¾ƷȵȂ‫ژ‬-ɱȲǠȵǠȄǒ‫ژيאא׎אژ‬wǠǹȽ‫
ژ‬ȽƩǚƷȄƨȵɓƩǵً‫ژ‬
-ȵȄƷȽɋȏ‫ژ‬ɓƌưȵȏȽ‫ٯ‬ÝƌȵǒƌȽً‫ژ‬%ƌɫǠư‫ژ‬°ِ‫ژ‬-ƨƷȵɋً‫ژ‬GȵƌƩƷ‫ژ‬kƷɬǠȽً‫ژژژژژژ‬
OǠȵȏȄȏȵǠ‫ژ‬ÞƌȽǚǠɼƌǵǠً‫ژ‬°ɋƷǑƌȄȏ‫ژ‬îƌȄƷȵȏ
¾ƷȵȂ‫ژ‬-ɱȲǠȵǠȄǒ‫ژيבא׎אژ‬eɲȏɋǠǵƌ‫
ژ‬ɋǚƌɫƌǹƷً‫¾ژ‬Ʒȵȵɲ‫ژ‬ƷȄɼƷǹً‫ژژژژژژژژژژژ‬
¾ƌǵƌǵȏ‫ژ‬OƌȽǚǠȂȏɋȏً‫ژ‬UȵƷȄƷ‫ژ‬¥ƌɼȏȽ‫ژ‬ÝǠƌȄƌً‫
ژ‬ȄȄƷɋɋƷ‫¨ژ‬ƷǠǹǹɲً‫ژژژژژ‬
%Ʒƨȏȵƌǚ‫ژ‬°ǠǹɫƷȵ
EXECUTIVE STAFF
Executive Director: Melissa
ِ‫ژ‬Russell‫ژ‬
Director, Governance & Associate Executive Director: ‫ژژژژژژژ‬
Anne Marie Kelly‫ژ‬
%ǠȵƷƩɋȏȵً‫ژ‬ȏȄǑƷȵƷȄƩƷ‫‚ژ‬ȲƷȵƌɋǠȏȄȽ‫ژي‬°ǠǹɫǠƌ‫ژ‬ƷƨƌǹǹȏȽ‫ژژژژژژژژژژژ‬
Director, Finance & Accounting: Sunny Hwang‫ژ‬
Director, Information Technology & Services: Sumit Kacker
Director, Marketing & Sales: Michelle Tubb‫ژ‬
Director, Membership ‫ژۯ‬-ưɓƩƌɋǠȏȄ: Eric Berkowitz
COMPUTER SOCIETY OFFICES
Washington, D.C.: 2001 L St., Ste. 700, Washington, D.C.
20036-4928ٕ Phone: +1 202 371 0101ٕ Fax: +1 202 728 9614ٕ‫ژژژژ‬
Email: ǚƷǹȲ‫ۮ‬ƩȏȂȲɓɋƷȵِȏȵǒ
Los Alamitos: 10662 Los Vaqueros Cir., Los Alamitos, CA 90720ٕ‫ژ‬
Phone: +1 714 821 8380ٕ Email: help@computer.org
u-u-¨°OU¥‫ژۯژ‬¥ÅkU
¾U‚w‫¨‚ژ‬%-¨°‫ژ‬
¥ǚȏȄƷ‫ژٕבבבגژזוהژ׎׎זژ׏ڹژي‬Fƌɱ‫ژٕ׏גהגژ׏אזژג׏וژ׏ڹژي‬
-ȂƌǠǹ‫ژي‬ǚƷǹȲ‫ۮ‬ƩȏȂȲɓɋƷȵِȏȵǒ
IEEE BOARD OF DIRECTORS
President: °ɓȽƌȄ‫ژ‬hِ‫ٹژ‬hƌɋǚɲ‫ژٺ‬kƌȄư
President-Elect: hِeِ‫¨ژ‬ƌɲ‫ژ‬kǠɓ
Past President: ¾ȏȽǚǠȏ‫ژ‬Fɓǵɓưƌ
Secretary: Kathleen
ِ‫ژ‬Kramer
Treasurer: uƌȵɲ‫ژ‬-ǹǹƷȄ‫¨ژ‬ƌȄưƌǹǹ
Director & President, IEEE-USA: hƌɋǚƷȵǠȄƷ‫ژ‬eِ‫ژ‬%ɓȄƩƌȄ‫ژ‬Director
& President, Standards Association: eƌȂƷȽ‫ژ‬uƌɋɋǚƷɬȽ‫ژ‬Director
& VP, Educational Activities: °ɋƷȲǚƷȄ‫ژ‬¥ǚǠǹǹǠȲȽ‫ژژ‬Director & VP,
Membership ‫ ۯ‬Geographic Activities:‫ژژژژژژژ‬
uƌǠǵƷ‫ژ‬kɓǠǵƷȄ
Director & VP, Publication Services & Products: kƌɬȵƷȄƩƷ‫ژ‬Oƌǹǹ‫ژ‬
Director & VP, Technical Activities: ¨ȏǒƷȵ‫ژ‬Åِ‫ژ‬FɓDZǠǠ
 IEEE COMPUTER SOCIETY

Computing
in Science &
Engineering
Subscribe to CiSE today!
www.computer.org/product/magazines/cise

The computational and data-centric problems faced by scientists

and engineers transcend disciplines. There is a need to share
knowledge of algorithms, software, and architectures, and
to transmit lessons-learned to a broad scientific audience.
Computing in Science & Engineering (CiSE) is a cross-disciplinary,
international publication that meets this need by presenting
contributions of high interest and educational value from a
variety of fields, including physics, biology, chemistry, and
astronomy. CiSE emphasizes innovative applications in cutting-
edge techniques. CiSE publishes peer-reviewed research articles,
as well as departments spanning news and analyses, topical
reviews, tutorials, case studies, and more.

www.computer.org/cise
 DEPARTMENT: MICRO ECONOMICS
The Economics of Confrontational
Conversation
Shane Greenstein , Harvard Business School, Boston, MA, 02163, USA
M
y favorite panel from Randall Munroe’s one-
panel comic, xkcd, is labeled “duty calls.” It
shows a lone stick figure at a desk,
hunched over, intensely staring at his computer
screen, while engaging in a staccato conversation
with an offscreen companion. She says: “Are you com-
ing to bed?” Him: “I can’t. This is important.” Her:
“What?” Him: “Someone is wrong on the internet.”
Like all good satire, this comic elicits both laughs
and winces. Nobody would ever engage in this behav-
ior in any physical place where a veneer of social
politeness predominates, such as standing in line at a
cashier or sitting in an airplane seat. On the internet
surfers jettison much of their social restraint, con-
fronting, and correcting perfect strangers. It leads to,
for example, edit wars on Wikipedia, condescending
insults on Reddit, and righteous putdowns on Twitter.
This behavior invites plenty of legal analysis, angry
editorializing, and technological proposals, but rarely
economic analysis. Let us address that gap. What eco-
nomic factors make confrontational conversation
more or less likely in our era?
TENSIONS WITH COMPROMISE
The first piece of relevant economics is the low cost of
scale. It is inexpensive to host terabytes of data, and
dirt-cheap to serve millions of users. Software can be
replicated many times at almost no cost, making it
possible for a platform to scale.
The second relevant economic factor complements
scale, and goes by the label “network effects.” These
are self-reinforcing advantages affiliated with being a
focal platform. Simplifying, a platform becomes focal in
one of two ways. In one form a platform attracts more
apps or content, and that attracts more users, which
then attracts more apps or content, and so on. In
another form a platform attracts more sellers, which
attracts more buyers, which attracts more sellers and
0272-1732 ß 2021 IEEE
Digital Object Identifier 10.1109/MM.2021.3060295
Date of current version 26 March 2021.
86 IEEE Micro Published by the IEEE Computer Society
so on. In either case, those attractions become self-
reinforcing.
A third crucial piece of economics shapes focal
platforms with large scale. Some confrontation is inev-
itable, and undermines the functionality of most (not
all!) platforms. It is possible to write volumes on how
to address confrontation with human moderation or
algorithmic processes, and whether specific practices
are legal or effective. Take a step back from that dis-
cussion, and recognize the broad economic facts. No
matter how it gets implemented, the processes are
expensive and imperfect.
ON THE INTERNET SURFERS
JETTISON MUCH OF THEIR SOCIAL
RESTRAINT, CONFRONTING, AND
CORRECTING PERFECT STRANGERS.
IT LEADS TO, FOR EXAMPLE, EDIT
WARS ON WIKIPEDIA,
CONDESCENDING INSULTS ON
REDDIT, AND RIGHTEOUS
PUTDOWNS ON TWITTER. THIS
BEHAVIOR INVITES PLENTY OF
LEGAL ANALYSIS, ANGRY
EDITORIALIZING, AND
TECHNOLOGICAL PROPOSALS, BUT
RARELY ECONOMIC ANALYSIS.
Two features of the present era drive up those
costs and exacerbate the imperfections. Anything
with long video is inherently expensive to moderate,
as managers at YouTube and Facebook can attest. In
addition, bots and misinformation farms have flooded
the modern experience, especially at focal plat-
forms—just ask Twitter and Facebook. Holding those
in check has become an endless and expensive
whack-a-mole for large platforms.
More to the point, addressing content moderation
has become this era’s key to achieving scale. Different
platforms have taken different approaches to this
challenge, and each approach comes with different
upsides and drawbacks.
March/April 2021

DIFFERENT APPROACHES
Apple’s approach to content moderation is the easiest
to understand, namely, it “sanitizes” its content during
a preapproval process for apps. Apple treats violent
content and confrontational language just like it
treats porn. It bans everything that violate some rules
and bounds, with the (possible) exception of some
video games.
Is that expensive? Yes, because humans must do
it. Moreover, the approval process is arduous, and
many developers dislike it. So why do it? Because it
protects the brand. Would most parents buy an
iPhone for a teenage child if it included an app for
quasi-Nazi nonsense? Of course not. As in any other
business, Apple targets a primary customer, and aligns
their operations with serving these customers.
Parts of this approach has migrated to other plat-
forms. Most platforms want nothing to do with Holo-
caust deniers and flat-earthers or violent terrorists of
any sort, so they ban them, using their terms of service
as a legal shield.
DURING THE ELECTION A NEW TYPE
OF BREAKAWAY EMERGED WHEN
POLITICAL OPINION SHAPERS
COUCHED THEIR APPEAL IN
RIGHTEOUS PATRIOTISM. LOVE THEM
OR HATE THEM, THEIR EXPERIENCE
ILLUSTRATES THE ECONOMICS.
Sanitizing alone does not work on all platforms in
all situations. That mixed success motivates another
approach, channeling the confrontations with clever
design accompanied by a few sanitizing rules about
indecent language. For example, a customer may hate
a restaurant for a good or crazy reason, but Yelp found
a way to guide the confrontations, and make them
valuable to readers. Yelp displays the average as stars,
and spotlights a colorful description, which users can
read or ignore.
A whole range of ratings sites use different varia-
tions on this approach. However, fake reviews and
offensive reviews (that just skirt the rules) have
become the pest in today’s version of whack-a-mole.
Without naming names, it undermines the usefulness
of some ratings sites.
Facebook illustrates a different approach, namely,
using cleverly designed algorithms to manipulate who
takes part in the confrontational conversation. One
might call it “divide but engage” because this
March/April 2021
MICRO ECONOMICS
approach tends to divide participants into like-minded
groups. Facebook came to this approach through test-
ing and refining its algorithms for the news feed to
maximize engagement, showing users content that
kept them for longer and motivated them to come
back.
As has been widely noted, this approach has one
major drawback. It creates insulated conversation
bubbles of outrage among like-minded friends. More-
over, a flood of misinformation and bots make this
problem worse. Some time ago Facebook started to
feel the downside in terms of user fatigue and disgust
with the site overall, so it has been experimenting with
policing the misinformation. It banned many fake
sites. Like other platforms, Facebook also imple-
mented labels for lies and for unverified rumors. Like
every imperfect approach, this one was costly, and it
frustrated plenty of users.
And, yet, they survive and still thrive. The economic
lesson? The site held together because the network
effects are strong enough.
A final approach melds clever design and sanitizing
into something more muddled. While there are many
examples, the most successful of these is Google’s
approach to its search engine. Google sanitizes its
ads, banning alcohol, gambling, porn, and illegal activi-
ties, as well as indecent language. At the same time,
Google has long taken a more permissive approach to
its organic search results. If a user searched for some-
thing shady they got back something, well, shady.
Users filter the offensive results if they want to. This
was (and still is) a clever and effective way to find any-
thing except the dark web.
Its downsides are also well known: It remains vul-
nerable to special pleading. Google fields endless
complaints about its organic results. Many govern-
ments around the world wants to regulate it, while
many politicians want to manipulate it.
PRESSURES AGAINST SCALE
We just described settings where network effects are
strong. What happens when they are not? Simply
stated, a breakaway community forms if users find an
entrepreneur who creates an alternative platform that
users prefer.
Breakaway communities with facets of indecency
have been a part of the internet for a long time. For
example, Reddit, 4chan, or many places in the dark
web wear their confrontational language as a badge of
honor. More to the point, the skills to manage one of
these sites are common, and it is a lot cheaper to
operate if the moderation is light. It is not my taste
IEEE Micro 87
MICRO ECONOMICS
and it may not be yours, but there is a large supply of
these breakaways.
During the election a new type of breakaway
emerged when political opinion shapers couched their
appeal in righteous patriotism. Love them or hate
them, their experience illustrates the economics.
It started with, say, Alex Jones and Infowars, who
got banned because it went beyond the pale. Many
mainstream sites thought it was the right thing to do,
and it aligned with the economics because the losses
of user were small compared to the risks to offending
too many disgusted customers. Once banned, Alex
Jones took his users with him and formed his own
community elsewhere.
AS DEEPFAKES BECOME MORE
COMMON, WHO WILL ADJUDICATE
WHETHER A DEEPFAKE OF A
POLITICIAN OR CELEBRITY IS REAL OR
NOT? HOW CAN ANYBODY DO THAT
IF ONLINE USERS HAVE SORTED
THEMSELVES INTO VARIOUS GROUPS
THAT DO NOT TRUST ONE ANOTHER?
Parler’s recent experience exemplifies the state of
play today. As a reminder, Parler came into existence
because many white nationalist groups tired of Face-
book’s and Twitter’s attempts to label their content as
factually problematic and potentially violent. Parler
opened on AWS, and made lack of moderation a fea-
ture instead of a bug, and it attracted a following
quickly.
The other mainstream platforms eventually reacted
in predictable ways. Apple was among the earliest to
88 IEEE Micro
ban Parler’s app, consistent with its sanitizing approach.
AWS eventually they took a similar action after notifying
Parler of their concerns about lack of moderation. Once
Parler gained prominence as a coordinator of the capital
riot, AWS saw no benefit to having its brand associated
with this group, and that was that.
Parler’s breakaway did not go well because man-
agement cut many corners and did not design a hack-
proof site, but here is the rub. Parler is trying again.
Even if they fail again, it is a good bet that somebody
else will get a version of this up and running.
Summarizing, the increasing frequency of break-
aways is a symptom that they are becoming cheaper
to build. Ergo, we should expect mainstream sites to
face increasing pressures towards fragmentation.
CONCLUSION
The trends toward fragmentation worries anyone who
wants to maintain civil society. Who will encourage the
confrontations that settle the public conversations?
Most worrisome, misinformation, and deep fakes
are becoming more widespread in breakaway commu-
nities, and especially on the dark web. Right now,
most users of deepfakes entertain themselves (You
do not really want to know the details.), but, as with
any frontier software, it will become mainstream soon
enough.
As deepfakes become more common, who will
adjudicate whether a deepfake of a politician or celeb-
rity is real or not? How can anybody do that if online
users have sorted themselves into various groups that
do not trust one another?
SHANE GREENSTEIN is a Professor at the Harvard Business
School. Contact him at sgreenstein@hbs.edu.
March/April 2021
Get Published in the IEEE Open
Journal of the
$PNQVUFS
4PDJFUZ

Submit a paper today to the

premier open access journal
in DPNQVUJOH
BOE
JOGPSNBUJPO

UFDIOPMPHZ.

Your research will benefit from

the IEEE marketing launch and
5 million unique monthly users
of the IEEE Xplore® Digital Library.
Plus, this journal is fully open
and compliant with funder
mandates, including Plan S.

Submit your paper today!

Visit XXXDPNQVUFSPSHPK to learn more.
IEEE
Computer
Society Has
You Covered!
WORLD-CLASS CONFERENCES — Stay
ahead of the curve by attending one of our
215+ globally recognized conferences.

DIGITAL LIBRARY — Easily access over 800k

articles covering world-class peer-reviewed
content in the IEEE Computer Society
Digital Library.

CALLS FOR PAPERS — Discover

opportunities to write and present your
ground-breaking accomplishments.

EDUCATION — Strengthen your resume

with the IEEE Computer Society Course
Catalog and its range of offerings.

ADVANCE YOUR CAREER — Search the

new positions posted in the IEEE Computer
Society Jobs Board.

NETWORK — Make connections that count

by participating in local Region, Section,
and Chapter activities.

Explore all of the member benefits

at www.computer.org today!