PURR: A Primitive for Reconfigurable Fast Reroute

Marco Chiesa
KTH Royal Institute of Technology
Code: bitbucket.org/marchiesa/purr

Joint work with:

Roshan Sedar
Gianni Antichi
Michael Borokhovich
Andrzej Kamisiński
Georgios Nikolaidis
Stefan Schmid
 PURR: Thanks to my coauthors!

Joint work with:

Roshan Sedar
Gianni Antichi
Michael Borokhovich
Andrzej Kamisiński
Georgios Nikolaidis
Stefan Schmid

2024-03-02 2
 Network resilience is a key yet challenging property

High resilience

Why is it that hard?

2024-03-02 Image credits: route by Philipp Petzka from the Noun Project 3
 Routing reconvergence takes time!
routing
restoration failure event

control-plane

time
detection & route
recomputation

data-plane update

Video shot taken from “Lemmings”

2024-03-02 designed and developed by DMA Design 5
 Centralized controllers make reconvergence slower

failure occurs

control-plane

time
detection & route
recomputation

Advanced SDN routing High resilience data-plane update

e.g., intent-based logically
centralized SDN

2024-03-02 Image credits: route by Philipp Petzka from the Noun Project 7
 Hard to recover within 50ms!

failure occurs

control-plane

time
detection & route
recomputation

Advanced SDN routing High resilience data-plane update

e.g., intent-based logically “[carrier-grade networks] level of
centralized SDN availability requires substantial over-
provisioning and fast reroute local
recovery, e.g., within 50 milliseconds“ [1]

[1] On low-latency-capable topologies, and their impact on the design of intra-domain routing. In SIGCOMM 2018
2024-03-02 Image credits: route by Philipp Petzka from the Noun Project 8
 Fast Reroute (FRR):
pre-computing failover paths
match/actions (e.g., at
X) C
1 dst = A >> FRR1
2nd
backup
FRR actions (e.g., at X B
X)FRR >> fwd A B C primary backup
control plane 1

FRR entails solving two orthogonal problems: A

1. control-plane: compute network-wide primary/backup forwarding rules

10
 Fast Reroute (FRR):
pre-computing failover paths
match/actions (e.g., at
X) C
1 dst = A >> FRR1
dst = B >> FRR2 backup

FRR actions (e.g., at X B

X)FRR >> fwd A B C primary
2nd
control plane 1
FRR2 >> fwd B C A backup

FRR entails solving two orthogonal problems: A

1. control-plane: compute network-wide primary/backup forwarding
rules
11
 Fast Reroute (FRR): header/metadata

deparser
pre-computing failover paths

parser
match/actions (e.g., at
X) C
1 dst = A >> FRR1
dst = B >> FRR2
2 packet pipeline
backup

FRR actions (e.g., at X B

X)FRR >> fwd A B C primary
2nd
control plane 1
FRR2 >> fwd C B A backup

FRR entails solving two orthogonal problems: A

1. control-plane: compute network-wide primary/backup forwarding
rules
2. data-plane: support conditional forwarding in each switch 12
The goal of this talk: implement a FRR header/metadata
primitive

deparser
that minimizes pipeline resource consumption

parser
dst = A >> FRR1 FRR1 >> fwd A B C
dst = B >> FRR2 FRR2 >> fwd C B A

PURR: A building block for implementing arbitrary FRR mechanisms

2
packet pipeline

Input
FRR actions (e.g., at Output
X)FRR >> fwd A B C
1
FRR2 >> fwd C B A

Cat by dDara from the Noun Project 13

 PURR applies to P4 programmable switches

Packet recirculation
simplified pipeline
Packets in

Packets out
Parser

stage …

stage N
stage 1
ingress
buffer
Advanced SDN Input
routing High availability
headers & High flexibility
(e.g., intent-based (e.g., within(e.g.,
50 reconfigurable
metadata
logically centralized milliseconds”[1])P4 dataplanes)
Resources: SDN)
- SRAM (exact matches), e.g., 2M entries Runtime P4
- TCAM (wildcard matches), e.g., 100K entries (Control plane)
- ALU
[1] On low-latency-capable topologies, and their impact on the design of intra-domain routing. In SIGCOMM 2018
2024-03-02
Image credits: route by Philipp Petzka from the Noun Project 14
 PURR applies to P4 programmable switches

Packet recirculation
simplified pipeline
Packets in

Packets out
Parser

stage …

stage N
stage 1
ingress
buffer
Advanced SDN Input
routing High availability
headers & High flexibility
(e.g., intent-based (e.g., within(e.g.,
50 reconfigurable
metadata
logically centralized milliseconds”[1])P4 dataplanes)
SDN)
Resources: How do we realize FRR in P4?
- SRAM (exact matches), e.g., 2M entries
No P4 built-in FRR primitive
- TCAM (wildcard matches), e.g., 100K entries but…
Runtime P4
(Control plane)
a programmable pipeline: many approaches are
- ALU possible!
[1] On low-latency-capable topologies, and their impact on the design of intra-domain routing. In SIGCOMM 2018
2024-03-02
Image credits: route by Philipp Petzka from the Noun Project 15
 First approach: packet recirculation
action
match
write &
tag fwd recirculate
Input 1 1 tag := 2 port 1 fails
FRR1 = 1 2 3 4
2 2 tag := 3 port 2 fails
3 3 tag := 4
4 4 -

throughput reduction
latency increase

2024-03-02 16
 FRR recirculation has high memory occupancy
action
match
write &
tag fwd recirculate
Input 1 1 tag := 2
FRR1 = 1 2 3 4
2 2 tag := 3
FRR2 = 4 3 2 1
3 3 tag := 4
4 4 -

throughput reduction 5 4 tag:= 6 port 4 fails

latency increase 6 3 tag = 7 port 3 fails
high memory overhead 7 2 tag = 8
8 1 -
2024-03-02 17
 PURR: a primitive for reconfigurable FRR
A building block for implementing arbitrary FRR mechanisms

high low forwarding efficient flexibility small

throughput latency reroute forwarding
tables

parallel search with TCAM model port status -> flip

support one single
arbitrary FRRbitmechanisms,
i.e., arbitrary FRR input sequences
2024-03-02 Cat by dDara from the Noun Project 18
 PURR: a primitive for reconfigurable FRR
A building block for implementing arbitrary FRR mechanisms
• intriguing connection to (algorithmic) “string theory”

high low forwarding efficient flexibility small

throughput latency reroute forwarding
tables

2024-03-02 Cat by dDara from the Noun Project 19

 PURR: Encoding FRR in the packet metadata
Input
FRR1 = 1 2 3 4

packet metadata match action

FRR = 1 port
port1 2isis port status fwd
active
active 1*** 1
*1** 2
port status
**1* 3
1111
***1 4
P4 register
TCAM wildcard
match memory

2024-03-02 20
 PURR: Encoding FRR in the packet metadata
Input
FRR1 = 1 2 3 4

packet metadata port status match action

port status fwd
FRR = 1 1111
1*** 1
*1** 2
**1* 3
***1 4

TCAM wildcard
match memory

2024-03-02 21
 PURR: Encoding FRR in the packet metadata
Input
FRR1 = 1 2 3 4

packet metadata port status match action

port status fwd
FRR = 1 0111
1*** 1

port 1 fails *1** 2

**1* 3
***1 4

TCAM wildcard
match memory

2024-03-02 22
 PURR: Encoding FRR in the packet metadata
Input
FRR1 = 1 2 3 4

packet metadata port status match action

port status fwd
FRR = 1 0011
1*** 1
*1** 2
port 2 fails **1* 3
***1 4

TCAM wildcard
match memory

2024-03-02 23
 PURR: One tempting option: “Duplication” TCAM
Input
FRR1 = 1 2 3 4 FRR2 = 2 3 4 1

packet metadata port status match action

FRR = 2 0011 FRR port status fwd

1 1*** 1
1 *1** 2
1 **1* 3

similar to FRR recirculation: 1 ***1 4

high memory overhead 2 *1** 2
2 **1* 3
2 ***1 4
2024-03-02 2 1*** 1 24
 PURR: Encoding FRR in the packet metadata
Input
FRR1 = 1 2 3 4 FRR2 = 2 3 4 1

packet metadata port status Encoding FRR input:

FRR = 1 1111 • add a packet metadata field frr_ports
action write • map bits to the switch ports
match
frr_ports • set bit to 1 to include a port
FRR = 1 b11b12b13b140b5 • set bit to 0 to skip a port
FRR = 2 00 11111 00
bit-to-port mapping
FRR = 3 10 20 31 41 1 1 0

FRR = 4 0001111
2024-03-02 25
 PURR: Encoding FRR in the packet metadata
Input
FRR1 = 1 2 3 4 FRR2 = 2 3 4 1

packet metadata port status match action

frr_ports port status fwd
FRR = 1 1111
1**** 1*** 1
action write *1*** *1** 2
match
frr_ports **1** **1* 3
FRR = 1 11110 ***1* ***1 4
FRR = 2 01111 ****1 1*** 1

bit-to-port mapping
12341
2024-03-02 26
 PURR: Encoding FRR in the packet metadata
Input
FRR1 = 1 2 3 4 FRR2 = 2 3 4 1

packet metadata port status match action

frr_ports port status fwd
FRR = 1 1111
1**** 1*** 1
action write *1*** *1** 2
match
frr_ports **1** **1* 3
FRR = 1 11110 ***1* ***1 4
FRR = 2 01111 ****1 1*** 1

bit-to-port mapping
12341
2024-03-02 27
 PURR: Encoding FRR in the packet metadata
Input
FRR1 = 1 2 3 4 FRR2 = 2 3 4 1

packet metadata port status match action

frr_ports port status fwd
FRR = 1 0111
1**** 1*** 1
action write *1*** *1** 2
match
frr_ports **1** **1* 3
FRR = 1 11110 ***1* ***1 4
FRR = 2 01111 ****1 1*** 1

bit-to-port mapping
12341
2024-03-02 28
 PURR: re-cycling TCAM entries
Input
FRR1 = 1 2 3 4 FRR2 = 2 3 4 1

packet metadata port status match action

frr_ports port status fwd
FRR = 2 1111
1**** 1*** 1
action write *1*** *1** 2
match
frr_ports **1** **1* 3
FRR = 1 11110 ***1* ***1 4
FRR = 2 01111 ****1 1*** 1

bit-to-port mapping
12341
2024-03-02 29
 The key problem: How to compute the bit-to-port
mapping that minimizes memory occupancy?
FRR1 = 2 3 1 0 FRR2 = 0 2 1 3 FRR3 = 3 0 2 1 FRR4 = 1 0 2 3

bit-to-port mapping = ?.2 3 1 0 2 1 3

FRR1 = 2 3 1 0
FRR2 = 0213
FRR3 = 3 0 2 1
FRR4 = 102 3
Shortest Common Supersequence (SCS) problem without repetitions
• SCS without repetitions is computationally hard (based on [2])
• Dynamic Programming (DPSCS) computes optimum in exponential time

2024-03-02 [2 ]T. Jiang, M. Li. On the approximation of shortest common supersequences and longest common subsequences. In Journal on Computing. 30
 Fast-Greedy: a heuristic for solving this specific SCS
idea: remove the most F1=2 3 1 0 F1=3 1 0 F1=1 0 F1=1 0
frequent left-most F2=2 0 1 3 F2=0 1 3 F2=0 1 3 F2=1 3
element among the F3=2 3 0 1 F3=3 0 1 F3=0 1 F3=1
longest sequences F4=3 1 2 0 F4=3 1 2 0 F4=1 2 0 F4=1 2 0
remove 2 remove 3 remove 0 remove 1
See the paper for:
• multi-table
optimization F1=0 F1=0 F1= ---
F2=3 F2=3 F2=3
F3= --- F3= --- F3= ---
F4=2 0 F4=0 F4= ---
remove 2 remove 0 remove 3
SCS = 2 3 0 1 2 0 3
2024-03-02 31
 Implementation feasibility

P4-based implementations:
• implemented different FRR mechanisms in P4 using
PURR (e.g., F10 [1], arborescences [2], BFS, DFS,
rotor router [3])
• compiled on Tofino

FPGA-based implementation:
• implemented PURR on the NetFPGA-SUME platform

[1] V. Liu et al. "F10: A Fault-Tolerant Engineered Network" in NSDI 2013

[2] M. Chiesa et al. "On the Resiliency of Randomized Routing Against Multiple Edge Failures" in Transactions on Networking 2016
2024-03-02 [3] Borokhovich et al ””Graph exploration algorithms” in HotSDN 2013 32
 Evaluation: How does the FRR implementation
impact memory and performance?
Two subquestions:
1. How much memory does PURR save?
2. How does performance in a datacenter vary depending
on how one implements a FRR primitive?

See the paper for:

• multi-table optimization
• random vs tree-based FRR sequences
• FPGA chip occupancy
• low-size FRR sequences
2024-03-02 33
 How much memory does PURR save?
The “circular sequences” case
Input:
• switch with k ports
• 10 circular set of FRR
sequences

“Duplication TCAM” FRR: • 10 Top-of-Rack switches in a

• k2 number of TCAM entries datacenter with F10 FRR [nsdi-13]
• 10 destinations with the "k arc-
disjoint" FRR mechanism [ton-16]
With PURR encoding:
• k-1 number of TCAM entries
[nsdi-13] V. Liu et al. "F10: A Fault-Tolerant Engineered Network" in NSDI 2013
2024-03-02 [ton-16] M. Chiesa et al. "On the Resiliency of Randomized Routing Against Multiple Edge Failures" in Transactions on Networking 2016 34
 How much memory does PURR save?
The “circular sequences” case
Input: For k = 24
• switch with k ports • 92% less TCAM entries
• 10 circular set of FRR • 470 instead of 5.760
sequences
For k = 48
“Duplication TCAM” FRR: • 96% less TCAM entries
• k2 number of TCAM entries • 950 instead of 23.040

With PURR encoding:

• k-1 number of TCAM entries
[nsdi-13] V. Liu et al. "F10: A Fault-Tolerant Engineered Network" in NSDI 2013
2024-03-02 [ton-16] M. Chiesa et al. "On the Resiliency of Randomized Routing Against Multiple Edge Failures" in Transactions on Networking 2016 35
 How much memory does PURR save?
Fast-greedy performs close to the optimum
avg. #TCAM entries

30 106

avg. time [ms]

105
Fast-greedy 104
20 103 DPSCS
DPSCS 102
10 101 Fast-greedy
100
0 10-1
2 3 4 5 6 7 8 2 3 4 5 6 7
number of FRR sequences number of FRR sequences

Input: randomly generated set of FRR sequences of length 7

2024-03-02 36
 How much memory does 32 PURR save?
factorial possible FRR sequences
Fast-greedy scales to large numberTCAM
The “duplication” of sequences
or recirculation FRR
approaches would not scale
107
#TCAMentries

[bit]
entries

10 3 k=8 sequence size=32

k=16 k=32
k=8 k=16 sequence
k=32size=32

bits
106
sequence size=16

cost
105 sequence size=16

TCAM
2
10 sequence size=8
avg. #TCAM

104

Memory
sequence size=8
103

avg.
1
10 102
101 102 103 104 105 101 102 103 104 105
number of sequences number of sequences

Input: randomly generated set of FRR sequences

2024-03-02 37
 How does FCT vary depending on the FRR primitive?
PURR improves both FCT and throughput
2.0
Small flows FCT [ms]

Throughput [Gbps]
1.5 101
FRR recirculation immediat
e reconve
1.0 rgence
2.4x 2.7x
0.5 FRR recirc
immediate reconvergence 100 ulation
0.0
10 20 30 40 50 60 70 10 20 30 40 50 60 70
Load [%] Load [%]

NS-3 simulations
Topology: 32-server Clos network, 10Gbps links
Workload: data-mining Transport: DCTCP One link failure at 0.5s
2024-03-02 38
 How does FCT vary depending on the FRR primitive?
PURR improves both FCT and throughput
2.0
Small flows FCT [ms]

Throughput [Gbps]
1.5 101
FRR recirculation immediat
e reconve
1.0 purr
rgence
2x
1.4x purr
0.5 FRR recirc
immediate reconvergence 100 ulation
0.0
10 20 30 40 50 60 70 10 20 30 40 50 60 70
Load [%] Load [%]

NS-3 simulations
Topology: 32-server Clos network, 10Gbps links
Workload: data-mining Transport: DCTCP One link failure at 0.5s
2024-03-02 39
 Conclusions: Keep calm and enjoy programmability
Fast Reroute is a critical functionality in today’s network
• requires high throughput, low latency, fast reactiveness, small forwarding tables
P4 does not define an FRR built-in primitive
• pipeline compilers and control-plane must program the P4 pipeline
PURR: We propose a lightweight TCAM-based FRR primitive
• an intriguing connection to algorithmic string theory
• no FRR-tailored hardware support
• improve performance by a factor of ~2x w.r.t. FRR recirculation
Marco Chiesa
Thank you! KTH Royal Institute of Technology
Code: bitbucket.org/marchiesa/purr
reusable
2024-03-02 Cat by dDara from the Noun Project 40