Netronome-Ovs Offload PDF

OVS Offload Models Used with NICs and SmartNICs: Pros and Cons - ... about:reader?url=https://www.netronome.com/blog/ovs-offload-models-...
netronome.com
OVS Offload Models Used with NICs

and SmartNICs: Pros and Cons -
Netronome
15-19 minutes
As SmartNICs become more popular, more decision makers at

companies are being asked to look at the way SmartNICs work –
specifically the different offload models related to the OVS
datapath, and the pros and cons of each model. In this blog, I will
go through and explain these various models. As we go through
each model, you will realize that not all of them require offload of
the OVS datapath to NIC hardware. In some cases the datapath is
actually moved up to the user space as well (some call it on-
loading). I will cover all of them using a consistent tabular format for
easy readability and comparison. Here we go…
Model 1: OVS datapath in the kernel
VNF with Open vSwitch

1 of 18 09-05-20, 10:37 AM
VNF with Open vSwitch

(kernel datapath)
VM VM
Guest OS Guest OS
Virtio Virtio
Open vSwitch
vswitchd
USER SPACE
OVS kernel module

KERNEL
NIC NIC
Default for OpenStack switching,

bonding, overlay, live migration
Summary of features, pros (in green) and cons (in red):
Most mature and proven model
Broadest vendor support

Location of OVS
Easier to integrate and enhance
datapath (kernel, Kernel
the datapath with other kernel-
user, NIC)
based networking features such
as Conntrack, BPF
2 of 18 09-05-20, 10:37 AM
Policy enforcement
Suitable for SDN and Network
using OVS datapath Yes
Virtualization
(yes, no)
Suitable for cloud-based
OpenStack deployments
Orchestration Yes
Supported with most OpenStack
support (yes, no)
distributions
Uses Virtio driver in the VM

VM hardware making the VMs hardware
independence (yes, Yes independent and enabling support
no) of broad array of guest operating
systems and live VM migration
Poor datapath performance
1-2 Performance degrades rapidly

Performance, CPU
Mpps with more flows and more or
core use for datapath
Using 4 complex policy rules
processing (Mpps,
CPU Consumes many CPU cores for
No. of cores)
Cores datapath performance resulting in
lowest server utilization and TCO
Model 2: SR-IOV is used where the OVS datapath is

bypassed
DPDK VNF with SR-IOV
VNF VNF
DPDK DPDK
Network Network
3 of 18 09-05-20, 10:37 AM
Network Network
App App
Guest OS Guest OS
VF PMD VM PMD
Compute KVM
Node
VFO &/VF1
2
NIC
Hardware dependent to the NIC

line-rate, no CPU overhead, ToR for switching
Kernel but not Mature and proven model

Location of OVS datapath
used (bypassed
(kernel, user, NIC) Broadest vendor support
using SR-IOV)
No Not suitable for SDN

Policy enforcement using Relies on policy
Not suitable Network
OVS datapath (yes, no) enforcement at
Virtualization at scale
TOR switch

OpenStack Orchestration
Yes deployments where SDN-
support (yes, no)
4 of 18 09-05-20, 10:37 AM
based policy enforcement is

not required
Supported with most

OpenStack distributions.
Uses vendor-specific driver

in the VM making the VMs
VM hardware hardware dependent
No
independence (yes, no)
Live VM migration is not
supported
Performance, CPU core Close to 30 Mpps

Excellent performance
use for datapath No CPU cores as
delivering packets from
processing (Mpps, No. of the OVS datapath
network ports to VMs
cores) is not used
Model 3: OVS datapath is moved to the user space

using DPDK
DPDK VNF with

Open vSwitch + DPDK
VNF VNF
DPDK DPDK
Network Network
App App
Guest OS Guest OS
5 of 18 09-05-20, 10:37 AM
Virtio Virtio
DPDK vhost-user
Compute
Node
Open vSwitch + DPDK
DPDK PMD
NIC
DPDK – Direct I/O to NIC or vNIC

switching, bonding, overlay
Broad vendor support
Location of OVS Difficult to enhance features

User
datapath (kernel, leveraging other kernel datapath
space
user, NIC) implementations such as in
Conntrack and BPF
Policy enforcement
Suitable for SDN and Network
Virtualization
(yes, no)
OpenStack Suitable for cloud-based
Orchestration Yes deployments
6 of 18 09-05-20, 10:37 AM
Supported with many OpenStack

support (yes, no)
distributions

independence (yes, Yes independent and enabling support
no) of broad array of guest operating
systems and live VM migration
Good datapath performance
6-8 Performance degrades rapidly

Performance, CPU
Mpps with more flows and more or
core use for datapath
Using 4 complex policy rules
processing (Mpps,
CPU Consumes many CPU cores for
No. of cores)
Cores datapath performance resulting in
low server utilization and TCO
Model 4: OVS datapath in the kernel is fully offloaded

to the SmartNIC, hardware-dependent VMs
OpenStack
Controller
Nova
Neutron
VNF VNF
VM VM
vNIC vNIC OVS Control
ovsdb-server
Hypervisor KVM ovs-vswitchd

USER SPACE
KERNEL
7 of 18 09-05-20, 10:37 AM
KERNEL
SR-IOV
VFs
TC Flower Offload
VF VF PF
vSwitch Offload
NIC NIC
NIC
Uses kernel-compliant and

upstreamed TC flower
based offload of the OVS
datapath
Offload mechanism is
Kernel and included in the RHEL 7.5
SmartNIC distribution
Location of OVS Fallback to
Kernel-based offload makes
datapath (kernel, kernel OVS for
it easier to enhance features
user, NIC) control traffic
leveraging other kernel
and new/first
datapath implementations
flow
such as in Conntrack and
BPF
Available in the latest kernel

releases only until
backported to older versions
8 of 18 09-05-20, 10:37 AM
Policy enforcement
Suitable for SDN and
Network Virtualization.
(yes, no)
Suitable for Cloud-based
deployments
Supported with newer

OpenStack distributions
OpenStack
such as Queens and
Orchestration Yes
RHOSP 13
support (yes, no)
Available in the latest
OpenStack releases only
until backported to older
versions
VM hardware in the VM making the VMs
independence (yes, Yes hardware dependent
no) Live VM migration is not

supported
Excellent datapath
performance
Performance, CPU 25-28 Mpps Performance is maintained

core use for No CPU cores well with more flows and
datapath used for more or complex policy rules
processing (Mpps, datapath Frees all CPU cores from
No. of cores) processing datapath processing
resulting in high server
utilization and TCO
9 of 18 09-05-20, 10:37 AM
Model 5: OVS datapath is fully offloaded to the

SmartNIC, VMs are hardware-independent
OVS Offload with OpenStack

Relay Agent in Userspace Controller
VNF VNF
VM VM
vNIC vNIC
OVS Control
ovsdb-server
Relay Agent
ovs-vswitchd
DPDK DPDK
USER SPACE Full Offload
KERNEL
Kernel and Uses kernel-compliant and

SmartNIC upstreamed TC flower-based
Location of OVS
Fallback to offload of the OVS datapath
datapath (kernel,
kernel OVS for Offload mechanism is
user, NIC)
control traffic included in the RHEL 7.5
and new/first
10 of 18 09-05-20, 10:37 AM
distribution
Kernel-based offload makes

it easier to enhance features
leveraging other kernel
datapath implementations
flow
such as in Conntrack and
BPF

releases only until
Policy enforcement
using OVS Yes
Network Virtualization
datapath (yes, no)
Suitable for Cloud-based
deployments
Supported with newer

OpenStack OpenStack distributions such
Orchestration Yes as Queens and RHOSP 13
support (yes, no) Available in the latest
OpenStack releases only
until backported to older
versions
independence (yes, Yes independent and enabling
no) support of broad array of

guest operating systems and
11 of 18 09-05-20, 10:37 AM
live VM migration
Uses one of the two modes:

Relay Agent with
vhost/DPDK and Virtio 1.0, or
vDPA (vhost datapath
acceleration) with Virtio 1.1
Consumes 1-3 CPU cores for

processing the Relay Agent
in user space
Excellent datapath
performance
Performance, CPU 25-28 Mpps Performance is maintained

core use for No CPU Cores well with more flows and
datapath used for more or complex policy rules
processing (Mpps, datapath Frees all CPU cores from
No. of cores) processing datapath processing resulting
in high server utilization and
TCO
Model 6: OVS datapath is offloaded to the SmartNIC,

VMs are hardware-independent
OpenStack
Controller
OVS Offload with
Direct Virtio to NIC Nova
Neutron
VNF VNF
VM VM
OVS Control
12 of 18 09-05-20, 10:37 AM
OVS Control
vNIC vNIC
ovsdb-server
*+# (,$Direct
( .+/
- 0 ovs-vswitchd
Virtio
USER SPACE Full Offload
KERNEL
Uses kernel-compliant and

upstreamed TC flower-based
offload of the OVS datapath
Offload mechanism is
Kernel and included in the RHEL 7.5
SmartNIC distribution
Location of OVS Fallback to Kernel-based offload makes

datapath (kernel, kernel OVS for it easier to enhance features
user, NIC) control traffic leveraging other kernel
and new/first datapath implementations
flow such as in Conntrack and
BPF

releases only until
13 of 18 09-05-20, 10:37 AM
Model 7: OVS datapath is in the user space and some

functions are offloaded to the SmartNIC
OVS-DPDK Partial Offload OpenStack

QoS, security groups, Controller
Conntrack, overlay Nova
VNF VNF Neutron

VM VM
vNIC vNIC
OVS Control
ovsdb-server
OVS-DPDK Bridge ovs-vswitchd
DPDK DPDK
USER SPACE Partial
Offload
KERNEL
User space Difficult to enhance features

Partial offload of leveraging kernel datapath
Location of OVS
datapath implementations and
datapath (kernel,
functions to NIC innovations such as in
user, NIC)
or SmartNIC Conntrack and BPF
14 of 18 09-05-20, 10:37 AM
hardware using
DPDK Flow API
Policy
enforcement using Suitable for SDN and
Yes
OVS datapath Network Virtualization
(yes, no)
OpenStack
Orchestration Yes
deployments
support (yes, no)
making the VMs hardware
VM hardware
independent and enabling
independence Yes
support of broad array of
(yes, no)
guest operating systems
and live VM migration
Good datapath
performance, full line rate
for 10GbE networks
Performance degrades
Performance, CPU 10-15 Mpps
rapidly with more flows and
core use for 4-8 CPU cores
more or complex policy
datapath used for
rules
processing (Mpps, datapath
Consumes many CPU
No. of cores) processing
cores for datapath
performance resulting in
low server utilization and
TCO
15 of 18 09-05-20, 10:37 AM
Model 8: Both the OVS datapath and control plane are

offloaded to the SmartNIC
OVS Embedded in NIC OpenStack

No host OVS Controller
Tighter integration testing
Nova
VNF VNF Neutron

VM VM
vNIC vNIC SDN

Controller
OVN
USER SPACE
KERNEL
SmartNIC Useful for bare metal cloud

Location of OVS Control plane applications where the
datapath (kernel, runs in the user service provider has no
user, NIC) space on an ARM control over what host
16 of 18 09-05-20, 10:37 AM
or MIPS CPU on
the SmartNIC
Datapath slow operating system is used on

path runs in the the server
kernel or user
Best when SDN and cloud
space on the
orchestration is implemented
ARM and fast
via the SmartNIC (not
path runs on
through the host)
accelerator chip
available on
some SmartNICs

Network Virtualization
Non-Bare Metal Use Case:

When the host runs OVS
Policy
control plane and datapath,
enforcement
the version that runs on the
using OVS Yes
SmartNIC can easily go out
datapath (yes,
of sync with the version that
no)
runs on the host, causing
anomalies and feature
inconsistencies related to
SDN deployments

OpenStack deployments
Orchestration Yes Non-Bare Metal Use Case:
support (yes, no) When the host runs OVS
control plane and datapath,
17 of 18 09-05-20, 10:37 AM
the version that runs on the

SmartNIC can easily go out
of sync with the version that
runs on the host, causing
anomalies and feature
inconsistencies related to
cloud orchestration.

in the VM making the VMs
VM hardware
independence No hardware dependent
(yes, no) Live VM migration is not

supported
15-20 Mpps
No CPU cores
used for datapath
Performance,
processing Good datapath performance,
CPU core use for
OVS control full line rate for 10GbE
datapath
plane processing networks for small packets
processing
on the host and 25GbE networks for
(Mpps, No. of
consumes less mid-sized packets
cores)
than a CPU core.
This is eliminated
by this approach
18 of 18 09-05-20, 10:37 AM

Netronome-Ovs Offload PDF

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Netronome-Ovs Offload PDF

Uploaded by

Copyright:

Available Formats

OVS Offload Models Used with NICs and SmartNICs: Pros and Cons - ... about:reader?url=https://www.netronome.com/blog/ovs-offload-models-...

OVS Offload Models Used with NICs

As SmartNICs become more popular, more decision makers at

Model 1: OVS datapath in the kernel

VNF with Open vSwitch

VNF with Open vSwitch

OVS kernel module

Default for OpenStack switching,

Most mature and proven model

Broadest vendor support

Uses Virtio driver in the VM

Poor datapath performance

1-2 Performance degrades rapidly

Model 2: SR-IOV is used where the OVS datapath is

DPDK VNF with SR-IOV

Hardware dependent to the NIC

Kernel but not Mature and proven model

No Not suitable for SDN

Suitable for cloud-based

based policy enforcement is

Supported with most

Uses vendor-specific driver

Performance, CPU core Close to 30 Mpps

Model 3: OVS datapath is moved to the user space

DPDK VNF with

DPDK – Direct I/O to NIC or vNIC

Broad vendor support

Location of OVS Difficult to enhance features

Supported with many OpenStack

Uses Virtio driver in the VM

Good datapath performance

6-8 Performance degrades rapidly

Model 4: OVS datapath in the kernel is fully offloaded

Hypervisor KVM ovs-vswitchd

Summary of features, pros (in green) and cons (in red):

Uses kernel-compliant and

Available in the latest kernel

Supported with newer

Uses vendor-specific driver

VM hardware in the VM making the VMs

independence (yes, Yes hardware dependent

no) Live VM migration is not

Performance, CPU 25-28 Mpps Performance is maintained

Model 5: OVS datapath is fully offloaded to the

OVS Offload with OpenStack

USER SPACE Full Offload

Summary of features, pros (in green) and cons (in red):

Kernel and Uses kernel-compliant and

Kernel-based offload makes

Available in the latest kernel

Supported with newer

Uses Virtio driver in the VM

VM hardware making the VMs hardware

independence (yes, Yes independent and enabling

no) support of broad array of

Uses one of the two modes:

Consumes 1-3 CPU cores for

Performance, CPU 25-28 Mpps Performance is maintained

Model 6: OVS datapath is offloaded to the SmartNIC,

Summary of features, pros (in green) and cons (in red):

Uses kernel-compliant and

Kernel and included in the RHEL 7.5

Location of OVS Fallback to Kernel-based offload makes