BRKSDN 2411

#CLUS
NFV Performance -
Challenges and Solutions
Ian Wells, Distinguished Engineer
Nikolai Pitaev, Engineer, Technical Marketing
BRKSDN-2411
#CLUS
“What’s in it for me?"
This session will help you to understand bottlenecks, key performance
parameters and optimization techniques for Virtual Network Functions (VNF) on
an NFV platform (NFVI and VIM).
In this session Out of scope
Introduction and overview: what is NFV, NFV
applications versus normal cloud applications, Generic introduction to virtualization basics.
performance measurement.
Bottlenecks on different levels:
1. Physical level (BIOS, NIC)
Detailed description of one specific VNF use
2. Host OS / Hypervisor
case.
3. IO and vSwitch (SR-IOV, VPP, OVS-DPDK)
4. VNF (using hugepages, vCPU pinning)
Performance optimization based on real life
Troubleshooting and debugging deep dive.
projects with Cisco VNFs (CSR1000V, XRv).
Target audience is technical attendees looking for basic understanding of VNF

performance challenges and solutions.
#CLUS BRKSDN-2411 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 3
Why should I care?
Real life example:
Top customer, huge project, complex feature set
200 Mbps per VM throughput without any optimization
800 Mbps per VM after performance optimization
Key factors used to improve performance:

CPU: vCPU pinning, Hyper-threading, ...
IO: Queue size, vSwitch worker/PMD threads, ...
Agenda
• Introduction
• Bottlenecks in a Linux/KVM/QEMU environment
• Methodology for Performance testing
• Finding optimal VNF setup
• Future and Conclusion
Cisco Webex Teams
Questions?
Use Cisco Webex Teams to chat
with the speaker after the session
How
1 Find this session in the Cisco Live Mobile App
2 Click “Join the Discussion”
3 Install Webex Teams or go directly to the team space
4 Enter messages/questions in the team space
Webex Teams will be moderated cs.co/ciscolivebot# BRKSDN-2411

by the speaker until June 16, 2019.
#CLUS © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 6
Introduction
What is Network Function Virtualization (NFV)
• NFV is a network architecture concept

that uses the technologies of IT
virtualization to virtualize entire classes
of network node functions.
• NFV MANO stands for management and

organization and is a working group of
the European Telecommunications
Standards Institute Industry
Specification Group (ETSI).
• NFV performance comes down to the

common denominator, which is
Linux/KVM/QEMU optimization.
What do you need from an NFV infrastructure?
The three pillars of cloud: Network, Compute and Storage

• The network is key and has to be fast and reliable
• Workloads rely on efficient and optimized compute†
• Storage needs vary by use case but is usually secondary
A good VNF will run badly on a poor NFVI – choose and tune your
software and hardware carefully.
† some more than others – see later, “realtime”
Why is it different from a compute-centric cloud?
Network centric use cases – like Virtual Packet Core, virtual Managed
Services, SD-WAN, vBNG – sit in the flow of traffic rather than
answering requests – and it changes the nature of the traffic.
HTTP, 1500 Byte Cloud
NFV
IMIX
Internet
Common VNF variations
Server
VM VM
Load balancer
Load balancer
VM VM
VM VM
Why it is different from physical
Physical Router: Virtual Router:

• Software and Hardware from a • Different vendors: HW, VIM,
single source VNF(s), Orchestration
• Verified system performance • Conservative (and sometimes not
comparable) performance numbers
• Clear process for Fixes / Features /
per component
Improvements
• Fixes / Features / Improvements
required from multiple vendors
NFV performance is needed in all markets and
segments
Service Providers
Public Cloud
Enterprise
Private Cloud
NFV Community Landscape
Standardization Open Source Vendors SP

Bodies Communities • Provide solutions • Compare solutions
• standards • Reference platforms • Contribute • Integrate
• recommendations & interfaces • Deploy
• Components • Contribute
• Interoperability
Different definitions, methodology and goals
Bottlenecks and
benchmarks
What do you need?
Performance Consistency
• Ultimate measurements will be • Maintaining this performance over
business relevant, e.g. customers extended periods
per server
• Some features of modern
• But they usually end up meaning hardware (e.g. SpeedStep and
‘fast network performance’ – Turbo Boost) will allow you to get
measured in packets per second or one VM running fast for short
Gbps periods but can’t maintain this on a
loaded system
• PPS is important for vSwitches –
processing cost and limitations are • How is it affected by failover,
related to number of packets upgrades, maintenance…
processed
Using public clouds for NFV
It can be done Use it wisely
• For instance, running CSRs inside of AWS or • Public cloud SLA – ‘it will work except when
Google cloud to terminate IPSec connections it doesn’t’
• We’ve even run a mobile packet core in • Performance consistency – machines can be
AWS overcommitted and VMs will slow down
when they get more full
• Location – not always in the flow of traffic;
sometimes thousands of miles out of the
way
• Traffic type – clouds work for TCP and UDP
– not for multicast, L2, MPLS, …
• DoS – test a VNF in a public cloud? That’s a
DoS attack!
Throughput interpretation
Compute node
1Mpps
VNF
vlan1
traffic
generator
vswitch
vlan2
1Mpps
VNF
sent by traffic generator per direction
What Throughput number to report?

A. NDR = 1 Mpps B. NDR = 2 Mpps
C. NDR = 4 Mpps D. All choices are valid
Bottlenecks exist on different levels
Differentiate between Platform performance x86 Host

and VNF performance Intra-VM Bottleneck
Guest Guest
User Kernel
VM1
Application .. VM n
Application
At platform level, other bottlenecks may Hypervisor Bottleneck I/O driver I/O driver
affect throughput
• Physical NIC capacity vNIC
Boundary Bottleneck vNIC
Host User
• Virtual Switching
/Qemu
• Hypervisor performance
vSwitch Bottleneck vSwitch
• CPU share
Kernel
• vNIC connection
KVM
Host
pNIC Driver pNIC Driver
In throughput testing, there will ALWAYS be pNIC Bottleneck pNIC pNIC
at least one bottleneck!

Pkt Pkt Pkt Pkt
Pkt Pkt Pkt Pkt
Pkt Pkt Pkt Pkt
Understand WHICH bottleneck is ‘active’ and

WHEN bottlenecks switch – the biggest
bottleneck will hide the others
Symptoms for active bottlenecks at different levels
* The simple version
Symptom Problem Solutions

Packets are dropped Can’t go fast enough Use fewer packets
Symptoms for active bottlenecks at different levels
* For later reference
Symptom Problem Solutions
Drops on the physical NIC Can’t get packets into host as Virtual switch issues – faster switch, or
fast as they’re arriving skip switch with PCI passthrough
Drops on tx from vSwitch to VM is struggling to accept VM starved of CPU – give it more
VM packets CPUs if it can use it, stop other
processes competing with it for CPU
VM is stalling – VM competing for CPU
VM generally accepting traffic with something else – find ways to
but occasional queue isolate it better
overflows Optimize VM placement
Faster VM code – DPDK
VM isn’t fast enough SRIOV
Drops on VM output vSwitch CPU problems Give the vSwitch more cores
NFVBench Test Tool
Aim:
• Simulate a VNF running on test infrastructure
• Find out its performance
Test Node • Find out why it doesn’t perform better
TOR-A
TRex
TOR-B
NFVbench
VPC Compute Node N
Compute Node 2
Compute Node 1
Different packet paths
single VNF chain (PVP) NIC Compute node

NIC Compute node A
DC-SW vswitch 2-VNF chain
traffic (inter-node PVVP) vswitch
VNF1a
generator VNF
traffic
DC-SW
generator
vswitch VNF1b
NIC Compute node B
NFVBench report
End to end view of drops in the whole path! Traffic generator (TRex)
+----------------------+----------+-----------------+---------------+---------------+-----------------+---------------+---------------+
| Interface | Device | Packets (fwd) | Drops (fwd) | Drop% (fwd) | Packets (rev) | Drops (rev) | Drop% (rev) |
| traffic-generator | trex | 3,561,150,633 | | | 3,561,152,091 |

Physical switch
+======================+==========+=================+===============+===============+=================+===============+===============+
0 | 0.0000% |
+----------------------+----------+-----------------+---------------+---------------+-----------------+---------------+---------------+
| vni-4096 | n9k | 3,561,150,633 | 0 | 0.0000% | 3,561,152,091 | 0 | 0.0000% |
+----------------------+----------+-----------------+---------------+---------------+-----------------+---------------+---------------+
| vxlan_tunnel1 | vpp | 3,561,150,433 | 200 | 0.0000% | 3,561,152,091 | 0 | 0.0000% |
+----------------------+----------+-----------------+---------------+---------------+-----------------+---------------+---------------+
| VirtualEthernet0/0/0 | vpp | 3,561,150,433 | 0 | 0.0000%
VPP and VNF
| 3,561,152,091 | 0 | 0.0000% |
+----------------------+----------+-----------------+---------------+---------------+-----------------+---------------+---------------+
| VirtualEthernet0/0/8 | vpp | 3,561,150,433 | 0 | 0.0000%
incoming and outgoing
| 3,561,152,091 | 0 | 0.0000% |
+----------------------+----------+-----------------+---------------+---------------+-----------------+---------------+---------------+
| vxlan_tunnel0 | vpp | 3,561,150,433 | 0 | 0.0000% | 3,561,152,091 | 199 | 0.0000% |
+----------------------+----------+-----------------+---------------+---------------+-----------------+---------------+---------------+
| vni-4097 | n9k | 3,561,150,433 | 0 | 0.0000% | 3,561,152,290 | 0 | 0.0000% |
+----------------------+----------+-----------------+---------------+---------------+-----------------+---------------+---------------+
| traffic-generator | trex | 3,561,150,431 | 2 | 0.0000% | 3,561,152,290 | | |
+----------------------+----------+-----------------+---------------+---------------+
Physical switch
Traffic generator (TRex)
Methodology for
performance
testing
Optimization depends on the use case and VNF
One high throughput VNF, which takes the whole server with 28
cores needs different optimization compared to 14 x 2 vCPU VNFs.
CSR 1000V as vBNG XRv 9000 as vBNG

2 vCPU VM: 28 vCPUs VM:
Scale 8,000 Sessions 32,000 Sessions
5 Gbps IMIX 80 Gbps IMIX
Use Cases vPTA, vLAC, vLNS, vLTS vPTA, LNS on radar
Methodology for VNF performance optimization
1. Define your test parameters:

RFC 2544 is an RFC, but which IMIX sizes, duration of test runs, PDR or NDR
2. Know your the baseline numbers:

VNF, vSwitch / IO, Hardware performance without optimization
3. Design your system for the best performance using bottom-up

approach:
BIOS: Hyper-threading, power consumption profile
Host / CPU / Memory: pinning, NUMA socket design BIOS Host
I/O / vSwitch: what is the best I/O for me?
VNF: what is the best VNF for me? IO
VNF
vSwitch
Example: vBNG CSR 1000V optimization
1. Test parameters:
RFC 2544 with 8 iterations, Cisco IMIX, 1 Min. per test run, PDR=0.01%
2. Select VNF and determine the baseline numbers:

CSR 1000V with 2.5 Gbps IMIX throughput with 2 vCPU, any IOS XE RLS
3. Design for the best performance using bottom-up approach:

BIOS: Hyper-threading off, Speedstep off
Host / CPU / Memory: RHEL generic tuning
IO / vSwitch: SR-IOV
VNF: IOS XE 16.3 run with pinning and hugepages
Result: 5 Gbps per VNF
Will faster CPU provide linear NFV performance
increase?
Two servers with:
Impact of Different server Core Speeds
• 16 core @ 3.2 GHz CSR 1000v, IMIX, SR-IOV, IOS XE 16.3
• 24 core @ 2.6 GHz
SR-IOV with 2 x 10 GE Ports used 7.367
3.2 GHz, 16 core
CEF (IP forwarding) tested 20
6.001
For 1 VM, performance increase 2.6 GHz, 24 core
18.101
proportional to the CPU Cycle
difference 3.2 7.4 0 5 10 15 20 25
≈ linear!
2.6 6 1x2vCPU 3x2vCPU
For 3 VMs - not proportional

• IO-Limit – bottleneck switched from CPU to IO
Drop Rate Definition has significant impact to
throughput
Typical definitions for Drop Rates Throughput as a func on of acceptable
Traffic Loss (%, normalized, KVM, XE 3.13)
• Non-drop Rate (NDR) = 0 packets lost 180%
• Partial Drop Rate (PDR) 0.01% or 0.05% 160%
Normalized Throughput (%, NDR = 100%)

are lost on average
140%
120%
100%
Small relaxation of PDR definition can lead 80%
60%
to significant higher throughput: 40%
• If your use case accepts PDR of 0.05%, 20%
you will measure ~40% higher throughput 0%

0.00 0.05 0.10 0.15 0.20 0.25 0.30 0.35 0.40 0.45 0.50 0.55 0.60 0.65 0.70 0.75
compared to NDR
% of acceptable traffic loss per VM
% increase in Throughput
RFC 2544 parameters impact performance
results
binary search = the next transmission rate is one half of the difference between the previous
failed and the previous successful rate.
Example: 800 𝑓𝑎𝑖𝑙 −0 (𝑠𝑢𝑐𝑐𝑒𝑠𝑠)
400 Mbps = 2
Key parameters: resolution, duration of single run, success criteria (drop rate)
Utilizing all resources:

One Flow vs. Multiple Flow – use all data plane cores
Optimizations
Key BIOS optimization parameters BIOS Host
IO
VNF
vSwitch
Configuration recommendation: choose carefully

Hyper-threading
allows a single physical core to behave as TWO logical cores
Does not double performance → different threads require different resources
at the same time
Configuration recommendation: Speedstep OFF
CPU Speedstep
Allows physical CPU clock speed to be changed dynamically by software
Configuration recommendation: Turbo boost OFF

CPU Turbo boost
Enables the processor to run above its base operating frequency via dynamic
control of the processor's clock rate
Key Host optimization parameters BIOS Host
IO
VNF
vSwitch
The infrastructure has to provide, and the VNF has to use:

Hugepages: bigger virtual memory page sizes
Default size is 4K, 2MB and 1GB can be used on x64 platforms
The good: they make memory access faster
The bad: you can’t overcommit memory
vCPU Pinning: strictly associate vCPUs with physical CPUs
Emulator thread pinning: strictly associate QEMU processing with pCPUs
CPU Isolation: Dedicate VM’s CPUs to VMs – don’t run system tasks on them
Why hugepages? BIOS Host
IO
VNF
vSwitch
• Minimum size ethernet frame: 84 bytes

inter-frame MAC MAC Minimum
CRC
gap preamble header Payload Size
4bytes
12 bytes 8 bytes 14 bytes 46 bytes
• 10GbE -> 14.88Mpps or 1 frame every 67.2ns

• 3Ghz CPU -> 203 cycles per frame
A TLB miss costs upward of 20 cycles per TLB miss!
How to avoid TLB misses BIOS Host
IO
VNF
vSwitch
TLB
4kb Page
4kb Page
4kb Page
4kb Page
...
... CPU
5 entries:
4kb Page
20kb Addressed Memory
How to avoid TLB misses BIOS Host
IO
VNF
vSwitch
TLB
2Mb Page
2Mb Page CPU
2 entries:
4Mb Addressed Memory

CPU scheduling BIOS Host
IO
VNF
vSwitch
If your VM loses time in big chunks, it will drop packets when the
input queue fills up
If your VM loses time in smaller chunks, it will underperform
• Remember that 67.2ns? ‘Small’ is a relative term
• (processes usually get scheduled for 25ms at a time)
CPU scheduling BIOS Host
IO
VNF
vSwitch
If you have two workloads on a CPU, the Linux kernel schedules them for
you
If you’re running a VM:
• It loses some slices of time while other processes run
• It loses some slices of time when the kernel runs
• It loses some slices of time when interrupts happen
• It can’t do anything about this
If you’re running more VMs cores than physical cores, this will happen
Real-time kernels BIOS Host
IO
VNF
vSwitch
There are many ways to define “real time”
• Hard realtime is what we’re aiming for – we get predictable CPU time
• Soft realtime is available from a special ‘realtime’ or ‘pre-emptive’ kernel – it keeps
the interruption lengths low so that your desktop doesn’t get jerky, but they still
happen and they still take the same amount of time away from the VM
The pre-emptive kernel doesn’t solve your problem
Tuning for isolation is required
• Keep other processes off of your VM CPUs
• Even if they have nothing to do, if they could run there, the kernel will check to see if they’re
ready
• Allocate your VMs to specific CPUs, and your CPUs to specific VMs
• Redirect hardware interrupts
• If your VM uses a clock interrupt, your CPU will receive a clock interrupt no matter what you do
Key IO / vSwitch optimization parameters BIOS Host
IO
VNF
vSwitch
Copying a packet from one area of memory to another using CPU is expensive
Avoid multiple packet copy operations by choosing I/O technology:
PCI passthrough (SRIOV*) lets the physical NIC copy packets straight into the VM
DPDK is a userspace library – using it bypasses the kernel, context switches
VPP (a DPDK app) works processes multiple packets in batches for CPU efficiency
OVS-DPDK is a DPDK-ized version of OVS
Kernel space forwarders like conventional OVS are very costly
Optimize QEMU queue size for better absorption of packet arrival rate - example will
follow later in the presentation
Pin vSwitch worker threads to dedicated physical cores
User space and kernel space forwarders BIOS Host
IO
VNF
vSwitch
Virtual Kernel Driver Virtual Kernel Driver Virtual Kernel Driver

Machine Machine Machine
virtqueue
QEMU FE virtqueue
QEMU FE
QEMU Driver User-space

Compute Host Compute Host Compute Host
User space User space switch User space
Tap Device
OVS / LB
Compute Host Compute Host Compute Host
Kernel space Kernel Drivers Kernel space Kernel space
eth1 eth1 eth1
Kernel space User space SRIOV
DPDK, the Data Plane Development Kit BIOS Host
IO
VNF
vSwitch
User space forwarding

Dedicated to fast forwarding above anything else
Steals a NIC from the kernel and talks directly to the hardware
• no context switches
• no problems with address spaces, no copying of data
Will use 100% of its cores to forward packets – even if there are no packets
available
• Interrupts cause context switches and take time; faster to just ask if there
are new packets when there’s nothing better to do
Can be used both as a vSwitch and inside the VM
DPDK forwarders BIOS Host
IO
VNF
vSwitch
OVS-DPDK: fd.io VPP:

- A DPDK implementation of OVS(!) - An open source forwarder written
- Works the same as OVS with flow exclusively for userspace packet
based packet matching and forwarding
forwarding on 5-tuple matches - Processes packets in batches
(vectors) to use the CPU more
Can be problematic with user-to- optimally
internet flows
- lots of users x lots of internet = Doesn’t care about flows; uses MAC
lots of flows address table based forwarding
How many DPDK worker threads do I need?
BIOS Host
IO
VNF
vSwitch
Number of DPDK worker threads can have positive impact on total system
throughput if I/O path is the bottleneck.
Placement of worker threads on sockets / NUMA nodes does matter!
Balance the interface association to worker threads on the sockets
System Throughput Effect of allocating different number of VPP Worker Threads
(2vCPU, CEF, 0.01% PLR, IOS XE 16.3)
25
System Throughput
20
15
10
0
1 2 3 4 5
1 Worker 6.31 5.286 4.395 4.047 3.979
2 Workers 6.13 8.365 8.199 8.919 9.077
4 Workers 6.328 11.933 23.32
Key VNF optimization parameters BIOS Host
IO
VNF
vSwitch
Don’t forget to use all those infrastructure features from before…

Plus:
Pin the CPUs inside your VMs
Use hugepages inside your VMs
Do one of these:
• Avoid overcommitting resources – if you need more CPU or memory than
you have your VMs will fight it out
• Depending on your NFV use case, consider multiple VMs per server and
oversubscription
Design your CPU mapping for better BIOS Host
performance IO
vSwitch
VNF
• x86 Server with 2 NUMA sockets, 8 cores each = 16 cores total

• VPP or OVS-DPDK as vSwitch, NICs are mapped to worker / PMD threads
• Select proper mapping of the physical NICs
• 6 CSR 1000V VMs with 2vCPU each
CSR1 CSR2 CSR3 CSR4 CSR5 CSR6
Linux VPP Linux VPP
Emul worker1 Emul worker2
vCPU0 vCPU1 vCPU0 vCPU1
vCPU0 vCPU1 vCPU0 vCPU1 vCPU0 vCPU1 vCPU0 vCPU1
CPU00 CPU01 CPU02 CPU03 CPU04 CPU05 CPU06 CPU07 CPU10 CPU11 CPU12 CPU13 CPU14 CPU15 CPU16 CPU17
Socket0 Socket1
Physical Physical
Interface 1 Interface 2
Same example, different design BIOS Host
IO
VNF
vSwitch
Do you see any room for improvement in following design?
CSR1 CSR2 CSR3 CSR4 CSR5 CSR6

Linux VPP VPP VPP
Emul worker1 worker2 main
vCPU0 vCPU1 vCPU0 vCPU1 vCPU0 vCPU1 vCPU0 vCPU1 vCPU0 vCPU1 vCPU0 vCPU1
CPU00 CPU01 CPU02 CPU03 CPU04 CPU05 CPU06 CPU07 CPU10 CPU11 CPU12 CPU13 CPU14 CPU15 CPU16 CPU17
Socket0 Socket1
To improve: Physical
Interface 1
Physical
Interface 2
1. physical NIC – VPP mismatch
2. CSR3 – socket crossing “tax”
3. Emulator pin for VMs 4-6 on different socket
Example
Example: Multi-VM and Multi-Feature CSR
1000V performance with SR-IOV and VPP
x86 Host (FD.io VPP)
Test methodology and Profile: VM 1
Application .. VM n
Application
Guest User
• Features
DPDK-VirtIO Ptr
DPDK-VirtIO
Ptr Ptr Ptr
 Bi-Directional NAT, Firewall, ... Ptr Ptr
Ptr
Ptr
Ptr
Ptr
Ptr
Ptr
 100 VRFs per VM
Kernel
Guest
• 2 min test time
• UDP IMIX IPv4 vNIC Shared Pkt Mem vNIC
Host User
/Qemu
(vHost_user) Pkt Pkt Pkt Pkt
(vHost_user)
Pkt Pkt Pkt Pkt
Server details:
FD.io VPP
Kernel
Host
• RHEL 7.2 on UCS C240 M4L pNIC Driver
• Intel(R) Xeon(R) CPU E5-2690 v3 @ 2.60GHz, 2 Pkt Pkt

pNIC
Pkt Pkt
sockets, 12 cores per socket Pkt

Pkt
Pkt
Pkt
Pkt
Pkt
Pkt
Pkt
Traffic Generator
Multi-Feature Test Results
Insignificant difference between SR- Total System Throughput vs. Number of VNFs
IOV and VPP 2vCPU, multi-feature Set, IOS XE 3.16, 0.01% PLR
20
18
Very good linearity 16
System Throughput (Gbps)

14
12
Reaching 6 Gbps with 10 VMs running 10
comprehensive Multi-Feature set 8

6
4
2
0
1 2 3 4 5 6 7 8 9 10 11
VPP 0.652 1.291 1.891 2.486 3.67 4.869 6.016
SR-IOV 0.669 1.309 1.931 2.536 3.778 5.01 6.35
Benchmark 0.669 1.34 2.01 2.68 3.35 4.02 4.69 5.36 6.03 6.7
Default QEMU queue size was the main bottleneck
Effect of Qemu Queue Size changes on total system throughput,

10x2vCPU CSR 1000v, IMIX, multi-feature configuration
7000
Total System Throughput
6000
5000
4000
3000
2000
1000
0
256x256 1024x1024 208x2048
IOX XE 3.16 675 6016
IOS XE 16.3 874 6030 6068
Queue sizes increase from 256 (default) to 1024 packets improved

performance by factor 10!
Insignificant benefit from 2048/2048 queue size
Impact of features to the system performance
CEF results with SR-IOV Total System Throughput vs. Number of VNFs
2vCPU, multi-feature Set, IOS XE 16.3, 0.01% PLR
Variability due to features 40
System Throughput (Gbps)

35
IPv6 GRE 30 CEF

25
NAT, Firewall, 20
15
VRF-aware firewall 10 Multi-Feature Set

5
Access Lists (for filtering NAT) 0

1 2 3 4 5 6 7 8 9 10 11
VPP 0.67 1.309 1.921 2.491 3.05 3.67 4.265 4.836 5.44 6.03
100 VRFs SR-IOV 0.67 1.347 2.003 2.602 3.74 5.01 6.29
Benchmark 0.67 1.34 2.01 2.68 3.35 4.02 4.69 5.36 6.03 6.7
CEF Benchmark 7.367 14.734 22.101 29.468 36.835 44.202 51.569 58.936 66.303 73.67 81.037
CEF VPP 6.273 12.343 23.894
The Future
Realtime workloads
Realtime has many definitions – here, we mean ‘guaranteeing to do work
before it is due 100% of the time’, hard realtime
• 5G is coming, and with it it brings Cloud RAN
• Cloud RAN puts cell site radio control into virtual machines
• Cell site radio likes to hear from its software regularly
• … like, 1000 times a second regularly
• Do you work within 1ms or don’t bother
• And if you don’t bother, everyone’s calls hang up
This is a new field for NFV, one that we’re already working in; we’ll keep you
updated as we learn more
Why are we talking about VMs?
VMs are the technology of today

• Broad industry adoption
• Actually well suited to VNFs
Containers are the next step forward
From there – who knows?
What are we looking for in a platform?
How to move packets around from workload to workload

How to use multiple workloads from multiple vendors – and still be
able to say where the problem lies
• On your laptop – is your laptop running slow, or is it the program
you’re using, or is it the virus scanner slowing everything down?
• We need accountability in the platform we use
Simplicity
Containers as we use them today
Host
Limited options for isolation

Container Container
Limited options for performance tuning
Interface Interface One (kernel TAP) interface per container

Kernel All traffic via kernel
NIC NIC owned by kernel
One vision for NFV
Host CPU pinning and hugepages
Shared memory packet paths (memif)

memif
PCI passthrough SRIOV devices to user
DPDK app
Container Container
DPDK app space forwarders
SRIOV Conventional networking for control traffic
Interface Interface
NIC
Fast traffic avoids kernel completely

Kernel
NIC
What is memif?
• Packet based shared memory interface for

user-mode application
• Container friendly (no privileged
containers needed)
• Support for multiple queues (incl.
asymmetric configurations)
• Take security seriously
• Lightweight library for apps - allows easy
creation of applications which
communicate over memif
Community progress
Divide and conquer – what NFV requires for packets is not what
Kubernetes offers for traditional microservices
Network service mesh project
• Inspired by Istio microservice mesh
• Delivers packets to containers based on service definitions
• Provides appropriate – SRIOV, memif or other – network
interfaces that are efficient at delivering packets
Summary: Key messages
1. Unlike hardware routers, any VNF setup has to be configured

for optimal performance on several levels:
BIOS Host
IO
VNF
vSwitch
2. It’s not rocket science, but needs proper design.
3. Use presented methodology to understand WHICH

bottleneck is ‘active’ and WHEN bottlenecks switch.
Complete your
online session • Please complete your session survey
evaluation after each session. Your feedback
is very important.
• Complete a minimum of 4 session
surveys and the Overall Conference
survey (starting on Thursday) to
receive your Cisco Live water bottle.
• All surveys can be taken
in the Cisco Live Mobile App.
Cisco Live sessions will be available for viewing
on demand after the event at ciscolive.cisco.com.
Continue your education
Demos in the
Walk-in labs
Cisco campus
Meet the engineer

Related sessions
1:1 meetings
Thank you
#CLUS
#CLUS

BRKSDN 2411

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

BRKSDN 2411

Uploaded by

Copyright:

Available Formats

#CLUS

Target audience is technical attendees looking for basic understanding of VNF

Key factors used to improve performance:

Webex Teams will be moderated cs.co/ciscolivebot# BRKSDN-2411

• NFV is a network architecture concept

• NFV MANO stands for management and

• NFV performance comes down to the

The three pillars of cloud: Network, Compute and Storage

† some more than others – see later, “realtime”

HTTP, 1500 Byte Cloud

Physical Router: Virtual Router:

Standardization Open Source Vendors SP

Different definitions, methodology and goals

What Throughput number to report?

C. NDR = 4 Mpps D. All choices are valid

Differentiate between Platform performance x86 Host

In throughput testing, there will ALWAYS be pNIC Bottleneck pNIC pNIC

at least one bottleneck!

Understand WHICH bottleneck is ‘active’ and

Symptom Problem Solutions

Symptom Problem Solutions

single VNF chain (PVP) NIC Compute node

| traffic-generator | trex | 3,561,150,633 | | | 3,561,152,091 |

Traffic generator (TRex)

CSR 1000V as vBNG XRv 9000 as vBNG

Use Cases vPTA, vLAC, vLNS, vLTS vPTA, LNS on radar

1. Define your test parameters:

2. Know your the baseline numbers:

3. Design your system for the best performance using bottom-up

2. Select VNF and determine the baseline numbers:

3. Design for the best performance using bottom-up approach:

Result: 5 Gbps per VNF

For 3 VMs - not proportional

• Partial Drop Rate (PDR) 0.01% or 0.05% 160%

Normalized Throughput (%, NDR = 100%)

Small relaxation of PDR definition can lead 80%

• If your use case accepts PDR of 0.05%, 20%

you will measure ~40% higher throughput 0%

Utilizing all resources:

Configuration recommendation: choose carefully

Configuration recommendation: Turbo boost OFF

The infrastructure has to provide, and the VNF has to use:

vCPU Pinning: strictly associate vCPUs with physical CPUs

Emulator thread pinning: strictly associate QEMU processing with pCPUs

• Minimum size ethernet frame: 84 bytes

• 10GbE -> 14.88Mpps or 1 frame every 67.2ns

A TLB miss costs upward of 20 cycles per TLB miss!

2Mb Page CPU

4Mb Addressed Memory

Pin vSwitch worker threads to dedicated physical cores

Virtual Kernel Driver Virtual Kernel Driver Virtual Kernel Driver

QEMU Driver User-space

eth1 eth1 eth1

Kernel space User space SRIOV

User space forwarding

OVS-DPDK: fd.io VPP:

Don’t forget to use all those infrastructure features from before…

• x86 Server with 2 NUMA sockets, 8 cores each = 16 cores total

CSR1 CSR2 CSR3 CSR4 CSR5 CSR6

 100 VRFs per VM

• Intel(R) Xeon(R) CPU E5-2690 v3 @ 2.60GHz, 2 Pkt Pkt

sockets, 12 cores per socket Pkt

Very good linearity 16