A REPORT

ON

DVS PROPERTY ENHANCEMENT OF ENS SWITCH IN

VMWARE NSX-T

By

Prakhar Gupta (2019H1030157H)

AT

VMware Software India Pvt. Ltd. , Bangalore

A Practice School II of

BIRLA INSTITUTE OF TECHNOLOGY & SCIENCE, PILANI

(March, 2021)
 A REPORT

ON

DVS PROPERTY ENHANCEMENT OF ENS SWITCH IN

VMWARE NSX-T

By

Name of the Student ID No. Discipline

Prakhar Gupta (2019H1030157H) M.E. Computer Science

Prepared in the partial fulfillment of the

Practice School II Course

AT

VMware Software India Pvt. Ltd. , Bangalore

A Practice School II of

BIRLA INSTITUTE OF TECHNOLOGY & SCIENCE, PILANI

(March, 2021)
 Abstract

BIRLA INSTITUTE OF TECHNOLOGY AND SCIENCE

PILANI (RAJASTHAN)

Practice School Division

Station: VMware Software India Pvt. Ltd Centre: Bangalore

Duration: 6 months Date of Start: 11th January, 2021

Date of Submission: 21st March, 2021

Title of the Project: DVS Property Enhancement of ENS Switch in VMware NSX-T

ID No. : 2019H1030157H
Name : Prakhar Gupta
Discipline : M.E. Computer Science

Name and Designation of the experts:

Mr. Binu Pillai - Senior Manager, NSX Data Plane for vSphere (Palo Alto)
Mr. Ankur Sharma - Staff Engineer II (Palo Alto)
Mr. Alex RN- Senior Software Engineer (Palo Alto)

Name of the PS Faculty: Mr. Chandra Shekar R K

Key Words: DVS, ENS, TLB, ESXi, DPDK

Project Areas: NSX Data Plane for vSphere, Enhanced Network Stack, Load Balancer, Network
Virtualization, Networking

Abstract: VMware developed an Enhanced Network Stack, So to meet the increasing

performance requirements of the NFV workloads. The ENS employs dedicated kernel threads to
process the network traffic. And these threads continuously poll the virtual and physical NIC
ports of the virtual switch for processing the packets. The Thread Load Balancer (TLB) uses
different metrics to periodically reallocate the ports to the kernel threads for increased packet rate
and lower packet drop.
This document describes the customer requirement and solutions which is setting default metric
for the system as well as for the two modes in which ENS enabled vSwitch works by doing an
enhancement of dvs property which can be used to apply properties on distributed switches.

Prakhar Gupta
Signature of Student Signature of PS Faculty
Date: 21st March, 2021

3
 BIRLA INSTITUTE OF TECHNOLOGY AND SCIENCE
PILANI (RAJASTHAN)

PRACTICE SCHOOL DIVISION

Response Option Sheet

Station: VMware Software India Pvt. Ltd Center: Bangalore

ID No. & Name: 2019H1030157H, Prakhar Gupta

Title of the Project: DVS Property Enhancement of ENS Switch in VMware NSX-T

Usefulness of the project to the on-campus courses of study in various disciplines. Project should
be scrutinized keeping in view the following response options. Write Course No. and Course
Name against the option under which the project comes.

Refer Bulletin for Course No. and Course Name.

Code No. Response Option Course No.(s) & Name
1. A new course can be designed out of this Linux Kernel Programming
project.
2. The project can help modification of the CS G525 - Advanced
course content of some of the existing Computer Networks
Courses
3. The project can be used directly in some of
the existing Compulsory Discipline Courses
(CDC)/ Discipline Courses Other than
Compulsory (DCOC)/ Emerging Area (EA),
etc. Courses
4. The project can be used in preparatory
courses like Analysis and Application
Oriented Courses (AAOC)/ Engineering
Science (ES)/ Technical Art (TA) and Core
Courses.
5. This project cannot come under any of the
above mentioned options as it relates to the
professional work of the host organization.

Prakhar Gupta ________________

Signature of Student Signature of Faculty
Date: 21st March, 2021 Date:

4
 Acknowledgement

I would like to take this opportunity to express my gratitude towards the head of this organization,

VMware, for providing me with this opportunity to be a part of VMware as an intern. I would like to

express my very great appreciation to Mr. Binu Pillai (Senior Manager) for his guidance,

motivation, valuable and constructive suggestions. I wish to acknowledge the assistance given

by My Mentor Mr. Ankur Sharma, (Staff Engineer II) for all the support and day to day

activities here. I express my warm regards to my other mentor Mr. Alex RN (Senior Software

Engineer) for helping me out with setting up the development environment and giving ramp up

sessions.

I would like to thank our PS faculty, Prof. Chandra Shekar R K for his constant support

during this program and for always being readily approachable when in need.

I would also like to thank my fellow interns who helped me understand various things.

5
 Table of Contents

I. Abstract 3

II. Response Option Sheet 4

III. Acknowledgement 5

1. Introduction 7

2. Literature Survey 8

3. Problem Statement 10

4. Development Setup 10
4.1 Development Build Setup 10
4.2 Building and deploying ESXi hypervisor 10
4.3 Building and deploying NSX data path 10
4.4 Development Flow 10

5. Proposed Solution 11

6. Learnings 11

7. References 12

6
1. Introduction

VMware provides cloud computing and virtualization software and services. And these
virtualization software creates an abstraction layer over computer hardware that allows the
hardware elements of a single computer to be divided into multiple virtual computers. And these
services and VMware products enable enterprises to build Software-defined data centers, which
extends virtualization concepts such as abstraction, pooling, and automation to all data center
resources: compute, storage and networking. VMware core product NSX provides network and
security virtualization products for the SDDC that enables networks to be virtualized and control
them logically in software.

For improving the network performance, VMware developed Enhanced Network Stack (ENS)
that provides a fast packet processing path for its Software Forwarding Element called the NSX
vSwitch. Inspired by DPDK, VMware uses few of its techniques in the ESXi kernel to improve
the data plane processing eciency. ENS has a simpler packet buffer representation, employs
dedicated polling contexts for processing the packets, performs NUMA aware memory accesses
and has lockless shared data structures.

An ENS enabled vSwitch employs dedicated kernel threads to poll, process the packets received
and forward the packets to the concerned virtual ports. These ports are logically connected to
either the virtual NICs of the VMs or the physical NICs of the server. Each port is polled by a
single kernel thread to avoid synchronization between the threads. However, a single kernel
thread may poll multiple ports. A single vSwitch may use a pool of kernel threads, each pinned
to a CPU core respectively. In order to further improve the packet processing eciency and to
lower the packet drop, ENS dynamically load balances the ports among the available kernel
threads. The ENS Thread Load Balancer (TLB) takes into consideration different costs such as
communication, stability, VM affinity and device affinity cost, for re-balancing the virtual ports.
Each vSwitch has a single TLB context assigned to it, which periodically runs to perform the
balancing if required.

This document describes the customer requirements and its solution which is setting up default
load balance metric for the system as well as for the two modes interrupt and polling in which
ENS enabled vSwitch works by doing an enhancement of dvs property which can be used to
apply properties on distributed switches. For this purpose net-dvs command is used on the ESXi
shell and which will also show a lot of information about the distributed virtual switch which is
connected to the hosts.

7
2. Literature Survey

VMware Enhanced Network Stack

VMware developed an Enhanced Network Stack (ENS) to better support the increasing
performance requirements of the NFV workloads. This approach re-architects the network stack
using few of the libraries given by Intel's Data Plane Development Kit (DPDK). The Software
Forwarding Element i.e. the vSwitch can be configured in the ENS mode. The virtual NICs of
the VMs and the physical NICs of the server are connected to these ports respectively. In this
mode, the virtual ports of the vSwitch are polled by a dedicated kernel thread for processing its
packets. A single port is polled by only one kernel thread, however, a single kernel thread may
poll multiple virtual ports. The kernel threads are pinned to the CPU cores in order to prevent
any scheduling delays. As compared to the default network stack, the new network stack uses a
simpler packet buffer representation, has optimized backend NIC emulation, improved packet
forwarding, NUMA aware memory accesses and lockless implementation of most of the shared
data structures. The below figure illustrates an ENS vSwitch with two kernel threads, three
virtual NIC ports and two physical NIC ports. Kernel thread 1 is polling the two ports and Kernel
thread 2 is polling three ports.

The kernel threads maintain a lot of statistics while it is processing the packets of its assigned
ports. It may happen that the ports being handled by a particular kernel thread are having a lot of
trac and the ports of some other kernel thread are having very low trac. In this case the CPU
power of the other kernel thread remains under-utilized, if the ports are statically allocated to the

8
kernel threads. For this, vSwitch has another thread called a Thread Load Balancer (TLB), which
periodically checks the statistics logged by the kernel threads, to load balance the ports among
the available kernel threads. It tries to increase the CPU utilization of the kernel threads, by re-
allocating the ports to different kernel threads depending on certain costs. These costs include -
the cost of communication between two ports, cost of re-allocating a port from one kernel thread
to another and cost of memory accesses. By increasing the CPU utilization the increased packet
rate and lower packet drop is achieved.

Net-dvs Command

The net-dvs command which can be used to apply properties on distributed switches and also can
be used to display information about the Distributed Switch configuration, which can be useful
in situations like troubleshooting.
when logged onto the ESXi shell, the net-dvs command can be run and will output a lot of
information about the dvSwitch that the host is connected to. The command displays data found
in the dvsdata.db file. This file is updated by the host at 5 minute intervals.

9
3. Problem Statement
DVS Property Enhancement of ENS Switch: Change the default load balance metric to CPU load
and implement an enhancement for dvs property through which users can set separate default
load balance metric for interrupt and polling mode of ENS enabled Switch.

4. Development Setup

4.1 Development Build Setup

A cluster which is a Linux-based system with all the tools required for developing the source and
building the necessary deployable objects. The cluster machine is accessed using SSH, via a
Putty SSH client.

4.2 Building and deploying ESXi hypervisor

ESXi source is cloned and a bootable image of the hypervisor is created. And that is deployed as
a VM on a cloud service provided by VMware.

4.3 Building and deploying NSX data path

Clone the NSX Source on the cluster using git. Built the project after required change that will
generate a packaged file containing the NSX data plane related kernel modules and user space
programs. And transfer the package to ESXi for installation and testing.

4.4 Development Flow

Flow involves setting up a dev environment, cloning, building and deploying ESXi and updating
the NSX source on the cluster machine, building the updated NSX package, transferring it to the
Nested ESXi, installing the package and testing the updated features.

5. Proposed Solution

10
 ● Changed the default metric to CPU load.
● Defined two dvs property for setting up default for interrupt and poll mode of ENS
Switch.
● Created a structure that will hold the default metric for both modes.
● Implemented callback write and clear for both modes that will set and unset the default
metric.
The assigned task has yet one last thing to be completed which is to make the implemented dvs
property consumable so that they can sync up with the ENS Switch.

Tools and Technologies used: C and Python programming, Git, Perforce, Bazel and Scons build
tools, Shell Scripting, SSH, Confluence, VMware ESXi, VMware vSphere, VMware NSX.

6. Learnings

It was a whole new learning experience as first I had to go through ramp up sessions and hands
on lab sessions through which I got familiar with the VMware core products and learned
concepts related to virtualization, enhanced network stack, thread load balancer.

The learning involves the

● VMware NSX-T datapath source files, how they are linked from its CLI to the kernel
backend
● The Workflow of building, deploying and testing the builds.
● The Review process of the updates made to the products.
● Documenting your work on confluence.

11
7. References

[1] VMware Inc. VMware Network Virtualization

https://www.vmware.com/topics/glossary/content/network-virtualization.

[2] VMware Inc. VMware VxLANs overview.

https://blogs.vmware.com/vsphere/2013/04/vxlan-series-different-components-part-1.html.

[3] Luo Louis (San Francisco CA US). Enhanced Network Stack, July 2020.

[4] Matthew Portnoy. Virtualization Essentials. SYBEX Inc., USA, 1st edition, 2012.

12