NET5521

vSphere Distributed Switch –

Design and Best Practices

Vyenkatesh (Venky) Deshpande, VMware

Marcos Hernandez, VMware

#NET5521
Session Objective

 New capabilities in VDS

 VDS can meet your design requirements

 Provide Common best practices while designing with VDS

2
Recommended Sessions & Labs

 VSVC4966 – vSphere Distributed Switch – Technical Deep Dive

 VSVC5103 - vSphere Networking and vCloud Networking Suite

Best Practices and Troubleshooting

 You can check out VSS to VDS Migration workflow and new VDS
features in the lab HOL-SDC-1302

 NET5266 - Bringing Network Virtualization to VMware environments

with NSX

 NET5654 - Troubleshooting VXLAN and Network Services in a

Virtualized Environment

3
Agenda
 Overview of VDS and New Features in 5.5

 Common Customer Deployments

 Design and Best Practices

 NSX and VDS

4
 VDS Overview and 5.5 Features

5
vSphere Distributed Switch (VDS)

vSphere Distributed Switch

 Manage a Datacenter wide switch vs. Individual switches per host

 Advanced feature support

 Higher Scale

 Foundation for your Network Virtualization Journey

6
 vSphere Distributed Switch (VDS) Architecture

VMware vCenter Server

Management Plane

vSphere Distributed Switch

Data Plane Data Plane

vSphere Distributed Switch

Legend :
vSphere vSphere
dvPG-A
dvPG-B
dvuplink1 dvuplink2
dvUplink PG

dvUplink
Host 1 Host 2
vmnic0 vmnic1 vmnic0 vmnic1

Management Plane : Allows to configure various parameters of the distributed switch

Data Plane : Handles the packet switching function

7
VDS Enhancements in vSphere 5.5

Performance and Scale

 Enhanced LACP
 Enhanced SR-IOV
 40 Gig NIC support
vSphere Distributed Switch
Packet Classification

 Traffic Filtering (ACLs)

 DSCP Marking (QoS)

Visibility & Troubleshooting

 Host Level Packet Capture

Tool (tcpdump). Available
for Standard Switch as well

8
 LACP Enhancements
Host
 Link Aggregation Control
Protocol
 Standards based – 802.3ad
 Automatic negotiation of link aggregation
vSphere Distributed Switch
parameters

 Advantages
vSphere
 Aggregates link BW and provides
redundancy
 Detects link failures and cabling mistakes
LACP and automatically reconfigures
Communication

 Enhancements
 Support for 64 LAGs per VDS and per
Host
 Support for 22 different hashing
algorithms
Physical switches

9
 Common Customer Deployments

10
VDS in the Enterprise
vCenter
VMware Server
vCentServer

VDS VDS VDS VDS VDS

Data Center

Cluster 1 Cluster 2 Cluster 3 Cluster 4

ROBO 1 ROBO 2

 Multiple VDS per VC (128)  Central Management for DC and

ROBO environments
 VDS can span multiple Clusters
 Role Based management control
 Hundreds of Hosts per VDS

11
 Design Best Practices

12
Infrastructure Design Goals

 Reliable

 Secure

 Performance

 Scalable

 Operational

13
Infrastructure Types Influence Your Design Decisions

 Available Infrastructure
• Type of Servers
• Type of Physical Switches

 Servers
• Rack mount or Blade
• Number of Ports and Speed. For example, Multiple 1 Gig or 2 – 10 Gig

 Physical Switches
• Managed and un-managed
• Protocol and features support

 Example Deployment – 2 – 10 Gig Server configuration

14
 Reliable - Connectivity

15
 Physical Connection Options
One Physical Switch Two Physical Switches One Physical Switch Two Physical Switches
with Ether Channel in MLAG configuration

VDS VDS VDS VDS

vSphere vSphere vSphere vSphere

MLAG/vPC

Port Group – Teaming Port Group – Teaming Port Group – Teaming Port Group – Teaming
Port ID, MAC Hash, Port ID, MAC Hash, IP Hash LACP
Explicit Failover, LBT Explicit Failover, LBT

16
Connectivity Best Practices

 Avoid Single point of Failure

• Connect two or more physical NICs to a VDS
• Preferably connect those physical NICs to separate physical switches

 Configure Port groups with appropriate teaming setting based on

the physical switch connectivity and configuration. For example
• Use IP hash when Ether channel is configured on Physical Switch

 Configure Port Fast and BPDU guard on Access Switch Ports

• No STP running on virtual switches
• No loop created by virtual switch

 Trunk all Port group VLANs on Access Switch ports

17
 Spanning Tree Protocol Boundary
Virtual Network

No Spanning Tree
Support vSphere Distributed Switch

No BPDU vSphere vSphere

generated

Switch Port Switch Port

Configuration: Configuration:
Port Fast Port Fast
BPDU Guard BPDU Guard
VLAN 10,20 VLAN 10.20

Physical Network

Spanning Tree Protocol Boundary

18
Teaming Best Practices

 Link Aggregation mechanisms do not double the BW

• Hashing algorithm performs better in some scenarios. For example
• Web servers accessed by different users have enough variation in IP Src and Dest
fields and can utilize links effectively
• However, few workloads accessing a NAS array doesn’t have any variation in
the packet header fields. Traffic might end up on only one physical NIC

 Why Load Based Teaming is better ?

 Takes into account link utilization
 Checks Utilization of Links every 30 seconds
 No special configuration required on the physical switches

19
Load Based Teaming

Network Traffic Bandwidth

vMotion traffic 7 Gig

VM1 traffic 5 Gig

VM2 traffic 2 Gig

VM1 VM2 VM1 VM2

vMotion vMotion

1 2 3 4 1 2 3 4

10 11 Rebalance 10 11

VDS VDS

10 Gig 2 Gig 7 Gig 7 Gig

20
 Security/Isolation

21
Traffic Types Running on a vSphere Host

NFS FT Mgmt vMotion

Traffic Traffic Traffic Traffic
vmk1 vmk2 vmk3 vmk4

VDS

PG-A PG-B PG-C PG-D PG-E

vSphere

Host

10 Gig 10 Gig

22
Security Best Practices

 Provide Traffic Isolation using VLANs

• Each Port group can be associated with different VLAN
 Keep default Security settings on the Port group
• Promiscuous Mode – Reject
• MAC address Changes – Reject
• Forged Transmit – Reject
 While utilizing PVLAN feature make sure Physical Switches are
also configured with Primary, Secondary VLAN configuration
 Enable BPDU filter property at Host level to prevent DoS attack
situation due to compromised virtual machines
 Make use Access Control List Feature (5.5)

23
 Performance

24
Why Should You Care About Performance?

 As more workloads are getting virtualized, 10 Gig pipes

are getting filled

 Some workloads have specific BW and latency requirements

• Business Critical applications
• VOIP applications
• VDI application

 Noisy Neighbors problem has to be addressed

• vMotion is very BW intensive and can impact other traffic types
• General Purpose VM traffic can impact other critical applications such
as VOIP application

25
 Network I/O Control
VM
Traffic Infrastructure Traffics

vMotion Mgmt FT NFS

vSphere Distributed
Port groups

Administrator Teaming Policy

vSphere Distributed Switch

Shaper Shaper
Traffic Shares Limit 802.1p
(Mbps)
Scheduler Scheduler
VM Traffic 30 - 4
Limits
Port 1
vMotion 20 -
4000 3 Host

Mgmt 5 - 7 10 Gig 10 Gig

Shares % Link BW
FT 10 - 6 BW 10 Gig
Port 2
30 30/50 3/5*10 = 6
NFS 20 - 5 20 20/50 2/5*10 = 4
Total 50

26
 Business Critical Applications and User Defined Traffic Types
App 1 App 2 VM
Traffic Traffic Traffic

vMotion Mgmt FT NFS

vSphere Distributed Port groups

Administrator Teaming Policy
vSphere Distributed Switch

Traffic Shares Limit 802.1p Shaper Shaper

(Mbps)
App1 10 - 7 Scheduler Scheduler

App2 10 - 6
Port 1 Host
VM Traffic 10 - 4

vMotion 20 - 3 10 Gig 10 Gig

Mgmt 5 - 7

FT 10 - 6 Port 2

NFS 20 - 5

27
End to End QoS

 How to make sure that the Application traffic flowing through

Physical Network Infrastructure is also Prioritized ?

 Two types of Tagging or Marking supported

• COS – Layer 2 Tag
• DSCP Marking – Layer 3 Tag

802.1Q Header IP Header

0x8100 COS D VLAN Version H Length TOS/DS P Length …..

16 bits 3 bits 1 bit 12 bits

DSCP ECN
6 bits 2 bits

28
 Tagging at Different Level

Guest Tagging Virtual Switch Tagging Physical Switch Tagging

DSCP
COS
DSCP
vSphere Switch vSphere Switch COS vSphere Switch

vSphere vSphere vSphere

DSCP
COS

Physical Physical Physical

Network Network Network

VDS can pass VM QoS
markings downstream
NIOC can’t assign
separate queue based
on the tag
Admins lose control
29
VDS implements 802.1p and/or QoS marking or remarking
DSCP marking done in the physical switch
Preferred option and/or router
Burdensome QoS management
Single Edge QoS enforcement
on each edge device (e.g. ToR)
point
Congestion Scenario in the Physical Network

Higher Tagged Traffic

Lower Tagged Traffic
vSphere Switch
vSphere Switch Un Tagged Traffic
vSphere
vSphere

Congested Switch

Physical Network

30
Per Port Traffic Shaping
VM
Traffic  Ingress and Egress
Parameters
vMotion Mgmt
 Average Bandwidth
Ingress Egress  Kbps
 Peak Bandwidth
 Kbps
 Burst Size
 Kbytes

10 Gig 10 Gig

Burst Size

Peak BW

Token Average BW
Bucket BW

Time

31
Other Performance Related Decisions

 Need more BW for Storage

• If iSCSI, utilize Multi-Pathing.
• MTU configuration – Jumbo frame
• LBT can’t work for iSCSI traffic because of port binding requirements

 Need more BW for vMotion

• Use Multi-NIC vMotion.
• LBT doesn’t split the vMotion traffic to multiple Physical NICs.

 Latency Sensitive application – Care about Micro seconds

• Utilize SR-IOV
• Doesn’t support vMotion, HA and DRS features

32
 Scalable

33
 Scale
 Scaling Compute Infrastructure
 Adding Hosts to Clusters
VDS
Data Center

 Adding new Clusters

 Impact on VDS Design
 VDS can span across 500 hosts
Cluster 1 Cluster 2 Cluster 3 Cluster 4

 Scaling number of users or

applications
 More Virtual Machines connected to
VDS
isolated networks (VLANs)
 Impact on VDS Design
Data Center

 Separate port groups for each application

– 10,000 port groups support
Cluster 1 Cluster 2 Cluster 3 Cluster 4  Number of virtual ports - 60,000
 Dynamic Port management (Static Ports)
34
 Operational

35
How to Operate Your Virtual Network?

 Major concerns
• Lost visibility into traffic from VM to VM on the same Host
• How do I troubleshoot configuration issues?
• How do I troubleshoot connectivity issues?

 Make use of VDS features

• Netflow and Port Mirroring
• Network Health Check detects mis-configuration across virtual
and physical switches
• Host level Packet Capture allows you to monitor traffic at vnic,
vmknic and vmnic level

36
 NSX and VDS

37
VMware NSX Functional System Overview

Tenant UI

Consumption API Operations

CMP

UI

API
Management Plane
NSX Manager API, config, etc.
vCenter Server
HA, scale-out
Logs/Stats

Control Plane NSX Controller Run-time state

vSwitch vSwitch vSwitch vSwitch

Data Plane vSphere vSphere vSphere vSphere

Hosts

38
VXLAN Protocol Overview
 Ethernet in IP overlay network
 Entire L2 frame encapsulated in
UDP
 50+ bytes of overhead
 Decouples Physical network
from the Logical
 24 bits VXLAN ID identifies 16 M
Logical networks
 VMs do NOT see VXLAN ID
 Physical Network devices don’t see
VMs MAC and IP address
39
 VTEP (VXLAN Tunnel End
Point)
 VMkernel interface which serves as
the endpoint for encapsulation/de-
encapsulation of VXLAN traffic
 VXLAN can cross Layer 3
network boundaries
 Technology submitted to IETF
for standardization
• With Cisco, Citrix, Red Hat,
Broadcom, Arista and Others
VXLAN Configuration on VDS

VXLAN Transport Subnet A 10.20.10.0/24 VXLAN Transport Subnet B 10.20.11.0/24

VM1 VM2 VM3 VM4

VXLAN 5001

vSphere Distributed Switch

VTEP1 10.20.10.10 VTEP2 10.20.10.11 VTEP3 10.20.11.10 VTEP4 10.20.11.11

vSphere Host vSphere Host vSphere Host vSphere Host

VXLAN Transport Network

40
 For More Details on VXLAN attend
NET5654 - Troubleshooting VXLAN and Network
Services in a Virtualized Environment

41
Key Takeaways

 VDS is flexible and scalable to meet your design requirements.

 VDS simplifies the deployment and operational aspects

of virtual network

 Make use of NIOC and LBT feature to improve utilization

of your I/O resources

 VDS is a key component of NSX Platform

42
 Q&A
Paper: http://www.vmware.com/resources/techresources/10250
http://blogs.vmware.com/vsphere/networking
@VMWNetworking

43
Other VMware Activities Related to This Session

 HOL:
HOL-SDC-1302
vSphere Distributed Switch from A to Z
 Group Discussions:
NET1000-GD
vSphere Distributed Switch with Vyenkatesh Deshpande

44
THANK YOU
 NET5521
vSphere Distributed Switch –
Design and Best Practices

Vyenkatesh Deshpande, VMware

Marcos Hernandez, VMware

#NET5521
 Backup: Example Design

48
VDS in Rack Server Deployment: Two 10 Gig Ports

Cluster 1 Cluster 2

vSphere Distributed Switch

ESXi ESXi ESXi ESXi

.................

Legend :
Access
Layer PG-A
L2 Switch PG-B

Aggregation
Layer

Router

49
Option1: Static Design – Port Group to NIC Mapping

Port Teaming Active Standby Unused

Traffic Type
Group Option Uplink Uplink Uplink
dvuplink1/
Virtual Machine PG-A LBT None None
dvuplink2
Explicit
NFS PG-B dvuplink1 dvuplink2 None
Failover
Explicit
FT PG-C Failover dvuplink2 dvuplink1 None

Explicit
Management PG-D Failover dvuplink2 dvuplink1 None

Explicit
vMotion PG-E Failover dvuplink2 dvuplink1 None

50
Option2: Dynamic Design –
Use NIOC and Configure Shares and Limits
 Need Bandwidth information for different traffic types
• NetFlow
 Bandwidth Assumption
• Management – Less than 1 Gig
• vMotion – 2 Gig
• NFS – 2 Gig
• FT – 1 Gig
• Virtual Machine – 2 Gig
 Shares calculation
• Equal shares to vMotion, NFS and Virtual Machine
• Lower shares to Management and FT

51
Option2: Dynamic Design –
Use NIOC and Configure Shares and Limits

Traffic Port Teaming Active Standby NIOC NIOC

Type Group Option Uplink Uplink Shares Limits
Virtual
PG-A LBT dvuplink1,2 None 20 -
Machine

NFS PG-B LBT dvuplink1,2 None 20 -

LBT
FT PG-C dvuplink1,2 None 10 -

LBT
Mgmt. PG-D dvuplink1,2 None 5 -

vMotion PG-E LBT dvuplink1,2 None 20 -

52
Dynamic Design Option with NIOC and LBT – Pros and Cons

 Pros
• Better utilized I/O resources through traffic management
• Logical separation of traffic through VLAN
• Traffic SLA maintained through NIOC shares
• Resiliency through Active-Active Paths

 Cons
• Dynamic traffic movement across physical infrastructure need all paths
to be available and handle any traffic characteristics.
• VLAN expertise

53