Best Practices for Virtual Networking

Karim Elatov
Technical Support Engineer, GSS

© 2009 VMware Inc. All rights reserved

Agenda

Best Practices for Virtual Networking

Virtual Network Overview

vSwitch Configurations
Tips & Tricks
Troubleshooting Virtual Networks

What’s New in vSphere 5.0

Network Design Considerations

2
Virtual Network Overview - Physical to Virtual

Physical Virtual

Physical
Physical
Switch

Virtual Switch
Physical
Switch

Conventional access, distribution, core design Under the covers, virtual network same as physical
Design with redundancy for enhanced availability Access layer implemented as virtual switches

3
Virtual Switch Options

Virtual Switch Model Details

vNetwork Standard Host based: - Same as vSwitch in VI3
Switch 1 or more per
ESX host
vNetwork Distributed Distributed: - Expanded feature set
Switch 1 or more per - Private VLANs
“Datacenter” - Bi-directional traffic shaping
- Network vMotion
- Simplified management
Cisco Nexus 1000V Distributed: - Cisco Catalyst/Nexus feature set
1 or more per - Cisco NXOS cli
“Datacenter” - Supports LACP

Virtual networking concepts similar with all virtual switches

4
 ESX Virtual Switch: Capabilities

MAC
address
 NIC Teaming of Physical NIC(s) [uplink(s)] associated
assigned to
vnic
with vSwitches
VM0 VM1
 Layer 2 - only forward frames VM <-> VM and VM <-

MAC a MAC b MAC c

> Uplink; No vSwitch <-> vSwitch or Uplink <-> Uplink
vSwitch
vSwitch
 vSwitch will not create loops affecting Spanning

Tree in the physical network

 Can terminate VLAN trunks (VST mode) or pass

Physical trunk through to VM (VGT mode)

Switches

5
Distributed Virtual Switch

Standard vSwitch vNetwork & dvSwitch

vCenter vCenter
Exist across 2 or more clustered hosts

•Provide similar functionality to vSwitches

•Reside on top of hidden vSwitches

vCenter owns the configuration of the dvSwitch

•Consistent host network configurations

6
 Port Groups
 Template for one or more ports with a common
configuration

• VLAN Assignment

• Security

• Traffic Shaping (limit egress traffic from VM)

• Failover & Load Balancing

 Distributed Virtual Port Group (Distributed Virtual Switch)

• Bidirectional traffic shaping (ingress and egress)

• Network VMotion—network port state migrated upon

VMotion

7
 NIC Teaming for Availability and Load Sharing

 NIC Teaming aggregates multiple physical uplinks:

VM0 VM1

• Availability—reduce exposure to single points of

failure (NIC, uplink, physical switch)

• Load Sharing—distribute load over multiple

vSwitch uplinks (according to selected NIC teaming

NIC Team
algorithm)

 Requirements:
• Two or more NICs on same vSwitch

• Teamed NICs must have same VLAN configurations

KB - NIC teaming in ESXi and ESX (1004088)

8
 NIC Teaming Options

Name Algorithm—vmnic Physical Network Considerations

chosen based upon:
Originating vnic port Teamed ports in same L2 domain
Virtual Port ID (BP: team over two physical
switches)
Source MAC MAC seen on vnic Teamed ports in same L2 domain
Address (BP: team over two physical
switches)
IP Hash* Hash(SrcIP, DstIP) Teamed ports configured in static
802.3ad “Etherchannel”
- no LACP (Nexus 1000v for LACP)
- Needs MEC to span 2 switches
Explicit Failover Highest order uplink Teamed ports in same L2 domain
Order from active list (BP: team over two physical
switches)
Best Practices:
•Originating Virtual PortID for VMs is the default, no extra configuration needed
•IP Hash, ensure that physical switch is properly configured for Etherchannel

*KB - ESX/ESXi host requirements for link aggregation (1001938)

*KB - Sample configuration of EtherChannel / Link aggregation with ESX/ESXi and Cisco/HP switches (1004048)

9
Cisco Nexus 1000v Overview

 Cisco Nexus 1000v is a software switch for vNetwork Distributed

Switches (vDS):
• Virtual Supervisor Module (VSM)
• Virtual Ethernet Module (VEM)

Things to remember:

• Virtual Ethernet Module (VEM)VSM uses external network fabric to

communicate with VEMs
• VSM does not take part in forwarding packets
• VEM does not switch traffic to other VEM without an uplink

10
Cisco Nexus 1000v Modules

Server 1 Server 2 Server 3

VM VM VM VM VM VM VM VM VM VM VM VM
#1 #2 #3 #4 #5 #6 #7 #8 #9 #10 #11 #12

VMware
VEMvSwitch Nexus VEM
VMware1000V vDS
vSwitch VMware
VEMvSwitch
VMware ESX VMware ESX VMware ESX

Nexus 1000V

VSM vCenter Server

Virtual Supervisor Module (VSM)
• Virtual or Physical appliance running
Cisco OS (supports HA)
• Performs management, monitoring, &
configuration
• Tight integration with VMware Virtual
Center
Virtual Ethernet Module (VEM)
• Enables advanced networking
capability on the hypervisor
• Provides each VM with dedicated
“switch port”
• Collection of VEMs = 1 DVS
Cisco Nexus 1000V Enables:
• Policy Based VM Connectivity
• Mobility of Network & Security
Properties
• Non-Disruptive Operational Model

11
vSwitch Configurations

Best Practices for Virtual Networking

Virtual Network Overview

vSwitch Configurations
Tips & Tricks
Troubleshooting Virtual Networks

What’s New in vSphere 5.0

Network Design Considerations

12
 Cisco ‘show run’ and ‘show tech-support’

Obtain configuration of a Cisco router or switch

•Run commands in priviliged EXEC mode

•’show run’ The following is a Cisco EtherChannel sample configuration:

interface Port-channel1
•‘show tech-support’ switchport
switchport access vlan 100
switchport mode access
no ip address
!
interface GigabitEthernet1/1
switchport
switchport access vlan 100
switchport mode access
no ip address
channel-group 1 mode on
!

KB - Troubleshooting network issues with the Cisco show tech-support command (1015437)

13
Traffic Types on a Virtual Network
Virtual Machine Traffic
• Traffic sourced and received from virtual machine(s)
• Isolate from each other based on service level
vMotion Traffic
• Traffic sent when moving a virtual machine from one ESX host to
another
• Should be isolated
Management Traffic
• Should be isolated from VM traffic (one or two Service Consoles)
• If VMware HA is enabled, includes heartbeats
IP Storage Traffic—NFS and/or iSCSI via vmkernel interface
• Should be isolated from other traffic types
Fault Tolerance (FT) Logging Traffic
• Low latency, high bandwidth
• Should be isolated from other traffic types
How do we maintain traffic isolation without proliferating NICs? VLANs
14
Traffic Types on a Virtual Network, cont.

 Port groups in dedicated VLANs on a management-only virtual

switch.

Service console/VMK Interface

vMotion storage mgmt

virtual machines
106 107 108
production management
virtual switch virtual switch

production management
vMotion storage

15
VLAN Tagging Options

EST – External Switch Tagging VGT – Virtual Guest Tagging VST – Virtual Switch Tagging

VLAN
assigned in
Port Group
policy

vSwitch vSwitch vSwitch

VLAN Tags PortGroup VLAN Tags

applied in set to VLAN applied in
Guest “4095” vSwitch

Physical Switch Physical Switch Physical Switch

External Physical
switch applies
VLAN tags VST is the best practice and
most common method

switchport access vlan switchport trunk switchport trunk

16
 DVS Support for Private VLAN (PVLAN)

 Enable users to restrict communications

DMZ network
• Between VMs on the same VLAN or network
segment
Web application database email document
server
Allow server the same
devices to share IP subnet while server
server server
being Layer 2 Isolated
 PVLAN Types
• Community
Benefits:
• VMs can communicate with VMs on isolated isolated
•Employ
community
Community Larger PVLAN
subnets
and Promiscuous(advantageous to hosting
PVLANenvironments)
PVLAN
• Isolated
•Reduce Management Overhead
• VMs can only communicate with VMs on
the Promiscuous
• Promiscuous
• VMs can communicate with all VMs
router in promiscuous PVLAN
KB - Private VLAN (PVLAN) on vNetwork Distributed Switch - Concept Overview (1010691)

17
PVLAN Cost Benefit

W2003EE-32-A W2003EE-32-B W2003EE-32-A W2003EE-32-B W2003EE-32-A W2003EE-32-B W2003EE-32-A W2003EE-32-B W2003EE-32-A W2003EE-32-B W2003EE-32-A W2003EE-32-B

PG PG PG PG PG PG PG PG PG PG PG PG

Distributed Virtual Switch

TOTAL COST: 12 VLANs (one per VM)

W2003EE-32-A W2003EE-32-B W2003EE-32-A W2003EE-32-B W2003EE-32-A W2003EE-32-B W2003EE-32-A W2003EE-32-B W2003EE-32-A W2003EE-32-B W2003EE-32-A W2003EE-32-B

PG (with Isolated PVLAN)

Distributed Virtual Switch

TOTAL COST: 1 PVLAN (over 90% savings…)

18
 Link Aggregation

EtherChannel
•Port trunking between two to eight
•Active Fast Ethernet, Gigabit Ethernet, or 10 Gigabit Ethernet ports

EtherChannel vs. 802.3ad

•EtherChannel is Cisco proprietary and 802.3ad is an open standard
Note: ESX implements 802.3ad Static Mode Link Aggregation

LACP (one of the implementations included in IEEE 802.3ad)

•Link Aggregation Control Protocol (LACP)
•Control the bundling of several physical ports into a single logical channel
•Only supported on Nexus 1000v
KB ESX/ESXi host requirements for link aggregation (1001938)

19
 Sample Link Aggregation Configuration

Supported switch Aggregation algorithm: IP-SRC-DST

Supported Virtual Switch NIC Teaming mode: IP HASH
KB - Sample configuration of EtherChannel / Link aggregation with ESX/ESXi andCisco/HP switches (1004048)

20
 Failover Configurations

Link Status relies solely on the network adapter link state

•Cannot detect configuration errors
•Spanning Tree Blocking
•Incorrect VLAN
•Physical switch cable pulls

Beacon Probing sends out and listens for beacon probes

•Broadcast frames (ethertype 0x05ff)

Beacon Probing Best Practice

•Use at least 3 NICs for triangulation
•If only 2 NICs in team, can’t determine link failed
•Leads to shotgun mode results
Figure — Using beacons to detect upstream
KB - What is beacon probing? (1005577) network connection failures.

21
 Spanning Tree Protocol (STP) Considerations

 Spanning Tree Protocol creates loop-free L2 tree

VM0 VM1
topologies in the physical network
• Physical links put in “blocking” state to construct
MAC a MAC b loop-free tree

vSwitch
 ESX vSwitch does not participate in Spanning Tree
and will not create loops with uplinks
• ESX Uplinks will not block, always active (full use
vSwitch drops
BPDUs of all links)
Physical
Switches
Recommendations for Physical Network Config:
1. Leave Spanning Tree enabled on physical network
and ESX facing ports (i.e. leave it as is!)
Switches sending 2. Use “portfast” or “portfast trunk” on ESX facing
BPDUs every 2s to
construct and
ports (puts ports in forwarding state immediately)
maintain Spanning 3. Use “bpduguard” to enforce STP boundary
Blocked link Tree Topology

KB - STP may cause temporary loss of network connectivity when a failover or failback event occurs (1003804)

22
Tips & Tricks

Best Practices for Virtual Networking

Virtual Network Overview

vSwitch Configurations
Tips & Tricks
Troubleshooting Virtual Networks

What’s New in vSphere 5.0

Network Design Considerations

23
Tips & Tricks

 Load-Based Teaming (LBT)

• Dynamically balance network load over available uplinks

• Triggered by ingress or egress congestion at 75% mean utilization over a 30

second period

• Configure on DVS via “Route based on physical NIC load”

*LBT is not available on the Standard vSwitch (DVS feature for ingress/egress traffic shaping)

 Network I/O Control (NetIOC)

• DVS software scheduler to isolate and prioritize specific traffic types
contending for bandwidth on the uplinks connecting ESX/ESXi 4.1 hosts with
the physical network.

24
 Tips & Tricks

Tip #1 – After physical to virtual migration, the VM MAC address can be

changed for Licensed Applications relying on physical MAC address. (KB
1008473)

Tip #2 – NLB Multicast needs physical switch Manual ARP resolution of NLB
cluster. (KB 1006525)

Tip #3 – Cisco Discovery Protocol (CDP) gives switchport configuration

information useful for troubleshooting (KB 1007069)

Tip #4 - Beacon Probing and IP Hash DO NOT MIX (duplicate packets and port
flapping) (KB 1017612 & KB 1012819)

Tip #5 – Link aggregation is never supported on disparate trunked switches – Use

VSS with MEC. (KB 1001938 & KB 1027731)

25
Tips & Tricks
Using 10GigE Ingress (into switch)
traffic shaping policy
control on Port Group
Variable/high High
b/w 2Gbps+ 1-2G b/w Low b/w

iSCSI NFS VMotion FT SC

 2x 10GigE common/expected
SC#2 • 10GigE CNAs or NICs
 Possible Deployment Method
vSwitch • Active/Standby on all Portgroups
Gbps
FCoE
10GE 10GE
FCoE
• VMs “sticky” to one vmnic
10
• SC/vmk ports sticky to other
• Use Ingress Traffic Shaping
FCoE
to control traffic type per
Port Group
FCoE Priority Group
• If FCoE, use
Best Practice: Ensure Drivers and Firmware are compatible forPriority
success
bandwidth reservation
Group
(in CNA config utility)
vSphere 4.1 supports up to (4) 10GigE NICs; 5.0 supports (8) reservation
bandwidth 10GigE NICs (on CNA
utility)
26
Troubleshooting Virtual Networks

Best Practices for Virtual Networking

Virtual Network Overview

vSwitch Configurations
Tips & Tricks
Troubleshooting Virtual Networks

What’s New in vSphere 5.0

Network Design Considerations

27
Network Troubleshooting Tips

 Troubleshoot one component at a time

• Physical NICs
• Virtual Switch
• Virtual NICs
• Physical Network

 Tools for Troubleshooting

• vSphere Client
• Command Line Utilities
• ESXTOP
• Third party tools
• Ping and Traceroute
• Traffic sniffers & Protocol
Analyzers
• Wireshark
• Logs
28
 Capturing Traffic

Best Practice: create a new management interface for this purpose

vSwitch must be in Promiscuous Mode (KBs 1004099 & 1002934)
ESXi uses tcpdump-uw (KB 1031186)
29
What’s New in vSphere 5.0

Best Practices for Virtual Networking

Virtual Network Overview

vSwitch Configurations
Tips & Tricks
Troubleshooting Virtual Networks

What’s New in vSphere 5.0

Network Design Considerations

30
 What’s New in vSphere 5?

Monitor and troubleshoot virtual infrastructure traffic

• NetFlow V5

• Port mirror (SPAN)

• LLDP (standard based link layer discovery protocol) support simplifies the
network configuration and management in non-Cisco switch environment.
Enhancements to the network I/O control (NIOC)

• Ability to create User-defined resource pool

• Support for vSphere replication traffic type; a new system traffic type that
carries replication traffic from one host to another.

• Support for IEEE 802.1p tagging

What’s New in VMware vSphere 5.0 Networking Technical Whitepaper

31
Network Design Considerations

Best Practices for Virtual Networking

Virtual Network Overview

vSwitch Configurations
Tips & Tricks
Troubleshooting Virtual Networks

What’s New in vSphere 5.0

Network Design Considerations

32
Network Design Considerations
How do you design the virtual network for
performance and availability but maintain isolation
between the various traffic types
(e.g. VM traffic, VMotion, and Management)?
• Starting point depends on:
• Number of available physical ports on server
• Required traffic types
• 2 NIC minimum for availability, 4+ NICs
per server preferred

• 802.1Q VLAN trunking highly recommended for logical scaling

(particularly with low NIC port servers)

• Examples are meant as guidance and do not represent strict

requirements in terms of design

• Understand your requirements and resultant traffic types and

design accordingly

33
Example 1: Blade Server with 2 NIC Ports

 Candidate Design:
SC vmkernel • Team both NIC ports
• Create one virtual switch
• Create three port groups:
Portgroup3 Portgroup1 Portgroup2
VLAN 30 VLAN 10 VLAN 20

vSwitch • Use Active/Standby policy

for each portgroup
vmnic0 vmnic1
• Portgroup1: Service Console (SC)
VLAN Trunks
• Portgroup2: VMotion
(VLANs 10, 20, 30)
• Portgroup3: VM traffic
• Use VLAN trunking
• Trunk VLANs 10, 20,
Active
30 on each uplink
Standby

Note: Team over dvUplinks with vDS

34
Example 2: Server with 4 NIC Ports
 Candidate Design:
• Create two virtual switches
• Team two NICs to each vSwitch
SC vmkernel
• vSwitch0 (use active/standby
Portgroup4
VLAN 40
Portgroup3
VLAN 30
Portgroup1
VLAN 10
Portgroup2
VLAN 20
for each portgroup):
vSwitch1 vSwitch0 • Portgroup1: Service Console (SC)

vmnic0 vmnic2 vmnic1 vmnic3

• Portgroup2: VMotion

• vSwitch1 (use Originating Virtual

VLANs VLANs PortID)
30, 40 10, 20
• Portgroup3: VM traffic #1

• Portgroup4: VM traffic #2
Active
Standby • Use VLAN trunking
Note: Team over dvUplinks with vDS • vmnic1 and vmnic3: Trunk VLANs 10, 20

• vmnic0 and vmnic2: Trunk VLANs 30, 40

35
Example 3: Server with 4 NIC Ports (Slight Variation)

 Candidate Design:
• Create one virtual switch
• Create two NIC teams
SC vmkernel
• vSwitch0 (use active/standby
Portgroup4
VLAN 40
Portgroup3
VLAN 30
Portgroup1
VLAN 10
Portgroup2
VLAN 20
for portgroups 1 & 2):
• Portgroup1: Service Console (SC)
vSwitch0

• Portgroup2: Vmotion
vmnic0 vmnic2 vmnic1 vmnic3

• Use Originating Virtual PortID

for Portgroups 3 & 4
VLANs VLANs
30, 40 10, 20
• Portgroup3: VM traffic #1

• Portgroup4: VM traffic #2
Active
Standby
• Use VLAN trunking
• vmnic1 and vmnic3: Trunk VLANs 10, 20
Note: Team over dvUplinks with vDS
• vmnic0 and vmnic2: Trunk VLANs 30, 40

36
Questions

37