Best Practices for Virtual Networking

Karim Elatov
Technical Support Engineer, GSS

2009 VMware Inc. All rights reserved

Agenda

Best Practices for Virtual Networking

Virtual Network Overview

vSwitch Configurations
Tips & Tricks
Troubleshooting Virtual Networks

Whats New in vSphere 5.0

Network Design Considerations

2
Virtual Network Overview - Physical to Virtual

Physical Virtual

Physical
Physical
Switch

Virtual Switch
Physical
Switch

Conventional access, distribution, core design Under the covers, virtual network same as physical
Design with redundancy for enhanced availability Access layer implemented as virtual switches

3
Virtual Switch Options

Virtual Switch Model Details

vNetwork Standard Host based: - Same as vSwitch in VI3
Switch 1 or more per
ESX host
vNetwork Distributed Distributed: - Expanded feature set
Switch 1 or more per - Private VLANs
Datacenter - Bi-directional traffic shaping
- Network vMotion
- Simplified management
Cisco Nexus 1000V Distributed: - Cisco Catalyst/Nexus feature set
1 or more per - Cisco NXOS cli
Datacenter - Supports LACP

Virtual networking concepts similar with all virtual switches

4
 ESX Virtual Switch: Capabilities

MAC
address
NIC Teaming of Physical NIC(s) [uplink(s)] associated
assigned to
vnic
with vSwitches
VM0 VM1
Layer 2 - only forward frames VM <-> VM and VM <-

MAC a MAC b MAC c

> Uplink; No vSwitch <-> vSwitch or Uplink <-> Uplink
vSwitch
vSwitch
vSwitch will not create loops affecting Spanning

Tree in the physical network

Can terminate VLAN trunks (VST mode) or pass

Physical trunk through to VM (VGT mode)

Switches

5
Distributed Virtual Switch

Standard vSwitch vNetwork & dvSwitch

vCenter vCenter
Exist across 2 or more clustered hosts

Provide similar functionality to vSwitches

Reside on top of hidden vSwitches

vCenter owns the configuration of the dvSwitch

Consistent host network configurations

6
 Port Groups
Template for one or more ports with a common
configuration

VLAN Assignment

Security

Traffic Shaping (limit egress traffic from VM)

Failover & Load Balancing

Distributed Virtual Port Group (Distributed Virtual Switch)

Bidirectional traffic shaping (ingress and egress)

Network VMotionnetwork port state migrated upon

VMotion

7
 NIC Teaming for Availability and Load Sharing

NIC Teaming aggregates multiple physical uplinks:

VM0 VM1

Availabilityreduce exposure to single points of

failure (NIC, uplink, physical switch)

Load Sharingdistribute load over multiple

vSwitch uplinks (according to selected NIC teaming

NIC Team
algorithm)

Requirements:
Two or more NICs on same vSwitch

Teamed NICs must have same VLAN configurations

KB - NIC teaming in ESXi and ESX (1004088)

8
 NIC Teaming Options

Name Algorithmvmnic Physical Network Considerations

chosen based upon:
Originating vnic port Teamed ports in same L2 domain
Virtual Port ID (BP: team over two physical
switches)
Source MAC MAC seen on vnic Teamed ports in same L2 domain
Address (BP: team over two physical
switches)
IP Hash* Hash(SrcIP, DstIP) Teamed ports configured in static
802.3ad Etherchannel
- no LACP (Nexus 1000v for LACP)
- Needs MEC to span 2 switches
Explicit Failover Highest order uplink Teamed ports in same L2 domain
Order from active list (BP: team over two physical
switches)
Best Practices:
Originating Virtual PortID for VMs is the default, no extra configuration needed
IP Hash, ensure that physical switch is properly configured for Etherchannel

*KB - ESX/ESXi host requirements for link aggregation (1001938)

*KB - Sample configuration of EtherChannel / Link aggregation with ESX/ESXi and Cisco/HP switches (1004048)

9
Cisco Nexus 1000v Overview

Cisco Nexus 1000v is a software switch for vNetwork Distributed

Switches (vDS):
Virtual Supervisor Module (VSM)
Virtual Ethernet Module (VEM)

Things to remember:

Virtual Ethernet Module (VEM)VSM uses external network fabric to

communicate with VEMs
VSM does not take part in forwarding packets
VEM does not switch traffic to other VEM without an uplink

10
Cisco Nexus 1000v Modules

Server 1 Server 2 Server 3

VM VM VM VM VM VM VM VM VM VM VM VM
#1 #2 #3 #4 #5 #6 #7 #8 #9 #10 #11 #12

VMware
VEMvSwitch Nexus VEM
VMware1000V vDS
vSwitch VMware
VEMvSwitch
VMware ESX VMware ESX VMware ESX

Nexus 1000V

VSM vCenter Server

Virtual Supervisor Module (VSM)
Virtual or Physical appliance running
Cisco OS (supports HA)
Performs management, monitoring, &
configuration
Tight integration with VMware Virtual
Center
Virtual Ethernet Module (VEM)
Enables advanced networking
capability on the hypervisor
Provides each VM with dedicated
switch port
Collection of VEMs = 1 DVS
Cisco Nexus 1000V Enables:
Policy Based VM Connectivity
Mobility of Network & Security
Properties
Non-Disruptive Operational Model

11
vSwitch Configurations

Best Practices for Virtual Networking

Virtual Network Overview

vSwitch Configurations
Tips & Tricks
Troubleshooting Virtual Networks

Whats New in vSphere 5.0

Network Design Considerations

12
 Cisco show run and show tech-support

Obtain configuration of a Cisco router or switch

Run commands in priviliged EXEC mode

show run The following is a Cisco EtherChannel sample configuration:

interface Port-channel1
show tech-support switchport
switchport access vlan 100
switchport mode access
no ip address
!
interface GigabitEthernet1/1
switchport
switchport access vlan 100
switchport mode access
no ip address
channel-group 1 mode on
!

KB - Troubleshooting network issues with the Cisco show tech-support command (1015437)

13
Traffic Types on a Virtual Network
Virtual Machine Traffic
Traffic sourced and received from virtual machine(s)
Isolate from each other based on service level
vMotion Traffic
Traffic sent when moving a virtual machine from one ESX host to
another
Should be isolated
Management Traffic
Should be isolated from VM traffic (one or two Service Consoles)
If VMware HA is enabled, includes heartbeats
IP Storage TrafficNFS and/or iSCSI via vmkernel interface
Should be isolated from other traffic types
Fault Tolerance (FT) Logging Traffic
Low latency, high bandwidth
Should be isolated from other traffic types
How do we maintain traffic isolation without proliferating NICs? VLANs
14
Traffic Types on a Virtual Network, cont.

Port groups in dedicated VLANs on a management-only virtual

switch.

Service console/VMK Interface

vMotion storage mgmt

virtual machines
106 107 108
production management
virtual switch virtual switch

production management
vMotion storage

15
VLAN Tagging Options

EST External Switch Tagging VGT Virtual Guest Tagging VST Virtual Switch Tagging

VLAN
assigned in
Port Group
policy

vSwitch vSwitch vSwitch

VLAN Tags PortGroup VLAN Tags

applied in set to VLAN applied in
Guest 4095 vSwitch

Physical Switch Physical Switch Physical Switch

External Physical
switch applies
VLAN tags VST is the best practice and
most common method

switchport access vlan switchport trunk switchport trunk

16
 DVS Support for Private VLAN (PVLAN)

Enable users to restrict communications

DMZ network
Between VMs on the same VLAN or network
segment
Web application database email document
server
Allow server the same
devices to share IP subnet while server
server server
being Layer 2 Isolated
PVLAN Types
Community
Benefits:
VMs can communicate with VMs on isolated isolated
Employ
community
Community Larger PVLAN
subnets
and Promiscuous(advantageous to hosting
PVLANenvironments)
PVLAN
Isolated
Reduce Management Overhead
VMs can only communicate with VMs on
the Promiscuous
Promiscuous
VMs can communicate with all VMs
router in promiscuous PVLAN
KB - Private VLAN (PVLAN) on vNetwork Distributed Switch - Concept Overview (1010691)

17
PVLAN Cost Benefit

W2003EE-32-A W2003EE-32-B W2003EE-32-A W2003EE-32-B W2003EE-32-A W2003EE-32-B W2003EE-32-A W2003EE-32-B W2003EE-32-A W2003EE-32-B W2003EE-32-A W2003EE-32-B

PG PG PG PG PG PG PG PG PG PG PG PG

Distributed Virtual Switch

TOTAL COST: 12 VLANs (one per VM)

W2003EE-32-A W2003EE-32-B W2003EE-32-A W2003EE-32-B W2003EE-32-A W2003EE-32-B W2003EE-32-A W2003EE-32-B W2003EE-32-A W2003EE-32-B W2003EE-32-A W2003EE-32-B

PG (with Isolated PVLAN)

Distributed Virtual Switch

TOTAL COST: 1 PVLAN (over 90% savings)

18
 Link Aggregation

EtherChannel
Port trunking between two to eight
Active Fast Ethernet, Gigabit Ethernet, or 10 Gigabit Ethernet ports

EtherChannel vs. 802.3ad

EtherChannel is Cisco proprietary and 802.3ad is an open standard
Note: ESX implements 802.3ad Static Mode Link Aggregation

LACP (one of the implementations included in IEEE 802.3ad)

Link Aggregation Control Protocol (LACP)
Control the bundling of several physical ports into a single logical channel
Only supported on Nexus 1000v
KB ESX/ESXi host requirements for link aggregation (1001938)

19
 Sample Link Aggregation Configuration

Supported switch Aggregation algorithm: IP-SRC-DST

Supported Virtual Switch NIC Teaming mode: IP HASH
KB - Sample configuration of EtherChannel / Link aggregation with ESX/ESXi andCisco/HP switches (1004048)

20
 Failover Configurations

Link Status relies solely on the network adapter link state

Cannot detect configuration errors
Spanning Tree Blocking
Incorrect VLAN
Physical switch cable pulls

Beacon Probing sends out and listens for beacon probes

Broadcast frames (ethertype 0x05ff)

Beacon Probing Best Practice

Use at least 3 NICs for triangulation
If only 2 NICs in team, cant determine link failed
Leads to shotgun mode results
Figure Using beacons to detect upstream
KB - What is beacon probing? (1005577) network connection failures.

21
 Spanning Tree Protocol (STP) Considerations

Spanning Tree Protocol creates loop-free L2 tree

VM0 VM1
topologies in the physical network
Physical links put in blocking state to construct
MAC a MAC b loop-free tree

vSwitch
ESX vSwitch does not participate in Spanning Tree
and will not create loops with uplinks
ESX Uplinks will not block, always active (full use
vSwitch drops
BPDUs of all links)
Physical
Switches
Recommendations for Physical Network Config:
1. Leave Spanning Tree enabled on physical network
and ESX facing ports (i.e. leave it as is!)
Switches sending 2. Use portfast or portfast trunk on ESX facing
BPDUs every 2s to
construct and
ports (puts ports in forwarding state immediately)
maintain Spanning 3. Use bpduguard to enforce STP boundary
Blocked link Tree Topology

KB - STP may cause temporary loss of network connectivity when a failover or failback event occurs (1003804)

22
Tips & Tricks

Best Practices for Virtual Networking

Virtual Network Overview

vSwitch Configurations
Tips & Tricks
Troubleshooting Virtual Networks

Whats New in vSphere 5.0

Network Design Considerations

23
Tips & Tricks

Load-Based Teaming (LBT)

Dynamically balance network load over available uplinks

Triggered by ingress or egress congestion at 75% mean utilization over a 30

second period

Configure on DVS via Route based on physical NIC load

*LBT is not available on the Standard vSwitch (DVS feature for ingress/egress traffic shaping)

Network I/O Control (NetIOC)

DVS software scheduler to isolate and prioritize specific traffic types
contending for bandwidth on the uplinks connecting ESX/ESXi 4.1 hosts with
the physical network.

24
 Tips & Tricks

Tip #1 After physical to virtual migration, the VM MAC address can be

changed for Licensed Applications relying on physical MAC address. (KB
1008473)

Tip #2 NLB Multicast needs physical switch Manual ARP resolution of NLB
cluster. (KB 1006525)

Tip #3 Cisco Discovery Protocol (CDP) gives switchport configuration

information useful for troubleshooting (KB 1007069)

Tip #4 - Beacon Probing and IP Hash DO NOT MIX (duplicate packets and port
flapping) (KB 1017612 & KB 1012819)

Tip #5 Link aggregation is never supported on disparate trunked switches Use

VSS with MEC. (KB 1001938 & KB 1027731)

25
Tips & Tricks
Using 10GigE Ingress (into switch)
traffic shaping policy
control on Port Group
Variable/high High
b/w 2Gbps+ 1-2G b/w Low b/w

iSCSI NFS VMotion FT SC

2x 10GigE common/expected
SC#2 10GigE CNAs or NICs
Possible Deployment Method
vSwitch Active/Standby on all Portgroups
Gbps
FCoE
10GE 10GE
FCoE
VMs sticky to one vmnic
10
SC/vmk ports sticky to other
Use Ingress Traffic Shaping
FCoE
to control traffic type per
Port Group
FCoE Priority Group
If FCoE, use
Best Practice: Ensure Drivers and Firmware are compatible forPriority
success
bandwidth reservation
Group
(in CNA config utility)
vSphere 4.1 supports up to (4) 10GigE NICs; 5.0 supports (8) reservation
bandwidth 10GigE NICs (on CNA
utility)
26
Troubleshooting Virtual Networks

Best Practices for Virtual Networking

Virtual Network Overview

vSwitch Configurations
Tips & Tricks
Troubleshooting Virtual Networks

Whats New in vSphere 5.0

Network Design Considerations

27
Network Troubleshooting Tips

Troubleshoot one component at a time

Physical NICs
Virtual Switch
Virtual NICs
Physical Network

Tools for Troubleshooting

vSphere Client
Command Line Utilities
ESXTOP
Third party tools
Ping and Traceroute
Traffic sniffers & Protocol
Analyzers
Wireshark
Logs
28
 Capturing Traffic

Best Practice: create a new management interface for this purpose

vSwitch must be in Promiscuous Mode (KBs 1004099 & 1002934)
ESXi uses tcpdump-uw (KB 1031186)
29
Whats New in vSphere 5.0

Best Practices for Virtual Networking

Virtual Network Overview

vSwitch Configurations
Tips & Tricks
Troubleshooting Virtual Networks

Whats New in vSphere 5.0

Network Design Considerations

30
 Whats New in vSphere 5?

Monitor and troubleshoot virtual infrastructure traffic

NetFlow V5

Port mirror (SPAN)

LLDP (standard based link layer discovery protocol) support simplifies the
network configuration and management in non-Cisco switch environment.
Enhancements to the network I/O control (NIOC)

Ability to create User-defined resource pool

Support for vSphere replication traffic type; a new system traffic type that
carries replication traffic from one host to another.

Support for IEEE 802.1p tagging

Whats New in VMware vSphere 5.0 Networking Technical Whitepaper

31
Network Design Considerations

Best Practices for Virtual Networking

Virtual Network Overview

vSwitch Configurations
Tips & Tricks
Troubleshooting Virtual Networks

Whats New in vSphere 5.0

Network Design Considerations

32
Network Design Considerations
How do you design the virtual network for
performance and availability but maintain isolation
between the various traffic types
(e.g. VM traffic, VMotion, and Management)?
Starting point depends on:
Number of available physical ports on server
Required traffic types
2 NIC minimum for availability, 4+ NICs
per server preferred

802.1Q VLAN trunking highly recommended for logical scaling

(particularly with low NIC port servers)

Examples are meant as guidance and do not represent strict

requirements in terms of design

Understand your requirements and resultant traffic types and

design accordingly

33
Example 1: Blade Server with 2 NIC Ports

Candidate Design:
SC vmkernel Team both NIC ports
Create one virtual switch
Create three port groups:
Portgroup3 Portgroup1 Portgroup2
VLAN 30 VLAN 10 VLAN 20

vSwitch Use Active/Standby policy

for each portgroup
vmnic0 vmnic1
Portgroup1: Service Console (SC)
VLAN Trunks
Portgroup2: VMotion
(VLANs 10, 20, 30)
Portgroup3: VM traffic
Use VLAN trunking
Trunk VLANs 10, 20,
Active
30 on each uplink
Standby

Note: Team over dvUplinks with vDS

34
Example 2: Server with 4 NIC Ports
Candidate Design:
Create two virtual switches
Team two NICs to each vSwitch
SC vmkernel
vSwitch0 (use active/standby
Portgroup4
VLAN 40
Portgroup3
VLAN 30
Portgroup1
VLAN 10
Portgroup2
VLAN 20
for each portgroup):
vSwitch1 vSwitch0 Portgroup1: Service Console (SC)

vmnic0 vmnic2 vmnic1 vmnic3

Portgroup2: VMotion

vSwitch1 (use Originating Virtual

VLANs VLANs PortID)
30, 40 10, 20
Portgroup3: VM traffic #1

Portgroup4: VM traffic #2
Active
Standby Use VLAN trunking
Note: Team over dvUplinks with vDS vmnic1 and vmnic3: Trunk VLANs 10, 20

vmnic0 and vmnic2: Trunk VLANs 30, 40

35
Example 3: Server with 4 NIC Ports (Slight Variation)

Candidate Design:
Create one virtual switch
Create two NIC teams
SC vmkernel
vSwitch0 (use active/standby
Portgroup4
VLAN 40
Portgroup3
VLAN 30
Portgroup1
VLAN 10
Portgroup2
VLAN 20
for portgroups 1 & 2):
Portgroup1: Service Console (SC)
vSwitch0

Portgroup2: Vmotion
vmnic0 vmnic2 vmnic1 vmnic3

Use Originating Virtual PortID

for Portgroups 3 & 4
VLANs VLANs
30, 40 10, 20
Portgroup3: VM traffic #1

Portgroup4: VM traffic #2
Active
Standby
Use VLAN trunking
vmnic1 and vmnic3: Trunk VLANs 10, 20
Note: Team over dvUplinks with vDS
vmnic0 and vmnic2: Trunk VLANs 30, 40

36
Questions

37