Size Matters: MTU Considerations

and Implications in the Data Center

Eric Spielman, Network Consulting Engineer

BRKDCN-1100
Agenda

• Overview of MTU within the Data Center

• Sizing across multiple protocols
• MTU interaction with Data Center technologies
• Ins & Outs of MTU related configuration
• Summary
Session Goal

BRKDCN-1100 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 4
Overview of MTU within the
Data Center
The Data Center Evolution

VXLAN, OTV

FabricPath
vPC / vPC+
STP

Fabric
Extension

Unified Fabric
FCoE, iSCSI

 STP Tiered Design  Full Link Utilization  Host & Network Overlays
 Separate SAN Infrastructure  Converged Fabrics  Increased Scalability

BRKDCN-1100 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 6
Role of MTU

• What is the maximum size of a

frame that a network device will
accept?
• Bidirectional attribute
• Hop-by-hop basis

BRKDCN-1100 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 7
Jumbo Frames 14 byte
Eth Hdr
20 byte
IP Hdr 14 byte
20 byte Eth Hdr
L4 Hdr 20 byte
IP Hdr

• Any payload size greater than Data Payload

20 byte
L4 Hdr
≤ 1460 bytes
1500 bytes
4 byte
• Does not include Ethernet headers Eth FCS

• Enables 6x as much payload data in

one transmit Data Payload
≤ 9176 bytes

• Nexus & UCS support payloads of

up to 9216 bytes

MTU 1500
4 byte
Eth FCS
MTU 9216

BRKDCN-1100 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 8
Why the Draw to Jumbo Frames?

• Larger MTU is needed for protocols and technologies like FCoE, OTV,
and recommended for apps such as vMotion
• Jumbo frames reduce the number of copies and context switches that
occur in the host due to packet overhead reduction
• A 9000 byte frame results in up to 4 context switches and 4 copies to message
in, process, and exit the host
• A typical 1500 byte frame would require 6x as many to receive the same
amount of data
• Improved forwarding and buffer utilization from network devices

BRKDCN-1100 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 9
Finding the Right Size
PMTUD for IPv4

MTU 9000 MTU 1500 MTU 9000

R1 R2
7K-1 1. Host sends 9000 byte packet to discover 7K-2
max MTU
2. R2-R3 link is only 1500 MTU & needs to
Host fragment Server
3. DF-bit prevents & ICMP unreachable is
sent back to Host with link MTU of 1500

Transmit with new MTU value

BRKDCN-1100 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 10
PMTUD
In real life...

ISP

7K-1 7K-2

(config-if)# no ip unreachables
Host Server

BRKDCN-1100 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 11
Effects of MTU Mismatches
• Performance issues:
• Lost packets
• Retransmits
• Truncation
• High delay
• Excess resource utilization

• Issues with dynamic routing

adjacencies
• Fragmentation...

BRKDCN-1100 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 12
The Issue(s) with Fragmentation...
• Breaks down large IP datagrams into smaller datagrams so
that it can be sent on a link with a smaller MTU
• Fragmentation introduces more router load
• Host is responsible for reassembling = more load on the host
• If one fragment is missing, the entire original packet must be
resent
• Only initial fragment contains upper-level protocol headers
• Non-initial fragments will match only source/destination IP
information in ACLs
• May be vulnerable to attack

BRKDCN-1100 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 13
The Bigger The Better?

DC02

• Jumbo frame support is not widespread P1.a

P1.b
• North / South and inter-DC traffic should be
cautious of going above 1500 MTU ☹️✖
• Never assume
DC01
• Doesn’t play well with real-time traffic
• More prone to performance issues in terms
of RTT and latency

P1

BRKDCN-1100 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 14
It’s Good Where It Makes Sense

• East / West traffic; the busy parts

of the network
• Storage traffic
• Workload mobility
• Isolated traffic

• Private WAN where there is control

& visibility into the complete traffic
path
• All stars need to align

BRKDCN-1100 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 15
Sizing Across Multiple Protocols
Making It All Flow...

BRKDCN-1100 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 17
NX-OS System QoS Policies
Policy Application Point
system ingress egress
Policy Type Function
qos interface interface
Define traffic
qos classification ✓ ✓
Apply queuing
strategy; strict
queuing priority, deficit ✓ ✓ ✓
weight round
robing
Define system
class: drop/no-
network-qos drop, MTU, ✓
buffer size

• Match and define policy using the respective type of class-maps and
policy-maps
BRKDCN-1100 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 18
 QoS Policy Types
*Platform dependent criteria

type qos type network-qos type queuing

Class-map Class-map Class-map
CoS, DSCP, ACL, IPP, CoS, Protocol CoS, DSCP
Protocol (TLV) qos-group qos-group

Policy-map Policy-map Policy-map

Marking, policing, mutation MTU, No-drop (lossless Ethernet) Priority, queueing, buffering,
scheduling
Set qos-group Congestion control

BRKDCN-1100 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 19
Type Network-QoS Policies

• One per system that is applied to the

Unified Crossbar and affects all L2
ports
• Exception is native FC interfaces
• Controls the shape of the traffic and
how it reacts
• Ability for varying MTU values among
classes
• Why use CoS?

Ethernet Wire

BRKDCN-1100 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 20
IPv4 Traffic
• MTU may be determined based on:
• Interface configuration <mtu x>
• network-qos policy: mapped into classes based on CoS/QoS marking or qos-groups
• Otherwise the default class value is used

• L3 hop where packet exceeds MTU: fragment except when the DF-bit is present
• Switched traffic exceeding the MTU is truncated or dropped

Consistent MTU scheme

Ingress Ingress Ingress

Module Module Module
Ingress Egress Ingress Egress Ingress Egress
Module Fabric Module Module Fabric Module Module Fabric
Module
Ingress Ingress Ingress
Module Module Module

BRKDCN-1100 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 21
IPv6

• No fragmentation supported on routers!

• Hosts are responsible for fragmentation
• Must rely on PMTUD, consistent MTU scheme, RA messages
• ICMPv6 Type2 messages must be allowed for PMTUD operation
• MTU for IPv6 traffic may be set with the interface-level MTU command
or within a network-qos class

BRKDCN-1100 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 22
IPv6
• NX-OS has one interface-level MTU command: mtu <size>
• Influences IPv4 & IPv6 alike
• Recommendation: have IPv6 routers send hosts the MTU that should be used
via Router Advertisements (ICMP packets)
N7706-A(config-if)# ipv6 nd mtu 1480
N7706-A(config-if)# sh ipv6 nd int e1/11
ICMPv6 ND Interfaces for VRF "default"
Instructs the host to use an MTU of Ethernet3/1, Interface status: protocol-down/link-
down/admin-down
1480 in packets it originates IPv6 address: 0dc3:0dc3:0000:0000:0218:baff:fed8:239d RA
ICMPv6 active timers:
Last Neighbor-Solicitation sent: never
Last Neighbor-Advertisement sent: never
Last Router-Advertisement sent:never
Next Router-Advertisement sent in: 0.000000
Router-Advertisement parameters:
Periodic interval: 200 to 600 seconds
Send "Managed Address Configuration" flag: false
Send "Other Stateful Configuration" flag: false
Send "Current Hop Limit" field: 64
Send "MTU" option value: 1480

BRKDCN-1100 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 23
iSCSI

• Block-level storage access over IP

• Can co-exist with LAN infrastructure
• TCP or PFC

• iSCSI with Jumbo Frames

• GOOD: Allows for entire SCSI data blocks in
packets = less buffer wait time and throughput is
consumed more efficiently
• NOT SO GOOD: MTU mismatches severely hinder
performance with potential disconnects and data
corruption

BRKDCN-1100 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 24
 iSCSI Best Practices
WAN
✓
Intra-DC iSCSI
with existing LAN Inter-DC iSCSI
Multiple L3 hops

Remote
iSCSI

LAN SAN
✓
iSCSI direct • Use Jumbo Frames when there is control
attach to UCS FI
over all transit devices
• Best practice: isolate iSCSI traffic (i.e.
dedicated replication/backup VLANs), apply
separate CoS value
BRKDCN-1100 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 25
Sample VLAN-based iSCSI Policy
Reference 5672 Platform
IP ACL matches the well-known
iSCSI ports for the QoS policy
Sets the internal qos-group for
system class mapping and
applied in the VLAN database
Network-qos policy matches the
iSCSI traffic with the qos-group
and increases the MTU
ip access-list iSCSI-ACL
permit tcp 10.0.0.0/24 any eq 860
permit tcp 10.0.0.0/24 any eq 3260
!
class-map type qos match-all class-iSCSI
match access-group name iSCSI-ACL
!
policy-map type qos iSCSI-QOS-policy
class class-iSCSI
set qos-group 2
set cos 4
class class-default
!
vlan configuration 10
service-policy type qos input iSCSI-QOS-policy
!
class-map type network-qos iSCSI-nq
match qos-group 2
!
policy-map type network-qos jumbo-nq-policy
class type network-qos c-nq-iSCSI
mtu 9000
pause no-drop
class type network-qos c-nq-default
mtu 1500
!
system qos
service-policy type network-qos jumbo-nq-policy

BRKDCN-1100 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 26
NAS
• File-level storage access over an IP
network
• NFS or SMB
• Storage can be shared simultaneously across
multiple computers
• TCP or UDP based
• May benefit from increased MTU but not
required LAN SAN
• Mismatches result in latency and slow
performance
• Similar to iSCSI principles, assign a consistent
CoS/QoS policy

BRKDCN-1100 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 27
Fibre Channel
• Supported on Unified Port (UP) switches that include the Nexus 5500/5600/6000
series switches as well as the 2348UPQ FEX
• Requires Jumbo Frames
• Maximum receive size of 2148 bytes with headers
• Maximum payload of 2112 bytes

• MTU adjustments are hard-coded when interface is converted to ‘fc’ and traffic is
identified/classified based on the physical interface it is received on

SOF FC Header SCSI Payload CRC EOF

Commands & Data
4bytes 24bytes 2112 bytes 4bytes 4bytes

Control information,
Frame delimiters src/dst addresses
BRKDCN-1100 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 28
Fibre Channel over Ethernet (FCoE)
• Still Fibre Channel
• FC frame is encapsulated with an FCoE header and regular Ethernet headers
• Classified based on its EtherType

• CoS 3 is the standard FCoE QoS marking and will auto-associate with this class
• ELP requires that rxbufsize match between peers, otherwise link fails

2180 Bytes

Ethernet
Header FCoE SCSI Payload
Header FC Header CRC FCSEOF
12bytes Commands & Data
16bytes 24bytes 2112 bytes 4bytes 4bytes 4bytes
+ 4bytes
.1Q tag

FCoE identified
with Ethertype
0x8906
BRKDCN-1100 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 29
FCoE Among Platforms

N7706-Admin-A# sh policy-map system

N56128-A# sh policy-map system
Type network-qos policy-maps
Type network-qos policy-maps
============================
============================
policy-map type network-qos jumbo-nq-7e-4q8q-policy template 7e-4q8q
policy-map type network-qos jumbo+_fcoe-nq-policy
class type network-qos c-nq-7e-4q8q-drop
class type network-qos class-fcoe
congestion-control tail-drop
pause no-drop
mtu 9000
mtu 2158
class type network-qos c-nq-7e-4q8q-ndrop-fcoe
class type network-qos class-default
pause
mtu 9000
mtu 2112

Header length is internally added and

implemented in hardware with fixed sizes N7K: Must use a dedicated Storage
2112 + 18 (eth_hdr&CRC) + 28 VDC and a network-qos policy
(FC_hdr&CRC) = 2158 bytes

BRKDCN-1100 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 30
Fibre Channel over IP (FCIP)

FC SAN
FC SAN

FCIP FCIP
end-point end-point

• Long distance SAN extension between FCIP end-points

• Available on MDS platforms
• Leverages TCP for resilience and congestion control

BRKDCN-1100 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 31
FCIP Frame

S
Ethernet IP TCP TCP FCIP EISL Opt
O FC Frame CRC
Header Header Header Opts Header Hdr Hdr
F

14 20 20 12 28 4 8 0-16 4

FCIP Overhead: 98 Bytes Maximum FC frame size:

Includes 94 byte header 2148 bytes
& 4 byte CRC

• FCIP will segment and reassemble FC frames if MTU exceeds

• Enabling Jumbo frames across the path will increase performance
• DF-bit is set between FCIP end-points

BRKDCN-1100 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 32
FCIP MTU & Discovery
MDS-A(config)# feature fcip
MDS-A(config)# int gi1/1
Increase MTU on physical MDS-A(config-if)# ip address 10.1.1.1/24
interface; will also increase MDS-A(config-if)# mtu 2300
on FCIP interface MDS-A(config)# fcip profile 10
• FCIP MTU MDS-A(config)# ip address 10.1.1.1
MDS-A(config)# int fcip 10
recommendation is MDS-A(config-if)# use-profile 10
MDS-A(config-if)# peer-info ipaddr10.1.1.2
2300 bytes MDS-A(config-if)# no shut
MDS-A(config-if)# show fcip profile 10
FCIP Profile 10
• Path MTU Internet Address is 10.1.1.1 (interface GigabitEthernet1/1)
Tunnels Using this Profile: fcip10
Discovery is Any MTU changes across Listen Port is 3225
TCP parameters
path will be discovered
enabled by default within 3600 secs
SACK is enabled
PMTU discovery is enabled, reset timeout is 3600 sec
FCIP end-points Keep alive is 60 sec
Minimum retransmission timeout is 200 ms
Maximum number of re-transmissions is 4
Send buffer size is 16384 KB
Maximum allowed bandwidth is 5000000 kbps
Minimum available bandwidth is 4000000 kbps
Configured round trip time is 1000 usec
Congestion window monitoring is enabled, burst size is 50 KB
Auto jitter detection is enabled

BRKDCN-1100 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 33
MTU Interaction with Data Center
Technologies
vPC
Overview

STP vPC Physical View vPC Logical View

7K-1 7K-2
7K-1 7K-2 7K-1 7K-2

SW1 SW1 SW1

• All links forwarding: no blocked ports, increased bandwidth, load sharing
• High availability & redundancy

BRKDCN-1100 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 35
vPC
MTU Areas of Interest
vPC Domain

• Peer-Link vPC Peer-Keepalive

• Internally programmed for 9216+

padding vPC Peer-Link
• Extra padding is due to Cisco Data
Center Ethernet Protocol (CDCE) vPC Member
Ports
• MAC-in-MAC
• EtherType 0x8903
• Overhead auto-accounted for vPC vPC vPC

• Downstream device
• Should have same MTU as vPC
Member Ports

BRKDCN-1100 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 36
vPC
Member Port MTU
• Applies to switches where interface-level MTU config is used
• MTU config must match on vPC port-channel member interfaces between
vPC peers
• Physical interfaces auto-inherit
• Mismatch causes forwarding interruption for that specific vPC (Type-1)
N7706-A# sh vpc consistency-parameters inter po 11

Legend:
Type 1 : vPC will be suspended in case of mismatch

Name Type Local Value Peer Value

------------- ---- ---------------------- -----------------------
mode 1 active active
Speed 1 10000 Mb/s 10000 Mb/s
Duplex 1 full full
Port Mode 1 trunk trunk
Native Vlan 1 1 1
MTU 1 9000 9000
Admin port mode 1 trunk trunk
BRKDCN-1100 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 37
vPC
Global Consistency Check
• Applies to Nexus switches with network-qos policies
• vPC still operates despite MTU mismatch within the network-qos policy
(Type-2)
• *No reason for it not to match
Nexus56128P-A# sh vpc consistency-parameters global

qos-group 0 (default) Legend:

Type 1 : vPC will be suspended in case of mismatch

Name Type Local Value Peer Value

------------- ---- ---------------------- -----------------------
QoS 2 ([], [3], [], [], [], ([], [3], [], [], [],
[]) [])
Network QoS (MTU) 2 (9216, 2196, 0, 0, 0, (9216, 2196, 0, 0, 0,
0) 0)
qos-group 1 (FCoE) Network Qos (Pause) 2 (F, T, F, F, F, F) (F, T, F, F, F, F)

BRKDCN-1100 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 38
Why Overlays Encapsulated Traffic

Virtual Switch Virtual Switch

Resilient Underlay Overlay Virtual Network

• High capacity interconnects • Scalable
• Transit devices are agnostic • Flexible
• Must be capable of handling • L2 or L3
overhead
BRKDCN-1100 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 39
FabricPath
Benefits

S10 S20 S30 S40 FabricPath

S100 S200 S300 S400 S500 S600

• Maintains Layer 2 connectivity without the use of STP

• Efficient while utilizing ECMP
• Enables flexibility within the Data Center and is highly scalable

BRKDCN-1100 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 40
FabricPath
Control Plane Operation
• Utilizes IS-IS to dictate the forwarding topology
• Essentially L3 routing to the L2 bridged domain
• Computes shortest path within the FabricPath domain
• Supports the use of equal-cost paths

• No L2 IS-IS configuration needed but is tunable

• IS-IS LSPs have an MTU of 1492 bytes
• Can adjust with lsp-mtu <size> command but no documented performance benefit
since it is strictly control traffic
• Recommend to leave at default

BRKDCN-1100 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 41
FabricPath
Data Plane

All fields retained, new CRC

S10 S20 S30 S40

OutDA OutSA FPTag DMAC SMAC .1Q Etype Payload CRC

S100 S200 S300 S400 S500 S600

16 bytes total overhead

within FP Domain DMAC SMAC .1Q Etype Payload CRC

FP header removed as
frame returns to CE domain
Classical Ethernet Frame
Ingress into FP Domain

BRKDCN-1100 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 42
FabricPath
Data Plane
• The 16 bytes of overhead is automatically accommodated by the ASICs when
the interface is converted to switchport mode fabricpath
• Same EtherType (CDCE) as on vPC Peer-Link
• MTU configurations and adjustments on the platform are reflected in the
FabricPath forwarding domain:

N7706-Spine-A# show fabricpath isis interface brief

Fabricpath IS-IS domain: default
Interface Type Idx State Circuit MTU Metric Priority Adjs/AdjsUp
--------------------------------------------------------------------------------
port-channel10 P2P 1 Up/Ready 0x01/L1 9000 10 64 1/1

port-channel20 P2P 3 Up/Ready 0x01/L1 9000 10 64 1/1

port-channel30 P2P 2 Up/Ready 0x01/L1 9000 10 64 1/1

BRKDCN-1100 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 43
 OTV
OTV
L3 DCI
Overview OTV Edge
OTV Edge

OTV Edge OTV Edge

OTV Inline
• LAN Extension over any network Core Core
Core
Core
• Dark Fiber, MPLS, IP... West DC East DC
• Multi-data center scalability with VLANs 100-109 VLANs 100-109
ability for workload mobility
OTV Join
• Network-based overlay with Interfaces
Core Transit
stateless tunneling Overlay Interface

OTV
Edge Core Device
OTV on a stick
Aggregation Device
Internal Interfaces

BRKDCN-1100 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 44
OTV
Packet Types

Packet Type Purpose IP Length

ISIS Hello; Establish OTV
1450 Bytes
Unicast Adjacency
ISIS Hello; Establish OTV *Recommended
1442 Bytes OTV control
Multicast* Adjacency
Added to all OTV
OTV Header Pkt_len+42 Bytes
Data traffic

• In order to forward a standard 1500 byte packet from a host application,

ensure transport supports at least a 1542 byte MTU end-to-end
• Alternatively the host MTU could be decreased to accommodate overhead

BRKDCN-1100 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 45
OTV
Data Packet Format
New L2 Header
• MAC-in-IP 14 Bytes
• 802.1Q tag is removed from original L2 New IP Header
header 20 Bytes
• VLAN ID copied to OTV Shim OTV OTV Shim
• 802.1P bits copied to new IP Header and Overhead 8 Bytes
OTV Shim Original L2 Header
14 Bytes
• Fragmentation is not supported due to L2
presumption and ‘hiding’ of L3 network Original
• DF bit is set on all OTV control and data Original IP Packet Frame
traffic between Edge devices Payload

• No PMTUD either
L2 FCS

BRKDCN-1100 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 46
OTV MTU ≥ 1542 bytes
Interface Config

7K-A-CORE(config-if)# int po10

7K-A-CORE(config-if)# description OTV Internal Int
7K-A-CORE(config-if)# mtu 1542
7K-A-CORE(config-if)# int e1/1
7K-A-CORE(config-if)# description Core Uplink
7K-A-CORE(config-if)# mtu 1542
OTV Edge
Device Core Device

Aggregation Device
7K-A-OTV(config-if)# int po10
7K-A-OTV(config-if)# description Join Interface
7K-A-OTV(config-if)# mtu 1542
7K-A-OTV(config-if)# int overlay1
7K-A-OTV(config-if)# otv join-interface po10

OTV Internal Interface

OTV Join Interface

BRKDCN-1100 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 47
VXLAN
What Does it Accomplish?

• Flexible host-based overlay with

mobility across data centers and
clouds
• Segmentation and Scale Underlay

• Stretches the L2 segment ID field to 24

bits = up to 16 million L2 segments
• Extends L2 segments over a L3
boundary – VXLAN encapsulates L2
frames in an IP/UDP header
• Allows for all links forwarding via L3 VXLAN VNI 100
ECMP

BRKDCN-1100 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 48
VXLAN
1500 byte packet
Overhead and Transport 14 byte inner-Ethernet header
8 byte VXLAN ID
8 byte UDP header
VXLAN Encapsulation 20 byte IP header
50 Bytes (possibly 54) 1550 byte minimum VXLAN MTU
MAC Header IP Header UDP Hdr
14 bytes (4 bytes optional)_ 20 Bytes 8 Bytes

Outer Outer Outer Outer VXLAN Inner Inner

Outer Outer Ethernet
MAC MAC IP IP ID MAC MAC CRC
802.1Q UDP Payload
DA SA DA SA (8 bytes) DA SA

VM MAC
addresses
Transport MAC UDP destination The original frame; at
addresses Transport port 4789 identifies VNI Field default 1514 bytes or
VLAN IP addresses the packet as identifies the L2 1518 bytes
(Optional) of the VTEPs VXLAN traffic; segment
source UDP is a
hash of inner frame

BRKDCN-1100 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 49
VXLAN
Operation
• Virtual Tunnel End Points
(VTEP) perform VXLAN
IPv4 Transport
encapsulation / de-capsulation
• Add 50 bytes to transit links to
VTEP support MTU of 1550 bytes
Hypervisor • Alternatively MTU could be
decreased by 50 bytes on
servers
VTEP VTEP

• When 1000v is used as VTEP,

LAN segment for bridging
& IP interface known in the it can assist with MTU issues
transport network (usually
a loopback)

BRKDCN-1100 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 50
Ins and Outs of MTU Related
Configurations
Logical Flow

Operating System Compute Switching

Storage

Nexus
OS Kernel Driver Adapter IOM FI
Switch
1000v IP

MTU

BRKDCN-1100 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 52
Influencing Traffic Size
IOS vs. NX-OS

IOS NX-OS
ip mtu mtu
mtu network-qos policy
ip tcp adjust-mss
system mtu
...

BRKDCN-1100 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 53
Nexus 7000 & 7700 Series

• Store-and-forward switch
• May use interface-level MTU
commands or network-qos policy
depending on interface
• Features such as FCoE and FEX
require network-qos policies

BRKDCN-1100 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 54
Interface-Level MTU
Layer 2
• Jumbo MTU is supported on the
switch by default but needs to be
enabled
• Interface-level must use the
standard default MTU of 1500 or
match the value specified in
system jumbomtu command
N7706-A(config)# sh run all | i jumbomtu
system jumbomtu 9216
N7706-A(config)# sh int e1/1
Ethernet1/1 is up
admin state is up, Dedicated Interface
Hardware: 1000/10000 Ethernet, address:
Description: 5K-A 1/4
MTU 1500 bytes, BW 10000000 Kbit, DLY 10 usec
reliability 255/255, txload 1/255, rxload 1/255
N7706-A(config)# int e1/1
N7706-A(config-if)# mtu 9216
N7706-A(config)# sh int e1/1
Ethernet1/1 is up
admin state is up, Dedicated Interface
Hardware: 1000/10000 Ethernet, address:
Description: 5K-A 1/4
MTU 9216 bytes, BW 10000000 Kbit, DLY 10 usec
reliability 255/255, txload 1/255, rxload 1/255

• Applies for port-channels as well

Nexus7706-A(config)# int e1/1
Nexus7706-A(config-if)# mtu 9000
ERROR: Ethernet1/1: MTU on L2 interfaces can only be set to default or system-jumboMTU

BRKDCN-1100 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 55
 N7K network-qos Policies
• Alternative MTU configuration is via the
‘network-qos’ policy in the Admin VDC
• Applies to L2 traffic in all VDCs
• Class-maps match on CoS and protocol
• Disruptive
• Default policy cannot be modified; copy
& modify a predefined template or
create one that conforms to a template
• Any interface-level MTU will take
precedence
Example policy with 7 drop
classes & 1 no-drop (FCoE)
N7706-A# qos copy policy-map type network-qos default-nq-7e-4q8q-policy
prefix CUSTOM-
N7706-A# conf
N7706-A(config)#policy-map type network-qos CUSTOM-nq-7e-4q8q-policy
N7706-A(config-pmap-nqos)# class type network-qos c-nq-7e-4q8q-drop
N7706A(config-pmap-nqos-c)# mtu 9216
N7706A(config-pmap-nqos-c)# class type network-qos c-nq-7e-4q8q-ndrop-
fcoe
N7706A(config-pmap-nqos-c)# mtu 2112
N7706A(config-pmap-nqos-c)# system qos
N7706-A(config-sys-qos)# service-policy type CUSTOM-nq-7e-4q8q-policy
!
N7706-A# sh policy-map system type network-qos
policy-map type network-qos CUSTOM-nq-7e-4q8q-policy template 7e-4q8q
class type network-qos c-nq-7e-4q8q-drop
congestion-control tail-drop
mtu 9216
class type network-qos c-nq-7e-4q8q-ndrop-fcoe
pause
mtu 2112
BRKDCN-1100 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 56
N7K network-qos Templates

Template Drop CoS Priority NoDrop CoS

• CoS 6 & 7 reserved for default-nq-

0,1,2,3,4,5,6,7 5,6,7 -
control traffic (non- 8e-policy
configurable) default-nq-
0,1,2,4,5,6,7 5,6,7 3
7e-policy
• CoS 3 reserved for
FCoE default-nq-
0,1,2,5,6,7 5,6,7 3,4
6e-policy
default-nq-
0,5,6,7 5,6,7 1,2,3,4
4e-policy

BRKDCN-1100 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 57
Nexus 7K
Layer 3 Interfaces
• Configured directly on any L3
interface: L3 ports, subinterfaces,
SVIs, port-channels, port-channel
subinterfaces
• Parent port-channel interface must have
an equal or greater size
• Values can differentiate among L3
N7706-A(config-if)# sh int po10.100
interfaces with up to 64 settings per
module
N7706-A(config)# int po10
N7706-A(config-if)# mtu 9000
N7706-A(config-if)# int po10.100
N7706-A(config-if)# mtu 2200
N7706-A(config-if)# sh int po10
port-channel10 is up
admin state is up
Hardware: Port-Channel, address:
Description: 7K-A Agg
MTU 9000 bytes, BW 10000000 Kbit, DLY 10 usec
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, medium is broadcast
Port mode is routed
port-channel10.100 is up
admin state is up, [parent interface is port-channel10]
Hardware: Port-Channel, address:
Internet Address is 10.1.1.1/30
MTU 2200 bytes, BW 10000000 Kbit, DLY 10 usec
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation 802.1Q Virtual LAN, Vlan ID 100, medium is broadcast
Port mode is routed
BRKDCN-1100 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 58
 Nexus 7K Verification & Troubleshooting
N7706-A# sh system internal qos network-qos hw-config | i MTU|slot|CoS
slot 1
UF Configs for CoS 0
MTU = 1500 [FCoE: No]
UF Configs for CoS 1
MTU = 1500 [FCoE: No]
UF Configs for CoS 2
Software verification of port MTU = 1500 [FCoE: No]
interface MTU UF Configs for CoS 3
MTU = 2112 [FCoE: Yes]
UF Configs for CoS 4
MTU = 1500 [FCoE: No]
N7706-A# sh system internal ethpm info interface po10 | i UF Configs for CoS 5
Inidicates whether port is
MTU MTU = 1500 [FCoE: No]
UF Configs for CoS 6 using intf level MTU or
medium (broadcast), snmp trap(on), MTU(9000),
MTU = 1500 [FCoE: No] inheriting the MTU from the
UF Configs for CoS 7 Virtual Link
MTU = 1500 [FCoE: No]
Interface Config Oper(VLs) MTU (value)
Ethernet1/1 Auto Off(0) Port-MTU (9000)
Ethernet1/2 Auto Off(0) Port-MTU (9000)
Ethernet1/3 Auto Off(0) Port-MTU (9000)
Ethernet1/4 Auto Off(0) VL-MTU (-)
Ethernet1/5 Auto Off(0) Port-MTU (9000)
Ethernet1/6 Auto Off(0) Port-MTU (9000)
Ethernet1/7 Auto Off(0) Port-MTU (9000)
Ethernet1/8 Auto Off(0) Port-MTU (9000)
Ethernet1/9 Auto Off(0) Port-MTU (9000)
Ethernet1/10 Auto Off(0) Port-MTU (9000)
Ethernet1/11 Auto Off(0) Port-MTU (9000)
Ethernet1/12 Auto Off(0) Port-MTU (9000)
Ethernet1/13 Auto Off(0) VL-MTU (-)
Ethernet1/14 Auto Off(0) VL-MTU (-)
BRKDCN-1100 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 59
 Nexus 7K Verification & Troubleshooting
Equals 9038 in decimal
(internal header
A configurable rate-limit for packets needing to be accommodation)
fragmented may be configured; by default it is 500 pps N7706-A-Admin# attach module 1
Attaching to module 1 ...
To exit type ‘exit’, to abort type ‘$.’
N7706-A# sh hardware rate-limiter layer-3 mtu module-1# sh hardware internal mac port 1 qos | i MTU
Port MTU (No dropping action): 9038
Units for Config: packets per second
Allowed, Dropped & Total: aggregated since last clear counters
rl-1: STP and Fabricpath-ISIS module-1# sh hardware internal mac port 1 sw-state | b MTU
rl-2: L3-ISIS and OTV-ISIS # Port MTU #
rl-3: UDLD, LACP, CDP and LLDP port adj_mtu=0x26 last_mtu=0x2316 qos_mtu=0x0 pm_mtu=0x234e
rl-4: Q-in-Q and ARP request vl0 adj_mtu=0x16 last_mtu=0x2328 qos_mtu=0x2328 pm_mtu=0x2600
rl-5: IGMP, NTP, DHCP-Snoop, Port-Security, Mgmt and Copy traffic vl1 adj_mtu=0x16 last_mtu=0x2328 qos_mtu=0x2328 pm_mtu=0x2600
vl2 adj_mtu=0x16 last_mtu=0x2328 qos_mtu=0x2328 pm_mtu=0x2600
vl3 adj_mtu=0x16 last_mtu=0x2328 qos_mtu=0x2328 pm_mtu=0x2600
Module: 1 vl4 adj_mtu=0x16 last_mtu=0x2328 qos_mtu=0x2328 pm_mtu=0x2600
vl5 adj_mtu=0x16 last_mtu=0x2328 qos_mtu=0x2328 pm_mtu=0x2600
Rate-limiter PG Multiplier: 1.00 vl6 adj_mtu=0x16 last_mtu=0x2328 qos_mtu=0x2328 pm_mtu=0x2600
vl7 adj_mtu=0x16 last_mtu=0x2328 qos_mtu=0x2328 pm_mtu=0x2600
R-L Class Config Allowed Dropped Total
+------------------+--------+---------------+---------------+-----------------+
L3 mtu 500 106 0 106

Any packets exceeding the rate-

limit will be dropped and counted

BRKDCN-1100 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 60
Nexus 9000 (Standalone)
MTU Overview

• Similar to 7K
• Configure MTU for L3 interfaces
directly on the interface
• Interface-level MTU for L2 interfaces is
supported (1500 or system
jumbomtu <size>)
• System policies (network-qos) may
also be used to apply to all L2
interfaces

BRKDCN-1100 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 61
Nexus 5500 & 5600
Architecture Overview

• Switching paradigms:
• Cut-through switch [default for 10G]: frames exceeding the
MTU are truncated and result in eventual CRC errors
• Store-and-forward: frames are dropped when MTU is
exceeded
• Layer 3 capabilities are integrated at line rate for 5600
series and are available with L3 module for 5500 series

BRKDCN-1100 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 62
Nexus 5500 & 5600
Switching Modes Src Int Dst Int Switching Mode

10G Ethernet 10G Ethernet Cut-through

10G Ethernet 1G Ethernet Cut-through

• When 10G
Ethernet is used 1G Ethernet 1G Ethernet Store-and-Forward
ingress, cut-
through switching 1G Ethernet 10G Ethernet Store-and-Forward
is used by default
FCoE Fibre Channel Cut-through

Fibre Channel FCoE Store-and-Forward

Fibre Channel Fibre Channel Store-and-Forward

FCoE FCoE Cut-through

BRKDCN-1100 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 63
Nexus 5500 & 5600
Layer 2 Interfaces
• Must use a nework-qos policy-map
• Per interface MTU not supported
• One system-wide policy MTU may be configured
on a per class basis
• qos-groups uniquely identify the system
classes internally
• 5 configurable, 1 default class and 2 reserved for
control traffic
• System defaults
• system jumbomtu 9216; configurable from
2158 to 9216 bytes
• Default network-qos policy cannot be modified

BRKDCN-1100 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 64
Nexus 5500 & 5600
Layer 2 MTU Configuration

N5672-A(config)# policy-map type network-qos jumbo-nq-policy

N5672-A(config-pmap-nq)# class type network-qos class-fcoe Both commands
N5672-A(config-pmap-nq-c)# pause no-drop required for
N5672-A(config-pmap-nq-c)# mtu 2158
N5672-A(config-pmap-nq-c)# class type network-qos class-default
FCoE class
N5672-A(config-pmap-nq-c)# mtu 9000
N5672-A(config)# system qos
N5672-A(config-sys-qos)# service-policy type network-qos jumbo-nq-policy
N5672-A# sh policy-map system

Type network-qos policy-maps

===============================

policy-map type network-qos jumbo-nq-policy

class type network-qos class-fcoe
match qos-group 1
pause no-drop
mtu 2158
class type network-qos class-default
match qos-group 0
mtu 9000

BRKDCN-1100 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 65
 Nexus 5500 & 5600
L2 MTU verification N5672-A# sh queuing int e1/37
Ethernet1/37 queuing information:
N5672-A# sh int e1/37 TX Queuing
Ethernet1/37 is up qos-group sched-type oper-bandwidth
Dedicated Interface 0 WRR 50
Hardware: 1000/10000 Ethernet, address: 1 WRR 50
Description: Dell VRTX Eth8/1
MTU 1500 bytes, BW 10000000 Kbit, DLY 10 usec RX Queuing
reliability 255/255, txload 1/255, rxload 1/255 qos-group 0
Encapsulation ARPA q-size: 100160, HW MTU: 9000 (9000 configured)
Port mode is trunk drop-type: drop, xon: 0, xoff: 0
full-duplex, 10 Gb/s, media type is 10G Statistics:
<output omitted> Pkts received over the port : 449082142
30 seconds input rate 93872 bits/sec, 38 packets/sec Ucast pkts sent to the cross-bar : 448848703
30 seconds output rate 170712 bits/sec, 101 packets/sec Mcast pkts sent to the cross-bar : 233439
Load-Interval #2: 5 minute (300 seconds) Ucast pkts received from the cross-bar : 244633695
input rate 80.43 Kbps, 15 pps; output rate 154.54 Pkts sent to the port : 317668644
Kbps, 78 pps Pkts discarded on ingress : 0
RX Per-priority-pause status : Rx (Inactive), Tx (Inactive)
38255265 unicast packets 256631 multicast
packets 418643 broadcast packets qos-group 1
38930539 input packets 23961814506 bytes q-size: 165120, HW MTU: 2158 (2158 configured)
11702772 jumbo packets 0 storm suppression bytes drop-type: no-drop, xon: 62720, xoff: 88320
<output omitted> Statistics:
TX Pkts received over the port : 3446851283
71224821 unicast packets 89774097 multicast Ucast pkts sent to the cross-bar : 3446902243
packets 4865803 broadcast packets Mcast pkts sent to the cross-bar : 0
165864721 output packets 84561892595 bytes Ucast pkts received from the cross-bar : 2610115540
45431246 jumbo packets Pkts sent to the port : 2610115564
<output omitted> Pkts discarded on ingress : 0
Per-priority-pause status : Rx (Active), Tx (Active)

BRKDCN-1100 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 66
 Nexus 5500 & 5600 Troubleshooting
• Using cut-through forwarding, frame is passed to egress ASIC once DA is
parsed
N56128-A# sh hardware internal bigsur port e1/48 counters rx
Excellent command for
5K is receiving 9216 byte seeing packet sizes / types
frames while configured for Port 10gb1/48 MAC statistics: on an interface
only 1500 MTU MAC Statistic | Value | Diff | Rate |
-------------------------------+-----------------------------------+----------------------+--------------+
<snip>
RX_PKT_SIZE_IS_8192_TO_9216 |181538 |181538 |0

N56128-A# sh hardware internal bigsur all-ports

Bigsur Port Info:

Determine which bigsur ASIC Port |asic|inst|inst|
the interface belongs to... name |idx |slot|asic|eport|logi|flag|adm|opr|if_index|diag|ucVer
---------+----+----+----+-----+----+----+---+---+--------+----+-----
<snip>
xgb1/48 |3 |0 |3 |11 p |47 |b3 |en |up |1a02f000|pass| 0.00

N56128-A# sh hardware internal bigsur asic 3 counters interrupt | i mtu

Switch is detecting
MTU violations; Slot 0 Bigsur3 interrupt statistics:
packets are truncated Interrupt name |Count |ThresRch|ThresCnt|Ivls
-----------------------------------------------+--------+--------+--------+----
<snip>
big_bmin_cl1_INT_p1_err_ig_mtu_vio |30 |0 |30 |0

BRKDCN-1100 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 67
Nexus 5500 & 5600 Troubleshooting

N56128-A# sh hardware internal bigsur port e1/4 counters rx

On the egress 5K interface;
output errors are also Port 10gb1/4 MAC statistics:
shown in ‘show interface’ MAC Statistic | Value | Diff | Rate |
-------------------------------+-----------------------------------+----------------------+--------------+
<snip>
TX_PKT_FRAME ERROR |181538 |181538 |0

N56128-A# sh hardware internal bigsur asic 0 counters interrupt | i crc

Slot 0 Bigsur0 interrupt statistics:

Frames are CRC stomped Interrupt name |Count |ThresRch|ThresCnt|Ivls
-----------------------------------------------+--------+--------+--------+----
<snip>
big_fwe_psr1_P2_INT_pkt_err_eth_crc_stomp |30 |0 |30 |0

BRKDCN-1100 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 68
Nexus 5500 & 5600
L3 Interfaces

• Set MTU directly on the interface

• Use mtu <size> syntax under the interface (physical, SVI,
Port-Channel)
• Can only support an MTU of up to 9192 on L3 interfaces
• Caused by 24 byte switch internal header
• Implies that other devices should be adjusted to
accommodate if L3 is needed on 5K

BRKDCN-1100 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 69
Nexus 3000, 3100, 3500 Series

Nexus 3000 Nexus 3100, 3500

• Per-port MTU configuration • network-qos policy must be used
supported for L2 and L3 interfaces
• May use to network-qos policy to
affect all ports

BRKDCN-1100 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 70
Nexus 2200 & 2300FEX
Architecture Overview

• Jumbo frame support is dependent

on the parent switch configuration
• Auto-inherit of network-qos policy
• No local switching on FEX
• FEX links use a VNTag (6 bytes) & an
802.1Q tag (4 bytes)
• Programed when NIFs are configured

BRKDCN-1100 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 71
Nexus 2200 & 2300FEX
7K Parent

Example enables jumbo

• 6.2(2)+:All FEX MTU controlled by
network-qos policy
• Prior releases only MTU modification was
needed on FPC
• Create a new network-qos policy in the
Default or Admin VDC
• Changes are automatically pushed
down to the FEX
frames for all traffic classes
N7706-Admin-A(config)# class-map type network-qos
match-any class-nq-8e
N7706-Admin-A(config-cmap-nqos)# match cos 0-7
N7706-Admin-A(config)# policy-map type network-qos
jumbo-nq-8e template 8e-4q8q
N7706-Admin-A(config-pmap-nqos)# class-map type
network-qos class-nq-8e
N7706-Admin-A(config-pmap-nqos-c)# mtu 9000
N7706-Admin-A(config-pmap-nqos-c)# congestion tail-
drop
N7706-Admin-A(config)# system qos
N7706-Admin-A(config-sys-qos)# service-policy type
network-qos jumbo-nq-8e

BRKDCN-1100 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 72
Nexus 1000v
VMware (N1K 5.x)
• system jumbomtu 9000 is a fixed global setting
• For jumbo frame forwarding, MTU config needed on ‘eth’ port-profiles OR
interfaces
N1kv-VSM-A(config)# port-profile type eth Uplink_PP Nexus1000v-VSM-A(config)# int e3/1
N1kv-VSM-A(config-port-prof)# mtu 9000 Nexus1000v-VSM-A(config-if)# mtu 9000

• Confirm on ESXi host:

[root@UCSC240:~] esxcfg-nics -l
Name PCI Driver Link Speed Duplex MAC Address MTU Description
vmnic0 0000:02:00.0 igb Up 1000Mbps Full a4:93:4c:aa:fb:6e 9000 Intel Corporation I350 Gigabit Network Connection
vmnic1 0000:02:00.1 igb Up 1000Mbps Full a4:93:4c:aa:fb:6f 9000 Intel Corporation I350 Gigabit Network Connection

BRKDCN-1100 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 73
Nexus 1000v
Microsoft Hyper-V
----------------------------------------------
• No management over MTU settings LTL: 17
----------------------------------------------
within the virtual modules Port ID: 4
NIC Index: 2
NIC Instance ID: {7C3A395E-ADDF-41B8-BE39-
• Physical adapters added to 1000v D05BC73C4699}
MAC address: a0:36:9f:6a:c8:92
should have desired MTU Port profile: UPLINK-PP
Port profile GUID: 6AF1BB76-F87A-4EAB-9480-
• Modify via the MS adapter properties, 8420AFB6CD02
Network GUID:
UCSM, or ILO where applicable VM/NIC name:
Adapter
Intel(R) Ethernet 10G 2P X520

• Value should not be changed after NIC is VM UUID:

NIC UUID:
added to the switch MTU:
Link state:
9000
UP
Duplex: Full
• MTU value not reflected on VSM/VEM Tx speed:
Rx speed:
1410065408
1410065408
interfaces; verify on host with Autoneg: Enabled
Link Params pending: No
.\vemcmd.exe show attach info Speed Capability 0x10
Duplex Capability 0x4

BRKDCN-1100 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 74
UCS Manager
Fabric Interconnect Processing

Host
Host

Trusted CoS Traffic

Ingress Traffic
L2/L3/L4 Details Classification

Egress Traffic
Marking CoS
Ingress
Policing
Egress Strict Priority or
Violation =
MTU Check Scheduler DWRR Scheduler
truncate or drop

Threshold reached =
-Tail drop for drop classes Per-class
Egress
-Pause for lossless buffer monitor
Queues
8 multicast Unicast VOQ
queues (8 per egress interface)

BRKDCN-1100 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 75
UCS Manager
QoS System Classes

Manually enter desired MTU

for each class
Match criteria ‘normal’: 1500 bytes

Auto-created / hard-set with

vHBA creation;
‘fc’: 2158 bytes

BRKDCN-1100 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 76
UCS Manager
Assigning to vNICs

Specify the MTU on the vNIC;

informs the host network adapter of
MTU value to be used

vNICs can be assigned 9000

specific priority; otherwise

will default to Best Effort
6248-FI-A# scope service-profile server 1/1
6248-FI-A /org/service-profile# show vnic

vNIC:
Name Fabric ID Dynamic MAC Addr Virtualization Preference
-------------- ---------- ----------------- -------------------------
1-host-vNIC-A A 00:25:B5:0A:00:01 NONE
1-host-vNIC-B B 00:25:B5:0B:00:01 NONE
6248-FI-A /org/service-profile# show vnic 1-host-eth-1 detail

vNIC:
<output omitted>
Host Interface Ethernet MTU: 9000

BRKDCN-1100 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 77
End-to-End Testing
• Verification across platforms
• Linux: ping 192.168.1.5 –M do –s 8972 OR tracepath
• ESXi: vmkping –d –s 8972 192.168.1.5
• Windows: mturoute OR

Subtract the IP header &

ICMP header (28 bytes)
from the ping size

• Be aware of default CoPP policy that rate-limits jumbo ICMP packets

destined to the local Nexus switch
BRKDCN-1100 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 78
Summary
In Conclusion

• Not all traffic is created equal

• Know the flow
• Align the stars

BRKDCN-1100 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 80
Complete Your Online Session Evaluation
• Give us your feedback to be
entered into a Daily Survey
Drawing. A daily winner will
receive a $750 Amazon gift card.
• Complete your session surveys
through the Cisco Live mobile
app or from the Session Catalog
on CiscoLive.com/us.

Don’t forget: Cisco Live sessions will be available

for viewing on-demand after the event at
CiscoLive.com/Online

BRKDCN-1100 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 81
Continue Your Education
• Demos in the Cisco campus
• Walk-in Self-Paced Labs
• Lunch & Learn
• Meet the Engineer 1:1 meetings
• Related sessions

Presentation ID © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 82
Thank you