Technical FAQ

10-Gigabit Ethernet FAQ

Frank Pleshe, NetApp July 2010 | Version 1.0 | TR-3863

TABLE OF CONTENTS

1 NETWORK INTERFACE CARDS SUPPORTED ...................................................................... 3 2 CABLING AND OPTICS SUPPORTED .................................................................................... 5 3 GENERAL TECHNICAL ............................................................................................................ 6 4 PROTOCOL OFFLOAD SUPPORT .......................................................................................... 6 5 UNIFIED TARGET ADAPTER (UTA) ........................................................................................ 7

10-Gigabit Ethernet FAQ

1 NETWORK INTERFACE CARDS SUPPORTED

HOW DO I KNOW IN WHICH SLOTS A 10-GIGABIT ETHERNET (10GBE) NETWORK INTERFACE CARD (NIC) CAN BE INSTALLED? Answer: The systems configuration guides located on the NOW site are the most updated reference to in which slots 10GbE NICs can be installed on supported storage controllers: http://now.netapp.com/NOW/knowledge/docs/hardware/NetApp/syscfg/index.shtml. CAN THE 10GBE NICS AUTONEGOTIATE TO GBE SPEED? Answer: Port speed negotiation to GbE speed is not supported. CAN THE 10GBE NICS BE CONNECTED IN A BACK-TO-BACK CONFIGURATION? Answer: It is possible to interconnect two NetApp storage controllers using a 10GbE NIC directly to one another without the use of an Ethernet switch. However, this is not a tested or supported configuration. WHY IS ONLY ONE 10GBE NIC SUPPORTED ON THE FAS2050? Answer: The FAS2050 has less power available to the expansion slots and requires that less heat be generated for system cooling. Therefore, the X1106A-R6 NIC was specifically introduced to provide 10GbE connectivity to the FAS2050. CAN A GBE SMALL FORM-FACTOR PLUGGABLE (SFP) BE USED IN PLACE OF A 10GBE SFP+? Answer: No. The current generation of 10GbE NICs that NetApp provides only supports the use of 10GbE SFP+ transceivers. DO NETAPP 10GBE NICS PROVIDE SUPPORT FOR 10GBASE-T? Answer: The current NetApp 10GbE NICs only support optical connections using 10Gbase-SR and copper connections using SFP+ copper twin-ax, which is also known as direct-attach cable (DAC). The other current limiting factors are: Current generation NICs require 46 watts of power needed to drive 10GBase-T. Copper cable, known as CAT 6a or CAT 7, is not generally available at this time. 10GBase-T is currently more expensive. IS THERE A FIELD-REPLACEABLE UNIT (FRU) FOR THE OPTICAL XFP TRANSCIEVERS IN THE X1005A-R5 OR X1006A-R6 NICS? Answer: No FRU parts are available for these NICs. The entire NIC must be replaced. WHAT 10GBE INTERFACES ARE AVAILABLE FROM NETAPP TODAY? Answer: Currently seven different 10GbE NICs are available from NetApp.

10-Gigabit Ethernet FAQ

NIC

Platform

Data ONTAP Release 7.2.3 7.3.x 8.0.x 7.2.3 7.3.x 8.0.x

Ports

Protocols

Bus

X1005A-R5, single port 10GbE, optical (Chelsio T210E) X1008A-R5, dual port 10GbE, optical (Chelsio T320E)

FAS3050 FAS60xx

One (1) optical short range with LC connector, 300m distance with OM3 cable Two (2) optical short range with LC connector, 300m distance with OM3 cable

IP only

PCI-X

FAS3040 FAS3070 FAS31xx FAS60xx SA300 SA600 V-Series FAS20501

IP only

PCIe Gen1 with 8 lanes

X1106A-R6, single port 10GbE, optical (Chelsio N310E) X1107A-R6, dual port 10GbE, SFP+ optical or copper (Chelsio S320E)

7.3.2

One (1) optical short range with LC connector, 300m distance with OM3 cable Two (2) optical short range with LC connector, 300m distance with OM3 cable

IP only

PCIe Gen1 with 8 lanes PCIe Gen1 with 8 lanes

FAS3040 FAS3070 FAS31xx FAS60xx SA300 SA600 V-Series FAS3040 FAS3070 FAS31xx FAS60xx V-Series FAS3040 FAS3070 FAS31xx FAS60xx V-Series

7.3.2 8.0.1

IP only

X1139A-R6, dual port 10GbE, unified target, SFP+ optical (Qlogic 8142)

7.3.2 8.0.x

Two (2) optical short range LC, 300m distance with OM3 cable

FCoE2 IP3

PCIe Gen1 with 8 lanes, PCIe Gen2 4 lanes PCIe Gen1 with 8 lanes, PCIe Gen2 4 lanes

X1140A-R6, dual port 10GbE, unified target, CU (Qlogic 8152)4

7.3.2 8.0.x

Two (2) copper twin-ax SFP+ ports supporting 1,3 or 5 meter cables

FCoE3 IP4

Note: The use of x for the Data ONTAP release indicates all minor releases of software.

1 2

This NIC is only supported on the FAS2050. In Data ONTAP 7.3.2, only FCoE is supported on this UTA. 3 In Data ONTAP 8.0.1, both FCoE and IP are supported for this UTA. 4 Only Cisco or Brocade (depending on switch connected) SFP+ copper twin-ax cables can be used with this UTA. Other third-party cables do not meet the signal-to-noise ratio and electrical properties required to operate.

10-Gigabit Ethernet FAQ

2 CABLING AND OPTICS SUPPORTED

CAN I USE ANY TYPE OF MULTIMODE FIBRE CABLE TO CONNECT MY 10GBE NICS? Answer: NetApp only supports the use of OM3 (ISO/IEC 11801) certified cable between a NetApp storage controller and a 10GbE Ethernet switch. OM3 cable is specifically designed for the high bandwidth required for 10GbE and can be quickly identified by its aqua color. The maximum distance supported with this cable is 300 meters when the storage controller NIC is directly connected to a 10GbE Ethernet switch. Note: A new generation of multimode cable specified by EIA/TIA 492AAAD, identified as OM4, has become available. The maximum effective distance for OM4 certified cable is 550 meters. NetApp currently does not sell OM4 cables, but it is acceptable to use these type cables for 10GbE connections. Refer to NetApp TR3552 for more detailed information on optical cabling: http://media.netapp.com/documents/tr-3552.pdf. WHAT CABLING OPTIONS ARE AVAILABLE FOR NETAPP 10GBE NICS? Answer: Two cable types are supported by NetApps current 10GbE NIC/UTA cards. They are short-range multimode fiber with LC connectors and SFP+ copper twin-ax. The NetApp price list provides certified cables that are supported. Because 10GbE is very sensitive to power, signal-to-noise ratio, and other factors, using cables from other sources can be problematic. Only approved cables should be used.
SFP+ Copper Twin-Ax X-SFP-H10GB-CU1M-R6 Cisco N50XX 10GBase Copper SFP+cable,1m X-SFP-H10GB-CU3M-R6 Cisco N50XX 10GBase Copper SFP+cable,3m X-SFP-H10GB-CU3M-R6 Cisco N50XX 10GBase Copper SFP+cable,5m X-TWX-0101-R6 Brocade, Cable, Twin-ax, 10GbE, SFP+, 1m,R6 X-TWX-0301-R6 Brocade, Cable, Twin-ax, 10GbE, SFP+, 3m,R6 X-TWX-0501-R6 Brocade, Cable, Twin-ax, 10GbE, SFP+, 5m,R6 OM3 Optical Cable X6553-R6 Cable,OPT,50u,2GHz/KM,MM,LC/LC,2M X6536-R6 Cable,Optical,50u,2000MHz/Km/MM,LC/LC,5M X6537-R6 Cable,Optical,50u,2000MHz/Km/MM,LC/LC,30M X6547-R6 Cable,Optical,50u,2000MHz/Km/MM,LC/SC,5M X6546-R6 Cable,Optical,50u,2000MHz/Km/MM,LC/SC,30M

Note: NetApp resells both Cisco and Brocade switches. You should always purchase cables available from the switch vendor being used. In other words, if you purchase a Cisco switch, then you should purchase the Cisco twin-ax cable for that NIC/UTA connection.

CAN THE SFP+ X6563-R OPTICAL TRANSCIEVER BE USED IN THE X1140A-R6 UTA? Answer: The X1140A-R6 can only be used with approved NetApp SFP+ twin-ax cables. ARE THE SHORT REACH (SR) OPTICS IN AN XFP MODULE INTEROPERABLE WITH THE SR OPTICS IN AN SFP+ MODULE? Answer: The XFP SR optics in an X1005A-R5 and X1008A-R6 are interoperable with SFP+ SR optics used in 10GbE Ethernet switches such as the Cisco Nexus 5010. DO NETAPP 10GBE NICS SUPPORT LONG REACH (LR) OPTICS? Answer: Currently NetApp 10GbE NICs only support SR fiber optic connections, with a maximum distance of 300 meters using OM3 cable and 550 meters using OM4 cable. CAN OTHER VENDORS' SFP+ TRANSCEIVERS BE USED? Answer: Each NIC has its own certified SFP+ optical transceivers. For example, the SFP+ transceiver used in the FAS62xx and FAS32xx cannot be installed in an X1107A-R6 NIC.

10-Gigabit Ethernet FAQ

CAN OTHER VENDORS' SFP+ TWIN-AX CABLES BE USED? Answer: Ethernet switch vendors have their own set of tolerances for twin-ax transceivers and cables. You should consult the switch vendors documentation to make sure the twin-ax cable you select is approved for use with that vendor's product.

3 GENERAL TECHNICAL
CAN A 10GBE NIC BE USED IN A VIRTUAL INTERFACE (VIF)? Answer: All NetApp 10GbE NICs can be used in a VIF. HOW MANY 10GBE PORTS CAN BE USED IN A VIF? Answer: The maximum number of Ethernet interfaces Data ONTAP supports in a VIF is 16. Note: The IEEE 802.3ad link aggregation specification only provides for a maximum of 8 interfaces. To utilize all 16 interfaces, you would need two VIFs with eight members in each VIF. WHY DO I SEE A MESSAGE THAT TOE HAS BEEN DISABLED WHEN I CREATE A VIF USING A 10GBE NIC? Answer: In Data ONTAP versions prior to 7.3.2, TCP offload engine (TOE) capabilities are disabled when a VIF is configured using a 10GbE interface that has TOE capabilities. This limitation is imposed because the Data ONTAP kernel must control the sessions in a VIF configuration and therefore disables the TOE functionality of the NIC. Data ONTAP 7.3.2 disables TOE capabilities by default, and in Data ONTAP version 8, TOE functionality has been completely removed. HOW MANY VLANS ARE SUPPORTED ON A 10GBE NIC? Answer: All NetApp NICs support 1 to 4,094 VLAN tags. NetApp does not impose a VLAN limitation based on the Ethernet NIC. Data ONTAP has a limitation based on system memory and processor speeds. Refer to the network management guide for your version of Data ONTAP to determine the current limitations: http://now.netapp.com/NOW/knowledge/docs/ontap/ontap_index.shtml. DO THE CURRENT DUAL PORT 10GBE NICS SUPPORT FULL LINE RATE? Answer: The X1008A-R6 and X1107A-R6 10GbE NICs have a PCIe Gen1 bus interface with 8 lanes operating at 2.5Ghz. Aggregate throughput for PCIe Gen1 with 8 lanes is 16Gb/s, so therefore these cards cannot achieve full line rate on both ports. The X1139A-R6 and X1140A-R6 10GbE UTA cards have a PCIe Gen2 bus interface with 4 lanes operating at 5.0Ghz. Doubling the operating frequency and reducing the lanes on these cards yield the same throughput of 16Gb/s, which again provides less than line rate due to the PCIe bus limitation. Note that there is associated PCIe signaling overhead with each of the cards, which reduces the total shared PCIe bandwidth to approximately 15Gb/s. Sharing of the PCIe bus is generally equal between the two ports but does not limit one port from operating at a line rate of 10Gb/s while the other operates with the remaining PCIe bandwidth.

4 PROTOCOL OFFLOAD SUPPORT

WHAT IS TOE? Answer: The term TOE refers to a TCP offload engine. NICs can provide TOE capabilities to offload many of the processor-intensive tasks such as: Connection establishment Acknowledgment of packets Sliding window calculations for acknowledgments Calculating checksum and sequence numbers Connection termination

10-Gigabit Ethernet FAQ

WHY WAS TOE DISABLED IN DATA ONTAP VERSION 7.3.2? Answer: TOE was disabled for several reasons. The most important factor was customer demand for VIF support on 10GbE. Overall performance was only slightly increased when using TOE with the current generation of NICs. Newer technologies such as TCP segmentation offload (TSO), receive side scaling (RSS), and receive side coalescing (RSC) are replacing TOE while providing similar performance benefits. WHAT IS TCP SEGMENTATION OFFLOAD (TSO)? Answer: TCP segmentation offload (TSO), also known as large segment offload (LSO), is a feature that allows the NIC to offload the storage controller CPU burden of segmenting large amounts of data for transmission to hosts. Enabling TSO on a NIC can increase overall system performance. TSO will be available for 10GbE NICs in Data ONTAP version 8.0.1. WHAT IS RECEIVE SIDE SCALING (RSS)? Answer: Receive side scaling (RSS) is a feature that allows data streams received for a specific connection to be shared across multiple processors or processor cores. Without this feature only a single processor is responsible for handling the data stream and can cause poor use of system cache memory. RSS will be available for 10GbE NICs in Data ONTAP version 8.0.1. WHAT IS RECEIVE SIDE COALESCING (RSC)? Answer: Receive side coalescing (RSC), also known as large receive offload (LRO), is a feature that allows the NIC to group or coalesce group packets from the same connection into a larger packet before sending to the processor. Enabling this feature reduces the overall processor utilization by processing fewer packets. RSC will be available for 10GbE NICs in a future release of Data ONTAP.

5 UNIFIED TARGET ADAPTER (UTA)

WHAT IS DATA CENTER BRIDGING (DCB)? Answer: A new standard that provides lossless capabilities to the existing Ethernet protocol. DCB is also called data center Ethernet (DCE) or converged enhanced Ethernet (CEE). DCB includes several standards in various stages of ratification: Priority flow control (PFC): allows PAUSE frames per priority level Enhanced transmission selection (ETS): provides a management framework for assignment of bandwidth based on 802.1p CoS traffic classes Congestion notification (QCN): common protocol for end-to-end congestion management

WHAT DOES THE TERM UNIFIED TARGET ADAPTER (UTA) MEAN? Answer: Unified target adapter is a target adapter card on the NetApp storage controllers that can process converged traffic using different protocols over the same wire, such as FCoE, NFS, CIFS, and iSCSI. WHAT IS THE DIFFERENCE BETWEEN A CONVERGED NETWORK ADAPTER (CNA) AND A UNIFIED TARGET ADAPTER (UTA)? Answer: A CNA is an initiator card that can be installed on host (server) equipment and supports different protocols (FCoE, CIFS, NFS, iSCSI) over a 10GbE link. The capabilities of the CNA and UTA are very similar, with the cleanest distinction being that the CNA is installed on the initiator/host side, while the UTA is installed on the target (NetApp) storage controller. WHAT FIBRE CHANNEL SPEEDS ARE SUPPORTED ON THE UTA? Answer: The UTA supports FCoE up to 10Gb/s. It must be connected to a 10Gb FCoE capable switch or a DCB switch, which is then connected to a switch providing Fibre Channel services. From these any speed FC device supported by the switch can be attached, such an 8Gb/s FC initiator card. You can also combine multiple FC initiator cards (HBAs) to drive up to (near) 10Gbps Fibre Channel traffic over the link to the UTA. There is a slight tax for Ethernet overhead, but the effective Fibre Channel rate is between 9Gbps and 10Gbps.

10-Gigabit Ethernet FAQ

HOW DO I CONFIGURE BANDWIDTH ON THE UTA? Answer: Bandwidth can be allocated for different traffic classes, such as FCoE, using enhanced transmission selection. ETS is a part of the DCB protocol, and its settings are configured on the DCB switch to which the UTA is connected. The UTA will accept the settings provided by the switch. FCoE uses traffic class 3. CAN I CONNECT A HOST CNA ADAPTER DIRECTLY TO A NETAPP UTA? Answer: No, direct connect is not supported today. The CNA has to be connected to a DCB/FCoE switch and then to the NetApp UTA. WHAT IS FCOE MULTIHOP, AND IS IT SUPPORTED? Answer: FCoE multihop is a network configuration where an FCoE frame traverses multiple (more than one) FCoE switches from the initiator to the target UTA. For example, a server can connect from a CNA to an FCoE top-of-rack (TOR) switch to another fixed port or director class switch prior to terminating at the UTA on the storage controller. It is expected that the switch vendors (Cisco and Brocade) will have multihop support implemented by end of 2010. The NetApp UTA requires no modification for multihop support. MAY I USE THE 10GBE UNIFIED TARGET ADAPTER FOR IP-BASED TRAFFIC ONLY? Answer: Yes. Please contact product management (Krister Eriksson) for an update about deployment considerations.

NetApp provides no representations or warranties regarding the accuracy, reliability or serviceability of any information or recommendations provided in this publication, or with respect to any results that may be obtained by the use of the information or observance of any recommendations provided herein. The information in this document is distributed AS IS, and the use of this information or the implementation of any recommendations or techniques herein is a customers responsibility and depends on the customers ability to evaluate and integrate them into the customers operational environment. This document and the information contained herein may be used solely in connection with the NetApp products discussed in this document.

Copyright 2010 NetApp, Inc. All rights reserved. No portions of this document may be reproduced without prior written consent of NetApp, Inc. Specifications are subject to change without notice. NetApp, the NetApp logo, Go further, faster, Data ONTAP, and 10-Gigabit NOW are FAQ Ethernet trademarks or registered trademarks of NetApp, Inc. in the United States and/or other countries. All other brands or products are trademarks or registered trademarks of their respective holders and should be treated as such. TR-3863