IBM Blade Center Virtual Fabric

Front cover
IBM BladeCenter Virtual Fabric Solutions

Covering IBM Virtual Fabric Mode and Switch Independent Mode Concepts and benefits of virtual NICs (vNICs) Step-by-step instructions for creating various vNIC solutions
David Watts Jure Arzensek Valeria Lima Norm Strole
ibm.com/redbooks
International Technical Support Organization IBM BladeCenter Virtual Fabric Solutions August 2011
SG24-7966-00
Note: Before using this information and the product it supports, read the information in Notices on page vii.
First Edition (August 2011) This edition applies to the following IBM BladeCenter I/O options: Emulex 10GbE Virtual Fabric Adapter Emulex 10GbE Virtual Fabric Adapter II Broadcom 2-port 10Gb Virtual Fabric Adapter BNT Virtual Fabric 10Gb Switch Module Cisco Nexus 4001I Switch Module
Copyright International Business Machines Corporation 2011. All rights reserved. Note to U.S. Government Users Restricted Rights -- Use, duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp.
Contents
Notices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . vii Trademarks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . viii Preface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ix The team who wrote this book . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ix Now you can become a published author, too! . . . . . . . . . . . . . . . . . . . . . . . . . xi Comments welcome. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xii Stay connected to IBM Redbooks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xii Chapter 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 1.1 Virtual NICs: overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4 1.2 Comparing modes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 1.3 Selecting the mode . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 1.4 Use of BladeCenter Open Fabric Manager . . . . . . . . . . . . . . . . . . . . . . . . . 7 1.5 The future of converged fabrics and FCoE capabilities. . . . . . . . . . . . . . . . 8 1.6 Using this book . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8 Part 1. IBM Virtual Fabric Mode . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9 Chapter 2. Solution architecture - IBM Virtual Fabric Mode . . . . . . . . . . . 11 2.1 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12 2.2 Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14 2.3 Planning considerations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17 Chapter 3. Supported products - IBM Virtual Fabric Mode . . . . . . . . . . . . 19 3.1 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20 3.2 Emulex 10Gb Virtual Fabric Adapters . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21 3.2.1 Emulex 10GbE Virtual Fabric Adapter . . . . . . . . . . . . . . . . . . . . . . . 21 3.2.2 Emulex 10GbE Virtual Fabric Adapter II . . . . . . . . . . . . . . . . . . . . . . 25 3.3 BNT Virtual Fabric 10Gb Switch Module . . . . . . . . . . . . . . . . . . . . . . . . . . 27 Chapter 4. Configuring the solution - IBM Virtual Fabric Mode . . . . . . . . 37 4.1 Configuring the Emulex 10GbE Virtual Fabric Adapter . . . . . . . . . . . . . . . 38 4.1.1 Upgrading firmware . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38 4.1.2 Configuring the adapter in the UEFI interface . . . . . . . . . . . . . . . . . . 39 4.2 Configuring the BNT Virtual Fabric 10Gb Switch Module . . . . . . . . . . . . . 44 4.2.1 BladeOS CLI . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45 4.2.2 isCLI . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51 4.2.3 Browser-based interface (BBI) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52
Copyright IBM Corp. 2011. All rights reserved.
iii
4.3 Creating virtual NICs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56 4.3.1 Before you configure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56 4.3.2 Enabling and disabling vNIC functionality on the server . . . . . . . . . . 56 4.3.3 Creating the vNICs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60 4.4 vNIC group configuration. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63 4.5 Configuring vNIC VLANs. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66 4.6 vNIC and VMready . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 69 4.7 High availability and vNICs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70 4.8 Configuring the operating system . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72 4.8.1 Configuring teams and VLANs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73 4.8.2 Configuring a new team . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74 4.8.3 Configuring a VLAN for a physical or team adapter . . . . . . . . . . . . . 76 Chapter 5. Examples of use - IBM Virtual Fabric Mode . . . . . . . . . . . . . . . 81 5.1 Example 1: vNICs only, no uplinks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82 5.2 Example 2: vNICs only, one uplink . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 86 5.3 Example 3: vNICs only, one uplink trunk. . . . . . . . . . . . . . . . . . . . . . . . . . 91 Part 2. Switch Independent Mode . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 99 Chapter 6. Solution architecture - Switch Independent Mode . . . . . . . . 101 6.1 Solution overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 103 6.2 Features . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 105 Chapter 7. Supported products - Switch Independent Mode . . . . . . . . . 109 7.1 Virtual Fabric adapters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 110 7.1.1 Emulex 10GbE Virtual Fabric Adapter II . . . . . . . . . . . . . . . . . . . . . 110 7.1.2 Broadcom 2-port 10-Gb Virtual Fabric Adapter. . . . . . . . . . . . . . . . 114 7.2 Virtual Fabric switch modules . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 116 7.2.1 BNT Virtual Fabric 10Gb Switch Module. . . . . . . . . . . . . . . . . . . . . 116 7.2.2 Cisco Nexus 4001I Switch Module . . . . . . . . . . . . . . . . . . . . . . . . . 124 Chapter 8. Configuring the solution - Switch Independent Mode. . . . . . 133 8.1 Configuring Virtual Fabric adapters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 135 8.1.1 Using BOFM Advanced V4.1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 135 8.1.2 Firmware upgrade of Emulex 10GbE Virtual Fabric Adapter II . . . . 135 8.1.3 Configuring the Emulex 10GbE Virtual Fabric Adapter II in UEFI . . 136 8.1.4 Configuring the Broadcom 2-port 10-Gb Virtual Fabric Adapter in UEFI . . . . . . . . . 141 8.2 Configuring the BNT Virtual Fabric 10Gb Switch Module . . . . . . . . . . . . 146 8.2.1 BladeOS CLI . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 147 8.2.2 isCLI . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 152 8.2.3 Browser-based interface (BBI) . . . . . . . . . . . . . . . . . . . . . . . . . . . . 153 8.3 Cisco Nexus 4001I Switch Module configuration . . . . . . . . . . . . . . . . . . 156
iv
8.3.1 Firmware upgrade . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 157 8.4 Creating virtual NICs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 160 8.4.1 Broadcom 2-port 10-Gb Virtual Fabric Adapter. . . . . . . . . . . . . . . . 161 8.4.2 Emulex 10GbE Virtual Fabric Adapter II . . . . . . . . . . . . . . . . . . . . . 164 8.5 Configuring vNIC VLANs. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 167 8.6 vNIC and VMready . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 169 8.7 High availability and vNICs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 169 8.8 Configuring the operating systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 169 8.8.1 vNICs on the Emulex 10GbE Virtual Fabric Adapter II . . . . . . . . . . 170 8.8.2 vNICs on the Broadcom 2-port 10-Gb Virtual Fabric Adapter . . . . . 172 8.8.3 Configuring teams with Emulex OneCommand Manager . . . . . . . . 179 8.8.4 Configuring teams with Broadcom BACS . . . . . . . . . . . . . . . . . . . . 184 Chapter 9. Examples of use - Switch Independent Mode . . . . . . . . . . . . 193 9.1 Configurations overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 194 9.1.1 Virtual LAN (VLAN) definitions . . . . . . . . . . . . . . . . . . . . . . . . . . . . 195 9.1.2 Switch/adapter vNIC correlation . . . . . . . . . . . . . . . . . . . . . . . . . . . 196 9.2 Cisco Nexus 4001I Switch Module configurations . . . . . . . . . . . . . . . . . 197 9.2.1 vNICs summary. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 197 9.2.2 Emulex adapter configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 198 9.2.3 Broadcom adapter configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . 208 9.2.4 Cisco Nexus 4001I Switch Module configuration . . . . . . . . . . . . . . 224 9.3 BNT Virtual Fabric 10-Gb Switch Module configurations . . . . . . . . . . . . 226 9.3.1 Emulex adapter configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 226 9.3.2 Broadcom adapter configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . 237 9.3.3 Configuring the BNT Virtual Fabric 10-Gb Switch Module . . . . . . . 252 Abbreviations and acronyms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 255 Related publications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 257 IBM Redbooks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 257 Other publications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 257 Online resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 258 Help from IBM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 258 Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 259
Contents
vi
Notices
This information was developed for products and services offered in the U.S.A. IBM may not offer the products, services, or features discussed in this document in other countries. Consult your local IBM representative for information on the products and services currently available in your area. Any reference to an IBM product, program, or service is not intended to state or imply that only that IBM product, program, or service may be used. Any functionally equivalent product, program, or service that does not infringe any IBM intellectual property right may be used instead. However, it is the user's responsibility to evaluate and verify the operation of any non-IBM product, program, or service. IBM may have patents or pending patent applications covering subject matter described in this document. The furnishing of this document does not give you any license to these patents. You can send license inquiries, in writing, to: IBM Director of Licensing, IBM Corporation, North Castle Drive, Armonk, NY 10504-1785 U.S.A. The following paragraph does not apply to the United Kingdom or any other country where such provisions are inconsistent with local law: INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS PUBLICATION "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESS OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF NON-INFRINGEMENT, MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. Some states do not allow disclaimer of express or implied warranties in certain transactions, therefore, this statement may not apply to you. This information could include technical inaccuracies or typographical errors. Changes are periodically made to the information herein; these changes will be incorporated in new editions of the publication. IBM may make improvements and/or changes in the product(s) and/or the program(s) described in this publication at any time without notice. Any references in this information to non-IBM websites are provided for convenience only and do not in any manner serve as an endorsement of those websites. The materials at those websites are not part of the materials for this IBM product and use of those websites is at your own risk. IBM may use or distribute any of the information you supply in any way it believes appropriate without incurring any obligation to you. Information concerning non-IBM products was obtained from the suppliers of those products, their published announcements or other publicly available sources. IBM has not tested those products and cannot confirm the accuracy of performance, compatibility or any other claims related to non-IBM products. Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products. This information contains examples of data and reports used in daily business operations. To illustrate them as completely as possible, the examples include the names of individuals, companies, brands, and products. All of these names are fictitious and any similarity to the names and addresses used by an actual business enterprise is entirely coincidental. COPYRIGHT LICENSE: This information contains sample application programs in source language, which illustrate programming techniques on various operating platforms. You may copy, modify, and distribute these sample programs in any form without payment to IBM, for the purposes of developing, using, marketing or distributing application programs conforming to the application programming interface for the operating platform for which the sample programs are written. These examples have not been thoroughly tested under all conditions. IBM, therefore, cannot guarantee or imply reliability, serviceability, or function of these programs.
vii
Trademarks
IBM, the IBM logo, and ibm.com are trademarks or registered trademarks of International Business Machines Corporation in the United States, other countries, or both. These and other IBM trademarked terms are marked on their first occurrence in this information with the appropriate symbol ( or ), indicating US registered or common law trademarks owned by IBM at the time this information was published. Such trademarks may also be registered or common law trademarks in other countries. A current list of IBM trademarks is available on the Web at http://www.ibm.com/legal/copytrade.shtml The following terms are trademarks of the International Business Machines Corporation in the United States, other countries, or both: AIX BladeCenter DS4000 IBM Redbooks Redbooks (logo) System Storage System x
The following terms are trademarks of other companies: BNT, VMready, and Server Mobility are trademarks or registered trademarks of Blade Network Technologies, Inc., an IBM Company. Intel, Intel logo, Intel Inside logo, and Intel Centrino logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States and other countries. Microsoft, Windows, and the Windows logo are trademarks of Microsoft Corporation in the United States, other countries, or both. Intel, Intel logo, Intel Inside, Intel Inside logo, Intel Centrino, Intel Centrino logo, Celeron, Intel Xeon, Intel SpeedStep, Itanium, and Pentium are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States and other countries. Linux is a trademark of Linus Torvalds in the United States, other countries, or both. Other company, product, or service names may be trademarks or service marks of others.
viii
Preface
The deployment of server virtualization technologies in data centers requires significant efforts in providing sufficient network I/O bandwidth to satisfy the demand of virtualized applications and services. For example, every virtualized system can host several dozen network applications and services, and each of these services requires certain bandwidth (or speed) to function properly. Furthermore, because of different network traffic patterns relevant to different service types, these traffic flows may interfere with each other, leading to serious network problems including the inability of the service to perform its functions. The IBM Virtual Fabric solution for IBM BladeCenter addresses these issues. The solution is based on the IBM BladeCenter H chassis with a 10-Gb Converged Enhanced Ethernet infrastructure built on 10-Gb Ethernet switch modules in the chassis and the Emulex or Broadcom Virtual Fabric Adapters in each blade server. This book is for clients who want to learn how to implement an IBM Virtual Fabric solution, either the IBM Virtual Fabric Mode offering or the Switch Independent Mode offering. It provides step-by-step instructions for configuring the adapters and switches.
The team who wrote this book

This book was produced by a team of specialists from around the world working at the International Technical Support Organization, Raleigh Center. David Watts is a Consulting IT Specialist at the IBM ITSO Center in Raleigh. He manages residencies and produces IBM Redbooks publications on hardware and software topics related to IBM System x and BladeCenter servers and associated client platforms. He has authored over 80 books, papers, and technotes. He holds a Bachelor of Engineering degree from the University of Queensland (Australia) and has worked for IBM both in the US and Australia since 1989. David is an IBM Certified IT Specialist and a member of the IT Specialist Certification Review Board. Jure Arzensek is an Advisory IT Specialist at IBM Slovenia and works for the EMEA level 2 BladeCenter support team. He has worked with IBM BladeCenter products since 2002 and has been with IBM since 1995. Jure holds a degree in Computer Science from the University of Ljubljana. His other areas of expertise
ix
include IBM System x servers, SAN, System Storage DS3000 and DS4000 products and network operating systems for the Intel platform. He has co-authored eight other IBM Redbooks publications. Valeria Lima was a member of STG Lab Services in IBM Brazil at the time of writing. Norm Strole is a Senior Technical Staff Member in Research Triangle Park, North Carolina. He is an IBM Master Inventor with 30 years of experience in the networking field. He holds a Ph.D. degree in Electrical Engineering from Duke University. His areas of expertise include Blade networking subsystems and networking protocols. He has written extensively on the IBM Token Ring, with publication in both the IBM Journal of Research and Development and the IBM Systems Journal.
The team (l-r): David, Jure, Valeria, and Norm
Thanks to the following people for their contributions to this project: From IBM Marketing: Richard Mancini Shekhar Mishra Michael Talplacido From IBM Development: Manzar Alam Ralph Geiger Marc Stracuzza Tony Tran
From Broadcom: Kevin Liebl Maziar Tamadon From Emulex: Tom Boucher Leanne Greene From IBM Redbooks: Karen Lawrence Linda Robinson Alfred Schwab This book is based in part on the document IBM BladeCenter Virtual Fabric Solutions, REDP-4673. Thanks to its authors: Tom Boucher Scott Lorditch Mark McCool David Watts
Now you can become a published author, too!

Here's an opportunity to spotlight your skills, grow your career, and become a published authorall at the same time! Join an ITSO residency project and help write a book in your area of expertise, while honing your experience using leading-edge technologies. Your efforts will help to increase product acceptance and customer satisfaction, as you expand your network of technical contacts and relationships. Residencies run from two to six weeks in length, and you can participate either in person or as a remote resident working from your home base. Find out more about the residency program, browse the residency index, and apply online at: ibm.com/redbooks/residencies.html
Preface
xi
Comments welcome
Your comments are important to us! We want our books to be as helpful as possible. Send us your comments about this book or other IBM Redbooks publications in one of the following ways: Use the online Contact us review Redbooks form found at: ibm.com/redbooks Send your comments in an email to: redbooks@us.ibm.com Mail your comments to: IBM Corporation, International Technical Support Organization Dept. HYTD Mail Station P099 2455 South Road Poughkeepsie, NY 12601-5400
Stay connected to IBM Redbooks

Find us on Facebook: http://www.facebook.com/IBMRedbooks Follow us on Twitter: http://twitter.com/ibmredbooks Look for us on LinkedIn: http://www.linkedin.com/groups?home=&gid=2130806 Explore new Redbooks publications, residencies, and workshops with the IBM Redbooks weekly newsletter: https://www.redbooks.ibm.com/Redbooks.nsf/subscribe?OpenForm Stay current on recent Redbooks publications with RSS Feeds: http://www.redbooks.ibm.com/rss.html
xii
Chapter 1.
Introduction
The deployment of server virtualization technologies in data centers requires significant efforts to provide sufficient network I/O bandwidth (or speed) to satisfy the demand of virtualized applications and services. For example, every virtualized system can host several dozen network applications and services, and each of these services requires a certain bandwidth to function properly. Furthermore, because of different network traffic patterns relevant to different service types, these traffic flows might interfere with each other, leading to serious network problems, including the inability of the service to perform its functions. Providing sufficient bandwidth and isolation to virtualized applications in a 1-Gb network infrastructure might be particularly challenging for blade-based deployments in which the number of physical I/O ports per blade is limited. For example, of the maximum of eight physical ports per blade, at least two ports are usually used to connect to the external storage. Therefore, only six 1-Gb Ethernet ports are available, with a total network bandwidth of 6 Gb per blade for Gigabit Ethernet infrastructures. This leaves no room for future growth. In addition, traffic flows are isolated on a physical port basis, and the bandwidth per interface is static with a maximum bandwidth of 1 Gb per flow, thus limiting flexibility of bandwidth usage. The IBM BladeCenter Virtual Fabric solutions address these issues by increasing the number of available Ethernet ports and providing more flexibility in allocating the available bandwidth to meet specific application requirements.
The BladeCenter Virtual Fabric solutions implement virtual network interface controllers (vNICs) and are based on the IBM BladeCenter H chassis with a 10-Gb Converged Enhanced Ethernet (CEE) switch infrastructure and 10-Gb Virtual Fabric Adapters (VFAs) installed in each blade server. The basic concept of the BladeCenter Virtual Fabric solution is to take the two 10-Gb ports that are on a 10-Gb VFA and split them into eight vNICs. This configuration allows each vNIC or virtual channel to be between 100 MB and 10 Gb in increments of 100 MB, such that the total of all eight vNICs does not exceed the 20-Gb total that the two 10-Gb ports offer. The separation of the virtual channels is continued into the switch module utilizing VLAN tagging to identify the various data streams. Two operational modes are available, dependent upon the type of 10-Gb expansion card installed in the server and the type of 10-Gb switch installed in the BladeCenter chassis: IBM Virtual Fabric Mode IBM Virtual Fabric Mode can be enabled on the combinations listed in Table 1-1.
Table 1-1 Supported combinations using IBM Virtual Fabric Mode Supported switches Supported adapters Emulex 10GbE Virtual Fabric Adapter, 49Y4235 Emulex 10GbE Virtual Fabric Adapter Advanced, 49Y4275 Emulex 10GbE Virtual Fabric Adapter II, 90Y3550 Emulex 10GbE Virtual Fabric Adapter II Advanced, 90Y3566 Broadcom 2-port 10Gb Virtual Fabric Adapter, 81Y3133 BNT Virtual Fabric 10Gb Switch Module Supported Supported Supported Supported Not supported Cisco Nexus 4001I Switch Module Not supported Not supported Not supported Not supported Not supported
In this configuration, each blade server has up to 20 Gb of available network bandwidth (two 10-Gb ports) and the ability to split this bandwidth among up to eight vNICs, starting at either 100 MB or 1 Gb (adapter-dependent), up to a maximum of 10 Gb per one vNIC channel with 100-MB increments. Channel bandwidth is configured in the BNT Virtual Fabric switch (VFS) for each port and passed to each of the VFAs. Dynamic bandwidth support allows the channel bandwidth to be increased or decreased under the control of the VFS without resetting the VFA. Also, this mode supports bidirectional bandwidth control, with both the server outbound and inbound virtual channel bandwidth limits being enforced by the VFA and VFS, respectively. NIC
Teaming at the VFA allows an individual vNIC channel to failover without impacting other vNIC channels on the same physical link. Switch Independent Mode Switch Independent Mode is, as the name suggests, more independent of the switch module used, and can be enabled on the combinations listed in Table 1-2. Note: Support for Switch Independent Mode by the Emulex 10GbE Virtual Fabric Adapter II will require a firmware upgrade, which is expected in 4Q/2011.
Table 1-2 Supported combinations using Switch Independent Mode Supported switches Supported adapters Emulex 10GbE Virtual Fabric Adapter, 49Y4235 Emulex 10GbE Virtual Fabric Adapter Advanced, 49Y4275 Emulex 10GbE Virtual Fabric Adapter II, 90Y3550 Emulex 10GbE Virtual Fabric Adapter II Advanced, 90Y3566 Broadcom 2-port 10Gb Virtual Fabric Adapter, 81Y3133 BNT Virtual Fabric 10Gb Switch Module Not supported Not supported Supported 4Q/2011a Supported 4Q/2011a Supported Cisco Nexus 4001I Switch Module Not supported Not supported Supported 4Q/2011a Supported 4Q/2011a Supported
a. Support for Switch Independent Mode will require a firmware upgrade, which is expected in 4Q/2011.
Switch Independent Mode extends the IEEE 802.1q VLAN domains in the network infrastructure to specific virtual channels at each server. Up to 8 channels can be configured for each VFA, with outbound bandwidths per channel from 100 Mb to 10 Gb in 100 Mb increments. The channel bandwidth can be configured using the adapter utility (future versions of BladeCenter Open Fabric Manager are also planned to offer this capability). Inbound channel bandwidth constraints (for example, from the switch to the VFA) are not required. Note that combining the Emulex 10GbE Virtual Fabric Adapter II and the BNT Virtual Fabric 10Gb Switch Module supports both modes. The actual mode used is specified in the Unified Extensible Firmware Interface (UEFI) menu of the adapter. The Switch Independent Mode support in the Emulex 10GbE Virtual
Chapter 1. Introduction
Fabric Adapter II will be made available in a future firmware update in the fourth quarter of 2011. With either mode, the integrated Gigabit Ethernet ports on the blade servers are still available, connecting to 1-Gb Ethernet switches installed in bays 1 and 2 of the chassis. Plus, with the addition of a CIOv expansion card in the blade servers and I/O modules in bays 3 and 4 of the BladeCenter H chassis, additional network ports or storage ports can be made available to applications to provide additional expansion capabilities. The combination of the integrated Gigabit Ethernet ports, the CIOv expansion card, and the eight vNICs of the Virtual Fabric Adapter means that up to 12 I/O ports are available to each blade in the chassis.
1.1 Virtual NICs: overview

A network interface controller (NIC) is a component in a blade server that allows the server to be connected to a network. The NIC provides the physical point of connection, as well as internal software for encoding and decoding network packets. With the original introduction of Virtual Fabric in 2009, IBM, Emulex, and BNT established the new vNIC feature. vNIC enables virtualization of the physical interface. Rather than eight physical connections between the adapter and the switch, there are two 10-Gb interfaces to two separate switch modules, and each 10-Gb interface can appear to the operating system as four separate physical interfaces. Tip: The vNIC concept described here is associated with the physical port and should not be confused with the concept of virtualization within a hypervisor-based operating system that provides virtual I/O interfaces to the virtual machines. Splitting up a single interface into four is achieved by following the PCIe standard of the PCI DeviceID. Each device can have one ID, and that device can have eight functions. This is not a new standard: many dual- or quad-port Ethernet or Fibre Channel Adapters have a single application-specific integrated circuit (ASIC) chip as well, but the physical interfaces are iterated as a function. So, in a sense, the VFA has virtualized Layer 1 of the Open Systems Interconnection (OSI) model, by virtualizing the physical interface. Virtualizing the NIC helps to resolve issues caused by limited NIC slot availability. By virtualizing a 10-Gbps NIC, its resources can be divided into multiple logical
instances or vNICs. Each vNIC appears as a regular, independent NIC to the server operating system or a hypervisor, with each vNIC using a portion of the overall bandwidth of the physical NIC. For example, a NIC partition with a maximum bandwidth of 4 Gbps appears to the host applications as a physically distinct 4-Gbps Ethernet adapter. The vNIC partitions provide traffic forwarding and port isolation based on the destination MAC address, the virtual local area network (VLAN) tag, or both. It does not provide bridging functions, either within a partition or across partitions.
1.2 Comparing modes

Table 1-3 compares the use of IBM Virtual Fabric Mode with Switch Independent Mode.
Table 1-3 Comparison of Virtual Fabric modes IBM Virtual Fabric Mode Number of virtual channels per physical 10-Gb port Total number of virtual channels per adapter Minimum virtual channel bandwidth (can also be set to 0 to disable the channel) Server-to-switch bandwidth limit per virtual channel Switch-to-server bandwidth limit per virtual channel Supported expansion cards 4 8 Emulex 10-Gb VFA: 100 Mba Emulex 10-Gb VFA II: 100 Mb Yes Yes Emulex 10GbE Virtual Fabric Adapter Emulex 10GbE Virtual Fabric Adapter II BNT Virtual Fabric 10Gb Switch Module Switch Independent Mode 4 8 100 MB
Yes No Emulex 10GbE Virtual Fabric Adapter II Broadcom 2-port 10Gb Virtual Fabric Adapter BNT Virtual Fabric 10Gb Switch Module Cisco Nexus 4001I Switch Module
Supported switches
IBM Virtual Fabric Mode Number of unique customer VLANs per virtual channel Maximum allowed by applications
Switch Independent Mode Varies by adapter Emulex VFA: 128 VLANs (4 channel) Broadcom VFA: Maximum allowed by applications Required Supported Not supported Broadcom VFA: No Emulex VFA II: Yes Yes Yesb Yes (BNT switch only) BNT: 1024 Cisco: 512
IEEE 802.1q VLAN tagging VLAN trunking on switch uplinks Isolated NIC teaming failover per vNIC channel iSCSI support - hardware initiator iSCSI support - software initiator Fibre Channel over Ethernet (FCoE) channel option Switch stacking Maximum number of virtual groups per switch
Optional Not supported Supported Emulex VFA I: Yes Emulex VFA II: Yes Yes Yesb Yes 32 (1 per virtual channel)c
a. Support for a minimum bandwidth allocation of 100 Mbps requires BNT Virtual Fabric 10Gb Switch Module firmware 6.5.3.0 or later and Emulex 10GbE Virtual Fabric Adapter firmware 2.103.269.3x or later. Otherwise, the minimum allocation is 1 Gbps. b. FCoE supported in a future firmware release. See the BladeCenter Interoperability Guide for the latest. c. One vNIC port per physical port per group.
1.3 Selecting the mode

You should use IBM Virtual Fabric Mode if your network environment has the following characteristics: You do not already implement IEEE 802.1q VLAN tags in your network infrastructure.For example, you may have implemented multiple IP subnets with Layer 3 separation among the Layer 2 broadcast domains provided by switches and routers. Servers may have multiple NIC adapters with attachments to physically separate switches or port-based VLAN groups to provide Layer 2 separation. You have implemented VLAN tagging only in the network infrastructure so that the tagging is transparent to the applications. This implementation
requires that the switches add and remove VLAN tags, rather than the server applications or NIC drivers doing so. You require overlapping IEEE 802.1q VLANs per NIC interface. Two (or more) individual vNICs on a given VFA should not share the same VLAN in either mode. You wish to implement a solution that takes advantage of Single Root IO Virtualization (SR-IOV). You should use Switch Independent Mode if your network environment has the following characteristics: You have elected to use the Cisco Nexus 4001I Switch Module in your BladeCenter H or HT chassis. You want to implement some of the advanced Layer 2 or Layer 3 BNT switch features that are not available in IBM Virtual Fabric Mode. You want to implement the Broadcom 2-port 10Gb Virtual Fabric Adapter to maintain driver compatibility with the base 1-Gb NIC cards.
1.4 Use of BladeCenter Open Fabric Manager

It is planned to also update BladeCenter Open Fabric Manager Advanced to support the deployment of Emulex and Broadcom 2-port 10Gb Virtual Fabric Adapters when configured for Switch Independent Mode of operation. This will allow you to configure the VFAs across multiple chassis from a central control point. Advanced BOFM pushes the parameters to a BOFM instance that is within the BladeCenter Advanced Management Module (AMM) in each chassis. This, in turn, pushes the parameters to the VFA on each blade. Note: This is a future capability that is not supported at the time of this publication. The VLAN configuration in the blade application drivers and the corresponding switch configuration must still be performed as described in Part 2 of this IBM Redbooks publication.
1.5 The future of converged fabrics and FCoE capabilities

The Emulex 10GbE Virtual Fabric Adapter and Emulex 10GbE Virtual Fabric Adapter II cards are planned to support FCoE in conjunction with vNIC channels with a future firmware release. This planned support will allow you to designate one of the four virtual channels on each of the two physical ports for the reliable transport of Fibre Channel storage traffic. This will also require a compatible FCoE-capable switch. FCoE support will be described in a subsequent revision of this book.
1.6 Using this book

This book includes two parts: The chapters in Part 1 cover the product detail and implementation for IBM Virtual Fabric Mode. The chapters in Part 2 cover the product detail and implementation for Switch Independent Mode. Essential information is repeated in both parts so that you need to only reference the part that applies to your hardware combination.
Part 1
Part
IBM Virtual Fabric Mode

The chapters in Part 1 cover the product detail and implementation for the IBM Virtual Fabric Mode. Chapters in this part are: Chapter 2, Solution architecture - IBM Virtual Fabric Mode on page 11 Chapter 3, Supported products - IBM Virtual Fabric Mode on page 19 Chapter 4, Configuring the solution - IBM Virtual Fabric Mode on page 37 Chapter 5, Examples of use - IBM Virtual Fabric Mode on page 81
10
Chapter 2.
Solution architecture - IBM Virtual Fabric Mode

The Virtual Fabric architecture is based on a virtual channel concept and is dependent upon both the Ethernet adapter and the Ethernet switch used to provide separation of the virtual channels on the shared 10-Gbps link. In this chapter, we introduce the technology and architecture of IBM Virtual Fabric Mode. Topics in this chapter are: 2.1, Overview on page 12 2.2, Architecture on page 14 2.3, Planning considerations on page 17
11
2.1 Overview
As described in Chapter 1, Introduction on page 1, a single 10-Gb link can be partitioned into up to four separate virtual channels. Each channel can be assigned a specified bandwidth. Traffic separation is maintained among the channels. The IBM Virtual Fabric Mode with the Emulex 10GbE Virtual Fabric Adapter (VFA) and the BNT Virtual Fabric 10Gb Switch Module (VFS) uses a double VLAN tagging scheme to maintain the separation of the virtual channels on the shared 10-Gbps link. Figure 2-1 shows an example of an implementation of IBM Virtual Fabric Mode and the VLAN tagging that is employed.
Virtual Fabric Mode Overview

VFA removes outer tag on VFA or VFS inserts outer frame egress tag for vSwitch Group VFS removes outer tag on frame egress
BNT 10G VFS

vSwitch Group 11
Emulex 10G VFA I or II Customer VLAN tag 10G Link
VLAN 10
vSwitch Group 22 Emulex 10G VFA I or II 10G Link vSwitch Group 33
VLAN 20
VLAN 30
Note: Inner tag denotes customer VLAN (optional)

Figure 2-1 IBM Virtual Fabric Mode overview
The VFA applies a unique outer tag to the packets, which allows the switch to properly handle the inbound frames and forward them to the appropriate subset
12
of ports. Likewise, the switch applies a corresponding outer tag on all packets that are received from the network infrastructure via the uplink ports and insures that these packets are only sent to VFAs that are members of the corresponding vNIC group. This double tag scheme allows the VLANs to be tunneled through the VFA and the switch. Clients should not need to make any concessions with regard to their existing network infrastructure Layer 2 networking scheme. Configuration is provided through the switch user interfaces. Parameters are passed to the adapter by the switch. Channel bandwidth is dynamic in that it may be changed (from the switch) without resetting the VFA. The VFA enforces bandwidth limits for each virtual channel. The VFS enforces the same virtual channel bandwidth limits on traffic flowing to the server. Bandwidth allocation and metering is controlled by the VFS, and, in cases when outgoing traffic flow for a particular vNIC exceeds established bandwidth limits, the switch may drop packets to maintain the traffic below configured thresholds. To isolate vNIC traffic flows from each other, vNICs can be assigned to vNIC logical groups (or vSwitches) that behave as independent switches inside the VFS. That is, vNICs in the same group can communicate with each other, whereas vNICs in different groups cannot. This vSwitch group ID is appended as an outer VLAN tag by either the VFS or the VFA. In addition to external or internal switch ports, external trunks can also be assigned to a particular vNIC group to isolate upstream connections. The VFA must be in vNIC mode to support the IBM Virtual Fabric Mode solution (vNIC mode is the default mode of operations for the VFA). In such a configuration, each physical port on the VFA is split into four virtual vNICs that are visible by the operating system as independent Ethernet controllers. The VFS recognizes these vNICs and manages and isolates network traffic flows between them. The BNT Virtual Fabric 10Gb Switch Module disables several Layer 2 and Layer 3 switch functions when supporting the IBM Virtual Fabric Mode. For example, ACL filters, VMReady, Layer 2 spanning tree, and Layer 3 routing are some of the functions that are not supported in conjunction with Virtual Fabric Mode operation. Note that these functions are disabled at the port level, not the entire switch. For example, you cannot configure vNIC and VMReady on the same switch port (blade server) but you can configure them on different ports on the same switch module. Note: These restrictions are not applicable with Switch Independent Mode, as described in Chapter 6, Solution architecture - Switch Independent Mode on page 101.
Chapter 2. Solution architecture - IBM Virtual Fabric Mode
13
The BNT switch will support stacking for both IBM Virtual Fabric Mode and Switch Independent Mode.
2.2 Architecture
Figure 2-2 on page 15 provides a general view of the vNIC solution architecture and offers an example of configuring the vNICs to having different bandwidth values. The different colors represent different virtual port groups to isolate traffic flows for different services and applications, and the thickness of each link represents the proportional amount of assigned bandwidth for the particular traffic flow. The Virtual Fabric architecture provides the capability to subdivide a single 10-Gb link into up to four virtual channels.
14
I/O Bay 1
I/O Bay 2
I/O Bay 4
Components and connectivity inside the IBM BladeCenter H chassis
BNT Virtual Fabric Switch (Bay 7)
Blade Server 14 Blade Server 1

FC Expansion Card Virtual Fabric Adapter On-Board 1GbE CIOv
INT1.1 INT1.2 INT1.3 INT1.4
I/O Bay 3
EXT1 EXT2 EXT3 EXT4 EXT5 EXT6 EXT7 EXT8 EXT9 EXT10 EXT1 EXT2 EXT3 EXT4 EXT5 EXT6 EXT7 EXT8 EXT9 EXT10
vSwitch1
vSwitch2
vSwitch3 vNIC1 vNIC2 vNIC3 vNIC4 vNIC1 vNIC2 vNIC3 vNIC4 INT14.1 INT14.2 INT14.3 INT14.4
vSwitch4
CFFh
BNT Virtual Fabric Switch (Bay 9)

INT1.1 INT1.2 INT1.3 INT1.4 vSwitch1
Legend
Standard I/O bays connections vNIC virtual connections External physical trunks vSwitch virtual trunk connections Mid-Plane
INT14.1 INT14.2 INT14.3 INT14.4
vSwitch2
vSwitch3
vSwitch4
Figure 2-2 BladeCenter Virtual Fabric vNIC solution diagram
Trunk failover is supported on a per-vNIC basis, so you can set up NIC teaming in a failover mode for vNICs. In case of external trunk or uplink failure, the switch shuts down only those vNICs that belong to the same vNIC group as the failed external trunk, whereas other vNICs on the same physical NIC (pNIC) port are untouched and continue to perform their functions. vNICs are configured on the BNT Virtual Fabric 10Gb Switch Module by dividing internal VFS ports into subinterfaces (or logical ports). Each internal interface of the VFS can be divided into up to four vNICs, and they maintain strict one-to-one mapping to vNICs on the VFA to a logical vSwitch group within the switch and the corresponding uplink port(s). Table 2-1 on page 16 is a vNIC mapping for VFA and VFS, as well as vNIC identification data, to differentiate vNICs on the host operating system (vNIC PCIe Function ID).
15
Table 2-1 vNIC mappings and identification vNIC PCIe function ID 0 2 4 6 1 3 5 7 VFA port number 0 0 0 0 1 1 1 1 VFS bay number 7 7 7 7 9 9 9 9 vNIC number 1 2 3 4 1 2 3 4 vNIC IDa INTx.1 INTx.2 INTx.3 INTx.4 INTx.1 INTx.2 INTx.3 INTx.4
a. A vNIC ID of x represents the VFS internal port number to which the VFA physical port is connected (x is 1 - 14).
General characteristics of this functionality are: Each server physical NIC (pNIC) port is divided into up to four virtual NICs (vNICs). Operating system (OS) configurations see eight unique NICs (2 ports x 4 vNICs). All vNIC parameters are configured from the VFS user interface. The Data Center Bridging Exchange (DCBX) protocol is used between the switch and NIC to convey configuration information. The user enables vNICs and allocates the bandwidth. The allowable vNIC bandwidth range is either 100 Mbps or 1 Gbps 10 Gbps (VFA model dependent): The increments are 100 Mbps The default bandwidth setting is 2.5 Gbps The sum of all four vNICs cannot exceed 10 Gbps
VFS-to-server bandwidth metering on a per vNIC per port basis. Server-to-VFS bandwidth metering on a per-vNIC basis. The user assigns vNICs and, optionally, uplinks to vNIC groups: Groups serve to isolate virtual NIC traffic flowing on the same physical port: Existing VLANs within the client network are not impacted.
16
No forwarding occurs between uplinks assigned to vNIC groups. Up to 32 vNIC groups are supported per Virtual Fabric Switch (VFS). An uplink (port or trunk) can belong to only one vNIC group. A server port (pNIC or vNIC) can belong to only one vNIC group.
The failover mechanism is virtual port aware. Key concurrency rules are: vNIC and VMready VMready is a unique function that enables the network to be virtual machine aware. The network can be configured and managed at the virtual port (vport) level, rather than just the physical port level. At the time of writing, you could not enable IBM Virtual Fabric Mode on either of the Emulex Virtual Fabric Adapters (VFAs) and implement BNT VMready on that server at the same time. However, certain ancillary functions of VMready that come with vCenter integration work with the VFA in either pNIC or vNIC mode. Running the VFA in pNIC mode and running VMready are supported concurrently. Check the latest release notes of the BNT Virtual Fabric 10Gb Switch Module firmware for the latest information. vNIC and switch stacking A stack is a group of BNT Virtual Fabric 10Gb Switch Modules that work together as a unified system and is thus managed as a single entity. The network views the stack as a single switch entity, and the stack can be accessed and managed as a whole using standard switch IP interfaces. Once the stacking links have been established, the number of ports available in a stack is equal to the total number of remaining ports of all the switches that are part of the stack.The numbers of available IP interfaces, VLANs, trunks, trunk links, and other switch attributes are not aggregated among the switches in a stack. The totals for the stack as a whole are the same as for any single switch configured in stand-alone mode. The BNT VFS supports stacking and Virtual Fabric Mode at the same time with the most recent firmware.
2.3 Planning considerations

You should take several considerations into account when planning a Virtual Fabric solution using IBM Virtual Fabric Mode: The adapter must operate in vNIC mode.
17
Up to four vNICs can be configured on each pNIC on the VFA (a total of eight vNICs on a 2-port VFA). The default bandwidth to be assigned to a vNIC is 2.5 Gbps. This can only be changed by configuration of the VFS. The maximum bandwidth that can be assigned to a vNIC interface is 10 Gbps. The minimum bandwidth that can be assigned to a vNIC depends upon the adapter used: Emulex 10GbE Virtual Fabric Adapter: 100-Mb minimum. Support for a minimum bandwidth allocation of 100 Mbps requies BNT Virtual Fabric 10-Gb Switch Module firmware 6.5.3.0 or later and Emulex 10GbE Virtual Fabric Adapter firmware 2.103.269.3x or later. Otherwise, the minimum allocation is 1 Gbps. Emulex 10GbE Virtual Fabric Adapter II: 100Mb minimum The sum of the bandwidth allocated for all four vNICs on the same pNIC cannot exceed 10 Gb. Up to 32 vNIC groups are supported on a VFS. vNICs on the same VFA must be assigned to separate vNIC groups at the VFS. Only one external port or one trunk made up of external ports is allowed per vNIC group. External trunks and ports can belong to, at most, one vNIC group at a time (there is no requirement that trunks and ports belong to a vNIC group). Internal ports not connected to a VFA in vNIC mode can belong to at most one vNIC group at a time. Virtual machines (VMs) and other virtual entities (VEs) associated with vNICs are automatically detected by VFS when VMready is enabled. However, these VEs cannot be assigned to any of the VMready VM groups, because vNIC groups used by vNICs are fully isolated from other switch entities. Currently, BladeCenter Open Fabric Manager is not supported for IBM Virtual Fabric Mode. Tip: All vNIC parameters are configured on the BNT Virtual Fabric 10Gb Switch Module.
18
Chapter 3.
Supported products - IBM Virtual Fabric Mode

In this chapter, we describe the products used in IBM Virtual Fabric Mode. This offering consists of the following supported adapters and switch modules: Emulex 10GbE Virtual Fabric Adapter and Virtual Fabric Adapter Advanced for IBM BladeCenter These are the first generation Emulex VFA cards for BladeCenter, and they are supported in IBM Virtual Fabric Mode offering only. Emulex 10GbE Virtual Fabric Adapter II and Virtual Fabric Adapter Advanced II for IBM BladeCenter These expansion cards are the second generation Virtual Fabric Adapters from Emulex, and can be used both in IBM Virtual Fabric Mode and in Switch Independent Mode. BNT Virtual Fabric 10Gb Switch Module The topics in this chapter are: 3.1, Overview on page 20 3.2, Emulex 10Gb Virtual Fabric Adapters on page 21 3.3, BNT Virtual Fabric 10Gb Switch Module on page 27
19
3.1 Overview
Table 3-1 lists the components supported by the IBM Virtual Fabric Mode solution.
Table 3-1 BladeCenter IBM Virtual Fabric Mode solution support matrix Description Chassis IBM BladeCenter H IBM BladeCenter HT Blade servers HS22 HS22V HX5 Virtual Fabric adapters Emulex 10GbE Virtual Fabric Adapter Emulex 10GbE Virtual Fabric Adapter II Virtual Fabric switches BNT Virtual Fabric 10Gb Switch Module External Virtual Fabric Switch connections Copper-based SFP+ Copper Direct Attach Cable - 1 M SFP+ Copper Direct Attach Cable - 3 M SFP+ Copper Direct Attach Cable - 7 M Fiber Optics-based IBM 10GBase-SR 10GbE 850 nm Fiber SFP+ Transceiver BLADE 1000Base-SX SFP IBM BNT 1000BASE-T SFP Transceiver 44W4408 81Y1622 81Y1618 1 per external port 1 per external port 1 per external port 59Y1936 59Y1940 59Y1944 1 per external port 1 per external port 1 per external port 46C7191 1 or 2 per chassis 49Y4235 90Y3550 1 per CFFh slot 1 per CFFh slot 7870 7871 7872 Varies Varies Varies 8852 8740/8750 Varies Varies Machine type/ part number Quantity
This chapter describes the features of these products.
20
3.2 Emulex 10Gb Virtual Fabric Adapters

You can use two generations of Emulex Virtual Fabric Adapters in IBM Virtual Fabric Mode: First generation Emulex 10GbE Virtual Fabric Adapter (part number 49Y4235) and Emulex 10GbE Virtual Fabric Adapter Advanced (part number 49Y4275) These adapters support IBM Virtual Fabric Mode only. They do not support Switch Independent Mode. The Emulex 10GbE Virtual Fabric Adapter II (part number 90Y3550) and Emulex 10GbE Virtual Fabric Adapter II Advanced (part number 90Y3566) In addition to IBM Virtual Fabric Mode, these adapters also support Switch Independent Mode. They also support Serial over LAN via 10-GB switches and support the technology preview of SR-IOV on Red Hat Enterprise Linux 6. The following sections describe these expansion cards in details.
3.2.1 Emulex 10GbE Virtual Fabric Adapter

The Emulex 10GbE Virtual Fabric Adapter for IBM BladeCenter is a dual-port 10-Gb Ethernet card that supports 1-Gbps or 10-Gbps traffic, or up to eight virtual NIC devices. The virtual NICs are configured to meet your mix of network connectivity and throughput demands for today's complex server application environments. Each physical 10-Gbps port can be divided into four virtual ports with bandwidth allocation in 100-Mbps increments to a maximum of 10 Gbps per physical port. The adapter is a CFFh expansion card; therefore, it requires that switch modules be installed in I/O bays 7 and 9. The Emulex 10GbE Virtual Fabric Adapter Advanced supports Fibre Channel over Ethernet (FCoE) and iSCSI hardware initiator functionality, in addition to the features available on the standard card. Table 3-2 lists the Emulex 10GbE Virtual Fabric Adapter ordering part numbers and feature codes.
Table 3-2 Ordering part numbers and feature codes Description Emulex 10GbE Virtual Fabric Adapter Emulex 10GbE Virtual Fabric Adapter Advanced Part number 49Y4235 49Y4275 Feature code 5755 2435
Chapter 3. Supported products - IBM Virtual Fabric Mode
21
Description Emulex 10GbE Virtual Fabric Adapter Advanced Upgrade
Part number 49Y4265
Feature code 2436
The Emulex 10GbE Virtual Fabric Adapter has the following features: Connection to either 1-Gb or 10-Gb data center infrastructure (1-Gb and 10-Gb auto-negotiation) PCI Express 2.0 x8 host interface IBM BladeCenter CFFh form factor, can be combined with a CIOv expansion card on the same blade server Operates either as an 8-port virtual NIC or as a 2-port 1/10Gb Ethernet adapter: Virtual port bandwidth allocation in 100-Mbps increments Supports up to eight virtual ports Wake On LAN support Full-duplex (FDX) capability Bus-mastering support Direct memory access (DMA) support Preboot Execution Environment (PXE) support IPv4/IPv6 TCP, UDP checksum offload Large send offload (LSO) Large receive offload Receive side scaling (RSS) IPv4 TCP Chimney Offload
VLAN insertion and extraction Jumbo frames up to 9000 bytes Load balancing and failover support including adapter fault tolerance (AFT), switch fault tolerance (SFT), adaptive load balancing (ALB), teaming support, and IEEE 802.3ad Enhanced Ethernet (draft) Enhanced Transmission Selection (ETS) (P802.1Qaz) Priority-based Flow Control (PFC) (P802.1Qbb) Data Center Bridging Capability Exchange Protocol (DCBX), CIN-DCBX and Converged Enhanced Ethernet- (CEE) DCBX (P802.1Qaz)
22
The Emulex Virtual Fabric Adapter Advanced provides FCoE and iSCSI support. The adapter is configurable from the BNT Virtual Fabric 10Gb Switch Module. This functionality is the basis for IBM Virtual Fabric Mode offering. The expansion card has two modes of operation: Virtual Fabric vNIC (or simply, vNIC) mode This mode is the basis for Virtual Fabric functionality. In vNIC mode, each physical port appears to the blade server as four virtual NICs. The default bandwidth for each vNIC is 2.5 Gbps. Bandwidth for each vNIC can be configured from 1 Gbps to 10 Gbps, in 100-Mbps increments. Total bandwidth of all vNICs combined cannot exceed 10 Gbps per physical port. vNICs can also be configured to have zero bandwidth if you must allocate the available bandwidth to fewer than eight vNICs. In IBM Virtual Fabric Mode, the expansion card communicates with the BNT Virtual Fabric 10Gb Switch Module, which provides independent control for each vNIC. Physical NIC (pNIC) mode In pNIC mode, the expansion card can operate as a standard 10-Gbps or 1-Gbps, 2-port Ethernet expansion card. When in pNIC mode, the expansion card functions with any supported 10-Gb switch installed in I/O module bays 7 and 9. In addition, the Emulex Virtual Fabric Adapter Advanced supports iSCSI hardware initiator functionality on both physical ports. When the ports operate in pNIC mode, you can use up to two iSCSI hardware initiators. When the ports are configured in vNIC mode, you can use two iSCSI initiators and six vNICs. The standard Emulex 10GbE Virtual Fabric Adapter can be upgraded to the same features as the Emulex 10GbE Virtual Fabric Adapter Advanced with the addition of the Advanced Upgrade option, part number 49Y4265.
23
The Emulex 10GbE Virtual Fabric Adapter is shown in Figure 3-1.
Figure 3-1 Emulex 10GbE Virtual Fabric Adapter
Figure 3-2 shows the placement of the adapter in an HS22 blade server. The adapter is installed in the CFFh expansion slot.
24
Emulex Virtual Fabric Adapter

Figure 3-2 Emulex Virtual Fabric Adapter placement in HS22 blade server
For more information, see the Emulex 10GbE Virtual Fabric Adapter At-a-Glance Guide at: http://www.redbooks.ibm.com/abstracts/tips0748.html
3.2.2 Emulex 10GbE Virtual Fabric Adapter II

This adapter enhances the existing BladeCenter Virtual Fabric portfolio. While its basic features are similar to the original Emulex 10GbE Virtual Fabric Adapter, the Emulex 10GbE Virtual Fabric Adapter II provides more flexible vNIC support: in addition to IBM Virtual Fabric Mode, it also supports Switch Independent Mode. The Emulex Virtual Fabric Adapter Advanced II offers FCoE and iSCSI hardware initiator functionality, in addition to the features available on the standard card. The Emulex 10GbE Virtual Fabric Adapter II is shown in Figure 3-3 on page 26.
25
Figure 3-3 Emulex 10GbE Virtual Fabric Adapter II
Table 3-3 lists the part numbers for ordering these expansion cards.
Table 3-3 Part numbers and feature codes for ordering Emulex adapters Description Emulex Virtual Fabric Adapter II (CFFh) Emulex Virtual Fabric Adapter Advanced II (CFFh) Emulex Virtual Fabric Adapter Advanced Upgrade Part number 90Y3550 90Y3566 49Y4265 Feature code A1XG A1XH 2436
Tip: Emulex Virtual Fabric Adapter Advanced Upgrade is the same part number (49Y4265) for both the Emulex 10GbE Virtual Fabric Adapter and the Emulex 10GbE Virtual Fabric Adapter II. The original Emulex 10GbE Virtual Fabric Adapter could be configured in pNIC or vNIC mode. The new Emulex 10GbE Virtual Fabric Adapter II now offers two flavors of vNIC: IBM Virtual Fabric Mode, which is the same as vNIC mode on the original Emulex VFA adapters. In this mode, the Emulex VFA card obtains the vNIC parameters from the switch via DCBX. This mode only works in conjunction with the BNT Virtual Fabric 10Gb Switch Module. Switch Independent Mode, where the adapter can work with BNT Virtual Fabric 10Gb Switch Module or the Cisco Nexus 4001I Switch Module.
26
Table 3-4 compares the Emulex 10GbE Virtual Fabric Adapter with the Emulex 10GbE Virtual Fabric Adapter II.
Table 3-4 Emulex Virtual Fabric Adapters: Feature comparison Function pNIC mode IBM Virtual Fabric Mode Switch Independent Mode iSCSI hardware initiator FCoE Serial over LAN (SoL) High energy efficiency Emulex Virtual Fabric Adapter Yes Yes No Yes (Advancedb) Yes (Advancedb ) No No Emulex Virtual Fabric Adapter II Yes Yes Yesa Yes (Advancedb ) Yes (Advancedb ) Yes Yesc
a. Planned for 4Q/2011 b. Available on Emulex Virtual Fabric Adapter Advanced II, or with Advanced Upgrade c. Emulex Virtual Fabric Adapter II is 20% more energy efficient
3.3 BNT Virtual Fabric 10Gb Switch Module

The BNT Virtual Fabric 10Gb Switch Module for IBM BladeCenter offers the most bandwidth of any blade switch, and it represents the perfect migration platform for clients who are still at 1 Gb outside the chassis by seamlessly integrating in the existing 1-Gb infrastructure. This is the first 10-Gb switch for IBM BladeCenter that supports converged networking (that is, the ability to transmit Converged Enhanced Ethernet (CEE) to a Fibre Channel over Ethernet (FCoE)-capable, top-of-rack switch). This feature is available with firmware release 6.1 and higher. CEE and FCoE functionality enables you to transfer storage, network, Voice over IP (VoIP), video, and other data over the common Ethernet infrastructure. With the use of the QLogic Virtual Fabric Extension Module, clients can achieve FCoE gateway functionality inside the BladeCenter chassis. In this publication, we concentrate on virtual fabric. The BNT Virtual Fabric 10Gb Switch Module can be used both in IBM Virtual Fabric Mode and Switch Independent Mode. In IBM Virtual Fabric Mode, all vNIC parameter configuration is performed on the BNT Virtual Fabric 10Gb Switch Module. Therefore, the switch module is a key part of the offering. This offering allows clients to form
27
eight vNICs from one physical NIC and to manage them in virtual groups. The switch module can be managed using a command-line interface or web browser interface, providing all of the benefits of I/O virtualization at 10-Gb speed. If you have a chassis with multiple servers, several operating at 1 Gbps, several at 10 Gbps, and several transmitting converged packets, this single switch can handle all of these workloads and can connect to a 1-Gb infrastructure to a 10-Gb infrastructure, or both. With the extreme flexibility of the BNT switch, you can take advantage of the technologies that they require for multiple environments. For 1-Gbps uplinks, they can take advantage of SFP transceivers. For 10-Gbps uplinks, they have a choice of either SFP+ transceivers (short range or long range for longer distances, or direct-attached copper (DAC) cables (also known as twinax active cables) for shorter distances. DAC cables are more cost-effective, consume less power, and can be up to 7 meters in length. They are ideal for connecting chassis, connecting to a top-of-rack switch, or even connecting to an adjacent rack. Figure 3-4 shows the switch module.
Figure 3-4 BNT Virtual Fabric 10Gb Switch Module
Table 3-5 lists the part number and feature code to use to order the module.
Table 3-5 BNT Virtual Fabric 10Gb Switch Module part number and feature code Description BNT 10Gb Virtual Fabric Switch Module for IBM BladeCenter Part number 46C7191 Feature code 1639
28
The part number includes the following items: One BNT Virtual Fabric 10Gb Switch Module Three-meter, mini-USB-to-DB9 serial console cable One filler module BNT Virtual Fabric 10Gb Switch Module Installation Guide BNT user license agreement Important Notices document Documentation CD-ROM Tip: Small form-factor pluggable plus (SFP+) transceivers are not included and must be purchased separately. To communicate outside of the chassis, you must have either SFP+ transceivers or SFP+ direct-attach copper (DAC) cables connected. DAC cables have SFP+ transceivers on both ends. You have the flexibility to expand your bandwidth as desired, using anywhere from one connection up to ten connections per switch. Table 3-6 on page 30 lists the part numbers for ordering the SFP+ transceivers, FC cables, and DAC cables. Tip: There are both 10-Gb and 1-Gb SFP+ modules available for uplink.
29
Table 3-6 IBM part numbers for ordering SFP+ transceivers, FC cables, and DAC cables Description 10Gb SFP+ IBM 10GBase-SR 10GbE 850 nm Fiber SFP+ Transceiver BNT SFP+ Transceiver 1Gb SFP+ BLADE 1000BASE-T (RJ45) SFP Transceiver BLADE 1000BASE-SX SFP Transceiver DAC cables 0.5 m Molex Direct Attach Copper SFP+ Cable 1 m Molex Direct Attach Copper SFP+ Cable 3 m Molex Direct Attach Copper SFP+ Cable 7 m Molex Direct Attach Copper SFP+ Cable FC cables 3 m Intel Connects Optical Cable 10 m Intel Connects Optical Cable 30 m Intel Connects Optical Cable 46D0153 46D0156 46D0159 3852 3853 3854 59Y1932 59Y1936 59Y1940 59Y1944 3735 3736 3737 3738 81Y1618 81Y1622 3268 3269 44W4408 46C3447 4942 5053 Part number Feature code
Table 3-7 lists additional transceivers and DAC cable options that are available directly from BNT.
Table 3-7 BNT part numbers for ordering SFP-based transceivers and cables Description BLADE 1000Base-T SFP BLADE 1000Base-SX SFP BLADE 10GBase-LR SFP+ SFP+ Copper Direct Attach Cable - 1 M SFP+ Copper Direct Attach Cable - 3 M SFP+ Copper Direct Attach Cable - 7 M Part number BN-CKM-S-T BN-CKM-S-SX BN-CKM-SP-LR BN-SP-CBL-1M BN-SP-CBL-3M BN-SP-CBL-7M
30
The BNT Virtual Fabric 10Gb Switch Module includes the following features and functions: Form-factor Single-height, high-speed switch module Internal ports Fourteen internal auto-negotiating ports: 1 Gbps or 10 Gbps to the server blades Two internal, full-duplex 100-Mbps ports connected to the management module External ports Up to ten 10-Gb SFP+ ports (also designed to support 1-Gb SFP if required, flexibility of mixing 1 Gb/10 Gb) Oversubscription ratio (14 internal ports to 10 external ports) is very low, and this makes the switch module suitable for the most performance-intensive environments. One 10/100/1000-Mb copper RJ45 used for management or data An RS-232 mini-USB connector for serial port that provides an additional means to install software and configure the switch module Scalability and performance Autosensing 1-Gb/10-Gb internal and external Ethernet ports for bandwidth optimization Non-blocking architecture with wire-speed forwarding of traffic and full line rate performance of 480-Gbps full duplex Media access control (MAC) address learning: automatic update, supports up to 32-Kb MAC addresses Up to 128 IP interfaces per switch Static, EtherChannel, and Link Aggregation Control Protocol (LACP) (IEEE 802.3ad) link aggregation, up to 100 Gb of total bandwidth per switch, up to 18 trunk groups, and up to eight ports per group Support for jumbo frames (up to 12288 bytes) Broadcast and multicast storm control IGMP snooping for limit flooding of IP multicast traffic (IGMP V1, V2, and V3) Configurable traffic distribution schemes over trunk links, based on source and destination IP addresses, MAC addresses, or both
31
Fast port forwarding and fast uplink convergence for rapid Spanning Tree Protocol (STP) convergence Stacking support - clients can stack up to eight BNT Virtual Fabric 10Gb Switch Module Availability and redundancy Virtual Router Redundancy Protocol (VRRP) for Layer 3 router redundancy IEEE 802.1D STP for providing Layer 2 redundancy with PVRST+ IEEE 802.1s Multiple STP (MSTP) for topology optimization, up to 128 STP instances supported by single switch IEEE 802.1w Rapid STP (RSTP), providing rapid STP convergence for critical delay-sensitive, traffic-like voice or video Layer 2 trunk failover to support active and standby configurations of network adapter teaming on blades Interchassis redundancy (Layer 2 and Layer 3) VLAN support Up to 1024 VLANs supported per switch, VLAN numbers ranging from 1 to 4095 (4095 is a dedicated VLAN used for the management module connection only) 802.1Q VLAN tagging support on all ports Protocol-based VLANs Security VLAN-based, MAC-based, and IP-based access control lists (ACLs) 802.1X port-based authentication Multiple user IDs and passwords User access control Radius, Terminal Access Controller Access-Control System Plus (TACACS+), Lightweight Directory Access Protocol (LDAP) Quality of Service (QoS) Up to eight queues per port Support for IEEE 802.1p, IP ToS/DSCP, and ACL-based (MAC/IP source and destination addresses, VLANs) traffic classification and processing Traffic shaping and remarking based on defined policies Eight Weighted Round Robin (WRR) priority queues per port for processing qualified traffic
32
Layer 3 functions IP forwarding IP filtering with ACLs (up to 4096 ACLs supported) VRRP for router redundancy Support for up to 128 static routes Routing protocol support (Router Information Protocol (RIP) V1, RIP V2, Open Shortest Path First protocol (OSPF)V1, V2, and V3, BGP-4), up to 1024 entries in routing table IPv6 routing including static routes and OSPFv3 (requires firmware V6.3 or higher) Support for Dynamic Host Configuration Protocol (DHCP) Relay IPv6 host management IPv6 forwarding based on static routes Manageability Simple Network Management Protocol (SNMP V1, V2, and V3) HTTP and HTTPS Browser-Based Interface (BBI) Industry standard CLI and BLADEOS/AlteonOS CLI Telnet interface for CLI SSH v1/v2 Serial interface for CLI Scriptable CLI Firmware image update (Trivial File Transfer Protocol (TFTP) and File Transfer Protocol (FTP)) Network Time Protocol (NTP) for switch clock synchronization BNT BLADEHarmony Manager support Monitoring Switch LEDs for external port status and switch module status indication Port mirroring for analyzing network traffic passing through switch Change tracking and remote logging with syslog feature Power-On Self Test (POST) diagnostics Special functions Serial over LAN (SOL)
33
Virtualization features VMready Virtual Fabric Adapter vNIC support Converged Enhanced Ethernet and FCoE features FCoE allows Fibre Channel traffic to be transported over Ethernet links. FCoE Initialization Protocol (FIP) snooping to enforce point-to-point links for FCoE traffic outside the regular Fibre Channel topology. Priority-Based Flow Control (PFC) (IEEE 802.1Qbb) extends the 802.3x standard flow control to allow the switch to pause traffic, based on the 802.1p priority value in each packet VLAN tag. Enhanced Transmission Selection (ETS) (IEEE 802.1Qaz) provides a method for allocating link bandwidth, based on the 802.1p priority value in each packet VLAN tag. DCBX (IEEE 802.1AB) allows neighboring network devices to exchange information about their capabilities. Supports the QLogic Virtual Fabric Extension Module for IBM BladeCenter, which provides FCoE gateway functionality inside the BladeCenter chassis. VMready is a unique solution that enables the network to be virtual machine-aware. The network can be configured and managed for virtual ports (v-ports), rather than just for physical ports. With VMready, as VMs migrate across physical hosts, so do their network attributes. Virtual machines can be added, moved, and removed, while retaining the same ACLs, QoS, and VLAN attributes. VMready allows for a define-once-use-many configuration that evolves as the server and network topologies evolve. VMready works with all virtualization products, including VMware, Hyper-V, Xen, and KVM, without modification of virtual machine hypervisors or guest operating systems. It is available as part of the 6.1 (and higher) software code. VMready compatibility with Virtual Fabric solutions is as follows: VMready is not supported with IBM Virtual Fabric Mode. VMready is supported with Switch Independent Mode. The switch module supports the following IEEE standards: IEEE 802.1D STP with PVRST+ IEEE 802.1s MSTP IEEE 802.1w RSTP IEEE 802.1p Tagged Packets
34
IEEE 802.1Q Tagged VLAN (frame tagging on all ports when VLANs are enabled) IEEE 802.1x port-based authentication IEEE 802.2 Logical Link Control IEEE 802.3ad Link Aggregation Control Protocol IEEE 802.3x Full-duplex Flow Control IEEE 802.3ab 1000BASE-T Gigabit Ethernet IEEE 802.3ae 10GBASE-SR 10-Gb Ethernet fiber optics short range IEEE 802.3ae 10GBASE-LR 10-Gb Ethernet fiber optics long range IEEE 802.3z 1000BASE-SX Gigabit Ethernet The following network cables are supported for the BNT Virtual Fabric 10Gb Switch Module: 10GBASE-SR for 10-Gb ports: 850Nm wavelength, multimode fiber, 50 or 62.5 (300 meters maximum), with LC duplex connector 1000BASE-T for RJ45 port: UTP Category 6 (100 meters maximum) UTP Category 5e (100 meters maximum) UTP Category 5 (100 meters maximum) EIA/TIA-568B 100-ohm STP (100 meters maximum)
More information is available in the following publications: BNT Virtual Fabric 10Gb Switch Module At-a-Glance Guide, available at: http://www.redbooks.ibm.com/abstracts/tips0708.html Documents in PDF format available at: http://www.ibm.com/support/docview.wss?uid=psg1MIGR-5080917 are: BNT Virtual Fabric 10Gb Switch Module Installation Guide Application Guide Command Reference isCLI Reference BBI (Browser-based Interface) Quick Guide Release Notes End User License Agreement
35
36
Chapter 4.
Configuring the solution IBM Virtual Fabric Mode

In this chapter, we describe how to configure the components for operating in IBM Virtual Fabric Mode. When configuring the solution in this mode, we configure vNIC parameters (such as bandwidth allocation, vNIC groups and VLANs) on the BNT Virtual Fabric 10Gb Switch Module. The switch transfers the configuration settings to the Emulex 10GbE Virtual Fabric Adapter using the Data Center Bridging eXchange (DCBX) protocol. The adapter requires very little configuring: you only need to enable vNIC mode in the blade server Unified Extensible Firmware Interface (UEFI) utility. The rest of the configuration is performed on the switch. The BNT Virtual Fabric 10Gb Switch Module can be configured using any of the following tools: BladeOS command line interface (CLI) Industry-standard CLI (isCLI) Browser-based interface (BBI) This chapter has the following topics: 4.1, Configuring the Emulex 10GbE Virtual Fabric Adapter 4.2, Configuring the BNT Virtual Fabric 10Gb Switch Module on page 44 4.3, Creating virtual NICs on page 56
37
4.4, vNIC group configuration on page 63 4.5, Configuring vNIC VLANs on page 66 4.6, vNIC and VMready on page 69 4.7, High availability and vNICs on page 70 4.8, Configuring the operating system on page 72
4.1 Configuring the Emulex 10GbE Virtual Fabric Adapter

This section describes how to upgrade firmware and configure the Emulex 10GbE Virtual Fabric Adapter cards. Note: Our discussion and screen shots show the process using the new Emulex 10GbE Virtual Fabric Adapter II. The procedures are similar on the older Emulex 10GbE Virtual Fabric Adapter cards, with minor differences.
4.1.1 Upgrading firmware

We recommended that you upgrade the firmware on the Emulex 10GbE Virtual Fabric Adapter II to the latest level before beginning these setup procedures. You can upgrade firmware in a number of ways: Windows Server 2008, 2008 R2 and 2003 online update Windows PE offline update Linux online code update Linux offline code update VMware ESX code update package Code update using a bootable DVD media UEFI code update The upgrade procedure is documented in the readme files for these products.
Perform the upgrade

In the example that follows, we use the Windows online package to upgrade the firmware. The package uses an executable file, elxflash.exe, to perform the update. Note: When using elxflash.exe for the firmware update, some parameters might need to be set prior to the update. These parameters are explained in the readme file for the executable.
38
1. Enter the elxflash /update command 2. Verify the results (Figure 4-1)
>elxflash /update Thu May 19 15:41:27 2011 HBA=OCe11100-NIC, Port Type=NIC, MAC=00-00-C9-B2-57-02, Update=Firmware, Image=oc11-4.0.200.10.ufi, New=4.0.200.10, Old=3.103.351.10, Status=Success Thu May 19 15:41:27 2011 HBA=OCe11100-NIC, Port Type=NIC, MAC=00-00-C9-B2-57-06, Update=Firmware, Image=oc11-4.0.200.10.ufi, New=4.0.200.10, Old=3.103.351.10, Status=Success (lines deleted for clarity) Thu May 19 15:41:28 2011 HBA=OCe11100-NIC, Port Type=NIC, MAC=00-00-C9-B2-57-09, Update=Firmware, Image=oc11-4.0.200.10.ufi, New=4.0.200.10, Old=3.103.351.10, Status=Success elxflash: All required updates succeeded - Return Code=0 Figure 4-1 Windows online code update, using the elxflash executable
The output is truncated here for clarity. The command produces eight sections, with three shown in Figure 4-1. Each section represents one vNIC.
4.1.2 Configuring the adapter in the UEFI interface

Note: The instructions and panel shots in this section are based on pre-GA level firmware of the Emulex 10GbE Virtual Fabric Adapter II. The steps may be different for you using the latest firmware. To configure the Emulex 10GbE Virtual Fabric Adapter II using the blade server UEFI interface: 1. Power on or restart the blade 2. When prompted during POST, press F1 to display the System Configuration and Boot Management panel (also known as the UEFI utility main menu) (Figure 4-2 on page 40).
Chapter 4. Configuring the solution - IBM Virtual Fabric Mode
39
Figure 4-2 Blade UEFI utility main menu
3. Select System Settings, then select Network in the subsequent window, to display the Network configuration panel, showing the Emulex 10GbE Virtual Fabric Adapter II ports as the two physical ports on the adapter (Figure 4-3 on page 41).
40
Figure 4-3 Network configuration panel in the blade UEFI utility
4. Select the first port to configure to display the configuration panel. To enable IBM Virtual Fabric Mode (vNIC) or Switch Independent Mode (vNIC2) mode, Multichannel needs to be enabled first. Unless Multichannel is enabled, the card operates as two physical 10-Gbps Ethernet ports. When you enable Multichannel, it is enabled on both of the 10-Gb physical ports. The process of enabling Multichannel is discussed in 9.3.1, Emulex adapter configuration on page 226. Tips: vNIC2 as listed here refers to Switch Independent Mode as discussed in Switch Independent Mode on page 99. Switch Independent Mode is not available on the older Emulex 10GbE Virtual Fabric Adapter. SR-IOV is disabled when using Switch Independent Mode. 5. With Multichannel enabled, select vNIC or vNIC2 mode, as needed (for our example, using IBM Virtual Fabric Mode, select vNIC (Figure 4-4 on page 42).
41
Figure 4-4 Emulex NIC selection
6. Select Multichannel Configuration to display the Function Configuration panel, listing the vNIC settings, where four functions represent four vNICs (Figure 4-5 on page 43).
42
Figure 4-5 Function Configuration panel
7. Select Function to display the Multichannel Configuration panel in which the parameters for that vNIC will be configured (Figure 4-6).
Figure 4-6 Set the Logical Port VLAN ID (LPVID) (optional, mode-dependent)
43
When in IBM Virtual Fabric Mode, there is only one parameter: Logical Port VLAN ID (LPVID). Configuration of this parameter is optional because vNIC functionality does not depend on it. This is the inner VLAN ID in double-tagged frames (the outer VLAN ID and bandwidth parameters come from the BNT Virtual Fabric 10Gb Switch Module). However, if we were configuring Switch Independent Mode (selecting vNIC2 in Figure 4-4 on page 42), LPVID would be a required parameter. 8. Save the settings after making any changes. With Multichannel enabled, it is enabled on both physical ports. We suggest that you review the vNIC settings for the second physical port to ensure that the configuration is correct. 9. Exit to the UEFI main menu, select Save Settings (Figure 4-7) and restart the blade server.
Figure 4-7 Save settings
4.2 Configuring the BNT Virtual Fabric 10Gb Switch Module

This section provides an overview of the tools available for configuring the BNT Virtual Fabric 10Gb Switch Module. This switch module can be configured using a broad-based web browser (BBI), a command line interface (CLI) (either
44
BladeOS or isCLI) or Simple Network Management Protocol (SNMP). Future plans include management of switch modules from BOFM. The examples in this section use BNT Virtual Fabric 10Gb Switch Module firmware version 6.5.3.0, the current version at the time of this writing.
4.2.1 BladeOS CLI

BladeOS CLI is a simple and intuitive, menu-based user interface that provides complete switch module administration. To use the BladeOS CLI on the switch module, use the following communication methods: Telnet session Serial connection SSH connection
Log in to the BladeOS

BladeOS CLI consists of a main menu and hierarchically organized submenus. An example of logging in to the switch module using telnet follows: 1. Use the telnet command to connect to the switch module with IP address 9.42.171.39. telnet 9.42.171.39 2. Type the password (the default password is admin). 3. The BladeOS main menu displays, serving as your starting point for all switch module management and configuration tasks (Figure 4-8 on page 46).
45
[Main Menu] May 25 14:01:06 9.44.16.209 info stats cfg oper boot maint diff apply save revert exit >> Main# Figure 4-8 BladeOS CLI main menu 9.42.171.39 NOTICE mgmt: admin(admin) login from host
Information Menu Statistics Menu Configuration Menu Operations Command Menu Boot Options Menu Maintenance Menu Show pending config changes [global command] Apply pending config changes [global command] Save updated config to FLASH [global command] Revert pending or applied changes [global command] Exit [global command, always available]
Upgrade the firmware

Here we show an example of the BNT Virtual Fabric 10Gb Switch Module firmware upgrade using BladeOS CLI. At the time of this writing, firmware 6.5.3.0 is the current level. Therefore, this procedure upgrades the switch module to this version. Late breaking news: BNT Virtual Fabric 10Gb Switch Module firmware 6.7.2 is now available and is required for the Emulex 10GbE Virtual Fabric Adapter II. This firmware level is also required if the Emulex 10GbE Virtual Fabric Adapter is also updated to the latest firmware. The instructions in this section, however, still refer to firmware 6.5.3.0. The firmware update package is located on the IBM support website at: http://ibm.com/support/entry/portal/docdisplay?lndocid=MIGR-5086594 The firmware code consists of three parts: Boot image OS image 1 OS image 2 When upgrading firmware, you need to upgrade the boot image and one of the two OS images. It is suggested that you leave the other OS image at the existing firmware level, so that you can easily activate it should any problems arise with the new level.
46
Any TFTP or FTP server can be used to transfer the boot and OS images to the BNT Virtual Fabric 10Gb Switch Module. In our example, we use the TFTP server available on the BladeCenter Advanced Management Module (AMM). From the firmware upgrade package, we moved two files into the tftproot directory in AMM local storage: GbESM-24-10G-6.5.3.0_Boot.img: This is the new boot image. GbESM-24-10G-6.5.3.0_OS.img: This is the new OS image. The contents of the tftproot directory in AMM local storage are shown in Figure 4-9.
Figure 4-9 Firmware files in the tftproot directory
After ensuring that TFTP server functionality is enabled on the AMM, proceed with the firmware upgrade using BladeOS CLI.
47
Upgrade the OS image

The command to transfer the new OS image into one of the image banks is: /boot/gtimg X TADDR GbESM-24-10G-6.5.3.3_OS.img where X represents the OS image bank (1 or 2) and TADDR represents the TFTP server IP address. 1. Enter the command to transfer the image to an image bank (in our example, use image bank 2), using the following command: /boot/gtimg 2 9.42.171.73 GbESM-24-10G-6.5.3.0_OS.img 2. Verify the results (Figure 4-10).
>> Main# /boot/gtimg 2 9.42.171.73 GbESM-24-10G-6.5.3.0_OS.img Enter username for FTP server or hit return for TFTP server: image2 currently contains Software Version 6.1.2 that was downloaded at 6:11:38 Thu Jan 5, 2000. New download will replace image2 with file "GbESM-24-10G-6.5.3.0_OS.img" from FTP/TFTP server 9.42.171.73. Confirm download operation [y/n]: y Starting download... File appears valid Download in progress ................................................................. ................................................................. ................................................................ Image download complete (5737743 bytes) Writing to flash...This takes about 90 seconds. Please wait Write complete (5737743 bytes), now verifying FLASH... Verification of new image2 in FLASH successful. image2 now contains Software Version 6.5.3 AMM CodeLevel 3 : 0605WMJ03000 BLADEOS Im2 11/17/2010 Switch is currently set to boot software image1. Do you want to change that to the new image2? [y/n] May 25 14:57:28 9.42.171.39 INFO mgmt: image2 downloaded from host 9.42.171.73, file 'GbESM-24-10G-6.5.3.0_OS.img', software version 6.5.3 y Next boot will use new software image2. >> May 25 14:57:46 9.42.171.39 NOTICE mgmt: boot image changed Boot Options# May 25 14:57:46 9.42.171.39 INFO mgmt: Firmware downloaded to image2 Figure 4-10 Upgrade of OS image 2
48
Figure 4-10 confirms that we have successfully updated OS image 2 to the new firmware version, version 6.5.3.0. Before the upgrade, the switch module was set to boot from image bank 1. Notice in Figure 4-10 that this has been changed to image bank 2 during the update process.
Upgrade the boot image

The command to transfer the new boot image is: /boot/gtimg boot TADDR GbESM-24-10G-6.5.3.0_Boot.img where TADDR represents the IP address of the TFTP server. To begin the transfer: 1. Enter the command to transfer the image: /boot/gtimg boot 9.42.171.73 GbESM-24-10G-6.5.3.0_Boot.img 2. Verify the results (Figure 4-11 on page 50).
49
>> Boot Options# /boot/gtimg boot 9.42.171.73 GbESM-24-10G6.5.3.0_Boot.img Enter username for FTP server or hit return for TFTP server: boot kernel currently contains Software Version 6.1.2 New download will replace boot kernel with file "GbESM-24-10G6.5.3.0_Boot.img" from FTP/TFTP server 9.42.171.73. Confirm download operation [y/n]: y Starting download... File appears valid Download in progress ................................................................. ................................................................. ................................................................. Boot image (FS, 7687332 bytes) download complete. Writing to flash...This can take up to 90 seconds. Please wait FS Sector now contains Software Version 6.5.3 Boot image (Kernel, 7687332 bytes) download complete. Writing to flash...This can take up to 90 seconds. Please wait Kernel Sector now contains Software Version 6.5.3 Boot image (Boot, 7687332 bytes) download complete. Writing to flash...This can take up to 90 seconds. Please wait Boot Sector now contains Software Version 6.5.3 >>May 25 15:13:17 9.42.171.39 INFO mgmt: boot kernel downloaded from host 9.42.171.73, file 'GbESM-24-10G-6.5.3.0_Boot.img', software version 6.5.3 Boot Options# May 25 15:13:17 9.42.171.39 INFO mgmt: Firmware downloaded to boot kernel Figure 4-11 Confirmation of the boot image upgrade
Reset the switch module

Enter the command to reset the switch module: /boot/reset
Assistance with BladeOS CLI commands

For more information about BladeOS CLI commands, see Command Reference 6.3, BNT Virtual Fabric 10 Gb Switch Module, available at: http://ibm.com/support/entry/portal/docdisplay?lndocid=MIGR-5080917
50
4.2.2 isCLI
BladeOS is a proprietary CLI environment, developed by BNT for configuring and managing their switch modules. System administrators who are more familiar with Cisco CLI have the option of using the industry standard CLI (isCLI). isCLI is a Cisco-like user interface, which allows the complete management and configuration support of BNT switch modules. BladeOS CLI is the default CLI mode, but you can easily change between BladeOS and isCLI.
Change your choice of CLI between BladeOS and isCLI

The following shows how to change from BladeOS to isCLI and from isCLI to BladeOS: To change from BladeOS to isCLI, use the following command: boot/mode iscli Restart the switch module after this command is run. To revert from isCLI to BladeOS CLI, run the command: boot cli-mode bladeos-cli Restart the switch module after the command is run. The selected CLI mode is preserved across reboots, even when you restore the switch module to factory defaults.
isCLI privilege levels

The isCLI mode has three privilege levels: User EXEC mode: This is the initial mode with the most limited privileges. You can perform basic tasks, such as list system information, but you cannot set or change operating parameters. With this privilege level set, the isCLI prompt ends with the greater than character, >, for example: Switch> Privileged EXEC mode: This mode has a higher privilege level. It allows you to set or change operating parameters of the switch module. With this privilege level set, the isCLI prompt ends with the number sign (also known as the pound sign), #, for example: Switch# Privileged EXEC mode is accessed from User EXEC mode with the command: enable Global Configuration Mode: This is the most privileged mode. When in this mode, you can make changes to the running configuration, and set or change
51
global parameters. Use the following command to enter this mode from Privileged EXEC mode: configure terminal To change the privilege level, enter the commands shown in Figure 4-12. Figure 4-12 is an example of enabling Privileged EXEC and then Global Configuration Mode. To enable a different privilege level, see isCLI privilege levels on page 51.
Router>enable Enable privilege granted. Router#configure terminal Enter configuration commands, one per line. Router(config)# Figure 4-12 isCLI: Enabling EXEC privileges End with Ctrl/Z.
Assistance with isCLI commands

For more information about isCLI commands, see isCLI Reference 6.3, BNT
Virtual Fabric 10 Gb Switch Module, available at:

http://www.ibm.com/support/docview.wss?uid=psg1MIGR-5080917
4.2.3 Browser-based interface (BBI)

The browser-based interface (BBI) can be used to perform most management, administration, and configuration tasks on the BNT Virtual Fabric 10Gb Switch Module. BBI is an easy-to-use and intuitive, web-based management interface and is enabled by default. It can be disabled using the CLI.
Launch the BBI

The following shows how to log in to the switch module and launch BBI: 1. Open a web browser, and connect to the switch module IP address. Alternatively, you can access BBI from the AMM web interface: a. Select the following options in the AMM web interface: I/O Module Tasks Configuration Advanced Options Start CLI/Web Session
52
b. Ensure that the Protocol field is set to Web (not Telnet), then click Start Session (Figure 4-13)
Ensure Web is selected as the Protocol, then click Start Session.
Figure 4-13 AMM web interface: Starting the CLI web session
2. Log in to the switch module (Figure 4-14 on page 54). Default credentials are: Username: admin Password: admin Note: Clients following best practice recommendations have likely changed the password from the default for added security.
53
Figure 4-14 Login panel for the BNT Virtual Fabric 10Gb Switch Module
3. After a successful login, the BBI Switch Dashboard is displayed (Figure 4-15 on page 55).
54
Figure 4-15 BNT Virtual Fabric 10Gb Switch Module dashboard
The switch dashboard is your starting point for switch module management.
Assistance with BBI commands

For more information about BBI commands, see the BBI Quick Guide, available at: http://ibm.com/support/entry/portal/docdisplay?lndocid=MIGR-5080917
55
4.3 Creating virtual NICs

In this section, we show how to create vNICs in a BNT Virtual Fabric 10Gb Switch Module, using three methods: BBI: a browser-based interface isCLI: the international standard command line interface (CLI) BladeOS: the BNT CLI Each vNIC is created and configured on the switch. The properties of each vNIC are communicated to the Emulex adapter using the Data Center Bridging eXchange (DCBX) protocol.
4.3.1 Before you configure

Consider the following when creating vNICs: Up to four vNICs can be created on any blade. There is no requirement to create all four. Only 10 Gb of total bandwidth can be allocated to the vNICs on one port on a server blade. Oversubscription is not supported. If less than 10 GB of total bandwidth is allocated to all of the defined vNICs on a port, then the unallocated bandwidth is not usable. Bandwidth allocations are policed on both inbound and outbound traffic. Unlike certain configurations that use quality of service, and unlike the virtual NIC functionality built into the IBM AIX operating system, there is no provision to allow a given vNIC to exceed its allocated bandwidth at any time. Bandwidth allocations can be changed from the switch command line or browser interface. These changes are propagated to the servers immediately and do not require that the network be brought down or that the server be rebooted. It is possible, for instance, to have one bandwidth allocation scheme in use during business hours and a different one during off hours when batch processing is in progress.
4.3.2 Enabling and disabling vNIC functionality on the server

To configure vNICs on one blade, begin by enabling vNIC functionality. This process uses a master toggle switch, which can be used to temporarily disable vNIC without removing all of the configuration details. This command is carried out once per switch, regardless of how many blades will use the vNIC.
56
Using BBI
You can enable vNIC mode using BBI by setting the Global vNIC On/Off parameter to On as shown in Figure 4-16 and clicking Submit. Disable it by setting the parameter to Off.
Figure 4-16 Enabling vNIC mode
Using isCLI
You can enable vNIC mode using isCLI with the command vnic enable as shown in Example 4-1.
Example 4-1 Enabling vNIC mode BNT Virtual Fabric 10Gb Switch Module for IBM BladeCenter. Enter password: System Information at 19:12:16 Sun Jan 8, 2000 Router> Jan 8 19:12:16 9.42.171.39 NOTICE mgmt: admin(admin) login from host 9.44.168.83 en Enable privilege granted. Router#configure terminal Enter configuration commands, one per line. End with Ctrl/Z. Router(config)#vnic enable
57
Router(config)#
You can disable vNIC mode using isCLI with the command no vnic enable as shown in Example 4-2.
Example 4-2 Disabling vNIC mode Router(config)#no vnic enable Router(config)#
Note: When you have successfully enabled or disabled vNIC mode using isCLI, no message is displayed.
Using BladeOS
Enabling vNIC mode using BladeOS is shown in Figure 4-17 and Figure 4-18 on page 59. 1. Navigate to /cfg/virt/vnic to display the Main Menu. 2. Enter cfg for the Configuration menu and press Enter (Figure 4-17).
[Main Menu] Jan 15 16:56:14 9.44.168.155 info stats cfg oper boot maint diff apply save revert exit >> Main# Jan 15 16:56:33 9.42. 171.253 cfg
9.42.171.39 NOTICE
mgmt: admin(admin) login from host
Information Menu Statistics Menu Configuration Menu Operations Command Menu Boot Options Menu Maintenance Menu Show pending config changes [global command] Apply pending config changes [global command] Save updated config to FLASH [global command] Revert pending or applied changes [global command] Exit [global command, always available] 9.42.171.39 WARNING ntp: cannot contact primary NTP server
Figure 4-17 The Configuration menu
3. Enter virt for the Virtualization menu and press Enter (Figure 4-18 on page 59).
58
[Configuration Menu] sys - System-wide Parameter Menu port - Port Menu qos - QOS Menu acl - Access Control List Menu pmirr - Port Mirroring Menu l2 - Layer 2 Menu l3 - Layer 3 Menu cee - CEE Configuration Menu fcoe - Fiber Channel Over Ethernet Configuration Menu rmon - RMON Menu virt - Virtualization Menu setup - Step by step configuration set up dump - Dump current configuration to script file ptcfg - Backup current configuration to FTP/TFTP server gtcfg - Restore current configuration from FTP/TFTP server cur - Display current configuration
Figure 4-18 The Virtualization menu
4. Enter vmpolicy for the Virtual Machines policy Configuration menu and press Enter (Figure 4-19 on page 60). 5. Enter on to enable vNIC functionality.
59
>> Configuration# virt -----------------------------------------------------------[Virtualization Menu] vmpolicy - Virtual Machines Policy Configuration Menu vnic - vNIC Configuration Menu vmgroup - Virtual Machines Groups Menu vmprof - Virtual Machine Profiles Menu vmware - VMware-specific Settings Menu enavmr - Enable VMready disvmr - Disable VMready cur - Display all current virtualization settings >> Virtualization# vnic -----------------------------------------------------------[VNIC Global Configuration Menu] port - Port vNIC Configuration Menu vnicgrp - VNIC Group Configuration Menu on - Globally turn vNIC feature ON off - Globally turn vNIC feature OFF cur - Display current vNIC configuration >> VNIC Global Configuration# on Current status: ON New status: ON >> VNIC Global Configuration# Figure 4-19 Using BladeOS Cli to enable vNIC
4.3.3 Creating the vNICs

In this section we describe how to create vNICs. In most installations, you will likely have two BNT Virtual Fabric 10Gb Switch Modules in your chassis, so this process needs to be done twice, once on each of the two switches. The same commands are used on both switches if the two physical ports are to be configured identically, especially if trunk failover is to be used for high availability. When issuing the commands, remember that: Blade numbers (internal port numbers) are 1 through 14, and vNIC numbers are 1 through 4. Bandwidth is specified in 100-MB increments, which also specifies the percentage of 10-Gb bandwidth allocated to a specific vNIC.
60
Using BBI
Using BBI, the configuration process is carried out using the Configuration tab, as shown in Figure 4-20. Select the vNIC, set the State to Enabled and set the Maximum bandwidth (in our case, we entered 30 for 3 Gbps).
Figure 4-20 Creating a vNIC
Using isCLI:
Using isCLI the command syntax is as follows: vnic port <port alias or number> index <1-4> bandwidth <1-100> enable or no enable For example, we want to do the following configuration tasks using isCLI: 1. 2. 3. 4. Enable vNIC # 8 on blade 8 Allocate 30% of the 10 Gb to this vNIC (3 Gb) Enable vNIC #2 on blade 8 Allocate 70% of the 10 Gb to this vNIC (7 Gb)
The commands to do these tasks and the output we got are shown in Figure 4-21 on page 62.
61
Jan 9 17:02:38 9.42.171.39 NOTICE mgmt: admin(admin) login from host 9.44.168.202 en Enable privilege granted. Router#configure terminal Enter configuration commands, one per line. End with Ctrl/Z. Router(config)#vnic port 8 index 1 Router(vnic_config)#band Router(vnic_config)#bandwidth 30 Router(vnic_config)#enable Router(config)#vnic port 8 index 2 Router(vnic_config)#bandwidth 70 Router(vnic_config)#enable Router(vnic_config)# Figure 4-21 Using isCLI to create a vNIC
Using BladeOS
Using BladeOS, the relevant commands are as follows: port <port number> -- Displays the port vNIC menu. vnic <1-4> -- Displays the vNIC menu for the selected vNIC. bw <1-100> -- Configures the maximum bandwidth allocated to this vNIC, in increments of 100 Mbps. For example, 1 = 100 Mbps, 10 = 1000 Mbps. ena -- enables the selected vNIC. dis -- disables the selected vNIC. cur -- displays the current vNIC port parameters. The commands we issued in our example and the output we got are shown in Figure 4-22 on page 63.
62
> Virtualization# vnic --------------------------------------------------------[VNIC Global Configuration Menu] port - Port vNIC Configuration Menu vnicgrp - VNIC Group Configuration Menu on - Globally turn vNIC feature ON off - Globally turn vNIC feature OFF cur - Display current vNIC configuration >> VNIC Global Configuration# port Enter port (INT1-INT14): 4 --------------------------------------------------------[Port INT4 vNICs Menu] vnic - VNIC Configuration Menu cur - Display current port vNIC configuration >> Port INT4 vNICs# vnic Enter vNIC (1-4): 2 --------------------------------------------------------[vNIC INT4.2 Menu] bw - Set maximum bandwidth of the vNIC ena - Enable vNIC dis - Disable vNIC cur - Display current vNIC configuration >> vNIC INT4.2# ena Current status: disabled New status: enabled Warning: "Tagging" is enabled on vNIC port INT4 vNIC INT4.2# bw 20 Current vNIC Maximum Bandwidth: 25 (increments of 100Mbps) Pending vNIC Maximum Bandwidth: 20 (increments of 100Mbps) Figure 4-22 Using BladeOS Cli to configure vNIC
4.4 vNIC group configuration

A vNIC group consists of one or more vNICs, (optionally) one or more uplink (EXTernal) ports, and (optionally) one or more non-vNIC server blade-facing (INTernal) ports. For a vNIC to communicate, it must be a member of a single vNIC group, and only one vNIC from a given port can be in any specific group. The overall steps for configuring a vNIC group are as follows. 1. Enable or disable a group. Up to 32 groups can be created. 2. Add a vNIC to the group. 3. Add a physical port to the group.
63
4. Add an external trunk group to the vNIC group. 5. Enable or disable trunk failover, as discussed in 4.7, High availability and vNICs on page 70. The commands for these steps are: /cfg/virt/vnic/vnicgrp /cfg/virt/vnic/vnicgrp /cfg/virt/vnic/vnicgrp /cfg/virt/vnic/vnicgrp /cfg/virt/vnic/vnicgrp <number>/{ena or dis} <number>/addvnic <blade#.vnic#> <number>/addport <INT or EXT port> <number>/addtrnk <trunk #> <number>/failover {ena or dis}
Tip: The remvnic, remport, and remtrnk commands can be used to remove the corresponding items from the group. The commands to add a vNIC and a trunk to a vNIC group are shown in Figure 4-23 on page 65.
64
>> Virtualization# vnic -----------------------------------------------------------[VNIC Global Configuration Menu] port - Port vNIC Configuration Menu vnicgrp - VNIC Group Configuration Menu on - Globally turn vNIC feature ON off - Globally turn vNIC feature OFF cur - Display current vNIC configuration >> VNIC Global Configuration# vnicgrp Enter vNIC Group (1-32): 8 -----------------------------------------------------------[vNIC Group 8 Menu] vnicvlan - Set VLAN number to vNIC group failover - Enable/disable uplink failover addvnic - Add vNIC to vNIC group remvnic - Remove vNIC from vNIC group addport - Add port to vNIC group remport - Remove port from vNIC group addtrnk - Add trunk to vNIC group remtrnk - Remove trunk from vNIC group ena - Enable vNIC group dis - Disable vNIC group del - Delete vNIC group cur - Display current vNIC group configuration >> vNIC Group 8# addvnic INT4.2 >> vNIC Group 8# addtrnk 3 >> vNIC Group 8# ena Current status: disabled New status: enabled >> vNIC Group 8# Figure 4-23 Using BladeOS Cli to create vNIC Group
The isCLI equivalent is: vnic vnicgroup <number 1-32> member INT<port>.<vnic number 1-4> port INT<port number or alias> or EXT <port number or alias> vlan <vlan number 1-4094> trunk <trunk number> failover enable exit The exit command is required to end the configuration for a specific group.
65
Note: The no member, no port, no trunk, no vlan, and no failover commands can be used to remove the corresponding items from the group. Example commands are shown in Figure 4-24.
Router>Jan 10 17:20:24 9.42.171.39 NOTICE mgmt: admin(admin) login from host 9.44.168.202 en Enable privilege granted. Router#configure terminal Enter configuration commands, one per line. End with Ctrl/Z. Router(config)#vnic vnicgroup 2 Router(vnic group config)#member INT8.1 Router(vnic group config)#vlan 4091 Warning: VLAN 4091 is moved to STG 1. Router(vnic group config)#failover Router(vnic group config)#trunk 2 Router(vnic group config)#enable Warning: STP is turned off for port EXT1 in STG 1 Warning: Changed the pvid of uplink port EXT1 in vNIC group 2 to 4091 Warning: Deleted port EXT1 from VLAN 1 Router(vnic group config)# Figure 4-24 Creating vNIC group
Currently, only one uplink port or one trunk group of uplink ports aggregated together can be part of a given vNIC group. Any uplink port or trunk can only be part of one vNIC group at a time. This means that it is not currently possible for multiple groups to share uplinks. At present, only static trunk groups are supported. Link Aggregation Control Protocol (LACP) trunks are supported in the Switch Independent Mode (see Part 2, Switch Independent Mode on page 99). Each vNIC group is associated with a vNIC VLAN, as discussed in Configuring vNIC VLANs on page 66.
4.5 Configuring vNIC VLANs

Every vNIC channel is associated with a vNIC group with a unique VLAN ID. These group VLANs differ from conventional VLANs that are configured on the Virtual Fabric switch in significant ways: vNIC VLANs are implemented as outer tags on the Ethernet frames that pass through the switch. These outer tags are never forwarded to a blade or to an
66
upstream switch. They serve solely to isolate traffic in a vNIC group from other vNIC groups and from conventional VLANs. Within a group, conventional VLANs configured by an operating system on a server blade (such as with the vconfig command in Linux) are implemented as inner tags. These inner tags pass through the group and the Virtual Fabric switch completely unchanged. The VLAN configuration on upstream switches needs to match that on the blades, but the Virtual Fabric switch simply passes the inner tags through (Figure 4-25).
802.1q "Double Tagging" and the BNT Virtual Fabric Switch

Upstream Router BNT Virtual Fabric Switch HS22 Blade
Payload
Payload
Payload
Ethernet Header EtherType (of payload) .1Q VLAN tag VLAN ID "A" EtherType = 0x8100 Ethernet Header SA DA
Tagged traffic for multiple VLANs is sent from the upstream switch. Upon entry to the Virtual Fabric switch, the vnicvlan value is inserted as the second tag (ID "B").
Ethernet Header EtherType (of payload) .1Q VLAN tag 2 VLAN ID "A" EtherType = 0x8100 .1Q VLAN tag 1 VLAN ID "B" EtherType = 0x8100 Ethernet Header SA DA
Traffic flows through the Virtual Fabric switch using the vnicvlan tag and is forwarded to the blade-facing port, where the vnicvlan "outer" tag is removed.
Ethernet Header EtherType (of payload) .1Q VLAN tag VLAN ID "A" EtherType = 0x8100 Ethernet Header SA DA
Figure 4-25 Double tagging
The same conventional VLAN can enter the chassis in more than one group, but traffic between groups never occurs within the Virtual Fabric switch. On the Virtual Fabric switch, a given VLAN number can be a conventional VLAN, a vNIC group, or a VMready group, but not more than one of these at the same time.
67
Additional notes about vNIC VLANs: Routing of traffic between vNIC VLANs is not supported, nor is routing between a vNIC VLAN and a conventional VLAN. When EXTernal (uplink) ports are added to a vNIC group, they are also added to the vNIC VLAN, and they are always aggregated together. At present, there is no way to split these ports and aggregate them in more than one trunk. When INTernal (blade-facing) ports are added to a vNIC group, it is assumed that there will be no vNICs created on those blades. (This is with the addport command, not the addvnic command.) Traffic can then flow between vNICs and these INTernal ports. It will be forwarded using the vNIC VLAN as the outer tag. Any inner tags associated with the non-VNIC interface on the ports will be unchanged, just as with the vNICs. vNIC VLANs are configured with the associated vNIC groups. The isCLI commands are: vnic vnicgroup <number 1-32> vlan <vlan number 1-4094> To configure vNIC VLANs, the BladeOS commands are: /cfg/virt/vnic/vnicgrp <number> vnicvlan <1-4094> The selected VLAN number cannot be used within the Virtual Fabric switch for any other purpose. However, it is of purely local significance in that it is never seen by the blades (other than by the vNIC driver), and never forwarded out of the Virtual Fabric switch to an upstream switch. Use high numbers that are not used in the clients network as vNIC VLANs (for example, 4000 and up).
68
Figure 4-26 shows the BBI window to configure a vNIC group and the associated VLAN.
Figure 4-26 Configuring the vNIC group and the associated VLAN
4.6 vNIC and VMready

Versions of BNT Virtual Fabric 10Gb Switch Module firmware prior to 6.7.2 did not support a blade-facing port being in a vNIC group and a VMready group at the same time. However, with the release of 6.7.2, this is now possible.
69
With versions prior to 6.7.2, it is possible, however, to use vNIC to support certain blades and VMready on other blades using the same switch and within the same chassis, just not the same port.
4.7 High availability and vNICs

The trunk failover function enables high-availability designs when used with standard ports. This failover function is also available when vNICs are used. With vNICs, if the uplink ports that are members of a group (vnicgrp) fail due to a failure on the upstream device or a cable failing or being erroneously removed, the associated vNIC is brought down by the Virtual Fabric switch. In Windows, this appears as a network cable unplugged condition on the server. To exploit the trunk failover function to deliver high availability, take the following steps: 1. The other Virtual Fabric switch in the chassis must also be configured to use vNICs for the server in question. It is a good idea, but not required, that the same vNIC numbers, associated groups, and bandwidth allocation be used on both switches. 2. The corresponding vNIC ports, as seen by the operating system, must be teamed together using the available tools for NIC teaming. In general, NIC teaming must be performed in active/standby mode, using the following specifications: For Windows servers, use the available utility from Emulex. Linux servers must have the native bonding driver. ESX has support for multiple NIC teaming modes using the network configuration GUI. Multiple vNICs that are part of the same physical 10-Gb port can support high availability, but each vNIC does so independently. In order to implement this, do the following: 1. Enable failover for all vNIC groups for which the server ports are members. Do so on both of the Virtual Fabric switches. 2. Configure NIC teaming (as above) on corresponding pairs of vNICs on the server. Make sure that one member of the team is physically connected to one of the Virtual Fabric switches, and the other member of the team to the other Virtual Fabric switch. 3. Configure uplink ports or trunks as members of the corresponding vNIC groups on the two switches. It is good practice to connect these uplinks to separate upstream physical switches, if possible.
70
Figure 4-27 is a sample configuration showing only one blade. The same configuration commands are required for each blade used for which failover is desired.
High Availability with vNIC

Bay 7 10-port switch Switch bay Port INT3
vNIC 3.1
vNIC 3.2
Blade #3 vNIC 3.1 vNIC 3.2
Router
Port INT3
Legends: RED vNIC Group 1, VLAN 2001 (outer tag) vNIC 3.1 and EXT10 on each switch GREEN vNIC Group 2, VLAN 2002 (outer tag) vNIC 3.2 and EXT9 on each switch BLACK Physical port INT3 on each switch
Bay 9 10-port switch
Figure 4-27 High availability with vNIC
Using isCLI, the commands for the blade configuration in Figure 4-27 are: vnic port INT3 index 1 enable vnic port INT3 index 2 enable vnic vnicgroup 1 failover vlan 2001 member INT3.1 port EXT10 enable exit
71
vnic vnicgroup 2 failover vlan 2002 member INT3.2 port EXT9 enable exit The BladeOS configuration for Figure 4-27 is identical on each of the two switches and includes: /cfg/virt/vnic /port 3/vnic 1/ena /cfg/virt/vnic/port 3/vnic 2/ena /cfg/virt/vnic/vnicgrp 1/ena failover ena vnicvlan 2001 addvnic 3.1 addport EXT10 /cfg/virt/vnic/vnicgrp 2/ena failover ena vnicvlan 2002 addvnic 3.2 addport EXT9 For more information, refer to BNT Virtual Fabric 10Gb Switch Module for IBM BladeCenter Application Guide, available from: http://ibm.com/support/entry/portal/docdisplay?lndocid=MIGR-5080917
4.8 Configuring the operating system

In this section, we show how to configure teams and VLANs with the Emulex 10GbE Virtual Fabric Adapter. The following operating systems support the Emulex 10GbE Virtual Fabric Adapter: Microsoft Windows Server 2003 (x86 and x64) Microsoft Windows Server 2008 (x86 and x64) Microsoft Windows Server 2008 Release 2 (x64 only) Red Hat Enterprise Linux 5.5 and 5.6 Red Hat Enterprise Linux 6.0 and 6.1 SUSE Linux Enterprise Server 10.3 and 10.4
72
SUSE Linux Enterprise Server 11.1 VMware ESX 4.0, 4.1, and 5.0 Citrix XenServer 5.6 Citrix XenServer 6.0 Note: Before installing the device drivers and firmware, make sure the following firmware for other BladeCenter devices is at the latest level. OneCommand Manager is the Emulex application that enables the centralized management of Emulex OneConnect Converged Network Adapters (CNAs) and Host Bus Adapters (HBAs) from a centralized management console. OneCommand Manager provides a GUI and a scriptable CLI for administration. You can use the OneCommand Manager to configure load balancing and teaming.
4.8.1 Configuring teams and VLANs

A team (or bond in Linux) of adapters functions as a single virtual network interface and appears the same as a nonteamed adapter to other network devices. The benefits of teaming are: Virtual adapters can balance a large network load across several adapters. Teams also improve system availability by providing a failover path for critical network connections. A protocol address, such as an IP address, is usually assigned to the physical adapter. However, when OneCommand NIC Teaming and Multiple VLAN Manager (OneCommand NIC Teaming Manager) is installed, the protocol address is assigned to the team adapter and not to the physical adapters that make up the team. The IPCONFIG /all command for Windows or IFCONFIG command for Linux shows the IP and MAC addresses of the virtual adapter and not of the individual physical adapters.
73
4.8.2 Configuring a new team

To configure a new team, start the OneCommand NIC Teaming Manager application and proceed as follows. 1. Use the Create Team button to create a new team configuration (Figure 4-28).
Figure 4-28 Configuring a new team using OneCommand NIC Teaming Manager
74
Figure 4-29 Configuring a new team
2. Enter a name of the new team in the Team Name field (Figure 4-29). 3. Choose a Team Type from the drop-down list. The default is Failover. 4. If necessary, change the Team Type criteria for Load Distributed By to one of the following: Default: Failover. Selects the port with the least traffic load for the session. Destination MAC Address: Performs an XOR on the destination MAC address to determine which port should carry the load. Destination IP Address: Performs an XOR on the destination and source IP address to determine which port should carry the load. 5. Enable or disable auto failback functionality. The default is enabled.
75
6. The Available Network Adapters area lists the adapters that are not members of any team. To add an adapter to the team, select the adapter from this list, and click Add to display the adapter in the Team Member Adapters list. 7. The Team Member Adapters area lists the adapters that are members of the team. 8. Click OK. With the team created, a window displays as in Figure 4-29, showing that the adapter has been added to the team.
4.8.3 Configuring a VLAN for a physical or team adapter

You can configure VLANs at either the physical adapter level or at the team level. Configuring the VLAN at both the physical and team levels may cause double tagging and should be avoided. As a result, all virtual adapter members of the team should have the VLAN disabled at the physical or operating system level. To configure a VLAN for either a physical or team adapter, do the following using the OneConnect software: 1. Select the physical or team adapter in the Team and Available Adapters window (Figure 4-30 on page 77) to which you want to add a VLAN, and click VLAN.
76
Figure 4-30 Configuring a new team
2. In the NIC Teaming and VLAN Manager dialog box that appears, Figure 4-31 on page 78, enter a valid VLAN ID in the range 1-4094, for example 50.
77
Figure 4-31 Configuring a VLAN
3. The VLAN Name shows the VLAN Name in the format Vlan_<VLAN ID>. Click Add to add the VLAN to the adapter, as shown in Figure 4-31. Note: You can create multiple VLANs for an adapter. The VLANs Configured list shows the list of all VLANs configured for the adapter.
78
Figure 4-32 Confirmation of configured VLAN
4. Click OK to complete the configuration. The VLAN is now created.
79
80
Chapter 5.
Examples of use - IBM Virtual Fabric Mode

In this chapter, we describe the steps to create three separate virtual fabric configurations, comprising combinations of: Virtual network interface controllers (vNICs) Uplink ports Uplink trunks All of the configuration performed here is done using interfaces into the BNT Virtual Fabric 10Gb Switch Module. Tip: You also need to ensure that vNIC mode is enabled in the Emulex Virtual Fabric Adapters. vNIC mode is the default. The topics in this chapter are: 5.1, Example 1: vNICs only, no uplinks on page 82 5.2, Example 2: vNICs only, one uplink on page 86 5.3, Example 3: vNICs only, one uplink trunk on page 91 Tip: For the Virtual Fabric Adapter, the PCIe function ID can be determined by running the ethtool -i command in Linux or by viewing the Network Adapter Properties in Windows Device Manager.
81
5.1 Example 1: vNICs only, no uplinks

Example 1 creates group VGRP 1, which consists solely of vNIC ports. The configuration is (Figure 5-1 on page 83): vNIC INT1.2 (blade in slot 1, vNIC 2) of the switch in I/O bay 7, set to a maximum bandwidth of 5 Gbps (50% of 10 Gbps) vNIC INT2.1 (blade 2, vNIC 1) of the switch in I/O bay 7, set to a maximum bandwidth of 2 Gbps (20% of 10 Gbps) One vNIC group containing these two vNICs (vNIC Group VLAN 127) The BladeOS command script in Example 5-1 can be used to implement this configuration.
Example 5-1 BladeOS script to create Scenario 1 /c/virt/vnic on /c/virt/vnic/port INT1/vnic 2 ena bw 50 /c/virt/vnic/port INT2/vnic 1 ena bw 20 /c/virt/vnic/vnicgrp 1 ena vnicvlan 127 addvnic INT1.2 addvnic INT2.1 apply
The isCLI command script in Example 5-2 can be used to implement this configuration.
Example 5-2 isCLI script to create Scenario 1 vnic enable vnic port INT1 index 2 enable bandwidth 50 vnic port INT2 index 1 enable bandwidth 20 vnic vnicgroup 1 enable vlan 127 member 1.2
82
member 2.1 exit
The Virtual Fabric Switch browser-based interface (BBI) can also be used to implement this configuration, as follows: 1. Click the Configure tab of the BBI interface. 2. Enable vNIC operation by expanding Virtualization VNIC General, and then select On and click Submit Apply Save. Tip: Click Submit Apply Save after every change to ensure that the setting persists after any switch reboot.
Figure 5-1 Example 1: Enable vNIC operations on the vNIC Configuration panel
Chapter 5. Examples of use - IBM Virtual Fabric Mode
83
3. Enable and configure each of the vNICs by selecting Virtualization VNICs (Figure 5-2).
Figure 5-2 Example 1: vNICs Configuration panel
4. For the first vNIC, select Enabled, and then enter the bandwidth allocation (Figure 5-3).
Figure 5-3 Example 1: Enabling vNIC INT1.2
84
5. For the second vNIC, select Enabled, and then enter the bandwidth allocation (Figure 5-4).
Figure 5-4 Example1: Enabling vNIC INT2.1
6. Enable and configure the vNIC group by clicking Virtualization VNIC VNIC Groups (Figure 5-5).
Figure 5-5 Example 1: vNIC Groups Configuration panel
7. Click vNIC Group 1 and Figure 5-6 on page 86 displays: a. b. c. d. Enable the group. Set the VLAN to 127. Select the two vNICs. Click Add.
85
Figure 5-6 Example 1: vNIC Group 1 Configuration panel
5.2 Example 2: vNICs only, one uplink

Example 2 creates group VGRP 3, which consists of vNIC ports and one uplink port. The configuration is (Figure 5-7 on page 88): vNIC INT1.3 (blade in slot 1, vNIC 3) of the switch in I/O bay 7, set to a maximum bandwidth of 2.5 Gbps (25% of 10 Gbps) vNIC INT2.2 (blade 2, vNIC 2) of the switch in I/O bay 7, set to a maximum bandwidth of 7 Gbps (70% of 10 Gbps) One vNIC group containing these vNICs (vNIC Group VLAN 500) One uplink port, EXT2 The BladeOS command script in Example 5-3 can be used to implement this configuration.
Example 5-3 BladeOS script for Example 2 /c/virt/vnic on /c/virt/vnic/port INT1/vnic 3 ena bw 25 /c/virt/vnic/port INT2/vnic 2
86
ena bw 70 /c/virt/vnic/vnicgrp 3 ena vnicvlan 500 addvnic INT1.3 addvnic INT2.2? addport EXT2 apply
Example 5-4 isCLI script for Example 2 vnic enable vnic port INT1 index 3 enable bandwidth 25 (default) vnic port INT2 index 2 enable bandwidth 70 vnic vnicgroup 3 enable vlan 500 member 1.3 member 2.2 port EXT2 exit
87
The Virtual Fabric Switch BBI can also be used to implement this configuration, as follows. 1. Click the Configure tab of the BBI interface. 2. Enable vNIC operations by expanding Virtualization VNIC General, and then select On and click Submit Apply Save. Tip: Click Submit Apply Save after every change to ensure that the setting persists after any switch reboot.
88
3. Enable and configure each vNIC by clicking Virtualization VNICs (Figure 5-8).
4. For the first vNIC, select Enabled, and then enter the bandwidth allocation (Figure 5-9).
Figure 5-9 Example 3: vNIC INT1.3 Configuration panel
89
5. For the second vNIC, select Enabled, and then enter the bandwidth allocation (Figure 5-10).
6. Enable and configure the vNIC group by selecting Virtualization VNIC VNIC Groups (Figure 5-11).
7. Click vNIC Group 3 and Figure 5-12 on page 91 displays: a. b. c. d. e. Enable the group. Set the VLAN to 500. Select the EXT2 uplink port. Select the vNICs. Click Add.
90
5.3 Example 3: vNICs only, one uplink trunk

Example 3 creates group VGRP 5, which consists of vNIC ports and one uplink trunk. The configuration is (Figure 5-13 on page 93): vNIC INT1.1 (blade in slot 1, vNIC 1) of the switch in I/O bay 9, set to a maximum bandwidth of 3 Gbps (30% of 10 Gbps) vNIC INT2.2 (blade 2, vNIC 2) of the switch in I/O bay 9, set to a maximum bandwidth of 3.3 Gbps (33% of 10 Gbps) One vNIC group containing these two vNICs (vNIC Group VLAN 925) One uplink trunk comprising external ports EXT1 and EXT3 The BladeOS command script in Example 5-5 can be used to implement this configuration.
Example 5-5 BladeOS script for Example 3 /c/l2/trunk 1 ena add EXT1 add EXT3 /c/virt/vnic on
91
/c/virt/vnic/port INT1/vnic 1 ena bw 30 /c/virt/vnic/port INT2/vnic 2 ena bw 33 /c/virt/vnic/vnicgrp 5 ena vnicvlan 925 addvnic INT1.1 addvnic INT2.2 addtrnk 1 apply
Example 5-6 isCLI script for Example 3 vnic enable portchannel 1 port EXT1 portchannel 1 port EXT3 portchannel 1 enable vnic port INT1 index 1 enable bandwidth 30 vnic port INT2 index 2 enable bandwidth 33 vnic vnicgroup 5 enable vlan 925 member 1.1 member 2.2 trunk 1 exit
92
The Virtual Fabric Switch BBI can also be used to implement this configuration as follows. 1. Click the Configure tab of the BBI interface. 2. Enable vNIC operation by expanding Virtualization VNIC General, and then select On and click Submit Apply Save. Tip: Click Submit Apply Save after every change to ensure that the setting persists after any switch reboot.
93
3. Enable and configure the vNICs by selecting Virtualization VNICs (Figure 5-14).
4. For the first vNIC, select Enabled, and then type the bandwidth allocation (Figure 5-15).
94
5. For the second vNIC, select Enabled, and then type the bandwidth allocation (Figure 5-16).
6. Enable the trunk group by clicking Layer 2 Trunk Groups and Trunk Group 1 (Figure 5-17).
Figure 5-17 Example 5: Trunk Groups Configuration panel
7. Enable the trunk group, select EXT1 and EXT3, and click Add (Figure 5-18 on page 96).
95
Figure 5-18 Example 5: Switch Trunk Group 1 Configuration panel
8. Enable and configure the vNIC group by clicking Virtualization VNIC VNIC Groups (Figure 5-19).
9. Click vNIC Group 5 and Figure 5-20 on page 97 displays: a. Enable the group.
96
b. c. d. e.
Set the VLAN to 925. Select the Trunk 1 uplink trunk. Select the vNICs. Click Add.
97
98
Part 2
Part
Switch Independent Mode

The chapters in Part 2 cover the product details and implementation for the Switch Independent Mode offering. Chapters in this part are: Chapter 6, Solution architecture - Switch Independent Mode on page 101 Chapter 7, Supported products - Switch Independent Mode on page 109 Chapter 8, Configuring the solution - Switch Independent Mode on page 133 Chapter 9, Examples of use - Switch Independent Mode on page 193
99
100
Chapter 6.
Solution architecture Switch Independent Mode

In Switch Independent Mode, the virtual channel separation on the shared 10-Gb link is maintained in both the Virtual Fabric adapter (VFA) and the switch. The adapter NICs enforce bandwidth limits for each virtual channel. However, in Switch Independent Mode, the switches do not enforce bandwidth limits to the servers, but will instead rely upon link level flow control to throttle traffic when necessary. The full features of the BNT and Cisco switches are still available when operating in Switch Independent Mode. In this mode, you will not require an outer VLAN tag to maintain channel separation. Instead, the mode uses the MAC address or the MAC address in combination with the IEEE 802.1Q VLAN tag to distinguish the individual vNIC channels on each link. This places a requirement on you and your network administrator to coordinate the configuration of the VFAs and the chassis switches to match the existing VLAN structure in your network. Figure 6-1 on page 102 shows an example of an implementation of Switch Independent Mode and the VLAN tagging that is employed.
101
Switch Independent Mode Overview

Customer VLAN tags maintain Layer 2 separation within switch
Emulex or Brdcm 10G VFA Customer VLAN tag 10G Link
VLAN 10
VLAN 10
VLAN 20 Emulex or Brdcm 10G VFA 10G Link
VLAN 20
VLAN 30
VLAN 30
Note: VLAN tag denotes customer VLAN (required)

Figure 6-1 Switch Independent Mode overview
The Emulex 10GbE Virtual Fabric Adapter II allows your network administrator to configure one VLAN ID per virtual channel interface. Up to 31 additional VLANs can be configured within the applications stack (VMware, and so on) for each virtual channel. These VLANs are then incorporated along with the MAC address or addresses on inbound frames to forward traffic to the correct interface. The Broadcom 2-port 10Gb Virtual Fabric Adapter includes one MAC address per virtual channel. You can assign up to 16 MAC addresses to each vNIC partition, for a total of up to 64 MAC addresses per each physical port. It is possible to allocate separate bandwidth allocations to each vNIC partition. The vNIC partition bandwidth allocation allows for fine tuning and division of available physical port bandwidth between the vNICs. Topics in this chapter are: 6.1, Solution overview 6.2, Features on page 105
102
6.1 Solution overview

Table 6-1 shows the possible configurations that support Switch Independent Mode.
Table 6-1 Supported combinations using Switch Independent Mode Supported switches Supported adapters Emulex 10GbE Virtual Fabric Adapter, 49Y4235 Emulex 10GbE Virtual Fabric Adapter Advanced, 49Y4275 Emulex 10GbE Virtual Fabric Adapter II, 90Y3550 Emulex 10GbE Virtual Fabric Adapter II Advanced, 90Y3566 Broadcom 2-port 10Gb Virtual Fabric Adapter, 81Y3133 BNT Virtual Fabric 10Gb Switch Module Not supported Not supported Supported 4Q/2011a Supported 4Q/2011a Supported Cisco Nexus 4001I Switch Module Not supported Not supported Supported 4Q/2011a Supported 4Q/2011a Supported
Figure 6-1 provides a general view of the vNIC solution architecture. Different colors represent different virtual port groups to isolate traffic flows for different services and applications, and the thickness of each link represents the amount of assigned bandwidth for a particular traffic flow. Switch Independent Mode provides the capability to subdivide a single 10-Gb link into up to four virtual channels.
Chapter 6. Solution architecture - Switch Independent Mode
103
I/O Bay 1
I/O Bay 2
I/O Bay 4
Components and connectivity inside the IBM BladeCenter H chassis
Cisco Virtual Fabric Switch (Bay 7)
Blade Server 14 Blade Server 1

FC Expansion Card Virtual Fabric Adapter
vNIC1 vNIC2 vNIC3 vNIC4 vNIC1 vNIC2 vNIC3 vNIC4
10Gb port
VLAN1
I/O Bay 3
EXT1 EXT2 EXT3 EXT4 EXT5 EXT6 EXT1 EXT2 EXT3 EXT4 EXT5 EXT6
On-Board 1GbE CIOv
VLAN2
VLAN3
10Gb port
VLAN4
CFFh
Cisco Virtual Fabric Switch (Bay 9)

10Gb port VLAN1
Legend
Standard I/O bays connections vNIC virtual connections External physical trunks vSwitch virtual trunk connections Mid-Plane
10Gb port
VLAN2
VLAN3
VLAN4
Figure 6-2 BladeCenter Virtual Fabric vNIC solution diagram with Cisco switch
Tip: An uplink can support multiple VLANs (not shown in Figure 6-2). vNICs are configured in the operating system using the Emulex or Broadcom software in the pre-boot environment. The VFA configuration and setup is performed inside the network controller rather than the switch. Each physical interface of a VFA can be divided into up to four vNICs. You should also create a corresponding VLAN within the chassis switch for each vNIC associated with a switch port. For example, a VFA port with four vNICs defined should correspond to four VLANs on the switch port. The switch VLANs should correspond to the VLANs that are defined throughout your network infrastructure. The vNIC interfaces become extensions of these network-wide VLANs.
104
You must take several considerations into account when planning to use the components in Switch Independent Mode: The Virtual Fabric adapter must operate in vNIC2 mode (Emulex) or in NIC Partition mode (Broadcom). Up to four vNICs can be configured on each physical NIC (pNIC) on the VFA (a total of eight vNICs on a two-port VFA). The bandwidth for each vNIC can be assigned in a range of 100 Mb to 10 Gb in 100-Mb increments (the default is 2.5 Gb) The total bandwidth of the four vNICs per port cannot exceed 10 Gbps, however, the Broadcom 2-port 10Gb Virtual Fabric Adapter supports oversubscription, whereby the aggregate bandwidth can be as high as 4x10 Gb allowing any of the vNICs to consume as much as 10 Gb where there is no contention or less when there is (based on the weight assigned to that vNIC). The sum of the bandwidth allocated for all four vNICs on the same pNIC cannot exceed 10 Gb. Support by BladeCenter Open Fabric Manager is planned but was not available at the time of writing. Tip: All parameters for vNIC2 and NIC Partition are configured on the Virtual Fabric adapter.
6.2 Features
The Emulex 10GbE Virtual Fabric Adapter II and Broadcom 2-port 10Gb Virtual Fabric Adapter are both high-performance, dual-port network adapters for 10 Gbps Ethernet (10 GbE) networks having the following general characteristics: Each server pNIC port is divided into up to four vNICs. The operating system configurations see eight unique NICs (2 ports x 4 vNICs). Figure 6-3 on page 106 shows how a configured Broadcom 2-port 10Gb Virtual Fabric Adapter appears to Windows Server 2008.
105
Figure 6-3 Showing NICs in network connections - Windows Server 2008
All vNIC parameters are configured from the VFA user interface. The user enables vNICs and allocates bandwidth: The allowable vNIC bandwidth range is from 100 Mbps to 10 Gbps: Bandwidth increments are in 100 Mbps. The default bandwidth is 2.5 Gbps. For the Emulex adapter, the sum of all four vNIC bandwidths cannot exceed 10 Gbps. For the Broadcom adapter, the sum of all four allocated vNIC bandwidths can exceed 10 Gbps because the adapter supports oversubscription. The total actual consumed vNIC bandwidths cannot exceed 10 Gbps.
106
Server-to-VFS bandwidth metering is on a per-vNIC basis. Figure 6-4 shows an example of a possible configuration.
Figure 6-4 Setting bandwidth
The user associates vNICs to VLAN groups within the switch by configuring the switch independent from the VFA. Optionally, switch uplinks can be assigned to VLAN groups: VLAN groups within the switch serve to isolate virtual NIC traffic flowing on the same physical port: Existing VLANs within the client network are not impacted. No forwarding occurs between uplinks assigned to VLAN groups. An uplink (port or trunk) can belong to multiple vNIC groups. A server port (pNIC or vNIC) can be associated with one or more VLAN groups.
The failover mechanism for switch-independent mode is not virtual port-aware.
107
Key concurrency rules are: vNIC and VMready VMready is a unique offering that enables the switches within the network to be virtual machine-aware. The network switches can be configured and managed for virtual ports (vports), rather than just for physical ports. Switch-independent mode will allow you to enable vNIC mode on the Emulex 10GbE Virtual Fabric Adapter II and implement VMready in the switch for that server at the same time. Further details are in Chapter 7, Supported products - Switch Independent Mode on page 109. vNIC and switch stacking A switch stack is a group of BNT Virtual Fabric 10Gb Switch Modules that work together as a unified system and can be managed as a single entity. The network views the stack as a single switch entity, and the stack can be accessed and managed as a whole using standard switch IP interfaces. Once the stacking links have been established, the number of ports available in a stack equals the total number of remaining ports of all the switches that are part of the stack. The number of available IP interfaces, VLANs, trunks, trunk links, and other switch attributes are not aggregated among the switches in a stack. The totals for the stack as a whole are the same as for any single switch configured in stand-alone mode. The BNT VFS supports stacking, with either of the vNIC modes at the same time. vNIC and iSCSI You can enable vNIC mode on the VFA and use iSCSI on that same adapter.
108
Chapter 7.
Supported products - Switch Independent Mode

In this chapter, we describe the products used in Switch Independent Mode. This mode consists of the supported combinations of the following adapters, switch modules, and chassis types: Virtual Fabric adapters Emulex 10GbE Virtual Fabric Adapter II Emulex 10GbE Virtual Fabric Adapter II Advanced Broadcom 2-port 10-Gb Virtual Fabric Adapter Notes: Support for Switch Independent Mode by the Emulex 10GbE Virtual Fabric Adapter II will require a firmware upgrade, which is expected in 4Q/2011 The Emulex 10GbE Virtual Fabric Adapter, 49Y4235, and Emulex 10GbE Virtual Fabric Adapter Advanced, 49Y4275, are not supported in the Switch Independent Mode offering. Use the Emulex 10GbE Virtual Fabric Adapter II or II Advanced, 90Y3550 or 90Y3566. 10-Gb Ethernet switch modules BNT Virtual Fabric 10Gb Switch Module
109
Cisco Nexus 4001I Switch Module BladeCenter H or HT chassis Table 7-1 lists the specific combinations that are supported.
Table 7-1 Supported combinations using Switch Independent Mode Supported switches Supported adapters Emulex 10GbE Virtual Fabric Adapter, 49Y4235 Emulex 10GbE Virtual Fabric Adapter Advanced, 49Y4275 Emulex 10GbE Virtual Fabric Adapter II, 90Y3550 Emulex 10GbE Virtual Fabric Adapter II Advanced, 90Y3566 Broadcom 2-port 10-Gb Virtual Fabric Adapter, 81Y3133 BNT Virtual Fabric 10Gb Switch Module Not supported Not supported Supported 4Q/2011a Supported 4Q/2011a Supported Cisco Nexus 4001I Switch Module Not supported Not supported Supported 4Q/2011a Supported 4Q/2011a Supported
These product features are described in the following sections. The topics covered in this chapter are: 7.1, Virtual Fabric adapters on page 110 7.2, Virtual Fabric switch modules on page 116
7.1 Virtual Fabric adapters

The Switch Independent Mode offering supports the use of the Emulex 10GbE Virtual Fabric Adapter II and the Broadcom 2-port 10-Gb Virtual Fabric Adapter. In this section, we discuss these two adapters.

The Emulex 10GbE Virtual Fabric Adapter II, part number 90Y3550, is a dual-port, 10-Gb Ethernet card that supports 1-Gbps or 10-Gbps traffic, or up to eight virtual NIC (vNIC) devices. The vNICs are configured to meet your mix of network connectivity and throughput demands in today's complex server application environments. Each physical 10-Gbps port can be divided into four virtual ports, with bandwidth allocation in 100 Mbps increments, to a maximum of
110
10 Gbps per physical port. The adapter is a CFFh expansion card. Therefore, it requires that switch modules are installed in I/O bays 7 and 9. The Emulex Virtual Fabric Adapter Advanced II, part number 90Y3566, is a separate adapter based on the same hardware, with additional support for Fibre Channel over Ethernet (FCoE) and Internet Small Computer System Interface (iSCSI) hardware initiator functionality, in addition to the features available on a standard card. Also available is the Emulex Virtual Fabric Adapter Advanced Upgrade, part number 49Y4265, which enables the Emulex 10GbE Virtual Fabric Adapter II to be upgraded to support the FCoE and iSCSI capabilities of the Advanced adapter. The Emulex 10GbE Virtual Fabric Adapter II is shown in Figure 7-1.
,
Figure 7-1 Emulex 10GbE Virtual Fabric Adapter II (CFFh)
Table 7-2 lists the Emulex Virtual Fabric Adapter (CFFh) ordering part numbers and feature codes.
Table 7-2 Ordering part numbers and feature codes Description Emulex Virtual Fabric Adapter II Emulex Virtual Fabric Adapter Advanced II \ Emulex Virtual Fabric Adapter Advanced Upgrade Part number 90Y3550 90Y3566 49Y4265 Feature code A1XG AIXH 2436
Chapter 7. Supported products - Switch Independent Mode
111
The Emulex 10GbE Virtual Fabric Adapter II has the following features: Connection to either 1-Gb or 10-Gb data center infrastructure (1-Gb and 10-Gb auto-negotiation) PCI Express 2.0 x8 host interface IBM BladeCenter CFFh form factor, which can be combined with a CIOv expansion card on the same blade server Operates either as an 8-port virtual NIC or as a 2-port, 1/10-Gb Ethernet adapter: Virtual port bandwidth allocation in 100-Mbps increments Supports up to eight virtual ports Wake On LAN support Full-duplex (FDX) capability Bus-mastering support Direct memory access (DMA) support Preboot Execution Environment (PXE) support IPv4/IPv6 TCP, UDP checksum offload Large send offload (LSO) Large receive offload Receive side scaling (RSS) IPv4 TCP Chimney Offload
VLAN insertion and extraction Jumbo frames up to 9000 bytes Load balancing and failover support, including adapter fault tolerance (AFT), switch fault tolerance (SFT), adaptive load balancing (ALB), teaming support, and IEEE 802.3ad Enhanced Ethernet (draft) Enhanced Transmission Selection (ETS) (P802.1Qaz) Priority-based Flow Control (PFC) (P802.1Qbb) Data Center Bridging Capabilities eXchange (DCBX) protocol, CIN-DCBX and Converged Enhanced Ethernet- (CEE-) DCBX (P802.1Qaz) The Emulex Virtual Fabric Adapter Advanced II provides FCoE and iSCSI hardware initiator support. You can configure the adapter to operate as two physical iSCSI ports. The adapter supports Serial over LAN functionality. The adapter supports both virtual fabric modes: IBM Virtual Fabric Mode and Switch Independent Mode.
112
Note: Support for Switch Independent Mode by the Emulex 10GbE Virtual Fabric Adapter II will require a firmware upgrade, which is expected in 4Q/2011. The original Emulex 10GbE Virtual Fabric Adapter can be configured in pNIC or vNIC mode. The new Emulex 10GbE Virtual Fabric Adapter II now offers two vNIC modes of operation: IBM Virtual Fabric Mode (sometimes referred to as vNIC1), which is the same as vNIC mode on original Emulex VFA adapters. This mode is used when configuring for the IBM Virtual Fabric Mode offering. We also call this mode switch-dependent mode, because the Emulex adapters rely on the switch module to obtain vNIC parameters (via DCBX). This mode only works in conjunction with BNT Virtual Fabric 10Gb Switch Module. Switch Independent Mode (sometimes referred to as vNIC2), which does not rely on any vNIC settings being passed to the adapter from the switch module. In this mode, the Emulex 10GbE Virtual Fabric Adapter II can work with these switches: BNT Virtual Fabric 10Gb Switch Module Cisco Nexus 4001I Switch Module
113
Figure 7-2 shows the placement of Emulex 10GbE Virtual Fabric Adapter II in an HS22 blade server. It is a high-speed expansion card, so you need to install it in the CFFh expansion slot.
Virtual Fabric Adapter

Figure 7-2 Placement of a CFFh card in the BladeCenter HS22 server
7.1.2 Broadcom 2-port 10-Gb Virtual Fabric Adapter

The Broadcom 2-port 10-Gb Virtual Fabric Adapter, part number 81Y3133, is a dual-port 10Gbps high-speed expansion card designed to operate in Switch Independent Mode. The adapter is shown in Figure 7-3 on page 115.
114
Figure 7-3 Broadcom 2-port 10-Gb Virtual Fabric Adapter
You can configure up to four partitions (vNICs) on each of the two 10-Gbps physical ports. vNIC partitioning is based on MAC addresses. In contrast with the Emulex 10GbE Virtual Fabric Adapter II, the Broadcom 2-port 10-Gb Virtual Fabric Adapter does not use VLAN IDs for vNIC partitioning. You can assign up to 16 MAC addresses to each vNIC partition, for a total of up to 64 MAC addresses per each physical port. It is possible to dynamically allocate bandwidth to each vNIC partition. vNIC partition bandwidth allocation allows for fine-tuning and division of available physical port bandwidth between the vNICs. Each enabled partition appears as a dedicated network card to the operating system. Broadcom 2-port 10-Gb Virtual Fabric Adapter features are as follows: Integrated dual 10-Gbps MAC and dual 10GBASE-CX4 PCI Express x8 2.0, 5 GT/s compliant IEEE 802.3ap Clause 73 compliant backplane operation IEEE 802.3xx Clause 37 compliant auto-negotiation for 1Gbps TCP/IP Offload Engine (TOE) Microsoft TCP chimney compliant Full FASTPATH TCP offload for IPv4 and IPv6 Network Controller Sideband Interface (NC-SI) PXE v2.1 remote boot Wake-on-LAN (WOL) Virtual LANsIEEE 802.1q VLAN tagging IEEE 802.3x flow control
115
The Broadcom 2-port 10-Gb Virtual Fabric Adapter form factor is CFFh, therefore it communicates with high-speed I/O modules installed in I/O bays 7 and 9. Figure 7-2 on page 114 shows placement of the adapter in an HS22 blade.
7.2 Virtual Fabric switch modules

Because Switch Independent Mode is based on the vNIC configuration capabilities of the Virtual Fabric adapters, it does not depend on the switch module model used. Currently, two switches are supported: BNT Virtual Fabric 10Gb Switch Module Cisco Nexus 4001I Switch Module
7.2.1 BNT Virtual Fabric 10Gb Switch Module

The BNT Virtual Fabric 10Gb Switch Module for IBM BladeCenter offers the most bandwidth of any blade switch and represents the perfect migration platform for clients who are still at 1 Gb outside the chassis by seamlessly integrating into the existing 1-Gb infrastructure. This is the first 10-Gb switch for IBM BladeCenter that supports converged networking (that is, is able to transmit Converged Enhanced Ethernet (CEE) to an FCoE-capable, top-of-rack switch). This feature is available with firmware release 6.1 and higher. CEE and FCoE functionality enables the clients to transfer storage, network, Voice Over Internet Protocol (VoIP), video, and other data over the common Ethernet infrastructure. With the use of QLogic Virtual Fabric Extension Module, clients can achieve FCoE gateway functionality inside the BladeCenter chassis. In this publication, we concentrate on virtual fabric. The BNT switch module can be used both in IBM Virtual Fabric Mode and Switch Independent Mode offerings. In an IBM Virtual Fabric Mode solution, all vNIC parameter configuration is performed on the BNT Virtual Fabric 10Gb Switch Module. Therefore, the switch module is a key part of the offering. This offering enables clients to form eight virtual network interface controllers (vNICs) from one physical NIC (pNIC) and to manage them in virtual groups. The switch module can be managed using a command-line interface (CLI) or a web browser interface and, in the future, with BladeCenter Open Fabric Manager (BOFM), provide all the benefits of I/O virtualization at 10-Gb speeds. If you have a chassis with multiple servers, several operating at 1 Gbps and several at 10 Gbps, and several transmitting converged packets, this single
116
switch can handle all of these workloads and can connect to a 1-Gb infrastructure or a 10-Gb infrastructure, or both. With the extreme flexibility of the BNT switch, you can take advantage of the technologies that they require for multiple environments. For 1-Gbps uplinks, they can take advantage of small form-factor pluggable (SFP) transceivers. For 10-Gbps uplinks, they have a choice of either enhanced small form-factor pluggable (SFP+) transceivers (short-range or long-range) for longer distances, or direct-attached copper (DAC) cables (also known as twinax active cables) for shorter distances. DAC cables are more cost-effective, consume less power, and can be up to 7 meters in length. They are ideal for connecting chassis together, connecting to a top-of-rack (TOR) switch, or even connecting to an adjacent rack. Figure 7-4 on page 117 shows the switch module.
Figure 7-4 BNT Virtual Fabric 10Gb Switch Module
Table 7-3 lists the part number to use to order the module.
Table 7-3 Part number and feature code Description BNT Virtual Fabric 10Gb Switch Module Part number 46C7191 Feature code 1639
The part number includes the following items: One BNT Virtual Fabric 10Gb Switch Module Three-meter, mini-USB-to-DB9 serial console cable One filler module BNT Virtual Fabric 10Gb Switch Module Installation Guide BNT user license agreement Important Notices document
117
Documentation CD-ROM Tip: Enhanced small form-factor pluggable plus (SFP+) transceivers are not included and must be purchased separately. To communicate outside of the chassis, you must have either SFP+ transceivers or SFP+ direct-attach copper (DAC) cables connected. DAC cables have SFP+ transceivers on both ends. You have the flexibility to expand your bandwidth as desired, using from one to ten connections per switch. Table 7-4 lists the part number to use to order the SFP+ transceivers, FC cables, and DAC cables.
Table 7-4 IBM part numbers for ordering SFP+ transceivers, FC cables, and DAC cables Description 10Gb SFP+ IBM 10GBase-SR 10GbE 850 nm Fiber SFP+ transceiver BNT SFP+ transceiver 1Gb SFP+ BLADE 1000BASE-T (RJ45) SFP transceiver BLADE 1000BASE-SX SFP transceiver DAC cables 0.5 m Molex DAC SFP+ cable 1 m Molex DAC SFP+ cable 3 m Molex DAC SFP+ cable 7 m Molex DAC SFP+ cable FC cables 3 m Intel Connects optical cable 10 m Intel Connects optical cable 30 m Intel Connects optical cable 46D0153 46D0156 46D0159 3852 3853 3854 59Y1932 59Y1936 59Y1940 59Y1944 3735 3736 3737 3738 81Y1618 81Y1622 3268 3269 44W4408 46C3447 4942 5053 Part number Feature code
Table 7-5 on page 119 lists additional transceivers and direct-attach cable options that are available directly from BNT.
118
Table 7-5 BNT part numbers for ordering SFP-based transceivers and cables Description BLADE 1000Base-T SFP BLADE 1000Base-SX SFP BLADE 10GBase-LR SFP+ SFP+ DAC - 1 M SFP+ DAC - 3 M SFP+ DAC - 7 M Part number BN-CKM-S-T BN-CKM-S-SX BN-CKM-SP-LR BN-SP-CBL-1M BN-SP-CBL-3M BN-SP-CBL-7M
The BNT Virtual Fabric 10Gb Switch Module includes the following features and functions: Form-factor Single-height, high-speed switch module Internal ports Fourteen internal, auto-negotiating ports: 1 Gbps or 10 Gbps to the server blades Two internal full-duplex 100-Mbps ports connected to the management module External ports Up to ten 10-Gb SFP+ ports (also designed to support 1-Gb SFP, if required; flexibility of mixing 1 Gb and 10 Gb) Oversubscription ratio (14 internal ports to 10 external ports) is very low, and this makes the switch module suitable for the most performance-intensive environments. One 10/100/1000-Mb copper RJ45 used for management or data An RS-232 mini-USB connector for serial port that provides an additional means to install software and configure the switch module Scalability and performance Autosensing 1-Gb and 10-Gb internal and external Ethernet ports for bandwidth optimization Non-blocking architecture with wire-speed forwarding of traffic and full line rate performance of 480-Gbps full duplex Media access control (MAC) address learning: automatic update, supports up to 32-Kb MAC addresses
119
Up to 128 IP interfaces per switch Static, EtherChannel, and Link Aggregation Control Protocol (LACP) (IEEE 802.3ad) link aggregation, up to 100 Gb of total bandwidth per switch, up to 18 trunk groups, and up to eight ports per group Support for jumbo frames (up to 12288 bytes) Broadcast/multicast storm control Internet Group Management Protocol (IGMP) snooping for limit flooding of IP multicast traffic (IGMP V1, V2, and V3) Configurable traffic distribution schemes over trunk links, based on source and destination IP addresses, MAC addresses, or both Fast port forwarding and fast uplink convergence for rapid Spanning Tree Protocol (STP) convergence Stacking support: you can stack up to eight BNT Virtual Fabric 10Gb Switch Modules Availability and redundancy Virtual Router Redundancy Protocol (VRRP) for Layer 3 router redundancy IEEE 802.1D STP for providing Layer 2 redundancy with PVRST+ IEEE 802.1s Multiple STP (MSTP) for topology optimization, up to 128 STP instances supported by single switch IEEE 802.1w Rapid STP (RSTP), providing rapid STP convergence for critical, delay-sensitive, traffic-like voice or video Layer 2 trunk failover to support active and standby configurations of network adapter teaming on blades Interchassis redundancy (Layer 2 and Layer 3) VLAN support Up to 1024 VLANs supported per switch, VLAN numbers ranging from 1 to 4095 (4095 is a dedicated VLAN used for the management module connection only) 802.1Q VLAN tagging support on all ports Protocol-based VLANs Security VLAN-based, MAC-based, and IP-based access control lists (ACLs) 802.1X port-based authentication Multiple user IDs and passwords
120
User access control Radius, Terminal Access Controller Access-Control System Plus (TACACS+), Lightweight Directory Access Protocol (LDAP) Quality of service (QoS) Up to eight queues per port Support for IEEE 802.1p, IP ToS/DSCP, and ACL-based (MAC and IP source and destination addresses, VLANs) traffic classification and processing Traffic shaping and remarking based on defined policies Eight Weighted Round Robin (WRR) priority queues per port for processing qualified traffic Layer 3 functions IP forwarding IP filtering with ACLs (up to 4096 ACLs supported) VRRP for router redundancy Support for up to 128 static routes Routing protocol support (Router Information Protocol (RIP) V1, RIP V2, Open Shortest Path First protocol (OSPF) V1, V2, and V3, BGP-4), up to 1024 entries in routing table IPv6 routing, including static routes and OSPFv3 (requires firmware V6.3 or higher) Support for Dynamic Host Configuration Protocol (DHCP) Relay IPv6 host management IPv6 forwarding based on static routes Manageability Simple Network Management Protocol (SNMP V1, V2, and V3) HTTP/HTTPS Browser-Based Interface (BBI) Industry standard CLI (isCLI) and BLADEOS or AlteonOS CLI Telnet interface for CLI SSH v1/v2 Serial interface for CLI Scriptable CLI Firmware image update (Trivial File Transfer Protocol (TFTP) and File Transfer Protocol (FTP))
121
Network Time Protocol (NTP) for switch clock synchronization BNT BLADEHarmony Manager support Monitoring Switch LEDs for external port status and switch module status indication Port mirroring for analyzing network traffic passing through switch Change tracking and remote logging with syslog feature Power-On Self Test (POST) diagnostics
Special functions Serial over LAN (SOL) Virtualization features VMready Virtual Fabric Adapter vNIC support Converged Enhanced Ethernet and FCoE features FCoE allows Fibre Channel traffic to be transported over Ethernet links. FCoE Initialization Protocol (FIP) snooping to enforce point-to-point links for FCoE traffic outside the regular Fibre Channel topology. Priority-Based Flow Control (PFC) (IEEE 802.1Qbb) extends 802.3x standard flow control to allow the switch to pause traffic based on the 802.1p priority value in each packets VLAN tag. Enhanced Transmission Selection (ETS) (IEEE 802.1Qaz) provides a method for allocating link bandwidth based on the 802.1p priority value in each packets VLAN tag. Data Center Bridging Capability Exchange Protocol (DCBX) (IEEE 802.1AB) allows neighboring network devices to exchange information about their capabilities. Supports the QLogic Virtual Fabric Extension Module for IBM BladeCenter, which provides FCoE gateway functionality inside the BladeCenter Chassis. VMready is a unique solution that enables the network to be virtual machine-aware. The network can be configured and managed for virtual ports (v-ports), rather than just for physical ports. With VMready, as VMs migrate across physical hosts, so do their network attributes. Virtual machines can be added, moved, and removed while retaining the same ACLs, QoS, and VLAN attributes. VMready allows for a define-once-use-many configuration that evolves as the server and network topologies evolve. VMready works with all virtualization products, including VMware, Hyper-V, Xen, and KVM, without
122
modification of virtual machine hypervisors or guest operating systems. It is available as part of the 6.1 (and higher) software code. VMready compatibility with Virtual Fabric solutions is as follows: VMready is not supported with IBM Virtual Fabric Mode VMready is supported with Switch Independent Mode The switch module supports the following IEEE standards: IEEE 802.1D STP with PVRST+ IEEE 802.1s MSTP IEEE 802.1w RSTP IEEE 802.1p Tagged Packets IEEE 802.1Q Tagged VLAN (frame tagging on all ports when VLANs are enabled) IEEE 802.1x port-based authentication IEEE 802.2 Logical Link Control IEEE 802.3ad Link Aggregation Control Protocol IEEE 802.3x Full-duplex Flow Control IEEE 802.3ab 1000BASE-T Gigabit Ethernet IEEE 802.3ae 10GBASE-SR 10Gb Ethernet fiber optics short range IEEE 802.3ae 10GBASE-LR 10Gb Ethernet fiber optics long range IEEE 802.3z 1000BASE-SX Gigabit Ethernet The following network cables are supported for the BNT Virtual Fabric 10Gb Switch Module: 10GBASE-SR for 10-Gb ports: 850Nm wavelength, multimode fiber, 50 or 62.5 (300 meters maximum), with LC duplex connector 1000BASE-T for RJ45 port: UTP Category 6 (100 meters maximum) UTP Category 5e (100 meters maximum) UTP Category 5 (100 meters maximum) EIA/TIA-568B 100-ohm STP (100 meters maximum)
More information is available in the following publications: BNT Virtual Fabric 10Gb Switch Module At-a-Glance Guide available at: http://www.redbooks.ibm.com/abstracts/tips0708.html
123
Documents in PDF format available at: http://www.ibm.com/support/docview.wss?uid=psg1MIGR-5080917 BNT Virtual Fabric 10Gb Switch Module Installation Guide Application Guide Command Reference Industry standard command line interface Reference (isCLI Reference) BBI (Browser-based Interface) Quick Guide Release Notes End User License Agreement
7.2.2 Cisco Nexus 4001I Switch Module

The Cisco Nexus 4001I Switch Module is a blade switch solution for the BladeCenter H and HT chassis, providing the server I/O solution required for high-performance, scale-out, virtualized, and non-virtualized x86 computing architectures. It is a line rate, extremely low-latency, non-blocking, Layer 2, 10-Gb Ethernet switch module that is fully compliant with FCoE and IEEE Data Center Bridging standards. The Cisco Nexus 4001I Switch Module enables a standards-based, high-performance Unified Fabric running over 10Gb Ethernet in the blade server environment. This Unified Fabric enables consolidation of LAN traffic, storage traffic (IP-based, such as iSCSI, network-attached storage (NAS), Fibre Channel SAN), and high-performance computing (HPC) traffic over a single 10-Gb Ethernet server network. This offering works with BladeCenter Open Fabric Manager (BOFM), providing all of the benefits of I/O virtualization at 10Gbps speed. Figure 7-5 shows the switch module.
Figure 7-5 Cisco Nexus 4001I Switch Module
124
Table 7-6 shows the part numbers to order these modules and additional options for them.
Table 7-6 Cisco Nexus 4001I Switch Module and related options part numbers and feature codes Description Cisco Nexus 4001I Switch Module for IBM BladeCenter Software upgrade license for the Cisco Nexus 4001I Switch Module Part number 46M6071 49Y9983 Feature code 0072 1744 Cisco part number N4K-4001I-XPX N4K-4001I-SSK9
The module part numbers include the following items: One Cisco Nexus 4001I Switch Module Cisco Console Cable RJ45-to-DB9 One filler panel Important Notices document Documentation CD-ROM
Software Upgrade License for the Cisco Nexus 4001I Switch Module
The Cisco Nexus 4001I Switch Module is designed to support both 10-Gb Ethernet and FCoE. The software upgrade license for the Cisco Nexus 4001I Switch Module, part number 49Y9983, enables the switch to work in FCoE mode. When connected to a converged adapter in the server, this switch can route CEE packets to an upstream FCoE switch, which can then route the packets to the LAN or SAN.
SFP+ transceivers and copper cables

The Cisco Nexus 4001I Switch Module does not ship with either SFP+ transceivers or SFP+ copper cables. These can be ordered from IBM, directly from Cisco Systems, or from authorized Cisco Systems resellers, as listed in Table 7-7.
Table 7-7 SFP+ transceivers and copper cables for the Cisco Nexus 4001I Switch Module Description IBM part number IBM feature code Cisco Systems part number
10Gb Ethernet SFP+ transceiver Cisco 10GBASE-SR SFP+ (MMF, 850-nm, LC) Cisco 10GBASE-LR SFP+ (SMF, 1310-nm, LC) 88Y6054 None A1A6 None SFP-10G-SR(=) SFP-10G-LR(=)
125
Description
IBM part number
IBM feature code
Cisco Systems part number
DAC Cables Cisco 1-m 10G SFP+ Twinax cable assembly, passive Cisco 3-m 10G SFP+ Twinax cable assembly, passive Cisco 5-m 10G SFP+ Twinax cable assembly, passive Gigabit Ethernet SFP+ Cisco 1000BASE-T SFP transceiver (Cat. 5 copper wire, RJ-45) Cisco 1000BASE-SX SFP transceiver (MMF, 850-nm, LC) Cisco 1000BASE-LX/LH SFP transceiver (MMF/SMF, 1300-nm, LC) 88Y6058 88Y6062 None A1A7 A1A8 None GLC-T(=) GLC-SX-MM(=) GLC-LH-SM(=) None None None None None None SFP-H10GB-CU1M(=) SFP-H10GB-CU3M(=) SFP-H10GB-CU5M(=)
The Cisco Nexus 4001I Switch Module includes the following features and functions: Form-factor Single-height high-speed switch module External ports Six 10-Gb SFP+ ports operating at wire speed. Also designed to support 1Gb SFP if required, with the flexibility of mixing 1 Gb and 10 Gb. Table 7-7 on page 125 lists supported transceivers and cables. One 10/100/1000-Mb Ethernet copper RJ45 used for management. An RS-232 to RJ45 connector for a serial port that provides an additional means to configure the switch module. The console cable is supplied with the switch module. Internal ports Fourteen internal auto-negotiating ports: 1 Gbps or 10 Gbps to the server blades Two internal full-duplex 100-Mbps ports connected to the management modules Scalability and performance Autosensing 1-Gb/10-Gb internal and external Ethernet ports for bandwidth optimization
126
Non-blocking architecture with wire-speed forwarding of traffic and full line rate performance of 400-Gbps full duplex Forwarding rate of 300 million packets per second (mpps). Low, predictable, and consistent latency of 1.5 microseconds regardless of packet size, traffic pattern, or enabled features on 10GbE interface MAC address learning: automatic update, supports up to 8 Kb MAC addresses EtherChannels and LACP (IEEE 802.3ad) link aggregation, up to 60Gbps of total uplink bandwidth per switch, up to seven trunk groups, and up to six ports per group Support for jumbo frames (up to 9216 bytes) Traffic suppression (unicast, multicast, and broadcast) IGMP snooping to limit flooding of IP multicast traffic (IGMP V2 and V3) Configurable traffic distribution schemes over EtherChannel links based on source/destination IP addresses, MAC addresses, or ports Spanning Tree edge ports (formerly PortFast) for rapid STP convergence Availability and redundancy Link state tracking to mirror the state of the external ports on the internal Ethernet links and to allow the failover of the processor blade traffic to an operational external link on a separate Cisco Ethernet switch. IEEE 802.1D STP for redundant backbone connections and loop-free networks. IEEE 802.1s MSTP for grouping VLANs into a spanning-tree instance, and provided for multiple forwarding paths for data traffic and load balancing. IEEE 802.1w Rapid STP (RSTP) for rapid convergence of the spanning tree by immediately transitioning root and designated ports to the converting state. Configurable Unidirectional link detection (UDLD) for detecting and disabling unidirectional links. This feature prevents a larger network failure in the event that a unidirectional link is detected, thus reducing downtime in these situations. VLAN support Up to 512 VLANs supported per switch; VLAN numbers ranging from 1 to 4000 802.1Q VLAN tagging support on all ports Private VLANs
127
Security VLAN-based, MAC-based, and IP-based access control lists (ACLs) Role-based access control (RBAC) to restrict the authorization of the user to perform switch management functions TACACS+, a proprietary feature for managing network security through a TACACS server RADIUS for verifying the identity of, granting access to, and tracking activities of remote users Quality of service Support for IEEE 802.1p CoS, IP ToS/DSCP, Protocol, IP Real Time Protocol, and ACL-based (MAC/IP source and destination addresses, VLANs) traffic classification and processing Trust boundaries on incoming interfaces to automatically classify incoming packets into system classes based on packet CoS value Traffic shaping, guaranteeing bandwidth, or prioritizing based on defined policies Up to eight egress queues per interface (one queue per CoS) for processing qualified traffic (support for strict priority and weighted round-robin (WRR) CoS policies) Fibre Channel over Ethernet Support for T11-compliant FCoE on all 10-Gigabit Ethernet interfaces. FCoE Initialization Protocol (FIP) snooping to enable the switch to operate as a loss-less Ethernet bridge transparently forwarding FCoE packets. 802.1Q VLAN tagging for FCoE frames. Priority-based flow control (IEEE 802.1Qbb) simplifies management of multiple traffic flows over a single network link and creates lossless behavior for Ethernet by allowing class-of-service (CoS)-based flow control. Enhanced Transmission Selection (IEEE 802.1Qaz) enables consistent management of QoS at the network level by providing consistent scheduling of different traffic types (IP, storage, and so on). DCBX Protocol (IEEE 802.1AB) simplifies network deployment and reduces configuration errors by providing auto-negotiation of IEEE 802.1 DCB features between the NIC and the switch and between switches.
128
IPv6 support IPv6 host support (IPv6 unicast addressing, IPv6 traffic processing, IPv6 applications support: syslog server, RADIUS, TACACS+, NTP, telnet, and ssh). IPv6 traffic forwarding at Layer 3 is not supported. Manageability Command line interface: You can configure switches using the CLI from an SSH V2 session, a Telnet session, or the console port. SSH provides a secure connection to the device. XML Management Interface over secure shell (SSH): You can configure switches using the XML management interface, which is a programming interface based on the NETCONF protocol that complements the CLI functionality. For more information see the Cisco NX-OS XML Interfaces User Guide. Cisco Data Center Manager support. SNMP V1, 2, and 3 support. Protected Mode feature to isolate switch management from the Advanced Management Module for increased security of the switch. Cisco Discovery Protocol (CDP) Versions 1 and 2 to aid in troubleshooting and reporting the misconfiguration of ports connecting to other devices supporting CDP. Monitoring Switch LEDs for external port status and switch module status indication RMON Change tracking and remote logging with syslog feature Online diagnostics Cisco Fabric Services Session Manager SPAN for local traffic monitoring
Special functions Serial over LAN (SOL) The switch module supports the following IEEE standards: IEEE 802.1D: Spanning Tree Protocol IEEE 802.1p: CoS Prioritization IEEE 802.1Q: VLAN Tagging IEEE 802.1s: Multiple VLAN Instances of Spanning Tree Protocol IEEE 802.1w: Rapid Reconfiguration of Spanning Tree Protocol IEEE 802.3ab: 1000Base-T (10/100/1000 Ethernet over copper) IEEE 802.3ad: Link Aggregation Control Protocol (LACP)
129
IEEE 802.3ae 10Gb Ethernet over fiber optics 10GBASE-SR 10GBASE-LR IEEE 802.3z Gigabit Ethernet over fiber optics 1000BASE-SX 1000BASE-LX IEEE 802.3ab 1000BASE-T over copper twisted pair Network cable requirements: 10GBASE-SR Table 7-8 lists the 10GBASE-SR cabling specifications.
Table 7-8 10GBASE-SR cabling specifications Wavelength 850 nm 850 nm 850 nm 850 nm 850 nm Cable type MMF MMF MMF MMF MMF Core size (microns) 62.5 62.5 50 50 50 Modal bandwidth (MHz/km) 160 200 400 500 2000 Maximum cable length 85 feet (26 m) 108 feet (33 m) 217 feet (66 m) 269 feet (82 m) 984 feet (300 m)
10GBASE-LR Table 7-9 lists the 10GBASE-LR cabling specifications.

Table 7-9 10GBASE-LR cabling specifications Wavelength 1310 nm Cable type SMF Core size (microns) G.652 Modal bandwidth (MHz/km) Not applicable Maximum cable length 6.2 miles (10 km)
1000BASE-SX Table 7-10 lists the 1000BASE-SX cabling specifications.

Table 7-10 1000BASE-SX cabling specifications Wavelength 850 nm 850 nm Cable type MMF MMF Core size (microns) 62.5 62.5 Modal bandwidth (MHz/km) 160 200 Maximum cable length 722 feet (220 m) 902 feet (275 m)
130
Wavelength 850 nm 850 nm
Cable type MMF MMF
Core size (microns) 50 50
Modal bandwidth (MHz/km) 400 500
Maximum cable length 1640 feet (500 m) 1804 feet (550 m)
1000BASE-LX Table 7-11 lists the 1000BASE-LX cabling specifications.

Table 7-11 1000BASE-LX cabling specifications Wavelength 1310 nm 1310 nm 1310 nm 1310 nm Cable type MMF MMF MMF SMF Core size (microns) 62.5 50.0 50.0 G.652 Modal bandwidth (MHz/km) 500 400 500 Not applicable Maximum cable length 1804 feet (550 m) 1804 feet (550 m) 1804 feet (550 m) 6.2 miles (10 km)
1000BASE-T UTP Category 6 (100 meters maximum) UTP Category 5e (100 meters maximum) UTP Category 5 (100 meters maximum) EIA/TIA-568B 100-ohm STP (100 meters maximum)
See the following documentation for more information: Configuration Guide for the Cisco Nexus 4001I Switch Module Hardware Installation Guide for the Cisco Nexus 4001I Switch Module Getting Started Guide for the Cisco Nexus 4001I Switch Module Command Reference Guide for the Cisco Nexus 4001I Switch Module These publications can be downloaded at: http://ibm.com/support/entry/portal/docdisplay?lndocid=MIGR-5082494 In addition, IBM Redbooks publication Cisco Nexus 4001I Switch Module for IBM BladeCenter, TIPS0754, contains useful information about the Cisco Nexus 4001I Switch Module: http://www.redbooks.ibm.com/abstracts/tips0754.html
131
132
Chapter 8.
Configuring the solution Switch Independent Mode

In this chapter, we describe how to configure components in Switch Independent Mode. When configuring for the IBM Virtual Fabric Mode solution, as described in Part 1, IBM Virtual Fabric Mode on page 9, most of the configuration steps are performed on the BNT Virtual Fabric 10Gb Switch Module. The switch module transfers configuration settings to the Emulex 10GbE Virtual Fabric Adapter using Data Center Bridging eXchange (DCBX). The Emulex 10GbE Virtual Fabric Adapter requires very little configuring; only that virtual network interface card (vNIC) mode needs to be enabled using the blade server Unified Extensible Firmware Interface (UEFI) utility. In contrast, Switch Independent Mode requires that you set vNIC parameters for the adapter using the blade server UEFI utility. In addition, you need to configure the switch module to match the virtual local area network (VLAN) ID configuration on the vNICs and the extended network infrastructure. The topics covered in this chapter are: 8.1, Configuring Virtual Fabric adapters on page 135 8.2, Configuring the BNT Virtual Fabric 10Gb Switch Module on page 146 8.3, Cisco Nexus 4001I Switch Module configuration on page 156 8.4, Creating virtual NICs on page 160
133
8.5, Configuring vNIC VLANs on page 167 8.6, vNIC and VMready on page 169 8.7, High availability and vNICs on page 169 8.8, Configuring the operating systems on page 169
134
8.1 Configuring Virtual Fabric adapters

In this section, we discuss firmware upgrade and adapter configuration in UEFI for both the Emulex 10GbE Virtual Fabric Adapter II and the Broadcom 2-port 10-Gb Virtual Fabric Adapter.
8.1.1 Using BOFM Advanced V4.1

BladeCenter Open Fabric Manager (BOFM) Advanced Version 4.1 is planned to support configuration of vNIC parameters on VFAs. The following vNIC parameters are due to be supported by BOFM: MAC address of virtual NIC VLAN ID Minimum bandwidth Maximum bandwidth Priority At the time of writing, details about BOFM Advanced V4.1 were not yet available. The next revision of this book will include detailed descriptions of vNIC parameter configuration using BOFM.
8.1.2 Firmware upgrade of Emulex 10GbE Virtual Fabric Adapter II

Upgrade the firmware on the Emulex 10GbE Virtual Fabric Adapter II to the latest level before starting this setup procedure. Note: Support for Switch Independent Mode by the Emulex 10GbE Virtual Fabric Adapter II will require a firmware upgrade, which is expected in 4Q/2011 You can upgrade firmware in a number of ways: Windows Server 2008, 2008 R2 and 2003 online update Windows PE offline update Linux online code update Linux offline code update VMware code update package Code update using a bootable DVD media UEFI code update These methods are well documented in the readme files. We show an example of upgrading the Emulex 10GbE Virtual Fabric Adapter II firmware with the
Chapter 8. Configuring the solution - Switch Independent Mode
135
Windows online package. The package uses an executable file, elxflash.exe, to perform the code update. Elxflash can have many parameters to set in order to carry out the code update in the desired way. These parameters are explained in the readme file. Figure 8-1 shows an example of updating the Emulex 10GbE Virtual Fabric Adapter II firmware update with the elxflash /update command.
>elxflash /update Thu May 19 15:41:27 2011 HBA=OCe11100-NIC, Port Type=NIC, MAC=00-00-C9-B2-57-02, Update=Firmware, Image=oc11-4.0.200.10.ufi, New=4.0.200.10, Old=3.103.351.10, Status=Success Thu May 19 15:41:27 2011 HBA=OCe11100-NIC, Port Type=NIC, MAC=00-00-C9-B2-57-06, Update=Firmware, Image=oc11-4.0.200.10.ufi, New=4.0.200.10, Old=3.103.351.10, Status=Success (lines deleted for clarity) Thu May 19 15:41:28 2011 HBA=OCe11100-NIC, Port Type=NIC, MAC=00-00-C9-B2-57-09, Update=Firmware, Image=oc11-4.0.200.10.ufi, New=4.0.200.10, Old=3.103.351.10, Status=Success elxflash: All required updates succeeded - Return Code=0 Figure 8-1 Windows online code update (elxflash executable)
The output in Figure 8-1 is truncated for clarity. Running the command produces eight sections; however, only three sections are shown in Figure 8-1. Each section represents one of the vNICs.
8.1.3 Configuring the Emulex 10GbE Virtual Fabric Adapter II in UEFI

To configure the Emulex 10GbE Virtual Fabric Adapter II using the UEFI interface, do the following: 1. Power on or restart the blade. 2. When prompted during POST, press F1, and the UEFI utility main menu displays (Figure 8-2 on page 137).
136
Figure 8-2 Blade UEFI utility - main menu
3. Select System Settings, then select Network on the subsequent panel, to display the Network panel (Figure 8-3).
Figure 8-3 Network configuration in the blade UEFI utility
137
The Network window shows two Emulex 10GbE Virtual Fabric Adapter II ports, which are the two physical ports present on the adapter. 4. Select one of the ports to configure, and the Emulex NIC Selection panel displays (Figure 8-4) 5. Enable Multichannel To enable vNIC or vNIC2 mode, you must first enable Multichannel. Unless Multichannel is enabled, the card will operate as two physical 10-Gbps Ethernet ports. When you enable Multichannel, it will be enabled on both 10-Gb physical ports. After enabling Multichannel, you can now select vNIC or vNIC2 mode: vNIC enables IBM Virtual Fabric Mode. vNIC2 enables Switch Independent Mode. We are configuring for the Switch Independent Mode solution. Therefore, we need to select vNIC2 mode. Next, we look at the settings that relate to vNICs (Figure 8-4).
Figure 8-4 Emulex vNIC settings
6. Select Multichannel Configuration to display the Function Configuration panel in which four functions represent four vNICs (Figure 8-5 on page 139).
138
7. Select a function to display the Multichannel Configuration panel where we will configure parameters for that vNIC (Figure 8-6). When in vNIC mode (used in IBM Virtual Fabric Mode solution), there is only one parameter: Logical Port VLAN ID (LPVID). Configuration of this parameter is optional, as vNIC functionality does not depend on it. This is the inner VLAN ID in double-tagged frames (the outer VLAN ID and bandwidth parameters come from the BNT Virtual Fabric 10Gb Switch Module). However, we are configuring for the Switch Independent Mode solution, based on vNIC2 mode. In this mode, LPVID is a required parameter. The adapter uses LPVID values to isolate traffic within each individual vNIC. For network traffic to pass correctly, ensure that all of the VLAN IDs specified on the four vNIC functions are allowed on the blade-facing port on the Virtual Fabric Switch Module. Another important parameter is the vNIC bandwidth allocation. You can enter values between 0 and 100 in this field. The value is the percentage of total bandwidth allocated to a particular vNIC.
139
Figure 8-6 Set vNIC2 mode parameters
8. Review the vNIC2 settings on the second physical port as well, to ensure that the configuration is correct. 9. Exit to the main UEFI menu and save your settings (Figure 8-7 on page 141).
140
Figure 8-7 Save settings
10.Restart the blade server.
8.1.4 Configuring the Broadcom 2-port 10-Gb Virtual Fabric Adapter in UEFI
This section describes the steps to configure vNIC partitions on the physical ports of the Broadcom 2-port 10-Gb Virtual Fabric Adapter. We use the blade server UEFI interface to configure the vNIC parameters. Alternatively, you can use the Broadcom VFA option ROM configuration utility, accessible by pressing Ctrl-S when prompted. Refer to 8.4.1, Broadcom 2-port 10-Gb Virtual Fabric Adapter on page 161 for details. 1. Power on or restart the blade. 2. When prompted during POST, press F1, and the UEFI utility main menu displays (Figure 8-8 on page 142).
141
Figure 8-8 Blade server UEFI utility - main menu
3. Select System Settings, then select Network on the next panel to display the Network panel (Figure 8-9).
Figure 8-9 Network panel
142
Four Broadcom NetXtreme II network ports are listed. Two, 1Gb Ethernet ports belong to the onboard Broadcom network controller. The two 10Gb network ports belong to the Broadcom 2-port 10-Gb Virtual Fabric Adapter. 4. Select the first 10-Gb network port to display the Broadcom Main Configuration window for that port (Figure 8-10).
Figure 8-10 Broadcom main configuration panel
5. Enable vNIC partitions by selecting Device Configuration Menu to display the Device Configuration panel Figure 8-11 on page 144).
143
Figure 8-11 Enable vNIC partitions
6. With the vNIC partitions enabled, press Esc to return to the previous menu, in which a new configuration option displays (NIC Partitioning Configuration Menu) (Figure 8-12).
Figure 8-12 New configuration option for vNIC partitioning
144
7. Select NIC Partitioning Configuration Menu to display the NIC Partition Configuration panel (Figure 8-13). The panel displays four vNICs, and an option to configure bandwidth allocations (Global Bandwidth Allocation Menu).
Figure 8-13 NIC Partition Configuration
8. Select the Global Bandwidth Allocation Menu. 9. As shown in Figure 8-14 on page 146, configure two values, as needed, for each NIC partition: a. Relative bandwidth weight We can use this parameter to specify the guaranteed bandwidth in an overload situation. For example, if the parameter is set to 20, the NIC partition will be guaranteed to transmit 2 Gbps of bandwidth (20% of total bandwidth). The sum of all four NIC partitions must be either zero or 100. If it is zero, then the relative bandwidth weight mechanism is disabled, and a round-robin bandwidth allocation mechanism is used instead. b. Maximum bandwidth This parameter sets the maximum bandwidth (as percentage of total physical port bandwidth) that the NIC partition can transmit. You can set Maximum Bandwidth to 100 on all four partitions. In this case, a partition will be able to use all 10 Gbps of bandwidth when no other partition is
145
communicating. If this parameter is set to a value lower than 100, then this is the maximum bandwidth available to the NIC partition.
Figure 8-14 Global Bandwidth Allocation Menu
10.When you have enabled NIC partitions and set the bandwidth allocation parameters, exit to the main menu. 11.Save your settings.
8.2 Configuring the BNT Virtual Fabric 10Gb Switch Module

This section provides an overview of the tools available for configuring the BNT Virtual Fabric 10Gb Switch Module in Switch Independent Mode. This switch module can be configured using a command-line interface (CLI) (for example, BladeOS or the industry standard CLI (isCLI)), a browser-based interface (BBI), or SNMP. Future plans include configuring the switch module using BOFM. Examples in this section are based on the BNT Virtual Fabric 10Gb Switch Module firmware version 6.5.3.0, which is the current version at the time of this writing.
146
When configuring the Switch Independent Mode solution, we need to configure VLANs on the BNT Virtual Fabric 10Gb Switch Module to match the existing VLAN structure in the network. In this section, we only provide an overview of configuration tools. The actual examples of configuring VLANs are in 9.3.3, Configuring the BNT Virtual Fabric 10-Gb Switch Module on page 252.
8.2.1 BladeOS CLI

BladeOS CLI is a simple and intuitive, menu-based user interface that provides complete switch module administration. To use the BladeOS CLI on the switch module, you can use the following communication methods: Telnet session Serial connection SSH connection BladeOS CLI consists of the main menu and hierarchically organized submenus. An example of logging in to the switch module with telnet follows: 1. In our example, we use the telnet command to connect to the switch module with IP address 9.42.171.39 telnet 9.42.171.39 2. Type the password to log in. The default password is admin. However, clients following best practice recommendations would have likely changed the password for added security. The BladeOS main menu displays (see Figure 8-15 on page 148), which is the starting point to all of the switch module management and configuration tasks.
147
[Main Menu] May 25 14:01:06 9.44.16.209 info stats cfg oper boot maint diff apply save revert exit >> Main# Figure 8-15 BladeOS CLI main menu 9.42.171.39 NOTICE mgmt: admin(admin) login from host Information Menu Statistics Menu Configuration Menu Operations Command Menu Boot Options Menu Maintenance Menu Show pending config changes [global command] Apply pending config changes [global command] Save updated config to FLASH [global command] Revert pending or applied changes [global command] Exit [global command, always available]
Firmware upgrade
Here we show an example of the BNT Virtual Fabric 10Gb Switch Module firmware upgrade using the BladeOS CLI. At the time of writing, firmware 6.5.3.0 is the current level. We will upgrade the switch module to this version. This firmware update package is available on the IBM support website: http://ibm.com/support/entry/portal/docdisplay?lndocid=MIGR-5086594 Late breaking news: BNT Virtual Fabric 10Gb Switch Module firmware 6.7.2 is now available and is required for the Emulex 10GbE Virtual Fabric Adapter II. This firmware level is also required if the Emulex 10GbE Virtual Fabric Adapter is also updated to the latest firmware. The instructions in this section, however, still refer to firmware 6.5.3.0. The firmware code consists of three parts: Boot image OS image 1 OS image 2 When upgrading firmware, upgrade the boot image and one of the two OS images. It is suggested that you leave the other OS image at the existing firmware level, so that you can easily revert back to it, if needed.
148
Use a Trivial File Transfer Protocol (TFTP) or a File Transfer Protocol (FTP) server to transfer the boot and OS images to the BNT Virtual Fabric 10Gb Switch Module. You can use any TFTP or FTP server. In our example, we use the TFTP server available on the BladeCenter Advanced Management Module (AMM). We put the following two files into the tftproot directory in AMM local storage: GbESM-24-10G-6.5.3.0_Boot.img: This is the new boot image. GbESM-24-10G-6.5.3.0_OS.img: This is the new OS image. The contents of the tftproot directory in AMM local storage are shown in Figure 8-16.
Figure 8-16 Firmware files in the tftproot directory
149
After ensuring that TFTP server functionality is enabled in AMM, proceed with the firmware upgrade using BladeOS CLI. The sequence that follows must be carried out in the order written. 1. Transfer the new OS image into one of the image banks (we use image bank 2 in our example) The BladeOS CLI command syntax is as follows: /boot/gtimg X TADDR GbESM-24-10G-6.5.3.3_OS.img The X represents the OS image bank (1 or 2) and TADDR represents the TFTP server IP address. For example, we used the following command: /boot/gtimg 2 9.42.171.73 GbESM-24-10G-6.5.3.0_OS.img Figure 8-17 shows the command output.
>> Main# /boot/gtimg 2 9.42.171.73 GbESM-24-10G-6.5.3.0_OS.img Enter username for FTP server or hit return for TFTP server: image2 currently contains Software Version 6.1.2 that was downloaded at 6:11:38 Thu Jan 5, 2000. New download will replace image2 with file "GbESM-24-10G-6.5.3.0_OS.img" from FTP/TFTP server 9.42.171.73. Confirm download operation [y/n]: y Starting download... File appears valid Download in progress ................................................................. ................................................................. ................................................................ Image download complete (5737743 bytes) Writing to flash...This takes about 90 seconds. Please wait Write complete (5737743 bytes), now verifying FLASH... Verification of new image2 in FLASH successful. image2 now contains Software Version 6.5.3 AMM CodeLevel 3 : 0605WMJ03000 BLADEOS Im2 11/17/2010 Switch is currently set to boot software image1. Do you want to change that to the new image2? [y/n] May 25 14:57:28 9.42.171.39 INFO mgmt: image2 downloaded from host 9.42.171.73, file 'GbESM-24-10G-6.5.3.0_OS.img', software version 6.5.3 y Next boot will use new software image2. >> May 25 14:57:46 9.42.171.39 NOTICE mgmt: boot image changed Boot Options# May 25 14:57:46 9.42.171.39 INFO mgmt: Firmware downloaded to image2 Figure 8-17 Upgrade of OS image 2
150
Figure 8-17 shows that we have successfully updated OS image 2 to the new firmware version 6.5.3.0. Before the upgrade, the switch module was set to boot from OS image bank 1. We changed this to OS image bank 2 during the update process. 2. Transfer the new boot image. The BladeOS CLI command to do this is: /boot/gtimg boot TADDR GbESM-24-10G-6.5.3.0_Boot.img TADDR represents the IP address of the TFTP server. In our example, the actual command is: /boot/gtimg boot 9.42.171.73 GbESM-24-10G-6.5.3.0_Boot.img Figure 8-18 shows an example of a successful boot image update.
>> Boot Options# /boot/gtimg boot 9.42.171.73 GbESM-24-10G6.5.3.0_Boot.img Enter username for FTP server or hit return for TFTP server: boot kernel currently contains Software Version 6.1.2 New download will replace boot kernel with file "GbESM-24-10G6.5.3.0_Boot.img" from FTP/TFTP server 9.42.171.73. Confirm download operation [y/n]: y Starting download... File appears valid Download in progress ................................................................. ................................................................. ................................................................. Boot image (FS, 7687332 bytes) download complete. Writing to flash...This can take up to 90 seconds. Please wait FS Sector now contains Software Version 6.5.3 Boot image (Kernel, 7687332 bytes) download complete. Writing to flash...This can take up to 90 seconds. Please wait Kernel Sector now contains Software Version 6.5.3 Boot image (Boot, 7687332 bytes) download complete. Writing to flash...This can take up to 90 seconds. Please wait Boot Sector now contains Software Version 6.5.3 >>May 25 15:13:17 9.42.171.39 INFO mgmt: boot kernel downloaded from host 9.42.171.73, file 'GbESM-24-10G-6.5.3.0_Boot.img', software version 6.5.3 Boot Options# May 25 15:13:17 9.42.171.39 INFO mgmt: Firmware downloaded to boot kernel Figure 8-18 Upgrade of boot image
151
3. Reset the switch module. The BladeOS CLI command is: /boot/reset More details about BladeOS CLI are documented in Command Reference 6.3, BNT Virtual Fabric 10Gb Switch Module, available at: http://www.ibm.com/support/docview.wss?uid=psg1MIGR-5080917
8.2.2 isCLI
BladeOS is a proprietary CLI environment developed by BNT for configuring and managing their switch modules. However, system administrators who are more used to Cisco CLI have the option of using the industry-standard CLI (isCLI) instead. isCLI is a Cisco-like user interface that enables complete management and configuration support of BNT switch modules. BladeOS CLI is the default CLI mode, but you can easily switch between BladeOS and isCLI: To change from BladeOS to isCLI, use the following command: boot/mode iscli Restart the switch module after running this command. To revert from isCLI to BladeOS CLI, use the command: boot cli-mode bladeos-cli Restart the switch module after the command. CLI mode is preserved across reboots, even when you restore the switch module to factory defaults. isCLI mode has three different privilege levels: User EXEC mode This is the initial mode with the most limited privileges. When you work in this mode, the isCLI prompt ends with the > character, as in: Switch> Privileged EXEC mode This mode has a higher privilege level, and is indicated by number (or pound) character # at the end of the isCLI prompt, as in: Switch#
152
Privileged EXEC mode is accessed from User EXEC mode with the command enable. Global Configuration Mode This is the most privileged mode. When in this mode, you can make changes to a running configuration. Use the following command to enter this mode from the Privileged EXEC mode: configure terminal An example of enabling Privileged EXEC and Global Configuration Mode is shown in Figure 8-19.
Router>enable Enable privilege granted. Router#configure terminal Enter configuration commands, one per line. End with Ctrl/Z. Router(config)# Figure 8-19 isCLI: Enabling EXEC privileges
More details about isCLI are in isCLI Reference 6.3, BNT Virtual Fabric 10Gb Switch Module, available at: http://www.ibm.com/support/docview.wss?uid=psg1MIGR-5080917
8.2.3 Browser-based interface (BBI)

You can use BBI to perform most management, administration and configuration tasks on the BNT Virtual Fabric 10Gb Switch Module. BBI is an easy-to-use, intuitive, web-based management interface, which is enabled by default (and can be disabled in CLI). We show an example of logging in to the switch module and launching BBI in the following steps: 1. Open a web browser window, and connect to the IP address of the switch module. Alternatively, you can access BBI from the AMM web interface, from which you can: a. Select the following options: I/O Module Tasks Configuration Advanced Options Start CLI/Web Session
153
b. Ensure the Protocol field is set to Web (not Telnet), then click Start Session, as shown in Figure 8-20.
Figure 8-20 AMM web interface - Start CLI/web session
2. Log in to the switch module (Figure 8-21 on page 155). Default credentials are: Username: admin Password: admin Note: Clients following best practice recommendations will likely have changed the password from the default for added security.
154
Figure 8-21 Log in to the BNT Virtual Fabric 10Gb Switch Module
3. After a successful login, the BBI Switch Dashboard displays, serving as the starting point for switch module management (Figure 8-22 on page 156).
155
Figure 8-22 The BNT Virtual Fabric 10Gb Switch Module dashboard
To find out more about BBI, see BBI Quick Guide 6.3, BNT Virtual Fabric 10Gb Switch Module, available at: http://www.ibm.com/support/docview.wss?uid=psg1MIGR-5080917
8.3 Cisco Nexus 4001I Switch Module configuration

You can configure the Cisco Nexus 4001I Switch Module using NX-OS CLI commands.
156
Log in to the switch module via Telnet with the default login credentials (the same credentials as on AMM): USERID PASSW0RD (with digit 0) After logging in, the panel shown in Figure 8-23 displays, and the switch module can be configured.
Nexus 4000 Switch ibm-switch-2 login: USERID Password: Cisco Nexus Operating System (NX-OS) Software TAC support: http://www.cisco.com/tac Copyright (c) 2002-2010, Cisco Systems, Inc. All rights reserved. The copyrights to certain works contained in this software are owned by other third parties and used and distributed under license. Certain components of this software are licensed under the GNU General Public License (GPL) version 2.0 or the GNU Lesser General Public License (LGPL) Version 2.1. A copy of each such license is available at http://www.opensource.org/licenses/gpl-2.0.php and http://www.opensource.org/licenses/lgpl-2.1.php ibm-switch-2# Figure 8-23 Cisco NX-OS CLI
When configuring a Switch Independent Mode solution, we need to configure VLANs on the Cisco Nexus 4001I Switch Module to match the existing VLAN structure in the network. In this section, we only provide an overview of configuration basics. The actual examples of configuring VLANs are in 9.2.4, Cisco Nexus 4001I Switch Module configuration on page 224. In the following section, we show the firmware upgrade procedure using CLI commands.
8.3.1 Firmware upgrade

At the time of writing, the latest NX-OS version for the Cisco Nexus 4001I Switch Module is 4.1.2.E1.1F. The firmware update package is available on the IBM support website: http://www.ibm.com/support/entry/portal/docdisplay?lndocid=MIGR-508654 When upgrading firmware on the BNT Virtual Fabric 10Gb Switch Module, we transferred the new code directly from the TFTP server to an image bank on the
157
switch module. In contrast, Cisco allows you to transfer the image from an FTP or TFTP server to a directory in the NX-OS file system. You can upgrade the firmware by installing the new kickstart and system image files. Steps for doing so follow. 1. Copy the kickstart and system image files from the FTP or TFTP server to the Cisco Nexus 4001I Switch Module file system: n4000-bk9-kickstart.4.1.2.E1.1f.bin: The NX-OS kickstart image file. n4000-bk9.4.1.2.E1.1f.bin: The actual NX-OS system image file. The IP address of our FTP server in our example is 9.42.171.50. The destination directory in the NX-OS file system is the bootflash: directory. We specify these as parameters in the following two copy commands: copy ftp://9.42.171.50/n4000-bk9-kickstart.4.1.2.E1.1f.bin bootflash: vrf chassis-management copy ftp://9.42.171.50/n4000-bk9.4.1.2.E1.1f.bin bootflash: vrf chassis-management Figure 8-24 shows the output of these commands. As you can see, we are prompted for the user name and password to access the FTP server.
ibm-switch-2# copy ftp://9.42.171.50/n4000-bk9-kickstart.4.1.2.E1.1f.bin bootflash: vrf chassis-management Enter username: cisco Password: ***** Transfer of file Completed Successfully ***** ibm-switch-2# copy ftp://9.42.171.50/n4000-bk9.4.1.2.E1.1f.bin bootflash: vrf chassis-management Enter username: cisco Password: ***** Transfer of file Completed Successfully ***** Figure 8-24 Copying the kickstart and system image files to a directory
2. Before installing the files, verify that they appear in the bootflash: directory, as shown in Figure 8-25 on page 159.
158
ibm-switch-2# dir 49 Jul 39 Jul 20624384 Jul 74274426 Jul 49152 Jul 20624384 Jul 20699136 May 74274426 Jul 74281634 May 4096 Jul 4096 Jul 4096 Jul
03 03 03 03 03 03 18 03 18 03 03 03
03:58:50 03:58:55 03:58:50 03:58:55 03:58:49 03:54:12 13:49:36 03:54:49 13:50:21 03:56:30 03:56:31 03:56:31
2010 2010 2010 2010 2010 2010 2011 2010 2011 2010 2010 2010
..tmp-kickstart ..tmp-system .tmp-kickstart .tmp-system lost+found/ n4000-bk9-kickstart.4.1.2.E1.1d.bin n4000-bk9-kickstart.4.1.2.E1.1f.bin n4000-bk9.4.1.2.E1.1d.bin n4000-bk9.4.1.2.E1.1f.bin vdc_2/ vdc_3/ vdc_4/
Usage for bootflash:// 333664256 bytes used 290123776 bytes free 623788032 bytes total Figure 8-25 Bootflash: directory
3. To upgrade the NX-OS with the new files, use the install all command, and specify the kickstart and system keywords and image files as parameters, for example: install all kickstart bootflash:n4000-bk9-kickstart.4.1.2.E1.1f.bin system bootflash:n4000-bk9.4.1.2.E1.1f.bin This is shown in Figure 8-26 on page 160.
159
ibm-switch-2# install all kickstart bootflash:n4000-bk9-kickstart.4.1.2.E1.1f.bin system bootflash:n4000-bk9.4.1.2.E1.1f.bin Verifying image bootflash:/n4000-bk9-kickstart.4.1.2.E1.1f.bin for boot variable "kickstart". [####################] 100% -- SUCCESS Verifying image bootflash:/n4000-bk9.4.1.2.E1.1f.bin for boot variable "system". [####################] 100% -- SUCCESS Lines deleted for clarity Switch will be reloaded for disruptive upgrade. Do you want to continue with the installation (y/n)? [n] Install is in progress, please wait. Setting boot variables. [####################] 100% -- SUCCESS Performing configuration copy. [####################] 100% -- SUCCESS Module 1: Refreshing compact flash and upgrading bios/loader/bootrom. Warning: please do not remove or power off the module at this time. [####################] 100% -- SUCCESS Finishing the upgrade, switch will reboot in 10 seconds. Figure 8-26 Upgrading firmware for the Cisco Nexus 4001I Switch Module
The switch module automatically reboots when the upgrade is finished. More details about Cisco Nexus 4001I Switch Module CLI commands are in Command Reference Guide for Cisco Nexus 4001I Switch Module, available at: http://www.ibm.com/support/entry/portal/docdisplay?lndocid=MIGR-5082494
8.4 Creating virtual NICs

vNICs using Switch Independent Mode are created and configured on the Virtual Fabric adapter directly, using either the UEFI interface (Emulex and Broadcom adapters) or a preboot environment (available with the Broadcom adapter only).
160
8.4.1 Broadcom 2-port 10-Gb Virtual Fabric Adapter

Configuration and support of the virtual NICs are carried out using a Broadcom application, available in the pre-boot environment (Ctrl+S during boot when prompted), or using UEFI. Here we describe the pre-boot method. To configure the Broadcom 2-port 10-Gb Virtual Fabric Adapter using the preboot utility, do the following: 1. Power on or restart the blade. 2. When prompted, press Ctrl+s to display the Device List, Figure 8-27.
Figure 8-27 Device List
3. Highlight the device of interest and press Enter. The Main Menu window appears, Figure 8-28 on page 162.
161
Figure 8-28 Device Main Menu
4. Select NIC Partition Configuration and press Enter. The NIC Partition Configuration window appears, Figure 8-29. 5. Highlight the NIC Partition field and press the right arrow to change the value to Enabled.
The four NIC partitions are displayed, with their MAC addresses: PF#0, MAC address 10:18:00:96:CE:90
162
PF#2, MAC address 10:18:00:96:CE:94 PF#4, MAC address 10:18:00:96:CE:98 PF#6, MAC address 10:18:00:96:CE:9C 6. Highlight one of the partitions you want to configure and press Enter. In our case we selected PF#0. Figure 8-30 displays.
Figure 8-30 Partition function menu
You can configure bandwidth parameters on this panel: Bandwidth Weight allows you to specify the guaranteed bandwidth in an overload situation. For example, if the parameter is set to 20, the NIC partition will be guaranteed to transmit 2 Gbps of bandwidth (20% of total bandwidth). The sum of all four NIC partitions must be either zero or 100. If it is zero, then the relative bandwidth weight mechanism is disabled, and a round-robin bandwidth allocation mechanism is used instead Maximum Bandwidth sets the maximum bandwidth (as percentage of total physical port bandwidth) that the NIC partition can transmit. You can set this to 100 on all four partitions. In this case, a partition will be able to use all 10 Gbps of bandwidth when no other partition is communicating. If this parameter is set to a value lower than 100, then this is the maximum bandwidth available to the NIC partition. 7. Once you have set the bandwidth values for NIC partition PF#0, press Esc to exit the menu, then set bandwidth parameters for the other three NIC partitions (PF#2, PF#4 and PF#6) following the same steps.
163

Emulex provides a UEFI interface for configuring virtual NICs. The vNIC Configuration Management Utility allows you to define rate-limiting parameters for each vNIC. A bandwidth value of 0 disables the associated vNIC interface on the hardware. The default configuration PCIe function numbers will automatically be assigned. Tip: Ensure that the network device associated with a disabled vNIC is also disabled in the OS. To enable the Emulex 10GbE Virtual Fabric Adapter II using the server UEFI interface, do the following: 1. Power on or restart the blade. 2. Press F1 when prompted during POST. The UEFI utility main menu displays. 3. Select System Settings, then select Network on the subsequent panel. The network configuration panel displays, Figure 8-31.
Figure 8-31 Network controllers
4. Highlight the Emulex adapter you wish to configure and press Enter. the Emulex NIC Selection menu appears, Figure 8-32 on page 165.
164
Figure 8-32 Emulex NIC Selection
5. To enable vNIC or vNIC2 mode, you must first set Multichannel to Enabled. Unless Multichannel is enabled, the card will operate as two physical 10-Gbps Ethernet ports. When you enable Multichannel, it will be enabled on both 10-Gb physical ports 6. After enabling Multichannel, you can now select vNIC or vNIC2 mode under Switch Configuration: Use vNIC mode in IBM Virtual Fabric Mode. Use vNIC2 mode for Switch Independent Mode. We are configuring for the Switch Independent Mode. Therefore, we need to set the Switch Configuration field to vNIC2. Next, we look at the settings that relate to vNICs. Tip: If you do not see the vNIC2 option, update the Emulex firmware. 7. Highlight Multichannel Configuration and press Enter to display the Function Configuration menu (Figure 8-33 on page 166).
165
Figure 8-33 Function selection
8. Functions 0 to 3 represent four vNICs. Select Function 0 and press Enter to configure parameters for the first vNIC. Figure 8-34 appears.
Figure 8-34 Multichannel Configuration
166
9. Figure 8-34 on page 166 shows the vNIC parameters configuration panel. Two key fields here are: Logical Port VLAN ID (the LPVID field) is a required parameter when configuring for Switch Independent Mode. The adapter uses LPVID values to isolate traffic within each individual vNIC. For network traffic to pass correctly, ensure that all of the VLAN IDs specified on the four vNIC functions are allowed on the blade-facing port on the Virtual Fabric Switch Module. Another important parameter is the vNIC bandwidth allocation, the Bandwidth field. You can enter values between 0 and 100 in this field. The value is the percentage of total bandwidth allocated to a particular vNIC 10.You need to set the LPVID and bandwidth values for the remaining three vNICs, and then configure vNICs on the second physical port as well. When done, exit to the main UEFI menu and save the settings.
8.5 Configuring vNIC VLANs

Switch Independent Mode simply extends the existing customer VLANs to the NIC interfaces. Switch Independent Mode vNICs do not require an outer tag to maintain channel separation. Instead, they utilize the MAC address in combination with the IEEE 802.1Q VLAN tag to distinguish the individual vNIC channels on each link. This requires that you coordinate the configuration of the VFAs and the chassis switches to match the existing VLAN structure in the network. Conventional VLANs are configured by an operating system on a server blade (such as with the vconfig command in Linux) are implemented as IEEE 802.1q VLAN tags. The VLAN configuration on upstream switches needs to match the VLAN configuration on the blades (Figure 8-35 on page 168).
167
Virtual Fabric Switch (Bay 7) 10 Gb port VLAN10

EXT1 EXT2 EXT3 EXT4 EXT5 VLAN30 EXT6
Blade 1
vNIC4
VLAN 40
. . .
10 Gb port
VLAN20
vNIC3
VLAN 30
VLAN40
Operating System
vNIC2 VLAN 20 10 Gb port Virtual Fabric Switch (Bay 9) VLAN10
EXT1 EXT2 EXT3 EXT4 EXT5 VLAN30 EXT6
vNIC1
VLAN 10
. . .
10 Gb port
VLAN20
VLAN40
Figure 8-35 Switch Independent Mode diagram
You will also configure the switch ports using switch configuration methods (for example, GUI, CLI, or SNMP). Presuming that multiple internal switch ports share common VLANs by default, you will create distinct VLAN port groups using conventional port VLAN schemes. Most ports will thus be mapped to multiple VLANs, with the ports configured as VLAN trunks. The VLANs can then be mapped to physically separate external ports, or two or more VLANs can share an uplink port (for example, a VLAN trunk). The switch configuration steps include: 1. Assign internal ports to each VLAN based on vNIC-to-VLAN mapping. A single port will be mapped to multiple VLANs. 2. Assign corresponding external ports to their respective VLAN. A VLAN can be mapped to a single port or a port group. Optionally, multiple VLANs can be mapped to a single port or a port group. The switch normally provides basic Layer 2 forwarding among ports that share a VLAN. Only the BNT Virtual Fabric 10Gb Switch Module is capable of L3 forwarding between separate VLANs within the chassis.
168
8.6 vNIC and VMready

You can use VMready only with the BNT Virtual Fabric 10Gb Switch Module. VMready allows for much greater control over the allocation of network resources. Using VMready, the server administrator can assign network policies at a Group level, as opposed to a physical server level. Access Control Lists (ACLs), Quality of Service (QoS) and VLAN settings in VMready are set at the Group level, and when virtual machines move as a result of a VMotion event, the ACLs, QoS, and VLAN assignments move with them. This ensures that appropriate network resources are allocated to the VMs, no matter where they reside on the network. Versions of BNT Virtual Fabric 10Gb Switch Module firmware prior to 6.7.2 did not support a blade-facing port being in a vNIC group and a VMready group at the same time. However, with the release of 6.7.2, this is now possible. With versions prior to 6.7.2, it is possible, however, to use vNIC to support certain blades and VMready on other blades using the same switch and within the same chassis, just not the same port. However, if you are running the adapters in Switch Independent Mode, you do not need to use vNIC groups and are thus free to implement VMready groups.
8.7 High availability and vNICs

The trunk failover function, which enables high-availability designs when used with standard ports, is also available when either the Emulex or Broadcom adapters in vNIC mode are used with either a BNT or Cisco switch. However, with Switch Independent Mode, if the uplink ports that are members of a VLAN group fail due to a failure on the upstream device, a cable failure, or a cable being erroneously removed, the associated internal port is brought down by the switch, thus impacting those vNIC channels that are in other VLAN groups. In Windows, this displays as a network cable unplugged condition on the server.
8.8 Configuring the operating systems

In this section, we cover how to configure drivers and perform other operating system-related activities.
169
Most configuration activities are performed by utilities provided by the supported operating systems. For the Broadcom 2-port 10-Gb Virtual Fabric Adapter, the centralized management application is the Broadcom Advanced Control Suite (BACS). The BACS application enables centralized management of Broadcom Virtual NICs throughout the data center from a centralized management console. The BACS application provides a graphical user interface (GUI) and a scriptable command line interface (CLI). You can use BACS to configure load balancing and teaming. Similarly, the Emulex OneCommand Manager utility is available to configure Emulex 10GbE Virtual Fabric Adapter II.
8.8.1 vNICs on the Emulex 10GbE Virtual Fabric Adapter II

Figure 8-36 shows the Network Connections menu. As we expect, there are eight vNICs listed, four on each 10-Gb physical port.
Figure 8-36 Network Connections
For clarity, we changed the names of the NICs from the default values to those listed in Table 8-1 on page 171. You can use the MAC addresses to match each vNIC with the virtual fabric adapter (VFA) displayed on the Network Connections panel.
170
Table 8-1 NIC names and matching vNICs NIC name Emulex VFA II #01 Emulex VFA II #02 Emulex VFA II #03 Emulex VFA II #04 Emulex VFA II #11 Emulex VFA II #12 Emulex VFA II #13 Emulex VFA II #14 10Gb physical port 0 0 0 0 1 1 1 1 vNIC 1 2 3 4 1 2 3 4 MAC address 00:00:C9:B2:57:02 00:00:C9:B2:57:03 00:00:C9:B2:57:04 00:00:C9:B2:57:05 00:00:C9:B2:57:06 00:00:C9:B2:57:07 00:00:C9:B2:57:08 00:00:C9:B2:57:09
Right-click a NIC and select Status to verify parameters, such as the MAC address and link speed. Figure 8-37 shows the status panel for the Emulex VFA II #01.
Figure 8-37 Emulex VFA II #01 status
171
Tip: Speed is reported as 4.0 Gbps. This means the bandwidth was set to 40 when configuring this vNIC in Unified Extensible Firmware Interface (UEFI) utility. To see the MAC address of this vNIC, click Details. Figure 8-38 shows the Network Connection Details panel, which displays the MAC address in the Physical Address field.
Figure 8-38 Network Connection Details panel
This MAC address indeed represents vNIC 1 on 10-Gb physical port 0. We can identify and verify MAC addresses and link speeds of the other vNICs in the same manner.
8.8.2 vNICs on the Broadcom 2-port 10-Gb Virtual Fabric Adapter

We now verify how the NIC partitions on the Broadcom 2-port 10-Gb Virtual Fabric Adapter are seen by the operating system (Windows Server 2008). We first show Windows built-in networking tools, and then the Broadcom Advanced Control Suite (BACS).
172
Windows networking tools

Figure 8-39 shows how the NIC partitions appear in the Network and Sharing Center.
Figure 8-39 Broadcom 2-port 10-Gb Virtual Fabric Adapter NIC partitions, viewed using Windows Server 2008
For clarity, we have changed the NIC names from the defaults according to Table 8-2. We used MAC addresses to match each NIC with the actual NIC partition.
Table 8-2 NIC names and matching NIC partitions NIC name Broadcom VFA 01 Broadcom VFA 02 Broadcom VFA 03 Broadcom VFA 04 10Gb physical port 0 0 0 0 NIC partition 1 2 3 4 MAC address
10:18:00:96:E3:80 10:18:00:96:E3:84 10:18:00:96:E3:88 10:18:00:96:E3:8C
173
NIC name Broadcom VFA 11 Broadcom VFA 12 Broadcom VFA 13 Broadcom VFA 14
10Gb physical port 1 1 1 1
NIC partition 1 2 3 4
MAC address
10:18:00:96:E3:82 10:18:00:96:E3:86 10:18:00:96:E3:8A 10:18:00:96:E3:8E
By clicking View Status (highlighted in Table 8-2 on page 173), we can verify parameters for each of the NIC partitions, such as link speed and MAC address. Figure 8-40 shows this for NIC Broadcom VFA 01.
Figure 8-40 Broadcom VFA 01 status
As you can see, speed is reported as 4.0 Gbps. This indicates the bandwidth was set to 40 on this NIC partition when configuring parameters in UEFI utility. Click Details to see the MAC address of this NIC partition. Figure 8-41 on page 175 shows the Network Connection Details panel in which the MAC address displays.
174
Figure 8-41 Network Connection Details panel showing the MAC address
This is indeed the MAC address of NIC partition 1 on 10-Gb physical port 0. Use the same method to verify MAC addresses and speed for the other NIC partitions on the Broadcom 2-port 10-Gb Virtual Fabric Adapter.
Broadcom Advanced Control Suite (BACS)

BACS is a Windows-based management utility that allows you to administer, configure, and diagnose Broadcom Virtual Fabric Adapters and other Broadcom NICs. Figure 8-42 on page 176 shows an example from BACS. You can see eight NIC partitions in the left pane. NIC partition 1 on physical port 0 is highlighted. Note the speed: 4 Gbps (this indicates the bandwidth parameter of 40 in the UEFI utility).
175
Figure 8-42 BACS - NIC partition 1
Using the BACS utility, test the configuration by running a basic connectivity test. To do so, click the Diagnostics tab, then ping to either an IP gateway or a user-specified IP address. See Figure 8-43 on page 177.
176
Figure 8-43 BACS - Diagnostics tab
The Statistics tab displays network traffic counters. These counters are useful when monitoring or troubleshooting NIC behavior. See Figure 8-44 on page 178.
177
Figure 8-44 BACS - Statistics tab
One of the functions of the BACS utility is to configure NIC teaming. You can use NIC partitions in teaming configurations the same way you would use physical NICs. To set up NIC teaming, click Team Management in the lower left corner (see Figure 8-45 on page 179). We cover BACS NIC teaming in 8.8.4, Configuring teams with Broadcom BACS on page 184.
178
Figure 8-45 BACS - Team Management
8.8.3 Configuring teams with Emulex OneCommand Manager

To configure NIC teaming with the Emulex 10GbE Virtual Fabric Adapter II, we can use the Emulex OneCommand Manager utility, which runs in Windows and Linux. It also supports VMware ESX/ESXi and Microsoft Hyper-V hypervisors. See the OneCommand Manager home page for details: http://www.emulex.com/products/management-software/device-management/on ecommand-manager/ Figure 8-46 on page 180 shows the OneCommand Manager main window.
179
Figure 8-46 Emulex OneCommand Manager utility
In our example, we will create a team with two members: Emulex VFA II #01 and Emulex VFA II #02. 1. Click Create team to begin the process. This launches the Create team window, shown in Figure 8-47 on page 181.
180
Figure 8-47 Emulex OneCommand Manager - Create team window
2. In this window, you can specify team parameters, such as team name and type, and select team members. We use the simplest form of teaming, Failover, so select FailOver from the Team Type field in Figure 8-47. This means the primary member will carry all the traffic, while the secondary member will have a standby role. Other team type options are: LoadBalancing 802.3ad static LACP 3. Select each adapter you wish to form into the team and click Add. 4. When done with setting parameters and selecting team members, click OK to continue. Figure 8-48 on page 182 shows the newly created team.
181
Figure 8-48 Emulex OneCommand Manager - new team created
5. By highlighting the new team and clicking VLAN, we can configure VLAN ID for the team. This opens up the Add/Remove VLAN window, shown in Figure 8-49 on page 183.
182
Figure 8-49 Emulex OneCommand Manager - Adding VLAN
6. After entering the VLAN ID and clicking Add, the VLAN is configured (see Figure 8-50 on page 184).
183
Figure 8-50 Emulex OneCommand Manager - VLAN configured
7. Click OK to return to the main window.
8.8.4 Configuring teams with Broadcom BACS

A team (or bond in Linux terminology) of adapters functions as a single virtual network interface and appears the same as a nonteamed adapter to other network devices. BACS allows you to team up to eight network adapters together in a single group. This group becomes a virtual network adapter with its own IP address. The benefits of teaming are as follows: Virtual adapters can balance a large network load across several adapters. The use of teams improves system availability by providing a failover path for critical network connections.
184
Types of teams supported by BACS are: Switch-dependent Link Aggregation Control Protocol (LACP) or IEEE 802.3ad GEC/FEC (Gigabit Etherchannel/Fast Etherchannel) Switch-dependent NIC teaming is not related to IBM Virtual Fabric Mode. This teaming type requires that LACP/Etherchannel is configured on an Ethernet switch module. Modules also need to be stacked, to act as a single switch. Switch-independent SLB (Smart Load Balancing) SLB with Auto Failover Disable Switch-independent NIC teaming is not related to Switch Independent Mode. The name indicates that switch-independent NIC teaming does not require any LACP/Etherchannel configuration on an Ethernet switch module. We show how to configure a new team in Windows using BACS application: 1. Launch the BACS application to open up main the BACS window (shown in Figure 8-51 on page 186).
185
Figure 8-51 BACS main window
2. Click the Team Management frame (lower left corner) to start the Broadcom Teaming Wizard. The Welcome window displays, shown in Figure 8-52 on page 187.
186
Figure 8-52 Using the teaming wizard - Welcome window
3. Click Next on the Teaming Wizard Welcome window. You now need to enter the name for the new team (see Figure 8-53 on page 188).
187
Figure 8-53 Using the teaming wizard (Creating/Modifying a Team: Team Name window)
4. After entering the name for the new team, click Next to display the Team Type window (Figure 8-54 on page 189).
188
Figure 8-54 Using the teaming wizard (Creating/Modifying a Team: Team Type window)
5. Select the type of team you wish to create and click Next. 6. At the prompt, verify and confirm that the switch is configured correctly (this is only applicable to teaming modes that require LACP/Etherchannel configuration on the switch modules); Figure 8-55.
Figure 8-55 Using the teaming wizard (Verification window)
7. Click OK in the verification window to proceed. You can now assign team members (Figure 8-56 on page 190).
189
Figure 8-56 Using the teaming wizard (Creating/Modifying a Team: Assigning Team Members window)
8. Highlight the adapter you wish to add to the team and click Add, then add the remaining team members in the same manner;Figure 8-57.
Figure 8-57 Using the teaming wizard (Creating/Modifying a Team: Assigning Team Members window)
190
9. Once all team members are added, click Next, and the Commit Changes window displays (Figure 8-58).
Figure 8-58 Using the teaming wizard (Commit changes window)
10.Optionally, select the Save changes and continue to manage more teams radio button, if needed, and click Next. 11.With all team members added, select the Commit changes to system and Exit the wizard radio button, then click Finish. 12.The Team Management window confirms the newly added team (Figure 8-59 on page 192).
191
Figure 8-59 Teaming established
This completes configuration of NIC teaming with the BACS application.
192
Chapter 9.
Examples of use - Switch Independent Mode

In this chapter, we show examples of Switch Independent Mode configurations, using the following combinations: Cisco Nexus 4001I Switch Module with: Emulex 10GbE Virtual Fabric Adapter II Broadcom 2-port 10-Gb Virtual Fabric Adapter These are covered in 9.2, Cisco Nexus 4001I Switch Module configurations on page 197. BNT Virtual Fabric 10-Gb Switch Module with Emulex 10GbE Virtual Fabric Adapter II Broadcom 2-port 10-Gb Virtual Fabric Adapter These are covered in 9.3, BNT Virtual Fabric 10-Gb Switch Module configurations on page 226.
193
9.1 Configurations overview

We use the following equipment to demonstrate the four configurations: Two BladeCenter H chassis (BC4 and BC5) Two Cisco Nexus 4001I Switch Modules in chassis BC4 Two BNT Virtual Fabric 10-Gb Switch Modules in chassis BC5 Two HS22 blades in each chassis Two Emulex 10GbE Virtual Fabric Adapter II cards Two Broadcom 2-port 10-Gb Virtual Fabric Adapter cards Chassis BC4 is used to demonstrate the following configurations, shown in Figure 9-1: Emulex 10GbE Virtual Fabric Adapter II and Cisco Nexus 4001I Switch Module Broadcom 2-port 10-Gb Virtual Fabric Adapter and Cisco Nexus 4001I Switch Module
Chassis midplane HS22 blade 1 Emulex VFA II
Cisco Nexus 4001I (switch bay 7) INT1 INT7
HS22 blade 7 Broadcom VFA
INT1 INT7 Cisco Nexus 4001I (switch bay 9)
BladeCenter H chassis BC5
Figure 9-1 BladeCenter H chassis BC4
Chassis BC5 is used to demonstrate the following configurations, shown in Figure 9-2 on page 195: Emulex 10GbE Virtual Fabric Adapter II and BNT Virtual Fabric 10-Gb Switch Module
194
Broadcom 2-port 10-Gb Virtual Fabric Adapter and BNT Virtual Fabric 10-Gb Switch Module
Chassis midplane HS22 blade 4 Emulex VFA II
BNT Virtual Fabric Switch Module (bay 7) INT4 INT8
HS22 blade 8 Broadcom VFA
INT4 INT8 BNT Virtual Fabric Switch Module (bay 9)
BladeCenter H chassis BC5
Figure 9-2 BladeCenter H chassis BC5
9.1.1 Virtual LAN (VLAN) definitions

We will enable the following VLANs on virtual fabric switch (VFS) modules: VLAN 10 Both blades use this VLAN for communication through external port 2 on the VFS. vNIC 1 on each blade server communicates on this VLAN. VLAN 20 The blades use this VLAN to communicate through external port 4. vNIC 2 on each blade server communicates on this VLAN. VLAN 30 This VLAN does not use any external port on the VFS module. The respective vNIC 3s on each blade server use this VLAN to communicate between themselves. VLAN 40 vNIC 4 on blade server 1 in chassis BC4 communicates through external port 2 on the VFS module. We use VLAN 40 for this. Similarly, vNIC 4 on blade server 4 in chassis BC5 communicates through external port 2 on the BNT 10-Gb Virtual Fabric Switch Module.
Chapter 9. Examples of use - Switch Independent Mode
195
VLAN 50 vNIC 4 on blade server 7 in chassis BC4 communicates through external port 4 on the VFS module. We use VLAN 50 for this. Similarly, vNIC 4 on blade server 8 in chassis BC5 communicates through external port 4 on the BNT 10-Gb Virtual Fabric Switch Module. Table 9-1 and Table 9-2 show these VLANs on Cisco Nexus 4001I Switch Module in chassis BC4 and on BNT 10-Gb Virtual Fabric Switch Module in chassis BC5.
Table 9-1 VLANs on Cisco Nexus 4001I Switch Module VLAN 10 20 30 40 50 Internal ports 1, 7 1, 7 1, 7 1 7 2 4 External ports 2 4
Because each internal (blade-facing) port on the switch modules needs to pass traffic in multiple VLANs, it is important to configure these ports in trunk mode. External ports can be configured in trunk mode when necessary (when traffic from multiple VLANs passes through a particular external port).
Table 9-2 VLANs on BNT 10Gb Virtual Fabric Switch Module VLAN 10 20 30 40 50 Internal ports 4, 8 4, 8 4, 8 4 8 2 4 External ports 2 4
9.1.2 Switch/adapter vNIC correlation

In IBM Virtual Fabric Mode (switch-dependent mode), there is a direct correlation between vNICs on the Emulex Virtual Fabric Adapter and vNIC definitions on the
196
BNT Virtual Fabric 10-Gb Switch Module. This is because vNICs are configured on the switch module, and settings are transferred to the adapter via DCBX. In Switch Independent Mode, this is different. We configure vNICs on the adapter itself, for example, using the Unified Extensible Firmware Interface (UEFI) utility. No vNIC settings need to be configured on the switch module. However, we do need to configure internal ports as trunk ports and associate one or more VLANs with each vNIC channel, thus allowing for multiple VLANs on each port to ensure that the traffic flows to the correct interfaces. The intent is to extend specific VLANs to specific vNIC interfaces within each server blade.
9.2 Cisco Nexus 4001I Switch Module configurations

In this section, we show examples using the Cisco Nexus 4001I Switch Module. In our examples, we have installed these modules in chassis BC4. The blade in slot 1 has Emulex 10GbE Virtual Fabric Adapter II, and the blade in slot 7 has Broadcom 2-port 10-Gb Virtual Fabric Adapter. We begin with the Emulex configuration on blade 1, then show the Broadcom configuration. Finally, we will set up the switch module. Topics in this section are: 9.2.1, vNICs summary on page 197 9.2.2, Emulex adapter configuration on page 198 9.2.3, Broadcom adapter configuration on page 208 9.2.4, Cisco Nexus 4001I Switch Module configuration on page 224
9.2.1 vNICs summary

The HS22 blade server in slot 1 contains an Emulex 10GbE Virtual Fabric Adapter II. On this adapter, we enable vNICs and set the following VLAN IDs for them: VLAN 10 for vNIC 1 VLAN 20 for vNIC 2 VLAN 30 for vNIC 3 VLAN 40 for vNIC 4 The HS22 blade server in slot 7 contains a Broadcom 2-port 10-Gb Virtual Fabric Adapter. We will enable vNICs on the adapter, but we do not specify any VLAN IDs on the Broadcom 2-port 10-Gb Virtual Fabric Adapter. However, we set the following VLANs for vNICs in the operating system: VLAN 10 for vNIC 1
197
VLAN 20 for vNIC 2 VLAN 30 for vNIC 3 VLAN 50 for vNIC 4
9.2.2 Emulex adapter configuration

Table 9-3 shows the configuration that we will implement on the Emulex 10GbE Virtual Fabric Adapter II in HS22 blade 1.
Table 9-3 vNIC configuration: Emulex 10GbE Virtual Fabric Adapter II vNIC function 1 2 3 4 Bandwidth 4Gbps 3Gbps 2Gbps 1Gbps VLAN ID 10 20 30 40 MAC address 00:00:C9:B2:57:02 00:00:C9:B2:57:03 00:00:C9:B2:57:04 00:00:C9:B2:57:05
We can identify the MAC addresses for each vNIC function using the UEFI utility, (see vNICs in the operating system on page 205). It is important to know the MAC addresses so that you can identify each vNIC function in the operating system. To configure parameters on the Emulex adapter, perform these steps: 1. Power on or restart HS22 blade 1. 2. When prompted during POST, press F1, and the UEFI utility main menu displays (Figure 9-3 on page 199).
198
Figure 9-3 UEFI utility main menu
3. Select System Settings, and then select Network on the subsequent panel to display the Network panel, showing two Emulex 10-Gb ports (Figure 9-4).
Figure 9-4 Network configuration in the blade server UEFI utility
199
4. Select the first port on which to configure vNICs, and the Emulex NIC Selection panel displays (Figure 9-5).
Figure 9-5 Set vNIC2 mode on the Emulex 10GbE Virtual Fabric Adapter II
5. Ensure that the following values are set: Multichannel must be enabled. vNIC2 mode must be set in the Switch Configuration field. 6. Configure VLAN and bandwidth settings for each of the four vNICs by selecting Multichannel Configuration, which displays the Function Configuration panel (Figure 9-6 on page 201).
200
7. Configure each of the four vNIC functions: a. Set 4 Gbps (40% of total bandwidth) for the first vNIC function with MAC address 00:00:C9:B2:57:02, and this vNIC will use VLAN 10 (Figure 9-7 on page 202).
201
Figure 9-7 Bandwidth and VLAN settings for the first vNIC
b. Set 3 Gbps (30% of bandwidth) for the second vNIC function with MAC address 00:00:C9:B2:57:03, and this vNIC will use VLAN 20 (Figure 9-8).
Figure 9-8 Bandwidth and VLAN settings for the second vNIC
202
c. Set 2 Gbps (20% of bandwidth) for the third vNIC function with MAC address 00:00:C9:B2:57:04), and this vNIC will use VLAN 30 (Figure 9-9).
Figure 9-9 Bandwidth and VLAN settings for the third vNIC
d. Set 1 Gbps (10% of bandwidth) for the fourth vNIC function with MAC address 00:00:C9:B2:57:05, and this vNIC will use VLAN 40 (Figure 9-10 on page 204).
203
Figure 9-10 Bandwidth and VLAN settings for the fourth vNIC
8. With all four vNIC functions configured, save your changes (Figure 9-11).
Figure 9-11 Save settings before exiting
204
Configure vNIC functions on the second 10-Gb physical port in the same manner. Use the steps in Emulex adapter configuration on page 198 as a guide. Remember to save your settings before exiting (Figure 9-12).
vNICs in the operating system

HS22 blade server 1 runs Windows Server 2008. We can now check how the vNICs appear to the operating system. Figure 9-13 on page 206 shows the Network Connections window, listing eight vNICs, four on each 10-Gb physical port.
205
Figure 9-13 Network Connections window
For clarity, we have changed the names of NICs from defaults to the values listed in Table 9-4. We used the MAC addresses to match each vNIC with the adapter displayed in the Network Connections window.
Table 9-4 NIC names and matching vNICs NIC name Emulex VFA II #01 Emulex VFA II #02 Emulex VFA II #03 Emulex VFA II #04 Emulex VFA II #11 Emulex VFA II #12 Emulex VFA II #13 Emulex VFA II #14 10Gb physical port 0 0 0 0 1 1 1 1 vNIC 1 2 3 4 1 2 3 4 MAC address 00:00:C9:B2:57:02 00:00:C9:B2:57:03 00:00:C9:B2:57:04 00:00:C9:B2:57:05 00:00:C9:B2:57:06 00:00:C9:B2:57:07 00:00:C9:B2:57:08 00:00:C9:B2:57:09
To verify parameters, including MAC address and link speed, right-click a NIC (for example, the Emulex VFA II #01) and select Status (Figure 9-14 on page 207).
206
Figure 9-14 Emulex VFA II #01 status
Speed is reported as 4.0 Gbps, because we set bandwidth to 40 on this vNIC (see Figure 9-7 on page 202). To view the MAC address, click Details to display the Network Connection Details panel (Figure 9-15 on page 208), in which the Physical Address field shows the MAC address.
207
Figure 9-15 Network Connection Details
This MAC address indeed belongs to vNIC 1 on 10-Gb physical port 0. Identify and verify MAC addresses and link speeds of all other vNICs in the same manner.
9.2.3 Broadcom adapter configuration

We set up the Broadcom 2-port 10-Gb Virtual Fabric Adapter in a manner similar to that for the Emulex adapter, with one difference: We do not specify any VLAN ID on the Broadcom 2-port 10-Gb Virtual Fabric Adapter,which only uses MAC addresses for virtual channel traffic separation. Table 9-5 lists the settings for the Broadcom 2-port 10-Gb Virtual Fabric Adapter.
Table 9-5 Broadcom 2-port 10-Gb Virtual Fabric Adapter vNIC configuration NIC partition 1 2 Bandwidth 4Gbps 3 Gbps MAC address 10:18:00:96:E3:80 10:18:00:96:E3:84
208
NIC partition 3 4
Bandwidth 2 Gbps 1 Gbps
MAC address 10:18:00:96:E3:88 10:18:00:96:E3:8C
We can identify the MAC addresses for each NIC partition using the UEFI utility (see Figure 9-7 on page 202). It is important to know the MAC addresses so that you can identify each NIC partition in the operating system. Follow these steps to configure the Broadcom 2-port 10-Gb Virtual Fabric Adapter parameters in the UEFI utility: 1. Power on or restart the blade server. 2. When prompted during POST, press F1 to display the UEFI utility main menu (Figure 9-16).
Figure 9-16 Blade server UEFI utility - main menu
3. Select System Settings, then Network on the subsequent panel, to display the Network panel (Figure 9-17 on page 210).
209
Figure 9-17 Network configuration - select the first 10-Gb port
4. Select the first of the two 10-Gb physical ports to configure, and this displays the Broadcom Main Configuration panel (Figure 9-18).
Figure 9-18 Broadcom Main Configuration panel
210
5. Select Device Configuration Menu to display the Device Configuration panel, Figure 9-19.
Figure 9-19 Device Configuration Menu with NIC Partition set to Enabled
6. Highlight NIC Partition and set it to Enabled. 7. With NIC partition enabled, press Esc to return to the Broadcom Main Configuration Page. Notice that a new option displays, the NIC Partitioning Configuration Menu option (Figure 9-20 on page 212).
211
Figure 9-20 New configuration option for NIC partitioning
8. Select the NIC Partitioning Configuration Menu option to display the NIC Partition Configuration panel (Figure 9-21).
Figure 9-21 NIC Partition Configuration panel
212
On this panel, you can identify the MAC addresses of all four NIC partitions: PF#0 represents the first NIC partition: L2=10180096E380, which means the MAC address is 10:18:00:96:E3:80 PF#2 is the second NIC partition: L2=10180096E384, which means the MAC address is 10:18:00:96:E3:84 PF#4 is the third NIC partition: L2=10180096E388, which means the MAC address is 10:18:00:96:E3:88 PF#6 is the fourth NIC partition: L2=10180096E38C, which means the MAC address is 10:18:00:96:E3:8C It is important to know these MAC addresses so that you can match the NICs presented in the operating system to the NIC partitions. 9. Highlight Global Bandwidth Allocation Menu and press Enter. Figure 9-22 appears.
10.Using this menu, you can set the bandwidth allocation parameters. In our example, we do not use relative bandwidth weight, so we leave these values at zero. However, we do configure maximum bandwidth, as follows: NIC partition 1: 4Gbps (40% of total bandwidth) NIC partition 2: 3Gbps (30% of total bandwidth)
213
NIC partition 3: 2Gbps (20% of total bandwidth) NIC partition 4: 1Gbps (10% of total bandwidth) 11.Configure NIC partitions and bandwidth parameters for the second 10-Gb physical port, starting with Figure 9-17 on page 210 12.Save your settings before exiting the UEFI utility (Figure 9-23).
Figure 9-23 Save settings before exiting the UEFI utility

We now verify how the vNIC partitions are seen in the operating system. Blade 7 runs Windows Server 2008. We first describe the Windows built-in networking tools, and then the Broadcom Advanced Control Suite (BACS).

Figure 9-24 on page 215 shows how the NIC partitions appear in Network and Sharing Center.
214
Figure 9-24 Broadcom 2-port 10-Gb Virtual Fabric Adapter NIC partitions in Windows Server 2008
For clarity, we have changed the NIC names from the defaults according to Table 9-6. We used MAC addresses to match each NIC with the actual NIC partition.
Table 9-6 NIC names and matching NIC partitions NIC name Broadcom VFA 01 Broadcom VFA 02 Broadcom VFA 03 Broadcom VFA 04 Broadcom VFA 11 Broadcom VFA 12 Broadcom VFA 13 10-Gb physical port 0 0 0 0 1 1 1 NIC partition 1 2 3 4 1 2 3 MAC address
10:18:00:96:E3:80 10:18:00:96:E3:84 10:18:00:96:E3:88 10:18:00:96:E3:8C 10:18:00:96:E3:82 10:18:00:96:E3:86 10:18:00:96:E3:8A
215
NIC name Broadcom VFA 14
10-Gb physical port 1
NIC partition 4
MAC address
10:18:00:96:E3:8E
To verify the parameters for each NIC partition, such as link speed and MAC address, click View Status (Figure 9-25 shows this for NIC Broadcom VFA 01).
As you can see, speed is reported as 4.0 Gbps, because we set bandwidth to 40 on this NIC partition (see Figure 9-22 on page 213). To view the MAC address of this NIC partition, click Details to display the Network Connection Details panel (Figure 9-26 on page 217).
216
Figure 9-26 Network Connection Details showing the MAC address
This is indeed the MAC address of NIC partition 1 on 10-Gb physical port 0. We can use the same method to verify MAC addresses and speed for all other NIC partitions on the Broadcom 2-port 10-Gb Virtual Fabric Adapter.
VLAN IDs in the operating system

You can define VLAN IDs for each NIC partition in the operating system. For example, follow these steps to define VLAN ID 10 on NIC named Broadcom VFA 01: 1. From the Networking in Windows Control Panel, launch the properties window for the Broadcom VFA 01 (Figure 9-27 on page 218).
217
Figure 9-27 Broadcom VFA 01 properties window
2. Click Configure, then click the Advanced tab, to display a window in which you can configure NIC settings, such as the VLAN ID (Figure 9-28 on page 219). 3. For this example, set the VLAN ID to 10.
218
Figure 9-28 VLAN ID configuration
4. Click OK to save the changes. 5. Use the same steps to configure the VLAN ID on other NIC partitions on 10-Gb physical port 0, as follows: Broadcom VFA 02 - VLAN ID 20 Broadcom VFA 03 - VLAN ID 30 Broadcom VFA 04 - VLAN ID 40 6. Configure VLAN IDs on NIC partitions on 10-Gb physical port 1 in the same manner, starting with Figure 9-27 on page 218.

Broadcom Advanced Control Suite (BACS) is a management utility that allows you to administer, configure, and diagnose Broadcom Virtual Fabric Adapters (VFAs) and other Broadcom NICs. Figure 9-29 on page 220 shows an example from BACS. You can see eight NIC partitions in the left pane, with NIC partition 1 on physical port 0 highlighted. Note that the speed is 4 Gbps, as we set it previously (see Figure 9-22 on page 213).
219
Figure 9-29 BACS: NIC partition 1
On the Configurations tab, you can check and set NIC parameters (Figure 9-30 on page 221). Notice that VLAN ID is highlighted and is set to 10 (as we set it in Figure 9-28). You can use BACS to configure the VLAN ID to a different value, or to change any other parameter.
220
Figure 9-30 BACS - configuration tab
On the Diagnostics tab, you can test configurations by running a basic connectivity test: ping either to an IP gateway or to a user-specified IP address (Figure 9-31 on page 222).
221
The Statistics tab displays network traffic counters (Figure 9-32 on page 223). These counters are useful when monitoring or troubleshooting NIC behavior.
222
One of the main functions of the BACS utility is to configure NIC teaming. You can use NIC partitions in teaming configurations in the same way you would use physical NICs. To set up NIC teaming, click Team Management (lower left corner) to start the teaming wizard (Figure 9-33 on page 224). From here, you can select the NICs (or NIC partitions) for the team, and then configure teaming parameters. The teaming wizard is discussed in 8.8.4, Configuring teams with Broadcom BACS on page 184.
223
Figure 9-33 BACS: Team Management
9.2.4 Cisco Nexus 4001I Switch Module configuration

We are now ready to configure VLANs on the Cisco Nexus 4001I Switch Module. The HS22 blade in slot 1 contains the Emulex adapter which is configured to use the following VLANs on vNICs: 10, 20, 30, and 40. So we need to set the internal (blade-facing) port 1 on the switch module to trunk mode, and allow VLANs 10, 20, 30, and 40 on it. The commands to perform these actions are in Figure 9-34 on page 225.
224
ibm-switch-2(config)# configure terminal ibm-switch-2(config)# interface Ethernet 1/1 ibm-switch-2(config-if)# switchport mode trunk ibm-switch-2(config-if)# switchport trunk allowed vlan 10, 20, 30, 40 Figure 9-34 Configuring internal port 1 on Cisco Nexus 4001I Switch Module
The HS22 blade in slot 7 contains the Broadcom adapter. There is no VLAN ID configuration in the UEFI settings for the Broadcom card, but we set the following VLANs in the operating system: 10, 20, 30, and 50. We now need to configure internal port 7 on the switch module to pass traffic in these VLANs. The commands are shown in Figure 9-35.
ibm-switch-2(config-if)# configure terminal ibm-switch-2(config)# interface Ethernet 1/7 ibm-switch-2(config-if)# switchport mode trunk ibm-switch-2(config-if)# switchport trunk allowed vlan 10, 20, 30, 50 Figure 9-35 Configuring internal port 7 on Cisco Nexus 4001I Switch Module
We need to allow VLANs also on external ports, so that the blade servers can communicate with devices outside of the chassis. If any external port needs to carry traffic from multiple VLANs, you need to set trunk port mode, and allow the corresponding VLANs on it. In our example, two external ports are used: External port 2 needs to carry VLANs 10 and 40. Figure 9-36 lists the commands to allow this.
ibm-switch-2(config-if)# configure terminal ibm-switch-2(config)# interface Ethernet 1/16 ibm-switch-2(config-if)# switchport mode trunk ibm-switch-2(config-if)# switchport trunk allowed vlan 10, 40 Figure 9-36 External port 2 VLANs
External port 4 will carry VLANs 20 and 50. Figure 9-37 on page 226 shows the commands to configure the port.
225
ibm-switch-2(config-if)# configure terminal ibm-switch-2(config)# interface Ethernet 1/18 ibm-switch-2(config-if)# switchport mode trunk ibm-switch-2(config-if)# switchport trunk allowed vlan 20, 50 Figure 9-37 External port 4 VLANs
This completes the configuration of VLANs on Cisco Nexus 4001I Switch Module.
9.3 BNT Virtual Fabric 10-Gb Switch Module configurations

In this section, we show the examples with BNT Virtual Fabric 10-Gb Switch Module. These modules are installed in chassis BC5 in our lab examples. Blade in slot 4 has Emulex 10GbE Virtual Fabric Adapter II, and blade in slot 8 has Broadcom 2-port 10-Gb Virtual Fabric Adapter. We begin with an Emulex configuration, and then also show the Broadcom configuration. Finally, we will set up the switch module. Topics we cover in this section are: 9.3.1, Emulex adapter configuration 9.3.2, Broadcom adapter configuration on page 237 9.3.3, Configuring the BNT Virtual Fabric 10-Gb Switch Module on page 252
9.3.1 Emulex adapter configuration

Table 9-3 shows the configuration we implemented on the Emulex 10GbE Virtual Fabric Adapter II on the HS22, blade 4.
Table 9-7 Emulex Virtual Fabric Adapter II vNIC configuration vNIC function 1 2 3 4 Bandwidth 4 Gbps 3 Gbps 2 Gbps 1 Gbps VLAN ID 10 20 30 40 MAC address 00:00:C9:B2:5B:3A 00:00:C9:B2:5B:3B 00:00:C9:B2:5B:3C 00:00:C9:B2:5B:3D
226
You can identify the MAC addresses for each vNIC function using the Unified Extensible Firmware Interface (UEFI) utility (see Figure 9-39 on page 228). With the MAC addresses, you can then identify each vNIC function in the operating system. To configure the parameters on the Emulex adapter: 1. Power on or restart HS22 blade 4. 2. When prompted during POST, press F1, and the System Configuration and Boot Management panel displays (also known as the UEFI utility Main Menu) (Figure 9-3).
Figure 9-38 The UEFI utility
3. Select System Settings, then select Network on the subsequent panel to display the Network panel that shows two Emulex 10-Gb ports (Figure 9-39 on page 228).
227
Figure 9-39 Network configuration in the blade server UEFI utility
4. Select the first port to configure the vNICs for, then press Enter to display the Emulex NIC Selection panel (Figure 9-5).
Figure 9-40 Set vNIC2 mode on the Emulex Virtual Fabric Adapter II
228
5. On the Emulex NIC Selection panel, ensure that the following values are set: Multichannel is set to Enabled. The Switch Configuration field is set to the vNIC2 mode. 6. Configure the VLAN and bandwidth settings for each of the four vNICs: a. Highlight Multichannel Configuration and press Enter to display the Function Configuration panel (Figure 9-6).
7. Configure each of the four vNIC functions by selecting each Function menu item in turn, then pressing Enter to display the Multichannel Configuration panel for that vNIC. a. Allocate 4 Gbps (40% of total bandwidth) to the first vNIC function, which has MAC address 00:00:C9:B2:5B:3A and will use VLAN 10 (Figure 9-42 on page 230).
229
Figure 9-42 Bandwidth and VLAN settings for the first vNIC
b. Allocate 3 Gbps (30% of total bandwidth) to the second vNIC function, which has MAC address 00:00:C9:B2:5B:3B and will use VLAN 20 (Figure 9-8).
Figure 9-43 Bandwidth and VLAN settings for the second vNIC
230
c. Allocate 2 Gbps (20% of total bandwidth) to the third vNIC function, which has MAC address 00:00:C9:B2:5B:3C and will use VLAN 30 (Figure 9-9).
Figure 9-44 Bandwidth and VLAN settings for the third vNIC
d. Allocate 1 Gbps (10% of total bandwidth) to the fourth vNIC function, which has MAC address 00:00:C9:B2:5B:3D and will use VLAN 40 (Figure 9-45 on page 232).
231
Figure 9-45 Bandwidth and VLAN settings for the fourth vNIC
8. With the four vNIC functions configured, save the settings (Figure 9-46).
232
9. Configure the vNIC functions on the second 10-Gb physical port in the same manner, by selecting the second 10-Gb port in Figure 9-39 on page 228 and following the similar procedure. 10.Save all settings before exiting (Figure 9-47).

The HS22 blade in bay 4 runs SLES 11. You can now find out how the vNICs appear to the operating system. The Network Settings panel is shown in Figure 9-48 on page 234, which lists eight vNICs, four on each 10-Gb physical port.
233
Figure 9-48 Network Settings
Tip: At the time of writing, Linux and VMware both report the Emulex 10GbE Virtual Fabric Adapter II adapter as having a ServerEngines chip. This will change to Emulex in a future firmware update. For clarity, we changed the names of the NICs from the default values to those listed in Table 9-4 on page 206. We used the MAC addresses to match each vNIC with the adapter as displayed on the Network Connections panel. To determine the MAC addresses, you can use OneCommand Manager (see OneCommand Manager on page 235) or the ifconfig command (see The ifconfig command on page 236).
234
Table 9-8 NIC names and matching vNICs NIC name eth4 eth6 eth8 eth10 eth5 eth7 eth9 eth11 10Gb physical port 0 0 0 0 1 1 1 1 vNIC 1 2 3 4 1 2 3 4 MAC address 00:00:C9:B2:5B:3A 00:00:C9:B2:5B:3B 00:00:C9:B2:5B:3C 00:00:C9:B2:5B:3D 00:00:C9:B2:5B:3E 00:00:C9:B2:5B:3F 00:00:C9:B2:5B:40 00:00:C9:B2:5B:41
OneCommand Manager
You can use OneCommand Manager to verify parameters such as MAC addresses and link speed. Figure 9-49 on page 236 is the status panel for NIC eth4.
235
Figure 9-49 Eth4 status
The speed is reported as 4.0 Gbps (see Figure 9-14 on page 207) because we set the bandwidth to 40 on this vNIC in step 7 on page 229.
The ifconfig command

Another method of displaying MAC addresses is to use the ifconfig command. Figure 9-50 on page 237 shows the results of the ifconfig command. The MAC addresses are displayed in the Hwaddr field.
236
eth10
Link encap:Ethernet HWaddr 00:00:C9:B2:5B:3D inet addr:10.10.0.70 Bcast:10.10.0.255 Mask:255.255.255.0 UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:0 errors:0 dropped:0 overruns:0 frame:0 TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:0 (0.0 b) TX bytes:0 (0.0 b) Memory:95a80000-95aa0000 Link encap:Ethernet HWaddr 00:00:C9:B2:5B:41 inet addr:10.10.0.80 Bcast:10.10.0.255 Mask:255.255.255.0 UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:0 errors:0 dropped:0 overruns:0 frame:0 TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:0 (0.0 b) TX bytes:0 (0.0 b) Memory:95ac0000-95ae0000
eth11
Figure 9-50 Results of the ifconfig command for viewing MAC addresses
We confirmed that this MAC address belongs to vNIC 4 on 10-Gb physical port 0 (eth10) and vNIC 4 on 10-Gb physical port 1 (eth11). You can identify and verify the MAC addresses and link speeds of all other vNICs in the same manner.
9.3.2 Broadcom adapter configuration

The Broadcom 2-port 10-Gb Virtual Fabric Adapter is set up in a similar manner as the Emulex 10GbE Virtual Fabric Adapter II, with one difference: You do not specify a VLAN ID for the Broadcom adapter. The Broadcom adapter uses only MAC addresses for virtual channel traffic separation. Table 9-5 lists our settings for the Broadcom 2-port 10-Gb Virtual Fabric Adapter.
Table 9-9 Broadcom Virtual Fabric Adapter vNIC configuration NIC partition 1 2 3 4 Bandwidth 4 Gbps 3 Gbps 2 Gbps 1 Gbps MAC address 10:18:00:96:CE:90 10:18:00:96:CE:94 10:18:00:96:CE:98 10:18:00:96:CE:9C
237
You can identify the MAC addresses for each NIC partition using the UEFI utility (see Figure 9-21 on page 212). It is important to determine the MAC addresses so that you can identify each NIC partition in the operating system. Follow these steps to configure the Broadcom adapter parameters using the UEFI utility: 1. Power on or restart the blade server. 2. When prompted during POST, press F1, and the UEFI utility Main Menu displays (Figure 9-16).
Figure 9-51 The UEFI utility Main Menu
3. Select System Settings, then select Network on the subsequent panel to display the Network configuration window.
238
Figure 9-52 Network configuration panel
4. Select the first of the two Broadcom NetXtreme II 10 Gigabit Ethernet entries, and press Enter to display the Broadcom Main Configuration Page (Figure 9-53).
Figure 9-53 Broadcom Main Configuration panel
5. Select Device Configuration Menu and press Enter to display the Device Configuration panel (Figure 9-54 on page 240).
239
Figure 9-54 Device Configuration panel with NIC Partition set to Enabled
6. Ensure that NIC Partition is set to Enabled. 7. Once the NIC partition parameter is enabled, press Esc to return to the Broadcom Main Configuration Page; Figure 9-55.
Figure 9-55 New configuration option: the NIC Partitioning Configuration Menu
8. A new option is available now, NIC Partitioning Configuration Menu. Highlight it and press Enter. Figure 9-56 on page 241 appears.
240
9. The NIC Partition Configuration panel is shown in Figure 9-56. You can identify the MAC addresses of all four NIC partitions on this panel: PF#0 represents the first NIC partition: L2=10180096CE90, which means the MAC address is 10:18:00:96:CE:90 PF#2 is the second NIC partition: L2=10180096CE94, which means the MAC address is 10:18:00:96:CE:94 PF#4 is the third NIC partition: L2=10180096CE98, which means the MAC address is 10:18:00:96:CE:98 PF#6 is the fourth NIC partition: L2=10180096CE9C, which means the MAC address is 10:18:00:96:CE:9C It is important to know these MAC addresses so that you can match the NICs presented in the operating system to NIC partitions. 10.Select Global Bandwidth Allocation Menu and press Enter to display that menu (Figure 9-57 on page 242).
241
11.Set the bandwidth allocation parameters in this panel. For our example: Do not use Relative Bandwidth Weight, so leave those values at zero. Configure maximum bandwidth as follows: NIC partition 1: 4 Gbps (40% of total bandwidth) NIC partition 2: 3 Gbps (30% of total bandwidth) NIC partition 3: 2 Gbps (20% of total bandwidth) NIC partition 4: 1 Gbps (10% of total bandwidth)
12.Repeat the steps to configure NIC partitions and bandwidth parameters for the second 10-Gb physical port, starting from step 4 on page 239. 13.Save your changes before exiting the UEFI utility (Figure 9-58 on page 243).
242
Figure 9-58 Save settings before exiting the UEFI utility
NIC partitions in the operating system

We can define how the NIC partitions are seen by the operating system. Blade 7 runs on Windows Server 2008. In this section, we describe the Windows built-in networking tools and the Broadcom Advanced Control Suite (BACS).

Figure 9-59 on page 244 shows how the NIC partitions appear in the Network and Sharing Center.
243
Figure 9-59 Broadcom Virtual Fabric Adapter NIC partitions in Windows Server 2008
For clarity, we have changed the NIC names from the defaults according to Table 9-6. We used MAC addresses to match each NIC with the actual NIC partition in Table 9-10.
Table 9-10 NIC names and matching NIC partitions NIC name Broadcom VFA 01 Broadcom VFA 02 Broadcom VFA 03 Broadcom VFA 04 Broadcom VFA 11 Broadcom VFA 12 Broadcom VFA 13 10-Gb physical port 0 0 0 0 1 1 1 NIC partition 1 2 3 4 1 2 3 MAC address 10:18:00:96:CE:90 10:18:00:96:CE:94 10:18:00:96:CE:98 10:18:00:96:CE:9C 10:18:00:96:CE:92 10:18:00:96:CE:96 10:18:00:96:CE:9A
244
NIC name Broadcom VFA 14
10-Gb physical port 1
NIC partition 4
MAC address 10:18:00:96:CE:9E
1. On the Network and Sharing Center panel, click View Status for each partition to verify the parameters, such as link speed and MAC address (see Figure 9-60 for parameters of the NIC Broadcom VFA 01). As you can see, speed is reported as 4.0 Gbps, because we set the bandwidth to 40 on this NIC partition.
2. Click Details, and the MAC address of the selected NIC partition displays on the Network Connection Details panel (Figure 9-61 on page 246).
245
The MAC address matches vNIC 1 on port 0 as listed in Figure 9-53 on page 239
Figure 9-61 Network Connection Details panel for a selected NIC partition
We can confirm from Figure 9-26 that this is the MAC address of NIC partition 1 (the Broadcom DCM57712 NeXtreme II) on 10-Gb physical port 0. 3. Repeat the steps to verify the MAC address and speed for the other NIC partitions on the Broadcom Virtual Fabric Adapter (see Figure 9-24 on page 215 through Figure 9-26 on page 217).
Setting VLAN IDs in the operating system in Windows

You can define VLAN IDs for each NIC partition in the operating system. For example, follow these steps to define VLAN ID 10 for NIC Broadcom VFA 01. 1. Open the Broadcom VFA 01 Properties panel ()(Figure 9-62 on page 247)
246
Figure 9-62 Broadcom VFA 01 properties window
2. Click Configure to configure the vNIC. 3. Click the Advanced tab, Figure 9-63 on page 248 appears. Scroll the Property box and select VLAN ID.
247
Figure 9-63 VLAN ID configuration
4. Set the VLAN ID to 10 as shown in Figure 9-63. 5. Repeat these steps to configure the remaining VLAN IDs for the other NIC partitions on 10-Gb physical port 0, as follows: Broadcom VFA 02 - VLAN ID 20 Broadcom VFA 03 - VLAN ID 30 Broadcom VFA 04 - VLAN ID 40

The BACS management utility allows you to administer, configure, and diagnose the Broadcom 2-port 10-Gb Virtual Fabric Adapter and other Broadcom NICs. Figure 9-64 on page 249 shows an example of BACS, indicating eight NIC partitions in the left pane. NIC partition 1 on physical port 0 is highlighted. Note the speed: 4 Gbps (as was set in the bandwidth parameter in the UEFI utility).
248
Figure 9-64 BACS - NIC partition 1
On the Configurations tab, you can check and set NIC parameters (Figure 9-65 on page 250). Note that the VLAN ID is set to 10 (as in Figure 9-28 on page 219). Use the BACS utility to configure the VLAN ID to a different value, or to change any other parameter.
249
Figure 9-65 BACS - Configuration tab
On the Diagnostics tab, you can test a configuration by running a basic connectivity test (either ping to IP gateway or a user-specified IP address) (Figure 9-66 on page 251).
250
The Statistics tab displays network traffic counters, which are useful when monitoring or troubleshooting NIC behavior (Figure 9-67 on page 252).
251
NIC Teaming using BACS

One of the main functions of the BACS utility is to configure NIC teaming. Use NIC partitions in teaming configurations in the same way as with physical NICs. By clicking the Team Management frame, you can start the Teaming Wizard. We discuss this in 8.8.4, Configuring teams with Broadcom BACS on page 184.
9.3.3 Configuring the BNT Virtual Fabric 10-Gb Switch Module

You are now ready to configure VLANs on the BNT Virtual Fabric 10-Gb Switch Module. The HS22 blade server in slot 4 contains Emulex 10GbE Virtual Fabric Adapter II, which is configured to use VLANs 10, 20, 30, and 40. Internal (blade-facing)
252
port 4 (INT4) on the BNT switch module needs to pass traffic in all these VLANs, so VLAN tagging must be enabled on this port. The HS22 blade server in slot 8 contains Broadcom Virtual Fabric Adapter, configured to use VLANs 10, 20, 30 and 50. Internal (blade-facing) port 8 (INT8) on the BNT switch module needs to pass traffic in all these VLANs. This means VLAN tagging must be enabled on this port. External port 2 (EXT2) must pass traffic in VLANs 10 and 40, and external port 4 (EXT4) must pass traffic in VLANs 20 and 50. Both EXT2 and EXT4 must have tagging enabled. By default, VLAN tagging is enabled on internal ports, but disabled on external ports. So you need to enable tagging on ports EXT2 and EXT4. Figure 9-68 shows isCLI commands needed for this.
interface port EXT2 tagging exit interface port EXT4 tagging exit Figure 9-68 Enable tagging on EXT2 and EXT4
The VLAN structure is defined in Table 9-2 on page 196. The sequence of isCLI commands to configure these VLANs is shown in Figure 9-69 on page 254.
253
vlan 10 enable name "VLAN 10" member INT4 member INT8 member EXT2 vlan 20 enable name "VLAN 20" member INT4 member INT8 member EXT4 vlan 30 enable name "VLAN 30" member INT4 member INT8 vlan 40 enable name "VLAN 40" member INT4 member EXT2 vlan 50 enable name "VLAN 50" member INT8 member EXT4 Figure 9-69 VLAN definitions on BNT Virtual Fabric Switch Module
This completes configuration of the BNT Virtual Fabric Switch Module.
254
Abbreviations and acronyms

AC ACL AFT ALB AMM ASIC BACS BBI BOFM CD-ROM CDP CEE CLI DAC DC DCB DCBX DHCP DMA ETS ETSI alternating current access control list adapter fault tolerance adaptive load balancing Advanced Management Module application-specific integrated circuit Broadcom Advanced Control Suite browser-based interface BladeCenter Open Fabric Manager compact disc read only memory Cisco Discovery Protocol Converged Enhanced Ethernet command-line interface dual address cycle domain controller Data Center Bridging Data Center Bridging Capability Exchange Dynamic Host Configuration Protocol direct memory access Enhanced Technical Support European Telecommunications Standard Industry Fibre Channel full duplex Fast Etherchannel LED LPVID LR LSO MAC MB MSTP KVM LACP LAN LDAP IP IPTV IT ITSO ID IEEE IGMP FIP FTP GEC GT GUI HPC HSSM I/O IBM File Transfer Protocol Gigabit Etherchannel Gigatransfers graphical user interface high performance computing high speed switch module input/output International Business Machines identifier Institute of Electrical and Electronics Engineers Internet Group Management Protocol Internet Protocol Internet Protocol Television information technology International Technical Support Organization keyboard video mouse Link Aggregation Control Protocol local area network Lightweight Directory Access Protocol light emitting diode Logical Port VLAN ID long range Large send offload media access control megabyte Multiple Spanning Tree Protocol
FC FDX FEC
255
NAS NC-SI NEBS NGN NIC NTP OS OSI OSPF PCI PDF PE PFC POST PXE RADIUS RBAC RIP RMON ROM RSA RSS RSTP SAN SATA SFP SFT SLB SLES SNMP
network attached storage Network Controller Sideband Interface Network Equipment Building System next-generation network network interface card Network Time Protocol operating system Open Systems Interconnect Open Shortest Path First Peripheral Component Interconnect Portable Document Format Preinstallation Environment Priority-based Flow Control power-on self test Preboot eXecution Environment Remote Authentication Dial In User Service Role Based Access Control Routing Information Protocol Remote Monitoring read-only memory Remote Supervisor Adapter Receive-side scaling Rapid Spanning Tree Protocol storage area network Serial ATA small form-factor pluggable switch fault tolerance Smart Load Balancing SUSE Linux Enterprise Server Simple Network Management Protocol
SOL SR SSH STP TACACS TADDR TCP TCP/IP TFTP TOE TOR UDLD UDP UEFI USB UTP VE VFA VFS VGRP VLAN VM VNIC VOIP VPD VRRP WOL WRR XML XOR
Serial over LAN short range Secure Shell Spanning Tree Protocol Terminal Access Controller Access Control System TFTP address Transmission Control Protocol Transmission Control Protocol/Internet Protocol Trivial File Transfer Protocol TCP offload engine top-of-rack UniDirectional link detection user datagram protocol Unified Extensible Firmware Interface universal serial bus unshielded twisted pair Virtualization Engine Virtual Fabric Adapter Virtual Fabric Switch vNIC Group virtual LAN virtual machine virtual network interface card Voice over Internet Protocol vital product data virtual router redundancy protocol Wake on LAN weighted round-robin Extensible Markup Language exclusive or
256
Related publications
The publications listed in this section are considered particularly suitable for a more detailed discussion of the topics covered in this book.
IBM Redbooks
The following IBM Redbooks publications provide additional information about the topic in this document. Note that some publications referenced in this list might be available in softcopy only. Emulex 10GbE Virtual Fabric Adapter and Virtual Fabric Adapter Advanced for IBM BladeCenter, TIPS0748 BNT Virtual Fabric 10Gb Switch Module for IBM BladeCenter, TIPS0708 Cisco Nexus 4001I Switch Module for IBM BladeCenter, TIPS0754 Broadcom 2-port 10Gb Virtual Fabric Adapter for IBM BladeCenter, TIPS0827 Emulex 10GbE Virtual Fabric Adapter II for IBM BladeCenter, TIPS0828 You can search for, view, download or order these documents and other Redbooks, Redpapers, Web Docs, draft and additional materials, at the following website: ibm.com/redbooks
Other publications
These publications are also relevant as further information sources: Publications for the BNT Virtual Fabric 10Gb Switch Module http://ibm.com/support/entry/portal/docdisplay?lndocid=MIGR-5080917 Product publications for the Cisco Nexus 4001I Switch Module http://ibm.com/support/entry/portal/docdisplay?lndocid=MIGR-5082494 Installation Guide for IBM 10 Gb Ethernet Passthru Module http://ibm.com/support/entry/portal/docdisplay?lndocid=MIGR-5080966
257
Online resources
These websites are also relevant as further information sources: BNT Virtual Fabric 10 Gigabit switch module firmware update v6.5.3.0 http://ibm.com/support/entry/portal/docdisplay?lndocid=MIGR-5086594 Cisco support and downloads http://www.cisco.com/tac Emulex OneCommand Manager http://www.emulex.com/products/management-software/device-management /onecommand-manager/
Help from IBM

IBM Support and downloads ibm.com/support IBM Global Services ibm.com/services
258
Index
Numerics
46C7191 46M6071 49Y4235 49Y4265 49Y4275 49Y9983 81Y3133 90Y3550 90Y3566 28, 117 125 21 22, 26, 111 21 125 114 21, 26, 110111 21, 26, 111 telnet 147 BNT Virtual Fabric 10Gb Switch Module 27, 116 BBI 52, 153 BladeOS 45, 147 browser-based interface 52, 153 cables 35, 123 CLI selection 51 configuration tools 37 configuring 44, 146 configuring vNICs 15 create vNIC groups 63 creating vNICs 56 DAC cables 30, 118 direct-attached cables 28, 117 features 31, 119 firmware update 46, 148 IEEE standards 34, 123 isCLI privilege levels 51, 152 Layer 3 functions 33, 121 management 33, 121 performance 31, 119 ports 31, 119 publications 50 QoS 32, 121 redundancy 32, 120 reset 50 scalability 31, 119 scenarios 226 security 32, 120 SFP+ transceivers 28, 117 shipgroup 29, 117 stacking 17 transceivers 28, 30, 117118 VLANs 32, 120 BOFM See BladeCenter Open Fabric Manager Broadcom 2-port 10-Gb Virtual Fabric Adapter oversubscription 105 Switch Independent Mode 102 teaming 184 Broadcom 2-port 10Gb Virtual Fabric Adapter 114 bandwidth 145 Bandwidth Weight 163 Broadcom Advanced Control Suite 172, 175
A
architecture IBM Virtual Fabric Mode 11, 14 Switch Independent Mode 103
B
bandwidth 1 IBM Virtual Fabric Mode 13 Switch Independent Mode 105106, 139 BBI 52, 153 creating vNICs 56, 61 enable vNIC mode 57 examples 81 logon 53, 154 Quick Guide 55, 156 Switch Dashboard 54, 155 VLANs 69 BladeCenter Open Fabric Manager 7 IBM Virtual Fabric Mode 18 Switch Independent Mode 105, 135 BladeOS 45, 147 command reference 50, 152 creating vNICs 56, 62 enable vNIC mode 58 examples 81 logon 45, 147 VLANs 68 vNIC groups 64 BNT Virtual Fabric 10-Gb Switch Module examples 226 isCLI 51, 152
259
configuring 141 creating vNICs 161 Device Configuration Menu 143 features 105, 115 Global Bandwidth Allocation Menu 145 MAC address maximums 102 MAC addresses 115 Maximum Bandwidth 163 partitions 144 relative bandwidth 145 Broadcom Advanced Control Suite 172, 175 browser-based interface 52, 153
BNT Virtual Fabric 10Gb Switch Module 44, 146 Broadcom 2-port 10Gb Virtual Fabric Adapter 141 Cisco Nexus 4001I Switch Module 156 Emulex 10GbE Virtual Fabric Adapter II 38, 136 IBM Virtual Fabric Mode 37 creating vNICs IBM Virtual Fabric Mode 56 Switch Independent Mode 139, 144
D
double tag scheme IBM Virtual Fabric Mode 13 drivers IBM Virtual Fabric Mode 169 Switch Independent Mode 169
C
channels 5 characteristics IBM Virtual Fabric Mode 16 Switch Independent Mode 101 choosing the mode 6 Cisco Nexus 4001I Switch Module 124 BladeCenter Open Fabric Manager 124 cables 130 command reference 160 configuring 156 DAC cables 125 examples 197 FCoE 125, 128 firmware 157 IEEE standards 129 IPv6 support 129 license upgrade 125 performance 126 ports 126 publications 131 scenarios 197 security 128 Software Upgrade License 125 telnet 157 transceivers 125 VLAN support 127 VLANs 157 comparison 5 compatibility 2 components IBM Virtual Fabric Mode 20 Switch Independent Mode 109 concept 2 configuring
E
elxflash 38, 136 Emulex 10GbE Virtual Fabric Adapter 21, 25 comparison 27 configuring 38 features 22 firmware 38 minimum bandwidth 18 modes supported 23 UEFI interface 39 vNIC mode in UEFI 41 Emulex 10GbE Virtual Fabric Adapter II 21, 110 configuring 38 creating vNICs 164 driver 171 features 105, 112 firmware 38, 135 minimum bandwidth 18 modes of operation 26 Multichannel Configuration 42 Switch Independent Mode 102 UEFI 164 UEFI interface 39 vNIC mode in UEFI 41 Emulex OneCommand Manager 74, 179 examples BNT Virtual Fabric 10-Gb Switch Module 226 Cisco Nexus 4001I Switch Module 197 IBM Virtual Fabric Mode 81 Switch Independent Mode 193
260
existing tagging 6
F
failover 6 IBM Virtual Fabric Mode 15 Switch Independent Mode 107, 169 FCoE 6, 8 firmware BNT Virtual Fabric 10Gb Switch Module 46, 148 Cisco Nexus 4001I Switch Module 157 Emulex 10GbE Virtual Fabric Adapter II 135 Emulex 10GbE Virtual Fabric Adapters 38 Function 139
G
Global Bandwidth Allocation Menu 145 Global Configuration Mode 51, 153 groups 6 gtimg command 49, 150
H
hardware IBM Virtual Fabric Mode 19 Switch Independent Mode 109 high availability IBM Virtual Fabric Mode 70 Switch Independent Mode 169 hypervisor 4
I
IBM Virtual Fabric Mode architecture 11, 14 bandwidth 13 BladeCenter Open Fabric Manager 18 BNT switch configuration 44 BNT Virtual Fabric 10Gb Switch Module 27 characteristics 16 comparison 5 components 20 configuring the solution 37 creating vNICs 56 double tag scheme 13 Emulex 10GbE Virtual Fabric Adapter 21, 25 Emulex 10GbE Virtual Fabric Adapter II 21 enabling vNICs 56 examples 81
failover 15 firmware updates 38, 46 hardware 2, 19 high availability 70 introduced 2 isCLI 51 Layer 2/3 functions disabled 13 LPVID 44 mapping of vNICs 15 minimum bandwidth 18 Multichannel Configuration 43 optics 20 outer tag 12 overview 12 planning 17 products 19 rules 16 scenarios 81 selection 6 SFP+ transceivers 20 stacking 14, 17 teams 73 transceivers 20 trunk failover 15 trunks 18, 70 virtual channels 12 VLAN tagging 12 VLANs, configuring 66 VMready 17, 69, 169 vNIC group creation 63 vNIC groups 18 vNIC mode 13 vSwitches 13 IEEE 802.1q 3, 6 VLAN tag 101 isCLI BNT Virtual Fabric 10Gb Switch Module 51, 152 command reference 52, 153 creating vNICs 56, 61 enable vNIC mode 57 examples 81 VLANs 68 vNIC groups 65 isCLI privilege levels 51, 152 iSCSI 6 Switch Independent Model 108
Index
261
J
jumbo frames 22
L
Layer 2/3 functions disabled IBM Virtual Fabric Mode 13 limits 5 Logical Port VLAN ID IBM Virtual Fabric Mode 44 Switch Independent Mode 139 LPVID IBM Virtual Fabric Mode 44 Switch Independent Mode 139
Switch Independent Mode 105 Privileged 51 Privileged EXEC mode 51, 152 products IBM Virtual Fabric Mode 19 Switch Independent Mode 109
R
Redbooks website 257 Contact us xii rules IBM Virtual Fabric Mode 16 Switch Independent Mode 108
M
mapping of vNICs IBM Virtual Fabric Mode 15 Switch Independent Mode 104 maximum VLANs 6 minimum bandwidth IBM Virtual Fabric Mode 18 Switch Independent Mode 106 modes 2 Multichannel 138 Multichannel Configuration 42
S
scenarios BNT Virtual Fabric 10-Gb Switch Module 226 Cisco Nexus 4001I Switch Module 197 IBM Virtual Fabric Mode 81 Switch Independent Mode 193 selection of the mode 6 SFP+ transceivers IBM Virtual Fabric Mode 20 Switch Independent Mode 118, 125 stacking 6, 14 IBM Virtual Fabric Mode 17 Switch Independent Mode 108 Switch Independent Mode 101 about 101 architecture 103 bandwidth 105106, 139 bandwidth limits 101 BladeCenter Open Fabric Manager 105, 135 BladeOS 45 BNT Virtual Fabric 10Gb Switch Module 109 Broadcom 2-port 10-Gb Virtual Fabric Adapter 102, 109, 141 Broadcom 2-port 10Gb Virtual Fabric Adapter 184 Broadcom Advanced Control Suite 172, 175 Cisco Nexus 4001I Switch Module 110 comparison 5 configuration 136 configuring 133 creating vNICs 160 drivers 169 Emulex 10GbE Virtual Fabric Adapter II 102, 109
O
OneCommand 74 optics IBM Virtual Fabric Mode 20 Switch Independent Mode 118, 125 outer tag IBM Virtual Fabric Mode 12 Switch Independent Mode 101 oversubscription Broadcom 2-port 10-Gb Virtual Fabric Adapter 105 overview 4 IBM Virtual Fabric Mode 12 Switch Independent Mode 103
P
partitions Broadcom 2-port 10Gb Virtual Fabric Adapter 144 planning IBM Virtual Fabric Mode 17
262
Emulex OneCommand Manager 179 examples 193 failover 107, 169 features 105 firmware 135 firmware updates 148 Function 139 hardware 3, 109 high availability 169 IEEE 802.1Q VLAN tag 101 introduction 3 isCLI 152 iSCSI 108 Logical Port VLAN ID 139 LPVID 139 MAC addresses 115 Multichannel 138 Multichannel Configuration 138 outer tag 101 overview 103 planning 105 products 109 scenarios 193 selection 7 stacking 108 trunk failover 169 UEFI configuration 136 VLAN groups 107 VLAN ID 139 VLAN tagging 101 VLANs 104, 167 VMready 108, 169 vNIC2 mode 138
IBM Virtual Fabric Mode 15 Switch Independent Mode 169 trunking 6 trunks IBM Virtual Fabric Mode 18, 70
U
User EXEC mode 51, 152
V
virtual channels 5 virtual groups 6 virtual NIC See vNIC virtualization 1 VLAN groups IBM Virtual Fabric Mode 18 Switch Independent Mode 107 VLAN ID Switch Independent Mode 139 VLAN tagging 2 IBM Virtual Fabric Mode 12 Switch Independent Mode 101 VLANs maximum 6 Switch Independent Mode 167 VMready 69, 169 IBM Virtual Fabric Mode 17 Switch Independent Mode 108, 169 vNIC overview 4 vNIC groups IBM Virtual Fabric Mode 18 vNIC2 See Switch Independent Mode selecting 138 vSwitches 13
T
teaming Broadcom 2-port 10Gb Virtual Fabric Adapter 184 creating 74 Emulex 10GbE Virtual Fabric Adapter II 179 IBM Virtual Fabric Mode 73, 179 telnet 45 BNT Virtual Fabric 10-Gb Switch Module 147 Cisco Nexus 4001I Switch Module 157 total ports 4 transceivers IBM Virtual Fabric Mode 20 trunk failover
Index
263
264
(0.5 spine) 0.475<->0.875 250 <-> 459 pages
Back cover
Covering IBM Virtual Fabric Mode and Switch Independent Mode Concepts and benefits of virtual NICs (vNICs) Step-by-step instructions for creating various vNIC solutions
The deployment of server virtualization technologies in data centers requires significant efforts in providing sufficient network I/O bandwidth to satisfy the demand of virtualized applications and services. For example, every virtualized system can host several dozen network applications and services, and each of these services requires certain bandwidth (or speed) to function properly. Furthermore, because of different network traffic patterns relevant to different service types, these traffic flows may interfere with each other, leading to serious network problems including the inability of the service to perform its functions. The IBM Virtual Fabric solution for IBM BladeCenter addresses these issues. The solution is based on the IBM BladeCenter H chassis with a 10-Gb Converged Enhanced Ethernet infrastructure built on 10-Gb Ethernet switch modules in the chassis and the Emulex or Broadcom Virtual Fabric Adapters in each blade server. This book is for clients who want to learn how to implement an IBM Virtual Fabric solution, either the IBM Virtual Fabric Mode offering or the Switch Independent Mode offering. It provides step-by-step instructions for configuring the adapters and switches.
INTERNATIONAL TECHNICAL SUPPORT ORGANIZATION
BUILDING TECHNICAL INFORMATION BASED ON PRACTICAL EXPERIENCE IBM Redbooks are developed by the IBM International Technical Support Organization. Experts from IBM, Customers and Partners from around the world create timely technical information based on realistic scenarios. Specific recommendations are provided to help you implement IT solutions more effectively in your environment.
For more information: ibm.com/redbooks

SG24-7966-00 ISBN 0738435961

IBM Blade Center Virtual Fabric

Uploaded by

Document Information

Original Description:

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

IBM Blade Center Virtual Fabric

Uploaded by

Copyright:

Available Formats

Front cover

IBM BladeCenter Virtual Fabric Solutions

David Watts Jure Arzensek Valeria Lima Norm Strole

Copyright IBM Corp. 2011. All rights reserved.

IBM BladeCenter Virtual Fabric Solutions

IBM BladeCenter Virtual Fabric Solutions

Copyright IBM Corp. 2011. All rights reserved.

IBM BladeCenter Virtual Fabric Solutions

The team who wrote this book

Copyright IBM Corp. 2011. All rights reserved.

The team (l-r): David, Jure, Valeria, and Norm

IBM BladeCenter Virtual Fabric Solutions

Now you can become a published author, too!

Stay connected to IBM Redbooks

IBM BladeCenter Virtual Fabric Solutions

Copyright IBM Corp. 2011. All rights reserved.

IBM BladeCenter Virtual Fabric Solutions

1.1 Virtual NICs: overview

IBM BladeCenter Virtual Fabric Solutions

1.2 Comparing modes

1.3 Selecting the mode

IBM BladeCenter Virtual Fabric Solutions

1.4 Use of BladeCenter Open Fabric Manager

1.5 The future of converged fabrics and FCoE capabilities

1.6 Using this book

IBM BladeCenter Virtual Fabric Solutions

IBM Virtual Fabric Mode

Copyright IBM Corp. 2011. All rights reserved.

IBM BladeCenter Virtual Fabric Solutions

Solution architecture - IBM Virtual Fabric Mode

Copyright IBM Corp. 2011. All rights reserved.

Virtual Fabric Mode Overview

BNT 10G VFS

Emulex 10G VFA I or II Customer VLAN tag 10G Link

vSwitch Group 22 Emulex 10G VFA I or II 10G Link vSwitch Group 33

Note: Inner tag denotes customer VLAN (optional)

IBM BladeCenter Virtual Fabric Solutions

Chapter 2. Solution architecture - IBM Virtual Fabric Mode

IBM BladeCenter Virtual Fabric Solutions

Components and connectivity inside the IBM BladeCenter H chassis

BNT Virtual Fabric Switch (Bay 7)

Blade Server 14 Blade Server 1

INT1.1 INT1.2 INT1.3 INT1.4

BNT Virtual Fabric Switch (Bay 9)

Figure 2-2 BladeCenter Virtual Fabric vNIC solution diagram

Chapter 2. Solution architecture - IBM Virtual Fabric Mode

IBM BladeCenter Virtual Fabric Solutions

2.3 Planning considerations

Chapter 2. Solution architecture - IBM Virtual Fabric Mode

IBM BladeCenter Virtual Fabric Solutions

Supported products - IBM Virtual Fabric Mode

Copyright IBM Corp. 2011. All rights reserved.

This chapter describes the features of these products.

IBM BladeCenter Virtual Fabric Solutions

3.2 Emulex 10Gb Virtual Fabric Adapters

3.2.1 Emulex 10GbE Virtual Fabric Adapter

Chapter 3. Supported products - IBM Virtual Fabric Mode

Description Emulex 10GbE Virtual Fabric Adapter Advanced Upgrade

Part number 49Y4265

Feature code 2436

IBM BladeCenter Virtual Fabric Solutions

Chapter 3. Supported products - IBM Virtual Fabric Mode

The Emulex 10GbE Virtual Fabric Adapter is shown in Figure 3-1.