Front cover

IBM BladeCenter Virtual Fabric Solutions
Covering IBM Virtual Fabric Mode and Switch Independent Mode Concepts and benefits of virtual NICs (vNICs) Step-by-step instructions for creating various vNIC solutions

David Watts Jure Arzensek Valeria Lima Norm Strole

ibm.com/redbooks

International Technical Support Organization IBM BladeCenter Virtual Fabric Solutions August 2011

SG24-7966-00

Note: Before using this information and the product it supports, read the information in “Notices” on page vii.

First Edition (August 2011) This edition applies to the following IBM BladeCenter I/O options: Emulex 10GbE Virtual Fabric Adapter Emulex 10GbE Virtual Fabric Adapter II Broadcom 2-port 10Gb Virtual Fabric Adapter BNT Virtual Fabric 10Gb Switch Module Cisco Nexus 4001I Switch Module

© Copyright International Business Machines Corporation 2011. All rights reserved. Note to U.S. Government Users Restricted Rights -- Use, duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp.

Contents
Notices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . vii Trademarks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . viii Preface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ix The team who wrote this book . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ix Now you can become a published author, too! . . . . . . . . . . . . . . . . . . . . . . . . . xi Comments welcome. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xii Stay connected to IBM Redbooks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xii Chapter 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 1.1 Virtual NICs: overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4 1.2 Comparing modes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 1.3 Selecting the mode . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 1.4 Use of BladeCenter Open Fabric Manager . . . . . . . . . . . . . . . . . . . . . . . . . 7 1.5 The future of converged fabrics and FCoE capabilities. . . . . . . . . . . . . . . . 8 1.6 Using this book . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8 Part 1. IBM Virtual Fabric Mode . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9 Chapter 2. Solution architecture - IBM Virtual Fabric Mode . . . . . . . . . . . 11 2.1 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12 2.2 Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14 2.3 Planning considerations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17 Chapter 3. Supported products - IBM Virtual Fabric Mode . . . . . . . . . . . . 19 3.1 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20 3.2 Emulex 10Gb Virtual Fabric Adapters . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21 3.2.1 Emulex 10GbE Virtual Fabric Adapter . . . . . . . . . . . . . . . . . . . . . . . 21 3.2.2 Emulex 10GbE Virtual Fabric Adapter II . . . . . . . . . . . . . . . . . . . . . . 25 3.3 BNT Virtual Fabric 10Gb Switch Module . . . . . . . . . . . . . . . . . . . . . . . . . . 27 Chapter 4. Configuring the solution - IBM Virtual Fabric Mode . . . . . . . . 37 4.1 Configuring the Emulex 10GbE Virtual Fabric Adapter . . . . . . . . . . . . . . . 38 4.1.1 Upgrading firmware . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38 4.1.2 Configuring the adapter in the UEFI interface . . . . . . . . . . . . . . . . . . 39 4.2 Configuring the BNT Virtual Fabric 10Gb Switch Module . . . . . . . . . . . . . 44 4.2.1 BladeOS CLI . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45 4.2.2 isCLI . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51 4.2.3 Browser-based interface (BBI) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52

© Copyright IBM Corp. 2011. All rights reserved.

iii

. . . . . . . . . . . . . . . . 103 6. . . . . . 153 8. . .1. . . . . . . . . . . . . . . . .Switch Independent Mode . . . .2. . . . . . . .8. . . . . . . . . . . . . . . . . . .1. . . . . .3. . . . . . . . . . . . . . . . . . . . . . . 76 Chapter 5. .2 Virtual Fabric switch modules . . . . . . . . . . . . 147 8. . . .1 BladeOS CLI . . . . . . . . . . . . . . . . . . . . no uplinks . . . . . . . . . . . . . . . . . . . 105 Chapter 7. . . . . . . . . . . . . . . . . . . . . .3 Browser-based interface (BBI) . . . . 63 4. . . . . . . . . . . . . . . . . . . . . . . . . .3 Creating the vNICs . 72 4. 56 4. . . 156 iv IBM BladeCenter Virtual Fabric Solutions . . . . . . . . . . 135 8. . . . . . . . . . . . . . . . . Supported products . . 110 7.2. . . Configuring the solution . . . . . . . . . . . .1 . . . 60 4. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 116 7. . . . .2 Firmware upgrade of Emulex 10GbE Virtual Fabric Adapter II . . . . . . . 70 4. . . . . .1 Configuring Virtual Fabric adapters . . . 110 7.2 Broadcom 2-port 10-Gb Virtual Fabric Adapter. . . 124 Chapter 8. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 133 8. . . . . . . 73 4. . . . . . .2. . . . . . . . . . . . . . . . . . .1 Example 1: vNICs only. . . . . . . . . .1 BNT Virtual Fabric 10Gb Switch Module. . . . . . . . . 136 8. . . . . . . . . . . . . . one uplink trunk. . . . . . . . .3. . . . . . . . . . .2 isCLI . . . . . Solution architecture . . . . . . . . . . . .1.6 vNIC and VMready . . . . . . . . . . . . . Switch Independent Mode . . . . . . . . . . . . . . . . . . . . . . . . . . . 91 Part 2. . 82 5. . . . . . . . . . . . . . . . . .Switch Independent Mode. . . . . .1 Solution overview . . . . . . . .2. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .3 Creating virtual NICs . . . . . . . . . . . . . . . . . . . . . . . . . 141 8. . . . . . . . . . . one uplink . . . . . . . . Examples of use . . . .1 Before you configure . . . . . . . . . . .3 Configuring the Emulex 10GbE Virtual Fabric Adapter II in UEFI . .1. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 86 5. . . . . . . . . . .8. . . . . . . . . . . . . . . . . . . . . .4. . .4 vNIC group configuration. . . . . . .1 Virtual Fabric adapters . . .8 Configuring the operating system . . . . . . . . . . . . . . . . . . . . . . . .1 Configuring teams and VLANs . . . . 56 4.1. . . . . . . .IBM Virtual Fabric Mode . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 135 8. . . 152 8. . . . . . . . . . . . . . . . . . . . . . . . . . 81 5. .5 Configuring vNIC VLANs. . . . . . . . . . . . . . . . . . . . . . .8. . . . . .1 Using BOFM Advanced V4. . . .4 Configuring the Broadcom 2-port 10-Gb Virtual Fabric Adapter in UEFI .1.2 Enabling and disabling vNIC functionality on the server . . . . . . . . . . . . . 56 4. . .3. . . .2 Features . 116 7. . 69 4. . . . .2 Example 2: vNICs only. . . . 146 8. . . . . . . . .2. . . . . . . 74 4. . 114 7. . . . . . . . . . . . . . . . . . . 109 7. . . . . . . . . . . . . . . . . 66 4. . . . . . . . . . . . . . . . . . . . . . . . 101 6. . . . . . .2 Configuring the BNT Virtual Fabric 10Gb Switch Module . . . . . . . . . . . . . . . . . . . . .7 High availability and vNICs . . . . . . 135 8. . . .Switch Independent Mode . . . . . . .3 Example 3: vNICs only. . . . . . . . . . . . . . . .2 Configuring a new team . . . . . . . . .3 Cisco Nexus 4001I Switch Module configuration . . .1 Emulex 10GbE Virtual Fabric Adapter II . . . . . .3 Configuring a VLAN for a physical or team adapter . . . . 99 Chapter 6. . . . . . . . . . .2 Cisco Nexus 4001I Switch Module . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . .1. . . .8. . . . . . . . . .8. . . . . . . . . . . . 255 Related publications .2 Cisco Nexus 4001I Switch Module configurations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .4. . . . . . . . . . . . . . . . . . . . . . .Switch Independent Mode . . . . 170 8. . . . . . . . . . . . . 259 Contents v . . . . . . . .3 Configuring teams with Emulex OneCommand Manager . . . . . . . . . .2. 226 9. . . . . . . . . 197 9.3 Configuring the BNT Virtual Fabric 10-Gb Switch Module . . . . . . . . . . . . . . . .3. . . . . . . . . . . 179 8. . . . . . . . . . 164 8. . . . . . . . . . . . . . . . . . . . . . . 167 8.3. . . . . . . . . . . . . . 197 9. . . . . . . . . .6 vNIC and VMready . . . . . . . . . . . . . . . . . . . . . . . . . 257 Online resources . . . . . . . . .8. . . .1 Broadcom 2-port 10-Gb Virtual Fabric Adapter. . . . . . . .2 Broadcom adapter configuration .1 Configurations overview . . 224 9. . . . . . . 172 8. . . . . . . . . . Examples of use . . . . . . . . . . . . . 198 9.2 Emulex 10GbE Virtual Fabric Adapter II . . . . . . . . . . . . 195 9. . . . . . . . .3 BNT Virtual Fabric 10-Gb Switch Module configurations . . . . . . . . . . .4 Configuring teams with Broadcom BACS . . . . . . . . . .8 Configuring the operating systems . . . . .1 Virtual LAN (VLAN) definitions . . . . . . . . . . . . . . 257 IBM Redbooks . . . . . 169 8. . . . . . . . . . 258 Index .1 vNICs summary. . . . . . . . . . . . . . . . . . . .1 Emulex adapter configuration . . . . . . . . . . .8. . . . . 237 9. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 257 Other publications . . . . .4 Cisco Nexus 4001I Switch Module configuration . . . . 208 9. . . . . . . . . . . . 169 8. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .2 vNICs on the Broadcom 2-port 10-Gb Virtual Fabric Adapter . . . . . . .1 vNICs on the Emulex 10GbE Virtual Fabric Adapter II . . . . . 160 8. . . . . . . . . . . 226 9. . . . . . . . . . . . . . . . . . . . . . . . .2. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 193 9. . . . . . . . . . . . . . . . . . 196 9.2 Emulex adapter configuration . . . . . . . . . . . . . . . . . . . .3. . . . . . . . . . . . . . . . . . . . . . . . .7 High availability and vNICs . . . . . . . . . . . . . . . . 161 8. . . . . . . . . . .4. . .5 Configuring vNIC VLANs. . . . . . . . .2. . . . . . . . . . 169 8. . . .1 Firmware upgrade . . . . . . . . . . . . . . .2 Switch/adapter vNIC correlation . . . . . . . . . .3 Broadcom adapter configuration . . .8. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .4 Creating virtual NICs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 194 9. . . . . . . . . . . .2. . . . . . . . . . . . . . . . . . . . . . . . . . . .1. . . . . . . . . . . . . . . . . 252 Abbreviations and acronyms . . . . . . . . . . . . . . . 258 Help from IBM . . . . . . . 157 8. . . . . . . . . . 184 Chapter 9. .3. . . . . . .

vi IBM BladeCenter Virtual Fabric Solutions .

or service that does not infringe any IBM intellectual property right may be used instead. THE IMPLIED WARRANTIES OF NON-INFRINGEMENT. These examples have not been thoroughly tested under all conditions. This information could include technical inaccuracies or typographical errors. modify. IBM may use or distribute any of the information you supply in any way it believes appropriate without incurring any obligation to you. vii . You can send license inquiries. these changes will be incorporated in new editions of the publication. or features discussed in this document in other countries. BUT NOT LIMITED TO. the examples include the names of individuals. program. or function of these programs. IBM has not tested those products and cannot confirm the accuracy of performance. which illustrate programming techniques on various operating platforms. marketing or distributing application programs conforming to the application programming interface for the operating platform for which the sample programs are written. All rights reserved. IBM. program. Any reference to an IBM product. Changes are periodically made to the information herein. EITHER EXPRESS OR IMPLIED. their published announcements or other publicly available sources. and distribute these sample programs in any form without payment to IBM. therefore. To illustrate them as completely as possible. therefore. The furnishing of this document does not give you any license to these patents. This information contains examples of data and reports used in daily business operations. to: IBM Director of Licensing. compatibility or any other claims related to non-IBM products. services.A.Notices This information was developed for products and services offered in the U. NY 10504-1785 U. this statement may not apply to you. All of these names are fictitious and any similarity to the names and addresses used by an actual business enterprise is entirely coincidental. Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products. serviceability. in writing. for the purposes of developing. it is the user's responsibility to evaluate and verify the operation of any non-IBM product. © Copyright IBM Corp. brands.S. MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.A. IBM may have patents or pending patent applications covering subject matter described in this document. or service is not intended to state or imply that only that IBM product. or service may be used. However. The following paragraph does not apply to the United Kingdom or any other country where such provisions are inconsistent with local law: INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS PUBLICATION "AS IS" WITHOUT WARRANTY OF ANY KIND. 2011. COPYRIGHT LICENSE: This information contains sample application programs in source language. Any functionally equivalent product. Consult your local IBM representative for information on the products and services currently available in your area. Some states do not allow disclaimer of express or implied warranties in certain transactions. IBM may make improvements and/or changes in the product(s) and/or the program(s) described in this publication at any time without notice. You may copy. companies. program.S. IBM may not offer the products. program. The materials at those websites are not part of the materials for this IBM product and use of those websites is at your own risk. INCLUDING. Armonk. Information concerning non-IBM products was obtained from the suppliers of those products. North Castle Drive. IBM Corporation. Any references in this information to non-IBM websites are provided for convenience only and do not in any manner serve as an endorsement of those websites. using. cannot guarantee or imply reliability. and products. or service.

Itanium. indicating US registered or common law trademarks owned by IBM at the time this information was published. viii IBM BladeCenter Virtual Fabric Solutions . an IBM Company. Intel Inside. and Pentium are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States and other countries. Intel logo. other countries. or both. other countries. Intel SpeedStep. Intel Inside logo. Intel Centrino.Trademarks IBM. or both.com are trademarks or registered trademarks of International Business Machines Corporation in the United States.ibm. or service names may be trademarks or service marks of others. Intel Inside logo. or both: AIX® BladeCenter® DS4000® IBM® Redbooks® Redbooks (logo) System Storage® System x® ® The following terms are trademarks of other companies: BNT. other countries. A current list of IBM trademarks is available on the Web at http://www. Intel Centrino logo. Celeron. other countries. Such trademarks may also be registered or common law trademarks in other countries. Other company. and ibm. These and other IBM trademarked terms are marked on their first occurrence in this information with the appropriate symbol (® or ™).shtml The following terms are trademarks of the International Business Machines Corporation in the United States. Intel.. Microsoft. Linux is a trademark of Linus Torvalds in the United States. VMready. Windows. Inc. the IBM logo. or both. and Intel Centrino logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States and other countries. and the Windows logo are trademarks of Microsoft Corporation in the United States.com/legal/copytrade. Intel logo. Intel. and Server Mobility are trademarks or registered trademarks of Blade Network Technologies. Intel Xeon. product.

leading to serious network problems including the inability of the service to perform its functions. He holds a Bachelor of Engineering degree from the University of Queensland (Australia) and has worked for IBM both in the US and Australia since 1989.Preface The deployment of server virtualization technologies in data centers requires significant efforts in providing sufficient network I/O bandwidth to satisfy the demand of virtualized applications and services. ix . these traffic flows may interfere with each other. David Watts is a Consulting IT Specialist at the IBM ITSO Center in Raleigh. every virtualized system can host several dozen network applications and services. This book is for clients who want to learn how to implement an IBM Virtual Fabric solution. and each of these services requires certain bandwidth (or speed) to function properly. The IBM® Virtual Fabric solution for IBM BladeCenter® addresses these issues. For example. The solution is based on the IBM BladeCenter H chassis with a 10-Gb Converged Enhanced Ethernet infrastructure built on 10-Gb Ethernet switch modules in the chassis and the Emulex or Broadcom Virtual Fabric Adapters in each blade server. papers. His other areas of expertise © Copyright IBM Corp. David is an IBM Certified IT Specialist and a member of the IT Specialist Certification Review Board. He has authored over 80 books. Jure Arzensek is an Advisory IT Specialist at IBM Slovenia and works for the EMEA level 2 BladeCenter support team. It provides step-by-step instructions for configuring the adapters and switches. He manages residencies and produces IBM Redbooks® publications on hardware and software topics related to IBM System x® and BladeCenter servers and associated client platforms. because of different network traffic patterns relevant to different service types. Jure holds a degree in Computer Science from the University of Ljubljana. 2011. either the IBM Virtual Fabric Mode offering or the Switch Independent Mode offering. All rights reserved. Furthermore. and technotes. The team who wrote this book This book was produced by a team of specialists from around the world working at the International Technical Support Organization. He has worked with IBM BladeCenter products since 2002 and has been with IBM since 1995. Raleigh Center.

SAN. He has co-authored eight other IBM Redbooks publications. He has written extensively on the IBM Token Ring. degree in Electrical Engineering from Duke University. North Carolina. with publication in both the IBM Journal of Research and Development and the IBM Systems Journal. The team (l-r): David. He is an IBM Master Inventor with 30 years of experience in the networking field. System Storage® DS3000 and DS4000® products and network operating systems for the Intel platform. He holds a Ph. Jure.D. and Norm Thanks to the following people for their contributions to this project: From IBM Marketing: Richard Mancini Shekhar Mishra Michael Talplacido From IBM Development: Manzar Alam Ralph Geiger Marc Stracuzza Tony Tran x IBM BladeCenter Virtual Fabric Solutions .include IBM System x servers. Valeria. Valeria Lima was a member of STG Lab Services in IBM Brazil at the time of writing. His areas of expertise include Blade networking subsystems and networking protocols. Norm Strole is a Senior Technical Staff Member in Research Triangle Park.

com/redbooks/residencies.From Broadcom: Kevin Liebl Maziar Tamadon From Emulex: Tom Boucher Leanne Greene From IBM Redbooks: Karen Lawrence Linda Robinson Alfred Schwab This book is based in part on the document IBM BladeCenter Virtual Fabric Solutions. Thanks to its authors: Tom Boucher Scott Lorditch Mark McCool David Watts Now you can become a published author. Find out more about the residency program. and become a published author—all at the same time! Join an ITSO residency project and help write a book in your area of expertise. too! Here's an opportunity to spotlight your skills. and apply online at: ibm. while honing your experience using leading-edge technologies. browse the residency index. as you expand your network of technical contacts and relationships.html Preface xi . Your efforts will help to increase product acceptance and customer satisfaction. grow your career. Residencies run from two to six weeks in length. and you can participate either in person or as a remote resident working from your home base. REDP-4673.

com/Redbooks. International Technical Support Organization Dept. Send us your comments about this book or other IBM Redbooks publications in one of the following ways: Use the online Contact us review Redbooks form found at: ibm.linkedin. residencies. and workshops with the IBM Redbooks weekly newsletter: https://www.ibm.ibm.com/redbooks Send your comments in an email to: redbooks@us.com/IBMRedbooks Follow us on Twitter: http://twitter.facebook.com/rss.html xii IBM BladeCenter Virtual Fabric Solutions .com/groups?home=&gid=2130806 Explore new Redbooks publications. NY 12601-5400 Stay connected to IBM Redbooks Find us on Facebook: http://www.Comments welcome Your comments are important to us! We want our books to be as helpful as possible.redbooks. HYTD Mail Station P099 2455 South Road Poughkeepsie.com/ibmredbooks Look for us on LinkedIn: http://www.ibm.com Mail your comments to: IBM Corporation.redbooks.nsf/subscribe?OpenForm Stay current on recent Redbooks publications with RSS Feeds: http://www.

and the bandwidth per interface is static with a maximum bandwidth of 1 Gb per flow. and each of these services requires a certain bandwidth to function properly. these traffic flows might interfere with each other. of the maximum of eight physical ports per blade. For example. Introduction The deployment of server virtualization technologies in data centers requires significant efforts to provide sufficient network I/O bandwidth (or speed) to satisfy the demand of virtualized applications and services. Furthermore. including the inability of the service to perform its functions. All rights reserved. Providing sufficient bandwidth and isolation to virtualized applications in a 1-Gb network infrastructure might be particularly challenging for blade-based deployments in which the number of physical I/O ports per blade is limited. leading to serious network problems. The IBM BladeCenter Virtual Fabric solutions address these issues by increasing the number of available Ethernet ports and providing more flexibility in allocating the available bandwidth to meet specific application requirements. because of different network traffic patterns relevant to different service types.1 Chapter 1. every virtualized system can host several dozen network applications and services. at least two ports are usually used to connect to the external storage. with a total network bandwidth of 6 Gb per blade for Gigabit Ethernet infrastructures. For example. 1 . In addition. thus limiting flexibility of bandwidth usage. 2011. © Copyright IBM Corp. only six 1-Gb Ethernet ports are available. Therefore. traffic flows are isolated on a physical port basis. This leaves no room for future growth.

this mode supports bidirectional bandwidth control. Dynamic bandwidth support allows the channel bandwidth to be increased or decreased under the control of the VFS without resetting the VFA. 49Y4235 Emulex 10GbE Virtual Fabric Adapter Advanced. starting at either 100 MB or 1 Gb (adapter-dependent). Channel bandwidth is configured in the BNT® Virtual Fabric switch (VFS) for each port and passed to each of the VFAs. Also. The separation of the virtual channels is continued into the switch module utilizing VLAN tagging to identify the various data streams. each blade server has up to 20 Gb of available network bandwidth (two 10-Gb ports) and the ability to split this bandwidth among up to eight vNICs. 49Y4275 Emulex 10GbE Virtual Fabric Adapter II. 81Y3133 BNT Virtual Fabric 10Gb Switch Module Supported Supported Supported Supported Not supported Cisco Nexus 4001I Switch Module Not supported Not supported Not supported Not supported Not supported In this configuration. respectively. such that the total of all eight vNICs does not exceed the 20-Gb total that the two 10-Gb ports offer. This configuration allows each vNIC or virtual channel to be between 100 MB and 10 Gb in increments of 100 MB. with both the server outbound and inbound virtual channel bandwidth limits being enforced by the VFA and VFS. Two operational modes are available. dependent upon the type of 10-Gb expansion card installed in the server and the type of 10-Gb switch installed in the BladeCenter chassis: IBM Virtual Fabric Mode IBM Virtual Fabric Mode can be enabled on the combinations listed in Table 1-1. Table 1-1 Supported combinations using IBM Virtual Fabric Mode Supported switches   Supported adapters Emulex 10GbE Virtual Fabric Adapter. The basic concept of the BladeCenter Virtual Fabric solution is to take the two 10-Gb ports that are on a 10-Gb VFA and split them into eight vNICs. up to a maximum of 10 Gb per one vNIC channel with 100-MB increments. 90Y3566 Broadcom 2-port 10Gb Virtual Fabric Adapter.The BladeCenter Virtual Fabric solutions implement virtual network interface controllers (vNICs) and are based on the IBM BladeCenter H chassis with a 10-Gb Converged Enhanced Ethernet (CEE) switch infrastructure and 10-Gb Virtual Fabric Adapters (VFAs) installed in each blade server. 90Y3550 Emulex 10GbE Virtual Fabric Adapter II Advanced. NIC 2 IBM BladeCenter Virtual Fabric Solutions .

Switch Independent Mode extends the IEEE 802. Up to 8 channels can be configured for each VFA. Note: Support for Switch Independent Mode by the Emulex 10GbE Virtual Fabric Adapter II will require a firmware upgrade. as the name suggests. Introduction 3 . 90Y3550 Emulex 10GbE Virtual Fabric Adapter II Advanced. Support for Switch Independent Mode will require a firmware upgrade. Switch Independent Mode Switch Independent Mode is. 81Y3133 BNT Virtual Fabric 10Gb Switch Module Not supported Not supported Supported 4Q/2011a Supported 4Q/2011a Supported Cisco Nexus 4001I Switch Module Not supported Not supported Supported 4Q/2011a Supported 4Q/2011a Supported a.1q VLAN domains in the network infrastructure to specific virtual channels at each server. 49Y4275 Emulex 10GbE Virtual Fabric Adapter II. The actual mode used is specified in the Unified Extensible Firmware Interface (UEFI) menu of the adapter. from the switch to the VFA) are not required. which is expected in 4Q/2011. The channel bandwidth can be configured using the adapter utility (future versions of BladeCenter Open Fabric Manager are also planned to offer this capability).Teaming at the VFA allows an individual vNIC channel to failover without impacting other vNIC channels on the same physical link. Inbound channel bandwidth constraints (for example. with outbound bandwidths per channel from 100 Mb to 10 Gb in 100 Mb increments. 49Y4235 Emulex 10GbE Virtual Fabric Adapter Advanced. and can be enabled on the combinations listed in Table 1-2. Table 1-2 Supported combinations using Switch Independent Mode Supported switches   Supported adapters Emulex 10GbE Virtual Fabric Adapter. Note that combining the Emulex 10GbE Virtual Fabric Adapter II and the BNT Virtual Fabric 10Gb Switch Module supports both modes. The Switch Independent Mode support in the Emulex 10GbE Virtual Chapter 1. more independent of the switch module used. 90Y3566 Broadcom 2-port 10Gb Virtual Fabric Adapter. which is expected in 4Q/2011.

With the original introduction of Virtual Fabric in 2009. Each device can have one ID. as well as internal software for encoding and decoding network packets. The combination of the integrated Gigabit Ethernet ports. and each 10-Gb interface can appear to the operating system as four separate physical interfaces. with the addition of a CIOv expansion card in the blade servers and I/O modules in bays 3 and 4 of the BladeCenter H chassis. vNIC enables virtualization of the physical interface. The NIC provides the physical point of connection.or quad-port Ethernet or Fibre Channel Adapters have a single application-specific integrated circuit (ASIC) chip as well. the VFA has virtualized Layer 1 of the Open Systems Interconnection (OSI) model. and that device can have eight functions. Plus. This is not a new standard: many dual. and BNT established the new vNIC feature. its resources can be divided into multiple logical 4 IBM BladeCenter Virtual Fabric Solutions . connecting to 1-Gb Ethernet switches installed in bays 1 and 2 of the chassis.1 Virtual NICs: overview A network interface controller (NIC) is a component in a blade server that allows the server to be connected to a network. With either mode. but the physical interfaces are iterated as a function. By virtualizing a 10-Gbps NIC. additional network ports or storage ports can be made available to applications to provide additional expansion capabilities. the CIOv expansion card. Emulex. and the eight vNICs of the Virtual Fabric Adapter means that up to 12 I/O ports are available to each blade in the chassis. the integrated Gigabit Ethernet ports on the blade servers are still available. 1. Rather than eight physical connections between the adapter and the switch. in a sense. IBM. Virtualizing the NIC helps to resolve issues caused by limited NIC slot availability. So.Fabric Adapter II will be made available in a future firmware update in the fourth quarter of 2011. Tip: The vNIC concept described here is associated with the physical port and should not be confused with the concept of virtualization within a hypervisor-based operating system that provides virtual I/O interfaces to the virtual machines. Splitting up a single interface into four is achieved by following the PCIe standard of the PCI DeviceID. there are two 10-Gb interfaces to two separate switch modules. by virtualizing the physical interface.

1. the virtual local area network (VLAN) tag. It does not provide bridging functions.instances or vNICs.2 Comparing modes Table 1-3 compares the use of IBM Virtual Fabric Mode with Switch Independent Mode. a NIC partition with a maximum bandwidth of 4 Gbps appears to the host applications as a physically distinct 4-Gbps Ethernet adapter. or both. Table 1-3 Comparison of Virtual Fabric modes IBM Virtual Fabric Mode Number of virtual channels per physical 10-Gb port Total number of virtual channels per adapter Minimum virtual channel bandwidth (can also be set to 0 to disable the channel) Server-to-switch bandwidth limit per virtual channel Switch-to-server bandwidth limit per virtual channel Supported expansion cards 4 8 Emulex 10-Gb VFA: 100 Mba Emulex 10-Gb VFA II: 100 Mb Yes Yes Emulex 10GbE Virtual Fabric Adapter Emulex 10GbE Virtual Fabric Adapter II BNT Virtual Fabric 10Gb Switch Module Switch Independent Mode 4 8 100 MB Yes No Emulex 10GbE Virtual Fabric Adapter II Broadcom 2-port 10Gb Virtual Fabric Adapter BNT Virtual Fabric 10Gb Switch Module Cisco Nexus 4001I Switch Module Supported switches Chapter 1. either within a partition or across partitions. Introduction 5 . For example. Each vNIC appears as a regular. independent NIC to the server operating system or a hypervisor. The vNIC partitions provide traffic forwarding and port isolation based on the destination MAC address. with each vNIC using a portion of the overall bandwidth of the physical NIC.

269.software initiator Fibre Channel over Ethernet (FCoE) channel option Switch stacking Maximum number of virtual groups per switch Optional Not supported Supported Emulex VFA I: Yes Emulex VFA II: Yes Yes Yesb Yes 32 (1 per virtual channel)c a.1q VLAN tags in your network infrastructure.0 or later and Emulex 10GbE Virtual Fabric Adapter firmware 2.103. Otherwise. You have implemented VLAN tagging only in the network infrastructure so that the tagging is transparent to the applications.For example. Support for a minimum bandwidth allocation of 100 Mbps requires BNT Virtual Fabric 10Gb Switch Module firmware 6. b.3 Selecting the mode You should use IBM Virtual Fabric Mode if your network environment has the following characteristics: You do not already implement IEEE 802. One vNIC port per physical port per group. FCoE supported in a future firmware release.hardware initiator iSCSI support .3.IBM Virtual Fabric Mode Number of unique customer VLANs per virtual channel Maximum allowed by applications Switch Independent Mode Varies by adapter Emulex VFA: 128 VLANs (4 channel) Broadcom VFA: Maximum allowed by applications Required Supported Not supported Broadcom VFA: No Emulex VFA II: Yes Yes Yesb Yes (BNT switch only) BNT: 1024 Cisco: 512 IEEE 802.5. c. Servers may have multiple NIC adapters with attachments to physically separate switches or port-based VLAN groups to provide Layer 2 separation. you may have implemented multiple IP subnets with Layer 3 separation among the Layer 2 broadcast domains provided by switches and routers.3x or later. See the BladeCenter Interoperability Guide for the latest. This implementation 6 IBM BladeCenter Virtual Fabric Solutions . the minimum allocation is 1 Gbps. 1.1q VLAN tagging VLAN trunking on switch uplinks Isolated NIC teaming failover per vNIC channel iSCSI support .

4 Use of BladeCenter Open Fabric Manager It is planned to also update BladeCenter Open Fabric Manager Advanced to support the deployment of Emulex and Broadcom 2-port 10Gb Virtual Fabric Adapters when configured for Switch Independent Mode of operation. You want to implement some of the advanced Layer 2 or Layer 3 BNT switch features that are not available in IBM Virtual Fabric Mode. 1. You want to implement the Broadcom 2-port 10Gb Virtual Fabric Adapter to maintain driver compatibility with the base 1-Gb NIC cards.1q VLANs per NIC interface. The VLAN configuration in the blade application drivers and the corresponding switch configuration must still be performed as described in Part 2 of this IBM Redbooks publication.requires that the switches add and remove VLAN tags. rather than the server applications or NIC drivers doing so. Introduction 7 . pushes the parameters to the VFA on each blade. This will allow you to configure the VFAs across multiple chassis from a central control point. You should use Switch Independent Mode if your network environment has the following characteristics: You have elected to use the Cisco Nexus 4001I Switch Module in your BladeCenter H or HT chassis. You require overlapping IEEE 802. This. You wish to implement a solution that takes advantage of Single Root IO Virtualization (SR-IOV). Two (or more) individual vNICs on a given VFA should not share the same VLAN in either mode. Note: This is a future capability that is not supported at the time of this publication. in turn. Advanced BOFM pushes the parameters to a BOFM instance that is within the BladeCenter Advanced Management Module (AMM) in each chassis. Chapter 1.

6 Using this book This book includes two parts: The chapters in Part 1 cover the product detail and implementation for IBM Virtual Fabric Mode. Essential information is repeated in both parts so that you need to only reference the part that applies to your hardware combination. 1.5 The future of converged fabrics and FCoE capabilities The Emulex 10GbE Virtual Fabric Adapter and Emulex 10GbE Virtual Fabric Adapter II cards are planned to support FCoE in conjunction with vNIC channels with a future firmware release.1. FCoE support will be described in a subsequent revision of this book. 8 IBM BladeCenter Virtual Fabric Solutions . This planned support will allow you to designate one of the four virtual channels on each of the two physical ports for the reliable transport of Fibre Channel storage traffic. This will also require a compatible FCoE-capable switch. The chapters in Part 2 cover the product detail and implementation for Switch Independent Mode.

IBM Virtual Fabric Mode” on page 81 © Copyright IBM Corp. Chapters in this part are: Chapter 2. 2011.IBM Virtual Fabric Mode” on page 11 Chapter 3. “Supported products .IBM Virtual Fabric Mode” on page 19 Chapter 4. “Examples of use .Part 1 Part 1 IBM Virtual Fabric Mode The chapters in Part 1 cover the product detail and implementation for the IBM Virtual Fabric Mode. All rights reserved. “Solution architecture .IBM Virtual Fabric Mode” on page 37 Chapter 5. 9 . “Configuring the solution .

10 IBM BladeCenter Virtual Fabric Solutions .

2. Solution architecture . 11 .IBM Virtual Fabric Mode The Virtual Fabric architecture is based on a virtual channel concept and is dependent upon both the Ethernet adapter and the Ethernet switch used to provide separation of the virtual channels on the shared 10-Gbps link. Topics in this chapter are: 2.1.3. “Planning considerations” on page 17 © Copyright IBM Corp. In this chapter. All rights reserved. we introduce the technology and architecture of IBM Virtual Fabric Mode. “Overview” on page 12 2. 2011. “Architecture” on page 14 2.2 Chapter 2.

Each channel can be assigned a specified bandwidth. Virtual Fabric Mode Overview VFA removes outer tag on VFA or VFS inserts outer frame egress tag for vSwitch Group VFS removes outer tag on frame egress BNT 10G VFS vSwitch Group 11 Emulex 10G VFA I or II Customer VLAN tag 10G Link VLAN 10 vSwitch Group 22 Emulex 10G VFA I or II 10G Link vSwitch Group 33 VLAN 20 VLAN 30 Note: Inner tag denotes customer VLAN (optional) Figure 2-1 IBM Virtual Fabric Mode overview The VFA applies a unique outer tag to the packets. The IBM Virtual Fabric Mode with the Emulex 10GbE Virtual Fabric Adapter (VFA) and the BNT Virtual Fabric 10Gb Switch Module (VFS) uses a double VLAN tagging scheme to maintain the separation of the virtual channels on the shared 10-Gbps link. a single 10-Gb link can be partitioned into up to four separate virtual channels.1 Overview As described in Chapter 1. Figure 2-1 shows an example of an implementation of IBM Virtual Fabric Mode and the VLAN tagging that is employed. “Introduction” on page 1. which allows the switch to properly handle the inbound frames and forward them to the appropriate subset 12 IBM BladeCenter Virtual Fabric Solutions . Traffic separation is maintained among the channels.2.

The VFA must be in vNIC mode to support the IBM Virtual Fabric Mode solution (vNIC mode is the default mode of operations for the VFA). you cannot configure vNIC and VMReady on the same switch port (blade server) but you can configure them on different ports on the same switch module. The VFA enforces bandwidth limits for each virtual channel. Bandwidth allocation and metering is controlled by the VFS. each physical port on the VFA is split into four virtual vNICs that are visible by the operating system as independent Ethernet controllers. Likewise. That is.Switch Independent Mode” on page 101. vNICs can be assigned to vNIC logical groups (or vSwitches) that behave as independent switches inside the VFS. This double tag scheme allows the VLANs to be tunneled through the VFA and the switch. Layer 2 spanning tree. and. as described in Chapter 6. not the entire switch. Note that these functions are disabled at the port level. The VFS recognizes these vNICs and manages and isolates network traffic flows between them. ACL filters. To isolate vNIC traffic flows from each other. Note: These restrictions are not applicable with Switch Independent Mode. in cases when outgoing traffic flow for a particular vNIC exceeds established bandwidth limits. the switch applies a corresponding outer tag on all packets that are received from the network infrastructure via the uplink ports and insures that these packets are only sent to VFAs that are members of the corresponding vNIC group. VMReady. The VFS enforces the same virtual channel bandwidth limits on traffic flowing to the server. For example. This vSwitch group ID is appended as an outer VLAN tag by either the VFS or the VFA. Clients should not need to make any concessions with regard to their existing network infrastructure Layer 2 networking scheme. The BNT Virtual Fabric 10Gb Switch Module disables several Layer 2 and Layer 3 switch functions when supporting the IBM Virtual Fabric Mode. Channel bandwidth is dynamic in that it may be changed (from the switch) without resetting the VFA.of ports. and Layer 3 routing are some of the functions that are not supported in conjunction with Virtual Fabric Mode operation. vNICs in the same group can communicate with each other. For example. In such a configuration.IBM Virtual Fabric Mode 13 . Chapter 2. In addition to external or internal switch ports. whereas vNICs in different groups cannot. external trunks can also be assigned to a particular vNIC group to isolate upstream connections. the switch may drop packets to maintain the traffic below configured thresholds. Parameters are passed to the adapter by the switch. “Solution architecture . Configuration is provided through the switch user interfaces. Solution architecture .

and the thickness of each link represents the proportional amount of assigned bandwidth for the particular traffic flow.The BNT switch will support stacking for both IBM Virtual Fabric Mode and Switch Independent Mode. The Virtual Fabric architecture provides the capability to subdivide a single 10-Gb link into up to four virtual channels. 2.2 Architecture Figure 2-2 on page 15 provides a general view of the vNIC solution architecture and offers an example of configuring the vNICs to having different bandwidth values. 14 IBM BladeCenter Virtual Fabric Solutions . The different colors represent different virtual port groups to isolate traffic flows for different services and applications.

1 INT14.4 I/O Bay 3 EXT1 EXT2 EXT3 EXT4 EXT5 EXT6 EXT7 EXT8 EXT9 EXT10 EXT1 EXT2 EXT3 EXT4 EXT5 EXT6 EXT7 EXT8 EXT9 EXT10 vSwitch1 vSwitch2 vSwitch3 vNIC1 vNIC2 vNIC3 vNIC4 vNIC1 vNIC2 vNIC3 vNIC4 INT14. and they maintain strict one-to-one mapping to vNICs on the VFA to a logical vSwitch group within the switch and the corresponding uplink port(s). vNICs are configured on the BNT Virtual Fabric 10Gb Switch Module by dividing internal VFS ports into subinterfaces (or logical ports).4 vSwitch2 vSwitch3 vSwitch4 Figure 2-2 BladeCenter Virtual Fabric vNIC solution diagram Trunk failover is supported on a per-vNIC basis.4 vSwitch1 Legend Standard I/O bays connections vNIC virtual connections External physical trunks vSwitch virtual trunk connections Mid-Plane INT14.IBM Virtual Fabric Mode 15 . the switch shuts down only those vNICs that belong to the same vNIC group as the failed external trunk.1 INT1.3 INT1.1 INT1.3 INT1.2 INT1.2 INT14. whereas other vNICs on the same physical NIC (pNIC) port are untouched and continue to perform their functions.1 INT14. Table 2-1 on page 16 is a vNIC mapping for VFA and VFS. so you can set up NIC teaming in a failover mode for vNICs.2 INT14.3 INT14. as well as vNIC identification data.3 INT14. Solution architecture . In case of external trunk or uplink failure. Each internal interface of the VFS can be divided into up to four vNICs.4 vSwitch4 CFFh BNT Virtual Fabric Switch (Bay 9) INT1. Chapter 2.I/O Bay 1 I/O Bay 2 I/O Bay 4 Components and connectivity inside the IBM BladeCenter H chassis BNT Virtual Fabric Switch (Bay 7) Blade Server 14 Blade Server 1 FC Expansion Card Virtual Fabric Adapter On-Board 1GbE CIOv INT1. to differentiate vNICs on the host operating system (vNIC PCIe Function ID).2 INT1.

4 a. Operating system (OS) configurations see eight unique NICs (2 ports x 4 vNICs).1 INTx.2 INTx.14).Table 2-1 vNIC mappings and identification vNIC PCIe function ID 0 2 4 6 1 3 5 7 VFA port number 0 0 0 0 1 1 1 1 VFS bay number 7 7 7 7 9 9 9 9 vNIC number 1 2 3 4 1 2 3 4 vNIC IDa INTx. The user assigns vNICs and. The Data Center Bridging Exchange (DCBX) protocol is used between the switch and NIC to convey configuration information.5 Gbps The sum of all four vNICs cannot exceed 10 Gbps – VFS-to-server bandwidth metering on a per vNIC per port basis. All vNIC parameters are configured from the VFS user interface.2 INTx. A vNIC ID of x represents the VFS internal port number to which the VFA physical port is connected (x is 1 . – Server-to-VFS bandwidth metering on a per-vNIC basis. uplinks to vNIC groups: – Groups serve to isolate virtual NIC traffic flowing on the same physical port: • Existing VLANs within the client network are not impacted. – The allowable vNIC bandwidth range is either 100 Mbps or 1 Gbps 10 Gbps (VFA model dependent): • • • The increments are 100 Mbps The default bandwidth setting is 2.1 INTx.4 INTx. General characteristics of this functionality are: Each server physical NIC (pNIC) port is divided into up to four virtual NICs (vNICs).3 INTx. optionally. 16 IBM BladeCenter Virtual Fabric Solutions . The user enables vNICs and allocates the bandwidth.3 INTx.

Check the latest release notes of the BNT Virtual Fabric 10Gb Switch Module firmware for the latest information. The BNT VFS supports stacking and Virtual Fabric Mode at the same time with the most recent firmware. trunk links. and the stack can be accessed and managed as a whole using standard switch IP interfaces. rather than just the physical port level. At the time of writing. 2. Once the stacking links have been established. The network views the stack as a single switch entity. vNIC and switch stacking A stack is a group of BNT Virtual Fabric 10Gb Switch Modules that work together as a unified system and is thus managed as a single entity. The totals for the stack as a whole are the same as for any single switch configured in stand-alone mode. trunks. VLANs. Chapter 2. the number of ports available in a stack is equal to the total number of remaining ports of all the switches that are part of the stack. – The failover mechanism is virtual port aware. An uplink (port or trunk) can belong to only one vNIC group. and other switch attributes are not aggregated among the switches in a stack.IBM Virtual Fabric Mode 17 . The network can be configured and managed at the virtual port (vport) level. However. Up to 32 vNIC groups are supported per Virtual Fabric Switch (VFS). certain ancillary functions of VMready that come with vCenter integration work with the VFA in either pNIC or vNIC mode. you could not enable IBM Virtual Fabric Mode on either of the Emulex Virtual Fabric Adapters (VFAs) and implement BNT VMready on that server at the same time.The numbers of available IP interfaces.• • • • No forwarding occurs between uplinks assigned to vNIC groups. Running the VFA in pNIC mode and running VMready are supported concurrently. A server port (pNIC or vNIC) can belong to only one vNIC group. Solution architecture . Key concurrency rules are: vNIC and VMready® VMready is a unique function that enables the network to be virtual machine aware.3 Planning considerations You should take several considerations into account when planning a Virtual Fabric solution using IBM Virtual Fabric Mode: The adapter must operate in vNIC mode.

– Emulex 10GbE Virtual Fabric Adapter II: 100Mb minimum The sum of the bandwidth allocated for all four vNICs on the same pNIC cannot exceed 10 Gb.5. Otherwise. Up to 32 vNIC groups are supported on a VFS. vNICs on the same VFA must be assigned to separate vNIC groups at the VFS.5 Gbps. at most.103. BladeCenter Open Fabric Manager is not supported for IBM Virtual Fabric Mode. Tip: All vNIC parameters are configured on the BNT Virtual Fabric 10Gb Switch Module.269. Support for a minimum bandwidth allocation of 100 Mbps requies BNT Virtual Fabric 10-Gb Switch Module firmware 6.0 or later and Emulex 10GbE Virtual Fabric Adapter firmware 2. Only one external port or one trunk made up of external ports is allowed per vNIC group. Currently. the minimum allocation is 1 Gbps. because vNIC groups used by vNICs are fully isolated from other switch entities. one vNIC group at a time (there is no requirement that trunks and ports belong to a vNIC group).3. The minimum bandwidth that can be assigned to a vNIC depends upon the adapter used: – Emulex 10GbE Virtual Fabric Adapter: 100-Mb minimum.Up to four vNICs can be configured on each pNIC on the VFA (a total of eight vNICs on a 2-port VFA). This can only be changed by configuration of the VFS. Virtual machines (VMs) and other virtual entities (VEs) associated with vNICs are automatically detected by VFS when VMready is enabled. 18 IBM BladeCenter Virtual Fabric Solutions . However. these VEs cannot be assigned to any of the VMready VM groups.3x or later. Internal ports not connected to a VFA in vNIC mode can belong to at most one vNIC group at a time. External trunks and ports can belong to. The maximum bandwidth that can be assigned to a vNIC interface is 10 Gbps. The default bandwidth to be assigned to a vNIC is 2.

This offering consists of the following supported adapters and switch modules: Emulex 10GbE Virtual Fabric Adapter and Virtual Fabric Adapter Advanced for IBM BladeCenter These are the first generation Emulex VFA cards for BladeCenter. “Overview” on page 20 3. and they are supported in IBM Virtual Fabric Mode offering only. Supported products . Emulex 10GbE Virtual Fabric Adapter II and Virtual Fabric Adapter Advanced II for IBM BladeCenter These expansion cards are the second generation Virtual Fabric Adapters from Emulex. All rights reserved. we describe the products used in IBM Virtual Fabric Mode. “BNT Virtual Fabric 10Gb Switch Module” on page 27 © Copyright IBM Corp. 19 . 2011. “Emulex 10Gb Virtual Fabric Adapters” on page 21 3.3 Chapter 3.3. BNT Virtual Fabric 10Gb Switch Module The topics in this chapter are: 3.IBM Virtual Fabric Mode In this chapter.2. and can be used both in IBM Virtual Fabric Mode and in Switch Independent Mode.1.

Table 3-1 BladeCenter IBM Virtual Fabric Mode solution support matrix Description Chassis IBM BladeCenter H IBM BladeCenter HT Blade servers HS22 HS22V HX5 Virtual Fabric adapters Emulex 10GbE Virtual Fabric Adapter Emulex 10GbE Virtual Fabric Adapter II Virtual Fabric switches BNT Virtual Fabric 10Gb Switch Module External Virtual Fabric Switch connections Copper-based SFP+ Copper Direct Attach Cable .3. 20 IBM BladeCenter Virtual Fabric Solutions .1 M SFP+ Copper Direct Attach Cable .3 M SFP+ Copper Direct Attach Cable .7 M Fiber Optics-based IBM 10GBase-SR 10GbE 850 nm Fiber SFP+ Transceiver BLADE 1000Base-SX SFP IBM BNT 1000BASE-T SFP Transceiver 44W4408 81Y1622 81Y1618 1 per external port 1 per external port 1 per external port 59Y1936 59Y1940 59Y1944 1 per external port 1 per external port 1 per external port 46C7191 1 or 2 per chassis 49Y4235 90Y3550 1 per CFFh slot 1 per CFFh slot 7870 7871 7872 Varies Varies Varies 8852 8740/8750 Varies Varies Machine type/ part number Quantity This chapter describes the features of these products.1 Overview Table 3-1 lists the components supported by the IBM Virtual Fabric Mode solution.

2. 3. The adapter is a CFFh expansion card. Supported products . The Emulex 10GbE Virtual Fabric Adapter Advanced supports Fibre Channel over Ethernet (FCoE) and iSCSI hardware initiator functionality. The Emulex 10GbE Virtual Fabric Adapter II (part number 90Y3550) and Emulex 10GbE Virtual Fabric Adapter II Advanced (part number 90Y3566) In addition to IBM Virtual Fabric Mode.1 Emulex 10GbE Virtual Fabric Adapter The Emulex 10GbE Virtual Fabric Adapter for IBM BladeCenter is a dual-port 10-Gb Ethernet card that supports 1-Gbps or 10-Gbps traffic. They do not support Switch Independent Mode. it requires that switch modules be installed in I/O bays 7 and 9.IBM Virtual Fabric Mode 21 . in addition to the features available on the standard card. Table 3-2 Ordering part numbers and feature codes Description Emulex 10GbE Virtual Fabric Adapter Emulex 10GbE Virtual Fabric Adapter Advanced Part number 49Y4235 49Y4275 Feature code 5755 2435 Chapter 3. these adapters also support Switch Independent Mode. They also support Serial over LAN via 10-GB switches and support the technology preview of SR-IOV on Red Hat Enterprise Linux 6. Table 3-2 lists the Emulex 10GbE Virtual Fabric Adapter ordering part numbers and feature codes. The virtual NICs are configured to meet your mix of network connectivity and throughput demands for today's complex server application environments. therefore. or up to eight virtual NIC devices.2 Emulex 10Gb Virtual Fabric Adapters You can use two generations of Emulex Virtual Fabric Adapters in IBM Virtual Fabric Mode: First generation Emulex 10GbE Virtual Fabric Adapter (part number 49Y4235) and Emulex 10GbE Virtual Fabric Adapter Advanced (part number 49Y4275) These adapters support IBM Virtual Fabric Mode only.3. The following sections describe these expansion cards in details. Each physical 10-Gbps port can be divided into four virtual ports with bandwidth allocation in 100-Mbps increments to a maximum of 10 Gbps per physical port.

(CEE) DCBX (P802.1Qaz) 22 IBM BladeCenter Virtual Fabric Solutions .Description Emulex 10GbE Virtual Fabric Adapter Advanced Upgrade Part number 49Y4265 Feature code 2436 The Emulex 10GbE Virtual Fabric Adapter has the following features: Connection to either 1-Gb or 10-Gb data center infrastructure (1-Gb and 10-Gb auto-negotiation) PCI Express 2. and IEEE 802. adaptive load balancing (ALB).0 x8 host interface IBM BladeCenter CFFh form factor.3ad Enhanced Ethernet (draft) – Enhanced Transmission Selection (ETS) (P802. can be combined with a CIOv expansion card on the same blade server Operates either as an 8-port virtual NIC or as a 2-port 1/10Gb Ethernet adapter: – Virtual port bandwidth allocation in 100-Mbps increments – Supports up to eight virtual ports Wake On LAN support Full-duplex (FDX) capability Bus-mastering support Direct memory access (DMA) support Preboot Execution Environment (PXE) support IPv4/IPv6 TCP. teaming support.1Qbb) – Data Center Bridging Capability Exchange Protocol (DCBX). CIN-DCBX and Converged Enhanced Ethernet. UDP checksum offload – – – – Large send offload (LSO) Large receive offload Receive side scaling (RSS) IPv4 TCP Chimney Offload VLAN insertion and extraction Jumbo frames up to 9000 bytes Load balancing and failover support including adapter fault tolerance (AFT).1Qaz) – Priority-based Flow Control (PFC) (P802. switch fault tolerance (SFT).

2-port Ethernet expansion card. When the ports operate in pNIC mode. When the ports are configured in vNIC mode. Total bandwidth of all vNICs combined cannot exceed 10 Gbps per physical port. In vNIC mode. the expansion card can operate as a standard 10-Gbps or 1-Gbps. When in pNIC mode. The expansion card has two modes of operation: Virtual Fabric vNIC (or simply. Supported products . The adapter is configurable from the BNT Virtual Fabric 10Gb Switch Module. the Emulex Virtual Fabric Adapter Advanced supports iSCSI hardware initiator functionality on both physical ports.5 Gbps. each physical port appears to the blade server as four virtual NICs. Physical NIC (pNIC) mode In pNIC mode. The standard Emulex 10GbE Virtual Fabric Adapter can be upgraded to the same features as the Emulex 10GbE Virtual Fabric Adapter Advanced with the addition of the Advanced Upgrade option. The default bandwidth for each vNIC is 2. part number 49Y4265. the expansion card functions with any supported 10-Gb switch installed in I/O module bays 7 and 9. which provides independent control for each vNIC. vNICs can also be configured to have zero bandwidth if you must allocate the available bandwidth to fewer than eight vNICs. This functionality is the basis for IBM Virtual Fabric Mode offering. Chapter 3. in 100-Mbps increments. you can use up to two iSCSI hardware initiators. the expansion card communicates with the BNT Virtual Fabric 10Gb Switch Module. you can use two iSCSI initiators and six vNICs. Bandwidth for each vNIC can be configured from 1 Gbps to 10 Gbps.The Emulex Virtual Fabric Adapter Advanced provides FCoE and iSCSI support.IBM Virtual Fabric Mode 23 . In addition. vNIC) mode This mode is the basis for Virtual Fabric functionality. In IBM Virtual Fabric Mode.

The Emulex 10GbE Virtual Fabric Adapter is shown in Figure 3-1. Figure 3-1 Emulex 10GbE Virtual Fabric Adapter Figure 3-2 shows the placement of the adapter in an HS22 blade server. The adapter is installed in the CFFh expansion slot. 24 IBM BladeCenter Virtual Fabric Solutions .

Emulex Virtual Fabric Adapter Figure 3-2 Emulex Virtual Fabric Adapter placement in HS22 blade server For more information.redbooks.2.IBM Virtual Fabric Mode 25 . The Emulex 10GbE Virtual Fabric Adapter II is shown in Figure 3-3 on page 26. see the Emulex 10GbE Virtual Fabric Adapter At-a-Glance Guide at: http://www.ibm. While its basic features are similar to the original Emulex 10GbE Virtual Fabric Adapter. the Emulex 10GbE Virtual Fabric Adapter II provides more flexible vNIC support: in addition to IBM Virtual Fabric Mode.com/abstracts/tips0748.2 Emulex 10GbE Virtual Fabric Adapter II This adapter enhances the existing BladeCenter Virtual Fabric portfolio.html 3. Chapter 3. The Emulex Virtual Fabric Adapter Advanced II offers FCoE and iSCSI hardware initiator functionality. Supported products . in addition to the features available on the standard card. it also supports Switch Independent Mode.

which is the same as vNIC mode on the original Emulex VFA adapters. In this mode. The original Emulex 10GbE Virtual Fabric Adapter could be configured in pNIC or vNIC mode. Switch Independent Mode. The new Emulex 10GbE Virtual Fabric Adapter II now offers two flavors of vNIC: IBM Virtual Fabric Mode. where the adapter can work with BNT Virtual Fabric 10Gb Switch Module or the Cisco Nexus 4001I Switch Module. 26 IBM BladeCenter Virtual Fabric Solutions . the Emulex VFA card obtains the vNIC parameters from the switch via DCBX.Figure 3-3 Emulex 10GbE Virtual Fabric Adapter II Table 3-3 lists the part numbers for ordering these expansion cards. Table 3-3 Part numbers and feature codes for ordering Emulex adapters Description Emulex Virtual Fabric Adapter II (CFFh) Emulex Virtual Fabric Adapter Advanced II (CFFh) Emulex Virtual Fabric Adapter Advanced Upgrade Part number 90Y3550 90Y3566 49Y4265 Feature code A1XG A1XH 2436 Tip: Emulex Virtual Fabric Adapter Advanced Upgrade is the same part number (49Y4265) for both the Emulex 10GbE Virtual Fabric Adapter and the Emulex 10GbE Virtual Fabric Adapter II. This mode only works in conjunction with the BNT Virtual Fabric 10Gb Switch Module.

Supported products . Available on Emulex Virtual Fabric Adapter Advanced II.IBM Virtual Fabric Mode 27 . top-of-rack switch). and it represents the perfect migration platform for clients who are still at 1 Gb outside the chassis by seamlessly integrating in the existing 1-Gb infrastructure. all vNIC parameter configuration is performed on the BNT Virtual Fabric 10Gb Switch Module. Emulex Virtual Fabric Adapter II is 20% more energy efficient 3. This feature is available with firmware release 6. and other data over the common Ethernet infrastructure. With the use of the QLogic Virtual Fabric Extension Module. Therefore. The BNT Virtual Fabric 10Gb Switch Module can be used both in IBM Virtual Fabric Mode and Switch Independent Mode. In this publication. This is the first 10-Gb switch for IBM BladeCenter that supports converged networking (that is. video. we concentrate on virtual fabric. Table 3-4 Emulex Virtual Fabric Adapters: Feature comparison Function pNIC mode IBM Virtual Fabric Mode Switch Independent Mode iSCSI hardware initiator FCoE Serial over LAN (SoL) High energy efficiency Emulex Virtual Fabric Adapter Yes Yes No Yes (Advancedb) Yes (Advancedb ) No No Emulex Virtual Fabric Adapter II Yes Yes Yesa Yes (Advancedb ) Yes (Advancedb ) Yes Yesc a.1 and higher. the ability to transmit Converged Enhanced Ethernet (CEE) to a Fibre Channel over Ethernet (FCoE)-capable. clients can achieve FCoE gateway functionality inside the BladeCenter chassis.3 BNT Virtual Fabric 10Gb Switch Module The BNT Virtual Fabric 10Gb Switch Module for IBM BladeCenter offers the most bandwidth of any blade switch. Voice over IP (VoIP). This offering allows clients to form Chapter 3. Planned for 4Q/2011 b. or with Advanced Upgrade c. In IBM Virtual Fabric Mode.Table 3-4 compares the Emulex 10GbE Virtual Fabric Adapter with the Emulex 10GbE Virtual Fabric Adapter II. CEE and FCoE functionality enables you to transfer storage. network. the switch module is a key part of the offering.

or both. DAC cables are more cost-effective. several operating at 1 Gbps. connecting to a top-of-rack switch. this single switch can handle all of these workloads and can connect to a 1-Gb infrastructure to a 10-Gb infrastructure. consume less power.eight vNICs from one physical NIC and to manage them in virtual groups. they can take advantage of SFP transceivers. they have a choice of either SFP+ transceivers (short range or long range for longer distances. or direct-attached copper (DAC) cables (also known as twinax active cables) for shorter distances. If you have a chassis with multiple servers. several at 10 Gbps. They are ideal for connecting chassis. Figure 3-4 BNT Virtual Fabric 10Gb Switch Module Table 3-5 lists the part number and feature code to use to order the module. and can be up to 7 meters in length. providing all of the benefits of I/O virtualization at 10-Gb speed. For 1-Gbps uplinks. Figure 3-4 shows the switch module. you can take advantage of the technologies that they require for multiple environments. or even connecting to an adjacent rack. The switch module can be managed using a command-line interface or web browser interface. With the extreme flexibility of the BNT switch. Table 3-5 BNT Virtual Fabric 10Gb Switch Module part number and feature code Description BNT 10Gb Virtual Fabric Switch Module for IBM BladeCenter Part number 46C7191 Feature code 1639 28 IBM BladeCenter Virtual Fabric Solutions . and several transmitting converged packets. For 10-Gbps uplinks.

Tip: There are both 10-Gb and 1-Gb SFP+ modules available for uplink. you must have either SFP+ transceivers or SFP+ direct-attach copper (DAC) cables connected. Table 3-6 on page 30 lists the part numbers for ordering the SFP+ transceivers. mini-USB-to-DB9 serial console cable One filler module BNT Virtual Fabric 10Gb Switch Module Installation Guide BNT user license agreement Important Notices document Documentation CD-ROM Tip: Small form-factor pluggable plus (SFP+) transceivers are not included and must be purchased separately. To communicate outside of the chassis. DAC cables have SFP+ transceivers on both ends. You have the flexibility to expand your bandwidth as desired. and DAC cables.The part number includes the following items: One BNT Virtual Fabric 10Gb Switch Module Three-meter. FC cables. using anywhere from one connection up to ten connections per switch. Chapter 3. Supported products .IBM Virtual Fabric Mode 29 .

FC cables. Table 3-7 BNT part numbers for ordering SFP-based transceivers and cables Description BLADE 1000Base-T SFP BLADE 1000Base-SX SFP BLADE 10GBase-LR SFP+ SFP+ Copper Direct Attach Cable .Table 3-6 IBM part numbers for ordering SFP+ transceivers.1 M SFP+ Copper Direct Attach Cable .7 M Part number BN-CKM-S-T BN-CKM-S-SX BN-CKM-SP-LR BN-SP-CBL-1M BN-SP-CBL-3M BN-SP-CBL-7M 30 IBM BladeCenter Virtual Fabric Solutions .5 m Molex Direct Attach Copper SFP+ Cable 1 m Molex Direct Attach Copper SFP+ Cable 3 m Molex Direct Attach Copper SFP+ Cable 7 m Molex Direct Attach Copper SFP+ Cable FC cables 3 m Intel Connects Optical Cable 10 m Intel Connects Optical Cable 30 m Intel Connects Optical Cable 46D0153 46D0156 46D0159 3852 3853 3854 59Y1932 59Y1936 59Y1940 59Y1944 3735 3736 3737 3738 81Y1618 81Y1622 3268 3269 44W4408 46C3447 4942 5053 Part number Feature code Table 3-7 lists additional transceivers and DAC cable options that are available directly from BNT.3 M SFP+ Copper Direct Attach Cable . and DAC cables Description 10Gb SFP+ IBM 10GBase-SR 10GbE 850 nm Fiber SFP+ Transceiver BNT SFP+ Transceiver 1Gb SFP+ BLADE 1000BASE-T (RJ45) SFP Transceiver BLADE 1000BASE-SX SFP Transceiver DAC cables 0.

full-duplex 100-Mbps ports connected to the management module External ports – Up to ten 10-Gb SFP+ ports (also designed to support 1-Gb SFP if required. and V3) – Configurable traffic distribution schemes over trunk links.IBM Virtual Fabric Mode 31 . supports up to 32-Kb MAC addresses – Up to 128 IP interfaces per switch – Static.The BNT Virtual Fabric 10Gb Switch Module includes the following features and functions: Form-factor – Single-height. up to 18 trunk groups. and up to eight ports per group – Support for jumbo frames (up to 12288 bytes) – Broadcast and multicast storm control – IGMP snooping for limit flooding of IP multicast traffic (IGMP V1. and this makes the switch module suitable for the most performance-intensive environments. and Link Aggregation Control Protocol (LACP) (IEEE 802. V2. high-speed switch module Internal ports – Fourteen internal auto-negotiating ports: 1 Gbps or 10 Gbps to the server blades – Two internal. MAC addresses. based on source and destination IP addresses. EtherChannel. Supported products . or both Chapter 3. flexibility of mixing 1 Gb/10 Gb) Oversubscription ratio (14 internal ports to 10 external ports) is very low.3ad) link aggregation. up to 100 Gb of total bandwidth per switch. – One 10/100/1000-Mb copper RJ45 used for management or data – An RS-232 mini-USB connector for serial port that provides an additional means to install software and configure the switch module Scalability and performance – Autosensing 1-Gb/10-Gb internal and external Ethernet ports for bandwidth optimization – Non-blocking architecture with wire-speed forwarding of traffic and full line rate performance of 480-Gbps full duplex – Media access control (MAC) address learning: automatic update.

Terminal Access Controller Access-Control System Plus (TACACS+). IP ToS/DSCP. and IP-based access control lists (ACLs) – 802.1D STP for providing Layer 2 redundancy with PVRST+ – IEEE 802.1X port-based authentication – Multiple user IDs and passwords – User access control – Radius.1s Multiple STP (MSTP) for topology optimization.1Q VLAN tagging support on all ports – Protocol-based VLANs Security – VLAN-based. VLAN numbers ranging from 1 to 4095 (4095 is a dedicated VLAN used for the management module connection only) – 802. providing rapid STP convergence for critical delay-sensitive. Lightweight Directory Access Protocol (LDAP) Quality of Service (QoS) – Up to eight queues per port – Support for IEEE 802.1p. up to 128 STP instances supported by single switch – IEEE 802. VLANs) traffic classification and processing – Traffic shaping and remarking based on defined policies – Eight Weighted Round Robin (WRR) priority queues per port for processing qualified traffic 32 IBM BladeCenter Virtual Fabric Solutions .1w Rapid STP (RSTP). and ACL-based (MAC/IP source and destination addresses. MAC-based. traffic-like voice or video – Layer 2 trunk failover to support active and standby configurations of network adapter teaming on blades – Interchassis redundancy (Layer 2 and Layer 3) VLAN support – Up to 1024 VLANs supported per switch.clients can stack up to eight BNT Virtual Fabric 10Gb Switch Module Availability and redundancy – Virtual Router Redundancy Protocol (VRRP) for Layer 3 router redundancy – IEEE 802.– Fast port forwarding and fast uplink convergence for rapid Spanning Tree Protocol (STP) convergence – Stacking support .

Layer 3 functions – IP forwarding – IP filtering with ACLs (up to 4096 ACLs supported) – VRRP for router redundancy – Support for up to 128 static routes – Routing protocol support (Router Information Protocol (RIP) V1. BGP-4). Open Shortest Path First protocol (OSPF)V1. RIP V2. Supported products . up to 1024 entries in routing table – IPv6 routing including static routes and OSPFv3 (requires firmware V6. V2.3 or higher) – Support for Dynamic Host Configuration Protocol (DHCP) Relay – IPv6 host management – IPv6 forwarding based on static routes Manageability – Simple Network Management Protocol (SNMP V1.IBM Virtual Fabric Mode 33 . V2. and V3. and V3) – HTTP and HTTPS Browser-Based Interface (BBI) – Industry standard CLI and BLADEOS/AlteonOS CLI – Telnet interface for CLI – SSH v1/v2 – Serial interface for CLI – Scriptable CLI – Firmware image update (Trivial File Transfer Protocol (TFTP) and File Transfer Protocol (FTP)) – Network Time Protocol (NTP) for switch clock synchronization – BNT BLADEHarmony Manager support Monitoring – Switch LEDs for external port status and switch module status indication – Port mirroring for analyzing network traffic passing through switch – Change tracking and remote logging with syslog feature – Power-On Self Test (POST) diagnostics Special functions – Serial over LAN (SOL) Chapter 3.

3x standard flow control to allow the switch to pause traffic.1AB) allows neighboring network devices to exchange information about their capabilities. – Supports the QLogic Virtual Fabric Extension Module for IBM BladeCenter.1p Tagged Packets 34 IBM BladeCenter Virtual Fabric Solutions .1p priority value in each packet VLAN tag. – Priority-Based Flow Control (PFC) (IEEE 802. and KVM. – Enhanced Transmission Selection (ETS) (IEEE 802. Xen.1w RSTP IEEE 802. while retaining the same ACLs. VMready is a unique solution that enables the network to be virtual machine-aware.1p priority value in each packet VLAN tag. It is available as part of the 6.1D STP with PVRST+ IEEE 802.1 (and higher) software code. as VMs migrate across physical hosts. rather than just for physical ports. VMready works with all virtualization products. which provides FCoE gateway functionality inside the BladeCenter chassis. VMready allows for a define-once-use-many configuration that evolves as the server and network topologies evolve. and removed. Virtual machines can be added. VMready compatibility with Virtual Fabric solutions is as follows: VMready is not supported with IBM Virtual Fabric Mode.1Qaz) provides a method for allocating link bandwidth. The switch module supports the following IEEE standards: IEEE 802. The network can be configured and managed for virtual ports (v-ports). – DCBX (IEEE 802. VMready is supported with Switch Independent Mode. so do their network attributes. and VLAN attributes. moved. without modification of virtual machine hypervisors or guest operating systems. Hyper-V. including VMware. – FCoE Initialization Protocol (FIP) snooping to enforce point-to-point links for FCoE traffic outside the regular Fibre Channel topology.1Qbb) extends the 802. QoS. based on the 802.Virtualization features – VMready – Virtual Fabric Adapter vNIC support Converged Enhanced Ethernet and FCoE features – FCoE allows Fibre Channel traffic to be transported over Ethernet links.1s MSTP IEEE 802. based on the 802. With VMready.

3ab 1000BASE-T Gigabit Ethernet IEEE 802.ibm.5µ (300 meters maximum).3x Full-duplex Flow Control IEEE 802.3ae 10GBASE-LR 10-Gb Ethernet fiber optics long range IEEE 802. 50µ or 62.1x port-based authentication IEEE 802.html Documents in PDF format available at: http://www.IBM Virtual Fabric Mode 35 .IEEE 802.redbooks.3ae 10GBASE-SR 10-Gb Ethernet fiber optics short range IEEE 802.1Q Tagged VLAN (frame tagging on all ports when VLANs are enabled) IEEE 802. available at: http://www. Supported products . with LC duplex connector 1000BASE-T for RJ45 port: – – – – UTP Category 6 (100 meters maximum) UTP Category 5e (100 meters maximum) UTP Category 5 (100 meters maximum) EIA/TIA-568B 100-ohm STP (100 meters maximum) More information is available in the following publications: BNT Virtual Fabric 10Gb Switch Module At-a-Glance Guide.ibm.com/abstracts/tips0708.3z 1000BASE-SX Gigabit Ethernet The following network cables are supported for the BNT Virtual Fabric 10Gb Switch Module: 10GBASE-SR for 10-Gb ports: 850Nm wavelength. multimode fiber.3ad Link Aggregation Control Protocol IEEE 802.2 Logical Link Control IEEE 802.wss?uid=psg1MIGR-5080917 are: – – – – – – – BNT Virtual Fabric 10Gb Switch Module Installation Guide Application Guide Command Reference isCLI Reference BBI (Browser-based Interface) Quick Guide Release Notes End User License Agreement Chapter 3.com/support/docview.

36 IBM BladeCenter Virtual Fabric Solutions .

“Creating virtual NICs” on page 56 © Copyright IBM Corp. “Configuring the BNT Virtual Fabric 10Gb Switch Module” on page 44 4.4 Chapter 4. vNIC groups and VLANs) on the BNT Virtual Fabric 10Gb Switch Module. Configuring the solution IBM Virtual Fabric Mode In this chapter. When configuring the solution in this mode. we configure vNIC parameters (such as bandwidth allocation. The switch transfers the configuration settings to the Emulex 10GbE Virtual Fabric Adapter using the Data Center Bridging eXchange (DCBX) protocol. The rest of the configuration is performed on the switch.1.3. 37 . All rights reserved. “Configuring the Emulex 10GbE Virtual Fabric Adapter” 4. The adapter requires very little configuring: you only need to enable vNIC mode in the blade server Unified Extensible Firmware Interface (UEFI) utility. we describe how to configure the components for operating in IBM Virtual Fabric Mode.2. The BNT Virtual Fabric 10Gb Switch Module can be configured using any of the following tools: BladeOS command line interface (CLI) Industry-standard CLI (isCLI) Browser-based interface (BBI) This chapter has the following topics: 4. 2011.

38 IBM BladeCenter Virtual Fabric Solutions .6. we use the Windows online package to upgrade the firmware. The procedures are similar on the older Emulex 10GbE Virtual Fabric Adapter cards.4. “vNIC and VMready” on page 69 4. Perform the upgrade In the example that follows. The package uses an executable file.8.1 Configuring the Emulex 10GbE Virtual Fabric Adapter This section describes how to upgrade firmware and configure the Emulex 10GbE Virtual Fabric Adapter cards. Note: Our discussion and screen shots show the process using the new Emulex 10GbE Virtual Fabric Adapter II.7. elxflash. Note: When using elxflash. You can upgrade firmware in a number of ways: Windows Server 2008.1. “vNIC group configuration” on page 63 4.exe. “Configuring vNIC VLANs” on page 66 4. with minor differences.4. 2008 R2 and 2003 online update Windows PE offline update Linux online code update Linux offline code update VMware ESX code update package Code update using a bootable DVD media UEFI code update The upgrade procedure is documented in the readme files for these products. 4. some parameters might need to be set prior to the update.5.1 Upgrading firmware We recommended that you upgrade the firmware on the Emulex 10GbE Virtual Fabric Adapter II to the latest level before beginning these setup procedures. “High availability and vNICs” on page 70 4.exe for the firmware update. to perform the update. These parameters are explained in the readme file for the executable. “Configuring the operating system” on page 72 4.

1.0.10.103.10. New=4. Old=3.0. Update=Firmware.0. press F1 to display the System Configuration and Boot Management panel (also known as the UEFI utility main menu) (Figure 4-2 on page 40).10. Status=Success elxflash: All required updates succeeded .0.10. The command produces eight sections.200.10. 4. Image=oc11-4.2 Configuring the adapter in the UEFI interface Note: The instructions and panel shots in this section are based on pre-GA level firmware of the Emulex 10GbE Virtual Fabric Adapter II. Image=oc11-4. New=4. with three shown in Figure 4-1. When prompted during POST. Image=oc11-4.200. New=4.IBM Virtual Fabric Mode 39 . MAC=00-00-C9-B2-57-09.Return Code=0 Figure 4-1 Windows online code update.0. Update=Firmware. Status=Success Thu May 19 15:41:27 2011 HBA=OCe11100-NIC. Chapter 4.10.200.351. The steps may be different for you using the latest firmware.1. Status=Success (lines deleted for clarity) Thu May 19 15:41:28 2011 HBA=OCe11100-NIC.0. Update=Firmware. Old=3.200. Port Type=NIC.200.10. Port Type=NIC.ufi. Configuring the solution . Port Type=NIC. MAC=00-00-C9-B2-57-02. using the elxflash executable The output is truncated here for clarity.351. Old=3.103. Power on or restart the blade 2. MAC=00-00-C9-B2-57-06. Enter the elxflash /update command 2.351.ufi.200. Each section represents one vNIC.103.10. To configure the Emulex 10GbE Virtual Fabric Adapter II using the blade server UEFI interface: 1. Verify the results (Figure 4-1) >elxflash /update Thu May 19 15:41:27 2011 HBA=OCe11100-NIC.10.ufi.

40 IBM BladeCenter Virtual Fabric Solutions . showing the Emulex 10GbE Virtual Fabric Adapter II ports as the two physical ports on the adapter (Figure 4-3 on page 41). then select Network in the subsequent window. to display the Network configuration panel. Select System Settings.Figure 4-2 Blade UEFI utility main menu 3.

using IBM Virtual Fabric Mode. “Emulex adapter configuration” on page 226. as needed (for our example. With Multichannel enabled. 5. select vNIC (Figure 4-4 on page 42). Unless Multichannel is enabled. Tips: vNIC2 as listed here refers to Switch Independent Mode as discussed in “Switch Independent Mode” on page 99. the card operates as two physical 10-Gbps Ethernet ports. SR-IOV is disabled when using Switch Independent Mode. Switch Independent Mode is not available on the older Emulex 10GbE Virtual Fabric Adapter.Figure 4-3 Network configuration panel in the blade UEFI utility 4.1.IBM Virtual Fabric Mode 41 . The process of enabling Multichannel is discussed in 9. it is enabled on both of the 10-Gb physical ports. When you enable Multichannel.3. To enable IBM Virtual Fabric Mode (vNIC) or Switch Independent Mode (vNIC2) mode. Chapter 4. Select the first port to configure to display the configuration panel. select vNIC or vNIC2 mode. Multichannel needs to be enabled first. Configuring the solution .

listing the vNIC settings. 42 IBM BladeCenter Virtual Fabric Solutions . Select Multichannel Configuration to display the Function Configuration panel. where four functions represent four vNICs (Figure 4-5 on page 43).Figure 4-4 Emulex NIC selection 6.

mode-dependent) Chapter 4. Figure 4-6 Set the Logical Port VLAN ID (LPVID) (optional. Configuring the solution .IBM Virtual Fabric Mode 43 .Figure 4-5 Function Configuration panel 7. Select Function to display the Multichannel Configuration panel in which the parameters for that vNIC will be configured (Figure 4-6).

Figure 4-7 Save settings 4. if we were configuring Switch Independent Mode (selecting vNIC2 in Figure 4-4 on page 42).When in IBM Virtual Fabric Mode. Exit to the UEFI main menu. select Save Settings (Figure 4-7) and restart the blade server. Configuration of this parameter is optional because vNIC functionality does not depend on it. This switch module can be configured using a broad-based web browser (BBI). it is enabled on both physical ports. 9. With Multichannel enabled. 8. Save the settings after making any changes. We suggest that you review the vNIC settings for the second physical port to ensure that the configuration is correct. This is the inner VLAN ID in double-tagged frames (the outer VLAN ID and bandwidth parameters come from the BNT Virtual Fabric 10Gb Switch Module). there is only one parameter: Logical Port VLAN ID (LPVID). However. LPVID would be a required parameter.2 Configuring the BNT Virtual Fabric 10Gb Switch Module This section provides an overview of the tools available for configuring the BNT Virtual Fabric 10Gb Switch Module. a command line interface (CLI) (either 44 IBM BladeCenter Virtual Fabric Solutions .

171.BladeOS or isCLI) or Simple Network Management Protocol (SNMP). Future plans include management of switch modules from BOFM. The BladeOS main menu displays.39. telnet 9. use the following communication methods: Telnet session Serial connection SSH connection Log in to the BladeOS BladeOS CLI consists of a main menu and hierarchically organized submenus. menu-based user interface that provides complete switch module administration. 4.39 2.171.IBM Virtual Fabric Mode 45 . The examples in this section use BNT Virtual Fabric 10Gb Switch Module firmware version 6. 3.42.5. serving as your starting point for all switch module management and configuration tasks (Figure 4-8 on page 46). Chapter 4.3. An example of logging in to the switch module using telnet follows: 1.2. Type the password (the default password is admin).1 BladeOS CLI BladeOS CLI is a simple and intuitive. Use the telnet command to connect to the switch module with IP address 9. To use the BladeOS CLI on the switch module. the current version at the time of this writing. Configuring the solution .0.42.

this procedure upgrades the switch module to this version.3.16.5.0.3.com/support/entry/portal/docdisplay?lndocid=MIGR-5086594 The firmware code consists of three parts: Boot image OS image 1 OS image 2 When upgrading firmware.42. The instructions in this section. Late breaking news: BNT Virtual Fabric 10Gb Switch Module firmware 6.39 NOTICE mgmt: admin(admin) login from host Information Menu Statistics Menu Configuration Menu Operations Command Menu Boot Options Menu Maintenance Menu Show pending config changes [global command] Apply pending config changes [global command] Save updated config to FLASH [global command] Revert pending or applied changes [global command] Exit [global command.7.0 is the current level. so that you can easily activate it should any problems arise with the new level. The firmware update package is located on the IBM support website at: http://ibm.5.171. however. still refer to firmware 6.2 is now available and is required for the Emulex 10GbE Virtual Fabric Adapter II. you need to upgrade the boot image and one of the two OS images. It is suggested that you leave the other OS image at the existing firmware level. This firmware level is also required if the Emulex 10GbE Virtual Fabric Adapter is also updated to the latest firmware. 46 IBM BladeCenter Virtual Fabric Solutions .44.[Main Menu] May 25 14:01:06 9. At the time of this writing. always available] Upgrade the firmware Here we show an example of the BNT Virtual Fabric 10Gb Switch Module firmware upgrade using BladeOS CLI. Therefore.209 info stats cfg oper boot maint diff apply save revert exit >> Main# Figure 4-8 BladeOS CLI main menu 9. firmware 6.

Configuring the solution .5. GbESM-24-10G-6.5. In our example.img: This is the new boot image.Any TFTP or FTP server can be used to transfer the boot and OS images to the BNT Virtual Fabric 10Gb Switch Module. proceed with the firmware upgrade using BladeOS CLI. From the firmware upgrade package.3. we use the TFTP server available on the BladeCenter Advanced Management Module (AMM). Figure 4-9 Firmware files in the tftproot directory After ensuring that TFTP server functionality is enabled on the AMM.img: This is the new OS image.0_Boot.3.0_OS. The contents of the tftproot directory in AMM local storage are shown in Figure 4-9. Chapter 4.IBM Virtual Fabric Mode 47 . we moved two files into the tftproot directory in AMM local storage: GbESM-24-10G-6.

.....img where X represents the OS image bank (1 or 2) and TADDR represents the TFTP server IP address....0_OS.171.3 y Next boot will use new software image2..42.............5..42............171......... Do you want to change that to the new image2? [y/n] May 25 14:57:28 9.3_OS.5.....2 that was downloaded at 6:11:38 Thu Jan 5. use image bank 2)..42. image2 now contains Software Version 6. 1...3.... Confirm download operation [y/n]: y Starting download. >> May 25 14:57:46 9.......5..... 2000.3....42.......Upgrade the OS image The command to transfer the new OS image into one of the image banks is: /boot/gtimg X TADDR GbESM-24-10G-6......42.5..42.......5.1..0_OS.171. Verification of new image2 in FLASH successful........171.... Enter the command to transfer the image to an image bank (in our example.3.......39 INFO mgmt: Firmware downloaded to image2 Figure 4-10 Upgrade of OS image 2 48 IBM BladeCenter Virtual Fabric Solutions ......171... Please wait Write complete (5737743 bytes)..img'..73.img 2.0_OS......5....42. Image download complete (5737743 bytes) Writing to flash.....171.3.39 NOTICE mgmt: boot image changed Boot Options# May 25 14:57:46 9.... Verify the results (Figure 4-10)......39 INFO mgmt: image2 downloaded from host 9.0_OS. file 'GbESM-24-10G-6......5........3 AMM CodeLevel 3 : 0605WMJ03000 BLADEOS Im2 11/17/2010 Switch is currently set to boot software image1.................... New download will replace image2 with file "GbESM-24-10G-6.171...3.img Enter username for FTP server or hit return for TFTP server: image2 currently contains Software Version 6. File appears valid Download in progress ......73 GbESM-24-10G-6.... ...73 GbESM-24-10G-6.73....... ...img" from FTP/TFTP server 9... using the following command: /boot/gtimg 2 9..... >> Main# /boot/gtimg 2 9. now verifying FLASH.....This takes about 90 seconds...... software version 6....

5. Configuring the solution . Notice in Figure 4-10 that this has been changed to image bank 2 during the update process.0_Boot.img where TADDR represents the IP address of the TFTP server.3. Verify the results (Figure 4-11 on page 50).img 2. the switch module was set to boot from image bank 1.3. Upgrade the boot image The command to transfer the new boot image is: /boot/gtimg boot TADDR GbESM-24-10G-6.0. version 6.Figure 4-10 confirms that we have successfully updated OS image 2 to the new firmware version.73 GbESM-24-10G-6.5.171. To begin the transfer: 1.0_Boot. Before the upgrade.42. Chapter 4. Enter the command to transfer the image: /boot/gtimg boot 9.5.3.IBM Virtual Fabric Mode 49 .

...........2 New download will replace boot kernel with file "GbESM-24-10G6........ Writing to flash.5..5.3 >>May 25 15:13:17 9.42......3...5...img" from FTP/TFTP server 9....171... 7687332 bytes) download complete..3 Boot image (Boot..42.42..........img'..0_Boot....3 Boot Options# May 25 15:13:17 9. ...5..This can take up to 90 seconds.........171..... Writing to flash...5... 7687332 bytes) download complete.....73....... file 'GbESM-24-10G-6........... available at: http://ibm.... File appears valid Download in progress ...>> Boot Options# /boot/gtimg boot 9... Please wait FS Sector now contains Software Version 6....This can take up to 90 seconds....171..img Enter username for FTP server or hit return for TFTP server: boot kernel currently contains Software Version 6...171. see Command Reference 6..39 INFO mgmt: boot kernel downloaded from host 9..... Boot image (FS.3. Please wait Kernel Sector now contains Software Version 6.3...39 INFO mgmt: Firmware downloaded to boot kernel Figure 4-11 Confirmation of the boot image upgrade Reset the switch module Enter the command to reset the switch module: /boot/reset Assistance with BladeOS CLI commands For more information about BladeOS CLI commands.. ........... software version 6..... BNT Virtual Fabric 10 Gb Switch Module..5.......3 Boot image (Kernel...0_Boot......1......com/support/entry/portal/docdisplay?lndocid=MIGR-5080917 50 IBM BladeCenter Virtual Fabric Solutions ....42..........73.......0_Boot......... Writing to flash...........171......3.... 7687332 bytes) download complete.. Please wait Boot Sector now contains Software Version 6...... Confirm download operation [y/n]: y Starting download.42...73 GbESM-24-10G6...This can take up to 90 seconds...5.

BladeOS CLI is the default CLI mode. for example: Switch> Privileged EXEC mode: This mode has a higher privilege level. isCLI privilege levels The isCLI mode has three privilege levels: User EXEC mode: This is the initial mode with the most limited privileges. for example: Switch# Privileged EXEC mode is accessed from User EXEC mode with the command: enable Global Configuration Mode: This is the most privileged mode. To revert from isCLI to BladeOS CLI. you can make changes to the running configuration. even when you restore the switch module to factory defaults. With this privilege level set. System administrators who are more familiar with Cisco CLI have the option of using the industry standard CLI (isCLI).2. but you can easily change between BladeOS and isCLI. but you cannot set or change operating parameters. “>”. The selected CLI mode is preserved across reboots.2 isCLI BladeOS is a proprietary CLI environment. With this privilege level set. the isCLI prompt ends with the greater than character.4. developed by BNT for configuring and managing their switch modules. such as list system information. “#”. and set or change Chapter 4. run the command: boot cli-mode bladeos-cli Restart the switch module after the command is run. which allows the complete management and configuration support of BNT switch modules. Configuring the solution . It allows you to set or change operating parameters of the switch module. the isCLI prompt ends with the number sign (also known as the pound sign).IBM Virtual Fabric Mode 51 . You can perform basic tasks. use the following command: boot/mode iscli Restart the switch module after this command is run. isCLI is a Cisco-like user interface. When in this mode. Change your choice of CLI between BladeOS and isCLI The following shows how to change from BladeOS to isCLI and from isCLI to BladeOS: To change from BladeOS to isCLI.

ibm. Figure 4-12 is an example of enabling Privileged EXEC and then Global Configuration Mode. available at: http://www. Select the following options in the AMM web interface: I/O Module Tasks  Configuration  Advanced Options  Start CLI/Web Session 52 IBM BladeCenter Virtual Fabric Solutions . Open a web browser. To enable a different privilege level.3 Browser-based interface (BBI) The browser-based interface (BBI) can be used to perform most management. BBI is an easy-to-use and intuitive.com/support/docview. Assistance with isCLI commands For more information about isCLI commands. web-based management interface and is enabled by default. enter the commands shown in Figure 4-12.2. and connect to the switch module IP address. Use the following command to enter this mode from Privileged EXEC mode: configure terminal To change the privilege level. one per line. Router#configure terminal Enter configuration commands. BNT Virtual Fabric 10 Gb Switch Module. administration. you can access BBI from the AMM web interface: a. see “isCLI privilege levels” on page 51. Router>enable Enable privilege granted. see isCLI Reference 6.3. and configuration tasks on the BNT Virtual Fabric 10Gb Switch Module.global parameters. Router(config)# Figure 4-12 isCLI: Enabling EXEC privileges End with Ctrl/Z. Alternatively. Launch the BBI The following shows how to log in to the switch module and launch BBI: 1.wss?uid=psg1MIGR-5080917 4. It can be disabled using the CLI.

Figure 4-13 AMM web interface: Starting the CLI web session 2. Chapter 4. then click Start Session. then click Start Session (Figure 4-13) Ensure Web is selected as the Protocol. Log in to the switch module (Figure 4-14 on page 54). Configuring the solution .b.IBM Virtual Fabric Mode 53 . Default credentials are: Username: admin Password: admin Note: Clients following best practice recommendations have likely changed the password from the default for added security. Ensure that the Protocol field is set to Web (not Telnet).

Figure 4-14 Login panel for the BNT Virtual Fabric 10Gb Switch Module 3. the BBI Switch Dashboard is displayed (Figure 4-15 on page 55). After a successful login. 54 IBM BladeCenter Virtual Fabric Solutions .

see the BBI Quick Guide.Figure 4-15 BNT Virtual Fabric 10Gb Switch Module dashboard The switch dashboard is your starting point for switch module management. Configuring the solution . Assistance with BBI commands For more information about BBI commands.IBM Virtual Fabric Mode 55 .com/support/entry/portal/docdisplay?lndocid=MIGR-5080917 Chapter 4. available at: http://ibm.

for instance. which can be used to temporarily disable vNIC without removing all of the configuration details.3. These changes are propagated to the servers immediately and do not require that the network be brought down or that the server be rebooted. we show how to create vNICs in a BNT Virtual Fabric 10Gb Switch Module. using three methods: BBI: a browser-based interface isCLI: the international standard command line interface (CLI) BladeOS: the BNT CLI Each vNIC is created and configured on the switch. Oversubscription is not supported. It is possible. there is no provision to allow a given vNIC to exceed its allocated bandwidth at any time. Bandwidth allocations are policed on both inbound and outbound traffic.2 Enabling and disabling vNIC functionality on the server To configure vNICs on one blade. Bandwidth allocations can be changed from the switch command line or browser interface.1 Before you configure Consider the following when creating vNICs: Up to four vNICs can be created on any blade. begin by enabling vNIC functionality. If less than 10 GB of total bandwidth is allocated to all of the defined vNICs on a port. 56 IBM BladeCenter Virtual Fabric Solutions . Only 10 Gb of total bandwidth can be allocated to the vNICs on one port on a server blade. 4. regardless of how many blades will use the vNIC.3 Creating virtual NICs In this section. This command is carried out once per switch. Unlike certain configurations that use quality of service. The properties of each vNIC are communicated to the Emulex adapter using the Data Center Bridging eXchange (DCBX) protocol.3. This process uses a master toggle switch. to have one bandwidth allocation scheme in use during business hours and a different one during off hours when batch processing is in progress. There is no requirement to create all four. then the unallocated bandwidth is not usable.4. and unlike the virtual NIC functionality built into the IBM AIX® operating system. 4.

IBM Virtual Fabric Mode 57 . Disable it by setting the parameter to Off. Example 4-1 Enabling vNIC mode BNT Virtual Fabric 10Gb Switch Module for IBM BladeCenter.39 NOTICE mgmt: admin(admin) login from host 9.171.42. Enter password: System Information at 19:12:16 Sun Jan 8.168. 2000 Router> Jan 8 19:12:16 9. Configuring the solution . End with Ctrl/Z.83 en Enable privilege granted. Figure 4-16 Enabling vNIC mode Using isCLI You can enable vNIC mode using isCLI with the command vnic enable as shown in Example 4-1.Using BBI You can enable vNIC mode using BBI by setting the Global vNIC On/Off parameter to On as shown in Figure 4-16 and clicking Submit. Router#configure terminal Enter configuration commands.44. Router(config)#vnic enable Chapter 4. one per line.

always available] 9. 1.253 cfg 9.171.171. 2. Enter cfg for the Configuration menu and press Enter (Figure 4-17).155 info stats cfg oper boot maint diff apply save revert exit >> Main# Jan 15 16:56:33 9.39 WARNING ntp: cannot contact primary NTP server Figure 4-17 The Configuration menu 3. no message is displayed.168. Using BladeOS Enabling vNIC mode using BladeOS is shown in Figure 4-17 and Figure 4-18 on page 59. Enter virt for the Virtualization menu and press Enter (Figure 4-18 on page 59).42. [Main Menu] Jan 15 16:56:14 9.42. 171. 58 IBM BladeCenter Virtual Fabric Solutions .Router(config)# You can disable vNIC mode using isCLI with the command no vnic enable as shown in Example 4-2.42.44.39 NOTICE mgmt: admin(admin) login from host Information Menu Statistics Menu Configuration Menu Operations Command Menu Boot Options Menu Maintenance Menu Show pending config changes [global command] Apply pending config changes [global command] Save updated config to FLASH [global command] Revert pending or applied changes [global command] Exit [global command. Example 4-2 Disabling vNIC mode Router(config)#no vnic enable Router(config)# Note: When you have successfully enabled or disabled vNIC mode using isCLI. Navigate to /cfg/virt/vnic to display the Main Menu.

Layer 3 Menu cee . Chapter 4.Restore current configuration from FTP/TFTP server cur . Enter vmpolicy for the Virtual Machines policy Configuration menu and press Enter (Figure 4-19 on page 60).Display current configuration Figure 4-18 The Virtualization menu 4.Virtualization Menu setup .Backup current configuration to FTP/TFTP server gtcfg .RMON Menu virt .System-wide Parameter Menu port .CEE Configuration Menu fcoe . Configuring the solution .Dump current configuration to script file ptcfg .QOS Menu acl . 5.Port Mirroring Menu l2 .IBM Virtual Fabric Mode 59 .Access Control List Menu pmirr .[Configuration Menu] sys .Step by step configuration set up dump .Layer 2 Menu l3 . Enter on to enable vNIC functionality.Fiber Channel Over Ethernet Configuration Menu rmon .Port Menu qos .

Enable VMready disvmr . In most installations.3. The same commands are used on both switches if the two physical ports are to be configured identically. and vNIC numbers are 1 through 4. When issuing the commands.VNIC Group Configuration Menu on .Port vNIC Configuration Menu vnicgrp . you will likely have two BNT Virtual Fabric 10Gb Switch Modules in your chassis.Virtual Machines Policy Configuration Menu vnic . 60 IBM BladeCenter Virtual Fabric Solutions . once on each of the two switches.Virtual Machine Profiles Menu vmware .Display all current virtualization settings >> Virtualization# vnic -----------------------------------------------------------[VNIC Global Configuration Menu] port .Globally turn vNIC feature ON off . especially if trunk failover is to be used for high availability.Display current vNIC configuration >> VNIC Global Configuration# on Current status: ON New status: ON >> VNIC Global Configuration# Figure 4-19 Using BladeOS Cli to enable vNIC 4.Globally turn vNIC feature OFF cur .3 Creating the vNICs In this section we describe how to create vNICs.vNIC Configuration Menu vmgroup . remember that: Blade numbers (internal port numbers) are 1 through 14.Disable VMready cur .Virtual Machines Groups Menu vmprof . so this process needs to be done twice.VMware-specific Settings Menu enavmr .>> Configuration# virt -----------------------------------------------------------[Virtualization Menu] vmpolicy . which also specifies the percentage of 10-Gb bandwidth allocated to a specific vNIC. Bandwidth is specified in 100-MB increments.

the configuration process is carried out using the Configuration tab. Enable vNIC # 8 on blade 8 Allocate 30% of the 10 Gb to this vNIC (3 Gb) Enable vNIC #2 on blade 8 Allocate 70% of the 10 Gb to this vNIC (7 Gb) The commands to do these tasks and the output we got are shown in Figure 4-21 on page 62. Select the vNIC. Configuring the solution . we want to do the following configuration tasks using isCLI: 1. set the State to Enabled and set the Maximum bandwidth (in our case.Using BBI Using BBI. Chapter 4. as shown in Figure 4-20. 3.IBM Virtual Fabric Mode 61 . 4. Figure 4-20 Creating a vNIC Using isCLI: Using isCLI the command syntax is as follows: vnic port <port alias or number> index <1-4> bandwidth <1-100> enable or no enable For example. 2. we entered 30 for 3 Gbps).

Displays the port vNIC menu.enables the selected vNIC.disables the selected vNIC. ena -.202 en Enable privilege granted. 62 IBM BladeCenter Virtual Fabric Solutions . End with Ctrl/Z. 10 = 1000 Mbps. dis -. Router#configure terminal Enter configuration commands. vnic <1-4> -.42. bw <1-100> -.168. the relevant commands are as follows: port <port number> -.Jan 9 17:02:38 9. cur -. Router(config)#vnic port 8 index 1 Router(vnic_config)#band Router(vnic_config)#bandwidth 30 Router(vnic_config)#enable Router(config)#vnic port 8 index 2 Router(vnic_config)#bandwidth 70 Router(vnic_config)#enable Router(vnic_config)# Figure 4-21 Using isCLI to create a vNIC Using BladeOS Using BladeOS. one per line.171. 1 = 100 Mbps.Displays the vNIC menu for the selected vNIC.39 NOTICE mgmt: admin(admin) login from host 9. The commands we issued in our example and the output we got are shown in Figure 4-22 on page 63.Configures the maximum bandwidth allocated to this vNIC. in increments of 100 Mbps.displays the current vNIC port parameters.44. For example.

1.2# bw 20 Current vNIC Maximum Bandwidth: 25 (increments of 100Mbps) Pending vNIC Maximum Bandwidth: 20 (increments of 100Mbps) Figure 4-22 Using BladeOS Cli to configure vNIC 4.Port vNIC Configuration Menu vnicgrp . and only one vNIC from a given port can be in any specific group.Display current vNIC configuration >> VNIC Global Configuration# port Enter port (INT1-INT14): 4 --------------------------------------------------------[Port INT4 vNICs Menu] vnic .IBM Virtual Fabric Mode 63 .2# ena Current status: disabled New status: enabled Warning: "Tagging" is enabled on vNIC port INT4 vNIC INT4.Display current port vNIC configuration >> Port INT4 vNICs# vnic Enter vNIC (1-4): 2 --------------------------------------------------------[vNIC INT4. Add a physical port to the group. it must be a member of a single vNIC group.VNIC Group Configuration Menu on . Up to 32 groups can be created.> Virtualization# vnic --------------------------------------------------------[VNIC Global Configuration Menu] port . and (optionally) one or more non-vNIC server blade-facing (INTernal) ports.Globally turn vNIC feature OFF cur . For a vNIC to communicate.2 Menu] bw . (optionally) one or more uplink (EXTernal) ports. 3. Configuring the solution .Display current vNIC configuration >> vNIC INT4.Disable vNIC cur .Globally turn vNIC feature ON off . Chapter 4.Enable vNIC dis . 2. Add a vNIC to the group. The overall steps for configuring a vNIC group are as follows. Enable or disable a group.4 vNIC group configuration A vNIC group consists of one or more vNICs.VNIC Configuration Menu cur .Set maximum bandwidth of the vNIC ena .

remport. 64 IBM BladeCenter Virtual Fabric Solutions . Enable or disable trunk failover. The commands to add a vNIC and a trunk to a vNIC group are shown in Figure 4-23 on page 65.7. “High availability and vNICs” on page 70.4. The commands for these steps are: /cfg/virt/vnic/vnicgrp /cfg/virt/vnic/vnicgrp /cfg/virt/vnic/vnicgrp /cfg/virt/vnic/vnicgrp /cfg/virt/vnic/vnicgrp <number>/{ena or dis} <number>/addvnic <blade#. and remtrnk commands can be used to remove the corresponding items from the group. Add an external trunk group to the vNIC group.vnic#> <number>/addport <INT or EXT port> <number>/addtrnk <trunk #> <number>/failover {ena or dis} Tip: The remvnic. as discussed in 4. 5.

Display current vNIC group configuration >> vNIC Group 8# addvnic INT4.Remove port from vNIC group addtrnk .Remove vNIC from vNIC group addport .Globally turn vNIC feature ON off . Configuring the solution .Enable vNIC group dis .Port vNIC Configuration Menu vnicgrp .Display current vNIC configuration >> VNIC Global Configuration# vnicgrp Enter vNIC Group (1-32): 8 -----------------------------------------------------------[vNIC Group 8 Menu] vnicvlan .Add port to vNIC group remport .Add trunk to vNIC group remtrnk .Remove trunk from vNIC group ena .VNIC Group Configuration Menu on .Set VLAN number to vNIC group failover .Disable vNIC group del .2 >> vNIC Group 8# addtrnk 3 >> vNIC Group 8# ena Current status: disabled New status: enabled >> vNIC Group 8# Figure 4-23 Using BladeOS Cli to create vNIC Group The isCLI equivalent is: vnic vnicgroup <number 1-32> member INT<port>.<vnic number 1-4> port INT<port number or alias> or EXT <port number or alias> vlan <vlan number 1-4094> trunk <trunk number> failover enable exit The exit command is required to end the configuration for a specific group.Globally turn vNIC feature OFF cur .>> Virtualization# vnic -----------------------------------------------------------[VNIC Global Configuration Menu] port .Delete vNIC group cur .IBM Virtual Fabric Mode 65 .Add vNIC to vNIC group remvnic .Enable/disable uplink failover addvnic . Chapter 4.

as discussed in “Configuring vNIC VLANs” on page 66. Router#configure terminal Enter configuration commands. Router(config)#vnic vnicgroup 2 Router(vnic group config)#member INT8. only static trunk groups are supported.Note: The no member. no vlan.171. Any uplink port or trunk can only be part of one vNIC group at a time. 4. Router>Jan 10 17:20:24 9. one per line.44.1 Router(vnic group config)#vlan 4091 Warning: VLAN 4091 is moved to STG 1.202 en Enable privilege granted. Router(vnic group config)#failover Router(vnic group config)#trunk 2 Router(vnic group config)#enable Warning: STP is turned off for port EXT1 in STG 1 Warning: Changed the pvid of uplink port EXT1 in vNIC group 2 to 4091 Warning: Deleted port EXT1 from VLAN 1 Router(vnic group config)# Figure 4-24 Creating vNIC group Currently. no trunk. no port. and no failover commands can be used to remove the corresponding items from the group.39 NOTICE mgmt: admin(admin) login from host 9.42. These outer tags are never forwarded to a blade or to an 66 IBM BladeCenter Virtual Fabric Solutions . Link Aggregation Control Protocol (LACP) trunks are supported in the Switch Independent Mode (see Part 2. Each vNIC group is associated with a vNIC VLAN. End with Ctrl/Z. Example commands are shown in Figure 4-24. At present.5 Configuring vNIC VLANs Every vNIC channel is associated with a vNIC group with a unique VLAN ID. These group VLANs differ from conventional VLANs that are configured on the Virtual Fabric switch in significant ways: vNIC VLANs are implemented as outer tags on the Ethernet frames that pass through the switch.168. This means that it is not currently possible for multiple groups to share uplinks. only one uplink port or one trunk group of uplink ports aggregated together can be part of a given vNIC group. “Switch Independent Mode” on page 99).

the vnicvlan value is inserted as the second tag (ID "B"). where the vnicvlan "outer" tag is removed.upstream switch. 802. a given VLAN number can be a conventional VLAN.1Q VLAN tag VLAN ID "A" EtherType = 0x8100 Ethernet Header SA DA Tagged traffic for multiple VLANs is sent from the upstream switch.1Q VLAN tag VLAN ID "A" EtherType = 0x8100 Ethernet Header SA DA Figure 4-25 Double tagging The same conventional VLAN can enter the chassis in more than one group. Within a group. or a VMready group. a vNIC group.1Q VLAN tag 2 VLAN ID "A" EtherType = 0x8100 . but traffic between groups never occurs within the Virtual Fabric switch. Chapter 4. Ethernet Header EtherType (of payload) . Configuring the solution .IBM Virtual Fabric Mode 67 . but not more than one of these at the same time. These inner tags pass through the group and the Virtual Fabric switch completely unchanged. They serve solely to isolate traffic in a vNIC group from other vNIC groups and from conventional VLANs. Upon entry to the Virtual Fabric switch. On the Virtual Fabric switch. but the Virtual Fabric switch simply passes the inner tags through (Figure 4-25).1Q VLAN tag 1 VLAN ID "B" EtherType = 0x8100 Ethernet Header SA DA Traffic flows through the Virtual Fabric switch using the vnicvlan tag and is forwarded to the blade-facing port. conventional VLANs configured by an operating system on a server blade (such as with the vconfig command in Linux) are implemented as inner tags. Ethernet Header EtherType (of payload) . The VLAN configuration on upstream switches needs to match that on the blades.1q "Double Tagging" and the BNT Virtual Fabric Switch Upstream Router BNT Virtual Fabric Switch HS22 Blade Payload Payload Payload Ethernet Header EtherType (of payload) .

(This is with the addport command.Additional notes about vNIC VLANs: Routing of traffic between vNIC VLANs is not supported. there is no way to split these ports and aggregate them in more than one trunk. Use high numbers that are not used in the client’s network as vNIC VLANs (for example. it is assumed that there will be no vNICs created on those blades. Any inner tags associated with the non-VNIC interface on the ports will be unchanged. 68 IBM BladeCenter Virtual Fabric Solutions . nor is routing between a vNIC VLAN and a conventional VLAN. the BladeOS commands are: /cfg/virt/vnic/vnicgrp <number> vnicvlan <1-4094> The selected VLAN number cannot be used within the Virtual Fabric switch for any other purpose. it is of purely local significance in that it is never seen by the blades (other than by the vNIC driver). vNIC VLANs are configured with the associated vNIC groups. When EXTernal (uplink) ports are added to a vNIC group. just as with the vNICs. not the addvnic command. 4000 and up).) Traffic can then flow between vNICs and these INTernal ports. and they are always aggregated together. When INTernal (blade-facing) ports are added to a vNIC group. and never forwarded out of the Virtual Fabric switch to an upstream switch. It will be forwarded using the vNIC VLAN as the outer tag. The isCLI commands are: vnic vnicgroup <number 1-32> vlan <vlan number 1-4094> To configure vNIC VLANs. they are also added to the vNIC VLAN. At present. However.

with the release of 6. Configuring the solution . Figure 4-26 Configuring the vNIC group and the associated VLAN 4.7.Figure 4-26 shows the BBI window to configure a vNIC group and the associated VLAN. However.IBM Virtual Fabric Mode 69 . Chapter 4. this is now possible.7.2.6 vNIC and VMready Versions of BNT Virtual Fabric 10Gb Switch Module firmware prior to 6.2 did not support a blade-facing port being in a vNIC group and a VMready group at the same time.

70 IBM BladeCenter Virtual Fabric Solutions . With vNICs. this appears as a network cable unplugged condition on the server. – ESX has support for multiple NIC teaming modes using the network configuration GUI. Configure uplink ports or trunks as members of the corresponding vNIC groups on the two switches. associated groups. This failover function is also available when vNICs are used. do the following: 1. if the uplink ports that are members of a group (vnicgrp) fail due to a failure on the upstream device or a cable failing or being erroneously removed. just not the same port.2. NIC teaming must be performed in active/standby mode. Configure NIC teaming (as above) on corresponding pairs of vNICs on the server. if possible. Make sure that one member of the team is physically connected to one of the Virtual Fabric switches. use the available utility from Emulex. but not required. It is a good idea. must be teamed together using the available tools for NIC teaming. – Linux servers must have the native bonding driver. 2. The corresponding vNIC ports. It is good practice to connect these uplinks to separate upstream physical switches. to use vNIC to support certain blades and VMready on other blades using the same switch and within the same chassis. In general. 4. Multiple vNICs that are part of the same physical 10-Gb port can support high availability. Do so on both of the Virtual Fabric switches.With versions prior to 6.7 High availability and vNICs The trunk failover function enables high-availability designs when used with standard ports. however. In Windows. 2. using the following specifications: – For Windows servers. as seen by the operating system. it is possible. take the following steps: 1. 3. the associated vNIC is brought down by the Virtual Fabric switch.7. To exploit the trunk failover function to deliver high availability. and bandwidth allocation be used on both switches. that the same vNIC numbers. In order to implement this. and the other member of the team to the other Virtual Fabric switch. but each vNIC does so independently. The other Virtual Fabric switch in the chassis must also be configured to use vNICs for the server in question. Enable failover for all vNIC groups for which the server ports are members.

1 and EXT10 on each switch GREEN – vNIC Group 2.1 vNIC 3.2 Blade #3 vNIC 3.IBM Virtual Fabric Mode 71 . VLAN 2001 (outer tag) vNIC 3.Figure 4-27 is a sample configuration showing only one blade. High Availability with vNIC Bay 7 10-port switch Switch bay Port INT3 vNIC 3.1 vNIC 3. the commands for the blade configuration in Figure 4-27 are: vnic port INT3 index 1 enable vnic port INT3 index 2 enable vnic vnicgroup 1 failover vlan 2001 member INT3.1 port EXT10 enable exit Chapter 4. The same configuration commands are required for each blade used for which failover is desired. VLAN 2002 (outer tag) vNIC 3.2 and EXT9 on each switch BLACK – Physical port INT3 on each switch Bay 9 10-port switch Figure 4-27 High availability with vNIC Using isCLI. Configuring the solution .2 Router Port INT3 Legends: RED – vNIC Group 1.

4 72 IBM BladeCenter Virtual Fabric Solutions .1 addport EXT10 /cfg/virt/vnic/vnicgrp 2/ena failover ena vnicvlan 2002 addvnic 3. we show how to configure teams and VLANs with the Emulex 10GbE Virtual Fabric Adapter.8 Configuring the operating system In this section.2 port EXT9 enable exit The BladeOS configuration for Figure 4-27 is identical on each of the two switches and includes: /cfg/virt/vnic /port 3/vnic 1/ena /cfg/virt/vnic/port 3/vnic 2/ena /cfg/virt/vnic/vnicgrp 1/ena failover ena vnicvlan 2001 addvnic 3.5 and 5.6 Red Hat Enterprise Linux 6.vnic vnicgroup 2 failover vlan 2002 member INT3.2 addport EXT9 For more information. available from: http://ibm.com/support/entry/portal/docdisplay?lndocid=MIGR-5080917 4.0 and 6. refer to BNT Virtual Fabric 10Gb Switch Module for IBM BladeCenter Application Guide.1 SUSE Linux Enterprise Server 10. The following operating systems support the Emulex 10GbE Virtual Fabric Adapter: Microsoft Windows Server 2003 (x86 and x64) Microsoft Windows Server 2008 (x86 and x64) Microsoft Windows Server 2008 Release 2 (x64 only) Red Hat Enterprise Linux 5.3 and 10.

You can use the OneCommand Manager to configure load balancing and teaming. 4. Configuring the solution . Teams also improve system availability by providing a failover path for critical network connections.0 Citrix XenServer 5. OneCommand Manager is the Emulex application that enables the centralized management of Emulex OneConnect Converged Network Adapters (CNAs) and Host Bus Adapters (HBAs) from a centralized management console. Chapter 4. is usually assigned to the physical adapter. The IPCONFIG /all command for Windows or IFCONFIG command for Linux shows the IP and MAC addresses of the virtual adapter and not of the individual physical adapters. The benefits of teaming are: Virtual adapters can balance a large network load across several adapters. A protocol address. and 5.0.1 Configuring teams and VLANs A team (or bond in Linux) of adapters functions as a single virtual network interface and appears the same as a nonteamed adapter to other network devices. such as an IP address. 4.8.IBM Virtual Fabric Mode 73 .1 VMware ESX 4.6 Citrix XenServer 6. make sure the following firmware for other BladeCenter devices is at the latest level.1.SUSE Linux Enterprise Server 11. when OneCommand NIC Teaming and Multiple VLAN Manager (OneCommand NIC Teaming Manager) is installed.0 Note: Before installing the device drivers and firmware. the protocol address is assigned to the team adapter and not to the physical adapters that make up the team. However. OneCommand Manager provides a GUI and a scriptable CLI for administration.

Use the Create Team button to create a new team configuration (Figure 4-28). 1. Figure 4-28 Configuring a new team using OneCommand NIC Teaming Manager 74 IBM BladeCenter Virtual Fabric Solutions .2 Configuring a new team To configure a new team. start the OneCommand NIC Teaming Manager application and proceed as follows.4.8.

Choose a Team Type from the drop-down list. The default is enabled.Figure 4-29 Configuring a new team 2. Chapter 4. 3. change the Team Type criteria for Load Distributed By to one of the following: – Default: Failover. Selects the port with the least traffic load for the session. Enable or disable auto failback functionality. 4. 5.IBM Virtual Fabric Mode 75 . The default is Failover. Enter a name of the new team in the Team Name field (Figure 4-29). – Destination IP Address: Performs an XOR on the destination and source IP address to determine which port should carry the load. If necessary. Configuring the solution . – Destination MAC Address: Performs an XOR on the destination MAC address to determine which port should carry the load.

all virtual adapter members of the team should have the VLAN disabled at the physical or operating system level.6. The Team Member Adapters area lists the adapters that are members of the team. do the following using the OneConnect software: 1. Select the physical or team adapter in the Team and Available Adapters window (Figure 4-30 on page 77) to which you want to add a VLAN. a window displays as in Figure 4-29. The Available Network Adapters area lists the adapters that are not members of any team. To configure a VLAN for either a physical or team adapter. Click OK. 7.8. With the team created. 4. 76 IBM BladeCenter Virtual Fabric Solutions . As a result. Configuring the VLAN at both the physical and team levels may cause double tagging and should be avoided. 8. and click Add to display the adapter in the Team Member Adapters list. and click VLAN.3 Configuring a VLAN for a physical or team adapter You can configure VLANs at either the physical adapter level or at the team level. To add an adapter to the team. select the adapter from this list. showing that the adapter has been added to the team.

Configuring the solution . Chapter 4.Figure 4-30 Configuring a new team 2. for example 50. Figure 4-31 on page 78. enter a valid VLAN ID in the range 1-4094.IBM Virtual Fabric Mode 77 . In the NIC Teaming and VLAN Manager dialog box that appears.

The VLAN Name shows the VLAN Name in the format Vlan_<VLAN ID>.Figure 4-31 Configuring a VLAN 3. Click Add to add the VLAN to the adapter. as shown in Figure 4-31. Note: You can create multiple VLANs for an adapter. 78 IBM BladeCenter Virtual Fabric Solutions . The VLANs Configured list shows the list of all VLANs configured for the adapter.

The VLAN is now created.IBM Virtual Fabric Mode 79 . Chapter 4.Figure 4-32 Confirmation of configured VLAN 4. Click OK to complete the configuration. Configuring the solution .

80 IBM BladeCenter Virtual Fabric Solutions .

Examples of use . one uplink” on page 86 5. 2011.1. vNIC mode is the default. “Example 3: vNICs only. we describe the steps to create three separate virtual fabric configurations. 81 . “Example 2: vNICs only.IBM Virtual Fabric Mode In this chapter. one uplink trunk” on page 91 Tip: For the Virtual Fabric Adapter. no uplinks” on page 82 5. “Example 1: vNICs only.2. The topics in this chapter are: 5.3. All rights reserved.5 Chapter 5. Tip: You also need to ensure that vNIC mode is enabled in the Emulex Virtual Fabric Adapters. the PCIe function ID can be determined by running the ethtool -i command in Linux or by viewing the Network Adapter Properties in Windows Device Manager. comprising combinations of: Virtual network interface controllers (vNICs) Uplink ports Uplink trunks All of the configuration performed here is done using interfaces into the BNT Virtual Fabric 10Gb Switch Module. © Copyright IBM Corp.

1 (blade 2. set to a maximum bandwidth of 2 Gbps (20% of 10 Gbps) One vNIC group containing these two vNICs (vNIC Group VLAN 127) The BladeOS command script in Example 5-1 can be used to implement this configuration. Example 5-2 isCLI script to create Scenario 1 vnic enable vnic port INT1 index 2 enable bandwidth 50 vnic port INT2 index 1 enable bandwidth 20 vnic vnicgroup 1 enable vlan 127 member 1.5. The configuration is (Figure 5-1 on page 83): vNIC INT1. set to a maximum bandwidth of 5 Gbps (50% of 10 Gbps) vNIC INT2. vNIC 1) of the switch in I/O bay 7. vNIC 2) of the switch in I/O bay 7.1 Example 1: vNICs only.2 addvnic INT2.1 apply The isCLI command script in Example 5-2 can be used to implement this configuration. Example 5-1 BladeOS script to create Scenario 1 /c/virt/vnic on /c/virt/vnic/port INT1/vnic 2 ena bw 50 /c/virt/vnic/port INT2/vnic 1 ena bw 20 /c/virt/vnic/vnicgrp 1 ena vnicvlan 127 addvnic INT1.2 82 IBM BladeCenter Virtual Fabric Solutions .2 (blade in slot 1. which consists solely of vNIC ports. no uplinks Example 1 creates group VGRP 1.

and then select On and click Submit  Apply  Save. Figure 5-1 Example 1: Enable vNIC operations on the vNIC Configuration panel Chapter 5. Enable vNIC operation by expanding Virtualization  VNIC  General.1 exit The Virtual Fabric Switch browser-based interface (BBI) can also be used to implement this configuration. Click the Configure tab of the BBI interface. as follows: 1.IBM Virtual Fabric Mode 83 . 2. Examples of use . Tip: Click Submit  Apply  Save after every change to ensure that the setting persists after any switch reboot.member 2.

and then enter the bandwidth allocation (Figure 5-3). Figure 5-3 Example 1: Enabling vNIC INT1. For the first vNIC. Figure 5-2 Example 1: vNICs Configuration panel 4. select Enabled.3. Enable and configure each of the vNICs by selecting Virtualization  VNICs (Figure 5-2).2 84 IBM BladeCenter Virtual Fabric Solutions .

IBM Virtual Fabric Mode 85 . Examples of use . Figure 5-4 Example1: Enabling vNIC INT2. and then enter the bandwidth allocation (Figure 5-4). Chapter 5.1 6.5. d. select Enabled. Set the VLAN to 127. c. Click Add. Enable the group. Enable and configure the vNIC group by clicking Virtualization  VNIC  VNIC Groups (Figure 5-5). For the second vNIC. Figure 5-5 Example 1: vNIC Groups Configuration panel 7. Click vNIC Group 1 and Figure 5-6 on page 86 displays: a. Select the two vNICs. b.

Example 5-3 BladeOS script for Example 2 /c/virt/vnic on /c/virt/vnic/port INT1/vnic 3 ena bw 25 /c/virt/vnic/port INT2/vnic 2 86 IBM BladeCenter Virtual Fabric Solutions .3 (blade in slot 1. The configuration is (Figure 5-7 on page 88): vNIC INT1. vNIC 3) of the switch in I/O bay 7.2 Example 2: vNICs only. set to a maximum bandwidth of 7 Gbps (70% of 10 Gbps) One vNIC group containing these vNICs (vNIC Group VLAN 500) One uplink port.2 (blade 2. one uplink Example 2 creates group VGRP 3. set to a maximum bandwidth of 2.5 Gbps (25% of 10 Gbps) vNIC INT2. vNIC 2) of the switch in I/O bay 7.Figure 5-6 Example 1: vNIC Group 1 Configuration panel 5. EXT2 The BladeOS command script in Example 5-3 can be used to implement this configuration. which consists of vNIC ports and one uplink port.

3 member 2. Examples of use .2? addport EXT2 apply The isCLI command script in Example 5-4 can be used to implement this configuration.2 port EXT2 exit Chapter 5.IBM Virtual Fabric Mode 87 .ena bw 70 /c/virt/vnic/vnicgrp 3 ena vnicvlan 500 addvnic INT1. Example 5-4 isCLI script for Example 2 vnic enable vnic port INT1 index 3 enable bandwidth 25 (default) vnic port INT2 index 2 enable bandwidth 70 vnic vnicgroup 3 enable vlan 500 member 1.3 addvnic INT2.

2. 1. and then select On and click Submit  Apply  Save. as follows. Figure 5-7 Example 3: Enable vNIC operations on the vNIC Configuration panel 88 IBM BladeCenter Virtual Fabric Solutions . Click the Configure tab of the BBI interface.The Virtual Fabric Switch BBI can also be used to implement this configuration. Enable vNIC operations by expanding Virtualization  VNIC  General. Tip: Click Submit  Apply  Save after every change to ensure that the setting persists after any switch reboot.

Examples of use . Figure 5-8 Example 3: vNICs Configuration panel 4. Figure 5-9 Example 3: vNIC INT1.IBM Virtual Fabric Mode 89 . select Enabled.3 Configuration panel Chapter 5. For the first vNIC. Enable and configure each vNIC by clicking Virtualization  VNICs (Figure 5-8).3. and then enter the bandwidth allocation (Figure 5-9).

b. c. Enable and configure the vNIC group by selecting Virtualization  VNIC  VNIC Groups (Figure 5-11). e. select Enabled. Select the vNICs. Figure 5-10 Example 3: vNIC INT2. d.2 Configuration panel 6. 90 IBM BladeCenter Virtual Fabric Solutions . Select the EXT2 uplink port. Click Add. Set the VLAN to 500. and then enter the bandwidth allocation (Figure 5-10). For the second vNIC. Enable the group.5. Click vNIC Group 3 and Figure 5-12 on page 91 displays: a. Figure 5-11 Example 3: vNIC Groups Configuration panel 7.

vNIC 1) of the switch in I/O bay 9. set to a maximum bandwidth of 3 Gbps (30% of 10 Gbps) vNIC INT2.2 (blade 2. Example 5-5 BladeOS script for Example 3 /c/l2/trunk 1 ena add EXT1 add EXT3 /c/virt/vnic on Chapter 5.Figure 5-12 Example 3: vNIC Group 3 Configuration panel 5. The configuration is (Figure 5-13 on page 93): vNIC INT1. set to a maximum bandwidth of 3. vNIC 2) of the switch in I/O bay 9.3 Gbps (33% of 10 Gbps) One vNIC group containing these two vNICs (vNIC Group VLAN 925) One uplink trunk comprising external ports EXT1 and EXT3 The BladeOS command script in Example 5-5 can be used to implement this configuration.3 Example 3: vNICs only.IBM Virtual Fabric Mode 91 . Examples of use . which consists of vNIC ports and one uplink trunk.1 (blade in slot 1. one uplink trunk Example 3 creates group VGRP 5.

1 member 2./c/virt/vnic/port INT1/vnic 1 ena bw 30 /c/virt/vnic/port INT2/vnic 2 ena bw 33 /c/virt/vnic/vnicgrp 5 ena vnicvlan 925 addvnic INT1.2 trunk 1 exit 92 IBM BladeCenter Virtual Fabric Solutions .1 addvnic INT2. Example 5-6 isCLI script for Example 3 vnic enable portchannel 1 port EXT1 portchannel 1 port EXT3 portchannel 1 enable vnic port INT1 index 1 enable bandwidth 30 vnic port INT2 index 2 enable bandwidth 33 vnic vnicgroup 5 enable vlan 925 member 1.2 addtrnk 1 apply The isCLI command script in Example 5-6 can be used to implement this configuration.

2. Tip: Click Submit  Apply  Save after every change to ensure that the setting persists after any switch reboot. Examples of use . Click the Configure tab of the BBI interface. and then select On and click Submit  Apply  Save. Enable vNIC operation by expanding Virtualization  VNIC  General.The Virtual Fabric Switch BBI can also be used to implement this configuration as follows.IBM Virtual Fabric Mode 93 . 1. Figure 5-13 Example 5: Enable vNIC operations on the vNIC Configuration panel Chapter 5.

select Enabled. Figure 5-15 Example 5: vNIC INT1. For the first vNIC. Figure 5-14 Example 5: vNICs Configuration panel 4.3. and then type the bandwidth allocation (Figure 5-15). Enable and configure the vNICs by selecting Virtualization  VNICs (Figure 5-14).1 Configuration panel 94 IBM BladeCenter Virtual Fabric Solutions .

and then type the bandwidth allocation (Figure 5-16). select Enabled. Figure 5-16 Example 5: vNIC INT2. Examples of use . Figure 5-17 Example 5: Trunk Groups Configuration panel 7. and click Add (Figure 5-18 on page 96).5. Enable the trunk group by clicking Layer 2  Trunk Groups and Trunk Group 1 (Figure 5-17). For the second vNIC. select EXT1 and EXT3.2 Configuration panel 6.IBM Virtual Fabric Mode 95 . Chapter 5. Enable the trunk group.

Figure 5-19 Example 5: vNIC Groups Configuration panel 9. Click vNIC Group 5 and Figure 5-20 on page 97 displays: a. Enable and configure the vNIC group by clicking Virtualization  VNIC  VNIC Groups (Figure 5-19). Enable the group. 96 IBM BladeCenter Virtual Fabric Solutions .Figure 5-18 Example 5: Switch Trunk Group 1 Configuration panel 8.

e. Select the Trunk 1 uplink trunk. Examples of use . d. Figure 5-20 Example 5: vNIC Group 5 Configuration panel Chapter 5. c.IBM Virtual Fabric Mode 97 . Select the vNICs. Set the VLAN to 925. Click Add.b.

98 IBM BladeCenter Virtual Fabric Solutions .

Switch Independent Mode” on page 101 Chapter 7. Chapters in this part are: Chapter 6.Switch Independent Mode” on page 109 Chapter 8.Switch Independent Mode” on page 133 Chapter 9. “Solution architecture .Switch Independent Mode” on page 193 © Copyright IBM Corp. 99 . “Supported products . “Examples of use . 2011. “Configuring the solution .Part 2 Part 2 Switch Independent Mode The chapters in Part 2 cover the product details and implementation for the Switch Independent Mode offering. All rights reserved.

100 IBM BladeCenter Virtual Fabric Solutions .

In this mode. The adapter NICs enforce bandwidth limits for each virtual channel. the virtual channel separation on the shared 10-Gb link is maintained in both the Virtual Fabric adapter (VFA) and the switch. 101 . in Switch Independent Mode. Instead. © Copyright IBM Corp. but will instead rely upon link level flow control to throttle traffic when necessary.1Q VLAN tag to distinguish the individual vNIC channels on each link.6 Chapter 6. Solution architecture Switch Independent Mode In Switch Independent Mode. The full features of the BNT and Cisco switches are still available when operating in Switch Independent Mode. you will not require an outer VLAN tag to maintain channel separation. the mode uses the MAC address or the MAC address in combination with the IEEE 802. This places a requirement on you and your network administrator to coordinate the configuration of the VFAs and the chassis switches to match the existing VLAN structure in your network. the switches do not enforce bandwidth limits to the servers. 2011. All rights reserved. Figure 6-1 on page 102 shows an example of an implementation of Switch Independent Mode and the VLAN tagging that is employed. However.

The Broadcom 2-port 10Gb Virtual Fabric Adapter includes one MAC address per virtual channel.1. Topics in this chapter are: 6.2. “Solution overview” 6. These VLANs are then incorporated along with the MAC address or addresses on inbound frames to forward traffic to the correct interface. “Features” on page 105 102 IBM BladeCenter Virtual Fabric Solutions .Switch Independent Mode Overview Customer VLAN tags maintain Layer 2 separation within switch Emulex or Brdcm 10G VFA Customer VLAN tag 10G Link VLAN 10 VLAN 10 VLAN 20 Emulex or Brdcm 10G VFA 10G Link VLAN 20 VLAN 30 VLAN 30 Note: VLAN tag denotes customer VLAN (required) Figure 6-1 Switch Independent Mode overview The Emulex 10GbE Virtual Fabric Adapter II allows your network administrator to configure one VLAN ID per virtual channel interface. for a total of up to 64 MAC addresses per each physical port. and so on) for each virtual channel. You can assign up to 16 MAC addresses to each vNIC partition. It is possible to allocate separate bandwidth allocations to each vNIC partition. Up to 31 additional VLANs can be configured within the applications stack (VMware. The vNIC partition bandwidth allocation allows for fine tuning and division of available physical port bandwidth between the vNICs.

81Y3133 BNT Virtual Fabric 10Gb Switch Module Not supported Not supported Supported 4Q/2011a Supported 4Q/2011a Supported Cisco Nexus 4001I Switch Module Not supported Not supported Supported 4Q/2011a Supported 4Q/2011a Supported a. Figure 6-1 provides a general view of the vNIC solution architecture. Different colors represent different virtual port groups to isolate traffic flows for different services and applications. 90Y3566 Broadcom 2-port 10Gb Virtual Fabric Adapter. Chapter 6.Switch Independent Mode 103 . 49Y4275 Emulex 10GbE Virtual Fabric Adapter II. Table 6-1 Supported combinations using Switch Independent Mode Supported switches   Supported adapters Emulex 10GbE Virtual Fabric Adapter. Support for Switch Independent Mode will require a firmware upgrade. 90Y3550 Emulex 10GbE Virtual Fabric Adapter II Advanced. Solution architecture . which is expected in 4Q/2011.1 Solution overview Table 6-1 shows the possible configurations that support Switch Independent Mode. 49Y4235 Emulex 10GbE Virtual Fabric Adapter Advanced. and the thickness of each link represents the amount of assigned bandwidth for a particular traffic flow. Switch Independent Mode provides the capability to subdivide a single 10-Gb link into up to four virtual channels.6.

The switch VLANs should correspond to the VLANs that are defined throughout your network infrastructure. You should also create a corresponding VLAN within the chassis switch for each vNIC associated with a switch port.I/O Bay 1 I/O Bay 2 I/O Bay 4 Components and connectivity inside the IBM BladeCenter H chassis Cisco Virtual Fabric Switch (Bay 7) Blade Server 14 Blade Server 1 FC Expansion Card Virtual Fabric Adapter vNIC1 vNIC2 vNIC3 vNIC4 vNIC1 vNIC2 vNIC3 vNIC4 10Gb port VLAN1 I/O Bay 3 EXT1 EXT2 EXT3 EXT4 EXT5 EXT6 EXT1 EXT2 EXT3 EXT4 EXT5 EXT6 On-Board 1GbE CIOv VLAN2 VLAN3 10Gb port VLAN4 CFFh Cisco Virtual Fabric Switch (Bay 9) 10Gb port VLAN1 Legend Standard I/O bays connections vNIC virtual connections External physical trunks vSwitch virtual trunk connections Mid-Plane 10Gb port VLAN2 VLAN3 VLAN4 Figure 6-2 BladeCenter Virtual Fabric vNIC solution diagram with Cisco switch Tip: An uplink can support multiple VLANs (not shown in Figure 6-2). vNICs are configured in the operating system using the Emulex or Broadcom software in the pre-boot environment. The vNIC interfaces become extensions of these network-wide VLANs. The VFA configuration and setup is performed inside the network controller rather than the switch. a VFA port with four vNICs defined should correspond to four VLANs on the switch port. 104 IBM BladeCenter Virtual Fabric Solutions . For example. Each physical interface of a VFA can be divided into up to four vNICs.

the Broadcom 2-port 10Gb Virtual Fabric Adapter supports oversubscription. Figure 6-3 on page 106 shows how a configured Broadcom 2-port 10Gb Virtual Fabric Adapter appears to Windows Server 2008.You must take several considerations into account when planning to use the components in Switch Independent Mode: The Virtual Fabric adapter must operate in vNIC2 mode (Emulex) or in NIC Partition mode (Broadcom). Support by BladeCenter Open Fabric Manager is planned but was not available at the time of writing.5 Gb) The total bandwidth of the four vNICs per port cannot exceed 10 Gbps.Switch Independent Mode 105 .2 Features The Emulex 10GbE Virtual Fabric Adapter II and Broadcom 2-port 10Gb Virtual Fabric Adapter are both high-performance. The operating system configurations see eight unique NICs (2 ports x 4 vNICs). 6. Up to four vNICs can be configured on each physical NIC (pNIC) on the VFA (a total of eight vNICs on a two-port VFA). The bandwidth for each vNIC can be assigned in a range of 100 Mb to 10 Gb in 100-Mb increments (the default is 2. The sum of the bandwidth allocated for all four vNICs on the same pNIC cannot exceed 10 Gb. Solution architecture . Chapter 6. dual-port network adapters for 10 Gbps Ethernet (10 GbE) networks having the following general characteristics: Each server pNIC port is divided into up to four vNICs. Tip: All parameters for vNIC2 and NIC Partition are configured on the Virtual Fabric adapter. however. whereby the aggregate bandwidth can be as high as 4x10 Gb allowing any of the vNICs to consume as much as 10 Gb where there is no contention or less when there is (based on the weight assigned to that vNIC).

the sum of all four vNIC bandwidths cannot exceed 10 Gbps. For the Broadcom adapter. The total actual consumed vNIC bandwidths cannot exceed 10 Gbps. For the Emulex adapter. the sum of all four allocated vNIC bandwidths can exceed 10 Gbps because the adapter supports oversubscription.Figure 6-3 Showing NICs in network connections . The default bandwidth is 2. 106 IBM BladeCenter Virtual Fabric Solutions .5 Gbps. The user enables vNICs and allocates bandwidth: – The allowable vNIC bandwidth range is from 100 Mbps to 10 Gbps: • • • • Bandwidth increments are in 100 Mbps.Windows Server 2008 All vNIC parameters are configured from the VFA user interface.

Switch Independent Mode 107 . Chapter 6. No forwarding occurs between uplinks assigned to VLAN groups. Figure 6-4 shows an example of a possible configuration. Solution architecture . Optionally. An uplink (port or trunk) can belong to multiple vNIC groups.– Server-to-VFS bandwidth metering is on a per-vNIC basis. A server port (pNIC or vNIC) can be associated with one or more VLAN groups. Figure 6-4 Setting bandwidth The user associates vNICs to VLAN groups within the switch by configuring the switch independent from the VFA. – The failover mechanism for switch-independent mode is not virtual port-aware. switch uplinks can be assigned to VLAN groups: – VLAN groups within the switch serve to isolate virtual NIC traffic flowing on the same physical port: • • • • Existing VLANs within the client network are not impacted.

108 IBM BladeCenter Virtual Fabric Solutions . The totals for the stack as a whole are the same as for any single switch configured in stand-alone mode. with either of the vNIC modes at the same time. “Supported products . and other switch attributes are not aggregated among the switches in a stack. Switch-independent mode will allow you to enable vNIC mode on the Emulex 10GbE Virtual Fabric Adapter II and implement VMready in the switch for that server at the same time. The network views the stack as a single switch entity.Switch Independent Mode” on page 109. vNIC and iSCSI You can enable vNIC mode on the VFA and use iSCSI on that same adapter. the number of ports available in a stack equals the total number of remaining ports of all the switches that are part of the stack. The BNT VFS supports stacking. rather than just for physical ports. VLANs. trunk links. Once the stacking links have been established. The network switches can be configured and managed for virtual ports (vports).Key concurrency rules are: vNIC and VMready VMready is a unique offering that enables the switches within the network to be virtual machine-aware. and the stack can be accessed and managed as a whole using standard switch IP interfaces. The number of available IP interfaces. vNIC and switch stacking A switch stack is a group of BNT Virtual Fabric 10Gb Switch Modules that work together as a unified system and can be managed as a single entity. Further details are in Chapter 7. trunks.

All rights reserved. we describe the products used in Switch Independent Mode. Use the Emulex 10GbE Virtual Fabric Adapter II or II Advanced. Supported products . 109 . and Emulex 10GbE Virtual Fabric Adapter Advanced. 90Y3550 or 90Y3566. 10-Gb Ethernet switch modules – BNT Virtual Fabric 10Gb Switch Module © Copyright IBM Corp. switch modules. This mode consists of the supported combinations of the following adapters. which is expected in 4Q/2011 The Emulex 10GbE Virtual Fabric Adapter. are not supported in the Switch Independent Mode offering. and chassis types: Virtual Fabric adapters – Emulex 10GbE Virtual Fabric Adapter II – Emulex 10GbE Virtual Fabric Adapter II Advanced – Broadcom 2-port 10-Gb Virtual Fabric Adapter Notes: Support for Switch Independent Mode by the Emulex 10GbE Virtual Fabric Adapter II will require a firmware upgrade. 49Y4235. 2011. 49Y4275.Switch Independent Mode In this chapter.7 Chapter 7.

Support for Switch Independent Mode will require a firmware upgrade. The topics covered in this chapter are: 7. 49Y4235 Emulex 10GbE Virtual Fabric Adapter Advanced.1.1. These product features are described in the following sections. 90Y3566 Broadcom 2-port 10-Gb Virtual Fabric Adapter. In this section.2. 7. we discuss these two adapters.1 Virtual Fabric adapters The Switch Independent Mode offering supports the use of the Emulex 10GbE Virtual Fabric Adapter II and the Broadcom 2-port 10-Gb Virtual Fabric Adapter. to a maximum of 110 IBM BladeCenter Virtual Fabric Solutions . part number 90Y3550.1 Emulex 10GbE Virtual Fabric Adapter II The Emulex 10GbE Virtual Fabric Adapter II. 81Y3133 BNT Virtual Fabric 10Gb Switch Module Not supported Not supported Supported 4Q/2011a Supported 4Q/2011a Supported Cisco Nexus 4001I Switch Module Not supported Not supported Supported 4Q/2011a Supported 4Q/2011a Supported a. 90Y3550 Emulex 10GbE Virtual Fabric Adapter II Advanced. which is expected in 4Q/2011. 10-Gb Ethernet card that supports 1-Gbps or 10-Gbps traffic. 49Y4275 Emulex 10GbE Virtual Fabric Adapter II. The vNICs are configured to meet your mix of network connectivity and throughput demands in today's complex server application environments. “Virtual Fabric switch modules” on page 116 7. Table 7-1 Supported combinations using Switch Independent Mode Supported switches   Supported adapters Emulex 10GbE Virtual Fabric Adapter. is a dual-port. or up to eight virtual NIC (vNIC) devices. Each physical 10-Gbps port can be divided into four virtual ports. with bandwidth allocation in 100 Mbps increments. “Virtual Fabric adapters” on page 110 7.– Cisco Nexus 4001I Switch Module BladeCenter H or HT chassis Table 7-1 lists the specific combinations that are supported.

it requires that switch modules are installed in I/O bays 7 and 9. is a separate adapter based on the same hardware. Figure 7-1 Emulex 10GbE Virtual Fabric Adapter II (CFFh) Table 7-2 lists the Emulex Virtual Fabric Adapter (CFFh) ordering part numbers and feature codes. which enables the Emulex 10GbE Virtual Fabric Adapter II to be upgraded to support the FCoE and iSCSI capabilities of the Advanced adapter. Therefore. Also available is the Emulex Virtual Fabric Adapter Advanced Upgrade. part number 49Y4265. part number 90Y3566.Switch Independent Mode 111 . Supported products . Table 7-2 Ordering part numbers and feature codes Description Emulex Virtual Fabric Adapter II Emulex Virtual Fabric Adapter Advanced II \ Emulex Virtual Fabric Adapter Advanced Upgrade Part number 90Y3550 90Y3566 49Y4265 Feature code A1XG AIXH 2436 Chapter 7. The adapter is a CFFh expansion card. in addition to the features available on a standard card.10 Gbps per physical port. with additional support for Fibre Channel over Ethernet (FCoE) and Internet Small Computer System Interface (iSCSI) hardware initiator functionality. The Emulex 10GbE Virtual Fabric Adapter II is shown in Figure 7-1. The Emulex Virtual Fabric Adapter Advanced II. .

adaptive load balancing (ALB).1Qaz) – Priority-based Flow Control (PFC) (P802. 1/10-Gb Ethernet adapter: – Virtual port bandwidth allocation in 100-Mbps increments – Supports up to eight virtual ports Wake On LAN support Full-duplex (FDX) capability Bus-mastering support Direct memory access (DMA) support Preboot Execution Environment (PXE) support IPv4/IPv6 TCP.(CEE-) DCBX (P802.The Emulex 10GbE Virtual Fabric Adapter II has the following features: Connection to either 1-Gb or 10-Gb data center infrastructure (1-Gb and 10-Gb auto-negotiation) PCI Express 2. including adapter fault tolerance (AFT). The adapter supports both virtual fabric modes: IBM Virtual Fabric Mode and Switch Independent Mode.3ad Enhanced Ethernet (draft) – Enhanced Transmission Selection (ETS) (P802. You can configure the adapter to operate as two physical iSCSI ports. The adapter supports Serial over LAN functionality.1Qaz) The Emulex Virtual Fabric Adapter Advanced II provides FCoE and iSCSI hardware initiator support. CIN-DCBX and Converged Enhanced Ethernet. switch fault tolerance (SFT). and IEEE 802.0 x8 host interface IBM BladeCenter CFFh form factor. 112 IBM BladeCenter Virtual Fabric Solutions . which can be combined with a CIOv expansion card on the same blade server Operates either as an 8-port virtual NIC or as a 2-port. teaming support. UDP checksum offload – – – – Large send offload (LSO) Large receive offload Receive side scaling (RSS) IPv4 TCP Chimney Offload VLAN insertion and extraction Jumbo frames up to 9000 bytes Load balancing and failover support.1Qbb) – Data Center Bridging Capabilities eXchange (DCBX) protocol.

which is the same as vNIC mode on original Emulex VFA adapters. This mode only works in conjunction with BNT Virtual Fabric 10Gb Switch Module.Switch Independent Mode 113 . which does not rely on any vNIC settings being passed to the adapter from the switch module.Note: Support for Switch Independent Mode by the Emulex 10GbE Virtual Fabric Adapter II will require a firmware upgrade. which is expected in 4Q/2011. Supported products . We also call this mode switch-dependent mode. Switch Independent Mode (sometimes referred to as vNIC2). In this mode. because the Emulex adapters rely on the switch module to obtain vNIC parameters (via DCBX). The original Emulex 10GbE Virtual Fabric Adapter can be configured in pNIC or vNIC mode. the Emulex 10GbE Virtual Fabric Adapter II can work with these switches: – BNT Virtual Fabric 10Gb Switch Module – Cisco Nexus 4001I Switch Module Chapter 7. The new Emulex 10GbE Virtual Fabric Adapter II now offers two vNIC modes of operation: IBM Virtual Fabric Mode (sometimes referred to as vNIC1). This mode is used when configuring for the IBM Virtual Fabric Mode offering.

part number 81Y3133. 114 IBM BladeCenter Virtual Fabric Solutions . It is a high-speed expansion card. The adapter is shown in Figure 7-3 on page 115. is a dual-port 10Gbps high-speed expansion card designed to operate in Switch Independent Mode. Virtual Fabric Adapter Figure 7-2 Placement of a CFFh card in the BladeCenter HS22 server 7. so you need to install it in the CFFh expansion slot.2 Broadcom 2-port 10-Gb Virtual Fabric Adapter The Broadcom 2-port 10-Gb Virtual Fabric Adapter.Figure 7-2 shows the placement of Emulex 10GbE Virtual Fabric Adapter II in an HS22 blade server.1.

It is possible to dynamically allocate bandwidth to each vNIC partition. vNIC partitioning is based on MAC addresses.3x flow control Chapter 7. Broadcom 2-port 10-Gb Virtual Fabric Adapter features are as follows: Integrated dual 10-Gbps MAC and dual 10GBASE-CX4 PCI Express x8 2.1 remote boot Wake-on-LAN (WOL) Virtual LANs—IEEE 802. 5 GT/s compliant IEEE 802. Each enabled partition appears as a dedicated network card to the operating system. vNIC partition bandwidth allocation allows for fine-tuning and division of available physical port bandwidth between the vNICs.Figure 7-3 Broadcom 2-port 10-Gb Virtual Fabric Adapter You can configure up to four partitions (vNICs) on each of the two 10-Gbps physical ports.3xx Clause 37 compliant auto-negotiation for 1Gbps TCP/IP Offload Engine (TOE) Microsoft TCP chimney compliant Full FASTPATH TCP offload for IPv4 and IPv6 Network Controller Sideband Interface (NC-SI) PXE v2. Supported products . the Broadcom 2-port 10-Gb Virtual Fabric Adapter does not use VLAN IDs for vNIC partitioning. In contrast with the Emulex 10GbE Virtual Fabric Adapter II. for a total of up to 64 MAC addresses per each physical port. You can assign up to 16 MAC addresses to each vNIC partition.3ap Clause 73 compliant backplane operation IEEE 802.0.Switch Independent Mode 115 .1q VLAN tagging IEEE 802.

top-of-rack switch). In this publication. is able to transmit Converged Enhanced Ethernet (CEE) to an FCoE-capable. with BladeCenter Open Fabric Manager (BOFM). it does not depend on the switch module model used. this single 116 IBM BladeCenter Virtual Fabric Solutions . The switch module can be managed using a command-line interface (CLI) or a web browser interface and. Figure 7-2 on page 114 shows placement of the adapter in an HS22 blade. This is the first 10-Gb switch for IBM BladeCenter that supports converged networking (that is. This feature is available with firmware release 6. With the use of QLogic Virtual Fabric Extension Module. several operating at 1 Gbps and several at 10 Gbps.The Broadcom 2-port 10-Gb Virtual Fabric Adapter form factor is CFFh. CEE and FCoE functionality enables the clients to transfer storage. The BNT switch module can be used both in IBM Virtual Fabric Mode and Switch Independent Mode offerings. video. clients can achieve FCoE gateway functionality inside the BladeCenter chassis. This offering enables clients to form eight virtual network interface controllers (vNICs) from one physical NIC (pNIC) and to manage them in virtual groups. we concentrate on virtual fabric. and other data over the common Ethernet infrastructure. 7. In an IBM Virtual Fabric Mode solution. Currently. and several transmitting converged packets.1 BNT Virtual Fabric 10Gb Switch Module The BNT Virtual Fabric 10Gb Switch Module for IBM BladeCenter offers the most bandwidth of any blade switch and represents the perfect migration platform for clients who are still at 1 Gb outside the chassis by seamlessly integrating into the existing 1-Gb infrastructure.2. If you have a chassis with multiple servers. provide all the benefits of I/O virtualization at 10-Gb speeds. all vNIC parameter configuration is performed on the BNT Virtual Fabric 10Gb Switch Module. therefore it communicates with high-speed I/O modules installed in I/O bays 7 and 9.2 Virtual Fabric switch modules Because Switch Independent Mode is based on the vNIC configuration capabilities of the Virtual Fabric adapters. in the future. the switch module is a key part of the offering. network.1 and higher. Voice Over Internet Protocol (VoIP). two switches are supported: BNT Virtual Fabric 10Gb Switch Module Cisco Nexus 4001I Switch Module 7. Therefore.

or even connecting to an adjacent rack.Switch Independent Mode 117 . DAC cables are more cost-effective. With the extreme flexibility of the BNT switch. Supported products . They are ideal for connecting chassis together. or both. Table 7-3 Part number and feature code Description BNT Virtual Fabric 10Gb Switch Module Part number 46C7191 Feature code 1639 The part number includes the following items: One BNT Virtual Fabric 10Gb Switch Module Three-meter. For 10-Gbps uplinks. For 1-Gbps uplinks. or direct-attached copper (DAC) cables (also known as twinax active cables) for shorter distances. Figure 7-4 BNT Virtual Fabric 10Gb Switch Module Table 7-3 lists the part number to use to order the module. consume less power. they can take advantage of small form-factor pluggable (SFP) transceivers. you can take advantage of the technologies that they require for multiple environments. and can be up to 7 meters in length. connecting to a top-of-rack (TOR) switch. mini-USB-to-DB9 serial console cable One filler module BNT Virtual Fabric 10Gb Switch Module Installation Guide BNT user license agreement Important Notices document Chapter 7. Figure 7-4 on page 117 shows the switch module. they have a choice of either enhanced small form-factor pluggable (SFP+) transceivers (short-range or long-range) for longer distances.switch can handle all of these workloads and can connect to a 1-Gb infrastructure or a 10-Gb infrastructure.

118 IBM BladeCenter Virtual Fabric Solutions . DAC cables have SFP+ transceivers on both ends.5 m Molex DAC SFP+ cable 1 m Molex DAC SFP+ cable 3 m Molex DAC SFP+ cable 7 m Molex DAC SFP+ cable FC cables 3 m Intel Connects optical cable 10 m Intel Connects optical cable 30 m Intel Connects optical cable 46D0153 46D0156 46D0159 3852 3853 3854 59Y1932 59Y1936 59Y1940 59Y1944 3735 3736 3737 3738 81Y1618 81Y1622 3268 3269 44W4408 46C3447 4942 5053 Part number Feature code Table 7-5 on page 119 lists additional transceivers and direct-attach cable options that are available directly from BNT. and DAC cables. using from one to ten connections per switch. Table 7-4 lists the part number to use to order the SFP+ transceivers. FC cables. FC cables. you must have either SFP+ transceivers or SFP+ direct-attach copper (DAC) cables connected. To communicate outside of the chassis. Table 7-4 IBM part numbers for ordering SFP+ transceivers.Documentation CD-ROM Tip: Enhanced small form-factor pluggable plus (SFP+) transceivers are not included and must be purchased separately. You have the flexibility to expand your bandwidth as desired. and DAC cables Description 10Gb SFP+ IBM 10GBase-SR 10GbE 850 nm Fiber SFP+ transceiver BNT SFP+ transceiver 1Gb SFP+ BLADE 1000BASE-T (RJ45) SFP transceiver BLADE 1000BASE-SX SFP transceiver DAC cables 0.

auto-negotiating ports: 1 Gbps or 10 Gbps to the server blades – Two internal full-duplex 100-Mbps ports connected to the management module External ports – Up to ten 10-Gb SFP+ ports (also designed to support 1-Gb SFP. and this makes the switch module suitable for the most performance-intensive environments.7 M Part number BN-CKM-S-T BN-CKM-S-SX BN-CKM-SP-LR BN-SP-CBL-1M BN-SP-CBL-3M BN-SP-CBL-7M The BNT Virtual Fabric 10Gb Switch Module includes the following features and functions: Form-factor Single-height.1 M SFP+ DAC .Switch Independent Mode 119 .Table 7-5 BNT part numbers for ordering SFP-based transceivers and cables Description BLADE 1000Base-T SFP BLADE 1000Base-SX SFP BLADE 10GBase-LR SFP+ SFP+ DAC . high-speed switch module Internal ports – Fourteen internal. if required. Supported products . – One 10/100/1000-Mb copper RJ45 used for management or data – An RS-232 mini-USB connector for serial port that provides an additional means to install software and configure the switch module Scalability and performance – Autosensing 1-Gb and 10-Gb internal and external Ethernet ports for bandwidth optimization – Non-blocking architecture with wire-speed forwarding of traffic and full line rate performance of 480-Gbps full duplex – Media access control (MAC) address learning: automatic update. flexibility of mixing 1 Gb and 10 Gb) Oversubscription ratio (14 internal ports to 10 external ports) is very low. supports up to 32-Kb MAC addresses Chapter 7.3 M SFP+ DAC .

based on source and destination IP addresses.1X port-based authentication – Multiple user IDs and passwords 120 IBM BladeCenter Virtual Fabric Solutions . EtherChannel.1w Rapid STP (RSTP). delay-sensitive. up to 100 Gb of total bandwidth per switch.3ad) link aggregation.1Q VLAN tagging support on all ports – Protocol-based VLANs Security – VLAN-based.– Up to 128 IP interfaces per switch – Static. VLAN numbers ranging from 1 to 4095 (4095 is a dedicated VLAN used for the management module connection only) – 802.1D STP for providing Layer 2 redundancy with PVRST+ – IEEE 802.1s Multiple STP (MSTP) for topology optimization. up to 128 STP instances supported by single switch – IEEE 802. V2. MAC addresses. up to 18 trunk groups. and IP-based access control lists (ACLs) – 802. and up to eight ports per group – Support for jumbo frames (up to 12288 bytes) – Broadcast/multicast storm control – Internet Group Management Protocol (IGMP) snooping for limit flooding of IP multicast traffic (IGMP V1. providing rapid STP convergence for critical. and Link Aggregation Control Protocol (LACP) (IEEE 802. and V3) – Configurable traffic distribution schemes over trunk links. MAC-based. or both – Fast port forwarding and fast uplink convergence for rapid Spanning Tree Protocol (STP) convergence – Stacking support: you can stack up to eight BNT Virtual Fabric 10Gb Switch Modules Availability and redundancy – Virtual Router Redundancy Protocol (VRRP) for Layer 3 router redundancy – IEEE 802. traffic-like voice or video – Layer 2 trunk failover to support active and standby configurations of network adapter teaming on blades – Interchassis redundancy (Layer 2 and Layer 3) VLAN support – Up to 1024 VLANs supported per switch.

and V3) – HTTP/HTTPS Browser-Based Interface (BBI) – Industry standard CLI (isCLI) and BLADEOS or AlteonOS CLI – Telnet interface for CLI – SSH v1/v2 – Serial interface for CLI – Scriptable CLI – Firmware image update (Trivial File Transfer Protocol (TFTP) and File Transfer Protocol (FTP)) Chapter 7. Terminal Access Controller Access-Control System Plus (TACACS+). Open Shortest Path First protocol (OSPF) V1.1p. Lightweight Directory Access Protocol (LDAP) Quality of service (QoS) – Up to eight queues per port – Support for IEEE 802. BGP-4). V2. VLANs) traffic classification and processing – Traffic shaping and remarking based on defined policies – Eight Weighted Round Robin (WRR) priority queues per port for processing qualified traffic Layer 3 functions – IP forwarding – IP filtering with ACLs (up to 4096 ACLs supported) – VRRP for router redundancy – Support for up to 128 static routes – Routing protocol support (Router Information Protocol (RIP) V1.– User access control – Radius. V2. IP ToS/DSCP. and V3. including static routes and OSPFv3 (requires firmware V6. and ACL-based (MAC and IP source and destination addresses.Switch Independent Mode 121 . up to 1024 entries in routing table – IPv6 routing. Supported products .3 or higher) – Support for Dynamic Host Configuration Protocol (DHCP) Relay – IPv6 host management – IPv6 forwarding based on static routes Manageability – Simple Network Management Protocol (SNMP V1. RIP V2.

– FCoE Initialization Protocol (FIP) snooping to enforce point-to-point links for FCoE traffic outside the regular Fibre Channel topology. – Priority-Based Flow Control (PFC) (IEEE 802. and removed while retaining the same ACLs. as VMs migrate across physical hosts. With VMready. – Supports the QLogic Virtual Fabric Extension Module for IBM BladeCenter. rather than just for physical ports.3x standard flow control to allow the switch to pause traffic based on the 802. QoS.– Network Time Protocol (NTP) for switch clock synchronization – BNT BLADEHarmony Manager support Monitoring – – – – Switch LEDs for external port status and switch module status indication Port mirroring for analyzing network traffic passing through switch Change tracking and remote logging with syslog feature Power-On Self Test (POST) diagnostics Special functions – Serial over LAN (SOL) Virtualization features – VMready – Virtual Fabric Adapter vNIC support Converged Enhanced Ethernet and FCoE features – FCoE allows Fibre Channel traffic to be transported over Ethernet links. without 122 IBM BladeCenter Virtual Fabric Solutions .1p priority value in each packet’s VLAN tag. which provides FCoE gateway functionality inside the BladeCenter Chassis. – Enhanced Transmission Selection (ETS) (IEEE 802. and KVM. Xen. Virtual machines can be added. – Data Center Bridging Capability Exchange Protocol (DCBX) (IEEE 802.1Qaz) provides a method for allocating link bandwidth based on the 802. VMready allows for a define-once-use-many configuration that evolves as the server and network topologies evolve. VMready works with all virtualization products.1p priority value in each packet’s VLAN tag. Hyper-V.1Qbb) extends 802. and VLAN attributes.1AB) allows neighboring network devices to exchange information about their capabilities. moved. VMready is a unique solution that enables the network to be virtual machine-aware. The network can be configured and managed for virtual ports (v-ports). so do their network attributes. including VMware.

1s MSTP IEEE 802. 50 µ or 62.1 (and higher) software code.redbooks. multimode fiber.3ae 10GBASE-LR 10Gb Ethernet fiber optics long range IEEE 802.3ab 1000BASE-T Gigabit Ethernet IEEE 802.modification of virtual machine hypervisors or guest operating systems.1p Tagged Packets IEEE 802. with LC duplex connector 1000BASE-T for RJ45 port: – – – – UTP Category 6 (100 meters maximum) UTP Category 5e (100 meters maximum) UTP Category 5 (100 meters maximum) EIA/TIA-568B 100-ohm STP (100 meters maximum) More information is available in the following publications: BNT Virtual Fabric 10Gb Switch Module At-a-Glance Guide available at: http://www.1w RSTP IEEE 802.3ad Link Aggregation Control Protocol IEEE 802.1Q Tagged VLAN (frame tagging on all ports when VLANs are enabled) IEEE 802.2 Logical Link Control IEEE 802. It is available as part of the 6.html Chapter 7. Supported products .3x Full-duplex Flow Control IEEE 802.3z 1000BASE-SX Gigabit Ethernet The following network cables are supported for the BNT Virtual Fabric 10Gb Switch Module: 10GBASE-SR for 10-Gb ports: 850Nm wavelength.5 µ (300 meters maximum).com/abstracts/tips0708.3ae 10GBASE-SR 10Gb Ethernet fiber optics short range IEEE 802. VMready compatibility with Virtual Fabric solutions is as follows: VMready is not supported with IBM Virtual Fabric Mode VMready is supported with Switch Independent Mode The switch module supports the following IEEE standards: IEEE 802.Switch Independent Mode 123 .1x port-based authentication IEEE 802.1D STP with PVRST+ IEEE 802.ibm.

Fibre Channel SAN). Figure 7-5 Cisco Nexus 4001I Switch Module 124 IBM BladeCenter Virtual Fabric Solutions . The Cisco Nexus 4001I Switch Module enables a standards-based.wss?uid=psg1MIGR-5080917 – – – – – – – BNT Virtual Fabric 10Gb Switch Module Installation Guide Application Guide Command Reference Industry standard command line interface Reference (isCLI Reference) BBI (Browser-based Interface) Quick Guide Release Notes End User License Agreement 7. such as iSCSI. providing the server I/O solution required for high-performance. 10-Gb Ethernet switch module that is fully compliant with FCoE and IEEE Data Center Bridging standards. Figure 7-5 shows the switch module. Layer 2.ibm.2 Cisco Nexus 4001I Switch Module The Cisco Nexus 4001I Switch Module is a blade switch solution for the BladeCenter H and HT chassis.2. storage traffic (IP-based. and high-performance computing (HPC) traffic over a single 10-Gb Ethernet server network. and non-virtualized x86 computing architectures.com/support/docview. providing all of the benefits of I/O virtualization at 10Gbps speed. It is a line rate. extremely low-latency.Documents in PDF format available at: http://www. non-blocking. network-attached storage (NAS). virtualized. scale-out. high-performance Unified Fabric running over 10Gb Ethernet in the blade server environment. This Unified Fabric enables consolidation of LAN traffic. This offering works with BladeCenter Open Fabric Manager (BOFM).

Table 7-6 shows the part numbers to order these modules and additional options for them.
Table 7-6 Cisco Nexus 4001I Switch Module and related options part numbers and feature codes Description Cisco Nexus 4001I Switch Module for IBM BladeCenter Software upgrade license for the Cisco Nexus 4001I Switch Module Part number 46M6071 49Y9983 Feature code 0072 1744 Cisco part number N4K-4001I-XPX N4K-4001I-SSK9

The module part numbers include the following items: One Cisco Nexus 4001I Switch Module Cisco Console Cable RJ45-to-DB9 One filler panel Important Notices document Documentation CD-ROM

Software Upgrade License for the Cisco Nexus 4001I Switch Module
The Cisco Nexus 4001I Switch Module is designed to support both 10-Gb Ethernet and FCoE. The software upgrade license for the Cisco Nexus 4001I Switch Module, part number 49Y9983, enables the switch to work in FCoE mode. When connected to a converged adapter in the server, this switch can route CEE packets to an upstream FCoE switch, which can then route the packets to the LAN or SAN.

SFP+ transceivers and copper cables
The Cisco Nexus 4001I Switch Module does not ship with either SFP+ transceivers or SFP+ copper cables. These can be ordered from IBM, directly from Cisco Systems, or from authorized Cisco Systems resellers, as listed in Table 7-7.
Table 7-7 SFP+ transceivers and copper cables for the Cisco Nexus 4001I Switch Module Description IBM part number IBM feature code Cisco Systems part number

10Gb Ethernet SFP+ transceiver Cisco 10GBASE-SR SFP+ (MMF, 850-nm, LC) Cisco 10GBASE-LR SFP+ (SMF, 1310-nm, LC) 88Y6054 None A1A6 None SFP-10G-SR(=) SFP-10G-LR(=)

Chapter 7. Supported products - Switch Independent Mode

125

Description

IBM part number

IBM feature code

Cisco Systems part number

DAC Cables Cisco 1-m 10G SFP+ Twinax cable assembly, passive Cisco 3-m 10G SFP+ Twinax cable assembly, passive Cisco 5-m 10G SFP+ Twinax cable assembly, passive Gigabit Ethernet SFP+ Cisco 1000BASE-T SFP transceiver (Cat. 5 copper wire, RJ-45) Cisco 1000BASE-SX SFP transceiver (MMF, 850-nm, LC) Cisco 1000BASE-LX/LH SFP transceiver (MMF/SMF, 1300-nm, LC) 88Y6058 88Y6062 None A1A7 A1A8 None GLC-T(=) GLC-SX-MM(=) GLC-LH-SM(=) None None None None None None SFP-H10GB-CU1M(=) SFP-H10GB-CU3M(=) SFP-H10GB-CU5M(=)

The Cisco Nexus 4001I Switch Module includes the following features and functions: Form-factor Single-height high-speed switch module External ports – Six 10-Gb SFP+ ports operating at wire speed. Also designed to support 1Gb SFP if required, with the flexibility of mixing 1 Gb and 10 Gb. Table 7-7 on page 125 lists supported transceivers and cables. – One 10/100/1000-Mb Ethernet copper RJ45 used for management. – An RS-232 to RJ45 connector for a serial port that provides an additional means to configure the switch module. The console cable is supplied with the switch module. Internal ports – Fourteen internal auto-negotiating ports: 1 Gbps or 10 Gbps to the server blades – Two internal full-duplex 100-Mbps ports connected to the management modules Scalability and performance – Autosensing 1-Gb/10-Gb internal and external Ethernet ports for bandwidth optimization

126

IBM BladeCenter Virtual Fabric Solutions

– Non-blocking architecture with wire-speed forwarding of traffic and full line rate performance of 400-Gbps full duplex – Forwarding rate of 300 million packets per second (mpps). – Low, predictable, and consistent latency of 1.5 microseconds regardless of packet size, traffic pattern, or enabled features on 10GbE interface – MAC address learning: automatic update, supports up to 8 Kb MAC addresses – EtherChannels and LACP (IEEE 802.3ad) link aggregation, up to 60Gbps of total uplink bandwidth per switch, up to seven trunk groups, and up to six ports per group – Support for jumbo frames (up to 9216 bytes) – Traffic suppression (unicast, multicast, and broadcast) – IGMP snooping to limit flooding of IP multicast traffic (IGMP V2 and V3) – Configurable traffic distribution schemes over EtherChannel links based on source/destination IP addresses, MAC addresses, or ports – Spanning Tree edge ports (formerly PortFast) for rapid STP convergence Availability and redundancy – Link state tracking to mirror the state of the external ports on the internal Ethernet links and to allow the failover of the processor blade traffic to an operational external link on a separate Cisco Ethernet switch. – IEEE 802.1D STP for redundant backbone connections and loop-free networks. – IEEE 802.1s MSTP for grouping VLANs into a spanning-tree instance, and provided for multiple forwarding paths for data traffic and load balancing. – IEEE 802.1w Rapid STP (RSTP) for rapid convergence of the spanning tree by immediately transitioning root and designated ports to the converting state. – Configurable Unidirectional link detection (UDLD) for detecting and disabling unidirectional links. This feature prevents a larger network failure in the event that a unidirectional link is detected, thus reducing downtime in these situations. VLAN support – Up to 512 VLANs supported per switch; VLAN numbers ranging from 1 to 4000 – 802.1Q VLAN tagging support on all ports – Private VLANs

Chapter 7. Supported products - Switch Independent Mode

127

Security – VLAN-based, MAC-based, and IP-based access control lists (ACLs) – Role-based access control (RBAC) to restrict the authorization of the user to perform switch management functions – TACACS+, a proprietary feature for managing network security through a TACACS server – RADIUS for verifying the identity of, granting access to, and tracking activities of remote users Quality of service – Support for IEEE 802.1p CoS, IP ToS/DSCP, Protocol, IP Real Time Protocol, and ACL-based (MAC/IP source and destination addresses, VLANs) traffic classification and processing – Trust boundaries on incoming interfaces to automatically classify incoming packets into system classes based on packet CoS value – Traffic shaping, guaranteeing bandwidth, or prioritizing based on defined policies – Up to eight egress queues per interface (one queue per CoS) for processing qualified traffic (support for strict priority and weighted round-robin (WRR) CoS policies) Fibre Channel over Ethernet – Support for T11-compliant FCoE on all 10-Gigabit Ethernet interfaces. – FCoE Initialization Protocol (FIP) snooping to enable the switch to operate as a loss-less Ethernet bridge transparently forwarding FCoE packets. – 802.1Q VLAN tagging for FCoE frames. – Priority-based flow control (IEEE 802.1Qbb) simplifies management of multiple traffic flows over a single network link and creates lossless behavior for Ethernet by allowing class-of-service (CoS)-based flow control. – Enhanced Transmission Selection (IEEE 802.1Qaz) enables consistent management of QoS at the network level by providing consistent scheduling of different traffic types (IP, storage, and so on). – DCBX Protocol (IEEE 802.1AB) simplifies network deployment and reduces configuration errors by providing auto-negotiation of IEEE 802.1 DCB features between the NIC and the switch and between switches.

128

IBM BladeCenter Virtual Fabric Solutions

IPv6 support IPv6 host support (IPv6 unicast addressing, IPv6 traffic processing, IPv6 applications support: syslog server, RADIUS, TACACS+, NTP, telnet, and ssh). IPv6 traffic forwarding at Layer 3 is not supported. Manageability – Command line interface: You can configure switches using the CLI from an SSH V2 session, a Telnet session, or the console port. SSH provides a secure connection to the device. – XML Management Interface over secure shell (SSH): You can configure switches using the XML management interface, which is a programming interface based on the NETCONF protocol that complements the CLI functionality. For more information see the Cisco NX-OS XML Interfaces User Guide. – Cisco Data Center Manager support. – SNMP V1, 2, and 3 support. – Protected Mode feature to isolate switch management from the Advanced Management Module for increased security of the switch. – Cisco Discovery Protocol (CDP) Versions 1 and 2 to aid in troubleshooting and reporting the misconfiguration of ports connecting to other devices supporting CDP. Monitoring – – – – – – – Switch LEDs for external port status and switch module status indication RMON Change tracking and remote logging with syslog feature Online diagnostics Cisco Fabric Services Session Manager SPAN for local traffic monitoring

Special functions – Serial over LAN (SOL) The switch module supports the following IEEE standards: IEEE 802.1D: Spanning Tree Protocol IEEE 802.1p: CoS Prioritization IEEE 802.1Q: VLAN Tagging IEEE 802.1s: Multiple VLAN Instances of Spanning Tree Protocol IEEE 802.1w: Rapid Reconfiguration of Spanning Tree Protocol IEEE 802.3ab: 1000Base-T (10/100/1000 Ethernet over copper) IEEE 802.3ad: Link Aggregation Control Protocol (LACP)

Chapter 7. Supported products - Switch Independent Mode

129

5 62.5 Modal bandwidth (MHz/km) 160 200 Maximum cable length 722 feet (220 m) 902 feet (275 m) 130 IBM BladeCenter Virtual Fabric Solutions . Table 7-10 1000BASE-SX cabling specifications Wavelength 850 nm 850 nm Cable type MMF MMF Core size (microns) 62.3ae 10Gb Ethernet over fiber optics – 10GBASE-SR – 10GBASE-LR IEEE 802.IEEE 802. Table 7-8 10GBASE-SR cabling specifications Wavelength 850 nm 850 nm 850 nm 850 nm 850 nm Cable type MMF MMF MMF MMF MMF Core size (microns) 62.652 Modal bandwidth (MHz/km) Not applicable Maximum cable length 6. Table 7-9 10GBASE-LR cabling specifications Wavelength 1310 nm Cable type SMF Core size (microns) G.5 62.3ab 1000BASE-T over copper twisted pair Network cable requirements: 10GBASE-SR Table 7-8 lists the 10GBASE-SR cabling specifications.5 50 50 50 Modal bandwidth (MHz/km) 160 200 400 500 2000 Maximum cable length 85 feet (26 m) 108 feet (33 m) 217 feet (66 m) 269 feet (82 m) 984 feet (300 m) 10GBASE-LR Table 7-9 lists the 10GBASE-LR cabling specifications.3z Gigabit Ethernet over fiber optics – 1000BASE-SX – 1000BASE-LX IEEE 802.2 miles (10 km) 1000BASE-SX Table 7-10 lists the 1000BASE-SX cabling specifications.

Supported products .652 Modal bandwidth (MHz/km) 500 400 500 Not applicable Maximum cable length 1804 feet (550 m) 1804 feet (550 m) 1804 feet (550 m) 6.0 50.redbooks.com/support/entry/portal/docdisplay?lndocid=MIGR-5082494 In addition.ibm.com/abstracts/tips0754.html Chapter 7.2 miles (10 km) 1000BASE-T – – – – UTP Category 6 (100 meters maximum) UTP Category 5e (100 meters maximum) UTP Category 5 (100 meters maximum) EIA/TIA-568B 100-ohm STP (100 meters maximum) See the following documentation for more information: Configuration Guide for the Cisco Nexus 4001I Switch Module Hardware Installation Guide for the Cisco Nexus 4001I Switch Module Getting Started Guide for the Cisco Nexus 4001I Switch Module Command Reference Guide for the Cisco Nexus 4001I Switch Module These publications can be downloaded at: http://ibm. contains useful information about the Cisco Nexus 4001I Switch Module: http://www.0 G.Wavelength 850 nm 850 nm Cable type MMF MMF Core size (microns) 50 50 Modal bandwidth (MHz/km) 400 500 Maximum cable length 1640 feet (500 m) 1804 feet (550 m) 1000BASE-LX Table 7-11 lists the 1000BASE-LX cabling specifications.5 50.Switch Independent Mode 131 . IBM Redbooks publication Cisco Nexus 4001I Switch Module for IBM BladeCenter. Table 7-11 1000BASE-LX cabling specifications Wavelength 1310 nm 1310 nm 1310 nm 1310 nm Cable type MMF MMF MMF SMF Core size (microns) 62. TIPS0754.

132 IBM BladeCenter Virtual Fabric Solutions .

most of the configuration steps are performed on the BNT Virtual Fabric 10Gb Switch Module.3. 133 . Configuring the solution Switch Independent Mode In this chapter. Switch Independent Mode requires that you set vNIC parameters for the adapter using the blade server UEFI utility. 2011.2. In addition.1. All rights reserved.8 Chapter 8. “Creating virtual NICs” on page 160 © Copyright IBM Corp. “IBM Virtual Fabric Mode” on page 9. “Cisco Nexus 4001I Switch Module configuration” on page 156 8. The topics covered in this chapter are: 8.4. The switch module transfers configuration settings to the Emulex 10GbE Virtual Fabric Adapter using Data Center Bridging eXchange (DCBX). In contrast. as described in Part 1. When configuring for the IBM Virtual Fabric Mode solution. The Emulex 10GbE Virtual Fabric Adapter requires very little configuring. we describe how to configure components in Switch Independent Mode. only that virtual network interface card (vNIC) mode needs to be enabled using the blade server Unified Extensible Firmware Interface (UEFI) utility. “Configuring the BNT Virtual Fabric 10Gb Switch Module” on page 146 8. you need to configure the switch module to match the virtual local area network (VLAN) ID configuration on the vNICs and the extended network infrastructure. “Configuring Virtual Fabric adapters” on page 135 8.

8. “vNIC and VMready” on page 169 8.7. “High availability and vNICs” on page 169 8.5.8. “Configuring vNIC VLANs” on page 167 8. “Configuring the operating systems” on page 169 134 IBM BladeCenter Virtual Fabric Solutions .6.

Note: Support for Switch Independent Mode by the Emulex 10GbE Virtual Fabric Adapter II will require a firmware upgrade. The following vNIC parameters are due to be supported by BOFM: MAC address of virtual NIC VLAN ID Minimum bandwidth Maximum bandwidth Priority At the time of writing.2 Firmware upgrade of Emulex 10GbE Virtual Fabric Adapter II Upgrade the firmware on the Emulex 10GbE Virtual Fabric Adapter II to the latest level before starting this setup procedure.1 is planned to support configuration of vNIC parameters on VFAs.1. Configuring the solution .1 BladeCenter Open Fabric Manager (BOFM) Advanced Version 4.8. 8.1 Configuring Virtual Fabric adapters In this section. we discuss firmware upgrade and adapter configuration in UEFI for both the Emulex 10GbE Virtual Fabric Adapter II and the Broadcom 2-port 10-Gb Virtual Fabric Adapter. We show an example of upgrading the Emulex 10GbE Virtual Fabric Adapter II firmware with the Chapter 8. which is expected in 4Q/2011 You can upgrade firmware in a number of ways: Windows Server 2008.Switch Independent Mode 135 . 8. 2008 R2 and 2003 online update Windows PE offline update Linux online code update Linux offline code update VMware code update package Code update using a bootable DVD media UEFI code update These methods are well documented in the readme files.1 were not yet available. The next revision of this book will include detailed descriptions of vNIC parameter configuration using BOFM. details about BOFM Advanced V4.1 Using BOFM Advanced V4.1.

10. press F1. MAC=00-00-C9-B2-57-06. to perform the code update.351. Old=3.103. Running the command produces eight sections. Power on or restart the blade.1.10.ufi. and the UEFI utility main menu displays (Figure 8-2 on page 137).351.ufi. The package uses an executable file. 8.10. Update=Firmware. however. MAC=00-00-C9-B2-57-09. Status=Success Thu May 19 15:41:27 2011 HBA=OCe11100-NIC.200.Return Code=0 Figure 8-1 Windows online code update (elxflash executable) The output in Figure 8-1 is truncated for clarity.10.200.10. When prompted during POST.ufi.103.200. These parameters are explained in the readme file.0.103.200. elxflash.0.10. Each section represents one of the vNICs. MAC=00-00-C9-B2-57-02. Update=Firmware. Port Type=NIC.exe. New=4. Status=Success elxflash: All required updates succeeded . New=4. Figure 8-1 shows an example of updating the Emulex 10GbE Virtual Fabric Adapter II firmware update with the elxflash /update command. Port Type=NIC. Image=oc11-4.3 Configuring the Emulex 10GbE Virtual Fabric Adapter II in UEFI To configure the Emulex 10GbE Virtual Fabric Adapter II using the UEFI interface. Image=oc11-4.10.10.0. Update=Firmware.200.10.0. Image=oc11-4. do the following: 1.0.200. 2. Old=3. New=4. only three sections are shown in Figure 8-1. >elxflash /update Thu May 19 15:41:27 2011 HBA=OCe11100-NIC. Elxflash can have many parameters to set in order to carry out the code update in the desired way.351. Port Type=NIC. Old=3.Windows online package. 136 IBM BladeCenter Virtual Fabric Solutions . Status=Success (lines deleted for clarity) Thu May 19 15:41:28 2011 HBA=OCe11100-NIC.0.

then select Network on the subsequent panel. Configuring the solution . Select System Settings. Figure 8-3 Network configuration in the blade UEFI utility Chapter 8.main menu 3.Switch Independent Mode 137 . to display the Network panel (Figure 8-3).Figure 8-2 Blade UEFI utility .

Enable Multichannel To enable vNIC or vNIC2 mode. you can now select vNIC or vNIC2 mode: – vNIC enables IBM Virtual Fabric Mode. Select Multichannel Configuration to display the Function Configuration panel in which four functions represent four vNICs (Figure 8-5 on page 139). which are the two physical ports present on the adapter. 138 IBM BladeCenter Virtual Fabric Solutions . Next. we need to select vNIC2 mode. After enabling Multichannel. you must first enable Multichannel. We are configuring for the Switch Independent Mode solution. Therefore. – vNIC2 enables Switch Independent Mode. Unless Multichannel is enabled. and the Emulex NIC Selection panel displays (Figure 8-4) 5. Figure 8-4 Emulex vNIC settings 6. we look at the settings that relate to vNICs (Figure 8-4). it will be enabled on both 10-Gb physical ports. 4. the card will operate as two physical 10-Gbps Ethernet ports. Select one of the ports to configure.The Network window shows two Emulex 10GbE Virtual Fabric Adapter II ports. When you enable Multichannel.

LPVID is a required parameter. You can enter values between 0 and 100 in this field. The adapter uses LPVID values to isolate traffic within each individual vNIC. The value is the percentage of total bandwidth allocated to a particular vNIC. ensure that all of the VLAN IDs specified on the four vNIC functions are allowed on the blade-facing port on the Virtual Fabric Switch Module. Another important parameter is the vNIC bandwidth allocation. For network traffic to pass correctly. However. there is only one parameter: Logical Port VLAN ID (LPVID). This is the inner VLAN ID in double-tagged frames (the outer VLAN ID and bandwidth parameters come from the BNT Virtual Fabric 10Gb Switch Module). When in vNIC mode (used in IBM Virtual Fabric Mode solution). based on vNIC2 mode. Configuration of this parameter is optional.Switch Independent Mode 139 . Select a function to display the Multichannel Configuration panel where we will configure parameters for that vNIC (Figure 8-6).Figure 8-5 Function Configuration panel 7. In this mode. we are configuring for the Switch Independent Mode solution. Configuring the solution . as vNIC functionality does not depend on it. Chapter 8.

9. Review the vNIC2 settings on the second physical port as well.Figure 8-6 Set vNIC2 mode parameters 8. Exit to the main UEFI menu and save your settings (Figure 8-7 on page 141). to ensure that the configuration is correct. 140 IBM BladeCenter Virtual Fabric Solutions .

Chapter 8. Power on or restart the blade. We use the blade server UEFI interface to configure the vNIC parameters. “Broadcom 2-port 10-Gb Virtual Fabric Adapter” on page 161 for details. you can use the Broadcom VFA option ROM configuration utility. accessible by pressing Ctrl-S when prompted. 1. Alternatively. press F1. 8.4.Restart the blade server.Figure 8-7 Save settings 10. and the UEFI utility main menu displays (Figure 8-8 on page 142). When prompted during POST.4 Configuring the Broadcom 2-port 10-Gb Virtual Fabric Adapter in UEFI This section describes the steps to configure vNIC partitions on the physical ports of the Broadcom 2-port 10-Gb Virtual Fabric Adapter. Refer to 8. 2.1.Switch Independent Mode 141 .1. Configuring the solution .

Select System Settings. Figure 8-9 Network panel 142 IBM BladeCenter Virtual Fabric Solutions .main menu 3.Figure 8-8 Blade server UEFI utility . then select Network on the next panel to display the Network panel (Figure 8-9).

Two. Chapter 8.Switch Independent Mode 143 . Enable vNIC partitions by selecting Device Configuration Menu to display the Device Configuration panel Figure 8-11 on page 144). 4. 1Gb Ethernet ports belong to the onboard Broadcom network controller. Configuring the solution . Select the first 10-Gb network port to display the Broadcom Main Configuration window for that port (Figure 8-10). The two 10Gb network ports belong to the Broadcom 2-port 10-Gb Virtual Fabric Adapter.Four Broadcom NetXtreme II network ports are listed. Figure 8-10 Broadcom main configuration panel 5.

With the vNIC partitions enabled. Figure 8-12 New configuration option for vNIC partitioning 144 IBM BladeCenter Virtual Fabric Solutions .Figure 8-11 Enable vNIC partitions 6. press Esc to return to the previous menu. in which a new configuration option displays (NIC Partitioning Configuration Menu) (Figure 8-12).

Switch Independent Mode 145 . as needed. Configuring the solution . and an option to configure bandwidth allocations (Global Bandwidth Allocation Menu). The sum of all four NIC partitions must be either zero or 100. then the relative bandwidth weight mechanism is disabled. The panel displays four vNICs. Figure 8-13 NIC Partition Configuration 8.7. for each NIC partition: a. a partition will be able to use all 10 Gbps of bandwidth when no other partition is Chapter 8. b. You can set Maximum Bandwidth to 100 on all four partitions. In this case. Maximum bandwidth This parameter sets the maximum bandwidth (as percentage of total physical port bandwidth) that the NIC partition can transmit. 9. If it is zero. Relative bandwidth weight We can use this parameter to specify the guaranteed bandwidth in an overload situation. the NIC partition will be guaranteed to transmit 2 Gbps of bandwidth (20% of total bandwidth). configure two values. As shown in Figure 8-14 on page 146. Select NIC Partitioning Configuration Menu to display the NIC Partition Configuration panel (Figure 8-13). Select the Global Bandwidth Allocation Menu. if the parameter is set to 20. For example. and a round-robin bandwidth allocation mechanism is used instead.

Future plans include configuring the switch module using BOFM. or SNMP.communicating. 146 IBM BladeCenter Virtual Fabric Solutions .2 Configuring the BNT Virtual Fabric 10Gb Switch Module This section provides an overview of the tools available for configuring the BNT Virtual Fabric 10Gb Switch Module in Switch Independent Mode. a browser-based interface (BBI). Examples in this section are based on the BNT Virtual Fabric 10Gb Switch Module firmware version 6. which is the current version at the time of this writing.5.When you have enabled NIC partitions and set the bandwidth allocation parameters. Figure 8-14 Global Bandwidth Allocation Menu 10.3. This switch module can be configured using a command-line interface (CLI) (for example. 8. 11. exit to the main menu. BladeOS or the industry standard CLI (isCLI)). then this is the maximum bandwidth available to the NIC partition.Save your settings.0. If this parameter is set to a value lower than 100.

171.42. clients following best practice recommendations would have likely changed the password for added security. To use the BladeOS CLI on the switch module.Switch Independent Mode 147 . The actual examples of configuring VLANs are in 9. we need to configure VLANs on the BNT Virtual Fabric 10Gb Switch Module to match the existing VLAN structure in the network. Chapter 8.3.3. we only provide an overview of configuration tools.39 telnet 9. Type the password to log in. The BladeOS main menu displays (see Figure 8-15 on page 148). “Configuring the BNT Virtual Fabric 10-Gb Switch Module” on page 252. you can use the following communication methods: Telnet session Serial connection SSH connection BladeOS CLI consists of the main menu and hierarchically organized submenus. menu-based user interface that provides complete switch module administration. However.2.1 BladeOS CLI BladeOS CLI is a simple and intuitive. In this section. we use the telnet command to connect to the switch module with IP address 9. An example of logging in to the switch module with telnet follows: 1.42. which is the starting point to all of the switch module management and configuration tasks. The default password is admin.When configuring the Switch Independent Mode solution.171. 8. In our example.39 2. Configuring the solution .

always available] Firmware upgrade Here we show an example of the BNT Virtual Fabric 10Gb Switch Module firmware upgrade using the BladeOS CLI. This firmware level is also required if the Emulex 10GbE Virtual Fabric Adapter is also updated to the latest firmware.16.5.com/support/entry/portal/docdisplay?lndocid=MIGR-5086594 Late breaking news: BNT Virtual Fabric 10Gb Switch Module firmware 6.39 NOTICE mgmt: admin(admin) login from host Information Menu Statistics Menu Configuration Menu Operations Command Menu Boot Options Menu Maintenance Menu Show pending config changes [global command] Apply pending config changes [global command] Save updated config to FLASH [global command] Revert pending or applied changes [global command] Exit [global command. This firmware update package is available on the IBM support website: http://ibm.[Main Menu] May 25 14:01:06 9.44. It is suggested that you leave the other OS image at the existing firmware level. still refer to firmware 6.0 is the current level. We will upgrade the switch module to this version.3. At the time of writing.0.7.5. 148 IBM BladeCenter Virtual Fabric Solutions . firmware 6.171. upgrade the boot image and one of the two OS images.42.3. The instructions in this section.209 info stats cfg oper boot maint diff apply save revert exit >> Main# Figure 8-15 BladeOS CLI main menu 9. so that you can easily revert back to it. however. The firmware code consists of three parts: Boot image OS image 1 OS image 2 When upgrading firmware.2 is now available and is required for the Emulex 10GbE Virtual Fabric Adapter II. if needed.

img: This is the new OS image.Use a Trivial File Transfer Protocol (TFTP) or a File Transfer Protocol (FTP) server to transfer the boot and OS images to the BNT Virtual Fabric 10Gb Switch Module. You can use any TFTP or FTP server. The contents of the tftproot directory in AMM local storage are shown in Figure 8-16.img: This is the new boot image. GbESM-24-10G-6.5.3. Configuring the solution .5. Figure 8-16 Firmware files in the tftproot directory Chapter 8.3.0_Boot. we use the TFTP server available on the BladeCenter Advanced Management Module (AMM). In our example.0_OS. We put the following two files into the tftproot directory in AMM local storage: GbESM-24-10G-6.Switch Independent Mode 149 .

39 INFO mgmt: Firmware downloaded to image2 Figure 8-17 Upgrade of OS image 2 150 IBM BladeCenter Virtual Fabric Solutions ..3....42.This takes about 90 seconds. 1...171.......39 NOTICE mgmt: boot image changed Boot Options# May 25 14:57:46 9. software version 6.73.img" from FTP/TFTP server 9..img Figure 8-17 shows the command output........img'.73 GbESM-24-10G-6..171.. Do you want to change that to the new image2? [y/n] May 25 14:57:28 9...5... File appears valid Download in progress .......... Confirm download operation [y/n]: y Starting download.171.... we used the following command: /boot/gtimg 2 9.......39 INFO mgmt: image2 downloaded from host 9.0_OS.42.........73..5.... >> Main# /boot/gtimg 2 9......3.0_OS....73 GbESM-24-10G-6.... Transfer the new OS image into one of the image banks (we use image bank 2 in our example) The BladeOS CLI command syntax is as follows: /boot/gtimg X TADDR GbESM-24-10G-6. >> May 25 14:57:46 9.....5...42...After ensuring that TFTP server functionality is enabled in AMM... For example...img Enter username for FTP server or hit return for TFTP server: image2 currently contains Software Version 6.171...... ........... image2 now contains Software Version 6....img The X represents the OS image bank (1 or 2) and TADDR represents the TFTP server IP address... Please wait Write complete (5737743 bytes)......42.. 2000....2 that was downloaded at 6:11:38 Thu Jan 5.....0_OS......171.42. The sequence that follows must be carried out in the order written..... proceed with the firmware upgrade using BladeOS CLI.. Image download complete (5737743 bytes) Writing to flash...... now verifying FLASH..... New download will replace image2 with file "GbESM-24-10G-6.1......3. ...3 AMM CodeLevel 3 : 0605WMJ03000 BLADEOS Im2 11/17/2010 Switch is currently set to boot software image1.5..3.................3 y Next boot will use new software image2.. Verification of new image2 in FLASH successful.....0_OS..5......5.171.........42............ file 'GbESM-24-10G-6.3.....171...42..3_OS..5.....

.5.2 New download will replace boot kernel with file "GbESM-24-10G6.. Writing to flash....73 GbESM-24-10G-6. Configuring the solution ........3.......... Transfer the new boot image. 7687332 bytes) download complete. File appears valid Download in progress ......5.0_Boot...42..0_Boot...img Enter username for FTP server or hit return for TFTP server: boot kernel currently contains Software Version 6..0........ file 'GbESM-24-10G-6..... The BladeOS CLI command to do this is: /boot/gtimg boot TADDR GbESM-24-10G-6.......5.. ...39 INFO mgmt: boot kernel downloaded from host 9.....5.73...... the actual command is: /boot/gtimg boot 9. 2.3 Boot Options# May 25 15:13:17 9. software version 6..171..171..42.5.5...img Figure 8-18 shows an example of a successful boot image update.......... ... 7687332 bytes) download complete........ Writing to flash....3..img TADDR represents the IP address of the TFTP server..42.. >> Boot Options# /boot/gtimg boot 9.... Before the upgrade..Figure 8-17 shows that we have successfully updated OS image 2 to the new firmware version 6.73 GbESM-24-10G6...5...... Please wait Boot Sector now contains Software Version 6.3 Boot image (Boot.39 INFO mgmt: Firmware downloaded to boot kernel Figure 8-18 Upgrade of boot image Chapter 8..........3......1......3.. Please wait Kernel Sector now contains Software Version 6...This can take up to 90 seconds...171.. Boot image (FS.. Writing to flash..5.42...3 Boot image (Kernel.3. Confirm download operation [y/n]: y Starting download......5.73.. Please wait FS Sector now contains Software Version 6...0_Boot....... In our example.....img'.3 >>May 25 15:13:17 9......This can take up to 90 seconds..42...42....0_Boot.171............ 7687332 bytes) download complete........171....img" from FTP/TFTP server 9..Switch Independent Mode 151 ......5.0_Boot.......3.... We changed this to OS image bank 2 during the update process.This can take up to 90 seconds.171. the switch module was set to boot from OS image bank 1......

2. BNT Virtual Fabric 10Gb Switch Module. use the command: boot cli-mode bladeos-cli Restart the switch module after the command. BladeOS CLI is the default CLI mode.ibm. When you work in this mode. isCLI is a Cisco-like user interface that enables complete management and configuration support of BNT switch modules. system administrators who are more used to Cisco CLI have the option of using the industry-standard CLI (isCLI) instead. use the following command: boot/mode iscli Restart the switch module after running this command. as in: Switch# 152 IBM BladeCenter Virtual Fabric Solutions . the isCLI prompt ends with the > character. and is indicated by number (or pound) character # at the end of the isCLI prompt.com/support/docview.3. Reset the switch module. even when you restore the switch module to factory defaults. isCLI mode has three different privilege levels: User EXEC mode This is the initial mode with the most limited privileges.3. as in: Switch> Privileged EXEC mode This mode has a higher privilege level. To revert from isCLI to BladeOS CLI. available at: http://www. The BladeOS CLI command is: /boot/reset More details about BladeOS CLI are documented in Command Reference 6.wss?uid=psg1MIGR-5080917 8.2 isCLI BladeOS is a proprietary CLI environment developed by BNT for configuring and managing their switch modules. CLI mode is preserved across reboots. but you can easily switch between BladeOS and isCLI: To change from BladeOS to isCLI. However.

3.wss?uid=psg1MIGR-5080917 8. Router(config)# Figure 8-19 isCLI: Enabling EXEC privileges More details about isCLI are in isCLI Reference 6.Privileged EXEC mode is accessed from User EXEC mode with the command enable.2.com/support/docview. Global Configuration Mode This is the most privileged mode. When in this mode. BBI is an easy-to-use. you can make changes to a running configuration. and connect to the IP address of the switch module. End with Ctrl/Z. Use the following command to enter this mode from the Privileged EXEC mode: configure terminal An example of enabling Privileged EXEC and Global Configuration Mode is shown in Figure 8-19. available at: http://www. Select the following options: I/O Module Tasks  Configuration  Advanced Options  Start CLI/Web Session Chapter 8. Router>enable Enable privilege granted.ibm. one per line. Configuring the solution . administration and configuration tasks on the BNT Virtual Fabric 10Gb Switch Module. which is enabled by default (and can be disabled in CLI). BNT Virtual Fabric 10Gb Switch Module. Open a web browser window.Switch Independent Mode 153 . We show an example of logging in to the switch module and launching BBI in the following steps: 1. from which you can: a.3 Browser-based interface (BBI) You can use BBI to perform most management. web-based management interface. Alternatively. intuitive. you can access BBI from the AMM web interface. Router#configure terminal Enter configuration commands.

154 IBM BladeCenter Virtual Fabric Solutions . Ensure the Protocol field is set to Web (not Telnet).b. Log in to the switch module (Figure 8-21 on page 155). Figure 8-20 AMM web interface . then click Start Session. Default credentials are: – Username: admin – Password: admin Note: Clients following best practice recommendations will likely have changed the password from the default for added security.Start CLI/web session 2. as shown in Figure 8-20.

Switch Independent Mode 155 . After a successful login. the BBI Switch Dashboard displays. Chapter 8. Configuring the solution .Figure 8-21 Log in to the BNT Virtual Fabric 10Gb Switch Module 3. serving as the starting point for switch module management (Figure 8-22 on page 156).

com/support/docview.ibm. BNT Virtual Fabric 10Gb Switch Module.3.Figure 8-22 The BNT Virtual Fabric 10Gb Switch Module dashboard To find out more about BBI.wss?uid=psg1MIGR-5080917 8. available at: http://www.3 Cisco Nexus 4001I Switch Module configuration You can configure the Cisco Nexus 4001I Switch Module using NX-OS CLI commands. 156 IBM BladeCenter Virtual Fabric Solutions . see BBI Quick Guide 6.

4. Nexus 4000 Switch ibm-switch-2 login: USERID Password: Cisco Nexus Operating System (NX-OS) Software TAC support: http://www.Log in to the switch module via Telnet with the default login credentials (the same credentials as on AMM): – USERID – PASSW0RD (with digit 0) After logging in. In the following section. A copy of each such license is available at http://www.org/licenses/gpl-2.1 Firmware upgrade At the time of writing.1.0 or the GNU Lesser General Public License (LGPL) Version 2. In this section. we show the firmware upgrade procedure using CLI commands.php ibm-switch-2# Figure 8-23 Cisco NX-OS CLI When configuring a Switch Independent Mode solution. Cisco Systems. and the switch module can be configured. we need to configure VLANs on the Cisco Nexus 4001I Switch Module to match the existing VLAN structure in the network. The firmware update package is available on the IBM support website: http://www. Certain components of this software are licensed under the GNU General Public License (GPL) version 2.2.ibm.1F.0. The actual examples of configuring VLANs are in 9. The copyrights to certain works contained in this software are owned by other third parties and used and distributed under license.2.org/licenses/lgpl-2. “Cisco Nexus 4001I Switch Module configuration” on page 224.cisco. we transferred the new code directly from the TFTP server to an image bank on the Chapter 8.opensource. Configuring the solution .3.com/support/entry/portal/docdisplay?lndocid=MIGR-508654 When upgrading firmware on the BNT Virtual Fabric 10Gb Switch Module.E1. Inc. the panel shown in Figure 8-23 displays.opensource. we only provide an overview of configuration basics.1. All rights reserved. 8.php and http://www.1. the latest NX-OS version for the Cisco Nexus 4001I Switch Module is 4.Switch Independent Mode 157 .com/tac Copyright (c) 2002-2010.

As you can see.1f.1.2.42.4.bin bootflash: vrf chassis-management Enter username: cisco Password: ***** Transfer of file Completed Successfully ***** ibm-switch-2# copy ftp://9. In contrast.50/n4000-bk9-kickstart. verify that they appear in the bootflash: directory.E1.bin bootflash: vrf chassis-management Enter username: cisco Password: ***** Transfer of file Completed Successfully ***** Figure 8-24 Copying the kickstart and system image files to a directory 2.171.4.2.1f.switch module.bin bootflash: vrf chassis-management Figure 8-24 shows the output of these commands.E1. The IP address of our FTP server in our example is 9. You can upgrade the firmware by installing the new kickstart and system image files.1f.42.1. Steps for doing so follow.171. as shown in Figure 8-25 on page 159.4.1.1f.E1.4. We specify these as parameters in the following two copy commands: – copy ftp://9.bin: The actual NX-OS system image file.42.4.50/n4000-bk9. Copy the kickstart and system image files from the FTP or TFTP server to the Cisco Nexus 4001I Switch Module file system: – n4000-bk9-kickstart.E1. ibm-switch-2# copy ftp://9.1.2. – n4000-bk9.42.50/n4000-bk9.50.4. 1.2.2. 158 IBM BladeCenter Virtual Fabric Solutions .bin bootflash: vrf chassis-management – copy ftp://9.171. Cisco allows you to transfer the image from an FTP or TFTP server to a directory in the NX-OS file system. Before installing the files.E1.2.50/n4000-bk9-kickstart.171. The destination directory in the NX-OS file system is the bootflash: directory. we are prompted for the user name and password to access the FTP server.42.bin: The NX-OS kickstart image file.1f.1.171.E1.1f.1.

4.1f.bin n4000-bk9-kickstart.4.1d.tmp-system .bin system bootflash:n4000-bk9. Configuring the solution . To upgrade the NX-OS with the new files.1.E1.E1..4.bin This is shown in Figure 8-26 on page 160.ibm-switch-2# dir 49 Jul 39 Jul 20624384 Jul 74274426 Jul 49152 Jul 20624384 Jul 20699136 May 74274426 Jul 74281634 May 4096 Jul 4096 Jul 4096 Jul 03 03 03 03 03 03 18 03 18 03 03 03 03:58:50 03:58:55 03:58:50 03:58:55 03:58:49 03:54:12 13:49:36 03:54:49 13:50:21 03:56:30 03:56:31 03:56:31 2010 2010 2010 2010 2010 2010 2011 2010 2011 2010 2010 2010 .1f.tmp-kickstart .1.tmp-kickstart .Switch Independent Mode 159 .4.E1.2.bin n4000-bk9.bin vdc_2/ vdc_3/ vdc_4/ Usage for bootflash:// 333664256 bytes used 290123776 bytes free 623788032 bytes total Figure 8-25 Bootflash: directory 3.4.tmp-system lost+found/ n4000-bk9-kickstart.E1.1f. use the install all command.1.. for example: install all kickstart bootflash:n4000-bk9-kickstart.2.1. Chapter 8.2.2.2.E1.1d.1f.2. and specify the kickstart and system keywords and image files as parameters.bin n4000-bk9.1.E1.4.1.

switch will reboot in 10 seconds.E1. [####################] 100% -.2.ibm.1. please wait.2. More details about Cisco Nexus 4001I Switch Module CLI commands are in Command Reference Guide for Cisco Nexus 4001I Switch Module.com/support/entry/portal/docdisplay?lndocid=MIGR-5082494 8.bin Verifying image bootflash:/n4000-bk9-kickstart. available at: http://www.E1. Figure 8-26 Upgrading firmware for the Cisco Nexus 4001I Switch Module y The switch module automatically reboots when the upgrade is finished.1f. Warning: please do not remove or power off the module at this time.SUCCESS Module 1: Refreshing compact flash and upgrading bios/loader/bootrom.4.E1. 160 IBM BladeCenter Virtual Fabric Solutions .4. Do you want to continue with the installation (y/n)? [n] Install is in progress.2.1.1.bin for boot variable "system". [####################] 100% -.bin system bootflash:n4000-bk9.4.2.SUCCESS Performing configuration copy.4. [####################] 100% -.4 Creating virtual NICs vNICs using Switch Independent Mode are created and configured on the Virtual Fabric adapter directly.SUCCESS Finishing the upgrade.1f.1f.ibm-switch-2# install all kickstart bootflash:n4000-bk9-kickstart. [####################] 100% -.SUCCESS Verifying image bootflash:/n4000-bk9. [####################] 100% -.1f.E1. Setting boot variables.SUCCESS Lines deleted for clarity Switch will be reloaded for disruptive upgrade.1.bin for boot variable "kickstart". using either the UEFI interface (Emulex and Broadcom adapters) or a preboot environment (available with the Broadcom adapter only).

8.Switch Independent Mode 161 . available in the pre-boot environment (Ctrl+S during boot when prompted). Power on or restart the blade. The Main Menu window appears. Figure 8-27 Device List 3.1 Broadcom 2-port 10-Gb Virtual Fabric Adapter Configuration and support of the virtual NICs are carried out using a Broadcom application. When prompted. Chapter 8. Configuring the solution . To configure the Broadcom 2-port 10-Gb Virtual Fabric Adapter using the preboot utility. 2.4. or using UEFI. do the following: 1. Highlight the device of interest and press Enter. Here we describe the pre-boot method. Figure 8-27. press Ctrl+s to display the Device List. Figure 8-28 on page 162.

MAC address 10:18:00:96:CE:90 162 IBM BladeCenter Virtual Fabric Solutions . Highlight the NIC Partition field and press the right arrow to change the value to Enabled. with their MAC addresses: – PF#0.Figure 8-28 Device Main Menu 4. Figure 8-29. 5. Select NIC Partition Configuration and press Enter. The NIC Partition Configuration window appears. Figure 8-29 NIC Partition Configuration The four NIC partitions are displayed.

In this case. Figure 8-30 displays. then set bandwidth parameters for the other three NIC partitions (PF#2. You can set this to 100 on all four partitions. In our case we selected PF#0. PF#4 and PF#6) following the same steps. a partition will be able to use all 10 Gbps of bandwidth when no other partition is communicating. the NIC partition will be guaranteed to transmit 2 Gbps of bandwidth (20% of total bandwidth). For example. Chapter 8. Figure 8-30 Partition function menu You can configure bandwidth parameters on this panel: – Bandwidth Weight allows you to specify the guaranteed bandwidth in an overload situation. if the parameter is set to 20. Highlight one of the partitions you want to configure and press Enter.– PF#2.Switch Independent Mode 163 . press Esc to exit the menu. and a round-robin bandwidth allocation mechanism is used instead – Maximum Bandwidth sets the maximum bandwidth (as percentage of total physical port bandwidth) that the NIC partition can transmit. MAC address 10:18:00:96:CE:98 – PF#6. MAC address 10:18:00:96:CE:9C 6. 7. If it is zero. Configuring the solution . The sum of all four NIC partitions must be either zero or 100. MAC address 10:18:00:96:CE:94 – PF#4. If this parameter is set to a value lower than 100. then the relative bandwidth weight mechanism is disabled. Once you have set the bandwidth values for NIC partition PF#0. then this is the maximum bandwidth available to the NIC partition.

Tip: Ensure that the network device associated with a disabled vNIC is also disabled in the OS. 3. Select System Settings. Power on or restart the blade. 164 IBM BladeCenter Virtual Fabric Solutions . Figure 8-32 on page 165. A bandwidth value of 0 disables the associated vNIC interface on the hardware. The network configuration panel displays.8. do the following: 1. Press F1 when prompted during POST. To enable the Emulex 10GbE Virtual Fabric Adapter II using the server UEFI interface. The UEFI utility main menu displays.4. Figure 8-31. 2. the Emulex NIC Selection menu appears. The vNIC Configuration Management Utility allows you to define rate-limiting parameters for each vNIC. Figure 8-31 Network controllers 4.2 Emulex 10GbE Virtual Fabric Adapter II Emulex provides a UEFI interface for configuring virtual NICs. The default configuration PCIe function numbers will automatically be assigned. Highlight the Emulex adapter you wish to configure and press Enter. then select Network on the subsequent panel.

Therefore. we need to set the Switch Configuration field to vNIC2. Chapter 8. – Use vNIC2 mode for Switch Independent Mode. We are configuring for the Switch Independent Mode. Highlight Multichannel Configuration and press Enter to display the Function Configuration menu (Figure 8-33 on page 166). When you enable Multichannel. Next.Switch Independent Mode 165 . you can now select vNIC or vNIC2 mode under Switch Configuration: – Use vNIC mode in IBM Virtual Fabric Mode. Configuring the solution . the card will operate as two physical 10-Gbps Ethernet ports. To enable vNIC or vNIC2 mode. you must first set Multichannel to Enabled. update the Emulex firmware. 7.Figure 8-32 Emulex NIC Selection 5. Unless Multichannel is enabled. After enabling Multichannel. it will be enabled on both 10-Gb physical ports 6. Tip: If you do not see the vNIC2 option. we look at the settings that relate to vNICs.

Select Function 0 and press Enter to configure parameters for the first vNIC. Functions 0 to 3 represent four vNICs.Figure 8-33 Function selection 8. Figure 8-34 appears. Figure 8-34 Multichannel Configuration 166 IBM BladeCenter Virtual Fabric Solutions .

1Q VLAN tag to distinguish the individual vNIC channels on each link. 8. Two key fields here are: – Logical Port VLAN ID (the LPVID field) is a required parameter when configuring for Switch Independent Mode.1q VLAN tags. Chapter 8. exit to the main UEFI menu and save the settings. the Bandwidth field. ensure that all of the VLAN IDs specified on the four vNIC functions are allowed on the blade-facing port on the Virtual Fabric Switch Module. The value is the percentage of total bandwidth allocated to a particular vNIC 10.5 Configuring vNIC VLANs Switch Independent Mode simply extends the existing customer VLANs to the NIC interfaces. The adapter uses LPVID values to isolate traffic within each individual vNIC. they utilize the MAC address in combination with the IEEE 802. Configuring the solution . Figure 8-34 on page 166 shows the vNIC parameters configuration panel.9. – Another important parameter is the vNIC bandwidth allocation. and then configure vNICs on the second physical port as well. The VLAN configuration on upstream switches needs to match the VLAN configuration on the blades (Figure 8-35 on page 168). For network traffic to pass correctly. You can enter values between 0 and 100 in this field.You need to set the LPVID and bandwidth values for the remaining three vNICs. Instead. Conventional VLANs are configured by an operating system on a server blade (such as with the vconfig command in Linux) are implemented as IEEE 802. When done. Switch Independent Mode vNICs do not require an outer tag to maintain channel separation.Switch Independent Mode 167 . This requires that you coordinate the configuration of the VFAs and the chassis switches to match the existing VLAN structure in the network.

A single port will be mapped to multiple VLANs. a VLAN trunk). The switch configuration steps include: 1. you will create distinct VLAN port groups using conventional port VLAN schemes. . Assign corresponding external ports to their respective VLAN. 168 IBM BladeCenter Virtual Fabric Solutions . Presuming that multiple internal switch ports share common VLANs by default. with the ports configured as VLAN trunks. 10 Gb port VLAN20 vNIC3 VLAN 30 VLAN40 Operating System vNIC2 VLAN 20 10 Gb port Virtual Fabric Switch (Bay 9) VLAN10 EXT1 EXT2 EXT3 EXT4 EXT5 VLAN30 EXT6 vNIC1 VLAN 10 . . GUI. Most ports will thus be mapped to multiple VLANs.Virtual Fabric Switch (Bay 7) 10 Gb port VLAN10 EXT1 EXT2 EXT3 EXT4 EXT5 VLAN30 EXT6 Blade 1 vNIC4 VLAN 40 . 10 Gb port VLAN20 VLAN40 Figure 8-35 Switch Independent Mode diagram You will also configure the switch ports using switch configuration methods (for example. CLI. multiple VLANs can be mapped to a single port or a port group. Only the BNT Virtual Fabric 10Gb Switch Module is capable of L3 forwarding between separate VLANs within the chassis. The VLANs can then be mapped to physically separate external ports. or SNMP). . . 2. or two or more VLANs can share an uplink port (for example. Optionally. The switch normally provides basic Layer 2 forwarding among ports that share a VLAN. Assign internal ports to each VLAN based on vNIC-to-VLAN mapping. A VLAN can be mapped to a single port or a port group.

Versions of BNT Virtual Fabric 10Gb Switch Module firmware prior to 6.8 Configuring the operating systems In this section. Access Control Lists (ACLs). and when virtual machines move as a result of a VMotion event. However.8. This ensures that appropriate network resources are allocated to the VMs. In Windows. this displays as a network cable unplugged condition on the server. Chapter 8. 8. 8. it is possible. thus impacting those vNIC channels that are in other VLAN groups. Configuring the solution .7.7. Quality of Service (QoS) and VLAN settings in VMready are set at the Group level.2. with Switch Independent Mode. if the uplink ports that are members of a VLAN group fail due to a failure on the upstream device. you do not need to use vNIC groups and are thus free to implement VMready groups. which enables high-availability designs when used with standard ports. this is now possible. the server administrator can assign network policies at a Group level. the associated internal port is brought down by the switch. if you are running the adapters in Switch Independent Mode. we cover how to configure drivers and perform other operating system-related activities. to use vNIC to support certain blades and VMready on other blades using the same switch and within the same chassis.7.2. However. the ACLs. however. QoS. with the release of 6. With versions prior to 6. Using VMready. is also available when either the Emulex or Broadcom adapters in vNIC mode are used with either a BNT or Cisco switch. just not the same port. a cable failure.6 vNIC and VMready You can use VMready only with the BNT Virtual Fabric 10Gb Switch Module. VMready allows for much greater control over the allocation of network resources.7 High availability and vNICs The trunk failover function. no matter where they reside on the network.2 did not support a blade-facing port being in a vNIC group and a VMready group at the same time. and VLAN assignments move with them. or a cable being erroneously removed.Switch Independent Mode 169 . However. as opposed to a physical server level.

Most configuration activities are performed by utilities provided by the supported operating systems. For the Broadcom 2-port 10-Gb Virtual Fabric Adapter, the centralized management application is the Broadcom Advanced Control Suite (BACS). The BACS application enables centralized management of Broadcom Virtual NICs throughout the data center from a centralized management console. The BACS application provides a graphical user interface (GUI) and a scriptable command line interface (CLI). You can use BACS to configure load balancing and teaming. Similarly, the Emulex OneCommand Manager utility is available to configure Emulex 10GbE Virtual Fabric Adapter II.

8.8.1 vNICs on the Emulex 10GbE Virtual Fabric Adapter II
Figure 8-36 shows the Network Connections menu. As we expect, there are eight vNICs listed, four on each 10-Gb physical port.

Figure 8-36 Network Connections

For clarity, we changed the names of the NICs from the default values to those listed in Table 8-1 on page 171. You can use the MAC addresses to match each vNIC with the virtual fabric adapter (VFA) displayed on the Network Connections panel.

170

IBM BladeCenter Virtual Fabric Solutions

Table 8-1 NIC names and matching vNICs NIC name Emulex VFA II #01 Emulex VFA II #02 Emulex VFA II #03 Emulex VFA II #04 Emulex VFA II #11 Emulex VFA II #12 Emulex VFA II #13 Emulex VFA II #14 10Gb physical port 0 0 0 0 1 1 1 1 vNIC 1 2 3 4 1 2 3 4 MAC address 00:00:C9:B2:57:02 00:00:C9:B2:57:03 00:00:C9:B2:57:04 00:00:C9:B2:57:05 00:00:C9:B2:57:06 00:00:C9:B2:57:07 00:00:C9:B2:57:08 00:00:C9:B2:57:09

Right-click a NIC and select Status to verify parameters, such as the MAC address and link speed. Figure 8-37 shows the status panel for the Emulex VFA II #01.

Figure 8-37 Emulex VFA II #01 status

Chapter 8. Configuring the solution - Switch Independent Mode

171

Tip: Speed is reported as 4.0 Gbps. This means the bandwidth was set to 40 when configuring this vNIC in Unified Extensible Firmware Interface (UEFI) utility. To see the MAC address of this vNIC, click Details. Figure 8-38 shows the Network Connection Details panel, which displays the MAC address in the Physical Address field.

Figure 8-38 Network Connection Details panel

This MAC address indeed represents vNIC 1 on 10-Gb physical port 0. We can identify and verify MAC addresses and link speeds of the other vNICs in the same manner.

8.8.2 vNICs on the Broadcom 2-port 10-Gb Virtual Fabric Adapter
We now verify how the NIC partitions on the Broadcom 2-port 10-Gb Virtual Fabric Adapter are seen by the operating system (Windows Server 2008). We first show Windows built-in networking tools, and then the Broadcom Advanced Control Suite (BACS).

172

IBM BladeCenter Virtual Fabric Solutions

Windows networking tools
Figure 8-39 shows how the NIC partitions appear in the Network and Sharing Center.

Figure 8-39 Broadcom 2-port 10-Gb Virtual Fabric Adapter NIC partitions, viewed using Windows Server 2008

For clarity, we have changed the NIC names from the defaults according to Table 8-2. We used MAC addresses to match each NIC with the actual NIC partition.
Table 8-2 NIC names and matching NIC partitions NIC name Broadcom VFA 01 Broadcom VFA 02 Broadcom VFA 03 Broadcom VFA 04 10Gb physical port 0 0 0 0 NIC partition 1 2 3 4 MAC address

10:18:00:96:E3:80 10:18:00:96:E3:84 10:18:00:96:E3:88 10:18:00:96:E3:8C

Chapter 8. Configuring the solution - Switch Independent Mode

173

NIC name Broadcom VFA 11 Broadcom VFA 12 Broadcom VFA 13 Broadcom VFA 14

10Gb physical port 1 1 1 1

NIC partition 1 2 3 4

MAC address

10:18:00:96:E3:82 10:18:00:96:E3:86 10:18:00:96:E3:8A 10:18:00:96:E3:8E

By clicking View Status (highlighted in Table 8-2 on page 173), we can verify parameters for each of the NIC partitions, such as link speed and MAC address. Figure 8-40 shows this for NIC Broadcom VFA 01.

Figure 8-40 Broadcom VFA 01 status

As you can see, speed is reported as 4.0 Gbps. This indicates the bandwidth was set to 40 on this NIC partition when configuring parameters in UEFI utility. Click Details to see the MAC address of this NIC partition. Figure 8-41 on page 175 shows the Network Connection Details panel in which the MAC address displays.

174

IBM BladeCenter Virtual Fabric Solutions

Figure 8-41 Network Connection Details panel showing the MAC address

This is indeed the MAC address of NIC partition 1 on 10-Gb physical port 0. Use the same method to verify MAC addresses and speed for the other NIC partitions on the Broadcom 2-port 10-Gb Virtual Fabric Adapter.

Broadcom Advanced Control Suite (BACS)
BACS is a Windows-based management utility that allows you to administer, configure, and diagnose Broadcom Virtual Fabric Adapters and other Broadcom NICs. Figure 8-42 on page 176 shows an example from BACS. You can see eight NIC partitions in the left pane. NIC partition 1 on physical port 0 is highlighted. Note the speed: 4 Gbps (this indicates the bandwidth parameter of 40 in the UEFI utility).

Chapter 8. Configuring the solution - Switch Independent Mode

175

Figure 8-42 BACS - NIC partition 1

Using the BACS utility, test the configuration by running a basic connectivity test. To do so, click the Diagnostics tab, then ping to either an IP gateway or a user-specified IP address. See Figure 8-43 on page 177.

176

IBM BladeCenter Virtual Fabric Solutions

Figure 8-43 BACS - Diagnostics tab

The Statistics tab displays network traffic counters. These counters are useful when monitoring or troubleshooting NIC behavior. See Figure 8-44 on page 178.

Chapter 8. Configuring the solution - Switch Independent Mode

177

We cover BACS NIC teaming in 8. click Team Management in the lower left corner (see Figure 8-45 on page 179). 178 IBM BladeCenter Virtual Fabric Solutions .Statistics tab One of the functions of the BACS utility is to configure NIC teaming.Figure 8-44 BACS .4. You can use NIC partitions in teaming configurations the same way you would use physical NICs.8. “Configuring teams with Broadcom BACS” on page 184. To set up NIC teaming.

which runs in Windows and Linux.Switch Independent Mode 179 .8.Team Management 8. we can use the Emulex OneCommand Manager utility.3 Configuring teams with Emulex OneCommand Manager To configure NIC teaming with the Emulex 10GbE Virtual Fabric Adapter II.com/products/management-software/device-management/on ecommand-manager/ Figure 8-46 on page 180 shows the OneCommand Manager main window.emulex. See the OneCommand Manager home page for details: http://www. It also supports VMware ESX/ESXi and Microsoft Hyper-V hypervisors. Configuring the solution . Chapter 8.Figure 8-45 BACS .

Click Create team to begin the process.Figure 8-46 Emulex OneCommand Manager utility In our example. This launches the Create team window. 180 IBM BladeCenter Virtual Fabric Solutions . 1. shown in Figure 8-47 on page 181. we will create a team with two members: Emulex VFA II #01 and Emulex VFA II #02.

so select FailOver from the Team Type field in Figure 8-47.3ad static – LACP 3. 4. Configuring the solution . Other team type options are: – LoadBalancing – 802. Failover. such as team name and type.Create team window 2. and select team members. Select each adapter you wish to form into the team and click Add. When done with setting parameters and selecting team members. click OK to continue. In this window. you can specify team parameters. Chapter 8. while the secondary member will have a standby role.Figure 8-47 Emulex OneCommand Manager . We use the simplest form of teaming.Switch Independent Mode 181 . Figure 8-48 on page 182 shows the newly created team. This means the primary member will carry all the traffic.

shown in Figure 8-49 on page 183. By highlighting the new team and clicking VLAN. 182 IBM BladeCenter Virtual Fabric Solutions .new team created 5. we can configure VLAN ID for the team. This opens up the Add/Remove VLAN window.Figure 8-48 Emulex OneCommand Manager .

Figure 8-49 Emulex OneCommand Manager . the VLAN is configured (see Figure 8-50 on page 184).Adding VLAN 6. Chapter 8. Configuring the solution .Switch Independent Mode 183 . After entering the VLAN ID and clicking Add.

8.Figure 8-50 Emulex OneCommand Manager . BACS allows you to team up to eight network adapters together in a single group.4 Configuring teams with Broadcom BACS A team (or bond in Linux terminology) of adapters functions as a single virtual network interface and appears the same as a nonteamed adapter to other network devices. The benefits of teaming are as follows: Virtual adapters can balance a large network load across several adapters.VLAN configured 7. 184 IBM BladeCenter Virtual Fabric Solutions .8. Click OK to return to the main window. The use of teams improves system availability by providing a failover path for critical network connections. This group becomes a virtual network adapter with its own IP address.

Configuring the solution .Switch Independent Mode 185 . Switch-independent – SLB (Smart Load Balancing) – SLB with Auto Failover Disable Switch-independent NIC teaming is not related to Switch Independent Mode. to act as a single switch. Chapter 8. The name indicates that switch-independent NIC teaming does not require any LACP/Etherchannel configuration on an Ethernet switch module. We show how to configure a new team in Windows using BACS application: 1. Launch the BACS application to open up main the BACS window (shown in Figure 8-51 on page 186). Modules also need to be stacked.3ad – GEC/FEC (Gigabit Etherchannel/Fast Etherchannel) Switch-dependent NIC teaming is not related to IBM Virtual Fabric Mode.Types of teams supported by BACS are: Switch-dependent – Link Aggregation Control Protocol (LACP) or IEEE 802. This teaming type requires that LACP/Etherchannel is configured on an Ethernet switch module.

186 IBM BladeCenter Virtual Fabric Solutions . Click the Team Management frame (lower left corner) to start the Broadcom Teaming Wizard.Figure 8-51 BACS main window 2. The Welcome window displays. shown in Figure 8-52 on page 187.

You now need to enter the name for the new team (see Figure 8-53 on page 188). Chapter 8. Configuring the solution .Welcome window 3. Click Next on the Teaming Wizard Welcome window.Figure 8-52 Using the teaming wizard .Switch Independent Mode 187 .

After entering the name for the new team. 188 IBM BladeCenter Virtual Fabric Solutions .Figure 8-53 Using the teaming wizard (Creating/Modifying a Team: Team Name window) 4. click Next to display the Team Type window (Figure 8-54 on page 189).

At the prompt. Select the type of team you wish to create and click Next. Figure 8-55. Figure 8-55 Using the teaming wizard (Verification window) 7. Configuring the solution . You can now assign team members (Figure 8-56 on page 190). 6. Click OK in the verification window to proceed. Chapter 8.Figure 8-54 Using the teaming wizard (Creating/Modifying a Team: Team Type window) 5. verify and confirm that the switch is configured correctly (this is only applicable to teaming modes that require LACP/Etherchannel configuration on the switch modules).Switch Independent Mode 189 .

Figure 8-57. then add the remaining team members in the same manner.Figure 8-56 Using the teaming wizard (Creating/Modifying a Team: Assigning Team Members window) 8. Figure 8-57 Using the teaming wizard (Creating/Modifying a Team: Assigning Team Members window) 190 IBM BladeCenter Virtual Fabric Solutions . Highlight the adapter you wish to add to the team and click Add.

With all team members added. select the Save changes and continue to manage more teams radio button.The Team Management window confirms the newly added team (Figure 8-59 on page 192).Optionally. select the Commit changes to system and Exit the wizard radio button. and the Commit Changes window displays (Figure 8-58). Chapter 8. 12. Configuring the solution . Once all team members are added. 11. and click Next. if needed. Figure 8-58 Using the teaming wizard (Commit changes window) 10. then click Finish.Switch Independent Mode 191 .9. click Next.

Figure 8-59 Teaming established This completes configuration of NIC teaming with the BACS application. 192 IBM BladeCenter Virtual Fabric Solutions .

Switch Independent Mode In this chapter. Examples of use . © Copyright IBM Corp. 193 . BNT Virtual Fabric 10-Gb Switch Module with – Emulex 10GbE Virtual Fabric Adapter II – Broadcom 2-port 10-Gb Virtual Fabric Adapter These are covered in 9. using the following combinations: Cisco Nexus 4001I Switch Module with: – Emulex 10GbE Virtual Fabric Adapter II – Broadcom 2-port 10-Gb Virtual Fabric Adapter These are covered in 9.3. “BNT Virtual Fabric 10-Gb Switch Module configurations” on page 226. “Cisco Nexus 4001I Switch Module configurations” on page 197.9 Chapter 9. we show examples of Switch Independent Mode configurations. 2011.2. All rights reserved.

shown in Figure 9-2 on page 195: Emulex 10GbE Virtual Fabric Adapter II and BNT Virtual Fabric 10-Gb Switch Module 194 IBM BladeCenter Virtual Fabric Solutions .1 Configurations overview We use the following equipment to demonstrate the four configurations: Two BladeCenter H chassis (BC4 and BC5) Two Cisco Nexus 4001I Switch Modules in chassis BC4 Two BNT Virtual Fabric 10-Gb Switch Modules in chassis BC5 Two HS22 blades in each chassis Two Emulex 10GbE Virtual Fabric Adapter II cards Two Broadcom 2-port 10-Gb Virtual Fabric Adapter cards Chassis BC4 is used to demonstrate the following configurations. shown in Figure 9-1: Emulex 10GbE Virtual Fabric Adapter II and Cisco Nexus 4001I Switch Module Broadcom 2-port 10-Gb Virtual Fabric Adapter and Cisco Nexus 4001I Switch Module Chassis midplane HS22 blade 1 Emulex VFA II Cisco Nexus 4001I (switch bay 7) INT1 INT7 HS22 blade 7 Broadcom VFA INT1 INT7 Cisco Nexus 4001I (switch bay 9) BladeCenter H chassis BC5 Figure 9-1 BladeCenter H chassis BC4 Chassis BC5 is used to demonstrate the following configurations.9.

VLAN 40 vNIC 4 on blade server 1 in chassis BC4 communicates through external port 2 on the VFS module. We use VLAN 40 for this.1. VLAN 20 The blades use this VLAN to communicate through external port 4. VLAN 30 This VLAN does not use any external port on the VFS module. Similarly.Broadcom 2-port 10-Gb Virtual Fabric Adapter and BNT Virtual Fabric 10-Gb Switch Module Chassis midplane HS22 blade 4 Emulex VFA II BNT Virtual Fabric Switch Module (bay 7) INT4 INT8 HS22 blade 8 Broadcom VFA INT4 INT8 BNT Virtual Fabric Switch Module (bay 9) BladeCenter H chassis BC5 Figure 9-2 BladeCenter H chassis BC5 9. vNIC 4 on blade server 4 in chassis BC5 communicates through external port 2 on the BNT 10-Gb Virtual Fabric Switch Module. Examples of use . Chapter 9. vNIC 2 on each blade server communicates on this VLAN.1 Virtual LAN (VLAN) definitions We will enable the following VLANs on virtual fabric switch (VFS) modules: VLAN 10 Both blades use this VLAN for communication through external port 2 on the VFS.Switch Independent Mode 195 . The respective vNIC 3s on each blade server use this VLAN to communicate between themselves. vNIC 1 on each blade server communicates on this VLAN.

2 Switch/adapter vNIC correlation In IBM Virtual Fabric Mode (switch-dependent mode). Similarly. 7 1. External ports can be configured in trunk mode when necessary (when traffic from multiple VLANs passes through a particular external port). 7 1. it is important to configure these ports in trunk mode. there is a direct correlation between vNICs on the Emulex Virtual Fabric Adapter and vNIC definitions on the 196 IBM BladeCenter Virtual Fabric Solutions .VLAN 50 vNIC 4 on blade server 7 in chassis BC4 communicates through external port 4 on the VFS module. We use VLAN 50 for this. 8 4. 7 1 7 2 4 External ports 2 4 Because each internal (blade-facing) port on the switch modules needs to pass traffic in multiple VLANs. Table 9-2 VLANs on BNT 10Gb Virtual Fabric Switch Module VLAN 10 20 30 40 50 Internal ports 4. 8 4 8 2 4 External ports 2 4 9. Table 9-1 and Table 9-2 show these VLANs on Cisco Nexus 4001I Switch Module in chassis BC4 and on BNT 10-Gb Virtual Fabric Switch Module in chassis BC5. 8 4. Table 9-1 VLANs on Cisco Nexus 4001I Switch Module VLAN 10 20 30 40 50 Internal ports 1. vNIC 4 on blade server 8 in chassis BC5 communicates through external port 4 on the BNT 10-Gb Virtual Fabric Switch Module.1.

“Emulex adapter configuration” on page 198 9. We will enable vNICs on the adapter. 9. for example. In Switch Independent Mode. “Broadcom adapter configuration” on page 208 9.2 Cisco Nexus 4001I Switch Module configurations In this section. Topics in this section are: 9. In our examples.Switch Independent Mode 197 .BNT Virtual Fabric 10-Gb Switch Module.3. we set the following VLANs for vNICs in the operating system: VLAN 10 for vNIC 1 Chapter 9. Examples of use . Finally.2. we do need to configure internal ports as trunk ports and associate one or more VLANs with each vNIC channel.1. but we do not specify any VLAN IDs on the Broadcom 2-port 10-Gb Virtual Fabric Adapter. However. using the Unified Extensible Firmware Interface (UEFI) utility. This is because vNICs are configured on the switch module. We configure vNICs on the adapter itself. “Cisco Nexus 4001I Switch Module configuration” on page 224 9. then show the Broadcom configuration. However.4. No vNIC settings need to be configured on the switch module.2. we have installed these modules in chassis BC4.2.2. we enable vNICs and set the following VLAN IDs for them: VLAN 10 for vNIC 1 VLAN 20 for vNIC 2 VLAN 30 for vNIC 3 VLAN 40 for vNIC 4 The HS22 blade server in slot 7 contains a Broadcom 2-port 10-Gb Virtual Fabric Adapter. and settings are transferred to the adapter via DCBX.1 vNICs summary The HS22 blade server in slot 1 contains an Emulex 10GbE Virtual Fabric Adapter II.2. The blade in slot 1 has Emulex 10GbE Virtual Fabric Adapter II. We begin with the Emulex configuration on blade 1. we show examples using the Cisco Nexus 4001I Switch Module. this is different. “vNICs summary” on page 197 9. The intent is to extend specific VLANs to specific vNIC interfaces within each server blade. and the blade in slot 7 has Broadcom 2-port 10-Gb Virtual Fabric Adapter. On this adapter. we will set up the switch module. thus allowing for multiple VLANs on each port to ensure that the traffic flows to the correct interfaces.2.

VLAN 20 for vNIC 2 VLAN 30 for vNIC 3 VLAN 50 for vNIC 4 9.2. Table 9-3 vNIC configuration: Emulex 10GbE Virtual Fabric Adapter II vNIC function 1 2 3 4 Bandwidth 4Gbps 3Gbps 2Gbps 1Gbps VLAN ID 10 20 30 40 MAC address 00:00:C9:B2:57:02 00:00:C9:B2:57:03 00:00:C9:B2:57:04 00:00:C9:B2:57:05 We can identify the MAC addresses for each vNIC function using the UEFI utility. To configure parameters on the Emulex adapter. perform these steps: 1. press F1.2 Emulex adapter configuration Table 9-3 shows the configuration that we will implement on the Emulex 10GbE Virtual Fabric Adapter II in HS22 blade 1. 198 IBM BladeCenter Virtual Fabric Solutions . (see “vNICs in the operating system” on page 205). and the UEFI utility main menu displays (Figure 9-3 on page 199). When prompted during POST. It is important to know the MAC addresses so that you can identify each vNIC function in the operating system. Power on or restart HS22 blade 1. 2.

and then select Network on the subsequent panel to display the Network panel.Figure 9-3 UEFI utility main menu 3. Select System Settings. showing two Emulex 10-Gb ports (Figure 9-4). Examples of use . Figure 9-4 Network configuration in the blade server UEFI utility Chapter 9.Switch Independent Mode 199 .

Figure 9-5 Set vNIC2 mode on the Emulex 10GbE Virtual Fabric Adapter II 5. – vNIC2 mode must be set in the Switch Configuration field. 6. and the Emulex NIC Selection panel displays (Figure 9-5). Configure VLAN and bandwidth settings for each of the four vNICs by selecting Multichannel Configuration.4. Ensure that the following values are set: – Multichannel must be enabled. 200 IBM BladeCenter Virtual Fabric Solutions . which displays the Function Configuration panel (Figure 9-6 on page 201). Select the first port on which to configure vNICs.

Figure 9-6 Function Configuration panel 7. and this vNIC will use VLAN 10 (Figure 9-7 on page 202). Examples of use . Configure each of the four vNIC functions: a. Chapter 9. Set 4 Gbps (40% of total bandwidth) for the first vNIC function with MAC address 00:00:C9:B2:57:02.Switch Independent Mode 201 .

and this vNIC will use VLAN 20 (Figure 9-8).Figure 9-7 Bandwidth and VLAN settings for the first vNIC b. Set 3 Gbps (30% of bandwidth) for the second vNIC function with MAC address 00:00:C9:B2:57:03. Figure 9-8 Bandwidth and VLAN settings for the second vNIC 202 IBM BladeCenter Virtual Fabric Solutions .

Chapter 9. and this vNIC will use VLAN 30 (Figure 9-9). Set 2 Gbps (20% of bandwidth) for the third vNIC function with MAC address 00:00:C9:B2:57:04). Examples of use . Figure 9-9 Bandwidth and VLAN settings for the third vNIC d.Switch Independent Mode 203 . and this vNIC will use VLAN 40 (Figure 9-10 on page 204).c. Set 1 Gbps (10% of bandwidth) for the fourth vNIC function with MAC address 00:00:C9:B2:57:05.

With all four vNIC functions configured. save your changes (Figure 9-11). Figure 9-11 Save settings before exiting 204 IBM BladeCenter Virtual Fabric Solutions .Figure 9-10 Bandwidth and VLAN settings for the fourth vNIC 8.

Examples of use . four on each 10-Gb physical port. Chapter 9.Switch Independent Mode 205 . listing eight vNICs. We can now check how the vNICs appear to the operating system.Configure vNIC functions on the second 10-Gb physical port in the same manner. Use the steps in “Emulex adapter configuration” on page 198 as a guide. Figure 9-12 Save settings before exiting vNICs in the operating system HS22 blade server 1 runs Windows Server 2008. Figure 9-13 on page 206 shows the Network Connections window. Remember to save your settings before exiting (Figure 9-12).

the Emulex VFA II #01) and select Status (Figure 9-14 on page 207). We used the MAC addresses to match each vNIC with the adapter displayed in the Network Connections window. we have changed the names of NICs from defaults to the values listed in Table 9-4. including MAC address and link speed. 206 IBM BladeCenter Virtual Fabric Solutions .Figure 9-13 Network Connections window For clarity. right-click a NIC (for example. Table 9-4 NIC names and matching vNICs NIC name Emulex VFA II #01 Emulex VFA II #02 Emulex VFA II #03 Emulex VFA II #04 Emulex VFA II #11 Emulex VFA II #12 Emulex VFA II #13 Emulex VFA II #14 10Gb physical port 0 0 0 0 1 1 1 1 vNIC 1 2 3 4 1 2 3 4 MAC address 00:00:C9:B2:57:02 00:00:C9:B2:57:03 00:00:C9:B2:57:04 00:00:C9:B2:57:05 00:00:C9:B2:57:06 00:00:C9:B2:57:07 00:00:C9:B2:57:08 00:00:C9:B2:57:09 To verify parameters.

in which the Physical Address field shows the MAC address. click Details to display the Network Connection Details panel (Figure 9-15 on page 208). To view the MAC address. because we set bandwidth to 40 on this vNIC (see Figure 9-7 on page 202). Examples of use .Figure 9-14 Emulex VFA II #01 status Speed is reported as 4.Switch Independent Mode 207 . Chapter 9.0 Gbps.

2. Table 9-5 lists the settings for the Broadcom 2-port 10-Gb Virtual Fabric Adapter. Identify and verify MAC addresses and link speeds of all other vNICs in the same manner. 9. Table 9-5 Broadcom 2-port 10-Gb Virtual Fabric Adapter vNIC configuration NIC partition 1 2 Bandwidth 4Gbps 3 Gbps MAC address 10:18:00:96:E3:80 10:18:00:96:E3:84 208 IBM BladeCenter Virtual Fabric Solutions .3 Broadcom adapter configuration We set up the Broadcom 2-port 10-Gb Virtual Fabric Adapter in a manner similar to that for the Emulex adapter.which only uses MAC addresses for virtual channel traffic separation. with one difference: We do not specify any VLAN ID on the Broadcom 2-port 10-Gb Virtual Fabric Adapter.Figure 9-15 Network Connection Details This MAC address indeed belongs to vNIC 1 on 10-Gb physical port 0.

2. Power on or restart the blade server. Chapter 9. Follow these steps to configure the Broadcom 2-port 10-Gb Virtual Fabric Adapter parameters in the UEFI utility: 1.Switch Independent Mode 209 . to display the Network panel (Figure 9-17 on page 210). then Network on the subsequent panel. When prompted during POST. press F1 to display the UEFI utility main menu (Figure 9-16). It is important to know the MAC addresses so that you can identify each NIC partition in the operating system.main menu 3. Select System Settings. Figure 9-16 Blade server UEFI utility . Examples of use .NIC partition 3 4 Bandwidth 2 Gbps 1 Gbps MAC address 10:18:00:96:E3:88 10:18:00:96:E3:8C We can identify the MAC addresses for each NIC partition using the UEFI utility (see Figure 9-7 on page 202).

Figure 9-18 Broadcom Main Configuration panel 210 IBM BladeCenter Virtual Fabric Solutions . and this displays the Broadcom Main Configuration panel (Figure 9-18). Select the first of the two 10-Gb physical ports to configure.Figure 9-17 Network configuration .select the first 10-Gb port 4.

With NIC partition enabled. press Esc to return to the Broadcom Main Configuration Page. Chapter 9. 7. Notice that a new option displays. Figure 9-19 Device Configuration Menu with NIC Partition set to Enabled 6. Highlight NIC Partition and set it to Enabled. the NIC Partitioning Configuration Menu option (Figure 9-20 on page 212). Select Device Configuration Menu to display the Device Configuration panel. Examples of use . Figure 9-19.Switch Independent Mode 211 .5.

Select the NIC Partitioning Configuration Menu option to display the NIC Partition Configuration panel (Figure 9-21).Figure 9-20 New configuration option for NIC partitioning 8. Figure 9-21 NIC Partition Configuration panel 212 IBM BladeCenter Virtual Fabric Solutions .

On this panel. Figure 9-22 appears. In our example. you can set the bandwidth allocation parameters. we do not use relative bandwidth weight. Highlight Global Bandwidth Allocation Menu and press Enter. Figure 9-22 Global Bandwidth Allocation Menu 10. we do configure maximum bandwidth. which means the MAC address is 10:18:00:96:E3:84 – PF#4 is the third NIC partition: L2=10180096E388. 9. which means the MAC address is 10:18:00:96:E3:8C It is important to know these MAC addresses so that you can match the NICs presented in the operating system to the NIC partitions. which means the MAC address is 10:18:00:96:E3:88 – PF#6 is the fourth NIC partition: L2=10180096E38C. as follows: – NIC partition 1: 4Gbps (40% of total bandwidth) – NIC partition 2: 3Gbps (30% of total bandwidth) Chapter 9. which means the MAC address is 10:18:00:96:E3:80 – PF#2 is the second NIC partition: L2=10180096E384.Using this menu.Switch Independent Mode 213 . Examples of use . you can identify the MAC addresses of all four NIC partitions: – PF#0 represents the first NIC partition: L2=10180096E380. However. so we leave these values at zero.

Windows networking tools Figure 9-24 on page 215 shows how the NIC partitions appear in Network and Sharing Center.Configure NIC partitions and bandwidth parameters for the second 10-Gb physical port. Figure 9-23 Save settings before exiting the UEFI utility vNICs in the operating system We now verify how the vNIC partitions are seen in the operating system.– NIC partition 3: 2Gbps (20% of total bandwidth) – NIC partition 4: 1Gbps (10% of total bandwidth) 11. starting with Figure 9-17 on page 210 12. 214 IBM BladeCenter Virtual Fabric Solutions . Blade 7 runs Windows Server 2008. and then the Broadcom Advanced Control Suite (BACS).Save your settings before exiting the UEFI utility (Figure 9-23). We first describe the Windows built-in networking tools.

Examples of use .Switch Independent Mode 215 . we have changed the NIC names from the defaults according to Table 9-6. We used MAC addresses to match each NIC with the actual NIC partition.Figure 9-24 Broadcom 2-port 10-Gb Virtual Fabric Adapter NIC partitions in Windows Server 2008 For clarity. Table 9-6 NIC names and matching NIC partitions NIC name Broadcom VFA 01 Broadcom VFA 02 Broadcom VFA 03 Broadcom VFA 04 Broadcom VFA 11 Broadcom VFA 12 Broadcom VFA 13 10-Gb physical port 0 0 0 0 1 1 1 NIC partition 1 2 3 4 1 2 3 MAC address 10:18:00:96:E3:80 10:18:00:96:E3:84 10:18:00:96:E3:88 10:18:00:96:E3:8C 10:18:00:96:E3:82 10:18:00:96:E3:86 10:18:00:96:E3:8A Chapter 9.

click View Status (Figure 9-25 shows this for NIC Broadcom VFA 01). 216 IBM BladeCenter Virtual Fabric Solutions . To view the MAC address of this NIC partition. speed is reported as 4. because we set bandwidth to 40 on this NIC partition (see Figure 9-22 on page 213). click Details to display the Network Connection Details panel (Figure 9-26 on page 217). Figure 9-25 Broadcom VFA 01 status As you can see.NIC name Broadcom VFA 14 10-Gb physical port 1 NIC partition 4 MAC address 10:18:00:96:E3:8E To verify the parameters for each NIC partition. such as link speed and MAC address.0 Gbps.

Switch Independent Mode 217 . Examples of use . follow these steps to define VLAN ID 10 on NIC named Broadcom VFA 01: 1. Chapter 9. We can use the same method to verify MAC addresses and speed for all other NIC partitions on the Broadcom 2-port 10-Gb Virtual Fabric Adapter. launch the properties window for the Broadcom VFA 01 (Figure 9-27 on page 218). From the Networking in Windows Control Panel.Figure 9-26 Network Connection Details showing the MAC address This is indeed the MAC address of NIC partition 1 on 10-Gb physical port 0. For example. VLAN IDs in the operating system You can define VLAN IDs for each NIC partition in the operating system.

then click the Advanced tab. set the VLAN ID to 10. 218 IBM BladeCenter Virtual Fabric Solutions . to display a window in which you can configure NIC settings. 3. For this example.Figure 9-27 Broadcom VFA 01 properties window 2. such as the VLAN ID (Figure 9-28 on page 219). Click Configure.

configure.VLAN ID 40 6. Chapter 9. Use the same steps to configure the VLAN ID on other NIC partitions on 10-Gb physical port 0. and diagnose Broadcom Virtual Fabric Adapters (VFAs) and other Broadcom NICs.VLAN ID 20 – Broadcom VFA 03 . Click OK to save the changes. Figure 9-29 on page 220 shows an example from BACS. Note that the speed is 4 Gbps. Examples of use . as follows: – Broadcom VFA 02 . Broadcom Advanced Control Suite (BACS) Broadcom Advanced Control Suite (BACS) is a management utility that allows you to administer.Switch Independent Mode 219 .VLAN ID 30 – Broadcom VFA 04 . starting with Figure 9-27 on page 218. Configure VLAN IDs on NIC partitions on 10-Gb physical port 1 in the same manner.Figure 9-28 VLAN ID configuration 4. You can see eight NIC partitions in the left pane. as we set it previously (see Figure 9-22 on page 213). with NIC partition 1 on physical port 0 highlighted. 5.

you can check and set NIC parameters (Figure 9-30 on page 221).Figure 9-29 BACS: NIC partition 1 On the Configurations tab. or to change any other parameter. 220 IBM BladeCenter Virtual Fabric Solutions . You can use BACS to configure the VLAN ID to a different value. Notice that VLAN ID is highlighted and is set to 10 (as we set it in Figure 9-28).

Examples of use .configuration tab On the Diagnostics tab. Chapter 9.Figure 9-30 BACS . you can test configurations by running a basic connectivity test: ping either to an IP gateway or to a user-specified IP address (Figure 9-31 on page 222).Switch Independent Mode 221 .

222 IBM BladeCenter Virtual Fabric Solutions .Figure 9-31 BACS .Diagnostics tab The Statistics tab displays network traffic counters (Figure 9-32 on page 223). These counters are useful when monitoring or troubleshooting NIC behavior.

“Configuring teams with Broadcom BACS” on page 184.Switch Independent Mode 223 . From here. and then configure teaming parameters.Statistics tab One of the main functions of the BACS utility is to configure NIC teaming.8. you can select the NICs (or NIC partitions) for the team. The teaming wizard is discussed in 8.4.Figure 9-32 BACS . Chapter 9. You can use NIC partitions in teaming configurations in the same way you would use physical NICs. click Team Management (lower left corner) to start the teaming wizard (Figure 9-33 on page 224). Examples of use . To set up NIC teaming.

Figure 9-33 BACS: Team Management

9.2.4 Cisco Nexus 4001I Switch Module configuration
We are now ready to configure VLANs on the Cisco Nexus 4001I Switch Module. The HS22 blade in slot 1 contains the Emulex adapter which is configured to use the following VLANs on vNICs: 10, 20, 30, and 40. So we need to set the internal (blade-facing) port 1 on the switch module to trunk mode, and allow VLANs 10, 20, 30, and 40 on it. The commands to perform these actions are in Figure 9-34 on page 225.

224

IBM BladeCenter Virtual Fabric Solutions

ibm-switch-2(config)# configure terminal ibm-switch-2(config)# interface Ethernet 1/1 ibm-switch-2(config-if)# switchport mode trunk ibm-switch-2(config-if)# switchport trunk allowed vlan 10, 20, 30, 40 Figure 9-34 Configuring internal port 1 on Cisco Nexus 4001I Switch Module

The HS22 blade in slot 7 contains the Broadcom adapter. There is no VLAN ID configuration in the UEFI settings for the Broadcom card, but we set the following VLANs in the operating system: 10, 20, 30, and 50. We now need to configure internal port 7 on the switch module to pass traffic in these VLANs. The commands are shown in Figure 9-35.
ibm-switch-2(config-if)# configure terminal ibm-switch-2(config)# interface Ethernet 1/7 ibm-switch-2(config-if)# switchport mode trunk ibm-switch-2(config-if)# switchport trunk allowed vlan 10, 20, 30, 50 Figure 9-35 Configuring internal port 7 on Cisco Nexus 4001I Switch Module

We need to allow VLANs also on external ports, so that the blade servers can communicate with devices outside of the chassis. If any external port needs to carry traffic from multiple VLANs, you need to set trunk port mode, and allow the corresponding VLANs on it. In our example, two external ports are used: External port 2 needs to carry VLANs 10 and 40. Figure 9-36 lists the commands to allow this.
ibm-switch-2(config-if)# configure terminal ibm-switch-2(config)# interface Ethernet 1/16 ibm-switch-2(config-if)# switchport mode trunk ibm-switch-2(config-if)# switchport trunk allowed vlan 10, 40 Figure 9-36 External port 2 VLANs

External port 4 will carry VLANs 20 and 50. Figure 9-37 on page 226 shows the commands to configure the port.

Chapter 9. Examples of use - Switch Independent Mode

225

ibm-switch-2(config-if)# configure terminal ibm-switch-2(config)# interface Ethernet 1/18 ibm-switch-2(config-if)# switchport mode trunk ibm-switch-2(config-if)# switchport trunk allowed vlan 20, 50 Figure 9-37 External port 4 VLANs

This completes the configuration of VLANs on Cisco Nexus 4001I Switch Module.

9.3 BNT Virtual Fabric 10-Gb Switch Module configurations
In this section, we show the examples with BNT Virtual Fabric 10-Gb Switch Module. These modules are installed in chassis BC5 in our lab examples. Blade in slot 4 has Emulex 10GbE Virtual Fabric Adapter II, and blade in slot 8 has Broadcom 2-port 10-Gb Virtual Fabric Adapter. We begin with an Emulex configuration, and then also show the Broadcom configuration. Finally, we will set up the switch module. Topics we cover in this section are: 9.3.1, “Emulex adapter configuration” 9.3.2, “Broadcom adapter configuration” on page 237 9.3.3, “Configuring the BNT Virtual Fabric 10-Gb Switch Module” on page 252

9.3.1 Emulex adapter configuration
Table 9-3 shows the configuration we implemented on the Emulex 10GbE Virtual Fabric Adapter II on the HS22, blade 4.
Table 9-7 Emulex Virtual Fabric Adapter II vNIC configuration vNIC function 1 2 3 4 Bandwidth 4 Gbps 3 Gbps 2 Gbps 1 Gbps VLAN ID 10 20 30 40 MAC address 00:00:C9:B2:5B:3A 00:00:C9:B2:5B:3B 00:00:C9:B2:5B:3C 00:00:C9:B2:5B:3D

226

IBM BladeCenter Virtual Fabric Solutions

You can identify the MAC addresses for each vNIC function using the Unified Extensible Firmware Interface (UEFI) utility (see Figure 9-39 on page 228). With the MAC addresses, you can then identify each vNIC function in the operating system. To configure the parameters on the Emulex adapter: 1. Power on or restart HS22 blade 4. 2. When prompted during POST, press F1, and the System Configuration and Boot Management panel displays (also known as the UEFI utility Main Menu) (Figure 9-3).

Figure 9-38 The UEFI utility

3. Select System Settings, then select Network on the subsequent panel to display the Network panel that shows two Emulex 10-Gb ports (Figure 9-39 on page 228).

Chapter 9. Examples of use - Switch Independent Mode

227

Figure 9-39 Network configuration in the blade server UEFI utility

4. Select the first port to configure the vNICs for, then press Enter to display the Emulex NIC Selection panel (Figure 9-5).

Figure 9-40 Set vNIC2 mode on the Emulex Virtual Fabric Adapter II

228

IBM BladeCenter Virtual Fabric Solutions

then pressing Enter to display the Multichannel Configuration panel for that vNIC. a. Examples of use . Figure 9-41 Function Configuration panel 7.5. ensure that the following values are set: – Multichannel is set to Enabled. Configure each of the four vNIC functions by selecting each Function menu item in turn.Switch Independent Mode 229 . – The Switch Configuration field is set to the vNIC2 mode. On the Emulex NIC Selection panel. Allocate 4 Gbps (40% of total bandwidth) to the first vNIC function. Highlight Multichannel Configuration and press Enter to display the Function Configuration panel (Figure 9-6). 6. Chapter 9. which has MAC address 00:00:C9:B2:5B:3A and will use VLAN 10 (Figure 9-42 on page 230). Configure the VLAN and bandwidth settings for each of the four vNICs: a.

Figure 9-43 Bandwidth and VLAN settings for the second vNIC 230 IBM BladeCenter Virtual Fabric Solutions . which has MAC address 00:00:C9:B2:5B:3B and will use VLAN 20 (Figure 9-8).Figure 9-42 Bandwidth and VLAN settings for the first vNIC b. Allocate 3 Gbps (30% of total bandwidth) to the second vNIC function.

Allocate 2 Gbps (20% of total bandwidth) to the third vNIC function.c. Chapter 9. Allocate 1 Gbps (10% of total bandwidth) to the fourth vNIC function. which has MAC address 00:00:C9:B2:5B:3D and will use VLAN 40 (Figure 9-45 on page 232). Figure 9-44 Bandwidth and VLAN settings for the third vNIC d. which has MAC address 00:00:C9:B2:5B:3C and will use VLAN 30 (Figure 9-9).Switch Independent Mode 231 . Examples of use .

With the four vNIC functions configured.Figure 9-45 Bandwidth and VLAN settings for the fourth vNIC 8. save the settings (Figure 9-46). Figure 9-46 Save settings before exiting 232 IBM BladeCenter Virtual Fabric Solutions .

Configure the vNIC functions on the second 10-Gb physical port in the same manner. by selecting the second 10-Gb port in Figure 9-39 on page 228 and following the similar procedure. Chapter 9.9. Examples of use .Switch Independent Mode 233 . The Network Settings panel is shown in Figure 9-48 on page 234.Save all settings before exiting (Figure 9-47). 10. which lists eight vNICs. Figure 9-47 Save settings before exiting vNICs in the operating system The HS22 blade in bay 4 runs SLES 11. You can now find out how the vNICs appear to the operating system. four on each 10-Gb physical port.

you can use OneCommand Manager (see “OneCommand Manager” on page 235) or the ifconfig command (see “The ifconfig command” on page 236). For clarity. Linux and VMware both report the Emulex 10GbE Virtual Fabric Adapter II adapter as having a ServerEngines chip. we changed the names of the NICs from the default values to those listed in Table 9-4 on page 206. This will change to Emulex in a future firmware update.Figure 9-48 Network Settings Tip: At the time of writing. We used the MAC addresses to match each vNIC with the adapter as displayed on the Network Connections panel. 234 IBM BladeCenter Virtual Fabric Solutions . To determine the MAC addresses.

Table 9-8 NIC names and matching vNICs NIC name eth4 eth6 eth8 eth10 eth5 eth7 eth9 eth11 10Gb physical port 0 0 0 0 1 1 1 1 vNIC 1 2 3 4 1 2 3 4 MAC address 00:00:C9:B2:5B:3A 00:00:C9:B2:5B:3B 00:00:C9:B2:5B:3C 00:00:C9:B2:5B:3D 00:00:C9:B2:5B:3E 00:00:C9:B2:5B:3F 00:00:C9:B2:5B:40 00:00:C9:B2:5B:41 OneCommand Manager You can use OneCommand Manager to verify parameters such as MAC addresses and link speed. Figure 9-49 on page 236 is the status panel for NIC eth4. Examples of use . Chapter 9.Switch Independent Mode 235 .

Figure 9-49 Eth4 status The speed is reported as 4. Figure 9-50 on page 237 shows the results of the ifconfig command. The MAC addresses are displayed in the Hwaddr field. 236 IBM BladeCenter Virtual Fabric Solutions . The ifconfig command Another method of displaying MAC addresses is to use the ifconfig command.0 Gbps (see Figure 9-14 on page 207) because we set the bandwidth to 40 on this vNIC in step 7 on page 229.

0 b) TX bytes:0 (0.0.255. 9. You can identify and verify the MAC addresses and link speeds of all other vNICs in the same manner.0.10.70 Bcast:10.255 Mask:255.3.255.10.0 b) Memory:95a80000-95aa0000 Link encap:Ethernet HWaddr 00:00:C9:B2:5B:41 inet addr:10. with one difference: You do not specify a VLAN ID for the Broadcom adapter.0 UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:0 errors:0 dropped:0 overruns:0 frame:0 TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:0 (0. Table 9-5 lists our settings for the Broadcom 2-port 10-Gb Virtual Fabric Adapter.0.Switch Independent Mode 237 .10.0 b) TX bytes:0 (0. The Broadcom adapter uses only MAC addresses for virtual channel traffic separation.2 Broadcom adapter configuration The Broadcom 2-port 10-Gb Virtual Fabric Adapter is set up in a similar manner as the Emulex 10GbE Virtual Fabric Adapter II. Examples of use . Table 9-9 Broadcom Virtual Fabric Adapter vNIC configuration NIC partition 1 2 3 4 Bandwidth 4 Gbps 3 Gbps 2 Gbps 1 Gbps MAC address 10:18:00:96:CE:90 10:18:00:96:CE:94 10:18:00:96:CE:98 10:18:00:96:CE:9C Chapter 9.0 b) Memory:95ac0000-95ae0000 eth11 Figure 9-50 Results of the ifconfig command for viewing MAC addresses We confirmed that this MAC address belongs to vNIC 4 on 10-Gb physical port 0 (eth10) and vNIC 4 on 10-Gb physical port 1 (eth11).255 Mask:255.10.255.0 UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:0 errors:0 dropped:0 overruns:0 frame:0 TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:0 (0.80 Bcast:10.eth10 Link encap:Ethernet HWaddr 00:00:C9:B2:5B:3D inet addr:10.0.255.

Figure 9-51 The UEFI utility Main Menu 3. press F1. It is important to determine the MAC addresses so that you can identify each NIC partition in the operating system. Power on or restart the blade server. Select System Settings. Follow these steps to configure the Broadcom adapter parameters using the UEFI utility: 1. 238 IBM BladeCenter Virtual Fabric Solutions . When prompted during POST. then select Network on the subsequent panel to display the Network configuration window. 2. and the UEFI utility Main Menu displays (Figure 9-16).You can identify the MAC addresses for each NIC partition using the UEFI utility (see Figure 9-21 on page 212).

Figure 9-53 Broadcom Main Configuration panel 5. Select the first of the two Broadcom NetXtreme II 10 Gigabit Ethernet entries.Figure 9-52 Network configuration panel 4. and press Enter to display the Broadcom Main Configuration Page (Figure 9-53).Switch Independent Mode 239 . Examples of use . Chapter 9. Select Device Configuration Menu and press Enter to display the Device Configuration panel (Figure 9-54 on page 240).

Figure 9-54 Device Configuration panel with NIC Partition set to Enabled 6. Figure 9-56 on page 241 appears. Highlight it and press Enter. press Esc to return to the Broadcom Main Configuration Page. Once the NIC partition parameter is enabled. A new option is available now. 240 IBM BladeCenter Virtual Fabric Solutions . 7. Ensure that NIC Partition is set to Enabled. Figure 9-55 New configuration option: the NIC Partitioning Configuration Menu 8. NIC Partitioning Configuration Menu. Figure 9-55.

The NIC Partition Configuration panel is shown in Figure 9-56. which means the MAC address is 10:18:00:96:CE:90 – PF#2 is the second NIC partition: L2=10180096CE94. You can identify the MAC addresses of all four NIC partitions on this panel: – PF#0 represents the first NIC partition: L2=10180096CE90.Switch Independent Mode 241 . Examples of use . which means the MAC address is 10:18:00:96:CE:9C It is important to know these MAC addresses so that you can match the NICs presented in the operating system to NIC partitions. which means the MAC address is 10:18:00:96:CE:98 – PF#6 is the fourth NIC partition: L2=10180096CE9C.Figure 9-56 NIC Partition Configuration 9. Chapter 9. 10.Select Global Bandwidth Allocation Menu and press Enter to display that menu (Figure 9-57 on page 242). which means the MAC address is 10:18:00:96:CE:94 – PF#4 is the third NIC partition: L2=10180096CE98.

Set the bandwidth allocation parameters in this panel. so leave those values at zero. starting from step 4 on page 239.Figure 9-57 Global Bandwidth Allocation Menu 11.Repeat the steps to configure NIC partitions and bandwidth parameters for the second 10-Gb physical port. For our example: – Do not use Relative Bandwidth Weight. 242 IBM BladeCenter Virtual Fabric Solutions .Save your changes before exiting the UEFI utility (Figure 9-58 on page 243). – Configure maximum bandwidth as follows: • • • • NIC partition 1: 4 Gbps (40% of total bandwidth) NIC partition 2: 3 Gbps (30% of total bandwidth) NIC partition 3: 2 Gbps (20% of total bandwidth) NIC partition 4: 1 Gbps (10% of total bandwidth) 12. 13.

we describe the Windows built-in networking tools and the Broadcom Advanced Control Suite (BACS).Figure 9-58 Save settings before exiting the UEFI utility NIC partitions in the operating system We can define how the NIC partitions are seen by the operating system. Examples of use .Switch Independent Mode 243 . Chapter 9. Windows networking tools Figure 9-59 on page 244 shows how the NIC partitions appear in the Network and Sharing Center. In this section. Blade 7 runs on Windows Server 2008.

We used MAC addresses to match each NIC with the actual NIC partition in Table 9-10. we have changed the NIC names from the defaults according to Table 9-6.Figure 9-59 Broadcom Virtual Fabric Adapter NIC partitions in Windows Server 2008 For clarity. Table 9-10 NIC names and matching NIC partitions NIC name Broadcom VFA 01 Broadcom VFA 02 Broadcom VFA 03 Broadcom VFA 04 Broadcom VFA 11 Broadcom VFA 12 Broadcom VFA 13 10-Gb physical port 0 0 0 0 1 1 1 NIC partition 1 2 3 4 1 2 3 MAC address 10:18:00:96:CE:90 10:18:00:96:CE:94 10:18:00:96:CE:98 10:18:00:96:CE:9C 10:18:00:96:CE:92 10:18:00:96:CE:96 10:18:00:96:CE:9A 244 IBM BladeCenter Virtual Fabric Solutions .

Examples of use .0 Gbps. On the Network and Sharing Center panel. click View Status for each partition to verify the parameters. because we set the bandwidth to 40 on this NIC partition. Figure 9-60 Broadcom VFA 01 status 2. and the MAC address of the selected NIC partition displays on the Network Connection Details panel (Figure 9-61 on page 246). Click Details. speed is reported as 4.Switch Independent Mode 245 .NIC name Broadcom VFA 14 10-Gb physical port 1 NIC partition 4 MAC address 10:18:00:96:CE:9E 1. Chapter 9. such as link speed and MAC address (see Figure 9-60 for parameters of the NIC Broadcom VFA 01). As you can see.

Repeat the steps to verify the MAC address and speed for the other NIC partitions on the Broadcom Virtual Fabric Adapter (see Figure 9-24 on page 215 through Figure 9-26 on page 217). 3.The MAC address matches vNIC 1 on port 0 as listed in Figure 9-53 on page 239 Figure 9-61 Network Connection Details panel for a selected NIC partition We can confirm from Figure 9-26 that this is the MAC address of NIC partition 1 (the Broadcom DCM57712 NeXtreme II) on 10-Gb physical port 0. For example. Open the Broadcom VFA 01 Properties panel ()(Figure 9-62 on page 247) 246 IBM BladeCenter Virtual Fabric Solutions . 1. Setting VLAN IDs in the operating system in Windows You can define VLAN IDs for each NIC partition in the operating system. follow these steps to define VLAN ID 10 for NIC Broadcom VFA 01.

3.Switch Independent Mode 247 . Figure 9-63 on page 248 appears.Figure 9-62 Broadcom VFA 01 properties window 2. Click Configure to configure the vNIC. Click the Advanced tab. Chapter 9. Scroll the Property box and select VLAN ID. Examples of use .

Repeat these steps to configure the remaining VLAN IDs for the other NIC partitions on 10-Gb physical port 0. NIC partition 1 on physical port 0 is highlighted. 5.Figure 9-63 VLAN ID configuration 4. as follows: – Broadcom VFA 02 .VLAN ID 30 – Broadcom VFA 04 . Note the speed: 4 Gbps (as was set in the bandwidth parameter in the UEFI utility). Set the VLAN ID to 10 as shown in Figure 9-63. 248 IBM BladeCenter Virtual Fabric Solutions . configure. and diagnose the Broadcom 2-port 10-Gb Virtual Fabric Adapter and other Broadcom NICs.VLAN ID 20 – Broadcom VFA 03 . Figure 9-64 on page 249 shows an example of BACS. indicating eight NIC partitions in the left pane.VLAN ID 40 Broadcom Advanced Control Suite (BACS) The BACS management utility allows you to administer.

Examples of use . Note that the VLAN ID is set to 10 (as in Figure 9-28 on page 219). or to change any other parameter. Use the BACS utility to configure the VLAN ID to a different value. Chapter 9.NIC partition 1 On the Configurations tab.Switch Independent Mode 249 .Figure 9-64 BACS . you can check and set NIC parameters (Figure 9-65 on page 250).

250 IBM BladeCenter Virtual Fabric Solutions .Configuration tab On the Diagnostics tab.Figure 9-65 BACS . you can test a configuration by running a basic connectivity test (either ping to IP gateway or a user-specified IP address) (Figure 9-66 on page 251).

Switch Independent Mode 251 . Chapter 9.Figure 9-66 BACS .Diagnostics tab The Statistics tab displays network traffic counters. which are useful when monitoring or troubleshooting NIC behavior (Figure 9-67 on page 252). Examples of use .

Figure 9-67 BACS . Use NIC partitions in teaming configurations in the same way as with physical NICs. 30.3. and 40. 9. “Configuring teams with Broadcom BACS” on page 184. 20. We discuss this in 8. you can start the Teaming Wizard.3 Configuring the BNT Virtual Fabric 10-Gb Switch Module You are now ready to configure VLANs on the BNT Virtual Fabric 10-Gb Switch Module. which is configured to use VLANs 10.Statistics tab NIC Teaming using BACS One of the main functions of the BACS utility is to configure NIC teaming.8.4. Internal (blade-facing) 252 IBM BladeCenter Virtual Fabric Solutions . By clicking the Team Management frame. The HS22 blade server in slot 4 contains Emulex 10GbE Virtual Fabric Adapter II.

By default. interface port EXT2 tagging exit interface port EXT4 tagging exit Figure 9-68 Enable tagging on EXT2 and EXT4 The VLAN structure is defined in Table 9-2 on page 196. 30 and 50. The sequence of isCLI commands to configure these VLANs is shown in Figure 9-69 on page 254. Figure 9-68 shows isCLI commands needed for this. 20. Chapter 9. configured to use VLANs 10. Internal (blade-facing) port 8 (INT8) on the BNT switch module needs to pass traffic in all these VLANs. The HS22 blade server in slot 8 contains Broadcom Virtual Fabric Adapter. This means VLAN tagging must be enabled on this port. Examples of use . So you need to enable tagging on ports EXT2 and EXT4.Switch Independent Mode 253 . Both EXT2 and EXT4 must have tagging enabled. but disabled on external ports. so VLAN tagging must be enabled on this port. External port 2 (EXT2) must pass traffic in VLANs 10 and 40.port 4 (INT4) on the BNT switch module needs to pass traffic in all these VLANs. VLAN tagging is enabled on internal ports. and external port 4 (EXT4) must pass traffic in VLANs 20 and 50.

254 IBM BladeCenter Virtual Fabric Solutions .vlan 10 enable name "VLAN 10" member INT4 member INT8 member EXT2 vlan 20 enable name "VLAN 20" member INT4 member INT8 member EXT4 vlan 30 enable name "VLAN 30" member INT4 member INT8 vlan 40 enable name "VLAN 40" member INT4 member EXT2 vlan 50 enable name "VLAN 50" member INT8 member EXT4 Figure 9-69 VLAN definitions on BNT Virtual Fabric Switch Module This completes configuration of the BNT Virtual Fabric Switch Module.

255 .Abbreviations and acronyms AC ACL AFT ALB AMM ASIC BACS BBI BOFM CD-ROM CDP CEE CLI DAC DC DCB DCBX DHCP DMA ETS ETSI alternating current access control list adapter fault tolerance adaptive load balancing Advanced Management Module application-specific integrated circuit Broadcom Advanced Control Suite browser-based interface BladeCenter Open Fabric Manager compact disc read only memory Cisco Discovery Protocol Converged Enhanced Ethernet command-line interface dual address cycle domain controller Data Center Bridging Data Center Bridging Capability Exchange Dynamic Host Configuration Protocol direct memory access Enhanced Technical Support European Telecommunications Standard Industry Fibre Channel full duplex Fast Etherchannel LED LPVID LR LSO MAC MB MSTP KVM LACP LAN LDAP IP IPTV IT ITSO ID IEEE IGMP FIP FTP GEC GT GUI HPC HSSM I/O IBM File Transfer Protocol Gigabit Etherchannel Gigatransfers graphical user interface high performance computing high speed switch module input/output International Business Machines identifier Institute of Electrical and Electronics Engineers Internet Group Management Protocol Internet Protocol Internet Protocol Television information technology International Technical Support Organization keyboard video mouse Link Aggregation Control Protocol local area network Lightweight Directory Access Protocol light emitting diode Logical Port VLAN ID long range Large send offload media access control megabyte Multiple Spanning Tree Protocol FC FDX FEC © Copyright IBM Corp. All rights reserved. 2011.

NAS NC-SI NEBS NGN NIC NTP OS OSI OSPF PCI PDF PE PFC POST PXE RADIUS RBAC RIP RMON ROM RSA RSS RSTP SAN SATA SFP SFT SLB SLES SNMP network attached storage Network Controller Sideband Interface Network Equipment Building System next-generation network network interface card Network Time Protocol operating system Open Systems Interconnect Open Shortest Path First Peripheral Component Interconnect Portable Document Format Preinstallation Environment Priority-based Flow Control power-on self test Preboot eXecution Environment Remote Authentication Dial In User Service Role Based Access Control Routing Information Protocol Remote Monitoring read-only memory Remote Supervisor Adapter Receive-side scaling Rapid Spanning Tree Protocol storage area network Serial ATA small form-factor pluggable switch fault tolerance Smart Load Balancing SUSE Linux Enterprise Server Simple Network Management Protocol SOL SR SSH STP TACACS TADDR TCP TCP/IP TFTP TOE TOR UDLD UDP UEFI USB UTP VE VFA VFS VGRP VLAN VM VNIC VOIP VPD VRRP WOL WRR XML XOR Serial over LAN short range Secure Shell Spanning Tree Protocol Terminal Access Controller Access Control System TFTP address Transmission Control Protocol Transmission Control Protocol/Internet Protocol Trivial File Transfer Protocol TCP offload engine top-of-rack UniDirectional link detection user datagram protocol Unified Extensible Firmware Interface universal serial bus unshielded twisted pair Virtualization Engine Virtual Fabric Adapter Virtual Fabric Switch vNIC Group virtual LAN virtual machine virtual network interface card Voice over Internet Protocol vital product data virtual router redundancy protocol Wake on LAN weighted round-robin Extensible Markup Language exclusive or 256 IBM BladeCenter Virtual Fabric Solutions .

TIPS0827 Emulex 10GbE Virtual Fabric Adapter II for IBM BladeCenter. view.com/support/entry/portal/docdisplay?lndocid=MIGR-5080917 Product publications for the Cisco Nexus 4001I Switch Module http://ibm. Web Docs.Related publications The publications listed in this section are considered particularly suitable for a more detailed discussion of the topics covered in this book. Redpapers. 257 . at the following website: ibm. Emulex 10GbE Virtual Fabric Adapter and Virtual Fabric Adapter Advanced for IBM BladeCenter. Note that some publications referenced in this list might be available in softcopy only. TIPS0828 You can search for.com/support/entry/portal/docdisplay?lndocid=MIGR-5082494 Installation Guide for IBM 10 Gb Ethernet Passthru Module http://ibm. draft and additional materials.com/redbooks Other publications These publications are also relevant as further information sources: Publications for the BNT Virtual Fabric 10Gb Switch Module http://ibm.com/support/entry/portal/docdisplay?lndocid=MIGR-5080966 © Copyright IBM Corp. TIPS0748 BNT Virtual Fabric 10Gb Switch Module for IBM BladeCenter. All rights reserved. 2011. IBM Redbooks The following IBM Redbooks publications provide additional information about the topic in this document. TIPS0754 Broadcom 2-port 10Gb Virtual Fabric Adapter for IBM BladeCenter. TIPS0708 Cisco Nexus 4001I Switch Module for IBM BladeCenter. download or order these documents and other Redbooks.

com/products/management-software/device-management /onecommand-manager/ Help from IBM IBM Support and downloads ibm.3.com/support IBM Global Services ibm.0 http://ibm.com/services 258 IBM BladeCenter Virtual Fabric Solutions .Online resources These websites are also relevant as further information sources: BNT Virtual Fabric 10 Gigabit switch module firmware update v6.com/support/entry/portal/docdisplay?lndocid=MIGR-5086594 Cisco support and downloads http://www.emulex.com/tac Emulex OneCommand Manager http://www.5.cisco.

152 creating vNICs 56. 117 stacking 17 transceivers 28. 116 BBI 52. 156 Switch Dashboard 54. 147 command reference 50. 117 shipgroup 29. 153 cables 35. 121 management 33. 135 BladeOS 45.Index Numerics 46C7191 46M6071 49Y4235 49Y4265 49Y4275 49Y9983 81Y3133 90Y3550 90Y3566 28. 117 features 31. 110–111 21. 30. 62 enable vNIC mode 58 examples 81 logon 45. 154 Quick Guide 55. 119 ports 31. 153 BladeOS 45. 119 publications 50 QoS 32. 146 configuring vNICs 15 create vNIC groups 63 creating vNICs 56 DAC cables 30. 147 VLANs 68 vNIC groups 64 BNT Virtual Fabric 10-Gb Switch Module examples 226 isCLI 51. 123 CLI selection 51 configuration tools 37 configuring 44. 119 firmware update 46. 120 SFP+ transceivers 28. 152 Layer 3 functions 33. 259 . 26. All rights reserved. 155 VLANs 69 BladeCenter Open Fabric Manager 7 IBM Virtual Fabric Mode 18 Switch Independent Mode 105. 2011. 123 isCLI privilege levels 51. 148 IEEE standards 34. 120 reset 50 scalability 31. 147 browser-based interface 52. 119 scenarios 226 security 32. 111 21 125 114 21. 14 Switch Independent Mode 103 B bandwidth 1 IBM Virtual Fabric Mode 13 Switch Independent Mode 105–106. 153 creating vNICs 56. 118 direct-attached cables 28. 121 performance 31. 175 A architecture IBM Virtual Fabric Mode 11. 26. 26. 120 BOFM See BladeCenter Open Fabric Manager Broadcom 2-port 10-Gb Virtual Fabric Adapter oversubscription 105 Switch Independent Mode 102 teaming 184 Broadcom 2-port 10Gb Virtual Fabric Adapter 114 bandwidth 145 Bandwidth Weight 163 Broadcom Advanced Control Suite 172. 139 BBI 52. 117 125 21 22. 152 © Copyright IBM Corp. 117–118 VLANs 32. 111 telnet 147 BNT Virtual Fabric 10Gb Switch Module 27. 121 redundancy 32. 61 enable vNIC mode 57 examples 81 logon 53.

25 comparison 27 configuring 38 features 22 firmware 38 minimum bandwidth 18 modes supported 23 UEFI interface 39 vNIC mode in UEFI 41 Emulex 10GbE Virtual Fabric Adapter II 21. 175 browser-based interface 52. 112 firmware 38. 136 IBM Virtual Fabric Mode 37 creating vNICs IBM Virtual Fabric Mode 56 Switch Independent Mode 139. 128 firmware 157 IEEE standards 129 IPv6 support 129 license upgrade 125 performance 126 ports 126 publications 131 scenarios 197 security 128 Software Upgrade License 125 telnet 157 transceivers 125 VLAN support 127 VLANs 157 comparison 5 compatibility 2 components IBM Virtual Fabric Mode 20 Switch Independent Mode 109 concept 2 configuring E elxflash 38. 135 minimum bandwidth 18 modes of operation 26 Multichannel Configuration 42 Switch Independent Mode 102 UEFI 164 UEFI interface 39 vNIC mode in UEFI 41 Emulex OneCommand Manager 74. 115 Global Bandwidth Allocation Menu 145 MAC address maximums 102 MAC addresses 115 Maximum Bandwidth 163 partitions 144 relative bandwidth 145 Broadcom Advanced Control Suite 172.configuring 141 creating vNICs 161 Device Configuration Menu 143 features 105. 146 Broadcom 2-port 10Gb Virtual Fabric Adapter 141 Cisco Nexus 4001I Switch Module 156 Emulex 10GbE Virtual Fabric Adapter II 38. 110 configuring 38 creating vNICs 164 driver 171 features 105. 136 Emulex 10GbE Virtual Fabric Adapter 21. 179 examples BNT Virtual Fabric 10-Gb Switch Module 226 Cisco Nexus 4001I Switch Module 197 IBM Virtual Fabric Mode 81 Switch Independent Mode 193 260 IBM BladeCenter Virtual Fabric Solutions . 144 D double tag scheme IBM Virtual Fabric Mode 13 drivers IBM Virtual Fabric Mode 169 Switch Independent Mode 169 C channels 5 characteristics IBM Virtual Fabric Mode 16 Switch Independent Mode 101 choosing the mode 6 Cisco Nexus 4001I Switch Module 124 BladeCenter Open Fabric Manager 124 cables 130 command reference 160 configuring 156 DAC cables 125 examples 197 FCoE 125. 153 BNT Virtual Fabric 10Gb Switch Module 44.

existing tagging 6 F failover 6 IBM Virtual Fabric Mode 15 Switch Independent Mode 107. 17 teams 73 transceivers 20 trunk failover 15 trunks 18.1q 3. 19 high availability 70 introduced 2 isCLI 51 Layer 2/3 functions disabled 13 LPVID 44 mapping of vNICs 15 minimum bandwidth 18 Multichannel Configuration 43 optics 20 outer tag 12 overview 12 planning 17 products 19 rules 16 scenarios 81 selection 6 SFP+ transceivers 20 stacking 14. 150 H hardware IBM Virtual Fabric Mode 19 Switch Independent Mode 109 high availability IBM Virtual Fabric Mode 70 Switch Independent Mode 169 hypervisor 4 I IBM Virtual Fabric Mode architecture 11. 46 hardware 2. 8 firmware BNT Virtual Fabric 10Gb Switch Module 46. 153 creating vNICs 56. 169 vNIC group creation 63 vNIC groups 18 vNIC mode 13 vSwitches 13 IEEE 802. 14 bandwidth 13 BladeCenter Open Fabric Manager 18 BNT switch configuration 44 BNT Virtual Fabric 10Gb Switch Module 27 characteristics 16 comparison 5 components 20 configuring the solution 37 creating vNICs 56 double tag scheme 13 Emulex 10GbE Virtual Fabric Adapter 21. 69. 152 command reference 52. 152 iSCSI 6 Switch Independent Model 108 Index 261 . configuring 66 VMready 17. 6 VLAN tag 101 isCLI BNT Virtual Fabric 10Gb Switch Module 51. 169 FCoE 6. 70 virtual channels 12 VLAN tagging 12 VLANs. 153 groups 6 gtimg command 49. 25 Emulex 10GbE Virtual Fabric Adapter II 21 enabling vNICs 56 examples 81 failover 15 firmware updates 38. 148 Cisco Nexus 4001I Switch Module 157 Emulex 10GbE Virtual Fabric Adapter II 135 Emulex 10GbE Virtual Fabric Adapters 38 Function 139 G Global Bandwidth Allocation Menu 145 Global Configuration Mode 51. 61 enable vNIC mode 57 examples 81 VLANs 68 vNIC groups 65 isCLI privilege levels 51.

109. 152 products IBM Virtual Fabric Mode 19 Switch Independent Mode 109 R Redbooks website 257 Contact us xii rules IBM Virtual Fabric Mode 16 Switch Independent Mode 108 M mapping of vNICs IBM Virtual Fabric Mode 15 Switch Independent Mode 104 maximum VLANs 6 minimum bandwidth IBM Virtual Fabric Mode 18 Switch Independent Mode 106 modes 2 Multichannel 138 Multichannel Configuration 42 S scenarios BNT Virtual Fabric 10-Gb Switch Module 226 Cisco Nexus 4001I Switch Module 197 IBM Virtual Fabric Mode 81 Switch Independent Mode 193 selection of the mode 6 SFP+ transceivers IBM Virtual Fabric Mode 20 Switch Independent Mode 118. 109 O OneCommand 74 optics IBM Virtual Fabric Mode 20 Switch Independent Mode 118. 125 outer tag IBM Virtual Fabric Mode 12 Switch Independent Mode 101 oversubscription Broadcom 2-port 10-Gb Virtual Fabric Adapter 105 overview 4 IBM Virtual Fabric Mode 12 Switch Independent Mode 103 P partitions Broadcom 2-port 10Gb Virtual Fabric Adapter 144 planning IBM Virtual Fabric Mode 17 262 IBM BladeCenter Virtual Fabric Solutions . 125 stacking 6. 135 BladeOS 45 BNT Virtual Fabric 10Gb Switch Module 109 Broadcom 2-port 10-Gb Virtual Fabric Adapter 102. 141 Broadcom 2-port 10Gb Virtual Fabric Adapter 184 Broadcom Advanced Control Suite 172. 14 IBM Virtual Fabric Mode 17 Switch Independent Mode 108 Switch Independent Mode 101 about 101 architecture 103 bandwidth 105–106. 139 bandwidth limits 101 BladeCenter Open Fabric Manager 105.J jumbo frames 22 L Layer 2/3 functions disabled IBM Virtual Fabric Mode 13 limits 5 Logical Port VLAN ID IBM Virtual Fabric Mode 44 Switch Independent Mode 139 LPVID IBM Virtual Fabric Mode 44 Switch Independent Mode 139 Switch Independent Mode 105 Privileged 51 Privileged EXEC mode 51. 175 Cisco Nexus 4001I Switch Module 110 comparison 5 configuration 136 configuring 133 creating vNICs 160 drivers 169 Emulex 10GbE Virtual Fabric Adapter II 102.

152 V virtual channels 5 virtual groups 6 virtual NIC See vNIC virtualization 1 VLAN groups IBM Virtual Fabric Mode 18 Switch Independent Mode 107 VLAN ID Switch Independent Mode 139 VLAN tagging 2 IBM Virtual Fabric Mode 12 Switch Independent Mode 101 VLANs maximum 6 Switch Independent Mode 167 VMready 69. 169 vNIC2 mode 138 IBM Virtual Fabric Mode 15 Switch Independent Mode 169 trunking 6 trunks IBM Virtual Fabric Mode 18. 169 vNIC overview 4 vNIC groups IBM Virtual Fabric Mode 18 vNIC2 See Switch Independent Mode selecting 138 vSwitches 13 T teaming Broadcom 2-port 10Gb Virtual Fabric Adapter 184 creating 74 Emulex 10GbE Virtual Fabric Adapter II 179 IBM Virtual Fabric Mode 73. 169 IBM Virtual Fabric Mode 17 Switch Independent Mode 108.1Q VLAN tag 101 introduction 3 isCLI 152 iSCSI 108 Logical Port VLAN ID 139 LPVID 139 MAC addresses 115 Multichannel 138 Multichannel Configuration 138 outer tag 101 overview 103 planning 105 products 109 scenarios 193 selection 7 stacking 108 trunk failover 169 UEFI configuration 136 VLAN groups 107 VLAN ID 139 VLAN tagging 101 VLANs 104. 70 U User EXEC mode 51. 179 telnet 45 BNT Virtual Fabric 10-Gb Switch Module 147 Cisco Nexus 4001I Switch Module 157 total ports 4 transceivers IBM Virtual Fabric Mode 20 trunk failover Index 263 . 167 VMready 108. 109 high availability 169 IEEE 802. 169 features 105 firmware 135 firmware updates 148 Function 139 hardware 3.Emulex OneCommand Manager 179 examples 193 failover 107.

264 IBM BladeCenter Virtual Fabric Solutions .

475”<->0.875” 250 <-> 459 pages .IBM BladeCenter Virtual Fabric Solutions (0.5” spine) 0.

.

.

INTERNATIONAL TECHNICAL SUPPORT ORGANIZATION BUILDING TECHNICAL INFORMATION BASED ON PRACTICAL EXPERIENCE IBM Redbooks are developed by the IBM International Technical Support Organization. This book is for clients who want to learn how to implement an IBM Virtual Fabric solution. Customers and Partners from around the world create timely technical information based on realistic scenarios. For example. Specific recommendations are provided to help you implement IT solutions more effectively in your environment. Experts from IBM. and each of these services requires certain bandwidth (or speed) to function properly.com/redbooks SG24-7966-00 ISBN 0738435961 . these traffic flows may interfere with each other. The solution is based on the IBM BladeCenter H chassis with a 10-Gb Converged Enhanced Ethernet infrastructure built on 10-Gb Ethernet switch modules in the chassis and the Emulex or Broadcom Virtual Fabric Adapters in each blade server. every virtualized system can host several dozen network applications and services.Back cover ® IBM BladeCenter Virtual Fabric Solutions ® Covering IBM Virtual Fabric Mode and Switch Independent Mode Concepts and benefits of virtual NICs (vNICs) Step-by-step instructions for creating various vNIC solutions The deployment of server virtualization technologies in data centers requires significant efforts in providing sufficient network I/O bandwidth to satisfy the demand of virtualized applications and services. It provides step-by-step instructions for configuring the adapters and switches. because of different network traffic patterns relevant to different service types. Furthermore. leading to serious network problems including the inability of the service to perform its functions. either the IBM Virtual Fabric Mode offering or the Switch Independent Mode offering. For more information: ibm. The IBM Virtual Fabric solution for IBM BladeCenter addresses these issues.

Sign up to vote on this title
UsefulNot useful