HP Bladesystem C-Class Ethernet Interconnect and Mezzanine Options

HP BladeSystem c-Class
Ethernet Interconnect and

Mezzanine Options
Module 5
he646s b.01
© 2008 Hewlett-Packard Development Company, L.P.

The information contained herein is subject to change without notice
Objectives
• Discuss network convergence technologies
• Describe the Ethernet network interconnect components
– Ethernet interconnect modules
• HP GbE2c Switch
• Cisco Catalyst Blade Switch 3020
• Pass-Thru Module
– Mezzanine cards
• NICs - (Broadcom and Intel)
– Virtual Connect
he646s b.01 © 2008 Hewlett-Packard Development Company, L.P. 2

Fabric convergence
• Data networking, storage, and clustering traffic have been on
different protocols and different network types
Ethernet
• Economical, lower bandwidth, higher latency
• Enabling technologies
– iSCSI for storage
– RDMA/TCP for clustering
InfiniBand
• Higher speed, lower latency, and less CPU load
Fibre Channel
• High speed, low overhead, hardware-intensive, limited distance

Shared I/O
• Using a fabric as the first stage of aggregating the connectivity
and sharing a few I/O devices among many hosts
• Shared I/O fabrics
– InfiniBand switches
– Encapsulated PCI Express switches
– Ethernet switches
– SAS expanders
• Top requirements for shared I/O fabrics
– Lower cost than traditional dedicated I/O per server
– Uncompromised performance
– Ease-of-management for deployment, provisioning, and failover

HP BladeSystem c-Class Ethernet interconnect
categories
• Interconnect switches
– Branded switches from Cisco and Blade Network Technologies
– Pass-Thru Modules
– Virtual Connect
• Mezzanine cards
– Gigabit NIC
– Multifunction NIC

Single-wide switch modules
Inta ke Air
(left or right)
Exha ust Air
Connector JP
E45
Alignment Server Server
E1 BL16 BL09
P8 D45
P7
Connector JS
P5
C
(signal ground)
P3 P2
P1 B45
Server Server
A1 A45
BL08 BL01

Double-wide switch modules
Server BL16 Server BL09
Connector JP2 E45
Connector JP1
P8 Lane-2 & -3 Lane-2 & -3 Lane-0 & -1 Lane-0 & -1 D45
E1
P7
P5 Connector JS2 Connector JS1
C
P3 (signal ground)
P1 P2 Lane-2 & -3 Lane-2 & -3 A1 Lane-0 & -1 Lane-0 & -1
B45
Alignment
A45
Server BL08 Server BL01

Interconnect bays
LAN switch or Pass-Thru LAN switch or Pass-Thru
SAN switch or Pass-Thru SAN switch or Pass-Thru
Rear view of enclosure

Interconnect bay numbering
Interconnect bay Server blade signal
1 NIC 1 and NIC 3

(embedded)
2 NIC 2 and NIC 4

(embedded)
3 and 4 Mezzanine 1
5 and 6 Mezzanine 2
7 and 8 Mezzanine 3

Ethernet
interconnect
modules
he646s
Rev. 6.21
b.01 © 2008 Hewlett-Packard Development
HP Restricted
Company, L.P. 11
GbE2c
Switch
he646s
Rev. 6.21
HP Restricted
Company, L.P. 12
GbE2c Switch
• Designed specifically for
c-Class BladeSystem by Blade
Network Technologies
• Based on HP p-Class GbE2
• Same Layer 2 features as found in
the GbE2 2.X firmware
• Trunking support for
– 16 internal downlink server ports
– Five RJ-45 external uplinks
– Two internal cross-connects
– Port 19 has a dedicated management
connection to the Onboard
Administrator (future release)

GbE2c Switch features
• System security • User accounts that permit limited
• Telnet, SNMP, SCP, TFTP file access to the switch
transfer, human read/write • Fast Uplink Convergence
configuration file, XModem, and • SSH 2
MIBs objects
• LACP
• Industry-standard protocol support
• Configurable Trunk Hash Algorithm
and compatibility
• Secure browser access (HTTPS) to
• 9K jumbo frames
management functions
• Configuration and management
• RSTP and MSTP
from any switch port
• PVST+
• Support for IGMP snooping for
multicasting • RADIUS and TACACS+
• RMON • HTTPS
• QoS • SNMP V1 and V3

GbE2c Switch redundancy
• Redundancy and failover features
– Up to four paired switches within one c-Class enclosure
– Five Gigabit Ethernet uplink ports per switch
– Server networking connections routed to each switch
– Redundant data path 10/100/1000 Ethernet cross connections between the pair
– STP support eliminates potential problems
– Redundant power supplies and cooling fans
– Redundant firmware images and configuration settings
– Redundant, configurable DNS clients, syslog servers, gateways, and community
strings and SNMP trap manager hosts
Redundant crosslinks
• Two switches are connected through redundant 10/100/1000 crosslinks
UFD and NIC teaming
• UFD with NIC teaming enables secondary path for continued access
• When used with UFD, NIC teams on the server blade must be configured for
switch redundancy

GbE2c Layer 2/3 Switch
• Designed specifically for
c-Class BladeSystem by Blade
Network Technologies
• Based on HP p-Class GbE2
• Same Layer 2 features as found in
the GbE2 2.X firmware
• Trunking support for
– 16 internal downlink server ports
– Five RJ-45 external uplinks
– Four optional SFP uplinks
– Two internal cross-connects
(disabled by default)
– Port 19 has a dedicated management
connection to the Onboard
Administrator
Cisco Catalyst
Blade Switch
3020
he646s
Rev. 6.21
HP Restricted
Company, L.P. 19
Catalyst Blade Switch 3020
• Cisco Systems Ethernet switch designed for the c-Class BladeSystem
• Same Layer 2+ features as found in the p-Class CGESM
• Integrated copper uplinks and optional SFPs for fiber support
• Ports
– 16 internal downlinks
– Eight uplinks
• Eight copper uplinks
• Four SFP (fibre) ports can be substituted for four copper ports
• Ports 23 and 24 can also be configured as internal cross connects
– Fa0 is internally connected to the Onboard Administrator for switch management
17x
17 Cisco Catalyst Ethernet Blade Switch 3020 17 21

23x
SYS
T
STA
T
DLX
SPD
MOD 20 20 24
CONSOLE UI 18 24x
E 17 19 20
D 18x

Catalyst Blade Switch 3020 features
• Performance
– Wire speed switching on 16 internal 1Gb ports
– Wire speed switching on 8 external 10/100/1000 ports
– Four external 10/100/1000 SFP based ports
– One external console port
• Management
– CiscoWorks software
– IOS CLI support
– Embedded RMON software agent
• Security
– VLAN –ACLs
– IBNS –802.1x Port Security
– ACS compatibility
Catalyst Blade Switch 3020 software
• QoS parameters
• IOS CLI support for commonality with Cisco routers and switches
• RMON software agent enhances traffic management, monitoring,
and analysis
Firmware
• The switch ships with a Cisco LAN Base IOS image
• IOS updates are available on the Cisco website
CiscoWorks
• Cisco SMARTnet technical support services provide:
– Access to Cisco IOS firmware and software updates
– Web access to technical support tools and repositories
– 24x7 telephone support through the Cisco TAC
– Advance replacement of hardware

Catalyst Blade Switch 3020 front bezel
Item Description Item Description

1 Switch module 6 Health LED
2 Release latch 7 SFP module port LED
3 Switch LED panel 8 SFP module ports for SX fiber
4 Mode button 9 Gigabit Ethernet port LEDs
5 UID LED 10 Gigabit Ethernet RJ-45 ports

InfiniBand
switch

HP 4X DDR InfiniBand Switch Module
• Based on the Mellanox InfiniHost InfiniScale III 4X DDR
InfiniBand switch chip
• The HP 4X DDR IB Switch Module for HP BladeSystem c-Class

provides 24 InfiniBand 4X DDR ports with 20 Gbps port-to-port
connectivity.
• The ports are arranged as 16 downlinks to connect up to 16 blade

servers in the enclosure, and 8 uplinks to connect to the external
InfiniBand switches to build an InfiniBand fabric

HP 4X IB Dual Port Mezzanine HCA
• HP 4X DDR IB Dual-port Mezzanine HCA for HP BladeSystem c-Class is based
on the new ConnectX IB technology from Mellanox
• The IB Dual-port Mezz HCA cards deliver low-latency and high-bandwidth for
performance-driven server and storage clustering applications in High-Performance
Computing (HPC) and enterprise data centers

Ethernet
Pass-Thru
module
he646s
Rev. 6.21
HP Restricted
Company, L.P. 28
Ethernet Pass-Thru Module
• Straight through LAN connectivity
– 1:1 NIC to patch panel connection
– Nonblocking and nonsensing
• Management
– No need for LAN management inside
the enclosure
– Basic status monitored by the
enclosure manager
• Performance
– Full duplex 1Gb Ethernet only
• No auto-negotiation
– Redundant connections from servers
to external switches with a pair of
Pass-Thru modules

Ethernet Pass-Thru Module Health LED
Health LED
Internal health
Status
LED color
Off Normal (not connected to power)
Green Normal (connected to power)
Amber Problem (a fault exists)

Mezzanine
cards and
NICs
he646s
Rev. 6.21
HP Restricted
Company, L.P. 31
BladeSystem c-Class mezzanine cards
• The half-height blade

supports two cards
• The full-height blade

supports three cards

NC326M PCI Express Dual-Port 1Gb Server
Adapter
• Type 1 Mezzanine
– 10/100/1000 NIC – Integrated on the BL480,680, 685
Physical Cable Software Functions

Cluster
Storage
Network

NC373M PCI Express Dual-Port Multifunction
Gigabit Server Adapter
Physical Cable Software Functions Feature Benefit

Cluster
Storage Provides access to storage
Simplified IT
Network over the same wire used for
infrastructure
networking
Removes the need for drives
• Type 1 Mezzanine iSCSI boot
in a server
– Integrated on the BL460,
480, 680 and 685 Provides access to SAN
Accelerated iSCSI
storage over TCP/IP
• Supports
– TOE Performance Frees CPU cycles for other
– iSCSI networking work
– RDMA
RDMA Enhances performance
• One connection, multiple
functions

NC325M PCI Express Quad-Port 1Gb Server
Adapter
– Quad-port - 10/100/1000 NICs
– Compatible with BL460c, BL465c, BL480c, BL680c and BL685c
– Must be installed in mezzanine slot 2 of a half-height server blade to
provide connectivity on all four ports
Physical Cable Software Functions
Cluster
Storage
Network

NC360M PCI Express Dual-Port 1Gb Server
Adapter – Intel Chipset
– 10/100/1000 NIC
– IEEE 802.1p, 802.1Q, 802.3, 802.3ad, and 802.3x
– ProLiant Teaming including Network Fault Tolerance, Transmit
Load Balancing, and Switch-Assisted Load Balancing
– TCP/IP checksum offload (TCO)

NC364M PCI Express Quad-Port 1Gb Server
Adapter – Intel Chipset
– 10/100/1000 NIC
– IEEE 802.1p, 802.1Q, 802.3, 802.3ad, and 802.3x
– ProLiant Teaming including Network Fault Tolerance, Transmit
Load Balancing, and Switch-Assisted Load Balancing
– TCP/IP checksum offload (TCO)

Virtual
Connect
he646s
Rev. 6.21
HP Restricted
Company, L.P. 38
What is Virtual Connect?
• Hardware plus a management agent
• Cable reduction without switch management
• Smart alternative to pass-through modules
• Plugs into interconnect bay
• Ships in Ethernet, Fibre Channel versions
• Allows server replacement and upgrade without showing any
change to the LAN or SAN
• Server changes no longer impact networks, so server, LAN, and
SAN administrators can be more productive

Virtual Connect architecture

server server server server server server
virtualizes the connections from
servers to LANs and SANs
Appears as one server to LAN and SAN Ethernet

Ethernet FC
FC Ethernet
Ethernet
FC
FC
• Streamlines the process of connecting server blades to LANs and SANs

• Enables upgrade, migrate, redeployment, and failover of server blades without
modification to LAN or SAN
• Reduces cables up to 94% without the addition of managed Ethernet or Fibre
Channel switches

Key Virtual Connect components
Ethernet Modules
X1 X2 1 2 3 4
5 6 7 8
HP Bla deSystem Ethernet Module
X1

X2
Bay 1
1
5
2
6
3
7
4
8
X1

X2
Bay 2
1
5
2
6
3
7
4
8
Fibre Channel Modules
Bay 3 Bay 4
1 2 3 4 1 2 3 4 1 2 3 4
HP Fibre Cha nnel Concentra tor Module HP Fibre Cha nnel Concentra tor Module
HP Fibre Cha nnel Concentra tor Module
Bay 5 Bay 6
Bay 7 Bay 8 Virtual Connect Manager

Virtual Connect Ethernet module
Face panel
2 x 10Gb Ethernet (CX4) 8 x 1Gb Ethernet (RJ-45)

for data center links Recessed module usable as data center links or
or stacking links reset button as stacking links
USB port for
Port number and status indicators indicates emergency firmware
Next button
a data center link (green), a stacking link update procedure
(unused in initial
(amber), or a highlighted port (blue) release firmware)

Virtual Connect Fibre module
Face panel

Administering a server failover —
The manual approach
1. Detect failure in A
2. Contact LAN
Server administrator Network administrator
administrator and have a
spare connected to the
same networks as A
A LAN 3. Have any MAC-based
security adjusted to reflect
B new MAC addresses
4. Contact SAN
Storage administrator administrator to allow
C spare to access SAN
volumes on server A
SAN
Spare 5. Migrate boot image from
server A to spare

Administering a server failover with
Virtual Connect
• Makes the changes invisible to the
LAN and SAN administrators ServerAdministrator
Server Administrator NetworkAdministrator
Network Administrator
– No visible change to Ethernet

MAC addresses A
A LAN
HP Virtual Connect
– No visible change to Fibre
Channel WWNs BB
• Reduces number of cables leaving StorageAdministrator
Storage Administrator
CC
the server administrator’s domain
SAN
• Looks like servers to the LAN and
Spare
Spare
SAN administrators

Recommended stacking links
Two Virtual Connect Ethernet modules
Four Virtual Connect Ethernet modules
Six Virtual Connect Ethernet modules

Virtual Connect Ethernet with stacked
enclosures
Server Blade 1 Server Blade 2 Server Blade 64 Virtual Connect

… Manager
pNIC pNIC pNIC pNIC pNIC pNIC
Virtual Connect
for Ethernet
works across
multiple stacked
enclosures
LAN 1 LAN 2 LAN 3 LAN 4
Data center networks

How do VC-Enet uplinks look from the data center
network perspective?
Virtual Connect
Server Blade 1 Server Blade 2 Server Blade 64 Manager (VCM)
…
pNIC pNIC pNIC pNIC pNIC pNIC
Virtual Connect
For Ethernet
works across
multiple stacked
enclosures
?
Does not participate in data center STP
‘LAN-safe’ Provides automatic loop prevention
Allows aggregation of links to data center networks
From the outside, (LACP and fail-over)
Supports VLAN tagging on egress or pass-through of
VC-Enet uplinks look a lot like VLAN tags
regular server connections Supports Link Layer Discovery
Protocol (LLDP)
Introduction to
Fibre Channel
N-port ID Virtualization
(NPIV)
he646s
Rev. 6.21
HP Restricted
Company, L.P. 49
HP Virtual Connect FC Module uses
N_Port_ID Virtualization
Defined by the T11 FC standards
• FC Device Attach (FC-DA) Specification, Section 4.13
• FC Link Services (FC-LS) Specification
“NPIV provides a Fibre Channel facility for assigning multiple N_Port_IDs

to a single N_Port, thereby allowing multiple distinguishable entities.”
Where is it used?
• When implemented within a FC HBA, allows unique WWNs & IDs for each
virtual machine within a server
• Can be used by a transparent HBA aggregator device (like HP VC-FC module)

to reduce cables in a vendor-neutral fashion.

N_Port_ID Virtualization As used
with an HBA aggregator by HP VC-FC
module
1 FLOGI X
ACC X
Server #1
(WWN A) 2a FLOGI A 2b Full
ACC A FDISC A
ACC A FC Switch
Server #2 HBA with
(WWN B) 3a FLOGI B Aggr. 3b NPIV
FDISC B
ACC B
ACC B Support
(WWN X)
Server #3 4b
(WWN C) 4a FLOGI C FDISC C
ACC C ACC C
1. Fabric Login using HBA aggregator’s WWN (WWN X).

Sets up the buffer credits for the overall link.
Receives overall Port ID.
2-4a Server HBA does normal fabric login with their WWNs.
2-4b Server HBA fabric logins translated to ‘FDISC’.
5. Traffic for all 4 N-port IDs carried on the same link.
N_Port_ID Virtualization As used
with an HBA aggregator by HP VC-FC
module
Server #1
(WWN A) A Full
A
FC Switch
Server #2 HBA with
(WWN B) B Aggr. B NPIV
Support
(WWN X)
C
Server #3
(WWN C) C
Uses standard FC HBAs (Qlogic, Once server is logged in, the FC Works with a wide range of
Emulex) frames pass through unchanged! Brocade, McData, or Cisco
No changes to server OS or device Doesn’t interfere with FC Switches
drivers server/SAN compatibility. (with recent firmware)

Virtual Connect FC Module
helps move the SAN fabric boundary
Enclosure
FC SAN Fabric
Standard Server 1 Brocade, Brocade,
Blade SAN Cisco, or
McData
Cisco, or
McData
Server 2
…
…
Configuration SAN Switch SAN Switch
Today Server 3
Brocade, Brocade,
…
Cisco, or Cisco, or
McData McData
…
…
Server 16 SAN Switch SAN Switch
N-Ports F-Ports E-Ports E-Ports
Enclosure
Blade SAN Server 1

Brocade, FC SAN
Cisco, or
Configuration HP VC-FC McData Fabric
…
Server 2 Module SAN Switch
…
with Virtual
Server 3
Connect FC Brocade,
…
HP VC-FC Cisco, or
Module McData
…
…
Server 16 SAN Switch
N-Ports F-Ports N-Ports F-Ports

with NPIV
HP Virtual Connect Profile Migration

Summary
Discussed:
• Network convergence technologies (Fibre Channel, Infiniband, Ethernet –
shared fabric)
• Ethernet network interconnect components
– Ethernet interconnect modules
• HP GbE2c Switch Layer 2 and Layer 2/3
• Cisco Catalyst Blade Switch 3020
– Pass-Thru Module
– Mezzanine cards
• NC326M PCI Express Dual-Port 1Gb Server Adapter
• NC325M PCI Express Quad-Port 1Gb Server Adapter
• NC373M PCI Express Dual-Port Multifunction Gb Server Adapter
• NC360M PCI Express Dual-Port 1Gb Server Adapter
• NC364M PCI Express Quad-Port 1Gb Server Adapter
– Virtual Connect

he646s
Rev. 6.21
HP Restricted
Company, L.P. 56

HP Bladesystem C-Class Ethernet Interconnect and Mezzanine Options

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

HP Bladesystem C-Class Ethernet Interconnect and Mezzanine Options

Uploaded by

Copyright:

Available Formats

HP BladeSystem c-Class

Ethernet Interconnect and

© 2008 Hewlett-Packard Development Company, L.P.

he646s b.01 © 2008 Hewlett-Packard Development Company, L.P. 2

he646s b.01 © 2008 Hewlett-Packard Development Company, L.P. 3

he646s b.01 © 2008 Hewlett-Packard Development Company, L.P. 5

he646s b.01 © 2008 Hewlett-Packard Development Company, L.P. 6

he646s b.01 © 2008 Hewlett-Packard Development Company, L.P. 7

Server BL16 Server BL09

Connector JP2 E45

P5 Connector JS2 Connector JS1

Server BL08 Server BL01

he646s b.01 © 2008 Hewlett-Packard Development Company, L.P. 8

LAN switch or Pass-Thru LAN switch or Pass-Thru

SAN switch or Pass-Thru SAN switch or Pass-Thru

SAN switch or Pass-Thru SAN switch or Pass-Thru

SAN switch or Pass-Thru SAN switch or Pass-Thru

Rear view of enclosure

he646s b.01 © 2008 Hewlett-Packard Development Company, L.P. 9

Interconnect bay Server blade signal

1 NIC 1 and NIC 3

2 NIC 2 and NIC 4

he646s b.01 © 2008 Hewlett-Packard Development Company, L.P. 10

he646s b.01 © 2008 Hewlett-Packard Development Company, L.P. 13

he646s b.01 © 2008 Hewlett-Packard Development Company, L.P. 14

he646s b.01 © 2008 Hewlett-Packard Development Company, L.P. 16

17 Cisco Catalyst Ethernet Blade Switch 3020 17 21

he646s b.01 © 2008 Hewlett-Packard Development Company, L.P. 20

he646s b.01 © 2008 Hewlett-Packard Development Company, L.P. 23

Item Description Item Description

he646s b.01 © 2008 Hewlett-Packard Development Company, L.P. 24

he646s b.01 © 2008 Hewlett-Packard Development Company, L.P. 25

• The HP 4X DDR IB Switch Module for HP BladeSystem c-Class

• The ports are arranged as 16 downlinks to connect up to 16 blade

he646s b.01 © 2008 Hewlett-Packard Development Company, L.P. 26

he646s b.01 © 2008 Hewlett-Packard Development Company, L.P. 27

he646s b.01 © 2008 Hewlett-Packard Development Company, L.P. 29

he646s b.01 © 2008 Hewlett-Packard Development Company, L.P. 30

• The half-height blade

• The full-height blade

he646s b.01 © 2008 Hewlett-Packard Development Company, L.P. 32

Physical Cable Software Functions

he646s b.01 © 2008 Hewlett-Packard Development Company, L.P. 33

Physical Cable Software Functions Feature Benefit

he646s b.01 © 2008 Hewlett-Packard Development Company, L.P. 34

he646s b.01 © 2008 Hewlett-Packard Development Company, L.P. 35

he646s b.01 © 2008 Hewlett-Packard Development Company, L.P. 36

he646s b.01 © 2008 Hewlett-Packard Development Company, L.P. 37

he646s b.01 © 2008 Hewlett-Packard Development Company, L.P. 39

Virtual Connect architecture

Appears as one server to LAN and SAN Ethernet

• Streamlines the process of connecting server blades to LANs and SANs

he646s b.01 © 2008 Hewlett-Packard Development Company, L.P. 40

HP Bla deSystem Ethernet Module

HP Bla deSystem Ethernet Module

HP Bla deSystem Ethernet Module

HP Fibre Cha nnel Concentra tor Module

he646s b.01 © 2008 Hewlett-Packard Development Company, L.P. 41

2 x 10Gb Ethernet (CX4) 8 x 1Gb Ethernet (RJ-45)

he646s b.01 © 2008 Hewlett-Packard Development Company, L.P. 42

he646s b.01 © 2008 Hewlett-Packard Development Company, L.P. 43

he646s b.01 © 2008 Hewlett-Packard Development Company, L.P. 44

– No visible change to Ethernet