2580D KTS I/O

Jaime Calle December 9, 2009

The charts that follow are extracted from this I/O Guidelines document

I/O Configuration Guidelines for 2580D Nodes (Bluefish R710 Platform)

007-0008216 Rev C 11/09/09

Jaime Calle Teradata Client and Platform Engineering Teradata Corporation

2580 Cabinet Components

4 Nodes (two 2-node cliques)
> > > > > > Dell R710 node; 2 2.66 GHz Nehalam CPUs 48GB Memory / node 2 Ethernet ports on node for BYNET 3 Quad 4Gb FC Cards / node 1 Quad copper Ethernet card / node SLES 10 SP1 OS
20
48 Port BYNET/SM Switch 48 Port BYNET/SM Switch

Disk Array 2 Disk Array 3 Disk Array 4 Disk Array 5 Disk Array 6
48 Port BYNET/SM Switch

Clique 1

the 2-clique cabinet 3 Optional TMS in the 1-clique cabinet 2 Optional Channel Servers in the 1-clique cabinet
3 > November 09, 2009 Teradata Confidential

19 18 17 16 15 14 13 12 12 Disk Arrays (one 12-disk tray per DA) 11 > 12 disk tray with built-in dual controllers 10 > 36 disks per node (3 12-disk arrays) 9 8 > SAS 15K 300GB disk drives 7 2 BYNET/SM Ethernet Switches 6 > Two 48 Port Ethernet Switches (Dell 6248) 5 4 1 SMWeb CMIC 1 Optional Teradata Managed Server (TMS) in 3 2
1

Disk Array 6 Disk Array 5 Disk Array 3 Disk Array 2 Disk Array 1 Node 2 Node 1 TMS CMIC Node 1 Node 2 Disk Array 1 Disk Array 4

Clique 2

TMS

TMS
Channel Server Channel Server TMS CMIC Node Node 12 Drive Enclosure 12 Drive Enclosure 12 Drive Enclosure 12 Drive Enclosure 12 Drive Enclosure 12 Drive 48 Port BYNET/SM Switch Enclosure AC box PD AC box U

AC box PD AC box

2-Clique Cabinet

1-Clique Cabinet with Channel and TMS

2580 Node 007-0008216 C

Dell R710 Rear View

1 -Size PCIe slot and 1 -Size PCIe slot on the Center PCI Riser 2 -Size PCIe slots on the Left PCI Riser Slots 1, 3, 4: 9.5 x 4.2 Slot 2: 6.6 x 4.2

Rear View

4 Gb Ethernet ports

4 > November 09, 2009

Teradata Confidential

2580 Node 007-0008216 C

2580D Rear View

Adapter Selection for Harness Cable Notice: HBA Port 0 on the right

Disk Arrays are connected to Adapters according to BIOS slot discovery sequence

A1

A3 A2

2580D Node (Dell)

SM Ports
Left = eth0 Right = eth1

BYNET Ports
Left = eth2 = BYNET 1 Right = eth3 = BYNET 0

BIOS slot discovery sequence: 2, 1, 4, 3

Slots numbered bottom up 5 3 2 1

Adapter Selection for Harness Cable A3 A2 A1

2555 Node

SM Ports

BYNET Ports
Left = eth0 = BYNET 0 Right = eth1 = BYNET 1

BIOS slot discovery sequence: 1, 4, 5, 2, 3

5 > November 09, 2009

Teradata Confidential

2580 Node 007-0008216 C

PCI Bus Structure with PCI Adapters In

Bluefish Server Board Database Node
Nehalem-EP (4 core) CPUs
QUAD CPU QPI
APIC 9 PCI 0

MEMORY

QUAD CPU QPI

MEMORY

ESI x4

0 4

Tylersburg-36D IOH
5 6 7 8 9 10
x4 x4 x4 x4 x4 x4 x4 x8 x8 8 C 10

1 x2 x4 2 3 x4

x2

PCI 1

Enet

SM3G

x4 x1 x1

PCIe SW APIC 8

PCI 2

Enet

BYNET

PCI 3

2xSATA 6xUSB LPC (SIO, CPLD)

ICH9
33 PCI 14

Center Riser
In HBA: PCI 9,A,B PCI 8 S1: Quad FC S2: Quad Enet PCI 4 PCI 10 PCI C

Left Riser
S3: Quad FC S4: Quad FC In HBA: PCI 11,12,13 In HBA: PCI D,E,F

iDRAC
Video

In HBA: PCI 5,6,7 PERC 6i SAS Ctlr

1- Size (9.5), 1- Half Size (6.6) PCIe2 slots

IOH = Input/Output Hub ICH = I/O Controller Hub (South Bridge) iDRAC = integrated Dell Remote Access Controller APIC = Advanced Programmable Interrupt Controller PERC = PowerEdge Expandable RAID Controller
6 > November 09, 2009 Teradata Confidential

2- Size (9.5) PCIe2 slots

2580 Node 007-0008216 C

PCI Slots and On-board Devices

Bluefish Server Board Database Node
PCIe2 x4 PCI 8 PCIe2 x8 PCI 10

R710 Node
Dev 1D Dev 1F Dev 1F F 0-3 Fun 0 Fun 2 PCI 32/33 PCI 0 Dev 0 Dev 0 Dev 0

SLOT 1 Dev 0

Quad FC

SLOT 3 Dev 0

Quad FC
Dev 3

PCIe x4 PCI 3

PCIe x4 PCI 1
Broadcom 5709C Ethernet

PCIe x4 PCI 2
Broadcom 5709C Ethernet

USB

LPC

SATA Ctrl

SAS Ctrl

SLOT 2 Dev 0

Quad Enet
Copper

PCIe2 x4 PCI 4

PCIe2 x8 PCI C

PCI 32/33 PCI 14 Matrox G200e VIDEO

SLOT 4 Dev 0

Quad FC

USB SIO, BMC

Internal SAS Disks

Size slot and Size Slot, Full Height

Size Slots, Full Height

Bluefish Server Board (Dell R710)

SM3G ports

BYNET ports

1 Storage HBA plus 1 Quad Ethernet adapter for client and BAR

2 Storage HBAs

7 > November 09, 2009

Teradata Confidential

2580 Node 007-0008216 C

PCI Slot Assignments

R710 PCIe FS Slots
Slots 1,4,3: Slot 2:

Database Node = R710; Channel Server Node = 5550

Three Storage adapters (quad 4Gb LSI7404E2) Quad Ethernet adapter (1Gb Ethernet PRO/1000 PT Quad) for SM3G connections (eth0, eth1) for BYNET connections (eth2, eth3)
*Rear view of chassis

R710 On-board 1Gb Ethernet/copper Ports (2)

2 ports left* 2 ports right*

5550 PCI-X FS Slots

Slot 1: Slot 2: Slot 3: Slot 4: Slot 5:

FICON (PXFA) or ESCON (PXSA4) adapter 2nd FICON (PXFA) or ESCON (PXSA4) adapter if needed Not used (limit 2 Channel adapters) Option slot (1Gb Ethernet (PRO/1000 PT Quad)) Option slot for SM3G connections

5550 PCIe LP Slots

5550 On-board 1Gb Ethernet/copper Ports (2)

2 ports 2 ports

5550 I/O Module 1Gb Ethernet/copper Ports (2)

for BYNET connections

8 > November 09, 2009

Teradata Confidential

2580 Node 007-0008216 C

2580 Features Supported

PCI adapters supported with the 2580 Node as of the revision date of this guidelines document. Newer PCI adapters may be approved in time.
Standard Adapters
Quad 4Gb FC Gb Ethernet copper quad

Name

LSI7404E2 PRO/1000 PT Quad

Form

PCIe2/HP PCIe/LP

Supplier

LSI Logic Intel

Feature #
9157-F248 9157-F237

Mfg Part #

007-9970566 007-9984264

Channel Adapters for 5550 Channel Servers

ESCON FICON Gb Ethernet copper quad PXSA4/GX PXFA/GX PRO/1000 PT Quad

PCI-X/HP PCI-X/HP PCIe/LP

BTI BTI Intel

9155-F225 9155-F226 9157-F237

007-9970454 007-9970456 007-9984264

System
2580

Class#
9157

TD Part # Class# + Feature#

9157-FXXX

PPL#

315-0601255

TD Part # PPL# + Feature#

315-0601255FXXX

9 > November 09, 2009

Teradata Confidential

2580 Node 007-0008216 C

Disk Array Configuration

For this configuration the ports on the host adapters are as follows:

LSI 7404 Quad Port 4 Gbit/s FC Host Adapter

Clique Diagram Note PCI Slot Numbers

2580D Clique
Quad 4Gb HBAs

2555 Clique
Quad 4Gb HBAs

Node 1
1 4 3

Node 2
1 4 3

Node 1

Node 2
1 2 3

PCI Slots

1 2

1 2

1 2

1 2

1 2

1 2

1 2

1 2

1 2

1 2

1 2

1 2

Array 1
(12 dsks)

Array 2
(12 dsks)

Array 3
(12 dsks)

Array 4
(12 dsks)

Array 5
(12 dsks)

Array 6
(12 dsks)

Array 1
(12 dsks)

Array 2
(12 dsks)

Array 3
(12 dsks)

Array 4
(12 dsks)

Array 5
(12 dsks)

Array 6
(12 dsks)

11 > November 09, 2009

Teradata Confidential

2580 Node 007-0008216 C

Rear View of Chassis

Adapter Selection for Harness Cable Notice: HBA Port 0 on the right

A1

A3
A2

Dell Node

SM Ports
Left = eth0 Right = eth1

BYNET Ports
Left = eth2 = BYNET 1 Right = eth3 = BYNET 0

BIOS slot discovery sequence: 2, 1, 4, 3

Disk Arrays are connected to Adapters according to BIOS slot discovery sequence

FC Port 1 FC Port 2

SAS Expansion Port

FC Port 1 FC Port 2

SAS Expansion Port

Controller A Disk Tray with Controllers

Controller B

12 > November 09, 2009

Teradata Confidential

2580 Node 007-0008216 C

Upper Storage Harness Cable

Cable No. 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 From Chassis 12, Adapter 1, Port 0 Chassis 12, Adapter 1, Port 1 Chassis 12, Adapter 1, Port 2 Chassis 12, Adapter 1, Port 3 Chassis 12, Adapter 2, Port 0 Chassis 12, Adapter 2, Port 1 Chassis 12, Adapter 2, Port 2 Chassis 12, Adapter 2, Port 3 Chassis 12, Adapter 3, Port 0 Chassis 12, Adapter 3, Port 1 Chassis 12, Adapter 3, Port 2 Chassis 12, Adapter 3, Port 3 Chassis 13, Adapter 1, Port 0 Chassis 13, Adapter 1, Port 1 Chassis 13, Adapter 1, Port 2 Chassis 13, Adapter 1, Port 3 Chassis 13, Adapter 2, Port 0 Chassis 13, Adapter 2, Port 1 Chassis 13, Adapter 2, Port 2 Chassis 13, Adapter 2, Port 3 Chassis 13, Adapter 3, Port 0 Chassis 13, Adapter 3, Port 1 Chassis 13, Adapter 3, Port 2 Chassis 13, Adapter 3, Port 3 Label A1-P0 A1-P1 A1-P2 A1-P3 A2-P0 A2-P1 A2-P2 A2-P3 A3-P0 A3-P1 A3-P2 A3-P3 A1-P0 A1-P1 A1-P2 A1-P3 A2-P0 A2-P1 A2-P2 A2-P3 A3-P0 A3-P1 A3-P2 A3-P3 To Chassis 14, Controller A, Port 1 Chassis 15, Controller A, Port 1 Chassis 16, Controller A, Port 1 Chassis 17, Controller A, Port 1 Chassis 18, Controller A, Port 1 Chassis 19, Controller A, Port 1 Chassis 14, Controller B, Port 1 Chassis 15, Controller B, Port 1 Chassis 16, Controller B, Port 1 Chassis 17, Controller B, Port 1 Chassis 18, Controller B, Port 1 Chassis 19, Controller B, Port 1 Chassis 14, Controller A, Port 2 Chassis 15, Controller A, Port 2 Chassis 16, Controller A, Port 2 Chassis 17, Controller A, Port 2 Chassis 18, Controller A, Port 2 Chassis 19, Controller A, Port 2 Chassis 14, Controller B, Port 2 Chassis 15, Controller B, Port 2 Chassis 16, Controller B, Port 2 Chassis 17, Controller B, Port 2 Chassis 18, Controller B, Port 2 Chassis 19, Controller B, Port 2 Label CA-P1 CA-P1 CA-P1 CA-P1 CA-P1 CA-P1 CB-P1 CB-P1 CB-P1 CB-P1 CB-P1 CB-P1 CA-P2 CA-P2 CA-P2 CA-P2 CA-P2 CA-P2 CB-P2 CB-P2 CB-P2 CB-P2 CB-P2 CB-P2 Cabinet Side Middle Middle Middle Middle Middle Middle Middle Middle Middle Middle Middle Middle Middle Middle Middle Middle Middle Middle Middle Middle Middle Middle Middle Middle

19

18 17 16 15 14

13 12

T3 Upper Storage Harness Cable

Note: Adapter 1 is in Slot 1 Adapter 2 is in Slot 4 Adapter 3 is in Slot 3

13 > November 09, 2009 Teradata Confidential 2580 Node 007-0008216 C

Lower Storage Harness Cable

Cable No. 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 From Chassis 9, Adapter 1, Port 0 Chassis 9, Adapter 1, Port 1 Chassis 9, Adapter 1, Port 2 Chassis 9, Adapter 1, Port 3 Chassis 9, Adapter 2, Port 0 Chassis 9, Adapter 2, Port 1 Chassis 9, Adapter 2, Port 2 Chassis 9, Adapter 2, Port 3 Chassis 9, Adapter 3, Port 0 Chassis 9, Adapter 3, Port 1 Chassis 9, Adapter 3, Port 2 Chassis 9, Adapter 3, Port 3 Chassis 8, Adapter 1, Port 0 Chassis 8, Adapter 1, Port 1 Chassis 8, Adapter 1, Port 2 Chassis 8, Adapter 1, Port 3 Chassis 8, Adapter 2, Port 0 Chassis 8, Adapter 2, Port 1 Chassis 8, Adapter 2, Port 2 Chassis 8, Adapter 2, Port 3 Chassis 8, Adapter 3, Port 0 Chassis 8, Adapter 3, Port 1 Chassis 8, Adapter 3, Port 2 Chassis 8, Adapter 3, Port 3 Label A1-P0 A1-P1 A1-P2 A1-P3 A2-P0 A2-P1 A2-P2 A2-P3 A3-P0 A3-P1 A3-P2 A3-P3 A1-P0 A1-P1 A1-P2 A1-P3 A2-P0 A2-P1 A2-P2 A2-P3 A3-P0 A3-P1 A3-P2 A3-P3 To Chassis 7, Controller A, Port 1 Chassis 6, Controller A, Port 1 Chassis 5, Controller A, Port 1 Chassis 4, Controller A, Port 1 Chassis 3, Controller A, Port 1 Chassis 2, Controller A, Port 1 Chassis 7, Controller B, Port 1 Chassis 6, Controller B, Port 1 Chassis 5, Controller B, Port 1 Chassis 4, Controller B, Port 1 Chassis 3, Controller B, Port 1 Chassis 2, Controller B, Port 1 Chassis 7, Controller A, Port 2 Chassis 6, Controller A, Port 2 Chassis 5, Controller A, Port 2 Chassis 4, Controller A, Port 2 Chassis 3, Controller A, Port 2 Chassis 2, Controller A, Port 2 Chassis 7, Controller B, Port 2 Chassis 6, Controller B, Port 2 Chassis 5, Controller B, Port 2 Chassis 4, Controller B, Port 2 Chassis 3, Controller B, Port 2 Chassis 2, Controller B, Port 2 Label CA-P1 CA-P1 CA-P1 CA-P1 CA-P1 CA-P1 CB-P1 CB-P1 CB-P1 CB-P1 CB-P1 CB-P1 CA-P2 CA-P2 CA-P2 CA-P2 CA-P2 CA-P2 CB-P2 CB-P2 CB-P2 CB-P2 CB-P2 CB-P2 Cabinet Side Middle Middle Middle Middle Middle Middle Middle Middle Middle Middle Middle Middle Middle Middle Middle Middle Middle Middle Middle Middle Middle Middle Middle Middle

9 8

7 6 5 4 3

T3 Lower Storage Harness Cable

Note: Adapter 1 is in Slot 1 Adapter 2 is in Slot 4 Adapter 3 is in Slot 3

14 > November 09, 2009 Teradata Confidential 2580 Node 007-0008216 C

Disk Tray Module Front View

12 Drive CRUs

Chassis Summary LEDs Locate Summary Fault Over Temp Power

15 > November 09, 2009 Teradata Confidential

Drive CRU LEDs Service Action Allowed Self ID / Fault In Place / Active

2580 Node 007-0008216 C

Disk Tray Module (cont.)

Drive Slot Numbering

Tray 0, Slot 1

Tray 0, Slot 12

Slots are numbered from left-to-right, top-to-bottom

16 > November 09, 2009

Teradata Confidential

2580 Node 007-0008216 C

LUN Layout

RAID-1 Drive Group Assignment

LUN LUN LUN LUN LUN LUN

0 1 2 3 4 5

Tray Tray Tray Tray Tray Tray

0 0 0 0 0 0

Slot Slot Slot Slot Slot Slot

1, 2, 5, 6, 9, 10,

Tray Tray Tray Tray Tray Tray

0 Slot 0 Slot 0 Slot 0 Slot 0 Slot 0 Slot

3 4 7 8 11 12

17 > November 09, 2009

Teradata Confidential

2580 Node 007-0008216 C

Backup

18 > November 09, 2009

Teradata Confidential

2580 Node 007-0008216 C

PCI Device Addressing (Hex)

Bus # PCI Bus 0
IOH/PCIe Port 0 (bus 0 to ICH10) IOH/PCIe Port 1 (to bus 1) (NIC 1,2) IOH/PCIe Port 2 (paired to port 1) IOH/PCIe Port 3 (to bus 2) (NIC 3,4) IOH/PCIe Port 4 (to bus 3) (SAS ctrl) IOH/PCIe Port 5 (to bus 4) (slot 2) IOH/PCIe Port 6 (to bus 8) (slot 1) IOH/PCIe Port 7 (to bus C) (slot 4) IOH/PCIe Port 8 (paired to port 7) IOH/PCIe Port 9 (to bus 10) (slot 3) IOH/PCIe Port A (paired to port 9) IOH/Host Bridge (disabled) IOH/Host Bridge (disabled) IOH/Host Bridge (disabled) IOH/Host Bridge (disabled) IOH/PIC (8259) IOH/PIC (8259) IOH/PIC (APIC) IOH/System Peripherals 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 2 3 4 5 6 7 8 9 A D D E E 10,11 14 15 16 0 0 0 0 0 0 0 0 0 0 0 0,1,2,3 6,7 0,1,2,3 4 0,1 0,1,2,3 0 0-7 8086 8086 8086 8086 8086 8086 8086 8086 8086 8086 8086 8086 8086 8086 8086 8086 8086 8086 8086 3406 3408 3409 340A 340B 340C 340D 340E 340F 3410 3411 343A-D 341A-B 341C-F 3439 3425-8 342E,22-23,38 342F 3430-33,29-2C

1028 = Dell 0235 = R710 Board Device ID SVID

1028

Device #

Func #

Vendor ID

SID
0235

1028

0235

ICH/USB Controllers ICH/USB 2.0 EHCI Controller ICH/PCI Bridges (to bus X - disabled) ICH/PCI Bridges (to bus X - disabled) ICH/PCI Bridges (to bus X - disabled) ICH/USB Controllers ICH/USB 2.0 EHCI Controller ICH/82801 PCI Bridge (to bus 14) (VGA ctrl) ICH/ISA bridge ICH/IDE bridge ICH/SMBus Controller

0 0 0 0 0 0 0 0 0 0 0

1A 1A 1C 1C 1C 1D 1D 1E 1F 1F 1F

0,1 7 0 4 5 0,1 7 0 0 2 3

8086 8086 8086 8086 8086 8086 8086 8086 8086 8086 8086

2937-8 293C 2940 2948 294A 2934-5 293A 244E 2918 2921 2930

1028 1028

0235 0235

1028 1028 1028 1028 1028

0235 0235 0235 0235 0235

19 > November 09, 2009

Teradata Confidential

2580 Node 007-0008216 C

PCI Device Addressing (Hex)

Bus # PCI Bus 1 (PCIe x4)
Broadcom 6709C Gb Dual Ethernet 1 2 3 0 0 0 0,1 0,1 0 14E4 14E4 1000 1639 1639 0060

1028 = Dell 0235 = R710 Board Device ID SVID

1028 1028 1028

Device #

Func #

Vendor ID

SID
0235 0235 1F0C

PCI Bus 2 (PCIe2 x4)

Broadcom 6709C Gb Dual Ethernet

PCI Bus 3 (PCIe2 x4)

Dell PowerEdge RAID (SAS6i) Controller

PCI Bus 4 (PCIe2 x4)

Slot 2 (half length) 4 (Quad Ethernet card has busses 5,6,7 internal) 0

PCI Bus 8 (PCIe2 x4)

Slot 1 (9.5 long) (Quad FC card has busses 9,A,B internal) 8 0

PCI Bus C (PCIe2 x8)

Slot 4 (Quad FC card has busses D,E,F internal) C 0

PCI Bus 10 (PCIe2 x8)

Slot 3 10 (Quad FC card has busses 11,12,13 internal) 0

PCI Bus 14 (PCI 32/33)

iDRAC 14 14 0 3 0 0 xxxx 102B xxxx 0532 1028 1028 0235 0235

PCI Bus 14 (PCI 32/33)

ATI Rage XL Video

+The PCI bus number will be higher if there are PCI bridged adapters on the lower numbered PCI busses.

20 > November 09, 2009

Teradata Confidential

2580 Node 007-0008216 C

PCI Adapters Information

Host Adapter
LSI7404E2 - PS (IDT switch) - 2devs 2p ea. (949E) PRO/1000 PT 4p (IDT switch) - 2ports (82571EB)

PCI W/F/K Size (LxH)

Vend ID

Dev ID

SVID

SID

Dev#

Fun#

Adapters for R710 Database Nodes

PCIe2 x8 PCIe x8 PCIe x4 PCIe x4 6.6x4.2 ---6.6x2.6 ---111D 1000 111D 8086 806A 0646 8018 10BC ---1000 ---8086 ---1070 ---11BC 2,4 (on 2 int. buses) 0,0 0,1 ea. dev 2,4 (on 2 int. buses) 0,0 0,1 ea. dev

Adapters for 5550 Channel Servers

PXSA4/GX PB (IBM bridge) - PPC (PPC440GX) PXFA/GX PB (IBM bridge) - PPC (PPC440GX) PRO/1000 PT 4p (IDT switch) - 2ports (82571GB) 64/133/3V 64/133 64/133/3V 64/133 PCIe x4 PCIe x4 7.5x4.2 ---7.5x4.2 ---6.6x2.6 ---1014 1014 1014 1014 111D 8086 01A7 027F 01A7 027F 8018 10BC ---1233 ---1233 ---8086 ---0202 ---0203 ---11BC
8 0

8 0 2,4 (on 2 int. buses) 0,0 0,1 ea. dev

Vendor IDs:
1000 LSI Logic 1233 BTI 1014 IBM 12D8 Pericom 101A - NCR 1B3E Teradata 111D Integrated Device Tech 8086 Intel

Legend:
W PCI width F PCI frequency K PCI key CLS Cache line size LxH Board length & height SVID Subsystem Vendor ID SID Subsystem ID Dev - Device Fun - Function

21 > November 09, 2009

Teradata Confidential

2580 Node 007-0008216 C

INTx Interrupt Handling

With PCI Express busses, the native interrupt mechanism is Message Signaled Interrupts (MSI and MSI-X)
> All PCI adapters will use MSI/MSI-X interrupts

The chipset supports emulated INTx type interrupts for legacy support which will be used by any PCI device that do not have MSI/MSI-X enabled See the INTx Interrupt Routing chart a couple of charts later
> PCIe devices that use INTx type interrupts would generate Assert INTx messages over the PCIe bus > In the node, Assert INTx messages are routed to either the I/OxAPIC in the ICH9 chip or the I/O APIC in the IOH chip depending upon if the I/OxAPIC in the IOH chip is disabled or enabled (by BIOS) It should be enabled! > ORing of some Assert INTx messages may occur in the I/OxAPIC in the IOH (see INTx Interrupt Routing chart)

OS support of MSI and MSI-X

> Linux SuSE supports MSI/MSI-X with SLES 10

22 > November 09, 2009

Teradata Confidential

2580 Node 007-0008216 C

INTx Interrupt Rotation

Assert INTx messages can get rotated on the way to the I/OxAPIC by PCIe switches

When Assert INTA rotates it become Assert INTB or Assert INTC or Assert INTD depending on the rotation amount
Similar rotation is followed by Assert INTB, Assert INTC and Assert INTD. Rotation past INTD goes back to INTA INTx rotation occurs in PCI bridges and follow the rotation rules established in the PCI Bridge specification. PCIe switches are represented as 2 levels of bridges and follow the same rules The amount of INTx rotation depends on the device # of the device on the secondary bus of the PCI bridge
> > > > INTx INTx INTx INTx for for for for devices devices devices devices 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, A, B, etc. do not rotate (INTA INTA) etc. rotate by 1 (INTA INTB) etc. rotate by 2 (INTA INTC) etc. rotate by 3 (INTA INTD)

INTx rotation may occur on every PCIe switch on the interrupt path from the originating device to the I/OxAPIC (switches are found within the adapters themselves)
23 > November 09, 2009 Teradata Confidential 2580 Node 007-0008216 C

INTx Routing with IOxAPIC 9 Enabled

Applies only if PCI Device has MSI/MSI-X Disabled
IOH Port

PCI 1
(PCIe x4)

Enet

F0 Assert INTA F1 Assert INTB F0 Assert INTA F1 Assert INTB

Assert INTA Assert INTB Assert INTC Assert INTD Assert INTA Assert INTB Assert INTC Assert INTD Assert INTA Assert INTB Assert INTC Assert INTD Assert INTA Assert INTB Assert INTC Assert INTD Assert INTA Assert INTB Assert INTC Assert INTD Assert INTA Assert INTB Assert INTC Assert INTD

1 3 4

PCI 2
(PCIe x4)

Enet

PCI 3
(PCIe x4)

SAS

PCI 5+
(PCIe x4)

1 2 3 4

Top

* * *

PCI 4
(PCIe x8)

Bottom

PCI 7+
(PCIe x8)

Top

PCI 6+
(PCIe x8)

0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23

Tylersburg IOH

QPI
IOxAPIC 9 MSI messages to the CPUs

Bottom

ICH9

PCI 9+
(32/33)

VGA

PCI 0
15
A B C D E F G H

IOxAPIC 8 MSI messages to the CPUs

Emulated INTx: The Assert INTx messages from PCI devices that do not have MSI/MSI-X enabled go to the IOxAPIC in the Tylersburg IOH. Note that the Assert INTx message can get rotated on the I/O Riser switch on the way to the I/OxAPIC as illustrated above. *If the PCI adapter has an internal PCIe switch, the Assert INTx messages can also get rotated in the adapter switch before leaving the adapter.
24 > November 09, 2009 Teradata Confidential

2580 Node 007-0008216 C

IOxAPIC 8 IRQ Assignments

PIN # INTR NMI IRQ0 IRQ1 IRQ2 IRQ3 IRQ4 IRQ5 IRQ6 IRQ7 IRQ8_L IRQ9 IRQ10 IRQ11 IRQ12 IRQ13 IRQ14 IRQ15 IRQA IRQB IRQC IRQD IRQE IRQF IRQG IRQH SERIAL
25 > November 09, 2009

Description Processor interrupt NMI to processor In APIC mode, 8259 cascade Keyboard In APIC mode, Timer

Real Time Clock (RTC) SCI, TCO, boot interrupt SCI, TCO SCI, TCO Mouse FERR# Logic

IDE, Enet-1, Enet-2 USB-1, USB-3 USB-2 Video

SIO3
Teradata Confidential 2580 Node 007-0008216 C