V2500/V2600 Familiarization Guide: HP 9000 Servers

V2500/V2600 Familiarization Guide
HP 9000 Servers
Final Edition
A5074A A5074-90001 (available via HP-Web) November 1999
Preface
The V2500 was HPs most powerful computer to date utilizing up to 32 of HPs most powerful PA8500 PCXW processor chip. HPs V2500 is mostly an extension to the previous V-Class products performance with few visible changes to existing FRUs. The most notable change is adding a 2nd CPU to each processor board; the most visible is adding 2 more cooling fans to accommodate for the additional heat and for n+1 support. A increase in processor speed for the V-Class product line was then needed for the V2500 product and the V2600 was created. The V2600 is a field and factory upgrade to the V2500 product which itself was a field and factory upgrade to the V2200. Relative to the V2500 product, the V2600 is a processor board upgrade, utilizing the PA8600, 552 MHz, PCXW+ processor chip. It will allow customers to simply replace their V2500 processor board assemblies with V2600 processor board assemblies and will require a newer version of HPUX (11.0, Extension Pack 9905, CD-ROM Date Code of 3917) to be installed. The V2600 product is conceptually similar to the V2250 upgrade for the V2200 product. The major differences are: The V2250 processor board used the same processor as the V2200 processor board, the V2250 was simply clocked to operate at a higher frequency. The V2600 uses a different revision of the PCXW processor with functional differences from the processor used for the V2500.
V2500 and V2600 systems are very similar to the V22x0 system, this Familiarization Guide is mostly review of what you already know and will be use learn what the differences are. How to use this document This document contains five chapters and one appendix to provide training information to all levels of HP field service personnel. For easy references, each chapters material is organized under the following headings: Processor
Memory I/O Mid-plane Power Cooling Mass Storage Service Support Processor (formerly Test Station)
Chapters 1 - 3 contain information primarily for CEs; chapters 4 - 5 contain information that assists troubleshooting and enhances functional knowledge for all (CEs, RCEs, CECs, WTEC). There is one appendix which contains both old and new V-Class server acronym definitions.
Contents
V2500/V2600 H/W Enhancements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Introduction. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Processor board (EWPB) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Memory board (EWMB) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . V-Class Memory Interleaves . . . . . . . . . . . . . . . . . . . . . . . . I/O Subsystem (SIOB) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Mid-Plane (MIB, SCUB, MIBPB) . . . . . . . . . . . . . . . . . . . . . . . Mid-plane Board (MIB) . . . . . . . . . . . . . . . . . . . . . . . . . . . . Utilities Board (SCUB). . . . . . . . . . . . . . . . . . . . . . . . . . . . . Mid-plane Power Board (MIBPB) . . . . . . . . . . . . . . . . . . . . Power Subsystem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Higher capacity +48VDC power supply (NPS) . . . . . . . . . . AC front-end changes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Cooling Subsystem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . N+1 fan system . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Mass Storage. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Service Support Processor . . . . . . . . . . . . . . . . . . . . . . . . . . . . . System Management Hardware . . . . . . . . . . . . . . . . . . . . . . System management Software . . . . . . . . . . . . . . . . . . . . . . . Universal Service Support Processor . . . . . . . . . . . . . . . . . . V2500/V2600 H/W Configurations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Introduction. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Processor Board Configurations . . . . . . . . . . . . . . . . . . . . . . . . . PCXW and PCXW+ Configurations . . . . . . . . . . . . . . . . . . Processor Board Installation . . . . . . . . . . . . . . . . . . . . . . . . . Single CPU in Processor Board Configuration. . . . . . . . . . . Mixed Configuration. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2 CPUs per Processor Board Configuration . . . . . . . . . . . . . Memory Configurations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . V2500/V2600 Memory Configurations . . . . . . . . . . . . . . . . . . . V2500/V2600 DIMM Configuration Rules . . . . . . . . . . . . . Using dcm to Check V2500/V2600 Memory Configuration V2500/V2600 Memory Board Configuration Rules . . . . . . I/O Subsystem Configurations . . . . . . . . . . . . . . . . . . . . . . . . . . Mid-Plane (MIB, SCUB, MIBPB) . . . . . . . . . . . . . . . . . . . . . . . MIB . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
1 1 2 3 3 5 6 6 6 6 7 7 7 8 8 9 10 10 10 10 13 13 14 14 15 15 16 17 19 20 22 23 25 27 28 28
Contents
SCUB . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28 MIBPB . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28 Power Subsystem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29 Cooling Subsystem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30 Mass Storage. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31 Service Support Processor . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32 Miscellaneous . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33 Configuration Limitations/Exclusions . . . . . . . . . . . . . . . . . 33 Upgradability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33 Quality . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34 Actions and Tasks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35 Introduction. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35 Processor Tasks. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36 Task: Deconfiguring Processors using xconfig . . . . . . . . . . 36 Task: Deconfiguring Processors with CPUconfig . . . . . . . . 37 Task: Deconfiguring Processors with SPPDSH . . . . . . . . . . 37 Task: Interpreting POST messages. . . . . . . . . . . . . . . . . . . . 42 Task: Interpreting the LCD Display during POST . . . . . . . . 42 Task: COPMOD changes to a V2500 EWPB & V2600 ELPB 48 Memory Tasks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50 Task: Deconfiguring Memory using xconfig . . . . . . . . . . . . 50 Task: Deconfiguring Memory with SPPDSH. . . . . . . . . . . . 50 SPDSH Commands. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50 SPPDSH Configuration Codes . . . . . . . . . . . . . . . . . . . . . . . 53 Task: Interpreting POST messages. . . . . . . . . . . . . . . . . . . . 59 I/O Subsystem Tasks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63 Task: Determine I/O Hardware Path . . . . . . . . . . . . . . . . . . 63 Mid-Plane (MIB, SCUB, MIBPB) Tasks . . . . . . . . . . . . . . . . . . 64 MIB . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64 SCUB . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64 MIBPB . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64 Power Subsystem Tasks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65 Task: Determining the State of Power Subsystem . . . . . . . . 65 Task: Hot Swap Replacement of Node Power Supplies. . . . 65 Cooling Subsystem Tasks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67 Task: On-Line replacement of Cooling Fans . . . . . . . . . . . . 67
Contents
Mass Storage Tasks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Service Support Processor Tasks . . . . . . . . . . . . . . . . . . . . . . . . Task: do_reset (new features) . . . . . . . . . . . . . . . . . . . . . . . . Task: pce vs. pce_util . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Task: Loading Firmware using PDCFL . . . . . . . . . . . . . . . . Task: Downloading Firmware to SCUB. . . . . . . . . . . . . . . . Troubleshooting Hints . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Introduction. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Some General Thoughts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Processor FRUs. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . EWPB & ELPB . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . PCXW & PCXW+ (or CPU) . . . . . . . . . . . . . . . . . . . . . . . . Mesa . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . CPU3000 Tests . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Executing CPU3000 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Memory FRUs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . V-Class DIMM Slot Identification . . . . . . . . . . . . . . . . . . . . Mesa . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Using the dcm script to troubleshoot a memory error . . . . . . . . MEM3000 Tests . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Executing MEM3000 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . I/O Subsystem FRUs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . SIOB . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . IO3000 Tests. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Executing IO3000 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Power Subsystem FRUs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . +48VDC Power Supplies . . . . . . . . . . . . . . . . . . . . . . . . . . . Cooling Subsystem FRUs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . N+1 Cooling Fan Panels . . . . . . . . . . . . . . . . . . . . . . . . . . . . Mass Storage FRUs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . PCI Cards . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . SCSI Devices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Service Support Processor Utilities . . . . . . . . . . . . . . . . . . . . . . CCMD Differences . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . EST Differences . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Hard_Logger Master ID Differences . . . . . . . . . . . . . . . . . .
68 69 69 70 71 73 75 75 76 77 77 77 77 78 79 81 81 83 84 85 88 90 90 90 92 97 97 98 98 99 99 99 100 100 101 102
Contents
Theory of Operation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Introduction. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Processor Modules . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . PA8500/PA8600 Adaptations for V2500/V2600 . . . . . . . . . Runway Bus Arbitration . . . . . . . . . . . . . . . . . . . . . . . . . . . . Memory Modules . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . V-Class Memory Addressing Formats . . . . . . . . . . . . . . . . . V22x0 Memory Interleaving . . . . . . . . . . . . . . . . . . . . . . . . V2500/V2600 Memory Interleaving . . . . . . . . . . . . . . . . . . Commercial vs. Technical Systems . . . . . . . . . . . . . . . . . . . I/O Modules . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Key Points. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Mid-Plane . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Functional review of main computing FRUs . . . . . . . . . . . . Power Modules . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Processor module power supply . . . . . . . . . . . . . . . . . . . . . . Cooling Modules. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Processor module thermal. . . . . . . . . . . . . . . . . . . . . . . . . . . Mass Storage Modules . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Service Support Processor . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Universal SSP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Acronyms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Introduction. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . V22x0 Acronym List . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . V2500/V2600 Acronym List . . . . . . . . . . . . . . . . . . . . . . . . . . .
105 105 108 108 109 110 111 114 115 116 117 117 119 119 120 120 121 121 122 123 123 125 125 126 130
List of Figures
EWPB with Two PCXW CPUs and their Power Supplies ........................2 SIOB Card Cage Cover for up to Seven PCI Slots....................................5 CPU Locations on V2500/V2600 Processor Board...................................14 DIMM Locations on V2500/V2600 EWMB .............................................22 Example dcm Script -- partial listing .........................................................24 DIMM Locations for Example dcm Script ................................................25 Xconfig SSP (or Test Station) Screen........................................................36 Menu Mode command CPUconfig ........................................................37 Example clist command.............................................................................39 Example cget command.............................................................................40 Example cput command.............................................................................41 POST processor messages .........................................................................42 LCD Display during POST........................................................................43 LCD Display ..............................................................................................47 COPMOD example....................................................................................49 Example clist command.............................................................................51 Example cget command.............................................................................52 Example query command ..........................................................................56 Example cput command.............................................................................57 Example toggle command .........................................................................57 POST messages from a V2500/V2600 system with 32 GB of memory....60 POST mapping of Quadrants, Buses and Rows ........................................60 POST messages from a V2500/V2600 system with 16 GB of memory....61 POST mapping of Quadrants, Buses and Rows ........................................61 Physical EWMB Layout ............................................................................62 SIOB Card Cage .......................................................................................63 Steps performed when hot swapping a nps................................................66 Example output from the pce command ....................................................70 Command line interface.............................................................................80 V2500/V2600 EWMB DIMM Slot Groupings: ........................................83 Example V2500/V2600 dcm script, partial listing ....................................85 Executing MEM3000 from the command line interface ...........................89 Parameter Words 8 through 19 ..................................................................94 Executing IO3000 from the command line interface.................................95 V22x0 Processing Module pair and EPAC................................................103 V2500/V2600 Processing Module pairs and SPAC ..................................103
V22x0 System Block Diagram ..................................................................106 V2500/V2600 System Block Diagram ......................................................107 Runway Bus Signal Sharing between PCXW CPUs .................................108 V22x0 Coherent Memory Address Modification ......................................112 V2500/V2600 Coherent Memory Address Modification ..........................113 V22x0 Memory Block Diagram ................................................................114 V2500/V2600 Memory Block Diagram ....................................................115
1)
V2500/V2600 H/W Enhancements

Introduction
The V2500/V2600 server contains the next generation of updated hardware compatible with the existing V22x0 platform. This consists of enhancements to each of these major subsystems in the platform: processor, memory and I/O. The mechanical design for V2500/V2600 is virtually identical to V22x0 except for addition of a card slot in the I/O card cage. Changes in some areas of the basic infrastructure hardware were required, including midplane, utilities board, power and thermal subsystems. These enhancements are bundled together to comprise the new V2500/V2600 server. Key Points Two keys to improving performance are increase parallelisms through twice as many CPUs and memory throughput through more banks. In total, these enhancements are intended for both new server deliveries and field upgrades to existing V22x0 servers. When upgrading a V22X0 server to V2500/V2600 functionality, all enhancements must be implemented. Separate configurability of one or more of the enhancements is not supported. See Chapter 5 Theory of Operations for details on improvements to parallelism and memory interleaving.
HP Company Confidential
V2500 H/W Enhancements Processor board (EWPB/ELPB)
Processor board (EWPB/ELPB)

The V2500 platform includes the new PA8500 PCXW CPU (processor chip) on the EWPB processor board. The V2600 platform includes the new PA8600 PCXW+ CPU (processor chip) on the ELPB processor board. Each processor board contains up to two CPUs (processor chips) interconnected to the same Runway bus. With two CPUs per processor board, single cabinet configurations of up to 32 CPUs are supported. The CPU for the V2500 operates at 440MHz and the CPU for the V2600 operates at 552 MHZ, while the Runway bus frequency remains at 120MHz.
Figure 1 EWPB with two PCXW CPUs and their Power Supplies
PBnR
DC to DC Power Supply DC to DC Power Supply PCXW CPU A PCXW CPU B DC to DC Power Supply DC to DC Power Supply PCXW CPU A PCXW CPU B
0
R U N W A Y
from X-Bar PnL Agent to X-Bar
PBnL
to SAGA
V25040.ppt
Key Points Both Instruction and Data Caches were moved from external to internal Caches within the PCXW and PCXW+. This makes room for installing one or two CPUs on a processor board. There is 0.5 MB of parity-protected I-Cache and 1.0 MB of ECC-protected D-Cache inside each PCXW or PCXW+ CPU chip. PCXU+ chips used in V22x0 EVPBs have 4.0 MB of external parityprotected Level 1 Cache. SAGA is an enhancement of the EPIC chip; more details are forthcoming later in this chapter and in later chapters.
2 HP Company Confidential
V2500 H/W Enhancements Memory board (EWMB)
Memory board (EWMB)

The V2500/V2600 memory board visibly looks just like the V22x0 EMB but consists of new memory board EWMB, new memory controller ASIC SMAC plus a new NBC chip that multiplexes four DIMMs to the SMAC, and a modified CTI controller ASIC (STAC). The new memory controller (SMAC) provides higher memory bandwidth by accessing more memory banks in parallel. This feature required substantial design changes to the layout of the memory board, but preserved compatibility with existing 32MB and 128MB DIMMs. CTI controller (STAC) changes were generally limited to those required for compatibility with the (SMAC) memory controller. Key Points Existing 32MB and 128MB DIMM types, and a new 256MB DIMM are supported. (There is no longer a restriction by HP-UX when only using 32MB DIMMs as contiguous memory is greater than 32MB.) The 256MB DIMM is only supported on V2500/V2600 servers. Use of this DIMM allows per-cabinet memory configurations up to 32GB.
V-Class Memory Interleaves

This section compares V22x0 and V2500/V2600 memory interleaving options. V22x0 Interleaving Options V22x0s memory boards (or EMBs) each support four banks; with up to four DIMMs per bank. Each EMB configuration allows 4/8/12/16 DIMMs per board; 4 banks per EMB provide 8/16/32 way interleaving.
V2500 H/W Enhancements Memory board (EWMB)
Sequential memory addresses are interleaved across 1/2/4 pairs of memory boards and their banks yielding the following memory interleaves: 4 DIMMs provide 8/16/32 way interleaving 8 DIMMs provide 8/16/32 way interleaving 12 DIMMs provide 8/16/32 way interleaving 16 DIMMs provide 8/16/32 way interleaving
NOTE Mixing DIMM sizes has no performance degradation with this interleaving format. Adding more or larger DIMMs per bank will increase performance.
V2500/V2600 Interleaving Options V2500/V2600 memory boards (or EWMBs) each support up to 32 banks; using eight individual buses with up to two DIMMs per bus. EWMB configurations also allow 4/8/12/16 DIMMs per board. Sequential memory addresses are interleaved across 1/2/4 pairs of memory boards, their buses and banks yielding the following memory interleaves: 4 same size DIMMs provide 16/32/64 way interleaving 8 same size DIMMs provide 32/64/128 way interleaving 12 same size DIMMs provide 32/64/128 way interleaving 16 same size DIMMs provide 64/128/256 way interleaving
NOTE Mixing DIMM sizes has performance degradation with this interleaving format. Adding larger DIMMs increases performance; adding DIMMS to every bus increases both interleaving factor and performance.
V2500 H/W Enhancements I/O Subsystem (SIOB)
I/O Subsystem (SIOB)

The V2500/V2600 I/O subsystem features a new I/O card cage supporting both 32/64-bit PCI I/O buses. A new ASIC (SAGA) replaces the EPIC and a new I/O backplane design provides the 64-bit PCI I/O adapter functionality while preserving compatibility with the current 32-bit PCI programming model. Key Points Shared I/O memory is expanded from 192K to 448K per I/O module. The new I/O chassis features one additional I/O controller card slot, for a total of 7 PCI slots in the chassis. SAGA supports +5V PCI signalling, providing compatibility with I/O controllers from V22x0. A new EMI cover for the I/O card cage allows more clearance for the cables due to the additional card slot.
Figure 2 SIOB Card Cage Cover for up to Seven PCI Slots
SAGA (B)
SAGA (A)
N E W 0
V25055.ppt
See Chapter 5 Theory of Operations for details of SAGA (A) and SAGA (B).
V2500 H/W Enhancements Mid-Plane (MIB, SCUB, MIBPB)
Mid-Plane (MIB, SCUB, MIBPB)

Mid-plane Board (MIB)
The V2500/V2600 mid-plane remains visually the same as the ENRB used by the V22x0 except that SPAC ASICs replace EPAC ASICs; ERAC ASICs have not changed for V2500/V2600, however. Some minor changes to processor module (EWPB or ELPB) interconnect, clock distribution, and other functions made necessary by the processor, memory and I/O improvements.
Utilities Board (SCUB)

The V2500/V2600 Utilities board includes increased (NVRAM) and (SRAM) storage for firmware additions required for the increased processor population. A partial list of (SCUB) changes, compared to the V22x0 (ECUB) utilities board includes: SRAM expanded from 128k to 256k NVRAM expanded from 32k to 128k 32 CPU support in CPU Report CSR in SMUC 32 CPU support in Semaphore CSR in SMUC 4 new resource semaphores in the SMUC Support of n+1 redundant fan operation and graceful shutdown
Mid-plane Power Board (MIBPB)

No functional changes between V22x0 and V2500/V2600.
V2500 H/W Enhancements Power Subsystem
Power Subsystem
Some V2500/V2600 power system changes include redesign of the processor module dc-dc converter power supply, development of a 2400 Watt +48VDC power supply (NPS), and minor changes to the ac front end. Current V22x0 servers employ three 2000 Watts front end supplies to generate the internal +48VDC distribution. An optional fourth supply provides n+1 power supply redundancy.
Higher capacity +48VDC power supply (NPS)

A 2400 Watts supply, in a form factor that is compatible with new and existing chassis, was developed as initial power calculations indicated that three 2000 Watt power supplies did not satisfy the +48VDC demand for a fully configured (32 processors, 32GB memory, internal peripherals) V2500/V2600 server. All V2500/V2600 servers employ four 2400 Watt front end power supplies to generate the internal +48VDC distribution. The fourth supply provides n+1 power supply redundancy.
AC front-end changes
Although both the V22x0 and V2500/V2600 incorporate modifications to support a 60 Amp AC input service, they both require the same 50 Amp AC input presently used by V22x0 servers today.
V2500 H/W Enhancements Cooling Subsystem
Cooling Subsystem
The V2500/V2600 cooling system features a redesigned processor heat pipe on each CPU, and increased airflow to cabinet FRUs with the deployment of six (instead of four) cooling fans. The addition of two more fans and mechanical interlocks enables the n+1 feature that supports (and encourages) on-line fan replacement.
N+1 fan system

A new fan panel design provides n+1 fan capability. This fan panel increases the number of fans in the server from 4 to 6, and implements features permitting power-down and replacement of individual fans without interrupting operation of the server. Key Points Operation with a failed fan will be supported for a finite duration sufficient to schedule a service call. Indefinite operation with a failed fan is not recommended due to long-term reliability degradation. The 6-fan panel is the only fan configuration supported in the V2500/ V2600 cooling subsystem.
V2500 H/W Enhancements Mass Storage
Mass Storage
Two new internally-mountable (or embedded) product offerings on V2500/V2600 servers include the following: New embedded DVD-ROM replaces the CD-ROM (sometime after SR). New Low Voltage Differential (LVD) type of 18GB Internal Disks. Key Points The new LVD type disks are ULTRA 2-SCSI and cannot be mixed on the same SCSI bus with earlier HVD type ULTRA-SCSI disks supported by V22x0 servers. All drives within the same disk tray must be all LVD type (requiring A5149A PCI SCSI card) or all HVD type (requiring A4800A PCI SCSI card). All LVD drives are labelled to indicate this difference.
V2500 H/W Enhancements Service Support Processor
Service Support Processor

System Management Hardware
A faster Service Support Processor (SSP or Test Station), HP model # B180L with more memory, is provided with new V2500/V2600 servers. Key Points Upgrades to V2500/V2600 keep their existing SSP and just update their software for V2500/V2600.
System management Software

The version of HP-UX supported on the SSP may be 10.20 or 11.x; in any case, the format used to display the V2500/V2600 screens now uses CDE instead of X.11 used with V22x0 screens. There are three distinct V2500/V2600 configurations offered to customers:
Individual V2500/V2600 with service support processor; Multiple V2500/V2600 single nodes sharing the same service support processor; Multiple V2500/V2600 single/multi nodes sharing the same service support processor.

All of the above configurations share the same System Management Software, utilities, user interface, and Firmware. See the Service Support Processor Installation and Service Guide for a description of these features and for examples of SSP connections that support two of the three configurations.
10
Additional Security All B180L SSP include a utility called xsecure that provides added security for customers that are hesitant about remote support.
NOTE
Syntax & man page information about the X-based security manager xsecure:
xsecure [-on | -off | -check] | [-display display_device_name]
xsecure may be run as a script or as an X-based program. It is used to separate the test station (SSP) from any modems or external ethernet buses in order to provide additional security into the system. In the secure mode, all network LANs other than the tsdart bus are disabled and the optional modem on the second serial port will be disabled. When in normal mode, all networks and modems are re-enabled. If the [-on | -off | -check] options are used, xsecure does not use the GUI interface and instead remains test based. These options allow the user to turn the secure mode on, off or allow the user to check the secure mode status. A simple button with a red or green secure mode indicator provides the user with secure mode status information. The red indicator shows that the SSP is secure. A green indicator shows that the network is available and the SSP may be accessed through the ethernet port. In order for xsecure to work properly, the SSP, console cables, terminal mux and modems must be configured in specific ways. The SSPs JTAG connections, OBP connections and an optional terminal mux must be connected to lan0 and identified in the /etc/hosts file as tsdart-d. The sppconsole serial cable from node 0 must be connected to SSPs serial port 0. An optional modem may be connected to SSPs serial port 1. If a modem is connected directly to the nodes utility board, it remains unsecure.
11
12
2)
V2500/V2600 H/W Configurations

Introduction
Hardware configuration guidelines are quite similar to those for V2200, except as follows:
New processor count configuration rules are necessary due to two processors per processor module and differing module requirements. New DIMM positioning configurations. Additional DIMM configuration rules for the new 256MB DIMM module. Maximum I/O controller configurations have increased by one controller per I/O chassis, due to the additional card slot.
See Chapter 5 Theory of Operations for details on balanced configurations through board slot and DIMM slot choices.
13
V2500 H/W Configurations Processor Board Configurations
Processor Board Configurations

The processor board for a V2500 or V2600 can contain up to two CPUs. Upgrade of a single-processor board to a dual-processor board will be supported through the addition of a second socketed processor to the board.
PCXW and PCXW+ Configurations

The following figure defines the physical locations of CPU A and CPU B on a processor board
Figure 3 CPU Locations on V2500/V2600 Processor Board
Power Brick for CPU A
CPU A
C O N N E C T O R
Power Brick for CPU B
CPU B
14
Processor Board Installation

When installing additional processor boards it is important to install them in the proper order for optimum performance. The processor boards will be populated with one CPU per board for the first 16. Additional CPUs will be installed on each board to create dual CPU boards in the same order as shown in Table 2.
NOTE The orders are recommended for optimum performance. The system will still operate if this order is not followed.
Single CPU in Processor Board Configuration

Guidelines for a single CPU per processor board configurations:
The cabinet may have 2 through 16 CPUs There may be 2 through 16 processor boards Every processor board must have exactly 1 CPU in socket A
Table 1 shows the order that processors with a single CPU should be added to the system.
Table 1 Single CPU per Processor Board Installation Sequence Order 1st 2nd 3rd 4th 5th 6th 7th 8th Slot Name PB0L PB4L PB1R PB5R PB2R PB6R PB3L PB7L Menu Mode Proc. ID 0 8 2 10 5 13 7 15 A A A A A A A A CPU Location
15
Order 9th 10th 11th 12th 13th 14th 15th 16th
Slot Name PB0R PB4R PB1L PB5L PB2L PB6L PB3R PB7R
Menu Mode Proc. ID 1 9 3 11 4 12 6 14 A A A A A A A A
CPU Location
Mixed Configuration
Guidelines for the mixed configuration of a single CPU per processor board and 2 CPUs per processor board configurations:
The cabinet may have 17 through 32 CPUs There must be exactly 16 processor boards Every processor board must have a minimum of 1 CPU
Table 2 shows the order that processors of a mixed configuration should be added to the system.
Table 2 Mixed Processor Board Installation Sequence Order 17th 18th 19th 20th Slot Name PB0L PB4L PB1R PB5R Menu Mode Proc. ID 16 24 18 26 B B B B CPU Location
16
Order 21st 22nd 23rd 24th 25th 26th 27th 28th 29th 30th 31th 32nd
Slot Name PB2R PB6R PB3L PB7L PB0R PB4R PB1L PB5L PB2L PB6L PB3R PB7R
Menu Mode Proc. ID 21 29 23 31 17 25 19 27 20 28 22 30 B B B B B B B B B B B B
CPU Location
2 CPUs per Processor Board Configuration

Guidelines for 2 CPUs per processor board configurations:
The cabinet may have 2 through 32 CPUs There may be 1 through 16 processor boards Every processor board must have exactly 2 CPU
Table 3 shows the order that processors with a 2 CPUs should be added to the system.
17
V2500 H/W Configurations Processor Board Configurations Table 3 Dual CPUs per Processor Board Installation Sequence Order 1st & 2nd 3rd & 4th 5th & 6th 7th & 8th 9th & 10th 11th & 12th 13th & 14th 15th & 16th 17th & 18th 19th & 20th 21st & 22nd 23th & 24th 25th & 26th 27th & 28th 29th & 30th 31th & 32nd Slot Name PB0L PB4L PB1R PB5R PB2R PB6R PB3L PB7L PB0R PB4R PB1L PB5L PB2L PB6L PB3R PB7R Menu Mode Proc. ID 0 & 16 8 & 24 2 & 18 10 & 26 5 & 21 13 & 29 7 & 23 15 & 31 1 & 17 9 & 25 3 & 19 11 & 27 4 & 20 12 & 28 6 & 22 14 & 30 CPU Location A&B A&B A&B A&B A&B A&B A&B A&B A&B A&B A&B A&B A&B A&B A&B A&B
18
V2500 H/W Configurations Memory Configurations
Memory Configurations
As with the V22x0, memory board configurations are limited to 2, 4 or 8 memory boards in a single cabinet system (multi-cabinet configurations require all 8 memory boards in each cabinet). These configurations may be selected independent of the number of processors present in the cabinet. DIMM configurations are possible with 1, 2, 3, or 4 sets of DIMMs populated per memory board. One set of DIMMS, consists of 4 DIMM modules that are identical in size. All memory boards must be populated identically.
NOTE Mixing of 32MB and 256MB DIMM module configurations are not supported in production configurations. 32MB DIMMs provide lower performance than the other DIMM modules.
The V2500/V2600 memory configuration is different from the existing V22x0 memory configuration in several ways. This section discusses those differences and how to check your current configuration.
19
V2500 H/W Configurations V2500/V2600 Memory Configurations
V2500/V2600 Memory Configurations

In the V2500/V2600 server, DIMMs are installed on Exemplar WBased Memory Boards (EWMBs). For the remainder of this document the EWMB will be referred to as the memory board. This section contains information about DIMMs and their positioning on memory boards. A V2500/V2600 memory board is organized by quadrants, rows, and buses. Each memory board has 4 quadrants, 4 rows and 8 buses. The following terms are used to describe a V2500/V2600 memory board, as shown in Table 4, 5 and Figure 4:
Slot The connectors location into which DIMMs are installed. There are 16 DIMM slots, each with a unique designator which denotes the slots quadrant and bus. A group of four DIMM slots staggered across the memory board. 8 buses span the four rows. Each DIMM within a quadrant is on a different bus. Each DIMM has SDRAMs on each side and represents two rows. For instance, the first DIMM installed in the system would represent row 0 bus 0 and row 1 bus 0. All DIMMs have the same SDRAMs on both sides. Therefore, rows 0 and 1 will have the same SDRAM size. Rows 2 and 3 will have the same SDRAM size. Bus interleaving can be configured to either 4 way or 8 way bus interleaving. 8 way provides the best performance. To achieve 8 way bus interleaving, all buses on a row must be populated with DIMMs having the same SDRAM size.
Quadrant Buses Rows
Table 4 shows the correlation between a DIMM slot and a row bus intersection. The first DIMM to be installed in a memory board, Q0B0, occupies row 0 bus 0 and row 1 bus 0 in quadrant 0.
20
V2500 H/W Configurations V2500/V2600 Memory Configurations Table 4 DIMM row/bus Relationships Rows 0 0 1 2 3 Q2B0 Q2B1 Q2B2 Q2B3 Q3B4 Q3B5 Q3B6 Q3B7 Q0B0 1 Q0B1 2 Q0B2 3 Q0B3 Buses 4 Q1B4 5 Q1B5 6 Q1B6 7 Q1B7
V2500/V2600 DIMM Quadrant Designations

Memory boards can be populated in increments of 4 DIMMs called quadrants. A quadrant is a group of four DIMM slots staggered across the memory board. Quadrants are similar to banks on the V22x0.
4 DIMMS provides 1/4 population 8 DIMMS provides 1/2 population 12 DIMMS provides 3/4 population 16 DIMMS provides full population
Table 5 shows the rows and buses associated with each quadrant (0-3) and Figure 4 shows how they are laid out on the memory board.
Table 5 Quadrant Assignments Rows 0 0 1 2 3 Quadrant 2 Quadrant 3 1 2 3 Buses 4 5 6 7
Quadrant 0
Quadrant 1
21
V2500 H/W Configurations V2500/V2600 Memory Configurations Figure 4 DIMM Locations on V2500/V2600 EWMB
Example:
Q2B3: Quadrant 2, Bus 3
V2500/V2600 DIMM Configuration Rules

Use the following rules to plan the memory board DIMM configuration:
All memory boards must be populated identically. Single cabinet memory boards may be populated in 1/4, 1/2, 3/4, or full increments. Multi cabinet memory boards may be populated in only 1/4, 1/2, or full increments. All DIMMs within a quadrant must be of the same size: 32 Mbyte, 128 Mbyte or 256 Mbyte.
22
V2500 H/W Configurations V2500/V2600 Memory Configurations DIMMs in quadrant 0 can be of a different size than DIMMs in quadrant 2 or 3 without degrading performance. DIMMs in quadrant 1 can be of a different size than DIMMs in quadrant 2 or 3 without degrading performance. DIMMS in quadrant 0 and 1 should be the same size for maximum performance. DIMMS in quadrant 2 and 3 should be the same size for maximum performance. DIMMs in quadrant 0 can be of a different size than DIMMs in quadrant 1. To allow this memory to be fully utilized, the bus interleave span will be reduced to 4 way bus interleaving. This will degrade performance. DIMMs in quadrant 2 can be of a different size than DIMMs in quadrant 3. To allow this memory to be fully utilized, the bus interleave span will be reduced to 4 way bus interleaving. This will degrade performance. Mixing of 32 Mbyte DIMMS and 256 Mbyte DIMMs is not supported. All quadrants on a given memory board do not have to be populated with DIMMs.
NOTE
Appendix A of V2500/V2600 Upgrade Guide contains diagrams of all possible DIMM configurations supported by the V2500/V2600.
Using dcm to Check V2500/V2600 Memory Configuration

Use the dcm script to check the existing memory configuration. Redirect the output of the dcm script to a file and print it out for future reference. Select the tsh window and enter:
dcm # > filename
where # is the cabinet number and filename is where you want to redirect the dcm scripts output. If there is only one cabinet, # should be 0. Print out the dcm script output. Enter:
lp filename
23
Figure 5 is an example of the part of a dcm script output showing the memory configuration of a system.
Figure 5 Example dcm Script -- partial listing
Memory: ======= Physical: L=128MB, M=64MB, S=16MB Logical: l=128MB, m=64MB, s=16MB
(If logical memory not specified, then it matches physical memory size)
* = Software Deconfigured EWMB0: ====== EWMB0: EWMB0: EWMB0: EWMB0: EWMB1: ====== EWMB1: EWMB1: EWMB1: EWMB1:
- = Not In Use
Q0B0 Q0B1 Q0B2 Q0B3
L/L L/L L/L L/L
Q1B4 Q1B5 Q1B6 Q1B7
L/L L/L L/L L/L
Q2B0 Q2B1 Q2B2 Q2B3
-/-/-/-/-
Q3B4 Q3B5 Q3B6 Q3B7
-/-/-/-/-
Q0B0 Q0B1 Q0B2 Q0B3
L/L L/L L/L L/L
Q1B4 Q1B5 Q1B6 Q1B7
L/L L/L L/L L/L Row 0 / Row 1
Q2B0 Q2B1 Q2B2 Q2B3
-/-/-/-/Row 2 / Row 3
Q3B4 Q3B5 Q3B6 Q3B7
-/-/-/-/Row 2 / Row 3
Row 0 / Row 1 Memory Board Quadrant # Bus #
Quadrant # Bus #
Quadrant # Bus #
Quadrant # Bus #
In the example dcm script above, a partial listing of the physical and logical memory information codes are defined. Physical memory represents memory that was not hardware or software deconfigured. Logical memory is memory that was altered by hardware or software to meet the memory configuration rules. The memory board location is displayed after the physical and logical memory size information. In Figure 5, two boards are shown, EWMB0 and EWMB1. Each DIMM slot location is also displayed with the status of all quadrants, buses and rows. Located to the right of each quadrant and bus number is the letter representing the size of the DIMM installed in that slot. Because these are Dual In-line Memory Modules, a 256 MB DIMM is represented as
24
L/L and contains ten 128 MB DRAM chips on each side of the DIMM. A 32 MB DIMM would be listed as S/S and contains ten 16 MB DRAM chips on each side of the DIMM. Figure 6 shows a V2500/V2600 memory board configured as reported by this dcm script example.
Figure 6 DIMM Locations for Example dcm Script
V2500/V2600 Memory Board Configuration Rules

The V-Class system supports up to eight memory boards. Valid configurations of memory boards include two, four, and eight. (A six memory board configuration is not supported.) When performing an upgrade, you could add either two, four, or six memory boards. The first two memory boards, as shown in Table 6, are located in slots MB0L and MB1L.
25
V2500 H/W Configurations V2500/V2600 Memory Configurations Table 6 Memory Board Configurations Order Minimum memory board configuration Upgrade to four memory boards Upgrade to eight memory boards MB0L MB1L MB6R MB7R MB2R MB3R MB4L MB5L Slot locations
26
V2500 H/W Configurations I/O Subsystem Configurations
I/O Subsystem Configurations

Unlike the V22x0, all four I/O card cages are installed, regardless of selected processor and memory configuration. Each card cage (or chassis) houses two 64-bit PCI I/O adapter ASICs, one of which will host a 3-slot 64-bit PCI bus, and the other a 4-slot 64-bit PCI bus, for a maximum of 7 controller cards per chassis. Due to power supply limitations in the I/O chassis, there are powerbudget configuration rules for controller population in each chassis. V2500/V2600 will not be released with EPIC support. Each I/O chassis shipped with V2500/V2600 will be the 64-bit PCI chassis. Field upgrades of a V22X0 to a V2500/V2600 include replacing each EPIC I/O chassis with a SAGA I/O chassis. All upgraded and new V2500/V2600 systems will have 4 I/O chassis installed, providing the customer with 28 expansion slots.
27
V2500 H/W Configurations Mid-Plane (MIB, SCUB, MIBPB)
Mid-Plane (MIB, SCUB, MIBPB)

MIB
No visible configuration changes between v22x0 and V2500/V2600. The MIB will always have 12 ASICs (8 SPACs and 4 ERACs) installed.
SCUB
The SCUB replaces the ECUB for the V2500/V2600. There are no visible configuration changes between ECUB and SCUB except the n+1 switch on the SCUB should always be in the down (enabled) position.
MIBPB
No visible configuration changes between v22x0 and V2500/V2600.
28
V2500 H/W Configurations Power Subsystem
Power Subsystem
Hot swapping of a +48 VDC node power supply (nps) is supported for the V2500/V2600. All V2500/V2600 systems will have 4 nps installed.
29
V2500 H/W Configurations Cooling Subsystem
Cooling Subsystem
There are visible configuration changes between v22x0 and V2500/ V2600 as there are now six cooling fans in each of the two cooling fan tray assemblies. This arrangement supports n+1 on-line fan replacement.
30
V2500 H/W Configurations Mass Storage
Mass Storage
Low Voltage Differential (LVD) SCSI devices will be supported on the V2500/V2600.
31
V2500 H/W Configurations Service Support Processor

The Common Desktop Environment (CDE) is implemented at the Service Support Processor (SSP or Test Station) software release version 4.3 vs. the X11 windows environment. The CDE windows environment will also be utilized on the SSP for V2500/V2600 systems. B180L For V2500/V2600 nodes, the Test LAN that connects between the SSP and the node is derived from lan0 on the B180L and not from lan1. This is not the case for V22x0 nodes as the Test LAN on the 712 SSP connects to lan1. In any case, the new SSP software for V2500/V2600 sorts this out automatically by routing to the correct LAN port for either model SSP that connects to a V2500/V2600 node.
32
V2500 H/W Configurations Miscellaneous
Miscellaneous
Configuration Limitations/Exclusions
The V2500/V2600 Stingray Hardware Program includes the first release of single-cabinet and multi-cabinet V2500/V2600 hardware platforms, with 440 MHz PCXW and 552 MHz PCXW+ processor chips. PCXW processors are supported in the V2500 and the PCXW+ processors and supported in the V2600.
Upgradability
The V2500/V2600 is field upgradable from previous versions of the V22x0 product line. The number of subassemblies to be swapped is substantial, including in all cases:
The mid-plane and utility boards. All processor modules. All memory modules (DIMMs are re-usable). All EPIC I/O chassis' (controllers are reusable). All +48VDC power supplies. All fans. A 50 Amp AC service is required for V2500/V2600 systems.
The V2600 is also field upgradable from the previous version of the V2500 product line. The assemblies to replace for this type of upgrade are:
All processor modules.
33
V2500 H/W Configurations Miscellaneous
Quality
Due to the similarities in hardware technologies, the V2500/V2600 hardware has greatly improved the AFR (Analyzed Failure Rate) characteristics over the V22x0. A significant reduction in AFR is anticipated with the introduction of the PCXW and PCXW+ processor chips which allow the elimination of external caches. Also a small reduction in AFR is achieved by the n+1 fan and hot swapping of the +48VDC power subsystem.
34
3)
Actions and Tasks

Introduction
The intent of this chapter is to identify the new tasks required for V2500/V2600 systems and those V22x0 tasks that had to be modified because of changes or enhancements made for V2500/V2600 support.
35
Actions and Tasks Processor Tasks
Processor Tasks
Task: Deconfiguring Processors using xconfig
Minor changes were necessary for the xconfig utility to support 32 processors. Xconfig operates in the same manner as with the V22x0 hardware. The xconfig utility now displays the status of up to 32 processors as seen in Figure 7.
Figure 7 Xconfig SSP (or Test Station) Screen
36
Task: Deconfiguring Processors with CPUconfig

OBPs menu mode command CPUconfig is used in the same manner as with V22x0 hardware. Figure 8 displays an example of the output from the CPUconfig command.
Figure 8 Menu Mode command CPUconfig
Command Description ----------------CPUconfig [<cpu>] [ON|OFF|SHOW] (De)Configure/Show Processor Command: CPU config show Proc type ---------HP,PA85000 HP,PA85000 HP,PA85000 HP,PA85000 HP,PA85000 HP,PA85000 HP,PA85000 HP,PA85000 HP,PA85000 HP,PA85000 Proc# ----0 1 2 3 4 5 6 7 8 9 Proc Rev -------1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 Speed ------440 MHz 440 MHz 440 MHz 440 MHz 440 MHz 440 MHz 440 MHz 440 MHz 440 MHz 440 MHz State ------Active Active Active Active Active Active Active Active Active Active Dcache ------1024 KB 1024 KB 1024 KB 1024 KB 1024 KB 1024 KB 1024 KB 1024 KB 1024 KB 1024 KB Icache ------512 KB 512 KB 512 KB 512 KB 512 KB 512 KB 512 KB 512 KB 512 KB 512 KB I-prefetch ---------Off Off Off Off Off Off Off Off Off Off
NOTE
Tables provided in the previous chapter (Configuration) assist mapping of the Menu Mode Proc. ID to the slot name where the processor is installed in the chassis.
Task: Deconfiguring Processors with SPPDSH

To deconfigure or reconfigure processors when dialed into a V22x0 system using a modem, it is necessary to use the CCMU utility. All of the CCMU functionality is now incorporated within SPPDSH for the V2500/V2600.
37
The definition of the acronym SPPDSH is Symmetrical Parallel Processing Diagnostic Shell. SPPDSH is a modified ksh with a large number of built-in functions to allow the reading and writing of V2500/V2600 node memory, configuration parameters and CSRs.
NOTE To use the SPPDSH functions it is necessary to execute the sppdsh command on the Service Support Processor (SSP) or use the pull down menu option and open a SPPDSH window.
SPPDSH Commands A man page is located on the SSP that contains detailed information about the commands available inside SPPDSH. Retrieve When modifying a configuration parameter, the first SPPDSH command necessary to use is retrieve. The retrieve command uploads the configuration parameters of the nodes NVRAM into a buffer in the SSP. Once the parameters have been uploaded they may be modified by using the SPPDSH command cput.
NOTE Syntax & man page information about the retrieve command:
retrieve <node#>
Uploads node configuration parameters to the SSP from NVRAM. Clist The clist command is used to provide a listing of the configuration parameters and will accept wildcards as a part of the parameter.
NOTE Syntax & man page information about the clist command:
clist <parameter>
Parameters are names representing POST configurable data.Wherever possible, these parameters should have the same names as those used in OBP. The example in Figure 9 displays how the clist command can be used to list parameters which start with the characters cpu.
38
Actions and Tasks Processor Tasks Figure 9 Example clist command

sppdsh$ clist cpu* cpu_0 cpu_1 cpu_10 cpu_11 cpu_12 cpu_13 cpu_14 cpu_15 cpu_16 cpu_17 cpu_18 cpu_19 cpu_2 cpu_20 cpu_21 cpu_22 cpu_23 cpu_24 cpu_25 cpu_26 cpu_27 cpu_28 cpu_29 cpu_3 cpu_30 * * *(This is not a complete listing of every parameter!!)* * *
Cget To obtain the value from a configuration parameter, the cget command is used. These values are reported in hexadecimal format. Table 7 provides a definition of the hex code values associated with processors.
NOTE Syntax & man page information about the cget command:
cget [-f | -s] <node #> <parameter>
Get the value stored in the configuration parameter name of the node from the SSPs buffer. [-f] forces a read overwriting current data. [-s] forces a read but does so silently. Figure 10 shows an example of how the cget command can be used to obtain the value stored in all configuration parameters that start with the letters cpu.
39
Actions and Tasks Processor Tasks Figure 10 Example cget command
sppdsh$ cget 0 cpu* cpu_0 0x01 cpu_1 0x01 cpu_10 0x01 cpu_11 0x01 cpu_12 0x01 cpu_13 0x01 cpu_14 0x01 cpu_15 0x01 cpu_16 0x30 cpu_17 0x30 cpu_18 0x30 cpu_19 0x30 cpu_2 0x01 cpu_20 0x30 cpu_21 0x30 * * *(This is not a complete listing of every parameter!!)* * *
SPPDSH Configuration Codes

Table 7
Configuration Codes
Hex Code Value 0xff 0x00 0x01 0x10 0x20 0x30 0x40 0x50 Hex Code Value Definition UNKNOWN RESERVED PASS FAIL DECONFIG EMPTY SW_DECONFIG INSTALLED
40
Cput Using the cput command allows the value of the configuration parameter to be modified in the SSPs buffer. Table 7 can be used to determine the correct hex value to be used during the parameter modification.
NOTE Syntax & man page information about the cput command:
cput <node #> <parameter> <value>
Set the node configuration parameter to a specific value in the SSPs buffer. Figure 11 displays an example of using the cput command to modify the hex value stored in parameter cpu_0 to contain 0x40. This will force the processor associated with parameter cpu_0 (physical location of PB0L, CPU socket # A) to become software deconfigured. The software deconfiguration will not occur until the replace command is executed and the system has been reset.
Figure 11 Example cput command
sppdsh$ cput 0 cpu_0 0x40 cpu_0 0x40 sppdsh$
WARNING
If a processor is deconfigured in either socket of a processor board, the processor which is sharing this runway bus will also be deconfigured. The same tools that were used on the V22x0 system will be used to differentiate which processor was hardware deconfigured vs. software deconfigured for the V2500/V2600 systems.
Replace When all modifications to the configuration parameters in the SSPs buffer are complete, the SPPDSH command replace is used. The replace command downloads the configuration parameters from the SSPs buffer to NVRAM. In order for the system to operate with the new values, a reset must be performed on the V2500/V2600.
41
Actions and Tasks Processor Tasks NOTE Syntax & man page information about the cput command:
replace <node#>
Downloads node # configuration parameters from the SSPs buffer to NVRAM.
Task: Interpreting POST messages

In the console window, POST messages will be displayed when a system is booting. By reading the POST messages during bootup, the amount of installed and configured processors can be determined. A failing processor could also be identified if it fails to pass POST. Figure 12 has an example of a V2500 with 32 processors that passed POST.
Figure 12 POST processor messages
POST Hard Boot on [0:18] Stingray POST, Revision 0.6.1.0 1998/08/20 11:04:54 LAB Probing CPUs: PB0L_A PB0R_A PB1R_A PB1L_A PB2L_A PB2R_A PB4R_A PB5R_A PB5L_A PB6L_A PB6R_A PB7R_A PB7L_A PB0L_B PB2L_B PB2R_B PB3R_B PB3L_B PB4L_B PB4R_B PB5R_B PB5L_B PB7L_B #0006 PB3R_A PB3L_A PB4L_A PB0R_B PB1R_B PB1L_B PB6L_B PB6R_B PB7R_B
Task: Interpreting the LCD Display during POST

POST formats the LCD into three sections:
Node Information Line CPU Status Lines Message Line
Figure 13 displays where the three sections are positioned on the LCD Display.
42
Actions and Tasks Processor Tasks Figure 13 LCD Display during POST
Node Information Line
0 (0,0) CPU Status Lines ------abcdefghijklm -------------------
Message Line
Node Information Line The first line is used to display node identification. This information is primarily used when two or more systems are connected together to form a multinode system. A single node system always has 0 (0,0) as the output on the LCD display. It is not the intent of this manual to provide training on a multinode system. H/W training on the multinode system will be presented in the form of a lecture lab course.The format of the Node Information Line is:
<node id> (<x>,<y>)
CPU Status Lines The two middle rows of the LCD display are dedicated to CPU status information. One character in each row corresponds to a unique CPU. The V22x0 hardware has sixteen characters for the sixteen possible CPUs that could be installed and the V2500/V2600 hardware now has thirty two. Table 8 defines the different states of the CPU when the status code is displayed on the LCD.
43
Actions and Tasks Processor Tasks Table 8 LCD CPU Status Line States character 0 1 2 3 4 5 6 7 8 9 a b R M I H Description CPU internal diagnostic register initialization. CPU early data cache initialization. CPU scratch RAM tests. CPU scratch RAM initialization. CPU BIST-based instruction cache initialization. CPU BIST-based data cache initialization. CPU internal register final initialization. CPU basic instruction set testing. CPU basic instruction cache testing. CPU basic data cache testing. CPU basic TLB testing. CPU post-selftest internal register cleanup. RUN: Performing system initialization operations. MONARCH: The main POST initialization CPU. IDLE: CPU is in an idle loop, awaiting a command HPMC: CPU has detected a high priority machine check (HPMC).
44
character T S D d ?
Description TOC: CPU has detected a transfer of control (TOC). SOFT_RESET: CPU has detected a soft RESET. Dead (failing) CPU Deconfigured CPU Empty CPU slot Unknown CPU status
Message Line Information about what state the system is in can be obtained in the message line. Table 9 contains a listing of the codes that define which test the system is performing during POST. When the system has finished: running POST, loading all of the necessary PDC files, and booted HP-UX, the message line will be used to display the system run codes as it did with the V22x0 hardware.
Table 9
Message Line States

Message a b c d Description Utilities board (SCUB) hardware initialization. CPU initialization/selftest rendezvous. Utilities board (SCUB) SRAM test. Utilities board (SCUB) SRAM initialization.
45
Message e f g h i j k l m n o p q r
Description Reading Node ID and serial number. Verifying non-volatile RAM (NVRAM) data structures. Probing system hardware (ASICs). Initializing system hardware (ASICs). Probing CPUs. Initialing, and optionally testing, remaining SCUB SRAM. Probing main memory. Initializing main memory. Verifying multi-node hardware configuration. Multinode initialization starting synchronization. Multinode hardware initialization. Multinode hardware verification. Multinode initialization ending synchronization. Enabling system error hardware.
LCD Command: Executed from the Test Station (SSP), the LCD command will print the contents of an LCD display for node 0 of current complex. If no arguments are specified, lcd will default to using the all parameter.
NOTE Syntax information about the lcd command:
46
lcd [<node id> | <complex name> | all] Determining Processor Hardware Paths The LCD display can be used to determine which slot the processors are installed in. Using the information provided in Figure 14 you can define the following:
CPU number used by menu mode. Physical slot location of the processor board. Locations of processor A and processor B. SPAC the processors connect to. A formula that can be used to calculate the hardware path of a processor using the path provided by the HPUX ioscan command.
Figure 14
LCD Display As used by Menu Mode:

0 1 2 3 - - - - - - 16 17 18 19 4 5 6 7 - - - - - - 20 21 22 23 8 9 10 11 - - - - - - 24 25 26 27 12 13 14 15 - - - - - - 28 29 30 31
CPU #: LCD Display, A Proc.: LCD Display, B Proc.: CPU #:
Formula used to calculate Hardware as viewed by HPUX:

CPU# + 16 = HPUX Hardware path
Using the LCD to determine a physical slot:
SPAC #:
2 3 / \ / \ - - - - - - L R R L
4 5 / \ / \ - - - - - - L R R L
6 7 / \ / \ - - - - - - L R R L
/ \ / \ LCD Display, A Proc.: LCD Display, B Proc.: CPU Location: - - - - - - L R R L
47
Task: COPMOD changes to a V2500 EWPB & V2600 ELPB

There are two types of configurations for a V2500 EWPB: one configuration is as a single processor EWPB (Assembly Part Number - A5491-60001) and the other as a dual processor EWPB (Assembly Part Number - A5492-60001). The EWPB will use the PA8500 PCXW processor chip. There are two types of configurations for a V2600 ELPB: one configuration is as a single processor ELPB (Assembly Part Number A5825-60001) and the other as a dual processor ELPB (Assembly Part Number - A5826-60001). The ELPB will use the PA8600 PCXW+ processor chip. If a EWPB or ELPB is converted from a single processor to a dual processor or visa versa, it will be necessary to change the assembly part number to the corresponding configuration. This is a new task for the V2500/V2600 only. Changing an assembly part number requires the use of the SPPDSH command copmod to update the final part number of an EWPB/ELPB after a CPU was installed or removed. The COP chip allows for electronic tracking of the EWPB/ELPB FRU for future references, etc. The SPPDSH command cop reads information stored inside a COP chip and the copmod command writes data into the COP chip.
NOTE Syntax & man page information about the copmod command: copmod <node#> <cop_id> [<-c> | <-n node#> | <-b board_serial_number> | <-a assembly_part_number> | <-r assembly_revision>]
Reading/writing of COP information is via JTAG logic on the SCUB over the Test LAN from the SSP. Note that data base information on the SSPs screen is derived from COP data.
48
Figure 15 displays an example of the steps that where taken to change a EWPBs assembly part number from a single processor board to a dual processor board. Using the appropriate part number, the same steps can also be used to update an ELPBs assembly part number.
Figure 15 COPMOD example
sppdsh$ sppdsh$ cop 0 pb1l Node Device Part Number Board Serial Number EDC Scan Artwork Rev --------------------------------------------------------------------------0 pb1l A5491-60001 1321478 3843 00 a sppdsh$ sppdsh$ sppdsh$ copmod 0 pb1l -p A5492-60001 sppdsh$ sppdsh$ sppdsh$ cop 0 pb1l Node Device Part Number Board Serial Number EDC Scan Artwork Rev --------------------------------------------------------------------------0 pb1l A5492-60001 1321478 3843 00 a sppdsh$ sppdsh$ sppdsh$ do_reset 0 1
49
Actions and Tasks Memory Tasks
Memory Tasks
Task: Deconfiguring Memory using xconfig
Minor changes were necessary for the xconfig utility to support the new positioning of DIMMS. The new slot naming convention of quadrants, buses and rows will be used within the xconfig utility. Xconfig operates in the same manner to display and deconfigure memory in the V2500 as with the V22x0. Figure 25 will assist determining were the physical slot is located on the EWMB.
Task: Deconfiguring Memory with SPPDSH

To deconfigure or reconfigure memory when dialed into a V22x0 system using a modem, it was necessary to use the CCMU utility. All of the CCMU functionality is now incorporated within SPPDSH for the V2500/V2600.
NOTE To use the SPPDSH functions it is necessary to execute the sppdsh command on the SSP or use the pull down menu option and open a SPPDSH window.
SPDSH Commands
A man page is located on the SSP that contains detailed information about the commands available inside SPPDSH. Retrieve When modifying a configuration parameter, the first SPPDSH command necessary to use is retrieve. The retrieve command uploads the configuration parameters of the nodes NVRAM into a buffer on the SSP. Once the parameters have been uploaded to a buffer on the SSP they may be modified by using the SPPDSH command cput.
NOTE
Syntax & man page information about the retrieve command:
50

retrieve <node#>
Uploads node configuration parameters to the SSP from NVRAM. Clist The clist command is used to provide a listing of the configuration parameters and will accept wildcards as a part of the parameter.
NOTE Syntax & man page information about the clist command:
clist <parameter>
Parameters are names representing POST configurable data.Where ever possible these parameters should have the same names as used in OBP. The example in Figure 16 displays how the clist command can be used to list parameters which start with the characters mb.
Figure 16 Example clist command
sppdsh$ clist mb* mb0l.q0b0r0 mb0l.q0b0r1 mb0l.q0b1r0 mb0l.q0b1r1 mb0l.q0b2r0 mb0l.q0b2r1 mb0l.q0b3r0 mb0l.q0b3r1 mb0l.q1b4r0 mb0l.q1b4r1 mb0l.q1b5r0 mb0l.q1b5r1 mb0l.q1b6r0 mb0l.q1b6r1 mb0l.q1b7r0 mb0l.q1b7r1 mb0l.q2b0r0 mb0l.q2b0r1 mb0l.q2b1r0 mb0l.q2b1r1 mb0l.q2b2r0 mb0l.q2b2r1 mb0l.q2b3r0 mb0l.q2b3r1 * * *(This is not a complete listing of every parameter!!)* * *
51
Cget There are two commands cget and query which can be used to determine what the status of the hardware is. Examples of both commands will be provided. To obtain the value from a configuration parameter, the cget command is used. These values are reported in hexadecimal format. Table 10 provides a definition of the hex code values.
NOTE Syntax & man page information about the cget command:
cget [-f | -s] <node #> <parameter>
Get the value stored in the configuration parameter name of the node from the SSPs buffer. [-f] forces a read overwriting current data. [-s] forces a read but does so silently. Figure 17 shows and example of how the cget command can be used to obtain the value stored in all configuration parameters that start with the characters mb0l.
Figure 17 Example cget command
sppdsh (hw2b):cget 0 mb0l* mb0l.q0b0r0 0x8a mb0l.q0b0r1 0x8a mb0l.q0b1r0 0x8a mb0l.q0b1r1 0x8a mb0l.q0b2r0 0x8a mb0l.q0b2r1 0x8a mb0l.q0b3r0 0x8a mb0l.q0b3r1 0x8a mb0l.q1b4r0 0x8a mb0l.q1b4r1 0x8a mb0l.q1b5r0 0x8a mb0l.q1b5r1 0x8a mb0l.q1b6r0 0x8a mb0l.q1b6r1 0x8a mb0l.q1b7r0 0x8a mb0l.q1b7r1 0x8a mb0l.q2b0r0 0x30 mb0l.q2b0r1 0x30 mb0l.q2b1r0 0x30 mb0l.q2b1r1 0x30 mb0l.q2b2r0 0x30 mb0l.q2b2r1 0x30 mb0l.q2b3r0 0x30 * * *(This is not a complete listing of every parameter!!)* * *
52
SPPDSH Configuration Codes

Table 10 Configuration Codes Hex Code Value 0xff 0x00 0x01 0x10 0x20 0x30 0x40 0x50 0x04 0x24 0x34 0x44 0x64 0x74 0x08 0x28 0x38 0x48 0x68 0x78 0x0c Hex Code Value Definition UNKNOWN RESERVED PASS FAIL DECONFIG EMPTY SW_DECONFIG INSTALLED 16MB_DECONFIG 16MB_DECONFIG_88_TO_80 16MB_DECONFIG_88 SW_16MB_DECONFIG SW_16MB_DECONFIG_88_TO_80 SW_16MB_DECONFIG_88 64MB_DECONFIG 64MB_DECONFIG_88_TO_80 64MB_DECONFIG_88 SW_64MB_DECONFIG SW_64MB_DECONFIG_88_TO_80 SW_64MB_DECONFIG_88 128MB_DECONFIG
53
Hex Code Value 0x2c 0x3c 0x4c 0x6c 0x7c 0x85 0xa5 0xb5 0x89 0xa9 0xb9 0xc9 0xe9 0xf9 0x8d 0xad 0xbd 0xcd 0xed 0xfd 0x8a 0xaa 0xba
Hex Code Value Definition 128MB_DECONFIG_88_TO_80 128MB_DECONFIG_88 SW_128MB_DECONFIG SW_128MB_DECONFIG_88_TO_80 SW_128MB_DECONFIG_88 16MB 16MB_88_TO_80 16MB_88 64MB_TO_16MB 64MB_TO_16MB_88_TO_80 64MB_TO_16MB_88 SW_64MB_TO_16MB SW_64MB_TO_16MB_88_TO_80 SW_64MB_TO_16MB_88 128MB_TO_16MB 128MB_TO_16MB_88_TO_80 128MB_TO_16MB_88 SW_128MB_TO_16MB SW_128MB_TO_16MB_88_TO_80 SW_128MB_TO_16MB_88 64MB 64MB_88_TO_80 64MB_88
54
Hex Code Value 0x8e 0xae 0xbe 0xce 0xee 0xfe 0x8f 0xaf 0xbf
Hex Code Value Definition 128MB_TO_64MB 128MB_TO_64MB_88_TO_80 128MB_TO_64MB_88 SW_128MB_TO_64MB SW_128MB_TO_64MB_88_TO_80 SW_128MB_TO_64MB_88 128MB 128MB_88_TO_80 128MB_88
Query To obtain the status of a memory DIMM, the query command could also be used. The query command reports the status of hardware using an english format.
NOTE Syntax & man page information about the query command: query <node id> <A|C|M|I|D#|S>
The A stands for agents or PACs, C stands for cpus, M stands for MACs, I stands for I/O devices, D stands for the dimms on memory board #, and S stands for SCI devices. The example in Figure 18 displays how the query command can be used to obtain the status of DIMMs located on memory board zero.
55
Actions and Tasks Memory Tasks Figure 18 Example query command

sppdsh (hw2b):query 0 D0 mb0l.q0b0r0: configured to 64M mb0l.q0b0r1: configured to 64M mb0l.q0b1r0: configured to 64M mb0l.q0b1r1: configured to 64M mb0l.q0b2r0: configured to 64M mb0l.q0b2r1: configured to 64M mb0l.q0b3r0: configured to 64M mb0l.q0b3r1: configured to 64M mb0l.q1b4r0: configured to 64M mb0l.q1b4r1: configured to 64M mb0l.q1b5r0: configured to 64M mb0l.q1b5r1: configured to 64M mb0l.q1b6r0: configured to 64M mb0l.q1b6r1: configured to 64M mb0l.q1b7r0: configured to 64M mb0l.q1b7r1: configured to 64M * * *(This is not a complete listing of the query output!!)* * *
Cput The two commands cput and toggle, can be used to deconfigure hardware. Examples of both commands will be provided. Using the cput command allows the value of the configuration parameter to be modified in the SSPs buffer. Table 10 can be used to determine the correct hexadecimal value needed for parameter modification.
cput <node #> <parameter> <value>
Set the node configuration parameter to a specific value in the SSPs buffer. Figure 19 displays an example of using the cput command to modify the hex value stored in parameter mb0l.q0b0r0 to contain 0x48. This will force the memory row associated with parameter mb0l.q0b0r0 (physical location of Q0B0r0) to become a 64 MB software deconfigured row. The software deconfiguration will not occur until the replace command is executed and the system has been reset.
56
Actions and Tasks Memory Tasks Figure 19 Example cput command

sppdsh$ cput 0 mb0l.q0b0r0 0x48 mb0l.q0b0r0 0x48 sppdsh$
WARNING
When a memory row is software deconfigured, POST will deconfigure the rows on this memory board and all other memory boards installed in the system for the most optimal memory interleaving. This assures that all memory boards within the system are populated identically.
Toggle To deconfigure a part in the system the toggle command could be used. The toggle command is only capable of activating or deactivating a component. toggle is not capable of deconfiguring a 128 Mb row to 64 Mb.
NOTE Syntax and man page information about the toggle command toggle <node id> <A#|C#|M#|D#|I#|S#>
The toggle command is used to change the state of a part in the system between active and disabled. The A stands for agents or PACs, C stands for cpus, M stands for MACs, I stands for I/O devices, D stands for the dimms on memory board #, and S stands for SCI devices. Figure 20 displays an example of using the toggle command to mark parameter mb0l.q0b0r0 to be software disabled. This will force the memory row associated with parameter mb0l.q0b0r0 (physical location of Q0B0r0) to become a 64 MB software deconfigured row. The software deconfiguration will not occur until the replace command is executed and the system has been reset.
Figure 20 Example toggle command
sppdsh$ toggle 0 Dmb0l.q0b0r0 mb0l.q0b0r0: deconfigured (0x48) sppdsh$
57
Actions and Tasks Memory Tasks WARNING When a memory row is software deconfigured, POST will deconfigure the rows on this memory board and all other memory boards installed in the system for the most optimal memory interleaving. This assures that all memory boards within the system are populated identically.
Replace When all modifications to the configuration parameters in the SSPs buffer are complete, the SPPDSH command replace is used. The replace command downloads the configuration parameters from the SSPs buffer to NVRAM. In order for the system to operate with the new values, a reset must be performed on the V2500/V2600.
replace <node#>
Downloads node # configuration parameters from the SSPs buffer to NVRAM.
58
Task: Interpreting POST messages

With the new orientation of DIMMS on the memory board, an enhanced set of row status character definition codes were adopted. Table 11 provides a description for each of the row status characters.
Table 11 Row Status Character DIMM Definitions Symbol # Description 0 MB Installed Hardware Deconfigured to 0 MB Software Deconfigured to 0 MB 32 MB Configured 64 MB Configured 128 MB Configured
. : |
59
Figure 21 displays the status of how a 32GB (256 Mb DIMMs) system was configured during POST without any memory deconfiguration problems.
Figure 21 POST messages from a V2500/V2600 system with 32 GB of memory
Probing Main Memory: MB0L MB1L MB2R MB3R MB4L MB5L MB6R MB7R Initializing Main Memory: Parallel memory initialization in progress r0 ||||][|||| ||||][|||| ||||][|||| ||||][|||| ||||][|||| ||||][|||| ||||][|||| ||||][|||| Map: r1 ||||][|||| ||||][|||| ||||][|||| ||||][|||| ||||][|||| ||||][|||| ||||][|||| ||||][|||| r2 ||||][|||| ||||][|||| ||||][|||| ||||][|||| ||||][|||| ||||][|||| ||||][|||| ||||][|||| r3 ||||] ||||] ||||] ||||] ||||] ||||] ||||] ||||]
PB1R_B MB0L [|||| PB1R_A MB1L [|||| PB2L_A MB2R [|||| PB3R_A MB3R [|||| PB4L_A MB4L [|||| PB5R_A MB5L [|||| PB6L_A MB6R [|||| PB7R_A MB7R [|||| Building Main Memory Booting OBP
Figure 22 defines how a row maps to the quadrant and bus associated with each one of the status characters displayed in the POST message.
Figure 22 POST mapping of Quadrants, Buses and Rows
Q1B6R0
Row: Status Character: Bus #:
r0 r1 r2 r3 PB1R_B MB0L [|||| ||||][|||| ||||][|||| ||||][|||| ||||] [0 1 2 3 4 5 6 7][0 1 2 3 4 5 6 7][0 1 2 3 4 5 6 7][0 1 2 3 4 5 6 7]
Quadrant #:
60
Figure 23 displays the status of how a 16GB (128 Mb DIMMs) system was configured during POST without any memory deconfiguration problems.
Figure 23 POST messages from a V2500/V2600 system with 16 GB of memory
Probing Main Memory: MB0L MB1L MB2R MB3R MB4L MB5L MB6R MB7R Initializing Main Memory: Parallel memory initialization in progress r0 ::::][:::: ::::][:::: ::::][:::: ::::][:::: ::::][:::: ::::][:::: ::::][:::: ::::][:::: Map: r1 ::::][---::::][---::::][---::::][---::::][---::::][---::::][---::::][---r2 ----][-------][-------][-------][-------][-------][-------][-------][---r3 ----] ----] ----] ----] ----] ----] ----] ----]
PB1R_B MB0L [:::: PB1R_A MB1L [:::: PB2L_A MB2R [:::: PB3R_A MB3R [:::: PB4L_A MB4L [:::: PB5R_A MB5L [:::: PB6L_A MB6R [:::: PB7R_A MB7R [:::: Building Main Memory Booting OBP
Figure 24 defines how a row maps to the quadrant and bus associated with each one of the status characters displayed in the POST message.
Figure 24 POST mapping of Quadrants, Buses and Rows
Q0B2R1
Row: Status Character: Bus #:
r0 r1 r2 r3 PB1R_B MB0L [:::: ::::][:::: ::::][---- ----][---- ----] [0 1 2 3 4 5 6 7][0 1 2 3 4 5 6 7][0 1 2 3 4 5 6 7][0 1 2 3 4 5 6 7]
Quadrant #:
Figure 25 can then be used to determine where the DIMM slot is located on the memory board. SPPDSH parameters are shown on the left and POST mapping is found on the right. The shaded slots represent one set of DIMMs installed in quadrant zero.
HP Company Confidential 61
Actions and Tasks Memory Tasks Figure 25 Physical EWMB Layout
Memory DIMM (EWMB) Slot Locations

X=memory board number (0-7)
SPPDSH Parameter
(Even numbered rows correspond to a side of the DIMM which is closest to the connector.)
Connector
POST Mapping
(Even numbered rows correspond to a side of the DIMM which is closest to the connector.)
60$&
mbXl.q0b3r0 - mbXl.q0b3r1 mbXl.q2b3r0 - mbXlq2b3r1 mbXl.q1b7r0 - mbXl.q1b7r1 mbXl.q3b7r0 - mbXl.q3b7r1
Q0B3 Q2B3 Q1B7 Q3B7
Q0B3R0-Q0B3R1 Q2B3R2-Q2B3R3 Q1B7R0-Q1B7R1 Q3B7R2-Q3B7R3
mbXl.q0b1r0 - mbXl.q0b1r1 mbXl.q2b1r0 - mbXl.q2b1r1 mbXl.q1b5r0 - mbXl.q1b5r1 mbXl.q3b5r0 - mbXl.q3b5r1 mbXl.q0b2r0 - mbXl.q0b2r1 mbXl.q2b2r0 - mbXl.q2b2r1 mbXl.q1b6r0 - mbXl.q1b6r1 mbXl.q3b6r0 - mbXl.q3b6r1
Q0B1 Q2B1 Q1B5 Q3B5 Q0B2 Q2B2 Q1B6 Q3B6
Q0B1R0-Q0B1R1 Q2B1R2-Q2B1R3 Q1B5R0-Q1B5R1 Q3B5R2-Q3B5R3 Q0B2R0-Q0B3R1 Q2B2R2-Q2B2R3 Q1B6R0-Q1B6R1 Q3B6R2-Q3B6R3
mbXl.q0b0r0 - mbXl.q0b0r1 mbXl.q2b0r0 - mbXl.q2b0r1 mbXl.q1b4r0 - mbXl.q1b4r1 mbXl.q3b4r0 - mbXl.q3b4r1
Q0B0 Q2B0 Q1B4 Q3B4
Q0B0R0-Q0B0R1 Q2B0R2-Q2B0R3 Q1B4R0-Q1B4R1 Q3B4R2-Q3B4R3
67$&
rvpr!$
Urqh9rprir $
(('
62
Actions and Tasks I/O Subsystem Tasks
I/O Subsystem Tasks

Task: Determine I/O Hardware Path
With the addition of a slot to each card cage, Figure 26 displays what the values are for the SAGA ASICs and the PCI slot numbers.
Figure 26
SIOB Card Cage
63
Actions and Tasks Mid-Plane (MIB, SCUB, MIBPB) Tasks
Mid-Plane (MIB, SCUB, MIBPB) Tasks

MIB
When replacing the MIB it is necessary to configure the customers SWID value into the new FRU.
SCUB
When replacing the SCUB it is necessary to restore the original OBP parameters to the state they were in before the SCUB was replaced. The SCUBs firmware versions must be verified to match those on the SSP and reload when applicable.
MIBPB
No functional changes between V22x0 power board for the ENRB and V2500/V2600 power board for the MIB.
64
Actions and Tasks Power Subsystem Tasks
Power Subsystem Tasks

The V2500/V2600 will allow hot swapping of a +48VDC node power supply when the n+1 rules apply. All V2500/V2600 systems will have 4 node power supplies. In order for n+1 to be in effect, the redundancy switch on the SCUB must be in the down position.
Task: Determining the State of Power Subsystem

This is a new task for V2500/V2600. Visual identification of the redundancy switch position on the SCUB is no longer necessary as this information is available from the T/S using the SPPDSH command pswitch.
NOTE Syntax & man page information about the pswitch command:
pswitch <node #>
Task: Hot Swap Replacement of Node Power Supplies

This is a new task for V2500/V2600. Hot swap replacement of a nps (node power supply) for the V2500/V2600 is given to the customer as a new feature. The V22x0 does not support hot swapping of a node power supply. The first step is to use the SPPDSH command pce. pce identifies a node power supply failure via four status codes 0x70 - 73. These status codes are the same values used for a V22x0 system and can be found in the leds man page on the SSP. The pce command is explained in detail during the SSP tasks portion of this chapter. Using the -r option of pce, set the system power flag to off. Figure 27 displays an example of using the -r option. Move the on/off switch of the failed nps to the off position. Perform the necessary procedures to remove and replace the nps. Move the on/ off switch of the new nps to the on position.
65
Actions and Tasks Power Subsystem Tasks
Using the -r option of pce, set the system power flag to on. Figure 27 displays an example of using the -r option. Verify the error has been removed from the system by executing the pce command. The SPPDSH command blink is then used to toggle the state of light bar (ELEDB) to stop flashing after the node power supply is replaced.
NOTE Figure 27 Syntax information about the blink command:
blink <node #>
Steps performed when hot swapping a nps

sppdsh$ pce 0 Node Clocks LEDS @C U SHPT Supply 1 Supply 2 Supply 3 Supply 4 ---- ------ ------------- ---- ------ -------- -------- -------- -------0 Normal ATTN 0x70 25 1 0000 Nominal Nominal Nominal Nominal sppdsh$ sppdsh$ pce 0 -r 0 Node Clocks LEDS @C U SHPT Supply 1 Supply 2 Supply 3 Supply 4 ---- ------ ------------- ---- ------ -------- -------- -------- -------0 Normal ATTN 0x70 25 1 0000 Nominal Nominal Nominal Nominal sppdsh$ sppdsh$ pce 0 -r 1 Node Clocks LEDS @C U SHPT Supply 1 Supply 2 Supply 3 Supply 4 ---- ------ ------------- ---- ------ -------- -------- -------- -------0 Normal 0x00 25 1 0010 Nominal Nominal Nominal Nominal sppdsh$ sppdsh$ pce 0 Node Clocks LEDS @C U SHPT Supply 1 Supply 2 Supply 3 Supply 4 ---- ------ ------------- ---- ------ -------- -------- -------- -------0 Normal 0x00 25 1 0010 Nominal Nominal Nominal Nominal sppdsh$ sppdsh$ blink 0 sppdsh$
66
Actions and Tasks Cooling Subsystem Tasks
Cooling Subsystem Tasks

The V2500/V2600 will allow hot swapping of cooling fan. All V2500/ V2600 systems will have 6 cooling fans.
Task: On-Line replacement of Cooling Fans

This is a new task for V2500/V2600. Hot swapping of a cooling fan is made possible with newly designed fan trays. The SPPDSH command pce identifies cooling fan failures via six status codes 0x5c - 61. The pce command is explained in detail during the SSP tasks portion of this chapter. Mechanical interlocks within each fan assembly control DC power on/off switching for safety during individual fan assembly replacement. The SPPDSH command blink is used to toggle the state of light bar (ELEDB) to stop flashing after the fan is replaced.
NOTE Syntax & man page information about the blink command:
blink <node #>
N+1 fans provided with V2500/V2600 will temporarily cool cabinet FRUs while replacing any 1 of 6 cooling fans. Each set of three fans in the Upper and Lower fan trays are marked (0, 1, 2) with silk screen.
67
Actions and Tasks Mass Storage Tasks
Mass Storage Tasks

No functional changes between V22x0 and V2500/V2600 with the exception that newer I/O devices are now supported on V2500/V2600.
68
Actions and Tasks Service Support Processor Tasks
Service Support Processor Tasks

Task: do_reset (new features)
do_reset operates in the same manner on the V2500/V2600 hardware, however there are a few new features to the do_reset command.
NOTE Syntax information for the do_reset command:
do_reset [# | complex | all] [level] [boot option]
The new boot option feature allows the system boot up to different levels that could only be accomplished with the use of ccmu or xconfig on V22x0 hardware. Table 12 provides a listing of the parameter with a definition that may be used with the do_reset command.
Table 12 Boot Option Parameters Parameter obp post_interactive tc_standalone tc_interactive loader NOTE Definition Reboots to OBP. Reboots to POST in the interactive mode. reboots to the test controller in stand-alone mode reboots to the test controller in interactive mode reboots to the firmware loader
Since tc_standalone and tc_interactive are now boot option parameters to the do_reset command, their associated scripts have been removed from the /spp/scripts directory.
69
The loader parameter is a new feature in the V2500/V2600. When the loader option is used, the V2500/V2600 system will: reboot, execute the POST firmware, and then load a new firmware loading utility called PDCFL. See Task: Loading Firmware using PDCFL on page 69.
Task: pce vs. pce_util

For V22x0 hardware, pce_util is used to obtain the value of the environmental LEDs, current power, clock, temperature states and margin clocks in the node. This capability is still available but it has been incorporated within SPPDSH and has the new name of pce.
NOTE The command syntax for pce is provided for you and can also be obtained from the SPPDSH man page that resides in the SSP.
pce <node#| complex#|all> [-c <n|u|e>] [-r <on|off>] [-p <l|n|u>] -c manipulate clocks on the current nodes. n[ominal] u[upper] e[xternal] -r sets the power flag to on or off nominal frequency upper frequency external connector
-p manipulate power on the current nodes.

n[ominal] u[upper] e[xternal] nominal frequency upper frequency external connector
Figure 28 displays an example of the output from the pce command.

Figure 28 Example output from the pce command
sppdsh$ pce 0 Node Clocks LEDS @C U SHPT Supply 1 Supply 2 Supply 3 Supply 4 ---- ------ ------------- ---- ------ -------- -------- -------- -------0 Normal 0x00 27 1 0000 Nominal Nominal Nominal Nominal
70
Task: Loading Firmware using PDCFL

PDCFL is a bootable module with the capabilities of loading other firmware modules into Flash RAM. If a V22x0 system has a problem booting OBP firmware, the system can not be used. load_eprom was the only utility available to re-install the OBP firmware into Flash RAM and would take over two hours to install. With PDCFL being a bootable module for V2500/V2600 systems, the system can be reset to boot PDCFL and re-install the OBP firmware. Re-installing OBP firmware with PDCFL will take approximately twelve minutes. Booting the system to load the PDCFL module can be performed by executing the following command on the SSP:
do_reset 0 1 loader
A successful boot of PDCFL displays a prompt similar to the shown below in the console window of the SSP:
PDCFL>
The fload command within PDCFL would then be used to re-install the OBP firmware into Flash RAM. Using the following command at the PDCFL prompt will reinstall OBP firmware:
fload obp2500.pdc 0xf0080000
After the fload command completes writing OBP to Flash RAM, resetting the system is necessary. The OBP parameters can then be restored using the nvrestore command at the OBP prompt. PDCFL can be loaded into Flash RAM, via load_eprom command, through the JTAG port. Once PDCFL has booted, it can also be used to load other modules, such as POST and Test Controller, into Flash RAM through the SONIC port. load_eprom supports a -f option for loading PDCFL into the appropriate sector in Flash RAM. Two OBP parameters need to be setup to allow PDCFL to communicate with the SSP. These are ts-ip# and obp-ip#. The usual values are

ts-ip# 15.99.111.99 obp-ip# 15.99.111.150
The SSP needs to be setup to act as a tftp server for the files you wish to load into Flash RAM. This is accomplished for you via the /spp/ scripts/ts.install command that is performed when the SSP software is upgraded. If additional information is required on how the SSP is configured as a tftp server, it can be obtained by reading the man page for PDCFL that resides on the SSP. From the PDCFL prompt, the following commands are supported: printenv [variable] - Prints configuration variables from NVRAM. setenv [variable] [value] - Allows setting configuration variables in NVRAM. reset [post] - Resets the node, optionally changing the boot vector to point to the POST module. lifls - Prints a listing of the LIF volume into Flash RAM. The listing includes the name of the module, the Flash RAM address at which the module starts, the size in LIF units, the date the module was last written, and the sectors included by the module. Example output from the PDCFL command lifls:
PDCFL> lifls LIF Volume FLASH4 Name Addr Size Date Sectors ----------------------------------------------------------------------POST 0xF0020000 0x400 04/09/97 4-5 TC 0xF0140000 0x300 04/09/97 16-17 CPU3000 0xF0170000 0x300 04/09/97 17-18 DIODC 0xF01A0000 0x300 04/09/97 19-20 MEM3000 0xF01D0000 0x2F0 04/09/97 20-21 RDR_DUMPE 0xF01FF000 0x10 04/09/97 21 IO3000 0xF0260000 0x500 04/09/97 25-27 INTER3000 0xF02B0000 0x300 04/09/97 27-28 PDCFL 0xF02E0000 0x200 04/09/97 29 DFDUTIL 0xF0300000 0x200 04/09/97 30
72
fload [file] [location] - loads a file from the SSP tftp directory to the address in Flash RAM specified in the lif directory by name. Location can also be a specific address given in hex to allow loading files that have not yet been entered in the LIF directory. If this form is used, the LIF directory will not be updated. Example output of installing POST firmware using the fload command:
PDCFL> fload post.fw POST TFTP server : 15.99.103.191 CUB IP : 15.99.111.150 Reading : post.fw Writing : POST (each . represents 4K copied) Sector erased 0xF0020000 ....................................... Sector erased 0xF0040000 ............ 148384 bytes transferred
Task: Downloading Firmware to SCUB

This task is accomplished using the same commands on the V2500/ V2600 as used on the V22x0. The primary commands used are load_eprom from the SSP and the OBP command source that uses the fwcp command from the /spp/scripts/dl-diags script. Specific steps for downloading the utilities can be obtained from the SSP software release notes or by reading the man page for fw_install.
73
74
4)
Troubleshooting Hints
Introduction
This chapter contains a little information about many things. Since this document wont be updated after its release, we are providing as much information as is known at this time.
75
Troubleshooting Hints Some General Thoughts
Some General Thoughts

Stand alone Diagnostics Just about every V22x0 Service Support Processor (SSP or Test Station) utility and offline diagnostic had to be revised and some even scrapped for the V2500/V2600 new CPUs and ASICs. Among the discarded were: ARCH3000 (some of its code now lives in CPU3000 and MEM3000); and INTRA3000 (some of its code now lives in INTER3000). What this all means is that it is likely that some (yet to be discovered) utility errors may occur that are not caused by H/W faults or configuration errors. Cheer up! There are still many H/W faults that are detected by the utilities and diagnostics. This is just a reminder that there are still some H/W faults that are not easy to troubleshoot even when the utilities find them.H/W faults during POST are the first that comes to mind; IO3000 is next; then there are always those hard errors from a Responseless Request. One very positive point is that the V2500/V2600 hardware supports HP-UX and Mesa in a way that V22x0 H/W could not and may never. HP-UX Chassis Codes Chassis codes displayed in the control panels LCD while HP-UX is running must be able to indicate greater than 16 CPUs. Chassis Code example: FnnF where (n) = (% CPU utilization) and (n)=(# of CPUs) Displaying greater than 16 CPUs requires stealing a bit from the previous digit. 16 CPUs is represented as F0FF; 32 CPUs as F1FF. Web-based Troubleshooting for V2500/V2600 URL: http://fesweb.corp.hp.com/dewold/v2500/v-2500.htm
Troubleshooting Hints Processor FRUs
Processor FRUs
EWPB & ELPB
Much more attention must now be paid to EWPB and ELPB faults as each PA8500 or PA8600 processor chip on the processor board is a Field Replaceable Unit and the EWPB or ELPB must be COPed when a CPU is both added or removed. The difficulty is not identifying that a CPU failed, but in interpreting the error datas naming convention that identifies CPU A/B on Runway 0/1 on PBnL or PBnR.
PCXW & PCXW+ (or CPU)

Thanks to a very robust cache design, D-Cache errors detected by PA8500 and PA8600 CPUs no longer HPMC. Oh, in case you didnt know, the floating point (FP) logic within every PA-RISC CPU has no error circuitry. Since POST does not test FP to any extent, it is necessary to run (CPU3000, subtest 570) periodically to ensure FP functionality. Also, PCXW and PCXW+ chips have one more reason to HPMC as they detect a new Runway Bus estat code: 0xC (ESTAT_SYNCHRONIZER_ERR) The Runway synchronizer has detected a synchronize rule violation. As such, there are minor changes to the PCXW or PCXW+ Status 0 and 1 register definitions. See Web-based Troubleshooting for V2500/V2600 (URL is on previous page).
Mesa
All PCXW and PCXW+ recoverable cache errors are logged as LPMCs by LOGTOOL within Mesa.
77
Troubleshooting Hints Processor FRUs
CPU3000 Tests
Class 1 subtests verify the most basic functionality of the processor chips. When testing processors it is recommended to execute all of the subtests within each class. The format for test times is (HH:MM:SS).
Table 13
Subtest 100 101 102 103 104 105 111 120 130 140 141 150
CPU Class 1 Subtests (General)

Description CPU Minimal Functionality Test CPU ALU Test CPU Branch Test CPU Arithmetic Conditions Test CPU Bitwise Operations Test CPU Control Registers Test CPU Interval Timer Test CPU Multimedia Test CPU Shadow Register Test CPU Local Diagnostic Registers Test CPU Remote Diagnostic Registers Test CPU Register Bypass Test 32 CPU test time 00:00:05 00:00:05 00:00:05 00:00:05 00:00:05 00:00:05 00:00:05 00:00:05 00:00:05 00:00:05 00:00:05 00:00:05
Table 14
Subtest 210
CPU Class 2 Subtests (Icache)

Description Icache Ram Test 32 CPU test time 00:05:00
Table 15
Subtest 310
CPU Class 3 Subtests (Dcache)

Description Dcache Ram Test 32 CPU test time 00:05:00
Table 16
Subtest 400
CPU Class 4 Subtests (TLB)

Description TLB Ram Test 32 CPU test time 00:00:30
78
Troubleshooting Hints Processor FRUs Table 17

Subtest 500 510 520 530 540 550 560 560 570
CPU Class 5 Subtests (Functional Tests)

Description Re-executes early selftests (excluding cache) from icache Verifies access widths Verifies cache flush and purge operations Verifies instructions can be encached from coh. mem. Verifies data can be encached from coh. mem. Verifies dcache parity errors can be detected and handled Verifies functionality of the dcache store queue Test functionality of the TLB Verifies the floating point unit 32 CPU test time 00:01:00 00:00:30 00:00:30 00:00:30 00:00:30 00:00:30 00:00:30 00:00:30 00:00:30
Executing CPU3000
Executing the offline test controller diagnostics for a V2500/V2600 system is performed the same way as it is on a V22x0 system. When the system is booted into test controller interactive mode, the CPU3000 diagnostic can be executed from the interactive menu driven interface within the console window. If the system is booted into test controller standalone mode, the CPU3000 diagnostic can be executed from the GUI interface running the command cxtest -d from a SSP shell window. The cxtest command also has a command line interface. Detailed information about the cxtest command can be found in the cxtest man page on the SSP. The figure on the following page shows an example of booting the system into test controller standalone mode and executing all of the CPU3000 subtests via the command line interface.
79
Troubleshooting Hints Processor FRUs Figure 29 Command line interface

fesmesst:/users/sppuser$ do_reset 0 1 tc_standalone 0:0xf000d00a90 0x01000000 0x01000000 fesmesst:/users/sppuser$ cxtest -cpu 1 Nodes found Command line 0 : -cpu : Number of args 1 Subtest 100 - CPU Minimal Functionality Test 0:00:05 passed Subtest 101 - CPU ALU Test 0:00:05 passed Subtest 102 - CPU Branch Test 0:00:05 passed Subtest 103 - CPU Arithmetic Conditions Test 0:00:05 passed Subtest 104 - CPU Bitwise Operations Test 0:00:06 passed Subtest 105 - CPU Control Registers Test 0:00:05 passed Subtest 111 - CPU Interval Timer Test 0:00:05 passed *****This is not a complete listing*****
0:0 0:0 0:0 0:0 0:0 0:0 0:0
80
Troubleshooting Hints Memory FRUs
Memory FRUs
V-Class DIMM Slot Identification
The DIMM slot connectors on the EWMB of the V2500/V2600 look similar to those on the EMB of the V22x0. However, each DIMM slot connector is identified with a different marking. This is because V22x0 memory boards are not compatible with V2500/V2600 memory boards. V22x0 EMB DIMM Slot Accesses With V22x0 EMBs, the connector markings use Bn to indicate Banks 0 - 3; and use Sn to indicate Slot 0 - 7.
Table 18
V22x0 DIMM Slot Markings

B0S0 B1S0 B2S4 B3S4 B0S1 B1S1 B2S5 B3S5 B0S2 B1S2 B2S6 B3S6 B0S3 B1S3 B2S7 B3S7
All Row 0/1 accesses to DIMMs in Banks 0 - 3 go to slots labeled B0S0, B1S0, B2S4, and B3S4. All Row 2/3 accesses to DIMMs in Banks 0 - 3 go to slots labeled B0S1, B1S1, B2S5, and B3S5. All Row 4/5 accesses to DIMMs in Banks 0 - 3 go to slots labeled B0S2, B1S2, B2S6, and B3S6. All Row 6/7 accesses to DIMMs in Banks 0 - 3 go to slots labeled B0S3, B1S3, B2S7, and B3S7.
81
Troubleshooting Hints Memory FRUs
V2500/V2600 EWMB DIMM Slot Accesses With V2500/V2600 EWMBs, the connector marking use Qn to indicate the Quadrant 0 - 3; and use Bn to indicate Bus 0 - 7.
Table 19
V2500/V2600 DIMM Slot Markings

Q0B0 Q1B4 Q2B0 Q3B4 Q0B1 Q1B5 Q2B1 Q3B5 Q0B2 Q1B6 Q2B2 Q3B6 Q0B3 Q1B7 Q2B3 Q3B7
All Row 0/1 accesses to Banks 0/1 in DIMMs on Buses 0 - 3 are within Quadrant 0; slots are labeled Q0B0 through Q0B3. All Row 0/1 accesses to Banks 0/1 in DIMMs on Buses 4 - 7 are within Quadrant 0; slots are labeled Q1B0 through Q1B3. All Row 2/3 accesses to Banks 0/1 in DIMMs on Buses 0 - 3 are within Quadrant 0; slots are labeled Q2B0 through Q2B3. All Row 2/3 accesses to Banks 0/1 in DIMMs on Buses 4 - 7 are within Quadrant 0; slots are labeled Q3B0 through Q3B3.
NOTE The two banks (or sides) of each DIMM are not needed to identify the slot number.
See the following figure to understand how Quadrant is used as a DIMM slot identifier.
82
Troubleshooting Hints Memory FRUs Figure 30 V2500/V2600 EWMB DIMM Slot Groupings:
Bus # Rows 0/1
0
Q0B0
2
Q0B2
4
Q1B4
6
Q1B6
Q0
Q0B1 Q2B0 Q2B2 Q0B3 Q3B4
Q1
Q1B5 Q3B6 Q1B7
Rows 2/3
Q2
Q2B1 Q2B3
Q3
Q3B5 Q3B7
V25020.ppt
Each box within the above figure represents a DIMM slot #. This memory design allows mixed DIMM sizes to be managed on a Quadrant basis instead of by a Bank basis like V22x0. Both memory designs provide limited DIMM loading flexibility. The first rule is that all DIMMs within a Quadrant must be the same size on all EWMBs. The second rule is that all EWMBs must have identical Quadrant configurations.
Mesa
All single-bit ECC memory errors detected while running HP-UX are logged by Mesa and are no longer reported in DMesg buffer. Double-bit ECC memory errors are logged into PDT after a hard error is signalled.
83
Troubleshooting Hints Using the dcm script to troubleshoot a memory error
Using the dcm script to troubleshoot a memory error

This section discusses one way to troubleshoot a memory board or DIMM error. If a screen similar to the following appears during POST run the dcm script to determine which DIMM (or DIMMs) may be faulty. The # character in the following example indicates memory has been deconfigured. For example:
r0 PB0L_A PB1R_A PB2L_A PB3R_A PB4L_A PB5R_A PB6L_A PB7R_A MB0L MB1L MB2R MB3R MB4L MB5L MB6R MB7R [#### [.... [.... [.... [.... [.... [.... [.... r1 ....][#### ####][.... ####][.... ####][.... ####][.... ####][.... ####][.... ####][.... r2 ....][____ ####][____ ####][____ ####][____ ####][____ ####][____ ####][____ ####][____ r3 ____][____ ____][____ ____][____ ____][____ ____][____ ____][____ ____][____ ____][____ ____] ____] ____] ____] ____] ____] ____] ____]
To run the dcm script: Bring up the sppdsh prompt at a teststation window by entering:
$ sppdsh
Run the dcm script from the sspdsh prompt by entering:

sppdsh$ dcm 0
Output similar to the following example will be displayed:
84
Troubleshooting Hints Using the dcm script to troubleshoot a memory error Figure 31 Example V2500/V2600 dcm script, partial listing Logical: l=128MB, m=64MB, s=16MB
Physical: L=128MB, M=64MB, S=16MB
(If logical memory not specified, then it matches physical memory size) * = Software Deconfigured EWMB0: ====== EWMB0: EWMB0: EWMB0: EWMB0: EWMB1: ====== EWMB1: EWMB1: EWMB1: EWMB1:
Unrecognized DIMM
- = Not In Use
Q0B0 Q0B1 Q0B2 Q0B3
-/L-/LL-/LL-/L-
Q1B4 Q1B5 Q1B6 Q1B7
L/L L/L L/L L/L
Q2B0 Q2B1 Q2B2 Q2B3
-/-/-/-/-
Q3B4 Q3B5 Q3B6 Q3B7
-/-/-/-/-
Deallocated DIMMS
Q0B0 Q0B1 Q0B2 Q0B3
L/L L/L L/L L/L
Q1B4 Q1B5 Q1B6 Q1B7
L-/LL-/LL-/LL-/L-
Q2B0 Q2B1 Q2B2 Q2B3
-/-/-/-/-
Q3B4 Q3B5 Q3B6 Q3B7
-/-/-/-/-
The above figure shows a DIMM labeled -/-. This indicates that the DIMM has not been recognized during the boot process. POST typically marks only one DIMM at a time as -/-. Because of the failure in Q0B0 on EWMB0, POST automatically deconfigured the DIMMs in Q0B1, Q0B2, and Q0B3. The code L-/L- indicates that the DIMMs have been deconfigured. Because all of the memory boards must be configured identically, POST will also deallocate the equivalent amount of memory from every other memory board installed in this system leaving Q1B4, Q1B5, Q1B6, and Q1B7 deallocated.
MEM3000 Tests
The estimated test times are based on a system with 32 processors and 32 GB of memory.
85
Troubleshooting Hints Using the dcm script to troubleshoot a memory error Table 20
Subtest 100 101 110 120 130 140 150
Memory Class 1 Subtests (Basic Functionality)

Description Diagnostic CSR Read/Write Test Other EMAC CSR Read/Write Test Memory Data Read/Write Test Memory ECC Read/Write Test Memory Tag Read/Write Test Memory Initialization Test First 32 Memory Lines Test 32 CPU test time 00:00:08 00:00:05 00:03:00 00:00:33 00:03:00 00:00:30 00:00:30
Table 21
Subtest 200 210 211 230 231 232 233 234 235 236 237 238
Memory Class 2 Subtests (TAG Bytes)

Description Tag Bank Test Tag Addressing Test Tag Byte Uniqueness Pattern Test Tag March-C Pattern 1 Test Tag March-C Pattern 2 Test Tag March-C Pattern 3 Test Tag March-C Pattern 4 Test Tag March-C Pattern 5 Test Tag March-C Pattern 6 Test Tag March-C Pattern 7 Test Tag March-C Pattern 8 Test Tag User Data Pattern Test 32 CPU test time 00:00:10 00:05:30 00:05:00 00:11:00 00:11:00 00:11:00 00:11:00 00:11:00 00:11:00 00:11:00 00:11:00 00:11:00
86
Subtest 300 310 311 330 331 332 333 334 335 336 337 338
Memory Class 3 Subtests (Data Bytes)

Description Memory Bank Test Memory Addressing Test Memory Byte Uniqueness Pattern Test Memory March-C Pattern 1 Test Memory March-C Pattern 2 Test Memory March-C Pattern 3 Test Memory March-C Pattern 4 Test Memory March-C Pattern 5 Test Memory March-C Pattern 6 Test Memory March-C Pattern 7 Test Memory March-C Pattern 8 Test Memory User Data Pattern Test 32 CPU test time 00:04:00 00:06:30 00:06:20 00:08:50 00:05:40 00:05:40 00:05:40 00:05:40 00:05:40 00:05:40 00:05:40 00:05:40
Table 23
Subtest 400 410 420
Memory Class 4 Subtests (Coherency and Semaphores)

Description Memory Data Flush Transaction Test Memory Load/Store Test Memory Semaphore Transaction Test 32 CPU test time 00:00:10 00:00:15 00:00:15
Table 24
Subtest 500 510 520 530
Memory Class 5 Subtests (ECC)

Description ECC Single Error Correction Test ECC Double Error Detection (coherent) Test ECC Double Error Detection (non-coherent) Test ECC Disable Test 32 CPU test time 00:00:20 00:00:45 00:00:20 00:00:20
87
Subtest 600 610 620 630
Memory Class 6 Subtests (Miscellaneous Functionality)

Description Memory Access Protection Test Memory Tag Test I EMAC System Configuration CSR Test 80 vs. 88 Bit DIMM Test 32 CPU test time 00:00:40 00:00:30 00:00:30 00:00:05
Executing MEM3000
Executing the offline test controller diagnostics for a V2500/V2600 system is performed the same way as it is on a V22x0 system. When the system is booted into test controller interactive mode, the MEM3000 diagnostic can be executed from the interactive menu driven interface within the console window. If the system is booted into test controller standalone mode, the MEM3000 diagnostic can be executed from the GUI interface by running the command cxtest -d from a SSP shell window. The cxtest command also has a command line interface. Detailed information about the cxtest command can be found in the cxtest man page on the SSP. The figure on the following page shows an example of booting the system into test controller standalone mode and executing all of the MEM3000 subtests via the command line interface.
88
Troubleshooting Hints Using the dcm script to troubleshoot a memory error Figure 32 Executing MEM3000 from the command line interface
fesmesst:/users/sppuser$ do_reset 0 1 tc_standalone 0:0xf000d00a90 0x01000000 0x01000000 fesmesst:/users/sppuser$ cxtest -mem 1 Nodes found Command line 0 : -mem : Number of args 1 Subtest 100 - Diagnostic CSR Read/Write Test 0:00:05 passed Subtest 101 - Other EMAC CSR Read/Write Test 0:00:05 passed Subtest 110 - Memory Data Read/Write Test 0:00:30 passed Subtest 120 - Memory ECC Read/Write Test 0:00:11 passed Subtest 130 - Memory Tag Read/Write Test 0:00:30 passed Subtest 140 - Memory Initialization Test 0:00:07 passed Subtest 150 - First 32 Memory Lines Test 0:00:11 passed *****This is not a complete listing*****
0:0 0:0 0:0 0:0 0:0 0:0 0:0
It is estimated that testing memory with MEM3000 will take approximately one quarter of the time to run on a V2500/V2600 as compared to a V22x0. This is due to the memory subsystem redesign.
89
Troubleshooting Hints I/O Subsystem FRUs
I/O Subsystem FRUs

SIOB
SIOB card cages are not interchangeable with previous EIOB card cages. Both, however, have the same DC power budget rules.
IO3000 Tests
Test times are estimates based on one SAGA and one peripheral selected. Test times are with parameters defaulted, this includes disk write enables set to False. Selecting more SAGA/devices or enabling write operations will increase test times.
Table 26
Subtest 100 105 110
IO3000 Class 1 Subtests (SAGA CSRs)

Description CSR reset test CSR read/write test Error CSR test 00:00:10 00:00:15 00:00:10 Test time (1 SAGA)
Table 27
Subtest 200 205 210 215 220 225 230 235 240 245
IO3000 Class 2 Subtests (Shared & Prefetch Memory)

Description Context/shared memory read/write test Context/shared memory access width test Context/shared memory march C- test Context/shared memory pattern test Context/shared memory parity detection test Prefetch memory read/write test Prefetch memory access width test Prefetch memory march C- test Prefetch memory pattern test Prefetch memory parity detection test Test time (1 SAGA) 00:00:10 00:00:10 00:03:00 00:03:00 00:00:10 00:00:10 00:00:10 00:01:30 00:01:30 00:00:10
90
Troubleshooting Hints I/O Subsystem FRUs Table 28

Subtest 500 505 510 515 520
IO3000 Class 5 Subtests (SCSI Disks)

Description SCSI Disk test unit ready test SCSI Disk inquiry test SCSI Disk read capacity test SCSI Disk read test SCSI Disk write test Test time (1 SAGA) 00:00:15 00:00:15 00:00:15 00:00:15 00:00:15
Table 29
Subtest 600 605 610 615 620 625 630 635 640 645 650
IO3000 Class 6 Subtests (Channel Builder)

Description Channel init, ATPR = 0x0 Channel build, ATPR = 0x0 Channel init, data prefetch, ATPR = 0x2 Channel build, ATPR = 0x2 Channel init, write tlb, ATPR = 0x8 Channel init, write tlb, data prefetch, ATPR Channel init, tlb prefetch, ATPR = 0xc Channel build, ATPR = 0xc Channel init, tlb & data prefetch, ATPR = 0xe Channel build, ATPR = 0xe Channel context access test Test time (1 SAGA) 00:00:15 00:00:15 00:00:15 00:00:15 00:00:15 00:00:15 00:00:15 00:00:15 00:00:15 00:00:15 00:00:15
Table 30
Subtest 700 705 710 715 720 725 730 735 740
IO3000 Class 7 Subtests (DMA)

Description External interrupt test DMA across page and channel Jump forward within a page Jump backward within a page Jump outside of a page (TLB encached) Jump outside of a page (TLB not encached) Jump outside of a channel Non contiguous TLBs DMA byte swapping Test time (1 SAGA) 00:00:15 00:00:15 00:00:15 00:00:15 00:00:15 00:00:15 00:00:15 00:00:15 00:00:15
91
Troubleshooting Hints I/O Subsystem FRUs Table 31

Subtest 800 805 810
IO3000 Class 8 Subtests (Multi-Disks)

Description Multi-disk non-mixed traffic Multi-disk mixed traffic, ATPR = 0xe Multi-disk mixed traffic, ATPR = 0xf Test time (1 SAGA) 00:03:30 00:03:30 00:03:30
Table 32
Subtest 1200 1205 1230 1240 1250
IO3000 Class 12 Subtests (Symbios SCSI PCI Cards)

Description Symbios SCSI PCI configuration space test Symbios SCSI PCI I/O and Memory space test Symbios SCSI Scripts RAM test Symbios SCSI Interrupt test Symbios SCSI DMA engine test Test time (1 SAGA) 00:00:10 00:00:10 00:00:10 00:00:40 00:00:10
Executing IO3000
Using IO3000 test controller diagnostic requires the use of a switch and few options to the cxtest command. These are: IO3000 Switch io - This switch causes cxtest to execute the IO3000 diagnostic. Options c - To select an entire class of subtests use the -c <number> option. The user can specify a range of classes by using a hyphen between the numbers. As an example -c 2-4, would run classes 2,3 and 4. The user can also specify a list of classes by placing a "," between the numbers. An example would be -c 4,7,2. This would run class 4 then class 7 and finally class 2. s - To select a subtest use the -s <number> option. The user can specify a range of subtests by using a hyphen between the numbers. As an example -s 100-150, would run subtests 100 through 150.
92
The user can also specify a list of subtests by placing a "," between the numbers. An example would be -s 100,150,140. This would run subtest 100 then subtest 150 and finally subtest 140. pa# - To select a parameter word use the -pa# <number> option. The # symbol represents the parameter word to use. The test controller environment allows IO3000 to have 20 words of user parameters. Words 8 19 are used for device specification. Due to Core Logic SRAM space limitations, only 20 devices per SAGA can be tested at a time. Up to 24 devices can be specified via parameter words 8-19. Each of the parameter words is broken down into 2 device specifications as explained in Figure 33. Devices are numbered according to their position in the parameter list. A device can be specified in any of the device specification locations in user parameter space. An unused device parameter should be initialized such that the slot field is 0xf (i.e. device spec of 0x0f00). Therefore if both device parameters in a given parameter word are unused, the parameter word would be set to 0x0f000f00. As an example, to specify a disk on SAGA 0x4, slot 0x2, SCSI id 0xa, SCSI lun 0x0, one might set parameter word 8 to 0x42a00f00. Note that the lower (right) half of the parameter word has the slot field set to the 0xf. The device number is 0 since it was entered in device 0 parameter location.
93
Troubleshooting Hints I/O Subsystem FRUs Figure 33 Parameter Words 8 through 19
Selecting Devices with Parameter Words 8 through 19

Parameter Word 8 Parameter Word 9 DEV 0 DEV 2 DEV 1 DEV 3
**************THROUGH**************
Parameter Word 19 LUN # SCSI ID # SLOT # EPIC #
DEV 22
DEV 23 EPIC # SLOT # SCSI ID # LUN #
Parameter Word 8 = 0 1 0 0 0 F 0 0 Parameter Word 9 = 4 2 A 0 5 1 6 0 0F00 = No device selected
The figure on the following page shows an example of booting the system into test controller standalone mode and executing IO3000 using the command line interface. The command line is using parameter word 8 to execute the class 12 Symbios SCSI Card tests for a the device installed on SAGA 1 in slot 0 and subtests 500 through 515 to test the SCSI disk drive which is installed on SAGA 1 in slot 0 at address 3 with a LUN number of 0.
WARNING In Table 28, notice that subtest 520 is a SCSI disk write test. If this test is selected, data on the disk drive will be destroyed. Also notice that the example in Figure 34 is not executing subtest 520.
94
Troubleshooting Hints I/O Subsystem FRUs Figure 34 Executing IO3000 from the command line interface
fesmesst:/users/sppuser$ do_reset 0 1 tc_standalone 0:0xf000d00a90 0x01000000 0x01000000 fesmesst:/users/sppuser$ cxtest -io -pa8 0x10300f00 -c 12 -s 500-515 1 Nodes found Command line 0 : -io : Number of args 7 Class 12 Subtest 1200 - Symbios SCSI PCI configuration space test 0:00:10 passed Subtest 1205 - Symbios SCSI PCI I/O and Memory space test 0:00:09 passed Subtest 1230 - Symbios SCSI Scripts RAM test 0:00:09 passed Subtest 1240 - Symbios SCSI Interrupt test 0:00:26 passed Subtest 1250 - Symbios SCSI DMA engine test 0:00:09 passed Class 12 passed Subtest 500 - SCSI Disk test unit ready test 0:00:15 passed Subtest 505 - SCSI Disk inquiry test 0:00:15 passed Subtest 510 - SCSI Disk read capacity test 0:00:16 passed Subtest 515 - SCSI Disk read test 0:00:12 passed
0:0 0:0 0:0 0:0 0:0
0:0 0:0 0:0 0:0
Table 33 provides a listing of values that could be used for parameter word 8 to test a single disk device for each one of the possible SAGA and slot numbers in a V2500/V2600.
NOTE
In the following table, x represents the SCSI ID number; y represents the LUN number
Table 33
Possible values for Parameter word 8. SAGA 4 4 4 0 0 0 0 1 2 0 1 2 Slot pa8 value 0x40xy0f00 0x41xy0f00 0x42xy0f00 0x00xy0f00 0x01xy0f00 0x02xy0f00
95
SAGA 0 5 5 5 1 1 1 1 6 6 6 2 2 2 2 7 7 7 3 3 3 3 3 0 1 2 0 1 2 3 0 1 2 0 1 2 3 0 1 2 0 1 2 3
Slot
pa8 value 0x03xy0f00 0x50xy0f00 0x51xy0f00 0x52xy0f00 0x10xy0f00 0x11xy0f00 0x12xy0f00 0x13xy0f00 0x60xy0f00 0x61xy0f00 0x62xy0f00 0x20xy0f00 0x21xy0f00 0x22xy0f00 0x23xy0f00 0x70xy0f00 0x71xy0f00 0x72xy0f00 0x30xy0f00 0x31xy0f00 0x32xy0f00 0x33xy0f00
96
Troubleshooting Hints Power Subsystem FRUs
Power Subsystem FRUs

+48VDC Power Supplies
New SPPDSH command pswitch reports on the state of the n+1 switch on the SCUB. The SPPDSH command pce is used to identify if an environmental error exist for a node power supply (NPS).The table below identifies the failed NPS when any of the four hexadecimal codes are found.
Table 34 Node Power Supply Status Code Identification Hex Code 0x70 0x71 0x72 0x73 Node Power Supply Upper Left Upper Right Lower Left Lower Right
97
Troubleshooting Hints Cooling Subsystem FRUs
Cooling Subsystem FRUs

N+1 Cooling Fan Panels
The table below identifies the failed cooling fan when any of the six hexadecimal codes are found using the SPPDSH command pce. Upper Right implies a fan on the right, inside the top most fan panel when viewed from the rear of the system. Lower Left implies a fan on the left, inside the bottom most fan panel when viewed from the rear of the system.
Table 35 Cooling Fan Status Code Identification Hex Code 5c 5d 5e 5f 60 61 NOTE Cooling Fan Upper Right Upper Middle Upper Left Lower Right Lower Middle Lower Left
Hexadecimal codes 5d and 60 are not implemented in V22x0 servers as they only use four cooling fans (two fans per panel).
98
Troubleshooting Hints Mass Storage FRUs
Mass Storage FRUs

PCI Cards
Two different choices here: F/W SCSI card for HVD devices and Ultra 2 SCSI card for LVD devices.
SCSI Devices
Both LVD and HVD disk types cannot be mixed on the same SCSI Bus.
99
Troubleshooting Hints Service Support Processor Utilities
Service Support Processor Utilities

CCMD Differences
ccmd is the daemon that maintains a shared memory buffer of information on the V2500/V2600 hardware. ccmd also monitors the system and reports any significant complex events back to the SSP. There are two types of related information in the shared memory buffer: cabinet information (node numbers and IP addresses) of the complex and scan information, which is derived from the hardware itself. ccmd periodically broadcasts to determine what cabinets are available. If ccmd cannot talk to a cabinet that it previously reached, it sends an event to the console and the log. If it establishes or reestablishes contact, or if a cabinet powers up, ccmd will read hardware information in EEPROM. At power on of the system, ccmd performs a scan operation to build the boot configuration map. The boot configuration map contains information about the installed components. POST uses the boot configuration map to test the installed components. Once running, ccmd checks for power-ups, power-downs, errors and environmental conditions at regular intervals. If at any time ccmd detects a change in the configuration, it changes the database, and writes the /spp/data/config/complex.cfg file for each applicable cabinet. These files are required for Excalibur System Test. If ccmd detects an error condition, it kicks off an error interrogator that logs and diagnoses error conditions. ccmd is no longer responsible for building the node_0.cfg database file that was used by EST. This responsibility has been replaced by the est_config command.
100
EST Differences
EST, est_config & the node_0.cfg file Though nothing about how EST works should have changed, there are some things that EST users need to be aware of for use on the V2500/V2600. EST will now use the est_config command to generate the node_0.cfg file EST uses to get information about the system under test. This is a change in the way the node_0.cfg file is generated. To scan a system, EST needs various information about that system. ccmd must be running on a SSP to gather certain data about the system. When ccmd is done checking a system, EST can be run. EST will then decide whether or not to run est_config to generate the node_0.cfg file in /spp/data. When ccmd has done its job and EST has decided (by itself or with user input) whether or not to run est_config, a user should be ready to start scanning. EST decides whether or not to run est_config by looking at the node_0.cfg and node_0.pwr files. If the machine was power cycled since est_config was last run, EST will automatically run est_config to generate the node_0.cfg file. If the node_0.cfg was updated after the power file, est_config will NOT be run. The node_0.cfg file is generated by est_config when run, and the node_0.pwr file is generated by ccmd when the machine is power cycled. Both the node_0.pwr and node_0.cfg files usually live in /spp/data. A user can tell EST to run est_config with the -Y option (ex. prompt% est -Y 0). This option runs est_config regardless of the timestamps on the power and config files. A user can also tell EST not to run est_config with the -Z option (ex. prompt% est -Z 0). Though using this option is not recommended, -Z will tell EST not to run est_config even if the power file is newer than the config file.
101
Hard_Logger Master ID Differences

The Master ID is a 6-bit field extracted from the failed MACs Error Info CSR bits[31:36] and must be manually decoded to identify the Initiator of the request. The Master ID value is placed into a hard_logger output when appropriate. In the case of the hard_logger event Responseless Request or Response to incorrect state, the extractor r_incorrstate_resples_req provides the user with the Master ID field.See Web-based Troubleshooting for V22x0/V2500/V2600. The 6-bit field is really two separate 3-bit fields: source (bits[31:33]) and device # (bits[34:36]). See the two tables and figures below for specific details.
Table 36 Decoding the Master ID Value for V22x0 FRUs
Source (Bits[31:33])
Value = 0 is Runway 0 (PBnL) Value = 1 is Runway 1 (PBnR) Value = 2 is I/O (EPIC) Value = 3 is (unused) Value = 4 is Data Mover Value = 5 is JTAG Value = 6 is ETAC Value = 7 is EMAC
Device # (Bits[34:36])
Value = 0 is ASIC #0 Value = 1 is ASIC #1 Value = 2 is ASIC #2 Value = 3 is ASIC #3 Value = 4 is ASIC #4 Value = 5 isASIC #5 Value = 6 is ASIC #6 Value = 7 is ASIC #7
102
Troubleshooting Hints Service Support Processor Utilities Figure 35

DC to DC Power Supply DC to DC Power Supply
V22x0 Processing Module pair and EPAC

D-Cache PCXU+ I-Cache DC to DC Power Supply DC to DC Power Supply
PBnR
D-Cache PCXU+ I-Cache
0
R U N W A Y
PBnL
1 to EPIC
V25035.ppt
Table 37
Decoding the Master ID Value for V2500/V2600 FRUs
Source (Bits[31:33])
Value = 0 is Runway 0, CPU A (PBnL) Value = 1 is Runway 1, CPU A (PBnR) Value = 2 is I/O (SAGA) Value = 3 is Runway 1, CPU B (PBnR) Value = 4 is Data Mover Value = 5 is JTAG Value = 6 is STAC Value = 7 is Runway 0, CPU B (PBnL)
Figure 36 V2500/V2600 Processing Module pairs and SPAC
PBnR
DC to DC Power Supply DC to DC Power Supply PCXW CPU A PCXW CPU B DC to DC Power Supply DC to DC Power Supply PCXW CPU A PCXW CPU B
Device # (Bits[34:36])
Value = 0 is ASIC #0 Value = 1 is ASIC #1 Value = 2 is ASIC #2 Value = 3 is ASIC #3 Value = 4 is ASIC #4 Value = 5 isASIC #5 Value = 6 is ASIC #6 Value = 7 is ASIC #7
0
R U N W A Y
PBnL
to SAGA
V25040.ppt
103
104
5)
Theory of Operation
Introduction
This chapter is not intended to explain all V2500/V2600 functions in detail but to give some explanation of some important functions to better understand the differences from V22x0 in the following areas: V-Class H/W block diagrams showing balanced parallelism PA8500/PA8600 adaptations V-Class memory addressing formats V-Class Memory Board block diagrams showing DIMM accessing Miscellaneous V2500/V2600 subsystem functions
105
Theory of Operations
Figure 37
1X PIC I O L F
2 1 0
V22x0 System Block Diagram

PB0R (B) 0 (A) 4 (B) 2 (A) 6 (B) 1 (A) 5 (B) 3 (A) 7
U U 1 0 P0L Agent P1R Agent R2R XBar P2L Agent P3R Agent R0L XBar MB0L EMAC MB2R EMAC MB4L EMAC MB6R EMAC R3R XBar MB1L EMAC MB3R P5R Agent R1L XBar EMAC MB5L EMAC MB7R P7R Agent EMAC CTI CTI CTI CTI CTI CTI CTI CTI
PB0L PB1R
U U
1 0
1X PIC
2 1 0
PB1L
U
PB2R
U
0 1
1X PIC I O R R
2 1 0
PB2L PB3R
U U
1 0
1X PIC
2 1 0
PB3L 3
U
PB4R
U
0 1 P4L Agent
2 1 0
1X PIC 1X PIC
PB4L 4
U
I O L R
PB5R
U
1 0
PB5L PB6R
U U
2 1 0
0 1 P6L Agent
2 1 0
1X PIC 1X PIC
PB6L 6
U
I O R F
PB7R
U
1 0
PB7L Single-PCXU+ EVPBs
2 1 0
32-bit PCI EIOBs
B0S0
B3S4 B1S0
MIDPLANE (ENRB)
B2S4
MEMORY EMBs
v25015.ppt
The above figure shows all the V22x0s main computing FRUs, their inter-connections and FRU numbering. DIMM slot locations are shown on the right side of the slide as the 16 short vertical lines within the eight Memory Boards. Note that the relative location of the first DIMM for Banks 0 - 3 are identified using 4 bolder vertical bars within each Memory Board.
Key Points
All EPIC (B) memory accesses use R0L and R2R X-Bar chips; all EPIC (A) memory accesses use R1L and R3R X-Bar chips. Maximum configuration of PCI cards and EIOBs are evenly distributed (or balanced) across all available memory configurations. Memory accesses are the bottleneck to increased performance.
106
Theory of Operation
Figure 38
(B) 0 (A) 4 (B) 2 (A) 6 (B) 1 (A) 5 (B) 3 (A) 7
V2500/V2600 System Block Diagram

PB0R
2X PIC W W W W 1 1 0 1 P1R Agent R2R XBar P2L Agent P3R Agent R0L XBar 0 P0L Agent MB0L SMAC MB2R SMAC MB4L SMAC MB6R SMAC R3R XBar MB1L SMAC MB3R P5R Agent R1L XBar SMAC MB5L SMAC MB7R P7R Agent SMAC CTI CTI CTI CTI CTI CTI CTI CTI
PB0L PB1R
W W W W
I O L F
3 2 1 0
2X PIC
2 1 0
PB1L
W W
PB2R
W W
0 1
2X PIC I O R R
3 2 1 0
PB2L PB3R
W W 3 W W
1 0
2X PIC
2 1 0
PB3L
W W
PB4R
W W
0 1 P4L Agent
2X PIC I O L R
3 2 1 0
PB4L 4
W W
PB5R
W W
1 0
2X PIC
2 1 0
PB5L PB6R
W W W W
0 1 P6L Agent
2X PIC I O R F
3 2 1 0
PB6L 6
W W
PB7R
W W
1 0
2X PIC
2 1 0
PB7L Dual-PCXW EWPBs
Q0B0
Q0B3 Q0B1
64-bit PCI SIOBs

v25010.ppt
MIDPLANE (MIB)
Q0B2
MEMORY EWMBs
The above figure shows all the V2500/V2600 main computing FRUs, their inter-connections and FRU numbering. DIMM slot locations are shown on the right side of the slide as the 16 short vertical lines within the eight Memory Boards. Note that the relative location of the four DIMMs within Quadrant 0 are identified using 4 bolder vertical bars within each Memory Board.
Key Points
All SAGA (B) memory accesses use R0L and R2R X-Bar chips; all SAGA (A) memory accesses use R1L and R3R X-Bar chips. Maximum configuration of PCI cards and SIOBs are not evenly distributed (or balanced) across all available memory configurations. X-Bar accesses are the bottleneck to increased performance.
107
Theory of Operations Processor Modules
Processor Modules
PA8500/PA8600 Adaptations for V2500/V2600
Several considerations are needed to allow sharing a common Runway Bus by two PCXW or PCXW+ CPUs on the same EWPB. For the V2500/V2600 implementation, a single PCXW/PCXW+ chip per board must be loaded in the socket for CPU A. The socket for CPU B does not function when the socket for CPU A is not loaded. See figure below for details.
Figure 39 Runway Bus Signal Sharing between PCXW CPUs
DC to DC Power Supply DC to DC Power Supply
0
T
64 32
CPU A
T
CPU B
32
R U N W A Y
from X-Bar PnL Agent
1
V25045.ppt
to X-Bar
to SAGA
When both CPUs are installed, a different set of 32 signal lines from the shared Runway Bus are terminated by each processor chip to equalize skew, loading, etc. This is not the case, however, when only CPU A socket is loaded as all 64 signal lines are terminated by CPU A. For some CPU A faults, the entire processor board may appear to not function and would be de-configured by POST.
108
Theory of Operation Processor Modules
Runway Bus Arbitration

Unlike K-Series where the host manages arbitration for all of Runways clients, Runway arbitration is determined by the CPU pairs. The Processor Agent Controller (SPAC), as the host for both Runway Buses, is not involved. Since Runway buses have not changed, single threaded applications that are Runway Bus limited today are not expected to have any significant performance increase. Applications that are not Runway Bus limited will see increased performance that is only limited by memory latency and internal CPU instruction execution rate. The increased CPU clock rate increases the internal execution rate by a proportionate amount relative to V22x0, although there is some degradation due to the smaller data and instruction caches within the PCXW and PCXW+. The maximum application performance increase is 1.8x (the ratio of the PCXU+ to PCXW core clock rate increase). Other internal CPU related changes that result in performance increases are: 4-way set associative data and instruction caches, more TLB entries, and a larger branch history table.
109
Theory of Operations Memory Modules
Memory Modules
The V2500/V2600 memory modules design greatly improves sustainable memory bandwidth. These improvements are a result of allowing more memory operations to occur concurrently as well as increasing the number of managed banks of coherent memory. The result of these improvements is an approximate 2x increase in maximum sustainable memory bandwidth (~1.7x for real applications) compared to SPP2000/V22x0 hardware. The latency of the memory system as measured from the processor for an idle system remains at roughly 500 nano-seconds. There is a small latency improvement due to the PCXW and PCXW+ clock rate increase (and thus a corresponding decrease in the amount of time to issue memory requests). Theres a large decrease in latency when the system is not idle due to the increase in sustainable bandwidth of the memory system. This latency decrease is observed as better scalability of multithreaded applications. The design requirement for the memory system yields a 2x sustainable memory bandwidth increase when comparing the latency versus bandwidth aspects of the SPP2000/V22x0 versus the V2500/V2600. With all the changes to increase memory bandwidth, the format for the 40-bit coherent memory address also changes for V2500/V2600.
110
Theory of Operation Memory Modules
V-Class Memory Addressing Formats

Like the PA8200 CPU, the PA8500 and PA8600 CPU conforms to the same 40-bit Runway Bus addressing format. Unlike the V22x0 before it, the V2500/V2600 architecture does not support the same 40-bit addressing format, it uses yet another format. Unfortunately, Runways addressing format is different than the 40bit addressing formats used by V-Class systems. Each V-Classs implementation is streamlined to get maximum performance from PA-RISC processors. As such, memory configurations only support balanced memory board and DIMM slot configurations that get maximum performance. The V22x0s addressing format maps a modified 40-bit Runway Bus memory address to a 16GB coherent memory address space; the V2500/V2600 addressing format maps it to a 32GB coherent memory space. Memory configurations with less than the maximum number of memory boards and or less than the maximum number and size of DIMMs will leave gaps or holes within the memory space. But noncontiguous coherent memory space does not inhibit performance. Both V-Class memory addressing formats require some knowledge of the memory interleaving algorithms used by EPAC/SPACs to modify the original 40-bit Runway Bus address to correctly map 32 byte cache lines into the appropriate DIMM on the appropriate board, bus/ bank, and row for storage and retrieval. Utilities are available either from the SSP or the web to decode either 40-bit coherent memory address formats. Figures 40 and 41 show the V22x0 and V2500/V2600 40-bit addressing formats and how they differ from Runways format. Figures 42 and 43 show the V22x0 and V2500/V2600 memory board DIMM accesses.
Theory of Operations Memory Modules Figure 40

24
Not Used
V22x0 Coherent Memory Address Modification

-30 - Memory Page 51-52 - Page Offset 51-52 - Page Offset
5 VR RI VB BI
58-59
63
- Line Offset 58-59-60-61- 63 - B - E - H- 00
24
28 -30 32-33 -36 -38
Node ID 0 - Row - VR- VB - Memory Page
Mem Block
3
Mem Bank Interleave
Memory Board Map
14
Note: Bit 29 of theV22x0s 40-bit memory adddress is not used.

V25025.ppt
All V22x0 EPACs must modify the Runway 40-bit coherent memory addresses from up to two CPUs to interleave all banks on all EMBs. This is accomplished prior to routing the memory addresses through the X-Bar to a specific (even/odd) EMB. Two decisions are derived by the interleaving algorithm that examines bits[33:37] and bits[52:58] of the 40-bit Runway address on top. Not shown are the different bit combinations chosen within the Page Offset bits dependent upon the total number of EMB pairs. The EMB # and the bank #, derived by the algorithm, are written into bits[33:37] and the bank value is repeated in bits[59:61] of the 40-bit V22x0 address. This new 40-bit address is included within the packet along with the transaction type and sent to the EMB # derived from bits[33:35]. Each EMAC, upon receiving the packet, uses Row # from bits[30:32] and Bank # from bits[59:61] to access the appropriate DIMM slot. The EPAC uses the unchanged value of bit [58] to determine which of the two X-Bar ports (even/odd) to route the packet to the EPAC.
Theory of Operation Memory Modules Figure 41

24
V2500/V2600 Coherent Memory Address Modification

-29 51-52 - Page Offset 51-52 Lower Page Page Offset
8
Not - Memory Page Used
58-59 63 - Line Offset 58-59 63 Line Offset
24 28-29 30-31 33 40-41 Node Upper Interleave 0 ID Page Row Base

8
Interleave Base
Interleave Index
Force Node ID Function
Memory Row Maps
Memory Bank Interleave
Memory Bus Interleave
Memory Board Interleave
3 4 2 2 2 3
Memory Board Map
11
Note: Bit 24 of the V2500s 40-bit memory address is not used.

V25030.ppt
All V2500/V2600 SPACs must modify the Runway 40-bit coherent memory addresses from up to four CPUs to interleave all buses and banks on all EWMBs. This is accomplished prior to routing the memory addresses through the X-Bar to a specific (even/odd) EWMB. Four decisions are derived by the interleaving algorithm that examines bits[33:40] and bits[51:58] of the 40-bit Runway address on top. Not shown are the different bit combinations chosen within the Page Offset bits dependent upon the total number of EWMB pairs. The EWMB #, the bus #, and the bank #, derived by the algorithm, are written into bits[33:40] of the 40-bit V2500/V2600 address. This new 40-bit address is included within the packet along with the transaction type and sent to the EWMB # derived from the algorithm. Each SMAC, upon receiving the packet, uses Row # from bits[31:32], Bank # from bits[33:34], Bus # from bits[35:37] and Board # from bits[38:40] to access the appropriate DIMM slot and side.
113
The SPAC uses the unchanged value of bit [58] to determine which of the two X-Bar ports (even/odd) to route the packet to the SMAC.
Figure 42 V22x0 Memory Block Diagram
Rows: 0
A
Banks 0
X X E T A C Y Y
E O B0 S0 E M A C
Even
E O B0 S1 E O B1 S1 E O B2 S5 E O B3 S5
E O B1 S0 E O B2 S4 E O B3 S4
Odd
DIMMs
V25060.ppt
V22x0 Memory Interleaving

Sequential addresses are interleaved up to 32-way in the following order: first by Board #, then by (Even/Odd) Bank #, and lastly by Row #. The two sides of a DIMM (Even/Odd) have a logical association with Row numbers (shown within each DIMM above).
NOTE The Row bits provide the starting point for interleavings modulo sequencing.
Supports 2/4 bank interleaving for both like/mixed DIMM sizes up to 32-way. The V22x0s memory boards can support up to two memory accesses (one Even Bank and one Odd Bank) simultaneously to any Row.
114
Theory of Operation Memory Modules
EMBs have many memory holes when all DIMM slot are not occupied with the largest DIMM size. Contiguous memory available to HP-UX is only 32 MB with the smallest DIMM size and is therefore not supported in Rows 0/1.
Figure 43 V2500/V2600 Memory Block Diagram
Q0B1
Q0B2
Q0B3
Q1B4
Q1B5
Q1B6
Rows: 0
1
A
0 1
Q0B0
0 1
1
0 1
2
0 1
3
DIMMs
0 1
4
0 1
5
0 1
6
Q1B7
0 1
7
X X S T A C
Bus: 0
NBC S M A C
Bus: 0
NBC
Quad 1
Quad 0
Quad 2
Quad 3
NBC
B
1 2 3 4
DIMMs
NBC
5 6 7
Y Y
Rows: 2
0 1
Q2B0
0 1
Q2B1
0 1
Q2B2
0 1
Q2B3
0 1
Q3B4
0 1
Q3B5
0 1
Q3B6
0 1
Q3B7
V25065.ppt
V2500/V2600 Memory Interleaving

Sequential addresses are interleaved up to 256-way in the following order: first by Board #, then by Bus #, and lastly by (Even/Odd or by Odd/Even) Bank #. The two sides of a DIMM (0 & 1) have a logical association with Bank numbers (shown within each DIMM above). A single DIMM on a Bus represents 2 Banks (0 & 1); two like sized DIMMs on the same Bus represent 4 Banks (0-3); two different sized DIMMs on the same Bus represent Banks (0 & 1) for each DIMM.
NOTE The Row bits provide the starting point for interleavings modulo sequencing. HP Company Confidential 115
For mixed DIMM sizes: supports 32 MB and 64 MB size DIMM combinations or 64 MB and 128 MB DIMM combinations or 32 MB and 128 MB DIMM combinations. For like DIMM sizes: supports Quadrants 0-3 (4/8-bus 2/4 bank) interleaving up to 256-way. Quadrants 0-3 may contain any supported DIMM size combination, however, the best performance is achieved with the most interleaves. The V2500/V2600 memory boards can support up to eight memory accesses (one per Bus) simultaneously to any Row or logical Bank. EWMBs have fewer memory holes when all DIMM slots are not occupied with the largest DIMM size. Contiguous memory available to HP-UX is now 64 MB with the smallest DIMM size and is supported in Rows 0/1 of Quadrant 0.
Commercial vs. Technical Systems

V2500/V2600 servers uses two modes of operation of the coherency protocol to optimize system performance for commercial versus technical systems. 1.) Commercial systems use two new Runway bus transactions. These transactions are notification of private clean cast outs and cache-tocache writes for 3rd party access to private clean data. The notification of private clean cast outs results in a 10-12% increase in performance for TPC-C applications. The cache-to-cache writes for 3rd party access to private clean data results in additional TPC-C application performance due to decreased access latency observed by a processor for 3rd party private clean accesses. 2.) For technical systems the coherency protocol includes the V22x0s read hint mechanism to increase sustainable memory bandwidth. Technical systems disable notification of private clean cast outs. These changes result in an increase in usable memory bandwidth of roughly 30%. This usable memory bandwidth increases SPECrate_fp95 results by 10%.
Theory of Operation I/O Modules
I/O Modules
The increase in peak I/O bandwidth for the SAGA I/O adapter allows a single PCI controller to achieve a larger peak bandwidth. However, in order to increase the aggregate system I/O throughput of the V2500/ V2600 requires reworking system software at a later date. The I/O subsystem supports the 2x PCI bus. This will double the peak I/O bandwidth capabilities of the hardware, to approximately 174MB/ sec peak sustainable data transfer rate (bus transfer rate is 240MB/sec, but doesnt count address and other overheads). I2O features for PCI cards are not supported by SAGA. The extensions required for supported 64-bit PCI cards are documented in the I/O chapter of the V2500/V2600 Architecture Specification.
Key Points
SAGA supports PCI to PCI bridges, albeit, with limited performance. The SAGA DMA engine is designed as a pipe between a PCI controller and main memory. Every time a controller switches its address stream, it incurs a start-up penalty, equivalent to the time it takes to re-prime or flush the pipe. For PCIbound DMA Writes, re-priming the pipe consists of updating SAGAs channel context state, fetching page table pointers from system memory, and then fetching data from system memory. For MEMbound DMA Reads, the flow consists of coherently flushing the previous data, updating the bridges channel context state, and fetching page table pointers from system memory. Due to these large overheads per address jump, the best performance is demonstrated by long transfers on PCI where the transfer time of the DMA Reads make this start-up overhead insignificant. A PCI to PCI bridge or multi-port controller implemented using a PCI to PCI bridge, has just the opposite transfer pattern from the preferred pattern described above. Since the PCI bridge is arbitrating between
Theory of Operations I/O Modules
multiple controllers on a secondary PCI bus, it turns long bursts from multiple controllers on the secondary bus into multiple small bursts (64 bytes is common) on the primary bus. The transfer time of these small bursts on the primary bus is dwarfed by the address switching start-up overhead, limiting the achievable performance through the bridge. This effect has been calculated to be between 7 to 15 MB per second maximum throughput when multiple controllers are active behind a bridge, as compared to a single controllers maximum supported throughput rate of 174 MB per second. Although SAGA will functionally support a PCI/PCI bridge controller, deliverable bandwidth will be limited to less than 15MB per second when switching between multiple addressing streams through the bridge.
118
Theory of Operation Mid-Plane
Mid-Plane
Functional review of main computing FRUs
There are up to 32 CPUs in a V2500/V2600 server with 1 or 2 CPUs per processor board. Two CPUs on the same board must share the Runway Bus. Each SPAC must accommodate up to two processor boards, via their Runway Buses, and a SAGA I/O ASIC. The X-Bar, implemented on the MIB or mid-plane, allows up to eight SPACs to simultaneously transfer 32 byte cache line packets to any of up to eight SMACs. The core V2500/V2600 operations are centered around the ASIC interactions that shuttle packets (of instructions or data cache lines) between coherent memory space and each CPUs cache memory. Other operations include SAGA DMA cache line transactions to coherent memory space and CPU (LOAD/STORE) and SPAC "unwedger" (get/put) transactions to CSR space. The V-Class version of Scalable Computing Architecture (SCA, formerly ccNUMA) implements cache coherency checking within each SMAC. The CPUs assume they are interacting with another CPU or Memory module over their shared Runway Bus. However, each CPU is only interacting directly with an SPAC. Each SPAC has access to all SMACs via the X-Bar. In order to maintain cache coherency, each SMAC must occasionally request a change of ownership of a private data cache line from one CPU to another CPU. This requires the SMAC to have access to all CPUs via the X-Bar.
119
Theory of Operations Power Modules
Power Modules
Processor module power supply
The V2500/V2600 processor module required a total redesign of the +48VDC to low voltage power conversion hardware on the module. The change removes the existing +3.3VDC @ 50 Amp, +1.5VDC @ 6.5 Amp, and -1.9VDC @ 1.5 Amp converters. Power for the new processor pair will be provided by two +2.0V @ 50 Amp converters or an equivalent single converter. A +1.6V @ 3 Amp converter will power termination and a +3.3VDC @ 0.5 Amp converter will power the SPHYR clock chip. The redesign lives within available power, thermal, and control line constraints and is compatible with the existing systems infrastructure.
120
Theory of Operation Cooling Modules
Cooling Modules
Processor module thermal
Processor boards are air cooled using forced convection with 250 fpm (minimum) across the (EWPBs). Each board may have up to two processors cooled by low cost aluminum heat sinks with heat pipes imbedded in the base. To account for the additional CPU on each module, the cooling area was expanded to utilize the space above the CPUs (in addition to the fin area used on V22x0 processor heat pipe). This compensates for the preheated air generated by the 3 upstream CPUs.
121
Theory of Operations Mass Storage Modules
Mass Storage Modules
122
Theory of Operation Service Support Processor

Universal SSP
123
Theory of Operations Service Support Processor
124
A)
Acronyms
Introduction
The release of the V2500/V2600 hardware platform introduces a new HP 9000 Series CPU for Hewlett Packard and new set of acronyms. During the design phase of this product a code name of Stingray was adopted and used in the acronyms. V2500/V2600 use the same system architecture as the V22x0 except that it has been enhanced. The acronyms for the V2500/V2600 have also been enhanced. The following tables list the acronyms for the V2500/V2600 and the V22x0 hardware platform. The tables provide you with a fully qualified explanation of the acronym, where the component is located within the system and some supporting text.
125
Acronyms V22x0 Acronym List
V22x0 Acronym List

Fully Qualified Is Located On
Acronym
Supporting Text
ECUB
Exemplar Core Utility Board Exemplar Node Routing Board Exemplar Memory Board Exemplar Processor Agent Controller Exemplar Memory Access Controller Exemplar PCI Controller
The Midplane The Midplane
The ECUB is the heart of the power on self test, environmental monitoring, scanning, system clocks, as well as many other support issues. The ENRB is the largest circuit board in the VClass system and is also known as the Midplane. This is where the HyperPlane Crossbar backplane is located as well as all of the Processor Agent Controllers (EPACs). The EMB has several plug-on modules to complete the functionality of the memory subsystem. Each pair of processors and one half (3 PCI slots) of an average I/O card cage connect to a Processor Agent Chip (or PAC). Each PAC chip communicates directly with the HyperPlane with NO bus waiting. There is a custom VLSI chip for the Memory Access Controller and there may be an optional VLSI chip for the V-Class technical SPU called Toroidal Access Controller (TAC). Each I/O Card Cage contains six total PCI slots for connecting PCI based I/O device adapter cards. There are two EPIC chips located inside each I/O card cage. Each EPIC chip controls three PCI slots and is connected to a Processor Agent Controller (PAC) chip which is located on the HyperPlane Crossbar Backplane.
ENRB
EMB
The Memory Board The Midplane
EPAC
EMAC
The Memory Board
EPIC
The EIOB
126
Acronym
Fully Qualified
Is Located On
Supporting Text
E48VB
Exemplar 48 Volt Board
The Upper Rear Chassis
The Node Power Supply (NPS) provides +48V DC to the two +48V Boards (E48VB) which simply route the +48V throughout the node to the fans and the DC - DC converters. The two E48VBs attach together to form a single "bus bar" type connection. The E48VBs are located in the rear chassis, behind the ENRB. The EVPB contains a circuit which detects when the +3.3V DC drops below specification and will signal a power failure to the Core Utilities Board (ECUB). The EVPB contains an overtemp sensor and reports overtemp conditions to the ECUB. The DC - DC converters are part of the EVPB. Failure of one of the converters will require replacement of the EVPB. The EMBPB sits atop the Memory Board and connects to the EMB through the Memory Power Board Extender (EMBPX). The EMBPX is a field replaceable unit. The EMUC is a non field replaceable FPGA located on the ECUB. It functions as the environmental monitor and power controller. Both the EMUC and power-on circuit work together to control power. The EPUC is a non field replaceable FPGA located on the ECUB. Core logic functions are controlled by the EPUC via the 8 EPAC ASICs. System initialization and CPU interrupt handing are the EPUCs main functions.
EVPB
Exemplar Vulcan Processor Board
The Processor Board
EMBPB
Exemplar Memory Board Power Board Environment al Monitor Utility Controller Exemplar Processor Utility Controller
The Memory Board
EMUC
The Utility Board (ECUB)
EPUC
The Utility Board (ECUB)
127
Acronym
Fully Qualified
Is Located On
Supporting Text
ERAC
Exemplar Routing Access Controller Exemplar I/O Board Exemplar Torrodial Access Controller
The Midplane
The ERAC is a field replaceable ASIC which is located on the Midplane. Four ERACs make up the HyperPlane XBAR. Each ERAC has 16 discrete ports with NO bus waiting! Each EIOB contains six PCI card slots. Up to four EIOBs can be installed in a V-Class system for a total of 24 PCI card slots. The ETAC ASIC will only be installed in XClass (multi-node) systems. Globally shared memory is accessed between nodes via the ETAC. Off node memory accesses are routed from the ETAC to either main memory or the XBAR through the EMAC. Up to 16 DIMMs can be installed on each memory board (EMB). 80 bit DIMMS can only be used in single node systems. 88 bit DIMMs are required in all multi-node systems. The extra 8 tag bits are needed for cache coherency between hypernodes. These DIMMs will function along side the 80 bit DIMMs in a single node system. The EMBPX is a small extension board that buses the 48VDC input and logic level voltages between the EMB and the EMBPB. The ECCB connects the keyswitch to all four power supplies and the RS232 ports from the bulk head connector to the ECUB. Environmental monitoring signals from the power supplies and fans also route through the ECCB to the ECUB.
EIOB
The I/O Card Cage The Memory Board
ETAC
EPMB
Exemplar Plug-in Memory Board Exemplar Plug-in Memory Board (Multi-node) Exemplar Memory Board Power Extender Exemplar Cable Connector Board
The 80 Bit DIMMs
EPMBM
The 88 Bit DIMMs
EMBPX
The Memory Board
ECCB
the Upper Rear Chassis
128
Acronym
Fully Qualified
Is Located On
Supporting Text
ELEDB
Exemplar LED Board
The left front side of Chassis
The ELEDB is the attention light for the system. The normal state is solid on across all segments. When flashing either an environment error or system crash has occurred. Test station utilities must be used to determine the error state when flashing. All ASICs are field replaceable and require a specified torque and torque sequence when installing. V-Class ASICs include the EMAC, EPAC and ERAC chips. The ETAC ASIC is only installed in multi-node systems. The XBAR is a non-blocking fully muxed data switch located on the ENRB. It is comprised of four ERAC (routing attachment chip) ASICs. Connected directly to the XBAR are up to eight EMACs (memory controllers) and eight EPACs (processor agent chips). With 16 discrete ports per ERAC, the XBAR has a total of 64 discrete ports with NO bus waiting!
ASIC
Application Specific Integrated Circuit Crossbar
12 on ENRB 1-2 on each EMB
XBAR
The Midplane
129
Acronyms V2500/V2600 Acronym List
V2500/V2600 Acronym List

Is Located On
Acronym
Fully Qualified
Supporting Text
SCUB
Stingray Core Utility Board Midplane Interface Board
The Midplane The Midplane
The ECUB is the heart of the power on self test, environmental monitoring, scanning, system clocks, as well as many other support issues. The MIB is the largest circuit board in the VClass system and is also known as the Midplane. This is where the HyperPlane Crossbar backplane is located as well as all of the Processor Agent Controllers (EPACs). The EMB has several plug-on modules to complete the functionality of the memory subsystem. Each pair of processors and one half (3 PCI slots) of an average I/O card cage connect to a Processor Agent Chip (or PAC). Each PAC chip communicates directly with the HyperPlane with NO bus waiting. There is a custom VLSI chip for the Memory Access Controller and there may be an optional VLSI chip for the V-Class technical SPU called Toroidal Access Controller (TAC).
MIB
EWMB
Exemplar WBased Memory Board Stingray Processor Agent Controller Stingray Memory Access Controller
The Memory Board Midplane Interface Board
SPAC
SMAC
The Memory Board
130
Acronym
Fully Qualified
Is Located On
Supporting Text
SAGA
Just another cool acronym that has no meaning.
The SIOB
Each Stingray I/O Card Cage contains seven total PCI slots for connecting PCI based I/O device adapter cards. There are two SAGA chips located inside each I/O card cage. One SAGA chip controls 4 slots (A side of the SIOB, e.g. iolf_a) and the other controls 3 slots (B side of the SIOB, e.g. iolf_b). Each SAGA chip connects to a Processor Agent Controller (PAC) chip which is located on the HyperPlane Crossbar Backplane. The Node Power Supply (NPS) provides +48V DC to the two +48V Boards (E48VB) which simply route the +48V throughout the node to the fans and the DC - DC converters. The two E48VBs attach together to form a single "bus bar" type connection. The E48VBs are located in the rear chassis, behind the ENRB. The EWPB contains a circuit which detects when the +3.3V DC drops below specification and will signal a power failure to the Core Utilities Board (SCUB). The EWPB contains an overtemp sensor and reports overtemp conditions to the SCUB. The DC - DC converters are part of the EWPB. Failure of one of the converters will require replacement of the EWPB. The EWPB operates at 440 MHZ and is the processor board used in a V2500 system
E48VB
Exemplar +48 Volt Board
EWPB
Exemplar WBased Processor Board
The Processor Board
131
Acronym
Fully Qualified
Is Located On
Supporting Text
ELPB
Exemplar Lanshark Processor Board
The Processor Board
The ELPB contains a circuit which detects when the +3.3V DC drops below specification and will signal a power failure to the Core Utilities Board (SCUB). The ELPB contains an overtemp sensor and reports overtemp conditions to the SCUB. The DC - DC converters are part of the ELPB. Failure of one of the converters will require replacement of the ELPB. The ELPB operates at 552 MHZ and is the processor board used a V2600 system. The EWMBPB sits a top the Memory Board and connects to the EWMB through the Memory Power Board Extender (EMBPX). The EMBPX is a field replaceable unit. The EMUC is a non field replaceable FPGA located on the SCUB. It functions as the environmental monitor and power controller. Both the EMUC and power-on circuit work together to control power. The EPUC is a non field replaceable FPGA located on the SCUB. Core logic functions are controlled by the EPUC via the 8 EPAC ASICs. System initialization and CPU interrupt handing are the EPUCs main functions. The ERAC is a field replaceable ASIC which is located on the Midplane. Four ERACs make up the HyperPlane XBAR. Each ERAC has 16 discrete ports with NO bus waiting! Each SIOB contains seven PCI card slots. Up to four SIOBs can be installed in a V-Class system for a total of 28 PCI card slots.
EWMBPB
Exemplar WBased Memory Board Power Board Environmental Monitor Utility Controller
The Memory Board
EMUC
The Utility Board (SCUB)
EPUC
Exemplar Processor Utility Controller Exemplar Routing Access Controller Stingray I/O Board
The Utility Board (SCUB)
ERAC
Midplane Interface Board The I/O Card Cage
SIOB
132
Acronym
Fully Qualified
Is Located On
Supporting Text
STAC
Stingray Torrodial Access Controller Exemplar Plug-in Memory Board Exemplar Plug-in Memory Board (Multi-node) Exemplar Memory Board Power Extender Exemplar Cable Connector Board
The Memory Board
Globally shared memory is accessed between nodes via the STAC. Off node memory accesses are routed from the STAC to either main memory or the XBAR through the SMAC. Up to 16 DIMMs can be installed on each memory board (EWMB). 80 bit DIMMS can only be used in single node systems. 88 bit DIMMs are required in all multi-node systems. The extra 8 tag bits are needed for cache coherency between hypernodes. These DIMMs will function along side the 80 bit DIMMs in a single node system. The EMBPX is a small extension board that buses the 48VDC input and logic level voltages between the EWMB and the EWMBPB. The ECCB connects the keyswitch to all four power supplies and the RS232 ports from the bulk head connector to the SCUB. Environmental monitoring signals from the power supplies and fans also route through the ECCB to the SCUB.
EPMB
The 80 Bit DIMMs The 88 Bit DIMMs
EPMBM
EMBPX
The Memory Board
ECCB
133
Acronym
Fully Qualified
Is Located On
Supporting Text
ELEDB
Exemplar LED Board
the left front side of Chassis
The ELEDB is the attention light for the system. The normal state is solid on across all segments. When flashing either an environment error or system crash has occurred. Test station utilities must be used to determine the error state when flashing. All ASICs are field replaceable and require a specified torque and torque sequence when installing. V-Class ASICs include the SMAC, SPAC and ERAC chips. The STAC ASIC is only installed in multi-node systems. The XBAR is a non-blocking fully muxed data switch located on the MIB. It is comprised of four ERAC (routing attachment chip) ASICs. Connected directly to the XBAR are up to eight SMACs (memory controllers) and eight SPACs (processor agent chips). With 16 discrete ports per ERAC, the XBAR has a total of 64 discrete ports with NO bus waiting!
ASIC
Application Specific Integrated Circuit Crossbar
12 on ENRB 1-2 on each EMB
XBAR
Midplane Interface Board
134

V2500/V2600 Familiarization Guide: HP 9000 Servers

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

V2500/V2600 Familiarization Guide: HP 9000 Servers

Uploaded by

Copyright:

Available Formats

V2500/V2600 Familiarization Guide

A5074A A5074-90001 (available via HP-Web) November 1999

68 69 69 70 71 73 75 75 76 77 77 77 77 78 79 81 81 83 84 85 88 90 90 90 92 97 97 98 98 99 99 99 100 100 101 102

V2500/V2600 H/W Enhancements

V2500 H/W Enhancements Processor board (EWPB/ELPB)

Processor board (EWPB/ELPB)

from X-Bar PnL Agent to X-Bar

V2500 H/W Enhancements Memory board (EWMB)

Memory board (EWMB)

V-Class Memory Interleaves

V2500 H/W Enhancements Memory board (EWMB)

V2500 H/W Enhancements I/O Subsystem (SIOB)

I/O Subsystem (SIOB)

V2500 H/W Enhancements Mid-Plane (MIB, SCUB, MIBPB)

Mid-Plane (MIB, SCUB, MIBPB)

Utilities Board (SCUB)

Mid-plane Power Board (MIBPB)

V2500 H/W Enhancements Power Subsystem

Higher capacity +48VDC power supply (NPS)

V2500 H/W Enhancements Cooling Subsystem

N+1 fan system

V2500 H/W Enhancements Mass Storage

V2500 H/W Enhancements Service Support Processor

Service Support Processor

System management Software

Service Support Processor

V2500 H/W Enhancements Service Support Processor

V2500 H/W Enhancements Service Support Processor

V2500/V2600 H/W Configurations

V2500 H/W Configurations Processor Board Configurations

Processor Board Configurations

PCXW and PCXW+ Configurations

Power Brick for CPU A

Power Brick for CPU B

V2500 H/W Configurations Processor Board Configurations

Processor Board Installation

Single CPU in Processor Board Configuration

V2500 H/W Configurations Processor Board Configurations

Order 9th 10th 11th 12th 13th 14th 15th 16th

Menu Mode Proc. ID 1 9 3 11 4 12 6 14 A A A A A A A A

V2500 H/W Configurations Processor Board Configurations

Menu Mode Proc. ID 21 29 23 31 17 25 19 27 20 28 22 30 B B B B B B B B B B B B

2 CPUs per Processor Board Configuration

V2500 H/W Configurations Memory Configurations

V2500 H/W Configurations V2500/V2600 Memory Configurations

V2500/V2600 Memory Configurations

Quadrant Buses Rows

V2500/V2600 DIMM Quadrant Designations

Q2B3: Quadrant 2, Bus 3

V2500/V2600 DIMM Configuration Rules

Using dcm to Check V2500/V2600 Memory Configuration

V2500 H/W Configurations V2500/V2600 Memory Configurations

Q0B0 Q0B1 Q0B2 Q0B3

L/L L/L L/L L/L

Q1B4 Q1B5 Q1B6 Q1B7

L/L L/L L/L L/L

Q2B0 Q2B1 Q2B2 Q2B3

Q3B4 Q3B5 Q3B6 Q3B7

Q0B0 Q0B1 Q0B2 Q0B3

L/L L/L L/L L/L

Q1B4 Q1B5 Q1B6 Q1B7

L/L L/L L/L L/L Row 0 / Row 1

Q2B0 Q2B1 Q2B2 Q2B3