Professional Documents
Culture Documents
DREADNOUGHT
STORAGE SYSTEM
Troubleshooter’s Guide
P/N XXX-XXX-XXX
DRAFT FOR PILOT CLASS 1/15/08
REV. A
No part of this publication may be reproduced or distributed in any form or by any means, or stored in a database
or retrieval system, without the prior written consent of EMC Corporation.
EMC believes the information in this publication is accurate as of its publication date. The information is subject to
change without notice.
THE INFORMATION IN THIS PUBLICATION IS PROVIDED "AS IS." EMC CORPORATION MAKES NO
REPRESENTATIONS OR WARRANTIES OF ANY KIND WITH RESPECT TO THE INFORMATION IN THIS
PUBLICATION, AND SPECIFICALLY DISCLAIMS IMPLIED WARRANTIES OF MERCHANTABILITY OR
FITNESS FOR A PARTICULAR PURPOSE.
Use, copying, and distribution of any EMC software described in this publication requires an applicable software
license
ICDA® (Integrated Cached Disk Array), EMC2® (the EMC logo), THE STORAGE ARCHITECTS®,
MOSAIC:2000®, Harmonix®, CLARiiON® and Symmetrix® are registered trademarks and EMC™ and SRDF™
and FLARE™ are trademarks of EMC Corporation.
__________________________________________________________________________________________
All other trademarks used herein are the property of their respective owners.
Trademark Information
EMC CONFIDENTIAL
Preface
Description This document looks at EMC’s application of the hardware used in the
Dreadnought systems. It is designed for personnel who have to repair
EMC’s CLARiiON hardware to component level.
It is assumed that readers have a knowledge of, or access to, the following
topics:
Interfaces & Protocols - Fibre Channel, Ethernet, I2C, RS232, PCI, PCI-X,
PCI-Express, iSCSI and SAS.
• Sylvia Dunne
• Eugene McVeigh
• Fred Lilienkamp
• Dave Caissie
• Roger Guilbault
• Dave Thomas
Also special thanks for the use of all the Engineering Documentation and
assistance from various department throughout EMC.
EMC CONFIDENTIAL
iii
EMC CONFIDENTIAL
iv Preface
Warnings and Cautions
This unit has more than one power supply cord. To reduce the risk of
electric shock, disconnect two (2) power supply cords before servicing.
Ground circuit continuity is vital for safe operation of the machine. Never
operate the machine with grounding conductors disconnected.
Remember to reconnect any grounding conductors removed for or
during any installation procedure.
Das Geraet hat mehr als eine Anschlussleitung. Zur Vermeidung der
Gefahr eines elektrischen Schlages sind vor dem öffnen beide
Anschlussleitungen vom Netz zu trennen.
Additional The system operates at high voltages. To protect against physical harm,
Warnings and power off the system whenever possible while servicing.
Cautions
In case of fire or other emergency, isolate the system's power involved
and alert appropriate personnel.
Exercise great care at all times when working on the machine. Remember
to:
• Use caution near any moving part and any part that may start unex-
pectedly such as fans, motors, solenoids, etc.
EMC CONFIDENTIAL
EMC CONFIDENTIAL
EMC CONFIDENTIAL
Contents vii
viii Contents
Figures
Figures ix
2-50 Interface Controller Hub (ICH) Clocks ................................................................................... 104
2-51 PLX Clocks................................................................................................................................... 105
2-52 LAN1 & LAN2 Clocks ............................................................................................................... 105
2-53 Server IO & Firmware Hub Clocks .......................................................................................... 106
2-54 Management Controller Clock ................................................................................................. 106
2-55 33Mhz to Uarts and IO Modules .............................................................................................. 107
2-56 Midplane and Debug connector Clocks .................................................................................. 107
2-57 Wildcat-S Interrupts Block ........................................................................................................ 108
2-58 Interrupts received by SIO & ICH7.......................................................................................... 109
2-59 SAN Cabinet with one Wildcat-S Chassis............................................................................... 112
2-60 Wildcat-S SAN Chassis - LAN Block Diagram ...................................................................... 113
2-61 BMC5751 LAN Chip Block Diagram ....................................................................................... 114
2-62 Wildcat-S Chassis - 2 Solar Flare FRU’s (rear view) .............................................................. 115
2-63 Solar Flare Management FRU Face Plate ................................................................................ 115
2-64 Solar Flare Interface Block......................................................................................................... 116
2-65 Dreadnought NAS System LAN - Example Only ................................................................. 117
2-66 Dreadnought NAS - LAN Block Diagram .............................................................................. 119
2-67 Earthquake Management FRU Face Plate............................................................................... 120
2-68 Earthquake Interface Block ....................................................................................................... 121
2-69 Wildcat-S RS232 UART Ports ................................................................................................... 124
2-70 DUART for IO4 & IO5 Diplexing ............................................................................................. 125
2-71 Quad UART Pinout .................................................................................................................... 126
2-72 Wildcat-S Chassis Midplane I2C Interfaces (SAN)................................................................ 130
2-73 SAN I2C Arbitration Signals..................................................................................................... 131
2-74 Midplane SAN I2C Access Arbitration Diagram................................................................... 132
2-75 Wildcat-S Chassis Midplane I2C Interfaces (NAS)................................................................ 133
2-76 NAS I2C Arbitration Signals..................................................................................................... 134
2-77 ICH7/MCH I2C Master Address Map..................................................................................... 136
2-78 Wildcat-S I2C Interfaces ............................................................................................................ 137
2-79 SAN MCU I2C Slave Port Access............................................................................................. 138
2-80 NAS MCU I2C Slave Port Access............................................................................................. 139
2-81 MCU I2C Master Port Address Map ....................................................................................... 140
2-82 MCU I2C Master Interface ........................................................................................................ 141
2-83 Blackford MCH FBDIMM I2C Address Map ......................................................................... 142
2-84 Blackford MCH Hot-Plug I2C Address Map ......................................................................... 142
2-85 Blackford MCH I2C Diagram ................................................................................................... 142
2-86 Wildcat-S I2C Reset, Arbitration & Attention Signals .......................................................... 143
3-87 Solar Flare .................................................................................................................................... 147
3-88 Solar Flare Block Diagram......................................................................................................... 148
3-89 Broadcom BCM5397 Switch...................................................................................................... 149
3-90 BCM5397 block diagram............................................................................................................ 150
3-91 Solar Flare Ethernet Crosslink .................................................................................................. 151
3-92 LPC2131 Microcontroller Block Diagram ............................................................................... 152
3-93 Philips LPC2131 Microcontroller ............................................................................................. 153
3-94 Serial Crosslink Block Diagram................................................................................................ 154
3-95 Solar Flare I2C Topology ........................................................................................................... 155
3-96 Slot Dependent I2C Addresses ................................................................................................. 155
3-97 Solar Flare Voltage Sequence.................................................................................................... 156
3-98 Command Power Sequencer..................................................................................................... 157
3-99 Broadcom BCM5397 pin-out..................................................................................................... 158
3-100 Broadcom BCM5397 pin-out (cont.) ........................................................................................ 159
3-101 Broadcom BCM5397 pin-out (cont.) ........................................................................................ 160
3-102 Philips LPC2131 pin-out ............................................................................................................ 161
3-103 Philips LPC2131 pin definitions ............................................................................................... 162
3-104 Philips LPC2131 pin definitions (cont.)................................................................................... 163
EMC CONFIDENTIAL
x Figures
Figures
EMC CONFIDENTIAL
Figures xi
EMC CONFIDENTIAL
xii Figures
Tables
EMC CONFIDENTIAL
Figures xiii
EMC CONFIDENTIAL
xiv Figures
1
Chapter Contents:
• System Intro............................................................................................... 16
EMC CONFIDENTIAL
15
1.1 System Intro
Dreadnought is a high-end converged Midrange SAN/NAS offering. The
Dreadnought system architecture in many ways will look similar to the
hammerhead since it is the next generation to the Hammerhead system.
The cabinet shown below will house the Wildcat-S Chassis. There will be
configurations which combine Wildcat-S Chassis, D15= (DAE) Disk Array
Enclosure and control stations (servers). The Dreadnought cabinet will be
populated according to the configuration guide.
8 DAE
(120 Drives)
Wildcat-S
Chassis
SPS
NAS & SAN Network Attached Storage (NAS) and Storage Area Network (SAN) are
two different storage architectures for accessing stored information.
NAS With NAS, the storage device is connected directly to the network,
normally through a network interface such as Gigabit Ethernet and iSCSI.
Our NAS architecture requires a processing engine to operate which is
called a ‘Data Mover’,
SAN The SAN implementations are used to connect storage to the network via
servers, but require very high speed access to these servers, so they are
normally connected via Fibre Optic cables and use the Fiber Channel and
FICON protocols. The SAN architecture requires a processing engine
which is called a ‘Storage Processor’.
Both processing engines use the Wildcat-S as its base board and the
differences occur when different IO Modules (SLIC’s) and Management
FRU’s are inserted into the motherboard. There are different portions of
the software used to operate the different configurations.
For example: (Data Mover & Storage Processor).
EMC CONFIDENTIAL
System Intro 17
1.2 Wildcat-S Chassis Overview
A Wildcat-S Chassis is made up of a number of different parts:
Power Power
Supply B Wildcat-S Blade B Supply A
Power & Led’s
Wildcat-S
Blade A
The Wildcat-S Chassis above is the SAN version which can be recognized
by the use of two Solar Flare Management FRU’s
EMC CONFIDENTIAL
The Nova test board is located at the bottom center of the Wildcat-S
Chassis in (Figure 1-3). This test board is used for Jtag & boundary scan
purposes in manufacturing only and will not be shipped to the customer.
A blank face plane will be installed for customer shipment.
EMC CONFIDENTIAL
Power Supply - A
Management FRU - A
Blower Fan
ITRAC
IO Module 3
ITRAC
IO Module 2
Wildcat-S
Storage Processor
A
ITRAC
IO Module 1
Blower Fan
ITRAC
IO Module 0
M
I
D
P
IO ANNEX
L
Blower Fan A
N
E
ITRAC
IO Module 3
ITRAC
IO Module 2
Wildcat-S
Blower Fan Storage Processor
B
ITRAC
IO Module 1
ITRAC
IO Module 0
Management FRU - B
Power Supply - B
Wildcat-S Blade The Wildcat-S Blade (also called a Data Mover in a NAS system) or
(SP in an SAN system) is the core component in the Wildcat-S Chassis. It
consists of 5 separate module:
Numbering Figure 1-6 below shows the 4 IO Modules and their module and port
numbering scheme. This diagram assumes 4 port IO Modules but there
Scheme are some 2 port modules.
Port 3
Port 2
ITRAC
IO Module 3
Port 1
ICH
Port 0
CPU 1 Port 3
Port 2
ITRAC
MCH IO Module 2
Port 1
Wildcat-S Port 0
GBX
Port 2
ITRAC
FB DIMM IO Module 1
Port 1
FB DIMM
CPU 0 FB DIMM Port 0
FB DIMM
Port 3
FB DIMM
FB DIMM Port 2
ITRAC
FB DIMM IO Module 0
Port 1
FB DIMM
Port 0
EMC CONFIDENTIAL
Wildcat-S The CPU Module (Wildcat-S) contains two Intel processors, eight slots of
Motherboard FBDIMM RAM (Fully Buffered Direct In-line Memory Module) and the
Intel Blackford Memory Controller Hub (MCH). Other components
include the Intel Hance Rapids I/O Controller Hub (ICH7), a (FWH)
Firmware Hub chip, dual Ethernet LAN chips, NVRAM, and Server I/O
chip.
I/O Modules The I/O Modules which will be used in the Dreadnought system are
available in a few different configurations, to allow for different interfaces
on the front-end and back-end of the system.
The IO Modules will not be covered in this manual and may be found in
the SLIC Small IO Card TSG.
EMC CONFIDENTIAL
Management At the sides of the chassis are two management boards. They can be used
to communicate with other Wildcat-S Chassis (NAS) and share
Boards management functions. These boards are available in two different
configurations:
These boards mount in a metal carrier, which slides into the chassis
underneath the Power Supply. The airdam has a Power and Fault LED,
Ethernet and RS-232 connectors.
Nova Test The Nova Test board is for factory use only. It is plugged into the front of
the chassis, where it performs three main functions:
Board
• it provides an interface to the voltage margining circuitry.
EMC CONFIDENTIAL
EMC CONFIDENTIAL
Acronyms 25
• SCSI - Small Computer System Interface
• SLIC - SmalL IO Card form factor
• SGPIO - Serial GPIO
• SMBus - System Management Bus - a protocol on top of an I2C bus for
chip-to-chip management.
• SP - Storage Processor
• SPA - Storage Processor is located in the lower slot of the DPE chassis (rear
view).
• SPB - Storage Processor is located in the upper slot of the DPE chassis (rear
view)
• SPS - Standby Power Supply — a CRU that provides battery backup when
AC power fails.
• STPSATA Tunneling Protocol — part of the SAS specification that allows
SATA traffic to pass from controller, through expanders, to drives. The
controller wraps SATA in STP headers that the final expander unwraps and
sends to the drive.
• TLA - Top Level Assembly
• TWI - Two Wire Interface - a low level serial chip-to-chip communications
bus developed by PMC, similar to I2C
• VPD - Vital Product Data — information programmed by the factory into a
“resume” EEPROM on some FRU’s, generally containing some unique
information on each part such as a WWN seed and serial number. The term
“VPD” is often used to refer to the EEPROM itself.
EMC CONFIDENTIAL
Chapter contents:
EMC CONFIDENTIAL
27
2.1 Wildcat-S Blade Introduction
• 1 Motherboard
• 4 IO Modules
• 1 Metal Tray
CP U- 0
Wildcat-S CP U-1
Motherboard
MC H
FB DI M
MS
ICH
IO Module #3
Connector
0
1
4 IO Modules 2
3
Figure 2-8. Wildcat-S Blade
Wildcat-S The Wildcat-S Blade is the base board for the Wildcat-S Chassis which is
used in the Dreadnought System. Each Wildcat-S Chassis contain 2
Motherboard motherboards which will be populated with two Clovertown Quad Core
processors. The processors have large heat sinks shown above which are
secured to the board with 4 screws. The Clovertown CPU uses a 771 pin,
Flip Chip Land Grid Array Chip (FC-LGA4) package. Each CPU chip is
inserted into an LGA771 socket which allows each processor to be
removed.
Table 2-2 illustrates the Wildcat-S Motherboard CPU populations for the
Dreadnought and Tigon system.:
System Type Base Board CPU Type # CPU Speed Front Side Bus
Dreadnought Wildcat-S Clovertown - Quad Core 2 2.33 Ghz 1333M/TSec
Tigon Wildcat-S Clovertown - Quad Core 2 2.33 Ghz 1333M/TSec
EMC CONFIDENTIAL
28 DREADNOUGHT SP BOARD
Wildcat-S Figure 2-8 on page 28, is a photo of the Wildcat-S Blade and below is an
block diagram of its major components and interfaces.
Interfaces Block
U8 U7
Wildcat-S
205-800-602c 2
U16 LAN1 LAN2
x1 PCIe
x1 PCIe
LPC
x8 PCIe CMI
x4 ESI
x8 PCIe IO
U49
INTEL CPU 1 x4 PCIe
Woodcrest= FSB
Clovertown=
U65 x4 PCIe J52_1S
FSB MCH x4 PCIe IO Card 2
J22
x4 PCIe
4Gb/sec each
U96 CH0 CH2
INTEL CPU 0 CH1 CH3
Woodcrest = J11 – FB DIMM
Clovertown= J12 – FB DIMM J51_1S
J9 – FB DIMM IO Card 1
J10 – FB DIMM
J7 – FB DIMM
U120
J8 – FB DIMM
J5 – FB DIMM
J6 – FB DIMM
LPC
COM 1 J50_1S
U141 U140 IO Card 0
COM 2 SIO FWH U149
NVRAM
Each Intel CPU Chip has its own front side bus (FSB) that connects to the
Blackford Memory Controller Hub (MCH).
The Blackford MCH functions as the “bridge” for the processors to the
rest of the system. It provides four fully buffered DIMM interfaces to the
system RAM (FBDIMM). The MCH is also connected to the ICH7 IO
Controller Hub and together these two chips are called the Intel Blackford
chipset.
• One 8 Lane PCIe Bus A to the PLX which is a PCIe to PCIe Switch. Each
PCIe port operates at 2.5Ghz.
-The PLX sends x8 PCIe to the Midplane connector for the CMI interface.
-The PLX sends x8 PCIe to the Midplane connector to IO ANNEX interface.
• Four x4 PCIe interfaces to each of the 4 IO Module Connectors. These
connectors may have various Modules inserted into them.
EMC CONFIDENTIAL
• Two x1 PCIe interfaces connect to the two Ethernet 5751 controllers. The
LAN 0 & LAN 1 Ethernet controllers are connected to the midplane
connector. From there each controller is routed to 1 Management FRU.
• The Low Pin Count (LPC) interface connects the ICH to the Server IO chip
which contains the two RS232 communication ports COM1 & COM2 out to
the midplane connector.
• The Firmware Hub which contains the BIOS & POST code is also
attached to the LPC interface.
• An USB2.0 flash controller interface connects to 256Mbyte of flash used for
the diagnostic code.
• Some of the GPIO pins are connected to a quad UART which are used for
the diplex circuits which allow communication to the DAE’s.
EMC CONFIDENTIAL
30 DREADNOUGHT SP BOARD
Wildcat-S The Wildcat-S Motherboard contains:
Components • Two Quad Core Clovertown 2.33Ghz CPU’s
2
• One Intel Blackford Memory Controller Hub (MCH)
• One Intel 82801 Gigabit IO Controller Hub 7 (ICH7)
• Eight slots of Fully Buffered Direct In-line Memory Modules (FBDIMM) up
to 16Gbytes of System Memory
• One (PLX) PCI Express bridge chip (PEX8524)
• Two 10/100/1000 Ethernet Chips which connect through the midplane
• 2MB Firmware Hub for BIOS and POST
• 256Mbytes of Nand Flash Memory connected through USB to the ICH
• 256K NVRAM
• One 87417 Server IO chip (SIO).
• Control Monitor Device (CMD) will provide on-board DCDC control and
monitoring
Wildcat-S Parts The below diagram shows most of the major parts reference designators
on the Wildcat-S boards. Only front-side parts are shown.
Reference
INTEL
U51 U50
Woodcrest =
Clovertown =
U65 J52_1S
MCH Y8
J22S Y7
U96
INTEL
Woodcrest= J11 – FB DIMM
Clovertown= J12 – FB DIMM J51_1S
J9 – FB DIMM
J10 – FB DIMM
J17
J7 – FB DIMM U2691
U121
U120
J8 – FB DIMM
U133 U131 U128 U125
J5 – FB DIMM
U127
J6 – FB DIMM
U143 U137
EMC CONFIDENTIAL
Quad Core The Clovertown Quad Core processor has four independent 2.33GHz
CPU cores with each pair of cores (0,1) & (2,3) sharing a 4MB L2 Cache
each. The four cores in each processor share an Intel proprietary FSB
interface to the Blackford MCH.
The CPU uses the FSB interface which consists of 144 signals and
supports up to 1333 million 64-bit transfers per second with a theoretical
bandwidth of 10.66GBytes/sec.
Blackford MCH The Blackford Memory Controller Hub (MCH) is also connected to the
processors FSB’s. It provides switching facilities between a number of
different components and has five distinct interfaces:
• one Intel proprietary x4 ESI (Enterprise South bridge Interface) to the ICH7
As its name suggests, the MCH is a hub, a device that connects to several
other devices and routes or directs information between them.
EMC CONFIDENTIAL
32 DREADNOUGHT SP BOARD
ICH7 IO The other chip in the chipset is the I/O Controller Hub called the ICH7.
This component provides the interface to I/O devices that a computer
Controller Hub would typically need, like keyboard, mouse, RS232 and boot ROM. It also
2
has a PCI interface which is used to add other controllers, such as USB or
Ethernet. All of these interfaces or controllers can be grouped together
effectively because they work in the kilohertz range or slower, and use
very little bandwidth compared to the processors and high-speed devices
on the MCH. The ICH7 on the Wildcat-S is connected to the Blackford
MCH via a 4 lane (ESI) interface.
Server IO The Server IO chip attaches to the Low Pin Count (LPC) bus from the
ICH7. The SIO incorporates the real time clock functions and provides
Controller connectivity to NVRAM and a DUART.
DDR2 RAM The Blackford MCH has 4 * FBDIMM channels and each channel will
accommodate up to two industry standard FBDIMM’s per channel which
FBDIMM’s are connected to the board via a 240 pin FBDIMM connector. Supported
DIMMs can be 1, 2 or 4 Gbytes and can be either single or dual rank using
either x4, or x8 DDR2 SRAM devices.
Midplane The midplane connector is an Gbx type connector. The connector supplies
power from 16 blades on the midplane to the motherboard. It also
Connector transports I2C communication and other signals used by the
motherboard.
EMC CONFIDENTIAL
34 DREADNOUGHT SP BOARD
2.2 Intel Clovertown CPU
Intel LPC
FWH
ICH7 33 MHz
clovertown_fsb_block.emf
• 64-bit capable
EMC CONFIDENTIAL
• Thermal monitoring
Operation The Front Side Bus (FSB) is a 1333 MHz quad-pumped bus running off a
333 MHz system clock, which results in 10.6 GBytes/s data transfer. The
processor transfers data four times per bus clock. Along with the 4X data
bus, the address bus can deliver addresses two times per bus clock. In
addition, the Request Phase completes in one clock cycle. The FSB is also
used to deliver interrupts to the processor.
Signals on the FSB use Assisted Gunning Transceiver Logic (AGTL+) level
voltages.The AGTL+ inputs require reference voltages (GTLREF_DATA_MID,
GTLREF_DATA_END, GTLREF_ADD_MID and GTLREF_ADD_END)
which are used by the receivers to determine if a signal is a logical 0 or a
logical 1.Termination resistors (RTT) for AGTL+ signals are provided on the
processor silicon and are always enabled on the processor to control
reflections on the transmission line.
The Clovertown is the latest of the x86 family of Intel CPU’s. While
offering a complex and high performance interface to the supporting
chipset and peripheral devices, from a software or Emulator point of
view it has many of the same methods of access to those supporting
devices, namely Memory Read/Write and I/O Read/Write cycles.
FSB_A_N<35:17> A B
FSB_A_N<16:3>
FSB_HREQ_N<4:0> A B
FSB_ADSTB_N0
FSB_ADSTB_N1
FSB_TRDY_N
FSB_DBSY_N
FSB_DRDY_N
EMC CONFIDENTIAL
36
Memory R/W In the normal course of events, the CPU needs to access either some
memory device (DDR SDRAM, Boot Flash, NVRAM) or some peripheral
device (Ethernet controller, FC Controller, COM port, PCI-E Bridge etc.)
2
attached to the MCH or ICH. The primary way of the CPU accessing a
device is by a Memory Read or Memory Write cycle, which is usually
initiated by a MOV instruction.
I/O R/W Some devices or functions are not mapped to a memory address but to
another space that the CPU can access, called I/O space. I/O space is
limited to 64 KBytes, since an I/O access only uses the low 16 address bits.
An I/O address is sometimes referred to as an I/O Port. I/O accesses are
initiated by IN or OUT instructions
CMI (Peer Memory space - FBDIMM) PLX 8000 000 - BFFF FFFF - -
EMC CONFIDENTIAL
CPU Boot A number of events must take place for the CPU to begin executing code
Sequence to bring the system up. To assist troubleshooting a dead Wildcat-S, the
basic sequence is provided here.
Power:
• The 1.05V and 3.3V regulators are enabled by the MC for the CPU’s
(3.3) and MCH (1.05 and 3.3). This also begins powering the CK410
Clock generator.
• The 1.5V regulator is enabled by the MC for the CPU’s and MCH.
• The processors drive the VID<6:0> lines to the VRM’s to specify the
exact voltage they require.
CPU Reset:
• The CPU performs its internal self check and initializes it’s registers.
• The MCH forwards the request to the ICH (over the ESI link)
• The ICH generates a Read cycle on the LPC bus to the FWH and
fetches the Boot vector to pass back up to the FSB.
• The Boot vector is used by the CPU to begin executing the BIOS code,
typically address FFF0 0000h.
EMC CONFIDENTIAL
38
The signals used by the processor are shown in Figure 2-13s
VTT _CPU
[22..1]
VCC_Core
[181..1]
VCC_FSB# VSSA FSB#_VCCPLL
2
FSB#_A_N[35:3]
FSB#_AP_N[1:0] PUP_FSB#_BSEL[2:0]
FSB#_ASDTB_N[1:0] CPU#_BCLK+/-
FSB#_ADS_N FSB_INIT_N Execution
FSB#_BR[1:0]_N FSB_STPCLK_N Control
FSB#_REQ_N[4:0] ICH_CPUSLP_N Signals
Address Signals
FSB#_RS_N[2:0]
FSB_INTR
FSB#_RSP_N FSB_NMI
FSB#_DEFER_N
FSB#_IERR _N
FSB#_HIT_N
FSB#_HITM_N
Front Side Busses
FSB#_LOCK_N
FSB#_BINIT_N FSB_FERR_N
FSB#_BNR_N FSB_IGNNE_N PC
FSB#_BPRI_N FSB_A20M_N Compatibility
Signals
FSB#_MCERR_N FSB_SMI_N
FSB#_RESET_N
FSB#_HD_N[63:0] Intel
FSB#_DP_N[3:0]
Clovertown
Data Signals
FSB#_DBI_N[3:0] FSB#_SKTOCC_N
FSB#_DSTBP_N[3:0] FSB#_VID_SEL
FSB#_DSTBN_N[3:0]
# = 0(U96) , 1(U49)
FSB#_VID [5..0]
FSB#_TRDY_N POWER
CPU_PWRGD
FSB#_DRDY_N CONTROL
FSB#_VCC_SENS
SIGNALS
FSB#_DBSY_N FSB#_VSS_SENS
FSB#_GTLREF_ADD_R
FSB#_GTLREF_DATA_R
CPU#_THERMDA XDP_TMS_MAIN
Thermal CPU#_THERMDC XDP_TRST_N
FSB#_THERMTRIP_R XDP_DBRST_N
clovertown_pinout.vsd
Vss (Gnd)
EMC CONFIDENTIAL
40
2.3 Blackford MCH
Overview The Blackford Memory Controller Hub (MCH) is a 1432-pin BGA which
functions as the "bridge" for the processor to the rest of the system. The
2
MCH provides interfaces to:
to CPU’s
64 64
CPU I/F
FSB 0 Bus: 0 Dev: 16
FSB 1
Fnc: 0,1,2
FB DIMM Ch0
14 MCH I/O MCH Error
Registers Registers
Bus:0 Dev:22 Fnc:0
4 8
PCI Bus 0
PCIe Port 2
FB DIMM Ch1 4 8
14 Bus: 0 Dev: 2 Fnc:0
Memory Controller
4 to PLX
10 CH 1 PCIe Port 3
4
Bus: 0 Dev: 3 Fnc:0
4
FB DIMM Ch2 PCIe Port 4
14 4 to SLIC 0
Bus: 0 Dev: 4 Fnc:0
Bus:0 Dev:21 Fnc:0
FBD Branch 1
10 CH 0 4
PCIe Port 5 to SLIC 1
4
Bus: 0 Dev: 5 Fnc:0
FB DIMM Ch3
14 4
CH 1 PCIe Port 6 to SLIC 2
10 4
Bus: 0 Dev: 6 Fnc:0
4
PCIe Port 7 to SLIC 3
I2C to 4
DIMM’s Bus: 0 Dev: 7 Fnc:0
DMA Controller
SMBus Bus: 0 Dev: 8 Fnc:0
Controller
Blackford
ESI MCH
Bus: 0 Dev: 0 Fnc:0
4 4
blackford_block.emf to ICH7
EMC CONFIDENTIAL
Blackford MCH 41
Blackford also contains DMA engines and numerous registers for I/O
access and error reporting. The MCH internal modules and interfaces
appear to the processors as several devices attached to PCI bus 0,
although physically not all are actual PCI devices and the physical
internal bus is not an actual PCI bus. Also, because the ICH7 is connected
to the ESI link, all the ICH internal modules are accessed as if they are on
PCI bus 0.
Front-Side Bus The Blackford MCH supports two Woodcrest or Clovertown processors,
with an 1333 MHz front side bus (FSB). Each FSB interface consists of 144
signals. The bus clock is 333 MHz, the address and request interface is
double pumped to 667 MHz. The 64 bit data bus is quad pumped to 1333
MHz. This provides a FSB bandwidth of 10.7 GB/s.
The CPU’s have the capability to arbitrate for the FSB between themselves
by asserting the BREQ0# signal to each other. The MCH has priority
access and gains ownership of the FSB by asserting BPRI# to the CPU’s.
System RAM System RAM consists of eight Fully Buffered DIMMs (FBDIMM’s) in
eight sockets. Total supported memory is up to 32 GBytes of DDR2-667
SDRAM
The four FBD channels are organized into two branches with two
channels on each branch. Each branch is supported by a separate memory
controller. The two channels in each branch operate in lock step to
increase FBD bandwidth. A branch transfers 16 bytes of payload per
frame on Southbound lanes (Blackford to FBD) and 32 bytes of payload
per frame on Northbound lanes (FBD to Blackford).
EMC CONFIDENTIAL
42
• Four Fully Buffered DDR memory channels
•
Supports up to eight dual-ranked FB DIMMs for a total of 32GBytes
The FBD link speed is 6x the speed of the DDR data transfer speed. A
4Gb/s FBD link speed supports DDR2-667 (FSB @ 1333MT/s)
2
• All memory devices are DDR2
• Detection of all two wire faults on the DIMMs. Includes any pair of
single bit errors.
FBD Memory Before any transfers to/from FBD can be supported, the MCH DRAM
Configuration registers must be initialized. Detection of memory type and size is
accomplished via the four System Management Bus interfaces on the
MCH (SMBus 1-4). SMBus is basically Intel's version of I2C. SMBus 1-4
are used for FBD channels 0-3 to extract the DRAM type and size
information from the Serial Presence and Detect Port on the DIMMs.
Hi-Speed I/F - Blackford has six 4-lane (x4) PCI Express (PCIe) ports. Each pair of x4
PCIe ports can be configured as an 8-lane (x8) PCIe port. PCIe ports are
PCI Express numbered 2-7. Ports 2 and 3 will be paired together as a single x8 port.
Ports 4, 5, 6, and 7 are connected to IO Modules as x4 ports.
EMC CONFIDENTIAL
Blackford MCH 43
Wildcat-S will also support one or two dual wide IO Modules by pairing
ports 4 with 5 and 6 with 7.
PCIe ports 2 and 3 have a four channel DMA engine. This DMA engine
allows transfers from one device connected to Blackford to another (such
as FBD to FBD, FBD to IO).
Enterprise The Enterprise South Bridge Interface (ESI) is the chip to chip interface
between the MCH and the ICH7. It is a four lane (x4) extension to the
Southbridge standard PCI Express specification with special commands/features
Interface (ESI) added. The ESI consists of two signal groups, inbound and outbound,
each having 4 differential signal pairs clocked at 2.5 GHz.
All CPU accesses that do not map to either the System RAM or one of the
PCIe interfaces (PLX or SLIC’s) will be directed to the ESI and
subsequently to the ICH. This would include processor boot firmware,
Ethernet and RS232 communications, and any other "downstream" I/O
devices.
The ESI is considered part of PCI bus 0, so functions within the ICH
which is attached to the ESI appear in the PCI configuration space on PCI
bus 0, along with the MCH internal functions, even though they are two
physical devices.
EMC CONFIDENTIAL
44
MCH Registers The Blackford MCH has a number of functions or devices which require
configuration before the system can be operational. This is done during
system boot through the PCI Configuration registers for each device.
2
Error information for each device is recorded in various status registers
which are also configured during system boot by the firmware. Access to
these configuration registers is done by accessing the I/O Address and
Data registers in the MCH. The main registers used in the system are
detailed below.
PCI Configuration All functions or devices within MCH appear as PCI devices and therefore
Registers have some common registers for configuration, as well as function-
specific ones.
EMC CONFIDENTIAL
Blackford MCH 45
I/O Access The Wildcat motherboard is designed to use the PCI mapping provided
Registers largely through the MCH and ICH to access most of the devices on board.
This requires support for PCI Memory R/W, PCI I/O R/W, and PCI
Configuration R/W cycles. However, the Pentium CPU can only generate
Memory and I/O cycles on its Front-Side Bus, so a way of generating
Configuration cycles is needed. The CPU uses I/O R/W cycles to a specific
I/O port address to accomplish this.
EMC CONFIDENTIAL
46
FSB #_A_N[35:3] EXP _FRM_MCHA_[7:0]_C+
2
FSB#_ASDTB_N[1:0] EXP _TO_MCHA_[7:0]
FSB#_ADS_N EXP _TO_MCHA_[7:0]
A U29,A P2
FSB#_BR[1:0]_N
FSB #_REQ_N[4:0]
Blackford MCH EXP _FRM_MCHB_[7:0]_C+
PCIe Bus
A V 34,A J9
FSB #_MCERR_N
A U34,A J 10
% = 0, 1, 2, 3 E6
MCH_PE_COMP V1_5
A J27,A H11 R12,P 12
FSB#_RESET _N MCH_100MHZ_CLK+
A N30,A E11 J2
MCH_100MHZ_CLK-
K2
FSB #_HD_N[63:0] V1_5
W10,W11, Y12, A A11
MCH_PEWIDTH [3:0]
FSB #_DP_N[3:0]
K1
MCH_PE_VCCA V1_5
Data Signals
FSB #_DBI_N[3:0]
MCH_PE_VSSA GND
L1
FSB#_DSTBP_N[3:0]
FSB#_DSTBN_N[3:0]
FBD_CH%_SB_[9:0]+
FSB#_TRDY_N
A T32, A K6
FBD_CH%_SB_[9:0]-
FSB#_DRDY_N
A T29, A M3
FBD_CH%_NB_[13:0]+
FSB #_DBSY_N
A R30,A M4
FBD_CH%_NB_[13:0]-
FBD_BRANCH#_CLK+
A A6
MCH_BCLK+
A N17 FBD_BRANCH#_CLK- FB-DIMMs
333 MHz W8
MCH_BCLK-
A P 17
CH 0,1,2,3
MCH_PSEL [2:0] FBD_BRANCH#_VCCA V1_5
Y 10
100b A B 1,A B2, A C1
HP_SMB_SDA
L12
ESI _FRM_MCH_[3:0]_C+
HP_SMB_SCL
SMBus K 13
A K9,A G8, A H6,A H7
ESI _FRM_MCH_[3:0]_C-
I2C MEM_SDA
J 14
A L8, A E8, A G6, A J6 ESI Link
ESI _TO_MCH_[3:0]
MEM_SCL K 14
A K9,A G8, A H6,A H7 to ICH
ESI _TO_MCH_[3:0]
A L8, A E8, A G6, A J6
1.5V MCH_RST_N
MCH_TRST_LV_N G17
A8 V1_5
MCH_PWROK
XDP_TMS_MAIN MCH_CORE_VCCA H17
A7
XDP_TCK
MCH_FSB_VCCA
F35
MCH_FBD_ICOMPBIAS Misc.
MCH_CORE_VSSA
JTAG MCH_TDI
A6
VTT_CPU E 36
MCH_FBD_RESIN Signals
B7
MCH_FBD_RGBIASEXT
MCH_TDO E 37
B6 3.3V
V3REF F13
Blackford _pinout.emf
Blackford MCH 47
.
FSB#_ADS_N AU29/AP2 I/O Address strobe to indicate the beginning of a bus cycle
FSB#_HREQ_N<4:0> I/O Request lines to define the current bus cycle type: Memory R/W, I/O
R/W, Deferred Reply, etc.
FSB#_RS_N<2:0> Out Response code indicating Retry, Deferred Response, Normal Data, etc.
FSB#_DEFER_N AV34/AJ9 Out Indicates a transaction will be accepted but not guaranteed to complete
in order. All I/O and PCI Config accesses are automatically Deferred
FSB#_HIT_N AU32/AK8 I/O Indicates a cache in the system contains data requested
FSB#_HITM_N AV33/AJ7 I/O Indicates a cache in the system has modified the data requested
FSB#_LOCK_N AT30/AL4 In Asserted by an agent during a Read-Modify-Write cycle
FSB#_BINIT_N AK27/AJ4 In Asserted by an agent to indicate that the FSB is not usable and will
re-initialize
FSB#_BNR_N AV30/AK3 I/O Asserted to Block the Next Request to stall the FSB until an agent is
available to handle more transactions
FSB#_BPRI_N AU34/AJ10 Out Asserted by MCH as the Bus Priority Request Interrupt
FSB#_MCERR_N AJ27/AH11 I/O Machine Check Error indicates an unrecoverable FSB fault
FSB#_RESET_N AN30/AE11 Out Hard Reset. Must be asserted for 1 ms after VCC and BCLK are stable
FSB#_DBI_N<3:0> I/O Data Bus invert, when low, indicates that the 16-bit portion of the data
bus is to be inverted (LO = 0, HI = 1)
FSB#_DSTBP_N<3:0> I/O Data Strobe Positive to latch a 16-bit portion of the Data bus
FSB#_DSTBN_N<3:0> I/O Data Strobe Negative to latch a 16-bit portion of the Data bus
FSB#_TRDY_N AT32/AK6 Out Target Ready asserted to indicate Write data may be sent by processor
FSB#_DRDY_N AT29/AM3 I/O Data Ready asserted by agent driving data on the FSB
FSB#_DBSY_N AR30/AM4 I/O Data Bus Busy asserted by agent driving data on the FSB
ESI Link
EMC CONFIDENTIAL
48
Table 2-1. MCH Signal Definitions
SCHEMATIC SIGNAL PIN I/O DESCRIPTION
EXP_TO_MCHA_<7:0>+
EXP_TO_MCHA_<7:0>-
EXP_FRM_MCHB_<7:0>_C+
In
In
Out
PCIe receive lines from PLX bridge
MISC PCI
FBDIMM
FBD_CH%_SB<9:0>+ Out Data & Address to FBDIMM (Southbound)
HP_SMB_SDA L12 I/O Serial data for Hot Plug I2C bus to detect SLIC & Annex insertion and
removal
HP_SMB_SCL K13 I/O Serial clock for Hot Plug I2C bus to detect SLIC & Annex insertion and
removal
MCH_SPD%_SDA I/O Serial data for Serial presence Detect I2C bus to FBDIMM channel %
MCH_SPD%_SCL I/O Serial clock for Serial presence Detect I2C bus to FBDIMM channel %
MISC SIGNALS
MCH_RST_N C2 In Reset for all internal MCH registers except for "sticky" bits
EMC CONFIDENTIAL
Blackford MCH 49
Table 2-1. MCH Signal Definitions
SCHEMATIC SIGNAL PIN I/O DESCRIPTION
MCH_PWROK E3 In Reset for all internal MCH registers including "sticky" bits
MCH_FBD_BGBIASEXT L12 In FB-DIMM Bypass bias input for band pass gap circuit
JTAG
ANALOG SIGNALS
EMC CONFIDENTIAL
50
2.4 FB DIMMS
This Chapter will examine the (FBDIMM) Fully Buffered Dual In-line
Memory Module which is used on the Wildcat-S motherboard.
2
The mother board has eight 240 pin FB DIMM connectors similar to
Figure 2-19 which accept 1 FBDIMM module each.
EMC CONFIDENTIAL
FB DIMMS 51
Memory Blackford MCH masters four Fully Buffered DIMM (FBD) memory
channels seen in Figure 2-20 below. Wildcat-S will support up to two
Controller DIMMs connected to each FBD channel for a total of eight DIMMs.
Wildcat-S
205-800-602c
U8 U7
U16 LAN1 LAN2
U49
INTEL CPU 1
Woodcrest= FSB
Clovertown=
U65 J52_1S
FSB MCH IO Card 2
J22
4Gb/sec each
U96 CH0 CH2
INTEL CPU 0 CH1 CH3
Woodcrest = J11 – FB DIMM
Clovertown= J12 – FB DIMM J51_1S
J9 – FB DIMM IO Card 1
J10 – FB DIMM
J7 – FB DIMM
U120
J8 – FB DIMM
J5 – FB DIMM
J6 – FB DIMM
J50_1S
U141 U140 IO Card 0
SIO FWH U149
NVRAM
EMC CONFIDENTIAL
52
FBDIMM Intro The Wildcat-S Motherboard has slots for 8 FBDIMM’s. Figure 2-21 below
illustrates how each channel has 2 FBDIMM with the first FBDIMM
connected to the MCH and the second FBDIMM daisy chained to the
2
first. There are 4 channels for at total of 8 FBDIMM’s. The clock signals are
also shown here. The last group of signals are the 4 I2C or (SMB)
interfaces which originate from the MCH who is the master.
DRAM DRAM
DRAM DRAM
DRAM DRAM
2 DRAM 2 DRAM 2
Blackford
s
u
DRAM DRAM
Memory
Control DRAM DRAM
DRAM DRAM
DRAM DRAM
DRAM DRAM
2 DRAM 2 DRAM 2
s
DRAM DRAM
DRAM DRAM
DRAM DRAM
MCH_SPD1_SMB
SPD SMBus 1
DRAM DRAM
DRAM DRAM
DRAM DRAM
2 DRAM 2 DRAM 2
s
DRAM DRAM
DRAM DRAM
DRAM DRAM
DRAM DRAM
DRAM DRAM
DRAM DRAM
2 DRAM 2 DRAM 2
s
DRAM DRAM
DRAM DRAM
2
DRAM DRAM
DB1200G
FBD Clock
Buffer
FB DIMMS 53
FBDIMM Wildcat-S will accommodate up to two industry standard FBDIMM per
Blackford MCH channel. The channel consists of 10 Southbound
Channel 0 differential pairs that transport commands and or data from the
Blackford MCH to the FBDIMM. The AMB also forwards the data to the
next FBDIMM. The second piece of the channel is made up of 14
Northbound differential pairs which move data and responses from the
FBDIMM to the Blackford MCH. This data or response will also be
forwarded from the second FBDIMM connected on that channel back to
the MCH.
Channel 0
MCH_SPD0_SCL
MCH_SPD0_SDA
J5 – FB DIMM J6 – FB DIMM
SCL
SCL
SDA
SDA
SA2
SA 2
FBD_00_CLK+ SA1 FBD_01_CLK+
333Mhz SA0
SA 1
CLK+ FBD_00_CLK- FBD_01_CLK- SA 0
CLK-
V3_3
14.318Mhz + 167Mhz
333Mhz
U75 - U121
Clk Clk
Driver Buffer FBD_BR0_RESET_N
U148
U65
Blackford MCH
EMC CONFIDENTIAL
54
Figure 2-23. MCH Branches with balanced DIMM Populated
The four FBDIMM channels are organized into two branches with two
channels on each branch. Each branch is supported by a separate memory
2
controller.
The two channels in each branch operate in lock step to increase FBDIMM
bandwidth. A branch transfers 16 bytes of payload per frame on
Southbound lanes (Blackford to FBDIMM) and 32 bytes of payload per
frame on Northbound lanes (FBDIMM to Blackford).
EMC CONFIDENTIAL
FB DIMMS 55
FBDIMM The FBDIMM direct signaling interface between the memory controller
and the DRAM chips is split into two independent signaling interfaces
Components with a (AMB) Advanced Memory Buffer between them. The interface
between the AMB and DRAM chips is the same one used in today’s DDR2
DIMMS shown in Figure 2-25.
DDR2 SDRAM (72) DATA / (18) STROBE DDR2 SDRAM (72) DATA / (18) STROBE
DDR2 SDRAM (29) Address / (4) Clk DDR2 SDRAM (29) Address / (4) Clk
DDR2 SDRAM (29) Address / (4) Clk DDR2 SDRAM (29) Address / (4) Clk
10
SouthBound Data-In 14 NorthBound Data-Out
Blackford
MCH
However, the interface between the memory controller and the buffer has
changed from a shared parallel interface to a point-to-point serial
interface. These two point-to-point interfaces are referred to as the South
Bound and North Bound interfaces as noted in Figure 2-25 above.
FBDIMM Pinout The pin assignments for the FBDIMM connector are detailed below. The
power and ground pins have been left out. See schematic for more detail.
EMC CONFIDENTIAL
56
Table 2-3. FBDIMM Connector Miscellaneous Signals
2
FBD_00_SA1 240 I2C Address / boot configuration
FBD_00_SA0 239 I2C Address / boot configuration
FBD_00_CLK+ 228 167Mhz Differential Clock
FBD_00_CLK- 229 167Mhz Differential Clock
FBD_BR0_RESET_N 17 RESET TO ALL FBDIMM’S
North Bound and South Bound data signals are shown below.
EMC CONFIDENTIAL
FB DIMMS 57
Table 2-5. FBDIMM Connector Differential Data Pins to next DIMM
EMC CONFIDENTIAL
58
DATA In non-mirrored, dual channel mode, Blackford supports the 18-device
DRAM code option for FBDIMM. This code has the following properties:
Protection
2
(ECC) • Correction of any x4 or x8 DRAM device failure
• Detection of all two wire faults on the DIMMs. Includes any pair of single
bit errors.
Single Device Blackford employs a single device data correction (SDDC) algorithm for
the memory subsystem that will recover from any single x8 or x4 DRAM
Data Correction device failure as well as detect any dual x4 device failure. The chip
(SDDC) disable is a 32-byte two phase code. The MCH also supports demand and
scrubbing.
EMC CONFIDENTIAL
FB DIMMS 59
FBDIMM SMBus SMBus is basically Intel's version of I2C. This SMBus port operates at
100Khz. FBDIMM size and configuration as well as AMB status are
(I2C) accessed by Blackford via a dedicated SMBus interface. Blackford has one
SMBus interface for each FBDIMM channel. Each FBDIMM has three
SMBus address pins (SA2,SA1 & SA0). The first FBDIMM on a channel
will have an address of b'000 and the second FBDIMM in the channel will
be have an address of b'001.
Channel 0
MCH_SPD0_SCL
SCL
SMBus 1 MCH_SPD0_SDA
SDA
SCL
SMBus 2 J5 – FB DIMM J6 – FB DIMM
SDA SCL
SCL
SCL
SMBus 3
SDA SDA
SDA
SCL
SMBus 4 EEPROM EEPROM
SDA
U65
Blackford
AMB AMB
MCH Advanced Advanced
Memory Buffer Memory Buffer
b’001
b’000
SA2
SA2
SA1
SA1
SA0
SA0
V3_3
The AMB has configuration registers that provide flexibility and allow for
testing and optimization of the device. Upon system (FBD_BR0_RST*),
configuration registers are reset to predetermined default states,
representing the minimum feature set required to successfully bring up a
normal channel. It is expected that the BIOS will properly determine and
program optimal configuration settings.
For all of these registers, the AMB supports register access mechanisms
through SMBus as well through in-band channel commands.
FBDIMM Before any transfers to/from FBDIMM can be supported, the MCH
DRAM registers must be initialized. The MCH uses its four System
Memory Management Bus interfaces (SMBus 1-4) to extract the DRAM type and
Configuration size information from the Serial Presence and Detect Port on the DIMMs.
EMC CONFIDENTIAL
60
Reset Each FBDIMM has a low true reset input that uses 1.5V CMOS signaling.
The reset can be asserted by ICH7 GPIO25 or by the MCU GPIO P1.16
Next FBDIMM
DDR2 SDRAM (29) Address / (4) Clk DDR2 SDRAM (29) Address / (4) Clk
10
SouthBound Data-In 14 NorthBound Data-Out
Blackford
MCH
Command, address, and data are transferred from the MCH to the AMB
over a high speed serial interface consisting of ten differential signals
called the South Bound Interface. Data returned from the AMB to the
MCH is transferred over a high speed serial interface consisting of
fourteen high speed differential signals called the North Bound Interface.
If command, address, and data from the memory controller is not meant
for the first FBDIMM, the AMB on that FBDIMM repeats the information
on a separate set of 10 differential signals to the next FBDIMM.
EMC CONFIDENTIAL
FB DIMMS 61
FBD Thermal The Advanced Memory Buffer (AMB) contains an internal thermal diode
to measure AMB/DIMM temperature. Upon detecting an over
Diode temperature, the AMB initiates a thermal throttling event.
Data Flow Data flows in two directions between the Blackford (MCH) and the
FBDIMM’s
South Bound A branch transfers 16 bytes of payload per frame on Southbound lanes
(Blackford to FBDIMM). 16x8=128 bits / 2 DIMM per branch = 64 bits per
DIMM.
EMC CONFIDENTIAL
62
South Bound The below is an example of what command / data may look like coming
across the Southbound 10 Lane serial interface. Transfers 0-3 from the
Data Example MCH sends out the following to the FBDIMM:
•
•
•
There are two bits which contain the Frame Type.
24 bits of command are sent to the FBDIMM.
14 bits contain the CRC associated with the 24 bit command. 2
10 SouthBound Bit Lanes
Transfer
9 8 7 6 5 4 3 2 1 0
0 CRC0 CRC7 CRC8 Frame CMD20 CMD16 CMD12 CMD8 CMD4 CMD0
Cmd/F-type Cmd/F-type Cmd/F-type Type 0
1 CRC1 CRC6 CRC9 Frame CMD21 CMD17 CMD13 CMD9 CMD5 CMD1
Cmd/F-type Cmd/F-type Cmd/F-type Type 1
2 CRC2 CRC5 CRC10 CRC13 CMD22 CMD18 CMD14 CMD10 CMD6 CMD2
Cmd/F-type Cmd/F-type Cmd/F-type Cmd/F-type
3 CRC3 CRC4 CRC11 CRC12 CMD23 CMD19 CMD15 CMD11 CMD7 CMD3
Cmd/F-type Cmd/F-type Cmd/F-type Cmd/F-type
4 CRC21 DATA DATA DATA DATA DATA DATA DATA DATA DATA
Cmd/DATA Cmd/DATA Cmd/DATA Cmd/DATA Cmd/DATA Cmd/DATA Cmd/DATA Cmd/DATA Cmd/DATA
5 CRC20 DATA DATA DATA DATA DATA DATA DATA DATA DATA
Cmd/DATA Cmd/DATA Cmd/DATA Cmd/DATA Cmd/DATA Cmd/DATA Cmd/DATA Cmd/DATA Cmd/DATA
6 CRC19 DATA DATA DATA DATA DATA DATA DATA DATA DATA
Cmd/DATA Cmd/DATA Cmd/DATA Cmd/DATA Cmd/DATA Cmd/DATA Cmd/DATA Cmd/DATA Cmd/DATA
7 CRC18 DATA DATA DATA DATA DATA DATA DATA DATA DATA
Cmd/DATA Cmd/DATA Cmd/DATA Cmd/DATA Cmd/DATA Cmd/DATA Cmd/DATA Cmd/DATA Cmd/DATA
8 CRC17 DATA DATA DATA DATA DATA DATA DATA DATA DATA
Cmd/DATA Cmd/DATA Cmd/DATA Cmd/DATA Cmd/DATA Cmd/DATA Cmd/DATA Cmd/DATA Cmd/DATA
9 CRC16 DATA DATA DATA DATA DATA DATA DATA DATA DATA
Cmd/DATA Cmd/DATA Cmd/DATA Cmd/DATA Cmd/DATA Cmd/DATA Cmd/DATA Cmd/DATA Cmd/DATA
10 CRC15 DATA DATA DATA DATA DATA DATA DATA DATA DATA
Cmd/DATA Cmd/DATA Cmd/DATA Cmd/DATA Cmd/DATA Cmd/DATA Cmd/DATA Cmd/DATA Cmd/DATA
11 CRC14 DATA DATA DATA DATA DATA DATA DATA DATA DATA
Cmd/DATA Cmd/DATA Cmd/DATA Cmd/DATA Cmd/DATA Cmd/DATA Cmd/DATA Cmd/DATA Cmd/DATA
Next Frame below contains the rest of the Cmd /DATA CRC
CRC0 CRC7 CRC8
Transfers 4 to 11 contain data and its 8 CRC which complete the frame.
In the next frame 14 bits of CRC are calculated on the CMD CRC and the
15 bits of DATA CRC by an XOR function.
I am not sure which CRC scheme we are using so Figure 2-28 is meant for
example only at this time.
EMC CONFIDENTIAL
FB DIMMS 63
North Bound A branch transfers 32 bytes of payload per frame on Northbound lanes
(FBDIMM to Blackford). 32x8=256 bits / 2 DIMM per branch = 128 bits per
DIMM.
EMC CONFIDENTIAL
64
Northbound
Data Example
Transfer
0
13
CRC
12
CRC
11
DATA
10
DATA
9
DATA
8
14 NorthBound Bit Lanes
DATA
7
DATA
6
DATA DATA
5 4
DATA
3
DATA
2
DATA
1
DATA
0
DATA
2
1 CRC CRC DATA DATA DATA DATA DATA DATA DATA DATA DATA DATA DATA DATA
2 CRC CRC DATA DATA DATA DATA DATA DATA DATA DATA DATA DATA DATA DATA
3 CRC CRC DATA DATA DATA DATA DATA DATA DATA DATA DATA DATA DATA DATA
4 CRC CRC DATA DATA DATA DATA DATA DATA DATA DATA DATA DATA DATA DATA
5 CRC CRC DATA DATA DATA DATA DATA DATA DATA DATA DATA DATA DATA DATA
6 CRC CRC DATA DATA DATA DATA DATA DATA DATA DATA DATA DATA DATA DATA
7 CRC CRC DATA DATA DATA DATA DATA DATA DATA DATA DATA DATA DATA DATA
8 CRC CRC DATA DATA DATA DATA DATA DATA DATA DATA DATA DATA DATA DATA
9 CRC CRC DATA DATA DATA DATA DATA DATA DATA DATA DATA DATA DATA DATA
10 CRC CRC DATA DATA DATA DATA DATA DATA DATA DATA DATA DATA DATA DATA
11 CRC CRC DATA DATA DATA DATA DATA DATA DATA DATA DATA DATA DATA DATA
I am not sure which CRC scheme we are using so Figure 2-29 is meant for
example only at this time.
EMC CONFIDENTIAL
FB DIMMS 65
EMC CONFIDENTIAL
66
2.5 PLX PEX8524 PCIe Switch
The PLX PEX8524 switch provides a x8 PCI Express CMI path between SP
A and SP B. It also provides an x8 (or dual x4) PCIe path across the mid
plane to its corresponding IO Annex slot.
The Wildcat-S implementation of the PLX PEX 8524 chip is shown in the
figure below.
MIDPLANE 8
SP A SP B
PLX PLX
CPU A CPU B U16 CPU A CPU B U16
U96 U49 U96 U49
8 8
PCI-E PCI-E
MCH Port A MCH Port A
MEM CTLR
MEM CTLR
DMA DMA
DIMM’s
DIMM’s
Engines Engines
U65 U65
plx_cmi_block.emf
Overview The PEX8524 (hereafter referred to as the PLX) is a multi-port PCIe switch
which may be used to connect PCIe devices in various ways such as a
fan-out from a single host to multiple endpoints, or as a communication
bridge between peer host controllers. In the Clariion Fleet family, the
peer-to-peer communication function is implemented so as to provide the
CMI (Command Management Interface) path between SP A and SP B.
EMC CONFIDENTIAL
PEX8524
PCI-E Ingress Egress PCI-E
Scheduler Scheduler
Port Port
Station 1 Station 0
Non-Blocking Crossbar
Upstream Crossbar Crossbar
8 Port Switch Switch
Ingress Egress
Switch Fabric
Dev 8
to MCH
8
Dev 0
to I/O
Annex
Crossbar Crossbar
NT Port Switch Switch
8 Egress Ingress
Dev 9
to CMI
Egress Ingress
Scheduler Scheduler
plx_port_block.emf
PLX Functions An important functionality of the PLX is the ability of it to allow a host to
access devices within the address space or domain of another host or
peer, while preventing those devices from being discovered or
enumerated by the first host. In a typical PCI/PCIe environment the BIOS
will enumerate the PCI busses, devices and functions in the system by
accessing the PCI Configuration Registers and cataloging the Device ID,
Vendor ID and Header Type of the ones that respond. The memory and
I/O addresses of the devices are assigned at this time, so typically this is
done once only at reset.
The Header Type indicates the type of the device, whether the device is a
regular PCI device (Header Type 0) or if it is a PCI Bridge of some kind
(Header Type 1). If it is a bridge, the software must then enumerate any
busses (called subordinate busses) and devices on the other side of the
bridge before continuing on with the next device after the discovered
bridge. If each host were to try to enumerate all the devices that each
could access in the system, including each other’s devices, the discovery
process would fail because the hosts could enumerate the devices more
than once with unpredictable results.
There is one upstream port that interfaces with the MCH and two
downstream ports that interface with the IO Annex card and the peer SP.
At this point it can be noted that both downstream ports operate
differently. Both ports perform a different type of bridging function:
EMC CONFIDENTIAL
68
• Transparent Bridging to the IO Annex
2
The PLX prevents redundant enumeration by configuring one of its ports
as Non-Transparent. This means that even though it functions as a PCI
Bridge (Header Type 1)which allows accesses to propagate over a
subordinate bus to the peer devices, it reports that port to the
enumerating software as an end device (Header Type 0), thus preventing
any discovery of any devices in the peer. Transparent ports are reported
as Header Type 1 and so allow discovery of any subordinate busses and
their connected devices.
The process is completely symmetric between the SP’s so that they both
see their own local RAM buffer beginning at 0x4000 0000 and their Peer’s
buffer beginning at 0x8000 0000. The translation process is described in
Figure 2-32.
1 SPA
8200 1000 Port 8 PLX Port 9
FSBCTL Upstream Port Virtual Port Link Port
Non-Transparent
PCI-E Port A Registers Registers Registers
C000 0XXX C001 0XXX C001 1XXX
MCH BFFF FFFF BFFF FFFF BFFF FFFF 7FFF FFFF
Memory Window Memory Window Memory Window Memory Window
ADDRESS
8200 1000 8200 1000 8200 1000 TRANSLATION 4200 1000
2 3
2 4
2 8000 000 4000 000
6000 000 6000 000
5
2
MIDPLANE
8
2 PCI-E Port A
Registers
C000 0XXX
Registers
C001 0XXX
Registers
C001 1XXX
BFFF FFFF BFFF FFFF BFFF FFFF 7FFF FFFF
MCH Memory Window Memory Window Memory Window Memory Window
7
2 4200 1000
plx_adr_xlate.emf
EMC CONFIDENTIAL
2. The PCIe port in the MCH claims the transaction since the address falls
within its memory window (0x6000 0000 - 0xBFFF FFFF) as defined by its
base and limit registers. It forwards the transaction onto the PCIe link to
the PLX.
3. The PLX upstream port (Port 8) receives the transaction from MCH and
determines that it is within its own memory window and so passes it to
the downstream Non-Transparent port (Port 9).
5. The Link Port modifies the address of the transaction to its own
memory window (0x4000 0000 - 0x7FFF FFFF) and sends the new
transaction on the PCIe bus to the Midplane and Peer SP.
6. The Peer PLX Link Port receives the transaction and determines
whether the address of the transaction falls within its memory window
(ox4000 0000 - 0x7FFF FFFF) and forwards it to the Virtual Port.
8. The Peer MCH PCIe Port 8 receives the transaction and determines that
it does NOT fall within its memory window so it also forwards it
upstream where the Memory Controller recognizes the address and
claims the transaction. The Peer Memory Controller then accesses the
DIMM’s at the modified address (0x4000 0000 - 0x7FFF FFFF).
PLX The PLX may be configured by several mechanisms. These are H/W pin
strapping, EEPROM and Configuration Registers.
Configuration
H/W Strapping At power on reset some of the configuration parameters for the PLX are
set to a default value by strapping certain I/O pins on the package high or
low by means of pull-up or pull-down resistors. These pins set bits in the
PLX to an initial value at power on reset, which may be overwritten later
by either the EEPROM or software. The functions defined by these pins
are detailed in Table 2-8.
EMC CONFIDENTIAL
70
Table 2-8. PLX H/W Strapping Pins
2
SCHEMATIC NAME PIN Description
CFG_ST0_PORTCFG[4:0] D25,E25, Configures the number of lanes in ports 0,1,2,3 (x0 = disabled) in Station 0
F25,G25, 00000 = x4, x4, x4, x4
H25 00001 = x16, x0, x0, x0
00010 = x8, x8, x0, x0
00011 = x8, x4, x4, x0
00100 = x8, x4, x2, x2
00101 = x8, x2, x2, x4
00110 = x8, x2, x4, x2
CFG_ST1_PORTCFG[3:0] D25,E25, Configures the number of lanes in ports 0,1,2,3 (x0 = disabled) in Station 1
F25,G25, 0000 = x4, x4, x4
H25 0001 = x0, x0, x0
0010 = x8, x0, x0
0011 = x4, x4, x0
0100 = x4, x2, x2
0101 = x2, x2, x4
0110 = x2, x4, x2
CFG_PLX_TESTM[3:0] J28,J27, Test Mode select: reserved for Factory testing only.
J30,J29 1111 = Default (Test Modes Disabled)
EEPROM The PLX also relies on an EEPROM to load other configuration data at
each power on reset. The EEPROM used in the Fleet family is an Atmel
AT25256AN 256Kbyte (32K x 8 bits internal) serial device. This device
uses the SPI bus to connect to the PLX. Contained in the EEPROM is the
initial values of the PCI Configuration Registers. Not all the registers
need to be set at this time, but in the Fleet systems the Device ID, Vendor
ID and Header Type are.
EMC CONFIDENTIAL
PLX J25
PLX_EE_CS_N
1 8
PLX_EE_SCK PUP_PLX_EEHLD
H26 6 EEPROM 3
PLX_EE_DO PDN_PLX_EEWP_N
J26 5 AT25256 AN 7
PLX_EE_DI
G26 2 4
PDN_PLX_EEPR_N
K26
plx_eeprom.emf
The SPI Master (PLX) will drive the Clock and Chip Select lines to the
Slave (EEPROM). Data driven by the PLX on the MOSI line (Command
byte followed by Address) is valid on the rising edge of the clock. If it is a
Write operation the data continues to be valid on rising clock edges. If it is
a Read operation the data driven by the EEPROM is valid on the falling
clock edge.
CLK
CS0 Valid on rising clock edge Write data valid on rising clock edge
Read data valid on falling clock edge Master terminates transfer
Master drives command and address Master drives write data
COMMAND ADDRESS Write DATA #1 Write DATA # N
MOSI
EMC CONFIDENTIAL
72
PLX Registers The PLX contains a number of registers that are used for configuring its
operation and reporting status and errors during operation. All registers
may be accessed by Memory read/write cycles from the CPU once the
2
PLX has been configured by the BIOS.
PCI Configuration At system reset the devices in the system must be enumerated by the
Registers BIOS. PCI devices are enumerated and discovered by their location in the
system topology and are identified by their particular Bus Number
(0-255), Device Number (0-31), and Function number (0-7). The BIOS
must then assign memory or I/O addresses to the devices.
38 Reserved
FF 3C Max_Lat Min_Gnt Int_Pin Int_Line
mch_pci_cfgspace.emf
Doorbell and To facilitate communication between peer hosts the PLX contains
Scratch pad registers for passing messages or interrupts to each other called the
Registers Doorbell registers. These are available when the port is configured as
Non-Transparent. Setting bits in a Set register will cause interrupt packets
to be sent on the link, only if the corresponding bits are clear in a Mask
register. Both the Virtual port and the Link port have Doorbell registers.
EMC CONFIDENTIAL
Scratch pad registers are similar in that they allow access from either the
Virtual port (local side) or the Link port (peer side). The main difference
is that writing to them will not cause an interrupt, so that they are
effectively dual-ported general-purpose registers.
EMC CONFIDENTIAL
74
PLX Signals The signal names used on the Dreadnought for the PLX is shown in
Figure 2-37 on page 75.
MCH
I/F
EXP_FRM_MCHA_<7:0>+
EXP_FRM_MCHA_<7:0>-
EXP_TO_MCHA_<7:0>_C+
EXP_TO_MCHA_<7:0>_C-
IO_EXP_R<7:0>+
IO_EXP_R<7:0>-
IO_EXP_T<7:0>_C+
IO_EXP_T<7:0>_C-
IO
ANNEX
PATH
2
CMI_EXP_R<7:0>+
PLX_EE_DI A G27
CMI_EXP_R<7:0>-
CMI
EEPROM
PLX_EE_DO
PLX_EE_CS_N
A G30
PLX CMI_EXP_T<7:0>_C+ PATH
I/F
PLX_EE_SCK
A G28
A G29
PEX8524 CMI_EXP_T<7:0>_C-
PDN_PLX_EEPR_N A H28
U16
VDD33A
PLX_PCI_RST_N
G3
V3_3
MISC. PLX_100MHZ_CLK_C+ V1
A J4
PLX_100MHZ_CLK_C- V1_A
A K4 POWER
V1_5
GND
P/U CFG_SERDES_MD
A C3
00010 b CFG_ST0_PORTCFG<4:0> G4
PLX_TMS
A C28, A C27,A D28,A D27,A E28
PLX_TDO
CONFIG 0010 b CFG_ST1_PORTCFG<3:0>
P29, N28, N29,M 28
H1
plx_pinout.emf
EMC CONFIDENTIAL
PCIeXPRESS SIGNALS
EEPROM INTERFACE
PLX_EE_DI AG27 In PLX input from the serial EEPROM data output.
PLX_EE_DO AG30 Out PLX output to the serial EEPROM data input.
PDN_PLX_EEPR_N AH28 In Serial EEPROM present input. Tied to GND to indicate presence of
EEPROM
MISCELLANEOUS
PLX_PCI_RST_N G3 In General reset for this PLX
PLX_100MHZ_CLK_C+ AJ4 In Positive half of 100-MHz PCI Express reference clock input signal pair
EMC CONFIDENTIAL
76
Table 2-10. PLX PEX8524 Signals
SCHEMATIC SIGNAL PIN I/O DESCRIPTION
PLX_100MHZ_CLK_C- AK4 In Negative half of 100-MHz PCI Express reference clock input signal pair
JTAG
PLX_TMS
PLX_TDO
PLX_TDI_MUX
G4
H1
J1
In
Out
In
JTAG Test Mode Select
POWER
EMC CONFIDENTIAL
78
2.6 ICH 7
Introduction The ICH 7 IO Controller Hub is based on the Intel IO Controller Hub 5
architecture. ICH 7 integrates much of the Super IO functions and is targeted at
2
embedded servers. It has several different interfaces:
I2C- ICH 7 provides both a master and slave I2C interface. In the SAN
environment the ICH 7 is the SMB master however in a NAS
environment the System Management Card provides the master for the
SMB and the ICH 7 acts as a slave.
EMC CONFIDENTIAL
ICH 7 79
.
AH28 FSB_A20M_N
ESI_FRM_MCH_<3:0>+/- FSB<1:0>_IERR_33_N
ESI_TO_MCH_<3:0>C+/- AG26 FSB_FERR_N
C25 AG22 FSB_IGNNE_N
DMI DMICOMP D25 AH22 FSB_STPCLK_N
(Blackford MCH) ICH_DMI_CLK+/- AF25 FSB_INTR
INTERFACE BOOT
MCH_ERR<2:0> AH24 FSB_NMI
CPU
AF23 FSB_SMI_N INTERFACE
AF22 FSB_INIT_N
L_FRAME_N AB3 AG23 ICH_RCIN_N N.U.
L_AD<3..0> AE22 ICH_A20GATE
FWH_INIT_N AG21
FWH & SIO LPC
L_DRQ0_N AC3
INTERFACE R4 FLT_BL_YLW_N
ICH_SER_IRQ AH21
AC22 SIO_ANNEX_FLT
I2C_ARB_A1_BUF_OUT
A2 I2C_ARB_A1_BUF_IN
LAN1
EXP_TO_ICH1+/- F25/F26 AF21 ICH_ALERT_BUF_N
EXP_FRM_ICH1+/- E27/E28 E22 CF_INS_N
LAN2 EXP_TO_ICH2+/- H25/H26 AE19 PLEASE_BIF_N
EXP_FRM_ICH2+/- G27/G28 D8 FBD_CLKEN_N GPIO
EXP_TO_ICH3+/- K25/K26 AH18 ICH_SWA_INS_N
DBG X1 PCI-E EXP_FRM_ICH3+/- AE20 ICH_SWB_INS_N
J27/J28
CONN E19 PS1_I2C_ATN_BUF_N
AD20 IO_ANNEX_INS_N
PUP_ICH_P_FRAME_N F16 E23 ICH_SYS_FLT_N
PCI PUP_ICH_P_PERR_N C9 B21 ICH_SW_FLT
INTERFACE
PUP_ICH_P_SERR_N B10
(Not Used)
ICH_33MHZ_CLK A9 AC20 ICH_USB_RST_N
ICH_PCIRST_N B18 D20 ICH_FBD_RST
A21 IO_I2C_BUFRST
RESETS
A13 PS1_I2C_BUFRST
ICH_SDA B22 C26 ICH_PLTRST_N
ICH_SCL C22 Y4 ICH_RSM_RST_N
SMB / I2C
ICH_SLAVE_SDA
ICH_SLAVE_SCL
A25
B25
ICH7 A22 ICH_RST_IN_N
INTERFACE PROCESSOR
P.U. ICH_LINKALERT_N A26 AF26 FSB_THERMTRIP_N
G7 TEMPERATURE
P.U. PUP_INTRUDER Y5 PROCHOT_<1:0>_33_N
INDICATORS
BTN_IS_SMI_N
F8 ICH_PIRQG_N
ICH_USB_R_<7:4>,<1:0>+/- G8 ICH_PIRQE_N
ICH_USB_OC<4:0>_N B5 ICH_PIRQD_N
USB INTERUPTS
C5 ICH_PIRQC_N
iNTERFACE ICH_USB_RBIAS
USB_48MHZ_CLK B4 ICH_PIRQB_N
B2
A3 ICH_PIRQA_N
C23 ICH_PWRBTN
ICH_SATA<3:0>_RX_C+/-
SATA AA4 ICH_PWROK
ICH_SATA<3:0>_TX_C+/- POWER
INTERFACE AC18 ICH_PSA_INS_N INTERFACE
ICH_SATA_CLK+/-
AC21 ICH_PSB_INS_N
N.U. PUP_ICH_SATA_LED
ICH_SATA_RBIAS AB1 ICH_RTCX1
AC2 ICH_RTCX2 CLOCKS
ALT_CMI_TX_BUF AF19
To MIDPLANE ALT_CMI_RX_BUF AC1 ICH_14MHZ_CLK
R3
C_FPGA_CONFIG0_N U2 A27 ICH_SUS_STAT_N POWER
C_FPGA_CNFDNE1_12_N C20 SIO_32KX1 MANAGEMENT
AH19 INTERFACE
C_FPGA_STATUS0_N AD21 AD22
PERSONALITY C_FPGA_DATA0_N PUP_ICH_VRMPG
AC19 F20
BOARD PUP_ICH_WAKE_N
C_FPGA_DCLK0_N AG18
SIGNALS AH20 PUP_MCH_SYNC_N
AF20 PUP_ICH_THRM_N
PUP_ICH_RI_N MISCELLANEOUS
CFG_ICH_INTVRM W4 A28
PUP_ICH_SPKR IDE_A<02:00> N.U.
A19
MISCELLANEOUS IDE_D<15:00> N.U.
PUP_RTC_RST_N AA3
ICH_SLP_S<5:3>_N N.U.
PUP_ICH_DXFTEST_N C21
PUP_ICH_TP0_N A3 ICH7_ACZ_SYNC
F21 HanceRapids _New_Pinout .vsd
A3 ICH7_ACZ_SDOUT
80
Signal
Descriptions
Table 2-11. ICH 7 Pin Descriptions
SCHEMATIC SIGNAL
Blackford MCH & DMI Interface
I/O Description 2
ESI_FRM_MCH_<3:0>+/- I One of two differential strobe lines used to transmit and receive data through the hub
interface
ESI_TO_MCH_<3:0>+/- O One of two differential strobe lines used to transmit and receive data through the hub
interface
DMICOMP I/O Determines the DMI input and output impedances
ICH_DMI_CLK+/- I 100MHz Differential clock used to run the DMI Interface.
MCH_ERR<2:0> I Indicates to the ICH 7 that the Blackford MCH has a fault condition.
Firmware Hub / LPC Interface
LFRAME* I/O FWH4/LFRAME is a muxed signal. LFRAME#:
Indicates the start of an LPC cycle, or an abort.
LAD(3..0) I/O LPC Muxed Address Data bus to the firmware hub FWH(3..0) or SIO chip
LAD(3..0). These signals have internal pull-ups inside the Hance Rapids.
LDRQ0_N I The Super IO will use this signal to request either a DMA or Bus master access of
the LPC interface.
FWH_INIT_N O Asserted by the ICH 7 for 16 PCI clocks to reset the FWH. It is a copy of the signal
FSB_INIT_N which is used to reset the processor’s.
ICH_SER_IRQ I/O An interrupt request to/from the Server I/O.
PCI Express Interface
EXP_TO_ICH<3:1>+/- I Serial PCI Express Interface to the LAN1, LAN2 and DBG X1 PCI-E CONN
EXP_FRM_ICH<3:1>+/- O Serial PCI Express Interface from the LAN1, LAN2 and DBG X1 PCI-E CONN
PCI Interface (Not Used)
PUP_ICH_P_FRAME_N I Not Used
PUP_ICH_P_PERR_N I Not Used
PUP_ICH_P_SERR_N I Not Used
ICH_33MHZ_CLK I Not Used
ICH_PCIRST_N O Not Used
SMB INTERFACE
ICH_SDA O ICH 7 (master) I2C clock
ICH_SCL I/O ICH 7 (master) I2C data.
ICH_SLAVE_SDA I ICH 7 (slave) I2C clock.
ICH_SLAVE_SCL I/O ICH 7 (slave) I2C data.
ICH_LINKALERT_N I/O P.U.
PUP_INTRUDER I P.U.
USB INTERFACE
ICH_USB_R_<7:4>,<1:0>+/- I/O Differential pairs used to transmit & receive Data/Address/Command signals for the
ICH 7 USB ports.
EMC CONFIDENTIAL
ICH 7 81
Table 2-11. ICH 7 Pin Descriptions
SCHEMATIC SIGNAL I/O Description
ICH_USB_OC<4:0>_N I Indicate to the USB controller that an over current condition exists
ICH_USB_RBIAS I/O USB Resistor Bias. Analog connection point for an external resistor. Used to set
transmit currents and internal load resistors.
ICH_48MHz_CLK I 48MHz Clock used to run the USB controller.
Serial ATA INTERFACE.
ICH_SATA<3:0>_RX_C+/- I Differential signals for receiving data over the serial ATA interface
ICH_SATA<3:0>_TX_C+/- O Differential signals for transmitting data over the serial ATA interface
ICH_SATA_CLK+/- I 100MHz clock used to run the SATA controller.
PUP_ICH_SATA_LED O N.U. This is an open-collector output pin driven during SATA command activity.
ICH_SATA_RBIAS I/O Analogue connection point for an external connection point to ground.
GPIO.
FLT_BL_YLW_N I A high on this pin and a low on the SIO_FAULT_LED pin will light the blade fault
LED blue.
SIO_ANNEX_FLT I A high on this signal will light the IO Annex fault LED.
I2C_ARB_A1_BUF_IN I A low on this pin indicates the A I2C bus is being requested.
I2C_ARB_A1_BUF_OUT_N O Driven high to request ownership of the shared A I2C bus.
ICH_ALERT_BUF_IN O Driven low by the ICH 7 to indicate a PS1 I2C failure exists.
CF_INS_N I Compact FLASH is inserted
PLEASE_BIF_N I A low indicates that the Tornado extension card wants the PLX configured for dual
84 PCIe.
FBD_CLKEN_N O A low will enable the FBD clock buffer.
ICH_SWA_INS_N I A low indicates that the A management board is inserted.
ICH_SWB_INS_N I A low indicates that the B management board is inserted.
PS1_I2C_ATN_BUF_N I A low indicates a PS1 I2C bus error.
IO_ANNEX_INS_N I A low indicates that the IO annex A ia inserted.
ICH_SYS_FLT_N O Asserted to light the System Fault LED
ICH_SW_FLT O Asserted to light the Management Fault LED
GPIO Resets
ICH_USB_RST_N O Driven low by the ICH 7 to reset the USB HUBs
ICH_FBD_RST O Driven high by the ICH 7 to reset the Fully Buffered DIMMs.
IO_I2C_BUFRST O Driven high by the ICH 7 to reset the IO Modules I2C buses
PS1_I2C_BUFRST I/O A high on this pin will reset the PS1 I2C bus. Default to 0
ICH_PLTRST_N O Platform Reset: The ICH 7 asserts PLTRST* to reset devices on the platform that is
the SIO & FWH. The ICH 7 asserts PLTRST* during power-up and when S/W
initiates a hard reset sequence through the Reset Control Register. The ICH 7 drives
PLTRST* inactive for a minimum of 1 ms after both PWROK and VRMPWRGD
are driven High.
ICH_RSM_IN_N I This signal is used to reset the resume power plane logic. When low the internal
LAN Controller will be put into reset.
ICH_RST_IN_N I This signal forces an internal reset after being de-bounced. The ICH 7 will reset
immediately if the SMBus is idle, otherwise, it will wait for up to 25ms+/- 2ms for
the SMBus to idle before forcing a reset on the system.
EMC CONFIDENTIAL
82
Table 2-11. ICH 7 Pin Descriptions
SCHEMATIC SIGNAL I/O Description
Boot CPU U47 Interface
FSB_A20M_N
FSB<1:0>_IERR_33_N
O
I
Mask A20: A20M# will go active based on either setting the appropriate bit in
the Port 92h register, or based on the A20GATE input being active.
Speed Strap: During the reset sequence, the ICH 7 drives A20M#
high when the corresponding bit is set in the FREQ_STRP register.
When asserted indicates that the relevant CPU is indicating an error condition.
2
FSB_FERR_N I Numeric Coprocessor Error: This signal is tied to the coprocessor error signal
on the processor. FERR# is only used when the ICH 7
coprocessor error reporting function is enabled in the General Control Register
FSB_IGNNE_N O Ignore Numeric Error: This signal is connected to the ignore error pin on the
CPU. IGNNE# is only used when the ICH 7 coprocessor error
reporting function is enabled in the General Control Register
FSB_INIT_N O Initialization: INIT# is asserted by the ICH 7 for 16 PCI clocks to reset the
processor. The Hance Rapids may be configured to support CPU BIST. In that case,
INIT# will be active when PXPCIRST# is active.
FSB_INTR O CPU Interrupt: INTR is asserted by the ICH 7 to signal the
processor that an interrupt request is pending and needs to be serviced. It is an
asynchronous output and normally driven low.
Speed Strap: During the reset sequence, the ICH 7 drives INTR
high when the corresponding bit is set in the FREQ_STRP register.
FSB_NMI O Non-Maskable Interrupt: NMI is used to force a non-Maskable interrupt to the
processor. The ICH 7 may generate an NMI when either SERR#
or IOCHK# is asserted
FSB_SMI_N O System Management Interrupt: SMI# is an active low output synchronous to
PCICLK. It is asserted by the ICH 7 in response to one of many
enabled hardware or software events.
FSB_STPCLK_N O Stop Clock Request: STPCLK# is an active low output synchronous to
PCICLK. It is asserted by the Hance Rapids in response to one of many
hardware or software events. When the processor samples STPCLK# asserted,
it responds by stopping its internal clock.
ICH_RCIN_N I Keyboard Controller Reset CPU: The keyboard controller may generate INIT#
to the processor. This saves the external OR gate with the ICH 7_other sources of
INIT#. When the ICH 7 detects the assertion of this signal, INIT# is generated for 16
PCI clocks.
ICH_A20GATE I A20 Gate: From the keyboard controller. Acts as an alternative method to force
the A20M# signal active.
Processor Temperature Indicators
FSB_THERMTRIP_N I When low, this signal indicates that a thermal trip from the processor occurred.
PROCHOT_<1:0>_33_N I A low indicates that either processor 0 or 1 has asserted PROCHOT.
Power Interface
ICH_PWRBTN I The Power button will indicate a system request to go to a sleep state. If the system
is already in a sleep state, this signal will cause a wake event to occur.
ICH_PWROK I When asserted, this signal is an indication to the ICH 7 that core power has been
stable for 99ms and that PCICLK has been stable for 1 ms. When PWOOK is
de-asserted the ICH 7 asserts ICH_PLTRST_N
ICH_PSA_INS_N I When asserted indicates that power supply A is inserted
ICH_PSB_INS_N I When asserted indicates that power supply B is inserted
Clocks
ICH_RTCX1 I This signal is connected to the 32.768kHz crystal.
ICH_RTCX2 I This signal is connected to the 32.768kHz crystal.
ICH_14MHZ_CLK I This 14MHz clocks used to run the 8254 timers internal to the ICH 7
EMC CONFIDENTIAL
ICH 7 83
Table 2-11. ICH 7 Pin Descriptions
SCHEMATIC SIGNAL I/O Description
84
Table 2-11. ICH 7 Pin Descriptions
SCHEMATIC SIGNAL I/O Description
PUP_RTC_RST_N I Routed to the CMOS Clear Jumper J15 CONN2. When asserted, this signal resets
2
register bits in the Real Time Clock (RTC) well.
PUP_ICH_DXFTEST_N I/O Pulled Up
PUP_ICH_TP0_N I This signal must have an external pull-up to V3_3.
MGMT_USB+/-
ICH_USB_R_0+/- USB
USB MIDPLAN
PORT 0 Controller
HUB ANNEX_USB+/- CONNECTO
MSYSTEMS
USB ICH_USB_R_<4,5>+/-
UDOC
PORT’S
4& 5 Connector
IO3_USB_+/-
USB ICH_USB_R_7+/- IO2_USB_+/-
PORT 7 USB 4-Port
Io Module
HUB IO1_USB_+/-
Controller Connector
IO0_USB_+/-
ICHUSBInterface.emf
EMC CONFIDENTIAL
ICH 7 85
ICH 7 USB Port
Port 4 & 5 Connected to a debug connector capable of
connecting to the MSystems uDoc Flash IO
Module.
Port 6 Connected to the Management controller
LAN Interfaces The Wildcat-S board will include two 10/100/1000 Mbit interfaces, each
utilizing a Broadcom Gigabit Ethernet controller BCM5751.Each LAN
interface will be attached to the ICH 7 using a PCI Express Root Port.
There is a serial EPROM connected to the chip to store the MAC address
for the interface. LAN boot will be supported by this interface.
Management Module A
RJ45
BCM
BCM5397
5751 RJ45
Ethernet
SW
BCM RJ45
5751
Wildcat A M
i IO Annex A
d
p
l
a
n
e IO Annex B
Wildcat B
Management Module B
BCM
5751 RJ45
BCM5397
Ethernet RJ45
BCM SW
5751
RJ45
Wcat_S_LAN.emf
EMC CONFIDENTIAL
86
There are two BCM5751 populated on the Wildcat-S motherboard,
referred to as LAN1 and LAN2. LAN1’s Ethernet port is connected to the
BCM5325E switch on the local switch via the midplane. LAN2’s Ethernet
port is connected to the BCM5325E switch on the peer switch also via the
2
midplane. They are both connected in parallel to the ICH 7 via the PCI
Express bus.
Operation On power up the device’s core logic exits the reset state 80ms after all
three power rails reach their respective minimum threshold levels.
The PCI-Express core logic will exit the reset state 600us after
LAN_RST_N de-asserts and V3.3V is above its minimum threshold level.
After exiting from the reset state, the boot processor begins executing
code from the internal start-up ROM image. Execution of the startup code
requires, the processor to load boot code from the external EPROM into
internal scratch pad memory. Start-up code will be executed when the
device is reset via the hardware reset signal or a software reset.
Clocks A single external free running clock source is required for the core
functions of the device to operate. The device requires a 25MHz clock
input from either an external crystal on the crystal signal pins
LAN1_X<1:0>. From this single clock source, the device, the device
derives a 125-MHz clock fro the Gigabit PHY, and a 62.5 MHz clock fro
the core.
Configuration After reset, the BCM5751 must be configured so that it may function in
the system. This is done by the system CPU reading and writing over the
PCI bus to the Configuration registers in the BCM5751. After the basic
Configuration registers are programmed, the CPU programs additional
registers so that the BCM5751can locate its data structures in system
RAM.
ICH 7 87
Transmit To transmit data, the System CPU will place the data payload and the
data structures describing it into system RAM. The CPU will then write
over the PCI-Express bus to a register in the BCM5751 to inform the
device that there is a packet for transmission. The BCM5751 will then
move the packet data from host memory to its internal buffer
memory.The MAC transmitter then initiates a transmission over the
MII/GMII interface to the Management module.
LAN_RST_N C2
ICH 7 LAN1_REFCLK_SEL F4 D8 LAN1_PCIE_TST
Interface
LAN1_100MHZ_CLK+/-
Ethernet Tranceiver
LAN1_TRD0_C+/-
LAN1_TRD1_C+/-
LAN1_TRD2_C+/-
LAN1_TRD3_C+/-
PDN_LAN1_SI E12
A10 LAN1_RDAC
SPI Interface PDN_LAN1_SCLK E11
Not Used PDN_LAN1_CS_N C12 LAN1_X1
P12
N12 LAN1_X0 CLOCKS
PDN_LAN1_LOW_PWR L6
LAN1_ATTN_BTTN_N A2
JTAG TEST
LAN1_TDI H12
LAN1_TDO D6
LAN1_TMS C11
JTAG LAN1_TRST_N D12
LAN1_TCK D7
BCM5751 _Pinout.emf
Signal
Descriptions
Table 2-13. BCM5751 Pin Descriptions
SCHEMATIC SIGNAL I/O Description
PCI-E Interface
EXP_FRM_ICH1+/- I PCI Express receive differential pair
EXP_TO_ICH1+/- O PCI Express transmit differential pair
EMC CONFIDENTIAL
88
Table 2-13. BCM5751 Pin Descriptions
SCHEMATIC SIGNAL I/O Description
LAN_RST_N I The PCI-E reset. The PCI_E core exits reset state 600us after LAN_RST_N is
2
de-asserted and the internal Power On Reset (POR) circuitry detects the correct
voltage threshold level on the V3_3 volt input (2.67V). The PCI-E core logic will
enter reset state if the 3.3V power rail drops below 2.67V or LAN_RST_N input is
asserted. Assertion of this signal does not reset the on-chip MAC or other modules
LAN1_REFCLK_SEL I This signal determines which input reference the clock the PLL will use to generate
its internal PCI-E clock. This pin is pulled to 3.3V on the Wildcat-S board which
configures the controller to use the PCI-E LAN1_100MHZ_CLK+/-
LAN1_100MHZ_CLK+/- I 100MHz differential clock which provides timing for all PCI-E transactions.
Firmware Hub / LPC Interface
PUP_LAN1_WAKE_N O Asynchronous output request for a change in power state.
LAN1_PCIE_TST I PCI-E test mode used for ATE testing. Must be pulled low for normal operation.
Failure to pull low will place the chip in non-operational mode.
Ethernet Transceiver Interface
LAN_TRD0_C+/- I/O Ethernet interface to the management module
LAN_TRD1_C+/- I/O Ethernet interface to the management module
LAN_TRD2_C+/- I/O Ethernet interface to the management module
LAN_TRD3_C+/- I/O Ethernet interface to the management module
Miscellaneous Signals
LAN1_EECLK I/O Serial EEPROM data.
LAN1_EEDAT I Serial EEPROM clock.
LAN1_RDAC I DAC Bias Resistor. This signal is used to adjust the current level of the DAC.
LAN1_X1 I 25MHz crystal input
LAN1_X0 O 25MHz crystal output
JTAG INTERFACE
LAN1_TDI I JTAG serial data in.
LAN1_TDO O JTAG serial data out.
LAN1_TMS I JTAG mode select.
LAN1_TRST_N I reset for the JTAG controller.
LAN1_TCK I/O JTAG serial clock
EMC CONFIDENTIAL
ICH 7 89
EMC CONFIDENTIAL
90
2.7 Server IO & Slow Devices
The Server I/O (SIO) chip is an integrated I/O controller which provides
the processor with a number of interfaces to various slow devices in the
system. The Server I/O chip may also be referred to as the Server I/O in
this manual. The other supporting devices attached to the SIO are:
2
• NVRAM
• BIOS / POST (FLASH)
• RS232 level Buffers
U141 U149
XAD<7:0>
U18
U143 XA<11:4>
SIO X-Bus
Buffer
LPC Bus
L_AD<3:0> NVRAM
ICH PC87417 XA<17:12>
U137
Buffer
4 XA<3:0> XA<17:0>
RS232
SIO_TXD1 TXD1_CBT
U140 U127
SIO_RXD1
MIDPLANE
BIOS/POST RXD1_CBT
FLASH
SIO_TXD2 TXD2_CBT
SIO_RXD2 RXD2_CBT
Server I/O The Server I/O device (SIO) used in Wildcat-S is the PC87417. It is a
128-pin PQFP package which provides the following functions and
interfaces:
EMC CONFIDENTIAL
EMC CONFIDENTIAL
92
Signal
Descriptions
Table 2-14. Super IO Chip Pin Descriptions
SCHEMATIC SIGNAL PIN PU/PD I/O
CLOCKS - RESET - BATTERY LOW SIGNALS
Description 2
SIO_33MHZ_CLK 114 I 33Mhz Clock from the U40 Clock Synthesizer.
SIO_32KX1 42 I Suspend Clock: (N/U) From the ICH RTC generator circuit (32.768 KHz).
SUSCLK will have a duty cycle that may be as low as 30% or as high as
70%.
SIO_PCI_RST_N 120 I The Hance Rapids ICH asserts ICH_PXPCIRST_N which becomes this
signal. This is hardware reset of SIO active low. The signal also resets the
FHW.
BATTERY_LOW_N 49 I From the U51 chip which monitors the Lithium battery.
Address / Data Multiplexed Bus & controls (XBUS Interface)
XAD<7..0> PU I/O Super IO multiplexed address / data bus. Goes to NVRAM & Port80 debug
latch chip which controls 8 leds. (see diagram)
XSTRB_O 34 O This signal latches out XA<11..4> to the NVRAM
XSTRB_1 33 O This signal latches out XA<16..12> to the NVRAM
XRD_N 14 O This signal controls the output enable to the NVRAM
NVRAM_CS_N 22 O This signal controls the chip select to the NVRAM
XWR_N 15 O This signal controls the write enable to the NVRAM and is also ORED with
PORT80_CS_N signal to control the Port 80 debug latch chip clock.
XA<3..0> O These signals go the NVRAM address pins XA<3..0>.
DUART_CS_N 23 O Chip Select for the IO Module located on the IO Annex card
PORT80_CS_N 21 O
QUART_CS_N 20 O
Firmware Hub / LPC Interface
L_FRAME_N 117 I/O FWH4/LFRAME is a muxed signal. LFRAME#:
Indicates the start of an LPC cycle, or an abort.
L_AD(3..0) I/O LPC Muxed Address Data bus to the firmware hub FWH(3..0) or Hance
Rapids chip LAD(3..0). These signals have internal pull-ups inside the Hance
Rapids.
L_DRQ0_N 118 O LPC Serial DMA/Master Request Input: To the Hance Rapids Chip to
request DMA or bus master access of the LPC bus.
ICH_SER_IRQ 119 I/O Serial Interrupt Request: This pin implements the serial interrupt protocol.
FAULT OUTPUTS
FAULT_REG<7..0> O These signals are outputs to the fault expander chip’s U73 and U74.
SIO_BLADE_FAULT 50 O Indicates blade fault when this signal goes active low. Causes the blue Fault
LED on Airdam to Blink.
When this signal is high it indicates no fault and disables the Fault LED.
SAFE2RM_HOT_LED 51 O Will light LED CR36_2S indicating that the FRU is too hot to remove
Keyboard and Mouse Signals to the Debug Compression Header which is Depopulated.
KB_CLK 125 O Keyboard clock. Pulled up.
KB_DATA 126 I/O Keyboard data. Pulled up
EMC CONFIDENTIAL
EMC CONFIDENTIAL
94
Table 2-14. Super IO Chip Pin Descriptions
SCHEMATIC SIGNAL PIN PU/PD I/O Description
SIO_DCD2_N 102 I Data Carrier Detected. Connected to the Switch Management Module via
2
the midplane.
INTERRUPTS
PS2_I2C_ATTN_N 54 I
IO0_2_4_INT_N 78 I
IO1_3_5_INT 10 I Asserted low when PSA detects no AC, also asserted when PSA is powered
by DC from SPS
POWER
BATTERY_LOW_N 49 PU I Asserted low when PSB detects no AC, also asserted when PSB is powered
by DC from SPS.
MARGIN_LO_N 56 PU O Output to on-board power circuits to margin voltages down 3%
MARGIN_HI_N 48 PU O Output to on-board power circuits to margin voltages up 3%
MFG_MODE_N 35 PU I Used by manufacturing to disable voltage monitoring while margining.
MISC. SIGNALS
SIO_WDO_N 55 O SIO_WDO needs to be configured by BIOS to be used as WD timer output to
U35 Micro controller. Pulses when a watchdog timeout occurs.
HLD_IN_PST_N 124 I
The below tables contains the chip selects that the Super IO chip outputs
to control other devices.
EMC CONFIDENTIAL
In the Wildcat-S system some of the SIO hardware strapping signals are
left at their default settings and therefore use the internal pull-up or
pull-down resistors within the chip to set the configuration parameters.
EMC CONFIDENTIAL
96
BIOS Chip The BIOS / POST chip which is sometimes referred to as the firmware hub
is a FLASH device which contains the BIOS and POST code for the
Wildcat-S board. It is a 2 Meg x 8-bit programmable non-volatile memory
2
which interfaces directly to the LPC bus. The clock for the LPC bus is
provided by the clock generator chip and runs at 33 MHz.
SIO ICH7
PC87417 L_FRAME_N
L_AD<3:0>
U140
U75 FWH_33MHZ_
CLK_R BIOS/POST
FLASH FWH_INIT_N
Sio _fwh_block.emf
The SIO_PCIRST_N signal is used to reset the device memory while the
FWH_INIT_N (CPU Reset) is used to reset the memory when the CPU is
reset.
SIO COM The Super IO connects to 8 COM Ports in total. They are:
PORTS Table 2-17. Super IO COM Ports
EMC CONFIDENTIAL
DUART A Dual UART connected to the Super IO provides two serial COM Ports
which are used to communicate with the FPGA on the IO Modules
connected to the Wildcat board via the IO Annex Module. The ports will
be used to communicate with the fibre ports using diplexed RS-232.
COM7_CS_N
DUART_CS_N
U135
COM8_CS_N
XAD<7:0>
DUART_TO/FRM_IO4 U136
XA<2:0>
DUART_TO/FRM_IO5 DUART
U143
XAD<7:0>
XAD<7:0>
U149 XA<17:12>
XA <11:4>
U137
NVRAM XA<3:0> U141
NVRAM_CE_N
XA<3:0>
DUART_TO/FRM_IO0
XAD<7:0> NVRAM_CS_N SIO
DUART_TO/FRM_IO1 U120 U13
XA<2:0> PC87417
DUART_TO/FRM_IO2 XWR_N
QUART
DUART_TO/FRM_IO3
XRD_N
PORT 80_CS_N
U120
PORT 80
CONN XAD<7:0>
COM<6:3>_CS_N QUART_CS_N
U93
Sio _port_80.emf
QUART The Quad UART is connected to the SuperIO XBUS and provides four
additional serial COM Ports. These COM Ports are used to communicate
to the FPGAs on each IO MOdule connected to the Wildcat via the IO
Module connectors. The COM ports will be used to communicate with
the fibre ports using diplexed RS-232.
EMC CONFIDENTIAL
98
PORT 80 The BIOS shall have the ability to transmit Port 80 codes as well as Fault
codes to the enclosure management micro controller. A port 80 debug
board may be inserted to view the Port 80 codes.
Dallas NVRAM The DS1314 ia an Nonvolatile Controller with Lithium Battery Monitor.
Controller This chip is a self-contained device which converts a standard low power
SRAM into a non-volatile memory. A precision voltage reference and
comparator monitors the Vcc input for an out-of-tolerance condition.
When an invalid Vcc condition occurs, the conditioned chip enable
output (NVRAM_CE_N) is forced inactive independent of the chip select
signal (NVRAM_CS_N) from the SIO chip. This effectively write-protects
the stored data in the SRAM.
During a power failure, the controller switches the SRAM power from the
Vcc pin (V3_3) to the battery to provide the energy required for data
retention. On a subsequent power-up, the SRAM remains write protected
until a valid power condition returns.
EMC CONFIDENTIAL
100
2.8 Clocks & Interrupts
Clocks and Figure 2-46 below shows the locations of the crystals and oscillators used
by the Wildcat-S Motherboard. The following is a list of their frequencies
2
Oscillator and the parts which use each one.
Locations
• Y1 - LAN 2 - 25Mhz
• Y2 - LAN 1 - 25Mhz
• Y3 - Flash Device - 24Mhz
• Y4 - ICH - 32.768Khz
• Y5 - Management Controller (MC)- 12Mhz
• Y6 - Depopulated and not show above
• Y7 - Uart, Quart & 4 IO Modules - 33.333Mhz
• Y8 - Clock Drive - 14.318Mhz
INTEL
U51 U50
Woodcrest =
Clovertown =
U65
J52_1S
MCH
Y8
J22S Y7
U96
INTEL
Woodcrest= J11 – FB DIMM
Clovertown= J12 – FB DIMM J51_1S
J9 – FB DIMM
J10 – FB DIMM
J17
J7 – FB DIMM U2691
U121
U120
J8 – FB DIMM
U133 U131 U128 U125
J5 – FB DIMM
U127
J6 – FB DIMM
U143 U137
EMC CONFIDENTIAL
The MCH also receives a 333Mhz clock signal from the driver.
A 100Mhz clock signal is sent from the driver to the U52 clock buffer
which is used to fan out up to 12 copies of this clock to various
components. Only the 100Mhz to the MCH is shown below.
The (U75) clock driver and both buffers (U52 & U121) are connected to
the I2C SMBus (not shown) which can be used by the ICH to read/write
to configuration command registers in each component.
U75 J17
IDTCV126 CPU0_BCLK_+/- 2 U96
BCLK0/1 CPU0 XDP Conn
CK410B 333Mhz (CPU)
BCLK0/1
CPU1_BCLK_+/- 2 U49 2 XDP_BCLK_+/-
BCLK0/1 CPU1
333Mhz 333 Mhz
PECLK_+/- 100Mhz
FREQ_SEL[2:0]
[b’100] = 333Mhz U52 MCH_100MHZ_CLK_+/-
DIF_0+/-
100Mhz
BYPASS
CK_DB1200 _SRC_+/-
CLK_IN+/-
DB1200G U121 167Mhz
Clk Buffer DIV2 DIF_9+/-
DB1200 G DIF_8+/- 167Mhz
Clk Buffer
333Mhz
DIF_10+/-
Y8 – Crystal DB1200 G_MEM_CLK_+/-
CLK_IN+/-
14.318MHz 333Mhz
EMC CONFIDENTIAL
102
FBDIMM Clocks The 8 FBDIMM’s slots each receive an 167Mhz Differential Clock from the
U121 clock buffer. The FBDIMM’s use this clock to re-time the data
signals on writes to the DDR2 chips used on the FBDIMM’s and it is also
2
used as the clock source to clock the data back over the serial interface
Lanes to the MCH on reads from the FBDIMM’s.
The clock buffer takes as its input frequency 333Mhz from the U75 clock
driver chip. The clock buffer is programed by the ICH over the I2C lines
for a gear ratio of 1/2 or divide by 2 out to the FBDIMM’s.
FBD_00_CLK_+/- 2
U75 U121 DIF_1+/- FBDIMM 00
FBD_11_CLK_+/- 2
DIF_2+/- FBDIMM 11
167Mhz
FBD_20_CLK_+/- 2
DIF_5+/- FBDIMM 20
FBD_21_CLK_+/- 2
DIF_4+/- FBDIMM 21
333Mhz
DB1200 G_MEM_CLK_+/-
CLK_IN+/- FBD_30_CLK_+/- 2
DIF_6+/- FBDIMM 30
FBD_31_CLK_+/- 2
Y8 – Crystal
DIF_7+/- FBDIMM 31
14.318MHz
FLASH Clock The Cypress NAND FLASH/ Controller module U2 is a plug-in device
connected to the USB interface to the ICH. It has a external 24Mhz
oscillator that supplies the clock signal through the connector for the U2
modules PLL circuit.
U2
Y3 – Crystal NAND
NAND
24MHz FLASH
FLASH
Controller
EMC CONFIDENTIAL
U75
CK410B
U18
ICH_SATA_CLK+/- ICH7
SRC_4+/- SATACLK_+/- PECLK_+/-
USB_48MHZ_CLK
USB48MHZ CLK48
ICH_33MHZ DMICLK_+/-
PCIF_1 PCI_CLK
ICH_14MHZ
REF_0 CLK14
Y4 – Crystal
FREQ_SEL[2:0] = [b’100] 32.768kHz
100Mhz U52
CK_DB1200 _SRC_+/- ICH_DMI_CLK+/-
DIF_1+/- 2
SRC_0+/- CLK_IN+/-
BYPASS
DB1200G
Clk Buffer
Y8 – Crystal
14.318MHz
EMC CONFIDENTIAL
104
PLX Clocks The PLX8532 receives an 100Mhz differential clock signal from the U52
clock buffer. It uses this clock for its PCI-Express interface controller. The
PLX has its own internal clock for other operations.
U75
2
CK410B FREQ_SEL[2:0]
[b’100] = 333Mhz
U52 PLX8532
DIF_0+/-
BYPASS
CK_DB1200_SRC_+/- PLX_100MHZ_CLK_+/-
SRC0+/- CLK_IN+/- DIF_2+/-
100 Mhz 100Mhz
Y8 – Crystal
14.318MHz
LAN Clocks LAN 1,2 chips each have an 25Mhz crystal oscillator connected to them.
This clock is used for operational purposes. Both LAN chips U8 & U7
receive an 100Mhz differential clock from the U52 clock buffer which they
use for their PCI-Express interface controllers.
U8
U75 Y2 – Crystal
25MHz LAN1
FREQ_SEL[2:0] = [b’100] BCM5751
CK410B
100Mhz U52 LAN1_100MHZ_CLK_+/- 2
CK_DB1200 _SRC_+/- DIF_4+/-
SRC_0+/- CLK_IN+/- LAN2_100MHZ_CLK_+/-
DIF_5+/- 2
BYPASS
DB1200G
Clk Buffer
U7
Y1 – Crystal
25MHz LAN2
BCM5751
EMC CONFIDENTIAL
U75
CK410B
U141
SIO
SIO_33MHZ_MHZ
REF_1 CLK
U140
FWH
FWH_33MHZ_CLK
PCIF_0 CLK
FREQ_SEL[2:0] = [b’100]
Y8 – Crystal
14.318MHz
MC Clock The Management Controller chip receives a 12Mhz clock from the Y5
crystal oscillator attached to it. This is the operational clock used by the
MC.
U35
Y5 – Crystal Management
12MHz Controller
LPC2146
EMC CONFIDENTIAL
106
IO Modules and There are 4 IO Module connectors on the Wildcat-S motherboard. Each
connector receives an 100Mhz differential clock which is used by the IO
Uarts Clocks Module (SLIC) inserted in each slot for PCI-Express Interface operations.
2
Each IO Module connector also receives an 33Mhz clock which may be
used by the different requirements of each IO Module.
The Dual and Quad UART’s both receive the 33Mhz signal which they
use as an operational clock. The 33Mhz signal is derived from the Y7
Oscillator which outputs a frequency of 33.333Mhz.
U52
U75 BYPASS
DB1200G
CK410B 100Mhz Clk Buffer
CK_DB1200 _SRC_+/- IO0_100MHZ_CLK_+/-
CLK_IN+/- DIF_6+/-
IO1_100MHZ_CLK_+/-
DIF_7+/-
IO2_100MHZ_CLK_+/-
DIF_8+/-
IO3_100MHZ_CLK_+/-
DIF_9+/-
Y8 – Crystal
14.318MHz
Midplane & The Debug Connector is not populated on the Wildcat-S motherboard but
it would have received the 100Mhz differential clock signals.
Debug
Connector Dreadnought’s midplane receives an 100Mhz differential clock signal
from U52 clock buffer.
Clocks
U52
U75 BYPASS
DB1200G
CK410B 100Mhz Clk Buffer
CK_DB1200 _SRC_+/- 2 DEBUG_100MHZ_CLK_+/-
CLK_IN+/- DIF_3+/- Debug Conn
Populate-false
2 MP_100MHZ_CLK_+/-
DIF_10+/- Midplane
Y8 – Crystal
14.31818 MHz
EMC CONFIDENTIAL
DUART_IO5_INT
B
DUART
0 QUART_IO1_INT INT[A,B]
Clovertown Firmware XIRQ A
QUART_IO3_INT
THERMTRIP_N
0
(CPU0) Hub
DUART_IO4_INT
SIO
MCERR_N
IGNNE_N
FERR_N
BINIT_N
IERR_N
SMI_N 0 QUART_IO0_INT B
INIT_N
ACK D
INTR
A QUART
QUART_IO2_INT
NMI
0
SERIRQ
C
INT[A,B,C,D]
Legacy IRQs
0
GTL IGNNE_N FERR_N SERIRQ PIRQB
0
to INIT3_3N PIRQD
TTL GPIO8 PIRQA
0
GPIO12 0
PIRQC
INIT_N
INTR MSI/INTx Root
NMI PCI Express LAN1
SMI_N
ICH 7 MSI/INTx Root
PCI Express LAN2
MSI/INTx Root
MCERR_N
BINIT_N
INIT_N
IGNNE_N
IERR_N
THERMTRIP_N
INTR
SMI_N
FERR_N
NMI
Debug
PCI Express
connector
GPIO
MCU_I2C MCU
IDEIRQ Compact Flash
Clovertown Reset PIRQF
MCU PIRQG
(CPU1) Expander
ESI GPIOs PIRQE PIRQH
X4 ESI
Memory Write (MSI/INTx
FSB
FSB
ICH to Blackford )
or No Connect
PLX Switch PCI Express A
PCI Express C
PCI Express B
PCI Express B
MSI/INTx
3:0
7:4
3:0
7:4
Wildcat-N/S Interrupt IO 0 IO 1 IO 2 IO 3
IO0_INS*
IO_ANNEX_INS*
BTN_IS_SMI_N from Management Module
SIO Interrupts The Server IO Chip receives two interrupt signals which are a combined
from the DUART & QUART chips as follows:
from DUART &
QUART • IO1_3_5_INT - Occurs when an incoming message is received by:
- the QUART from IO Module 1 or IO Module 3
- or the DUART from IO Module 5.
• IO0_2_4_INT - Occurs when an incoming message is received by:
- the QUART from IO Module 0 or IO Module 2
- or the DUART from IO Module 4.
EMC CONFIDENTIAL
108
ICH7 Interrupts The interrupts received by the ICH7 are described below:
2
a message is in the receiver buffer.
• ICH_PIRQB interrupt is received from the QUART from IO Module 2 when
a message is in the receiver buffer.
• ICH_PIRQC interrupt is received from the QUART from IO Module 1 when
a message is in the receiver buffer.
• ICH_PIRQD interrupt is received from the QUART from IO Module 0 when
a message is in the receiver buffer.
• ICH_PIRQE interrupt is received from the U79 SMBUS Expander when a
IO Module or IO Annex Module insert signal changes voltage level.
• ICH_PIRQF interrupt is received from the I2C Expander when a Force SMI
is received from the MCU.
• ICH_PIRQG interrupt is received from the IO Annex.
• ICH_PIRQH interrupt is received from the MCU on the Management FRU
when the reset button has been depressed.
SIO
DUART_IO4_INT 0
QUART_IO0_INT ACK
0
QUART_IO2_INT
DUART_IO5_INT 0
QUART_IO1_INT XIRQ
0
QUART_IO3_INT SERIRQ
V3_3 ICH7
SERIRQ
0
PIRQA
0
PIRQC
0
LPC2146 PIRQB
0
MCU I2C V3_3 PIRQD
EXPNDR PIRQE
FORCE_SMI_N PIRQF
V3_3
14.3K
ICH_PIRQG_N IO_ANNEX_INT_N PIRQG
BTN_IS_SMI_N PIRQH
V3_3
14.3K
MP_BTN_IS_SMI_N
Other Interrupts The interrupts not covered here may be covered in other sections of the
manual if time allows.
EMC CONFIDENTIAL
110
2.9 Chassis Management (LAN)
EMC CONFIDENTIAL
SAN Chassis The Dreadnought system utilizes a distributed switch architecture for
management of the system. Each Wildcat-S Chassis has two Management
LAN Cabling FRU’s. Each FRU has a Management LAN port which is used to connect
to the control station. The service port may be used for servicing the
system. Figure 2-59 shows an example of the Ethernet cabling for a SAN
system comprising a single Wildcat-S Chassis and two control stations.
The below is for example only. See the Dreadnought System Configuration Guide for
actual configurations
DAE
DAE
DAE
DAE
Wildcat-S Blade B
Wildcat-S Blade A
Control Station
Control Station
SPS SPS
EMC CONFIDENTIAL
112
The Wildcat-S SAN Chassis in Figure 2-60 shows the LAN Ethernet
signals within a SAN type chassis. The chassis has two Solar Flare
Management FRU’s which contain an 6-port Ethernet switch to provide
the redundant fabrics. See the section about “Solar Flare - Management
2
FRU (SAN)” on page 147 for more details.
Power Supply - A
Solar Flare Management FRU – A
CROSSLINK_TD<3..0>+/- BMC5397 Switch Management
PORT 0 PORT 3
LAN Port
MGT_A_BLD_A_TD<3..0>+/-
PORT 2
J13 J1
J3 Service
PORT 1 PORT 4
LAN Port
Blower Fan
ITRAC
U8 U7 IO Module 3
LAN 1 LAN 2
ITRAC
IO Module 2
Wildcat-S
J5 Storage Processor
A
ITRAC
MGT_B_BLD_A_TD<3..0>+/-
IO Module 1
Blower Fan
ITRAC
IO Module 0
M
I J21 IO A
D
MGT_A_BLD_B_TD<3..0>+/-
Nova
Test J8 P IO ANNEX
Card
L
J7 IO B
Blower Fan A
N
E
ITRAC
U8 U7 IO Module 3
LAN 1 LAN 2
ITRAC
IO Module 2
Wildcat-S
MGT_B_BLD_B_TD<3..0>+/-
IO Module 1
ITRAC
IO Module 0
J14
Solar Flare Management FRU – A
BMC5397Switch
PORT 2 PORT 3 Management
PORT 1 LAN Port
J1 CROSSLINK_TD<3..0>+/-
J3 Service
PORT 0 PORT 4
LAN Port
Power Supply - B
Figure 2-60. Wildcat-S SAN Chassis - LAN Block Diagram
EMC CONFIDENTIAL
LAN1_TRD0_C+ EXP_TO_ICH1_C+
LAN1_TRD0_C- BCM5751 EXP_TO_ICH1_C-
U18
LAN1_TRD1_C+ U8 EXP_FRM_ICH1_C+
ICH7
J22S LAN1_TRD1_C- EXP_FRM_ICH1_C-
Midplane LAN1_TRD2_C+
Connector LAN1_TRD2_C- LAN1_100MHZ_CLK+ U52
LAN1_TRD3_C+ LAN1_100MHZ_CLK+ Clk
Driver
LAN1_TRD3_C-
LAN1_EEDAT
LAN1_EECLK U15
EEPROM
LAN1_REFCLK_SEL LAN1_EEWP
V3_3
LAN1_ATTN_BTTN_N LAN1_XI
25 MHz Y2
LAN1_RDAC
LAN1_XO
LAN1_PCIE_TST
LAN1_TRST_N
LAN1_TCK
LAN1_TDI JTAG
U18 ICH_PLTRST_A_N LAN_RST_N LAN1_TMS
ICH7 V3_3 LAN1_TDO
EMC CONFIDENTIAL
114
SAN In an SAN system there is only one Wildcat-S Chassis which has 2 Solar
Flare Management FRU’s installed. It may also include DAE’s, SPS and
Control Station.
SP B Blade
2
Empty SP A Blade slot
Service
LAN Port
Management
LAN Port
Solar Flare SAN IO Annex Solar Flare SAN
Mgnt. FRU A Slots Mgnt. FRU B
Solar Flare FRU Each Solar Flare FRU has two 10/100/1000 Ethernet ports:
• 1 LAN Service port can be used to access the system by service person.
• 1 LAN Management which connects to the customer LAN network for
management purposes.
Fault LED Power OK
LED
Link LED
Activity
LED
Service
LAN
Console Port
Management
LAN
SPS Port
Activity
LED
Link LED
Figure 2-63. Solar Flare Management FRU Face Plate
Each RJ45 ethernet connector has a yellow port activity LED and a green
Link State LED.
EMC CONFIDENTIAL
Resume CMD
Prom Module
Peer MGMT.
Crosslink-
I2C
I2C - SDA/SCL
Blade A
Blade B
Isolation
RS232
RS232
Switch
USB
Midplane Connector
See“Solar Flare - Management FRU (SAN)” on page 147 for more details.
EMC CONFIDENTIAL
116
NAS Chassis In an NAS Dreadnought system there can be multiple Wildcat-S Chassis
which uses 2 EarthQuake Management FRU’s each. The NAS system may
LAN Cabling also include DAE’s, SPS and Control Station.
The below is for example only. See the Dreadnought System Configuration Guide for
actual configurations.
Wildcat-S Blade B
Wildcat-S Blade A
IO ANNEX-B IO ANNEX-A
U U
Wildcat-S Blade B
Wildcat-S Blade A
IO ANNEX-B IO ANNEX-A
UP-LINK U U
CS / UPS
Control Station
Control Station
UPS UPS
EMC CONFIDENTIAL
118
The Wildcat-S NAS Chassis in Figure 2-60 shows the LAN Ethernet
signals. The chassis has two Earthquake Management FRU’s which have
three ethernet LAN ports. See the section about “Earthquake -
Management FRU (NAS)” on page 169 for more details.
Power Supply - A
PORT 2 PORT 3 CS1 or UPS Port
2
MGT_A_BLD_A_TD<3..0>+/-
J2 BMC5397
J13
J4 EarthQuake Mgnt FRU B Switch PORT 0 Up-Link Port
PORT 1 PORT 4 Down-Link Port
Blower Fan
ITRAC
U8 U7 IO Module 3
LAN 1 LAN 2
ITRAC
IO Module 2
Wildcat-S
J5 Storage Processor
A
ITRAC
MGT_B_BLD_A_TD<3..0>+/-
IO Module 1
Blower Fan
ITRAC
IO Module 0
M
I
D J21 IO A
MGT_A_BLD_B_TD<3..0>+/-
Nova P
Test J8
L IO ANNEX
Card
A J7 IO B
Blower Fan N
E
U8 U7
ITRAC
IO Module 3
LAN 1 LAN 2
ITRAC
IO Module 2
Wildcat-S
MGT_B_BLD_B_TD<3..0>+/-
IO Module 1
ITRAC
IO Module 0
Power Supply - B
Figure 2-66. Dreadnought NAS - LAN Block Diagram
EMC CONFIDENTIAL
EMC CONFIDENTIAL
120
Earthquake Block The Earthquake Management Module is an (NAS) type FRU. It has three
Diagram RJ45 connectors which is used for:
• Up-link Port
2
• Down-link or Control Station Port
• Second Control Station or UPS Port.
These three ports are connected to a BCM5397 ethernet switch which
allows the ports to communicate with:
Resume CMD
Prom Module
Peer MGMT.
Crosslink-
I2C
Blade A
Blade B
Isolation
PS1 - I2C
PS2 - I2C
RS232
Switch
USB
Midplane Connector
EMC CONFIDENTIAL
122
2.10 Chassis Management (RS232)
EMC CONFIDENTIAL
IO Module 1 IO Module 5
DUART_FRM_IO1 DUART_TO_IO5
Diplex Diplex
DUART_TO_IO1 DUART_FRM_IO5
FPGA FPGA
Midplane
Fogbow
QUART
IO Module 2
DUART_FRM_IO2 Management Module
Diplex
XCVR
DUART_TO_IO2 SIO_RXD0/TXD0 To/From
FPGA
SPS
SIO
IO Module 3
DUART_FRM_IO3
XCVR
Diplex
DUART_TO_IO3
SIO_RXD1/TXD1 To/From
FPGA Console
MCU
MCU_RX0/TX0
HDR
XCVR
PC_RX/TX
EMC CONFIDENTIAL
124
DUART A Texas Instruments TL16C752B Dual UART is connected to the Super IO
XBUS to provide two added serial ports for Wildcat. These two serial
ports are used to communicate to an FPGA on each IO Module connected
2
to Wildcat via the IO Annex Module, that will handle the communication
for diplexing to the Fibre ports.
U141 XA2
26
U136 5
DUART_FRM_IO4
SERIAL A
Server IO XA1
27
DUART To Midplane 7
DUART_TO_IO4
Y7
33.33Mhz
Only TX, RX, and RTS will be supported. Other signals are not routed.
EMC CONFIDENTIAL
U141 XA2
28
U120 77
DUART_FRM_IO0
DUART_IO0_CDA_N
79
DUART_IO0_DSRA_N
Serial A 3
COM3_CS_N DUART_IO0_DTRA_N
9 STATUS 5
COM4_CS_N CTRL PUP_DUART_IO0_RIA_N
13 78
U93 COM5_CS_N PDN_DUART_IO0_CTSA_N
4
49
QUART_CS_N COM6_CS_N
53 DUART_IO1_CDB_N
23
DUART_IO1_DSRB_N
Serial B 19
U18 Misc. DUART_RESET_1 STATUS 17 DUART_IO1_DTRB_N
33 CTRL
ICH7 Circuitry
24
PUP_DUART_IO1_RIB_N
PDN_DUART_IO1_CTSB_N
18
QUART_33MHZ_CLK DUART_IO2_CDC_N
U78 31 39
DUART_IO2_DSRC_N
Serial C 43
STATUS DUART_IO2_DTRC_N
45
CTRL PUP_DUART_IO2_RIC_N
Y7 38
33.33Mhz PDN_DUART_IO2_CTSC_N
44
DUART_IO3_CDD_N
63
QUART_CLKSEL DUART_IO3_DSRD_N
PU 26 Serial D 59
DUART_IO3_DTRD_N
STATUS 57
CTRL PUP_DUART_IO3_RID_N
64
PDN_DUART_IO3_CTSD_N
58
EMC CONFIDENTIAL
126
The Diplex FPGA on the IO Modules are capable of supporting 8 separate
ports for Diplexing. The DUART ports could support these 8 ports
running at 19.2K BAUD.
2
Only TX, RX, and RTS (flow control) will be supported. Other signals are
not routed.
EMC CONFIDENTIAL
128
2.11 Chassis Management (I2C)
SAN vs NAS I2C Wildcat-S supports both Dreadnought SAN and NAS system
management architectures which are incompatible. This is done by using
Architecture two different types of Management FRU’s and having the Wildcat-S
board read the type of Management FRU’s installed and the midplane
resume (VPD) chip which contains the system type. Once the Wildcat-S
knows this information it will load use the appropriate software code
which configures the system as NAS or SAN.
I2C Access There are two I2C access types used in a Wildcat-S Chassis.
Types • The first type is a simple scheme and only involves accessing I2C buses
local to the Wildcat-S Blade and its components. The blade includes local
I2C interfaces to the IO Modules. Arbitration signals are required to be used
for some but not all I2C buses.
• The second type involves access to the midplane I2C interfaces (PS_A_I2C
& PS_B_I2C) which will require arbitration between multiple I2C masters
which are located on different Modules within the Wildcat-S chassis.
EMC CONFIDENTIAL
IO Annex A PIC
Solar Flare A PIC
Solar Flare B IO Annex B
0x60 CMD CMD
0x60
0x?? 0x??
Resume Resume
PROM 0xAE PROM 0xAE
0 1 0 1
Sw 0 xEA Sw 0 xEA
I2C I2C
Devices Devices
PS A PS B
Micro Micro
0x52 Resume 0x52
PROM 0xAA
Resume PS_A_I2C Resume
PROM0xAC PROM0xAC
PS_B_I2C
Fogbow
MidPlane
Wildcat-S A
8 bit I2 C
expander
0x4E
Redundant
Reset
Hold Post
Wildcat-S B 8 bit I2 C
expander
0x4E
Redundant
Reset
Hold Post
CMD CMD
Master for SAN U23 Master for SAN 2
U23
I2 C switch 0X24 I C switch 0X24
PECI 0xE4 PECI
0xE4 0x54 0x54
ICH7 1 3 4 5 6 0
ICH7 1 3 4 5 6 0
0x88 0x88
U18 slave I2C switch U18 slave I 2C switch
master 0XE6 master 0XE6
1 of 4 MUX SEL0 To I2C I2C I2C I2C 1 of 4 MUX SEL 0 To I2C I2C I2C I2C
SEL1 SIO SEL 1 SIO
Devices Devices Devices Devices Devices Devices Devices Devices
11 01 10 00 11 01 10 00
Resume Resume
(0xA4) (0xA4)
Clock Clock
MCH MCH
0XC0 0XC0
EMC CONFIDENTIAL
130
SAN - I2C In a SAN Dreadnought system each Wildcat-S chassis may contain 2
Solar Flare Management Modules, 2 Wildcat-S Blades which include IO
Modules, IO Annex Modules and 2 power supplies which are not show in
2
Figure 2-73. Each Wildcat-S Blade has one I2C master which is the ICH7.
The Solar Flare does not have a I2C master and the arbitration signals are
not used by the Solar Flare FRU as indicated in Figure 2-73. The ICH7
(I2C master) on the Wildcat-S Blades use the I2C_ARB_A1 &
I2C_ARB_B1 signals to arbitrate for access to the (PS_A_I2C & PS_B_I2C)
midplane I2C buses. See Figure 2-74. This will allow the ICH7 to access
components on both itself and its Peer Wildcat-S Blade as well as the
midplane resume (VPD), Management FRU, both power supplies and IO
Annex modules.
I2C_ARB_A1_BUF_0_N U27
U18
ITRAC
IO Module 3
ICH I2C_ARB_A1_BUF_IN
U28
MP_I2C_ARB_A1
U99
ITRAC
I2C_ARB_A1
IO Module 2
J5
Wildcat-S SP
A
ITRAC
MP_I2C_ARB_B1
IO Module 1
I2C_ARB_B1
U104
M U144 U141
ITRAC
I SIO_ARB_B1 SIO IO Module 0
D
I2C_ARB_B1
P I2C_ARB_A1_BUF_0_N U27
U18
ITRAC
L ICH I2C_ARB_A1_BUF_IN
IO Module 3
A U28
MP_I2C_ARB_A1
N U99
ITRAC
IO Module 2
E
J5
Wildcat-S SP
B
ITRAC
MP_I2C_ARB_B1
IO Module 1
I2C_ARB_B1
U104
U144 U141
ITRAC
SIO IO Module 0
SIO_ARB_B1
J2
J4 SolarFlare Mgnt FRU B
EMC CONFIDENTIAL
YES
NO
Pull Control Signal t(n) = n*t1
Low for T = t(n) t(n) – Time delay for
each host trying
to get on I2C bus.
Report Failure on
Is NO one Control Signal n – Host Number
Control Signal Blade A = 1
Continue using only
Low?
remaining Blade B = 2
YES
Release
t1 = 20us
Control Signal t2 = 10us
for T = t2
Is
Control Signal NO
High?
YES
The Control Signal will be either
Pull Control Signal I2C_ARB_A1 or I2C_ARB_B1
Low until done
depending upon which I2C bus
with I2C bus
is being arbitrated for.
Is
Control Signal NO
Low? t2
MP_I2C_ARB_(A/B)1
YES
One Control
Done, Success Signal
Host can
t(n) connect to
Wildcat-S SAN _I2C Arbitration Flow .emf I2C Bus
Figure 2-74. Midplane SAN I2C Access Arbitration Diagram
EMC CONFIDENTIAL
132
Midplane I2C The midplane I2C buses for the power supplies are crossed on the
Wildcat-S Chassis midplane. This prevents I2C conflicts since both blades
Buses (NAS) will have a CMD, resume prom, switch, and 8 bit Fault expanders. It also
2
prevents confusion as to which power supply is actually on that bus. For
example, blade A would see PSA on the PS_A I2C bus and blade B would
see PSA on the PS_B I2C bus.
IO Module 4
0B D00 4 CDx
Thunderbolt
Expander 0xC2
IO Module 4
Thunderbolt
7
Resume 0xA8
7
Resume 0xA8
XEP 5 05 8 w(s )
CMD 0x20
dsPIC33F
XEP 50 58 w(s )
CMD 0x20
0x38 0xDC
PCA9552
dsPIC33F
0xDC
24LC32A
PCA9552
0x38
24LC32A
6
6
PCA9548
Sw 0xE0
MCB 5 175 (s )
w
PCA9548
MCB 517 5 (s )
w
Sw 0xE0
SFPs 0xA0
SFPs 0xA0
MCU
5
MCU
5
0
MCB 5 17 5 (s )
w
0
MCB 517 5 (s )
w
0x90
4
0x90
4
2 3
3
24LC32A
2 PF S
1 PCA Resume
2 PF S
PCA 9557 0x32 PCA 9557 0x30 9545 1
2
Resume PROM0xAE
1 PF S
9545
1 PF S
1
PROM0xAE
1
2 2
SW
0
SW
0
IO Module 5
0B D00 4 CDx
Thunderbolt
IO Module 5
Expander 0xC2
Thunderbolt
Expander 0xC2
7
Resume 0xA8
7
Resume 0xA8
3 PCA 9546 PCA 9546 CMD PCA 9546 PCA 9546 CMD 3
XEP 5 05 8 w(s )
CMD 0x20
dsPIC33F
XEP 50 58 (s )
w
PCA9552
CMD 0x20
24LC32A
PCA9552
dsPIC33F
0x22
24LC32A
6
6
PCA9548
MCB 51 75 (s )
w
PCA9548
Sw 0xE0
MCB 51 75 (s )
w
Sw 0xE0
SFPs 0xA0
SFPs 0xA0
0 0 0 0
5
24LC32A
5
24LC32A
MCB 51 75 (s )
w MCB 51 75 (s )
w
4
Resume
4
Resume
3
0xA6
3
2 PF S
2 PF S 0xA6
2
2
1 PF S 1 PF S
Tornado A Tornado B
1
1
0
0
24LC32A
Resume Fogbow Midplane
0xAA
PS A PS B
Micro PS_A_I2C
0x52 Micro
24LC32A PS_B_I2C 0x52
Resume 24LC32A
PROM 0xAC Resume
PROM 0xAC
Wildcat-S A Wildcat-S B
Redundant FAULT_REG (Bank 0 bits 7:0) Redundant FAULT_REG (Bank 0 bits 7:0)
16 bit I2C RST_FRM_PEER_N (Bank 1 bit 5) 16 bit I2C RST_FRM_PEER_N (Bank 1 bit 5)
expander PERSIST_RST_N (Bank 1 bit 6) expander PERSIST_RST_N (Bank 1 bit 6)
Master for SAN 0x4E HLD_IN_PST_N (Bank 1 bit 4) dsPIC33F Master for SAN 0x4E HLD_IN_PST_N (Bank 1 bit 4)
PCA9673 CMD PCA9673 dsPIC33F
ICH slave Sequencer ICH slave CMD
0x88 0x88 Sequencer
7 I2C switch 0x24 7 I2C switch
master PCA9548 PECI master PCA9548 PECI 0x24
0xE4 0xE4
0 1 2 3 4 5 6
MAX6621 0 1 2 3 4 5 6
MAX6621
0x54 0x54
1 1
PCA9543 PCA9543
1 of 4 MUX SEL0 I2C switch 1 of 4 MUX SEL0 I2C switch
To 0XE6 To 0XE6
CBTLV3253 SEL1 SIO CBTLV3253 SEL1 SIO
11 01 10 00 11 01 10 00
PEX8505 (sw)
MCU
PEX8505 (sw)
PEX8505 (sw)
MCU
DB400 0xDC
DB400 0xDC
BCM5715 (sw)
BCM5715 (sw)
BCM5715 (sw)
BCM5715 (sw)
DB400 0xDC
DB400 0xDC
BCM5715 (sw)
BCM5715 (sw)
BCM5715 (sw)
BCM5715 (sw)
0x92 0x92
Clocks 0xD2, 0xD6, 0xDE Clocks 0xD2, 0xD6, 0xDE
SFP1
SFP2
SFP1
SFP2
SFP1
SFP2
SFP1
SFP2
EMC CONFIDENTIAL
J4 U21
MCU
J2
U15
I2C_ARB_A1_BUF_0_N U27
U18
ITRAC
IO Module 3
ICH I2C_ARB_A1_BUF_IN
I2C_ARB_A1
U28
MP_I2C_ARB_A1
U99
I TRAC
IO Module 2
J5
Wildcat-S SP
A
I2C_ARB_B1
ITRAC
MP_I2C_ARB_B1
IO Module 1
I2C_ARB_B1
U104
M
U141
I U144
ITRAC
SIO IO Module 0
SIO_ARB_B1
D
P
I2C_ARB_A1_BUF_0_N U27
L U18
ITRAC
IO Module 3
ICH I2C_ARB_A1_BUF_IN
A U28
N MP_I2C_ARB_A1
U99
ITRAC
E IO Module 2
J5
Wildcat-S SP
B
ITRAC
MP_I2C_ARB_B1
IO Module 1
I2C_ARB_B1
U104
U144 U141
ITRAC
SIO IO Module 0
SIO_ARB_B1
U15
J2
J4 U21
MCU
U6
EMC CONFIDENTIAL
134
NAS I2C Bus • The I2C Master on one of the two Wildcat-S Blades or one of the two Earth
Quake Management FRU’s installed in the Chassis pulls down either
Arbitration I2C_ARB_A1 or I2C_ARB_B1 signal to begin arbitrating for either
2
Flowchart PS_A_I2C or PS_B_I2C I2C bus.
• The master device then reads back the value through its feedback loop to
verify the arbitration signal is low.
• The I2C master releases the control signal and checks the value through its
feedback loop to verify the signal is high indicating its peer is not trying to
access the same I2C bus.
• I2C master pulls the arbitration signal low again and checks its value.
• The I2C master does its I2C read/write and releases the arbitration signal.
Is
Control Signal NO
High?
YES
The Control Signal will be either
Pull Control Signal I2C_ARB_A1 or I2C_ARB_B1
Low until done
depending upon which I2C bus
with I2C bus
is being arbitrated for .
Is
Control Signal NO
Low? t2
MP_I2C_ARB_(A/B)1
YES
One Control
Done, Success Signal
Host can
t(n) connect to
Wildcat-S I2C Arbitration Flow .emf I2C Bus
Midplane I2C Access Arbitration Diagram
EMC CONFIDENTIAL
• Bus 01 is connected to the U35 MCU I2C slave port. If the CPU (via the
ICH7) wants to access the MCU slave port, it must arbitrate for ownership of
the PS2_I2C bus first. This will lock out access to the MCU slave port from
its peer CPU and the 2 EarthQuake Management FRU’s in a NAS system by
holding the U64 8-port I2C switch in reset. See Figure 2-78.
• Bus 10 provides access to the PS_A_I2C bus. On this bus resides power
supply A or B depending which slot the Wildcat-S Blade is inserted. The bus
also has connectivity to the Wildcat-S midplane resume proms, a 16 bit I2C
fault expander U74 which is connected to the U141 Server IO GPIO status
bits and is used to monitor the state of the CPU Module. The CPU must
arbitrate for the PS_A_I2C bus by using the I2C_ARB_A1 control signal out
of the ICH7.
• Bus 00 provides access to the PS_B_I2C bus. On this bus resides power
supply B, the 16 bit AUX expander U73 (PCA9673) used as a redundant
Fault Register, Wildcat-S devices U23 Control Monitor Device (CMD)
which provides information on the state of the processors and on board
voltages. The U66 MAX6621 which supplies a PECI interface to the 2
CPU’s. An 8-port I2C switch U64 provides access to I2C components on the
IO Modules (0-3), and the ICH7's slave interface. The CPU must arbitrate
for the PS_B_I2C bus by using the I2C_ARB_B1 control signal out of the
Server IO chip.
EMC CONFIDENTIAL
136
I2C Interface The below illustration shows the I2C interfaces on the Wildcat-S
motherboard. There are two interfaces connected out to the midplane
Block MP_PS1_I2C & MP_PS2_I2C. They connect to all the various modules
2
installed in the Wildcat-S Chassis. There are four I2C interfaces connected
one each to the Wildcat-S Blades 4 IO Module connectors.
NOTE:
Signals ending in SMB represent two signals ending in
Blackford SDA and SCL
MCH Signals ending in I2C represent two signals ending in
V3_3 SDA and SCL
4.7k DB1200G
IOANNEX_INS* 10k
PCA9554 IDTCV126
HP_SMB_SMB FBD CLK Buffer
IO[3:0]_INS* SMBus ADR 0xDE Div/2
Expander
ADR 0x40 U79
MEM_I2C
11
14.3k
CLOCK_I2C
V3_3
V3_3 1 of 4
Mux
4.7k 01
U83 24LC32A
ICH7 Resume VCC_CMD
CBTLV3253
V3_3
SMBus 0 Ohm Prom
U18 TRUNC_I2C
Master ADR 0xA4
14.3k
14.3k
SMLink 0/1
ADR 0x88 Slave PS1_I2C CBTLV MP_PS1_I2C
10 3245
14.3k
V3_3
V3_3 V3_3
LPC2146 PCA9673
Management 14.3k I2C Primary
4.7k
Controller
Midplane
PCA9543 Expander 11
U35 S[1:0] ISO_PS2_I2C 0 Fault_REG[7:0] FROM SIO
MCU_I2C_R I2C U74
00 ADR 0x4C
HLD_IN_PST*
Master Slave Switch 1 RST_FRM_PEER*
ADR 0x92 SMB_SEL[1:0] ADR 0xE6
SIO PERSIST_RST*
PCA9673
Reset U141 I2C 11
VCC_CMD
Exp. PCA9548 Expander
ICH_SLAVE_I2C
U56 0 Aux
U73 14.3k
ADR 0x4E
IO(3-0) 1
IO3_I2C MP_PS2_I2C
PwrEn IO3 6 PS2_I2C CBTLV
I2C
U59 IO2_I2C 3245
Switch
IO(3-0) IO2 5 U69
Flt. Leds IO1_I2C
U61 IO1 4
dsPIC33F Note: All components within
IO0_I2C ADR 0xE4 CMD the dotted red line area will
IO(3-0) IO0 3
Reset-Out U23 be powered by VCC_CMD
Clovertown ADR 0x24 which is generated by a
U60 MAX6621
CPU 1 PECI linear regulator connected
IO(3-0) PECI
ISSP Header to V+12_SBY (A OR B)
PwrGd Clovertown U66
U57 CPU 0 ADR 0x54
IO(3-0)
Insert
U58
The Wildcat-S Resume PROM and the CMD (Wildcat-S) will be powered
by VCC_CMD which is powered by 12V Standby A OR B even if the
Wildcat-S Blade itself is powered off. This allows one Wildcat-S Blade to
read the Resume and CMD of its peer blade at any time, even if the peer
board fails to power up completely. The CMD controls the power enable
and power good signals to/from the DC-DC power circuits. The CMD
may have power fault and other information in its registers.
EMC CONFIDENTIAL
One port is used as a slave only port connected to the U83 MUX port '01'
and the 8 port I2C switch U69 by the CLOCK_I2C interface. The slave
only port is used to communicate with and control the MCU.
The second I2C port will be configured as a I2C master port. The master
port connects to a number of I2C expanders and will be discussed in the
section on “MCU Master I2C Port” on page 140.
Local Wildcat-S In a SAN or NAS system when the CPU wants to access the local MCU
I2C slave port, the CPU needs to arbitrate for the PS2_I2C interface. Then
CPU Access to the CPU instructs the Server IO chip to set its U83 MUX (SMB_SEL<1:0>)
MCU Slave I2C lines to branch '01' this connects the MUX to the MCU I2C slave port. A
logic decode inside the Server IO will select and hold the 8 port I2C
Port switch (0xE4) in reset. This reset isolates the MCU from the global I2C bus
in the event of a failure on that global I2C bus.
The CPU can now use the ICH7 to communicate over the CLOCK_I2C
bus to the MCU slave port and control the MCU and its master I2C port.
MEM_I2C
11
PEER
14.3k
WILDCAT-S
CLOCK_I2C SP WILDCAT-S
V3_3
V3_3 01
SP
4.7k
U83
ICH7
CBTLV3253
V3_3
0 Ohm
U18 SMBus TRUNC_I2C
Master
14.3k
SMLink 0/1
ADR 0x88 Slave PS1_I2C
10
14.3k
V3_3
V3_3
1 of 4
Mux
14.3k
ISO_PS2_I2C PCA9543
S[1:0] I2C 0
00
Switch 1
SMB_SEL[1:0] ADR 0xE6
SIO
LPC2146 PCA9673
U141 VCC_CMD
Management I2C
Midplane
ISSP Header
EMC CONFIDENTIAL
138
Peer Wildcat-S In a SAN or NAS system when the Peer Wildcat-S CPU Module wants to
access its peers’ MCU I2C slave bus it must first arbitrate for ownership of
CPU Access to the global I2C bus that the MCU I2C slave bus is connected to.
2
MCU Slave I2C
Which global I2C bus the MCU is connected to is dependent on which
Bus slot the Wildcat-S is located in. See Figure 2-72, Wildcat-S Chassis Midplane
I2C Interfaces (SAN).
In order for the Peer CPU Module to access the MCU I2C slave port via
the 8 port I2C switch U69 it must select branch 1 to connect it to the
CLOCK_I2C bus in which the MCU resides. The peer CPU can now
access the MCU slave port and control the MCU and its master I2C port.
NAS In a NAS system when one of the Earthquake Management FRU’s wants
to access the MCU Slave I2C bus it must first arbitrate for ownership of
Earthquake the global I2C bus that the MCU I2C slave bus is connected to.
Management
In order for the EarthQuake to access the MCU I2C slave port via the 8
FRU Access to port I2C switch U69 it must select branch 1 to connect it to the
MCU Slave I2C CLOCK_I2C bus in which the MCU resides. The EarthQuake
management FRU can now access the MCU slave port and control the
Bus MCU and its master I2C port.
EarthQuake
MEM_I2C NAS
11
Management
FRU-A
14.3k
WILDCAT-S
CLOCK_I2C SP
V3_3
V3_3 01
0 Ohm V3_3
U18 SMBus TRUNC_I2C
Master
14.3k
SMLink 0/1
ADR 0x88 Slave PS1_I2C
10
PEER
WILDCAT -S
14.3k
V3_3
SP
1 of 4 V3_3
Mux
14.3k
PCA9543
S[1:0] ISO_PS2_I2C 0
00 I2C
Switch 1
SMB_SEL[1:0] ADR 0xE6
SIO
LPC2146 PCA9673
Management U141 I2C VCC_CMD
Midplane
EMC CONFIDENTIAL
The LPC2146 Management Controller (MC) U35 has one I2C master port.
The master port connects to a number of I2C expanders which are briefly
described below. The I2C Master port connections are illustrated in
Figure 2-82.
EMC CONFIDENTIAL
140
MCU I2C Master The MCU is
Interface
I2C Addr:
0x34 1 IO0_FLT_N
IO1_FLT_N
IO2_FLT_N
2
PCA9557
IO3_FLT_N
5 IO_ANNEX_FLT_N
IO_ICH_PCI_RST_N
I2C Addr:
0xE4 1 IO0_PWRGD
IO1_PWRGD
PCA9538 IO2_PWRGD
PWRGD_EXP_INT_N IO3_PWRGD
5 IO_ANNEX_PWRGD
EXP_RST_N
Fault Registers
EMC CONFIDENTIAL
Blackford also has an I2C bus (referred to as the Hot Plug SMBus) that
can be used on Wildcat-S to monitor the insertion or removal of IO
Modules and the IO Annex.
Blackford
V3_3
MCH
V3_3
4.7k
IOANNEX_INS*
PCA9554
SMBus HP_SMB_SMB
From IO (3-0) & IO IO[3:0]_INS*
Expander
Annex Connectors
ADR 0x40 U79
EMC CONFIDENTIAL
142
I2C Reset, The below diagram shows the mapping of the Wildcat-S Blade I2C Reset,
Arbitration and Attention signals. These signals go out onto the midplane
Arbitration & and connect to the two Management FRU’s, PSA, PSB, IO Annex and the
2
Attention Nova Test Board Slot.
Signals
V+3.3
IO Conn
1k IO Conn
IO
IO Conn
IO_I2C_BUFRST Conn
IO_I2C_RST_N
V+3.3
EN_CBT_N
EN 1.5k
GPIO56 I2C_ARB_B1_BUF_IN CBT I2C_ARB_B1
GPIO30
PS1_I2C_ATN_BUF_N PS1_I2C_ATTN_N
GPIO13
PS1_I2C_BUFRST PS1_I2C_RST_N
GPIO22
I2C
I2C_EXP_RST_N EXP
0x34
V+3.3
I2C
14.3k I2C_AUX_RST_N EXP
0x36
CMD
I2C IO
I2C_MUX_RST_N Switch
0xE4
PS2_I2C_ATTN_N
GPIO11
SIO
V+3.3
SIO_ARB_A1
GPIO52
1.5k
I2C_ARB_A1
GPIO51
MP_I2C_ARB_B1
I2C_PWRON_RST_N
MP_PS1_ATTN_N
V+3.3 Isolation
M
Switch A MP_PS1_I2C_RST_N
1k I
PECI I2C_CBT_EN_N D
EN
P
L
MP_PS2_ATTN_N A
N
MCU V+3.3 MP_PS2_I2C_RST_N E
Isolation
V+3.3 Switch B
MP_I2C_ARB_A1
8.2k
SET EN_CBT
4.7k D Q EN
EN_CBT_CLK
EN_CBT_N SIO COM
CLR Q CBT
062-000-931
(062-000-892)
Reset Signals The reset signals will follow a similar path as the attention signals above
and reset the MCU’s and some I2C buffers on various modules. It will
also reset the expander chips and I2C devices on the IO Modules.
Arbitration The arbitration signals were described earlier in this chapter for both a
SAN and NAS type system.
Signals
EMC CONFIDENTIAL
EMC CONFIDENTIAL
144
Chapter 3 MANAGEMENT FRUS
Chapter contents:
•
•
•
Introduction................................................88
Solar Flare...................................................89
Earthquake...............................................110 3
EMC CONFIDENTIAL
145
3.1 Introduction
There are two types of management FRU’s used in the Wildcat-S chassis
which are used in the Dreadnought systems.
• Earthquake (NAS)
The Management FRU is used to monitor and control the chassis. RS-232
communications for the console and SPS is another important feature of
the Management FRU. It provides a path between the airdam and the
Wildcat-S CPU module. Management card redundancy is achieved by
ensuring that each chassis always contains two management FRU’s -
should one fail, the other can take over the management duties of the
entire enclosure.
EMC CONFIDENTIAL
Introduction
The Solar Flare SAN management card provides various management
functionalities to the Wildcat-S chassis running in a SAN configuration.
There are two Management Modules per chassis.
3
Each Solar Flare interfaces with a Wildcat-S blade, power supplies and IO
modules via the system midplane (Fogbow). Communication between
the two management modules is supported by an Ethernet Crosslink
which provides service personnel the capability to plug into either
management module’s service jack and have access to both Wildcat-S
blades.
Power OK LED
Fault LED
USB 2.0 port
Link LED
Activity LED
Service
LAN
EMC CONFIDENTIAL
LED’s
USB Micro DB-9 RJ-45
Power Fault
Resume CMD
Prom Module
Peer MGMT.
Crosslink-
I2C
Blade A
Blade B
Isolation
Switch
Midplane Connector
EMC CONFIDENTIAL
The Ethernet switch allows for the connection of two blades within a
Wildcat-S chassis to two Mag Jack Ethernet connectors. One of the
connectors is utilized for the management connection while the second is
3
for service use in a traditional SAN configuration.
In figure 3-51 below, the Ethernet switch and it’s surrounding logic is
shown.
BCM5397 Switch
TRD0_0+/-
Ethernet Crosslink TRD1_0+/-
TRD2_0+/- PORT 0 TRD0_3+/-
TRD3_0+/- TRD1_3+/- TDO-3 +/-
PORT 3 RJ45
TRD2_3+/-
TRD0_1+/- TRD3_3+/-
MidPlane SPA TRD1_1+/-
TRD2_1+/-
PORT 1 TRD0_4+/-
TRD3_1+/- TRD1_4+/- TDO-3 +/-
PORT 4 TRD2_4+/- RJ45
TRD0_2+/-
SPB TRD3_4+/-
TRD1_2+/-
TRD2_2+/-
PORT 2
TRD3_2+/-
JTAG
Philips
I2C
LPC2131
P0_4
SCK
SCL P0_5 MISO
SDA P0_6 MOSI SPI
P0_7
Bus SS
JTAG
Switch
POWER_GOOD
RESET CMD
TRST
TMS
TCK JTAG
TDI
TDO
EMC CONFIDENTIAL
10/100/1000
TRD[3:0]_0+/- GMAC Register
PHY Space
10/100/1000
TRD[3:0]_1+/- GMAC
PHY
Packet Buffer
10/100/1000
TRD[3:0]_2+/- GMAC
PHY
10/100/1000 Address
TRD[3:0]_3+/- GMAC Management
PHY
10/100/1000
TRD[3:0]_4+/- GMAC LED
PHY LED
Interface
GMII / RvMII / RGMII GMAC
EEPROM
EEPROM
MII MAC Interface
Switch
Configuration
The Broadcom 5397 Ethernet switch connects the three gigabit Ethernet
ports coming from the midplane and the two external ports. The switch is
able to partition the ports into separate networks, creating the concept of
the management and the service LAN. The service LAN is comprised of
the external service port, the service port coming from the midplane
(which is connected to the MAC on the peer's SP), and the Ethernet
crosslink between the two management boards. The second LAN
connects the external management port to the local SP's management
MAC.
By default the Micro controller will program the Switch for a SAN
configuration. Wildcat-S blades will use POST or FLARE to program the
micro controller via I2C for all other configurations. The micro controller
will use an SPI interface to configure the switch.
EMC CONFIDENTIAL
Ethernet Crosslink
3
GigE Switch GigE Switch
Management A Management B
Crosslink
SP A SP B
EMC CONFIDENTIAL
LPC2131 Microcontroller
EMC CONFIDENTIAL
EMC CONFIDENTIAL
SP A SP B
RX TX TX RX
I2C I2C
MIDPLANE
RX TX TX RX
There is another serial port shared between the A and B Solar Flare micro
controllers. This link could be used so that the blade that is up only needs
to tell one Solar Flare to flip its MUX and the micro controller can inform
the micro controller on the peer Solar Flare to flip MUX the serial stream
over.
EMC CONFIDENTIAL
3
I2C Bus
The figure below shows the I2C bus topology on Solar Flare with the
devices' I2C addresses. There are two system I2C busses, each
management module connects to one of those busses depending on the
slot the management module resides in.
Resume
PROM
(0xAE) I2C
CMD
M Sequencer
Switch S0
I
SCL D (0xF2)
SDA S1
P
RST
(SlotA =0xEA)
L
A (SlotB =0xE8) 4.7K
(0x90)
N
E 3.3V
Philips MCU
I2C Isolation The Philips PCA9546 allows the Master to disconnect the microcontroller
from the bus in the case that the microcontroller's I2C bus has become
stuck. The microcontroller also has the ability to disconnect itself from the
bus. If the microcontroller does not detect a valid slot ID code, it will
disconnect itself from the bus to prevent possible addressing conflicts
with the other management unit. This is necessary because one of the
switch address pins is slot dependent. If the slot ID is stuck one way or
the other it can create an addressing problem in the design.
EMC CONFIDENTIAL
Power Sequencing
& Monitoring
The CMD module provides sequencing, margining and other power
management functions to the system. This device uses PMBus, based on
SMBus, to communicate status of the power subsystem. Using the CMD
provides an easy and consistent method to control the DC/DC converters
on the board. Other benefits of the CMD are: provides both power good
and fault signals to the system, margining capability to both 3% and 5%
without modifying resistors and DC regulation can be maintained at
better than 1%.
Broadcom recommends that both the 2.5 and 1.0V rails come up
simultaneously, but to err on the side of caution, the higher voltage comes
up after the 1.0V is up. Power is sequenced according to the following
figure.
All other voltages do not affect the Broadcom chip and do not have a
sequencing requirement.
The Solar Flare board monitors the 5.0, 3.3, 2.5 and 1.0 voltages using the
Command Module. The module only asserts POWER_GOOD when all
voltages are within specification.
EMC CONFIDENTIAL
EMC CONFIDENTIAL
BCM5397
EMC CONFIDENTIAL
EMC CONFIDENTIAL
EMC CONFIDENTIAL
EMC CONFIDENTIAL
EMC CONFIDENTIAL
EMC CONFIDENTIAL
EMC CONFIDENTIAL
EMC CONFIDENTIAL
EMC CONFIDENTIAL
EMC CONFIDENTIAL
Introduction
The Earthquake NAS management card provides various management
functionalities to a Dreadnought enclosure running in a NAS
configuration. There are two Management Modules per enclosure.
3
Each Earthquake interfaces with a Wildcat Storage Processor (SP), power
supplies and IO modules via the system midplane (Fogbow).
Communication between the two management modules is supported by
two serial paths between the two modules.
EMC CONFIDENTIAL
Riser Card
Pwr/ NMI
Micro RJ45 RJ45 RJ45 USB
Flt 7 Seg Button
Debug DB9 Service Mgmt Uplink Service
LED’s
EMU Conn
4 3 0
USB 2.0
MII
HUB
Motorola Coldfire SPI BCM5397
Service/
(MCF5282) 1Gb Enet Switch 2
Debug JTAG Chain
Muxing 1
I2C
SDA/SCL
SPB
Arb
Arb
Debug
Midplane Connector
EMC CONFIDENTIAL
170
Figure 3-51 shows the block diagram for the Earthquake NAS
management FRU. The major components on this board are the Freescale
MCF5282 Coldfire Micro controller, the Broadcom BCM5397 Gigabit
Ethernet Switch, the CMD Power Sequencer and the USB2.0 Hub. Also,
due to the lack of PCB real-estate on the front panel, a riser card (Tremor)
is used to bring some of the features to the front panel. The DB9, LED's
and Seven Segment Display reside on the Tremor which mates to
Earthquake through a connector.
Broadcom
BCM5397
Gigabit
The Broadcom BCM 5397 10/100/1000 BASE-TX is a six-port Ethernet
Ethernet Switch Switch. The device has five integrated 10/100/1000 transceivers while the
sixth port is provided with an industry standard MII interface. Each of
the five gigabit Ethernet ports are comprised of 4 differential pairs, or 8
wires each. These pairs carry a 125MHz signal utilizing PAM-5 encoding
to transfer the data. The switch has an SPI interface for diagnostics,
configuration, and switch management.
EMC CONFIDENTIAL
The figure below is a high-level block diagram of the Ethernet switch and
its surrounding logic.
M
A
SPA G Port 1
N
Midplane E
Port 0
RJ45
T
SPB I Port 2 Uplink
C
S
I2C
Port 3
RJ45
Mgmt
JTAG
MII
Switch BCM5397
1Gb Enet Switch Port 4
RJ45
SPI
I2C Service
244 Coldfire
Buffer MCF5282 ~RST
TDI
TDO TDI
JTAG
JTAG
TDO
EMC CONFIDENTIAL
172
BCM5397
10/100/1000
TRD[3:0]_0+/- GMAC Register
PHY Space
10/100/1000
TRD[3:0]_1+/- GMAC
PHY
Packet Buffer
10/100/1000
TRD[3:0]_2+/- GMAC
PHY
10/100/1000 Address
TRD[3:0]_3+/- GMAC Management
PHY
3
10/100/1000
TRD[3:0]_4+/- GMAC LED
PHY LED
Interface
GMII / RvMII / RGMII GMAC
EEPROM
EEPROM
MII MAC Interface
The block diagram of the BCM5397 shows the internal logic of the switch.
Switch
Configuration
The Broadcom 5397 Ethernet switch connects the two gigabit Ethernet
ports coming from the midplane and the three external ports. The switch
is able to partition the ports into separate networks, creating the concept
of the management and the service LAN. The service LAN is comprised
of the external service port and the service port coming from the
midplane (which is connected to the MAC on the peer's SP). The second
LAN connects the external management port to the local SP's
management MAC.
By default the Micro controller will program the Switch for a NAS
configuration. POST or FLARE will need to program the micro controller
via I2C for all other configurations. The micro controller will use an SPI
interface to configure the switch. The SPI interface is a simple four-wire
bus, which allows access to the internal registers within the Broadcom
switch. Upon power up the Coldfire processor will be required to do
some basic configuration and run diagnostics on the switch over its SPI
interface. The Forward Enable feature should be active within the
Broadcom switch so traffic can move thru the device without dependency
on the Coldfire for configuration if there is an issue with a device on
board.
EMC CONFIDENTIAL
- 64KB SRAM
- 3 UARTs
- SPI Interface
- I2C Controller
The following figure depicts the block diagram of the Coldfire processor.
EMC CONFIDENTIAL
174
Serial UART Earthquake is connected to both of the local SP's UART's, but has only
one external micro db9 port located on the riser (Tremor) card. Thus,
MUX several MUX's are used to select which serial port is connected to the
external connector. In addition, the Coldfire has the ability to attach the
console serial port to itself, and direct the debug port to the external
connector.
3
UARTS UARTS
DBG DBG
CNSL CNSL
RX TX RX TX TX RX TX RX
MIDPLANE
1 1
2
Coldfire Coldfire
MUX
MUX
0
0
RX RX
3 3
TX
2 2 TX
2 2
1
1
MUX
MUX
MUX
MUX
MUX
MUX
0
1
1
RX TX TX RX
The MUX Select lines default to port 0 with pull downs. This connects the
SP's console UART to the external micro db9 connector. The Coldfire
debug UART is not externally accessible but it is located on the Tremor
riser card, that will give access to this for lab bring up and debug. The
Coldfire has the ability to change the select lines on the MUX to connect
the debug UART to the front panel.
There are two serial links between the Earthquake boards in a chassis.
The second link can also be used to access the Coldfire debug UART.
When a cable is inserted into a debug header on the Tremor riser card, it
automatically flip's the MUX and connects the second UART to this
header.
EMC CONFIDENTIAL
V3_3 V3_3
4.7K I2C_ARB_A1_O 1.5K C_I2C_ARB_A1 CBTLV PS_1_I2C_ARB
3384
C_I2C _ARB_A1 V3_3
MC _CBT_EN 1
Midpla ne
4.7K GND
MC _CBT_EN 2 GND
4.7K
V3_3 V3_3 EN
GND
4.7K I2C_ARB_B1_O 1.5K C _I2C_ARB_B1 PS_2_I2C_ARB
Coldfire
C_I2C_ARB_B1 V3_3
C_PS_2_I2C _ATN_N 4.7K C_PS_2_I2C_ATN_N PS_2_I2C_ ATTN_N
CBTLV
3384
DB G Header
V+3_3
ZERO
ADR 0x 90 ZERO
PCA 95 xx ZERO 14.3K
SW_I2C I2 C PS_1_I2C
Switch 0
2
1 V+3_3
ADR 0 xEC , 0xE 8
Resume
PCA 95 xx 14.3K
ADR 0 xAE
I2 C PS_2_I2C
Switch 0
1
ADR 0 xEE , 0 xEA
CMD
ZERO
ADR 0xF 2
ISSP Header
EMC CONFIDENTIAL
176
I2C Isolation The Coldfire allows the Master to disconnect the micro controller from
the bus in the case that the micro controller’s I2C bus has become stuck.
The micro controller also has the ability to disconnect itself from the bus.
If the micro controller does not detect a valid slot ID code, it will remain
disconnected from the bus to prevent possible addressing conflicts with
the other management unit. This is necessary because one of the switch
address pins is slot dependent. If the slot ID is stuck one way or the other
it can create an addressing problem in the design.
3
Figure 3-117. Slot Dependent I2C Addresses
Enclosure
Identification The Earthquake board will display the enclosure ID from the Coldfire
CPU with a value of 1 to 8 respectively. Software is responsible for
creating and maintaining each value. A seven segment display is
mounted to a riser card (Tremor) plugged into Earthquake to display the
enclosure information.
EMC CONFIDENTIAL
POWER _ A _SENSE
Coldfire
POWER _ B _SENSE MCF5282
Power
7.15K 7.15K
To Voltage
Plane / Islands
3.3V
V + 12 _ A Soft V + 12 _ SS _ A V + 12 _ ORED DC-DC V + 3 .3
Start Converter
2.5V
V + 12 _ B Soft V + 12 _ SS _ B DC-DC V +2.5
Start Converter
1.0V
V+ 1
DC-DC
3.3V
3.0Vdc Voltage Converter
DC-DC
M id pl a n e
Reference Divider
Converter
V + 3 .3 _CMD 5.0V V +5
DC-DC
Converter
4.7K Analog Monitor X Warning
Control
1K Monitoring
Device
(CMD)
Sequencer / Monitor
Power Sequencing
& Monitoring
The CMD module provides sequencing, margining and other power
management functions to the system. This device uses PMBus, based on
SMBus, to communicate status of the power subsystem. Using the CMD
provides an easy and consistent method to control the DC/DC converters
on the board. Other benefits of the CMD are: provides both power good
and fault signals to the system, margining capability to both 3% and 5%
without modifying resistors and DC regulation can be maintained at
better than 1%.
178
Broadcom recommends that both the 2.5 and 1.0V rails come up
simultaneously, but to err on the side of caution, the higher voltage comes
up after the 1.0 is up. Power is sequenced according to the following
figure.
3
All other voltages do not affect the Broadcom chip and do not have a
sequencing requirement.
The Earthquake board monitors the 5.0, 3.3, 2.5 and 1.0 voltages using the
Command Module. The module only asserts POWER_GOOD when all
voltages are within specification. This signal is tied to the reset scheme
into the on board micro controller (Coldfire).
EMC CONFIDENTIAL
V+3_3
V3_3 4.7K
4.7K
C_PS_1_I2C_RST_N CBTLV PS_1_I2C_RST_N
3384
V3_3
PCA9557
1K
V3_3 USB 0 V3_3 EN
4.7K HUB 1 1K
2
4.7K V3_3
100NF
I2C Exp GND 1K GND
GND 3
4 4.7K
V3_3 5 GND
1K
6
EXP_RST
7 V3_3
ADR 0x32, 0x30 1K
Midplane
4.7K
GND
MC_CBT_EN2 GND
4.7K 4.7K
V3_3
GND
EN
4.7K C_PS_2_I2C_RST_N PS_2_I2C_RST_N
Coldfire
MC_NMI_ASSERT CBTLV
V3_3 V3_3
3384
4.7K MC_BCM_RST_N MC_NMI_N 4.7K V+3_3
I2C_SW_RST1
I2C_SW_RST2
PCA95xx 14.3K
I2C PS_1_I2C
V3_3
ZERO Switch 0
1K
V3_3 1 V+3_3
ADR 0xEC, 0xE8
MC_RST_O_N 4.7K MAX6816
V3_3
NMI_SWITCH
Debounce
MC_RST_I_N 1K
PCA95xx
EXT_WD
ZERO
14.3K
V3_3 I2C ZERO PS_2_I2C
V3_3
WatchDog 4.7K Switch 0
1K
Timer 1
V3_3
CBT
4.7K NMI_N
V3_3 ADR 0xEE, 0xEA 4.7K CBT_NMI_EN_N
GND EN 1NF
4.7K
Emulator PWR_LED_XTOR_C GND
V3_3 1K
GND CMD
GND
BCM_RST_N 1K GigE
CMD_POWER_GOOD
ZERO
ADR 0xF2
Switch
4.7K
GND ISSP Header
MC_RESET_N
A reset of the Coldfire will result in the entire board resetting. The USB
device is also reset controlled from the Coldfire and is configured thru a
dedicated EEPROM.
EMC CONFIDENTIAL
180
Power and Fault The Earthquake board contains a fault LED and a power LED located on
the riser (Tremor) card. The fault LED is yellow and will be controlled by
LEDs both a remote board signal and the Coldfire micro controller. The micro
controller will assert the fault LED under certain unrecoverable situations
such as switch failure.
The power LED is green and is connected to the 3.3V voltage source. On
power-up, the CMD device turns on the LED when all voltages are up
and within specification.
The management module also has the ability to assert the LED's that
reside on the Midplane and the Local/Peer Blades. The diagram below
3
illustrates the connectivity thru the midplane.
SP A MIDPLANE IO ANNEX B
ICH_SW_FLT
I 1k 1k
C
H ICH_SYS _FLT_N
7
V+3.3
MGMT B
10 k
IO_ANNEX_FLT MC_FAULT_LED
MCU ZERO
Fault 1k 1k
10K
EXP
Trem or
BLADE_FLT R iser
SIO V+3.3
1k 1k 56
CMD_POWER _GOOD
CMD
1k
4.7k
C 2.2 k 2 .2k
O
L
MC_BLD_FLT _LED _B MC_BLD_FLT _LED _B
D
F
I 2.2 k 2 .2k
R
E MC_ANX_FLT_LED _A MC_ANX_FLT_LED _A
2.2 k 2 .2k
MC_ANX_FLT_LED _B MC_ANX_FLT_LED _B
2.2 k 2. 2k
V+3.3 SP B
10 k
MC_FAULT_LED ICH_SW _FLT
ZERO I
1k 1k C
ICH_SYS _FLT_N
H
7
Trem or
R iser
V+3.3
IO_ANNEX_FLT
56 MCU
CMD_POWER _GOOD Fault
CMD
10 k EXP
1k
4.7k
BLADE_FLT
SIO
1k 1k 1k 1k
IO ANNEX A
EMC CONFIDENTIAL
JTAG_TDO MCU_TDO
244
JTAG_ENABLE_N EN
M
V3_3
I
D 4.7K JTAG_EN
DePOP
V3_3
P
L 1K ZERO
4.7K
A
GND
N
E EN
JTAG_TRST_N BCM_TRST_N
JTAG_TCK BCM_TCK
244 BCM
JTAG_TMS BCM_TMS
JTAG_TDI BCM_TDI TDO
TDI
1K BCM_TDO
1K
GND
GND EN JTAG_EN
MCU_TRST_N
MCU_TCK
244 ColdFire
MCU_TMS
ZERO
V3_3
4.7K
Emul
EMC CONFIDENTIAL
182
Signal Lists
Motorola Coldfire
MCF5282
EMC CONFIDENTIAL
EMC CONFIDENTIAL
184
Motorola Coldfire
MCF5282 (cont.)
EMC CONFIDENTIAL
EMC CONFIDENTIAL
186
Motorola Coldfire
MCF5282 (cont.)
EMC CONFIDENTIAL
EMC CONFIDENTIAL
188
Tremor Riser
Card_RS-232
Buffer
EMC CONFIDENTIAL
190
Chapter 4 CHASSIS POWER
EMC CONFIDENTIAL
191
4.1 Wildcat-S Chassis Power
The Wildcat-S Chassis receives power from two power supplies (PS-A &
PS-B). Both power supplies plug into the chassis midplane. Figure 4-130
shows the rear view of the Wildcat-S Chassis with the two power supplies
inserted.
PS-A
PS-B
EMC CONFIDENTIAL
Power Supply - A
V+12_SBY_A
Blower Fan
ITRAC
IO Module 3
ITRAC
IO Module 2
Wildcat-S
J5 Storage Processor
A
ITRAC
IO Module 1
Blower Fan
ITRAC
4
IO Module 0
M
I
D J21 IO A
Nova P
Test J8 IO ANNEX
Card L
A J7 IO B
Blower Fan
N
E
ITRAC
IO Module 3
ITRAC
IO Module 2
Wildcat-S
Blower Fan J6 Storage Processor
B
ITRAC
IO Module 1
V+12_SBY_B
ITRAC
IO Module 0
Power Supply - B
M CR52_16S
I
D
V+12_CMD U24
VCC_CMD
P VCC
LT1763 3.3volts
L
A
N CR52_17S
E
U100
V+12_SBYB_FUSE RESUME
1A
F2
VREF_3P0
To PS1 & PS2 I2C U25
V+12_SBYB SCL/SDA LINES VREF_3P0_RET
Power up the The 2 management FRU’s receives standby power from the two power
supplies. This enables the two FRU’s to have communication with the
Wildcat-S outside world through LAN ethernet ports. In order to turn on the 12v
Chassis main power to the two Wildcat-S blades and the rest of the system the
user needs to send a power on command through the LAN port which is
addressed in such a way to be sent over the SPI interface to the (MC)
Micro controller in the management FRU. That MC sends out an I2C
command to the two power supplies (PS-A & PS-B) to enable their 12v
main power to the Wildcat-S Chassis.
EMC CONFIDENTIAL
Power Supply - A
ITRAC
IO Module 3
ITRAC
IO Module 2
Wildcat-S
J5 Storage Processor
A
ITRAC
IO Module 1
Blower Fan
4
ITRAC
IO Module 0
M
I
D J21 IO A
Nova P
Test J8 IO ANNEX
Card L
A J7 IO B
Blower Fan
N
E
ITRAC
IO Module 3
ITRAC
IO Module 2
Wildcat-S
Blower Fan J6 Storage Processor
V+12_BLADEB_IOB
B
ITRAC
IO Module 1
ITRAC
IO Module 0
Power Supply - B
The output of the DC-DC, VRMS and other voltages are fed back to the
(CMD) Control & Monitoring Device and are measured to ensure that
they stay within specifications.
V+12_POL_MEAS
VCC15_MEAS
V12 F1 V12_G CM1
V1_5
M 1.5V
F5 VCC09_MEAS
I V12_B U138
D 0.9V V0_9
P
L V12_B CM2 VCC18_MEAS
A 1.8V
V1_8
N VCC1P05_MEAS
E U29
1.05V
V1_05
F4
V12_R VCC33_MEAS
U1
3.3V V3_3
U23
VCC12_MEAS (CMD)
V12_R CM3
VTT_CPU SEQUENCER /
1.2V
MONITOR
VCC10_MEAS
V12_R U9
1.0V V1
V12 VCC50_MEAS
V12 U32
J50_1S V12_IO0 F6 5V USB V5_USB
IO Card 0
U32
5V USB
U138 EN_V1_8
0.9V EN18
FET
CIRCUIT
CM2 EN_V1_8
1.8V
4
(CMD)
U29 EN_V1_05 FET EN1P05 SEQUENCER /
1.05V CIRCUIT
MONITOR
IO_ANNEX_PWREN CM3 EN_V1_2 FET EN12
J22 CIRCUIT
Midplane 1.2V
IO0_PWREN J3 VRM_OUTEN_A
J50_1S VRM-A
IO Card 0 For CPU 0
IO1_PWREN U J4
J51_1S VRM_OUTEN_B
IO Card 1 6 VRM-B
0 For CPU 1
J52_1S IO2_PWREN
IO Card 2 MCU_SDA
U35 WILDCAT_PWRGD
IO3_PWREN MCU_SCL MCU
J53_1S
IO Card 3
EMC CONFIDENTIAL
EMC CONFIDENTIAL
VTT_CPU VTT_CPU
U131 /U128
VID_MUX_SEL
CPU_VID<6..0>
U23
(CMD)
SEQUENCER /
4
MONITOR
EMC CONFIDENTIAL
MARGIN_HI_N
U23
SERVER MFG_MODE_N
(CMD)
IO
MARGIN_LO_N
M
I
MP_MARGIN_HI_N
D
P MP_MFG_MODE_N
L
MP_MARGIN_LO_N
A
N
E
Figure 4-138. Manufacturing Mode Signals used for Margining
The margin signals are also sent out to the IO Modules (SLICS) to
margining the DC-DC power circuits on each module.
EMC CONFIDENTIAL
U9
1.0V
TRIM+_V1_0 RC
CIRCUIT
MGN_10
4
BUF_MARGIN_HI_N
BUF_MARGIN_LO_N
BUF1_MARGIN_HI_N BUF_MFG_MODE_N
J50_1S BUF1_MARGIN_LO_N
IO Card 0
BUF1_MFG_MODE_N
BUF1_MARGIN_HI_N
J51_1S BUF1_MARGIN_LO_N
IO Card 1 BUF1_MFG_MODE_N
U97
BUF1_MARGIN_HI_N
J52_1S BUF1_MARGIN_LO_N MARGIN_HI_N
IO Card 2 BUF1_MFG_MODE_N MARGIN_LO_N U41
MFG_MODE_N
SIO
BUF1_MARGIN_HI_N
J53_1S BUF1_MARGIN_LO_N
IO Card 3
BUF1_MFG_MODE_N
EMC CONFIDENTIAL
V+12_POL_MEAS
VCC50_MEAS
EN33
VCC33_MEAS
EN18
VCC18_MEAS
EN15
DC-DC & VRM VCC15_MEAS DC-DC & VRM
EN12
Measurement VCC12_MEAS Output Enables
EN1P05
VCC1P05_MEAS
EN10
VCC10_MEAS
VCC09_MEAS
VRM_OUTEN_A
VCPU_A_MEAS VRM Output
VRM_OUTEN_B Enables
VCPU_B_MEAS
BUF_MFG_MODE_N
ICD_SDA Margin Signals
To Depop. debug BUF_MARGIN_HI_N
Connector ICD_SCL From SIO
BUF_MARGIN_LO_N
MP_PS2_SDA
U23
Midplane I2C BUS (CMD) MGN_33
MP_PS2_SCL
SEQUENCER / MGN_18
MONITOR MGN_15
Margin Signals
OVP_TRIP MGN_12
To Blade_Size FET to DC-DC circuits
MGN_1P05
FSB0_VRHOT_R_N MGN_10
From CPU VRM’s
FSB1_VRHOT_R_N
VID_MUX_SEL
BOARD_SEATED
VCC_CMD
From DEPOP. ICD_MCLR FBD_FAN_TACH
VCC_CMD
Debug Conn.
VREF_3P0
U25 Standby
VREF_3P0_RET VCC_CMD = 3.3v VCC_CMD is derived
Power Regulator VCC from 12V Standby
EMC CONFIDENTIAL
Chapter Content:
EMC CONFIDENTIAL
203
5.1 Tornado IO Annex Extender Card
Introduction There are two IO Annex slots in the Dreadnought chassis, IO Annex A
and IO Annex B. Each slot can house one IO Annex card known as the.
Tornado Board and each Tornado board can be populated with two IO
modules (SLIC’s.
4 IO Module Slots
DAERear.emf
2 IO Annex Slots
BLADE A BLADE B
MCH MCH
PLX PLX
4 Lane PCI-
Express
Foxbow
Midplane
IO ANNEX A IO ANNEX B
The other main interface on the Tornado board is the I2C bus which is
used for MCU, Resume and fan control.
EMC CONFIDENTIAL
LED
FAN A FAN B
Tornado Block
Figure 5-143.
multi-fan fault Regulator 1 I2C 1:4
Enable Switch
2 I2C
3
RS232
M idplane
Connector
RS232
I2C Address
OTHER CARDS
0xB0, 1011 000x PM C
X4 Lanes
EMC CONFIDENTIAL
X4 Lanes SATA
Buffer
I2C Address
100 MHZ 0xB2, 1011 001x
Local or PM C
X4 Lanes
X4 Lanes SATA
33MHz USB
Clock
Power Enable
V+12
PM8380 There are two PMC PM8380 Quad SATA/SAS device are used on the each
IO Annex tornado board. The function of each PM8380 is to buffer the
PCI-Express signals from the blade to the SLIC’s and from the SLIC’s to
the blade.
5
Blade A
Blade PCIe 0-7 IO PCIe 0-7
IO Blade IO Blade
PCIe 0-3 PCIe 0-3 PCIe 4-7 PCIe 4-7
IO0_PMC_RST_N TXA0-3+/- RXA0-3+/- TXA0-3+/- RXA0-3+/- IO1_PMC_RST_N
27.7MHZ PMC0_CLK PMC1_CLK 27.7MHZ
MCU_MSTR_SCL MCU_MSTR_SCL
MCU_MSTR_SDA MCU_MSTR_SDA
RXA0-3+/- TXA0-3+/- RXA0-3+/- TXA0-3+/-
PMCTxRxBuffer.emf
IO Module 0 IO Module 1
Each PM8380, one channel receives 4 lines of serial data from the blade
which it buffers and then outputted to the attached SLIC. On the second
channel, 4 transmit lines of serial data are received from the SLIC,
buffered and transmitted to the blade.The PM8380s are configured using
the I2C bus.
EMC CONFIDENTIAL
PMC0_TXAPRE<1:0>
IO 0_PMC_RST_N B2 99
PMC0_TXBPRE<1:0>
27.7MHZ PMC0_CLK B1 97 PMC0_RXAEQ<1:0>
PMC PMC0_RXBEQ<1:0>
95
MCU_MSTR_SCL J1
I2C from PMC0_RXSWIZ
MCU_MSTR_SDA J2 100
MCU PMC0_SASLVL
94
PDN_PMC0_SEL J1 RBIAS_B0
Pulled up/ 101
down PDN_PMC0_ADR<2:1> J2
PMC_Pinout.emf
Signal
Descriptions
Table 5-20. PM8380 Pin Descriptions
SCHEMATIC SIGNAL I/O Description
PCI Express Interface
EXP_FRM_BLD<3:0>+/- I PCI Express receive serial differential data from the blade.
EXP_TO_BLD<3:0>+/- O PCI Express transmit serial differential data to the blade.
EXP_FRM_IO0/1_<3:0>+/- I PCI Express receive serial differential data from the SLIC.
EXP_TO_IO0/1_<3:0>+/- O PCI Express transmit serial differential data to the blade.
Configuration, Reset and Clocks Signals
IO0/1_PMC_RST_N* I When this reset signal is asserted (set low) all the digital logic is reset and all
registers return to their default values. When this signal is not asserted the device is
operational.
PMC0_CLK I The System clock input accepts a free running clock with a nominal frequency of
37.5 MHz.
PMC0/1_RXAEQ<1:0> I The Receive equalization weight selection configures the amount of equalization on
the RXA+/-<3:0> serial interface The receive equalization may also be over-ridden
using the TWI Interface
PMC0/1_RXBEQ<1:0> I The Receive equalization weight selection configures the amount of equalization on
the RXB+/-<3:0> serial interface The receive equalization may also be over-ridden
using the TWI Interface
PMC0/1_TXAPRE<1:0> I The transmit pre-emphasis weight selection configures the amount of pre-emphasis
on the TXA+/-<3:0> serial interface.The pre-emphasis may also be over-ridden
using the TWI Interface.
PMC0/1_TXBPRE<1:0> I The transmit pre-emphasis weight selection configures the amount of pre-emphasis
on the TXB+/-<3:0> serial interface.The pre-emphasis may also be over-ridden
using the TWI Interface.
EMC CONFIDENTIAL
RBIAS_B0 I Analogue Bias Reference RBIAS. A 20 Kohm +/-1% resister must be connected
between RBIAS and adjacent VSS ball as close to the device as possible. The
external reference resistor pin creates a calibrated current for controlling the
high-speed serial interface thresholds, amplitudes and termination networks.
I2C Interface
MCU_MSTR_SCL I This signal is part of the devices 2-wire serial control interface and is used to clock
the data transfer to and from the registers.
MCU_MSTR_SDA I/O This signal is part of the 2-wire serial control interface and operates as the
bi-directional serial data port for TWI transfers.
LPC2131 The Tornado IO Annex board uses a Micro controller (MCU) to provide
additional functionality and diagnostic capability to the system. It
Tornado Micro Provides several monitoring functions for both the Tornado and any SLIC
controller
5
inserted into the Tornado. These functions include:
• Voltage margining.
Local margining The MCU will be able to margin the board high and low. Through the I2C
bus, the master will be able to put the MCU into manufacturing mode by
writing to a register. Once in manufacturing mode the MCU can be told to
margin the system high or low by writing to the manufacturing mode
register. The MCU will ensure that the system is nor margined high or
low simultaneously.
Led Control The MCU will have control of the fault LED on the front of the tornado
card. If the MCU fails to initialize, the fault LED will stay on
approximately 1 second before the watchdog trips shutting off the board.
The MCU will assert the fault LED when it encounters a fault. The fault
will be kept in a register that can be accessed via the I2C bus.
The MCU will also have control of the SLIC’s fault LEDS either
autonomously or through the I2C bus.
EMC CONFIDENTIAL
Fuse Monitoring The MCU will monitor the two fuses through which power is supplied to
the SLIC’s. If no power is detected on the side of the fuse the MCU will
light the fault LED on the Tornado IO Annex card only.
Diplex FPGA The main responsibility of the MCU is to program the Diplex FPGA on
Programming any SLIC that may be inserted into the Tornado.The MCU holds a copy of
the Diplex FPGA image at all times. It will be up the POST or application
software to verify that the version of Diplexed FPGA software is up to
date. If the version is not up to date the POST or application software will
have to send down the latest version to the MCU via the I2C bus.
I2C Interface The MCU uses an I2C interface to communicate with the CPU. The
MCU’s hardware I2C interface is compatible with the 16KHz speed used
in the Dreadnought system.
There is an isolation switch between the MCU and the master on the bus.
This serves two purposes. The first is to allow us to isolate the MCU from
the bus in case for some reason that the MCU becomes stuck and holds
either the data or clock lines low. The second is to allow two devices with
the same address to be on the bus. The switch allows for either the MCU
in slot A or the MCU in slot B to be disconnected from the bus while the
other is communicating with the master. The switch address is dependent
on the slot the switch is in.
EMC CONFIDENTIAL
MCU_WD_OUT IO_ANNEX_PWRGD_IN
MCU_WD_ENABLE IO_FUSES_OK
MCU_I2C_ATTN_N
MCU_IO1_RST_N
IO 0/1_FLT
LPC2136
MCU_TXD0 MCU_JTAG_MSTR
MCU_RXD0 MCU_PWRGD_CNTRL
MCU_I2C_RST_N
MCU_MSTR_SCL IO0/1_PWRGD
MCU_MSTR_SDA IO 0/1_PWREN
MCU_SCL IO0_INS_N
MCU_SDA IO_ANNEX_FLT_LED
IO_ANNEX_RST_N
5
MCU_TDO MP_SLOT _ID
MCU_TDI
LCL_MFG_MODE_N
MCU_TCK
LCL_MARGIN_LOW_N
MCU_TMS LPC2136 _Pinout.emf LCL_MARGIN_HIGH_N
MCU_TRST_N
LPC2131 Signal
Descriptions
Table 5-23. LPC2131 Signal Descriptions
SCHEMATIC SIGNAL I/O Description
Miscellaneous
Vref I 3.3V
MCU_RST_N I When asserted this signal resets the MCU.
XTAL1/2 I 12MHz Clock
Control, Monitoring and Status Signals
PWM_CNTRL_EN_N O Output enable for the Bus FET Switch for the Fan controls.
FAN_PWM_CBT O Control the FAN speed.
FAN_A_TACH I Tachometer output signal from the fans which is monitored by the MCU. If this
signal drops below the minimum acceptable operating range the fan will be declared
faulted
FAN_B_TACH I Tachometer output signal from the fans which is monitored by the MCU. If this
signal drops below the minimum acceptable operating range the fan will be declared
faulted
MCU_WD_OUT I MCU watchdog monitor data latch
EMC CONFIDENTIAL
EMC CONFIDENTIAL
Tornado to
Midplane
Interface
5
EXP_FRM_BLD_< 7:4>+/- XAIO_EXPT <7:4>_+/- XAIO_EXPT <7:4>_+/-
EXP_TO_BLD_<7:4>+/- XAIO_EXPR <7:4>_+/- XAIO_EXPR <7:4>_+/-
PMC 1 Blade A
MP_USB+/- ANNEX_A_USB_+/- G6/H6 ANNEX_A_USB_+/-
MP100MHZ_CLK+/_ IOA_100MHZ_CLK_+/- G5/H5 IOA_100 MHZ_CLK_+/_
IO0/1_UART_TO_BLD RS232 _TX0/1_TO_BLDA RS232_TX0/1_TO_BLDA
IO0/1_UART_FRM_BLD RS232 _RX0/1_FR_BLDA RS232_RX0/1_FR_BLDA
IO_ANNEX_RST_N IO_ANNEX_A_RST_N I16 IO_ANNEX_A_RST_N
IO_ANNEX_PWRGD IO_ANNEX_A_PWRGD I19 IO_ANNEX_A_PWRGD
IO0/1_FLOWCNTL_N FLOW_CNTRL_AUAR_0/1 E16/E17 FLOW_CNTRL_AUAR_0/1
IO_ANNEX_PWR_EN IO_ANNEX_A_PWR_EN E15 IO_ANNEX_A_PWR_EN
MP_I2C_RST_N PS_A_I2C_RST_N PS_A_I2C_RST_N Blade A &B, MGMT A&B, PSA , Nova Tst Card
MP_RTCK ANNEX_A_MCU_RTCK ANNEX_A_MCU_RTCK
MP_MCU_RST_N ANNEX_A_MCU_RST_N ANNEX_A_MCU_RST_N Nova Tst Card
MP_JTAG_EN_N IOA_JTAG_EN_N IOA_JTAG_EN_N
PLEASE_BIFURC_N PLEASE_BIFURC_A_N PLEASE_BIFURC_A_N Blade A &B, IO Annex B
MARGIN_HIGH_N V_MARGIN_A_HIGH_N V_MARGIN_A_HIGH_N
Blade A , MGMT A,
MARGIN_LOW_N V_MARGIN_A_LOW_N V_MARGIN_A_LOW_N
Nova Tst Card
MFG_MODE_N A_MFG_MODE_N A_MFG_MODE_N
MP_SLOT_ID GND GND Slot ID
IO_ANNEX_INS_IN_N GND GND Insert GND loop start
Midplane_Pinout.emf
EMC CONFIDENTIAL
Introduction The Dreadnought test board code name is the NOVA. It preforms four
major functions;
The Nova test card plugs into the front of the unit into the midplane.
EMC CONFIDENTIAL
JTAG ALT
MFG A MCU A MCU B MFG B
POD ITP
017-001-097 017-006-250 017-017-217 017-006-250 017-006-250 017-001-097
EMC CONFIDENTIAL
216
POD TDO
SW1 P3 BLDA_ITP_ DBRST_N 1.2V This enable anded with
JTAG enable
017-017-303
BLDA_ITP_TCK0 1.2V o Pod TCK
30 PINS
BLDA_ITP_TRST_N 1.2V 3.3v to o Pod TRST JTAG
1.2v POD
Blade A BLDA_ITP_TMS_MAIN 1.2V Level o Pod TMS
Shift From JTAG Diagram 1
BLDA_ITP_TDI_MAIN 1.2V o Pod TDI
X
BLDA_ITP_TDO_MAIN 1.2V 1.2v to 3.3v
M
SW2 P3
Level Shift
SW3 P1 BLDA_ITP_DBRST_N 1.2V This enable anded with
JTAG enable SW1 P1
BLDB_ITP_TCK0 1.2V o
I Blade B
BLDB_ITP_TRST_N 1.2V
BLDB_ITP_TMS_MAIN 1.2V
3.3v to
1.2v
Level
o
o
0 1
U51 P15
JTAG Select
D
U51 P6
Shift
BLDB_ITP_TDI_MAIN 1.2V o
U52 P15
BLDB_ITP_TDO_MAIN 1.2V 1.2v to 3.3v
P
U51 P2
Level Shift
To Blade A Reset
SW1 P3
0 1
A
plus BLDB_FSB_RST, BPM's
3,4,5 from Proc 0 and 1 ITP B
HEADER
5
HEADER
A_MFG_MODE_N B_MFG_MODE_N 1
V_MARGIN_A_HIGH V_MARGIN_B_HIGH 2
V_MARGIN_A_LOW V_MARGIN_B_LOW 3
PS_A_I2C_RST_N PS_B_I2C_RST_N 4
3.3V_BLADEA -SENSE 3.3V_BLADEB_SENSE 5
GND GND 6
PS_A_I2C_SCL PS_B_I2C_SCL 7
PS_A_12C SDA PS_A_12C SDA 8
I2C_ARB_A1 I2C ARB_B1 9
EMC CONFIDENTIAL
EMC CONFIDENTIAL
218
BLADE A ENABLE_N 3.3V ITP enable and JTAG enable
017-017-893
Anded for this inputt
BLADE A TCK 3.3V o Pod TCK
120 PINS
BLAD E A TRST 3.3V o Pod TRST JTA G
POD
Blade A BLADE A TMS 3.3V o Pod TM S
M
SW 2 P2
BLADE B ENABLE _N 3.3V ITP enable and JTAG enable
Anded for this inputt SW 2 P4
BLADE B TCK 3.3V o
I Blade B
BLAD E B TRST 3.3V
o
0 1
U51 P1
JTAG SELECT_N
D
U51 P7
BLADE B TDI 1.2V 3.3 to 1.2 Level o
U52 P5
Shift
BLAD E B TDO 3.3V
P
U52 P7
IO A ENABLE_N 3.3V
o 0 1 SW 4 P1
IO A TCK 3.3V
L IO A
IO A TRST 3.3V
IO A TM S 3.3V
o
o
SW 4 P3
A IO A TDI 3.3V
IO A TDO 3.3V
o
N IO B ENABLE_N 3.3V
IO B TCK 3.3V o
0 1
E IO B
IO B TRST 3.3V
IO B TM S 3.3V
o
IO B TDI 3.3V o
IO B TDO 3.3V
V+3.3 V+1.2
V +12_SBY A 2AM PS 0 1
Blocking V+12 3.3V 1.2V
V +12_SBY B 2AM PS Diodes Regulator Regulator To JTAG Block Diagram
below
G ND 4 AM PS GND X
5
017-017-893
120 PINS
I MGT A
MGT A TRST 3.3V
o
SW5 P2
P
U53 P1
MGT B ENABLE_N 3.3V
0 1
MGT B TCK 3.3V o
L MGT B
MGT B TRST 3.3V
0 1
N
E
X To XDP Block Diagram
EMC CONFIDENTIAL
EMC CONFIDENTIAL
220
A Side MCU Select
BLADE
SW5 P3
M BLADEA_MCU_RST_N
BLADEA_JTAG_TDO
o
I BLADEA_MCU_RTCK
o
D 073-006-221
P MGMT_A_JTAG_TCK ATCK 9 J8
o
MGT_A_JTAG_TRST_N ATRST 3
MGT A o
L MGMT_A_JTAG_TMS ATMS 7
MCU
o Header
MGMT_A_JTAG_TDI ATDI 5 20 pins
o
A
15 A
MGT_A_JTAG_MCU_RS ARST
Side
o
MGMT_A_JTAG_TDO ATDO 13
o
N MGT_A_JTAG_MCU_CK ARTCK 11
o
E IOA_JTAG_TCK
o
IOA_JTAG_TRST_N
Annex A IOA_JTAG_TMS
o
o
IOA_JTAG_TDI
o
ANNEX_A_MCU_RST_N
5
o
IOA_JTAG_TDO
o
ANNEX_A_MCU_RTCK
o
MCU Connector There are two headers for MCU emulation. one for the A Field
replaceable Units (FRU’s) the other for the B side FRU’s. Each side has 3
J8 & J11 FRU’s that can be emulated, one at a time. These are the Blade,
Management board and Annex for each side. This can be done with the
configuration switches manually or via program control.
EMC CONFIDENTIAL
222
Index
A P
Acronyms 25 PCI 79
PCI-Express 41
B Port 80 99
POST 91
battery 99
BIOS 91, 97 R
C RS232 91, 98
COM 1 98 S
COM 2 98
SIO 91
E strapping 96
Super IO 79
Ethernet 79, 114
H
HubLink 41
I
I2C 79
ICH 41
L
LAN 114
Lithium Battery 99
Low Pin Count (LPC) 79
LPC 79, 91
N
NVRAM 91, 99
EMC CONFIDENTIAL
Index 1
EMC CONFIDENTIAL
2 Index