You are on page 1of 224

Dreadnought Storage System

DREADNOUGHT
STORAGE SYSTEM
Troubleshooter’s Guide
P/N XXX-XXX-XXX
DRAFT FOR PILOT CLASS 1/15/08
REV. A

EMC Corporation 171 South Street, Hopkinton, MA 01748-9103


Corporate Headquarters: (508) 435-1000, (800) 424-EMC2
Fax: (508) 435-5374, Service: (800) SVC-4EMC
EMC CONFIDENTIAL
Copyright © [2008] EMC Corporation. All Rights Reserved.
Printed January 2008
First Edition

No part of this publication may be reproduced or distributed in any form or by any means, or stored in a database
or retrieval system, without the prior written consent of EMC Corporation.

EMC believes the information in this publication is accurate as of its publication date. The information is subject to
change without notice.

THE INFORMATION IN THIS PUBLICATION IS PROVIDED "AS IS." EMC CORPORATION MAKES NO
REPRESENTATIONS OR WARRANTIES OF ANY KIND WITH RESPECT TO THE INFORMATION IN THIS
PUBLICATION, AND SPECIFICALLY DISCLAIMS IMPLIED WARRANTIES OF MERCHANTABILITY OR
FITNESS FOR A PARTICULAR PURPOSE.

Use, copying, and distribution of any EMC software described in this publication requires an applicable software
license

ICDA® (Integrated Cached Disk Array), EMC2® (the EMC logo), THE STORAGE ARCHITECTS®,
MOSAIC:2000®, Harmonix®, CLARiiON® and Symmetrix® are registered trademarks and EMC™ and SRDF™
and FLARE™ are trademarks of EMC Corporation.

__________________________________________________________________________________________

All other trademarks used herein are the property of their respective owners.

Trademark Information

EMC CONFIDENTIAL
Preface

Description This document looks at EMC’s application of the hardware used in the
Dreadnought systems. It is designed for personnel who have to repair
EMC’s CLARiiON hardware to component level.

It begins by introducing the EMC CLARiiON Dreadnought Enclosure


system, followed by an introduction to the Wildcat-S which is the SP or
motherboard. The Wildcat-S components will then be examined to the
component level in chapter 2. Chapters 3 will examine the Management
FRU’s used in the enclosure. Chapter 4 will identify what IO Modules
(SLIC’s) will be used in the Dreadnought. Chapter 5 examines enclosure
power and the Wildcat-S power. Chapter 6 finishes the manual with an
examination of the IO ANNEX extender board and NOVA test board
used for boundary scan at the system level.

Prerequisites This manual is not a substitute for hardware specifications and


schematics. Rather it aims to be the central point of reference when
debugging Dreadnought boards. Refer to documentation from Design
Engineering for precise details on signals and functioning of any of the
components.

It is assumed that readers have a knowledge of, or access to, the following
topics:

Interfaces & Protocols - Fibre Channel, Ethernet, I2C, RS232, PCI, PCI-X,
PCI-Express, iSCSI and SAS.

As the Wildcat-S controller is part of the CLARiiON family, readers


should be familiar with these systems overall architecture and operation.

Acknowledgments The Dreadnought Troubleshooter Guide editorial team would like to


acknowledge the contributions making this guide possible from the
following people:

• Sylvia Dunne
• Eugene McVeigh
• Fred Lilienkamp
• Dave Caissie
• Roger Guilbault
• Dave Thomas

Also special thanks for the use of all the Engineering Documentation and
assistance from various department throughout EMC.

NOTICE: This document contains sensitive technical information which is


for use solely by EMC employees in the execution of their duties within the
EMC Corporation. Any use, duplication or distribution outside the
Corporation is strictly prohibited.

EMC CONFIDENTIAL

iii
EMC CONFIDENTIAL

iv Preface
Warnings and Cautions

The following warnings and cautions pertain throughout this manual.

WARNING Trained service personnel only.

This unit has more than one power supply cord. To reduce the risk of
electric shock, disconnect two (2) power supply cords before servicing.

Ground circuit continuity is vital for safe operation of the machine. Never
operate the machine with grounding conductors disconnected.
Remember to reconnect any grounding conductors removed for or
during any installation procedure.

ATTENTION Resérvé au personnel autorisé.

Cet appareil comporte plus d'un cordon d'alimentation. Rafin de prévenir


les chocs électriques, débrancher les deux cordons d'alimentation avant
de faire le dépannage.

Un circuit de terre continu est essentiel en vue du fonctionnement


sécuritaire de l'appareil. Ne jamais metre l'appareil en marche lorsque le
conducteur de mise a la terre est débranché.

WARNUNG Nur für Fachpersonal.

Das Geraet hat mehr als eine Anschlussleitung. Zur Vermeidung der
Gefahr eines elektrischen Schlages sind vor dem öffnen beide
Anschlussleitungen vom Netz zu trennen.

STROMSTREUVERLUST: Gerät muss geerdet werden, bevor es am


Stromnetz angeschlossen wird.

Additional The system operates at high voltages. To protect against physical harm,
Warnings and power off the system whenever possible while servicing.
Cautions
In case of fire or other emergency, isolate the system's power involved
and alert appropriate personnel.

Exercise great care at all times when working on the machine. Remember
to:

• Remove rings, watches, or other jewelery and neckties before you


begin any procedures.

• Use caution near any moving part and any part that may start unex-
pectedly such as fans, motors, solenoids, etc.

EMC CONFIDENTIAL

Warnings and Cautions v


• Always use the correct tools for the job.

• Always use the correct replacement parts.

• Keep all paperwork, including incident reports, up to date, complete,


and accurate.

Static EMC incorporates state-of-the-art technology in its designs, including the


Precautions use of LSI and VLSI components. These chips are very susceptible to
damage caused by static discharge and need to be handled accordingly.

Unless specifically designed for non-disruptive replacement, never plug


or unplug boards with the power on. Severe component damage may
result.

EMC CONFIDENTIAL

vi Warnings and Cautions


Contents

Chapter 1 Dreadnought System Architecture.................................. 15


1.1 System Intro ........................................................................................... 16
1.2 Wildcat-S Chassis Overview ............................................................... 18
1.3 Wildcat-S Chassis Components .......................................................... 20
1.4 Dreadnought Code Names.................................................................. 24
1.5 Acronyms ............................................................................................... 25

Chapter 2 Dreadnought SP Board..................................................... 27


2.1 Wildcat-S Blade Introduction .............................................................. 28
2.2 Intel Clovertown CPU .......................................................................... 35
2.3 Blackford MCH ..................................................................................... 41
2.4 FB DIMMS.............................................................................................. 51
2.5 PLX PEX8524 PCIe Switch................................................................... 67
2.6 ICH 7 ....................................................................................................... 79
2.7 Server IO & Slow Devices .................................................................... 91
2.8 Clocks & Interrupts............................................................................. 101
2.9 Chassis Management (LAN) ............................................................. 111
2.10 Chassis Management (RS232) ........................................................... 123
2.11 Chassis Management (I2C)................................................................ 129

Chapter 3 Management FRUS ......................................................... 145


3.1 Introduction ......................................................................................... 146
3.2 Solar Flare - Management FRU (SAN)............................................. 147
3.3 Earthquake - Management FRU (NAS) ........................................... 169

Chapter 4 Chassis Power................................................................. 191


4.1 Wildcat-S Chassis Power.................................................................... 192

Chapter 5 Other Cards ..................................................................... 203


5.1 Tornado IO Annex Extender Card ................................................... 204
5.2 Nova Test Card.................................................................................... 215

EMC CONFIDENTIAL

Contents vii
viii Contents
Figures

1-1 CLARiiON Dreadnought System front view ........................................................................... 16


1-2 Wildcat-S Chassis rear view........................................................................................................ 18
1-3 Wildcat-S Chassis Front View .................................................................................................... 19
1-4 Wildcat-S Chassis Components flat layout............................................................................... 20
1-5 Wildcat-S blade with 4 IO Modules........................................................................................... 21
1-6 Wildcat-S & IO Modules Numbering Scheme ......................................................................... 21
1-7 Wildcat-S Blade Drawing ............................................................................................................ 22
2-8 Wildcat-S Blade............................................................................................................................. 28
2-9 Wildcat-S Interfaces Block Diagram .......................................................................................... 29
2-10 Wildcat-S Parts Layout ................................................................................................................ 31
2-11 Clovertown Front Side Bus ......................................................................................................... 35
2-12 Front Side Bus Signal Timing ..................................................................................................... 36
2-13 Clovertown CPU Pinout.............................................................................................................. 39
2-14 Blackford MCH Block Diagram.................................................................................................. 41
2-15 Generic PCI Configuration Register Format ............................................................................ 45
2-16 PCI Configuration Registers ....................................................................................................... 46
2-17 Blackford MCH Signals ............................................................................................................... 47
2-18 FBDIMM photo Samsung............................................................................................................ 51
2-19 FBDIMM Connector ..................................................................................................................... 51
2-20 Wildcat-S FBDIMM Interface...................................................................................................... 52
2-21 Wildcat-S with 8 FBDIMM connected to 4 Channels .............................................................. 53
2-22 FBDIMM Channel 0 Interface to the Blackford MCH............................................................. 54
2-23 MCH Branches with balanced DIMM Populated.................................................................... 55
2-24 MCH Branches with Memory Controllers................................................................................ 55
2-25 FBDIMM Components................................................................................................................. 56
2-26 MCH SMBus 1-4 interface to Channel 0 FBDIMM’s ............................................................... 60
2-27 (AMB) Advanced Memory Buffer.............................................................................................. 61
2-28 South Bound Data Format........................................................................................................... 63
2-29 North Bound Data Format in 14 Lane Mode............................................................................ 65
2-30 PLX in the Peer-to-Peer CMI Path.............................................................................................. 67
2-31 PLX Block Diagram ...................................................................................................................... 68
2-32 Non-Transparent Port Address Translation............................................................................. 69
2-33 PLX EEPROM Interface ............................................................................................................... 72
2-34 SPI Bus Timing.............................................................................................................................. 72
2-35 Generic PCI Configuration Register Format ............................................................................ 73
2-36 PLX Doorbell and Scratch pad Registers .................................................................................. 74
2-37 PLX Signal Pins ............................................................................................................................. 75
2-38 Hance Rapids - ICH...................................................................................................................... 80
2-39 USB Interfaces ............................................................................................................................... 85
2-40 LAN Ethernet Communications Connections.......................................................................... 86
2-41 Broadcom BCM5751 Pinout ........................................................................................................ 88
2-42 Wildcat-S Server IO Block Diagram........................................................................................... 91
2-43 Wildcat-S Server IO Pin out ........................................................................................................ 92
2-44 BIOS / POST Block Diagram ....................................................................................................... 97
2-45 NVRAM, DUART, QUART & PORT80 CONN Interface ...................................................... 98
2-46 Crystals and Oscillators are shown in bright Yellow............................................................ 101
2-47 CPU1, CPU0 & Blackford MCH Clocks .................................................................................. 102
2-48 FBDIMM Clocks.......................................................................................................................... 103
2-49 NAND FLASH Module Clock .................................................................................................. 103
EMC CONFIDENTIAL

Figures ix
2-50 Interface Controller Hub (ICH) Clocks ................................................................................... 104
2-51 PLX Clocks................................................................................................................................... 105
2-52 LAN1 & LAN2 Clocks ............................................................................................................... 105
2-53 Server IO & Firmware Hub Clocks .......................................................................................... 106
2-54 Management Controller Clock ................................................................................................. 106
2-55 33Mhz to Uarts and IO Modules .............................................................................................. 107
2-56 Midplane and Debug connector Clocks .................................................................................. 107
2-57 Wildcat-S Interrupts Block ........................................................................................................ 108
2-58 Interrupts received by SIO & ICH7.......................................................................................... 109
2-59 SAN Cabinet with one Wildcat-S Chassis............................................................................... 112
2-60 Wildcat-S SAN Chassis - LAN Block Diagram ...................................................................... 113
2-61 BMC5751 LAN Chip Block Diagram ....................................................................................... 114
2-62 Wildcat-S Chassis - 2 Solar Flare FRU’s (rear view) .............................................................. 115
2-63 Solar Flare Management FRU Face Plate ................................................................................ 115
2-64 Solar Flare Interface Block......................................................................................................... 116
2-65 Dreadnought NAS System LAN - Example Only ................................................................. 117
2-66 Dreadnought NAS - LAN Block Diagram .............................................................................. 119
2-67 Earthquake Management FRU Face Plate............................................................................... 120
2-68 Earthquake Interface Block ....................................................................................................... 121
2-69 Wildcat-S RS232 UART Ports ................................................................................................... 124
2-70 DUART for IO4 & IO5 Diplexing ............................................................................................. 125
2-71 Quad UART Pinout .................................................................................................................... 126
2-72 Wildcat-S Chassis Midplane I2C Interfaces (SAN)................................................................ 130
2-73 SAN I2C Arbitration Signals..................................................................................................... 131
2-74 Midplane SAN I2C Access Arbitration Diagram................................................................... 132
2-75 Wildcat-S Chassis Midplane I2C Interfaces (NAS)................................................................ 133
2-76 NAS I2C Arbitration Signals..................................................................................................... 134
2-77 ICH7/MCH I2C Master Address Map..................................................................................... 136
2-78 Wildcat-S I2C Interfaces ............................................................................................................ 137
2-79 SAN MCU I2C Slave Port Access............................................................................................. 138
2-80 NAS MCU I2C Slave Port Access............................................................................................. 139
2-81 MCU I2C Master Port Address Map ....................................................................................... 140
2-82 MCU I2C Master Interface ........................................................................................................ 141
2-83 Blackford MCH FBDIMM I2C Address Map ......................................................................... 142
2-84 Blackford MCH Hot-Plug I2C Address Map ......................................................................... 142
2-85 Blackford MCH I2C Diagram ................................................................................................... 142
2-86 Wildcat-S I2C Reset, Arbitration & Attention Signals .......................................................... 143
3-87 Solar Flare .................................................................................................................................... 147
3-88 Solar Flare Block Diagram......................................................................................................... 148
3-89 Broadcom BCM5397 Switch...................................................................................................... 149
3-90 BCM5397 block diagram............................................................................................................ 150
3-91 Solar Flare Ethernet Crosslink .................................................................................................. 151
3-92 LPC2131 Microcontroller Block Diagram ............................................................................... 152
3-93 Philips LPC2131 Microcontroller ............................................................................................. 153
3-94 Serial Crosslink Block Diagram................................................................................................ 154
3-95 Solar Flare I2C Topology ........................................................................................................... 155
3-96 Slot Dependent I2C Addresses ................................................................................................. 155
3-97 Solar Flare Voltage Sequence.................................................................................................... 156
3-98 Command Power Sequencer..................................................................................................... 157
3-99 Broadcom BCM5397 pin-out..................................................................................................... 158
3-100 Broadcom BCM5397 pin-out (cont.) ........................................................................................ 159
3-101 Broadcom BCM5397 pin-out (cont.) ........................................................................................ 160
3-102 Philips LPC2131 pin-out ............................................................................................................ 161
3-103 Philips LPC2131 pin definitions ............................................................................................... 162
3-104 Philips LPC2131 pin definitions (cont.)................................................................................... 163
EMC CONFIDENTIAL

x Figures
Figures

3-105 Philips LPC2131 pin definitions (cont.)................................................................................... 164


3-106 Philips LPC2131 pin definitions (cont.)................................................................................... 165
3-107 CMD_Power Sequencer pin-out............................................................................................... 166
3-108 CMD Power Sequencer Pin Definitions .................................................................................. 167
3-109 Earthquake................................................................................................................................... 169
3-110 Earthquake Block Diagram ....................................................................................................... 170
3-111 Tremor Riser Card ...................................................................................................................... 171
3-112 Earthquake Ethernet Block Layout .......................................................................................... 172
3-113 BCM5397 block diagram............................................................................................................ 173
3-114 Motorola Coldfire Block Diagram............................................................................................ 174
3-115 Serial Mux Control ..................................................................................................................... 175
3-116 Earthquake I2C Topology ......................................................................................................... 176
3-117 Slot Dependent I2C Addresses ................................................................................................. 177
3-118 Earthquake Power Distribution................................................................................................ 178
3-119 Earthquake Voltage Sequence .................................................................................................. 179
3-120 Earthquake Reset Diagram........................................................................................................ 180
3-121 Earthquake LED Control ........................................................................................................... 181
3-122 Earthquake JTAG Topology...................................................................................................... 182
3-123 Coldfire BGA pinout .................................................................................................................. 183
3-124 Coldfire BGA pinout (cont.)...................................................................................................... 184
3-125 Coldfire BGA pinout (cont.)...................................................................................................... 185
3-126 Coldfire BGA pinout (cont.)...................................................................................................... 186
3-127 Coldfire BGA pinout (cont.)...................................................................................................... 187
3-128 Earthquake CMD_Power_Sequencer ...................................................................................... 188
3-129 Tremor Riser Card RS-232......................................................................................................... 189
4-130 Wildcat-S Chassis Rear View .................................................................................................... 192
4-131 Wildcat-S Chassis 12 Volt Standby power.............................................................................. 193
4-132 Wildcat-S Blade Standby Power Usage................................................................................... 194
4-133 Wildcat-S Chassis 12 Volt Main power ................................................................................... 195
4-134 Wildcat-S 12 volt Main power to DC-DC & IO Connectors................................................. 196
4-135 Wildcat-S Power Enable Signals............................................................................................... 197
4-136 Wildcat-S Power up Good Sequence ....................................................................................... 198
4-137 Wildcat-VID Monitoring ........................................................................................................... 199
4-138 Manufacturing Mode Signals used for Margining ................................................................ 200
4-139 Voltage Margin Signals.............................................................................................................. 201
4-140 CMD Pinout................................................................................................................................. 202
5-141 Rear side of the Dreadnought Enclosure ................................................................................ 204
5-142 SP to IO Annex Card Interface ................................................................................................. 204
5-143 Tornado Block Diagram............................................................................................................. 206
5-144 PCI-Express Tx/Rx Buffer Block Module ................................................................................ 207
5-145 PM8380 Pinout ............................................................................................................................ 208
5-146 Tornado Micro controller Pinout ............................................................................................. 211
5-147 Tornado to Midplane Interface................................................................................................. 213
5-148 Nova Test Card ........................................................................................................................... 215
5-149 Airdam on the Nova Test Card ................................................................................................ 216
5-150 MFG & XDP BLOCK DIAGRAM............................................................................................. 217
5-151 JTAG BLOCK DIAGRAM FOR A\B FRU’s ........................................................................... 219
5-152 MCU Headers J8 & J11............................................................................................................... 221

EMC CONFIDENTIAL

Figures xi
EMC CONFIDENTIAL

xii Figures
Tables

1-1 Dreadnought I/O Cards (SLIC’s)................................................................................................ 22


2-2 Wildcat-S SP CPU Configurations Comparison ...................................................................... 28
2-2 FB DIMM Wildcat-S ..................................................................................................................... 51
2-3 FBDIMM Connector Miscellaneous Signals ............................................................................. 56
2-4 FBDIMM Connector Differential Data Pins to/from MCH .................................................... 57
2-5 FBDIMM Connector Differential Data Pins to next DIMM.................................................... 58
2-6 Southbound Lane Operational Modes ...................................................................................... 62
2-7 Northbound Modes...................................................................................................................... 64
2-8 PLX H/W Strapping Pins............................................................................................................. 71
2-9 PLX EEPROM SPI Signals ........................................................................................................... 72
2-10 PLX PEX8524 Signals ................................................................................................................... 76
2-11 ICH 7 Pin Descriptions ................................................................................................................ 81
2-12 Power Rail Minimum Threshold Levels ................................................................................... 87
2-13 BCM5751 Pin Descriptions.......................................................................................................... 88
2-14 Super IO Chip Pin Descriptions ................................................................................................. 93
2-15 SIO Chip Selects........................................................................................................................... 95
2-16 SIO Hardware Strapping Signals ............................................................................................... 96
2-17 Super IO COM Ports .................................................................................................................... 97
5-18 SLIC PCI Express Lane Configuration .................................................................................... 207
5-19 Blade PCI Express Lane Configuration ................................................................................... 207
5-20 PM8380 Pin Descriptions........................................................................................................... 208
5-21 MCU’s response to onboard temperature changes .............................................................. 210
5-22 Isolation Switch I2C Addresses ................................................................................................ 210
5-23 LPC2131 Signal Descriptions .................................................................................................... 211
5-24 MFG Connector Pins .................................................................................................................. 217
5-25 XDP Pinout .................................................................................................................................. 218
5-26 JTAG Connector Pins ................................................................................................................. 220
5-27 MCU Connector Pins ................................................................................................................. 221
5-28 Switch Settings ............................................................................................................................ 222

EMC CONFIDENTIAL

Figures xiii
EMC CONFIDENTIAL

xiv Figures
1

Chapter 1 DREADNOUGHT SYSTEM


ARCHITECTURE

Chapter Contents:

• System Intro............................................................................................... 16

EMC CONFIDENTIAL

15
1.1 System Intro
Dreadnought is a high-end converged Midrange SAN/NAS offering. The
Dreadnought system architecture in many ways will look similar to the
hammerhead since it is the next generation to the Hammerhead system.
The cabinet shown below will house the Wildcat-S Chassis. There will be
configurations which combine Wildcat-S Chassis, D15= (DAE) Disk Array
Enclosure and control stations (servers). The Dreadnought cabinet will be
populated according to the configuration guide.

8 DAE
(120 Drives)

Wildcat-S
Chassis

SPS

Figure 1-1. CLARiiON Dreadnought System front view


This manual deals mainly with the Wildcat-S Chassis. The chassis will be
described starting from the system level and will examine the individual
boards down to component level. The architecture and functionality will
be discussed. This will give the reader an understanding of how the
Dreadnought system works, so as to allow the debug of faulty modules.
EMC CONFIDENTIAL

16 DREADNOUGHT SYSTEM ARCHITECTURE


Operating The Dreadnought will use the Flare operating system. This will include
embedded firmware (BIOS, POST). Flare will be shipped with the system.
1
System

NAS & SAN Network Attached Storage (NAS) and Storage Area Network (SAN) are
two different storage architectures for accessing stored information.

NAS With NAS, the storage device is connected directly to the network,
normally through a network interface such as Gigabit Ethernet and iSCSI.
Our NAS architecture requires a processing engine to operate which is
called a ‘Data Mover’,

SAN The SAN implementations are used to connect storage to the network via
servers, but require very high speed access to these servers, so they are
normally connected via Fibre Optic cables and use the Fiber Channel and
FICON protocols. The SAN architecture requires a processing engine
which is called a ‘Storage Processor’.

Both processing engines use the Wildcat-S as its base board and the
differences occur when different IO Modules (SLIC’s) and Management
FRU’s are inserted into the motherboard. There are different portions of
the software used to operate the different configurations.
For example: (Data Mover & Storage Processor).

EMC CONFIDENTIAL

System Intro 17
1.2 Wildcat-S Chassis Overview
A Wildcat-S Chassis is made up of a number of different parts:

• Wildcat-S Chassis with midplane


• 2 Power Modules A & B
• 2 Management FRU’s A & B.
• 2 Wildcat-S Blades, each with 2 processors, up to 8 FBDIMM’s and Intel
chipset
• There are 2 types of Management FRU’s used in the Wildcat-S Chassis
- SAN configurations - uses two Solar Flare Management FRU’s
- NAS configurations - uses two Earth Quake Management FRU’s
• 2 Tornado I/O Annex Extender cards can be inserted into the I/O Annex Slot.
Each Tornado can accept two I/O Modules (SLIC’s) for a total of 4.
• 2 Power Supply units A & B
• 4 Blower units
• Up to 12 I/O modules - (SLIC’s) Small Form Factor I/O Cards per Chassis.

A Wildcat-S blade is made up of one Wildcat-S motherboard with 4 IO


Modules inserted. As always, there are two blades for redundancy, the
lower one is labelled blade A. The upper is blade B. See Figure 1-2 for
more details.

Power Power
Supply B Wildcat-S Blade B Supply A
Power & Led’s

Wildcat-S
Blade A

B - Solar Flare 8 I/O Modules A - Solar Flare


Management FRU 2 IO Annex Slots w /
Management &
Console & SPS Ports 2 I/O Modules each
Service Ports
Figure 1-2. Wildcat-S Chassis rear view

In the rear of the Wildcat-S Chassis at the bottom there is an IO ANNEX


slot which may have 2 half width I/O Annex extender cards. Each
extender card can accept two IO Modules (SLIC’s) as shown above.

The Wildcat-S Chassis above is the SAN version which can be recognized
by the use of two Solar Flare Management FRU’s

EMC CONFIDENTIAL

18 DREADNOUGHT SYSTEM ARCHITECTURE


Wildcat-S Figure 1-3 below shows the front of the Wildcat-S Chassis with the two
Power Supplies, four blowers and the Nova test board inserted. The
1
Chassis Front Chassis power and fault LEDs are show under power supply B. These
View LED’s are on the midplane and brought to the front of the chassis using
two light pipes.

Power Supply A 4 Fan Blowers Power Supply B

Nova Test Chassis Power


Board Slot Fault Led on Led

Figure 1-3. Wildcat-S Chassis Front View

The Nova test board is located at the bottom center of the Wildcat-S
Chassis in (Figure 1-3). This test board is used for Jtag & boundary scan
purposes in manufacturing only and will not be shipped to the customer.
A blank face plane will be installed for customer shipment.

EMC CONFIDENTIAL

Wildcat-S Chassis Overview 19


1.3 Wildcat-S Chassis Components

Power Supply - A

Management FRU - A

Blower Fan

ITRAC
IO Module 3

ITRAC
IO Module 2
Wildcat-S
Storage Processor
A

ITRAC
IO Module 1
Blower Fan

ITRAC
IO Module 0
M
I
D
P
IO ANNEX
L
Blower Fan A
N
E

ITRAC
IO Module 3

ITRAC
IO Module 2
Wildcat-S
Blower Fan Storage Processor
B
ITRAC

IO Module 1
ITRAC

IO Module 0

Management FRU - B

Power Supply - B

Figure 1-4. Wildcat-S Chassis Components flat layout

Wildcat-S Blade The Wildcat-S Blade (also called a Data Mover in a NAS system) or
(SP in an SAN system) is the core component in the Wildcat-S Chassis. It
consists of 5 separate module:

• 1 CPU Module (Wildcat-S)


• 4 IO Modules
The CPU module is essentially a processor motherboard, with two
processors, RAM and associated components. The four I/O Modules are
inserted into connectors on the Wildcat-S motherboard. Once connected,
the 5 boards are collectively known as a ‘Blade’
EMC CONFIDENTIAL

20 DREADNOUGHT SYSTEM ARCHITECTURE


Wildcat-S Blade In Figure 1-5 below is an example of a Wildcat-S blade. It has three IO
Modules (SLIC’s) installed in Module positions (0,1,2). The 4th module is
1
Photo partially removed from position (3).

Figure 1-5. Wildcat-S blade with 4 IO Modules

Numbering Figure 1-6 below shows the 4 IO Modules and their module and port
numbering scheme. This diagram assumes 4 port IO Modules but there
Scheme are some 2 port modules.
Port 3

Port 2
ITRAC

IO Module 3
Port 1
ICH
Port 0

CPU 1 Port 3

Port 2
ITRAC

MCH IO Module 2
Port 1

Wildcat-S Port 0
GBX

Storage Processor Port 3

Port 2
ITRAC

FB DIMM IO Module 1
Port 1
FB DIMM
CPU 0 FB DIMM Port 0
FB DIMM
Port 3
FB DIMM
FB DIMM Port 2
ITRAC

FB DIMM IO Module 0
Port 1
FB DIMM
Port 0

Figure 1-6. Wildcat-S & IO Modules Numbering Scheme

EMC CONFIDENTIAL

Wildcat-S Chassis Components 21


Wildcat-S CPU In the Dreadnought system the Wildcat-S Chassis will use Wildcat-S
Blades with 2 Quad Core Clovertown CPU’s.
Configurations
.

Figure 1-7. Wildcat-S Blade Drawing

Wildcat-S The CPU Module (Wildcat-S) contains two Intel processors, eight slots of
Motherboard FBDIMM RAM (Fully Buffered Direct In-line Memory Module) and the
Intel Blackford Memory Controller Hub (MCH). Other components
include the Intel Hance Rapids I/O Controller Hub (ICH7), a (FWH)
Firmware Hub chip, dual Ethernet LAN chips, NVRAM, and Server I/O
chip.

I/O Modules The I/O Modules which will be used in the Dreadnought system are
available in a few different configurations, to allow for different interfaces
on the front-end and back-end of the system.

The IO Modules will not be covered in this manual and may be found in
the SLIC Small IO Card TSG.

Table 1-1. Dreadnought I/O Cards (SLIC’s)

Name Type Ports Speed


Tomahawk FC 4 Copper or Optical SFP 1,2,4 Gb/s - Per Port
Harpoon iSCSI 2 Copper RJ45 1 Gb/s - Per Port
Quicksand GigE/IPsec 2 Optical SFP 1.25 Gb/s - Per Port
Poseidon Ethernet 2 Copper or Optical SFP 10 Gb/s - Per Port
Coromandel SAS / SATA 4 Ports - 2 Lanes Each 6 Gb/sec - Per Lane

EMC CONFIDENTIAL

22 DREADNOUGHT SYSTEM ARCHITECTURE


Power Supplies The Wildcat-S Chassis uses two Tanker Power Supplies which provide
1200 Watts of 12-Volt Main, 12-Volt Auxiliary, and 24V Blower power.
1
Control and status is implemented through an I2C interface, rather than
dedicated signals (there is a separate I2C bus for each supply). The Power
Supplies monitor/control the Blower Modules, and also contain
Power/Fault LEDs and Over-Temp sensors.

Management At the sides of the chassis are two management boards. They can be used
to communicate with other Wildcat-S Chassis (NAS) and share
Boards management functions. These boards are available in two different
configurations:

• NAS configuration - Earth Quake

• SAN configuration - Solar Flare

These boards mount in a metal carrier, which slides into the chassis
underneath the Power Supply. The airdam has a Power and Fault LED,
Ethernet and RS-232 connectors.

On the NAS configuration, an 7 segment display is also present to


indicate the chassis number.

Nova Test The Nova Test board is for factory use only. It is plugged into the front of
the chassis, where it performs three main functions:
Board
• it provides an interface to the voltage margining circuitry.

• it provides an interface to the ITP port for the Emulator.

• it provides an interface for Jtag & Boundary Scan

EMC CONFIDENTIAL

Wildcat-S Chassis Components 23


1.4 Dreadnought Code Names
• Dreadnought - Is the name for the cabinet and any modules installed.
• Wildcat-S - This is the motherboard which plugs into the midplane of the
Chassis. It contains the CPU complex which is what controls how the system
operates.
• Solar Flare - SAN Management FRU (Field Replaceable Unit) which is used
to route Ethernet, RS232 and I2C to the SP’s in the Wildcat-S Chassis.
• EarthQuake - NAS Management FRU (Field Replaceable Unit) which is
used to route Ethernet, RS232 and I2C to the SP’s in the Wildcat-S Chassis.
• Fogbow - Wildcat-S Chassis Midplane
• Tornado - IO ANNEX Extender card which accepts 2 IO Modules (SLIC).
Two may be inserted in the IO Annex Slot.
• NOVA Test Board - JTAG board used for boundary scan and Jtag. Only used
in manufacturing - does not ship with system.

EMC CONFIDENTIAL

24 DREADNOUGHT SYSTEM ARCHITECTURE


1
1.5 Acronyms
• 6Gb - Six Giga-bit
• ADC - Analog to Digital Converter
• ATA - advanced Technology Attachment
• BE - Back End
• BIOS - Basic Integrated Operating System
• CMI - Common Management Interface
• CPLD - Complex Programmable Logic Device
• DAE - Disk Array Enclosure. Used for expansion. Contains disk drives.
• DFM - Designed for Manufacture
• DFT - Designed for Test
• DPE - Disk Processor Enclosure. Contains one or two SP boards and disk
drives.
• E3 - Extended Embedded Exerciser
• FBDIMM - Fully Buffered Dual In line Memory Module
• FC - Fibre Channel
• Flare - Operating system used in Clariion Midrange products.
• FRU - Field Replaceable Unit
• FWH - Firmware Hub which holds BIOS & POST code used by the CPU.
• Gb - Gigabits
• GB - GigaBytes
• GigE - Gigabit Ethernet
• GPIO - General Purpose IO
• HA - High Availability - generally meaning that no single point of hardware
failure or one-time software failure will deny access to data (although
performance could be degraded).
• I2C - A low level serial chip-to-chip communications bus/protocol
developed by Philips.
• IO - Input/Output
• iSCS - ISCSI over IP (Internet Protocol)
• LCC - Link Control Card — used to refer to associated software in Flare
used in DAEs on CX arrays. Legacy code uses this abbreviation internally
and it will persist in Mamba.
• LED - Light Emitting Diode
• MC - Management Controller — a complex of one or more chips that
performs management functions on the motherboard.
• MUX - A term that normally means “multiplexor
• NAS - Network Attached Storage
• NC - No Connection
• NDU - Non Destructive Upgrade
• OOB - Out-of-Band
• PCI-e, PCI-E or PCI-Express — a serial point-to-point addition to the PCI
standard that allows high bandwidth through the use of up to 32 parallel 2.5
Gb/s lanes.
• POST - Power On Self Test
• PS - Power Supply
• PSA - Power Supply that is located on the right of the chassis (rear view)
• PSB - Power Supply that is located on the left of the chassis (rear view)
• SAN - Storage Area Network
• SAS - Serial Attached SCSI (ANSI T10 standard) — a standard for serial
point-to-point SCSI.
• SATA - Serial ATA
EMC CONFIDENTIAL

Acronyms 25
• SCSI - Small Computer System Interface
• SLIC - SmalL IO Card form factor
• SGPIO - Serial GPIO
• SMBus - System Management Bus - a protocol on top of an I2C bus for
chip-to-chip management.
• SP - Storage Processor
• SPA - Storage Processor is located in the lower slot of the DPE chassis (rear
view).
• SPB - Storage Processor is located in the upper slot of the DPE chassis (rear
view)
• SPS - Standby Power Supply — a CRU that provides battery backup when
AC power fails.
• STPSATA Tunneling Protocol — part of the SAS specification that allows
SATA traffic to pass from controller, through expanders, to drives. The
controller wraps SATA in STP headers that the final expander unwraps and
sends to the drive.
• TLA - Top Level Assembly
• TWI - Two Wire Interface - a low level serial chip-to-chip communications
bus developed by PMC, similar to I2C
• VPD - Vital Product Data — information programmed by the factory into a
“resume” EEPROM on some FRU’s, generally containing some unique
information on each part such as a WWN seed and serial number. The term
“VPD” is often used to refer to the EEPROM itself.

EMC CONFIDENTIAL

26 DREADNOUGHT SYSTEM ARCHITECTURE


2

Chapter 2 DREADNOUGHT SP BOARD

Chapter contents:

• Wildcat-S Blade Introduction.................................................................. 28

EMC CONFIDENTIAL

27
2.1 Wildcat-S Blade Introduction

Introduction The Wildcat-S Blade consists of

• 1 Motherboard
• 4 IO Modules
• 1 Metal Tray

CP U- 0
Wildcat-S CP U-1
Motherboard
MC H
FB DI M
MS

ICH
IO Module #3
Connector

0
1
4 IO Modules 2
3
Figure 2-8. Wildcat-S Blade

Wildcat-S The Wildcat-S Blade is the base board for the Wildcat-S Chassis which is
used in the Dreadnought System. Each Wildcat-S Chassis contain 2
Motherboard motherboards which will be populated with two Clovertown Quad Core
processors. The processors have large heat sinks shown above which are
secured to the board with 4 screws. The Clovertown CPU uses a 771 pin,
Flip Chip Land Grid Array Chip (FC-LGA4) package. Each CPU chip is
inserted into an LGA771 socket which allows each processor to be
removed.

Table 2-2 illustrates the Wildcat-S Motherboard CPU populations for the
Dreadnought and Tigon system.:

Table 2-2. Wildcat-S SP CPU Configurations Comparison

System Type Base Board CPU Type # CPU Speed Front Side Bus
Dreadnought Wildcat-S Clovertown - Quad Core 2 2.33 Ghz 1333M/TSec
Tigon Wildcat-S Clovertown - Quad Core 2 2.33 Ghz 1333M/TSec

EMC CONFIDENTIAL

28 DREADNOUGHT SP BOARD
Wildcat-S Figure 2-8 on page 28, is a photo of the Wildcat-S Blade and below is an
block diagram of its major components and interfaces.
Interfaces Block

U8 U7
Wildcat-S
205-800-602c 2
U16 LAN1 LAN2

PLX x8 PCIe U18 J53_1S


ICH IO Card 3

x1 PCIe
x1 PCIe
LPC
x8 PCIe CMI

x4 ESI
x8 PCIe IO

U49
INTEL CPU 1 x4 PCIe
Woodcrest= FSB
Clovertown=
U65 x4 PCIe J52_1S
FSB MCH x4 PCIe IO Card 2

J22
x4 PCIe
4Gb/sec each
U96 CH0 CH2
INTEL CPU 0 CH1 CH3
Woodcrest = J11 – FB DIMM
Clovertown= J12 – FB DIMM J51_1S
J9 – FB DIMM IO Card 1

J10 – FB DIMM
J7 – FB DIMM
U120
J8 – FB DIMM
J5 – FB DIMM
J6 – FB DIMM
LPC
COM 1 J50_1S
U141 U140 IO Card 0
COM 2 SIO FWH U149
NVRAM

Figure 2-9. Wildcat-S Interfaces Block Diagram

Each Intel CPU Chip has its own front side bus (FSB) that connects to the
Blackford Memory Controller Hub (MCH).

The Blackford MCH functions as the “bridge” for the processors to the
rest of the system. It provides four fully buffered DIMM interfaces to the
system RAM (FBDIMM). The MCH is also connected to the ICH7 IO
Controller Hub and together these two chips are called the Intel Blackford
chipset.

In our application the MCH is configured as 5 PCI Express Interfaces:

• One 8 Lane PCIe Bus A to the PLX which is a PCIe to PCIe Switch. Each
PCIe port operates at 2.5Ghz.
-The PLX sends x8 PCIe to the Midplane connector for the CMI interface.
-The PLX sends x8 PCIe to the Midplane connector to IO ANNEX interface.
• Four x4 PCIe interfaces to each of the 4 IO Module Connectors. These
connectors may have various Modules inserted into them.

EMC CONFIDENTIAL

Wildcat-S Blade Introduction 29


The ICH7 is the interface to the slower IO devices with these interfaces:

• Two x1 PCIe interfaces connect to the two Ethernet 5751 controllers. The
LAN 0 & LAN 1 Ethernet controllers are connected to the midplane
connector. From there each controller is routed to 1 Management FRU.
• The Low Pin Count (LPC) interface connects the ICH to the Server IO chip
which contains the two RS232 communication ports COM1 & COM2 out to
the midplane connector.
• The Firmware Hub which contains the BIOS & POST code is also
attached to the LPC interface.
• An USB2.0 flash controller interface connects to 256Mbyte of flash used for
the diagnostic code.
• Some of the GPIO pins are connected to a quad UART which are used for
the diplex circuits which allow communication to the DAE’s.

EMC CONFIDENTIAL

30 DREADNOUGHT SP BOARD
Wildcat-S The Wildcat-S Motherboard contains:
Components • Two Quad Core Clovertown 2.33Ghz CPU’s

2
• One Intel Blackford Memory Controller Hub (MCH)
• One Intel 82801 Gigabit IO Controller Hub 7 (ICH7)
• Eight slots of Fully Buffered Direct In-line Memory Modules (FBDIMM) up
to 16Gbytes of System Memory
• One (PLX) PCI Express bridge chip (PEX8524)
• Two 10/100/1000 Ethernet Chips which connect through the midplane
• 2MB Firmware Hub for BIOS and POST
• 256Mbytes of Nand Flash Memory connected through USB to the ICH
• 256K NVRAM
• One 87417 Server IO chip (SIO).
• Control Monitor Device (CMD) will provide on-board DCDC control and
monitoring

Wildcat-S Parts The below diagram shows most of the major parts reference designators
on the Wildcat-S boards. Only front-side parts are shown.
Reference

CM1 1.5V J14


J4 - VRM
Y1
J21
Wildcat-S U8 U7 U2
Y2
U16 LAN1 LAN2 Y4 Y3
205-800-602c U18 U12
PLX J53_1S
ICH
U26 U23 J20
U29 Y5 U28 U27
CM3 1.2V
U39
U49 U35

INTEL
U51 U50
Woodcrest =
Clovertown =
U65 J52_1S
MCH Y8
J22S Y7

U96
INTEL
Woodcrest= J11 – FB DIMM
Clovertown= J12 – FB DIMM J51_1S
J9 – FB DIMM
J10 – FB DIMM
J17

J7 – FB DIMM U2691
U121

U120
J8 – FB DIMM
U133 U131 U128 U125

J5 – FB DIMM
U127

J6 – FB DIMM
U143 U137

J3 - VRM CM2 U141 J U140 U136 J50_1S


1.8V SIO 1 FWH
8

Figure 2-10. Wildcat-S Parts Layout

EMC CONFIDENTIAL

Wildcat-S Blade Introduction 31


Processors The Wildcat-S board is a dual processor motherboard. It uses two Intel
processors, and the Intel Blackford chipset, in a similar fashion to generic
computer motherboards. The package for the Clovertown devices are 771
pin, Flip Chip Land Grid Array Chip (FC-LGA4) packages that use the
LGA771 socket.

Quad Core The Clovertown Quad Core processor has four independent 2.33GHz
CPU cores with each pair of cores (0,1) & (2,3) sharing a 4MB L2 Cache
each. The four cores in each processor share an Intel proprietary FSB
interface to the Blackford MCH.

The CPU uses the FSB interface which consists of 144 signals and
supports up to 1333 million 64-bit transfers per second with a theoretical
bandwidth of 10.66GBytes/sec.

Blackford MCH The Blackford Memory Controller Hub (MCH) is also connected to the
processors FSB’s. It provides switching facilities between a number of
different components and has five distinct interfaces:

• two (FSB) Front Side Bus interface’s to the two processors

• four channel interfaces to the System RAM (FBDIMM’s)

• one Intel proprietary x4 ESI (Enterprise South bridge Interface) to the ICH7

• four 4x PCIe interfaces to the high-speed / high-bandwidth IO Modules.

• one 8x PCIe interface to the Midplane (CMI / Aux)

As its name suggests, the MCH is a hub, a device that connects to several
other devices and routes or directs information between them.

High Speed On a generic computer motherboard, the biggest drain of processing


power and bandwidth is the graphics hardware, so the interface to this
Devices - PCI hardware is normally the fastest. On the Wildcat-S board and similar
Express server motherboards, the high-speed interface is used for moving data,
from the Front-End interface, through cache, to the Back-End interface
and in the reverse direction. The most suitable interface at the time of
design was PCI Express (PCIe), a serial version of PCI.

EMC CONFIDENTIAL

32 DREADNOUGHT SP BOARD
ICH7 IO The other chip in the chipset is the I/O Controller Hub called the ICH7.
This component provides the interface to I/O devices that a computer
Controller Hub would typically need, like keyboard, mouse, RS232 and boot ROM. It also

2
has a PCI interface which is used to add other controllers, such as USB or
Ethernet. All of these interfaces or controllers can be grouped together
effectively because they work in the kilohertz range or slower, and use
very little bandwidth compared to the processors and high-speed devices
on the MCH. The ICH7 on the Wildcat-S is connected to the Blackford
MCH via a 4 lane (ESI) interface.

Server IO The Server IO chip attaches to the Low Pin Count (LPC) bus from the
ICH7. The SIO incorporates the real time clock functions and provides
Controller connectivity to NVRAM and a DUART.

DDR2 RAM The Blackford MCH has 4 * FBDIMM channels and each channel will
accommodate up to two industry standard FBDIMM’s per channel which
FBDIMM’s are connected to the board via a 240 pin FBDIMM connector. Supported
DIMMs can be 1, 2 or 4 Gbytes and can be either single or dual rank using
either x4, or x8 DDR2 SRAM devices.

Midplane The midplane connector is an Gbx type connector. The connector supplies
power from 16 blades on the midplane to the motherboard. It also
Connector transports I2C communication and other signals used by the
motherboard.

IO Module The Wildcat-S motherboard has 4 IO Module connectors. Each connector


can accept 1 (SLIC) Small IO Card. Each connector has an interface from
Connectors the MCH which consists of x4 Lanes of PCIe. The connector also supplies
12VDC to the IO Modules DC-DC circuits which create the required
voltages for the components on each module. An I2C interface is also
present at the connector for communication purposes which include the
resume eeprom.

EMC CONFIDENTIAL

Wildcat-S Blade Introduction 33


EMC CONFIDENTIAL

34 DREADNOUGHT SP BOARD
2.2 Intel Clovertown CPU

Introduction The Wildcat-S motherboard supports two Intel Xeon 5300-series


Clovertown processors. The processors will be packaged in an FC-LGA6
Land Grid Array package with 771 pins. This utilizes a surface mount
2
LGA771 socket.

The Clovertown is a quad-core 64-bit server/workstation processor


utilizing four Intel micro architecture cores. These processors are based
on Intel’s 65 nanometer process technology for power efficiency. The
Clovertown package consists of two die, each containing two processor
cores

Intel Clovertown CPU Intel Clovertown CPU


U96 U49

CPU CPU CPU CPU CPU CPU CPU CPU


Core 0 Core 1 Core 2 Core 3 Core 4 Core 5 Core 6 Core 7

L1 Cache L1 Cache L1 Cache L1 Cache L1 Cache L1 Cache L1 Cache L1 Cache


32 KB 32 KB 32 KB 32 KB 32 KB 32 KB 32 KB 32 KB

L2 Cache L2 Cache L2 Cache L2 Cache


4 MB 4 MB 4 MB 4 MB

FSB0 Intel FSB1


333 MHz 333 MHz
Blackford
MCH

ESI 2.5 GHz

Intel LPC
FWH
ICH7 33 MHz
clovertown_fsb_block.emf

Figure 2-11. Clovertown Front Side Bus

Features • Core-speed up to 2.33 GHz

• 64-bit capable

• 1333 MHZ Front-Side Bus (quad-pumped) yields 6.4 GB/s

• 8 MB L2 Cache (4 MB/ die)

• Extended Instruction Set (MMX2, SSE2)

• Deeper Pipeline depths

EMC CONFIDENTIAL

Intel Clovertown CPU 35


• Speed-Step technology - power management

• Thermal monitoring

Operation The Front Side Bus (FSB) is a 1333 MHz quad-pumped bus running off a
333 MHz system clock, which results in 10.6 GBytes/s data transfer. The
processor transfers data four times per bus clock. Along with the 4X data
bus, the address bus can deliver addresses two times per bus clock. In
addition, the Request Phase completes in one clock cycle. The FSB is also
used to deliver interrupts to the processor.

Signals on the FSB use Assisted Gunning Transceiver Logic (AGTL+) level
voltages.The AGTL+ inputs require reference voltages (GTLREF_DATA_MID,
GTLREF_DATA_END, GTLREF_ADD_MID and GTLREF_ADD_END)
which are used by the receivers to determine if a signal is a logical 0 or a
logical 1.Termination resistors (RTT) for AGTL+ signals are provided on the
processor silicon and are always enabled on the processor to control
reflections on the transmission line.

The Clovertown is the latest of the x86 family of Intel CPU’s. While
offering a complex and high performance interface to the supporting
chipset and peripheral devices, from a software or Emulator point of
view it has many of the same methods of access to those supporting
devices, namely Memory Read/Write and I/O Read/Write cycles.

In older x86 architectures a different strobe signal was asserted for


Memory or I/O cycles to latch the address in the chipset. In the
Clovertown, FSB_ADSTB_N0 and FSB_ADSTB_N1 are asserted along
with the 36-bit address and FSB_HREQ_N<4:0>. The type of cycle is
encoded in the FSB_HREQ_N<4:0> signals and is decoded by the MCH to
be passed on to the target device or memory.

FIRST TRANSACTION SECOND TRANSACTION


CPU#_BCLK0
CPU#_BCLK1 ADDRESS & TYPE ATTRIBUTES & BYTE ENABLES

FSB_A_N<35:17> A B

FSB_A_N<16:3>
FSB_HREQ_N<4:0> A B

FSB_ADSTB_N0

FSB_ADSTB_N1

FSB_TRDY_N

FSB_DBSY_N

FSB_DRDY_N

DATA (FIRST TRANSACTION ) DATA (SECOND TRANSACTION )


FSB_HD_N<63:0>
FSB_DSTBP_N<3:0>
FSB_DSTBN_N<3:0>
fsb_operation.emf

Figure 2-12. Front Side Bus Signal Timing

EMC CONFIDENTIAL

36
Memory R/W In the normal course of events, the CPU needs to access either some
memory device (DDR SDRAM, Boot Flash, NVRAM) or some peripheral
device (Ethernet controller, FC Controller, COM port, PCI-E Bridge etc.)

2
attached to the MCH or ICH. The primary way of the CPU accessing a
device is by a Memory Read or Memory Write cycle, which is usually
initiated by a MOV instruction.

This means a device’s registers are mapped, or assigned, an address or


range of addresses in the CPU memory space. The system memory
(except the Boot Flash) is also assigned address ranges where it can be
accessed. The size of the space seen by the CPU is limited by the width of
the address bus, and in the case of the Intel Clovertown, 36 address bits
giving a maximum of 64 GBytes of physically addressable memory space.

I/O R/W Some devices or functions are not mapped to a memory address but to
another space that the CPU can access, called I/O space. I/O space is
limited to 64 KBytes, since an I/O access only uses the low 16 address bits.
An I/O address is sometimes referred to as an I/O Port. I/O accesses are
initiated by IN or OUT instructions

I/O cycles are used to generate Configuration Cycles during POST in


order to assign the addresses of the memory and peripheral devices. see
“I/O Access Registers” on page 46 to see this mechanism detailed.

Table 2-3. Address Map of Wildcat-S Motherboard


I/O Config Address
Device Name Location Memory Address Address Bus : Dev : Func

BIOS/POST FW Hub 000E 0000 - 000F FFFF - -


NVRAM NVRAM FFD8 0000 - FFD9 FFFF - -

Main Memory (DDR SDRAM) DIMM 0000 0000 - FFFF FFFF* - -

SMBus Controller ICH - 0x1380 00 : 1F : 03


PLX Bridge PLX FC00 000 0x2000 10 : 00: 00

CMI (Peer Memory space - FBDIMM) PLX 8000 000 - BFFF FFFF - -

LPC Interface ICH - - 00 : 1F : 00

LAN1 BCM CC10 0000 - 4E : 00 : 00

LAN2 BCM CC20 0000 - 4F : 00 : 00

PCI-E Port A Interface MCH F000 0000 - 02 : 00 : 00

PCI-E Port B Interface MCH D000 0000 - 04 : 00 : 00

PCI-E Port B1 Interface MCH D600 0000 - 05 : 00 : 00

PCI-E Port C Interface MCH C000 0000 - 06 : 00 : 00

PCI-E Port C1 Interface MCH C600 0000 - 07 : 00 : 00

PCI-E Port B (PCI-E Interface - I/O 0) SLIC 0 D000 0000 - 26 : 00 : 0X

PCI-E Port B1 (PCI-E Interface - I/O 1) SLIC 1 D600 0000 - 30 : 00 : 0X

PCI-E Port C (PCI-E Interface - I/O 2) SLIC 2 C000 0000 - 3A : 00 : 0X

EMC CONFIDENTIAL

Intel Clovertown CPU 37


Table 2-3. Address Map of Wildcat-S Motherboard
I/O Config Address
Device Name Location Memory Address Address Bus : Dev : Func

PCI-E Port C1 (PCI-E Interface - I/O 3) SLIC 3 C600 0000 - 44 : 00 : 0X

PCI-E Port A (I/O 4) Annex F000 0000 - 12 : 00 : 00

PCI-E Port A (I/O 5) Annex F600 0000 - 1C : 00 : 00

CPU Boot A number of events must take place for the CPU to begin executing code
Sequence to bring the system up. To assist troubleshooting a dead Wildcat-S, the
basic sequence is provided here.

Power:

• When the chassis is connected to AC power the main 12V and


Standby 5V comes from the Midplane. This powers the MC and CMD
on the Wildcat-S for powering up the rest of the Wildcat components.

• The 1.05V and 3.3V regulators are enabled by the MC for the CPU’s
(3.3) and MCH (1.05 and 3.3). This also begins powering the CK410
Clock generator.

• The 1.5V regulator is enabled by the MC for the CPU’s and MCH.

• VTT_CPU (1.2V) is enabled by the CMD.

• The processors drive the VID<6:0> lines to the VRM’s to specify the
exact voltage they require.

• The VRM’s for each processor are enabled by the MC to output


VCC_FSB# for each processor.

• The MC asserts CPU_PWRGD to each processor once all the voltages


have been enabled and are stable.

CPU Reset:

• Once the power is present the MCH de-asserts FSB#_RESET_N to the


CPU

• The CPU performs its internal self check and initializes it’s registers.

• The CPU outputs FFFF FFF0h on the FSB to the MCH.

• The MCH forwards the request to the ICH (over the ESI link)

• The ICH generates a Read cycle on the LPC bus to the FWH and
fetches the Boot vector to pass back up to the FSB.

• The Boot vector is used by the CPU to begin executing the BIOS code,
typically address FFF0 0000h.

EMC CONFIDENTIAL

38
The signals used by the processor are shown in Figure 2-13s

VTT _CPU
[22..1]
VCC_Core
[181..1]
VCC_FSB# VSSA FSB#_VCCPLL
2
FSB#_A_N[35:3]
FSB#_AP_N[1:0] PUP_FSB#_BSEL[2:0]
FSB#_ASDTB_N[1:0] CPU#_BCLK+/-
FSB#_ADS_N FSB_INIT_N Execution
FSB#_BR[1:0]_N FSB_STPCLK_N Control
FSB#_REQ_N[4:0] ICH_CPUSLP_N Signals
Address Signals

FSB#_RS_N[2:0]
FSB_INTR
FSB#_RSP_N FSB_NMI
FSB#_DEFER_N
FSB#_IERR _N
FSB#_HIT_N
FSB#_HITM_N
Front Side Busses

FSB#_LOCK_N
FSB#_BINIT_N FSB_FERR_N
FSB#_BNR_N FSB_IGNNE_N PC
FSB#_BPRI_N FSB_A20M_N Compatibility
Signals
FSB#_MCERR_N FSB_SMI_N
FSB#_RESET_N

FSB#_HD_N[63:0] Intel
FSB#_DP_N[3:0]
Clovertown
Data Signals

FSB#_DBI_N[3:0] FSB#_SKTOCC_N
FSB#_DSTBP_N[3:0] FSB#_VID_SEL
FSB#_DSTBN_N[3:0]
# = 0(U96) , 1(U49)
FSB#_VID [5..0]
FSB#_TRDY_N POWER
CPU_PWRGD
FSB#_DRDY_N CONTROL
FSB#_VCC_SENS
SIGNALS
FSB#_DBSY_N FSB#_VSS_SENS

FSB#_GTLREF_ADD_R

FSB#_GTLREF_DATA_R

GTL+ FSB#_GTLREF_ADD FSB#_BPMB[5.. 0]_N


FSB#_GTLREF_DATA FSB#_BPM_N[5..0]
CPU#_COMP[3..0] XDP_TCK
JTAG/Test
FSB#_TDI
Signals
FSB#_PROCHOT_N FSB#_TDO

CPU#_THERMDA XDP_TMS_MAIN
Thermal CPU#_THERMDC XDP_TRST_N

FSB#_THERMTRIP_R XDP_DBRST_N

clovertown_pinout.vsd

Vss (Gnd)

Figure 2-13. Clovertown CPU Pinout

EMC CONFIDENTIAL

Intel Clovertown CPU 39


EMC CONFIDENTIAL

40
2.3 Blackford MCH

Overview The Blackford Memory Controller Hub (MCH) is a 1432-pin BGA which
functions as the "bridge" for the processor to the rest of the system. The
2
MCH provides interfaces to:

• 2 Clovertown processors, using independent Front Side Busses (FSB)

• System RAM (FB-DIMM), using 4 memory channels

• Hi-speed devices using PCI express

• ICH 7 using the ESI (Enterprise South bridge Interface)

The MCH supports 36-bit host addressing, allowing up to 64 GB of


physical memory. CPU I/O cycles are decoded to PCI Express, ESI
interface or MCH configuration registers. CPU memory cycles are
decoded to PCI Express, ESI or system memory. Processor interrupts are
sent over the FSB’s.

to CPU’s
64 64

CPU I/F
FSB 0 Bus: 0 Dev: 16
FSB 1
Fnc: 0,1,2

FB DIMM Ch0
14 MCH I/O MCH Error
Registers Registers
Bus:0 Dev:22 Fnc:0

10 CH 0 Bus:0 Dev:19 Fnc:0


FBD Branch 0

4 8
PCI Bus 0

PCIe Port 2
FB DIMM Ch1 4 8
14 Bus: 0 Dev: 2 Fnc:0
Memory Controller

4 to PLX
10 CH 1 PCIe Port 3
4
Bus: 0 Dev: 3 Fnc:0

4
FB DIMM Ch2 PCIe Port 4
14 4 to SLIC 0
Bus: 0 Dev: 4 Fnc:0
Bus:0 Dev:21 Fnc:0
FBD Branch 1

10 CH 0 4
PCIe Port 5 to SLIC 1
4
Bus: 0 Dev: 5 Fnc:0
FB DIMM Ch3
14 4
CH 1 PCIe Port 6 to SLIC 2
10 4
Bus: 0 Dev: 6 Fnc:0
4
PCIe Port 7 to SLIC 3
I2C to 4
DIMM’s Bus: 0 Dev: 7 Fnc:0

DMA Controller
SMBus Bus: 0 Dev: 8 Fnc:0
Controller
Blackford
ESI MCH
Bus: 0 Dev: 0 Fnc:0

4 4
blackford_block.emf to ICH7

Figure 2-14. Blackford MCH Block Diagram

EMC CONFIDENTIAL

Blackford MCH 41
Blackford also contains DMA engines and numerous registers for I/O
access and error reporting. The MCH internal modules and interfaces
appear to the processors as several devices attached to PCI bus 0,
although physically not all are actual PCI devices and the physical
internal bus is not an actual PCI bus. Also, because the ICH7 is connected
to the ESI link, all the ICH internal modules are accessed as if they are on
PCI bus 0.

Accesses to non operational or non-existent devices, internal or external


to the MCH or ICH, are master aborted. This means that writes are
dropped and reads return all 1’s.

Front-Side Bus The Blackford MCH supports two Woodcrest or Clovertown processors,
with an 1333 MHz front side bus (FSB). Each FSB interface consists of 144
signals. The bus clock is 333 MHz, the address and request interface is
double pumped to 667 MHz. The 64 bit data bus is quad pumped to 1333
MHz. This provides a FSB bandwidth of 10.7 GB/s.

The Blackford MCH is responsible for decoding the memory address


range for the other devices on the Wildcat. During power-up and reset,
the CPUs are the last devices to come out of reset. The Blackford MCH
holds the CPU in reset until all the other devices have powered-up
properly, and drives the lines used to configure the CPUs during
power-up.

The CPU’s have the capability to arbitrate for the FSB between themselves
by asserting the BREQ0# signal to each other. The MCH has priority
access and gains ownership of the FSB by asserting BPRI# to the CPU’s.

System RAM System RAM consists of eight Fully Buffered DIMMs (FBDIMM’s) in
eight sockets. Total supported memory is up to 32 GBytes of DDR2-667
SDRAM

Blackford masters four Fully Buffered DIMM (FBD) memory channels.


Wildcat will support up to two DIMMs connected to each FBD channel
for a total of eight DIMMs.

The four FBD channels are organized into two branches with two
channels on each branch. Each branch is supported by a separate memory
controller. The two channels in each branch operate in lock step to
increase FBD bandwidth. A branch transfers 16 bytes of payload per
frame on Southbound lanes (Blackford to FBD) and 32 bytes of payload
per frame on Northbound lanes (FBD to Blackford).

Two branches may be operated in mirrored (RAID 1) or non-mirrored


mode. Wildcat will have mirrored mode disabled.

Key features of the MCH FBD memory interface on Wildcat-S are:

EMC CONFIDENTIAL

42
• Four Fully Buffered DDR memory channels

• Branch channels are paired together in lock step for bandwidth


requirements


Supports up to eight dual-ranked FB DIMMs for a total of 32GBytes

The FBD link speed is 6x the speed of the DDR data transfer speed. A
4Gb/s FBD link speed supports DDR2-667 (FSB @ 1333MT/s)
2
• All memory devices are DDR2

• ECC Data protection for correction of any x4 or x8 DRAM device


failure

• Detection of all two wire faults on the DIMMs. Includes any pair of
single bit errors.

• Detection of all permutations of two x4 DRAM failures

Blackford employs a single device data correction (SDDC) algorithm for


the memory subsystem that will recover from any single x8 or x4 DRAM
device failure as well as detect any dual x4 device failure. The chip
disable is a 32-byte two phase code. The MCH also supports demand and
scrubbing.

A scrub corrects a correctable error in memory. A four byte ECC is


attached to each 32 byte payload. An error is detected when the ECC
calculated from the payload does not match the ECC read from the
memory. The error is corrected by modifying the ECC, the payload, or
both and writing both the ECC and the payload back into memory.

FBD Memory Before any transfers to/from FBD can be supported, the MCH DRAM
Configuration registers must be initialized. Detection of memory type and size is
accomplished via the four System Management Bus interfaces on the
MCH (SMBus 1-4). SMBus is basically Intel's version of I2C. SMBus 1-4
are used for FBD channels 0-3 to extract the DRAM type and size
information from the Serial Presence and Detect Port on the DIMMs.

FB DIMMs contain a Serial Presence Detect (SPD) Port that consists of a


serial data (SDA), serial clock (SCL) and a 4 bit address field (SA [3:0]).
Blackford integrates a 100 KHz SPD to access the DIMM SPD EEPROM.

The FB-DIMM technology is detailed in chapter 2.

Hi-Speed I/F - Blackford has six 4-lane (x4) PCI Express (PCIe) ports. Each pair of x4
PCIe ports can be configured as an 8-lane (x8) PCIe port. PCIe ports are
PCI Express numbered 2-7. Ports 2 and 3 will be paired together as a single x8 port.
Ports 4, 5, 6, and 7 are connected to IO Modules as x4 ports.

As can be seen in Figure 2-14, PCIe ports 2 and 3 will be connected to a


PLX PEX8524 as a single x8 PCIe interface, PCIe ports 4, 5, 6 and 7 will be
connected to IO Module 0, 1, 2 and 4 as single x4 PCIe interfaces.

EMC CONFIDENTIAL

Blackford MCH 43
Wildcat-S will also support one or two dual wide IO Modules by pairing
ports 4 with 5 and 6 with 7.

PCIe ports 2 and 3 have a four channel DMA engine. This DMA engine
allows transfers from one device connected to Blackford to another (such
as FBD to FBD, FBD to IO).

Blackford PCIe ports will support widths of x8 (when paired together),


x4, x2, and x1 but during training will always attempt to operate at the
largest width it is configured for (based on MCH_PEWIDTH<3:0>) as
shown in Table 3. Maximum payload is 128 bytes and each port only
supports one virtual channel.

Loopback mode is supported for testability, as well as Lane Reversal and


polarity Inversion.

Table 3. MCH PCIe Port Configuration

MCH_PEWIDTH<3:0> Port 0 Port 2 Port 3 Port 4 Port 5 Port 6 Port 7


(ESI)
0000 x4 x4 x4 x4 x4 x4 x4
0001 x4 x4 x4 x4 x4 x8
0010 x4 x4 x4 x8 x4 x4
0011 x4 x4 x4 x8 x8
other reserved
1000 x4 x8 x4 x4 x4 x4
1001 x4 x8 x4 x4 x8
1010 x4 x8 x8 x4 x4
1011 x4 x8 x8 x8
other reserved
1111 x4 Determined by link negotiation

Enterprise The Enterprise South Bridge Interface (ESI) is the chip to chip interface
between the MCH and the ICH7. It is a four lane (x4) extension to the
Southbridge standard PCI Express specification with special commands/features
Interface (ESI) added. The ESI consists of two signal groups, inbound and outbound,
each having 4 differential signal pairs clocked at 2.5 GHz.

All CPU accesses that do not map to either the System RAM or one of the
PCIe interfaces (PLX or SLIC’s) will be directed to the ESI and
subsequently to the ICH. This would include processor boot firmware,
Ethernet and RS232 communications, and any other "downstream" I/O
devices.

The ESI is considered part of PCI bus 0, so functions within the ICH
which is attached to the ESI appear in the PCI configuration space on PCI
bus 0, along with the MCH internal functions, even though they are two
physical devices.

EMC CONFIDENTIAL

44
MCH Registers The Blackford MCH has a number of functions or devices which require
configuration before the system can be operational. This is done during
system boot through the PCI Configuration registers for each device.

2
Error information for each device is recorded in various status registers
which are also configured during system boot by the firmware. Access to
these configuration registers is done by accessing the I/O Address and
Data registers in the MCH. The main registers used in the system are
detailed below.

PCI Configuration All functions or devices within MCH appear as PCI devices and therefore
Registers have some common registers for configuration, as well as function-
specific ones.

PCI devices are assigned a particular Bus Number (0-255), Device


Number (0-31), and Function number (0-7). Most modules within both
MCH and ICH are considered to be on PCI Bus 0 even though they exist
in two separate physical packages, since the HubLink I/F is transparent to
any configuration or memory accesses.

One major purpose of the configuration registers is to map (assign a


memory address to) the device so the CPU’s can use normal memory R/W
cycles to access the device. This is done by performing a Configuration
Write to the Base Address Registers (BAR) in the PCI Configuration
space at offset 0x10 to 0x24. The generic PCI Configuration Registers are
shown in Figure 2-15.

Address Required fields


00 Offset
00 Device ID Vendor ID
04 Status Command
Header
08 Class Code Rev ID
3C 0C BIST Header Typ Latency Cache Line
40
10 Base Address 0
14 Base Address 1
18 Base Address 2
1C Base Address 3
20 Base Address 4
Device 24 Base Address 5
Specific
28 Cardbus CIS Pointer
2C Subsystem ID Command
30 Expansion ROM Base Address
34 Reserved Capabilities
38 Reserved
FF 3C Max_Lat Min_Gnt Int_Pin Int_Line
mch_pci_cfgspace.emf

Figure 2-15. Generic PCI Configuration Register Format

EMC CONFIDENTIAL

Blackford MCH 45
I/O Access The Wildcat motherboard is designed to use the PCI mapping provided
Registers largely through the MCH and ICH to access most of the devices on board.
This requires support for PCI Memory R/W, PCI I/O R/W, and PCI
Configuration R/W cycles. However, the Pentium CPU can only generate
Memory and I/O cycles on its Front-Side Bus, so a way of generating
Configuration cycles is needed. The CPU uses I/O R/W cycles to a specific
I/O port address to accomplish this.

Configuration cycles are generated by the CPU performing an I/O Write


of bit 31 = 1, the PCI bus number, Device number, Function number and
register offset to the CONFIG_ADDRESS register (I/O address 0xCF8),
then performing an I/O Read or Write to the CONFIG_DATA register
(I/O address 0xCFC) with the data. The MCH or ICH will then generate a
PCI Configuration Read or Write bus cycle to the specified device. The
format of these registers is shown in Figure 2-16.

Figure 2-16. PCI Configuration Registers

EMC CONFIDENTIAL

46
FSB #_A_N[35:3] EXP _FRM_MCHA_[7:0]_C+

FSB#_AP_N[1:0] EXP _FRM_MCHA_[7:0]_C-


PLX

2
FSB#_ASDTB_N[1:0] EXP _TO_MCHA_[7:0]
FSB#_ADS_N EXP _TO_MCHA_[7:0]
A U29,A P2
FSB#_BR[1:0]_N
FSB #_REQ_N[4:0]
Blackford MCH EXP _FRM_MCHB_[7:0]_C+

U65 EXP _FRM_MCHB_[7:0]_C-


Address Signals

FSB #_RS_N[2:0] SLIC


EXP _TO_MCHB_[7:0]
FSB#_RSP_N
A N27,A K2
0 &1
EXP _TO_MCHB_[7:0]
FSB#_DEFER_N

PCIe Bus
A V 34,A J9

FSB#_HIT_N A U32,A K8 EXP _FRM_MCHC_[7:0]_C+


FSB #_HITM_N
Front Side Busses

A V 33,A J7 EXP _FRM_MCHC_[7:0]_C- SLIC


FSB#_LOCK_N
A T30, A L4 EXP _TO_MCHC_[7:0] 2 &3
FSB #_BINIT_N
A K 27,A J4 EXP _TO_MCHC_[7:0]
FSB #_BNR_N
A V 30,A K3
FSB#_BPRI_N
# = 0, 1 PUP_EXPHPINTER _N V3_3

FSB #_MCERR_N
A U34,A J 10
% = 0, 1, 2, 3 E6
MCH_PE_COMP V1_5
A J27,A H11 R12,P 12
FSB#_RESET _N MCH_100MHZ_CLK+
A N30,A E11 J2
MCH_100MHZ_CLK-
K2
FSB #_HD_N[63:0] V1_5
W10,W11, Y12, A A11
MCH_PEWIDTH [3:0]
FSB #_DP_N[3:0]
K1
MCH_PE_VCCA V1_5
Data Signals

FSB #_DBI_N[3:0]
MCH_PE_VSSA GND
L1
FSB#_DSTBP_N[3:0]

FSB#_DSTBN_N[3:0]
FBD_CH%_SB_[9:0]+
FSB#_TRDY_N
A T32, A K6
FBD_CH%_SB_[9:0]-
FSB#_DRDY_N
A T29, A M3
FBD_CH%_NB_[13:0]+
FSB #_DBSY_N
A R30,A M4
FBD_CH%_NB_[13:0]-

FBD_BRANCH#_CLK+
A A6
MCH_BCLK+
A N17 FBD_BRANCH#_CLK- FB-DIMMs
333 MHz W8
MCH_BCLK-
A P 17
CH 0,1,2,3
MCH_PSEL [2:0] FBD_BRANCH#_VCCA V1_5
Y 10
100b A B 1,A B2, A C1

VREF _MCH_FSB#_DATA FBD_BRANCH#_VSSA GND


A M3
800 mV
MCH_FSB_CRES
A T35
MCH_FSB_SLWCRES MCH_SPD %_SDA
A U35
MCH_FSB _ODTCRES MCH_SPD %_SCL
A R34
MCH_FSB _SLWCTRL
V1_5 A V 13

HP_SMB_SDA
L12
ESI _FRM_MCH_[3:0]_C+
HP_SMB_SCL
SMBus K 13
A K9,A G8, A H6,A H7

ESI _FRM_MCH_[3:0]_C-
I2C MEM_SDA
J 14
A L8, A E8, A G6, A J6 ESI Link
ESI _TO_MCH_[3:0]
MEM_SCL K 14
A K9,A G8, A H6,A H7 to ICH
ESI _TO_MCH_[3:0]
A L8, A E8, A G6, A J6

1.5V MCH_RST_N
MCH_TRST_LV_N G17
A8 V1_5
MCH_PWROK
XDP_TMS_MAIN MCH_CORE_VCCA H17
A7
XDP_TCK
MCH_FSB_VCCA
F35
MCH_FBD_ICOMPBIAS Misc.
MCH_CORE_VSSA
JTAG MCH_TDI
A6
VTT_CPU E 36
MCH_FBD_RESIN Signals
B7
MCH_FBD_RGBIASEXT
MCH_TDO E 37
B6 3.3V
V3REF F13
Blackford _pinout.emf

Figure 2-17. Blackford MCH Signals


EMC CONFIDENTIAL

Blackford MCH 47
.

Table 2-1. MCH Signal Definitions


SCHEMATIC SIGNAL PIN I/O DESCRIPTION

FRONT SIDE BUS SIGNALS

FSB#_A_N<35:3> I/O Processor address bus.

FSB#_AP_N<1:0> I/O Parity for processor address bus.

FSB#_ADSTB_N<1:0> I/O Strobes to qualify FSB_A<35:3> and FSB_HREQ<4:0> on rising edge

FSB#_ADS_N AU29/AP2 I/O Address strobe to indicate the beginning of a bus cycle

FSB#_BREQ0_N I/O Bus Request for ownership of the FSB

FSB#_BREQ1_N In Request from other agent for ownership of FSB

FSB#_HREQ_N<4:0> I/O Request lines to define the current bus cycle type: Memory R/W, I/O
R/W, Deferred Reply, etc.

FSB#_RS_N<2:0> Out Response code indicating Retry, Deferred Response, Normal Data, etc.

FSB#_RSP_N AN27/AK2 Out Parity for the Response code lines

FSB#_DEFER_N AV34/AJ9 Out Indicates a transaction will be accepted but not guaranteed to complete
in order. All I/O and PCI Config accesses are automatically Deferred

FSB#_HIT_N AU32/AK8 I/O Indicates a cache in the system contains data requested

FSB#_HITM_N AV33/AJ7 I/O Indicates a cache in the system has modified the data requested
FSB#_LOCK_N AT30/AL4 In Asserted by an agent during a Read-Modify-Write cycle

FSB#_BINIT_N AK27/AJ4 In Asserted by an agent to indicate that the FSB is not usable and will
re-initialize

FSB#_BNR_N AV30/AK3 I/O Asserted to Block the Next Request to stall the FSB until an agent is
available to handle more transactions

FSB#_BPRI_N AU34/AJ10 Out Asserted by MCH as the Bus Priority Request Interrupt

FSB#_MCERR_N AJ27/AH11 I/O Machine Check Error indicates an unrecoverable FSB fault
FSB#_RESET_N AN30/AE11 Out Hard Reset. Must be asserted for 1 ms after VCC and BCLK are stable

FSB#_HD_N<63:0> I/O FSB Data bus

FSB#_DP_N<3:0> I/O Parity for FSB Data bus

FSB#_DBI_N<3:0> I/O Data Bus invert, when low, indicates that the 16-bit portion of the data
bus is to be inverted (LO = 0, HI = 1)

FSB#_DSTBP_N<3:0> I/O Data Strobe Positive to latch a 16-bit portion of the Data bus

FSB#_DSTBN_N<3:0> I/O Data Strobe Negative to latch a 16-bit portion of the Data bus

FSB#_TRDY_N AT32/AK6 Out Target Ready asserted to indicate Write data may be sent by processor

FSB#_DRDY_N AT29/AM3 I/O Data Ready asserted by agent driving data on the FSB

FSB#_DBSY_N AR30/AM4 I/O Data Bus Busy asserted by agent driving data on the FSB

MCH_BCLK+/- AN17/AP17 In Differential Bus Clock for FSB (333 MHz)

ESI Link

ESI_FRM_MCH_<3:0>_C+ I/O PCIe transmit lines to ICH7

ESI_FRM_MCH_<3:0>_C+ E31 I/O PCIe transmit lines to ICH7

ESI_TO_MCH_<3:0>+ D32 I/O PCIe receive lines from ICH7

ESI_TO_MCH_<3:0>- K25 I/O PCIe receive lines from ICH7

PCIe x8 BRIDGE TO PLX SIGNALS


EXP_FRM_MCHA_<7:0>_C+ Out PCIe transmit lines to PLX bridge

EMC CONFIDENTIAL

48
Table 2-1. MCH Signal Definitions
SCHEMATIC SIGNAL PIN I/O DESCRIPTION

EXP_FRM_MCHA_<7:0>_C- Out PCIe transmit lines to PLX bridge

EXP_TO_MCHA_<7:0>+

EXP_TO_MCHA_<7:0>-

PCIe x8 BRIDGE TO SLIC 0 & 1 SIGNALS

EXP_FRM_MCHB_<7:0>_C+
In

In

Out
PCIe receive lines from PLX bridge

PCIe receive lines from PLX bridge

PCIe transmit lines to SLIC 0 & 1


2
EXP_FRM_MCHB_<7:0>_C- Out PCIe transmit lines to SLIC 0 & 1

EXP_TO_MCHB_<7:0>+ In PCIe receive lines from SLIC 0 & 1

EXP_TO_MCHB_<7:0>- In PCIe receive lines from SLIC 0 & 1

PCIe x8 BRIDGE TO SLIC 2 & 3 SIGNALS

EXP_FRM_MCHC_<7:0>_C+ Out PCIe transmit lines to SLIC 2 & 3

EXP_FRM_MCHC_<7:0>_C- Out PCIe transmit lines to SLIC 2 & 3

EXP_TO_MCHC_<7:0>+ In PCIe receive lines from SLIC 2 & 3

EXP_TO_MCHC_<7:0>- In PCIe receive lines from SLIC 2 & 3

MISC PCI

PUP_EXPHPNTER_N E6 In Hot-Swap Interrupt input

MCH_PE_COMP R12,P12 In PCIe Compensation resistor

MCH_100MHZ_CLK+/- J2,K2 In PCIe clock (100 MHz)

MCH_PEWIDTH<3:0> W10,W11, In PCIe Port configuration straps


Y12,AA11
MCH_PE_VCCA K1 1.5 V Supply for PCIe ports

MCH_PE_VSSA L1 Ground for PCIe ports

FBDIMM
FBD_CH%_SB<9:0>+ Out Data & Address to FBDIMM (Southbound)

FBD_CH%_SB<9:0>- Out Data & Address to FBDIMM (Southbound)

FBD_CH%_NB<13:0>+ In Data & Address from FBDIMM (Northbound)


FBD_CH%_NB<13:0>- In Data & Address from FBDIMM (Northbound)

FBD_BRANCH#_CLK+ In Clock for Memory Branch #

FBD_BRANCH#_CLK- In Clock for Memory Branch #

FBD_BRANCH#_VCCA 1.5 V Supply for Memory Branch #

FBD_BRANCH#_VSSA Ground for Memory Branch #

SMBUS / I2C SIGNALS

MEM_SDA J14 I/O Serial data for Blackford slave SMBus

MEM_SCL K14 I/O Serial clock for Blackford slave SMBus

HP_SMB_SDA L12 I/O Serial data for Hot Plug I2C bus to detect SLIC & Annex insertion and
removal

HP_SMB_SCL K13 I/O Serial clock for Hot Plug I2C bus to detect SLIC & Annex insertion and
removal

MCH_SPD%_SDA I/O Serial data for Serial presence Detect I2C bus to FBDIMM channel %

MCH_SPD%_SCL I/O Serial clock for Serial presence Detect I2C bus to FBDIMM channel %

MISC SIGNALS

MCH_RST_N C2 In Reset for all internal MCH registers except for "sticky" bits

EMC CONFIDENTIAL

Blackford MCH 49
Table 2-1. MCH Signal Definitions
SCHEMATIC SIGNAL PIN I/O DESCRIPTION

MCH_PWROK E3 In Reset for all internal MCH registers including "sticky" bits

MCH_FBD_ICOMPBIAS F35 Out FB-DIMM transmitter swing bias

MCH_FBD_RESIN E36 In FB-DIMM on-die impedance compensation

MCH_FBD_BGBIASEXT L12 In FB-DIMM Bypass bias input for band pass gap circuit

JTAG

MCH_TRST_LV_N J9 Out JTAG TAP reset

XDP_TMS_MAIN F3 In JTAG TAP mode select


XDP_TCK D2 In JTAG serial clock

MCH_TDI G5 In JTAG serial data in

MCH_TDO G6 JTAG serial data out

ANALOG SIGNALS

VREF_MCH_FSB#_DATA In Address/control signal Voltage Reference input (800 mV typical)

MCH_FSB_CRES AT35 In Common return for bus compensation resistors

MCH_FSB_SLWCRES AU35 In Compensation Resistor for bus Slew Rate

MCH_FSB_ODTCRES AR34 In Compensation Resistor for On-Die Termination

MCH_FSB_SLWCTRL AV13 Processor slew rate control

MCH_PSEL<3:0> Processor speed select

EMC CONFIDENTIAL

50
2.4 FB DIMMS
This Chapter will examine the (FBDIMM) Fully Buffered Dual In-line
Memory Module which is used on the Wildcat-S motherboard.
2

Figure 2-18. FBDIMM photo Samsung


Supported FBDIMM’s are 1GByte, 2GByte, or 4Gbyte and can be either
single rank or dual rank using either x4 or x8 DDR2 SDRAM devices. The
FBDIMM speed grade is 667MHz DDR2 devices.

Table 2-2. FB DIMM Wildcat-S

Description Pins FB DIMM Size Number System Size


FB DIMM / With 667 Mhz DDR2 SRAM 240 1GByte 8 8GByte
FB DIMM / With 667 Mhz DDR2 SRAM 240 2GByte 8 16GByte
FB DIMM / With 667 Mhz DDR2 SRAM 240 4GByte 8 32GByte

The mother board has eight 240 pin FB DIMM connectors similar to
Figure 2-19 which accept 1 FBDIMM module each.

Figure 2-19. FBDIMM Connector


The FBDIMM modules have two sided gold fingers. One side contains
pins 1-120 and the second side has pins 121-240 which are used to carry
signals.

EMC CONFIDENTIAL

FB DIMMS 51
Memory Blackford MCH masters four Fully Buffered DIMM (FBD) memory
channels seen in Figure 2-20 below. Wildcat-S will support up to two
Controller DIMMs connected to each FBD channel for a total of eight DIMMs.

Wildcat-S
205-800-602c
U8 U7
U16 LAN1 LAN2

PLX U18 J53_1S


ICH IO Card 3

U49
INTEL CPU 1
Woodcrest= FSB
Clovertown=
U65 J52_1S
FSB MCH IO Card 2

J22

4Gb/sec each
U96 CH0 CH2
INTEL CPU 0 CH1 CH3
Woodcrest = J11 – FB DIMM
Clovertown= J12 – FB DIMM J51_1S
J9 – FB DIMM IO Card 1

J10 – FB DIMM
J7 – FB DIMM
U120
J8 – FB DIMM
J5 – FB DIMM
J6 – FB DIMM

J50_1S
U141 U140 IO Card 0
SIO FWH U149
NVRAM

Figure 2-20. Wildcat-S FBDIMM Interface

EMC CONFIDENTIAL

52
FBDIMM Intro The Wildcat-S Motherboard has slots for 8 FBDIMM’s. Figure 2-21 below
illustrates how each channel has 2 FBDIMM with the first FBDIMM
connected to the MCH and the second FBDIMM daisy chained to the

2
first. There are 4 channels for at total of 8 FBDIMM’s. The clock signals are
also shown here. The last group of signals are the 4 I2C or (SMB)
interfaces which originate from the MCH who is the master.

DRAM DRAM
DRAM DRAM
DRAM DRAM

14 Diff Pairs DRAM DRAM


14 Diff Pairs
Channel 0 10 Diff Pairs AMB 10 Diff Pairs AMB

2 DRAM 2 DRAM 2
Blackford
s

u
DRAM DRAM
Memory
Control DRAM DRAM

Hub DRAM DRAM


MCH_SPD0_SMB
SPD SMBus 0

DRAM DRAM
DRAM DRAM
DRAM DRAM

14 Diff Pairs DRAM 14 Diff Pairs DRAM

Channel 1 10 Diff Pairs AMB 10 Diff Pairs AMB

2 DRAM 2 DRAM 2
s

DRAM DRAM

DRAM DRAM
DRAM DRAM
MCH_SPD1_SMB
SPD SMBus 1
DRAM DRAM
DRAM DRAM

DRAM DRAM

14 Diff Pairs DRAM DRAM


14 Diff Pairs
Channel 2 10 Diff Pairs AMB 10 Diff Pairs AMB

2 DRAM 2 DRAM 2
s

DRAM DRAM

DRAM DRAM
DRAM DRAM

SPD SMBus 2 MCH_SPD2_SMB

DRAM DRAM
DRAM DRAM

DRAM DRAM

14 Diff Pairs DRAM 14 Diff Pairs DRAM

Channel 3 10 Diff Pairs AMB 10 Diff Pairs AMB

2 DRAM 2 DRAM 2
s

DRAM DRAM

DRAM DRAM
2
DRAM DRAM

SPD SMBus 3 MCH_SPD3_SMB


CK410B

DB1200G
FBD Clock
Buffer

Figure 2-21. Wildcat-S with 8 FBDIMM connected to 4 Channels


EMC CONFIDENTIAL

FB DIMMS 53
FBDIMM Wildcat-S will accommodate up to two industry standard FBDIMM per
Blackford MCH channel. The channel consists of 10 Southbound
Channel 0 differential pairs that transport commands and or data from the
Blackford MCH to the FBDIMM. The AMB also forwards the data to the
next FBDIMM. The second piece of the channel is made up of 14
Northbound differential pairs which move data and responses from the
FBDIMM to the Blackford MCH. This data or response will also be
forwarded from the second FBDIMM connected on that channel back to
the MCH.

Channel 0
MCH_SPD0_SCL
MCH_SPD0_SDA

J5 – FB DIMM J6 – FB DIMM
SCL
SCL
SDA
SDA

U65 FBD_CH0_NB_<13..0>+/- DIMM00_01_NB_<13..0>+/-


Blackford
AMB AMB
MCH Advanced Advanced
Memory Buffer Memory Buffer
FBD_CH0_SB_<9..0>+/- DIMM00_01_SB_<9..0>+/-

SA2
SA 2
FBD_00_CLK+ SA1 FBD_01_CLK+
333Mhz SA0
SA 1
CLK+ FBD_00_CLK- FBD_01_CLK- SA 0
CLK-
V3_3

14.318Mhz + 167Mhz
333Mhz
U75 - U121
Clk Clk
Driver Buffer FBD_BR0_RESET_N
U148

Figure 2-22. FBDIMM Channel 0 Interface to the Blackford MCH


FBDIMM configurations on one channel must match FBDIMM
configurations for the other channel within the Branch. Configurations
from one branch to another can be different as long as mirroring is
disabled, which is the case for Wildcat-S. The FBDIMM specification
allows as many as eight FBDIMM’s to be on one memory channel. Intel's
Blackford shown in Figure 2-23 allows up to four FBDIMM’s on a
channel. The Wildcat-S will only have slots for two FBDIMM’s on a
channel as illustrated in the below diagram.

J6 – FBDIMM J8 – FBDIMM J10 – FBDIMM J12 – FBDIMM

J5 – FBDIMM J7 – FBDIMM J9 – FBDIMM J11 – FBDIMM

CHANNEL 0 CHANNEL 1 CHANNEL 2 CHANNEL 3


BRANCH 0 BRANCH 1

U65
Blackford MCH
EMC CONFIDENTIAL

54
Figure 2-23. MCH Branches with balanced DIMM Populated

The four FBDIMM channels are organized into two branches with two
channels on each branch. Each branch is supported by a separate memory

2
controller.

J6 – FBDIMM J8 – FBDIMM J10 – FBDIMM J12 – FBDIMM

J5 – FBDIMM J7 – FBDIMM J9 – FBDIMM J11 – FBDIMM

CHANNEL 0 CHANNEL 1 CHANNEL 2 CHANNEL 3


BRANCH 0 BRANCH 1
Memory Controller Memory Controller
U65
Blackford MCH

Figure 2-24. MCH Branches with Memory Controllers

The two channels in each branch operate in lock step to increase FBDIMM
bandwidth. A branch transfers 16 bytes of payload per frame on
Southbound lanes (Blackford to FBDIMM) and 32 bytes of payload per
frame on Northbound lanes (FBDIMM to Blackford).

Two branches may be operated in mirrored (RAID 1) or non-mirrored


mode. Wildcat will have mirrored mode disabled.

EMC CONFIDENTIAL

FB DIMMS 55
FBDIMM The FBDIMM direct signaling interface between the memory controller
and the DRAM chips is split into two independent signaling interfaces
Components with a (AMB) Advanced Memory Buffer between them. The interface
between the AMB and DRAM chips is the same one used in today’s DDR2
DIMMS shown in Figure 2-25.

DDR2 SDRAM (72) DATA / (18) STROBE DDR2 SDRAM (72) DATA / (18) STROBE

DDR2 DDR2 DDR2 DDR2 AMB DDR2 DDR2 DDR2 DDR2


Component Component Component Component Advanced Component Component Component Component
Memory Buffer

DDR2 SDRAM (29) Address / (4) Clk DDR2 SDRAM (29) Address / (4) Clk

SouthBound Data-Out10 14 NorthBound Data-In


DDR2 SDRAM (72) DATA / (18) STROBE DDR2 SDRAM (72) DATA / (18) STROBE

DDR2 DDR2 DDR2 DDR2 AMB DDR2 DDR2 DDR2 DDR2


Component Component Component Component Advanced Component Component Component Component
Memory Buffer

DDR2 SDRAM (29) Address / (4) Clk DDR2 SDRAM (29) Address / (4) Clk
10
SouthBound Data-In 14 NorthBound Data-Out

Blackford
MCH

Figure 2-25. FBDIMM Components

However, the interface between the memory controller and the buffer has
changed from a shared parallel interface to a point-to-point serial
interface. These two point-to-point interfaces are referred to as the South
Bound and North Bound interfaces as noted in Figure 2-25 above.

FBDIMM Pinout The pin assignments for the FBDIMM connector are detailed below. The
power and ground pins have been left out. See schematic for more detail.

Table 2-3. FBDIMM Connector Miscellaneous Signals

SIGNAL NAME PIN Description


MCH_SPD0_SCL 120 I2C from Blackford MCH
MCH_SPD0_SDA 119 I2C from Blackford MCH
FBD_00_SA2 118 I2C Address / boot configuration

EMC CONFIDENTIAL

56
Table 2-3. FBDIMM Connector Miscellaneous Signals

SIGNAL NAME PIN Description

2
FBD_00_SA1 240 I2C Address / boot configuration
FBD_00_SA0 239 I2C Address / boot configuration
FBD_00_CLK+ 228 167Mhz Differential Clock
FBD_00_CLK- 229 167Mhz Differential Clock
FBD_BR0_RESET_N 17 RESET TO ALL FBDIMM’S

North Bound and South Bound data signals are shown below.

Table 2-4. FBDIMM Connector Differential Data Pins to/from MCH

(+) DATA SIGNAL PIN (-) DATA SIGNAL PIN


FBD_CH0_NB_13+ 40 FBD_CH0_NB_13- 41
FBD_CH0_NB_12+ 48 FBD_CH0_NB_12- 49
FBD_CH0_NB_11+ 66 FBD_CH0_NB_11- 67
FBD_CH0_NB_10+ 63 FBD_CH0_NB_10- 64
FBD_CH0_NB_9+ 60 FBD_CH0_NB_9- 61
FBD_CH0_NB_8+ 57 FBD_CH0_NB_8- 58
FBD_CH0_NB_7+ 54 FBD_CH0_NB_7- 55
FBD_CH0_NB_6+ 51 FBD_CH0_NB_6- 52
FBD_CH0_NB_5+ 37 FBD_CH0_NB_5- 38
FBD_CH0_NB_4+ 34 FBD_CH0_NB_4- 35
FBD_CH0_NB_3+ 31 FBD_CH0_NB_3- 32
FBD_CH0_NB_2+ 28 FBD_CH0_NB_2- 29
FBD_CH0_NB_1+ 25 FBD_CH0_NB_1- 26
FBD_CH0_NB_0+ 22 FBD_CH0_NB_0- 23
(+) DATA SIGNAL PIN (-) DATA SIGNAL PIN
FBD_CH0_SB_9+ 90 FBD_CH0_SB_9- 91
FBD_CH0_SB_8+ 102 FBD_CH0_SB_8- 103
FBD_CH0_SB_7+ 99 FBD_CH0_SB_7- 100
FBD_CH0_SB_6+ 96 FBD_CH0_SB_6- 97
FBD_CH0_SB_5+ 93 FBD_CH0_SB_5- 94
FBD_CH0_SB_4+ 82 FBD_CH0_SB_4- 83
FBD_CH0_SB_3+ 79 FBD_CH0_SB_3- 80
FBD_CH0_SB_2+ 76 FBD_CH0_SB_2- 77
FBD_CH0_SB_1+ 73 FBD_CH0_SB_1- 74
FBD_CH0_SB_0+ 70 FBD_CH0_SB_0- 71

EMC CONFIDENTIAL

FB DIMMS 57
Table 2-5. FBDIMM Connector Differential Data Pins to next DIMM

(+) DATA SIGNAL PIN (-) DATA SIGNAL PIN


DIMM00_01_NB_13+ 160 DIMM00_01_NB_13- 161
DIMM00_01_NB_12+ 168 DIMM00_01_NB_12- 169
DIMM00_01_NB_11+ 186 DIMM00_01_NB_11- 187
DIMM00_01_NB_10+ 183 DIMM00_01_NB_10- 184
DIMM00_01_NB_9+ 180 DIMM00_01_NB_9- 181
DIMM00_01_NB_8+ 177 DIMM00_01_NB_8- 178
DIMM00_01_NB_7+ 174 DIMM00_01_NB_7- 175
DIMM00_01_NB_6+ 171 DIMM00_01_NB_6- 172
DIMM00_01_NB_5+ 157 DIMM00_01_NB_5- 158
DIMM00_01_NB_4+ 154 DIMM00_01_NB_4- 155
DIMM00_01_NB_3+ 151 DIMM00_01_NB_3- 152
DIMM00_01_NB_2+ 148 DIMM00_01_NB_2- 149
DIMM00_01_NB_1+ 145 DIMM00_01_NB_1- 146
DIMM00_01_NB_0+ 142 DIMM00_01_NB_0- 143
(+) DATA SIGNAL PIN (-) DATA SIGNAL PIN
DIMM00_01_SB_9+ 210 DIMM00_01_SB_9- 211
DIMM00_01_SB_8+ 222 DIMM00_01_SB_8- 213
DIMM00_01_SB_7+ 219 DIMM00_01_SB_7- 220
DIMM00_01_SB_6+ 216 DIMM00_01_SB_6- 217
DIMM00_01_SB_5+ 213 DIMM00_01_SB_5- 214
DIMM00_01_SB_4+ 202 DIMM00_01_SB_4- 203
DIMM00_01_SB_3+ 199 DIMM00_01_SB_3- 200
DIMM00_01_SB_2+ 196 DIMM00_01_SB_2- 197
DIMM00_01_SB_1+ 193 DIMM00_01_SB_1- 194
DIMM00_01_SB_0+ 190 DIMM00_01_SB_0- 191

EMC CONFIDENTIAL

58
DATA In non-mirrored, dual channel mode, Blackford supports the 18-device
DRAM code option for FBDIMM. This code has the following properties:
Protection

2
(ECC) • Correction of any x4 or x8 DRAM device failure

• Detection of 99.986% of all single bit failures that occur in addition to a x8


DRAM failure. The MCH will detect a series of failures on a specific
DRAM and use this information in addition to the information provided by
the code to achieve 100% of these cases.

• Detection of all two wire faults on the DIMMs. Includes any pair of single
bit errors.

• Detection of all permutations of two x4 DRAM failures

Single Device Blackford employs a single device data correction (SDDC) algorithm for
the memory subsystem that will recover from any single x8 or x4 DRAM
Data Correction device failure as well as detect any dual x4 device failure. The chip
(SDDC) disable is a 32-byte two phase code. The MCH also supports demand and
scrubbing.

A scrub corrects a correctable error in memory. A four byte ECC is


attached to each 32 byte payload. An error is detected when the ECC
calculated from the payload does not match the ECC read from the
memory. The error is corrected by modifying the ECC, the payload, or
both and writing both the ECC and the payload back into memory.

Only one demand or patrol scrub can be in process at a time.

The attributes of the SDDC are:

• "Two phase code over 32-bytes of data


• "100% correction for all single x4 or x8 device failures
• "100% detection of all double x4 device failures
• "Between 99.986% and 100% detection of double x8 device failures
depending on the type of failure.

EMC CONFIDENTIAL

FB DIMMS 59
FBDIMM SMBus SMBus is basically Intel's version of I2C. This SMBus port operates at
100Khz. FBDIMM size and configuration as well as AMB status are
(I2C) accessed by Blackford via a dedicated SMBus interface. Blackford has one
SMBus interface for each FBDIMM channel. Each FBDIMM has three
SMBus address pins (SA2,SA1 & SA0). The first FBDIMM on a channel
will have an address of b'000 and the second FBDIMM in the channel will
be have an address of b'001.

Channel 0
MCH_SPD0_SCL
SCL
SMBus 1 MCH_SPD0_SDA
SDA
SCL
SMBus 2 J5 – FB DIMM J6 – FB DIMM
SDA SCL
SCL
SCL
SMBus 3
SDA SDA
SDA
SCL
SMBus 4 EEPROM EEPROM
SDA

U65
Blackford
AMB AMB
MCH Advanced Advanced
Memory Buffer Memory Buffer
b’001
b’000

SA2
SA2
SA1
SA1
SA0
SA0

V3_3

Figure 2-26. MCH SMBus 1-4 interface to Channel 0 FBDIMM’s

The AMB has configuration registers that provide flexibility and allow for
testing and optimization of the device. Upon system (FBD_BR0_RST*),
configuration registers are reset to predetermined default states,
representing the minimum feature set required to successfully bring up a
normal channel. It is expected that the BIOS will properly determine and
program optimal configuration settings.

For all of these registers, the AMB supports register access mechanisms
through SMBus as well through in-band channel commands.

FBDIMM Before any transfers to/from FBDIMM can be supported, the MCH
DRAM registers must be initialized. The MCH uses its four System
Memory Management Bus interfaces (SMBus 1-4) to extract the DRAM type and
Configuration size information from the Serial Presence and Detect Port on the DIMMs.

FBDIMM’s contain a Serial Presence Detect (SPD) Port that consists of a


serial data (SDA), serial clock (SCL) and a 4 bit address field (SA [3:0]).
Blackford MCH can read/write to the DIMM SPD EEPROM and AMD
during configuration.

EMC CONFIDENTIAL

60
Reset Each FBDIMM has a low true reset input that uses 1.5V CMOS signaling.
The reset can be asserted by ICH7 GPIO25 or by the MCU GPIO P1.16

Clocking Each FBDIMM is clocked by a DB1200G compliant clock buffer. The


FBDIMM clock frequency will be the half the frequency of the CPU front
side bus clock frequency. See Figure 2-22 for clock signal and speeds.
2
Advanced The FBDIMM buffer is referred to as the AMB (Advanced Memory
Buffer). The AMB is a single-chip component located in the center of each
Memory Buffer FBDIMM that acts as a repeater and buffer for all signals and commands
(AMB) exchanged between the memory controller and the DDR2 SDRAM
devices, including data input and output. The AMB communicates with
the host controller and adjacent FBDIMM’s on a system board using an
industry-standard, high-speed, differential point-to-point interface at 1.5
V.

Next FBDIMM

SouthBound Data-Out 10 14 NorthBound Data-In


DDR2 SDRAM (72) DATA / (18) STROBE DDR2 SDRAM (72) DATA / (18) STROBE

DDR2 DDR2 DDR2 DDR2 AMB DDR2 DDR2 DDR2 DDR2


Component Component Component Component Advanced Component Component Component Component
Memory Buffer

DDR2 SDRAM (29) Address / (4) Clk DDR2 SDRAM (29) Address / (4) Clk
10
SouthBound Data-In 14 NorthBound Data-Out

Blackford
MCH

Figure 2-27. (AMB) Advanced Memory Buffer

Command, address, and data are transferred from the MCH to the AMB
over a high speed serial interface consisting of ten differential signals
called the South Bound Interface. Data returned from the AMB to the
MCH is transferred over a high speed serial interface consisting of
fourteen high speed differential signals called the North Bound Interface.
If command, address, and data from the memory controller is not meant
for the first FBDIMM, the AMB on that FBDIMM repeats the information
on a separate set of 10 differential signals to the next FBDIMM.

FBDIMM technology offers better RAS (reliability, availability,


serviceability) by extending the currently available ECC (error check
code, a method of checking the integrity of data in DRAM) to include
protection of commands and address data. Additionally, FBDIMM
technology automatically retries when an error is detected, allowing for
uninterrupted operation in case of transient errors.

EMC CONFIDENTIAL

FB DIMMS 61
FBD Thermal The Advanced Memory Buffer (AMB) contains an internal thermal diode
to measure AMB/DIMM temperature. Upon detecting an over
Diode temperature, the AMB initiates a thermal throttling event.

Data Flow Data flows in two directions between the Blackford (MCH) and the
FBDIMM’s

• Southbound - MCH --> FBDIMM


• Northbound - FBDIMM --> MCH.

South Bound A branch transfers 16 bytes of payload per frame on Southbound lanes
(Blackford to FBDIMM). 16x8=128 bits / 2 DIMM per branch = 64 bits per
DIMM.

There are a number of configurations which can be used to format the


way data is transferred across the 10 Southbound Lanes between the
Blackford (MCH) and the FBDIMM’s. The interface can use all 10 lanes to
transfer data/command and if any lane develops problems the whole
interface to that memory channel becomes useless.

If the MCH is configures the interface to operate in fail-over mode lane 9


is not used and held as a backup lane. If any of the 9 remaining lanes fail
it can use lane 9 to replace the failed lane and continue to operate. Only
single lane failures are supported in the mode. If more then one lane fails
then the interface for that memory channel becomes inoperative.

Table 2-6. Southbound Lane Operational Modes

Southbound Bit Lane Options


Mode Memory Channel CRC Fail-Over Frame Size
ECC Capable
10 - Lane Mode 8 Bit ECC 12 Bit CRC Yes 168 Bit
9 - Lane Mode (Failover) 8 Bit ECC 6 Bit CRC No 156 Bit

EMC CONFIDENTIAL

62
South Bound The below is an example of what command / data may look like coming
across the Southbound 10 Lane serial interface. Transfers 0-3 from the
Data Example MCH sends out the following to the FBDIMM:




There are two bits which contain the Frame Type.
24 bits of command are sent to the FBDIMM.
14 bits contain the CRC associated with the 24 bit command. 2
10 SouthBound Bit Lanes
Transfer
9 8 7 6 5 4 3 2 1 0
0 CRC0 CRC7 CRC8 Frame CMD20 CMD16 CMD12 CMD8 CMD4 CMD0
Cmd/F-type Cmd/F-type Cmd/F-type Type 0
1 CRC1 CRC6 CRC9 Frame CMD21 CMD17 CMD13 CMD9 CMD5 CMD1
Cmd/F-type Cmd/F-type Cmd/F-type Type 1
2 CRC2 CRC5 CRC10 CRC13 CMD22 CMD18 CMD14 CMD10 CMD6 CMD2
Cmd/F-type Cmd/F-type Cmd/F-type Cmd/F-type

3 CRC3 CRC4 CRC11 CRC12 CMD23 CMD19 CMD15 CMD11 CMD7 CMD3
Cmd/F-type Cmd/F-type Cmd/F-type Cmd/F-type

4 CRC21 DATA DATA DATA DATA DATA DATA DATA DATA DATA
Cmd/DATA Cmd/DATA Cmd/DATA Cmd/DATA Cmd/DATA Cmd/DATA Cmd/DATA Cmd/DATA Cmd/DATA

5 CRC20 DATA DATA DATA DATA DATA DATA DATA DATA DATA
Cmd/DATA Cmd/DATA Cmd/DATA Cmd/DATA Cmd/DATA Cmd/DATA Cmd/DATA Cmd/DATA Cmd/DATA

6 CRC19 DATA DATA DATA DATA DATA DATA DATA DATA DATA
Cmd/DATA Cmd/DATA Cmd/DATA Cmd/DATA Cmd/DATA Cmd/DATA Cmd/DATA Cmd/DATA Cmd/DATA

7 CRC18 DATA DATA DATA DATA DATA DATA DATA DATA DATA
Cmd/DATA Cmd/DATA Cmd/DATA Cmd/DATA Cmd/DATA Cmd/DATA Cmd/DATA Cmd/DATA Cmd/DATA

8 CRC17 DATA DATA DATA DATA DATA DATA DATA DATA DATA
Cmd/DATA Cmd/DATA Cmd/DATA Cmd/DATA Cmd/DATA Cmd/DATA Cmd/DATA Cmd/DATA Cmd/DATA

9 CRC16 DATA DATA DATA DATA DATA DATA DATA DATA DATA
Cmd/DATA Cmd/DATA Cmd/DATA Cmd/DATA Cmd/DATA Cmd/DATA Cmd/DATA Cmd/DATA Cmd/DATA

10 CRC15 DATA DATA DATA DATA DATA DATA DATA DATA DATA
Cmd/DATA Cmd/DATA Cmd/DATA Cmd/DATA Cmd/DATA Cmd/DATA Cmd/DATA Cmd/DATA Cmd/DATA

11 CRC14 DATA DATA DATA DATA DATA DATA DATA DATA DATA
Cmd/DATA Cmd/DATA Cmd/DATA Cmd/DATA Cmd/DATA Cmd/DATA Cmd/DATA Cmd/DATA Cmd/DATA

Next Frame below contains the rest of the Cmd /DATA CRC
CRC0 CRC7 CRC8

CRC1 CRC6 CRC9

CRC2 CRC5 CRC10 CRC13

CRC3 CRC4 CRC11 CRC12

Figure 2-28. South Bound Data Format

Transfers 4 to 11 contain data and its 8 CRC which complete the frame.

In the next frame 14 bits of CRC are calculated on the CMD CRC and the
15 bits of DATA CRC by an XOR function.

I am not sure which CRC scheme we are using so Figure 2-28 is meant for
example only at this time.

EMC CONFIDENTIAL

FB DIMMS 63
North Bound A branch transfers 32 bytes of payload per frame on Northbound lanes
(FBDIMM to Blackford). 32x8=256 bits / 2 DIMM per branch = 128 bits per
DIMM.

There are a number of configurations which can be used to format the


way data is transferred across the 14 Northbound Lanes where data
moves from the FBDIMM’s to the Blackford (MCH). The interface can use
all 14 lanes to transfer data/command and if any lane develops problems
the whole interface to that memory channel becomes useless.

If the MCH is configures the interface to operate in 13 lane fail-over mode


lane 13 is not used and held as a backup lane. If any of the 13 remaining
lanes fail it can use lane 13 to replace the failed lane and continue to
operate. Only single lane failures are supported in the mode. If more then
one lane fails then the interface for that memory channel becomes
inoperative

If the MCH is configures the interface to operate in 12 lane fail-over mode


lane 12 is not used and held as a backup lane. If any of the 12 remaining
lanes fail it can use lane 12 to replace the failed lane and continue to
operate. Only single lane failures are supported in the mode. If more then
one lane fails then the interface for that memory channel becomes
inoperative

Table 2-7. Northbound Modes

Northbound Bit Lane Options


Mode Memory Channel CRC Fail-Over Frame
ECC Capable Size
14 - Lane Mode 8 Bit ECC 12 Bit CRC Yes 168 Bit
14 - Lane Mode (Failover) 8 Bit ECC 6 Bit CRC No 156 Bit
13 - Lane Mode 8 Bit ECC 6 Bit CRC Yes 156 Bit
13 - Lane Mode (Failover) 8 Bit ECC No CRC No 144 Bit
12 - Lane Mode (Non ECC DIMMS) No ECC 6 Bit CRC No 144 Bit

EMC CONFIDENTIAL

64
Northbound
Data Example

Transfer

0
13

CRC
12

CRC
11

DATA
10

DATA
9
DATA
8
14 NorthBound Bit Lanes

DATA
7
DATA
6
DATA DATA
5 4
DATA
3
DATA
2
DATA
1
DATA
0
DATA
2
1 CRC CRC DATA DATA DATA DATA DATA DATA DATA DATA DATA DATA DATA DATA

2 CRC CRC DATA DATA DATA DATA DATA DATA DATA DATA DATA DATA DATA DATA
3 CRC CRC DATA DATA DATA DATA DATA DATA DATA DATA DATA DATA DATA DATA
4 CRC CRC DATA DATA DATA DATA DATA DATA DATA DATA DATA DATA DATA DATA

5 CRC CRC DATA DATA DATA DATA DATA DATA DATA DATA DATA DATA DATA DATA

6 CRC CRC DATA DATA DATA DATA DATA DATA DATA DATA DATA DATA DATA DATA

7 CRC CRC DATA DATA DATA DATA DATA DATA DATA DATA DATA DATA DATA DATA
8 CRC CRC DATA DATA DATA DATA DATA DATA DATA DATA DATA DATA DATA DATA
9 CRC CRC DATA DATA DATA DATA DATA DATA DATA DATA DATA DATA DATA DATA

10 CRC CRC DATA DATA DATA DATA DATA DATA DATA DATA DATA DATA DATA DATA

11 CRC CRC DATA DATA DATA DATA DATA DATA DATA DATA DATA DATA DATA DATA

Figure 2-29. North Bound Data Format in 14 Lane Mode

I am not sure which CRC scheme we are using so Figure 2-29 is meant for
example only at this time.

EMC CONFIDENTIAL

FB DIMMS 65
EMC CONFIDENTIAL

66
2.5 PLX PEX8524 PCIe Switch
The PLX PEX8524 switch provides a x8 PCI Express CMI path between SP
A and SP B. It also provides an x8 (or dual x4) PCIe path across the mid
plane to its corresponding IO Annex slot.

This switch provides a configurable non-transparent bridge for Wildcat-S


2
to Wildcat-S communications (CMI path). The transparent bridge will
allow each Wildcat-S Blackford to properly enumerate the PCI devices
connected to the IO Annex. Both Wildcat PLX switches provide a
non-blocking crossbar switch architecture.

The Wildcat-S implementation of the PLX PEX 8524 chip is shown in the
figure below.

MIDPLANE 8

SP A SP B
PLX PLX
CPU A CPU B U16 CPU A CPU B U16
U96 U49 U96 U49

8 8

PCI-E PCI-E
MCH Port A MCH Port A
MEM CTLR

MEM CTLR

DMA DMA
DIMM’s

DIMM’s

Engines Engines

U65 U65

plx_cmi_block.emf

Figure 2-30. PLX in the Peer-to-Peer CMI Path

Overview The PEX8524 (hereafter referred to as the PLX) is a multi-port PCIe switch
which may be used to connect PCIe devices in various ways such as a
fan-out from a single host to multiple endpoints, or as a communication
bridge between peer host controllers. In the Clariion Fleet family, the
peer-to-peer communication function is implemented so as to provide the
CMI (Command Management Interface) path between SP A and SP B.

Another function that can be performed by the PLX is to allow an SP to


mirror Write Data from a host to the peer SP DIMM’s through the PLX’s to
provide redundancy. If the first SP fails before data is written to the DAE,
the mirrored copy can be written to the DAE by the second SP. The PLX is
physically packaged as a 644-pin BGA which uses 3.3 V I/O signals and
1.5 V and 1.0 V for the high speed serial interfaces.

EMC CONFIDENTIAL

PLX PEX8524 PCIe Switch 67


The PLX contains 24 PCIe links which may be grouped and configured as
up to six ports or interfaces. Any port may be up to x8 in width, with the
remaining ports being limited by the number of available links left.Both
the MCH interface and the CMI path uses one x8 PCIe port each. The
configuration and functions of the PLX is set by a combination of
strapping pins, EEPROM data and software-accessible configuration
registers.

PEX8524
PCI-E Ingress Egress PCI-E
Scheduler Scheduler
Port Port
Station 1 Station 0

Non-Blocking Crossbar
Upstream Crossbar Crossbar
8 Port Switch Switch
Ingress Egress

Switch Fabric
Dev 8
to MCH
8
Dev 0
to I/O
Annex
Crossbar Crossbar
NT Port Switch Switch
8 Egress Ingress
Dev 9
to CMI
Egress Ingress
Scheduler Scheduler

plx_port_block.emf

Figure 2-31. PLX Block Diagram

PLX Functions An important functionality of the PLX is the ability of it to allow a host to
access devices within the address space or domain of another host or
peer, while preventing those devices from being discovered or
enumerated by the first host. In a typical PCI/PCIe environment the BIOS
will enumerate the PCI busses, devices and functions in the system by
accessing the PCI Configuration Registers and cataloging the Device ID,
Vendor ID and Header Type of the ones that respond. The memory and
I/O addresses of the devices are assigned at this time, so typically this is
done once only at reset.

The Header Type indicates the type of the device, whether the device is a
regular PCI device (Header Type 0) or if it is a PCI Bridge of some kind
(Header Type 1). If it is a bridge, the software must then enumerate any
busses (called subordinate busses) and devices on the other side of the
bridge before continuing on with the next device after the discovered
bridge. If each host were to try to enumerate all the devices that each
could access in the system, including each other’s devices, the discovery
process would fail because the hosts could enumerate the devices more
than once with unpredictable results.

There is one upstream port that interfaces with the MCH and two
downstream ports that interface with the IO Annex card and the peer SP.
At this point it can be noted that both downstream ports operate
differently. Both ports perform a different type of bridging function:
EMC CONFIDENTIAL

68
• Transparent Bridging to the IO Annex

• Non-Transparent Bridging to the Peer SP

2
The PLX prevents redundant enumeration by configuring one of its ports
as Non-Transparent. This means that even though it functions as a PCI
Bridge (Header Type 1)which allows accesses to propagate over a
subordinate bus to the peer devices, it reports that port to the
enumerating software as an end device (Header Type 0), thus preventing
any discovery of any devices in the peer. Transparent ports are reported
as Header Type 1 and so allow discovery of any subordinate busses and
their connected devices.

Address Another function performed by the PLX is to translate addresses of


Translation accesses between the peers. This allows one SP to use a particular
memory address to access memory at a different address in the other SP.
In the Fleet family the Peer SP is accessed by reading or writing to the
address range 0x8000 0000 to 0xBFFF FFFF. The corresponding location in
the Peer’s RAM that it can access for verifying the data is 0x4000 0000 to
0x7FFF FFFF.

The process is completely symmetric between the SP’s so that they both
see their own local RAM buffer beginning at 0x4000 0000 and their Peer’s
buffer beginning at 0x8000 0000. The translation process is described in
Figure 2-32.

1 SPA
8200 1000 Port 8 PLX Port 9
FSBCTL Upstream Port Virtual Port Link Port
Non-Transparent
PCI-E Port A Registers Registers Registers
C000 0XXX C001 0XXX C001 1XXX
MCH BFFF FFFF BFFF FFFF BFFF FFFF 7FFF FFFF
Memory Window Memory Window Memory Window Memory Window
ADDRESS
8200 1000 8200 1000 8200 1000 TRANSLATION 4200 1000
2 3
2 4
2 8000 000 4000 000
6000 000 6000 000
5
2
MIDPLANE

4200 1000 SPB (PEER)


DIMMs

Port 8 PLX Port 9


MEMCTL Upstream Port Virtual Port Link Port
4200 1000 Non-Transparent

8
2 PCI-E Port A
Registers
C000 0XXX
Registers
C001 0XXX
Registers
C001 1XXX
BFFF FFFF BFFF FFFF BFFF FFFF 7FFF FFFF
MCH Memory Window Memory Window Memory Window Memory Window

7
2 4200 1000

8000 000 4000 000 6


2
6000 000 6000 000

plx_adr_xlate.emf

Figure 2-32. Non-Transparent Port Address Translation

EMC CONFIDENTIAL

PLX PEX8524 PCIe Switch 69


1. The CPU generates a transfer to or from an address within the Peer
memory window (0x8000 0000 - 0xBFFF FFFF) and drives it on the FSB.

2. The PCIe port in the MCH claims the transaction since the address falls
within its memory window (0x6000 0000 - 0xBFFF FFFF) as defined by its
base and limit registers. It forwards the transaction onto the PCIe link to
the PLX.

3. The PLX upstream port (Port 8) receives the transaction from MCH and
determines that it is within its own memory window and so passes it to
the downstream Non-Transparent port (Port 9).

4. Port 9 is configured as being Non-Transparent which gives it two


interfaces: a Virtual Port connecting to the upstream port, and a Link Port
which connects to the PCIe interface. The Virtual Port determines if the
transaction falls within its memory window (0x8000 0000 - 0xBFFF FFFF)
and passes it on to the Link Port.

5. The Link Port modifies the address of the transaction to its own
memory window (0x4000 0000 - 0x7FFF FFFF) and sends the new
transaction on the PCIe bus to the Midplane and Peer SP.

6. The Peer PLX Link Port receives the transaction and determines
whether the address of the transaction falls within its memory window
(ox4000 0000 - 0x7FFF FFFF) and forwards it to the Virtual Port.

7. The address of the transaction is checked by the Virtual port and is


determined to NOT fall within its memory window. Because all the PCIe
ports use subtractive decode the transaction is, by default, automatically
sent "upstream". When the upstream port receives it the same action is
taken, it is forwarded "upstream" to the MCH.

8. The Peer MCH PCIe Port 8 receives the transaction and determines that
it does NOT fall within its memory window so it also forwards it
upstream where the Memory Controller recognizes the address and
claims the transaction. The Peer Memory Controller then accesses the
DIMM’s at the modified address (0x4000 0000 - 0x7FFF FFFF).

PLX The PLX may be configured by several mechanisms. These are H/W pin
strapping, EEPROM and Configuration Registers.
Configuration

H/W Strapping At power on reset some of the configuration parameters for the PLX are
set to a default value by strapping certain I/O pins on the package high or
low by means of pull-up or pull-down resistors. These pins set bits in the
PLX to an initial value at power on reset, which may be overwritten later
by either the EEPROM or software. The functions defined by these pins
are detailed in Table 2-8.

EMC CONFIDENTIAL

70
Table 2-8. PLX H/W Strapping Pins

2
SCHEMATIC NAME PIN Description

CFG_PLX_MODESEL[1:0] G2,G1 Mode Select pins:


00 = reserved
01 = Intelligent Adapter mode
10 = Dual-Host mode
11 = Transparent mode

CFG_SERDES_MD AC3 Factory Test pin - tied High

CFG_ST0_PORTCFG[4:0] D25,E25, Configures the number of lanes in ports 0,1,2,3 (x0 = disabled) in Station 0
F25,G25, 00000 = x4, x4, x4, x4
H25 00001 = x16, x0, x0, x0
00010 = x8, x8, x0, x0
00011 = x8, x4, x4, x0
00100 = x8, x4, x2, x2
00101 = x8, x2, x2, x4
00110 = x8, x2, x4, x2

CFG_ST1_PORTCFG[3:0] D25,E25, Configures the number of lanes in ports 0,1,2,3 (x0 = disabled) in Station 1
F25,G25, 0000 = x4, x4, x4
H25 0001 = x0, x0, x0
0010 = x8, x0, x0
0011 = x4, x4, x0
0100 = x4, x2, x2
0101 = x2, x2, x4
0110 = x2, x4, x2

CFG_USTRM_PSEL[3:0] C6,C7, Selects the Upstream port:


B6,B7 0000 = Port 0
0001 = Port 1
0010 = Port 2
:
1000 = Port 8
CFG_NT_USTRM_PS[3:0] B19,B17, Selects the Non-Transparent port:
B18,C17 0000 = Port 0
0001 = Port 1
0010 = Port 2
:
1000 = Port 8
1001 = Port 9

CFG_PLX_TESTM[3:0] J28,J27, Test Mode select: reserved for Factory testing only.
J30,J29 1111 = Default (Test Modes Disabled)

EEPROM The PLX also relies on an EEPROM to load other configuration data at
each power on reset. The EEPROM used in the Fleet family is an Atmel
AT25256AN 256Kbyte (32K x 8 bits internal) serial device. This device
uses the SPI bus to connect to the PLX. Contained in the EEPROM is the
initial values of the PCI Configuration Registers. Not all the registers
need to be set at this time, but in the Fleet systems the Device ID, Vendor
ID and Header Type are.

EMC CONFIDENTIAL

PLX PEX8524 PCIe Switch 71


V+3.3 V+3.3

PLX J25
PLX_EE_CS_N
1 8

PLX_EE_SCK PUP_PLX_EEHLD
H26 6 EEPROM 3
PLX_EE_DO PDN_PLX_EEWP_N
J26 5 AT25256 AN 7
PLX_EE_DI
G26 2 4

PDN_PLX_EEPR_N
K26

plx_eeprom.emf

Figure 2-33. PLX EEPROM Interface

The Serial Peripheral Interconnect (SPI) bus is a Motorola-designed


four-wire multi-master bus running at 7.8 MHz as implemented on the
PLX. The signal definitions and protocol are shown in Figure 2-9 and
Figure 2-34.

Table 2-9. PLX EEPROM SPI Signals


Signal name Description Driven by
PLX_EE_SCK Serial Clock PLX

PLX_EE_CS Chip Select PLX


PLX_EE_SO MOSI (Master Out / Slave In Data) PLX

PLX_EE_SI MISO (Master In / Slave Out Data) EEPROM

The SPI Master (PLX) will drive the Clock and Chip Select lines to the
Slave (EEPROM). Data driven by the PLX on the MOSI line (Command
byte followed by Address) is valid on the rising edge of the clock. If it is a
Write operation the data continues to be valid on rising clock edges. If it is
a Read operation the data driven by the EEPROM is valid on the falling
clock edge.

CLK

CS0 Valid on rising clock edge Write data valid on rising clock edge
Read data valid on falling clock edge Master terminates transfer
Master drives command and address Master drives write data
COMMAND ADDRESS Write DATA #1 Write DATA # N
MOSI

Read DATA #1 Read DATA # N


MISO

bcm_spi_proto.emf Slave drives read data

Figure 2-34. SPI Bus Timing

The Configuration registers are detailed on page 70.

EMC CONFIDENTIAL

72
PLX Registers The PLX contains a number of registers that are used for configuring its
operation and reporting status and errors during operation. All registers
may be accessed by Memory read/write cycles from the CPU once the

2
PLX has been configured by the BIOS.

PCI Configuration At system reset the devices in the system must be enumerated by the
Registers BIOS. PCI devices are enumerated and discovered by their location in the
system topology and are identified by their particular Bus Number
(0-255), Device Number (0-31), and Function number (0-7). The BIOS
must then assign memory or I/O addresses to the devices.

The PCI Configuration Registers are used to map (assign a memory


address to) the device so the CPU’s can use normal memory R/W cycles to
access the device. This is done by performing a Configuration Write to the
Base Address Registers (BAR) in the PCI Configuration space at offset
0x10 to 0x24. The generic PCI Configuration Registers are shown in
Figure 2-35.

Address Required fields


00 Offset
00 Device ID Vendor ID
04 Status Command
Header
08 Class Code Rev ID
3C 0C BIST Header Typ Latency Cache Line
40
10 Base Address 0
14 Base Address 1
18 Base Address 2
1C Base Address 3
20 Base Address 4
Device 24 Base Address 5
Specific
28 Cardbus CIS Pointer
2C Subsystem ID Command
30 Expansion ROM Base Address
34 Reserved Capabilities

38 Reserved
FF 3C Max_Lat Min_Gnt Int_Pin Int_Line
mch_pci_cfgspace.emf

Figure 2-35. Generic PCI Configuration Register Format

Doorbell and To facilitate communication between peer hosts the PLX contains
Scratch pad registers for passing messages or interrupts to each other called the
Registers Doorbell registers. These are available when the port is configured as
Non-Transparent. Setting bits in a Set register will cause interrupt packets
to be sent on the link, only if the corresponding bits are clear in a Mask
register. Both the Virtual port and the Link port have Doorbell registers.

EMC CONFIDENTIAL

PLX PEX8524 PCIe Switch 73


PEX8524
CPU Port 9
Virtual Port Link Port

C001 1090 Set Link I/ F IRQ


Port 8 C001 1094 Clear Link I /F IRQ
C001 1098 Set Link I /F IRQ Mask
C001 109 C Clear Link I /F IRQ Mask

Set Virtual I /F IRQ C001 0090


Clear Virtual I / F IRQ C001 0094
IRQx PACKET
Set Virtual I / F IRQ Mask C001 0098
MCH Clear Virtual I /F IRQ Mask C001 009 C

C001 00 B0 NT Port Scratchpad 0


C001 00 B4 NT Port Scratchpad 1
C001 00 B8 NT Port Scratchpad 2
C001 00 BC NT Port Scratchpad 3
C001 00 C0 NT Port Scratchpad 4
C001 00 C4 NT Port Scratchpad 5
C001 00 C8 NT Port Scratchpad 6
C001 00 CC NT Port Scratchpad 7
plx_doorbell.emf

Figure 2-36. PLX Doorbell and Scratch pad Registers

Scratch pad registers are similar in that they allow access from either the
Virtual port (local side) or the Link port (peer side). The main difference
is that writing to them will not cause an interrupt, so that they are
effectively dual-ported general-purpose registers.

EMC CONFIDENTIAL

74
PLX Signals The signal names used on the Dreadnought for the PLX is shown in
Figure 2-37 on page 75.

MCH
I/F
EXP_FRM_MCHA_<7:0>+
EXP_FRM_MCHA_<7:0>-
EXP_TO_MCHA_<7:0>_C+
EXP_TO_MCHA_<7:0>_C-
IO_EXP_R<7:0>+
IO_EXP_R<7:0>-
IO_EXP_T<7:0>_C+
IO_EXP_T<7:0>_C-
IO
ANNEX
PATH
2
CMI_EXP_R<7:0>+
PLX_EE_DI A G27
CMI_EXP_R<7:0>-
CMI
EEPROM
PLX_EE_DO
PLX_EE_CS_N
A G30
PLX CMI_EXP_T<7:0>_C+ PATH
I/F
PLX_EE_SCK
A G28

A G29
PEX8524 CMI_EXP_T<7:0>_C-

PDN_PLX_EEPR_N A H28

U16

VDD33A
PLX_PCI_RST_N
G3
V3_3
MISC. PLX_100MHZ_CLK_C+ V1
A J4
PLX_100MHZ_CLK_C- V1_A
A K4 POWER
V1_5
GND

01b CFG_PLX_MODSEL<1:0> G2,G1

P/U CFG_SERDES_MD
A C3
00010 b CFG_ST0_PORTCFG<4:0> G4
PLX_TMS
A C28, A C27,A D28,A D27,A E28
PLX_TDO
CONFIG 0010 b CFG_ST1_PORTCFG<3:0>
P29, N28, N29,M 28
H1

STRAPS 1000 b CFG_USTRM_PS<3:0> J1


PLX_TDI_MUX JTAG
L2,K 2,L3, K3
PLX_TCK
1001 bCFG_NT_USTRM_PS<3:0> M29, L29L28K 28
H2
PLX_TRST_N
1111 b CFG_PLX_TESTM<3:0> H4
J28, J27, J 30, J 29

plx_pinout.emf

Figure 2-37. PLX Signal Pins

The function of the pins are described in Table 2-10.

EMC CONFIDENTIAL

PLX PEX8524 PCIe Switch 75


Table 2-10. PLX PEX8524 Signals
SCHEMATIC SIGNAL PIN I/O DESCRIPTION

PCIeXPRESS SIGNALS

CMI_EXP_R<7:0>+/- G27/G26, In PCIe receive lines from Backplane/Peer PLX


D30/E30,
D28/E28,
D26/E26,
D24/E24,
D22/E22,
D20/E20,
D18/E18

CMI_EXP_T<7:0>_C+/- G29/G30, Out PCIe transmit lines to Backplane/Peer PLX


B30/A30,
B28/A28,
B26/A26,
B24/A24,
B22/A22,
B20/A20,
B18/A18

EXP_FRM_MCHA<7:0>+/- D2/E2, In PCIe receive lines from MCH


D4/E4,
D6/E6,
D8/E8,
D10/E10,
D12/E12,
D14/E14,
D16/E16

EXP_TO_MCHA_C<7:0>+/- B2/A2, Out PCIe transmit lines to MCH


B4/A4,
B6/A6,
B8/A8,
B10/A10,
B12/A12,
B14/A14,
B16/A16,
B18/A18
IO_EXP_R<7:0>+/- AF20/AG20, In PCIe receive lines from I/O Annex
AF18/AG18,
AF16/AG16,
AF14/AG14,
AF12/AG12,
AF10/AG10,
AF8/AG8,
AF6/AG6

IO_EXP_T_C<7:0>+/- AJ20/AK20, Out PCIe transmit lines to I/O Annex


AJ18/AK18,
AJ16/AK16,
AJ14/AK14,
AJ12/AK12,
AJ10/AK10,
AJ8/AK8,
AJ6/AK6

EEPROM INTERFACE

PLX_EE_DI AG27 In PLX input from the serial EEPROM data output.

PLX_EE_DO AG30 Out PLX output to the serial EEPROM data input.

PLX_EE_CS_N AG28 Out Serial EEPROM active-Low chip select output.

PLX_EE_SCK AG29 Out 7.8 MHz serial EEPROM clock output.

PDN_PLX_EEPR_N AH28 In Serial EEPROM present input. Tied to GND to indicate presence of
EEPROM

MISCELLANEOUS
PLX_PCI_RST_N G3 In General reset for this PLX

PLX_100MHZ_CLK_C+ AJ4 In Positive half of 100-MHz PCI Express reference clock input signal pair
EMC CONFIDENTIAL

76
Table 2-10. PLX PEX8524 Signals
SCHEMATIC SIGNAL PIN I/O DESCRIPTION

PLX_100MHZ_CLK_C- AK4 In Negative half of 100-MHz PCI Express reference clock input signal pair
JTAG

PLX_TMS

PLX_TDO
PLX_TDI_MUX
G4

H1
J1
In

Out
In
JTAG Test Mode Select

JTAG Test Data Output


JTAG Test Data Input
2
PLX_TCK H2 In JTAG Test Clock Input

PLX_TRST_N H4 In JTAG Reset

POWER

V3_3 - 3.3 V Power for I/O logic functions

VDD33A AF1 - 3.3 V Power for PLL circuits


V1_5 - SerDes termination pins tied to SerDes termination voltage (typically 1.5 V)

V1 - 1.0 V Power for Core Logic

V1_A - 1.0 V Power for Core Logic

CONFIGURATION STRAP SIGNALS - See Table 2-8 on page 71

EMC CONFIDENTIAL

PLX PEX8524 PCIe Switch 77


EMC CONFIDENTIAL

78
2.6 ICH 7

Introduction The ICH 7 IO Controller Hub is based on the Intel IO Controller Hub 5
architecture. ICH 7 integrates much of the Super IO functions and is targeted at
2
embedded servers. It has several different interfaces:

• Direct Media Interface (DMI) to the Blackford MCH

• Low Pin Count (LPC) bus to the Super IO and FWH

• PCI Express interface to Ethernet controllers and a debug header

I2C- ICH 7 provides both a master and slave I2C interface. In the SAN
environment the ICH 7 is the SMB master however in a NAS
environment the System Management Card provides the master for the
SMB and the ICH 7 acts as a slave.

EMC CONFIDENTIAL

ICH 7 79
.

AH28 FSB_A20M_N
ESI_FRM_MCH_<3:0>+/- FSB<1:0>_IERR_33_N
ESI_TO_MCH_<3:0>C+/- AG26 FSB_FERR_N
C25 AG22 FSB_IGNNE_N
DMI DMICOMP D25 AH22 FSB_STPCLK_N
(Blackford MCH) ICH_DMI_CLK+/- AF25 FSB_INTR
INTERFACE BOOT
MCH_ERR<2:0> AH24 FSB_NMI
CPU
AF23 FSB_SMI_N INTERFACE
AF22 FSB_INIT_N
L_FRAME_N AB3 AG23 ICH_RCIN_N N.U.
L_AD<3..0> AE22 ICH_A20GATE
FWH_INIT_N AG21
FWH & SIO LPC
L_DRQ0_N AC3
INTERFACE R4 FLT_BL_YLW_N
ICH_SER_IRQ AH21
AC22 SIO_ANNEX_FLT
I2C_ARB_A1_BUF_OUT
A2 I2C_ARB_A1_BUF_IN
LAN1
EXP_TO_ICH1+/- F25/F26 AF21 ICH_ALERT_BUF_N
EXP_FRM_ICH1+/- E27/E28 E22 CF_INS_N
LAN2 EXP_TO_ICH2+/- H25/H26 AE19 PLEASE_BIF_N
EXP_FRM_ICH2+/- G27/G28 D8 FBD_CLKEN_N GPIO
EXP_TO_ICH3+/- K25/K26 AH18 ICH_SWA_INS_N
DBG X1 PCI-E EXP_FRM_ICH3+/- AE20 ICH_SWB_INS_N
J27/J28
CONN E19 PS1_I2C_ATN_BUF_N
AD20 IO_ANNEX_INS_N
PUP_ICH_P_FRAME_N F16 E23 ICH_SYS_FLT_N
PCI PUP_ICH_P_PERR_N C9 B21 ICH_SW_FLT
INTERFACE
PUP_ICH_P_SERR_N B10
(Not Used)
ICH_33MHZ_CLK A9 AC20 ICH_USB_RST_N
ICH_PCIRST_N B18 D20 ICH_FBD_RST
A21 IO_I2C_BUFRST
RESETS
A13 PS1_I2C_BUFRST
ICH_SDA B22 C26 ICH_PLTRST_N
ICH_SCL C22 Y4 ICH_RSM_RST_N
SMB / I2C
ICH_SLAVE_SDA
ICH_SLAVE_SCL
A25
B25
ICH7 A22 ICH_RST_IN_N
INTERFACE PROCESSOR
P.U. ICH_LINKALERT_N A26 AF26 FSB_THERMTRIP_N
G7 TEMPERATURE
P.U. PUP_INTRUDER Y5 PROCHOT_<1:0>_33_N
INDICATORS
BTN_IS_SMI_N
F8 ICH_PIRQG_N
ICH_USB_R_<7:4>,<1:0>+/- G8 ICH_PIRQE_N
ICH_USB_OC<4:0>_N B5 ICH_PIRQD_N
USB INTERUPTS
C5 ICH_PIRQC_N
iNTERFACE ICH_USB_RBIAS
USB_48MHZ_CLK B4 ICH_PIRQB_N
B2
A3 ICH_PIRQA_N

C23 ICH_PWRBTN
ICH_SATA<3:0>_RX_C+/-
SATA AA4 ICH_PWROK
ICH_SATA<3:0>_TX_C+/- POWER
INTERFACE AC18 ICH_PSA_INS_N INTERFACE
ICH_SATA_CLK+/-
AC21 ICH_PSB_INS_N
N.U. PUP_ICH_SATA_LED
ICH_SATA_RBIAS AB1 ICH_RTCX1
AC2 ICH_RTCX2 CLOCKS
ALT_CMI_TX_BUF AF19
To MIDPLANE ALT_CMI_RX_BUF AC1 ICH_14MHZ_CLK
R3
C_FPGA_CONFIG0_N U2 A27 ICH_SUS_STAT_N POWER
C_FPGA_CNFDNE1_12_N C20 SIO_32KX1 MANAGEMENT
AH19 INTERFACE
C_FPGA_STATUS0_N AD21 AD22
PERSONALITY C_FPGA_DATA0_N PUP_ICH_VRMPG
AC19 F20
BOARD PUP_ICH_WAKE_N
C_FPGA_DCLK0_N AG18
SIGNALS AH20 PUP_MCH_SYNC_N
AF20 PUP_ICH_THRM_N
PUP_ICH_RI_N MISCELLANEOUS
CFG_ICH_INTVRM W4 A28
PUP_ICH_SPKR IDE_A<02:00> N.U.
A19
MISCELLANEOUS IDE_D<15:00> N.U.
PUP_RTC_RST_N AA3
ICH_SLP_S<5:3>_N N.U.
PUP_ICH_DXFTEST_N C21
PUP_ICH_TP0_N A3 ICH7_ACZ_SYNC
F21 HanceRapids _New_Pinout .vsd
A3 ICH7_ACZ_SDOUT

Figure 2-38. Hance Rapids - ICH


EMC CONFIDENTIAL

80
Signal
Descriptions
Table 2-11. ICH 7 Pin Descriptions
SCHEMATIC SIGNAL
Blackford MCH & DMI Interface
I/O Description 2
ESI_FRM_MCH_<3:0>+/- I One of two differential strobe lines used to transmit and receive data through the hub
interface

ESI_TO_MCH_<3:0>+/- O One of two differential strobe lines used to transmit and receive data through the hub
interface
DMICOMP I/O Determines the DMI input and output impedances
ICH_DMI_CLK+/- I 100MHz Differential clock used to run the DMI Interface.
MCH_ERR<2:0> I Indicates to the ICH 7 that the Blackford MCH has a fault condition.
Firmware Hub / LPC Interface
LFRAME* I/O FWH4/LFRAME is a muxed signal. LFRAME#:
Indicates the start of an LPC cycle, or an abort.
LAD(3..0) I/O LPC Muxed Address Data bus to the firmware hub FWH(3..0) or SIO chip
LAD(3..0). These signals have internal pull-ups inside the Hance Rapids.
LDRQ0_N I The Super IO will use this signal to request either a DMA or Bus master access of
the LPC interface.
FWH_INIT_N O Asserted by the ICH 7 for 16 PCI clocks to reset the FWH. It is a copy of the signal
FSB_INIT_N which is used to reset the processor’s.
ICH_SER_IRQ I/O An interrupt request to/from the Server I/O.
PCI Express Interface
EXP_TO_ICH<3:1>+/- I Serial PCI Express Interface to the LAN1, LAN2 and DBG X1 PCI-E CONN
EXP_FRM_ICH<3:1>+/- O Serial PCI Express Interface from the LAN1, LAN2 and DBG X1 PCI-E CONN
PCI Interface (Not Used)
PUP_ICH_P_FRAME_N I Not Used
PUP_ICH_P_PERR_N I Not Used
PUP_ICH_P_SERR_N I Not Used
ICH_33MHZ_CLK I Not Used
ICH_PCIRST_N O Not Used
SMB INTERFACE
ICH_SDA O ICH 7 (master) I2C clock
ICH_SCL I/O ICH 7 (master) I2C data.
ICH_SLAVE_SDA I ICH 7 (slave) I2C clock.
ICH_SLAVE_SCL I/O ICH 7 (slave) I2C data.
ICH_LINKALERT_N I/O P.U.
PUP_INTRUDER I P.U.
USB INTERFACE
ICH_USB_R_<7:4>,<1:0>+/- I/O Differential pairs used to transmit & receive Data/Address/Command signals for the
ICH 7 USB ports.

EMC CONFIDENTIAL

ICH 7 81
Table 2-11. ICH 7 Pin Descriptions
SCHEMATIC SIGNAL I/O Description
ICH_USB_OC<4:0>_N I Indicate to the USB controller that an over current condition exists
ICH_USB_RBIAS I/O USB Resistor Bias. Analog connection point for an external resistor. Used to set
transmit currents and internal load resistors.
ICH_48MHz_CLK I 48MHz Clock used to run the USB controller.
Serial ATA INTERFACE.
ICH_SATA<3:0>_RX_C+/- I Differential signals for receiving data over the serial ATA interface
ICH_SATA<3:0>_TX_C+/- O Differential signals for transmitting data over the serial ATA interface
ICH_SATA_CLK+/- I 100MHz clock used to run the SATA controller.
PUP_ICH_SATA_LED O N.U. This is an open-collector output pin driven during SATA command activity.
ICH_SATA_RBIAS I/O Analogue connection point for an external connection point to ground.
GPIO.
FLT_BL_YLW_N I A high on this pin and a low on the SIO_FAULT_LED pin will light the blade fault
LED blue.
SIO_ANNEX_FLT I A high on this signal will light the IO Annex fault LED.
I2C_ARB_A1_BUF_IN I A low on this pin indicates the A I2C bus is being requested.
I2C_ARB_A1_BUF_OUT_N O Driven high to request ownership of the shared A I2C bus.
ICH_ALERT_BUF_IN O Driven low by the ICH 7 to indicate a PS1 I2C failure exists.
CF_INS_N I Compact FLASH is inserted
PLEASE_BIF_N I A low indicates that the Tornado extension card wants the PLX configured for dual
84 PCIe.
FBD_CLKEN_N O A low will enable the FBD clock buffer.
ICH_SWA_INS_N I A low indicates that the A management board is inserted.
ICH_SWB_INS_N I A low indicates that the B management board is inserted.
PS1_I2C_ATN_BUF_N I A low indicates a PS1 I2C bus error.
IO_ANNEX_INS_N I A low indicates that the IO annex A ia inserted.
ICH_SYS_FLT_N O Asserted to light the System Fault LED
ICH_SW_FLT O Asserted to light the Management Fault LED
GPIO Resets
ICH_USB_RST_N O Driven low by the ICH 7 to reset the USB HUBs
ICH_FBD_RST O Driven high by the ICH 7 to reset the Fully Buffered DIMMs.
IO_I2C_BUFRST O Driven high by the ICH 7 to reset the IO Modules I2C buses
PS1_I2C_BUFRST I/O A high on this pin will reset the PS1 I2C bus. Default to 0
ICH_PLTRST_N O Platform Reset: The ICH 7 asserts PLTRST* to reset devices on the platform that is
the SIO & FWH. The ICH 7 asserts PLTRST* during power-up and when S/W
initiates a hard reset sequence through the Reset Control Register. The ICH 7 drives
PLTRST* inactive for a minimum of 1 ms after both PWROK and VRMPWRGD
are driven High.
ICH_RSM_IN_N I This signal is used to reset the resume power plane logic. When low the internal
LAN Controller will be put into reset.
ICH_RST_IN_N I This signal forces an internal reset after being de-bounced. The ICH 7 will reset
immediately if the SMBus is idle, otherwise, it will wait for up to 25ms+/- 2ms for
the SMBus to idle before forcing a reset on the system.

EMC CONFIDENTIAL

82
Table 2-11. ICH 7 Pin Descriptions
SCHEMATIC SIGNAL I/O Description
Boot CPU U47 Interface
FSB_A20M_N

FSB<1:0>_IERR_33_N
O

I
Mask A20: A20M# will go active based on either setting the appropriate bit in
the Port 92h register, or based on the A20GATE input being active.
Speed Strap: During the reset sequence, the ICH 7 drives A20M#
high when the corresponding bit is set in the FREQ_STRP register.
When asserted indicates that the relevant CPU is indicating an error condition.
2
FSB_FERR_N I Numeric Coprocessor Error: This signal is tied to the coprocessor error signal
on the processor. FERR# is only used when the ICH 7
coprocessor error reporting function is enabled in the General Control Register
FSB_IGNNE_N O Ignore Numeric Error: This signal is connected to the ignore error pin on the
CPU. IGNNE# is only used when the ICH 7 coprocessor error
reporting function is enabled in the General Control Register
FSB_INIT_N O Initialization: INIT# is asserted by the ICH 7 for 16 PCI clocks to reset the
processor. The Hance Rapids may be configured to support CPU BIST. In that case,
INIT# will be active when PXPCIRST# is active.
FSB_INTR O CPU Interrupt: INTR is asserted by the ICH 7 to signal the
processor that an interrupt request is pending and needs to be serviced. It is an
asynchronous output and normally driven low.
Speed Strap: During the reset sequence, the ICH 7 drives INTR
high when the corresponding bit is set in the FREQ_STRP register.
FSB_NMI O Non-Maskable Interrupt: NMI is used to force a non-Maskable interrupt to the
processor. The ICH 7 may generate an NMI when either SERR#
or IOCHK# is asserted
FSB_SMI_N O System Management Interrupt: SMI# is an active low output synchronous to
PCICLK. It is asserted by the ICH 7 in response to one of many
enabled hardware or software events.
FSB_STPCLK_N O Stop Clock Request: STPCLK# is an active low output synchronous to
PCICLK. It is asserted by the Hance Rapids in response to one of many
hardware or software events. When the processor samples STPCLK# asserted,
it responds by stopping its internal clock.
ICH_RCIN_N I Keyboard Controller Reset CPU: The keyboard controller may generate INIT#
to the processor. This saves the external OR gate with the ICH 7_other sources of
INIT#. When the ICH 7 detects the assertion of this signal, INIT# is generated for 16
PCI clocks.
ICH_A20GATE I A20 Gate: From the keyboard controller. Acts as an alternative method to force
the A20M# signal active.
Processor Temperature Indicators
FSB_THERMTRIP_N I When low, this signal indicates that a thermal trip from the processor occurred.
PROCHOT_<1:0>_33_N I A low indicates that either processor 0 or 1 has asserted PROCHOT.
Power Interface
ICH_PWRBTN I The Power button will indicate a system request to go to a sleep state. If the system
is already in a sleep state, this signal will cause a wake event to occur.
ICH_PWROK I When asserted, this signal is an indication to the ICH 7 that core power has been
stable for 99ms and that PCICLK has been stable for 1 ms. When PWOOK is
de-asserted the ICH 7 asserts ICH_PLTRST_N
ICH_PSA_INS_N I When asserted indicates that power supply A is inserted
ICH_PSB_INS_N I When asserted indicates that power supply B is inserted
Clocks
ICH_RTCX1 I This signal is connected to the 32.768kHz crystal.
ICH_RTCX2 I This signal is connected to the 32.768kHz crystal.
ICH_14MHZ_CLK I This 14MHz clocks used to run the 8254 timers internal to the ICH 7

EMC CONFIDENTIAL

ICH 7 83
Table 2-11. ICH 7 Pin Descriptions
SCHEMATIC SIGNAL I/O Description

Power Management Interface


ICH_SUS_STAT_N O This signal is asserted by the ICH 7 to indicate that the system will be entering a low
power state soon. It is monitored by devices with memory that need to switch from
norma refresh to suspend refresh mode. It can also be used by other peripherals as an
indication that they should isolate their outputs that may be going to powered-off
planes.
SIO_32KX1 O An output from the ICH 7 internal RTC generator circuit and used by other chips as
a refresh clock.
Signals going to the Midplane
ALT_CMI_TX_BUF O Transmit signal for the alternate CMI path. the signal is inverted before it gets to the
midplane
ALT_CMI_RX_BUF I Receive signal for the alternate CMI path. the signal is inverted before it gets to the
midplane
Personality Board Signals
C_FPGA_CNFDNE1_12_N I Configuration done signal for programming the UART Multiplexing FPGA on IO
card 0
C_FPGA_DCLK0 O Data clock for programming the FPGA on IO card 1.
C_FPGA_DATA0 I/O Data line for programming the FPGA on IO card 1.
C_FPGA_CONFIG0_N O Configuration line for programming the FPGA on IO card 1.
C_FPGA_STATUS0_N I Status line for programming the FPGA on IO card 1.
C_FPGA_CNFDNE0_N I Configuration done signal for programming the FPGA on the Personality module
Interrupts
BTN_IS_SMI_N I This signal is from the Reset/NMI button on the AirDam. This allows the user to
issue a Reset or NMI to the CPU blade. The button will function as a Reset until
POST is complete and after this point it will function as an NMI button.
ICH_PIRQA_N I Interrupt from the IO Annex card
ICH_PIRQB_N I Interrupt from the IO Annex card
ICH_PIRQC_N I Interrupt from IO Card 1
ICH_PIRQD_N I Interrupt from IO Card 0
ICH_PIRQE_N I Not Implemented.
ICH_PIRQG_N I Interrupt from the IO Annex card
Miscellaneous Signals
PUP_ICH_VRMPG I Pulled Up
PUP_ICH_WAKE_N I Pulled Up
PUP_MCH_SYNC_N I Pulled Up
PUP_ICH_THRM_N I Pulled Up
PUP_ICH_RI_N I Pulled Up
ICH7_ACZ_SYNC O Pulled Up
ICH7_ACZ_SDOUT O Pulled Up
CFG_ICH_INTVRM I This signal enables the internal 1.05v suspend regulator when connected to
VccRTC. When connected to Vss, the internal regulator is disabled.
PUP_ICH_SPKR O Pulled Up
EMC CONFIDENTIAL

84
Table 2-11. ICH 7 Pin Descriptions
SCHEMATIC SIGNAL I/O Description
PUP_RTC_RST_N I Routed to the CMOS Clear Jumper J15 CONN2. When asserted, this signal resets

2
register bits in the Real Time Clock (RTC) well.
PUP_ICH_DXFTEST_N I/O Pulled Up
PUP_ICH_TP0_N I This signal must have an external pull-up to V3_3.

USB Interface The USB interfaces are connected as follows:

MGMT_USB+/-
ICH_USB_R_0+/- USB
USB MIDPLAN
PORT 0 Controller
HUB ANNEX_USB+/- CONNECTO

USB ICH_USB_R_1+/- NAND ND_FLASH_D0 FLASH


FLASH
PORT 1
Controller
ICH 7

MSYSTEMS
USB ICH_USB_R_<4,5>+/-
UDOC
PORT’S
4& 5 Connector

USB ICH_USB_R_6+/- Management


Controller
PORT 6

IO3_USB_+/-
USB ICH_USB_R_7+/- IO2_USB_+/-
PORT 7 USB 4-Port
Io Module
HUB IO1_USB_+/-
Controller Connector
IO0_USB_+/-
ICHUSBInterface.emf

Figure 2-39. USB Interfaces

ICH 7 USB Port


Port 0 Connected to a two port USB HUB that connects
to the Solar Flare Management Module front
panel connector and the IO Annex via the
midplane.
Port 1 Connected to the Emulation Flash using the a
Cypress USB Flash controller.

EMC CONFIDENTIAL

ICH 7 85
ICH 7 USB Port
Port 4 & 5 Connected to a debug connector capable of
connecting to the MSystems uDoc Flash IO
Module.
Port 6 Connected to the Management controller

Port 7 Connected to a 4-port USB HUB that connects to


each IO Module.

LAN Interfaces The Wildcat-S board will include two 10/100/1000 Mbit interfaces, each
utilizing a Broadcom Gigabit Ethernet controller BCM5751.Each LAN
interface will be attached to the ICH 7 using a PCI Express Root Port.
There is a serial EPROM connected to the chip to store the MAC address
for the interface. LAN boot will be supported by this interface.

Management Module A
RJ45
BCM
BCM5397
5751 RJ45
Ethernet
SW
BCM RJ45
5751

Wildcat A M
i IO Annex A
d
p
l
a
n
e IO Annex B

Wildcat B

Management Module B
BCM
5751 RJ45
BCM5397
Ethernet RJ45
BCM SW
5751
RJ45

Wcat_S_LAN.emf

Figure 2-40. LAN Ethernet Communications Connections

BCM5751 The Broadcom BCM5751 is a triple-speed 10/100/100BASE-T Ethernet


LAN controller which combines a Media Access Controller (MAC),
Ethernet Ethernet transceivers, buffer memory, and a PCI Express interface in one
Controller physical package. It has a serial memory interface for non-volatile data
storage such as a serial EEPROM, and it also has an integrated RISC
processor which may be used for optional processing of Ethernet data.

EMC CONFIDENTIAL

86
There are two BCM5751 populated on the Wildcat-S motherboard,
referred to as LAN1 and LAN2. LAN1’s Ethernet port is connected to the
BCM5325E switch on the local switch via the midplane. LAN2’s Ethernet
port is connected to the BCM5325E switch on the peer switch also via the

2
midplane. They are both connected in parallel to the ICH 7 via the PCI
Express bus.

Operation On power up the device’s core logic exits the reset state 80ms after all
three power rails reach their respective minimum threshold levels.

Power Rail Minimum


Threshold Level
1.2V 0.85V
2.5V 2.0V
3.3V 2.8V

Table 2-12. Power Rail Minimum Threshold Levels

The PCI-Express core logic will exit the reset state 600us after
LAN_RST_N de-asserts and V3.3V is above its minimum threshold level.

After exiting from the reset state, the boot processor begins executing
code from the internal start-up ROM image. Execution of the startup code
requires, the processor to load boot code from the external EPROM into
internal scratch pad memory. Start-up code will be executed when the
device is reset via the hardware reset signal or a software reset.

Clocks A single external free running clock source is required for the core
functions of the device to operate. The device requires a 25MHz clock
input from either an external crystal on the crystal signal pins
LAN1_X<1:0>. From this single clock source, the device, the device
derives a 125-MHz clock fro the Gigabit PHY, and a 62.5 MHz clock fro
the core.

In addition to the core clock, the BCM5751 supports an external reference


clock front end PCI-Express interface. The PCI-Express interface clock is a
100MHz differential clock input on the LAN1_100MHz_CLK+/-.

Configuration After reset, the BCM5751 must be configured so that it may function in
the system. This is done by the system CPU reading and writing over the
PCI bus to the Configuration registers in the BCM5751. After the basic
Configuration registers are programmed, the CPU programs additional
registers so that the BCM5751can locate its data structures in system
RAM.

Receive When an Ethernet packet (frame) is received, the BCM5751 will


automatically validate and strip the header and CRC from it and store the
payload in its internal buffer memory. It will access the PCI-Express bus
to read some data structures from system RAM to tell it where the data
buffers are located. It will then write over the PCI-Express bus to move
the data payload into system RAM.
EMC CONFIDENTIAL

ICH 7 87
Transmit To transmit data, the System CPU will place the data payload and the
data structures describing it into system RAM. The CPU will then write
over the PCI-Express bus to a register in the BCM5751 to inform the
device that there is a packet for transmission. The BCM5751 will then
move the packet data from host memory to its internal buffer
memory.The MAC transmitter then initiates a transmission over the
MII/GMII interface to the Management module.

PCI-E Interface A6 PUP_LAN1_WAKE_N


EXP_FRM_ICH1+/-
EXP_TO_ICH1+/-

LAN_RST_N C2
ICH 7 LAN1_REFCLK_SEL F4 D8 LAN1_PCIE_TST
Interface
LAN1_100MHZ_CLK+/-

Ethernet Tranceiver
LAN1_TRD0_C+/-
LAN1_TRD1_C+/-
LAN1_TRD2_C+/-
LAN1_TRD3_C+/-

D10 PUP_LAN1_SCL N.U.


PDN_LAN1_GPIO_0 G12 D9 PUP_LAN1_SDA N.U.
GPIO
LAN1_EEWP
PDN_LAN1_GPIO_2
G13
H13
BCM5751 K11 LAN1_EECLK
L10 LAN1_EEDAT EPROM

PDN_LAN1_SI E12
A10 LAN1_RDAC
SPI Interface PDN_LAN1_SCLK E11
Not Used PDN_LAN1_CS_N C12 LAN1_X1
P12
N12 LAN1_X0 CLOCKS
PDN_LAN1_LOW_PWR L6
LAN1_ATTN_BTTN_N A2

JTAG TEST

LAN1_TDI H12
LAN1_TDO D6
LAN1_TMS C11
JTAG LAN1_TRST_N D12
LAN1_TCK D7
BCM5751 _Pinout.emf

Figure 2-41. Broadcom BCM5751 Pinout

Signal
Descriptions
Table 2-13. BCM5751 Pin Descriptions
SCHEMATIC SIGNAL I/O Description
PCI-E Interface
EXP_FRM_ICH1+/- I PCI Express receive differential pair
EXP_TO_ICH1+/- O PCI Express transmit differential pair

EMC CONFIDENTIAL

88
Table 2-13. BCM5751 Pin Descriptions
SCHEMATIC SIGNAL I/O Description
LAN_RST_N I The PCI-E reset. The PCI_E core exits reset state 600us after LAN_RST_N is

2
de-asserted and the internal Power On Reset (POR) circuitry detects the correct
voltage threshold level on the V3_3 volt input (2.67V). The PCI-E core logic will
enter reset state if the 3.3V power rail drops below 2.67V or LAN_RST_N input is
asserted. Assertion of this signal does not reset the on-chip MAC or other modules
LAN1_REFCLK_SEL I This signal determines which input reference the clock the PLL will use to generate
its internal PCI-E clock. This pin is pulled to 3.3V on the Wildcat-S board which
configures the controller to use the PCI-E LAN1_100MHZ_CLK+/-
LAN1_100MHZ_CLK+/- I 100MHz differential clock which provides timing for all PCI-E transactions.
Firmware Hub / LPC Interface
PUP_LAN1_WAKE_N O Asynchronous output request for a change in power state.
LAN1_PCIE_TST I PCI-E test mode used for ATE testing. Must be pulled low for normal operation.
Failure to pull low will place the chip in non-operational mode.
Ethernet Transceiver Interface
LAN_TRD0_C+/- I/O Ethernet interface to the management module
LAN_TRD1_C+/- I/O Ethernet interface to the management module
LAN_TRD2_C+/- I/O Ethernet interface to the management module
LAN_TRD3_C+/- I/O Ethernet interface to the management module
Miscellaneous Signals
LAN1_EECLK I/O Serial EEPROM data.
LAN1_EEDAT I Serial EEPROM clock.
LAN1_RDAC I DAC Bias Resistor. This signal is used to adjust the current level of the DAC.
LAN1_X1 I 25MHz crystal input
LAN1_X0 O 25MHz crystal output
JTAG INTERFACE
LAN1_TDI I JTAG serial data in.
LAN1_TDO O JTAG serial data out.
LAN1_TMS I JTAG mode select.
LAN1_TRST_N I reset for the JTAG controller.
LAN1_TCK I/O JTAG serial clock

EMC CONFIDENTIAL

ICH 7 89
EMC CONFIDENTIAL

90
2.7 Server IO & Slow Devices
The Server I/O (SIO) chip is an integrated I/O controller which provides
the processor with a number of interfaces to various slow devices in the
system. The Server I/O chip may also be referred to as the Server I/O in
this manual. The other supporting devices attached to the SIO are:
2
• NVRAM
• BIOS / POST (FLASH)
• RS232 level Buffers

U141 U149
XAD<7:0>

U18
U143 XA<11:4>
SIO X-Bus
Buffer
LPC Bus

L_AD<3:0> NVRAM
ICH PC87417 XA<17:12>
U137
Buffer
4 XA<3:0> XA<17:0>

RS232

SIO_TXD1 TXD1_CBT
U140 U127
SIO_RXD1

MIDPLANE
BIOS/POST RXD1_CBT
FLASH
SIO_TXD2 TXD2_CBT

SIO_RXD2 RXD2_CBT

Figure 2-42. Wildcat-S Server IO Block Diagram

Server I/O The Server I/O device (SIO) used in Wildcat-S is the PC87417. It is a
128-pin PQFP package which provides the following functions and
interfaces:

• Low Pin Count (LPC) bus from ICH chip


• X-Bus (8-bit to NVRAM)
• Real Time Clock (RTC) with 242 byte CMOS RAM
• 51 GPIO ports
• Watchdog Timer
• 2 UART’s which connect to the midplane
• Floppy Drive Interface (not used)
• IEEE 1284 Parallel Port (not used)
The Floppy Drive and Parallel port interfaces are not used in Wildcat-S.

EMC CONFIDENTIAL

Server IO & Slow Devices 91


Server IO Pin The below illustration shows most of the Server IO (SIO) signals.
out

SIO_33MHZ_CLK SERIAL 1 96 SIO_RXD1


114
SIO_32KX1 TO COM1 SIO_TXD1
42 CLKS- Socket 98
SIO_PCI_RST_N Reset-
120 SIO_CTS1_N
PS2_I2C_RST_N 99
9 SIO_RTS1_N
HLD_IN_PST_N 97
124 SIO_DSR1_N
SERIAL 1 95
XAD<7:0> STATUS SIO_DTR1_N
24-31 CTRL 100
XSTRB_<1,0> SIO_DCD1_N
33-34 94
XRD_N PUP_SIO_RI1
14 101
NVRAM_CS_N
22 SIO_RXD2
XWR_N SERIAL 2 104
15 X-Bus
TO COM2 SIO_TXD2
Interface
XA<3:0> Socket 106
DUART_CS_N
23 SIO_CTS2_N
PORT80_CS_N 107
21 SIO_RTS2_N
QUART_CS_N 105
20 SERIAL 2
SIO_DSR2_N
STATUS 103
Server CTRL SIO_DTR2_N
L_AD<3:0> 108
110-113
L_FRAME_N I/O 102
SIO_DCD2_N
117 LPC
L_DRQ0_N PUP_SIO_RI2
Interface 109
118
ICH_SER_IRQ BATTERY_LOW_N
119 49
MFG_MODE_N
SIO_BLADE_FAULT 35
50 Power MARGIN_HI_N
SAFE2RM_HOT_LED 48
51 FAULTS MARGIN_LO_N
FAULT_REG_<7:0> 56
1..8
FPGA1_CBT_OE
36
SIO_WDO_N FPGA0_CBT_OE
55 45
MISC ICH_RCIN_R_N
Misc.
122
ICH_A20GATE
123
SIO_ARB_B1
PS2_I2C_ATTN_N 38
54 I2C_ARB_B1
IO0_2_4_INT_N I2C 37
78 Interrupts (SMB Bus) 52
SMB_SEL0
IO1_3_5_INT SMB_SEL1
10 53
VBAT, VSB, VDD TIED VSS(4:0) Tied
TO V3_3 to GND

Figure 2-43. Wildcat-S Server IO Pin out

EMC CONFIDENTIAL

92
Signal
Descriptions
Table 2-14. Super IO Chip Pin Descriptions
SCHEMATIC SIGNAL PIN PU/PD I/O
CLOCKS - RESET - BATTERY LOW SIGNALS
Description 2
SIO_33MHZ_CLK 114 I 33Mhz Clock from the U40 Clock Synthesizer.
SIO_32KX1 42 I Suspend Clock: (N/U) From the ICH RTC generator circuit (32.768 KHz).
SUSCLK will have a duty cycle that may be as low as 30% or as high as
70%.
SIO_PCI_RST_N 120 I The Hance Rapids ICH asserts ICH_PXPCIRST_N which becomes this
signal. This is hardware reset of SIO active low. The signal also resets the
FHW.
BATTERY_LOW_N 49 I From the U51 chip which monitors the Lithium battery.
Address / Data Multiplexed Bus & controls (XBUS Interface)
XAD<7..0> PU I/O Super IO multiplexed address / data bus. Goes to NVRAM & Port80 debug
latch chip which controls 8 leds. (see diagram)
XSTRB_O 34 O This signal latches out XA<11..4> to the NVRAM
XSTRB_1 33 O This signal latches out XA<16..12> to the NVRAM
XRD_N 14 O This signal controls the output enable to the NVRAM
NVRAM_CS_N 22 O This signal controls the chip select to the NVRAM
XWR_N 15 O This signal controls the write enable to the NVRAM and is also ORED with
PORT80_CS_N signal to control the Port 80 debug latch chip clock.
XA<3..0> O These signals go the NVRAM address pins XA<3..0>.
DUART_CS_N 23 O Chip Select for the IO Module located on the IO Annex card
PORT80_CS_N 21 O
QUART_CS_N 20 O
Firmware Hub / LPC Interface
L_FRAME_N 117 I/O FWH4/LFRAME is a muxed signal. LFRAME#:
Indicates the start of an LPC cycle, or an abort.
L_AD(3..0) I/O LPC Muxed Address Data bus to the firmware hub FWH(3..0) or Hance
Rapids chip LAD(3..0). These signals have internal pull-ups inside the Hance
Rapids.
L_DRQ0_N 118 O LPC Serial DMA/Master Request Input: To the Hance Rapids Chip to
request DMA or bus master access of the LPC bus.
ICH_SER_IRQ 119 I/O Serial Interrupt Request: This pin implements the serial interrupt protocol.
FAULT OUTPUTS
FAULT_REG<7..0> O These signals are outputs to the fault expander chip’s U73 and U74.
SIO_BLADE_FAULT 50 O Indicates blade fault when this signal goes active low. Causes the blue Fault
LED on Airdam to Blink.
When this signal is high it indicates no fault and disables the Fault LED.
SAFE2RM_HOT_LED 51 O Will light LED CR36_2S indicating that the FRU is too hot to remove
Keyboard and Mouse Signals to the Debug Compression Header which is Depopulated.
KB_CLK 125 O Keyboard clock. Pulled up.
KB_DATA 126 I/O Keyboard data. Pulled up

EMC CONFIDENTIAL

Server IO & Slow Devices 93


Table 2-14. Super IO Chip Pin Descriptions
SCHEMATIC SIGNAL PIN PU/PD I/O Description
MS_CLK 127 O Mouse clock. Pulled up.
MS_DATA 128 I/O Mouse data, Pulled up
ICH_RCIN_R_N 122 O Keyboard Controller Reset to ICH7. The keyboard controller may
generate INIT# to the ICH7. When the ICH7 detects the assertion of this
signal for 16 PCI clocks will cause a Interrupt to the CPU.
ICH_A20GATE 123 O A20 Gate: From the keyboard controller. Acts as an alternative method to
force
the A20M# signal active.
I2C INTERFACES & CONTROLS
I2C_ARB_B1 37 I SIO monitors this line to see if I2C shared bus is free.
45 I SIO monitors this line to see if I2C shared bus is free.
38 O If SIO sees the I2C shared bus is free it can then arbitrate for the shared I2C
buses
PS2_I2C_ATTN_N 54 O If SIO sees the I2C shared bus is free it can then arbitrate for the shared I2C
buses
PS2_I2C_RST_N 9 O This line goes to the PIC and U42 ColdFire.
SMB_SEL0 52 PU O Select line to the U83 MUX to allow the ICH7 access to one of the I2C
buses.
SMB_SEL1 53 PU O Select line to the U83 MUX to allow the ICH7 access to one of the I2C
buses.
SERIAL COM PORT 1 & STATUS SIGNALS
SIO_RXD1 96 I RS232 serial data received from the Diplex FPGA across patch cable from
the DAE.
SIO_TXD1 98 O RS232 serial data sent to the Diplex FPGA to be delivered to the DAE across
patch cable.
SIO_CTS1_N 99 I Clear to send. Connected to the Switch Management Module via the
midplane
SIO_RTS1_N 97 O Request to send..Connected to the Switch Management Module via the
midplane
SIO_DSR1_N 95 I Data Set Ready. Connected to the Switch Management Module via the
midplane
SIO_DTR1_N 100 O Data Terminal Ready. Connected to the Switch Management Module via
the midplane
SIO_DCD1_N 94 I Data Carrier Detected. Connected to the Switch Management Module via
the midplane
SERIAL COM PORT 2 & STATUS SIGNALS
SIO_RXD2 104 I RS232 serial data received from the Diplex FPGA across patch cable from
the DAE.
SIO_TXD2 106 O RS232 serial data received from the Diplex FPGA across patch cable from
the DAE.
SIO_CTS2_N 107 I Clear to send. Connected to the Switch Management Module via the
midplane
SIO_RTS2_N 105 O Request to send. Connected to the Switch Management Module via the
midplane
SIO_DSR2_N 103 I Data Set Ready. Connected to the Switch Management Module via the
midplane
SIO_DTR2_N 108 O Data Terminal Ready. Connected to the Switch Management Module via
the midplane

EMC CONFIDENTIAL

94
Table 2-14. Super IO Chip Pin Descriptions
SCHEMATIC SIGNAL PIN PU/PD I/O Description
SIO_DCD2_N 102 I Data Carrier Detected. Connected to the Switch Management Module via

2
the midplane.
INTERRUPTS
PS2_I2C_ATTN_N 54 I
IO0_2_4_INT_N 78 I
IO1_3_5_INT 10 I Asserted low when PSA detects no AC, also asserted when PSA is powered
by DC from SPS
POWER
BATTERY_LOW_N 49 PU I Asserted low when PSB detects no AC, also asserted when PSB is powered
by DC from SPS.
MARGIN_LO_N 56 PU O Output to on-board power circuits to margin voltages down 3%
MARGIN_HI_N 48 PU O Output to on-board power circuits to margin voltages up 3%
MFG_MODE_N 35 PU I Used by manufacturing to disable voltage monitoring while margining.
MISC. SIGNALS
SIO_WDO_N 55 O SIO_WDO needs to be configured by BIOS to be used as WD timer output to
U35 Micro controller. Pulses when a watchdog timeout occurs.
HLD_IN_PST_N 124 I

The below tables contains the chip selects that the Super IO chip outputs
to control other devices.

Table 2-15. SIO Chip Selects


CHIP SCHEMATIC
SEL. SIGNAL PIN PU/PD I/O Description
SUPER IO CHIP SELECTS
XCS0 DUART_CS_N 23 PU O Chip Select to the Personality Connector
XCS1 NVRAM_CS_N 22 O This signal controls the chip select to the NVRAM
XCS2 PORT80_CS_N 21 O This signal is ORED with XWR_N and controls the Port 80 debug
latch chip clock pin. Port 80 controls the (POST) LEDS.
XCS3 QUART_CS_N 20 PU O Chip Enable output to FPGA.

EMC CONFIDENTIAL

Server IO & Slow Devices 95


SIO H/W Strapping Because the SIO has many internal functions it is necessary to select a
basic set of configuration parameters at power up. This is done by
"strapping", or tying certain signals to a pull-up or pull-down resistor.
These signal pins are read by the SIO at reset so it can configure the
proper internal functions accordingly.

In the Wildcat-S system some of the SIO hardware strapping signals are
left at their default settings and therefore use the internal pull-up or
pull-down resistors within the chip to set the configuration parameters.

The relevant signals are shown in Table 2-16

Table 2-16. SIO Hardware Strapping Signals


Schematic Signal Name Pin PU/PD Strap Name & Function

SIO_DTR1_N 100 PU Base Address. Sampled at VDD Power-Up reset to determine


the base address of the configuration Index-Data register pair,
as follows:
No pull-up resistor = 2Eh-2Fh
External pull-up resistor = 4Eh-4Fh

SIO_RTS1_N 97 PU TRI-STATE Device. Sampled at VDD Power-Up reset to force


the device to float all its output and I/O pins, as follows:
No pull-up resistor = pins active
External pull-up resistor = pins floating

SIO_WDO_N 55 PU CLKIN 48 MHz. Sampled at VSB Power-Up reset to determine


the presence of the 48 MHz input clock at the CLKIN pin, as
follows:
No pull-up resistor = no clock
External pull-up resistor = 48 MHz clock

XSTB2_N 32 NC X-Bus Default Configuration. Sampled at VSB Power-Up reset


to set the configuration of the X-Bus transactions.
XSTB1_N 33 PU Pins.. Functionality
210
XSTB0_N 34 PU 0 x x = No BIOS
1 0 0 = With BIOS, XA11-4 multiplexed, XRDY disabled
1 0 1 = With BIOS, XA11-4 multiplexed, XRDY enabled
1 1 0 = With BIOS, XA11-4 direct, XRDY disabled
1 1 1 = With BIOS, XA11-4 direct, XRDY enabled
No pull-up resistor = 0 (internal pull down)
External pull-up resistor = 1

MARGIN_HI_N 48 PU ACCESS.bus Slave Address. Sampled at VSB Power-Up reset


to determine the slave address of the device on the
ACCESS.bus, as follows
No pull-up resistor = D8h, D9h
External pull-up resistor = 60h, 61h

EMC CONFIDENTIAL

96
BIOS Chip The BIOS / POST chip which is sometimes referred to as the firmware hub
is a FLASH device which contains the BIOS and POST code for the
Wildcat-S board. It is a 2 Meg x 8-bit programmable non-volatile memory

2
which interfaces directly to the LPC bus. The clock for the LPC bus is
provided by the clock generator chip and runs at 33 MHz.

U141 SIO _PCI_RST_N ICH_PLTRST


U18

SIO ICH7
PC87417 L_FRAME_N

L_AD<3:0>

U140
U75 FWH_33MHZ_
CLK_R BIOS/POST
FLASH FWH_INIT_N

Sio _fwh_block.emf

Figure 2-44. BIOS / POST Block Diagram

The SIO_PCIRST_N signal is used to reset the device memory while the
FWH_INIT_N (CPU Reset) is used to reset the memory when the CPU is
reset.

SIO COM The Super IO connects to 8 COM Ports in total. They are:
PORTS Table 2-17. Super IO COM Ports

COM Port Device IO Space


1 Debug 0x2F8
2 SPS 0x3F8
3 IO Module 0 0x300
4 IO Module 1 0x308
5 IO Module 2 0x310
6 IO Module 3 0x318
7 IO Module 4 (Located on 0x320
the Tornado)
8 IO Module 5 (Located on 0x328
the Tornado)

EMC CONFIDENTIAL

Server IO & Slow Devices 97


RS232 Two of the Super IO COM Ports are connected to the Switch Management
Module serial connectors via the midplane. One COM port is used to
communicate with the SPS and the other is used for debug.

The RS232 interface is connected to the mini-DB9 connector (J9) located


on the front airdam of the management module. This interface is also
referred to as COM1 and COM2 and is used for debug and maintenance.

• COM 1 is attached to the console


Diagnostics, such as Converge Diags are run via the console port.
• COM 2 is for the Standby Power Supply (SPS) if used.

DUART A Dual UART connected to the Super IO provides two serial COM Ports
which are used to communicate with the FPGA on the IO Modules
connected to the Wildcat board via the IO Annex Module. The ports will
be used to communicate with the fibre ports using diplexed RS-232.

COM7_CS_N
DUART_CS_N
U135

COM8_CS_N

XAD<7:0>
DUART_TO/FRM_IO4 U136
XA<2:0>
DUART_TO/FRM_IO5 DUART
U143
XAD<7:0>
XAD<7:0>
U149 XA<17:12>

XA <11:4>
U137
NVRAM XA<3:0> U141
NVRAM_CE_N

XA<3:0>
DUART_TO/FRM_IO0
XAD<7:0> NVRAM_CS_N SIO
DUART_TO/FRM_IO1 U120 U13
XA<2:0> PC87417
DUART_TO/FRM_IO2 XWR_N
QUART
DUART_TO/FRM_IO3
XRD_N

PORT 80_CS_N

U120
PORT 80
CONN XAD<7:0>

COM<6:3>_CS_N QUART_CS_N
U93
Sio _port_80.emf

Figure 2-45. NVRAM, DUART, QUART & PORT80 CONN Interface

QUART The Quad UART is connected to the SuperIO XBUS and provides four
additional serial COM Ports. These COM Ports are used to communicate
to the FPGAs on each IO MOdule connected to the Wildcat via the IO
Module connectors. The COM ports will be used to communicate with
the fibre ports using diplexed RS-232.
EMC CONFIDENTIAL

98
PORT 80 The BIOS shall have the ability to transmit Port 80 codes as well as Fault
codes to the enclosure management micro controller. A port 80 debug
board may be inserted to view the Port 80 codes.

NVRAM The non-volatile memory on the Wildcat SP is implemented by a Cypress


CY62128V SRAM memory. It is a 256K x 8-bit CMOS Static SRAM device.
2
Normally, this is a volatile memory but in the Wildcat SP it is attached to
a battery to provide power when the main power supplies are shut off,
thus preserving any data.

Dallas NVRAM The DS1314 ia an Nonvolatile Controller with Lithium Battery Monitor.
Controller This chip is a self-contained device which converts a standard low power
SRAM into a non-volatile memory. A precision voltage reference and
comparator monitors the Vcc input for an out-of-tolerance condition.
When an invalid Vcc condition occurs, the conditioned chip enable
output (NVRAM_CE_N) is forced inactive independent of the chip select
signal (NVRAM_CS_N) from the SIO chip. This effectively write-protects
the stored data in the SRAM.

During a power failure, the controller switches the SRAM power from the
Vcc pin (V3_3) to the battery to provide the energy required for data
retention. On a subsequent power-up, the SRAM remains write protected
until a valid power condition returns.

EMC CONFIDENTIAL

Server IO & Slow Devices 99


EMC CONFIDENTIAL

100
2.8 Clocks & Interrupts

Clocks and Figure 2-46 below shows the locations of the crystals and oscillators used
by the Wildcat-S Motherboard. The following is a list of their frequencies
2
Oscillator and the parts which use each one.
Locations
• Y1 - LAN 2 - 25Mhz
• Y2 - LAN 1 - 25Mhz
• Y3 - Flash Device - 24Mhz
• Y4 - ICH - 32.768Khz
• Y5 - Management Controller (MC)- 12Mhz
• Y6 - Depopulated and not show above
• Y7 - Uart, Quart & 4 IO Modules - 33.333Mhz
• Y8 - Clock Drive - 14.318Mhz

CM1 1.5V J14


J4 - VRM
Y1
J21
Wildcat-S U8 U7 U2
Y2
U16 LAN1 LAN2 Y4 Y3
205-800-602c U18 U12
PLX J53_1S
ICH
U26 U23 J20
U29 Y5 U28 U27
CM3 1.2V
U39
U49 U35

INTEL
U51 U50
Woodcrest =
Clovertown =
U65
J52_1S
MCH
Y8
J22S Y7

U96
INTEL
Woodcrest= J11 – FB DIMM
Clovertown= J12 – FB DIMM J51_1S
J9 – FB DIMM
J10 – FB DIMM
J17

J7 – FB DIMM U2691
U121

U120
J8 – FB DIMM
U133 U131 U128 U125

J5 – FB DIMM
U127

J6 – FB DIMM
U143 U137

J3 - VRM CM2 U141 J U140 U136 J50_1S


1.8V SIO 1 FWH
8

Figure 2-46. Crystals and Oscillators are shown in bright Yellow

EMC CONFIDENTIAL

Clocks & Interrupts 101


CPU & MCH U75 is a IDTCV127 clock driver designed for P4 processors. It uses an
14.318Mhz oscillator located at Y8 as its input frequency. The clock driver
Clocks has its Freq_sel<2..0> set at b’100 which translates to a 333Mhz CPU clock
output to the two processors.

The MCH also receives a 333Mhz clock signal from the driver.

A 100Mhz clock signal is sent from the driver to the U52 clock buffer
which is used to fan out up to 12 copies of this clock to various
components. Only the 100Mhz to the MCH is shown below.

U121 clock buffer is configured to be a DIVIDE / 2 which takes as input


333Mhz from the U75 clock driver and outputs 167Mhz to the MCH and
other components on the board. The only exception is the DIF_<10,11>
outputs are setup as bypass ports which output 333Mhz. One 333Mhz
signal is connected to the J17 XDP test interface connector.

The (U75) clock driver and both buffers (U52 & U121) are connected to
the I2C SMBus (not shown) which can be used by the ICH to read/write
to configuration command registers in each component.

U75 J17
IDTCV126 CPU0_BCLK_+/- 2 U96
BCLK0/1 CPU0 XDP Conn
CK410B 333Mhz (CPU)
BCLK0/1
CPU1_BCLK_+/- 2 U49 2 XDP_BCLK_+/-
BCLK0/1 CPU1
333Mhz 333 Mhz

MCH_BCLK_+/-2 Blackford MCH FBD_BRANCH0_CLK_+/-


CORECLK_+/- FBD01CLK_+/-
333Mhz
FBD_BRANCH1_CLK_+/-
PSEL[2:0] PSEL[2:0]
U65 FBD23CLK_+/-

PECLK_+/- 100Mhz
FREQ_SEL[2:0]
[b’100] = 333Mhz U52 MCH_100MHZ_CLK_+/-
DIF_0+/-
100Mhz
BYPASS
CK_DB1200 _SRC_+/-
CLK_IN+/-
DB1200G U121 167Mhz
Clk Buffer DIV2 DIF_9+/-
DB1200 G DIF_8+/- 167Mhz
Clk Buffer
333Mhz
DIF_10+/-
Y8 – Crystal DB1200 G_MEM_CLK_+/-
CLK_IN+/-
14.318MHz 333Mhz

Figure 2-47. CPU1, CPU0 & Blackford MCH Clocks

EMC CONFIDENTIAL

102
FBDIMM Clocks The 8 FBDIMM’s slots each receive an 167Mhz Differential Clock from the
U121 clock buffer. The FBDIMM’s use this clock to re-time the data
signals on writes to the DDR2 chips used on the FBDIMM’s and it is also

2
used as the clock source to clock the data back over the serial interface
Lanes to the MCH on reads from the FBDIMM’s.

The clock buffer takes as its input frequency 333Mhz from the U75 clock
driver chip. The clock buffer is programed by the ICH over the I2C lines
for a gear ratio of 1/2 or divide by 2 out to the FBDIMM’s.

U75 clock driver has an 14.31818Mhz crystal oscillator (Y8) connected to


it supplying its input clock. FREQ_SEL<2..0> lines are hard wired b’100
which results in an output clock speed of 333Mhz.

FBD_00_CLK_+/- 2
U75 U121 DIF_1+/- FBDIMM 00

DIV2 DIF_0+/- FBD_01_CLK_+/- 2


CK410B FBDIMM 01
DB1200G
FREQ_SEL[2:0]
Clk Buffer FBD_10_CLK_+/- 2
[b’100] = 333 Mhz DIF_3+/- FBDIMM 10

FBD_11_CLK_+/- 2
DIF_2+/- FBDIMM 11
167Mhz

FBD_20_CLK_+/- 2
DIF_5+/- FBDIMM 20

FBD_21_CLK_+/- 2
DIF_4+/- FBDIMM 21
333Mhz
DB1200 G_MEM_CLK_+/-
CLK_IN+/- FBD_30_CLK_+/- 2
DIF_6+/- FBDIMM 30

FBD_31_CLK_+/- 2
Y8 – Crystal
DIF_7+/- FBDIMM 31
14.318MHz

Figure 2-48. FBDIMM Clocks

FLASH Clock The Cypress NAND FLASH/ Controller module U2 is a plug-in device
connected to the USB interface to the ICH. It has a external 24Mhz
oscillator that supplies the clock signal through the connector for the U2
modules PLL circuit.

U2
Y3 – Crystal NAND
NAND
24MHz FLASH
FLASH
Controller

Figure 2-49. NAND FLASH Module Clock

EMC CONFIDENTIAL

Clocks & Interrupts 103


ICH Clocks The ICH has a 32.768Khz oscillator connected to it which is used to
generate its real time clock. This real time clock is sent out of the ICH
SUSCLK pin which becomes SIO_32KX1 out to the Server IO chip. Other
clocks include:

• 48Mhz clock is used for USB controller operations.


• 33Mhz clock is used for the LPC interface.
• 14Mhz reference clock.
• 100Mhz clock is used for the PCI Express operations.

U75
CK410B
U18
ICH_SATA_CLK+/- ICH7
SRC_4+/- SATACLK_+/- PECLK_+/-
USB_48MHZ_CLK
USB48MHZ CLK48
ICH_33MHZ DMICLK_+/-
PCIF_1 PCI_CLK
ICH_14MHZ
REF_0 CLK14

Y4 – Crystal
FREQ_SEL[2:0] = [b’100] 32.768kHz

100Mhz U52
CK_DB1200 _SRC_+/- ICH_DMI_CLK+/-
DIF_1+/- 2
SRC_0+/- CLK_IN+/-

BYPASS
DB1200G
Clk Buffer

Y8 – Crystal
14.318MHz

Figure 2-50. Interface Controller Hub (ICH) Clocks

EMC CONFIDENTIAL

104
PLX Clocks The PLX8532 receives an 100Mhz differential clock signal from the U52
clock buffer. It uses this clock for its PCI-Express interface controller. The
PLX has its own internal clock for other operations.

U75
2
CK410B FREQ_SEL[2:0]
[b’100] = 333Mhz
U52 PLX8532
DIF_0+/-

BYPASS

CK_DB1200_SRC_+/- PLX_100MHZ_CLK_+/-
SRC0+/- CLK_IN+/- DIF_2+/-
100 Mhz 100Mhz

Y8 – Crystal
14.318MHz

Figure 2-51. PLX Clocks

LAN Clocks LAN 1,2 chips each have an 25Mhz crystal oscillator connected to them.
This clock is used for operational purposes. Both LAN chips U8 & U7
receive an 100Mhz differential clock from the U52 clock buffer which they
use for their PCI-Express interface controllers.

U8
U75 Y2 – Crystal
25MHz LAN1
FREQ_SEL[2:0] = [b’100] BCM5751
CK410B
100Mhz U52 LAN1_100MHZ_CLK_+/- 2
CK_DB1200 _SRC_+/- DIF_4+/-
SRC_0+/- CLK_IN+/- LAN2_100MHZ_CLK_+/-
DIF_5+/- 2

BYPASS
DB1200G
Clk Buffer
U7
Y1 – Crystal
25MHz LAN2
BCM5751

Figure 2-52. LAN1 & LAN2 Clocks

EMC CONFIDENTIAL

Clocks & Interrupts 105


FWH & SIO The Server IO (SIO) and Firmware Hub (FWH) chips both receive an
33Mhz clock from the U75 clock driver. This 33Mhz clock is used for
Clocks operational purposes and to run the LPC interface to the ICH.

U75
CK410B
U141
SIO
SIO_33MHZ_MHZ
REF_1 CLK

U140
FWH
FWH_33MHZ_CLK
PCIF_0 CLK

FREQ_SEL[2:0] = [b’100]

Y8 – Crystal
14.318MHz

Figure 2-53. Server IO & Firmware Hub Clocks

MC Clock The Management Controller chip receives a 12Mhz clock from the Y5
crystal oscillator attached to it. This is the operational clock used by the
MC.

U35
Y5 – Crystal Management
12MHz Controller
LPC2146

Figure 2-54. Management Controller Clock

EMC CONFIDENTIAL

106
IO Modules and There are 4 IO Module connectors on the Wildcat-S motherboard. Each
connector receives an 100Mhz differential clock which is used by the IO
Uarts Clocks Module (SLIC) inserted in each slot for PCI-Express Interface operations.

2
Each IO Module connector also receives an 33Mhz clock which may be
used by the different requirements of each IO Module.

The Dual and Quad UART’s both receive the 33Mhz signal which they
use as an operational clock. The 33Mhz signal is derived from the Y7
Oscillator which outputs a frequency of 33.333Mhz.

U52
U75 BYPASS
DB1200G
CK410B 100Mhz Clk Buffer
CK_DB1200 _SRC_+/- IO0_100MHZ_CLK_+/-
CLK_IN+/- DIF_6+/-
IO1_100MHZ_CLK_+/-
DIF_7+/-
IO2_100MHZ_CLK_+/-
DIF_8+/-
IO3_100MHZ_CLK_+/-
DIF_9+/-
Y8 – Crystal
14.318MHz

CBT3_33MHZ_CLK IO3_33MHZ_CLK J53_1S


OSC_33MHZ_CLK IO_33MHZ_CLK IO Card 3
Y7 – OSC Conn.
33.333MHz
U78 U77 CBT2_33MHZ_CLK U88 IO2_33MHZ_CLK J52_1S
IO Card 2
U136 Conn.
CLK CLK CLK
Dual DUART_33MHZ_CLK BUFFER BUFFER CBT1_33MHZ_CLK BUFFER IO_33MHZ_CLK J51_1S
UART IO Card 1
Conn.
U120
QUART_33MHZ_CLK CBT0_33MHZ_CLK IO0_33MHZ_CLK J50_1S
Quad
IO Card 0
UART Conn.

Figure 2-55. 33Mhz to Uarts and IO Modules

Midplane & The Debug Connector is not populated on the Wildcat-S motherboard but
it would have received the 100Mhz differential clock signals.
Debug
Connector Dreadnought’s midplane receives an 100Mhz differential clock signal
from U52 clock buffer.
Clocks

U52
U75 BYPASS
DB1200G
CK410B 100Mhz Clk Buffer
CK_DB1200 _SRC_+/- 2 DEBUG_100MHZ_CLK_+/-
CLK_IN+/- DIF_3+/- Debug Conn
Populate-false

2 MP_100MHZ_CLK_+/-
DIF_10+/- Midplane
Y8 – Crystal
14.31818 MHz

Figure 2-56. Midplane and Debug connector Clocks

EMC CONFIDENTIAL

Clocks & Interrupts 107


Interrupts Figure 2-57 shows the various interrupts on the Wildcat-S Blade. These
include hard wired interrupts, PCI Message Interrupts and similar
signals below.

DUART_IO5_INT
B
DUART
0 QUART_IO1_INT INT[A,B]
Clovertown Firmware XIRQ A
QUART_IO3_INT

THERMTRIP_N
0
(CPU0) Hub
DUART_IO4_INT
SIO
MCERR_N

IGNNE_N
FERR_N
BINIT_N

IERR_N
SMI_N 0 QUART_IO0_INT B
INIT_N

ACK D
INTR

A QUART
QUART_IO2_INT
NMI

0
SERIRQ
C
INT[A,B,C,D]
Legacy IRQs

0
GTL IGNNE_N FERR_N SERIRQ PIRQB
0
to INIT3_3N PIRQD
TTL GPIO8 PIRQA
0
GPIO12 0
PIRQC
INIT_N
INTR MSI/INTx Root
NMI PCI Express LAN1
SMI_N
ICH 7 MSI/INTx Root
PCI Express LAN2
MSI/INTx Root
MCERR_N
BINIT_N
INIT_N

IGNNE_N

IERR_N

THERMTRIP_N
INTR

SMI_N

FERR_N
NMI

Debug
PCI Express
connector
GPIO

MCU_I2C MCU
IDEIRQ Compact Flash
Clovertown Reset PIRQF
MCU PIRQG
(CPU1) Expander
ESI GPIOs PIRQE PIRQH

X4 ESI
Memory Write (MSI/INTx
FSB

FSB

ICH to Blackford )

FSB1_MCERR_N ESI ERR<2:0>


FSB1_BINIT_N
INT
FSB0_BINIT_N I2C
FSB0_MCERR_N
Expander
Memory Write (MSI/INTx Blackford to Processors 1) FSB1
Blackford
Memory Write (MSI/INTx Blackford to Processors 0) FSB0 MCH
Connector

PCI Express CMI Path


Mid Plane

or No Connect
PLX Switch PCI Express A

PCI Express MSI/INTx


or sRIO IO ANNEX Upstream
PCI Express C

PCI Express C
PCI Express B

PCI Express B

MSI/INTx
3:0

7:4

3:0

7:4

MSI/INTx MSI/INTx MSI/INTx


Upstream Upstream Upstream Upstream

Wildcat-N/S Interrupt IO 0 IO 1 IO 2 IO 3

Block Diagram Connector Connector Connector Connector


IO3_INS*
Feb. 15, 2006 IO2_INS*
IO1_INS*

IO0_INS*

IO_ANNEX_INS*
BTN_IS_SMI_N from Management Module

PIRQG From IO ANNEX

Figure 2-57. Wildcat-S Interrupts Block

SIO Interrupts The Server IO Chip receives two interrupt signals which are a combined
from the DUART & QUART chips as follows:
from DUART &
QUART • IO1_3_5_INT - Occurs when an incoming message is received by:
- the QUART from IO Module 1 or IO Module 3
- or the DUART from IO Module 5.
• IO0_2_4_INT - Occurs when an incoming message is received by:
- the QUART from IO Module 0 or IO Module 2
- or the DUART from IO Module 4.

EMC CONFIDENTIAL

108
ICH7 Interrupts The interrupts received by the ICH7 are described below:

• ICH_PIRQA interrupt is received from the QUART from IO Module 3 when

2
a message is in the receiver buffer.
• ICH_PIRQB interrupt is received from the QUART from IO Module 2 when
a message is in the receiver buffer.
• ICH_PIRQC interrupt is received from the QUART from IO Module 1 when
a message is in the receiver buffer.
• ICH_PIRQD interrupt is received from the QUART from IO Module 0 when
a message is in the receiver buffer.
• ICH_PIRQE interrupt is received from the U79 SMBUS Expander when a
IO Module or IO Annex Module insert signal changes voltage level.
• ICH_PIRQF interrupt is received from the I2C Expander when a Force SMI
is received from the MCU.
• ICH_PIRQG interrupt is received from the IO Annex.
• ICH_PIRQH interrupt is received from the MCU on the Management FRU
when the reset button has been depressed.

SIO
DUART_IO4_INT 0
QUART_IO0_INT ACK
0
QUART_IO2_INT

DUART_IO5_INT 0
QUART_IO1_INT XIRQ
0
QUART_IO3_INT SERIRQ

V3_3 ICH7
SERIRQ
0
PIRQA
0
PIRQC
0
LPC2146 PIRQB
0
MCU I2C V3_3 PIRQD
EXPNDR PIRQE
FORCE_SMI_N PIRQF
V3_3

14.3K
ICH_PIRQG_N IO_ANNEX_INT_N PIRQG
BTN_IS_SMI_N PIRQH
V3_3
14.3K
MP_BTN_IS_SMI_N

Figure 2-58. Interrupts received by SIO & ICH7

Other Interrupts The interrupts not covered here may be covered in other sections of the
manual if time allows.

EMC CONFIDENTIAL

Clocks & Interrupts 109


EMC CONFIDENTIAL

110
2.9 Chassis Management (LAN)

EMC CONFIDENTIAL

Chassis Management (LAN) 111


SAN and NAS The Dreadnought storage system can be configured to operate as either
an (SAN) Storage Area Network or (NAS) Network Attached Storage
type system. This is done by using different combinations of IO Modules
and Management FRU’s and Chassis.

SAN Chassis The Dreadnought system utilizes a distributed switch architecture for
management of the system. Each Wildcat-S Chassis has two Management
LAN Cabling FRU’s. Each FRU has a Management LAN port which is used to connect
to the control station. The service port may be used for servicing the
system. Figure 2-59 shows an example of the Ethernet cabling for a SAN
system comprising a single Wildcat-S Chassis and two control stations.

The below is for example only. See the Dreadnought System Configuration Guide for
actual configurations

Dreadnought SAN System

DAE

DAE

DAE

DAE

Wildcat-S Blade B

IO Module IO Module IO Module IO Module


PS-B PS-A

Wildcat-S Blade A

Solar Flare IO Module IO Module IO Module IO Module Solar Flare

Service IO ANNEX-B IO ANNEX-A


LAN S S

Management M IO Module IO Module IO Module IO Module M


LAN

Control Station

Control Station

SPS SPS

Figure 2-59. SAN Cabinet with one Wildcat-S Chassis

EMC CONFIDENTIAL

112
The Wildcat-S SAN Chassis in Figure 2-60 shows the LAN Ethernet
signals within a SAN type chassis. The chassis has two Solar Flare
Management FRU’s which contain an 6-port Ethernet switch to provide
the redundant fabrics. See the section about “Solar Flare - Management

2
FRU (SAN)” on page 147 for more details.

Power Supply - A
Solar Flare Management FRU – A
CROSSLINK_TD<3..0>+/- BMC5397 Switch Management
PORT 0 PORT 3
LAN Port
MGT_A_BLD_A_TD<3..0>+/-
PORT 2
J13 J1
J3 Service
PORT 1 PORT 4
LAN Port
Blower Fan

ITRAC
U8 U7 IO Module 3
LAN 1 LAN 2

ITRAC
IO Module 2
Wildcat-S
J5 Storage Processor
A

ITRAC
MGT_B_BLD_A_TD<3..0>+/-

IO Module 1
Blower Fan

ITRAC
IO Module 0

M
I J21 IO A
D
MGT_A_BLD_B_TD<3..0>+/-

Nova
Test J8 P IO ANNEX
Card
L
J7 IO B
Blower Fan A
N
E
ITRAC

U8 U7 IO Module 3
LAN 1 LAN 2
ITRAC

IO Module 2
Wildcat-S
MGT_B_BLD_B_TD<3..0>+/-

Blower Fan J6 Storage Processor


B
ITRAC

IO Module 1
ITRAC

IO Module 0

J14
Solar Flare Management FRU – A
BMC5397Switch
PORT 2 PORT 3 Management
PORT 1 LAN Port
J1 CROSSLINK_TD<3..0>+/-
J3 Service
PORT 0 PORT 4
LAN Port
Power Supply - B
Figure 2-60. Wildcat-S SAN Chassis - LAN Block Diagram

EMC CONFIDENTIAL

Chassis Management (LAN) 113


Wildcat-S LAN The below diagram in Figure 2-61 shows one of the two Broadcom
BCM5751 chips located on the Wildcat-S Blade. The BCM5751 chips use a
Ethernet Chip 1xPCI-e link to the ICH7 Interface Controller Hub. These two LAN chips
will be used to provide a 10/100/1000Mbit Ethernet MAC LAN
management interface from the Wildcat-S motherboard to the midplane.
The signals are then routed to the two BMC5397 Switches located on the
two Management FRU’s. The switch will use MAC addressing to direct
the ethernet traffic to/from the J45 Ethernet Connectors on the
Management FRU’s airdam.

LAN1_TRD0_C+ EXP_TO_ICH1_C+
LAN1_TRD0_C- BCM5751 EXP_TO_ICH1_C-
U18
LAN1_TRD1_C+ U8 EXP_FRM_ICH1_C+
ICH7
J22S LAN1_TRD1_C- EXP_FRM_ICH1_C-
Midplane LAN1_TRD2_C+
Connector LAN1_TRD2_C- LAN1_100MHZ_CLK+ U52
LAN1_TRD3_C+ LAN1_100MHZ_CLK+ Clk
Driver
LAN1_TRD3_C-
LAN1_EEDAT
LAN1_EECLK U15
EEPROM
LAN1_REFCLK_SEL LAN1_EEWP
V3_3
LAN1_ATTN_BTTN_N LAN1_XI
25 MHz Y2
LAN1_RDAC
LAN1_XO
LAN1_PCIE_TST
LAN1_TRST_N
LAN1_TCK
LAN1_TDI JTAG
U18 ICH_PLTRST_A_N LAN_RST_N LAN1_TMS
ICH7 V3_3 LAN1_TDO

Figure 2-61. BMC5751 LAN Chip Block Diagram

EMC CONFIDENTIAL

114
SAN In an SAN system there is only one Wildcat-S Chassis which has 2 Solar
Flare Management FRU’s installed. It may also include DAE’s, SPS and
Control Station.

SP B Blade
2
Empty SP A Blade slot

Service
LAN Port
Management
LAN Port
Solar Flare SAN IO Annex Solar Flare SAN
Mgnt. FRU A Slots Mgnt. FRU B

Figure 2-62. Wildcat-S Chassis - 2 Solar Flare FRU’s (rear view)

Solar Flare FRU Each Solar Flare FRU has two 10/100/1000 Ethernet ports:

• 1 LAN Service port can be used to access the system by service person.
• 1 LAN Management which connects to the customer LAN network for
management purposes.
Fault LED Power OK
LED

USB 2.0 port

Link LED

Activity
LED

Service
LAN
Console Port
Management
LAN
SPS Port
Activity
LED
Link LED
Figure 2-63. Solar Flare Management FRU Face Plate
Each RJ45 ethernet connector has a yellow port activity LED and a green
Link State LED.

EMC CONFIDENTIAL

Chassis Management (LAN) 115


Solar Flare Block The Solar Flare Management Module is an (SAN) type FRU. It has an
Diagram dual RJ45 connector which is used for Service and Management Ports.
These two ports are connected to a BCM5397 ethernet switch which
allows both ports to communicate with:

• Wildcat-S Blade A ethernet chip


• Wildcat-S Blade B ethernet chip
• Its peer Solar Flare Management Module - using the crosslink interface

Solar Flare Management Module (SAN)

USB Micro DB-9 LED’s Dual RJ-45


Power Fault
Service Service Debug Service Mgmt

Philips BCM5397 Gigabit


MUX LPC2131 Ethernet Switch
Microcontroller
USB 2.0
Hub
From Peer MGMT UART

Resume CMD
Prom Module

Peer MGMT.
Crosslink-
I2C
I2C - SDA/SCL

Blade A

Blade B
Isolation
RS232

RS232

Switch
USB

Midplane Connector

Figure 2-64. Solar Flare Interface Block

See“Solar Flare - Management FRU (SAN)” on page 147 for more details.

EMC CONFIDENTIAL

116
NAS Chassis In an NAS Dreadnought system there can be multiple Wildcat-S Chassis
which uses 2 EarthQuake Management FRU’s each. The NAS system may
LAN Cabling also include DAE’s, SPS and Control Station.

The system utilizes a distributed switch architecture for management of


the system. Each Wildcat-S Chassis has two Management Modules
containing an 6-port Ethernet switch to provide the redundant fabrics.
Figure 2-65 illustrates the Ethernet cabling for a NAS system comprising
2
multiple Wildcat-S Chassis and two control stations.

The below is for example only. See the Dreadnought System Configuration Guide for
actual configurations.

Dreadnought NAS System


To Next Wildcat-S To Next Wildcat-S
Chassis Chassis

Wildcat-S Blade B

IO Module IO Module IO Module IO Module


PS-B PS-A

Wildcat-S Blade A

Earthquake IO Module IO Module IO Module IO Module Earthquake

IO ANNEX-B IO ANNEX-A
U U

D C IO Module IO Module IO Module IO Module D C

Wildcat-S Blade B

IO Module IO Module IO Module IO Module


PS-B PS-A

Wildcat-S Blade A

Earthquake IO Module IO Module IO Module IO Module Earthquake

IO ANNEX-B IO ANNEX-A
UP-LINK U U

D C IO Module IO Module IO Module IO Module D C


DOWN-LINK

CS / UPS
Control Station

Control Station

UPS UPS

Figure 2-65. Dreadnought NAS System LAN - Example Only


EMC CONFIDENTIAL

Chassis Management (LAN) 117


The distributed switch architecture allows for unlimited growth of a
Dreadnought NAS system since every Wildcat-S Chassis provides two
more switches to the overall system. Cabling for a system is always
started with the control station(s) wired to the first Chassis and the
remaining Chassis are cascaded via the uplink and downlink ports. The
last Chassis in the system will always have an unused uplink port to
allow for future expansion.

The design incorporates two completely independent fabrics for


redundancy with no interconnections between them. Both the control
station(s) and the Blades of the system have redundant MACs so that
they have a connection to both fabrics. By doing so, a failure on either
fabric will not prevent the operation of the system. The only device,
which resides only on one fabric, is the management micro controller.
Since there are two management controllers within each Wildcat-S
Chassis redundancy is handled by the second micro controller in the
event that a fabric fails.

EMC CONFIDENTIAL

118
The Wildcat-S NAS Chassis in Figure 2-60 shows the LAN Ethernet
signals. The chassis has two Earthquake Management FRU’s which have
three ethernet LAN ports. See the section about “Earthquake -
Management FRU (NAS)” on page 169 for more details.

Power Supply - A
PORT 2 PORT 3 CS1 or UPS Port
2
MGT_A_BLD_A_TD<3..0>+/-
J2 BMC5397
J13
J4 EarthQuake Mgnt FRU B Switch PORT 0 Up-Link Port
PORT 1 PORT 4 Down-Link Port

Blower Fan

ITRAC
U8 U7 IO Module 3
LAN 1 LAN 2

ITRAC
IO Module 2
Wildcat-S
J5 Storage Processor
A

ITRAC
MGT_B_BLD_A_TD<3..0>+/-

IO Module 1
Blower Fan

ITRAC
IO Module 0
M
I
D J21 IO A
MGT_A_BLD_B_TD<3..0>+/-

Nova P
Test J8
L IO ANNEX
Card
A J7 IO B
Blower Fan N
E
U8 U7
ITRAC

IO Module 3
LAN 1 LAN 2
ITRAC

IO Module 2
Wildcat-S
MGT_B_BLD_B_TD<3..0>+/-

Blower Fan J6 Storage Processor


B
I TRAC

IO Module 1
ITRAC

IO Module 0

PORT 2 PORT 3 CS1 or UPS Port


J14 J2 BMC5397
J4 EarthQuake Mgnt FRU B Switch PORT 0 Up-Link Port
PORT 1 PORT 4 Down-Link Port

Power Supply - B
Figure 2-66. Dreadnought NAS - LAN Block Diagram

EMC CONFIDENTIAL

Chassis Management (LAN) 119


EarthQuake Each EarthQuake FRU has three 10/100/1000 Ethernet ports:
FRU • 1 Uplink port
• 1 Down link port
• 1 Control Station or UPS LAN port.

Figure 2-67. Earthquake Management FRU Face Plate

EMC CONFIDENTIAL

120
Earthquake Block The Earthquake Management Module is an (NAS) type FRU. It has three
Diagram RJ45 connectors which is used for:

• Up-link Port

2
• Down-link or Control Station Port
• Second Control Station or UPS Port.
These three ports are connected to a BCM5397 ethernet switch which
allows the ports to communicate with:

• Wildcat-S Blade A ethernet chip


• Wildcat-S Blade B ethernet chip
• And other Wildcat-S Chassis in the Dreadnought NAS System
Earthquake Management Module (NAS)
7 Segment
USB Micro DB-9 Display LED’s Three RJ-45 Ports
Power Fault
NMI Down-Link Service -
Service Service Up-Link
Button Service UPS

Motorola BCM5397 Gigabit


MUX Coldfire Ethernet Switch
Microcontroller
USB 2.0
Hub
From Peer MGMT UART

Resume CMD
Prom Module

Peer MGMT.
Crosslink-
I2C
Blade A

Blade B
Isolation
PS1 - I2C

PS2 - I2C
RS232

Switch
USB

Midplane Connector

Figure 2-68. Earthquake Interface Block

EMC CONFIDENTIAL

Chassis Management (LAN) 121


EMC CONFIDENTIAL

122
2.10 Chassis Management (RS232)

EMC CONFIDENTIAL

Chassis Management (RS232) 123


RS232 The Wildcat-S has 9 Serial RS232 UART Ports used for communication on
the Wildcat-S board:
Interfaces
• 4 Ports from the QUAD UART connect to IO Modules 0-3. These interfaces
are used for back-end diplexing when communicating to DAE’s.
• 2 Ports from the DUAL UART connect to IO Modules 4,5. These interfaces
pass through the Tornado IO Annex Expander board to the 2 IO Modules
and are used for Diplexing communication to DAE.
• 2 Ports from the Super IO chip are wired to the Management FRU and
connect to the Console and SPS Mini DB9 sockets on the FRU’s airdam.
• The last serial ports connect the MCU via a header to a debug PC to allow
MCU code debug. This will most likely be depopulated in future revisions.
Wildcat-S UART Ports
IO Module 0 Wildcat-S Tornado IO Module 4
DUART_FRM_IO0 DUART_TO_IO4 IO Annex
Diplex Diplex
DUART_TO_IO0 DUART_FRM_IO4
FPGA FPGA
DUART

IO Module 1 IO Module 5
DUART_FRM_IO1 DUART_TO_IO5
Diplex Diplex
DUART_TO_IO1 DUART_FRM_IO5
FPGA FPGA

Midplane
Fogbow
QUART

IO Module 2
DUART_FRM_IO2 Management Module
Diplex

XCVR
DUART_TO_IO2 SIO_RXD0/TXD0 To/From
FPGA
SPS
SIO

IO Module 3
DUART_FRM_IO3
XCVR

Diplex
DUART_TO_IO3
SIO_RXD1/TXD1 To/From
FPGA Console
MCU

MCU_RX0/TX0
HDR

XCVR

PC_RX/TX

Figure 2-69. Wildcat-S RS232 UART Ports

EMC CONFIDENTIAL

124
DUART A Texas Instruments TL16C752B Dual UART is connected to the Super IO
XBUS to provide two added serial ports for Wildcat. These two serial
ports are used to communicate to an FPGA on each IO Module connected

2
to Wildcat via the IO Annex Module, that will handle the communication
for diplexing to the Fibre ports.

U141 XA2
26
U136 5
DUART_FRM_IO4
SERIAL A
Server IO XA1
27
DUART To Midplane 7
DUART_TO_IO4

XA0 IO4 SLIC FLOWCTL_TO_IO4


28 33
XAD<7:0> U108 To
DUART_FRM_IO5 Switch Midplane
XRD_N SERIAL B 4
19 DUART_TO_IO5 IO4 & IO5
To Midplane 8
XWR_N IO5 SLIC FLOWCTL_TO_IO5
15 22
DUART_IO4_INT
30
DUART_CDA_N
DUART_IO5_INT 40
29 DUART_DSRA_N
SERIAL A 39
COM7_CS_N DUART_DTRA_N
10 STATUS 34
DUART_CS_N U135 COM8_CS_N CTRL PUP_DUART_RIA_N
11 41
PDN_DUART_CTSA_N
38

U18 Misc. DUART_RESET_0


36 DUART_CDB_N
ICH7 Circuitry 16
DUART_DSRB_N
SERIAL B 20
DUART_DTRB_N
STATUS 35
DUART_33MHZ_CLK CTRL PUP_DUART_RIB_N
U78 13 21
PDN_DUART_CTSB_N
23

Y7
33.33Mhz

Figure 2-70. DUART for IO4 & IO5 Diplexing

The Diplex FPGA is capable of supporting 8 separate ports for Diplexing.


The DUART ports could support these 8 ports running at 19.2K BAUD.

Only TX, RX, and RTS will be supported. Other signals are not routed.

EMC CONFIDENTIAL

Chassis Management (RS232) 125


QUART A Texas Instruments TL16C754B Quad UART is connected to the Super
IO XBUS to provide four added serial ports for Wildcat.

U141 XA2
28
U120 77
DUART_FRM_IO0

Server IO XA1 QUART To IO0 DUART_TO_IO0 IO0


29 Connector 10 Conn.
XA0 FLOWCTL_TO_IO0
30 7
XAD<7:0>
DUART_FRM_IO1
XRD_N 4
51 To IO1 DUART_TO_IO1 IO1
Connector 12 Conn.
XWR_N FLOWCTL_TO_IO1
11 25
QUART_IO0_INT DUART_FRM_IO2
8 37
To IO2 DUART_TO_IO2 IO2
QUART_IO2_INT Connector 50 Conn.
48 FLOWCTL_TO_IO2
47
QUART_IO1_INT
14 DUART_FRM_IO3
65
To IO3 DUART_TO_IO3 IO3
QUART_IO3_INT Connector 52
54 FLOWCTL_TO_IO3 Conn.
55

DUART_IO0_CDA_N
79
DUART_IO0_DSRA_N
Serial A 3
COM3_CS_N DUART_IO0_DTRA_N
9 STATUS 5
COM4_CS_N CTRL PUP_DUART_IO0_RIA_N
13 78
U93 COM5_CS_N PDN_DUART_IO0_CTSA_N
4
49
QUART_CS_N COM6_CS_N
53 DUART_IO1_CDB_N
23
DUART_IO1_DSRB_N
Serial B 19
U18 Misc. DUART_RESET_1 STATUS 17 DUART_IO1_DTRB_N
33 CTRL
ICH7 Circuitry
24
PUP_DUART_IO1_RIB_N
PDN_DUART_IO1_CTSB_N
18

QUART_33MHZ_CLK DUART_IO2_CDC_N
U78 31 39
DUART_IO2_DSRC_N
Serial C 43
STATUS DUART_IO2_DTRC_N
45
CTRL PUP_DUART_IO2_RIC_N
Y7 38
33.33Mhz PDN_DUART_IO2_CTSC_N
44

DUART_IO3_CDD_N
63
QUART_CLKSEL DUART_IO3_DSRD_N
PU 26 Serial D 59
DUART_IO3_DTRD_N
STATUS 57
CTRL PUP_DUART_IO3_RID_N
64
PDN_DUART_IO3_CTSD_N
58

Figure 2-71. Quad UART Pinout


These four serial ports are used to communicate to an FPGA on each IO
Module connected to Wildcat-S via the4 IO Module connectors. These
serial ports will handle the communication for diplexing when a Fibre
SLIC is present and used as a back-end ports.

EMC CONFIDENTIAL

126
The Diplex FPGA on the IO Modules are capable of supporting 8 separate
ports for Diplexing. The DUART ports could support these 8 ports
running at 19.2K BAUD.

2
Only TX, RX, and RTS (flow control) will be supported. Other signals are
not routed.

EMC CONFIDENTIAL

Chassis Management (RS232) 127


EMC CONFIDENTIAL

128
2.11 Chassis Management (I2C)

SAN vs NAS I2C Wildcat-S supports both Dreadnought SAN and NAS system
management architectures which are incompatible. This is done by using
Architecture two different types of Management FRU’s and having the Wildcat-S
board read the type of Management FRU’s installed and the midplane
resume (VPD) chip which contains the system type. Once the Wildcat-S
knows this information it will load use the appropriate software code
which configures the system as NAS or SAN.

I2C Access There are two I2C access types used in a Wildcat-S Chassis.
Types • The first type is a simple scheme and only involves accessing I2C buses
local to the Wildcat-S Blade and its components. The blade includes local
I2C interfaces to the IO Modules. Arbitration signals are required to be used
for some but not all I2C buses.

• The second type involves access to the midplane I2C interfaces (PS_A_I2C
& PS_B_I2C) which will require arbitration between multiple I2C masters
which are located on different Modules within the Wildcat-S chassis.

EMC CONFIDENTIAL

Chassis Management (I2C) 129


Midplane I2C The midplane I2C buses for the power supplies are crossed on the
Wildcat-S Chassis midplane. This prevents I2C conflicts since both blades
Buses (SAN) will have a CMD, resume prom, switch, and 8 bit Fault expanders. It also
prevents confusion as to which power supply is actually on that bus. For
example, blade A would see PSA on the PS_A I2C bus and blade B would
see PSA on the PS_B I2C bus.

Wildcat-S Chassis SAN - PSA & PSB I2C

IO Annex A PIC
Solar Flare A PIC
Solar Flare B IO Annex B
0x60 CMD CMD
0x60
0x?? 0x??

Resume Resume
PROM 0xAE PROM 0xAE
0 1 0 1
Sw 0 xEA Sw 0 xEA

I2C I2C
Devices Devices

PS A PS B
Micro Micro
0x52 Resume 0x52
PROM 0xAA
Resume PS_A_I2C Resume
PROM0xAC PROM0xAC
PS_B_I2C

Fogbow
MidPlane

Wildcat-S A
8 bit I2 C
expander
0x4E
Redundant
Reset
Hold Post
Wildcat-S B 8 bit I2 C
expander
0x4E
Redundant
Reset
Hold Post
CMD CMD
Master for SAN U23 Master for SAN 2
U23
I2 C switch 0X24 I C switch 0X24
PECI 0xE4 PECI
0xE4 0x54 0x54
ICH7 1 3 4 5 6 0
ICH7 1 3 4 5 6 0
0x88 0x88
U18 slave I2C switch U18 slave I 2C switch
master 0XE6 master 0XE6

1 of 4 MUX SEL0 To I2C I2C I2C I2C 1 of 4 MUX SEL 0 To I2C I2C I2C I2C
SEL1 SIO SEL 1 SIO
Devices Devices Devices Devices Devices Devices Devices Devices
11 01 10 00 11 01 10 00

Resume Resume
(0xA4) (0xA4)
Clock Clock

IO Module IO Module IO Module IO Module IO Module IO Module IO Module IO Module


8 bit I2 C 0 1 2 3 8 bit I2 C 0 1 2 3
expander GPIO/ Status bits expander GPIO/ Status bits
(0x4C) Reset (0x4C) Reset
Reset PIC Hold Post Reset PIC Hold Post
0x3E 0x3E

MCH MCH
0XC0 0XC0

Figure 2-72. Wildcat-S Chassis Midplane I2C Interfaces (SAN)

EMC CONFIDENTIAL

130
SAN - I2C In a SAN Dreadnought system each Wildcat-S chassis may contain 2
Solar Flare Management Modules, 2 Wildcat-S Blades which include IO
Modules, IO Annex Modules and 2 power supplies which are not show in

2
Figure 2-73. Each Wildcat-S Blade has one I2C master which is the ICH7.
The Solar Flare does not have a I2C master and the arbitration signals are
not used by the Solar Flare FRU as indicated in Figure 2-73. The ICH7
(I2C master) on the Wildcat-S Blades use the I2C_ARB_A1 &
I2C_ARB_B1 signals to arbitrate for access to the (PS_A_I2C & PS_B_I2C)
midplane I2C buses. See Figure 2-74. This will allow the ICH7 to access
components on both itself and its Peer Wildcat-S Blade as well as the
midplane resume (VPD), Management FRU, both power supplies and IO
Annex modules.

J4 SolarFlare Mgnt FRU A

I2C_ARB_A1_BUF_0_N U27
U18

ITRAC
IO Module 3
ICH I2C_ARB_A1_BUF_IN
U28
MP_I2C_ARB_A1
U99

ITRAC
I2C_ARB_A1

IO Module 2

J5
Wildcat-S SP
A

ITRAC
MP_I2C_ARB_B1
IO Module 1
I2C_ARB_B1
U104

M U144 U141
ITRAC
I SIO_ARB_B1 SIO IO Module 0

D
I2C_ARB_B1

P I2C_ARB_A1_BUF_0_N U27
U18
ITRAC

L ICH I2C_ARB_A1_BUF_IN
IO Module 3
A U28
MP_I2C_ARB_A1
N U99
ITRAC

IO Module 2
E
J5
Wildcat-S SP
B
ITRAC

MP_I2C_ARB_B1
IO Module 1
I2C_ARB_B1
U104

U144 U141
ITRAC

SIO IO Module 0
SIO_ARB_B1

J2
J4 SolarFlare Mgnt FRU B

Figure 2-73. SAN I2C Arbitration Signals

EMC CONFIDENTIAL

Chassis Management (I2C) 131


SAN I2C Bus • The I2C Master on one of the two Wildcat-S Blades installed in the Chassis
pulls down either I2C_ARB_A1 or I2C_ARB_B1 signal to begin arbitrating
Arbitration for either PS_A_I2C or PS_B_I2C I2C bus.
Flowchart • It then reads back the value through its feedback loop to verify the
arbitration signal is low.
• The I2C master releases the control signal and checks the value through its
feedback loop to verify the signal is high indicating its peer is not trying to
access the same I2C bus.
• I2C master pulls the arbitration signal low again and checks its value.
• I2C master does its I2C read/write and releases the arbitration signal.

Start Request for I2C Bus

Is NO Did Signal YES Wait >=10 * t1


Control Signal go High before (Prevents
High? Timeout ? Deadlock)

YES
NO
Pull Control Signal t(n) = n*t1
Low for T = t(n) t(n) – Time delay for
each host trying
to get on I2C bus.
Report Failure on
Is NO one Control Signal n – Host Number
Control Signal Blade A = 1
Continue using only
Low?
remaining Blade B = 2
YES

Release
t1 = 20us
Control Signal t2 = 10us
for T = t2

Is
Control Signal NO
High?

YES
The Control Signal will be either
Pull Control Signal I2C_ARB_A1 or I2C_ARB_B1
Low until done
depending upon which I2C bus
with I2C bus
is being arbitrated for.

Is
Control Signal NO
Low? t2
MP_I2C_ARB_(A/B)1
YES
One Control
Done, Success Signal

Host can
t(n) connect to
Wildcat-S SAN _I2C Arbitration Flow .emf I2C Bus
Figure 2-74. Midplane SAN I2C Access Arbitration Diagram
EMC CONFIDENTIAL

132
Midplane I2C The midplane I2C buses for the power supplies are crossed on the
Wildcat-S Chassis midplane. This prevents I2C conflicts since both blades
Buses (NAS) will have a CMD, resume prom, switch, and 8 bit Fault expanders. It also

2
prevents confusion as to which power supply is actually on that bus. For
example, blade A would see PSA on the PS_A I2C bus and blade B would
see PSA on the PS_B I2C bus.

Dreadnought NAS I2C with Wildcat-S CPU Module Rev 1.7

Earth Quake A Earth Quake B


0B D0 04 C Dx

IO Module 4
0B D00 4 CDx

Thunderbolt

Expander 0xC2
IO Module 4
Thunderbolt

PCA 9557 CY28400


Expander 0xC2

CY28400 PCA 9557

7
Resume 0xA8
7

Resume 0xA8

XEP 5 05 8 w(s )

CMD 0x20
dsPIC33F
XEP 50 58 w(s )
CMD 0x20

0x38 0xDC

PCA9552
dsPIC33F

0xDC

24LC32A
PCA9552

0x38
24LC32A

6
6

PCA9548
Sw 0xE0
MCB 5 175 (s )
w
PCA9548

MCB 517 5 (s )
w
Sw 0xE0

SFPs 0xA0
SFPs 0xA0

MCU

5
MCU
5

0
MCB 5 17 5 (s )
w
0
MCB 517 5 (s )
w
0x90

4
0x90
4

Coldfire 24LC32A Coldfire PCA

2 3
3

24LC32A
2 PF S
1 PCA Resume
2 PF S
PCA 9557 0x32 PCA 9557 0x30 9545 1
2

Resume PROM0xAE
1 PF S
9545
1 PF S

1
PROM0xAE
1

2 2
SW

0
SW
0

0xE2 0xE2 0B D0 04 CDx

IO Module 5
0B D00 4 CDx

Thunderbolt
IO Module 5

Expander 0xC2
Thunderbolt
Expander 0xC2

7
Resume 0xA8
7

Resume 0xA8

3 PCA 9546 PCA 9546 CMD PCA 9546 PCA 9546 CMD 3
XEP 5 05 8 w(s )

CMD 0x20
dsPIC33F
XEP 50 58 (s )
w

PCA9552
CMD 0x20

24LC32A
PCA9552

dsPIC33F

0x22
24LC32A

0xEE 0xEA 0x22 0xEA 0xE8

6
6

PCA9548
MCB 51 75 (s )
w
PCA9548

Sw 0xE0
MCB 51 75 (s )
w
Sw 0xE0

SFPs 0xA0
SFPs 0xA0

0 0 0 0

5
24LC32A
5

24LC32A
MCB 51 75 (s )
w MCB 51 75 (s )
w

4
Resume
4

Resume

3
0xA6
3

2 PF S
2 PF S 0xA6

2
2

1 PF S 1 PF S

Tornado A Tornado B

1
1

0
0

24LC32A
Resume Fogbow Midplane
0xAA
PS A PS B
Micro PS_A_I2C
0x52 Micro
24LC32A PS_B_I2C 0x52
Resume 24LC32A
PROM 0xAC Resume
PROM 0xAC

Wildcat-S A Wildcat-S B
Redundant FAULT_REG (Bank 0 bits 7:0) Redundant FAULT_REG (Bank 0 bits 7:0)
16 bit I2C RST_FRM_PEER_N (Bank 1 bit 5) 16 bit I2C RST_FRM_PEER_N (Bank 1 bit 5)
expander PERSIST_RST_N (Bank 1 bit 6) expander PERSIST_RST_N (Bank 1 bit 6)
Master for SAN 0x4E HLD_IN_PST_N (Bank 1 bit 4) dsPIC33F Master for SAN 0x4E HLD_IN_PST_N (Bank 1 bit 4)
PCA9673 CMD PCA9673 dsPIC33F
ICH slave Sequencer ICH slave CMD
0x88 0x88 Sequencer
7 I2C switch 0x24 7 I2C switch
master PCA9548 PECI master PCA9548 PECI 0x24
0xE4 0xE4
0 1 2 3 4 5 6
MAX6621 0 1 2 3 4 5 6
MAX6621
0x54 0x54
1 1
PCA9543 PCA9543
1 of 4 MUX SEL0 I2C switch 1 of 4 MUX SEL0 I2C switch
To 0XE6 To 0XE6
CBTLV3253 SEL1 SIO CBTLV3253 SEL1 SIO
11 01 10 00 11 01 10 00

Thunderbolt Thunderbolt Tomahawk Tomahawk Thunderbolt Thunderbolt Tomahawk Tomahawk


IO Module 0 IO Module 1 IO Module 2 IO Module 3 IO Module 0 IO Module 1 IO Module 2 IO Module 3
Resume 0xA8 Resume 0xA8 Resume Resume Resume 0xA8 Resume 0xA8 Resume Resume
24LC32A 24LC32A 24LC32A 0xA8 0xA8 24LC32A 24LC32A 24LC32A 0xA8 0xA8
Resume 24LC32A 24LC32A Resume 24LC32A 24LC32A
PROM 0xA4 CMD 0x20 CMD 0x20 CMD 0x20 CMD 0x20
CMD CMD PROM 0xA4 CMD CMD
dsPIC33F dsPIC33F dsPIC33F dsPIC33F
0x20 0x20 0x20 0x20
Expander 0xC2 Expander 0xC2 dsPIC33F dsPIC33F Expander 0xC2 Expander 0xC2 dsPIC33F dsPIC33F
PCA9673 FAULT_REG<7:0> PCA9552 PCA9552 PCA9673 FAULT_REG<7:0> PCA9552 PCA9552
16 bit I2C RST_FRM_PEER _N Sw 0xE0 Sw 0xE0 16 bit I2C RST_FRM_PEER_N
PERSIST_RST_N PERSIST_RST_N
Sw 0xE0 Sw 0xE0
expander PCA9548 PCA9548 expander
HLD_IN_PST HLD_IN_PST PCA9548 PCA9548
0x4C 0 1 2 3 4 5 6 7 0 1 2 3 4 5 6 7 0x4C 0 1 2 3 4 5 6 7 0 1 2 3 4 5 6 7
LPC2146 LPC2146
MCU Slv MCU Slv
PEX8505 (sw)

PEX8505 (sw)

MCU
PEX8505 (sw)

PEX8505 (sw)

MCU
DB400 0xDC

DB400 0xDC
BCM5715 (sw)
BCM5715 (sw)

BCM5715 (sw)
BCM5715 (sw)

DB400 0xDC

DB400 0xDC
BCM5715 (sw)
BCM5715 (sw)

BCM5715 (sw)
BCM5715 (sw)

0x92 0x92
Clocks 0xD2, 0xD6, 0xDE Clocks 0xD2, 0xD6, 0xDE
SFP1
SFP2

SFP1
SFP2

SFP1
SFP2

SFP1
SFP2

MCH 0XC0 MCH 0XC0


LM75 0X90 LM75 0X90

SFPs 0xA0 SFPs 0xA0 SFPs 0xA0 SFPs 0xA0

Figure 2-75. Wildcat-S Chassis Midplane I2C Interfaces (NAS)

EMC CONFIDENTIAL

Chassis Management (I2C) 133


NAS - I2C The NAS Dreadnought system has four I2C masters within the system.
Each Wildcat-S Blade has one ICH7 each which is an I2C master and the 2
Earth Quake Management Modules use an Coldfire Micro controller as a
I2C Master. The Coldfire MCU’s are connected to the two arbitration
signals I2C_ARB_A1 & I2C_ARB_B1 as well as the two Wildcat-S Blades.
The 4 I2C masters use the arbitration signal to avoid bus contention when
more then one master wants to access the same I2C bus at the same time.

EarthQuake Mgnt FRU B


U6

J4 U21
MCU
J2
U15

I2C_ARB_A1_BUF_0_N U27
U18

ITRAC
IO Module 3
ICH I2C_ARB_A1_BUF_IN
I2C_ARB_A1

U28
MP_I2C_ARB_A1
U99

I TRAC
IO Module 2

J5
Wildcat-S SP
A
I2C_ARB_B1

ITRAC
MP_I2C_ARB_B1
IO Module 1
I2C_ARB_B1
U104

M
U141
I U144

ITRAC
SIO IO Module 0
SIO_ARB_B1
D
P
I2C_ARB_A1_BUF_0_N U27
L U18
ITRAC

IO Module 3
ICH I2C_ARB_A1_BUF_IN
A U28
N MP_I2C_ARB_A1
U99
ITRAC

E IO Module 2

J5
Wildcat-S SP
B
ITRAC

MP_I2C_ARB_B1
IO Module 1
I2C_ARB_B1
U104

U144 U141
ITRAC

SIO IO Module 0
SIO_ARB_B1

U15
J2

J4 U21
MCU
U6

EarthQuake Mgnt FRU B


Figure 2-76. NAS I2C Arbitration Signals

EMC CONFIDENTIAL

134
NAS I2C Bus • The I2C Master on one of the two Wildcat-S Blades or one of the two Earth
Quake Management FRU’s installed in the Chassis pulls down either
Arbitration I2C_ARB_A1 or I2C_ARB_B1 signal to begin arbitrating for either

2
Flowchart PS_A_I2C or PS_B_I2C I2C bus.
• The master device then reads back the value through its feedback loop to
verify the arbitration signal is low.
• The I2C master releases the control signal and checks the value through its
feedback loop to verify the signal is high indicating its peer is not trying to
access the same I2C bus.
• I2C master pulls the arbitration signal low again and checks its value.
• The I2C master does its I2C read/write and releases the arbitration signal.

Start Request for I2C Bus

Is NO Did Signal YES Wait >=10 * t1


Control Signal go High before (Prevents
High? Timeout ? Deadlock)
YES
NO t(n) = n*t1
Pull Control Signal
Low for T = t(n)
t(n) – Time delay for
each host trying
to get on I2C bus.
Report Failure on n – Host Number
Is NO one Control Signal Blade A = 1
Control Signal
Continue using only
Low? Blade B = 2
remaining
YES
Earthquake A = 3
Earthquake B = 4
Release t1 = 20us
Control Signal t2 = 10us
for T = t2

Is
Control Signal NO
High?

YES
The Control Signal will be either
Pull Control Signal I2C_ARB_A1 or I2C_ARB_B1
Low until done
depending upon which I2C bus
with I2C bus
is being arbitrated for .

Is
Control Signal NO
Low? t2
MP_I2C_ARB_(A/B)1
YES
One Control
Done, Success Signal

Host can
t(n) connect to
Wildcat-S I2C Arbitration Flow .emf I2C Bus
Midplane I2C Access Arbitration Diagram
EMC CONFIDENTIAL

Chassis Management (I2C) 135


Wildcat-S ICH7 The ICH7 U18 has its Master I2C port connected to the 1x4 MUX U83 for
system wide access to I2C components. See Figure 2-78 for visual
I2C Master reference. The 4 Mux buses are as follows:
Access
• Bus 11 of the U83 MUX is connected to the Blackford MCH I2C slave port,
the LM75 temperature sensor, the CK410B clock buffer, the PCI Express
DB1200G clock buffer, and the FBDIMM BD1200G clock buffer.

• Bus 01 is connected to the U35 MCU I2C slave port. If the CPU (via the
ICH7) wants to access the MCU slave port, it must arbitrate for ownership of
the PS2_I2C bus first. This will lock out access to the MCU slave port from
its peer CPU and the 2 EarthQuake Management FRU’s in a NAS system by
holding the U64 8-port I2C switch in reset. See Figure 2-78.

• Bus 10 provides access to the PS_A_I2C bus. On this bus resides power
supply A or B depending which slot the Wildcat-S Blade is inserted. The bus
also has connectivity to the Wildcat-S midplane resume proms, a 16 bit I2C
fault expander U74 which is connected to the U141 Server IO GPIO status
bits and is used to monitor the state of the CPU Module. The CPU must
arbitrate for the PS_A_I2C bus by using the I2C_ARB_A1 control signal out
of the ICH7.

• Bus 00 provides access to the PS_B_I2C bus. On this bus resides power
supply B, the 16 bit AUX expander U73 (PCA9673) used as a redundant
Fault Register, Wildcat-S devices U23 Control Monitor Device (CMD)
which provides information on the state of the processors and on board
voltages. The U66 MAX6621 which supplies a PECI interface to the 2
CPU’s. An 8-port I2C switch U64 provides access to I2C components on the
IO Modules (0-3), and the ICH7's slave interface. The CPU must arbitrate
for the PS_B_I2C bus by using the I2C_ARB_B1 control signal out of the
Server IO chip.

On power up, the 1x4 mux defaults to bus 11.

Figure 2-77. ICH7/MCH I2C Master Address Map

EMC CONFIDENTIAL

136
I2C Interface The below illustration shows the I2C interfaces on the Wildcat-S
motherboard. There are two interfaces connected out to the midplane
Block MP_PS1_I2C & MP_PS2_I2C. They connect to all the various modules

2
installed in the Wildcat-S Chassis. There are four I2C interfaces connected
one each to the Wildcat-S Blades 4 IO Module connectors.

NOTE:
Signals ending in SMB represent two signals ending in
Blackford SDA and SCL
MCH Signals ending in I2C represent two signals ending in
V3_3 SDA and SCL

FBDIMM #01 FBDIMM #00 4.7k 4.7k 4.7k 4.7k


EEPROM ADR 0xA2 EEPROM ADR 0xA0 U65
AMB ADR 0xB2 AMB ADR 0xB0
MCH_SPD0_SMB CPU0 CPU1
SPD SMBus 0 LM75
VRM11 VRM11
FBDIMM #11 FBDIMM #10
EEPROM ADR 0xA2 EEPROM ADR 0xA0 ADR 0xE0 ADR 0xE2 ADR 0x90
AMB ADR 0xB2 AMB ADR 0xB0
MCH_SPD1_SMB
SPD SMBus 1 0 0
FBDIMM #21 FBDIMM #20
EEPROM ADR 0xA2 EEPROM ADR 0xA0 CFG SMBus
AMB ADR 0xB2 AMB ADR 0xB0
MCH_SPD2_SMB ADR 0xC0
SPD SMBus 2
FBDIMM #31 FBDIMM #30 CK410B
EEPROM ADR 0xA2 EEPROM ADR 0xA0 IDTCV126
AMB ADR 0xB2 AMB ADR 0xB0 CLK Generator
MCH_SPD3_SMB SPD SMBus 3 ADR 0xD2
GPIO SMBus
DB1200G
IDTCV128
V3_3 PCIe CLK Buffer
V3_3
ADR 0xD6 Bypass

4.7k DB1200G
IOANNEX_INS* 10k
PCA9554 IDTCV126
HP_SMB_SMB FBD CLK Buffer
IO[3:0]_INS* SMBus ADR 0xDE Div/2
Expander
ADR 0x40 U79

MEM_I2C
11
14.3k

CLOCK_I2C
V3_3

V3_3 1 of 4
Mux
4.7k 01
U83 24LC32A
ICH7 Resume VCC_CMD
CBTLV3253

V3_3
SMBus 0 Ohm Prom
U18 TRUNC_I2C
Master ADR 0xA4
14.3k
14.3k
SMLink 0/1
ADR 0x88 Slave PS1_I2C CBTLV MP_PS1_I2C
10 3245
14.3k

V3_3

V3_3 V3_3
LPC2146 PCA9673
Management 14.3k I2C Primary
4.7k
Controller

Midplane
PCA9543 Expander 11
U35 S[1:0] ISO_PS2_I2C 0 Fault_REG[7:0] FROM SIO
MCU_I2C_R I2C U74
00 ADR 0x4C
HLD_IN_PST*
Master Slave Switch 1 RST_FRM_PEER*
ADR 0x92 SMB_SEL[1:0] ADR 0xE6
SIO PERSIST_RST*
PCA9673
Reset U141 I2C 11
VCC_CMD
Exp. PCA9548 Expander
ICH_SLAVE_I2C
U56 0 Aux
U73 14.3k
ADR 0x4E
IO(3-0) 1
IO3_I2C MP_PS2_I2C
PwrEn IO3 6 PS2_I2C CBTLV
I2C
U59 IO2_I2C 3245
Switch
IO(3-0) IO2 5 U69
Flt. Leds IO1_I2C
U61 IO1 4
dsPIC33F Note: All components within
IO0_I2C ADR 0xE4 CMD the dotted red line area will
IO(3-0) IO0 3
Reset-Out U23 be powered by VCC_CMD
Clovertown ADR 0x24 which is generated by a
U60 MAX6621
CPU 1 PECI linear regulator connected
IO(3-0) PECI
ISSP Header to V+12_SBY (A OR B)
PwrGd Clovertown U66
U57 CPU 0 ADR 0x54

IO(3-0)
Insert
U58

Figure 2-78. Wildcat-S I2C Interfaces

The Wildcat-S Resume PROM and the CMD (Wildcat-S) will be powered
by VCC_CMD which is powered by 12V Standby A OR B even if the
Wildcat-S Blade itself is powered off. This allows one Wildcat-S Blade to
read the Resume and CMD of its peer blade at any time, even if the peer
board fails to power up completely. The CMD controls the power enable
and power good signals to/from the DC-DC power circuits. The CMD
may have power fault and other information in its registers.

EMC CONFIDENTIAL

Chassis Management (I2C) 137


MCU I2C The LPC2146 Management Controller (MC) U35 has two I2C ports.

One port is used as a slave only port connected to the U83 MUX port '01'
and the 8 port I2C switch U69 by the CLOCK_I2C interface. The slave
only port is used to communicate with and control the MCU.

The second I2C port will be configured as a I2C master port. The master
port connects to a number of I2C expanders and will be discussed in the
section on “MCU Master I2C Port” on page 140.

Local Wildcat-S In a SAN or NAS system when the CPU wants to access the local MCU
I2C slave port, the CPU needs to arbitrate for the PS2_I2C interface. Then
CPU Access to the CPU instructs the Server IO chip to set its U83 MUX (SMB_SEL<1:0>)
MCU Slave I2C lines to branch '01' this connects the MUX to the MCU I2C slave port. A
logic decode inside the Server IO will select and hold the 8 port I2C
Port switch (0xE4) in reset. This reset isolates the MCU from the global I2C bus
in the event of a failure on that global I2C bus.

The CPU can now use the ICH7 to communicate over the CLOCK_I2C
bus to the MCU slave port and control the MCU and its master I2C port.

MEM_I2C
11
PEER
14.3k

WILDCAT-S
CLOCK_I2C SP WILDCAT-S
V3_3

V3_3 01
SP
4.7k
U83
ICH7
CBTLV3253

V3_3
0 Ohm
U18 SMBus TRUNC_I2C
Master
14.3k
SMLink 0/1
ADR 0x88 Slave PS1_I2C
10
14.3k

V3_3

V3_3
1 of 4
Mux
14.3k

ISO_PS2_I2C PCA9543
S[1:0] I2C 0
00
Switch 1
SMB_SEL[1:0] ADR 0xE6
SIO
LPC2146 PCA9673
U141 VCC_CMD
Management I2C
Midplane

Controller PCA9548 Expander


ICH_SLAVE_I2C
U35 0 U73 14.3k
Master Slave ADR 0x4E
1
ADR 0x92
IO3_I2C MP_PS2_I2C I2C
6 PS2_I2C CBTLV
I2C
IO2_I2C Switch 3245
5 U69
IO1_I2C
4
dsPIC33F
IO0_I2C ADR 0xE4 CMD
3
U23
ADR 0x24

ISSP Header

Figure 2-79. SAN MCU I2C Slave Port Access

EMC CONFIDENTIAL

138
Peer Wildcat-S In a SAN or NAS system when the Peer Wildcat-S CPU Module wants to
access its peers’ MCU I2C slave bus it must first arbitrate for ownership of
CPU Access to the global I2C bus that the MCU I2C slave bus is connected to.

2
MCU Slave I2C
Which global I2C bus the MCU is connected to is dependent on which
Bus slot the Wildcat-S is located in. See Figure 2-72, Wildcat-S Chassis Midplane
I2C Interfaces (SAN).

• If the Wildcat-S is inserted in slot A - its peer Wildcat-S CPU would


arbitrate for use of the PS_B_I2C midplane bus.
• If the Wildcat-S is inserted in slot B - its peer Wildcat-S CPU would arbitrate
for use of the PS_A_I2C midplane bus.

In order for the Peer CPU Module to access the MCU I2C slave port via
the 8 port I2C switch U69 it must select branch 1 to connect it to the
CLOCK_I2C bus in which the MCU resides. The peer CPU can now
access the MCU slave port and control the MCU and its master I2C port.

NAS In a NAS system when one of the Earthquake Management FRU’s wants
to access the MCU Slave I2C bus it must first arbitrate for ownership of
Earthquake the global I2C bus that the MCU I2C slave bus is connected to.
Management
In order for the EarthQuake to access the MCU I2C slave port via the 8
FRU Access to port I2C switch U69 it must select branch 1 to connect it to the
MCU Slave I2C CLOCK_I2C bus in which the MCU resides. The EarthQuake
management FRU can now access the MCU slave port and control the
Bus MCU and its master I2C port.

EarthQuake
MEM_I2C NAS
11
Management
FRU-A
14.3k

WILDCAT-S
CLOCK_I2C SP
V3_3

V3_3 01

4.7k I2C Coldfire


U83 MCU
ICH7
CBTLV3253

0 Ohm V3_3
U18 SMBus TRUNC_I2C
Master
14.3k
SMLink 0/1
ADR 0x88 Slave PS1_I2C
10
PEER
WILDCAT -S
14.3k

V3_3

SP
1 of 4 V3_3
Mux
14.3k
PCA9543
S[1:0] ISO_PS2_I2C 0
00 I2C
Switch 1
SMB_SEL[1:0] ADR 0xE6
SIO
LPC2146 PCA9673
Management U141 I2C VCC_CMD
Midplane

Controller PCA9548 Expander


ICH_SLAVE_I2C
U35 0 U73 14.3k
Master Slave ADR 0x4E
1
ADR 0x92
IO3_I2C MP_PS2_I2C I2C
6 PS2_I2C CBTLV
I2C
IO2_I2C Switch 3245
5 U69
IO1_I2C EarthQuake
4
dsPIC33F NAS
IO0_I2C
3
ADR 0xE4 CMD Management
U23 FRU-B
ADR 0x24
I2C Coldfire
ISSP Header MCU

Figure 2-80. NAS MCU I2C Slave Port Access

EMC CONFIDENTIAL

Chassis Management (I2C) 139


MCU Master The MCU which includes an master I2C port may be controlled by the
Wildcat-S CPU, Peer Wildcat-S CPU or if in a NAS system by the Earth
I2C Port Quake Management FRU. This is done through the MCU slave I2C port.

The LPC2146 Management Controller (MC) U35 has one I2C master port.
The master port connects to a number of I2C expanders which are briefly
described below. The I2C Master port connections are illustrated in
Figure 2-82.

• U56 - I2C Expander receives the following signals:


- reset from peer
- persist reset which tells the board to reset but preserve cache
- NMI signal from management module
- CMD_Warning indicating the CMD has detected an out of spec voltage.
- Sleep signal indicating the board is in sleep mode.
- The only output is an Force_SMI signal out to the ICH.
• U59- I2C Expander sends out the power enables to:
- IO Module (3-0) slots
- IO Annex slot.
• U61- I2C Expander sends out the fault led signals to:
- IO Module (3-0) slots
- IO Annex slot.
• U60- I2C Expander sends out the PCI reset signals to:
- IO Module (3-0) slots
- IO Annex slot.
• U57- I2C Expander receives the power good signals from:
- IO Module (3-0) slots
- IO Annex slot.

• U58- I2C Expander receives the IO Module Insert signals from:
- IO Module (3-0) slots
- IO Annex slot.

Figure 2-81. MCU I2C Master Port Address Map

EMC CONFIDENTIAL

140
MCU I2C Master The MCU is
Interface

I2C Addr:
0x34 1 IO0_FLT_N
IO1_FLT_N
IO2_FLT_N
2
PCA9557
IO3_FLT_N
5 IO_ANNEX_FLT_N
IO_ICH_PCI_RST_N

MCU_SDA I2C Addr:


MCU_SCL 0x30 1 IO0_PWREN
IO1_PWREN
PCA9557 IO2_PWREN
IO3_PWREN
5 IO_ANNEX_PWREN
EXP_RST_N

CLOCK_SDA I2C Addr:


CLOCK_SCL Slave 0x32 IO0_RST_N
I2C Addr: 1
IO1_RST_N
0x3E
MCU_SDA_R IO2_RST-N
PCA9557 To/From IO
Master IO3_RST_N
MCU_SCL_R Module
5 IO_ANNEX_RST_N
IO_ICH_PCI_RST_N
Connectors
and IO Annex

MCU_SDA I2C Addr:


MCU_SCL 0xE2 1 IO0_INS_N
IO1_INS_N
IO2_INS_N
PCA9538
IO3_INS_N
INS_EXP_INT_N
5 IO_ANNEX_INS_N
EXP_RST_N
MCU 7 SLOT_A
U35

I2C Addr:
0xE4 1 IO0_PWRGD
IO1_PWRGD
PCA9538 IO2_PWRGD
PWRGD_EXP_INT_N IO3_PWRGD
5 IO_ANNEX_PWRGD

EXP_RST_N
Fault Registers

RST_EXP_INT_N I2C Addr: 0


0xE6 FORCE_SMI PS1_SDA
SLEEP PS1_SCL
PCA9539 PS2_SDA
PCA9538 PCA9539 PS2_SCL
BTN_IS_RST_N
PERSIST_RST_N 14
EXP_RST_N
7 RST_FROM_PEER_N 13

Figure 2-82. MCU I2C Master Interface

EMC CONFIDENTIAL

Chassis Management (I2C) 141


Blackford Blackford has a separate I2C bus for each FBDIMM channel. Each
FBDIMM has a Serial Presence Detect (SPD) prom that contains such
Controlled I2C information as memory size, memory chip width, CAS latency, etc.

Figure 2-83. Blackford MCH FBDIMM I2C Address Map

Blackford also has an I2C bus (referred to as the Hot Plug SMBus) that
can be used on Wildcat-S to monitor the insertion or removal of IO
Modules and the IO Annex.

Figure 2-84. Blackford MCH Hot-Plug I2C Address Map

Blackford
V3_3
MCH

FBDIMM #01 FBDIMM #00 4.7k 4.7k 4.7k 4.7k


EEPROM ADR 0xA2 EEPROM ADR 0xA0 U65
AMB ADR 0xB2 AMB ADR 0xB0
MCH_SPD0_SMB
SPD SMBus 0
FBDIMM #11 FBDIMM #10
EEPROM ADR 0xA2 EEPROM ADR 0xA0
AMB ADR 0xB2 AMB ADR 0xB0
MCH_SPD1_SMB
SPD SMBus 1
FBDIMM #21 FBDIMM #20
EEPROM ADR 0xA2 EEPROM ADR 0xA0
AMB ADR 0xB2 AMB ADR 0xB0
MCH_SPD2_SMB
SPD SMBus 2
FBDIMM #31 FBDIMM #30
EEPROM ADR 0xA2 EEPROM ADR 0xA0
AMB ADR 0xB2 AMB ADR 0xB0
MCH_SPD3_SMB SPD SMBus 3
GPIO SMBus

V3_3

4.7k
IOANNEX_INS*
PCA9554
SMBus HP_SMB_SMB
From IO (3-0) & IO IO[3:0]_INS*
Expander
Annex Connectors
ADR 0x40 U79

Figure 2-85. Blackford MCH I2C Diagram

EMC CONFIDENTIAL

142
I2C Reset, The below diagram shows the mapping of the Wildcat-S Blade I2C Reset,
Arbitration and Attention signals. These signals go out onto the midplane
Arbitration & and connect to the two Management FRU’s, PSA, PSB, IO Annex and the

2
Attention Nova Test Board Slot.
Signals
V+3.3

IO Conn
1k IO Conn
IO
IO Conn
IO_I2C_BUFRST Conn
IO_I2C_RST_N
V+3.3

EN_CBT_N
EN 1.5k
GPIO56 I2C_ARB_B1_BUF_IN CBT I2C_ARB_B1
GPIO30

GPIO23 I2C_ARB_B1_BUF_O_N V+3.3 V+3.3 V+3.3 V+3.3


1k
GND 14.3k 1k 14.3k 14.3k
ICH_ALERT_BUF_N PS2_I2C_ATTN_N
ICH GPIO20

PS1_I2C_ATN_BUF_N PS1_I2C_ATTN_N
GPIO13

PS1_I2C_BUFRST PS1_I2C_RST_N
GPIO22
I2C
I2C_EXP_RST_N EXP
0x34
V+3.3
I2C
14.3k I2C_AUX_RST_N EXP
0x36
CMD
I2C IO
I2C_MUX_RST_N Switch
0xE4

PS2_I2C_RST_N I2C Iso


GPIO10 Switch
0xE6

PS2_I2C_ATTN_N
GPIO11
SIO
V+3.3
SIO_ARB_A1
GPIO52
1.5k
I2C_ARB_A1
GPIO51

MP_I2C_ARB_B1
I2C_PWRON_RST_N
MP_PS1_ATTN_N
V+3.3 Isolation
M
Switch A MP_PS1_I2C_RST_N
1k I
PECI I2C_CBT_EN_N D
EN
P
L
MP_PS2_ATTN_N A
N
MCU V+3.3 MP_PS2_I2C_RST_N E
Isolation
V+3.3 Switch B
MP_I2C_ARB_A1
8.2k
SET EN_CBT
4.7k D Q EN
EN_CBT_CLK
EN_CBT_N SIO COM
CLR Q CBT
062-000-931
(062-000-892)

Figure 2-86. Wildcat-S I2C Reset, Arbitration & Attention Signals

Reset Signals The reset signals will follow a similar path as the attention signals above
and reset the MCU’s and some I2C buffers on various modules. It will
also reset the expander chips and I2C devices on the IO Modules.

Arbitration The arbitration signals were described earlier in this chapter for both a
SAN and NAS type system.
Signals

EMC CONFIDENTIAL

Chassis Management (I2C) 143


Attention When a power supply, management FRU or IO Annex Extender board
has an error condition it sends out an attention signal (interrupt) out to
Signals the midplane. Each Wildcat-S Blade (A or B) receives 2 attention signals
from the midplane I2C buses. When ever the wildcat blade receives this
signal it must arbitrate for the PSA_I2C or PSB_I2C bus in which the
attention signal is associated with and read the appropriate registers from
the devices on that bus which are described below. The software takes it
from here.

If the Wildcat-S Blade is inserted in Slot A then:

• Wildcat-S ICH receives PS_A_ATTN_N from the midplane. This signal


originates from:
- The (A) Power Supply MCU
- In an SAN - MCU on the (A) Solar Flare Management FRU
- In an NAS - MCU on the (A&B) Earth Quake Management FRU’S
- The MCU on the (A) Tornado IO Annex Extender Board
• Wildcat-S SIO receives PS_B_ATTN_N from the midplane. This signal
originates from:
- The (B) Power Supply MCU
- In an SAN - MCU on the (B) Solar Flare Management FRU
- In an NAS - MCU on the (B&A) Earth Quake Management FRU’S
- The MCU on the (B) Tornado IO Annex Extender Board

If the Wildcat-S Blade is inserted in Slot B then:

• Wildcat-S ICH receives PS_B_ATTN_N from the midplane. This signal


originates from:
- The (B) Power Supply MCU
- In an SAN - MCU on the (B) Solar Flare Management FRU
- In an NAS - MCU on the (A&B) Earth Quake Management FRU’S
- The MCU on the (B) Tornado IO Annex Extender Board
• Wildcat-S SIO receives PS_A_ATTN_N from the midplane. This signal
originates from:
- The (A) Power Supply MCU
- In an SAN - MCU on the (A) Solar Flare Management FRU
- In an NAS - MCU on the (B&A) Earth Quake Management FRU’S
- The MCU on the (A) Tornado IO Annex Extender Board

EMC CONFIDENTIAL

144
Chapter 3 MANAGEMENT FRUS

Chapter contents:




Introduction................................................88
Solar Flare...................................................89
Earthquake...............................................110 3

EMC CONFIDENTIAL

145
3.1 Introduction
There are two types of management FRU’s used in the Wildcat-S chassis
which are used in the Dreadnought systems.

• Solar Flare (SAN)

• Earthquake (NAS)

The Management FRU is used to monitor and control the chassis. RS-232
communications for the console and SPS is another important feature of
the Management FRU. It provides a path between the airdam and the
Wildcat-S CPU module. Management card redundancy is achieved by
ensuring that each chassis always contains two management FRU’s -
should one fail, the other can take over the management duties of the
entire enclosure.

EMC CONFIDENTIAL

146 MANAGEMENT FRUS


3.2 Solar Flare - Management FRU (SAN)

Introduction
The Solar Flare SAN management card provides various management
functionalities to the Wildcat-S chassis running in a SAN configuration.
There are two Management Modules per chassis.

3
Each Solar Flare interfaces with a Wildcat-S blade, power supplies and IO
modules via the system midplane (Fogbow). Communication between
the two management modules is supported by an Ethernet Crosslink
which provides service personnel the capability to plug into either
management module’s service jack and have access to both Wildcat-S
blades.

Solar Flare’s front-panel connectivity allows the outside world access to


the chassis in several ways: The design includes two 1000Mb Ethernet
connections for LAN management ports, dual micro DB-9 connectors to
provide COM port and SPS connectivity for the blades as well a USB port
for external connections.

Power OK LED
Fault LED
USB 2.0 port

Link LED

Activity LED

Service
LAN

Management Console Port


LAN
SPS Port
Activity LED
Link LED NMI Reset

Figure 3-87. Solar Flare

EMC CONFIDENTIAL

Solar Flare - Management FRU (SAN) 147


Logical Layout Figure 3-88 shows the block diagram for the Solar Flare SAN
management FRU. The major components on this board are:

• a 10/100/1000Mb integrated switch (the Broadcom BCM5397) to


connect the external management ports, blade Ethernet ports and a
link between the two management modules.

• a Philips micro controller to perform switch configuration and


diagnostics. Both the LAN ports are configured from the default of
auto, to fixed at 10/100/1000Mb, half / full duplex via the micro
controller.

• a I2C Isolation Switch

• CMD Power Sequencer

• USB2.0 Hub - used for

• Resume EEPROM - used to

SAN Management Module

LED’s
USB Micro DB-9 RJ-45
Power Fault

Service Service Debug Service Mgmt

Philips BCM5397 Gigabit


MUX LPC2131 Ethernet Switch
Microcontroller
USB 2.0
Hub
From Peer MGMT UART

Resume CMD
Prom Module
Peer MGMT.
Crosslink-

I2C
Blade A

Blade B

Isolation
Switch

Midplane Connector

Figure 3-88. Solar Flare Block Diagram

EMC CONFIDENTIAL

148 MANAGEMENT FRUS


Broadcom The Broadcom BCM5397 10/100/1000 Base-TX is a six-port Ethernet
Switch. The device has five integrated 10/100/1000 transceivers while the
BCM5397 sixth port is provided with an industry standard MII interface. Each of
Gigabit the five gigabit Ethernet ports are comprised of 4 differential pairs, or 8
wires each. These pairs carry a 125Mhz signal utilizing PAM-5 encoding
Ethernet Switch to transfer data. The switch has an SPI interface for diagnostics,
configuration, and switch management, which is used and controlled by
the Philips micro controller.

The Ethernet switch allows for the connection of two blades within a
Wildcat-S chassis to two Mag Jack Ethernet connectors. One of the
connectors is utilized for the management connection while the second is

3
for service use in a traditional SAN configuration.

In figure 3-51 below, the Ethernet switch and it’s surrounding logic is
shown.

BCM5397 Switch

TRD0_0+/-
Ethernet Crosslink TRD1_0+/-
TRD2_0+/- PORT 0 TRD0_3+/-
TRD3_0+/- TRD1_3+/- TDO-3 +/-
PORT 3 RJ45
TRD2_3+/-
TRD0_1+/- TRD3_3+/-
MidPlane SPA TRD1_1+/-
TRD2_1+/-
PORT 1 TRD0_4+/-
TRD3_1+/- TRD1_4+/- TDO-3 +/-
PORT 4 TRD2_4+/- RJ45
TRD0_2+/-
SPB TRD3_4+/-
TRD1_2+/-
TRD2_2+/-
PORT 2
TRD3_2+/-
JTAG

Philips
I2C

LPC2131
P0_4
SCK
SCL P0_5 MISO
SDA P0_6 MOSI SPI
P0_7
Bus SS
JTAG
Switch
POWER_GOOD
RESET CMD
TRST
TMS
TCK JTAG
TDI
TDO

Figure 3-89. Broadcom BCM5397 Switch

EMC CONFIDENTIAL

Solar Flare - Management FRU (SAN) 149


BCM5397

10/100/1000
TRD[3:0]_0+/- GMAC Register
PHY Space
10/100/1000
TRD[3:0]_1+/- GMAC
PHY
Packet Buffer
10/100/1000
TRD[3:0]_2+/- GMAC
PHY
10/100/1000 Address
TRD[3:0]_3+/- GMAC Management
PHY
10/100/1000
TRD[3:0]_4+/- GMAC LED
PHY LED
Interface
GMII / RvMII / RGMII GMAC
EEPROM
EEPROM
MII MAC Interface

Block Diagram BCM5397 Ethernet Switch


Figure 3-90. BCM5397 block diagram
Figures 3-52 above shows the internal workings of the Broadcom switch.

Switch
Configuration
The Broadcom 5397 Ethernet switch connects the three gigabit Ethernet
ports coming from the midplane and the two external ports. The switch is
able to partition the ports into separate networks, creating the concept of
the management and the service LAN. The service LAN is comprised of
the external service port, the service port coming from the midplane
(which is connected to the MAC on the peer's SP), and the Ethernet
crosslink between the two management boards. The second LAN
connects the external management port to the local SP's management
MAC.

By default the Micro controller will program the Switch for a SAN
configuration. Wildcat-S blades will use POST or FLARE to program the
micro controller via I2C for all other configurations. The micro controller
will use an SPI interface to configure the switch.

EMC CONFIDENTIAL

150 MANAGEMENT FRUS


Ethernet The Wildcat-S chassis has an Ethernet link between the two management
modules, introducing the concept of an Ethernet crosslink. This allows
Crosslink service personnel to plug into either management module's service jack,
and have access to both service processors. Access to the SP’s can be
disabled at the Broadcom chip if this functionality is not required.

Ethernet Crosslink

Management Service Service Management

3
GigE Switch GigE Switch

Management A Management B

Crosslink

GbE GbE GbE GbE


MAC 0 MAC 1 MAC 1 MAC 0

SP A SP B

Figure 3-91. Solar Flare Ethernet Crosslink

EMC CONFIDENTIAL

Solar Flare - Management FRU (SAN) 151


Philips The Solar Flare uses a microcontroller to provide added functionality and
diagnostic capability to the system. The LPC2131 microcontroller is based
Microcontroller on a 32/16-bit CPU with 32kB-256kB of embedded high speed Flash
LPC2131 memory and 8kb RAM. The controller operates from 0~60 MHz at 3.3v.
Serial interfaces for the chip include: 2 x UART, 2 x I2C, 2 x SPI.

The Philips microcontroller has two main functions. The first is to


configure the Broadcom Ethernet Switch. The other is to provide board
status via I2C. The micro will continuously poll the Broadcom switch for
status. If an error condition is detected, it will try to recover from the error
if possible and report the status the next time the device is polled via I2C.

LPC2131 Microcontroller

Figure 3-92. LPC2131 Microcontroller Block Diagram

EMC CONFIDENTIAL

152 MANAGEMENT FRUS


3

Figure 3-93. Philips LPC2131 Microcontroller

EMC CONFIDENTIAL

Solar Flare - Management FRU (SAN) 153


Serial Crosslink The way the serial is routed on Solar Flare allows for one Wildcat-S blade
to monitor the other blade’s serial output or console. In order to do this,
the SP that is up will need to write to the micro controller on Solar Flare
via I2C. The micro controller will then hit a MUX to pipe the serial stream
of the booting blade to the one that is already up. The Solar Flare
associated with the peer blade will then also need to switch the MUX to
complete the loop. With the peer blade booting, the blade that is up will
need to tell both Solar Flare boards to switch the serial in order to monitor
the peer's console.

Black Widow/Wildcat SPA Black Widow/Wildcat SPB

SP A SP B

RX TX TX RX

I2C I2C

MIDPLANE

Sel MUX MUX Sel


uController uController
0 1 1 0

RX TX TX RX

Micro DB9 Connector Micro DB9 Connector


Console A Console B

Akula/Solar Flare A Akula/Solar Flare B

Figure 3-94. Serial Crosslink Block Diagram

There is another serial port shared between the A and B Solar Flare micro
controllers. This link could be used so that the blade that is up only needs
to tell one Solar Flare to flip its MUX and the micro controller can inform
the micro controller on the peer Solar Flare to flip MUX the serial stream
over.

In order to prevent this crosslink from getting stuck on permanently,


there will be a timeout in the micro controller to switch the crosslink back
to its default state. Once the command to switch the MUX over has been
received, the micro controller will give the SP 10 seconds and then switch
the MUX back to the default. In order to keep the crosslink up, the blade
will need to write to the micro controller before that 10-second timeout
expires.

EMC CONFIDENTIAL

154 MANAGEMENT FRUS


Microcontroller There is a serial bus connection between the two Solar Flare management
Private Serial Bus modules in a Wildcat-S chassis. This allows for a simplified Serial
Crosslink implementation as mentioned previously. This also allows for
status sharing between Solar Flare management modules. In cases where
the I2C bus is hung or other faults where I2C communication is disabled,
this allows for status to be retrieved from the peer Solar Flare.

3
I2C Bus
The figure below shows the I2C bus topology on Solar Flare with the
devices' I2C addresses. There are two system I2C busses, each
management module connects to one of those busses depending on the
slot the management module resides in.

Resume
PROM

(0xAE) I2C
CMD
M Sequencer
Switch S0
I
SCL D (0xF2)
SDA S1
P
RST
(SlotA =0xEA)
L
A (SlotB =0xE8) 4.7K
(0x90)
N
E 3.3V
Philips MCU

Figure 3-95. Solar Flare I2C Topology

I2C Isolation The Philips PCA9546 allows the Master to disconnect the microcontroller
from the bus in the case that the microcontroller's I2C bus has become
stuck. The microcontroller also has the ability to disconnect itself from the
bus. If the microcontroller does not detect a valid slot ID code, it will
disconnect itself from the bus to prevent possible addressing conflicts
with the other management unit. This is necessary because one of the
switch address pins is slot dependent. If the slot ID is stuck one way or
the other it can create an addressing problem in the design.

Figure 3-96. Slot Dependent I2C Addresses

EMC CONFIDENTIAL

Solar Flare - Management FRU (SAN) 155


Power The Solar Flare board is supplied by two 12V voltage rails, A & B. The
voltage is always on and redundant from the power system. The Solar
Requirements Flare board uses diode OR'ing in order to create one 12V voltage rail for
the board. The OR'ed voltage is then fed through soft start circuitry as to
enable the board to be hot plugged. In order to prevent a possible board
failure from dragging down both 12V buses, active over-current circuitry
is used to disable the board from the power rail in the event of a short
circuit.

Power Sequencing
& Monitoring
The CMD module provides sequencing, margining and other power
management functions to the system. This device uses PMBus, based on
SMBus, to communicate status of the power subsystem. Using the CMD
provides an easy and consistent method to control the DC/DC converters
on the board. Other benefits of the CMD are: provides both power good
and fault signals to the system, margining capability to both 3% and 5%
without modifying resistors and DC regulation can be maintained at
better than 1%.

The CMD module uses a 6-bit CMD_BOARD_ID for determining which


voltages to set up and monitor. Solar Flare's CMD_BOARD_ID[6-1] is
"000010".

Broadcom recommends that both the 2.5 and 1.0V rails come up
simultaneously, but to err on the side of caution, the higher voltage comes
up after the 1.0V is up. Power is sequenced according to the following
figure.

Figure 3-97. Solar Flare Voltage Sequence

All other voltages do not affect the Broadcom chip and do not have a
sequencing requirement.

The Solar Flare board monitors the 5.0, 3.3, 2.5 and 1.0 voltages using the
Command Module. The module only asserts POWER_GOOD when all
voltages are within specification.

The Command Module is shown in Figure 3-60.

EMC CONFIDENTIAL

156 MANAGEMENT FRUS


CMD Power Sequencer

Figure 3-98. Command Power Sequencer

EMC CONFIDENTIAL

Solar Flare - Management FRU (SAN) 157


Signal Lists

BCM5397

Figure 3-99. Broadcom BCM5397 pin-out

EMC CONFIDENTIAL

158 MANAGEMENT FRUS


BCM5397 (cont.)

Figure 3-100. Broadcom BCM5397 pin-out (cont.)

EMC CONFIDENTIAL

Solar Flare - Management FRU (SAN) 159


BCM5397 (cont.)

Figure 3-101. Broadcom BCM5397 pin-out (cont.)

EMC CONFIDENTIAL

160 MANAGEMENT FRUS


LPC2131

Figure 3-102. Philips LPC2131 pin-out

EMC CONFIDENTIAL

Solar Flare - Management FRU (SAN) 161


LPC2131 _Pin
Description

Figure 3-103. Philips LPC2131 pin definitions

EMC CONFIDENTIAL

162 MANAGEMENT FRUS


LPC2131 _Pin
Description (cont.)

Figure 3-104. Philips LPC2131 pin definitions (cont.)

EMC CONFIDENTIAL

Solar Flare - Management FRU (SAN) 163


LPC2131 _Pin
Description (cont.)

Figure 3-105. Philips LPC2131 pin definitions (cont.)

EMC CONFIDENTIAL

164 MANAGEMENT FRUS


LPC2131 _Pin
Description (cont.)

Figure 3-106. Philips LPC2131 pin definitions (cont.)

EMC CONFIDENTIAL

Solar Flare - Management FRU (SAN) 165


CMD_Power
Sequencer Pin-out

Figure 3-107. CMD_Power Sequencer pin-out

EMC CONFIDENTIAL

166 MANAGEMENT FRUS


CMD Power
Sequencer pin
definitions

Figure 3-108. CMD Power Sequencer Pin Definitions

EMC CONFIDENTIAL

Solar Flare - Management FRU (SAN) 167


EMC CONFIDENTIAL

168 MANAGEMENT FRUS


3.3 Earthquake - Management FRU (NAS)

Introduction
The Earthquake NAS management card provides various management
functionalities to a Dreadnought enclosure running in a NAS
configuration. There are two Management Modules per enclosure.

3
Each Earthquake interfaces with a Wildcat Storage Processor (SP), power
supplies and IO modules via the system midplane (Fogbow).
Communication between the two management modules is supported by
two serial paths between the two modules.

Earthquake uses a Broadcom 10/100/1000 BASE-T Ethernet switch as well


as the Freescale Coldfire microprocessor to form the core of the board.
The Earthquake board provides external connections to the Storage
Processors' (SP) serial ports, Ethernet, and a single USB port.

Earthquake's front-panel connectivity allows the outside world access to


the enclosure in several ways: The design includes three 1Gb Ethernet
connections for management and uplink ports, a single micro DB-9
connector to provide COM port access, a NMI button, and a USB port for
external connections. The board also has a 7 segment display for the
enclosure ID as well as the usual power and fault LEDs. The Earthquake
utilizes a Freescale Coldfire microcontroller to perform switch
configuration and out of band enclosure management.

Figure 3-109. Earthquake

EMC CONFIDENTIAL

Earthquake - Management FRU (NAS) 169


Logical Layout

Riser Card
Pwr/ NMI
Micro RJ45 RJ45 RJ45 USB
Flt 7 Seg Button
Debug DB9 Service Mgmt Uplink Service
LED’s

EMU Conn

4 3 0
USB 2.0
MII
HUB
Motorola Coldfire SPI BCM5397
Service/
(MCF5282) 1Gb Enet Switch 2
Debug JTAG Chain
Muxing 1
I2C
SDA/SCL

Bld I2C Bld I2C


JTAG Ins Ins JTAG
Size Ctl Size Ctl

SPB

244 8-bit I/O 1:4 I2C 1:4 I2C 244


8-bit CBT 8-bit CBT Expander Switch Switch
Buf EN EN
Buf SPA
Uart Xlink B,
Debug,
Cnsl RX Uart Xlink A,
Cnsl TX
Resume
PS1 CMD PS2
Debug PROM I2C I2C

Arb
Arb
Debug

Midplane Connector

Figure 3-110. Earthquake Block Diagram

EMC CONFIDENTIAL

170
Figure 3-51 shows the block diagram for the Earthquake NAS
management FRU. The major components on this board are the Freescale
MCF5282 Coldfire Micro controller, the Broadcom BCM5397 Gigabit
Ethernet Switch, the CMD Power Sequencer and the USB2.0 Hub. Also,
due to the lack of PCB real-estate on the front panel, a riser card (Tremor)
is used to bring some of the features to the front panel. The DB9, LED's
and Seven Segment Display reside on the Tremor which mates to
Earthquake through a connector.

Figure 3-111. Tremor Riser Card

Broadcom
BCM5397
Gigabit
The Broadcom BCM 5397 10/100/1000 BASE-TX is a six-port Ethernet
Ethernet Switch Switch. The device has five integrated 10/100/1000 transceivers while the
sixth port is provided with an industry standard MII interface. Each of
the five gigabit Ethernet ports are comprised of 4 differential pairs, or 8
wires each. These pairs carry a 125MHz signal utilizing PAM-5 encoding
to transfer the data. The switch has an SPI interface for diagnostics,
configuration, and switch management.

EMC CONFIDENTIAL

Earthquake - Management FRU (NAS) 171


The Ethernet switch allows for the connection of two blades within a
Dreadnought enclosure to three MagJack Ethernet connectors. One of the
connectors is utilized for the management connection, another is used for
service and the third is used for an Uplink. Ethernet connections utilize
magnetics to drive signals to the two blades.

The figure below is a high-level block diagram of the Ethernet switch and
its surrounding logic.

M
A
SPA G Port 1
N
Midplane E
Port 0
RJ45
T
SPB I Port 2 Uplink
C
S

I2C
Port 3
RJ45
Mgmt
JTAG
MII
Switch BCM5397
1Gb Enet Switch Port 4
RJ45
SPI
I2C Service
244 Coldfire
Buffer MCF5282 ~RST

TDI
TDO TDI
JTAG

JTAG

TDO

Figure 3-112. Earthquake Ethernet Block Layout

EMC CONFIDENTIAL

172
BCM5397

10/100/1000
TRD[3:0]_0+/- GMAC Register
PHY Space
10/100/1000
TRD[3:0]_1+/- GMAC
PHY
Packet Buffer
10/100/1000
TRD[3:0]_2+/- GMAC
PHY
10/100/1000 Address
TRD[3:0]_3+/- GMAC Management
PHY

3
10/100/1000
TRD[3:0]_4+/- GMAC LED
PHY LED
Interface
GMII / RvMII / RGMII GMAC
EEPROM
EEPROM
MII MAC Interface

Block Diagram BCM5397 Ethernet Switch


Figure 3-113. BCM5397 block diagram

The block diagram of the BCM5397 shows the internal logic of the switch.

Switch
Configuration
The Broadcom 5397 Ethernet switch connects the two gigabit Ethernet
ports coming from the midplane and the three external ports. The switch
is able to partition the ports into separate networks, creating the concept
of the management and the service LAN. The service LAN is comprised
of the external service port and the service port coming from the
midplane (which is connected to the MAC on the peer's SP). The second
LAN connects the external management port to the local SP's
management MAC.

By default the Micro controller will program the Switch for a NAS
configuration. POST or FLARE will need to program the micro controller
via I2C for all other configurations. The micro controller will use an SPI
interface to configure the switch. The SPI interface is a simple four-wire
bus, which allows access to the internal registers within the Broadcom
switch. Upon power up the Coldfire processor will be required to do
some basic configuration and run diagnostics on the switch over its SPI
interface. The Forward Enable feature should be active within the
Broadcom switch so traffic can move thru the device without dependency
on the Coldfire for configuration if there is an issue with a device on
board.

EMC CONFIDENTIAL

Earthquake - Management FRU (NAS) 173


Motorola The Motorola MCF5282 Coldfire processor was chosen for the
management micro controller on the Earthquake board. The device is a
Coldfire 32-bit processor based on the 68K architecture capable of running up to
MCF5282 66MHz at the core. The following lists some of the features:

- 512KB Embedded Flash Memory

- 64KB SRAM

- 10/100 Ethernet MAC

- 3 UARTs

- SPI Interface

- 8-channel Analog-to-Digital Converter

- I2C Controller

- 256-pin 1mm pitch BGA (17mm x 17mm)

The following figure depicts the block diagram of the Coldfire processor.

Figure 3-114. Motorola Coldfire Block Diagram

EMC CONFIDENTIAL

174
Serial UART Earthquake is connected to both of the local SP's UART's, but has only
one external micro db9 port located on the riser (Tremor) card. Thus,
MUX several MUX's are used to select which serial port is connected to the
external connector. In addition, the Coldfire has the ability to attach the
console serial port to itself, and direct the debug port to the external
connector.

Black Widow/Wildcat SPA Black Widow/Wildcat SPB

3
UARTS UARTS
DBG DBG
CNSL CNSL
RX TX RX TX TX RX TX RX

MIDPLANE

1 1
2
Coldfire Coldfire
MUX

MUX
0

0
RX RX
3 3
TX
2 2 TX

2 2
1

1
MUX

MUX
MUX

MUX

MUX

MUX
0

1
1

RX TX TX RX

DB-9 Tremor Coldfire


2 Coldfire Tremor DB-9
Riser DEBUG DEBUG Riser

Nimitz/Earth Quake Nimitz/Earth Quake

Figure 3-115. Serial Mux Control

The MUX Select lines default to port 0 with pull downs. This connects the
SP's console UART to the external micro db9 connector. The Coldfire
debug UART is not externally accessible but it is located on the Tremor
riser card, that will give access to this for lab bring up and debug. The
Coldfire has the ability to change the select lines on the MUX to connect
the debug UART to the front panel.

There are two serial links between the Earthquake boards in a chassis.
The second link can also be used to access the Coldfire debug UART.
When a cable is inserted into a debug header on the Tremor riser card, it
automatically flip's the MUX and connects the second UART to this
header.

EMC CONFIDENTIAL

Earthquake - Management FRU (NAS) 175


I2C Bus The figure below shows the I2C bus topology on Earthquake with the
devices' I2C addresses. There are two system I2C busses, each
management module connects to one of those busses depending on the
slot the management module resides in

V3_3 V3_3
4.7K I2C_ARB_A1_O 1.5K C_I2C_ARB_A1 CBTLV PS_1_I2C_ARB
3384
C_I2C _ARB_A1 V3_3

C_PS_1_I2C_ATN_N 4.7K C_PS_1_I2C _ATN_N PS_1_I2C_ ATTN_N


V3_3
V3_3 PCA 9557
1K
4.7K 0 V3_3 EN
1 1K
2
4.7K V3_3
I2 C Exp GND 1K
3 GND
4 4.7K
5 GND
V3_3 6
4.7K
7 V3_3
SLOT _ID_0 ADR 0 x 32 , 0x 30
1K

MC _CBT_EN 1

Midpla ne
4.7K GND
MC _CBT_EN 2 GND
4.7K
V3_3 V3_3 EN
GND
4.7K I2C_ARB_B1_O 1.5K C _I2C_ARB_B1 PS_2_I2C_ARB
Coldfire
C_I2C_ARB_B1 V3_3
C_PS_2_I2C _ATN_N 4.7K C_PS_2_I2C_ATN_N PS_2_I2C_ ATTN_N
CBTLV
3384
DB G Header

V+3_3
ZERO

ADR 0x 90 ZERO
PCA 95 xx ZERO 14.3K

SW_I2C I2 C PS_1_I2C
Switch 0
2
1 V+3_3
ADR 0 xEC , 0xE 8

Resume
PCA 95 xx 14.3K
ADR 0 xAE

I2 C PS_2_I2C
Switch 0
1
ADR 0 xEE , 0 xEA

CMD
ZERO

ADR 0xF 2

ISSP Header

Figure 3-116. Earthquake I2C Topology

EMC CONFIDENTIAL

176
I2C Isolation The Coldfire allows the Master to disconnect the micro controller from
the bus in the case that the micro controller’s I2C bus has become stuck.
The micro controller also has the ability to disconnect itself from the bus.
If the micro controller does not detect a valid slot ID code, it will remain
disconnected from the bus to prevent possible addressing conflicts with
the other management unit. This is necessary because one of the switch
address pins is slot dependent. If the slot ID is stuck one way or the other
it can create an addressing problem in the design.

3
Figure 3-117. Slot Dependent I2C Addresses

Enclosure
Identification The Earthquake board will display the enclosure ID from the Coldfire
CPU with a value of 1 to 8 respectively. Software is responsible for
creating and maintaining each value. A seven segment display is
mounted to a riser card (Tremor) plugged into Earthquake to display the
enclosure information.

EMC CONFIDENTIAL

Earthquake - Management FRU (NAS) 177


Power The Earthquake board is supplied by two 12V voltage rails, standby A &
B. The voltage is always on and redundant from the power system. The
Requirements Earthquake board uses diode OR'ing in order to create one 12V voltage
rail for the board. The two 12 voltage rails are fed through soft start
circuitry allowing the board to be hot plugged. In order to prevent a
possible board failure from dragging down both 12V buses, active
over-current circuitry is used to disable the board from the power rail in
the event of a short circuit.

POWER _ A _SENSE
Coldfire
POWER _ B _SENSE MCF5282
Power

4.7K 7.15K 7.15K 4.7K

7.15K 7.15K
To Voltage
Plane / Islands
3.3V
V + 12 _ A Soft V + 12 _ SS _ A V + 12 _ ORED DC-DC V + 3 .3
Start Converter

2.5V
V + 12 _ B Soft V + 12 _ SS _ B DC-DC V +2.5
Start Converter

1.0V
V+ 1
DC-DC
3.3V
3.0Vdc Voltage Converter
DC-DC
M id pl a n e

Reference Divider
Converter
V + 3 .3 _CMD 5.0V V +5
DC-DC
Converter
4.7K Analog Monitor X Warning

Board ID Reference_ Input


6

Control
1K Monitoring
Device
(CMD)
Sequencer / Monitor

Sense Loss of 1 Supply


4 .7K Analog Monitor Y
Zero

4.7K Board Inserted


V + 12 _Soft Start Sense

1K On Board DCDC Enables

On Board DCDC Voltage Senses

Figure 3-118. Earthquake Power Distribution

Power Sequencing
& Monitoring
The CMD module provides sequencing, margining and other power
management functions to the system. This device uses PMBus, based on
SMBus, to communicate status of the power subsystem. Using the CMD
provides an easy and consistent method to control the DC/DC converters
on the board. Other benefits of the CMD are: provides both power good
and fault signals to the system, margining capability to both 3% and 5%
without modifying resistors and DC regulation can be maintained at
better than 1%.

The CMD module uses a 6-bit CMD_BOARD_ID for determining which


voltages to set up and monitor. Earthquake's CMD_BOARD_ID[6:1] is
"000010".
EMC CONFIDENTIAL

178
Broadcom recommends that both the 2.5 and 1.0V rails come up
simultaneously, but to err on the side of caution, the higher voltage comes
up after the 1.0 is up. Power is sequenced according to the following
figure.

Figure 3-119. Earthquake Voltage Sequence

3
All other voltages do not affect the Broadcom chip and do not have a
sequencing requirement.

The Earthquake board monitors the 5.0, 3.3, 2.5 and 1.0 voltages using the
Command Module. The module only asserts POWER_GOOD when all
voltages are within specification. This signal is tied to the reset scheme
into the on board micro controller (Coldfire).

EMC CONFIDENTIAL

Earthquake - Management FRU (NAS) 179


Reset & The only time Earthquake will see a reset is a power-on reset. The CMD
will hold the Broadcom chip in reset until power good is asserted. Once
Initialization power good is released, the Coldfire micro controller will load the default
NAS settings into the Broadcom chip after some delay to allow the
Broadcom chip to stabilize after reset. Once the Broadcom chip is loaded
with the default configuration, it will be up to software to tell the micro
controller to allow the switch to begin forwarding packets.

V+3_3

V3_3 4.7K

4.7K
C_PS_1_I2C_RST_N CBTLV PS_1_I2C_RST_N
3384

V3_3
PCA9557
1K
V3_3 USB 0 V3_3 EN
4.7K HUB 1 1K
2
4.7K V3_3
100NF
I2C Exp GND 1K GND
GND 3
4 4.7K
V3_3 5 GND
1K
6
EXP_RST
7 V3_3
ADR 0x32, 0x30 1K

MC_USB_RST_N MC_CBT_EN1 V+3_3

Midplane
4.7K
GND
MC_CBT_EN2 GND

4.7K 4.7K
V3_3
GND
EN
4.7K C_PS_2_I2C_RST_N PS_2_I2C_RST_N
Coldfire

MC_NMI_ASSERT CBTLV
V3_3 V3_3
3384
4.7K MC_BCM_RST_N MC_NMI_N 4.7K V+3_3
I2C_SW_RST1

I2C_SW_RST2
PCA95xx 14.3K
I2C PS_1_I2C
V3_3
ZERO Switch 0
1K
V3_3 1 V+3_3
ADR 0xEC, 0xE8
MC_RST_O_N 4.7K MAX6816
V3_3
NMI_SWITCH
Debounce
MC_RST_I_N 1K
PCA95xx
EXT_WD

ZERO

14.3K
V3_3 I2C ZERO PS_2_I2C
V3_3
WatchDog 4.7K Switch 0
1K
Timer 1
V3_3
CBT
4.7K NMI_N
V3_3 ADR 0xEE, 0xEA 4.7K CBT_NMI_EN_N
GND EN 1NF
4.7K
Emulator PWR_LED_XTOR_C GND

V3_3 1K
GND CMD
GND
BCM_RST_N 1K GigE
CMD_POWER_GOOD
ZERO

ADR 0xF2
Switch
4.7K
GND ISSP Header
MC_RESET_N

Figure 3-120. Earthquake Reset Diagram

The Motorola Coldfire has an internal watchdog timer to prevent


software runaway and a hang condition. In order not to rely on a single
chip for the processor and watchdog, an external watchdog is
incorporated in the design.

A reset of the Coldfire will result in the entire board resetting. The USB
device is also reset controlled from the Coldfire and is configured thru a
dedicated EEPROM.

EMC CONFIDENTIAL

180
Power and Fault The Earthquake board contains a fault LED and a power LED located on
the riser (Tremor) card. The fault LED is yellow and will be controlled by
LEDs both a remote board signal and the Coldfire micro controller. The micro
controller will assert the fault LED under certain unrecoverable situations
such as switch failure.

The power LED is green and is connected to the 3.3V voltage source. On
power-up, the CMD device turns on the LED when all voltages are up
and within specification.

The management module also has the ability to assert the LED's that
reside on the Midplane and the Local/Peer Blades. The diagram below

3
illustrates the connectivity thru the midplane.

SP A MIDPLANE IO ANNEX B
ICH_SW_FLT
I 1k 1k
C
H ICH_SYS _FLT_N
7
V+3.3
MGMT B
10 k
IO_ANNEX_FLT MC_FAULT_LED

MCU ZERO
Fault 1k 1k
10K
EXP

Trem or
BLADE_FLT R iser
SIO V+3.3

1k 1k 56
CMD_POWER _GOOD
CMD
1k
4.7k

V+3.3 MGMT A V+3.3


10 k 10 k
MC_SYS _FLT _LED C
MC_SYS_FLT_LED
O
L
D
F
MC_PR_MNG_FLT_LED MC_PR_MNG_FLT _LED I
R
2.2 k 2.2k E

MC_BLD_FLT _LED _A MC_BLD_FLT _LED _A

C 2.2 k 2 .2k
O
L
MC_BLD_FLT _LED _B MC_BLD_FLT _LED _B
D
F
I 2.2 k 2 .2k
R
E MC_ANX_FLT_LED _A MC_ANX_FLT_LED _A

2.2 k 2 .2k

MC_ANX_FLT_LED _B MC_ANX_FLT_LED _B

2.2 k 2. 2k

V+3.3 SP B
10 k
MC_FAULT_LED ICH_SW _FLT
ZERO I
1k 1k C
ICH_SYS _FLT_N
H
7
Trem or
R iser
V+3.3
IO_ANNEX_FLT

56 MCU
CMD_POWER _GOOD Fault
CMD
10 k EXP
1k
4.7k

BLADE_FLT
SIO

1k 1k 1k 1k

IO ANNEX A

Figure 3-121. Earthquake LED Control

EMC CONFIDENTIAL

Earthquake - Management FRU (NAS) 181


JTAG The following figure shows the JTAG connections on the Earthquake
board. A serial chain is present which is accessed thru the midplane. The
Coldfire has a connector on the Emulation/JTAG signals where JTAG
functionality is enabled with an active High input on JTAG_EN.

JTAG_TDO MCU_TDO

244
JTAG_ENABLE_N EN

M
V3_3
I
D 4.7K JTAG_EN
DePOP
V3_3
P
L 1K ZERO
4.7K
A
GND
N
E EN
JTAG_TRST_N BCM_TRST_N
JTAG_TCK BCM_TCK
244 BCM
JTAG_TMS BCM_TMS
JTAG_TDI BCM_TDI TDO
TDI
1K BCM_TDO
1K
GND
GND EN JTAG_EN
MCU_TRST_N
MCU_TCK
244 ColdFire
MCU_TMS

100 TDI TDO


V3_3
4.7K
GND DePOP

ZERO
V3_3
4.7K
Emul

Figure 3-122. Earthquake JTAG Topology

EMC CONFIDENTIAL

182
Signal Lists

Motorola Coldfire
MCF5282

Figure 3-123. Coldfire BGA pinout

EMC CONFIDENTIAL

Earthquake - Management FRU (NAS) 183


Motorola Coldfire
MCF5282 (cont.)

Figure 3-124. Coldfire BGA pinout (cont.)

EMC CONFIDENTIAL

184
Motorola Coldfire
MCF5282 (cont.)

Figure 3-125. Coldfire BGA pinout (cont.)

EMC CONFIDENTIAL

Earthquake - Management FRU (NAS) 185


Motorola Coldfire
MCF5282 (cont.)

Figure 3-126. Coldfire BGA pinout (cont.)

EMC CONFIDENTIAL

186
Motorola Coldfire
MCF5282 (cont.)

Figure 3-127. Coldfire BGA pinout (cont.)

EMC CONFIDENTIAL

Earthquake - Management FRU (NAS) 187


CMD Power .
Sequencer Pinout .

Figure 3-128. Earthquake CMD_Power_Sequencer

EMC CONFIDENTIAL

188
Tremor Riser
Card_RS-232
Buffer

Figure 3-129. Tremor Riser Card RS-232

EMC CONFIDENTIAL

Earthquake - Management FRU (NAS) 189


EMC CONFIDENTIAL

190
Chapter 4 CHASSIS POWER

EMC CONFIDENTIAL

191
4.1 Wildcat-S Chassis Power
The Wildcat-S Chassis receives power from two power supplies (PS-A &
PS-B). Both power supplies plug into the chassis midplane. Figure 4-130
shows the rear view of the Wildcat-S Chassis with the two power supplies
inserted.

PS-A
PS-B

Figure 4-130. Wildcat-S Chassis Rear View

EMC CONFIDENTIAL

192 CHASSIS POWER


12 volt Standby Both power supplies PS-A & PS-B supply 12 volt standby power to (SPA
& SPB), (Management FRU A & B) and the Nova test card. The 12 volt
Power standby power from PS-A is also routed to the A IO Annex slot. Standby
12 volt power from PS-B is also routed to the B IO Annex slot. See
Figure 4-131 below for more details.

Power Supply - A

J13 J1 Management FRU - A

V+12_SBY_A
Blower Fan

ITRAC
IO Module 3

ITRAC
IO Module 2
Wildcat-S
J5 Storage Processor
A

ITRAC
IO Module 1
Blower Fan

ITRAC
4
IO Module 0
M
I
D J21 IO A

Nova P
Test J8 IO ANNEX
Card L
A J7 IO B
Blower Fan
N
E
ITRAC

IO Module 3
ITRAC

IO Module 2
Wildcat-S
Blower Fan J6 Storage Processor
B
ITRAC

IO Module 1
V+12_SBY_B

ITRAC

IO Module 0

J14 J2 Management FRU - B

Power Supply - B

Figure 4-131. Wildcat-S Chassis 12 Volt Standby power


EMC CONFIDENTIAL

Wildcat-S Chassis Power 193


Wildcat-S The Wildcat-S Chassis receives AC power from the cabinet through the
power cords to each power supply. This causes the two power supplies to
Standby Power start sending out 12 volt standby power. Each Wildcat-S blade receives
standby power from both power supplies (PS-A & PS-B) as described in
Figure 4-132 below. The (CMD) Command & Monitoring Device is
powered by the output of U24 which is VCC_CMD with a value of
3.3volts. The resume eeprom also uses VCC_CMD as well as the (PS-A &
PS-B) I2C SCL/SDA lines pull-up resistors.

V+12_SBYA Wildcat-S Blade (A or B) Standby Power


U23
(CMD)
F3
1A SEQUENCER /
V+12_SBYA_FUSE V12_CMD_IN V+12_CMD_MEAS
MONITOR
VCC_CMD_MEAS

M CR52_16S
I
D
V+12_CMD U24
VCC_CMD
P VCC
LT1763 3.3volts
L
A
N CR52_17S
E

U100
V+12_SBYB_FUSE RESUME
1A
F2

VREF_3P0
To PS1 & PS2 I2C U25
V+12_SBYB SCL/SDA LINES VREF_3P0_RET

Figure 4-132. Wildcat-S Blade Standby Power Usage

Power up the The 2 management FRU’s receives standby power from the two power
supplies. This enables the two FRU’s to have communication with the
Wildcat-S outside world through LAN ethernet ports. In order to turn on the 12v
Chassis main power to the two Wildcat-S blades and the rest of the system the
user needs to send a power on command through the LAN port which is
addressed in such a way to be sent over the SPI interface to the (MC)
Micro controller in the management FRU. That MC sends out an I2C
command to the two power supplies (PS-A & PS-B) to enable their 12v
main power to the Wildcat-S Chassis.

EMC CONFIDENTIAL

194 CHASSIS POWER


12 volt Main The two power supplies supply 12 volt main power to the Wildcat-S
Chassis. Power supply A supplies power to SP A and IO Annex slot A. It
Power is also connected to the B power supply. If PSB malfunctions it can rout
some of PSA’s power to SP B and IO Annex B to keep the chassis powered
up. Power supply B supplies power to SP B and IO Annex slot B. It also is
connected to the A power supply in case of an malfunction, it can route
power to SP A and IO Annex A.

Power Supply - A

J13 J1 Management FRU - A

Blower Fan V+12_BLADEA_IOA

ITRAC
IO Module 3

ITRAC
IO Module 2
Wildcat-S
J5 Storage Processor
A

ITRAC
IO Module 1
Blower Fan

4
ITRAC
IO Module 0
M
I
D J21 IO A

Nova P
Test J8 IO ANNEX
Card L
A J7 IO B
Blower Fan
N
E
ITRAC

IO Module 3
ITRAC

IO Module 2
Wildcat-S
Blower Fan J6 Storage Processor
V+12_BLADEB_IOB

B
ITRAC

IO Module 1
ITRAC

IO Module 0

J14 J2 Management FRU - B

Power Supply - B

Figure 4-133. Wildcat-S Chassis 12 Volt Main power


EMC CONFIDENTIAL

Wildcat-S Chassis Power 195


Wildcat-S The Wildcat-S Blade receives V12 from one of the two power supplies
(PS-A or PS-B) used in the Wildcat-S Chassis. This V12 volt power will be
Power converted on the Wildcat-S motherboard to various voltages which are
required by its components. The CMD monitors 12v, 5.0V, 3.3V, 2.5V, 1.8V,
1.5V, 1.2V, 1.05V, 1.0V, and 0.9V. Finally 2 VRM’s take V12 volts as input
and create the necessary supply voltage required for the 2 CPU’s. There
are many fuses located on the Wildcat-S board some of which are shown
in Figure 4-134 below. The V12 also has 4 fused branches which flow to
the 4 IO Module Connectors. The IO Modules will convert this V12 into
the various voltages required by each IO Modules components.

The output of the DC-DC, VRMS and other voltages are fed back to the
(CMD) Control & Monitoring Device and are measured to ensure that
they stay within specifications.

V+12_POL_MEAS

VCC15_MEAS
V12 F1 V12_G CM1
V1_5
M 1.5V
F5 VCC09_MEAS
I V12_B U138
D 0.9V V0_9
P
L V12_B CM2 VCC18_MEAS
A 1.8V
V1_8
N VCC1P05_MEAS
E U29
1.05V
V1_05
F4
V12_R VCC33_MEAS
U1
3.3V V3_3
U23
VCC12_MEAS (CMD)
V12_R CM3
VTT_CPU SEQUENCER /
1.2V
MONITOR
VCC10_MEAS
V12_R U9
1.0V V1

V12 VCC50_MEAS
V12 U32
J50_1S V12_IO0 F6 5V USB V5_USB
IO Card 0

J51_1S V12_IO1 F7 VCPU_A_MEAS


V12 J3
IO Card 1 VRM-A VCC_FSB0
For CPU 0
F8
J52_1S V12_IO2
IO Card 2
VCPU_B_MEAS
V12 J4
J53_1S V12_IO3 F9 VRM-B VCC_FSB1
IO Card 3 For CPU 1

Figure 4-134. Wildcat-S 12 volt Main power to DC-DC & IO


Connectors
EMC CONFIDENTIAL

196 CHASSIS POWER


Wildcat-S DC The onboard DC-DC converters have enable signals which allow the
(CMD) Control & Monitoring Device to control the power-up sequence.
Power Enables Once all the onboard power is stable the CMD sends Wildcat_Pwrgd to
the MCU to indicate that it can now turn on the power to the IO Modules
and IO Annex.

U32
5V USB

U1 EN_V3_3 FET EN33


3.3V CIRCUIT

CM1 EN_V1_5 FET EN15


1.5V CIRCUIT

U138 EN_V1_8
0.9V EN18
FET
CIRCUIT
CM2 EN_V1_8
1.8V

U9 EN_V1_0 FET EN10


1.0V CIRCUIT
U23

4
(CMD)
U29 EN_V1_05 FET EN1P05 SEQUENCER /
1.05V CIRCUIT
MONITOR
IO_ANNEX_PWREN CM3 EN_V1_2 FET EN12
J22 CIRCUIT
Midplane 1.2V

IO0_PWREN J3 VRM_OUTEN_A
J50_1S VRM-A
IO Card 0 For CPU 0

IO1_PWREN U J4
J51_1S VRM_OUTEN_B
IO Card 1 6 VRM-B
0 For CPU 1
J52_1S IO2_PWREN
IO Card 2 MCU_SDA
U35 WILDCAT_PWRGD
IO3_PWREN MCU_SCL MCU
J53_1S
IO Card 3

Figure 4-135. Wildcat-S Power Enable Signals

EMC CONFIDENTIAL

Wildcat-S Chassis Power 197


Wildcat-S 12 volt standby power is present as soon as the power supplies receive
AC input voltage. The CMD which controls the below sequence uses 3.3v
Power-up which was derived from the 12 volt standby power. Its the CMD which
Sequence regulates the power-up sequence shown below. Once one of the
management FRU’s receive the power on command it instructs the power
supplies to turn on the V12 to both Wildcat-S blades. The first thing
which happens is the 5 volt regulator powers on when it receives the V12.
The CMD regulates the power up sequence using enables to the DC-DC
circuits from this point forward.

Figure 4-136. Wildcat-S Power up Good Sequence

EMC CONFIDENTIAL

198 CHASSIS POWER


CPU VID Each CPU had a dedicated (VRM) Voltage Regulator Modules which
supplies power (VCC_FSB0 & VCC_FSB1) for that CPU. Each VRM
Monitoring receives a 6-bit VID value from its corresponding CPU. This VID value
allows each CPU to select / control its own voltage source. All 1’s on the
VID lines will result in the VRM output being disabled. The (CMD)
Command/Monitor Device U23 in Figure 4-137 monitors the VID lines
from each CPU. The two groups of VID lines CPU0 VID<6:1> and CPU1
VID<6:1> are muxed to the CMD by using a single CMD GPIO pin from
the CMD chip. This muxed value, CPU_VID<6..0> is monitored by the
CMD.

VTT_CPU VTT_CPU

FSB1_VID_SEL U49 U96 FSB0_VID_SEL


J4 INTEL CPU 0 J3
VRM-B INTEL CPU 1 VRM-A
Woodcrest =
For CPU 1 VCC_FSB1 Woodcrest = VCC_FSB0 For CPU 0
Clovertown=
Clovertown=
FSB1_VID<6..0> FSB0_VID<6..0>

U125 CPU1_VID<6..0> CPU0_VID<6..0> U133

U131 /U128
VID_MUX_SEL

CPU_VID<6..0>
U23
(CMD)
SEQUENCER /
4
MONITOR

Figure 4-137. Wildcat-VID Monitoring

EMC CONFIDENTIAL

Wildcat-S Chassis Power 199


Wildcat-S Voltage Margining is controlled by three signals, named MFG_MODE_N,
MARGIN_HI_N and MARGIN_LO_N. These signals can be driven by
Margining the Nova Test Card connected to the midplane or by software which
controls the Server I/O GPIO pins.

MARGIN_HI_N

U23
SERVER MFG_MODE_N
(CMD)
IO

MARGIN_LO_N

M
I
MP_MARGIN_HI_N
D
P MP_MFG_MODE_N
L
MP_MARGIN_LO_N
A
N
E
Figure 4-138. Manufacturing Mode Signals used for Margining

MFG_MODE_N, MARGIN_HI_N and MARGIN_LO_N are presented to


the CMD which when asserted, command the CMD to margin all
non-VRM type POLs high or low, respectively. The default mode, when
neither signal is asserted or when both are asserted, is a non-margin state.
If both are asserted, the CMD should generate an error condition.

Margining shall be implemented by signal from the CMD driving the


POL trim pin. The signal is generated by the CMD, and is configurable
depending on the amount of margining, type of POL, and trim value, if
any. The analog margin allows margin amounts and trim values to be
programmable, eliminating the need for rework if trim or margin values
need to be changed. Refer to Figure 4-139. The two VRM’s which supply
power for the CPU’s will not be margined.

The margin signals are also sent out to the IO Modules (SLICS) to
margining the DC-DC power circuits on each module.

EMC CONFIDENTIAL

200 CHASSIS POWER


U1 TRIM+_V3_3 RC MGN_33
3.3V CIRCUIT

CM2 TRIM+_V1_8 RC MGN_18


1.8V CIRCUIT

CM1 TRIM+_V1_5 RC MGN_15


1.5V CIRCUIT
U23
CM3 TRIM+_V1_2 RC MGN_12 (CMD)
1.2V CIRCUIT SEQUENCER /
MONITOR
U29 TRIM+_V1_05 RC MGN_1P05
1.05V CIRCUIT

U9
1.0V
TRIM+_V1_0 RC
CIRCUIT
MGN_10
4
BUF_MARGIN_HI_N
BUF_MARGIN_LO_N
BUF1_MARGIN_HI_N BUF_MFG_MODE_N
J50_1S BUF1_MARGIN_LO_N
IO Card 0
BUF1_MFG_MODE_N

BUF1_MARGIN_HI_N
J51_1S BUF1_MARGIN_LO_N
IO Card 1 BUF1_MFG_MODE_N
U97
BUF1_MARGIN_HI_N
J52_1S BUF1_MARGIN_LO_N MARGIN_HI_N
IO Card 2 BUF1_MFG_MODE_N MARGIN_LO_N U41
MFG_MODE_N
SIO
BUF1_MARGIN_HI_N
J53_1S BUF1_MARGIN_LO_N
IO Card 3
BUF1_MFG_MODE_N

Figure 4-139. Voltage Margin Signals

EMC CONFIDENTIAL

Wildcat-S Chassis Power 201


CMD Pinout Below shows the CMD input and output signals grouped by function.
Block

V+12_POL_MEAS
VCC50_MEAS
EN33
VCC33_MEAS
EN18
VCC18_MEAS
EN15
DC-DC & VRM VCC15_MEAS DC-DC & VRM
EN12
Measurement VCC12_MEAS Output Enables
EN1P05
VCC1P05_MEAS
EN10
VCC10_MEAS
VCC09_MEAS
VRM_OUTEN_A
VCPU_A_MEAS VRM Output
VRM_OUTEN_B Enables
VCPU_B_MEAS

BUF_MFG_MODE_N
ICD_SDA Margin Signals
To Depop. debug BUF_MARGIN_HI_N
Connector ICD_SCL From SIO
BUF_MARGIN_LO_N
MP_PS2_SDA
U23
Midplane I2C BUS (CMD) MGN_33
MP_PS2_SCL
SEQUENCER / MGN_18
MONITOR MGN_15
Margin Signals
OVP_TRIP MGN_12
To Blade_Size FET to DC-DC circuits
MGN_1P05
FSB0_VRHOT_R_N MGN_10
From CPU VRM’s
FSB1_VRHOT_R_N

From CPU <1,0>


FSB_THERMTRIP_33 WILDCAT_PWRGD To MCU
MUXED WITH VID
WILDCAT_CMD_WARN To I2C Reset Expander
CPU_VID<6..0>

VID_MUX_SEL
BOARD_SEATED
VCC_CMD
From DEPOP. ICD_MCLR FBD_FAN_TACH
VCC_CMD
Debug Conn.
VREF_3P0
U25 Standby
VREF_3P0_RET VCC_CMD = 3.3v VCC_CMD is derived
Power Regulator VCC from 12V Standby

Figure 4-140. CMD Pinout

EMC CONFIDENTIAL

202 CHASSIS POWER


Chapter 5 OTHER CARDS

Chapter Content:

• Tornado IO Annex Extender Card ....................................................... 204


• Nova Test Card ....................................................................................... 215

EMC CONFIDENTIAL

203
5.1 Tornado IO Annex Extender Card

Introduction There are two IO Annex slots in the Dreadnought chassis, IO Annex A
and IO Annex B. Each slot can house one IO Annex card known as the.
Tornado Board and each Tornado board can be populated with two IO
modules (SLIC’s.

4 IO Module Slots

DAERear.emf
2 IO Annex Slots

Figure 5-141. Rear side of the Dreadnought Enclosure

The Tornado is a PCI Express based adapter board. It provides a


connection between each Blade and the two IO SLIC modules housed on
it, thereby providing IO port expansion for the dreadnought NAS and
SAN systems. Each blade communicates with a SLIC via a 4 lane PCI
Express lane.

BLADE A BLADE B

MCH MCH

PLX PLX

4 Lane PCI-
Express
Foxbow
Midplane
IO ANNEX A IO ANNEX B

PMC PMC PMC PMC


4 Lane PCI-
Express
IO 0 IO 1 IO 0 IO 1
Mid_2_Tornado.emf

Figure 5-142. SP to IO Annex Card Interface


EMC CONFIDENTIAL

204 OTHER CARDS


Blade A is connected to IO Annex A via two 4-lane PCI Express paths,
one of which goes to SLIC IO0 while the other goes to SLIC IO1. Blade B
is connected to IO Annex B via two 4-lane PCI Express paths, one of
which goes to SLIC IO0 while the other goes to SLIC IO1.

The other main interface on the Tornado board is the I2C bus which is
used for MCU, Resume and fan control.

EMC CONFIDENTIAL

Tornado IO Annex Extender Card 205


206
From Blade and Both
FAULT Management FRU's
Diagram

LED
FAN A FAN B
Tornado Block

POWER ANNEX_PWRGD I2C Address


LED
0xA6, 1010 011x 017-017-891
V+3.3
Aux
V+3.3V V+12
Aux Phillips Micro Resume
Fan Controller Prom
and Chiclet
Power Control
Watchdog
Latch Master Slave
Timer I2C Address
Bus Bus
I2C Address 0xE2, 1110 001x
Shuts off voltage regulators I2C Address Temp 0x90, 1001 000x
when Watchdog has failed or IO_ANNEX Sensor 0
0x90, 1001 000x

Figure 5-143.
multi-fan fault Regulator 1 I2C 1:4
Enable Switch
2 I2C
3
RS232
M idplane
Connector
RS232

I2C Address

OTHER CARDS
0xB0, 1011 000x PM C
X4 Lanes

EMC CONFIDENTIAL
X4 Lanes SATA
Buffer
I2C Address
100 MHZ 0xB2, 1011 001x
Local or PM C
X4 Lanes
X4 Lanes SATA

Tornado Block Diagram


Remote
Driver Buffer
V+12_Blade

33MHz USB
Clock
Power Enable
V+12

Chiclet 0 Connector Chiclet 1 Connector V+3.3 Soft


Start
Voltage Regulators
Chiclet card plugs Chiclet card plugs
into this connector into this connector 1.8V
USB
HUB
PCI Express PCI Express is used to communicate between the each blade and the two
SLIC’s on the blades corresponding IO Annex card. PCI Express uses the
term lanes, where a lane is made up of 4 signals, TX+, TX-, RX+ and
RX-.The PCI Bus between the Blade and the SLIC’s can be configured as
either 8-lanes wide for 1 SLIC or 2 *4-lanes for 2 SLIC’s. In the
Dreadnought/Tigon application the PCI Express is configure as 2 *
4-lanes with each 4-lane group used to access 1 SLIC. Both Blade and
SLIC have signals that define their PCI Express lane size configuration.

Table 5-18. SLIC PCI Express Lane Configuration


SLIC PCI Express Configuration
IO0/1_WIDTH_0 1 Input from each SLIC 0 = 2 * 4 Lane.
Both signals drive the
PLEASE_BIFURC_N to Midplane

Table 5-19. Blade PCI Express Lane Configuration


SLIC PCI Express Configuration
IO_SPW_0, 1 Input from each SLIC 00 = 4 Lane of PCI Express
IO_SPW_1 Pulled down on the Tornado Card.

PM8380 There are two PMC PM8380 Quad SATA/SAS device are used on the each
IO Annex tornado board. The function of each PM8380 is to buffer the
PCI-Express signals from the blade to the SLIC’s and from the SLIC’s to
the blade.
5
Blade A
Blade PCIe 0-7 IO PCIe 0-7

IO Blade IO Blade
PCIe 0-3 PCIe 0-3 PCIe 4-7 PCIe 4-7
IO0_PMC_RST_N TXA0-3+/- RXA0-3+/- TXA0-3+/- RXA0-3+/- IO1_PMC_RST_N
27.7MHZ PMC0_CLK PMC1_CLK 27.7MHZ
MCU_MSTR_SCL MCU_MSTR_SCL
MCU_MSTR_SDA MCU_MSTR_SDA
RXA0-3+/- TXA0-3+/- RXA0-3+/- TXA0-3+/-

PMCTxRxBuffer.emf

IO Module 0 IO Module 1

Figure 5-144. PCI-Express Tx/Rx Buffer Block Module

Each PM8380, one channel receives 4 lines of serial data from the blade
which it buffers and then outputted to the attached SLIC. On the second
channel, 4 transmit lines of serial data are received from the SLIC,
buffered and transmitted to the blade.The PM8380s are configured using
the I2C bus.
EMC CONFIDENTIAL

Tornado IO Annex Extender Card 207


PM8380 Pinout

EXP_FRM_BLD_<3:0>+/- EXP_FRM_IO 0_<3:0>+/-

EXP_TO_BLD_<3:0>+/- EXP_TO_IO0_< 3:0>+/-

PMC0_TXAPRE<1:0>
IO 0_PMC_RST_N B2 99
PMC0_TXBPRE<1:0>
27.7MHZ PMC0_CLK B1 97 PMC0_RXAEQ<1:0>
PMC PMC0_RXBEQ<1:0>
95
MCU_MSTR_SCL J1
I2C from PMC0_RXSWIZ
MCU_MSTR_SDA J2 100
MCU PMC0_SASLVL
94
PDN_PMC0_SEL J1 RBIAS_B0
Pulled up/ 101
down PDN_PMC0_ADR<2:1> J2

PMC_Pinout.emf

Figure 5-145. PM8380 Pinout

Signal
Descriptions
Table 5-20. PM8380 Pin Descriptions
SCHEMATIC SIGNAL I/O Description
PCI Express Interface
EXP_FRM_BLD<3:0>+/- I PCI Express receive serial differential data from the blade.
EXP_TO_BLD<3:0>+/- O PCI Express transmit serial differential data to the blade.
EXP_FRM_IO0/1_<3:0>+/- I PCI Express receive serial differential data from the SLIC.
EXP_TO_IO0/1_<3:0>+/- O PCI Express transmit serial differential data to the blade.
Configuration, Reset and Clocks Signals
IO0/1_PMC_RST_N* I When this reset signal is asserted (set low) all the digital logic is reset and all
registers return to their default values. When this signal is not asserted the device is
operational.
PMC0_CLK I The System clock input accepts a free running clock with a nominal frequency of
37.5 MHz.
PMC0/1_RXAEQ<1:0> I The Receive equalization weight selection configures the amount of equalization on
the RXA+/-<3:0> serial interface The receive equalization may also be over-ridden
using the TWI Interface
PMC0/1_RXBEQ<1:0> I The Receive equalization weight selection configures the amount of equalization on
the RXB+/-<3:0> serial interface The receive equalization may also be over-ridden
using the TWI Interface
PMC0/1_TXAPRE<1:0> I The transmit pre-emphasis weight selection configures the amount of pre-emphasis
on the TXA+/-<3:0> serial interface.The pre-emphasis may also be over-ridden
using the TWI Interface.
PMC0/1_TXBPRE<1:0> I The transmit pre-emphasis weight selection configures the amount of pre-emphasis
on the TXB+/-<3:0> serial interface.The pre-emphasis may also be over-ridden
using the TWI Interface.

EMC CONFIDENTIAL

208 OTHER CARDS


Table 5-20. PM8380 Pin Descriptions
SCHEMATIC SIGNAL I/O Description
PMC0/1_RXSWIZ I Pin Strap Connector Port B Swizzle. The receive port swizzle configures the receive
port ordering on the RXB+/-<3:0> serial interfaces. The RXSIIZB input may also be
over-ridden using the TWI interface.
PMC0/1_SASLVL I Pin Strap Transmit Swing level Control for SAS interfaces. The transmit swing level
control configures the maximum SAS level for SAS interfaces. The SASLVL input
may also be over-ridden by the TWI interface.

RBIAS_B0 I Analogue Bias Reference RBIAS. A 20 Kohm +/-1% resister must be connected
between RBIAS and adjacent VSS ball as close to the device as possible. The
external reference resistor pin creates a calibrated current for controlling the
high-speed serial interface thresholds, amplitudes and termination networks.
I2C Interface
MCU_MSTR_SCL I This signal is part of the devices 2-wire serial control interface and is used to clock
the data transfer to and from the registers.
MCU_MSTR_SDA I/O This signal is part of the 2-wire serial control interface and operates as the
bi-directional serial data port for TWI transfers.

LPC2131 The Tornado IO Annex board uses a Micro controller (MCU) to provide
additional functionality and diagnostic capability to the system. It
Tornado Micro Provides several monitoring functions for both the Tornado and any SLIC
controller

5
inserted into the Tornado. These functions include:

• Programming the Diplexed FPGA on the SLIC’s

• Monitoring the temperature and controlling the fan speeds.

• SLIC monitoring and status reporting

• Monitoring the fuses going to the SLIC’s

• Voltage margining.

Communication to the MCU will be through I2C/SMBUS supported


protocol.

Local margining The MCU will be able to margin the board high and low. Through the I2C
bus, the master will be able to put the MCU into manufacturing mode by
writing to a register. Once in manufacturing mode the MCU can be told to
margin the system high or low by writing to the manufacturing mode
register. The MCU will ensure that the system is nor margined high or
low simultaneously.

Led Control The MCU will have control of the fault LED on the front of the tornado
card. If the MCU fails to initialize, the fault LED will stay on
approximately 1 second before the watchdog trips shutting off the board.
The MCU will assert the fault LED when it encounters a fault. The fault
will be kept in a register that can be accessed via the I2C bus.

The MCU will also have control of the SLIC’s fault LEDS either
autonomously or through the I2C bus.

EMC CONFIDENTIAL

Tornado IO Annex Extender Card 209


Fan Control Fan control will be done autonomously by the MCU with the ability to be
overridden using I2C commands. The MCU will speed up the fans or
slow them down depending on the onboard temperature as measured by
a temperature sensor. The MCU monitors an output tachometer signal
from each fan and if the signal drops below the minimum acceptable
operating range the fan will be declared faulted. If only one fan fails the
MCU will continue to regulate the onboard temperature with the
remaining good fan, and report status to the application software.

Table 5-21. MCU’s response to onboard temperature changes


Temperature
(degrees C) MCU Response to temperature change
o Fans at minimum Speed
15 Fans set to 1/4 the max speed
25 Fans set to 1/2 the max speed
40 Fans set to 3/4 the max speed
55 Fans set to max speed. High Temp Warning Bit set
75 Board shuts down in Clariion Case. Fans set to Max
Speed. Over-Temp bit set.

Fuse Monitoring The MCU will monitor the two fuses through which power is supplied to
the SLIC’s. If no power is detected on the side of the fuse the MCU will
light the fault LED on the Tornado IO Annex card only.

Diplex FPGA The main responsibility of the MCU is to program the Diplex FPGA on
Programming any SLIC that may be inserted into the Tornado.The MCU holds a copy of
the Diplex FPGA image at all times. It will be up the POST or application
software to verify that the version of Diplexed FPGA software is up to
date. If the version is not up to date the POST or application software will
have to send down the latest version to the MCU via the I2C bus.

I2C Interface The MCU uses an I2C interface to communicate with the CPU. The
MCU’s hardware I2C interface is compatible with the 16KHz speed used
in the Dreadnought system.

There is an isolation switch between the MCU and the master on the bus.
This serves two purposes. The first is to allow us to isolate the MCU from
the bus in case for some reason that the MCU becomes stuck and holds
either the data or clock lines low. The second is to allow two devices with
the same address to be on the bus. The switch allows for either the MCU
in slot A or the MCU in slot B to be disconnected from the bus while the
other is communicating with the master. The switch address is dependent
on the slot the switch is in.

Table 5-22. Isolation Switch I2C Addresses


SLOT A SLOT b
Switch Address 0xE2 0xE2
PIC Address 0x94 0x94

EMC CONFIDENTIAL

210 OTHER CARDS


Tornado MCU
Pinout
IO0_FPGA_PRG_EN_N
V3.3 VREF
MCU_RST_N IO1_FPGA_PRG_EN_N

12Mhz XTAL1/2 MCU_FPGA_CFGDN_N


MCU_FPGA_STATUS_N
PWM_CNTRL_EN _N
MCU_FPGA _CONFIG_N
FAN_PWM_CBT
MCU_FPGA_DCLK
FAN_A_TACH
FAN_B_TACH MCU_FPGA_DATA

MCU_WD_OUT IO_ANNEX_PWRGD_IN
MCU_WD_ENABLE IO_FUSES_OK
MCU_I2C_ATTN_N
MCU_IO1_RST_N
IO 0/1_FLT
LPC2136
MCU_TXD0 MCU_JTAG_MSTR
MCU_RXD0 MCU_PWRGD_CNTRL
MCU_I2C_RST_N
MCU_MSTR_SCL IO0/1_PWRGD
MCU_MSTR_SDA IO 0/1_PWREN
MCU_SCL IO0_INS_N
MCU_SDA IO_ANNEX_FLT_LED
IO_ANNEX_RST_N

5
MCU_TDO MP_SLOT _ID
MCU_TDI
LCL_MFG_MODE_N
MCU_TCK
LCL_MARGIN_LOW_N
MCU_TMS LPC2136 _Pinout.emf LCL_MARGIN_HIGH_N
MCU_TRST_N

Figure 5-146. Tornado Micro controller Pinout

LPC2131 Signal
Descriptions
Table 5-23. LPC2131 Signal Descriptions
SCHEMATIC SIGNAL I/O Description
Miscellaneous
Vref I 3.3V
MCU_RST_N I When asserted this signal resets the MCU.
XTAL1/2 I 12MHz Clock
Control, Monitoring and Status Signals
PWM_CNTRL_EN_N O Output enable for the Bus FET Switch for the Fan controls.
FAN_PWM_CBT O Control the FAN speed.
FAN_A_TACH I Tachometer output signal from the fans which is monitored by the MCU. If this
signal drops below the minimum acceptable operating range the fan will be declared
faulted
FAN_B_TACH I Tachometer output signal from the fans which is monitored by the MCU. If this
signal drops below the minimum acceptable operating range the fan will be declared
faulted
MCU_WD_OUT I MCU watchdog monitor data latch
EMC CONFIDENTIAL

Tornado IO Annex Extender Card 211


Table 5-23. LPC2131 Signal Descriptions
SCHEMATIC SIGNAL I/O Description
MCU_WD_ENABLE I Watchdog enable.
IO_FUSES_OK I Indicates that the two fuses through which power is supplied to the SLIC’s are OK
MCU_I2C_ATTN_N O I2C Interrupt
MCU_IO0/1_RST_N O Output from the MCU to reset the SLIC’s
IO0/1_FLT O Output from the MCU to the SLIC indicating that the Tornado IO Annex card has a
fault condition.
MCU_JTAG_MSTR O Controls the Mux’s on the IO ANNEX JTAG Scan Chain.
MCU_PWRGD_CNTRL
IO_ANNEX_PWRGD_IN O Will light the power Good LED on the Tornado board. Also indicates power status to
the blade via the midplane.
MCU_I2C_RST_N O Used to reset the I2C isolation switch on the Tornado board.
IO0/1_PWRGD I Input power good signal from the SLIC to the MCU
IO0/1_PWREN O Power enable signal to the SLIC.
IO0_INS_N I Input signal from the SLIC indicating that it is inserted
IO_ANNEX_FLT_LED O Sets the Fault LED on the Tornado IO Annex board.
IO_ANNEX_RST_N I This reset signal comes from the blade and will reset the IO annex card and the
attached SLIC’s.
MP_SLOT_ID I Midplane connection indicating whether this is IO Annex A or B.
LCL_MFG_MODE_N O This signal indicates to the SLIC’s that the MCU is in manufacturing mode. The
MCU can be put into manufacturing mode by writing to a register in the MCU via
the I2C bus. Once in manufacturing mode the MCU can then be told to margin the
system voltage either high or low.
LCL_MARGIN_LOW_N I From the midplane to the MCU to margin the voltage low.
LCL_MARGIN_HIGH_N I From the midplane to the MCU to margin the voltage high.
FPGA Programming Signals
IO0_FPGA_PRG_EN_N O Enables the buffer for programming SLIC 1
IO1_FPGA_PRG_EN_N O Enables the buffer for programming SLIC 0
MCU_FPGA_CFGDN_N I Configuration Done status from the SLIC FPGA.
MCU_FPGA_STATUS_N I Status information during FPGA programming.
MCU_FPGA_CONFIG_N O Enable program mode for the FPGA.
MCU_FPGA_DCLK O Data Signal for programming the FPGA.
MCU_FPGA_DATA O Clock Signal for programming the FPGA.
I2C Interface Signals
MCU_TXD0 O UART Interface that goes to a header and used for debugging the board.
MCU_RXD0 I UART Interface that goes to a header and used for debugging the board.
MCU_MSTR_SCL I/O Master I2C bus clock used to communicate with the two PMC’s and the Terminal
sensor on the Tornado board
MCU_MSTR_SDA I/O Master I2C bus data signal used to communicate with the two PMC’s and the
Terminal sensor on the Tornado board
MCU_SCL I/O Slave I2C bus clock from the midplane used to communicate with the MCU,
Resume PROM and the two SLIC’s connected.

EMC CONFIDENTIAL

212 OTHER CARDS


Table 5-23. LPC2131 Signal Descriptions
SCHEMATIC SIGNAL I/O Description
MCU_SDA I/O Slave I2C data bus from the midplane used to communicate with the MCU, Resume
PROM and the two SLIC’s connected.
JTAG Interface Signals
MCU_TDO O JTAG Interface
MCU_TDI I JTAG Interface
MCU_TCK I JTAG Interface
MCU_TMS I JTAG Interface
MCU_TRST_N I JTAG Interface

Tornado to
Midplane
Interface

Tornado Midplane BLADE A


EXP_FRM_BLD_<3:0>+/- XAIO_EXPT <3:0>_+/- XAIO_EXPT <3:0>_+/-
PMC 0 EXP_TO_BLD_<3:0>+/- XAIO_EXPR<3:0>_+/- XAIO_EXPR <3:0>_+/-

5
EXP_FRM_BLD_< 7:4>+/- XAIO_EXPT <7:4>_+/- XAIO_EXPT <7:4>_+/-
EXP_TO_BLD_<7:4>+/- XAIO_EXPR <7:4>_+/- XAIO_EXPR <7:4>_+/-
PMC 1 Blade A
MP_USB+/- ANNEX_A_USB_+/- G6/H6 ANNEX_A_USB_+/-
MP100MHZ_CLK+/_ IOA_100MHZ_CLK_+/- G5/H5 IOA_100 MHZ_CLK_+/_
IO0/1_UART_TO_BLD RS232 _TX0/1_TO_BLDA RS232_TX0/1_TO_BLDA
IO0/1_UART_FRM_BLD RS232 _RX0/1_FR_BLDA RS232_RX0/1_FR_BLDA
IO_ANNEX_RST_N IO_ANNEX_A_RST_N I16 IO_ANNEX_A_RST_N
IO_ANNEX_PWRGD IO_ANNEX_A_PWRGD I19 IO_ANNEX_A_PWRGD
IO0/1_FLOWCNTL_N FLOW_CNTRL_AUAR_0/1 E16/E17 FLOW_CNTRL_AUAR_0/1
IO_ANNEX_PWR_EN IO_ANNEX_A_PWR_EN E15 IO_ANNEX_A_PWR_EN

MP_I2C_SCL PS_A_I2C_SCL PS_A_I2C_SCL


Blade A &B, MGMT A&B, PSA ,
MP_I2C_SDA PS_A_I2C_SDA PS_A_I2C_SDA Nova Tst Card , Resume
IO_ANNEX_INS_N IO_ANNEX_A_INS_N IO_ANNEX_A_INS_N
Blade A , MGMT A&B,
ANNX_FLT_LED_R
J1
IO_ANEX_A_FLT_LED J21 IO_ANEX_A_FLT_LED

MP_I2C_RST_N PS_A_I2C_RST_N PS_A_I2C_RST_N Blade A &B, MGMT A&B, PSA , Nova Tst Card
MP_RTCK ANNEX_A_MCU_RTCK ANNEX_A_MCU_RTCK
MP_MCU_RST_N ANNEX_A_MCU_RST_N ANNEX_A_MCU_RST_N Nova Tst Card
MP_JTAG_EN_N IOA_JTAG_EN_N IOA_JTAG_EN_N
PLEASE_BIFURC_N PLEASE_BIFURC_A_N PLEASE_BIFURC_A_N Blade A &B, IO Annex B
MARGIN_HIGH_N V_MARGIN_A_HIGH_N V_MARGIN_A_HIGH_N
Blade A , MGMT A,
MARGIN_LOW_N V_MARGIN_A_LOW_N V_MARGIN_A_LOW_N
Nova Tst Card
MFG_MODE_N A_MFG_MODE_N A_MFG_MODE_N
MP_SLOT_ID GND GND Slot ID
IO_ANNEX_INS_IN_N GND GND Insert GND loop start

JTAG_IO0_SEL_N ANEXA_CHAIN_SEL _0 ANEXA_CHAIN_SEL_0


MP_TDO IOA_JTAG_TDO IOA_JTAG_TDO
MP_TDI IOA _JTAG_TDI IOA_JTAG_TDI
Blade A, Nova Tst Card
MP_TCK IOA _JTAG_TCK IOA_JTAG_TCK
MP_TMS IOA_JTAG_TMS IOA_JTAG_TMS
MP_TRST_N IOA_JTAG_TRST_N IOA_JTAG_TRST_N

Midplane_Pinout.emf

Figure 5-147. Tornado to Midplane Interface

EMC CONFIDENTIAL

Tornado IO Annex Extender Card 213


EMC CONFIDENTIAL

214 OTHER CARDS


5.2 Nova Test Card

Introduction The Dreadnought test board code name is the NOVA. It preforms four
major functions;

• Connectivity for Manufacturing mode/Margining for Blades A & B.

• Connectivity for CPU emulation for blades A & B using eXtended


Debug port (XDP) ports, or Blade A or B (configurable) using ALT
ITP port on the airdam.

• Connectivity for 8 JTAG loops.

• Connectivity for MCU Emulation for Blades A & B.

Figure 5-148. Nova Test Card

The Nova test card plugs into the front of the unit into the midplane.

It resides in the front side opposite the IO Annex Slots.

EMC CONFIDENTIAL

Nova Test Card 215


The XDP connectors for Blades A & B are located inboard behind the
respective MCU connectors and provide an XDP interface for CPU
emulation. The ALT ITP connector on the airdam is normally not
connected but may be configured to Blade A or B by populating some 0
ohm resistors. this adapts the newer XDP interface to the older interface
and allows direct connection of an ITT uMaster 3040 DPA. The following
diagram shows the Airdam, JTAG, XDP and MCU connections.

JTAG ALT
MFG A MCU A MCU B MFG B
POD ITP
017-001-097 017-006-250 017-017-217 017-006-250 017-006-250 017-001-097

OOOOOOOOOO OOOOOOOOOOOOO OOOOOOOOOO


OOOOOOOOOO OOOOOOOOOOOO OOOOOOOOOO

Figure 5-149. Airdam on the Nova Test Card

EMC CONFIDENTIAL

216
POD TDO
SW1 P3 BLDA_ITP_ DBRST_N 1.2V This enable anded with
JTAG enable
017-017-303
BLDA_ITP_TCK0 1.2V o Pod TCK
30 PINS
BLDA_ITP_TRST_N 1.2V 3.3v to o Pod TRST JTAG
1.2v POD
Blade A BLDA_ITP_TMS_MAIN 1.2V Level o Pod TMS
Shift From JTAG Diagram 1
BLDA_ITP_TDI_MAIN 1.2V o Pod TDI
X
BLDA_ITP_TDO_MAIN 1.2V 1.2v to 3.3v

M
SW2 P3
Level Shift
SW3 P1 BLDA_ITP_DBRST_N 1.2V This enable anded with
JTAG enable SW1 P1
BLDB_ITP_TCK0 1.2V o

I Blade B
BLDB_ITP_TRST_N 1.2V

BLDB_ITP_TMS_MAIN 1.2V
3.3v to
1.2v
Level
o

o
0 1
U51 P15
JTAG Select

D
U51 P6
Shift
BLDB_ITP_TDI_MAIN 1.2V o
U52 P15
BLDB_ITP_TDO_MAIN 1.2V 1.2v to 3.3v

P
U51 P2
Level Shift
To Blade A Reset
SW1 P3
0 1

L Same 6 signals as above


To Blade B Reset SW3 P1

A
plus BLDB_FSB_RST, BPM's
3,4,5 from Proc 0 and 1 ITP B
HEADER

N Same 6 signals as above plus


BLDA_FSB_RST, BPM's 3,4,5
from Proc 0 and 1 ITP A

E MFG mode, Margin high,


Margin low, 3.3v blade
HEADER

sense, I2C SCL,SDA,


RST_N and ARB_A1 MFG A
HEADER
MFG mode, Margin high,
Margin low, 3.3v blade
sense, I2C SCL,SDA,
RST_N and ARB_B1 MFG B

5
HEADER

Figure 5-150. MFG & XDP BLOCK DIAGRAM

MFG Connectivity for Manufacturing Mode/margining for Blades A & B.


Connectors J6 The following table will give the pinout mapping for the DB9 connectors
& J7 at J6 & J7.

Table 5-24. MFG Connector Pins


MFG A SIGNAL NAME J6 MFG B S IGNAL NAME J7 PIN NUMBER

A_MFG_MODE_N B_MFG_MODE_N 1
V_MARGIN_A_HIGH V_MARGIN_B_HIGH 2
V_MARGIN_A_LOW V_MARGIN_B_LOW 3
PS_A_I2C_RST_N PS_B_I2C_RST_N 4
3.3V_BLADEA -SENSE 3.3V_BLADEB_SENSE 5
GND GND 6
PS_A_I2C_SCL PS_B_I2C_SCL 7
PS_A_12C SDA PS_A_12C SDA 8
I2C_ARB_A1 I2C ARB_B1 9

EMC CONFIDENTIAL

Nova Test Card 217


XDP Connectivity for CPU emulation for blades A & B using eXtended Debug
port (XDP) ports, or Blade A or B (configurable) using ALT ITP port on
Connectors J3 the airdam. These connectors are intended for American Aerium CPU
& J4 Emulation as well as the MicroMaster ITT emulation

Table 5-25. XDP Pinout


XDP A SIGNAL NAME XDP B SIGNAL NAME PIN NUMBER
BLADEA_P1_BPM5 BLADEB_P1_BPM5 3
BLADEA_P1_BPM4 BLADEB_P1_BPM4 5
BLADEA_P1_BPM3 BLADEB_P1_BPM3 9
BLADEA_P0_BPM5 BLADEB_P0_BPM5 21
BLADEA_P0_BPM4 BLADEB_P0_BPM4 23
BLADEA_P0_BPM3 BLADEB_P0_BPM3 27
VTTA_CPU VTTB_CPU 39
VTTA_CPU VTTB_CPU 43
VTTA_CPU VTTB_CPU 44
BLDA_ITP_FSB_RST BLDB_ITP_FSB_RST 46
BLDA_ITP_DBRST_N BLDB_ITPDBRST_N 48

EMC CONFIDENTIAL

218
BLADE A ENABLE_N 3.3V ITP enable and JTAG enable
017-017-893
Anded for this inputt
BLADE A TCK 3.3V o Pod TCK
120 PINS
BLAD E A TRST 3.3V o Pod TRST JTA G
POD
Blade A BLADE A TMS 3.3V o Pod TM S

BLADE A TDI 1.2V 3.3 to 1.2 Level o Pod TDI


Shift
BLAD E A TDO 3.3V

M
SW 2 P2
BLADE B ENABLE _N 3.3V ITP enable and JTAG enable
Anded for this inputt SW 2 P4
BLADE B TCK 3.3V o

I Blade B
BLAD E B TRST 3.3V

BLADE B TMS 3.3V


o

o
0 1
U51 P1
JTAG SELECT_N

D
U51 P7
BLADE B TDI 1.2V 3.3 to 1.2 Level o
U52 P5
Shift
BLAD E B TDO 3.3V

P
U52 P7
IO A ENABLE_N 3.3V
o 0 1 SW 4 P1
IO A TCK 3.3V

L IO A
IO A TRST 3.3V

IO A TM S 3.3V
o

o
SW 4 P3

A IO A TDI 3.3V

IO A TDO 3.3V
o

N IO B ENABLE_N 3.3V

IO B TCK 3.3V o
0 1

E IO B
IO B TRST 3.3V

IO B TM S 3.3V
o

IO B TDI 3.3V o

IO B TDO 3.3V

V+3.3 V+1.2
V +12_SBY A 2AM PS 0 1
Blocking V+12 3.3V 1.2V
V +12_SBY B 2AM PS Diodes Regulator Regulator To JTAG Block Diagram
below
G ND 4 AM PS GND X

5
017-017-893

120 PINS

From JTAG Block Diagram


above

M MGT A ENABLE_N 3.3V

MGT A TCK 3.3V o SW5 P1

I MGT A
MGT A TRST 3.3V

MGT A TMS 3.3V


o

o
SW5 P2

D MGT A TDI 3.3V

MGT A TDO 3.3V


o
U53 P15
JTAG SELECT_N

P
U53 P1
MGT B ENABLE_N 3.3V
0 1
MGT B TCK 3.3V o

L MGT B
MGT B TRST 3.3V

MGT B TMS 3.3V


o

A MGT B TDI 3.3V

MGT B TDO 3.3V


o

0 1

N
E
X To XDP Block Diagram

Figure 5-151. JTAG BLOCK DIAGRAM FOR A\B FRU’s

EMC CONFIDENTIAL

Nova Test Card 219


JTAG There are Eight JTAG chains within the Dreadnought chassis. Each one is
independent of the others and any combination of them can be used via
Connector J5 the enable signals. This allows total flexibility and the ability to handle
any configuration of inserted FRU’s. The eight are BladeA, BladeB, IO
Annex A, IO Annex B, XDPA, XDPB, MGMT A and MGMT B. The Blade
chains do not include the processors as they are included in the XDP
chains. The XDP chains are tapped into for the BLADE JTAG chains as the
XDP signals run from the midplane to the XDP headers directly. The
Blade chains include the two double wide or four single wide SLIC IO
boards that plug into the blades I/O slots directly.

Table 5-26. JTAG Connector Pins


JTAG CONNECTOR
SIGNAL NAME J5 PIN NUMBER
POD_TMS 1
POD_TRST 2
POD_TDI 3
V+3.3 6
POD_TDO 7
POD_TCK 12
SELECT_0_N 15
SELECT_1_N 16
SELECT_2_N 19
SELECT_3_N 20

EMC CONFIDENTIAL

220
A Side MCU Select

BLADE
SW5 P3

017-017-893 062-000-428 MGMT


SW6 P3
BLADEA_JTAG_TCK
120 PINS
o
BLADEA_JTAG_TRST ANNEX
o SW6 P1
BLADEA_JTAG_TMS
Blade A
o
BLADEA_JTAG_TDI 3.3 to 1.2V level shift needed
o

M BLADEA_MCU_RST_N

BLADEA_JTAG_TDO
o

I BLADEA_MCU_RTCK
o

D 073-006-221

P MGMT_A_JTAG_TCK ATCK 9 J8
o
MGT_A_JTAG_TRST_N ATRST 3
MGT A o

L MGMT_A_JTAG_TMS ATMS 7
MCU
o Header
MGMT_A_JTAG_TDI ATDI 5 20 pins
o

A
15 A
MGT_A_JTAG_MCU_RS ARST
Side
o
MGMT_A_JTAG_TDO ATDO 13
o

N MGT_A_JTAG_MCU_CK ARTCK 11
o

E IOA_JTAG_TCK
o
IOA_JTAG_TRST_N
Annex A IOA_JTAG_TMS
o

o
IOA_JTAG_TDI
o
ANNEX_A_MCU_RST_N

5
o
IOA_JTAG_TDO
o
ANNEX_A_MCU_RTCK
o

Figure 5-152. MCU Headers J8 & J11

MCU Connector There are two headers for MCU emulation. one for the A Field
replaceable Units (FRU’s) the other for the B side FRU’s. Each side has 3
J8 & J11 FRU’s that can be emulated, one at a time. These are the Blade,
Management board and Annex for each side. This can be done with the
configuration switches manually or via program control.

Table 5-27. MCU Connector Pins


MCU A SIGNAL NAME J8 MCU B SIGNAL NAME J11 PIN NUMBER
V+3.3 V+3.3 1
V+3.3 V+3.3 2
MCU_A_TRST MCU_B_TRST 3
MCU_A_TDI MCU_B_TDI 5
MCU_A_TMS MCU_B_TMS 7
MCU_A_TCK MDC_B_TCK 9
MCU-a-RTCK MCU_B_RTCK 11
MCU_A_TDO MCU_B_TDO 13
MCU_A_RST MCU_B_RST 15
MCU_A_PDN1 MCU_B_PND1 17
MCU_A_PDN2 MCU_B_PDN2 19
EMC CONFIDENTIAL

Nova Test Card 221


Nova Test card The NOVA card is only compatible with the Dreadnought chassis
containing the Fogbow midplane. It is designed to support connections
Switch Settings for the following emulators / test pods without opening the Dreadnought
chassis.

Asset-InterTech boundary Scan Test system; PCI-100, PCI-400, USB-10 up


to 4 American Arium Emulators: ITP-700 and XDP pods up to 16 MHz.
ITT MicroMaster 4030: ITP-700 with Universal DPA pod up to 20 MHz.
Keil Micro Vision 3: MCU emulator

Table 5-28. Switch Settings


FUNCTION OR
OPERATON SWITCHES >ON CONNECTOR
JTAG BLADE A -No SLIC’s SW2-1,2,4 J5
JTAG Blade A -SLIC 0 SW2-1,2,4 + SW5-1,2 J5
JTAG Blade A -SLIC1 SW2-1,2 + SW5-1,2 J5
JTAG Blade A -SLIC 2 SW2-1,2,4 + SW5-2 J5
JTAG Blade A -SLIC 3 SW2-1,2 + SW5-1 J5
JTAG Blade B- NO-SLIC 0 SW5-3,4 +SW6-2 J5
JTAG Blade B -SLIC 0 SW5-3,4 + SW6-2,3,4 J5
JTAG Blade B -SLIC1 SW5-3,4 + SW6-2,3 J5
JTAG Blade B -SLIC 2 SW5-3,4 + SW6-2,4 J5
JTAG Blade BA -SLIC 3 SW5-3,4 + SW6-2 J5

JTAG MGMNT A SW4-1 J5


JTAG MGMNT B SW4-2 J5
JTAG ANNEXA_SLIC0 SW3-1,2 J5
JTAG ANNEXA_SLIC1 SW3-1 J5
JTAG ANNEXB)SLIC0 SW3-3,4 J5
JTAG ANNEXB_SLIC1 SW3-3 J5

EMC CONFIDENTIAL

222
Index

A P
Acronyms 25 PCI 79
PCI-Express 41
B Port 80 99
POST 91
battery 99
BIOS 91, 97 R

C RS232 91, 98

COM 1 98 S
COM 2 98
SIO 91
E strapping 96
Super IO 79
Ethernet 79, 114

H
HubLink 41

I
I2C 79
ICH 41

L
LAN 114
Lithium Battery 99
Low Pin Count (LPC) 79
LPC 79, 91

N
NVRAM 91, 99

EMC CONFIDENTIAL

Index 1
EMC CONFIDENTIAL

2 Index

You might also like