x4640 Server Architecture

SUN FIRE™ X4640
SERVER ARCHITECTURE
Highly-Scalable x64 Computing Based on Powerful
Six-Core AMD Opteron™ Processors
White Paper
November 2009
Sun Microsystems, Inc.
Table of Contents
Executive Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
Delivering Massive x64 Single-System Scalability . . . . . . . . . . . . . . . . . . . . . . . . . 2

The Sun Fire X4640 Server . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
System Comparisons . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
Enterprise-Class Capabilities . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
AMD Opteron Processor Technology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11

Innovative Processor Technology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
Forth-Generation Six-Core AMD Opteron Processors . . . . . . . . . . . . . . . . . . . . . . . . 12
Coherent HyperTransport 3 (cHT3) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
HyperTransport Assist (HT Assist) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
AMD-P Power Management Technology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
AMD-V Virtualization Technology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
Server Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
System-Level Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
CPU/Memory Modules . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22
Sun Fire X4640 Server Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24
Enterprise-Class Operating System and Management Software . . . . . . . . . . . . . 30

Solaris 10 OS Support . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30
Integrated Lights Out Management (ILOM) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32
Sun xVM Ops Center . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36
Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38
For More Information . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39
This Page Intentionally Left Blank
1 Executive Summary Sun Microsystems, Inc.
Executive Summary
Even as the x64 computing architecture has grown in ubiquity, organizations continue
to demand new levels of scalability from their computing platforms. Databases need to
process ever-larger number of transactions. Growing data warehouses require larger
analysis efforts to extract useful information. High performance computing (HPC)
applications are increasingly pressed into service to answer our most critical business
needs and scientific inquiries. While the latest x64 processors offer very impressive
levels of performance, organizations need systems that offer balanced scalability and
performance, to preserve and capitalize on investments.
At the same time, many organizations are responding to new levels of server sprawl
and complexity that have resulted from deploying large numbers of legacy systems.
Administrative, energy, and real estate costs continue to grow unabated — severely
straining IT budgets. Increasingly, many are responding with consolidation strategies
that combine ever more powerful standard x64 systems with innovative new
approaches to virtualization. Not only does consolidation offer to improve vital resource
utilization, but it can also greatly enhance business agility by providing computational
resources where they are needed most.
The Sun Fire™ X4640 server employs the latest high-performance AMD Opteron™
processors in a massively-scalable system that demonstrates Sun's proven datacenter-
centric design focus. Ideal for databases, HPC applications, and mission-critical
applications alike, this server delivers scalable and manageable x64 computing in
dense and servicable rackmount enclosures. Using the latest Six-Core AMD Opteron
processors, the Sun Fire X4640 server is particularly well suited for consolidation and
virtualization initiatives. With compelling compute power, memory capacity, and I/O
density, as well as considerable energy efficiency, organizations can live within their
means even as they grow their IT infrastructure.
With a choice of operating systems — including the Solaris™ Operating System (OS),
Linux, Microsoft Windows, and VMware — the Sun Fire X4640 server effectively
combines Six-Core AMD Opteron processors with balanced system design. Large
memory support, considerable I/O bandwidth, and up to eight processor sockets (up to
48 processor cores) give this server the capacity to handle the most demanding
applications. With datacenter operation as a fundamental design assumption, the
server also offers redundant and hot-swappable components, efficient front-to-back air
flow, highly-efficient power supplies, and built-in system management tools. Beyond
supporting a large number of processors, the Sun Fire X4640 server is engineered to
provide mission-critical application availability, even as it conserves valuable energy
resources and lowers operational expenses. This document details the systems
architecture of Sun Fire X4640 server, along with key software components.
2 Delivering Massive x64 Single-System Scalability Sun Microsystems, Inc.
Chapter 1
Delivering Massive x64 Single-System Scalability
To stay competitive, organizations must deploy applications quickly, delivering compute

power where it is needed most, all with increasingly little margin for error. Databases,
high performance computing applications, and the growth of Web-centric business
models are all placing significant demands on IT infrastructure. At the same time,
systems deployed in the datacenter must now offer considerable investment protection
and agility, scaling gracefully under pressure even as they deliver business-critical and
mission-critical applications. Regrettably, many datacenters have become sprawling
and complex, and most are up against very real constraints in terms of power, cooling,
and real estate.
Consolidation through virtualization has emerged as an effective strategy for

addressing the very real need for computing scalability while increasing the work that
can be done in a given power, thermal, and physical footprint. Consolidation can
improve resource utilization, reduce administrative complexity, and drive down IT costs.
Consolidating many smaller legacy servers into fewer more powerful systems can also
help to minimize administrative workloads while increasing capacity and conserving
valuable datacenter floor space. Energy costs can be drastically reduced, vastly
improving available performance relative to the amount of energy consumed.
Nonetheless, effective consolidation strategies require truly robust, scalable, and
available computing platforms.
The Sun Fire™ X4640 Server

To help IT managers address the challenge of increasing capacity while managing
datacenter growth, the Sun Fire X4640 server provides industry-leading scalability and
performance based on the latest AMD Opteron processors. This server features high
performance and unprecedented density in an energy-efficient and compact four rack
unit (4U) form-factor. With capabilities that complement the rest of the Sun™ server
product line, the Sun Fire X4640 server raises the bar for 32- and 64-bit enterprise-class
computing. These systems offer:
• Best-In-Class Performance
The Sun Fire X4640 server features the latest Six-Core AMD Opteron processors.
With a sophisticated cache hierarchy, and on-chip memory management, these
processors offer high system performance and throughput compared with systems
based on earlier-generation x86 chipsets. Making the most of the AMD Direct
Connect Architecture, the Sun Fire X4640 server can house up to eight Six-Core
AMD Opteron processors — yielding up to 48 AMD Opteron cores.
• Remarkable Compute and Memory Density

Density is the cornerstone of Sun’s server designs, and the Sun Fire X4640 server is
no exception. When populated in a standard 40 rack unit enclosure, this server
facilitates a single rack with up to 480 cores, 640 DIMM slots, and 60 PCI Express
slots. In addition, each server can house up to four internal 2.5-inch SAS disk
drives. The Sun Fire X4640 server provides the densities needed to achieve
consolidation and virtualization efficiencies. Such density facilitates the
consolidation of many smaller servers, helping to conserve real estate, lower
energy expense, and make the most of costly administrative talent. In addition,
these servers support multiple operating systems, helping to simplify
consolidation efforts and diminish server sprawl.
• Extensive System Expandability
The ability to expand and upgrade a server over time reduces the need for
additional capital acquisitions and lowers application lifecycle costs. The Sun Fire
X4640 server supports two, four, six, or eight processor sockets per system and
provides maximum memory configurations up to 512 GB (using 8 GB DIMMs). Four
Gigabit Ethernet ports are standard, as are five USB ports (two front, two back,
one internal), and one video port (HD-15). With breakthrough system densities,
this server is designed to scale to support new users, more transactions, or new
32-bit or 64-bit applications — enhancing system longevity and increasing overall
return on investment (ROI).
• Improved Energy Efficiency
Sun offers a portfolio of eco-responsible products and computing solutions to
address a broad range of infrastructure requirements. In Sun Fire X4640 servers,
AMD Opteron processors incorporate new technologies that minimize power use
and enhance energy efficiency. AMD PowerNow! technology optimizes processor
performance relative to the power consumed, allowing CPU clock frequency to be
adjusted to the needs of applications. Enhanced AMD PowerNow! features in Six-
Core AMD Opteron processors offer features such as Independent Dynamic Core
technology and Dual Dynamic Power Management. AMD Smart Fetch technology
allows cores to sleep while still providing full data access to other cores. High-
efficiency power supplies in the server chassis lessen overall power consumption.
Variable-speed fans, effective disk carrier design, and front-to-back air flow in the
chassis help to effectively cool the system and maintain appropriate ambient
temperatures for both the processor and the system.
• Enterprise-Class High Availability

The Sun Fire X4640 server is designed with enterprise-class reliability, availability,
and serviceability (RAS) features. To maximize uptime, systems include redundant
hot-swappable fans and can be configured with redundant hot-swappable power
supplies. Internal SAS disk drives can be configured for RAID 0 or 1, and disk drives
are also hot-swappable. Four integrated Gigabit Ethernet ports enhance network
availability and can be utilized in failover configurations. On-board system
management tools encourage proactive remote monitoring and intervention.
• Tightly-Integrated Management
To support out-of-band management, the Sun Fire X4640 server incorporates an
Integrated Lights Out Management (ILOM) service processor. This built-in
hardware-based management functionality allows administrators to monitor and
manage systems remotely, letting them take corrective action as necessary to
minimize unplanned downtime.
Sun Fire X4640 servers combine best-in-class performance with noteworthy compute,
memory, and I/O capacities. As a result, these systems are designed to scale up, scale
out, and scale within, enabling deployment in a wide range of application
architectures:
• Scale-up architectures — With up to 48 processor cores available per system, this
server is well suited to scale for growing workloads that deliver Web, database, and
other key infrastructure services.
• Scale-out architectures — With large memory capacities, significant internal storage,
four Gigabit Ethernet ports, and high-bandwidth PCI Express expansion, these servers
can scale to solve complex computing problems that demand intensive computing
power and data bandwidth.
• Scale-within capabilities — With the ability to support a wide range of virtualization
technologies, Sun Fire X4640 servers are ideal systems to consolidate multiple
applications within a single extensible platform.
Figure 1 illustrates the 4U Sun Fire X4640 server enclosure.
Figure 1. The Sun Fire X4640 server offers up to 48 AMD Opteron cores in only four
rack units (4U).
System Comparisons
The Sun Fire X4640 server complements Sun’s extended x64 server product line. Sun
provides a wide range of AMD Opteron based servers with diverse capabilities. Table 1
compares the features of Sun Fire X4140, X4440, and X4640 servers.
Table 1. Sun Fire X4140, X4440, and X4640 server feature comparison
Feature Sun Fire X4140 Server Sun Fire X4440 Server Sun Fire X4640 Server
Processors One or two Six-Core AMD Two or four Six-Core AMD Two, four, six, or eight Six-Core
Opteron series 2000 processors Opteron Series 8000 processors AMD Opteron Series 8000
processors
Memory capacity Up to 128 GB Up to 256 GB Up to 512 GB
(2, 4, or 8 GB DDR2 DIMMs) (2, 4, or 8 GB DDR2 DIMMs) (2, 4, or 8 GB DDR2 DIMMs)
Maximum internal Up to eight SFF 2.5-inch SAS 73 Up to eight SFF 2.5-inch SAS 73 Up to four SFF 2.5-inch SAS 73 or
disk drives or 146 GB disk drives, choice of or 146 GB disk drives, choice of 146 GB disk drives, internal RAID
RAID HBAsa RAID HBAsa
RAID Hardware RAID 0, 1, 1E, 10, 5, Hardware 0, 1, 1E, 10, 5, 5EE, Hardware 0 or 1
5EE, 50, 6, and 60 with SAS 50, 6, and 60 with SAS RAID
RAID HBA HBA
Removable and Slimline DVD+/-RW Slimline DVD-R Slimline DVD/CD-RW
pluggable I/O Five USB 2.0 ports (Two front, Five USB 2.0 ports (Two front, Five USB 2.0 ports (Two front,
two rear, one internal) two rear, one internal) two rear, one internal)
Expansion slots Two x8 PCI Express slots, One x4 PCI Express slot, Two PCI-X slots,
One x16 PCI Express slot Four x8 PCI Express slots, and Four x8 PCI Express slots, and
One x16 PCI Express slot Two x4 PCI Express slots
Ethernet Four on-board Gigabit Ethernet Four on-board Gigabit Ethernet Four on-board Gigabit Ethernet
ports (10/100/1000Base-T) ports (10/100/1000Base-T) ports (10/100/1000Base-T)
Power supplies Dual redundant hot-swappable Dual redundant hot-swappable Four redundant hot-swappable
AC 650 W power supply units AC 1050 W power supply units AC 1133 W power supply units
(N+N redundancy) (N+N redundancy) (N+N redundancy)
Fans Redundant, hot-swappable fan Redundant, hot-swappable fan Redundant, hot-swappable fan
modules (N+N redundancy) modules (N+N redundancy) modules (N+N redundancy)
Form factor 1 rack unit (1U) 2 rack units (2U) 4 rack units (4U)
a.The RAID HBA consumes a single PCI Express slot
Features of the Sun Fire X4640 server include:

• Support for multiple Six-Core AMD Opteron processors
• Large memory capacities (up to 512 GB)
• Internal storage, with support for four internal SAS disk drives
• Internal hardware RAID support provided
• PCI-X and PCI Express expandability
• Four external and one internal USB 2.0 ports
• Built-in quad Gigabit Ethernet support, built-in VGA video port
• Integrated Lights Out Management (ILOM) service processor and firmware
• Support for multiple operating systems
Multiple off-the-shelf configurations are available, along with a wide spectrum of

options to tailor the system for specific workload requirements.
Enterprise-Class Capabilities
Ultimately, datacenter requirements go beyond mere performance and scalability.
Effective servers must provide considerable flexibility to serve a wide variety of needs in
an agile fashion. The Sun Fire X4640 server is designed for datacenter operation —
running multiple operating systems and providing significant reliability, availability,
and availability (RAS) features. Ultimately, Sun’s innovative chassis design across its x64
product line contributes directly to the datacenter utility of these powerful systems.
A Choice of Operating Systems

In order to provide maximum flexibility and investment protection, Sun Fire X4640
server supports a choice of operating systems, including:
• The Solaris Operating System
• The Linux operating system (64-bit Red Hat or SuSE Linux)
• Microsoft Windows
• VMware ESX Server
Table 2 lists supported OS releases supported by the Sun Fire X4640 server as of this
writing. Please see sun.com/x64 for the latest supported operating systems and
environments.
Table 2. Supported OS releases for the Sun Fire X4640 server
OS Release 32-bit or 64-bit Six-Core

Solaris 10 OS 64-bit Update 7 (5/09)
OpenSolaris™ 2009.06 32/64-bit Yes
Red Hat Enterprise Linux 4, Update 8 minimum 32/64-bit Yes (64-bit only)
Red Hat Enterprise Linux (RHEL) 5.3 64-bit Yes
SuSE Linux Enterprise Server 10, SP1 and SLES 11 64-bit Yes
Microsoft Windows Server 2003 Standard or Enterprise 32/64-bit Yes
Edition, SP1 or later
Microsoft Windows Server 2008 R2 Data Center, 64-bit Yes
Enterprise, and standard
VMware vSphere 4, ESX 4, and ESXI 4.0 64-bit Yes
• The Solaris™ Operating System

Distributed under a commercial and open source licensing model, the Solaris 10
OS offers many innovative technologies that change the equation for
organizations needing to reduce costs, minimize complexity, and help eliminate
risk. The Solaris 10 OS is optimized for Sun systems and is supported on over one
thousand third-party x86/x64 systems. In addition, the Solaris 10 OS is free for
download without requirement to purchase a support contract, providing an
economic advantage over other community-based operating system offerings.
Taking advantage of the Solaris 10 OS can bring added flexibility and power to the
enterprise. Supporting systems from laptops and single-board computers to
datacenter and cluster installations, the Solaris 10 OS serves applications ranging
from military command and control systems, to telecommunication switch gear,
to stock trading. The Solaris 10 OS also includes more than 180 applications from
the free and open source software (F/OSS) community, and thousands of others
are freely available for download over the Internet.
Provided on all Sun systems at no charge, the Solaris OS delivers performance,

security, scalability, and reliability advantages for scale-out computing
environments. Underlying technologies, such as a high-performance networking
stack, advanced file system, and modern memory model combine to optimize the
performance of hosted applications. A suite of security features previously only
found in Sun’s military-grade Trusted Solaris™ operating system are now included
to fortify the commercial enterprise.
The Solaris OS supports near linear scalability from 1 to 72 CPUs and
addressability of up to 264 bytes of memory, well beyond the physical memory
limits of even Sun’s largest server. In addition, by providing the ability to
automatically recover from hardware faults, the Solaris OS provides maximum
data and application availability. The Solaris 10 OS is free for download without
requirement to purchase a support contract, offering an economic advantage over
other community-based operating system offerings.
• Linux Environments
Sun offers and supports the leading Linux variants on Sun Fire x64 servers,
including Red Hat Enterprise Linux and Novell SUSE Linux Enterprise Server. As the
leader in enterprise services for UNIX®, Sun brings decades of expertise to Linux
environments. Sun support contracts for Linux provide all front-line support and
transparent access to back-line support from Red Hat and Novell.
Sun is one of the largest contributors to the open-source community. Areas of

contribution include OpenOffice.org, Mozilla, GNOME, and X.org. In addition, Sun
provides key software offerings for Linux, including:
– Lustre™ parallel file system

– Sun Ray™ Server Software
– StarOffice™ productivity suite
– Sun Java™ Desktop Powered Program
– Sun Studio, Sun Java Studio Creator, and NetBeans™ IDE software
– MySQL™ database
• Microsoft Windows Environments

Organizations are constantly seeking to reduce the variety of platforms in the
datacenter, even when a wide range of workloads are present. To address this
need, the Sun Fire X4640 server is ideal for Microsoft SQL and Microsoft Windows
consolidation and virtualization efforts. This server has passed stringent Microsoft
compatibility test suites, achieving the Designed for Windows certification and a
listing in Microsoft Windows catalogs. Support contracts for Microsoft Windows
are also available from Sun. This certification and support demonstrates Sun’s
commitment to provide the best platforms to run not only the Solaris OS and
Linux, but Microsoft Windows as well.
• VMware
Ground-breaking virtualization solutions from VMware help improve asset
utilization, operational efficiency, and business agility. Sun offers the VMware
software on Sun hardware systems with full support from Sun. VMware
virtualization technology also combines with key Solaris 10 OS features such as
DTrace, Solaris Containers, and Solaris Predictive Self Healing software. As a
result, organizations can create breakthrough approaches to virtualization. In
fact, utilizing VMware virtual infrastructure software with the Solaris 10 OS for
consolidation projects can increase system utilization by up to ten times.
Effective and Consistent System Design for the Datacenter

Beyond the capabilities of individual systems, Sun understands that datacenters have
unique and pressing needs that require attention on the part of system designers.
Density, performance, and scalability are all essential considerations, but systems must
also fit in with modern datacenter strategies that consider power, cooling, and
serviceability. The Sun Fire X4640 server shares an innovative design philosophy that
extends across Sun’s volume x86/x64 and SPARC® server platforms.
Principals of this design philosophy include:
• Common chassis design — Shared chassis design leverages key system innovations
across multiple architectures, provides for common components and subassemblies,
and greatly simplifies administration for those deploying multiple processor
architectures.
• Maximum compute density — Sun’s volume servers provide leading density in terms
of CPU cores, memory, storage and I/O. This focus on density often lets Sun’s smaller
rackmount servers replace larger rackmount servers, for a considerable space
savings.
• Common shared management — The Sun Fire X4640 server is designed for ease of
management and serviceability with service processors in common with other Sun
volume server platforms. Systems and components are designed for easy
identification and hot-swap components facilitate on-line replacement.
• Continued investment protection — Sun designs for maximum investment

protection. Even with breakthrough x64 technology, Sun’s Solaris Binary
Compatibility Guarantee means that applications simply run without modification.
Chassis Design Innovations

Sun volume server platforms feature basic chassis designs that are shared with other
Sun x64 and SPARC server platforms. This approach not only provides a consistent look
and feel across the product line, but it simplifies administration through consistent
component placement and shared components. Beyond mere consistency, this
approach provides a datacenter design focus that places key technology where it can
make a difference for the operations.
• Enhanced System and Component Serviceability
Finding and identifying servers and components in a modern datacenter can be a
challenge. The Sun Fire X4640 server is optimized for lights-out datacenter
configurations with easy-to-identify servers and modules. Color-coded operator
panels provide straightforward diagnostics. Systems are designed for deployment
in hot-isle / cold-isle multiracked deployments with both front and rear diagnostic
LEDs and Fault Remind features to help identify faulty or failed components.
Consistent connector layouts for power, networking, and management make

moving between Sun’s systems straightforward. All hot-plug components are tool-
less and easily available for serviceability. For instance, Sun Fire X4640 servers
provides direct access to fan modules so that fans can be serviced without
exposing sensitive components, or causing unnecessary downtime.
• Robust Chassis, Component, and Subassembly Design
Sun’s volume servers share chassis that are carefully designed to provide
reliability and cool operation. In spite of their computational, I/O, and storage
density, Sun’s servers are able to maintain adequate cooling using conventional
technologies. Efficient modular fan assemblies keep the chassis within an effective
operating temperature range.
• Minimized Cabling for Maximized Airflow
To minimize cabling and increase reliability, Sun Fire X4640 servers are designed
with a minimum of cabling. This approach not only provides more reliable
connectors, but it maximizes airflow through the chassis for more effective
cooling.
• Leading Reliability, Availability, and Serviceability (RAS)

The Sun Fire X4640 server provides excellent reliability, availability, and
serviceability (RAS) characteristics. Highly-reliable parts and a relatively low total
component count minimize the opportunity for system errors.
The following key design elements help increase the dependability of IT services:
– Reduced parts count

– Built-in RAID capabilities
– Redundancy and hot-swap components
– Parity protection and error correction capabilities
– Integrated Lights Out Management (ILOM) service processor
– Superior energy efficiency
– Robust virtualization technology
– Comprehensive fault management
11 AMD Opteron™ Processor Technology Sun Microsystems, Inc.
Chapter 2
AMD Opteron™ Processor Technology
The Sun Fire X4640 server is powered by the latest AMD Opteron processors, utilizing
AMD’s Direct Connect Architecture and NVIDIA chipsets for scalability and fast I/O
throughput. The server supports Forth-Generation Six-Core AMD Opteron processors.
The sections that follow describe the architecture and feature set of supported AMD
Opteron processors.
Innovative Processor Technology

The AMD Opteron processor extends the ubiquitous x86 architecture to accommodate
64-bit processing. Formerly known as x86-64, AMD’s enhancements to the x86
architecture allow seamless migration to the superior performance of 64-bit
technology. These processors offer a common core architecture that is consistent across
multisocket systems, and is also consistent with previous AMD Opteron processors. This
strategy helps organizations minimize the cost of transitions while they maximize past
investments in software and hardware optimization.
AMD Opteron processors are offered in three series as shown below. Sun Fire X4640
servers employ 8000 Series AMD Opteron processors.
• 1000 Series — Single socket
• 2000 Series — Up to two sockets (one coherent HyperTransport 3 link)
• 8000 Series — Four to Eight sockets (three coherent HyperTransport 3 links)
AMD Opteron processors offer considerable advantages, including:

• AMD64 technology — AMD64 technology lets 64-bit operating systems provide full,
transparent, and simultaneous 32-bit and 64-bit platform application multitasking.
This approach lets systems run the existing installed-base of 32-bit applications and
operating systems at peak performance, while providing a 64-bit migration path.
• Direct Connect Architecture — AMD’s Direct Connect Architecture helps to reduce the
very real challenges and bottlenecks of system architecture.
– Memory is directly connected to the processor, optimizing memory performance
– I/O is directly connected to the processor, for more balanced throughput and I/O
– Processors are directly connected to other processors, allowing for more linear
symmetrical multiprocessing
• Integrated DDR2 memory controller — A 144-bit wide, on-chip DDR2 memory
controller provides 128 bits for data and 16 bits for ECC and Enhanced ECC
technologies, while providing low-latency memory bandwidth that scales as
processors are added.
• AMD HyperTransport technology — AMD HyperTransport Technology provides a
scalable bandwidth interconnect between processors, I/O subsystems, and other
chipsets.
• AMD Virtualization (AMD-V) — AMD Virtualization reduces overhead by selectively

intercepting instructions destined for guest environments while the Direct Connect
Architecture helps guest operating systems run at near native speed. A virtualization-
aware integrated memory controller provides efficient isolation of virtual machine
memory.
• Enhanced performance per watt — Energy-efficient DDR2 memory uses up to 30%
less power than DDR1 memory, and up to 58% less power than FB-DIMM memory. In
addition, AMD PowerNow! technology with Optimized Power Management can
deliver performance on demand, while minimizing power consumption.
Fourth-Generation Six-Core AMD Opteron Processors

Using proven 45 nm technology, Six-Core AMD Opteron processors provide a higher core
count, improved cache indexing, and coherent HyperTransport 3 support — all within
the same thermal envelopes and physical socket (Socket F (1207)) as previous-
generation AMD Opteron processors. A block-level diagram of the Six-Core AMD Opteron
processor is shown in Figure 2.
Forth-Generation Six-Core AMD Opteron
Core 1 Core 2 Core 3 Core 4 Core 5 Core 6
512 KB
128 128 KB 128 KB 128 KB 128 KB 128 KB
L2 Cache
L1 L1 Cache L1 Cache L1 Cache L1 Cache L1 Cache
512 KB 512 KB 512 KB 512 KB 512 KB 512 KB
L2 Cache L2 Cache L2 Cache L2 Cache L2 Cache L2 Cache
6 MB L3 Cache
System Request Interface
Crossbar Switch
DDR2 HyperTransport 0 HyperTransport 2

Memory
Controller HyperTransport 1
Figure 2. Support for six cores increases computational throughput in the same
thermal envelope as previous-generation AMD Opteron processors.
The Six-Core AMD Opteron processor offers a number of key advantages, including:
• Six cores, each with 128 KB of Level-1 cache and 512 KB of Level-2 cache
• 6 MB Level-3 cache shared by all processor cores
• HyperTransport 3 support for up to 4.8 GT/sec. (19.2 GB/sec.) per link
• HyperTransport Assist (HT Assist) to help improve overall system performance in
multisocket systems
• Dual Dynamic Power Management support
• AMD-V and AMD-P support
Fourth-Generation Six-Core AMD Opteron processors are labeled as 24xx and 84xx CPUs
distinguishing them from earlier Third-Generation Quad-Core AMD Opteron processors
(23xx and 83xx CPUs).
Coherent HyperTransport 3 (cHT3)

HyperTransport interconnect technology is a high-performance and fully-differential
point-to-point link with low latency. In addition to HyperTransport 1, the enhanced Six-
Core AMD Opteron processor supports Coherent HyperTransport 3 (cHT3).
The benefits of this upgraded HyperTransport technology include the following:

• Support for higher processor frequencies (2 GHz)
• Higher maximum aggregate bandwidth (16 GB/sec)
• Better reliability through features that can minimize crosstalk and electromagnetic
interference (EMI)
• Greater packet transfer efficiency through an enhanced error retry algorithm
Six-Core AMD Opteron processors provide Coherent HyperTransport 3 support on

processor-to-processor links, with processor-to-I/O links supported via HyperTransport 1.
A hypothetical four-socket implementation is depicted in Figure 3.
PCIe
Bridge
Coherent HyperTransport 1
HyperTransport 3 2.0 GT/sec.
4.8 GT/sec.
PCIe
Bridge
Figure 3. Coherent HyperTransport 3 (cHT3) can be provided between processors,

with HyperTransport 1 interconnects to I/O bridges.
Note – Six-Core AMD Opteron processors can only be installed on systems with a motherboard
that supports HyperTransport 3, excluding earlier models of the Sun Fire X4600 M2 server. Ear-
lier Sun Fire X4600 M2 servers cannot be upgraded to support coherent HyperTransport 3. Sys-
tem CPU/Memory modules are specially keyed so that they may not be inserted in to
incompatible systems.
HyperTransport Assist (HT Assist)

In previous-generation architectures, a given core had to search each cache to find the
latest version of a piece of data. This process often resulted in latency and extra probe
filter traffic. HT Assist provides indices in the cache on each processor so that the core
can simply look at the cache registry to find the location of the data. A lookup that
might have required as many as 10 HyperTransport transactions without HT Assist can
be reduced to just two with HT Assist (Figure 4).
CPU 1 CPU 2 CPU 1 CPU 2
CPU 3 CPU 4 CPU 3 CPU 4
Without HT Assist With HT Assist

(10 Transactions) (2 Transactions)
Figure 4. HT Assist sets up indices in the cache on each processor, greatly reducing
probe filter traffic.
HT Assist saves time, reduces probe filter traffic, and can increase performance, by:
• Reducing latency from inter-processor communication
• Increasing STREAM memory bandwidth by as much as 60% (application dependent)
• Providing significant improvements in HPC, database, and virtualization workloads
• Increasing performance for cache-sensitive applications on systems four-socket
systems
AMD-P Power Management Technology

Six-Core AMD Opteron processors continue to provide industry-leading power
management technology, including:
• Enhanced AMD PowerNow! Technology with Independent Dynamic Core Technology
allows each core to vary its frequency, based on the specific needs of the application.
This ability allows for more precise power management to reduce data center energy
consumption and thereby reduce total cost of ownership (TCO).
• Dual Dynamic Power Management allows each processor to maximize the power-
saving benefits of Enhanced AMD PowerNow! technology without compromising
performance. Dual Dynamic Power Management can reduce idle power consumption
and allow for per-processor power management in multi-socket systems to decrease
power consumption.
• AMD CoolCore Technology evaluates which parts of the die — the cores, the memory,
or both — are needed to support currently running applications. AMD CoolCore
Technology can cut power to unused transistor areas to reduce power consumption
and lower heat generation.
• AMD Smart Fetch Technology lets inactive cores write contents of their L1 and L2
caches to the shared L3 cache. This feature can allow the inactive cores to enter a
“halt” state and draw less power, reducing CPU power consumption while still
making the contents of their caches available to other active cores.
AMD-V Virtualization Technology

Like their predecessors, Six-Core AMD Opteron processors also employ a full
complement of AMD-V virtualization technology, including:
• Rapid Virtualization Indexing and Tagged-TLB were developed to allow the translation
of virtual to physical memory addresses in hardware instead of software. This
capability helps decrease latency, improve performance, and increase the number of
virtual machines running on each physical machine — allowing for a higher
performing, more flexible IT environment.
• Extended Migration allows movement of a virtual machine between two physical
machines running AMD Opteron processors on many virtualization software
solutions. AMD Extended Migration is backward compatible with Single-, Dual-, and
Quad-Core AMD Opteron processors.
16 Server Architecture Sun Microsystems, Inc.
Chapter 3
Server Architecture
The Sun Fire X4640 server is designed to provide multisocket scalability and high
performance while offering high reliability and low power consumption. The sections
that follow detail physical and architectural aspects of the system.
System-Level Architecture
Figure 5 provides a high-level block diagram of the Sun Fire X4640 server motherboard,
illustrating a relatively simple and elegant design. The server supports up to eight Six-
Core AMD Opteron processors on CPU/Memory modules that insert directly into the
motherboard. Eight on-board DIMM slots are provided on each CPU/Memory module,
and are directly connected to the processor. CPU/Memory Modules are added in pairs,
and are interconnected by dedicated Coherent HyperTransport links (4.8 GT/sec.)
System configurations with two, four, six, and eight processors are supported. All
processors in a system must be the same type (six-core, same stepping, and the same
frequency).
CPU Module G
CPU Module B
CPU Module C
CPU Module D
CPU Module H
CPU Module A
CPU Module E
CPU Module F
CPU 0 CPU 2 CPU 1 CPU 3 CPU 4 CPU 5 CPU 6 CPU 7
Coherent HT3 links - 4.8 GT/sec

Non-Coherent HT1 links - 2.0 GT/sec
Internal USB
+ +
USB USB
USB
PCI-X USB
+
8132 + CK8-04 IO-04
Super
SATA I/O
Shared
100 MHz Bus
32 MB
Flash
+
LSI
+ DVI Video
over LAN
SAS 1064 USB Video Redirect MPC8248
Hub Rage XL SP
+ +
64 MB
DDR
SDRAM
BCM
5221 Managed
ILOM Power Supplies
PCI-X 100 MHz
PCI-X 100 MHz
PCI-E 8-Lane
PCI-E 8-Lane
PCI-E 4-Lane
PCI-E 8-Lane
PCI-E 8-Lane
PCI-E 4-Lane
SAS 4 x 1 GB Rear Front SATA VGA Management Serial

HDDs Ethernet USB USB CD/DVD Video Ethernet
Figure 5. Sun Fire X4640 server motherboard block-level diagram

The CPU/Memory modules are connected via non-coherent HyperTransport links to a

set of components that provide I/O for the system, including:
• A tunneling HyperTransport bridge (AMD-8132 HyperTransport PCI-X Tunnel)
• A HyperTransport-attached Southbridge component (NVIDIA nForce4 Professional
2200/CK8-04)
• A HyperTransport-attached Southbridge component (NVIDIA nForce4 Professional
2050/IO-04)
AMD Direct Connect Architecture Implementation with

HyperTransport Links
AMD’s Direct Connect Architecture provides system designers with a powerful tool for
building tightly-coupled systems with large numbers of processors. Because processors
are directly connected to memory, I/O, and each other via HyperTransport links,
multisocket systems can be designed with low latency and without extra bus and chip
overhead. The Sun Fire X4640 server leverages Coherent HyperTransport links and the
AMD Direct Connect Architecture to create a design that scales easily from two to eight
sockets.
The design of a HyperTransport topology for larger numbers of processors must take
hop distance into account. In particular, average hop distance increases when building
eight-socket systems when compared to that of four-socket systems. Minimizing the
number of hops is ideal, and Sun Fire X4640 servers accomplish minimum hop distance
by enhancing the simple ladder topology as shown in Figure 6.
Ladder Twisted Ladder Enhanced Twisted Ladder
Figure 6. Enhanced Twisted Ladder topologies yield reduced hop counts for larger
numbers of processors
Sun’s approach using Coherent HyperTransport technology minimizes hop count and
latency between processors. Since memory is controlled by each processor, latency for
remote memory accesses is also reduced. Figure 7 illustrates the hop counts for two-
socket, four-socket, six-socket, and eight-socket Sun Fire X4640 server configurations.
Though many choose to divide the resources of Sun Fire X4640 servers using
virtualization technology, workloads with higher CPU components and lower memory
latency/bandwidth needs scale best beyond four sockets.
2-Socket 4-Socket 6-Socket 8-Socket
CPU 0 1 CPU 0 1 2 3 CPU 0 1 2 3 4 5 CPU 0 1 2 3 4 5 6 7
0 0 1 0 0 1 1 2 0 0 1 1 2 2 3 0 0 1 1 2 2 2 2 3
1 1 0 1 1 0 1 1 1 1 0 2 1 1 2 1 1 0 2 1 2 2 1 2
2 1 1 0 1 2 1 2 0 1 1 2 2 1 2 0 2 1 1 2 2
A, B
3 2 1 1 0 3 2 1 1 0 2 1 3 2 1 2 0 1 1 2 2
4 2 1 1 2 0 1 4 2 2 1 1 0 2 1 2
A, D, E, H
5 3 2 2 1 1 0 5 2 2 1 1 2 0 2 1
6 2 1 2 2 1 2 0 2
A, B, D, E, F, H
7 3 2 2 2 2 1 1 0
A, B, C, D, E, F, G, H
Figure 7. Hops between processors in Sun Fire X4640 servers
Connectivity between CPU/Memory modules changes depending on the actual number

of modules installed. Figure 8 illustrates the various connectivity scenarios.
CPU CPU CPU CPU

Module B Module F Module F Module H
CPU CPU CPU CPU

Module D Module E Module D Module E
CPU CPU CPU CPU

Module C Module G Module A Module B
CPU CPU
Module A Module H + +
PCI-X
+
8132 + IO-04
+ +
PCI-X
+
8132 + IO-04
CK8-04P
CK8-04P
8 CPU Modules 6 CPU Modules
CPU CPU CPU

Module E Module H Module B
CPU CPU CPU

Module A Module D Module A
+ + + +
PCI-X PCI-X
+
8132 + IO-04 +
8132 +
CK8-04P CK8-04P
4 CPU Modules 2 CPU Modules
Figure 8. Hypertransport topology in Sun Fire X4640 servers for 2-processor,

4-processor, 6-processor, and 8-processor configurations
I/O Subsystem
The Sun Fire X4640 I/O subsystem is connected to the CPU/Memory complex through a
HyperTransport link from processors A and H. The I/O bridges are implemented with the
NVIDIA NForce4 Professional 2200 (CK8-04) and the NVIDIA nForce4 Professional 2050
(IO-04) media and communications processors (MCPs). Other I/O components connect
to these three main chipsets, including:
• A quad-channel SAS RAID controller (LSI SAS 1064) attached to one of the AMD-8132
PCI-X tunnels on a shared bus with the PCI-X slot 0.
• Two 10/100/1000 dual Gigabit Ethernet MAC/PHY devices (Intel FW82546GB NIC)
• An ATI Rage XL video controller device attached to the PCI bus of the NVIDIA nForce
Professional 2200
• A Super I/O device connecting the Integrated Lights Out Management (ILOM) service
processor to the NVIDIA nForce Professional 2200 chip.
AMD-8132 HyperTransport PCI-X Tunnel

The AMD-8132 HyperTransport PCI-X 2.0 tunnel is a high-speed device that provides two
independent, high-performance PCI-X bus bridges, integrated with a high-speed
HyperTransport technology tunnel. The AMD-8132 component brings high-performance
PCI-X 2.0 connectivity to AMD Opteron processor-based systems with increased
throughput, improved RAS capabilities, robust data management, and enhanced
HyperTransport technology connectivity.
As shown in Figure 9, the front-end HyperTransport interface (Side A) provides a

configurable 16-bit wide communication path to the host, offering up to 8 GB/sec. of
aggregate bandwidth. Similarly, the back-end HyperTransport interface (Side B)
provides a configurable 16-bit wide communication path to a downstream device,
offering up to 8 GB/sec. of aggregate bandwidth.
The AMD-8132 HyperTransport PCI-X Tunnel provides high-speed PCI-X capability and
offers the following features:
• 16-bit HyperTransport interfaces (Side A and B) offering a maximum aggregate
bandwidth up to 8 GB/sec. on each side
• Two PCI-X bridges (A and B), each of which supports a 64-bit data bus
• Each bridge also supports Mode 1 PCI-X, conventional PCI protocol, and Mode 2
operation
• Each bridge supports independent transfer rates and operational modes:
– In PCI-X mode, the bridge supports transfer rates of 133, 100, 66, and 50 MHz
– In PCI mode, the bridge supports transfer rates of 66, 50, 33, and 25 MHz
AMD-8132 Device
Side A Side B
HyperTransport Link HyperTransport Link Downstream
Host Tunnel
16 bits upstream 16 bits upstream Device
16 bits downstream 16 bits downstream
PCI-X PCI-X
Bridge A Bridge B
Slots and Devices
Figure 9. AMD-8132 HyperTransport PCI-X tunnel architecture block diagram
NVIDIA nForce Professional Media and Communications Processors (MCPs)

Both MCPs provide I/O connectivity to a CPU via a 1 GHz 8 GB/sec. HyperTransport
connection. Each MCP provides 20 lanes of PCI Express I/O divided into 3 links (two x8
links and one x4 link) for a total of 40 lanes of PCI Express I/O bandwidth. These links
are all directly wired to PCI Express expansion slots exposed through the rear of the
chassis. The CK8-04 bridge also implements the southbridge functionality including
multiple USB2.0 ports, a serial port, and an IDE bus.
The NVIDIA nForce Professional 2200 (CK8-04) supports:

• Three PCI Express ports (two x8 and one x4)
• Independent IDE connection to the DVD/CD-RW player
• LPC connection to the Super I/O chip
• Six USB 2.0 ports (three for external use and three for the ILOM service processor)
• A PCI connection to the ATI Rage XL video controller
The NVIDIA nForce Professional 2050 (IO-04) supports three PCI Express ports (two x8
and one x4).
LSI SAS1064 SAS Controller

The Sun Fire X4640 server includes an integrated LSI SAS1064 4-port 3-GB/sec. SAS HW
RAID disk controller. The LSI SAS1064 is a versatile controller that provides 4 SAS ports
capable of 3 GB/second data transfers for each PHYS, for a total maximum bandwidth
of 12 GB/second. SAS features of the LSI SAS1064 include:
• Four fully-independent PHYS
• Support for 3 GB/second SAS data transfers for each PHYS
• High-performance, serial, point-to-point, enterprise-level storage interface
• Simplified cabling between devices
• Data transfers using SCSI information units
• 133 MHz 64-bit PCI-X interface (shared bus with PCI-X 100 MHz slots)
• Integrated RAID0 and RAID1 solutions provide Integrated Mirroring technology and
Integrated Striping technology
• Up to four hot-swappable 2.5-inch SAS disk drives are supported.
Intel 82546GB Dual-Port Gigabit Ethernet Controllers

The Sun Fire X4640 server is equipped with two Intel 82546GB Dual Port Gigabit
Ethernet controllers that provide quad Gigabit Ethernet interfaces to the server
platform. The Intel 82546GB Dual Port Gigabit Ethernet controller provides two 64-bit
fully-integrated Gigabit Ethernet Media Access Control (MAC) and physical layer (PHY)
functions. The controller is capable of transmitting and receiving data at
10/100/1000 Mb/second data rates with half or full duplex capabilities. Additional
features of the Intel 82546GB Dual Port Gigabit Ethernet controller include:
• 802.3ab PHY compliance and compatibility (CAT-5 use)
• 802.3ab auto-negotiation
• 802.3x full-duplex flow control
• 802.9q VLAN tag insertion, stripping, and packet filtering
• Preboot eXecution Environment (PXE) Flash Interface support (32- and 64-bit)
ATI Rage XL graphics Controller

The Sun Fire X4640 server employs an ATI Rage XL graphics controller to deliver 2D and
3D graphics acceleration. The DVI port is connected to the ILOM service processor for
remote graphics redirection. The ATI Rage XL has the following features:
• 64-bit 125 MHz memory clock
• 8 MB external memory
• 64-bit SDR (SDRAM/SGRAM) memory path
• 64-bit AGP/PCI bus
• Integrated TMDS: DVI, DFP, and VESA P&D interface
• Support for 24-bit TTL
• 1600 x 1200 maximum resolution
• 16.7 M maximum color depth
• 1024 x 768 resolution
ILOM Daughter Card

Each Sun Fire X4640 server provides an Integrated Lights Out Management (ILOM)
service processor located on a daughter card. This service processor is similar to those
used in other Sun x86/x64 and SPARC servers. ILOM provides multiple in-band and out-
of-band management solutions. IPMI and SNMP OS-resident agents provide in-band
management through the host operating system or platform. Out-of-band
management is provided through the serial port or dedicated Ethernet port. A
command line interface (CLI) is provided for out-of-band serial port connectivity. The
dedicated Ethernet connectivity to out-of-band management features include:
• Web interface
• CLI via SSH
• IPMI 2.0
• SNMP v1, v2c, and v3
The ILOM daughter card is a standalone system with no dependencies on the operating
system. Physical interfaces are provided through the connections listed above. The
actual connected devices, however, are located on the main system board. More
information on ILOM capabilities is provided in Chapter 4. ILOM daughter card
connections include:
• DVI output port connected to the ATI Rage XL via FPGA to support remote graphics
redirection from the ILOM service with a maximum VGA resolution of 1024 x 768
• Three USB connections to the NVIDIA nForce Professional 2200 to provide remote
keyboard, mouse, and storage functionality
• LPC connection to the NVIDIA nForce Professional 2200 via an FPGA
• 10/100 MB Ethernet connected to the on-board Broadcom BCM5221 10/100 Ethernet
controller, which automatically detects straight or cross-over Ethernet cables
• External serial port multiplexed with the main system serial port for access to the
ILOM CLI and system console using serial port redirection (S-o-L, Serial-over-LAN)
CPU/Memory Modules
The Sun Fire X4640 server supports up to eight Six-Core AMD Opteron Series 8000
processors — each provided on a CPU/Memory module with associated slots for
memory DIMMs. CPU/Memory modules are also known as "books", and each contains:
• AMD Opteron CPU and heatsink
• 8 DDR2 memory DIMM slots
• CPU/Memory Voltage Regulator Module
• Airflow redirection baffles for greatest cooling efficiency
• A Fault Remind button and per-DIMM LEDs to help identify a failed component.
The CPU/Memory module (Figure 10) is connected to the system motherboard using a
pair of ganged connectors. Each processor receives core voltage input from an
individual voltage regulator module (VRM). The VRM regulates the system current and
voltage according to the VID code output by the CPU package. The VRM is a DC-to-DC
point-of-load convertor specified for +12 Volt input and programmable 100 Amp output.
Sun Fire X4640 CPU/Memory modules also support split-plane operation for support of
AMD Dual Dynamic Power Management.
Sun Fire X4640 CPU/Memory modules are not compatible with previous-generation Sun
Fire X4600 M2 servers. Likewise, the previous-generation CPU/Memory modules made
for Sun Fire X4600 M2 servers are not compatible with the Sun Fire X4640 server. CPU/
Memory modules are physically keyed to prevent accidental deployment in non-
compatible systems.
Figure 10. The Sun Fire X4640 server CPU/Memory Module (book) provides support
for a Six-Core AMD Opteron processor and up to eight DDR2 DIMMs.
Memory Subsystem
Each AMD Opteron processor includes a low-latency, high-bandwidth, integrated
memory controller that reduces latencies during memory access over traditional front-
side bus-based memory controllers, and each supports up to eight ECC Registered
DDR2/667 memory modules. Although the AMD Opteron processor's memory controller
works in 64-bit or 128-bit mode ECC operation, for best performance results, it is
recommended to run 128-bit ECC operation mode. To run in 128-bit mode, DIMMs are
populated in pairs such that they occupy one-half of the AMD Opteron processor's 128-
bit controller interface. The controller supports 1 bit per byte ECC, and the Sun Fire
X4640 server uses DDR2/667 (PC2-5300) registered DDR SDRAM modules.
Eight DDR DIMM slots are provided per processor, and they are color-coded white and
black to indicate population order (white = 0,1 and black = 2,3; white pair first). LED
fault indicators controlled by the ILOM Service Processor provide the ability to easily
identify failed DIMM modules. Memory capacity scales with the number of processors.
As a result, a dual-socket system can support a maximum of sixteen DIMMs (up to
128 GB with 8 GB DIMMs). A fully populated, eight-processor server supports a
maximum of 64 DIMMS or up to 512 GB of memory with 8 GB DIMMs.
Other features of the Sun Fire X4640 server memory architecture include:
• Dedicated on-die 128-bit wide DDR memory controller
• Memory bandwidth up to 10.7 GB/sec. @ DDR2/667
• Under 80 ns direct memory latency
• Registered ECC DDR2/667 (PC2-5300) DIMMs supported
• Up to 32 GB per CPU with 4 GB DIMMs in 8 DIMM slots per CPU
A Broad Range of Configuration Options

The Sun Fire X4640 server can support two, four, six, or eight processors through
HyperTransport3 links. Table 3 illustrates the CPU population rules and numbering.
Systems with two processors simply deploy two CPU/Memory modules, leaving the rest
of the slots empty. Systems deploying four and six CPU/Memory modules require filler
modules that bridge the HyperTransport connections and help direct air-flow through
the chassis. In eight-processor systems, all of the slots are populated.
Table 3. CPU module population and numbering
Front of the Server

Slot A Slot B Slot C Slot D Slot E Slot F Slot G Slot H
8 Processors Module 0 Module 2 Module 1 Module 3 Module 4 Module 5 Module 6 Module 7
6 Processors Module 0 Module 1 Filler Module 2 Module 3 Module 4 Filler Module 5
4 Processors Module 0 Filler Filler Module 1 Module 2 Filler Filler Module 3
2 Processors Module 0 Module 1 Empty Empty Empty Empty Empty Empty
Rear of the Server
Sun Fire X4640 Server Overview

The Sun Fire X4640 server provides considerable computational power and scalability in
a space-efficient 4U rackmount package. By closely coupling up to eight Six-Core AMD
Opteron processors, these systems extend the ubiquitous x64 architecture for mission-
critical applications and OLTP database workloads as well as high performance
computing and consolidation. The server is designed to address the challenges of
modern datacenters with reduced power consumption and a small physical footprint.
Depending on the model selected, the Sun Fire x4640 server features two, four, six, or
eight Six-Core AMD Opteron processors and up to 512 GB of memory.
System Motherboard and Chassis Perspective

Figure 11 provides a top-down perspective of the Sun Fire X4640 server with the top
cover removed and a full complement of CPU/Memory modules installed. For some
four-socket and six-socket configurations, filler modules are installed in empty slots to
help ensure proper airflow and front-to-back cooling. All eight PCI Express slots are low
profile, and are wired to either x8 or x4 PCI Express interfaces.
ILOM Daughter Card
CPU and Memory Modules
Hot-Swap Fan Modules
Figure 11. Sun Fire X4640 server top view
Four system fans insert from the top of the chassis, and four power supplies insert from
the rear of the chassis. The four power supplies provide N+N redundancy. The system
can continue to operate at full capacity with any combination of two of the four power
supplies.1
Enclosure
The 4U Sun Fire X4640 server enclosure is designed for use in a standard 19-inch rack
(Table 4).
Table 4. Dimensions and weight of the Sun Fire X4640 server
Dimension U.S. International

Height 6.9 inches (4 RU) 176 millimeters
Width 17.5 inches 445 millimeters
Depth 24.75 inches 629 millimeters
Maximum standalone weight 88 pounds 40 kilograms
(without rackmount kit and cable
management arm)
1.Some restrictions apply when apply when all 64 DIMM slots are populated with 137 Watt CPUs.
The Sun Fire X4640 server includes the following major components:
• Support for two, four, six, or eight CPU/Memory modules, with each module
containing one Six-Core AMD Opteron 8000 series processor
• Eight DIMM slots per CPU/Memory module, supporting Up to 512 GB of memory with
Registered ECC DDR2/667 (PC2-5300) DIMMs
• Four drive bays for SAS disk drives and additional DVD-ROM
• Four on-board 10/100/1000Base-T Ethernet ports
• Five USB 2.0 ports (two forward, two rear facing, one internal)
• Six PCI Express expansion slots (four x8, and two x4)
• Two 64-bit PCI-X slots @ 100 MHz
• ILOM service processor daughter card with 10/100Base-T Ethernet port
• 19-inch rackmount kit
• Cable management arm (optional)
• Four 1133 Watt AC power supplies (hot-swappable in N+N redundant configuration)
with integral fans
• Four (N+1 redundant) hot-swappable cooling fans under environmental monitoring
and control, accessed directly from the top of the chassis
Front and Rear Perspectives

Figure 12 illustrates the front and rear panels of the Sun Fire X4640 server.
System status indicators

Locate Button/LED
CD/DVD drive
USB 2.0 Ports Hard Disk Drives
Serial and network

Management ports
VGA port
Redundant (N+N)
power supply units
PCI-X slots USB ports

PCI Express slots 10/100/1000BaseT PCI Express slots
Ethernet ports
Figure 12. Sun Fire X4640 server front and rear panel perspectives
External features of the Sun Fire X4640 server include:

• Front and rear system and component status indicator lights provide locator (white),
service required (amber), and activity status (green) for the system.
• Four hot-plug SAS disk drives insert through the front panel of the system.
• One slimline, slot-accessible DVD/CD-RW drive is accessed through the front panel.
• Five USB 2.0 ports are provided, two on the front panel, two on the rear, and one
internal.
• Four hot-plug/hot-swap (N+N) power supplies with integral fans insert from the rear.
• Rear power-supply indicator lights convey the status of each power supply.
• A single AC plug is provided on each hot-plug/hot-swap power supply.
• Four 10/100/1000BaseT autosensing Ethernet ports are provided.
• A 15-pin dinn connector is provided for VGA port.
• A total of six PCI Express card slots are provided, along with two PCI-X slots.
• Two management ports are provided for use with the ILOM system controller. The
RJ-45 serial management port provides the default connection to the ILOM controller.
Cooling and Power

The Sun Fire X4640 server is designed with stringent datacenter power and cooling
requirements in mind.
Cooling
Because heat is so destructive to electronic components, lowering operating
temperature can substantially increase reliability. Sun's cooling design enables the
CPUs to operate approximately 15 percent more efficiently than in some competitive
systems.
PSU Airflow
Chassis Divider
Motherboard
Airflow
Motherboard
Airflow
Figure 13. Sun Fire X4640 servers provide front-to-back airflow with airflow
segregated between power supplies and the main chassis
As shown, the Sun Fire X4640 server is partitioned by a chassis divider that segregates
airflow into two distinct chambers:
• The motherboard/PCI and fan chamber, and
• The Power Supply Unit (PSU) and storage chamber
Both areas are air cooled front-to-back in concert with typical modern datacenter
practice. The PSU and storage chamber is cooled by individual fans on the back of each
power supply. Air is drawn through the front of the server by these rear-located fans.
The motherboard/PCI and fan chamber is cooled by redundant rows of fans mounted in
front of the server behind the bezel (Figure 14). Fans are hot swappable, and each fan
has an LED fault indicator. Fans are accessible from the top of the server so that they
can be accessed without interrupting system operation.
Figure 14. Four N+1 redundant fans are accessed through the top of the Sun Fire
X4640 server.
Power
The Sun Fire X4640 server is powered by dual redundant hot-swappable power supply
units (PSUs) with integral fans. The server requires two PSUs, but up to four PSUs can
be used per system for redundancy. For optimal redundancy, separate circuits can be
used to eliminate the dependence on a single electrical circuit, and to provide for
consistent N+N redundancy with the PSUs themselves. Power supplies are monitored
and managed by the ILOM service processor. PSU fans are not individually managed by
ILOM, but are instead considered part of the PSU unit. As a result, the failure of a PSU
fan is reported by ILOM as a failure of the entire PSU.
30 Enterprise-Class Operating System and Management Software Sun Microsystems, Inc.
Chapter 4
Enterprise-Class Operating System and
Management Software
Unlike many x86/x64 systems, Sun Fire X4640 server is ideally suited for enterprise
environments. Not only are the systems designed with the datacenter in mind, but the
operating system and management software provided with these systems allow them
to serve the most important and mission-critical applications. The sections that follow
describe the Solaris 10 OS and key enterprise-grade management technology.
Solaris 10 OS Support
Among the available operating systems, the Solaris OS is ideal for large-scale enterprise
deployments. Supported on all of Sun’s x86/x64 and SPARC platforms, the Solaris OS
has specific features that can enhance flexibility and performance — as described in
the sections that follow.
• Solaris Containers for Consolidation, Secure Partitioning, and Virtualization
Solaris Containers comprise a group of technologies that work together to
efficiently manage system resources, virtualize the system, and provide a
complete, isolated, and secure runtime environment for applications. Solaris
Containers can be used to partition and allocate the considerable computational
resources of Sun Fire X4640 servers. Solaris containers are comprised of Solaris
Zones and Solaris Resource Management which work together with the Solaris
fair-share scheduler.
– Solaris Zones — Solaris Zones can be used to create an isolated and secure envi-
ronment for running applications. A zone is a virtualized operating system envi-
ronment created within a single instance of the Solaris OS. Zones can be used to
isolate applications and processes from the rest of the system. This isolation
helps enhance security and reliability since processes in one zone are prevented
from interfering with processes running in another zone.
– Resource Management — Resource management tools provided with the
Solaris OS lets administrators dedicate resources such as CPU cycles to specific
applications. CPUs in multicore multiprocessor systems — such as Sun Fire
X4640 servers — can be logically partitioned into processor sets and bound to a
resource pool, and can ultimately be assigned to a Solaris zone. Resource pools
provide the capability to separate workloads so that consumption of CPU
resources does not overlap. Resource pools also provide a persistent configura-
tion mechanism for processor sets and scheduling class assignment. In addition,
the dynamic features of resource pools let administrators adjust system
resources in response to changing workload demands.
• Solaris Dynamic Tracing (DTrace) to Instrument and Tune Live Software Environments
When production systems exhibit nonfatal errors or sub-par performance, the
sheer complexity of modern distributed software environments can make accurate
root-cause diagnosis extremely difficult. Unfortunately, most traditional
approaches to solving this problem have proved time-consuming and inadequate,
leaving many applications languishing far from their potential performance levels.
The Solaris DTrace facility provides dynamic instrumentation and tracing for both
application and kernel activities — even allowing tracing of application
components running in a Java Virtual Machine (JVM™)1. DTrace lets developers and
administrators explore the entire system to understand how it works, track down
performance problems across many layers of software, or locate the cause of
aberrant behavior. Tracing is accomplished by dynamically modifying the
operating system kernel to record additional data at locations of interest. Best of
all, although DTrace is always available and ready to use, it has no impact on
system performance when not in use, making it particularly effective for
monitoring and analyzing production systems.
• NUMA Optimization in the Solaris OS

With memory managed by each processor on Sun Fire X4640 servers, the
implementation represents a non-uniform memory access (NUMA) architecture. In
NUMA architectures, the speed needed for a processor to access its own memory
is slightly different than that required to access memory managed by another
processor. The Solaris OS provides technology that can specifically help
applications improve performance on NUMA architectures.
– Memory Placement Optimization (MPO) — The Solaris 10 OS uses MPO to

improve the allocation and placement of memory across the physical memory of
a server, resulting in increased performance. Through MPO, the Solaris 10 OS
works to help ensure that memory is as close as possible to the processors that
access it, while still maintaining enough balance within the system. As a result,
many database and HPC applications are able to run considerably faster with
MPO.
– Hierarchical lgroup support (HLS) — HLS improves the MPO feature in the
Solaris OS. HLS helps the Solaris OS optimize performance for systems with
more complex memory latency hierarchies. HLS lets the Solaris OS distinguish
between the degrees of memory remoteness, allocating resources with the low-
est possible latency for applications. If local resources are not available by
default for a given application, HLS helps the Solaris OS allocate the nearest
remote resources.
1.The terms "Java Virtual Machine" and "JVM" mean a Virtual Machine for the Java platform.
• Solaris ZFS™
Solaris ZFS offers a dramatic advance in data management, automating and
consolidating complicated storage administration concepts and providing
unlimited scalability with the world’s first 128-bit file system. ZFS is based on a
transactional object model that removes most of the traditional constraints on I/O
issue order, resulting in dramatic performance gains. ZFS also provides data
integrity, protecting all data with 64-bit checksums that detect and correct silent
data corruption.
• A Secure and Robust Enterprise-Class Environment
Best of all, the Solaris OS doesn’t require arbitrary sacrifices. The Solaris Binary
Compatibility Guarantee helps ensure that existing Solaris applications continue
to run unchanged, protecting investments. Certified multilevel security protects
Solaris environments from intrusion. Moreover, Sun’s comprehensive Fault
Management Architecture means that elements such as Solaris Predictive Self
Healing can communicate directly with the hardware to help reduce both planned
and unplanned downtime.
Integrated Lights Out Management (ILOM)

The Sun Fire X4640 server offers an Integrated Lights Out Manager (ILOM) service
processor on a removable daughter card, allowing remote management for all
activities that do not require physically touching the system. Industry standards are
embraced throughout, letting these systems easily integrate into existing
environments. In addition, since the ILOM service processor is a core component of Sun
Fire X4640 servers, there is no additional charge for this functionality.
On-board ILOM Firmware and Connections

The ILOM service processor connects to all major components via on-board interfaces
such as I2C with a separate management network provided for remote access. Equipped
with field-upgradeable firmware, the ILOM service processor supplies management
functions for fan speed control and diagnostic LEDs, and provides a wealth of
connections to individual server components. Sensors can use the ILOM service
processor to generate entries in the system event log when the sensor crosses a certain
threshold value. Examples in Sun Fire X4640 servers include:
• Chassis sensors for intrusion, power supply failure, temperature failure, or fan failure
• Front and back panel sensors sensing the state of various LEDs and system locate
button
• Motherboard temperature sensors to monitor the ambient temperature chip on the
motherboard
• Power supply sensors to determine whether power supplies are present, connected
to AC power, and/or powering the system
• Hard disk drive backplane sensors to determine the presence and health of the disk
backplane
• Fan sensors to determine the presence of the fan tray, the speed of individual fans,
and to detect the failure of individual fans.
ILOM Communication Channels, User Management, and Security

Access to ILOM functionality on Sun Fire X4640 servers can be made through a variety of
both out-of-band and in-band communication channels. Out-of-band communication
helps ensure that effective management can take place even in the event of hardware
or networking failures, and includes:
• A management serial port that provides direct console access via a command line
interface (CLI)
• A dedicated Ethernet port that provides a web-based GUI (over HTTPS), a CLI via SSH,
IPMI 2.0, and SNMP v1, v2c, and v3
In-band communication to the ILOM service processor is provided via the host OS
running on the server. A variety of management tools can be used to access
management information on individual servers:
• Intelligent Platform Management Interface (IPMI) with IPMItool
IPMItool is a simple command-line interface to systems that support the
Intelligent Platform Management Interface (IPMI) v2.0 specification. IPMItool
provides the ability to read the sensor data repository and print sensor values,
display the contents of the system event log, print field-replaceable unit
information, read and set LAN configuration parameters, and perform remote
chassis power control. IPMItool was originally written to take advantage of IPMI-
over-LAN interfaces but it is also capable of using the system interface as provided
by a Linux kernel device driver such as OpenIPMI or a Solaris OS driver called BMC
that is provided with the Solaris 10 OS. IPMItool is available under a BSD
compatible license.
IPMItool is not designed to replace the OpenIPMI library but instead provides a
completely command-line oriented tool that can be used by administrators in
conjunction with other tools. Where possible, IPMItool supports comma-
separated values for output to facilitate parsing by other scripts or programs.
IPMItool is designed to run quick command-response functions that can be as
simple as turning the system on or off, or as complex as reading in the sensor data
records while extracting and printing detailed sensor information for each record.
• Simple Network Management Protocol (SNMP) Management

SNMP management provides remote access by SNMP-compliant entities to
monitor and control network devices and manage configurations including
statistics collection, performance, and security on a network. SNMP is a network
management protocol used almost exclusively in TCP/IP networks. Sun Fire X4640
servers provide SNMP MIBs to manage and monitor the servers using any SNMP-
capable network management system, such as HP OpenView Network Node
Manager (NNM), Tivoli, CA Unicenter, or IBM Director. The MIB data describes the
information being managed, reflects current and recent server status, and
provides server statistics.
SNMP v1, v2c, and v3 are supported with v3 selected by default (v1 and v2c are
disabled by default). SNMP “sets” can be selected or disabled (default). An IPMI-
specific trap called a Platform Event Trap, or PET, may also be generated. The
following SNMP MIBs are supported:
– SNMP-FRAMEWORK-MIB
– SNMP-USER-BASED-SM-MIB
– SNMP-MPD-MIB
– ENTITY-MIB
– SUN-PLATFORM-MIB
• Role-Based Administration
Different management users can be defined with corresponding roles and
responsibilities. Up to 10 user IDs can be created locally on the service processor
with each user ID consisting of a user name and the roles that are allowed. By
default, Administrator and Operator roles are defined. Authentication is carried
out against a local service processor database. Alternately, an LDAP client is
implemented in the ILOM service processor as well to allow authentication
against an LDAP server (LDAP groups must be mapped to service processor roles).
Up to 10 concurrent active sessions are supported on the service processor,
including serial, secure shell and web clients. User accounts can be authenticated
through LDAP, Radius, and Active Directory.
Remote Keyboard, Video, Mouse, and Storage (RKVMS)

To facilitate effective and full-featured remote management, the ILOM service processor
provides remote keyboard, video, mouse, and storage (RKVMS) support that is tightly
integrated with Sun Fire X4640 servers. Together these capabilities allow the servers to
be administered remotely, while accessing keyboard, mouse, video and storage devices
local to the administrator (Figure 15). ILOM Remote Console support is provided on the
ILOM service processor and can be downloaded and executed on the management
console. Input/output of virtual devices is handled between ILOM on the server and
ILOM Remote Console on the web-based client management console.
ILOM Remote Console

Displays Remote Video in Video
Application Window (Up to 1024x768@60Hz)
Graphics Redirect Over Ethernet
Local Mouse and

Keyboard
Sun Fire X4640

Management
ILOM Remote Console Server
Console Connected to ILOM Over
Management Ethernet
Floppy Disk or Keyboard, Mouse, CDROM,

Floppy Image and Floppy are Seen as
USB Devices by BIOS and OS
CDROM, DVDROM
or .iso Image Remote Keyboard, Mouse and Storage
Emulated as USB Devices by ILOM
Figure 15. Remote keyboard, video, mouse, and storage (RKVMS) support in the
ILOM service processor allows full-featured remote management for Sun Fire
X4640 servers.
• Remote Keyboard and Mouse Support

The Sun Fire X4640 server detects a USB keyboard and mouse through the ILOM
service processor. ILOM Remote Console captures mouse and keyboard input on
the management console and sends it to the ILOM service processor. The service
processor then transmits mouse and keyboard inputs on the respective USB buses
on the server. The server receives keyboard entries and mouse movements as if
they were generated by local USB devices.
• Remote Video Support

The server incorporates a VGA graphics controller. Graphics from the ATI graphics
controller is sent to the ILOM service processor. The service processor then
redirects the video signal to ILOM Remote Console running on the management
system over a network connection, where the video is displayed on the
management console. ILOM Remote Console supports 16-bit video to
accommodate higher quality.
• Remote Virtual Storage

The Sun Fire X4640 server detects two USB storage devices through the ILOM
service processor that can be set up in the BIOS as floppy, CD/DVDROM, or disk
image (floppy and CD/DVDROM by default). When the server tries to access either
the virtual floppy disk or the virtual CDROM, the ILOM service processor redirects
the request over the Ethernet connection to ILOM Remote Console on the
management console. ILOM Remote Console accesses the content from the
physical floppy disk drive, CD/DVDROM, or disk image, returning it across the
network to the server. The ILOM service processor presents the data to the server
as if it were coming from a local USB storage device. This capability allows the
server to boot remotely from a virtual USB storage device. The content of the
storage device can be an actual CD, floppy disk, or disk image (ISO or IMG).
Sun Ops Center

Beyond local and remote management capabilities, datacenter infrastructure needs to
be agile and flexible, allowing not only fast deployment but streamlined redeployment
of resources as required. Sun Ops Center technology provides an IT infrastructure
management platform for integrating and automating management of thousands of
heterogeneous systems. To improve life-cycle and change management, Sun Ops
Center supports the management of applications and the servers on which they run,
including the Sun Fire X4640 server.
Sun Ops Center simplifies infrastructure life-cycle management by letting

administrators perform standardized actions across logical groups of systems. Sun Ops
Center can automatically discover and group bare-metal systems, performing actions
on the entire group as easily as operating on a single system. Sun Ops Center remotely
installs and updates firmware and operating systems, including support for:
• Solaris 8, 9, and 10 on SPARC systems
• Solaris 10 on x86/x64 platforms
• Red Hat and SuSE Linux distributions
In addition, the software provides considerable lights-out monitoring of both hardware

and software, including fans, temperature, disk and voltage levels — as well as swap
space, CPU utilization, memory capacity, and file systems. Role-based access control
lets IT staff grant specific management permissions to specific users. A convenient
hybrid user interface integrates both a command-line interface (CLI) and an easy-to-use
graphical user interface (GUI), providing remote access to manage systems from
virtually anywhere.
Sun Ops Center provides advanced management and monitoring features to the Sun
Fire X4640 server. The remote management interface discovers and presents the the
servers, making operations, detailed inventory, and status pages available to
administrators. Servers can be discovered and organized into logical groups. Organizing
servers into groups also allows features such as OS deployment across multiple
systems.
Some of the functions available through Sun Ops Center software include operating
system provisioning, firmware updates (for both the BIOS and ILOM service processor
firmware), and health monitoring. In addition, Sun Ops Center includes a framework
allowing administrators to easily access inventory information, simplifying the task of
running jobs on multiple servers with server grouping functionality.
38 Conclusion Sun Microsystems, Inc.
Chapter 5
Conclusion
To curb the impact of sprawling datacenters, many organizations need x64 servers with
truly massive levels of scalability. Whether serving large database or high performance
computing applications, or consolidating multiple lower-powered servers, systems
must scale gracefully. The balanced design of Sun Fire X4640 server lets them scale in
processing power, memory, and I/O bandwidth, without arbitrary limitations.
The Sun Fire X4640 server features an innovative enterprise-class design that makes the
most of AMD’s Direct Connect Architecture. Based on innovative Six-Core AMD Opteron
processors, this server can supply a powerful building block to help scale, virtualize,
and consolidate datacenter infrastructure. Offering best-in-class performance,
remarkable density, and extensive system expandability, the Sun Fire X4640 server
makes the most of constrained datacenter resources. In addition, the server offers
extensive reliability, availability, and serviceability features along with improved energy
efficiency. Integrated Lights Out Management technology provides advanced
monitoring and management — at no extra cost.
Ultimately, even the most capable servers require enterprise-class operating systems
and tools. With a choice of the Solaris OS, Linux, Microsoft Windows, and VMware
virtualization software, these servers provide a wealth of popular options. The
Solaris OS in particular offers advanced and innovative features such as Solaris
Containers and DTrace that can help consolidate and tune enterprise deployments.
With powerful management tools such as Sun Ops Center, organizations can deploy
Sun Fire X4640 servers quickly with confidence, knowing that their choices can improve
their results as well as the bottom line.
39 Conclusion Sun Microsystems, Inc.
For more information

To learn more about Sun products and the benefits of Sun Fire X4640 server, contact a
Sun sales representative, or consult the related documents and Web sites listed in
Table 5.
Table 5. Related Websites
Web Site URL Description

sun.com/x64 Sun Fire X4640 server
sun.com/solaris The Solaris Operating System
sun.com/software/products/opscenter Sun Ops Center
sun.com/vmware VMware virtualization software
This Page Intentionally Left Blank
Sun Fire X4640 Server Architecture On the Web sun.com/x64
Sun Microsystems, Inc. 4150 Network Circle, Santa Clara, CA 95054 USA Phone 1-650-960-1300 or 1-800-555-9SUN (9786) Web sun.com
© 2009 Sun Microsystems, Inc. All rights reserved. Sun, Sun Microsystems, Java, JVM, Lustre, MySQL, NetBeans,Solaris, Sun Fire, Sun Ray, and StarOffice are trademarks or registered trademarks of Sun Microsystems, Inc. or
its subsidiaries in the United States and other countries. All SPARC trademarks are used under license and are trademarks or registered trademarks of SPARC International, Inc. in the US and other countries. Products
bearing SPARC trademarks are based upon an architecture developed by Sun Microsystems, Inc. AMD and Opteron are trademarks or registered trademarks of Advanced Micro Devices, Inc. Information subject to change
without notice. Printed in USA SunWIN #: 470898 11/09

x4640 Server Architecture

Uploaded by

Document Information

Original Description:

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

x4640 Server Architecture

Uploaded by

Copyright:

Available Formats

SUN FIRE™ X4640

Delivering Massive x64 Single-System Scalability . . . . . . . . . . . . . . . . . . . . . . . . . 2

AMD Opteron Processor Technology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11

Enterprise-Class Operating System and Management Software . . . . . . . . . . . . . 30

To stay competitive, organizations must deploy applications quickly, delivering compute

Consolidation through virtualization has emerged as an effective strategy for

The Sun Fire™ X4640 Server

• Remarkable Compute and Memory Density

• Enterprise-Class High Availability

Figure 1 illustrates the 4U Sun Fire X4640 server enclosure.

a.The RAID HBA consumes a single PCI Express slot

Features of the Sun Fire X4640 server include:

Multiple off-the-shelf conﬁgurations are available, along with a wide spectrum of

A Choice of Operating Systems

Table 2. Supported OS releases for the Sun Fire X4640 server

OS Release 32-bit or 64-bit Six-Core

• The Solaris™ Operating System

Provided on all Sun systems at no charge, the Solaris OS delivers performance,

Sun is one of the largest contributors to the open-source community. Areas of

– Lustre™ parallel ﬁle system

• Microsoft Windows Environments

Effective and Consistent System Design for the Datacenter

• Continued investment protection — Sun designs for maximum investment

Chassis Design Innovations

Consistent connector layouts for power, networking, and management make

• Leading Reliability, Availability, and Serviceability (RAS)

– Reduced parts count

Innovative Processor Technology

AMD Opteron processors offer considerable advantages, including:

• AMD Virtualization (AMD-V) — AMD Virtualization reduces overhead by selectively

Fourth-Generation Six-Core AMD Opteron Processors

Forth-Generation Six-Core AMD Opteron

Core 1 Core 2 Core 3 Core 4 Core 5 Core 6

System Request Interface

DDR2 HyperTransport 0 HyperTransport 2

Coherent HyperTransport 3 (cHT3)

The beneﬁts of this upgraded HyperTransport technology include the following:

Six-Core AMD Opteron processors provide Coherent HyperTransport 3 support on

Figure 3. Coherent HyperTransport 3 (cHT3) can be provided between processors,

HyperTransport Assist (HT Assist)

CPU 1 CPU 2 CPU 1 CPU 2

CPU 3 CPU 4 CPU 3 CPU 4

Without HT Assist With HT Assist

AMD-P Power Management Technology

AMD-V Virtualization Technology

CPU 0 CPU 2 CPU 1 CPU 3 CPU 4 CPU 5 CPU 6 CPU 7

Coherent HT3 links - 4.8 GT/sec

SAS 4 x 1 GB Rear Front SATA VGA Management Serial

Figure 5. Sun Fire X4640 server motherboard block-level diagram

The CPU/Memory modules are connected via non-coherent HyperTransport links to a

AMD Direct Connect Architecture Implementation with

Ladder Twisted Ladder Enhanced Twisted Ladder

Figure 7. Hops between processors in Sun Fire X4640 servers

Connectivity between CPU/Memory modules changes depending on the actual number

CPU CPU CPU CPU

CPU CPU CPU CPU

CPU CPU CPU CPU

8 CPU Modules 6 CPU Modules

CPU CPU CPU

CPU CPU CPU

4 CPU Modules 2 CPU Modules

Figure 8. Hypertransport topology in Sun Fire X4640 servers for 2-processor,

AMD-8132 HyperTransport PCI-X Tunnel