Electrical Design of Energy Efficient Data Centers PDF

ENERGY EFFICIENT DATACENTERS
ELECTRICAL DESIGN
Michael Ryan, Brett Rucker, Dean Nelson, Petr Vlasaty,
Ramesh KV, Serena DeVito, and Brian Day
Sun Global Lab & Datacenter Design Services
Sun BluePrints™ Online
Part No 820-6213-10
Revision 1.0, 3/11/09
Sun Microsystems, Inc.
Table of Contents
Introduction. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .1
About This Article . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
Traditional Datacenter Electrical Design . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .3

Utility Power and Transformer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
Switchgear . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
Backup Power . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
Power Distribution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
Rack Power Distribution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
Datacenter Tier Levels. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
Sun’s Approach to Electrical Design . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
Increasing Electrical Distribution Efficiency . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .8

Component Selection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
Equipment Power Supplies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
Power Characteristics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
Appropriate Levels of Redundancy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
Flexibility Through Modularity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17

Datacenter Power Distribution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
Extending the Datacenter Lifecycle . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21

Electrical Yard . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
Building Infrastructure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22
Pod Power Distribution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22
Sun’s Electrical Design At Work. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24

Santa Clara, California . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24
Broomfield, Colorado . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29
Sun Modular Datacenter. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34
Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37
About The Authors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38
Acknowledgments. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43
Ordering Sun Documents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43
Accessing Sun Documentation Online . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43
1 Introduction Sun Microsystems, Inc.
Chapter 1
Introduction
Good datacenter electrical design supports increased energy efficiency, stability, long-
term growth, and business agility. As the following examples indicate, electrical design
has become paramount in this equation:
• Electrical design that supports growth gives datacenters a longer life. Many
datacenters built even as recently as 5–10 years ago have reached a premature end of
life because their infrastructure was overtaken by density, power, and cooling
demands that were not planned for.
• Modular designs, from the electrical yard to server racks themselves, can support
both rapid growth and rapid change, giving organizations the agility they need to
adapt to changing business conditions.
• System efficiency can be increased while not comprising availability by careful system
design and component selection, such as choosing high-efficiency transformers and
uninterruptible power supplies (UPS).
• Appropriate use and implementation of redundancy can raise efficiency and reduce
construction and operational cost over the life cycle of the datacenter. Everyone
wants a Tier 4 datacenter, but there is inherent capital and operational inefficiency in
having two complete distribution systems. Instead, a flexible and modular electrical
design can support multiple Tier levels in the same datacenter, allowing the cost of
the higher service level to be spent only on the systems requiring it.
• Metering at the right places in the electrical distribution system is a key feature
that’s needed in order to assess operating conditions of the system and components.
Good metering design practices are imperative in the datacenter to continuously
assess system heath datacenter efficiency. Metering should be implemented starting
at the utility feeds and cascading through the system down to the rack level.
About This Article

This Sun BluePrints™ article discusses the electrical design principles that Sun’s Global
Lab & Datacenter Design Services (GDS) organization incorporated into its energy-
efficient, pod-based datacenters. Sun’s GDS organization is responsible for Sun’s
technical infrastructure (datacenters, labs, server and communication rooms and
wiring closets) worldwide, and has ushered in a new generation of energy-efficient
datacenters at Sun.
• Chapter 2, “Traditional Datacenter Electrical Design” on page 3, discusses how
datacenter electrical systems are typically designed.
• Chapter 3, “Increasing Electrical Distribution Efficiency” on page 8, highlights the
opportunities that we have found to improve modularity and efficiency in datacenter
electrical design.
2 Introduction Sun Microsystems, Inc.
• Chapter 4, “Flexibility Through Modularity” on page 17, shows how our design
choices create flexible datacenters that contribute to business agility.
• Chapter 5, “Extending the Datacenter Lifecycle” on page 21, discusses the
importance of coordinating expected datacenter lifecycles with equipment refresh
cycles. The chapter shows the many ways in which datacenters can be built from day
one to support future expansion
• Chapter 6, “Sun’s Electrical Design At Work” on page 24, provides a tour of our
datacenter designs from the perspective of the one-line diagrams that describe their
electrical systems. We focus on Sun’s Santa Clara, California datacenters, including
our most recent addition of Sun™ Modular Datacenters at the site.
3 Traditional Datacenter Electrical Design Sun Microsystems, Inc.
Chapter 2
Traditional Datacenter Electrical Design
From an electrical design standpoint, a datacenter is a hierarchy of electrical devices

that transmit power from a utility feed to server racks (Figure 1). One or more feeds
arrive from the electrical utility before or after their voltages are transformed to usable
levels. Switchgear provides, among other things, a disconnect point for the utility feed.
Where needed, UPS and generators provide transient and longer-term backup power.
Conduit and wireways feed distribution systems in the datacenters themselves. Power
distribution mechanisms deliver electricity to racks and standalone systems. Within
racks, smaller distribution systems provide power to individual servers. This chapter
describes the various steps in the distribution chain; the next chapter describes the
ways in which efficiency, modularity, and flexibility can be introduced into each step.
FACILITY TRANSFORMER SWITCHGEAR UPS 480V PDUs RACKS

POWER SWITCHGEAR
GENERATOR
MORE THAN ONE OF EACH DEVICE INSTALLED AT EACH LAYER
Figure 1. A typical datacenter power distribution system transforms and delivers utility power to equipment racks.
Utility Power and Transformer

Power from the electric utility is usually delivered at voltages anywhere from 480V to
21 kV and higher. The first step in the distribution process is to transform this voltage to
a level that is usable by the building and datacenter equipment. The transformer may
be owned by the utility but installed on a pad provided by the customer or it may be
owned by the customer. Both models are used at Sun, and Figure 2 illustrates
transformers owned by Sun at Sun’s Santa Clara, California campus.
In many parts of the world, a single transformer reduces medium voltage from the
utility to 400/230V, with 230 the final voltage supplied to datacenter equipment. (The
400/230V notation indicates three-phase power where the first number indicates phase
to phase and the second number indicates phase to neutral voltage). In the United
States, voltage is reduced in two steps. A first step to 480/277V is used to supply devices
including HVAC equipment and lighting throughout the building. A second step to
208/120V brings the voltage down to that used by compute equipment. Each time
voltage is reduced, some energy losses are introduced into the power distribution
system.
Generators
Transformers
Figure 2. This view of the main switch yard in Sun’s Santa Clara, California facility
shows four transformers connected to three separate utility feeds and switchgear,
Backup generators are in the background.
Switchgear
In industrial electrical design terminology, Datacenters use switchgear and distribution boards to safely distribute power from the
“medium voltage” ranges from 1-33kV. utility to the datacenter floor. The switchgear includes circuit breakers and switches for
managing medium and low voltages and they are typically used to distribute large
amounts of power to various locations within datacenters and buildings.
The circuit breakers installed in switchgear tend to be more sophisticated than breakers
in distribution panels. Switchgear breakers typically have intelligence for monitoring
and control, have configurable trip points, and can include ground fault interruption
circuitry (depending on the design). The purpose of the breakers is to interrupt and
disconnect the load from the power source in the event of abnormally high current
conditions that could damage the electrical system or create unsafe operating
conditions such as a phase-to-phase or phase-to-ground faults. Switchgear may also
include tie breakers and cross-connection schemes that allow the power path to be
reconfigured through the same set of switchgear, or around it in case of a fault. For
example, in a dual-path power system design, it is important to be able to re-route
power around a fault in the system. Tie breaker schemes allow switchgear that would
normally be fed from one source to be switched over to a secondary source. Automatic
transfer switches can also be used to automate this process.
At Sun, a tie breaker scheme allows power supplied to buildings to be doubled. By

installing double-ended switchgear with internal tie breakers, an additional power feed
can be incorporated by installing a new transformer at the opposite end of the
switchgear. The tie breakers then separate the switchgear from one bus into two
distinct buses.
Backup Power
Backup generators are used to power critical equipment in the event of a utility power
failure. Backup power is sometimes supplied to the main switchgear, sometimes it is
provided further down in the distribution system.
Backup power generators cannot start instantly, so another form of backup power
needs to continue to power datacenter equipment during the interval of eight seconds
or less that is needed to start the generators and ready them to carry the datacenter
and support loads. The traditional approach is to use battery backup to support the
datacenter load during this time. Rotational UPS that draw power from a flywheel are
starting to prove themselves superior to battery backup. Both approaches introduce
losses that can be mitigated as discussed in the next chapter.
Power Distribution
Conduit and wireway distribute the conductors that run from the switchgear to the
datacenter itself, where typically one of three techniques are used to distribute power
to racks. The trade-offs between these approaches are discussed in detail in “Datacenter
Power Distribution” on page 17.
• Distribution panel boards use standard wall-mounted distribution panels at the
datacenter’s perimeter with any combination of conduit, wire mold, raceways,
and armored cable whips running from the breaker panels to individual racks.
Power is typically delivered underneath a raised floor in the datacenter, but also
can be delivered overhead.
• Power distribution units (PDUs) typically consist of multiple distribution panel

boards and can also include internal transformers. (PDUs without transformers are
typically called remote power panels (RPPs). PDUs tend to have a significant
amount of built-in intelligence and monitoring capabilities. PDUs are typically
installed on the datacenter floor throughout the space. PDUs can be used in both
slab and raised-floor datacenters.
• Busway provides a modular track above or below rows of racks. ‘Cans’ combine
breakers and short whips that can be plugged into the busway to provide power
directly to rack-mounted power strips. Busways are capable of distributing three-
phase power across the datacenter floor. The cans are designed to use any
combination of the three phases, allowing different types of equipment to be
powered by plugging in different cans, and for power consumption to be balanced
across phases. Sun has found that busway power distribution provides the most
efficient and modular way to distribute power to the racks and has standardized
on technology for new datacenters.
Rack Power Distribution

Inside each rack is a power strip or rack-mounted power distribution unit (rPDU) that
distributes power to the IT equipment installed in the rack. Sun has standardized on
rPDUs that monitor power and allow each plug to be powered on and off over the
network. These rPDUs provide data for power monitoring as well as allowing remote
users the ability to power cycle equipment. As the datacenter continues to evolve,
power monitoring and control of individual outlets will increase in importance.
Knowing actual power loads at the server and rack level enables the datacenter
manager to see the power utilization and distribution across the datacenter space. This
information can then be used in the IT equipment provisioning process. It can also be
used to dynamically adjust loads based on server utilization as compute load changes.
Datacenter Tier Levels

As defined by Uptime Institute's white Datacenters typically are classified into four Tier levels that are defined by the levels of
paper Tier Classifications Define Site redundancy in their power and cooling systems. From the perspective of electrical
Infrastructure Performance, Tier levels
design, the Tier levels are defined as follows:
range from 1–4, with Tier 4 being the most
fault tolerant. • Tier 1 uses a single path for power distribution, without redundant components.
• Tier 2 uses a single path for power distribution. Redundant components include
multiple UPS and generator power.
• Tier 3 uses multiple power distribution paths along with redundant components.
This allows one distribution path to be taken out of service for maintenance or
repairs without having to take the datacenter offline.
• Tier 4 uses fault tolerant multiple power system and system distribution paths,
redundant components, and uses only equipment with dual power supplies so
that any distribution path or complete system can fail completely without
affecting datacenter uptime.
Sun’s Approach to Electrical Design

The Green Grid’s definition of PUE and DCiE is Sun’s approach is to use the opportunities that exist within a datacenter’s electrical
available in their white paper Data Center design to increase efficiency, flexibility, and provide room for growth. Efficiency
Power Efficiency Metrics at:
increases help to reduce power consumption and increase the datacenter’s Power
http://www.thegreengrid.org
Usage Effectiveness (PUE) or its reciprocal DCiE. Building in flexibility allows a
datacenter to change rack power configurations in minutes, rather than days or
months, contributing to the company’s ability to quickly adapt to changing business
conditions and product development efforts. Preparing for future growth extends
datacenter life by anticipating rapid hardware refresh cycles and the increased power
requirements that accompany them.
8 Increasing Electrical Distribution Efficiency Sun Microsystems, Inc.
Chapter 3
Increasing Electrical Distribution Efficiency
A typical datacenter can easily lose 10 percent of its power in the power distribution
system. Transformers, UPS, and distribution contribute to power loss. Using a number
of the techniques discussed in this section, we have reduced the power distribution
system loss in our Santa Clara, California software organization datacenter from ten
percent to approximately four percent of the total datacenter facility load.
The calculation is as follows: A savings of six percent may not seem particularly significant. Consider one of our
800 kW x .06 = 48 kW datacenters that has an IT load of 800 kilowatts (kW). Reducing overhead by six percent
48 kW x 8760 hours x $0.08 = $33,638
is a savings of 48 kW, or $33,638 per year. Any loss due to inefficiency is converted to
heat, so the savings is even greater when one considers that greater efficiency means
less cooling is needed to remove that heat, another cost reduction.
Component Selection
The low-hanging fruit for increasing electrical system efficiency is choosing high-
efficiency components from transformers and UPS to server power supplies. This
straightforward choice is often overlooked due to higher initial costs. However, if the
total cost of ownership is evaluated, the payback period proves to be very short. With
the current green initiative trend, these component costs are starting to come down,
making the decision easier to justify.
High-Efficiency Transformers
In the United States, most datacenters have at least two power conversion steps that
require transformers. Typical transformers impose about a 3 percent loss, making them
around 97 percent efficient. Using a high-efficiency NEMA TP1-rated transformer can
increase performance to the mid-98 percent efficient range. This not only saves energy
in the electrical system but since all transformer loss are converted to heat it also
reduces electrical room cooling loads.
High-Efficiency UPS
There are many factors that contribute to the efficiency of UPS systems. However, a
common factor across most UPS systems is that the higher the utilization, the more
efficiently they operate. For example, if we look at a typical UPS efficiency curve, the
rated efficiency of the unit is usually stated for loads greater 70% of capacity. Because
of UPS redundancy requirements, UPS system are rarely operated in such a high region.
Performance of the UPS begins to drop off when loads are 50 percent or less of rated
loads. At around 30 percent utilization, UPS efficiency drops very quickly as utilization is
reduced. Since UPS systems are a major contributor to electrical system losses it is
important to make sure the UPS system is designed so that it can be operated at a
higher utilization level and that the efficiency of the unit is maximized for the expected
design operations.
APC white paper #1 describes UPS topologies. There are many different UPS topologies that can be used in datacenter applications,
Please refer to: http://www.apc.com including double on-line conversion, delta conversion, and line interactive. In our Santa
Clara datacenter, we chose to use a delta-conversion UPS that uses a modular approach
to support capacity increases. This allows us to right size the UPS for day one needs and
then grow the UPS using 200 kW power modules as the load increases, all without
major construction. This choice provides us with UPS efficiency into the 97 percent
range which is typically about 3–4 percent better than the traditional double-
conversion UPS.
Alternatives to Battery-Based UPS

Battery-based UPS are costly to maintain. Batteries must be replaced over time,
sometimes as early as when they are five years old; rooms must be vented to prevent
hydrogen gas build up; catch pans must be in place to protect against acid spills; and
battery-based UPS take a significant amount of space. Batteries are also not very
environmentally friendly and require controls on the hazardous materials both when
handling and disposing of them.
A low-maintenance, more environmentally friendly approach uses the rotational inertia

of a flywheel to generate electricity during the gap between a power failure and
generator startup (Figure 3). There are two common approaches to rotational UPS:
Figure 3. A line-interaction UPS is • The traditional approach uses a typical UPS topology, such as a double-conversion or
in use at Sun’s Broomfield, line-interactive model, replacing the batteries with a flywheel that stores energy
Colorado datacenter. through rotational inertia. When utility power is lost, the energy stored in the
flywheel powers the datacenter. It can substitute directly for a battery-based UPS
within a building or datacenter.
• Another approach couples a diesel generator with an induction clutch. When power
fails, the flywheel turns the generator until the diesel engine starts and comes up to
speed.
Flywheel-based UPS are finding increasing acceptance throughout the industry because
they greatly reduce maintenance costs and require less space in the UPS room than
traditional battery-based UPS systems. The biggest concern of datacenter operators is
the relatively short ride-through time provided by the flywheel. Battery UPS system are
typically sized to support loads for 10–15 minutes while the generators come on line. A
typical generator, however, is ready to accept full load in 8–10 seconds.
If a generator does not start up in 8–10 seconds, there is typically very little that can be
done to repair it during the 15 minutes the batteries can deliver power. While rotational
UPS do require maintenance on mechanical parts such as bearings, they take far less
space and maintenance than battery backup systems. The current drawback is that
flywheel UPS systems tend to have a higher initial cost as compared to a traditional
battery UPS. Reduced maintenance costs usually offset the difference in a short time.
Equipment Power Supplies

The power supplies on server, storage, and networking equipment also contribute to
power distribution losses. Power supplies are always sized to handle the worst-case
scenario: a server with the maximum memory, internal disk, and peripheral cards
installed and with the CPUs and fans running at full power. Many servers, however,
actually operate at 50 percent or less of their name plate power (Figure 4), a range
where historically, average power supplies can be relatively inefficient.
100
Efficient
Percent Efficiency
Average
45
0 Percent of Name Plate Power Output 100
Figure 4. Power supplies such as those used by Sun are more efficient across a
broader output range.
Information on the 80 Plus program is With today’s focus on energy efficiency, server vendors including Sun are striving to
available at http://www.80plus.org. make their power supplies more efficient at low utilization levels. In the U.S., an electric
utility-funded program called “80 Plus” is promoting the use of power supplies that are
at least 80 percent efficient at 20, 50, and 100 percent of their rated load. Most of the
power supplies on servers from Sun exceed this specification, with some of them 93
percent efficient at an 80 percent load.
Another issue to consider is the fact that systems with redundant power supplies
operate at far less than 50 percent load, often putting them into an inefficient range of
operation. If the application supported by the systems is designed to be resilient to
single points of failure, then servers with single power supplies can be used. This allows
their power supplies to run in a much more efficient portion of their operating range.
Regardless of whether single or dual power supplies are used, care needs to be taken to
ensure that each power supply is plugged into a dedicated circuit.
Power Characteristics
An organization’s choice of power characteristics — voltages and AC vs. DC —has an
impact on datacenter power distribution efficiency.
208V Vs. 230V

This topic is discussed in detail in the Green In the U.S., we typically reduce the utility’s medium-voltage supply in two steps, using
Grid’s white paper 16 titled: Quantitative two stages of transformers, down to 208/120V. Other parts of the world use higher
Efficiency Analysis of Power Distribution
voltages, such as single-phase 230V (line to neutral), yielding some efficiency benefits.
Systems for Data Centers The paper is
available at: http://www.thegreengrid.org The higher voltage can eliminate the inefficiency of a second transformer step, and it
allows the use of less copper, as distribution paths internal to the datacenter can carry
more power at lower amperage.
Virtually all datacenter equipment is built to accept a range of input power from 100–
240V, so the choice of which voltage to use is one that is dependent on comfort levels
and local electrical codes. Had we had more time to evaluate, test and educate the
project team when we built our datacenters in California, we may have been able to
increase efficiency even further by deploying 230V throughout.
Alternating Vs. Direct Current

There is a long-running controversy regarding the relative merits of using high-voltage
DC versus AC power in the datacenter. The argument is this: high-voltage (380–575V) DC
eliminates the requirement for large amounts of copper in the distribution system,
making this aspect competitive with AC power. Using DC throughout the datacenter
eliminates three AC-DC power conversions that add to inefficiency (Figure 5):
• 480 VAC is converted to direct current at the UPS to charge the batteries.
• Battery power is inverted to 480 VAC to supply the datacenter.
• The first stage of datacenter equipment power supplies converts incoming AC power
to high-voltage DC, which is then regulated down to the low DC voltages used within
the server, storage, or networking equipment.
Power Inefficiencies 380 VDC (typical)
AC/DC DC/AC AC/DC DC/DC

480 VAC
Voltages
Internal
Supply
Power Power Supply Unit (PSU)
Lead-Acid Batteries UPS PDU Server
Figure 5. The argument for DC power in the datacenter is that using DC power
throughout eliminates multiple inefficient AC-DC power conversions.
The final report resulting from this project is In conjunction with a California Energy Commission-sponsored project, Sun
available at: implemented a proof-of-concept that demonstrated the power savings of using DC in
http://hightech.lbl.gov/documents/
the datacenter environment. Sun created two identical datacenters, one with AC and
DATA_CENTERS/DCDemoFinalReport.pdf
one with DC power distribution. Off-the-shelf equipment from Sun, Cisco, and Intel was
used, with the power supplies modified by hand to eliminate the power conversion
circuitry that converts alternating to direct current. Incoming power was rectified,
distributed at 380 VDC, and fed directly into the modified power supplies. This was a
proof-of-concept demonstration only, as 380 VDC power supplies are not available as
products from Sun. (Sun does offer 48 VDC-powered equipment through its Netra™
product line designed for communication-carrier environments). The results showed a
2–5 percent improvement in power supply efficiency, and up to a 10–20 percent
improvement when the use of a flywheel vs. a battery-based UPS was factored into the
equation.
We believe that the benefits demonstrated by this experiment are superseded by

improvements in power supply efficiency and that similar improvements are possible
without the use of DC power. The equipment used in this experiment used older
equipment equipped with power supplies where a greater improvement could be
shown compared to today’s more efficient power supplies.
Another approach that is being discussed is using the telco standard of 48 VDC.
Regardless of which DC power approach is used, we see some key disadvantages:
• Equipment that accepts high-voltage DC directly is not available on the market today.
There is a limited selection of equipment that can use 48 VDC.
• There is a relative lack of experience and comfort using DC power in the datacenter.
Distribution systems, system power connections, and breakers are all different, and
everyone from electrical contractors to system administrators would need training.
• Maintenance and arc-flash suppression has not been uniformly addressed.
• For most companies, the ability to leverage the newest and best server, storage, and
network technology is key to maintaining a competitive edge. Today, the choice of DC
power restricts equipment choices from almost all major manufacturers.
Appropriate Levels of Redundancy

Datacenter power efficiency can be increased by using redundancy in appropriate ways.
First, choosing a level of redundancy that is matched to the organization’s mission, and
their level of accepted risk, increases efficiency by reducing the amount of redundant
equipment that is necessary, and also by reducing layers of components (such as UPS)
that add to inefficiency. Second, redundancy can be implemented so that equipment is
used in a more efficient part of its operating range.
Choosing Tier Levels

Everyone wants to have a Tier 4 datacenter, with two complete power distribution
systems so that either can be taken out of service without affecting datacenter uptime.
In fact, most organizations only require a small portion of their datacenter to be built to
Tier 4 specifications — if any at all. The cost of implementing a Tier 4 design and
continuous operation at that level can be more than five times greater than a Tier 1
facility.
Although our electrical design approach can support any Tier level, our datacenters at
Sun are used for internal product development and typically require Tier 1 or Tier 2
support. Our IT specific datacenters provide more critical services, and therefore require
a Tier 3 level of service.
A pod, as defined in the Sun BluePrints Series Our datacenters can support multiple Tier levels in the same room. Different
article The Role of Modularity in Datacenter configurations of our modular busway can feed multiple parallel power distribution
Design, is a small, self-contained group of
paths down to individual racks. Since most of our equipment does not actually require
racks and/or benches that optimize power,
cooling, and cabling efficiencies. A typical UPS power, we can run two busways, one with UPS power and one without, to each of
pod is a collection of 20–24 racks with a our pods. Racks that require redundancy and UPS power can connect to both busways.
common hot or cold aisle along with a Those that don’t require UPS power do not. In the end, only approximately 20 percent
modular set of infrastructure for it to handle
of our systems actually require UPS power, helping to increase efficiency and
its own power, cooling, and cabling.
significantly reduce the capital and operating costs associated with uninterruptible
power.
Our pods are self-sufficient from a power, cooling, and cabling perspective, so a Tier 3
pod can be placed in the same room with Tier 1 and Tier 2 equipment. The same would
be true if the corporation required Tier 4 as long as a dual-path system with or without
redundancy was carried back to the central plant.
Using Distributed Redundancy

Using an appropriate level of redundancy in the electrical distribution system is one
means to increase efficiency; the way in which you implement redundancy also has an
impact on efficiency.
2N Redundancy
A typical UPS configuration uses 2N redundancy, where one of two UPS systems can
support the entire load should one UPS fail. This model operates each UPS at less than
50 percent utilization. Typical maximum utilization levels are 40–50 percent so that
when the UPS has to support 100 percent of the datacenter load it does not exceed its
full capacity rating and trip offline. It should also be noted that UPS systems operating
in this range are relatively inefficient especially if operated at less than expected full
design loads. A hypothetical configuration illustrated in Figure 6 shows two UPS
powering a datacenter consisting of three racks.
Figure 6. In a 2N redundant model, each UPS is sized to support 100 percent of the
workload but operates at less than a 50 percent utilization level.
N+1 Redundancy (Parallel Redundant)

This model uses N+1 UPS to support a load where one UPS can fail and the remaining N
are sufficient. The example in Figure 7 shows a three UPS N+1 UPS system, each UPS
supports 33 percent of the power draw, but is sized to support 50 percent in the event
that one fails. This puts each UPS at a more favorable 66 utilization level. Because each
UPS is sized too small to power the entire server workload, the outputs must be
synchronized to a common bus, and it is a single point of failure for this model.
Distributed Redundancy
A distributed redundancy model uses three or more UPS connected with multiple paths
to servers. A distributed redundancy UPS system model can be used to achieve multiple
distribution path redundancy where required without the need for a 2N system.
(Figure 8). For example, a three-UPS distributed system can achieve a utilization level of
66 percent. This is the same utilization level as the N+1 model, but the configuration
eliminates the need to tie outputs together and eliminates the bus as a single point of
failure. The benefits can include lower capital cost as compared to a 2N system,
increased efficiency, and dual power paths to servers.
Figure 7. In an N+1 redundant model, all UPS are connected to all servers, but the
bus is a single point of failure.
Figure 8. In a distributed redundant model, each UPS supports one third of the
servers and operates at a 66 percent utilization level. There is no single point of
failure.
As with all approaches to UPS redundancy, distributed redundancy requires adjusting

the balance of servers to UPS, and this requires a flexible, modular power distribution
system that can adapt as a datacenter grows. Modularity and Flexibility are the topics
of the next chapter.
17 Flexibility Through Modularity Sun Microsystems, Inc.
Chapter 4
Flexibility Through Modularity
Modularity is a key datacenter design principle that enables flexibility and the ability to
grow. At Sun, our modular, pod-based design creates energy-efficient building blocks
that can be duplicated easily in any size or Tier-level datacenter worldwide. The pod
design allows different modules to be selected and configured to best meet client
requirements as a datacenter changes over time. Following our modular, pod-based
approach, different pods in the same room can to be built to different Tier-level
specifications because our electrical design can support it. The modular approach
allows pods, and racks within pods, to have radically different densities, with a rack
using 4 kW next to one that requires 30 kW. The modular approach is flexible so that a
4 kW rack can be replaced with one using 30 kW — or vice versa — in a matter
of hours. The same modular design that supports brick-and-mortar datacenters also can
be used to support Sun Modular Datacenters as discussed in “Sun Modular Datacenter”
on page 34
Datacenter Power Distribution

For a more detailed discussion of the trade- The feature that most supports the day-to-day flexibility of our datacenters is our choice
offs summarized here, see the Sun BluePrints of power distribution on the datacenter floor itself. The challenge here is two-fold: the
article: Energy Efficient Datacenters: The
power distribution system must be able to support growth so that the datacenter can
Benefits of Modular Design.
handle increased density at every equipment refresh cycle. The power distribution
system also must be able to handle day-to-day equipment churn that significantly
changes the power draw at the rack level. At Sun, this is extremely important, as spaces
used for new hardware bring-up must allow for the next generation of products and
even higher densities to be rolled in and powered up without long delays. Our power
distribution system helps future Sun products to get to market faster because all outlet
types and loads are supported. Our datacenters have products in them today that our
customers won't see for two years.
Hard-Wired Power Distribution

The oldest, and the least-flexible power distribution system is one that uses circuit
breaker panels at the datacenter perimeter with conduit, wire moulding, or armored-
cable whips running from the breaker panels to individual racks or groups of racks.
This approach limits flexibility, as a significant amount of electrical work is
necessary in order to add or change service to a rack. Every time a panel is opened,
all of the equipment attached to it is at risk.
Power Distribution Units

PDUs combine transformers and breaker panels in units that are placed on the
datacenter floor. PDUs are typically installed throughout the datacenter floor, with
individual flexible whips powering individual racks. PDUs occupy valuable
datacenter floor space and add to the datacenter’s heat load. Whips are difficult to
manage, and every time that a PDU is opened to add or change a configuration, it
introduces the possibility of human error, putting everything the PDU powers at risk.
Although we have used PDUs in some of our earlier designs, we now see it as a
limiting factor in those datacenters’ flexibility. This is because all changes require an
electrician, the PDU must be powered down, much more time is required for changes,
and the cost can be up to ten times more over the life of the electrical system when
compared to busway systems.
The Benefits of Modular Busway

Our pod design removes all transformers and power distribution units from the
datacenter floor. Instead, we use a hot-pluggable busway overhead or under a raised
access floor to distribute power to each rack (Figure 9).
Figure 9. Modular busway is installed overhead or under a raised access floor that
is not used for air distribution.
Each busway includes three phases, one neutral, and one ground conductors that are
accessed by ‘cans’ that tap into the right combinations for the power to be delivered.
The busway supports an intermixing of 120V, single-phase power up to 208V, three-
phase power (or the equivalent geographical power configuration for datacenters
located outside of the U.S.). Despite its capacity to handle up to 100A through a single
can, an electrician can install a new circuit in a matter of minutes. This flexibility is key
in our datacenters where equipment churn occurs on a regular basis.
This modular approach can be used to support different Tier levels of service on a pod-
by-pod basis. A Tier 1 pod can be supported with two busways, one over each row of
racks in a pod. Tiers 2–4 can be supported with four busways, with the second pair of
busways providing access to a redundant power source. This allows us to provide the
required level of service to our customers while saving the cost and inefficiency of
providing redundant power to every single rack. The pod illustrated in Figure 10 shows
four busways delivering redundant power to the pod.
Two busways above

each rack
Busway ‘cans’
Figure 10. A pod using four busways provides two redundant power paths that
support Tier 2-4 levels of availability.
We have found busway to work equally well whether used with the in-row cooling
approach illustrated in Figure 10, or with overhead cooling units as illustrated in
Figure 11.
Two busways above

each rack
Busway ‘cans’
Figure 11. Busway can be used with overhead cooling units as easily as with in-
row cooling.
21 Extending the Datacenter Lifecycle Sun Microsystems, Inc.
Chapter 5
Extending the Datacenter Lifecycle
The industry is seeing a new wave of datacenter construction that is driven by

premature obsolescence of existing datacenters that has resulted from the high
demand driven by the global consumption of technology and services. Facilities built
during the late 1990s were expected to last 15–20 years, and most have reached a
premature end of life. While these facilities may have sufficient space, their power and
cooling systems no longer can support the IT equipment in them.
The well-known reason for this problem is a mismatch between the datacenter’s
projected life span and the frequency and impact of equipment refresh cycles. If a
datacenter is built to last 15 years, then it must be built to accommodate equipment
refresh cycles that range from 3–5 years. For example, a high-density datacenter in the
year 2000 consumed around 3–4 kW per rack. Now 3–4 kW per rack is below the
industry average, and today’s high-density racks are exceeding 20–30 kW per rack!
In our datacenters at Sun, we have incorporated expansion capacity to support three-

year equipment refresh cycles. From day one, we have built a modular system that can
accommodate growth through the addition of new modules, and we have built extra
capacity in terms of additional equipment locations and power paths. This “future
proofing” of infrastructure required an additional investment of 10–15 percent in pipe
work for the mechanical/electrical systems. But this scalability enabled Sun's agility
and paid for itself within the first year.
This chapter follows the path of electricity from the utility to servers. It illustrates how
datacenters can be built to support the rapid change and increases in density that
occur over time without requiring major construction inside or outside of in the
datacenter itself.
Electrical Yard
The electrical yard is where the utility power is provided to the datacenter. When
building a datacenter, or retrofitting an existing building, it’s important to plan for the
future with the electrical utility: be sure that long-term power needs can be satisfied by
the utility and under what terms. In other words, design backwards from the amount of
power you will require or can receive to reach your end state. Build your electrical yard
with expansion in mind:
• Plan for additional transformers that connect to new utility feeds by preparing
concrete pads for them from the beginning.
• Design switchgear so that new feeds can be installed and cut over seamlessly.
• Configure switchgear so that there is room for growth. One technique is to use cross-
connect (cross tie) schemes to allow a bus to be split in half and powered by two
separate utility feeds in the future.
• Leave room in the switchgear for additional breakers, and run empty conduits —
sized to accommodate the largest possible wire — from the electrical yard to future
equipment locations in buildings. Trench and run conduit underground only once. At
Sun’s Santa Clara facility, we have installed conduit from the electrical yard to
datacenter spaces to accommodate growth within datacenters. We also have run
conduit from the electrical yard to existing office spaces that may become
datacenters in the future.
• The electrical yard is typically home to one or more backup generators; plan space for
growth by allowing space for additional generator capacity to support your end state.
Building Infrastructure
Each datacenter has a significant amount of electrical distribution infrastructure that
needs to be designed and sized with room for expansion:
• Prepare to expand UPS systems as electrical loads grow. This requires not only a UPS
design that is modular and can be expanded, but also floor space for additional UPS
modules and battery racks or flywheel cabinets.
• Use transformers, not PDUs. Our modular, pod-based design uses transformers in
separate rooms rather than PDUs on the datacenter floor. Reserve space for
additional transformers that growth in electrical loads will require.
• Design power distribution with room for expansion and preinstall conduit to support
the future growth. For example, if the goal is to support 100 percent growth over the
datacenter lifecycle, then design distribution boards to be 50 percent full on day one
to allow for easy expansion over time. Empty conduits to support future expansion
should also be run at the time of original construction to reduce the impact of the
expansion on the operating datacenter.
• Use industrial busway to provide power from the distribution panels to the pods. This
allows power feeds to be changed as power consumption within pods grows.
Pod Power Distribution

Each of our pods use a modular overhead busway to deliver power from the distribution
system to the equipment racks. Modular busway contributes to a datacenter’s agility by
making it possible to reconfigure equipment racks for completely different power
requirements — voltage, phase, and amperage — in a matter of minutes rather than
hours, days, weeks, or months. Busway also enables a datacenter’s expansion capacity
by supporting increasing power draws as equipment is replaced gradually over time.
Rather than sizing the overhead busway to the maximum expected power draw, we
have designed a strategy that can double the power provided to a pod in two phases —
allowing the power draw to increase by a factor of four over time.
For example, consider a pod with two busways, one above each row of racks (Figure 12).
• Our initial configuration is to jumper the two busways together through a conduit
bridge and provide a single power feed for each pair of busways.
Phase One:
One feed for two
busways using
jumper
Phase Two:
One feed for
each busway
Phase Three:
One feed for
each split
busway
Figure 12. As a pod’s power draw increases, its supply can be increased in two
phases, each phase doubling the previous power capacity.
• The pod’s power feed can be doubled by removing the jumper and providing a power
feed to each busway.
• The pod’s power feed can be doubled again by splitting each busway in half and
providing a separate feed for each half.
This expansion strategy avoids the up-front cost of purchasing high-amperage busway,
while providing a way to support a four-fold increase in power consumption over time
with minimal disruption or construction costs. This level of expansion in a traditional
datacenter electrical distribution system would require a complete repowering of the
datacenter space along with significant downtime and expense. Keep in mind that this
strategy requires your conduit to be larger and pathways to be pre-defined to support
the end state quantity and size of conductors. Choosing one size busway that supports
the final load will allow the re-use of cans over the functional life of the datacenter.
These busways also support higher voltages as discussed in the previous chapter.
24 Sun’s Electrical Design At Work Sun Microsystems, Inc.
Chapter 6
Sun’s Electrical Design At Work
The design principles that we have discussed to this point are embodied in our
datacenter designs worldwide. This chapter provides a top-to-bottom overview of our
Santa Clara, California datacenters followed by a discussion of Sun’s newest facility in
Broomfield, Colorado, and an overview of Sun’s use of Sun Modular Datacenters at the
Santa Clara site.
Santa Clara, California

Our electrical design tour of our Santa Clara datacenter is through a series of one-line
diagrams. These diagrams illustrate the various components and how they are
connected without the detail of showing each wire for each phase — hence the term
“one line” diagram.
Electrical Yard
Our Santa Clara project, one of the largest datacenter consolidation efforts in Sun’s
history, allowed us to construct an entirely new electrical service yard to supply power
to new datacenters and buildings.
This electrical yard receives two 12 kV power feeds from Silicon Valley Power, each of
which allows us to draw 4.5 MVA of continuous power. To ensure that we have power to
grow over time, we have negotiated with the local utility to provide up to 24 MW of
additional power to the electrical yard through additional 4.5 MVA feeders.
A set of medium-voltage switchgear allows the additional feeds to be incorporated into

the electrical yard without disrupting existing service. The medium-voltage switchgear
feeds three similar sets of 480V switchgear through step-down transformers, one of
which is illustrated in Figure 13.
The switchgear supplies 480V power to office buildings and datacenters on the Santa
Clara campus. The switchgear includes breakers for existing spaces and for future
expansion, with conduit runs already installed to the future spaces.
We accommodate expansion at this level of the distribution hierarchy by supporting

each of the three sets of switchgear with a single transformer. Each set of switchgear is
configured with two halves that are connected with a tie breaker that is currently
closed. When the time comes to double the power supply, a second transformer will be
connected to a new utility power feed, and the tie breaker will be opened. Now each
half of the switchgear has a 3 MW continuous operating capacity.
Existing transformer supplies Future transformer allows

switchgear with 480/277V a doubling of capacity
Tie breaker remains closed

until future transformer is installed
Breakers, future breaker locations, and power feeds to buildings and datacenters
Figure 13. Each of three sets of 480V switchgear can support a doubling of power by adding a second transformer and
opening a tie breaker.
A photograph of the Santa Clara switch yard gives a visual perspective (Figure 14).
• In the foreground are three rows of similar switchgear. From right to left is the 12 kV
switching upstream of the transformer. Next is the transformer, followed by the 480V
switchgear. Note the empty space at the end of two of the rows to accommodate an
additional transformer and switchgear. One already has been installed.
• Behind the 480V switchgear is the medium-voltage, 12 kV switchgear that
accommodates existing and future utility feeds.
• Also visible in the photograph are two diesel backup generators.
Load
Bank Generators
12 kV Utility Feeds
and Switchgear
Expansion 480V
Locations Switchgear
12kV Switchgear
and Transformers
Figure 14. A view of the Santa Clara electrical yard shows the 12 kV to 480V
transformers and switchgear in the foreground.
Backup Generators
Because the majority of our datacenters are for research and development efforts, the
pods in them are designed to Tier 1–2 specifications. For 20 percent of the load, we
deploy UPS to support more critical operations. As operational risks increase, we deploy
generators behind the UPS to provide continuous power.
Figure 15 shows a portion of the one-line diagram showing our backup generator
configuration. Two backup generators are currently installed, with space, conduit, and
breaker positions for a third. This provides 2 MVA of power today, with 3 MVA when the
third generator is installed. The generators provide power to a set of paralleling
switchgear that combines the generator outputs. A 1000 kW load bank is installed to
provide a full load when testing the generators. Regular 100 percent load testing of
generators is important to make sure the generators are operating correctly and always
ready to accept load.
The paralleling switchgear currently feeds one 800 kVA UPS with provisions for future
growth. In the event of a failure, the feed is switched through automatic transfer
switches (ATS). Two automatic transfer switches (ATS) switch the power feed for the UPS
from utility power to generator power in the event of a power failure. A third ATS
supplies power to keep the cooling system operating.
Feed from
480V
switchgear
Existing two
generators
Automatic
transfer
switch
Future
generator
Supply to UPS
Figure 15. Our Santa Clara electrical yard includes three generators that provide
backup power to UPS.
Battery-Based UPS
Our Santa Clara site is equipped to accommodate a fourth UPS to bring the total to
3.2 MVA. The site uses three 800 kVA UPS capable of providing up to 2.4 MVA of
uninterruptible power. We chose APC Symmetra MW units for the Santa Clara site for
their modularity and efficiency. The APC UPS is unique in that it enables the UPS system
to grow in 200 kW increments. At the Santa Clara site, two of the three UPS are sized
with 800 kW frames with only 400 kW of power modules installed, right sizing the UPS
to our day one needs. As our loads grow, we will add additional power modules that
grow the UPS to its full 800 kW capacity with only minimal construction in the UPS
room to match battery capacity to UPS capacity.
Power supply from

automatic transfer
switch
To transformers and
distribution panels
Overhead busways providing uninterruptible

power to pods
Figure 16. An 800 kVA UPS supplies power to three transformers, each of which
supplies 208/120V uninterruptible power to a set of busways.
The UPS illustrated in Figure 16 takes its input power from the automatic transfer
switch illustrated in Figure 15. Its output feeds a distribution panel that connects to
three transformers that step down the 480/277V power to the 208/120V power that
feeds the busways. One transformer is illustrated in Figure 16. It supplies power to a
distribution panel that feeds a set of 225A and 400A busways inside the datacenter
spaces.

For the majority of our pods that only require Tier 1 service, a single 480V feed from the
switchgear supports multiple transformers that bring the voltage down to the 208/120V
supplied to pods. The example shown in Figure 17 illustrates pairs of busway jumpered
together as described in “Pod Power Distribution” on page 22.
Supply from
480V switchgear
480V transformer
Figure 17. Power is distributed to non-critical loads through transformers that

feed 208/120V power to overhead busways.
Metering and Power Management

One of the most important aspects of datacenter electrical design is an integrated
metering and energy management system. These systems are critical because they
provide the ability to evaluate how power is being used in the datacenter, power
quality or disturbance events, how efficiently the datacenter is operating, and what
impact any energy-saving measures have on overall efficiency.
As we discussed in the Sun BluePrints Series article Energy Efficient Datacenters: The
Role of Modularity in Datacenter Design, power usage effectiveness (PUE) is the ratio of
total facility power to the IT equipment load. In order to obtain this ratio, and to be
able to observe energy consumption of the various datacenter subsystems, appropriate
metering capabilities must be placed at strategic locations cascaded through the
electrical system. In our Santa Clara datacenters, we have metering and power
management in place at the following locations:
• The utility switchgear
• All major switchgear
• All UPS, ATS, and generators
• Each overhead busway
• Each rack is equipped with a rack-mounted power distribution unit (rPDU) that
monitors overall power consumption and also provides remote on/off per-outlet
control. These rPDUs can be monitored over the network, providing fine-grained data
on IT equipment loads.
Broomfield, Colorado
Our Broomfield, Colorado location is the site of our most recent datacenter
construction project. Spurred by the need to consolidate the former StorageTek site into
our Broomfield campus, there are two significant differences between this datacenter’s
electrical design and the Santa Clara design described above:
• No electrical yard. In Broomfield, the utility provides 480V power through a set of
primary voltage transformers located at the campus perimeter. For redundancy
purposes, these transformers are loop fed through utility company-owned switch
cabinets. This is an example of where the utility owns the primary voltage feeders,
transformers, and switch cabinets.
• Flywheel UPS. The Broomfield campus is the largest installation of rotational UPS
at Sun. The double-conversion UPS used in Broomfield supplies 900 kVA of backup
power, sufficient to power critical loads between the time of a power failure and
diesel generator startup. The UPS pictured in Figure 18 provides 15 seconds of
backup power to critical loads, sufficient to cover the 8 seconds between a power
failure and diesel generator startup.
Main Power Distribution

Power to datacenter equipment located on floors 1–4 of the Broomfield building
originates from utility transformers located at one end of the building. One
transformer feeds power to a 4000A, 480V main distribution board that serves
equipment loads on the first and fourth floors. The second transformer feeds power to
a 5000A, 480V main distribution board that serves equipment loads on the second and
third floors. The two electrical services combined are capable of delivering 7.1 MW to
datacenter equipment on floors 1–4. A transient voltage surge suppression device is

installed at each service entry point to protect the building electrical system from
voltage surges generated outside the building.
Figure 18. The rotational UPS used in Broomfield supplies 960 KW kVA of backup
power, sufficient to power critical loads between the time of a power failure and
diesel generator startup.
The main switchboards are equipped with spare circuit breakers and spaces with
conduit runs to the secondary electrical rooms located on the second and fourth floors
for future expansion. A one-line diagram illustrating this design is illustrated in
Figure 19.
The central plant transformers are fed from a new utility transformer located northeast
of the building. It feeds power to a 2500A, 480V main distribution board which serves
mechanical loads supporting the datacenter. The electrical service is capable of
delivering 2.2 MW to mechanical load in the central plant.

The two main distribution switchboards distribute 480V power to a series of 225 kVA
and 500 kVA high-efficiency dry-type transformers located in secondary electrical rooms
on each floor. We sized these transformers so the would have a much greater load than
the industry standard of 30 percent. High-efficiency transformers were selected because
at a higher load factor, efficiency tends to remain constant and at a higher level. In
contrast, the efficiency of standard dry-type transformers decline sharply beyond the 30
percent load factor.
Figure 19. Broomfield datacenter main power distribution
Each transformer feeds a large 120/208V distribution panelboard also located in the
secondary electrical rooms. A transient voltage suppression system (TVSS) device is
installed at each distribution panelboard which protects sensitive datacenter
equipment downstream from voltage surges internally generated within the building.
The distribution panelboards are equipped with spare circuit breakers, spaces, and
conduits routed to locations of future datacenter panelboards. The distribution
panelboards feed 120/208V panelboards that distribute power to all 225A busways in
the datacenter. The diagram shown in Figure 20 illustrates how pod power is distributed
from a typical secondary electrical room to datacenter busway.
Figure 20. Pod power is supplied from a 500 kVA transformer located in a
secondary electrical room.
Rotational UPS
A 960 kW/1200 kVA, 480V centralized rotational UPS provides approximately 15 seconds
of back-up power for critical datacenter equipment located in the mainframe, server
room and one pod in the CTC lab located on the second floor. The UPS has four
modular 240 kW/300 kVA flywheels in which three of the four flywheels are needed to
support the critical loads. This N+1 design approach provides redundancy where one
flywheel can be completely taken offline for maintenance purposes. The UPS is
equipped with a full-maintenance wrap-around bypass so that it can be taken out of
service without interrupting power to critical loads. At 97 percent, the efficiency of the
rotational UPS is higher than a traditional battery-based system due to the fact that the
two electrical conversions do not occur under normal operating conditions — only
when the flywheel is carrying the critical load.
The UPS illustrated in Figure 21 takes its input and maintenance by-pass power from a
1600A, 480V emergency distribution panel. This distribution panel is equipped with a
TVSS device which protects sensitive datacenter equipment downstream from voltage
surges internally generated within the building. The UPS output feeds a 1600A, 480V
main distribution panel that distributes power to a large 225 kVA and 500 kVA high-
efficiency dry-type step-down transformers. The transformers feed 120/208V
panelboards that distribute power to all 225A busway in the datacenter.
Back-up Generator
Emergency back-up power to the datacenter and central plan is served from an existing
1250 kW/1562.5 kVA diesel generator located outside the building. The back-up
generator feeds three emergency main distribution boards located in main electrical
room which supports life safety loads and critical secondary mechanical equipment
that supports cooling to critical datacenter and the rotational UPS. It also feeds an
emergency main distribution board that supports critical primary mechanical
equipment in the new central chiller plant.
Metering and Power Management

The Broomfield datacenter has metering and power management throughout. Three-
phase energy and power measurement with data logging and power quality analysis
can be obtained locally from each meter location or remotely from the campus central
control room or off campus from a Web browser. Metering has been installed at the
following locations:
• The two 480V main distribution switchboards are monitored to determine total power
consumption for the four floors of the datacenter.
• Each branch distribution feeder in the 480V main distribution switchboards is
monitored.
Figure 21. The Broomfield rotational UPS are integrated into the power design so
that any one of them can be taken out of service without disrupting loads.
• Input and output distribution panelboards monitor the flywheel UPS power and
quality.
• Each overhead busway incorporates power monitoring, allowing a per-pod evaluation
of power consumption of equipment.
• Each rack is equipped with an rPDU that monitors overall rack power consumption.
These in-rack PDUs can be monitored over the network, providing fine-grained data
on IT equipment loads.
Sun Modular Datacenter

For more information on the Sun Modular Recent expansion at our Santa Clara site has used two Sun Modular Datacenters (Sun
Datacenter, please visit MDs). This unit is a self-contained containerized datacenter configured in an enhanced
http://www.sun.com/sunmd
20-foot shipping container with integrated high-efficiency cooling and power
infrastructure. Sun MD is available in 2 configurations: the original eight-rack S20
model, and a new seven-rack D20 edition. Both support a 200 kW capacity in only 160
square feet of space. The Sun MD can typically be deployed much more quickly than a
traditional datacenter; all that is required is to provide appropriate chilled water,
networking and power. This section describes the product power features and an
overview of the external power requirements for deploying a Sun MD.
Power Overview
The Sun MD is available in two different electrical configurations. The low-voltage
version operates on 190 to 220 VAC, three-phase, 50 or 60 Hz electrical service. The high-
voltage version operates on 380 to 415 VAC, three-phase, 50 or 60 Hz electrical service
Power Distribution in the Sun MD

Power is routed into the Sun MD through two electrical feeder entrances, one on each
side of the container, and can be provided by one or two external power sources.
The Sun MD has a primary and a secondary internal power distribution panel
corresponding to the two electrical feeder entrances (Figure 22). The internal power
panels provide redundant power to each of the racks. Each panel supplies power
through a set of circuit breakers to each of the racks and to each of the HX fan control
units in the Sun MD.
Supplying Power to the Sun MD

Power can be supplied to a Sun MD through a commercial utility, an on-site utility
power generator, or a standalone power generator in any combination. There are
several factors to consider in designing and deploying external power infrastructure for
the Sun MD, including requirements for redundancy, uptime, mobility, and existing site
power infrastructure.
Figure 22. Sun Modular Datacenter power distribution panels. This view of a low-
voltage unit is from the back of the Sun MD, with A panel on the left and the B
panel on the right.
Uptime is typically a primary concern in the deployment of a datacenter. In order to

provide the highest levels of service, a means of maintaining power in the event that an
external power source fails is highly recommended. This involves a design using an
automatic transfer switch to switch from the primary source of power to a secondary or
backup source of power such as a generator. A UPS is used to supply power to the Sun
MD during the time interval when the primary power fails and when the secondary
source comes online.
While the actual configuration of the electrical power distribution system will vary, a
typical external power infrastructure will have many of the following components:
• Commercial electric utility service
• Site distribution panel
• Automatic transfer switch (if required)
• Chiller/heater sub-panel (if required by building code)
• Transformer (if required to match line voltage)
• Generator (optional)
• Uninterruptible power supply (optional)
• UPS power distribution panel (if required by building code)
When we deployed Sun Modular Datacenters at our Santa Clara site, we utilized the
flexibility of the electrical infrastructure described in previous sections and literally
snapped in the modular datacenters. We installed high-efficiency transformers outside
next to each container to convert 480V to 208V. Installing the transformers outside
eliminated the need to provide cooling for the transformers. The containers in Santa
Clara are installed downstream of our UPS system which is currently providing power to
both the Sun MDs as well as our building-based datacenters.
37 Summary Sun Microsystems, Inc.
Chapter 7
Summary
A datacenter’s electrical design plays a critical role whose influence ranges from
protecting real estate investments to reducing day-to-day power consumption.
Traditional electrical designs can limit growth, reduce business agility, and add, rather
than subtract, from skyrocketing utility bills. In contrast, the datacenter designs that
we have used at Sun facilitate growth, increase agility, and provide significant savings
in energy consumption. Rather than being a static aspect of datacenter design that
hampers business operations, our datacenter electrical design flexibility helps to better
align our datacenters and business operations.
Our modular electrical design provides greater flexibility that helps our datacenters
better support our business operations. Our modular design allows our engineering
organization to roll in and power a next-generation server in a matter of minutes. The
same modularity that allows us to support different Tier levels within the same
datacenter also saves costs by delivering only the required amount of redundancy.
Our philosophy of preparing for future growth from day one has a dramatic impact on
the datacenter lifecycle. By building a datacenter electrical system that can support
future generations of IT equipment, we have better matched equipment refresh cycles
with our planned datacenter life cycles. This not only helps prepare our business for the
future, it helps to protect the immense investment in real estate by increasing the
longevity of our datacenters. This agility has become a competitive advantage for Sun.
Ultimately, a datacenter electrical system must deliver power from the electric utility to
each piece of IT equipment. Taking advantages of the opportunities to increase
efficiency along the way, making sure not to overprovision for day-one capacity, and
through careful monitoring, we have been able to achieve about a six percent greater
efficiency that amounts to an annual savings of $33,638 in one datacenter alone. From
energy-efficient transformers to appropriate implementations of redundancy,
datacenter electrical designs offer abundant opportunity to reduce power consumption
and overall operating costs.
One of the best aspects of our datacenter electrical design principles is that they are
straightforward and easy to implement, so organizations outside of Sun can reap
similar benefits. While our designs initially seemed radical compared to traditional
designs, the choices that we have made are now embodied in mainstream practices
at Sun. These practices are also starting to become prevalent in the industry.
About The Authors

Michael Ryan
Mike Ryan is a Senior Staff Engineer in Sun’s Global Lab & Datacenter Design Services
organization. Mike graduated from San Jose State University with a degree in
mechanical engineering. He is a licensed professional mechanical engineer and has
been involved in the design, construction, and operation of mission-critical facilities for
the past 18 years. His experience spans numerous industries such as gas turbine co-
generation, semi-conductor manufacturing, and mission-critical datacenters. For the
last seven years he has focused on the design and operation of mechanical and
electrical infrastructure systems supporting high-availability datacenters. Mike joined
Sun as a Staff Engineer for the GDS organization in March of 2006. Mike is the primary
author of the GDS physical standards and was the lead design engineer for the Santa
Clara, California datacenter project, one of the largest and most complex in Sun's
history. He also provided supporting engineering services for the Broomfield, Colorado
datacenter project. Mike's design work supports the Act portion of Sun's Eco strategy,
and was showcased at Sun's Eco Launch in August 2007.
Mike is a member of ASHRAE (American Society of Heating, Refrigeration, and Air-
conditioning Engineers). He also participates in the Data Center Pulse and the Critical
Facilities Round Table group. Mike was the recipient of Sun's prestigious Innovation
Award presented by Sun's CEO, Jonathan Schwartz ad CTO, Greg Papadopoulos in July of
2008.
Dean Nelson
Dean is the Senior Director of Global Lab & Datacenter Design Services (GDS) in the
Work Environments (WE) business unit of Sun Microsystems. The GDS organization
bridges the gap between Facilities, IT, and Engineering, and is responsible for managing
Sun’s multi-billion dollar global technical infrastructure portfolio including datacenter
design, standards and strategy. The GDS work resides in the “Act” portion of Sun’s Eco
strategy, and was showcased at Sun’s Eco Launch in August, 2007, and the Colorado
Data Center grand opening in January, 2009.
Dean has been in the technology industry for 19 years, of which 16 have been with Sun.
He spent four years in Sun manufacturing in roles ranging from component-level debug
to managing quality. This included helping the drive to achieve ISO 9002 certification.
Dean joined the Sun Engineering community in 1993. He led systems and network
administration support for some of Sun’s largest and most complex R&D lab
environments.
Dean left Sun in 2000 to join a networking startup company called Allegro Networks. At
Allegro, he built a world-class QA team, state of the art global R&D lab environments
and fully integrated automation system.
In 2003, Dean returned to Sun joining the newly formed N1™ Software organization. He
orchestrated the integration of Terraspring and Center Run R&D labs and the merger of
Sun™ Cluster software into N1 technology. In mid-2003, Dean took over management all
of the N1 organization’s R&D labs, build engineering, automation, and capital budget
responsibilities world-wide. In 2004, he became a leading member of the Global Lab &
Datacenter Design Services (GDS) team tasked with creating a strategy to standardize
Sun’s multi-billion dollar technical infrastructure portfolio. He was the architect of the
GDS operating model and lead design engineer for lab and datacenter projects world-
wide. In 2006, he became the GDS Director. Since taking over the group, Dean has
delivered the GDS strategy, executing the largest technical infrastructure consolidation
in Sun’s history compressing the 1.4 million square foot portfolio by 48 percent, a $250
million dollar investment.
Dean holds numerous industry technical and business board positions including
Founder of Data Center Pulse (http://datacenterpulse.org) - an exclusive datacenter
owner community. Dean was featured in Contrarian Minds, a focus on the engineers,
scientists and dreamers of Sun Microsystems. Dean was also the recipient of Sun’s
prestigious Innovation Award presented by Sun’s CEO, Jonathan Schwartz and CTO,
Greg Papadopoulos in July of 2008. Dean lives in Northern California with his wife and
daughter.
Serena Devito
Serena DeVito is a Datacenter Design Engineer in Sun’s Global Lab & Datacenter Design
Services organization. Serena joined Sun over ten years ago and has spent the bulk of
her career as a system administrator within the IT organization where she designed,
deployed, and supported critical infrastructure for the company. She then moved to the
Software organization and managed the beta testing project for the Sun Cluster
product group. Serena acted in a customer capacity running early releases of Sun
Cluster software on production home directory clusters with over one hundred active
users as well as clustered web services. Serena identified, documented, and filed bugs
while supporting one of Sun's largest and complex engineering lab environments.
Serena has been a core member of the GDS team since 2004. In December 2005 she
joined the GDS organization to drive lab and datacenter consolidation projects world-
wide. Along with building numerous labs and datacenters in the Americas, Serena was
the technical lead and customer engagement engineer on the Santa Clara, California
datacenter consolidation project, one of the largest and most complex datacenter
consolidations in Sun's history. For this work, Serena was the recipient of Sun's
prestigious Innovation Award presented by Sun's CEO, Jonathan Schwartz ad CTO, Greg
Papadopoulos in July of 2008.
Serena is a graduate of the University of Adelaide (Australia) with a Bachelors of
Architectural Studies and a focus in Computer-Aided Design. She is a Sun Certified
Solaris Administrator for the Solaris™ 7, 8 and 9 Operating Systems. She also loves
traveling and riding horses.
Ramesh KV
Ramesh has been in the technology industry for over 13 years and has held a variety of
technical & managerial roles since joining Sun in 2001.
Ramesh began his career as a customer support engineer at Kumdev computers, where
his clients included companies in the manufacturing, pharmaceuticals, telecom and IT
industries. He later joined SAS (now know as SASKEN) as a Systems Engineer, where he
supported the NORTEL infrastructure. He then moved on to TATA Elxsi as a Design and
Development Specialist, where he was involved in various key infrastructure projects.
Ramesh spent the initial years at Sun working for the RPE organization as Global Lab
Manager, where he drove the global lab model and designed, deployed, and supported
critical infrastructure across USA, EMEA & India. In early 2006, Ramesh moved into the
role of IT/ Lab Council head for the India Engineering Centre (IEC) reporting to the site
VP as well as the Global Lab & Datacenter Services (GDS) lead for the APAC region. In
this role Ramesh led the physical and technical design for the Bangalore, India
Datacenter consolidation project and supported all APAC lab and datacenter projects. In
2007 Ramesh became a Regional Work Place Manager (RWPM) covering Asia South and
India in addition to his GDS responsibilities. In this role he has been responsible for all
workplace resources activities across Asia South & India, including portfolio
management, real estate services & facilities management. Ramesh has been a critical
member of the GDS technology team since 2004 and has designed multiple labs and
datacenters across Asia Pacific. Ramesh is the recipient of a research disclosure on
“Web Based Multi OS Installation,” PC Quest IT Implementation project of the year
award in 2007 and has received 12 internal awards globally including Sun's leadership
award in 2005 and 2007. Ramesh was the recipient of Sun's prestigious Innovation
Award presented by Sun's CEO, Jonathan Schwartz ad CTO, Greg Papadopoulos in July of
2008. Helives in Bangalore, India with his wife and daughter.
Petr Vlasaty
Petr has been working in the IT industry for more than 12 years. His experience that
started as an implementation engineer and project coordinator in 1996 has grown into
a variety of people, project, process and service management areas of datacenter
operations and technical support services. Petr worked for ANECT to define and develop
the System Support and Maintenance department responsible for systems
implementations and support for key customers including the Czech Ministry of
Finance, Commercial Bank, and others. Petr formed a 12-person team that defined
internal and external processes, established contracts, and represented support
services to customers. In November 2004, Petr joined the Sun Software organization as
R&D Datacenter Manager and Team Lead for the Prague Datacenter Management
Team. Petr designed and implemented two internal datacenter projects in Prague in
partnership with the GDS team. He applied the GDS processes and standards for the
new Prague datacenter expansion that was completed in June 2006. Due to Petr's
outstanding performance in Prague, he was assigned as the technical lead for the
Louisville, Colorado consolidation project in August of 2006. He spent four months in
Colorado developing the very detailed and complex plan. Petr joined the GDS
organization full time in January of 2007. He is responsible for the Europe, Middle-East
and Africa (EMEA) region and continues to act as technical lead for the Colorado
datacenter consolidation project, the largest and most complex in Sun's history.
Petr is a Staff Engineer, Datacenter Architect, and the GDS Technical Infrastructure Lead
for the EMEA region. He loves sports like mountain hiking (year round, especially
winter) and rock climbing. Petr lives with his wife Vera and 10 month-old son Erik in
Prague, Czech Republic.
Petr is a CCSE (Check Point Certified Security Engineer), CCSA (Check Point Certified
Security Administrator) and SCSA (Sun Certified Solaris Administrator). Petr was the
recipient of Sun's prestigious Innovation Award presented by Sun's CEO, Jonathan
Schwartz ad CTO, Greg Papadopoulos in July of 2008.
Brett Rucker
Brett Rucker is a Staff Engineer in Sun’s Global Lab & Datacenter Design Services
organization where he focuses on physical environment issues. Brett has been in the
Corporate Engineering, Construction and Management business for 25 Years. A
Graduate of the University of Colorado, Boulder, and a licensed Professional Engineer,
Brett began his career with Ball Aerospace Systems Division, a rich opportunity that
gave him complete design and management responsibility for all construction projects.
He supervised products including building additions, clean rooms, X-Ray rooms,
datacenters, laboratories, anechoic chambers, secure areas, gallium arsenide facilities,
high elevation satellite dish structural mounts, RF shielded rooms, and office,
conference, and customer show areas. Highlights of Brett’s time at Ball include design
and construction of large secured program area in 7-days to support critical Corp.
project, stick build design and construction of MAMA Lab Clean Room facility for 1/3
market cost (still in operation today with one of Ball's premier products), metal finish
Area with a $30,000 per year energy savings in 1992 base, and creation of the Facility
Engineering Program at Ball Aerospace which grew to take over Ball corporate facilities.
In 1992, Brett joined a newly forming engineering group at StorageTek to address

facility backlog and project issues. Brett and his team created corporate building
specifications, drawing standards, and quickly eliminated all backlog. Brett drove the
high tech areas, and HVAC infrastructure as well as general projects. He represented
StorageTek on refrigerant conversion and reported on this project at an ASHRAE
function. Brett became manager of the New Project Management Group in 2000
directing projects at headquarters and in the field. His responsibilities soon included
StorageTek’s energy management program, which deployed processes and delivered
high-quality, low-cost projects with consistent high metrics with limited staff support.
This group lead all project designs, construction and project budgets. The energy
management group delivered ISO Certification, installed monitoring systems, metrics,
monthly log and issue reviews, best payback energy projects and commodity contracts
that beat the market consistently. Brett joined Sun’s GDS team in 2006 as a design
engineer and is the physical design lead for the largest and most complex datacenter
consolidation in Sun's history, the StorageTek move from Louisville, to Broomfield,
Colorado.
Brett is a member of ASHRAE, and AEE, and he was the recipient of Sun's prestigious
Innovation Award presented by Sun's CEO, Jonathan Schwartz ad CTO, Greg
Papadopoulos in July of 2008. Brett is active in developing youth programs and
coaching his daughter’s competitive fast-pitch team with great support from his wife
and two daughters.
Brian Day
Brian Day is a Senior Program Manager in Sun’s Global Lab & Datacenter Design
Services organization. Brian has been working in the technology industry for over 10
years, of which more than five have been with Sun. He spent over two years in Sun’s
Systems group, developing automation programs for server, storage, and software QA
functions. This work included testing, configuring, and administering some of Sun's
largest systems at the time, including the Sun Enterprise™ 10000 and Sun Enterprise
6500 servers, and Sun StorEdge™ D1000 storage device. Brian then served in a similar
role for iPlanet™ software, developing server side Java™ technology-based tools that
automated core team processes (ticketing system, equipment tracking, on-call
notification).
Brian left Sun in 2000 to pursue his MBA full time. He then joined Adobe Systems in
2001 as a Program Manager, leading the development of several Digital Imaging plug-
ins for Adobe Acrobat. In 2003 he took over full time program management of the
Photoshop Elements product, releasing an all new version (3.0) in the fall of 2004. This
included the coordination of all key product teams (engineering, QA, marketing,
localization, documentation) and ensuring a successful release for the critical holiday
shopping season. Brian returned to Sun in early 2005 to program manage the GDS
initiative. As Senior Program Manager & Chief of Staff for GDS, Brian is responsible for
first engagement with new customers, ensuring world-wide GDS projects are resourced,
deliverable expectations are set, and commitments honored. Brian's work in GDS has
significantly contributed to Sun's modular design and the teams success. Brian was the
recipient of Sun's prestigious Innovation Award presented by Sun's CEO, Jonathan
Schwartz ad CTO, Greg Papadopoulos in July of 2008. Brian holds a Bachelor of Science
in Computer Science and a Masters of Business Administration.
Acknowledgments
The authors would like to thank Steve Gaede, an independent technical writer and
engineer, for his probing questions, deep insights, and ability to create a coherent story.
Steve is a frequent contributor to Sun Microsystems technical documents, including
many Sun BluePrints articles. He is a member of ACM and USENIX, and is active in the
Boulder Colorado professional community, having been a coordinator of the Front
Range UNIX Users Group since 1984.
Thanks to Mark Monroe for providing input on power supply efficiency trends, and to
James Monahan for his detailed review comments. Thanks also to the Sun MD team
members who spent time contributing to this paper, including Maurice Cloutier, Liz
From, Brian Kowalski, and Bob Schilmoeller.
References
More information on Sun’s energy-efficient datacenters can be found at:
http://www.sun.com/datacenterdesign.
Ordering Sun Documents

The SunDocsSM program provides more than 250 manuals from Sun Microsystems, Inc.
If you live in the United States, Canada, Europe, or Japan, you can purchase
documentation sets or individual manuals through this program.
Accessing Sun Documentation Online

The docs.sun.com web site enables you to access Sun technical documentation
online. You can browse the docs.sun.com archive or search for a specific book title or
subject. The URL is
http://docs.sun.com/
To reference Sun BluePrints OnLine articles, visit the Sun BluePrints OnLine Web site at:
http://www.sun.com/blueprints/online.html
Energy Efficient Datacenters: Electrical Design On the Web sun.com
Sun Microsystems, Inc. 4150 Network Circle, Santa Clara, CA 95054 USA Phone 1-650-960-1300 or 1-800-555-9SUN (9786) Web sun.com
© 2009 Sun Microsystems, Inc. All rights reserved. Sun, Sun Microsystems, the Sun logo, BluePrints, iPlanet, Java, N1, Solaris, StorEdge, Sun Enterprise, Sun Fire, and SunDocs are trademarks or registered trademarks
of Sun Microsystems, Inc. or its subsidiaries in the U.S. and other countries.Information subject to change without notice. Printed in USA 3/2009

Electrical Design of Energy Efficient Data Centers PDF

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Electrical Design of Energy Efficient Data Centers PDF

Uploaded by

Copyright:

Available Formats

ENERGY EFFICIENT DATACENTERS

Sun BluePrints™ Online

Traditional Datacenter Electrical Design . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .3

Increasing Electrical Distribution Efficiency . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .8

Flexibility Through Modularity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17

Extending the Datacenter Lifecycle . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21

Sun’s Electrical Design At Work. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24

About This Article

From an electrical design standpoint, a datacenter is a hierarchy of electrical devices

FACILITY TRANSFORMER SWITCHGEAR UPS 480V PDUs RACKS

MORE THAN ONE OF EACH DEVICE INSTALLED AT EACH LAYER

Utility Power and Transformer

At Sun, a tie breaker scheme allows power supplied to buildings to be doubled. By

• Power distribution units (PDUs) typically consist of multiple distribution panel

Rack Power Distribution

Datacenter Tier Levels

Sun’s Approach to Electrical Design

Alternatives to Battery-Based UPS

A low-maintenance, more environmentally friendly approach uses the rotational inertia

Equipment Power Supplies

208V Vs. 230V

Alternating Vs. Direct Current

Power Inefficiencies 380 VDC (typical)

AC/DC DC/AC AC/DC DC/DC

Lead-Acid Batteries UPS PDU Server

We believe that the benefits demonstrated by this experiment are superseded by

Appropriate Levels of Redundancy

Choosing Tier Levels

Using Distributed Redundancy

N+1 Redundancy (Parallel Redundant)

As with all approaches to UPS redundancy, distributed redundancy requires adjusting

Datacenter Power Distribution

Hard-Wired Power Distribution

Power Distribution Units

The Benefits of Modular Busway

Two busways above

Two busways above

The industry is seeing a new wave of datacenter construction that is driven by

In our datacenters at Sun, we have incorporated expansion capacity to support three-

Pod Power Distribution

Santa Clara, California

A set of medium-voltage switchgear allows the additional feeds to be incorporated into

We accommodate expansion at this level of the distribution hierarchy by supporting

Existing transformer supplies Future transformer allows

Tie breaker remains closed

Power supply from

Overhead busways providing uninterruptible

Pod Power Distribution

Figure 17. Power is distributed to non-critical loads through transformers that

Metering and Power Management

Main Power Distribution

datacenter equipment on floors 1–4. A transient voltage surge suppression device is

Pod Power Distribution

Figure 19. Broomfield datacenter main power distribution

Metering and Power Management

Sun Modular Datacenter

Power Distribution in the Sun MD

Supplying Power to the Sun MD

Uptime is typically a primary concern in the deployment of a datacenter. In order to

About The Authors

In 1992, Brett joined a newly forming engineering group at StorageTek to address

Ordering Sun Documents

Accessing Sun Documentation Online