Professional Documents
Culture Documents
System Administration
Data Centers
Topics
Data Center: A facility for housing a large
amount of computer or communications
equipment.
1. Racks
2. Power
3. PUE
4. Cooling
5. Containers
6. Economics
1
Google DC in The Dalles
2
Data Center is composed of:
• A physically safe and secure space
• Racks that hold computer, network, and
storage devices
• Electric power sufficient to operate the
installed devices
• Cooling to keep the devices within their
operating temperature ranges
• Network connectivity throughout the data
center and to places beyond
3
Racks: The Skeleton of the DC
• 19” rack standard
– EIA-310D
– Other standard numbers.
• NEBS 21” racks
– Telecom equipment.
• 2-post or 4-post
• Air circulation (fans)
• Cable management
• Doors or open
Rack Units
Rack Sizes
http://www.gtweb.net/rackframe.html
4
Rack Purposes
Organize equipment
– Increase density with vertical stacking.
Cooling
– Internal airflow in rack cools servers.
– Data center airflow determined by
arrangement of racks.
Wiring Organization
– Cable guides keep cables within racks.
Rack-Mount Servers
1U
4U
5
Blade Servers
Buying a Rack
Buy the right size
– Space for servers.
– power, patch panels, etc.
Be sure it fits your servers.
– Appropriate mounting rails.
– Shelves for non-rack servers.
Environment options
– Locking front and back doors
– Sufficient power and cooling.
– Power/environment monitors.
– Console if needed.
Space
Aisles
Wide enough to move equipment.
Separate hot and cold aisles.
Hot spots
Result from poor air flow.
Servers can overheat when average
room temperature is too low.
Work space
A place for SAs to work on servers.
Desk space, tools, etc.
Capacity
Room to grow.
6
Data Center Power Distribution
http://www.42u.com/power/data-center-
power.htm
Standby UPS
• Power will be briefly interrupted during switch
• Computers may lockup/reboot during interruption
• No power conditioning
• Short battery life
• Very inexpensive
http://myuninterruptiblepowersupply.com/toplogy.htm
7
Online UPS
• AC -> DC -> AC conversion design
• True uninterrupted power without switching
• Extremely good power conditioning
• Longer battery life
• Higher price
http://myuninterruptiblepowersupply.com/toplogy.htm
8
The Power Problem
• 4-year power cost = server purchase price.
• Upgrades may have to wait for electricity.
• Power is a major data center cost
– $5.8 billion for server power in 2005.
– $3.5 billion for server cooling in 2005.
– $20.5 billion for purchasing hardware in 2005.
9
Data Center Energy Usage
10
Computer Room Air Conditioning
• Large scale, highly
reliable air
conditioning units
from companies like
Liebert.
• Cooling capacity
measured in tons.
11
Hot-Cold Aisle Architecture
• Server air intake from cold aisles
• Server air exhaust into hot aisles
• Improve efficiency by reducing mixture of
hot/cold
Free Cooling
• Cooling towers dissipate
heat by evaporating water,
reducing or eliminating need
to run chillers
• Google Belgium DC uses
100% free cooling
12
Server PUE (SPUE)
Primary sources of inefficiency
– Power Supply Unit (PSU) (70-75% efficiency)
– Voltage Regulator Modules (VRMs)
• Can lose more than 30% power in conversion losses
– Cooling fans
• Software can reduce fan RPM when not needed
SPUE ratios of 1.6-1.8 are common today
13
Server Utilization ~10-50%
It is surprisingly hard
to achieve high levels
“The Case for of utilization of typical
Energy-Proportional servers (and your home
Computing,” PC is even worse)
Luiz André Barroso,
Urs Hölzle,
IEEE Computer,
December 2007
Figure 1. Average CPU utilization of more than 5,000 servers during a six-month period. Servers
are rarely completely idle and seldom operate near their maximum utilization, instead operating
most of the time at between 10 and 50 percent of their maximum
Figure 2. Server power usage and energy efficiency at varying utilization levels, from idle to
peak performance. Even an energy-efficient server still consumes about half its full power
when doing virtually no work.
Utilization 100%
14
Improving Power Efficiency
15
Containers
Data Center in a
shipping container.
– 4-10X normal data
center density.
– 1000s of servers.
– 100s of kW of power.
Advantages
– Efficient cooling
– High server density
– Rapid deployment
– Scalability
16
Data Center Failure Events
0.8
0.6
0.4
0.2
0
1 11 21 31 41 51 61 71 81 91
Fault-Tolerant Architecture
Must use fault-tolerant software architecture
– Hardware must detect faults
– Hardware must notify software in timely fashion
Fault-tolerant architecture reduces costs
– Choose hardware reliability level that maximizes
cost efficiency, not just reliability
Fault-tolerant architecture can improve perf
– Spreading processing and storage across many
servers improves bandwidth and CPU capacity
17
Causes of Service Disruptions
18
Operational Costs
Operational costs include
– Electricity
– Salaries for personnel
– Server maintenance contracts
– Software licenses
Larger data centers are cheaper
– Smaller number of sysadmins per server
– Fixed number of security guards
For multi-MW data center, $0.02-$0.08/month
Case Study
Tier 3 multi-MW data center
– Dell 2950 III EnergySmart servers (300W, $6000)
– Cost of electricity is 6.2₵/kW
– Servers financed with 3-year loan @ 12%
– Cost of DC construction is $15/W, 12-yr lifetime
– DC opex is 4₵/month
– PUE = 2.0
– Server lifetime is 3 years
– Server maintenance is 5% of capex
– Server avg power = 75% peak
Key Points
Data center components
– Physically secure space
– Racks, the DC skeleton
– Power, including UPS and PDU
– Cooling
– Networking
Power efficiency (server cost = 4 years power on avg)
– PUE = Data center power / IT equipment power
– Most power in traditional DC goes to cooling, UPS
– SPUE = Server PUE; inefficiencies from PSU, VRM, fans
Cooling
– Heat load estimation
– Air flow control (hot/cold aisle architecture or containers)
– Higher cold air temperatures (27C vs. 20C)
– Free cooling (cooling towers)
TCO = DC depr + DC opex + Svr depr + Svr opex
19
References
1. Luiz Andre Barroso and Urs Holzle, The Case for Energy-
Proportional Computing, IEEE Computer, Vol 40, Issue 12,
December 2007.
2. Luiz Andre Barroso and Urs Holzle, The Datacenter as a
Computer: An Introduction to the Design of Warehouse-Scale
Machines, 1st edition, Morgan and Claypool Publishers
3. Xiaobo Fan, Wolf-Dietrich Weber, Luiz Andre Barroso, Power
provisioning for a warehouse-sized computer, ISCA '07:
Proceedings of the 34th annual international symposium on
Computer architecture
4. Thomas A. Limoncelli, Christina J. Hogan, and Strata R.
Chalup, The Practice of System and Network Administration,
Second Edition, Addison-Wesley Professional, 2007.
5. Evi Nemeth, Garth Snyder, Trent R. Hein, Ben Whaley, UNIX
and Linux System Administration Handbook, 4th edition,
Prentice Hall, 2010.
20