ATDEN22 021ebook

Accredited Tier Designer
1
Select portions of the curriculum presentation are not provided in this booklet due to permissions granted or withheld
by companies providing actual examples for educational purposes only. Please do not copy or distribute this booklet in
either hard copy or soft copy. Please refrain from taking pictures or videotaping any portion of this course without the
instructor’s prior written permission.
Copyright ©2009-2022 by Uptime Institute, LLC

405 Lexington Avenue, 9th Floor
New York, NY 10174
All rights reserved.
Uptime Institute’s publications are protected by international copyright law. Uptime Institute requires written requests
at each and every occasion that Uptime Institute’s intellectual property or portions of Uptime Institute’s intellectual
property are reproduced or used. Uptime Institute copyright extends to all media—paper, electronic, and video content
—and includes use in other publications, internal company distribution, company websites and marketing materials,
and handouts for seminars and courses. For more information, please visit www.uptimeinstitute.com/resources to
download a Copyright Reprint Permission Request Form.
2
Table of Contents
Uptime Institute Education....................................................................................................................4
Session 1: Tier Standard: Topology......................................................................................................6
Session 2: Cooling Systems................................................................................................................22
Session 3: Electrical Systems.............................................................................................................42
Session 4: Ancillary Systems..............................................................................................................65
Session 5: Common Discrepancies....................................................................................................85
Session 6: Design Impacts to Operational Sustainability................................................................95
Session 7: Myths and Misconceptions.............................................................................................109
Session 8: Design Review Exercise.................................................................................................. 114
Abbreviations...................................................................................................................................... 118
Contacts............................................................................................................................................... 123
Additional Learning Opportunities...................................................................................................125
For our Tier Standards and Technical papers, please visit www.uptimeinstitute.com
3
Uptime Institute Education
Each of our three core programs is designed to allow students to start with the essential knowledge they need to
do their current jobs and then grow in their ability to meet more demanding challenges over time. Once enrolled
in a specific program, successful completion of an initial course will result in the student’s base-level Accredited
credential. Through additional advanced course completion, the student can gain Professional designations in
these programs based upon their demonstrated mastery of the concepts.
Core Accreditation
The Accredited Tier Designer (ATD) program provides instruction to licensed professional engineers, designers,
and project managers in design management roles on practical application of the Uptime Institute Tier Standard:
Topology.
The Accredited Tier Specialist (ATS) program provides instruction for the management team directly
responsible for data center uptime on the Uptime Institute Tier Standards: Topology and Operational
Sustainability as well as the importance of staffing, training, planning, and operation of critical facilities. This
course can also serve as a fundamentals introductory course to the digital infrastructure industry.
The Accredited Operations Specialist (AOS) program is based on the Uptime Institute Tier Standard:
Operational Sustainability and provides instruction to achieve a complete understanding of the concepts and
criteria developed in a comprehensive world class Management & Operations program for a critical facility.
Professional Accreditation
The Accredited Tier Professional (ATP) program provides advanced analysis, problem-solving, and design
guides to Accredited Tier Designer (ATD) alumni, equipping them with detailed application of the Tiers, including
comprehensive approaches to enable fast-tracked achievement of Uptime Institute Tier Certification of Design
Documents (TCDD) and Tier Certification of Constructed Facility (TCCF).
The Accredited Operations Professional (AOP) program equips executives responsible for the management
of multiple data center properties and managers and team leaders of data center operations teams with the
strategies, concepts, and principles fundamental to cost-efficient, high-reliability, and sustainable operations.
Course participants are provided insight into successful operations strategies, methods for translating those
strategies into actionable goals and initiatives, and practical examples of the benefits of the Uptime Institute
approach to Operational Sustainability.
All courses require completing in-depth training and receiving a passing score on the final examination.
© 2009-2022 Uptime Institute, LLC. All rights reserved
4
Uptime Institute Training Courses Qualify for CPD Credits
CPD Accreditation means taking Uptime Institute Accredited Training Courses qualifies
as continuing education credits toward earning and maintaining professional certifications
and designations associated with your profession (see www.uptimeinstitute.com for more
information).
© 2009-2022 Uptime Institute, LLC. All rights reserved
5
Session 1
6
Session 1 Tier Standard: Topology
Tier Standards
• Tier Standard: Topology
› Defines Tier Classification System
› Performance Requirements for each functionality objective
• Tier Standard: Operational Sustainability

› Defines Expected Operational Behaviors
› Aligned by Tier
Both Are Owner Standards
Distinction Between Courses

• Accredited Tier Designer (ATD) course is for data center experts that
generate designs
› Course principally dedicated to the Tier Standard: Topology
› Detailed presentation of the Uptime Institute Tier topology concepts
› “Deep dive” into proper application of Tiers
• Accredited Operations Specialist (AOS) course is for data center

owners, operators, and outside resources
› Course principally dedicated to the Tier Standard: Operational
Sustainability
› Management practices critical to Operational Sustainability
• Accredited Tier Specialist (ATS) course is for data center owners,

operators, and outside resources
› Broad overview of both Tier Standards rather than deep dives
7
ATD Course Goals & Context

Local codes and requirements form the foundation for design
examples based on minimum Tier criteria, not best practices
• Enhance design professionals’understanding of the practical
requirements of Uptime Institute Tier-based design
• Facilitate consistent application of Uptime Institute Tier topology
concepts
• Provide valuable education and recognition for industry
participants in the Uptime Institute Tier Program
Accreditation is not an endorsement for Certification
The Concept of “N”

• The concept of N: Definition of N sensitive to the context it is
used
› Refers to required number of components to meet “need”
› Refers to capacity of system in kilowatts (kW) when discussing the
design load or design demand
• “R” typically appears whenever N is stated

› Refers to “redundant” number of components
8
Overview
• Tier Classification Genesis • Engine-Generator Systems
• Tier Classification Objectives • Ambient Temperature Design
Points
• Tier Topology Categories
• Makeup Water
• Key Tier Topology Principles
• Telecommunications
• Capacity Component
• Operational Sustainability
• Distribution Path
• Tier Classifications
• Public Utility Systems
Tier Classification Genesis

• An Owner’s Request
Data Center Performance and Investment Criteria
• An Industry Solution
Tier Classifications Define Site Infrastructure Performance
• An International Standard
Data Center Site Infrastructure Tier Standard
9
Tier Classification Objectives

• Provide a common understanding
and language of data center
infrastructure concepts
• Identify expected data center
performance by differences in
topology
› Recognize that all data centers are
not alike
▪ nor need to be
› Refers to a single operations site
• Tier concepts are simple; application requires extreme diligence
Tier Topology Categories

• Tier Classifications represent broad topology categories
› Redundant capacity components
› Redundant (diverse) distribution paths
› Classification based on Maintenance opportunity and Failure
response
• Fractional “concepts” are not rationalized

› No Standard for Tier III.6
› Tier III + is undefined
› Site Classification based on lowest system rating
• Differentiation within a Tier enhances Operational Sustainability
10
Key Tier Topology Principles

• Begin and end at the IT critical environment
• Build upon the previous Tier
• Provide facility operation and maintenance opportunities
• Considers only the built environment
Capacity Component
• Active devices
• Moves heat from one location to another via the use of active
devices
• Generates electrical power from an energy source (fuel, stored
energy)
11
Distribution Path
• Not a Capacity Component
• Distribution path elements such as
› Passive devices
› Conductors
› Pipes
› Valves
› Transformers
Tier Classifications
• Tier I – Basic Capacity
• Tier II – Redundant Components
• Tier III – Concurrently Maintainable
› Applies to Each and Every component and path
• Tier IV – Fault Tolerant
› Considers a Single event, along with Consequential impact
12
Tier I – Basic Capacity

• Summary
› Non-redundant capacity components (N only)
▪ Critical environment power and cooling systems
› Single distribution path
• Operations and Maintenance Considerations

› Site infrastructure and critical environments must be shut down for
annual maintenance and repair work
› Installation or construction of capacity may disrupt the critical
environment
Tier I – Operational Risks

• Any capacity component or distribution
path element failure will disrupt the
critical environment
• All or portions of the critical environment are susceptible to
disruption due to planned and unplanned activities
• Operations (human) errors have high likelihood of site disruption
• Deferred maintenance to avoid downtime increases the risk and
severity of disruptions in the critical environment
13
Tier II – Redundant Components

• Summary
› Redundant capacity components (N+R)
▪ Engine generators, UPS modules, IT, and UPS cooling
› Single distribution path

› Capacity components can be maintained or repaired with limited
impact to the critical environment
› Site infrastructure and critical environments must be shut down for
annual maintenance and repair work
› Installation or replacement of capacity components may disrupt the
critical environment
Tier II – Operational Risks

• A capacity component failure may disrupt
the critical environment
• A distribution path element failure will disrupt
the critical environment
• All or portions of the critical environment are susceptible to
disruption due to planned and unplanned activities
• Operations (human) errors have high likelihood of site disruption
• Deferred maintenance to avoid downtime increases the risk and
severity of disruptions in the critical environment
14
Tier III – Concurrently Maintainable

• Summary
› Redundant capacity components and independent distribution
paths (transformers and transfer switches are path elements)
› Some elements of a distribution path may be inactive
› Predicated on dual-cord IT equipment
› No runtime limits on engine-generator capacity at design load

› Each and Every capacity component and distribution path element
can be taken out of service for maintenance, repair, or replacement
on a planned basis without impacting the critical environment or IT
processes
Tier III – Practical Insight

• Each and Every extends to:
› Valves and fittings
› Switchgear and panels
• Maintenance focus requires:
› Dead lugs for safety during electrical activities
▪ Completely de-energized electrical devices
› Dry pipes to avoid liquid spills
▪ Mechanically isolated to allow draining of the section or equipment
• Single Points-of-Failure are not eliminated
15
Tier III – Operational Risks

• All or portions of the critical environment
are susceptible to disruption due to failures
or unplanned activities
• Scheduled maintenance activities occur on redundant
components, distribution paths, and systems—which will reduce
redundancy and may elevate risk of disruption
• Operations (human) errors may lead to site disruption
• Single-corded IT equipment or incorrect installation may defeat
the intent of the Tier III electrical infrastructure for that IT
equipment
Tier IV – Fault Tolerant

• Summary
› Redundant capacity components
› Redundant active distribution paths
› Compartmentalization of both capacity components and
distribution paths
› N after any failure
› Continuous Cooling for critical IT and UPS systems
› No runtime limits on engine-generator capacity at design load

› Each and Every capacity component and distribution path element
can sustain a failure, error, planned, or unplanned event without
impacting the critical environment or IT processes
16
Tier IV – Practical Insight

• Single event with consequential impact
› Loss of a switchboard impacts everything “downstream”
powered by that switchboard
› Replacing a valve requires a dry pipe on both sides
• Design considerations for Continuous Cooling are consistent
with UPS for IT equipment power
• Most human errors are considered failure events
› Exceptions
▪ Emergency power off (EPO) activations
▪ Fire suppression activations
▪ Failure to properly connect IT loads
Tier IV – Operational Risks

• The critical environment is not susceptible
to disruption due to failure of any single
capacity component, distribution element,
site infrastructure system, or single human error
• Scheduled maintenance activities occur on redundant
components, elements, and systems—which may create a risk
of disruption
• Operation of the EPO system, activation of the fire protection
system, or malicious human interaction may lead to site
disruption
• Single-corded IT equipment or incorrect installation may defeat
the intent of the Tier IV electrical infrastructure for that IT
equipment
17
Public Utility Systems

• The number or lack of utility service entrances does not affect
Tier Classification
• The on-site power production system is the source of electrical
power for site availability and reliability
› Engine generators
› Turbine generators
› Fuel cells
• The loss of any public utility (power, water, natural gas, district
heating, district cooling, etc.) is not considered a failure
› The data center must autonomously detect the utility loss and
autonomously respond by switching to on-site sources
› In select designs this may require the autonomous restart of
equipment
Engine-Generator Systems
• Concurrent Maintainability and Fault Tolerance
› The engine-generator system, along with its power paths and other
supporting elements, shall meet the applicable Concurrently Maintainable
and Fault Tolerant criteria while they are carrying the site load on engine-
generator power
• Manufacturers' Runtime Limits
› Manufactures’limitation on consecutive hours of operation at the design
load is not permissible for Tier III and IV
› Prime and Standby engine-generator ratings include runtime and load
limits
• Regulatory Runtime Limits
› Annual Authority Having Jurisdiction (AHJ) regulatory limits on operating
hours driven by emissions do not affect the capacity or redundancy of
engine generators in Tier design or Certification
18
Ambient Temperature Design Points

• Power and cooling equipment capacities shall be determined at
extreme recorded outdoor temperatures
• The ASHRAE Handbook—Fundamentals lists appropriate
outdoor design points
› The design wet bulb (WB) temperature shall be the
“Extreme Maximum Wet Bulb”
› The design dry bulb (DB) temperature shall be the “n=20 years”
value
• Cooling coil capacity shall be determined at anticipated

computer room or critical environment return air conditions
• Minimum and maximum dry bulb temperatures shall be
considered
Makeup Water
• Applies to direct and indirect evaporative cooling solutions
• Must provide for 12 hours consumption from
on-site sources at N demand
• Makeup water system shall meet other Tier criteria (as required
for Tier II‒IV) to the point of consumption
• More information in Session 4 and detailed examples address
water volumes involved in the Accredited Tier Designer
Technical Paper Series: Makeup Water
19
Telecommunications
• Requirements
› Power and Cooling to active telecommunication and
communication equipment must meet Tier requirements
• Recommendations
› Consider designing communications conveyance (vaults, cable
ways, conduit) in accordance with Concurrent Maintainability (Tier
III) or Fault Tolerance (Tier IV) from the property line to the data
center communication demarcation point
• Points beyond the property line are beyond control of the owner
• Distribution within the data center is adequately covered by
other standards
Operational Sustainability
• Complements Tier Standard: Topology
• Differentiates the value of investment within a Tier level
• Focuses on the effectiveness of investment over the life of the
facility
• Aggregates Positive and Negative impact of design and
management choices
› Management and Operations
› Built Environment
› Site Location
20
Tier Classification Recap

• The four Tiers are progressive
› Tier I is the foundation
› Tier II includes Tier I, adds redundant components
› Etc.
• Each and Every considerations are exhaustive
• Engine-generator ratings are more restrictive for Tier III and Tier
IV
• Tier III may have Single Points-of-Failure
• Tier III is all about Maintenance Opportunities and not Faults
• Tier IV requires Continuous Cooling and Compartmentalization
21
Session 2
22
Session 2 Cooling Systems
Cooling Systems
Nominal Capacity: 300 GPM
300 300 300
N= 300 GPM NN+2N

+N1:
1=300
300GPM,
GPM,Redundant
Redundant
(gallons per minute)
150 150 150 150 150
N + 1= 300 GPM, Redundant
Basic Redundant
23
Single Distribution Path
Basic
Capacity
N= 300 GPM
Redundant
Components
Redundant Distribution Path
Concurrently
Maintainable
N= 300 GPM
24
Redundant Distribution Path
A A A A A A
Fault
Tolerant
N= 300 GPM
A A A A A A
au·ton·o·mous: (adj.) Not controlled by others or outside forces; independent
Compromised Distribution Path
A A A A A A
Basic
Capacity
N= 330 GPM
A A A A A A
25
Valves are a Risk

• A common “rule of
thumb” is to avoid valves
whenever possible
› Engineers for nuclear
power plants and
submarines minimize
valves where possible
• Both nuclear plants and
submarines are shut
down for an extended
period on a regular basis
for maintenance
(Tier II equivalent)
Tier III and Tier IV data centers are NOT shut down on
a regular basis for maintenance!
Dual Distribution Path
Concurrently
Maintainable
N= 300 GPM
System A System B
26
Dual Distribution Path
Fault
Compartmentalization
Tolerant
N= 300 GPM
System A System B
Basic Capacity (Tier I)

Computer Room Cooling
27
Redundant Components (Tier II)

Concurrently Maintainable (Tier III)

Concurrently Maintainable
28

Concurrently Maintainable System

Concurrently Maintainable Dual-Coil
29

Concurrently Maintainable 4-Pipe
Fault Tolerant (Tier IV)

30

Chilled Water Distribution
• The topology may look like Tier III solutions
• Tier IV must autonomously respond to failures— such as leaky
valves or fittings—to prevent further impact to the site
• Compartmentalization of chilled water pipes and control elements
outside the computer room is required

Chilled Water System
31

Chilled Water System
Tie Valves in a Chilled Water System

Compliant
32
Glycol Systems
CRAC CRAC CRAC
Glycol System Tier Progression
Basic Capacity N Components

CRAC
33
Redundant
Components CRAC CRAC N+1 Components
Concurrently
Maintainable CRAC CRAC N+1 Components
34
Add Capacity
Larger Capacity
Pumps
Concurrently
Maintainable CRAC CRAC CRAC N+1 Components
Split-System CRAC Cooling
• Avoids issues of multiple valves

and piping systems
• Integrity of power sources is a
significant design and Certification
consideration (Session 3)
• Ambient (outdoor) design
temperatures have a large affect on
capacity (Session 5) CRAC CRAC
• Continuous Cooling, required for

Tier IV, has proven difficult to
achieve
35
Continuous Cooling
• Continuous Cooling is the capability to maintain steady state in the
critical environments without site power until the mechanical
system is providing rated cooling at the extreme ambient conditions
› Computer rooms, network rooms, UPS rooms. . .
• To complement uninterrupted power for IT devices

• Continuous Cooling is required to meet Tier IV criteria
› Part of the Uptime Institute Tier Standard
Why Continuous Cooling?

• Provides IT devices thermal stability without site power until the
mechanical system is providing rated cooling to the IT devices at
the extreme ambient conditions
• Lockouts occur frequently
› Chillers/CRACs /Switchgear relays
› Consider a UPS battery plant is often designed for 15 minutes or more
• “Normal” Sequence of Operations can take several minutes
› Programmed engine-generator start delay
› Start and parallel sequence
› Switchgear transfer
› Chiller or CRAC restart sequence
36
Continuous Cooling Case Study

ASHRAE 2011 Recommendations
• Recommended operating envelope for IT equipment
› Temperature
64.4‒80.6°F dry bulb
(18‒27°C)
› Temperature Rate-of-Change
9°F per hour
(5°C)
› Moisture
≤ 60% relative humidity (RH) and
41.9‒59°F dew point
(5.5‒15°C)
Continuous Cooling Case Study

ASHRAE 2015 Recommendations
• Recommended operating envelope for IT equipment
› Temperature
64.4‒80.6°F dry bulb
(18‒27°C)
› Temperature Rate-of-Change
32°F per hour
(20°C, but no more than 5°C in any 15 minutes)
› Moisture
≤ 60% relative humidity (RH) and
15.8‒59°F dew point
(-9°C – 15°C)
37
Temperature Rate-of-Change Test

Test Protocol
• Steady state configuration and conditions
› Load = 250 cabinets at 6 kW average or 1,500 kW
› Temperature sensors indicate ≤ 80°F (27°C)
› Power and cooling system normal
• Shut down all (14) CRAH units
• Record temperature rate-of-rise to 90°F (32°C)
• Recover
Temperature Rate-of-Change Results

• First Data Point ~ 1 Minute
› Temperature sensor at 7 feet AFF = 90°F (32°C)
› Temperature sensor at ceiling = 110°F (43°C)
Owner aborts test!
• Last Data Point ~ 20 Minutes

› Temperature sensor at 7 feet AFF returns to 78°F (25.5°C)
38
Tier IV Continuous Cooling

• Cooling technology drives the solution
• Requires, as appropriate for cooling solution
› Uninterruptible power for cooling fans
› Uninterruptible power secondary chilled water pumps or glycol system
pumps
› Thermal storage or uninterruptible refrigeration
• Uninterruptible CRAC-based Continuous Cooling is conceivable
Noncritical Functions
• Some facility infrastructure components do not directly impact
availability of critical environments
• These systems may be isolated on failure or for maintenance for
extended periods
• Duration of “extended period” is equivalent to on-site stores of
engine-generator fuel and makeup water
39
Common Noncritical Functions

• Building pressurization (makeup air systems)
• Humidification
• Fuel polishing (fuel storage filtration systems)
• Reverse-osmosis systems
• “Free cooling” or economizer systems
• Building automation (Tier I, Tier II, Tier III)
• Support area comfort HVAC
• Battery room ventilation
Unintended Impacts
• Noncritical system connections to critical systems may create
noncompliant topology
• Cautions
∅ Chilled water connections for building air handlers
∅ Free cooling heat exchanger integration
∅ Fuel polishing system isolation
40
Utility Sources
• Tier I‒IV data center systems must be able to autonomously
respond to a loss of external cooling generation, such as district
cooling
• This is similar to how engine generators must be able to
autonomously respond to loss of utility power from the grid
Cooling Systems Recap

• Chilled water and glycol systems require careful attention to piping
and valve configurations
• Direct expansion (DX) systems simplify (avoid) piping challenges,
but require careful attention to power and Continuous Cooling
solutions
• Integration of noncritical systems can create
Concurrent Maintainability and Fault Tolerance discrepancies
41
Session 3
42
Session 3 Electrical Systems
Critical Electrical Systems

• Critical Distribution – Computer Room Power
› UPS output to server cabinets
› Includes other critical environments
• UPS Systems
• Power Backbone
› Site power source to UPS input and mechanical systems
• Engine Generators
• Utility Power Source
Alternate Solutions not Reflected in Diagrams

• Other possible UPS solutions
› Direct current rectifier
› Diesel rotary uninterruptible power supply (DRUPS)
• Other site power sources (could be only source of power)
› Turbines
› Fuel cells
43
Single Power Path Utility
Risks of Failure Multiple Power Sources
• UPS system level failure UPS

One Power Path
• Major circuit breaker
• Minor circuit breaker
• Plug or receptacle
PDU
• Electrical connection (Function)
Computer
Hardware
• Human error
• EPO
• Fire
Dual Power Path

Utility Utility
G G
Risks of Failure
Multiple Power Sources
• A/B circuiting error
• UPS step load failure
UPS UPS
Two Power Paths
• EPO
• Fire
PDU PDU
(Function) (Function)
Computer
Hardware
44
Static Transfer Switches (STS)

Utility Utility
G
G
• Large impact upon
failure
UPS UPS • Input to computer
hardware is not
Two Power Paths
One Power Path

below this point
› Concurrently
Maintainable
Static Transfer
Switch
› Fault Tolerant
Computer
Hardware • Ensure STS
source-switch algorithm
will not transfer a fault to
the alternate source!
Point-of-Use Transfer Switch (PTS)

Utility Utility
G G
• Tier-compliant
accommodation
UPS UPS
Two Power Paths • Provides redundancy close
to load
• Flexible solution added or
removed as required
Computer • Risk transferred to IT
Hardware
architecture
Dual Power Path

PTS to this point
45
Critical IT Loads
• Computer room loads provide the precedent
• Network infrastructure/switch rooms
• Network operations center (NOC)
› Consider impact to system continuous availability requirements for
IT operations
• Building automation or management systems (BAS/BMS)
› Essential to Autonomous Tier IV system response
UPS Capacity –
Need (N) versus Redundant
500
Basic
N=400 kW
500 Sys A1
Sys A1
Redundant Sys A2
500 Sys B1 N=400 kW Sys A
46
Basic Capacity
UPS and Critical Distribution
• N=3 UPS modules
• Single path
Redundant Components
LV A Source
UPS A Input Bus • N=3 UPS modules

• Single path
=
~
=
~
=
~
=
~
• Adds redundant UPS
module
= = = =
~ ~ ~ ~
UPS A Parallel Bus
UPS A Output Bus
Critical Distribution
Bus A 600-V
480-V
Device
240-V
120/208-V
Device
PDU Ax
47
• N=3 UPS modules
• Adds second critical
distribution path
• Adds means to maintain
UPS system
Fault Tolerant
• N=3 UPS modules

• Dual path
48
Tier III and IV Shortfall

Main-Tie-Main
UPS A Output Bus UPS B Output Bus
Critical Distribution Critical Distribution

Bus A 600-V Bus B
480-V
Device
240-V
120/208-V
Device
PDU Ax PDU Bx
Tier III and IV Method

Main-Tie-Tie-Main
Automatically closing into a fault will compromise Tier IV
49
Electrical Backbone
• Includes the elements that distribute power from the site power
sources (utility and engine-generator systems) to the UPS input
and mechanical and building loads
• Tier concepts apply similarly to both medium- and low-voltage
segments
Basic Capacity
Electrical Backbone
50
Electrical Backbone
Electrical Backbone
51
Fault Tolerant
Electrical Backbone
Minimum Fault Tolerant

Electrical System
52
Basic Capacity
Cooling Components Power
Mechanical Distribution A
Computer Room Cooling A
Condenser
CRAH
CRAH
CRAC
N=3 Cooling units
Mechanical Distribution A
Computer Room Cooling A

Condenser
Condenser
CRAH
CRAH
CRAC
CRAC
N=3 Cooling units Redundant Capacity Components
53
Concurrently Maintainable & Fault Tolerant

Condenser
Condenser
Condenser
CRAH
CRAH
CRAH
Tier IV requires autonomous
CRAC
CRAC
CRAC
response to failure
Redundant Capacity Components Redundant Capacity Components
N=3 Cooling units

Alternate Power

response to failure
N=3 Cooling units
54

Chilled Water Plant Power
Manual Transfer Switch

Automatic Tier IV requires autonomous
response to failure
N=3 Chillers
Engine-Generator Systems
• Consider engine generators as on-site power source
› Tier III requires Concurrent Maintenance while carrying the load
› Tier IV requires Fault Tolerance while carrying the load
• Manufacturers’ Engine Ratings
› Tier III and Tier IV engine-generator capacity is based on
manufacturers’ unlimited runtime capacity at site conditions
› Standby and non-derated Prime-rated units are acceptable for Tier
I and Tier II using the capacity at site conditions
• Runtime Limits
› Manufacturers’ runtimes represent capability constraints
› Government regulated (typically environmental) runtimes represent
regulatory constraints
55
Definitions
• Definition of an engine generator
› A generating set consists of one or more Reciprocating Internal
Combustion (RIC) engines used to produce mechanical energy and
one or more generators to convert the mechanical energy into
electrical energy. The generating set includes any components
used for coupling the mechanical prime mover(s) and electrical
generator(s) (e.g., couplings, gearbox) and, where applicable, any
load-bearing and mounting components.
– International Standards Organization (ISO)® 8528-1, 2005
Assumptions
• Applies only to engine generators as defined
• Purposes of an engine-generator plant
› Provide reliable power for data centers
› Provide power when utility is not available
› Provide power during maintenance activities
› Provide power after infrastructure failure
▪ Loss of utility is not considered a failure but is a normal
operational condition
• Engine-generator plant could be the only source of power
• Capacity is a separate discussion from environmental permitting
constraints or emissions ratings
56
Engine-Generator Ratings
• International Standards Organization (ISO) 8528-1 is the
governing document
• 4 rating classifications
› Emergency Standby
› Limited-Time Running
› Prime
› Continuous
• Major differences in operating hours and power output

capacities
Standby Rating
• Definition
› The maximum power available during a variable electrical power
sequence, under the stated operating conditions, for which a
generating set is capable of delivering in the event of a utility power
outage or under test conditions for up to 200 hours of operation
per year – ISO 8528-1
• Caveat for manufacturer’s maintenance intervals

• But
› The permissible average power output over 24 hours of operation
cannot exceed 70% of the standby rating unless otherwise agreed
by the manufacturer – ISO 8528-1
57
Prime Rating
• Definition
› The maximum power which a generating set is capable of
delivering continuously while supplying a variable electrical load
when operated for an unlimited number of hours per year – ISO 8528-1

• But
› The permissible average power output over 24 hours of operation
cannot exceed 70% of the prime rating unless otherwise agreed by
the manufacturer – ISO 8528-1
Continuous Rating
• Definition
› The maximum power which the generating set is capable of
delivering continuously while supplying a constant electrical load
when operated for an unlimited number of hours per year – ISO 8528-1
58
Manufacturer's Engine Generator

Rating Definitions
Electrical Hours of Tier Suitability
Rating Remarks
Load Operation I II III IV
• Permissible output over 24
hours cannot exceed 70%
✗
unless otherwise agreed by
Standby Variable 200* ✓ ✓ ✗ manufacturer
• Caveat for manufacturer’s
maintenance intervals
• Permissible output over 24
hours cannot exceed 70%
unless otherwise agreed by
Prime Variable Unlimited ✓ ✓ ✓ ✓ manufacturer
maintenance intervals

Continuous Constant Unlimited ✓ ✓ ✓ ✓ maintenance intervals
* - Manufacturers can attest to the unlimited runtime for Tiers III & IV applications
✓ - Suitable after derating
Manufacturer's Engine Generator

Rating Definitions
• ISO 8528 definitions are the default ratings for Tier Certifications
• Manufacturers may attest that their units exceed the ISO 8528
definitions for
› Average power capability – higher than 70% for standby and prime
units
› Unlimited runtime – standby units without hours limitations
› Manufacturers must provide a letter from a technical authority
• Manufacturers are moving to specific definitions for data center
applications
• Must utilize the site conditions including altitude and ambient
conditions
59
Engine-Generator System Topology

• Tier I requires an engine-generator system providing adequate
capacity (N) for critical load
› At least one engine generator or equivalent
› No specific technology mandated
• Tier II requires at least N+1 capacity components
› At least two engine generators or equivalent
Engine-Generator Concepts
N=3 Engine Generators
60
Engine-Generator Concepts (Alternate)
N=3 Engine Generators
Fault Tolerant
Engine-Generator Concepts
N=3 Engine Generators response to failure
Engine-Generator Engine-Generator
Paralleling Bus A Utility Bus Paralleling Bus B
Main Distribution A Main Distribution B
UPS Input Bus A Mech Distribution Bus A Mech Distribution Bus B UPS Input Bus B
61
Utility Power Sources

• Provides an economical alternative to on-site power generation
• Utility power solutions are not required to meet criteria for any
Tier
• Due diligence is required to ensure utility transformer and bus
configuration do not compromise
› Dual path requirements of Tier III
› Compartmentalization requirements of Tier IV
Utility Power Sources

• Tier I‒IV data center systems must be able to autonomously
respond to a loss of external power generation, such as from the
grid
• This is similar to how cooling systems must be able to
autonomously respond to loss of district cooling
62
Sample Schematic
23 Racks 22 Racks 22 Racks 22 Racks 22 Racks
124.2 KW 118.8 KW 118.8 KW 118.8 KW 118.8 KW
23 Racks 23 Racks 22 Racks 22 Racks 22 Racks

124.2 KW 124.2 KW 118.8 KW 118.8 KW 118.8 KW
PDU PDU PDU PDU PDU PDU PDU PDU PDU PDU PDU PDU
1A 1B 2A 2B 3A 3B 4A 4B 5A 5B 1C 1D
UPS-1 UPS-2 UPS-3 UPS-4 UPS-5

450 KW 450 KW 450 KW 450 KW 450 KW
Distribution Distribution Distribution Distribution

Board 1 Board 2 Board 3 Board 4
Exercise
• Split the class into 2 teams
1. Electrical
2. Mechanical
• Each team to explain their understanding of the cooling unit

redundancy based on their design
• Each team then assumes the other was right, and explain
how to fix/improve their design
63
Critical Environment Cooling System

Schematic
Electrical System Recap

• Tier progression
› Basic Capacity
› Redundant Components
› Concurrent Maintenance
› Fault Tolerance
• Begin power path design in the computer room and continue to
the site power source, including each and every component
and element
• Electrical topology can defeat mechanical topology
64
Session 4
65
Session 4 Ancillary Systems
Makeup Water System

• Backup on-site makeup water storage is required for sites using
evaporative cooling
• The makeup water system must meet applicable topology
criteria by Tier to point of delivery for a minimum duration of
12-hours
• Technical Paper Series provides additional information about
typical water volumes
Makeup Water Tier Criteria
12 Hour Capacity
12 Hour Capacity
Bulk Water Storage –

No Topology Requirements
66
Makeup Water Tier Criteria – Tier III
N+1 Components
Makeup Water Tier Criteria – Tier III
N+1 Components
67
Makeup Water Tier Criteria – Tier IV
2N Components
Engine-Generator Fuel System

• Storage tanks
• Piping systems
• Pumps
• Day tanks
• Controls
• Filtration
68
Fuel System Tier Progression

• Tier I – 12-hours fuel storage for engine generator(s)
• Tier II – Redundant tanks and pumps for 12-hour supply (at least
2)
• Tier III – Concurrently Maintainable 12-hour fuel supply system
to “N” engine generators while in operation
› Identical to minimum makeup water system storage requirement
• Tier IV – Autonomous control response to component or path
failure
Fuel System Tier Criteria – Tier I & II

Bulk Fuel Storage –
No Topology Requirements
Tier I –
Total of 12 Hours Fuel
Tier II –
Redundant 12 Hours Fuel
Tier III and Tier IV –

Other Fuel System Requirements
69
Fuel System Tier Criteria – Tier III
N+1 Components
Fuel System Tier Criteria – Tier III
N+1 Components
70
Fuel System Tier Criteria – Tier IV
2N Components
Fuel System Consideration
• A single equalization valve is not Concurrently Maintainable nor

Fault Tolerant
• Use double valves for Concurrent Maintainability
71
Fuel System Consideration
• A single equalization valve is not Concurrently Maintainable nor

Fault Tolerant
• Use double valves for Concurrent Maintainability
• Provide Compartmentalization for Fault Tolerance
• Manually operated “Normally Closed” valves avoid Fault Tolerant
automation requirements
Fuel Polishing Systems

• Fuel polishing (filtration and bio-treatment) is not required
for any Tier
› Fuel polishing is an Operational Sustainability issue
• Fuel polishing must be carefully integrated to avoid
compromising Tier III or Tier IV criteria
72
Fuel Polishing Configuration
• This configuration is limited to supplying

no more than the redundant number of
engine-generator subsystems
Fuel Polishing Configuration
Filter
System
Engine Fuel Supply
• This fuel filtration configuration can be

isolated without affecting fuel supply to
engine generators
Fuel Storage Tank
73
Building Automation
• Includes
› Supervisory control and data acquisition (SCADA)
› Plant controls: building automation system (BAS) and building
management system (BMS)
› Emergency power off (EPO)
• Principal Tier IV Consideration
› Tier IV requires autonomous response to failure
Tier IV Autonomous Response

• Operator intervention shall not be required to respond to single
system failure
• Control system failure shall not disrupt critical environment
› Critical environment must remain stable with failed control system
• Tier IV data center facility infrastructure control systems shall
› Detect system failure
› Isolate and contain failure
› Sustain “N” capacity after failure of any component or path
74
Electrical System Control Considerations

• No special consideration for Tier I or Tier II
• Control maintenance or a single control failure shall not disrupt
subordinate control systems (e.g., chiller plant control)
• Control systems must comply with Tier criteria while engine
generators are supporting the site load
› Concurrently Maintainable for Tier III
› Fault Tolerant for Tier IV
› A single failure (including software instance) shall not prevent
autonomous engine-generator start/operation in Tier IV sites
Plant and Temperature Controls

• Common names • Scope of system
› Building automation system › “Head-end” server
(BAS) › Communications (may include site network)
› Building management systems › Building node control panel,
(BMS) local control panel
› Direct digital controls (DDC) › Sensor, actuator, input/output
cards, power supplies
75
Plant and Temperature Controls

• No special controls requirements for Tier I and Tier II
• Tier III requires that the critical environment must remain stable
with any control element out of service for calibration, repair, or
replacement on a scheduled basis
• Tier IV data center facility infrastructure control systems shall
› Detect system failure
› Isolate and contain failure
› Sustain “N” capacity after failure of any component or path
Active Control Systems

• Active: Requires continuous modulation and feedback to
manage system
Will fail
• Control elements that must change state in response to failure

must be physically independent of failed component
› Power source
› Control cabinet or node
› Sensors
76
Active Control Impact
• Design represents traditional

Primary–Secondary loops
• “Normal” operation uses Valve “A”
valve “A” to control
temperature in secondary
loop

Primary–Secondary loops
• “Normal” operation uses Valve “B”
valve “A” to control
temperature in secondary
loop
• Variations in primary loop
pressure adversely impact
secondary loop temperature
stability
• “Abnormal” operation uses
valve “B” to control loop
temperature
77

Primary – Secondary loops
• “Normal” operation uses valve
“A” to control temperature in
secondary loop
• Variations in primary loop
pressure adversely impact
secondary loop temperature
stability
• “Abnormal” operation uses
valve “B” to control loop
temperature
• Control failure drives system to
full-cold—and loss of sensible
capacity
Fire Protection Considerations

• Gaseous fire suppression is permissible but not required by any
Uptime Institute Tier Classification level
• Integration of fire detection system into other facility
infrastructure systems shall be
› Concurrently Maintainable (Tier III) or
› Fault Tolerant (Tier IV)
• Combined fire detection and EPO systems are suspect
› Detracts from Operational Sustainability
78
Fire Protection Considerations

• The connection of the fire systems to the cooling units can defeat Concurrent
Maintainability and Fault Tolerance
• The location and state of the relays when the fire system is maintained or
suffers a fault are a critical consideration
FACP
+
NO
CRAH
NARM
- NC
Network Addressable Relay Module
NO
NO
CRAH
Relay 1
NC
NARM CRAH
NC
NO
Network Addressable Relay Module

Relay 2
CRAH
NC
Fire Alarm Control Panel
EPO Considerations
• The Tier Standard does not require an EPO if not required by
AHJ
• Standing-voltage EPO systems have not been shown to be
compliant with Tier III or Tier IV
› Often integrated into distributed components
› Centralized Applied Voltage systems are effective
▪ Applies to both the push-button activation circuit and the shunt-trip
circuit
• The EPO system shall be
› Concurrently Maintainable for Tier III
› Fault Tolerant for Tier IV
79
• Applies to complementary systems and distribution paths in
Tier IV topology
• Tier IV requires physical isolation to prevent a single event from
simultaneously impacting more than the number of redundant
components or systems
• Each compartment shall contain no more than the number of
redundant components
Effective Compartmentalization
Secondary Chilled Water Pumps N=3 (R=1) Chilled Water Machines and Primary Pumps N=4 (R=2)
80
Electrical Compartmentalization
Electrical Compartmentalization
81
Telecommunication Routing
• These recommendations apply to communications connections
from site property line to data center communications point of
demarcation, including intermediate vaults, rooms, and closets
› Tier III recommends diverse conveyances, but may share path
› Tier IV recommends diverse conveyances and physical isolation to
prevent a single event from simultaneously impacting more than the
number of redundant paths
• Remember, power and cooling for active communication
components must meet the requirements of the Tier objective
Site Communications Path
POP A POP A
POP B POP B
Section
Section
• Concurrently Maintainable
paths recommended for Tier III
82
Site Communications Path
POP A POP A
POP B POP B
• Compartmentalized path
recommended for Tier IV
Other Ancillary Systems

• No Tier-level specific or Certification criteria
› Building pressurization (Makeup air systems)
› Battery room ventilation
› Reverse-osmosis systems
› Free-cooling or economizer systems
› Lightning protection
› Grounding
› Load bank
• Integrate carefully!
83
Ancillary Systems Recap

• Compartmentalization is Required for Tier IV
• Critical ancillary systems must meet same criteria as power and
cooling system at each Tier level
› Tier II – Redundant Components
› Tier III – Concurrently Maintainable
› Tier IV – Fault Tolerant
• Reminder: Some ancillary systems are not critical but may
inadvertently compromise Tier
84
Session 5
85
Session 5 Design Discrepancies
Design Point: Uptime Institute Tier Standard

Ambient Temperature Design Points
“The effective capacity for data center facilities infrastructure
equipment shall be determined at the peak demand conditions
based on the climatological region and steady state operating
set points for the data center. All manufacturers’ equipment
capacities shall be adjusted to reflect the extreme observed
temperatures and altitude at which the equipment will operate
to support the data center.”
—Tier Standard: Topology
Design Point vs. Owners Objectives

• Owners’ initial expectations
› No water in the computer room
› Small infrastructure footprint
› Sustainable refrigerant
› 75°F (24°C) return air temperature at 50% RH
• Supports owners’ planned 500-kW server room
86
Design Point: Manufacturer’s Product Data

• Design Objective: 30-ton, air-cooled, DX unit
› Equivalent to 105 kW
kW
• Largest capacity range of a popular provider
› Nominal total capacity 104 kW ≈ Sensible capacity 83 kW
▪ 75°F (24°C) DB return air at 50% RH
› R22 refrigerant
› Semi-hermetic compressor
› Water-cooled compressor (120 GPM @ 85°F (29.4°C) EWT)
› “Capacity data is factory certified to be within 5% tolerance.”
› Alternative Model Total capacity 101 kW~ Sensible capacity 89 kW
▪ 75°F (24°C) DB at 45% RH
• 104-kW DX unit yields less than 85 kW of useful cooling
Capacity Determination
• Capacity at ASHRAE rating criteria must be adjusted for

the project
87
ASHRAE Design Conditions
• Reference: 2009 ASHRAE Handbook—Fundamentals

(Updated and published every 4 years)
ASHRAE Design Conditions for DENVER STAPLETON INTERNATIONAL

AIRPORT, DENVER, CO, USA
Extreme Annual
Monthly Design Dry Bulb Temperatures Profile (July)
Design Conditions
5% 2% 0.4% n=20 years
91.4°F 94.2°F 97.4°F 103.5°F
(Probable) Hours Exceeded

Unlikely
30 hour 15 hour 3 hour
(20-year period)
ASHRAE Design Conditions
• Reference: 2009 ASHRAE Handbook—Fundamentals

(Updated and published every 4 years)
ASHRAE Design Conditions for DENVER STAPLETON INTERNATIONAL

AIRPORT, DENVER, CO, USA
Extreme Annual
Monthly Design Dry Bulb Temperatures Profile (July)
Design Conditions
5% 2% 0.4% n=20 years
33°C 34.6°C 36.4°C 39.7°C
(Probable) Hours Exceeded

Unlikely
30 hour 15 hour 3 hour
(20-year period)
88
Capacity Validation
• Manufacturer’s data for 30-ton nominal unit, R407 refrigerant,
manufacturer’s recommended air-cooled refrigerant condenser
Outside DB Inside DB Sensible Sensible Capacity
Altitude Inside RH
Temperature Temperature Capacity (kW) (ton)
95°F (35°C) Sea Level 80°F (26.7°C) 40% 93.7 26.6
• Manufacturer’s data for n=20 extreme design condition,

adjusted for altitude (above 5,000 feet), using owners’ set points
Outside DB Inside DB Sensible Sensible Capacity
Altitude Inside RH
Temperature Temperature Capacity (kW) (ton)
104°F (40°C) 5,000 feet 75°F (24°C) 50% 64.9 18.5
Design Point:
500-kW Computer Room Demand
• Standard Capacity ~ N=6 (562 kW) Redundant=2
93.7 kW 93.7 kW 93.7 kW 93.7 kW
93.7 kW 93.7 kW 93.7 kW 93.7 kW
• Ambient Conditions ~ N=8 (519 kW) Redundant=0

64.9 kW 64.9 kW 64.9 kW 64.9 kW
64.9 kW 64.9 kW 64.9 kW 64.9 kW
89
Design Point:
500-kW Computer Room Demand
• Standard Capacity ~ N=6 (562 kW) Redundant=2
93.7 kW 93.7 kW 93.7 kW 93.7 kW
93.7 kW 93.7 kW 93.7 kW 93.7 kW
• 95% of Baseline ~ N=8+ (493 kW) Redundant=0

61.6 kW 61.6 kW 61.6 kW 61.6 kW
61.6 kW 61.6 kW 61.6 kW 61.6 kW
• Adjusted for Ambient conditions and variation ~ unit capacity is

60% of Nominal 105 kW (30 tons)
Mechanical System Power
Sub 7 Line 1 Public Utility Sub 9 Line 4

UTIL A
UTIL B
RM 12
RM 16
1 2 3 4 5 6
SYNC A
SYNC B
RM 12
RM 16
ATS ATS
3 4
ATS ATS ATS ATS

DIST B
1 2 5 6
RM 25
DIST C
DIST A
PNL 09 EG A Access PNL 11 EG B Access

RM 23
RM 27
Fuel Pumps Fuel Pumps

MCC 3
MCC 4
UPS 1A See E401 UPS 1B See E401

RM 34
RM 34
UPS A Cooling See E203 UPS B Cooling See E204

UPS 2A See E402 UPS 2B See E402
MCC 1
MCC 6
RM 30
RM 38
MCC 2
MCC 5
MCU 401
MCU 402
MCU 403
MCU 404
MCU 405
MCU 406
MCU 506
MCU 505
MCU 504
MCU 503
MCU 502
MCU 501
RM 30
RM 38
90
Tell-Tale Panel Schedules

Panel UMHVAC2A Panel URHVAC14 Panel UMHVAC2B
Rating: 480Y/277 3P 4W 600A 42 KAIC Rating: 480Y/277 3P 4W 400A 42 KAIC Rating: 480Y/277 3P 4W 600A 42 KAIC
Located: Mechanical Mezzanine Located: Roof Penthouse Located: Mechanical Mezzanine
Fed From: Fed From: Fed From:
Remarks: GND Bus Remarks: GND Bus Remarks: GND Bus
1 2 1 2 1 2
CRCU 01 3 4 CRCU 03 ACC 01 3 4 ACC 02 CRCU 02 3 4 CRCU 04

5 6 5 6 5 6
7 8 7 8 7 8
CRCU 05 9 10 CRCU 07 ACC 03 9 10 ACC 04 CRCU 06 9 10 CRCU 08

11 12 11 12 11 12
13 14 13 14 13 14
CRCU 09 CRCU 11 CRCU 10 CRCU 12

15 16 ACC 05 15 16 ACC 06 15 16
(Future) (Future) (Future) (Future)
17 18 17 18 17 18
19 20 19 20 19 20
ACC 09 ACC 11 ACC 10 ACC 12

21 22 ACC 07 21 22 ACC 08 21 22
(Future) (Future) (Future) (Future)
23 24 23 24 23 24
• Current Tier functionality limited by Penthouse Panel

• Future plan impacted as well
Piping Topology
CRCU - 07 CRCU - 18 CRCU - 05 CRCU - 16

(Future) (Future)
8 CWS 8 CWS
8 CWR 8 CWR
CRCU - 19 CRCU - 06 CRCU - 17 CRCU - 04

(Future) (Future)
Computer Room
UPS A Room UPS B Room
UCU - 27 UCU - 29
• Initial condition requires 4 redundant computer room cooling units
91
Valve Placement Guidelines

N+1 Air Cooled Chillers N+1 Secondary N+2 CRAH
Pumps
• N+1 devices in looped systems shall be separated by at least 2 valves

• N+2 devices in looped systems shall be separated by at least 1 valve
• Tier IV requires autonomous valve operation
Condenser Water Schematic
N+1 Components
92
Compartmentalization:
Uptime Institute Tier Standard
“Complementary systems and distribution paths must be

physically isolated from one another (Compartmentalized)
to prevent any single event from simultaneously impacting
both systems or distribution paths.”
—Tier Standard: Topology
Compartmentalization:
Tier IV Requirement
Secondary Chilled Water Pumps:
N=3 (R=1)
Chilled Water Machines, Primary Pumps:
ATS A ATS B ATS C ATS D
SCHWP A SCHWP B SCHWP C
VFD C
SCHWP D N=4 (R=2)
VFD A VFD B VFD D
MCC C
MCC A
MCC B
CHWM 1A CHWM 2A CHWM 1B CHWM 2B CHWM 1C CHWM 2C
• No more than the redundant number of like devices shall be

located in a single space
• Complementary power distribution components shall not be
located in the same physical space
93
Design Discrepancies Recap

• Just like a utility outage, hot days are not an accident
› Select equipment based on Extreme (hot and cold) ambient
conditions
• Integrate power topology with mechanical topology
› Redundant units are dependent on power source and path
• The number and location of isolation valves required is based on
equipment redundancy
› Ensure capacity is available with any element isolated
• Power path and mechanical equipment shall be
Compartmentalized for Tier IV
94
Session 6
95
Session 6 Design Impacts to Operational Sustainability
Strategic Concepts
• Start With The End In Mind
› Operations
› Commissioning
96
Operational Sustainability Overview

• Management and Operations
• Building Characteristics
• Site Location
Building Characteristics Categories

• Infrastructure • Pre-Operational
› Flexibility for Incremental › Commissioning
Capacity Increases
› Infrastructure to Support
Operations
› Ease of Maintenance
• Building Features
› Purpose Built
› Support and Specialty
Spaces
› Security and Access
› Setback
97
Infrastructure to Support Operations

• Simplicity of design has a major impact on the availability of the
data center by reducing human error
› Ease to Operate and Manage
› Ease to Understand Configuration
› Ease to Troubleshoot
98
20 EG Rooms 30 UPS Rooms 20 SW Rooms

Design Example: 2N Option
• Allows the UPS and PDUs to be loaded at maximum of 50%

• Easiest layout to manage loads and connections
99

Design Example: 3 to Make 2 Option
• Allows the UPS and PDUs to be loaded at maximum of 66.6%

• Requires strict management of loads and connections
Infrastructure Support to Operations

Labeling conventions
Color: All Infrastructure
(engine generators, panels, switchgear, piping, conduits, or cables)
Labels: Prefix AXXX

Suffix XXXA
System Labels System B XXXX
Identification Must:
1. Be easily understood
2. Identify system
3. Identify location
4. Sized for easy reading
5. Consistently located
100
Ease of Maintenance
• Maintenance space is necessary to enable proper maintenance
› Different space requirements for maintenance, repairs, or
replacement
▪ Allows installation or replacement of critical components without
affecting operations
› Provides craftsmen the ability to work safely and with freedom of
motion, thus reducing the risk of human error
Support and Specialty Spaces

• Support and specialty spaces decrease risks
• IT hardware receiving, storage, and staging area provides place
for
› Uncrating equipment—keeps packing materials out of computer
room
› Testing equipment before putting it on critical power
› Storing supplies—keeps contaminants and combustibles out of
computer room and saves valuable computer room space
101
Support and Specialty Spaces

• Building automation system (BAS) control room
› Space for Facilities to monitor the data center infrastructure
• Command/disaster recovery center
› Dedicated space to work any data center issues
• Dedicated spaces for parts and tool storage
› Necessary to store critical spare parts and tools for quick response
› Allows for better management of these resources
• Engineering and Facility shop space
› Office space for Engineering support (Facilities, IT) is necessary to
enable quick response times to issues
› Shop space, if necessary, to repair equipment or build power whips
away from the computer room floor
Commissioning
• A process designed to test both individual components and
whole systems to ensure that they operate as the designer
intended
• Commissioning (Cx) is the ‘official’ beginning of the facility hand
off from the Contractor and Designer to Maintenance and
Operations
• A commissioning agent (CxA) is generally appointed to lead the
team through the commissioning process
102
Commissioning
• Commissioning is not only for new data center builds
• Scalable infrastructure designs need to be commissioned when
they are installed, even though there is critical IT load
› This needs to be considered during the design phase in order to
minimize the risk to the critical IT load
• Re-commissioning should take place after major repairs to
critical systems
103
CxA
• The CxA should be hired during the design phase to begin
coordinating and planning the commissioning process
• Most effective when the CxA is an unbiased, third party who
works directly for the owner
• Responsible for the following
› Directing the planning of all phases of commissioning
› Scripting all tests for the Integrated Systems Testing (IST)
› Gathering documentation from all sources and compiling into a
final commissioning report
Levels of Commissioning
Level 5: Integrated
Systems Testing (IST)
Level 4:
Functional
System Testing
(FST)
Level 3:
Functional Testing
Level 2: Receipt,
Installation, and Post-
Installation Testing
Level 1: Factory
Witness Testing
(FWT)
104
• Level 1: Factory Witness Testing (FWT)
› Factory engineers test equipment in a controlled environment
against national standards (e.g., ASHRAE, ARI, ANSI, NETA, etc.)
› Can be witnessed by the owner if desired
• Level 2: Receipt, Installation, and Post-Installation Testing
› Contractor installs the equipment and ensures that the individual
components are installed per the design drawings
• Level 3: Functional Testing
› Contractors test very basic operation of individual components
(i.e., power on, proper rotation, etc.)
• Level 4: Functional System Testing (FST)
› Factory representatives come on site to start up and test
equipment against company standards
› Test and Balancing (TAB)
› Generally starts the warranty period
• Level 5: Integrated Systems Testing (IST)
› Full systems are tested against the documented Sequences of
Operation
› Tests prove that each and every component works within the
system as it is supposed to
› Should be tested against all possible operating
conditions/situations
105
Commissioning
Failure Rate Bathtub Curve
Initial Useful Life End of Life

Mortality
Time
Commissioning must surpass the initial mortality period
Problems Found in the Commissioning

of Tier III and Tier IV Data Centers
• Engine generator coolant and oil leaks
• Inability of engine generators to produce stated capacity at
ambient conditions
• UPS inverter explosion
• Loss of power source to mechanical controller enabled the
mechanical plant emergency off system
• Explosion of current transformer in switchgear
• Air handling unit mode change from free cooling to recirculation
was not Fault Tolerant
106
Data Center Site Selection

• Site Location issues include
› Risks posed by natural disasters
› Risks posed by man-made features, which can change over time
• Investigate site risks
› Online tools
› On-site and area investigation
Data Center Site Selection

• Risk mitigation methodologies should be employed when and
where possible, consistent with business objectives
107
Operational Sustainability Recap

• Start With The End In Mind
• Building Characteristics impact Operational Sustainability
› Infrastructure to support operations
› Ease of maintenance
› Support and specialty spaces
• An intensive commissioning program is the key to successfully
bringing a new data center into operation
• Site Location
› Identify and mitigate risk
108
Session 7
109
Session 7 Myths and Misconceptions
Tiers Does Not Require a Raised Floor
• Uptime Institute:
› Choice of cooling delivery (underfloor or overhead) is an owner
decision based on operational preference
› Decisions such as: raised floor or on-slab, Cold Aisle/Hot Aisle,
containment of Cold Aisle/Hot Aisle, or gallery cooling can effect
the efficiency of computer room cooling, but none of them are
mandated by Tiers
For Tier III or IV, the Engine-Generator Plant

Must Provide Power to the Load at all Times
› Tiers does not require that the engine-generator plant run at all
times
› Data centers will typically utilize the public utility a majority of the
time
› However, the engine-generator plant must be properly configured
and sized to carry the critical load without runtime limitations
› Meeting the above criteria requires special attention to engine-
generator capacity ratings and power distribution
110
Tiers has No Physical Security Requirements
› The level of security of a data center is largely determined by
▪ Industry practices and requirements
▪ The criticality of IT functions preformed at the site
▪ Owner’s security policies
› A number of physical security measures that are best practices
should not be confused with Tier requirements
› Access Control is an expected Behavior for Operational
Sustainability
Tier IV is the Best
› Tiers reflects a performance-based, business-objective-driven,
data center topology benchmarking system
› An organization’s tolerance for risk determines the appropriate Tier
to support the business objective
› If no business objective is defined, then often Tiers is misused to
rationalize unnecessary investment
111
Utility Feeds Determine Tier Level
› The number of utility feeds, substation, and power grids that
provide public power to a data center neither predicts nor
influences Tier level
› Utility power is not even required for Tiers
Component Count Determines Tier Level
› N+1, N+2, N+N, or 2(N+1) does not determine Tier level
› It is possible to achieve Tier IV with just N+1 components for some
systems
112
Site Location Affects Tier Level

› Although a critical consideration for the life-cycle operation of the
facility and in determining, evaluating, and mitigating risk to the
data center, the geographical location does not affect Tier
› Site location does impact Operational Sustainably
113
Session 8
114
Session 8 Design Review Exercise
Tier-Based Design Review

• Lesson Objectives
› This session will be a mentored exercise to review drawings
submitted for Uptime Institute Tier Certification
› Individual drawings are used by the permission of the copyright
owner for
the ATD course and shall not be removed from the room
› Return all drawings at the completion of the exercise to Uptime
Institute staff
Exercise Format
• Form teams of 4 or 5
› Multiple discipline teams are best
• Review of 1st assigned system
› Allocate 25 minutes for review
› Take 5 minutes to confirm notes for presentation
• Present summaries
› Each team allocated 10 minutes to present summary
• Review of 2nd assigned system
› 4 sets of drawings; allocate 25 minutes for each review
› Take 5 minutes to confirm notes for presentation
• Present summaries
› Each team allocated 10 minutes to present summary
• Discussion may continue after the session, but drawings must remain
in the room
115
Review Objectives
1. Determine the design Tier intent
2. Determine highest level of Tier compliance
3. Identify characteristics that must be modified to comply
with next Tier
4. Identify characteristics that exceed Tier level determined in
Step 2
D1-1 first
Stage 1 Review Objectives

1. Group leaders will group, and decide on one outcome
2. Group Leaders will return to discuss changes with their team
3. Group Leaders will present upfront on screen as 1 new team
116
Stage 2 Review Objectives

1. New groups with the leaders forming a new team
2. Then in each group, each person will be in charge of
presenting at least 1 drawing (e.g., C1-1)
3. Team leaders will present upfront on screen as 1 new team
117
Abbreviations
118
Abbreviation List
–A–
ACC Air-Cooled Condenser
ACE Assistant Chief Engineer
AFF Above Finished Floor
AHJ Authority Having Jurisdiction
AIR Abnormal Incident Report
AL Advanced Level (of mitigation)
ANSI American National Standards Institute
AOS Accredited Operations Specialist
APZ Accident Potential Zone
ASHRAE American Society of Heating, Refrigerating and Air-Conditioning Engineers
ATD Accredited Tier Designer
ATS Accredited Tier Specialist
ATS Automatic Transfer Switch
–B–
BAS Building Automation System
BC Building Characteristics (element of Operational Sustainability)
BL Basic Level (of mitigation)
BMS Building Management System
BOD Board of Directors
–C–
CapEx Capital Expenditure, Capital Expense
CE Chief Engineer
CEO Chief Executive Officer
CFD Computational Fluid Dynamics
CFO Chief Financial Officer
CIO Chief Information Technology Officer
COO Chief Operating Officer
CM Corrective Maintenance
CMMS Computerized Maintenance Management System
CRAC Computer Room Air Conditioner (DX)
CRAH Computer Room Air Handler (Chilled Water)
CRCU Computer Room Cooling Unit
CRE Corporate Real Estate
CTO Chief Technology Officer
Cx Commissioning
CxA Commissioning Agent
119
–D–
DB Dry Bulb Temperature
DC Data Center
DDC Direct Digital Controls
DR Disaster Recovery
DRUPS Diesel (or Dynamic) Rotary Uninterruptible Power Supply
DX Direct Expansion
–E–
EOP Emergency Operating Procedure
EoR or EOR Engineer-of-Record
EPA Environmental Protection Agency (U.S.)
EPO Emergency Power Off
EWT Entering Water Temperature f
–F–
FAA Federal Aviation Administration (U.S.)
FM Facility Manager
FORCSS Financial, Opportunity, Risk, Compliance, Sustainability, Service Quality
ft2 square foot
FTE Full-time Equivalent (full-time employee)
FWT Factory Witness Test
–G–
GPM Gallons Per Minute
–H–
HQ Headquarters
HR Human Resources (personnel management department)
HRO High Reliability Organization
HVAC Heating, Ventilation, Air Conditioning
–I–
ID Identification/Identify
IEEE Institute of Electrical and Electronics Engineers
IL Intermediate Level (of mitigation)
IR Infrared
ISO International Organization for Standardization
IST Integrated Systems Test
IT Information Technology
–K–
KPI Key Performance Indicator
kVA Kilovolt Amperes
kW Kilowatts
120
–L–
LAP Local Administrative Procedure
LEED Leadership in Energy and Environmental Design (U.S.)
–M–
M&O Management & Operations (element of Operational Sustainability)
M/H Man-Hours
MBWA Management By Walking Around
MCC Motor Control Center
MMS Maintenance Management System
MOP Method Of Procedure
m/s Meters per second (m/s2, meters per second squared)
MTBF Mean Time Between Failuers
MTR Mean Time to Restore
MTS Manual Transfer Switch
MW Megawatt
MWO Maintenance Work Order
–N–
N Nominal Capacity or Units
NETA InterNational Electrical Testing Agency
NOC Network Operations Center
NW Northwest
–O–
O&M Operations and Maintenance
OEM Original Equipment Manufacturer
OJT On-the-Job Training
OpEx Operating Expense
OPR Owner’s Project Requirements
OS Operational Sustainability
OSHA Occupational Safety and Health Administration (U.S.)
–P–
PDU Power Distribution Unit
PE Professional Engineer
PF Power Factor
PM Preventive Maintenance
PMP Program Management Plan
POP Point Of Presence
PTS Point-of-use Transfer Switch
PUE Power Usage Effectiveness
121
–R–
R Redundant Capacity or Units
RACI Responsible, Accountable, Consulted, Informed
RFP Request For Proposal
ROI Return on Investment
R&R Remove and Replace
RH Relative Humidity
RTO Return To Operations
–S–
SCADA Supervisory Control And Data Acquisition
SCP Site Configuration Procedure
SL Site Location (element of Operational Sustainability)
SLA Service Level Agreement
SOP Standard Operating Procedure
SOW Scope of Work
STS Static Transfer Switch
SW Southwest
SZ Seismic Zone
–T–
TAB Test, Adjust, & Balance
TCCF Tier Certification of Constructed Facility
TCDD Tier Certification of Design Documents
TCOS Tier Certification of Operational Sustainability
TIA Telecommunications Industry Association
TST Thermal Storage Tank
–U–
UCU UPS Cooling Unit
UFAD Underfloor Air Distribution
UL Underwriters Laboratories
UPS Uninterruptible Power Source
U.S. United States
US$ United States Dollars
–W–
WB Wet Bulb Temperature
WH Water Heater
WO Work Order
122
Contacts
123
Course Contacts
General Information
5470 Shilshole Avenue NW
Suite 500
Seattle, WA 98107
+1 206.783.0510 (Seattle, WA)
info@uptimeinstitute.com
Dr. Hussein Shehata Cynthia Bendula

Director – Education Program Membership and Education Program Manager
Dubai, UAE Seattle, Washington
+971 55 2834202 +1.206.294.7643
+971 4 452 0940 ext 202 cbendula@uptimeinstitute.com
hshehata@uptimeinstitute.com
Mozart Mello Alexey Solodovnikov

Managing Director – Brasil Managing Director – Russia and CIS
Brasil Russia
+55.11.99993.0043 +7.495.225.9393
mmello@uptimeinstitute.com asolodovnikov@uptimeinstitute.com
http://uptimeinstitute.com/brasil http://uptimeinstitute.com/russia
Philip Hu Ali Moinuddin

Managing Director – North Asia Managing Director – Europe
Taiwan United Kingdom
+886.932.031.088 +44.7973.629.704
phu@uptimeinstitute.com amoinuddin@uptimeinstitute.com
Matt Stansberry Patrick Chan

Vice President, North America Vice President, South Asia
North Carolina United Kingdom
+1.720.214.6634 +6012.2063288
mstansberry@uptimeinstitute.com pchan@uptimeinstitute.com
124
Additional Learning Opportunities
Advanced Credentials
Each of our three core programs is designed to allow students to start with the essential knowledge they need to do
their current jobs, and then grow in their ability to meet more demanding challenges over time. Once enrolled in a
specific program, successful completion of initial courseware will result in the student’s base-level credential being
authorized, “Accredited”. Through additional course material completion and associated evidence submission
requirements, the student can gain “Professional” and then “Expert” designations based upon their demonstrated
mastery of the concepts. This applies to all designers, specialists and operations focused individuals.
https://uptimeinstitute.com/education/accredited-training/advanced-seminars
Continuing Professional Development (CPD)

Uptime Institute has earned CPD Accreditation through the CPD Standards Office. Uptime Institute Accredited Training
Courses qualifies as continuing education credits toward earning and maintaining professional certifications and
designations associated with your profession.
https://uptimeinstitute.com/education/overview
Webinars
The Uptime Institute webinar series aims to share that industry knowledge with the broader community to help drive data
center reliability, sustainability and efficiency.
https://uptimeinstitute.com/webinars
Network
The Uptime Institute Network offers meaningful peer-to-peer interaction and a forum for knowledge transfer free from the
influence of vendors or concern over trade secrets. Members benefit from sharing experiences, lessons learned, and solutions.
https://uptimeinstitute.com/ui-network/network-overview
Events
Uptime Institute is committed to sharing knowledge, insights, and proven methodologies with the industry at large–both at our
own corporate events and by participating as thought leaders in events around the globe.
https://uptimeinstitute.com/events
125
Uptime Institute is a division of The 451 Group, a leading technology
industry analyst and data company. Uptime Institute has office
locations in the U.S., Mexico, Costa Rica, Brazil, U.K., Spain, U.A.E.,
Russia, Taiwan, Singapore, and Malaysia.
Visit www.uptimeinstitute.com for more information.
© 2022 Uptime Institute, LLC. All rights reserved
126

ATDEN22 021ebook

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

ATDEN22 021ebook

Uploaded by

Copyright:

Available Formats

Accredited Tier Designer

Copyright ©2009-2022 by Uptime Institute, LLC

All rights reserved.

Uptime Institute Education....................................................................................................................4

Session 1: Tier Standard: Topology......................................................................................................6

Session 2: Cooling Systems................................................................................................................22

Session 3: Electrical Systems.............................................................................................................42

Session 4: Ancillary Systems..............................................................................................................65

Session 5: Common Discrepancies....................................................................................................85

Session 6: Design Impacts to Operational Sustainability................................................................95

Session 7: Myths and Misconceptions.............................................................................................109

Session 8: Design Review Exercise.................................................................................................. 114

Additional Learning Opportunities...................................................................................................125

© 2009-2022 Uptime Institute, LLC. All rights reserved

© 2009-2022 Uptime Institute, LLC. All rights reserved

• Tier Standard: Operational Sustainability

Both Are Owner Standards

Session 1 Tier Standard: Topology

Distinction Between Courses

• Accredited Operations Specialist (AOS) course is for data center

• Accredited Tier Specialist (ATS) course is for data center owners,

ATD Course Goals & Context

Accreditation is not an endorsement for Certification

Session 1 Tier Standard: Topology

The Concept of “N”

• “R” typically appears whenever N is stated

Session 1 Tier Standard: Topology

Tier Classification Genesis

Tier Classification Objectives

Session 1 Tier Standard: Topology

Tier Topology Categories

• Fractional “concepts” are not rationalized

• Differentiation within a Tier enhances Operational Sustainability

Key Tier Topology Principles

Session 1 Tier Standard: Topology

Session 1 Tier Standard: Topology

Tier I – Basic Capacity

• Operations and Maintenance Considerations

Session 1 Tier Standard: Topology

Tier I – Operational Risks

Tier II – Redundant Components

• Operations and Maintenance Considerations

Session 1 Tier Standard: Topology

Tier II – Operational Risks

Tier III – Concurrently Maintainable

• Operations and Maintenance Considerations

Session 1 Tier Standard: Topology

Tier III – Practical Insight

Tier III – Operational Risks

Session 1 Tier Standard: Topology

Tier IV – Fault Tolerant

• Operations and Maintenance Considerations

Tier IV – Practical Insight

Session 1 Tier Standard: Topology

Tier IV – Operational Risks

Public Utility Systems

Session 1 Tier Standard: Topology

Ambient Temperature Design Points

• Cooling coil capacity shall be determined at anticipated

Session 1 Tier Standard: Topology

Session 1 Tier Standard: Topology

Tier Classification Recap

Session 2 Cooling Systems