Uptime Institute’ Accredited Tier Designer Technical Paper Series: Continuous Cooling October 2017Abstract This technical paper clarifies the requirements for Continuous Cooling in the context of Uptime Institute's Tier Standard: Topology. Tier IV is the only Tier that requires Continuous Cooling However, Uptime Institute recommends Continuous Cooling at densities beyond 4 kilowatts (kW) per rack, regardless of the Tier level The Role of Continuous Cooling {As the power densities in data centers continue to increase, the need for Continuous Cooling becomes more profound. The risk of a loss of cooling during an uninterruptible power supply (UPS) ride-through event and the associated impact can be catastrophic to a business. IT equioment may fail or be damaged Depending on the cooling or UPS technology deployed in a facility, the requirements for Continuous Cooling can widely differ, This paper clarifies the definition of Continuous Cooling and details deployment considerations for various cooling technologies. Continuous Cooling is defined as the ability to provide a stable thermal environment for the critical IT equipment without any interruption. Continuous Cooling requires a consistent server inlet temperature for the amount of time it takes for the mechanical system to restart after any interruption of power to the cooling systems (including the time for a transfer to engine generators, if applicable). It also requires adequate maintenance of the as-designed Cold Aisle temperature, Maintaining a stable thermal environment helps mitigate the chances of sudden increases to the UPS output load Increases in load may occur due to increased server fan power consumption to help make up for a temporary loss of cooling. if the load is not well managed, the increase could overload the UPS, impacting the operation of the entire facility. ‘The American Society of Heating, Refrigerating, and Air-Conditioning Engineers (ASHRAE), in concert with major IT equipment manufacturers, established Thermal Guidelines for Data Processing Environments that include recommended computer equipment inlet air temperatures necessary to enable reliable operation of servers, storage equipment, and network devices. As of 2015, the ASHRAE Guideline (accepted worldwide) recommends that the device inlet be maintained between dry bulb temperatures of 18-27°C (66-81°F}, with dew point temperatures of -9°C to 16°C (-15-69°F) and relative humidity below 60% to meet the manufacturer's established criteria, It should be noted that the requirements of Continuous Cooling are independent of the ASHRAE Guideline for actual server inlet temperature. Compliance with the ASHRAE Guideline for server inlet temperature is an owner decision based on an organization's individual needs. However, Uptime Institute Continuous Cooling requirements are tied to the ASHRAE Guideline for allowable rate of change as it relates to the definition of a stable thermal environment. ASHRAE Guidelines state the maximum allowable change in niet temperature for IT equipment, For a data center with tape storage (specifically) the maximum allowable temperature change for typical IT equipment is limited to 5°C in an hour. All other IT equipment is limited to maximum allowable temperatures of up to 20°C in an hour. For a data center with any kind of devices, ASHRAE limits this rate of change to 5" within any 18-minute period. It is important to note that this \s not a rate of change, but rather a discrete temperature nerease/decrease limitation Continuous Cooling | October 2017‘A Continuous Cooling solution must be capable of providing the stable thermal environment for the entire amount of time it takes for the mechanical cooling system to restart. ‘These parameters have many impacts in application. As an example: Consider a data center with Cold Aisle containment where chilled water computer room air handling (CRAH) units operate on supaly air temperature control methodologies and with a normal set point of 20°C. Ifthe control deadband is set at 1°C, this means that during normal operations the unit would provide an IT equipment inlet temperature of 19-21°C, assuming no air mixing between the CRAH unit and the IT equipment. Ita faut or loss of utility power occurs while the unit is supplying 19°C air to the critical environment, and the fault causes the CRAH unit to increase its temperature to 24°C at its peak, then the inlet temperature cannot, within any 16-minute period, deviate between 19-24°C. This means that ifthe unit recovers within @ 15-minute period, the unit cannot avercoo! to the point where the supply air temperature drops below 19°C. Maintaining this temperature range necessitates close attention to the control algorithms used to direct the cooling of the CRAH units. Using the same example, if a fault were to instead cause a peak supply air temperature of 22°C before recovery, the recovery could cool down to 17°C in order to stay within the 5° deadband in any 15-minute period Consideration must also be given to the time required to restore mechanical cooling, While ASHRAE uses 15-minute periods to define the maximum allowed temperature changes, the Tier Standard: Topology additionally requires that a Continuous Cooling solution is capable of providing the stable thermal environment forthe entire amount of time it takes for the mechanical cooling system to restart following any interruption of the cooling production or the loss of ulilty power. The mechanical system restart timeis measured from the moment of the loss of utility up until the engine generators (or other on-site power production systems) start and close into the critical load, when the mechanical system has power restored, is in operation, and is providing rated cooling ina steady state ‘operating condition. For example, a chiler is not officially operational untilitis restarted and running with normal supply and return water conditions and flow rates Although manufacturers are reducing equioment restart times, the interval betwee! the loss of power and a resumption of the system's ability to produce stable cooling needs to be incorporated as a data point in determining the ride-through time, For example, ifit takes 10 minutes to resume stable mechanical cooling after a loss of power, then the thermal energy storage (TES) must be able to provide 10 minutes of chilled water storage. Although Tier IV is the only Tier level that requires Continuous Cooling, data centers with higher than average IT load densities should consider Continuous Cooling to mitigate large temperature increases due to a loss of utlity power or component failure. As a point of reference, Uptime Institute conducted a demonstration on a 6-kW/rack average computer room. Intake air temperatures in the computer room exceeded the top value in this range within 60 seconds after a loss of cooling or even after just loss of air movement, Additionally, this demonstration showed that a T-minute loss of cooling required 20 minutes to recover. Continuous Cooling | October 2017Continuous Cooling provides the bridge to enable the thermal environment to remain stable until the mechanical or other cooling system resumes, Consider the scenario of a public utility failure during which the UPS continues to power the IT devices, but mechanical plant operation is interrupted, Depending on the technology of the cooling deployed, this interruption may continue for several minutes. During this time, any elevation in temperatures in the computer room may damage IT equipment. Continuous Cooling provides the bridge to enable the thermal environment to remain stable until the mechanical or other cooling system resumes. A properly designed Continuous Cooling solution will prevent any increase in the average server inlet air temperature Application of Continuous Cooling Requirements Tier IV is the only Tier that requires Continuous Cooling, The following are a few ‘examples of ways to achieve Continuous Cooling. The Tier Standards are not prescriptive but provide outcome- and performance-based standards, thus other methods may exist to accomplish the goal of Continuous Cooling beyond those described below. Example 1: Continuous Cooling for a chilled water system is generally accomplished with TES capability (also known as chilled water storage). Secondary pumps and CRAHS are required to be on Fault Tolerant UPS power sources as well. The power source can be the IT UPS or a separate, Concurrently Maintainable and Fault Tolerant UPS system that is dedicated to the mechanical systems. If the cooling system is in a primary-direct configuration, then the primary pumps are required to be on a UPS, Adcitionally, consideration must be given to the thermal storage tanks and how they are connected to the chilled water distribution. Depending on the tanks’ ‘connection to the chilled water distribution system, supply and return (hence warmer) chilled water might be mixed, thereby potentially decreasing the amount of time that a chilled water storage tank can support a stable thermal environment, for the IT equipment. Example 2: Continuous Cooling for direct expansion (DX) systems requires both the computer room air conditioners (CRACs) and the external condensers to be on a Concurrently Maintainable and Fault Tolerant UPS system. Additionally, DX systems may require some additional engineering analysis to meet Continuous Cooling requirements. Compressor cycling during the normal operation of CRACS can cause the compressors to pause operation for several minutes for compressor protection. In higher than average density environments, these gaps may cause large temperature swings during both normal operations and during faults and loss of utility power, making the stable thermal environment more difficult to achieve. Continuous Cooling | October 2017Continuous Cooling is a justifiable safeguard for any facility with average densities above 4 kW in light of the potential damage to facilities and IT assets. Example 3: Continuous Cooling for evaporative systems such as direct evaporative or indirect evaporative heat exchangers, requires the water pumps (secondary or system Circulating) and the distribution fans to be on Fault Tolerant UPS power sources Example 4: Continuous Cooling for 100% outside air systems that can provide cooling throughout the year require the fans (or the systern that delivers the air to the ‘computer room) to be on Fault Tolerant UPS power sources. When rotary UPS systems are deployed as the IT UPS, and the cooling systems are on a no-break bus, additional Continuous Cooling measures may not be needed because the mechanical system would net experience an interruption during @ transfer from utilty to engine-generator (or other on-site) power. If there is a chilled water system deployed in tandem with a rotary UPS with no batteries, the site must demonstrate that the average server inlet temperature will not increase beyond the limits stated previously if there is no TES installed. Each specific case should be reviewed to ensure the requirement for a stable thermal environment will be met during a loss of power event Conclusion By providing thermal stability to the IT environment during any interruption in the cooling system, such as the transition from utility outage to engine-generator power, Continuous Cooling ensures that a utility event does not result in costly heat damage to IT hardware or critical equipment, nor increase the UPS oulput load to the point of averload, Continuous Cooling is a requirement only for Tier IV Certification, but is a justifiable safeguard for any facility with average densities above 4 kW in light of the potential damage to facilities and IT assets. Continuous Cooling | October 2017ATD Technical Paper Series: Continuous Cooling, Version 8. All updates specific to this version are effective October 2017, Related Publications Tier Standard: Topology Accredited Tier Designer Technical Paper Series Further information can be found at www.uptimeinstitute.com, About Uptime Institute Uptime Institute is an unbiased advisory organization focused on improving the performance, efficiency, and reliability of business critical infrastructure through innovation, collaboration, and independent certifications. Uptime institute serves all stakeholders responsible for IT service availability through industry leading standards, education, peer-to-peer networking, consulting, and award programs delivered to enterprise organizations and third-party operators, manufacturers, and providers. Uptime Institute is recognized globally for the creation and administration of the Tier Standards & Certifications for Data Center Design, Construction, and Operational Sustainability along with its Management & Operations reviews, FORCSS® methodology, and Efficient IT Stamp of Approval. Visit www. uptimeinstitute.com for more information. utes cision of The 451 Gros, a leadng technology indy anayet ne dat company. Uptime nati has ofcelocationsinthe US, Mex, faz UK span, URE, Rustin Tawan Sngapors a Maye Uptime ntti Le A ght reseved 1roe25.1.07 8