rrean016 Data Center Cooling: CRACICRAH redundancy, capacity, and selection metrics - Uptime rss evournal Uptime Institute Data Center Cooling: CRAC/CRAH redundancy, capacity, and selection metrics in Executive Striking the appropriate balance between cost and reliability is a business decision that requires metrics By Dr. Hussein Shehata This paper focuses on cooling limitations of down-flow computer room air conditioners/air handlers (CRACS/CRAHS) with dedicated heat extraction solutions in high-density data center cooling applications. The paper also explains how higher redundancy can increase total cost of ownership (TCO) while supporting only very light loads and proposes a metric to help balance the requirements of achieving higher capacities and efficient space utilization With several vendors proposing passive high-density technologies (eg, cabinet hot air removal as a total resolution to the challenge of high density), this analysis shows that such solutions are only possible for a select few cabinets in each row and not for full deployments. ‘The vendors claim that the technologies can remove heat loads exceeding 20 kilowatts (kW) per cabinet, but our study disproves that claim; passive-cooling units cannot extract more heat than the cold air supplied by the CRACs. For the efficient design of a data center, the aim is to increase the number of cabinets and the total IT load, with the minimal necessary supporting cooling infrastructure. hips curnal apt ensue comida cerer-coning-redundancy-capacity-selecton-metics! sr0rraan016 Data Center Cooling: CRACICRAK recusdancy, capacity, and selection mesrcs- Uptime Insti eJoural See Figure 1 + NUMBER OF CABINETS + LOAD PER CABINET ri (ay - NUMBER OF CRAC UNITS rt a ry BS https /ournal uptimeinstitute.com/iyp-content/uploads/20%4/1 1/shehataa pe! Figure 1, The relationship between IT and supporting spaces Passive Hot Air Removal Data center design continually evolves towards increasing capacity and decreasing spatial volume, increasing energy density. High-end applications and equipment have higher energy density than standard equipment; however, the high-performance models of any technology have historically become the market standard with the passage of time, which in the case of the IT industry is a short period. As an example, every 3 years the worla's faste supercomputers offer 10 times the performance of the previous generation, a trend that has been documented over the past 20 years. Cooling high-density data centers is mostly commonly achieved by: + Hot Air Removal (HAR) via cabinet exhaust ducts—active and passive See Figure 2 hips jcurnal apt ensue comida cerer-coning-redundancy-capacty-selecton-metics!rraan016 Data Center Cooling: CRACICRAH redundancy, capacity, and selection metrics - Uptime rss evournal ho timeinstitute.com/up-cont Figure 2, 84s Figure 2. HAR via cabinet echaust ducts (active and passive). Courtesy APC + Dedicated fan-powered cooling units (i.., chilled water cabinets). See Figure 3, thites Hiournal uptimeinstitute.com/up- ‘content/uploads/2014/11/. Hussein Figure. 3.jpe Figure 3. Dedicated fan-powered cooling wits This paper focuses on HAR/CRAC technology using an underfloor air distribution plenum, Approach High-density data centers require cooling units that are capable of delivering the highest cooling capacity using the smallest possible footprint. The high-powered CRACS in the smallest footprints available from the major manufacturer offer a net sensible cooling capacity of approximately 90 kW but require 3x1-meter (m) (width by depth) hips curnal apt ensue comida cerer-coning-redundancy-capacity-selecton-metics! 30rraan016 Data Center Cooling: CRACICRAK recusdancy, capacity, and selection mesrcs- Uptime Insti eJoural footprints. (Appendix C includes the technical specifications for the example CRAQ), Excluding a detailed heat load estimate and air efficiency distribution effectiveness, the variables of CRAC capacity, cabin quantity, and cabinet capacity may be related in the following Note: The formula is simplified and focused on IT cooling requirements, excluding other loads such as lighting and solar gains. CRAC Capacity = Number of IT cabinets x kW/cabinet (1) Example 1 for N Capacity: If a 90-kW CRAC cools 90 cabinets, the average cooling delivered per cabinet is 1 kW. 90 kW= 90 cabinets x 1 kW/cabinet (2) Example 2 for N Capacity: If a 90-kW CRAC cools two cabinets, the average cooling delivered per cabinet is 45 kW. 90 kW= 2 cabinets x 45 kW/cabinet (3) ‘The simplified methodology, however, does not provide practical insight into space usage and heat extraction capability. In Example 1, lone CRAC would struggle to efficiently deliver air evenly to all 90 cabinets due to the practical constraints of CRAC airflow throw; In most circumstances the cabinets farthest from the CRAC would likely receive less air then the closer cabinets (assuming practical raised: floor heights and minimal abstructions to under floor airflow), In Example 2, one CRAC would be capable of supplying sufficient cooling to both cabinets; however, the ratio of space utilization of the CRAG, service access space, and airflow throw buffer would result in a high space usage for the infrastructure compared to prime white space (IT cabinets), Other constraints, such as allocating sufficient perforated floor tiles/grills in case of a raised-floor plenum or additional Cold Aisle containment for maximum air distribution effectiveness may lead to extremely large Cold Aisles that again render the data center space utilization inefficient. hips jcurnal apt ensue comida cerer-coning-redundancy-capacty-selecton-metics!rraan016 Data Center Cooling: CRACICRAH redundancy, capacity, and selection metrics - Uptime rss evournal Raleod Floor Underder Supply Ale Coll Ase Wot (https-/Hournal uptime nbap-conten 4 Figure 4 imagesjpes Figure 4. Typieal Cod Aisle/tHot Aisle arrangement (ASHRAE TC3.9) ‘Appendix 8 includes a number of data center layouts generated to illustrate these concepts. The strategic layouts in this study considered maximum (18 m), average (14 m) and minimal (10 m) practical CRAC air throw, with CRACS installed perpendicular to cabinet rows on one and two sides as recommended in ASHRAE TC9.9, The front-to-back airflow cabinets are assumed to be configured to the best practice of Cold Aisle/Hot Aisle arrangement (See Figure 4), Variation in throw resulted in low, medium, and high cabinet count, best defined as high density, average density, and maximum packed (high number of cabinets) for the same data center whitespace area and electrical load (see Figure 5) Inthe example layouts, CRACS were placed close ‘together, with the rinimal 500- millimeter (ren) maintenance space on one side and 1,000 mm on the long side (see Figure 6). Note that each CRAC manufacturer might have different unit ener a clearance requirements. A minimal 2-m buffer between the nearest cabinet hitps-Hhournal uptimeinstitute.com/up. ontent/uploads/2014/1 1.-Hussein Figures: hips jcurnal apt ensue comida cerer-coning-redundancy-capacity-selecton-metics!rraan016 Data Center Cooling: CRACICRAH redundancy, capacity, and selection metics - Uptime rstute eJournal tmageS.jpel and each CRAC Figure 5. CRAC throw area unit prevents entrainment of warm air into the cold air plenum. Cold and Hot aisle widths were modeled on approximately 1,000 mm (hot) and 1,200 mm (cold) as. recommended in ASHRAE TC9.9 literature. In the context of this study, CRAC footprint is defined as the area occupied by CRACS (including maintenance and airflow throw buffer); cabinet footprint is defined as the area occupied by cabinets {and their aistes). These two areas have been compared to analyze the use of prime footprint within the data center hall. Tier level requires each and every power and cooling component and path to fulfil the Tier requirements; in the context of this paper the redundancy configuration reflects the Tier level of CRAC capacity components only, excluding considerations to other subsystems required for the facility's operation. Tier | would not require redundant components, hence N CRAC units are employed. Tiers Il, Ill and IV would require redundant CRACS; therefore N¥1 and N+2 configurations were also considered, Abasic analysis shows that using aCRAC as described above would require a 14-m? area (including throw buffer), which would generate 25.7 kW of cooling for every 1m of active CRAC perimeter at N redundancy, 19.3, KW for one-sided N+ redundancy and two-sided N42 redundancy, 22.5 kW for two-sided N+1 redundancy, and 12.9 kW for one-sided N12 redundancy. However, data center halls are not predominantly selected and designed based on perimeter length, but rather on floor area images.joal Figure 6. CRAC maintenance zone ‘The study focused on identifying the area required by CRAC units, compared to that occupied by IT cabinets, and defines it as a ratio. Figure 7 shows Tier I(N) one-sided CRACs in a high-density cabinet configuration. Appendix A includes the other configuration models. hips curnal apt ensue comida cerer-coning-redundancy-capacity-selecton-metics! 520rraan016 Data Center Cooling: CRACICRAH redundancy, capacity, and selection metrics - Uptime rss evournal Furthermore, a metric has been derived to help determine the appropriate cabinet footprint at the required Tier level (considering RAC redundancy only). institute. com/up-conte 84 As Figure, Figure 7. Tier 1 (N) one-sided CRACS in high-density cabinet configuration Cabinet capacity to footprint factor C2F= kw/cabinet / C2C (4) Where CRAC to Cabinet factor C2C= CRAC footprint / Cabinet footprint (5) For multiple layout configurations, the higher the C2F, the more IT capacity can be incorporated into the space. Higher capacity could be established by more cabinets at lower densities or by fewer cabinets at higher densities. However, the C2F is closely linked to the necessary CRAC footprint, which as analyzed in this paper, could be a major limiting factor (see Figure 8). ow w/eabinet {hitps:/fournal uptimeinstitute, comvup-content/uploads/2014/1 1/shehatab jog} Figure 8. CoF versus cabinet load (kW) for various CRAC redundancies hips curnal apt ensue comida cerer-coning-redundancy-capacity-selecton-metics! m0Troe Data Certr Coalng: CRAGICRAK recundaney, capacity, and selection meres - Ustime tte evourral Results The detailed results appear in Appendix B. The variations analyzed included reference CRACs with no redundancy, with one redundant Unit, and with two redundant units. For each of the CRAC configurations, three cabinet layouts were considered: maximum packed, average density, and high density). Results showed that the highest C2F based on the six variations within each of the three redundancy configurations is as follows: + Tier | (N}-one-sided CRAC deployment: C2F= 13, + Tier IHIV (N+1)}-two-sided CRAC deployment: C2F = 11 + Tier IHIV (N+2 and above)-two-sided CRAC deployment: C2F= 9.8 ‘The noteworthy finding is that the highest C2F in all 18-modeled variations was for high-density implementation and at a CRA cabinet (C20) area ratio of 0.46 (Le., CRACS occupy 32% of the entire space) and a cabinet footprint of 2.3 m? per cabinet. This is supporting evidence that, although high-density cabinets would require more cooling footprint, high density is the most efficient space utilization per kW of IT Example 3 illustrates how the highest C2F on a given CRAC redundancy and one- or two-sided layout may be utilized for sizing the footprint and capacity within an average-sized 186-m? data center hall for a Tier ILIV (N#2, C2F=9.8, C2C=0.5, and cabinet, footprint of 2.3 m2) deployment. The space is divided into a net 124- rm? data hall for cabinets, and 62 m? of space for CRAC units by Utilizing the resutting ideal C2C of 0.46. Example 3: ff a net 124-m? data hall for cabinets and 62 m? of space for CRAC units is available, the highest achievable capacity would be 45 kWicabinet. 9.8= 4.5 kWicabinet/59 m? ; 127 m? (6) To determine the number of cabinets and CRACs, the CRAC cooling capability will be used rather than the common method of dividing the area by cabinet footprint. ‘The total area occupied by a CRAC is 14 m2; hence approximately four CRACs would occupy the 59-m2 space. Two CRACS are duty, since N+2 is utilized; therefore, the available capacity would be 90 kW x 2.= 180 kW. The number of cabinets that could then be installed in this 186-mn2 t tal area would be 180/4.5 = 40 cabinets. Thi tal effective space used by the 40 cabinets is 92 m2 (40 x 2.3 m? ) that is 72% of the available cabinet dedicated area. This shows hips curnal apt ensue comida cerer-coning-redundancy-capacity-selecton-metics! a0Troe Data Certr Coalng: CRAGICRAK recundaney, capacity, and selection meres - Ustime tte evourral that higher redundancy may be resilient but does not fully utilize the space efficiently. This argurnent highlights the importance of the debate between resilience and space utilization, Example 4 illustrates how C2F may be utilized for sizing the footprint and capacity within the same data center hall but at a lower redundancy of N#1 configuration Example 4: By applying the same methodology, the highest achievable capacity would be 5.2 kW/cabinet. The total area occupied by a CRAC is 14 m? (including CRAC throw and maintenance); hence approximately four CRACs would occupy 59 m? of space, Three CRACS would be on duty, since N+1 is utilized; therefore, the available capacity would be 90 KW x 3 = 270 kW. The number of cabinets that could then be installed in this 186-m? total area would be 270/5.2 ~ 52 cabinets. The total effective space used by the 52 cabinets is 120 m? (52x 2.3 rm? ), which is 95% of the space. The comparison of Example 3 to Example 4 shows that less redundancy provides more efficient space utilization fntps:/iournal,uptimeinsttute.com/iyp-content/uploads/2014/1 Vi) Hussein. (Gizwre-2-imazescopv ioe! igure 9, Summary ofthe results ‘The analysis shows that taking into consideration the maximurn C2F results obtained for each redundancy type and then projecting output on a given average load per cabinet, an example average high-density cabinet of 20 kW would require the CRAC units to occupy double the IT cabinet space in an N+2 configuration, hence lowering the effective use of such prime IT floor space (See Figure 9). hips curnal apt ensue comida cerer-coning-redundancy-capacity-selecton-metics! 90Troe Data Certr Coalng: CRAGICRAK recundaney, capacity, and selection meres - Ustime tte evourral Additional Metrics Additional metrics for design purposes have been derived from the illustrated graphs and resultant formulae. ‘The derived formula could be documented as follows: PEK/L+M46.4 x RIS) (8) Where P = Cooling per perimeter meter (kW/m) K = CRAC net sensible capacity (KW) L=CRAC length (m) M= CRAC manufacturer side maintenance clearance (rm) R= CRAC redundancy S=One- or two-sided CRAC layout Condusion Approximately 50% (270 KW/180 kW) more capacity, 30% more cabinets, and 16% higher-cabinet load density could be utlized in the same space with only one redundant CRAC and may stil fulfil Tier ILIV component redundancy requirements. This is achievable at no additional investment cost as the same number of CRACS (4) is installed within the same available footprint of 2,000 ft2, The analysis also showed that the highest average practical load per cabinet should not exceed 6 KW if efficient space utilization is sought by maintaining a C2C of 0.46. This study shows that an average high-density cabinet load may not be cooled efficiently with the use of only CRACs or even with CRACs coupled with passive heat-extraction solutions. The data supports the necessary implementation of row- and cabinet-based active cooling for high-density data center applications, The first supercomputers used cooling water; however, the low- density data centers that were commissioned closer to a decade ago (below 2 KW per cabinet) almost totally eliminated liquid cooling, This was due to reservations about the risks of water leakage within live, critical data centers. Data centers of today are considered to be medium-density facilities. Some of these data centers average below 4 kW per cabinet, Owners and operators that have higher demands and are ahead of the average market typically dedicate only a portion of the data center space to high-density cabinets. With server density increasing every day and high-density cabinets (approaching 40 kW and above) becoming a potential future deployment, data centers seem likely to experience soaring heat loads that will demand comprehensive liquid-cooling infrastructures, hips curnal apt ensue comida cerer-coning-redundancy-capacity-selecton-metics! 10720rraan016 Data Center Cooling: CRACICRAH redundancy, capacity, and selection metrics - Uptime rss evournal With future high-density requirements, CRAC units may become secondary cooling support or even more drastically, CRAC units may become obsolete! Appendix A {https /Hournal uptimeinstitute.com/up-content/uploads/2014/1 1/- Hussein. AppendieAt.ioe] “Appendix At. One-sided CRAC, maximum-throw, maxximum-packed cabinets hips jcurnal apt ensue comida cerer-coning-redundancy-capacty-selecton-metics! 0rraan016 Data Center Cooling: CRACICRAH redundancy, capacity, and selection metrics - Uptime rss evournal ix-A2-image 4 Appendix A2. One sided CRAC, average-throw, meatum cabinets {https /iournal uptimeinstitute,comvp-content/uploads/2014/11/I- Hussein. Append A3image15.pe1 Appendix Ag. One-sded CRAC, minimum-throw high-density cabinets hips jcurnal apt ensue comida cerer-coning-redundancy-capacty-selecton-metics!rraan016 Data Center Cooling: CRACICRAH redundancy, capacity, and selection metrics - Uptime rss evournal {https:/hournsl uptimeinstitute.corv/up-content/uploads/2014/11/ Hussein. AppendicAdjpg] Appendix Ag, Two-sided CRAC, maximum-throw, maximum-packed cabinets. hips jcurnal apt ensue comida cerer-coning-redundancy-capacty-selecton-metics! +320rraan016 Data Center Cooling: CRACICRAH redundancy, capacity, and selection metrics - Uptime rss evournal {bitps:/hournal uptimeinstitute.comvup-content/uploads/2014/1 1} Hussein: Aopendix-AS-imaze17 ipa Appendix As. Two-sided CRAC, average throw, medium packed cabinets hips jcurnal apt ensue comida cerer-coning-redundancy-capacty-selecton-metics! +420rraan016 Data Center Cooling: CRACICRAH redundancy, capacity, and selection metrics - Uptime rss evournal [hetps /iournal uptimeinstitute.com/up-content/upioads/2014/1H)LHUssen Aovendic AS-imazet ine Appendix A6. Two-sided CRAC, minimum-throw, high density eabinets ‘Appendix 8 ttps://journal_ uptimeinstitute, com/wp-content/uploads/2014/11/ hips curnal apt ensue comida cerer-coning-redundancy-capacity-selecton-metics! +20rraan016 Data Center Cooling: CRACICRAH redundancy, capacity, and selection metrics - Uptime rss evournal Hussein-Appendix-B1b-image19,ipg] {https /ournal uptimeinstitute,com/up-content/uploads/2014/1 1]. HUssein- Appendix B13-image20 jpg] Appendix Br. Tier I(N) CRAC modeling results Note 1: HD = High Density Note 2: MP = Max Packed Note 3: *= CRAC Area includes maintenance and throw buffer Note 4:6 = 27 m? area is deducted from total area, as itis already included in the throw buffer https://journal.uptimeinstitute.com/wp-content/uploads/2014/11/ Hussein.Appendix.B2b.image?1 Jog! {hips Hiournal uptimeinstitute,com/up-content/uploads/2014/1 1) Hussein. ‘Aopencix-623-image22 jpg] Appendix Be. Tier I-IV (N+2) CRAC modeling results hips curnal apt ensue comida cerer-coning-redundancy-capacity-selecton-metics! +620rraan016 Data Center Cooling: CRACICRAH redundancy, capacity, and selection metrics - Uptime rss evournal Note 1: HD = High Density Note 2: MP = Max Packed Note 3: * = CRAC Area includes maintenance and throw buffer Note 4: = 27 m? area is deducted from total area, as itis already Included in the throw buffer ournal, cont I ay https: /ournal uptimelnstitute,com/up-content/uploads/2014/1 1). HUsseln. Appendix B33-image?4 pz] Appendix Bg. Tier I-IV (N2) CRAC modeling results ‘Appendix https://journal.uptimeinstitute.com/wp-content/uploads/2014/11/ ssi XC image?5.i hips curnal apt ensue comida cerer-coning-redundancy-capacity-selecton-metics! s7e0rraan016 Data Center Cooling: CRACICRAH redundancy, capacity, and selection metrics - Uptime rss evournal ‘ais Conte Dara https://journal.uptimeinstitute, com/wp-content/uploads/2014/11/ Hussein-Appendix-Cé-image?8.ipg] s//journal com/wp-content/uploads/2014 Hussein-Appendix-C3-image27. Liebert CRAC Technical Specification Note: Net sensible cooling will be reduced by 7.5 kW x 3 = 22.5 kW hips curnal apt ensue comida cerer-coning-redundancy-capacity-selecton-metics! +820rraan016 Data Center Cooling: CRACICRAK recusdancy, capacity, and selection mesrcs- Uptime Insti eJoural for fans; 68.7 kW for Model DH/VH380A https://journal.uptimeinstitute.com/wp- content/uploads/2014/1 1/hussein jog’ Dr Hussein Shehata, BA, PhD, CEng, PGDip, MASHRAE, MIET, MCIBSE, is the technical director, EMEA, Uptime Institute Professional Services (UIPS). Dr Shehata is a U.K. Chartered Engineer who joined Uptime Institute Professional Services in 2011. He is based in Dubai, serving the EMEA region. From 2008-2011, Hussein was vice president & AsiaPacific DC Engineering, Architecture & Strategy Head at JP Morgan in Japan. Prior to that, he co-founded, managed, and operated as a subject matter expert (SME) at PTS Consulting Japan, He graduated in Architecture, followed by a PhD in HVAC, and a diploma in Higher Education that focused on multi-discipline teaching, with a focus on Engineers and Architects Share this entry f v & in és = You might also like RT Miya Putting Proper Data TELUS Identifyfetainer How Russia's DCIMto Data Center SIDCS Lurking Improvebusines: First Work Center Outages provide Vulneral the practice TierIV forvYou Staffing Incident supnort inthe Effective can Cortifica isKey and fo Worlds ofEC impar of to Industrywidespr Best» plugs data Design Reliable TranspaCanadia Run center Docume Operatic network Data availabil Centers hips curnal apt ensue comida cerer-coning-redundancy-capacity-selecton-metics! 1920rraan016 Data Center Cooling: CRACICRAH redundancy, capacity, and selection metrics - Uptime rss evournal hips curnal apt ensue comida cerer-coning-redundancy-capacity-selecton-metics! 200