You are on page 1of 78

Front cover

Draft Document for Review January 30, 2018 12:30 pm REDP-5472-00

IBM Power System AC922
Introduction and Technical
Overview
Alexandre Bicas Caldeira

Redpaper

Draft Document for Review January 30, 2018 12:30 pm 5472edno.fm

International Technical Support Organization

IBM Power System AC922 Introduction and Technical
Overview

October 2017

REDP-5472-00

2018 12:30 pm Note: Before using this information and the product it supports.fm Draft Document for Review January 30. Note to U.S. © Copyright International Business Machines Corporation 2017. First Edition (October 2017) This edition applies to the IBM Power System AC922 server models 8335-GTG and 8335-GTW. Government Users Restricted Rights -.Use. All rights reserved.5472edno. duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp. read the information in “Notices” on page v. . This document was created or updated on January 30. 2018.

. . . . . . . . . . . . . . . . . . . . . . . . . . .v Trademarks . . . . . . . . . . . . . . . . . . . . . . . .4. . . . . . . . . . . . . . . . . . . . . . . . . .4 I/O subsystem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .1 Memory features and placement rules .7 Internal storage . . . . .1 POWER9 processor overview. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .2. . . . .3 Minimum features . . . . . . . . . . . . . . .1 PCIe . . . . . . . . . . . . . . . . . . . . . . . . . . . . . iii . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 1. . . . . . . . . . . . . . . . . . . . . . viii Stay connected to IBM Redbooks . . . . . . . . . . . . . . . . . . . . . . . . . .2. . . . . . . . . . . . . . . . . . . . . . . . . . . . . .2 Power AC922 server model 8335-GTW . . . . .5 PCI adapters . .8 External I/O subsystems . . . . . . . . . . . . . . . . . . . . . . . . . vii Now you can become a published author. . . . . . . . . . . . . . . . too! . . . . . . . . . . . . . . . . . . . .5 NVLINK 2. . . . . . . . . . . . . . . . . . . . . . . . . . . . 9 2. . .4 The NVIDIA Tesla V100 . . . . . . 33 2. . . . . . . . . . . . . . .Draft Document for Review January 30. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .6 Flash storage adapters . . . . . . . . . . . . . . . . . . . .3 Fibre Channel adapters . . . . . . . . . . . . .4. 28 2. . . . . . . . . . . . . . . . . . . . .1 Key server features . . . . . . .2 Processor subsystem . . . . .5. . . . . . . . . . . . . . . . . . . . . 15 2. . . 36 2. . .10 IBM System Storage . . .5. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .4. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .5 Compute intensive accelerator . .4. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .1 System architecture . . . . . . . . . . . . . . . . . . . . . . . . . . 6 Chapter 2. . . . . . . . . . . . . . .3 Memory subsystem . . . . . . . . . 36 2. . . . . . . . . . . . . . . . . . . . . .4. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . viii Chapter 1. . . . . . . . . . . . . . . . . . . . . . . . .2 Red Hat Enterprise Linux . . . . . . . . . . 4 1. . . . . . . . . . . . . . . . . . . .2. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32 2. . . . . . .fm Contents Notices . . . . . . .11 Operating system support . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .5. . . . . . . . . . . . . . . . . . . . . . 20 2. .5. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34 2. . . . . . . . . . . . . . . . . . . . . . . . viii Comments welcome. . . . . . . . . . . 15 2. . . . . . . . . 35 2. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . System architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . vii Authors . . . . . .2 Memory bandwidth . . . . . . . . . . . . . . . 18 2. . . . . . .1 Power AC922 server model 8335-GTG . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19 2. . . 1 1. . . . . . . . . . . . . . . . . . . . . . .7. . . . . . . . . . 36 2. . . . . . . . . .5. . . . . 33 2. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .5. . . . . . . . . . . . . . 33 2. . . . . . . . . . . . 22 2. . . . . . . . .3 Additional information . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Product summary . . . . . . . . . . . . . . . . . . . . . . . 32 2. . . . . . . . . . .2 IBM CAPI2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .2. . . . . . . . . . . . . . . . . . . . .1 Slot configuration . . . . . . . 25 2. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .9 Location Codes . . . . . 17 2. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36 2. . . . . 31 2. . . . . . . . . . . . . . . 2017. . . . . . . . . . . . . . . . . . . . . . .11. . . . . . . . . . . . . . . .2. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20 2. . . . . . . . . 32 2. . . . . . . . . . .11. . . . . . . . . .2 LAN adapters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18 2. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .3. . 37 © Copyright IBM Corp. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10 2. . . . . . . . . . . . . . . . 2018 12:30 pm 5472TOC. . . . . . .12 Java. . . . . . . . . . . . . . . . . . . . .4 CAPI-enabled InfiniBand adapters . . . . . . . . . . . . . . . . 5 1. . . . 21 2. All rights reserved. . . . . 34 2. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .0 . . . . .2 Server models . . . . . . . . . . . 31 2. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .3. . . . . . . . . . . . . . . . . .6 System ports . . . 2 1. . . . . . . . . . . . . . . . . . . 29 2. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . vi Preface . . . . .2 Processor feature codes . . . . . . . .1 Disk and media features . . . . 29 2. . . .11. . . . . . . . . . . .1 Ubuntu . . . . . . .3 OpenCAPI . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39 3. . . . 45 3. . . . . . . . .5. . . . .5. . . . . . . . . .3 Rack-mounting rules . . . . . . . 49 3.1 IBM Enterprise Slim Rack 7965-S42. . . . . . . . . . . . 52 Chapter 4. . . . .4 OEM racks .1 Deep learning frameworks included and requirements . . . . . . . . . . . .2 AC power distribution units . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63 Online resources . . . 60 4. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .4 System cooling . . . . . . . . . . . . . . . . . . . . . . . . 58 4. . . 63 Help from IBM . 2018 12:30 pm Chapter 3. . . . . . . . . . . . . . . . . . . . . . . . . . . .5472TOC. .3 System power . . . . . . . .5. . . . . . . . . . . . . . . IBM PowerAI . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .3 Distributed deep learning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .1 Operating environment . . . . . . . . . . . . . . . . . . 46 3. 41 3. . . . . . . . . . 62 Related publications .5 Software download . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .5. . . . .fm Draft Document for Review January 30. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .2 PowerAI Vision . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40 3. . . . . . . . . . . . .2 Physical package . . . . . . . . . . . . . . . . . . . . . . . . . . . 60 4. . . 58 4. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63 IBM Redbooks . . . . . . . 41 3. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .5 Rack specifications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63 iv IBM Power System AC922 Introduction and Technical Overview . . . . . . . . . . . . . . . . . . . . . . . . . . .4 Large model support . . . . . . . . . Physical infrastructure. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57 4. . . . . . 40 3. . . . . . . . . . . 52 3. . . . . . . . . .

or features discussed in this document in other countries. This information could include technical inaccuracies or typographical errors. for the purposes of developing. v . IBM shall not be liable for any damages arising out of your use of the sample programs. or service is not intended to state or imply that only that IBM product. marketing or distributing application programs conforming to the application programming interface for the operating platform for which the sample programs are written. Armonk. program. program. or function of these programs. and distribute these sample programs in any form without payment to IBM. it is the user’s responsibility to evaluate and verify the operation of any non-IBM product. IBM has not tested those products and cannot confirm the accuracy of performance. serviceability. This material might be available from IBM in other languages. companies. Changes are periodically made to the information herein. or service may be used. Some jurisdictions do not allow disclaimer of express or implied warranties in certain transactions. in writing. brands. and products. the examples include the names of individuals. you may be required to own a copy of the product or product version in that language in order to access it. Any reference to an IBM product. cannot guarantee or imply reliability. services. This information contains examples of data and reports used in daily business operations. US INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS PUBLICATION “AS IS” WITHOUT WARRANTY OF ANY KIND. INCLUDING. MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. North Castle Drive. IBM may make improvements and/or changes in the product(s) and/or the program(s) described in this publication at any time without notice. without warranty of any kind. © Copyright IBM Corp. EITHER EXPRESS OR IMPLIED. IBM may use or distribute any of the information you provide in any way it believes appropriate without incurring any obligation to you. Actual performance results may vary depending on specific configurations and operating conditions. therefore. Statements regarding IBM’s future direction or intent are subject to change or withdrawal without notice. therefore.fm Notices This information was developed for products and services offered in the US. or service. which illustrate programming techniques on various operating platforms. and represent goals and objectives only. All rights reserved. program. using. However. However. IBM. 2017. their published announcements or other publicly available sources. modify. All of these names are fictitious and any similarity to actual people or business enterprises is entirely coincidental. To illustrate them as completely as possible.Draft Document for Review January 30. Consult your local IBM representative for information on the products and services currently available in your area. Any references in this information to non-IBM websites are provided for convenience only and do not in any manner serve as an endorsement of those websites. These examples have not been thoroughly tested under all conditions. COPYRIGHT LICENSE: This information contains sample application programs in source language. The sample programs are provided “AS IS”. or service that does not infringe any IBM intellectual property right may be used instead. Any functionally equivalent product. The furnishing of this document does not grant you any license to these patents. You can send license inquiries. IBM may not offer the products. The materials at those websites are not part of the materials for this IBM product and use of those websites is at your own risk. this statement may not apply to you. BUT NOT LIMITED TO. NY 10504-1785. to: IBM Director of Licensing. The performance data and client examples cited are presented for illustrative purposes only. Information concerning non-IBM products was obtained from the suppliers of those products. IBM may have patents or pending patent applications covering subject matter described in this document. You may copy. program. MD-NC119. IBM Corporation. 2018 12:30 pm 5472spec. Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products. these changes will be incorporated in new editions of the publication. compatibility or any other claims related to non-IBM products. THE IMPLIED WARRANTIES OF NON-INFRINGEMENT.

and ibm. 2018 12:30 pm Trademarks IBM. or service names may be trademarks or service marks of others. Java.shtml The following terms are trademarks or registered trademarks of International Business Machines Corporation.com are trademarks or registered trademarks of International Business Machines Corporation.ibm. and might also be trademarks or registered trademarks in other countries. vi IBM Power System S822LC for High Performance Computing Introduction and Technical Overview .com/legal/copytrade. the IBM logo. product. Other product and service names might be trademarks of IBM or other companies. other countries.fm Draft Document for Review January 30. A current list of IBM trademarks is available on the web at “Copyright and trademark information” at http://www. AIX® POWER® Redbooks® DS8000® Power Systems™ Redpaper™ Easy Tier® POWER9™ Redbooks (logo) ® EnergyScale™ PowerHA® Storwize® IBM® PowerLinux™ System Storage® IBM FlashSystem® PowerVM® XIV® OpenCAPI™ Real-time Compression™ The following terms are trademarks of other companies: Linux is a trademark of Linus Torvalds in the United States.5472spec. and all Java-based trademarks and logos are trademarks or registered trademarks of Oracle and/or its affiliates. Other company. registered in many jurisdictions worldwide. or both.

This paper introduces the major innovative AC922 server features and their relevant functions: 򐂰 Powerful IBM POWER9™ processors that offer 16 cores at 2. marketing. This paper does not replace the latest marketing materials and configuration tools.0 GHz with 2.0. can be used to enhance your knowledge of IBM server solutions. OpenCAPI™ and second generation NVIDIA NVLink technology for exceptional processor-to-accelerator intercommunication 򐂰 Up to six dedicated NVIDIA Tesla V100 GPUs This publication is for professionals who want to acquire a better understanding of IBM Power Systems products and is intended for the following audience: 򐂰 Clients 򐂰 Sales and marketing professionals 򐂰 Technical support professionals 򐂰 IBM Business Partners 򐂰 Independent software vendors This paper expands the set of IBM Power Systems™ documentation by providing a desktop reference that offers a detailed technical description of the AC922 server.fm Preface This IBM® Redpaper™ publication is a comprehensive guide that covers the IBM Power System AC922 server (8335-GTG and 8335-GTW models). Authors This paper was produced by a team of specialists from around the world working at the International Technical Support Organization. PMP © Copyright IBM Corp.87 GHz turbo 򐂰 IBM CAPI 2. and high-performance computing. high-performance analytics.09 GHz turbo performance or 20 cores at 2.Draft Document for Review January 30. He holds a degree in Computer Science from the Universidade Estadual Paulista (UNESP) and an MBA in Marketing. Alexandre has more than 20 years of experience working on IBM Systems Solutions and has worked also as an IBM Business Partner on Power Systems hardware. 2017. The AC922 server is the next generation of the IBM Power processor based systems. Alexandre Bicas Caldeira is a Certified IT Specialist and a former Product Manager for Power Systems Latin America. The project that produced this publication was managed by: Scott Vetter Executive Project Manager. vii . 2018 12:30 pm 5472pref. together with existing sources. IBM AIX®. sales. and technical sales support. It is intended as an additional source of information that. and IBM PowerVM® virtualization products. His major areas of focus are competition. Austin Center. All rights reserved.6 GHz with 3. designed for deep learning and artificial intelligence.

as you expand your network of technical contacts and relationships. Cesar Diniz Maciel.com/groups?home=&gid=2130806 򐂰 Explore new Redbooks publications.com/ibmredbooks 򐂰 Look for us on LinkedIn: http://www. Volker Haug. Your efforts will help to increase product acceptance and customer satisfaction. Jeff Stuecheli IBM Now you can become a published author. too! Here’s an opportunity to spotlight your skills.com/redbooks 򐂰 Send your comments in an email to: redbooks@us. browse the residency index. residencies.com 򐂰 Mail your comments to: IBM Corporation. and workshops with the IBM Redbooks weekly newsletter: viii IBM Power System AC922 Introduction and Technical Overview . grow your career.html Comments welcome Your comments are important to us! We want our papers to be as helpful as possible.com/redbooks/residencies.com/IBMRedbooks 򐂰 Follow us on Twitter: http://twitter. Send us your comments about this paper or other IBM Redbooks publications in one of the following ways: 򐂰 Use the online Contact us review Redbooks form found at: ibm.fm Draft Document for Review January 30. Find out more about the residency program. Residencies run from two to six weeks in length. Ann Lund. International Technical Support Organization Dept. Chris Mann.5472pref. 2018 12:30 pm Thanks to the following people for their contributions to this project: Adel El-Hallak.linkedin. and become a published author—all at the same time! Join an ITSO residency project and help write a book in your area of expertise. NY 12601-5400 Stay connected to IBM Redbooks 򐂰 Find us on Facebook: http://www.ibm.facebook. and you can participate either in person or as a remote resident working from your home base. Scott Soutter. while honing your experience using leading-edge technologies. HYTD Mail Station P099 2455 South Road Poughkeepsie. and apply online at: ibm.

redbooks.ibm.redbooks.com/Redbooks.nsf/subscribe?OpenForm 򐂰 Stay current on recent Redbooks publications with RSS Feeds: http://www.html Preface ix .Draft Document for Review January 30.com/rss.fm https://www.ibm. 2018 12:30 pm 5472pref.

2018 12:30 pm x IBM Power System AC922 Introduction and Technical Overview .fm Draft Document for Review January 30.5472pref.

All rights reserved. Product summary The IBM Power System AC922 is the next generation of the IBM POWER® processor based systems. 1 . Among the new technologies the system provides. designed for deep learning and artificial intelligence. Therefore. the system is available in two distinct models: 򐂰 8335-GTG up to four GPUs and air cooled 򐂰 8335-GTW up to six GPUS and water cooled © Copyright IBM Corp. delivering up to 100 TFlops each. The system is a co-designed with OpenPOWER Foundation ecosystem members and it will be deployed at the most powerful supercomputer on the planet with a partnership between IBM. It provides the latest technologies available for highest performance computing. lower latency. among others.Draft Document for Review January 30. a five fold improvement versus the previous generation 򐂰 Second generation NVLINK with 2x throughput compared with the first generation Due to the massive computing capacity packed into just 2Us of rack space. high-performance analytics.fm 1 Chapter 1. improving even more the movement of data from memory to GPU accelerator cards and back. special cooling was designed to support the largest configurations. and high-performance computing. 2017. data processing. allowing for faster. in order to accommodate distinct datacenter infrastructure requirements. NVIDIA. Mellanox. the most significative are: 򐂰 Two POWER9 processors with up to 40 cores and improved buses 򐂰 1 TB of DDR4 memory with improved speed 򐂰 Up to six NVIDIA Tesla V100 (Volta) GPUs. 2018 12:30 pm 5472ch01.

2018 12:30 pm Figure 1-1 shows the front and rear view of an Power System AC922 server:  .fm Draft Document for Review January 30.5472ch01.

         .

1 Key server features The Power System AC922 was designed to address the demanding needs of deep learning and AI.09 GHz turbo) • Clock speeds for 20-core chip of 2. The system includes several features to improve performance: 򐂰 POWER9 Processors – Each POWER9 processor module has either 16 or 20 cores based 64-bit architecture • Clock speeds for 16-core chip of 2. – 7. based on the NVIDIA SXM2 form factor connectors. and up to 120 MB of L3 cache per chip – Up to 4 threads per core – 120 GB/s memory bandwidth per chip – 64 GB/s SMP interconnect between POWER9 chips 򐂰 DDR4 Memory – Sixteen dual in-line memory module (DIMM) memory slots – Maximum of 1024 GB DDR4 system memory – Improved clock from 1333 MHz to 2666 MHz for reduced latency 򐂰 NVIDIA Tesla V100 GPUs – Up to six NVIDIA Tesla V100 GPUs.   Figure 1-1 Front and rear view of the Power System AC922 1. An updated list of ported HPC applications that can exploit the IBM POWER technology can be seen in http://ibm. designed for deep learning 2 IBM Power System AC922 Introduction and Technical Overview .6 GHz (3. and high-performance computing.7 TFLOPs per GPU for single precision – 125 TFLOPs per GPU for deep learning • New 640 Tensor Cores per GPU.8 TFLOPs per GPU for double precision – 15. high-performance analytics.0 GHz (2.biz/hpcapplications.87 GHz turbo) – 512 KB of L2 cache per core.

0 capable slots Figure 1-2 shows the physical locations of the main server components. twice the throughput from PCIe Gen3 • Three CAPI 2.fm – 16 GB HBM2 internal memory with 900 GB/s bandwidth.Draft Document for Review January 30. storage controllers. 2018 12:30 pm 5472ch01.   0 .0 – Twice the throughput.5x the bandwidth compared to Pascal P100 – Liquid cooling for six GPUs configurations to improve compute density 򐂰 NVLink 2. compared to 32 GB/s of traditional PCIe Gen3 򐂰 OpenCAPI 3. compared to the previous generation of NVLink – Up to 200 GB/s of bi-directional bandwidth between GPUs – Up to 300 GB/s of bi-directional bandwidth per POWER9 chip and GPUs. and advanced memory technologies – Up to 100 GB/s of bi-directional bandwidth between CPUs and OpenCAPI devices 򐂰 PCIe Gen4 Slots – Four PCIe Gen4 slots up to 64 GB/s bandwidth per slot.0 – Open protocol bus to allow for connections between the processor system bus in a high speed and cache coherent manner with OpenCAPI compatible devices like accelerators. network controllers. 1.

% & '*-$( 0 6'%"' 0 =9  .

.

&$%&#' 0  0 68.5 .

 . ! '.$( 0 7)& #'&)6. #(%# + "  0 6)& #')= #(%# + "  0 6)& #')9 #(%#  % !.

 .

#" '+$( 0 7755 0 77< 0 #'($ .

! 0 $'#8$%>$ 0 .

7#%!'#%   0 855 0 #'&($ 0  " 7.5 .

 ..

 '+$( 0 6.#%75#% 0 .

 0 *&'! -& Figure 1-2 Location of server main components 1.9 !! !% 0 68.51=88:/#" *2 0 7)&.2 Server models The IBM Power AC922 is manufactured in two distinct models: Chapter 1. Product summary 3 .

2. 2018 12:30 pm Table 1-1 Summary of AC922 server available models Server POWER9 Maximum Maximum Cooling Model Chips Memory GPU Cards 8335-GTG 2 1 TB 4 Air cooled 8335-GTW 2 1 TB 6 Water cooled 1.6 GHz processor module – 20-core 2.1 Power AC922 server model 8335-GTG This summary describes the standard features of the Power AC922 model 8355-GTG: 򐂰 19-inch rack-mount (2U) chassis 򐂰 Two POWER9 processor modules: – 16-core 2. Coherent Accelerator Processor Interface (CAPI) enabled – One PCIe x8 Gen4 Low Profile slot.5472ch01.0 port for general use – One system port with RJ45 connector 򐂰 Two power supplies (both are required) 4 IBM Power System AC922 Introduction and Technical Overview . based on the NVIDIA SXM2 form factor connectors air-cooled 򐂰 Integrated features: – IBM EnergyScale™ technology – Hot-swap and redundant cooling – Two 1 Gb RJ45 ports – One front USB 3.0 GHz processor module – Up to 1024 GB of 2666 MHz DDR4 error correction code (ECC) memory 򐂰 Two small form factor (SFF) bays for hard disk drives (HDDs) or solid state drives (SSDs) that support: – Two 1 TB 7200 RPM NL SATA disk drives (#ELD0) – Two 2 TB 7200 RPM NL SATA disk drives (#ES6A) – Two 960 GB SATA SSDs (#ELS6) – Two 1.fm Draft Document for Review January 30.0 port for general use – One rear USB 3.84 TB SATA SSDs (#ELU0) 򐂰 Integrated SATA controller 򐂰 Four Peripheral Component Interconnect Express (PCIe) Gen4 slots: – Two PCIe x16 Gen4 Low Profile slot. CAPI enabled – One PCIe x4 Gen4 Low Profile slot 򐂰 Two or four NVIDIA Tesla V100 GPU (#EC4J).92 TB SATA SSDs (#ELSZ) – Two 3.

0 GHz processor module – Up to 1024 GB of 2666 MHz DDR4 error correction code (ECC) memory 򐂰 Two small form factor (SFF) bays for hard disk drives (HDDs) or solid state drives (SSDs) that support: – Two 1 TB 7200 RPM NL SATA disk drives (#ELD0) – Two 2 TB 7200 RPM NL SATA disk drives (#ES6A) – Two 960 GB SATA SSDs (#ELS6) – Two 1. 2018 12:30 pm 5472ch01.2. Figure 1-3 AC922 server model 8335-GTG fully populated with four GPUs 1.84 TB SATA SSDs (#ELU0) 򐂰 Integrated SATA controller 򐂰 Four Peripheral Component Interconnect Express (PCIe) Gen4 slots: – Two PCIe x16 Gen4 Low Profile slot.6 GHz processor module – 20-core 2.2 Power AC922 server model 8335-GTW This summary describes the standard features of the Power AC922 model 8335-GTW: 򐂰 19-inch rack-mount (2U) chassis 򐂰 Two POWER9 processor modules: – 16-core 2. Coherent Accelerator Processor Interface (CAPI) enabled – One PCIe x8 Gen4 Low Profile slot.fm The internal view of the fully populated server with four GPUs can be seen in Figure 1-3. CAPI enabled – One PCIe x4 Gen4 Low Profile slot Chapter 1.Draft Document for Review January 30. Product summary 5 .92 TB SATA SSDs (#ELSZ) – Two 3.

3 Minimum features The minimum initial order for the Power AC922 model 8355-GTG must include the following minimum features: 򐂰 Two processor modules with at least 16 cores each 򐂰 256 GB of memory (sixteen 16 GB memory DIMMs) 򐂰 Two HDDs or SSDs 򐂰 Two #EC3L PCIe 2-port 100 Gbps Ethernet adapters 򐂰 Two #EC4J compute-intensive accelerators (NVIDIA V100) 򐂰 Two power supplies and power cords (both are required) 򐂰 An Linux OS indicator 6 IBM Power System AC922 Introduction and Technical Overview . 2018 12:30 pm 򐂰 Four or six NVIDIA Tesla V100 GPU (#EC4J).0 port for general use – One system port with RJ45 connector 򐂰 Two power supplies (both are required) The internal view of the fully populated server with six GPUs and water cooling system installed can be seen in Figure 1-4. based on the NVIDIA SXM2 form factor connectors water-cooled 򐂰 Integrated features: – IBM EnergyScale technology – Hot-swap and redundant cooling – Two 1 Gb RJ45 ports – One rear USB 3.fm Draft Document for Review January 30.5472ch01.2. Figure 1-4 AC922 server model 8335-GTW fully populated with six GPUs Note: A hardware management console is not supported on the AC922 server. 1.

Draft Document for Review January 30. 2018 12:30 pm 5472ch01.fm 򐂰 A rack integration indicator 򐂰 A Language Group Specify Chapter 1. Product summary 7 .

2018 12:30 pm 8 IBM Power System AC922 Introduction and Technical Overview .fm Draft Document for Review January 30.5472ch01.

System architecture This chapter describes the overall system architecture for the AC922 server. Note: The speeds that are shown are at an individual component level. The bandwidths that are provided throughout the section are theoretical maximums that are used for reference. 2018 12:30 pm 5472ch02. Always do the performance sizing at the application-workload environment level and evaluate performance by using real-world performance measurements and production workloads.Draft Document for Review January 30. © Copyright IBM Corp. 9 . Multiple components and application implementation are key to achieving the preferred performance. 2017.fm 2 Chapter 2. All rights reserved.

Additionally. Each SCM is attached eight memory RDIMM slots. GPUs and PCIe slots can be seen in Figure 2-1. three of these slots are CAPI-capable. The server has a maximum capacity of 16 memory DIMMs which allows for up to 1024 GB of memory. the server has a total of four PCIe Gen3 slots.1 System architecture The AC922 server is a two single-chip module (SCM) system. 2018 12:30 pm 2. memory DIMMs. each 300 Watt capable.     . The system board has sockets for four or six GPUs depending on the model.fm Draft Document for Review January 30.5472ch02. A diagram with the location of the processors.

This bus also drives the integrated Ethernet and USB ports. 10 IBM Power System AC922 Introduction and Technical Overview .    "#   !  Figure 2-1 Component location for four GPU and six GPU system planars An integrated SATA controller is fed through a dedicated PCI bus on the main system board and allows for up to two SATA HDDs or SSDs to be installed. The location of the integrated SATA connector can be seen in Figure 2-2 on page 11.

fm    . 2018 12:30 pm 5472ch02.Draft Document for Review January 30.

 .

       .

 .

.

Chapter 2.  Figure 2-2 Integrated SATA connector The POWER9 processor brings enhanced memory and I/O connectivity. System architecture 11 . A diagram with the external processor connectivity can be seen in Figure 2-3 on page 12.0. improved chip to chip communication and a new bus called NVLINK 2.

$     $%&    $%&   "'()*  . 2018 12:30 pm * --()...   --().fm Draft Document for Review January 30.%!2))3%")* *&)("$ 00)2 *)+'(.)/ 0/1%/.5472ch02.)/ 0/1%/.$ '+& .

 .)("'00)".' "$%&  *&)("$ 00)2    . *0.

.

.

developed by IBM. Symmetric Multiprocessing chip-to-chip interconnect is done via a four channel SMP bus with 64 GB/s bidirectional bandwidth. The Figure 2-4 on page 13 compares the POWER9 implementation of NVLINK 2. This link provides up to 5x more communication bandwidth between CPUs and GPUs (when compared to traditional PCIe Gen3) and allows for faster data transfer between memory and GPUs and between GPUs..$ *&)() )0  0) ) 2'.4. The connection between GPUs and between CPUs and GPUs is done via a link called NVLINK 2. The latest PCIe Gen4 interconnect doubles the channel bandwidth from previous PCIe Gen3 generation.0. allowing for the 48 PCIe channels to drive total of 192 GB/s bidirectional bandwidth between I/O adapters and the POWER9 chip. For more information about NVLINK 2.* Figure 2-3 POWER9 chip external connectivity Faster DDR4 memory DIMMs at 2666 MHz are connected to two memory controllers via eight channels with a total bandwidth of 120 GB/s.0 please see 2. “NVLINK 2.0 with traditional processor chips using PCIe and NVLINK.  *) )0  0)*  0)*  0)* --().)/ 0/1%/. 12 IBM Power System AC922 Introduction and Technical Overview .5. NVIDIA and the OpenPOWER Foundation. Each POWER9 CPU and each GPU have six NVLINK channels. These channels can be aggregated to allow for more bandwidth or more peer to peer connections.0” on page 28. Complex and data hungry algorithms like the ones used in machine learning can benefit from having these enlarged pipelines for data transfer once the amount of data needed to be processed is many times larger than the GPU internal memory. called Bricks. each one delivering up to 50 GB/s bi-directional bandwidth.

%  4*.Draft Document for Review January 30.4 (!/&.fm (!/&.)+ 4*.%) )0 2&.&'0!2'00)".&1&.4  '00)".)+ 4*.%  2&.&1&.&'0!2'00)". 2018 12:30 pm 5472ch02.&1&.)+ )+'(4 )+'(4 )+'(4 .4 2&.

* .

*  * )+'(40* )+'(40* )+'(40* ('")**'( ('")**'(   %&& %&& %&&  * * * . having more than two GPUs per processor would potentially create a bottleneck on the data flow from system memory to GPUs.! *  * . Once the processor has to handle all the GPU to GPU communication and GPU to system memory communication. The goal is to move data from system memory to GPU’s 16 GB internal memory as fast as possible so that GPU processing doesn’t have to stop and wait for data to be moved to be able to continue.0 implementation on POWER9 goes beyond traditional implementation by implementing 1.' .! '2. reducing the dependency on the PCIe bus to exchange data between GPUs but still depending on PCIe Gen3 to GPU to system memory communications.0 POWER9 implementation versus traditional architectures On traditional processors.! '2. Chapter 2. where the six NVLINK Bricks are divided in groups of three allowing for 150 GB/s buses between GPUs. communication is done via PCIe Gen3 buses. System architecture 13 . To reduce this impact on the GPU to GPU communication. there are two distinct logical diagrams depending on the amount of maximum GPUs supported per system. Once NVLINK Bricks are combined differently depending on the server having four or six GPUs (with two POWER9 processors) to maximize bandwidth.! '2. Figure 2-5 on page 14 shows the logical system diagram for the AC922 server (8335-GTG) with four GPUs.! '2. NVLINK brings a 50 GB/s direct link between GPUs. reducing potential bottlenecks throughout the system.! '2.' ) )0  ) )0   (&"'* '2.5x more memory bandwidth and aggregating NVLINK Bricks to allow for up to 3x faster communication between GPUs and system memory to GPU. The NVLINK 2.' (&"'  (&"'* Figure 2-4 NVLINK 2.

2018 12:30 pm NVLink 2. 14 IBM Power System AC922 Introduction and Technical Overview . allowing for 100 GB/s buses between GPUs but allowing for more connected GPUs.CAPI PCIe Gen4 x8 CAPI PCIe Gen4 x8 PCIe Gen4 x4 PCIe Gen2 x4 PCIe Gen2 x4 PCIe Gen2 x4 PEX PCIe Gen2 x2 PCIe Gen2 x1 2 x 1Gbps Ethernet Internal Storage Broadcom BMC USB Controller 2x Internal Rear Front IPMI VGA RJ-45 USB USB USB Figure 2-5 The AC922 server model 8335-GTG logical system diagram Figure 2-5 shows the logical system diagram for the AC922 server (8335-GTW) with six connected GPUs.5472ch02. where the six NVLINK Bricks are divided in three groups of two Bricks.0 150 GB/s aggregated bandwidth (3 Bricks) 15 GB/s per channel DDR4 DIMM DDR4 DIMM DDR4 DIMM DDR4 DIMM DDR4 DIMM DDR4 DIMM DDR4 DIMM DDR4 DIMM POWER9 X Bus POWER9 DDR4 DIMM CPU 0 64 GB/s CPU 1 DDR4 DIMM DDR4 DIMM DDR4 DIMM DDR4 DIMM DDR4 DIMM DDR4 DIMM DDR4 DIMM PCIe Gen4 x8 PCIe Gen4 x16 .fm Draft Document for Review January 30.CAPI PCIe Gen4 x16 .0 NVIDIA NVIDIA NVIDIA NVIDIA VOLTA VOLTA VOLTA VOLTA GPU GPU GPU GPU 50 GB/s per channel (Brick) NVLink 2.

1 POWER9 processor overview The POWER9 chip has four variations. The POWER9 processor in the AC922 server is the latest generation of the POWER processor family. depending on the server scalability and whether their are being used on Linux ecosystem designed servers or PowerVM ecosystem servers. 2018 12:30 pm 5472ch02.CAPI PCIe Gen4 x8 CAPI PCIe Gen4 x8 PCIe Gen4 x4 PCIe Gen2 x4 PCIe Gen2 x4 PCIe Gen2 x4 PEX PCIe Gen2 x2 PCIe Gen2 x1 2 x 1Gbps Ethernet Internal Storage Broadcom BMC USB Controller 2x Internal Rear Front IPMI VGA RJ-45 USB USB USB Figure 2-6 The AC922 server model 8335-GTW logical system diagram 2.fm 100 GB/s aggregated bandwidth (2 Bricks) NVLink 2.CAPI PCIe Gen4 x16 . maximum core count.0 (Brick) 15 GB/s per channel DDR4 DIMM DDR4 DIMM DDR4 DIMM DDR4 DIMM DDR4 DIMM DDR4 DIMM DDR4 DIMM DDR4 DIMM POWER9 X Bus POWER9 DDR4 DIMM CPU 0 64 GB/s CPU 1 DDR4 DIMM DDR4 DIMM DDR4 DIMM DDR4 DIMM DDR4 DIMM DDR4 DIMM DDR4 DIMM PCIe Gen4 x8 PCIe Gen4 x16 . and memory connection. System architecture 15 .0 NVIDIA NVIDIA NVIDIA NVIDIA NVIDIA NVIDIA VOLTA VOLTA VOLTA VOLTA VOLTA VOLTA GPU GPU GPU GPU GPU GPU 50 GB/s per channel NVLink 2. Chapter 2. Table 2-1 on page 16 compares the chip variations.2.2 Processor subsystem This section introduces the latest processor in the Power Systems product family and describes its main characteristics and features in general. The main differences reside in the scalability. Based on the 14 nm FinFET Silicon-On-Insulator (SOI) architecture. SMT capability.Draft Document for Review January 30. the chip size is 685 mm x 685 mm and contains eight billion transistors. 2.

"+&.&+&Y"! '1"(  '*4*.fm Draft Document for Review January 30."+&. &002 '*4*. 2018 12:30 pm Table 2-1 POWER9 chip variations Variation Maximum SMP Maximum Memory Memory Cores Connections SMT Connection Bandwidth Scale-Out Linux 24 2 sockets SMT4 Direct 120 GB/s Scale-Out PowerVM 24 2 sockets SMT4 Direct 120 GB/s Scale-Up Linux 12 16 sockets SMT8 Memory 230 GB/s Buffer Scale-Up PowerVM 12 16 sockets SMT8 Memory 230 GB/s Buffer The main reason for this differentiation between Linux Ecosystem and PowerVM ecosystem is that Linux Ecosystems have a greater need of granularity and core counts per chip while PowerVM Ecosystem has a greater need of stronger threads and higher per core performance for better efficiency on licensing.&+&Y"!   '("*%&&  '("*%&&  .5472ch02. A diagram reflecting the main differences can also be seen in Figure 2-7.

             2"0."( '00" .  %"0. PCIe Gen4 I/O controllers.  %"0. and an interconnection system that connects all components within the chip at 7 TB/s. The interconnect also extends through module and system board technology to other POWER9 processors in addition to DDR4 memory and various I/O devices."( '00" ."( '00" . Q *' '"."( '00" . The POWER9 chip contains two memory controllers."+'(4 .. Q 0##"("!"+'(4 .% "     "     0##"( 0##"(  &0'* &0'*  &0'* &0'* Figure 2-7 POWER9 chip variations The AC922 server utilizes the Scale-out Linux version. with up to 24 cores and SMT4. Q &(" . Q  *' '". Each core has 256 KB of L2 cache.(4. and all cores share 120 MB of L3 embedded DRAM (eDRAM). 16 IBM Power System AC922 Introduction and Technical Overview .                          %"0. % &"0  &"0  "+'(4  "+'(4          Q "*&$0"! Q *"+'(40!1&!.0!(! Q *"+'(40!1&!. % &"0  &"0  "+'(4  "+'(4          Q 0!0*.% "     "        &0'* &0'*  &0'* &0'*              2"&                          %"0..

2018 12:30 pm 5472ch02. 2. Theoretically. Figure 2-8 The 24-core POWER9 processor Each POWER9 processor has eight memory channels. Each buffer chip also includes an L4 cache to reduce the latency of local memory accesses.Draft Document for Review January 30. System architecture 17 .2. Due to the current state of memory DIMM technology. Processor features must be in quantity two and cannot be mixed. the largest available DIMM is 64 GB and therefore the largest amount of memory supported on the AC922 is 1024 GB (64 GB x 8 channels x 2 processors).2 Processor feature codes The AC922 server supports two processor configurations only. Figure 2-8 shows the POWER9 processor with 24 cores. Chapter 2. designed to address up to 512 GB of memory. scale-up POWER9 processor-based systems use memory buffer chips to interface between the POWER9 processor and DDR4 memory.fm While scale-out variations have direct memory connections. a two-socket server could address up to 8 TB of memory and a 16-socket system could address up to 64 TB of memory.

1 Memory features and placement rules Each feature code equates to a single memory DIMMs.87 GHz Turbo) . Table 2-4 on page 19 shows total memory and how it can be accomplished by the quantities of each memory feature code. Memory features equate to a single memory DIMM. 18 IBM Power System AC922 Introduction and Technical Overview . 2666 MHz DDR4 memory modules (#EM61).190W 2/2 Linux 2. The server supports a maximum of 16 DDR4 RDIMMs slots in the main system planar directly connected to the POWER9 processor. 򐂰 The base memory is 256 GB with sixteen 16 GB. 2018 12:30 pm Table 2-2 POWER9 processor features supported Feature Description Min/Max OS code support EP0K POWER9 16-core 2. 2.5472ch02. 򐂰 Memory features cannot be mixed.3 Memory subsystem The AC922 server is a two-socket system that supports two POWER9 SCM processor modules.00 GHz (2.3. Memory feature codes that are supported are as follows: 򐂰 16 GB DDR4 򐂰 32 GB DDR4 򐂰 64 GB DDR4 Plans for future memory growth needs should be taken into account when deciding which memory feature size to use at the time of initial system order once an upgrade will require a full replacement of the installed DIMMs. All memory DIMMs must be populated and mixing of different memory feature codes is not supported. For the AC922 server (models 8335-GTG and 8335-GTW).09 GHz Turbo) . On the Table 2-3 it is shown the available memory feature codes for ordering: Table 2-3 Memory features supported Feature Description Min/Max OS code support EM61 16 GB DDR4 2666 MHz DDR4 RDIMM 16/16 Linux EM63 32 GB DDR4 2666 MHz DDR4 RDIMM 16/16 Linux EM64 64 GB DDR4 2666 MHz DDR4 RDIMM 16/16 Linux The supported maximum memory is 1024 GB by installing a quantity of 16 #EM64 memory DIMMs.190W 2/2 Linux EP0M POWER9 20-core 2.60 GHz (3.fm Draft Document for Review January 30. the following requirements apply: 򐂰 All the memory DIMMs must be populated.

2018 12:30 pm 5472ch02.860 GHz 16 cores @ 3.fm Table 2-4 Supported memory feature codes for AC922 server Total installed memory Memory features 256 GB 512 GB 1024 GB 16 GB (#EM61) 16 32 GB (#EM63) 16 64 GB (#EM64) 16 2. memory.57 GBps The bandwidth figures for the caches are calculated as follows: 򐂰 L1 cache: In one clock cycle.3.860 GHz Core: (2 x 16 B + 1 x 16 B) x 2. Table 2-5 The AC922 server single-core bandwidth estimates Single core 8335-GTG and 8335-GTW 2.259 GHz = 208. Table 2-5 shows the maximum bandwidth estimates for a single core on the server.57 GBps Table 2-6 shows the overall bandwidths for the entire AC922 server populated with the two processor modules.860 GHz 3.Draft Document for Review January 30.860 GHz = 183.43 GBps 򐂰 L2 cache: In one clock cycle. System architecture 19 .259 GHz = 156.259 GHz Core: (1 x 32 B + 1 x 32 B) x 3. The value varies depending on the clock of the core.259 GHz Core: (1 x 32 B + 1 x 16 B) x 3.28 GBps – 3.28 GBps 156.259 GHz L1 (data) cache 137.860 GHz = 137.259 GHz L1 (data) cache 2746 GBps 2503 GBps L2 cache 2746 GBps 2503 GBps Chapter 2. and the formula is as follows: – 2. The value varies depending on the clock of the core. and interconnect bandwidths. Table 2-6 The AC922 server total bandwidth estimates Total bandwidths 8335-GTG 20 cores @ 2.04 GBps – 3.28 GBps 156.43 GBps L2 cache 137.2 Memory bandwidth The POWER9 processor has exceptional cache. one 32-byte load operation and one 16-byte store operation can be accomplished. and the formula is as follows: – 2.43 GBps 򐂰 L3 cache: One 32-byte load operation and one 32-byte store operation can be accomplished at half-clock speed.860 GHz Core: (1 x 32 B + 1 x 16 B) x 2.43 GBps L3 cache 183.04 GBps 208.28 GBps – 3. and the formulas are as follows: – 2.860 GHz Core: (1 x 32 B + 1 x 32 B) x 2.259 GHz = 156. two 16-byte load operations and one 16-byte store operation can be accomplished.259 GHz Core: (2 x 16 B + 1 x 16 B) x 3.860 GHz = 137.

where the number is effectively the number of lanes. In these configurations.259 GHz L3 cache 3661 GBps 3337 GBps Total memory 240 GBps 240 GBps SMP interconnect 64 GBps 64 GBps PCIe interconnect 272 GBps 272 GBps Where: 򐂰 Total memory bandwidth: Each POWER9 processor has eight memory channels running at 15 GBps. but larger speed adapters are not compatible in smaller connector sizes (that is. x2.4. and x16 speed adapters in the same size connector slots first before mixing adapter speed with connector slot size. The bandwidth formula is calculated as follows: 1 X bus * 4 bytes * 16 GHz = 64 GBps 򐂰 PCIe interconnect: Each POWER9 processor has 34 PCIe lanes running at 16 Gbps full-duplex.4 I/O subsystem The key components of the I/O subsystem are discussed in the following. 20 IBM Power System AC922 Introduction and Technical Overview . PCIe Gen4 slots also support previous generation (Gen3 and Gen2) adapters. A single PCIe serial link is a dual-simplex connection that uses two pairs of wires.5472ch02. which operate at lower speeds according to the following rules: 򐂰 Place x1. or x32. x8. x4. x12. 2018 12:30 pm Total bandwidths 8335-GTG 20 cores @ 2. 2.1 PCIe PCIe uses a serial interface and allows for point-to-point interconnections between devices by using a directly wired interface between these connection points. If you attempt to force an adapter into the wrong type of slot. A PCIe link can consist of multiple lanes. PCIe adapters use a different type of slot than PCI adapters. The bandwidth formula is calculated as follows: 34 lanes x 2 processors x 16 Gbps x 2 = 272 GBps 2. and can transmit only one bit per cycle.860 GHz 16 cores @ 3. x16. you might damage the adapter or the slot. the connection is labeled as x1. The bandwidth formula is calculated as follows: 8 channels x 15 GBps = 120 GBps per processor module 򐂰 SMP interconnect: The POWER9 processors are connected using an X-bus. 򐂰 Adapters with lower speeds are allowed in larger sized PCIe connectors. x8.fm Draft Document for Review January 30. These two pairs of wires are called a lane. a x16 adapter cannot go in an x8 PCIe slot connector). The AC922 supports the new PCIe Gen4. which are capable of 32 GBps simplex (64 GBps duplex) on a single x16 interface. one pair for transmit and one pair for receive.

ibm. Figure 2-9 shows a comparison of the traditional model. use the IBM prerequisite website: https://www-912. where the accelerator has to go thru the processor to access memory. see the following website: http://www. To obtain this information. ensure that you have the software that is required to support the new feature and determine whether there are existing update prerequisites to install.2 IBM CAPI2 IBM CAPI2 is the evolution of CAPI and defines a coherent accelerator interface structure for attaching special processing devices to the POWER9 processor bus.ibm. For more information about the System Planning Tool. CAPI allows a specialized hardware accelerator to be seen as an additional processor in the system with access to the main system memory and coherent communication with other processors in the system.4. Applications can also have customized functions by using the same effective addresses (pointers) they use for any threads running on a host processor. use the System Planning Tool to validate the new adapter configuration.Draft Document for Review January 30. Applications can have customized functions in Field Programmable Gate Arrays (FPGAs) and enqueue work requests directly in shared memory queues to the FPGA.nsf The following sections describe other I/O technologies that enhance or replace the PCIe interface. Before adding or rearranging adapters. 2. CAPI2 can attach accelerators that have coherent shared memory access with the processors in the server and share full virtual address translation with these processors. with CAPI. 2018 12:30 pm 5472ch02. From a practical perspective.com/systems/support/tools/systemplanningtool/ If you are installing a new feature. As the original CAPI.  . due to the restricted height of the server.fm POWER9-based servers support PCIe low profile (LP) cards.com/e_dir/eServerPreReq. now using a standard PCIe Gen4 buses with twice the bandwidth compared to the previous generation.

.

  .

   .

.

                 .

  .

 Figure 2-9 CAPI accelerator attached to the POWER9 processor The benefits of using CAPI include the ability to access shared memory blocks directly from the accelerator. the ability to perform memory transfers directly between the accelerator and processor cache. This reduction in the code path length might occur because the adapter is not Chapter 2. and the ability to reduce the code path length between the adapter and the processors. System architecture 21 .

5472ch02. and cache areas as though they were a processor in the system. By removing these layers. storage and other devices to be connected to the processor through a high bandwidth. consisting of an FPGA or an ASIC. The custom processors on the system board. becoming the interface of choice for connecting high performance devices. low latency interface. use expanded memory beyond server internal capacity. CAPI also presents a simpler programming model.fm Draft Document for Review January 30. network. It also eliminates the need for direct memory access that requires system calls to the OS kernel. With this in mind. it is treated as a processor. the OpenCAPI Consortium was created. Some of these use cases and examples can be seen in Figure 2-10 on page 23. Instead of treating the hardware accelerator as an I/O device. which eliminates the requirement of a device driver to perform communication. or PSL). the data transfer operation requires fewer clock cycles in the processor. By providing a high bandwidth low latency connection to devices. 2018 12:30 pm operating as a traditional I/O device.4. see 2. 22 IBM Power System AC922 Introduction and Technical Overview . “CAPI-enabled InfiniBand adapters” on page 32. use this layer to access shared memory regions. OpenCAPI allows several applications to improve networking. OpenCAPI aims to allow memory. This ability enhances the performance of the data access for the device and simplifies the programming effort to use the device. opening the CAPI interface to other hardware developers and extending its capabilities. 2. and there is no device driver layer to perform processing. with low latency. improving the I/O performance.5. The PCIe standard is pervasive to every processor technology. They do so by using a hardware accelerator with a simplified programming model and efficient communication with the processor and memory resources. do not allow the attachment of memory tor load/store operations. Companies use the performance of the POWER9 processor for general applications and the custom acceleration of specific functions.3 OpenCAPI While CAPI is a technology present in IBM POWER processors and depending on IBM’s intellectual property (the Processor Service Layer. For a list of supported CAPI adapters. accelerators. make use of FPGA accelerators. which provides address translation and system memory cache for the accelerator functions. and reduce latency to storage devices. with the goal of defining a device attachment interface. The implementation of CAPI on the POWER9 processor allows hardware companies to develop solutions for specific application demands. The accelerator adapter implements the Power Service Layer (PSL). including memory attachment. several industry solutions would benefit from having a mechanism of connecting different devices to the processor. but its design characteristics and latency.4.

2018 12:30 pm 5472ch02. which encompasses a 25 GB/s signaling and protocol built to enable very low latency interface on CPU and attached devices. That alone show a significant enhancement in OpenCAPI when compared to traditional PCIe interconnects. the electrical interface is based on the design from the 25G workgroup within the OpenPower Foundation. Open CAPI is agnostic to processor architecture and as such the electrical interface is not being defined by the OpenCAPI consortium or any of its workgroups. The current design for the adapter is based on a PCIe card that draws power from the PCIe slot. In contrast.10 ns). PCIe latency is 10 times bigger (around 100 ns). On the POWER9 processor.Draft Document for Review January 30. while connecting to the OpenCAPI port on the planar through a 25 GBs/s cable. System architecture 23 . as seen in the Figure 2-11 on page 24. Future capabilities include increase speeds to 32 GB/s and 56 GB/s signaling.fm Figure 2-10 OpenCAPI uses cases The design of OpenCAPI allows for very low latency in accessing attached devices (in the same range of DDR memory access . Chapter 2. which enables memory to be connected through OpenCAPI and serve as main memory for load/store operations.

5472ch02.fm Draft Document for Review January 30. 2018 12:30 pm   !! .

 ! .

 " .

 ! !    ! !    ! %     ! !    ! !    ! % .  ! "!   !   ! # ! Figure 2-11 OpenCAPI compatible adapter and 25G link The OpenCAPI interface uses the same electrical interconnect as NVLink 2. The use of OpenCAPI adapters limits the mount of NVLINK ports available for GPU communication. Each POWER9 chip has six NVLINK ports. out of which four can be alternatively be used for OpenCAPI adapters as shows in Figure 2-12. or both. an OpenCAPI-attached device.0 – systems can be designed to have an NVLink-attached GPU.

"
!#
$#

Figure 2-12 OpenCAPI and NVLINK shared ports on the POWER9 chip

24 IBM Power System AC922 Introduction and Technical Overview

Draft Document for Review January 30, 2018 12:30 pm 5472ch02.fm

2.4.4 The NVIDIA Tesla V100
NVIDIA’s new NVIDIA Tesla V100 accelerator, code named Volta, takes GPU computing to
the next level. This section describes the Tesla V100 accelerator, as shown in Figure 2-13.

Figure 2-13 NVIDIA Tesla V100 for NVLINK accelerator

NVIDIA Tesla V100 is the world’s most advanced data center GPU ever built to accelerate AI,
HPC, and graphics. Powered by NVIDIA Volta, the latest GPU architecture, Tesla V100 offers
the performance of 100 CPUs in a single GPU - enabling data scientists, researchers, and
engineers to tackle challenges that were once impossible

The Tesla V100 includes the following key features:
򐂰 Volta architecture
By pairing CUDA Cores and Tensor Cores within a unified architecture, a single server
with Tesla V100 GPUs can replace hundreds of commodity CPU servers for traditional
HPC and Deep Learning.
򐂰 Tensor core
Equipped with 640 Tensor Cores, Tesla V100 delivers 125 TeraFLOPS of deep learning
performance. That’s 12X Tensor FLOPS for DL Training, and 6X Tensor FLOPS for DL
Inference when compared to NVIDIA Pascal GPUs.
򐂰 Next generation NVLINK
NVIDIA NVLink in Tesla V100 delivers 2X higher throughput compared to the previous
generation. Up to eight Tesla V100 accelerators can be interconnected at up to 300 GB/s
to unleash the highest application performance possible on a single server.
򐂰 Maximum efficiency mode

Chapter 2. System architecture 25

5472ch02.fm Draft Document for Review January 30, 2018 12:30 pm

The new maximum efficiency mode allows data centers to achieve up to 40% higher
compute capacity per rack within the existing power budget. In this mode, Tesla V100 runs
at peak processing efficiency, providing up to 80% of the performance at half the power
consumption.
򐂰 HBM2
With a combination of improved raw bandwidth of 900 GB/s and higher DRAM utilization
efficiency at 95%, Tesla V100 delivers 1.5X higher memory bandwidth over Pascal GPUs
as measured on STREAM.
򐂰 Programmability
Tesla V100 is architected from the ground up to simplify programmability. Its new
independent thread scheduling enables finer-grain synchronization and improves GPU
utilization by sharing resources among small jobs.

The Tesla V100 is built to deliver exceptional performance for the most demanding compute
applications. It delivers the following performance benefits:
򐂰 7.8 TFLOPS of double-precision floating point (FP64) performance
򐂰 15.7 TFLOPS of single-precision (FP32) performance
򐂰 125 Tensor TFLOP/s of mixed-precision.

With 640 Tensor Cores, Tesla V100 is the world’s first GPU to break the 100 teraflops
(TFLOPS) barrier of deep learning performance. The next generation of NVIDIA NVLink™
connects multiple V100 GPUs at up to 300 GB/s to create the world’s most powerful
computing servers. AI models that would consume weeks of computing resources on
previous systems can now be trained in a few days. With this dramatic reduction in training
time, a whole new world of problems will now be solvable with AI.

Multiple GPUs are common in workstations, as are the nodes of high-performance computing
clusters and deep-learning training systems. A powerful interconnect is extremely valuable in
multiprocessing systems. NVIDIA’s new NVIDIA Tesla V100 doesn’t rely on traditional PCIe
for data transfers, instead using the new NVLINK 2.0 bus that creates an interconnect for
GPUs that offer higher bandwidth than PCI Express Gen3 (PCIe) and are compatible with the
GPU ISA to support shared memory multiprocessing workloads.

Once PCIe buses are not used for data transfer, the GPU cards don’t need to comply with the
traditional PCIe card format. In order to improve density in the AC922 server the GPUs have a
different form factor called SXM2. This form factor allows for the GPU to be connected directly
on the system planar. Figure 2-14 shows the SXM2 GPU module top and bottom views and
the connectors used for the GPU modules.

26 IBM Power System AC922 Introduction and Technical Overview

fm           .Draft Document for Review January 30. 2018 12:30 pm 5472ch02.

.    Figure 2-14 SXM2 GPU module views The location of the GPUs on the AC922 system planar can be seen in Figure 2-15.

  .

     .

  .

System architecture 27 .  Figure 2-15 GPU location in a six GPU configuration Chapter 2.

enabling much tighter data sharing and improved application scaling. Multiple Links can be combined to form Gangs for even higher-bandwidth connectivity between processors. NVHS transmits data over a link called Brick that connects two processors (GPU-to-GPU or GPU-to-CPU).0 is NVIDIA’s new generation high-speed interconnect technology for GPU-accelerated computing.com/parallelforall/inside-volta 2. Although traditional NVLINK implementation primarily focuses on interconnecting multiple NVIDIA Tesla V100s together.nvidia. The NVLINK implementation in Tesla V100 supports up to six Links. Support for the GPU ISA allows programs running on NVLINK-connected GPUs to execute directly on data in the memory of another GPU and on local memory. 28 IBM Power System AC922 Introduction and Technical Overview .fm Draft Document for Review January 30. under POWER9 it also connects Tesla V100 GPUs with IBM POWER9 CPUs allowing for direct system memory access providing GPUs with an extended memory orders of magnitude larger than the internal 16 GB memory. allowing for a gang with an aggregate maximum theoretical bandwidth of 300 GB/s bidirectional bandwidth. “Physical infrastructure” on page 39. NVLink significantly increases performance for both GPU-to-GPU communications and for GPU access to system memory.4. Bricks are always combined to provide the highest bandwidth possible. 2018 12:30 pm Cooling for four GPU configurations (AC922 model 8335-GTG) is done via air while for six GPU configurations (model 8335-GTW) are water based.0 uses NVIDIA’s new High-Speed Signaling interconnect (NVHS). Figure 2-16 compares the bandwidth of the POWER9 processor connected with two GPUs and three GPUs. On Power implementation.5472ch02. A single Brick supports up to 50 GB/s of bidirectional bandwidth between the endpoints. NVLINK 2. Supported on SXM2-based Tesla V100 accelerator boards.5 NVLINK 2. For more information on server water cooling please see Chapter 3.0 NVLINK 2. For more information on the Tesla V100 please visit Inside Volta Parallel for All blog: https://devblogs. GPUs can also perform atomic memory operations on remote GPU memory addresses.

fm .Draft Document for Review January 30. 2018 12:30 pm 5472ch02.

   .

  .

 .

 .

 .

 .

   .

   .

     .

   .

 .

   .

The PCIe interface also contain the side band communication for status. Figure 2-17 shows a rear-view diagram of the AC922 server with its PCIe slots. Chapter 2.5 PCI adapters This section describes the types and functions of the PCI adapters that are supported by the AC922 server. power management. System architecture 29 .0 All the initialization of the GPU is through the PCIe interface. allowing for 32 GB/s unidirectional and 64 GB/s bi-directional bandwidth. Note: PCIe adapters on the AC922 server are not hot-pluggable. all data communication is using the NVLink.1 Slot configuration The AC922 server has four PCIe Gen4 slots. and so on.5. The AC922 server leverages the latest PCIe Gen4 technology. 2. 2. Once the GPU is up and running.        Figure 2-16 CPU to GPU and GPU to GPU interconnect using NVLink 2.

fm Draft Document for Review January 30.5472ch02. 2018 12:30 pm .

 !% •   '$" .

 !% • $ •   '$" .

!! %!$! %!#$(%#!!!$ •   • $ .

"$"  $#"& # •   &$!" "$!$ !(#&""$ (   .

% #' %!#$    .

 !%  •   "'$" &"$ • $ •   .

 !% •   '$" • $ Figure 2-17 Rear-view PCIe slots and main components Table 2-7 provides the PCIe Gen4 slot configuration. Yes half length Slot 2 has a shared connection between the two POWER9 CPUs. Table 2-7 The AC922 server PCIe slot properties Slot Description Card size CAPI capable Slot 1 PCIe Gen4 x4 Half height. When using a dual channel Mellanox InfiniBand ConnectX5 (IB-EDR) Network Interface Card (#EC64) it allows for each CPU to have direct access to the Infiniband card. 30 IBM Power System AC922 Introduction and Technical Overview . If the #EC64 card is not installed. the shared slot will operate as a single x8 PCIe Gen4 slot attached to processor 0. Figure 2-18 on page 31 shows the logical diagram of the slot 2 connected to the two POWER9 processors. Yes half length Slot 4 PCIe Gen4 x16 Half height. Yes half length Slot 3 PCIe Gen4 x16 Half height. No half length Slot 2 PCIe Gen4 x8 Shared Half height.

5.Draft Document for Review January 30. A x8 adapter can be placed in a x16 slot. Table 2-9 on page 32 summarizes the available Fibre Channel adapters. Table 2-8 lists the supported LAN adapters for the server. an LC-SC 50 micron fibre converter cable (#2456) or an LC-SC 62.3 Fibre Channel adapters The AC922 server supports direct or SAN connection to devices that use Fibre Channel adapters. Chapter 2. Table 2-8 Supported LAN adapters Feature Description Max OS code support EC2R PCIe3 LP 2-port 10GbE (NIC& RoCE) SFP28 Adapter x8 3 Linux EC2T PCIe3 LP 2-port 25/10GbE (NIC& RoCE) SFP28 Adapter x8 2 Linux EC3L PCIe3 LP 2-port 100GbE (NIC& RoCE) QSFP28 Adapter x16 2 Linux EL3Z PCIe2 LP 2-port 10/1GbE BaseT RJ45 Adapter x8 3 Linux EL4M PCIe2 LP 4-port 1GbE Adapter 4 Linux EN0T PCIe2 LP 4-Port (10Gb+1GbE) SR+RJ45 Adapter x8 3 Linux EN0V PCIe2 LP 4-port (10Gb+1GbE) Copper SFP+RJ45 Adapter x8 2 Linux 2.fm POWER9 X Bus POWER9 CPU 0 64 GB/s CPU 1 PCIe Gen4 x8 PCIe Gen4 x8 CAPI PCIe Gen4 x8 Figure 2-18 Shared PCIe slot 2 logical diagram Only LP adapters can be placed in LP slots. 2018 12:30 pm 5472ch02. which all have LC connectors. If you are attaching a device or switch with an SC-type fiber connector.5 micron fibre converter cable (#2459) is required.2 LAN adapters To connect the AC922 server to a local area network (LAN). System architecture 31 .5. you can use the LAN adapters that are supported in the PCIe slots of the system unit. but a x16 adapter cannot be placed in a x8 slot. 2.

5. The AC922 server aims to deliver a new class of technology that maximizes performance and efficiency for all types of scientific. Table 2-12 Available flash storage adapters Feature CCIN Description Max OS code support EC5A PCIe3 1.5472ch02. the server can offload processor-intensive operations to a GPU accelerator and boost performance. Table 2-11 Graphics processing units adapters that are supported Feature Description Max OS code support EC4J One air-cooled NVIDIA Tesla V100 GPUs 16 GB 4 Linux (model 8335-GTG only EC4H One water-cooled NVIDIA Tesla V100 GPUs 16 GB 6 Linux (model 8335-GTW only) 2.5 Compute intensive accelerator Compute intensive accelerators are GPUs that are developed by NVIDIA.4 CAPI-enabled InfiniBand adapters Table 2-10 shows the available CAPI adapters. machine learning. Table 2-11 lists the available compute intensive accelerators.6 TB NVMe Flash Adapter 3 Linux 32 IBM Power System AC922 Introduction and Technical Overview . Table 2-10 Available CAPI adapters Feature Description Max OS code support EC64 PCIe4 LP 2-port 100 Gb EDR InfiniBand Adapter x16 3 Linux EC62 PCIe4 LP 1-port 100 Gb EDR InfiniBand Adapter x16 3 Linux 2. deep learning. Java. engineering. artificial intelligence. and other technical computing workloads.6 Flash storage adapters The available flash storage adapters are shown in Table 2-12.5. 2018 12:30 pm Table 2-9 Fibre Channel adapters supported Feature Description Max OS code support EL43 PCIe3 LP 16 Gb 2-port Fibre Channel Adapter x8 3 Linux EL5V PCIe3 LP 32 Gb 2-port Fibre Channel Adapter x8 3 Linux 2.5.fm Draft Document for Review January 30. big-data analytics. With NVIDIA GPUs.

The integrated system ports are supported for modem and asynchronous terminal connections with Linux.Draft Document for Review January 30.fm 2. Table 2-13 Summary of features for the integrated SATA disk controller Option Integrated SATA disk controller Supported RAID types None . All SFF-4 bays support concurrent maintenance or hot-plug capability.JBOD Disk bays Two SFF Gen4 (HDDs/SDDs) SATA controllers Single IBM Easy Tier® capable controllers No External SAS ports No Split backplane No The 2.7 Internal storage The internal storage on the AC922 server contains the following features: 򐂰 A storage backplane for two 2. Table 2-14 on page 34 lists the supported devices that can be installed.1 Disk and media features The server supports the attachment of up to two SATA storage devices. Table 2-13 presents a summarized view of these features. Disk features cannot be mixed. 򐂰 The storage split backplane feature is not supported. SFF-2 or SFF-3 drives do not fit in an SFF-4 bay. Disks that are used in other Power Systems servers usually have an SFF-3 or SFF-2 carrier and are not compatible with this system. and a VGA port shown in Figure 2-17 on page 30. Any other application that uses serial ports requires a serial port adapter to be installed in a PCI slot. The VGA port does not support cable lengths that exceed three meters. one rear USB 3.5 inch or SFF SAS bays can contain SATA drives (HDD or SSD) that are mounted on a Gen4 tray or carrier (also knows as SFF-4). one Intelligent Platform Management Interface (IPMI) port.5-inch SFF Gen4 SATA HDDs or SDDs. 2018 12:30 pm 5472ch02. 򐂰 One integrated SATA disk controller (non-RAID). The integrated system ports do not support IBM PowerHA® configurations. Limitation: The disks use an SFF-4 carrier. 2.0 port).0 port (model 8335-GTG also has one front USB 3. 2.6 System ports The system board has two 1 Gbps Ethernet port. System architecture 33 . Chapter 2.7.

2018 12:30 pm Table 2-14 Supported storage devices Feature Description Max OS code support ELD0 1 TB 7.CAPI P1C14 P1C3 – PCIe Gen4 x16 .9 Location Codes The location codes for the server main components can be seen on Figure 2-19.CAPI POWER9 P1C10 CPU P1C4 – PCIe Gen4 x8 (shared) – CAPI NVIDIA Tesla V100 P1C5 – PCIe Gen4 x4 FAN GPU P1C27 P1C28 P1C29 P1C30 P1C11 P1D1 .SATA HDD/SSD Power Button / Front USB Figure 2-19 AC922 main components location codes 34 IBM Power System AC922 Introduction and Technical Overview .5472ch02.8 External I/O subsystems The AC922 server does not support external PCIe Gen3 I/O expansion drawers nor EXP24S. P1C15 DDR4 Memory DIMM P1C16 P1C6 P1C17 P1C18 NVIDIA FAN Tesla V100 GPU P1C13 POWER9 P1C7 CPU NVIDIA Tesla V100 GPU FAN P1C19 P1C20 P1C21 P1C22 P1C8 NVIDIA FRONT P1C23 Tesla V100 REAR P1C24 GPU P1C9 P1C25 FAN P1C26 NVIDIA Tesla V100 GPU P1C2 – PCIe Gen4 x16 .92 TB SSD SFF-4 disk drive 2 Linux ELU6 3.fm Draft Document for Review January 30. It does not support an internal DVD drive.2k RPM SATA SFF-4 disk drive 2 Linux ELU4 960 GB SSD SFF-4 disk drive 2 Linux ELU5 1.84 TB SSD SFF-4 disk drive 2 Linux ES6A 2 TB 7. 2. 2. and EXP24SX storage drawers. EXP12X.2k RPM 5xx SATA SFF-4 disk drive 2 Linux The AC922 server is designed for network installation or USB media installation.SATA HDD/SSD Power Supplies (x2) NVIDIA Tesla V100 GPU P1D2 .

Simple scaling.com/systems/storage/network IBM Storwize family The IBM Storwize® family is the ideal solution to optimize the data architecture for business flexibility and data storage efficiency.10 IBM System Storage The IBM System Storage® disk systems products and offerings provide compelling storage solutions with superior value for all levels of business. and tight integration with hypervisors and the OpenStack platform enable optimal storage agility for cloud environments. IBM Network Attached Storage IBM Network Attached Storage (NAS) products provide a wide-range of network attachment capabilities to a broad range of host and client systems. and IBM Storwize V7000. For more information. flexibility. 2018 12:30 pm 5472ch02. The storage system can manage a broad scope of storage workloads that exist in today’s complex data center. heterogeneous storage environments. For more information. scalability.ibm. see the following website: http://www. high-capacity.com/systems/storage/disk/xiv/index. IBM offers a comprehensive flash portfolio with the IBM FlashSystem family. Easy Tier. and total overall value for the most demanding. such as IBM Scale Out Network Attached Storage and the IBM System Storage N series. see the following website: http://www. offer storage virtualization.com/systems/storage/storwize IBM FlashSystem family The IBM FlashSystem® family delivers extreme performance to derive measurable economic value across the data architecture (servers. from entry-level to high-end storage systems.ibm. IBM Storwize V5000.com/systems/storage/flash IBM XIV Storage System The IBM XIV® Storage System is a high-end disk storage system. resiliency. see the following website: http://www. XIV Storage Systems extend ease of use with integrated management for large and multi-site XIV deployments. Chapter 2.ibm.com/systems/storage/disk The following sections describe sample System Storage offerings. applications. such as the IBM Storwize V3700.ibm. and many more functions. IBM Real-time Compression™. and secure storage system that delivers the highest levels of performance. doing it effectively and efficiently. see the following website: http://www. software. and storage).Draft Document for Review January 30.ibm. see the following website: http://www. heterogeneous workloads.html IBM System Storage DS8000 The IBM System Storage DS8800 storage system is a high-performance. reducing operational complexity and enhancing capacity planning. For more information. helping thousands of enterprises meet the challenge of data growth with hotspot-free performance and ease of use. high service levels for dynamic.fm 2. Different models. For more information about the hardware and software. For more information about the various offerings. System architecture 35 .

For more information about Ubuntu for POWER9. Red Hat provides separate builds and licenses for big endian and little endian versions for Power. and also provide the highest levels of reliability and system uptime.ubuntu. For more information.html 2. see the following website: http://www.3 Additional information For more information about the IBM PowerLinux™ Community.com/developerworks/group/tpl For more information about the features and external devices that are supported by Linux.com/systems/power/software/linux/index. see the Linux on Power Systems website: http://www. the IBM System Storage DS8000® storage system includes a range of features that automate performance optimization and application quality of service.com/ecosystem/hardware/2689861 2.2 Red Hat Enterprise Linux Red Hat Enterprise Linux 7.11. see the following website: https://www.fm Draft Document for Review January 30. which provides a UNIX-like implementation across many computer architectures.com/systems/storage/disk/ds8000/index. see the following website: https://www.04.1 Ubuntu Ubuntu Server 16.ibm. cross-platform OS. 2. For more information about RHEL for POWER9.11.ibm.1. 2018 12:30 pm Additionally.5472ch02.ibm.redhat.11 Operating system support The AC922 server supports Linux. Linux on Power Systems is the only Linux infrastructure that offers both scale-out and scale-up choices. see the following website: http://www. Starting with Red Hat Enterprise Linux 7. For more information about the software that is available on Power Systems servers.11.html 36 IBM Power System AC922 Introduction and Technical Overview .4 for Power LE (POWER9) and any subsequent updates are supported. It is supported on every Power Systems server IBM sells.com/systems/power/software/linux/index.com/download/server 2. see the following website: https://access.03 LTS and any subsequent updates are supported.html The Linux operating system is an open source.ibm.

For more information. the prepackaged Java that is part of a Linux distribution is designed to meet the most common requirements. there are several resources available.12 Java When running Java applications on the POWER9 processor. For current information about IBM Java and tested Linux distributions. see the following website: http://cr.net/ubuntu/+source/openjdk-9 https://launchpad.net has resources for Ubuntu builds.net/~simonis/ppc-aix-port/ Launchpad.fm 2. If you require a different level of Java. see the following webistes: https://launchpad.html For additional information about the OpenJDK port for Linux on PPC64 LE and pregenerated builds.com/developerworks/java/jdk/linux/tested.net/ubuntu/+source/openjdk-7 Chapter 2.Draft Document for Review January 30.net/ubuntu/+source/openjdk-8 https://launchpad.ibm.openjdk. 2018 12:30 pm 5472ch02.java. System architecture 37 . see the following website: https://www.

5472ch02. 2018 12:30 pm 38 IBM Power System AC922 Introduction and Technical Overview .fm Draft Document for Review January 30.

© Copyright IBM Corp.Draft Document for Review January 30. Physical infrastructure The objective of this section is to summarize all the physical infrastructure requirements regarding the AC922 servers. 2018 12:30 pm 5472ch03. All rights reserved.fm 3 Chapter 3. Additional information can be found in the Knowledge Center for the AC922 server. 39 . 2017.

60°C (41 .fm Draft Document for Review January 30. Tip: The maximum measured value is expected from a fully populated server under an intensive workload.80% 8 . Use the IBM Systems Energy Estimator to obtain a heat output estimate that is based on a specific configuration. 2018 12:30 pm 3.27 °C (64 .com/see/EnergyEstimator 3.3 in.0 mm (33.2 Physical package Table 3-2 shows the physical dimensions of the chassis.80% Maximum dew point 24°C (75° F) 27°C (80°F) Operating voltage 200 .240 V AC N/A Operating frequency 50 .ibm.4 in.6 kVA maximum N/A Thermal output 8872 BTU/hr maximum N/A Maximum altitude 3050 m N/A (10.1 Operating environment Table 3-1 provides the operating environment specifications for the AC922 server.5 mm (17.3 Hz N/A Power consumption 2300 watts maximum N/A Power source loading 2.140°F) Recommended: 18 .5472ch03.000 ft.6/6. The maximum measured value also accounts for component tolerance and operating conditions that are not ideal.7 bels operating/idling N/A a. Power consumption and heat load vary greatly by server configuration and usage. The estimator is available at the following website: http://www-912. Heavy workloads might see some performance degradation above 35°C if internal temperatures trigger a CPU clock reduction.60 Hz +/.) Noise level and sound power 7.) Depth 845. The server is available only in a rack-mounted form factor and requires 2U (2 EIA units) of rack space.80 °F) Relative humidity 8 . Table 3-2 Dimension The AC922 server models 8335-GTG and 8335-GTW Width 441.40°Ca 1 .104°F) (34 . Table 3-1 Operating environment for the 4-GPU 8335-GTG AC922 server Server operating environment Description Operating Non-operating Temperature Allowable: 5 .) 40 IBM Power System AC922 Introduction and Technical Overview .

3 System power The AC922 server is powered by two 2200W power supplies located in the rear of the unit.) Table 3-3 Physical dimensions for the AC922 server 3. An example of the power cable with its connectors can be seen in Figure 3-1. The power supplies are designed to provide redundancy in case of a power supply failure. The PDU connector type (IEC C20 or IEC C19) depends on the selected rack PDU. A new power cable to connect the power supplies to the PDUs in the rack is required rending the reuse of existing power cables not viable. 2018 12:30 pm 5472ch03.4 in. throttling may occur in case of a power supply failure when six GPUs are installed. Once GPUs are the largest power consuming devices in the server. depending on the configuration and utilization.) Weight (maximum configuration) 30 kg (65 lbs. The power supplies on the server use a new Rong Feng 203P-HP connector. .fm Dimension The AC922 server models 8335-GTG and 8335-GTW Height 86.0 mm (3. In this case system remain operational but may experience reduced performance until the power supply is replaced.Draft Document for Review January 30.

.

   .

 .

“Compute intensive accelerator” on page 32 for a list of available GPUs.5. If 1-phase PDUs are selected a minimum of 5 PDUs are required to support a full rack of 18 AC922 servers with four GPUs configuration. Chapter 3. Figure 3-1 AC922 power cables with the Rong Feng connector Both 1-phase and 3-phase PDUs are supported.5. Once the 1-phase PDUs are limited to 48A. a total of 4 PDUs will be required to support a full rack with 18 AC922 servers configured with four GPUs. Physical infrastructure 41 .5. For more information see 3. When opting for 3-phase 60A PDUs. 3. no more than four AC922 servers can be connected to a single PDU. See 2. “AC power distribution units” on page 49.2.4 System cooling Air or water cooling depends on the model of the server and the GPUs feature codes selected.

the system that supplies the cooling water must be able to measure the room dew point and automatically adjust the water temperature accordingly. all components are air cooled.5472ch03.6°C (39°F . In air cooled systems (8335-GTG). customers is responsible for providing the system that supplies the chilled conditioned water to the rack. Typical primary chilled water is too cold for use in this application because building chilled water can be as cold as 4°C .43°F). A picture with the internal view of the server and the two processors and four GPUs heatsinks installed can be seen in Figure 3-2. including processors and GPUs that use heatsinks. When using water cooled systems. the water temperature must be above the maximum dew point for that data center installation.  . Otherwise.fm Draft Document for Review January 30. 2018 12:30 pm Rack requirement: The IBM 7965-S42 rack with feature #ECR3 or #ECR4 installed supports the water cooling option for the AC922 server (see “Optional water cooling” on page 47). Once water condensation can occur in low temperatures.

  .

Water lines carrying cool water in and warm water out are also shipped. 42 IBM Power System AC922 Introduction and Technical Overview . Cold plates to cool two processor modules and up to six GPUs are shipped. PCIe adapters and power supplies are cooled using traditional air cooling systems.  Figure 3-2 AC922 air cooled model internal view In water cooled systems (8335-GTW) processors and GPUs are cooled using water while other components like memories. This feature is installed in the system unit when the server is manufactured and is not installed in the field.

fm When ordering the AC922 model 8335-GTW a cooling kit is required.           . The feature code #XXX will provide with the internal cooling system of the server as shows in Figure 3-3. It contain the pipes.Draft Document for Review January 30. 2018 12:30 pm 5472ch03. coldplates and splitters required to cool the system.

    Figure 3-3 Internal cooling components for the 8335-GTW server A view of the cooling system installed in the server can be seen in Figure 3-4.  .

         .

Physical infrastructure 43 .    Figure 3-4 Internal cooling installed in a 8335-GTW model server Chapter 3.

fm Draft Document for Review January 30. 2018 12:30 pm A detailed view on a processor and three GPUs cooling can be seen in Figure 3-5.5472ch03. .

         .

     .

A picture showing cold water in blue and warm water in red can be seen in Figure 3-6. In each system the water flows in one direction passing first by the processor and then through three GPUs.  Figure 3-5 Processor and GPU water cooling details Water enter the system and pass through a splitter to two separate systems. Then. the warm water returns to a return line splitter and out the server.      .

The GPU air-cooled and water-cooled servers have the following ordering differences: 򐂰 With an air-cooled server. an air-cooled server cannot be changed into a water-cooled server. and a water-cooled server cannot be changed into an air-cooled server.     Figure 3-6 Cold and warm water flow through the AC922 system When shipped from IBM. 44 IBM Power System AC922 Introduction and Technical Overview . Customer setup is not supported for water-cooled systems. (8335-GTG). an initial order can be ordered with two GPUs or four GPUs feature #EC4J.

Order information: The AC922 server cannot be integrated into these racks during the manufacturing process and are not ordered together with servers.5.fm 򐂰 With a water-cooled server (8335-GTW). see the following website: http://www. serviceable. model 8335-GTW server only mounts in the 42U IBM Enterprise Slim Rack (7965-S42). weight.4.ibm. cable management.4.com/support/knowledgecenter/POWER8/p8had/p8had_83x_watercool. Responsibility: The client is responsible for ensuring that the installation of the drawer in the preferred rack or cabinet results in a configuration that is stable. Ordering this model with slide rails is not supported. Maintenance of components other than power supplies and fans must be done on a bench with the server unplugged from the cooling system.5.htm 3. water-cooled options have just one rack supported as can be seen in Table 3-4: Table 3-4 Supported racks by model Type-Model Description Supported by Supported by 8335-GTG 8335-GTW 7014-T00 IBM 7014 Rack Model T00 Yes No 7014-T42 IBM 7014 Rack Model T42 Yes No 7014-S25 IBM Entry Rack Cabinet Model S25 Yes No 7965-94Y IBM 42U Slim Rack Yes No 7965-S42 IBM Enterprise Slim Rack Yes Yes N/A OEM 19” Rack See 3.5 Rack specifications Depending on the model chosen for the AC922 server. a quantity of four or six feature #EC4H GPUs must be ordered. For more information about the water cooling option. Physical infrastructure 45 . safe. 2018 12:30 pm 5472ch03. These racks are built to the 19 inch EIA 310D standard. Chapter 3. and compatible with the drawer requirements for power. If a system is installed in a rack or cabinet that is not an IBM rack. IBM does not offer integration of the server into the rack before shipping. While the air cooled model (8335-GTG) supports a wide variety of racks. Note: AC922 model 8335-GTW only offers the fixed rail kit option. If the server and any of the supported IBM racks are ordered together.Draft Document for Review January 30. there are different racks supported. and rail security. “OEM No racks” on page 52 Note: Due to the water cooling system. they are shipped at the same time in the same shipment but in separate packing material. “OEM racks” on page 52. ensure that the rack meets the requirements that are described in 3. cooling.

filler panels are mounted in the front of the rack in empty EIA locations and the rack offers perforated front and rear door designs. Figure 3-7 IBM 7965-S42 racks front view 46 IBM Power System AC922 Introduction and Technical Overview .) width combined with its 1070 mm (42.5472ch03.) depth plus its 42 EIA enclosure capacity provides great footprint efficiency for your systems and allows it to be easily placed on standard 24-inch floor tiles.5. “AC power distribution units” on page 49 To allow maximum airflow through the datacenter and the rack cabinets. The IBM power distribution units (PDU) are mounted vertically in four (4) side bays. non-blade servers. 2018 12:30 pm 3.1 in. Its 600 mm (23. two (2) on each side. and I/O drawers. For more information on IBM PDUs. After the side bays have been filled.5.0-meter (79-inch) Model 7965-S42 is compatible with past and present Power Systems servers and provides an excellent 19-inch rack enclosure for your data center.2. Another difference between the 7965-S42 model rack and the 7014-T42 model rack is that the “top hat” is on the 40U and 41U boundary instead of the 36U and 37U boundary in the 7014-T42 model. PDUs can be mounted horizontally at the rear of the rack. A front view of the 7965-S42 rack can be seen in Figure 3-7 on page 46. please see 3. This is a 19" rack cabinet that provides 42U of rack space for use with rack-mounted. allowing for better thermal and cable management capabilities.6 in.fm Draft Document for Review January 30.1 IBM Enterprise Slim Rack 7965-S42 The new 2.

Since the hose exits may require some space inside the rack. it is recommended that a 2U space must be left vacant on the top or bottom of the rack depending on the location of the hoses chosen. The feature #ECR4 can be used to order the manifold with water input and output at the bottom of the rack. The feature #ECR3 indicates the manifold with water input and output at the top of the rack. 2018 12:30 pm 5472ch03. water cooling is mandatory and therefore the 7965-S42 rack must be ordered with the water cooling option (features #ECR3 or #ECR4). These features represents a manifold for water cooling and provides water supply and water return for one to 20 servers mounted in a 7965-S42 Enterprise Slim Rack.   . There are no MES for these features in the field.Draft Document for Review January 30. Figure 3-8 on page 47 shows both options of water input and output.fm Ballasts for additional stability will be available therefore it is expected that the 7965-S42 racks will not require the depopulate rules above the 32 EIA location as required with 7014-T42 rack models. Optional water cooling When opting for model 8335-GTW of the AC922 server.

.

     .

.

            Figure 3-8 Top and bottom water input and output for 7965-S42 rack Chapter 3. Physical infrastructure 47 .

48 IBM Power System AC922 Introduction and Technical Overview . The manifold does not interfere with the placement of servers or other I/O drawers. Figure 3-10 on page 49 shows a manifold for the 7965-S42 rack. Quick connect fittings are located every 2U on the manifold for water supply and return providing 20 pairs of fittings.fm Draft Document for Review January 30. 2018 12:30 pm Figure 3-9 on page 48 shows a datacenter rock if 7965-S42 racks with water input and output at the top of the rack. Figure 3-9 Datacenter rack row with water input and output at the top of racks The manifold is mounted on the right side of the rack as viewed from the rear and extends for 40U.5472ch03.

PDUs include the AC power distribution unit #7188 and the AC Intelligent PDU+ #7109. or use vertically mounted PDUs on the left side of the rack.com/support/knowledgecenter/POWER8/p8had/p8had_7965s_watercool.Draft Document for Review January 30. one console serial port. The Intelligent PDU+ is identical to #7188 PDUs.fm Figure 3-10 Manifold for the 7965-S42 rack The servers are connected to the manifold through quick-connects. The manifold makes access to PDU impossible.htm Important: Avoid vertically mounted PDUs on the right side as viewed from the rear of the rack. and one RS232 serial port for power monitoring. You must provide a 1-inch female national pipe thread (FNPT) fitting for each hose and must provide treated water.5 mm (1. Outer diameter of the hoses is approximately 34. see the site and hardware planning documentation at: https://www. Supply and return hoses from the manifold to the server are provided as part the server cooling feature.ibm. Use either horizontally mounted PDUs. not generic building water. Chapter 3.25 m (14-foot) hose kits are provided with the manifold to connect water supply and return. but it is equipped with one Ethernet port.36 in). 2018 12:30 pm 5472ch03. Physical infrastructure 49 . For more information.2 AC power distribution units AC power distribution is done via power distribution units (PDUs). Two 4. 3.5. The manifold has one cold water inlet that leads to the rack and one warm water outlet.

Each 50 IBM Power System AC922 Introduction and Technical Overview . Each outlet is rated up to 10 amps. two additional PDUs can be installed horizontally in the T00 rack and three in the T42 rack. the preferred approach is to use fillers in the EIA units that are occupied by these PDUs to facilitate the correct airflow and ventilation in the rack. Each PDU requires one PDU-to-wall power cord. When mounting PDUs horizontally. which must be ordered separately. Six groups of two outlets are fed by six circuit breakers. In the rear of the rack. but each group of two outlets is fed from one 15 amp circuit breaker. 2018 12:30 pm The PDUs have 12 client-usable IEC 320-C13 outlets. The four vertical mounting locations are filled first in the T00 and T42 racks. Various power cord features are available for various countries and applications by varying the PDU-to-wall power cord. Four PDUs can be mounted vertically in the back of the T00 and T42 racks. Mounting PDUs horizontally consumes 1U per PDU and reduces the space that is available for other racked components. Figure 3-11 shows the placement of the four vertically mounted PDUs.fm Draft Document for Review January 30. Rack Rear View Circuit breaker reset Status LED 3 4 1 2 Figure 3-11 PDU placement and PDU view The PDU receives power through a UTG0247 power-line connector.5472ch03.

32A 6657 PDL 230 .ibm. To better enable electrical redundancy. Canada. 32A 6654 NEMA L6-30 200 . For detailed power requirements and power cord details about the 7014 racks. LA. which are not included in the base order.208. LA.fm power cord provides the unique design characteristics for the specific power requirements. Canada. 230 1 63 amps EMEA 63A Notes: Ensure that the correct power cord feature is configured to support the power that is being supplied.htm Chapter 3. Table 3-5 shows the available wall power cord options for the PDU and iPDU features. Canada. the PDU can supply 4.8 . Physical infrastructure 51 .Draft Document for Review January 30. The power of all of the drawers that are plugged into the PDU must not exceed the power cord limitation. To match new power requirements and save previous investments. 230 1 24 amps EMEA P+N+G. 230 3 16 amps/phase Internationally available 3P+N+G. 240 1 48 amps US. Based on the power cord that is used. see IBM Knowledge Center: http://www.208. 16A 6489 IEC309 230 3 32 amps/phase EMEA 3P+N+G. and Japan 60A 6491 IEC 309. For maximum availability.208. 2018 12:30 pm 5472ch03.com/support/knowledgecenter/api/redirect/powersys/v3r1m5/topic/p7ha d/p7hadrpower. 240 1 24 amps US. The Universal PDUs are compatible with previous models.240 1 32 amps Australia and New Zealand 6658 Korean plug 220 1 30 amps North and South Korea 6492 IEC 309.2 kVA. 200 . 2P+G. P+N+G. Table 3-5 Wall power cord options for the PDU and iPDU features Feature Wall plug Rated voltage Phase Rated amperage Geography code (Vac) 6653 IEC 309. 240 1 24 amps US. which must be ordered separately. a preferred approach is to connect power cords from the same system to two separate PDUs in the rack.19. and Japan (watertight) 6656 IEC 309. these power cords can be requested with an initial order of the rack or with a later upgrade of the rack features. each server has two power supplies that must be connected to separate PDUs. and Japan 6655 RS 3750DP 200 . and to connect each PDU to independent power sources. LA.

jsp The website mentions the following key points: 򐂰 The front rack opening must be 450 mm wide ± 0.). 3.75 mm (17. This standard is published by the Electrical Industries Alliance.com/support/knowledgecenter/api/redirect/systems/scope/hw/index. 52 IBM Power System AC922 Introduction and Technical Overview .ibm.5472ch03.4 OEM racks The system can be installed in a suitable OEM rack if that the rack conforms to the EIA-310-D standard for 19-inch racks.fm Draft Document for Review January 30. 2018 12:30 pm For detailed power requirements and power cord details about the 7965-94Y rack. ± 0.5.Figure 0-1 on page 53 is a top view that shows the specification dimensions. For rack stability.htm 3.ibm.72 in. 򐂰 Any remaining space in the rack can be used to install other systems or peripheral devices if the maximum permissible weight of the rack is not exceeded and the installation rules for these devices are followed.com/support/knowledgecenter/api/redirect/powersys/v3r1m5/topic/p7ha d/p7hadkickoff795394x. start filling a rack from the bottom.3 Rack-mounting rules Consider the following primary rules when you mount the system into a rack: 򐂰 The system can be placed at any location in the rack.5. see IBM Power Systems Hardware IBM Knowledge Center: http://www. For more information. see IBM Knowledge Center: http://www. be sure to follow the rack manufacturer’s safety instructions regarding rack stability. 򐂰 Before placing the system into the service position.03 in.

) apart on-center (horizontal width between the vertical columns of holes on the two front-mounting flanges and on the two rear-mounting flanges) as seen on Figure 3-12 on page 54.03 in.Draft Document for Review January 30. Chapter 3.fm Figure 0-1 Top view of rack specification dimensions (not specific to IBM) 򐂰 The rail-mounting holes must be 465 mm ± 0. Physical infrastructure 53 . 2018 12:30 pm 5472ch03.3 in. ± 0.8 mm (18.

2018 12:30 pm Figure 3-12 Mounting flange dimensions 򐂰 The vertical distance between the mounting holes must consist of sets of three holes spaced (from bottom to top) 15.fm Draft Document for Review January 30. which makes each three-hole set of vertical hole spacing 44. ± 0.28 in.67 mm (0. 15. Figure 0-2 on page 55 shows the top front specification dimensions.1 mm (0.45 mm (1.).1 mm ± 0.75 in.). Rail-mounting holes must be 7.9 mm (0.5 in.) apart on center.9 mm (0. 54 IBM Power System AC922 Introduction and Technical Overview .625 in.625 in.004 in. and 12.) in diameter.) on-center.5472ch03.

99 in.) of depth within the rack from the rear rack mount flange to the frame line as shows in Figure 3-13 on page 56.) is needed behind the installed system for maintenance.Draft Document for Review January 30. Physical infrastructure 55 . 2018 12:30 pm 5472ch03.fm Figure 0-2 Rack specification dimensions top front view 򐂰 A minimum rack opening width of 500 mm (19.69 in. service and cable management.) for a depth of 330 mm (12. Chapter 3. Recommended depth is at least 254 mm (10 in.

2018 12:30 pm Figure 3-13 OEM rack opening depth 56 IBM Power System AC922 Introduction and Technical Overview .fm Draft Document for Review January 30.5472ch03.

and artificial intelligence to reason and act upon data that couldn’t be used until now like structure. © Copyright IBM Corp. Deep Learning consists of algorithms that permit software to train itself by exposing multilayered neural networks to vast amounts of data. in all forms. 򐂰 Easy updates: code updates arrive via repository. 򐂰 Distributed as easy to install binaries. and moving the model into production. configuration and adoption of deep learning. videos and sounds. many data points.fm 4 Chapter 4. which develop that neural model of understanding by building weights and connections between many. testing. optimizing. IBM IBM Cognitive Systems created an offering called PowerAI. Taking advantage of a distributed architecture. 57 . On the other hand.Draft Document for Review January 30. Cognitive computing brings value to your organization’s data by an entirely new approach to problems: using deep learning. and training the model. is expanding as a resource to be utilized. It is most frequently used to perform tasks like speech and image recognition but can be used to virtually any kind of data. often millions in a training data set. machine learning. IBM PowerAI Data. unstructured. It offers many optimizations that can ease installation and management. Incorporating complete lifecycle management from installation and configuration. 2017. to inference. The intelligence in the process sits within the deep learning software frameworks themselves. 򐂰 Validated deep learning platform with each release. images. In order to ease the process of installations. All rights reserved. PowerAI brings a suite of capabilities from the open source community and combines them in to a single enterprise distribution of software. in many industries. 2018 12:30 pm 5472ch04. 򐂰 Includes all dependencies and libraries. and can accelerate performance: 򐂰 Ready-to-use deep learning frameworks (Tensorflow and IBM Caffe). data ingest and preparation. building. the data explosion is outstripping the human capacity to understand the meaning hidden within that data. PowerAI can help enable your teams to quickly iterate through the training cycle on more data to help continuously improve the model over time.

1 Deep learning frameworks included and requirements IBM PowerAI release 4 has the following requirements for the operating system: 򐂰 Ubuntu 16. 򐂰 NVIDIA CUDA version 8.3.0. 8.0 4. Table 4-1 Packages included in IBM PowerAI release 4 Package Version Tensorflow 1.9.0 IBM Caffe 1.0rc1 Chainer 1. 6. Distributed Deep Learning.20 is recommended The Table 4-1 lists the deep learning frameworks included in the current release.23 Torch 7 BVLC Caffe 1.61 is recommended 򐂰 NVIDIA cuDNN 6.0 is required.0 NCCL 1.15. IBM PowerAI brings a set of unique and innovative tools to allow for companies to adopt Cognitive computing like PowerAI Vision.fm Draft Document for Review January 30. 򐂰 Designed for enterprise scale with multisystem cluster performance and large memory support.0.5472ch04. 2018 12:30 pm 򐂰 Dedicated support teams for deep learning.0 NVCaffe 0.0 TensorFlow Operator for IBM Distributed DL Library (DDL) 1.2.0.1.44 or higher is required.0. 4.2 PowerAI Vision Most of enterprises are facing challenges adopting Artificial Intelligence techniques to understand and act upon their data.04 (Architecture: ppc64le) 򐂰 NVIDIA driver version 384.66 or higher is required.0.14 DIGITS 05.0rc1 IBM Distributed Deep Learning Library Docs (DDL) 1.19 Theano 0. 򐂰 Supported with IBM Power servers with NVLink attached CPUs and NVIDIA GPUs. The most common challenges are: 򐂰 No experience on development team on computer vision applications 򐂰 Lack of skills on Deep Neural Networks (DNN) design and development 58 IBM Power System AC922 Introduction and Technical Overview .2 OpenBLAS 0.0. and Large Model Support covered in the following chapters.

  !    "! "  "               "" #            "   ! "# .fm 򐂰 Poor understanding on how to build a platform to support enterprise scale deep learning. training. including data preparation. PowerAI Vision focus on optimizing the workflow required for deep learning as shown on Figure 4-1. allowing users with little experience in deep learning to perform complex image analysis with virtually no coding necessary.Draft Document for Review January 30. 2018 12:30 pm 5472ch04. and inference PowerAI vision comes to help solve these issues by bringing a tool to train models to classify images and deploy models to inference real time images.

most of the time in deep learning is spent in preparing the data to be ingested by the system. PowerAI vision is distributed as a Technology Preview For more information please see: https://developer. PowerAI Vision allow the user to prepare the data for deep learning training with the following pre-processing features: 򐂰 Copy bundled sample datasets to jump start your learning of PowerAI Vision 򐂰 Create categories. Individual image analysis. deploying and engaging trained models to infer categories and detect occurrences of trained objects in test and real-time datasets Note: At the time of writing. IBM PowerAI 59 .com/linuxonpower/deep-learning-powerai/technology-previews/ Chapter 4. resizing and cropping Once the model is trained. load datasets for training by dragging and dropping images from local disks 򐂰 Create labels. Train models with the labelled objects 򐂰 Optionally import datasets for training in ZIP formats 򐂰 Preprocess images including rotating. categorization and data transformation are some of the steps involved in this phase. !  #"# "  "  # "  Figure 4-1 Traditional deep learning steps required optimized by PowerAI Vision Usually.ibm. mark objects on images and assign to labels. PowerAI Vision allows the use of a simple GUI interface to trigger deep learning inference thru APIs.

As a result of this capability PowerAI with Distributed Deep Learning can scale jobs across large numbers of cluster resources with very little loss to communication overhead.5472ch04. data scientists are forced to make trade-offs to stay within the 16 GB memory limits of each GPU. For more information please see: https://developer. the amount of data required to be processed exceeds the capacity processing of a single server. Distributed Deep Learning is avail-able as a technology preview with PowerAI 4. forcing data scientists to split the problem being solved into hundreds of smaller problems. This consumes even more time on preparation and modeling once not all data can be analyzed at once. In other cases the training time could potentially be reduced by splitting the tasks among a cluster of servers. PowerAI is unique in that this capability is incorporated in to the Deep Learning frameworks as an integrated binary. reducing complexity for clients as they bring in high performance cluster capability. when data scientists develop a deep learning workload the structure of matrices in the neural model and the data.0.ibm. 2018 12:30 pm 4. To accelerate the time dedicated to training a model. As models grow in complexity (deeper neural networks contain more layers and larger matrices) and data sets increase in size (high definition video vs. the PowerAI stack uses new technologies to deliver exceptional training performance by distributing a single training job across a cluster of servers. web scale images).4 Large model support One of the challenges customers face in the deep learning space is that they are limited by the size of memory available within GPUs. Note: At the time of writing.fm Draft Document for Review January 30.3 Distributed deep learning Many times. and is compatible with bundled TensorFlow and IBMCaffe frameworks.com/linuxonpower/deep-learning-powerai/releases/ 4. PowerAI’s Distributed Deep Learning (DDL) brings intelligence about the structure and layout of the underlying cluster (topology) which includes information about the location of the cluster’s different compute resources such as Graphical Processing Units (GPUs) and CPUs and the data on each node. Today. elements which train the model (in a batch) must sit within the memory on GPU. 60 IBM Power System AC922 Introduction and Technical Overview . This is a problem because today’s GPUs memory is mostly limited to 16 GB.

2018 12:30 pm 5472ch04.fm                    .Draft Document for Review January 30.

 .

data elements and batch or set sizes significantly.                                                                                                                                                                                      Figure 4-2 Traditional approach splitting the problem into 16 GB chunks With Large Model Support. enabled by PowerAI’s unique NVLink connection between CPU (memory) and GPU. with the outcome of executing far larger models and expanding up to nearly 1 TB of system memory across all GPUs. Users now have the ability to increase model sizes. ! !   . the entire model and dataset can be loaded in to system memory and cached down to the GPU for action.

 .

   .

         .

 .

 .

 .

 .

 .

 .

 .

.

.

.

.

 .

.

IBM PowerAI 61 .  Figure 4-3 Large Model Support approach with larger than 16 GB data being consumed by GPUs Chapter 4.

fm Draft Document for Review January 30.ibm.5 Software download If you have an IBM Power AC922 and would like to test and use IBM PowerAI.ibm. increasing organizational efficiency.5472ch04. and is compatible with bundled TensorFlow and IBMCaffe frameworks For more information please see: https://developer. Large Model Support is available as a technology preview with PowerAI 4. Note: At the time of writing. 2018 12:30 pm This capability is unique to PowerAI and opens up the opportunity to address larger challenges and get much more work done within a PowerAI single server.com/linuxonpower/deep-learning-powerai/releases/ 4.04 LTS from the following source: https://ibm. you can download PowerAI distributed as a binary for Ubuntu 16.com/linuxonpower/deep-learning-powerai/releases/ 62 IBM Power System AC922 Introduction and Technical Overview .biz/powerai Installation guides and other pertinent informations can be obtained on https://developer.0.

view. 63 .com/en-us/data-center/tesla-v100/ 򐂰 NVIDIA Tesla V100 Performance Guide http://images.org/ 򐂰 NVIDIA Tesla V100 https://www. REDP-5405 򐂰 IBM PowerAI: Deep Learning Unleashed on IBM Power Systems.POWER9 Monza Module https://www-355. IBM Redbooks The following IBM Redbooks publications provide additional information about the topic in this document.com/redbooks Online resources These websites are also relevant as further information sources: 򐂰 OpenPOWER Foundation https://openpowerfoundation. 2018 12:30 pm 5472bibl. 򐂰 IBM Power System S822LC for High Performance Computing Introduction and Technical Overview.com/services © Copyright IBM Corp.nvidia. 2017. Web Docs. All rights reserved.fm Related publications The publications listed in this section are considered particularly suitable for a more detailed discussion of the topics covered in this paper. Redpapers.com/support IBM Global Services ibm.pdf 򐂰 IBM Portal for OpenPOWER .com/content/pdf/volta-marketing-v100-performance-guide-us- r6-web. download or order these documents and other Redbooks. at the following website: ibm.Draft Document for Review January 30.xhtml?al iasId=POWER9_Monza 򐂰 OpenCAPI http://opencapi. draft and additional materials.org/technical/use-cases/ Help from IBM IBM Support and downloads ibm. SG24-8409 You can search for.nvidia. Note that some publications referenced in this list might be available in softcopy only.com/systems/power/openpower/tgcmDocumentRepository.ibm.

fm Draft Document for Review January 30. 2018 12:30 pm 64 IBM Power System AC922 Introduction and Technical Overview .5472bibl.

.

Back cover Draft Document for Review January 30. ® ibm. 2018 12:34 pm REDP-5472-00 ISBN DocISBN Printed in U.S.com/redbooks .A.