Ey SAPNote 3102813 - SAP HANA on VMware vSphere 7.0 U2/U3 with up to 12 TB DRAM 448 vCPUs VM sizes ‘Component: BC-OP-LNX-ESX (Linux on VmWare ESX), Version: 19, Released On: 13.12.2023 | Symptom You want to instal SAP HANA platform in a VMware vSphen TB with 224 vCPUs and up to 12 TB DRAM with 448 vCPUs. 0 U2/U3 virtualized environment with VM sizes more than 6 | Other Terms ESXi, VMW, vSphere, virtualized SAP HANA, VMware, 12 TB VM | Reason and Prerequisites SAP declares support for VMware vSphere 7.0 U2/U3 (U3 requires at least version Use) fo virtual deployments in production scenarios of SAP HANA on either certified appliances or through SAP HANA tailored data center integration (TDD verified hardware configurations. The SAP HANA start release defined as "SAP HANA 1.0 SPS xx" includes support for SAP HANA 2.0. SAP always recommends the use of HANA 2.0. Certified Hardware can be found in the Certified and ‘Supported SAP HANA® Hardware directory (3) In case of TDI, SAP HANA certified Linux versions (httns://launchpad support sap.com/# /notes/2095584) are also supported inside a VMware vSphere VM. | Solution By issuing this SAP Note, SAP extends support for SAP HANA on VMware vSphere 7.0 Ua/U3 with VM sizes up to 12.TB DRAM (PMEM restrictions are dseribed in note 2013430), providing the following conditions are met: SAP HANA start release: + Intel Cascade Lake and Cooper Lake: SAP ITANA 1.0 SPS 12 Revision 122.19; SAP TIANA 2.0 recommended Important information: The validations have shown a limitation with OLTP workload on 8 socket VMs. For $/4HANA. standard workload, OLTP transactional request times show an overhead of up to 100 ms when accessed via the ‘VMware virtualized network adapter (VMXNET3). ‘This overhead leads to a transactional throughput loss, which varied between 8% and 12%, running at a very high system load, when compared to the underlying bare metal environment. ‘VMware has identified a missing network optimization and is working on a solution. Documentation and timeline are provided by VMware in this KB article. Further questions on optimizations to lower the network latencies need to be addressed to the VMware support team on component BC-OP-LNX-ESX. For performance critical OLTP workload, the VMXNET3 adapter should therefore not be used without further validation, A PCI passthrough network device did not show this impact on OLTP transactional request times and, transactional throughput and have fulfilled the SAP HANA validation requirements. Due to a potential misalignment of memory with VM sizes > 8 TB, in combination with passthrough network devices, customers need to verify the VM memory configuration as described in VMware KB 87730, When a memory misalignment gets detected, then install the KB attached hot patch. If this is not possible, then apply the deseribed workaround to align the VM memory size manually Note: Some virtual machine operations like vMotion become unavailable when @ PCI or PCTe passthrough device gets added to a virtual machine. See further remarks on these different configurations below. Single and multiple SAP HANA production virtual machines on a single physical server CPU architecture (Intel glueless architecture only): + Intel Cascade Lake processors of TDIvg Model 2 & 4 & 6 & 8 sockets + Intel Cooper Lake processors of TDIvs Model 2 & 4 & 6 & 8 sockets Important information for CPU configurations not following the Intel glueless reference architecture: ‘VMware with SAP HANA is supported on 4- and 8-socket HPe Superdome Flex servers with glued architecture and Cascade Lake CPUs. In these configurations, the maximum supported RAM size of a VM is 6TB. A single VM must not span across the chassis. Details see http://www hpe.com/psnow/doe/as0001355¢n0The minimum size of a virtual SAP HANA instance is a half socket (represented by at least 8 physical cores) and 128 GBofRAM. The maximum size of a virtual SAP HANA instance is 448 vCPUs with vSphere 7 U2/U3, The maximal memory is limited to 12 TB. For the maximal supported 8-socket VM size, OLAP workload is supported up to 6 T'B of memory (Class L) or 12 TB (Class M - customers must size accordingly) ‘The standard memory limitations of 4 and up to 8 socket-wide VMs can be mitigated via workload-based sizing Details see SAP note 2779240. ‘YM size -full socket + Minimum 1 socket / maximum 8 sockets + 1.25.35, 45,57 6, 7-,and 8-socket VMs on up to 8-socket server (fully & partly QDI meshed) VM size ~ “half socket” (NUMA Node sharing VM) + NUMA Node sharing VMs on 2-, 4-, 6-,and &-socket server + Noodd multiples of 0.5 (half) sockets like 1.5 socket VMs, ‘ocket VMs ete. SAP HANA production virtual machines as part of an SAP HANA Seale-Out Cluster CPU architecture (on 8 socket hosts): + Intel Cascade Lake processors of TDIvs Model 8 socket, + Intel Cooper Lake processors of TDIvs Model 8 socket, YM size — full socket + Node size 6 TB (Cascade Lake and Cooper Lake) © S socket VM on 8 socket hosts (fully and partly UPI meshed) © Upto master plus 7 worker nodes are possible. HA nodes should be added according to the custersize. EAQ: SAP HANA High Availabilty [2] © Upto 6 TB of RAM per Scale-Out node (VM) « Sizing up to class M workloads Yor sealeout wth ¢-socket Vs, please refer to SAP note 2537606 Configurations and Sizing of SAP HANA VMs For performance critical workload, a passthrough network device configuration is recommended. All 448 vCPUs could be leveraged, and the overall virtualization performance impact is below 10% (CPU) when compared to physical deployed SAP HANA systems on the same hardware configuration (CPU and memory). For sizing purposes customers are encouraged to use 2.10% CPU sizing buffer for virtual SAP HANA systems, Please also refer to the information provided atthe top regarding the virtualized network overhead on OTLP transactions and the linked VMware KB article, especially the missing support for Motion and the requirement for a passthrough network device per network performance critical VM needs to get considered The VMXNETS setup, which provides additional support for features like VMware vMotion and flexible Multi VM, requires special configurations and has a larger impact on the performance. ‘VMware recommends to reserve vCPUs for OLTP workload (e. g. only 416 vCPUs could be used on a 448 hyperthreaded bare ‘metal system) and to optimize the VM NIC configuration as documented in this VMware >log: https://blogs.vmware,com/apps/2023/10/19tb-sap-hana-vms html (the release of the updated best practice documentation is still pending). OLAP scenarios with 8 socket VMs and up to 6 TB and 416 vCPUs are supported with a Class 1 sizing. A performance impact for OLAP queries was observed with 14%. For larger amounts of memory (up to 12 TB are supported with a Class M sizing), the validation has also shown a larger impact during the data loading phase of BW scenarios (BWH benchmark phase 1). A virtualization overhead of 22% was observed. This overhead can be different in concrete customer scenarios, With 448 vCPUs the performance impact during the query phase (BWH benchmark phase 2) was within a 10% sizing buffer). For pure OLAP scenarios, a CPU reduction is not required in general. However, in case of mixed OLTP/OLAP or pure OLTP workload, the configuration described above is recommended. Each SAP HANA instance / virtual machine is sized according to the existing SAP HANA sizing guidelines and VMware recommendations, ‘SAPs general sizing recommendation is to Scale-Up first. Rules for virtualized SAP HANA on VMware vSphere 7.0 ‘+ CPU and Memory over-commitment must not be used, + SAP HANA VMs can get co-deployed with SAP non-production HANA or any other workload VMs on the same vSphere i host, as long as the production SAP HANA VMs are not negatively impacted by the co-deployed VMs. In case of negative impact on SAP HANA, SAP may ask to remove any other workload. No NUMA node sharing between SAP HANA and non-HANA allowed. This limitation is also valid for the VMware Clustering Services vCLS, see [8]_vSphere ‘Glustering Service (vCLS). + Remarks about NUMA Node sharing SAP HANA VMs: © Incase two VMs share a NUMA Node, both VMs need to fulfill the minimal resource requirements for an SAP HANA system: 128 GB RAM and 8 pCores per VM.¢ An additional performance impact for NUMA-node sharing VMs should be considered and a sizing buffer of at least 15% (CPU) should get used. + The Time Stamp Counter (TSC) must be synchronized between all sockets/cores. + Certified OS versions (from SUSE and Redhat) and server combinations as listed on the SAP HANA hardware directory are supported when virtualized. + VMware vMotion, VMware Distributed Resource Scheduled (DRS), as well as VMware HA capabilities ean be used to achieve operational performance and availability + SAP HANA Tailored Datacenter Integration (TDD) [3] delivery methods are supported for SAP HANA on VMware vSphere. The hardware must be listed on the SAP HANA hardware directory. The SAP HANA installation must done by an SAP certified engineer, qualified as "SAP Certified Technology Specialist - Required exam: C_HANATEC_a1 (*)- See [4] SAP Training and Certification Shop The SAP HANA Installation on SAP HANA certified hardware must be successfully verified with the [5] SAP HANA Hardware and Cloud Measurement Tools. + Configuration and overall setup must comply with the latest version of SAP HANA on VMware vSphere (6] and the ‘most recent best practice guide [9] SAP HANA on VMware vSphere Best Practices and Reference Architecture Guide. + Ina classical setup like described in this note, the SAP HANA core-to-memory ratio as reflected in the HANA Hardware Directory must be considered. This also applies to virtualized SAP HANA guests. Starting with vSphere 6.7 U2, there is also an alternative sizing possibility based on the SAP HANA workload. More information in SAP note 279240. + SAP and VMware will jointly support virtual SAP HANA in production adhering to the SLAs defined in the customer support contract. Ifa reported problem is a known SAP HANA issue with a validated fix, SAP support will recommend the appropriate fix directly to the customer. For all other performance related issues, the customer will be referred within SAP's OSS system to VMware for support. VMware will take ownership and work with SAP HANA HW/OS. partner, SAP and the customer to identify the root cause. SAP support may request that additional details be gathered by the customer or the SAP HANA HW partner to help with troubleshooting issues. In rare cases - SAP may require ‘VMware to reproduce the issue on SAP HANA running in a bare metal environment. + Toallow for an easier distinction to be made between scenarios which ean be considered a good fit against those scenarios which seem to be less suitable for virtualization, more scenario specific examples are being worked out and described in the VMware Best Practices Guide, + For VMware vSAN support information see SAP note 2718982 + For VMware PMEM support information see SAP note 2913410 + ‘TSX is disabled by default. Further information can be requested via support ticket on component BC-OP-LNX-ESX. Additional documentation and knowledge [1] Certified and Supported SAP HANA Hardware directory [2] EAQ: SAP HANA High Availability (3] SAP HANA Tailored Data Conter Integration - Frequently Asked Questions [4] SAP Training and Certification Shy [5] SAP HANA Hardware and Cloud Measurement Tools, [6] SAP HANA on VMware vSphere wiki [7] SAP HANA on VMware vSphere best practices guide white paper [8] ySphere Clustering Service (vCLS) [9] SAP HANA on VMware vSphere Best Practices and Reference Architecture Guide | Attributes Key Value Other Components HAN-DB (SAP HANA Database)