Ey SAPNote 2937606 - SAP HANA on VMware vSphere 7.0 in production ‘Component: HAN-DB (SAP HANA Database), Version: 29, Released On: 10.10.2023 | Symptom ‘You want to install SAP HANA platform in a virtualized environment using VMware vSphere 7.0 virtual machines. | Other Terms ESXi, VMW, vSphere, virtualized SAP HANA, VMware | Reason and Prerequisites SAP declares support for VMware vSphere 7.0 (incl. Ut, U2, and >=U3e) for virtual deployments in production scenarios of SAP HANA on either certified appliances or through SAP HANA tailored data center integration (TDD verified hardware configurations. The SAP HANA start release defined as "SAP HANA 1.0 SPS xx" includes support for SAP HANA 2.0, SAP always recommends the use of HANA 2.0. Certified Hardware can be found in the Certified and Supported SAP HANA Hardware directory [1] In case of TDI, SAP HANA certified Linux versions (SAP note 2235581) are also supported inside a VMware vSphere VM. | Solution By issuing this SAP Note, SAP extends support for SAP HANA on VMware vSphere 7.0 (inel. Ut, U2, and U3), providing the following conditions are met: SAP HANA start release: + Intel Skylake: SAP HANA 1.0 SPS 12 Revision 122.19; SAP HANA 2.0 recommended + Intel Cascade Lake: SAP HANA 1.0 SPS 12 Revision 122.19; SAP HANA 2.0 recommended «+ Intel Cooper Lake: SAP HANA 1.0 SPS 12 Revision 122.19; SAP HANA 2.0 recommended + Intel Tee Lake: SAP HANA 1.0 SPS 12 Revision 122.19; SAP HANA 2.0 recommended Important information: Recent revalidations have shown a limitation with OLTP workload on 4 socket VMs, independent of the CPU type. For S/4HANA standard workload, OLTP transactional request times show an overhead of up to 100 ms when accessed via the VMware virtualized network adapter (VMXNET3). This overhead leads to a transactional throughput loss by not more than 10% as compared to the underlying bare metal environment. VMware has identified a missing network optimization and is working on a solution. Documentation and timeline is provided by VMware in this article. Further questions on work-arounds or tuning options need to be addressed to the VMware support team on ‘component BC-OP-LNX-ESX. ‘Warning for Intel Ice Lake CPUs: A VM spanning more than 64 vCPUs (more than 32 cores per socket with hyperthreading) does not show a correct number of CPUs which is caused by a limitation of the VMware hypervisor. As an example, a 2-socket lee Lake system with 40 cores per socket is represented as a 4-socket virtual system. This can be verified via command “Iscpu, which shows 4 sockets (wrong) and 2 NUMA nodes (correct). The validation tests did not show any functional problems or an additional performance impact besides the aforementioned VMXNETS issue. Since all tests rely on consistent information, SAP cannot guarantee that this behaviour will not have any negative impact in a customer landscape. Customers should be aware that issues caused by this CPU representation need to be handled by VMware support. This incorrect behavior is described in more detail in VMware kb articles 89881 and 89879. Single and multiple SAP HANA production virtual machines on a single physical server CPU architecture (Intel glueless architecture only): + Intel Skylake processors of TDlvs Model 2 & 4 & 8 sockets, + Intel Cascade Lake processors of TDlv5 Model 2 & 4 & 6 & 8 sockets + Intel Cooper Lake processors of TDIv5 Model 2 & 4 & 6 & 8 sockets, requires vSphere 7.0 U2/U3, + Intel Ice Lake processors of TDIvs Model 2 sockets, requires vSphere 7.0 U3 (>= U3e) Important information for CPU configurations not following the Intel glueless reference architecture: ‘VMware with SAP HANA is supported on 4- and 8-socket HPe Superdome Flex servers with glued architecture and Cascade Lake CPUs. In these configurations, the maximum supported RAM size of a VM is 6TB. A single VM must not span across the chassis. Details see htins:/ www hpe.com/psnow/doc/as0001355enw The minimum size of a virtual SAP HANA instance is a half socket (represented by at least 8 physical cores) and 128 GB of RAM.Skylake, Cascade Lake, Cooper Lake: The maximum size of a virtual SAP HANA instance is 224 vCPUs. The maximal memory is limited to 6 TB. For the maximal supported 4-socket VM size, OLAP workload is supported up to 3 ‘TB of memory (Class L) or 6 TB (Class M - customers must size accordingly). Ice Lake: The maximum size of a virtual SAP HANA instance is 160 vCPUs. The maximal memory is limited to 4'TB. For the maximal supported 2-socket VM size (4 socket, see warning regarding VMware limitation mentioned before), OLAP workload is supported up to 2 TB of memory (Class 1) or 4 TB (Class M - customers must size accordingly). Larger VM sizes are covered by SAP note 3102813. The memory limitations can be mitigated via workload-based sizing, Details see SAP note 2779240. VM size -full socket ‘+ Minimum 1 socket / maximum 4 sockets (2 sockets for Ice Lake) + 1-,2-,g- and 4-socket VMs on up to 8-socket server (fully & partly UPI meshed). 1- and 2-sockets for Ice Lake. + 8.socket VMs: see SAP note 3102813. VM size ~ “half socket” (NUMA Node sharing VM) + NUMA Node sharing VMs on 2-, + Noodd multiples of 0.5 (half) sock and 8-socket server (2- ts like 1.5 socket VMs, socket server for Tee Lake) 5 socket VMs ete. Remark on Broadwell support: Tests with vSphere 7 on this CPU type have shown a higher performance impact of the virtualization compared to the general SAP expectations. However, customers who are already running vSphere 6.7 on Broadwell and do not observe any performance issues with their current workload can upgrade to vSphere 7.0 (>= Uge). The supported memory and vCPU configurations are described in SAP note 2393017, New installations of ‘Sphere 7 on Broadwell are not recommended. In this case, more recent CPU generations should be used if customers need to run vSphere 7.0 with SAP HANA. VMware provides more details in this blog. SAP HANA production virtual machines as part of an SAP HANA Seale-Out Cluster CPU architecture (on minimum 4 socket hosts): + Intel Skylake processors of TDIv5 Model 4 & 8 socket + Intel Cascade Lake processors of TDIvs Model 4 & 8 socket + Intel Cooper Lake processors of TDIv5 Model 4 & 8 socket, YM sire fll scket + Node size 2 TB © 4 socket VM on up to 8 socket hosts (fully and partly UPI meshed) © Upto master plus 15 worker nodes are possible. H1A nodes should be added according to the cluster-size. FAQ: ‘SAP HANA High Availability (2) © Upto2 TB of RAM per Seale-Out node (VM). © Sizing up to class L workloads + Node size 3 TB © 4 socket VM on up to & socket hosts (fully and partly UPI meshed) © Upto1master plus 7 worker nodes are possible. HA nodes should be added according to the cluster-size. FAQ: SAP HANA High Availability [2] © Uptog TB of RAM per Seal © Sizing up to class M workloads + Regarding node size 6 TB, refer to details in SAP note 3102813) Sizing of SAP HANA VMs + Overall, the virtualization impact on CPU is in most cases below 10% when compared to physical deployed SAP HANA systems on the same hardware configuration for CPU and memory. For sizing purposes customers are encouraged to use 210% CPU sizing buffer for virtual SAP ITANA systems. Please also refer to the information provided at the top regarding the virtualized network overhead on OTLP transactions and the linked VMware KB article. + Each SAP HANA instance / virtual machine is sized according to the existing SAP HANA sizing guidelines and VMware recommendations, + SAPs general sizing recommendation is to Scale-Up first. Rules for virtualized SAP HANA on VMware vSphere 7.0 + CPU and Memory over-commitment must not be used. + SAP HANA VMs can get co-deployed with SAP non-production HANA or any other workload VMs on the same vSphere ESXi host, as long as the production SAP HANA VMs are not negatively impacted by the co-deployed VMs. In case of negative impact on SAP HANA, SAP may ask to remove any other workload. No NUMA node sharing between SAP HANA and non-HANA allowed. This limitation is also valid for the VMware Clustering Services vCLS, see [8] Sphere (Clustering Service (vCLS). + Remarks about NUMA Node sharing SAP HANA VMs ©. Incase two VMs share a NUMA Node, both VMs need to fulfill the minimal resource requirements for an SAP HANA system: 128 GB RAM and 8 pCores per VM. © An additional performance impact for NUMA-node sharing VMs should be considered and a CPU sizing buffer of at least 15% should get used. ‘+ The Time Stamp Counter (TSC) must be synchronized between all socket Out node (VM).+ Certified OS versions (from SUSE and Redhat) and server combinations as listed on the SAP HANA hardware directory are supported when virtualized. + VMware vMotion, VMware Distributed Resource Scheduled (DRS), as well as VMware HA capabilities can be used to achieve operational performance and availability. + SAP HANA Tailored Datacenter Integration (TDI) [3] delivery methods are supported for SAP HANA on VMware vSphere. The hardware must be listed on the SAP HANA hardware directory. The SAP HANA installation must done by an SAP certified engineer, qualified as "SAP Certified Technology Specialist - Required exam: C_HANATEC_t1 (*)- See [4] SAP Taining and Certification Shop The SAP HANA Installation on SAP HANA certified hardware must be successfully verified with the + Configuration and overall setup must comply with the latest version of SAP HANA on VMware vSphere [6] and the most recent best practice guide [9] SAP HANA on VMware vSphere Best Practices and Reference Architecture Guide. + Ina classical setup like described in this note, the SAP HANA core-to-memory ratio as reflected in the HANA Hardware Directory must be considered. This also applies to virtualized SAP HANA guests, Starting with Sphere 6.7 Ua, there is also an alternative sizing possibility based on the SAP HANA workload, More information in SAP note 2779240. + SAP and VMware will jointly support virtual SAP HANA in production adhering to the SLAs defined in the customer support contract. Ifa reported problem is a known SAP HANA issue with a validated fix, SAP support will recommend the appropriate fix directly to the customer. For all other performance related issues, the customer will be referred within SAP's OSS system to VMware for support. VMware will take ownership and work with SAP HANA HW/OS: partner, SAP and the customer to identify the root cause. SAP support may request that additional details be gathered by the customer or the SAP HANA HW partner to help with troubleshooting issues. In rare cases ~ SAP may require ‘VMware to re-produce the issue on SAP HANA running in a bare metal environment. + Toallow for an easier distinction to be made between scenarios which can be considered a good fit against those scenarios which seem to be less suitable for virtualization, more scenario specific examples are being worked out and described in the VMware Best Practices Guide. + For VMware vSAN support information see SAP note 2718082 + For VMware PMEM support information see SAP note 2913410 + SX is disabled by default. Further information can be requested via support ticket on component BC-OP-LNX-ESX. Additional documentation and knowledge [1] Certified and Supported SAP HANA® Hardware directory [2] FAQ: SAP HANA High Availability [3] SAP HANA Tailored Data Center Integration - Frequently Asked Questions [4] SAP ‘Training and Certification Shop, (sl. (6) SAP HANA on VMware vSphere wiki (71SAP HANA on VMware vSphere best practices guide white paper 8] wSphere Clustering Service (vCLS) [9] SAP HANA on VMware vSphere Best Practices and Reference Architecture Guide P HANA Hardware and Cloud Measurement Tools | Attributes Key Value Other Components ‘BC-OP-LNX-ESX (Linux on VmWare ESX)