Professional Documents
Culture Documents
FAQ SAP HANA Performance Optimization
FAQ SAP HANA Performance Optimization
Symptom
The SAP HANA performance is not optimal.
Environment
SAP HANA
Cause
1. Where do I find information about SAP HANA performance optimization provided by SAP?
2. Which prerequisites should be fulfilled in order to be able to perform a detailed performance analysis?
3. How can I check if a performance issue is related to SAP HANA?
4. How can the main contributors to the SAP HANA response time be identified?
5. Which types of SAP HANA performance views exist?
6. Which general checks can be performed in order to optimize the performance of the SAP HANA database?
7. How can the performance of backup, restore and recovery be analyzed?
8. How can long SAP HANA startup times be analyzed?
9. Which SAP services are available for analyzing and optimizing the performance of SAP HANA?
10. What should be considered if SAP HANA hangs completely?
11. How can the performance of commit operations be optimized?
12. How can the performance of rollbacks be optimized?
13. How can the performance of DDL operations be optimized?
14. How can a representative workload be replayed in a different environment?
15. How can automatic actions be triggered in case of bad performance?
16. What are reasons for a non-responsive SAP HANA database without obvious resource or locking issue?
17. Which factors have an impact on the performance of shutdowns?
18. What do I have to consider in terms of database trigger performance?
19. How can I optimize the performance of migrations based on SUM / SWPM / R3load?
20. How can performance issues establishing connections be monitored?
Resolution
1. Where do I find information about SAP HANA performance optimization provided by SAP?
The SAP HANA Troubleshooting and Performance Analysis Guide contains comprehensive information how to optimize
performance of SAP HANA databases.
The SAP HANA Performance Guide for Developers contains performance related information that is particularly useful for
developers.
2. Which prerequisites should be fulfilled in order to be able to perform a detailed performance analysis?
The performance analysis capabilities significantly improve with newer SAP HANA patch levels (e.g. more comprehensive
history information, utilization of STATEMENT_HASH for SQL statements), so it is of advantage to have a recent SAP
HANA patch level in place. See SAP Note 2115815 for more information related to SAP HANA patches and upgrades.
Make sure that the SAP HANA parameters are set based on the SAP recommendations (SAP Note 2186744)
because in this way you implicitly make sure that important monitoring features are enabled, e.g. the expensive statement
trace (SAP Note 2180165) and the thread sample details (SAP Note 2114710).
As of SAP HANA Revision 74 it is recommended to switch to the embedded statistics server (SAP Note 2147247). Among
others this change allows to store important thread sample information for longer times in
HOST_SERVICE_THREAD_SAMPLES.
In order to make sure that sufficient historic information is kept and also comparisons with activities of the previous month
are possible (e.g. comparison of current month's end closing activities with previous month's end closing activities) the
statistics server history retention should be set to 42 days as recommended in SAP Note 2147247. Be aware that this
change increases the memory requirements of the statistics server.
4. How can the main contributors to the SAP HANA response time be identified?
In order to focus on the most important performance factors it is required to understand where most of the time is lost. On
SAP HANA server side this time distribution can be determined based on the THREAD_STATE (and in case of locks
LOCK_WAIT_NAME) information in views M_SERVICE_THREADS and M_SERVICE_THREAD_SAMPLES. See SAP Note
2114710 for more information.
M_* Views Memory Monitoring views containing information since last startup
M_*_RESET Views Memory Resettable counterparts of monitoring views containing information since last res
et
HOST_* and GLOBAL_* View Memory + Dis History views populated by statistics server (see SAP Note 2147247)
s k
For more details see the SAP HANA SQL and System Views Reference.
6. Which general checks can be performed in order to optimize the performance of the SAP HANA database?
The following areas should be checked in order to make sure that the SAP HANA database runs as efficiently as possible:
Area Details
SAP HANA pat Current SAP HANA patch levels usually include additional performance optimizations and resolve performance bu
ch level gs. Therefore you should make sure that you are on a reasonably new revision level. See SAP Note 2115815 for more
details.
SAP HANA and See SAP Notes 2186744 and 2600030 and make sure that the SAP HANA and SAP ABAP parameters are configure
SAP ABAP para d optimally.
meter settings
Statement hints Make sure that the statement hints delivered via SAP Note 2700051 are properly implemented so that known scena
rios of wrong optimizer decisions are eliminated.
Configuration o See SAP Note 2000003 ("How can the configuration and performance of the SAP HANA hardware, firmware and o
f operating syst perating system be checked?") and make sure that all recommendations related to operating system and hardware
em and hardwa are implemented. Pay particular attention for SAP Note 2131662 and make sure that transparent hugepages are dis
re abled on Linux level.
With SLES >= 12.2 you also have to make sure that the entry [Login] -> UserTasksMax=infinity is defined in /etc/s
ystemd/logind.conf.d/sap.conf as suggested in SAP Note 2205917. Otherwise the operating system may restrict the
number of threads that can be opened and this can result in various issues:
Terminations of thread starts with "Thread not started: <thread_type> not started" or "<thread_type> threa
d creation error: 11 (Resource temporarily unavailable)" errors
Threads can be stuck in a long time waiting for JobWorker threads (e.g. "JobBarrier Wait for Jobs").
Network Make sure that the network bandwidth and latency fulfills your expectations, both between SAP HANA scale-out n
odes and between SAP HANA and the client application servers. See SAP Note 1100926 for more information.
Make sure that SAP HANA statement routing is configured optimally, so that the amount of network communicatio
ns is minimized (see SAP Note 2200772).
I/O analysis Slow I/O can impact startup times, savepoints, commits or reloads. See SAP Note 1999930 for more information.
SQL statement Check for expensive SQL statements and optimize them from SAP HANA or application perspective. See SAP Note
optimization 2000002 ("FAQ: SAP HANA SQL Optimization") for more information.
Lock analysis Check for lock waits and optimize them from SAP HANA or application perspective. See SAP Note 1999998 ("FAQ:
SAP HANA Lock Analysis") for more information.
Activated traces Activated traces can have an impact on performance, so you should only activate as few traces as possible. The activ
ation and modification state of some (but not of all) traces can be checked using SQL: "HANA_Traces_ActivatedA
ndModifiedTraces" (SAP Note 1969700).
Writing database or user trace information (SAP Note 2403162) can result in many threads being blocked by trace f
lushes. In this case the call stack contains modules like:
TrexTrace::endl
TrexTrace::TraceStream::flush
stream& ltt::impl::ostreamFlush
Diagnose::impl::TraceBuffer::sync
Diagnose::TraceStream::flushTraceBuffer
Diagnose::TraceOutputFileHandlerImpl::doOutput
Diagnose::TraceSegment::reserveEntry
See SAP Note 2380176 and make sure that the database trace (and related traces like the user-specific trace) are on
ly activated temporarily and as focused as possible.
Connection han SAP Note 2385992 describes situations and optimizations when initial connections to the database require signific
dling ant time.
Garbage collecti See SAP Note 2169283 and make sure that garbage collection works correctly and isn't permanently blocked.
on
Application opt Make sure that the application efficiently utilizes the SAP HANA database. For S/4HANA environments you can ch
imization eck SAP Note 2689405 for a collection of performance best practices.
NUMA See SAP Note 2470289 in order to make sure that NUMA is properly configured and doesn't cause issues.
Several of these checks can be executed by running the SAP HANA Mini Checks (SQL: "HANA_Configuration_MiniChecks",
SAP Note 1969700). See SAP Note 1999993 for more information.
7. How can the performance of backup, restore and recovery be analyzed?
SAP Note 1835075 describes information that needs to be collected in case of performance issues in the area of backup,
restore and recovery. In many cases the performance is disk I/O bound (SAP Note 1999930). SAP Note 2930882 describes a
problem that can result in increased backup runtimes with SAP HANA 2.00.040 - 2.00.047 due to small I/O requests.
If recovery takes long, you can check for progress information in the database trace files (SAP Note 2380176). Typically you
can find out what kind of operation currently takes place (e.g. recovery, rollback or garbage collection).
Additionally you can generate context lists with hdbcons (SAP Note 2222218) using 'context l -s' in order to understand what
the threads are actually doing. Depending on the call stacks (typically of the LogRecoveryQueue<id> threads) you can draw
the following conclusions:
This call stack indicates that a delta storage container is loaded. This can be a consequence of frequent delta
AttributeEngine:
:BTreeNodeGroupN merges during recovery with SAP HANA 1.0 because the delta storage of concat attributes / multi column ind
ode::getEntryInd exes isn't persisted (SAP Note 2160391 -> "Are indexes persisted to disk?") while at the same time the log repl
ex ay maintains persistence pages. Therefore the delta storage of the concat attribute has to be recreated repeate
AttributeEngine: dly.
:BtreeAttribute:
If you repeatedly suffer from this scenario for some tables, you can consider to adjust the auto merge decision
:Delta::DeltaBTr
ee::findLeafNode
function or the smart merge logic in order to minimize delta merges during normal production operation (SA
AttributeEngine: P Note 2057046).
:BtreeAttribute: Starting with SAP HANA 2.0 SPS 01 the log replay updates the in-memory representation and so the overhea
:Delta::DeltaBTr
d no longer exists.
ee::insert_load
AttributeEngine:
:BtreeAttribute:
:Delta::DeltaBTr
ee::load
AttributeEngine:
:BtreeAttribute:
:DeltaContainer:
:load
AttributeEngine:
:BTreeAttribute:
:load
AttributeEngine:
:BTreeAttribute:
:open
AttributeEngine:
:openAvc
AttributeEngine:
:AttributeStore:
:loadAttribute
AttributeEngine:
:AttributeApi::l
oadAttribute
AttributeEngine:
:AttributeApi::l
oadAttributes
__pread_nocancel
This call stack indicates that a rollback takes place. See "How can the performance of rollbacks be optimized?"
System::UX::prea for details.
d
FileAccess::file
Read
FileAccess::Loca
lFile::read
PageAccess::Data
VolumeImpl::gene
ricPageIO
PageAccess::Data
VolumeImpl::read
Page
PageAccess::Page
IOImpl::readPage
PageAccess::Logi
calPageAccessImp
l::loadPageInter
nal
PageAccess::Logi
calPageAccess::l
oadPage
DataContainer::P
ageChainContaine
rImpl::deletePag
eImpl
DataContainer::V
irtualFileImpl::
deletePageIntern
alDoWork
DataContainer::V
irtualFileImpl::
deletePageIntern
al
DataContainer::V
irtualFileImpl::
deletePageInUndo
DataContainer::V
irtualFileUndo::
undoDoWork
DataContainer::V
irtualFileUndo::
undo
DataAccess::Undo
FileAnchor::proc
ess
DataAccess::Pers
istenceSession::
txRollbackCallba
ck
TransactionManag
er::RollbackLogR
ecord::redo
This step at the end of a restore makes sure that all restored pages are flushed to disk. A SubmitThread-DATA
syscall
Synchronization: -0 is triggered that performs the actual I/O writes in FileAccess::ioSubmit.
:Semaphore::time From the trace files it can appear like there is no real progress for some time. The last lines of the backup log i
dWait ndicate that the recovery progress is already 100 %:
ResourceManager:
:ResourceFlushCo 2019-10-06T17:05:29+05:30 INFO RECOVERY progress of service: indexserver, hanahost:32103,
unterAndState::w volume: 3, 95% 448824082432/468017217536
aitFor 2019-10-06T17:07:04+05:30 INFO RECOVERY progress of service: indexserver, hanahost:32103,
ResourceManager: volume: 3, 100% 468017217536/468017217536
:WaitAndSwitchCo
unterResource::w
In the indexserver trace the following messages are printed on a regular basis:
aitForFlush
DataAccess::Pers
i Service_Startup GlobalTransCommImpl.cpp(06358) : Processing Slave_GetMasterLogPos for volu
istenceManagerIm
me 2
pl::RecoveryHand
i Service_Startup GlobalTransCommImpl.cpp(06366) : Finished processing Slave_GetMasterLogPos
lerImpl::finish
, lsn=0xffffffffffffffff
DataAccess::Data
RecoveryHandler:
This scenario can remain for a longer time in case of bottlenecks in the I/O area. You can check the SAP HAN
:finish
A I/O behavior based on SAP Note 1999930. In context of the restore you can use the following hdbcons com
Backup::BackupSr
mand to extract I/O figures from the database:
c_CommonSnapshot
::executeRestore
hdbcons 'statreg print -h -n M_VOLUME_IO_TOTAL_STATISTICS'
Backup::BackupEx
e_JobRestore::ru
nBackupJob Before being written down to disk, restored pages are stored in the following heap allocator (SAP Note 19999
97, name is version dependent):
Pool/PersistenceManager/PersistentSpace/DefaultLPA/Page
Pool/PersistenceManager/PersistentSpace/DefaultLPA/DataPage
9. Which SAP services are available for analyzing and optimizing the performance of SAP HANA?
The most important service for analyzing and optimizing the performance of SAP HANA is:
SAP Technical Performance Optimization - HANA (TPO)
As part of this service SAP collects current architecture, runtime and performance data, identifies issues and risks related to
SAP HANA and analyzes the longest running SQL statements. It can also include an empowering component, i.e. a workshop
with a joint SAP HANA analysis and detailed technical discussions.
Several other SAP services can also include SAP HANA performance analysis:
Business Process Performance Optimization Service (BPPO)
Volume Test Optimization Service (VTO)
Customer Program Optimization Service (CPO)
SAP GoingLive Support Service (GLS)
SAP EarlyWatch Check
SAP HANA System Administration Service
Solution Manager Self Service "SQL Statement Tuning" (SAP Note 1601951)
For more information see the SAP Support Programs and Services Overview.
Reaso Details
n
Disk I/ During commits the change information is written to the log files. If writing down the data is slow, the commit times can in
O write crease. See SAP Note 1999930 and make sure that writing to the logs is performed with an acceptable performance.
perform
ance to l
ogs
Synchro If a synchronous system replication mode is activated, changes are propagated to the replication side during a commit, and
nous sys the local session has to wait for an acknowledgement. If you suffer from long commit times in a system with activated sync
tem repl hronous system replication, you should at first check if a switch to a less critical system replication mode (e.g. SYNC -> SY
ication NCMEM or SYNCMEM -> ASYNC) improve the performance. If yes, you can analyze if there are unnecessary replication d
mode elays, e.g. due to limited network bandwidth or high latency times. See SAP Note 1999880 for more information related to
SAP HANA system replication.
Synchro If a transaction performs changes on a table with activated optimistic synchronous table replication (OSTR), the replicas n
nous ta eed to be refreshed during commit. This can have an adverse impact on the commit performance.
ble repli
See SAP Note 2340450 for more information related to SAP HANA table replication.
cation
Wait sta You can check the commit related thread states (e.g. using SQL: "HANA_Threads_ThreadSamples_FilterAndAggregatio
tes n", THREAD_METHOD = 'CommitTrans', available via SAP Note 1969700) and analyze states different from "Running" w
ith a significant contribution of thread samples more in detail. See SAP Note 2114710 for more details. For example, signifi
cant "Network Poll" waits can indicate weaknesses in the infrastructure or configuration area (SAP Note 2222200).
High a An unnecessary high amount of commit requests can result in overhead and performance regressions. You can use the foll
mount o owing analysis commands available via SAP Note 1969700 to do a drill-down:
f commi
SQL: "HANA_Workload" (column COMMIT_PER_S): Commit frequency over time
t reques
SQL: "HANA_Connections_Statistics" (columns COMMIT, COM_PER_S): Commits per connection
ts
High a The commit performance can be significantly impacted by a high amount of active versions. See SAP Note 1694864 for mo
mount o re details.
f active
versions
Integrat If a SAP HANA integrated liveCache is used, the SAP HANA commit performance can be impacted by related commits on l
ed liveC iveCache side. You can use SQL: "HANA_liveCache_Procedures" (OBJECT_NAME = 'TRANS-END') available via SAP No
ache te 1969700 to check for a commit time breakdown on liveCache side.
The time for the 'Kernel-Commit' method is part of the SAP HANA commit time. Other related times like 'Flush-Cache', 'C
ommit-Invalidate-Callback' and 'Validate-Callback' have to be considered on top of the SAP HANA commit time.
During a liveCache commit a potentially large amount of data has to be flushed, so in case of high commit times you should
always consider the amount of flushed data to judge if it is more an I/O issue or a data volume issue.
See SAP Note 2593571 for more details related to the integrated liveCache.
Log vol Encryption of log volumes imposes a certain overhead when writing down redo log data to disk and so commit times can in
ume enc crease. Usually the overhead is limited and acceptable. See SAP Note 2159014 for more information.
ryption
Encryption of log volumes is available starting with SAP HANA 2.0. You can use SQL: "HANA_Encryption_Overview" (S
AP Note 1969700) in order to check the encryption configuration.
Log bac When log backups are executed, I/O read requests are executed against the log file system that can content with the I/O wr
kups ite requests of commits. This contention can be reduced by lowering the log backup buffer size (default: 128 MB) e.g. to 1
MB (SAP Note 2618564):
Be aware that this reduction is only useful when commit time peaks correspond to times of log backups.
SAP HA The following SAP HANA bugs can be responsible for increased commit times:
NA bug
SAP Note Impacted Revision Details
s
2439289 122.06 Overhead in internal commit processing design
2.00.000
If commits take very long, a termination with error "snapshot timestamp synchronization failed" is possible (SAP Note
2399990).
Action Details
Generat See SAP Note 2313619 and generate call stacks of the SAP HANA threads in order to identify the activities of threads res
e call sta ponsible for the rollback.
cks
The threads are called LogRecoveryQueue<id> and a typical call stack would be:
54220[thr=120221]: LogRecoveryQueue53 at
1: 0x00007fef1f223113 in __pread_nocancel+0x27 (libpthread.so.0)
2: 0x00007fef228504e5 in System::UX::pread (libhdbbasis.so)
3: 0x00007fef22665bb5 in FileAccess::fileRead (libhdbbasis.so)
4: 0x00007fef22643a03 in FileAccess::LocalFile::read (libhdbbasis.so)
5: 0x00007fef2d25837e in PageAccess::DataVolumeImpl::genericPageIO (libhdbdataaccess.so)
6: 0x00007fef2d258fcd in PageAccess::DataVolumeImpl::readPage (libhdbdataaccess.so)
7: 0x00007fef2d29e0a9 in PageAccess::PageIOImpl::readPage (libhdbdataaccess.so)
8: 0x00007fef2d27dbab in PageAccess::LogicalPageAccessImpl::loadPageInternal (libhdbdataaccess.so)
9: 0x00007fef2d27ea01 in PageAccess::LogicalPageAccess::loadPage (libhdbdataaccess.so)
10: 0x00007fef2d0d1c85 in DataContainer::PageChainContainerImpl::deletePageImpl (libhdbdataaccess.so)
11: 0x00007fef2d0f1597 in DataContainer::VirtualFileImpl::deletePageInternalDoWork (libhdbdataaccess.s
o)
12: 0x00007fef2d0f1a47 in DataContainer::VirtualFileImpl::deletePageInternal (libhdbdataaccess.so)
13: 0x00007fef2d10507b in DataContainer::VirtualFileImpl::deletePageInUndo (libhdbdataaccess.so)
14: 0x00007fef2d156c46 in DataContainer::VirtualFileUndo::undoDoWork (libhdbdataaccess.so)
15: 0x00007fef2d15b897 in DataContainer::VirtualFileUndo::undo (libhdbdataaccess.so)
16: 0x00007fef2cfe5bc8 in DataAccess::UndoFileAnchor::process (libhdbdataaccess.so)
17: 0x00007fef2cf6c521 in DataAccess::PersistenceSession::txRollbackCallback (libhdbdataaccess.so)
18: 0x00007fef2d2bb0da in TransactionManager::RollbackLogRecord::redo (libhdbdataaccess.so)
19: 0x00007fef2d1e2685 in DataRecovery::ProxyPseudoLogRecord::redo (libhdbdataaccess.so)
20: 0x00007fef2d205c33 in DataRecovery::RecoveryQueue::run (libhdbdataaccess.so)
21: 0x00007fef22601f29 in Execution::Thread::staticMainImp (libhdbbasis.so)
22: 0x00007fef22602dfd in Execution::Thread::staticMain (libhdbbasis.so)
Modules like __pread_nocancel indicate read I/O activities (SAP Note 1999930).
Check m Once you have identified the main activities during rollback based on the call stacks, you can check SAP HANA monitori
onitorin ng views for details. During startup no SQL can be executed against the database, but instead you can use hdbcons for t
g views hat purpose. For example, you can evaluate M_VOLUME_IO_DETAILED_STATISTICS via hdbcons in the following w
ay in case the call stacks indicate I/O activities (SAP Note 2222218):
Action Command
You can then load the resulting output into Excel and determine the times spent for different I/O types and page sizes. I
n this way you can find out how many pages of which size have been read, how long the average read time was and how l
ong reading took in total.
Optimiz If you conclude that I/O is the main contributor to the rollback runtime, the following options exist to improve it:
e I/O
If the average I/O times determined in the last step are higher than expected: Check in collaboration with your OS
and hardware partners if there is a bottleneck or misconfiguration in the I/O stack that has a negative impact on t
he average I/O times (SAP Note 1999930).
If you face a high number of reasonably fast I/O requests, you can only consider a - temporary - move of the under
lying data file to a really fast storage (e.g. SSD) or a RAM disk. Prefetching and page cache on Linux level doesn't h
elp, because SAP HANA always performs direct I/O bypassing the operating system page cache.
Implem Starting with SAP HANA 1.00.122.11 and 2.00.020 the performance of rollback is optimized, so an upgrade to these min
ent SAP imum Revision levels may help to speed up rollbacks.
HANA fi
xes
Proactiv In order to avoid critical rollback times in the future, you should consider the following:
e steps
Avoid long running update transactions with a very high number of modifications from a business perspective, bec
ause a rollback in case of a termination or crash can take significant times.
Monitor the system - in particular before performing a planned SAP HANA shutdown - in order to detect a signific
ant rollback / undo generation, e.g. via check ID M0856 ("Max. undo size of current transaction (MB)" of the SAP
HANA Mini Checks (SAP Note 1999993) or via SQL: "HANA_Transactions_UndoAndRedoLog" (SAP Note 19697
00).
A rollback of row store table information typically requires more I/O activities than a rollback of column store tabl
es. Therefore you should check if you can move critical tables to the column store. See SAP Note 2222277 for more
information.
A high amount of individual 4 KB page reads can be linked to row store tables with hybrid LOBs (SAP Note 22206
27), because every LOB container is read from disk individually. Check if you can move tables with a high amount
of disk LOB activity from row store to column store (SAP Note 2222277). An optimization is available with SAP H
ANA 1.00.121.
DD SA Optimizations
Lt P
ype N
ot
e
Ind 21 Consider the following optimizations:
ex c 60
If possible, create the index before the underlying table is populated with data.
reat 39
If the underlying uniqueness check executed by the join engine takes quite long, multi-threaded execution may imp
ion 1
rove the performance. This can be enforced by setting the parameter indexserver.ini -> [joins] -> single_thread_exe
cution_for_partitioned_tables to 'false'.
In case of a column store table and a unique / primary index it is helpful to perform a delta merge (SAP Note 20570
46) and make sure that the delta storage remains empty while the index is created. This eliminates the need to com
bine main and delta dictionaries required for uniqueness check purposes.
Starting with SAP HANA 2.00.045 inverted value indexes are created bulk wise to reduce memory consumption. Th
is can negatively impact the performance. You can set the parameter indexserver.ini -> [global] -> create_index_op
timize_bulksize (default: 40000000 for SAP HANA <= 2.0 SPS 06, 0 respectively individual calculation based on
memory limits for SAP HANA >= 2.0 SPS 07) to a higher value in order to shift resource consumption from CPU to
memory. A value of 0 or higher than the table rows deactivates bulk processing completely and provides optimal per
formance at the cost of high memory consumption. Be aware that this setting only applies to column loadable index
es, so if page loadable / NSE (SAP Note 2799997) is active for the index, it can't be used.
For indexes on partitioned tables the parameter indexserver.ini -> [ddl] -> max_number_of_data_jobs can be used
to balance between performance, CPU and memory. Per default, this parameter is only limited by the general concu
rrency settings (SAP Note 2222250), resulting in potentially high CPU and memory consumption and good perform
ance. By reducing it you can reduce the number of concurrently processed partitions and as a consequence also the
peak memory utilization. For example, max_number_of_data_jobs = 2 makes sure that only two partitions are pro
cessed at the same time and so also the CREATE INDEX memory overhead is lower compared to a parallel processi
ng of more than two partitions. At the same time the overall CREATE INDEX runtime can increase.
Tab 20 See SAP Note 2044468 ("How can the performance of repartitioning activities be influenced?") for details.
le p 44
arti 46
tion 8
ing
16. What are reasons for a non-responsive SAP HANA database without obvious resource or locking issue?
In most cases a slow or hanging SAP HANA database is caused by lock scenarios (SAP Note 1999998) or high resource
consumption (e.g. CPU, SAP Note 2100040). Nevertheless there can also be scenarios where the database is slow or stuck and
where you can find neither obvious locks nor high resource consumption:
Wrong parame 2600 Wrong parameters (e.g. a very small max_concurrency value or a restricted affinity setting) can significantly
ter settings 030 restrict the resources that can be used by SAP HANA so you have to make sure that all parameters are config
ured based on SAP best practices.
CPU restrictio In some scenarios it is possible that on layers below SAP HANA the amount of assigned CPUs or the CPU aff
ns on lower lay inity is limited for SAP HANA. Make sure that no unintended limitations exist.
ers
Call stack gene 2313 If you use gstack to generate SAP HANA call stacks, it can block the whole SAP HANA instance while it is ex
ration via gsta 619 ecuted. Neither CPU consumption is increased nor there are any dedicated locks visible. All activities are sus
ck pended until gstack is finished.
17. Which factors have an impact on the performance of shutdowns?
Starting with SAP HANA 2.0 SPS 06 you can use SQL: "HANA_Startup_StepDuration" (DATA_SOURCE = 'HISTORY',
PHASE = 'STOP%') to list steps that were done during historic service shutdowns including runtimes. This can sometimes
help to identify reasons for increased shutdown durations.
The following overview lists typical scenarios that can result in long SAP HANA shutdown times. When a shutdown takes too
long, it may be terminated with a timeout. See SAP Note 2177064 ("What happens when a shutdown takes a long time?") for
more details.
Sce Details
nari
o
Roll In case transactions with many changes were still running at the time of the shutdown, they have to be rolled back. You can us
back e SQL: "HANA_Transactions_UndoAndRedoLog" (SAP Note 1969700) to check how much undo was already created by runn
ing transactions.
Me Releasing allocated memory back to the operating system requires some time. Typically it takes around 1 to 2 minutes to relea
mor se 1 TB of memory. So on systems with a particularly high amount of allocated memory a successful shutdown can be delayed
y rel significantly.
ease
You can use SQL: "HANA_Memory_Overview" (SAP Note 1969700) to check for the memory that is currently allocated and
used by SAP HANA. If the allocated memory is much higher than the used memory, you can consider a manual memory defra
gmentation using the "mm gc -f" option of hdbcons (SAP Note 2222218) before doing the actual shutdown. With this comman
d the allocated memory will already be defragmented and released while the system is still up and running and so the subsequ
ent memory release time during shutdown will be shorter.
See SAP Note 1999997 for more information related to SAP HANA memory.
If you try to start the same service again while the previous one is still releasing memory (being in "defunct" state), the startup
can fail with the following errors (SAP Note 2222200):
19. How can I optimize the performance of migrations based on SUM / SWPM / R3load?
The document Best Practice Guide - Classical Migration of SAP NetWeaver AS ABAP to SAP HANA provides a lot of useful
details including aspects relevant for performance. SAP Note 2000002 -> "How can the performance of INSERTs and data
loads be tuned?" can be helpful to optimize the import performance.
CONNECTION ESTABLISHMENT:
ESTABLISHED OPEN_TS=1667287667374733 (2022-11-01 08:27:47.374733)
ESTABLISHED_TS=1667287671737762 (2022-11-01 08:27:51.737762)
TOTAL_ELAPSED_TIME=4363029
- TOTAL_CLIENT_ROUND_TIME=1958
- TOTAL_SERVER_TIME=4361071
[0] INITIAL REQUEST TIME=4342083
- RECEIVE_TS=1667287667375027 (2022-11-01 08:27:47.375027)
- RECEIVE_TIME=4341754
- RECEIVE_CALL_TIME=4322898
- RECEIVE_CALL_COUNT=2
- SERVER_PROCESSING_TIME=306
- JOB_DISPATCH_WAIT_TIME=86
- SSL_ACTIVATE_TIME=18853
- SEND_TIME=23
- SENT_TS=1667287671716816 (2022-11-01 08:27:51.716816)
[1] AUTHENTICATE REQUEST TIME=737
- CLIENT_ROUND_TIME=1704
- RECEIVE_TS=1667287671718520 (2022-11-01 08:27:51.718520)
- RECEIVE_TIME=49
- RECEIVE_CALL_TIME=42
- RECEIVE_CALL_COUNT=1
- SERVER_PROCESSING_TIME=644
- JOB_DISPATCH_WAIT_TIME=74
- AUTHENTICATE_EVALUATE_TIME=413
- AUTHENTICATE_METHOD=SCRAMSHA256=0
- SEND_TIME=44
- SENT_TS=1667287671719257 (2022-11-01 08:27:51.719257)
[2] CONNECT REQUEST TIME=18251
- CLIENT_ROUND_TIME=254
- RECEIVE_TS=1667287671719511 (2022-11-01 08:27:51.719511)
- RECEIVE_TIME=15
- RECEIVE_CALL_TIME=13
- RECEIVE_CALL_COUNT=1
- SERVER_PROCESSING_TIME=18206
- JOB_DISPATCH_WAIT_TIME=50
- AUTHENTICATE_EVALUATE_TIME=49
- AUTHENTICATE_TIME/PRE=134
- AUTHENTICATE_TIME/POST/LDAP_AUTHORIZATION=68
- AUTHENTICATE_TIME/POST=12202
- AUTHENTICATE_TIME=12414
- SEND_TIME=30
- SENT_TS=1667287671737762 (2022-11-01 08:27:51.737762)
In this example the overall connection establishment time is around 4.36 seconds and most of the time is spent with the SAP
HANA database server (+ network transfer time). Thus, you can start to analyze the SAP HANA database for (temporary)
performance issues and bottlenecks that can contribute to increased server times.
Example: (time spent on client side)
CONNECTION ESTABLISHMENT:
ESTABLISHED OPEN_TS=1667295051307880 (2022-11-01 10:30:51.307880)
ESTABLISHED_TS=1667295053945292 (2022-11-01 10:30:53.945292)
TOTAL_ELAPSED_TIME=2637412
- TOTAL_CLIENT_ROUND_TIME=2574608
- TOTAL_SERVER_TIME=62804
[0] INITIAL REQUEST TIME=42182
- RECEIVE_TS=1667295051308140 (2022-11-01 10:30:51.308140)
...
In this case the client (SAP Note 2393013) and network connectivity (SAP Note 2222200) need to be inspected.
Example: (JobWorker bottleneck)
CONNECTION ESTABLISHMENT:
ABORTED OPEN_TS=1680790399893438 (2023-04-06 16:13:19.893438)
TOTAL_ELAPSED_TIME=88343191
- TOTAL_CLIENT_ROUND_TIME=78068
...
- SENT_TS=1680790400121801 (2023-04-06 16:13:20.121801)
[2] CONNECT REQUEST TIME=88059526 - ABORTED
- CLIENT_ROUND_TIME=55302
- RECEIVE_TS=1680790400177103 (2023-04-06 16:13:20.177103)
- RECEIVE_TIME=53
- RECEIVE_CALL_TIME=21
- RECEIVE_CALL_COUNT=1
- SERVER_PROCESSING_TIME=88059473
- JOB_DISPATCH_WAIT_TIME=87831032
- AUTHENTICATE_EVALUATE_TIME=62
- AUTHENTICATE_TIME/PRE=230
- AUTHENTICATE_TIME/POST/LDAP_AUTHORIZATION=92
- AUTHENTICATE_TIME/POST=181988
- AUTHENTICATE_TIME=182338
- SEND_TIME=0
- SENT_TS=0 (<none>)
High JOB_DISPATCH_WAIT_TIME values typically indicate JobWorker bottlenecks. Make sure that workload settings are
properly in place (SAP Note 2222250), the system isn't overloaded by database requests (SAP Note 2000002) and NUMA is
properly configured (SAP Note 2470289).
Keywords
SAP HANA performance stuck hang troubleshooting slow
Attributes
Key Value
Products
Products
2399979 How-To: Configuring automatic SAP HANA Data Collection with SAP HANASitter
2200772 FAQ: SAP HANA Statement Routing and Client Distribution Mode
2177604
2159435 How-To: Keeping SAP HANA Row Store in Memory when restarting
2057046
2930882 Potentially Increased Data Backup Or Full Data Shipment Runtime in SAP HANA 2 SPS04
2205917 SAP HANA DB: Recommended OS settings for SLES 12 / SLES for SAP Applications 12
Best Practice Guide - Classical Migration of SAP NetWeaver AS ABAP to SAP HANA
3051992 SAP HANA takes longer to start after a crash or ungraceful shutdown
3065607 Performance tips & tricks for SAP S/4HANA Migration Cockpit: Migrate Data Using Staging Tables
2346988 Long Runtime in transport step "Generation of Programs and Screens" <SourceSID>G9<number>.<targetSID>
2666775 DML Operations through SAP Replication Server (SRS) to HANA are sub-optimal
2908549 SAP HANA Database requests fail with error code 1038
2399979 How-To: Configuring automatic SAP HANA Data Collection with SAP HANASitter