Sap Hana For Opentext Content Server 16.2.X: Best Practices

Product: OpenText Content Server
Version: 16.2.x
Task/Topic Deployment, Administration, Performance
Audience: Administrators
Platform: SAP Hana 2.0
Document ID: 500380
Updated: June 20, 2019
Best Practices
SAP Hana for OpenText Content Server
16.2.x
Michelle Schwan, Database Architect
OpenText Performance Engineering Team

SAP Hana for OpenText Content Server 16.2.x
Contents
Audience ...................................................................................................................... 4
Introduction ................................................................................................................. 5
General Overview of Content Server Database Connections ................................ 6
SAP Hana Setup Best Practices ................................................................................ 7
SAP Hana Database Sizing.................................................................................... 7
CPU Sizing ....................................................................................................... 7
Disk Sizing ....................................................................................................... 7
RAM Size ......................................................................................................... 8
Memory Allocations ................................................................................................ 8
Hardware Configurations ........................................................................................ 9
Disks ................................................................................................................ 9
File system ....................................................................................................... 9
Linux Packages ................................................................................................ 9
Linux Kernel Parameters ................................................................................. 9
Storage Best Practices ........................................................................................... 9
SAP Hana Database Installation Configurations .................................................... 9
Dynamic Tiering ............................................................................................... 9
Scale-Out ....................................................................................................... 10
Installation Options ........................................................................................ 10
Data Volume Management ................................................................................... 10
SAP Hana Configuration Settings ........................................................................... 11
Parameter Settings ........................................................................................ 11
Changing Parameter Settings ........................................................................ 11
File Configurations ................................................................................................ 12
Global.ini ............................................................................................................... 12
Communication .............................................................................................. 12
TCP Backlog ............................................................................................ 12
Execution ....................................................................................................... 12
Job Workers For Database Requests ..................................................... 12
Max Concurrency .................................................................................... 12
Max Concurrency Dynamic Minimum Percentage .................................. 13
Memory Manager ........................................................................................... 13
Allocation Limit ........................................................................................ 13
Global Allocation Limit ............................................................................. 14
Statement Memory Limit ......................................................................... 14
Persistence .................................................................................................... 15
Consistent Change Lock ......................................................................... 15
Enable Auto Log Backup ......................................................................... 15
Log Mode ................................................................................................. 16
Parallel Persistence Garbage Collection ................................................. 16
The Information Company™ 2

Indexserver.ini ...................................................................................................... 17
Session .......................................................................................................... 17
Idle Connection Timeout .......................................................................... 17
SQL ................................................................................................................ 17
Plan Cache Enabled ................................................................................ 17
Plan Cache Size ...................................................................................... 18
Statisticsserver.ini ................................................................................................. 18
Memory Manager ........................................................................................... 18
Allocation Limit ........................................................................................ 18
Content Server Installation Options ....................................................................... 19
Performance Tuning Tips ......................................................................................... 21
Statement Memory ............................................................................................... 21
Memory Manager ................................................................................................. 21
Shared Memory .................................................................................................... 21
Monitoring and Benchmarking ................................................................................ 22
Large Memory Usage of Statistics Server ............................................................ 22
Monitoring ............................................................................................................. 22
Consistency Checks ............................................................................................. 22
Identifying Worst-performing Queries .................................................................... 23
Content Server Connect Logs .............................................................................. 23
Performance Analyzer ............................................................................. 23
Blocked Transactions ........................................................................................... 24
Workload Management ........................................................................................ 24
Administration Recommendations ......................................................................... 25
Backup and Recovery .......................................................................................... 25
Appendices ................................................................................................................ 26
Appendix A - References ...................................................................................... 26
Appendix B – SAP Hana Bugs of Interest ............................................................ 27
Appendix C – Troubleshooting Queries ............................................................... 28
Column Table Compression .................................................................... 28
Column Table Memory Usage ................................................................. 28
Heap Memory .......................................................................................... 28
Host Resource Utilization ........................................................................ 29
Long Running Cursors ............................................................................ 29
Row Table Memory Usage ...................................................................... 29
Service Memory....................................................................................... 30
Shared Memory ....................................................................................... 30
SQL Plan Cache ...................................................................................... 31
Statistics Queries..................................................................................... 31

Audience
The document is intended for a technical audience that is planning an
implementation of OpenText™ products. OpenText recommends consulting with
OpenText Professional Services who can assist with the specific details of individual
implementation architectures.
Disclaimer
The tests and results described in this document apply only to the OpenText
configuration described herein. For testing or certification of other configurations,
contact OpenText Corporation for more information.
All tests described in this document were run on equipment located in the OpenText
Performance Laboratory and were performed by the OpenText Performance
Engineering Group. Note that using a configuration similar to that described in this
document, or any other certified configuration, does not guarantee the results
documented herein. There may be parameters or variables that were not
contemplated during these performance tests that could affect results in other test
environments.
For any OpenText production deployment, OpenText recommends a rigorous
performance evaluation of the specific environment and applications to ensure that
there are no configuration or custom development bottlenecks present that hinder
overall performance.
All results in this paper are based on server–side measurements and do not capture
browser rendering of results. Actual timings including client side (e.g. browsers)
timings may vary significantly depending on the client machine specifications, the client
network, browser variations and other conditions of the user’s environment.
4
Introduction
This document covers best practices for OpenText™ Content Server when SAP Hana
is used as the metadata repository. Recommendations are for SAP Hana 2.0 latest
SPS version.
Most common performance issues can be solved by ensuring that the hardware used
to deploy SAP Hana Server has sufficient CPU, RAM and fast I/O devices, properly
balanced. SAP Hana is an in-memory column storage database and requires large
amounts of RAM and CPU.
Topics here explore non-default options available when simple expansion of resources
is ineffective and discuss some best practices for administration of Content Server’s
database. It concentrates on non-default options, because in general, as a
recommended starting point, Content Server on SAP Hana Server installations uses
default deployment options. Usage profiles vary widely, so any actions taken based
on topics discussed in this paper must be verified in your own environment prior to
production deployment, and a rollback plan must be available should adverse effects
be detected.
All recommendations should be tested and evaluated on a test or staging site and
should be evaluated with respect to your database and site characteristics. Testing
and impact analysis should occur before deployment on a production site and should
be done in adherence to your company’s change management procedures.
These recommendations are not intended to replace the services of an experienced
and trained SAP Hana database administrator (DBA), and do not cover standard
operational procedures for SAP Hana database maintenance, but rather offer advice
specific to Content Server on the SAP Hana platform.
DISCLAIMER: This Best Practices document contains procedures

and sample SQL code. All procedures and sample code in this
area is specific to the scenario presented in the document and
delivered as is and is for educational purposes only. It is presented
as a guide to supplement official OpenText product documentation.
Procedures on deployments using other operating systems and
database application platforms may differ in some areas.
Deployments with additional optional modules may also add to the
complexity.
5
General Overview of Content Server Database

Connections
Content Server 16.2.x creates three persistent database connections per thread.
These are threads that handle user requests to the application. All user requests are
pooled in this manner and use the database sessions established using the database
schema owner account. For example, perhaps your Content Server site supports
10,000 users. Each of these application users has their own account in most cases,
but at the database level all requests are coming in on perhaps a few dozen
connections, all owned by the schema owner. This is a typical web application
architecture.
In addition to the front-end threads, there is also a database connection for certain
background threads, such as the notifications/agents thread.
In Content Server 10.5 and above, there is a framework called "Distributed Agents".
This is a framework of threads used to parallelize large batch operations in the
background, such as the calculation of facet counts for the new faceted browse
functionality. Distributed Agents will also create database connections.
6
SAP Hana Setup Best Practices

OpenText recommends that you install and configure SAP Hana with an SAP Database
professional using the following recommendations for best performance. This section
covers many SAP Hana settings and refers to SAP Hana documentation where
applicable.
SAP Hana can only be installed on Linux platforms. For more information about what
platforms are supported by SAP Hana, see SAP Product Availability Matrix.
SAP Hana Database Sizing

SAP has a tool called SAP Quick Sizer tool to calculate CPU, disk, memory and I/O
resources. This tool now Hana specific sizing. To use the SAP Quick Sizer tool, you
can follow the instructions in SAP Hana Central - Quick Sizer.
CPU Sizing
A CPU sizing only needs to be performed, in addition to the memory sizing, if an extra-
large number of users are working on a relatively small amount of data.
CPU sizing is user-based. SAP Hana supports 300 SAP Application Performance
Standard (SAPS) for each concurrently active user.
The CPU sizing should be adjusted so that the server load does not exceed 65% on
average.
The minimum recommended for SAP Hana is 8 CPUs.
Calculate SAPS: fully processed order line items per hours * 5%
Maximum number of users per server: SAPS * 65%.
CPU total: CPU Total = SAPS / active users
Disk Sizing
Disk sizing consists of two categories: the Persistence Layer (Data Volume); and the
Disk Log (Log Volume).
By default, SAP Hana writes the data from memory onto the Persistence Layer (Data
Volume) every five minutes.
The Disk Log (Log Volume) saves the log files to ensure that changes are durable.
This is the redo log.
Disk Total = 5 * RAM Total
For scale-out SAP Hana installations, the recommended disk size requirements for
/hana/shared/<SID> (default directory) depends on the number of worker nodes.
Worker nodes, in a scale-out system use a disk space of one RAM per four worked
nodes.
Disk Total = (4 * RAM Total) + (nodes/4 rounded down * RAM Total)
7
SAP recommends using half of the required disk space for the SAP Hana application.
Note: SAP Hana will most likely take less space than the Data and Log Volumes.
RAM Size
RAM is divided into Static and Dynamic RAM. The static RAM is the amount of main
memory that is used for the column table data. Dynamic RAM is used for new data,
queries and other dynamic objects. The Dynamic and Static RAM should be the same
size.
The minimum amount RAM recommended by SAP Hana is 128GB, however, SAP
recommends that you use the calculation or the SAP Quick Sizer tool to get the correct
amount (see SAP Hana Database Sizing).
The column table data will be compressed automatically. The compression rate is set
at seven. SAP recommends taking half of the size of the disk-based database plus a
20% buffer plus 50GB for other services.
RAM Total = (Disk Size (from above) / 2 * 1.20) + 50 GB
SAP Hana licensing is done by the RAM or is unlimited. Please keep this in mind when
determining the RAM size.
Memory Allocations
Due to the special manner of SAP Hana’s memory management, the usage of
transparent hugepages in Linux may lead to hanging situations and performance
degradations including high CPU usage.
The steps below are for SuSE Linux SLES.
1. Check to see if Transparent Huge Pages are activated. If always is in square
brackets and the output of Kilobytes is greater than 0, then some processes use
THPs.
cat /sys/kernel/mm/transparent_hugepage/enabled
cat /proc/meminfo | grep AnonHugePages
2. Deactivate THP
echo never > /sys/kernel/mm/transparent_hugepage/enabled
3. Prevent THP from activating on reboot with the parameter:
transparent_hugepage=never
a. Add the parameter to the kernel boot line in /etc/default/grub
GRUB_CMDLINE_LINUX="quiet transparent_hugepage=never"
b. Rebuild the /boot/grub2/grub.cfg file
grub2-mkconfig -o /boot/grub2/grub.cfg –BIOS-based
grub2-mkconfig -o /boot/efi/EFI/redhat/grub.cfg –UEFI-
based
c. And/or integrate the parameter within your system boot scripts
8
Hardware Configurations
Please make sure you follow all the requirements for the hardware for the SAP Hana
server. For more information, see SAP Hana Platform.
Disks
Make use of SSD drives. They have a performance advantage of five to seven times
faster than spinning drives.
File system
SAP recommends using XFS (64-bit journaling file system) for disks holding the data
files and Hana database installation itself.
Linux Packages
SAP recommends updating all the packages installed on the Linux server. Bad
packages or older versions of packages can cause problems with installation and/or
running SAP Hana in both fresh installs and upgrades on major versions.
Linux Kernel Parameters

SAP recommends changing some of the default settings in Linux. For more details,
see Optimizing the Network Configuration on Hana and OS.
Storage Best Practices

This section covers a few specific topics related to storage, but it is not meant to be a
comprehensive guide for storage planning.
OpenText recommends that all the directories and files are on the same disk.
Avoid putting the Data and Log directories on the operating system drive/mount point.
If the Log and Data files for the database are on the same directory as the SAP Hana
installation and Log/Data files use the maximum amount of disk space, SAP Hana will
stop working.
Refer to Recommended File System Layout to understand the storage format at the
levels of files and directories.
SAP Hana Database Installation Configurations
Dynamic Tiering
SAP Hana can be set up to use Dynamic Tiering. This should be used in the case
where there are billions of rows that are accessed infrequently. These tables of data
do not need to be kept in memory.
9
Scale-Out
SAP Hana can be installed as a scale-out system. However, SAP recommends that
you avoid scaling-out and scale the machine up. If you require a scale-out system,
SAP recommends that your slave (worker) and/or failover (stand by worker) databases
are configured the same as the master database.
Content Server will function on multi-container and scale-out databases.
Installation Options
When installing SAP Hana, in the system properties, make sure that the Restrict
Maximum Memory Allocation is set to NO (or unchecked in the GUI install). This will
allow SAP Hana to use the maximum memory of your machine and license limit. This
can also be changed after installation, see Memory Manager.
SAP recommends that the SYSTEM user be deactivated after the installation of SAP
Hana. See Content Server Installation Options on how to create a user which can be
used to install Content Server.
For more details on installation, see SAP Hana Server Installation and Update Guide.
Data Volume Management

SAP Data Volume Management can help control the growth rate and existing sizes of
the SAP Hana database.
SAP has a variety of tools and processes to manage the SAP Hana database.
For more information, see Data Volume Management.
10
SAP Hana Configuration Settings

Configuration can be done in the Administration Console for each Container. Access
to the Administration Console is done with the SYSTEM id or having IniFile Admin and
Monitoring Granted Roles. These steps should only be done by a SAP Hana Database
Administrator.
Parameters configured at the system level apply to the complete system and all
databases (tenant or container depending on where the change is made). Parameters
configured at the database level apply only to the specified database.
Access to the INI files should be done via Hana Cockpit. SAP recommends that Hana
Cockpit is installed on a server separate from SAP Hana.
The configuration files are in /usr/sap/<sid>/SYS/global/hdb/custom/config. The
Container files are under this directory. Tenant configuration files are listed as folders
by the tenant name under this directory.
For more details on the Administration console, see SAP Hana Administration Guide.
Parameter Settings
To determine if there are settings that your tenant and/or container database require,
do the following steps.
1. Get HANA_Configuration_Parameters_~.txt from the
SQLStatements.zip file attached to the SAP Knowledge Base Note 1969700 -
SQL Statement Collection for SAP Hana.
2. Run this script in the container database. Once the suggested results are returned,
review the changes. Some of the suggestions may not be required (i.e. The script
recommends enabling the parameter expensive_statements. This activates a
trace which do you not want as default.).
3. Make the appropriate changes to the container (for more details, see Changing
Parameter Settings).
4. Repeat the steps 2 and 3 for each tenant
Changing Parameter Settings

Parameter changes can be done using SAP Hana Studio – Configuration and
Monitoring – Administration - Configuration or by command line:
ALTER SYSTEM ALTER CONFIGURATION ('<service>.ini', 'SYSTEM')
SET ('<section>', '<parameter_name>') = '<value>'
WITH RECONFIGURE;
Example:
ALTER SYSTEM ALTER CONFIGURATION ('global.ini', 'SYSTEM')
SET ('persistence', 'non_trans_cch_block_size')='16777216'
WITH RECONFIGURE;
To remove a configuration, you can use SAP Hana Studio or by command line:
ALTER SYSTEM ALTER CONFIGURATION ('<service>.ini', 'SYSTEM')
UNSET ('<section>', '<parameter_name>') WITH RECONFIGURE;
11
File Configurations
These settings are specific to the Content Server database.
Global.ini
Communication
TCP Backlog
Description The tcp_backlog is used for setting the size of connection
backlogs. If many requests are sent in a short timeframe, the backlog
could become full and cause a connection broken error.
Default 128
Recommendation Set this parameter to 2048. If you are still receiving connection
broken errors, increase the size to a maximum of 8192.
Change Level Container only
Notes This change requires a restart.
Execution
Job Workers For Database Requests
Description The default_statement_concurrency_limit parameter
controls the maximum overall parallelism for a single database
request.
Default 0 – No Limit
Recommendation Set this parameter to 30 to 50% of available CPU threads, but no

higher than max_concurrency.
Change Level Tenant only
Notes Parameters max_concurrency_hint should be smaller or equal to

the default_statement_concurrency_ limit.
Max Concurrency
Description The max_concurrency parameter is the number of CPU threads
overall that can be used by parallelized requests.
Default 0 – number of logical CPU cores of host
Recommendation Level as default
Notes You can use the parameter max_concurrency_dyn_min_pct for

concurrency reduction.
12
Max Concurrency Dynamic Minimum Percentage

Description The max_concurrency_dyn_min_pct parameter is the number of
CPU threads overall that can be used by parallelized requests.
Default Not set
Recommendation Level as default
Change Level Tenant
Notes Monitor the CPU and if CPU is high for the Tenant database or SAP
Hana, alter this parameter – starting at 50%.
Memory Manager
Allocation Limit
Description The allocation_limit limits the memory consumption of the
related SAP Hana process.
Default Not set
Recommendation For container, global_allocation_limit

If only one tenant, leave as is, else set a reasonable limit based on
usage. Be prepared to increase the limit
Change Level Container
Notes This amount should not exceed the amount declared in the
global_allocation_limit.
Setting the limit to a value above the limit of your license can cause
a database lockdown.
The total of all the tenant limits should equal the container limit, but it
is possible to exceed it.
If the tenant limit is not restricted, it can impact other tenants within
the container.
For more information, see Define Memory Allocation Limits.
Tenant changes must be done at the container level. Example: set
the tenant to 78GB of RAM
ALTER SYSTEM ALTER CONFIGURATION
('global.ini', 'DATABASE', 'CSARC')
SET ('memorymanager','allocationlimit') = '79872'
WITH RECONFIGURE;
13
Global Allocation Limit

Description The global_allocation_limit, or Restrict Maximum Memory
Allocation option when installing, is used to limit the amount of
memory that can be used by the database.
Default 0 - (90% * 64GB) + (97% * Total RAM – 64GB)
Recommendation Leave container setting as is
Change Level Container and/or Tenant
Notes For non-production systems, the default should be changed to restrict

the amount of RAM used.
Setting the limit to a value above the limit of your license can cause
a database lockdown.
Statement Memory Limit

Description The statement_memory_limit allows you to limit the memory
consumption of a single SQL Statement. The
statement_memory_limit_threshold makes sure that the
statement_memory_limit only takes effect if the overall SAP
Hana memory allocation exceeds a defined percentage of the
global_allocation_limit.
Default Not set
Recommendation 30% of the first 500GB of the global_allocation_limit, and

15% of the remaining memory.
Change Level Tenant
Notes This can be set by user and workload class as well.

You should test the effects of these settings carefully to avoid issues
like termination of queries or backups.
It is generally useful to start with a rather high memory limit.
14
Persistence
The Persistence section in the Global.ini file has the information pertaining to the data, log,
backup data, backup log, and table optimization writes.
Consistent Change Lock

Description The non_trans_cch_block_size parameter is the time an
internal activity, like table optimization, holds a Consistent Change
Lock.
Default 256 MB
Recommendation A reduction to a smaller value can make sure that the Consistent
Change Lock is released more frequently.
16777216 to 134217728
Change Level Tenant
Notes This change should only be done if recommended by the

HANA_Configuration_Parameters script.
Enable Auto Log Backup

Description The enable_auto_log_backup allows the creation of redo log
backups at regular intervals.
Default yes
Recommendation SAP recommends leaving the default value
Change Level Container or Tenant database
Notes During normal system operations (log_mode = normal), SAP strongly

recommends that enable_auto_log_backup remain at the
default of yes (activated). If automatic log backups are disabled, the
log area will grow until the file system is full.
For more details, see Enable or Disable Automatic Log Backup.
15
Log Mode
Description The log_mode parameter is how the log segments are treated when
backing up.
Default normal
Recommendation Leave as default but monitor the log file creation
Change Level Container or Tenant database
Notes The normal setting is recommended to provide support for point-in-

time recovery. The Normal option backs up the log segments
automatically enable_auto_log_backup with enabled.
When savepoints are written, the log segments are immediately freed
to be overwritten by new log entries when using the overwrite option.
To avoid disk space issues, the log_mode can be changed to
overwrite.
See Log Modes for more details.
Parallel Persistence Garbage Collection

Description The max_gc_parallelity parameter controls the parallelism of
the garbage collector clean up.
Default 0 – The total number of logical CPU cores of the host.
Recommendation ½ of the total CPUs
Change Level Container or Tenant
Notes By reducing the value in max_gc_parallelity you can reduce the

overhead, but a small value may result in garbage collection delays.
For this reason, the change should be done in the Content Server
tenant first.
To trigger an immediate garbage collection, run:
hdbcons 'mm gc -f'
A restart is required for this change to take effect.
For more details, see SAP Hana Garbage Collection.
16
Indexserver.ini
Session
Idle Connection Timeout
Description The idle connection timeout parameters are set for all of SAP Hana
and specific applications within SAP Hana. These parameters set
the length of idle connections before they are cancelled.
Default idle_connection_timeout = 1440

idle_connection_timeout_application_hdbstudio = -1
idle_connection_timeout_application_hdbwlreplayer
= 0
idle_connection_timeout_application_statisticsserv
er = 60
Recommendation To avoid a disconnection of the client idle connections and potentially

cancelling running statements, set the
idle_connection_timeout to 0 in the Content Server tenant
database.
Change Level Container or Tenant
Notes Change the timeout if you are running into timeout errors within
Content Server. Monitor long running cursors after change.
For more details, see How to handle Hana Alert 42.
SQL
Plan Cache Enabled
Description The plan_cache_enabled parameter allows the execution plans
of SQL queries to be parsed and stored.
Default true
Recommendation Leave as default
Change Level Container and Tenant
17
Plan Cache Size

Description SQL Plan Cache is an area allocated in memory where the executive
plans of SQL queries are parsed and stored.
Default 2 to 5 GB
The actual size may vary.
Recommendation 0.90 * (current plan_cache_capacity)/(current plan_cache_hit_ratio)
Change Level Tenant
Notes The plan_cache_enabled parameter must be set to true for the

plan_cache_size to be implemented.
If it is undersized, it can result in Alert 58: Plan Cache Size alerts.
The Plan Cache Hit Ratio from the view SQL Plan Cache should be
over 90 percent.
SQL Plan Cache values over 20GB is not recommended.
A restart of Hana may be required to affect any changes or by
resetting the view to get the correct statistics:
ALTER SYSTEM RESET MONITORING VIEW
SYS.M_SQL_PLAN_CACHE_RESET;
For more details, see Guide to Tuning SQL Plan Cache.
For more information on the Plan Cache, see SAP Hana Parsing.
Statisticsserver.ini
Memory Manager
Allocation Limit
Description The allocation_limit is the percentage amount of memory
which is the maximum that the Statistics Server will use.
Default 5%
Recommendation 10%, no higher than 20%
Change Level Container
Notes This should be set per tenant as required. For more details, see
Statistics Server runs out of memory.
18
Content Server Installation Options

Consider the following tips and guidelines when initially installing the SAP Hana for use
with Content Server:
• SAP Hana is a Container Database. This will require Content Server to be
installed on a Tenant Database. OpenText recommends that Content Server is in
its own Tenant.
• Although the Content Server installation has a built-in mechanism for creating the
tablespace and database account, it is also fine to create the user account in
advance. The Content Server installation will allow you to install using these.
The creation of a new user will create the database and schema required for
Content Server. No other permissions need to be added to the new Content
Server user account.
Warning: Content Server installs in the schema with the same

name as the database/user name. OpenText recommends not to
change the schema name for the Content Server database.
Warning: Content Server installed tables as Column Store. This

allows Content Server to utilize the in memory and compression
features of SAP Hana. Do not change column tables to row tables
as this will affect the performance of Content Server on SAP Hana.
• For SAP Hana, SAP recommends that you deactivate the SYSTEM account.
The SAP Hana DBA can create the following OTADMIN user for Content
Server to have the ability to create its own user without having all the
permissions of SYSTEM. The user will need to be created in the Tenant
Database which will be used for Content Server.
The OTADMIN user can be added through SAP Hana Cockpit or SAP Hana
Studio.
Create the OTADMIN user SQL:
create user OTADMIN password Welcome1
no force_first_password_change;
Granting permissions to the user is best done through the menus as not all the
permissions can be added through scripts.
Permissions for OTADMIN:
o Package Privileges – Root Package
 All the Repo objects grantable to others
o Objects Privileges – Repository_Rest (Sys) - Execute grantable to
others
o Granted Roles – Content_Admin
19
o System Privileges (all grantable to other users and roles)

 Catalog Read, Create Schema, Credential Admin, Data
Admin, Export, Import, Repo.Export, Repo.Import,
Repo.Maintain_Delivery_Units, Repo.Modify_Change,
Repo.Modify_Foreign_Contribution,
Repo.Modify_Own_Contribution,
Repo.Work_In_Foreign_Workspace, Session Admin, Table
Admin, User Admin
After the above account has been created the Content Server
administrator can login using the OTADMIN account to create the Content
Server user.
• After installation, the SAP Hana DBA can remove the superuser rights to the
OTADMIN account. The superuser will not be required in any upgrades to
the system.
20
Performance Tuning Tips
Statement Memory
Protect SAP Hana from excessive memory usage due to uncontrolled queries by
limiting the amount of memory used by single statement executions per host.
Statement executions that require more memory will be aborted when they reach the
limit.
To trace these statements, turn on resource tracking by changing the parameters listed
in Workload Management in the Content Server tenant database. The trace should
only be turned on when troubleshooting issues.
Memory Manager
The Memory Manager section of the global.ini files contains parameters that allow you
to control the memory consumption of SAP Hana.
The global allocation limit limits the amount of memory that can be used by the
container (in megabytes). The default value is 0 – which uses default settings. The
global allocation limit is calculated by default as: 90% of the first 64GB of available
physical memory on the host + 97% of each further GB.
The allocation limit limits the amount of memory that can be used by an individual
process (in megabytes). The default allocation limit is calculated as above. When the
limit is reached, a dump file is created with compositelimit_oom in the name. The
statement is aborted.
The statement memory limit is applied if the current SAP Hana memory consumption
exceeds the statement memory limit threshold as a percentage of the global allocation
limit.
See setting suggestions in, Memory Manager.
Shared Memory
Shared memory is where system information and row store tables are stored. High
usage in shared memory can cause memory shortage with SAP Hana.
Verify that all tables are column store tables.
It may be necessary to increase system memory.
Note that memory usage can spill between tenants. It may be necessary to have
Content Server on a separate instance from other SAP Hana databases.
21
Monitoring and Benchmarking
Large Memory Usage of Statistics Server

Large memory usage can happen when the majority of memory used by the Statistics
Server has too many alerts or is undersized in memory. Big Statistics Server table
leads to performance impact on the system.
Check the size of the table _SYS_STATISTICS.STATISTICS_ALERTS_BASE and
truncate STATISTICS_ALERTS_BASE from hdbsql.
Monitoring
The System Monitor provides an overview of all the SAP Hana databases at a glance.
Each database can be drilled down into an individual system in the Administration
editor.
To see the System Monitor views, you must have the MONITORING role.
The Administration editor allows you to identify problems and avoid disruptions.
For more information, see Monitoring in SAP Hana Cockpit or Managing and
Monitoring Performance in SAP Hana Cockpit.
Consistency Checks
SAP Hana recommends that you perform consistency checks on a regular basis to find
any corruptions.
Running these checks can affect performance, so it is recommended to run these
checks on a maintenance window.
For more details, see Table and Catalog Consistency Checks and Technical
Consistency Checks.
22
Identifying Worst-performing Queries

There are several ways to identify poorly-performing SQL.
Load Monitor, system views and SQL Plan cache are SAP Hana tools that can help
troubleshoot issues that are currently happening.
For more information, see SAP Hana SQL Optimization.
Content Server Connect Logs

Content Server connect logging (SQL Connect logs) generates logs that include every
Content Server transaction and the SQL statements that they issue. The logs provide
timings in microseconds for each statement. Note that connect logs can use
substantial space if left enabled for significant periods of time, which can add up to
25% overhead.
The connect logs can be found in Content Server Administration – Server Configuration
– Configure Log Settings. A log level of 3 – Debug will give you the most detail about
the SQL statements generated.
Please note: you may wish to only have the log level at the high setting to capture the
performance issues. Set the level to a lower level or 0 – off once the issue has been
captured.
WARNING: Any ad-hoc tuning that involves the modification,
introduction, or removal of indexes can be detrimental to overall
performance of Content Server.
If there are any indexing concerns, report them to OpenText
Content Server customer support.
Performance Analyzer
OpenText Performance Analyzer is a tool that works with Content Server Logs. This
tool allows you to open connect logs, see the percentage mixture, and see the relative
importance of your site’s usage profile.
The Raw Data tab allows you to sort the transactions by overall execution time and
SQL time. This will show you which Content Server transactions are taking the most
SQL time (not just the individual statements that they issue).
To see the individual SQL statements (from the Raw Data tab), right-click on the line
and select Show SQL. Performance Analyzer will display each SQL statement that the
transaction issues, how long each one took to execute, and how many rows were
affected.
23
Blocked Transactions
Currently Blocked Transaction Monitoring can only be done through SAP Hana Studio.
They are indicated in the Administration editor on the Performance – Threads tab.
Use the Load Monitor tool to isolate the exact time frame where the issue happened.
From there, investigate what happened at the specific time frame by checking:
_SYS_STATISTICS.HOST_BLOCKED_TRANSACTIONS – analyze the columns
WAITING_SCHEMA_NAME, WAITING_TABLE_NAME, and WAITING_RECORD_ID
SYS.M_DELTA_MERGE_STATISTICS (or _SYS_STATISTICS.HOST_DELTA_
MERGE_STATEMENTS) – columns START_TIME and EXECUTION_TIME provide
information if there was a Delta Table Merge running
SYS.SAVEPOINTS (or _SYS_STATISTICS.HOST_SAVEPOINTS) – check if a
savepoint was written during the time period
Only check entries that have a TOTAL_LOCK_WAIT_COUNT > 0 and compare the
MAX_CURSOR_DURATION against AVG_CURSOR_DURATION. If there is a significant
difference, then the transactions took longer than average.
For more information, see Blocked Transaction Monitoring.
Workload Management
In SAP Hana, it is possible to turn on tracing to monitor memory and CPU issues with
SAP Hana. The trace should only be turned on when troubleshooting issues.
To turn on tracing, change the following parameters:
SET ('expensive_statement', 'enable') = 'true'
WITH RECONFIGURE;
SET ('resource_tracking', 'enable_tracking') = 'on'
WITH RECONFIGURE;
SET ('resource_tracking', 'memory_tracking') = 'on'
WITH RECONFIGURE;
SET ('resource_tracking',
'service_thread_sampling_monitor_enabled')='true'
WITH RECONFIGURE;
SET ('memorymanager', 'statement_memory_limit') = '12'
WITH RECONFIGURE;
For more details, see SAP Hana Memory and SAP Hana CPU.
24
Administration Recommendations
This chapter provides recommendations for administering a SAP Hana instance that
hosts a Content Server repository.
Backup and Recovery

Ideally, a cold backup of both the external file store (EFS) and a frequent hot or online
backup of the SAP Hana database and archive logs provides the best insurance for a
completely consistent and in-synch backup of the Content Server site.
Details about SAP Hana Backup and Recovery are available in Availability and
Scalability.
Only primary systems can be recovered with data and log backups.
Whenever a point-in-time recovery is done of the primary system, the secondary site
(if active) must be newly registered for SAP Hana System Replication.
25
Appendices
Appendix A - References
SAP Hana Administration Guide

https://help.sap.com/viewer/6b94445c94ae495c83a19646e7c3fd56/2.0.03/en-
US/330e5550b09d4f0f8b6cceb14a64cd22.html
SAP Hana Platform (Core)
http://help.sap.com/hana_appliance
SAP Hana Troubleshooting and Performance Analysis Guide
https://help.sap.com/viewer/bed8c14f9f024763b0777aa72b5436f6/2.0.03/en-
US/7d28bc8c4e54413caf2716731494da88.html
SAP Hana Wiki
https://wiki.scn.sap.com/wiki/display/SAPHANA
SQL Statement Collection for SAP Hana

https://launchpad.support.sap.com/#/notes/1969700

Appendix B – SAP Hana Bugs of Interest
• The page cache of a database is larger than expected (for more details, see
https://launchpad.support.sap.com/#/notes/2403124).
Workaround: Turn off the optimization of keeping column table main persistent
pages in memory (hidden parameter, default True).
ALTER SYSTEM ALTER CONFIGURATION ('global.ini','SYSTEM')
SET ('persistence','internal_caching_for_main') = 'false'
with reconfigure;
• Larger Memory Utilization Than Specified - affected releases SAP Hana 2

0.24.01 to 0.24.07 (SPS02) and 030.00 to 034.00 (SPS03), see
https://launchpad.support.sap.com/#/notes/2729005.
Workaround: Turn off the sharing allocator
ALTER SYSTEM ALTER CONFIGURATION ('global.ini','SYSTEM')
SET ('memorymanager',
'enable_sharing_allocator_for_implicit') = 'false'
WITH RECONFIGURE;
Solution: Apply revisions - 0.24.08 (SPS02) or 035.00 (SPS03) or higher

• Alert 58 – Plan Cache Size is too small (see SAP Hana Parsing #2) has been
corrected but still receiving the alert
This alert is based on the evicted count. The count needs to be reset.
Solution: Review the evicted count number from the view SQL Plan Cache.
Review the evicted plan count from Statistics Queries.
Increase the threshold amount in plan_cache_parameter_threshold.
• Trace files *.*oom* generated due to memory problems (see SAP Hana
Memory)

Appendix C – Troubleshooting Queries

These queries should be run by the DBA with SYSTEM type privileges. They can be
run in either the Tenant or Container database.
Column Table Compression
Description Generates SQL to compress column tables which have high memory
consumption
Sample SELECT SCHEMA_NAME, TABLE_NAME,

SUM(RAW_RECORD_COUNT_IN_MAIN) Records,
'UPDATE "'||SCHEMA_NAME||'"."'||TABLE_NAME
||'" with parameters ('''||'optimize_compression'
||''''||'='||''''||'force'|| ''');' Command
FROM M_CS_TABLES
WHERE LAST_COMPRESSED_RECORD_COUNT = 0
AND RAW_RECORD_COUNT_IN_MAIN > 10000000
AND NOT (SCHEMA_NAME LIKE '_SYS%'
OR SCHEMA_NAME LIKE 'SYS%'
OR SCHEMA_NAME LIKE 'SAP%'
OR SCHEMA_NAME LIKE 'HANA%')
GROUP BY SCHEMA_NAME, TABLE_NAME
ORDER BY SUM(RAW_RECORD_COUNT_IN_MAIN) DESC;
Column Table Memory Usage

Description List schemas within a tenant and the total memory used for column
tables
Sample SELECT host, schema_name,

ROUND(SUM(MEMORY_SIZE_IN_TOTAL)/1024/1024, 2)
"Column Tables MB Used"
FROM M_CS_TABLES
WHERE NOT (schema_name LIKE '_SYS%'
OR schema_name LIKE 'SYS%'
OR schema_name LIKE 'SAP%'
OR schema_name LIKE 'HANA%')
GROUP BY host, schema_name
ORDER BY SUM(MEMORY_SIZE_IN_TOTAL) DESC;
Heap Memory
Description List the used allocators that are responsible for the highest heap
memory consumption (top 15)
Sample SELECT TOP 15 M_SERVICES.HOST, SERVICE_NAME, CATEGORY,

ROUND(INCLUSIVE_SIZE_IN_USE/1024/1024/1024, 2)
"Inclusive In Use GB",
ROUND(EXCLUSIVE_SIZE_IN_USE/1024/1024/1024, 2)
"Exclusive In Use GB"
FROM M_HEAP_MEMORY INNER JOIN M_SERVICES
ON M_HEAP_MEMORY.PORT = M_SERVICES.PORT
AND M_HEAP_MEMORY.HOST = M_SERVICES.HOST
ORDER BY EXCLUSIVE_SIZE_IN_USE DESC;
Notes Watch to see if the Inclusive Size to close/same as the Exclusive size

Host Resource Utilization

Description List the physical memory used/available
Sample SELECT HOST,

ROUND(USED_PHYSICAL_MEMORY/1024/1024/1024, 2)
"Resident GB",
ROUND((USED_PHYSICAL_MEMORY+FREE_PHYSICAL_MEMORY)/
1024/1024/1024, 2) "Physical Memory GB",
ROUND(INSTANCE_TOTAL_MEMORY_USED_SIZE/
1024/1024/1024, 2) "Used Memory GB"
FROM M_HOST_RESOURCE_UTILIZATION;
Long Running Cursors

Description Run this query after receiving a long lived cursor in the alerts
Sample SELECT P.HOST, P.PORT, C.LOGICAL_CONNECTION_ID,

C.CLIENT_HOST, C.CLIENT_PID, P.STATEMENT_STATUS,
P.STATEMENT_STRING,
SECONDS_BETWEEN(P.LAST_EXECUTED_TIME,
CURRENT_TIMESTAMP) IDLE_TIME,
P.START_MVCC_TIMESTAMP
FROM M_PREPARED_STATEMENTS P, M_CONNECTIONS C
WHERE P.HOST = C.HOST AND P.PORT = C.PORT
AND P.CONNECTION_ID = C.CONNECTION_ID
AND C.CONNECTION_ID > 0
AND SECONDS_BETWEEN(P.LAST_EXECUTED_TIME,
CURRENT_TIMESTAMP) > 0
AND P.STATEMENT_STATUS <> 'NONE';
Row Table Memory Usage

Description List schemas within a tenant and the total memory used for row tables
Sample SELECT host, schema_name,

ROUND(SUM(USED_FIXED_PART_SIZE +
USED_VARIABLE_PART_SIZE)/1024/1024, 2)
"Row Tables MB Used"
FROM M_RS_TABLES
WHERE NOT (schema_name LIKE '_SYS%'
OR schema_name LIKE 'SYS%'
OR schema_name LIKE 'SAP%'
OR schema_name LIKE 'HANA%')
GROUP BY host, schema_name
ORDER BY SUM(USED_FIXED_PART_SIZE +
USED_VARIABLE_PART_SIZE) DESC;

Service Memory
Description List the heap (column) and shared (row) memory usage
Sample SELECT HOST, PORT, SERVICE_NAME,

ROUND(ALLOCATION_LIMIT/1024/1024/1024, 2)
"Allocation Limit GB",
ROUND(EFFECTIVE_ALLOCATION_LIMIT/1024/1024/1024, 2)
"Effective Allocation Limit GB",
ROUND(HEAP_MEMORY_ALLOCATED_SIZE/1024/1024/1024, 2)
"Heap Memory Allocated GB",
ROUND(SHARED_MEMORY_ALLOCATED_SIZE/1024/1024, 2)
"Shared Memory Allocated MB",
ROUND((HEAP_MEMORY_ALLOCATED_SIZE+
SHARED_MEMORY_ALLOCATED_SIZE)/1024/1024/1024, 2)
"Total Memory Allocated GB",
ROUND(CODE_SIZE/1024/1024/1024, 2) "Code Size GB",
ROUND(HEAP_MEMORY_USED_SIZE/1024/1024/1024, 2)
"Heap Memory Used GB",
ROUND(SHARED_MEMORY_USED_SIZE/1024/1024, 2)
"Shared Memory Used MB",
ROUND(TOTAL_MEMORY_USED_SIZE/1024/1024/1024, 2)
"Total Memory Used GB",
SHARED_MEMORY_ALLOCATED_SIZE/
EFFECTIVE_ALLOCATION_LIMIT * 100
"Shared Memory Allocated PCT",
SHARED_MEMORY_USED_SIZE/TOTAL_MEMORY_USED_SIZE * 100
"Shared Memory Used PCT",
HEAP_MEMORY_ALLOCATED_SIZE/
EFFECTIVE_ALLOCATION_LIMIT * 100
"Shared Memory Allocated PCT",
HEAP_MEMORY_USED_SIZE/TOTAL_MEMORY_USED_SIZE * 100
"Shared Memory Used PCT"
FROM M_SERVICE_MEMORY
ORDER BY SHARED_MEMORY_USED_SIZE DESC;
Shared Memory
Description List the shared memory distribution
Sample SELECT HOST, PORT, CATEGORY,

ROUND(ALLOCATED_SIZE/1024/1024, 2) "Allocated MB",
ROUND(USED_SIZE/1024/1024, 2) "Used MB",
ROUND(FREE_SIZE/1024/1024, 2) "Free MB"
FROM SYS.M_SHARED_MEMORY
ORDER BY USED_SIZE DESC;

SQL Plan Cache

Description List the Plan Cache Used memory
Sample – Plan SELECT HOST, PORT, USER_NAME, STATEMENT_STRING,

Cache Used By ROUND(PLAN_MEMORY_SIZE/1024/1024, 2)
"Plan Memory MB",
Statement PARAMETER_COUNT, TOTAL_EXECUTION_TIME
FROM M_SQL_PLAN_CACHE
WHERE NOT (USER_NAME LIKE 'SYS%'
OR USER_NAME LIKE '_SYS%')
ORDER BY PLAN_MEMORY_SIZE DESC;
Sample – Plan SELECT HOST, PORT, STATISTICS_COLLECTION_ENABLED,

Cache Used ROUND(PLAN_CACHE_CAPACITY/1024/1024, 2)
"Plan Cache Capcity MB",
ROUND(CACHED_PLAN_SIZE/1024/1024, 2)
" Plan Cache Size MB",
ROUND(CACHED_PLAN_TOTAL_EXECUTION_TIME/1000000
/60, 2) "Plan Cache Total Execution Time mins",
ROUND(PLAN_CACHE_HIT_RATIO * 100, 2)
"Plan Cache Hit Ratio %",
EVICTED_PLAN_COUNT
FROM M_SQL_PLAN_CACHE_OVERVIEW;
Statistics Queries
Description Review information in statistics
Sample – SELECT *
Statistic Alerts FROM _SYS_STATISTICS.STATISTICS_ALERT_THRESHOLDS
WHERE ALERT_ID = 58;
(Plan Cache Size)
Sample – SELECT TOP 500 HOST, SERVER_TIMESTAMP,

Memory Usage ROUND(INSTANCE_TOTAL_MEMORY_USED_SIZE/1024/1024/
1024, 2) "Used Memory GB"
FROM _SYS_STATISTICS.HOST_RESOURCE_UTILIZATION_
STATISTICS
WHERE SERVER_TIMESTAMP >= '2019-03-01'
ORDER BY INSTANCE_TOTAL_MEMORY_USED_SIZE DESC;

About OpenText
OpenText enables the digital world, creating a better way for organizations to work with information, on premises or in the
cloud. For more information about OpenText (NASDAQ: OTEX, TSX: OTC) visit opentext.com.
Connect with us:
OpenText CEO Mark Barrenechea’s blog

Twitter | LinkedIn
32
www.opentext.com/contact
Copyright © 2019 Open Text SA or Open Text ULC (in Canada).
All rights reserved. Trademarks owned by Open Text SA or Open Text ULC (in Canada).

Sap Hana For Opentext Content Server 16.2.X: Best Practices

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Sap Hana For Opentext Content Server 16.2.X: Best Practices

Uploaded by

Copyright:

Available Formats

Product: OpenText Content Server

OpenText Performance Engineering Team

The Information Company™ 2

The Information Company™ 3

DISCLAIMER: This Best Practices document contains procedures

General Overview of Content Server Database

SAP Hana Setup Best Practices

SAP Hana Database Sizing

Linux Kernel Parameters

Storage Best Practices

SAP Hana Database Installation Configurations

Data Volume Management

SAP Hana Configuration Settings

Changing Parameter Settings

Change Level Container only

Notes This change requires a restart.

Recommendation Set this parameter to 30 to 50% of available CPU threads, but no

Change Level Tenant only

Notes Parameters max_concurrency_hint should be smaller or equal to

Default 0 – number of logical CPU cores of host

Recommendation Level as default

Notes You can use the parameter max_concurrency_dyn_min_pct for

Max Concurrency Dynamic Minimum Percentage

Default Not set

Recommendation Level as default

Change Level Tenant

Default Not set

Recommendation For container, global_allocation_limit

Change Level Container

Global Allocation Limit

Default 0 - (90% * 64GB) + (97% * Total RAM – 64GB)

Recommendation Leave container setting as is

Change Level Container and/or Tenant

Notes For non-production systems, the default should be changed to restrict

Statement Memory Limit

Default Not set

Recommendation 30% of the first 500GB of the global_allocation_limit, and

Change Level Tenant

Notes This can be set by user and workload class as well.

Consistent Change Lock

Change Level Tenant

Notes This change should only be done if recommended by the

Enable Auto Log Backup

Recommendation SAP recommends leaving the default value

Change Level Container or Tenant database

Notes During normal system operations (log_mode = normal), SAP strongly

Recommendation Leave as default but monitor the log file creation

Change Level Container or Tenant database

Notes The normal setting is recommended to provide support for point-in-

Parallel Persistence Garbage Collection

Default 0 – The total number of logical CPU cores of the host.

Recommendation ½ of the total CPUs

Change Level Container or Tenant

Notes By reducing the value in max_gc_parallelity you can reduce the

Default idle_connection_timeout = 1440

Recommendation To avoid a disconnection of the client idle connections and potentially

Change Level Container or Tenant

Recommendation Leave as default

Change Level Container and Tenant

Plan Cache Size

Recommendation 0.90 * (current plan_cache_capacity)/(current plan_cache_hit_ratio)

Change Level Tenant

Notes The plan_cache_enabled parameter must be set to true for the