How to use SAP transaction ST06 for SAP performance analysis

In my previous post, I have covered how to run ST06, navigate through important ST06 screens and understand those
screens. In this post, I would talk about how to use SAP transaction ST06 for performance analysis:
1. Use SAP ST06 to do CPU workload analysis
2. Use SAP ST06 to do SAP system memory load analysis
3. Other analysis
I mainly focus on CPU and Memory analysis which are often showed up as a concern to application/system performance.
SAP ST06 itself cannot provide all data needed for CPU load and memory performance analysis. SAP ST06 can be used to
find out when and where there is a performance concern related to CPU and memory etc for further analysis. Working in
conjunction with other performance tools like SAP workload monitor ST03/ST03N, database load monitor- ST04, SAP
transaction statistics tool STAD and SAP snapshots collector -/SDF/MON etc, you can identify SAP work processes ,
jobs/programs, users who contributed to CPU/Memory performance issue, then solutions/options can be reviewed and worked
on subsequently.

1 Use SAP ST06 to do CPU workload analysis

CPU workload analysis is to make sure that CPU workload is not exceeding recommended utilization 80% and maintaining
CPU workload balance by analyzing peak load period and load distribution pattern.

1.1 Review CPU load to make sure CPU utilization under the recommended threshold
One of important goals for CPU workload analysis is to make sure that CPU utilization is not over general recommendation
Sum of User and Sys CPU utilization should not over 80% on hourly basis. For application server, system utilization portion of
CPU should not be over 10% and for data base server, it should not be over 20%. It is more critical to maintain at least 20%
CPU idle rate for database server since database server is a central instance and all SAP transactions/programs need to
access database server one way or another.
You can run ST06 and refresh the main screen to see whether CPU is over utilized now. You can use hourly CPU utilization
history for the past 24 hours or daily average in the past 30 days provided by SAP ST06 to see whether CPU has been over
utilized in the past. If there are situations that those thresholds are exceeded, you need to look into the details on who (jobs,
processes or users) are contributing most to the over utilization so you can review the situation and identify the solution.
Then you might ask how to identify top jobs, processes or users who is consuming a lot of CPU power. Unfortunately, SAP
ST06 itself cannot provide SAP jobs or processes names. You need to use different SAP or non SAP tools to complete the
performance analysis dependant on whether you are reviewing a current CPU load concern or past CPU load concern. Since
we are dealing with SAP system performance and the CPU overutilization is mostly related to processes from SAP in most
cases, this post would cover details how to identify top SAP jobs, processes or users. There are cases when non SAP
processes are making significant contribution to CPU overutilization, I would briefly mention this.

1.1.1 How to identify TOP CPU SAP user, processes, programs/jobs related to current CPU load
issue
First, you need to refresh ST06 Top CPU process screens to know which ST06 process is a top CPU process. Following is a
part of ST06 Top CPU processes screen:

Figure 1 ST06 TOP CPU processes

If a process is continuously showed at top of above screen during the observation period, then it is an expensive process. If it
is related SAP work process or a SAP database session/process. You can find corresponding sap user, sap work process or
database session, SAP job/program name via following method.
For a process in ST06 Top CPU processes screen which is related to a SAP work process, you can use following way to
identify related SAP user, program or jobs Get the Process ID from ST06 Top CPU processes screen.
Run SAP work process monitor SM50 and locate SAP work process based on Process id. From SAP SM50 screen,
you can know the SAP process type, program name and SAP userID.
For SAP background process, you can find corresponding SM37 job name: Run SE16 against SAP table TBTCO, put
process-id into field TBTCO-WPPROCID, execute it, you would get SM37 job name. There are other ways to get
background job name based on process ID.
For a process in ST06 Top CPU processes screen which is related to a SAP database process, you can get the corresponding
database process and further the SAP work process if applied:

Get the Process ID from ST06 Top CPU processes screen.

Run SAP ST04 database process/session monitor. Process ID in step 1 is actually Operating System PID in ST04
session monitor. So you can identify the corresponding database session to know the SQL statement which is
causing the load on the database server from there, you can review the SQL statement to identify any possible
improvement opportunity. Please refer to my post on how to tune SQL.
Most of database processes are result of running SAP application jobs or programs. You can identify the
corresponding job or programs via client ID in ST04 session monitor screen. Client ID is actually mapped to
SAP50/SM66 work process ID.
You can identify corresponding SAP job if it is a background process (see above).

If a process in ST06 Top CPU processes screen is a non-SAP process and you do not have access to operating system shell,
you need to talk with your system-level administrator to find out operating system level program/script/job name.
ST06 process name like DW.* is related to SAP work process in SM50. ST06 process name like ORACLE* is related to a
database session/process in ST04. This has been covered in my previous post how to run ST06, navigate through important
ST06 screens and understand those screens.

1.1.1 How to identify TOP CPU SAP user, processes, program/jobs related to past CPU load
For past CPU overload, you need to use SAP transaction ST03/ST03N SAP system workload monitor to analyze the
workload for the specific period to identify top offenders. SAP transaction ST03/ST03N is an important performance tool. You
can use ST06 data as an input for ST03/ST03N analysis. If /SDF/MON is scheduled in the system, you can use /SDF/MON
system performance snapshots to do further analysis.

1.2 Review CPU load to achieve better load balance

Another goal of CPU workload analysis is to achieve CPU load balance including vertical and horizontal balance. Vertical
balance is to make sure that the same SAP server has proper load balance over time period such as a day or a week. The
objective of vertical CPU load review is to spread CPU load out over time so a server CPU power can be used equally in
different time period as much as possible. The objective of horizontal CPU load review is to spread CPU load among all
servers/instances of a SAP system so each server of a SAP system has equal utilization at any specific moment whenever
possible. In reality, it is impossible to achieve absolute vertical or horizontal CPU utilization balance due to various reasons
business office hours and non business office hours, business pattern(seasonal ), different level of resource consumption by
different programs/jobs and different level of resource consumption by the same program due to volume difference etc.
SAP ST06 can display hourly CPU utilization for past 24 hours. It can display daily average CPU utilization for a single server
and across systems for the past 30 days. This information is an input for load balance analysis. SAP ST06 cannot tell you
which users and processes are creating those loads. You need to work with SAP workload monitor ST03/ST03N to find further
details.
Load balance issue is normally caused by many concurrent running SAP work process as a result of parallel solution,
immediate processed EDI/ALE interface in my experience and incoming RFC calls etc. SAP CPU workload balance analysis
should look into those areas first. It is really rare that culprit is singe thread of SAP work process.

1.3 Understanding CPU load and application/system performance

CPU load is an important factor for application/system performance. But there is no simple equation to say: high CPU load =
bad performance, low CPU load = good performance; High CPU load = bad user experience, Low CPU load = good user
experience. You can refer to my previous post to understand other factors which might influence a program performance.
However I would assume here that CPU is only factor which influence performance to make discussion simple.
Even hourly average CPU utilization is under 80%, this does not mean all business transactions executed in that hour would
have normal performance. Whether there is a performance impact, this really depends on CPU load distribution in that hour
and how sensitive a business transaction is to runtime. For example, if CPU utilization for 1 st half hour is 90% and 2nd half hour
utilization is 10%, then average CPU utilization is 50% for that hour well below the recommended utilization, however
performance of a business transaction which is running in the 1st half hour would be very likely impacted. This would also
depends on normal utilization If normal utilization is 50%, then CPU utilization jumps to 80%, an application or transaction
performance might be impacted. If a job/program normally runs for several days, then single hour CPU over utilization on
runtime of such job or program can be ignored. In another way, when we evaluate CPU impact on a long running
transaction/job runtime, we need to check CPU load history instead of current CPU load.
Also if a SAP system has separate database server and application server, one server has overloaded but another server is
ok.. the performance impact on a transaction might depends on where the transaction spends most time. If 99% of time is
spent on database server and database server is not over loaded , CPU contention on an application server is very likely have
no impact on the performance of transaction where it is executed on application server. So on.
Even performance of all business transactions in the SAP system are impacted from technical analysis point view. This might
be different from what business users feel. Since each business transaction and process has an acceptable runtime range
that is SAP users expectation as well. Business users would only feel the performance impact if business transaction or
performance has a notable deviation from its normal runtime range. For example, a background job has a run time range of 1
2 hours based on volume, business might not feel difference if what should be finished in 1 hour normally for small volume
takes 1 and half hour due to CPU overutilization. However, business would feel the performance issue if big volume happens
to kick in while there is a CPU overutilization in this case.
Technically, CPU load would impact a program/job performance if a business job needs CPU but have to be put to wait due to
CPU contention. When a job/program spends a lot of time waiting for CPU, its performance is impacted. SAP transaction
STAD would show statistical data of job/program runtime which break job runtime down to further components. Analysis of
those components can help you to understand the impact of CPU contention. For example, in server with 4 processors
running only 4 jobs and each of 4 jobs is purely doing calculation w/o disc for hours, You should see CPU 100% utilization but
each of 4 jobs spends no time in waiting for CPU.

Under current SAP technology, we normally do not need to worry about individual CPU/processor utilization showed in
Snapshot CPU screen (Not TOP CPU screen which shows a list of work processes). However, this could be a concern if
one or more of server CPUs are reserved for specific purpose/application.

1.4 Other CPU workload analysis tools

There are other tools for CPU load analysis. I have mentioned SAP workload monitor SAP transaction ST03/ST03N in
previous section. It can be used to do workload analysis as well. You can use ST06 to identify peak load period and use SAP
workload monitor ST03/ST03N for further analysis. SAP database load monitor ST04 can be used to monitor SQL operation
load on database server. I would cover SAP workload monitor ST03/ST03N and database load monitor ST04 in the future.
My favorite tool for load monitor analysis is SAP performance snapshot program /SDF/MON. /SDF/MON is not good for
current CPU contention but it can be used to collect CPU load snap-shots for historical utilization analysis. ST ST06 is good
for current CPU load analysis but depends on other tools to find process, sap job, user and program information. So you can
ST06 to know when there is a CPU contention, then use SAP /SDF/MON transaction to collect snapshot for that particular
period for details analysis. You can refer to my post SAP /SDF/MON workload analysis for details on how to use SAP
transaction /SDF/MON in load analysis.

1.5 Process relation among SAP ST06, SM50/SM66 and ST04

This section would tell you the linkage among ST06 TOP CPU process, SM50 SAP work processes and ST04 database
process/sessions via examples. Exception where a process exists in one tool but not in another tool due to SAP job
cancellation etc is not discussed here.

1.5.1 How is a ST06 database related process linked to a SAP SM50 work process

Figure 2 ST06 DB related process -> SM50 process

1.5.1 How is a ST06 SAP-related process linked to a SM50 SAP work process

Figure 3 ST06 OS process -> SM50 process

1.6 Possible solution for mitigating CPU work load and achieving better load balance
Through above analysis, you identify the top peak period and top business transactions/jobs, you can consider following
solutions to reduce CPU load and achieve better load balance:

Schedule run t job in different time when system is less busy or change job frequency

Tune job/program logic and database operation So the job/program is more efficient and use less CPU. This can
include job variant change.
Workload balance/redistribution run the job/program in another server/instance of the SAP system.
Upgrading hardware more powerful server or more CPUs/Processors.

Upgrading hardware should be the last resort after all other options have been tried. For tuning ABAP program performance,
you can refer to my post how to do performance trace using SAP ST12 and how to tuning ABAP program performance if
you like.
If external process is using a lot of CPU power, then you can use similar solutions as what we have for SAP related
processes. Or you can simply terminate it and stop it depending on your scenario.

2 Use SAP ST06 to do Memory load analysis

Memory workload analysis is to make sure that there is enough free memory and free swap space (Virtual memory) available
in the system by analyzing peak memory and memory usage pattern.
SAP ST06 can show you current server memory utilization and hourly average memory utilization for past 24 hours. Please
refer to my post how to run ST06 on ST06 screen navigation and understanding. Following is a part of past 24 hours
memory history screen:

Figure 4 ST06 past 24 hr memory history

To know current SAP server memory utilization status, you can just run ST06 transaction and refreshing the main screen
every 10 seconds. If you would like to know which SAP process/user is consuming the server memory, you need to access
mode list screen of SAP transaction ST02 SAP buffer/memory monitor. ST06 provides top CPU processes but
unfortunately it cannot provide Top memory processes.

Figure 5 ST02 mode-list

To know history memory usage, you can run SAP ST06 screen and navigate to past 24 hours memory screen. However, if you
observed a particular period when the memory usage is a concernthat is all you can get from SAP ST06. You can use this
information to schedule SAP transaction /SDF/MON to get memory snapshots for the period concerned, Then based on
/SDF/MON memory snapshots, you can figure who and which processes are top memory consumption processes/users for
the period in questions. That is why SAP /SDF/MON stands out as one of my favorite performance tool both for SAP

production support and SAP development. Please refer to my post on how to run /SDF/MON for memory analysis if you would
like to know more on it.

Figure 6 /SDF/MON snapshots

Memory load analysis also pays attention to swap space. I normally just check whether there are enough free swap space and
hourly paging out rate. Hourly paging out rate for UX and Linux OS should not be over 20% of physical memory in general.
Here I assume that all memory consumption in a server are from SAP users or sap processes. SAP ST02 and SAP
/SDF/MON is for analyzing memory consumption from SAP users in the instance/system where it is started. If there are other
instances on one server or there are operating system level processes which consume a lot of memory, they would not be
captured by SAP ST02 or SAP /SDF/MON. UX level or other sap instances need to be checked at the same time.
Possible solutions to mitigate performance concerns due to memory overload and achieve better SAP memory load balance
are:
1.
2.
3.
4.
5.

Reduce volume you might need to educate end business users or change job variant.
Schedule Run job/transaction in different time when there is less memory load.
Redeployment run the job in different instance or server.
Tune the SAP job/program to reduce memory usage.
More capacity add in additional physical memory or swap space.

As for CPU load analysis, adding more capacity should the last resort when other options have been reviewed. If external
process/task is causing issue, you need to work with owner and look into solution depends on actual scenario.

4 Other analysis Via SAP ST06

Based on ST06, we can do IO analysis based on and file space analysis. Objective of this analysis is to make sure that there
is no hot spot or IO bottleneck by proper IO configuration such as spreading data out among system disc properly. You can
refresh ST06 Main screen and pay attention to DISC with highest response time to observe whether a particular disc has high
utilization for a significant duration and has a lot of process in waiting (queue length). You also can check past 24 hours disc
usage history to find out whether a particular disc has an extremely high utilization and queue length. I have not run into such
situation in my SAP performance work so far. No further insight can be shared. SAP system IO configuration is not my area as
well.

Figure 7 -ST06 Disc with Highest response time

File space analysis is to check that we have enough free space at OS directory level. This might be critical for successful
running of jobs/programs which need to output file to operating system level.

4. Further clarification
Up to now, I have covered on how to use SAP operating system monitor ST06 to do performance analysis: Identify
existing/potential problems using SAP operating system monitor ST06 and continue ST06 analysis with other SAP
performance tools. From ST06, you can access other SAP tools such as network traffic check which can be handy for
troubleshooting network related performance issue.