You are on page 1of 54

IBM Americas Advanced Technical Support

IBM SAP Technical Brief

IBM - SAP Java Monitoring & Tuning Guide

Damir Rubic IBM Americas Advanced Technical Support SAP & Oracle Solutions

Version: 1.0 Date: December 21, 2007

© 2007 International Business Machines, Inc.

IBM Americas Advanced Technical Support 1.  PREFACE ............................................................................................................................................. 3  1.1  1.2  1.3  1.4  1.5   1.6  2.  2.1.  2.2.  3.  3.1.  3.2.  4.  ACKNOWLEDGEMENTS.................................................................................................................... 3  DISCLAIMERS .................................................................................................................................. 3  COPYRIGHTS ................................................................................................................................... 3  FEEDBACK ...................................................................................................................................... 3  VERSION UPDATES.......................................................................................................................... 3  WHO SHOULD READ THIS DOCUMENT.............................................................................................. 4  SAP J2EE ENGINE SYSTEM ACHITECTURE ..................................................................................... 5  JAVA CLUSTER ACHITECTURE ...................................................................................................... 10  SAP J2EE ENGINE ADMINISTRATION TOOLS................................................................................ 13  SOFTWARE DEPLOYMENT MANAGER (SDM) ............................................................................... 19 

JAVA TECHNOLOGY IN THE SAP APPLICATION SERVER ................................................. 5 

SAP JAVA SERVER ADMINISTRATION.................................................................................... 13 

SAP JAVA SERVER MONITORING............................................................................................. 21  4.1.  TECHNIQUES FOR PROBLEM DETECTION ....................................................................................... 22  4.1.1  Monitoring of the Log and Trace files .................................................................................... 22  4.1.2  Availability Monitoring of SAP Java engine and Java based applications (GRMG)............. 26  4.1.3  Distributed Statistics Records (DSRs) .................................................................................... 28  4.1.4  State Monitoring ..................................................................................................................... 31  4.2.  TECHNIQUES FOR PROBLEM ANALYSIS ......................................................................................... 34  4.2.1  Performance Trace ................................................................................................................. 34  4.2.2  Single Activity Trace / JARM .................................................................................................. 37  4.2.3  Single Activity Trace ............................................................................................................... 40  4.2.4  SQL Trace ............................................................................................................................... 42  4.2.5  Application Trace.................................................................................................................... 44  4.2.6  Conclusion .............................................................................................................................. 47 

5. 

IBM JAVA TECHNOLOGY FOR SAP .......................................................................................... 49  5.1  5.2  5.3  IBM JVM GARBAGE COLLECTION ............................................................................................... 50  IBM JVM PARAMETERS ............................................................................................................... 51  MONITORING THE JVM ACTIVITY ................................................................................................. 51 

PUBLICATIONS: ...................................................................................................................................... 54 

© 2007 International Business Machines, Inc. Page 2

IBM Americas Advanced Technical Support

1. Preface
1.1 Acknowledgements

I would like to thank Mark Gordon and Jim Dilley for all their help regarding this and many other Oracle & SAP & pSeries performance-related subjects.

1.2

Disclaimers

IBM has not formally reviewed this paper. While effort has been made to verify the information, this paper may contain errors. IBM makes no warranties or representations with respect to the content hereof and specifically disclaim any implied warranties of merchantability or fitness for any particular purpose. IBM assumes no responsibility for any errors that may appear in this document. The information contained in this document is subject to change without any notice. IBM reserves the right to make any such changes without obligation to notify any person of such revision or changes. IBM makes no commitment to keep the information contained herein up to date.

1.3

Copyrights

SAP and R/3 are copyrights of SAP A.G. RS/6000, pSeries, and AIX are copyrights of IBM Corporation. Oracle is a copyright of Oracle Corporation.

1.4

Feedback

Please send comments or suggestions for changes to drubic@us.ibm.com.

1.5

Version Updates

Version 1.0 – initial version

© 2007 International Business Machines, Inc. Page 3

IBM Americas Advanced Technical Support

1.6

Who should read this document

This paper is primarily intended for pSeries Sales Specialists who would like to know more about the new SAP Technology Application Framework – NetWeaver 2004s. The goal of this paper is to provide the audience with material that will help them understand the most important elements of the SAP NetWeaver Application Framework, and how to combine these elements with the strengths of the IBM pSeries platform. The SAP NetWeaver Application Platform presents a new framework consisting of several different elements. This paper will focus on the J2EE engine, which is the central and most interesting element of the latest SAP Java announcement. This paper will provide the reader with an overview of the most important features of the J2EE engine. The J2EE engine also presents opportunities for extension into multiple areas. I would be happy to accept suggestions regarding exploring these additional areas for development.

© 2007 International Business Machines, Inc. Page 4

IBM Americas Advanced Technical Support

2. Java Technology in the SAP Application Server
2.1. SAP J2EE Engine System Achitecture

As a system, the J2EE Engine consists of three logical layers: 1. Java Enterprise Runtime – comprised of low-level subsystems that provide functions such as class loading, cluster communication, persistent configuration data management, etc. 2. J2EE Engine components – consists of interfaces, libraries and services components that provide various runtime functions and programming APIs 3. Applications – refers to the applications that are deployed and run on the J2EE Engine The J2EE Engine architecture is based on the following general rule: components from the higher level can use components from lower level; whereas components from the lower level are not aware of the APIs of the components from the higher level and therefore cannot use them. This rule is reflected by the starting order of the system modules: runtime is first, then services (libraries are loaded, and interfaces resolved at this phase), and lastly, applications. The system is considered functional when all runtime managers and core services components are started in the proper order. Components from the higher level use a set of defined APIs to utilize the functions of the components from the lower level. The J2EE Engine components use the Framework APIs to connect to the SAP Java Enterprise Runtime. Applications use the J2EE Engine components by the APIs that are defined by J2EE 1.3 Specification (and supporting specifications) and SAP-proprietary APIs.

© 2007 International Business Machines, Inc. Page 5

Inc. These are: • • • • • • • • Log Manager – manages the process of logging system events Ports Manager – manages the process of opening various communication ports on the system Pool Manager – manages the process of pooling Java objects (this does not refer to pooling JDBC connections) Application Thread Manager – handles threads in which client applications’ source code is executed Thread Manager – handles threads in which the J2EE Engine system operations are executed Connections Manipulator Manager – manages the process of opening TCP/IP sockets from the client to the J2EE Engine Locking Manager – handles locking as interface to the Enqueue Service Configuration Manager – manages the process of storing and reading persistent configuration data (such as properties and deployment data) to and from a relational database © 2007 International Business Machines. It is built of several low-level subsystems called Managers. Page 6 .IBM Americas Advanced Technical Support Figure 1-1 SAP J2EE Engine Architecture • SAP Java Enterprise Runtime The Java Enterprise Runtime provides the core functions of the system.

Inc. Page 7 . It provides runtime lifecycle management of those components The Service Manager is the container of all J2EE Engine components of the system. Figure 1-2 Service Manager © 2007 International Business Machines. It also manages components at runtime. Using the functions of other runtime subsystems. This is why the Service Manager is the last runtime subsystem to be started. it loads and starts the components when the J2EE Engine starts up.IBM Americas Advanced Technical Support • Class-loading Manager – controls activities for registering and unregistering of loaders and references between them. and stops them when the J2EE Engine is being stopped. This is necessary for locating and loading Java classes • Licensing Manager – handles SAP licenses • Cluster Manager – manages communication (in terms of cluster elements status and modules available on it at runtime) between J2EE Engine cluster elements • Service Manager – represents a container of pluggable components. as well as for accessing loaders referenced by another loader.

the Service Manager stops all started components. Resolves components for which the previous steps are successfully completed 4. If not. they provide the system with their name and classes (no objects). Interface components can also be implemented directly by applications. 2. Other library components or services components typically access those using static methods. Checks whether all core components are resolved. It is conducted exclusively by the Service Manager. Starts service components 7. If not. classes and objects to the system. 3. or when an object is first requested. Built on top of the runtime and able to communicate and use each other. Reads the components persistent data from the database 2. these components form the complete system infrastructure to run both J2EE and SAP proprietary applications. At runtime. Libraries – provide name. This includes checks for duplication of component names. © 2007 International Business Machines.IBM Americas Advanced Technical Support The process of starting (loading) the J2EE Engine components is strictly defined. They provide the system with their name. Services – more powerful than the other two types of components. unloads them and stops the whole system • SAP J2EE Engine Components The J2EE Engine components build the second level of the system. Service components can access and utilize functions of the runtime through the Framework API. The runtime objects are registered on the system once the components classes have been loaded. These components do not provide any runtime functions. for availability and consistency of components deployment descriptors. the Service Manager stops the entire system 5. It performs the following steps: 1. Page 8 . They are used by services components that provide their implementation. classes and runtime objects. Performs checks for data integrity. Performs a check on whether all core service components are started. Inc. These objects are created by the system when it loads the library. Creates the class loaders of the components and loads them 6. and so on 3. Interfaces – “contracts” that define how different components of the system work together. Three types of J2EE Engine components are defined: 1.

mySAP component DB Server DB Server EJB Server Legacy Figure 1-3 SAP J2EE Engine Containers © 2007 International Business Machines. These components reside in different containers on the J2EE Engine as shown in Figure 1-3..IBM Americas Advanced Technical Support • Applications The applications form the third level of the J2EE Engine architecture. The boundary between the applications level and the J2EE Engine components level is defined by the J2EE 1. Page 9 . Each of these containers provides runtime services and life cycle management for the application components.. Browser Web Server Presentation Layer Web Container Servlet Java Server Page J2EE Application Server J2EE Services JDBC JNDI JAAS JMS JTA JCA Business Layer EJB Container Session Bean Entity Bean . Applications use these APIs to utilize the functions of the different types of components on the lower level in the system.3 APIs along with a few SAP proprietary APIs. An enterprise application consists of several types of application components. Inc.

IBM Americas Advanced Technical Support 2. Java Cluster Achitecture A Java cluster installation consists of: • One or more instances of the AS Java • The Central Services. which also form an instance • One or more databases The dispatchers and servers can be split up among different physical servers. Figure 1-4 Layers of the SAP J2EE Engine © 2007 International Business Machines. An HTTP request to execute a servlet runs through several layers in the J2EE engine.2. Central Services are a special example of a Java instance. The Java dispatcher receives the request. stopped. and is then indicated as the central instance. An instance is identified by the system ID (SID) and the instance number. selects a server process for the processing and establishes the connection from the client to the server. The cluster elements that form an instance run on one physical machine. This instance usually runs with the database and Central Services on the same machine. Inc. and monitored. • Java Instance A Java instance is a unit in the AS Java cluster which can be individually started. The Central Services (Message Service and Enqueue Service) are installed on a host that meets the requirements for high availability. Page 10 . It is also possible to run several instances on one physical machine. A Java instance normally is comprised of one Java Dispatcher and several server processes. A Java instance is started and stopped by the Java Startup and Control Framework. Another special instance is one that runs the SDM (Software Deployment Manager).

A system installed in this way can only process requests to Java applications. Page 11 .IBM Americas Advanced Technical Support • Minimum Cluster Installation The following graphic shows the simplest installation of usage type AS Java. Inc. and Software Deployment Manager (SDM) • The Central Services instance • The Database A Java Instance consists of (with the exception of Central Services): • • a Java Dispatcher One or several server processes © 2007 International Business Machines. a server process. Figure 1-5 Minimum Cluster Installation This minimal AS Java installation consists of: • A Java central instance with a dispatcher.

IBM Americas Advanced Technical Support • Large Cluster Installation The graphic below shows a larger Java cluster installation. and monitored separately. each of which has an instance number and can be started. Inc. It consists of four instances. Page 12 . stopped. Figure 1-6 Large Cluster Installation © 2007 International Business Machines.

Shell Console Administrator. Config Tool and SAP Netweaver Administrator. Visual Administrator enables: o Obtaining general information about a service. Page 13 . cluster elements. SAP J2EE Engine Administration Tools The SAP J2EE Engine provides the following basic administration options: o offline – enables the configuration and administration of J2EE Engine cluster elements when the server is not running o online – enables the administration of J2EE Engine cluster elements at runtime o remote – enables configuration of the J2EE Engine from a distance. Inc. group. It provides remote monitoring and management of managers. manager. SAP Java Server Administration 3. and interfaces working on each element in a single GUI. libraries. interface or library (for example. its name.1. and so on) o Logging on to the SAP J2EE Engine via the Visual Administrator tool o Administrating and changing common properties or those specific to a service or manager o Configuring the global properties o Runtime administration and control © 2007 International Business Machines.IBM Americas Advanced Technical Support 3. • Visual Administrator SAP J2EE Engine Visual Administrator is a graphical user interface (GUI) that enables administration of the entire cluster. services. that is. the administration tool runs on one machine and the server that must be managed is running on a different machine These options are implemented in the administration tools for the J2EE Engine: Visual Administrator. and modules running on them.

the user can access and use all shell commands available on the different J2EE Engine cluster elements. the Telnet shell is opened on the dispatcher. as well as continuous deployment of applications and additional libraries.IBM Americas Advanced Technical Support Figure 1-7 Visual Administrator To start the Visual Administrator. By default. Unlike the Visual Administrator. Inc. it is not a GUI and the runtime control and administration is done using specific commands in the shell language. The Console Administrator enables remote administration through Telnet clients or applets that simulate a Telnet client. The commands are entered on the command line of the console where the cluster element is running. Once connected. Page 14 . This tool enables monitoring of the processes started on the different elements of the cluster and provides an opportunity for prompt and adequate reaction whenever problems occur. • Shell Console Administrator The J2EE Engine Console Administrator is an alternative to the Visual Administrator. enter the following command: “/usr/sap/<SID>/<Instance Name>/j2ee/admin/go.sh” (or go.bat for Win based environment). Use the LSC command to © 2007 International Business Machines.

The JUMP command is available for Telnet administration only. enter the following command: “telnet hostname telnet_provider_service_port” or. as an example: “telnet ibm_node_01 50008”.IBM Americas Advanced Technical Support display all server components with their ID. use the JUMP command and specify the ID of the target element. For example. executing jump 4001 enables the remote administration of a cluster element with ID 4001. © 2007 International Business Machines. component name. The first component displayed is the current one. and type. host. port. To close the Telnet connection type exit on the command line. Inc. To pass over from one component to another. Page 15 . Figure 1-8 Shell (Telnet) Administration To connect to the J2EE dispatcher.

The J2EE Engine GUI Config Tool enables configuration of J2EE Engine cluster elements concerning properties. it connects to the database and scans the server configuration. Figure 1-9 The GUI ConfigTool © 2007 International Business Machines. After installing the J2EE Engine a configtool directory is created. containing a configtool script file. it is not required to have a SAP J2EE Engine running. Page 16 . Inc.sh” (or configtool. To start the GUI ConfigTool enter the following command: “/usr/sap/<SID>/<Instance Name>/j2ee/configtool/configtool. as the Config Tool connects directly to database and applies the changes. The information provided is passed to the corresponding interface that is used for configuration. When running the J2EE Engine Config Tool via a GUI or text-only interface. and exporting the system configuration to an XML file. adding new elements. When working offline.bat for the Win based environment).IBM Americas Advanced Technical Support • ConfigTool The Config Tool enables offline configuration of the J2EE Engine cluster elements. It is XML-based and allows the user to export for later use.

where it is capable of operating an entire system landscape containing ABAP and Java systems as the application platform of SAP NetWeaver. troubleshooting and problem analysis of SAP NetWeaver system landscape Central administration tool available landscape-wide for both Java and ABAP systems. Inc. Web Dynpro runs in a normal web browser Interface allows seamless navigation to other SAP NetWeaver administration tools (User Management Engine. for starting and stopping instances. Page 17 . System Landscape Directory) For Java. The most important advantages of the NWA are: • • No longer need to switch between different tools for administration. The SAP NetWeaver Administrator can be used in a central scenario. is easy-to-use.IBM Americas Advanced Technical Support • SAP Netweaver Administrator SAP NetWeaver Administrator (NWA) is a new Web Dynpro-based tool for administration and monitoring. The NWA unifies the most important administration and monitoring tools both for Java and ABAP systems. the NWA represents a crossover from various expert tools to an integrated. which are difficult to use and integrate • • • • © 2007 International Business Machines. simple and clear solution. offering a central point of entry point into the SAP NetWeaver system landscape. The NWA also completes the integration of the data sources for monitoring For ABAP. the NWA represents a crossover from many different expert transactions. and monitoring error-free functioning of components Interface follows the current guidelines for interface design. checking configuration settings and logs. task-oriented and complete.

© 2007 International Business Machines. open a following http page: “http://<fully qualified hostname>:50000+(100 * instance number)/nwa” or in the following example “http://ibm_node_01:50010/nwa”. Page 18 .IBM Americas Advanced Technical Support Figure 1-10 Netweaver Administrator To connect to the Web Administrator. Inc.

SDM resides in the SAP Web Application Server.SDAs or SCAs . war) and an SAP manifest. It also provides modification support on demand. The SDA contains the manifest information . If an EAR archive contains a SAP manifest. whose quantity describes a precisely-defined version level.that is. • Software Deployment Archive (SDA) The Software Deployment Archive (SDA) is the delivery format for SAP applications in programming languages other than ABAP. Furthermore.of its archives (such as jar. from where it then manages the installed archives.sda. Inc. which contains additional software logistics information. • Software Component Archive (SCA) A Software Component Archive (SCA) is the physical representation of a version of a software component. A SCA update always results in a new version level of the software component. the CBS consults the SDM on the target system. The SDM uses the Deployment Manager to control the deployment of SDAs/SCAs for the following o o o o J2EE applications File system content Database content SAP J2EE Engine Additional Libs © 2007 International Business Machines. When deploying SDAs/SCAs.2. The SDM recognizes the EAR archive as a SDA. package-related data . • Deployment The deployment is the final step in the software delivery process. The SDM is responsible for installing and updating software components into the SAP J2EE Engine. The SDM recognizes dependencies between archives and provides support when installing and maintaining shared applications. The EAR archive is a special case in the J2EE context. This process can be initiated via the SAP NetWeaver Developer Studio. It contains a specific number of SDAs. but does not rename the archive extension as <archive_name>.in the runtime environment of the SAP Systems. SDA is the smallest unit that can be deployed. the SDA is the smallest unit for which patches can be created and delivered. Each time a new component version is deployed. it is also a SDA.IBM Americas Advanced Technical Support 3. the Software Deployment Manager stores the data in the SDM Repository. Page 19 . It is a ZIP-compatible archive format that can be used as a container for other archives. it involves the deployment of available software packages . Software Deployment Manager (SDM) The Software Deployment Manager (SDM) is a tool to deploy and manage software packages from SAP.

Page 20 . Inc. enter the following commands: “/usr/sap/<SID>/<Instance Name>/SDM/program/StartServer.IBM Americas Advanced Technical Support Figure 1-11 Software Deployment Manager (SDM) To start the SDM GUI tool. “/usr/sap/<SID>/<Instance Name>/SDM/program/sdm. © 2007 International Business Machines.sh” (if server not started).sh remotegui”.

Instead of investigating potential configuration problems or performance bottlenecks through individual system logging. while log files and performance data can be accessed from one location to simplify administration. the data are collected and presented in a central monitoring system (CEN). Inc.IBM Americas Advanced Technical Support 4. Figure 1-12 Central Monitoring of the SAP NW04s System Landscape To address the different aspects of Java monitoring. Page 21 . SAP Java Server Monitoring The previous chapters have provided information about the basic architectural elements of the SAP Java landscape. two different task groups have been defined: • Problem detection • Problem analysis © 2007 International Business Machines. The basic Java monitoring architecture is developed and organized centrally. This chapter will cover the different aspects of the Java monitoring techniques. and in most of cases. Problems will be displayed as soon as they occur. will take advantage of the SAP CCMS monitoring architecture. The following tools and techniques should be utilized to achieve a robust and scalable environment through higher availability and optimized performance of the monitored applications.

IBM Americas Advanced Technical Support 4. during typical Java engine activity. performance related to the DB. The log files for administrators contain only the information needed for a broad overview of typical Java engine related activities.1 Techniques for Problem detection Monitoring of the Log and Trace files The first step in the problem detection phase is the search for information contained in the log files generated by the Java engine. Typical elements of interest include status of the heap memory and program stack. SAP has done an excellent job of separating the log files in two groups.) On the other hand. Thus. CPU and memory utilization. System administrators need to scan a wide range of engine related elements to determine current status (overall system availability. Those three configuration elements are: • Log Manager – manages the logging of system events (Figure 1-13) Figure 1-13 Log Manager in Visual Administrator © 2007 International Business Machines.1. Inc. Page 22 . All the important events that describe either the startup or typical workload of the engine are contained in this set of log files.1. etc. 4. developers require detailed information about engine-related activities and applications they are working on. two types of the files are created: • Log files for the administrators • Trace files for the developers The Visual Administrator contains all the necessary tools needed to configure and review Java engine generated files. providing different levels of the information that support the needs of two different types of users: administrators and developers.

Page 23 . is defined by the values of the variables circled in the Figure 1-13. for example: “/usr/sap/SID/JC<InstNo>/j2ee/cluster/dispatcher<No> or server<No>”. Figure 1-14 Log Configurator in Visual Administrator © 2007 International Business Machines. Inc. The benefit is that the Java engine doesn’t have to be started to use this tool. however. the engine has to be restarted. • Log Configurator – used to configure individual logging and tracing settings (Figure 1-14) This element enables the user to manage logs and traces for individual Java components. The location of the generated trace file. all SAP applications come with predefined categories and locations. There are two modes of work on this panel (circled on Figure 1-14): “simple mode” for use on production systems and “advanced mode” for use on development systems. Once the specific set of the parameters has been changed and applied.IBM Americas Advanced Technical Support Each of the generated log files are positioned into a separate “home” directory of the specific node (dispatcher or server). The functionality of the Log Manager is also provided in the Configtool. Even though it is possible to create customized elements (category objects for logs and location objects for traces).

out: o Information about the SDM server process startup and shutdown. Page 24 . exists only on the J2EE central instance © 2007 International Business Machines. Inc. the “Log Viewer” functionality is also provided by other tools: o as a standalone Java application contained in the directory: “/usr/sap/SID/JC<InstNo>/j2ee/admin/logviewer-standalone” (Figure 1-15) o Command line log viewer o Log Viewer functionality integrated in the SAP Netweaver (Figure 1-16) Figure 1-15 Standalone Log Viewer Attached is a list of the most important log & trace files to be reviewed in the problem resolution process.out: o Information about the specific J2EE server process startup and shutdown • std_sdm.out: o Information about the J2EE dispatcher process startup and shutdown • std_server<ServNo>. They are located in the: “/usr/sap/SID/JC<InstNo>/work” directory. Files providing basic information: • INSTSTAT: o Information about J2EE instance startup • log_bootstrap_ID_<InstNo>: o Information about the files that are synchronized during the engine startup o Every server process creates a separate log file • std_dispatcher.IBM Americas Advanced Technical Support • Log Viewer – used to review Java engine generated logs and traces.

IBM Americas Advanced Technical Support Files providing more detailed information: • dev_bootstrap_ID<InstNo> o Detailed information about the J2EE bootstrap process including the corresponding JLaunch processes and startup parameters • dev_dispatcher o Detailed information about the startup and shutdown of the J2EE dispatcher process • dev_server<ServNo> o Detailed information about the startup and shutdown of the J2EE server process and startup parameters • dev_sdm o Detailed information about the SDM server process startup and shutdown. exists only on the J2EE central instance • jvm_bootstrap_ID<InstNo>. Inc. Page 25 . files that are synchronized and synchronization statistics Figure 1-16 Log Viewer in SAP Netweaver Administrator © 2007 International Business Machines.out o Detailed information about the J2EE bootstrap process.

GRMG contains two parts: • GRMG Infrastructure. to send. Inc. The following figures show how to configure and activate the GRMG for the monitoring of the Enterprise Portal infrastructure. receive and display GRMG communication elements in the alert monitor • GRMG application performs the actual application monitoring o Central monitoring system periodically calls a URLs – the GRMG applications o GRMG checks its component or scenario o GRMG application send a XML message back to the CEN o Message received is transferred into a CCMS monitoring tree and displays availability and messages of the component Figure 1-17 Availability Monitoring using GRMG Most current Java based SAP applications (EP.IBM Americas Advanced Technical Support 4.1. Page 26 . BI etc) use the GRMG infrastructure.2 Availability Monitoring of SAP Java engine and Java based applications (GRMG) This technique is based on the so-called “Heartbeat” (GRMG) central infrastructure. © 2007 International Business Machines. which is used to check the availability of the Java engine and Java based applications. XI.

Page 27 . Figure 1-18) Figure 1-18 GRMG customizing in Visual Administrator • • Customize GRMG information in the CEN system using the GRMG transaction (Figure 1-19) Display the availability of the activated scenarios in the RZ20 sub-tree Figure 1-19 GRMG customizing in CEN © 2007 International Business Machines.IBM Americas Advanced Technical Support Required set of the steps are to be executed: • Maintain the GRMG customizing file in the Visual Administrator • Transfer the GRMG customizing file into the CCMS agent (use the “Upload” button. Inc.

) the generated statistics will be transferred to the CEN to which that agent was registered. Detailed explanation of these configuration steps can be found on the SAP Global Help site (http://help. primarily the SAP J2EE engine.IBM Americas Advanced Technical Support 4.1. Figure 1-20 provides an overview of the elements and communication paths required for performance data collection. aggregation and display (Java Standalone System and Double-Stack System): Figure 1-20 Collection. © 2007 International Business Machines. Once the data collection agents are successfully activated (SAPCCMSR with the –j2ee option. Inc. this functionality was extended by using Distributed Statistics Records to monitor the non-ABAP components.3 Distributed Statistics Records (DSRs) SAP Performance Statistics Records are generated to monitor the performance of an ABAP system and its components. Generated statistics records provide information about the system workload and the resources used by a specific action.sap. Recently. Page 28 . Aggregation and Display of the DSR’s To activate the J2EE service to generate statistics records.com). a certain set of configuration activities must be executed in the Visual Administrator.

single or double J2EE stack. with the local ABAP system and the CEN In both types of activation. the SAPCCMSR agent is launched automatically after the J2EE engine is started successfully © 2007 International Business Machines. the agent has to be registered twice. Inc. Page 29 .IBM Americas Advanced Technical Support There are several steps that DSRs go through before they are displayed in the CCMS of the CEN: • • • • Activated DSR service in the J2EE engine starts generating the records Generated records are written to the specific location of the file system containing J2EE relevant objects Activated agent reads the records and sends them to the CEN Raw transferred data can be collected and aggregated by the DSR collector (runs on CEN) and then presented in the Global Workload Monitor (ST03G) (Figure 1-21) Figure 1-21 Aggregated DSR presentation in ST03G • • • Raw transferred data can be presented directly (without aggregation) in the CEN using the STATTRACE transaction. this single record data overview will be used as a functional trace to provide a finer granularity of statistic files (Figure 1-22) In the case of a double-stack system.

Inc.IBM Americas Advanced Technical Support Figure 1-22 Detailed DSR presentation in STATTRACE © 2007 International Business Machines. Page 30 .

etc.4 State Monitoring The most important elements of a system management discipline are the diagnostic tools and procedures that monitor system performance. databases and applications. Figure 1-23 JMX Implementation in SAP Netweaver Figure 1-23) provides an overview of the steps that are executed during the runtime data collection: o Monitored resource (Java application. Page 31 . This chapter will focus on the JMX (Java Management Extensions) monitoring infrastructure implemented in the J2EE engine. the monitored systems can include computers.1.IBM Americas Advanced Technical Support 4. This infrastructure provides a consolidated and automated process to monitor the system state. storage. in addition to SAP Netweaver applications. Within a global corporate IT infrastructure. along with the important parameters of SAP Netweaver and Java applications running on those systems.) creates a monitoring object (MBeans) o Developers are responsible for creating and registering MBeans in the application code o Data collection can be passive (polled by monitor) or active (pushed by resource) o Monitoring objects are registered with the JMX Agent (MBean Server) o MBean server in the SAP Java engine is presented through the JMX Adapter Service o MBean Server manages the MBeans information and makes it available to the JMX clients through the monitor Bean o JMX clients are Monitoring Services in the Visual Administrator and RZ20 in CCMS © 2007 International Business Machines. Inc. networks. JMX is an open Java standard that can be used to monitor all Java-based web applications. service. operating systems. Using JMX will decrease the frequency and granularity of the tasks required to retrieve necessary information about the system status.

Figure 1-24 JMX Adapter Service Global Properties in the Visual Administrator The data generated by the link established between the system (or Java application) and the MBeans are grouped. the local configuration should be applied. Page 32 . Each of these nodes has a predefined group of sub-nodes that consolidates a monitoring structure for a specific application or system resource (MBean). If specific nodes need to be tested. Inc. two different tools are available: Java engine ConfigTool and Visual Administrator (Figure 1-24). The local change has a priority over the global change. A color-coded scheme (red = problem. consolidated and presented in one of the administrator tools (Visual Administrator or CCMS). the global configuration should be applied. Every time the property is changed. Services and System. green = OK) signals down to the root node once a certain threshold is reached. Properties of the service may be changed on the global and local level. Every resource has a predefined threshold which can be changed using the “Configuration” button (Figure 1-25). © 2007 International Business Machines. To control the behavior of this service. Performance. Kernel. If the properties need to be replicated across all nodes. The monitoring tree consists of five separate “nodes:” Applications. the affected nodes have to be restarted. yellow = warning.IBM Americas Advanced Technical Support The JMX Adapter service links the data providers implemented in the shape of MBeans and the JMX clients used to monitor the system (or Java applications).

Inc.IBM Americas Advanced Technical Support Figure 1-25 Visual Administrator Monitoring Service Besides the Visual Administrator. agent registration and configuration has to be executed. Page 33 . © 2007 International Business Machines. the CCMS transaction RZ20 (Alert Monitor) can also be used to present data submitted by the source Java-related elements. Before this is established. One of the benefits of the Alert Monitor is that like Java. The unique value presented through the state monitoring infrastructure is a simplification of the anomoly detection generated in the system and Java applications. it can present the data generated by the ABAP instances.

At this point in time.1 Performance Trace Performance tracing is the “advanced” version of the activities discussed in the description of the DSR collection. Once the cause of the problem is determined.2. Figure 1-26 Data presentation in the STATTRACE and ST03G The functional trace tool (transaction STATTRACE) is used to display raw statistical data (individual records) and traces from the non-ABAP systems. since it presents such an important prerequisite. Techniques for Problem Analysis The previous chapter presented tools and procedures used to detect Java-based system performance and/or configuration issues. Page 34 . the problem analysis phase will start. © 2007 International Business Machines.2. I will assume that the basic knowledge about the system architecture of the monitored environment and the Java programming language is already present. The source (monitored) system must have the Java engine “Distributed Statistics” service enabled. Inc. 4. This section contains a set of tools that will provide detailed information about any system anomaly. The elements are presented in the Figure 1-26. and the CCMS agent SAPCCMSR registered with the Central Monitoring System (CEN). The same performance data collection elements will be used.IBM Americas Advanced Technical Support 4.

a new DSR with a GUID is generated and sent to the SAPCCMSR agent for presentation in CEN. © 2007 International Business Machines. displaying only raw statistical data from the ABAP systems o ST03G (Global Workload Monitor). DSRs can only measure the duration of the processing between the entry points of the “local” J2EE service presented on Figure 1-27. Inc. Page 35 . Every time the LUW is executed on the host. read directly by the functional trace tool from the statistics files accessed by the remote CCMS agent. the following Visual Administrator menu is selected: Performance Tracing Service -> Runtime -> Trace Config. The granularity of the information in the functional trace is finer than that of the data in the Global Workload Monitor. Figure 1-27 DSR’s across Multiple Components It is possible to trace the actions of the business process across several components and system boundaries using the communication element called Passport. If no user is specified. each time a remote call is made. The chosen page will present a set of the functional modules that can be activated to generate a trace for a specific user. .IBM Americas Advanced Technical Support It is an extension of the following two transactions: o STAD. Figure 1-28 provides an overview of the Passport function. a trace is activated for all users. displaying aggregated statistics data collected by the specific collector Data path (C) in Figure 1-26 represents a single record display option for the data. If the business process spans multiple hosts with multiple LUWs. To activate the performance trace. a passport is sent with the previously generated GUID.

Page 36 .IBM Americas Advanced Technical Support Figure 1-28 DSR’s across Multiple Components © 2007 International Business Machines. Inc.

The JARM displays not only the response time. attention should be paid to the Requests per Second counter. JARM data is shown in the Performance Tracing service of the J2EE Engine Visual Administrator tool. Page 37 . The data are presented in several different views: o Request summary This tab displays data accumulated since system start or data reset. JARM monitors must be implemented by the developer into the coding of the application. Figure 1-29 JARM Data in Visual Administrator The JARM data are additionally available in the CCMS and can be displayed using the RZ20 transaction. The most important counters are: Average Response Time of a Request and Number of Requests.2. The purpose of this monitoring is to track the status of the Java system. Figure 1-30 JARM Data – Request Summary © 2007 International Business Machines. This function is designed to allow performance measurement in productive systems and to identify areas causing problems. The data delivers an overview of components passed by a thread.IBM Americas Advanced Technical Support 4. They are very low overhead and can be used during runtime without any restrictions. but also the user who sent the request and the amount of data that was transferred. If system is permanently under load.2 Single Activity Trace / JARM The Java Application Response Time Measurement (JARM) is a method for collecting response time data from Java applications. along with the time remaining in the corresponding component. Inc.

The net time metric identifies the slowest components and the levels indicated how deep component is nested. outbound data. The data can be sorted by average gross time.IBM Americas Advanced Technical Support o Request Overview Most important requests and components may be selected by different criteria such as response time or amount of transferred data. The slowest response time gives an indication of where performance analysis may be necessary. average net time or the number of calls. Page 38 . If one of the requests is chosen. Figure 1-32 JARM Data – Component Overview © 2007 International Business Machines. Figure 1-31 JARM Data – Request Summary o Component Overview Shows the data collected for each component in a condensed way. the detailed information in the pane below is displayed. Inc.

Figure 1-34 JARM Data – Thread Overview © 2007 International Business Machines. The number of threads indicates the current load on the server.IBM Americas Advanced Technical Support o User Overview Shows the username with average response time. Figure 1-33 JARM Data – User Overview o Thread Overview Lists all the threads currently running in applications instrumented with JARM. Page 39 . This helps in locating long running or blocked requests. number of requests and time of the first and last request. Inc. This tab also tracks the time a request is running and the active time of components.

The difference between SAT and JARM is that JARM only provides the top value data. or to understand the logical processing steps inside an application. Figure 1-36 SAT Activation © 2007 International Business Machines.IBM Americas Advanced Technical Support 4.3 Single Activity Trace The Single Activity Trace (SAT) is one usage of Java Application Response time Measurement (JARM). This file can be viewed in the Log Viewer. Inc. Use JARM to analyze a single user’s request to find performance problems in an application. while SAT provides all the trace data generated for the selected request. To write a Single Activity Trace. The performance data collected are automatically written in a trace file using the SAP Logging API. Figure 1-35 SAT Functionality To activate the Single Activity Trace choose: Server -> Services -> Performance Tracing -> Runtime -> Trace Config tab. Page 40 . every component has to be instrumented with JARM.2.

IBM Americas Advanced Technical Support SAT log files are displayed in a format different from general logs: o Steps are displayed in hierarchical manner o Username indicates the beginning of a request or an action o Displays within the structure how deep a component is nested (first column shows the level) Figure 1-37 SAT log files © 2007 International Business Machines. Inc. Page 41 .

2.IBM Americas Advanced Technical Support 4. SQL trace is more suitable tool for developers than for detecting general performance issues. application. Inc. Besides using the “OpenSQLMonitors” Web application. the results of the trace can be reviewed using the Log Viewer. its duration. as well as context information. To launch the SQL Trace enter the URL of the application in your browser: “http://<hostname>:<http_port>/OpenSQLMonitors” Only an administrator has authorization to use this tool.4 SQL Trace SQL trace is a convenient tool for detecting problems when working with the persistent layer. It provides traces for JDBC methods that access the database or are otherwise important for database connectivity. The typical output is presented in the Figure 1-39. Figure 1-38 Accessing the SQL Trace SQL trace can be switched on and off dynamically. input parameters and results. SQL can trace the execution of SQL statements in both system and application threads. Page 42 . To evaluate the SQL Trace follow the Web path: “Main Page” -> “Trace Evaluation” -> Evaluate the selected trace for the specific node. © 2007 International Business Machines. The SQL Trace can also be activated or deactivated using the Visual Administrator. The log format is database independent. Besides the SQL statement text. It is not recommended to keep constantly activated as it consumes additional resources. each log record contains information about the time when the statement was executed. method and session in which the statement was executed. The function provides detailed information about traced statements such as a thread. Follow the Web browser path to turn the SQL Trace on and off (it is turned off by default): “Main Page” -> “Switch on and off SQL Trace” -> “SQL Trace on or off”.

© 2007 International Business Machines.htm If additional information is needed.10 kernel based NW.com/saphelp_nw2004s/helpdata/en/24/d4ff0eccf33a49a5c26a280a119b14/content. SQL Trace does not offer the functionality and flexibility of the equivalent ABAP ST05 transaction. Page 43 .IBM Americas Advanced Technical Support Figure 1-39 SQL Trace output For a detailed description of the data columns in the presented sample output. Figure 1-40 Detailed SQL Trace output At this time. Inc.sap. but further improvements are expected. especially with the upcoming 7. click on the specific SQL statement and the detailed output will be presented. please use the following link: http://help.

Page 44 . For every method. When the application is setup in the “bytecode-modified” mode.5 Application Trace Application Tracing is aimed primarily for the developers. the application can be started again in the “normal” mode. “Invocations”. The runtime control provides everything needed for proper application tracing. This can be done without putting the VM into the debug mode or redeploying the application. “Application” sub-tab – area provides a list of the applications that are deployed on the server.2. no instrumentation is required from the application developer. follow the path: Server -> Services -> Performance Tracing -> Runtime -> Application tracing tab (Figure 1-41). it is restarted and its new class loader knows there are changes to be made in the bytecodes. © 2007 International Business Machines. Once the debugging is over. Inc. Figure 1-41 Application Trace Menu in Visual Administrator Sub-tabs “Application”.IBM Americas Advanced Technical Support 4. To access the runtime control in the Visual Administrator. Right mouse button click on the specific application will provide an option for the apps restart in the “bytecodemodified” or “normal” mode (Figure 1-42). “Objects” and “Plug-ins” are used to control the tracing activities for the application. The application therefore must be started in the “bytecode-modified” mode. It allows the user to see all method invocations during a process flow. the used time is measured in microseconds. To use this tracing method. “Classes”.

Page 45 . Figure 1-43 Classes Sub-tab “Invocations” sub-tab presents the methods invoked. Inc. Figure 1-44 Invocations Sub-tab © 2007 International Business Machines. To check the content of the loaded classes. Figure 1-44.IBM Americas Advanced Technical Support Figure 1-42 Application Sub-tab “Classes” sub-tab presents the Java bytecode-modified classes loaded. the action from the Figure 1-43 has to be executed (“View Source”). combined with the length of the execution timed in microseconds.

Page 46 . Figure 1-46.IBM Americas Advanced Technical Support “Objects” sub-tab provides a more detailed overview of the Java object graph on the target virtual machine. Inc. Figure 1-45. Figure 1-46 Plug-ins Sub-tab © 2007 International Business Machines. and support paged views of arrays. show the inheritance of classes. Figure 1-45 Objects Sub-tab “Plug-ins” sub-tab provides the information about the “bytecode-modified” plug-ins. It can display the values of the fields of every object.

2. © 2007 International Business Machines. how and which set of performance techniques should be utilized to properly locate and repair application anomalies on the system? A daily review of the system can be conducted using the following set of the tools: o Application availability information is provided using the GRMG instrumentation (RZ20) o DSR statistics in the ST03G (CEN) provide a basic set of the system performance related data o State Monitors in Visual Administrator (or CCMS in CEN. o Properly configured Log Monitoring is another valuable source of the system status related information. Inc.6 Conclusion Figure 1-47 When to use different performance tools (1) After reviewing the various problem detection and problem analysis techniques. RZ20) present the state of the Java engines. this paper will examine the following: When. Page 47 .IBM Americas Advanced Technical Support 4.

an initial set of the problem analysis techniques will be executed: o Performance trace using the STATTRACE transaction o Single Activity Trace using the Visual Administrator and Log Viewer o SQL Trace if anomaly was discovered in the JDBC layer Once the problem has been properly discovered and described.IBM Americas Advanced Technical Support If problem is detected using one of the previously mentioned steps. Page 48 . the application developers will determine the problem solution path primarily using the Application Trace and Application Debugging functionality in the Visual Administrator and SAP Netweaver Developer Studio. Inc. An overview of the previously described steps is presented in Figure 1-48. Figure 1-48 When to use different performance tools (2) © 2007 International Business Machines.

Page 49 . Inc. it provides a “heap memory recycling” mechanism. it is recommended to use the new IBM 1. It is called a garbage collection and.2 Hybrid JVM with J9 technology. It is important to state that the application of the recommendations from the following chapters should only be executed after careful review of the specific group of the SAP OSS notes. in the simple words.4. Additionally. we will provide the overview for one of the most important activities executed inside the JVM working area.2 implementation as an underlying JVM mechanism.4. there is a set of the OSS notes that should be reviewed before the recommendations are applied: o If planning to a new installation or an upgrade of SAP NW2004 SR1 or SAP NW2004s SR1. To check the details of the announcement. Following paragraphs provide detailed description of the IBM JDK prerequisites and configuration details for the successful Java based SAP Netweaver implementation (from the SAP Web AS perspective). enhances memory management and improves debugging capability. It provides more consistent runtime behavior. follow the instructions in SAP Note 1008311 o For SAP NW2004 SP14 and above or SAP NW2004s SP6.2 for the application stacks of SAP Netweaver 2004 and SAP Netweaver 2004s. but the good starting point is the OSS note 716927.IBM Americas Advanced Technical Support 5.2 Hybrid JVM and it is available for NW2004 and NW2004s on AIX 5L. © 2007 International Business Machines. It is also 100% upward compatible for all SAP business applications. IBM Java Technology for SAP As mentioned before. SAP J2EE Engine utilizes IBM JDK 1. IBM recently exchanged the JDK version 1.4. The new JDK version is called 1. follow the instructions in SAP Note 1024700 o For new installations of SAP NW2004s SR2. follow the instructions in SAP Note 1024539 Before describing the IBM JVM parameters. check the SAP OSS Note 1039948.4. Most of these notes will be mentioned in this chapter.

Garbage collection can then begin. the Garbage Collector can consider compacting the resulting set of objects to remove the spaces between them. the length of the object at the start of the sequence is examined to determine the amount of free space to be released. in turn. The mark vector must be a subset of the allocbits vector. the bitsweep technique avoids the need to scan objects that are in the heap and therefore avoids the associated overhead cost for paging. the mark vector contains a bit for every reachable object that is in the heap. Because compaction can take a long time. the appropriate allocbit and markbit was tested to locate the garbage. which probably identify free space. the second part of the process is to scan each object for other references it makes. The first task of the Garbage Collector is to collect all the garbage that is in the heap. o Sweep phase After the mark phase. This process starts when any thread calls the Garbage Collector. all objects that are referenced from the thread stacks. Each of those objects can. It occurs in three phases: o Mark o Sweep o Compaction (optional) o Mark phase In the mark phase. When such a long sequence is found. and visit each object in turn. Page 50 . a rare event. interned strings and JNI references are identified. and the Garbage Collector is invoked. or directly by a specific call to System.1 IBM JVM Garbage Collection When the JVM cannot allocate an object from the current heap because of lack of space. Now. © 2007 International Business Machines. Inc. either indirectly as a result of allocation failure.IBM Americas Advanced Technical Support 5. All the other threads are then suspended. These two processes together generate a vector that defines live objects. The length of each object was held in the text that immediately preceded it on the heap. the Garbage Collector tries to avoid it if possible. which is allocated from the heap and later used by an individual thread to meet any allocation requirements. This step ensures that other threads are not suspended while they are holding critical locks. At each object. o Compaction phase When the garbage has been removed from the heap. Compaction is. The first step is to retrieve all the locks the garbage collection process needs. This action creates the root set of objects the JVM references. objects that have been allocated but are no longer referenced. therefore.gc(). reference others. statics. Therefore. a memory allocation fault occurs. Objects are not normally allocated from the heap itself but from thread local heap. In the bitsweep technique. the mark vector is examined directly to look for long sequences of zeros (not marked). that is. The task of the sweep phase is to identify the intersection of these vectors. The original technique for this sweep phase was to start a scan from the bottom of the heap.

3 Monitoring the JVM activity The initial set of the steps suggested to be used to check the status of the JVM (or the SAP JLaunch process) are based on AIX system performance tools. all Garbage Collection activities are logged into a file named standard server out (or std_serverX.out.IBM Americas Advanced Technical Support 5. o -Verbose:gc. Page 51 . nmon. Handles short-lived objects differently than objects that are long-lived. o -Xdisableexplicitgc. vmstat.) where X is the number of the corresponding server process (Jlaunch ID. ps and svmon will provide a basic overview of process related activity. © 2007 International Business Machines.2 IBM JVM parameters The following is an overview of the Hybrid JVM related parameters used in the previously mentioned OSS notes. one of the four garbage collection policies. this option is used to obtain information about the Java Object heap in real time while running Java applications. This list is a subset of all parameters available. Inc.) o -Xgcpolicy:gencon. o -Xmx<size>. In some cases. the previously explained Verbose:gc JDK option must be utilized. No parameters for the “old” JVM version will be discussed. sets the initial memory size.gc call Another example is included to show how the content of the std_output file is changed if heap memory allocation happens. Applications that have many short-lived objects can experience shorter pause times with this policy while still producing good throughput. When this option is used.gc to request garbage collection. o -Xms<size>.gc call. o -Xmn<size>. switch that tells the JVM not to run in the “classic” mode. To view detailed information about the JLaunch process linked with the IBM JVM. Figure 1-49 Verbose:gc content as a result of the System. 5. If the default values of the presented parameters need to be changed. This is a sample file simulating the content generated using a System. sets the maximum memory size. it must be done under the close supervision of either the SAP and/or IBM support service. these calls can degrade performance time through premature garbage collection and compactions.) Every server process has its own std_output file that resides in the “/usr/sap/<SID>/JCXX/work” directory. disables the explicit calls to System. Topas. This call is used to manually initiate a Garbage Collection. The content of the simplified sample std_output file is presented in Figure 1-49. o -Xj9. Figure 1-50. sets both the values of the parameters –Xmns (the initial size of the new space) and –Xmnx (the max size of the new space.

“Pattern Modeling and Analysis Tools for Java Garbage Collector”.ibm.IBM Americas Advanced Technical Support Figure 1-50 Verbose:gc content as a result of the Heap memory allocation failure Browsing the content of the sample file is a complex and time consuming process. analysis of Java heap usage. The tool that I use most often is called PMAT. PMAT can be downloaded from the IBM AlphaWorks Web site: http://www. Figure 1-51 Initial screen of the PMAT tool © 2007 International Business Machines. Page 52 . Inc.com/tech/pmat. and for recommending key configurations based on pattern modeling of Java heap usage. PMAT is used for parsing verbose GC traces. Several different free tools exist that can help present the GC generated data in a much more understandable way.alphaworks.

Inc.IBM Americas Advanced Technical Support One of the best options in the PMAT is the graphical overview of the JVM Garbage Collection related activities. I didn’t cover the specific tuning challenges around the most popular SAP solution. Page 53 . © 2007 International Business Machines. presented in the Figure 1-52: Figure 1-52 PMAT tool (Graphical overview of the GC activities) This chapter provided a basic overview of the IBM JVM tuning steps around the Garbage Collection process. Enterprise Portal since a document about it already exists and is mentioned in the Publications Appendix.

com Developer Works (IBM’s Resource for Developers) http://www-130.ibm.boulder.com/support/techdocs/atsmastr.3 and Power5”.IBM Americas Advanced Technical Support Publications: The publications listed in this section are considered particularly suitable for a more detailed discussion of the topics covered in this white paper.com SAP Help Portal http://help.ibm.com/infocenter/pseries/index.com/support/techdocs/atsmastr. Page 54 .ibm.sap.nsf/WebIndex/WP100740 “Tuning SAP R/3 with Oracle on pSeries”.jsp © 2007 International Business Machines. Inc. Publications • • “SAP Enterprise Portal on AIX 5.ibm. 1.nsf/WebIndex/WP100377 2. Online Resources • • • • • SAP Service Marketplace http://service. Mark Gordon (IBM) http://www-03.com/java IBM Infocenter for AIX: http://publib.ibm.com/developerworks/java/ AlphaWorks (Java Technology) http://www. Walter Orb (IBM) http://www-03.alphaworks.sap.