This action might not be possible to undo. Are you sure you want to continue?
0 End to End Root Cause Analysis
Powered by SAP NetWeaver 04s
Troubleshooting Guide Wily Introscope
Document Version 1.21 – October 2008 Valid starting with SP13
SAP AG Neurottstraße 16 69190 Walldorf Germany T +49/18 05/34 34 24 F +49/18 05/34 34 20 www.sap.com
© Copyright 2006 SAP AG. All rights reserved. No part of this publication may be reproduced or transmitted in any form or for any purpose without the express permission of SAP AG. The information contained herein may be changed without prior notice. Some software products marketed by SAP AG and its distributors contain proprietary software components of other software vendors. Microsoft, Windows, Outlook, and PowerPoint are registered trademarks of Microsoft Corporation. IBM, DB2, DB2 Universal Database, OS/2, Parallel Sysplex, MVS/ESA, AIX, S/390, AS/400, OS/390, OS/400, iSeries, pSeries, xSeries, zSeries, z/OS, AFP, Intelligent Miner, WebSphere, Netfinity, Tivoli, and Informix are trademarks or registered trademarks of IBM Corporation in the United States and/or other countries. Oracle is a registered trademark of Oracle Corporation. UNIX, X/Open, OSF/1, and Motif are registered trademarks of the Open Group. Citrix, ICA, Program Neighborhood, MetaFrame, WinFrame, VideoFrame, and MultiWin are trademarks or registered trademarks of Citrix Systems, Inc. HTML, XML, XHTML and W3C are trademarks or registered trademarks of W3C®, World Wide Web Consortium, Massachusetts Institute of Technology. Java is a registered trademark of Sun Microsystems, Inc.
SAP, R/3, mySAP, mySAP.com, xApps, xApp, SAP NetWeaver, and other SAP products and services mentioned herein as well as their respective logos are trademarks or registered trademarks of SAP AG in Germany and in several other countries all over the world. All other product and service names mentioned are the trademarks of their respective companies. Data contained in this document serves informational purposes only. National product specifications may vary.
These materials are subject to change without notice. These materials are provided by SAP AG and its affiliated companies ("SAP Group") for informational purposes only, without representation or warranty of any kind, and SAP Group shall not be liable for errors or omissions with respect to the materials. The only warranties for SAP Group products and services are those that are set forth in the express warranty statements accompanying such products and services, if any. Nothing herein should be construed as constituting an additional warranty.
Disclaimer Some components of this product are based on Java™. Any code change in these components may cause unpredictable and severe malfunctions and is therefore expressively prohibited, as is any decompilation of these components. Any Java™ Source Code delivered with this product is only to be used by SAP’s Support Services and may not be modified or altered in any way.
These are words or characters that you enter in the system exactly as they appear in the documentation. SELECT and INCLUDE. pushbuttons labels. graphic titles.T ypographic Conventions Type Style Example Text Represents Words or characters quoted from the screen. Keys on the keyboard. screen titles. These include report names. for example. Cross-references to other documentation. transaction codes. Angle brackets indicate that you replace these words and characters with appropriate entries to make entries in the system. source text. names of variables and parameters. messages. These include field names. table names. upgrade and database tools. Icons Icon Meaning Caution Example Note Recommendation Syntax Example text EXAMPLE TEXT Example text Example text <Example text> EXAMPLE TEXT . Variable user entry. Technical names of system objects. and key concepts of a programming language when they are surrounded by body text. Output on the screen. menu paths. F2 or ENTER. and menu options. for example. Emphasized words or phrases in body text. and table titles. program names. Exact user entry. This includes file and directory names and their paths. and names of installation. menu names.
.......................... 12 SOLUTION MANAGER DIAGNOSTICS INTEGRATION ......... 22 Wily Introscope I ......... 16 ENTERPRISE MANAGER SIZING AND TUNING ................................ 1 3 4 5 6 7 8 9 WEBVIEW ON NETWEAVER TROUBLESHOOTING .............................. 5 JAVA BYTE CODE AGENT TROUBLESHOOTING .....Contents October 2008 Contents 1 GETTING THE SOFTWARE AND DOCUMENTATION ........................................................................................ 12 HOSTAGENT PROBLEMS .................................................................................................................................................................................................................... 14 MISSING DATA PROBLEMS....................................................................................................................................... 8 EPAGENT INSTALLATION TROUBLESHOOTING ................................... 20 10 USING INTROSCOPE............................................... 1 2 ENTERPRISE MANAGER AND WEBVIEW INSTALLATION TROUBLESHOOTING..................................................................
1p9 Enterprise Manager may fail. Go to the Advanced tab Performance Settings Data Execution Prevention.2 Q: Where else can I find information on Introscope? A: Check http://service. There is an FAQ section on Introscope which will not be replicated here.1) A: The installation wizard is considered as virus by the Windows 2003 virus protection. You will also find references to SAP-internal training material here. A2: The property USER_INSTALL_DIR in file installer. 7. Workstation. The Agents can have the same version as the EM or an older version.properties has trailing blanks.1 Q: Where do I get info about Introscope installation and software download? A: SAP Note 797147 has all the needed info about downloading Introscope from Service Market Place including installation instructions and end user documentation. 4.0 vs. 2 Enterprise Manager and WebView Installation Troubleshooting 2. Perform the following steps to allow Windows to trust the Introscope installer: 1. Starting with Introscope 7. Please remove them. Wily Introscope 1 .1 Q: Installation Wizards Terminates Immediately on Windows 2003. Click Add.End to End Root Cause Analysis October 2008 1 Getting the Software and Documentation 1. but they cannot be newer than the Enterprise Manager. Enterprise Manager. Please do not edit this property. (Note: this should not happen any more for versions starting with 7. 1.2 Q: Installation Wizards Terminates with NullPointerException A1: The file installer. additional information is available in SAP Note 792999.1? A: You should always use the latest version of Introscope (Agent including instrumentation. Browse to the Introscope installer and add it to the list. Enterprise Manager. For SAP internal purpose. This allows you to upgrade the Enterprise Manager at any time you want and after this to upgrade the bytecode agents in the monitored systems whenever a maintenance window allows this. Right-click My Computer and choose Properties.properties is not set to silent. and WebView must even have the same patch level.1p5 Workstation to a 7. 2.properties is not in the same directory as the installer. The easiest way to ensure version compatibility is to launch the Workstation via WebStart (http://emhost:8081/workstation) or to launch WebView from the Enterprise Manager (http://emhost:8081/webview).com/diagnostics on the Service Marketplace. Restart Windows and run the Introscope installer. 1.1p5. 2.sap. A3: The property INSTALLER_UI in file installer. Enterprise Manager and WebView or Workstation must be of the same version. 3. Connecting a 7.3 Q: Version Compatibility 7. and WebView).
The columns Performance. Wily Introscope 2 . 8/16/07 08:21:50 PM CEST [INFO] [Manager] CA Wily Introscope(R) Version 7. Check the EM log file (under the logs directory in the EM installation). Depending on the situation.txt in the EM directory. The error message will look like [ERROR] [Manager. 8/16/07 08:21:50 PM CEST [INFO] [Manager] Using Introscope installation at: d:\usr\sap\ccms\wilyintroscope\..6 Q: The Enterprise Manager seems to run very unstable / seems to have OutOfMemory problems. change the EM ports (please remember to make a similar change on the agent and workstation side. 2.xml 2. The first line shows the release and patch level of the Enterprise Manager installation: 8/16/07 08:21:50 PM CEST [INFO] [Manager] Introscope Enterprise Manager Release 7. Screenshots from the problem.NumberOfAgents and Performance. If so.Agent.0_08" from Sun Microsystems Inc. check that you have the latest agent instrumentation (ISAG*.Agent. All files in this directory are needed. Address already in use: JVM_Bind If the EM had a problem with a management module you will see an error in the EM log file. please open a customer message for component XX-PART-WILY.NumberOfMetrics are of interest. log/ directory.1. If EM could not open/listen to one of the ports. A: A common source for problems is an older Introscope agent on an Oracle-based Netweaver installation.4 Q: What files/tools are important to troubleshoot an Introscope Enterprise Manager installation? A: The following items should at least be collected.SAR).1 2. This number is typically less than 5000 metrics per agent. what do I do? A: This can be caused by a couple of things: It is possible that the ports that the EM is using to connect to the agents or the workstation are already taken or that one of the management modules has problems. you can try to copy and use the module from the config/modules-backup directory or recopy the original management module jar file from service Marketplace.5.3 Q: How can I determine the version of my Enterprise Manager? A: Check the first few lines of the logfile config/IntroscopeEnterpriseManager.log. If the number of metrics is beyond 80000 and only a few agents (<10) are reporting. install/ directory. In some cases. the contents of the complete /data directory may be needed.P9 (Build 292911) 8/16/07 08:21:50 PM CEST [INFO] [Manager] Using Java VM version "Java HotSpot(TM) Server VM 1. The error message will look like [ERROR] [Manager.Bootstrap] XML problem while parsing d:\usr\sap\wilyintroscope\config\modules\ManagementModule. All files in this directory are needed.Environment] Failed to establish RMI traffic on port 6001. Check the file perflog.5 Q: My Enterprise Manager is not starting.End to End Root Cause Analysis October 2008 2. additional files may be requested.
A: You have to restart the Enterprise Manager after adding the modules. you possibly have to increase the Java heap for the Enterprise Manager (-Xmx option). As preparation. 2. 2.IOException: Cannot locate antRun script: Property 'ant.io.ipaddress). In case of a virtual host environment you may have to replace localhost with the virtual host name that the Enterprise Manager is listening to (property introscope. but they are still not loaded by the Enterprise Manager.exec.installer. /usr/sap/ccms/wilyintroscope/lib). ISC_HOME=/usr/sap/ccms/wilyintroscope JAVA_HOME=$ISC_HOME/jre JAVA_OPTS="-Duser=Admin -Dpassword=Admin89 -Dhost=localhost Dport=6001" $JAVA_HOME/bin/java $JAVA_OPTS -jar $ISC_HOME/lib/CLWorkstation.bat to remove the Windows Service Launch the Introscope Uninstall Wizard (if available) or just drop the EM installation directory. If the EM is running as service (default on Windows).xml:307: Execute failed: java. use the Windows “Services” Control Panel to restart the Enterprise Manager. This will lead to corruption of the historical data that is stored in the /data directory! The other option is to use the so-called Introscope command line workstation. Do not use kill -9 (kill -SIGKILL) to terminate the EM.10 Q: Non-Fatal error for Enterprise Manager Installation – Do I have to care about it? A: The installation log may show an error message similar to the text below.End to End Root Cause Analysis October 2008 If the number of metrics is beyond 80000 and more than 10 agents are reporting.7 Q: I installed the management modules into config/modules. Then use the following shell command to shut down the EnterpriseManager.jar from the workstation installation to the lib directory of the Enterprise Manager (e. please copy the file lib/CLWorkstation. You can ignore this error.g.enterprisemanager.ant15. 2. Additional Notes: ERROR file:/usr/sap/ccms/wilyintroscope/install/em.home' not found Wily Introscope 3 .8 Q: How can I stop the Enterprise Manager on Unix? A: There are two options to shutdown the Enterprise Manager on Unix. One option is to use the kill command to send termination signal to the java process running the Enterprise Manager (kill <PID> or kill -SIGTERM <PID>).9 Q: How can I uninstall the Enterprise Manager A: Procedure for Windows: stop the Enterprise Manager call DeRegisterEMService.jar shutdown 2.
even if an Internet connection is available.awt. 2.bin 2. perl -pi -e 's/export LD_ASSUME_KERNEL/#xport LD_ASSUME_KERNEL/' introscope7. On some systems using the Sun Java VM it was sufficient to set the parameter -Dswing. Install the JRE 5 manually on a machine where you want to launch the Workstation via WebStart or use the explicit installation of the Workstation. the only work-around is to set the VM parameter Djava. 2.13 Q: Installer fails on SuSE Linux SLES 10 This is a problem in the Install Anywhere installer that is used by Introscope. Solution 1: Use the platform independent installer to install the Enterprise Manager (See installation guide).html and install it. By default. If there is no valid X server on the machine.sh or Introscope_Enterprise_Manager.15 WebView just displays empty frames WebView requires the Adobe SVG plugin to display data. use the Xvfb dummy X server.HeadlessException. There is also an error message in the log file IntroscopeEMWebView.End to End Root Cause Analysis October 2008 2.awt. the browser opens a popup asking about the download of the SVG plugin. you may get the error message "An error occurred displaying the dashboard".log: [ERROR] [WebView] java.1. Solution 2: Modify the installer by issuing the following command on the installer::.nohup.sh) If this does not help. If you have any popup-blocker active or any other restriction keeps you from automatic installation of the SVG plugin. The problem is caused by bugs in the headless mode of the Java VM. Xvfb is normally provided as optional package for the operating system.14 The Console Lens sometimes does not work in WebView on Unix: "An Error occurred displaying the dashboard. download it manually from http://www.P4linuxAMD64SAPEndUser.com/svg/viewer/install/mainframed.12 Q: Launching the Workstation via WebStart Fails The workstation requires a Java 5 VM to run. Otherwise the logoff from the X Windows server may also terminate the Enterprise Manager (which acts as X Windows client). For some reason the JDK is not installed automatically if it is not in place.lax) and to set the DISPLAY environment variable of the user that launches the Enterprise Manager to a valid X display. Try setting a different temp directory: IATEMPDIR=/your/free/space/directory export IATEMPDIR 2.headless=false in the file containing the VM parameters for the EM (on Unix in runem.11 Q: Installation Fails due to Unsufficient Space in TEMP If you get the following warning: WARNING: /tmp does not have enough disk space! Attempting to use /home/sidadm for install base and tmp dir.volatileImageBufferEnabled=false in the launch script (runem.nohup. 2.adobe.16 Failed to start SNMP Agent You may see the following error message in the log: Wily Introscope 4 . Note that the X server defined via the DISPLAY variable should not be connected to an interactive user session." A: When setting the Console Lens in WebView on some Unix platforms.
No effect for data that is already stored in BI.data: illegal footer Detailed solution for Trace DB (/tracesdirectory): Shut down the Enterprise Manager and reindex the trace database with the following command (traces is the directory containing the trace database): IndexRebuilder[.enable=false 2.io.snmp. To change the WebView log level: Wily Introscope 5 . make sure you always shut down the Enterprise Manager normally.IOException: Invalid header in data/archive/1217894400000. To avoid this behavior. in particular on 64-bit Windows. corrupted trace files can be repaired by a reindexing run.sh] traces To avoid data file corruption. Then delete each file mentioned in the log as corrupt in the file system and restart the Enterprise Manager: 8/08/08 09:21:17 AM GMT [ERROR] [Manager. 3.wily. Quick resolution: Delete the directories /traces and /data (default locations). What Information do you need to support? A: Set the WebView log level to DEBUG and additionally trigger at least 5 full thread dumps while WebView is hanging. Also check Q2.8 for a scriptable shut down procedure on Unix.introscope. these corrupt files may prohibit startup of the Enterprise Manager or lead to an immediate shutdown after a start attempt. add the Java VM option -Xrs to EMService.SnmpException: SNMP Agent failed to start on port "161" using "Introscope-xxx" MIB. Then restart your Enterprise Manager.End to End Root Cause Analysis October 2008 4/08/08 05:30:35 PM CEST [ERROR] [Manager.server.spec. You can ignore this error message. 2.bat|.data to reperiodize it: java. Corrupted data files cannot be repaired and should be deleted.conf and restart the Enterprise Manager. To get rid of the message. Detailed solution for SmartStor (/data directory): check the log file IntroscopeEnterpriseManager.snmp.SnmpService] Failed to start SNMP Agent com.toolkit. In some case.enterprisemanager. The side effect is that you will lose all historical data that is stored in the Enterprise Manager.SnmpTransportException: Address already in use: Cannot bind The SNMP adapter is not needed for normal operation.SmartStor] Can't query data/archive/1217894400000.1 Q: WebView on Netweaver is Hanging / returns Errors.properties and change this property: introscope.beans.engine. 3 WebView on Netweaver Troubleshooting Note: It is not recommended any more to run WebView on Netweaver.snmp.log for entries like the following. the Enterprise Manager is terminated whenever the user logs off from the console. Please use this one instead of WebView on Netweaver.18 Enterprise Manager terminates on Windows On some Windows systems.17 Enterprise Manager Corrupted Data Files When the Enterprise Manager is terminated abnormally. edit the file config/IntroscopeEnterpriseManager. The Enterprise Manager contains a WebView installation in its internal web server that is ready-to-run without any deployment or configuration steps. Caused by: monfox. some of the files which are used for historical trace and metric data may get corrupted.
6 Q: WebView on NetWeaver: Transaction Tracer does not work A: There is a compatibility problem between the latest versions of WebView and the SAP J2EE server.g. or.End to End Root Cause Analysis October 2008 3. There seems to be a connection problem between WebView and Enterprise Manager.log to SAP. 5. 3. Retry loading webview in a new browser session 6.properties in a text editor 2.wily.WebView as follows: log4j. if you do not know how to do that.logger. 1.g.Log4JSeverityLevel.com\com. Either use WebView running on the Enterprise Manager or open a message to receive a fixed ear file.WebView=DEBUG#com. Restart the webview application. Please send the log file WebView log file e.5 Q: WebView on NetWeaver does not work on AIX A: WebView on AIX is only supported running in the EnterpriseManager. C:\usr\sap\SID\JC00\j2ee\cluster\server0\IntroscopeWebView.util. Change the value of the property log4j.logger. 3. restart the complete J2EE node which runs WebView. Open the file WebView properties file e. WebView running on Netweaver is not supported for AIX.wilytech.logf ile 3.4 Q: WebView on NetWeaver cannot connect to the Enterprise Manager: “This WebView instance is connected to the Enterprise Manager on unconnected” A: Please follow this procedure to obtain more verbose log information from WebView. C:\usr\sap\SID\JC00\j2ee\cluster\server0\apps\wilytech.webview\ser vlet_jsp\webview\root\WEB-INF\classes\log4j.feedback. Wily Introscope 7 . Save the file 4.
Server node details and Current VM Parameters show you the current configuration.g.agent. via the SAP MMC snapin (developer trace) or via the File System Browser of SMD. One of them is present if the VM is instrumented with Introscope and the path to the Agent. Typical installation directories are: /usr/sap/ccms/wily for manual installation /usr/sap/SMD/Jxx/SMDAgent/applications.End to End Root Cause Analysis October 2008 4 Java Byte Code Agent Troubleshooting 4. agent version. including log file location. The screenshot below shows a typical example.wily/ISAgent/ISAGENT71_13_0/wily for automated installations via Solution Manager Diagnostics If the monitored solution is maintained in SMD. Wily Introscope 8 .sap. and agent installation directory (derived from VM parameter). you can also use the SMD Introscope setup application to find out the Introscope configuration: Go to Diagnostics Setup Managed Systems Introscope Agent and select the J2EE node of interest.config/com. If you are not sure where the agent is located.smd. e.1 Q: How can I Determine the Agent Installation Directory? The agent installation directory contains the main agent file (Agent. To find out the current Java VM parameters of a SAP J2EE server node.jar passed via parameter clearly identifies the Agent installation directory.4).app lication.jar) and all important configuration files. check the log file /usr/sap/sid/instance/work/dev_server0 . check the Java VM parameter -javaagent (Java 5) or -Xbootclasspath (Java 1.
e. 2.1.g. you have to apply special options when installing the Introscope agents.End to End Root Cause Analysis October 2008 4. re-creating the connector is still recommended.4 Q: What files are important to troubleshoot an Introscope Agent Installation? Agent log files as described above (IntroscopeAgent.o01. Under the agent.20).P8 (Build 292801) 4. The files are called dev_server0 and std_server0.out (assuming server0 is the J2EE node name). Jxx or DVEBMGSxx) or /usr/sap/SID/j2ee/j2ee_xx/os_libs/work (6. e. the agent log file may also be located in the working directory of the VM.agentname.2 Q: Where can I find log files from the Java Byte Code Agent? There are two important agent log files to be collected: IntroscopeAgent. As a special case.x). instance may be JCxx. By default these log files are located in the subdirectory logs/ of the agent installation directory (see previous question). You need to find the files which were written when the problem occurred. via the Workstation: Open the Investigator (Workstation Investigator) and navigate to the agent of interest. Nonetheless.1. What do I do? A: Every time you change/upgrade the JDK you have to recreate the so-called connector to fit the new JDK version.log). Please refer to the “Creating an AutoProbe Connector” section in the installation guide. Reasons for problems are usually clearly stated in these files. agent log entries can be found in std_server0.40 and up. now my server is not starting up any more. they might be called std_server.3 Q: How can I determine the version of the Byte Code Agent? There are two common ways to determine the agent version: 1. they are located in /usr/sap/SID/instance/work (6. select the node Agent Stats|Build and Release. From the log file: In the log file IntroscopeAgent.log search for a line similar to the line below. Starting with Introscope 6. the agent log directory has been moved to /usr/sap/SMD/Jxx/SMDAgent/temp/wily/logs in order to overcome permission problems on Unix.out. create a subdirectory logs/ under the agent installation directory and assign write permission for group sapsys to this directory. the JDK version check is less strict and failing to re-create will no longer prohibit the startup of server nodes.g. Please check SAP Note 1015184. Wily Introscope 9 .log and Autoprobe. In case the agent has problems to write log files or other severe problems. Release 7. the log files may be missing due to permission problems. The value of this node shows the patch level of the agent.5 Q: I have changed/upgraded my JDK/JVM.log. 4. It shows the agent release including patch level: 6/05/07 01:38:31 PM CEST [INFO] [IntroscopeAgent] Introscope Agent Release 7. For old releases of the agent (6.agentname.log and Autoprobe. If the agent was installed via Solution Manager Diagnostics <= SP12 on Unix.P9 (Build 292911). To resolve the problem. If the startup framework is used. 4.agentname. Logs containing the output of the Java VM. if you upgrade your IBM VM on AIX from classic to J9 hybrid. For installations done via the Solution Manager Diagnostics starting with SP13. /usr/sap/SID/Jxx/j2ee/cluster/server0.o00 or std_server.1.
7 Q: How do I prevent type errors when putting the Introscope parameters in the configtool or service. Wily Introscope 10 . what do I do? A: Please trigger at least three full thread dumps on the monited system that seems to hang due to the agent and provide the log files listed in Question 4.5: There is no longer an Autoprobe connector. make sure you use semi-colon (“.“) on Windows and colon (“:“) on UNIX. 3. In the server log file you will be able to see what is “java home“ or “java path“ and use this path with the –jvm flag when you create the connector. Make sure the connector you created is pointing to the right JVM (in some cases you will also see an error in the SAP logs saying something like AutoProbe is expecting one JVM version is expected and another is being used. e. /usr/sap/SID/JCxx/j2ee/cluster/server0. Verify what java version is being used to run the J2EE engine. including Windows.4.jar and sap_IntroscopeAgent. additionally provide files named javacore*.ini A: 1.g. Agent. always use forward slash “/“ on all platforms. instead the option -javaagent is used.6 Q: My server does not start after I added the Introscope Agent. They are located in the working directory of the server node. “no such file or directory“ or “The system cannot find the path specified”.profile (for example /usr/sap/ccms/wily/Agent. what do I do? A: Check if this is not a server issue (If server doesn’t start without introscope please check SAP logs to see if there is an indication of the problem).End to End Root Cause Analysis October 2008 A better solution is available with JDK 1.6 Q: My server seems to hang after I added the Introscope Agent. Please refer to the “Creating an AutoProbe Connector” section in the installation guide for the process of recreating the connector 4. 4. 4. In the configtool. 2.jar) and do “dir“ or “ls“ on it.jar. For IBM platforms. Check the VM output (typically dev_serverx) for any error message like the following error. Take the full path you are using to AutoProbeConnector. if that file does NOT exist or if anything in the path is wrong you will get “file not found“. There may be messages about illegal VM parameters (=typo) or mismatches in the AutoProbe connector:.
profile to make sure you are connecting to the right EM.11 Q: I have a cluster installation across multiple host and instrumented all nodes. but I can’t see the agent reporting in the workstation. for Windows) and make sure all files exist. . Check the Agent profile (e. You will usually see the agent trying to connect to the EM and failing (EM host/port will be specified there). For a complete removal of the agent.9 Q: My server started. follow this procedure: Remove the Java VM parameters that enable the agent: -Xbootclasspath.10 Q: Why is the agent renamed to A<agentname>? A: Wily Introscope has a naming convention for agents. 4. Double check that you used the right separator (: for Unix.jar in the directory /usr/java14_64/jre/lib. The introscope agent log will tell you which Agent profile it is using.End to End Root Cause Analysis October 2008 4.log in the node’s working directory. 4. Both can be achieved via the Workstation by right-clicking on the agent. -Dcom. we have currently no recommendation. please note that you have to change it both at the Agent profile file and in the enterprise manager properties. Wily Introscope 11 . you may first want to shut down the agent or shut off all metrics. Delete the file rt. Make sure there is no firewall between the Agent and the EM.out).4 VM and confuses the Introscope autoprobe connector. if there is one try to ask them to open port 6001 (or the port you want to use) or check if you can do SSH tunneling. Then the server should start. 4. Make sure port 6001 is not already taken. if it is used you can change the port that the agents report to the EM on. we recommend using the directory /sapmnt/SID/global/wily which is available on all machines. It will list all jar files loaded in the VM log (std_server0. Check Introscope agent log file. A: The reason is probably that there is no connection between Agent and EM. Otherwise the agent is renamed automatically.12 Q: How can I uninstall the Agent? A: If you just want to reduce the overhead temporarily. 4. If you do not find a mistake here. For Windows. add the Java VM option –verbose:class to the node and restart. sap_IntroscopeAgent.* Restart the J2EE instance where you changed the parameters. This file is not used by the IBM 1.jar in /usr/java14_64/jre/lib and re-create the connector.g. 4. Remove the agent installation directory after the restart.13 Q: IBM AIX: J2EE Server does not start after adding the Introscope agent A: On some systems we have seen an old (JDK 1. it is not sufficient to restart the J2EE node since parameter changes are only activated during the bootstrap process of an instance. The agent name must begin with a letter.8Q: My server starts.3) rt. but I do not see any agents connecting to the Enterprise Manager and there is no IntroscopeAgent.wily. Check configtool to make sure you are pointing to the right Agent profile. Note that for the SAP J2EE startup framework. A: You probably mistyped some entries in the -Xbootclasspath option. How can I avoid copying the agent files to each machine? A: For Unix installations.
even without deployment.15 Q: Agent Setup Problems with SAP IPC A: Please check SAP Note 886600.End to End Root Cause Analysis October 2008 4.1p9 and the JmxService. See also SAP Note 943031.1p9.jar any more!). 5 EPAgent Installation Troubleshooting 5.introscope.1 retains too much memory. Please replace the line export DIR_INSTANCE=<TREX instance> with DIR_INSTANCE=<TREX instance> export DIR_INSTANCE 6 Solution Manager Diagnostics Integration 6.ear can be undeployed (see Introscope setup guide).ParseException: Wily Introscope 12 .agent. change the following properties in the agent profile to false and restart the server nodes. and uses the same mechanisms as before.sap. (no connector.wily.CannotParseException: java.jar is used.16 Polling of JMX Data with Introscope Prior to release 7. Starting with release 7. If you want to explicitly disable JMX data collection.1 Q: Introscope Data Extraction Fails with CannotParseException? A: If the log files for SMD contain the following entries: com.2 Q: TREX Plugin fails to start on Unix A: There is a bug in the delivered script python_epa.enable=false The only reason to disable JMX may be to work around the problem described in Note 1107279: The Introscope JMX adapter in release 7. 5. This deployment is no longer needed as of 7. Please see note 998967. The plugin calls “saposcol –xml” which might crash with segmentation violation or gives back non UTF-8 characters.jmx.1p9.text. 4.spec. the Investigator node "JMX" will be greyed out and metrics regarding threads and sessions will disappear: introscope.14 Q: J2EE Server does not start with Java 5 / JDK 1.5 / SAP VM A: The Java VM parameters for the Introscope agent are different for Java 5.enable=false com.jdbc. 4.ear was needed to activate polling of JMX data. the explicit deployment of JmxService.autojmx.1 Q: SAPOSCOL Plugin fails to start A: Please install latest available saposcol.introscope. The option – Xbootclasspath disappears. Attached to the note is a patch that fixes the problem.sh. the JMX data collection is always active.server.agent.beans. As a consequence. Instead the option –javaagent:d:/usr/sap/ccms/wily/Agent.
set the log location com.wily. To ensure proper mapping of Introscope agents into the SMD Solutions. Wily Introscope 13 . Either upgrade the Enterprise Manager to release 7.NoSuchMethodError: com. upgrade the bytecode agent that is instrumenting SMD to the latest release.jdbc (for SP14 and up) to log level Debug.jdbc.Ljava/lang/Class. if this does not help. 6. not found at com.wily.wily.introscope.smd.sql.agent. not found A:.service.jdbc.This may occur if the SMD itself is instrumented with an Introscope bytecode agent that is not compatible to the JDBC driver used to access the EM.IntroscopeDriver. the corresponding SMD agents on the monitored systems must be running.java:186) at com.wily.3 Q: E2E Data Extraction Fails with RetrieveException: Cannot get Introscope Agent hostname for host A:.This error is caused by SMD agents that cannot be contacted.getConnection(DriverManager.lang.6 Q: "No Enterprise Manager found for Queries" is displayed when trying to jump to Introscope The message indicates that currently no valid Enterprise Manager connection data is available.2 Q: E2E Workload Data Extraction on AIX takes several hours? A: On AIX some of the queries from Solution Manager Diagnostics to Introscope may hang for a time range from 10 minutes up to 1 hour.1p9 or higher or force it into an English locale by adding the following VM parameters: -Duser.MessageServiceFactory: method getService(Lcom/wily/isengard/postoffice/PostOffice.service.)Ljava/lang/Obje ct.isengard.java:189) 6.java:90) at java.connect(IntroscopeDriver. On the SMD.getConnection(DriverManager.introscope.messageprimitives.IntroscopeConnection. Run this self-check for the solutions and systems where Introscope data extraction is causing problems and provide the resulting XML file to SAP. The defaultTrace will then contain a log of all queries against Introscope.isengard.sap.jdbc. A work-around for the problem will be available with SMD SP12 Patch 4.java:53) at com.)Ljava/lang/Object.sql. The parts of the stack trace look like this: java. remove the bytecode agent from the monitoring system (SMD).Ljava/lang/Class.java:559) at java.4 Q: How to Troubleshoot E2E / EWA Data Extraction from Introscope? A:.language=en -Duser. or.IntroscopeConnection. Make sure you provide this log when reporting problems in this area. For further troubleshooting there is a self-check "Wily Introscope Agents" (starting with SMD SP13).admin.DriverManager.plugins.country=EN 6.messageprimitives.introscope. As resolution.sup. It must run in an English localization to be accessible for the SMD. 6.End to End Root Cause Analysis October 2008 Then the reason is that the IS Enterprise Manager runs in a non-English locale.lang.MessageServiceFactory: method getService(Lcom/wily/isengard/postoffice/PostOffice.wily.open(IntroscopeConnection.<init>(IntroscopeConnection.DriverManager.sap.NoSuchMethodError: com. 6.wily (for SMD up to SP13) or com.5 Q: Connection to the Introscope EM fails with this Exception: java.wily.
however.This file is similar to the profile of the Bytecode instrumentation agent.hostagent.1 Q: How do I get the HostAgent and what is it? A: The Introscope host agent is integrated into the SMDAgent starting with LMSERVICE SP13. SAP_ABAP 7.. defining which data should be polled for which system types. Connection information from the HostAgent when trying to connect the EM can be found in /usr/sap/SMD/Jxx/work/jvm_SMDAgent. All plugins formerly provided by SAP for the EPAgent are now part of the HostAgent. WilyHostAgentTemplate. which did not allow deep integration with the SMDAgent. It runs in the same Java VM as the SMDAgent and is administrated centrally (and automatically) via the SMD. The main configuration files are: IntroscopeSapAgent. invoke the setup wizard for each managed system to activate the HostAgent. Wily Introscope 14 .This file contains host-specific customization. SAP_GC. The check will force a re-validation of all EM connection data and the bottom result table gives details about the connection status to all EMs that are configured. Typically. The setup wizard populates this file. which ABAP systems should be polled and which GC files should be scanned. The following plugins currently available: Saposcol. The definition of the "saposcol" action is also contained here. 7 HostAgent Problems 7.profile . After upgrading to LMSERVICE13. 7. and pick any system in the left table.3 Q: Where are the configuration files for the HostAgent? A: The agent application "com. See also SAP Note 1159369. choose "Introscope EM" as check group in the right table. It's enough to select "Expert Mode". Configuration files are stored in the SMD Repository as resources for this application and replicated to the SMDAgents. It is a replacement of the EPAgent. As soon as the property "com. SAP_TREX.smd.sap.wilyhost" contains the HostAgent.trc.g.sap. the EM connection status is cached and not fully up to date. this file is not customized for specific hosts.This file is considered as generic "content template". The setup wizard populates this file.application. connection problems to ABAP stacks) can be found in /usr/sap/SMD/Jxx/SMDAgent/dev_jrfc.xml .agent.enable=true" is contained the HostAgent will be started. Use the File system browser of SMD to view the file. Additional RFC trace info (e. To get a fully up-to-date view on the Enterprise Manager configuration run a self check.out. Connection parameters (host and port) to the EM are listed here.xml . 7.2 Q: How do I activate the HostAgent? A: The HostAgent is activated automatically by the managed systems setup wizard in SMD.End to End Root Cause Analysis October 2008 In some cases.g. SapAgentConfig. e. The files can be uploaded and downloaded centrally via Diagnostics (Diagnostics Administration Managed Systems Agent Administration Applications Configuration).4 Q: Where can I find log information? Log information goes to the SMDAgent application log and can be viewed by the Agent Log Viewer in SMD: Diagnostics Administration Managed Systems Agent Administration Agent Log Viewer.
hostagent.If you want to see all data. conflicts may occur between different saposcol versions. Initial setting of the parameters (as long as the HostAgent is not active) become effective immediately. 7. raise the metric limit for saposcol in the HostAgent by modifying WilyHostAgentTemplate. The plugin calls “saposcol –xml” which might crash with segmentation violation or gives back non UTF-8 characters. The HostAgent plugins will restart automatically.Implement SAP Note 498112 to exclude some file systems from saposcol monitoring. however.wily.Disable Filesystem and/or disk polling on HostAgent level. Furthermore. The HostAgent uses the binary in directory /usr/sap/SMD/exe to launch the daemon (if it is not active yet) and to poll data from saposcol's shared memory segment. Edit the file 3.profile.hostagent. For this purpose.g.Filesystems.MonitoredProcesses"/> </action> . Changes to the Introscope connection parameters (after the HostAgent has been activated already) become effective only after a reboot. 2. Options to resolve this: . changes become effective immediately except for changes to the Introscope connection parameters: If you update them in SapIntroscopeAgent. If your monitored host has too many file systems / disks.7 Q: SAPOSCOL data is not displayed A: Please install latest available saposcol. this may reach the limit of 1000. conflicts may occur. upload the file as customized resource again (make sure to select the right SMDAgent scope).SapOsColAction" Wily Introscope 15 . modify WilyHostAgentTemplate.smd.xml) via Diagnostics Administration Managed Systems Agent Administration Application Configuration. stop the saposcol daemon and delete the shared memory segment (saposcol -k).sap.action. SapAgentConfig. Download the configuration file (e. .Disks.g. To solve the conflict. synchronize the saposcol binaries under SMD and the managed system to use the latest version.6 Q: The HostAgent does not show up in the Investigator although I corrected some typo in the Introscope Connection Parameters Reboot the SMDAgent. by "stop/start apps" in the Agent Administration).xml and remove Disks or Filesystems in the following example: <action prefix="SAP OsCol" name="SAPOsCol" class="com.End to End Root Cause Analysis October 2008 7. 4. may still hit the limit of saposcol and that it may impose some load on the Enterprise Manager if you have too many agents reporting a large number of saposcol metrics: <action prefix="SAP OsCol" name="SAPOsCol" class="com. Note that this solution.5 Q: How can I manually change the HostAgent configuration? 1. 7.smd. SelfMonitoring / agent log files indicate "metric limit exceeded" A: There is a built-in limit of 1000 metrics for each host agent action.sap.8 Q: SAPOSCOL data is not displayed.SapOsColAction" destination="Process_Saposcol" period="60000"> <property name="sections" value="CPU.xml.action.wily. and re-launch the SMD HostAgent (e.LAN. If the saposcol shared memory segment is created by some SAP instance using a different saposcol version. Please see note 998967 for the latest version.Memory. a reboot of the SMDAgent is required! 7.
Please refer to the Installation guide to the section “Configuring SAP NetWeaver for WebView Deployment“. what do I do? A: This can either be problems in PBL/PBD. If that file was not processed make sure it was added to the agent profile or to the right pbl file. 8. Check the introscope agent log file to see if it has any errors with using pbl. Check if autoprobe.0.windows" value="applications. what do I do? A: agent. Memory and CPU in the investigator tree. what do I do? A: This is due to a JVM bug related to running applications in headless mode.” for windows). what do I do? A: Traffic light can not report historical mode. sap_typical.application. “.1 Q: I see just Host. Check the AutoProbe. make sure you are in Live mode.4 Q: I see empty (white) traffic lights in WebView/Workstation. I:/usr/sap/QP1/TRX00 8 Missing Data Problems 8.config\com. Introscope will only show classes that were exercised.End to End Root Cause Analysis October 2008 destination="Process_Saposcol" period="60000" metriclimit="5000"> 7. Furthermore. the customizing for the correspond data collector Action may be wrong. see if there are any errors in the file.headless=true on Windows platforms. Wily Introscope 16 .agent.g.3 Q: I see empty graphs in WebView on Netweaver.log is created in the wily agent directory.jar or instrumentation (pbl or pbds) is NOT picked up.26 or later is required to collect data via the HostAgent. 8. make sure it is not commented and that the server was restarted after the change was done. make sure there is the right separator between the connector jar and the agent jar (“:” for UNIX .9 Q: TREX data collection does not work A: TREX 7.awt. no JDBC/Portal/J2EE or any other metrics.g. You have to remove the VM option –Djava.sap. pbd or jar files (e. Check the file SapAgentConfig. <property name="command.log file to make sure it processed the PBD you created. e.jar). class/classes were not exercised yet or server was not restarted after the change in instrumentation.bat"/> The installation path for TREX must always be given using forward slash. portal J2EE or custom instrumentation that I created. Check Introscope agent log file to see if it had any problem with one of the PBL or PBD files (you may get syntax errors in the pbd or errors indicating you used the wrong separator between the directive files in the agent profile. 8.smd. In the configtool.2 Q: When I look at the Wily Investigator tree I am missing some nodes. Try to exercise the part of the application you expect to see.xml for the corresponding HostAgent to make sure that it has the same spelling with respect to \ and / as below:.wilyhost\trex\python_epa.
DataSource SAPSIDDB If any counter is not active here.sap. only role administrators is allowed to access JMX at … This error is due to missing privileges to the user configured to retrieve the JMX data.6 Q: JmxService J2EE Application Causes Problems for Deployment / Upgrades The. After this.7 Q: JMX data (Threads. JDBC connections) shows only 0 values A: First of all.JmxSecurityException: Caller Guest not authorized.1p9. make sure that the JmxService helper application is deployed on the monitored systems (see Introscope installation guide). Use the SMD Setup Wizard or check the setup guide to activate the saposcol plugin. For older agents. Go into the “Security Provider” service . sessions. If you are running on Win32 or Linux 32 Bit.End to End Root Cause Analysis October 2008 8. a JmxService J2EE helper application had to be deployed on the monitored system. Next. 8.1p9. Below the different steps in order to fix it: 1. sessions. check the Introscope agent log. check in Visual Administrator Server Monitoring Service if the appropriate counters are active (reporting useful) data here. Wily Introscope 17 . If you are running on a 64 Bit JVM you will not see CPU usage in the Investigator tree and CPU per process data will not be available. Please undeploy the JmxService application and upgrade your agent to 7. look at the Introscope agent log file and see if there is an error/warning about platform monitor. 3. 1/27/08 09:37:33 AM CET [VERBOSE] [IntroscopeAgent] com. the counter should become active.1p9 or later. For all platforms. 2. The tree locations to check are: Kernel Kernel Services Services Application Threads Pool System Threads Pool Web Container Connector Service Stand Alone javax. React on Resource Failure = ignore.exception. Select “Runtime” tab and “Policy Configurations” tab.deployment descriptor of the JmxService J2EE helper application causes problems for JSPM. While before. click Configuration Edit. note that the mechanism to enable JMX data retrieval has changed starting with agent release 7.5 Q: I can’t see CPU in my Investigator tree / Dashboard Charts. 8. Open the Visual Administrator. 8. JDBC connections) does not appear If the JMX node does not appear in the Investigator. select the first instance. It is recommended to undeploy it with agents that have at least version 7.services. Save.jmx.sql.engine.8 Q: JMX data (Threads. this is no longer needed. you can obtain CPU (and other OS-related data) via the saposcol plugin of the HostAgent. If it contains this error message follow the procedure below: 1/27/08 09:37:33 AM CET [ERROR] [IntroscopeAgent] An error occurred polling JMX data. What do I do? A: Platform Monitor (the part that is in charge of getting this info) doesn’t currently support 64 Bit.
7. 6.9 Q: For a Chart. 8.2k. In the components list. Check that the step was applied for all the other instances if applicable. the Currently Displayed Value is 45. Based on this logic 46322 / 1024 gives 45.2k. A: Introscope divides metrics with high values by powers of 1024 to convert it into kilo / mega / giga. 8. select “sap. Although I Would Expect 46322. 5. A: The SAP_GC metrics are mirrored from the HostAgent to the Bytecode agent. The prerequisite for this to work is that the agent name in the HostAgent SAP_GC subtree and the bytecode agent name match: The value for the placeholder <agent_name> must be identical in these two paths: …|SAP Netweaver|<agent_name> and …|SAP HostAgent Process|SAP HostAgent Jxx|SAP_GC|<agent_name> Wily Introscope 18 . change the “Run As Identity“ textbox with an appropriate user belonging to the Administrators group. For all the list of Security roles. For this select “Change“ button. add it with the button ‘Add’ above.com/SQL Trace*OpenSQLMonitors” as shown below. select “OpenSQLMonitor”. If no user is available. Select “Security Roles” tab and switch to edit mode by clicking the pencil in the toolbar above the tabs.End to End Root Cause Analysis October 2008 4.10 Q: SAP_GC Appears for the HostAgent. but not for the Netweaver (bytecode) agent.
g.End to End Root Cause Analysis October 2008 8. please open a message on component XX-PART-WILY.11 Q: SAP_GC Data does not show metrics for Tenured and Perm Heap Spaces. If there are Full GCs in the trace and they do not show up as metrics in the Investigator tree. A: These metrics only appear after the first Full GC has been performed in the monitored system and after this GC has been logged to the monitored trace file (e. Wily Introscope 19 . std_server0.out).
the Java heap size of the EM must be increased as well.5GB RAM should be reserved for the Enterprise Manager. 9. 9. For an increasing number of metrics. open the Investigator and check the tree Custom Metric Host Custom Metric Process Custom Metric Agent. at least 1. currently used Java heap. An Enterprise Manager on a Windows 32 bit server system can typically handle at least 200000 metrics if the heap is extended to 1500MB. resource requirements for the Enterprise Manager heavily depend on the number of Introscope agents connected and on the workload of the monitored systems. To check the current capacity of the Enterprise Manager. Wily Introscope 20 .End to End Root Cause Analysis October 2008 9 Enterprise Manager Sizing and Tuning In general.2 Q: How many CPUs are needed for the Enterprise Manager? A: 1-2 CPUs should be reserved for the Enterprise Manager. 9.3 Q: How Much RAM is needed for the Enterprise Manager? A: Calculate about 1. 9.4 Q: How Many Agents Can the Enterprise Manager Handle? A: The capacity of the Enterprise Manager (EM) depends mainly on the total number of metrics that are reported by all agents. This contains the current capacity. For a heap of 1024MB. and the number of active agents and metrics.5-2 times the Java heap size (-Xmx). Select the node Enterprise Manager as shown below in the screenshot. a maximum of ~500MB per reporting agent could be occupied for historical data.1 Q: What disk space is required for the Enterprise Manager? A: In addition to the installation files (<200MB).
5 Q: What If I want to Connect More Agents than the Enterprise Manager Can Handle? A: Even on 64 bit systems the EM cannot scale arbitrarily to handle any number of agents. 9. Portal) and workloads (production.directory.8 Q: How Can I Gather Detailed Support Data for the Enterprise Manager? A: Please follow SAP Note 1105109 to collect support and performance information from the Enterprise Manager and provide the resulting ZIP file to SAP.dedicatedcontroller=true 9.smartstor. For this purpose. 9. please open a message on component XX-PART-WILY. To still get a unified view on all agents of all Enterprise Managers. the EM can handle ~40 agents. A: The Enterprise Manager is doing housekeeping for all log files and for the persistence directories (/data and /traces). an additional Enterprise Manager can be installed as Manager of Managers (MoM). Wily Introscope 21 . In more detail. A good guess for SAP Netweaver agents is 5000 metrics per agent. change the following properties in the file config/IntroscopeEnterpriseManager. If you need such a so-called clustered EM installation. 9. Assuming 3000 metrics per agent.enterprisemanager. the EM can handle ~65 agents. After these periods. the corresponding directories should stop growing (assuming a constant workload). Each agent has a metric “Agent Stats Metric Count”. QA system) will have different metric counts. Based on the numbers you observe (metrics per agent) and on the rough limit of the EM (~200000 metrics per EM) you can derive the maximum capacity of the EM: Assuming 5000 metrics per agent.7 Q: How Can I Tune the Enterprise Manager Performance? A: Put the /data directory on a different disk drive / disk controller.enterprisemanager.smartstor. The files are condensed automatically and removed after some time.archive=/your/separate/drive/ archive introscope.enterprisemanager. Different usage types (XI.directory=/your/separate/drive/data introscope. the /traces directory keeps a history for 14 days and the /data directory keeps a history of 31 days. If the capacity of a single EM is exhausted. you can install multiple Enterprise Managers on different hosts or with different ports on the same host.smartstor.End to End Root Cause Analysis October 2008 The number of metrics per agent can be determined in the Investigator as well.6 Q: Do I have to perform any housekeeping tasks? It seems like the data subdirectory of the EM is growing fast.properties: introscope.
transaction traces are saved automatically on the Enterprise Manager for 14 days. . delete the directory /traces. additionally type a search term like host:myhost or url:/irj. open a Workstation and connect it to any Enterprise Manager (e.Transaction Traces with more than 50000 components are dropped automatically on agent side and do not show up in the UI. select a time frame. Increase the trace period to several hours to see if this makes a difference. Also note that filters like catching are specific user ID are case-sensitive. Shutdown the EM. See also Note 1107756. recording will cease for this transaction.In some rare cases the tracer database on the Enterprise Manager may be corrupted. . select some traces in the summary table. and save it to some file. Open Workstation Query Historical Events.You may see strange behavior if the Workstation is run in a different time zone than the agent.g. A: There are several things to check: . To find out if this is the case check the agent log for entries like Transaction trace component limit of 50000 reached. To re-import the XML file into a different Workstation. You can redisplay them using the Trace tab in the Investigator or using the screen Workstation Query Historical Events.1 Q: How can I save and Re-Display Transaction Traces? A: First of all. For transporting the traces between systems. Trace Open Saved Events (XML) and select the XML file you created. . and restart the EM. 10.Double check your trace configuration and try different or more general configurations. Wily Introscope 22 .2 Q: I get data in the Investigator but no Transaction Traces. In both cases. via the Menu choose Trace Save as. a "dummy" EM on your desktop PC).End to End Root Cause Analysis October 2008 10 Using Introscope 10. For the query historical events screen.
This action might not be possible to undo. Are you sure you want to continue?
We've moved you to where you read on your other device.
Get the full title to continue reading from where you left off, or restart the preview.