SAP Solution Manager 7.

0 EhP1

End-to-End Diagnostics
Version 1.5, June 2009

Diagnostics agent troubleshooting

END-TO-END DIAGNOSTICS
DIAGNOSTICS AGENT TROUBLESHOOTING

Typographic Conventions
Type Style Example Text Represents Words or characters quoted from the screen. These include field names, screen titles, pushbuttons labels, menu names, menu paths, and menu options. Cross-references to other documentation. Example text
EXAMPLE TEXT

Icons
Icon Meaning Caution Example Note Recommendation Syntax

Emphasized words or phrases in body text, graphic titles, and table titles. Technical names of system objects. These include report names, program names, transaction codes, table names, and key concepts of a programming language when they are surrounded by body text, for example, SELECT and INCLUDE. Output on the screen. This includes file and directory names and their paths, messages, names of variables and parameters, source text, and names of installation, upgrade and database tools. Exact user entry. These are words or characters that you enter in the system exactly as they appear in the documentation. Variable user entry. Angle brackets indicate that you replace these words and characters with appropriate entries to make entries in the system. Keys on the keyboard, for example, F2 or ENTER.

Example text

Example text

<Example text>

EXAMPLE TEXT

END-TO-END DIAGNOSTICS
DIAGNOSTICS AGENT TROUBLESHOOTING

Contents
1 2 Introduction ......................................................................................................................................... 4 Pre-requisites ...................................................................................................................................... 5 2.1 OS support ................................................................................................................................... 5 2.2 JDK compatibility......................................................................................................................... 6 2.3 Managed System.......................................................................................................................... 8 2.4 SLD production system............................................................................................................... 9 Agent installation strategy................................................................................................................ 10 3.1 Standard environment............................................................................................................... 10 3.2 HA/virtualization environment .................................................................................................. 10 Agent installer.................................................................................................................................... 13 4.1 SAPinst Documentation ............................................................................................................ 13 4.2 Troubleshooting with SAPinst .................................................................................................. 13 4.3 OS configuration checks........................................................................................................... 14 4.4 JVM parameters ......................................................................................................................... 14 4.5 Startup scripts ........................................................................................................................... 15 Agent connectivity ............................................................................................................................ 18 5.1 Managing system....................................................................................................................... 18 5.2 Managed system ........................................................................................................................ 21 5.3 Full Qualified Name resolution issue ....................................................................................... 24 5.4 SAP router.................................................................................................................................. 24 5.5 Self-Monitoring .......................................................................................................................... 25 Agent Maintenance............................................................................................................................ 26 6.1 Upgrade SAP Kernel (saposcol update)................................................................................... 26 6.2 Configuration ............................................................................................................................. 26 6.2.1 Change Path of the JDK....................................................................................................... 27 6.2.2 Change JVM Parameters .................................................................................................... 27 6.2.3 Change the virtual host name............................................................................................... 27 6.2.4 Usage of smdsetup script..................................................................................................... 28 6.2.5 Change Agent log level ........................................................................................................ 31 6.3 Managing J2EE user.................................................................................................................. 32 Tracking agent issues ....................................................................................................................... 33 7.1 Deprecated SAP notes .............................................................................................................. 33 7.2 OSS message pre-requisites..................................................................................................... 33 7.3 Agent crash investigation ......................................................................................................... 34 7.4 Thread dump .............................................................................................................................. 35 7.5 Activate P4 logs ......................................................................................................................... 36 7.6 Telnet SMD server commands .................................................................................................. 37 Scalability........................................................................................................................................... 37 8.1 Managing java stack settings.................................................................................................... 37

3

4

5

6

7

8

End-to-End Diagnostics Diagnostic agent troubleshooting guide

4

1

Introduction
The goal of this guide is to help Solution Manager Users troubleshoot failures encountered when installing, configuring and using the Diagnostics Agents. It contains the description of the resolution of the most common known issues related to the Diagnostics Agents. It gives also information, tooltips and procedure to isolate and identify the root cause of an issue related to the Diagnostics Agent. This guide is separated in different areas that correspond to the different phases of the Solution Manager life cycle. The chapters of this document follow a logical order corresponding to the Diagnostics Agent’s life cycle. However, in order to successful troubleshoot a Diagnostics Agent’s issue, you should be familiar with all parts of this document. This guide covers the following area: Pre-requisites: Helps users to check that all pre-requisites are fulfilled in order to complete a correct installation of the Diagnostics Agents. Agent Installation Strategy: Answers to the most frequent questions rose during the planning phase of Diagnostics Agent installations. Agent Installer: Lists all common issues encountered during the installation of the Diagnostics Agents. Agent Connectivity: Help users to understand and troubleshoot the connection mechanisms flow between: o The Diagnostics and the Managing System. o The Diagnostics agent and the Managed systems. It includes also the main resolution procedures related to network configuration issues. Agent Maintenance: Describes the procedure and common errors related to the operation of the Diagnostics Agent. It concerns the administration of the Diagnostics Agents including a description of the smdsetup script. Tracking Agent Issues: Explains where to find information related to the Diagnostics Agent activities. It explains how to interpret this information in order to establish a correct diagnostics when troubleshooting an Agent issue. Scalability: Contains scalability and performance notes related to the Diagnostics Agent.

©SAP AG 2009

End-to-End Diagnostics Diagnostic agent troubleshooting guide

5

2
2.1

Pre-requisites
OS support

The following table describes the list of different OS supported by the Diagnostics agent: OS
Server 2003 on IA32 32bit Server 2003 on IA64 64bit Server 2003 on x64 64bit 5.3 64bit on Power 4, 5 6.x 64bit on Power 4, 5 11.31 on IA64 64bit 11.11, PA-RISC 64bit 10 on SPARC 64bit IA32 32bit IA64 64bit x86_64 64bit Power 64bit SLES9 only on zSeries 64bit V5R3, V5R4 64bit 1.6, 1.7, 1.8, 1.9 64bit

Diagnostics Agent 7.00 / 7.01 X X X X X (SAP Kernel > 133) X X X X X X X X X -

Diagnostics Agent 7.11 X X X X X X X X X X X X X X -

Windows

AIX

HP

Solaris

Linux

I5/OS Z/OS TRU64

©SAP AG 2009

End-to-End Diagnostics Diagnostic agent troubleshooting guide

6

2.2

JDK compatibility
OS
Server 2003 on IA32 32bit Server 2003 on IA64 64bit Server 2003 on x64 64bit 5.3 64bit on Power 4, 5 6.x 64bit on Power 4, 5 11.31 on IA64 64bit 11.11, PA-RISC 64bit 10 on SPARC 64bit 10 on x86_64 IA32 32bit IA64 64bit x86_64 64bit Power 64bit SLES9 only on zSeries 64bit V5R3, V5R4 64bit 1.6, 1.7, 1.8, 1.9 64bit

The following table describes the list of different JDK supported by the Diagnostics agent for each platform: Diagnostics Agent 7.00 / 7.01 SUN 1.4.2.13 to 1.4.2.17b061 or 1.4.2_21+ SUN 1.4.2.13 to 1.4.2.17b061 or 1.4.2_21+ SUN 1.4.2.13 to 1.4.2.17b061 or 1.4.2_21+ IBM 1.4.2 SR12 caix64142-20080923 IBM 1.4.2 SR12 caix64142-20080923 HP 1.4.2.15 to 1.4.2.172 or 1.4.2.21+ HP 1.4.2.15 to 1.4.2.172 or 1.4.2.21+ SUN 1.4.2.13 to 1.4.2.17b06 or 1.4.2_21+ SUN 1.4.2.13 to 1.4.2.17b06 or 1.4.2_21+ SUN 1.4.2.13 to 1.4.2.17b061 or 1.4.2_21+ SUN 1.4.2.13 to 1.4.2.17b061 or 1.4.2_21+ IBM 1.4.2 SR10 j9xa64142-20080130 IBM 1.4.2 SR12 cxp64142-20080923 IBM 1.4.2 SR12 cxz64142-20080923 IBM IT4J JVM Diagnostics Agent 7.11 SAPJVM5 last version SAPJVM5 last version SAPJVM5 last version SAPJVM5 last version SAPJVM5 last version SAPJVM5 last version SAPJVM5 last version SAPJVM5 last version SAPJVM5 last version SAPJVM5 last version SAPJVM5 last version SAPJVM5 last version SAPJVM5 last version SAPJVM5 last version SAPJVM5 last version -

Windows

AIX

HP

Solaris

Linux

I5/OS Z/OS TRU64

For Diagnostics agent 711, the SAPJVM 5 is included with the SAP Kernel DVD. Therefore, you don’t need to install separately the SAPJVM5 before installing the 711 agent. The following SAP notes can be used to determine the JDK version but do not apply the JVM parameters proposed in those sap notes since they are only specific to the sapj2ee engine. Note 716604 for the Sun JDK (Windows, Linux, Solaris) Note 716926 for the HP JDK (HP-UX) Note 716927 for the IBM JDK (AIX) Note 717376 for the IBM JDK (iSeries, OS/400) Note 746299 for the IBM JDK (zSeries) Note 810008 for the IBM JDK (Linux on POWER) Note 861215 for the IBM JDK (Linux on AMD64/EMT46)

1 Until further notice please don't use 1.4.2.17b12, 1.4.2_18 and 1.4.2_19. 2 Until further notice please don't use 1.4.2.20 (SAP note 1301397)

©SAP AG 2009

End-to-End Diagnostics Diagnostic agent troubleshooting guide

7

Most common JDK problems with agent Timeout occurred This error is logged in SMDSystem.log
Exception: Time out occurred when calling method 'configure' on object [com.sap.smd.api.IAgentContext_Stub@49f949f9] after 10000 ms. possible cause: com.sap.smd.server.util.concurrent.TimeoutException at com.sap.smd.server.exec.asio.AsioInvocationHandler.invoke (AsioInvocationHandler.java:130)

This timeout issue is caused either by a wrong JDK Version or wrong JVM parameters. The timeout occurs often on old IBM SDK or when the J9 technology is not enabled in the java parameters of the Diagnostics agent. Check also the java stack thread settings (see chapter 8.1 Managing java stack settings) Agent applications are not updated This error is still logged in SMDSystem.log
Exception: [Thread[Thread-5,5,main]] Error [com.sap.smd.om.impl.broker.ObjectProxyImpl@76933bcb (states:core.patch [proxy])]: error during load of states:core.patch [EXCEPTION] com.sap.smd.om.ObjectLoadException: failed to instantiate implementation class at com.sap.smd.core.plugin.service.ServiceEntityHandle.entityLoad(ServiceEntityHandl e.java:159) ... ... Caused by: java.lang.UnsatisfiedLinkError: inode0 at sun.nio.ch.FileChannelImpl.inode0(Native Method) at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:868)

This problem occurs only with HP JDK; re-install a JDK version recommended for Diagnostics agent (currently 1.4.2.17 or 1.4.2.21+).

©SAP AG 2009

End-to-End Diagnostics Diagnostic agent troubleshooting guide

8

2.3

Managed System

The following list describes the SAP systems for which the SAPinst installs automatically a Diagnostics agent at the end of the SAP instance (central or dialog) installation procedure. System NW 2007 Abap stack NW 2007 Java stack Netweaver Double stack Solution Manager NW 711 Abap stack NW 711 Java stack PI 7.11 CE 7.11 PI 7.10 NW 7.10 Java stack SAPinst 7.0 SR3 X X X SAPInst 7.01 Ehp1 X X X SAPInst 7.11 Ehp1 (X) Not scheduled yet (X) Not scheduled yet X X -

If the Diagnostics Agent has not been installed automatically, you have to install it as a standalone engine using the standard SAPinst (Please refer the Agent Setup Guide for more information). To avoid libraries conflicts it is highly recommended to install a Diagnostics agent which has the same kernel release than the SAP systems already installed on the host. Ex: On a NW04 or NW04s Managed System, you should install a Diagnostics agent 700 or 701, On a 711 or 710 Managed System, you should install a Diagnostics agent 711, … Note for System NW2004 (640), the recommendation is to install a Diagnostics agent NW2004s (700) only if no previous Diagnostics agent 640 has been installed on system. Ex: on a host with a sap system 640, if no agent is present -> install a new diagnostics agent 700(SR3) on a host with sap system 640, if one or more agent 640 are present -> install a new 640 agent.

©SAP AG 2009

End-to-End Diagnostics Diagnostic agent troubleshooting guide

9

2.4

SLD production system

To register a Diagnostics agents to a productive SLD, the CIM Model version of the productive SLD must be 1.5.26 or higher. How to determine which version of the CIM Model is installed? Go in SLD with the url: http://sldhostname:port/sld Logon with a J2EE user having the role SAP_SLD_ADMINISTRATOR Click on link ‘About’. The CIM Model version is displayed in a window popup. How to upgrade the CIM Model? Refer to SAP Note 669669

Updating the SAP Component Repository in the SLD

How to check if the Diagnostics agent is registered in SLD? Go in SLD with the url: http://sldhostname:port/sld Logon with a J2EE user having the role SAP_SLD_ADMINISTRATOR Go in the section ‘Administration’ Click on link ‘Content Maintenance’ Select the subset ‘All with Instances and class ‘Diagnostics Agent Instance’ If the class ‘Diagnostics Agent Instance’ does not appear in drop down list, No Diagnostics agent is registered In column Name of table, check if the Diagnostics agent is present. Or Go in Agent Administration tool and click on link “agent candidates” Enter the information to connect to productive SLD The list of Diagnostics agents registered in this SLD will be displayed. Which is the J2EE Group to assign to the SLD user used by the Diagnostics Agent? The SAP_SLD_CONFIGURATOR group or the security role DataSupplierLD should be assigned to the user used by the Diagnostics Agent. What is the SLD registration frequency of the Diagnostics Agents? Every 12 hours, the Diagnostics agent sends an SLD request to update the agent model information. What is the frequency used by the Diagnostics Agent to check if an association is present in SLD? If the Diagnostics agent is not connected to the Managing System and registered in SLD, it will check every 20 minutes if an association has been created in SLD. If the Diagnostics agent is still connected to Managing System and registered in SLD, it is going to check every hour if a new association has been created in SLD. Why the Diagnostics agent is not visible in Agent Admin tool? You need to create an SLD association between the Diagnostics agent and the Managing system. To create this association go in Agent Admin tool, click on link “agent candidate”. Enter the SLD connection data where the Diagnostics agent is registered then select the agent and click on the “attach” button.

©SAP AG 2009

End-to-End Diagnostics Diagnostic agent troubleshooting guide

10

3
3.1

Agent installation strategy
Standard environment

The following check list must be validated before installing a new Diagnostics agent on a managed host: Note: Don’t install a Diagnostics Agent on a Database or an SCS host without any Central Instance. This is not necessary in the current E2E RCA scenarios.

Check if a Diagnostic agent has already been installed on the managed system host. Verify if the managed system has been installed with a SAPInst version greater than NW04S SR3 Check on file system if the folder /usr/sap/DA or the folder /usr/sap/SMD exist. In both cases, an agent is already installed on the system. If the Diagnostics agent is already installed on system, check if the agent is present in the SLD server where the managed system is registered. In this case attach the agent to the Solution Manager System using the agent admin tool (agent candidate view) or connect the agent with the script ‘smdsetup’ (see part 6.2.4). If the Diagnostics agent is not installed, retrieve the virtual host or physical host name of the managed system. The rule is to install one diagnostics agent per virtual host. Go in the sap profile of the managed system (/usr/sap/SID/SYS/profile). Retrieve the hostname in the profile file name. (Ex: for the value W46_JC93_NCED60185755A, the hostname is nced60185755a). The Diagnostics agent must be installed with this hostname. When the installation is finished, on UNIX systems add the Diagnostics agent in OS startup/shutdown script (see chapter 4.5 Startup scripts) For UNIX system, check these Operating System limits of OS agent user (see part 4.2).

3.2

HA/virtualization environment

In High Availability environment, the Diagnostics agents must be installed on each virtual host used by the SAP Instance (Except for the hosts running the database or the Central Services). The Diagnostics Agents must be integrated in the cluster fail-over scripts used to control the managed systems. For this reason, the recommendation is to install the Diagnostics agent on a shared file system. During a failover scenario, the SAP instance of the managed system and the Diagnostic agent must be moved together. When the SAP instance and the agent are moved to another system node, you should mount the agent instance partition before the SAP instance partition. This is due to the fact that the classpath of the SAP java instance depends on some jar files contained in the agent folder for the Wily IS instrumentation.

©SAP AG 2009

End-to-End Diagnostics Diagnostic agent troubleshooting guide

11

Microsoft Cluster (MSCS)
With MSCS you enable critical system components, Single Points of Failure (SPOFs), to be automatically switched from one machine to the other in the event of hardware or software problems, or system maintenance affecting one machine. Until now, SAP has only supported the installation of one clustered SAP system in one MSCS cluster with two MSCS nodes. It is necessary to install a Diagnostics agent for each central instance and all dialog instances. The Diagnostics agents must be installed on a local disk. On the failover node, you should activate the windows service used to start the Diagnostics Agent.

Figure 1: ABAP or Java System with SPOFs, where non-SPOFs are installed locally on the MSCS Cluster Nodes

©SAP AG 2009

End-to-End Diagnostics Diagnostic agent troubleshooting guide

12

Figure 2: ABAP+Java System where the non-SPOFs are installed on hosts outside of the MS Cluster

More information is available in SAP Note 112266 - SAP and MS Cluster Server: Frequent questions and tips.

Flexframe
A FlexFrame-based server landscape for SAP usually consists of the following components: 2 highly available control nodes (without SAP software) One or more network appliance filer(s) (IO subsystem, NFS server) Application-node with Linux or Solaris operating systems. Implemented high availability of SAP components (myAMC - FlexFrame autonomous agents). Virtualization of SAP services (at least 1 IP address for each service irrespective of the server IP address). This can result in an instance being "changed" to another server. In addition, a check would be carried out to see whether the instance in question is already running on another application node. On this platform, install a Diagnostics agent for each service hosted on the application nodes identified by virtual IP addresses.

Other virtualization solutions
Please refer to SAP Note 674851 - Virtualization on Windows to get a list of supported virtualization solutions on Windows.

©SAP AG 2009

End-to-End Diagnostics Diagnostic agent troubleshooting guide

13

4
4.1

Agent installer
SAPinst Documentation

The following SAP Notes contain the most recent version of the ‘Agent Setup guide’ documentation and describes the problems that could occur during the installation: SAP Note 1234387: Inst. 7.1 EhP1 - Diagnostics Agent installation SAP Note 1234384: Inst. NW/SOLMAN 7.0 EhP1 - Diagnostics Agent installation SAP Note 1144737 Inst. NW 7.0 SR3/SOLMAN 7.0 - Diagnostics Agent installation

Don’t use the SAPInst SPS13 for SMD AGENT 7.00 or lower (SMDAGENT13_0-20001420.SAR), this installer is deprecated. If you need to install more than 20 agents, it is recommended to use the Mass Deployement option of the Diagnostics agent. SAP Note 1172764: Mass Deployment of SAP Diagnostics Agent 7.0 & 7.01 SAP Note 1294713: Mass Deployment of SAP Diagnostics Agent 7.11.

4.2

Troubleshooting with SAPinst

This section describes how to proceed when errors occur during the installation of the Diagnostics Agent with SAPinst. If an error occurs, SAPinst will stop the installation and display a dialog informing you about the error. In that case, the procedure is the following: a. To view the log file, choose View Logs. b. If an error occurs during the dialog or processing phase, do either of the following: Try to solve the problem Abort the installation with Exit Continue the installation by choosing Retry. c. Check the log files of SAPinst in the installation directory sapinst_instdir SAPinst normally creates this installation directory directly below the temporary directory. SAPinst finds the temporary directory by checking the value of the environment variables TEMP, TMP, or TMPDIR.

If no value is set for these variables, SAPinst uses /tmp/sapinst_instdir as default installation directory on UNIX platforms and %ProgramFiles%\sapinst_instdir on windows platforms. If you want to use an alternative installation directory, set the environment variable TEMP, TMP, or TMPDIR to the required directory.

©SAP AG 2009

End-to-End Diagnostics Diagnostic agent troubleshooting guide

14

4.3

OS configuration checks

After the installation of the Diagnostics agent under UNIX platform, check the following OS parameters corresponding to the Diagnostics agent OS user (ex: smdadm) Check the Diagnostics agent OS user limit : o Logon with the agent OS user ‘su – smdadm’ o Open ‘sh ‘shell with command ‘sh’ o Execute the command ‘ulimit –a’ o Compete the values with the following recommended values

time(seconds) file(blocks) data(kbytes) stack(kbytes) memory(Kbytes) coredump(blocks) nofiles(descriptors)

unlimited unlimited unlimited 4194304 unlimited 4108901 8092

If the user’s limit parameters for the OS agent user are not correct, some OutOfMemory exceptions can occur in the extractor jobs of the Diagnostics agents. Check File System access right: If during an extractor job (datacollect), an access denied occurs in the agent log ‘SMDApplication.log’, check if the folder /usr/sap/<SID_AGT>/JXX/SMDAgent/temp is writable for the sapsys group. If is not the case, execute the command ‘chmod –R 775 /usr/sap/<SID_AGT>/JXX/SMDAgent/temp’. Furthermore the file system of the Managed system must be readable by the Diagnostics agent. Logon with an agent OS user (su – smdadm), go in the managed system folder (ex: /usr/sap/SID/SYS/profile), and open one profile with the command ‘more SID_INSTANCE_HOST’. If the “more” command is not working, the access rights are not correct for the Diagnostics agents. In that case, check with your system Administrator to adapt the file system rights.

4.4

JVM parameters
JDK JVM Parameters -DP4ClassLoad=P4Connection -Xj9 -Xgcpolicy:gencon -Xmn50m -Xmx256m -Xms256m -DP4ClassLoad=P4Connection -Xmx128m -Xms128m -DP4ClassLoad=P4Connection -Xmx128m -Xms128m -DP4ClassLoad=P4Connection -Xmx128m -Xms128m SAP Note 1093831 n.a n.a n.a

In this section, you will find the recommended Diagnostics agent’s JVM parameters for each JDK vendor.

IBM SR10 or higher SUN JDK HP JDK SAP JVM5

Don’t use the java parameter verbose:gc in the Diagnostics agents JVM parameters, because this parameter will steadily increase the size of the file work/std_SMDAgent.out

©SAP AG 2009

End-to-End Diagnostics Diagnostic agent troubleshooting guide

15

4.5

Startup scripts
Information about Diagnostics agent 7.0x
On Windows platform the SAP MMC allows to start or stop the Diagnostics agent. The NT service of the Diagnostics agent is configured to start automatically during Windows startup.

The green status in SAP MMC only indicates that the agent is running. It doesn’t report if the agent is connected to the Managing system. If the status is yellow after the agent start, it is probably due to the fact that the agent process was stopped abnormally (see 7.3 Agent crash investigation) On UNIX platform, the commands smdstart.sh, smdstop.sh and smdadmin.sh allow to operate the Diagnostics agent. Since the SR3 release of SAPInst, those commands are included in the PATH variable of the OS agent user. In that case, the commands can be called from every folder (not only in the agent script folder). Command smdstart.sh <SID> <InstanceNumber> smdstop.sh <SID> <InstanceNumber> smdadmin.sh <SID> <instanceNumber> Action Starts the Diagnostics agent and the sapstartsrv process. Stops the Diagnostics agent and the sapstartsrv process Launches jcmon tool to administrate the Diagnostics agent instance

On Solaris platform, an error (/usr/sap/SMD/exe/smdstart.sh: usage: ulimit [ -HSacdfnstv ] [ limit ]) occurs when the smdstart.sh is executed. Check the solution in sap note 1109366. On HP-UNIX platform, an error (!! ERRROR !! current user is not the owner of this instance ) occurs when the smdstart.sh is executed. Check the solution in sap note 1109366.

The following table gives the list of the return code of the smdstart.sh script. Exit code of smdstart.sh 0 2 5 4 3 Reason Success to launch the agent process (This is not the confirmation that the agent is connected to SMD). Input Parameters are invalid. Agent is already running. User limit are not set correctly. Current OS user is invalided to start the agent.

For UNIX system, Diagnostics agents do not start automatically when the system reboots because the agent startup scripts are not included in the system startup scripts. The following command line can be used in the system startup scripts:

su -c /usr/sap/SMD/J98/script/smdstart.sh > /dev/null 2>&1

-s /bin/sh - smdadm

You will find below the command line for the shutdown sequence:

su -c /usr/sap/SMD/J98/script/smdstop.sh > /dev/null 2>&1

-s /bin/sh - smdadm

©SAP AG 2009

End-to-End Diagnostics Diagnostic agent troubleshooting guide The following procedure describes how to insert an agent in the boot sequence for a Linux SUSE 10 system: Create a file diagnostics_agents in folder /etc/init.d Determine the init run level in file /etc/inittab Create a symbolic link named S80diagnostics_agents in the folder /etc/init.d/rc<INIT_LEVEL>.d/ to the file diagnostics_agents (The agent has to be started after the initialization of the network services). Create a symbolic link named S80diagnostics_agents in the folder /etc/init.d/rc6.d to the file diagnostics_agents (runlevel 6 corresponds to the System reboot).

16

When smdstart.sh and smdstop.sh are embedded in the system startup scripts, the exit code of the smd scripts cannot be retrieved. Further it is important to redirect the standard output and standar error with a pipe like 2>&1 to avoid that the smdstart.sh and smdstop.sh are blocking the parent script.

#!/bin/sh ### BEGIN INIT INFO # Provides: diagnostics_agents # Required-Start: $network # Required-Stop: $network # Default-Start: 2 3 5 # Description: start diagnostics agents ### END INIT INFO case "$1" in 'start') su -c /usr/sap/SMD/J98/script/smdstart.sh > /dev/null 2>&1 -s /bin/sh - smdadm ;; 'stop') su -c /usr/sap/SMD/J98/script/smdstop.sh > /dev/null -s /bin/sh - smdadm ;; 'status') su -c /usr/sap/SMD/J98/script/smdadmin.sh -s /bin/sh - smdadm ;; *) echo "Usage: $0 { start | stop | status }" ;; esac exit 0

©SAP AG 2009

End-to-End Diagnostics Diagnostic agent troubleshooting guide

17

Information about Diagnostics agent 7.11 or higher
With Diagnostics agent 711 the startup system is compliant to the SAP startup mechanism. It means that the commands smdstart.sh and smdstop.sh have been removed. Under Windows the SAP MMC allows to start or stop the Diagnostics agent. The NT service of Diagnostics agent is configured to start automatically during the Windows startup.

The green status in SAP MMC only indicates that the agent is running. It doesn’t report if the agent is connected to the Managing system. If the status is yellow after the agent start, it is probably due to the fact that the agent process was stopped abnormally (see 7.3 Agent crash investigation)

Under UNIX with an agent OS User, the commands startsap and stopsap allow to operate the Diagnostics agent. Command startsap SMDA<Instance_Number> [-vhost <virtual hostname>] stopsap SMDA<Instance_Number> [-vhost <virtual hostname>] Action Start the Diagnostics agent. Stop the Diagnostics agent.

The sapstartsrv process of Diagnostics agent is automatically started during the boot sequence of system if the sapinit program has been installed on system. In that case, you can start or stop the Diagnostics agent with a SAP MMC (remote).

About sapstartsrv process
SAP Note 927637 - Web service authentication in sapstartsrv as of Release 7.00 Note 823941 - SAP Start Service on Unix platforms Description The Web service interface of sapstartsrv requires a user authentication from the client (MMC, SAP MC, sapcontrol and so on) for protected operations such as starting and stopping the SAP instance. After an SAP upgrade, the SAP Start Service (sapstartsrv) on UNIX platforms is not automatically started when the operating system is started.

©SAP AG 2009

End-to-End Diagnostics Diagnostic agent troubleshooting guide

18

5
5.1

Agent connectivity
Managing system

In this section, you will find a description of the different connection modes used from the agent to the Managing System. Two modes are available: Connection through the SAPJ2EE message server (RECOMMENDED mode) Direct connection to SAPJ2EE p4 dispatcher.

Connection through the SAPJ2EE Message server It is RECOMMENDED to connect the Diagnostics agent to Solution Manager with the SPA J2EE message server mode. To establish this connection, the information about the hostname of the J2EE Message server (SCS instance) and the HTTP port of message server must be collected from the SCS instance profile.

The agent connection mechanism can be described plainly: A. During the installation of the Diagnostic agent, the SAPJ2EE Message server connection mode must be selected. B. This information is stored in the agent configuration (runtime.properties) and the credentials are stored in the agent secure store file (secstore.properties). C. When the Diagnostics Agent starts, the P4 framework will retrieve all J2EE entries point available from the SCS message server (on HTTP port ex: 81xx). D. Once the list of J2EE entries point has been collected, one P4 entry point is chosen from the list. E. The Diagnostics agent will then establish a P4 connection to the selected J2EE entry point with the host name and the port mapped to the P4 entry point.

©SAP AG 2009

End-to-End Diagnostics Diagnostic agent troubleshooting guide

19

Troubleshooting Message server connection Before starting the agent installation or before executing the smdsetup script, you can check the content of the SCS message server with the following procedure: 1. Open the URL: http://msgserver:81xx/msgserver/text/logon (xx corresponding to instance number of the SCS instance) 2. Check if a P4 entry point is available, 3. On the managed system, open a shell with the <SID>adm user of the Managed system or the SMD Agent. 4. Execute the command “ping <host of P4 entry point>”: The host must be accessible, 5. Execute the command “telnet <host of P4 entry point> <P4 port>”. If no error message occurs (blank screen) the connection is OK. Most common agent connection problems All connection problems are logged in a file named SMDSystem.X.log (see chapter 6.2.5). Check this log to find the root exception of the issue: Once the exception is found, find the root cause of the exception by locating the last “caused by” string.

No Entry point found
Exception: com.sap.engine.services.jndi.persistent.exceptions.NamingException: Exception while trying to get InitialContext. [Root exception is com.sap.engine.interfaces.cross.DestinationException: there are no access points for service: P4 registered on the message server]

No P4 entry point has been reported by the Message Server. This is probably due to the fact that it is not the correct message server: it must be the Java and not the ABAP one. So check the content of the Message server with the URL http://msgserver:81xx/msgserver/text/logon and verify if a P4 entry point is available (as shown below).

Unknown host
Exception: [sapms:// yupxxxxx.domain,com:8136] Checking server availability... Connecting to SMD server sapms://yupxxxxx.domain,com:8136 failed - error counter: 0 - com.sap.engine.services.jndi.persistent.exceptions. NamingException: Exception while trying to get InitialContext. [Root exception is com.sap.engine.interfaces.cross.DestinationException: cannot establish connection with any of the available instances #Nested exceptions are: #java.net.UnknownHostException: yupxxxxx: yupxxxxx ]

This is a network resolution problem due to the fact that the selected P4 hostname retrieved from the SCS Message server list cannot be resolved from the Managed system host. Retrieve the list of the P4 entries point in the Message server as: version 1.0 J2EE933894500 J2EE NCED6018XXXXA.dhcp.ncel.sap.corp J2EES NCED6018XXXXA.dhcp.ncel.sap.corp P4 NCED6018XXXXA.dhcp.ncel.sap.corp P4S NCED6018XXXXA.dhcp.ncel.sap.corp P4HTTP NCED6018XXXXA.dhcp.ncel.sap.corp

59300 59301 59304 59306 59305

LB=1 LB=1 LB=1 LB=1 LB=1

©SAP AG 2009

End-to-End Diagnostics Diagnostic agent troubleshooting guide Execute the command “ping <hostname_of_P4_entry_point_5xx04>” in a Managed system shell. If the host cannot be reached, then change the host definition in your DNS or in the /etc/hosts. You may alternatively fix you engine installation to report the right host information in the message server (see chapter 5.3).

20

Direct Connection to P4 Dispatcher

Solution Manager in cluster environment If the solution manager system is installed on a High Availability environment, this direct connection cannot be used because the agent would always establish a connection to the same java instance corresponding to the same p4 port. To enable HA, you must select the SAP J2EE Message server connection mode. In that case, the connection of the diagnostics agent will be load-balanced to the different available Java instances.

©SAP AG 2009

End-to-End Diagnostics Diagnostic agent troubleshooting guide

21

5.2

Managed system

All connections to the Managed systems from the Diagnostics agent are executed locally on the same host. It is the reason why it is really imperative to install the Diagnostics agent on the same virtual host than the SAP Managed system. The agent connects to the following different SAP System layers: Java stack on port P4 (5xx04) Sapstartsrv http port of Managed System (5xx13) J2EE Message server (39xx and 81xx) RFC call to Managed Abap stack

Troubleshooting P4 connection problems with Managed system Check on which host the P4 socket is bound o Run the OS command “netstat –an” to find the entry line containing the P4 port of the managed system:
TCP 0.0.0.0:59304 0.0.0.0:0 LISTENING

Or
tcp 0 0 *.50304 *.* LISTEN

In this example we see that the p4 port is bound for all hosts where a p4 request can be emitted (* or 0.0.0.0), if the port is bound like “10.45.192.204:50304” only the p4 requests emitted from the host 10.45.192.204 will be accepted. It explains why the Diagnostic agent must be installed on virtual host 10.45.192.204 with SAPInst. The situation can be fixed by changing the profile property SAPLOCALHOST (see 6.2.3 Change the virtual host name). o After verifying the port in the OS configuration, check in the J2ee dispatcher configuration that the property “bindHost” is set to 0.0.0.0. This should be visible with the J2EE configuration tool of the managed system->Instance_xxx->dispatcher->services->p4. If the value is not configured to 0.0.0.0 (which accepts all hosts), add the agent IP or install the agent on one of the listed IP.

©SAP AG 2009

End-to-End Diagnostics Diagnostic agent troubleshooting guide

22

Check if the agent can access the J2ee message server (SCS) of the Managed system with the host and the http port stored in the agent configuration (see the next section). For example : o o Open a shell with the agent OS user (smdadm) on the managed system. Execute the command “msprot -l -mshost <host_j2ee_msg_srv> -msserv 39<SCS> or 36<SCS> -d J2EE –s”, the values of host and port must be retrieved from the Agent configuration (see the next section). The command must return the following content :
SAP Message Server Protocol Program, Version 1.3 (built: Jan 24 2008 01:27:19) INFO CLUSTER-ID HOST ---> LIST ---> LIST ---> LIST ---> LIST ---> LIST ---> LIST ---> LIST ---> LIST ---> LIST ---> LIST ---> LIST ---> LIST ---> LIST ---> LIST ---> LIST ---> LIST ---> LIST ---> LIST ---> LIST GROUP-ID STATE J2EE-SERVICE-MASK ---------------------------------------------------------------------------------------------SERVICE:00000 NAME:ConfigurationManager SERVICE:00001 NAME:_CacheManager SERVICE:00003 NAME:{service_manager_deploy_distributor} SERVICE:00004 NAME:{service_manager_internal_connection} SERVICE:00005 NAME:shell SERVICE:00006 NAME:jmx_notification SERVICE:00007 NAME:p4 SERVICE:00008 NAME:webservices SERVICE:00010 NAME:iiop SERVICE:00011 NAME:r3startup SERVICE:00012 NAME:telnet SERVICE:00013 NAME:log_configurator SERVICE:00014 NAME:jmx SERVICE:00016 NAME:http SERVICE:00022 NAME:com.sap.security.core.ume.service SERVICE:00023 NAME:security SERVICE:00024 NAME:tc~sec~certrevoc~service SERVICE:00025 NAME:servlet_jsp SERVICE:00026 NAME:webdynpro 0000041052 RUNNING 0000041052 RUNNING XXXXXXXXOXXOXXXXXOXXXXXXXXXXXX XXXXXXOXXXXXXXXXXOOOOOOOOOOOOO

o

LIST 0004105250 ncevhost.mydomain LIST 0004105200 ncevhost.mydomain

In this content, the P4 service is listed as well as the j2ee nodes with hostnames (e.g. ncevhost.mydomain). Check if those hostnames can be resolved on your network by executing the command “ping ncevhost.mydomain” o Open the J2EE Visual admin of the Managed system and connect with the j2ee user used by the Diagnostics agent (agent configuration see next section) to check if the user can be authenticated and if it has the Administrator role. Check also the Managed system defaulttrace file to find some exception linked to P4 connection.

o

©SAP AG 2009

End-to-End Diagnostics Diagnostic agent troubleshooting guide

23

Connection configuration for managed systems

The following table lists all properties generated and updated by the Managed Setup Wizard allowing the Diagnostics agent to connect to the Managed system in order to complete the E2E RCA operations. Those properties are stored in the Diagnostics agent configuration under the scope named “server name”. o Managed systems connection information: Connection Type to Managed system
P4 port http port J2EE Message server Intern Port J2EE Message server Http port J2EE Message server host

Port
5xx04 5xx00 39xx 81xx <Hostname>

Configuration stored in agelet
com.sap.smd.agent.application.global.configuration com.sap.smd.agent.application.global.configuration com.sap.smd.agent.application.global.configuration com.sap.smd.agent.application.global.configuration com.sap.smd.agent.application.global.configuration

Property
[SID]/[INB]/sapj2ee/P4/port [SID]/[INB]/sapj2ee/http/port [SID]/sapj2ee/msgserver/port [SID]/[INB]/sapj2ee/http/port [SID]/sapj2ee/msgserver/host

o

Credentials : Configuration stored in agelet
com.sap.smd.agent.application.global.configuration

Credential
J2EE credentials

Property user
[SID]/sapj2ee/admin/user

Property password
[SID]/sapj2ee/admin/pwd

Credential
ABAP credentials

Configuration stored in agelet
com.sap.smd.agent.application.global.configuration

Property client
[SID]/its/client

Property user
[SID]/abap/com/user

Property password
[SID]/its/client

Those values might be only changed in the agent Admin tool in case of debugging. The general recommendation is to update those values with the Managed system setup wizard.

©SAP AG 2009

End-to-End Diagnostics Diagnostic agent troubleshooting guide

24

5.3

Full Qualified Name resolution issue

In most of the cases, a connection issue from the Diagnostics Agent to the Managing system or the managed reveals a network configuration issue. You will find below a list of SAP notes which can help to solve some network resolution problems in different part of SAP products. SAP Note Note 611361 Hostnames of SAP servers Note 962955 Use of virtual TCP/IP host names Note 773830 FQHN determination in ICM Note 962955 Use of virtual TCP/IP host names Note 1282975 Use of virtual TCP/IP host names in Windows Note 129997 Hostname and IP address lookup Note 1052122 Host names in SLD Note 979277 Incorrect registration of computer systems in the SLD Resume The use of hostnames that are too long or contain illegal characters may result in undetermined errors You want to use one or more virtual host names for SAP software within an SAP server landscape to hide the physical network identities for the pieces of software from each other. It is not clear which fully qualified host name (FQHN) the Internet Communication Manager (ICM) is using. You want to use one or more virtual host names for SAP software within an SAP server landscape to hide the physical network identities for the pieces of software from each other. You want to use one or more virtual host names for SAP software within an SAP server landscape to hide the physical network identities for the pieces of software from each other. Information about IP addresses lookup. You wonder if a hostname that occurs in the System Landscape Directory (SLD) is correct. In the SLD, hostnames primarily occur in technical systems and destinations In computer systems registered by the ABAP data supplier in the SLD, attributes may be missing in certain cases (for example, details of the operating system).

5.4

SAP router

In some network configuration, the Diagnostics agent has to connect to the Managing system though a SAP router. This feature will be fully supported in Solution Manager 7 Ehp2 (7.02). In order to enable this feature in Ehp1, please contact the Diagnostics Agent support team to get information on how to setup the agent and on which patch level you can perform the installation.

©SAP AG 2009

End-to-End Diagnostics Diagnostic agent troubleshooting guide

25

5.5

Self-Monitoring

During a self-monitoring setup of Solution Manager, some special configurations in the different setup wizard are needed due to some restrictions about the local agent connectivity. All information about the setup configuration of the “self-monitoring” is available in the following SAP Notes:

SAP Note 1273028 - Introscope 8 Release Notes 917080 - SMD Auto-Monitoring : single user for SMD Agent setup 1292811 - Wily setup wizard is failing in auto-monitoring system

Resume BCI Agent 8.0 cannot be used for SMD self-monitoring. Please use Java agent version 7.2 for self monitoring of the Solution Manager Java stack.
DEPRECATED, DO NOT USE ANYMORE

An error occurs when performing the setup of a wily introscope agent on solution manager itself (self-monitoring) although the user and password are correct.

Error in Managed system setup wizard in Self-monitoring

End of stream is reached unexpectedly during input from Socket
Exception: com.sap.smdagent.plugins.connectors.p4.exceptions.P4AuthorizationException: Access is denied: Please make sure the <user> and <password> parameters are correct.; nested exception is: com.sap.engine.services.jndi.persistent.exceptions.NamingException: Exception during getInitialContext operation.No server is running. [Root exception is com.sap.engine.services.security.exceptions.BaseLoginException: Exception in creating new RemoteLoginContext instance.] at com.sap.smd.agent.plugin.connectors.p4.P4ConnectionService.getP4Connection(P4ConnectionServ ice.java:69) ... at java.lang.Thread.run(Thread.java:838) Caused by: com.sap.engine.services.jndi.persistent.exceptions.NamingException: Exception during getInitialContext operation.No server is running. Root exception is com.sap.engine.services.security.exceptions.BaseLoginException: Exception in creating new RemoteLoginContext instance. at com.sap.engine.services.security.remote.login.RemoteLoginContextExt.<init>(RemoteLoginConte xtExt.java:34) ... at java.lang.Thread.run(Thread.java:838) Caused by: java.io.EOFException: End of stream is reached unexpectedly during input from Socket [addr=/56.zzz.xx.xx,port=50204,localport=22326] at com.sap.engine.services.rmi_p4.Connection.run(Connection.java:423)

The problem is linked to the value if the host name registered in the Solution Manager SCS message server. Check the hostname stored in SCS message server with the following url http://scs_host:81<SCS_instance_number>/msgserver/text/logon. The agent hostname reported in agent admin and SCS hostname of entry points must be equal. Since 7.01 Ehp1 SPS19, add the property <SID>/selfmonitoring/enabled=true in the agelet global configuration of the Diagnostics agent XD XD which do the self monitoring to avoid some auto-monitoring connection issue.

©SAP AG 2009

End-to-End Diagnostics Diagnostic agent troubleshooting guide

26

6
6.1

Agent Maintenance
Upgrade SAP Kernel (saposcol update)

For specific agent issues, you might be asked to upgrade the SAP kernel binaries of the Diagnostics agent (Ex: This is requested when upgrading saposcol). In that case, follow the procedure:

For 7.00 and 7.01
Determine the SID and instance number of the Diagnostics agent to upgrade. Determine the SAP Kernel version by opening a shell with <sid>adm for Unix or command prompt Windows in folder /usr/sap/<SID_AGT>/exe Execute the command “sapcontrol -prot NI_HTTP -nr <Instance_Number_AGT> -function GetVersionInfo”
GetVersionInfo OK Filename, VersionInfo, Time C:\usr\sap\SMD\J98\..\exe\sapstartsrv.exe, 700, patch 144, changelist 957098, optU, NTintel, C:\usr\sap\SMD\J98\..\exe\msg_server.exe, 700, patch 144, changelist 957098, optU, NTintel, C:\usr\sap\SMD\J98\..\exe\enserver.exe, 700, patch 144, changelist 957098, optU, NTintel, C:\usr\sap\SMD\J98\..\exe\jcontrol.exe, 700, patch 144, changelist 957098, optU, NTintel, C:\usr\sap\SMD\J98\..\exe\jlaunch.exe, 700, patch 144, changelist 957098, optU, NTintel,

Stop all Diagnostics agents with same SID (because they share the binaries) Download the latest version of the SAP Kernel on service market place. Be careful if the agent kernel version is 700 you must upgrade to the latest available version of the 700 kernel. You shouldn’t upgrade a 700 kernel to the latest version of the 701 kernel. Extract the SAR File (ex: SAPEXE_179-20000978.SAR) in folder /usr/sap/<SID_AGT>/exe

For 711
Determine the SID and instance number of the Diagnostics agent to upgrade. Determine the SAP Kernel version, by opening a shell with <sid>adm for Unix or command prompt Windows in folder /usr/sap/<SID_AGT>/SYS/exe/uc/<OS>. Execute the command “sapcontrol -prot NI_HTTP -nr <Instance_Number_AGT> -function GetVersionInfo”
/usr/sap/DAA/SMDA96/exe/sapstartsrv, 711, patch 9, changelist 1010123, optu; sun_64, /usr/sap/DAA/SMDA96/exe/gwrd, 711, patch 9, changelist 1010123, optu , sun_64, /usr/sap/DAA/SMDA96/exe/icman, 711, patch 9, changelist 1010123, optu , sun_64, /usr/sap/DAA/SMDA96/exe/jcontrol, 711, patch 9, changelist 1010123, optu , sun_64, /usr/sap/DAA/SMDA96/exe/jlaunch, 711, patch 9, changelist 1010123, optu , sun_64, /usr/sap/DAA/SMDA96/exe/jstart, 711, patch 9, changelist 1010123, optu , sun_64,

Stop all Diagnostics agents with same SID. Download the latest version of the SAP Kernel 711 on service market place. (do not use SAP Kernel 710) Extract the SAR File (ex: SAPEXE_20-20003746.SAR) in the folder /usr/sap/<SID_AGT>/SYS/exe/uc/<OS>

6.2

Configuration

This section describes all the procedures to update the Diagnostics agent configuration elements which cannot be changed centrally from Solution Manager.

©SAP AG 2009

End-to-End Diagnostics Diagnostic agent troubleshooting guide

27

6.2.1 Change Path of the JDK
If the JDK path changes after the Diagnostics agent installation, you need to update the path of JDK in the agent profile file.

For 7.00 and 7.01
Open the profile <SID_AGT>_J<INSTANCE_NUMBER>_<hostname> in folder
/usr/sap/<SID_AGT>/sys/profile.

Update the parameter jstartup/vm/home by putting the absolute path to the JDK home. Remove the file /usr/sap/<SID_AGT>/JXX/profile/smd.properties.vmprop. Restart the Diagnostics agent.

For 7.11
As the Diagnostics agent 711 runs only on SAPJVM5, it is not recommended to change the path of the JDK.

6.2.2 Change JVM Parameters
If the JVM parameters need to be changed, you have to update the agent instance properties file.

For 7.00 and 7.01
Open the file smd.properties in folder /usr/sap/<SID_AGT>/J<Instance_Number>/profile. Update the parameter “smdagent.javaParameters” (see 4.4 JVM parameters). Remove the file /usr/sap/<SID_AGT>/JXX/profile/smd.properties.vmprop. Restart the Diagnostics agent.

For 7.11
Open the file smdagent.properties in the folder /usr/sap/<SID_AGT>/SMDA<Instance_Number>/SMDAgent. Update the parameter “smdagent.javaParameters” (see 4.4 JVM parameters). Remove the file /usr/sap/<SID_AGT>/JXX/profile/smd.properties.vmprop. Restart the Diagnostics agent.

6.2.3 Change the virtual host name
If the Diagnostics agent host name reported in the Agent Admin tool is not the expected name, you can change it in the agent instance profile.

For 7.00, 7.01 and 711
Open the profile <SID_AGT>_J<INSTANCE_NUMBER>_<hostname> in the folder /usr/sap/<SID_AGT>/sys/profile. Update or add the parameter SAPLOCALHOST by setting the short hostname desired (not the FQN).

©SAP AG 2009

End-to-End Diagnostics Diagnostic agent troubleshooting guide

28

6.2.4 Usage of smdsetup script
The script ’smdsetup’ allows to update some configuration parameters for Diagnostics agent. This script is delivered in the Diagnostics Agent since Solution Manager SPS16. However, this script is always automatically patched to the latest version when the Diagnostics agent connects to the SMD server. So if the LMSERVICE has a version greater than SPS15, the smdsetup script of the Diagnostics agents will be updated to the last available version. To find out what is the version of the smdsetup script, you have to execute the command “smdsetup.sh” or “smdsetup.bat” without any parameters o If the command output asks 5 parameters like “smdsetup.sh $JAVA_HOME smdserverhost 53004 j2eadmin abc123 myhost”, it is a smdsetup with a version less than SPS13. Example of smdsetup SP13 output:
SMD Agent Setup expects the following parameters: 1) Path to the Java Virtual Machine - JDK 1.4 is required 2) Host Name (or IP Address) of the SMD Server (managing SYSTEM) Note: Make sure to prefix the host name by sapms:// when using the message server port. 3) Message Server HTTP Port (ex: 8131) or P4 Port (ex: 50004) of the SMD Server 4) User (J2EE User member of 'Administrator' Role) 5) Password of the J2ee User 6) The so called 'SMSY Server Name' which is the unique key used to declared the host in Solution Manager.

o

If the output proposes some action as “managingconf”, the version of smdsetup version is greater than SPS14. Example of smdsetup SP14+ output:
>> Help Information ---------------------Script which allows to update some diagnostics agent configuration. smdsetup <action> <parameters> List of actions: * smdsetup secstorej2ee user:"<value>" pwd:"<value>" * smdsetup secstoresld user:"<value>" pwd:"<value>" * smdsetup sldconf hostname:"<fqn>" port:"<http port>" [optional user:"<value>" pwd:"<value>"] * smdsetup managingconf hostname:"sapms://<fqn>" port:"<MS httpPort>" [optional user:"<...>" pwd:"<...>" servername:"<...>"] * smdsetup managingconf hostname:"<fqn>" port:"<p4 port>" [optional user:"<value>" pwd:"<value>" servername:"<value>"] ….

SMDSetup version <= SPS13
The smdsetup script in this version is located in /usr/sap/<SID_AGT>/J<Instance_Number>/SMDAgent.

©SAP AG 2009

End-to-End Diagnostics Diagnostic agent troubleshooting guide

29

In this version the script permits only to change the connection information and the credentials at the same time. It means that all the input parameters are mandatory. Classpath error In case of a classpath error, you need to adapt the classpath of the smdsetup script SP12 or lower, because in SPS13 the logging.jar has moved from the SMDAgent/lib/launcher directory to the SMDAgent/lib directory.

SMDSetup version >= SPS14
In 7.0 SR3 and 7.0 the smdsetup script in this version is located in the following folder: o /usr/sap/<SID_AGT>/J<Instance_Number>/script In 7.11 the smdsetup script is located in the following folder: o /usr/sap/<SID_AGT>/SMDA<Instance_Number>/script

Action managingconf sldconf secstorej2ee secstoresld changeservername addsaprouter saprouterpass removesaprouter supportlogs

Description Change the managing system of the agent (host and port), restart agent after the changes. Change the SLD server where the agent is registered (host and port), restart agent after the changes. Change only the J2EE user used to connect to the Managing system. (restart after the changes) Change only the J2EE user used to register the agent to the SLD system. (restart after the changes) Change the server name of agent. (restart after the changes) Full Implementation in Ehp2, so don’t use this command Full Implementation in Ehp2, so don’t use this command Full Implementation in Ehp2, so don’t use this command Execute some agent health-checks and generate a zip containing all agents logs (including the agent check report)

Availability SPS14 SPS14 SPS14 SPS14 SPS14 SPS18 SPS18 SPS18 SPS17 patch 3

MANAGINGCONF SYNTAX

To execute this command you need to stop the agent. The command updates the connection settings to the Managing system: Connect the agent by P4 direct connection:
smdsetup managingconf hostname:”myhost” port:”53004”

Connect the agent by J2EE Message server http port (not internal message server port 36xx):
smdsetup managingconf hostname:”sapms://myhost.domain.corp” port:”8101”

Optionally, you can add the j2ee user information in the command line to update also the agent secure store:
smdsetup managingconf hostname:”myhost.domain,corp” port:”53004” user:”SMD_ADMIN” pwd:”init1234”

©SAP AG 2009

End-to-End Diagnostics Diagnostic agent troubleshooting guide
SLDCONF SYNTAX

30

To execute this command you need to stop the agent. The command updates the connection settings to the SLD system of the agent registration: Connect the agent to the SLD system by the http port of java stack.
smdsetup sldconf hostname:”mySLDhost.domain.corp” port:”50000”

Optionally, you can add the sld user information in the command line to update the agent secure store:
smdsetup sldconf hostname:”mySLDhost.domain.corp” port:”50000” user:”slddsuser” pwd:”xxxxx”

SECSTOREJEE SYNTAX

To execute this command you need to stop the agent. The command updates the credentials of the connection to Managing system: Update the agent secstore file:
smdsetup secstorej2ee user:”SMD_ADMIN” pwd:”init1234”
SECSTORESLD SYNTAX

To execute this command you need to stop the agent. The command updates the credentials of the connection to the SLD system: Update the agent secstore file
smdsetup secstoresld user:”slddsuser” pwd:”xxxxxx”
CHANGESERVERNAME SYNTAX

To execute this command you need to stop the agent. The command updates the “servername” attribute of the agent: Update the agent configuration file
smdsetup changeservername servername:"myservername"
SUPPORTLOGS SYNTAX

The Diagnostics agent must be started before executing this action. This command retrieves all agent log files and executes some basics tests to check the agent pre-requisites as: jdk version check jvm parameters check hostname resolution check tcp port check p4 protocol check message server connectivity check To execute the command
smdsetup supportlogs

At the end of the execution of the command, a Zip file is generated in /usr/sap/<AGENT_SID>/<AGENT_INSTANCE_NUMBER>/SMDAgent/supportLogs_<TIMESTAMP>.zip. The zip content is the following: SMD Agent log files SMD Agent profile SAPInst logs Work directory All properties files contained in the SMD agent folder.

©SAP AG 2009

End-to-End Diagnostics Diagnostic agent troubleshooting guide

31

6.2.5 Change Agent log level
The Diagnostics agent logs are stored in /usr/sap/<SID_AGT>/<InstNbr>/SMDAgent/log folder and organized as follow: Filename SMDSystem.log Default settings 10 files of 1MBytes Default level INFO Description This file reports all information about agent framework like connection to managing system, SLD report, agelets loading and patch sequence. This file reports all information about the agelets used by RCA applications, Setup Wizards and Wily IS setup. This file reports all information about agent connection layer with managing system.

SMDAgentApplication.log

10 files of 1Mbytes

WARNING

P4.log

10 files of 1MBytes

NONE

When changing the log level of the agent logging, you need to consider two cases: o The Diagnostic agent is connected to the Solution Manager System: Go in Agent Administration tool, select the tab “Agent Log viewer” Select the agent click on display button Select the location In section “Log Level Control”, select the level and click on the “apply” button. o The Diagnostics agent is not connected to the Solution Manager System: Stop the agent Go in folder /usr/sap/<SID_AGT>/<InstNbr>/SMDAgent/configuration Open the file log-configuration.xml Adapt the level (attribute effective-severity) in following line (level values: ALL,DEBUG,INFO,WARNING,ERROR,NONE): For Filename SMDSystem.log SMDAgentApplication.log P4.log

<log-controller effective-severity="INFO" maximum-severity="ALL" minimumseverity="DEBUG" name="/SMDlogger/System"> <log-controller effective-severity="WARNING" maximum-severity="ALL" minimumseverity="DEBUG" name="/SMDlogger/AgentApp"> <log-controller effective-severity="NONE" maximum-severity="ALL" minimumseverity="ALL" name="com.sap.engine.services.rmi_p4">

Save the file and restart the agent.

©SAP AG 2009

End-to-End Diagnostics Diagnostic agent troubleshooting guide

32

6.3

Managing J2EE user

Information about the agent j2ee user who connects to the Solution Manager System, is described in the “User Administration guide”, section 2.3.3 [SOLMAN.AGT.COM]: Diagnostics agent Communication User. If the J2EE user used by the Diagnostic agent cannot be authenticated (wrong password, missing role assignment, user locked), the agent will shutdown automatically to avoid locking the corresponding account. Most common agent connection problems Authentication failed all authentication problems are logged in file SMDSystem.log.
Exception: at com.sap.engine.core.thread.impl3.SingleThread.run (SingleThread.java:170) Caused by: com.sap.engine.services.security.exceptions.BaseLoginException: Authentication did not succeed. at com.sap.engine.services.security.login.ModulesProcessAction.run (ModulesProcessAction.java:175) ... 13 more

Before restarting the agent It is important to execute some checks related to the J2EE user as follow: 1. Start the ABAP transaction SU01 on Solution Manager. Check if the J2EE user (SMD_ADMIN) is not locked Check that the Role assigned is compliant to SAP recommendations done in the “User Administration guide”. 2. Logon to the Agent administration tool with the J2EE user (SMD_ADMIN) to check that the logon module of the Java stack works correctly. 3. Call the smdsetup to update only the credentials to be sure the password is up-to-date. 4. Restart the Diagnostic agent. In dev_smdagent.log, if you find the exit code -11112, it is an authentication problem as described above. [Thr 880] Wed Feb 04 09:28:47 2009
[Thr 880] JLaunchIStartFunc: end of main method (rc = 0) [Thr 6000] JLaunchIExitJava: exit hook is called (rc = -11112) [Thr 6000] ************************************************************* *** ERROR => The Java VM terminated with a non-zero exit code. *** Please see SAP Note 943602 , section 'J2EE Engine exit codes' *** for additional information and trouble shooting. ********************************************************************** [Thr 6000] JLaunchCloseProgram: good bye (exitcode = -11112)

©SAP AG 2009

End-to-End Diagnostics Diagnostic agent troubleshooting guide

33

7
7.1

Tracking agent issues
Deprecated SAP notes

The following table contains a list of SAP notes which are now deprecated:

SAP Note 985140 906248 987092 954861 985138 1037767 1055798 1069588

Short description SMD Agent/SAPinst: Virtual Host is not taken into account Problem with missign CA directory with NW04s SP5 SMD install Install an SMD Agent <= SP11 on IBM System i (i5/OS, OS400) SMD agent installation with SAP Inst failed (SSL problem) SMD Agent/SAPinst/IBM JDK: Java version wrongly detected SMD Agent/SAPinst: DNS Domain name error on AIX smdagent.sh start: java.lang.NoClassDefFoundError SMD Agent/SAPinst: How to use an alternate System Name

State Deprecated Deprecated Deprecated Deprecated Deprecated Deprecated Deprecated Deprecated

7.2

OSS message pre-requisites
The Diagnostics agent has been installed with SAPInst SR3, Ehp1 or 7.11. The OS limit for Unix platform are compliant with the agent OS user (<AgentSID>adm), JDK vendor and version is supported by Diagnostics agent, JVM parameters of Diagnostics agent are compliant, The Diagnostics agent connection to the Managing system is of type J2EE Message Server connection, Recommendations for the auto-monitoring scenario Managed setup wizard execution have been applied. The Java dispatcher of Solution Manager is correctly sized to avoid some agent disconnection (see part 8.1 Managing java stack settings.)

Before forwarding any OSS messages to the SAP support organization, it is important to check the following points:

If those check points cannot solve the problem, follow the SAP Note 1076573 to create the OSS message. If the LMService version is greater or equal to NW04S SPS17 Patch 3, attach the support logs Zip file generated by the smdsetup script (see part 6.2.4).

©SAP AG 2009

End-to-End Diagnostics Diagnostic agent troubleshooting guide

34

7.3

Agent crash investigation

When the system process of the Diagnostics agent shutdowns abnormally, the procedure to determine the cause of this crash is the following: Go in folder /usr/sap/<SID_AGT>/J<INSTANCE_NUMBER>/work Open the log file dev_jcontrol (This is the log file of the jcontrol process which is the controller of the java agent process) to check if the jcontrol has started the jlaunch process as described below:
[Thr 3288] JControlICheckProcessList: process SMDAgent started (PID:3616)

If the process jlaunch cannot be start, the error is probably located in SAP profile parameters

The message JStartupICheckFrameworkPackage: can't find framework package C:\usr\sap\SMD\exe\jvmx.jar is NOT a critical error. The Diagnostics agent does not use specific SAP kernel functions and this message can be ignored. If the jlaunch process has been started, open the file dev_SMDAgent which corresponds to the jlaunch process output :
Thr 3860] JLaunchISetState: change state from [Initial (0)] to [Initial (0)] [Thr 3860] JLaunchISetState: change state from [Initial (0)] to [Waiting for start (1)] [Thr 3860] Thu Dec 25 12:11:35 2008 [Thr 3860] JLaunchISetState: change state from [Waiting for start (1)] to [Starting (2)] [Thr 3860] JLaunchISetState: change state from [Starting (2)] to [Running (3)]

In this file, all information about the java of the Diagnostics agent is stored. You will information related to JVM loading, process state, crash of VM or bad JVM parameters. In case of process crash, you will find in this file, the corresponding Exit Code or Error message. JVM Loading problem all JVM loading problems are logged in file dev_SMDAgent.
Error: [Thr 1084238144] *** ERROR => DlLoadLib: dlopen()= libjvm.so: cannot open shared object file: No such file or directory -> DLENOACCESS (0, Success) [dlux_mt.c 314] [Thr 1084238144] *** ERROR => Can't load VM shared library (libjvm.so) (rc=-2) [jhvmxx_mt.c 1576] [Thr 1084238144] *** ERROR => Cannot load DLL for Java VM [jlnchxxi_mt.717]

You will find potential JDK installation issues like : Wrong JDK version for Agent JDK is not compatible with SAP Kernel (32bits/64bits) Agent OS user have not rights to access to JDK Libraries The SAP Note 940893 - Analysis of errors within the startup of J2EE Engine 6.40 can help to troubleshoot JVM loading problems.

If the JVM is loaded correctly, now check the file std_SMDAgent.out and jvm_SMDAgent.out to find some errors like OutOfMemory, StackOverflow or Nullpointer which could have stopped the agent process.

©SAP AG 2009

End-to-End Diagnostics Diagnostic agent troubleshooting guide

35

7.4

Thread dump

Facing a deadlock situation or a Diagnostic agent freeze, the only correct way to understand the cause of the problem is to perform a thread dump on the agent process.

Make a Thread dump under Windows
The Diagnostics agent is like any other SAP system, available in the SAP MMC.

In SAP MMC, open the node “AS Java Process Table” of the Diagnostics agent identified by the SID and the instance number. Only one process is displayed in the right part: It is the agent process. Select this process and right click on it to display the context menu, press the item “Dump stack Trace”.

Make a Thread dump under Unix
Under UNIX platform, you need to use the command smdadmin.sh.

©SAP AG 2009

End-to-End Diagnostics Diagnostic agent troubleshooting guide

36

Start in an UNIX shell with the OS user of the Diagnostics agent. Execute the command “smdadmin.sh XX” (XX is the instance number of Diagnostics agent). For old agent installation, it is necessary to execute the command “../smdadmin.sh” from the folder /usr/sap/<SID_AGT>/J<InstanceNumber>/script. Once the command is executed, the JCMON menu is displaying, select the entry “20: Local administration menu” then execute the action “10: Dump stacktrace” on process 0 (This is the process of the SMDAgent).

Where is the thread dump stored?
Once the thread dump is done, the output is written in file /usr/sap/<SID_AGT>/J<InstanceNumber>/work/
std_SMDAgent.out for Sun and HP-UNIX JDK.

For IBM JDK, the thread dump output is written in java.core.<timestamp>.txt in folder usr/sap/<SID_AGT>/J<InstanceNumber>/SMDAgent.To find the absolute path of an IBM thread dump, open the file std_SMDAgent.out and find the line beginning with “Java Dump written to”.

7.5

Activate P4 logs

In case of a persistent p4 connection problem with the managing system or the managed system (like timeout, connection reset, ..) , you should enable the P4 logs before sending the OSS message to the Development Support organization.

Connection problem to Managing System or Managed system
The procedure to enable the P4 logs on the Diagnostics agent is the following: Stop the agent Clean the log folder under SMDAgent directory Set the log level (effective-severity) of SMDSystem location to ALL (see 6.2.5 Change Agent log level) Set the log level (effective-severity) of P4 location to ALL (see 6.2.5 Change Agent log level) Add the Java parameter –Ddebug=true in agent configuration (6.2.2 Change JVM Parameters) Once the p4 log enabled on agent, go to Managing system or Managed system to execute the following instructions: Open the Visual Administrator of Java stack Select the server node Under SID node Select the service “Log Configurator” Go in tab Locations Select the location com.sap.engine.services.rmi_p4 Increase the log level to DEBUG and save the modification. Once all P4 locations are configured, restart the agent to reproduce the problem and attach the agent logs (use the command ‘smdsetup supportlogs’) and the current defaulttrace.log of the Managing or the Managed system to the OSS message. Performance Impact It is important to restore All log locations changed to their default level because the procedure above has a severe impact on the performance of the Managed system or the Managing system.

©SAP AG 2009

End-to-End Diagnostics Diagnostic agent troubleshooting guide

37

7.6

Telnet SMD server commands

Some telnet commands are available in the Solution Manager Java stack to check the installation and configuration of the SMD server. To use those commands, start a telnet client to the SAP J2EE server of Solution Manager (default port 5xx18) and go on server node with the command ‘jump <nodeid>’. Add the group smdserver to enable the SMD commands by typing use the command ‘add smdserver’. Once the group smdserver is added, execute the command ’man –g smdserver’ to see all available SMD server commands. SMD server telnet command Smdstatus Smdstatus –a list_agents find_agent <servername> or <fqn> mm ON mm OFF kill_agent <servername> or <fqn> start_agent <servername> or <fqn> stop_agent <servername> or <fqn> host_mapping known_agents -c known_agents -n as es Description Display the version and running mode of SMD server. Check connectivity of all agents Display all diagnostics agents connected to SMD server with their status. Retrieve an agent from name, fully qualified domain name or IP. Enable the maintenance mode of SMD server Disable the maintenance mode of SMD server Reboot the Diagnostics agent process Start the agent applications of selected Diagnostics agent Stop the agent applications of selected Diagnostics agent Display the mapping between server name and network names for all agents. Displays the list of known agents. (Agents that have been registered at least once to the SMD Server) Displays the list of known agents by FQN. Provides ping statistics for all agents. Provide statistics usage for the SMD server.

8
8.1

Scalability
Managing java stack settings

We recommend adapting the J2EE configuration of Solution Manager Java stack if an important number of Diagnostics agents will connect to Solution Manager. The Java dispatcher has an important role in the agent connection’s stability. Therefore, you need to adapt the maximum number of threads in the pool and in the heap size configuration of the dispatcher to avoid some timeout connection issues. Dispatcher Parameter Java Max Heap Size (-Xmx) ThreadManager->MaxThreadCount Recommended Value Number of agents installed * 1.28 (minimum 128MB) ~ Number of agents installed * 1.75

Default value of MaxThreadCount After an installation of Solution Manager, the default value of parameter MaxThreadCount in Java Dispatcher configuration is set to 70. These settings allow by default to connect a maximum of 40 agents.

©SAP AG 2009

Sign up to vote on this title
UsefulNot useful