®

IBM Software Group

Diagnosing WebSphere Application Server Hangs on AIX
Ricky Marley, Advisory Software Engineer, WebSphere Application Server L2 Support Team Lead

WebSphere® Support Technical Exchange

IBM Software Group

Agenda
Background/Overview What Data Should be Collected to Diagnose the Problem Javacores - Basics - Thread Analysis - Lock Analysis - Understanding Javacores from a WebSphere AppServer Summary Questions?

WebSphere® Support Technical Exchange

2

IBM Software Group What is a Hang or Degradation in Performance? Hang is a condition where the Java Virtual Machine (JVM) become becomes unresponsive for client requests When a client complains about hang.Transports : 0" (37c18e37) has been active for 680. WebSphere® Support Technical Exchange 3 . There are 1 threads in total in the server that may be hung. the below message will be written into SystemOut file when a hung thread is detected: [7/15/04 15:03:11:502 EDT] 3c3b4e37 ThreadMonitor W WSVR0605W: Thread "Servlet.839 milliseconds and may be hung.1.Engine.1. first thing to understand is “What type of requests to WebSphere are unresponsive?” Starting V5.

Web Container (JVM). or database Synchronization of Java code in the JVM Web Container waiting for a response from an external resource Limitation of resources CPU Memory (typically Java heap) WebSphere® Support Technical Exchange 4 .IBM Software Group What Can Cause a Hang? Circular dependency in application code causing a deadlock in JVM Bottleneck caused by: Improper tuning of the webserver.

ibm.IBM Software Group Threadpools and Their Role in the Appserver Within the WebSphere Application Server process. there are various types of thread pools that serve client requests: WebContainer Thread pool Object Request Broker (ORB) Thread pool Data Replication Service (DRS) Thread pool Java Message Service (JMS) Thread pool Alarm Thread pool that will be reused for various purposes SOAP Connector Thread pool Application defined Thread pool The following link illustrates the various states in which you may find a WebContainer thread: http://www-1.com/support/docview.wss?uid=swg21137491 WebSphere® Support Technical Exchange 5 .

tprof Memory utilization – verbose GC data WebSphere 5. Documentation that needs to be collected at the time of the problem can be found by clicking on the link for “Hangs / Performance Degradation” at the following website: http://www-1.X – /usr/WebSphere/AppServer WebSphere 6.wss?uid=swg21145599 These documents include instructions for gathering: Javacores • Will be found in the AppServer Working Directory netstat CPU utilization .IBM Software Group Data Needed to Debug an AppServer Hang Issue Application logs alone cannot be used to debug this type of problem.ibm.X ./usr/WebSphere/AppServer/profiles/server1 WebSphere® Support Technical Exchange 6 .com/support/docview.

Java™ thread dump or a thread dump. All of the monitors on a JVM. Some useful information about the system that the JVM runs under. The data presented in the following sections: All of the threads that run on a Java™ Virtual Machine (JVM).IBM Software Group Introduction to Javacores – The Basics A javacore . WebSphere® Support Technical Exchange 7 . is a file that contains a Large Amount of useful information about the JVM. also known as a Java™ dump.

IBM Software Group Introduction to Javacores – Dump Routines The following chart presents the logical subcomponents in which a Javacore presents information from the JVM. Title XHPI Subcomponent Tags CI Subcomponent LK Subcomponent XM Subcomponent CL Subcomponent WebSphere® Support Technical Exchange 8 . To debug a performance degradation. focus should be placed in the LK and XM subcomponents.

The TITLE subcomponent contains the date and time of occurrence and the location within your file system.IBM Software Group Introduction to Javacores – Dump Routines The following figure is an example of the “Title” dump routine which presents the reason the JVM produced the Javacore file.txt ------------------------------------------------------------------------ WebSphere® Support Technical Exchange 9 .1156877224. NULL 0SECTION NULL 1TISIGINFO 1TIDATETIME 1TIFILENAME NULL -----------------------------------------------------------------------TITLE subcomponent dump routine =============================== signal 3 received Date: 2006/08/29 at 14:47:04 Javacore filename: /usr/WebSphere/AppServer/javacore29534. You can use this information to determine if the Java dump was created by a user or was system generated.

IBM Software Group Introduction to Javacores – Dump Routines The following figure is an example of the “XHPI subcomponent” dump routine.4.17 OS Level : AIX 5.0 Environment Variables --------------------MANPATH=/usr/dt/man:/usr/share/man WebSphere® Support Technical Exchange 10 .115.rtp.raleigh.3. J2RE 1.2 IBM AIX build ca1420-20040626 Operating Environment --------------------Host : aixwas3. This section will identify the operating environment.42.com:9. user limits. memory information.0. -----------------------------------------------------------------------XHPI subcomponent dump routine ============================== Tue Aug 29 14:47:04 2006 SIGQUIT received at 0x0 in <unknown>. NULL 0SECTION NULL 1XHTIME 1XHSIGRECV 1XHFULLVERSION … 1XHOPENV NULL 2XHHOSTNAME 2XHOSLEVEL … 1XHENVVARS NULL 2XHENVVAR …. and other operating system related details.ibm.

.net..2 IBM AIX build ca1420-20040626 Running as a standalone JVM /usr/WebSphere/AppServer/java/bin/java -Djava. Use this section to identify the JVM build./jre/bin Sys Classpath: /usr/WebSphere/AppServer/java/bin/.preferIPv4Stac Java Home Dir: /usr/WebSphere/AppServer/java/bin/. NULL 0SECTION NULL 1CIJAVAVERSION 1CIRUNNINGAS 1CICMDLINE 1CIJAVAHOMEDIR 1CIJAVADLLDIR 1CISYSCP 1CIUSERARGS 2CIUSERARG 2CIUSERARG 2CIUSERARG 2CIUSERARG 2CIUSERARG 2CIUSERARG ---------------------------------------------------------------CI subcomponent dump routine ============================ J2RE 1./jre/lib/co UserArgs: vfprintf 0x30000EF4 -Djava..4.preferIPv4Stack=true -Dwas. the JVM system property variables.net.socket=35676 -Xbootclasspath/p:/usr/WebSphere/AppServer/java/jre/li -Xms50m -Xmx256m WebSphere® Support Technical Exchange 11 . the class path that the JVM uses./jre Java DLL Dir: /usr/WebSphere/AppServer/java/bin/.IBM Software Group Introduction to Javacores – Dump routines The following figure is an example of the “CI subcomponent” dump routine.status. and other system information related to the JVM.

Transports : 0" (0x450914A0) WebSphere® Support Technical Exchange 12 . 0SECTION LK subcomponent dump routine NULL ============================ NULL 1LKPOOLINFO Monitor pool info: 2LKPOOLINIT Initial monitor count: 32 2LKPOOLEXPNUM Minimum number of free monitors before expansion: 5 2LKPOOLEXPBY Pool will next be expanded by: 182 2LKPOOLTOTAL Current total number of monitors: 364 2LKPOOLFREE Current number of free monitors: 163 NULL 1LKMONPOOLDUMP Monitor Pool Dump (flat & inflated object-monitors): 2LKMONINUSE sys_mon_t:0x44F79558 infl_mon_t: 0x00000000: 3LKMONOBJECT java. This Component provides information for all of the monitors in the JVM. You can use this section to analyze how resources are being used by various threads.Object@359F1130/359F1138: Flat locked by thread ident 0x26 3LKNOTIFYQ Waiting to be notified: 3LKWAITNOTIFY "Servlet.lang.Engine.IBM Software Group Introduction to Javacores – Dump Routines The following figure is an example of the “LK subcomponent” dump routine.

java:36) 4XESTACKTRACE at TE02Servlet.service(HttpServlet. .Engine. You can use the stack trace to relate the current activity of thread with the source code. sys_thread_t:0x412CDAA0.HttpServlet. This section provides a detailed list of all the threads and their states. sys_thread_t:0x4660CD20 4XESTACKTRACE at TE02Servlet.http.java:853) .2 IBM AIX build ca1420-20040626 3XMTHREADINFO "Servlet.servlet.service(HttpServlet. WebSphere® Support Technical Exchange 13 .java:18) 4XESTACKTRACE at javax. It also provides the current stack trace for each thread listed.doLock02(TE02Servlet.IBM Software Group Introduction to Javacores – Dump Routines The following figure is an example of the “XM subcomponent” dump routine.doPost(TE02Servlet. 0SECTION XM subcomponent dump routine NULL ============================ NULL 1XMCURTHDINFO Current Thread Details NULL ---------------------3XMTHREADINFO "Signal dispatcher" (TID:0x300FB960.stack address not valid1 XMTHDINFO All Thread Details NULL -----------------2XMFULLTHDDUMP Full thread dump Classic VM (J2RE 1. .java:77) 4XESTACKTRACE at TE02Servlet.Transports : 3" (TID:0x30285098.http.java:740) 4XESTACKTRACE at javax.doGet(TE02Servlet.HttpServlet.4. state:R 3XHNATIVESTACK Native StackNULL -----------3XHSTACKLINEERR unavailable .servlet.

the current total number of monitors. and so on. JVM System Monitor Dump Java Object Monitor Dump Thread Identifiers WebSphere® Support Technical Exchange 14 . The same as the monitor pool dump except that additional JVM internal information is provided. too many threads waiting on a monitor. You can use this section to identify any problems. it lists all of the system monitors on the JVM for which the Java dump was captured. Similar to the monitor pool dump except that instead of listing any monitor. Provides an association between the XM dump routine and the monitor pool dump. Lists the monitors that exist in the JVM at the time of the Java dump along with the threads waiting for that particular monitor and the owner of each monitor. such as. such as. You can use this section as a pointer to the location of the problem identified in the monitor pool dump. Monitors can also be owned or not owned which will be indicated in the file.IBM Software Group Introduction to javacores – Monitor analysis The following table shows how the LK Dump routine is broken down into five subsections: Monitor Pool Info Monitor Pool Dump Provides basic monitor information.

for example.IBM Software Group Introduction to javacores – Thread analysis Thread States State R CW Name Runnable Conditional Wait Description Thread that has the ability to run or is running. Thread waiting on a monitor lock Thread suspended. MW S Monitor Wait Suspended WebSphere® Support Technical Exchange 15 . Typically threads in this state are waiting for a certain condition to occur. a thread waiting for a resource to become available. Thread waiting on a condition variable.

Transport threads doing? A.ibm. It is suggested to use the following methodology in the order listed: Is there a deadlock? Is there a monitor that has a long list of threads waiting for it? What are the Servlet.html) WebSphere® Support Technical Exchange 16 .IBM Software Group Analyzing Javacores There are many ways to approach analyzing the Javacores to determine the cause of a hang or performance degradation. Are they idle waiting for requests? B. Are some or most of the threads busy and they all appear to be doing something different? Javacores can be analyzed manually or by using the ThreadAnalyzer tool (http://www-128.Engine.com/developerworks/websphere/downloads/thread_analyzer. Are they busy and many of them have the same or similar stacks? C.

Deadlock The Javacore will report a deadlock as follows: 1LKDEADLOCK Deadlock detected !!! NULL --------------------NULL 2LKDEADLOCKTHR Thread "Servlet.Transports : 3" (0x4660CD20) 3LKDEADLOCKWTR is waiting for: 4LKDEADLOCKMON sys_mon_t:0x4532E7C8 infl_mon_t: 0x45170CC0: 4LKDEADLOCKOBJ java.Engine.Object@359F1130/395F1138: 3LKDEADLOCKOWN which is owned by: 2LKDEADLOCKTHR Thread "Servlet.Object@359F1120/395F1128: 3LKDEADLOCKOWN which is owned by: 2LKDEADLOCKTHR Thread "Servlet.Engine.lang.Transports : 3" (0x4660CD20) WebSphere® Support Technical Exchange 17 .IBM Software Group Analyzing javacores .Engine.lang.Transports : 0" (0x450914A0) 3LKDEADLOCKWTR which is waiting for: 4LKDEADLOCKMON sys_mon_t:0x44F79558 infl_mon_t: 0x00000000: 4LKDEADLOCKOBJ java.

Engine.Transports : 5" (0x4760CF50) WebSphere® Support Technical Exchange 18 .IBM Software Group Analyzing Javacores – Threads waiting on Monitor The following output from a Javacore shows several Servlet. entry count 1 3LKNOTIFYQ Waiting to be notified: 3LKWAITNOTIFY "Servlet.Object@359F1140/359F1148 This monitor is owned by thread identifier 0x25 2LKMONINUSE sys_mon_t:0x44F75D48 infl_mon_t: 0x00000000: 3LKMONOBJECT java.Engine.Transport threads waiting on a monitor named java.Transports : 3" (0x4660CD20) 3LKWAITNOTIFY "Servlet.lang.Engine.Engine.Transports : 0" (0x450914A0) 3LKWAITNOTIFY "Servlet.Engine.Object@359F1140/359F1148: Flat locked by thread ident 0x25.Engine.lang.Transports : 1" (0x450EBAA0) 3LKWAITNOTIFY "Servlet.Transports : 4" (0x4670CA14) 3LKWAITNOTIFY "Servlet.

Engine.Engine.Transports : 2” 1LKFLATMONDUMP Thread identifiers (as used in flat monitors): 2LKFLATMON ident 0x26 "Servlet.Engine.Engine.Transports : 2" 2LKFLATMON ident 0x14 "Servlet. thread ident 0x25 is “Servlet.Transports : 0" (0x4660CD20) (0x464900A0) (0x450EBAA0) (0x450914A0) WebSphere® Support Technical Exchange 19 .Transports : 1" 2LKFLATMON ident 0x13 "Servlet.IBM Software Group Analyzing Javacores – Threads waiting on Monitor Examine the “Thread identifiers” section of the LK subcomponent dump to determine the name of the thread associated with “thread ident 0x25.” In this case.Engine.Transports : 3" 2LKFLATMON ident 0x25 "Servlet.

http.servlet.HttpServlet. native ID:0x2438) prio=5 4XESTACKTRACE at java.http. The Java stack trace for thread is as follows: 3XMTHREADINFO "Servlet. sys_thread_t:0x464900A0.java:740) 4XESTACKTRACE at javax.servlet.Transports : 2” is found in the XM component section of the Javacore. all threads waiting for this monitor will be blocked Review subsequent Javacores and monitor the state of this thread WebSphere® Support Technical Exchange 20 .HttpServlet.Thread.java:64) 4XESTACKTRACE at TE02Servlet.sleep(Native Method) 4XESTACKTRACE at TE02Servlet.doGet(TE02Servlet.lang. state:CW.Engine.doLock01(TE02Servlet.doPost(TE02Servlet. Until this thread releases the monitor.Transports : 2" (TID:0x3018A658.IBM Software Group Analyzing Javacores – Threads waiting on Monitor “Servlet.java:853) The TE02Servlet has captured the lock and has gone to sleep.service(HttpServlet.Engine.java:18) 4XESTACKTRACE at javax.java:33) 4XESTACKTRACE at TE02Servlet.service(HttpServlet.

Threads are idle waiting for requests Idle threads could indicate that requests are not making it into the JVM. Is the transport thread present in the JVM? Are there any established connections to the webserver or application server? Review the webserver’s plugin log for potential errors connecting to the application server WebSphere® Support Technical Exchange 21 . you need to determine if any HTTP clients have been able to establish connections to the webserver/plugin or WebSphere transport.IBM Software Group Analyzing Javacores – Servlet Engine Transport Threads A. At this point.

Most of the threads are Runnable (State: R) and the top of their stack is java.socketRead This indicates that there is not a bottleneck regarding synchronized java resources. Oracle. What does the stack of the thread look like? What is the thread waiting on (LDAP.net.IBM Software Group Analyzing Javacores – Servlet Engine Transport Threads B. they are most likely in socketRead waiting for a response from a remote server.SocketInputStream. If the threads have similar stacks and Runnable. Threads are busy and many of them have the same or similar stacks 1. etc)? WebSphere® Support Technical Exchange 22 . HttpURLConnection.

this could cause a hang or performance degradation WebSphere® Support Technical Exchange 23 . none of the threads that are waiting on that monitor have been able to continue processing. – Does the stack of this thread change? If enough threads are attempting to pass through the code that is synchronized and the thread that holds the monitor takes a relatively long time to release it.IBM Software Group Analyzing Javacores – Servlet Engine Transport Threads B. – Does the stack of this thread remain the same? If so. Most of the threads are in monitor wait or conditional wait (State: MW or CW) indicating a potential bottleneck in synchronized code What does the stack look like for the threads that are in MW or CW? What thread holds the monitor that is blocking these threads? This thread should be reviewed in each thread dump. Threads are busy and many of them have the same or similar stacks 2.

and they all appear to be doing something different. some or most of the available threads are in use.IBM Software Group Analyzing Javacores – Servlet Engine Transport Threads C. Is CPU utilization high? High CPU will cause threads to contend for CPU cycles What does memory utilization look like? What does VerboseGC data look like? When GC runs. Threads are not idle. all other threads are suspended. This can indicate a resource issue. WebSphere® Support Technical Exchange 24 .

IBM Software Group Summary Identifying when a JVM is encountering a performance degradation Collecting the documentation required to debug the problem Locating the data collected Introduction to the content of javacores Analyzing javacores for a WebSphere Application Server JVM WebSphere® Support Technical Exchange 25 .

com/software/support/einfo.html Sign up to receive weekly technical My support emails: www.com/software/support/viewlet/probsub/ESR_Overview_viewlet_swf .ibm. webcasts and podcasts at: www.org Access key product show-me demos and tutorials by visiting IBM Education Assistant: ibm.ibm.com/software/info/education/assistant Learn about the Electronic Service Request (ESR) tool for submitting problems electronically: www. participate in technically-focused briefings.html Join the Global WebSphere User Group Community: www.com/software/websphere/events_1. conferences and events: www.ibm.html WebSphere® Support Technical Exchange 26 .websphere.com/developerworks/websphere/community/ Learn about other upcoming webcasts.IBM Software Group Additional WebSphere Product Resources Discover the latest trends in WebSphere Technology and implementation.ibm.

IBM Software Group Questions and Answers WebSphere® Support Technical Exchange 27 .

Sign up to vote on this title
UsefulNot useful