Oracle Weblogic Server

Best Practices for Troubleshooting Performance

Issues - CON8307

Laurent Goldsztejn
Fusion Middleware Proactive Support
September 2014
@weblogicsupport

Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |

Safe Harbor Statement
The following is intended to outline our general product direction. It is intended for
information purposes only, and may not be incorporated into any contract. It is not a
commitment to deliver any material, code, or functionality, and should not be relied upon
in making purchasing decisions. The development, release, and timing of any features or
functionality described for Oracle’s products remains at the sole discretion of Oracle.

Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |

Program Agenda

1 Performance issues
2 Best practices with server logs
3 Best practices with thread dumps
4 Deconstructing Java threads
5 Examples & Demo

Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |

Performance Issues

Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |

Performance Issues
Server hanging

The server does not respond to new requests

Requests time out

Requests take longer and longer to process

The server is no longer reported as running

Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |

Performance Issues
Issues that could lead to server hang or other severe performance issues
Application code or WLS internal issue

Too high number of file descriptors or open sockets

JVM spending too much time in GC

Long running JDBC requests

Lock contention between NM and Managed Server

Excessive or unneeded Remote JNDI lookup

WLS Session Replication problem

Memory leak

Deadlock

Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |

Performance Issues
Data needed to analyze a performance issue
Server logs, -verbose:gc to print information at every collection

Stdout redirected to server log:

Servers->Logging->General->Advanced-> Redirect Stdout Logging Enabled
-Dweblogic.log.RedirectStdoutToServerLogEnabled=true

Java Flight Recordings (JFR)

Collections of general data about the application (GCs, Heap, Optimizations, Objects stats etc…)

RDA collections or configuration files

Thread dumps / Heap dumps

WLDF to collect metrics, setup watch and notifications, and to define instrumentation

Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |

Best practices with server logs

Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |

Best practices with server logs
Errors in Server Log file

Check for
Check for Stuck Check for Too Many
OutOfMemory
Threads Open Files
messages

Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |

Best practices with server logs
Example of Stuck Threads

<Error> <WebLogicServer> <BEA-000337> <[STUCK] ExecuteThread: '6' for queue:

'weblogic.kernel.Default(self-tuning)' has been busy for "196" seconds working on the request
"Http Request: /abc/services/MyServicesImpl", which is more than the configured time
(StuckThreadMaxTime) of "180" seconds.

oracle.jdbc.driver.OracleStatement.getColumnIndex(OracleStatement.java:3288)
oracle.jdbc.driver.OracleResultSetImpl.findColumn(OracleResultSetImpl.java:1914)
oracle.jdbc.driver.OracleResultSet.getTimestamp(OracleResultSet.java:1661)
weblogic.jdbc.wrapper.ResultSet_oracle_jdbc_driver_OracleResultSetImpl.getTimestamp(Unknown Source)
com.foo.sqlmap.engine.type.DateTypeHandler.getResult(DateTypeHandler.java:44)
…
The thread is blocked and can't return to the thread pool in a given period of time

Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |

Best practices with server logs
Too Many Open Files
<Critical> <Server> <BEA-002616> <Failed to listen on channel "Default" on 172.1.2.3:7001, failure
count: 1, failing for 0 seconds, java.net.SocketException: Too many open files>

• Socket connections use file descriptors

• A socket can remain in TIME_WAIT after they are closed to guarantee that delayed packets in the network reached
the corresponding socket
tcp_time_wait_interval on Linux (default 4 minutes)
HKLM\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters on Windows (default 1 minute)

• If many connections are being opened and closed quickly then socket's in TIME_WAIT may begin to accumulate on
a system
– View sockets in TIME_WAIT using netstat

Support Pattern: How To Troubleshoot Too Many Open Files Problems [ID 867492.1]

Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |

Best practices with server logs
OutOfMemory

• JVM throws OOM if it is not able allocate memory to java objects

• The application may handle this error and decide to shut itself down in a safe way or decide to run ignoring this error

• -verbose:gc -XX:+PrintGCDetails, -XX:+PrintGCDateStamps and -

XX:+PrintGCTimeStamps to logs details on GC cycles

• Memory Leak Detectors and heap dumps can be used to identify causes for leaks in applications

Support Pattern: Troubleshooting Out of Memory and Memory Leak Problems [ID 877172.1]

Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |

Best practices with thread dumps

Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |

Best practices with thread dumps
Definitions
• A java thread dump is a snapshot that shows what every thread in the JVM
process is doing at a particular point in time

Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |

Best practices with thread dumps
What does a Thread Dump contain?

• Date and • Vendor

time the • Version
dump was • Build
created
• Architecture
• Mode
Time Stamp JVM Info

Stack Threads

• Classes • State
• Packages • Name,
• Method Calls Queue
• Locks • tid, nid, prio

Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |

Best practices with thread dumps
Different ways to take thread dumps

jcmd <pid>
Remote Diagnostic Unix Command-line print_threads
WLST threadDump()
Agent (RDA) kill -3 <pid>
Hotspot JDK 7

jcmd
<pid> Thread.print
WLS Admin Console Jstack <pid> Java VisualVM
(Java 7 mission
control)

Different ways to take thread dumps in WebLogic Server [ID 1098691.1]

How to Use RDA to Generate WLS Thread Dumps At Specified Intervals?
https://blogs.oracle.com/fmwinstallproactive/entry/how_to_use_rda_to

Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |

Best practices with thread dumps
Viewing a Thread Dump

• Different JVM Vendors display the data in different formats

• Markers for start/end of thread dumps
• Reporting of locks
• Thread states and method signatures

• The underlying data exposed remains the same across vendors

• Thread dumps can contain lots of data

• Trying to read them in a text editor can be very tricky

Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |

Best practices with thread dumps
Free parsing tools

Tool Description
Samurai Light weight open source tool that extract
http://yusuke.homeip.net/samurai/en/index.ht thread dumps and GC stats from log files,
ml and presents them in different table views
with color code for easy visual analysis

Thread Dump Analyzer (TDA) Swing GUI for analyzing Thread Dumps and
http://java.net/projects/tda/pages/Home Heap Information
Thread Logic Built upon TDA by adding logic for common
http://java.net/projects/threadlogic/ patterns found in application servers

Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |

Troubleshooting
Thread Dump Analysis – Five High level steps

Look at default and custom execute threads

Identity the cause of stuck threads by looking at the last method and java class invoked

Ignore the idle threads

Check thread dumps from different managed servers to find potential relationships between
the stuck threads

In most cases, WLS internal threads, such as muxer and listen threads, can be ignored

Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |

Deconstructing java threads

Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |

Deconstructing java threads
Thread hanging on SocketRead from JDBC call
<[STUCK] ExecuteThread: '4' for queue: 'weblogic.kernel.Default (self-tuning)' id=76 idx=0x128 tid=15140 prio=1 alive, in native, daemon
java.net.SocketInputStream.socketRead0(Native Method)
java.net.SocketInputStream.read(SocketInputStream.java:129) id (or tid) is the thread identifier, a unique process-
oracle.net.ns.Packet.receive(Unknown Source) wide number that identifies this thread within the
JVM process.
oracle.net.ns.DataPacket.receive(Unknown Source) nid is the OS-level native thread identifier. It can
oracle.net.ns.NetInputStream.getNextPacket(Unknown Source) be used to correlate with high CPU usage threads
identified at the OS level
… idx is the thread index in the threads array.
oracle.jdbc.driver.PhysicalConnection.<init>(PhysicalConnection.java:420) prio refers to the thread priority, a number
inherited from the thread that created it.
oracle.jdbc.driver.T4CConnection.<init>(T4CConnection.java:165) alive refers to the fact that this thread has not
oracle.jdbc.driver.T4CDriverExtension.getConnection(T4CDriverExtension.java:35) ended yet and is still active.
in native means that the thread uses the operating
oracle.jdbc.driver.OracleDriver.connect(OracleDriver.java:801) system's native ability to manage multi-threaded
java.sql.DriverManager.getConnection(DriverManager.java:140) processes.
daemon indicates that this thread can't prevent
com.foo.sbm.util.ConnectionPool.createConnection(Unknown Source) the JVM from exiting.

Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |

Deconstructing java threads
Thread.State
State Description

NEW Available thread ready to process work request for any work manager

Thread blocked by another and waiting for the other thread to complete its operations and
release its locks. If more than one threads attempt to acquire the monitor of a particular
BLOCKED
object then only one thread (selected by the JVM scheduler) is granted the monitor and all
other threads are put into BLOCKED state

RUNNING Thread actively working processing a task and therefore using some CPU

Thread in sleep, wait, join or park method and in a state of monitoring (waiting on monitor
TIMED_WAITING condition) - waiting for another thread to perform an action for up to a specified waiting
time is in this state
The thread is waiting by using a wait, join or park method without any maximum - waiting
WAITING
indefinitely for another thread to perform a particular action is in this state

Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |

Deconstructing java threads
Java monitors
• Mechanism used by Java to support thread synchronization
– Mutual exclusion
• Supported via object locks
• Enables multiple threads to independently work on shared data without interfering with each other
– Cooperation
• Supported via the wait and notify methods of class Object
• Enables threads to work together towards a common goal

• Package java.util.concurrent
– Utility classes commonly useful in concurrent programming

Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |

Deconstructing java threads
Java Monitors - Example
"Thread-0" … in Object.wait()[…] "Thread-0" is waiting for a
at java.lang.Object.wait(Native Method) notification after it called
- waiting on <id2> (a a.b.c.thread.MyClass) Object.wait()
at java.lang.Object.wait(Unknown Source)
at a.b.c.thread.Drop.take(ClassB.java:44)
- locked <id1> (a a.b.c.thread.ClassB)
at a.b.c.thread.exec(Test.java:15)
at java.lang.Thread.exec(Unknown Source)

"Thread-1" … waiting on condition[…]

at java.lang.Thread.sleep(Native Method)
"Thread-1" is sleeping on a
at x.y.z.thread.run(ClassA.java:33) condition after it called
at java.lang.Thread.run(Unknown Source) Thread.sleep

Conditions provide a means for one thread to suspend execution and wait until
notified by another thread that some state condition may now be true

Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |

Deconstructing java threads
Java locks

• Locks are used to prevent that more than one thread accesses and works the same
object

• Only one thread can own a Lock object at a time

• Lock objects support a wait/notify mechanism

• A lock is called Thin with little or no contention

• A lock is promoted to Fat when contention increases

Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |

Deconstructing java threads
Thread Requesting Locks
• If the lock is not held by another thread then the thread becomes the new holder of the
lock until it relinquishes it

• If the lock is currently held then the thread requesting it joins the blocked list of
contenders for this lock

• Threads holding locks should relinquish them as soon as they stop needing them to
avoid impairing on other threads

Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |

Deconstructing java threads
Deadlocks
• Deadlocks describe a situation where two or more threads are blocked waiting for each
other (circular dependency). Neither block will ever end because each thread is waiting
for the other

• Deadlocks can occur when multiple threads need the same locks, at the same time, but
obtain them in different order

• Restart of the server is the only option

Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |

Examples & Demo

Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |

Examples and demo
Thread hanging on SocketRead from JDBC call
<[STUCK] ExecuteThread: '4' for queue: 'weblogic.kernel.Default (self-tuning)' id=76 idx=0x128 tid=15140 prio=1 alive, in native, daemon
java.net.SocketInputStream.socketRead0(Native Method)
java.net.SocketInputStream.read(SocketInputStream.java:129)
oracle.net.ns.Packet.receive(Unknown Source)
oracle.net.ns.DataPacket.receive(Unknown Source)
The thread is stuck while waiting to
oracle.net.ns.NetInputStream.getNextPacket(Unknown Source) receive data back from an Oracle
Database
…

oracle.jdbc.driver.PhysicalConnection.<init>(PhysicalConnection.java:420) This can be due to blocked

database sessions or a
oracle.jdbc.driver.T4CConnection.<init>(T4CConnection.java:165) communication issue with the
oracle.jdbc.driver.T4CDriverExtension.getConnection(T4CDriverExtension.java:35) backend DB server leaving the
socket idle
oracle.jdbc.driver.OracleDriver.connect(OracleDriver.java:801)
java.sql.DriverManager.getConnection(DriverManager.java:140)
com.foo.sbm.util.ConnectionPool.createConnection(Unknown Source)

Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |

Examples and demo
Thread hanging on SocketRead from SMTP call
[STUCK] ExecuteThread: '13' for queue: 'weblogic.kernel.Default (self-tuning)'" id=76 idx=0x128 tid=15140 prio=1 alive, in native, daemon
at jrockit/net/SocketNativeIO.readBytesPinned(Ljava/io/FileDescriptor;[BIII)I(Native Method)
at jrockit/net/SocketNativeIO.socketRead(SocketNativeIO.java:46)[optimized] The thread is stuck waiting to
at java/net/SocketInputStream.socketRead0(Ljava/io/FileDescriptor;[BIII)I(SocketInputStream.java)[inlined] receive data from the remote
mail server. Wily introscope is
at java/net/SocketInputStream.read(SocketInputStream.java:129)[optimized] used as instrumentation. The
communication with mail server
…
needs to be checked
at java/io/BufferedInputStream.read(BufferedInputStream.java:235)[optimized]
^-- Holding lock: java/io/BufferedInputStream@0x291bf2b8[thin lock]
A lock is placed on a
at com/sun/mail/util/LineInputStream.readLine(LineInputStream.java:75) BufferInputStream object to
at com/sun/mail/smtp/SMTPTransport.readServerResponse(SMTPTransport.java:1440) safeguard its content

…
at javax/mail/Service.connect(Service.java:105)
at javax/mail/Transport.send0(Transport.java:168)
…

Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |

Examples and demo
Thread hanging on LDAP response
"[STUCK] ExecuteThread: '2' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=10 tid=0x000000004c8a2000 nid=0x7559 in Object.wait()
[0x000000004296c000]
java.lang.Thread.State: WAITING (on object monitor) The thread is waiting to enter a
synchronized block and acquire a
at java.lang.Object.wait(Native Method) Java monitor.
- waiting on <0x00002aaaeced3718> (a netscape.ldap.LDAPSearchListener)
Its wait will be over when it
at java.lang.Object.wait(Object.java:485) receives the message back from
at netscape.ldap.LDAPMessageQueue.waitForMessage(LDAPMessageQueue.java:200) the LDAP server.

- locked <0x00002aaaeced3718> (a netscape.ldap.LDAPSearchListener) The health of the LDAP server

at netscape.ldap.LDAPMessageQueue.waitFirstMessage(LDAPMessageQueue.java:101) needs to be checked to
understand why it’s performing
… slowly
at weblogic.security.providers.authentication.LDAPAtnDelegate.authenticate(LDAPAtnDelegate.java:3625)
at weblogic.security.providers.authentication.LDAPAtnLoginModuleImpl.login(LDAPAtnLoginModuleImpl.java:220)
at com.bea.common.security.internal.service.LoginModuleWrapper$1.run(LoginModuleWrapper.java:110)
at java.security.AccessController.doPrivileged(Native Method)

Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |

Examples and demo
Idle Thread waiting for new task
"[ACTIVE] ExecuteThread: '96' for queue: 'weblogic.kernel.Default (self-tuning)'" id=313028 idx=0x1d0 tid=4686 prio=5 alive, in native,
waiting, daemon
-- Waiting for notification on: weblogic/work/ExecuteThread@0x739b42e0[fat lock]
at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
at jrockit/vm/Locks.wait(Locks.java:2229)[inlined]
at java/lang/Object.wait(Object.java:474)[inlined] The thread is waiting to be
notified of a new task assignment
at weblogic/work/ExecuteThread.waitForRequest(ExecuteThread.java:156)[inlined] Several available or free threads
are waiting on the same
at weblogic/work/ExecuteThread.run(ExecuteThread.java:177)[optimized] ExecuteThread object which
^-- Lock released while waiting: weblogic/work/ExecuteThread@0x739b42e0[fat lock] results in a fat lock

at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
-- end of trace

Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |

Examples and demo
Deadlock
"[ACTIVE] ExecuteThread: '44' for queue: 'weblogic.kernel.Default (self-tuning)'" id=372 idx=0xfa tid=4257
prio=5 alive, in native, blocked, daemon
Thread #134 is holding a lock
-- Blocked trying to get lock: weblogic/utils/KeyTable@0x16939098[thin lock] that thread #44 needs and at
the same time is requesting a
at jrockit/vm/Threads.sleep(I)V(Native Method)
lock that thread #44 has locked
...
- Holding lock: weblogic/rjvm/ResponseImpl@0x4385fc78[fat lock]
at weblogic/rjvm/ResponseImpl.getTxContext()Ljava/lang/Object;(ResponseImpl.java:100)[inlined]
...
"[ACTIVE] ExecuteThread: '134' for queue: 'weblogic.kernel.Default (self-tuning)'" id=473785 idx=0x398 tid=32638 prio=5 alive, in native, blocked,
daemon
-- Blocked trying to get lock: weblogic/rjvm/ResponseImpl@0x4385fc78[fat lock]
Both threads are currently
at jrockit/vm/Threads.waitForSignal(J)Z(Native Method) blocked waiting to get a lock
...
^-- Holding lock: weblogic/utils/KeyTable@0x16939098[thin lock]
...
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
 Drinks. Food. Fun.
My Oracle Support Monday Mix
Tonight!
Monday, September 29
6:00 to 8:00 p.m.
ThirstyBear Brewing Company
(only ½ block from Moscone Center)

Join us for a relaxing Happy Hour after a busy day at Oracle OpenWorld!
•Take a break and unwind with your peers
•Get to know the Oracle support engineers you depend on
•Meet My Oracle Support executives and developers
•Enjoy drinks and hors d’oevres
•Admission is free with your Oracle OpenWorld badge
Event details at:
www.oracle.com/goto/mondaymix
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | Oracle Confidential – Internal/Restricted/Highly Restricted 35
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |