Join The War On: Classloader Leaks

Join the war on
Classloader leaks
Mattias Jiderhamn
Mattias Jiderhamn – java.jiderhamn.se

:-(
java.lang.OutOfMemoryError:
Metaspace
PermGen space

Local dev env

Continuous Deploy
.war

Agenda
What does it mean?
Why does it happen?
Where does it leak?
How to avoid?
OSS examples
Training!
What?
JVM memory
PermGen /
Stack
Metaspace
Heap

JVM memory
Per thread
Stack Local variables and
method parameters

JVM memory
Object instances
Heap
Young generation
Eden Survivor Old generation /
space space tenured space

JVM memory
Permanent Generation
java.lang.Class instances PermGen /
Metaspace
etc
Named before JEE and
class unloading
Renamed Metaspace in Java 8
Aka Method area
What?
PermGen space / Metaspace
= Too many classes are loaded

Why?
Reason for OOME
PermGen space / Metaspace
1. Your application is too large
-XX:MaxPermSize=256M
Java 8: Metaspace auto increase by default
2. java.lang.Class instances could not
be garbage collected after redeploy

Reference types
• Strong (i.e. normal) reference
– Never GC:ed if reachable
• Soft reference
– GC:ed before OutOfMemoryError
• Weak reference (WeakHashMap)
– GC:ed when no Strong or Soft refs
• Phantom reference
– … won’t prevent GC
Example
WeakHashMap
Map m = new WeakHashMap();

Foo myFoo = new Foo();
m.put(myFoo, ”bar”);
myFoo = null;
Foo

GC reachability
GC roots
Redeploy
Application Server
ClassLoader ClassLoader ClassLoader
app.war app.war’ app.war’’

ClassLoader refs
ClassLoader
Class Class Class
Instance

How leaks happen
Application Server
ClassLoader
Class Class Class
Instance
GC root

Where?
Application Server
• Application Server bugs
?
• Logging frameworks
– Apache Commons Logging
Unless LogFactory.release()
– Log4j - some configs
Unless LogManager.shutdown()
– java.util.logging custom Level
• Bean Validation API (JSR 303)
• Unified Expression Language (javax.el)

GC roots
• Class loaded by system ClassLoader
– static field in JDK classes (java.* etc)
• Live thread
– Stack – local vars, method params
– java.lang.Thread instance
• Object held as synchronization monitor
• JNI references
• JVM specials…
System classes
• java.sql.DriverManager
• Bean introspection cache, shutdown hooks,
custom default Authenticator, custom
security Provider, custom MBeans, custom
ThreadGroup, custom property editor, …
• Reference to contextClassLoader of first caller

DriverManager

DriverManager
JRE
Application Server
DriverManager
1) …
ClassLoader
com.mysql.jdbc.Driver
app.war
mysql-jdbc.jar
… or 2)
DriverManager.deregisterDriver(…)
Context shutdown
public class MyCleanupListener implements
javax.servlet.ServletContextListener {
...
/** Called when application is undeployed */

public void contextDestroyed(
ServletContextEvent servletContextEvent) {
DriverManager.deregisterDriver(…);
}
}

Threads
• Thread stack
– Local variables
– Method parameters
• MyThread extends Thread
MyRunnable implements Runnable
• contextClassLoader +
AccessControlContext inherited
– Example HTTP 1.1 Keep-Alive-Timer

Context shutdown
public class MyCleanupListener implements
javax.servlet.ServletContextListener {
...
/** Called when application is undeployed */

public void contextDestroyed(
ServletContextEvent servletContextEvent) {
DriverManager.deregisterDriver(…);
// Stop threads here!
}
}

Stopping threads
public class MyThread extends Thread
{
public void run() {
while(true) { // Bad idea!
// Do something
}
}
}
Stopping threads
public class MyThread extends Thread
{
private volatile boolean running = true;
private boolean running = true;
public void run() {
while(running) { // Until stopped
// Do something
}
}
public void shutdown() {
running = false;
} Kabutz / Java Specialists’ - The Law of the Blind Spot
Heinz
} Mattias Jiderhamn – java.jiderhamn.se
Threads
• Thread stack
– Local variables
– Method parameters
• MyThread extends Thread
MyRunnable implements Runnable
• contextClassLoader +
AccessControlContext inherited
– Example HTTP 1.1 Keep-Alive-Timer
• ThreadLocal
ThreadLocal
ThreadLocal
WeakHashMap<Thread, ?>
Thread Foo

ThreadLocal
ThreadLocal JavaDoc:
”Each thread holds an implicit
reference to its copy of a thread-
local variable as long as the thread
is alive and the ThreadLocal
instance is accessible; …”

ThreadLocal
Thread
put() ThreadLocalMap
Entry
ThreadLocal Foo
set()
ThreadLocal (?)
Pooled threads:
• Threads may outlive ClassLoader
• ThreadLocal → ThreadGlobal!

ThreadLocal
Thread
ThreadLocalMap
Entry
ThreadLocal Foo
static
Class ClassLoader Class

ThreadLocal
Thread
ThreadLocalMap
Entry
Stale entry
ThreadLocal Foo
ClassLoader Class

ThreadLocal
ThreadLocalMap JavaDoc:
”However, since reference queues
are not used, stale entries are
guaranteed to be removed only
when the table starts running out
of space”

ThreadLocal
• Custom value (incl references)
–static ThreadLocal = leak
–otherwise = unpredicted GC
• Custom ThreadLocal = no leak

ThreadLocal
Always clear your ThreadLocals!
try {
myThreadLocal.set(foo);
…
}
finally {
myThreadLocal.remove();
}
Known offenders
Apache ActiveMQ JarURLConnection
Apache Axis Java Advanced Imaging (JAI)
Apache Batik Javassist
Apache Commons Pool / DBCP Java Cryptography Architecture (JCA)
Apache CXF Java Server Faces 2
AWT Toolkit javax.security.auth.Policy/login.Configuration
Bean Validation API / JSR 303 JGroups
CGLIB (Hibernate / Spring / JBoss / Logback
Apache Geronimo) JAXB
dom4j Mozilla Rhino
EclipseLink MVEL
GeoTools OpenOffice Java Uno RunTime (JURT)
Google Guice Oracle JDBC
Groovy RMI
GWT / javax.imageio Serialization (<= JDK 1.4.2)
Hessian Spring framework
iCal4J Unified Expression Language / javax.el
Infinispan URLConnection + HTTP 1.1
IntrospectionUtils
XML parsing (DocumentBuilderFactory)
How?
Leak Prevention Lib
• Application server independent
– Version 2 core module is pure Java SE!
• Covers a lot more than Tomcat
– System class references avoided/cleared
– Threads are stopped
– ThreadLocals are cleared
– Known offenders handled
• Logs warnings
• Apache 2 licensed
– You may modify and redistribute
Leak Prevention Lib
Zero runtime dependencies
No required config
Maven pom.xml (Servlet 3.0+):
<dependency>
<groupId>se.jiderhamn.classloader-leak-prevention</groupId>
<artifactId>classloader-leak-prevention-servlet3</artifactId>
<version>2.0.1</version>
</dependency>

Tomcat
Cleanup
Thread Thread
remove() get()
ThreadLocalMap set()
Unsafe! remove()
Removed in 6.0.27
Bugzilla #48895
Tomcat 7.0.6+ renews the thread pool

whenever an application is redeployed.
Disposable threads for lifecycle events.
Bugzilla #49159

Leak Prevention Lib
Cleanup Thread
Thread
set(null) ThreadLocalMap
Entry
Stale
ThreadLocal Foo

Open Source
examples
Bean Validation API
Version 1.0.0.GA
javax.validation.Validation
Application Server
validation-api-1.0.0.GA.jar
app.war
hibernate-validator.jar

javax.el API
javax.el.BeanELResolver

Apache CXF
org.apache.cxf.transport.http.CXFAuthenticator
CXF-5442

OpenOffice JURT
com.sun.star.lib.util.AsynchronousFinalizer

Training!
Gear up!

Free tool
Eclipse Memory Analyzer
aka MAT
eclipse.org/mat

Heap dump
Run application server with
-XX:+HeapDumpOnOutOfMemoryError
-XX:HeapDumpPath=/temp

Links
java.jiderhamn.se @mjiderhamn
github.com/mjiderhamn
• ClassLoader Leak Prevention Library
• Step by step guide: acquire heap
dump and analyze for leaks using
Eclipse Memory Analyzer (MAT)
• List of known offenders and links to bug reports
• Submit reports and pull requests!
???
Join the fight!

Join The War On: Classloader Leaks

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Join The War On: Classloader Leaks

Uploaded by

Copyright:

Available Formats

Join the war on

Mattias Jiderhamn – java.jiderhamn.se

Mattias Jiderhamn – java.jiderhamn.se

Mattias Jiderhamn – java.jiderhamn.se

Mattias Jiderhamn – java.jiderhamn.se

Mattias Jiderhamn – java.jiderhamn.se

Mattias Jiderhamn – java.jiderhamn.se

Mattias Jiderhamn – java.jiderhamn.se

= Too many classes are loaded

Mattias Jiderhamn – java.jiderhamn.se

Mattias Jiderhamn – java.jiderhamn.se

Map m = new WeakHashMap();

Mattias Jiderhamn – java.jiderhamn.se

ClassLoader ClassLoader ClassLoader

app.war app.war’ app.war’’

Mattias Jiderhamn – java.jiderhamn.se

Class Class Class

Mattias Jiderhamn – java.jiderhamn.se

Class Class Class

Mattias Jiderhamn – java.jiderhamn.se

Mattias Jiderhamn – java.jiderhamn.se

Mattias Jiderhamn – java.jiderhamn.se

Mattias Jiderhamn – java.jiderhamn.se

/** Called when application is undeployed */

Mattias Jiderhamn – java.jiderhamn.se

Mattias Jiderhamn – java.jiderhamn.se

/** Called when application is undeployed */

Mattias Jiderhamn – java.jiderhamn.se

Mattias Jiderhamn – java.jiderhamn.se

Mattias Jiderhamn – java.jiderhamn.se

Mattias Jiderhamn – java.jiderhamn.se

Class ClassLoader Class

Mattias Jiderhamn – java.jiderhamn.se

Mattias Jiderhamn – java.jiderhamn.se

Mattias Jiderhamn – java.jiderhamn.se

Mattias Jiderhamn – java.jiderhamn.se

Mattias Jiderhamn – java.jiderhamn.se

Tomcat 7.0.6+ renews the thread pool

Mattias Jiderhamn – java.jiderhamn.se

Mattias Jiderhamn – java.jiderhamn.se

Mattias Jiderhamn – java.jiderhamn.se

Mattias Jiderhamn – java.jiderhamn.se

Mattias Jiderhamn – java.jiderhamn.se

Mattias Jiderhamn – java.jiderhamn.se

Mattias Jiderhamn – java.jiderhamn.se

Mattias Jiderhamn – java.jiderhamn.se

Mattias Jiderhamn – java.jiderhamn.se

You might also like