Find Those Java Memory Leaks Fast!

Oracle JRockit Mission Control 4.0 Mark Prichard

Java Platform Group, Oracle Fusion Middleware mark.prichard@oracle.com

The following is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, and timing of any features or functionality described for Oracles products remains at the sole discretion of Oracle.

JRockit Product Family

JROCKIT MISSION CONTROL
Complete insight into application

JROCKIT REAL TIME

High-performance real-time

JROCKIT VIRTUAL EDITION

Fly-weight Java container for

& JVM behavior

Zero performance overhead in

solution for standard Java

Industry leading Deterministic

virtualized environments
Improve datacenter efficiency - do

production environments
No application modification or

Garbage Collector
Millisecond response times with

more with less

Simpler and more powerful VM

configuration required

five nines guarantee

Improve application

management
Released Oracle WebLogic Suite

performance & latency with unique tooling

Virtualization Option

JROCKIT JVM
World-class performance Powerful diagnostics Full support from Oracle

JRockit Mission Control

Monitor health & performance in production A suite of powerful tools for
Performance Tuning Diagnostics Support

JRMC 4.0 Demos

Lets look at some JRMC + Flight Recorder demo scenarios

Demo - Memleak
Leaking char[] We suspect there may be a problem with Leak $DemoObject

Demo - Memleak
(right-click) Add to Type Graph Expand backwards until the leak stops Hashtable$Entry looks problematic

Demo - Memleak
(right-click on relation from Hashtable$Entry -> DemoObject) List Referring Instances to see instances pointing to DemoObject Hashtable$Entry instances listed in Instances View

Demo - Memleak
(right-click) Expand to Root Shows DemoThread with field table referring to Hashtable contining the leaking DemoObject

Demo Latency Analyzer

Threads spending most of the time blocking each other

Demo Latency Analyzer

Much better (removed synchronized logger calls)

Demo GC Detailed Analysis

GC profile is bad frequent (long) full collections Finalizers are the problem avoid them where possible

Demo GC Detailed Analysis

Much better simply by removing finalize() altogether Finalizers can often interfere with GC

Demo Code Analysis

Flight Recorder -> Code -> Hot Methods LinkedList is inefficient here, better to use HashMap instead

Demo Code Analysis

Flight Recorder -> Code -> Exceptions Exceptions are expensive to process: abused by programmer here

JRMC 4.0 Whats New?

JRockit Flight Recorder Support Historical Data Option & View WebLogic Triggers Memory Leak Advances
Same Look & Feel (SWT/JFace vs. Swing) Classloader support

Regular Expressions (query) Bug Fixes Eclipse Integration updated to Ganymede/Galileo

2010 Oracle Corporation Proprietary and Confidential 16

What is the JRockit Flight Recorder?

New in JRockit R28 Circular buffer in JRockit JVM that stores diagnostic data
Always on New data comes in and is stored, old data dropped off

New Data

Built-in integration with JRMC

Replaces JRMC Runtime Analyzer and Latency Analyzer

Time

Very low/near zero overhead

Uses data already used by JVM

Data can include events from the JVM and from any other event producer
WebLogic Server (WLDF) Fusion Middleware (DMS)

Flight Recording

Old Data

JRockit Flight Recorder Use Cases

What it is designed for?
Provide diagnostic information in running production systems Capture most recent activity to enable analysis leading up to an issue Capture data from all levels JVM, WLS, DMS, etc Offline/offsite analysis can be done using the JRMC GUI JRockit dumps capture information to assist in crash-analysis

What it is not designed for?

Large event payloads or very high volumes of events Long history Not a replacement for Debug logging or the server logging

JRockit Flight Recorder Summary

Short summary: JRA + LAT combined Collects diagnostic data, called events Continuous recording of events to RAM or Disk Low overhead, always on recording of basic events to RAM possible There is a special Default profile. It has id=0, and is designed to have an overhead around 1%. It writes to memory unless otherwise specified. It is off by default.

Flight Recorder will help you

Solve JVM tuning issues Help you figure out why JRockit does something strange Help you look under the hood of JRockit Help you look under the hood of software running on top of JRockit, for example WLS

Flight Recorder Diagnostic Commands You interact with Flight Recorder through Mission Control Or with the jrcmd diagnostic commands:
start_flightrecording stop_flightrecording check_flightrecording dump_flightrecording

start_flightrecording
jrcmd <pid> start_flightrecording
name settings defaultrecording delay duration filename compress maxage maxsize - Name of recording (string) - Settings file or preset (string) - Starts default recording (bool, false) - Delay recording start (time, 0s) - Duration of recording (time, 0s) - Resulting recording filename (string) - GZip-compress the resulting recording file (bool, false) - Maximum age of buffer data (time, 0s) - Maximum size of buffers in bytes (long, 0)

To start the default recording:

$> jrcmd 4711 start_flightrecording defaultrecording

Or start JRockit with the command:

$> java -XX:FlightRecorderOptions=defaultrecording=true

start_flightrecording (cont.)
For example, to start a recording connecting to a running JRockit JVM with pid 4711, with the settings named jrockit-se.jfs, and with the recording named Troubleshoot outputting to the file recording.jfr use: $> jrcmd 4711 start_flightrecording name=Troubleshoot settings=jrockit-se.jfs filename=recording.jfr

dump_flightrecording
Stores a snapshot of a running recording Jrcmd <pid> dump_flightrecording
name recording copy_to_file compress_copy - Recording name (string) - Recording id (long, -1) - Copy recording data to file (string) - GZip-compress "copy_to_file" destination (bool, false)

Example:
jrcmd 16749 dump_flightrecording recording=0 copy_to_file=cont.jfr

Will store a copy of recording with id=0 to a file named cont.jfr. Note: The recording ends up in the working directory of the JVM you attach to! (This is probably not the directory from where you run 'jrcmd'.)

stop_flightrecording
jrcmd <pid> stop_flightrecording
name recording discard copy_to_file compress_copy - Recording name (string) - Recording id (long, -1) - Discards the recording data (bool, false) - Copy recording data to file (string) - GZip-compress "copy_to_file" destination (bool, false)

Will stop the recording. Can save the stopped recording to a file.

WebLogic Events in JFR

Simple configuration: enabled and configured using a volume control per server:
Off (default), Low, Medium, High

Each level has a predetermined data set

Determined through consultation with dev groups and Support Data set was adjusted based on performance testing High volume includes user info and stack traces, which are expensive data gathering operations

WLDF/JFR Integration Architecture

WLST, Console, WLDF Watch/Notification

EJB WebApp JDBC JTA WS Etc WLDF Instrumentation Configuration (Instrumented at Build Time) WLDF Instrumentation Engine

WLST Image Manager

WLDF Monitors and Action Groups WLDF Flight Recorder Image Source
Instrumentation Extension WLDF Flight Recorder Actions
JRFR Producer API

JRFR Consumer API

Extension Actions

JRockit Flight Recorder Diagnostic Image File With Flight Recorder Data Diagnostic Image File Without Flight Recorder Data Extension supplied JRockit Supplied WLDF File/Artifact Available if JRockit is Used Extension Data

WLDF/JFR Integration Use Cases

What it is designed for?
Provide diagnostic information in running production systems Capture most recent activity to enable analysis leading up to an issue Capture data from all levels JVM, WLS, DMS, etc Offline/offsite analysis can be done using the JRMC GUI JRockit dumps capture information to assist in crash-analysis

What it is not designed for?

Large event payloads or very high volumes of events Long history Not a replacement for Debug logging or the server logging

WLS 10.3.3, JFR R28 and JRMC 4.0.1

Quick Start Guide

Turn on JFR default recording

-XX:FlightRecorderOptions=defaultrecording=true Flight recorder command line documentation

Configure WLS to use Platform MBean Server

Admin console: domain->configuration->general tab (Advanced) Check Platform MBean Server Enabled (default in 10.3.3) Check Platform MBean Server Used (off by default) NB: Platform MBean Server security -

Working with JRockit Flight Recorder Start JRMC, connect to WLS Configure WLS Diagnostic Volume (admin console: server->configuration->general, High) Right mouse, select Dump Default Recording

JRMC + Flight Recorder

JRMC + Flight Recorder

JRMC + Flight Recorder

JRMC + Flight Recorder

Recommended Reading
Written by two senior members of JRockit Engineering team Highly Recommended
Chapter 1 - Getting started Chapter 2 - Adaptive Code Generation Chapter 3 - Adaptive Memory Management Chapter 4 - Threads and synchronization Chapter 5 - Benchmarking and tuning Chapter 6 - JRockit Mission Control Chapter 7 - The Management Console Chapter 8 - The Runtime Analyzer Chapter 9 - The Flight Recorder Chapter 10 - The Memory Leak Detector Chapter 11 - JRCMD Chapter 12 - Using the JRockit Management APIs Chapter 13 - JRockit Virtual Edition Appendix A - Bibliography Appendix B - Glossary

36