Java tuning on z (CICS, IMS, DB2

Scott Clee - CICS Test Architect

Java on the mainframe has reached a level of maturity. Big mainframe players support it and its popularity has radically increased due skills availability and competitive pricing models on the z platform. Having answered the question "Can we run it?". Many people are now asking "How can we run it well?". This informative presentation brings together Java tuning information from a collection of CICS, IMS and DB2 experts. Audience: Sys Prog Skill level: Intermediate, Advanced




Important Disclaimer




Java considerations JVM tuning JVM on z CICS tuning DB2 tuning IMS tuning Tooling Links




Java considerations




Similarities and differences between JVMs and platforms The IBM J9 JVM is the same across all platforms – Tools and techniques available on distributed platforms are applicable to z/OS – In the past CICS would only support specific JVM parameters but this is not the case now Different vendor JVMs may have varying parameters and default settings – IBM J9 JVM runs in server mode – Sun HotSpot JVM defaults to client mode Certain aspects of JVM tuning are common across all platforms – Application tuning – Heap management – Garbage collection 03/06/2009 GUIDE SHARE EUROPE 6 .

JVM tuning 03/06/2009 GUIDE SHARE EUROPE 7 .

Main areas for performance consideration Server performance can be impacted by: – Application’s behaviour with objects – Use of heap – Frequency and type of garbage collection Performance can be increased with proper configuration of JVM parameters – This provides one of the biggest opportunities for performance gains Application Garbage collection 03/06/2009 Memory usage 8 GUIDE SHARE EUROPE .

Application tuning Java frees the programmer from the burden of memory management – Ensures applications are more robust – Brings in its own set of considerations and practices Watch out for object abuse! Verify these main areas: – Application is not over utilizing objects – Application is not leaking objects – Java heap parameters are set properly to handle a given object usage pattern 03/06/2009 GUIDE SHARE EUROPE 9 .

but it occurs less frequently.Heap size The Java heap is where the objects of a Java program live. dead objects. the application runs longer before a garbage collection occurs. a larger heap also takes longer to compact and causes garbage collection to take longer GUIDE SHARE EUROPE 03/06/2009 10 . and free memory. Because a large heap takes longer to fill. The JVM heap size determines how often and how long the VM spends collecting garbage. the object is garbage. The goal of tuning your heap size is to minimize the time that you spend doing garbage collection while maximizing the number of clients that you can handle at a given time. It is a repository for live objects. but occurs more frequently. If you set your heap size in accordance with your memory needs. full garbage collection is slower. However. If you set a large heap size. The Java heap parameters also influence the behavior of garbage collection. When an object can no longer be reached from any pointer in the running program. full garbage collection is faster. An acceptable rate for garbage collection is application-specific and should be adjusted after analyzing the actual time and frequency of garbage collections. Increasing the heap size supports more object creation.

It is not unusual for the total memory consumption of the VM to exceed the value of -Xmx. -Xmx limits the java heap size. you may see the amount of memory you use exceed the amount you have specified for Xmx. You achieve best performance by individually tuning each of your applications. It is good practice with server-side Java applications like Resin to set the minimum -Xms and maximum -Xmx heap sizes to the same value. 03/06/2009 GUIDE SHARE EUROPE 11 . and `g' or `G' to indicate gigabytes. For performance analysis. java will allocate memory for other things. If you monitor your java process with an OS tool like top or taskmanager. `m' or `M' to indicate megabytes. the initial and maximum heap sizes should be equal. including a stack for each thread.Setting the heap size -Xmx -Xms Append the letter `k' or `K' to the value to indicate kilobytes.

1. • Specifying Heap Size Values. See Turning On Verbose Garbage Collection and Redirecting Output. 5. If you find that you have a large amount of RAM remaining. Analyze the following: 1. • Typically.log file. you should use 80% of the available RAM (not taken by the operating system or other processes) for your JVM. If you find that your system is spending too much time collecting garbage (your allocated "virtual" memory is more than your RAM can handle). Make sure that the heap size is not larger than the available free RAM on your system. Turn on verbose garbage collection output for your Java VM and redirect both the standard error and standard output to a log file. what does the heap settle back down to after each full garbage collection? If the heap always settles to 85% free.3 Java HotSpot JVM. 8. What is your average memory footprint? In other words. set generation sizes. run more WebLogic Servers on your machine. 3. Use the -verbosegc option to measure exactly how much time and resources are put into garbage collection. Use as large a heap size as possible without causing your system to "swap" pages to disk. How long is garbage collection taking? Full garbage collection should not take longer than 3 to 5 seconds. 3. If you are using 1. compare the time stamps around the garbage collection. Monitor the performance of WebLogic Server under maximum load while running your application. lower your heap size. 7. you might set the heap size smaller. The amount of free RAM on your system depends on your hardware configuration and the memory requirements of running processes on your machine. 9. See your system administrator for help in determining the amount of free RAM on your system. How often is garbage collection taking place? In the weblogic. 6. 4. 03/06/2009 GUIDE SHARE EUROPE 12 . 2.Determining heap size 1. 2.

the state of steady memory utilization is If the heap free space settles at 85% or more.doc/info/exp/ae/rprf_javamemory. There is always a cost for garbage collection in Java applications.Garbage collection patterns For all three configurations.html 03/06/2009 GUIDE SHARE EUROPE 13 . consider decreasing the maximum heap size values because the application server and the application are underutilizing the memory allocated for heap. This example illustrates an important concept about the Java heap and its relationship to object Garbage collection also occurs less frequently and for short duration. If the application is not over utilizing objects and has no memory the total time in garbage collection is approximately 15%.boulder.jsp?topic=/com.

the more garbage collection it goes through. These objects are said to have "infant mortality. For information about specifying the size and ratios of the New and Old generation areas. and then they are moved out of the New area and into the Old. Garbage collection can occur in each generation when it fills up. live objects in Eden are copied into the next survivor space. The heap is divided into two general areas: New and Old." The longer an object survives. Attempt to cache objects for re-use. and the slower garbage collection becomes. If you can arrange for most of your objects to survive less than one collection. Many objects become garbage shortly after being allocated. 03/06/2009 GUIDE SHARE EUROPE 14 . The rate at which your application creates and releases objects determines how often garbage collection occurs. Eden is the area where new objects are allocated. generational garbage collection considers the lifetime of an object to avoid extra work. garbage collection is very efficient. impacting your performance. whenever possible. When garbage collection occurs. While naive garbage collection examines every living object in the heap. see Specifying Heap Size Values. Objects are copied between survivor spaces in this way until they exceed a maximum threshold. The New generation area is sub-divided further into Eden and two survivor spaces. When you manage memory in generations.3 Java HotSpot JVM uses generational garbage collection.Generational garbage collection The 1. Knowing that a majority of objects die young allows you to tune for efficient garbage collection. Poorly sized generations cause frequent garbage collection. rather than creating new objects. you create memory pools to hold objects of different ages.

Resin won't be able to respond to requests and they'll start to back up. Since this process "stops the world". you don't want the Eden to be too big or it will take too long for the GC to look through it for space that can be reclaimed. Does the IBM JVM support -Xmn ? 03/06/2009 GUIDE SHARE EUROPE 15 .Garbage collection There are essentially two GC threads running. The amount allocated for the Eden generation is the value specified with -Xmn. One is a very lightweight thread which does "little" collections primarily on the Eden (a. eventually the older generation will start to run out of room causing the Full GC thread to run (nearly) continuously. If there is a memory leak or inadequate heap allocated. Young) generation of the heap. The other is the Full GC thread which traverses the entire heap when there is not enough memory left to allocate space for objects which get promoted from the Eden to the older generation(s). The amount allocated for the older generation is the value of -Xmx minus the Xmn.k. Generally.a.

set the minimum and maximum heap sizes to the same value. 03/06/2009 GUIDE SHARE EUROPE 16 . For this type of investigation. allowing you to separate your garbage collection messages from the rest of the messages on stderr. Choose a representative. To ensure meaningful statistics. you gain insight as to whether the application is over-utilizing objects. repetitive workload that matches production usage as closely as possible. user errors included.You can use garbage collection to evaluate application performance health.out – This option redirects -verbosegc messages to a file. -Xverbosegc:file=/tmp/gc$$. run the fixed workload until the application state is steady. It usually takes several minutes to reach a steady state. – It also provides a performance advantage because writes to files are buffered better than writes to a character stream like stderr. Garbage collection can even detect the presence of memory leaks. By monitoring garbage collection during the execution of a fixed workload.

Use a Java profiler to determine which objects to target. adding memory. there are two ways to clear the bottleneck. If you can not optimize the application. Additional memory allows each clone to maintain a reasonable heap size. 03/06/2009 GUIDE SHARE EUROPE 17 . processors and clones might help. Additional processors allow the clones to run in parallel.If the information indicates a garbage collection bottleneck. The most cost-effective way to optimize the application is to implement object caches and pools.

compact: 0 ms> <GC(32): refs: soft 0 (age <= 32). sweep: 2 ms. 30% free> <AF[7]: completed in 1 ms> 03/06/2009 GUIDE SHARE EUROPE 18 . need 16400 bytes. 56% free (7763712/13761024). final 0.> <GC(32): freed 5676656 bytes. phantom 0> Bad garbage collection messages – – – – – – – – – – <AF[7]: Allocation Failure.Good garbage collection messages – – – – – <GC(32): GC cycle started Fri Jun 12 09:20:25 2009 <GC(32): freed 0 bytes from Transient Heap 100% free (523776/523776) and. 770 ms since last AF> <AF[7]: managing allocation failure. action=3 (114624/9697792)> <GC(8): need to expand mark bits for 13761024-byte heap> <GC(8): expanded mark bits by 63488 to 215016 bytes> <GC(8): need to expand alloc bits for 13761024-byte heap> <GC(8): expanded alloc bits by 63488 to 215016 bytes> <GC(8): need to expand FR bits for 13761024-byte heap> <GC(8): expanded FR bits by 126976 to 430036 bytes> <GC(8): expanded heap by 4063232 to 13761024 bytes. weak 0.. in 12 ms> <GC(32): mark: 10 ms..

0" ?> <verbosegc version="20081027_AB"> <initialized> <attribute name="gcPolicy" value="-Xgcpolicy:optthruput" /> <attribute name="maxHeapSize" value="0x4000000" /> <attribute name="initialHeapSize" value="0x4000000" /> <attribute name="compressedRefs" value="false" /> <attribute name="pageSize" value="0x1000" /> <attribute name="requestedPageSize" value="0x1000" /> </initialized> </verbosegc> 03/06/2009 GUIDE SHARE EUROPE 19 .VERBOSE=gc Writes XML to stderr Example: <?xml version="1.

Memory leaks Memory leaks in the Java language cause excessive garbage collection – More damaging than memory overuse because it ultimately leads to system instability. Garbage collection occurs more frequently until the heap is exhausted and Java code fails with a fatal out-of-memory exception Memory leaks occur when an unused object has references that are never freed – Commonly occur in collection classes such as Hashtable Can still hold a reference to an object even after real references are deleted 03/06/2009 GUIDE SHARE EUROPE 20 .

there is a memory leak.Tips for spotting memory leaks Repeat runs with increasing duration Graph of used memory should have a sawtooth shape – Each drop corresponds to a garbage collection There is a memory leak if one of the following occurs: – Sawtooth pattern looks like a staircase – Sawtooth pattern has an irregular shape. 03/06/2009 GUIDE SHARE EUROPE 21 . Look at the difference between number of freed and allocated objects – If the gap between the two increases over time.

JVM on z 03/06/2009 GUIDE SHARE EUROPE 22 .

2 JVM storage (Classcache=YES) Worker JVM Subsequent JVMs Worker JVM Master JVM Classcache 1st JVM + Master Common code XMX +10 meg XMX +10 meg XMX +10 meg JVMCCSIZE 18 meg Storage Cushion 20 meg ELSQA/ESWA Shrlibrgnsize 03/06/2009 GUIDE SHARE EUROPE 23 .JDK 1.4.

Java 5 JVM storage (Classcache=YES) JVM Subsequent JVMs JVM JVM Classcache 1st JVM Common code XMX +18 meg XMX +18 meg XMX +18 meg JVMCCSIZE 5 meg Storage Cushion 20 meg ELSQA/ESWA Shrlibrgnsize 03/06/2009 GUIDE SHARE EUROPE 24 .

JDK 1.4.2 JVM storage (Classcache=No) JVM subsequent JVMs JVM JVM 1st JVM Common code XMX +26 meg XMX +26 meg XMX +26 meg 10 meg Storage Cushion 20 meg ELSQA /ESWA Shrlibrgnsize 03/06/2009 GUIDE SHARE EUROPE 25 .

Java 5 JVM storage (Classcache=No) JVM subsequent JVMs JVM JVM 1st JVM Common code XMX +20 meg XMX +20 meg XMX +20 meg 5 meg Storage Cushion 20 meg ELSQA /ESWA Shrlibregionsize 03/06/2009 GUIDE SHARE EUROPE 26 .

Tuning Unix System Services (USS) for your JVM Set your JVM and CICS executables to be in a USS shared library (Shared Library Extended) – extattr +l jvm* – extattr +l cics*.so Put directory containing application classes into zFS 03/06/2009 GUIDE SHARE EUROPE 27 .

CICS tuning 03/06/2009 GUIDE SHARE EUROPE 28 .

JVM modes Single use Continuous (reusable but not reset) 03/06/2009 GUIDE SHARE EUROPE 29 .

Single use JVM REUSE=NO in JVM Profile Cannot be used with shared classcache JVM destroyed and rebuilt between program invocations 03/06/2009 GUIDE SHARE EUROPE 30 .

Continuous mode JVM REUSE=YES in JVM Profile Can be used with shared classcache JVM not reset between program invocations Highest level of performance. Less protection from effects of an earlier program invocation in the same JVM 03/06/2009 GUIDE SHARE EUROPE 31 .

“bar” Heap JVM Heap JVM Heap JVM 16 Mb 31-bit . there is an MVS JVM running in the CICS address space 03/06/2009 GUIDE SHARE EUROPE 32 .“line” For each JVM program running in CICS.Java resource usage 2Gb 64-bit .

4.Setting up the shared classcache Specify the following SIT parameters: – JVMCCSTART=YES – JVMCCSIZE=64M Specify the following in the JVM profile – CLASSCACHE=YES JIT’d classes are not cached due to a new JIT optimization method For set up info on Java 1.2 which use master and worker JVMs see – zAAP and the art of CICS Java tuning (GSE Nordic 2005) 03/06/2009 GUIDE SHARE EUROPE 33 .gse-nordic.

Run workload for long enough to ensure all classes are loaded and JITed 2.Tuning the shared classcache 1. Use CEMT INQ CLASSCACHE to check on the amount of storage 3. Set JVMCCSIZE SIT parameter large enough to contain the amount of storage used 03/06/2009 GUIDE SHARE EUROPE 34 .

2) – Occurs at a target heap utilisation GC_HEAP_THRESHOLD parameter in the JVM profile (default 85%) 100% implies no CICS-scheduled GC – Performed asynchronously in a CICS system task Application does not suffer bad response times simply because a scheduled GC takes place New transaction CJGC Unscheduled GC within the JVM can still occur at any time – But with a little tuning you can minimise the likelihood of this happening 03/06/2009 GUIDE SHARE EUROPE 35 .The CICS garbage collection model New Garbage Collection scheduling algorithm (introduced in CICS TS 3.

Tuning LE enclaves for your CICS JVMs JVM LE enclave uses storage allocations set by CICS Use user exit DFHJVMRO to modify default settings – Assembler URM – Invoked before invoking CEEPIPl to create new LE enclave for JVM 03/06/2009 GUIDE SHARE EUROPE 36 .

DB2 tuning 03/06/2009 GUIDE SHARE EUROPE 37 .

mx=200M 03/06/2009 GUIDE SHARE EUROPE 38 .mx=700M ms=200M.Environment tuning hints Tune your JVM heap Default heap size is – – ms = 1M mx = 8M Values between 300M and 400M are common in a production environment Set ms and mx to an equal value Heap Size Study Trans/sec ms=700M.

Heappools Set environment variable: _cee_runopts = "heappools(on)“ Controls an optional heap storage management algorithm – Designed to improve the performance of multithreaded C/C++ applications – Targetted at applications with high usage of malloc(). and free() 03/06/2009 GUIDE SHARE EUROPE 39 . realloc(). calloc().

Environment tuning hints Recommended DB2 BIND options – DYNAMICRULES(BIND) table access privileges of the binder used during program execution – QUALIFIER creator (schema name) for unqualified tables and views Use dynamic SQL statement caching – – – – – Avoids full cost of preparing SQL Processing cost close to static SQL Recommended for JDBC/SQLJ Cursor controlled update/delete executed dynamically in SQLJ 03/06/2009 GUIDE SHARE EUROPE 40 .

use ISOLATION(CS) Use SQLJ and/or JDBC batching and multi row operations for bulk processing Use JCC Type 2 driver if application server local (same LPAR) as DB2.General tips Turn off JDBC autocommit and cursor with hold Use VARCHAR or VARGRAPHIC as SQL column data type for string columns If most of the database processing is from Java applications then use Unicode as the database encoding scheme instead of EBCDIC Use matching java data type that matches SQL column data type Only read and update SQL columns as absolutely necessary Code to explicitly close resources (SQL results sets. prepared statements. otherwise use JCC Type 4 driver for application server connecting over distributed connection to DB2. callable statements) Use SQLJ (static SQL) instead of JDBC (dynamic SQL) Turn on dynamic statement caching and tune size of cache for JDBC (dynamic SQL) For JDBC (dynamic SQL) used in web based transaction processing use parameter markers rather than literals Use BIND options of ISOLATION(CS) and CURRENTDATA(NO) for SQLJ packages Use BIND option of CURRENTDATA(NO) for ISOLATION(CS) package for JDBC Do not allow JDBC isolation level to default . 03/06/2009 GUIDE SHARE EUROPE 41 .

IMS tuning 03/06/2009 GUIDE SHARE EUROPE 42 .

4 seconds (Java tuning and WFI) – Check region parameter to avoid out of memory exceptions – REGION=0M uses less storage than REGION=392M – When calculating Region size take into account that garbage collection requires up to – mx parameters amount of additional storage 03/06/2009 GUIDE SHARE EUROPE 43 .Environment tuning Software pre-reqs – IBM Java 5 JDK for z/OS – IMS V9 with enabling PTF for Java 5 JDK or IMS V10 JVM requires tuning – Don’t start with un-tuned JVM Customer case CPU time (SMF based) of 140 seconds (Java Defaults) versus 78 seconds (with Java tuning) versus 2.

POSIX=(ON) LE Parameters Tuning RPTSTG=(ON) will display usage and suggested values 03/06/2009 GUIDE SHARE EUROPE 44 .10.0.128.ANY.0.16K.32.10.Environment tuning Default LE Settings are unsuitable for Java Consider customizing LE option like recommended for WAS on z/OS: ANYHEAP(2M.4M.1M.128K).ANY.KEEP). 10).0.10.128K.10.10.10. XPLINK=(ON).NONE.16K. TERMTHDACT=(UADUMP). HEAPPOOLS(ON. HEAP(80M.10.ANY.10. STORAGE(NONE.ANY.KEEP.NONE.KEEP.10.64K. THREADSTACK(OFF.10. STACK(64K.128K).128K.2048.10.0.1024.

execute Java in Batch where profiling is available GUIDE SHARE EUROPE 45 Java Profiling – – – – 03/06/2009 .Environment tuning JVM tuning in MPR – – – Environment variable for COBOL JVM Options must be used for JVM parameters: COBJVMINITOPTIONS=-Xoptionsfile=/u/itso12/ file) Sample file: -Xcodecache=16M -Xquickstart -Xshareclasses:name=imscobol -Xscmx32M -Xmx96M -Xms64M Method to find the Java code where most CPU is spend and which Java methods are executed how often during program invocation Currently not available in IMS regions Eclipse TPTP can be In PL/I JVM Options are supplied at JVM API invokation as string (recommendation to point to jvm. but has some pitfalls Suggesting.

Tooling 03/06/2009 GUIDE SHARE EUROPE 46 .

authenticate=false -Dcom.port=<port number> Article on using jConsole – Free tool supplied with the JVM – Can be found in bin directory Connects to be a long running JVM – or a JVM in a CICS pool Add these parameters to server JVM 03/06/2009 GUIDE SHARE EUROPE 47 -Dcom.jmxremote.sun.sun.Real-time analysis .management.

MemoryManagerMXBean – The management interface for a memory manager. MemoryMXBean – The management interface for the memory system of the Java virtual machine. 03/06/2009 GUIDE SHARE EUROPE 48 . CompilationMXBean – The management interface for the compilation system of the Java virtual machine.Write your own .management ClassLoadingMXBean – The management interface for the class loading system of the Java virtual GarbageCollectorMXBean – The management interface for the garbage collection of the Java virtual machine.

lang. RuntimeMXBean – The management interface for the runtime system of the Java virtual ThreadMXBean – The management interface for the thread system of the Java virtual machine.Write your own . OperatingSystemMXBean – The management interface for the operating system on which the Java virtual machine is MemoryPoolMXBean – The management interface for a memory pool. 03/06/2009 GUIDE SHARE EUROPE 49 .

com/javase/6/docs/api/java/lang/management/ – an example Example program to print heap usage: public static void main(String[] args) { java. } Output from laptop JVM: init = 4194304(4096K) used = 643944(628K) committed = 4194304(4096K) max = 536870400(524287K) http://java.MemoryMXBean memory = ManagementFactory. System.lang.getMemoryMXBean().sun.html 03/06/2009 GUIDE SHARE EUROPE 50 .println(memory.getHeapMemoryUsage())

Offline dump analysis tools Heap analyzer – Graphical tool for discovering possible heap leaks – Eclipse memory analyzer tool (MAT) – http://www.html 03/06/2009 GUIDE SHARE EUROPE 51 .org/mat/ IBM DTFJ adapter for Eclipse MAT –

com/developerworks/forums/ D=13940438 IBM Monitoring and Diagnostic Tools for Java – http://www.html 03/06/2009 GUIDE SHARE EUROPE 52 .ibm.Offline garbage collection analysis tools Extensible Verbose Toolkit (EVTK) – – Data visualizer for verbose GC and GC trace

Links 03/06/2009 GUIDE SHARE EUROPE 53 .

Useful Java blogs and posts /dev/websphere – Automated Heap Dump Analysis: Finding Memory Leaks with One Click – 03/06/2009 GUIDE SHARE EUROPE 54 .com/devwebsphere/ Eclipse MAT blog – The Power of Aggregation: Making Sense of the Object in a Heap Dump – http://dev.eclipse.

com/developerworks/ibm/library/i-garbage2/ Understanding the IBM Java Garbage Collector part 3 – Understanding the IBM Java Garbage Collector part 2 – http://www.html 03/06/2009 GUIDE SHARE EUROPE 55 .3-part tutorial on IBM JVM garbage collector Understanding the IBM Java Garbage Collector part 1 – http://www.

html Fine-tuning Java garbage collection performance – WAS – Java memory tuning tips – http://publib.0/docs/tooldocs/ 03/06/2009 GUIDE SHARE EUROPE 56 – Monitoring and management tools – WAS – Tuning Java virtual machines – ere.doc/info/exp/ae/tprf_tunejvm.html#manage Monitoring and management for the Java platform – http://java.jsp?topic=/com.sun.jsp?topic=/

zSeries PD/CICS/Icing Sales .CICS Communities CICS Communities and Information CICS Transaction Server V4. etc. report problems. Links and resources. demos and other cool stuff CICSFluff . suggest improvements.1 & CICS Explorer – – – ibm. Ask questions.Other CICS videos Link Home page CICS Explorer Forum CICS Explorer Public Forum New! CICS Hub on the Rational COBOL Café – – CICS Hub Twitter CICS Blog – – – – – Twitter CICS eNews YouTube channels Blog Facebook eNews YouTube 03/06/2009 GUIDE SHARE EUROPE 57 . chat IBM developerWorks forum with FAQs. ISV Subscribe to the IBM_System_z channel IBM_CICS CICSFluff to get CICS news flashes Comment and opinion at Subscribe for news about CICS and related products CICS Explorer .

Summary Java considerations JVM tuning JVM on z CICS tuning DB2 tuning IMS tuning Tooling Links 03/06/2009 GUIDE SHARE EUROPE 58 .

Questions? 03/06/2009 GUIDE SHARE EUROPE 59 .

Sign up to vote on this title
UsefulNot useful