You are on page 1of 42

1

Copyright 2012, Oracle and/or its affiliates. All rights reserved.

Insert Information Protection Policy Classification from Slide 8

Triage and Diagnostics for E-Business Suite Systems


2 Copyright 2012, Oracle and/or its affiliates. All rights reserved.

Dave Suri, Project Lead, Applications Performance Gustavo Jimenez, Senior Development Manager, ATG
Insert Information Protection Policy Classification from Slide 8

Agenda
Architecture Patches/Certified Browsers/JDK versions JVM Tuning and Tools Forms Tools (strace/FRD) Java Concurrent Program Performance Case studies

Copyright 2012, Oracle and/or its affiliates. All rights reserved.

Architecture Overview
The applications tier connects to the database via JDBC connections The application connections are pooled, so a database session is used to service requests from multiple users throughout the lifetime of the connection Java concurrent programs run in their own JVM - each has a special configuration. Forms runs in Socket mode (Forms Listener) or Servlet mode (HTTP/s to the JVM) Apache HTTP forwards requests to OC4J through mod_oc4j. OC4J routes them to the right oacore JVM while OPMN helps keep the uptime high with death detection.
4 Copyright 2012, Oracle and/or its affiliates. All rights reserved.

Architecture Overview - R12 OA-Framework Flow

Copyright 2012, Oracle and/or its affiliates. All rights reserved.

Architecture Overview - R12 OA-Framework Flow

Copyright 2012, Oracle and/or its affiliates. All rights reserved.

Architecture Overview - R12 OA-Framework Flow

Copyright 2012, Oracle and/or its affiliates. All rights reserved.

Architecture Overview Forms Servlet Mode


The Forms Listener Servlet is a Java servlet runs inside the OC4J container The listener servlet delivers the ability to run Forms applications over HTTP The listener servlet manages the creation of Forms Runtime process (frmweb) for each client and manages any further communications between the client and its designated runtime process. The main difference is the traffic goes through HTTP/HTTPS and leverages load balancing/security capabilities of the MT. Does not require a special port to be open

Copyright 2012, Oracle and/or its affiliates. All rights reserved.

Architecture Overview Forms Socket Mode


Forms Direct Socket Connection
Client requests forms application OHS receives requests & sends the request to the forms listener Listener contacts a forms runtime engine (frmsrv/f60srvm) frmsrv/f60srvm spawns a runtime process, creates connection to the database & passes the forms params (menu, sid, form name) forms runtime communicates with the database & gets the forms logic (pl/sql,etc). forms service sends applet & the response back to client forms is running, page is rendered and direct connection to runtime process

Copyright 2012, Oracle and/or its affiliates. All rights reserved.

Agenda
Architecture Patches/Certified Browsers/JDK versions JVM Tuning and Tools Forms Tools (strace/FRD) Java Concurrent Program Performance Case studies

10

Copyright 2012, Oracle and/or its affiliates. All rights reserved.

Patches, Browsers & Versions


Document ID 244040.1 Recommended Performance Patches for the Oracle E-Business Suite Recommended performance patches for all the modules and technology stack components are consolidated in this note Apply latest Release Update Packs Release 12.1.3 - Patch 9239090 Document ID 1080973.1 Oracle E-Business Suite Release 12.1.3 Readme Release 12.1 - Patch 7303030 Document ID 752619.1 Oracle E-Business Suite Release 12.1.1 Maintenance Pack Installation Instructions Release 12.0 - RUP6/7 Patch 6728000 / 7577660 Document ID 743368.1 Oracle E-Business Suite Release 12.1.1 Maintenance Pack Installation Instructions

11

Copyright 2012, Oracle and/or its affiliates. All rights reserved.

Patches, Browsers & Versions


Upgrade to the latest certified technology stack EBS 12.1 is certified with 11gR2, Oracle Apps Server 10gR3, JRE 6.0. Document ID 561580.1 Document ID 380482.1 Oracle E-Business Suite Release 12 Technology Stack Documentation Roadmap Central document for installation, apps, database, security and other resources. OA Framework Applications Recommended Patches for Applications: note 275880.1 (Framework Roadmap) Oracle E-Business Suite Recommended Performance Patches Document id: 244040.1 Oracle E-Business Suite Recommended Performance Patches

For a list of certifications, use Document ID 561580.1 Oracle E-Business Suite Release 12.1 Release Content Documents

12

Copyright 2012, Oracle and/or its affiliates. All rights reserved.

Patches, Browsers & Versions


Browsers, OAF Pages in R12.x

Firefox 3.6 is supported in 11i, R12 Windows 7 / Internet Explorer 8 / Firefox 3.6 supported in 11i/R12 Oracle E-Business Suite 12.1.1 or higher R12.0 => use R12.ATG_PF.A.DELTA.6 & Forms 10.1.2.3 or higher

11.5.10 => CU2, 11i.ATG_PF.H Rollup 7 with Document ID 783600.1 Document ID 1103276.1, 389422.1 (R12), 285218.1 (11i)
R12.1.2 have ajax customizable homepage for increased performance R12.1.3 available as patch #9239090

13

Copyright 2012, Oracle and/or its affiliates. All rights reserved.

Agenda
Architecture Patches/Certified Browsers/JDK versions JVM Tuning and Tools Forms Tools (strace/FRD) Java Concurrent Program Performance Case studies

14

Copyright 2012, Oracle and/or its affiliates. All rights reserved.

JVM Tuning and Tools


Response Time/JVM Usage OutOfMemoryErrors Deadlocks Some great tools Jstat : monitor JVM Eclipse MAT : analyze heap dumps IBM Thread Dump Analyzer : analyze thread dumps Visual VM : profile, monitor, dump the JVM Jconsole : similar to Visual VM except for profiling Jrockit Mission Control : flight recorder, profiler, dump Thread Dump Analyzer
15 Copyright 2012, Oracle and/or its affiliates. All rights reserved.

JVM Tuning and Tools


Response Time/JVM Usage
First, rule out any SQL issues. Document ID 357597.1 How To Generate A SQL Trace In OA Framework For Oracle Applications
Monitor Request Response Time
Modify $ORA_CONFIG_HOME/10.1.3/Apache/Apache/conf/httpd.conf
LogFormat "%h %T Logs: $LOG_HOME/ora/10.1.3/Apache/access_log*

If there are no database-related issues, take a look into the JVM


Techniques you can use Thread dumps Send to Oracle Support for analysis GC logs for sizing, Profilers - Send to Oracle Support for analysis

16

Copyright 2012, Oracle and/or its affiliates. All rights reserved.

JVM Tuning and Tools


Response Time/JVM Usage
GC tuning Review the frequency of collections, especially major collections (i.e. Full GC) in the logs for healthy and unhealthy activity Use jstat tool with the gc option to review FCGT count Important columns to pay attention to are survivor spaces, eden and old gen Jstat column details (observe the grey highlighted columns) Timestamp, S0C, S1C, S0U, S1U, EC, EU, OC, OU, PC, PU, YGC, YGCT, FGC, FGCT, GCT 933.9,59904.0,57728.0,0.0,40504.5,230784.0,0.0,699072.0, 666307.1,131072.0,119088.8,31,2.054,1,0.000,2.054

17

Copyright 2012, Oracle and/or its affiliates. All rights reserved.

JVM Tuning and Tools


Response Time/JVM Usage
GC tuning Bigger heaps => Full GC will take longer Longer GCs => users may experience pauses Increase the number of JVMs used may provide the ability to handle more users with faster GC times. Each JVM has a smaller Xmx,Xms memory footprint so GCs will be faster! Also try to change the JVM collectors used and track the performance with Jconsole Enable the ParallelGC collector with ParallelGCThreads = 2

18

Copyright 2012, Oracle and/or its affiliates. All rights reserved.

JVM Tuning and Tools


Response Time/JVM Usage
GC tuning Unhealthy JVM with FullGC activity 898753.288: [Full GC 825983K->743065K(913024K), 15.8442715 secs] 898772.243: [Full GC 825983K->750548K(913024K), 13.3512933 secs] 898798.124: [Full GC 825983K->731902K(913024K), 13.3466676 secs] 898832.082: [Full GC 825983K->738500K(913024K), 13.1591655 secs] 898857.239: [Full GC 825973K->728250K(913024K), 16.4536985 secs] Healthy JVM with FullGC activity 902130.499: [GC 829181K->716211K(935296K), 0.2895464 secs] 902220.498: [GC 833715K->721542K(935296K), 0.3210096 secs] 902400.049: [GC 839046K->725265K(936768K), 0.3448279 secs] 902400.394: [Full GC 725265K->652064K(936768K), 8.5214844 secs] 902455.778: [GC 771040K->661918K(935296K), 0.1618026 secs]

19

Copyright 2012, Oracle and/or its affiliates. All rights reserved.

JVM Tuning and Tools


Response Time/JVM Usage
GC tuning Unhealthy JVM with FullGC activity 898753.288: [Full GC 825983K->743065K(913024K), 15.8442715 secs] 898772.243: [Full GC 825983K->750548K(913024K), 13.3512933 secs] 898798.124: [Full GC 825983K->731902K(913024K), 13.3466676 secs] 898832.082: [Full GC 825983K->738500K(913024K), 13.1591655 secs] 898857.239: [Full GC 825973K->728250K(913024K), 16.4536985 secs] Healthy JVM with FullGC activity Infrequent 902130.499: [GC 829181K->716211K(935296K), 0.2895464 secs] and shorter 902220.498: [GC 833715K->721542K(935296K), 0.3210096 secs] 902400.049: [GC 839046K->725265K(936768K), 0.3448279 secs] times! 902400.394: [Full GC 725265K->652064K(936768K), 8.5214844 secs] 902455.778: [GC 771040K->661918K(935296K), 0.1618026 secs]

Full GC too frequent!!!

20

Copyright 2012, Oracle and/or its affiliates. All rights reserved.

JVM Tuning and Tools


Out of Memory Errors
Common causes of OutOfMemoryError Sizing Issues Heap is undersized for expected load Memory Leak Some objects not cleaned up Strong reached references, connection objects, etc.
Memory Hemorrhage Reading large data sets consuming vast amounts of memory in any period of time Connection leaks leading to vast memory/stack allocations SQL Statements/caching may lead to excess memory used in JDBC libraries

21

Copyright 2012, Oracle and/or its affiliates. All rights reserved.

JVM Tuning and Tools


Out of Memory Errors
Out of Memory Heap Dump Example
This will be present in the $LOG_HOME/ora/10.1.3/opmn/default_group~oacore~default_group~1.log

10067.492: [Full GC 118783K->118783K(124928K), 1.2127460 secs] 10068.706: [Full GC 118783K->118783K(124928K), 1.2166770 secs]

10069.924: [Full GC 118783K->117977K(124928K), 1.2416330 secs]


java.lang.OutOfMemoryError: GC overhead limit exceeded Dumping heap to java_pid8515.hprof ... Heap dump file created [149076109 bytes in 10.240 secs]

Heap Location cd $INST_TOP/../../../apps/tech_st/10.1.3/j2ee/home ls -l *hprof -rw------- 1 oraperf dba 149076109 Jan 26 11:11 java_pid8515.hprof

22

Copyright 2012, Oracle and/or its affiliates. All rights reserved.

JVM Tuning and Tools

23

Copyright 2012, Oracle and/or its affiliates. All rights reserved.

JVM Tuning and Tools


JVM Tools
JDK 6 Jconsole Enable heap dumps to be collected from the attached JVM Goto java.sun.management | HotSpotDiagnostic | Operations | dumpHeap P0 : absolute file name (directory + filename) P1: leave to true so that only reachable objects are dumped
Use any binary hprof heap analyzer to analyze the heap dump and see which objects take up the most space JDK 6u7 and above has VisualVM Does thread/heap dumps, profiling, graphs

24

Copyright 2012, Oracle and/or its affiliates. All rights reserved.

JVM Tuning and Tools


JVM Tools
Jstat Shows the JVM heap internals Shows Timestamps and Capacity/Utilization for Survivor Spaces where live objects become old enough to copy Eden new objects are created here Old Generation long lived objects are kept here Young Generation composed of eden + 2 survivor spaces Perm Generation native heap and other non-heap objects Shows the Full GC and Young GC collection count To get the data from PID nnnnn every 60s for 2 hours jstat -gc -t nnnnn 60000 120 > jstat_gcutil_pid_31041 &
25 Copyright 2012, Oracle and/or its affiliates. All rights reserved.

JVM Tuning and Tools

26

Copyright 2012, Oracle and/or its affiliates. All rights reserved.

Agenda
Architecture Patches/Certified Browsers/JDK versions JVM Tuning and Tools Forms Tools (strace/FRD) Java Concurrent Program Performance Case studies

27

Copyright 2012, Oracle and/or its affiliates. All rights reserved.

Forms Tools (strace/FRD)


Forms Next, use strace (linux), trace (solaris) to attach to the pid: strace -o formsR12_frmweb.log -p 22874 strace -o formsR12_frmsrv.log -p 17347 The logs will show the activity of the forms server as well as the listener while the user is performing their actions. FRD, Tracing, Logging for Forms Document ID 438652.1 FRD gives clues into forms code execution, parameters, variables, item values Along with strace, FRD can provide valuable clues into researching issues.
28 Copyright 2012, Oracle and/or its affiliates. All rights reserved.

Agenda
Architecture Patches/Certified Browsers/JDK versions JVM Tuning and Tools Forms Tools (strace/FRD) Java Concurrent Program Performance Case studies

29

Copyright 2012, Oracle and/or its affiliates. All rights reserved.

Java Concurrent Program Performance


Observe the concurrent program log file after adding verbose:gc to see if the JVM is healthy or is it suffering from low memory
Increase the Memory when you encounter OOME in the logs location: Concurrent -> Program -> Define

Remember it runs in a separate JVM from oacore


All the regular java support utilities will work (jstat, eclipse mat, tda, etc) Can specify regular JVM commands in the define field to observe the JCP behaviour during the run

30

Copyright 2012, Oracle and/or its affiliates. All rights reserved.

Agenda
Architecture Patches/Certified Browsers/JDK versions JVM Tuning and Tools Forms Tools (strace/FRD) Java Concurrent Program Performance Case studies

31

Copyright 2012, Oracle and/or its affiliates. All rights reserved.

Case Studies
Oracle Advanced Product Catalog
Issue Slow Performance with Rules Enabled vs Rules Disabled
Tool Used IBM Thread Dump Analyzer Analyzes thread dumps Can compare multiple thread dumps Shows different thread states with different colors Primary area of concentration is : AJPRequestHandlerHTTPThreadGroup-* These thread groups are the Apache/AJP Requests coming to oacore Solution
Thread dumps showed JVM waiting for DB to send results. This eventually lead to the creation of a cache to improve Rules Enabled performance
32 Copyright 2012, Oracle and/or its affiliates. All rights reserved.

Case Studies

33

Copyright 2012, Oracle and/or its affiliates. All rights reserved.

Case Studies
Saving a Oracle Service Request
Issue Error saving a service request
Tools Used Forms FRD

FRD showed a NULL being passed to a variable named control.mfg_org_id Executing USER_EXIT Built-in: In Argument 0 - Type: String Value: FND FFLEX PRE-UPDATE CSXSRISR, 17, Built-In, Entry, -1424992664, USER_EXIT Analyzed the CSXSRISR_LIBRARY.Init() method to find the NULL Analyzed the SR_CONTACTS.PRE_UPDATE to find how the NULL was passed to the control.mfg_org_id
Solution
Product team notified of the issue and a test case was given to isolate the affected code
34 Copyright 2012, Oracle and/or its affiliates. All rights reserved.

Case Studies

Java Concurrent Program Issue


Out of Memory error experienced when running a Java Concurrent Program

Tool Used Eclipse MAT Analyzed heap dump Showed which package consumes significant percentage of the heap Helps derive clues and questions as to what is happening in the tech stack layers Can see what objects are occupying space and their values Solution
The Java Concurrent Programs max JVM memory was increased to 1GB to allow the program to finish successfully

35

Copyright 2012, Oracle and/or its affiliates. All rights reserved.

Case Studies

36

Copyright 2012, Oracle and/or its affiliates. All rights reserved.

Case Studies

37

Copyright 2012, Oracle and/or its affiliates. All rights reserved.

Case Studies

38

Copyright 2012, Oracle and/or its affiliates. All rights reserved.

Useful Documents
Patches 244040.1 561580.1 Roadmap 380482.1 275880.1 Browsers 1103276.1 389422.1 Oracle E-Business Suite Recommended Performance Patches Oracle E-Business Suite Release 12.1 Release Content Documents Oracle E-Business Suite Release 12 Technology Stack Documentation Roadmap Oracle Application Framework Release 11i Documentation Road Map Is Windows 7 Certified With Oracle E-Business Suite? Recommended Browsers for Oracle E-Business Suite Release 12

285218.1
Setup Diagnosis 362851.1 357597.1 1160285.1 ATG 438652.1

Recommended Browsers for Oracle E-Business Suite 11i


Guidelines to setup the JVM in Apps E-Business Suite 11i and R12 How To Generate A SQL Trace In OA Framework For Oracle Applications R12: Forms Runtime Diagnostics (FRD), Tracing And Logging For Forms In Oracle Applications Application Technology Group (ATG) Product Information Center

39

Copyright 2012, Oracle and/or its affiliates. All rights reserved.

Q&A

40

Copyright 2012, Oracle and/or its affiliates. All rights reserved.

41

Copyright 2012, Oracle and/or its affiliates. All rights reserved.

42

Copyright 2012, Oracle and/or its affiliates. All rights reserved.

Insert Information Protection Policy Classification from Slide 8