Professional Documents
Culture Documents
---------------------------------------------------------------------------
Here are just three examples where you can quickly answer important questions about your system with
The main performance information is the time taken to serve a request. This is not included in the
access_log by default , but can easily be added by adding the following line to the httpd.conf file via an
This line can be added anywhere in the file, but would sensibly be placed after the existing LogFormat
entries. This directive adds an extra column into the access_log to show the time in seconds it took from
receiving a request to sending back a response. Valid log formats are described in the Apache
documentation.
The access_log includes the HTTP status code of the request. They are listed in full in RFC 2616 but the
codes we would be immediately concerned about are any in the 400 or 500 range.
For example:
• Status 500 (internal server error) may typically be seen for a JServ request and often means the
JVM has some kind of problem or has died.
For example: This entry may indicate that the JServ JVM is not responding to any requests:
• Status 403 (forbidden) could typically be seen for oprocmgr requests and often means there is a misconfiguration that
needs to be resolved.
For example: This entry in access_log may indicate a problem with system configuration (oprocmgr.conf):
Page: 1
3. Are users having problems accessing pages?
The status code of 200 means the request was successful, however a dash ( - ) for the "Bytes sent"
column, normally means that the request hit the Apache timeout. You would also see a time taken to serve
If you see this situation occurring regularly then your users are either navigating away from the browser
page before it has rendered, or are likely to be getting a "white screen of death" in their browser window
In this situation you need to identify why the requests are not being processed in good time, which is a large
subject in itself.
Hopefully these examples will inspire you to want to analyse your access_log, but I hear you ask, "Where
Luckily the access_log is a simple text file, so if you do not have commercial monitoring software you can
use some "quick and dirty" scripts to report just the exceptions you are interesting in seeing. For example I
## Start of script
##
## Check for HTTP statuses in 400 or 500 range for JServ
## or PLSQL requests only
##
awk ' $9>=400 && $9<=599 { print $0 }' access_log* | grep -e
"servlet" -e "\/pls\/" | grep -v .gif
##
## Check for requests taking more than 30 seconds to be returned
##
awk ' $11>30 {print $0} ' access_log*
##
## This one is not an exception report, you need to manually check
## Look for when the JVMs are restarting
##
grep "GET /oprocmgr-service?cmd=Register" access_log*
##
## End of script
[Editor: Due to formatting restrictions , if you're cutting-and-pasting this script, you must manually join the
line above ending with grep -e with the following line starting with "servlet" into a single unbroken line.]
Page: 2
Conclusion
When you call Oracle Support with a problem like apj12 errors in your mod_jserv.log, middle-tier Java
Virtual Machines (JVMs) crashing, or poor middle-tier performance, then it will often be suggested to
increase the number of JVM processes. So, the key question that likely occurs to you is, "How many JVMs
First, some quick background: web requests received by Oracle HTTP Server (Apache) to process Java
code is sent to one of four different types of JVM groups to be processed. You can see this in the
jserv.conf file:
The number of JVMs for each group is signified by the first number on each line.
• OACoreGroup is the default group. This is where most Java requests will be serviced
• DiscoGroup is only used for Discoverer 4i requests
• FormsGroup is only used for Forms Servlet requests
• XmlSvcsGrp is for XML Gateway, Web Services, and SOAP requests
In the example above, I have two JVMs configured for OACoreGroup and one JVM configured for each of
Page: 3
Factors Affecting Number of JVMs Required
Determining how many JVMs to configure is a complex approximation, as many factors need to be taken
into account. These include:
Luckily, Oracle Development have undertaken various performance tests to establish some rules of thumb
that can be used to configure the initial number of JVMs for your system.
OACoreGroup
DiscoGroup
o Use the capacity planning guide from Note 236124.1 "Oracle 9iAS 1.0.2.2 Discoverer 4i: A
Capacity Planning Guide"
FormsGroup
XmlSvcsGrp
In addition to this, Oracle generally recommends no more than 2 JVMs per CPU. You also need to confirm
there are enough operating system resources (e.g. physical memory) to cope with any additional JVMs.
The general guidelines above are just that -- they're very broad estimates, and your mileage will vary. The
Applications Technology Group is working on a JVM Sizing whitepaper that will provide guidelines based on
Until then, it's critical that you test your environment under load, using transactional tests that closely mirror
Page: 4
Here are a couple of quick-and-dirty tools that might be useful in sizing your JVMs.
REM
REM
REM
REM
Check the number of f60webmx processes on the Middle Tier server. For example:
Conclusion
• The number of required JVMs is extremely site-specific and can be complex to predict
• Use the rules of thumb as a starting point, but benchmark your environment carefully to see if
they're adequate
• Proactively monitor your environment to determine the efficiency of the existing settings and
reevaluate if required
Page: 5
Investigating java.lang.OutOfMemoryError with Apps 11i Middle Tier JVM
---------------------------------------------------------------------------------------------------------
If you had to guess what the error "java.lang.OutOfMemoryError" indicates when seen in your Apache
In most cases this message is telling you that the Java process has exhausted available memory and
cannot process some requests successfully. In many cases additional symptoms will be:
The Java Virtual Machine (JVM) will sometimes be restarted by oprocmgr as performance is so poor it
presumes the JVM has died.
There can be other causes. For example, this message can sometimes be given where there is the lack of
some other resource such as threads per process (kernel parameter "max_thread_proc" on HP-UX) but
this case should be easily identifiable from the exact message written out.
Every object created in Java takes some memory. Once all java code that references the java object has
completed, then the Garbage Collector (GC) automatically removes the object and claims back the memory.
• Insufficient memory allocated to the JVM to cope with the amount of work
• Suboptimal GC processing
• A memory leak -- Java code is not releasing an object when it should
• A Java code or operating system defect
• jserv.conf
This file defines how many JVMs are configured, as discussed
• jserv.properties
This file controls the JVM parameters in the "wrapper.bin.parameters=" entry. When you
install Apps originally or update the JDK version, the default settings are updated in your
CONTEXT.xml file ("s_jvm_options" for the OACoreGroup JVM) and controlled by AutoConfig
thereafter.
Page: 6
What can I do about OutOfMemoryError?
Configurator can sometimes require huge chunks of memory for a relatively long period of time, so Oracle
now recommends setting up separate JVMs specifically for Configurator. For more details see Setting Up a
Dedicated JServ for Running Only Oracle Configurator" (Metalink Note 343677.1).
An obvious choice, you may think, and this will often at least delay the effect of the issue, but may not
always solve it. You can achieve this by either increasing the number of JVMs or the memory settings ("-
Xmx512M" for example). In general, I would not recommend going much over 640MB per JVM, in order to
keep the Garbage Collection time to a reasonable level.
• Tuning GC
You can influence how Java performs Garbage Collection. I would normally recomend reviewing
and possibly backing out any non-default JVM parameters first, as these are sometimes just carried
forward because they were used with previous JDK versions. As always, any tuning should be
With multi-CPU servers using JDK 1.4 and higher, the following parameters have been seen to
-XX:+UseConcMarkSweepGC -XX:+UseParNewGC
Not always as easy as it sounds. Generating class histograms or java profiling may be the only
options.
Page: 7
What data should I gather to investigate further ?
The initial investigation should focus on trying to identify the pattern of memory usage building up and how
frequent GC is occurring. The tools and information described below may help to achieve this
-XX:+PrintGCTimeStamps -XX:+PrintGCDetails
-Xverbosegc
This allows you to see how much of each memory pool is being used. For example you may find it is
the "Permanent Generation" space that is running out of memory, rather than the "Tenured
Generation". If this turns out to be the case, the "-XX:MaxPermSize" parameter could be
-XX:+PrintClassHistogram
Implemented in JDK 1.4 and higher by some vendors. Can be very useful if available. Beware of
using this on HP-UX platforms running JDK 1.4 as it seems to crash the JVM rather than output the
data!
• Use JConsole
• Java Profiling
-Xrunhprof
Page: 8
This should tell you exactly where the memory is being used, but has a huge impact on performance
and is therefore not likely to be possible to gather this data, except for a lightly used test
environment.
• Other clues
Does the DMS output for the JVM show increasing active connections or threads?
Do JDBC connections to the database keep increasing? What module names do they relate to?
Does temporarily disabling "AM Pool" reduce memory consumption (Profile option "FND:
Are you seeing PL/SQL "cursor leaks"? See, "Understanding and Tuning the Shared Pool" (Metalink
Note 62143.1) in section "Checking hash chain lengths" for SQL to check this.
Are Java processes spinning to 100% CPU or is CPU use very low when the error occurs?
Review your operating system vendor's web site to look for known issues or required platform
My Production system is failing every few hours, what can I do right now?
A possible quick fix would be to increase memory and/or number of JVMs. Be sure to ensure that this won't
You could also consider a scheduled bounce of Apache every 12 or 24 hours if possible.
These steps may allow you to minimise the immediate impact on the business so you can investigate and
There are many resources on the Internet to describe how Java uses memory and what steps to take for
performance tuning. The best place to start is your hardware vendors web site as the options available
depend on their Java implementation and also the specific Java version you are using.
Page: 9
For example:
Conclusion
Dealing with Java processes running out of memory can sometimes be as simple as increasing the memory
Although each case will have its own unique considerations, I hope that the information in this article has
given you some ideas as to the general approach to take should the need arise.
Additional References
------------------------------------------------------------
Whenever you are planning to upgrade components in your E-biz 11i environment, you shouldn't forget to
review the Technology Stack components as well as the Applications 11i code.
There are some compelling technology benefits justifying the upgrade of your Apps Middle Tiers to JDK
version 1.5, but the best reason for me is the JConsole GUI monitoring tool, which I think is really cool
Page: 10
For a better description of JConsole and its capabilities you can review this Sun article, but in summary
provides a lightweight, always-on monitoring capability for your JVMs. Using JConsole, you can view
graphically the Heap usage, Threads, classes and other statistics in real time, which gives you a great idea
of activity inside the JVM process, delivered in an easily digestible form. I can happily sit and watch my
JVMs chug away for hours using JConsole, and is more fun than watching TV!
Page: 11
JConsole with Apps Release 11i
You can run JConsole either locally (from the server that you are monitoring) or remotely via a network
connection. With Apps 11i, the safest and easier option is to run locally.
Whether running local or remote, you need to add the following lines to your jserv.properties file. You
should make this change following "Customizing an AutoConfig Environment" (Metalink note 270519.1) to
wrapper.bin.parameters=-Dcom.sun.management.jmxremote
wrapper.bin.parameters=-Dcom.sun.management.jmxremote.ssl=false
Page: 12
Connecting Locally
Once you have started your JVMs using the new parameters described above, you simply launch JConsole
as the same operating system user, which presents you with a list of all the JVM Processes for you to
Connecting Remotely
Running JConsole remotely is a little more complex and certainly not so secure... meaning it may introduce
unacceptable risk for a production environment. Firstly, authentication is enabled by default so you will
either need to configure your username/passwords or could choose to defeat authentication entirely by
using the jserv.properties value below. The following change allows ANYONE to attach to your JVM
wrapper.bin.parameters=-Dcom.sun.management.jmxremote.authenticate=false
The second issue to overcome is that each JVM needs its own TCP port to accept connections. One
script and provide a port number range from which you allocate a tcp port number. The sample script
modification below is not perfect (as it doesn't resolve port number clashes) but may give you some ideas as
####
####
mJMXPORTMIN=9600
mJXMPORTMAX=9650
Page: 13
mRANDOM=$(((($RANDOM*${mNUMPORTS})/32767)+1))
fi
After starting your JVMs you then need to identify the TCP port they are listening on. You can do this using
You can then startup JConsole from your PC (or any other machine) and using the "Remote" tab you enter
the machine name and port you wish to connect to and also the username/password if appropriate. You
then get the same monitoring capabilities as if you had connected locally.
Conclusion
The real time graphical monitoring provided by JConsole, along with its ease of implementation and use,
provides a useful tool to help you understand the way your JVMs are being utilised. This knowledge can
You can use load-balancing routers (LBRs) to protect your E-Business Suite from similar types of system
failures. Load-balancers increase your environment's fault-tolerance and scalability by distributing load
across a pool of application servers like this:
Page: 14
Besides fault-tolerance and scalability, another appealing benefit is that you can use load-balancing to
substitute expensive SMP boxes with clusters of inexpensive Linux-based commodity servers.
• HTTP Layer
• DNS-based
• Apache Jserv Layer
• Forms Metric Server
• Concurrent Processing Layer
• Database Layer via Real Application Clusters
HTTP Layer load-balancing is the most common method used in E-Business Suite environments.
Page: 15
In this configuration, end-users navigate to a specific Web Entry Point that represents your E-Business
Suite's domain name. An HTTP Layer load-balancer routes all subsequent traffic for a specific user to a
specific Web Node.
HTTP Layer load-balancers may use heartbeat checks for node death detection and restart, and
sophisticated algorithms for load-balancing.
DNS-Based Load-Balancing
When an end-user's browser attempts to access your E-Business Suite environment, your local Domain
Name Server (DNS) can direct that user to a specific application server in a pool based on available
capacity:
Traffic for that user's session will be handled by the application server 10.10.10.10, while other users' traffic
Page: 16
may be directed to other application servers in the pool. Like HTTP layer load-balancers, many DNS-based
load-balancers use heartbeat checks against nodes and sophisticated algorithms for load-balancing.
Our larger enterprise-class customers combine DNS-based and HTTP layer load-balancers to support their
business continuity plans. In the event of a disaster, end-users are directed via a DNS-based load-balancer
from the primary E-Business Suite environment to an offsite standby site.
The minimum requirement is that a load-balancer support session persistence, where a client's initial HTTP
connection is directed to a particular application server, then subsequent HTTP requests from that client are
directed to the same server. As long as a load-balancer is able to handle session persistence (also referred
to as "stickiness"), it's likely to work with the E-Business Suite.
Page: 17