Oracle Application Framework Troubleshooting Guide: Release 11i To Release 12.2

Oracle Application Framework
Troubleshooting Guide
Release 11i to Release 12.2
September 2016
Table of Contents
PART 1: OVERVIEW ........................................................................................................................1
General Health Checks for Any OA Framework Issue .............................................................. 2
PART 2: INTRODUCTION TO THE OA FRAMEWORK ARCHITECTURE ................................................... 5
Section 1: The Model ................................................................................................................ 5
Section 2: The View .................................................................................................................. 5
Section 3: The Controller ........................................................................................................... 5
Section 4: Three-tier Architecture .............................................................................................. 6
Section 5: A Graphical Look at the Architecture ........................................................................ 7
PART 3: DIAGNOSING THE CLIENT OR "PRESENTATION LAYER"....................................................... 9
Section 1: HTTP Traffic and Cookies ........................................................................................ 9
Section 2: REST Service Interface .......................................................................................... 16
Section 3: JavaScript Errors .................................................................................................... 18
Section 4: Presentation Layer Troubleshooting Checklist ....................................................... 19
Section 5: User Interface Issues ............................................................................................. 20
PART 4: TROUBLESHOOTING THE MIDDLEWARE ............................................................................ 31
Section 1: Troubleshooting the Controller ............................................................................... 31
Section 2: Troubleshooting the Model ..................................................................................... 37
Options to Debug the Model ................................................................................................ 38
On-Screen Logging .......................................................................................................... 38
BC4J Logging................................................................................................................... 40
Verifying Profile Options................................................................................................... 42
Verifying Database Connectivity ...................................................................................... 42
iii
Table of Contents
Analysis of BC4J Objects ................................................................................................. 43
Section 3: Diagnosing Personalizations .................................................................................. 44
Considerations .....................................................................................................................45
Verifying Personalizations ................................................................................................... 45
What Can Go Wrong With Personalizations ........................................................................ 48
Section 4: Middle-tier Performance and Scalability Diagnostics ............................................. 50
Common Performance Issues ............................................................................................. 50
Response Time ................................................................................................................ 51
Heap Exhaustion and Memory Leaks .............................................................................. 52
Thread Dumps and Deadlocks......................................................................................... 60
Section 5: Troubleshooting the Java Object Cache and the Caching Framework .................. 64
Diagnosing the Java Caching Infrastructure ........................................................................ 65
Cache Loaders Failing to Load Data................................................................................ 66
Cache Member Communication Problems ...................................................................... 66
Additional Troubleshooting Tools ..................................................................................... 68
Clearing the Cache ....................................................................................................... 68
Available Logging Facilities .......................................................................................... 68
CacheWatchUtil ............................................................................................................ 70
Section 6: Middleware Troubleshooting Checklists ................................................................. 77
Exceptions Checklist............................................................................................................ 77
Personalizations Checklist ................................................................................................... 81
Authentication, Timeout or HTML Error Code Checklist ...................................................... 82
Performance Checklist......................................................................................................... 84
Appendix A: Making the Most of BC4J Logging ...................................................................... 85
iv
Table of Contents
Appendix B: Standard output/Standard error files in the containers........................................ 89
Appendix C: Deploying debug classes, jar files and .jsps ....................................................... 91
Release 11i ..........................................................................................................................91
Release 12.0 and 12.1......................................................................................................... 92
Release 12.2 and Above ..................................................................................................... 93
Common to All Releases/Technologies ............................................................................... 94
Appendix D: Oracle E-Business Suite Control Scripts ............................................................ 95
Resources ............................................................................................................................... 97
Listing of Visual Elements ....................................................................................................... 99
Change Log ........................................................................................................................... 100
v
Oracle Application Framework Troubleshooting
Guide
The most current version of this document can be obtained in My Oracle Support (MOS)
Knowledge <Document 1496055.1>.
There is a change log at the end of this document.
Part 1: Overview
Oracle E-Business Suite is an Enterprise Resource Planning/Customer Relationship
Management (ERP/CRM) suite of incredible complexity and includes many different
technologies, of which Oracle Application Framework is the foundation for HTML-based
modules within the suite.
Although we subject OA Framework to world class quality control, given the configurability and
adaptability of Oracle E-Business Suite, OA Framework gets exposed to endless possible
permutations of configuration, hardware and use cases that can cause errors. This document
helps to resolve those cases when feasible, as well as provide proper guidance on applying
successful diagnostic techniques to quickly resolve errors that might surface during regular
operations.
This document focuses on OA Framework-based applications and related technologies such as

J2EE container, BC4J, UIX, Java, JavaScript, and many others that are involved in the proper
functioning of HTML-based applications built with Oracle Application Framework (OAF).
Although this document provides an overview of some of these different technologies, it is
important that you are very familiar with them and we therefore encourage you to learn more
about these topics using external sources.
This document uses certain scenarios to illustrate examples of where to apply different
troubleshooting techniques, but given the number of possible permutations of configuration, use
cases, and so on, these examples are by no means a complete list of potential problems.
Additionally, there are inherent differences to the scenarios when you run into issues during
design time using the JDeveloper IDE, versus during application deployment in a test or
production environment. This document makes that distinction whenever possible.
Unless otherwise noted, all of these examples are applicable across Releases 11i, 12.0, 12.1
and 12.2 of OA Framework.
Good companions to this publication are the Oracle Application Framework Developer’s Guide,
MOS Knowledge Document 1315485.1, Oracle Application Framework Personalization Guide,
available as part of the Online Documentation Library on the Oracle Technology Network, and
the specific Release Notes for the OA Framework release that correspond to your installation,
available on My Oracle Support.
Oracle Corporation welcomes your comments and suggestions on the quality and usefulness of
this guide. Your input is an important part of the information used for revisions.
1
Oracle Application Framework Troubleshooting Guide
General Health Checks for Any OA Framework Issue
If you encounter an OA Framework issue, the following list of basic health checks will assist
Oracle Support in troubleshooting the problem.
Health Check 1: Are you using the latest certified components for Oracle E-Business
Suite?
1. First verify the version of components you are using by selecting the About this Page
link that appears at the bottom of each OA Framework page (the link appears after you
set the "FND: Diagnostics" profile option to "Y", preferably at the User level). The
Technology Components tab displays the relevant information or you may select the
Generate Bug Report button on the "About" page to output this information and send it
to Oracle Support.
2. If you are not running a supported version of Oracle E-Business Suite, you should
upgrade to a supported version and verify if the issue reproduces on the supported
version.
For Release 11i: Refer to the E-Business Suite 11.5.10 Minimum Patch Level
and Extended Support Information Center, MOS Knowledge Document
1199724.1.
For Release 12.0: ATG RUP6 + Latest CPUs (12.0.x is currently desupported.)
For Release 12.1: ATG RUP3 + Latest CPUs for OA Framework. Note the
Release 12.1 minimum prerequisites:
• By July 1, 2011: You must apply the Applications Technology Release

Update Pack (RUP) 12.1.2 (R12.ATG_PF.B.Delta.2, MOS Knowledge
Document 845809.1)
• By Feb. 1, 2012: You must apply the Applications Technology Release
Update Pack (RUP) 12.1.3 (R12.ATG_PF.B.Delta.3, MOS Knowledge
Document 1066312.1)
• By Feb. 1, 2013: You must apply the suite-wide E-Business Suite 12.1.3
Release Update Pack
Refer to EBS 12.0 Minimum Baseline Required by Feb 2012 for Extended
Support for more information.
3. Refer to Script to find Apache, Java, JRE, Forms version for Oracle E-Business Suite
R12, MOS Knowledge Document 468311.1, to identify the iAS and Java versions you
are using. Send the results of the script to Oracle Support, so Oracle can review the
techstack versions you are using.
4. Ensure you are on the Application Server 10g (10.1.3.5) and Java 1.6. Refer to the
following MOS Knowledge documents for more information:
o Document 454811.1, Upgrading to the Latest OracleAS 10g 10.1.3.x Patch Set in
Oracle E-Business Suite Release 12
o Document 455492.1, Using Latest Update of Java 6.0 with Oracle E-Business
Suite Release 12
2
o Document 418664.1, Overview of Using Java with Oracle E-Business Suite

Release 12
Health Check 2: Do you have customizations that are not working correctly?
1. Post your issue on the "Technology - OA Framework" Discussion Forum on the Oracle
Technology Network (OTN).
o Use the forum to post questions and exchange information with other customers
working on custom extensions in the OTN community. The OA Framework
development team and Oracle Support monitors and participates in the
discussion threads of this forum.
o Additionally, consider participating in the OTN "JDeveloper and ADF" forum for
usage questions concerning Oracle9i JDeveloper.
2. Please note that Oracle's support policy does not include assistance in creating,
deploying or troubleshooting custom code within Oracle E-Business Suite.
Customizations that involve either the modification of existing code or the calling of API,
JSP, or JDeveloper Extensions/custom code within Oracle E-Business Suite may affect
the integrity of the product. Direct all efforts to write, troubleshoot or debug custom code,
setup, or make recommendations that may affect specific business needs to Oracle
Consulting. Refer to Oracle Application Framework Support Guidelines for Customers
Release 12, Document 395441.1, for additional information about Oracle's support
policy.
3. For help diagnosing issues with personalizations created from the OA Personalization
Framework UI, refer to Part 4: Troubleshooting the Middleware > Section 3: Diagnosing
Personalizations.
3
4
Part 2: Introduction to the OA Framework Architecture

OA Framework is based on the model-view-controller (MVC) design pattern. There are three
independent parts that are interconnected to form the framework:
• The Model, which encapsulates all logic pertaining to the data model of your application.
• The View, which encapsulates logic pertaining to the rendering of data as viewed by the
user.
• The Controller, which contains the logic that responds to a user's interactions with the
View.
The sections that follow show the different technology components that are reused and
extended in OA Framework for each of the MVC layers:
• Section 1: The Model

• Section 2: The View
• Section 3: The Controller
• Section 4: Three-tier Architecture
• Section 5: A Graphical Look at the Architecture
Section 1: The Model
OA Framework uses Oracle Application Development Framework (ADF) Business Components

(BC), formerly known as BC4J: Business Components for Java, or JBO: Java Business Objects,
for data model interaction. ADF BC creates components such as Entity Objects (EO), View
Objects (VO) and Application Modules (AM), that map to the objects pertinent to your data
model. For example, a TABLE object maps to an EO, which ensures object persistence, row
locking and other transaction management capabilities. A database query maps to a VO, which
is a container of the rows and the cursor pointer. All of these objects, in turn, are encapsulated
in an AM Java object. An AM is a container of all objects required to perform a transaction over
your tables and is also a container for the database connection.
Section 2: The View
OA Framework relies on Oracle UIX (User Interface XML) as a provider of HTML widgets (also
known as web beans) with data binding capabilities. OA Framework has also extended the base
functionality of UIX to accommodate Oracle E-Business Suite-specific use cases and security.
OA Framework saves all designed pages and regions as XML documents. The JDeveloper
integrated development environment (IDE) reads these XML documents from the file system.
When the pages and regions are ready for deployment, you import and store the XML
documents in the Meta Data System (MDS) repository of the database.
Section 3: The Controller
OA Framework provides an interface, OAController.java, and a class,

OAControllerImpl.java, as a starting point for any controller. When you write a controller,
you should subclass OAControllerImpl.java and write its logic in the resulting Java class.
The method processRequest or processFormRequest is invoked whenever you request a
5
page to the server. The serving thread executes the logic defined in one of these methods,
depending on whether the request is an HTTP POST (processFormRequest) or an HTTP
GET (processRequest). The method processRequest always executes the first time a
page renders. The method processFormRequest always executes when the user interacts
with the page, such as when a user selects the 'Submit' button to submit a transaction.
Section 4: Three-tier Architecture
OA Framework runs on a three-tier architecture.
The first tier, known as the client or “presentation tier”, uses a web browser installed on a client
PC or device. This tier reacts to user interactions to make requests to the server (second tier).
Additionally, some low-level logic executes in the client via UIX- or OA Framework-provided
JavaScript libraries.
The second tier or server, also known as the “application tier” or “middle tier”, is a complex
system where multiple technologies interact. The middle tier consists of an application server,
Oracle Fusion Middleware (FMW), that includes:
• An HTTP server
• A J2EE container -
o For Release 11i: Apache JServ
o For Release 12.0 and 12.1: Oracle Containers For Java (OC4J)
o For Release 12.2: Oracle WebLogic Server (WLS)
• Related interconnection technologies -
o mod_jserv for communication between Apache and JServ
o mod_oc4j for communication between Apache and OC4J
o mod_weblogic for communication between Apache and WebLogic
• JDBC drivers for database connectivity
All of OA Framework and its components run within the boundaries of the J2EE container. The
majority of issues that customers encounter occur on this middle tier, and as such, will be the
focus of this document. It is very important for you to be familiar with middle tier concepts so you
can successfully diagnose OA Framework issues.
The complexity of this tier may also increase exponentially when horizontal scalability is
factored into the deployment. Such a deployment introduces more complexity by enhancing the
network topology through the inclusion of hardware load balancers, multiple Application servers
and additional technology components such as Single Sign-On and Oracle Web Cache. When
you contact Oracle Support or Oracle E-Business Suite Development for assistance, please
indicate in great detail, the layout of your network topology and any third-party products that
interact with Oracle E-Business Suite in your installation. If your installation fits this scenario, we
will ask you to remove these additional network components whenever possible and test
whether the problem reproduces. If the problem persists, we will direct you to contact the
vendors of these components for additional assistance.
The third tier is the database tier. OA Framework applications run on the middle tier and use the
database as:
6
• A transactional data store.

• A page/region metadata repository via the MDS.
• A means of communication between different cache members from the distributed Java
Object Cache that is part of Oracle E-Business Suite that leverages the RDBMS server's
Advance Queuing technology through the Business Event System.
Section 5: A Graphical Look at the Architecture
The following graphics illustrate some of the concepts reviewed in this introduction:
Figure 1. Release 12.1 Three-tier logical architecture
Figure 2. Release 12.2 Three-tier logical architecture
7
Figure 3. OA Framework Component Stack
8
Part 3: Diagnosing the Client or "Presentation Layer"

To properly solve a problem, you need to do a thorough characterization of the issue and
identify the layer where the error occurred. For OA Framework, most of the interesting activity
occurs on the middle tier. However there are certain errors that can occur on the client:
• Presentation issues - For example, fonts do not look right or fonts do not match those
on the rest of the page.
• Mishandling of cookies - Since the HTTP protocol is a stateless protocol, it uses
cookies as a mechanism to simulate “session” persistence. This is very important but is
often overlooked during diagnostic exercises. If your client is incapable of handling
cookies properly and does not reflect session cookies back to the server, you may find
yourself encountering a broad range of errors. Errors may also occur when a user opens
multiple browser windows under the same process ID, such as opening multiple tabs or
multiple windows with the same applications session. This is a known limitation and is
not explicitly supported, as it causes cookies to overwrite each other. Proper education
can help users avoid such situations, which may lead to data corruption as well as
undetectable and unpredictable errors whose causes may be difficult to identify.
• Use of unsupported clients - Make sure you use the latest certified browsers listed in
the Recommended Browsers for Oracle E-Business Suite Release 12, MOS Knowledge
Document 389422.1.
• Errors with other clients - Other clients include diagnostics of REST requests, OA
Framework portlet provider, and so on.
• JavaScript errors - Be aware that some browsers do not report JavaScript errors in a
visible fashion and require you to be more alert to changes on the page or browser itself.
For example, Microsoft Internet Explorer (IE) version 6.0 shows a small Attention icon
on the browser's status bar when a JavaScript error occurs. To obtain relevant
information about the error, you must double-click on the Attention icon to display the
additional information.
The sections below discuss these potential problems in more detail:
• Section 1: HTTP Traffic and Cookies

• Section 2: REST Service Interface
• Section 3: JavaScript Errors
• Section 4: Presentation Layer Troubleshooting Checklist
• Section 5: User Interface Issues
Section 1: HTTP Traffic and Cookies
HTML pages generated by OA Framework make extensive use of cascading style sheets to
comply with well defined UI standards. These style sheets, mostly generated dynamically at
runtime, are based on a number of factors such as the Look-and-Feel and the version of OA
Framework and Oracle UIX in use. To avoid performance issues, these dynamically generated
style sheets are stored on the file system in a cache, under $OA_HTML/cabo/styles/cache.
We recommend you delete the contents of this directory every time you apply a new OA
Framework patch to your system to ensure that these files get regenerated using the latest,
recently patched class files from OA Framework and Oracle UIX.
9
It is also important to consider HTTP traffic when you encounter problems related to style
sheets. For example, the client may send a request to the server for a CSS or an XSS file but
the file cannot be found. To verify if HTTP traffic is the culprit in this case, you need to use an
HTTP traffic analyzer tool to examine what is happening to the request. Is the request being
cached locally in your browser or is the server unable to find the file and responds with a
different HTTP response code other than 200?
Example
The following example illustrates how to analyze HTTP traffic based on the rendering cycle of
the Login page.
Note: While this example uses the freeware Fiddler as the HTTP traffic analyzer
tool, it is not a recommendation. You may use any tool as long as it is capable of:
1. Reading request and response headers and bodies.

2. Analyzing HTTP over SSL, in case your installation is configured to use such a
set up.
Figure 4. Fiddler, after launching
1. First stop capturing traffic. Since this application is designed to capture all HTTP traffic
from all applications, the output may become confusing. Navigate to File > Capture
Traffic and uncheck this option.
2. Clear all sessions and leave all the sessions blank by clicking on the icon from the
toolbar and selecting Remove All from the context menu. You may also select the
following from the menu: Edit > Remove > All Sessions.
10
3. Finally, filter traffic specific to your server by selecting the Filters tab. Check the Use
Filters checkbox and enter the host name of your middle tier server or entry point as
shown in Figure 5.
Figure 5. Using Fiddler to filter traffic specific to your server
6. Open your browser window and select the process you want to analyze by clicking on
the Any Process selector icon and dragging the mouse to the browser window you
selected as shown in Figure 6.
Figure 6. Selecting the browser process for Fiddler
11
7. Once you select the process, you may start your tests. Signal Fiddler to start capturing
traffic by selecting the menu option File > Capture Traffic or by pressing the F12 key
when the focus is on the Fiddler window.
8. In the browser window, enter the URL to navigation to your Oracle E-Business Suite
environment and perform the steps that reproduce the problem. In this example, the
Login screen renders with style sheet issues as shown in Figure 7.
9. Once the problem reproduces, analyze the data captured by Fiddler. In Fiddler, switch to
the Inspectors tab and for both the request and response headers, select Raw as shown
in Figure 8. If Fiddler displays the message “Response is encoded and may need to be
decoded before inspection. Click here to transform”, in a text box with a yellow
background, go ahead and click it.
Note: To learn more about Fiddler, refer to the documentation available at

http://www.fiddler2.com/.
Figure 7. Login screen rendering with style sheet issues
12
Figure 8. Data captured by Fiddler
10. Examine the information captured by Fiddler as shown in Figure 8. In reviewing the Web
Sessions view, you see four HTTP requests due to the logic of the login process even
though the browser only points to the AppsLogin servlet. The request to the AppsLogin
servlet initiates a check on the settings in your environment. Depending on whether or
not you implemented Oracle Single Sign-on, the request is redirected either to the SSO
login page or to the AppsLocalLogin JSP, as shown by the redirect status code 302
(FOUND). If standard Oracle E-Business Security is implemented (non-SSO), the
request is then redirected to fndvald.jsp to verify and validate the credentials and to
create the user’s session if the username and password challenge passes.
13
11. Later, the request redirects to RF.jsp with a function code so that the Login page can
render within the context of an executed function to maintain security. Once the RF.jsp
redirect renders, which is when the page's HTML gets sent to the browser, the server
responds with status code 200.
Note: For more information, please check the status codes table published by
the W3C consortium at http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html.
12. You should also check for the following at this diagnostic stage:
• Missing objects - Note that a request sent to /OA_HTML/test.jsp is flagged in red in

Figure 8. In this example, the request is to a non-existent page, so the server responds
with a 404 status code (NOT FOUND). When you capture HTTP traffic, always check
for requests flagged in red.
• Cookies - Analyze every request to ensure that it contains at least 2 special cookies:
JSESSIONID and the Oracle E-Business Suite session cookie. To determine the
name of the Oracle E-Business Suite session cookie, perform the following query
against your database:
SELECT FND_WEB_CONFIG.DATABASE_ID FROM DUAL;
Check for the presence of this cookie on all requests to the server.
Figure 9. Detail of the request header showing the Cookie header
Figure 10. Detail of cookies on "Headers" view where cookies are shown individually
14
Figure 10 shows both the JSESSIONID cookie (used by the container to keep
track of the HTTP session) and the cookie named oraoracle, which is the
Oracle E-Business Suite session cookie in this example.
Cookies are paramount to secured HTML applications as they identify who you
are and your associated privileges. HTTP servers only understand requests and
responses and are not able to maintain a “stateful” session. By using cookies, we
can simulate a stateful session within a browser. However, if your instance is not
properly configured, cookies will be wrong or non-existent and will cause
unpredictable consequences.
Cookies may also get overwritten. One cause of cookie overwriting is using
multiple browser windows (or tabs) against the same Oracle E-Business
Suite session. This is not supported and leads to unpredictable behavior. If
you need multiple browser windows, ensure that you launch separate processes
in each or use one browser for one session and a different browser for the other.
• Session timeouts - Make sure session timeouts are properly set. Check the profile
option ICX:Session Timeout / ICX_SESSION_TIMEOUT, which governs the "inactivity
timeout" for Oracle E-Business Suite sessions. The value is expressed in minutes. A
value between 120 to 180 should be enough.
Check the inactivity timeout of the HTTP session. In the case of the OC4J
container, check the file:
$ORA_CONFIG_HOME/10.1.3/oacore/application-
deployments/oacore/html/orion-web.xml.
15
Make sure the following entry is in the file:
<session-timeout>30</session-timeout>
This setting signals the container to abandon sessions that stop showing activity
by the amount of minutes specified by the 'session-timeout' parameter. As a
result, all objects allocated in memory that pertain to those sessions are
discarded. This setting is important because it has a direct impact on
performance via resource utilization. The more time set before session timeout,
the bigger the memory or heap space the container has to allocate to these
objects for that extra time. Thirty minutes is an appropriate value for this setting.
Section 2: REST Service Interface
Although the REST interface uses HTTP to communicate between the client and the server, you
should also analyze the traffic between the two to verify that the XML server response is in the
expected format. The REST provider is basically an HTTP Servlet whose request and response
includes XML content in the body.
One caveat with the REST service is that if you implement SSL over HTTP (HTTPS) with self-
signed certificates, the client may not offer visual feedback if handshake errors are
encountered. Make sure that a potential client always first accepts the certificate in the
certificate store as a trusted certificate. Also check the documentation of your advanced
topologies to confirm that your configuration is certified and supported by the Oracle E-Business
Suite Technology Stack team.
Refer to the OA Framework Developer’s Guide, Document 1315485.1, for more information on
how to implement the REST service.
The following section provides an overview of the REST service architecture with useful
troubleshooting suggestions. In the example shown in Figure 11, the client application (in this
case, the Home page) sends a REST request with the parameters in XML format and the server
responds with the results of the request.
Figure 11. REST request and response generated in the home page (menu tree)
16
The challenge with a REST request is that it can come from almost any client, such as a mobile
client or a third party application, etc.
Figure 12. Architecture of REST service interface
17
To troubleshoot this layer, you may want to focus on the following:
1. The request - Make sure the session cookies (for both Oracle E-Business Suite and
HTTP session) are present on the request.
2. The format of the response - Make sure the payload returned from the server contains
the expected data that the client can parse and understand.
3. How the client application makes use of the payload.
For 1 and 2, perform the diagnostics via a HTTP traffic analyzer or a debugging proxy. For 3,
how to perform the diagnostics will depend on the client application itself.
Section 3: JavaScript Errors
The reporting of JavaScript errors varies greatly depending on the browser you are using, the
browser version, and the kind of plug-ins you installed in your browser.
Some browsers even come with JavaScript debuggers that allow you step through the code to
capture additional information that may be invaluable for reporting a product defect to
development. For example, Microsoft Internet Explorer 8 and above provides you with a set of
debugging tools when you press the F11 key, whereas in Microsoft Internet Explorer 7, you can
enable the JavaScript debugger by performing the following steps:
1. From the View menu, select Script Debugger > Open.

2. Select View > Script Debugger > Break on Next Statement at the moment of
reproducing the error.
3. Once you capture the error, write down the page that caused the error, the JavaScript
(.js) file that shows the error, and the line number when the error occurs.
Refer to your browser documentation to learn more about the JavaScript debugging facilities
available to your specific browser.
JavaScript logic, in general, is highly dependent on the underlying UIX technology. It is

important to know the version of UIX in your instance and whether the problem began after a
18
patch was applied, or an upgrade was performed. We recommend that you keep current on all
OA Framework-delivered patches that update technology components. To find the version of
UIX in your instance, refer to the “About this page” functionality described later in this document.
Avoid modifying the .js files yourself. Always report your problem to the product development
team that owns the page showing the undesired behavior.
Figure 13. Microsoft Script Debugger showing a breakpoint
While JavaScript errors are rare, when they do occur, they usually surface in the following
scenarios:
• In a page that uses Partial Page Refresh (PPR) where a region of the page may be
refreshed based on a user event (such as a user pressing a key or tabbing out of a
field).
• In a page that uses rich clients within a Rich Content Container web bean.
Refer to the OA Framework Developer’s Guide, Document 1315485.1, for more information
about the PPR and Rich Content Container features.
Section 4: Presentation Layer Troubleshooting Checklist
19
If a page does not render correctly or you encounter a Javascript issue, review the following
checklist to troubleshoot the issue.
1. Identify the navigation path for the page in question and capture a screenshot of the
page.
2. For Release 12, refer to the Font And Links Have Changed After Patching, MOS
Knowledge Document 1348791.1. This document identifies known presentation layer
issues that may occur after applying an OA Framework patch and provides a work-
around for the issues.
3. Are you using a Custom Look and Feel (CLAF) created by the Customizing Look and
Feel Administrator responsibility? Does the problem reproduce if you do not use the
custom skin (CLAF)?
Refer to the following MOS Knowledge documents when creating a Custom Look and
Feel:
2.
o Tips For Personalizing The E-Business Suite R12 Login Page (MainLoginPG),
Document 741459.1.
o How To Change Look And Feel and Colors Of Oracle Applications 11i and R12,
Document 210670.1.
For CLAF issues, review and refer the customer to the OA Framework
Personalization Guide, which they can download from
http://www.oracle.com/technetwork/documentation/applications-167706.html.
4. Are you using a browser listed in the Recommended Browsers for Oracle E-Business
Suite Release 12, MOS Knowledge Document 389422.1?
5. If you have a Javascript error or a non-functioning LOV, you may also find using the
Microsoft Internet Explorer Developer Toolbar helpful in diagnosing the problem. Be sure
to enable "Break on Error".
Section 5: User Interface Issues
This section lists known issues identified for the user interface.
• Attachments
• Infotile
• Hide / Show Subtab Layout
• Home Page
• Release 12.2.5 Alta Look-and-Feel
• Rich Table Interactions
• SubTab Navigation
• Tabs / Navigation
Release 12.2.5 Alta Look-and-Feel
Issue Cause Solution

1. OA Framework The “Oracle Applications • Verify the value of “Oracle Applications
20
pages do not Look and Feel” profile is Look and Feel” profile.
render in Alta explicitly set to a different • This profile needs to be set to “Alta
Look-and-Feel. Look and Feel. Look and Feel” or left blank for Alta
Look and Feel to take effect.
2. OA Framework • The browser cache • Clear the browser cache.

page styles look has reference to old • If clearing the browser cache does not
corrupted. CSS files. resolve the issue, perform the following
• The styles cache steps on the RUN file edition:
folder on the server o Stop oacore container.
has references to o Delete / rename
old CSS files (when $OA_HTML/cabo/styles/ca
OA Framework che folder.
patches are applied o Start oacore container and
in hot patch mode). verify the page.
3. Custom skins The custom skin is not Custom skin needs to be reviewed for
are not taking compliant with the seeded compliance with the Alta Skin and rewritten if
effect. Alta skin. necessary.
4. Some The custom skin is not Custom skin needs to be reviewed for
components do compliant with the seeded compliance with Alta skin implementation for
not render Alta skin for the affected the affected components and rewritten if
properly while components. necessary.
rendering OA
Framework
pages in a
custom skin.
5. The following A possible cause is that the The page structure needs to be reviewed to
error is seen page structure is not HTM5- ensure that the components follow the
while performing compliant. prescribed structure. Please review the topic
form submit “Page Layout (How to place content)” in
actions on an OA "Chapter 4: Implementing Specific UI
Framework page: Features" of the Oracle Application Framework
Developer’s Guide, MOS Document
“The page you 1315485.1, for detailed information on this
are trying to topic.
access is no
longer
active”
6. Page-level As per the Alta This is the expected behavior.
buttons are not specifications, page-level
repeated at the buttons are not repeated at
bottom. the bottom of the page.
7. The width of The icon of the LOV or Date This is the expected behavior as per the Alta
LOV and Date Picker field now renders Look-and-Feel specifications.
Picker fields are inside the text input field as
slightly larger part of the Alta Look-and-
than other Feel. In older Look-and-
21
message Feels, the icons for LOV

components. and Date Picker rendered
outside the text input field.
8. Alignment of The page structure is The page structure needs to be reviewed to
page layout incorrect and the correct ensure that the components follow the
contents is not layout styles are not used. prescribed structure. Please review the topic
correct. “Page Layout (How to place content)” in
"Chapter 4: Implementing Specific UI
Features" of the Oracle Application Framework
1315485.1, for detailed information on this
topic.
Tabs / Navigation

1. OA Framework Profile "FND: Top Menu Display • Administrator can change
pages still show the Type” is set to Links only. profile at the preferred levels.
Release 12.2.4-style • End users can navigate to
menu display. preferences and verify that
the Top-level Menu Display
Style is set to Icons and
Links.
2. OA Framework No Icon has been defined for the To replace a default icon with a
pages shows only function / menu. specific icon, please review the topic
default icons. “Tabs / Navigation” in "Chapter 4:
Implementing Specific UI Features"
of the Oracle Application Framework
1315485.1, for detailed information
on this topic.
3. A broken image is The image URI in the menu / Verify that the image URI mentioned
rendered for the function definition is incorrect. in the menu definition is present
menu item. under the $OA_MEDIA directory.
Rich Table Interactions

1. The column freeze The table does not have a This is the expected behavior. The
icon is disabled for horizontal scrollbar. column freeze icon is enabled once a
the table. horizontal scrollbar is seen for the table
(as a result of data or resizing of
existing columns).
2. The column freeze The following are possible • This is the expected behavior
icon is not seen for causes: for the mentioned cases.
the table. • The accessibility mode is set to
22
• The table is created “Screen Reader Optimized”

programmatically. unintentionally.
• The table has columns o Administrator set the
added “Self Service
programmatically. Accessibility Features”
• The table is part of an profile to “None” or
OA Framework page “Standard
embedded within a JTT Accessibility” at the
page. desired level.
• The page is rendered in o End users can change
“Screen Reader the Accessibility value
Optimized” accessibility to “None” or “Standard
mode. Accessibility” through
the Preferences page.
3. The table does not The possible causes are: • This is the expected behavior
show the vertical for the mentioned cases.
scrollbar. • The page is rendered in • If the accessibility mode is set
“Screen Reader to “Screen Reader Optimized”
Optimized” accessibility unintentionally.
mode. o Administrator set the
• The table is inside the “Self Service
LOV modal window. Accessibility Features”
• The table is part of profile to “None” or
Attachment Table “Standard
component. Accessibility” at the
desired level.
o End users can change
the Accessibility value
to “None” or “Standard
Accessibility” through
Preferences page.
4. Table’s vertical Safari on IOS does not render This is the browser behavior. The
scrollbar is not the scroll bars by default. scrollbars are highlighted when user
visible on the iPad. taps anywhere within the table area.
5. User wants to The Row Navigation Administrators can personalize the
revert to the Release Policy property is set to table and set the Row Navigation
12.2.4 style of table SCROLL by default. Policy to PAGINATION at the
row navigation desired level to revert back to the
(show previous/ next Release 12.2.4 style of row navigation.
links)
6. Rich interactions, The page may contain an Rename any item ID that contains a
such as reordering incorrectly named table ID that suffix such as ___n to something
or resizing a table includes the suffix "___n" in it's unique. For example, change the ID
column, are not name. For such IDs, OA TableResults___1 to TableResults1.
saved after a user Framework removes the suffix
navigates out the "___n" from the webbean in
page. These memory. As a result, a
23
changes should be duplicate ID in the web bean

retained even after tree occurs and the correct web
the user navigates bean containing the rich
out of the page. interaction changes will not be
found.
Attachments

1. The inline Inline Attachment Ensure that the Inline
Attachment UI does Enabled property is set to false Attachment Enabled property is
not appear even if the at the attachment component enabled at the attachment
"FND: Disable Inline level. component level. Please review the
Attachments" profile “Attachments” topic under "Chapter
is set to False. 4: Implementing Specific UI
Features" of the Oracle Application
Framework Developer’s Guide, MOS
Document 1315485.1 for detailed
information on this topic.
2. In Release 12.2.6, Release 12.2.6 is certified only Ensure that you install or upgrade to
attachments are not with Oracle Autovue Release Oracle Autovue Release 20.2.3 and
printing even though 20.2.3. set the profile option "FND
the profile option Attachment AutoVue Server".
"FND Attachment If the Oracle E-Business Suite
AutoVue Server" is instance is running with the If the Oracle E-Business Suite
enabled to use the HTTPS protocol, then you must instance is running with the HTTPS
AutoVue Document also import the SSL certificate of protocol, then import the SSL
Print Service. the Oracle E-Business Suite certificate of the Oracle E-Business
instance into the AutoVue Suite instance into the AutoVue
server. server.
3. Release 12.1.3 and Long text attachments are not None
later updates to the supported in Release 12.1.3 and
12.1 codeline do not later updates to the 12.1
support long text codeline.
attachments. As a
result, a user can not
add, edit or view long
text attachments in
these releases of OA
Framework. In
addition, users will
not be able to
properly view long
text attachments that
were added to the
attachment entity
from the Oracle
Form-based UI.
24
Attachments Behavior
The following table illustrates Inline Attachments behavior for the different combination of values
set for the 'FND: Disable Inline Attachments' profile option and the Inline Attachment
Enabled attachment item property.
Value of Inline Value of 'FND: Behavior prior to Release Behavior in Release

Attachment Disable Inline 12.2.5 12.2.5 and above
Enabled property Attachments'
profile
default false 1. Mouse hover / tap - Mouse click / tap /
Inline Attachment focus + [Enter] - Inline
pop-up appears. Attachment pop-up
2. Mouse click / double- appears.
tap / focus + [Enter] -
Attachment Page No other interaction is
appears. supported.
default true Mouse click / double-tap / Mouse click / tap /

focus + [Enter] - Attachment focus + [Enter] -
Page appears. Attachment Page
appears.
No other interaction is
supported.
true false 1. Mouse hover / tap - Mouse click / tap /
Inline Attachment focus + [Enter] - Inline
pop-up appears. Attachment pop-up
2. Mouse click / double- appears.
appears. supported.
true true 1. Mouse hover / tap - Mouse click / tap /

Inline Attachment focus + [Enter] key -
pop-up appears. Inline Attachment
2. Mouse click / double- pop-up appears.
appears. supported.
false false Mouse click / double-tap / Mouse click / tap /

appears.
false true Mouse click / double-tap / Mouse click / tap /
25
appears.
You may also set these attachment region item properties to control the behavior of the Inline
Attachments feature:
• Update Allowed - set to true to enable the Update icon in the Inline View
Attachments pop-up window.
• Delete Allowed - set to true to enable the Delete icon in the Inline View Attachments
pop-up window.
Aligning Attachment Prompts in a messageComponentLayout
The following instructions describe how to place an attachmentLink or

messageInlineAttachment item style in a messageComponentLayout region so that it's
prompt aligns properly with respect to other elements of the layout region.
Step 1. Create a messageLayout region as a child of the messageComponentLayout region.
Step 2. In the messageLayout region, set the Prompt property value to Attachments or some
other value.
Step 3. Create an attachmentLink or a messageInlineAttachment item under the

messageLayout region.
Step 4. For proper horizontal alignment of the attachment item, set the Prompt property of the
attachmentLink or messageInlineAttachment item as null.
Step 5. By default, the messageLayout region 'top' aligns its prompt and the attachmentLink
and messageInlineAttachment items 'middle' align their prompts. The only way to resolve
these vertical alignment differences is by programmatically setting the vertical alignment of the
messageLayout region, as shown in the code example below.
OAMessageLayoutBean msgLayout =
(OAMessageLayoutBean)webBean.findIndexedChildRecursive("<MessageLayout
BeanID>");
msgLayout.setAttributeValue(V_ALIGN_ATTR, "middle");
Subtab Navigation

1. The following error An invalid display type is set for Please review the “Subtab
message is seen on the chosen orientation. For Navigation” topic under "Chapter 4:
the page containing a example - Icon Only is not a Implementing Specific UI Features"
26
subtab layout: valid display type in horizontal of the Oracle Application Framework
orientation. Developer’s Guide, MOS Document
"You have set an 1315485.1 for detailed information on
invalid Display this topic.
Type in
&ORIENTATION
orientation"
Infotile

1. Content added The content is larger than the tile Please review the tile content. Since
within a tile is not size so the content gets hidden the tiles are rendered with a fixed
visible or is only along tile edges. size, ensure that the content fits
partially visible within the tile area.
2.Selecting a tile • The contentRegionId • Ensure that the
does not display any property of the tile is not contentRegionId property
content. mapped to any region. of the tile is set to a valid
• Partial Page Rendering is region ID.
disabled. • Ensure that the profile "FND:
Disable Partial Page
Rendering"’ is not set to True.
3.Tile description The font size, color and The tile content needs to be explicitly
does not align alignment properties are missing styled based on the requirements.
properly within the or incorrect for the tile contents.
tiles.
Hide / Show Subtab Layout

1. The following error The Icon URI is missing for the The Icon URI property is
message is seen on Hide / Show subtab layout links. mandatory for the Hide / Show
the page containing a subtab layout definition. Ensure that a
Hide / Show subtab valid image URI is provided for the
layout: subtab links.
"You have defined

a link under sub
tab layout
&REGION_NAME
without
specifying an
icon on it. If
you set
Orientation to
vertical and
27
Display Type to
Icon Only, then
you must define
an icon for all
links under
subtab layout
region”
Home Page

1. Functions and Form functions do not run on the If the profile Self Service Personal
submenus are touch devices supported by Home Page Mode is set to
missing from a Oracle E-Business Suite. Framework Tree, the following
responsibility's behavior will be observed on the
submenu on the Oracle E-Business Suite Home page
Home page when for the touch device:
Oracle E-Business
Suite is run from a • If a responsibility's submenu
touch device. contains only form functions,
that submenu does not
render.
• if a responsibility's submenu
contains a secondary
submenu that contains only
form functions, that secondary
submenu does not render.
• If a responsibility contains
only form functions, a "No
pages found" message
displays when you select the
responsibility.
• Form functions set as
favorites do not render in the
Favorites drop down list on
the Universal Global Header.
2. Favorites are not Form functions do not run on the If the profile Self Service Personal
rendering from the touch devices supported by Home Page Mode is set to
Simple Home Page or Oracle E-Business Suite. Framework Simplified, the following
Universal Global behavior will be observed on the
Header when Oracle Oracle E-Business Suite Home page
E-Business Suite is for the touch device:
run from a touch
device. • Favorites icons representing
form functions do not render.
• Form functions set as
favorites do not render in the
Favorites drop down lists on
28
the Universal Global Header.
29
30
Part 4: Troubleshooting the Middleware

In the introduction, we discuss the architecture of OA Framework and how it is built using the
Model/View/Controller design pattern. These three layers run in Java in the middleware’s J2EE
container. It is very important that you have a certain level of knowledge about built-in
debugging facilities for both OA Framework and the container.
Additionally, existing knowledge of the Java language for diagnosing and troubleshooting
purposes, and an architectural understanding of J2EE applications will go a long way in helping
you understand the following sections:
• Section 1: Troubleshooting the Controller

o Java Exceptions
• Section 2: Troubleshooting the Model
o Options to Debug the Model
 On-Screen Logging
 BC4J Logging
 Verifying Profile Options
 Verifying Database Connectivity
 Analysis of BC4J Objects
• Section 3: Diagnosing Personalizations
o Considerations
o Verifying Personalizations
o What Can Go Wrong With Personalizations
• Section 4: Middle-tier Performance and Scalability Diagnostics
o Common Performance Issues
 Response Time
 Heap Exhaustion and Memory Leaks
 Thread Dumps and Deadlocks
• Section 5: Troubleshooting the Java Object Cache and the Caching Framework
o Diagnosing the Java Caching Infrastructure
o Cache Loaders Failing to Load Data
o Cache Members Communication Problems
o Additional Troubleshooting Tools
 Clearing the Cache
 Available Logging Facilities
 CacheWatchUtil
• Section 6: Middleware Troubleshooting Checklists
o Exceptions Checklist
o Personalizations Checklist
o Authentication, Timeout or HTML Error Code Checklist
o Performance Checklist
Section 1: Troubleshooting the Controller
The controller is a Java class that extends OAControllerImpl, which implements the
interface OAController.
31
When a user interacts with an OA Framework-based application, the browser sends a request
to the middle tier server which delegates logic to the controller. The interactions can be
classified as POST and GET, as defined by the HTTP protocol.
For example, when a user navigates to a certain page, the browser sends a GET request and
the entire rendering cycle of the page starts. Control is transferred to the controller at a certain
point in the rendering cycle and the logic of the controller contained in the method
processRequest()executes. Usually controllers on processRequest contain the logic to
programmatically get references to AMs, VOs, and so on.
Similarly, when a user fills out a web page with data and clicks on the Submit button to post
changes, the browser sends a POST request. The page’s rendering cycle, at some point,
transfers control to the method processFormRequest() in the controller.
It is important to distinguish between both methods as the logic branches, as described above.
Refer to the OA Framework Developer’s Guide, Document 1315485.1, to review the guidelines
and recommendations for controller implementation.
When a controller fails, it usually throws a Java exception with a stack trace that should be
examined. These exceptions are by no means generic as each controller is contained within its
own class. A page may have multiple controllers that correspond to multiple regions. When
reporting an issue to Oracle Support or Oracle Development, always provide the Java stack
trace in its entirety. Additionally, it is important you identify the version of the Java class that
contains the controller to make sure there isn't already a higher version that provides a fix for
the condition.
Figure 14. Determining a file's version using ADIDENT
All controllers deploy under the $JAVA_TOP directory. You can use the ADIDENT utility
provided by Oracle to identify the version of a controller. The syntax is:
adident Header <full path to file>
where:
• adident is the name of the utility, in lowercase.

• Header is the string that adident will search for within the file.
• The third parameter is the name of the file to search.
Please provide the resulting controller version when you report any controller issue to Oracle
Support. In order to identify all the controllers associated with a page or region, please refer to
the "About this page" link at the bottom of the page. The "About this page" functionality renders
32
automatically for all OA Framework pages if the Diagnostics feature is enabled using the FND:
Diagnostics / FND_DIAGNOSTICS profile option.
Java Exceptions
The handling of Java exceptions is a topic of itself. The amount of information provided by a
Java stack trace may seem overwhelming to an untrained eye but to the developer who owns
the controller code, it, along with the version information of the controller, can help pinpoint the
exact location in the logic that triggers the unexpected condition.
Chances are, if you run into an error with a faulty controller, it will produce a Java exception.
However, If you configure your environment with tight security by setting the profile option FND:
Diagnostics to No, then no exception will be reported on the user interface. Only a System
error message is shown, directing you to contact your system administrator.
If you set the FND: Diagnostics profile option to Yes, then when a Java exception occurs, you
will get a link on the page that directs you to more error details.
The following is an example of a stack trace thrown by the PreferencesPageCO.java, the

controller of the Preferences page:
Error Page
Exception Details.
oracle.apps.fnd.framework.OAException: java.lang.NullPointerException
at
oracle.apps.fnd.framework.OAException.wrapperException(OAException.jav
a:896)
at
oracle.apps.fnd.framework.webui.OAWebBeanHelper.processRequest(OAWebBe
anHelper.java:616)
at
oracle.apps.fnd.framework.webui.OAWebBeanContainerHelper.processReques
t(OAWebBeanContainerHelper.java:251)
at
oracle.apps.fnd.framework.webui.OAPageLayoutHelper.processRequest(OAPa
geLayoutHelper.java:1166)
at
oracle.apps.fnd.framework.webui.beans.layout.OAPageLayoutBean.processR
equest(OAPageLayoutBean.java:1569)
at
oracle.apps.fnd.framework.webui.OAWebBeanHelper.processRequestChildren
(OAWebBeanHelper.java:968)
at
at
anHelper.java:659)
at
33
at
oracle.apps.fnd.framework.webui.beans.form.OAFormBean.processRequest(O
AFormBean.java:385)
at
at
at
anHelper.java:659)
at
at
oracle.apps.fnd.framework.webui.beans.OABodyBean.processRequest(OABody
Bean.java:353)
at
oracle.apps.fnd.framework.webui.OAPageBean.processRequest(OAPageBean.j
ava:2513)
at
oracle.apps.fnd.framework.webui.OAPageBean.preparePage(OAPageBean.java
:1894)
at
:538)
at
:426)
at _OA._jspService(_OA.java:212)
at
com.orionserver.http.OrionHttpJspPage.service(OrionHttpJspPage.java:59
)
at oracle.jsp.runtimev2.JspPageTable.service(JspPageTable.java:379)
at
oracle.jsp.runtimev2.JspServlet.internalService(JspServlet.java:594)
at oracle.jsp.runtimev2.JspServlet.service(JspServlet.java:518)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:856)
at
com.evermind.server.http.ResourceFilterChain.doFilter(ResourceFilterCh
ain.java:64)
at
oracle.apps.jtf.base.session.ReleaseResFilter.doFilter(ReleaseResFilte
r.java:26)
at
com.evermind.server.http.EvermindFilterChain.doFilter(EvermindFilterCh
ain.java:15)
at
oracle.apps.fnd.security.AppsServletFilter.doFilter(AppsServletFilter.
java:318)
34
at
com.evermind.server.http.ServletRequestDispatcher.invoke(ServletReques
tDispatcher.java:621)
at
com.evermind.server.http.ServletRequestDispatcher.forwardInternal(Serv
letRequestDispatcher.java:370)
at
com.evermind.server.http.HttpRequestHandler.doProcessRequest(HttpReque
stHandler.java:871)
at
com.evermind.server.http.HttpRequestHandler.processRequest(HttpRequest
Handler.java:453)
at
com.evermind.server.http.AJPRequestHandler.run(AJPRequestHandler.java:
313)
at
199)
at
oracle.oc4j.network.ServerSocketReadHandler$SafeRunnable.run(ServerSoc
ketReadHandler.java:260)
at
com.evermind.util.ReleasableResourcePooledExecutor$MyWorker.run(Releas
ableResourcePooledExecutor.java:303)
at java.lang.Thread.run(Thread.java:619)
## Detail 0 ##
java.lang.NullPointerException
at
oracle.apps.fnd.preferences.webui.PreferencesPageCO.processRequest(Pre
ferencesPageCO.java:623)
at
anHelper.java:600)
at
at
oracle.apps.fnd.framework.webui.OAPageLayoutHelper.processRequest(OAPa
geLayoutHelper.java:1166)
at
oracle.apps.fnd.framework.webui.beans.layout.OAPageLayoutBean.processR
equest(OAPageLayoutBean.java:1569)
at
at
at
anHelper.java:659)
at
35
at
oracle.apps.fnd.framework.webui.beans.form.OAFormBean.processRequest(O
AFormBean.java:385)
at
at
at
anHelper.java:659)
at
at
oracle.apps.fnd.framework.webui.beans.OABodyBean.processRequest(OABody
Bean.java:353)
at
oracle.apps.fnd.framework.webui.OAPageBean.processRequest(OAPageBean.j
ava:2513)
at
:1894)
at
:538)
at
:426)
at _OA._jspService(_OA.java:212)
at
com.orionserver.http.OrionHttpJspPage.service(OrionHttpJspPage.java:59
)
at oracle.jsp.runtimev2.JspPageTable.service(JspPageTable.java:379)
at
oracle.jsp.runtimev2.JspServlet.internalService(JspServlet.java:594)
at oracle.jsp.runtimev2.JspServlet.service(JspServlet.java:518)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:856)
at
com.evermind.server.http.ResourceFilterChain.doFilter(ResourceFilterCh
ain.java:64)
at
oracle.apps.jtf.base.session.ReleaseResFilter.doFilter(ReleaseResFilte
r.java:26)
at
com.evermind.server.http.EvermindFilterChain.doFilter(EvermindFilterCh
ain.java:15)
at
oracle.apps.fnd.security.AppsServletFilter.doFilter(AppsServletFilter.
36
java:318)
at
com.evermind.server.http.ServletRequestDispatcher.invoke(ServletReques
tDispatcher.java:621)
at
com.evermind.server.http.ServletRequestDispatcher.forwardInternal(Serv
letRequestDispatcher.java:370)
at
com.evermind.server.http.HttpRequestHandler.doProcessRequest(HttpReque
stHandler.java:871)
at
com.evermind.server.http.HttpRequestHandler.processRequest(HttpRequest
Handler.java:453)
at
313)
at
199)
at
oracle.oc4j.network.ServerSocketReadHandler$SafeRunnable.run(ServerSoc
ketReadHandler.java:260)
at
com.evermind.util.ReleasableResourcePooledExecutor$MyWorker.run(Releas
ableResourcePooledExecutor.java:303)
at java.lang.Thread.run(Thread.java:619)
Detail 0, shown in the stack trace above, indicates that the problem is caused at runtime by a
“Null pointer exception” in line 623 of PreferencesPageCO.java, in the
processRequest() method. A null pointer exception means that an object being used hasn’t
been initialized. It is possible for the initialization of an object to be skipped if configuration is
incorrect or there is flawed logic within the controller, such as conditional initialization.
The developer must evaluate the controller to determine why it is failing. In order to help with
the debugging phase, the Oracle product development team may provide you with a debug
version of the controller in question and have you deploy it manually in your environment to
obtain more information about the conditions that triggered the error. In certain cases, it is
important to provide a testcase that can run in JDeveloper so the developer can apply advanced
debugging techniques that help to greatly reduce debugging time.
Note that the amount of information you provide when reporting an issue can mean the
difference between a long debugging process and a quick resolution within initial contact.
Section 2: Troubleshooting the Model
The model layer leverages Oracle ADF Business Components (Oracle ADF BC), which maps
database entities to Java objects. This solid framework has been tested for many years. Oracle
ADF BC is also formerly known as Business Components for Java (BC4J) or Java Business
Objects (JBO), and we continue to refer to the Java objects it creates as BC4J objects.
37
There are different kinds of BC4J objects used in any given page. A page or a region may map
to an Application Module (AM) that contains View Objects (VOs) that hold the results of queries.
Each row returned is held in a View Row Object (VORowImpl). Tables that are updatable also
contain Entity Objects (EO) that handle the transactions against the database. Please refer to
the BC4J or Oracle ADF BC documentation for more information about these products. From
this point on, this section focuses on how to troubleshoot or diagnose this model layer as used
by OA Framework.
Options to Debug the Model
Oracle E-Business Suite is highly dependent on “connection state”. When the AOL/J layer
creates a connection, the information is stored in a connection pool. When a user requests a
JDBC database connection against the pool, AOL/J configures this connection by setting the
PL/SQL variables (from the FND_GLOBAL package) that are specific to the connection, to the
context of the user session. For example, Operating Unit, Responsibility, User, and so on. Data
or queries are then filtered by these variables to enforce security.
During design time, these queries may be executed outside of this security model so their
results may differ from those obtained during run time.
The best way to determine which queries are being executed and if the results returned by such
queries are in line with expected results is by enabling On-Screen Logging.
For cases where On-Screen Logging is not available, you should use BC4J logging.
See the examples below and refer to Appendix A for more information on how to read a BC4J
log.
• On-Screen Logging
• BC4J Logging
• Verifying Profile Options
• Verifying Database Connectivity
• Analysis of BC4J Objects
On-Screen Logging
On-Screen Logging is a diagnostic facility provided by OA Framework to enable the printing of

debugging information beneath the very page that is being rendered.
To enable On-Screen Logging
1. Enable the profile option FND: Diagnostics to display the Diagnostics link in the Global
menu on the top right of the page.
2. Select the Diagnostics link to navigate to the Diagnostics page.
3. Set Diagnostic to Show Log on Screen.
4. Set Log Level accordingly. To capture all the queries being executed, set Log Level to
Statement (1).
5. All the pages you navigate to now display debugging as well as BC4J runtime
information at the bottom of the page as shown in Figure 15 and Figure 16.
38
Once a page renders, you can save the HTML file from the browser and upload it, if necessary,
to My Oracle Support for Oracle Support and the Oracle product development team to analyze.
Figure 15. Flow to enable On-Screen Logging
Figure 16. Detail of BC4J logging data when On-Screen Logging is enabled
39
BC4J Logging
If On-Screen Logging is not feasible to activate, as is the case when a page completely fails to
render, you can enable BC4J logging, which causes the BC4J layer to redirect all debug
information to the middleware’s log files. To be included, the JVM requires a system property to
be set via the -D parameter.
To enable BC4J Logging in Releases 11i, 12.0 and 12.1:
1. Edit the file oc4j.properties pertaining to the OACORE instance used by OA Framework
to run. The file is located under:
$ORA_CONFIG_HOME/10.1.3/j2ee/oacore/config
2. Add the following line to the file using the same capitalization as shown:
jbo.debugoutput=console
3. Bounce the OACORE instance:
cd $INST_TOP/admin/scripts
adopmnctl.sh stop
adopmnctl.sh start
If you have multiple middle tiers, this step is required across all the middle tiers.
4. Once the bounce completes, reproduce the problem.
5. After reproducing the problem, locate the corresponding log file:
cd $INST_TOP/log/ora/10.1.3/oacore_default_group_*/
vi oacorestd.out
40
Note: oacore_default_group_* is used to denote the name of the directory,

where * may be replaced with a number from 1 to the number of JVMs available
in your instance.
Note: If more than one JVM is enabled per middle tier then check the appropriate
oacorestd.out log file that was used to reproduce the issue. To find the
appropriate log file:
a. With the profile option FND: Diagnostics set to Yes, an “About this page” link
appears on the bottom left of the page. Select this link.
b. In the “About” page, select the Java System Properties tab.
c. Locate the system property named oracle.ons.indexid.
d. Replacing * in oacore_default_group_* with the value specified by the
oracle.ons.indexid system property. For example, if oracle.ons.indexid shows a
value of “1” then you should search the log file under
$INST_TOP/log/ora/10.1.3/oacore_default_group_1.
7. After reproducing the problem, undo any changes and bounce the middle tier as
specified in steps 1 to 3 above.
The resulting log file will provide valuable insight as to whether the problem was due to issues at
the model layer or to some other cause. Whenever possible, please provide this log file to the
engineer working on your Service Request.
To enable BC4J Logging in Release 12.2:
Enabling logging in Release 12.2 is a little more complicated. In releases prior to 12.2, OC4J
enabled stdout/stderr by specifing the -out and -err parameters (which OPMN used to set
automatically).
In Release 12.2, the process is a little bit different for Oracle WebLogic Server (WLS). First
output redirection should be enabled in the WebLogic Server via the WLS console as follows:
1. WLS config does allow for outputting System.out.println() – to enable it to use the
following steps in the WLS console:
2. On the left hand side panel, in [domain name] > Environment > Servers > [Server
Name] > Logging Tab -> General (open advanced), check the boxes "Redirect stdout
logging enabled" and "Redirect stderr logging enabled".
3. This requires a WLS restart for changes to take effect.
NOTE 1: The standard default username and password for Release 12.2 WLS
instances is weblogic/welcome1.
NOTE 2: The server should be named oacore[something].
4. To access the console, use the following URL:

http://[server].[domain]:[port]/[console]. Note that the port in this URL
should NOT be the same port used by the application deployment.
41
If you don’t know the port in use by the console application, look in
$CONTEXT_FILE for s_wls_adminport.
5. Start and stop the middle tier services under $ADMIN_SCRIPTS_HOME.

a. Once logging has been enabled, perform the following steps:
b. Edit $IAS_ORACLE_HOME/../user_projects/domains/[domain
name]/bin/startWebLogic.sh
c. Edit $IAS_ORACLE_HOME/../user_projects/domains/[domain
name]/bin/startWebLogic.sh
d. Locate a line that starts as SAVE_JAVA_OPTIONS=”${JAVA_OPTIONS}”
e. Modify this line to SAVE_JAVA_OPTIONS=”${JAVA_OPTIONS} -
Djbo.debugoutput=console”
f. Save the changes.
g. Start up WebLogic Server (via adstrtal.sh, being the safer option).
h. Reproduce the error.
i. Log file should be under
$IAS_ORACLE_HOME/../user_projects/domains/[domain
name]/servers/oacore*1/logs/
j. Search on the log files to ensure that you upload the file with the correct BC4J
information (log files are rotated).
k. To disable logging, undo all the steps performed.
Verifying Profile Options
BC4J behavior is very sensitive to certain profile options. Issues may occur that require you to
alter certain profiles to improve performance. Please refer to the "Profile Options" appendix in
the OA Framework Developer’s Guide, Document 1315485.1 for the comprehensive list of
profiles that affect OA Framework.
If you are uncertain whether a problem you encounter is related to a profile setting, please
revert that profile option to its default value and verify if the problem still occurs. Certain settings,
if modified, can cause JDBC connections to be released before a transaction is complete.
Although this is desirable as database connections can be reused more often, this can cause
applications that are highly dependent on PL/SQL state to fail with unpredictable errors. Once a
connection is returned to the JDBC pool, the PL/SQL state of such a connection is lost and can
not be reconstituted. If you need to improve the performance of an application, before you
change any profile option documented on the OA Framework Developer’s Guide, you should
first consult with Oracle Support to ensure your application is compatible with the changes you
plan to perform.
Verifying Database Connectivity
BC4J requires a reliable database connection at the JDBC layer. To ensure that a database
transaction works as expected and as designed, check the following:
• If you have a firewall between your middle tier and RDBMS server, make sure the
ports opened on the firewall do not timeout. The AOL/J layer pools the database
connections and even when they are not being used, the pool keeps a reference to
these connections. If these connection are severed prematurely, the pool will fail when
trying to reuse one of these connections. While you might change the pool's behavior by
42
altering options in the DBC file, prematurely severing database connections can cause
problems for the regular lifecycle of a transaction such as causing it to fail to commit. If
you need to tweak the behavior of the JDBC connection pool, please contact Oracle
Support for advice or follow the steps of Diagnosing and tuning AOL/J JDBC Pool in
Oracle e-Business Suite 11i, MOS Knowlege Document 278868.1.
• Disable any database profiles that involve session disconnection. The reason for
this advice is the same as that stated in the above item. To check for these database
profiles, execute the following queries:
1.
a. To determine if a user has IDLE_TIME set:
SELECT resource_name, resource_type, LIMIT

FROM dba_profiles
WHERE profile =
(SELECT profile
FROM dba_users
WHERE username = 'APPS')
AND resource_name = 'IDLE_TIME'
b. If idle_time is set, use the following SQL statements to disable it:
ALTER profile [profile_name]

LIMIT idle_time unlimited;
Note: Execute the SQL above as the SYSTEM database user.
• Remove any implemented timeout or automatic connection/disconnection

mechanisms. If you have any concerns regarding performance or the capacity of your
database server specific to the number of JDBC connections, you should resort to other
configuration mechanisms or re-evaluate your server capacity.
Analysis of BC4J Objects
To examine the BC4J objects used by a certain page or flow, select the About this page link at
the bottom left of that page. Under the Page subtab of the "About" page, select the Business
Components References Details hide/show region to expand and show all the business
components associated with the page.
Note: To enable Diagnostics, set the profile option FND: Diagnostics to Yes. Enabling
Diagnostics also displays the About this page link and functionality.
Please familiarize yourself with the "About" page functionality as it is very important. You can
determine the system properties for the middleware's Java process, your current security
context information, a technology stack component's version, and even the queries that each of
the VOs execute. Refer to Figure 17 fo a quick overview of the "About" page and to Figure 18
for a focused view of a page's BC4J information.
Figure 17. How to check for page details in the "About" page
43
Figure 18. Details of BC4J objects for a page as shown in the "About" page
Section 3: Diagnosing Personalizations
44
“Personalization” is a very powerful feature of OA Framework. It allows you (a user or system

administrator) to tailor how a page displays and allows you to make quick modifications to an
application without incurring the high customization costs. Personalizations, as long as there
are no major changes to the base page, are considered patch-safe. If Oracle removes a region
or item from a newer version of a page that it delivers as a patch or an upgrade, OA Framework
simply ignores the reference to that removed region or item in the personalization.
Administrators may create user-level personalizations to tailor the result of queries or alter the
sort order of data displayed in certain pages.
Administrators have a wider range of options when they create administrative-level (or admin-
level) personalizations. These options vary from changing labels in the page to adding complete
regions to the page with additional information.
All personalizations, along with the pages themselves, are stored in the Oracle MDS Repository
(Meta Data Services). When OA Framework’s rendering engine fetches a page from MDS, it
also obtains its personalizations (if any) and applies them on top of the page so that the page
metadata is already personalized by the time it reaches the core rendering engine.
Like OA Framework pages, personalizations are stored in MDS as XML documents. To perform
certain diagnostics steps, you need access to the database as the APPS account. If this is not
feasible, you may debug using the “About" page Personalization tab, or On-Screen Logging.
The scope of this section does not focus on the details of personalizations, but rather on
techniques to debug and troubleshoot personalizations. If you need more information about the
feature, please refer to the Oracle Application Framework Personalization Guide, available from
the Online Documentation Library.
• Considerations
• Verifying Personalizations
• What Can Go Wrong With Personalizations
Considerations
An important consideration for Personalizations is that you are only allowed to personalize a
page or a component if the page and its elements are defined as “personalizable”. For
example, you may personalize a page by adding "Record History" to a table in the page, as long
as the table’s Admin Personalization property is enabled.
If you feel that a page or page component needs to be personalizable due to a specific business
case, you should discuss this with the Oracle product team that owns that page and component
by filing an Enhancement Request through Oracle Support.
Verifying Personalizations
To get a glimpse of the personalizations that have been added to a page, you can enable
Diagnostics and access a page's details using the About this page link. If you select the About
this page link for the page in Figure 20, you can view the personalizations available to this page
as shown in Figure 21.
45
Figure 19. Example Base Page
Figure 20. Same example base page showing a personalized table ("Code" column is removed)
Figure 21. Using the "About" page to display details of personalizations applied to a base page
46
Figure 21 shows that a Customized Element (Code) that has been hidden.
To learn more about a personalization, you can access the database as follows:
1. Execute the following PL/SQL script to identify the personalizations applied to a given
page:
SQL> exec
jdr_utils.listcustomizations('/oracle/apps/fnd/security/menufunction/w
ebui/FunctionSearchPG');
/oracle/apps/fnd/security/menufunction/webui/customizations/site/0/Fun
ctionSearchPG
PL/SQL procedure successfully completed.
2. Next, execute the script below to display the XML content of the personalization returned
by the previous step:
SQL> exec
jdr_utils.printdocument('/oracle/apps/fnd/security/menufunction/webui/
customizations/site/0/FunctionSearchPG');
<?xml version='1.0' encoding='UTF-8'?>
47
<customization xmlns="http://xmlns.oracle.com/jrad"
version="9.0.6.0.0_50" xml:lang="en-US"
customizes="/oracle/apps/fnd/security/menufunction/webui/FunctionSearc
hPG">
<modifications>
<modify element="Code" rendered="false"/>
</modifications>
</customization>
PL/SQL procedure successfully completed.
As you can see in the above example, the first script obtains the list of personalization
documents associated with the base page. In this case, only one personalization document
defined at the site level exists.
The second script prints the XML content of the personalization document, and in this example,
it correctly shows that the element “Code” has the rendered property set to “false”.
What Can Go Wrong With Personalizations
In our experience, many of the personalization-related issues that occur are due to poor
personalization implementations that affect the logic of the application or to syntax errors in EL
(Expression Language) expressions added to a personalization. The latter may wreak havoc in
a page's rendering cycle.
Use the following remedial actions to verify whether a personalization is the cause of the
problem encountered within your page:
1. Turn off personalizations. Set the profile option Disable Self-Service

Personalizations (FND_DISABLE_OA_CUSTOMIZATIONS) to Yes. This causes the
logic that fetches page metadata from MDS to ignore the personalizations applied to the
page. If this resolves the problem, then analyze the personalizations applied to this
page and re-implement them in a way that works. After you complete your testing, be
sure to revert the profile option back to its original value.
2. Delete the personalization.
a. Navigate to the “Functional Administrator” responsibility and select the
Personalization tab.
b. In the Application Catalog page, specify the document path of the problem
page as shown on Figure 22.
Figure 22. Searching for Personalizations applied to a page using the Functional Administrator
responsibility
48
c. Select Go.
d. Once the table returns values, select the Manage Personalizations icon for the
specific problem page.
e. After locating the personalization, check the Select checkbox and select the
Delete Personalizations button as shown on Figure 23.
Figure 23. Managing personalizations via the Functional Administrator responsibility
3. Delete the personalization via SQL*Plus.
Note: Only use this method if you are not able to access the Oracle E-Business
Suite user interface.
Once you identify the document that contains the offending personalization,
delete the personalization by executing:
49
Exec JDR_UTILS.DELETEDOCUMENT(‘<name of the

personalization document>’);
Please ensure that you delete only the personalization document and not the
base page.
Note: You should export the document before deleting it in case you need to re-
introduce the personalization. Please refer to the Oracle Application Framework
Personalization Guide, available from the Online Documentation Library for
details on how to export a personalization document.
Section 4: Middle-tier Performance and Scalability Diagnostics
Autoconfig in Oracle E-Business Suite provides a thoroughly tested standard configuration that
suffices for most installations and offers very good response times. However, there may be
certain cases specific to the usage patterns of the installation where performance has room for
improvement or where the user community's perception that the application isn't responding as
expected needs to be addressed.
We define performance as the time it takes for the system to perform a certain action (such as
rendering a page or completing a transaction) and scalability as the ability of the application to
maintain a steady response time regardless of the utilization of the application. In the enterprise
world, performance and scalability are paramount as they translate into lower cost (in
resources) per transaction.
In an enterprise application such as Oracle E-Business Suite, it is important to consider

performance long before the application goes into production. Completing a good benchmarking
and sizing exercise right from the start goes a long way in ensuring the best possible
performance from the application.
This section discusses when and how to apply performance troubleshooting techniques in
certain scenarios, such as when an organization grows its original user base. While this section
does not cover how to properly size or benchmark an application, it does discuss what to look
for when users find transaction times unacceptable or what to do when you see an increase in
the occurrence of the middle tier running out of heap space.
We highly recommend that you familiarize yourself with J2EE application performance concepts
and with how Java handles memory space.
Common Performance Issues
The following is a list of common performance issues. While these issues tend to surface more
than others, this list is by no means complete. We'll go into some details for the first three items.
1. Response time - The response time for a given transaction is not acceptable.
2. Heap Exhaustion and Memory leaks - Memory allocated for certain processes is not
released after it has been used.
3. Thread Dumps and Deadlocks - Competing threads lock each other and cause the
application to stop responding to user interactions.
50
4. Inadequate configuration - The sizing or configuration is not appropriate for the

intended use.
One of the most important resources you can use for diagnosing performance issues are log
files. Log files help you profile the characteristics of the application. Some log files may require
more expertise than others to interpret. When in doubt, check the documentation for the
available logging facility and refer to the Appendices at the end of this document.
Response Time
To help you track response time, you can configure the HTTP server to log the time it takes to
service a request. Simply edit httpd.conf to add the %T option to the log format, as shown:
LogFormat "%{ClientIP}i %l %u %t [ecid: %{Oracle-ECID}i] \"%r\" %>s

%b [%T (secs)]" common
This above log format example results in the following optional output:
• %T - provides the response time (in seconds) for a requested page to be sent back to the
client. Note that this number does not include factors such as network latency, but
rather only the documented time taken by the HTTP server.
• [ecid: %{Oracle-ECID}i] - Allows a trace to the OC4J request to determine
whether the OC4J request succeeded or failed. For more information, please refer to
How to Configure OHS and OC4J Access Logs to Include Execution Context ID (ECID),
MOS Knowlege Document 266662.1.
After editing httpd.conf as described above, the log file output (access_log or any of its
variants) appears as follows:
148.87.19.51 1 - - [01/May/2007:11:38:49 -0700] "GET

/OA_HTML/AppsLocalLogin.jsp HTTP/1.1" 200 5986
148.87.19.51 0 - - [01/May/2007:11:38:53 -0700] "POST

/OA_HTML/fndvald.jsp HTTP/1.1" 302 259
148.87.19.51 5 - - [01/May/2007:11:38:58 -0700] "GET

/OA_HTML/OA.jsp?OAFunc=OAHOMEPAGE HTTP/1.1" 200 41609
148.87.19.51 5 - - [01/May/2007:11:39:15 -0700] "GET

/OA_HTML/OA.jsp?OAFunc=OAHOMEPAGE&
akRegionApplicationId=0&navRespId=20420&navRespAppId=1&navSecGrpId=0&t
ransactionid=36910577&
51
oapc=2 HTTP/1.1" 200 94506
To interpret the output, the first column shows the IP address of the client that originated the
request. The second column (the single underlined digit after the IP address) shows the time in
seconds that the server took to send the response back to the client.
Note that the log file does not display those requests that are still in process or are completely
frozen.
If you find that pages are rendering too slowly, you should isolate the layer where the
performance issue is occurring by checking the following:
• Rule out any SQL issues by enabling a SQL trace. For more information, see How To
Generate A SQL Trace In OA Framework For Oracle Applications, MOS Knowledge
Document 357597.1.
• Check for issues at the JVM layer:
o Collect thread dumps to rule out deadlocks (threads competing for resources that
lock each other).
o Analyze garbage collection information in the log files to ensure there are no
memory leaks and/or the heap space assigned to the JVM is sufficient for the
load.
o Use profiling tools such as Java, Middleware or other 3rd party tools
o Record, analyze, and verify heap dumps.
Heap Exhaustion and Memory Leaks
Java allocates in heap space (memory), the objects that a program creates. When it no longer
references an object, Java discards it via a "Garbage Collection" (GC) process. The heap is
divided into four spaces:
• Eden - Space where newly created objects reside.

• Survivor - Space where objects reside after they survive a number of garbage collections
because they are still referenced.
• Tenured (also known as Old) - Space where all long-lived objects reside. Most long-lived
objects are related to the J2EE container or other technology components that may have
started the JVM.
• Permanent - Space used to store loaded class files.
When the Eden space is full, Java’s memory management system triggers a minor collection,
also known as a “Partial Garbage Collection” (PGC), that performs the following:
• Moves objects residing in Eden with strong references to the Survivor space.
• Moves objects residing in the Survivor space that have been in this space for a long time
to the Tenured space.
When the Tenured space is full, it triggers a major collection or Full GC. In all your tuning
exercises, your objective should always be to keep the number of Full GCs to a minimum.
During a Full GC, the JVM usually halts all thread execution (at runtime) until the entire heap is
cleaned up, and may cause users to perceive that the application is freezing.
52
Monitoring GC Activity with JVM Options
The best way to monitor GC activity is to record GC information in your log files. You can
accomplish this by setting GC-specific JVM options in your context files. By default, when you
define these options in your context file, autoconfig propagates them into both jserv.properties
(for Apache Jserv in Oracle E-Business Suite 11i) and opmn.xml (for Oracle E-Business Suite
Release 12.0 and Release 12.1).
The following JVM options are specific to GC activity monitoring:
• -verbose:gc - enables logging of GC activity in the standard output files of both

Apache Jserv and OC4J as shown in the example in Figure 24. Note that you may see
some minor variations in the output related to your platform and your environment's
version of Java. For more information, please check the Java documentation specific to
your platform and Java version.
Figure 24. Sample output in log files with -verbose:gc JVM option set
• –XX:+PrintGCDetails - provides more details at GC to help you understand what is

happening with the different areas defined in the heap on each GC cycle. This
information may help you better tweak an object’s generations within the heap. Note
that tweaking the heap allocation requires an advanced understanding of Java’s heap
management and of the JVM options available.
The following is an example of the output produced when you enable the –
XX:+PrintGCDetails JVM option:
53
6.037: [Full GC [PSYoungGen: 1718K->0K(12480K)]

[PSOldGen: 3488K->5178K(113856K)] 5207K->5178K(126336K)
[PSPermGen: 10241K->10241K(20736K)], 0.0691040 secs]

9.187: [GC [PSYoungGen: 423K->80K(12480K)] 5602K-
>5258K(126336K), 0.0011820 secs]
9.188: [Full GC [PSYoungGen: 80K->0K(12480K)]

[PSOldGen: 5178K->5179K(113856K)] 5258K->5179K(126336K)
[PSPermGen: 10242K->10242K(23296K)], 0.0629940 secs]
Monitoring GC Activity with Other Tools
You can also use other tools to monitor GC activity:
• jstat - a command line utility for JVM statistics monitoring. Refer to its extensive
documentation for more information.
• JConsole - a graphical monitoring tool available in JDK 1.5.0_22 and above. This tool
provides real time data when connected to a JVM. It requires a graphical desktop
(usually Virtual Network Computing (VNC) is enough) and leverages Java Management
Extensions (JMX) technology so that any Management Bean available to the JVM could
show information to the user. To use JConsole:
1. Start JConsole in the same directory as your JVM installation

($JAVA_HOME/bin). Any user with proper privileges may launch it.
JConsole connects to the JVM via RMI and if you launch it from the same
machine running the JVM, you only need the JVM’s process ID (pid) to
connect.
2. Use the following command to determine the process ID:
$INST_TOP/admin/scripts/adopmnctl.sh status
For OA Framework-based applications, you need only the process

ID for the OC4J:oacore process. In the example shown in Figure
25, that process ID is 5702.
Figure 25. Using adopmnctl.sh to determine OACORE JVM's process ID
54
3. Launch JConsole with the process ID. For example:
jconsole 5702 [Enter]
Figure 26 - JConsole command line options
4. The JConsole graphical screen appears, as shown in Figure 27.
Figure 27. JConsole start window
55
Refer to the JConsole documentation and experiment with the tool

to familiarize yourself with JConsole. Note that there are some
differences between JConsole shipped with JDK 5.0 and JDK 6.0.
5. To view heap utilization, select the Memory tab, as shown in Figure 28.
Important: In general, you should keep full GCs to a minimum. A

high number of full collections is an indication of inadequate heap
space. Either increase heap size (using the -XMX parameter) or
consider increasing capacity by adding more middle tier servers.
GC time at the bottom of the screen shows the number of full
collections. PS MarkSweep stands for Full Collections whereas
PS Scavenge stands for minor collections.
Figure 28. Heap (Memory) monitoring via JConsole
56
Analyzing Heap Exhaustion
The JVM throws an OutOfMemoryError (Heap exhaustion) if after executing a Full GC the
JVM is unable to free more memory. While this error may surface at any possible place, it is
important to understand that the flow showing this error most likely has nothing to do with the
error itself. The cause of the error may vary from inadequately assigned heap space to memory
leaks or memory hemorrhage and will require a thorough analysis to identify.
If the error is caused by inadequate heap space, you might realize that the problem reproduces
only at high loads or only in specific flows that consume a big portion of the heap space. On the
other hand, if the error is caused by a memory leak/hemorrhage, the error may reproduce at any
time regardless of the load.
Resolving Inadequate Heap Space
For inadequate heap space, you need to analyze the heap utilization, determine the utilization
peaks, and size accordingly. For 32 bit systems, heap space has a 2GB limit. If you hit this
limit, the best approach is to add additional JVMs via autoconfig (parameter oacore_nprocs in
the context file), as long as your OS and hardware resources allow for it. The other option
would be the addition of more middle tier servers.
Resolving Memory Leaks/Hemorrhage
Analyzing a memory leak/hemorrhage is a bit more complex but there are steps that you can
follow to determine its cause. We define a memory leak/hemorrhage as the steady increase of
heap space utilization over a period of time. A memory leak occurs over a relatively long period
of time, whereas hemorrhage occurs over a short period of time, in minutes or even seconds.
57
The only appropriate action you can take for a memory leak/hemorrhage is to find the source of
the leak. Anything else, such as preventative cycling of the JVM, is a workaround until you
identify the root cause.
The easiest way to find the source of a leak is to set some JVM options. The following JVM
options signal the JVM to create a heap dump when it consumes the entire heap:
• -XX:+HeapDumpOnOutOfMemoryError
• -XX:HeapDumpPath=/<some>/<path>
Set these JVM options in opmn.xml (for Oracle E-Business Suite Release 12 and above) within
the oacore section. Then if the JVM throws an OutOfMemoryError, it will dump the contents
of the heap into a file in the directory specified by the -XX:HeapDumpPath option. You can
then open the heap dump file with Heap Analysis Tool (HAT), jhat (Java Heap Analysis Tool
available in JDK 6), or any tool that supports the binary heap dump format, such as Eclipse
Memory Analyzer. Please also check the Java documentation for your platform for any
changes to these options.
The example in Figure 29 illustrates heap dump analysis using Eclipse Memory Analyzer.
Please refer to the extensive Eclipse Memory Analyzer documentation for details of what you
can analyze, and how to analyze it.
Important: Eclipse Memory Analyzer requires a 64 bit OS to open a heap dump file If the heap
size is 1.5GB or more.
Figure 29. Eclipse Memory Analyzer
The following are important factors to consider when you analyze a heap dump:
58
1. Objects with an abnormal amount of allocations.

2. Objects that occupy a high amount of heap and that have possibly been instantiated
several times.
You should also upload the heap dump file (.hprof) to My Oracle Support for analysis when you
face one of these conditions.
For any other heap utilization concerns, we recommend you get a series of five or six heap
dumps from JConsole and use a tool such as Eclipse Memory Analyzer to compare the dumps
and check for unexplained peaks of heap utilization. To get ad-hoc heap dumps:
1. Start JConsole.
2. Select the MBeans tab.
3. On the left navigation pane, expand the tree folder com.sun.management > Hotspot
Diagnostics > Operations > dumpHeap
4. Enter a full path for the heap dump file in the p0 field and remember to include the .hprof
extension.
5. Select the dumpHeap button. If there are no issues, a “Method successfully invoked”
message appears.
6. Verify that the heap dump file specified in the p0 field now exists.
7. Take additional heap dumps but name the heap dump files 2.hprof, 3.hprof and so on, to
ensure that these heap dump files don’t get overwritten).
8. Open the heap dump files in Eclipse Memory Analyzer to detect any unwanted behavior.
Figure 30. Using JConsole to obtain an ad-hoc heap dump
59
Thread Dumps and Deadlocks
A J2EE container is basically a multi-threaded Java program, where many threads are in
standby for new user connections. Once a thread is taken by a user, all processing occurs
within the confines of that thread until the server finishes preparing the page for client rendering.
Once the thread finishes, it returns to the thread pool to wait for a new user connection, that is,
a new request for a page.
Problems can occur, however, when multiple threads compete for the same limited number of
resources. Under normal circumstances, a thread waits on the JVM's thread monitor for a
minimal amount of time until the resource it awaits becomes available. However, in extreme
cases, a JVM can be so overloaded that for all practical purposes, the resource that the threads
compete for becomes “never available”. This condition is called “thread contention” and while its
cause may vary, its effect is always the same, that is, pages take longer than usual to render.
Fortunately Java can provide a thread dump, a full stack trace of the job that a thread is
performing. When you request a thread dump on the J2EE container (Apache Jserv, OC4J, or
Oracle WebLogic), Java can report each running thread and the stack trace of each.
Additionally, the thread dump can show which threads are: locked, locking resources and
deadlocked.
60
Note that a thread dump provides a snapshot over a given period time, so to perform a
consistent analysis, you should request a series of thread dumps at a regular interval of time.
The presence of a locked thread is normal in a given snapshot. What is not normal is when the
same thread is constantly locked waiting for the same resource over several snapshots.
Requesting a Thread Dump
There are 3 ways to request a thread dump from the container. All examples below assume you
are connected to the middle tier server as the user owner of the Oracle E-Business Suite
installation (that is, as applmgr) and that the environment of your installation has been
sourced.
• Method 1 - send a termination signal (SIGQUIT) to the JVM process.

1. Refer to Figure 25 for how to find the process ID of the current OACORE
instance.
2. Once you identify the process ID, you can issue the following command:
$ Kill -3 <PID>
3. This causes the JVM to dump all threads and their stack traces into the STDOUT
files. Refer to Appendix B to find the location of these files for your specific
environment.
• Method 2 - use the jstack tool provided by Java (5.0 and above), located in the
directory where Java is installed. This is probably the easiest method since jstack
provides the thread dump through the standard output, allowing you to redirect it to a file
of your choosing so there's no need to search for it among the container’s standard
output files.
3. To use jstack, you must identify the process ID (PID) for the OACORE instance
as shown in Figure 25, and the location of jstack in the file system. Use the
following command to show the location of the JAVA executable file. Jstack is
installed in that same directory, which we will call $JAVA_HOME/bin in our
example.
$ cat `which java.sh` | grep JSERVJAVA
2. Execute the following command to generate the thread dump and redirect
it into the file ThreadDump_1.txt.
61
$ $JAVA_HOME/bin/jstack <PID> >> ThreadDump_1.txt
• Method 3 - use JConsole as shown in Figure 31.

1. Select the MBeans tab.
2. Expand the “java.lang” branch to Threading > Operations.
3. Select the dumpAllThreads operation if you wish to obtain thread information.
4. Additionally, navigate to the Threads tab and inspect each available thread in the
system graphically, as shown on Figure 32. This method only shows information
on the JConsole user interface. While it is convenient to use, especially in cases
where there is no access to the backend server via ssh or telnet, it does require
you to enable remote access to the JVM console.
Figure 31. JConsole access to threading methods to detect deadlocks and getting thread
information
This method allows you to quickly detect deadlocks between threads.

We define a deadlock as the following situation: Thread A has a lock on
resource 1 and is waiting on resource 2, which in turn is locked by thread
B. However, thread B is waiting on resource 1 before it can release
resource 2. Since no thread can win in this situation, you should report it
as a product defect. Although a deadlock is rare, it can occur on highly
utilized systems or with poorly coded customizations.
You can be easily identify a deadlock in a thread dump or in JConsole as

shown in Figure 32. The effects of a deadlock are notorious as it can
62
cause the current thread to hang, giving the user the impression
that the system is frozen. Additionally, thread cleanup by the JVM can
cause system instability.
Figure 32. Deadlock shown in a thread dump
Figure 33. Thread operations in JConsole
In summary, if JConsole detects a deadlock, the thread dump shows

information about the classes that are locked or locking. You should
open a Service Request with Oracle Support for further analysis.
63
Section 5: Troubleshooting the Java Object Cache and the Caching

Framework
Oracle E-Business Suite is dependent on a number of objects stored in the database that
include profile options, lookup values, message dictionary, and language dictionary, among
others. Architecturally speaking, accessing many objects in a database may be detrimental to
the overall response of an application since each access involves a round trip to the database to
fetch the data. To overcome this issue, Oracle E-Business Suite caches in the middle tier, many
components like those mentioned above that do not change during a session's lifcycle. Once
the application fetches the object from the database it remains in memory for the period of time
specified by its Time To Live (TTL).
This caching architecture includes three specific layers:
• Product-specific cache loaders (upper layer)

• Java Caching Framework (intermediate common layer)
• Java Object Cache (JOC) (lowest layer)
Cache Loaders
If a database object, that an application depends on, meets the criteria for caching, the
application defines a cache component for that object. In other words, it registers those objects
that are going to be cached with the Java Caching Framework. The registration process
assigns a cache loader class for the declared component. The cache loader class checks for
the entity within the cache or loads the entity from the database in case of a cache miss. The
loaded entity remains in the cache until it reaches its TTL.
Each application product team, from the perspective of supportability, owns their own cache
components and cache loaders.
Caching Framework
The Caching Framework is the layer within the architecture that provides a common set of APIs
that wraps the Java Object Cache (JOC) functionality. It defines a configuration that makes the
JOC suitable for use with Oracle E-Business Suite by making the whole infrastructure "aware"
of existing facilities that can be leveraged to increase the performance and functionality of the
JOC. Usually all Oracle E-Business Suite interactions with the JOC go through this layer,
making the JOC completely transparent to client applications. It also initializes the entire
architecture once the first cached component loads into the cache.
Java Object Cache (JOC)
The JOC is a component delivered by the technology stack (Oracle iAS/Fusion Middle Ware) on
which the Java Caching Framework is based on and built. To dig deeper into the architecture of
the Java Caching Framework, please refer to the Oracle E-Business Suite Java Caching
Framework Developer's Guide, MOS Knowledge Document 1108093.1 for Release 12 and
above, or MOS Knowledge Document 275879.1 for Release 11i.
64
Additionally, you should also familiarize yourself with the Java Object Cache documentation in
the Oracle Application Server Containers for J2EE Services Guide 10g available from the
Online Documentation Library.
Important: Always check that you have installed the latest JOC patch along with the latest JTF
Cache infrastructure patch for your environment.
Figure 34. Architecture overview of Java Object Cache stack
• Diagnosing the Java Caching Infrastructure

• Cache Loaders Failing to Load Data
• Cache Members Communication Problems
• Additional Troubleshooting Tools
o Clearing the Cache
o Available Logging Facilities
o CacheWatchUtil
Diagnosing the Java Caching Infrastructure
Although Oracle thoroughly tests the Java Caching Framework, the distributed nature of the
Java Object Cache causes it to be very sensitive to the number of middle tiers and network
topology in which Oracle E-Business Suite is deployed.
For example, individual JVMs connect to each other via network sockets but the first JVM to
come online in the system becomes the “coordinator” of the JVM pool. If a JVM in the system
loads a component by requesting a specific cached entity via its unique key, that JVM stores the
data pertaining to that entity for the period specified by its TTL. That JVM also notifies the
"coordinator" of the cache operation. The "coordinator" then takes charge of transferring the
data from one JVM to the next and keeps a table of the data loaded so if a second JVM
requests the same cached entity, it can reuse that object on the second JVM without having to
requery the database.
65
At some point, if the data changes, the JVM that triggers the data change, sends a cache
invalidation event to the "coordinator", which in turn tells the JVM holding the cached entity in
memory to replace the old data with the new data. In the case where a non-Java application
modifies the entity, such as when a Forms-based application triggers a business event via the
Business Event System (BES), the triggered business event notifies a specific JVM container
attached to the concurrent manager. That JVM container then notifies the "coordinator" of the
data change in the cached object. This applies to profile options as well as user-responsibility
mappings modified from the Oracle Forms interface.
Overall, the Java Caching Framework requires quite a few components to function properly.
These include:
• The Network
• The JOC itself
• The Business Event System
• Java processes
• Cache loaders
The most common problems that occur within the Java Caching infrastructure are:
• Cache loaders not loading data.

• Cache invalidation events not reaching the JOC.
Cache Loaders Failing to Load Data
If cache loaders are not loading data, you should analyze carefully whether this is due to:
a. A general failure in the cache infrastructure.

b. ALL cache loaders not working.
c. A subset of cache loaders not working.
If all cache loaders are failing, then this is a signal that something is wrong in the underlying
layer. In this case, even the middle tiers are unable to start so you should focus your
troubleshooting efforts on the network architecture and communication between the
"coordinator", all cache members, and the Business Event System.
If only specific cache loaders are failing, you should contact the Oracle product team that owns
those cache loaders and focus on specifics such as the queries executed and database
connectivity, i.e., JDBC Connection pool exhausted.
Cache Member Communication Problems
As mentioned in the introduction, all JVMs in the system are interconnected via the JOC. These
include:
• OACORE JVMs.
• Java concurrent programs.
• Java containers.
66
• Any standalone Java process using ATG libraries, such as Oracle Mobile Web
Application (MWA) Telnet Server.
One of the most common issues occur when a firewall exists between multiple middle tiers and
the JOC communication ports are closed. In this scenario, there are two cache areas, the
internal zone and the DMZ ("demilitarized zone"), and you must decide whether to open the
firewall ports or leave them closed with a warning that no cache invalidation events can go from
one zone to the other.
The following high level description helps you understand which ports need to be open in the
firewall:
The "coordinator" JVM tries to bind to the port number specified by the variable
s_java_object_cache_port in the Oracle E-Business Suite context file ($CONTEXT_FILE).
Additionally, each non-coordinator JVM also tries to bind to client ports. Although these client
ports are randomly assigned, you can refer to the context file variable
s_fnd_cache_port_range to predict which ports to open on the firewall.
It is very important that no other service should bind to these ports. If in doubt, assign high
values for these port numbers. For example, 60001 for the variable
s_java_object_cache_port and 60010-60015 for the variable
s_fnd_cache_port_range to open five ports. Make sure that the range of ports specified for
the variable s_fnd_cache_port_range contain enough ports for all cache members.
Once you identify the total number of ports to open, you should open the same number of ports
on the firewall in case the DMZ configuration includes a full blown middle tier installed on the
DMZ. If the DMZ only uses a reverse proxy, then no configuration changes are required.
Please review the following MOS Knowledge Documents for more information:
• Advanced Configurations and Topologies for Enterprise Deployments of E-Business

Suite 11i, MOS Knowlege Document 217368.1
• Tips and Queries for Troubleshooting Advanced Topologies, MOS Knowledge
Document 364439.1
• DMZ Configuration with Oracle E-Business Suite 11i, MOS Document 287176.1
• Using Load-Balancers with Oracle E-Business Suite Release 12, MOS Document
380489.1
• Oracle E-Business Suite R12 Configuration in a DMZ, MOS Document 380490.1.
Important: A common misunderstanding about the Java caching infrastructure is the belief that
you can disable it by setting the Java system property LONG_RUNNING_JVM to a value of
“false”. The Java caching infrastructure CANNOT be disabled. The only effect that changing this
property has is that it signals the entire infrastructure to work in STANDALONE MODE as
opposed to DISTRIBUTED MODE, which is the default and recommended setting. This setting
is appropriate ONLY for short lived JVMs, for example, standalone Java concurrent programs.
The side effect that results from such a configuration is that there will be no invalidation events
triggered between the JVMs available in the system. This can lead to “dirty” reads, such as
obtaining an “old” value for an entity, and may lead to unexpected issues such as corrupted
data.
67
Oracle recommends that you leave this parameter set to its default value of “true” for all
middle tier containers in the system.
Additional Troubleshooting Tools
To gather additional information when debugging cache infrastructure problems, you can use
the Oracle E-Business Suite-provided tool to help clear the cache as required in different
scenarios or enable different logging mechanisms.
Clearing the Cache
Use the Caching Framework administration page to clear the cache as required by first signing
in as a user with access to the Functional Administrator responsibility.
1. Once in the responsibility, select the Core Services tab and then the Caching
Framework sub tab.
2. Select Global Configuration from the side navigation menu to navigate to the Global
Cache Configuration page.
3. Under the Cache Policy region, select the Clear All Cache button and answer the
prompt that follows to completely flush the cache.
Note that this process cannot be undone. Also, depending on the number of objects in the
cache, you may have some performance impact until the cache replenishes the entities in use
across all JVMs and users. Please refer to the respective Oracle E-Business Suite Java
Caching Framework Developer's Guide for your release for more information about the Global
Cache Configuration page.
Figure 35. Global Cache Configuration page that allows users with the Functional Administrator responsibility to clear the cache
Available Logging Facilities
You may use the Java Object Cache built-in logging facilities to obtain additional information
about the Java cache. Check the file named $APPLRGF/javacache.log (for both Release 11i
68
and 12), for error messages when required. During normal operations, exceptions are logged
into this file. However, if you need to do further debugging, you may modify the Java System
Property named IASCACHELOGLEVEL to increase or decrease the log level and the type of
information being logged. The Java Caching Framework sets the default value of this property
to 3 at startup. The following table describes the meaning of the possible values:
DESIRED LOG LEVEL VALUE

DEBUG 15
DEFAULT 4
ERROR 3
FATAL 0
INFO 10
OFF -1
TRACE 7
WARNING 6
To modify the current log level, in addition to altering the log level of the system property you
must also bounce the JVM.
• For Release 11i - edit $IAS_ORACLE_HOME/Apache/Jserv/etc/jserv.properties.

Locate the first line that starts with wrapper.bin.parameters and add the line below.
Save your changes and restart Apache.
wrapper.bin.parameters=-DIASCACHELOGLEVEL=15
• For R12 - edit $ORA_CONFIG_HOME/10.1.3/j2ee/oacore/config/oc4j.properties and

add the line below to the end of the file. Save your changes and restart OACORE, not
Apache. In Release 12, bouncing Apache has no effect on the Java containers.
IASCACHELOGLEVEL=15
Once the container restarts, reproduce the error and look at the javacache.log file for
debugging messages. Note that this log file only captures errors with the JOC itself and with
lower layers such as networking. To debug and troubleshoot the Java Caching Framework that
is one level above the JOC, use Oracle E-Business Suite logging.
Generally speaking, you enable Oracle E-Business Suite logging by setting a specific group of
profile options. However, if you have issues with the JOC, this will not be an option since profile
options are cached entities and will not be readable in this case. Thankfully the Oracle E-
Business Suite logging framework can handle such a situation by redirecting logging information
to a file instead of the FND_LOG_MESSAGES database table. To enable logging in this
situation, edit either jserv.properties (for Release 11i) or oc4j.properties (for Release 12) as
described above by adding the following system properties:
69
AFLOG_ENABLED=true
AFLOG_LEVEL=statement
AFLOG_MODULE=jtf%
AFLOG_FILENAME=/tmp/some_log_file.log
As an example, in Release 11i, you would add the line:
wrapper.bin.parameters=-DAFLOG_ENABLED=true
Note that in Release 11i, you must precede every system property with a
"wrapper.bin.parameters=-D" string.
As an example in Release 12, you would add the line:
AFLOG_ENABLED=true
Once you save these settings to the appropriate file, bounce Apache (Release 11i) or opmn
(Release12):
• For Apache, use the command: adapcctl.sh stop

• For opmn, use the command: adopmnctl.sh shutdown|startall
Once the server bounces, reproduce your error and inspect the log file specified by the system
property AFLOG_FILENAME for additional error messages.
Important: After you collect all debugging information, remember to undo the changes you
make to the configuration file as logging generates very large files that may be detrimental to
your system's performance.
CacheWatchUtil
CacheWatchUtil is a command line tool that allows you to inspect the current caches in the
system, the cache content, and the cache configuration. This section provides a high level
overview of the utility. Refer to the Oracle iAS/Fusion Middle Ware documentation to familiarize
yourself with the tool so you can interpret the information provided by this utility. Currently, the
documentation for CacheWatchUtil may be found under the "Monitoring and Debugging" section
in the "Java Object Cache" chapter of the Oracle Application Server Containers for J2EE
Services Guide (Part No. B14012-02).
To start CacheWatchUtil, please create a shell script as follows:
In Unix (For Release 12 and above):
#!/bin/bash
70
java -cp $IAS_ORACLE_HOME/lib/dms.jar:$CLASSPATH -jar \
$ORACLE_HOME/../10.1.3/javacache/lib/cache.jar watch \
-config=$ORA_CONFIG_HOME/10.1.3/javacache/admin/javacache.xml
In Windows (For Release 12 and above):
@echo off
setlocal
REM the below line is a single line
java -cp %IAS_ORACLE_HOME%\lib\dms.jar:%CLASSPATH% -jar
%ORACLE_HOME%\..\10.1.3\javacache\lib\cache.jar watch
-config=%ORA_CONFIG_HOME%\10.1.3\javacache\admin\javacache.xml
REM until here
endlocal
The shell script makes the following assumptions:
1. The location of the Java executable is included in the PATH environment variable.
2. The Oracle E-Business Suite environment file is sourced before execution.
Launch the shell script to get the following command line utility interface:
Figure 36. CacheWatchUtil command line tool after being launched
71
At the command prompt, type “help [Enter]” to show the following options:
cache> help
Usage:
list caches : print reachable Cache(s) info including CacheId(s)
lc : shortcut for "list caches"
list objects [CacheId] [region=<region>] [sort=<0..7>]: print

objects
list
lo [CacheId] [region=<region>] [sort=<0..7>]: shortcut for "list

objects"
0: By region name
1: By object name
2: By group name
3: By object type
4: By valid status
5: By reference count
6: By access count
7: By expiration
id : print this CacheWatcher ID set severity=n [CacheId] : set

logger severity to n
set timeout=n : set Group Communication timeout value to n
get config [CacheId] : get configuration info
dump [CacheId] : cause Cache(s) dump its contents to the logger
invalidate: cause Cache to invalidate all objects in the cache
destroy: cause Cache to destroy all objects in the cache
72
iostat: distribute cache I/O statistics for this JVM
help or ? : Display this help
quit or exit: quit
cache>
Enter the command “List Caches”, to show the members included in the distributed cache as
follows:
cache> list caches
View Id: 12
View CId: 0
View Members:
#1. [10.242.50.224:15210, pos=0, uid=0, pri=0,
def=prop:/mnt/m0/r12/121x/slc00tmy/fmk1213a/apps/tech_st/10.1.3
oacore.default_group.1]
#2. [10.242.50.224:56407, pos=1, uid=1, pri=0,
def=prop:/mnt/m0/r12/121x/slc00tmy/fmk1213a/apps/tech_st/10.1.3
oacore.default_group.2]
#3. [10.242.50.224:13916, pos=2, uid=2, pri=0, def=prop:NA]
#8. [10.242.50.224:59960, pos=7, uid=9, tag=14105722, pri=0,

def=prop:NA]
View BitMap:
73
1111111100000000000000000000000000000000000000000000000000000000
cache>
You may use the above output to verify whether all the expected servers are connected to the
distributed cache via its IP address and port in use. You may use the ordinal number next to
each definition to obtain additional details about the member process. For example, to list
details of cache member #1, enter the command "lo 1" as follows:
cache> lo 1
Cache 1 at slc00tmy.us.oracle.com:56407
REGION OBJNAME GROUP TYPE REFCNT ACCCNT EXPIRE VALID

LOCK
-------- --------- ------- ------ -------- -------- -------- -------

------
[FND/GRANT_CACHE] [FND/GRANT_CACHE] [FND] Region 0

12385
None true null
[FND/FUNCTION_ID_CACHE] [FND/FUNCTION_ID_CACHE] [FND] Region 0

171
None true null
[CTRL_APP_NAME] [CTRL_APP_NAME] [null] Region 0 3 None

true
null
[FND/MENU_INFO_CACHE] [FND/MENU_INFO_CACHE] [FND] Region 0

73
None true null
74
[JTF] [JTF] [null] Region 0 3 None true

null
[FND/RESP_INFO_CACHE] [FND/RESP_INFO_CACHE] [FND] Region 0

227
None true null
[oracle_java_cache_service] [oracle_java_cache_service]
[null] Region 1
3 None true null
[FND/SECURITY_GROUP_INFO_CACHE] [FND/SECURITY_GROUP_INFO_CACHE]
[FND] Region 0
7 None true null
[CTRL_APP_NAME/CTRL_CACHECOMPONENT_KEY]
[CTRL_APP_NAME] Region 0 3 None true null
[FND/PROFILE_OPTION_VALUE_CACHE]
[FND/PROFILE_OPTION_VALUE_CACHE]
[FND] Region 0 1796 None true null
[FND/FUNCTION_INFO_CACHE] [FND/FUNCTION_INFO_CACHE] [FND]

Region 0
228 None true null
[FND/USER_INFO_CACHE] [FND/USER_INFO_CACHE] [FND] Region 0

16
None true null
[FND/MENU_ID_CACHE] [FND/MENU_ID_CACHE] [FND] Region 0

13
None true null
[FND/mds_region] [FND/mds_region] [FND] Region 0

385
None true null
75
[FND/FND_OA_POPLIST] [FND/FND_OA_POPLIST] [FND] Region 0

67
None true null
[FND/PROFILE_OPTION_CACHE] [FND/PROFILE_OPTION_CACHE] [FND]

Region 0
363 None true null
[FND] [FND] [null] Region 0 16 None true

null
[FND/LOOKUP_CACHE] [FND/LOOKUP_CACHE] [FND] Region 0

27
None true null
[FND/FUNCTION_ANCESTOR_CACHE] [FND/FUNCTION_ANCESTOR_CACHE] [FND]

Region 0
184 None true null
[JTF/JTF_PROPERTYMANAGER_CACHE] [JTF/JTF_PROPERTYMANAGER_CACHE]
[JTF] Region 0
3 None true null
cache>
The above command displays all the objects loaded into that particular cache member. The
output should differ across all members as entities are loaded into a particular cache and
shared across all members, including the coordinator. Using the same command for member #2
displays:
cache> lo 2
Cache 2 at slc00tmy.us.oracle.com:13916
REGION OBJNAME GROUP TYPE REFCNT ACCCNT EXPIRE VALID

LOCK
76
-------- --------- ------- ------ -------- -------- -------- -------

------
[CTRL_APP_NAME] [CTRL_APP_NAME] [null] Region 0 3 None

true
null
[oracle_java_cache_service] [oracle_java_cache_service] [null]

Region 1
3 None true null
[CTRL_APP_NAME] Region 0 5 None true null
cache>
Please refer to the CacheWatchUtil documentation mentioned above to learn more about the
different commands you can use. These include commands to dump information, including data,
about the cache objects stored on each member, to see the configuration of a cache member,
and so on. Use this tool to rule out connectivity issues and verify that all intended distributed
cache members are connected to the cache.
Section 6: Middleware Troubleshooting Checklists
The issues that arise in the middleware may vary greatly. Refer to the troubleshooting checklist
that best addresses your issue.
Exceptions Checklist
If you encounter an exception:
1. Make note of the navigation path that led to the issue and get a screen capture of the
issue, if possible.
2. Provide Oracle Support with the complete error stack. The information that appears after
the "Details" section in the error stack is the most relevant.
3. Provide Oracle Support with the following log and configuration files (include the
navigation path to the error and the date/time of the error, if possible).
A. Log files:
2.
77
 Tar and zip $LOG_HOME/ora/10.1.3/opmn/*

 Tar and zip $LOG_HOME/ora/10.1.3/j2ee/oacore/*
 Tar and zip $LOG_HOME/ora/10.1.3/j2ee/forms/*
 Tar and zip $LOG_HOME/ora/10.1.3/Apache/*
 Tar and zip $LOG_HOME/appl/admin/log/ad*
 $APPLRGF/javacache.log
2.
B. Configuration files:
2.
 tar cvf configFiles_`hostname`.tar
$ORA_CONFIG_HOME/10.1.2/* \
$CONTEXT_FILE \
$ADMIN_SCRIPTS_HOME \
$APPL_TOP/*.env \
$ORA_CONFIG_HOME/10.1.3/Apache \
$ORA_CONFIG_HOME/10.1.3/config \
$ORA_CONFIG_HOME/10.1.3/j2ee/oacore \
$ORA_CONFIG_HOME/10.1.3/j2ee/forms \
$ORA_CONFIG_HOME/10.1.3/j2ee/oafm \
$ORA_CONFIG_HOME/10.1.3/network \
$ORA_CONFIG_HOME/10.1.3/opmn \
$INST_TOP/ora/10.1.3/*.env
3. Does an error occur in the $APPLRGF/javacache.log when the application fails or
are there errors that occur on the date of the application failure?
3.
A. Are there firewalls between any nodes?
A.
o If there are firewalls, you must perform step "5.8: Enable Distributed Oracle Java
Object Cache Functionality" described in Oracle E-Business Suite R12
Configuration in a DMZ, MOS Knowledge Document 380490.1.
o Make sure all JOC ports are open in the firewall. If there are firewalls and the
distributed ports are not listed in the query below, then distributed JOC
functionality is not set up.
REM
REM START OF SQL
REM
set echo on
set timing on
set feedback on
set long 10000
set linesize 120
set pagesize 132
78
column SHORT_NAME format A30

column NAME format A40
column LEVEL_SET format a15
column CONTEXT format a30
column VALUE format A60 wrap
select p.profile_option_name SHORT_NAME,
n.user_profile_option_name NAME,
decode(v.level_id,
10001, 'Site',
10002, 'Application',
10003, 'Responsibility',
10004, 'User',
10005, 'Server',
10007, 'SERVRESP',
'UnDef') LEVEL_SET,
decode(to_char(v.level_id),
'10001', '',
'10002', app.application_short_name,
'10003', rsp.responsibility_key,
'10005', svr.node_name,
'10006', org.name,
'10004', usr.user_name,
'10007', 'Serv/resp',
'UnDef') "CONTEXT",
v.profile_option_value VALUE
from fnd_profile_options p,
fnd_profile_option_values v,
fnd_profile_options_tl n,
fnd_user usr,
fnd_application app,
fnd_responsibility rsp,
fnd_nodes svr,
hr_operating_units org
where p.profile_option_id = v.profile_option_id (+)
and p.profile_option_name = n.profile_option_name
and p.profile_option_name in
('FND_CACHE_PORT_RANGE','JTF_DIST_CACHE_PORT','FLEXFIELDS:U
SE_JAVA_CACHE')
and usr.user_id (+) = v.level_value
and rsp.application_id (+) =
v.level_value_application_id
and rsp.responsibility_id (+) = v.level_value
and app.application_id (+) = v.level_value
and svr.node_id (+) = v.level_value
and org.organization_id (+) = v.level_value
order by short_name, level_set;
REM
REM END OF SQL
REM
79
• Is there an error in the database alert.log at the time of the application error?
A. Search for an ORA-00600 error in the database alert.log.
• Enable FND Logging and search on the error:
A. To enable FND_Logging, set the following profile options at the User level:
o FND: Debug Log Enabled = Y
o FND: Debug Log Level = Statement
o FND: Debug Log Module = %
o FND: Debug Log Filename = <path>/fwk.log
Set <path> to somewhere writable, like /tmp. Log out and log back in.
B. Run the following query and note its value:
select max(log_sequence) from fnd_log_messages;
C. Reproduce the problem.

D. Run the following query and provide output in an Excel spreadsheet.
select * from fnd_log_messages

where log_sequence> {value noted in step B above}
E. Search for the error message in the <path>/fwk.log. Look for the date and time of
reproduction.
• Review the BC4J logging information if there is a failing query in the error stack.
A. Perform the following steps:

1. Edit
$ORA_CONFIG_HOME/10.1.3/j2ee/oacore/config/oc4j.properties.
2. Add the following line:
jbo.debugoutput=console
3. Save the changes.

4. Bounce the OACORE container:
$INST_TOP/admin/scripts/adoacorectl.sh stop | start
5. Reproduce your problem.

6. Locate the logging info in the stdout/stderr files:
80
$INST_TOP/logs/ora/10.1.3/oacore_default_group_*/oacor
estd.out|.err
B. Find the error in the $LOG_HOME/ora/10.1.3/opmn/oacore/ oacore output file

and review.
(Show example of output here and in the demo) Is an example available??
C. Perform the following query to determine if IDLE_TIME has been set:
SELECT resource_name, resource_type, LIMIT

FROM dba_profiles
WHERE profile =
(SELECT profile
FROM dba_users
WHERE username = 'APPS')
AND resource_name = 'IDLE_TIME'
If idle time is set, use the following SQL to disable it:
ALTER profile <profile_name>

LIMIT idle_time unlimited;
Note: Execute both SQL statements above as the SYSTEM database user.
D. Make sure your shared file systems have the 'nolock' option. For example, on Linux, it
should be using the "nolock" option and on Solaris, it should be using the "llock" option.
• Select the About this Page link on your page and review the information in the
Technology Components tab. You may also output this information by selecting the
Generate Bug Report button. The About this Page link appears at the bottom of every
OA Framework page when the "FND: Diagnostics" profile option is set to "Y", preferably
at the User Level.
• Turn off all personalizations at the Site level by setting the profile option "Disable Self-
service Personal" to "Yes" and then bouncing the middle tier services.
Personalizations Checklist
If after personalizing a page, you observe unexpected or non-working behavior, review the
following troubleshooting checklist.
1. Are your custom personalizations active on the page?
Obtain this information by selecting the About this Page link on your page and reviewing
the Personalizations tab. You may also output this information by selecting the
Generate Bug Report button. The About this Page link appears at the bottom of every
OA Framework page when the "FND: Diagnostics" profile option is set to "Y", preferably
at the User Level. .
81
2. Turn off personalizations at the Site level by setting the profile option "Disable Self-
service Personal" to "Yes" and then bouncing the middle tier services. Does the issue
still persist?
3. Review specific personalizations functionality in the OA Framework Personalization
Guide, available for download from
http://www.oracle.com/technetwork/documentation/applications-167706.html.
4. Remove the personalization if:

o You are unable to navigate to the page that was personalized.
o The personalization is suspected or confirmed to be the cause of the issue.
Refer to How to Remove an OA Framework Personalization, MOS Knowledge

Document 304670.1 for instructions.
Authentication, Timeout or HTML Error Code Checklist
If you encounter an authentication, timeout or HTML error code:
1. Provide Oracle Support with a diagram of your network topology.

A. How many Web servers do you have?
B. Are there load balancers, firewalls, or other network components in your
networked setup of Oracle E-Business Suite?
C. Are you using Oracle Real Application Clusters (Oracle RAC)?
2. If there is a load balancer in your network, refer to MOS Knowledge Document
601694.1, How To Check Session Persistence On BigIP F5, Cisco Ace, Citrix Netscaler
or Radware AppDirector Load Balancer Appliances.
3. If there is a firewall in your network:
A. Is the firewall configured as described in MOS Knowledge Document 380490.1,
Oracle E-Business Suite R12 Configuration in a DMZ?
B. Refer to MOS Knowledge Document 276557.1, Firewall BLACKOUT and JDBC
connections with Oracle Applications 11i and 12, to determine if there is a dead
connection and also to set the FND_JDBC_USABLE_CHECK flag to 'true' in the
DBC file to verify that the connection can be used.
4. Provide Oracle Support with log and configuration files.
A. Provide the date and time of the error and the navigation path that led to the
error.
B. Log files:
3.
 Tar and zip $LOG_HOME/ora/10.1.3/opmn/*
 Tar and zip $LOG_HOME/ora/10.1.3/j2ee/oacore/*
 Tar and zip $LOG_HOME/ora/10.1.3/j2ee/forms/*
 Tar and zip $LOG_HOME/ora/10.1.3/Apache/*
 Tar and zip $LOG_HOME/appl/admin/log/ad*
 $APPLRGF/javacache.log
3.
82
C. Configuration files:
3.
tar cvf configFiles_`hostname`.tar
$ORA_CONFIG_HOME/10.1.2/* \
$CONTEXT_FILE \
$ADMIN_SCRIPTS_HOME \
$APPL_TOP/*.env \
$ORA_CONFIG_HOME/10.1.3/Apache \
$ORA_CONFIG_HOME/10.1.3/config \
$ORA_CONFIG_HOME/10.1.3/j2ee/oacore \
$ORA_CONFIG_HOME/10.1.3/j2ee/forms \
$ORA_CONFIG_HOME/10.1.3/j2ee/oafm \
$ORA_CONFIG_HOME/10.1.3/network \
$ORA_CONFIG_HOME/10.1.3/opmn \
$INST_TOP/ora/10.1.3/*.env
4. Provide Oracle Support with HTTP traffic analyzer output (such as from Fiddler).
4.
A. Download and install Fiddler from www.fiddler2.com.
B. Navigate to Tools>Fiddler Options>HTTPS and make sure Decrypt Https traffic
is checked.
C. Filter traffic specific to your server by selecting the Filters tab. Select the Use
Filters checkbox and enter the host name of your middle tier server or entry
point.
D. Signal Fiddler to begin capturing traffic. Make sure to delete all sessions by
choosing Remove>All Sessions from the right mouse menu.
E. Reproduce the steps that lead to your problem.
F. After you complete the test case, save the Fiddler file and send it to Oracle
Support.
5. Analyze the HTTP traffic analyzer output to make sure every request contains at least
two special cookies: JSESSIONID and Oracle E-Business Suite session cookie. Perform
the following query against your database to determine the name of the Oracle E-
Business Suite session cookie:
SELECT FND_WEB_CONFIG.DATABASE_ID FROM DUAL;
Verify that these two special cookies maintain their value.
7. Review timeouts. The value of the "ICX_SESSION_TIMEOUT" profile option should

agree with the session timeout in
deployments/oacore/html/orion-web.xml. We recommend you set the timeout
to 30 minutes. You should set "ICX_SESSION_TIMEOUT" only at the Site level.
8. Check the inactivity timeout of the HTTP session. In the case of the OC4J container,
check
deployments/oacore/html/orion-web.xml. Make sure the following entry is in
the file:
83
<session-timeout>30</session-timeout>
Performance Checklist
To diagnose performance issues:
1. Does your JVM conform to the guidelines described in MOS Knowledge Document
362851.1, JVM: Guidelines to setup the Java Virtual Machine in Apps Ebusiness Suite
11i and R12? Specifically, does your JVM following the guideline for the number of
oacore processes running given the number of users and number of machines and
CPUs on each machine on the middle tier?
2. Provide Oracle Support with:
A. $CONTEXT_FILE.
B. The number of users during peak operation.
C. The number of web servers and the number of CPUs on each web server.
D. After performing the following steps, do you have 3 leaked connections stacks in
the leaked connections?
1. Connect as SYSADMIN and select the System Administration
responsibility.
2. Navigate to Diagnostics > AOL/J Database Connection Pool Status.
3. Select leaked connections.
4. Are there any java threads in red?
5. Monitor every couple of hours and take screen shots. Send the
screenshots to Oracle Support.
E. Review the database alert.log at the time of the error.
F. Make sure the database parameters conform to the recommendations in MOS
Knowledge Document 174605.1, bde_chk_cbo.sql - EBS initialization parameters
- Healthcheck. Provide Oracle Support with the BDE_CHK_CBO report as
described in the MOS Knowledge Document 174605.1.
84
Appendix A: Making the Most of BC4J Logging

A BC4J log file may show the following content when executing a query:
12/05/31 13:06:16 [2534] SELECT * FROM (SELECT DISTINCT
FunNetBatchesAllEO.BATCH_ID,
FunNetBatchesAllEO.BATCH_NAME,
FunNetBatchesAllEO.BATCH_NUMBER,
FunNetBatchesAllEO.BATCH_CURRENCY,
FunNetBatchesAllEO.BATCH_STATUS_CODE,
FunNetBatchesAllEO.SETTLEMENT_DATE,
HOU.NAME OPERATING_UNIT,
FL.MEANING BATCH_STATUS,
FNA.AGREEMENT_NAME,
DECODE(FunNetBatchesAllEO.BATCH_STATUS_CODE,
'SELECTED', 'UpdateEnabled',
'SUSPENDED', 'UpdateEnabled',
'REJECTED', 'UpdateEnabled',
'ERROR', 'UpdateEnabled',
'UpdateDisabled') UPDATE_BATCH_SWITCHER,
'RUNNING', 'RepDisabled',
'CLEARING', 'RepDisabled',
'RepEnabled') REPORT_SWITCHER,
85
'SELECTED', 'DelEnabled',
'SUSPENDED', 'DelEnabled',
'REJECTED', 'DelEnabled',
'ERROR', 'DelEnabled',
'CANCELLED', 'DelEnabled',
'COMPLETE', 'RevEnabled',
'REVERSED', 'RevDisabled',
'DelDisabled') DELETE_BATCH_SWITCHER,
FNA.PARTNER_REFERENCE,
HP.PARTY_NAME CUST_ACCT_NAME,
POV.VENDOR_NAME SUPPLIER_NAME
FROM FUN.FUN_NET_BATCHES_ALL FunNetBatchesAllEO,
FUN_NET_AGREEMENTS_ALL FNA,
FUN.FUN_NET_SUPPLIERS_ALL FNS,
FUN_NET_CUSTOMERS_ALL FNC,
HR_OPERATING_UNITS HOU,
FUN_LOOKUPS FL,
HZ_PARTIES HP,
HZ_CUST_ACCOUNTS_ALL HCA,
PO_VENDORS POV,
PO_VENDOR_SITES_ALL POVS
WHERE FNA.AGREEMENT_ID = FunNetBatchesAllEO.AGREEMENT_ID
AND FNA.AGREEMENT_ID = FNS.AGREEMENT_ID(+)
AND FNA.AGREEMENT_ID = FNC.AGREEMENT_ID(+)
AND HOU.ORGANIZATION_ID = FNA.ORG_ID
86
AND FL.LOOKUP_TYPE(+) = 'FUN_NET_BATCH_STATUS'
AND FL.LOOKUP_CODE(+) = FunNetBatchesAllEO.BATCH_STATUS_CODE
AND MO_GLOBAL.CHECK_ACCESS(HOU.ORGANIZATION_ID) = 'Y'
AND FNC.CUST_ACCOUNT_ID = HCA.CUST_ACCOUNT_ID
AND HCA.PARTY_ID = HP.PARTY_ID
AND FNS.SUPPLIER_ID = POV.VENDOR_ID
AND POV.VENDOR_ID = POVS.VENDOR_ID
AND MO_GLOBAL.CHECK_ACCESS(POVS.ORG_ID) = 'Y') QRSLT WHERE ((

UPPER(AGREEMENT_NAME) like
UPPER(:1) AND (AGREEMENT_NAME like :2 OR AGREEMENT_NAME like :3

OR AGREEMENT_NAME like
:4 OR AGREEMENT_NAME like :5))) ORDER BY BATCH_ID
12/05/31 13:06:16 [2535] Bind params for ViewObject:

NetBatchSearchResultVO1
12/05/31 13:06:16 [2536] Binding param 1: GJ Internal Aus%
12/05/31 13:06:16 [2537] Binding param 2: GJ%
12/05/31 13:06:16 [2538] Binding param 3: gj%
12/05/31 13:06:16 [2539] Binding param 4: Gj%
12/05/31 13:06:16 [2540] Binding param 5: gJ%
One of the most common issues identified from BC4J logging is query results that differ from
expected results. If this occurs, you should copy the query from the log file, replace the bindings
by each parameter, and execute the query to verify for differences.
The problem with the example log file shown above is that the WHERE clause in the query may
be able to contradict itself.
87
88
Appendix B: Standard output/Standard error files in the

containers
This document contains plenty of references to standard output (stdout) and standard error
(stderr) log files. This section explains how these files are generated and where in the file
system you can find these files. Given the non-interactive nature of J2EE containers running as
background services, it is necessary to redirect stdout and stderr to log files so that important
information can be captured for later review when debugging difficult problems or corner cases.
The location of these files will differ based on the technology involved, such as Apache Jserv,
OC4J and Weblogic.
Additionally, these files are important because some debugging and troubleshooting techniques
described in this document redirect information to the standard output. Also debugging classes
will most likely include System.out.println statements that go into the standard output file.
Release 11i
In Release 11i, STDOUT files are located under:
$IAS_ORACLE_HOME/Apache/Jserv/logs/jvm
The naming of these files follows this logic:
[group_name].[instance_ID].std[out|err]
For example, the stderr file name for the OACORE instance 5, is called:
OACoreGroup.5.stderr
The instance ID is set for each individual JVM running on the same Apache instance. For an
instance running with a single JVM, the stderr file is named OACoreGroup.0.stderr (0 being the
ID of the first JVM). If there are multiple middle tiers attached to the same Oracle E-Business
Suite instance, then each machine will have its own stdout|stderr files. For more information,
please refer to the Oracle iAS/Fusion Middle Ware documentation. Additionally, refer to the
MOS Knowledge documents written by the Oracle E-Business Suite technology stack team in
regards to Oracle E-Business Suite-specific configuration for Apache JServ.
Release 12.0 and 12.1
In Release 12.0 and 12.1 when OC4J is involved, the stderr and stdout files are redirected to
the following location:
$INST_TOP/logs/ora/10.1.3/opmn/oacore_default_group_N/oacorestd.out|er
r
Note that "N" replaces the OACORE instance ID. If there is a single JVM in the system, then
you would review the files in the following directory:
89
$INST_TOP/logs/ora/10.1.3/opmn/oacore_default_group_1/
The standard output file is named oacorestd.out and the standard error file is named
oacorestd.err.
Each JVM creates its own directory. If there are multiple middle tier servers attached to the
same Oracle E-Business Suite instance, each server will have its own files. For more
information, please refer to the Oracle iAS/Fusion Middle Ware documentation and the MOS
Knowledge documents written by the Oracle E-Business Suite technology stack team regarding
Oracle E-Business Suite-specific configuration for 0C4J.
Release 12.2 and Above
For Oracle Weblogic, you need to access the administration console and enable output
redirection since it is disabled by default for performance purposes. The administration console
directs you to the location of the log files. Given the level of complexity and functionality
available in the Oracle Weblogic console application, you should refer to the Oracle Fusion
Middle Ware documentation to learn more about this feature.
90
Appendix C: Deploying debug classes, jar files and .jsps

If you need to deploy debug files during the course of debugging and troubleshooting, it is
important you understand where to place such files so they don’t interfere with the standard
classes already deployed. This information is also helpful if you need to debug your own files
by collecting additional information during the execution flow of any Java class.
Before deploying your debug files, you should understand how classloading works with the
J2EE containers that Oracle supports. As with any Java program, the classpath specification
ensures the proper class is picked up by the classloader and ensures a successful debugging
session. The sections that follow illustrate how to deploy debug files for specific releases:
• Release 11i
• Release 12.0 and 12.1
• Release 12.2 and Above
• Common to All Releases/Technologies
Release 11i
Whenever a JAR file or a class file is deployed on the filesystem, the J2EE container needs to
know the location of that file. In Release 11i, the file jserv.properties specifies the location
where a container can pick up the class files. For each classpath, the file contains a
“wrapper.classpath” entry pointing to the library. That entry must be added at the beginning
of jserv.properties, ideally as the first classpath entry, to ensure the container picks up the
debug class or JAR file instead of the standard files.
For a JAR file, you may deploy it anywhere on the file system that the user applmgr can
access. When you edit jserv.properties, locate the first “wrapper.classpath” entry and insert
a new “wrapper.classpath” entry immediately above that first entry with the location of your
JAR file.
JAR file
As an example, suppose you have the following JAR file:
• Debug JAR file: mydebug.jar

• Deployment location on the file system: /tmp
The entry to include in jserv.properties should appear as follows:
wrapper.classpath=/tmp/mydebug.jar
Note: Debug JAR files may also be zip files.
Class file
As an example, suppose you have the following class file:
91
• Name of the class file: MyClass.class

• Fully qualified package name for the class file:
oracle.apps.myproduct.subset1.subset2.MyClass
• Deployment location on the file system: /tmp
In this case, the entry jserv.properties should appear as follows:
wrapper.classpath=/tmp
Important: To deploy a class file, Jserv needs the starting point of the package name. To
deploy the class file in the example above, you must move the class file to
/tmp/oracle/apps/myproduct/subset1/subset2/MyClass.class since the entire
directory structure for the package is required by the class loader to properly load the class.
Simply moving Myclass.class to /tmp will result in errors.
As you can see, the methods for editing jserv.properties differ greatly depending whether you
deploy a JAR/zip file or a class file. For JAR/zip files, you must include the full path to the JAR
file along with the JAR file name. For class files, you need the location where the package
name structure is expanded.
Once you modify jserv.properties, save your changes and restart Apache. Once you collect the
debugging information requested by Oracle Support, undo the changes to jserv.properties and
restart Apache again.
Release 12.0 and 12.1
In Release 12.0 and 12.1, OC4J loads the classes from the file orion-application.xml,
located under $ORA_CONFIG_HOME/10.1.3/ /j2ee/oacore/application-
deployments/oacore.
Note: OC4J is compliant with J2EE standards and the J2EE standards indicate that any J2EE
container can run more than one application. As a result, J2EE containers load common
libraries before any application code and it is very important that any changes you make do not
affect the common libraries.
To deploy your debug files:
1. Edit the file orion-application.xml with the editor of your choice and locate the
following line in the file:
<library path="" />
Add your entries immediately below this line. Do not overwrite this line.
2. To deploy a JAR file, your entry should look like this:
<library path="/path/to/my/jar/file/MyJar.jar" />
92
3. To deploy a single class file, your entry should look like this:
<library path="/u01/some/dir" />
If the class is named MyClass.class and belongs to the package

oracle.apps.mymodule, then the classloader will expect that the actual class
file is located under
/u01/some/dir/oracle/apps/mymodule/MyClass.class.
4. Save your changes and restart OPMN, using the command:
adopmnctl.sh shutdown|startall
5. Perform your debug activities to collect the information you expect.

6. After you finish debugging, undo the changes you made to the file orion-
application.xml and bounce OPMN again to bring the instance back its previous
state.
Note: Debugging classes originally deployed on shared libraries is unnecessary under any
circumstance.
Release 12.2 and Above
In Release 12.2 and above, debugging on the Oracle WebLogic container is a bit simpler since
it uses the standard J2EE configuration. Classes are deployed under $OA_HTML/WEB-
INF/classes and JAR files are deployed under $OA_HTML/WEB-INF/lib. Any entry under
$OA_HTML/WEB-INF/classes should be placed in a way that the directory structure matches
the package information. For example, the class file MyClass.class, which belongs to the
package oracle.apps.mymodule should be deployed under $OA_HTML/WEB-
INF/classes/oracle/apps/mymodule/MyClass.class.
Special consideration for JAR files - Since Oracle WebLogic is configured in production mode
by default, all deployed JAR files must be signed. The adpatch utility automatically adds a
signature to newly created JAR files. To successfully deploy your JAR files, you must change
the configuration of Oracle WebLogic so that the classloader doesn’t check for signatures by
editing the following file:
$FMW_HOME/user_projects/domains/<name of your EBS

domain>/bin/setDomainEnv.sh
Search for the following line in the file:
JAVA_OPTIONS="${JAVA_OPTIONS} ${JAVA_PROPERTIES} -
Dwlw.iterativeDev=${iterativeDevFlag} -
Dwlw.testConsole=${testConsoleFlag} -
Dwlw.logErrorsToConsole=${logErrorsToConsoleFlag}”
Replace the line above as follows:
93
JAVA_OPTIONS="${JAVA_OPTIONS} ${JAVA_PROPERTIES} -
Dwlw.iterativeDev=${iterativeDevFlag} -
Dwlw.testConsole=${testConsoleFlag} -
Dwlw.logErrorsToConsole=${logErrorsToConsoleFlag} -
Dweblogic.classloader.noJarSigners=true"
Additionally, edit "startWeblogic.sh" and add the option:
-Dweblogic.classloader.noJarSigners=true
Save your changes and restart Oracle WebLogic to pick up your changes.
After you finish your debugging session, make sure to undo all the changes above and restart
Oracle WebLogic again. Refer to Appendix B for information on how to locate the stdout and
stderr files for Oracle WebLogic.
The task to disable JAR signature verification is by far the simplest option to perform. There are
other options as well, documented in the OA Framework Developer’s Guide under the chapter
"Debugging OA Framework Applications", in the section named “Adding Debug Classes to the
Classpath of the Release 12.2 File System Structure”.
Common to All Releases/Technologies
To debug JSP files, you should deploy them under $JAVA_TOP. Since production mode is
enabled, the container does not check for timestamps to recompile the JSP file nor unload a
JSP class after it is loaded, so be sure to manually compile the class with ojspCompile.pl
and bounce the container accordingly to uptake your changes.
For Release 11i, use the following command:
$JTF_TOP/admin/scripts/ojspCompile.pl --compile -s <your JSP file> --

flush
For Release 12 (including Release 12.2 and above), use:
$FND_TOP/patch/115/bin/ojspCompile.pl --compile -s <your jsp file> --

flush
Important: Make a backup copy of the original JSP file before replacing it with your debug
version.
94
Appendix D: Oracle E-Business Suite Control Scripts

This document mentions several scripts that you can use to stop and start services. These
scripts reside in different locations depending on your version of Oracle E-Business Suite and
are documented in detail in the Oracle E-Business Suite System Administrator's Guide
Documentation Set for Release 12.1.2 and above (formerly called Oracle Applications System
Administrator's Guide Documentation Set in Release 11i to Release 12.1.1) and in the Oracle
E-Business Suite Maintenance Guide and Oracle E-Business Suite Setup Guide for Release
12.2 and above. The table below serves as a quick guide to these scripts' basic usage.
Location of scripts in Release 11i:
$OAH_TOP/admin/scripts/<instance_name>
Location of scripts in Release 12 and above:
$INST_TOP/admin/scripts
Script name Purpose Parameters

adstrtal.sh Starts all
application
services Username and password.
adstpall.sh For example: adstrtal.sh <username>/<pwd> or
Stops all adstpall.sh <username>/<pwd>
application
services
adapcctl.sh
Starts and For Release 11i:
stops start|stop|status|graceful|restart|configtest|h
Apache elp
For Release 12: start|stop|status
adopmnctl.sh
Controls
Oracle
Process
Manager
(opmn).
Only valid
start|stop|status|shutdown|startall
for
Release
12. Used
to stop or
start OC4J
instances
adoacorectl.s
Starts and
h
stops only start|stop
the
OACORE
95
OC4J
instance
java.sh
Wrapper
script for
the Java
executable
All parameters are passed to the Java executable. Any
. Used by
parameter may be passed as long as it’s understood by Java.
the
containers
to run
Java.
Please refer to the system administration documentation mentioned above for details about
other scripts residing on the same directory.
96
Resources
• MOS Knowledge Document 1315485.1 - Oracle Application Framework Developer’s
Guide,
• Oracle Application Framework Personalization Guide, available as part of the Online
Documentation Library on the Oracle Technology Network
• Oracle E-Business Suite System Administrator's Guide Documentation Set (Release
12.1.2 and above), Oracle Application System Administrator's Guide Documentation Set
(Release 11i to Release 12.1.1), Oracle E-Business Suite Maintenance Guide and
Oracle E-Business Suite Setup Guide (Release 12.2 and above), available as part of the
Online Documentation Library on the Oracle Technology Network
• MOS Knowledge Document 454178.1 - Oracle Application Server Diagnostic Tools and
Log Files in Applications Release 12
• MOS Knowledge Document 455194.1 - Diagnosing database invalidation issues with
Java Cache for eBusiness Suite
• MOS Knowledge Document 386568.1 - Diagnosing Java caching issues with Apps 11i
and Release 12
• MOS Knowledge Document 217368.1 - Advanced Configurations and Topologies for
Enterprise Deployments of E-Business Suite 11i
• MOS Knowledge Document 364439.1 - Tips and Queries for Troubleshooting Advanced
Topologies
• MOS Knowledge Document 380489.1 - Using Load-Balancers with Oracle E-Business
Suite Release 12
• MOS Knowledge Document 380490.1 - Oracle E-Business Suite R12 Configuration in a
DMZ
• MOS Knowledge Document 287176.1 - DMZ Configuration with Oracle E-Business Suite
11i
• MOS Knowledge Document 278868.1 - Diagnosing and tuning AOL/J JDBC Pool in
Oracle E-Business Suite 11i
• MOS Knowledge Document 266662.1 - How to Configure OHS and OC4J Access Logs
to Include Execution Context ID (ECID)
• MOS Knowledge Document 357597.1 - How To Generate A SQL Trace In OA
Framework For Oracle Applications
• MOS Knowledge Document 11.1 - Demo It To Oracle
97
98
Listing of Visual Elements

• Figure 1. Release 12.1 Three-tier logical architecture
• Figure 2. Release 12.2 Three-tier logical architecture
• Figure 3. OA Framework Component Stack
• Figure 4. Fiddler, after launching
• Figure 5. Using Fiddler to filter traffic specific to your server
• Figure 6. Selecting the browser process for Fiddler
• Figure 7. Login screen rendering with style sheet issues
• Figure 8. Data captured by Fiddler
• Figure 9. Detail of the request header showing the Cookie header
• Figure 10. Detail of cookies on "Headers" view where cookies are shown individually
• Figure 11. REST request and response generated in the home page (menu tree)
• Figure 12. Architecture of REST service interface
• Figure 13. Microsoft Script Debugger showing a breakpoint
• Figure 14. Determining a file's version using ADIDENT
• Figure 15. Flow to enable On-Screen Logging
• Figure 16. Detail of BC4J logging data when On-Screen Logging is enabled
• Figure 17. How to check for page details in the "About" page
• Figure 18. Details of BC4J objects for a page as shown in the "About" page
• Figure 19. Example Base Page
• Figure 20. Same example base page showing a personalized table ("Code" column is
removed)
• Figure 21. Using the "About" page to display details of personalizations applied to a
base page
• Figure 22. Searching for Personalizations applied to a page using the Functional
Administrator responsibility
• Figure 23. Managing personalizations via the Functional Administrator responsibility
• Figure 24. Sample output in log files with -verbose:gc JVM option set
• Figure 25. Using adopmnctl.sh to determine OACORE JVM's process ID
• Figure 26 - JConsole command line options
• Figure 27. JConsole start window
• Figure 28. Heap (Memory) monitoring via JConsole
• Figure 29. Eclipse Memory Analyzer
• Figure 30. Using JConsole to obtain an ad-hoc heap dump
• Figure 31. JConsole access to threading methods to detect deadlocks and getting thread
information
• Figure 32. Deadlock shown in a thread dump
• Figure 33. Thread operations in JConsole
• Figure 34. Architecture overview of Java Object Cache stack
• Figure 35. Global Cache Configuration page that allows users with the Functional
Administrator responsibility to clear the cache
• Figure 36. CacheWatchUtil command line tool after being launched
99
Change Log
Date Description
Sept. 19, Initial document created.
2013
Jan. 14, Added Troubleshooting Checklists
2015
Jan. 14, Added instructions for BC4J Logging in Release 12.2.
2015
Sept. 21, Added General Health Checks for Any OA Framework Issue to Part 1: Overview.
2015 Added Section 5: User Interface Issues to Part 3: Diagnosing the Client or
“Presentation Layer”.
Jun 3, Added Rich Table Issue in Section 5: User Interface Issues.
2016
Documentation Notices
Copyright © 2013, 2016 Oracle and/or its affiliates. All rights reserved.
This software and related documentation are provided under a license agreement containing
restrictions on use and disclosure and are protected by intellectual property laws. Except as
expressly permitted in your license agreement or allowed by law, you may not use, copy,
reproduce, translate, broadcast, modify, license, transmit, distribute, exhibit, perform, publish, or
display any part, in any form, or by any means. Reverse engineering, disassembly, or
decompilation of this software, unless required by law for interoperability, is prohibited.
The information contained herein is subject to change without notice and is not warranted to be
error-free. If you find any errors, please report them to us in writing.
If this is software or related documentation that is delivered to the U.S. Government or anyone
licensing it on behalf of the U.S. Government, the following notice is applicable:
U.S. GOVERNMENT END USERS: Oracle programs, including any operating system,
integrated software, any programs installed on the hardware, and/or documentation, delivered
to U.S. Government end users are "commercial computer software" pursuant to the applicable
Federal Acquisition Regulation and agency-specific supplemental regulations. As such, use,
duplication, disclosure, modification, and adaptation of the programs, including any operating
system, integrated software, any programs installed on the hardware, and/or documentation,
shall be subject to license terms and license restrictions applicable to the programs. No other
rights are granted to the U.S. Government.
This software or hardware is developed for general use in a variety of information management
applications. It is not developed or intended for use in any inherently dangerous applications,
including applications that may create a risk of personal injury. If you use this software or
hardware in dangerous applications, then you shall be responsible to take all appropriate fail-
safe, backup, redundancy, and other measures to ensure its safe use. Oracle Corporation and
100
its affiliates disclaim any liability for any damages caused by use of this software or hardware in
dangerous applications.
Oracle and Java are registered trademarks of Oracle and/or its affiliates. Other names may be
trademarks of their respective owners.
Intel and Intel Xeon are trademarks or registered trademarks of Intel Corporation. All SPARC
trademarks are used under license and are trademarks or registered trademarks of SPARC
International, Inc. AMD, Opteron, the AMD logo, and the AMD Opteron logo are trademarks or
registered trademarks of Advanced Micro Devices. UNIX is a registered trademark of The Open
Group.
This software or hardware and documentation may provide access to or information on content,
products, and services from third parties. Oracle Corporation and its affiliates are not
responsible for and expressly disclaim all warranties of any kind with respect to third-party
content, products, and services. Oracle Corporation and its affiliates will not be responsible for
any loss, costs, or damages incurred due to your access to or use of third-party content,
products, or services.
For information about Oracle's commitment to accessibility, visit the Oracle Accessibility
Program website at http://www.oracle.com/pls/topic/lookup?ctx=acc&id=docacc.
Access to Oracle Support
Oracle customers have access to electronic support through My Oracle Support. For
information, visit http://www.oracle.com/pls/topic/lookup?ctx=acc&id=info or visit
http://www.oracle.com/pls/topic/lookup?ctx=acc&id=trs if you are hearing impaired.
101
103

Oracle Application Framework Troubleshooting Guide: Release 11i To Release 12.2

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Oracle Application Framework Troubleshooting Guide: Release 11i To Release 12.2

Uploaded by

Copyright:

Available Formats

Oracle Application Framework

PART 1: OVERVIEW ........................................................................................................................1

General Health Checks for Any OA Framework Issue .............................................................. 2

PART 2: INTRODUCTION TO THE OA FRAMEWORK ARCHITECTURE ................................................... 5

Section 1: The Model ................................................................................................................ 5

Section 2: The View .................................................................................................................. 5

Section 3: The Controller ........................................................................................................... 5

Section 4: Three-tier Architecture .............................................................................................. 6

Section 5: A Graphical Look at the Architecture ........................................................................ 7

PART 3: DIAGNOSING THE CLIENT OR "PRESENTATION LAYER"....................................................... 9

Section 1: HTTP Traffic and Cookies ........................................................................................ 9

Section 2: REST Service Interface .......................................................................................... 16

Section 3: JavaScript Errors .................................................................................................... 18

Section 4: Presentation Layer Troubleshooting Checklist ....................................................... 19

Section 5: User Interface Issues ............................................................................................. 20

PART 4: TROUBLESHOOTING THE MIDDLEWARE ............................................................................ 31

Section 1: Troubleshooting the Controller ............................................................................... 31

Section 2: Troubleshooting the Model ..................................................................................... 37

Options to Debug the Model ................................................................................................ 38

On-Screen Logging .......................................................................................................... 38

Verifying Profile Options................................................................................................... 42

Verifying Database Connectivity ...................................................................................... 42

Analysis of BC4J Objects ................................................................................................. 43

Section 3: Diagnosing Personalizations .................................................................................. 44

Verifying Personalizations ................................................................................................... 45

What Can Go Wrong With Personalizations ........................................................................ 48

Section 4: Middle-tier Performance and Scalability Diagnostics ............................................. 50

Common Performance Issues ............................................................................................. 50

Response Time ................................................................................................................ 51

Heap Exhaustion and Memory Leaks .............................................................................. 52

Thread Dumps and Deadlocks......................................................................................... 60

Diagnosing the Java Caching Infrastructure ........................................................................ 65

Cache Loaders Failing to Load Data................................................................................ 66

Cache Member Communication Problems ...................................................................... 66

Additional Troubleshooting Tools ..................................................................................... 68

Clearing the Cache ....................................................................................................... 68

Available Logging Facilities .......................................................................................... 68

Section 6: Middleware Troubleshooting Checklists ................................................................. 77

Personalizations Checklist ................................................................................................... 81

Authentication, Timeout or HTML Error Code Checklist ...................................................... 82

Appendix A: Making the Most of BC4J Logging ...................................................................... 85

Appendix B: Standard output/Standard error files in the containers........................................ 89

Appendix C: Deploying debug classes, jar files and .jsps ....................................................... 91

Release 11i ..........................................................................................................................91

Release 12.0 and 12.1......................................................................................................... 92

Release 12.2 and Above ..................................................................................................... 93

Common to All Releases/Technologies ............................................................................... 94

Appendix D: Oracle E-Business Suite Control Scripts ............................................................ 95

Listing of Visual Elements ....................................................................................................... 99

Change Log ........................................................................................................................... 100

There is a change log at the end of this document.

This document focuses on OA Framework-based applications and related technologies such as

General Health Checks for Any OA Framework Issue

• By July 1, 2011: You must apply the Applications Technology Release

o Document 418664.1, Overview of Using Java with Oracle E-Business Suite

Part 2: Introduction to the OA Framework Architecture

• Section 1: The Model

Section 1: The Model

OA Framework uses Oracle Application Development Framework (ADF) Business Components

Section 2: The View

Section 3: The Controller

OA Framework provides an interface, OAController.java, and a class,

Section 4: Three-tier Architecture