1.

Gather background information about the environment

a) Run the following SQL to provide information about the nodes

REM
REM START OF SQL
REM
set echo on
set timing on
set feedback on
set long 10000
set pagesize 132
set linesize 80
col PLATFORM_CODE form a5
col HOST form a20
col DOMAIN form a30
col WEBHOST form a30
col VIRTUAL_IP form a20
col status form a20
col ConcMgr form a8
col Forms form a8
col WebServer form a8
col Admin form a8
col Database form a8
col last_monitored form a40
--
select
NODE_NAME,
to_char(CREATION_DATE, 'DD-MON-RR HH24:MI') creation_date,
PLATFORM_CODE,
decode(STATUS,'Y','ACTIVE','INACTIVE') Status,
decode(SUPPORT_CP,'Y', 'ConcMgr','No') ConcMgr,
decode(SUPPORT_FORMS,'Y','Forms', 'No') Forms,
decode(SUPPORT_WEB,'Y','Web', 'No') WebServer,
decode(SUPPORT_ADMIN, 'Y','Admin', 'No') Admin,
decode(SUPPORT_DB, 'Y','Rdbms', 'No') Database,
to_char(LAST_MONITORED_TIME, 'DD-MON-RR HH24:MI:SS') last_monitored,
NODE_MODE,
SERVER_ADDRESS,
HOST,
DOMAIN,
WEBHOST,
VIRTUAL_IP,
SERVER_ID
from fnd_nodes
where node_name != 'AUTHENTICATION'
/
REM
REM END OF SQL
REM

b) Provide the following Apache configuration files

Release 11i

All files located in the $IAS_ORACLE_HOME/Apache/Apache/conf directory

All files located in the $IAS_ORACLE_HOME/Apache/Jserv/etc directory
$IAS_ORACLE_HOME/Apache/Apache/bin/java.sh
$FND_TOP/secure/<SID>_<Hostname>/<SID>.dbc
$APPL_TOP/admin/<SID>_<Hostname>.xml
You could use a script like that below to quickly gather this information:-

tar cvf mzConfigFiles_`hostname`_`date '+%m%d%y_%H%M'`.tar \

$IAS_ORACLE_HOME/Apache/Apache/conf/* \
$IAS_ORACLE_HOME/Apache/Jserv/etc/* \
$IAS_ORACLE_HOME/Apache/modplsql/cfg/* \
$IAS_ORACLE_HOME/Apache/Apache/bin/java.sh \
$FND_TOP/secure/*/*.dbc \
$APPL_TOP/admin/*.xml

Collate the latest versions of all configuration files for Apache and OC4J configuration files, which are
mainly located at $ORACLE_CONFIG_HOME

You can use the following command to package up these files:-

tar cvf configFiles_`hostname`.tar $ORA_CONFIG_HOME/10.1.2/* \

$CONTEXT_FILE \
$ADMIN_SCRIPTS_HOME \
$APPL_TOP/*.env \
$ORA_CONFIG_HOME/10.1.3/Apache \
$ORA_CONFIG_HOME/10.1.3/config \
$ORA_CONFIG_HOME/10.1.3/j2ee/oacore \
$ORA_CONFIG_HOME/10.1.3/j2ee/forms \
$ORA_CONFIG_HOME/10.1.3/j2ee/oafm \
$ORA_CONFIG_HOME/10.1.3/network \
$ORA_CONFIG_HOME/10.1.3/opmn \
$INST_TOP/ora/10.1.3/*.env

c) Confirm your network architecture and whether you are using a DMZ: A network topology diagram
would be useful, otherwise please describe the eBusiness environment network architecture

2. Check latest Java Cache patches have been applied

Release 11i

Consider applying these patches :

Bug 7527878 "FIX FOR BUG 7484199 FOR ORACLE EBS 11i (JOC 10.1.3.3 for iAS 1.0.2.2)"
Bug 7437603 "BACKPORT:6746642:RUP6:11I.ATG_PF.H:11.5.10.2:AITR12:INTERMITTENT
OACORE APPLICA"
OR
Bug 7389573 "STANDALONE RELEASE OF BUG 7286131 ON RUP5"

The above patches are all included in ATG RUP7

3. Confirm all eBiz nodes have been configured correctly

a) Use the following SQL to list the relevant profile options for Java Caching

REM
REM START OF SQL
REM
 set echo on
set timing on
set feedback on
set long 10000
set linesize 120
set pagesize 132
column SHORT_NAME format A30
column NAME format A40
column LEVEL_SET format a15
column CONTEXT format a30
column VALUE format A60 wrap
select p.profile_option_name SHORT_NAME,
n.user_profile_option_name NAME,
decode(v.level_id,
10001, 'Site',
10002, 'Application',
10003, 'Responsibility',
10004, 'User',
10005, 'Server',
10007, 'SERVRESP',
'UnDef') LEVEL_SET,
decode(to_char(v.level_id),
'10001', '',
'10002', app.application_short_name,
'10003', rsp.responsibility_key,
'10005', svr.node_name,
'10006', org.name,
'10004', usr.user_name,
'10007', 'Serv/resp',
'UnDef') "CONTEXT",
v.profile_option_value VALUE
from fnd_profile_options p,
fnd_profile_option_values v,
fnd_profile_options_tl n,
fnd_user usr,
fnd_application app,
fnd_responsibility rsp,
fnd_nodes svr,
hr_operating_units org
where p.profile_option_id = v.profile_option_id (+)
and p.profile_option_name = n.profile_option_name
and p.profile_option_name in
('FND_CACHE_PORT_RANGE','JTF_DIST_CACHE_PORT','FLEXFIELDS:USE_JAVA_CACHE')
and usr.user_id (+) = v.level_value
and rsp.application_id (+) = v.level_value_application_id
and rsp.responsibility_id (+) = v.level_value
and app.application_id (+) = v.level_value
and svr.node_id (+) = v.level_value
and org.organization_id (+) = v.level_value
order by short_name, level_set;
REM
REM END OF SQL
REM

The JTF_DIST_CACHE_PORT profile option must be consistent for all nodes and needs to match the entry
in jserv.properties. FND_CACHE_PORT_RANGE profile option should only be set when using DMZ
configurations and is explained in Note 287176.1 "DMZ Configuration with Oracle E-Business Suite 11i"
in section "5.9: Enable Distributed Oracle Java Object Cache Functionality"
In a DMZ environment, the port numbers used by both these two profile options need to be opened
through the firewall. Also bear in mind that a common problem is to miscalculate the number of ports
required so pay particular attention to the FND_CACHE_PORT_RANGE calculation

b) Use the following SQL to check the AQ setup and Workflow Agent Listener/WF Java Deferred Agent
Listeners are running

REM
REM START OF SQL
REM
set echo on
set timing on
set feedback on
set pagesize 132
set linesize 100
col name form a30
col queue_table form a30
col enqueue_enabled form a7
col dequeue_enabled form a7
col retention form a20
--
select name,
queue_table,
enqueue_enabled,
dequeue_enabled,
retention
from dba_queues
where owner = 'APPLSYS'
and queue_type = 'NORMAL_QUEUE'
order by 1
/
--
set pagesize 132
set linesize 90
col COMPONENT_NAME form a50
col COMPONENT_STATUS form a15
col COMPONENT_TYPE form a25
col STARTUP_MODE form a10
col CONTAINER_TYPE form a5
col INBOUND_AGENT_NAME form a20
col OUTBOUND_AGENT_NAME form a20
col CORRELATION_ID form a30
col MAX_IDLE_TIME form 9999999
rem
select
COMPONENT_NAME,
COMPONENT_STATUS,
COMPONENT_TYPE,
STARTUP_MODE,
CONTAINER_TYPE,
INBOUND_AGENT_NAME,
OUTBOUND_AGENT_NAME,
CORRELATION_ID,
MAX_IDLE_TIME
from FND_SVC_COMPONENTS
order by COMPONENT_NAME
/
REM
REM END OF SQL
REM
c) Check Event Group 'oracle.apps.fnd.bes.control.group' is registered and enabled in the Workflow
System

REM
REM START OF SQL
REM
REM List events related to Apps 11i Java Caching
REM
set echo on
set feedback on
set pagesize 132
set linesize 100
col name form a60
col stat form a8
col evt.licensed_flag form a1
col generate_function form a65
col ecl form a6
col sub_stat form a8
col sub_lic form a1
col sub_cl form a6
col rule_function form a45
col source_type form a8
--
select evt.name,
evt.type,
evt.status stat,
evt.licensed_flag,
evt.generate_function,
evt.java_generate_func,
sub.STATUS sub_stat,
sub.LICENSED_FLAG sub_lic,
sub.SOURCE_TYPE source_type,
sub.phase,
sub.description,
sub.rule_function rule_function,
sub.java_rule_func
from wf_events evt, wf_event_subscriptions sub
where evt.guid = sub.event_filter_guid
and evt.name = 'oracle.apps.fnd.bes.control.group'
order by evt.name,evt.type, sub.source_type desc
/
--
-- check which events registered to bes.control.group
--
select name, status
from wf_events
where guid in (select member_guid
from wf_event_groups
where group_guid =
(select guid
from wf_events
where name = 'oracle.apps.fnd.bes.control.group'))
order by 2,1
/
---
--- Check 'oracle.apps.fnd.wf.ds.userRole.updated' event
---
select event_name,
subscription_source_type,
 subscription_phase,
subscription_rule_data,
subscription_rule_function
from WF_ACTIVE_SUBSCRIPTIONS_V
where event_name = 'oracle.apps.fnd.wf.ds.userRole.updated'
/
REM
REM END OF SQL
REM

d) List the nodes currently registered for Java Caching and the cache configuration details

REM
REM START OF SQL
REM
set echo on
set timing on
set feedback on
set long 10000
set pagesize 132
set linesize 90
col APPLICATION_ID form 9999999
col COMP_NAME form a40
col COMPONENT_KEY form a40
col LOADER_CLASS_NAME form a60
col BUSINESS_EVENT_NAME form a80
col HOSTNAME form a50
REM
REM Collect information about the registered nodes
SELECT B.HOSTNAME,
decode(B.LOAD_PICK_UP_FLAG,'t','TRUE','FALSE') load_pickup_flag,
decode(B.PREFAB_ENABLED_FLAG,'t','TRUE','FALSE') prefab_enabled_flag,
B.CACHE_SIZE,
B.WSH_TYPE,
T.DESCRIPTION,
B.WEIGHT,
to_char(B.LAST_UPDATE_DATE,'DD-MON-RR HH24:MI') last_update_date
from JTF_PREFAB_WSH_POES_TL T, JTF_PREFAB_WSH_POES_B B
WHERE B.WSH_PO_ID = T.WSH_PO_ID (+)
order by B.HOSTNAME
/
REM
REM Information about the cache component items
REM
select
COMP_NAME,
COMPONENT_KEY,
LOADER_CLASS_NAME,
decode(TIMEOUT_TYPE,'it','IdleTime','ttl','TimeToLive','unknown') timeout_type,
TIMEOUT,
TIMEOUT_UNIT,
APPLICATION_ID,
decode(SGID_ENABLED_FLAG,'t','TRUE','FALSE') SGID_ENABLED,
decode(DISTRIBUTED_FLAG,'t','TRUE','FALSE') Distributed,
BUSINESS_EVENT_NAME
from JTF_PREFAB_CA_COMPS_B
order by APPLICATION_ID,COMP_NAME
/
REM
 REM END OF SQL
REM

e) Run the following SQL to identify how frequently the listeners run and their service parameters

REM
REM START OF SQL
REM
set echo on
set timing on
set feedback on
set long 10000
set pagesize 132
set linesize 100
col CONCURRENT_QUEUE_NAME form a20
col USER_CONCURRENT_QUEUE_NAME form a35
col MIN_PROCESSES form 9999
col MAX_PROCESSES form 9999
col SLEEP_SECONDS form 9999
col SERVICE_PARAMETERS form a95
--
select
FCQ.CONCURRENT_QUEUE_NAME,
FCQ.USER_CONCURRENT_QUEUE_NAME,
FCQS.MIN_PROCESSES,
FCQS.MAX_PROCESSES,
FCQS.SLEEP_SECONDS,
FCQS.SERVICE_PARAMETERS
from FND_CONCURRENT_QUEUE_SIZE FCQS, FND_CONCURRENT_QUEUES_TL FCQ
where FCQS.CONCURRENT_QUEUE_ID = FCQ.CONCURRENT_QUEUE_ID
and FCQ.CONCURRENT_QUEUE_NAME like 'WF%'
order by 1
/
REM
set linesize 110
col COMPONENT_NAME form a40
col COMPONENT_STATUS form a8
col parameter_name form a30
col parameter_value form a25
--
select c.COMPONENT_NAME,
c.COMPONENT_STATUS,
p.parameter_name,
v.parameter_value
from FND_SVC_COMP_PARAM_VALS v,
fnd_svc_comp_params_b p,
fnd_svc_components c
where v.component_id = c.component_id
and v.parameter_id = p.parameter_id
and upper(c.COMPONENT_NAME) like upper('Workflow Java Deferred Agent
Listener')
order by 1,3
/
REM
REM END OF SQL
REM

f) Check the file system (Release 11i only)

For each Apps Middle Tier node, run the following Unix script. This script will pull out the most relevant
configuration file settings and most relevant Java code version information

#####################################################################
### START OF SCRIPT ###
#####################################################################
LOG_FILE="mzJavaCacheChecks_`hostname`"
(
echo "Run started "
date
echo " "
##
echo "Checking jserv.conf"
echo "-------------------"
grep "^ApJServGroup " $IAS_ORACLE_HOME/Apache/Jserv/etc/jserv.conf
echo " "
echo "Checking jserv.properties"
echo "-------------------------"
grep -e "IASCacheProvidercacheProvider" -e "LONG_RUNNING_JVM" -e "CACHEMODE" -e
"JTFDBCFILE" $IAS_ORACLE_HOME/Apache/Jserv/etc/jserv.properties
echo " "
echo "Checking zone.properties"
echo "-------------------------"
grep "^servlets.startup" $IAS_ORACLE_HOME/Apache/Jserv/etc/zone.properties
##
echo " "
echo "Checking Java class versions"
echo "----------------------------"
adident Header $OA_JAVA/oracle/apps/jtf/cache/*.class
##
echo " "
echo "Run completed "
date
) 2>&1 | tee ${LOG_FILE}.txt
#####################################################################
### END OF SCRIPT ###
#####################################################################

NOTE - if the line below is returned from this script, then it would be worth re-testing after commenting
it out, as this has been known to contribute to Java Caching issues
servlets.startup=oracle.apps.fnd.framework.OAStartupServlet

g) Run the Java Diagnostic tests and upload the results

These can be accessed by either :-

a) CRM HTML Administration responsibility

Diagnostics--> CRM Foundation--> Caching Framework
then select the "Run without Prerequisite" button to launch the tests
this will produce two test reports which you can then view and "Upload to Support" or "Save" as a file

b) Direct navigation to the diagnostic tests

http://<hostname>:<port>/OA_HTML/jtfqalgn.htm
4. Identify nature and scope of the issue
This will confirm the extent of the issue by gathering the results of the following different tests. This will
confirm the extent of the issue by gathering the results of the following different tests. Use the same
browser session for all these tests to ensure the same JServ session is used for each test

THE FOLLOWING HIGHTLIGHTED ITEMS HAVE BEEN CONFIRMED – These are the issues happening in
our environment:
4.1. New responsibilities for existing users not showing
a. Login as an existing user, (for example TESTUSER1) then logout. This is to ensure the Java
Cache has been initialized and to set the JServ session cookie
b. Login as SYSADMIN user to add new self service framework responsibility to an existing user
(TESTUSER1)
b. Log out
d. Log on as the TESTUSER1 user
e. Confirm if the new responsibility is available or not
f. If not, bounce Apache or clear the Java cache
g. Login as the TESTUSER1 again, and confirm the new responsibility is now available

4.2 Changes to responsibilities not showing immediately

a. Disable or re-enable self service framework responsibility. (blank out end date or introduce
an end date of yesterday)
b. Log off
c. Log on
d. Confirm if the change to the responsibility has taken effect or not
e. Bounce Apache
f. Re-check to see if change to the responsibility has taken effect or not

4.6. Time based test

If any of the above tests failed to display a change without bouncing Apache, re-do the same test but
this time wait for 60 minutes after making the change before you login again to test if the change has
taken effect.

This test is to confirm whether the change is not happening at all, or is just taking a long time to process

THE FOLLOWING HIGHTLIGHTED ITEMS HAVE BEEN TESTED IN OUR ENVIRONMENTS – Results were
inconsistent, in that sometimes executing this process helped, in others there was no resolution of the
problem:

Check which of the following potential workaround have a positive impact

a) After making a change that has not been propogated, run the following three concurrent
requests, in order and confirm if this provides a workaround to the problem

a. "Sync responsibility role data into the WF table."

b. "Synchronize WF LOCAL tables"
 c. "Workflow Directory Services User/Role Validation" with the 'Fix dangling user/roles' and 'Add
missing user/role assignments' set to "yes"

THESE two options WERE NOT TESTED as the access is not available to us:

b) In the "Functional Administrator" responsibility->core Services->Caching Framework->Global

Configuration

Use the "Clear All" cache option to manually invalidate the Java Cache

c) Does bouncing Apache solve the problem

BELIEVE THIS COULD ONLY BE DONE IN CONJUNCTION WITH PLANNED OUTAGE:

5. Gather additional data to help identify the failure point
These steps will ensure the appropriate data is gathered to track the error whilst it is occurring.

a) Release 11i only Enable Apache debug level logging by following note 249669.1 "How To Collect
Apache and Jserv Debugging Details For Applications 11i" In particular the jserv.log logging should be
enabled.

b) Release11i only Shutdown Apache and archive the following log files

The following files from the $IAS_ORACLE_HOME/Apache/Apache/logs directory

i) access_log.<ID Number>
ii) error_log
The following files from the $IAS_ORACLE_HOME/Apache/Jserv/logs directory
iii) mod_jserv.log
iv) jserv.log
v) All files from the $IAS_ORACLE_HOME/Apache/Jserv/logs/jvm directory
These will have filenames of the form <JVM GroupName>.<JVM Instance>.stdout|stderr
For example OACoreGroup.0.stderr
vi) $COMMON_TOP/rgf/<Instance>_<Hostname>/javacache.log

c) Run the following Unix script to gather data from the operating system with the environment still
shutdown.

This will allow us to confirm that there are no rogue processes still running and all the tcp ports are free
#####################################################################
### START OF SCRIPT ###
#####################################################################
LOG_FILE="OS_`hostname`_`date '+%m%d%y_%H%M'`.txt"
echo "**********************************************" >>$LOG_FILE
echo "Run started " >> $LOG_FILE
date >>$LOG_FILE
echo "Running as `id`" >> $LOG_FILE
echo "**********************************************" >>$LOG_FILE
echo "uname -a" >>$LOG_FILE
uname -a >>$LOG_FILE
echo "++++++++++++++++++++++++++++++++++++++++++++++++" >> $LOG_FILE
 echo "ulimit -a" >>$LOG_FILE
ulimit -a >>$LOG_FILE
echo "++++++++++++++++++++++++++++++++++++++++++++++++" >> $LOG_FILE
echo "netstat -a">> $LOG_FILE
netstat -a >> $LOG_FILE
echo "++++++++++++++++++++++++++++++++++++++++++++++++" >> $LOG_FILE
echo "ps -e -o user,pid,ppid,s,pcpu,pmem,vsz,rss,stime,time,args" >> $LOG_FILE
ps -e -o user,pid,ppid,s,pcpu,pmem,vsz,rss,stime,time,args >>$LOG_FILE
echo "++++++++++++++++++++++++++++++++++++++++++++++++" >> $LOG_FILE
echo "Run completed " >> $LOG_FILE
date >> $LOG_FILE
#####################################################################
### END OF SCRIPT ###
#####################################################################

d) Release 11i only Startup Apache

e) In OAM navigate to Workflow Manager

Service Components--> select "Workflow Java Deferred Agent Listener" then click "Edit" button
Click "Next" button --> Change "Log Level" to STATEMENT
Click "Finish" button
Restart the "Workflow Java Deferred Agent Listener" for the change to take effect

f) In OAM navigate to Workflow Manager

Service Components--> select "Workflow Agent Listener Service" on the right hand side of the screen
In the next screen it will list three Service Instances
Select "Workflow Agent Listener Service" and stop the service
Select "Workflow Agent Listener Service" then click "Edit" button
Select the "Edit Service Parameters" button
Change the value
SVC_CONTAINER_LOG_LEVEL=4
to be
SVC_CONTAINER_LOG_LEVEL=1
Click OK to save this change
Click Save button
Select "Workflow Agent Listener Service" and start the service

The resulting log files are located in the $COMMON_TOP/admin/log/<instance>_<hostname> directory.

Look for files of the filename format FNDCPGSC<ID Number>.txt after reproducing the issue and upload
the updated files

g) Reproduce the problem then provide the following information:-

h) Rerun the Unix script from step (c) above

ii) Output from the following SQL

REM
REM START OF SQL
REM
REM Check entries for WF_JAVA_DEFERRED, WF_JAVA_ERROR and WF_CONTROL queues
REM
set echo on
set timing on
set feedback on
set long 10000
set pagesize 132
set linesize 105
col queue_name form a17
col state form a15
col how_many form 9999999999
--
select
'wf_java_deferred' queue_name,
decode(state,
0,'0 (Ready)',
1,'1 (Delayed)',
2, '2 (Retained)',
3, '3 (Exception)',
to_char(state|| ' (Other)')) state,
count(*) how_Many
from wf_java_deferred
group by state
UNION ALL
select
'wf_java_error' queue_name,
decode(state,
0,'0 (Ready)',
1,'1 (Delayed)',
2, '2 (Retained)',
3, '3 (Exception)',
to_char(state|| ' (Other)')) state,
count(*) how_Many
from wf_java_error
group by state
UNION ALL
select
'wf_control' queue_name,
decode(state,
0,'0 (Ready)',
1,'1 (Delayed)',
2, '2 (Retained)',
3, '3 (Exception)',
to_char(state|| ' (Other)')) state,
count(*) how_Many
from wf_control
group by state
/
REM
REM Check the difference between enqueue and dequeue time.
REM
REM This script queries the last 24 hours.
REM
col corr_id form a60
col retry_count form 999
col msg_state form a12
col enq_time form a18
col deq_time form a18
col Elapsed_Sec form 999999999
select 'wf_java_deferred' queue_name,
retry_count retrys,
msg_state,
to_char (a.enq_time, 'DD-MON-RR HH24:MI:SS') enq_time,
to_char (a.deq_time, 'DD-MON-RR HH24:MI:SS') deq_time,
 to_char ((nvl (a.deq_time, sysdate) - a.enq_time) * 24*60*60, '999999.99')
Elapsed_Sec,
corr_id
from applsys.aq$wf_java_deferred a
where a.enq_time >= sysdate - (24/24) -- arbitrary 24 hours
UNION ALL
select 'wf_java_error' queue_name,
retry_count retrys,
msg_state,
to_char (a.enq_time, 'DD-MON-RR HH24:MI:SS') enq_time,
to_char (a.deq_time, 'DD-MON-RR HH24:MI:SS') deq_time,
to_char ((nvl (a.deq_time, sysdate) - a.enq_time) * 24*60*60, '999999.99')
Elapsed_Sec,
corr_id
from applsys.aq$wf_java_error a
where a.enq_time >= sysdate - (24/24) -- arbitrary 24 hours
UNION ALL
select 'wf_control' queue_name,
retry_count retrys,
msg_state,
to_char (a.enq_time, 'DD-MON-RR HH24:MI:SS') enq_time,
to_char (a.deq_time, 'DD-MON-RR HH24:MI:SS') deq_time,
to_char ((nvl (a.deq_time, sysdate) - a.enq_time) * 24*60*60, '999999.99')
Elapsed_Sec,
corr_id
from applsys.aq$wf_control a
where a.enq_time >= sysdate - (24/24) -- arbitrary 24 hours
order by 5,6,7
/
REM
REM END OF SQL
REM

iii) The output from the following SQL to gather the STATEMENT level debug information

REM
REM START OF SQL
REM
set echo on
set timing on
set feedback on
set long 10000
set pagesize 132
set linesize 100
col time_stamp form a17
col user_name form a12
col log_level form a10
col module form a57 wrap
col message_text form a98 wrap
col session_id form 999999999999
REM
REM Show all messages for the last 1 hour
REM
select to_char(m.timestamp, 'DD-MM-RR HH24:MI:SS') time_stamp,
u.user_name,
l.meaning log_level,
m.module,
m.message_text
from fnd_log_messages m, fnd_user u, fnd_lookup_values l
where m.user_id = u.user_id
and l.lookup_type = 'AFLOG_LEVELS'
and l.lookup_code = m.log_level
 and m.timestamp >= sysdate - (1/24) -- arbitary show all messages for last 1
hour
order by m.timestamp, m.log_sequence
/
REM
REM END OF SQL
REM

iv) Collect Java Thread Dump for the OACoreGroup JVMs

For each of the OACoreGroup JVMs, run the following operating system command

kill -3 <PID>

If you need help to identify the correct JVM process to run the kill command for, you can use the
following procedure

Release 11i

cd $IAS_ORACLE_HOME/Apache/Jserv/logs/jvm
/sbin/fuser OACoreGroup.*.stdout
Release 12

ps -ef | grep CLIENT | grep oacore

This should return the process id (PID) for the OACoreGroup JVM(s)

v) Output from the following SQL which collects data from the Queue tables

REM
REM START OF SQL
REM
set echo on
set timing on
set feedback on
set long 10000
set pagesize 1000
set linesize 120
col CORR_ID form a45
col MSG_STATE form a13
col ENQ_TIME form a18
col DEQ_TIME form a18
col RETRY_COUNT form 99999
col EXPIRATION_REASON form a19
col USER_DATA form a10000
REM
select
CORR_ID,
MSG_STATE,
to_char(ENQ_TIME,'DD-MON-RR HH24:MI:SS') enq_time,
to_char(DEQ_TIME,'DD-MON-RR HH24:MI:SS') deq_time,
RETRY_COUNT,
EXPIRATION_REASON,
 USER_DATA
from applsys.aq$wf_java_deferred
order by ENQ_TIME, DEQ_TIME
/
REM
REM END OF SQL
REM

vi) Apache log files as listed below

Release 11i
The following files from the $IAS_ORACLE_HOME/Apache/Apache/logs directory
i) access_log.<ID Number>
ii) error_log
The following files from the $IAS_ORACLE_HOME/Apache/Jserv/logs directory
iii) mod_jserv.log
iv) jserv.log
v) All files from the $IAS_ORACLE_HOME/Apache/Jserv/logs/jvm directory
These will have filenames of the form <JVM GroupName>.<JVM Instance>.stdout|stderr
For example OACoreGroup.0.stderr
vi) $COMMON_TOP/rgf/<Instance>_<Hostname>/javacache.log