2012 Feb EBS Concurrent Processing Troubleshooting

1 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
My Oracle Support – Advisor Webcast Program

• EBS Concurrent Processing Troubleshooting
Techniques
Wednesday, 15 February, 2012 Upcoming Webcasts in EBS Technology area :
06:00 PM CET (GMT +2)
o March 2012 - EBS Patching Overview & Troubleshooting
Teleconference Access:
North America:
1866 966 9439
International (UK): Do you have any requests for future EBS - Technology
+44 (0) 1452 562 665 Webcast Events? Please email your suggestions to
ruediger.ziegler@oracle.com,subject: Topics of Interest.
Conference ID : 39485824

AGENDA
• Presentation and Demo

approximately 60 - 75 minutes
• Q&A Session
maximum 15 minutes
Web attendees can ask questions via Q&A panel

Phone attendees can ask questions via Q&A panel or phone

2 Who to ask?
3 Send
your question
Q&A panel
1
type your question here

your question pops-up here

ATTENTION – AUDIO INFORMATION
• Voice streaming/Audio broadcast is available. For full audio access,
• please join the telephone conference.
Teleconference Connect details:
1. Conference ID: 39485824

2. International dial in: +44 (0) 1452 562 665
3. List with national toll free numbers is available in Note 1148600.1
Note:
You can view this info anytime using WebEx menu from your WebEx-Session :
 Select ‘Communicate’ --> ‘Join Teleconference’.

Safe Harbor Statement
The following is intended to outline our general product direction.

It is intended for information purposes only, and may not be
incorporated into any contract. It is not a commitment to deliver any
material, code, or functionality, and should not be relied upon in making
purchasing decisions. The development, release, and timing of any
features or functionality described for Oracle’s products remains at the
sole discretion of Oracle.

EBS Concurrent Processing Troubleshooting Techniques
Presenters: Kandasamy Nainamalai, Kalaivani Murugesan and Rakesh Chithuluri
Q&A Panel Experts : Amira Abdelkader , Vishwanatha Rao,Thiagarajan, Martin Fritz

Program Agenda
• Concurrent Manager - Troubleshooting.
1. Manager Not Running (inactive NoManager)
2. How to run cmclean(Pending Requests)
3. ICM Running , Other Managers are down
4. Only OPP & Workflow services Down
5. DBMS_LOCK Error - ICM Log
6. After CLONE Managers Down/FNDSM Errors/CONC-SM TNS FAIL
7. Managers down - Status "System Hold, Fix Manager"
8. Managers down - "Target node/queue unavailable“
9. How to Enable diag=y
• Concurrent Request – Troubleshooting

1. ORA-20100 Error Troubleshooting
2. Rep-3000 Error troubleshooting
3. Unable to View output/log files
4. How to Collect a Trace and Tkprof
• XML Publisher Reports - Troubleshooting

1. XML Publisher Introduction
2. How to Obtain OPP Log
3. How to enable & collect XDO Log
4. BI Publisher - Troubleshooting

Concurrent Manager - Troubleshooting.
1. Manager Not Running (inactive NoManager)
2. How to run cmclean(Pending Requests)

Concurrent Manager – Troubleshooting
Concurrent Request Stays “Inactive NoManager”

Manager Not Running (Inactive NoManager)
• System Administrator Responsibility -> Concurrent -> Manager ->

• Administer Form -> Verify the Concurrent Managers Status
• Login into Backend CP Node
Verify ps –ef | FNDLIBR process
• Start Concurrent Manager (adcmctl.sh)
• Ensure All Managers are running now.

How to Run CMCLEAN
• Ensure Concurrent Managers are down.

• Collect cmclean.sql script from below document.
Note.134007.1 Ext/Pub CMCLEAN.SQL - Non Destructive Script to Clean Concurrent Manager Tables
• Execute the cmclean.sql script.

• Start the Concurrent Manager (adcmctl.sh)

Concurrent Manager Troubleshooting
Demonstration – part1

Concurrent Manager - Troubleshooting.
1. ICM Running , Other Managers are down

2. Only OPP & Workflow services Down
3. DBMS_LOCK Error - ICM Log
4. After CLONE Managers Down/
FNDSM Errors/CONC-SM TNS FAIL
5. Managers down - Status "System Hold, Fix Manager"
6. Managers down - "Target node/queue unavailable“
7. How to Enable diag=y

ICM Running , Other Managers are down

Solution :
• 1. Check listener Status (adalnctl.sh status)
• 2. Stop Concurrent Manager (adcmctl.sh stop)
• 3. Start Listener (adalnctl.sh start)
• 4. Start Concurrent Manager (adcmctl.sh start)

Concurrent Manager Troubleshooting
Demonstration – part2

Only OPP & Workflow services Down

Solution :
• Ensure Concurrent:GSM Enabled profile is set to ‘Y’
• Verify Service Manager status in Administer Form.
• Verify Service Manager Definition.
• Ensure FNDSM Entries available in FND_CONCURRENT_QUEUES Table
• FNDSM entry should be correct in Tnsnames.ora file and tnsping

FNDSM_hostname should work fine.
• Then Bounce the Services.

DBMS_LOCK Error – Internal Concurrent Manager Log

Solution :
• Ensure Concurrent Managers are down.

Ensure no FNDLIBR process running
ps –ef|grep FNDLIBR
• Run Cmclea.sql ( commit is mandatory)
• Then Start the services – adstrtal.sh apps/apps
• Now you can see Managers Up and running.

After CLONE Managers Down
• After CLONE Concurrent Manager not Starting
ICM logfile shows the following errors:
Error#1:
CONC-SM TNS FAIL

Call to StopProcess failed for FNDCPGSC
CONC-SM TNS FAIL
Call to PingProcess failed for FNDOPP
Error#2:
Could not contact Service Manager FNDSM_DHSRVMDS518_SDBA. The TNS alias could not be located, the listener process
on DHSRVMDS518 could not be contacted, or the listener failed to spawn the Service Manager process.
Error#3:
Cause: cleanup_node failed due to ORA-01427: single-row subquery returns more than one row
ORA-06512: at "APPS.FND_CP_FNDSM", line 29
ORA-06512: at line 1.

Solution:
• Ensure Database is running

• Ensure Managers are down.
• Connect SQLPLUS as APPS user and run the following :
EXEC FND_CONC_CLONE.SETUP_CLEAN;
COMMIT;
EXIT;
• Run Autoconfig on DB node and then Apps Node
• Run $FND_TOP/patch/115/sql/afdcm037.sql script
• Start Services (adstrtal.sh)

References:
• Concurrent Processing - After Cloning all the Concurrent Managers do not start for the cloned
Instance (Doc ID 555081.1)
• ICM log file shows 'CONC-SM TNS FAIL', 'Call to PingProcess failed', and/or 'Call to StopProcess
failed' for FNDCPGSC/FNDOPP (Doc ID 1312632.1)

Managers down - Status "System Hold, Fix Manager"

Solution:
• Ensure Concurrent :GSM Enabled profile is set to ‘Y’

• Run $FND_TOP/patch/115/sql/afdcm037.sql
• Go to $FND_TOP/bin
adrelink.sh force=y “fnd FNDLIBR”
adrelink.sh force=y “fnd FNDSM”
adrelink.sh force=y “fnd FNDFS”
adrelink.sh force=y “fnd FNDCRM”
• Run cmclean.sql
• Start Application Service (adstrtal.sh)

Managers down – “Target node/queue unavailable”

Solution:
Ensure Database is Running and Middle Tier Services are down.
o Connect SQLPLUS as APPS user and run the following :
EXEC FND_CONC_CLONE.SETUP_CLEAN;
COMMIT;
EXIT;
o Run AutoConfig on all tiers, firstly on the DB tier and then the APPS tiers and webtier to repopulate the required system
tables
o Run the CMCLEAN.SQL script from the referenced note below (don't forget to commit).
Note.134007.1 – ‘CMCLEAN.SQL - Non Destructive Script to Clean Concurrent Manager Tables‘
o Start the middle tier services including your concurrent manager.
o Retest the issue.

How to Enable diag=y
• Stop Concurrent Manager

• Take backup of adcmctl.sh
• Edit adcmctl.sh to append diag=y
$FND_TOP/bin/startmgr sysmgr="$unpw@$DB_NAME" $@ mgrname=$MGR_NAME diag=y
else
$FND_TOP/bin/startmgr sysmgr="$unpw@$DB_NAME" mgrname=$MGR_NAME diag=y
fi;
• Start Concurrent Manager

Concurrent Request – Troubleshooting
1. ORA-20100 Error Troubleshooting

2. Rep-3000 Error troubleshooting
3. Unable to View output/log files
4. How to Collect a Trace and Tkprof

Concurrent Request Troubleshooting
Demonstration – Part1

ORA-20100 Error Troubleshooting

Solution:
• Ensure Tmp Directory is valid. ($APPLPTMP)
• Tmp Directory should have enough space.
• Applmgr user does not have valid permission (Read/Write/Execute)
• Different instances use same Tmp Directory
• UTL_FILE_DIR Database Parameter does not have APPLPTMP Entry

Rep-3000 Error troubleshooting
Any Bitmap Reports Failing with following Error:
APPLLCSP Environment Variable set to :
XML_REPORTS_XENVIRONMENT is :
/u01/app/protoora/8.0.6/guicommon6/tk60/admin/Tk2Motif_UTF8.rgb
XENVIRONMENT is set to: /u01/app/protoora/8.0.6/guicommon6/tk60/admin/Tk2Motif_UTF8.rgb
Current NLS_LANG and NLS_NUMERIC_CHARACTERS Environment Variables are :

American_America.UTF8
''
REP-3000: Internal error starting Oracle Toolkit.

REP-3000: Internal error starting Oracle Toolkit.

Solution:
• Ensure an Xserver configured for CP server

• Ensure DISPLAY is set to valid Xserver
• Set DISPLAY in Xserver as root user and confirm Xclock is working
• Run xhost +
• From Backend CP Node Set DISPLAY as applmgr User
• Run Xclock from CP Node to verify connection between CP Node and Xserver
• Go to Front end and Verify DISPLAY Value
• If DISPLAY is incorrect in Front end then,
• Go to Oracle Application Manager and Change the s_display.
• Then Run Autoconfig
• Bounce Concurrent Manager.

Unable to View output/log files
Ensure RRA: Enabled profile is Set to ‘Yes’

Unable to View output/log files
• Ensure Listener is Running fine.

adalnctl.sh status

How to Collect a Trace and Tkprof
• Program Level Trace (Not Recommended)
Go to Concurrent Program -> Define form -> Query for Concurrent Program Enable Trace
• Request Level Trace (Recommended)
Ensure Concurrent: Allow Debugging profile is Set to ‘Yes’

On Submission Request Form -> Click on Debug Options Button
Select the Required Trace Level -> Click Ok.

Get Trace File Name
Run the following SQL to find out the Raw trace name and location for the concurrent program. The SQL prompts the user for the request id
prompt
accept request prompt 'Please enter the concurrent request id for the appropriate concurrent program:'
prompt
column traceid format a8

column tracename format a80
column user_concurrent_program_name format a40
column execname format a15
column enable_trace format a12
set lines 80
set pages 22
set head off
SELECT 'Request id: '||request_id ,

'Trace id: '||oracle_Process_id,
'Trace Flag: '||req.enable_trace,
'Trace Name:
'||dest.value||'/'||lower(dbnm.value)||'_ora_'||oracle_process_id||'.trc',
'Prog. Name: '||prog.user_concurrent_program_name,
'File Name: '||execname.execution_file_name|| execname.subroutine_name ,
'Status : '||decode(phase_code,'R','Running')
||'-'||decode(status_code,'R','Normal'),
'SID Serial: '||ses.sid||','|| ses.serial#,
'Module : '||ses.module
from fnd_concurrent_requests req, v$session ses, v$process proc,
v$parameter dest, v$parameter dbnm, fnd_concurrent_programs_vl prog,
fnd_executables execname
where req.request_id = &request
and req.oracle_process_id=proc.spid(+)
and proc.addr = ses.paddr(+)
and dest.name='user_dump_dest'
and dbnm.name='db_name'
and req.concurrent_program_id = prog.concurrent_program_id
and req.program_application_id = prog.application_id
--- and prog.application_id = execname.application_id
and prog.executable_application_id = execname.application_id
and prog.executable_id=execname.executable_id;

Get TkProf File
Syntax :
• $ tkprof <Trace Filename> <Tkprof Filename> explain=apps/password sort=‘(prsela,exeela,fchela)’

XML Publisher Reports
Introduction to XML Publisher Reports
OPP log file
XDO log file
Troubleshooting

XML Publisher - Troubleshooting
Demonstration

OPP Log File
Possible ways to obtain the OPP log file:
1. Directly from the file system based after identifying the corresponding OPP log file name using the following SQL
statement:
SELECT fcpp.concurrent_request_id req_id, fcp.node_name, fcp.logfile_name

FROM fnd_conc_pp_actions fcpp, fnd_concurrent_processes fcp
WHERE fcpp.processor_id = fcp.concurrent_process_id
AND fcpp.action_type = 6
AND fcpp.concurrent_request_id = &&request_id
Note: The SQL statement will only return data for those Concurrent Requests having a Layout specified (Upon
Completion... window).
2. Via the Forms application:

• 1.Login to the application as SYSADMIN
• 2. Responsibility: System Administrator
• 3. Function: Concurrent --> Manager --> Administration
• 4. Select the Output Post Processor
• 5. Click on the Processes button
• 6. Select the Concurrent Process which was active during the time that the request ran
• 7. Click on the Manager Log button to open the Output Post Processor log file

3. the Oracle Application Manager (OAM)
• 1. Login to the application as SYSADMIN
• 2. Responsibility: System Administration
• 3. Function: Oracle Applications Manager --> Concurrent Managers
• 4. Select the Output Post Processor Service and click on View Details
• 5. Click on View Processes
• 6. Select the Concurrent Process which was active during the time that the request ran
• 7. Click on the Log button to open the Output Post Processor log file

XML Publisher Core Engine Debug logs
The XDO log consist of complete debug information generated by XML Publisher Core engine. Follow the steps given
below to enable debug
1. Connect to the concurrent manager server as Application Owner(applmgr).

2. Create an $XDO_TOP/temp and an $XDO_TOP/resource directory.
3. Create an xdodebug.cfg file in the $XDO_TOP/resource directory, containing the following 2 lines:
LogLevel=STATEMENT
LogDir=[full XDO_TOP]/temp
4. Restart the concurrent managers in case of single step publishing method (not required for two step publishing method).
5. Reproduce the problem.
Note: The xdodebug.cfg file can also be created in the $AF_JRE_TOP/jre/lib OR $AF_JRE_TOP/lib directory.
With the above steps several debug files will be created under the $XDO_TOP/temp directory after reproducing the
Problem
xdo.log : XML Publisher Core engine debug log file

xdo_...xsl : XSL-FO version of the RTF template
xdo_...xml : XML data file
xdo_...fo : temporary file created by the FO Engine (only RTF / XSL templates)
xdo_...out : output file (may not exist in case of complete failure).

Troubleshooting BI Publisher Issues
Out Of Memory Error :
Symptoms: The Report completes with an Error The log file includes a message similar to the following:
+------------- 1) PUBLISH -------------+
Beginning post-processing of request 744134 on node R12P660 at 10-JAN-2012
14:50:47. Post-processing of request 744134 failed at 10-JAN-2012 14:51:15 with the
Error message: One or more post-processing actions failed. Consult the OPP service log for
details.
OR
'****Warning!!! Due to high volume of data, got out of memory exception...***
'****Please retry with scalable option or modify the Data template to run in scalable mode...***
OR
java.lang.OutOfMemoryError: Java heap space
Memory Issues can be at 2 levels:
1)Xml Data generation

2)Report Generation from XML data

Xml Data Generation
If the XML data generation is not proper, follow the steps given below
Steps:
Increase the memory from CP Definition
• Go to "System Administrator" responsibility.

• Navigate to Concurrent - > Program - > Define.
• Search for the report in the Concurrent Program (CP) window.
• Set the Options field to –Xmx2048M (Note: Don’t miss the dash before the X)
• Press Save
• Bounce the Concurrent Manager (optional).

Report Generation from XML data
XML Generation is fine but OPP fails with java.lang.OutOfMemoryError exception.
1.Check the current OPP JVM Heap size using following SQL query
SQL> select DEVELOPER_PARAMETERS from FND_CP_SERVICES where SERVICE_ID = (select

MANAGER_TYPE from FND_CONCURRENT_QUEUES where CONCURRENT_QUEUE_NAME = 'FNDCPOPP'
2. The default should be:

J:oracle.apps.fnd.cp.gsf.GSMServiceController:-mx512m
3. Increase the Heap Space per Process to 1024:

SQL> update FND_CP_SERVICES
set DEVELOPER_PARAMETERS = 'J:oracle.apps.fnd.cp.gsf.GSMServiceController:-mx1024m'
where SERVICE_ID =
(select MANAGER_TYPE
from FND_CONCURRENT_QUEUES
where CONCURRENT_QUEUE_NAME = 'FNDCPOPP');
SQL>Commit;
You need to restart the Concurrent Manager to make it effective. The maximim heap size should not
exceed 2048.

Bi Publisher Enhancement Parameters
Temporary Directory : Set this to a valid path, make sure that the temp directory is at least 5GB or 20x larger
than largest data file running. The application user should have read & write permissions to this directory
Navigation Path : XML Publisher Administrator responsibility

-> Choose Configuration tab
->General
->Temporary directory
There are few FoProcessing Parameters which can be fine tuned for better performance
Use XML Publisher's XSLT processor : Set this value to True

Enable scalable feature of XSLT processor : Set this value to False at Site level. For large reports set this value to true at data
definition level and template level.
Enable XSLT runtime optimization : Set this value to True

Specific Situations
There are situations for which the concurrent manager process (FNDLIBR) does produce an XML data file but the post-processing of
the request fails due to a different issue encountered, some of which the OPP does not even get invoked in the first place:
1. The Output Post Processor is not running or does not pick up the request:
... +------------- 1) PUBLISH -------------+
Unable to find an Output Post Processor service to post-process request 260006.
Check that the Output Post Processor service is running.
+--------------------------------------+ ...
The OPP log file cannot be determined using the SQL statement described earlier, as the OPP did not perform any action of the given
request.
Cause: OPP Manger is Down/Deactivated.
Action: Verify that the Output Post Processor is enabled and active
1. Profile Option "Concurrent: GSM Enabled" must be set to Y

2. Navigate to: Concurrent -> Managers -> Define
3. Query for Manager = 'Output Post Processor’ or Short Name = FNDCPOPP
4. Check the checkbox "Enable", save the changes.
5. Navigate to: Concurrent -> Managers -> Administer
6. Make sure the Output Post Processor Status is not 'Deactivated',Set Status to 'Activate' if necessary.
7. Bounce the concurrent manager for the changes to get reflected.

2. The Output Post Processor is running but has not picked up the request:
...
+------------- 1) PUBLISH -------------+
The Output Post-processor is running but has not picked up this request.
No further attempts will be made to post-process this request, and the request will be marked
with Warning status.
+--------------------------------------+
The OPP log file cannot be determined using the SQL statement described earlier, as the OPP did not perform
any action of the given request.
Cause: The number of concurrent requests that the Output Post Processor can handle in parallel depends upon the number of
Processes and the number of Threads Per Process. The default values are 2 Processes and 5 Threads Per Process so a total of
10 reports can be processed in parallel. In case there are other concurrent requests running which have already invoked the
OPP then it might happen that no additional requests can be picked up for a period of time.
Action:
1.Logon to Applications with "System Administrator" responsibility
2.Navigate to Concurrent -> Manager -> Define
3.Query for "Output Post Processor"
4.Click on "Work Shifts" and Increase the number of increase the number threads from 5 to 10
oracle.apps.fnd.cp.opp.OPPServiceThread:2:0:max_threads=5
to
oracle.apps.fnd.cp.opp.OPPServiceThread:2:0:max_threads=10
5.Bounce the concurrent Manager for the changes to get reflected.
Reference document for Process Vs Thread

Doc ID 1399454.1- Tuning Output Post Processor (OPP) to Improve Performance

3. The concurrent manager has timed out waiting for the Output Post-processor to finish this request.
...
+------------- 1) PUBLISH -------------+
The concurrent manager has timed out waiting for the Output Post-processor to finish this request.
Check that there are enough Output Post-processor service processes running.
More information may be found in the service process logfile.
+--------------------------------------+
The concurrent manager process has successfully invoked the OPP but it a timeout is encountered as the OPP takes too long to
complete the job
Cause: Once the Output Post Processor picks up the request, the BI Publisher engine is invoked to generate the final output file. The
time that this takes will depends on various elements such as size of the XML data file, complexity of the template, performance of
the server.By default a timeout will occur if it takes longer then 300 seconds (5 min.) for the BI Publisher engine to generate the
output file. The concurrent request will complete with status Warning and the request log file will contain above error Message.
Action Increase the value of the Concurrent:OPP Process Timeout as follows:
1.Logon to Applications with "System Administrator" responsibility

Navigate to Profile -> System
2.Query for "Concurrent:OPP Process Timeout".
Set the value to 10800 sec (3 hours)
3. Query for "Concurrent:OPP Response Timeout".
If the value of the Concurrent:OPP Response Timeout is 120 increase it to 240. (Value * 2)

Please provide the following information while raising SR with oracle support for XML Publisher Reports
1.Request log file of the failed concurrent program.
2.Out Post Processer log File

E-Business Suite Technology Group - Webcasts
 March 2012 : EBS Patching Overview & Troubleshooting
o EMEA-Session : 13th March 2012 at 10:00 CET (GMT +2)
o US-Session : 14th March 2012 at 18:00 CET (GMT +2)
For complete details on all upcoming Oracle Advisor Webcast Events, please see
Note 740966.1, Oracle Advisor Webcast Schedule.
For EBS Technology Specific Webcasts please check Note 1186338.1.
Do you have any requests for future ATG Advisor Webcast Events ?
Please email your suggestions to me :
ruediger.ziegler@oracle.com, subject: Topics of Interest.

Communities in My Oracle Support

EBS Technology Communities in MOS available
Following Communities are available in My Oracle Support
Oracle E-Business Suite

...
BI Publisher - Business Intelligence Products (not only EBS)
Core Concurrent Processing - Anything around Concurrent Processing and Concurrent Managers
Core Workflow - Any Workflow isse not only E-Business Suite
Diagnostic Tools - Anything around EBS Diagnostics
E-Business Customizations - Your Customizations
Installation - Fresh Install of the E-Business Suite
Patch Review EBS - Review of Patches around the E-Business Suite
Performance - EBS Performance
Upgrade - EBS Upgrade
User Produktivity Kit - User Productivity Kit (UPK) available for the E-Business Suite
Utilities - Utilities / generic EBS DBA issues
...
This is the current list for the E-Business Suite – Applications Technology Group

EBS ATG Product Information Center Note 1160285.1

Q&A

THANK YOU
for attending our Advisor Webcast!


2012 Feb EBS Concurrent Processing Troubleshooting

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

2012 Feb EBS Concurrent Processing Troubleshooting

Uploaded by

Copyright:

Available Formats

1 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.

My Oracle Support – Advisor Webcast Program

2 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.

• Presentation and Demo

Web attendees can ask questions via Q&A panel

3 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.

4 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.

5 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.

Teleconference Connect details:

1. Conference ID: 39485824

 Select ‘Communicate’ --> ‘Join Teleconference’.

6 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.

The following is intended to outline our general product direction.

7 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.

Presenters: Kandasamy Nainamalai, Kalaivani Murugesan and Rakesh Chithuluri

Q&A Panel Experts : Amira Abdelkader , Vishwanatha Rao,Thiagarajan, Martin Fritz

• Concurrent Request – Troubleshooting

• XML Publisher Reports - Troubleshooting

9 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.

1. Manager Not Running (inactive NoManager)

2. How to run cmclean(Pending Requests)

10 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.

11 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.

• System Administrator Responsibility -> Concurrent -> Manager ->

12 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.

• Ensure Concurrent Managers are down.

• Execute the cmclean.sql script.

13 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.

14 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.

1. ICM Running , Other Managers are down

15 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.

16 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.

• 1. Check listener Status (adalnctl.sh status)

• 2. Stop Concurrent Manager (adcmctl.sh stop)

• 3. Start Listener (adalnctl.sh start)

• 4. Start Concurrent Manager (adcmctl.sh start)

17 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.

18 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.

19 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.

• Ensure Concurrent:GSM Enabled profile is set to ‘Y’

• Verify Service Manager status in Administer Form.

• Verify Service Manager Definition.

• Ensure FNDSM Entries available in FND_CONCURRENT_QUEUES Table

• FNDSM entry should be correct in Tnsnames.ora file and tnsping

• Then Bounce the Services.

20 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.

21 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.

• Ensure Concurrent Managers are down.

• Run Cmclea.sql ( commit is mandatory)

• Then Start the services – adstrtal.sh apps/apps

• Now you can see Managers Up and running.

22 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.

ICM logfile shows the following errors:

CONC-SM TNS FAIL

23 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.

• Ensure Database is running

24 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.

25 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.

26 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.

• Ensure Concurrent :GSM Enabled profile is set to ‘Y’

27 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.

28 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.

Ensure Database is Running and Middle Tier Services are down.