All about ORACLE EBS Concurrent

Manager –Part 1
Here is function of each type of concurrent managers.

1. The Internal Concurrent Manager (ICM) starts, sets the number of active
processes, monitors, and terminates all other concurrent processes through
requests made to the Service Manager, including restarting any failed processes.
The ICM also starts and stops, and restarts the Service Manager for each node.
The ICM will perform process migration during an instance or node failure.
The ICM will be started one node in RAC and will be started by Internal Monitor in
case of node failure.
2. Internal Monitor (FNDIMON process) - Communicates with the Internal Concurrent
Manager. The Internal Monitor (IM) monitors the Internal Concurrent Manager, and
restarts any failed ICM on the local node. During a node failure in a PCP
environment the IM will restart the ICM on a surviving node (multiple ICM's may
be started on multiple nodes, but only the first ICM started will eventually
remain active, all others will gracefully terminate). There should be an
Internal Monitor defined on each node
3. Service Manager (FNDSM process) - Communicates with the Internal Concurrent
Manager, Concurrent Manager, and non-Manager Service processes.
4. Standard Manager (FNDLIBR process) - Communicates with the Service Manager and
any client application process. The Standard Manager is a worker process that
initiates, and executes client requests on behalf of Applications batch, and OLTP
clients.
5. Transaction Manager - Communicates with the Service Manager, and any user
process initiated on behalf of a Forms, or Standard Manager Request. See Note:
240818.1 regarding Transaction Manager communication and setup requirements
for
RAC.
Major Scripts $FND_TOP/sql Scripts

• afimchk.sql
Tells the status of the ICM
• afcmstat.sql
Lists active manager processes
• afrqrun.sql
Lists all the running, waiting and Terminating requests
• afrqwait.sql
Lists requests that are constrained and waiting for the ICM
to release them.
• afrqscm.sql
Prints log file name of managers that can run a given
request.
afcmcreq.sql
Prints the log file name of the manager that processed
the request
• afrqstat.sql
Summary of completed concurrent requests grouped by
completion status and execution type.
• afimlock.sql
Lists locks that the ICM is waiting to get
• afcmrrq.sql

Lists managers that currently are running a request

Major Concurrent Managers Tables Tables

o FND_CONCURRENT_PROCESSES
o FND_CONCURRENT_PROCESSORS
o FND_CONCURRENT_PROGRAMS
o FND_CONCURRENT_PROGRAMS_TL
o FND_CONCURRENT_PROGRAM_SERIAL
o FND_CONCURRENT_QUEUES
o FND_CONCURRENT_QUEUES_TL
o FND_CONCURRENT_QUEUE_CONTENT
o FND_CONCURRENT_QUEUE_PARAMS
o FND_CONCURRENT_QUEUE_SIZE
o FND_CONCURRENT_REQUESTS
How to define custom CM
Overnight: Only allow selected programs to run overnight

Create a request type Overnight

Create a manager that runs only overnight (work shifts)
Disallow the request type from standard / Allow in the overnight manager
Assign the program you wish to run overnight to the Overnight request type

1. Navigate to Concurrent / Manager / Define.

2.Manager Field: Custom Manager.

3.Short Name: CUSTOM.
4.Type: Concurrent Manager.
5.Program Library: FNDLIBR.
6.Enter desired cache.
7.Work Shifts: Standard.
8.Enter number of Processes.
9.Provide Specialization Rules (you can include or exclude
program, id, user, types or combination).
10. Save.
Navigate to Concurrent / Manager / Administer.
Activate the Custom Manager.
PCP (Parallel Concurrent Processing) in a RAC/NON RAC Environment.

The following query from sqlplus will show the current database Instance all the active
concurrent managers (queues) are connected to:

select concurrent_queue_name, p.node_name, db_instance

from fnd_concurrent_processes p, fnd_concurrent_queues q, fnd_cp_services s
where p.queue_application_id = q.application_id
and p.concurrent_queue_id = q.concurrent_queue_id
and q.manager_type = s.service_id
and p.process_status_code = 'A';

In a PCP Implementation we have an Internal Monitor running on each Concurrent

Manager Node, Internal Monitor is like a local police monitoring the process on the
respective concurrent tier. The job of IM to restart Internal Concurrent Manager in
case of a lost of database connection. IM enabled in Autoconfig XML File
"s_cp_reviver". Review Metalink Note 466752.1 for more information on FND
Reviver.

If you turn Profile Option "Concurrent: PCP Instance Check" to ON then when your
Database goes down then the concurrent managers connecting to the database node
which went down will failover to the failover node defined at the concurrent manager
level
 2. If you turn Profile Option "Concurrent: PCP Instance Check" to OFF then when
your Database goes down then the Reviver Process on the Middle Tier tries to restart
the Internal Concurrent Manager by connecting to the next surviving database node

Similarly if your Concurrent Tier goes down the concurrent manager should
failover after 2-3 PMON Cycles, however if you notice that managers didnt
failover then Login to OAM and make the apps tier which went down "Offline"

Login -> OAM -> Click on the Apps Tier which went down -> Turn it "Offline"

6. The Internal Concurrent Manager (ICM) starts, sets the number of active
processes, monitors, and terminates all other concurrent processes through
requests made to the Service Manager, including restarting any failed processes.
The ICM also starts and stops, and restarts the Service Manager for each node.
The ICM will perform process migration during an instance or node failure.
7. The Standard Manager is a worker process, that initiates, and executes client
requests on behalf of Applications batch, and OLTP clients.

8. Transaction Manager - Communicates with the Service Manager, and any user
process initiated on behalf of a Forms, or Standard Manager request. See Note:
240818.1 regarding Transaction Manager communication and setup requirements for
RAC.

How to Set Up Parallel Concurrent Processing (PCP) in Apps 11i?

1. Ensure all nodes have been defined correctly in each tnsnames.ora file on each of the nodes.
2. Ensure the Apps 806 listener up on each node.
3. Ensure the Internal Monitor manager up on each node.
4. Configuring Concurrent Managers in APPS:
5. Log into Oracle Applications as SYSADMIN with System Administrator responsibility.
6. Navigate to Install\Nodes and register the names of all the potential nodes in the PCP setu
7. Navigate to Concurrent\Manager\Define and set up the primary node for your Internal
Concurrent Manager. This is the node on which the ICM prefers to run, when the node is
available.
8. Navigate to Concurrent\Manager\Define, query the Internal Monitor Manager and make sure
the Enabled box is checked.
9. Make sure there is Internal Monitor Manager defined for each node.
10. Finally, set up the primary and secondary node names for all the concurrent managers
according to the desired distribution, via Concurrent\Manager\Define Form.
Concurrent Processing Tuning

Best Practices for Performance for Concurrent Managers in E-Business Suite 1057802.1

Provide the best practices to achieve better performance for concurrent manager in Oracle E-Business
Suite.

Sleep Seconds - is the number of seconds your Concurrent manager waits between checking the list of
pending concurrent request

Set the sleep time to be very brief during periods when the number of requests submitted is expected to
be high. Otherwise set the sleep time to a high number (e.g. 2 minutes) . This avoids constant polls to
check for new requests.
Increase the cache size (number of requests cached) to at least twice the number of target
processes.

Create specialized concurrent managers to dedicate certain process either short or long running
programs to avoid queue length.

To maximize throughput consider reducing the sleep time of the Conflict Resolution
Manager (CRM). The default value is 60 seconds. You can consider setting to 5 or 10
seconds.

Ensure "Purge Concurrent Request and/or Manager Data, FNDCPPUR," is run at regular
intervals with "Entity" parameter as "ALL". A high number of records in FND_CONCURRENT
tables can degrade the performance.

Ensure that the log/out files are removed from the locations shown below as you run "Purge
Concurrent Request and/or Manager Data program".

$APPLCSF/$APPLLOG
$APPLCSF/$APPLOUT
Defragment the tables periodically to reclaim unused space / improve performance

FND_CONCURRENT_REQUESTS
FND_CONCURRENT_PROCESSES
FND_CRM_HISTORY
FND_ENV_CONTEXT
FND_TEMP_FILES
TUNING CONCURRENT MANAGER

There are 5 ways to faster your CM

1.PMON cycle, queue size, and sleep time.

2.Purging Concurrent Requests
3.Adjusting the Concurrent Manager Cache Size
4.Analyzing Oracle Apps Dictionary Tables for High Performance
5. Number of Standard Managers

1.PMON cycle, queue size, and sleep time.

The ICM performance is affected by the three important Oracle parameters

PMON cycle

This is the number of sleep cycles that the ICM waits between the time it checks for concurrent
managers failures, are having problems with abnormal terminations.

Queue Size

The queue size is the number of PMON cycles that the ICM waits between checking for disabled or
new concurrent managers. The default for queue size of 1 PMON cycle should be used.

Sleep Time
The sleep time parameter indicates the seconds that the ICM should wait between checking for
requests that are waiting to run. The default sleep time is 60, but you can lower this number if you
see you have a lot of request waiting (Pending/Normal). However, reducing this number to a very low
value many cause excessive cpu utilization.

2.Purging Concurrent Requests

One important area of Concurrent Manager tuning is monitoring the space usage for the subsets
within each concurrent manager.
When the space in FND_CONCURRENT_PROCESSES and FND_CONCURRENT_REQUESTS exceed 50K,
you can start to experience serious
performance problems within your Oracle Applications. When you experience these space problems, a
specific request called “Purge Concurrent Requests And/Or Manager Data” should be scheduled to run
on a regular basis. This request can be configured to purge the request data from the FND tables as
well as the log files and output files on accumulate on
disk.

3.Adjusting the Concurrent Manager Cache Size

Concurrent manager performance can also be enhanced by increasing the manager cache size
to be at lease twice the number of target processes. The
cache size specifies the number of requests that will be cached each time the concurrent manager
reads from the FND_CONCURRENT_REQUESTS table. Increasing the cache size will boost the
throughput of the managers by attempting to avoid sleep time.

4. Gather stats /Analyze AOL check FND Tables

5. Number of Standard Managers

Some of environments are copies of other environments, and you may find that the number of
Standard Concurrent Managers are just 5. You can increase this to say 10 or 15, this will help the
pending requests queue is not getting too long & the Conflict Resolution Manager will have less load.
 To maximize throughput for jobs which spawn parallel workers (i.e. Auto Invoice, Payroll),
consider reducing the sleep time of the Conflict Resolution Manager (CRM)
• Default is 60s, consider 5 or 10 seconds

Transaction Managers - TMs

– Used for synchronous online processing(ex:Inventory Transactions)
– Ensure enough TMs exist to service the request load
• Set the profile “Concurrent:Wait for Available TM” to 1 (second).
– Set the sleep time on the TMs to a high number (e.g. 10 minutes)
• Avoids constant polls to check for shutdown requests

Concurrent Processing Tuning Assistant ( CPA)

CPA helps to diagnose and report historical data and assist in providing throughput of concurrent
managers

There are two ways to run the Tuning Assistant:

Against the Oracle Application Object Library tables

Against the Concurrent Processing Tuning Assistant repository

Running against AOL/FND tables gives most up to date information about the concurrent managers

One method for improving Tuning Assistant performance in retrieving data is to periodically use
the SQL language ANALYZE command to compute statistics on tables that the Tuning Assistant
accesses. These tables include:

FND_APPLICATION_TL
FND_LOOKUP_VALUES
FND_CONCURRENT_PROCESSES
FND_CONCURRENT_PROGRAMS
FND_CONCURRENT_PROGRAMS_TL
FND_CONCURRENT_QUEUES
FND_CONCURRENT_QUEUES_TL
FND_CONCURRENT_REQUESTS
FND_PRINTER_STYLES
FND_PRINTER_STYLES_TL
FND_PRODUCT_INSTALLATIONS
FND_USER
FND_CONC_PP_ACTIONS (for Apps 11.x)
FND_CONC_RELEASE_CLASSES (for Apps 11.x)
 FND_CONC_RELEASE_CLASSES_TL (for Apps 11.x)

CM diagnostic

1. Routine AFPEIM Encountered an Error While Starting Concurrent Manager on

Cloned- 230121.1

The tnsnames.ora & listener.ora on the target machine does not

contain the FNDSM_<target machine hoshname>_<SID> connection
strings,
but the Profile > System Option "Concurrent: GSM Enabled =Y"
and FND_NODES table is populated with the registered node_name.

2. Clean Source node or network toplogy after clone

API FND_CONC_CLONE.Setup_CLEAN ( DB Tier)

Execute autoconfig.sh on both tier

3. How to start GSM and Debug, it must be on 11.5.10 onwards

Login to OAM and Navigate to:

Site Map > Generic Services > Status Overview > View All > select Debug Service > select
Start from LOV > click Go

Enable GSM Debug Service logging:

Site Map > Generic Services > Status Overview > View All > select Debug Service
> View Details > View Status > click 'Set Debug On' Button -> OK

4 How to Reduce the Wait Time between Spawned Concurrent Requests [ID
270458.1]
 Set the cache size of the concurrent manger to the number of requests
in the request set, and then it will queue them all up to run at once.

4. How to find Concurrent manager node

select node_name from apps.fnd_nodes where support_cp='Y';

Concurrent Manager Processes Taking Lots of CPU- 114380.1

Check which manager from OS process and check which manager . Check how
many process and sleep for that particular manager.
if you define for example 10 processes, and specify a sleep time of 3,
this means that each of these 10 processes will verify every 3 seconds
if there is anything to process. This can cause serious performance
degradation. In this case, it is better to increase the sleep time
(the default is 30 seconds). Lower the number of processes, or
increase the sleep time.

5. How To Enable Debugging For Concurrent Managers? [ID 888275.1]

Enable DIAG for the ICM, by modifying script

$COMMON_TOP/admin/scripts/<SID>_<hostname>/adcmctl.sh
as follows :
Replace line :
CONTEXT_PARAMS="diag=N wait=N"
With :
CONTEXT_PARAMS="diag=Y wait=N"
Save file.

2. Enable the Debug Service if not yet enabled :

- Login with System Administrator responsibility
- Navigate to Concurrent / Manager /Define
- Query manager with name "Debug Service"
- tick the "Enabled" checkbox if not yet ticked
- save

3. Stop and restart the concurrent managers

SLEEP holds the number of seconds that the manager should wait between checks for new
requests.
MGRNAME is the name of the manager for locking and log purposes. So you can find if this
is icm(Internal Concurrent Manager), sm(Service Manager)
RESTART is set to N if the manager should not restart itself after a crash. Otherwise, it is an
integer number of minutes. The manager will attempt a restart after an abnormal termination if
the past invocation lasted for at least RESTART minutes.
PMON is the duration of time between process monitor checks (checks for failed workers).
References
Concurrent Manager Processes Taking Lots of CPU- 114380.1

Concurrent Manager Setup and Configuration Requirements in an 11i RAC Environment [ID
241370.1]

Best Practices for Performance for Concurrent Managers in E-Business Suite 1057802.1