You are on page 1of 22

ASM

OnDemand Concurrent Manager Analysis Programs
Author: Creation Date: Last Updated: Version: Approvals: Martin Fitzgerald Friday, June 12 2009 Monday, June 15 2009 1.0

Table of Contents
Concurrent Manager Analysis..................................................................................................3 Structure...............................................................................................................................3 Installation................................................................................................................................5 Parameters............................................................................................................................6 Actions.................................................................................................................................6 PLSQL tasks...................................................................................................................6 SQLPLUS tasks..............................................................................................................6 Execution Frequency...........................................................................................................9 Job Execution Reports............................................................................................................10 Common Concepts............................................................................................................10 Job Execution flow.......................................................................................................10 Execution Time vs Actual Runtime........................................................................11 Common Filter Parameters...........................................................................................11 Switching Data Store....................................................................................................12 OOD: Concurrent Program Execution Summary.............................................................13 Parameters.....................................................................................................................13 Purpose..........................................................................................................................13 OOD: Program execution counts over time......................................................................14 Parameters.....................................................................................................................14 Purpose..........................................................................................................................14 OOD: Detailed Program runtime history..........................................................................15 Parameters.....................................................................................................................15 Purpose..........................................................................................................................15 Manager Execution Reports...................................................................................................17 OOD: Show detailed Manager activity.............................................................................17 Parameters.....................................................................................................................17 Example Report............................................................................................................17 Report Heading section...........................................................................................18 Queue Heading section............................................................................................19 Worker Detail section..............................................................................................20 Manager summary section.......................................................................................21 Known Bugs...........................................................................................................................22

• OOD_CONCURRENT_QUEUE_CONTENT . There are 7 tables associated with the historical data store : • OOD_CONCURRENT_REQUESTS – This table holds much of the information originally contained in the FND_CONCURRENT_REQUESTS table. Structure In order to use these reports we rely on some tables to store the historical data. Using this data we can keep track on when worker targets were changed.Concurrent Manager Analysis Oracle OnDemand has a number of tools to analyze Concurrent Manager performance. • For the most part the historical data is summarized and it can be difficult getting the detail you need to compare points in time over vast stretches. To keep space usage low and make access of the table faster only a small subset of the columns are pulled over – mostly date columns however we also capture some of characteristics of the job to determine its unique characteristics. This table tries to get the real “run” times of a job and uses that to report against. As time goes by queues can be created and deleted. • OOD_CONC_REQ_SETS – It turns out the start and end times of a job are not the real story. • OOD_CONCURRENT_PROCESSES – This table holds the historical data for the FND_CONCURRENT_PROCESSES table. Without these tables existing none of the reports will work. • OOD_CONCURRENT_QUEUES – This table holds the historical data for the FND_CONCURRENT_QUEUES table. The manner in which these tables are populated is described later on. • OOD_CONCURRENT_QUEUE_SIZE. At present these reports are purely SQL*PLUS generated reports. By keeping a secondary copy we keep the connection between jobs that ran and the queue they ran in even if the queue no longer exists. Unfortunately most of these tools have 2 main drawbacks : • For the most part the tools are only accessible by OnDemand personnel and while that does help us to look at what happened in an environment it can leave a customer frustrated if they just want to understand what is happening on their environment. • . To resolve these issues and to make it easier in general to understand what is happening on any given environment we have provided a set of basic reports. the poll time and the workshifts for a queue.This table holds the historical data for the FND_CONCURRENT_QUEUE_SIZE table which is used to provide the number of workers.This table holds the historical data for the FND_CONCURRENT_QUEUE_CONTENT table which is used to define the includes and excludes of all the queues. These scripts represent the key scripts we use to help understand the performance of the Concurrent Manager subsystem separate from the performance of the jobs executed by the Concurrent Manager.

.• OOD_CONC_POP – This table holds the dates of when the historical tables were last updated. The basic idea is that the key tables used to hold live data will be accessible via the historical tables once populated and most scripts should only need to change the prefix from FND to OOD to view the information.

Alternatively you can login to the database via sqlplus as the APPS user and run the “ood_ctables” script o sqlplus apps/password @ood_ctables 700 Once these steps are completed the reports are installed and ready for use. For ease of use just past this into an SR when asking for a CRT: ______________________________________________________________________ To install the reports : • • • cd into the newly created cm_jobs directory o cd /autofs/upgrade/OHSPERF/OOD_CMJOBS/cm_jobs And run the install. ______________________________________________________________________ .ksh Once installed the reports cannot be used until the “OOD: Create and Update Historical CM tables” job is executed.Installation TO install follow the directions below.ksh script – you will need to provide the apps password when prompted. o ksh install. To do this log in as System Administrator and run the job from the SRS screen.

2. This program has the following characteristics: Parameters • Historical Days to Keep – This parameter is used to delete entries older than a certain period. An example output is below : o An initial Execution will look similar to the following : 14:40:39 14:40:45 14:40:45 14:40:47 14:40:48 14:40:49 Create Create Create Create Create Create table synonym index index index index : : : : : : OOD_CONCURRENT_REQUESTS OOD_CONCURRENT_REQUESTS ood_concurrent_requests_U1 on OOD_CONCURRENT_REQUESTS ood_concurrent_requests_n3 on OOD_CONCURRENT_REQUESTS ood_concurrent_requests_n6 on OOD_CONCURRENT_REQUESTS ood_concurrent_requests_n90 on OOD_CONCURRENT_REQUESTS . • Table sizes – we run a script to look at the table sizes. All activities for managing the space taken is handled by the Concurrent manager program “OOD: Create and Update Historical CM tables”. create_table – If the table does not currently exist it will create it. purge_table – Using the parameter provided on submission . This requires accurate statistics so it may not be accurate if stats were not gathered recently. create_indexes – We create a number of indexes (especially on the OOD_CONCURRENT_REQUESTS table ) based on the access patterns of the provided reports. analyze_table – If any of the previous actions resulted in a structure change ( a new table or new index) then the table is analyzed. SQLPLUS tasks To help manage how much space we use for these table we also report on the size of the tables and their contents.Managing the Historical Data The historical data of most systems using the reduced column set taken from the fnd_concurrent_requests table should be quite small. Each time the program is run we check to see if all expected indexes still exist 3. the program now deletes older entries. update_table – we then update the contents with the current data from the live tables 4. PLSQL tasks For each table explicitly listed above this program will perform the following actions in sequence: 1. 5. At present the default is 800 – which means we will keep just over 2 years worth of history Actions This program is split into 2 sections – a PLSQL procedure that manages the tables and a couple of sql statements that report on the contents.

• Table contents – Next we show how many records are in the largest of the tables (OOD_CONCURRENT_REQUESTS) on a day by day basis.14:40:49 14:40:50 14:40:51 14:40:52 14:40:52 14:40:53 14:40:55 14:40:55 14:41:21 14:41:21 14:41:21 14:41:21 14:41:21 14:41:21 14:41:21 14:41:21 14:41:22 14:41:22 14:41:22 14:41:22 14:41:22 14:41:22 14:41:22 14:41:22 14:41:22 14:41:22 14:41:22 14:41:23 14:41:23 14:41:24 14:41:24 14:41:24 14:41:24 14:41:24 14:41:24 14:41:24 14:41:24 14:41:24 14:41:24 14:41:24 14:41:24 - Create index Create index Create index Create index Create index Updating table Purging table Analyze table Create table Create synonym Create index Create index Create index Updating table Purging table Analyze table Create table Create synonym Create index Updating table Purging table Analyze table Create table Create synonym Create index Create index Updating table Purging table Analyze table Create index Updating table Purging table Analyze table Create index Updating table Purging table Analyze table Create synonym Create index Purging table Analyze table : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : ood_concurrent_requests_n91 on OOD_CONCURRENT_REQUESTS ood_concurrent_requests_n96 on OOD_CONCURRENT_REQUESTS ood_concurrent_requests_n97 on OOD_CONCURRENT_REQUESTS ood_concurrent_requests_n98 on OOD_CONCURRENT_REQUESTS ood_concurrent_requests_n99 on OOD_CONCURRENT_REQUESTS OOD_CONCURRENT_REQUESTS OOD_CONCURRENT_REQUESTS OOD_CONCURRENT_REQUESTS CONCURRENT_PROCESSES OOD_CONCURRENT_PROCESSES ood_concurrent_processes_N1 on OOD_CONCURRENT_PROCESSES ood_concurrent_processes_N2 on OOD_CONCURRENT_PROCESSES ood_concurrent_processes_u1 on OOD_CONCURRENT_PROCESSES OOD_CONCURRENT_PROCESSES OOD_CONCURRENT_PROCESSES OOD_CONCURRENT_PROCESSES CONCURRENT_QUEUES OOD_CONCURRENT_QUEUES ood_concurrent_queues_N1 on OOD_CONCURRENT_QUEUES OOD_CONCURRENT_QUEUES OOD_CONCURRENT_QUEUES OOD_CONCURRENT_QUEUES ood_conc_req_sets OOD_CONC_REQ_SETS ood_conc_req_sets_u1 on OOD_CONC_REQ_SETS ood_conc_req_sets_n1 on OOD_CONC_REQ_SETS OOD_CONC_REQ_SETS OOD_CONC_REQ_SETS OOD_CONC_REQ_SETS ood_concurrent_queue_size_n1 on OOD_CONCURRENT_QUEUE_SIZE OOD_CONCURRENT_QUEUE_SIZE OOD_CONCURRENT_QUEUE_SIZE OOD_CONCURRENT_QUEUE_SIZE ood_concurrent_queue_cont_n1 on OOD_CONCURRENT_QUEUE_CONTENT OOD_CONCURRENT_QUEUE_CONTENT OOD_CONCURRENT_QUEUE_CONTENT OOD_CONCURRENT_QUEUE_CONTENT OOD_CONC_POP ood_conc_pop_n1 on OOD_CONC_POP OOD_CONC_POP OOD_CONC_POP o 23:37:22 23:37:40 23:37:40 23:37:40 23:37:41 23:37:41 23:37:41 23:38:11 23:38:12 23:38:12 23:38:12 23:38:12 23:38:12 - Subsequent execution would look like this : : : : : : : : : : : : : : OOD_CONCURRENT_REQUESTS OOD_CONCURRENT_REQUESTS OOD_CONCURRENT_PROCESSES OOD_CONCURRENT_PROCESSES OOD_CONCURRENT_QUEUES OOD_CONCURRENT_QUEUES OOD_CONC_REQ_SETS OOD_CONC_REQ_SETS OOD_CONCURRENT_QUEUE_SIZE OOD_CONCURRENT_QUEUE_SIZE OOD_CONCURRENT_QUEUE_CONTENT OOD_CONCURRENT_QUEUE_CONTENT OOD_CONC_POP Updating table Purging table Updating table Purging table Updating table Purging table Updating table Purging table Updating table Purging table Updating table Purging table Purging table Note that in the second execution we didn’t have to create the indexes or analyze the tables. An example output is below : .

---------OOD_CONCURRENT_QUEUE_SIZE . Example output is below: Number of Processing DATES Jobs Hours --------.01 .--------.---------53.12 2 OOD_CONCURRENT_PROCESSES .30 . This will also help to have a basic measure to see if jobs are increasing over time.92 .16 63.13 .---------.---------22-MAY-09 19426 419 23-MAY-09 9045 186 24-MAY-09 6559 199 25-MAY-09 7029 118 26-MAY-09 22005 223 27-MAY-09 21198 203 28-MAY-09 21749 333 29-MAY-09 19846 356 30-MAY-09 4954 298 31-MAY-09 8205 625 01-JUN-09 17249 356 02-JUN-09 23811 404 03-JUN-09 20586 211 04-JUN-09 20596 234 05-JUN-09 18288 272 06-JUN-09 8302 281 07-JUN-09 7325 221 08-JUN-09 21082 204 09-JUN-09 20192 249 10-JUN-09 30229 249 11-JUN-09 29758 241 From this information I can see we are running about 20.15 10. .10 .35 19818 OOD_CONCURRENT_REQUESTS 52.00 . The number of processing hours just shows how many hours it would take to run all the jobs single threaded (ie it is the combined run time of all jobs).89 65.75 .---------.08 201 OOD_CONCURRENT_QUEUES .Sizes are in MB OWNER -------APPLSYS APPLSYS APPLSYS APPLSYS APPLSYS APPLSYS APPLSYS sum TABLE_NAME LOGICAL PHYSICAL DIFFERENCE NUM_ROWS -----------------------------.---------.17 3495 OOD_CONC_REQ_SETS .000. • The last step in the program is to review how many records are in the DB in a day by day basis.80 These contents then would say that we are using about 66MB of space.01 .94 1.09 169 OOD_CONC_POP .10 .09 83 OOD_CONCURRENT_QUEUE_CONTENT .10 .99 372033 ---------.000 jobs a day although the last 2 days had a significant increase going all the way to 30.02 .

.Execution Frequency Since this program manages the population and deletion of records from the historical data store its important that it be run frequently. At present I recommend executing nightly.

1. 5. In general then we are trying to track and understand the times associated with the following phases : Status Code Q I Status meaning Standby Pending Common Interpretation Incompatible Waiting All the reports will break out the times by these 3 states. When a worker wakes up to poll it will check to see if there are any jobs waiting to run. When viewing any of these reports Use the “View Ouput” button.Job Execution Reports Once installed there will be 3 Job Execution reports available : • OOD: Concurrent Program Execution Summary • OOD: Program execution counts over time • OOD: Detailed Program runtime history This section of the document will first go over some common concepts used by these reports and will then go into the specifics of each of these reports. If there are any incompatibilities between that job and any other job on the system it will have a status code of ‘Q’ for standby – otherwise it will have a status code of ‘I’ 3. To reconcile this issue we take the greater of these 2 dates to determine the effective requested_start_date. Once a job is submitted it will have a phase code of ‘P’. update it to running mode move it to running phase and begin executing the job. 2. When the sysdate is greater than the requested_start_date one of 2 things happens a. Job Submission – The time a job is submitted into the system is recorded in the FND_CONCURRENT_REQUESTS table as the “REQUEST_DATE”. When the job completes the manager takes care of any cleanup work (printing. Once the CRM releases the job it updates CRM_RELEASE_DATE column to the current time and the job goes into Pending normal mode. If the status is ‘Q’ the job remains in standby mode until the Conflict Resolution Manager (CRM) releases it. b. For Example a third party system could submit a job at 11am Jan 2nd 2009 with a requested_start_date of 11am Jan 1st 2009. The output may exist in both the log and out files however only the output in the “View Output” button is properly formatted. R Running Executing . If the job is already in Pending normal mode it will be available to be picked up by a running manager. At this time it will lock the record. Common Concepts Job Execution flow Before Analyzing these reports its important to understand the flow of how a job begins executing. rescheduling …) and completes the job by setting it to a complete phase. The requested_start_date of a job request can actually be in the past. 4.

o Starting Day o Ending Day Business Day definition – Most of the time we are most concerned about jobs running during the business day. The master program submits a child and waits for the child to complete. Job execution times – The default for the following 3 parameters is 0. Remember that GL is actually SQLGL and AP is SQLAP. Similarly to exclude jobs run at night bring the End of business day time earlier to say 5pm (eg 17:00:00) o Start of Business Day o End of Business Day o Exclude Weekends – Excluding weekends also may make sense. application. o Program Short Name – Find the short name from the Concurrent  Program  Define screen. As an example lets take a job which executes 10 child processes. However the actual completion time may be hours later than the actual start time. o Application Short Name – Program Application Short name (For example AR. then it submits the next process.This is the Concurrent Manager short name as seen in the second in the Concurrent Manager  Define screen. o Min Standby Time o Min Wait Time o Min Run Time Timeframe .Execution Time vs Actual Runtime The main issue with execution time is that it often doesn’t represent the load a job puts on the system. Where possible we use this metric to account for jobload rather than just the “completion time – start time”. To exclude jobs running in the morning set the Start of business day to a more appropriate time (for example 08:00:00). Queue or middle tier CM node in the system. Waiting for a worker (Pending time) or running for a long time (running time. Starting Day begins at 00:00:00 of the specified day and Ending day ends at 23:59:59 of the specified day. Times are in whole minutes. • Job Filters – These filters on the job being executed are useful in getting a handle on the most used jobs. Submitting User filters o Application User Name – Select only a specifica Application users jobs • • • • . o Concurrent Queue . Overall the master programs load on the system is tiny – perhaps accounting for a few seconds of runtime. ONT).Start and end days for the reports to be looking for jobs. Setting this to Y will exclude jobs which ran on Saturday and Sunday. Common Filter Parameters These 3 jobs share a number of parameters that can be used to filter the selection criteria. Essentially these are used to grab any jobs which were in conflict for a long time (Standby time). To reconcile this we have created a table called the OOD_CONC_REQ_SETS table which tries to capture the real load runtime of a job instead of the recorded runtime. Only relevant in a PCP configuration. o Hostname – Concurrent Manager host name.

o Application Resp Nam – Select only those jobs submitted from a a specific Responsibility. Switching Data Store All the reports include the following flag: • Live or Historical Tables – By default the reports run against the historical data store populated by the job “OOD: Create and Update Historical CM tables”. By switching this to “Live” it will instead go against the live FND tables. .

. What machine is getting the most load Given the options of common filters and the addition of 3 unique groupings it should be fairly easy to drill down into what jobs/users/responsibilities are causing the most load on the environment. Who has been submitting all those FSG reports? 3. Each of the first 3 columns can be any of the above values. o JOBCOUNT – Orders the report by Number of executions o JOBLOAD – Orders the report by summed runtime. This allows you to switch the sort column by either the number of jobs run or by the load the jobs represented on the system.OOD: Concurrent Program Execution Summary Parameters Besides the filters mentioned in the common concepts sections the following parameters are used : • This report adds the capability of providing 3 unique column definitions which will be used as both the first 3 columns and the group by clause. Purpose The point of this report is to try and answer some potentially complex questions. For example: 1. What job is run out of the GL application the most during the day? 2. • The other unique parameter is the “Order By” parameter. The choices which can be used are as follows : o QUEUE – Concurrent manager queue short name o Application – Program Application short name o Hostname – Concurrent Manager Host name o Responsibility – The Responsibility of the submitting user o Username – The Username of the submitting user o Short_Program_name – Concurrent program name o Full_program_name – User familiar concurrent program name o NONE – No filter on this column.

.OOD: Program execution counts over time Parameters Besides the filters mentioned in the common concepts sections the following parameters are used : • Interval – This parameter will display a record per interval detailing the job performance for any selected jobs within the interval. Avalable options are : o HOURS – The Default value. Shows job performance by hour o MINUTES .Shows job performance by minute o DAYS – A day by day comparison o WEEKS – Provides a week by week comparison using the week number of the year o MONTHS – A month by month comparison Purpose This report can be used to compare execution times over specific time periods. Its usefull in determining execution rates and load incurred by jobs and how that has changed over long periods of time.

. Purpose This report lists all the jobs line by line that meet the filter criteria. 09-06-14 15:11:34....... 09-06-14 15:22:58.. 09-06-14 15:22:58.If you know the specific request ID you are interested in you can specify it here. 09-06-14 15:21:42.. This report is more useful to analyze the details of specific jobs rather than trying to aggregate them up where the averages can distort the analysis.. 09-06-14 15:19:26. 09-06-14 15:22:15.. 09-06-14 15:19:27... 09-06-14 15:19:26.. 09-06-14 15:21:42... 09-06-14 15:22:15... 09-06-14 15:19:26. 09-06-14 15:19:26. 09-06-14 15:18:54. 09-06-14 15:22:00. Included in the report will be: • Requested start date • CRM Release date • Actual start date • Actual completion date In addition the times between each milestone are displayed...... 09-06-14 15:21:46.. 09-06-14 15:19:27......... For example the following report shows me all the child process of request ID 52065923: REQUEST_ID -------------------52065923 52065967 52065968 52065974 52065975 52065976 52065977 52065978 52065969 52065970 52065971 52065972 52065973 52065994 52065996 52065997 52065998 52066000 52066001 52066002 52066003 52066004 52066005 52066006 52066011 52066012 52066010 QUE_NAME --------------CUST_MRP CUST_MRP CUST_MRP CUST_MRP CUST_MRP CUST_MRP CUST_MRP CUST_MRP CUST_MRP CUST_MRP CUST_MRP CUST_MRP CUST_MRP CUST_MRP CUST_MRP CUST_MRP CUST_MRP CUST_MRP CUST_MRP CUST_MRP CUST_MRP CUST_MRP CUST_MRP CUST_MRP CUST_MRP CUST_MRP CUST_MRP CONCURRENT_PROG --------------MRCNSP MRCMON MRCSDW MRCSDW MRCSDW MRCSDW MRCSDW MRCSDW MRCNSW MRCNSW MRCNSW MRCNSW MRCNSW MRCSLD MRCSLD MRCSLD MRCSLD MRCSLD MRCSLD MRCSLD MRCSLD MRCSLD MRCSLD MRCNEW MRPEXPWF MRPAUREL MRCEAP USER_NAME ---------PPCBATCH PPCBATCH PPCBATCH PPCBATCH PPCBATCH PPCBATCH PPCBATCH PPCBATCH PPCBATCH PPCBATCH PPCBATCH PPCBATCH PPCBATCH PPCBATCH PPCBATCH PPCBATCH PPCBATCH PPCBATCH PPCBATCH PPCBATCH PPCBATCH PPCBATCH PPCBATCH PPCBATCH PPCBATCH PPCBATCH PPCBATCH REQUEST_DATE .. 09-06-14 15:21:49..OOD: Detailed Program runtime history Parameters Besides the filters mentioned in the common concepts sections the following parameters are used : • REQUEST_ID .. 09-06-14 15:19:27.. 09-06-14 15:22:16......... 09-06-14 15:19:27. 09-06-14 15:22:16..... 09-06-14 15:19:27.. If the REQUEST ID parameter is specified then the report will display the request ID and all its child processes in an indented fashion. 09-06-14 15:22:54.. 09-06-14 15:19:26.. -----------------. 09-06-14 15:19:26. 09-06-14 15:21:17........ 09-06-14 15:21:37..

Using this option we can see that job 52065923 kicked off job 52065967 which kicked off job 52065968 … The indented feature allows us to see the relationship between parent. . sibling and child for these processes.

...| \___________________________________ 298667 52065927-MRCMON 15:12:16 15:12:44 15:12:46 15:18:06|* | 298669 52065960-MRCNEW 15:18:01 15:18:16 15:18:23 15:18:29|* | | 52065963-MRPEXPWF 15:18:29 15:18:29 15:19:03|** | | 52065994-MRCSLD 15:21:17 15:21:34 15:21:36| * | | --.--------------------------.Mgr Process(1298664) 06/07/09 09:13:32 |******.. Pending=-.Manager Execution Reports The following report is more about the execution of the concurrent manager rather than the jobs within the concurrent manager.....Setting this to LIVE will switch the program to look at the live FND tables rather than the historical date store...*.-------...-------. The report name is “OOD: Show detailed Manager activity” and the output is a somewhat graphical depiction of the activity of the manager during a short period of time (can be between 1 and 120 minutes). Max is 120 mins (2 hours).but sleeping=z *** Manager Flags .Running=*.... • Queue Name to display – Setting this to something other than % will limit the program to show only those specific queues that meet the filter..-------. • Live or Historical Tables. Standby=_.--------|---------------| 298663 52065923-MRCNSP 15:11:34 15:18:47 15:18:54 15:22:55|zzzz* | | 52066010-MRCEAP 15:22:54 15:22:55 15:22:55| * | | 52066012-MRPAUREL 15:22:58 15:23:19 15:23:25 15:23:25| _* | | --. /* Processing requests for concurrent queue: CUST_MRP | | Manager Request CRM Actual Actual | 2 3 | Proc.. Paused=~ *** Running..Mgr Process(1298663) 06/07/09 09:13:32 |zzzz**.....Mgr Process(1298669) 06/07/09 09:13:32 |**... Idle =. Request-Program short name Start Release Start End |890123456789012| -----.. OOD: Show detailed Manager activity Parameters There are only a few parameters used by this program: • Starting Date and Time – Start time for when the analysis should begin • Reporting time in mins – Amount of time to report on..Mgr Process(1298666) 06/07/09 09:13:32 |*****. Example Report Below is an example report and what follow is an explanation on how to interpret it: (below report is from an actual customer however the job names have been replaced): Looking at jobs submitted between 06/14/2009 15:18:00 and 06/14/2009 15:33:00 *** Process Flags .| \___________________________________ 298664 52065931-MRCNSW 15:12:46 15:13:14 15:13:21 15:18:01|* | | 52065959-MRCSLD 15:18:01 15:18:01 15:18:01|* | | 52065977-MRCSDW 15:19:27 15:19:32 15:19:33| * | | 52065970-MRCNSW 15:19:26 15:19:47 15:20:03 15:21:49| -** | | 52066000-MRCSLD 15:21:46 15:21:49 15:21:49| * | | 52066001-MRCSLD 15:21:49 15:21:49 15:21:50| * | | 52066006-MRCNEW 15:22:16 15:22:18 15:22:20 15:22:58| * | | 52066011-MRPEXPWF 15:22:58 15:22:58 15:23:28| ** | | --........| \___________________________________ 298666 52065919-MRCNSP 15:11:24 15:12:13 15:12:16 15:18:28|* | | 52065962-MRCEAP 15:18:27 15:18:28 15:18:29|* | | 52065974-MRCSDW 15:19:27 15:19:29 15:19:33| * | | 52065978-MRCSDW 15:19:27 15:19:33 15:19:33| * | | 52065971-MRCNSW 15:19:26 15:19:47 15:20:03 15:22:16| -*** | | 52066005-MRCSLD 15:22:16 15:22:16 15:22:17| * | | --...| \___________________________________ ...Running=*.

..****.....298673 52065967-MRCMON 15:18:54 15:19:17 15:19:25 15:22:17|_zzz* | 298676 52065968-MRCSDW 15:19:26 15:19:26 15:19:43| * | | 52065997-MRCSLD 15:21:42 15:21:43 15:21:44| * | | 52065998-MRCSLD 15:21:42 15:21:44 15:21:45| * | | 52066003-MRCSLD 15:22:15 15:22:15 15:22:15| * | | 52066004-MRCSLD 15:22:15 15:22:16 15:22:16| * | | --...| \___________________________________ 298680 52065969-MRCNSW 15:19:26 15:19:47 15:19:55 15:22:01| **** | | 52066002-MRCSLD 15:22:00 15:22:01 15:22:01| * | | --......| | 1298680 2 14 06/07/09 09:13:32 |......| | 1298674 0 0 06/07/09 09:13:32 |..13 06/07/09 09:13:32 |..........................*.....but sleeping=z *** Manager Flags ...| | 1298665 0 0 06/07/09 09:13:32 |.....| \___________________________________ 298681 52065996-MRCSLD 15:21:37 15:21:38 15:21:39| * | 298682 52065975-MRCSDW 15:19:27 15:19:31 15:19:33| * | | 52065972-MRCNSW 15:19:26 15:19:47 15:20:03 15:21:53| -** | | 52065964-MRPAUREL 15:18:29 15:23:19 15:23:23 15:23:25|_____* | | --......****.........| | 1298678 0 0 06/07/09 09:13:32 |.......**...8 06/07/09 09:13:32 |zzzz**...| \___________________________________ ****** Process Jobs Busy% Manager Summary for CUST_MRP | -------..-----.| | 1298679 0 0 06/07/09 09:13:32 |...*.............13 06/07/09 09:13:32 |..| | 1298682 3 12...........................| | 1298667 1 ....| | 1298671 0 0 06/07/09 09:13:32 |..................Mgr Process(1298677) 06/07/09 09:13:32 |...zzz*.......****..Mgr Process(1298680) 06/07/09 09:13:32 |....| | 1298669 3 4.| | 1298681 1 .... The heading section indicates the start and end time of the report which should match the criteria provided at submission time.....****. Instead it is listed as Running by the concurrent manager and is taking ......| | 1298666 6 18.Mgr Process(1298676) 06/07/09 09:13:32 |..13 06/07/09 09:13:32 |.67 06/07/09 09:13:32 |*.| | 1298670 0 0 06/07/09 09:13:32 |.......67 06/07/09 09:13:32 |**.***.....*.......... In addition it details the characters used to represent various stats a program can be in : • Running – Process has begun processing by the concurrent manager • Pending – Process is waiting for a manager to run the job • Standby – Process is in conflict with another job and cannot run • Paused – Job has submitted a child process and is now paused waiting for the child process to complete • Running … but sleeping – Job has submitted a child process but did not go into pause state..............| | 1298675 0 0 06/07/09 09:13:32 |..| \___________________________________ 298677 52065976-MRCSDW 15:19:27 15:19:31 15:19:34| * | | 52065973-MRCNSW 15:19:26 15:19:47 15:20:04 15:22:16| -*** | | --.. Paused=~ *** Running..Running=*..*... Idle =.......*.............**.67 06/07/09 09:13:32 |*****..6 06/07/09 09:13:32 |........***.73 06/07/09 09:13:32 |******.| \___________________________________ Report Heading section Looking at jobs submitted between 06/14/2009 15:18:00 and 06/14/2009 15:33:00 *** Process Flags ...| | 1298672 0 0 06/07/09 09:13:32 |.......| | 1298664 8 19..... Standby=_. Pending=-...| | 1298673 1 19.| | 1298668 0 0 06/07/09 09:13:32 |........| | 1298677 2 15 06/07/09 09:13:32 |......*.......Mgr Process(1298682) 06/07/09 09:13:32 |....-----| 1298663 3 26..Running=*.| | 1298676 5 2....

up a slot in the queue. If the report was set to run against 120 minutes the report would actually be 120 columns wide for this section alone. For the purposes of this report then we will be sorting the jobs by the worker process that executed that job.-------. In this case we can see that this queue is meant to display the activities of the CUST_MRP queue. In this case we can see that the report spanned both the 20 minute and 30 minute marks.20.40 or 50 minutes into the report. At every 10 minute interval we indicate whether it is 10. The job is actually executing a “sleep” command as it waits for the child process to complete.--------|---------------| Each queue will have its own heading section.This queue provides ther request ID and a substring of the job being executed by the manager process.30. For that reason the report is limited to only 120 minutes at max. The Queue is started with a number of workers and each worker can be either executing a job or waiting to execute a job. Request-Program short name Start Release Start End |890123456789012| -----. – A Queue spawns several workers to deal with the jobs in the queue.-------. The workers will each have a unique “Manager Process ID”. The columns are as follows: • Manager Proc. • The next 4 columns provides the key milestones of a job: o Request Start – Greater of REQUEST_DATE and REQUESTED_START_DATE o CRM Release . In this case the flag to represent the 2 states are also displayed : • Running – Manager is actually processing a job • Idle – Manager has no work to do and is not processing any jobs. • Request-Program short name .-------. .May be blank if there are no incompatibilities otherwise it will contain the time the CRM released the job to run o Actual Start – Actual Start time o Actual End – Actual Completion time • The next set of columns represent the time in minutes. Tracking the performance of a queue then means tracking the performance of each of the workers and making sure that the workers are picking up jobs quickly.--------------------------. Since this report was run for a 15 minute period there are 15 columns in this section – each column will represent 1 of those 15 minutes of the report. Queue Heading section /* Processing requests for concurrent queue: CUST_MRP | | Manager Request CRM Actual Actual | 2 3 | Proc.

Also It ran so fast it took less than a second to run.Worker Detail section 298663 52065923-MRCNSP 15:11:34 15:18:47 15:18:54 15:22:55|zzzz* | | 52066010-MRCEAP 15:22:54 15:22:55 15:22:55| * | | 52066012-MRPAUREL 15:22:58 15:23:19 15:23:25 15:23:25| _* | | --.The first worker in the list ran 3 jobs. In addition if the worker only ran 1 job then there is no need to show a summary line of that worker as well. is not incompatible with anything so the CRM Release column is blank. Each subsequent line executed by the same worker will have a “|” instead of the worker number to indicate the they are part of the same worker flow • LINE3 . As can be seen.. In addition it will pull together all the statuses from the individual jobs so its clear how busy the worker might have been. Even so we show it as having been run during the minute. The intent is to show some representation of the “busy –ness” of a queue and how jobs reacted to that activity. It details out when the worker started and if it has exited it will show its end time as well.| \___________________________________ Each worker is displayed in numerical order. For example the following section shows worker 298667. The first job though spent pretty much the entire time running child processes as we can see it was mainly in a “Running … but Sleeping” mode the entire time (as denoted by the “z” character). If the worker ran no jobs then it wont be displayed in this section. In addition we can see that the time between requested_start_date and actual start_date is 1 second. for fast jobs many of them can be run in the same minute. with the job name of MRCEAP. • LINE2 . 298667 52065927-MRCMON 298669 52065960-MRCNEW | 52065963-MRPEXPWF … 15:12:16 15:12:44 15:12:46 15:18:06|* 15:18:01 15:18:16 15:18:23 15:18:29|* 15:18:29 15:18:29 15:19:03|** | | | • In these cases the summary line would equal the worker line so there is no need to show the summary line separately. LINE4 ..Mgr Process(1298663) 06/07/09 09:13:32 |zzzz**. Since it was requested to run at 15:22:58 it fell into the Standby section for the minute designated 22.. However since it only ran the single job it skipped the summary line and just went to the next worker... . • LINE5 .The second job.. Review the details of the report example in the previous section to see what it was waiting on. • LINE1. The detailed milestone dates are also supplied to give context to the graphical representation. It was released 21 seconds later but based on when it was submitted and released it will show up in the report with an “_” to designate standby mode.The last line of each worker is just a report separation line to indicate the end of the worker It is important to note that not every worker will be represented..The Next line sums up the activity of the worker that ran these jobs.. Clearly the graph isn’t exact.The 3rd job does have incompatibilities defined and it had to wait.

..| |......-----.........Manager summary section ****** Process Jobs Busy% Manager Summary for CUST_MRP | -------.| |*****. • The last column will be a repeat of the section seen earlier when the worker was originally displayed Note that in this section even workers which never ran a job will be portrayed.............13 06/07/09 09:13:32 | 1298674 0 0 06/07/09 09:13:32 | 1298675 0 0 06/07/09 09:13:32 | 1298676 5 2....-----| 1298663 3 26.......| |......... This makes it easier to see if capacity was available during specified times..| |.| |.| |....6 06/07/09 09:13:32 \___________________________________ |zzzz**...... ....| |...........67 06/07/09 09:13:32 | 1298667 1 ...| |******.............****....****...67 06/07/09 09:13:32 | 1298668 0 0 06/07/09 09:13:32 | 1298669 3 4.......*.....| |**.. Each line has the following columns: • Process – The unique worker ID as described before • Jobs – The number of jobs the worker processed in the timeframe specified • Busy% .........67 06/07/09 09:13:32 | 1298670 0 0 06/07/09 09:13:32 | 1298671 0 0 06/07/09 09:13:32 | 1298672 0 0 06/07/09 09:13:32 | 1298673 1 19...........| |................*...........| |......| |..... • The start and end times of the queue are then provided....***....73 06/07/09 09:13:32 | 1298665 0 0 06/07/09 09:13:32 | 1298666 6 18..8 06/07/09 09:13:32 | 1298664 8 19........| |.zzz*. This is useful to see if the queue is maxed out.13 06/07/09 09:13:32 | 1298677 2 15 06/07/09 09:13:32 | 1298678 0 0 06/07/09 09:13:32 | 1298679 0 0 06/07/09 09:13:32 | 1298680 2 14 06/07/09 09:13:32 | 1298681 1 .............**.| |......| |*....| |.............*............| |....*.....13 06/07/09 09:13:32 | 1298682 3 12....... As mentioned earlier just having a * in each column may only mean that 1 second of every minute was taken – the Busy% will be a far more accurate gauge f the amount of time the worker spent doing work.| The manager summary section provides an indication of the activity of the workers in the queue......| |..This number takes the total time available in the timeframe specified and indicates how often that worker was actually actively running a job..| |......

• Some of the logic determining the values for actual start time and job load may need to be refined. This is an anomaly of the job and as such its hard to account for it. . • Outages and system downtimes will cause huge spikes in both standby time and Pending time.Known Bugs These scripts try and mine the job history tables to show how the concurrent manager is performing. • Some jobs seem to resubmit themselves using the same request ID. However there are known issues in the data. They seem to work pretty well however it’s a bit convoluted and there may be conditions where the logic isn’t getting the right values.