ASM

OnDemand Concurrent Manager Analysis Programs
Author: Creation Date: Last Updated: Version: Approvals: Martin Fitzgerald Friday, June 12 2009 Monday, June 15 2009 1.0

Table of Contents
Concurrent Manager Analysis..................................................................................................3 Structure...............................................................................................................................3 Installation................................................................................................................................5 Parameters............................................................................................................................6 Actions.................................................................................................................................6 PLSQL tasks...................................................................................................................6 SQLPLUS tasks..............................................................................................................6 Execution Frequency...........................................................................................................9 Job Execution Reports............................................................................................................10 Common Concepts............................................................................................................10 Job Execution flow.......................................................................................................10 Execution Time vs Actual Runtime........................................................................11 Common Filter Parameters...........................................................................................11 Switching Data Store....................................................................................................12 OOD: Concurrent Program Execution Summary.............................................................13 Parameters.....................................................................................................................13 Purpose..........................................................................................................................13 OOD: Program execution counts over time......................................................................14 Parameters.....................................................................................................................14 Purpose..........................................................................................................................14 OOD: Detailed Program runtime history..........................................................................15 Parameters.....................................................................................................................15 Purpose..........................................................................................................................15 Manager Execution Reports...................................................................................................17 OOD: Show detailed Manager activity.............................................................................17 Parameters.....................................................................................................................17 Example Report............................................................................................................17 Report Heading section...........................................................................................18 Queue Heading section............................................................................................19 Worker Detail section..............................................................................................20 Manager summary section.......................................................................................21 Known Bugs...........................................................................................................................22

By keeping a secondary copy we keep the connection between jobs that ran and the queue they ran in even if the queue no longer exists. To resolve these issues and to make it easier in general to understand what is happening on any given environment we have provided a set of basic reports.Concurrent Manager Analysis Oracle OnDemand has a number of tools to analyze Concurrent Manager performance.This table holds the historical data for the FND_CONCURRENT_QUEUE_SIZE table which is used to provide the number of workers. • .This table holds the historical data for the FND_CONCURRENT_QUEUE_CONTENT table which is used to define the includes and excludes of all the queues. Structure In order to use these reports we rely on some tables to store the historical data. • OOD_CONCURRENT_QUEUE_SIZE. The manner in which these tables are populated is described later on. • OOD_CONCURRENT_QUEUE_CONTENT . This table tries to get the real “run” times of a job and uses that to report against. Without these tables existing none of the reports will work. • OOD_CONC_REQ_SETS – It turns out the start and end times of a job are not the real story. To keep space usage low and make access of the table faster only a small subset of the columns are pulled over – mostly date columns however we also capture some of characteristics of the job to determine its unique characteristics. • For the most part the historical data is summarized and it can be difficult getting the detail you need to compare points in time over vast stretches. As time goes by queues can be created and deleted. There are 7 tables associated with the historical data store : • OOD_CONCURRENT_REQUESTS – This table holds much of the information originally contained in the FND_CONCURRENT_REQUESTS table. At present these reports are purely SQL*PLUS generated reports. These scripts represent the key scripts we use to help understand the performance of the Concurrent Manager subsystem separate from the performance of the jobs executed by the Concurrent Manager. the poll time and the workshifts for a queue. • OOD_CONCURRENT_QUEUES – This table holds the historical data for the FND_CONCURRENT_QUEUES table. • OOD_CONCURRENT_PROCESSES – This table holds the historical data for the FND_CONCURRENT_PROCESSES table. Unfortunately most of these tools have 2 main drawbacks : • For the most part the tools are only accessible by OnDemand personnel and while that does help us to look at what happened in an environment it can leave a customer frustrated if they just want to understand what is happening on their environment. Using this data we can keep track on when worker targets were changed.

The basic idea is that the key tables used to hold live data will be accessible via the historical tables once populated and most scripts should only need to change the prefix from FND to OOD to view the information. .• OOD_CONC_POP – This table holds the dates of when the historical tables were last updated.

ksh Once installed the reports cannot be used until the “OOD: Create and Update Historical CM tables” job is executed. For ease of use just past this into an SR when asking for a CRT: ______________________________________________________________________ To install the reports : • • • cd into the newly created cm_jobs directory o cd /autofs/upgrade/OHSPERF/OOD_CMJOBS/cm_jobs And run the install. Alternatively you can login to the database via sqlplus as the APPS user and run the “ood_ctables” script o sqlplus apps/password @ood_ctables 700 Once these steps are completed the reports are installed and ready for use.ksh script – you will need to provide the apps password when prompted. ______________________________________________________________________ . To do this log in as System Administrator and run the job from the SRS screen. o ksh install.Installation TO install follow the directions below.

This program has the following characteristics: Parameters • Historical Days to Keep – This parameter is used to delete entries older than a certain period. SQLPLUS tasks To help manage how much space we use for these table we also report on the size of the tables and their contents. An example output is below : o An initial Execution will look similar to the following : 14:40:39 14:40:45 14:40:45 14:40:47 14:40:48 14:40:49 Create Create Create Create Create Create table synonym index index index index : : : : : : OOD_CONCURRENT_REQUESTS OOD_CONCURRENT_REQUESTS ood_concurrent_requests_U1 on OOD_CONCURRENT_REQUESTS ood_concurrent_requests_n3 on OOD_CONCURRENT_REQUESTS ood_concurrent_requests_n6 on OOD_CONCURRENT_REQUESTS ood_concurrent_requests_n90 on OOD_CONCURRENT_REQUESTS . the program now deletes older entries. 5. This requires accurate statistics so it may not be accurate if stats were not gathered recently. create_table – If the table does not currently exist it will create it. At present the default is 800 – which means we will keep just over 2 years worth of history Actions This program is split into 2 sections – a PLSQL procedure that manages the tables and a couple of sql statements that report on the contents. PLSQL tasks For each table explicitly listed above this program will perform the following actions in sequence: 1. • Table sizes – we run a script to look at the table sizes. analyze_table – If any of the previous actions resulted in a structure change ( a new table or new index) then the table is analyzed.Managing the Historical Data The historical data of most systems using the reduced column set taken from the fnd_concurrent_requests table should be quite small. update_table – we then update the contents with the current data from the live tables 4. create_indexes – We create a number of indexes (especially on the OOD_CONCURRENT_REQUESTS table ) based on the access patterns of the provided reports. purge_table – Using the parameter provided on submission . All activities for managing the space taken is handled by the Concurrent manager program “OOD: Create and Update Historical CM tables”. Each time the program is run we check to see if all expected indexes still exist 3. 2.

An example output is below : . • Table contents – Next we show how many records are in the largest of the tables (OOD_CONCURRENT_REQUESTS) on a day by day basis.14:40:49 14:40:50 14:40:51 14:40:52 14:40:52 14:40:53 14:40:55 14:40:55 14:41:21 14:41:21 14:41:21 14:41:21 14:41:21 14:41:21 14:41:21 14:41:21 14:41:22 14:41:22 14:41:22 14:41:22 14:41:22 14:41:22 14:41:22 14:41:22 14:41:22 14:41:22 14:41:22 14:41:23 14:41:23 14:41:24 14:41:24 14:41:24 14:41:24 14:41:24 14:41:24 14:41:24 14:41:24 14:41:24 14:41:24 14:41:24 14:41:24 - Create index Create index Create index Create index Create index Updating table Purging table Analyze table Create table Create synonym Create index Create index Create index Updating table Purging table Analyze table Create table Create synonym Create index Updating table Purging table Analyze table Create table Create synonym Create index Create index Updating table Purging table Analyze table Create index Updating table Purging table Analyze table Create index Updating table Purging table Analyze table Create synonym Create index Purging table Analyze table : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : ood_concurrent_requests_n91 on OOD_CONCURRENT_REQUESTS ood_concurrent_requests_n96 on OOD_CONCURRENT_REQUESTS ood_concurrent_requests_n97 on OOD_CONCURRENT_REQUESTS ood_concurrent_requests_n98 on OOD_CONCURRENT_REQUESTS ood_concurrent_requests_n99 on OOD_CONCURRENT_REQUESTS OOD_CONCURRENT_REQUESTS OOD_CONCURRENT_REQUESTS OOD_CONCURRENT_REQUESTS CONCURRENT_PROCESSES OOD_CONCURRENT_PROCESSES ood_concurrent_processes_N1 on OOD_CONCURRENT_PROCESSES ood_concurrent_processes_N2 on OOD_CONCURRENT_PROCESSES ood_concurrent_processes_u1 on OOD_CONCURRENT_PROCESSES OOD_CONCURRENT_PROCESSES OOD_CONCURRENT_PROCESSES OOD_CONCURRENT_PROCESSES CONCURRENT_QUEUES OOD_CONCURRENT_QUEUES ood_concurrent_queues_N1 on OOD_CONCURRENT_QUEUES OOD_CONCURRENT_QUEUES OOD_CONCURRENT_QUEUES OOD_CONCURRENT_QUEUES ood_conc_req_sets OOD_CONC_REQ_SETS ood_conc_req_sets_u1 on OOD_CONC_REQ_SETS ood_conc_req_sets_n1 on OOD_CONC_REQ_SETS OOD_CONC_REQ_SETS OOD_CONC_REQ_SETS OOD_CONC_REQ_SETS ood_concurrent_queue_size_n1 on OOD_CONCURRENT_QUEUE_SIZE OOD_CONCURRENT_QUEUE_SIZE OOD_CONCURRENT_QUEUE_SIZE OOD_CONCURRENT_QUEUE_SIZE ood_concurrent_queue_cont_n1 on OOD_CONCURRENT_QUEUE_CONTENT OOD_CONCURRENT_QUEUE_CONTENT OOD_CONCURRENT_QUEUE_CONTENT OOD_CONCURRENT_QUEUE_CONTENT OOD_CONC_POP ood_conc_pop_n1 on OOD_CONC_POP OOD_CONC_POP OOD_CONC_POP o 23:37:22 23:37:40 23:37:40 23:37:40 23:37:41 23:37:41 23:37:41 23:38:11 23:38:12 23:38:12 23:38:12 23:38:12 23:38:12 - Subsequent execution would look like this : : : : : : : : : : : : : : OOD_CONCURRENT_REQUESTS OOD_CONCURRENT_REQUESTS OOD_CONCURRENT_PROCESSES OOD_CONCURRENT_PROCESSES OOD_CONCURRENT_QUEUES OOD_CONCURRENT_QUEUES OOD_CONC_REQ_SETS OOD_CONC_REQ_SETS OOD_CONCURRENT_QUEUE_SIZE OOD_CONCURRENT_QUEUE_SIZE OOD_CONCURRENT_QUEUE_CONTENT OOD_CONCURRENT_QUEUE_CONTENT OOD_CONC_POP Updating table Purging table Updating table Purging table Updating table Purging table Updating table Purging table Updating table Purging table Updating table Purging table Purging table Note that in the second execution we didn’t have to create the indexes or analyze the tables.

---------53. Example output is below: Number of Processing DATES Jobs Hours --------.80 These contents then would say that we are using about 66MB of space.17 3495 OOD_CONC_REQ_SETS .09 169 OOD_CONC_POP .Sizes are in MB OWNER -------APPLSYS APPLSYS APPLSYS APPLSYS APPLSYS APPLSYS APPLSYS sum TABLE_NAME LOGICAL PHYSICAL DIFFERENCE NUM_ROWS -----------------------------.01 .12 2 OOD_CONCURRENT_PROCESSES . .--------. The number of processing hours just shows how many hours it would take to run all the jobs single threaded (ie it is the combined run time of all jobs).16 63.01 .99 372033 ---------.15 10.00 .---------.75 .92 .10 .08 201 OOD_CONCURRENT_QUEUES .02 .---------OOD_CONCURRENT_QUEUE_SIZE . This will also help to have a basic measure to see if jobs are increasing over time.10 .000 jobs a day although the last 2 days had a significant increase going all the way to 30.---------.13 .---------22-MAY-09 19426 419 23-MAY-09 9045 186 24-MAY-09 6559 199 25-MAY-09 7029 118 26-MAY-09 22005 223 27-MAY-09 21198 203 28-MAY-09 21749 333 29-MAY-09 19846 356 30-MAY-09 4954 298 31-MAY-09 8205 625 01-JUN-09 17249 356 02-JUN-09 23811 404 03-JUN-09 20586 211 04-JUN-09 20596 234 05-JUN-09 18288 272 06-JUN-09 8302 281 07-JUN-09 7325 221 08-JUN-09 21082 204 09-JUN-09 20192 249 10-JUN-09 30229 249 11-JUN-09 29758 241 From this information I can see we are running about 20.89 65.000.94 1.---------.09 83 OOD_CONCURRENT_QUEUE_CONTENT .10 .35 19818 OOD_CONCURRENT_REQUESTS 52. • The last step in the program is to review how many records are in the DB in a day by day basis.30 .

Execution Frequency Since this program manages the population and deletion of records from the historical data store its important that it be run frequently. . At present I recommend executing nightly.

Job Execution Reports Once installed there will be 3 Job Execution reports available : • OOD: Concurrent Program Execution Summary • OOD: Program execution counts over time • OOD: Detailed Program runtime history This section of the document will first go over some common concepts used by these reports and will then go into the specifics of each of these reports. In general then we are trying to track and understand the times associated with the following phases : Status Code Q I Status meaning Standby Pending Common Interpretation Incompatible Waiting All the reports will break out the times by these 3 states. To reconcile this issue we take the greater of these 2 dates to determine the effective requested_start_date. Common Concepts Job Execution flow Before Analyzing these reports its important to understand the flow of how a job begins executing. b. Job Submission – The time a job is submitted into the system is recorded in the FND_CONCURRENT_REQUESTS table as the “REQUEST_DATE”. When viewing any of these reports Use the “View Ouput” button. If the job is already in Pending normal mode it will be available to be picked up by a running manager. rescheduling …) and completes the job by setting it to a complete phase. 1. update it to running mode move it to running phase and begin executing the job. If there are any incompatibilities between that job and any other job on the system it will have a status code of ‘Q’ for standby – otherwise it will have a status code of ‘I’ 3. 2. 4. When the sysdate is greater than the requested_start_date one of 2 things happens a. Once the CRM releases the job it updates CRM_RELEASE_DATE column to the current time and the job goes into Pending normal mode. Once a job is submitted it will have a phase code of ‘P’. 5. At this time it will lock the record. When the job completes the manager takes care of any cleanup work (printing. When a worker wakes up to poll it will check to see if there are any jobs waiting to run. The output may exist in both the log and out files however only the output in the “View Output” button is properly formatted. The requested_start_date of a job request can actually be in the past. If the status is ‘Q’ the job remains in standby mode until the Conflict Resolution Manager (CRM) releases it. R Running Executing . For Example a third party system could submit a job at 11am Jan 2nd 2009 with a requested_start_date of 11am Jan 1st 2009.

Overall the master programs load on the system is tiny – perhaps accounting for a few seconds of runtime. o Starting Day o Ending Day Business Day definition – Most of the time we are most concerned about jobs running during the business day. Remember that GL is actually SQLGL and AP is SQLAP.Start and end days for the reports to be looking for jobs. Submitting User filters o Application User Name – Select only a specifica Application users jobs • • • • . Times are in whole minutes. Common Filter Parameters These 3 jobs share a number of parameters that can be used to filter the selection criteria. To reconcile this we have created a table called the OOD_CONC_REQ_SETS table which tries to capture the real load runtime of a job instead of the recorded runtime. Where possible we use this metric to account for jobload rather than just the “completion time – start time”. Queue or middle tier CM node in the system. o Program Short Name – Find the short name from the Concurrent  Program  Define screen. Similarly to exclude jobs run at night bring the End of business day time earlier to say 5pm (eg 17:00:00) o Start of Business Day o End of Business Day o Exclude Weekends – Excluding weekends also may make sense. Job execution times – The default for the following 3 parameters is 0. Setting this to Y will exclude jobs which ran on Saturday and Sunday. • Job Filters – These filters on the job being executed are useful in getting a handle on the most used jobs. The master program submits a child and waits for the child to complete. Waiting for a worker (Pending time) or running for a long time (running time. o Min Standby Time o Min Wait Time o Min Run Time Timeframe . then it submits the next process. To exclude jobs running in the morning set the Start of business day to a more appropriate time (for example 08:00:00). o Hostname – Concurrent Manager host name. However the actual completion time may be hours later than the actual start time. Essentially these are used to grab any jobs which were in conflict for a long time (Standby time). ONT). Starting Day begins at 00:00:00 of the specified day and Ending day ends at 23:59:59 of the specified day.Execution Time vs Actual Runtime The main issue with execution time is that it often doesn’t represent the load a job puts on the system. o Concurrent Queue . o Application Short Name – Program Application Short name (For example AR. Only relevant in a PCP configuration.This is the Concurrent Manager short name as seen in the second in the Concurrent Manager  Define screen. application. As an example lets take a job which executes 10 child processes.

By switching this to “Live” it will instead go against the live FND tables. Switching Data Store All the reports include the following flag: • Live or Historical Tables – By default the reports run against the historical data store populated by the job “OOD: Create and Update Historical CM tables”. .o Application Resp Nam – Select only those jobs submitted from a a specific Responsibility.

The choices which can be used are as follows : o QUEUE – Concurrent manager queue short name o Application – Program Application short name o Hostname – Concurrent Manager Host name o Responsibility – The Responsibility of the submitting user o Username – The Username of the submitting user o Short_Program_name – Concurrent program name o Full_program_name – User familiar concurrent program name o NONE – No filter on this column.OOD: Concurrent Program Execution Summary Parameters Besides the filters mentioned in the common concepts sections the following parameters are used : • This report adds the capability of providing 3 unique column definitions which will be used as both the first 3 columns and the group by clause. What job is run out of the GL application the most during the day? 2. . Each of the first 3 columns can be any of the above values. Who has been submitting all those FSG reports? 3. • The other unique parameter is the “Order By” parameter. Purpose The point of this report is to try and answer some potentially complex questions. For example: 1. What machine is getting the most load Given the options of common filters and the addition of 3 unique groupings it should be fairly easy to drill down into what jobs/users/responsibilities are causing the most load on the environment. o JOBCOUNT – Orders the report by Number of executions o JOBLOAD – Orders the report by summed runtime. This allows you to switch the sort column by either the number of jobs run or by the load the jobs represented on the system.

OOD: Program execution counts over time Parameters Besides the filters mentioned in the common concepts sections the following parameters are used : • Interval – This parameter will display a record per interval detailing the job performance for any selected jobs within the interval. Its usefull in determining execution rates and load incurred by jobs and how that has changed over long periods of time.Shows job performance by minute o DAYS – A day by day comparison o WEEKS – Provides a week by week comparison using the week number of the year o MONTHS – A month by month comparison Purpose This report can be used to compare execution times over specific time periods. Avalable options are : o HOURS – The Default value. . Shows job performance by hour o MINUTES .

. 09-06-14 15:21:46.... 09-06-14 15:19:27. 09-06-14 15:21:42.OOD: Detailed Program runtime history Parameters Besides the filters mentioned in the common concepts sections the following parameters are used : • REQUEST_ID .. 09-06-14 15:19:26.. 09-06-14 15:19:26.. 09-06-14 15:22:58.. 09-06-14 15:21:49.... 09-06-14 15:18:54..... 09-06-14 15:19:27... 09-06-14 15:19:27. 09-06-14 15:22:16.. 09-06-14 15:22:15..... 09-06-14 15:19:26.... 09-06-14 15:22:58. 09-06-14 15:22:00. 09-06-14 15:21:37........ Purpose This report lists all the jobs line by line that meet the filter criteria... If the REQUEST ID parameter is specified then the report will display the request ID and all its child processes in an indented fashion. -----------------. 09-06-14 15:19:26.. For example the following report shows me all the child process of request ID 52065923: REQUEST_ID -------------------52065923 52065967 52065968 52065974 52065975 52065976 52065977 52065978 52065969 52065970 52065971 52065972 52065973 52065994 52065996 52065997 52065998 52066000 52066001 52066002 52066003 52066004 52066005 52066006 52066011 52066012 52066010 QUE_NAME --------------CUST_MRP CUST_MRP CUST_MRP CUST_MRP CUST_MRP CUST_MRP CUST_MRP CUST_MRP CUST_MRP CUST_MRP CUST_MRP CUST_MRP CUST_MRP CUST_MRP CUST_MRP CUST_MRP CUST_MRP CUST_MRP CUST_MRP CUST_MRP CUST_MRP CUST_MRP CUST_MRP CUST_MRP CUST_MRP CUST_MRP CUST_MRP CONCURRENT_PROG --------------MRCNSP MRCMON MRCSDW MRCSDW MRCSDW MRCSDW MRCSDW MRCSDW MRCNSW MRCNSW MRCNSW MRCNSW MRCNSW MRCSLD MRCSLD MRCSLD MRCSLD MRCSLD MRCSLD MRCSLD MRCSLD MRCSLD MRCSLD MRCNEW MRPEXPWF MRPAUREL MRCEAP USER_NAME ---------PPCBATCH PPCBATCH PPCBATCH PPCBATCH PPCBATCH PPCBATCH PPCBATCH PPCBATCH PPCBATCH PPCBATCH PPCBATCH PPCBATCH PPCBATCH PPCBATCH PPCBATCH PPCBATCH PPCBATCH PPCBATCH PPCBATCH PPCBATCH PPCBATCH PPCBATCH PPCBATCH PPCBATCH PPCBATCH PPCBATCH PPCBATCH REQUEST_DATE . 09-06-14 15:22:15. 09-06-14 15:19:26. Included in the report will be: • Requested start date • CRM Release date • Actual start date • Actual completion date In addition the times between each milestone are displayed..... 09-06-14 15:22:54..... This report is more useful to analyze the details of specific jobs rather than trying to aggregate them up where the averages can distort the analysis. 09-06-14 15:19:27... 09-06-14 15:11:34... 09-06-14 15:19:26...If you know the specific request ID you are interested in you can specify it here. 09-06-14 15:21:17.. 09-06-14 15:21:42.. 09-06-14 15:22:16.. 09-06-14 15:19:27.... ....

sibling and child for these processes.Using this option we can see that job 52065923 kicked off job 52065967 which kicked off job 52065968 … The indented feature allows us to see the relationship between parent. .

. Example Report Below is an example report and what follow is an explanation on how to interpret it: (below report is from an actual customer however the job names have been replaced): Looking at jobs submitted between 06/14/2009 15:18:00 and 06/14/2009 15:33:00 *** Process Flags .. Paused=~ *** Running. Max is 120 mins (2 hours).| \___________________________________ 298667 52065927-MRCMON 15:12:16 15:12:44 15:12:46 15:18:06|* | 298669 52065960-MRCNEW 15:18:01 15:18:16 15:18:23 15:18:29|* | | 52065963-MRPEXPWF 15:18:29 15:18:29 15:19:03|** | | 52065994-MRCSLD 15:21:17 15:21:34 15:21:36| * | | --...--------|---------------| 298663 52065923-MRCNSP 15:11:34 15:18:47 15:18:54 15:22:55|zzzz* | | 52066010-MRCEAP 15:22:54 15:22:55 15:22:55| * | | 52066012-MRPAUREL 15:22:58 15:23:19 15:23:25 15:23:25| _* | | --..... Pending=-...Mgr Process(1298666) 06/07/09 09:13:32 |*****...Mgr Process(1298669) 06/07/09 09:13:32 |**.-------.Manager Execution Reports The following report is more about the execution of the concurrent manager rather than the jobs within the concurrent manager.Mgr Process(1298664) 06/07/09 09:13:32 |******....| \___________________________________ ..but sleeping=z *** Manager Flags .. Idle =.Running=*.. Standby=_... OOD: Show detailed Manager activity Parameters There are only a few parameters used by this program: • Starting Date and Time – Start time for when the analysis should begin • Reporting time in mins – Amount of time to report on..Mgr Process(1298663) 06/07/09 09:13:32 |zzzz**.Setting this to LIVE will switch the program to look at the live FND tables rather than the historical date store..*. • Live or Historical Tables. • Queue Name to display – Setting this to something other than % will limit the program to show only those specific queues that meet the filter.| \___________________________________ 298664 52065931-MRCNSW 15:12:46 15:13:14 15:13:21 15:18:01|* | | 52065959-MRCSLD 15:18:01 15:18:01 15:18:01|* | | 52065977-MRCSDW 15:19:27 15:19:32 15:19:33| * | | 52065970-MRCNSW 15:19:26 15:19:47 15:20:03 15:21:49| -** | | 52066000-MRCSLD 15:21:46 15:21:49 15:21:49| * | | 52066001-MRCSLD 15:21:49 15:21:49 15:21:50| * | | 52066006-MRCNEW 15:22:16 15:22:18 15:22:20 15:22:58| * | | 52066011-MRPEXPWF 15:22:58 15:22:58 15:23:28| ** | | --. The report name is “OOD: Show detailed Manager activity” and the output is a somewhat graphical depiction of the activity of the manager during a short period of time (can be between 1 and 120 minutes).... Request-Program short name Start Release Start End |890123456789012| -----.-------.| \___________________________________ 298666 52065919-MRCNSP 15:11:24 15:12:13 15:12:16 15:18:28|* | | 52065962-MRCEAP 15:18:27 15:18:28 15:18:29|* | | 52065974-MRCSDW 15:19:27 15:19:29 15:19:33| * | | 52065978-MRCSDW 15:19:27 15:19:33 15:19:33| * | | 52065971-MRCNSW 15:19:26 15:19:47 15:20:03 15:22:16| -*** | | 52066005-MRCSLD 15:22:16 15:22:16 15:22:17| * | | --..........-------.--------------------------.Running=*.. /* Processing requests for concurrent queue: CUST_MRP | | Manager Request CRM Actual Actual | 2 3 | Proc...

.****.13 06/07/09 09:13:32 |......but sleeping=z *** Manager Flags ............13 06/07/09 09:13:32 |.......| | 1298679 0 0 06/07/09 09:13:32 |.....| | 1298671 0 0 06/07/09 09:13:32 |...| | 1298682 3 12....| \___________________________________ Report Heading section Looking at jobs submitted between 06/14/2009 15:18:00 and 06/14/2009 15:33:00 *** Process Flags ....| | 1298666 6 18...... In addition it details the characters used to represent various stats a program can be in : • Running – Process has begun processing by the concurrent manager • Pending – Process is waiting for a manager to run the job • Standby – Process is in conflict with another job and cannot run • Paused – Job has submitted a child process and is now paused waiting for the child process to complete • Running … but sleeping – Job has submitted a child process but did not go into pause state...****..Mgr Process(1298682) 06/07/09 09:13:32 |.| \___________________________________ 298677 52065976-MRCSDW 15:19:27 15:19:31 15:19:34| * | | 52065973-MRCNSW 15:19:26 15:19:47 15:20:04 15:22:16| -*** | | --.....73 06/07/09 09:13:32 |******.........67 06/07/09 09:13:32 |*****......| | 1298676 5 2......Running=*.......298673 52065967-MRCMON 15:18:54 15:19:17 15:19:25 15:22:17|_zzz* | 298676 52065968-MRCSDW 15:19:26 15:19:26 15:19:43| * | | 52065997-MRCSLD 15:21:42 15:21:43 15:21:44| * | | 52065998-MRCSLD 15:21:42 15:21:44 15:21:45| * | | 52066003-MRCSLD 15:22:15 15:22:15 15:22:15| * | | 52066004-MRCSLD 15:22:15 15:22:16 15:22:16| * | | --......67 06/07/09 09:13:32 |*.........| | 1298677 2 15 06/07/09 09:13:32 |............Mgr Process(1298676) 06/07/09 09:13:32 |....**.....| | 1298675 0 0 06/07/09 09:13:32 |..........Running=*.| | 1298673 1 19......| | 1298678 0 0 06/07/09 09:13:32 |...****..... Pending=-......*.....| | 1298665 0 0 06/07/09 09:13:32 |.67 06/07/09 09:13:32 |**.*...Mgr Process(1298680) 06/07/09 09:13:32 |....| | 1298670 0 0 06/07/09 09:13:32 |..*......8 06/07/09 09:13:32 |zzzz**.***.. Instead it is listed as Running by the concurrent manager and is taking .****....13 06/07/09 09:13:32 |.zzz*................Mgr Process(1298677) 06/07/09 09:13:32 |......*...| \___________________________________ 298681 52065996-MRCSLD 15:21:37 15:21:38 15:21:39| * | 298682 52065975-MRCSDW 15:19:27 15:19:31 15:19:33| * | | 52065972-MRCNSW 15:19:26 15:19:47 15:20:03 15:21:53| -** | | 52065964-MRPAUREL 15:18:29 15:23:19 15:23:23 15:23:25|_____* | | --.| | 1298674 0 0 06/07/09 09:13:32 |. Paused=~ *** Running.......| | 1298667 1 .-----| 1298663 3 26...**.........| | 1298681 1 .***......... The heading section indicates the start and end time of the report which should match the criteria provided at submission time........... Standby=_.| | 1298668 0 0 06/07/09 09:13:32 |..... Idle =.....| | 1298672 0 0 06/07/09 09:13:32 |....| \___________________________________ ****** Process Jobs Busy% Manager Summary for CUST_MRP | -------...................6 06/07/09 09:13:32 |.....*...........| \___________________________________ 298680 52065969-MRCNSW 15:19:26 15:19:47 15:19:55 15:22:01| **** | | 52066002-MRCSLD 15:22:00 15:22:01 15:22:01| * | | --...| | 1298664 8 19..| | 1298669 3 4........-----...*.| | 1298680 2 14 06/07/09 09:13:32 |...........

up a slot in the queue. In this case we can see that this queue is meant to display the activities of the CUST_MRP queue. If the report was set to run against 120 minutes the report would actually be 120 columns wide for this section alone. In this case we can see that the report spanned both the 20 minute and 30 minute marks. – A Queue spawns several workers to deal with the jobs in the queue.-------. The workers will each have a unique “Manager Process ID”.--------------------------.-------. . Since this report was run for a 15 minute period there are 15 columns in this section – each column will represent 1 of those 15 minutes of the report.May be blank if there are no incompatibilities otherwise it will contain the time the CRM released the job to run o Actual Start – Actual Start time o Actual End – Actual Completion time • The next set of columns represent the time in minutes. The columns are as follows: • Manager Proc. • The next 4 columns provides the key milestones of a job: o Request Start – Greater of REQUEST_DATE and REQUESTED_START_DATE o CRM Release . The job is actually executing a “sleep” command as it waits for the child process to complete. In this case the flag to represent the 2 states are also displayed : • Running – Manager is actually processing a job • Idle – Manager has no work to do and is not processing any jobs. For the purposes of this report then we will be sorting the jobs by the worker process that executed that job.30. Request-Program short name Start Release Start End |890123456789012| -----.-------. The Queue is started with a number of workers and each worker can be either executing a job or waiting to execute a job.--------|---------------| Each queue will have its own heading section.40 or 50 minutes into the report. For that reason the report is limited to only 120 minutes at max. • Request-Program short name .20. At every 10 minute interval we indicate whether it is 10.This queue provides ther request ID and a substring of the job being executed by the manager process. Queue Heading section /* Processing requests for concurrent queue: CUST_MRP | | Manager Request CRM Actual Actual | 2 3 | Proc. Tracking the performance of a queue then means tracking the performance of each of the workers and making sure that the workers are picking up jobs quickly.

As can be seen.The 3rd job does have incompatibilities defined and it had to wait.. For example the following section shows worker 298667. The detailed milestone dates are also supplied to give context to the graphical representation.. • LINE1.Mgr Process(1298663) 06/07/09 09:13:32 |zzzz**. 298667 52065927-MRCMON 298669 52065960-MRCNEW | 52065963-MRPEXPWF … 15:12:16 15:12:44 15:12:46 15:18:06|* 15:18:01 15:18:16 15:18:23 15:18:29|* 15:18:29 15:18:29 15:19:03|** | | | • In these cases the summary line would equal the worker line so there is no need to show the summary line separately. However since it only ran the single job it skipped the summary line and just went to the next worker. • LINE5 . The intent is to show some representation of the “busy –ness” of a queue and how jobs reacted to that activity... .The last line of each worker is just a report separation line to indicate the end of the worker It is important to note that not every worker will be represented. Since it was requested to run at 15:22:58 it fell into the Standby section for the minute designated 22. In addition if the worker only ran 1 job then there is no need to show a summary line of that worker as well. It details out when the worker started and if it has exited it will show its end time as well. • LINE2 .The Next line sums up the activity of the worker that ran these jobs. Review the details of the report example in the previous section to see what it was waiting on. with the job name of MRCEAP. Even so we show it as having been run during the minute. is not incompatible with anything so the CRM Release column is blank. The first job though spent pretty much the entire time running child processes as we can see it was mainly in a “Running … but Sleeping” mode the entire time (as denoted by the “z” character).. If the worker ran no jobs then it wont be displayed in this section.The first worker in the list ran 3 jobs.The second job. for fast jobs many of them can be run in the same minute. In addition we can see that the time between requested_start_date and actual start_date is 1 second....| \___________________________________ Each worker is displayed in numerical order. In addition it will pull together all the statuses from the individual jobs so its clear how busy the worker might have been. It was released 21 seconds later but based on when it was submitted and released it will show up in the report with an “_” to designate standby mode. Clearly the graph isn’t exact. Also It ran so fast it took less than a second to run. Each subsequent line executed by the same worker will have a “|” instead of the worker number to indicate the they are part of the same worker flow • LINE3 . LINE4 .Worker Detail section 298663 52065923-MRCNSP 15:11:34 15:18:47 15:18:54 15:22:55|zzzz* | | 52066010-MRCEAP 15:22:54 15:22:55 15:22:55| * | | 52066012-MRPAUREL 15:22:58 15:23:19 15:23:25 15:23:25| _* | | --.

......| |........Manager summary section ****** Process Jobs Busy% Manager Summary for CUST_MRP | -------.... This makes it easier to see if capacity was available during specified times.| The manager summary section provides an indication of the activity of the workers in the queue.| |...| |**......| |........| |..........**.....| |....This number takes the total time available in the timeframe specified and indicates how often that worker was actually actively running a job. Each line has the following columns: • Process – The unique worker ID as described before • Jobs – The number of jobs the worker processed in the timeframe specified • Busy% .........13 06/07/09 09:13:32 | 1298677 2 15 06/07/09 09:13:32 | 1298678 0 0 06/07/09 09:13:32 | 1298679 0 0 06/07/09 09:13:32 | 1298680 2 14 06/07/09 09:13:32 | 1298681 1 ............| |*****..-----| 1298663 3 26.......| |........-----..****.......8 06/07/09 09:13:32 | 1298664 8 19.......... • The start and end times of the queue are then provided...........*.. • The last column will be a repeat of the section seen earlier when the worker was originally displayed Note that in this section even workers which never ran a job will be portrayed.*.| |.*..............6 06/07/09 09:13:32 \___________________________________ |zzzz**.........| |.....| |..| |*.........| |******................73 06/07/09 09:13:32 | 1298665 0 0 06/07/09 09:13:32 | 1298666 6 18.. This is useful to see if the queue is maxed out...*......zzz*..| |....| |.67 06/07/09 09:13:32 | 1298668 0 0 06/07/09 09:13:32 | 1298669 3 4.......| |.67 06/07/09 09:13:32 | 1298670 0 0 06/07/09 09:13:32 | 1298671 0 0 06/07/09 09:13:32 | 1298672 0 0 06/07/09 09:13:32 | 1298673 1 19..| |......67 06/07/09 09:13:32 | 1298667 1 ....***............. ........13 06/07/09 09:13:32 | 1298682 3 12............| |..13 06/07/09 09:13:32 | 1298674 0 0 06/07/09 09:13:32 | 1298675 0 0 06/07/09 09:13:32 | 1298676 5 2...| |................****. As mentioned earlier just having a * in each column may only mean that 1 second of every minute was taken – the Busy% will be a far more accurate gauge f the amount of time the worker spent doing work.

Known Bugs These scripts try and mine the job history tables to show how the concurrent manager is performing. • Outages and system downtimes will cause huge spikes in both standby time and Pending time. • Some of the logic determining the values for actual start time and job load may need to be refined. • Some jobs seem to resubmit themselves using the same request ID. They seem to work pretty well however it’s a bit convoluted and there may be conditions where the logic isn’t getting the right values. This is an anomaly of the job and as such its hard to account for it. However there are known issues in the data. .

Sign up to vote on this title
UsefulNot useful