Understanding AutoSys Job Scheduling and Monitoring

Concept of AutoSys - Apr 2010
Syed Mohani
INVESTMENT BANK TECHNOLOGY
Agenda / Topics to be Covered
Topics:-
----------------------------------
--- What is autosys how it works
--- How to check Autosys is running.
--- Jil (Job Information Language)
--- How to setup a new/delete/upgrade Jobs
--- Autosys vs Cron jobs
--- Common commands used (Operate Prospective)
--- Automatic conversion from Cron jobs to Autosys jobs
--- Changes from Solaris to Linux Environment AutoSys jobs
--- Automation on Monitoring

--- Demo
--- Q & A
2
What is Autosys
AutoSys is an automated job management system that lets operate schedule, monitor and report.
This presentation is intended for Operate who will be responsible for

defining, monitoring and managing jobs run by AutoSys.
Using Autosys utilities, we can define a job by assigning it a name and specifying the arrributes
that describe its associated behavior.
--- The AutoSys job information Language (JIL)

3
Software Components
From a software perspective, the AutoSys environment is composed these main components:-
--- Event Server (Autosys Database)
--- Event Processor
--- Remote Agent
4
Jobs Types
--- Command Jobs

--- File Watcher Jobs
--- Box Jobs
5
Jobs Types
--- Box Jobs

A Box is a container of jobs with like starting conditions, either Date/time conditions
or job dependency conditions.
6
AutoSys Commands and Monitors
There are several commands unique to AutoSys that can be used to control, configure, and
report on the system behavior.
---- chk_auto_up (to check connectivity with the database)

---- autoping ( -x returns ver info -D check the database info
---- autorep (to check the job status)
---- autostatus (Status of the job)
7
AutoSys Job Definition
There are four major JIL subcommands.
--- insert_job Allows to add a new job to AutoSys

--- update_job Allows to change an existing job
--- delete_job Allows to remove an existing job
--- delete_box Removes a box job and all of its subordinate jobs
8
AutoSys Job Definition
A job performs a particular action based on a set of instructions called a job definition.
Starting Conditions:
--- Data and time condition
--- days_of_week
--- start_times
--- run_window
--- Job dependency conditions
--- success(job_name)
--- failure(job_name)
--- notrunning(job_name)
--- done(job_name)
9
Job Status
At every point in processing, a job has what is called its current status or “state”.
--- STARTING ST (The job is in the process of ‘coming up”)
--- RUNNING RU (The job is running)
--- INACTIVE IN (Job has not been processed, has never been run)
--- SUCCESS SU (Exit code equal)
--- FAILURE FA (Job exited with an exit code)
--- TERMINATED TE (Job terminated while in the RUNNING state)
--- RESTARTED RE (Job was unable to start (or failed) and schedule to restart)
--- ACTIVATED AC (Job is itself has not yet started.)
--- ON_ICE OI (Job is removed from all conditions and logic, but is still defined.
--- ON_HOLD OH (Job is on hold and will not be run until it

receives the JOB_OFF_HOLD.
The difference between "on hold" and "on ice" is that when an "on hold" job is taken off hold, if its
starting conditions are already satisfied, it will be scheduled to run, and it will run. On the other
hand, if an "on ice" job is taken "off ice," it will not start, even if its starting conditions are already
satisfied. This job will not run until its starting conditions reoccur.
10
Common Commands
# autorep -j %lmc%
Job Name Last Start Last End ST Run Pri/Xit
uat_lmc_live_rate_Dly 04/14/2010 14:26:03 ----- RU 1868/1
Syntax
autorep {-J job_name -M machine_name -G global_name} [-s -d -q -o over_num] [-r run_num]
11
Common Commands
# autostatus -j <job-name> i.e test-for-autosys-job-creation

SUCCESS
Checking on Jobs in the STARTING state.:-

# chase -E -A
Display all jobs scheduled to run between these two dates:
job_depends -t -J ALL -F "04/01/2010 00:00" -T "12/26/2010 00:00"
Get version information:-
# autoflags -a
INC2 LINUX ANY 11.0 SP2 46a989d1 snjcqacriba01
To Check AutoSys is up and running:-

# chk_auto_up
Primary Scheduler is RUNNING on machine: <Machine-name>
# ps –ef |grep autosys
# autoping -m <Machine-name>
12
Common Commands
Checking system Status

--- autoflags -a (info about autosys and system configuration)
--- autoping -m ALL -D (Verifies versions AutoSys comm are correctly)
--- autosyslog -e (Display the Event Processor log files)
--- chase –A –E (Verifies autosys database are running)
--- chk_auto_up (Inspects the DB /environment variables)
Definuing Calendars
--- autocal
--- autocal_asc (Adds, delete, and print custom calendar)
Maintaining Databases
--- archive_events (Removes old info from autosys database)

--- clean_files –d days (Removed Remote Agent Log Files)
--- dbstatistics
Managing Security
--- autosys_secure (Maintain Autosys Edit and Exec super user owner)
13
Common Commands
Reporting Job Dependencies and Conditions

--- job_depends
Reporting Job status
--- autorep
--- autostatus –j <job-name> (Reports current status of the job)
AutoSys job status send
--- sendevent
Backup of All jobs
--- autorep –J ALL –q > dump_file
14
Common Commands
To check the jobs schedule between two Dates:-
# job_depends -t -j ALL -F "04/01/2010 00:00" -T "04/30/2010 00:00“
Job Forecast Report

From: 04/01/2010 00:00:00 To: 04/30/2010 00:00:00
Job Name Next Start Atomic Start Conditions

______________________________ _______________
qa_cri_batch02_queue_Health_c_min 04/21/2010 18:00:00 -------

qa_cri_batch03_queue_Health_c_min 04/21/2010 18:00:00 -------
15
Common Commands
To specify that the jobs can be run on any day except those days specified in the <name-
holidays> calendar.
exclude_calendar : <name-hoilidays>
Sendevent:-
sendevent -J <job-name> -E CHANGE_STATUS -s TERMINATED

sendevent -J <job-name> -E STARTJOB
sendevent -J <job-name> -E FORCE_STARTJOB
sendevent -J <job-name> -E JOB_ON_ICE
sendevent -E DELETEJOB -J <job-name>
16
Common Commands
Logs:-
# autosyslog -e -J <job-name>
17
SendEvents Options
To start or force start a job manually using sendevent :
sendevent –E FORCE_STARTJOB -J <job-name>
sendevent -E STARTJOB -J <job-name>
To put jobs on OFF ICE or ON ICE :
sendevent -E OFF_ICE -J <job-name>
sendevent -E ON_ICE -J <job-name>

18
SendEvents Options
Sendevents to AutoSys for a variety of purposes, including starting or stopping AutoSys jobs,
stopping the Event processor, and putting a job on hold. This command is also used to set AutoSys
global variables or cancel a scheduled event. sendevent is normally used with "-E" & -J option
-J job_name : Specifies the name of the job to which the specified event should be sent.
-E event :Specifies the event to be sent. This option is required. Any one of the following events
may be specified as:
STARTJOB
KILLJOB
DELETEJOB
FORCE_STARTJOB
JOB_ON_ICE
JOB_OFF_ICE
JOB_ON_HOLD
JOB_OFF_HOLD
CHANGE_STATUS
STOP_DEMON
CHANGE_PRIORITY
COMMENT
ALARM
SET_GLOBAL
SEND_SIGNAL
19
Command Attributes
AUTOREP
USAGE: autorep [-J JobName] [-d | -s | -q | -o OverRide # | -a | -w]

[-B GroupName] [-I ApplicationName] [-t]
[-X ExternalInstanceName] [-Y JobType]
[-R Run_num] [-L PrintLevel] [-N Retry] [-x]
[-z GlobalBlobName] [-f BlobDownloadDirectory]
[-G GlobalName] [-M MachineName] [-D DataSource] [-?]
20
Job Physical loaction
# pwd
/cri_staging/syed/jil/lmc
*** 2193 smccuatcriwa06 citwl:ugcrmcit /cri_staging/syed/jil/lmc
# ls -altr
total 20
drwxr-xr-x 3 citwl ugcrmcit 4096 Apr 13 17:27 ..
-rwxr-xr-x 1 citwl ugcrmcit 423 Apr 13 18:31 uat_lmc_live_rate_Dly.bak
-rwxr-xr-x 1 citwl ugcrmcit 423 Apr 14 11:42 uat_lmc_live_rate_Dly.jil.20100414
-rwxrwxrwx 1 citwl ugcrmcit 395 Apr 14 12:07 uat_lmc_live_rate_Dly.jil
21
Jobs Streams
A job stream (or Flow) refers to the logic of how related AutoSys jobs are run.
---- Date and Time conditions
---- Job dependency conditions
lob_backup Command job has date and time condition (Runs every day of the week at 01:01 am
Job_clean Command job has a job dependency condition specified for its starting parameters.
It will run only on the successful completions of job_backup.
22
Jobs Streams
File Watcher Jobs:-

A file watcher job is used to watch for an incoming file. Job has date and time conditions specified
For its starting condition. Example job is to be run every day of the week at 2:00 pm. It watches
For a file arriving. The job_Fwatch job will check every 15 second for the creation of the incoming
File. When condition is met, job_Fwatch will notify the server and complete with a SUCCESS status.
23
Defining and Running jobs using Job Information Language (JIL)
We can use AutoSys proprietary scripting language --- JIL to create jobs
definitions. This job definition must be submitted to the AutoSys database
before the job can be run.
JIL Sub-Commands:-
--- insert_job
--- update_job
--- delete_job
Jil Attributes Types:-

--- Job Name (insert_job: <job-Name>)
--- Job Type (c f and b) (job_type: c)

--- Machine Name (machine: <machine-name>)
--- Starting Conditions (days_of_week: mo, we, fr)
(condition: success(my_job)
--- File to Watch For (watch_file: /tmp/<file-name>
24
JIL Syntax
Writing a JIL script, follow the syntax rules:-

1 – sub_commnad: job_name (insert_job: <user-define-job-name>
2 – attribute_keyword: value
Example:-
Creating a job:
vi <test-for-autosys-job-creation.jil>
# JIL Script
# This script will re-direct a message to the file /tmp/test-output.out
update_job: test-for-autosys-job-creation job_type: c
machine: cri_batch_job_queue
permission: gx,wx,mx
command: /bin/echo "AutoSys KT demo" > /cri_staging/syed/jil/demo/test-output.out
std_out_file: /cri_staging/syed/jil/demo/test-output.out
Saving a job:
Jil < test-for-autosys-job-creation.jil
25
Jobs Details
# autorep -j uat_lmc_live_rate_Dly -q
/* ----------------- uat_lmc_live_rate_Dly ----------------- */
insert_job: uat_lmc_live_rate_Dly job_type: c
command: /apps/ss/userprojects/lmc/live_rate.sh
machine: cri_web_job_queue
owner: citwl@smccuatcriwa06
permission: gx,wx,mx
date_conditions: 1
days_of_week: mo,tu,we,th,fr
start_times: "22:00"
description: "Live Rate Job"

std_out_file: /local/ss/cit/userprojects/log/lmc/lmc_liverate.stdout
std_err_file: /local/ss/cit/userprojects/log/lmc/lmc_liverate.stderr
alarm_if_fail: 1
26
JIL Syntax
Delete a Job:
At the unix Command Prompt
# jil <enter-key>
At the jil prompt

jil>>1> delete_job: <job-name>
exit
27
Job Delete thru jil:-
Delete the Job:-
Step 1 --- Vi the jil file and replace command with delete_job:
/* ----------------- prd_cit_cap_CPPmonthRun ----------------- */
delete_job: prd_cit_cap_CPPmonthRun job_type: c
command: `cd /home/capbg/scripts; ./CPP_monthly.sh`
machine: snjprdcria1.jpmchase.com
#owner: capbg@snjprdcria1.jpmchase.com
permission: gx,ge,wx
condition: success(prd_cit_cap_CPPfile_fwatch)
std_out_file: /home/capbg/scripts/CPP_month.stdout
std_err_file: /home/capbg/scripts/CPP_month.stderr
alarm_if_fail: 1
Step 2 --- Save the jil file into Database.

# jil < [Jil-file-name]
Step 3 --- received the outpuy from save

Attempting to Delete Job: prd_cit_cap_CPPfile_fwatch
Jobs which are dependent on the
DELETED Job: prd_cit_cap_CPPfile_fwatch
prd_cit_cap_CPPmonthRun
Delete was successful!
Exit Code = 0
28
Auto Transfer Cronjobs into Autosys jobs
Commands:-
cron2jil - Translates crontab files into JIL format
cron2jil -f crontab_file [-d output_directory] [-i include_file] [-m machine] [-p prefix]
---- input as crontab file

---- # cron2jil –f file-name
---- convert files names i.e. cp_[123]
run_calendar: <callendar-name>
autocal_asc -s test_demo -E test_export.txt

Export the calendar entries into file
autocal_asc -lc
List the cycle calendar
Note:
cron2jil does not generate JIL files for jobs that are defined in crontab to start every minute
After the *.jil or *.cal files are generated, you must submit them to the database using the
jil and the autocal_asc commands, respectively.
29
Combing Calendars:-
Calendars can be combined in a number of ways. For expample, we can create a calendar
that includes all the dates that are in either one calendar or another.
We can import calendar text files. Calendars contained in ASCII test files can be imported
Into AutoSys database. These test files may contain multiple calendars, each of which must
be delimited with the calendar: Calendar_name attributes.
calendar: Q1paydays
01/01/2010
01/15/2010
02/01/2010
02/15/2010
03/01/2010
06/15/2010
calendar: Q1holidays
01/01/2010
30
Date Range:-
The following resource sets the number of years in the data range of the calendar, as a
default at start up. This can be overridden manually by way of the Date Range option from
the options menu.
! 1 = prior year plus current year.

! 2 = thru next year, 3 = thru third year,
! 4 = thru forth year, 5 = thru fifth year,
! 10 = thru tenth year
Autocal.dateRange:
31
New Setup for Linux Environment
Reporting Job Dependencies and Conditions
Virtual Machine Physical Machine

crrt_job_queue citcrrtapp[12]-qa.us.jpmchase.net
crm_crws_job_queue citportalapp[123]-qa.us.jpmchase.net
crm_rad_job_queue citportalapp[12]-qa.us.jpmchase.net
cri_batch_job_queue citssbatchapp[1234]-qa.us.jpmchase.net
cri_web_job_queue citsswebapp[1234]-qa.us.jpmchase.net
machine: cri_web_job_queue <------ Virtual Machine Name

permission: gx,wx,mx <------ Job Permissions add “mx”
To convert existing Jobs definitions

Jil out the current job definition from existing environment into a text file
Update machine name with virtual machine name (refer next slide )
update existing permission to add additional permission “mx”
Jil in the job definition to new environment.

32
AutoSys Jobs naming convention
Get the AutoSys jobs status thru web tool.

While apply the naming convention we can write a cgi script and use apache to get
status on web.
Env-App-JobName
XX-XXXXX-XXXXXXXXXXXXXXXXXXX
XX = [DR/ST/QA/UT/PR]
XXXXX = [CMART/CAPLO/BASEL/IBFTP/……]
XXXXXXXXXXXXXXXXXXXX = JOB Name
33
AutoSys vs CronJob
max_run_alarm: <mins>
Specifies the max run time that a job should require to finish normally. This test
can catch an error. Such as the applications stuck in a loop.
n_retrys: 5
To set the job to be automatically restarted up to five times after an application
Failure.
Priority: 1
Set the job priority for run.
run_calender: <calendar-name>
Running job on the last business day of the every month.
Run_window: <time-time>
Before starting the job autosys will check the window time.
watch_file: <path>
Attributes used as Watch file min file size and watch interval
34
INVESTMENT BANK TECHNOLOGY Q&A
35

Understanding AutoSys Job Scheduling and Monitoring

Uploaded by

Document Information

Original Description:

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Understanding AutoSys Job Scheduling and Monitoring

Uploaded by

Copyright:

Available Formats

Concept of AutoSys - Apr 2010

--- Automation on Monitoring

This presentation is intended for Operate who will be responsible for

--- The AutoSys job information Language (JIL)

--- Command Jobs

--- Box Jobs

---- chk_auto_up (to check connectivity with the database)

There are four major JIL subcommands.

--- insert_job Allows to add a new job to AutoSys

--- ON_HOLD OH (Job is on hold and will not be run until it

uat_lmc_live_rate_Dly 04/14/2010 14:26:03 ----- RU 1868/1

# autostatus -j <job-name> i.e test-for-autosys-job-creation

Checking on Jobs in the STARTING state.:-

To Check AutoSys is up and running:-

Checking system Status

--- archive_events (Removes old info from autosys database)

Reporting Job Dependencies and Conditions

To check the jobs schedule between two Dates:-

# job_depends -t -j ALL -F "04/01/2010 00:00" -T "04/30/2010 00:00“

Job Forecast Report

Job Name Next Start Atomic Start Conditions

qa_cri_batch02_queue_Health_c_min 04/21/2010 18:00:00 -------

sendevent -J <job-name> -E CHANGE_STATUS -s TERMINATED

To start or force start a job manually using sendevent :

sendevent –E FORCE_STARTJOB -J <job-name>

sendevent -E STARTJOB -J <job-name>

To put jobs on OFF ICE or ON ICE :

sendevent -E OFF_ICE -J <job-name>

sendevent -E ON_ICE -J <job-name>

USAGE: autorep [-J JobName] [-d | -s | -q | -o OverRide # | -a | -w]

File Watcher Jobs:-

Jil Attributes Types:-

--- Job Type (c f and b) (job_type: c)

Writing a JIL script, follow the syntax rules:-

description: "Live Rate Job"

At the jil prompt

/* ----------------- prd_cit_cap_CPPmonthRun ----------------- */

delete_job: prd_cit_cap_CPPmonthRun job_type: c

command: `cd /home/capbg/scripts; ./CPP_monthly.sh`

Step 2 --- Save the jil file into Database.

Step 3 --- received the outpuy from save

---- input as crontab file

autocal_asc -s test_demo -E test_export.txt

! 1 = prior year plus current year.

Reporting Job Dependencies and Conditions

Virtual Machine Physical Machine

machine: cri_web_job_queue <------ Virtual Machine Name

To convert existing Jobs definitions

Get the AutoSys jobs status thru web tool.

You might also like