Standards for ETL UNIX Shell Scripts for use with PowerMart 6.

2
7/28/03 K. Gillenwater (Updated 10/6/04)*

Background:

The upgrade of the Development and Production PowerMart repositories and servers necessitates changes
be made to ETL UNIX shell scripts used with a job scheduling tool. These changes, in large part, due to
the location of PowerMart installation directories and PowerMart command line syntax changes provide
an opportunity to not only change existing scripts, but also provide new standards for script development.
The following provides information as to what current scripts contain, and the information that needs to
be changed/included in the updated/new scripts. These standards need be followed to ensure that the jobs
work in the intended fashion.

Current Scripting (prior to the upgrade of PowerMart):

The following example is a UNIX shell script used prior to the upgrade of the PowerMart server to
version 6.2:

export SHLIB_PATH=$ORACLE_HOME/lib:/usr/local/pmserver/informatica/pm/
export MACHINE=`hostname`

cd /usr/local/pmserver/informatica/pm/

pmcmd start <username> <password> $MACHINE:4001 bs_siqrefqa_refresh 0 1

There are a couple of items that should be noted in the script. One, the first three lines of the script set up
variables and change the directory to point to the PowerMart installation directory where the pmcmd
executable resides to start the scheduled job and run pmserver. Secondly, the pmcmd command line lists
the username and shows the password. You will note in the following section outlining the new standards
that there are several changes to remove much of the hard coding in the scripts for ease of maintenance
and to remove the passwords from the script itself.

New Scripting Standards (for use with PowerMart version 6.2):

New scripting standards include the use of a UNIX shell script which the scheduling tool uses to start the
PowerMart job, and a separate file which contains the username and password for the user called in the
script. The following is a template that should be used to create scripts for new scheduled jobs, as well as
updating current jobs. Please note that once the upgrade is complete, existing scripts will no longer
function. All scripts will need to be updated prior to the upgrade of the Development and Production
instances of PowerMart. Following is the script and explanation. This script has been provided for
example and is named etl_unix_shell_script.sh.

# Program: etl_unix_shell_script.sh
# Author: Kevin Gillenwater
# Date: 7/24/2003
# Purpose: Sample UNIX shell script to load environment variables
# needed to run PowerMart jobs, pass username and password variables
# and start the job using the pmcmd command line.
#
# $1 = Project Id Parameter (ie. ud, hr, dss, siq, etc.)
#
# Example usage: etl_unix_shell_script.sh dss
#
# NOTE: Enter the Project ID parameter that is designated in the
# directory structure for your team
# (ie. dss would be used for the DWDAS team as the
# directory is /usr/local/autopca/dss/)

#-----------------------------------------------------------------
# Call the script to set up the Informatica Environment Variables:
#-----------------------------------------------------------------

. /usr/local/bin/set_pm_var.sh

#-----------------------------------------------------------------
# Read ETL configuration parameters from a separate file:
#-----------------------------------------------------------------

ETL_CONFIG_FILE=$JOBBASE/$1/remote_accts/test_etl.config
ETL_USER=`grep ETL_USER $ETL_CONFIG_FILE | awk -F: '{print $2}'`
ETL_PWD=`grep ETL_PWD $ETL_CONFIG_FILE | awk -F: '{print $2}'`

#-----------------------------------------------------------------
# Start the job
#-----------------------------------------------------------------

$PM_HOME/pmcmd startworkflow -u $ETL_USER -p $ETL_PWD -s $MACHINE:4001 -f DWDAS_

LOAD_dssqa wait s_m_CENTER_INSTITUTE

#-----------------------------------------------------------------
# Clean up script to remove files older than 30 days
#-----------------------------------------------------------------

/usr/bin/find /usr/local/autopca/dss/tmp -mtime +30 -exec rm {} \;

Notes Regarding the Script/Standards:

1. The beginning of each script should call and execute the script set_pm_var.sh to set up the PowerMart
variables used in the session (. /usr/local/bin/set_pm_var.sh). The set_pm_var.sh located on each machine
(Leisure for Production and Glance for Development) will provide for ease of maintenance if changes
need to be made to PowerMart variables, and will provide for one source for the scripts on a machine. The
following is the code in the script. You will not need to do anything with this script, it is for informational
purposes only:

# Program: set_pm_var.sh
# Author: Kevin Gillenwater
# Date: 7/3/2003
# Purpose: UNIX script which sets the variables for running PowerMart 6.2
# when called from a shell script (ie. script run by Autosys).

#-------------------------------------------
#Set up the Informatica Variables
#-------------------------------------------

export MACHINE=`hostname`
export SHLIB_PATH=/usr/local/pmserver/informatica/pm/infoserver
export PM_HOME=/usr/local/pmserver/informatica/pm/infoserver

#---------------------------------------------------------------------
# Set the environment variables needed for scheduling jobs.
# The value of JOBBASE differs based on the account. For AUTOSYS
# and AUTODBA, the variable should evaluate to /usr/local/autopca.
# For all other accounts, it should evaluate to their $HOME variable.
#---------------------------------------------------------------------

case $HOME in

/home/autopca/autopca) JOBBASE=/usr/local/autopca ;;
/home/autopca/autodba) JOBBASE=/usr/local/autopca ;;
*) JOBBASE=$HOME ;;

esac

export JOBBASE

2. The second section of etl_unix_shell_script.sh sets up ETL parameters for username and password
usage by PowerMart when running the workflow. The username and password are no longer stored as part
of the main shell script. In the script example, the filename test_etl_config contains the username and
password to be used when running the workflow. Following is the contents of test_etl_config:

ETL_USER:etlguy
ETL_PWD:ou812

This file (your password file) must be located in your team directory under the remote_accts folder (i.e.
/usr/local/autopca/dss/remote_accts/). The folder permissions on the Production machine will only permit
PCAs access to this folder. The permissions should be 6-4-0 (rw,r,-) on this file.

3. The third section of etl_unix_shell_script.sh contains the command to run the workflow in
PowerMart. Please follow the script exactly. The only thing that will need to be changed in this section is
the folder name and the workflow within the folder that is being executed
(i.e. ... DWDAS_LOAD_dssqa s_m_CENTER_INSTITUTE).

4. The final section of etl_unix_shell_script.sh contains code to clean up the scheduling tools log files
that are older than 30 days according to standards. Simply change the directory to point to your teams
specific folder (i.e. /usr/local/autopca/dss/tmp ).

*8/27/03 Added -wait parameter to the pmcmd command line to start the scheduled job.
*10/6/04 - Updated to generalize the scheduling tool used to run the shell scripts as UC4 has been chosen
to replace Autosys as the scheduling tool used to run PowerMart workflows.