Informatica – Historical catch-

up load in Informatica using

UNIX script

Priya Ravikumar

(524390)
Business Requirement:
As per the business requirement, we need to extract last one year data from source API and merge all
the data, generate them as a single file. Source API has 4 input parameter
Start date -from date
End date -to date
Limit -number of records
Page -Page number
Note: Start date and end date range should not exceed more than 7 days, API throws exception since it
could not fetch huge data.

Implementation:
To overcome the API exception, we have followed the below steps to complete the full load.

Step 1: Create a date list file which consist of last one year date, contains start and end date split into
7days each in a line. Need to make sure the date list file have CRLF at the end of each line.
Screenshot of date list file:

Step 2:
 Create 2 parameter files (static and dynamic) in which the Dynamic param file should be used as
Workflow param file.

 Static parameter file should have all required variables other than start and end date.

 Dynamic param file should be created using the below script.

cat wf_gcss_API_Serenova_NA1.prm > wf_gcss_API_Serenova_NA.prm

cat GCSS_SERENOVA_DATETIME >> wf_gcss_API_Serenova_NA.prm

Step 3:
Create a UNIX script which does the below:

1.Read the first line from the date list file and write it to a temp file(GCSS_SERENOVA_DATETIME).

2.Create a new dynamic param file with data from static file(wf_gcss_API_Serenova_NA1.prm) and temp
file (GCSS_SERENOVA_DATETIME).
3. Use pmcmd command to trigger the wf_gcss_API_Serenova_NA; once after the successful
completion of workflow, it automatically removes the first set of date from the date list file.

4. Since the date is passed in the FOR LOOP, script will go in loop to assign the next date and step no:3
will be executed until it reaches the last line of the date list file.

5. All the target files generated by the workflow should be with the timestamp, once the workflow run is
completed for all the date set, merge them into single file.

UNIX script:

#!/bin/bash

#set -x

fn=Date_List_NA

Spath=?

param=?

Tgtfiles=?

FILELIST=`cat $param/$fn.txt`

LOG_FILE=Serenova_NA_historical_load.log

echo $FILELIST

for DATE in $FILELIST

do

echo "DATE Started for $DATE" >> $LOG_DIR/$LOG_FILE

Start_Date=$(echo $DATE | cut -d ',' -f1)

End_Date=$(echo $DATE | cut -d ',' -f2)

echo ""\$\$"START_DT=$Start_Date" > $param/GCSS_SERENOVA_DATETIME

echo ""\$\$"END_TIME=$End_Date" >> $param/GCSS_SERENOVA_DATETIME

cd $param;

###########Workflow details###############

cat wf_gcss_API_Serenova_NA1.prm > wf_gcss_API_Serenova_NA.prm

cat GCSS_SERENOVA_DATETIME >> wf_gcss_API_Serenova_NA.prm

echo "Dates updated"

WF='wf_gcss2edw_serenova_NA'

#############################################
MAIL_LIST='XXX'

REPNAME='REP_DEV_GLBL_EU'

UNAME='VRaviP1'

IS='XXX'

FOLDER='XX'

sh -x $Spath/wf_trigger.sh $WF $MAIL_LIST $REPNAME $UNAME $IS $FOLDER

sleep 120;

ext_code=$? >> $LOG_DIR/$LOG_FILE

if [ $ext_code -eq 0 ]

then

sleep 30;

cd $param;

sed -i '1d' $fn.txt

rm $param/GCSS_SERENOVA_DATETIME;

else

echo -e "Workflow execution failed" >> $LOG_DIR/$LOG_FILE

fi

done
Wf_trigger.sh

Rollback and restart strategy:

Script can be restarted in case of any failure.