EDL Migration Form

1.      Summary

High Level Information

Application Name/Appcode PPA
ISM-IRIS Ticket No. (REQ#/CR#/INC#) CHG000010593825
Requested By Abhilash
Requestor J&J WWID  
Requestor Contact Phone 9844374272
Requestor Contact Email AChhipa@ITS.JNJ.com
Package reviewed by Amit
Jira reference
Short Description Migration from DEV to QA
Source Environnent (Dev/QA/Prod) Dev
Source Objects
(Folder/Scripts/Database/Table/View HQL Scripts,Workflows
/UDF/HBase/SOLR etc.)
Target Environnent (Dev/QA/Prod) QA
Target Objects
(Folder/Scripts/Database/Table/View HQL Scripts, workflows
/UDF/HBase/SOLR etc.)
Migration Date 14-04-2020
Purpose Migration of HQL Scripts changes for PPA Sprint 76 from
DEV to QA
 

2.      Functionalities deployed as part of the package:

package_prisma_0001_2020-04-13_07-59-01
Ticket Description Type
 ABFY-17640 DEV - Leadtime Report - negotiated lead time - Defect (USEMs) Bug
ABFY-17342 DEV - Forecast Price derivation from Standard Cost Task
ABFY-17875 DEV - LT - APAC region missing Bug
ABFY-17834 DEV - BOM.3.2.3 Scrap Amount Story
ABFY-17994 Tru Covid Report - Packaging Task
ABFY-17995 Tru Covid Report - FG Task

package_icertis_0001_2020-04-13_07-59-01
Ticket Description Type
 ABFY- PND.3.2.10 RM Master Mapping Changes to Existing Columns - Spec No -
Task
17861 EMEA HBG
PND.3.2.9 RM Master Mapping Changes to Existing Columns -
ABFY-17735 Task
Category/Subcategory - EMEA HBG

package_00094_2020-04-13_07-40-41
Ticket Description Type
 ABFY-17849 DEV - Hide PCI from OVC/Volume/BOM/Planning view screens Task
ABFY-17824 DEV - BPCS Y2K fix- Sezanne Validation Task
PF.2.3.7. Actual Payment Duration Attributes - AP LYNX Integration EMEA
ABFY-17762 Task
HBG
PF.2.3.7. Actual Payment Duration Attributes - AP LYNX Integration EMEA
ABFY-17761 Task
BPCS
ABFY-17911 BKT.1.1.1 Data Elements - DM Story
ABFY-17912 BKT.1.1.1 Data Elements - EM Story
ABFY-17875 DEV - LT - APAC region missing Bug
PND.3.2.10 RM Master Mapping Changes to Existing Columns - Spec No -
 ABFY-17861 Task
EMEA HBG
PND.3.2.9 RM Master Mapping Changes to Existing Columns -
ABFY-17735 Task
Category/Subcategory - EMEA HBG
ABFY- 17642 DEV - EM_PO fix in Vendor XREF logic Task

3.      Prerequisites & Assumptions

1.       Ensure Kerberos authentication is done before executing the scripts
2.       Ensure permission are set to a+x for all sh scripts before executing
3.       Login using service account

4.      Evidence of successful execution in lower environment

Package execution evidence from lower environment

package_prisma_0001_2020-04-13_07-59-01

package_icertis_0001_2020-04-13_07-59-01
 package_00094_2020-04-13_07-40-41

Workflow execution evidence

Will be updated with CR. 

Migration Execution Steps

1.      Rollback Strategy:

Please create the backup directory and copy the files;

 hadoop fs -mkdir
/qa/edl/sc/consumer/lpfg/appcode/scripts/PPA_HIVE_QA/PPA_BACKUP/PPA_BACKUP_APRIL14
_PKG94
  hadoop fs -cp /qa/edl/sc/consumer/lpfg/appcode/scripts/PPA_HIVE_QA/HIVESCRIPTS
/qa/edl/sc/consumer/lpfg/appcode/scripts/PPA_HIVE_QA/PPA_BACKUP/PPA_BACKUP_APRIL14
_PKG94/

         hadoop fs -cp /qa/edl/sc/consumer/lpfg/appcode/scripts/PPA_HIVE_QA/oozie_wf

/qa/edl/sc/consumer/lpfg/appcode/scripts/PPA_HIVE_QA/PPA_BACKUP/PPA_BACKUP_APRIL14_
PKG94/

         hadoop fs -cp /qa/edl/sc/consumer/lpfg/appcode/scripts/PPA_HIVE_QA/oozie_coord

/qa/edl/sc/consumer/lpfg/appcode/scripts/PPA_HIVE_QA/PPA_BACKUP/PPA_BACKUP_APRIL14_
PKG94/

 hadoop fs -cp /qa/edl/sc/consumer/lpfg/appcode/scripts/jars/TruReportSpark.jar

/qa/edl/sc/consumer/lpfg/appcode/scripts/jars/leadtime.jar
/qa/edl/sc/consumer/lpfg/appcode/scripts/jars/PaymentTerm.jar
/qa/edl/sc/consumer/lpfg/appcode/scripts/jars/SpendByFranchise.jar
/qa/edl/sc/consumer/lpfg/appcode/scripts/jars/BomExtractor-2.4-jar-with-dependencies.jar
/qa/edl/sc/consumer/lpfg/str/lpfg_stg/bpcs_config
/qa/edl/sc/consumer/lpfg/str/lpfg_core/gbl_bom_im_report_history
/qa/edl/sc/consumer/lpfg/str/lpfg_core/Payment_Term_Report_Table_Previous
/qa/edl/sc/consumer/lpfg/str/lpfg_stg/PLANTNEW
/qa/edl/sc/consumer/lpfg/str/lpfg_stg/MICOE_VOLUME_INPUT_FLAT
/qa/edl/sc/consumer/lpfg/appcode/scripts/PPA_HIVE_QA/PPA_BACKUP/PPA_BACKUP_APRIL14_
PKG94/

2.      Pre-Deployment steps:

Please overwrite the below files from Source to Target

Source /dev/edl/sc/consumer/lpfg/appcode/scripts/jars/TruReportSpark.jar
Destination /qa/edl/sc/consumer/lpfg/appcode/scripts/jars/TruReportSpark.jar
   
Source /dev/edl/sc/consumer/lpfg/appcode/scripts/jars/RMBucketReport.jar
Destination /qa/edl/sc/consumer/lpfg/appcode/scripts/jars/
 
Source /dev/edl/sc/consumer/lpfg/appcode/scripts/jars/FGBucketReport.jar
Destination /qa/edl/sc/consumer/lpfg/appcode/scripts/jars/

Source /dev/edl/sc/consumer/lpfg/appcode/scripts/jars/leadtime.jar
destination /qa/edl/sc/consumer/lpfg/appcode/scripts/jars/leadtime.jar
   
Source /dev/edl/sc/consumer/lpfg/appcode/scripts/jars/PaymentTerm.jar
Destination /qa/edl/sc/consumer/lpfg/appcode/scripts/jars/PaymentTerm.jar
   
Source /dev/edl/sc/consumer/lpfg/str/lpfg_core/Payment_Term_Report_Table_Previous
destination /qa/edl/sc/consumer/lpfg/str/lpfg_core/Payment_Term_Report_Table_Previous

Source /dev/edl/sc/consumer/lpfg/appcode/scripts/jars/SpendByFranchise.jar
destination /qa/edl/sc/consumer/lpfg/appcode/scripts/jars/SpendByFranchise.jar
 Source /dev/edl/sc/consumer/lpfg/str/lpfg_core/gbl_bom_im_report_history
destination /dev/edl/sc/consumer/lpfg/str/lpfg_core/gbl_bom_im_report_history_zz

/dev/edl/sc/consumer/lpfg/appcode/scripts/jars/BomExtractor-2.4-jar-with-
Source dependencies.jar
/qa/edl/sc/consumer/lpfg/appcode/scripts/jars/BomExtractor-2.4-jar-with-
destination dependencies.jar

Source /dev/edl/sc/consumer/lpfg/str/lpfg_stg/bpcs_config
destination /qa/edl/sc/consumer/lpfg/str/lpfg_stg/bpcs_config

Source /dev/edl/sc/consumer/lpfg/str/lpfg_stg/PLANTNEW
Destination /qa/edl/sc/consumer/lpfg/str/lpfg_stg/PLANTNEW

Source /dev/edl/sc/consumer/lpfg/appcode/scripts/jars/TruCovidFGReport.jar
Destination /qa/edl/sc/consumer/lpfg/appcode/scripts/jars/

Source /dev/edl/sc/consumer/lpfg/appcode/scripts/jars/TruCovidPKGReport.jar
Destination /qa/edl/sc/consumer/lpfg/appcode/scripts/jars/

Source /dev/edl/sc/consumer/lpfg/str/lpfg_stg/MICOE_VOLUME_INPUT_FLAT
Destination /qa/edl/sc/consumer/lpfg/str/lpfg_stg/MICOE_VOLUME_INPUT_FLAT

  Run Below Commands sequentially

 sqoop import --connect

"jdbc:sqlserver://ITSBEBEWSP01663;database=APAC_ECC;username=DBL_SQOOP;password=L38pfr
0g^&" --query "SELECT * from BKPF where \$CONDITIONS" --target-dir
“/qa/edl/sc/consumer/lpfg/str/lpfg_stg/APAC/MASTER/BKPF” --fields-terminated-by '|' -m 1
--delete-target-dir

  sqoop import --connect "jdbc:sqlserver://ITSBEBEWSP01663:1433;database=STF_ECC" --username

DBL_SQOOP --password 'L38pfr0g^&' --query "select * from BSAK where \$CONDITIONS" --target-dir
"/qa/edl/sc/consumer/lpfg/str/lpfg_stg/SAP_STFGAR_BSAK" --fields-terminated-by '|' -m 1 --delete-
target-dir

 sqoop import --connect

"jdbc:sqlserver://ITSBEBEWSP01663:1433;database=APAC_ECC;username=DBL_SQOOP;password=L
38pfr0g^&" --query "SELECT * from BSEG where \$CONDITIONS" --target-dir
“/qa/edl/sc/consumer/lpfg/str/lpfg_stg/APAC/MASTER/BSEG” --fields-terminated-by '|' -m 1
--delete-target-dir

  sqoop import --driver com.sap.db.jdbc.Driver --connect

"jdbc:sap://AWSAASNVA1023:30015/SAP_ECC_LA" --username SERVICE_GLOBAL --password
'Gl0bal998!' --query "select * from SAP_ECC_LA.BSEG where koart = 'K' and \$CONDITIONS" --target-
dir “/qa/edl/sc/consumer/lpfg/str/lpfg_stg/LATAM/TRANSACTION/BSEG” --fields-terminated-by '|'
-m 1 --delete-target-dir
  sqoop import --driver com.sap.db.jdbc.Driver --connect
"jdbc:sap://AWSAASNVA1023:30015/SAP_ECC_LA" --username SERVICE_GLOBAL --password
'Gl0bal998!' --query "select * from SAP_ECC_LA.BKPF where \$CONDITIONS" --target-dir
“/qa/edl/sc/consumer/lpfg/str/lpfg_stg/LATAM/TRANSACTION/BKPF” --fields-terminated-by '|' -m
1 --delete-target-dir

  sqoop import --connect "jdbc:sqlserver://ITSBEBEWSP01663:1433;database=STF_ECC" --username

DBL_SQOOP --password 'L38pfr0g^&' --query "select * from BKPF where BUKRS IN ('8494','8525')
and \$CONDITIONS" --target-dir "/qa/edl/sc/consumer/lpfg/str/lpfg_stg/SAP_STFGAR_BKPF"
--fields-terminated-by '|' -m 1 --delete-target-dir

  sqoop import --connect "jdbc:sqlserver://ITSBEBEWSP01663:1433;database=STF_ECC" --username

DBL_SQOOP --password 'L38pfr0g^&' --query "select * from BSAK where BUKRS IN ('8494','8525')
and \$CONDITIONS" --target-dir "/qa/edl/sc/consumer/lpfg/str/lpfg_stg/SAP_STFGAR_BSAK"
--fields-terminated-by '|' -m 1 --delete-target-dir

 sqoop import --driver com.ibm.as400.access.AS400JDBCDriver --connect

"jdbc:as400://BECSC010EU.CONBE.JNJ.COM/GRCCORPFP" --username LEAPFROGPE --password
'FROGPE17' --query "select * from AML where trim(AMLPDS)>='870112' and \$CONDITIONS"
--target-dir /qa/edl/sc/consumer/lpfg/str/lpfg_stg/BPCS/TRANSACTION/GRC/BPCS_GRC_AML"
--fields-terminated-by '|' -m 1 --delete-target-dir

 sqoop import --driver com.ibm.as400.access.AS400JDBCDriver --connect

"jdbc:as400://BECSC010EU.CONBE.JNJ.COM/GRCCORPFP" --username LEAPFROGPE --password
'FROGPE17' --query "select * from AMH where trim(AMHTDA)>='870101' and \$CONDITIONS"
--target-dir "/qa/edl/sc/consumer/lpfg/str/lpfg_stg/BPCS/TRANSACTION/GRC/BPCS_GRC_AMH"
--fields-terminated-by '|' -m 1 --delete-target-dir

 sqoop import --driver com.ibm.as400.access.AS400JDBCDriver --connect

"jdbc:as400://BECSC011.CONBE.JNJ.COM/FRMCORTFT" --username LEAPFROGTE --password
FROGTE17 --query "SELECT * from ITH where \$CONDITIONS" --target-dir
/qa/edl/sc/consumer/lpfg/str/lpfg_stg/BPCS/DELTATRANSACTION/ITH/source=BPCS/region=FRM/
--hive-drop-import-delims --fields-terminated-by '|' -m 1 --delete-target-dir

 sqoop import --driver com.ibm.as400.access.AS400JDBCDriver --connect

"jdbc:as400://BECSC011.CONBE.JNJ.COM/FRMCORTHT" --username LEAPFROGTE --password
FROGTE17 --query "SELECT * from ITHSV where \$CONDITIONS" --target-dir
/qa/edl/sc/consumer/lpfg/str/lpfg_stg/BPCS/DELTATRANSACTION/ITH/source=BPCS/region=FRM/
--hive-drop-import-delims --fields-terminated-by '|' -m 1 --append

3.      Copy package

Copy following packages from source to target

package_prisma_0001_2020-04-13_07-59-01

package_icertis_0001_2020-04-13_07-59-01

package_00094_2020-04-13_07-40-41
4.      Package deployment steps

Source packages folder

1.              Source: DEV

2.                  Base Package folder:

/data/data01/dev/edl/sc/consumer/lpfg/appcode/scripts/PPA_Deployment/PPA/

Target packages folder

1.              Target: QA

2.              Target Package Folder:

/data/data01/qa/edl/sc/consumer/lpfg/appcode/scripts/PPA_Deployment/PPA/

5. Run the below commands:

 impala-shell --ssl --ca_cert=/opt/cloudera/security/CAcerts/combinedtrust.pem -k -i

edlimpalaqa.jnj.com:21000 -QDECIMAL_V2=false -q 'invalidate metadata
lpfg_stg.bpcs_grc_amh; invalidate metadata lpfg_stg.bpcs_grc_aml; invalidate
metadata lpfg_stg.bpcs_regionwise_date_config;'

 6.      Execute packages

Execute the below command;

         cd
/data/data01/qa/edl/sc/consumer/lpfg/appcode/scripts/PPA_Deployment/PPA/package_prisma_00
01_2020-04-13_07-59-01/

         sh QAPACKAGE_EXE.sh

cd
/data/data01/qa/edl/sc/consumer/lpfg/appcode/scripts/PPA_Deployment/PPA/package_icertis_
0001_2020-04-13_07-59-01/

         sh QAPACKAGE_EXE.sh

cd
/data/data01/qa/edl/sc/consumer/lpfg/appcode/scripts/PPA_Deployment/PPA/package_00094_20
20-04-13_07-40-41/

         sh QAPACKAGE_EXE.sh
8. Run these workflows in Sequence:

EM_L1_L2_WF

 oozie job -oozie https://edloozieqa.jnj.com:11443/oozie -rerun 0838711-200330091727999-

oozie-oozi-W -Doozie.wf.rerun.failnodes=false

PPA_WEEKEND_MASTER_UMB_WF

 oozie job -oozie https://edloozieqa.jnj.com:11443/oozie -rerun 0835718-

200330091727999-oozie-oozi-W -Doozie.wf.rerun.failnodes=false

PPA_L1_TO_L4_UMB_WF

 oozie job -oozie https://edloozieqa.jnj.com:11443/oozie -config

/data/data01/qa/edl/sc/consumer/lpfg/appcode/scripts/PPA_Deployment/PPA/package_00
084_2019-10-18_06-15-44/FINAL/oozie_wf/PPA_L1_TO_L4_UMB_WF/job.properties -run

BOM_WF

 oozie job -oozie https://edloozieqa.jnj.com:11443/oozie -rerun 0836516-200330091727999-

oozie-oozi-W -Doozie.wf.rerun.failnodes=false

BOM_COSTING_WF

 oozie job -oozie https://edloozieqa.jnj.com:11443/oozie -rerun 0818768-200330091727999-

oozie-oozi-W -Doozie.wf.rerun.failnodes=false

MICOE_WF

 oozie job -oozie https://edloozieqa.jnj.com:11443/oozie -rerun 0818642-200330091727999-

oozie-oozi-W -Doozie.wf.rerun.failnodes=false

PRISMA_REPORTS_SPARK_UTILITY_WF

 oozie job -oozie https://edloozieqa.jnj.com:11443/oozie -config

/data/data01/qa/edl/sc/consumer/lpfg/appcode/scripts/PPA_Deployment/PPA/00082_201
9-09-06_06-50-18/FINAL/oozie_wf/PRISMA_REPORTS_SPARK_UTILITY_WF/job.properties
-run

PRISMA_DAILY_SPARK_REPORTS_WF

 oozie job -oozie https://edloozieqa.jnj.com:11443/oozie -config

/data/data01/qa/edl/sc/consumer/lpfg/appcode/scripts/PPA_Deployment/PPA/ package_0
0083_2019-09-27_10-18-
17/FINAL/oozie_wf/PRISMA_DAILY_SPARK_REPORTS_WF/job.properties -run
 END OF DEPLOYMENT

Updated Design_and_Release_Review_Checklist can be found in below mentioned location

https://jnj.sharepoint.com/:x:/r/teams/CON-TEAM-
SHARED_2/STFProjects2010onwards/2015%20Projects/_layouts/15/Doc.aspx?sourcedoc=
%7B707908FA-6A9A-4F99-832E-
64410F3A06F9%7D&file=Design_and_Release_Review_Checklist.xlsx&action=default&mobileredi
rect=true

Resolution Information [To be filled by ITS]

Implemented Date  
Change Control No.  
 

Migration Task Screenshots [To be filled by ITS]

5.       Frequently Asked Questions:

 
1.       How to submit a Service request to the EDL team?
 
Please duly fill the above section and attach the migration form to a service request (SR) in IRIS.
Please follow the below mentioned “how-to” document to raise a SR in IRIS.
 
http://wise.jnj.com/SharedServicesNonERP/BI/ClouderaEDL/Shared%20Documents/How%20to
%20documents/How%20to%20Submit%20a%20Service%20Request%20to%20EDL%20Team.pdf
 
2.       How to contact the EDL team?

Mail to DL-NCSUS-ITS-DATS-EDL-SUPPORT@ITS.JNJ.com