P. 1
ETL Development Standards

ETL Development Standards

|Views: 273|Likes:
Published by Chandrasekhar Reddy

More info:

Published by: Chandrasekhar Reddy on Oct 29, 2010
Copyright:Attribution Non-commercial

Availability:

Read on Scribd mobile: iPhone, iPad and Android.
download as PDF, TXT or read online from Scribd
See more
See less

03/27/2014

pdf

text

original

ETL Team Development Standards - DRAFT

Table of Contents
ETL Development Checklist ................................................................................................................................................................................ 2
Technical Specification Development Guidelines ..................................................................................................................................................................................2 Informatica Naming Conventions...........................................................................................................................................................................................................2 Informatica Version Control Guidelines.................................................................................................................................................................................................3 Documentation........................................................................................................................................................................................................................................4 Miscellaneous Guidelines .......................................................................................................................................................................................................................4 Optimization/Security Considerations ....................................................................................................................................................................................................4 Change Management Considerations .....................................................................................................................................................................................................4

Appendix A: Informatica Administration and Maintenance ............................................................................................................................... 5
Informatica Server Reboot Procedures ...................................................................................................................................................................................................5

Appendix B: Appendix C: Appendix D: Appendix E: Appendix F:

BAIRS Outages.............................................................................................................................................................................. 6 Issues Log....................................................................................................................................................................................... 6 Month-End Procedures .................................................................................................................................................................. 7 Archiving Procedures ..................................................................................................................................................................... 7 Student Data Warehouse Support................................................................................................................................................... 7

Access Request Process ..........................................................................................................................................................................................................................7

Appendix G – Gaelen Standards........................................................................................................................................................................... 7 Appendix H - Peoplesoft Table Considerations ................................................................................................................................................... 8 Appendix I - Environment Objectives.................................................................................................................................................................. 8

Page 1

) Area_TargetName_Qualifier_Action_vX_Y TargetName is Action is Page 2 . copy mappings from info_prod repository to info_dev repository within Designer tool. Review design with teammates. per Informatica Naming conventions. For new Informatica mappings. Create new folder as necessary. Develop new Informatica map in development folder.ETL Team Development Standards . Update “Informatica Maps in Development” list on ETL Sharepoint site – or drag separate copy of map into “In Development” folder. maps should be named using the following template: where: Area is stg – Staging dw – EDW fact/dimension tables the final target table name all in upper case del ins updt scd copy Delete Insert Update Slowly Changing Dimensions Copy (no transformation logic between sources and targets – mainly used for source to stage copies.edu/sites/RAPO/EDW/reporting/Deliverable%20Templates/Forms/AllItems.berkeley.DRAFT ETL Development Checklist Develop and publish Technical specifications based on functional requirements Create Informatica mapping specifications from ETL template Publish to Sharepoint ETL Home Shared Documents. but should be migrated to the dev copy of the relevant production folder after initial development is complete and before QA testing begins. For revisions to existing Informatica mappings. Maps may be developed in personal folders within the info_dev repository. creation of test data. use the PROJECT NAME Data Requirements template under Data Warehouse Groups > Deliverable Templates: https://calshare. Change version number per version control guidelines below. and ad hoc data movement. create mapping in info_dev repository using Designer tool.aspx Informatica Naming Conventions Map Naming Conventions Within Informatica Designer. Technical Specification Development Guidelines To develop technical specifications for a project.

transformations and/or targets.g.g. sessions should be named using the following template: s_MappingName_Qualifier where: s MappingName is Qualifier is stands for “session” the name of the Informatica mapping associated with a given session A description of the functionality of the session. e. Workflow Naming Conventions Within Informatica Workflow Manager. the major version number remains constant.ETL Team Development Standards . the minor version number is re-set to “0”. e. a change to a Filter transformation condition or a change to derived values within an Expression tranformation). new sources. Workflows should be named using the following template: wf_WorkflowName_Frequency where: wf WorkflowName is Frequency is stands for “Workflow” a description of the functionality contained within the workflow. replacing or significantly augmenting existing functionality. For minor mapping revisions. “Y” Session Naming Conventions Within Informatica Workflow Manager. Informatica Version Control Guidelines Version control in Informatica is managed as follows: Page 3 .DRAFT Qualifier is “v” “X” A description of the functionality of the mapping. “Daily”. It is initially set to 1 when a map is first created and is incremented by one for each subsequent major change to that mapping. “Weekly”.g. Major changes involve fundamental changes to a map design. stands for version is the major version number. It is initially set to 0 when a map is first created and is incremented by one for each minor change to a given mapping (e. This only needs to be added if the mapping is associated with multiple sessions. is the minor version number. “HR_ADM_WKFORCE” how often the workflow runs e. When the Major Version number (“X” above) is incremented. “Monthly”. “Daily” can be used for workflows which run Monday through Saturday or Monday through Friday.g. This only needs to be added if multiple mappings use the same target table.

• Report Inventories: https://bearshare.contains three documents containing report inventories for BAIRS. Security.edu/sites/RAPO/EDW/reporting/Reports/Shared%20Documents/Forms/AllItems. Optimization/Security Considerations • • Developer Roles (ADM_RO and ADM_HR_RO) should always be given read access to new database objects. Include considerations of shortcut folder management when existing maps are to be modified.ETL Team Development Standards .edu/C4/Implementing%20BearShare/default. Update HRMS_OPR_XREF table (in QA) to allow access. Miscellaneous Guidelines • • • • Source to Stage Mapping Guidelines Update strategy: maintenance of code values not matching in the source system Workflows should be updated in info_dev repository to match production before new/revised maps. Documentation Documentation should be posted on the appropriate BearShare site: BearShare is backed up nightly with 2 hour snapshots taken during the day – more info is available at : https://bearshare. BIS and HR.DRAFT • • Maps and associated sessions should share the same version number See naming conventions above for details on how version numbers should be maintained for mappings and sessions. Example below: Update BAIR_HRMS_OPR_XREF Set userid = ‘BISWJC’ Where oprid = ‘011502567’ (or ‘011504738’) 2. Michael Leefers is contact for bearshare questions. Administer Workforce) Change Management Considerations Resources: Page 4 .aspx. Access privileges may prevent developers from being able to view data contained in database Views.aspx . All documents on this list should be updated as part of any new development work.berkeley. Two ways to deal with this: 1. Apply for security access through SARA (HRMS Dept. It would be useful to augment these spreadsheets by adding the underlying tables associated with listed views.berkeley. sessions and/or workflows are moved to production.

berkeley.INCLIB to ASD.STAGE. • In Server Manager. Report Migrations Contact bairpthelp@berkeley.CTM.DRAFT • ASD_EDW_change_process_notification_flow. Non-standard between 5-6 PM .AEVARS). • Insure all developers have saved work and closed desktop clients. 8-9 AM.edu for report migration requests. connect to modoc • Repositories info_prod stop • Modoc shutdown (repository server) To bring up the Informatica server: • From Unix as informat on modoc: Page 5 .PUB. To request mainframe production changes: send 2 emails: • asdhelp@berkeley.BIS.P. Appendix A: Informatica Administration and Maintenance Informatica Server Reboot Procedures To bring down the Informatica server: • Confirm that no jobs are running.aspx ). Report users/ESS staff to review before general access is allowed. right-click on modoc_712 icon. (e. • Choose “Shut down server” • Log onto modoc as informat • ps –ef | grep pm • Open Admin console.g.edu/sites/RAPO/EDW/reporting/Shared%20Documents/Forms/AllItems.ETL Team Development Standards .edu Enter objects to be moved into TSO MIGMGR .describe what needs to be moved and when.berkeley.edu • ist-as-production@lists. under Data Warehouse Groups Shared Documents (https://bearshare. move members xxx from EDW.

tell Quin/DBA’s: please reset access so users can run reports (take the ucbdw1p database out of restricted access mode) • Add servicedesk@berkeley. Post a message in the report portal as soon as it is available.1./server .2/repository_server pmrepserver pmrepserver.DRAFT cd /apps/informatica7. Issues include • Reconciliation problems • Informatica production map failures • Etc.edu/sites/RAPO/EDW/reporting/Lists/DW%20Production%20Issues/AllItems.cfg • From Unix as informat on tehama: cd /apps/informatica7. • When loads have completed./pmserver pmserver_prod./pmserver pmserver_prod.cfg Appendix B: BAIRS Outages If there’s a problem loading EDW • Ask Quin/oracledoctor to put the database in restricted mode so users cannot run reports.edu to any BAIRS outage notifications so Kevin Haney can post it on the CCS Status page. Page 6 .2/ pmrepserver pmrepserver.cfg cd server . • Ask Michael to make specific reports or folders unavailable to users so we can allow other users to run reports until fin/pb load is complete.aspx.cfg cd ..ETL Team Development Standards . • Send listserves per doc in Support folder.1.berkeley. Appendix C: Issues Log All production issues should be entered in the Sharepoint issues log at https://bearshare.

Tapes are stored by Iron Mountain. retained for 1 year Backups are not encrypted. retained for 30 days • Weekly backups.ETL Team Development Standards . Dennis or designate is requested to provide a user to have access removed. Appendix G – Gaelen Standards Worksheet ALL – contains all table. DBA's grant/remove appropriate access and send notification to Dennis. See X:\RAPO\EDW\CCS-EDW\Support\Financials\ProductionSupport\AP PO Archive procedure. If user count is greater than 24. retained for 90 days • Monthly backups. Appendix F: Student Data Warehouse Support Access Request Process • • • • • • Request for access to pilot SDW is approved through Dennis or designate.let me know if you have any more questions.Next time we can avoid this by waiting an extra day before starting the monthly process for Foundation JD data. columns. This is due to our current daily process loads 2-day worth of Fact JD data in to the Open table. Approval is forwarded to Oracle DBA's DBA's verify that the allotted number of users (24) is not exceeded. BAIRDIM and BAIRFACT broken out as follows For example owner = BAIRDIM Table = ACCOUNT_TREE Page 7 .doc for details on AP/PO archiving procedures. “words” from HRMSDIM. Hope this helps . Appendix E: Archiving Procedures EDW data is backed up to tape on the following schedule: • Nightly backups.DRAFT Appendix D: Month-End Procedures Duplicate rows in Fact_JD_Open were inserted on 10/21. Dennis or designate notifies user. the day after we processed Foundation JD data for Period 3 close.

All columns would end with a class word. DW_FEFF_DT and DW_LEFF_DT.DRAFT Column = ACCOUNT_CODE Abbreviation = ACCOUNT Full English Name = ACCOUNT Worksheet Glossary – contains all the distinct abbreviations and full English name. Worksheet Class words – Contains a set of class words I am proposing to use here at Berkeley. Views will be set up for the codes as the need arises. Production • Ready for general user access Page 8 . with Current flag. then moved to the ODS/DW.ETL Team Development Standards . changing target names where necessary. If you could fill out as much as you know and send it back I would appreciate it Appendix H .Environment Objectives Development • Shared environment for DBA’s. ETL and Report Developers. One view for the current code set another view of the active code set. • Not for data validation • Not intended for functional users QA • • • Data validation Performance testing Non-production data can be loaded for test cases and then refreshed with production data (must be coordinated). Worksheet Questions – a number of abbreviations I do not know the “full English name”. The ones highlighted in red are outstanding questions that I would like to go over.Peoplesoft Table Considerations Translate table in People Soft will be extracted into the staging area. Appendix I .

You're Reading a Free Preview

Download
scribd
/*********** DO NOT ALTER ANYTHING BELOW THIS LINE ! ************/ var s_code=s.t();if(s_code)document.write(s_code)//-->