Professional Documents
Culture Documents
ETL Design
ETL Design
1. Document Control
1.1. Revision History Date Version Number Author Details Of Change
3/4/2013 Page 2 of 14
Table of Contents
<Any change to this ETL design document should be followed up by an update to the TOC field below. To update: right-click anywhere in the TOC field and select Update Field, then select Entire Table and click ok.>
1. Document Control..................................................................................................................................2 1.1. Revision History..........................................................................................................................2 1.2. Document Reviewers..................................................................................................................2 1.3. Document Approvers..................................................................................................................2 2. High level overview................................................................................................................................4 2.1. Purpose / Overview.....................................................................................................................4 2.2. ETL / Technical Architecture.......................................................................................................4 3. Specification Details..............................................................................................................................5 3.1. Table / View Structures..............................................................................................................5 Table Name: Table 1...........................................................................................................5 Table Name: Table 2...........................................................................................................5 Source Table Name: Table 1...............................................................................................6 Source Table Name: Table 2...............................................................................................6 Target Table Name: Table 3................................................................................................6 Target Table Name: Table 4................................................................................................7 3.2. Program List................................................................................................................................7 <Provide the name of the package/program/mapping>........................................................8
Subprogram 1 <Provide the name of the package/program >......................................................8 Subprogram 2 <Provide the name of the package/program>......................................................9 Source Qualifier Transformation 1 <Provide the transformation name>.......................................9 <Provide Transformation Type> Transformation 1 <Provide the transformation name>............10 Workflow 1 <Provide the name of the workflow >.......................................................................10 Session 1 <Provide the name of the session >...........................................................................10 Session 2 <Provide the name of the session >...........................................................................10
3.3. Detailed Column Mapping Specification....................................................................................11 3.4. Job Configuration and Scheduling Details................................................................................13 Job Name 1........................................................................................................................13 Job Name 2........................................................................................................................13 4. Appendix..............................................................................................................................................14 4.1. Livelink URL..............................................................................................................................14
3/4/2013 Page 3 of 14
3/4/2013 Page 4 of 14
3. Specification Details
3.1. Table / View Structures
< Provide the list all the tables/views that are used in this measure/subject area/project.>
Database:
Layer: By: 1 yr
IN DW
Column / Field
Format
Null
Description
Database:
Layer:
By:
1 yr
IN DW
Column / Field
Format
Null
Description
3/4/2013 Page 5 of 14
< If identifying the table usage as Source, Target, Staging adds clarity to the ETL design, feel free to indicate that in the Table Name field as Source, Target, Staging, etc. However, if the table is used as a combination of table usage types, then using the more generic table specification template above would be preferred, and then clarify the table usage in the Program sections below..>
Database:
Layer: By: 1 yr
IN DW
Column / Field
Format
Null
Description
Database:
Layer:
By:
1 yr
IN DW
Column / Field
Format
Null
Description
Database:
Layer: By: 1 yr
3/4/2013 Page 6 of 14
ion
IN DW
Column / Field
Format
Null
Description
Database:
Layer:
By:
1 yr
IN DW
Column / Field
Format
Null
Description
3/4/2013 Page 7 of 14
< Indicate the ETL technology used to develop the program. Examples of ETL technologies may include Informatica, OWB, PL/SQL, etc. >
<Provide a textual description about what this package/program does. Also include any other diagrams or flow charts to help describe the ETL process.> <Describe any table/dataset level filters> <Provide the types of extraction being used by this program/package, i.e. Incremental or Complete. Describe in details if the extraction is incremental.> <Describe the types of load being used by this program/package, i.e. Insert, Update, or Delete. It may be a combination of these strategies. Is this an Incremental load versus a complete table rebuild? > <Describe the error strategy to be followed within the Extraction Strategy and Load Strategy.> Error handling within Extraction Strategy Error handling within Load Strategy
Error Strategy
<Copy and paste the following sections to describe the ETL process as needed. Sub programs sections are applicable to documenting procedural program design. Transformation sections have been provided for documenting ETL designs for implementation using Informatica (current ETL standard within the EDW environment. NOTE: Depending on your ETL design you may have either 1- All procedural sub programs 2- All Informatica transformatoins or 3- Mixture of procedural and Informatica ETL. The sub program and transformation templates below should be used according to your needs to document the ETL design. For example, if additional fields are required, then add them. Or, if your ETL design will be fully deployed with Informatica, then delete the sub program sections. Or, if you need to design a different type of transformation, such as an Aggregator transformation, copy the Transformation template below and tailor the fields according to the transformation being designed. > Sub Program Description Sources Subprogram 1 <Provide the name of the package/program > <Describe what this package/program does> <Source Table 1 Name> <Source Table 2 Name> <Source Table 3 Name>
3/4/2013 Page 8 of 14
Targets Sub Program Level Filters Table Join Condition DFD Reference BRD Reference Sub Program Description Sources Targets Sub Program Level Filters Table Join Condition DFD Reference BRD Reference
<Target Table 1 Name> <Target Table 2 Name> <Target Table 3 Name> <Describe any table/dataset level filters> <Describe the basic join information>
Subprogram 2 <Provide the name of the package/program> <Describe what this package/program does> <Source Table 1 Name> <Source Table 2 Name> <Source Table 3 Name> <Target Table 1 Name> <Target Table 2 Name> <Target Table 3 Name> <Describe any table/dataset level filters> <Describe the basic join information>
< Templates for documenting ETL designs to be implemented within Informatica. Only Source Qualifier and generic templates are provided for examples. Feel free to create new templates for different types of Transformations.> Transformation Table / Synonym / View Source Qualifier Transformation 1 <Provide the transformation name> <Provide the table, synonym, or view which the Source Qualifier references>
3/4/2013 Page 9 of 14
<Provide filters or join conditions which should be applied to the SQ query> <Provide psuedocode or the exact SQL which should drive this transformation>
<Provide Transformation Type> Transformation 1 <Provide the transformation name> <Field name required to clearly and accurately articulate the transformation design.> <Field name required to clearly and accurately articulate the transformation design.>
< This section is specific to Informatica and can be used to document ETL design implemented within Workflows and Sessions which uses the Mapping (Program) of this chapter. These sections will typically be used when a generic mapping has been defined which can be used by multiple Sessions along with special SQL filters or conditions defined in each session. > WorkFlow Description Session Targets Session Level Filters Session Targets Session Level Filters Workflow 1 <Provide the name of the workflow > < Provide a textual description about what this workflow does. Include the workflow diagram which consist of various tasks that are run concurrently or sequentially > Session 1 <Provide the name of the session >
Table Name Insert Update as Update Update as Insert Update else insert Delete Truncate Table
<Target table 1> <Target table 2> <Describe any table/dataset level filters> Session 2 <Provide the name of the session >
Table Name Insert Update as Update Update as Insert Update else insert Delete Truncate Table
<Target table 1> <Target table 2> <Describe any table/dataset level filters>
3/4/2013 Page 10 of 14
3/4/2013 Page 11 of 14
3/4/2013 Page 12 of 14
Job Name
Existing Job?
Scheduling Specs
Frequency Day/Time
Job Name
Existing Job?
Frequency
Day/Time
3/4/2013 Page 13 of 14
4. Appendix
4.1. Livelink URL This latest version of this document can be found on LiveLink at:
http://ework.cisco.com/Livelink/livelink.exe?func=ll&objId=2248909&objAction=Open
3/4/2013 Page 14 of 14