You are on page 1of 20

www.charles-chen.

com

SSIS ETL Framework by Tracking GUIDs in Event Handlers

SSIS ETL FRAMEWORK BY TRACKING GUIDS IN EVENT HANDLERS
I. II. Overview ......................................................................................................................................... 2 Minimum Software Requirements ................................................................................................... 3

III. Installation....................................................................................................................................... 4 IV. Functionalities ................................................................................................................................. 6 a. b. c. V. a. b. c. Logging to SQL tables ................................................................................................................... 6 Sending Email Notifications .......................................................................................................... 6 Writing variables value to text files for troubleshooting ............................................................... 7 Demonstrations ............................................................................................................................... 9 Sol_1_TGIEH_SSIS_ETL_Framework_Template........................................................................... 10 Sol_2_AWDataWarehouseRefresh_Using_TGIEH_SEF_V1 .......................................................... 11 Sol_2_AWDataWarehouseRefresh_Using_TGIEH_SEF_V2 .......................................................... 12

VI. SSIS Development Using the SSIS ETL Framework .......................................................................... 15 VII. Further Developments ................................................................................................................... 17 a. b. c. ETL Logging Reporting Creation .................................................................................................. 17 Email Notification Customization ............................................................................................... 17 Debug Output Text Files Customization...................................................................................... 18

VIII. Suggestions ................................................................................................................................... 19

1|P ag e

A C# script task exists within the four event handlers (OnPreExecute. Based on this. Overview For .NET development. I highly recommend evaluating this utility and use it. it writes the variables to the log files for troubleshooting and passes the GUIDs to the DLL. However. SourceParentGUID = {DC7801F1-5F0E-4667-AA77-FE6A60D2067C} ExecutionInstanceGUID = {9822F74D-07CD-4319-99AE-5DFE89FCB3D1} ParentContainerGUID = {11BF118C-FCBB-4006-82D3-B77238FE8A6E} PackageID = {DC7801F1-5F0E-4667-AA77-FE6A60D2067C} SourceID = {97160538-AE6C-4D29-83A9-D7A1E9B5C305} So how do we ensure the uniqueness of GUID after copying objects –Tasks or Packages? I always use BIDS Helper to reset GUIDs (see Figure 1 . like below. Microsoft Enterprise Library that can be used for logging and many other functions. I created the DLL to track GUIDs (that pass from the SSIS Package OnPreExecute and OnPostExecute Event Handlers).NET DLL – SSIS ETL Framework by Tracking GUIDs in Event Handlers.SSIS ETL FRAMEWORK BY TRACKING GUIDS IN EVENT HANDLERS I. While saving the information to SQL Server. Even though SSIS comes with logging functionality. and OnVariableValueChanged) of the SSIS ETL Framework package. OnError. Figure 1 . significant effort is required to configure it. Every object.BIDS Helper – Reset GUIDs) before deploying the packages. BIDS Helper can do more than reset GUIDS. in SSIS package has a GUID. In the script tasks. OnPostExecute. Task and Package. If you’re a SSIS or BI developer. and saved the start and stop time to SQL tables. identified the relationship for Job -> Package -> Task. Besides tracking run time information. Of course. the DLL also sends out email notifications per configuration. there are no SSIS Framework for logging and email notification. I created the SSIS ETL framework with a . and it will pass its GUID to the Event Handlers whenever it’s running. For this reason. the DLL also track errors from OnError Event Handler and variables changed from OnVariableValueChanged Event Handler.BIDS Helper – Reset GUIDs 2|P ag e .

charles-chen.com) 6) Database Mail Setup If you want to setup Database Mail with Gmail account. Minimum Software Requirements This framework was created for SSIS developers. Figure 2 – Database Mail Configuration Sample is a sample setup for your reference.com/en-us/library/ms160706(v=sql. The following are the minimum software requirements to use the framework: 1) 2) 3) 4) Operating System. Windows 7.SSIS ETL FRAMEWORK BY TRACKING GUIDS IN EVENT HANDLERS II. Windows Vista.microsoft. Figure 2 – Database Mail Configuration Sample 3|P ag e .90). Windows 2008 or Windows 2008 R2 SQL Server Business Intelligence Development 2008 SQL Server 2008 or SQL Server 2008 R2 with AdventureWorks and AdvWorksDWX databases SQL Server 2005/2008 AWDataWarehouseRefresh Package Sample (http://msdn.com) 5) Excel Add-in SQL Utilities Utility available for download at (http://www.aspx) (http://sqlserversamples.codeplex.

SSIS ETL FRAMEWORK BY TRACKING GUIDS IN EVENT HANDLERS III. Figure 3 – TGIEH_SSIS_ETL_Framework Folder Structure 3) Restore TGIEH_SSIS_ETL_Framework . 6) Review SSIS XML configuration file C:\TGIEH_SSIS_ETL_Framework\TGIEH_SSIS_ETL_Framework_Solution\TGIEH_SSIS_ETL_Fra 4|P ag e .com 2) Save the download to a folder.cmd to setup System Environment variable TGIEH_SSIS_ETL_Framework and define the path and file name for SSIS XML configuration file 5) Run C:\TGIEH_SSIS_ETL_Framework\Commands\DeployDLL_Laptop.charles-chen. then you will need to modify the command.cmd to deploy the DLL If your computer is NOT 64 bit. Installation Follow the steps below for installation: 1) Download SSIS ETL Framework by Tracking GUIDs in Event Handler from www. unzip it to C:\ (See Figure Figure 3 – TGIEH_SSIS_ETL_Framework Folder Structure). AdventureWorks and AdvWorksDWX databases from C:\TGIEH_SSIS_ETL_Framework\Backup 4) Run C:\TGIEH_SSIS_ETL_Framework\Commands\SetSysVar.

AdvWorksDWX]. Database Mail Profile Name (if Database Profile name is not Database Mail)  \Package.TGIEH_SEF_Configurations in TGIEH_SSIS_ETL_Framework and change the following: a.Pro perties[Value]  \Package.dtsConfig and change the Data Source if the SQL Server installed is not default instance 7) Review SSIS configuration table dbo.Propert ies[Value]  \Package.Properties[ConnectionString] 5|P ag e . SQL Server Name (if SQL Server installed is not default instance)  \Package.Variables[User::ETL_Db_Mail_Send_Mail_Package_Profile_Name].AdventureWorks].Properti es[Value] c.Variables[User::ETL_Db_Mail_Send_Mail_Job_Recipients].)  \Package. Email Recipients (Please change it to your Email.Properties[ConnectionString]  \Package.Connections[localhost.Properties[ Value] b.SSIS ETL FRAMEWORK BY TRACKING GUIDS IN EVENT HANDLERS mework_Solution.Connections[localhost.Variables[User::ETL_Db_Mail_Send_Mail_Job_Profile_Name].Connections[localhost.Properties[ConnectionString]  \Package.Variables[User::ETL_Db_Mail_Send_Mail_Package_Recipients].master].

FK1 PK.xlsx  C:\TGIEH_SSIS_ETL_Framework\Outputs\Sol_2_AWDataWarehouseRefresh_Using_ TGIEH_SEF_V1. Logging to SQL tables Logging SSIS packages run information into five tables: ETL_Job.xlsx b. ETL_Job PK Job_Id Machine_Name User_Name Package_Name Job_Description System_PackageID Execution_Id Start_Date End_Date Job_Status Duration PK.xlsx  C:\TGIEH_SSIS_ETL_Framework\Outputs\Sol_2_AWDataWarehouseRefresh_Using_ TGIEH_SEF_V2. ETL_Error. Sending Email Notifications 6|P ag e .SSIS ETL FRAMEWORK BY TRACKING GUIDS IN EVENT HANDLERS IV. see Figure 4 – TGIEH_SSIS_ETL_Framework Database Diagram.FK1 PK.FK1 ETL_Error Job_Id Package_Id Task_Id Error_Code Error_Description ETL_Rowcount PK.sysssislog table. and writing variables value to text files for troubleshooting.FK1 PK ETL_Package Job_Id Package_Id Package_Name System_PackageID Execution_Id Start_Date End_Date Package_Status Duration PK.FK1 PK.FK1 PK.FK1 PK Job_Id Package_Id Object_Name RowCount Figure 4 – TGIEH_SSIS_ETL_Framework Database Diagram Sample output files by using Excel Add-in SQL Utilities:  C:\TGIEH_SSIS_ETL_Framework\Outputs\Sol_1_TGIEH_SSIS_ETL_Framework_Templ ate. Therefore. you no longer need to use SSIS Logging and dbo. The following are the explanations: a.FK1 PK ETL_Task Job_Id Package_Id Task_Id Task_Sequence Execution_Id Source_Name SourceDescription Source_Id Source_Parent_Id Start_Date End_Date Task_Status Duration PK. ETL_Rowcount. sending Email Notifications. Functionalities The SSIS ETL Framework provides three major functions: logging to SQL tables. ETL_Task. ETL_Package.

TGIEH_SEF_Configurations of database TGIEH_SSIS_ETL_Framework. Email Recipients.usp_ETL_SendMail and sends out email notifications. Figure 5 – Email Notifications Sample Database Mail Profiles. and the Event Flags in SSIS configuration table dbo. Email Recipients. Writing variables value to text files for troubleshooting 7|P ag e . The stored procedure uses Database Mail of SQL Server to send out email but you need to configure Database Mail Profile Name. and the Event Flags: ETL_Db_Mail_Send_Mail_Job_Profile_Name <Database Mail Profile Name for Job> ETL_Db_Mail_Send_Mail_Job_Recipients <Database Mail Recipient for Job> ETL_Db_Mail_Send_Mail_Package_Profile_Name <Database Mail Profile Name Package Job> ETL_Db_Mail_Send_Mail_Package_Recipients <Database Mail Recipient for Package> ETL_Db_Mail_Send_Mail_Event_Job_Flag 0 1 2 3 ETL_Db_Mail_Send_Mail_Event_Job_Flag 1 2 3 ETL_Db_Mail_Send_Mail_Event_Package_Flag 1 2 3 Description Don’t Send Before Only After Only Before and After Description Success Only Failure Only Completion (Success or Failure) Description Success Failure Completion (Success or Failure) c.SSIS ETL FRAMEWORK BY TRACKING GUIDS IN EVENT HANDLERS The DLL calls the stored procedure dbo. see Figure 5 – Email Notifications Sample.

zip  C:\TGIEH_SSIS_ETL_Framework\Outputs\Sol_2_AWDataWarehouseRefresh_Using_ TGIEH_SEF_V1_Log.zip 8|P ag e . see Figure 6 – Output Files Name Sample and Figure 7 – Output Sample. Figure 6 – Output Files Name Sample Figure 7 – Output Sample Full Output Files Sample:  C:\TGIEH_SSIS_ETL_Framework\Outputs\Sol_1_TGIEH_SSIS_ETL_Framework_Templ ate_Log.SSIS ETL FRAMEWORK BY TRACKING GUIDS IN EVENT HANDLERS If the flag ETL_Write_Text_Log is set to True and the file name ETL_Log_File is assigned with path and file name.zip  C:\TGIEH_SSIS_ETL_Framework\Outputs\Sol_2_AWDataWarehouseRefresh_Using_ TGIEH_SEF_V2_Log.txt. The output file name format is (File name of ETL_Log_File without file extention)(Package name)(Task name)(Event handler name)(Run time). the C# script task in the Event Handlers writes variables information to the text files for troubleshooting.

Figure 8 – AWDWRefresh. Since you’re a SSIS developer. you need to follow the Installation instruction and install it. I am assuming that you’re familiar with SSIS Package Configurations. Connections. If you want to reset the tables. you can run C:\TGIEH_SSIS_ETL_Framework\SQL\Create Tables. If you have any questions. package sample of Integration Services from SQL Server 2005/2008. Variables. Demonstrations The first time I opened AWDataWarehouseRefresh package. and Properties setup and going to skip the setup.SSIS ETL FRAMEWORK BY TRACKING GUIDS IN EVENT HANDLERS V. Ensure that you are familiar with this utility as you will often use it to output the results into worksheets (See Figure 9 – Use Excel add-in SQL utilities to output the SQL tables). I am going to assume you’ve installed Excel Add-in SQL Utilities as outlined in the minimum software requirements. In addition.dtsx Package Before you run the demonstration. 9|P ag e . my monitor was not big enough to view it all (see Figure 8 – AWDWRefresh. please send me an email.sql.dtsx Package)!  I am going to use the SSIS framework with AWDataWarehouseRefresh package to capture the tasks run sequence and group the tasks by function in a package so it’s easy to view.

Sol_1_TGIEH_SSIS_ETL_Framework_Template The first demo shows the SSIS framework with the package having only one task . Run the SQL script Create Tables. execute the package. Hint: Check the error in ETL_Error table.SSIS ETL FRAMEWORK BY TRACKING GUIDS IN EVENT HANDLERS Figure 9 – Use Excel add-in SQL utilities to output the SQL tables a. Change the SQL Statement to SELECT * FROM SYS. and use Excel Add-in SQL Utilities to output the tables to worksheets.sql to reset tables. SSIS Codes: SSIS Codes Folder SSIS Project SSIS Package Output Sample: Output Sample Folder 10 | P a g e C:\TGIEH_SSIS_ETL_Framework\TGIEH_SSIS_ETL_Framework_Solu tion\Sol_1_TGIEH_SSIS_ETL_Framework_Template Sol_1_TGIEH_SSIS_ETL_Framework_Template. What happened? I am going to leave this to you to find out yourself.Execute SQL Task.dtproj TGIEH_SEF_Template_Pkg.TABLESxx in the Execute SQL Task in the package and run the test one more time.dtsx C:\TGIEH_SSIS_ETL_Framework\Outputs .

and use Excel Add-in SQL Utilities to output the tables to worksheets.DimPromotion Source Description Sequence Container Foreach Loop Container Execute SQL Task Execute SQL Task Execute SQL Task Execute SQL Task Execute SQL Task Foreach Loop Container Execute SQL Task Foreach Loop Container Execute SQL Task Bulk Insert Task Bulk Insert Task Bulk Insert Task Bulk Insert Task Bulk Insert Task Bulk Insert Task Bulk Insert Task Bulk Insert Task Bulk Insert Task Bulk Insert Task Bulk Insert Task Bulk Insert Task Data Flow Task Data Flow Task Data Flow Task Data Flow Task Grouping PreLoad TempTables PreLoad TempTables PreLoad PreLoad PreLoad PreLoad PreLoad TempTables TempTables TempTables TempTables TempTables TempTables TempTables TempTables TempTables LoadDim LoadDim LoadDim LoadDim LoadDim LoadDim LoadDim LoadDim LoadDim 11 | P a g e .sql to reset tables.SSIS ETL FRAMEWORK BY TRACKING GUIDS IN EVENT HANDLERS SQL Tables Output Sample Text Files Outpue Sample Sol_1_TGIEH_SSIS_ETL_Framework_Template.DimProduct Data Flow Task .DimDepartmentGroup Data Flow Task . Similar to Demo one.zip b. AWDWRefresh Tasks Grouping: Task Sequence 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 Source Name Prepare New Database Drop temp tables Drop database Execute SQL Task Create database Set database options Create udfMinimumDate function Create tables Execute SQL Task Create temp tables Execute SQL Task Bulk Insert tempSpecialOffer-ForeignData Bulk Insert tempProduct-ForeignNames Bulk Insert tempCustomer-YearlyIncome Bulk Insert tempIndividual-ForeignData Bulk Insert tempProductSubcategory-ForeignNames Bulk Insert tempStore-MinPayment Bulk Insert tempProductCategory-ForeignNames Bulk Insert AdventureWorksDWBuildVersion Bulk Insert DimTime Bulk Insert DimOrganization Bulk Insert DimScenario Bulk Insert DimAccount Data Flow Task .ETL_Task and group the similar tasks together (see below AWDWRefresh Tasks Grouping).xlsx Sol_1_TGIEH_SSIS_ETL_Framework_Template_Log. execute the package.DimGeography Data Flow Task . run the SQL script Create Tables. Sol_2_AWDataWarehouseRefresh_Using_TGIEH_SEF_V1 The second demo is to show you the SSIS framework and the package AWDWRefresh. Review SQL table dbo.

DimProductCategory Data Flow Task .DimCustomer Data Flow Task .DimCurrency Bulk Insert FactFinance Data Flow Task .xlsx Sol_2_AWDataWarehouseRefresh_Using_TGIEH_SEF_V1_Log.dtsx.FactSalesQuota Data Flow Task . AWDWRefresh_LoadDim.FactResellerSales Data Flow Task .dtsx C:\TGIEH_SSIS_ETL_Framework\Outputs Sol_2_AWDataWarehouseRefresh_Using_TGIEH_SEF_V1.SSIS ETL FRAMEWORK BY TRACKING GUIDS IN EVENT HANDLERS 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 Data Flow Task .DimReseller Data Flow Task .DimSalesReason Data Flow Task . by using TGIEH_SEF_Template_Pkg and put tasks into the 12 | P a g e .dtsx.dtsx.FactInternetSalesReason Data Flow Task .dtsx.zip c. AWDWRefresh_TempTables.DimSalesTerritory 1 Data Flow Task .DimProductSubcategory Add Primary Keys Add Indexes Drop udfMinimumDate Function Drop temp tables 1 Execute SQL Task Add Foreign Key Constraints Data Flow Task Data Flow Task Data Flow Task Data Flow Task Bulk Insert Task Data Flow Task Data Flow Task Data Flow Task Data Flow Task Data Flow Task Data Flow Task Data Flow Task Data Flow Task Data Flow Task Data Flow Task Data Flow Task Execute SQL Task Execute SQL Task Execute SQL Task Foreach Loop Container Execute SQL Task Execute SQL Task LoadDim LoadDim LoadFact LoadDim LoadFact LoadDim LoadDim LoadDim LoadFact LoadFact LoadDim LoadDim LoadDim LoadFact LoadFact LoadDim PostLoad PostLoad PostLoad PostLoad PostLoad PostLoad SSIS Codes: SSIS Codes Folder SSIS Project SSIS Package Output Sample: Output Sample Folder SQL Tables Output Sample Text Files Outpue Sample C:\TGIEH_SSIS_ETL_Framework\TGIEH_SSIS_ETL_Framework_Solu tion\Sol_2_AWDataWarehouseRefresh_Using_TGIEH_SEF_V1 Sol_2_AWDataWarehouseRefresh_Using_TGIEH_SEF_V1.DimSalesTerritory 2 Data Flow Task .dtsx. AWDWRefresh_PreLoad.dtproj AWDWRefresh_Using_TGIEH_SEF.DimDepartmentGroup 1 Data Flow Task . AWDWRefresh_LoadFact.FactInternetSales Data Flow Task . Sol_2_AWDataWarehouseRefresh_Using_TGIEH_SEF_V2 I created five packages.FactCurrencyRate Data Flow Task . and AWDWRefresh_PostLoad.DimEmployee Data Flow Task .

execute the package. For Data Flow Task. I used Row Count to set the variables.dtsx AWDWRefresh_PreLoad. I can see AWDataWarehouseRefresh Project clearly now and it’s easy for me to maintain it. Like above demos. Okay.dtproj AWDWRefresh_Master.sql to reset tables. Run the SQL script Create Tables. I setup a SQL Task SQL .DimAccount Row Count to set the RowCount_DimAccount.  SSIS Codes: SSIS Codes Folder SSIS Project SSIS Package (Master) SSIS Package (PreLoad) SSIS Package (Load Temp Tables) SSIS Package (Load Dimensions) SSIS Package (Load Fact Tables) SSIS Package 13 | P a g e C:\TGIEH_SSIS_ETL_Framework\TGIEH_SSIS_ETL_Framework_Solu tion\Sol_2_AWDataWarehouseRefresh_Using_TGIEH_SEF_V2 Sol_2_AWDataWarehouseRefresh_Using_TGIEH_SEF_V2.dtsx .dtsx AWDWRefresh_LoadFact.ETL_Rowcount? Below are the steps to capture the record count for DimAccount:  Create the variable: RowCount_DimAccount   Change RaiseChangedEvent True Because it’s used Bulk Insert Task. Have you noticed the records in dbo.dtsx AWDWRefresh_TempTables. and use Excel Add-in SQL Utilities to output the tables to worksheets.dtsx AWDWRefresh_PostLoad.dtsx AWDWRefresh_LoadDim.SSIS ETL FRAMEWORK BY TRACKING GUIDS IN EVENT HANDLERS packages according to above AWDWRefresh Tasks Grouping.

SSIS ETL FRAMEWORK BY TRACKING GUIDS IN EVENT HANDLERS (Poast Load) Output Sample: Output Sample Folder SQL Tables Output Sample Text Files Outpue Sample C:\TGIEH_SSIS_ETL_Framework\Outputs Sol_2_AWDataWarehouseRefresh_Using_TGIEH_SEF_V2.zip 14 | P a g e .xlsx Sol_2_AWDataWarehouseRefresh_Using_TGIEH_SEF_V2_Log.

0\Common7\IDE\PrivateAssemblies\ProjectItems\DataTransformationProject\DataTransf ormationItems (64 bit machine) and save it as a template so you can add it to your SSIS project (see Figure 10 – Add New Item from Visual Studio install templates). add a Integration Services Project.: E Package template. T Package and L Package Template.e. and add the Framework package TGIEH_SEF_Template_Pkg.dtsx) to the project 3) Review and setup SSIS Package Configurations and SSIS XML configuration file 4) Review and define SSIS Variables 5) Review and create SSIS Connections 6) Review and Setup SSIS Package Properties 7) Review and setup SSIS configuration table dbo.dtsx (C:\TGIEH_SSIS_ETL_Framework\TGIEH_SSIS_ETL_Framework_Solution\Sol_1_TGIEH_SSIS_ ETL_Framework_Template\TGIEH_SEF_Template_Pkg.TGIEH_SEF_Configurations in TGIEH_SSIS_ETL_Framework 8) Reset GUIDs by using BIDS Helper 9) Copy the package to C:\Program Files (x86)\Microsoft Visual Studio 9. Figure 10 – Add New Item from Visual Studio install templates 10) Start your SSIS package development 11) You might create many templates by functionalities. 15 | P a g e . i. SSIS Development Using the SSIS ETL Framework Recommend to use the following steps for SSIS ETL development using the SSIS ETL Framework: 1) Deploy the Framework DLL (Refer to Installation Instructions) 2) Create a Visual Studio Solution.SSIS ETL FRAMEWORK BY TRACKING GUIDS IN EVENT HANDLERS VI.

16 | P a g e . you might change above to accommodate your development and environment.SSIS ETL FRAMEWORK BY TRACKING GUIDS IN EVENT HANDLERS Once you’re familiar with the framework.

ETL Logging Reporting Creation In above demos. ETL_Rowcount. a. Further Developments There are three customizable components and I am going leave the creations and customizations to you because you know your environment better. b.ETL_Job. I used my SQL development tool . You can modify the subject and content (body) format of email in the stored procedure to suit your environment.SSIS ETL FRAMEWORK BY TRACKING GUIDS IN EVENT HANDLERS VII. see Figure 11 – Email Notifications Content Sample. 17 | P a g e . Email Notification Customization The DLL sends out Email notifications by calling the stored procedure dbo.sql). and ETL_Error -. ETL_Package. ETL_Task.for ETL logging and deploy them to SQL Server Report Server so users can view the logs through SQL Server Report Manager.Excel add-in SQL utilities to show you the results but you should create SSRS Reports with five tables -.usp_ETL_SendMail (C:\TGIEH_SSIS_ETL_Framework\SQL\usp_ETL_SendMail.

Debug Output Text Files Customization The C# script task in the Event Handlers not only calls the DLL but also write variables information to the text files if the flag ETL_Write_Text_Log is set to True and the file name ETL_Log_File is assigned with path and file name. You can customize the output by modifying the Write to Log File section (see Figure 12 – C# Write to Log File Section) in the C# script to meet your troubleshooting needs. Figure 12 – C# Write to Log File Section 18 | P a g e .SSIS ETL FRAMEWORK BY TRACKING GUIDS IN EVENT HANDLERS Figure 11 – Email Notifications Content Sample c.

Happy SSIS Coding! 19 | P a g e . please send me an email to charles138@gmail. Suggestions If you have any comments or suggestions regarding SSIS ETL Framework by Tracking GUIDs in Event Handlers.com.SSIS ETL FRAMEWORK BY TRACKING GUIDS IN EVENT HANDLERS VIII.