SSIS Best Practices, Part 2 And as promised, here is my personal list of SQL Server Integration Services best practices

. This is by no means comprehensive (please see my previous post for links to other best practice resources online) but it's a great starting point to avoiding my mistakes (and the pain that came with them) when you're working with SSIS. Here goes: • Get your metadata right first, not later: The SSIS data flow is incredibly dependent on the metadata it is given about the data sources it uses, including column names, data types and sizes. If you change the data source after the data flow is built, it can be difficult (kind of like carrying a car up a hill can be difficult) to correct all of the dependent metadata in the data flow. Also, the SSIS data flow designer will often helpfully offer to clean up the problems introduced by changing the metadata. Sadly, this "cleanup" process can involve removing the mappings between data flow components for the columns that are changed. This can cause the package to fail silently - you have no errors and no warnings but after the package has run you also have no data in those fields. • Use template packages whenever possible, if not more often: Odds are, if you have a big SSIS project, all of your packages have the same basic "plumbing" - tasks that perform auditing or notification or cleanup or something. If you define these things in a template package (or a small set of template packages if you have irreconcilable differences between package types) and then create new packages from those templates you can reuse this common logic easily in all new packages you create. • Use OLE DB connections unless there is a compelling reason to do otherwise: OLE DB connection managers can be used just about anywhere, and there are some components (such as the Lookup transform and the OLE DB Command transform) that can only use OLE DB connection managers. So unless you want to maintain multiple connection managers for the same database, OLE DB makes a lot of sense. There are also other reasons (such as more flexible deployment options than the SQL Server destination component) but this is enough for me.

but that does not mean that the package developer can be lazy. Annotate like you mean it: You've heard of "test first development. By following this best practice you can keep your packages simpler and more modular. There are plenty of deployment-related tools that ship with SSIS. since access to external resources (file system." I've learned over the years that if I can't describe something in English. and much more maintainable. You can then use expressions based on these variables to use them anywhere within the package. If there is any external resource used by your package." right? This is good. but there is not a lot that you can do to ease the pain related to deployment unless your packages are truly location independent. and it has served me very well over the years. then you will have a consistent configuration approach that is self-documenting and resistant to change. so I tend to go very heavy on the comments in my procedural code. one annotation per data flow component and any additional annotations that make sense for a given design surface. but think of what you would want someone to do if you were going to open up their packages and try to figure out what they were trying to . I've carried this practice over into SSIS. file or folder path. If it’s external. One target table per package: This is a tip I picked up from the great book The Microsoft Data Warehouse Toolkit by Joy Mundy of The Kimball Group.) is performed (almost) consistently through connection managers.• • • • Only Configure package variables: If all of your package configurations target package variables. etc. you need to drive the values for the connection information (database connection string. whatever) in a package configuration so they can be updated easily in any environment without requiring modification to the packages. I'm going to struggle doing it in C# or whatever programming language I'm using. and like to have one annotation per task. deployment tops the list. This may seem like overkill. database. configure it: Of all of the aspects of SSIS about which I hear people complain. The design of SSIS goes a long way to making this possible. but I believe in "comment first development. URL.

But there are some data flow transformations that perform row-by-row operations. As with row-based operations. Question everything.• • • do. you need to aggressively pursue alternative approaches. Avoid asynchronous transforms: In short. This just does not scale for larger (or even "large-ish") data volumes. Odds are you will never understand a complex . Although there are valid uses for these transforms. Avoid row-based operations (think “sets!”): The SSIS data flow is a great tool for performing set-based operations on huge volumes of data . Really know your data – really! If there is one lesson I've learned (and learned again and again .that's why SSIS performs so well. and how.see my previous blog post about real world experience and the value of pain in learning .-) it is that source systems never behave the way you expect them to and behave as documented even less frequently. So annotate liberally and you won't be "that guy" . make sure that you've exhausted the alternatives and that you do performance testing early with real data volumes. and when. so if you find yourself thinking about using them.but also the dynamic nature of the data . they can easily cause data flow performance to grind to a halt. they tend to be very few and far between. and why. and make sure that you're testing early with representative volumes of data. and then test to validate the answers you retrieve. and although they have their uses. the Slowly Changing Dimension transform and the tried-and-true Lookup transform when used in non-cached mode. The danger here is that these transforms will work (and possibly even work well) with small number of records. and what processes initiate those changes. any fully-blocking asynchronous data flow transformation (such as Sort and Aggregate) is going to hold the entire set of input rows in memory before it produces any output rows to be consumed by downstream components. but completely choke and die when you need them to do the heavy lifting. You need to understand not only the static nature of the data .how it changes when it changes. the Fuzzy Lookup transform. These transformations include the OLE DB Command transform.the one everyone swears about when he's not around.what is stored where .

I mean the . I don't mean data source components.I think that was Barry Manilow) and have incredibly sophisticated capabilities work efficiently with huge volumes of data. Treat your packages like code: Just as relational databases are mature and well-understood.Microsoft. This will be the best road forward for a smooth deployment story. Oracle and the rest of the database vendors have invested millions of man years and billions of dollars[1] in tuning their databases. Really.they are a feature of Visual Studio. if your data source is a relational database. merging or performing other expensive operations in your data flow when you could do it in the data source as part of your select statement? It is almost always significantly faster to perform these operations in the data source. And if you are pulling data from sources like flat files which do not provide any such capabilities there are still occasions when it is faster to load the data into SQL Server and sort. So why would you consider sorting.it has amazing performance when used properly . Instead. and this is a significant difference.ds files that you can add to your SSIS projects in Visual Studio in the "Data Sources" node that is there in every SSIS project you create. so make sure you are very friendly (may I recommend including a line item for chocolate and/or alcohol in your project budget?) with the business domain experts for the systems from which you will be extracting data. whereas using Data Sources is a dead-end road. use package configurations to store the connection string for the connection managers in your packages. Why not use that investment when you can? Don’t use Data Sources: No. Do it in the data source: Relational databases have been around forever (although they did not write the very first song .but also don't think that it is the right tool for every job. Remember that Data Sources are not a feature of SSIS . aggregating. Remember . Please do not think that SSIS data flow doesn't perform well .• • • source system well enough. To nowhere. so is the value of using a repeatable process and tools like source code control and issue tracking software to manage the software development . aggregate and join your data there before pulling it back into SSIS.

lifecycle for software development projects. All of these tools. But now we're using Visual Studio for SSIS development and have many of the same capabilities to do things right as we do when working with C# or C++ or Visual Basic. processes and lessons apply to SSIS development as well! This may sound like an obvious point. often through pain) over the years. . but with DTS it was very difficult to "do things right" and many SSIS developers are bringing with them the bad habits that DTS taught and reinforced (yes. Some of the details may be different. but all of the principles apply.