Delta Fact Table Load Speed Tests

26/5/2014 Delta Fact Table Load Speed Tests
http://www.bidn.com/blogs/MarkGStacey/bidn-blog/606/delta-fact-table-load-speed-tests 1/10
Delta Fact Table Load Speed Tests
Who is online? 0 guests and 1 members
Member login | Become a member
-- search
Home
Articles
Forums
Blogs
Jobs
Events
Files
Spaces
People
login
register
Home Blogs MarkGStacey Delta Fact Table Load Speed Tests
Delta Fact Table Load Speed Tests
change text size: A A A
Comments (1)
posted 4/25/2010 by MarkGStacey - Views: [6198]
In the BI environment , an incremental load of a Fact table (also called a delta) is a process for which Microsoft has not provided us with a prebuilt tool.

I'm going to be writing an article comparing several methods for doing these incremental loads, what I've listed below are 4 example techniques, as well as a comparison to a 3rd party tool
Print
from Pragmatic works.

OUT of scope are the Change Data Capture techniques that only draw changed data from the source system. The assumption here is that these are loads that are either reloads or from file
dumps. I am going to be testing from a Landing table where the data is unscrubbed. Have a look here for CDC http://www.bidn.com/articles/data-warehousing-design/31/incremental-
loads-with-change-data-capture-%E2%80%93-part-1-enabling-change-data-capture

I have listed 4 representative scenarios, an incremental load, a full reload without clearing the warehouse, a full reload with clearing the warehouse, and a batch reload scenario.

Please let me know any other techniques or scenarios you believe are essential to cover!!



Technique 1
Scrub to stage table
Orphan handle
Merge into Prod
Technique 2
Scrub to Stage table
Orphan handle
Delete matches from prod
Insert into prod
Technique 3
Scrub in memory
Orphan containing records to stage table
Full cache In-memory lookup of existing fact records (filtered by date)
New records direct to prod
Unchanged records ignored
Changed records to stage
Orphan handling
Merge into prod
Technique 4
Scrub in memory
Orphan handling
Insert into prod
Technique 5
Scrub in memory
3rd party delta load tool
Scenario 1 - Incremental load
Production : 10 million rows
100k new rows
Scenario 2 - Full reload without clear
100k new rows
8 million unchanged rows
2 million changed rows
Scenario 3 - Batch reload
100k new rows
800k unchanged rows
200k changed rows
Scenario 4 - Full reload with clear
Production : 0 rows
10 million 100k new rows

Technique 1
Scrub to stage table
Orphan handle
Merge into Prod
Technique 2
Scrub to Stage table
Orphan handle
Insert into prod
Technique 3
Scrub in memory
Orphan handling
Merge into prod
Technique 4
Scrub in memory
Orphan handling
Insert into prod
Technique 5
Scrub in memory
3rd party delta load tool
Scenario 1 - Incremental load
100k new rows
Scenario 2 - Full reload without clear
100k new rows
8 million unchanged rows
2 million changed rows
Scenario 3 - Batch reload
100k new rows
800k unchanged rows
200k changed rows
Scenario 4 - Full reload with clear
Name:* Email:* Website:
Production : 0 rows
10 million 100k new rows

tags : ssis delta fact load incremental

0
/5
Avg: 0/5: (0 votes)
Comments (1)
tamzyn said:
This is a great idea for an article. I am currently working on Secinario 1 and technique 2. I think this is the most efficient method. Would be great to hear what worked best for you.
Question is Have you written it yet?
5/17/2010

Comment

Add a new comment
Type the characters you see in the image: *
Add comment
MarkGStacey's latest blog posts
Persisting DAX queries
11/3/2012 by MarkGStacey
Microsoft Integration Stack
Visual Intelligence
MarkGStacey.NET
Visio Shapes for SQL 2012 - SSAS (OLAP & Tabular), Master Data Services, and Data Quality Services Servers
Latest community blog posts
Automotive code scanners Diagnose vehicles Step base on Year/Model
5 hours ago by fcar
Isuzu Engine Couldnt Throttle Up During Driving
5 hours ago by fcar
nike free 5.0+ mens after some time
9 hours ago by naicha
SQL Saturday #298 Session Material Now Available (SSAS Tabular Models)
13 days ago by DustinRyan
Using Power Query to tell your story form your Facebook Data.
20 days ago by Anil
Copyright 2014 Axero Solutions LLC.
Contact Us
Business Intelligence Developer Network powered by Communifire Version 2.5.4240.2964
Choose your language: English

Delta Fact Table Load Speed Tests

Uploaded by

Document Information

Original Description:

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Delta Fact Table Load Speed Tests

Uploaded by

Copyright:

Available Formats

26/5/2014 Delta Fact Table Load Speed Tests

You might also like