Take the 2-minute tour ×
Database Administrators Stack Exchange is a question and answer site for database professionals who wish to improve their database skills and learn from others in the community. It's 100% free, no registration required.

What is the best extraction strategy for approximately 200 SQL Server 2005 source databases (same schema) for a daily load into a staging area in preparation for data warehouse cleansing, de-duplication, and transformations?

So far I have envisioned the following possibilities:

  1. Transactional replication: create 200 SQL Server 2008 R2 subscribers pulling data from their respective publishers in 2005. Enable Change Data Capture on the required tables in the subscriber and the shadow tables in order to perform incremental loads into our staging database.
  2. Rowversion: add a rowversion column on each required source table and use that in conjunction with an SSIS process to extract the change data to the staging database directly.
  3. BCP files: create an automated task to do a nightly dump of BCP files from all source tables. Use SSIS to load these tables into the staging database as part of a full load (as opposed to incremental).

Additional thoughts:

  1. I am nervous about the administrative and hardware overhead required to support an entirely new transactional replication topology on 200 databases.
  2. The total combined size of the databases is about 100GB. But a majority of that is part of the transaction logs and other tables which won't be used in any fact or dimension. In other words, the BCP files won't be enormous which is why I'm considering a full extraction strategy even though everything I've read recommends against it.
  3. I'm open to suggestions, documents, etc.
share|improve this question
add comment

3 Answers 3

up vote 5 down vote accepted

If you have 200 identical sources then you can parameterise a SSIS package with the data source and kick off nultiple threads. These can be controlled within the package by a foreach loop or from an external source that kicks off the extractors with a parameter.

You could consider a full load for relatively small dimensional sources and an incremental load for transactional data. This would require you to have persistent dimensions, but this is fairly straightforward to do with MERGE operations, or a pre-load area and dimension handler if you need slowly-changing dimensions.

You may wish to consider giving each source its own staging area (maybe a schema for each source in the staging database). This eliminates locking issues on the staging tables. Build a set of views over the staging tables (essentially just set of unions that correspond to each of the source tables) that includes data source information. These can be generated fairly easily, so you don't have to manually cut and paste 200 different queries into the union. Once you've staged the data then ETL process can read the whole lot from the view.

This allows the ETL to run in one hit, although you will have to come up with a strategy to deal with extract failures from individual systems. For this, you might want to look into an architecture that deals with late arriving data gracefully, so you can catch up individual feeds that had transient issues.

BCP

For 200 simple extracts, BCP is probably a good way to go. The sources are all identical, so the BCP files will be the same across sources. You can build a load controller with SSIS. Getting multiple threads to read the top off a common list would require you to implement synchronised access to the list. The SSIS process has a bunch of loops running in parallel in a sequence container that pop the next item, execute it and update the corresponding status.

Implementing the 'next' function uses a sproc running in a serializable transaction that pops the 'next' eligible source off the list and marks it as 'in progress' within the transaction. This is a 'table as queue' problem, but you don't have to implement synchronised inserts - a whole batch can be pushed into the table at the start of the run.

Structure the individual extract process so that it tries once or twice again if the first attempt fails. This will mitigate a lot of failures caused by transient errors. Fail the task if it fails twice, and structure the ETL so it is resilient to individual extraction failures.

Incremental loads

An incremental loader is probably not worth bothering for dimension tables unless you have a really big dimension that shows real performance issues. For the fact table data sources it probably is worth it. If you can add a row version to the application table with a timestamp column or some such, you can pick up stuff that's new. However, you will need to track this locally to record the last timestamp. If there is an insert or update date on the data you may be able to use that instead.

Full Loads

What could possibly go wrong?

200 processes kicking off to do a full load places a load spike on the network and possibly the staging database. This could lead to all sorts of transient issues like timeouts. For small dimension tables it's probably not such a big issue. However for 100GB there are quite a wide variety of issues - WAN saturation, locking (although the right staging architecture will mitigate that), availability of sources. The longer the extract process has to run the bigger influence environmental factors have on the reliability of the process.

There are quite a lot of imponderables here, so YMMV. I'd suggest an incremental load for the larger tables if possible.

share|improve this answer
    
Thanks for the detailed answer. Do you have any thoughts on using replication as a low impact means of mirroring the source data? –  8kb Apr 30 '12 at 19:08
    
@8kb - Maybe if you have to extract over a slow WAN connection you could replicate to a local server and run the ETL off that. –  ConcernedOfTunbridgeWells Apr 30 '12 at 20:22
add comment

You can use an SSIS package which loops through the source databases exporting the needed data to the destination database. With some work you could make this multi-threaded and do multiple databases at once.

share|improve this answer
add comment

The IRI Workbench (Eclipse GUI) supports bulk (multi-table) extraction, mapping (transformations, if needed, with rules), and bcp loads; specifically, its' reorg wizard is mentioned here. I wouldn't start with 200 sources at once, but see how it goes for the first 10 to determine if its multi-script approach (which I like for portability and easy modification) works there. Then build on that approach and see how it goes; IRI tools are for bulk data movement; the GUI is for convenience in specifying their use in multi-source situations

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.