'Extract, Transform and Load': Data load processing, normally in the context of data warehouse systems.

learn more… | top users | synonyms

1
vote
1answer
1k views

Loading multiValued fields into Solr via flat file, and possibly value position preservation in those fields

A hobby web project of mine uses data from many sources, some flat, some highly normalized. My ETL process boils all sources down to one logical table (which is soon to include some multi-valued ...
6
votes
1answer
2k views

Duplicating some rows of a data flow in SSIS

I have a data flow to migrate data from an old database to a new one. The old design had all the data and historical information (changes) stored in a single table with a "version" (incrementing ...
8
votes
4answers
755 views

Is there a standard language/interface for programmatic ETL in SQL Server?

I'm currently in the process of creating ETLs for our data warehouse. We're using SSIS 2008, but we're running into issues, the biggest of which is the difficulty in re-using components. We have ...
1
vote
1answer
473 views

What are the best practices for ETL of data from one OLTP system to another, Part 2?

Ok, I originally started a question a ways back called "What are the best practices for ETL of data from one OLTP system to another?". And as you can probably tell from most of my questions so far, ...
1
vote
1answer
615 views

ActiveX script throwing runtime error inside DTS package

I am a DBA. We have migrated SQL 2000 databases + DTS packages to SQL 2008. One of the DTS packages runs an ActiveX script to open an Excel file (which has runs macros when you open it). Coding ...
4
votes
1answer
925 views

SQL 2008: agent job error in opening excel files using DTS package

I have a SQL 2008 server (SQL2008S1) with Excel installed on it. I had to migrate databases and DTS packages from a SQL 2000 server (SQL2000S2) to this server. 1 DTS package is trying to access an ...
2
votes
1answer
739 views

What are the best practices for ETL of data from one OLTP system to another?

First off, I want to post that I am a developer and not a DBA, so I am trying to approach this from the standpoint of what is the best thing to do, rather than just assuming I can do everything ...
19
votes
7answers
5k views

Any Open Source / free ETL out there?

I was using Pentaho Data Integration even before Pentaho bought it and call it that. I have the last free version. I went on their website recently to see if they had released another version only to ...
4
votes
2answers
445 views

What is the best practice for mapping from natural keys to integer-based keys? (ETL)

This question originates more from an "enterprise architecture" point of view. (Reason being, our company is just starting to get into data management, and actually having DBA's -- I know, horrific ...
4
votes
1answer
127 views

Is there a non-DDL way to preserve back-references when staging data into production?

Let's say I've got this hypothetical schema: Source (OLTP) DB: Table Orders ------------ OrderID int IDENTITY (PK), CustomerID int NOT NULL, OrderAmount decimal NOT NULL Destination (DSS) DB: ...
3
votes
3answers
508 views

How can I use an ETL?

I know that those letters mean Extract, Transform, and Load. But, when I used it at first, I thought that during the Transform phase I could do plenty of different joins on data that I've extracted ...