'Extract, Transform and Load': Data load processing, normally in the context of data warehouse systems.

learn more… | top users | synonyms

2
votes
3answers
1k views

Is it good idea to change Simple Recover Model into Bulk-Logged on ETL system?

We are having an ETL process which inserts lots of data into tables. This database is set to Simple Recovery Model and the transaction log is growing a lot. I was thinking that would it help to set ...
8
votes
2answers
5k views

PostgreSQL for high volume transactions and for Data warehousing

Am quite new to PostgreSQL, I have never done a large deployment using it before. But, I have good experience in enterprise solutions and I want to try and apply some of what I learned using ...
2
votes
1answer
85 views

Data Extract and submit across the network

I've been given a somewhat vague requirement regarding a data extract process which I need some help with. I'm fairly new to database warehousing etc. but I will try to explain everything relevant. ...
2
votes
3answers
954 views

Loading a FACT table per day

I have a database with 13 billion rows, per day I have around 20-30 mio rows. On top of this I have one cube, one of its dimensions is DateTime that goes down to milliseconds. To load the fact table I ...
4
votes
3answers
1k views

How can I make my SSIS process consume more resources and run faster?

I have a daily ETL process in SSIS that builds my warehouse so we can provide day-over-day reports. I have two servers - one for SSIS and the other for the SQL Server Database. The SSIS server ...
1
vote
1answer
1k views

Loading multiValued fields into Solr via flat file, and possibly value position preservation in those fields

A hobby web project of mine uses data from many sources, some flat, some highly normalized. My ETL process boils all sources down to one logical table (which is soon to include some multi-valued ...
6
votes
1answer
2k views

Duplicating some rows of a data flow in SSIS

I have a data flow to migrate data from an old database to a new one. The old design had all the data and historical information (changes) stored in a single table with a "version" (incrementing ...
8
votes
4answers
820 views

Is there a standard language/interface for programmatic ETL in SQL Server?

I'm currently in the process of creating ETLs for our data warehouse. We're using SSIS 2008, but we're running into issues, the biggest of which is the difficulty in re-using components. We have ...
1
vote
1answer
491 views

What are the best practices for ETL of data from one OLTP system to another, Part 2?

Ok, I originally started a question a ways back called "What are the best practices for ETL of data from one OLTP system to another?". And as you can probably tell from most of my questions so far, ...
1
vote
1answer
632 views

ActiveX script throwing runtime error inside DTS package

I am a DBA. We have migrated SQL 2000 databases + DTS packages to SQL 2008. One of the DTS packages runs an ActiveX script to open an Excel file (which has runs macros when you open it). Coding ...
4
votes
1answer
954 views

SQL 2008: agent job error in opening excel files using DTS package

I have a SQL 2008 server (SQL2008S1) with Excel installed on it. I had to migrate databases and DTS packages from a SQL 2000 server (SQL2000S2) to this server. 1 DTS package is trying to access an ...
2
votes
1answer
799 views

What are the best practices for ETL of data from one OLTP system to another?

First off, I want to post that I am a developer and not a DBA, so I am trying to approach this from the standpoint of what is the best thing to do, rather than just assuming I can do everything ...
19
votes
7answers
6k views

Any Open Source / free ETL out there?

I was using Pentaho Data Integration even before Pentaho bought it and call it that. I have the last free version. I went on their website recently to see if they had released another version only to ...
4
votes
2answers
463 views

What is the best practice for mapping from natural keys to integer-based keys? (ETL)

This question originates more from an "enterprise architecture" point of view. (Reason being, our company is just starting to get into data management, and actually having DBA's -- I know, horrific ...
4
votes
1answer
129 views

Is there a non-DDL way to preserve back-references when staging data into production?

Let's say I've got this hypothetical schema: Source (OLTP) DB: Table Orders ------------ OrderID int IDENTITY (PK), CustomerID int NOT NULL, OrderAmount decimal NOT NULL Destination (DSS) DB: ...
3
votes
3answers
551 views

How can I use an ETL?

I know that those letters mean Extract, Transform, and Load. But, when I used it at first, I thought that during the Transform phase I could do plenty of different joins on data that I've extracted ...