'Extract, Transform and Load': Data load processing, normally in the context of data warehouse systems.

learn more… | top users | synonyms

0
votes
3answers
38 views

Can data be replicated from a slave MySQL back to a master without the slave containing all the master data?

We have a PHP/Symfony 2 application using MySql as the database. The database contains a large amount of data. The application is deployed to a separate server(s) and an empty/clean version of the ...
3
votes
1answer
76 views

How do I load data from a single table to multiple tables & keep referential integrity

Using SQL Server 2008R2. I have a single table with a bunch of columns. I have built a new database with multiple tables to copy the data too. How do I copy the data, and still keep the ...
2
votes
1answer
43 views

Are there best practices or standardized syntax for writing transformation logic?

This may not be the appropriate forum, but its the closest I could find. I write a lot of source to target mappings for work. These documents just describe the operations/logic/transformations/etc. ...
1
vote
1answer
73 views

SQL Server : ETL Stage Index

We have a very large staging table (> 80 GB). From our source system we load invoice data in the staging table. From the staging we transform the data and load into DWH/Fact. Every day we delete the ...
1
vote
1answer
63 views

DB2 Database Conversion Workbench Issue

I am currently trying to convert a SQL Server database to DB2. I am using the Data Conversion Workbench (DCW) for IBM Data Studio. The issue I have is that DCW can not seem to correctly extract data ...
1
vote
1answer
1k views

Dimensional Modeling and ETL in Redshift

I have been researching Amazon's Redshift database as a possible future replacement for our data warehouse. My experience has always been in using dimensional modeling and Ralph Kimball's methods, so ...
0
votes
1answer
56 views

ssis surrogate key in parent child tables

I have two tables Person & Ownership having one to many relationship (each person have many ownerships). Primary Key is person_id of type GUID. I am developing SSIS package to load data from both ...
3
votes
0answers
962 views

SSIS Recordset Destination error

I have an SSIS package targeted at SQL Server 2012. I have it deployed into the Stored Packages in Integration Services and then have a SQL Server Agent job which executes it. The first part of the ...
2
votes
0answers
202 views

Transfer data from DynamoDB to SQL server 2008?

I want to transfer data from amazon dynamo DB to SQL server 2008. What are the tools or methods which can be used to do this ?
2
votes
0answers
929 views

How to do automated archiving for a SQL Server database based on interpretation of database metadata?

I am looking for a solution giving me automated archiving of SQL Server databases based on a smart interpretation of table relationships that is able to archive records from related tables minimizing (...
1
vote
0answers
27 views

Retrieve data from OLTP MySQL to SQL Server DWH using SSIS

I'm running a BI solution currently reading data from an OLTP SQL Server 2008 to a SQL Server 2012 data warehouse. The ETL phase is implemented with SSIS and runs twice a day. This phase relies on a ...
1
vote
0answers
34 views

SSIS Guarantee that all data will be moved to target when running data flows in parallel

My package has the following two tasks which run in parallel. DF - Move source data to target loads a copy of all source data to target via OLEDB Dest - Target and also loads a copy of source PK ...
1
vote
0answers
25 views

Logging within a procedure's transaction

I have an ETL process running inside a PostgreSQL 9.3 procedure on AWS RDS. The procedure normally takes about ten minutes to run, but I'm having a rare issue where it takes several hours and never ...
1
vote
0answers
113 views

Connection refused when inserting

Description: Machine 1: Linux, PostgreSQL 9.3, on local, 10.12.10.123 Machine 2: Linux, PostgreSQL 9.3, on cloud, 104.10.124.123 Configuration (on both): + postgresql.conf : listen_addresses ="*" + ...
1
vote
0answers
120 views

ETL flow in Fact tables

I've got a simple theoretical question... If we discuss the principles, elements, design and implementation of a typical ETL flow for a dimension table then, in my understanding, (in short) I'd say ...
1
vote
0answers
227 views

ETL - Moving data from Postgres/Oracle to a TripleStore (RDF)

I'm looking for pointers on how to handle an ETL process that I have to implement for my organization. I know how to query SQL database but i'm not SQL DB specialist. I work more in the sphere of the ...
1
vote
0answers
263 views

ETL import more dimensions from only 1 column

I was requested at work to learn ETL. I have already created a database without using ETL with some dimensions and consolidations, etc, containing above 1 billion numerical values. Now I have to ...
0
votes
0answers
13 views

Data Cleansing Using DQS

My question is regarding the installation of DQS server. I have a production server and a data warehouse server, both are synced. Is it the best practice to cleanse the data in production using the ...
0
votes
0answers
25 views

Loading Data from url

I am trying to create a plan for data load and wanted to bounce off best practice ideas and learn about amazon RDS. I have multiple url sources of data and a RDS MYSQL instance to store them. I want ...
0
votes
0answers
36 views

Do Queries on a MySQL Federated table lock the remote table?

We are trying to use Federated tables as an experiment where a remote server copies from the Federated tables to a local db with OLAP designed tables. My question is whether the SELECT queries on the ...
0
votes
0answers
13 views

Isolation Level Impact on Selecting a Row Moving Between Two Tables

I have two tables (identical structure, one for current data and one for archive) being used as a source for a datamart staging table. I construct a SSIS data flow that runs SELECTs on both tables ...
0
votes
0answers
38 views

commonly used diagram for ETL

A the title indicates. Am i wondering what is the most commonly used diagram to explain an ETL process? In my case the process is rather linear, would a flowchart suffice? or is a Use-case also ...
0
votes
0answers
195 views

A “best practice” Postgresql function template for ETL processes

I use SQL Server in my day job, and have a personal project to build a star schema data warehouse in Postgres 9.4 on Ubuntu 14.04 to extend my knowledge (it'll be approximately 300GB uncompressed). I'...
0
votes
0answers
142 views

How to import data from Sqlite into SQL Server 2012

I need to import data from sqlite into SQL Server so we can include it in our data warehouse. Based on previous system volume. Id expect there to be 50-100,000 rows after a couple of years. For ...
0
votes
0answers
21 views

SSIS - different versions of the project

I am designing the ETL process. I created a package that extracts the data from the source using stored procedures. The SQL query looks like this: EXEC BI.ProcedureName The procedure returns data ...
0
votes
0answers
69 views

Batch Processing in Talend open studio for data integration

I am using TOS for data integration to transfer a data between mysql and postgres. Its transferring data between small tables successfully. But a table has 3GB data. while trying to transfer data its ...
0
votes
0answers
10 views

Pentaho getfilename error

I am try to send 20 mail with attachments using pentahi DI, Jobs are like below Get File Names--->Add constants--->Mail But some mails are not send with below error, Thay say files are not ...
0
votes
0answers
67 views

Tools to insert data faster in MySql database

I am trying to insert data from a flat file which is around 9 GB with about 15 million rows, I used SQl-server 2012 for that. I noticed data insert is much faster with sql server and it took around 30 ...
0
votes
0answers
553 views

Best way to run an ETL on a production MongoDB database

We have the following environment: 3 servers + 3 replica set servers Each server has 3 shards Our main collection has around 40,000,000 documents that average at around 6kb Our shard key is hashed(...
0
votes
0answers
41 views

Tables with same schema in DB for different sources. Bad pattern?

The main purpose of the system being developed is to collect data from different sources, process and standardize it, and then add it to the DB. After information from all sources is collected it ...
0
votes
0answers
186 views

Selectively load data using Oracle 11g r2 External Table Preprocessor

I am using the Preprocessor feature in Oracle 11g r2 to load several data files in a table. However, the raw files seem to have certain rows that shouldn't be imported with the rest. For example: ...
0
votes
0answers
1k views

ssis ado.net destination has failed to acquire the connection

I am Transferring data from sql server source db to sql server destination db (cross db). but some times getting error "ssis ado.net destination has failed to acquire the connection.The connection may ...
0
votes
0answers
48 views

Merge data using Integration Service

Please Consider this scenario: I have a table in my database. I want move this data in my OLAP database using SSIS.I can move all record from my table to OLAP database.The problem is I don't know how ...
0
votes
0answers
77 views

Inserting Records - How to get next incremented PK for each insert

I'm using spoon to look for Devices that have several attributes that define its location, and then create a Cabinet in that location if one doesn't exist. Is there an operation that lets me get a ...