'Extract, Transform and Load': Data load processing, normally in the context of data warehouse systems.

learn more… | top users | synonyms

0
votes
1answer
8 views

Setting the column width of a column in a table created by logparser

I'm trying to use LogParser for ETL. I am trying to import this file with the following command: "c:\Program Files (x86)\Log Parser 2.2\LogParser.exe" "SELECT Field1 AS CountryCode, Field2 AS Zip ...
1
vote
0answers
43 views

Connection refused when inserting

Description: Machine 1: Linux, PostgreSQL 9.3, on local, 10.12.10.123 Machine 2: Linux, PostgreSQL 9.3, on cloud, 104.10.124.123 Configuration (on both): + postgresql.conf : listen_addresses ="*" + ...
1
vote
1answer
22 views

Are there best practices or standardized syntax for writing transformation logic?

This may not be the appropriate forum, but its the closest I could find. I write a lot of source to target mappings for work. These documents just describe the operations/logic/transformations/etc. ...
1
vote
0answers
19 views

Best way to perform real time analytics on a growing collection of json file ( very large ) [closed]

I have a very large collection of JSON file (tens of millions) on S3 and I need to have some kind of aggregation (perform continuously as the collection grows). The aggregation is mainly counter ...
0
votes
0answers
14 views

Does an Object literal always mean the Predicate and Object form a key-value property on the Subject?

A triple is Subject-Predicate-Object, e.g. Post Office - Not Open - Sunday. Consider another example, Post Office - Size in Square Feet - "4000" For this example, I have two observations: the ...
0
votes
2answers
23 views

Can data be replicated from a slave MySQL back to a master without the slave containing all the master data?

We have a PHP/Symfony 2 application using MySql as the database. The database contains a large amount of data. The application is deployed to a separate server(s) and an empty/clean version of the ...
2
votes
2answers
57 views

How to pass parameter in select query(In with clause) and not in where condition in SQl Server SSIS Package?

I want to pass a parameter (@date1) in my SSIS OLE DB Source so I created a variable and I tried to pass the parameter using '?' but it's showing 'Syntax error, permission violation or other ...
0
votes
0answers
22 views

ETL flow in Fact tables

I've got a simple theoretical question... If we discuss the principles, elements, design and implementation of a typical ETL flow for a dimension table then, in my understanding, (in short) I'd say ...
0
votes
1answer
33 views

Dimensional Modeling and ETL in Redshift

I have been researching Amazon's Redshift database as a possible future replacement for our data warehouse. My experience has always been in using dimensional modeling and Ralph Kimball's methods, so ...
0
votes
1answer
36 views

String to timestamp in datastage 9.1.2

Apologies for asking a question that I suppose is very basic. I'm just starting out with Streams. I'm trying to read a sequential files containing a timestamp field and a value field, like this: ...
1
vote
1answer
26 views

How to load .dat files to oracle database

I have .DAT files extracted from telecom switch. I want to upload this file to an Oracle server using IBM data-stage ETL tool. Unfortunately these files are not readable format. This is a picture ...
0
votes
0answers
45 views

Best way to run an ETL on a production MongoDB database

We have the following environment: 3 servers + 3 replica set servers Each server has 3 shards Our main collection has around 40,000,000 documents that average at around 6kb Our shard key is ...
0
votes
0answers
35 views

Tracking row level status of SSIS ETL process

I have an ETL Process in SSIS in which I am loading around 150 tables in each run - Truncate and Insert. I have four packages each from different sources. Each package loads different tables. ...
1
vote
1answer
47 views

Insert/ Update Scalability (more records, continually longer lookups — solution?)

posted this before but never received a clear answer. I'm working on an ETL project and am basically feeding JSON records of "tickets" into a MySQL database. (actually come to think of it, may be ...
0
votes
1answer
47 views

How to keep the data warehouse synchronised with the source systems? [closed]

I'm a DBA who is responsible for exporting data from an ERP system for Business Intelligence purposes. The data from ERP system is being sent to the data warehouse and they are used in the ETL ...
1
vote
1answer
62 views

SSIS Transaction Problem

I am trying to use SQL transaction with my SSIS package so if any error comes changes are rollback. I am using an "Sequence Container" in Control Flow for this and setting its "Transaction Option" to ...
0
votes
2answers
106 views

SSIS import data or insert data if no match

I have a situation in which I need to import data from one table to another. The challenge here is in tables I have some columns which are references to other table values (kind of foreign key). I ...
0
votes
2answers
86 views

Process staging table by getting unique rows only (incremential)

I had simple problem with etl on staging table that was answered here: Process staging table by getting unique rows only Now after apllying solution I spotted that will be unusable in production ...
-2
votes
1answer
53 views

Process staging table by getting unique rows only [closed]

I have several staging tables where I bulk import data with duplicates and sometimes I upload historical data mixed with real-time data, just importing everything without caring if it is already in ...
3
votes
1answer
98 views

Metadata Headache

A process that I don't have control over is dropping and re-creating tables in a MySQL database every night. This wouldn't be a problem (I think), if it re-created the tables identically every time. ...
4
votes
1answer
168 views

SSIS Multicast performance question

I'm wondering how multicast actually works. So here is my setup. I have one source flat file(csv) of about 100k rows, maybe about 250-300 columns, and I'm multicasting that file into 7 one-to-one ...
1
vote
0answers
107 views

ETL - Moving data from Postgres/Oracle to a TripleStore (RDF)

I'm looking for pointers on how to handle an ETL process that I have to implement for my organization. I know how to query SQL database but i'm not SQL DB specialist. I work more in the sphere of the ...
1
vote
1answer
764 views

SSIS Packages with SQL Server Express 2008 R2

I understand from various questions [1] [2] that SQL Server Express 2008 R2 doesn't support SSIS packages and that there's only primitive support for ad-hoc data imports / exports. I have already got ...
0
votes
0answers
36 views

Tables with same schema in DB for different sources. Bad pattern?

The main purpose of the system being developed is to collect data from different sources, process and standardize it, and then add it to the DB. After information from all sources is collected it ...
-1
votes
3answers
354 views

How to load multiple values in cell from Excel to SQL Server many to many table [closed]

I have values in excel like this. StudentNumber| Subjects 1| english, science, maths 2| maths, biology 3| history, zoology I have Student table in database ...
1
vote
1answer
399 views

How to compare data between staging and operational tables?

I just want to ask how to compare two database tables, staging vs. operational tables. After we did the transformation and convert it to correct datatype like convert flag column (Y/N) nvarchar to ...
1
vote
0answers
1k views

How can I load csv data into mongodb and choose the types I'd like to have created for each column?

I'm looking for a way to load csv data in mongodb and specify the types that I'd like to have created for each column? I've been using mongoimport but I find it quite painful because there doesn't ...
-1
votes
2answers
44 views

how to identify why lookup failed in SSIS

i have created an ETL package and having a lookup to match userid in db. i checked in dataviewer what data is going as input to my lookup. the data is correct and the table i used in lookup to match ...
0
votes
0answers
122 views

Selectively load data using Oracle 11g r2 External Table Preprocessor

I am using the Preprocessor feature in Oracle 11g r2 to load several data files in a table. However, the raw files seem to have certain rows that shouldn't be imported with the rest. For example: ...
1
vote
2answers
612 views

What is a good solution for reliably transferring data between SQL Server and PostgreSQL? [closed]

In analyzing some of our business data, we're realizing that we need to join data across disparate data sources. For example, our application data is warehoused in Postgres (ported from MongoDB via ...
1
vote
1answer
198 views

Clustered Index with nightly ETL process

We have a nightly ETL process that involves data from many different sources being flattened, and loaded into a completely denormalized table on SQL Server 2008 for reporting. The clustered index on ...
1
vote
4answers
511 views

Looking for ideas on maintaining data warehouse in close to real-time - ETL tools, etc

I have done a fair number of traditional data warehouse implementations where the loads were done in a batch-oriented manner, i.e. the data is refreshed nightly or, at most, every few hours. I now ...
1
vote
1answer
69 views

How to deal with multiple error columns in data flow

I'm writing a simple package to load data into tables and a I have multiple destinations within the same package. I'm attempting to first attempt batch inserts but want to fall back to row by row in ...
0
votes
0answers
731 views

ssis ado.net destination has failed to acquire the connection

I am Transferring data from sql server source db to sql server destination db (cross db). but some times getting error "ssis ado.net destination has failed to acquire the connection.The connection may ...
0
votes
0answers
44 views

Merge data using Integration Service

Please Consider this scenario: I have a table in my database. I want move this data in my OLAP database using SSIS.I can move all record from my table to OLAP database.The problem is I don't know how ...
4
votes
1answer
4k views

MySQL failing and very slow on importing a large file

I have a 47 GB MySQL dump of a single table: http://dumps.wikimedia.org/commonswiki/latest/commonswiki-latest-image.sql.gz I ultimately want it into PostgreSQL, but since I didn't figure out an easy ...
0
votes
2answers
84 views

General ETL principle - ETL from unknown number of tables? [closed]

AFAIK, ETL integrates data from a fixed number of sources/tables. Is it possible that ETL can integrate data from an unknown number of tables which might be added in the future, given that these ...
1
vote
0answers
46 views

Data Integration [closed]

This is not exactly a DBA question but definitely a DB question and I think StackExchange DBA is my best bet for this question. Here I go : are there any frameworks ( free / open source preferred ) ...
4
votes
1answer
361 views

Large difference in disk space used by partitions

I have a partitioned table that contains about 155 million records total. It's partitioned by year and the partitions for both the current and previous years contain about 25 million records each. ...
2
votes
4answers
401 views

SQL Server: Database integration and synchronization

I have n SQL Server databases with the same data schema. On the other hand i have a web application that needs to retrieve, for some entities, the union of the information stored into the n ...
2
votes
2answers
193 views

Best Practices for Testing Application Dashboard Data

We have a dashboard that contains metrics based on a 6 month reporting period. Users log in and see their metrics based on the current 6-month reporting period. Every 6 months, we load new metrics ...
4
votes
4answers
2k views

best ETL design to transfer transaction tables records into the data-warehouse

I have 2 type of tables to populate the data-warehouse with every day, lookup tables or configuration tables with few 100s records, and thats easy where i just truncate and refill the table. but for ...
3
votes
1answer
106 views

Replace column values?

I recently got into SSIS and haven't been able to find what I am trying to achieve anywhere on the web. I am trying to take some rows as input, and then replace one specific column's value with ...
4
votes
1answer
5k views

Importing a large column (8000 chars) from Excel using SSIS

I'm trying to set up a regular import of an excel spreadsheet that we get from a vendor. I'm using SQL 2008 R2 SSIS to import it into a table. The problem connection manager is an OLE DB connection ...
3
votes
2answers
910 views

Is this a good strategy for importing a large amount of data and decomposing as an ETL?

I have a set of five tables (a highly decomposed schema for an ETL if I understand the nomenclature) that I'm going to load via bulk import, then run some inserts from those five tables into a SLEW of ...
7
votes
3answers
133 views

Tablespace on ephemeral storage

For performance reasons in some scenarios, e.g. Amazon EC2, you have access to a faster and cheaper storage device, which loses all its data on reboots, so it is called "ephemeral". This question is ...
0
votes
1answer
516 views

How to perform ETL from RDBMS to Neo4j

I have a mature 50+ tables web application based on mySQL. In order to do some advanced data mining I want to use Neo4j and the goodnes of cypher. However I'm having a hard time migrating my data from ...
1
vote
1answer
121 views

Tips for catching changes in table data over time interval for export to Data Warehouse

I've recently been tasked with exporting data from one of our databases to a central Data Warehouse. The export is supposed to be incremental since a full load every 24 hours would be a massive waste ...
0
votes
0answers
69 views

Inserting Records - How to get next incremented PK for each insert

I'm using spoon to look for Devices that have several attributes that define its location, and then create a Cabinet in that location if one doesn't exist. Is there an operation that lets me get a ...
1
vote
0answers
147 views

Transfer data from DynamoDB to SQL server 2008?

I want to transfer data from amazon dynamo DB to SQL server 2008. What are the tools or methods which can be used to do this ?