'Extract, Transform and Load': Data load processing, normally in the context of data warehouse systems.

learn more… | top users | synonyms

0
votes
0answers
34 views

What is the best ways to filter data on the fly

We are implementing a new plan for ETL database, but we do not want to store the intermediate table on our local sql server (only want those records that we need). any body know the best ways to do ...
1
vote
1answer
37 views

DB2 Database Conversion Workbench Issue

I am currently trying to convert a SQL Server database to DB2. I am using the Data Conversion Workbench (DCW) for IBM Data Studio. The issue I have is that DCW can not seem to correctly extract data ...
6
votes
1answer
105 views

How do you index alphanumeric foreign keys on large fact tables?

I have a field that is an alphanumeric field that ideally is a encrypted field for a non-unique identifier. It's used to relate other rather large fact tables in a many-to-many relationship. I do not ...
5
votes
1answer
57 views

Are there practical scenarios where inserts + deletes are faster than updates? (SQL Server)

I'm just wondering if there are common scenarios where an insert/ delete combination is faster than an update else insert function. Here is my specific example. I have to update a database with ...
6
votes
2answers
78 views

SQL server Database statistics degrading after execution

I have been experiencing timeouts while querying my SQL Server DB in particular one table. I tried updating the statistics and it fixed the performance issues for a couple of days. I started digging ...
1
vote
1answer
37 views

How to set SSIS flat file connection row delimiter to blank line?

I have a text file with the following format: DataColumn1 DataColumn2 DataColumn3 DataColumn1 DataColumn2 DataColumn3 DataColumn1 DataColumn2 DataColumn3 Each row with text in the file delimits a ...
0
votes
0answers
24 views

commonly used diagram for ETL

A the title indicates. Am i wondering what is the most commonly used diagram to explain an ETL process? In my case the process is rather linear, would a flowchart suffice? or is a Use-case also ...
0
votes
0answers
54 views

A “best practice” Postgresql function template for ETL processes

I use SQL Server in my day job, and have a personal project to build a star schema data warehouse in Postgres 9.4 on Ubuntu 14.04 to extend my knowledge (it'll be approximately 300GB uncompressed). ...
0
votes
1answer
66 views

Extract data from Oracle DBF files

I'm completely new to Oracle and I was provided with what appears to be an Oracle database backup (170 DBF files and 3 CTL files). I'd like to extract the information using SSIS or Pentaho's Kettle ...
0
votes
0answers
50 views

How to import data from Sqlite into SQL Server 2012

I need to import data from sqlite into SQL Server so we can include it in our data warehouse. Based on previous system volume. Id expect there to be 50-100,000 rows after a couple of years. For ...
1
vote
0answers
29 views

SSIS Guarantee that all data will be moved to target when running data flows in parallel

My package has the following two tasks which run in parallel. DF - Move source data to target loads a copy of all source data to target via OLEDB Dest - Target and also loads a copy of source PK ...
2
votes
1answer
78 views

Running scheduled SSIS pkg with SQL Server 2012 Web Edition

I have a client with SQL Server 2012 Web Edition. I believe it has SQL Server Agent, but I see on the following Microsoft page that many integration services are not supported: ...
1
vote
0answers
20 views

Logging within a procedure's transaction

I have an ETL process running inside a PostgreSQL 9.3 procedure on AWS RDS. The procedure normally takes about ten minutes to run, but I'm having a rare issue where it takes several hours and never ...
0
votes
0answers
17 views

SSIS - different versions of the project

I am designing the ETL process. I created a package that extracts the data from the source using stored procedures. The SQL query looks like this: EXEC BI.ProcedureName The procedure returns data ...
0
votes
0answers
43 views

Batch Processing in Talend open studio for data integration

I am using TOS for data integration to transfer a data between mysql and postgres. Its transferring data between small tables successfully. But a table has 3GB data. while trying to transfer data its ...
2
votes
3answers
24 views

Data Warehouse attribute that describes its timeframe?

I am writing a document about a Data Warehouse system, and would like to know if is there a single word (noun) that describes the DW attribute of how many time it contains? I mean, if a DW contains ...
4
votes
3answers
64 views

is automated data movement via Excel considered ETL?

I am wondering if informal methods of automatically extracting, transforming, and loaded data constitute "ETL." For example, I code Excel VBA to extract from databases onto a spreadsheet, transform ...
1
vote
1answer
66 views

When to run ETL for dimensions

We have a suite of restaurant web applications that stores data in SQL Server 2012 on database server. We have now started to move data for reporting onto a separate data warehouse server. The data ...
2
votes
0answers
45 views

Hash values vs. column values in an incremental load [closed]

I have a situation where I need to do an incremental load on multiple files every month. The files have thousands-to-millions of rows, and between 5 and 50 columns. I've read articles about hashing ...
1
vote
0answers
88 views

Using SSIS vs writing your own ETL [closed]

I have two data sources. Data from Questionaires laying in a datatable in the data hosters database (queriable via REST in JSON) Data about clients from salesforce (also queriable via REST in JSON) ...
0
votes
0answers
9 views

Pentaho getfilename error

I am try to send 20 mail with attachments using pentahi DI, Jobs are like below Get File Names--->Add constants--->Mail But some mails are not send with below error, Thay say files are not ...
0
votes
0answers
16 views

Update source table from SSIS after Inserting in Destination [duplicate]

I am using SSIS 2012 from visual studio to extract all email information from source tables as follows: Data flow to take data from Notifications table in App1 Store that in SQL Destination. ...
0
votes
0answers
52 views

Tools to insert data faster in MySql database

I am trying to insert data from a flat file which is around 9 GB with about 15 million rows, I used SQl-server 2012 for that. I noticed data insert is much faster with sql server and it took around 30 ...
1
vote
1answer
50 views

What's the term that describes our data processes [closed]

We provide an online subscription product that allows our users to research and analyze financial data of their interested industries. As such, we have various tools for loading data from various ...
0
votes
1answer
275 views

Synchronise Oracle Tables to SQL Server

Our ERP is using an Oracle database while our BI system is based on SQL Server. We need to get the changed data in Oracle and copy it in our staging area. At the moment we get the changed rows by ...
2
votes
1answer
58 views

Introduce or not introduce SSIS to an ETL project [closed]

Given is an ETL process that works fine using traditional SQL. The process is well optimized making use of ALTER TABLE permissions and performing partition switching, index creation, partition ...
3
votes
1answer
97 views

Stored procedure deployment - push or pull into database? [closed]

Using stored procedures as part of an ETL process to populate tables in another database, which method would be considered a best practice (if any) in terms of deployment, maintenance, visibility and ...
1
vote
1answer
77 views

SSIS Transaction Scope

My question is in regards to the scope of a transaction when using SSIS transactions. If I have a sequence container and it is marked as "transaction required" and I have tasks in that sequence ...
1
vote
1answer
26 views

Setting the column width of a column in a table created by logparser

I'm trying to use LogParser for ETL. I am trying to import this file with the following command: "c:\Program Files (x86)\Log Parser 2.2\LogParser.exe" "SELECT Field1 AS CountryCode, Field2 AS Zip ...
1
vote
0answers
71 views

Connection refused when inserting

Description: Machine 1: Linux, PostgreSQL 9.3, on local, 10.12.10.123 Machine 2: Linux, PostgreSQL 9.3, on cloud, 104.10.124.123 Configuration (on both): + postgresql.conf : listen_addresses ="*" + ...
1
vote
1answer
37 views

Are there best practices or standardized syntax for writing transformation logic?

This may not be the appropriate forum, but its the closest I could find. I write a lot of source to target mappings for work. These documents just describe the operations/logic/transformations/etc. ...
0
votes
0answers
123 views

SQL Server 2012/SSIS - How to Import Partitioned Table Data to Reporting DB?

What's a recommended SSIS process flow for importing a daily partitioned table to a reporting DB table (SQL Server 2012)? Will be switching partitions on main DB to a staging table each day and ...
1
vote
0answers
41 views

Best way to perform real time analytics on a growing collection of json file ( very large ) [closed]

I have a very large collection of JSON file (tens of millions) on S3 and I need to have some kind of aggregation (perform continuously as the collection grows). The aggregation is mainly counter ...
0
votes
0answers
19 views

Does an Object literal always mean the Predicate and Object form a key-value property on the Subject?

A triple is Subject-Predicate-Object, e.g. Post Office - Not Open - Sunday. Consider another example, Post Office - Size in Square Feet - "4000" For this example, I have two observations: the ...
0
votes
3answers
37 views

Can data be replicated from a slave MySQL back to a master without the slave containing all the master data?

We have a PHP/Symfony 2 application using MySql as the database. The database contains a large amount of data. The application is deployed to a separate server(s) and an empty/clean version of the ...
2
votes
2answers
585 views

How to pass parameter in select query(In with clause) and not in where condition in SQl Server SSIS Package?

I want to pass a parameter (@date1) in my SSIS OLE DB Source so I created a variable and I tried to pass the parameter using '?' but it's showing 'Syntax error, permission violation or other ...
1
vote
0answers
70 views

ETL flow in Fact tables

I've got a simple theoretical question... If we discuss the principles, elements, design and implementation of a typical ETL flow for a dimension table then, in my understanding, (in short) I'd say ...
1
vote
1answer
700 views

Dimensional Modeling and ETL in Redshift

I have been researching Amazon's Redshift database as a possible future replacement for our data warehouse. My experience has always been in using dimensional modeling and Ralph Kimball's methods, so ...
0
votes
1answer
126 views

String to timestamp in datastage 9.1.2

Apologies for asking a question that I suppose is very basic. I'm just starting out with Streams. I'm trying to read a sequential files containing a timestamp field and a value field, like this: ...
1
vote
1answer
84 views

How to load .dat files to oracle database

I have .DAT files extracted from telecom switch. I want to upload this file to an Oracle server using IBM data-stage ETL tool. Unfortunately these files are not readable format. This is a picture ...
0
votes
0answers
312 views

Best way to run an ETL on a production MongoDB database

We have the following environment: 3 servers + 3 replica set servers Each server has 3 shards Our main collection has around 40,000,000 documents that average at around 6kb Our shard key is ...
0
votes
0answers
81 views

Tracking row level status of SSIS ETL process

I have an ETL Process in SSIS in which I am loading around 150 tables in each run - Truncate and Insert. I have four packages each from different sources. Each package loads different tables. ...
1
vote
1answer
78 views

Insert/ Update Scalability (more records, continually longer lookups — solution?)

posted this before but never received a clear answer. I'm working on an ETL project and am basically feeding JSON records of "tickets" into a MySQL database. (actually come to think of it, may be ...
0
votes
1answer
76 views

How to keep the data warehouse synchronised with the source systems? [closed]

I'm a DBA who is responsible for exporting data from an ERP system for Business Intelligence purposes. The data from ERP system is being sent to the data warehouse and they are used in the ETL ...
1
vote
1answer
174 views

SSIS Transaction Problem

I am trying to use SQL transaction with my SSIS package so if any error comes changes are rollback. I am using an "Sequence Container" in Control Flow for this and setting its "Transaction Option" to ...
0
votes
2answers
717 views

SSIS import data or insert data if no match

I have a situation in which I need to import data from one table to another. The challenge here is in tables I have some columns which are references to other table values (kind of foreign key). I ...
0
votes
2answers
149 views

Process staging table by getting unique rows only (incremential)

I had simple problem with etl on staging table that was answered here: Process staging table by getting unique rows only Now after apllying solution I spotted that will be unusable in production ...
-2
votes
1answer
69 views

Process staging table by getting unique rows only [closed]

I have several staging tables where I bulk import data with duplicates and sometimes I upload historical data mixed with real-time data, just importing everything without caring if it is already in ...
3
votes
1answer
213 views

Metadata Headache

A process that I don't have control over is dropping and re-creating tables in a MySQL database every night. This wouldn't be a problem (I think), if it re-created the tables identically every time. ...
4
votes
1answer
408 views

SSIS Multicast performance question

I'm wondering how multicast actually works. So here is my setup. I have one source flat file(csv) of about 100k rows, maybe about 250-300 columns, and I'm multicasting that file into 7 one-to-one ...