Questions tagged [etl]
'Extract, Transform and Load': Data load processing, normally in the context of data warehouse systems.
216
questions
0
votes
0answers
14 views
How to establish ETL connection between Oracle 9 and Google Cloud (BigQuery)? [closed]
I just acquired Google Cloud, and would like to create a data warehouse. My first task will be to import (automatically, preferably on a daily basis) data from an old Oracle 9 database, to a GCP, ...
1
vote
0answers
34 views
Tool to help breaking multitenant into a single tenant [closed]
Oracle 11.2.0.4
I've been tasked with breaking out a single customer from our multitenant system into a separate system. And the data will have to be continuously updated. This is a large system ...
0
votes
2answers
42 views
Strategies/Resources for implementing data lineage in an ETL process?
I'm struggling to articulate my need here.
I've been tasked with developing a significant ETL process dealing with warehouse item costs in a major grocery chain using the full Microsoft Stack.
I'd ...
1
vote
0answers
69 views
Database - limit of reading a table with a lot of concurrent batch read and writes
We have this table. It contains data which will be picked up by an hourly batch job for processing. Five times a day, there are also multiple batch ETL jobs which get data from other source table and ...
0
votes
0answers
19 views
How can I run an ETL job only on rows added since the last run?
I currently have a flow in Google Dataprep (Trifacta) that operates on my datasource and stores the output in BigQuery. Running the job on the entire data is slow, expensive and unnecessary; I'd like ...
1
vote
1answer
13 views
Data Pipeline load bulk tables in one time
I am using AWS Data Pipeline for copying my RDS MySQL Database to Redshift. I need to create separate pipeline for each table and each pipeline create new EC2 instance. (Process takes time).
Problem
...
1
vote
0answers
7 views
Date getting truncated on PDI 7.1 for salesforce DB - Need workaround
We are working on transformation which inserts date into a salesforce DB and are seeing issues.
Below are the issues: A transformation working in PDI CE 6.1 do not work in CE 7.1 or CE 8.3.
Error :
...
0
votes
1answer
55 views
TIMESTAMP OR DateUpdate
I want to implement incremental ETL on some of my sql server tables, so I need a new column like UpdatedDate to track change dates of my records. I want to transfer only updated and inserted records, ...
1
vote
1answer
64 views
Does loading data to temp table improve performance?
When I write stored procedure I use views that have calculation inside them and so if I have to reuse the view multiple times in my procedure, I load it to a temp table and then use the temp table ...
1
vote
0answers
25 views
SSRS managed shared subscriptions data from from database
While running an ETL process we need to pause the shared managed SSRS subscriptions and then enable them and run them accordingly when our ETL has progressed. In order to do that we take a backup of ...
0
votes
2answers
461 views
SSIS Slow Data Transfer between Oracle and SQL Server 2016
I am trying to design a process that transfers data from Oracle to SQL Server 2016 by using SSIS.
It is done with the aim of removing productive processes from an SQL Server 2000 instance, where the ...
0
votes
2answers
344 views
Specifying Order of Executing Stored Procedures
Issue:
The order in which my stored procedures are called produces no errors if run independently, but when run as once, yields the following error:
Msg 2627, Level 14, State 1, Procedure ...
1
vote
0answers
91 views
Parallel insert and delete into staging tables
I work on a small system that receives quite a lot of data from multiple sources (hundreds of them). Depending on few factors I can get anywhere from few rows to a couple of thousands (in fact ...
0
votes
0answers
31 views
Are temporary columns a good solution for loading data for a small scale data migration?
Our team is facing the following scenario. We're trying to import data from a denormalized data set in MySQL into a normalized MySQL database and are unsure of what a good solution would be for ...
0
votes
1answer
43 views
Transactional Replication data during rebuild
To offload reporting I have transactional replication running between my prod box and my reporting server (top image #1). This works great until the snapshot has to be rebuilt. During this time users/...
0
votes
1answer
319 views
SSIS data flow gets stuck reading table with no errors
When I'm doing ETL in SSIS, the data flow task "pauses" reading from the source transactional Live IO DB (the one on the left). It stops reading the data and "pauses" on random rows and then does ...
0
votes
3answers
80 views
Window Function to get pairs of records and compute the time difference
I'm working with an event dataset as the one in the example below:
Event_Type Event_Timestamp Is_Active
A 2010-10-01 00:00:00 1
B 2010-10-01 00:00:01 1
A ...
1
vote
1answer
28 views
Is it faster to truncate destination and copy all source data or lookup and keep only non-matching data with SSIS transfer packages?
I am building an Azure SQL DB database and to do this I will have to regularly (weekly-monthly) pull data from various other servers. I am designing SSIS packages to do this and whilst most of the ...
2
votes
2answers
985 views
Linked server returned message “Query timeout expired.”
We have two separate SQL Servers. On one server we have a data warehouse (DWH), on the other we have sales information database.
Now on the DWH server there is an ETL job that collects the ...
1
vote
1answer
84 views
SSIS comparison load between new, updated and deleted records and populate a staging database
So far,
My control flow looks like:
Execute SQL Task
- This executes a statement that creates a staging table. If one exists, it is dropped so everytime the package is run, the staging table will be ...
0
votes
1answer
89 views
What do I need to develop an integration service to extract data from an Oracle DB to a SQL Server DB using Visual Studio 2015? [closed]
I have to develop an ETL to extract some data from an Oracle DB 11g to a SQL Server 2017 DB. I am pretty new at all this technology so I would like some advices. The strongest problem so far is to ...
3
votes
3answers
569 views
How to handle schema changes in source systems when running an ETL?
How do people typically handle schema changes in source systems when running an ETL as part of an EDW? For example when a column you were operating on disappears and your ETL breaks.
3
votes
0answers
34 views
Migrating Access Saved Import Specifications to SSIS Package
We currently receive a multi-hundred column flat file from a vendor, and the tech prior to me had created an import specification in Microsoft Access to automatically split the file into the right ...
1
vote
2answers
316 views
Can Transactional Replication and triggers be used for near-realtime ETL [closed]
We want to have a data warehouse synced with our production OLTP database in near realtime. Is it possible to do this with Transactional Replication and triggers?
How we currently ETL
We use sql ...
2
votes
1answer
246 views
ETL, Production, Development — Should these be different servers? Instances?
You can say I've become an "accidental DBA" as people have left this company and I'm managing many aspects of the BI system now suddenly.
My predecessor left quite the mess regardless but I'm trying ...
0
votes
0answers
48 views
How to Represent a VARCHAR Column with XML Data in a Fact Table?
I have SQL Server 2016 environment and I have decided to design a source table with holds some sort of event data as a transactional fact table. The table has about a million rows, and generates about ...
30
votes
4answers
1k views
What is a scalable way to simulate HASHBYTES using a SQL CLR scalar function?
As part of our ETL process, we compare rows from staging against the reporting database to figure out if any of the columns have actually changed since the data was last loaded.
The comparison is ...
1
vote
1answer
168 views
When to Disable and Recreate Indexes in Data Warehouse ETL?
We are moving data from Staging Database into Kimball Data Warehouse DB.
I am hearing different ideas regarding when to disable and rebuild indexes.
Stage tables are truncated every day for each new ...
-2
votes
1answer
208 views
Datawarehouse Staging Tables: Null or Not Null
We are loading data from OLTP into Staging Tables with SSIS, then into Kimball data warehouse with ETL process.
We are inquiring about staging tables, if they should have null/not null constraints. ...
3
votes
2answers
202 views
Structured .TXT Data To A Proper DBMS [closed]
I'm presented with multiple .txt documents (each with its own set of columns) containing rows separated by some type of delimiter (which is different across the files). Each .txt file has different ...
0
votes
1answer
899 views
Query or ETL process to insert data into fact tables in SSMS
I'm working on creating a star schema for my project. I want to use it for reporting. I have used Google Play Store dataset from Kaggle.
The thing that I'm having trouble loading data into my fact ...
1
vote
0answers
110 views
Increase frequency of updates (ETL process) to Data Warehouse
Our data warehouse is updated only once every 24 hours (at night). The ETL process takes a little over an hour to complete. I am wondering how to increase the frequency of the ETL process - so running ...
6
votes
2answers
922 views
SQL Staging Tables: Primary Key Clustered or Heap
We are taking legacy flat txt files and inserting them into stage tables with SSIS.The question arose whether table should have primary clustered key index or not. This is direct flat file import with ...
0
votes
1answer
159 views
SSIS Installation on Dedicated Server
We are planning on installing SSIS 2016 on a new dedicated ETL server.
The packages will be called from SQL Server Agent (on a separate server), or from an application (again on a separate server).
...
0
votes
1answer
65 views
What's your data ETL methodology? [closed]
I think this is more like an open question: How do you do the ETL for your company?
We are using MySQL as our database management system and we have a production schema. Every month, when we received ...
2
votes
1answer
1k views
Efficiently bulk upsert unrelated rows
As part of a manual replication process between databases with different but related schemas, for each table, we identify new and modified records in the source database and feed them via a table ...
1
vote
1answer
290 views
How to copy selected table data from sql server to mysql on a daily basis? [closed]
I am having a retail pos application with MSSQL database. Now I need to transfer selected table data from SQL server to MYSQL database on a daily basis like job.
How to achive this?
1
vote
2answers
643 views
SSIS ETL or CDC - real time DWH reporting
I am currently working on SSIS package for continuous incremental data loads of the tables (40 tables now and expected to increase in near future) from OLTP system to a Reporting database residing on ...
1
vote
1answer
3k views
SQL Fastest Way to find Data Difference Between Two Tables
I am trying to locate fastest way to find difference between two tables. The first table is populated from text files daily, the second table is maintained in our database. We need to add data that ...
1
vote
1answer
306 views
reseed auto_increment to specific number?
Is it possible to set the identity column of a table to start from a specific value?
I'm converting an ETL stage from mysql to a sql server database where a staging database loads records with a ...
-2
votes
1answer
85 views
Can ETL access Data while Rebuild Index Operation is running
Can I get some help to understand below questions:
We have a ETL job running (Datawarehouse Server) and weekly index operations (Production Server)(Ola Hallengren script) , below questions are ...
1
vote
2answers
62 views
How do I run a stored procedure only on newest data loaded?
My database has a table where the data is loaded from an Excel upload. Then, a stored procedure is run on the data to validate it and insert it to another table.
Currently, my stored procedure will ...
2
votes
1answer
148 views
Handling variable fields in a CSV file with JSON?
Im trying to figure out the engineering to get a CSV file into PostgreSQL.
Requirements:
The CSV has X number of predetermined required fields
The CSV file has Y number of dynamic fields
Example:
...
2
votes
1answer
68 views
Is it bad practice to use a data-warehouse for providing reorganized non-BI data?
The Primary goal is the case-optimized data representation. The idea is to gather data from several databases, save the data in a data-warehouse and restructure/connect the data in data marts(similar ...
0
votes
1answer
517 views
SQL Bulk import vs SSIS in Simplicity
We are deciding between bulk insert or SSIS. We have a simple spreadsheet with 5 columns, customerid, cost, date, product name, etc. The spreadsheet is generally only around 200 rows every week.
All ...
2
votes
1answer
3k views
Is it impossible to force MariaDB users to SSL?
First of all, my english is really bad, so sorry for my english mistakes.
I have a MariaDB 10.2.15. I installed SSL, and its working fine, but I cant force use SSL to users.
my.cnf:
[mysqld]
ssl
ssl-...
2
votes
0answers
46 views
SSIS - dynamically merging a group of files
I'm trying to create an import package for our new mssql database and I'm having to merge a group of 20 files together to be fed into a table on the database. However not all of those 20 files will be ...
0
votes
1answer
290 views
Multiple Agent Jobs Running same SSIS Package
I have the following doubt, is there a possibility to have faild a Job when are running simultaneously as the following example:
4 SQL Agent Jobs running the same SSIS in different schedules, I've ...
3
votes
2answers
4k views
Character Conversion Issue Importing From CSV File
After doing a load of CSV files, there are various words that are "written" to the database incorrectly.
Some examples:
Diã¡ria should be Diária
Crã©dito should be Crédito
Ligaã§ãµes should be ...
4
votes
1answer
9k views
Login failed for user 'NT AUTHORITY\ANONYMOUS LOGON' for SSIS
I have a SSIS package which is perform data migration between MySQL database and SQL server database. When I manually run it from visual studio or from SQL Server Management Studio, it ran ...
