Questions tagged [azure-data-factory]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
0
votes
0answers
12 views

Azure data factory - ON CONFLICT DO NOTHING

I am building a data pipeline with Azure data factory with source from Azure storage account where the data is in json files and sink as a postgresql database. My json files will contain data that ...
-1
votes
2answers
30 views

Accessing SQL server on-Prem database views from remote server through Azure Data Factory

We need to copy the data from client's remote SQL server (on-prem) database to our Azure SQL server through Azure data factory. So we can automate the data pull on regular basis. Client tried to offer ...
2
votes
0answers
43 views

Exactly-once FIFO queue in Synapse

Creating a queue table in SQL Server is a much-studied problem. However, I would like to implement one in Azure Synapse where many of the building blocks do not exists. Specifically no table hints (...
3
votes
2answers
85 views

ADF patterns to minimise batch run elapsed time

We use Azure Data Factory (ADF) to pull a number of source tables from an on-prem SQL Server DB into Azure Data Lake (DL). We've made this data-driven using the Lookup-ForEach pattern. There is one ...
2
votes
2answers
149 views

How to run legacy SSIS packages with Azure Synapse

I would like to know how to run an SSIS package from Azure Synapse Studio or whether it's at all possible. Apparently there is no support for SSIS runtime in Synapse but I suppose there should be a ...
2
votes
1answer
59 views

How are tasks assigned to ForEach iterations

The Lookup-ForEach pattern is common in Azure Data Factory (ADF). How are items produced by the Lookup allocated to the ForEach's workers, the number of which is controlled by Batch Count?
0
votes
0answers
45 views

Azure Data Factory pipeline accessing MariaDB

I have MariaDB running in a virtual machine in Azure. I need to move data from a file (in Azure storage) to a table in MariaDB using Azure Data Factory (ADF). There's no connector in ADF to write to ...
0
votes
1answer
42 views

Data Factory keeps getting "untrusted connection" error when logging into on prem SQL Server via IR on VM

We have: A local SQL Server database A VM with the Integration Runtime installed on it A Data Factory sharing this IR, working (according to the ADF interface) Logins stored in Key Vault for SQL ...
0
votes
0answers
68 views

Azure SQL Db to DB2 Sync

I need to automatically synchronize changes made in an Azure SQL database to our DB2 system. Whenever a change is made in Azure, DB2 should also be modified. Azure DB creation is not started and ...
0
votes
1answer
54 views

SQL Server 2016 compress function details - external decompression

According to SQL Server 2016 docs COMPRESS and DECOMPRESS methods are just a blackbox - you put the data in and after some magic it gets compressed or decompressed. The problem is that I need to find ...
0
votes
1answer
56 views

Alternative methods for client data integration (Azure SQL Database)

I have an application that ingests data from clients on a daily/weekly basis (two different data sets, one daily and the other weekly) into a SQL Azure Database. The clients' data source depends on ...
0
votes
1answer
301 views

Migrating Azure SQL database to Azure sql Managed instance

What are different options to move database from "Azure sql server" to "Azure sql managed instance", Looks like below options are not possible 1) Azure migration service - Not supporting azure sql ...
2
votes
1answer
25 views

Azure SSIS Deploying in SQL Azure

Does deploying SSIS really required an azure data lake house when deploying on Azure database?. I am currently working on an SSIS project that its database is currently deployed on Azure and I am ...
1
vote
1answer
38 views

Stored procedure to set value to a column must accept returned values from functions

I created a very simple table in Data Mart to store ETL diagnostic information: CREATE TABLE [internal].[etl_status] ( [property_name] VARCHAR(100) NOT NULL PRIMARY KEY, [property_value] ...
1
vote
0answers
50 views

How to automatically append updated data

I am currently working with Azure Data Factory to create pipelines that feed from a dataset into tables located in Microsoft SQL Server Management Studio. My issue is that whenever new data is fed ...
2
votes
1answer
50 views

How to implement updates from mySQL operational DB to Azure SQL DB for reporting

We have an operational mySQL DB running on AWS for a transactional system and an Azure SQL DB for reporting with PowerBI. Now I'd like to regularly (e.g. every night) do an update of certain tables ...
0
votes
1answer
264 views

What type of data ingestion does Azure Data Factory use?

This article data-ingestion says that there are two types of data ingestion Data can be streamed in real time or ingested in batches. What type of data ingestion does Azure Data Factory's ...
1
vote
1answer
3k views

Parameter value in Data Factory Triggered pipeline

I have a pipeline configured in azure data factory which basically does create a backup file (JSON) from a cosmosDB dataset and it's saved in a blob storage, my problem comes when I want to schedule ...
2
votes
1answer
2k views

Azure Data Factory 'Failed Validation' with Folder does not exist

I uploaded whole set of .txt files in my blob storage with folder structure Year/Month/day. I circled the folder structure. Then in the pipeline I have following dataset created. "type": "...