Questions tagged [azure-data-factory]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
0 votes
1 answer
116 views

In ADF, are copy data activities wrapped in transactions?

I have a copy data activity that is moving data from a managed instance to a sql database. The flow of the process is: truncate a staging table on the sql database as a distinct activity call a ...
user avatar
  • 101
0 votes
0 answers
52 views

Replicating on-prem MS SQL databases with Azure DataFactory

Is it possible to replicate a database from an on-prem SQL server to a SQL Server running on an Azure VM (SQL on IaaS, not a SQL Managed Instance or an AzureSQL database) using Azure DataFactory? Most ...
user avatar
  • 101
1 vote
1 answer
888 views

How to call python file in repo in databricks from data factory outside DBFS?

In Azure Databricks I have I have a repo cloned which contains python files, not notebooks. In Azure Data Factory I want to configure a step to run a Databricks Python file. However when I enter the /...
user avatar
0 votes
0 answers
40 views

Azure data factory - ON CONFLICT DO NOTHING

I am building a data pipeline with Azure data factory with source from Azure storage account where the data is in json files and sink as a postgresql database. My json files will contain data that ...
user avatar
-1 votes
2 answers
130 views

Accessing SQL server on-Prem database views from remote server through Azure Data Factory

We need to copy the data from client's remote SQL server (on-prem) database to our Azure SQL server through Azure data factory. So we can automate the data pull on regular basis. Client tried to offer ...
user avatar
2 votes
0 answers
84 views

Exactly-once FIFO queue in Synapse

Creating a queue table in SQL Server is a much-studied problem. However, I would like to implement one in Azure Synapse where many of the building blocks do not exists. Specifically no table hints (...
user avatar
3 votes
3 answers
204 views

ADF patterns to minimise batch run elapsed time

We use Azure Data Factory (ADF) to pull a number of source tables from an on-prem SQL Server DB into Azure Data Lake (DL). We've made this data-driven using the Lookup-ForEach pattern. There is one ...
user avatar
2 votes
3 answers
799 views

How to run legacy SSIS packages with Azure Synapse

I would like to know how to run an SSIS package from Azure Synapse Studio or whether it's at all possible. Apparently there is no support for SSIS runtime in Synapse but I suppose there should be a ...
user avatar
  • 121
2 votes
1 answer
122 views

How are tasks assigned to ForEach iterations

The Lookup-ForEach pattern is common in Azure Data Factory (ADF). How are items produced by the Lookup allocated to the ForEach's workers, the number of which is controlled by Batch Count?
user avatar
0 votes
1 answer
172 views

Data Factory keeps getting "untrusted connection" error when logging into on prem SQL Server via IR on VM

We have: A local SQL Server database A VM with the Integration Runtime installed on it A Data Factory sharing this IR, working (according to the ADF interface) Logins stored in Key Vault for SQL ...
user avatar
  • 1,606
0 votes
1 answer
92 views

SQL Server 2016 compress function details - external decompression

According to SQL Server 2016 docs COMPRESS and DECOMPRESS methods are just a blackbox - you put the data in and after some magic it gets compressed or decompressed. The problem is that I need to find ...
user avatar
0 votes
1 answer
93 views

Alternative methods for client data integration (Azure SQL Database)

I have an application that ingests data from clients on a daily/weekly basis (two different data sets, one daily and the other weekly) into a SQL Azure Database. The clients' data source depends on ...
user avatar
  • 11
0 votes
1 answer
352 views

Migrating Azure SQL database to Azure sql Managed instance

What are different options to move database from "Azure sql server" to "Azure sql managed instance", Looks like below options are not possible 1) Azure migration service - Not supporting azure sql ...
user avatar
2 votes
1 answer
32 views

Azure SSIS Deploying in SQL Azure

Does deploying SSIS really required an azure data lake house when deploying on Azure database?. I am currently working on an SSIS project that its database is currently deployed on Azure and I am ...
user avatar
  • 29
1 vote
1 answer
43 views

Stored procedure to set value to a column must accept returned values from functions

I created a very simple table in Data Mart to store ETL diagnostic information: CREATE TABLE [internal].[etl_status] ( [property_name] VARCHAR(100) NOT NULL PRIMARY KEY, [property_value] ...
user avatar
  • 681
1 vote
0 answers
55 views

How to automatically append updated data

I am currently working with Azure Data Factory to create pipelines that feed from a dataset into tables located in Microsoft SQL Server Management Studio. My issue is that whenever new data is fed ...
user avatar
2 votes
1 answer
67 views

How to implement updates from mySQL operational DB to Azure SQL DB for reporting

We have an operational mySQL DB running on AWS for a transactional system and an Azure SQL DB for reporting with PowerBI. Now I'd like to regularly (e.g. every night) do an update of certain tables ...
user avatar
0 votes
1 answer
294 views

What type of data ingestion does Azure Data Factory use?

This article data-ingestion says that there are two types of data ingestion Data can be streamed in real time or ingested in batches. What type of data ingestion does Azure Data Factory's ...
user avatar
1 vote
1 answer
3k views

Parameter value in Data Factory Triggered pipeline

I have a pipeline configured in azure data factory which basically does create a backup file (JSON) from a cosmosDB dataset and it's saved in a blob storage, my problem comes when I want to schedule ...
user avatar
  • 189
2 votes
1 answer
2k views

Azure Data Factory 'Failed Validation' with Folder does not exist

I uploaded whole set of .txt files in my blob storage with folder structure Year/Month/day. I circled the folder structure. Then in the pipeline I have following dataset created. "type": "...
user avatar
  • 12.3k