Questions tagged [azure-data-factory]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
0 votes
0 answers
19 views

Local SQL to Managed Cloud SQL via ADF. Copy Data throughput fell off a cliff

I have an on-site mini PC acting as a data extraction and replication SQL server that copies data nightly from the ERP production database that's running some legacy linux architecture. The copying is ...
funkyman50's user avatar
0 votes
0 answers
11 views

How do I exclude files in ADF when they don't have a file extension?

Sorry guys if this is the wrong place I couldn't find a separate data wranging stack. eed to SFTP files generated every 15 minutes up to an external SAAS product. Problem is the app generating these ...
Transitioned's user avatar
0 votes
1 answer
93 views

How to archive a on-premise sql server database to azure

We have an on-premise database of about eight terabytes that we are trying to archive to Azure. How can we go about it? This archive needs to be accessible because we might need to pull data from it ...
user avatar
0 votes
1 answer
197 views

is it a good idea to replace azure sql MI replication with CDC and ADF or Azure Function?

The company I work for put everything on azure and we use sql server replication to move data from one big collection db server (a azure sql MI) to our other database servers (say 20 in total). Every ...
jerry xu's user avatar
2 votes
1 answer
411 views

What permissions are needed for a user running Ola Hallengren's IndexOptimize stored procedure?

I'm planning to orchestrate my index maintenance with other jobs in my Azure SQL Database (serverless) using Azure Data Factory. The job will be run by the Managed Identity of my ADF service, and the ...
Widforss's user avatar
0 votes
1 answer
572 views

Azure Data Factory - copy data activity into a Postgresql database (on an IaaS Windows Azure vm, not Azure Database for Postgresql (PaaS))

I have a Postgresql database on a Windows vm (not a PaaS Azure Database), and I need to import csv files into it daily. ADF seems like a perfect fit, but I see that a standard Postgresql db is not a ...
Bobogator's user avatar
0 votes
1 answer
295 views

ADF: TempDb to user table very slow

I have a Azure Data Factory Data Flow that writes ~250 Mio. rows (~100 GB) to a MS Sql Server 2019. Currently, I have set the 'Use TempDB' option to true: The first part of the job, i.e. writing all ...
guid's user avatar
  • 1
0 votes
1 answer
874 views

In ADF, are copy data activities wrapped in transactions?

I have a copy data activity that is moving data from a managed instance to a sql database. The flow of the process is: truncate a staging table on the sql database as a distinct activity call a ...
Schmocken's user avatar
  • 101
0 votes
0 answers
307 views

Replicating on-prem MS SQL databases with Azure DataFactory

Is it possible to replicate a database from an on-prem SQL server to a SQL Server running on an Azure VM (SQL on IaaS, not a SQL Managed Instance or an AzureSQL database) using Azure DataFactory? Most ...
Marcel's user avatar
  • 111
1 vote
1 answer
2k views

How to call python file in repo in databricks from data factory outside DBFS?

In Azure Databricks I have I have a repo cloned which contains python files, not notebooks. In Azure Data Factory I want to configure a step to run a Databricks Python file. However when I enter the /...
Brendan Hill's user avatar
-1 votes
2 answers
529 views

Accessing SQL server on-Prem database views from remote server through Azure Data Factory

We need to copy the data from client's remote SQL server (on-prem) database to our Azure SQL server through Azure data factory. So we can automate the data pull on regular basis. Client tried to offer ...
user237624's user avatar
2 votes
0 answers
226 views

Exactly-once FIFO queue in Synapse

Creating a queue table in SQL Server is a much-studied problem. However, I would like to implement one in Azure Synapse where many of the building blocks do not exists. Specifically no table hints (...
Michael Green's user avatar
3 votes
3 answers
993 views

ADF patterns to minimise batch run elapsed time

We use Azure Data Factory (ADF) to pull a number of source tables from an on-prem SQL Server DB into Azure Data Lake (DL). We've made this data-driven using the Lookup-ForEach pattern. There is one ...
Michael Green's user avatar
2 votes
3 answers
2k views

How to run legacy SSIS packages with Azure Synapse

I would like to know how to run an SSIS package from Azure Synapse Studio or whether it's at all possible. Apparently there is no support for SSIS runtime in Synapse but I suppose there should be a ...
Jayvee's user avatar
  • 121
2 votes
1 answer
432 views

How are tasks assigned to ForEach iterations

The Lookup-ForEach pattern is common in Azure Data Factory (ADF). How are items produced by the Lookup allocated to the ForEach's workers, the number of which is controlled by Batch Count?
Michael Green's user avatar
-1 votes
1 answer
569 views

Data Factory keeps getting "untrusted connection" error when logging into on prem SQL Server via IR on VM

We have: A local SQL Server database A VM with the Integration Runtime installed on it A Data Factory sharing this IR, working (according to the ADF interface) Logins stored in Key Vault for SQL ...
blobbles's user avatar
  • 1,621
0 votes
1 answer
193 views

SQL Server 2016 compress function details - external decompression

According to SQL Server 2016 docs COMPRESS and DECOMPRESS methods are just a blackbox - you put the data in and after some magic it gets compressed or decompressed. The problem is that I need to find ...
Paweł Sopel's user avatar
0 votes
1 answer
211 views

Alternative methods for client data integration (Azure SQL Database)

I have an application that ingests data from clients on a daily/weekly basis (two different data sets, one daily and the other weekly) into a SQL Azure Database. The clients' data source depends on ...
brad's user avatar
  • 11
1 vote
1 answer
496 views

Migrating Azure SQL database to Azure sql Managed instance

What are different options to move database from "Azure sql server" to "Azure sql managed instance", Looks like below options are not possible 1) Azure migration service - Not supporting azure sql ...
Dyaneshwaran S's user avatar
2 votes
1 answer
37 views

Azure SSIS Deploying in SQL Azure

Does deploying SSIS really required an azure data lake house when deploying on Azure database?. I am currently working on an SSIS project that its database is currently deployed on Azure and I am ...
Nathan's user avatar
  • 29
1 vote
1 answer
73 views

Stored procedure to set value to a column must accept returned values from functions

I created a very simple table in Data Mart to store ETL diagnostic information: CREATE TABLE [internal].[etl_status] ( [property_name] VARCHAR(100) NOT NULL PRIMARY KEY, [property_value] ...
pmdci's user avatar
  • 681
1 vote
0 answers
73 views

How to automatically append updated data

I am currently working with Azure Data Factory to create pipelines that feed from a dataset into tables located in Microsoft SQL Server Management Studio. My issue is that whenever new data is fed ...
Mitchell's user avatar
2 votes
1 answer
131 views

How to implement updates from mySQL operational DB to Azure SQL DB for reporting

We have an operational mySQL DB running on AWS for a transactional system and an Azure SQL DB for reporting with PowerBI. Now I'd like to regularly (e.g. every night) do an update of certain tables ...
Jörg Brenninkmeyer's user avatar
0 votes
1 answer
392 views

What type of data ingestion does Azure Data Factory use?

This article data-ingestion says that there are two types of data ingestion Data can be streamed in real time or ingested in batches. What type of data ingestion does Azure Data Factory's ...
George Cernat's user avatar
1 vote
1 answer
4k views

Parameter value in Data Factory Triggered pipeline

I have a pipeline configured in azure data factory which basically does create a backup file (JSON) from a cosmosDB dataset and it's saved in a blob storage, my problem comes when I want to schedule ...
user avatar
2 votes
1 answer
3k views

Azure Data Factory 'Failed Validation' with Folder does not exist

I uploaded whole set of .txt files in my blob storage with folder structure Year/Month/day. I circled the folder structure. Then in the pipeline I have following dataset created. "type": "...
SqlWorldWide's user avatar
  • 13.1k