Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,584 questions
2 answers
Copy Activity SFTP sometimes copies files and sometimes does not
I run once a day a pipeline that has a 'Get Metadata' returning Child items and I filter the result to CSV type. The result is then past to a For Each which has a Copy data task and for each file @item().name it picks up the file from an SFTP and puts…
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(44.800000000000004, 31%, 14%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EJ%3C/text%3E%3C/svg%3E)
0 answers
ADF Copy Activity : Data load from ADLS (parquet files) to Azure Synapse Table Failing after running for 24 hours
ADF pipeline copy failed after running for 24 hours. Timeout is set to 7days. Copy Activity is trying to copy data from ADLS (Parquet files) to Azure Synapse Table on Production Environment . Could you please suggest what could be the possible issue for…
asked 2024-05-02T08:48:59.7533333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(313.6, 24%, 40%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EKS%3C/text%3E%3C/svg%3E)
Kapoor, Sipra (S)
5
Reputation points
asked 2024-05-02T08:48:59.7533333+00:00
Kapoor, Sipra (S)
5
Reputation points
3 answers
Mapping Data Flow - Null value issue in JSON which removes the complete column from json
Azure data factory dataflow not present null values in JSON format I am looking for how to present my null values with the Azure Data Factory Dataflow activity. Currently, I am using standard dataflow for modelling my data from Azure SQL database and…
asked 2021-07-09T14:36:18.787+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(16, 52%, 11%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EBS%3C/text%3E%3C/svg%3E)
Bhardwaj, Sorabh
1
Reputation point
answered 2024-05-02T08:41:10.57+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(246.4, 85%, 33%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EVT%3C/text%3E%3C/svg%3E)
VIJAY TALARI
0
Reputation points
1 answer
I am trying to read data from oracle table into azure blob using copy activity. It throws ORA-01426 numeric outflow error. Any suggestions on how to get it resolved?
I am trying to read data from oracle table into azure blob using copy activity. It throws ORA-01426 numeric outflow error. Any suggestions on how to get it resolved? Particulary there is one column causing this issue. its data type in oracle is…
asked 2024-05-01T15:39:05.63+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(96, 75%, 19%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESS%3C/text%3E%3C/svg%3E)
Satwik Srivastava
0
Reputation points
answered 2024-05-02T07:38:31.94+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(150.4, 64%, 24%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ENR%3C/text%3E%3C/svg%3E)
Nehruji R
1,976
Reputation points Microsoft Vendor
2 answers
CosmosDB Analytical store delete changes in sql sink
Hi I am trying to capture delete changes from Azure cosmosdb analytical store using change data capture in Azure datafactory, Source is Cosmosdb, sink is Azure sql database. In between I am flattening my file using dataflow. My insert, update is…
asked 2024-04-30T13:52:46.2133333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(112.00000000000001, 15%, 20%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EKB%3C/text%3E%3C/svg%3E)
Kuldeep Bhati
0
Reputation points
commented 2024-05-02T07:24:09.3266667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(12.8, 48%, 11%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EH%3C/text%3E%3C/svg%3E)
Harishga
3,430
Reputation points Microsoft Vendor
1 answer
One of the answers was accepted by the question author.
Retrieve video transcript from Azure Video Indexer using API
What API can I use to retrieve the transcript of a video from the Video Indexer portal? I'm looking for guidance on how to do this programmatically.
asked 2024-05-01T14:23:59.8933333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(92.8, 74%, 18%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EVA%3C/text%3E%3C/svg%3E)
Vishnoi, Astha
120
Reputation points
commented 2024-05-02T05:28:38.0533333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(153.6, 24%, 24%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ENM%3C/text%3E%3C/svg%3E)
navba-MSFT
17,125
Reputation points Microsoft Employee
0 answers
Remove BackSlash in result of Lookup Activity.
Hi All, I am having an order table, I have written a simple query to convert it to my required schema and it works perfect in my SSMS. Query: select(select JSON_OBJECT( 'OrderNumber': OrderNumber ) as 'values' from order for Json Path )as…
asked 2024-05-01T16:36:06.22+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(137.6, 79%, 23%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EMK%3C/text%3E%3C/svg%3E)
Muthu Kugan Muthusivakumar (INFOSYS LIMITED)
0
Reputation points Microsoft Vendor
edited the question 2024-05-02T05:13:54.7066667+00:00
Muthu Kugan Muthusivakumar (INFOSYS LIMITED)
0
Reputation points Microsoft Vendor
0 answers
When we deploy ADF,we are getting "TooManyFactoryUpdaterequests" in Create or Replace GlobalParameter through Azure Devops
When we deploy ADF, we are getting "TooManyFactoryUpdaterequests" in Create or Replace GlobalParameter using CI/CD Azure devops
asked 2024-05-01T16:52:46.6233333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(118.4, 57.99999999999999%, 21%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ERH%3C/text%3E%3C/svg%3E)
Ramesh Hariharan
0
Reputation points
edited a comment 2024-05-02T04:55:13.8+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(288, 61%, 37%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EP%3C/text%3E%3C/svg%3E)
phemanth
5,830
Reputation points Microsoft Vendor
0 answers
Data Factory to Salesforce using JWT
Hi All, We've been set with the requirement of authenticating with Salesforce via JWT. For the most part, this is set up and we can get an access token returned, however, how do we access Salesforce once we have the token? Does it have to be through a…
asked 2024-05-02T04:49:26.07+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(96, 30%, 19%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ERA%3C/text%3E%3C/svg%3E)
Ryan Abbey
0
Reputation points
asked 2024-05-02T04:49:26.07+00:00
Ryan Abbey
0
Reputation points
0 answers
How to read nested xlsx files in Azure Data Factory
How we can read these types of nested xlsx files in ADF
asked 2024-04-27T03:51:29.27+00:00
Abhishek Tripathi
0
Reputation points
commented 2024-05-02T04:25:46.98+00:00
Harishga
3,430
Reputation points Microsoft Vendor
0 answers
Copy Activity ADF pipeline data not showing up
Hi Team I created pipeline to copy activity from one table to other table in same log analytics workspace. I am using source Linked service Rest connector using managed Identity to get data from exceptions table then I am using Destination linked…
asked 2024-04-30T02:05:04.8933333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(144, 39%, 23%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ERS%3C/text%3E%3C/svg%3E)
Rishabh Srivastava
40
Reputation points Microsoft Employee
edited a comment 2024-05-02T04:15:33.91+00:00
Rishabh Srivastava
40
Reputation points Microsoft Employee
0 answers
Activity stuck in queue status
Starting from today morning, I am facing problem that any pipeline run, by trigger or manually or in debug mode, doesn't start. To be precise, pipeline starts ("In progress", but first activity stays in "Queue" status). Doesn't matter…
asked 2024-04-30T07:54:01.1866667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(195.2, 13%, 28%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EGE%3C/text%3E%3C/svg%3E)
Gużewski, Jacek
55
Reputation points
commented 2024-05-02T03:27:11.7566667+00:00
PRADEEPCHEEKATLA-MSFT
77,516
Reputation points Microsoft Employee
0 answers
Stored procedure transformation always on queued
Hello , I have 2 environment dev and test. There is a stored procedure which run successfully on dev bot queued on test for hours. What are the possible causes for that ? It runned successfully on dev in 1m on datafactory and 5 second on database.
asked 2024-04-30T09:25:25.9133333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(108.80000000000001, 38%, 20%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EMI%3C/text%3E%3C/svg%3E)
Mustafa Ispir
46
Reputation points
commented 2024-05-02T03:27:00.4133333+00:00
PRADEEPCHEEKATLA-MSFT
77,516
Reputation points Microsoft Employee
0 answers
Azure Data Factory pipelines stuck on first activity for 7+ hours
Hi, I just woke up to monitor all the runs, and suddenly saw that all of my runs were stuck on the first activity, or not even starting the first activity some for more than 7+ hours. This happened over-night as yesterday everything was fine and I didn't…
asked 2024-04-30T10:04:21.0433333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(185.6, 7.000000000000001%, 27%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EMG%3C/text%3E%3C/svg%3E)
Marcus Ghebrehiwet
0
Reputation points
commented 2024-05-02T03:26:25.8966667+00:00
PRADEEPCHEEKATLA-MSFT
77,516
Reputation points Microsoft Employee
0 answers
Custom libraries (wheel) for ADF Databricks Python activity run on serverless compute
I want to be able to execute Python scripts (via Databricks Python) from Azure Data Factory using serverless compute. Serverless compute does not support cluster level (compute scoped) libraries. In databricks workflows, it is being done as…
asked 2024-05-01T12:30:52.2366667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(268.8, 31%, 36%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EKP%3C/text%3E%3C/svg%3E)
Krzysztof Przysowa
0
Reputation points
commented 2024-05-02T02:34:43.2433333+00:00
PRADEEPCHEEKATLA-MSFT
77,516
Reputation points Microsoft Employee
2 answers
One of the answers was accepted by the question author.
Azure Data Factory copy activity stuck on queued status using integration run time self-hosted
I have been experiencing an issue, where the data factory copy activities are stuck in 'queued' status. When I try to use a database with the azure integrated run time, works normally. When I try to use one of the integration run time self-hosted, they…
asked 2024-05-01T14:06:20.4366667+00:00
Antônio Farias
0
Reputation points
accepted 2024-05-01T22:15:15.93+00:00
Antônio Farias
0
Reputation points
0 answers
How to copy the data from Amazon S3 to Azure SQL database incrementally.
Hello team, I wanted to wanted to copy data from Amazon S3(JSON format) to Azure SQL database. I wanted to learn the mechanism for incrementally updating the data from Amazon S3 to Azure SQL. Also, the environment has only SHIR deployed, so data flows…
asked 2024-05-01T20:06:01.4533333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(131.20000000000002, 0%, 22%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESC%3C/text%3E%3C/svg%3E)
Shweta Chavan
40
Reputation points
commented 2024-05-01T21:33:16.9633333+00:00
Oury Ba-MSFT
16,241
Reputation points Microsoft Employee
0 answers
How do I use the Script activity in ADF, so it uses Azure Databricks SQL Warehouse
I want to be able to use ADF Script activity to execute SQL statements on the Azure Databricks SQL warehouses (including the serverless kind). https://learn.microsoft.com/en-us/azure/data-factory/transform-data-using-script Azure Databricks SQL…
asked 2024-05-01T12:21:53.01+00:00
Krzysztof Przysowa
0
Reputation points
commented 2024-05-01T20:08:24.46+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(86.4, 36%, 18%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EBM%3C/text%3E%3C/svg%3E)
BhargavaGunnam-MSFT
26,226
Reputation points Microsoft Employee
2 answers
One of the answers was accepted by the question author.
Parsing row counts from json output from an ADF script to pass to Stored Procedure
I have a ADF script that simply performs the following from a ForEach Loop. @Anonymous (concat(concat('select count(*) as num_rows from ', item().table_owner_schema), '.'), item().table_name) This works fine and returns the row count in the…
asked 2022-09-14T13:46:46.027+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(153.6, 30%, 24%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EHR%3C/text%3E%3C/svg%3E)
Haliburton, Russell
21
Reputation points
accepted 2024-05-01T18:28:53.0233333+00:00
Haliburton, Russell
21
Reputation points
0 answers
How to use next link for pagination in adf (I have to use pagination for web activity)
Hi, I am trying to build up an adf pipeline to pull data from NetSuite. The result of the api call of NetSuite is like this in postman As we can see that this returns next link with the data and now we have to us this next link for pagination in adf. …
asked 2024-05-01T17:29:46.6833333+00:00
Amar Agnihotri
891
Reputation points
edited the question 2024-05-01T17:31:05.45+00:00
Amar Agnihotri
891
Reputation points