Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,805 questions
1 answer
Export JSON string but exclude key-value pairs with null/empty values
I'm attempting to output some data originally in csv format from ADF as a JSON string but I need to exclude any key-value pairs where the value is null or empty (e.g. "Report": []). First I am using an Aggregate transformation to convert 3…
asked 2024-05-30T20:35:28.12+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(60.8, 54%, 15%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EDL%3C/text%3E%3C/svg%3E)
David Lang
20
Reputation points
answered 2024-05-30T20:35:28.1566667+00:00
Q&A Assist
1 answer
Obvious behavior of dynamic mapping in ADF
I have a single copy activity that moves data from a source parquet file to an azure sql database. When I used auto mapping and tried to change the name of source or sink field it failed, however when I copy the json mapping generated and change one of…
asked 2024-05-29T06:44:25.6133333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(134.4, 61%, 23%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EBB%3C/text%3E%3C/svg%3E)
Bens Bernard
0
Reputation points
commented 2024-05-30T15:58:12.75+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(86.4, 36%, 18%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EBM%3C/text%3E%3C/svg%3E)
BhargavaGunnam-MSFT
27,976
Reputation points Microsoft Employee
1 answer
One of the answers was accepted by the question author.
Split column data into another column
Hi There, I want to extract data from a column , which has key-value pairs separated by commas, into respective columns using Azure Data Factory data flow. Example Data of Column containing key value…
asked 2024-05-30T04:31:35.6766667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(262.40000000000003, 26%, 35%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EKS%3C/text%3E%3C/svg%3E)
Kothuri, Sravya
20
Reputation points
edited a comment 2024-05-30T12:39:28.2966667+00:00
Kothuri, Sravya
20
Reputation points
1 answer
One of the answers was accepted by the question author.
ADF does not allow MI based auth for Azure table storage. Are there any alternatives for this?
I am trying to move away from key-based access for an azure table storage in ADF, but the only auth options I get are account key and SAS URI. Is there no way to use MI/SP for this? Blob storage has MI based auth but not table, why is this so?
asked 2024-05-29T12:15:25.3933333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(128, 45%, 22%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EAA%3C/text%3E%3C/svg%3E)
Akshita Arora
20
Reputation points Microsoft Employee
accepted 2024-05-30T11:01:19.96+00:00
Akshita Arora
20
Reputation points Microsoft Employee
2 answers
How to migrate bigger repo to azure devops
Hello, I am migrating from GIT to Devops. My repo is quite huge and while doing git push I am getting error "HTTP 413 curl 22 the requested url retuned error: 413". I am using Https to do migration. SOmewhere I saw ssh can solve this issue.…
asked 2024-05-24T11:32:14.2133333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(217.60000000000002, 28.999999999999996%, 31%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EJS%3C/text%3E%3C/svg%3E)
Jamesphilip, Sijith (Baker Hughes Contractor)
0
Reputation points
commented 2024-05-30T09:42:10.6166667+00:00
Amira Bedhiafi
16,951
Reputation points
2 answers
Default TImezone in the Dataflow
Hi Team, Is there anyway to change the default timezone in the datafactory? Currently, when I am creating a column(loadtime) with currentTimestamp() in derived column transformation in the dataflow, then I get the time in UTC. But, I want the default…
asked 2024-05-27T20:45:04.3433333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(83.2, 24%, 18%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EV%3C/text%3E%3C/svg%3E)
Vaibhav
105
Reputation points
edited a comment 2024-05-30T09:30:00.5133333+00:00
Amira Bedhiafi
16,951
Reputation points
0 answers
Issue with Copy Dataverse data into Azure SQL using Synapse Link Template
I am experiencing issues using ADF I am trying to use the template "Copy Dataverse data into Azure SQL using Synapse Link" to move f&o data from datalake to Azure…
asked 2024-05-30T07:08:10.6766667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(275.2, 20%, 36%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EDO%3C/text%3E%3C/svg%3E)
Dipo Onibile
0
Reputation points
edited a comment 2024-05-30T09:13:48.52+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(288, 61%, 37%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EP%3C/text%3E%3C/svg%3E)
phemanth
6,885
Reputation points Microsoft Vendor
1 answer
Hubspot connector doesn't work for deal properties: 414 Request-URI Too Large
I am getting this error message when trying to copy data from Deal properties from Hubspot: Failure happened on 'Source' side. ErrorCode=UserErrorOdbcOperationFailed,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=ERROR [HY000]…
asked 2024-05-21T07:12:35.4133333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(32, 49%, 13%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EBB%3C/text%3E%3C/svg%3E)
Boglarka Barany
0
Reputation points
commented 2024-05-30T09:13:36.9733333+00:00
Amira Bedhiafi
16,951
Reputation points
1 answer
All ADF dataflow stages disappear after publish
Hello, I made a source for my Dataflow and added several stages to the existing one. I published and Dataflow showed an error with a new source. I tried to delete it, but could not delete it. I published again and all my stages disappeared. It is not…
asked 2024-05-29T11:22:44.82+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(275.2, 3%, 36%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EDV%3C/text%3E%3C/svg%3E)
Darjuš Vasiukevič
20
Reputation points
commented 2024-05-30T09:11:23.1133333+00:00
Amira Bedhiafi
16,951
Reputation points
0 answers
How to pass the correct audience when calling mssparkutils.credentials.getToken on Azure China (Mooncake) cloud?
I'm using Microsoft Spark Utilities (MSSparkUtils) with linked service to authenticate into Azure SQL using System Assigned Managed Identity (Synapse Workspace) on Azure China (Mooncake) cloud. However, when I call gettoken with the audience type…
asked 2024-05-21T04:20:59.6566667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(246.4, 75%, 33%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EAB%3C/text%3E%3C/svg%3E)
Abhishek Bhatt
0
Reputation points Microsoft Employee
commented 2024-05-30T08:06:00.3766667+00:00
PRADEEPCHEEKATLA-MSFT
80,491
Reputation points Microsoft Employee
1 answer
One of the answers was accepted by the question author.
How can I force a failure when testing an ADF Webhook?
I would like to try to force a failure when testing an ADF WebHook activity so it follows the failure path. Is there a way I can easily do this? I can try to swap the on success and on failure paths to test the path, but I'd really like to force an…
asked 2024-05-30T03:46:34.86+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(44.800000000000004, 47%, 14%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EJ%3C/text%3E%3C/svg%3E)
JB
41
Reputation points
commented 2024-05-30T07:05:39.8666667+00:00
PRADEEPCHEEKATLA-MSFT
80,491
Reputation points Microsoft Employee
1 answer
Delete the file from SharePoint location
Hi All, I am trying to copy the files from Share Point to ADLS and referring to the below URL pipeline to achieve the copy functionality. https://www.syntera.ch/blog/2022/10/10/copy-files-from-sharepoint-to-blob-storage-using-azure-data-factory/ I need…
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(96, 24%, 19%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EA%3C/text%3E%3C/svg%3E)
1 answer
How to add Auth Headers in REST Linked Service in Azure Data Factory (ADF) when using oAuth2 Client Credentials?
My API provider requires an API-Key additional header when using oAuth2 Client Credential to generate access token. In Azure DataFactory, when I create a REST linked service using OAuth2 Client Credential as authentication type, I also include Auth…
asked 2024-03-28T18:18:03.04+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(64, 89%, 16%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EEM%3C/text%3E%3C/svg%3E)
Eduardo Moralles
0
Reputation points
edited a comment 2024-05-30T05:15:15.33+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(12.8, 48%, 11%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EH%3C/text%3E%3C/svg%3E)
Harishga
4,485
Reputation points Microsoft Vendor
1 answer
How to fix "Failure happened on 'Source' side A task was canceled" error
Data load from Salesforce object with 1million records fails with the below error in Azure Data Factory: Failure happened on 'Source' side. 'Type=System.Threading.Tasks.TaskCanceledException,Message=A task was canceled.,Source=mscorlib,' The data load…
asked 2024-05-28T18:42:40.6+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(300.8, 64%, 39%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ETK%3C/text%3E%3C/svg%3E)
thara kollu
0
Reputation points
commented 2024-05-30T04:32:45.88+00:00
Harishga
4,485
Reputation points Microsoft Vendor
1 answer
ADF lost access to Dynamics CRM / Dataverse after MFA was turned on
Today my ADF pipelines started throwing errors: Operation on target Process order failed: ErrorCode=DynamicsFailedToConnect,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=Failed to connect to Dynamics: Unable to Login to…
asked 2024-05-09T13:00:44.1466667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(99.2, 20%, 19%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3E14%3C/text%3E%3C/svg%3E)
11-4688
61
Reputation points
edited the question 2024-05-30T04:21:05.7133333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(281.6, 17%, 37%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EGM%3C/text%3E%3C/svg%3E)
Givary-MSFT
28,756
Reputation points Microsoft Employee
0 answers
Azure data factory pipelines stopped working because return pipeline variables are not longer recognized run time
Hi, My pipelines code has not changed for over 3 weeks and until yesterday there were no issues. All pipelines fail due to same issue: pipeline return variable set in child pipeline is not longer recognized in parent pipeline. System complains that…
asked 2024-05-26T19:08:40.2+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(86.4, 61%, 18%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3E6%3C/text%3E%3C/svg%3E)
61521973
31
Reputation points
commented 2024-05-30T04:04:22.79+00:00
AnnuKumari-MSFT
31,731
Reputation points Microsoft Employee
1 answer
Failed to convert the value in 'container' property to 'System.String' type. Please make sure the payload structure and value are correct.
We are creating a Pipeline consisting of some Lookup Activity, Set Variables and Copy Activity using REST APIs provided by Azure Data Factory. Pipeline has some parameters and Variables defined at Pipeline level and at Data set levels. Pipeline and its…
asked 2024-05-09T09:14:54.8866667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(0, 78%, 10%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EKS%3C/text%3E%3C/svg%3E)
Karthik S Gothe
5
Reputation points
commented 2024-05-30T04:02:14.6366667+00:00
AnnuKumari-MSFT
31,731
Reputation points Microsoft Employee
1 answer
filter and lookup transformation
Hi, have 2 tables table1 .jobid 1 table2 tjob2.jobid1 and jobname='test' how to filter jobid1 where jobname<>'test'
asked 2024-05-16T10:11:47.8166667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(204.8, 13%, 29%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EVS%3C/text%3E%3C/svg%3E)
Vineet S
205
Reputation points
commented 2024-05-30T03:45:05.6866667+00:00
AnnuKumari-MSFT
31,731
Reputation points Microsoft Employee
0 answers
Authorization Error
Can you provide guidance on resolving an authorization error that is preventing the refresh of a Power BI dataset within an Azure Synapse Workspace? The error message received is "The Web Activity failed with HttpStatusCode - 'Unauthorized', and…
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(64, 18%, 16%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EK%3C/text%3E%3C/svg%3E)
2 answers
One of the answers was accepted by the question author.
Not able to connect to on premise sql server using self integration run time in azure data factory
I am able connect to on premise sql server using SSMS tool but while connecting through azure data factory using self hosted integration runtime and linked service getting below error. No mistakes in user name and password .Please help me on this. { …
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(243.2, 32%, 33%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ER%3C/text%3E%3C/svg%3E)