Azure Data Lake Storage
An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.
1,380 questions
1 answer
Delete the file from SharePoint location
Hi All, I am trying to copy the files from Share Point to ADLS and referring to the below URL pipeline to achieve the copy functionality. https://www.syntera.ch/blog/2022/10/10/copy-files-from-sharepoint-to-blob-storage-using-azure-data-factory/ I need…
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(96, 24%, 19%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EA%3C/text%3E%3C/svg%3E)
1 answer
How to add Auth Headers in REST Linked Service in Azure Data Factory (ADF) when using oAuth2 Client Credentials?
My API provider requires an API-Key additional header when using oAuth2 Client Credential to generate access token. In Azure DataFactory, when I create a REST linked service using OAuth2 Client Credential as authentication type, I also include Auth…
asked 2024-03-28T18:18:03.04+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(64, 89%, 16%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EEM%3C/text%3E%3C/svg%3E)
Eduardo Moralles
0
Reputation points
edited a comment 2024-05-30T05:15:15.33+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(12.8, 48%, 11%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EH%3C/text%3E%3C/svg%3E)
Harishga
4,485
Reputation points Microsoft Vendor
1 answer
How to fix "Failure happened on 'Source' side A task was canceled" error
Data load from Salesforce object with 1million records fails with the below error in Azure Data Factory: Failure happened on 'Source' side. 'Type=System.Threading.Tasks.TaskCanceledException,Message=A task was canceled.,Source=mscorlib,' The data load…
asked 2024-05-28T18:42:40.6+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(300.8, 64%, 39%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ETK%3C/text%3E%3C/svg%3E)
thara kollu
0
Reputation points
commented 2024-05-30T04:32:45.88+00:00
Harishga
4,485
Reputation points Microsoft Vendor
1 answer
ADF lost access to Dynamics CRM / Dataverse after MFA was turned on
Today my ADF pipelines started throwing errors: Operation on target Process order failed: ErrorCode=DynamicsFailedToConnect,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=Failed to connect to Dynamics: Unable to Login to…
asked 2024-05-09T13:00:44.1466667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(99.2, 20%, 19%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3E14%3C/text%3E%3C/svg%3E)
11-4688
81
Reputation points
edited the question 2024-05-30T04:21:05.7133333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(281.6, 17%, 37%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EGM%3C/text%3E%3C/svg%3E)
Givary-MSFT
28,756
Reputation points Microsoft Employee
0 answers
Azure data factory pipelines stopped working because return pipeline variables are not longer recognized run time
Hi, My pipelines code has not changed for over 3 weeks and until yesterday there were no issues. All pipelines fail due to same issue: pipeline return variable set in child pipeline is not longer recognized in parent pipeline. System complains that…
asked 2024-05-26T19:08:40.2+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(86.4, 61%, 18%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3E6%3C/text%3E%3C/svg%3E)
61521973
31
Reputation points
commented 2024-05-30T04:04:22.79+00:00
AnnuKumari-MSFT
31,731
Reputation points Microsoft Employee
1 answer
Failed to convert the value in 'container' property to 'System.String' type. Please make sure the payload structure and value are correct.
We are creating a Pipeline consisting of some Lookup Activity, Set Variables and Copy Activity using REST APIs provided by Azure Data Factory. Pipeline has some parameters and Variables defined at Pipeline level and at Data set levels. Pipeline and its…
asked 2024-05-09T09:14:54.8866667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(0, 78%, 10%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EKS%3C/text%3E%3C/svg%3E)
Karthik S Gothe
5
Reputation points
commented 2024-05-30T04:02:14.6366667+00:00
AnnuKumari-MSFT
31,731
Reputation points Microsoft Employee
1 answer
filter and lookup transformation
Hi, have 2 tables table1 .jobid 1 table2 tjob2.jobid1 and jobname='test' how to filter jobid1 where jobname<>'test'
asked 2024-05-16T10:11:47.8166667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(204.8, 13%, 29%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EVS%3C/text%3E%3C/svg%3E)
Vineet S
205
Reputation points
commented 2024-05-30T03:45:05.6866667+00:00
AnnuKumari-MSFT
31,731
Reputation points Microsoft Employee
2 answers
One of the answers was accepted by the question author.
Not able to connect to on premise sql server using self integration run time in azure data factory
I am able connect to on premise sql server using SSMS tool but while connecting through azure data factory using self hosted integration runtime and linked service getting below error. No mistakes in user name and password .Please help me on this. { …
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(243.2, 32%, 33%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ER%3C/text%3E%3C/svg%3E)
1 answer
how to split a csv file based on columns.
Hi Team, There is a scenario in which we have to split columns of a CSV file into unique files in blob. for example let's say I need to create each file based on the combination of (TimeStart + TimeEnd + A) i.e. three columns per file and filename would…
asked 2024-05-22T10:50:06.97+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(204.8, 72%, 29%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EGN%3C/text%3E%3C/svg%3E)
Gorakh Nath Sharma
0
Reputation points
commented 2024-05-29T22:39:08.49+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(86.4, 36%, 18%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EBM%3C/text%3E%3C/svg%3E)
BhargavaGunnam-MSFT
27,976
Reputation points Microsoft Employee
0 answers
How can we run the different level of folders in a heirarchy in batches?
I have a doubt if we can approach something like this in ADF. I have created a for each activity where I'm iterating through an array of folders. Let's say I'm passing folder I'd of A and then I'm calling a pipeline in for each that will return sub…
asked 2024-05-28T16:27:31.67+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(256, 35%, 34%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EMY%3C/text%3E%3C/svg%3E)
Mansi Yadav
40
Reputation points
commented 2024-05-29T19:28:53.4466667+00:00
BhargavaGunnam-MSFT
27,976
Reputation points Microsoft Employee
2 answers
One of the answers was accepted by the question author.
Data Factory - No erorrs
Hi In Data Preview tab I'm getting a message like "Transformation stream is invalid, fix errors to enable data preview. Click to see errors" But when I click to see errors there is no errors. What should I do?
asked 2022-09-30T14:47:33.637+00:00
Deniz Alkan
101
Reputation points
answered 2024-05-29T19:09:41.8533333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(0, 0%, 10%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EBB%3C/text%3E%3C/svg%3E)
Braun, Bill
5
Reputation points
1 answer
Solutions Architect Expert Certification
Hello, I unfortunately lost access to my microsoft learn account because of my previous company, so I could not renew my AZ104 exam. Today I just passed the AZ-305 exam, I would kindly ask you to provide me the opportunity to renew my AZ-104 exam so I…
asked 2024-05-23T14:25:00.76+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(198.4, 94%, 29%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EWG%3C/text%3E%3C/svg%3E)
Werner György Obring
0
Reputation points
edited the question 2024-05-29T15:51:18.3466667+00:00
Givary-MSFT
28,756
Reputation points Microsoft Employee
1 answer
Using Entra ID Auth Only in ADF + Azure SSIS IR solution
Hi, We're currently working on complying with Microsoft cloud security benchmark in our tenant and ran into an issue with our Azure Data Factory service. To comply, we need to have only Entra ID Auth on our Azure SQL DBs. We're using Azure Data Factory…
asked 2024-05-24T09:37:29.6033333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(281.6, 77%, 37%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EJL%3C/text%3E%3C/svg%3E)
Johan Lindholm
0
Reputation points
edited the question 2024-05-29T15:50:45.2+00:00
Givary-MSFT
28,756
Reputation points Microsoft Employee
0 answers
Takes more time for a data to get uploaded
The data that I upload it takes more time to index, that was around only 2kb but it is 1 day now still it is indexing is there anything I should do
asked 2024-05-28T07:36:16.64+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(204.8, 9%, 29%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EGS%3C/text%3E%3C/svg%3E)
Gopinath S
0
Reputation points
commented 2024-05-29T15:36:54.9766667+00:00
BhargavaGunnam-MSFT
27,976
Reputation points Microsoft Employee
1 answer
One of the answers was accepted by the question author.
pipeline fail during the sink of a copy data to parquet
I have an issue with a pipeline that works just fine with the same resources (storage account, linked server and integration runtime) in a different ADF (TEST env), in PREPROD, the difference is that the IR is shared as it comes from the TEST env, it…
asked 2021-10-07T20:17:08.02+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(288, 12%, 37%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ERD%3C/text%3E%3C/svg%3E)
Ruben Dario Reyes Monsalve
41
Reputation points
commented 2024-05-29T15:14:53.5033333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(195.2, 50%, 28%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EGB%3C/text%3E%3C/svg%3E)
Gokul B
20
Reputation points
5 answers
One of the answers was accepted by the question author.
Copy Data Activity failes on datatype uniqueidentifier when Staging is enabled and no column mapping configured.
Using Azure Data Factory Copy Data Activity from Azure SQL Database to Azure Synapse Analytics. The Copy Data Activity failed with InvalidCastException error 'The given value of type String from the data source cannot be converted to type…
asked 2020-06-30T14:49:54.783+00:00
XchangeVisions
301
Reputation points
commented 2024-05-29T12:50:24.26+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(179.20000000000002, 33%, 27%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EUB%3C/text%3E%3C/svg%3E)
Umar Butt
0
Reputation points
1 answer
One of the answers was accepted by the question author.
ADF deployment issue
Hi, I am using Azure data factory and setup different branches for developers. I have setup adf publish branch as well so that when developers are done with their changes, we can publish the changes to adf publish branch and then from that branch I can…
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(185.6, 60%, 27%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ER%3C/text%3E%3C/svg%3E)
1 answer
One of the answers was accepted by the question author.
Azure Data Factory: Fetch xml to return all the attributes not working properyl
Hi, Thanks in advance for your response. I am using copy activity to copy the data from one environment to another environment in dynamics 365. If I dint use any filter in the fetch xml query, it works fine. Otherwise i am getting missing key…
asked 2024-05-28T14:31:05.47+00:00
Maniraj K V
20
Reputation points
commented 2024-05-29T11:59:50.3333333+00:00
Maniraj K V
20
Reputation points
1 answer
"No result available for stream" error when performing join and previous step uses cached lookup
I have a data flow where I look up and cache rows that have been updated as part of an ETL pipeline. I then want to join that updated data with the existing data in the target database. The problem seems to be that when doing the join operation the part…
asked 2022-06-23T20:01:38.977+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(307.2, 46%, 39%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EJN%3C/text%3E%3C/svg%3E)
John Neubecker
6
Reputation points
commented 2024-05-29T09:06:02.9666667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(294.40000000000003, 92%, 38%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EMD%3C/text%3E%3C/svg%3E)
Moon Do
0
Reputation points
0 answers
Unable to resolve Azure database for mysql name from Data Factory
I am creating a linked service from my Azure Data Factory environment, to access my Azure Database for Mysql database. I select "Azure database for Mysql", and I can see all my mysql servers. I select "From Azure subscription" as the…
asked 2024-05-07T07:39:15.9766667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(249.60000000000002, 71%, 34%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EIS%3C/text%3E%3C/svg%3E)
Ignacio Sales
5
Reputation points
commented 2024-05-29T07:59:20.8933333+00:00
PRADEEPCHEEKATLA-MSFT
80,491
Reputation points Microsoft Employee