Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,400 questions
1 answer
ADF using the new Postgresql connector is sometimes getting the error: Message=Could not load type 'NpgsqlTypes.NpgsqlInterval' from assembly 'Npgsql, Version=4.0.9.2. This has been running without error for months.
We have been using the new ADF Postgresql connector for months without issue but in the last ELT we are getting the error: Message=Could not load type 'NpgsqlTypes.NpgsqlInterval' from assembly 'Npgsql, Version=4.0.9.2. We are able to successfully…
asked Apr 4, 2025, 2:46 PM
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(44.800000000000004, 47%, 14%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EJ%3C/text%3E%3C/svg%3E)
JB
116
Reputation points
commented Apr 6, 2025, 2:41 AM
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(217.60000000000002, 5%, 31%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EJA%3C/text%3E%3C/svg%3E)
Josh Ashwood
96
Reputation points
3 answers
How to update the odata linked service with access token dynamically in azure data factory
We are trying to connect to share point using odata connector from azure data factory with below Authorization mode. Authorization : Anonymous Additional Headers : Bearer $accesstoken Is there a way to replace the access token in the linked…
asked Oct 13, 2021, 8:30 AM
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(240, 23%, 33%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EPU%3C/text%3E%3C/svg%3E)
Praveena Uppalapati [ELB]
51
Reputation points
commented Apr 5, 2025, 9:32 PM
Seth Coleman
51
Reputation points
1 answer
How do I prevent subsequent pagination calls from being written on separate lines in JSON file?
I have a copy data activity in Azure Data Factory that is copying data from a paginated REST API source and to a JSON file sink. When it completes, the first line of the JSON file is an array of all data aggregated together from each paginated pull. …
asked Apr 4, 2025, 11:00 PM
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(105.60000000000001, 69%, 20%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EST%3C/text%3E%3C/svg%3E)
Sean Tressler
0
Reputation points
answered Apr 5, 2025, 10:33 AM
Vinodh247
31,436
Reputation points MVP
1 answer
Problems connecting ADF to ServiceNow with large tables.
Good day to you all, We use ADF to retrieve many data from many systems. Also from SNOW (ServiceNow) we had this working and the end of November 2024. Now we encounter problems with the tables who are big. Like sn_customersrvice_case When I now try to…
asked Jan 14, 2025, 10:32 AM
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(70.4, 53%, 16%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EMR%3C/text%3E%3C/svg%3E)
Marc Rijnders
10
Reputation points
edited a comment Apr 4, 2025, 10:39 PM
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(291.2, 3%, 38%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EJN%3C/text%3E%3C/svg%3E)
Jason Naccarato
0
Reputation points
3 answers
Unable to Delete an Azure Data Factory Ghost Managed Private Endpoint
I am working to connect an Azure mysql instance to our new ADF and using Managed Private Endpoints to do it. During the testing process, I used the Linked Services GUI to create a link to our Mysql server. Later, I needed to test without the managed…
asked Apr 2, 2025, 3:21 PM
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(169.60000000000002, 55.00000000000001%, 26%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EHF%3C/text%3E%3C/svg%3E)
Hunter French
0
Reputation points
commented Apr 4, 2025, 4:35 PM
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(99.2, 65%, 19%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ECB%3C/text%3E%3C/svg%3E)
Chandra Boorla
10,900
Reputation points Microsoft External Staff
1 answer
ADF PII Detection and Masking - Schema Drift Issue
I am following the tutorial for setting up a pipeline for PII detection and masking (linked below). All steps have been followed exactly as described, without any changes to the pipeline settings or configurations. However, I am encountering an issue…
asked Nov 15, 2024, 7:42 PM
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(121.6, 7.000000000000001%, 21%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EMA%3C/text%3E%3C/svg%3E)
Muhammad Abdulbaqi
20
Reputation points
commented Apr 4, 2025, 3:24 PM
Chandra Boorla
10,900
Reputation points Microsoft External Staff
0 answers
How to fetch Schema of a file stored in Azure Blob using ADF
Hi, I'm sending data from Azure Blob to AWS S3 (As AWS S3 isn't supported as sink dataset in Copy Data Activity I'm using Mapping Dataflows). I need to write the schema to the sink, so my pipeline has to write data in one directory and schema file in…
asked Apr 3, 2025, 7:47 AM
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(89.60000000000001, 82%, 18%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EMM%3C/text%3E%3C/svg%3E)
Madan M R
0
Reputation points
commented Apr 4, 2025, 1:58 PM
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(195.2, 50%, 28%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EVR%3C/text%3E%3C/svg%3E)
Venkat Reddy Navari
725
Reputation points Microsoft External Staff
2 answers
Connectivity from Azure Data Factory to SFTP via NAT Gateway fails
I have to reach to the client sftp through my adf linked service. Client needs a single static ip to whitelist on their end. I created a SHIR within adf in a subnet that further is associated with a NAT gateway public ip. I requested the client to…
asked Mar 17, 2025, 12:31 PM
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(22.400000000000002, 64%, 12%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EPP%3C/text%3E%3C/svg%3E)
Pratima Patil
30
Reputation points
commented Apr 4, 2025, 11:53 AM
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(6.4, 57.00000000000001%, 10%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ERG%3C/text%3E%3C/svg%3E)
Rakesh Govindula
580
Reputation points Microsoft External Staff
1 answer
ADF & ServiceNow V2 connector with Dynamic pipeline success
I am new to ADF and looking to utilise the ServiceNow V2 connector. This is to replace on existing on-prem SSIS package using the ServiceNow ODBC driver. I have the required ServiceNow permissions & created a simple pipeline to test copying data from…
asked Apr 1, 2025, 1:42 PM
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(64, 4%, 16%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EDS%3C/text%3E%3C/svg%3E)
darren smith
0
Reputation points
commented Apr 4, 2025, 10:46 AM
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(233.6, 87%, 32%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESD%3C/text%3E%3C/svg%3E)
Smith, Darren (IT-LON)
0
Reputation points
1 answer
How to copy azure storage tables into a file with the types of every entity?
Hi, I have a pipeline that copies azure storage tables into csv file and I'm losing the types of every entity when I copying it from file to table. How can I save the entity with all the types?
asked Apr 1, 2025, 8:44 AM
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(121.6, 81%, 21%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EOM%3C/text%3E%3C/svg%3E)
OGINTZ Marina
130
Reputation points
commented Apr 4, 2025, 9:34 AM
Rakesh Govindula
580
Reputation points Microsoft External Staff
0 answers
How to dynamically use column NAMES (in value expression) from a parameter in ADF Dataflow Derived Column transformation?
I'm working with Azure Data Factory Dataflow and trying to implement a solution where I need to dynamically reference column names in a Derived Column transformation. Current Working Solution (Hard-coded) Currently, I have this pattern that works…
asked Apr 2, 2025, 1:24 AM
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(201.6, 91%, 29%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EJS%3C/text%3E%3C/svg%3E)
Jaewon Song
0
Reputation points
commented Apr 4, 2025, 9:15 AM
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(288, 61%, 37%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EP%3C/text%3E%3C/svg%3E)
phemanth
14,980
Reputation points Microsoft External Staff
0 answers
How to integrated Azure in powerplatform.
Connect Azure database to powerapps as currently unable to connect
asked Apr 4, 2025, 8:07 AM
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(278.4, 83%, 36%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EGC%3C/text%3E%3C/svg%3E)
Gudivaka, Chandra Sekhar
0
Reputation points
commented Apr 4, 2025, 8:09 AM
Nandan Hegde
34,696
Reputation points MVP
0 answers
ADF Dataflow failed to update lookup field in Dataverse
Hi, I need some help with Azure Data Factory. I built a flow to update one of the fields inside Dataverse at Sink step and I am getting the following error message: Job failed due to reason: Error response from server:…
asked Apr 3, 2025, 9:45 AM
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(16, 64%, 11%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EHC%3C/text%3E%3C/svg%3E)
Henry Choi
0
Reputation points
commented Apr 4, 2025, 8:01 AM
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(268.8, 8%, 36%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EST%3C/text%3E%3C/svg%3E)
Smaran Thoomu
21,700
Reputation points Microsoft External Staff
0 answers
Existing Dataset in ADF pipeline not working for Data Flow
I have a sql server db on a vm that I remote into from the Azure portal using bastion. The DB has private endpoints and so forth as part of it's security arrangements. I have set up a linked service in the ADF that allows me to connect pipelines with…
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(80, 10%, 17%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EB%3C/text%3E%3C/svg%3E)
0 answers
Connecting ADF to Private Key Vault and VM in VNET with Corporate Firewall
A scenario involves a private Key Vault, private SQL, private storage account, private Azure Data Factory (ADF), and a private VM with SHIR hosted on it. All resources have private endpoints created within the same VNET. The SHIR hosted on the VM has…
asked Apr 2, 2025, 7:27 AM
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(169.60000000000002, 46%, 26%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EVA%3C/text%3E%3C/svg%3E)
Vishesh Agarwal
0
Reputation points
commented Apr 4, 2025, 12:39 AM
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(252.8, 99%, 34%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EJN%3C/text%3E%3C/svg%3E)
J N S S Kasyap
1,175
Reputation points Microsoft External Staff
3 answers
ADF pipeline is failed connecting to Azure SQL
we are running a job to load the data from SQL Source to D365 Sink which runs for several hours and loads the data but intermittently SQL source connection was forcibly closed by the error message posted below SQL DB is configured with 100 DTUs and…
asked Sep 23, 2022, 7:44 AM
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(128, 74%, 22%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EMK%3C/text%3E%3C/svg%3E)
Mandya Krishna, Vikyath
1
Reputation point
commented Apr 3, 2025, 10:01 PM
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(304, 27%, 39%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EPH%3C/text%3E%3C/svg%3E)
Peter Holmes
0
Reputation points
1 answer
One of the answers was accepted by the question author.
replace special character PU2 from a file in dataflow
I have source file in dataflow and few rows in the file that have PU2 special character. How do i replace this using the derived transformation in dataflow?
asked Mar 31, 2025, 2:28 PM
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(137.6, 23%, 23%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EPS%3C/text%3E%3C/svg%3E)
Peruka, Sikander reddy
155
Reputation points
accepted Apr 3, 2025, 3:11 PM
Peruka, Sikander reddy
155
Reputation points
1 answer
How to upsert data into salesforce using data factory?
I am trying to perform an upsert into salesforce from data factory. I am getting this error........ Failure happened on 'Sink' side. ErrorCode=InvalidParameter,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=The value of the…
asked Mar 28, 2025, 3:23 PM
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(153.6, 55.00000000000001%, 24%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EBD%3C/text%3E%3C/svg%3E)
Blasko, Dan W
40
Reputation points
commented Apr 3, 2025, 12:38 PM
Blasko, Dan W
40
Reputation points
1 answer
ADLS Folder Deletion Issue in ADF Pipeline
I am encountering an issue while creating a pipeline in Azure Data Factory (ADF) to delete folders in Azure Data Lake Storage (ADLS). The pipeline includes a 'Get Metadata' activity to retrieve a list of folders to delete, but it fails with the following…
asked Mar 31, 2025, 4:13 PM
Jake Atkins
20
Reputation points
commented Apr 2, 2025, 6:13 PM
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(201.6, 18%, 29%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EGG%3C/text%3E%3C/svg%3E)
Ganesh Gurram
5,875
Reputation points Microsoft External Staff
1 answer
One of the answers was accepted by the question author.
Is it possible to join the ADFSandboxActivityRun and AzureActivity tables in ADF Monitoring Logs?
Good Day Everyone! I would love to combine the "caller" details from the AzureActivity table and the details from the ADFSandboxActivityRun table within ADF Logs. However, I'm not seeing a field on which I could join. I do see…
asked Mar 29, 2025, 5:48 PM
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(265.6, 88%, 35%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EBB%3C/text%3E%3C/svg%3E)
Brittany Blackwell
25
Reputation points
commented Apr 2, 2025, 4:45 PM
Vinodh247
31,436
Reputation points MVP