Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,806 questions
1 answer
how to call Oracle procedure in ADF lookup activity
We need call Oracle Procedure from the Azure Data Factory. It seems that in lookup activity only Select is supported. How can we call the Oracle Procedure without the modification on Oracle side?
asked 2024-01-04T13:38:07.0533333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(6.4, 93%, 10%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESK%3C/text%3E%3C/svg%3E)
Shi, Kangwen
0
Reputation points
edited a comment 2024-05-31T14:55:32.31+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(262.40000000000003, 97%, 35%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EMN%3C/text%3E%3C/svg%3E)
Maltais, Nancy
0
Reputation points
2 answers
how to check who created the log analytics workspace in azure but it created more then 90n days ago we can't able to check in activity log how to we check that?
how to check who created the log analytics workspace in azure ? it created more then 90n days ago we can't able to check in activity log how to we check in another way? kindly please share me how can we check that
asked 2024-05-31T11:00:29.9166667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(48, 16%, 14%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ETS%3C/text%3E%3C/svg%3E)
Tamil Selvan M
0
Reputation points
edited an answer 2024-05-31T13:04:39.7066667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(304, 1%, 39%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EVV%3C/text%3E%3C/svg%3E)
Vidya Viraktamath
85
Reputation points Microsoft Employee
2 answers
how to fix Error: Spark job failed: { "text/plain": "{\"runId\":\"2325e724-f898-471d-b9b3-1f28fc560b44\",\"sessionId\":\"9c4abfad-3cc6-4429-b30a-d70b25537d29\",\"status\":\"Failed\",\"payload\":{\"statusCode\":400,\"shortMessage\":\"java.lang.Exception:
while i am trying to Data preview getting this error
asked 2024-05-29T14:14:58.2933333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(198.4, 0%, 29%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EN%3C/text%3E%3C/svg%3E)
NK
0
Reputation points
commented 2024-05-31T12:28:37.3866667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(268.8, 8%, 36%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EST%3C/text%3E%3C/svg%3E)
Smaran Thoomu
11,045
Reputation points Microsoft Vendor
2 answers
couldnt able to download the files completely by using copy activity download from website always downloading partially
whenever i try to download the files from website by using copy activity its downloaded partially 124.kb instead if 240MB - 350MB ..
asked 2024-05-25T00:50:10.38+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(272, 21%, 36%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EAB%3C/text%3E%3C/svg%3E)
Anilkumar Balakumaran (z)
0
Reputation points
commented 2024-05-31T09:41:55.29+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(281.6, 32%, 37%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EAB%3C/text%3E%3C/svg%3E)
Anilkumar Balakumaran
0
Reputation points
2 answers
One of the answers was accepted by the question author.
How to apply Wildcard REGEX filename from SFTP using ADF ?
I have a list of files on an SFTP server that I need to copy and move to a blob storage container. The filenames should match the following wildcard pattern: (FullName)[0-9](-)[0-9](-)[0-9]*(.)(CSV). Here are the steps I've taken: Created a new…
4 answers
Azure Synapse Analytics Failed to setup debug session
Right now I'm not able to run a data flow task on our Azure Synapse environment. When I try to start a debug session I get the message Failed to setup debug session. When I trigger the pipeline instead the task fails with "Operation on target…
asked 2022-09-20T12:40:49.163+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(128, 8%, 22%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EA%3C/text%3E%3C/svg%3E)
Anonymous
answered 2024-05-31T09:04:57.9933333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(83.2, 74%, 18%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESC%3C/text%3E%3C/svg%3E)
srivally chintakindi
0
Reputation points
2 answers
One of the answers was accepted by the question author.
Understand costs AZ Databricks job
I have an Azure Data Factory pipeline that executes multiple Databricks Notebooks using job clusters. I need to track the cost of these job clusters, including both the Databricks and the underlying VM costs, specifically for this set of jobs. Currently,…
asked 2024-05-31T07:58:49.75+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(0, 64%, 10%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EW%3C/text%3E%3C/svg%3E)
WeirdMan
200
Reputation points
answered 2024-05-31T08:55:07.06+00:00
Vinodh247-1375
11,726
Reputation points
1 answer
Call API with dynamic URL and store JSON results in DataLake Storage
I am trying to make implement the following scenario using Data Factory: I am making multiple API calls, relying on a dyanmic URL (e.g. "url.com/api/{ID}" for a list of IDs). The resulting JSON from each of the API calls should be stored as a…
asked 2024-05-30T11:55:42.17+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(16, 13%, 11%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ELK%3C/text%3E%3C/svg%3E)
Lucas Kluge
0
Reputation points
answered 2024-05-31T08:47:11.1566667+00:00
Amira Bedhiafi
17,201
Reputation points
1 answer
Choosing an Approach for Incremental Loading with Watermark in Azure Data Factory: Efficiency and Cost Considerations
Hi all, I'm working on implementing an Azure Data Factory pipeline for incremental data loading using a watermark table approach. I have identified two different approaches but am unsure which one is considered the best practice in terms of…
asked 2024-01-13T01:55:43.14+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(310.4, 87%, 40%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ELM%3C/text%3E%3C/svg%3E)
Lucas Medina
20
Reputation points
edited a comment 2024-05-31T08:16:33.87+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(214.4, 75%, 30%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EA4%3C/text%3E%3C/svg%3E)
AbhishekKrishna-4295
0
Reputation points
1 answer
How to compare two dates in ADF to see which is greater?
Hi there, I am using Azure Data Factory to compare two dates (one is stored in a string variable and one its utcnow) and I need to know when the utcnow is greater than the date in the stored string variable. Both are just dates no timestamps I looked at…
asked 2024-05-30T12:03:14.8966667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(160, 57.99999999999999%, 25%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EML%3C/text%3E%3C/svg%3E)
Mike Langlois
5
Reputation points
commented 2024-05-31T08:02:21.2066667+00:00
Smaran Thoomu
11,045
Reputation points Microsoft Vendor
1 answer
Data flow failing at Sink Step when pipeline is scheduled
I have Pipeline with Data flow debug where data is sinked into Data set. When run manually it runs fine and the parquet file is created. However when scheduled in Pipeline it gives following error. Pipeline runs fine only gives error on schedule…
asked 2024-05-30T07:21:04.75+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(140.8, 32%, 23%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ENZ%3C/text%3E%3C/svg%3E)
Nazish Zahra
0
Reputation points
commented 2024-05-31T07:47:28.8633333+00:00
Amira Bedhiafi
17,201
Reputation points
0 answers
Except Unique Assert Transformation in Dataflow
I am trying to use Asset transformation in dataflow to check the dups in source. I am not able to get the correct result when using parameters that will have single or multiple columns as primary keys (example shared below) when using below expression. …
asked 2024-05-26T08:02:22.9033333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(83.2, 24%, 18%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EV%3C/text%3E%3C/svg%3E)
Vaibhav
105
Reputation points
commented 2024-05-31T07:39:10.81+00:00
Smaran Thoomu
11,045
Reputation points Microsoft Vendor
2 answers
One of the answers was accepted by the question author.
Azure HTTP Linked service ADF Username only no password
I am trying to use Azure HTTP Linked services to connect an API which uses just a username and no password for basic auth. The password input is mandatory in Azure. Anything i can do?
asked 2023-05-22T16:38:33.0066667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(272, 48%, 36%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ET%3C/text%3E%3C/svg%3E)
TeriTech03
26
Reputation points
answered 2024-05-30T21:05:19.4166667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(96, 48%, 19%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EDR%3C/text%3E%3C/svg%3E)
Dmitriy Ryabin
1
Reputation point
1 answer
Obvious behavior of dynamic mapping in ADF
I have a single copy activity that moves data from a source parquet file to an azure sql database. When I used auto mapping and tried to change the name of source or sink field it failed, however when I copy the json mapping generated and change one of…
asked 2024-05-29T06:44:25.6133333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(134.4, 61%, 23%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EBB%3C/text%3E%3C/svg%3E)
Bens Bernard
0
Reputation points
commented 2024-05-30T15:58:12.75+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(86.4, 36%, 18%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EBM%3C/text%3E%3C/svg%3E)
BhargavaGunnam-MSFT
27,976
Reputation points Microsoft Employee
1 answer
One of the answers was accepted by the question author.
Split column data into another column
Hi There, I want to extract data from a column , which has key-value pairs separated by commas, into respective columns using Azure Data Factory data flow. Example Data of Column containing key value…
asked 2024-05-30T04:31:35.6766667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(262.40000000000003, 26%, 35%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EKS%3C/text%3E%3C/svg%3E)
Kothuri, Sravya
20
Reputation points
edited a comment 2024-05-30T12:39:28.2966667+00:00
Kothuri, Sravya
20
Reputation points
1 answer
One of the answers was accepted by the question author.
ADF does not allow MI based auth for Azure table storage. Are there any alternatives for this?
I am trying to move away from key-based access for an azure table storage in ADF, but the only auth options I get are account key and SAS URI. Is there no way to use MI/SP for this? Blob storage has MI based auth but not table, why is this so?
asked 2024-05-29T12:15:25.3933333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(128, 45%, 22%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EAA%3C/text%3E%3C/svg%3E)
Akshita Arora
20
Reputation points Microsoft Employee
accepted 2024-05-30T11:01:19.96+00:00
Akshita Arora
20
Reputation points Microsoft Employee
2 answers
How to migrate bigger repo to azure devops
Hello, I am migrating from GIT to Devops. My repo is quite huge and while doing git push I am getting error "HTTP 413 curl 22 the requested url retuned error: 413". I am using Https to do migration. SOmewhere I saw ssh can solve this issue.…
asked 2024-05-24T11:32:14.2133333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(217.60000000000002, 28.999999999999996%, 31%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EJS%3C/text%3E%3C/svg%3E)
Jamesphilip, Sijith (Baker Hughes Contractor)
0
Reputation points
commented 2024-05-30T09:42:10.6166667+00:00
Amira Bedhiafi
17,201
Reputation points
2 answers
Default TImezone in the Dataflow
Hi Team, Is there anyway to change the default timezone in the datafactory? Currently, when I am creating a column(loadtime) with currentTimestamp() in derived column transformation in the dataflow, then I get the time in UTC. But, I want the default…
asked 2024-05-27T20:45:04.3433333+00:00
Vaibhav
105
Reputation points
edited a comment 2024-05-30T09:30:00.5133333+00:00
Amira Bedhiafi
17,201
Reputation points
1 answer
Hubspot connector doesn't work for deal properties: 414 Request-URI Too Large
I am getting this error message when trying to copy data from Deal properties from Hubspot: Failure happened on 'Source' side. ErrorCode=UserErrorOdbcOperationFailed,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=ERROR [HY000]…
asked 2024-05-21T07:12:35.4133333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(32, 49%, 13%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EBB%3C/text%3E%3C/svg%3E)
Boglarka Barany
0
Reputation points
commented 2024-05-30T09:13:36.9733333+00:00
Amira Bedhiafi
17,201
Reputation points
1 answer
One of the answers was accepted by the question author.
How can I force a failure when testing an ADF Webhook?
I would like to try to force a failure when testing an ADF WebHook activity so it follows the failure path. Is there a way I can easily do this? I can try to swap the on success and on failure paths to test the path, but I'd really like to force an…
asked 2024-05-30T03:46:34.86+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(44.800000000000004, 47%, 14%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EJ%3C/text%3E%3C/svg%3E)
JB
41
Reputation points
commented 2024-05-30T07:05:39.8666667+00:00
PRADEEPCHEEKATLA-MSFT
80,496
Reputation points Microsoft Employee