Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,805 questions
3 answers
Is there a way to handle SQL insert errors in copy activities within a pipelines?
Say when I am copying a sql table from one database to another (azure sql). I have 1000 rows, but one fails with a SQL exception on insert. Is there a way to handle these errors so 999 rows are inserted and the other one is saved in logs or azure…
asked 2020-08-06T17:51:46.837+00:00
Jose Sanchez Rico
1
Reputation point Microsoft Employee
answered 2020-08-06T22:08:04.597+00:00
Jose Sanchez Rico
1
Reputation point Microsoft Employee
2 answers
One of the answers was accepted by the question author.
To create a Pipeline that generates ‘incremental’ folder names
I have the following task: I have a Pipeline in an Azure Data factory (V2). The Pipeline contains several Copy data elements. Each of them copies data from an SQL database into an FTP folder. As a result, the Pipeline generates files like…
asked 2020-07-31T16:50:15.283+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(16, 57.99999999999999%, 11%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EVS%3C/text%3E%3C/svg%3E)
Victor Sotnikov
21
Reputation points
commented 2020-08-06T17:17:30.403+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(172.8, 70%, 26%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EMM%3C/text%3E%3C/svg%3E)
MartinJaffer-MSFT
26,041
Reputation points
1 answer
One of the answers was accepted by the question author.
Access Azure DataLake G2 from Azure subscription
Hello I 'm deploying an ETL scenario to move data from SQL server to Azure DataLake G2. When starting the process, I can see that specifying Azure DataLake G2 as target is defining Azure Blob Storage as target. Is this the right procedure ? …
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(153.6, 34%, 24%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EC%3C/text%3E%3C/svg%3E)
1 answer
One of the answers was accepted by the question author.
Data Factory cant see the Stored Procedure in my Azure SQL database
In SQL I have created a user and rule with full access to my schemas -- Create the user with Password for users that authenticate at the database CREATE USER DBowner WITH PASSWORD = 'pwrd'; --Im going to create a role and I want that rule to have…
asked 2020-08-06T10:29:43.197+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(204.8, 27%, 29%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EDE%3C/text%3E%3C/svg%3E)
Debbie Edwards
521
Reputation points
accepted 2020-08-06T10:48:08.96+00:00
Debbie Edwards
521
Reputation points
2 answers
File system Linked Service User Name and password
I am trying to move On premises files to an azure data lake using the File System Linked Service as the source My Files are in One drive and I have set up Self Hosted Integration run time (Hopefully) My first Question is, if Its One Drive can I still…
asked 2020-07-31T09:45:20.8+00:00
Debbie Edwards
521
Reputation points
commented 2020-08-06T04:16:15.193+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(108.80000000000001, 9%, 20%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EHM%3C/text%3E%3C/svg%3E)
HarithaMaddi-MSFT
10,136
Reputation points
2 answers
Azure data factory concatenation of columns
Hi, for one of the requirement in azure data factory we need to derive a primary key column based on the parameter p_primary_key_cols Input data: int_col|varchar_col|date_col 10|xyz|2020-02-13 20|abc|2020-03-12 pipeline…
asked 2020-06-05T16:16:02.697+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(131.20000000000002, 8%, 22%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESE%3C/text%3E%3C/svg%3E)
saikumar ellendula
6
Reputation points
commented 2020-08-05T08:49:10.16+00:00
saikumar ellendula
6
Reputation points
4 answers
Azure Data Factory: precision and scale as parameters to toDecimal() Function
Hi, providing the precision and scale as parameters to toDecimal() function in Azure data factory is giving "Should be integer literal error". please provide a way to resolve this issue. Thanks, Saikumar
asked 2020-06-08T10:40:55.62+00:00
saikumar ellendula
6
Reputation points
answered 2020-08-05T08:39:50.457+00:00
saikumar ellendula
6
Reputation points
1 answer
One of the answers was accepted by the question author.
In spite of having an azure runtime allocated, each activity has a separate "AcquiringCompute" step?
Question: In our pipeline, we have around 10 mapping data flow activities, in serial fashion (one after another). Each of them are configured to use the same integration runtime (azure managed runtime). In spite of this configuration, each activity shows…
asked 2020-08-04T17:55:37.84+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(73.60000000000001, 71%, 17%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EAM%3C/text%3E%3C/svg%3E)
Akshay Mahajan
21
Reputation points
commented 2020-08-04T21:53:59.247+00:00
Akshay Mahajan
21
Reputation points
1 answer
To create a Pipeline that generates ‘incremental’ folder names.
I have the following task: I have a Pipeline in an Azure Data factory (V2). The Pipeline contains several Copy data elements. Each of them copies data from an SQL database into an FTP folder. As a result, the Pipeline generates files like…
asked 2020-07-31T17:23:54.053+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(163.2, 0%, 25%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EAM%3C/text%3E%3C/svg%3E)
Amita Menon
1
Reputation point Microsoft Employee
answered 2020-08-04T20:36:11.607+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(134.4, 9%, 23%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EKM%3C/text%3E%3C/svg%3E)
KranthiPakala-MSFT
46,427
Reputation points Microsoft Employee
1 answer
Cannot connect to SQL database (ADF) - Pipeline -> DataFlow -> Sink
Hi Team, {"message":"at Sink 'XXXXXXXXXXSink': java.lang.RuntimeException: Cannot connect to SQL database: 'jdbc:sqlserver://XXXXXXXXXX.database.windows.net;database=XXXXXXXXXX', 'User: XXXXXXXXXX'. Please check the linked service…
asked 2020-07-30T05:55:32.507+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(179.20000000000002, 24%, 27%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EKR%3C/text%3E%3C/svg%3E)
K Rameshwar Arya
1
Reputation point
commented 2020-08-04T20:31:56.823+00:00
KranthiPakala-MSFT
46,427
Reputation points Microsoft Employee
1 answer
Key vault Azure data factory problem
Hello I've created a Key vault service to store a secret with the credentials to access to my cosmos DB collection. the secret value is as i show you : …
asked 2020-08-03T10:21:47.5+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(96, 71%, 19%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ECG%3C/text%3E%3C/svg%3E)
CORONADO GRANADOS Diana Milena
126
Reputation points
answered 2020-08-03T14:34:49.997+00:00
Alex Drenea
6
Reputation points Microsoft Employee
1 answer
Mapping data flow design question
Hi all- I am a long time .net developer, doing some adf work. I am trying to design a data flow in which data from an azure sql database table is transformed and inserted into another azure sql database table. If the record is successfully inserted, I…
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(44.800000000000004, 52%, 14%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EL%3C/text%3E%3C/svg%3E)
1 answer
ADF Copy Issue - Long File Path names
Hello, I am using copy activity to copy the files from on premise to Azure Storage Account . The job is failing as there is a long file path name at the source. can someone help me how can i fix this issue I am getting the following error …
asked 2020-07-27T15:06:24.047+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(83.2, 75%, 18%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EKN%3C/text%3E%3C/svg%3E)
Krishna Nagesh Kukkadapu
21
Reputation points
commented 2020-07-31T20:47:48.053+00:00
KranthiPakala-MSFT
46,427
Reputation points Microsoft Employee
1 answer
Azure Data Factory Linked Service to On-Premises SQL Server with named instance
Hello, I'm trying to connect from Azure Data Factory to an on-premises SQL Server using the SQL Server linked service (e.g. servername\instance1). I already have an integration runtime installed, configured, and working (tested with other servers). …
asked 2020-07-23T15:14:12.393+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(38.4, 56.00000000000001%, 13%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EN%3C/text%3E%3C/svg%3E)
Nestor
46
Reputation points
commented 2020-07-31T20:45:07.17+00:00
KranthiPakala-MSFT
46,427
Reputation points Microsoft Employee
1 answer
One of the answers was accepted by the question author.
Pipeline fails at data flow
I have created a 'copy data' and 'data flow' and have added them to my pipeline. When debugging the pipeline, copy data runs successfully, however, it fails at data flow with error: {"message":"at Sink 'OutputStreamName':…
asked 2020-07-30T08:51:55.913+00:00
Rose Kipgen
26
Reputation points Microsoft Employee
commented 2020-07-31T12:51:27.627+00:00
HarithaMaddi-MSFT
10,136
Reputation points
2 answers
One of the answers was accepted by the question author.
Azure data factory copy activity cosmos db error
Hello I'm getting the following mistake when I try to make a mapping of an array on copy data activity from csv file to cosmos db. The structure on cosmos db for the item should be as follows: "InsuredItems": [ { "Accomodation":…
asked 2020-07-22T17:19:38.367+00:00
CORONADO GRANADOS Diana Milena
126
Reputation points
accepted 2020-07-31T06:28:17.603+00:00
CORONADO GRANADOS Diana Milena
126
Reputation points
1 answer
REST API JSON DATE FORMAT
I am trying to copy data from rest api source using the azure copy activity. I have used Rest Api as source and AZ SQL DB as target. But the json response I am receiving is having the date as below format: {Createddate: /date(345667999)/} But when I…
asked 2020-07-16T13:59:05.157+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(281.6, 0%, 37%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EMD%3C/text%3E%3C/svg%3E)
Mark de Jonge
1
Reputation point
commented 2020-07-30T18:49:37.733+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(192, 26%, 28%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EHM%3C/text%3E%3C/svg%3E)
HimanshuSinha-msft
19,381
Reputation points Microsoft Employee
1 answer
One of the answers was accepted by the question author.
How to insert data in Azure Synapse Analytics through Azure Data factory without PolyBase
I have tried using Move and Transfar activity in data factory but I'm getting 2 error at publishing: ' Data flow activity 'dataflow1' requires a stagingLinkedService Data flow activity 'dataflow1' requires a stagingStoragePath ' I don't want to…
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(185.6, 95%, 27%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EMD%3C/text%3E%3C/svg%3E)
1 answer
dual factor authentication in Data Factory
Hi We have a plan to setup Azure data factory to connect sftp , for this we require dual factor authentication that means both sftp user id and ssh public key exchange method . so is this possible in current Azure Data factory V2 ? I observed…
asked 2020-07-23T05:51:50.72+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(233.6, 63%, 32%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EDS%3C/text%3E%3C/svg%3E)
Devara, Santosh (Cloud Architect)
1
Reputation point
commented 2020-07-30T06:15:24.23+00:00
KranthiPakala-MSFT
46,427
Reputation points Microsoft Employee
1 answer
One of the answers was accepted by the question author.
How to copydata from RestApi using datafactory and save it in Datalake?
Hi Everyone, I'm trying to fetch data from REST API and save the json string it into DataLake and I'm getting an error. I've followed the steps mentioned here https://learn.microsoft.com/en-us/azure/data-factory/connector-rest &…
asked 2020-07-27T11:18:38.973+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(32, 19%, 13%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ES%3C/text%3E%3C/svg%3E)
stramzik
161
Reputation points
commented 2020-07-29T17:20:33.883+00:00
KranthiPakala-MSFT
46,427
Reputation points Microsoft Employee