question

EddyNava-8056 avatar image
0 Votes"
EddyNava-8056 asked PRADEEPCHEEKATLA-MSFT commented

Azure Data Factory V2 Bug, Scheduled triggered result gives an error but manually triggered works fine

I created a pipeline using copy data to extract from one database to another database

First both source and sink databases had a pk key of 3 columns, when copying data manually it worked like a charm, but when triggered it would give a pk duplicated error, so what i did was deleting the pk key on the sink side, and now it doesn't give error but it duplicates rows as it doesn't when manually

azure-data-factory
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

1 Answer

PRADEEPCHEEKATLA-MSFT avatar image
0 Votes"
PRADEEPCHEEKATLA-MSFT answered PRADEEPCHEEKATLA-MSFT commented

Hello @EddyNava-8056,

Welcome to the Microsoft Q&A platform.

Try to make sure you load data only once either manual execution or trigger execution based on your requirement.

If we don't want duplicate entries then we should have primary key or unique key constrain on table. Please Note, Once you have primary key constrain on table when you try to load duplicate data then pipeline will throw error saying duplicates not allowed.

For more details, refer to the below information:

Primary key violation when writing to SQL Server/Azure SQL Database/Azure Cosmos DB.

For example: Copy data from a SQL server to a SQL database. A primary key is defined in the sink SQL database, but no such primary key is defined in the source SQL server. The duplicated rows that exist in the source cannot be copied to the sink. Copy activity copies only the first row of the source data into the sink. The subsequent source rows that contain the duplicated primary key value are detected as incompatible and are skipped.

Hope this helps. Do let us know if you any further queries.


Please don’t forget to Accept Answer and Up-Vote wherever the information provided helps you, this can be beneficial to other community members.


· 2
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

I found there was another pipeline automatically triggered, scheduled to run at the same time, inserting to the same sink database causing the duplicated values

Thank you so much for your time

1 Vote 1 ·

Hello @EddyNava-8056,

Glad to know that your issue has resolved.

0 Votes 0 ·