ADF Copy Activity : Data load from ADLS (parquet files) to Azure Synapse Table Failing after running for 24 hours

Question

ADF pipeline copy failed after running for 24 hours. Timeout is set to 7days.

Copy Activity is trying to copy data from ADLS (Parquet files) to Azure Synapse Table on Production Environment . Could you please suggest what could be the possible issue for it failing after running for a long time. Below is the error snippets.

Pipeline run id - f4fc2e67-ea18-4dd9-8d1c-1831a79e888b

ADF PipelineCopy Error .png

Pipeline_Snippet_with_time.png

Error codeUserErrorSqlDWCopyCommandError

Failure typeUser configuration issue

DetailsErrorCode=UserErrorSqlDWCopyCommandError,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=SQL DW Copy Command operation failed with error 'Cannot bulk load because the file "/187e9772-466a-40d0-96fd-5729dd322865/SynapseImportCommand/part-00000-tid-7576321053363675784-972a1d61-48dc-4829-9abe-df7d2e2a1693-235-1-c000.snappy.txt" is incomplete or could not be read. Operating system error code 5(Access is denied.).',Source=Microsoft.DataTransfer.ClientLibrary,''Type=System.Data.SqlClient.SqlException,Message=Cannot bulk load because the file "/187e9772-466a-40d0-96fd-5729dd322865/SynapseImportCommand/part-00000-tid-7576321053363675784-972a1d61-48dc-4829-9abe-df7d2e2a1693-235-1-c000.snappy.txt" is incomplete or could not be read. Operating system error code 5(Access is denied.).,Source=.Net SqlClient Data Provider,SqlErrorNumber=4862,Class=16,ErrorCode=-2146232060,State=5,Errors=[{Class=16,Number=4862,State=5,Message=Cannot bulk load because the file "/187e9772-466a-40d0-96fd-5729dd322865/SynapseImportCommand/part-00000-tid-7576321053363675784-972a1d61-48dc-4829-9abe-df7d2e2a1693-235-1-c000.snappy.txt" is incomplete or could not be read. Operating system error code 5(Access is denied.).,},],'

Answer 1

Hi @Kapoor, Sipra (S)

Thank you for reaching out to the community forum with your query.

Based on the error message you provided, it seems that the copy activity is failing due to an access denied error. The error message indicates that the file "/187e9772-466a-40d0-96fd-5729dd322865/SynapseImportCommand/part-00000-tid-7576321053363675784-972a1d61-48dc-4829-9abe-df7d2e2a1693-235-1-c000.snappy.txt" is incomplete or could not be read, and the operating system error code is 5 (Access is denied).

Here are some possible reasons for this error:

1. The account used to access the file does not have the necessary permissions to read the file.
2. The file is locked by another process or user.
3. The file is corrupted or incomplete.

To resolve this issue, you can try the following steps:

1. Check the permissions of the account used to access the file. Ensure that the account has the necessary permissions to read the file.
2. Check if the file is locked by another process or user. If so, wait for the lock to be released and try again.
3. Check if the file is corrupted or incomplete. If so, try to repair or replace the file.

If the issue persists, you can try to copy the file to a different location and then try to copy it to the Azure Synapse table. You can also check the Azure Synapse table's firewall settings to ensure that the IP address of the machine running the copy activity is allowed to access the table.

Hope this helps. Do let us know if you any further queries.

---

If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.