question

ChiDave-5749 avatar image
0 Votes"
ChiDave-5749 asked ChiDave-5749 commented

ADF Copy Data activity issue

Hello,

We have an ADF pipeline that is triggered daily for moving data from the on-prem server to the data lake storage. The data to be moved is incremental data from several tables. The pipeline runs usually go well, but on some days we run into an error. If we rerun the pipeline from failed activity then the run will succeed. This, however, involves manual action. Especially on the weekends, we don't want this to happen.

121166-image.png

The error occurs in the Copy data activity. Within this activity, the data from an on-prem SQL Server table will be moved to the Data Lake Storage in a parquet file. We are using a self-hosted integration runtime.

We are getting different error messages:
121128-image.png

121153-image.png

121154-image.png


Would you be able to help us out on this issue?

azure-data-factoryazure-data-lake-storage
image.png (10.7 KiB)
image.png (38.7 KiB)
image.png (47.8 KiB)
image.png (29.1 KiB)
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

1 Answer

VaibhavChaudhari avatar image
0 Votes"
VaibhavChaudhari answered ChiDave-5749 commented

It's showing memory related errors so that it something you can look into it.

Troubleshooting guide for Self hosted IR is here - https://docs.microsoft.com/en-us/azure/data-factory/self-hosted-integration-runtime-troubleshoot-guide

And you can set up the retry on Copy activity to rerun when it fails


Please don't forget to Accept Answer and Up-vote if the response helped -- Vaibhav

· 5
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Hi @chidave-5749,

We haven't heard back from you. Did @vaibhavchaudhari's response helped you or do you still require help on this issue ?

Thanks
Saurabh

0 Votes 0 ·

Hi @VaibhavChaudhari , @SaurabhSharma-msft ,

Thank you for the retry suggestion. This will help when the activity fails.

Regarding the memory issue, it is not clear to me how to exactly resolve this. The provided documentation doesn't give an elaborate explanation.

0 Votes 0 ·

@vaibhavchaudhari Any suggestions on this ?

Thanks
Saurabh

0 Votes 0 ·
Show more comments