ADF Copy Activity with Excel Dataset as a Source Fails Randomly due to File Size Error

Gaurav 1 Reputation point
2021-09-21T04:23:14.317+00:00

We have a copy data activity in one of our data ingestion flows, which uses Excel as the Source and Azure Synapse DW as the destination. This load runs multiple times during the day and we've noticed it failing sometimes with either of the following errors -
1.
133826-image.png

  1. 133853-image.png

The successful runs usually last anywhere between 5 min to 40 min, while the failures run for at least an hour before giving the error..The file is roughly 35.5 MB in size and the occurrence and the type of the error is very random. The execution utilizes the AutoResolve IR and we've observed the other parallel loads are not a reason for this. Many times, reprocessing the file right after the load's failed, gets completed with no issues. Are we missing something somewhere or are there any upper limits to the file size that can be handled by the Excel Dataset in the ADF?

Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
4,420 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,642 questions